How to stream audio chunks from the browser to the server?

Questions & Help Questions
, , , , ,
1

Hi all,

I have the following apps:

  • an application server that host my Elixir, Phoenix app hosted on Fly.io
  • an AI server hosting OpenAI Whisper Large V3 on a dedicated HuggingFace endpoint

In the app, users can record audio and have it transcribed.

I would like that the user’s browser sends chunks of audio data of a configurable duration (for ex. 5 seconds each – later I would like that each chunks overlaps each other based of a configurable duration, for ex. 2 seconds). The idea is that both servers (app and ai) never holds in memory the entirety of the audio + speed up inference by sending smaller audio files.

I know that there are some examples of Phoenix Liveview with Bumblebee (see this one), however:

  1. I’m not using Bumblebee
  2. Those examples send the entirety of the data to the AI model

I’ve tried the following in my microphone.js hook:

const SAMPLING_RATE = 16_000;
const CHUNK_DURATION_IN_MS = 5_000;
const CHUNK_OVERLAP_DURATION_IN_MS = 2_000;
const MIME_TYPE = "audio/ogg;codecs=opus";

const MicrophoneV2 = {
  mounted() {
    this.mediaRecorder = null;
    this.recording = false;

    this.el.addEventListener("click", () => {
      if (this.isRecording()) {
        this.stopRecording();
      } else {
        this.startRecording();
      }
    });
  },

  startRecording() {
    this.audioChunks = [];

    navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
      this.mediaRecorder = new MediaRecorder(stream, { mimeType: MIME_TYPE });

      this.mediaRecorder.addEventListener("dataavailable", (event) => {
        if (event.data.size > 0) {
          this.audioChunks.push(event.data);
          this.processChunks();
        }
      });

      this.mediaRecorder.start(CHUNK_DURATION_IN_MS);

      this.updateInterval = setInterval(() => {
        this.mediaRecorder.requestData();
      }, CHUNK_DURATION_IN_MS);
    });
  },

  stopRecording() {
    this.mediaRecorder.addEventListener("stop", () => {
      this.processChunks();
    });

    this.mediaRecorder.stop();
    clearInterval(this.updateInterval);
  },

  processChunks() {
    if (this.audioChunks.length < 1) return;

    const audioBlob = new Blob(this.audioChunks, { type: MIME_TYPE });

    audioBlob
      .arrayBuffer()
      .then((buffer) => {
        const context = new AudioContext({ sampleRate: SAMPLING_RATE });

        context.decodeAudioData(buffer, (audioBuffer) => {
          const pcmBuffer = this.audioBufferToPcm(audioBuffer);
          const buffer = this.convertEndianness32(
            pcmBuffer,
            this.getEndianness(),
            this.el.dataset.endianness
          );
          this.upload("audio", [new Blob([buffer])]);
        });
      })
      .then(() => {
        this.audioChunks = [];
      })
      .catch((error) => {
        console.error("Error decoding audio data", error);
      });
  },

  isRecording() {
    return this.mediaRecorder && this.mediaRecorder.state === "recording";
  },

  audioBufferToPcm(audioBuffer) {
    const numChannels = audioBuffer.numberOfChannels;
    const length = audioBuffer.length;
    const size = Float32Array.BYTES_PER_ELEMENT * length;
    const buffer = new ArrayBuffer(size);
    const pcmArray = new Float32Array(buffer);
    const channelDataBuffers = Array.from(
      { length: numChannels },
      (x, channel) => audioBuffer.getChannelData(channel)
    );

    // Average all channels upfront, so the PCM is always mono
    for (let i = 0; i < pcmArray.length; i++) {
      let sum = 0;

      for (let channel = 0; channel < numChannels; channel++) {
        sum += channelDataBuffers[channel][i];
      }

      pcmArray[i] = sum / numChannels;
    }

    return buffer;
  },

  convertEndianness32(buffer, from, to) {
    if (from === to) return buffer;

    // If the endianness differs, we swap bytes accordingly
    for (let i = 0; i < buffer.byteLength / 4; i++) {
      const b1 = buffer[i];
      const b2 = buffer[i + 1];
      const b3 = buffer[i + 2];
      const b4 = buffer[i + 3];

      buffer[i] = b4;
      buffer[i + 1] = b3;
      buffer[i + 2] = b2;
      buffer[i + 3] = b1;
    }

    return buffer;
  },

  getEndianness() {
    const buffer = new ArrayBuffer(2);
    const int16Array = new Uint16Array(buffer);
    const int8Array = new Uint8Array(buffer);

    int16Array[0] = 1;

    if (int8Array[0] === 1) {
      return "little";
    } else {
      return "big";
    }
  },
};

export { MicrophoneV2 };

The problem is in the processChunks method, when I empty the audioChunks like so:

        this.audioChunks = [];

The audioContext.decodeAudioData doesn’t work on partial content (seen here); it only works with the entirety of the audio:

Uncaught (in promise) DOMException: The buffer passed to decodeAudioData contains invalid content which cannot be decoded successfully.

I’ve attempted this but to no avail and I would like to avoid complicated binary handling.

How to send a bunch of audio chunks from the browser to the Phoenix server?