Injecting AI-generated speech (TTS → FFmpeg → PlainTransport) — two gotchas worth knowing

Integration
1

I recently added an AI voice to an existing MediaSoup SFU — an LLM generates text,
Google Cloud TTS produces PCM, FFmpeg encodes to Opus RTP, and it lands on a
PlainTransport producer so viewers consume it like any other participant.

Two things cost me time that I didn’t find written down anywhere:

1. Keep one long-lived FFmpeg process, not one per utterance.

My first version spawned FFmpeg per TTS segment. Every spawn generates a new SSRC
and resets the RTP sequence number, so to the consumer’s jitter buffer each
utterance looks like a brand new stream — audible glitches and decoder resets at
every boundary. Keeping a single process alive and writing each PCM chunk into its
stdin gives continuous SSRC and sequence numbers, and the artifacts disappear.

The SSRC passed to FFmpeg also has to match the one declared in the producer’s
rtpParameters:

ffmpeg -re -f s16le -ar 48000 -ac 1 -i pipe:0 \
  -ac 2 -c:a libopus -b:a 128k -application voip \
  -frame_duration 20 -packet_loss 5 -vbr on \
  -f rtp -payload_type 111 -ssrc {same as producer} \
  rtp://127.0.0.1:{rtpPort}

2. Warm the pipeline with silence before the first utterance.

When FFmpeg first starts, the first real utterance can arrive with a hiccup while
the jitter buffer settles. Writing ~100ms of zeroed PCM in and waiting briefly
before speaking primes it — the viewer hears nothing and the first word arrives
cleanly.

Also worth noting for anyone piping Google TTS: it sometimes returns WAV-wrapped
PCM rather than raw, so a header check before writing to stdin saves confusion.

Full writeup with the transport setup, backpressure handling on stdin, and the
latency breakdown: https://medium.com/@sumitsinha1007/adding-an-ai-voice-to-a-mediasoup-sfu-notes-from-actually-doing-it-c6c9241a6b8b

Curious whether others doing server-side injection keep one process alive or
handle SSRC continuity some other way.

2

Yes, I’m doing something very similar. I discussed my approach in this post:

I also published the code as an npm package and on GitHub:

GitHub - Hilokal/audio-rtp-tools: Tools for consuming and publishing PCM audio data from an RTP stream · GitHub

The module is designed to integrate ergonomically with mediasoup.

Instead of spawning an FFmpeg process, I use the FFmpeg C API through Node-API. This has a few advantages:

  1. I rebase the outgoing RTP timestamps to account for the wall-clock gap between speech utterances. It has been a while since I tested this, but I seem to remember that the receiving browser would stutter if packets arrived with timestamps that were too far behind the current playout timeline. I don’t believe -re handles this, although I could be mistaken.
  2. I integrate with libopus directly rather than using FFmpeg’s libopus wrapper. This allows me to take advantage of the opus PLC features (really only useful if you’re decoding), and also take advantage of the resampler that’s built into libopus.
  3. If you are running many FFmpeg pipelines on one machine, using threads within a single process can reduce overhead compared with spawning a separate process for each stream. The threads share the same address space, and fewer buffer copies are required between Node.js and the media pipeline.

Your idea about sending 100ms of silence to prime the jitter buffer is interesting. I hadn’t thought of that. I’m going to give this a try.