Injecting AI-generated speech (TTS → FFmpeg → PlainTransport) — two gotchas worth knowing

[jbaudanza (Jonathan Baudanza)]

Yes, I’m doing something very similar. I discussed my approach in this post:

Module for sending/receiving PCM data from PlainTransport

This is a simple node module for connecting to a PlainTransport and encoding/decoding Opus packets to/from PCM. It’s particularly helpful if you’re integrating with an AI model like a chatbot or transcription model, but it can be used for anything. I’ve been using this in production for a while. Recently, I figured I’d clean it up and publish it. Let me know if you find this useful or have any feedback. It depends on ffmpeg and libopus.

I also published the code as an npm package and on GitHub:

GitHub - Hilokal/audio-rtp-tools: Tools for consuming and publishing PCM audio data from an RTP stream · GitHub

The module is designed to integrate ergonomically with mediasoup.

Instead of spawning an FFmpeg process, I use the FFmpeg C API through Node-API. This has a few advantages:

1. I rebase the outgoing RTP timestamps to account for the wall-clock gap between speech utterances. It has been a while since I tested this, but I seem to remember that the receiving browser would stutter if packets arrived with timestamps that were too far behind the current playout timeline. I don’t believe -re handles this, although I could be mistaken.
2. I integrate with libopus directly rather than using FFmpeg’s libopus wrapper. This allows me to take advantage of the opus PLC features (really only useful if you’re decoding), and also take advantage of the resampler that’s built into libopus.
3. If you are running many FFmpeg pipelines on one machine, using threads within a single process can reduce overhead compared with spawning a separate process for each stream. The threads share the same address space, and fewer buffer copies are required between Node.js and the media pipeline.

Your idea about sending 100ms of silence to prime the jitter buffer is interesting. I hadn’t thought of that. I’m going to give this a try.