Recording: Silence audio and video when producer is paused

[sebinbabu]

Greetings,
Mediasoup has helped me develop an app that lets me teach at my school. Thanks a lot for this!
I’ve to implement a record feature. Seems straightforward, there are examples around, and I’ve gone through them.
I’m trying to fill black video frames and silence audio to my recording when no RTP packets are received (that is when producer.pause() is called or producer just simply stops sending tracks).
I’m aware that back in 2017, @ibc too was looking for a solution to this - as mentioned here

While I know mediasoup is just an SFU that forwards what it receives, is there a way to send silence frames when a pause action is triggered (except for using producer.replaceTrack({ track: silenceTrack }) in the client)?

Furthermore, suppose the producer is closed and the plainTransport consuming the producer is not, is it possible to keep appending silence frames until a plainTransport.consume() is called on a different producer?

Has anyone been able to create such a Gstreamer/Ffmpeg pipeline that does this?

[jmillan (José Luis Millán Villegas)]

I believe that Gstreamer/FFmpeg pipeline will fill the time with no incoming RTP with silence and black image already. Did you already give it a try?

[ibc (Iñaki Baz Castillo)]

@sebinbabu

While I know mediasoup is just an SFU that forwards what it receives, is there a way to send silence frames when a pause action is triggered (except for using producer.replaceTrack({ track: silenceTrack }) in the client)?

Just don’t call producer.pause() in server side, so silence RTP frames sent by the client will be delivered to Consumers.

@sebinbabu

suppose the producer is closed and the plainTransport consuming the producer is not, is it possible to keep appending silence frames until a plainTransport.consume() is called on a different producer?

No. mediasoup does not generate fake RTP packets just to make recording systems happy. That’s something you must deal with in the recorder side (not an expert about that).

[nazar-pc (Nazar Mokynskyi)]

If you just pause without DTX and without zero RTP mode it is easier because browser will automatically send black rectangle for video and silence in audio. If you replace track with null or use zero RTP mode then what will most likely happen with video is that it will just freeze on last frame and probably the same for audio, but for audio I didn’t check it myself.

So you may want to pause it first (to generate black frame and silence) and then replace with null to prevent RTP traffic from flowing.

[sebinbabu]

@jmillan

I believe that Gstreamer/FFmpeg pipeline will fill the time with no incoming RTP with silence and black image already. Did you already give it a try?

Yes I did. While ffmpeg seems to get the actual timestamp offsets right (so there’s sudden gaps in final WEBM), gstreamer just saves the parts where the RTP was received (no gaps in final WEBM).

@ibc

Just don’t call producer.pause() in server side, so silence RTP frames sent by the client will be delivered to Consumers.

This worked, thanks!

@nazar-pc

So you may want to pause it first (to generate black frame and silence) and then replace with null to prevent RTP traffic from flowing.

I’m pausing the the producer for now, I’ll check the behavior when replaced with null.