Why do we need to consume videos as paused?

mediasoup libraries
1

Some times consuming a video fails (video is not loaded) if I consume the video as paused = false. Do we really have to send an extra request to resume the video?

And how about audio?

2

The demo app creates the video Consumer with paused: true as an optimization to wait for the Consumer to be also created in the browser app before mediasoup requests a keyframe to the corresponding Producer.

Some times consuming a video fails (video is not loaded) if I consume the video as paused = false .

It should not fail. However the consumer browser may have to wait a bit more for a second keyframe. I expect this is what you see sometimes.

3

Thanks for the explanation :ok_hand:

4

Anyway, even if you don’t start the video consumer with paused: true, it should work (with some potential delay in video rendering). Doesn’t it?

5

Not sure, maybe it is some thing else like simulcast. Need to do more tests.

6

I’m seeing something that might be related.

For me, if I don’t create video Consumers with paused: true, the second client-side Consumer that is created in each client always receives a track that can’t play. (The play() call hangs forever. If the video element’s srcObject is changed, after attempting to play, the browser throws a DOMException.)

The first Consumer, and every Consumer after the second Consumer, work fine. If I follow the same practice as the demo and create the Consumer as paused, initially, then everything (including the second client-side Consumer that is created) works fine.

I just tracked this down and it’s reproducible for me on all browsers. I can create a minimal test case, if that’s helpful.

7

I can’t figure out nor why video.play() hangs forever neither why changing its srcObject throws. Anyway, yes, a minimal test case would help.

Said that, the proper way of creating a video consumer is by doing it in paused mode until its parameters are transmitted to the browser and the browser side consumer is created (as the demo does).

8

Upps, it makes sense that the second consumer fails:

  • When the first consumer is created, the WebRTC transport is not yet connected (it’s done upon creation of the first consumer in mediasoup-client) so mediasoup does not yet request a keyframe to the producer.
  • Once ICE+DTLS is connected, mediasoup will request a keyframe to the producer and that keyframe will properly reach the mediasoup-client consumer (which is already ready to process it).
  • However when a second consumer is created in the server (with paused: false) the transport is already connected so mediasoup will immediately request a keyframe to the producer, and that keyframe may reach the consuming mediasoup-client before its local side consumer is created.
  • If so, the browser will drop/ignore such a keyframe and, once the local consumer is created, it will receive RTP packets that cannot render (keyframe is missing). Theoretically the PeerConnection should send a PLI or FIR to request a keyframe.

BTW I’ve improved the doc about this: https://mediasoup.org/documentation/v3/mediasoup/api/#ConsumerOptions