Simulcast behavior with Firefox producer

mediasoup libraries
1

I am looking at the simple simulcast scenario: VP8, two spatial layers set as:

encodings: [
 { scaleResolutionDownBy: 4 },
 { scaleResolutionDownBy: 1 }
]

If the producer is in Chrome, everything works correctly. However, if the producer is in Firefox, the video is tending to freeze. The log looks like this (I added output from IncreaseLayer and ApplyLayers):

mediasoup:Channel request succeeded [method:consumer.resume, id:18] +0ms
mediasoup:Channel [pid:2227] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:385770193, seq:512] +10ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::SendRtpPacket() | sync key frame received +1ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::SendRtpPacket() | sending sync packet [ssrc:695875555, seq:1, ts:2257638956] from original [ssrc:961059511, seq:512, ts:2257638956] +0ms
mediasoup:Channel [pid:2227] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:3082833855, seq:22146] +83ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ProducerRtcpSenderReport() | first SenderReport [ssrc:385770193] +846ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ProducerRtcpSenderReport() | first SenderReport [ssrc:3082833855] +57ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 0:0 [virtual bitrate:600000, required bitrate:244550] +1ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 0:1 [virtual bitrate:355450, required bitrate:43005] +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 0:2 [virtual bitrate:312445, required bitrate:223846] +1ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 1:0 [virtual bitrate:88599, required bitrate:1] +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 1:1 [virtual bitrate:88598, required bitrate:2368] +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::IncreaseLayer() | setting provisional layers to 1:2 [virtual bitrate:86230, required bitrate:9837] +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:2d636d47-3681-4581-bd27-1ae9b3f11ac1] +0ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:737112356, media ssrc:695875555] +968ms
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +1s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s
mediasoup:Channel [pid:2227] RTC::SimulcastConsumer::ApplyLayers() | +2s

I wonder if the sequence in bold is something normally expected? If I prevent UpdateTargetLayers from setting layers to -1 at this moment, then the video does not freeze.

2

Hunted this down to the reversed encodings order (#152)

1 Like
3

We have a fix now, great. But what about the original effect? If the ApplyLayers method is called twice in sequence, without setting some provisional layers in between, then the second call sets the target layers to -1, preventing any producer’s streams from going to the consumer. I didn’t look too deeply in the algorithm and was not sure whether this could happened under normal circumstances or was caused by a combination of the error with the encodings order and my setup, but why was it allowed anyway? Isn’t it better to take a precaution and explicitly prevent ApplyLayers from setting target layers to -1? When I did so, the video at least continued to run, though spatial layers were mixed up.

4

Why should we prevent that? If there is no available bandwidth even for the lowest layer then it must select -1 so no video is sent at all.

5

Ok, I got it now.