Chrome and VP9 SVC support

We have been experimenting with VP9 SVC and can reproduce issues we are having in the mediasoup demo app. For screen sharing, what we would like to do is to provide multiple temporal layers with scalability mode L1T3 or higher temporal layers, such that frame rate is reduced rather than the user experiencing quality or spacial reductions in bandwidth limited situations. We have been experimenting with Chrome 80 and 84 canary and find that we are only ever seeing temporal layer 0 being forwarded by the SFU. We are seeing that by looking at the consumer remote stats on the client side, or looking at the consumer stats in the server side CLI. Its not clear that Chrome is producing more than just one layer, but on the client side we have verified the encodings going to the peerconnection are correct. We are using --force-fieldtrials=WebRTC-SupportVP9SVC/EnabledByFlag_1SL3TL/ on the command line. If we add spacial layers such as with L3T3, we see movement between spacial layers, but never see bandwidth on any temporal layers beyond 0.

Digging into the worker code a bit I find this bit of code in SvcConsumer::IncreaseLayer, around line 401 of SvcConsumer.cpp

                        temporalLayer = 0;

                        // Check bitrate of every temporal layer.
                        for (; temporalLayer < this->producerRtpStream->GetTemporalLayers(); ++temporalLayer)
                                // Ignore temporal layers lower than the one we already have (taking into account
                                // the spatial layer too).
                                // clang-format off
                                if (
                                        spatialLayer == this->provisionalTargetSpatialLayer &&
                                        temporalLayer <= this->provisionalTargetTemporalLayer
                                // clang-format on

                                requiredBitrate =
                                  this->producerRtpStream->GetLayerBitrate(nowMs, spatialLayer, temporalLayer);

                                  "testing layers %" PRIi16 ":%" PRIi16 " [virtual bitrate:%" PRIu32
                                  ", required bitrate:%" PRIu32 "]",

                                // If active layer, end iterations here. Otherwise move to next spatial layer.
                                if (requiredBitrate)
                                      goto done;

Two observations. The first is that the only layer that returns a non-zero requiredBitrate from the producerRtpStream is layer 0. Maybe the bug is in that code, or maybe Chrome isn’t sending data on those layers? The second is that it seems this loop will either jump to done or break out of this loop, so there isn’t any way to ever choose anything but temporalLayer 0 in the case that spatialLayer != this->provisionalTargetSpatialLayer. From my testing, provisional layers are -1:-1 in the loop. If I remove the bottom if condition, I see it cycle through all the layers, but I don’t get a working stream. I smell a bug in here… Any thoughts on a fix? Thanks for your help!

I’m pretty sure there is no bug in that code.

BTW: We already enable VP9 SVC in mediasoup-client without having to run Chrome with a flag. You may need such a flag to limit the number of spatial or temporal layers. Anyway, this is not Chrome public API and we cannot give much more support on it (things may change tomorrow).

BTW just run the demo with VP9 (running Chrome without any flag):

  • Open webcam, and check “Video Producer Remote Stats”. You will see 3 spatial with 3 temporal layers each one.

  • Start screen sharing and check “Video Producer Remote Stats”. You will see 3 spatial layers with just a single temporal layer each. No idea why, this is chrome stuff, but this is what mediasoup is receiving.

Also, remember that in order to use VP9 you have to call sendTransport.produce() with a single encoding with scalabilityMode.

  • For webcam video it must be ‘S3T3_KEY’.
  • For sharing it must be ‘S3T3’ and you must also set dtx: true in the encoding. This is explained somewhere in the “RTP Parameters” section of the doc.
  • Those are the values that match default VP9 settings in Chrome.
  • If you pass a command lien flag changing that, you should reflect those values in scalabilityMode string.

So, to clarify, instead of looking at receiving side (consumer code in mediasoup C++) look at the Producer stats. Those will tell you what Chrome is sending.

Hey @ibc.
do you have any idea if Simulcast supports VP9 nowadays?
i came across this link which claims its in M113, there is a demo attached too.

Ticket created: Must join all XxxConsumer classes into a single one · Issue #1359 · versatica/mediasoup · GitHub