Hello,
We tried several times to deploy H264 simulcast in a real world scenario with a few hundred viewers, as instructed (rtcp feedback, transport-cc, etc.). Every time the results were disappointing due to inaccurate Bandwidth Estimation (BWE), leading to oscillating quality even on perfect fiber connections. Sometimes it would black out or freeze.
Iñaki @ibc reported that Mediasoup was dependent on libwebrtc BWE and it was a black box.
Lorenzo wrote an in-depth blog on BWE. Same observation that libwebrtc’s BWE is sophisticated without solving BWE properly.
Let’s qualify this setup as “Server-side controlled simulcast” as the SFU decides which layer to send according to BWE.
Another approach, like HLS, is for the client to select the simulcast layer according to the data it receives.
We implemented this client-side approach by using WebRTC stats, specifically using framesDecoded and bytesReceived on inbound-rtp stats.
When the number of framesDecoded per second drops under a threshold (e.g. 5 fps) the client switches to the lower resolution.
Conversely we establish a data connection and send an increasing number of small packets to check whether the bandwidth is large enough to switch back to a higher resolution.
The result is stable, remains on the best resolution for good connections, and recovers well on fluctuations.