web client can not receive video when comsume prefer spatial layer 2 while produce setMaxSpatialLayer 1 or 0
Your environment
Operating system: ubuntu1~16.04.12
Node version: v13.14.0
npm version:
gcc/clang version:
mediasoup version : 3.6.24
mediasoup-client version: 3.6.21
Issue description
1 web client can not receive video when comsume prefer spatial layer 2 while produce setMaxSpatialLayer 1 or 0
2 the mediasoup log show:
mediasoup:Channel [pid:2432] (trace) RTC::SimulcastConsumer::UpdateTargetLayers() +0ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:6edcf06e-01a4-4837-b3a2-3a601903c81c] +0ms
The normal log should be :
mediasoup:Channel [pid:2432] (trace) RTC::SimulcastConsumer::UpdateTargetLayers() +1ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | using spatial layer 1 as RTP timestamp reference +0ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:1, temporal:0, consumerId:681d18b2-47c3-4e64-85c9-31e4afe47d70] +0ms
3 the video will not be recovered forever when this happen.
The reasons may be that the condition is not satisfied:
“RTP streams of our TS reference spatial layer and the given spatial layer, have Sender Report” on
inline bool SimulcastConsumer::CanSwitchToSpatialLayer(int16_t spatialLayer) const
The scenario is that,
The first peer join (no other peer see the first peer) the first peer shoul send spatial layer 0 or pauss.
If all other peers prefer one peer’s spatial layer 1, the peer should send max spatial layer 1.
If nobody in the room see the peer, it also should send spatial layer 0 or pauss.
If one prefer the peer’s spatial layer 2, it should send max spatial layer 2.
We have recently discovered an issue, which I think also explains the report above.
The issue is that, in SimulcastConsumer, if tsReferenceSpatialLayer is set, and no RTCP Server Reports are ever received for that spatial layer, then it is not possible to ever switch layers away from the selected reference spatial layer. This can happen, for example, if the client sets the max spatial layer to a lower value immediately after sending the first RTP packets.
We are testing a fix which waits until after a Sender Report is received before setting tsReferenceSpatialLayer (so some related logic pieces need to change to handle things differently at the beginning).
We will be happy to contribute/submit a patch once we have tested things on our end.
The problem happens if the higher spatial layer is never resumed. For example:
Start with 3 spatial layers
SimulcastConsumer will select layer 2 for tsReferenceSpatialLayer and currentSpatialLayer. (I believe this is only since the change in 3.6.17 (SimulcastConsumer: Prefer the highest spatial layer initially (PR #450).)
Client sets max spatial layer to 0 before any Sender Reports are sent. (And leaves it at this forever).
SimulcastConsumer will eventually notice layer 2 as inactive and set currentSpatialLayer to -1, but the logic in CanSwitchToSpatialLayer will not allow any other layer to be selected since GetProducerTsReferenceRtpStream()->GetSenderReportNtpMs() will remain false.