web client can not receive video when comsume prefer spatial layer 2 while produce setMaxSpatialLayer 1 or 0

mediasoup libraries
1

Bug Report

web client can not receive video when comsume prefer spatial layer 2 while produce setMaxSpatialLayer 1 or 0

Your environment

  • Operating system: ubuntu1~16.04.12
  • Node version: v13.14.0
  • npm version:
  • gcc/clang version:
  • mediasoup version : 3.6.24
  • mediasoup-client version: 3.6.21

Issue description

1 web client can not receive video when comsume prefer spatial layer 2 while produce setMaxSpatialLayer 1 or 0
2 the mediasoup log show:
mediasoup:Channel [pid:2432] (trace) RTC::SimulcastConsumer::UpdateTargetLayers() +0ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:6edcf06e-01a4-4837-b3a2-3a601903c81c] +0ms

The normal log should be :
mediasoup:Channel [pid:2432] (trace) RTC::SimulcastConsumer::UpdateTargetLayers() +1ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | using spatial layer 1 as RTP timestamp reference +0ms
mediasoup:Channel [pid:2432] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:1, temporal:0, consumerId:681d18b2-47c3-4e64-85c9-31e4afe47d70] +0ms

3 the video will not be recovered forever when this happen.

The reasons may be that the condition is not satisfied:
“RTP streams of our TS reference spatial layer and the given spatial layer, have Sender Report” on
inline bool SimulcastConsumer::CanSwitchToSpatialLayer(int16_t spatialLayer) const

2

Please paste more log lines before…

RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:6edcf06e-01a4-4837-b3a2-3a601903c81c] +0ms

…so we can have more context.

What’s the scenario? Are you limiting the spatial layers since the very beginning?

3

The scenario is that,
The first peer join (no other peer see the first peer) the first peer shoul send spatial layer 0 or pauss.
If all other peers prefer one peer’s spatial layer 1, the peer should send max spatial layer 1.
If nobody in the room see the peer, it also should send spatial layer 0 or pauss.
If one prefer the peer’s spatial layer 2, it should send max spatial layer 2.

4

Also provide the information we have asked for.

5

Hi,

We have recently discovered an issue, which I think also explains the report above.
The issue is that, in SimulcastConsumer, if tsReferenceSpatialLayer is set, and no RTCP Server Reports are ever received for that spatial layer, then it is not possible to ever switch layers away from the selected reference spatial layer. This can happen, for example, if the client sets the max spatial layer to a lower value immediately after sending the first RTP packets.
We are testing a fix which waits until after a Sender Report is received before setting tsReferenceSpatialLayer (so some related logic pieces need to change to handle things differently at the beginning).
We will be happy to contribute/submit a patch once we have tested things on our end.

Best,
Lewis

6

I don’t understand the problem. Sender Reports will be received eventually for the stopped spatial layer (stream) if resumed later.

Can you please elaborate the issue a bit more?

7

The problem happens if the higher spatial layer is never resumed. For example:

  • Start with 3 spatial layers
  • SimulcastConsumer will select layer 2 for tsReferenceSpatialLayer and currentSpatialLayer. (I believe this is only since the change in 3.6.17 (SimulcastConsumer: Prefer the highest spatial layer initially (PR #450).)
  • Client sets max spatial layer to 0 before any Sender Reports are sent. (And leaves it at this forever).
  • SimulcastConsumer will eventually notice layer 2 as inactive and set currentSpatialLayer to -1, but the logic in CanSwitchToSpatialLayer will not allow any other layer to be selected since GetProducerTsReferenceRtpStream()->GetSenderReportNtpMs() will remain false.
8

Got it, can you please report this in GitHub (with all collected data above) now that we know it’s a real bug? We know how to fix it. Thanks.

9

Friendly ping. Are you gonna create the GH issue for this, please?

10

Sorry for the delay - busy with other things. It is created now:

11

Thanks