Testing simulcast level switching

Hi,
I’m experimenting various settings for Simulcast. I found that in some cases, the spatial layer setting sent with setPreferredLayers is ignored. This always happens when the video source is a videoclip (not a real webcam source) sent as a fake system webcam, or using plain RTP transports.
I prepared a modified GStreamer broadcaster script (https://gist.github.com/vpalmisano/5716e8b52b258386fdd5c4fb93c97f11). You can test it using the well known Big Buck Bunny sequence (https://peach.blender.org/download/). In my local setup I found that level=0 is never sent, even if manually selected using setPreferredLayers.

Does level=0 mean spatial 0 or temporal 0?

BTW the proper way to diagnose this is by checking server side Producer's stats to see what the producer is currently sending. You’ll see an array of stats, one for each ongoing stream (if simulcast and if the sender is really sending N streams).

Spatial. I’m using only one temporal layer in the GStreamer script.

producer.getStats():
[
  {
    "bitrate": 53859,
    "byteCount": 1448769,
    "firCount": 0,
    "fractionLost": 0,
    "jitter": 0,
    "kind": "video",
    "mimeType": "video/VP8",
    "nackCount": 0,
    "nackPacketCount": 0,
    "packetCount": 6497,
    "packetsDiscarded": 0,
    "packetsLost": 0,
    "packetsRepaired": 0,
    "packetsRetransmitted": 0,
    "pliCount": 0,
    "score": 10,
    "ssrc": 2222,
    "timestamp": 1627638221,
    "type": "inbound-rtp"
  },
  {
    "bitrate": 139811,
    "byteCount": 3926095,
    "firCount": 0,
    "fractionLost": 0,
    "jitter": 0,
    "kind": "video",
    "mimeType": "video/VP8",
    "nackCount": 0,
    "nackPacketCount": 0,
    "packetCount": 6897,
    "packetsDiscarded": 0,
    "packetsLost": 0,
    "packetsRepaired": 0,
    "packetsRetransmitted": 0,
    "pliCount": 0,
    "score": 10,
    "ssrc": 2223,
    "timestamp": 1627638221,
    "type": "inbound-rtp"
  },
{
    "bitrate": 467568,
    "byteCount": 12412844,
    "firCount": 0,
    "fractionLost": 0,
    "jitter": 0,
    "kind": "video",
    "mimeType": "video/VP8",
    "nackCount": 0,
    "nackPacketCount": 0,
    "packetCount": 12025,
    "packetsDiscarded": 0,
    "packetsLost": 0,
    "packetsRepaired": 0,
    "packetsRetransmitted": 0,
    "pliCount": 1,
    "score": 10,
    "ssrc": 2224,
    "timestamp": 1627638221,
    "type": "inbound-rtp"
  }
]
producer.dump():
{
  "id": "f4b3e65a-45f5-4093-8c1e-4aa39629263c",
  "kind": "video",
  "paused": false,
  "rtpMapping": {
    "codecs": [
      {
        "mappedPayloadType": 101,
        "payloadType": 101
      }
    ],
    "encodings": [
      {
        "mappedSsrc": 225588197,
        "rid": null,
        "ssrc": 2222
      },
      {
        "mappedSsrc": 225588198,
        "rid": null,
        "ssrc": 2223
      },
      {
        "mappedSsrc": 225588199,
        "rid": null,
        "ssrc": 2224
      }
    ]
  },
  "rtpParameters": {
    "codecs": [
      {
        "clockRate": 90000,
        "mimeType": "video/VP8",
        "parameters": {},
        "payloadType": 101,
        "rtcpFeedback": [
          {
            "parameter": "fir",
            "type": "ccm"
          },
          {
            "type": "nack"
          },
          {
            "parameter": "pli",
            "type": "nack"
          }
        ]
      }
    ],
    "encodings": [
      {
        "codecPayloadType": 101,
        "ksvc": false,
        "maxBitrate": 512000,
        "scalabilityMode": "S1T1",
        "spatialLayers": 1,
        "ssrc": 2222,
        "temporalLayers": 1
      },
      {
        "codecPayloadType": 101,
        "ksvc": false,
        "maxBitrate": 1024000,
        "scalabilityMode": "S1T1",
        "spatialLayers": 1,
        "ssrc": 2223,
        "temporalLayers": 1
      },
      {
        "codecPayloadType": 101,
        "ksvc": false,
        "maxBitrate": 2048000,
        "scalabilityMode": "S1T1",
        "spatialLayers": 1,
        "ssrc": 2224,
        "temporalLayers": 1
      }
    ],
    "headerExtensions": [],
    "rtcp": {
      "cname": "5ab6b651",
      "reducedSize": true
    }
  },
  "rtpStreams": [
    {
      "params": {
        "clockRate": 90000,
        "cname": "5ab6b651",
        "mimeType": "video/VP8",
        "payloadType": 101,
        "spatialLayers": 1,
        "ssrc": 2223,
        "temporalLayers": 1,
        "useDtx": false,
        "useFir": true,
        "useInBandFec": false,
        "useNack": true,
        "usePli": true
      },
      "score": 10
    },
    {
      "params": {
        "clockRate": 90000,
        "cname": "5ab6b651",
        "mimeType": "video/VP8",
        "payloadType": 101,
        "spatialLayers": 1,
        "ssrc": 2224,
        "temporalLayers": 1,
        "useDtx": false,
        "useFir": true,
        "useInBandFec": false,
        "useNack": true,
        "usePli": true
      },
      "score": 10
    }
  ],
  "traceEventTypes": "",
  "type": "simulcast"
}

consumer.setPreferredLayers() will request a keyframe for the selected spatial layer (simulcast stream in this case) and become effective once such a keyframe is received in mediasoup, so I think that’s not happening.

You can use produce.enableTrace() to enable keyframes in the producer and check it. Documented in the Debugging section of mediasoup.

I modified the GStreamer script to produce 1 keyframe per second, nothing changed.

I will try now.

Using the browser as client and a fake webcam video source, I found that only 2 (over a total of 3) pli trace events are captured when I try to change levels. In the case of regular webcam source, I got 3 pli events, and the layer is always changed. This happens only when the video is generated, maybe a browser problem?

This is the script I’m using to start a fake video source (on Linux): https://gist.github.com/vpalmisano/17c3647e497629da5ec4aa9c15410a42

Edit:
Using VP9, I get always 1 pli event (using SVC), but spatial layer is never changed as requested.

mediasoup sends a PLI for the ssrc of the stream for which it needs a keyframe. This’s been working fine for years. You can also use the trace event to log PLI requests sent by mediasoup to the sender:

In VP9 there is a single SSRC so a single stream.

Found the problem: the fake webcam source resolution should be at least 1280x720 in order to make simulcast work in a 3 levels configuration, with [4, 2, 1] scaling. It may be the same if you have a webcam with resolution < 720p.

Nice. But the the missing spatial layer (the one you could not receive) was 2 and not 0.

Level = 0 for me was the one with lowest resolution.

Exactly, so that’s always sent, but you said it wasn’t. Lowest one must be the first in the encodings array given to the server side Producer BTW

I think that when the fake webcam source sends a resolution that is less than the resolution requested by getUserMedia, the lowest level is not sent at all…

A question: do you know if there are some max/min limitation to the configurable parameters for simulcast? For instance, can I set scaleResolutionDownBy=16, or maxBitrate=10 ?

There are ugly issues in libwebrtc if you specify encodings with too low maxBitrate, such as having zero RTP as result. I reported one of them to libwebrtc. So caution.

No I don’t know these value limits.

1 Like