Confusion with non-first codec failing with simulcast

mediasoup libraries
1

Hi there!

I’ve just found a new situation with mediasoup and have a bit of a suggestion to improve error messages, also a question about what’s the best way to proceed when several codecs are offered by a WebRTC agent.

This is a simulcast Producer that gets several codecs negotiated with Chrome, and Chrome always choses to send the very first codec so everything went well so far. But now I’m dealing with a WebRTC tool that not always sends the first codec, and the issue was hard to debug.

Here’s a look at the RtpParameters passed to the Producer:

{
  "mid": "0",
  "codecs": [
    { "mimeType": "video/VP8", "payloadType": 96, (...) },
    { "mimeType": "video/H264", "payloadType": 98, (...) },
    { "mimeType": "video/H264", "payloadType": 100, (...) },
    { "mimeType": "video/H264", "payloadType": 102, (...) }
  ],
  "encodings": [
    { "rid": "div4" },
    { "rid": "div2" },
    { "rid": "div1" }
  ]
}

Problem is, if the tool connects and decides to send H.264 PT 100 (for example), mediasoup will reject all incoming RTP packets with these messages:

ignoring packet with unknown RID (RID lookup)
no stream found for received packet [ssrc:11223344]

This happens because there is a check

isMediaPacket = (mediaCodec->payloadType == payloadType)

that fails (src), given that mediaCodec->payloadType == 96 (always set to the very first one of the given codecs) but in our example case, payloadType == 100.

As you’ll understand, these error messages were extremely confusing because they don’t really communicate the reason for them happening. So I’d like to suggest 2 changes:

  • Reword them to hint better about what’s the problem. The RID lookup is alright, it’s the PayloadType lookup that’s failing.
  • Print on the first message the RID string and the PT that are causing issues.

Each of the encodings have an undocumented field codecPayloadType (src) not mentioned in either API or RTP Parameters and Capabilities docs. When unset, mediasoup implicitly sets it to the PT of the very first media codec.

This was very surprising, in a bad way (principle of least surprise and all that). I guess most people just leave codecPayloadType unset, so any mediasoup app that doesn’t set it, is risking malfunction when a WebRTC agent decides to send a codec which was not the first one.

What is the “correct” way to use this field? Should we all be always creating N copies of each encoding, one for each possible N PayloadTypes?

  • My suggestion here is obviously to kindly add some docs.

But going a bit further, I wonder if it wouldn’t make even more sense to get rid of RtpEncodingParameters.codecPayloadType altogether. I believe with standard WebRTC it’s not possible to declare different encodings for different codecs (e.g. you wouldn’t be able to negotiate VP8 with 3 simulcast layers, and H.264 separately with 2 layers, all for the same media section, aka. same mediasoup Producer). So maybe removing it makes sense (or I probably lack more extended context about the feature).

Kind regards

2

I cannot answer to all questions/topics above now because I’m OOO with poor availability, but I want to make it clear that you cannot pass more than 1 codec to transport.produce() in mediasoup side. You can pass 1 video codec + its associated RTX codec, nothing else. Room for improvement here? Maybe, but we didn’t find any real use case so far. When you call transport.produce() you are supposed to pass the exact codec that mediasoup will receive. RTP parameters given to produce() are not like in SDP land where you announce the remote party “everything you can send or receive”.

3

Well, that’s a great insight! It also explains a lot of the behavior. Definitely my suggestions still apply in case other people find themselves in same situation, and now that you’ve clarified this, it’s something that would also be important to explain in docs (and even maybe enforce by code).

Anyway I now have some useful info, thanks for taking the time to reply!

4

Hello , maybe we can support dynamic codec change. For example , user A pub AV1 by default , user B don’t support AV1 , when B join room , I can dynamic change A’s codec to VP9 . It’s done on Google Meet now.

5

We are not ready for that. It will be done in a future refactor.

6

When you call transport.produce() you are supposed to pass the exact codec that mediasoup will receive.

Unfortunately I somehow missed that requirement and I can’t find it in documentation (I might be looking into wrong place). So I’d spend some time to debug the mediasoup and eventually found this thread.

Maybe, but we didn’t find any real use case so far.

The one I had tried to implement: I have native app which supposed to negotiate H264 and VP8 codec with mediasoup and preferably use H264 encoder and if it misbehaves it fallback to VP8 (hello to some of the Android devices). Until I’ve discovered that thread I assumed that it supposed to work without re-negotiation with mediasoup.
But the requirement to provide only single codec to produce() method make it clear what I’ve tried to do unfeasible.

Another use case is to switch codecs dynamically without re-negotiation when some conditions are met like available bitrate via VideoEncoderSelector interface: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/sdk/android/api/org/webrtc/VideoEncoderFactory.java;l=17?q=VideoEncoderSelector&sq=

PS. Does it make sense to make a PR which will add validation to the produce() API to fail early? (until proper handling of multiple codecs in produce is implemented).

7

I don’t know who said this and where but that’s not true at all.

8

I understood this as requirement to pass single video codec into produce(), am I wrong? I’m using mediasoup directly without wrapper like mediasoup-client.

9

Ok now I understand. It was not about mediasup-client. Yes, transport.produce() in mediasoup server must be called with only one codec + optional RTX codec.

1 Like