Confusion with non-first codec failing with simulcast

mediasoup libraries
1

Hi there!

I’ve just found a new situation with mediasoup and have a bit of a suggestion to improve error messages, also a question about what’s the best way to proceed when several codecs are offered by a WebRTC agent.

This is a simulcast Producer that gets several codecs negotiated with Chrome, and Chrome always choses to send the very first codec so everything went well so far. But now I’m dealing with a WebRTC tool that not always sends the first codec, and the issue was hard to debug.

Here’s a look at the RtpParameters passed to the Producer:

{
  "mid": "0",
  "codecs": [
    { "mimeType": "video/VP8", "payloadType": 96, (...) },
    { "mimeType": "video/H264", "payloadType": 98, (...) },
    { "mimeType": "video/H264", "payloadType": 100, (...) },
    { "mimeType": "video/H264", "payloadType": 102, (...) }
  ],
  "encodings": [
    { "rid": "div4" },
    { "rid": "div2" },
    { "rid": "div1" }
  ]
}

Problem is, if the tool connects and decides to send H.264 PT 100 (for example), mediasoup will reject all incoming RTP packets with these messages:

ignoring packet with unknown RID (RID lookup)
no stream found for received packet [ssrc:11223344]

This happens because there is a check

isMediaPacket = (mediaCodec->payloadType == payloadType)

that fails (src), given that mediaCodec->payloadType == 96 (always set to the very first one of the given codecs) but in our example case, payloadType == 100.

As you’ll understand, these error messages were extremely confusing because they don’t really communicate the reason for them happening. So I’d like to suggest 2 changes:

  • Reword them to hint better about what’s the problem. The RID lookup is alright, it’s the PayloadType lookup that’s failing.
  • Print on the first message the RID string and the PT that are causing issues.

Each of the encodings have an undocumented field codecPayloadType (src) not mentioned in either API or RTP Parameters and Capabilities docs. When unset, mediasoup implicitly sets it to the PT of the very first media codec.

This was very surprising, in a bad way (principle of least surprise and all that). I guess most people just leave codecPayloadType unset, so any mediasoup app that doesn’t set it, is risking malfunction when a WebRTC agent decides to send a codec which was not the first one.

What is the “correct” way to use this field? Should we all be always creating N copies of each encoding, one for each possible N PayloadTypes?

  • My suggestion here is obviously to kindly add some docs.

But going a bit further, I wonder if it wouldn’t make even more sense to get rid of RtpEncodingParameters.codecPayloadType altogether. I believe with standard WebRTC it’s not possible to declare different encodings for different codecs (e.g. you wouldn’t be able to negotiate VP8 with 3 simulcast layers, and H.264 separately with 2 layers, all for the same media section, aka. same mediasoup Producer). So maybe removing it makes sense (or I probably lack more extended context about the feature).

Kind regards

2

I cannot answer to all questions/topics above now because I’m OOO with poor availability, but I want to make it clear that you cannot pass more than 1 codec to transport.produce() in mediasoup side. You can pass 1 video codec + its associated RTX codec, nothing else. Room for improvement here? Maybe, but we didn’t find any real use case so far. When you call transport.produce() you are supposed to pass the exact codec that mediasoup will receive. RTP parameters given to produce() are not like in SDP land where you announce the remote party “everything you can send or receive”.

3

Well, that’s a great insight! It also explains a lot of the behavior. Definitely my suggestions still apply in case other people find themselves in same situation, and now that you’ve clarified this, it’s something that would also be important to explain in docs (and even maybe enforce by code).

Anyway I now have some useful info, thanks for taking the time to reply!

4

Hello , maybe we can support dynamic codec change. For example , user A pub AV1 by default , user B don’t support AV1 , when B join room , I can dynamic change A’s codec to VP9 . It’s done on Google Meet now.

5

We are not ready for that. It will be done in a future refactor.