Specify payloadType for consumer

mediasoup libraries
1

In order to simplify negotiation, I would like to always match the client’s preferred payload types. In principle, this should be easy, mediasoup is already able to translate the payload types between the producer and consumer, and the rtpCapabilities option of WebRtcTransport#consume allows specifying a preferredPayloadType.
Unfortunately, this value seems to be completely ignored. The payload type returned in the rtp parameters of the consumer is always the one defined in the router’s mediaCodecs, be it an explicit preferredPayloadType or the default incremental ones.
Is this intended behavior? Is it not possible to override the payloadType for the consumer?

To exemplify,

const consumer = await this.transport?.consume({
            producerId,
            rtpCapabilities: {
                codecs: [
                    {
                        kind: 'audio',
                        mimeType: 'audio/opus',
                        preferredPayloadType: 111,
                        clockRate: 48000,
                        channels: 2,
                        parameters: {
                            'useinbandfec': 1,
                            'maxaveragebitrate': 64000,
                            'stereo': 1,
                        },
                    },
                ],
                headerExtensions: [],
            },
})!;
console.log(consumer.rtpParameters)
/* {
  codecs: [
    {
      mimeType: 'audio/opus',
      payloadType: 100,   <- 100, as it's the first value in the router's mediaCodecs
      clockRate: 48000,
      channels: 2,
      parameters: [Object],
      rtcpFeedback: []
    }
  ],
  headerExtensions: [],
  encodings: [ { ssrc: 504721282, maxBitrate: 64000 } ],
  rtcp: { cname: '51b0ba15', reducedSize: true },
  mid: '0'
} */
2

The preferredPayloadType in consume() is not what you expect it to be. Actually it MUST much the expected value. It’s only optional in the codecs given to worker.createRouter(). This is explained in the docs somewhere.

There is already a task to allow per consumer payload type and RTP extension ids. BTW this is far from “an easy task”:

3

Thanks for the reply and information. I have changed my approach to have the server inform the client of the payload types instead, which is working nicely. That leads me to a subsequent question, though: What are the strategies for managing a large variety of codecs?
As far as I know, it is not possible to add media codecs to an existing router, which, especially for codecs with many profile variations, leaves us in an awkward situation of trying to fill in ahead of time all codecs that a client may attempt to use. I was hoping for a more dynamic approach where, as long as agreed to by all participants, any sensible combination of parameters could be appended to the router and used.
Thank you again in advance.

4

It could be feasible but I don’t see how it would be useful. If router codecs could change dynamically then all connected clients should be notified about the new codecs, the RtpCapabilities in the Device of mediasoup-client should be recomputed, etc.

Typically in a SFU environment you want to specify the exact codecs you want to support to cover the capabilities of your users. If you add recording feature you also need to limit those codecs for your recording system to support them, etc.

5

I suppose my use is mostly out of scope as I’m not using mediasoup-client. I have somewhat inverted the negotiation process: When a new client connects, it informs the server of all it’s supported capabilities; the server then finds the best match across all clients in one room and broadcasts the new selected codec. Initially, I was handling all of the configuration on the client, and expecting the server to conform with the payload types selected, hence my initial post. I have since changed this so that the server also responds to the initial capability offer with it’s matching capabilities. While the majority of cases can be easily covered, there is still a gap in the event that all clients support, for example, a higher H264 profile, but the router was not configured to handle it.
For now, I am logging these ‘missed capabilities’ so I can complete the configuration over time, any widely supported ones should be quickly covered so it is not a concern, I was just wondering for a cleaner approach.

6

Please create a Feature Request ticket in mediasoup GitHub with the rationale above.

7

So many feature requests die here in the forum when I kindly ask for a feature request ticket in GitHub. It’s ok.

8

My apologies, it has been posted #1564

1 Like