In order to simplify negotiation, I would like to always match the client’s preferred payload types. In principle, this should be easy, mediasoup is already able to translate the payload types between the producer and consumer, and the rtpCapabilities option of WebRtcTransport#consume allows specifying a preferredPayloadType.
Unfortunately, this value seems to be completely ignored. The payload type returned in the rtp parameters of the consumer is always the one defined in the router’s mediaCodecs, be it an explicit preferredPayloadType or the default incremental ones.
Is this intended behavior? Is it not possible to override the payloadType for the consumer?
The preferredPayloadType in consume() is not what you expect it to be. Actually it MUST much the expected value. It’s only optional in the codecs given to worker.createRouter(). This is explained in the docs somewhere.
There is already a task to allow per consumer payload type and RTP extension ids. BTW this is far from “an easy task”:
Thanks for the reply and information. I have changed my approach to have the server inform the client of the payload types instead, which is working nicely. That leads me to a subsequent question, though: What are the strategies for managing a large variety of codecs?
As far as I know, it is not possible to add media codecs to an existing router, which, especially for codecs with many profile variations, leaves us in an awkward situation of trying to fill in ahead of time all codecs that a client may attempt to use. I was hoping for a more dynamic approach where, as long as agreed to by all participants, any sensible combination of parameters could be appended to the router and used.
Thank you again in advance.
It could be feasible but I don’t see how it would be useful. If router codecs could change dynamically then all connected clients should be notified about the new codecs, the RtpCapabilities in the Device of mediasoup-client should be recomputed, etc.
Typically in a SFU environment you want to specify the exact codecs you want to support to cover the capabilities of your users. If you add recording feature you also need to limit those codecs for your recording system to support them, etc.
I suppose my use is mostly out of scope as I’m not using mediasoup-client. I have somewhat inverted the negotiation process: When a new client connects, it informs the server of all it’s supported capabilities; the server then finds the best match across all clients in one room and broadcasts the new selected codec. Initially, I was handling all of the configuration on the client, and expecting the server to conform with the payload types selected, hence my initial post. I have since changed this so that the server also responds to the initial capability offer with it’s matching capabilities. While the majority of cases can be easily covered, there is still a gap in the event that all clients support, for example, a higher H264 profile, but the router was not configured to handle it.
For now, I am logging these ‘missed capabilities’ so I can complete the configuration over time, any widely supported ones should be quickly covered so it is not a concern, I was just wondering for a cleaner approach.
I’m having issue with video/AV1 RTP streamed from ffmpeg (PT does not matter) and corresponding producer created, but then when I connect Chrome client it only works if preferredPayloadType in the router is set to 45 (this is what Chrome expects in SDP offer) but apparently does not work for FireFox which is expecting PT=99 (from SDP offer). Consequently, mediasoup sends RTP (PT=45, cpatured with Wireshark) to both Chrome and Firefox. RTP packets fly, but video frames are not decoded. Is AV1 more restrictive in the browsers as compared to other codecs?
As you can see in the mediasoup demo (which uses mediasoup-client in the browser app) this is not true. AV1 is properly received by Chrome despite payload type is 107: