Using Mediasoup client with h264

mediasoup libraries
1

Hi,
While trying to use the h264 codec on the browser side, a strange thing is happening with my mediasoup video producer on the send transport.
I am explicitly asking the producer to produce h264 with ‘profile-level-id’: ‘42e01f’ while calling the produce method on the send Transport. I have checked the device rtpCapabilities and my device supports this profile-level-id and I also can consume consumers with the same h264 profile id. But when I check the rtpParameters in sendTransport.on(“produce”) event, the profile-level-id is somehow changed to “profile-level-id”:“4d001f” . This is happening in both the latest chrome and edge browser but with firefox this issue is not happening. I am getting the desired h264 “profile-level-id”:“42e01f” in rtpParameters of sendTransport.on(“produce”).

I am using medisoup-client 3.7.6.

Am I doing something wrong here or missing something?

Sorry for being ignorant.

2

Well, it depends on the codecs you pass to worker.createRouter() in server side and the order of those codecs.

Alternatively you can force a codec (if supported by the Router) in client side by passing the codec option to transport.produce(). See the docs.

3

Thank you for your quick response. I am actually forcing the above mentioned h264 codec with profile id: “42e01f” in the client side while doing transport.produce. But it gets changed to “4d001f” when I receive them from transport.on(“produce”) only in case of chrome and edge but not in the case of firefox.
Will the order of codecs I pass while creating router can make a difference here?

4

If you add that h264 codec with same parameters (same profile level id and other important parameters) to the Router and then you properly pas that same codec in produce() in client then it must work and the order of codecs in the Router is irrelevant, otherwise you are doing something wrong. This is heavily tested.

5

Here is the codec that the remote device supports according to it’s rtpParameters.

mimeType: 'video/H264',
  kind: 'video',
 preferredPayloadType: 107,
  clockRate: 90000,
   parameters: {
    'level-asymmetry-allowed': 1,
    'packetization-mode': 1,
  'profile-level-id': '42e01f'
 },
 rtcpFeedback: [
     { type: 'goog-remb', parameter: '' },
    { type: 'transport-cc', parameter: '' },
   { type: 'ccm', parameter: 'fir' },
    { type: 'nack', parameter: '' },
    { type: 'nack', parameter: 'pli' }
 ]

These are the 2 h264 codec options I get when I log the router rtpCapabilities.

{
    kind: 'video',
    mimeType: 'video/H264',
   clockRate: 90000,
    parameters: {
     'level-asymmetry-allowed': 1,
    'packetization-mode': 1,
     'profile-level-id': '4d0032',
    'x-google-start-bitrate': 1000
    },
    rtcpFeedback: [ [Object], [Object], [Object], [Object], [Object] ],
    preferredPayloadType: 105
   },
      {
  kind: 'video',
  mimeType: 'video/H264',
  clockRate: 90000,
  parameters: {
       'level-asymmetry-allowed': 1,
       'packetization-mode': 1,
     'profile-level-id': '42e01f',
     'x-google-start-bitrate': 1000
     },
   rtcpFeedback: [ [Object], [Object], [Object], [Object], [Object] ],
   preferredPayloadType: 107
 },

The profile id “42e01f” is available in them both.

When I am trying to produce from my browser for consumption by the remote device while forcing the transport.produce to use this h264 profile of ‘42e01f’, It used to fail in the router.cancosume(producerId,remoteDeviceRTPCapabilities). When I dig deep, I realised that both chrome/edge are changing this profile-id from “42e01f” to “4d001f” in the rtpParameters while listening to the transport.on(“produce”). But this change doesn’t happen in case of firefox. For firefox, it successfully creates the video consumer for the remote device. But some other error seems to come in the client side.

rtp_transport_cc(196):Failed to demux RTP packet:PT=127 SSRC=1234 MID:probator
rtp_transport_cc(196):Failed to demux RTP packet:PT=127 SSRC=1234 MID:probator

internal_decoder_factory.cc(59) : Trying to create decoder for unsupported format.

I am able to consume the video sent by the remote device using the same parameters with h264 profile id “42e01f” but While sending it is not working. I am sure I am doing something wrong somewhere.

6

In transport.produce() in client side you must pass a codec that must be a codec entry in device.rtpCapabilities. It cannot be a codec object created from scratch.

7

This is properly documented:

8

Sure. This is how I am trying to create a codec for tnansport.produce.

 codec = this._device.rtpCapabilities.codecs.find(
            (c) =>
              c.mimeType.toLowerCase() === "video/h264" &&
              c.parameters["profile-level-id"] === "42e01f"
          );

I use this codec variable in transport.produce({track,encodings, codecOptions,codec}). Please let me know if I am missing something here.

9

Here is the screenshot when I run the app in localhost.
The first log is just before calling transport.produce() with the selected codec and the second log is inside the listener transport.on(produce).
image
This is with chrome Version 123.0.6312.122 (Official Build) (64-bit).
Here is how the same logs looks like in firefox.
image
With firefox, though I get the same profile level id but the payload type is different.

10

I also ran into this issue when trying to explicitly set the h264 codec with profile-level-id 42e01f when generating a producer. I was getting the same results as you - the h264 codec with profile-levle-id 4d001f was returned. I was able to get around it by changing the order of the device’s codecs in it’s rtpCapabilities prior to loading it.

This might be true for you too - the h264 codec with profile-levle-id 4d001f came before the one with 42e01f in the codecs in the devices rtpCapabilities. Looking at the code in the mediasoup client for the produce() function and the send() call for your handler (I’m working with Chrome 111), you can see it runs through the codecs and tries to find a match with the codec passed in. However, It does not appear to perform a “strict” check, meaning the criteria for a match does not look at parameters and fields like profile-level-id. So the 42e01f codec passed in matches with the first h264 it finds, which is the 4d001f codec.

Switching the order of the codecs before the device loads does work but I think it would be very nice to have finer control over what codecs are chosen when create a producer. It would be great if we could enable that “strict” check here when matching codecs. Happy to look at submitting a PR for this.

11

It should do strict checking. Can you please fill a bug in mediasoup-client GitHub?

12

Will do.

13

The payload type is different because sending and receiving payload types are usually different and device.rtpCapabilities contains codecs with the receiving payload types. Not a bug.

1 Like
14

Hi,
Thank you a lot for your response. It’s great to know that you made it working. After looking at the send function in mediasoup client, I was going for a very similar conclusion. When you say changing the order, do mean to say changing the order in the config file used for router creation because ultimately the routerRTPCapabilities fetched from server are loaded to the device in the client side. Am I missing something here?

15

I insist that if this is failing then it’s a bug, I already asked above to please fill an issue in GitHub with that specific use case so I can fix it and release a new version.

1 Like
16

Either @szt217 or me, one of us are going to file a bug for this for sure. Before filing the bug, I wish to verify if making this order change really works for me or not. Ideally it should work but I wish to verify before filing a bug. But a bug is going to be file by either one of us for sure so that the future users of this h264 profile needn’t face the same issue.
Thank you.

17

Yes, changing the order of the codecs will magically hide the bug and make it work as expected. But the bug is still there.

1 Like
18

Filed the bug here.

19

Fixed and version 3.7.7 released.

2 Likes
20

That’s super quick. Thank you a lot for fixing it promptly. For the future readers, if you are facing this issue, either upgrade to mediasoup-client version 3.7.7 or if you can’t upgrade for some reason, then change the order of your codecs, in the config file used for creating routers.

1 Like