Mediasoup with h264 on iOS

Hello.

I have compiled the Mediasoup library with WebRTC adding logs to see the device’s capabilities and I have added the library to iOS project with this wrapper: https://github.com/ethand91/mediasoup-ios-client

I check the codecs that WebRTC supports from my app:

let defaultRTC = RTCDefaultVideoDecoderFactory()
let defaultEncodeRTC = RTCDefaultVideoEncoderFactory()

for codec in defaultRTC.supportedCodecs() {
   print(codec.name)
   print(codec.parameters)
}

for codec in defaultEncodeRTC.supportedCodecs() {
   print(codec.name)
   print(codec.parameters)
}

LOGS:

H264
["profile-level-id": "640c33", "level-asymmetry-allowed": "1", "packetization-mode": "1"]
H264
["profile-level-id": "42e033", "level-asymmetry-allowed": "1", "packetization-mode": "1"]
VP8
VP9

I get the capabilities from our server and it supports h264 but when I pass them to Mediasoup it doesn’t load h264.

I have consulted the offer created by Mediasoup from the library and it doesn’t include h264.

json Handler::GetNativeRtpCapabilities(const PeerConnection::Options* peerConnectionOptions)
	{
		MSC_TRACE();

		std::unique_ptr<PeerConnection::PrivateListener> privateListener(
		  new PeerConnection::PrivateListener());
		std::unique_ptr<PeerConnection> pc(
		  new PeerConnection(privateListener.get(), peerConnectionOptions));

		(void)pc->AddTransceiver(cricket::MediaType::MEDIA_TYPE_AUDIO);
		(void)pc->AddTransceiver(cricket::MediaType::MEDIA_TYPE_VIDEO);

		webrtc::PeerConnectionInterface::RTCOfferAnswerOptions options;

		// May throw.
		auto offer                 = pc->CreateOffer(options);
		auto sdpObject             = sdptransform::parse(offer);
		auto nativeRtpCapabilities = Sdp::Utils::extractRtpCapabilities(sdpObject);

		MSC_DEBUG("GetNativeRtpCapabilities offer:\n%s", offer.dump(4).c_str());

		return nativeRtpCapabilities;
	}

LOGS
GetNativeRtpCapabilities offer:

v=0

o=- 3864646501880913761 2 IN IP4 127.0.0.1

s=-

t=0 0

a=group:BUNDLE 0 1

a=msid-semantic: WMS

m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126

c=IN IP4 0.0.0.0

a=rtcp:9 IN IP4 0.0.0.0

a=ice-ufrag:JuqU

a=ice-pwd:zlVuwktX7FdAJYbqRyP2QXCE

a=ice-options:trickle

a=fingerprint:sha-256 E8:82:F2:DC:B4:90:A9:EF:5D:F4:4B:8D:B5:51:CA:DB:EC:CD:93:A1:85:0A:20:80:89:35:43:72:DB:E9:19:25

a=setup:actpass

a=mid:0

a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level

a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time

a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01

a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid

a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id

a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id

a=sendrecv

a=msid:- a1845c7b-cddd-4790-86ad-dfd5f65f3a31

a=rtcp-mux

a=rtpmap:111 opus/48000/2

a=rtcp-fb:111 transport-cc

a=fmtp:111 minptime=10;useinbandfec=1

a=rtpmap:103 ISAC/16000

a=rtpmap:104 ISAC/32000

a=rtpmap:9 G722/8000

a=rtpmap:102 ILBC/8000

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:106 CN/32000

a=rtpmap:105 CN/16000

a=rtpmap:13 CN/8000

a=rtpmap:110 telephone-event/48000

a=rtpmap:112 telephone-event/32000

a=rtpmap:113 telephone-event/16000

a=rtpmap:126 telephone-event/8000

a=ssrc:1051213426 cname:wWzqvgrPSbEVsUBN

a=ssrc:1051213426 msid:- a1845c7b-cddd-4790-86ad-dfd5f65f3a31

a=ssrc:1051213426 mslabel:-

a=ssrc:1051213426 label:a1845c7b-cddd-4790-86ad-dfd5f65f3a31

m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127 124 125

c=IN IP4 0.0.0.0

a=rtcp:9 IN IP4 0.0.0.0

a=ice-ufrag:JuqU

a=ice-pwd:zlVuwktX7FdAJYbqRyP2QXCE

a=ice-options:trickle

a=fingerprint:sha-256 E8:82:F2:DC:B4:90:A9:EF:5D:F4:4B:8D:B5:51:CA:DB:EC:CD:93:A1:85:0A:20:80:89:35:43:72:DB:E9:19:25

a=setup:actpass

a=mid:1

a=extmap:14 urn:ietf:params:rtp-hdrext:toffset

a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time

a=extmap:13 urn:3gpp:video-orientation

a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01

a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay

a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type

a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing

a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07

a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space

a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid

a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id

a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id

a=sendrecv

a=msid:- 1a22e567-5fa0-4048-8bbe-8d896140457a

a=rtcp-mux

a=rtcp-rsize

a=rtpmap:96 VP8/90000

a=rtcp-fb:96 goog-remb

a=rtcp-fb:96 transport-cc

a=rtcp-fb:96 ccm fir

a=rtcp-fb:96 nack

a=rtcp-fb:96 nack pli

a=rtpmap:97 rtx/90000

a=fmtp:97 apt=96

a=rtpmap:98 VP9/90000

a=rtcp-fb:98 goog-remb

a=rtcp-fb:98 transport-cc

a=rtcp-fb:98 ccm fir

a=rtcp-fb:98 nack

a=rtcp-fb:98 nack pli

a=fmtp:98 profile-id=0

a=rtpmap:99 rtx/90000

a=fmtp:99 apt=98

a=rtpmap:100 AV1X/90000

a=rtcp-fb:100 goog-remb

a=rtcp-fb:100 transport-cc

a=rtcp-fb:100 ccm fir

a=rtcp-fb:100 nack

a=rtcp-fb:100 nack pli

a=rtpmap:101 rtx/90000

a=fmtp:101 apt=100

a=rtpmap:127 red/90000

a=rtpmap:124 rtx/90000

a=fmtp:124 apt=127

a=rtpmap:125 ulpfec/90000

a=ssrc-group:FID 627761831 1328663359

a=ssrc:627761831 cname:wWzqvgrPSbEVsUBN

a=ssrc:627761831 msid:- 1a22e567-5fa0-4048-8bbe-8d896140457a

a=ssrc:627761831 mslabel:-

a=ssrc:627761831 label:1a22e567-5fa0-4048-8bbe-8d896140457a

a=ssrc:1328663359 cname:wWzqvgrPSbEVsUBN

a=ssrc:1328663359 msid:- 1a22e567-5fa0-4048-8bbe-8d896140457a

a=ssrc:1328663359 mslabel:-

a=ssrc:1328663359 label:1a22e567-5fa0-4048-8bbe-8d896140457a

Why is Mediasoup not loading the h264 codec?
How could I get Mediasoup to work with WebRTC and h264 codec from iOS?

Thank you!

If you want to have just H264 video in all cases, you need to configure router with H264 as the only video option, see mediasoup :: API

Otherwise, you may want to make sure producerOptions on the client have H264 as the first (and hence preferred) codec in the list.

mediasoup-ios-client is not an official library, can’t suggest anything about it specifically, but I’m sure someone will be able to and above concepts still apply.

Hi @nazar-pc thanks for your answer

Our server only sends us the opus audio and h264 and rtx video. Should we remove the rtx codec?

{
    "codecs" : [
      {
        "parameters" : {

        },
        "kind" : "audio",
        "channels" : 2,
        "preferredPayloadType" : 100,
        "mimeType" : "audio\/opus",
        "clockRate" : 48000,
        "rtcpFeedback" : [
          {
            "type" : "transport-cc",
            "parameter" : ""
          }
        ]
      },
      {
        "mimeType" : "video\/H264",
        "parameters" : {
          "packetization-mode" : 1,
          "profile-level-id" : "640c33",
          "x-google-start-bitrate" : 1000,
          "level-asymmetry-allowed" : 1
        },
        "clockRate" : 90000,
        "kind" : "video",
        "preferredPayloadType" : 101,
        "rtcpFeedback" : [
          {
            "type" : "nack",
            "parameter" : ""
          },
          {
            "type" : "nack",
            "parameter" : "pli"
          },
          {
            "type" : "ccm",
            "parameter" : "fir"
          },
          {
            "type" : "goog-remb",
            "parameter" : ""
          },
          {
            "type" : "transport-cc",
            "parameter" : ""
          }
        ]
      },
      {
        "mimeType" : "video\/rtx",
        "parameters" : {
          "apt" : 101
        },
        "clockRate" : 90000,
        "kind" : "video",
        "preferredPayloadType" : 102,
        "rtcpFeedback" : [

        ]
      },
      {
        "mimeType" : "video\/H264",
        "parameters" : {
          "profile-level-id" : "42e01f",
          "packetization-mode" : 1,
          "x-google-start-bitrate" : 1000,
          "level-asymmetry-allowed" : 1
        },
        "clockRate" : 90000,
        "kind" : "video",
        "preferredPayloadType" : 103,
        "rtcpFeedback" : [
          {
            "type" : "nack",
            "parameter" : ""
          },
          {
            "type" : "nack",
            "parameter" : "pli"
          },
          {
            "type" : "ccm",
            "parameter" : "fir"
          },
          {
            "type" : "goog-remb",
            "parameter" : ""
          },
          {
            "type" : "transport-cc",
            "parameter" : ""
          }
        ]
      },
      {
        "mimeType" : "video\/rtx",
        "parameters" : {
          "apt" : 103
        },
        "clockRate" : 90000,
        "kind" : "video",
        "preferredPayloadType" : 104,
        "rtcpFeedback" : [

        ]
      }
    ],
    "headerExtensions" : [
      {
        "direction" : "sendrecv",
        "preferredId" : 1,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:sdes:mid",
        "kind" : "audio"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 1,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:sdes:mid",
        "kind" : "video"
      },
      {
        "direction" : "recvonly",
        "preferredId" : 2,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id",
        "kind" : "video"
      },
      {
        "direction" : "recvonly",
        "preferredId" : 3,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id",
        "kind" : "video"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 4,
        "preferredEncrypt" : false,
        "uri" : "http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time",
        "kind" : "audio"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 4,
        "preferredEncrypt" : false,
        "uri" : "http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time",
        "kind" : "video"
      },
      {
        "direction" : "recvonly",
        "preferredId" : 5,
        "preferredEncrypt" : false,
        "uri" : "http:\/\/www.ietf.org\/id\/draft-holmer-rmcat-transport-wide-cc-extensions-01",
        "kind" : "audio"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 5,
        "preferredEncrypt" : false,
        "uri" : "http:\/\/www.ietf.org\/id\/draft-holmer-rmcat-transport-wide-cc-extensions-01",
        "kind" : "video"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 6,
        "preferredEncrypt" : false,
        "uri" : "http:\/\/tools.ietf.org\/html\/draft-ietf-avtext-framemarking-07",
        "kind" : "video"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 7,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:framemarking",
        "kind" : "video"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 10,
        "preferredEncrypt" : false,

        "uri" : "urn:ietf:params:rtp-hdrext:ssrc-audio-level",
        "kind" : "audio"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 11,
        "preferredEncrypt" : false,
        "uri" : "urn:3gpp:video-orientation",
        "kind" : "video"
      },
      {
        "direction" : "sendrecv",
        "preferredId" : 12,
        "preferredEncrypt" : false,
        "uri" : "urn:ietf:params:rtp-hdrext:toffset",
        "kind" : "video"
      }
    ]
  }

Mediasoup only loads the audio with the opus codec but there is no video codec loaded.

Thank you very much for your help.

Why should you remove it?

Because @nazar-pc commented that I have to configure h264 as the only option in the router.

How does removing RTX codecs deal with what the API documentation says? Where do you want to remove RTX codec from? This makes no sense at all. This is not what Nazar told you.

That’s why I asked :sweat: I had understood that there should only be the h264 codec.

Initially my question was what could be happening so that Mediasoup does not load the h264 codec?

Is it possible you have to do something in WebRTC to enable encoding / decoding h264 in iOS? I understand that by default it should use the h264 hardware codec

libwebrtc in iOS may support a specific set of profile-level-id values for H264.

RTX is not a codec as such (yes, WebRTC API is a bit confusing in that regard), don’t remove it.

1 Like