Using GStreamer webrtcbin as MediaSoup client

Hi all,
I’m trying to use the webrtcbin (https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad/html/gst-plugins-bad-plugins-webrtcbin.html) GStreamer element for communicating with MediaSoup.
Without entering into the GStreamer details, the element expects to exchange the SDP info with a webrtc peer (a web example here https://github.com/centricular/gstwebrtc-demos/blob/master/sendrecv/js/webrtc.js).
I’m trying to reconstruct the SDP message from the JSON returned from mediasoup server. The communication starts, but I get these errors:

RTC::Transport::ReceiveRtpPacket() | no suitable Producer for received RTP packet [ssrc:824347444, payloadType:96]

The ssrc value is the same sent into the rtpParameters when creating the producer. I don’t understand if the mid and rid values can be used if the ssrc is not found into ssrcTable.

Anyway, what I need to know is how medasoup can interoperate with a client without using the mediasoup-client library.
Thanks

mediasoup follows ORTC rules for matching a RTP stream against the corresponding RTCRtpReceiver (Producer in mediasoup):

  • If the ssrc of the RTP packet was announced in the rtpParameters then it’s associated Producer will exist in the ssrcTable.
  • If the ssrc is not found in the ssrcTable then mediasoup will check its MID RTP extension (if any) and look for the associated Producer in the midTable. If found, it will enter the ssrc into the ssrcTable.
  • Otherwise it will do the same with the ridTable.

More in detail:

You can print const dump = await transport.dump() to print those tables (among other info). This is Chrome sending audio and video (with MID in both audio/video Producers and RID based simulcast in video Producer):

  "rtpListener": {
    "midTable": {
      "0": "2b39a73f-a4d3-40de-a3bf-c61383cff686",
      "2": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04"
    },
    "ridTable": {
      "r0": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04",
      "r1": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04",
      "r2": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04"
    },
    "ssrcTable": {
      "80064589": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04",
      "1190225763": "2b39a73f-a4d3-40de-a3bf-c61383cff686",
      "1884500992": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04",
      "2712523004": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04",
      "2747658646": "cfe2c7cc-84ef-426c-b36d-a1d77e7feb04"
    }
  },

You can also print const dump = await producer.dump() to see its effective rtpParameters and see if they match your expectations. Also, producer.getStats() will show stats only for the RtpStreams created in the Producer, this is, those for which at least a valid RTP packet has been received.

Ok, I fixed a code error. I have no errors from mediasoup, but the video flow from GStreamer seems stopped.
One thing I don’t understand is the candidates exchange: I see that mediasoup sends back to the client its candidates list after WebRtcTransport creation, but the client never sends its own candidates list to the server. Is it right?

Resolved (problem on GStreamer pipeline). Now seems to work perfectly.

mediasoup is ICE Lite server so it doesn’t need client candidates

1 Like

Hi Vittorio,

Could you please share your working GStreamer pipeline?

Thanks,

I’m integrating webrtcbin into https://github.com/vpalmisano/mediasoupbin
What you need to do is converting the GStreamer generated SDP message into the dtlsParameters and rtpParameters structures and send them to the produce call to MediaSoup.

1 Like

Hi all,
I’m trying to finish the integration between GStreamer webrtcbin element and Mediasoup and I’ve found a blocking problem when I try to consume a WebRTC flow from Mediasoup.
When used as receiver, webrtcbin configures its ICE connection as controlled (https://github.com/GStreamer/gst-plugins-bad/blob/1.16.2/ext/webrtc/gstwebrtcbin.c#L2721) and mediasoup emits this warning: “peer indicates ICE-CONTROLLED in STUN Binding Request”, maybe because using ICE-lite?

EDIT: this seems to be fixed in the latest release: https://github.com/GStreamer/gst-plugins-bad/blob/master/ext/webrtc/gstwebrtcbin.c#L4469

Yes, this was the issue I faced and on my request it is already fixed in master branch.
Please note though that RTX is not working with renegotiations in that case though, so make sure to have do-nack=false.

Do you mean this: https://github.com/centricular/gstwebrtc-demos/blob/4351ec1e2b27117051409c63ce1a049f5a54cb31/android/app/src/main/jni/webrtc.c#L321 ?

Yes, you can set that on transceiver, but it is already false by default, so should be fine out of the box.

I checked the do-nack property in the transceiver, it is False by default. Anyway, after I setup the pipeline, I receive only some initial rtp packets, then the nicesrc element inside webrtcbin doesn’t receive nothing. I checked on Mediasoup side, the consumer is emitting rtp traces, so all seems fine on server side.

My receiver setup flow:

signalling.createWebRtcTransport -> transport
signalling.consume(transport.id, producerId) -> consumer

[transform the transport json into a SDP offer representation]

webrtcbin.emit('set-remote-description', sdp_offer)
webrtcbin.emit('add-ice-candidate', <transport iceCandidates>)
webrtcbin.emit('create-answer') -> client answer sdp
webrtcbin.emit('set-local-description', answer)

[get the dtlsParameters from answer sdp]

signalling.transportConnect(transport, dtlsParameters)

Similar flow works for me, but I think I didn’t use add-ice-candidate, but rather had them in SDP itself. Essentially I wrote serializer to SDP such that it produces identical result to Chrome 74 handler from mediasoup-client.
Also did you create paused consumer that is resumed afterwards just like with browsers?

Yes,
this is an offer example:

v=0
o=mediasoup-client 10000 1 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-lite
a=fingerprint:sha-512 62:E3:4B:82:7C:9E:9E:82:FC:B0:16:D:FC:86:F3.XX.XX.XX
a=msid-semantic: WMS *
a=group:BUNDLE 0
m=video 7 UDP/TLS/RTP/SAVPF 101 106
c=IN IP4 127.0.0.1
a=rtpmap:101 VP8/90000
a=rtpmap:106 rtx/90000
a=rtcp-fb:101 nack 
a=rtcp-fb:101 nack pli
a=setup:active
a=mid:0
a=sendonly
a=ice-ufrag:XXXXXXXX
a=ice-pwd:XXXXXXXXXXXXXXXXXXXX
a=candidate:hostcandidate 1 udp 1076558079 127.0.0.1 16885 typ host
a=candidate:hostcandidate 1 udp 1076532479 10.73.0.137 10908 typ host
a=candidate:hostcandidate 1 udp 1076506879 192.168.0.2 18188 typ host
a=candidate:hostcandidate 1 udp 1076481279 10.0.0.137 10442 typ host
a=ice-options:renomination
a=rtcp-mux
a=rtcp-rsize
a=ssrc:332426526 cname:346d56e9
a=ssrc-group:FID 332426526
a=msid:346d56e9 b0d78830-b2cc-4e87-9763-8210a2680d9c

and webrtcbin answer:

v=0
o=- 10000 1 IN IP4 0.0.0.0
s=-
t=0 0
a=group:BUNDLE 0
m=video 9 UDP/TLS/RTP/SAVPF 101
c=IN IP4 0.0.0.0
a=ice-ufrag:/RaaVE9/USqq2c0fuM9T36/sxXXXXXX
a=ice-pwd:7AaL26nZTyghXXXXXXXXXXXXXX
a=mid:0
a=rtcp-mux
a=setup:passive
a=rtpmap:101 VP8/90000
a=rtcp-fb:101 nack pli
a=recvonly
a=fingerprint:sha-256 F9:B9:8A:EA:XXXXX

I think a=fmtp:102 apt=101 is missing and try to change a=setup:active to a=setup:actpass, a=end-of-candidates would be nice to have as well.

There is nothing really secret in serializing, so here is exactly what I’m using and what works (it is also covered by tests, but I didn’t publish them), hopefully it helps: https://gist.github.com/nazar-pc/58ffd40db680ecae07198b15b919057e

Nothing changed after adding this line. The strange thing is that the peerconnection starts and I receive the first video frames that are correctly decoded. After some 5-10 received frames, the nicesrc element seems stalled, maybe there be some threading problem in the pipeline.

With this setting a get this error:

dtlsconnection gstdtlsconnection.c:964:handle_error:<GstDtlsConnection@0x7fead80098c0> Fatal SSL error

Just tried, same result.

I found the problem: webrtcbin element requires always a source element, even if in receive mode only (https://github.com/centricular/gstwebrtc-demos/blob/master/sendonly/webrtc-recvonly-h264.c#L289).
I fixed for now using this pipeline:

audiotestsrc is-live=true num-buffers=100 
    ! opusenc 
    ! rtpopuspay 
    ! webrtcbin name=webrtcbin bundle-policy=max-bundle
    ! rtpvp8depay
    ! vp8dec name=sink

This is not true. I’m not having any sink pads on webrtcbin and it works just fine.