Audio Producer created but not sending media using Sdp Bridge

emdede · March 25, 2022, 4:15am

Hey everyone. First of all the obligatory praise for mediasoup. It’s an amazing library and I appreciate your work on this project! I have already completed a Virtual Office project using Mediasoup and it’s working nicely.

My new assignment, though, causes me a bit of a headache. We are trying to integrate WebRTC using Mediasoup SFU in Unity.

After many days of trying and debugging I finally got it to work. However, it only works with Video. The audio that is sent from the application is not being transmitted ( Empty Producer Stats ). Which is puzzling to me as in the core there shouldn’t be any difference, no?

I am turning to you guys in hopes that someone could enlighten me where the problem could be. Now I know that Mediasoup is only one part of my setup and the error could be somewhere else along the line but I am out of ideas at this point.

My setup is:
MixedReality WebRTC for Unity
OpenVidu’s Mediasoup SDP Bridge fork
Mediasoup

The process is simple. MR WebRTC works with a mostly vanilla approach. It’s a standard Offer-Answer model with trickle-ice.
I translate the SDP messages using the SDP bridge and populate Mediasoup transports.

Again, for Video it works absolutely fine already. However, the Audio track is not being sent.
I have already tried different things, like:

Remove video track and send only Audio (same result)
Use different codecs (same result)
Combed through MR WebRTC Issues on GH and this forum here (no specific results)

If I use the Unity App in vanilla mode (direct p2p via a PeerConnection) audio works fine. That’s why I’m thinking the problem is probably server-side somewhere.

I would greatly appreciate any hints that could lead me to resolve this issue. I’ll attach several bits of information that could help find the problem.

SDP Offer/Answer from the broken Audio connection:

Offer
v=0
o=- 8534608570643004729 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:xvmp
a=ice-pwd:DysEU3DkZiiT2J7mU4S2Vtb8
a=ice-options:trickle
a=fingerprint:sha-256 13:FB:6B:28:1D:B9:BF:CF:B3:3D:78:06:4C:5E:F5:D0:91:75:9F:D0:A3:46:0B:77:32:38:5A:2B:B0:EB:D8:49
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- da6e57c5-9b77-4f99-b477-445eea6d9780
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:2171558851 cname:KmmmiQWE7gsElCJV
a=ssrc:2171558851 msid: da6e57c5-9b77-4f99-b477-445eea6d9780
a=ssrc:2171558851 mslabel:
a=ssrc:2171558851 label:da6e57c5-9b77-4f99-b477-445eea6d9780

Answer
v=0
o=mediasoup-client 10000 1 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-lite
a=fingerprint:sha-512 E2:16:9F:63:40:AC:E1:AE:62:B8:AA:22:E1:C3:4E:04:FE:9A:EC:10:09:EF:19:67:23:8B:84:9F:BD:39:2B:42:93:EC:A6:61:D8:0E:80:8A:45:98:E9:C2:EB:0F:56:D0:DE:BE:C7:74:A8:23:68:77:BE:02:84:E0:77:84:A7:29
a=msid-semantic: WMS *
a=group:BUNDLE 0
m=audio 7 UDP/TLS/RTP/SAVPF 111
c=IN IP4 127.0.0.1
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtcp-fb:111 transport-cc
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=setup:active
a=mid:0
a=recvonly
a=ice-ufrag:lmvojj3yknu0tf5v
a=ice-pwd:abohg4ovhgr9ekuwfopu5p94614enxjp
a=candidate:udpcandidate 1 udp 1076302079 XXXXXXXXX 43009 typ host
a=candidate:tcpcandidate 1 tcp 1076302079 XXXXXXXXX 40267 typ host tcptype passive
a=end-of-candidates
a=ice-options:renomination
a=rtcp-mux
a=rtcp-rsize

As a comparison, here’s the Answer from when I try it with the vanilla implementation (audio works fine)

data=v=0
o=- 6722375024708538588 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=msid-semantic: WMS
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 102 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:oFYw
a=ice-pwd:Q05S8q0lleRv8SlllM2veqQw
a=ice-options:trickle
a=fingerprint:sha-256 8D:9D:F1:DE:51:79:72:EB:AF:C5:CA:36:70:E3:7D:D3:7A:78:3A:00:AE:F7:46:D0:F9:38:14:EC:30:A9:1D:9D
a=setup:active
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:- f572eaa1-7b6c-4a1c-94c9-8b594eb61cf5
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:270015488 cname:4vlIriUyG5mguaM/

Finally, here are some server logs of a test run including parsed producerParameters

new producer created [worker.pid:16464, router.id:e7b241ca-5150-4209-a25a-17620978ea3e, transport.id:fc3e619e-dc71-4ae3-a7b3-7975aebc2094, producer.id:b59463af-8712-4abe-911
b-92e675f1c15a]
PRODUCERPARAMS
 {"mid":"0",
  "codecs":[{"mimeType":"audio/opus","payloadType":111,"clockRate":48000,"channels":2,"parameters": "minptime":10,"useinbandfec":1},"rtcpFeedback":[{
"type":"transport-cc","parameter":""}]}],
  "headerExtensions":[{"uri":"urn:ietf:params:rtp-hdrext:ssrc-audio-level","id":1,"encrypt":false,"parameters":{}},{"uri":"urn:ietf:pa
rams:rtp-hdrext:sdes:mid","id":9,"encrypt":false,"parameters":{}}],
  "encodings":[{"ssrc":3187872821,"dtx":false}],
  "rtcp":{"cname":"IfkdEfLUVL+AjF+H","reducedSize":true,"mux": true}}
[SdpEndpoint.processOffer] mediasoup Producer created, kind: audio, type: simple, paused: false
[SdpEndpoint.createAnswer] Make 'recvonly' SDP Answer
mediasoup:Channel [pid:16464] RTC::IceServer::HandleTuple() | transition from state 'new' to 'connected' [hasUseCandidate:false, hasNomination:false, nomination:0] +1s
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::OnIceServerSelectedTuple() | ICE selected tuple +0ms
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::OnIceServerConnected() | ICE connected +0ms
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::MayRunDtlsTransport() | running DTLS transport in local role 'client' +1ms
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::OnDtlsTransportConnecting() | DTLS connecting +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::Run() | running [role:client] +0ms
transport ICE state changed to connected
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | DTLS handshake start +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'before SSL initialization'] +1ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write client hello'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | role: client, waiting:'SSLv3/TLS write client hello'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write client hello'] +108ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read server hello'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read server certificate'] +1ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read server key exchange'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read server certificate request'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read server done'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write client certificate'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write client key exchange'] +1ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write certificate verify'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write change cipher spec'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write finished'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | role: client, waiting:'SSLv3/TLS write finished'] +0ms
  mediasoup:Channel [pid:16464] RTC::IceServer::HandleTuple() | transition from state 'connected' to 'completed' [hasUseCandidate:true, hasNomination:false, nomination:0] +3
3ms
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::OnIceServerCompleted() | ICE completed +0ms
mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS write finished'] +75ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read change cipher spec'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | [role:client, action:'SSLv3/TLS read finished'] +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::OnSslInfo() | DTLS handshake done +1ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::CheckRemoteFingerprint() | valid remote fingerprint +0ms
  mediasoup:Channel [pid:16464] RTC::DtlsTransport::GetNegotiatedSrtpCryptoSuite() | chosen SRTP crypto suite: SRTP_AES128_CM_SHA1_80 +0ms
  mediasoup:Channel [pid:16464] RTC::WebRtcTransport::OnDtlsTransportConnected() | DTLS connected +0ms
mediasoup:Producer getStats() +5s
  mediasoup:Channel request() [method:producer.getStats, id:6] +4s
  mediasoup:Channel request succeeded [method:producer.getStats, id:6] +0ms
Audio Producer Stats in 5 s interval
 []

Thank you so much for any help in advance. If there’s something else I could provide to help with this issue, please let me know.

emdede · March 27, 2022, 6:23am

Quick update.

After some more testing I think the MixedReality WebRTC implementation is the culprit (or it’s a WebRTC concept I’m missing). Meaning, it’s probably not mediasoup-related. I’d still like to keep this open, if that’s okay for you, to see if someone’s dealt with this case before or maybe has a better general understanding of the matter and can point me to the source of the problem. Unfortunately, the MR WebRTC package has just officially entered “Deprecation” a few days ago and is no longer maintained, so there’s no use trying to reach out on their platform.

It seems to be connected to the send direction. If I run the Vanilla test (direct p2p) and have a sendonly direction from the connection that creates the Offer and a recvonly counterpart, the issue is exactly the same. Video gets through fine, but Audio is missing. But it works if both sides have a sendrecv direction. From my understanding of the MR WebRTC code the Track should be attached as a mediaLine just fine (it’s also present in the SDP offer). If anyone’s got an understanding of what’s happening here, I’d greatly appreciate if you could point out what might cause this.

Topic		Replies	Views
Unity integration for the client Integration	10	1434	September 22, 2022
Using mediasoup server with a standard WebRTC client mediasoup libraries	5	388	November 19, 2021
I have video, but no audio mediasoup-demo	6	878	February 10, 2022
Using libmediasoupclient with mediasoup v2 server side mediasoup libraries	9	1051	January 21, 2020
mediasoup-client: Peer to Peer mediasoup libraries	5	2484	May 22, 2020

Audio Producer created but not sending media using Sdp Bridge

Related topics