Firefox pipeToRouter problems?

mediasoup libraries
1

TL;DR - In Firefox, when User1 has producer & consumer transports on different Mediasoup routers, User1 cannot see/hear other producers, but everyone can see/hear User1. When User1’s producer & consumer transports are on the same router, all is well.

For smaller audiences, only 1 router/worker is used. There are no problems here.

For larger audiences, 1 router/worker is used for producers, X routers/workers are used for consumers. Producers are piped through to all consumer routers.

For example:
If Kim, James, and Natalie are part of a broadcast together, and a bunch of other people are just watching, it could look like:

  • router1: producerTransports for Kim, James, Natalie
  • router2: consumerTransports for Kim and users 4-50
  • router3: consumerTransports for James, Natalie, and users 51-100

If James is using Firefox, he can’t hear/see Kim or Natalie, but everyone can see/hear him. If James joins using Chrome or Safari on the same computer, all is well.

Additional notes:

  1. If James is producing but only router1 is used (for producers and consumers), Firefox works fine.
  2. If James is NOT producing / part of the broadcast, and is just viewing Kim+Natalie, all is well (he can see/hear them).

I am at a loss as to how the browser could be a factor here…any ideas?

2

I remember some issues in Firefox when creating two PeerConnections in the same website and the remote endpoint uses different DTLS certificates. So if you use two mediasoup Routers in different mediasoup Workers, each Worker will have its own random DTLS certificate.

NOTE: If “James” is not producing, then the sendTransport does not even connect ICE+DTLS to mediasoup. That happens once the first Producer (or DataProducer) is created. So this explains the issue if my above suspicion is true.

So:

  1. Ensure ICE and DTLS connects in both sendTransport and recvTransport when running Firefox in your failing scenario. This is critical, please don’t do bypass of this step. If I’m right one of those transports will fail at DTLS level.
  2. Create a TLS certificate and use it in your mediasoup Workers, so all Workers will use the same one and Firefox may not complain.
  3. If any of those is true, please comment here but also report the issue in the Firefox issue tracker.
1 Like
3

You are a genius! That did it, thank you.

I found this 2017 discussion where it seems the [Firefox] issue was first spotted. You said:

Good catch @kolychev!
Yep, the browser should not make assumptions regarding DTLS certificates no matter the URL origin.

I now specify my own DTLS certificates when creating the workers, and everything works as expected.

I did not check to see if any transports were failing, mainly because I wouldn’t know how to do this off the top of my head. I simply jumped straight to using my own DTLS certs since that seemed like the clear culprit after reading the above discussion.

4

Great. But please, report it to Firefox.

5

Assuming I’m understanding things correctly, the discussion linked above mentions this Firefox bug which I think covers this case.

That bug is 10+ years old though, so instead of waiting on a fix, I wonder if it might make sense for mediasoup to use the same cert for all workers on the same machine? This might be a terrible idea / break things elsewhere, but thought I’d toss it out there.

6

That bug is indeed +10 years old and it does not even mention WebRTC or PeerConnection. Unclear for me whether it’s related. IMHO a new bug should be created in Firefox tracker telling the specific problem above described: the same Firefox tab cannot connect 2 PeerConnections if the remote DTLS endpoints use a different DTLS certificate.

No, we will never do hacks in mediasoup to mitigate real bugs in browsers like this one. You can already make mediasoup use the same DTLS certificates for all workers if you wish.

Please confirm whether you will report this issue in Firefox tracker or not.

7

Got it, I will submit the bug in Firefox tracker and link it here once I do.

1 Like
8

I was in this world of Firefox/DTLS certificate negotiation failures a while back, https://bugzilla.mozilla.org/show_bug.cgi?id=1397177 was the most relevant bug I found.

9

I’ve commented in that issue in Firefox tracker:

After looking at this a bit, I’m less sure what I said in c6 is correct. Firefox should be good with two certificates with the same CN as long as they have different issuer/serials.

In https://mediasoup.org we create random DTLS certificates with random name/subject and random serial number:

However the issue reported above happens as shown here:

A second PeerConnection connecting to a different worker instance of mediasoup (which has a different random DTLS certificate) fails. And the issue does not happen if all worker instances of mediasoup are provided with the same DTLS certificate (instead of letting them create a random one).

If anyone can go deeper into this, please check how mediasoup creates a random DTLS certificate (if no certificate is provided via Worker config):

10

Interestingly, I don’t see such issues on Linux neither in Firefox stable nor in Nightly.
Have one router for sending data and another for receiving.
Also tried Firefox stable on macOS Catalina and didn’t have such problems either.
Setup is similar: 1 router for producers, X for consumers.

11

The issue just happens if you connect to different Workers with different DTLS certificates (which is the default behavior unless you provide the same cert to all Workers).

12

I have 1 router per worker, so workers are different too.
And I do not provide any custom certs to different workers.

13

I see. No idea honestly. This is Firefox specific. No idea why it depends on, but we do things right in mediasoup.

14

Sorry for disappearing, was on holiday. Now back at it!
And I don’t think the “fix” I mentioned was actually a fix…

I thought this fixed things, but then found out anyone not on the producer’s network cannot hear/see anything, so I reverted the change the next day.
So if…

  • User 1 is producing
  • User 2 is consuming, but is on the same WiFi network as User 1.
  • User 3+4 are consuming from a remote location.

…everything will appear fine for Users 1+2, but Users 3+4 won’t see/hear anything.

The REALLY strange part is if User 2 is on an iPad and disabled WiFi / switches to 4G, it STILL works for him! It’s as if proximity is somehow a factor… :exploding_head:

Right now, I am remoted into a laptop 0.2 miles away on its own network, and continue to see a black box when using my own certs.

For context, here is the worker call:

const worker = await mediasoup.createWorker({
    logLevel: mediaserver_environment.mediasoup.worker.logLevel,
    logTags: mediaserver_environment.mediasoup.worker.logTags,
    rtcMinPort: mediaserver_environment.mediasoup.worker.rtcMinPort,
    rtcMaxPort: mediaserver_environment.mediasoup.worker.rtcMaxPort,
    dtlsCertificateFile: '/etc/letsencrypt/live/mysite.com/fullchain.pem',
    dtlsPrivateKeyFile: '/etc/letsencrypt/live/mysite.com/privkey.pem'
});

When I comment out those dtls lines, everything works fine (other than the Firefox bug).

Does anyone have an idea of what could be causing it to not work across networks? I must be doing something stupid. The dtls key can simply be the same as the ssl key for the Node server, correct? I am using the same key.

15

Talking about “users not seeing/hearing anything” is completely useless for me. You have tools (getters and events in client and server side) to check transport ICE and DTLS status. If transport is not connected at ICE and DTLS levels then, of course, audio and video won’t work (it won’t even be sent) but the opposite is not necessarily true.