Firefox doesn't work sometimes until the consumer side run getUserMedia or getDisplayMedia

mediasoup libraries
1

Before I created a ticket Firefox with proxy: Mediasoup demo is perfect, but mediasoup-sample-app has a little issue

After some testing recently, I found that:
If we build a webpage, that the broadcaster broadcasts, others only watch, then if the watcher is using firefox, sometimes (e.g., he is using some type of proxy), the client side cannot successful connect to the transport.
Unless: he grants permission to mic, or cam, or screen.

I also found some code in the official demo:

// NOTE: Stuff to play remote audios due to browsers' new autoplay policy.
//
// Just get access to the mic and DO NOT close the mic track for a while.
// Super hack!
{
    const stream = await navigator.mediaDevices.getUserMedia({ audio: AUDIO_CONSTRAINS });
    const audioTrack = stream.getAudioTracks()[0];
    audioTrack.enabled = false;
    setTimeout(() => audioTrack.stop(), 120000);
}

If I comment these lines, the demo also can not always work properly, suppose the user doesn’t not enable any device.

Is there a best practice to fix this?

2

Learn about browsers autoplay policy.

4

Hi ibc,
Thanks for the reply, now I understand that these lines are for enabling the autoplay in a smart way.

The problem I met was is the transport “connected” (of connectionstatechange) event is not triggered.

And only when I use a proxy, this issue happens.

5

Shall I do some more detailed check (debug)?

I’m not familiar with the C++ development, also does not understand deeply of the core webrtc, and without any experience debugging Firefox internal stuff, I may do things ineffectively.

But I’m still happy to contribute my time.

6

And to emphasis, if

await navigator.mediaDevices.getUserMedia({ ...});

or

await navigator.mediaDevices.getDisplayMedia({ ..});

passed before the consume related code is executed, then things can work.

Hope this can be a clue.

7

I checked sdp, suppose

await navigator.mediaDevices.getUserMedia({ video: true });

is done, then the local sdp contains extra video information, may this be the point?

I will keep debugging.

8

For the cases that I do not use a proxy or grant device permission.
We can see the webrtc log

... setting pair to state WAITING ...
... setting pair to state IN_PROGRESS ...
...

STUN-CLIENT(5nDo|IP4:xxxxxx:xxxxx/UDP|IP4:xxxxxx:xxxxx/UDP(prflx|candidate:udpcandidate 1 udp 1076302079 xxxxxx xxxxx typ host)): Received response; processing

And then connected.
The stun server address is the mediasoup server address.

For the unsuccessful case, it just pend at

no checks to start
9

User should not need to enable their permissions for audio/video to be able to consume/view others. The autoplay permission/feature is not necessary in testing as you can click play.

A proxied user may be limited by what IPs/Ports/Protocols are allowed in their configuration so a TURN server or some sort would be required to be able to connect. I’d know these things while testing at least to narrow in on it being code issues/setup.

Can’t speak on your code but would recommend using the demo to understand the producer/consumer side of things both server-side and client. It’s fairly straight forward and issues like these aren’t imaginable.

10

However the tricky things was, if i run (on the watcher side)

await navigator.mediaDevices.getUserMedia({ ...});

or

await navigator.mediaDevices.getDisplayMedia({ ..});

then things starts to work.

I still doubt that firefox have a very tiny buggy thing.

(I’ve set up a coturn server, and uses it in the tranport.iceServers, that does not help).

I extracted the entire media-soup related stuff from roomclient.js (client) and room.js (server).
In the end I found this.

I know it’s very hard to connect these things, proxy is network, device is a resource.
But maybe they can have some indirect relationship during establishing peerconnection.

Joke: The Butterfly Effect

11

If you’re confident the stream is making to the viewer, check AudioContext state for suspended on load and if so, force the user to click before anything shows up otherwise you may have the stream to view but it never starts till interacted with (Thus auto-play fails).

Not sure where you’re applying this mediaDevices but if it’s initial start of script or something it forces the interaction before anything loads.

12

The direct problem is (in my failure case):
there is no transport “connected” (of connectionstatechange) event

Which means it’s impossible to continue.
Again this only happens on firefox with a socks5 proxy.

So it really seems just a network issue.

The strange thing is, if

await navigator.mediaDevices.getUserMedia({ ...});

is done, then the issue will not happen.

I know it’s hard to explain my situation. I will use a customized FirefoxHandler to play with.
I will also download firefox source code and some network tool to check the detailed stuff.

In the coming few days, I will have plenty of time to play with this.

Just hope I can be lucky.

Many thanks! CosmosisT

13

Check the local SDP of Firefox (specially local ICE candidates) in both cases: doing getUserMedia before and not doing it.

14

Thanks, will do.

15

(Meant to reply to jackwoods)
Maybe you’re stuck awaiting an action within the client. Possibly an issue with sync code working against asynchronous code.

I see this happen far too much and could maybe be a clue. You can console.log/debug and see what line of code pauses.

16

In my case, it seems not this reason, I will do further check everywhere anyways.

17

Also before I continue, I will spend some days understanding:
https://datatracker.ietf.org/doc/html/rfc5245

18

Hi, ibc,
if "navigator.mediaDevices.getUserMedia({ …}) " is not executed,
then the answer sdp looks like:

a=candidate:0 1 TCP 2105524479 6df806e3-97a6-7e45-8a3b-958cd272b750.local 9 typ host tcptype active

after "navigator.mediaDevices.getUserMedia({ …}) " is executed, the sdp changes to be like:

a=candidate:0 1 UDP 2122252543 192.168.0.2 61034 typ host
a=candidate:1 1 TCP 2105524479 192.168.0.2 9 typ host tcptype active

This is the only change.

While I’m not using a proxy, firefox can connect to the server in the previous case.
But if I use a proxy, then it cannot. Seems a turn server may help, I will make some testing.

Sad, on firefox, the log says:
Skipping TURN server because of address type mis-match

This turn server can work perfectly for chrome (requires producer & consumer side both use turn server)

About the coturn stuff, I will spend some time do further study.