Digital Ocean mediasoup deployment not showing live-stream

Deployment & Scalability
1

Our mediasoup implementation for our streaming app is not showing the host’s live-stream in our staging environment. It is working perfectly in localhost.

We are using Digital Ocean with a loadbalancer (tcp ports are forwarded) with kubernetes droplets (firewall has ports open and a Dockerfile also exposes those ports).
announcedIp is the loadbalancer public IP. The droplets inside have a public and private IP. Do i need to add them to listenIps?
And lastly, we enabled hostNetwork in kubernetes’ deployment.yaml file.
Also, we’re using helm charts. Not sure if that makes it more complicated.
Is there anything else im missing? Has anyone had a similar deployment that could help out?

The config file:

const config = {
    mediaSoup:
    {
        // WORKER SETTINGS.
        worker:
        {
            rtcMinPort: 32503, // MINIMUM NUMBER OF PORT.
            rtcMaxPort: 32505, // MAXIMUM NUMBER OF PORT.
            logLevel: 'error', // ONLY LOG ERRORS.
            // logTags: ['info', 'ice', 'dtls', 'rtp', 'srtp', 'rtcp',],
        },
        // ROUTER SETTINGS.
        router:
        {
            mediaCodecs:
                [
                    {
                        kind: 'audio',
                        mimeType: 'audio/opus',
                        clockRate: 48000,
                        channels: 2
                    },
                    {
                        kind: 'video',
                        mimeType: 'video/VP8',
                        clockRate: 90000,
                        parameters: { 'x-google-start-bitrate': 1000 }
                    },
                ]
        },
        // WEB RTC TRANSPORT SETTINGS
        webRtcTransport:
        {
            listenIps: [{ ip: '0.0.0.0', announcedIp: process.env.IP }], // SERVER IP ADDRESS.
            enableUdp: false,
            enableTcp: true,
            preferUdp: false,
            maxIncomingBitrate: 1500000,
            initialAvailableOutgoingBitrate: 1000000,
            minimumAvailableOutgoingBitrate: 600000,
            maxSctpMessageSize: 262144,
        }
    }
};
1 Like
2

Go to the debugging tools in your browser and see what fails and why: chrome://webrtc-internals/ in Chromium-based browser or about:webrtc in Firefox-based.
Check candidate pairs first, if they don’t connect then port forwarding doesn’t work properly.

hostNetwork in k8s should do, helm or not is irrelevant. Dockerfile ports are also irrelevant if you use host network.

P.S. Using TCP only is generally not a good idea, prefer UDP if at all possible.

2 Likes
3

Why using such a small port range?
Does the Digital Ocean machine has a public IP or are you using a frontend/proxy in front of it?

2 Likes
4

@nazar-pc i would love to use UDP, but Digital Ocean does not have UDP port forwarding on their loadbalancer. (if i dont need to forward those ports and i can use network host, i will definitely use UDP).

@vpalmisano i cannot specify a range to open or forward ports in digital ocean, kubernetes, and dockerfile. so i just picked 3 ports.

The load balancer has an public IP and the droplets have public and private IPs with firewalls and their ports open for mediasoup connection.

I hope that answers your question @vpalmisano

5

Unless you have a single client you will need more ports. You can put TURN server in front of Mediasoup and use that with a single public port as a proxy with as many ports as needed within internal network between TURN server and Mediasoup.

2 Likes
6

Are you sure that using a floating IP (without a load balancer) doesn’t allow to listen on all ports?

1 Like
7

@nazar-pc, that is the approach we ended up going with today. However, still not getting a feed. And i am not too familiar with chrome and firefox webrtc debugging tools. im not really sure what im looking for in those logs.

@vpalmisano Im not sure if i understand your question but i am currently not using a floating IP on the droplet. And the application needs to use a load balancer. The firewall on the droplet has the port range open for mediasoup.

I have the staging environment running if anyone is available to take a look at it. i completely understand if you dont want to. i know its asking a lot. :sweat_smile:

8

You should probably have a load balancer for signaling instead and no load balancer for actual media (use direct IP and UDP for that if possible). Otherwise TCP connection established by browser will end up on a random instance.

1 Like
9

If I may add, given it’s been a few days.

I’ve found a lot of these providers won’t actually give you a public facing address when you ask for it but one that’s internal to their networks. So maybe changing announcedIP to that of the public facing one. I know this has been an issue and to solve I run a resolver to find my facing IP before I launch server.

1 Like
10

Hey sorry for the late response! thats what we ended up going with after investigating webrtc behind a load balancer and reading this post: https://stackoverflow.com/a/61170138

We separated the webrtc portion into a separate repo, hosted it without a load balancer. It worked fine when not in docker and when the port range is only 20 ports. We are investigating the issue with docker-compose and port range at the moment.
Its definitely just a docker issue now. We exposed the ports, enabled hostNetwork, enabled privileged mode.

After this, we need to figure out how to scale this, but that will be after the beta.

@BronzedBroth, Digital Ocean provides both public and private IPs for the Doplet and a public IP for the load balancer, we’ve tried all combinations.

Also, we really appreciate the help.

11

You don’t want to open large number of individual ports or ranges with Docker. It will consume a ton of RAM and will not result in anything good. Either use host networking or TURN server in front of it. Don’t enabled privileged mode except very specific edge cases, it is a very bad idea for security, and try to not run an app as root either, better run as nobody:nogroup.

1 Like
12

thank you for the tips! So we did just that. Removed privledged mode and the large number of ports on docker. We kept hostNetworking and setup a turn server. All outside of the loadbalancer. The application is live using mediasoup (https://codecrow.io). Sorry for the shameless advertisement lol.
Our next attempt at scaling this would be to test docker swarm on a dedicated server. And again, thank you for your help. i’ve learned a lot.

1 Like
13

A bit off-topic to this particular issue but I noticed you mentioned the amount of open ports being low (both you and @vpalmisano).

What kind of calculations go into finding an optimal number of ports?

14

There is no “optimal”, you allocate as many as you expect you’d need. And yes, it has nothing to do with this thread at all.