Help Me Understand the MediaSoup Integration Flow

Integration
1

I’ve been trying to implement group video calling using MediaSoup, following the documentation carefully, but it’s not working as expected. Here’s the flow of my app:

  • Client 1 (Sender):
  • Joins a room, gets RTP capabilities, and loads the device.
  • Creates a WebRTC transport on the server for receiving media (consumer=true) and sends transport details to the client.
  • Creates a recvTransport on the client and attaches a connect event to send the transport ID and DTLS parameters back to the server.
  • The server connects the transport and checks for producers in the room. If producers exist, it consumes their media and sends the data to the client.
  • The client maps through the consumed data and calls consume on the recvTransport.
  • If camera permission is denied, the process stops. Otherwise:
    • Creates a WebRTC transport on the server for sending media (consumer=false) and uses the returned transport data to create a sendTransport on the client.
    • Attaches connect and produce event listeners to the sendTransport and sends the necessary parameters to the server to connect the transport and create a producer.
  • Client 2 (Receiver):
  • Joins the same room, gets RTP capabilities, and loads the device.
  • Since camera permission is off, only creates a WebRTC transport on the server for receiving media (consumer=true) and sends transport details to the client.
  • Creates a recvTransport on the client and attaches a connect event to send the transport ID and DTLS parameters back to the server.
  • The server connects the transport, checks for producers in the room, consumes their media, and sends the consumed data to the client.
  • The client maps through the consumed data and calls consume on the recvTransport.
2

Can you detail the issue you are encountering, like the error you are getting, if any.

4

From my understanding, the media produced by the sender is not being sent to the server. Here is a short video of the Chrome WebRTC internals when the sender’s camera is turned on and the producer is generating data. However, on the receiving end, the receiver is receiving the media track from the server, and I tried resuming the media track, but it’s not displaying anything

5

Unfortunately, the video just shows that the sender is sending media (to mediasoup). I am much more interested on what is happening on the receiver end since that is where the issue could be coming from. How is the receiver consuming the produced media? Any code snippet you could share for both client and server side of this? and also, are there any errors logged on the client side, like the browser console?

6

This is how i structured storing all the needed data for a group video call.

image
image701×856 64.3 KB

This is the code block for creating consumers in server.

async createConsumerFromTransport(data: any, roomId: string, userId: string) {
    const { rtpCapabilities }: { rtpCapabilities: RtpCapabilities } = data;

    const room = this.rooms.get(roomId);
    if (!room || !room.router) {
        throw new Error("Room or router not found");
    }

    let consumersInfo: any[] = []
    for (const user of room.users.values()) {
        if (user.id === userId) continue; //Prevent consuming same user stream

        for (const producerId of user.producersIds) {
            const producer = room.producers.get(producerId);
            if (!producer) continue;

            if (!room.router.canConsume({ producerId, rtpCapabilities })) {
                console.error(`Router cannot consume for producer ${producerId}`);
                continue;
            }

            const transport = room.transports.get(producer.transportId);
            if (!transport) {
                console.error(`Transport not found for producer ${producerId}`);
                continue;
            }
            try {
                const consumer = await transport.transport.consume({
                    producerId,
                    rtpCapabilities,
                    paused: true,
                });

                user.consumersIds.push(consumer.id);
                room.consumers.set(consumer.id, {
                    consumer,
                    producerId,
                    transportId: transport.transport.id,
                    userId,
                });
                consumersInfo.push({
                    id: consumer.id,
                    producerId: producerId,
                    kind: producer.kind,
                    rtpParameters: consumer.rtpParameters,
                    userId: producer.userId
                })
            } catch (err) {
                console.error(`Failed to create consumer for producer ${producerId}:`, err);
            }
        }
    }
    return consumersInfo
}

And this my client side for consuming media

const receiveAllVideoFromServer = async () => {
    console.log("Emitting createTransport for consuming ")

    socket.emit('createTransport', { consumer: true }, (data: any) => {

        const recvTRansport = device.current.createRecvTransport(data);

        // Connect will be triggerd on the first call to recvTransport.consume()
        recvTRansport.on('connect', async ({ dtlsParameters }, callback, errback) => {
            console.log('Recv transport connect event fired internally');
            try {
                console.log(`Emitting transportConnect event with id=${recvTRansport.id} and dtlsParameters`);
                socket.emit("transportConnect", {
                    dtlsParameters: dtlsParameters,
                    transportId: recvTRansport.id,
                    consumer: true
                }, (stat: any) => {
                    console.log("Got response from server for setting dtls on consumer transport", stat)
                    callback()
                })
            } catch (err) {
                console.log("Error occured", err)
                errback(err as Error)
            }
        });

        socket.emit('transportConsume', {
            rtpCapabilities: device.current.rtpCapabilities,
        }, async (consumeData: any) => {
            console.log("Ready to consume", consumeData);

            // Create an array to store all consumer operations
            const consumers: any[] = []
            for (const obj of consumeData) {
                let consumer = await recvTRansport.consume({
                    id: obj.id,
                    producerId: obj.producerId,
                    kind: obj.kind,
                    rtpParameters: obj.rtpParameters,
                })
                consumers.push(consumer)
            }

            console.log(consumers)               

            consumers.forEach((obj: any) => {
                console.log(obj)
                socket.emit('resumeConsumeTransport', {
                    consumerId: obj.id
                })
            })
          
            //Some ui related react code to render videos on screen

        });
    })
}
7

At this point and with the information at hand, it is quite diffcult to pinpoint what the issue is. The code snippet you pasted looks good so far, perhapes the issue is from another part of the code. You can try any or all of the following ( in no particular order) to get further information.

  • Check the webrtc internals at the receiver’s browser to see if packets are being received.
  • Enable logging for mediasoup both for client and server to see if any interesting message gets logged to the console.
  • Enable trace event at one of the consumers of the receiving peer (server side). Use with caution. This can help determine, from mediasoup end, if the consumer is actually getting RTP packets among other things.
8

While debugging the Mediasoup client on the receiving device, I found that the ICE connection state transitions to “connected,” and the MediaStreamTrack is successfully loaded into the browser’s video element.

image
image1920×1054 351 KB

Upon inspecting the receiving client using the Chrome WebRTC debugger, I observed incoming data. However, I also noticed outgoing data, even though I joined from the receiver device with both video and audio turned off

image
image1844×704 60.6 KB

PS: I am running both clients in same computer and connected to same network.
and this is how my config looks like

    let transport = await room.router.createWebRtcTransport({
        listenInfos: [
            {
                ip: "0.0.0.0",
                announcedIp: "127.0.0.1",
                protocol: "udp"
            },
            {
                ip: "0.0.0.0",
                announcedIp: "127.0.0.1",
                protocol: "tcp"
            }
        ],
        enableTcp: true,
        enableUdp: true,
        preferUdp: true,
    })

And one more thing is that i have attached a onLoadMetaData event listener to the video element and when the recieving client loads the streames video this event is not firing


                                <video
                                    ref={participant.ref}
                                    onLoadedMetadata={(e) => {
                                        console.log("Video metadata loaded")
                                        console.log(e)
                                    }}
                                    autoPlay
                                    playsInline
                                    muted={participant.id === user?.id}
                                    className="w-full h-full object-cover"
                                />
9

Your mediasoup setup should not be listening on localhost IP address 127.0.0.1 as can be seen in your announcedIp. This seems to be the issue you are encountering. Here is an earlier post on this issue.

10

I changed the announcedIp from 127.0.0.1 to my local ip which is 192.168.0.110 . But nothing changed the ice stills connects and i can see data packetsReceived in chrome webrtc internals. This is my current config
let transport = await room.router.createWebRtcTransport({
listenInfos: [
{
ip: “192.168.0.110”,
announcedIp:“192.168.0.110”,
protocol: “udp”
},
{
ip: “192.168.0.110”,
announcedIp:“192.168.0.110”,
protocol: “tcp”
}
],
enableTcp: true,
enableUdp: true,
preferUdp: true,
})

Here are some logs which shows the connection is success

image
image1873×973 246 KB

11

I think there is something wrong with the track. because when i set the srcObject to the track after consuming the receivers video. The onloadmetadata is not getting fired. So is there any things missing in track

12

Please don’t try to find external culprits. You have been said what the main problem is. Please read in depth the documentation about announcedIp in mediasoup website (there is also extra doc in the FAQ section).

1 Like
13

If you’re using the latest Mediasoup, ensure you include the portRange. Try this example:

let transport = await room.router.createWebRtcTransport({
    listenInfos: [
        {
            protocol: 'udp',
            ip: '0.0.0.0',
            announcedAddress: '192.168.0.110',
            portRange: { min: 40000, max: 40100 }
        },
        {
            protocol: 'tcp',
            ip: '0.0.0.0',
            announcedAddress: '192.168.0.110',
            portRange: { min: 40000, max: 40100 }
        }
    ],
    enableUdp: true,
    enableTcp: true,
    preferUdp: true,
    iceConsentTimeout: 20,
    initialAvailableOutgoingBitrate: 1000000
});
14

I tried adding a port range, but there were no changes. The ICE connection was successful, and here are the client logs.

image
image1813×975 221 KB

15

Your project is open source?, so if shared, others could analyze it. I believe the issue lies more within your internal logic or a misconfiguration, rather than with mediasoup itself.

16

Yes my repo is public. Here the link to my repo GitHub - Naveenravi07/mediasoup-video-stream.

However it contains some extra code. So i will give the relevant file links.

Mediasoup helper in server :- mediasoup-video-stream/apps/api/src/mediasoup/mediasoup.service.ts at main · Naveenravi07/mediasoup-video-stream · GitHub

Mediasoup controller wiith socket events handlers :- mediasoup-video-stream/apps/api/src/mediasoup/mediasoup.gateway.ts at main · Naveenravi07/mediasoup-video-stream · GitHub

Client code :- mediasoup-video-stream/apps/web/app/meet/[id]/page.tsx at main · Naveenravi07/mediasoup-video-stream · GitHub