Consumer intermittently drops connection to stream

mediasoup libraries
1

I have a fairly basic mediasoup setup: Ffmpeg sends a stream over rtp to a mediasoup plain transport. The plain transport is consumed by a webrtc consumer, which is sent to the client. This path is taken by two separate streams – one for audio, one for video. (FWIW, this is a desktop video and audio capture stream). FFmpeg is running on one machine and sending it over my local network to my machine, which hosts the mediasoup server as well as the client.

FFmpeg → stream → Plain Transport → Producer → Consumer → WebRTC Transport → stream → Client

Everything works mostly fine (took a while to get to this point though :grinning_face_with_smiling_eyes:), except that intermittently the client stops receiving data from the video stream, but the audio stream continues normally i.e. the video freezes on the last-received frame while the audio continues to play. That drop in stream seems to correspond to this log message on the server-side:

mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:ef8b3d31-8569-4726-b8e8-9a0e151ff5c5] +5s

Occasionally, I might see this message precede the drop as well:

  mediasoup:WARN:Channel [pid:51983] RTC::TransportCongestionControlClient::MayEmitAvailableBitrateEvent() | high BWE value decrease detected, notifying the listener [now:910000, before:1433600] +18s

Eventually (usually ~2-3s), the stream comes back up and I see this message:

mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:1, temporal:0, consumerId:ef8b3d31-8569-4726-b8e8-9a0e151ff5c5] +2s

The logs for a full cycle of down to back up looks something like this:

mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:17729] +8s
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:18292] +8s
mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:-1, temporal:-1, consumerId:ef8b3d31-8569-4726-b8e8-9a0e151ff5c5] +7s
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:19135] +2s
mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:1, temporal:0, consumerId:ef8b3d31-8569-4726-b8e8-9a0e151ff5c5] +329ms
mediasoup:Channel [pid:51983] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:1, media ssrc:856678269] +987ms
mediasoup:WARN:Channel [pid:51983] RTC::TransportCongestionControlClient::MayEmitAvailableBitrateEvent() | high BWE value decrease detected, notifying the listener [now:1433600, before:1974110] +2m
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:19997] +7s
mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::SendRtpPacket() | sync key frame received +0ms
mediasoup:Channel [pid:51983] RTC::SimulcastConsumer::SendRtpPacket() | sending sync packet [ssrc:856678269, seq:7318, ts:1533553142] from original [ssrc:955204770, seq:19997, ts:1533553142] +0ms
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:20895] +8s
mediasoup:WARN:Channel [pid:51983] RTC::TransportCongestionControlClient::MayEmitAvailableBitrateEvent() | high BWE value decrease detected, notifying the listener [now:910000, before:1433600] +18s
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:21755] +8s
mediasoup:Channel [pid:51983] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2121, seq:22549] +8s

From my perspective, there are a few potential issues:

  1. Ffmpeg is dropping frames/bitrate. This doesn’t appear to be the case. I’ve left the stream running for >1 hour and FFmpeg reports 6 total frames dropped and there have been many more than 6 stream dropouts.
  2. The mediasoup server is unable to handle the IO bandwidth. This doesn’t seem to be the case to me because I’m not seeing any producer-related or plain transport-related logs indicating that. I’m also on a fairly new (beefy) machine that theoretically should easily handle this rate of transfer.
  3. The mediasoup server is halting the stream. I’m unsure on this one because if I have two clients consuming, the stream usually will only halt for one and not the other. That said, it’s still possible it’s a server-side issue because maybe the server is halting the stream just for one consumer for whatever reason.
  4. The mediasoup client is requesting to halt the stream. Not sure why this would be happening, but this is my best guess. I’m not very familiar with BWE, but it seems that the BWE is prompting the client to temporarily halt the video stream. It’s also hard to tell because there doesn’t seem to be anything in the client logs that indicate anything changing. I have the debug option set to mediasoup-client:* – I see all the logs related to setting up the stream and connecting, but nothing after that, and especially nothing when the stream halts.

I’m not exactly looking for a full solution, just more so where to look/how to debug this. How can I tell where the issue is coming from? Is there a way I can better track/debug BWE? Are there client logs that I’m missing somehow/not turned on? Are there settings/configs I can/should tinker with to see if it fixes the issue?

This is the first time I even considered reaching out here for help in my mediasoup journey, so I know I’m really stumped. Please let me know if there is any additional information that would help here. Any help to point me in the right direction would be much appreciated. Thanks!

Stream info:
According to FFmpeg, the combined output stream bitrate is ~250 kbps. This is for a 720p 30fps fast preset h.264 video stream and 2 channel 128k 48000 opus audio stream.

Transport configs:

webRtcTransport: {
	listenIps: [{ ip: '127.0.0.1' }],
	maxIncomingBitrate: 10 * 1000 * 1000,
	minimumAvailableOutgoingBitrate : 600 * 1000,
	initialAvailableOutgoingBitrate: 600 * 1000,
	maxSctpMessageSize: 262144,
	enableUdp: true,
	enableTcp: true,
	preferUdp: true,
}

plainTransport: {
	listenIp: { ip: '0.0.0.0' },
	comedia: true,
	rtcpMux: false,
	maxSctpMessageSize: 262144,
}
2

I don’t understand why you are announcing simulcast (more than 1 stream) in the Producer associated to ffmpeg.

3

errr, I didn’t know I was :sweat_smile:. I just assumed it was off by default. How do I not do that? Is there something I’m setting/not setting?

4

Check the encodings array you are passing in produce(). BTW there is huge doc about this in the website.

5

Ahh, yes, that worked! I misunderstood how the encodings parameter worked in the producer options.

So what was happening is that the client detected some kind of change in the BWE and requested a different track on the stream, of which there were none available. Is that a correct understanding?

Either way, I’m seeing smooth sailing on the video stream now – no drops! :sunglasses: Thanks!