Very high packet loss with ffmpeg broadcasting

Integration
1

Hi, I followed the guide at https://mediasoup.org/documentation/v3/communication-between-client-and-server/#guidelines-for-node-sctp to try to broadcast through WebRTC to mediasoup-client in the browser. Eventually the goal is to livestream from x11 and alsa, but for demonstrating the issue I’m facing, I have set up a simple example with just a looping .webm file instead.

I’m experiencing unusually high videos stream packet loss: pretty much every single packet is lost save for a few small chunks spread far apart. Here are some graphs from chrome://webrtc-internals:

packet-loss
ohshit.png2560×1440 270 KB

My ffmpeg options look something like:

ffmpeg \
  -re \
  -stream_loop -1 \
  -i ~/a.webm \
  -map 0:a:0 \
  -c:a libopus -ab 48k -ac 2 -ar 48000 -application lowdelay -cutoff 12000 \
  -map 0:v:0 \
  -pix_fmt yuv420p
  -c:v libx264 -tune zerolatency -preset ultrafast -threads 0 -crf 23 -minrate 5M -maxrate 5M -bufsize 10M \
  -f tee \
  "[select=a:f=rtp:ssrc=11111111:payload_type=101]rtp://127.0.0.1:3301?rtcpport=4502|[select=v:f=rtp:ssrc=22222222:payload_type=102]rtp://127.0.0.1:3501?rtcpport=2989"

I’m not sure what the source of the problem is exactly. These options seem to work perfectly fine when simply outputing to a file. Is there some detail I’m missing?

Thanks

2

We cannot provide support about proper ffmpeg usage. Said that, you are just looking at ffmpeg and receiver browser side.

mediasoup provides lot of stats in server side you can check. For instance producer.getStats() which will tell you how good the RTP transmission from ffmpeg to mediasoup is.

BTW fmpeg has nice problems sending RTP over UDP. I’ve seen that it looses LOT of RTP packets due to its own buffers. I hope ffmpeg’s output tells you something, but right now your topic has zero info about mediasoup itself.

1 Like
3

Yeah, I wasn’t too sure exactly what kind of information to post here. If you want some other information I’ll post it here. I pretty much followed the guide to the T, except for the ffmpeg stuff, which is a little different. But looks like producer.getStats() is a good starting point to troubleshoot this issue. I’ll let you know if I find something, thanks

4

Check the packetLost in producer.getStats() and take into account that, if any packet is lost between ffmpeg and mediasoup, mediasouop will send a NACK to ffmpeg (if enabled in the producer rtpParameters), however ffmpeg does NOT support NACK as far as I know, so ti won’t retransmit the packet, producing the packet lost in the receiving browser and frozen video until ffmpeg sends a video key frame (which may happen every 2 seconds or more, depending on ffmpeg command line settings).

I would use gstreamer instead. AFAIK it properly supports NACK and so on.

1 Like
5

Actually I think I accidentally forgot to include the -tune zerolatency option I’m using in my post, which I have now edited to add. It essentially tells x264 to make every frame a keyframe to avoid the need to retransmit, since the very next frame would make up for the lost frames, so I actually don’t mind a packet being lost even now and then. What I don’t understand is if you see the graph, pretty much >90% of the packets are being dropped, which baffles me. Of course that is in the browser after going through mediasoup. Once I test with producer.getStats() I’ll let you know the details.

6

Hey @ibc, thanks a lot for showing me the light! I have ditched ffmpeg in favour of gstreamer (which is wayy better with support for NACK), and after some tweaking around using output from producer.getStats() like you suggested, things are looking pretty awesome, I have ~40 clients streaming simultaneously with hardly any dropped packets. And of course, thanks for this amazing piece of software!

7

Glad to hear that! :slight_smile:

BTW this is a common question. It would be great if you could contribute to mediasoup by writing your gstreamer usage as we do with ffmpeg here. You may directly create a PR in the mediasoup-website project by adding a new section " Example: Inject Audio and Video using GStreamer".

8

Sure, makes sense, that way everyone can enjoy the goodness

2 Likes
9

Hi @ambyjkl

I’m exactly experiencing the same issue with ffmpeg. Would you please share the equivalent GStreamer command you’ve used?

10

@ibc

For me the producer stats (ffmpeg => mediasoup) is fine but there are lots of packet loss in the consumer stats. This is while consuming webcam or shared screen is fine too. What do you think?

11

What I saw in the past is that ffmpeg does not loose RTP packets but video frames while encoding or buffering, so at RTP level everything seems to be good (no packet lost in the Producer) but those continuous RTP packets do not contain continuous video frames. I cannot help much more on the ffmpeg side.

12

Thanks for the tip. :ok_hand:

13

Hi @mkh I’m sorry I’ve been away. Had exams and such, but now I’m free again. I’ll make a PR with my gstreamer pipeline and share the link here

14

Great! Thank you @ambyjkl.

15

@ambyjkl If you could provide a GStreamer command, that would also be helpful if you are busy for preparing the PR.

16

Hey @mkh so sorry, the command is quite hairy and would need quite a bit of explanation so people understand what’s going on, so I didn’t yet get the chance to create a PR that I was happy with. But here’s the gist, I set up the server like so:

const audioTransport = await router.createPlainRtpTransport({
    listenIp: LOCALIP,
    rtcpMux: false,
  })

  const videoTransport = await router.createPlainRtpTransport({
    listenIp: LOCALIP,
    rtcpMux: false,
  })
  audioTransport.connect({
    ip: LOCALIP,
    port: 5006,
    rtcpPort: 5007,
  })

  videoTransport.connect({
    ip: LOCALIP,
    port: 5004,
    rtcpPort: 5005,
  })

  const audioProducer = await audioTransport.produce({
    kind: 'audio',
    rtpParameters: {
      codecs: [audioCodec],
      encodings: [{ ssrc: AUDIO_SSRC }]
    },
  })

  const videoProducer = await videoTransport.produce({
    kind: 'video',
    rtpParameters: {
      codecs: [videoCodec],
      encodings: [{ ssrc: VIDEO_SSRC }],
    },
  })
  // Read the transport local RTP port.
  const audioRtpPort = audioTransport.tuple.localPort
  const audioRtcpPort = audioTransport.rtcpTuple.localPort
  const videoRtpPort = videoTransport.tuple.localPort
  const videoRtcpPort = videoTransport.rtcpTuple.localPort

and I start gstreamer as a subprocess, feeding in the information. I have tweaked my command so it picks up your webcam and your microphone:

gst-launch-1.0 \
rtpbin name=rtpbin rtp-profile=avpf \
v4l2src device=/dev/video0 \
! queue \
! videorate ! video/x-raw,framerate=30/1 \
! videoconvert ! video/x-raw,format=I420,framerate=30/1 \
! x264enc tune=zerolatency speed-preset=1 dct8x8=true quantizer=23 pass=qual \
! rtph264pay ssrc=${VIDEO_SSRC} pt=${VIDEO_PAYLOAD_TYPE} \
! rtprtxqueue max-size-time=2000 max-size-packets=0 \
! rtpbin.send_rtp_sink_0 \
rtpbin.send_rtp_src_0 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} bind-port=5004 port=${videoRtpPort} \
rtpbin.send_rtcp_src_0 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} port=${videoRtcpPort} sync=false async=false udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0 \
\
alsasrc device="hw:<your mic device>" \
! queue \
! opusenc bandwidth=superwideband bitrate-type=vbr \
! rtpopuspay ssrc=${AUDIO_SSRC} pt=${AUDIO_PAYLOAD_TYPE} \
! rtprtxqueue \
! rtpbin.send_rtp_sink_1 \
rtpbin.send_rtp_src_1 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} bind-port=5006 port=${audioRtpPort} \
rtpbin.send_rtcp_src_1 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} port=${audioRtcpPort} sync=false async=false udpsrc port=5007 ! rtpbin.recv_rtcp_sink_1
17

Thank you @ambyjkl for sharing the code :smiley::pray:.

I modified your GStreamer command to read from file as input rather than webcam (video only):

gst-launch-1.0 \
rtpbin name=rtpbin rtp-profile=avpf \
filesrc location="${filename}" ! decodebin \
! queue \
! videoconvert \
! x264enc tune=zerolatency speed-preset=1 dct8x8=true quantizer=23 pass=qual \
! rtph264pay ssrc=${VIDEO_SSRC} pt=${VIDEO_PAYLOAD_TYPE} \
! rtprtxqueue max-size-time=2000 max-size-packets=0 \
! rtpbin.send_rtp_sink_0 \
rtpbin.send_rtp_src_0 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} bind-port=5004 port=${videoRtpPort} \
rtpbin.send_rtcp_src_0 ! udpsink bind-address=${LOCALIP} host=${LOCALIP} port=${videoRtcpPort} sync=false async=false udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0 \

However I could not get it to work with H264 since mediasoup complains about unsupported codec while I’m passing H264 as mediaCodecs argument when creating the router:

{
  kind: 'video',
  mimeType: 'video/H264',
  clockRate: 90000,
  parameters: {
    'packetization-mode': 1,
    'profile-level-id': '42e01f',
    'level-asymmetry-allowed': 1,
  },
}

I guess it’s about profile-level-id. @ibc How should I properly set profile-level-id? I found nothing useful in the docs.

OTOH it works with VP8 codec but in slow motion (2x or 3x slower)!

18

@mkh
Could you try adding the below caps after “x264enc”?
“x264enc ! video/x-h264,profile=constrained-baseline,level=(string)3.1”

19

No magic response for that. H264 profile level matching is SUPER hard. In mediasoup and mediasoup-client we use this lib to properly match profile-level-id values.

More here.

20

How about the mediasoup side? What should I pass as the codec profile-level-id?