Solved - Can't convince FFMPEG to understand mediasoup's RTP

I’m trying to port a solution I developed a while ago for Kurento. There it was possible to connect an RtpEndpoint to a WebRtcEndpoint and feed the output of the RtpEndpoint to a spawned FFMPEG process for further handling (storing to file, pushing to an RTSP server or whatever)

I tried something similar with mediasoup and so far all looks fine - just FFMPEG does neither write to file nor to any other output…

This is the server side sequence for tests. The code is invoked when the client asks the server to transport.produce (very q&d for now):

 const producer = await transport.produce({ kind, rtpParameters, appData: appData })

 const rtpTransport = await room.router.createPlainTransport({
       comedia: false,
       rtcpMux: false,
       listenIp: { ip: '127.0.0.1', announcedIp: null }
 })

const rtpPort = 50000
const rtcpPort = rtpPort + 1

await rtpTransport.connect({
    ip: '127.0.0.1',
    port: rtpPort,
    rtcpPort: rtcpPort,
    rtcpMux: false
})

logger.info(`RTP transport connected. local: ${rtpTransport.tuple.localIp}:${rtpTransport.tuple.localPort}, remote: ${rtpTransport.tuple.remoteIp}:${rtpTransport.tuple.remotePort}, ${rtpTransport.tuple.protocol}`)
logger.info(`RTCP transport connected. local: ${rtpTransport.rtcpTuple.localIp}:${rtpTransport.rtcpTuple.localPort}, remote: ${rtpTransport.rtcpTuple.remoteIp}:${rtpTransport.rtcpTuple.remotePort}, ${rtpTransport.rtcpTuple.protocol}`)

Finally I’m consuming it in the same sequence:

const rtpConsumer = await rtpTransport.consume({
       producerId: producer.id,
       rtpCapabilities: room.router.rtpCapabilities,
       paused: true,
})

All these steps succeed and if I check the stats regularly I’m getting valid stuff:

setInterval(() => {
   rtpConsumer.getStats().then(stats => console.warn(stats));
}, 2000)

Produces output like:

[
  {
    bitrate: 3065069,
    byteCount: 23645462,
    firCount: 0,
    fractionLost: 0,
    kind: 'video',
    mimeType: 'video/VP8',
    nackCount: 0,
    nackPacketCount: 0,
    packetCount: 20510,
    packetsDiscarded: 0,
    packetsLost: 0,
    packetsRepaired: 0,
    packetsRetransmitted: 0,
    pliCount: 0,
    rtxPacketsDiscarded: 0,
    rtxSsrc: 352172676,
    score: 10,
    ssrc: 352172675,
    timestamp: 1218815675,
    type: 'outbound-rtp'
  },
  {
    bitrate: 3043693,
    byteCount: 23481562,
    firCount: 0,
    fractionLost: 0,
    jitter: 0,
    kind: 'video',
    mimeType: 'video/VP8',
    nackCount: 0,
    nackPacketCount: 0,
    packetCount: 20510,
    packetsDiscarded: 0,
    packetsLost: 0,
    packetsRepaired: 0,
    packetsRetransmitted: 19,
    pliCount: 0,
    roundTripTime: 1.2664794921875,
    rtxPacketsDiscarded: 0,
    rtxSsrc: 3648546369,
    score: 10,
    ssrc: 3638399178,
    timestamp: 1218815675,
    type: 'inbound-rtp'
  }
]

The logger.info above traces also useful info, for instance:

webrtc:INFO:server RTP transport connected. local: 127.0.0.1:58111, remote: 127.0.0.1:50000, udp +4s
webrtc:INFO:server RTCP transport connected. local: 127.0.0.1:40809, remote: 127.0.0.1:50001, udp +0ms

When I fire up Wireshark I can see both UDP (RTP) and RTCP flowing between the mentioned ports over my loopback interface…

With Kurento I now spawned an FFMPEG process, fed it with an SDP file describing the input and wrote FFMPEG’s output either to disk or posted it to a chunked-POST HTTP server or forwarded it to an RTSP server.

That doesn’t work, at least obviously not with my simple SDP (as it seems), so I suppose I’m missing some describing things, either in the SDP or in the FFMPEG command line.

For the stream above the SDP is like so:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=KMS WebRTC to recorder bridge
c=IN IP4 127.0.0.1
t=0 0
m=video 50000 RTP/AVP 100
a=rtpmap:100 VP8/90000

(stored in a tmp file) and the FFMPEG command line is:

ffmpeg -loglevel debug -protocol_whitelist file,udp,rtp -i /var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-59000-NmigzKyGTDnf -vcodec copy /some/path/to/file.webm

FFMPEG creates an empty file.webm in the specified directory but that was it. No error, no nothing. No growing file either…:frowning:

Debug info traced by FFMPEG (I’m capturing it by an ffmpeg.on('data') handler:

ffmpeg.stderr.on('data', (data) => {
  logger.debug(`ffmpeg for recording on_data: ${data.toString()}`)
})
  webrtc:DEBUG:server ffmpeg for recording on_data: ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
  webrtc:DEBUG:server   built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  webrtc:DEBUG:server   configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/5.1.2_6 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-neon
  webrtc:DEBUG:server  +0ms
  webrtc:DEBUG:server ffmpeg for recording on_data:   libavutil      57. 28.100 / 57. 28.100
  webrtc:DEBUG:server   libavcodec     59. 37.100 / 59. 37.100
  webrtc:DEBUG:server   libavformat    59. 27.100 / 59. 27.100
  webrtc:DEBUG:server   libavdevice    59.  7.100 / 59.  7.100
  webrtc:DEBUG:server  +1ms
  webrtc:DEBUG:server ffmpeg for recording on_data:   libavfilter     8. 44.100 /  8. 44.100
  webrtc:DEBUG:server   libswscale      6.  7.100 /  6.  7.100
  webrtc:DEBUG:server   libswresample   4.  7.100 /  4.  7.100
  webrtc:DEBUG:server   libpostproc    56.  6.100 / 56.  6.100
  webrtc:DEBUG:server Splitting the commandline.
  webrtc:DEBUG:server Reading option '-loglevel' ... +0ms
  webrtc:DEBUG:server ffmpeg for recording on_data:  matched as option 'loglevel' (set logging level) with argument 'debug'.
  webrtc:DEBUG:server Reading option '-protocol_whitelist' ... +0ms
  webrtc:DEBUG:server ffmpeg for recording on_data:  matched as AVOption 'protocol_whitelist' with argument 'file,udp,rtp'.
  webrtc:DEBUG:server Reading option '-i' ... matched as input url with argument '/var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-59000-NmigzKyGTDnf'.
  webrtc:DEBUG:server Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'copy'.
  webrtc:DEBUG:server Finished splitting the commandline.
  webrtc:DEBUG:server Parsing a group of options: global .
  webrtc:DEBUG:server Applying option loglevel (set logging level) with argument debug.
  webrtc:DEBUG:server Successfully parsed a group of options.
  webrtc:DEBUG:server ffmpeg for recording on_data: Parsing a group of options: input url /var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-59000-NmigzKyGTDnf.
  webrtc:DEBUG:server Successfully parsed a group of options.
  webrtc:DEBUG:server Opening an input file: /var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-59000-NmigzKyGTDnf.
  webrtc:DEBUG:server ffmpeg for recording on_data: [NULL @ 0x13801f1a0] Opening '/var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-59000-NmigzKyGTDnf' for reading
  webrtc:DEBUG:server ffmpeg for recording on_data: [sdp @ 0x13801f1a0] Format sdp probed with size=2048 and score=50
  webrtc:DEBUG:server ffmpeg for recording on_data: [sdp @ 0x13801f1a0]  +1ms
  webrtc:DEBUG:server ffmpeg for recording on_data: video codec set to: vp8
  webrtc:DEBUG:server ffmpeg for recording on_data: [udp @ 0x137616610] end receive buffer size reported is 393216
  webrtc:DEBUG:server ffmpeg for recording on_data: [udp @ 0x137616320] end receive buffer size reported is 393216
  webrtc:DEBUG:server ffmpeg for recording on_data: [sdp @ 0x13801f1a0] setting jitter buffer size to 500
  webrtc:DEBUG:server ffmpeg for recording on_data: [sdp @ 0x13801f1a0] Before avformat_find_stream_info() pos: 115 bytes read:115 seeks:0 nb_streams:1

That also doesn’t point to an obvious error. Also ffpmeg.on('close') or ffmpeg.on('error') don’t fire…

I hope that doesn’t sound too weird. I’m currently not able to see the probably obvious. Anybody able to help me here?

Forgot to mention, what the last command after launching ffmpeg is:

 rtpConsumer.resume()

EDIT: When I close my producer client-side, these trace appears and ffmpeg closes:

Output file #0 does not contain any stream
2023-05-16T15:39:19.511Z webrtc:INFO:server ffmpeg for recording on_close stream 1

So ffmpeg seems to react on the input at least. It seems to not understand it…

EDIT 2: Weird output when I stop the stream:

  webrtc:DEBUG:server ffmpeg for recording on_data: [sdp @ 0x140c05520] Could not find codec parameters for stream 0 (Video: vp8, 1 reference frame, yuv420p): unspecified size
  webrtc:DEBUG:server Consider increasing the value for the 'analyzeduration' (100000000) and 'probesize' (100000000) options
  webrtc:DEBUG:server [sdp @ 0x140c05520] After avformat_find_stream_info() pos: 114 bytes read:114 seeks:0 frames:0
  webrtc:DEBUG:server  +41s
  webrtc:DEBUG:server ffmpeg for recording on_data: Input #0, sdp, from '/var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-76491-ldbfvSi4ysDf':
  webrtc:DEBUG:server   Metadata:
  webrtc:DEBUG:server     title           : Recorder
  webrtc:DEBUG:server   Duration: N/A, bitrate: N/A
  webrtc:DEBUG:server   Stream #0:0, 0, 1/90000: Video: vp8, 1 reference frame, yuv420p, 90k tbr, 90k tbn
  webrtc:DEBUG:server Successfully opened the file.
  webrtc:DEBUG:server Parsing a group of options: output url /some/path/to/file.webm.
  webrtc:DEBUG:server Applying option vcodec (force video codec ('copy' to copy stream)) with argument copy.
  webrtc:DEBUG:server Applying option f (force format) with argument webm.
  webrtc:DEBUG:server Successfully parsed a group of options.
  webrtc:DEBUG:server Opening an output file: /some/path/to/file.webm.
  webrtc:DEBUG:server Output #0, webm, to '/some/path/to/file.webm':
  webrtc:DEBUG:server Output file #0 does not contain any stream
  webrtc:DEBUG:server [AVIOContext @ 0x140c05b70] Statistics: 114 bytes read, 0 seeks

The file is empty…:frowning:

OK, a little step forward: I changed my stream to H.264. While checking with Wireshark I found two UDP streams: One from my web client to the mediasoup server and the other, plain RTP stream.

I made Wireshark to decode these packets as RTP and found valid RTP sequences. The “dynamic payload type” indicated was 103, so I changed SDP description from 100 to 103. In parallel I enabled Wireshark to decode RTP with payload type 103 as H.264 and was able to extract the video (with a Lua plugin) to a H.264 dump (NAL units), which I could convert to MP4. And playback.

So I’m still seeking for the proper FFMPEG command line to understand that programmatically.

The manual conversion of the H.264 dump gave this output:

~ $ ffmpeg -i dump.264 output.mp4
ffmpeg version 5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/5.1.2_6 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-neon
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, h264, from 'dump.264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1280x720, 25 fps, 25 tbr, 1200k tbn
File 'output.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x147705db0] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x147705db0] profile High, level 3.1, 4:2:0, 8-bit
[libx264 @ 0x147705db0] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=15 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf59.27.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 1280x720, q=2-31, 25 fps, 12800 tbn
    Metadata:
      encoder         : Lavc59.37.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  942 fps=342 q=-1.0 Lsize=    5079kB time=00:00:37.56 bitrate=1107.8kbits/s speed=13.6x    
video:5068kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.233872%
[libx264 @ 0x147705db0] frame I:5     Avg QP:14.75  size: 34346
[libx264 @ 0x147705db0] frame P:247   Avg QP:17.44  size: 10741
[libx264 @ 0x147705db0] frame B:690   Avg QP:19.98  size:  3426
[libx264 @ 0x147705db0] consecutive B-frames:  1.8%  1.1%  1.6% 95.5%
[libx264 @ 0x147705db0] mb I  I16..4: 47.9% 41.0% 11.1%
[libx264 @ 0x147705db0] mb P  I16..4: 13.1% 15.3%  0.8%  P16..4: 23.3%  4.8%  2.0%  0.0%  0.0%    skip:40.7%
[libx264 @ 0x147705db0] mb B  I16..4:  2.2%  1.9%  0.0%  B16..8: 25.1%  1.0%  0.1%  direct: 7.7%  skip:62.0%  L0:48.0% L1:50.8% BI: 1.2%
[libx264 @ 0x147705db0] 8x8 transform intra:50.3% inter:82.3%
[libx264 @ 0x147705db0] coded y,uvDC,uvAC intra: 22.3% 68.6% 14.9% inter: 6.2% 19.9% 0.1%
[libx264 @ 0x147705db0] i16 v,h,dc,p: 47% 26% 17% 10%
[libx264 @ 0x147705db0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 13% 57%  2%  2%  2%  2%  2%  2%
[libx264 @ 0x147705db0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 20% 28%  5%  5%  4%  5%  4%  5%
[libx264 @ 0x147705db0] i8c dc,h,v,p: 48% 26% 23%  3%
[libx264 @ 0x147705db0] Weighted P-Frames: Y:5.7% UV:3.6%
[libx264 @ 0x147705db0] ref P L0: 60.3% 10.0% 19.5% 10.0%  0.3%
[libx264 @ 0x147705db0] ref B L0: 82.7% 14.0%  3.3%
[libx264 @ 0x147705db0] ref B L1: 92.6%  7.4%
[libx264 @ 0x147705db0] kb/s:1101.58
~ $ 

To be continued :slight_smile:

EDIT: The WS capture:

Finally this FFMPEG command line wrote an MP4 file to disk (in case of H.264 input).

But just ONE TIME :slight_smile:

 cmdLine = `ffmpeg -protocol_whitelist file,udp,rtp -i ${file.name} -c copy -y /some/path/to/file.mp4`

The SDP in this case:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=Recorder
c=IN IP4 127.0.0.1
t=0 0
m=video 62000 RTP/AVP 103
a=rtpmap:103 H264/90000
a=fmtp:103 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f;

Most the time I’m getting tons of errors like

[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] non-existing PPS 0 referenced
[h264 @ 0x149f09370] decode_slice_header error
[h264 @ 0x149f09370] no frame!

after it finally gives up with:

sdp @ 0x149f089e0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[sdp @ 0x149f089e0] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

stderr: Input #0, sdp, from '/var/folders/7l/rc18f0m564qgtmjlzn6b_n5m0000gn/T/tmp-365-I4NN42nkwEqk':
  Metadata:
    title           : Recorder
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264, none, 90k tbr, 90k tbn

stderr: [mp4 @ 0x139f09df0] dimensions not set
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Error initializing output stream 0:0 -- 
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
    Last message repeated 1 times

It seems, that the output is missing SPS/PPS for a long time. This is understandable, given the fact, that the encoder might not produce that for a long time (I guess 2 minutes for Google encoders).

I would need to request an intra frame the moment I start recording.

Is there a way to generate a PLI request to the sender through mediasoup?

OK, making the consumer to requestKeyFrame() one second after resume after having spawned FFMPEG does the job. Same time I’m running resume on the consumer