Overlay rtp stream from medisaoup with ffmpeg

Integration
#1

The problem occurred while trying to overlay a stream from mediasoup to another stream (udp, rtmp, etc.).
According to ffmpeg logs, it starts working, processes for a while and stops (line 957 of the log file)
Sending one stream works without issue

I suspect there is something wrong with the stream from mediaasoup

Start command:
ffmpeg -strict experimental -loglevel debug -protocol_whitelist pipe,udp,rtp -fflags +genpts -flags low_delay -f sdp -analyzeduration 10000000 -probesize 10000000 -i pipe:0 -fflags +nobuffer -analyzeduration 10000000 -probesize 10000000 -i rtmp://eu-02.re.stream.mvs.wtf:1936/monitor/pV2ErmcZIscDsaoU8AS4KuYYYZG_-wPDiH_sDeu6U58 -filter_complex [0:a]asetpts=PTS-STARTPTS[a0];[1:a]asetpts=PTS-STARTPTS[a1];[0:v]setpts=PTS-STARTPTS,scale=iw/3:ih/3[pip];[1:v]setpts=PTS-STARTPTS[v1];[v1][pip]overlay=0:0:shortest=0[vo];[a0][a1]amix=inputs=2[ao] -map [vo] -map [ao] -c:a aac -c:v h264 -s 640x480 -r 30 -x264opts keyint=60:keyint_min=60:no-scenecut -preset superfast -profile:v high -level:v 4.1 -tune zerolatency -f flv -max_muxing_queue_size 4096 rtmp://publish.stream.mvs.wtf/push/LFcDPV65OWznyhAjcSvOqhcxN9BRFBijP7jFakjweR0

sdp:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=FFmpeg
c=IN IP4 127.0.0.1
t=0 0
m=video 26692 RTP/AVP 101
a=rtpmap:101 VP8/90000
a=sendonly
m=audio 26267 RTP/AVP 100
a=rtpmap:100 opus/48000/2
a=sendonly

Logs: https://gist.github.com/Able1991/b59bd18cc44a1b9b08c3df4cd5dbe08a

I studied the output and do not see the reason for this behavior

#2

Yes, why not. Let’s blame mediasoup.

No, there is nothing wrong with the stream from mediasoup.

#3

This is a good answer, and most importantly it helped =)

I’m not saying that mediasoup is to blame for everything, but there is such a problem that I am talking about

#4

I cannot help with ffmpeg stuff, but most of the “mediasoup + ffmpeg issues” reported in this forum are always related to wrong ffmpeg usage or expectations.

mediasoup (in server side) provides you with tons of debug logs and stats for you to diagnose how good the stream (Consumer) RTP transmission is. Use them.

Also, mediasoup is a SFU rather than a RTP endpoint. It means that the RTP stream that mediasoup forwards to your ffmpeg will be as good (but no more) than the stream that mediasoup receives from the corresponding producer. So if there is packet disorder (generated by the network or due to packet loss) your ffmpeg will receive unordered RTP packets. That’s something you have to “fix” in the receiver side (by setting a RTP jitter buffer, etc). No idea about how to do that in ffmpeg. This is not a forum about ffmpeg. But I know it can be done with GStreamer.

#5

Maybe the overlay filter is encountering a missing PTS from the RTP stream and getting stuck? It looks like there’s some packet loss around line 723.

ffmpeg’s RTP demuxer should deal with out-of-order packets, but if there are dropped packets, you’ll end up with gaps in the PTS timestamps that are emitted from the demuxer. Like ibc said, this isn’t a MediaSoup issue, it’s just a reality of processing RTP. You’ll have to account for these gaps if you’re trying to mix with other streams.

I only work with audio. I need to add aresample=async=1 to any RTP streams before I do any other filtering. This will resample the audio, inserting silent audio frames where there are PTS gaps. You’ll probably need a similar strategy for filling in missing video frames.

Also, the PTS stream coming out of the RTP demuxer should always start at 0. I don’t think the asetpts=PTS-STARTPTS filter is necessary. It’s probably not hurting though.

1 Like