Hi, we’ve been told that some users are experiencing delayed video when consuming it, regardless simulcast/SVC is used or not. This is, received video is delayed for many seconds behind the audio.
Recently there were some RTCP improvements and fixes in mediasoup and we don’t know if there is some regression in those changes that may cause the above issue:
I have been, you can test my website if you need to; it would appear once I get around 60% CPU on 4vCORE at about 120Mbp/s outgoing there’s hiccups in video (audio seems fine).
Just PM I’ll share what I need to, only occurs when I start loading up the server usage.
My network is capable of 500Mbp/s each server worse case, and CPU is not exceeded, the trace-route is always 100ms to much of the users.
I was on early versions 3.6.35 before doing a major jump in builds, I did update the other day pushing an update say 7 days ago now, this did not appear to fix any problems if this helps any.
The delay in video is a sub-second to few seconds while audio remains fine; as said only happens when I start pushing my cores. I’m strictly VP8 @ 200KB/s each stream and do use Secure PipeTransports.
I wouldn’t know if there’s other valuable input. But suspicions match up here.
Does the issue just happen with a certain number of video consumers? or does it just depend on CPU usage?
Do you mean that you were on version 3.6.5 without this issue and then you updated to (which version exactly?) and the delayed video problem shown up since then?
Tough to say if this by consumers, some rooms could be performing a 1x2 or 3x3 or some doing 24x48. Maybe it’s a producer issue? Only was able to notice this when CPU overall is a bit high but not maxed out by far.
Yeah no issues on version 3.6.35; Funny enough I am running a normal domain with that running still. I am working on an alpha site so testing all latest stuff now and mediasoup version 3.10.11
Really unsure what’s causing it, but in meantime I had routed a few servers and piped them to minimize it; but if I catch it tonight (only 7AM; users just waking up) I’ll try to capture some ideas/numbers maybe what’s going on. The traffic is around 100-300 users concurrent nightly so very easy to replicate.
I would like to know how I can help, I’m not very comfortable with debugging these processes yet without assistance but willing to learn what’s necessary if there’s extra details you require. Just let me know!
(Just thinking if this is in production, only fix is running more servers; I’d love to help very much so however I can possible)
You’re in luck, I was doing reboots this morning and found a chance to take a few servers offline to revert them and switch users over slowly to test 3.10.8 for you.
I’ll probably have results by tonight if it solves the problem. Not easy getting off-peak chances like this, but no issues anything to help out.
I have recently experienced this but cannot reproduce consistently. I experienced this while we were doing bandwidth adaptation testing. We are still on version 3.7.17.
The setup for testing was as such:
producer sending stream at 5MB bitrate
consumer receiving at 5MB
While streaming, used router to constrain producer bandwidth to 1MB
Experienced some freezing/video glitching, then video resumed at lower bitrate, but ~10 seconds behind
I removed bandwidth constraint on producer
Video quality upgraded/resumed - however, the video delay remained, consumer was receiving video ~10 seconds delayed
Disconnecting/reconnecting the consumer fixed the delay issue
My belief is that something was going wrong with some queue delivering the packets/frames to the consumer on the SFU.
The server was NOT CPU constrained (was the only session on it).
It is difficult to reproduce.
Would be helpful if you could point me at some metric / buffer / queue size or anything else to monitor to try and catch this along with some server evidence of bad behavior.
We are about to confirm that there is no real regression in mediasoup in latest versions. Our issue was related to the addition of an experimental RTP header extension in our Electron based app. Your issue is just related to the fact that BWE in mediasoup requires lot of love as documented many times in the forum. I mean, the same issue would have happened to you with any version of mediasoup as soon as you start doing those tests limiting the available bandwidth.
Are you suggesting we raise our bitrates to lessen video pausing? Currently at 200KB/s was actually trying to drive bitrate as low as possible for users prior 500KB/s.
I’m taking the chance right now and boosting it to 500KB/s users will hate it if they got crap internet, but I’ll report once servers fill up (updating this post) if there’s improvement. So far got them 20-40% 3 servers no issues except users who lag naturally.
Tested! Those lagging lag. Those who have good connections stay fluid. Some hiccups but less noticeable; Increasing bitrate helped but 500KB/s for me is 5-20TB a day per 4vCore server, can we do any better?
I’m all for keeping it low but good, user’s cam at any resolution but limited to bitrate + camera box they’re set in.
To clarify: the original issue described in this topic “delayed video” was supposed to mean lip sync problems (received audio and video are out of sync).
We have identified the issue and it’s not happening to any user but just in our Electron based application since we enable abs-capture-time RTP header extension, which is not enabled in Chrome or libwebrtc by default, so I insist: nobody is affected by this issue but just those that customize libwebrtc C++ code into their apps to enable such a RTP header extension.
The problem you are describing is not related to this, but to the bandwidth estimation (congestion controller, etc) in mediasoup, which is a known issue we are working on. We cannot fix that issue within this topic because it is unrelated.
I thought my issue was relevant, BWE could use correction I do see this but I don’t think it ties to my issue fully. Everything is fine with many transports and consumers, but as soon as it gets to a larger volume it almost seems as the loops are code-blocking in ways where things aren’t on t ime or discarded. Given Audio transmits perfectly still though I’m wondering if packets are just being tossed out if they exceed a buffer limit/etc unsure haven’t gone debugging yet.
A bit confusing, but noticeable once I get many producers/consumers going. I do PipeTransport remotely often but never seeing issues between the pipes latency wise/etc.
If this is BWE related, I’ll drop it but almost unsure cause my server and clients can handle the rates no issues.
The topic was about a specific possible regression just in latest versions of mediasoup and it doesn’t require simulcast or complex scenarios or high load to be reproduced. As explained before this was relayed to the usage of an experimental abs-capture-time RTP header extension in our application (the problem is in libwebrtc and not in mediasoup). Your perf problem is not related to this false issue or recent versions. Please report it separately in a new topic if you wish.
No need, my problem turned out to be very silly, I wasn’t testing enough end-points of network quality and found a few end-points operating at 50% packet loss. I had updated the night before this post and seeing some freezing I thought this was the issue.
Thanks for explaining. Apologies for asking, it’s brilliant work and I think I have a bit more time ahead of me before I get my head wrapped around it all.
