audio-video desync

mediasoup libraries
1

First off, I’m not entirely sure if this is a mediasoup issue or a webRTC issue, but any help would be greatly appreciated.

When attempting to do a long-term session of screen sharing, I find that the audio and video go out of sync intermittently. It does self-correct sometimes, but more often than not, there is clear de-sync. The video is almost always lagging behind audio. So far, I have tried VP8 and H264 for the video codec and while H264 is more resilient to de-sync, it still occurs at some point.

Initially, I thought this might be a CPU bottleneck, since video encoding/decoding takes time, but the de-sync occurs even on a modern CPU (Ryzen 9 3900X) under minimal load.

What could be causing this? Are there any parameters/codecs/settings that I can use to minimize the de-sync?

2

What browser are you using?

3

This happens to consumers on both, Chrome and Firefox.
The producer is on Chrome, since only Chrome supports sharing screen with audio. I believe both browsers are at their latest stable versions.

4

I have not measured a delay between video and audio when screen sharing. Do you get the same result when sharing camera?

5

No, at this point, we’ve done hundreds of hours of webcam and have not observed any noticeable de-sync.

That said, I will also point out that my webcam video resolution was 320x240, whereas the screen resolution was at 1920x1080.

6

Try setting a lower resolution. Or better yet, turn on ‘dtx’ in the encodings options of the producer, and set a lower frame rate say 5 fps. I think the decoding of 1080p video in real time vs audio which is much easier is the issue.

7

Chrome can just sync a received audio and a received video from the same source, so if the screen sharer is sending also mic, there is a good chance for audio and video from sharing to not be rendered in sync.

8

I’m not sure what ‘same source’ means here, but I am getting them both from one navigator.mediadevices.getDisplayMedia call:

navigator.mediaDevices.getDisplayMedia({
  audio: {
    noiseSuppression: false,
    autoGainControl: false,
    echoCancellation: false,
    latency: 0,
    sampleRate: 48000
  },
  video: {
    width: 1920,
    height: 1080,
    frameRate: 30,
    latency: 0
  }
})

Is this not considered as the same source?

9

Try with just mic and webcam. There is no desync, right? mediasoup is not responsible if Chrome does not send sharing audio and video in sync.