I have a proprietary app that has a WebRTC (m83) module which is used to broadcast a screencast of the app’s window. Instead of the app’s built-in WebRTC module I want to use the mediasoup suite.
I am able to access the origin of the screencast and I successfully hooked it to libmediasoupclient which sends the stream to mediasoup. Then I consume it in mediasoup-client in a modern browser. This works well.
But the app’s screencast setup seems to be optimized for its case - the video being static most of the time: if there is no change from the previous frame, it just displays black frames. In a way, this is similar to mediasoup-client’s “zeroRtpOnPause”. My libmediasoup enabled application just sends whatever it receives from the proprietary app, so there’s a black screen most of the time in the browser.
Do you know if this a default optimization of WebRTC for sending screencasts (VideoTrackSource->is_screncast(): true)?
Ideally I would like to take advantage of this optimization and adapt my setup such that the screen image in mediasoup-client is updated only when there’s a new non-black frame. Otherwise it should just display the last received frame.
Is this possible, or do I need to take care of the optimization at the application level?
There is a wrong assumption above. Temporal lack of RTP does not mean that the receiver should render black frames. That’s not true. If there is no RTP for a while the receiver should render the last received frame, and that’s what video decoders do.
So I don’t know what you mean with “black frames”.
The proprietary app is providing me “black frames” when there is no change from the previous frame. Somehow the current implementation of the app’s WebRTC module accounts for this. I want to do the same and I don’t know yet how.
Indeed, this is that I think is happening here as well. The proprietary app’s WebRTC module simply doesn’t deliver a new frame to the encoder if it is unchanged.
I have managed to get the same behaviour in my libmediasoupclient app by delivering a new frame from my custom video capturer only if it has changed from the previous frame.
Mediasoup complains (righthfully) about this:
mediasoup:WARN:Channel [pid:4289] RTC::RtpStreamRecv::OnTimer() | RTP inactivity detected, resetting score to 0 [ssrc:3035133469] +3m
mediasoup:Channel [pid:4289] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:676617052, media ssrc:676617052] +1ms
(and no key frame is sent until the frame actually changes)
And in spite of this, there is still a black screen displayed sometimes in the browser. But I am sure there is no black frame sent now.
It actually might not even be the encoding side. The machine where I’m running the webclient seems overloaded at times. Could it be that Chrome just decides not to display the last frame if the system is overloaded?
Unfortunately still no success. The image just flickers back to a black image when there are no new frames. CPU usage seems fine now. But I still see requests for keyframes in mediasoup logs
mediasoup:Channel [pid:13480] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:272601360, media ssrc:272601360] +940ms
mediasoup:Channel [pid:13480] RTC::RtpStreamRecv::RequestKeyFrame() | sending PLI [ssrc:1617147820] +1ms
mediasoup:Channel [pid:13480] RTC::RtpStreamRecv::RequestKeyFrame() | sending PLI [ssrc:1617147820] +999ms