I’ve been researching on ways to smooth out video and audio in a mediasoup router, potentially those that make mediasoup reselient against network congestions and other disruptions considering the fact that now people are streaming and having meetings more than ever.
From what I understand this requires changes to the sdp and appending a line that configures it on the sender’s side. How would one go about adding this to mediasoup and testing it out, because I believe it can have a major impact on perceived performance, especially in very low bandwidth scenarios.
And since this is related to WebRTC and not necessarily related to mediasoup, I’ve chosen to post this in the ‘Off Topic’ group.
Any and all the help / questions / critique will be appreciated.
Not off topic. This would indeed requiere additions in mediasoup (to support the new RTP header extension) and in mediasoup-client (an API to set/read it).
Awesome. I would love to work on this and add it, this in my opinion will be a big win. But I think there is going to be some waiting before this gets any further, even if someone implemented this in mediasoup there still needs to be support on the clients.
Chrome is by far the only one that has added support for this. I did P2P tests where I streamed a video from my phone to a Chrome client and using the ‘playoutDelayHint’ I was able to smooth the jittery video out using a delay of 5 seconds.
I think there’s a way to get the underlying peer connection object for a consumer. I hope to do a Chromium test for this on mediasoup, I should add a ‘playoutDelayHint’ to the receiving peer connection objects and see if that makes any difference.
Feel free to create a PR in mediasoup server if you work on this. Take as example the processing of other RTP header extensions if you wish.
This is not done in the PeerConnection but in the specific RTCRtpReceiver, and there is already an API to retrieve the RTCRtpReceiver of a Consumer in mediasoup-client.
BTW what I meant (in addition to that) is "an API to configure such a delay hint in the RTP sender, this is, in the mediasoup-client Producer class.
The only problem I see with that is currently in the docs I’ve not come across a way to set this parameter from the RTCRtpSender. the ‘playoutDelayHint’ is only available on chrome and can be set on the receiver.
I’m reading RFCs and docs and trying to figure out a way to attach the RTP header that will set this from mediasoup as it sends data out to the consumers. But that too is an assumption, I don’t have the knowledge yet to check weather the RTP header is supported on all clients specially ‘Safari’ - iOS is hectic af when it comes to support.
The craziest way that I can think of is packaging the encoder / decoder as a web assembly module and using data channels to send this data off and using canvas to render the video… That’s the only sure way of surely knowing that we have support for all that shit…
It would be great if you can point me to an implementation for this in the mediasoup - server side - library. I have configured the build environment locally I’m now going through code to understand where the changes will be required.
Honestly, telling you to how to implement this in mediasoup would take longer for me than implementing it. The problem is that we (mediasoup developers) are kinda busy nowadays with other priorities.
If you wish, create a feature request in mediasoup GitHub by referencing the corresponding draft/spec and current implementation in libwebrtc and/or browsers.
That’s not what I meant. You’ve said that I can look into the implementations for other header extensions and how MS handles that, I thought you might have a link to some resource that might point me in the right direction.
No worries though. Its all about the will, I’ll try my best. Thanks for all the advice. Much appreciated
There is no “link to resources” but just source code that deals with many other RTP header extensions in mediasoup. They are mainly in Producer.cpp and Transport.cpp.
That’s the only sure way of surely knowing that we have support for all that shit…
