I have an abnormal use case wherein I’d like to perform graphically intensive video post-processing entirely server-side to minimize the burden on client devices. While an MCU architecture is what’d probably work best in concept, I wanted to take advantage of the UI flexibility of mediasoup’s RFU implementation. Can we modify the video data in the router?
The only other requirement is to get a client to receive its own post-processed video. I.e. stream my face through a webcam to the server, have the server apply some effects to the video, and then send it out to all clients including back to the original client. Looking through the docs it seems like this should be possible since the API is pretty flexible as producer/consumers are ID-based.
Well, you will not be able to do this directly as mediasoup is SFU, it just receives the packet and routes them without decoding them, so you will not be able to do post video processing directly.
You should keep the processing of camera streams on client side at it’s most. Like you may want to change background of video then you should do it on client side there is not reason for doing this on server side, it will make it complicated.
If you want something like recording the call then you should receive the streams from mediaosup server on separate server as a client just like a normal ghost participant and then merge those streams using canvas and record it. You can use headless browser and puppeteer library for this purpose.
You can use ffmpeg, gstreamer etc as well to processing as well.
There is only one way which is to get the raw packets from mediasoup through direct or plain transport (not sure which one) and then consume them in ffmpeg, gstreamer, headless browser and process them and inject them back to mediasoup in the same way and then mediasoup will router it to the participants.
One simple way of doing this in your case can be that you use headless browser on server and then consume the video which you want to process and then process it using html canvas etc and then reproduce that stream from here to mediasoup again and then consume this processed stream in the rest of the participants just like normal stream.
There are plenty of resources available you can google for ‘headless browser’ to know about it. You will be able to open page from headless browser then the rest of the process is same you will join the call from headless browser as a normal user the only difference will be that it will be kind of a bot and will not be visible to other participants. Once you are in there then get the streams you want and then draw on canvas etc to record them.