# Processing video server-side, inside a router? **URL:** \ **Category:** mediasoup libraries\ **Created:** [February 15, 2023, 11:06pm UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959 "2023-02-15T23:06:29Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![dksdev](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dksdev](https://mediasoup.discourse.group/u/dksdev)\ **Post date:** [February 15, 2023, 11:06pm UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/1 "2023-02-15T23:06:29Z")
I have an abnormal use case wherein I’d like to perform graphically intensive video post-processing entirely server-side to minimize the burden on client devices. While an MCU architecture is what’d probably work best in concept, I wanted to take advantage of the UI flexibility of mediasoup’s RFU implementation. Can we modify the video data in the router? The only other requirement is to get a client to receive its own post-processed video. I.e. stream my face through a webcam to the server, have the server apply some effects to the video, and then send it out to all clients including back to the original client. Looking through the docs it seems like this should be possible since the API is pretty flexible as producer/consumers are ID-based. ---
**Author:** ![zaidiqbal](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/zaidiqbal/32/1498_2.png) [@zaidiqbal](https://mediasoup.discourse.group/u/zaidiqbal)\ **Post date:** [February 16, 2023, 6:36am UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/2 "2023-02-16T06:36:18Z")
Well, you will not be able to do this directly as mediasoup is SFU, it just receives the packet and routes them without decoding them, so you will not be able to do post video processing directly. You should keep the processing of camera streams on client side at it’s most. Like you may want to change background of video then you should do it on client side there is not reason for doing this on server side, it will make it complicated. If you want something like recording the call then you should receive the streams from mediaosup server on separate server as a client just like a normal ghost participant and then merge those streams using canvas and record it. You can use headless browser and puppeteer library for this purpose. You can use ffmpeg, gstreamer etc as well to processing as well. ---
**Author:** ![dksdev](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dksdev](https://mediasoup.discourse.group/u/dksdev)\ **Post date:** [February 16, 2023, 5:27pm UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/3 "2023-02-16T17:27:42Z")
Without getting into detail, the video processing we do is really performance intensive for phones, so I cant change the requirement. Is there no way to modify the video before mediasoup routes them? ---
**Author:** ![zaidiqbal](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/zaidiqbal/32/1498_2.png) [@zaidiqbal](https://mediasoup.discourse.group/u/zaidiqbal)\ **Post date:** [February 16, 2023, 5:37pm UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/4 "2023-02-16T17:37:29Z")
There is only one way which is to get the raw packets from mediasoup through direct or plain transport (not sure which one) and then consume them in ffmpeg, gstreamer, headless browser and process them and inject them back to mediasoup in the same way and then mediasoup will router it to the participants. One simple way of doing this in your case can be that you use headless browser on server and then consume the video which you want to process and then process it using html canvas etc and then reproduce that stream from here to mediasoup again and then consume this processed stream in the rest of the participants just like normal stream. Some things are described here: > [@Server-side processing of WebRTC streams](https://mediasoup.discourse.group/t/server-side-processing-of-webrtc-streams/4284): > > Hello everyone I’ve been using mediasoup for some time now, and I have to say, it’s really a life saver. I just want to know if it is possible, while using mediasoup to perform some AI processing on the video & audio streams using SDKs such as NVIDIA Maxine, on the server. My idea of going about it was to use a PlainRTPTransport to obtain the raw stream data and pass to the SDK, and then get the output from the SDK, and re-inject it into mediasoup using another PlainRTPTransport, but I can’t se… > [@How can I record in another computer instead of the computer which the medaisoup server is running?](https://mediasoup.discourse.group/t/how-can-i-record-in-another-computer-instead-of-the-computer-which-the-medaisoup-server-is-running/2650): > > There is a problem when I use ethand91/mediasoup3-record-demo([GitHub - ethand91/mediasoup3-record-demo: Simple Record Demo using Mediasoup 3 and GStreamer](https://github.com/ethand91/mediasoup3-record-demo)) to record.But the problem is not his demo,I really helpless that I ask for help here. I changed his demo,separate the ffmpeg part,becuse I want to record in another computer instead of the computer which the mediasoup server is running.The code ran well and I got video in the computer which the server is running,but I can’t got video in anot… > [@Get raw audio from RTP packet with DirectTransport](https://mediasoup.discourse.group/t/get-raw-audio-from-rtp-packet-with-directtransport/2715): > > Hi! does anyone have experience manipulating the RTP packets with DirectTransport with rtp.js library? I would like to extract the raw audio of the specific producer to manipulate in Nodejs const room = Room.getRoom(roomId); const router = room.router; const directTransport = await router.createDirectTransport(); const audioProducer = getProducer(roomId, id, 'audio'); const consumer = await directTransport.consume({ producerId: audioProducer.id, rtpCapab… > [@Recorded Audio (FFMPEG or GStreamer) removes gaps/ silent phases](https://mediasoup.discourse.group/t/recorded-audio-ffmpeg-or-gstreamer-removes-gaps-silent-phases/2930): > > I am testing recording of video and audio by creating a plain transport which is then consumed by FFMPEG process spawned by my app. When I play the recorded video, I see that the audio seems to jump ahead and gets out of sync with Video over time. The longer the video, the larger the displacement/ desync. To debug this further, I removed the video producer/ consumer and just recorded the audio stream. I tested the audio by speaking every 5 seconds over a 30 second period and when I play the rec… ---
**Author:** ![mathieu](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/mathieu/32/1691_2.png) [@mathieu](https://mediasoup.discourse.group/u/mathieu)\ **Post date:** [August 30, 2023, 7:49am UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/5 "2023-08-30T07:49:00Z")
@dksdev were you able to get this to work? ---
**Author:** ![zaidiqbal](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/zaidiqbal/32/1498_2.png) [@zaidiqbal](https://mediasoup.discourse.group/u/zaidiqbal)\ **Post date:** [August 30, 2023, 7:58am UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/6 "2023-08-30T07:58:28Z")
this is the way to go: > [@Processing video server-side, inside a router?](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/4): > > There is only one way which is to get the raw packets from mediasoup through direct or plain transport (not sure which one) and then consume them in ffmpeg, gstreamer, headless browser and process them and inject them back to mediasoup in the same way and then mediasoup will router it to the participants. One simple way of doing this in your case can be that you use headless browser on server and then consume the video which you want to process and then process it using html canvas etc and then… ---
**Author:** ![mathieu](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/mathieu/32/1691_2.png) [@mathieu](https://mediasoup.discourse.group/u/mathieu)\ **Post date:** [August 30, 2023, 8:18am UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/7 "2023-08-30T08:18:20Z")
Got it, thanks. Is there an example somewhere by any chance? ---
**Author:** ![zaidiqbal](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/zaidiqbal/32/1498_2.png) [@zaidiqbal](https://mediasoup.discourse.group/u/zaidiqbal)\ **Post date:** [August 30, 2023, 8:30am UTC](https://mediasoup.discourse.group/t/processing-video-server-side-inside-a-router/4959/8 "2023-08-30T08:30:24Z")
There are plenty of resources available you can google for ‘headless browser’ to know about it. You will be able to open page from headless browser then the rest of the process is same you will join the call from headless browser as a normal user the only difference will be that it will be kind of a bot and will not be visible to other participants. Once you are in there then get the streams you want and then draw on canvas etc to record them.