Processing Video Serverside and streaming via ffmpeg

Integration
1

I am tryping to process a video on the server side using Mediasoup PlainRTPTransport, i am piping my RTP stream to ffmpeg to gather the image frames


  const cmdArgStr = [
    "-nostdin",
    "-protocol_whitelist file,rtp,udp",
    "-fflags +genpts",
    `-i ${`configs/input-${audioPortRtp}-vp8.sdp`}`,
    "-vf fps=30",
    "-f image2pipe", // Output as image pipe
    "-",
  ]
    .join(" ")
    .trim();

  console.log(`Run command: ${cmdProgram} ${cmdArgStr}`);

This is how i convert my rtp stream to image Frame datas ,

Later i process it via tfjs , and conver it back to an image Buffer, Now i wanna conver this image back to a video and inject it into my media soup router.

I followed this tutorial 

const ffmpeg = spawn(FFmpegStatic, [
   "-f",
   "image2pipe",
   // "-framerate",
   // "30",
   "-i",
   "-", // Read input from stdin
   "-vf",
   "format=yuv420p", // Required for RTP streaming
   "-c:v",
   "libvpx", // Video codec
   "-preset",
   "ultrafast", // Preset for faster encoding
   // "-tune",
   // "zerolatency", // Tune for zero latency
   "-f",
   "rtp", // Output format RTP
   "-ssrc",
   "22222222",
   "-payload_type",
   "96",
   `rtp://127.0.0.1:${videoRTPPort}?rtcpport=${videoRTCPPort}`,
   // "tee",
   // `[select=v:f=rtp:ssrc=22222222:payload_type=96]rtp://127.0.0.1:${videoRTPPort}?rtcpport=${videoRTCPPort}`,
 ]);

i am using this configuration for streaming my local image as rtp stream , I’ve checked the stream with ffplay , it seems to work fine , but when i try to view the steram on the clinet side it just doent work and i get a empty video loading …


  const videoTransport = await router.createPlainTransport({
      listenIp: "127.0.0.1",
      rtcpMux: false,
      comedia: true,
    });

    // Read the transport local RTP port.
    const videoxRtpPort = videoTransport.tuple.localPort;
    // Read the transport local RTCP port.
    const videoxRtcpPort = videoTransport.rtcpTuple?.localPort ?? 0;
    console.log({ videoxRtpPort, videoxRtcpPort });
 const videoxProducer = await videoTransport.produce({
      kind: "video",
      rtpParameters: {
        codecs: [
          {
            mimeType: "video/VP8",
            clockRate: 90000,
            payloadType: 96,
            rtcpFeedback: [], // FFmpeg does not support NACK nor PLI/FIR.
            parameters: {
              "x-google-start-bitrate": 1000,
            },
          },
        ],
        encodings: [{ ssrc: 22222222 }],
      },
    });
    Room.getRoom(roomName)?.addProducer(
      getId(socket) + "altered",
      videoxProducer,
      "video"
    );
2 Likes