# "re-encoder" server side / keyFrameRequestDelay 5s

**URL:** <https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790>\
**Category:** mediasoup libraries\
**Created:** [April 12, 2020, 10:08pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790 "2020-04-12T22:08:11Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kaplan](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/kaplan/32/1116_2.png) [@Kaplan](https://mediasoup.discourse.group/u/Kaplan)\
**Post date:** [April 12, 2020, 10:08pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/1 "2020-04-12T22:08:11Z")

</div>

Hi, Just spent the weekend contemplating migrating a project from Janus to MediaSoup. I read through the docs, and watched the JanusCon presentation. Love how well though out MediaSoup is. Congrats! Great work!  
My first task is to create a large few-to-many broadcast that scales beyond a single host. I’ve read through the scalability section, especially the “Red” section [https://mediasoup.org/documentation/v3/scalability/](https://mediasoup.org/documentation/v3/scalability/)  
I have 2 question that I am hoping somebody here can answer:

1. has the server side “re-encoder” been written??? if so, is it available somewhere?
2. I am already used to having upto 5 seconds of “black screen” by shielding the presenter with a max PLI rate of 5s. This is acceptable for my use case. Does keyFrameRequestDelay work for this?

---

<div class="post-metadata">

**Author:** ![ibc](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/ibc/32/1540_2.png) [@ibc](https://mediasoup.discourse.group/u/ibc)\
**Post date:** [April 12, 2020, 11:25pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/2 "2020-04-12T23:25:30Z")

</div>

> [@Kaplan](#):
>
> as the server side “re-encoder” been written??? if so, is it available somewhere?

There is no a “promised reencoder that we will write”. However you can use libmediasoupclient C++ library for that.

> [@Kaplan](#):
>
> I am already used to having upto 5 seconds of “black screen” by shielding the presenter with a max PLI rate of 5s. This is acceptable for my use case. Does keyFrameRequestDelay work for this?

Yes.

---

<div class="post-metadata">

**Author:** ![novacreator](https://avatars.discourse-cdn.com/v4/letter/n/df788c/32.png) [@novacreator](https://mediasoup.discourse.group/u/novacreator)\
**Post date:** [July 2, 2021, 6:31am UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/3 "2021-07-02T06:31:15Z")

</div>

Hi guys. I will also need to implement such an re encoder. But it this point the documentation really falls short. Leaving you at a point where one does not know how to go further. What is this re encoder? Hiw can it be implemented? I have no freaking idea…

---

<div class="post-metadata">

**Author:** ![copiltembel](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@copiltembel](https://mediasoup.discourse.group/u/copiltembel)\
**Post date:** [July 2, 2021, 8:02pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/4 "2021-07-02T20:02:06Z")

</div>

Initially you say you need such a re encoder. Then you go on and ask:

> [@novacreator](#):
>
> What is this re encoder?

So you need it although you don’t know what it is?!

Do re-check the documentation. It’s very clear IMHO what it is and how you COULD implement such a reencoder.

---

<div class="post-metadata">

**Author:** ![novacreator](https://avatars.discourse-cdn.com/v4/letter/n/df788c/32.png) [@novacreator](https://mediasoup.discourse.group/u/novacreator)\
**Post date:** [July 3, 2021, 7:30pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/5 "2021-07-03T19:30:31Z")

</div>

As far as i understand this re encoder is somewhere beteween the producer and consumer and has to regenerate keyFrames as requested by the client. It is then itself a consumer and produce? What does it has to do with tge c++ lib? Do i have ti write my own video encoder in c++ to get this thing running?

---

<div class="post-metadata">

**Author:** ![BronzedBroth](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@BronzedBroth](https://mediasoup.discourse.group/u/BronzedBroth)\
**Post date:** [July 3, 2021, 7:43pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/6 "2021-07-03T19:43:13Z")

</div>

What does it has to do with the c++ lib?  
Lots…

You will need to write your own process but you can do so with the help of the given library. Mediasoup documents well but for these purposes they’re left in the developers hands to create what it is they exactly want. There’s no pre-mades outside the demo’s.

---

<div class="post-metadata">

**Author:** ![ibc](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/ibc/32/1540_2.png) [@ibc](https://mediasoup.discourse.group/u/ibc)\
**Post date:** [July 3, 2021, 10:50pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/7 "2021-07-03T22:50:09Z")

</div>

The reencoder is or maybe a libmediasoupclient based app that consumes a track from a real client (via webrtcTransport.consume() in mediasoup server), gets the consumer.track and passes it to webrtcTransport.produce() and injects it back in mediasoup for real clients to consume it, so it’s decoding and encoding again. It may even consume VP8 and reencode it using H264 or whatever.

---

<div class="post-metadata">

**Author:** ![gurupras](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/gurupras/32/738_2.png) [@gurupras](https://mediasoup.discourse.group/u/gurupras)\
**Post date:** [October 2, 2022, 10:21am UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/8 "2022-10-02T10:21:35Z")

</div>

Has anyone managed to write up an re-encoder? I don’t understand enough to build one out 🙁

Would I be correct in assuming that this cannot be written using the nodeJS APIs? I say this because I don’t find a method of extracting the track from a consumer.

---

<div class="post-metadata">

**Author:** ![kwindla](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/kwindla/32/41_2.png) [@kwindla](https://mediasoup.discourse.group/u/kwindla)\
**Post date:** [October 2, 2022, 7:32pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/9 "2022-10-02T19:32:59Z")

</div>

Hi Kaplan. How many is “many” in your use case. In general, you can scale to quite a large number of receiving clients before you need a re-encoder. This is dependent on your `encodings` rtp parameters, goals, and use case, of course. But especially if you’re willing to set `keyFrameRequestDelay` to 5 seconds, you may not need to re-encode.

---

<div class="post-metadata">

**Author:** ![BronzedBroth](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@BronzedBroth](https://mediasoup.discourse.group/u/BronzedBroth)\
**Post date:** [October 2, 2022, 11:20pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/10 "2022-10-02T23:20:12Z")

</div>

> [@kwindla](#):
>
> But especially if you’re willing to set `keyFrameRequestDelay` to 5 seconds, you may not need to re-encode.

No?

> [@gurupras](#):
>
> Would I be correct in assuming that this cannot be written using the nodeJS APIs? I say this because I don’t find a method of extracting the track from a consumer.

No? Many references on using FFMPEG/GStreamer.

As for write-ups there’s a few however your app may vary from others but there’s plent of FFMPEG/GStreamer usage examples to get you started. Process is transcoding, you decode and encode stream.

---

<div class="post-metadata">

**Author:** ![gurupras](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/gurupras/32/738_2.png) [@gurupras](https://mediasoup.discourse.group/u/gurupras)\
**Post date:** [October 2, 2022, 11:24pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/11 "2022-10-02T23:24:49Z")

</div>

Assuming I’m not changing encoding formats or any of that, is that really all that I would need to do? Somehow feed a WebRTC-transport-based producer into FFMPEG, get the resulting RTP stream into mediasoup via plainRTP-transport and advertise/use this producer instead of the WebRTC one?

What about the parts where it receives+reacts to a PLI/FIR? Do I get that for free via FFMPEG and plainRTP-transport? Sorry, very new to all of this

---

<div class="post-metadata">

**Author:** ![kwindla](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/kwindla/32/41_2.png) [@kwindla](https://mediasoup.discourse.group/u/kwindla)\
**Post date:** [October 2, 2022, 11:56pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/12 "2022-10-02T23:56:55Z")

</div>

> What about the parts where it receives+reacts to a PLI/FIR? Do I get that for free via FFMPEG and plainRTP-transport? Sorry, very new to all of this

Mediasoup will handle the PLIs for you.

Last time I looked at the ffmpeg rtp source code, it wasn’t possible to configure it to handle FIRs. (That could have changed, of course.)

It’s definitely possible to wire up RTCP FIR stuff back to the GStreamer encoding pipeline. In theory it’s fairly simple – both GStreamer and Mediasoup just need to be configured to know which RTCP options to enable and need to know the right ports. But figuring out the details might require reading through the `rtpsession` or `rtpbin` module source code to figure out (I don’t remember where the RTCP code is). If this project is important to you, you might want to consider hiring [Asymptotic](https://asymptotic.io/), who do GStreamer and WebRTC consulting, have reasonable rights, are very professional, and really, really know their stuff.

---

<div class="post-metadata">

**Author:** ![BronzedBroth](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@BronzedBroth](https://mediasoup.discourse.group/u/BronzedBroth)\
**Post date:** [October 3, 2022, 12:18am UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/13 "2022-10-03T00:18:27Z")

</div>

You tell us, I think you’ll find your answer quickly. 🙂

---

<div class="post-metadata">

**Author:** ![gurupras](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/gurupras/32/738_2.png) [@gurupras](https://mediasoup.discourse.group/u/gurupras)\
**Post date:** [October 3, 2022, 12:45pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/14 "2022-10-03T12:45:41Z")

</div>

I’m sharing this here in case someone finds this useful.  
It’d be very helpful if someone can provide feedback on whether what I’m doing even makes any sense… I definitely lost track of all of the ports more than once.

Note, if this even works, this is most likely brittle and probably only handles VP8 and/or H264. Let me know if I should put this up elsewhere so folks can contribute to making it better.

> **ReEncoder.js**
>
> ```nohighlight
> import execa from 'execa'
> import { getPort, releasePort } from './utils'
> 
> export default class ReEncoder {
> /**
> *
> * @param {import('mediasoup').types.Producer} producer
> * @param {import('mediasoup').types.Router} router
> */
> constructor (producer, router) {
> /** @type {import('mediasoup').types.Producer} */
> this.producer = producer
> /** @type {import('mediasoup').types.Router} */
> this.router = router
> 
> /** @type {import('mediasoup').types.PlainTransport} */
> this.plainRTPConsumerTransport = undefined
> /** @type {import('mediasoup').types.PlainTransport} */
> this.plainRTPProducerTransport = undefined
> 
> /** @type {import('mediasoup').types.Consumer} */
> this.consumer = undefined
> 
> /** @type {number} */
> this.remoteRTPPort = undefined
> /** @type {number} */
> this.remoteRTCPPort = undefined
> /** @type {number} */
> this.localRTCPPort = undefined
> 
> /** @type {execa.ExecaChildProcess<string>} */
> this.process = undefined
> }
> 
> async start () {
> this.producer.observer.on('close', async () => {
> this.plainRTPConsumerTransport?.close()
> this.plainRTPProducerTransport?.close()
> })
> this.plainRTPConsumerTransport = await router.createPlainTransport({
> listenIp: '127.0.0.1',
> comedia: false,
> rtcpMux: false,
> appData: {
> reencoder: true,
> forProducer: this.producer.id
> }
> })
> 
> const remoteRTPPort = this.remoteRTPPort = await getPort()
> const remoteRTCPPort = this.remoteRTCPPort = await getPort()
> await this.plainRTPConsumerTransport.connect({
> ip: '127.0.0.1',
> port: remoteRTPPort,
> rtcpPort: remoteRTCPPort
> })
> this.localRTCPPort = this.plainRTPConsumerTransport.tuple.localPort
> this.plainRTPConsumerTransport.observer.on('close', () => {
> releasePort(remoteRTPPort)
> releasePort(remoteRTCPPort)
> })
> 
> const codecs = []
> const routerCodec = this.router.rtpCapabilities.codecs.find(x => x.kind === this.producer.kind)
> codecs.push(routerCodec)
> 
> this.consumer = await this.plainRTPConsumerTransport.consume({
> producerId: this.producer.id,
> rtpCapabilities: this.router.rtpCapabilities,
> paused: false,
> appData: {
> forProducer: this.producer.id,
> reencoder: true
> }
> })
> 
> this.consumer.observer.on('close', async () => {
> this.process?.kill('SIGKILL')
> })
> 
> this.plainRTPProducerTransport = await router.createPlainTransport({
> listenIp: '127.0.0.1',
> rtcpMux: false,
> comedia: true
> })
> 
> await this.createReEncoder()
> 
> // await this.consumer.resume()
> 
> this.reEncodedProducer = await this.plainRTPProducerTransport.produce({
> kind: this.producer.kind,
> rtpParameters: this.consumer.rtpParameters,
> paused: false,
> appData: {
> ...this.producer.appData,
> forProducer: this.producer.id,
> reencoder: true,
> pid: this.process.pid
> }
> })
> }
> 
> async createReEncoder () {
> // Get the SDP
> const sdp = createSDP(this.consumer, this.remoteRTPPort, this.remoteRTCPPort)
> // Now start the ffmpeg process
> const cmdlineArray = [
> 'ffmpeg',
> '-y',
> '-hide_banner',
> '-loglevel',
> 'error',
> '-protocol_whitelist',
> 'file,pipe,udp,rtp',
> '-fflags',
> '+genpts',
> '-f',
> 'sdp',
> '-i',
> 'pipe:0'
> ]
> 
> const ssrc = this.consumer.rtpParameters.encodings[0].ssrc
> const payloadType = this.consumer.rtpParameters.codecs[0].payloadType
> 
> let selectChar
> 
> switch (this.producer.kind) {
> case 'video':
> cmdlineArray.push(...[
> '-map',
> '0:v:0',
> '-c:v',
> 'copy'
> ])
> selectChar = 'v'
> break
> case 'audio':
> cmdlineArray.push(...[
> '-map',
> '0:a:0',
> '-strict',
> '-2',
> '-c:a',
> 'copy'
> ])
> selectChar = 'a'
> break
> }
> 
> const {
> plainRTPProducerTransport: {
> tuple: {
> localIp: destIP,
> localPort: destPort
> },
> rtcpTuple
> }
> } = this
> const { localPort: destRTCPPort } = rtcpTuple
> 
> cmdlineArray.push(...[
> '-f',
> 'tee',
> `[select=${selectChar}:f=rtp:ssrc=${ssrc}:payload_type=${payloadType}]rtp://${destIP}:${destPort}?rtcpport=${destRTCPPort}`
> ])
> const cmdline = cmdlineArray.join(' ')
> this.process = execa.command(cmdline, {
> input: sdp
> })
> }
> }
> 
> /**
> *
> * @param {import('mediasoup').types.Consumer} consumer
> * @param {number} rtpPort
> * @param {number} rtcpPort
> * @returns
> */
> export function createSDP (consumer, rtpPort, rtcpPort) {
> const { rtpParameters, kind } = consumer
> const codecInfo = getCodecInfoFromRTPParameters(kind, rtpParameters)
> 
> const sdp = [
> 'v=0',
> 'o=- 0 0 IN IP4 127.0.0.1',
> 's=FFmpeg',
> 'c=IN IP4 127.0.0.1',
> 't=0 0'
> ]
> switch (kind) {
> case 'video':
> sdp.push(...[
> `m=video ${rtpPort} RTP/AVP ${codecInfo.payloadType}`,
> `a=rtpmap:${codecInfo.payloadType} ${codecInfo.codecName}/${codecInfo.clockRate}`
> ])
> break
> case 'audio':
> sdp.push(...[
> `m=audio ${rtpPort} RTP/AVP ${codecInfo.payloadType}`,
> `a=rtpmap:${codecInfo.payloadType} ${codecInfo.codecName}/${codecInfo.clockRate}/${codecInfo.channels}`
> ])
> break
> }
> sdp.push('a=sendonly')
> sdp.push('')
> return sdp.join('\n')
> }
> 
> /**
> *
> * @param {'video' | 'audio'} kind
> * @param {import('mediasoup').types.RtpParameters} rtpParameters
> * @returns
> */
> export function getCodecInfoFromRTPParameters (kind, rtpParameters) {
> return {
> payloadType: rtpParameters.codecs[0].payloadType,
> codecName: rtpParameters.codecs[0].mimeType.replace(`${kind}/`, ''),
> clockRate: rtpParameters.codecs[0].clockRate,
> channels: kind === 'audio' ? rtpParameters.codecs[0].channels : undefined
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![kwindla](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/kwindla/32/41_2.png) [@kwindla](https://mediasoup.discourse.group/u/kwindla)\
**Post date:** [October 9, 2022, 3:16pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/15 "2022-10-09T15:16:02Z")

</div>

In my experience, one of the ways this approach may be brittle is that `+genpts` and `copy` don’t play nicely together.

It’s worth testing heavily with packet loss in the incoming stream to surface timestamp and a/v sync issues.

---

<div class="post-metadata">

**Author:** ![gurupras](https://yyz2.discourse-cdn.com/free1/user_avatar/mediasoup.discourse.group/gurupras/32/738_2.png) [@gurupras](https://mediasoup.discourse.group/u/gurupras)\
**Post date:** [October 16, 2022, 5:12pm UTC](https://mediasoup.discourse.group/t/re-encoder-server-side-keyframerequestdelay-5s/790/16 "2022-10-16T17:12:08Z")

</div>

I tried piping the re-encoded stream from one node to another node and then creating a webrtc consumer for the re-encoded producerID on the second node. I end up with a black video that _eventually_ displays a frame before becoming stuck on that frame. ☹  
If I pipe the original producer, everything works. If I pipe the re-encoded producer, then I get a black screen.

Would really appreciate some help on how to implement a re-encoder using ffmpeg for VP8.  
On the client-side, this is what the inbound RTP looks like:

> **RTCInboundRTPVideoStream\_483466091 (inbound-rtp, mid=0, VP8)**
>
> ```auto
> Statistics RTCInboundRTPVideoStream_483466091
> timestamp	10/16/2022, 1:10:49 PM
> ssrc	483466091
> kind	video
> trackId	DEPRECATED_RTCMediaStreamTrack_receiver_308
> transportId	RTCTransport_0_1
> codecId	RTCCodec_0_Inbound_101
> [codec]	VP8 (101)
> mediaType	video
> jitter	0.002
> packetsLost	2072
> trackIdentifier	ba57c64c-c118-4c7c-9e13-01dc639eb419
> mid	0
> packetsReceived	967059
> [packetsReceived/s]	222.0064575166266
> bytesReceived	1351309984
> [bytesReceived_in_bits/s]	2502400.787498492
> headerBytesReceived	30945888
> [headerBytesReceived_in_bits/s]	56833.65312425641
> lastPacketReceivedTimestamp	1665940249506
> [lastPacketReceivedTimestamp]	10/16/2022, 1:10:49 PM
> jitterBufferDelay	27.622
> [jitterBufferDelay/jitterBufferEmittedCount_in_ms]	0
> jitterBufferEmittedCount	1639
> framesReceived	1634
> [framesReceived/s]	0
> [framesReceived-framesDecoded-framesDropped]	-6
> frameWidth	1280
> frameHeight	960
> framesDecoded	1640
> [framesDecoded/s]	0
> keyFramesDecoded	23
> [keyFramesDecoded/s]	0
> framesDropped	0
> totalDecodeTime	9.156385
> [totalDecodeTime/framesDecoded_in_ms]	0
> totalProcessingDelay	123.34863899999999
> [totalProcessingDelay/framesDecoded_in_ms]	0
> totalAssemblyTime*	2.955362
> [totalAssemblyTime*/framesAssembledFromMultiplePackets*_in_ms]	0
> framesAssembledFromMultiplePackets*	1640
> totalInterFrameDelay	3877.6129999999985
> [totalInterFrameDelay/framesDecoded_in_ms]	0
> totalSquaredInterFrameDelay	923334.1427090003
> [interFrameDelayStDev_in_ms]	0
> decoderImplementation	libvpx
> firCount	0
> pliCount	1850
> nackCount	83744
> qpSum	54086
> [qpSum/framesDecoded]	0
> minPlayoutDelay*	0
> 
> ```