Remote PipeTransport and layerschange / volume events

mediasoup libraries
1

Hi all,
I have replicated the pipeToRouter function for piping a remote producer to a local one.
In particular, I start a consumer on the remote side and a producer on the local one (exactly like this: https://github.com/versatica/mediasoup/blob/2d8f4ebac131370b98a59f6c150d4e7440cb65d1/src/Router.ts#L612).
Two things are not clear:

  1. the local producer, if added to an AudioLevelObserver instance, doesn’t produce no volume event;
  2. consumers created on the local copy don’t emit the layerschange event. The question is if the PipeTransport sends all the layers exactly as sent by the origin producer.
2

You mean consumer in local side and producer in remote one, right?

Assuming you mean “remote producer” (the one that receives in router2 the RTP from pipeConsumer in router1), volume event should work. Can you print producer.dump()?

What is local copy? BTW the local pipeConsumer in router1 does not emit layerschange event because it sends all layers and streams, that’s exactly what a pipeConsumer does.

3

In my case I should consume on the side where I have the producer (the remote side) and produce on the local side (remoteMediasoup is my signaling server and mediasoup is a proxy object, inside mediasoup.transportProduce I add the producer to the AudioLevelObserver):

    // consume from remote
    let pipeConsumer = await this.remoteMediasoup.request('transportConsume', {
      transportId: remotePipeTransport.id,
      producerId,
      rtpCapabilities,
      appData: {},
    });
    // produce on local
    let pipeProducer = await mediasoup.transportProduce({
        id: producerId,
        transportId: localPipeTransport.id,
        kind: pipeConsumer.kind,
        rtpParameters: pipeConsumer.rtpParameters,
        paused: pipeConsumer.producerPaused,
        appData: {},
    });
4

How is pipeProducer.dump()?

5 
{
  id: 'dfe729b6-861e-4db8-94f5-c0d4183d9966',
  kind: 'video',
  paused: false,
  rtpMapping: {
    codecs: [
      { mappedPayloadType: 101, payloadType: 101 },
      { mappedPayloadType: 102, payloadType: 102 },
      [length]: 2
    ],
    encodings: [
      { mappedSsrc: 709772240, rid: null, ssrc: 963353905 },
      { mappedSsrc: 709772241, rid: null, ssrc: 963353906 },
      { mappedSsrc: 709772242, rid: null, ssrc: 963353907 },
      [length]: 3
    ]
  },
  rtpParameters: {
    codecs: [
      {
        clockRate: 90000,
        mimeType: 'video/VP8',
        parameters: {},
        payloadType: 101,
        rtcpFeedback: [
          { type: 'nack' },
          { parameter: 'pli', type: 'nack' },
          { parameter: 'fir', type: 'ccm' },
          [length]: 3
        ]
      },
      {
        clockRate: 90000,
        mimeType: 'video/rtx',
        parameters: { apt: 101 },
        payloadType: 102,
        rtcpFeedback: [ [length]: 0 ]
      },
      [length]: 2
    ],
    encodings: [
      {
        codecPayloadType: 101,
        ksvc: false,
        maxBitrate: 256000,
        scalabilityMode: 'S1T3',
        spatialLayers: 1,
        ssrc: 963353905,
        temporalLayers: 3
      },
      {
        codecPayloadType: 101,
        ksvc: false,
        maxBitrate: 512000,
        scalabilityMode: 'S1T3',
        spatialLayers: 1,
        ssrc: 963353906,
        temporalLayers: 3
      },
      {
        codecPayloadType: 101,
        ksvc: false,
        maxBitrate: 1024000,
        scalabilityMode: 'S1T3',
        spatialLayers: 1,
        ssrc: 963353907,
        temporalLayers: 3
      },
      [length]: 3
    ],
    headerExtensions: [
      {
        encrypt: false,
        id: 6,
        parameters: {},
        uri: 'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07'
      },
      {
        encrypt: false,
        id: 7,
        parameters: {},
        uri: 'urn:ietf:params:rtp-hdrext:framemarking'
      },
      {
        encrypt: false,
        id: 11,
        parameters: {},
        uri: 'urn:3gpp:video-orientation'
      },
      {
        encrypt: false,
        id: 12,
        parameters: {},
        uri: 'urn:ietf:params:rtp-hdrext:toffset'
      },
      [length]: 4
    ],
    rtcp: { cname: '42c6a333', reducedSize: true }
  },
  rtpStreams: [ [length]: 0 ],
  traceEventTypes: '',
  type: 'simulcast'
}
6

I’m lost. How do you expect that a video Producer generates audio volume events? :slight_smile:

7 
{
  id: '8d9dfc43-a020-4062-8f44-8ef64a5d16be',
  kind: 'audio',
  paused: false,
  rtpMapping: {
    codecs: [ { mappedPayloadType: 100, payloadType: 100 }, [length]: 1 ],
    encodings: [
      { mappedSsrc: 401555356, rid: null, ssrc: 567848468 },
      [length]: 1
    ]
  },
  rtpParameters: {
    codecs: [
      {
        channels: 2,
        clockRate: 48000,
        mimeType: 'audio/opus',
        parameters: { minptime: 10, useinbandfec: 1 },
        payloadType: 100,
        rtcpFeedback: [ [length]: 0 ]
      },
      [length]: 1
    ],
    encodings: [ { codecPayloadType: 100, ssrc: 567848468 }, [length]: 1 ],
    headerExtensions: [
      {
        encrypt: false,
        id: 10,
        parameters: {},
        uri: 'urn:ietf:params:rtp-hdrext:ssrc-audio-level'
      },
      [length]: 1
    ],
    rtcp: { cname: '42c6a333', reducedSize: true }
  },
  rtpStreams: [ [length]: 0 ],
  traceEventTypes: '',
  type: 'simple'
}
8

Everything looks good. Honestly no idea why you don’t get “volume” event for that audio pipeProducer. But it works. I’ve just tested by adding a second audioLevelObserver in the mediasoupRouter2 in the mediasoup-demo pipe branch (which is kinda uncompleted) and “volumes” event is fired in the audio pipeProducer.

Obviously the AudioLevelObserver must be created in router2 when pipeProducer has been created.

9

In your demo you are piping using pipeToRouter (https://github.com/versatica/mediasoup-demo/blob/pipe/server/lib/Room.js#L1026), between routers on the same host, is it right?

10

Yes, but it should not matter as that info is in RTP packets. Anyway I’ve had to add more code in that branch to create a second AudioLevelObserver in the second router (that’s not in the branch).