Using PlainRtpTransport for Simulcast ingesting

mediasoup libraries
1

Hi,
I’m trying to use PlainRtpTransport for ingesting multiple RTP flows from a GStreamer pipeline in order to obtain a Simulcast producer using VP8 codec.
I changed the “encodings” array in the rtpParameters structure, adding 3 levels setting scalabilityMode = “S1T1” and adding 3 different ssrc values, used by the GStreamer pipeline.
I see the following behaviors:

  1. using low encoding bitrates (~2 Mbps for the higher level), the consumers are able to receive all SVC levels, as in the case of a WebRTC transport.
  2. increasing the bitrate and the video resolution, I can receive only the lower level. I think that the SVC consumers require that all levels are perfectly synchronized in order to enable the level switch. Is that correct?

I have some questions about the DistributeAvailableOutgoingBitrate function (https://github.com/versatica/mediasoup/blob/5c2208d5e70324ef28a91106afdeb67330ac70bd/worker/src/RTC/WebRtcTransport.cpp#L891): how Mediasoup estimates the total outgoing bitrate? Using REMB packets? How can I force a level switch bypassing the server side control?

2

Maybe related to this: Problem using SimulcastConsumer on piped transports
I’m investigating…

3

In order for a Consumer to be able to switch from the current stream to a desired stream, mediasoup must have received Sender Reports for both streams. Otherwise switching is not possible (otherwise lipsync would happen).

4

I’ll reply to the other questions next week.

5

Is it correct sending all the RTCP packets (for each level, with his own SSRC) on the same ip:port returned by createPlainRtpTransport?

6

You must send all RTP from the same port to the transport.tuple.localPort, and all RTCP from the same port to the transport.tuple.localPort (or localRtcpPort if rtcpMux is not enabled).

The origin port of RTCP must be the same as in RTP if rtcpMux is enabled, or different if not. And then you can play with the comedia mode.

Can you tell me the source/destination ports of RTP and RTCP and how you create the transport and call connect() (if comedia is not set)?

7

rtcpMux is disabled and comedia is enabled.
Since I’m using the transport with comedia enabled, I don’t have to call connect(), right?

I’m using:

  1. createPlainRtpTransport(comedia=true)
    this returns ip, rtpPort and rtcpPort that I will use as destinations for sending RTP and RTCP data from the GStreamer pipeline
  2. transportProduce(transportId, rtpParameters={…})
    then I start the send pipeline
  3. from the same RTCP socket used for sending rtcp reports, I receive the RTCP packets sent by mediasoup
8

You can not call connect() if comedia is set so that’s ok.

When comedia is enabled and rtcpMux is not (which is its default value), mediasoup won’t send you any RTCP until you send a RTCP to mediasoup. That’s how the transport learns your RTCP port when comedia is enabled and rtcpMux is not.

9

Everything seems ok: RTCP packets are sent and received.
The only problem seems that the DistributeAvailableOutgoingBitrate method doesn’t evaluate correctly the availableBitrate.
If REMB are disabled, how the server can evaluate the availableBitrate value?
If I force this value to something high, I can receive always the maximum level and I can switch using client side messages.

10

Right now in Napoli talking about mediasoup v3 :slight_smile:
Let me answer in the next days.

11

Yes, I see from conference streaming :slight_smile:

More investigation in progress.

1 Like
12

Everything seems ok: RTCP packets are sent and received.

Can you anyway check SimulcastConsumer::CanSwitchToSpatialLayer() to see whether it returns true or false?

The only problem seems that the DistributeAvailableOutgoingBitrate method doesn’t evaluate correctly the availableBitrate.
If REMB are disabled, how the server can evaluate the availableBitrate value?

DistributeAvailableOutgoingBitrate() is not called if there is no REMB support since there is no bandwidth estimator (no this->rembClient in Transport.cpp). So the app must decide which layer to receive by using consumer.setPreferredLayers(). The SimulcastConsumer will also move to a lower stream if it detects low “score” in the currently selected producer’s stream. You can log consumer.on("score") event and also periodically check consumer.score to see what value it has.

If I force this value to something high, I can receive always the maximum level and I can switch using client side messages.

Which value do you mean?

13

The client is a web browser, so the consumer has REMB support, right?

I configured the simulcast bitrates in the rtp producer as follows:

'encodings': [
  {'ssrc': 1000, 'active': True, 'maxBitrate': 2000000, 'scaleResolutionDownBy': 4, 'scalabilityMode': 'S1T1'}, 
  {'ssrc': 1001, 'active': True, 'maxBitrate': 4000000, 'scaleResolutionDownBy': 2, 'scalabilityMode': 'S1T1'}, 
  {'ssrc': 1002, 'active': True, 'maxBitrate': 8000000, 'scaleResolutionDownBy': 1, 'scalabilityMode': 'S1T1'}
]

CanSwitchToSpatialLayer returns true, here is a log of a simulcast consumer:

mediasoup:Channel[pid:52] request() [method:consumer.resume, id:12]
mediasoup:worker[pid:52] RTC::RembClient::CalculateProbationTargetBitrate() | probation enabled [bitrate:0, availableBitrate:600000, factor:0.000000, probationTargetBitrate:600000]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | before iterations [availableBitrate:600000]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | main bitrate for Consumer [priority:3, bitrate:600000, consumerId:4d1b5fb4-6ee9-40b6-a935-05f213f6cc18]
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | CanSwitchToSpatialLayer(0) returns true
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | choosing layers -1:-1 [bitrate:600000, virtualBitrate:648000, usedBitrate:0, consumerId:4d1b5fb4-6ee9-40b6-a935-05f213f6cc18]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after first main iteration [remainingBitrate:600000]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | layer bitrate for Consumer [bitrate:600000, consumerId:4d1b5fb4-6ee9-40b6-a935-05f213f6cc18]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after layer-by-layer iteration [remainingBitrate:600000]

The consumer doesn’t receive anything after this point. I see that the lower level has a maxBItrate=2000000 > availableBitrate:600000

Changing these lines in https://github.com/versatica/mediasoup/blob/5c2208d5e70324ef28a91106afdeb67330ac70bd/worker/src/RTC/WebRtcTransport.cpp#L919 makes the consumer works:

		uint32_t availableBitrate{ 1000000000 };

		if (this->rembClient)
		{
			//availableBitrate = this->rembClient->GetAvailableBitrate();

			// Resechedule next REMB event.
			this->rembClient->ResecheduleNextEvent();
		}

mediasoup:Channel[pid:52] request() [method:consumer.resume, id:27]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | before iterations [availableBitrate:1000000000]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | main bitrate for Consumer [priority:3, bitrate:1000000000, consumerId:f0d8fd8b-2e6c-4a4d-aa71-ad4a631d3634]
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | CanSwitchToSpatialLayer(0) returns true
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | CanSwitchToSpatialLayer(1) returns true
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | CanSwitchToSpatialLayer(2) returns true
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UseAvailableBitrate() | choosing layers 2:0 [bitrate:1000000000, virtualBitrate:1080000000, usedBitrate:7997632, consumerId:f0d8fd8b-2e6c-4a4d-aa71-ad4a631d3634]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after first main iteration [remainingBitrate:992002368]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | layer bitrate for Consumer [bitrate:992002368, consumerId:f0d8fd8b-2e6c-4a4d-aa71-ad4a631d3634]
mediasoup:worker[pid:52] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after layer-by-layer iteration [remainingBitrate:992002368]
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UpdateTargetLayers() | using spatialLayer:2 as RTP timestamp reference
mediasoup:worker[pid:52] RTC::SimulcastConsumer::UpdateTargetLayers() | target layers changed [spatial:2, temporal:0, consumerId:f0d8fd8b-2e6c-4a4d-aa71-ad4a631d3634]
mediasoup:worker[pid:52] RTC::RtpStreamRecv::RequestKeyFrame() | sending PLI [ssrc:1002]
14

Adding some debug messages, I found that in my case (3 spatial and 1 temporal layers), this loop exits here (https://github.com/versatica/mediasoup/blob/5c2208d5e70324ef28a91106afdeb67330ac70bd/worker/src/RTC/SimulcastConsumer.cpp#L443):

				if (requiredBitrate > virtualBitrate)
					goto done;

				// Set provisional layers and used bitrate.
				this->provisionalTargetSpatialLayer  = spatialLayer;
				this->provisionalTargetTemporalLayer = temporalLayer;

and so provisionalTargetSpatialLayer and provisionalTargetTemporalLayer have the initial value (-1).

Seems that the problem is that the requiredBitrate is higher than the client bitrate. This mechanism works only if the consumer starts immediately after the producer: in this case, the requiredBitrate includes only the audio bitrate (because first video packets are produced after audio, i think).

15

OK OK I see! And actually it’s nice that you are trying this. So:

  • I thought your consumer was not using REMB, my fault.

  • With current just-REMB BWE in mediasoup v3 and the current probation system (based on real video packet retransmissions) your scenario is not possible.

  • Actually you can set the initialOutgoingAvailableBitrate in the consumer transport (which defaults to 600000 bps) to a higher value, something higher than the low stream you are producing (2 mbps). That will let the low stream reach the receiver browser.

  • However that’s far from perfect. There will be a bitrate of 2 mbps in the downlink plus some extra bitrate due mediasoup’s RTP probation, but it will never make the remote browser send REMB feedbacks with a bitrate of 4 mbps, so it won’t be able to up switch spatial layer.

  • Even worse, if the network goes eventually down and there is packet loss and so on, the receiver may decrease the bitrate value of its REMB feedbacks to something lower than 2 mbps, so mediasoup wouldn’t be able to send the low stream / spatial-layer, and the world ends there.

The good news is:

  • We are working on sender side transport-cc in a private “devel” branch (we’ll move it to GitHub in next days, probably next week).

  • We have the very similar scenario as you: an endpoint producing a simulcast video with 2mbps / 5mbps / 8mbps.

  • It works fine! The transport-cc based sender side BWE makes that scenario possible!

  • We are using some files of libwebrtc. After adapting them to mediasoup and after some modifications (for instance to allow RTP probation even before sending real RTP packets), it works and produces very high availableBitrate values.

  • Once we have it perfectly working, we’ll try to make our own sender side BWE + probation implementation by processing received transport-cc feedbacks (we already have a proper parser to extract info for those feedbacks). And may be we’ll implement NADA instead of trying to figure out the algorithm of libwebrtc (based on goog_transport_control, which is not documented anywhere).

So, may be you know people with good knowledge about this topic to help us? :slight_smile: :slight_smile: :slight_smile:

16

Maybe initialAvailableOutgoingBitrate. Setting this option the problem is resolved.

I think that using simulcast, at least the smaller level should be sent in any case. Without sending any data, you can’t probe the available bandwidth and you suffer the downward spiral effect caused by the lack of bandwith probation.

Very busy in this period, I will try :wink:

17

You can find a comparison here:

18

But it’s not perfect. You may not be able to up switch to the next layer later.

Imagine the browser is wishing to receive 20 simulcast videos with a low stream of 500bps. That’s 20x500 = 10mbps. If the downlink speed is just 5mbps the connection will be broken. So we cannot let the low stream always pass.

Anyway, as I said using transport-cc for sender side BWE (in our private “devel” branch), your scenario does perfectly work because mediasoup creates proper probation (even if initialAvailableOutgoingBitrate is 600000). So just wait a bit please :slight_smile:

19

Oh cool! something to read during my flight tomorrow :slight_smile:

Beside it, do you think NADA is the good approach? Well, GCC algorithm is not documented so…

20

Luca (the paper author) says that NADA (at the time of the publication) had some queueing problems. He doesn’t know if these problems are fixed in the latest release.