Hi,
After the latest release updates, I’m not able to manually set the spatial layer on VP8 and H.264 simulcast consumers. After looking on server logs, I found that the SimulcastConsumer::UseAvailableBitrate method performs updates based on the packet loss percentage. How the auto-update mechanism can be turned off? Maybe calling SetExternallyManagedBitrate?
Are you using a browser as consumer? Or some gstreamer endpoint to consume?
Also, do you see that issue in the online official demo?
Are you using a browser as consumer? Or some gstreamer endpoint to consume?
Browser
Also, do you see that issue in the online official demo?
No, in the official demo I can manually change the spatial layer.
Maybe the rembClient is not enabled due to some configuration option?
I’ve just tried to disable REMB support in mediasoup (by removing it from lib/supporttedRtpCapabilities.js and tested it in the demo. The consumer can perfectly change the preferred layers and those are changed.
When there is not REMB, the consumer changes its layers based on preferred ones (set via API) and the RTP stream score. If score >= 7 (it’s bewteeen 0 and 10) then it can upgrade to a higher spatial layer.
There is no option to enable or disable it other than the negotiated rtcpFb in the codec.
The problem seems related to running mediasoup server inside a Docker container. I’m investigating on this more in deep.
We run mediasoup inside Dockers without any problem by using network="host".
Just in case, if the mediasoup WebRtcTransport does not receive REMB packets and other RTCP reports from the remote consumer, it cannot do neither auto nor manual layers change. So you may want to check whether the RTCP communication between the remote endpoint and mediasoup is fine.
No I’m wrong, the problem is related to using multiple workers and piping the producers from one router to another. If I use one worker, I can manually change levels.
When I consume the stream from a piped producer, I get these errors:
2019-07-04T09:03:16.937Z mediasoup:worker[pid:56] RTC::Transport::ReceiveRtcpPacket() | no Consumer found for received Receiver Report [ssrc:514687286]
I see also this:
mediasoup_1 | 2019-07-04T09:06:24.502Z mediasoup:worker[pid:56] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after first main iteration [remainingBitrate:312872]
mediasoup_1 | 2019-07-04T09:06:24.502Z mediasoup:worker[pid:56] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | layer bitrate for Consumer [bitrate:312872, consumerId:cbd9573c-1440-4e1f-9c7e-268903ceb9cb]
mediasoup_1 | 2019-07-04T09:06:24.502Z mediasoup:worker[pid:56] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after layer-by-layer iteration [remainingBitrate:312872]
mediasoup_1 | 2019-07-04T09:06:26.504Z mediasoup:worker[pid:51] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:2623493711, seq:58523]
mediasoup_1 | 2019-07-04T09:06:26.504Z mediasoup:worker[pid:56] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:151369818, seq:56720]
mediasoup_1 | 2019-07-04T09:06:26.509Z mediasoup:worker[pid:56] RTC::WebRtcTransport::OnRembClientAvailableBitrate() | outgoing available bitrate [bitrate:840256bps]
mediasoup_1 | 2019-07-04T09:06:26.509Z mediasoup:worker[pid:56] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | before iterations [availableBitrate:840256]
mediasoup_1 | 2019-07-04T09:06:26.509Z mediasoup:worker[pid:56] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | main bitrate for Consumer [priority:2, bitrate:840256, consumerId:cbd9573c-1440-4e1f-9c7e-268903ceb9cb]
mediasoup_1 | 2019-07-04T09:06:26.509Z mediasoup:worker[pid:56] RTC::SimulcastConsumer::UseAvailableBitrate() | choosing layers 1:2 [bitrate:840256, virtualBitrate:907476, usedBitrate:520600, consumerId:cbd9573c-1440-4e1f-9c7e-26890
In your demo I see that each Room runs on the same worker (https://github.com/versatica/mediasoup-demo/blob/v3/server/server.js#L466), so you can’t see the problem.
P.S.
the SVC consumer works as expected also on a piped transport.
Definitely we need to do more real tests with PipeTransport. Can you please provide a dump() of the PipeConsumer in the PipeTransport_1 in router_1, a dump() of the generated Producer in PipeTransport_2 in router_2, a dump() of the new Consumer in router_2, and a dump() of both PipeTransport instances?
pipeConsumer:
{ consumableRtpEncodings:
[ { ksvc: false,
maxBitrate: 750000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314559,
temporalLayers: 3 },
{ ksvc: false,
maxBitrate: 1500000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314560,
temporalLayers: 3 },
{ ksvc: false,
maxBitrate: 3000000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314561,
temporalLayers: 3 },
[length]: 3 ],
id: '08d1dc98-930a-40fd-8304-055a84050a06',
kind: 'video',
paused: false,
producerPaused: false,
rtpParameters:
{ codecs:
[ { clockRate: 90000,
mimeType: 'video/H264',
parameters:
{ 'level-asymmetry-allowed': 1,
'packetization-mode': 1,
'profile-level-id': '42e01f' },
payloadType: 107,
rtcpFeedback:
[ { parameter: 'pli', type: 'nack' },
{ parameter: 'fir', type: 'ccm' },
[length]: 2 ] },
[length]: 1 ],
encodings:
[ { codecPayloadType: 107,
ksvc: false,
maxBitrate: 750000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314559,
temporalLayers: 3 },
{ codecPayloadType: 107,
ksvc: false,
maxBitrate: 1500000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314560,
temporalLayers: 3 },
{ codecPayloadType: 107,
ksvc: false,
maxBitrate: 3000000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314561,
temporalLayers: 3 },
[length]: 3 ],
headerExtensions:
[ { encrypt: false,
id: 6,
parameters: {},
uri:
'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07' },
{ encrypt: false,
id: 7,
parameters: {},
uri: 'urn:ietf:params:rtp-hdrext:framemarking' },
{ encrypt: false,
id: 11,
parameters: {},
uri: 'urn:3gpp:video-orientation' },
{ encrypt: false,
id: 12,
parameters: {},
uri: 'urn:ietf:params:rtp-hdrext:toffset' },
[length]: 4 ],
rtcp: { cname: '6IrDM+1JnoYZUHIc', reducedSize: true } },
rtpStreams:
[ { params:
{ clockRate: 90000,
cname: '6IrDM+1JnoYZUHIc',
mimeType: 'video/H264',
payloadType: 107,
spatialLayers: 1,
ssrc: 499314561,
temporalLayers: 3,
useDtx: false,
useFir: true,
useInBandFec: false,
useNack: false,
usePli: true },
score: 10,
totalReportedLoss: 0,
totalSourceLoss: 0 },
{ params:
{ clockRate: 90000,
cname: '6IrDM+1JnoYZUHIc',
mimeType: 'video/H264',
payloadType: 107,
spatialLayers: 1,
ssrc: 499314560,
temporalLayers: 3,
useDtx: false,
useFir: true,
useInBandFec: false,
useNack: false,
usePli: true },
score: 10,
totalReportedLoss: 0,
totalSourceLoss: 0 },
{ params:
{ clockRate: 90000,
cname: '6IrDM+1JnoYZUHIc',
mimeType: 'video/H264',
payloadType: 107,
spatialLayers: 1,
ssrc: 499314559,
temporalLayers: 3,
useDtx: false,
useFir: true,
useInBandFec: false,
useNack: false,
usePli: true },
score: 10,
totalReportedLoss: 0,
totalSourceLoss: 0 },
[length]: 3 ],
supportedCodecPayloadTypes: [ 107, [length]: 1 ],
type: 'pipe' }
pipeProducer:
{ id: 'c87a4064-2ee7-4dde-8687-464639a2885d',
kind: 'video',
paused: false,
rtpMapping:
{ codecs:
[ { mappedPayloadType: 107, payloadType: 107 }, [length]: 1 ],
encodings:
[ { mappedSsrc: 779636838, rid: null, ssrc: 499314559 },
{ mappedSsrc: 779636839, rid: null, ssrc: 499314560 },
{ mappedSsrc: 779636840, rid: null, ssrc: 499314561 },
[length]: 3 ] },
rtpParameters:
{ codecs:
[ { clockRate: 90000,
mimeType: 'video/H264',
parameters:
{ 'level-asymmetry-allowed': 1,
'packetization-mode': 1,
'profile-level-id': '42e01f' },
payloadType: 107,
rtcpFeedback:
[ { parameter: 'pli', type: 'nack' },
{ parameter: 'fir', type: 'ccm' },
[length]: 2 ] },
[length]: 1 ],
encodings:
[ { codecPayloadType: 107,
ksvc: false,
maxBitrate: 750000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314559,
temporalLayers: 3 },
{ codecPayloadType: 107,
ksvc: false,
maxBitrate: 1500000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314560,
temporalLayers: 3 },
{ codecPayloadType: 107,
ksvc: false,
maxBitrate: 3000000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 499314561,
temporalLayers: 3 },
[length]: 3 ],
headerExtensions:
[ { encrypt: false,
id: 6,
parameters: {},
uri:
'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07' },
{ encrypt: false,
id: 7,
parameters: {},
uri: 'urn:ietf:params:rtp-hdrext:framemarking' },
{ encrypt: false,
id: 11,
parameters: {},
uri: 'urn:3gpp:video-orientation' },
{ encrypt: false,
id: 12,
parameters: {},
uri: 'urn:ietf:params:rtp-hdrext:toffset' },
[length]: 4 ],
rtcp: { cname: '6IrDM+1JnoYZUHIc', reducedSize: true } },
rtpStreams: [ [length]: 0 ],
type: 'simulcast' }
new Consumer in router_2:
{ consumableRtpEncodings:
[ { ksvc: false,
maxBitrate: 750000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 779636838,
temporalLayers: 3 },
{ ksvc: false,
maxBitrate: 1500000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 779636839,
temporalLayers: 3 },
{ ksvc: false,
maxBitrate: 3000000,
scalabilityMode: 'S1T3',
spatialLayers: 1,
ssrc: 779636840,
temporalLayers: 3 },
[length]: 3 ],
currentSpatialLayer: -1,
currentTemporalLayer: -1,
id: '0f2c5aa7-1ddc-4a50-85d2-a5ba56a44c65',
kind: 'video',
paused: true,
preferredSpatialLayer: 2,
preferredTemporalLayer: 2,
producerPaused: false,
rtpParameters:
{ codecs:
[ { clockRate: 90000,
mimeType: 'video/H264',
parameters:
{ 'level-asymmetry-allowed': 1,
'packetization-mode': 1,
'profile-level-id': '42e01f' },
payloadType: 107,
rtcpFeedback:
[ { type: 'goog-remb' },
{ parameter: 'fir', type: 'ccm' },
{ type: 'nack' },
{ parameter: 'pli', type: 'nack' },
[length]: 4 ] },
{ clockRate: 90000,
mimeType: 'video/rtx',
parameters: { apt: 107 },
payloadType: 108,
rtcpFeedback: [ [length]: 0 ] },
[length]: 2 ],
encodings:
[ { codecPayloadType: 107,
ksvc: false,
rtx: { ssrc: 374288032 },
scalabilityMode: 'S3T3',
spatialLayers: 3,
ssrc: 363047489,
temporalLayers: 3 },
[length]: 1 ],
headerExtensions:
[ { encrypt: false,
id: 4,
parameters: {},
uri: 'http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time' },
{ encrypt: false,
id: 6,
parameters: {},
uri:
'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07' },
{ encrypt: false,
id: 11,
parameters: {},
uri: 'urn:3gpp:video-orientation' },
{ encrypt: false,
id: 12,
parameters: {},
uri: 'urn:ietf:params:rtp-hdrext:toffset' },
[length]: 4 ],
rtcp: { cname: '6IrDM+1JnoYZUHIc', reducedSize: true } },
rtpStream:
{ params:
{ clockRate: 90000,
cname: '6IrDM+1JnoYZUHIc',
mimeType: 'video/H264',
payloadType: 107,
rtxPayloadType: 108,
rtxSsrc: 374288032,
spatialLayers: 3,
ssrc: 363047489,
temporalLayers: 3,
useDtx: false,
useFir: true,
useInBandFec: false,
useNack: true,
usePli: true },
score: 10,
totalReportedLoss: 0,
totalSourceLoss: 0 },
supportedCodecPayloadTypes: [ 107, [length]: 1 ],
targetSpatialLayer: -1,
targetTemporalLayer: -1,
type: 'simulcast' }
Do you have a RTCP warning with the specific not found ssrc in that exact scenario?
No, in both scenarios, so this shouldn’t be the main issue.
Here a log comparison, with 1 worker:
mediasoup:worker[pid:31344] RTC::WebRtcTransport::OnRembClientAvailableBitrate() | outgoing available bitrate [bitrate:3912928bps] +2s
mediasoup:worker[pid:31344] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | before iterations [availableBitrate:3912928] +0ms
mediasoup:worker[pid:31344] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | main bitrate for Consumer [priority:3, bitrate:3912928, consumerId:e3b41158-eb5f-4127-baf0-ed38d3ee0de3] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 0:0 [virtualBitrate:4225962, requiredBitrate:34040] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 0:1 [virtualBitrate:4225962, requiredBitrate:48256] +1ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 0:2 [virtualBitrate:4225962, requiredBitrate:62128] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:0 [virtualBitrate:4225962, requiredBitrate:229000] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:1 [virtualBitrate:4225962, requiredBitrate:337280] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:2 [virtualBitrate:4225962, requiredBitrate:450416] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 2:0 [virtualBitrate:4225962, requiredBitrate:985800] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 2:1 [virtualBitrate:4225962, requiredBitrate:1710336] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 2:2 [virtualBitrate:4225962, requiredBitrate:2558760] +0ms
mediasoup:worker[pid:31344] RTC::SimulcastConsumer::UseAvailableBitrate() | choosing layers 2:2 [bitrate:3912928, virtualBitrate:4225962, usedBitrate:2558760, consumerId:e3b41158-eb5f-4127-baf0-ed38d3ee0de3] +0ms
mediasoup:worker[pid:31344] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after first main iteration [remainingBitrate:1354168] +0ms
mediasoup:worker[pid:31344] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | layer bitrate for Consumer [bitrate:1354168, consumerId:e3b41158-eb5f-4127-baf0-ed38d3ee0de3] +0ms
mediasoup:worker[pid:31344] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after layer-by-layer iteration [remainingBitrate:1354168] +0ms
With multiple workers:
mediasoup:worker[pid:31785] RTC::WebRtcTransport::OnRembClientAvailableBitrate() | outgoing available bitrate [bitrate:754144bps] +2s
mediasoup:worker[pid:31785] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | before iterations [availableBitrate:754144] +0ms
mediasoup:worker[pid:31785] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | main bitrate for Consumer [priority:3, bitrate:754144, consumerId:939529d0-55ec-4396-800c-f18382ce236c] +0ms
mediasoup:worker[pid:31785] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:0 [virtualBitrate:814475, requiredBitrate:141744] +0ms
mediasoup:worker[pid:31785] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:1 [virtualBitrate:814475, requiredBitrate:199768] +0ms
mediasoup:worker[pid:31785] RTC::SimulcastConsumer::UseAvailableBitrate() | testing layers 1:2 [virtualBitrate:814475, requiredBitrate:246688] +0ms
mediasoup:worker[pid:31785] RTC::SimulcastConsumer::UseAvailableBitrate() | choosing layers 1:2 [bitrate:754144, virtualBitrate:814475, usedBitrate:246688, consumerId:939529d0-55ec-4396-800c-f18382ce236c] +0ms
mediasoup:worker[pid:31785] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after first main iteration [remainingBitrate:507456] +0ms
mediasoup:worker[pid:31785] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | layer bitrate for Consumer [bitrate:507456, consumerId:939529d0-55ec-4396-800c-f18382ce236c] +0ms
mediasoup:worker[pid:31785] RTC::WebRtcTransport::DistributeAvailableOutgoingBitrate() | after layer-by-layer iteration [remainingBitrate:507456] +0ms
In the latest scenario, I’m able to change only the temporal layer, using spatialLayer=1
Seems that spatial layers 0 and 2 are excluded from the iteration inside the SimulcastConsumer::UseAvailableBitrate method.
I see. It would be super useful if you could provide a dump() of PipeTransports in router_1 and router_2. I know that the router.pipeToRouter() API does not expose those internally created PipeTransports, but you can use:
router_1.observer.on('newtransport', (transport) =>
{
if (transport.constructor.name === 'PipeTransport')
{
const dump = await transport.dump();
// print dump
}
});
Same for PipeTransport in router_2. May you provide that info please? it would be super useful since the `transport.dump()` prints also a `mapSsrcConsumerId` which is a map of Consumers indexed by their outbound SSRCs.Since there is no RTP streams (yet?) in the PipeProducer, I assume you did the dump on it before receiving any RTP packet, right?
BTW: is your code somewhere and can be tested to reproduce the issue? By looking at the dumps above everything seems correct, but obviously it’s not…
Here you can find my repository: https://github.com/vpalmisano/mediasoup-3-test
Run using this command line:
DEBUG=app*,media* \
MAX_WORKERS=2 \
MS_LOG_LEVEL=debug \
PORT=10000 \
MS_LISTEN_IPS='[{"ip":"0.0.0.0","announcedIp":"<IP ADDRESS>"}]' \
MS_MIN_PORT=10001 \
MS_MAX_PORT=19999 \
yarn run debug
The same issue would happen if you create a single worker with 2 piped routers, right?
I get this error:
transportConsume TypeError: cannot use this Router as destination