libmediasoupclient for Android - no image

mediasoup libraries
1

Hi @haiyangwu,

I’ve been trying to adapt your demo to a test project that I am doing. The setup is the following:

1 ffmpeg video producer
1 mediasoup server
1 mediasoup-client consumer (web)
1 mediasoup-client-android based app following the demo that you’ve provided here

The first three elements seem to work together. I can see the video in the browser. Unfortunately I cannot get the mediasoup-client-android to work with this setup.

The connection seems to be established correctly between the mediasoup server and the mediasoup-client-android, but then I receive a lot of messages like this and I don’t see any video:

[…]
mediasoup:worker[pid:19367] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:333259308, media ssrc:333259308] +50ms
mediasoup:worker[pid:19367] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:333259308, media ssrc:333259308] +50ms
mediasoup:worker[pid:19367] RTC::Producer::ReceiveRtpPacket() | key frame received [ssrc:55059, seq:18436] +29ms
mediasoup:worker[pid:19367] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:333259308, media ssrc:333259308] +20ms
mediasoup:worker[pid:19367] RTC::Transport::HandleRtcpPacket() | PLI received, requesting key frame for Consumer [sender ssrc:333259308, media ssrc:333259308] +50ms
[…]

One thing that is different in my app is that I didn’t use the @BindingAdapter(“edias_render”) for the method

public static void render(SurfaceViewRenderer renderer, VideoTrack track) {
    Log.d(TAG, "edias_render: " + (track != null));
    if (track != null) {
        track.addSink(renderer);
        renderer.setVisibility(View.VISIBLE);
    } else {
        renderer.setVisibility(View.GONE);
    }
}

Basically it seems that the client never receives any key frame so it keeps on sending PLIs to the server. The producer does not support PLIs but it still sends keyframes at regular intervals.

Please let me know if you have any ideas how I could debug this.

2

Depending on how you use ffmpeg, it will send keyframe every N seconds or not. That’s how the poor RTP implementation of ffmpeg works.

Regarding why your implementation cannot render received video, honestly no idea. You may check the consumer.getStats() in mediasoup server.

3

I do manually set an interval, which is also visible in the mediasoup debug messages (i.e. keyframe received).

OK, I’ll have a look, thanks!

4

I mean check the stats not just to see PLIs but also other values such as sent RTP bytes, packets, etc.

5

These are the consumer stats that I see on the server:

[
{
bitrate: 541373,
byteCount: 2988561,
firCount: 0,
fractionLost: 0,
kind: ‘video’,
mimeType: ‘video/H264’,
nackCount: 0,
nackPacketCount: 0,
packetCount: 2892,
packetsDiscarded: 0,
packetsLost: 0,
packetsRepaired: 0,
packetsRetransmitted: 0,
pliCount: 660,
roundTripTime: 0.6103515625,
rtxSsrc: 319635064,
score: 10,
ssrc: 618044048,
timestamp: 7365341,
type: ‘outbound-rtp’
},
{
bitrate: 534307,
byteCount: 2953857,
firCount: 0,
fractionLost: 0,
jitter: 0,
kind: ‘video’,
mimeType: ‘video/H264’,
nackCount: 0,
nackPacketCount: 0,
packetCount: 2892,
packetsDiscarded: 0,
packetsLost: 0,
packetsRepaired: 0,
packetsRetransmitted: 0,
pliCount: 0,
score: 10,
ssrc: 91579,
timestamp: 7365341,
type: ‘inbound-rtp’
}
]

To me all looks good except for the very high PLI count. The client seems to be bombarding the server with this kind of requests.

But the problem is for sure on the client side. Since I don’t have any experience with the Android platform, it might be that I am doing something wrong over there. I will try to enable the debug logs as you guys describe it in the documentation and see if that helps me to identify the problem.

6

Check the remote SDP offer your are passing to your consumer client and verify if the sscrs and so on match the values in await consumer.dump() in server side.

7

Unsure how to do that, but I will have a look.

UPDATE: Just tried with VP8 which works fine. My guess now is that the H264 profile that I was using does not completely match the expected one by libmediasoup.

Seems very hard to find the definitive way of setting the H264 profile in libav …

It also seems impossible to find out what chrome is seeing as the full H264 profile (I can only see “Baseline” in webrtc-internals).

Anyway, I will try to have a look at the SDP offer, maybe I can see it there.

8

I’m afraid H264 negotiation is not easy. That’s why we use h264-profile-level-id in mediasoup and mediasoup-client to match H264 codecs:

9

The h264 codec is correctly matched by mediasoup (I even stepped with the debugger to see it).

I think we’re on to something in the other thread.