Client on IPv6 NAT can only complete ICE with send transports

Deployment & Scalability
1

I’m running MediaSoup on AWS ECS with both ipv4 and ipv6 listening address. My listenIps looks like this:

[
    { "ip":"2406:da12:f97:b302:e5de:b9ff:627d:d412" },
    { "ip":"172.31.2.28", "announcedIp":"13.124.39.172" }
]

I’m not using a STUN or TURN server.

I have a user that is coming from an ISP which gives him an IPv6 IP. However, he appears as an IPv4 IP to the rest of the Internet, and he is only able to connect to other IPv4 IPs. So he’s behind some kind of IPv6 NAT. He’s on a wired connection and using react-native-webrtc on iOS.

This user is able to create WebRTCTransports with my server but only in the receive direction.

When he creates a transport, both send and receive try to select IPv6 tuples first. The receive transport then falls back to the (correct) IPv4 tuple. The send transport seems stuck on the IPv6 tuple.

Log for the recv transport that worked:

2021-04-16T16:47:44.669000+00:00  [WebRtcTransport created]              direction=recv
2021-04-16T16:47:45.081000+00:00  [consumer created]                     consumerId=70253e88-8ea7-4344-9302-5ff328e9ba90 
2021-04-16T16:47:46.280000+00:00  [iceselectedtuplechange - transport]   tuple=[2406:da12:f97:b302:e5de:b9ff:627d:d412]:45819 <--> [2804:499c:1010:5900:f9d6:d37f:79f8:205e]:50070 (udp)
2021-04-16T16:47:46.280000+00:00  [iceState - transport]                 iceState=connected
2021-04-16T16:47:46.280000+00:00  [dtlsState - transport]                dtlsState=connecting
2021-04-16T16:47:46.655000+00:00  [iceselectedtuplechange - transport]   tuple=172.31.2.28:45703 <--> 45.229.212.158:2224 (udp)
2021-04-16T16:47:46.946000+00:00  [dtlsState - transport]                dtlsState=connected
2021-04-16T16:47:47.549000+00:00  [iceState - transport]                 iceState=completed
2021-04-16T16:48:08.419000+00:00  [socket-io closes]

Mediasoup log while created send transport (Gets stuck):

2021-04-16T16:47:49.486000+00:00  [WebRtcTransport created]                 direction=send
2021-04-16T16:47:50.176000+00:00  [producer created]                        producerId=fbec1fa0-4c5b-470d-9ac2-463f6566bc68 
2021-04-16T16:47:50.198000+00:00  [iceselectedtuplechange - transport]      tuple=[2406:da12:f97:b302:e5de:b9ff:627d:d412]:42634 <--> [2804:499c:1010:5900:f9d6:d37f:79f8:205e]:54058 (udp)
2021-04-16T16:47:50.198000+00:00  [iceState - transport]                    iceState=connected
2021-04-16T16:47:50.198000+00:00  [dtlsState - transport]                   dtlsState=connecting
2021-04-16T16:48:08.420000+00:00  [socket-io close]

Interestingly, on the client side I can look at getStats() for the send transport. It looks like the client side has already moved onto the correct IPv4 tuple, but I haven’t gotten the corresponding iceselectedtuplechange on mediasoup.

// stats for the correct ipv4 tuple
"RTCIceCandidatePair_JMS4xwQ6_CE13iTBU": {
        "id": "RTCIceCandidatePair_JMS4xwQ6_CE13iTBU",
        "type": "candidate-pair",
        "state": "succeeded",
        "priority": "4623671794094325247",
        "writable": "1",
        "bytesSent": "0",
        "nominated": "1",
        "timestamp": 1618591684453.631,
        "transportId": "RTCTransport_audio_1",
        "requestsSent": "2",
        "bytesReceived": "0",
        "responsesSent": "0",
        "localCandidateId": "RTCIceCandidate_JMS4xwQ6",
        "requestsReceived": "0",
        "remoteCandidateId": "RTCIceCandidate_CE13iTBU",
        "responsesReceived": "10",
        "totalRoundTripTime": "3.037",
        "consentRequestsSent": "8",
        "currentRoundTripTime": "0.286"
},

// stats for the failed ipv6 tuple
"RTCIceCandidatePair_R3mjjoRD_rpvkPlCQ": {
        "id": "RTCIceCandidatePair_R3mjjoRD_rpvkPlCQ",
        "type": "candidate-pair",
        "state": "in-progress",
        "priority": "4623781745794109951",
        "writable": "0",
        "bytesSent": "0",
        "nominated": "0",
        "timestamp": 1618591684453.631,
        "transportId": "RTCTransport_audio_1",
        "requestsSent": "141",
        "bytesReceived": "0",
        "responsesSent": "0",
        "localCandidateId": "RTCIceCandidate_R3mjjoRD",
        "requestsReceived": "0",
        "remoteCandidateId": "RTCIceCandidate_rpvkPlCQ",
        "responsesReceived": "0",
        "totalRoundTripTime": "0",
        "consentRequestsSent": "0"
},

My questions:

What is the difference between the ice handshake on send transports and receive transports? I had thought they were pretty much the same, but this user seems to be consistently able to create recv transports, but not sends. Anyone have a good tip on where to look next?

Thank you!

2

ICE is the same in both send and receive transports because mediasoup is ICE Lite so the client is the only that sends ICE Binding requests to the server. DTLS roles may change depending on whether the transport is for send or receive, but it does not look to be your problem.

3

Or actually it is because mediasoup tries to send DTLS handshake messages from the IPv6 which does not seem to work in your scenario.

4

OK thank you! I will look more into DTLS roles.

I had considered not offering IPv6 candidates to clients that come in over a IPv4 connection over socketio, but I want to exhaust all my options with letting ICE figure things out first.

5

mediasoup has been tested with IPv6 but I don’t about those IPv6-IPv4 tunnels and how they can affect things.

6

The problem in the failing case is that mediasoup did properly receive an ICE request in the IPv6 and selected such a tuple. Then it tries to send DTLS to client over it and it does not reach the client. You should verify whether the client is receiving ICE responses on that tuple. You may have to change announcedIp for that IPv6.

7

OK, so it sounds like maybe [Client IPv6] → [Mediasoup IPv6] is working, but [Mediasoup IPv6] → [Client IPv6] doesn’t work. And this is why mediasoup selected that tuple first.

I don’t think the client is receiving any responses, because “bytesReceived” and “responsesReceived” are all zero in the client report. I’ll keep investigating!

8

Makes sense. mediasoup selects a tuple based on the last ICE candidate it receives from the client. It may change dynamically. And mediasoup replies to those candidates using the same tuple but, obviously, it doesn’t know whether the client received the response (it should if the networking is ok).

9

OK, for the sake of anyone that finds this, I will summarize what I think the issue is and how I ended up solving it.

Network situation:
Client is in some kind of IPv6-in-IPv4 NAT
IPv6 tuple works, but only in the direction of client → server.
IPv4 tuple works in both directions.

Transport situation.
recv-transports are able to successfully land on the ipv4 tuple and work properly.
send-transports never finish the dtls handshake. Presumably this is caused by the one-way nature of the IPv6 tuple. This only seems to be an issue when “role” = “server” in dtlsParameters, which is the case with send transports (not receive transports).

Solution:
I am using socketio as my signaling mechanism. I’m using the ipv4/ipv6 state of the socket remote address to act as a gatekeeper to all my IPv6 listenIps. I figure, if a client’s network isn’t capable of doing a ipv6 tcp handshake with my websocket server, there’s a good chance WebRTC isn’t going to go well.

This solution seems to work for now. A more graceful solution might be to figure out how to get DTLS to recover and switch to the IPv4 tuple.

Here are some abbreviated code snippets:

import { isIPv6 } from "net";

function socketClientAddress(socket: Socket): string {
  // If you are using a load balancer, you probably need to look for x-forwarded-for.
  // If not, you should get the address directly off the socket.
  const headerName = "x-forwarded-for";
  const headers: any = socket.handshake.headers;
  if (typeof headers[headerName] === "string") {
    return headers[headerName];
  } else {
    return socket.handshake.address;
  }
}

async function createTransport(socket: Socket) {
    const socketAddress = socketClientAddress(socket);
    const enableIPV6 = !socketAddress.startsWith("::ffff") && isIPv6(socketAddress);

    const networkConfig = await getNetworkConfig();

    let listenIps;
    if (enableIPV6) {
      listenIps = networkConfig.listenIps;
    } else {
      listenIps = networkConfig.listenIps.filter((iter) => !isIPv6(iter.ip));
    }

    const transport = await router.createWebRtcTransport({
      listenIps: listenIps,
      ... other parameters
    });
}

Thanks again for your help ibc!