AWS, cloudfront, and s3 buckets

I have gotten myself a strange issue that I am not sure I understand enough to figure out. Hopefully I will figure it out while writing this post, but if that doesn’t happen, thanks for reading. :smiley:

It seems I am unable to connect webrtc transports when hosting my mediasoup-powered webapp with an aws s3 bucket/cloudfront.

I build my application, upload it to the s3 bucket, and distribute it globally via cloudfront. The signaling server is its own ec2 instance. The mediasoup server is also a separate ec2 instance. The issue arises when producing data or media to the mediasoup server from the client. Maybe a little more clearly: – client gets webapp from cloudfront/s3, client to signaling server for signaling dialog, and client to mediasoup servers for media exchange, according to what signaling tells client to do. This arrangement fails to connect to mediaservers, or even try ice candidates.

However, if I build the webapp on the signaling server itself and use the signaling server as the host for the webapp, everything works as expected.

If i build my webapp locally, and point my signaling to the signaling server, everything connects and works as expected.

This leads me to think that something am overlooking in the cloudfront/s3 or possibly the network handling mediasoup does. For instance, is there anything comparable to CORS headers that mediasoup client would need to communicate media to a server other than what it gets signaling from? I imagine this would be not the case because I pass the signaling from the mediasoup server back to the client untouched, so I think that would not matter but I don’t know it. Maybe there is something in the dtls fingerprints or some other value that gets generated client side by what domain you are currencly on or… guessing at that point.

When setup on the cloudfront/s3 bucket fails, I do indeed get ice parameters back with the correct IP of the mediasoup server with correct ports. And from what i read in the interface about:webrtc it has not even tried the ice(lite) candidates it gets in the offer/answer exchange. I do get the produce event on the client transport, and send up connection params to the mediasoup server and it returns an ID which is given to the produce callback. But… no dice on the client successfully connecting the transport. Ice/stun failure.

I am have also tried this with and without a paid-for stun/turn server, and it does not change the results.

My announce IP in the mediasoup config is set to the public IP of the aws server that mediasoup is running on and the local IP is the local interface address of the ec2 instance (local ipv4 address).

Does anyone know why I would get correct ice-lite in my sdp, but the client would not attempt to connect to them? I would hope to see all failed tries of the ice candidates at least…

You mean there are some remote ice candidates received from the server and passed to the browser, some local ice candidates, and the browser does not try to connect? What is it doing then?

After the client gets ice candidates from sdp negotiation, it tells me ice/sturn/turn failed to connect in the console. When i check about:webrtc in my browser, it says that it has not tried any ice candidates, neither failed or passed.

From “about:webrtc” I judge this is Firefox? There should be at least something in the “Connection Log” section.

Yup, about:webrtc in firefox. This is the dump from connection log. Ive changed the ipv4 and host names for obscurity. :smiley:
I dont see anything in this log tha jumps out at me, other than the ice candidates not being tried. They are being gotten though, but not used somehow. I suspect the URL parameter in the peer connection has something to do with this… but I feel like this would have been an issue for others if that was the case.

+++++++ BEGIN (process id 6973) ++++++++

Exit UDP socket connected

/builds/worker/checkouts/gecko/dom/media/webrtc/transport/third_party/nICEr/src/net/nr_socket_multi_tcp.c:617 function nr_socket_multi_tcp_listen failed with error 3

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): failed to create passive TCP host candidate: 3

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta..awss3host.com/techtruth):default) has no stream matching stream PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) no streams with non-empty check lists

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) no streams with pre-answer requests

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) no checks to start

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) pairing local trickle ICE candidate host(IP6:[2600:8800:7916:1800:428d:5cff:febc:86c4]:41252/UDP)

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) pairing local trickle ICE candidate host(IP6:[2600:8800:7916:1800:428d:5cff:febc:86c4]:50031/TCP) active

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): All candidates initialized

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth)): peer (PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default) Trickle grace period is over; marking every component with only failed pairs as failed.

ICE-PEER(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default)/STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a)/COMP(1): All pairs are failed, and grace period has elapsed. Marking component as failed.

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a): state dump

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a): Local component 1 - dumping candidates

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a)/CAND(+V5S): host(IP6:[2600:8800:7916:1800:428d:5cff:febc:86c4]:41252/UDP)

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a)/CAND(x+4o): host(IP6:[2600:8800:7916:1800:428d:5cff:febc:86c4]:50031/TCP) active

ICE-PEER(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default)/STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a): state dump

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a): Remote component 1 in state 3 - dumping candidates

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a)/CAND(E95c): candidate:udpcandidate 1 udp 1076558079 <publicIP> 10085 typ host

ICE(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth))/ICE-STREAM(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth) transport-id=transport_0 - e6b9f5b6:afbb2c92cd27e598c76ab001654b0f8a)/CAND(UAJq): candidate:tcpcandidate 1 tcp 1076302079 <publicIP> 10038 typ host tcptype passive

ICE-PEER(PC:{922a15f6-7678-42be-83ef-b2a6db278da8} 1613008771806614 (id=6442450950 url=https://beta.awss3host.com/techtruth):default): all checks completed success=0 fail=1

+++++++ END (process id 6973) ++++++++

Local cadidates are ok (host). But then the log looks exactly as if a localhost address was announced by the server. That <publicIP> was really public, not localhost by any chance? This is very strange, because you can place any other arbitrary IP address (including any private one) in the remote SDP, and the browser pairs it with the local candidates and starts trying.

@techtruth, I saw the log in the mail, and noticed that the remote address was IPv4, but there were only IPv6 local candidates. Probably this is the problem.

Ah! What a thing to overlook… It makes sense that it would not try the addresses if it had none to try!

I will give ipv6 some attention and see what I get into.

I was under the impression that if I supplied ipv4 addresses to the anounce and local IP that I would get ipv4 in the ice candidates. … Ill check that as well when I am in the code.

Hope I can mark this as solved soon. :slight_smile: