Kernel adjustment required to prevent dropped packets with WebRTCServer

Deployment & Scalability
1

I recently came across this thread and it piqued my curiosity:

Our servers were using the kernels defaults for rmem_default and wmem_default, and sure enough we were dropping a lot of packets unnecessarily.

You can check for dropped packets with this command:

ss -aum '(  sport > :4000 and sport < :5000 )'

Replace 4000 and 5000 with whatever you’re using for webrtc ports.

Sockets with a healthy buffer setting should say d0. Anything higher means
that the kernel is dropping packets.

These are the settings we use now. We could probably get away with something smaller, but so far we haven’t had any packet loss with these:

net.core.rmem_default = 2097152
net.core.rmem_max = 2097152
net.core.wmem_default = 2097152
net.core.wmem_max = 2097152

This requirement wasn’t obvious to me, so I thought I’d share it with the community.

If you are using the new WebRTCServer API, and you haven’t adjusted the default buffer size, then your kernel is probably dropping packets unnecessarily. The reason is that all of transports
are sharing the same set of socket buffers, so a bigger buffer is required.

If instead you’re using the listenIps option on createWebRtcTransport, then a new socket is created for each transport, and this isn’t as much of an issue.

This might be a good note to add under the WebRTCServer API docs.

5 Likes
2

This was quite useful… we had already done some tweaking to our kernel buffers and I thought things look good/better. However I was using ethtool, /proc/net/snmp and /proc/net/dev to check for errors and drops, all of which reported none. However ss is showing us still dropping some packets!

We bumped rmem/wmem vales to 512,000 previously but since we have plenty of mem I guess we’ll bump them again.

Just curious if you have any idea why, for example, SNMP stats, or netstats etc. would not reflect these drops?

3

When I was debugging this, I also noticed we had errors in using netstat -s and cat /proc/net/snmp | grep Udp:, so I don’t know why yours would be clean.

If it helps, the constants that track these stats in the Kernel are called UDP_MIB_RCVBUFERRORS and UDP_MIB_SNDBUFERRORS. They are defined here:

And incremented here:

So any tool that tracks these constants should work.

I suspect it might be possible to refactor mediasoup to bind each new transport to a new socket, even if they all share the same server port. But I don’t know enough about the mediasoup internals. Having every transport share the same socket buffer seems like it might come with some other inefficiencies that we haven’t been discovered yet.

1 Like