Memory issues after upgrading mediasoup to 3.18.0

Deployment & Scalability
1

We observed that the Mediasoup server’s memory usage started spiking after upgrading from version 3.12.0 to 3.18.0.

We implemented all the required breaking changes during the upgrade, such as updating listenIps to listenInfos, replacing announcedIp with announceAddress, and manually defining UDP or TCP priorities.

The graphs below show the server’s memory performance with Mediasoup 3.12.0 and after upgrading to 3.18.0. Following repeated crashes caused by high memory usage, we reverted back to version 3.12.0, after which the system stabilized again.
Are we missing something ?

mediasoup-client - 3.15.2
mediasoup server - 3.18.0 then reverted to 3.12.0
Node - 20.14.0

and there was openssl error that was happening in 3.12.0 but reduces in 3.18.0, but was still happening

image
image1264×774 182 KB

1 Like
2

3.12 was released on May 2023… Check which process (node|C++ worker) it using the memory. You are most probably having a leak on your application.

3

The thing is that they are using the very same Node application.

4

Node version we are at is - 20.14.0

5

I would suggest to try some intermediary version until you get the exact version where this memory usage behaviour changed. Also, don’t forget to check which process is taking the memory (node or mediasoup-worker)

6

We upgraded to 3.18.0 after checking the changelog for all intermediary versions.
There were memory spikes in 3.12.0 but very rare. to resolve those we upgraded to 3.18.0 because there very many fixes related to memory leak that got fixed between 3.12.0 to 3.18.0
We find 3.18.0 very stable based on changelog info and at last we have to revert back

7

Please try my previous suggestions.

8

Sure, checking the process

9

Probably not the culprit, but mediasoup latest versions require Node >= 22.

10

We have checked that the MediaSoup cores are taking memory and not releasing it even after the workers are closed. It will keep increase till the server crashed.

11

Closing a worker should terminate the spawned C++ process, and terminating a process will release it’s memory.

Can you confirm that calling `worker.close()` does not terminate the underlaying `mediasoup-worker` process?

12

The FlatBufferBuilder instances are pretty hefty, builder.clear() doesn’t actually free the memory, it just resets it for reuse so over time that may grow till the thread/worker is closed. If there’s a “leak” beyond that, I’d have to keep scouting.

This may explain why crashing could occur if there’s no memory available for retransmission or whatever scenario needs to fire off.

13

FlatBufferBuilder is either, `thread_local` or a private member for `ChannelSocket`, meaning it is released when the thread/process terminates or the `ChannelSocket` is destructed.

The buffers are only cleared so they are reused and there’s no need to re-allocate the buffer for each message. I don’t see any problem there. The buffers will be released once the `FlatBufferBuilder` destroys. We would have a lot of trouble if that was not the case.

If you want to discard this is the problem, then revert the PR that introduced that, here.

There are some unresponded questions. Please try to respond to the questions we make, so we can help.

14

AFAIK this has never happened and nobody has reported it. If memory exhausts then anything can fail obviously.