Hi team,
After upgrading to Mediasoup v3.16.1, I started experiencing a critical issue where the worker dies unexpectedly when a user attempts to join a room.
Here’s a snippet from my logs:
[6/28/2025, 08:01:52:620] [Server] Mediasoup worker died, exiting in 2 seconds...
Has anything changed in v3.16.1 that could be related to this behavior?
Note: On mediasoup 3.16.0 seems not happen.
Thank you!
Do you have an SRT in the mediasoup-demo? Otherwise, can you please share the coredump?
Hi jmillan,
Actually mediasoup-demo is experiencing an issue with WebSocket.
On my end, would the following log be sufficient for your investigation, or do you also need the core dump to analyze it further?
mediasoup:ERROR:Worker (stderr) libc++abi: terminating due to uncaught exception of type std::out_of_range: absl::container_internal::raw_hash_map<>::at undefined
mediasoup:Consumer resume()
mediasoup:Channel request() [method:CONSUMER_RESUME]
mediasoup:Channel Consumer Channel ended by the worker process
mediasoup:Channel Producer Channel ended by the worker process
mediasoup:ERROR:Worker worker process died unexpectedly [pid:9639, code:null, signal:SIGABRT]
mediasoup:Worker workerDied() [error:Error: [pid:9639, code:null, signal:SIGABRT]]
mediasoup:Channel close()
mediasoup:Router workerClosed()
mediasoup:Transport routerClosed()
mediasoup:Producer transportClosed()
or do you also need the core dump to analyze it further?
If you can yes, please. We would go directly to the problem.
We don’t need the core dump itself. The output of bt full would suffice.
From the backtrace, the crash is a SIGABRT triggered inside:
RTC::SimulcastConsumer::SendRtpPacket()
with an abort caused by:
absl::lts_20240722::base_internal::ThrowStdOutOfRange(char const*)
which means somewhere in the mediasoup code (or its dependencies) there’s an out-of-range access, very likely with an absl::flat_hash_map::at() call.
coredumps:
{"app_name":"mediasoup-worker","timestamp":"2025-06-28 11:59:37.00 +0200","app_version":"","slice_uuid":"54898629-572b-3955-828d-3fa88217d309","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 15.5 (24F74)","roots_installed":0,"incident_id":"EBB5F196-8901-43A2-8EF1-90C4923BB6EB","name":"mediasoup-worker"}
{
"uptime" : 10000,
"procRole" : "Unspecified",
"version" : 2,
"userID" : 501,
"deployVersion" : 210,
"modelCode" : "MacBookPro15,1",
"coalitionID" : 2119,
"osVersion" : {
"train" : "macOS 15.5",
"build" : "24F74",
"releaseType" : "User"
},
"captureTime" : "2025-06-28 11:59:36.6392 +0200",
"codeSigningMonitor" : 0,
"incident" : "EBB5F196-8901-43A2-8EF1-90C4923BB6EB",
"pid" : 50949,
"cpuType" : "X86-64",
"roots_installed" : 0,
"bug_type" : "309",
"procLaunch" : "2025-06-28 11:59:20.6195 +0200",
"procStartAbsTime" : 10846413003861,
"procExitAbsTime" : 10862431507259,
"procName" : "mediasoup-worker",
"procPath" : "\/Users\/USER\/*\/mediasoup-worker",
"parentProc" : "node",
"parentPid" : 50946,
"coalitionName" : "com.googlecode.iterm2",
"crashReporterKey" : "FAAB67DF-A122-934B-5267-290C11A9C28E",
"appleIntelligenceStatus" : {"reasons":["deviceNotCapable"],"state":"unavailable"},
"responsiblePid" : 8197,
"responsibleProc" : "iTerm2",
"codeSigningID" : "",
"codeSigningTeamID" : "",
"codeSigningValidationCategory" : 0,
"codeSigningTrustLevel" : 4294967295,
"codeSigningAuxiliaryInfo" : 0,
"bootSessionUUID" : "4362D546-CFD9-446E-8755-2EDD6AF7D367",
"wakeTime" : 10645,
"bridgeVersion" : {"build":"22P5072","train":"9.5"},
"sleepWakeUUID" : "96D743C7-5298-4259-BA18-C1A2CA59EE0C",
"sip" : "enabled",
"exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
"termination" : {"flags":0,"code":6,"namespace":"SIGNAL","indicator":"Abort trap: 6","byProc":"mediasoup-worker","byPid":50949},
"asi" : {"libsystem_c.dylib":["abort() called"]},
"extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
"faultingThread" : 0,
"threads" : [{"triggered":true,"id":281405,"threadState":{"r13":{"value":206158430216},"rax":{"value":0},"rflags":{"value":582},"cpu":{"value":0},"r14":{"value":259},"rsi":{"value":6},"r8":{"value":140701958816560},"cr2":{"value":0},"rdx":{"value":0},"r10":{"value":140704319547840,"symbolLocation":0,"symbol":"_os_xbs_chrooted"},"r9":{"value":0},"r15":{"value":22},"rbx":{"value":6},"trap":{"value":133},"err":{"value":33554760},"r11":{"value":582},"rip":{"value":140703216101446,"matchesCrashFrame":1},"rbp":{"value":140701958816912},"rsp":{"value":140701958816872},"r12":{"value":140704319515264,"symbolLocation":0,"symbol":"__stderrp"},"rcx":{"value":140701958816872},"flavor":"x86_THREAD_STATE","rdi":{"value":259}},"queue":"com.apple.main-thread","frames":[{"imageOffset":30790,"symbol":"__pthread_kill","symbolLocation":10,"imageIndex":1},{"imageOffset":23318,"symbol":"pthread_kill","symbolLocation":259,"imageIndex":2},{"imageOffset":526142,"symbol":"abort","symbolLocation":126,"imageIndex":3},{"imageOffset":65920,"symbol":"abort_message","symbolLocation":241,"imageIndex":4},{"imageOffset":4836,"symbol":"demangling_terminate_handler()","symbolLocation":240,"imageIndex":4},{"imageOffset":115447,"symbol":"_objc_terminate()","symbolLocation":104,"imageIndex":5},{"imageOffset":62923,"symbol":"std::__terminate(void (*)())","symbolLocation":6,"imageIndex":4},{"imageOffset":74327,"symbol":"__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*)","symbolLocation":27,"imageIndex":4},{"imageOffset":74300,"symbol":"__cxa_throw","symbolLocation":69,"imageIndex":4},{"imageOffset":1126600,"symbol":"absl::lts_20240722::base_internal::ThrowStdOutOfRange(char const*)","symbolLocation":56,"imageIndex":0},{"imageOffset":583781,"symbol":"decltype(absl::lts_20240722::container_internal::FlatHashMapPolicy<unsigned int, short>::value(std::__1::pair<unsigned int const, short>* std::__1::addressof[abi:v160006]<std::__1::pair<unsigned int const, short>>(std::__1::pair<unsigned int const, short>&)(decltype(std::__declval<std::__1::pair<unsigned int const, short>>(0)) std::__1::declval<std::__1::pair<unsigned int const, short>&>()()))) absl::lts_20240722::container_internal::raw_hash_map<absl::lts_20240722::container_internal::FlatHashMapPolicy<unsigned int, short>, absl::lts_20240722::hash_internal::Hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, short>>>::at<unsigned int, absl::lts_20240722::container_internal::FlatHashMapPolicy<unsigned int, short>>(unsigned int const&)","symbolLocation":277,"imageIndex":0},{"imageOffset":581221,"symbol":"RTC::SimulcastConsumer::SendRtpPacket(RTC::RtpPacket*, std::__1::shared_ptr<RTC::RtpPacket>&)","symbolLocation":53,"imageIndex":0},{"imageOffset":583418,"symbol":"RTC::SimulcastConsumer::SendRtpPacket(RTC::RtpPacket*, std::__1::shared_ptr<RTC::RtpPacket>&)","symbolLocation":2250,"imageIndex":0},{"imageOffset":415445,"symbol":"RTC::Router::OnTransportProducerRtpPacketReceived(RTC::Transport*, RTC::Producer*, RTC::RtpPacket*)","symbolLocation":485,"imageIndex":0},{"imageOffset":352307,"symbol":"RTC::Producer::ReceiveRtpPacket(RTC::RtpPacket*)","symbolLocation":787,"imageIndex":0},{"imageOffset":648166,"symbol":"RTC::Transport::ReceiveRtpPacket(RTC::RtpPacket*)","symbolLocation":118,"imageIndex":0},{"imageOffset":740192,"symbol":"RTC::WebRtcTransport::OnRtpDataReceived(RTC::TransportTuple*, unsigned char const*, unsigned long)","symbolLocation":160,"imageIndex":0},{"imageOffset":747254,"symbol":"non-virtual thunk to RTC::WebRtcTransport::OnUdpSocketPacketReceived(RTC::UdpSocket*, unsigned char const*, unsigned long, sockaddr const*)","symbolLocation":166,"imageIndex":0},{"imageOffset":3796808,"symbol":"uv__udp_io","symbolLocation":904,"imageIndex":0},{"imageOffset":3803956,"symbol":"uv__io_poll","symbolLocation":2084,"imageIndex":0},{"imageOffset":3732785,"symbol":"uv_run","symbolLocation":369,"imageIndex":0},{"imageOffset":22940,"symbol":"DepLibUV::RunLoop()","symbolLocation":28,"imageIndex":0},{"imageOffset":61361,"symbol":"Worker::Worker(Channel::ChannelSocket*)","symbolLocation":385,"imageIndex":0},{"imageOffset":16849,"symbol":"mediasoup_worker_run","symbolLocation":465,"imageIndex":0},{"imageOffset":1122786,"symbol":"main","symbolLocation":274,"imageIndex":0},{"imageOffset":25904,"symbol":"start","symbolLocation":3056,"imageIndex":6}]},{"id":281406,"name":"SCTP iterator","threadState":{"r13":{"value":4294967552},"rax":{"value":260},"rflags":{"value":583},"cpu":{"value":0},"r14":{"value":4395950488,"symbolLocation":128,"symbol":"sctp_it_ctl"},"rsi":{"value":4294967552},"r8":{"value":0},"cr2":{"value":0},"rdx":{"value":0},"r10":{"value":0},"r9":{"value":160},"r15":{"value":0},"rbx":{"value":123145391951872},"trap":{"value":133},"err":{"value":33554737},"r11":{"value":582},"rip":{"value":140703216084726},"rbp":{"value":123145391951744},"rsp":{"value":123145391951592},"r12":{"value":123145391951616},"rcx":{"value":123145391951592},"flavor":"x86_THREAD_STATE","rdi":{"value":4395950488,"symbolLocation":128,"symbol":"sctp_it_ctl"}},"frames":[{"imageOffset":14070,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":25210,"symbol":"_pthread_cond_wait","symbolLocation":988,"imageIndex":2},{"imageOffset":3931115,"symbol":"sctp_iterator_thread","symbolLocation":75,"imageIndex":0},{"imageOffset":24049,"symbol":"_pthread_start","symbolLocation":99,"imageIndex":2},{"imageOffset":6231,"symbol":"thread_start","symbolLocation":15,"imageIndex":2}]}],
"usedImages" : [
{
"source" : "P",
"arch" : "x86_64",
"base" : 4389924864,
"size" : 5046272,
"uuid" : "54898629-572b-3955-828d-3fa88217d309",
"path" : "\/Users\/USER\/*\/mediasoup-worker",
"name" : "mediasoup-worker"
},
{
"source" : "P",
"arch" : "x86_64",
"base" : 140703216070656,
"size" : 248656,
"uuid" : "dab10aa4-8afa-3d02-9cde-6023554ac858",
"path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
"name" : "libsystem_kernel.dylib"
},
{
"source" : "P",
"arch" : "x86_64",
"base" : 140703216320512,
"size" : 48592,
"uuid" : "a6d1f05a-0743-31b7-9fe2-268f06ccd51a",
"path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
"name" : "libsystem_pthread.dylib"
},
{
"source" : "P",
"arch" : "x86_64",
"base" : 140703214895104,
"size" : 560296,
"uuid" : "19f23df9-b4a5-3be3-8777-5c8f0c5a43c8",
"path" : "\/usr\/lib\/system\/libsystem_c.dylib",
"name" : "libsystem_c.dylib"
},
{
"source" : "P",
"arch" : "x86_64",
"base" : 140703215976448,
"size" : 94200,
"uuid" : "b125268e-2b0b-3441-8015-be6e7f16c513",
"path" : "\/usr\/lib\/libc++abi.dylib",
"name" : "libc++abi.dylib"
},
{
"source" : "P",
"arch" : "x86_64h",
"base" : 140703212208128,
"size" : 282716,
"uuid" : "de35b5ee-5872-3e5a-a1f0-eb18c5b0434c",
"path" : "\/usr\/lib\/libobjc.A.dylib",
"name" : "libobjc.A.dylib"
},
{
"source" : "P",
"arch" : "x86_64",
"base" : 140703212494848,
"size" : 632104,
"uuid" : "3771ea6a-0fe5-3b63-961d-c09e01d5e680",
"path" : "\/usr\/lib\/dyld",
"name" : "dyld"
},
{
"size" : 0,
"source" : "A",
"base" : 0,
"uuid" : "00000000-0000-0000-0000-000000000000"
}
],
"sharedCache" : {
"base" : 140703184052224,
"size" : 30064771072,
"uuid" : "57b0c2b8-36d6-3cf0-8bd0-0ac6133ec8b3"
},
"vmSummary" : "ReadOnly portion of Libraries: Total=402.5M resident=0K(0%) swapped_out_or_unallocated=402.5M(100%)\nWritable regions: Total=1.0G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.0G(100%)\n\n VIRTUAL REGION \nREGION TYPE SIZE COUNT (non-coalesced) \n=========== ======= ======= \nKernel Alloc Once 8K 1 \nMALLOC 1.0G 36 \nMALLOC guard page 24K 6 \nSTACK GUARD 56.0M 2 \nStack 8712K 2 \n__DATA 4619K 279 \n__DATA_CONST 22.3M 299 \n__DATA_DIRTY 436K 91 \n__FONT_DATA 2352 1 \n__LINKEDIT 163.6M 2 \n__OBJC_RO 61.3M 1 \n__OBJC_RW 2395K 2 \n__TEXT 238.9M 305 \n__TPRO_CONST 16 2 \nshared memory 36K 4 \n=========== ======= ======= \nTOTAL 1.5G 1033 \n",
"legacyInfo" : {
"threadTriggered" : {
"queue" : "com.apple.main-thread"
}
},
"logWritingSignature" : "03a3a4c51244be6792f2c3889778fd6e72e32cab",
"trialInfo" : {
"rollouts" : [
{
"rolloutId" : "63f9578e238e7b23a1f3030a",
"factorPackIds" : {
},
"deploymentId" : 240000005
},
{
"rolloutId" : "661464ecda55e5192b100804",
"factorPackIds" : {
},
"deploymentId" : 240000005
}
],
"experiments" : [
]
}
}
Thank you!
Can we please move this to GitHub issues?
Issue created here: Worker crash: 'std::out_of_range' in `absl::container_internal::raw_hash_map<>::at undefined` · Issue #1554 · versatica/mediasoup · GitHub
I’ve included a core dump in the issue: Worker crash: 'std::out_of_range' in `absl::container_internal::raw_hash_map<>::at undefined` · Issue #1554 · versatica/mediasoup · GitHub
Fix here after a one2one session with @jmillan:
I’m still experiencing a crash in the mediasoup worker for same reason, even after upgrading to the latest version (v3.16.3).
I did tons of tests and it never crashes. Please comment in the GH issue and paste the error log you see in there:
And also comment (in the GH issue) which codec you are using for video, etc.
Bug has been completely fixed in PR `Consumer` classes: Really fix target layer retransmission buffer by ibc · Pull Request #1558 · versatica/mediasoup · GitHub, now merged. New release soon.
It seems the issue is no longer occurring, I’ll continue to monitor it throughout the day.
Thank you so much for the great work!