ok it makes sense because it’s a node_modules folder within another node_modules. Yes, make should work.
The devel branch has serious issues. LOTS of crash and I had to switch back to the master branch. 
Opss, have you got any code dump or error logs?
Bunch of them 
. There are 111 core dumps of type sig6 and sig11. I will send the first and the last ones in separate posts due to the limit of body size.
The first core dump (int6):
gdb /var/conference/server/node_modules/mediasoup-devel/worker/out/Release/mediasoup-worker /tmp/cores/core.mediasoup-worke.sig6.28172
GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/conference/server/node_modules/mediasoup-devel/worker/out/Release/mediasoup-worker...done.
warning: core file may not match specified executable file.
[New LWP 28172]
[New LWP 28174]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/conference/server/node_modules/mediasoup/worker/out/Release/mediasoup-work'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f8b3d9ba428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8b3ee93740 (LWP 28172))]
(gdb) bt full
#0 0x00007f8b3d9ba428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
resultvar = 0
pd = <optimized out>
pid = 28172
selftid = 28172
#1 0x00007f8b3d9bc02a in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0, 4604418534313441280, 0,
4602678819172646912, 0, 140236014527939, 140236022984560, 0, 140236011074569, 140236014527808, 140236014527808, 10,
36204256, 140236022984560, 140236011075611, 140236014528256}}, sa_flags = 1045631264,
sa_restorer = 0x7f8b3dd4a700 <stderr>}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007f8b3e526242 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3 0x00007f8b3e531e86 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4 0x00007f8b3e531ed1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5 0x00007f8b3e532105 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6 0x00007f8b3e528b7a in std::__throw_bad_function_call() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#7 0x00000000005d6403 in std::function<void (bool)>::operator()(bool) const (this=0x7fff39131270, __args#0=true)
at /usr/include/c++/5/functional:2266
No locals.
#8 0x00000000005d6378 in TcpConnection::OnUvWrite(int, std::function<void (bool)> const&) (this=0x1ea34e0, status=0, onDone=...)
at ../src/handles/TcpConnection.cpp:487
No locals.
#9 0x00000000005d505e in onWrite (req=0x2d34a80, status=0) at ../src/handles/TcpConnection.cpp:47
writeData = 0x2d34a80
handle = 0x1ea08c0
connection = 0x1ea34e0
onDone = 0x7fff39131270
#10 0x0000000000758a22 in uv__write_callbacks (stream=0x1ea08c0) at ../deps/libuv/src/unix/stream.c:953
req = 0x2d34a80
q = 0x2d34ad8
pq = {0x7fff39131810, 0x7fff39131810}
__PRETTY_FUNCTION__ = "uv__write_callbacks"
#11 0x000000000075977a in uv__stream_io (loop=0x1d30d40, w=0x1ea0948, events=4) at ../deps/libuv/src/unix/stream.c:1329
stream = 0x1ea08c0
__PRETTY_FUNCTION__ = "uv__stream_io"
#12 0x000000000075eef5 in uv__io_poll (loop=0x1d30d40, timeout=36) at ../deps/libuv/src/unix/linux-core.c:384
max_safe_timeout = 1789569
events = {{events = 4, data = {ptr = 0x23, fd = 35, u32 = 35, u64 = 35}}, {events = 1, data = {ptr = 0x23, fd = 35,
u32 = 35, u64 = 35}}, {events = 1, data = {ptr = 0xe, fd = 14, u32 = 14, u64 = 14}}, {events = 1, data = {
ptr = 0x12, fd = 18, u32 = 18, u64 = 18}}, {events = 1, data = {ptr = 0xe, fd = 14, u32 = 14, u64 = 14}}, {
--Type <RET> for more, q to quit, c to continue without paging--c
events = 1, data = {ptr = 0x20, fd = 32, u32 = 32, u64 = 32}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 668 times>, {events = 30743600, data = {ptr = 0x1d528a800000000, fd = 0, u32 = 0, u64 = 132056466096914432}}, {events = 0, data = {ptr = 0x499b75 <BN_CTX_get+85>, fd = 4823925, u32 = 4823925, u64 = 4823925}}, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x8, fd = 8, u32 = 8, u64 = 8}}, {events = 30746768, data = {ptr = 0x1d51c3000000000, fd = 0, u32 = 0, u64 = 132042756561305600}}, {events = 0, data = {ptr = 0x4a9da7 <bn_sqr_fixed_top+103>, fd = 4890023, u32 = 4890023, u64 = 4890023}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 16, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 14 times>, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x1d52890, fd = 30746768, u32 = 30746768, u64 = 30746768}}, {events = 30747768, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x4a338f <bn_from_montgomery_word+735>, fd = 4862863, u32 = 4862863, u64 = 4862863}}, {events = 4294967295, data = {ptr = 0x1d52c80ffffffff, fd = -1, u32 = 4294967295, u64 = 132060696639700991}}, {events = 0, data = {ptr = 0x1d52830, fd = 30746672, u32 = 30746672, u64 = 30746672}}, {events = 8, data = {ptr = 0x1d5289000000000, fd = 0, u32 = 0, u64 = 132056363017699328}}, {events = 0, data = {ptr = 0x1d51c30, fd = 30743600, u32 = 30743600, u64 = 30743600}}, {events = 30746768, data = {ptr = 0x391344c000000000, fd = 0, u32 = 0, u64 = 4112706476143804416}}, {events = 32767, data = {ptr = 0x1d52830, fd = 30746672, u32 = 30746672, u64 = 30746672}}, {events = 30748144, data = {ptr = 0x391344c000000000, fd = 0, u32 = 0, u64 = 4112706476143804416}}, {events = 32767, data = {ptr = 0x4a3684 <bn_from_mont_fixed_top+84>, fd = 4863620, u32 = 4863620, u64 = 4863620}}, {events = 30746672, data = {ptr = 0x3913453000000000, fd = 0, u32 = 0, u64 = 4112706957180141568}}, {events = 32767, data = {ptr = 0x7fff391344c0, fd = 957564096, u32 = 957564096, u64 = 140734150952128}}, {events = 30743600, data = {ptr = 0x391344e000000000, fd = 0, u32 = 0, u64 = 4112706613582757888}}, {events = 32767, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 957564208, data = {ptr = 0x391344c000007fff, fd = 32767, u32 = 32767, u64 = 4112706476143837183}}, {events = 32767, data = {ptr = 0x1d51c30, fd = 30743600, u32 = 30743600, u64 = 30743600}}, {events = 4828985, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x3ec85e1400000000, fd = 0, u32 = 0, u64 = 4523969265686020096}}, {events = 32651, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 832, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 75 times>, {events = 0, data = {ptr = 0x3913448000000000, fd = 0, u32 = 0, u64 = 4112706201265897472}}, {events = 32767, data = {ptr = 0x7fff39134470, fd = 957564016, u32 = 957564016, u64 = 140734150952048}}, {events = 9127685, data = {ptr = 0x391345a800000000, fd = 0, u32 = 0, u64 = 4112707472576217088}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 1033709869, data = {ptr = 0x7f8b, fd = 32651, u32 = 32651, u64 = 32651}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 117, data = {ptr = 0x39133fe800000000, fd = 0, u32 = 0, u64 = 4112701150384357376}}, {events = 32767, data = {ptr = 0x7fff39133fd8, fd = 957562840, u32 = 957562840, u64 = 140734150950872}}, {events = 0, data = {ptr = 0x7f8b, fd = 32651, u32 = 32651, u64 = 32651}}, {events = 0, data = {ptr = 0x7fff39133ff0, fd = 957562864, u32 = 957562864, u64 = 140734150950896}}, {events = 0, data = {ptr = 0x3e495d803ffa8000, fd = 1073381376, u32 = 1073381376, u64 = 4488221309061988352}}, {events = 32651, data = {ptr = 0x7fff39134500, fd = 957564160, u32 = 957564160, u64 = 140734150952192}}, {events = 957564144, data = {ptr = 0x8b470500007fff, fd = 32767, u32 = 32767, u64 = 39203108563222527}}, {events = 0, data = {ptr = 0x7fff39134628, fd = 957564456, u32 = 957564456, u64 = 140734150952488}}, {events = 0, data = {ptr = 0x3d9d292d00000000, fd = 0, u32 = 0, u64 = 4439750080907444224}}, {events = 32651, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x3913447500000000, fd = 0, u32 = 0, u64 = 4112706154021257216}}, {events = 0, data = {ptr = 0x7fff39134068, fd = 957562984, u32 = 957562984, u64 = 140734150951016}}, {events = 957562968, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 1, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 957562992, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 1073397760, data = {ptr = 0xb00000000, fd = 0, u32 = 0, u64 = 47244640256}}, {events = 0, data = {ptr = 0x8b472000000000, fd = 0, u32 = 0, u64 = 39203224527306752}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x391340803ffac000, fd = 1073397760, u32 = 1073397760, u64 = 4112701804292784128}}, {events = 32767, data = {ptr = 0x7fff00000000, fd = 0, u32 = 0, u64 = 140733193388032}}, {events = 10, data = {ptr = 0x300000000, fd = 0, u32 = 0, u64 = 12884901888}}, {events = 0, data = {ptr = 0x7fff391344b5, fd = 957564085, u32 = 957564085, u64 = 140734150952117}}, {events = 3, data = {ptr = 0x8b470800000000, fd = 0, u32 = 0, u64 = 39203121448091648}}, {events = 0, data = {ptr = 0x100000000, fd = 0, u32 = 0, u64 = 4294967296}}, {events = 957563088, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 0, data = {ptr = 0xffffffffffffffff, fd = -1, u32 = 4294967295, u64 = 18446744073709551615}}, {events = 0, data = {ptr = 0x391347300000000b, fd = 11, u32 = 11, u64 = 4112709156203397131}}, {events = 32767, data = {ptr = 0x8b4709, fd = 9127689, u32 = 9127689, u64 = 9127689}}, {events = 27, data = {ptr = 0x3913474000000000, fd = 0, u32 = 0, u64 = 4112709224922873856}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 1046262287, data = {ptr = 0x3913446000007f8b, fd = 32651, u32 = 32651, u64 = 4112706063826976651}}, {events = 32767, data = {ptr = 0x1b, fd = 27, u32 = 27, u64 = 27}}, {events = 9, data = {ptr = 0x1000000000, fd = 0, u32 = 0, u64 = 68719476736}}, {events = 48, data = {ptr = 0x7fff39134700, fd = 957564672, u32 = 957564672, u64 = 140734150952704}}, {events = 957564480, data = {ptr = 0x3913412000007fff, fd = 32767, u32 = 32767, u64 = 4112702490414186495}}, {events = 0, data = {ptr = 0x7fff39134730, fd = 957564720, u32 = 957564720, u64 = 140734150952752}}, {events = 3337404145, data = {ptr = 0x3913410fffff8000, fd = -32768, u32 = 4294934528, u64 = 4112702421694644224}}, {events = 32767, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 9, data = {ptr = 0x32, fd = 50, u32 = 50, u64 = 50}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 91, data = {ptr = 0x6e, fd = 110, u32 = 110, u64 = 110}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 119, data = {ptr = 0x391344600000007c, fd = 124, u32 = 124, u64 = 4112706063826944124}}, {events = 32767, data = {ptr = 0x7fff39134730, fd = 957564720, u32 = 957564720, u64 = 140734150952752}}, {events = 57, data = {ptr = 0x3913474000000000, fd = 0, u32 = 0, u64 = 4112709224922873856}}, {events = 32767, data = {ptr = 0x100, fd = 256, u32 = 256, u64 = 256}}, {events = 9150464, data = {ptr = 0x3913416000000000, fd = 0, u32 = 0, u64 = 4112702765292060672}}, {events = 32767, data = {ptr = 0x5ec72f <std::array<unsigned char, 400ul>::operator[](unsigned long) const+35>, fd = 6211375, u32 = 6211375, u64 = 6211375}}, {events = 256, data = {ptr = 0x3e5cc77c00000000, fd = 0, u32 = 0, u64 = 4493685863598784512}}, {events = 32651, data = {ptr = 0x7fff391341a0, fd = 957563296, u32 = 957563296, u64 = 140734150951328}}, {events = 47583152, data = {ptr = 0x5faeb600000000, fd = 0, u32 = 0, u64 = 26932219494793216}}, {events = 0, data = {ptr = 0x5faee5 <nlohmann::detail::output_string_adapter<char, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::write_characters(char const*, unsigned long)+47>, fd = 6270693, u32 = 6270693, u64 = 6270693}}, {events = 957563296, data = {ptr = 0x2400007fff, fd = 32767, u32 = 32767, u64 = 154618855423}}, {events = 0, data = {ptr = 0x7fff391344aa, fd = 957564074, u32 = 957564074, u64 = 140734150952106}}, {events = 47583152, data = {ptr = 0x391342c000000000, fd = 0, u32 = 0, u64 = 4112704277120548864}}, {events = 32767, data = {ptr = 0x5e7afc <nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> >::dump_escaped(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+2680>, fd = 6191868, u32 = 6191868, u64 = 6191868}}, {events = 120, data = {ptr = 0xa932fc0000000000, fd = 0, u32 = 0, u64 = 12192084218175553536}}, {events = 1685028352, data = {ptr = 0x2218c70, fd = 35753072, u32 = 35753072, u64 = 35753072}}, {events = 957563984, data = {ptr = 0x3913424000007fff, fd = 32767, u32 = 32767, u64 = 4112703727364767743}}, {events = 32767, data = {ptr = 0x6666130030, fd = 1712521264, u32 = 1712521264, u64 = 439799185456}}, {events = 36, data = {ptr = 0x2400000000, fd = 0, u32 = 0, u64 = 154618822656}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 36, data = {ptr = 0x3913424000000000, fd = 0, u32 = 0, u64 = 4112703727364734976}}, {events = 0, data = {ptr = 0x2d60fb0, fd = 47583152, u32 = 47583152, u64 = 47583152}}, {events = 6270646, data = {ptr = 0x2000000000, fd = 0, u32 = 0, u64 = 137438953472}}, {events = 0, data = {ptr = 0x7fff391342c0, fd = 957563584, u32 = 957563584, u64 = 140734150951616}}, {events = 4251328, data = {ptr = 0x39134c8000000000, fd = 0, u32 = 0, u64 = 4112714997358919680}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x3da0953c00000000, fd = 0, u32 = 0, u64 = 4440713317517885440}}, {events = 32651, data = {ptr = 0x22a8120, fd = 36340000, u32 = 36340000, u64 = 36340000}}, {events = 957563984, data = {ptr = 0x23d940000007f00, fd = 32512, u32 = 32512, u64 = 161447889376149248}}, {events = 0, data = {ptr = 0x23d93a0, fd = 37589920, u32 = 37589920, u64 = 37589920}}, {events = 957563664, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff39134c80, fd = 957566080, u32 = 957566080, u64 = 140734150954112}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7f8b3da0953c <__GI___libc_free+76>, fd = 1033934140, u32 = 1033934140, u64 = 140236011115836}}, {events = 957566080, data = {ptr = 0xa932fc0000007fff, fd = 32767, u32 = 32767, u64 = 12192084218175586303}}, {events = 1685028597, data = {ptr = 0x7fff391342d0, fd = 957563600, u32 = 957563600, u64 = 140734150951632}}, {events = 6031467, data = {ptr = 0x391342f000000000, fd = 0, u32 = 0, u64 = 4112704483278979072}}, {events = 32767, data = {ptr = 0x23d93a0, fd = 37589920, u32 = 37589920, u64 = 37589920}}, {events = 957563664, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff39134c80, fd = 957566080, u32 = 957566080, u64 = 140734150954112}}, {events = 6067736, data = {ptr = 0x3913431000000000, fd = 0, u32 = 0, u64 = 4112704620717932544}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 31055520, data = {ptr = 0x204845000000000, fd = 0, u32 = 0, u64 = 145386567114948608}}, {events = 0, data = {ptr = 0x7fff39134340, fd = 957563712, u32 = 957563712, u64 = 140734150951744}}, {events = 6064579, data = {ptr = 0x1d9dec000000000, fd = 0, u32 = 0, u64 = 133382580199227392}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 31055520, data = {ptr = 0x204845000000000, fd = 0, u32 = 0, u64 = 145386567114948608}}, {events = 0, data = {ptr = 0x7fff39134360, fd = 957563744, u32 = 957563744, u64 = 140734150951776}}, {events = 6061325, data = {ptr = 0x1d9dea000000000, fd = 0, u32 = 0, u64 = 133382442760273920}}, {events = 0, data = {ptr = 0x23d93a0, fd = 37589920, u32 = 37589920, u64 = 37589920}}, {events = 957563904, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff39134c80, fd = 957566080, u32 = 957566080, u64 = 140734150954112}}, {events = 0, data = {ptr = 0x1ea094800000000, fd = 0, u32 = 0, u64 = 137932943430516736}}, {events = 0, data = {ptr = 0x1d30d40, fd = 30608704, u32 = 30608704, u64 = 30608704}}, {events = 957563840, data = {ptr = 0x7582fa00007fff, fd = 32767, u32 = 32767, u64 = 33076582528614399}}, {events = 0, data = {ptr = 0x7fff391345a0, fd = 957564320, u32 = 957564320, u64 = 140734150952352}}, {events = 957564320, data = {ptr = 0x3913464000007fff, fd = 32767, u32 = 32767, u64 = 4112708125411278847}}, {events = 32767, data = {ptr = 0x1ea08c0, fd = 32114880, u32 = 32114880, u64 = 32114880}}, {events = 957564096, data = {ptr = 0x7586d900007fff, fd = 32767, u32 = 32767, u64 = 33080838841204735}}, {events = 0, data = {ptr = 0x439134c80, fd = 957566080, u32 = 957566080, u64 = 18137435264}}, {events = 32114880, data = {ptr = 0x225c14800000000, fd = 0, u32 = 0, u64 = 154742277195956224}}, {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 1024, data = {ptr = 0x4800000000, fd = 0, u32 = 0, u64 = 309237645312}}, {events = 0, data = {ptr = 0x7fff391345f8, fd = 957564408, u32 = 957564408, u64 = 140734150952440}}, {events = 957564320, data = {ptr = 0x3913462000007fff, fd = 32767, u32 = 32767, u64 = 4112707987972325375}}, {events = 32767, data = {ptr = 0x225a56c, fd = 36021612, u32 = 36021612, u64 = 36021612}}, {events = 2565593683, data = {ptr = 0x7c39c1a4b98470ca, fd = -1182502710, u32 = 3112464586, u64 = 8951398647606833354}}, {events = 4093283211, data = {ptr = 0xfeef98333e459f96, fd = 1044750230, u32 = 1044750230, u64 = 18370068750923046806}}, {events = 267846134, data = {ptr = 0xb8fd94867ad13af8, fd = 2060532472, u32 = 2060532472, u64 = 13329973777393597176}}, {events = 455262307, data = {ptr = 0x225a510, fd = 36021520, u32 = 36021520, u64 = 36021520}}, {events = 0, data = {ptr = 0x1e7559000000000, fd = 0, u32 = 0, u64 = 137172390621741056}}, {events = 0, data = {ptr = 0x1e754f0, fd = 31937776, u32 = 31937776, u64 = 31937776}}, {events = 36926504, data = {ptr = 0x3913449000000000, fd = 0, u32 = 0, u64 = 4112706269985374208}}, {events = 4, data = {ptr = 0x2337418, fd = 36926488, u32 = 36926488, u64 = 36926488}}, {events = 36926504, data = {ptr = 0x113452800000000, fd = 0, u32 = 0, u64 = 77481656696438784}}, {events = 0, data = {ptr = 0x1134508 <RTC::EncryptBuffer+8>, fd = 18040072, u32 = 18040072, u64 = 18040072}}, {events = 2099355835, data = {ptr = 0x175fbef9bf4bba9f, fd = -1085556065, u32 = 3209411231, u64 = 1684274765525400223}}, {events = 2940461600, data = {ptr = 0x2408e6ae61b05d62, fd = 1638948194, u32 = 1638948194, u64 = 2596578821816737122}}, {events = 2324013938, data = {ptr = 0x1e7573c8c87ebb1, fd = -1937249359, u32 = 2357717937, u64 = 137174231225461681}}, {events = 0, data = {ptr = 0xc394475aaa130646, fd = -1441593786, u32 = 2853373510, u64 = 14092967588675126854}}, {events = 533523633, data = {ptr = 0x6d8bec4cc1435b9, fd = -871090759, u32 = 3423876537, u64 = 493353911643813305}}, {events = 2768510197, data = {ptr = 0xb74b1a6afdbbe1c8, fd = -38018616, u32 = 4256948680, u64 = 13207679379019915720}}, {events = 3258592114, data = {ptr = 0x1e756e0625de2ad, fd = 1650320045, u32 = 1650320045, u64 = 137173835381072557}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 31938272, data = {ptr = 0x3913463000000000, fd = 0, u32 = 0, u64 = 4112708056691769344}}...}
pe = 0x7fff39131990
e = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
real_timeout = 36
q = 0x1ea0960
w = 0x1ea0948
sigset = {__val = {0 <repeats 16 times>}}
psigset = 0x0
base = 9612754458
have_signals = 0
nevents = 0
count = 48
nfds = 1
fd = 35
op = 3
i = 0
__PRETTY_FUNCTION__ = "uv__io_poll"
#13 0x000000000074eb0d in uv_run (loop=0x1d30d40, mode=UV_RUN_DEFAULT) at ../deps/libuv/src/unix/core.c:373
timeout = 36
r = 1
ran_pending = 0
#14 0x00000000005b2300 in DepLibUV::RunLoop () at ../src/DepLibUV.cpp:52
__FUNCTION__ = "RunLoop"
#15 0x00000000005be519 in Worker::Worker (this=0x7fff39134af0, channel=0x1d31130) at ../src/Worker.cpp:31
No locals.
#16 0x000000000074676c in main (argc=10, argv=0x7fff39134c88) at ../src/main.cpp:123
worker = {<Channel::UnixStreamSocket::Listener> = {_vptr.Listener = 0xc9f858 <vtable for Worker+16>}, <SignalsHandler::Listener> = {_vptr.Listener = 0xc9f890 <vtable for Worker+72>}, channel = 0x1d31130, signalsHandler = 0x1d5e510, mapRouters = std::unordered_map with 2 elements = {["24bac81d-5344-47ec-8361-309f54f778da"] = 0x1e42ae0, ["1a14b1be-3555-4abf-b714-9c748b7c9c2e"] = 0x1d61c00}, closed = false}
__FUNCTION__ = "main"
version = "3.2.5"
channel = 0x1d31130
(gdb)
The last core dump (int11):
gdb /var/conference/server/node_modules/mediasoup-devel/worker/out/Release/mediasoup-worker /tmp/cores/core.mediasoup-worke.sig11.3748
GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/conference/server/node_modules/mediasoup-devel/worker/out/Release/mediasoup-worker...done.
warning: core file may not match specified executable file.
[New LWP 3748]
[New LWP 3750]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/conference/server/node_modules/mediasoup/worker/out/Release/mediasoup-work'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000001aa6740 in ?? ()
[Current thread is 1 (Thread 0x7fec340f0740 (LWP 3748))]
(gdb) bt full
#0 0x0000000001aa6740 in ?? ()
No symbol table info available.
#1 0x00000000005d6426 in std::function<void (bool)>::operator()(bool) const (this=0x7fff06e97d40, __args#0=true)
at /usr/include/c++/5/functional:2267
No locals.
#2 0x00000000005d6378 in TcpConnection::OnUvWrite(int, std::function<void (bool)> const&) (this=0x1fb8d90, status=0, onDone=...)
at ../src/handles/TcpConnection.cpp:487
No locals.
#3 0x00000000005d505e in onWrite (req=0x3718df0, status=0) at ../src/handles/TcpConnection.cpp:47
writeData = 0x3718df0
handle = 0x1fb0a50
connection = 0x1fb8d90
onDone = 0x7fff06e97d40
#4 0x0000000000758a22 in uv__write_callbacks (stream=0x1fb0a50) at ../deps/libuv/src/unix/stream.c:953
req = 0x3718df0
q = 0x3718e48
pq = {0x7fff06e982e0, 0x7fff06e982e0}
__PRETTY_FUNCTION__ = "uv__write_callbacks"
#5 0x000000000075977a in uv__stream_io (loop=0x19a6d40, w=0x1fb0ad8, events=4) at ../deps/libuv/src/unix/stream.c:1329
stream = 0x1fb0a50
__PRETTY_FUNCTION__ = "uv__stream_io"
#6 0x000000000075eef5 in uv__io_poll (loop=0x19a6d40, timeout=13) at ../deps/libuv/src/unix/linux-core.c:384
max_safe_timeout = 1789569
events = {{events = 4, data = {ptr = 0x50, fd = 80, u32 = 80, u64 = 80}}, {events = 1, data = {ptr = 0x10, fd = 16,
u32 = 16, u64 = 16}}, {events = 1, data = {ptr = 0x3e, fd = 62, u32 = 62, u64 = 62}}, {events = 1, data = {
ptr = 0x1a, fd = 26, u32 = 26, u64 = 26}}, {events = 1, data = {ptr = 0x4c, fd = 76, u32 = 76, u64 = 76}}, {
events = 1, data = {ptr = 0x18, fd = 24, u32 = 24, u64 = 24}}, {events = 1, data = {ptr = 0x28, fd = 40, u32 = 40,
u64 = 40}}, {events = 1, data = {ptr = 0x1a, fd = 26, u32 = 26, u64 = 26}}, {events = 0, data = {ptr = 0x0, fd = 0,
u32 = 0, u64 = 0}} <repeats 666 times>, {events = 27032624, data = {ptr = 0x19c88a800000000, fd = 0, u32 = 0,
u64 = 116117945540673536}}, {events = 0, data = {ptr = 0x499b75 <BN_CTX_get+85>, fd = 4823925, u32 = 4823925,
u64 = 4823925}}, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {
ptr = 0x8, fd = 8, u32 = 8, u64 = 8}}, {events = 27035792, data = {ptr = 0x19c7c3000000000, fd = 0, u32 = 0,
u64 = 116104236005064704}}, {events = 0, data = {ptr = 0x4a9da7 <bn_sqr_fixed_top+103>, fd = 4890023,
u32 = 4890023, u64 = 4890023}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 16, data = {
ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0,
u64 = 0}} <repeats 14 times>, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {
events = 0, data = {ptr = 0x19c8890, fd = 27035792, u32 = 27035792, u64 = 27035792}}, {events = 27036792, data = {
ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {
ptr = 0x4a338f <bn_from_montgomery_word+735>, fd = 4862863, u32 = 4862863, u64 = 4862863}}, {events = 4294967295,
data = {ptr = 0x19c8c80ffffffff, fd = -1, u32 = 4294967295, u64 = 116122176083460095}}, {events = 0, data = {
ptr = 0x19c8830, fd = 27035696, u32 = 27035696, u64 = 27035696}}, {events = 8, data = {ptr = 0x19c889000000000,
fd = 0, u32 = 0, u64 = 116117842461458432}}, {events = 0, data = {ptr = 0x19c7c30, fd = 27032624, u32 = 27032624,
u64 = 27032624}}, {events = 27035792, data = {ptr = 0x6e9af9000000000, fd = 0, u32 = 0, u64 = 498122266811301888}},
{events = 32767, data = {ptr = 0x19c8830, fd = 27035696, u32 = 27035696, u64 = 27035696}}, {events = 27037168, data = {
ptr = 0x6e9af9000000000, fd = 0, u32 = 0, u64 = 498122266811301888}}, {events = 32767, data = {
ptr = 0x4a3684 <bn_from_mont_fixed_top+84>, fd = 4863620, u32 = 4863620, u64 = 4863620}}, {events = 27035696,
--Type <RET> for more, q to quit, c to continue without paging--c
data = {ptr = 0x6e9b00000000000, fd = 0, u32 = 0, u64 = 498122747847639040}}, {events = 32767, data = {ptr = 0x7fff06e9af90, fd = 115978128, u32 = 115978128, u64 = 140733309366160}}, {events = 27032624, data = {ptr = 0x6e9afb000000000, fd = 0, u32 = 0, u64 = 498122404250255360}}, {events = 32767, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 115978240, data = {ptr = 0x6e9af9000007fff, fd = 32767, u32 = 32767, u64 = 498122266811334655}}, {events = 32767, data = {ptr = 0x19c7c30, fd = 27032624, u32 = 27032624, u64 = 27032624}}, {events = 4828985, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x33ee2e1400000000, fd = 0, u32 = 0, u64 = 3741979003825684480}}, {events = 32748, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 76 times>, {events = 0, data = {ptr = 0x6e9af5000000000, fd = 0, u32 = 0, u64 = 498121991933394944}}, {events = 32767, data = {ptr = 0x7fff06e9af40, fd = 115978048, u32 = 115978048, u64 = 140733309366080}}, {events = 9127685, data = {ptr = 0x6e9b07800000000, fd = 0, u32 = 0, u64 = 498123263243714560}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 851638573, data = {ptr = 0x7fec, fd = 32748, u32 = 32748, u64 = 32748}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 117, data = {ptr = 0x6e9aab800000000, fd = 0, u32 = 0, u64 = 498116941051854848}}, {events = 32767, data = {ptr = 0x7fff06e9aaa8, fd = 115976872, u32 = 115976872, u64 = 140733309364904}}, {events = 0, data = {ptr = 0x7fec, fd = 32748, u32 = 32748, u64 = 32748}}, {events = 0, data = {ptr = 0x7fff06e9aac0, fd = 115976896, u32 = 115976896, u64 = 140733309364928}}, {events = 0, data = {ptr = 0x336f2d803ffa4000, fd = 1073364992, u32 = 1073364992, u64 = 3706231047201636352}}, {events = 32748, data = {ptr = 0x7fff06e9afd0, fd = 115978192, u32 = 115978192, u64 = 140733309366224}}, {events = 115978176, data = {ptr = 0x8b470500007fff, fd = 32767, u32 = 32767, u64 = 39203108563222527}}, {events = 0, data = {ptr = 0x7fff06e9b0f8, fd = 115978488, u32 = 115978488, u64 = 140733309366520}}, {events = 0, data = {ptr = 0x32c2f92d00000000, fd = 0, u32 = 0, u64 = 3657759819047108608}}, {events = 32748, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x6e9af7500000000, fd = 0, u32 = 0, u64 = 498122150847184896}}, {events = 0, data = {ptr = 0x7fff06e9ab38, fd = 115977016, u32 = 115977016, u64 = 140733309365048}}, {events = 115977000, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 1, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 115977024, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 1073397760, data = {ptr = 0xb00000000, fd = 0, u32 = 0, u64 = 47244640256}}, {events = 0, data = {ptr = 0x8b472000000000, fd = 0, u32 = 0, u64 = 39203224527306752}}, {events = 0, data = {ptr = 0x7fff00000000, fd = 0, u32 = 0, u64 = 140733193388032}}, {events = 0, data = {ptr = 0x6e9ab503ffac000, fd = 1073397760, u32 = 1073397760, u64 = 498117594960281600}}, {events = 32767, data = {ptr = 0x7fff00000000, fd = 0, u32 = 0, u64 = 140733193388032}}, {events = 10, data = {ptr = 0x300000000, fd = 0, u32 = 0, u64 = 12884901888}}, {events = 0, data = {ptr = 0x7fff06e9af85, fd = 115978117, u32 = 115978117, u64 = 140733309366149}}, {events = 3, data = {ptr = 0x8b470800000000, fd = 0, u32 = 0, u64 = 39203121448091648}}, {events = 0, data = {ptr = 0x100000000, fd = 0, u32 = 0, u64 = 4294967296}}, {events = 115977120, data = {ptr = 0x7fff, fd = 32767, u32 = 32767, u64 = 32767}}, {events = 0, data = {ptr = 0xffffffffffffffff, fd = -1, u32 = 4294967295, u64 = 18446744073709551615}}, {events = 0, data = {ptr = 0x6e9b2000000000b, fd = 11, u32 = 11, u64 = 498124946870894603}}, {events = 32767, data = {ptr = 0x8b4709, fd = 9127689, u32 = 9127689, u64 = 9127689}}, {events = 27, data = {ptr = 0x6e9b21000000000, fd = 0, u32 = 0, u64 = 498125015590371328}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 864190991, data = {ptr = 0x6e9af3000007fec, fd = 32748, u32 = 32748, u64 = 498121854494474220}}, {events = 32767, data = {ptr = 0x1b, fd = 27, u32 = 27, u64 = 27}}, {events = 9, data = {ptr = 0x1000000000, fd = 0, u32 = 0, u64 = 68719476736}}, {events = 48, data = {ptr = 0x7fff06e9b1d0, fd = 115978704, u32 = 115978704, u64 = 140733309366736}}, {events = 115978512, data = {ptr = 0x6e9abf000007fff, fd = 32767, u32 = 32767, u64 = 498118281081683967}}, {events = 0, data = {ptr = 0x7fff06e9b200, fd = 115978752, u32 = 115978752, u64 = 140733309366784}}, {events = 4178990113, data = {ptr = 0x6e9abdfffff8000, fd = -32768, u32 = 4294934528, u64 = 498118212362141696}}, {events = 32767, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 9, data = {ptr = 0x32, fd = 50, u32 = 50, u64 = 50}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 91, data = {ptr = 0x6e, fd = 110, u32 = 110, u64 = 110}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 119, data = {ptr = 0x6e9af300000007c, fd = 124, u32 = 124, u64 = 498121854494441596}}, {events = 32767, data = {ptr = 0x7fff06e9b200, fd = 115978752, u32 = 115978752, u64 = 140733309366784}}, {events = 57, data = {ptr = 0x6e9b21000000000, fd = 0, u32 = 0, u64 = 498125015590371328}}, {events = 32767, data = {ptr = 0x100, fd = 256, u32 = 256, u64 = 256}}, {events = 9150464, data = {ptr = 0x6e9ac3000000000, fd = 0, u32 = 0, u64 = 498118555959558144}}, {events = 32767, data = {ptr = 0x5ec72f <std::array<unsigned char, 400ul>::operator[](unsigned long) const+35>, fd = 6211375, u32 = 6211375, u64 = 6211375}}, {events = 256, data = {ptr = 0x3382977c00000000, fd = 0, u32 = 0, u64 = 3711695601738448896}}, {events = 32748, data = {ptr = 0x7fff06e9ac70, fd = 115977328, u32 = 115977328, u64 = 140733309365360}}, {events = 30012320, data = {ptr = 0x5faeb600000000, fd = 0, u32 = 0, u64 = 26932219494793216}}, {events = 0, data = {ptr = 0x5faee5 <nlohmann::detail::output_string_adapter<char, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::write_characters(char const*, unsigned long)+47>, fd = 6270693, u32 = 6270693, u64 = 6270693}}, {events = 115977328, data = {ptr = 0x2400007fff, fd = 32767, u32 = 32767, u64 = 154618855423}}, {events = 0, data = {ptr = 0x7fff06e9af7a, fd = 115978106, u32 = 115978106, u64 = 140733309366138}}, {events = 30012320, data = {ptr = 0x6e9ad9000000000, fd = 0, u32 = 0, u64 = 498120067788046336}}, {events = 32767, data = {ptr = 0x5e7afc <nlohmann::detail::serializer<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> >::dump_escaped(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+2680>, fd = 6191868, u32 = 6191868, u64 = 6191868}}, {events = 120, data = {ptr = 0xabbead0000000000, fd = 0, u32 = 0, u64 = 12375519041572306944}}, {events = 2479770368, data = {ptr = 0x1c86440, fd = 29910080, u32 = 29910080, u64 = 29910080}}, {events = 115978016, data = {ptr = 0x6e9ad1000007fff, fd = 32767, u32 = 32767, u64 = 498119518032265215}}, {events = 32767, data = {ptr = 0x6565e90000, fd = 1709768704, u32 = 1709768704, u64 = 435501465600}}, {events = 36, data = {ptr = 0x2400000000, fd = 0, u32 = 0, u64 = 154618822656}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 36, data = {ptr = 0x6e9ad1000000000, fd = 0, u32 = 0, u64 = 498119518032232448}}, {events = 0, data = {ptr = 0x1c9f3a0, fd = 30012320, u32 = 30012320, u64 = 30012320}}, {events = 6270646, data = {ptr = 0x2000000000, fd = 0, u32 = 0, u64 = 137438953472}}, {events = 0, data = {ptr = 0x7fff06e9ad90, fd = 115977616, u32 = 115977616, u64 = 140733309365648}}, {events = 4251328, data = {ptr = 0x6e9b75000000000, fd = 0, u32 = 0, u64 = 498130788026417152}}, {events = 32767, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x32c6653c00000000, fd = 0, u32 = 0, u64 = 3658723055657549824}}, {events = 32748, data = {ptr = 0x1c1ad50, fd = 29470032, u32 = 29470032, u64 = 29470032}}, {events = 115978016, data = {ptr = 0x1c553b000007f00, fd = 32512, u32 = 32512, u64 = 127600179829309184}}, {events = 0, data = {ptr = 0x1c55530, fd = 29709616, u32 = 29709616, u64 = 29709616}}, {events = 115977696, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff06e9b750, fd = 115980112, u32 = 115980112, u64 = 140733309368144}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7fec32c6653c <__GI___libc_free+76>, fd = 851862844, u32 = 851862844, u64 = 140652440872252}}, {events = 115980112, data = {ptr = 0xabbead0000007fff, fd = 32767, u32 = 32767, u64 = 12375519041572339711}}, {events = 2479770509, data = {ptr = 0x7fff06e9ada0, fd = 115977632, u32 = 115977632, u64 = 140733309365664}}, {events = 6031467, data = {ptr = 0x6e9adc000000000, fd = 0, u32 = 0, u64 = 498120273946476544}}, {events = 32767, data = {ptr = 0x1c55530, fd = 29709616, u32 = 29709616, u64 = 29709616}}, {events = 115977696, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff06e9b750, fd = 115980112, u32 = 115980112, u64 = 140733309368144}}, {events = 6067736, data = {ptr = 0x6e9ade000000000, fd = 0, u32 = 0, u64 = 498120411385430016}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 29709488, data = {ptr = 0x1bd6cf000000000, fd = 0, u32 = 0, u64 = 125376142684192768}}, {events = 0, data = {ptr = 0x7fff06e9ae10, fd = 115977744, u32 = 115977744, u64 = 140733309365776}}, {events = 6064579, data = {ptr = 0x1c554d000000000, fd = 0, u32 = 0, u64 = 127601416779857920}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 29709488, data = {ptr = 0x1bd6cf000000000, fd = 0, u32 = 0, u64 = 125376142684192768}}, {events = 0, data = {ptr = 0x7fff06e9ae30, fd = 115977776, u32 = 115977776, u64 = 140733309365808}}, {events = 6061325, data = {ptr = 0x1c554b000000000, fd = 0, u32 = 0, u64 = 127601279340904448}}, {events = 0, data = {ptr = 0x1c55530, fd = 29709616, u32 = 29709616, u64 = 29709616}}, {events = 115977936, data = {ptr = 0x40dec000007fff, fd = 32767, u32 = 32767, u64 = 18259314724601855}}, {events = 0, data = {ptr = 0x7fff06e9b750, fd = 115980112, u32 = 115980112, u64 = 140733309368144}}, {events = 0, data = {ptr = 0x1f862a800000000, fd = 0, u32 = 0, u64 = 141971861956198400}}, {events = 0, data = {ptr = 0x19a6d40, fd = 26897728, u32 = 26897728, u64 = 26897728}}, {events = 115977872, data = {ptr = 0x7582fa00007fff, fd = 32767, u32 = 32767, u64 = 33076582528614399}}, {events = 0, data = {ptr = 0x7fff06e9b070, fd = 115978352, u32 = 115978352, u64 = 140733309366384}}, {events = 115978352, data = {ptr = 0x6e9b11000007fff, fd = 32767, u32 = 32767, u64 = 498123916078776319}}, {events = 32767, data = {ptr = 0x1f86220, fd = 33055264, u32 = 33055264, u64 = 33055264}}, {events = 115978128, data = {ptr = 0x7586d900007fff, fd = 32767, u32 = 32767, u64 = 33080838841204735}}, {events = 0, data = {ptr = 0x406e9b750, fd = 115980112, u32 = 115980112, u64 = 17295849296}}, {events = 33055264, data = {ptr = 0x1e8070800000000, fd = 0, u32 = 0, u64 = 137367519575932928}}, {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 1024, data = {ptr = 0x4800000000, fd = 0, u32 = 0, u64 = 309237645312}}, {events = 0, data = {ptr = 0x7fff06e9b0c8, fd = 115978440, u32 = 115978440, u64 = 140733309366472}}, {events = 115978352, data = {ptr = 0x6e9b0f000007fff, fd = 32767, u32 = 32767, u64 = 498123778639822847}}, {events = 32767, data = {ptr = 0x1a6d470, fd = 27710576, u32 = 27710576, u64 = 27710576}}, {events = 27710352, data = {ptr = 0x1e8091000000000, fd = 0, u32 = 0, u64 = 137369752958926848}}, {events = 0, data = {ptr = 0x1e80870, fd = 31983728, u32 = 31983728, u64 = 31983728}}, {events = 31983368, data = {ptr = 0x46026000000000, fd = 0, u32 = 0, u64 = 19705859709861888}}, {events = 4, data = {ptr = 0x1e806f8, fd = 31983352, u32 = 31983352, u64 = 31983352}}, {events = 31983368, data = {ptr = 0x1fb845000000000, fd = 0, u32 = 0, u64 = 142853292324552704}}, {events = 0, data = {ptr = 0x1fb83b0, fd = 33260464, u32 = 33260464, u64 = 33260464}}, {events = 30114312, data = {ptr = 0xab86f97500000000, fd = 0, u32 = 0, u64 = 12359840508271394816}}, {events = 4, data = {ptr = 0x1cb81f8, fd = 30114296, u32 = 30114296, u64 = 30114296}}, {events = 30114312, data = {ptr = 0x113452800000000, fd = 0, u32 = 0, u64 = 77481656696438784}}, {events = 0, data = {ptr = 0x1134508 <RTC::EncryptBuffer+8>, fd = 18040072, u32 = 18040072, u64 = 18040072}}, {events = 2756901046, data = {ptr = 0x215dca574084024f, fd = 1082393167, u32 = 1082393167, u64 = 2404300252179071567}}, {events = 2716885968, data = {ptr = 0x320f134c504413a3, fd = 1346638755, u32 = 1346638755, u64 = 3607123045032137635}}, {events = 897455979, data = {ptr = 0x1fb806cd461e05e, fd = -731783074, u32 = 3563184222, u64 = 142849018100310110}}, {events = 0, data = {ptr = 0x360fdb2285279025, fd = -2061004763, u32 = 2233962533, u64 = 3895573144008101925}}, {events = 810881864, data = {ptr = 0xff948617d351aee2, fd = -749621534, u32 = 3545345762, u64 = 18416492213112516322}}, {events = 3377328891, data = {ptr = 0x26938fd832b8061e, fd = 850920990, u32 = 850920990, u64 = 2779723553744356894}}, {events = 2463481837, data = {ptr = 0x1fb801076ccd3d9, fd = 1993135065, u32 = 1993135065, u64 = 142848621393269721}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}...}
pe = 0x7fff06e98460
e = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
real_timeout = 13
q = 0x1f862c0
w = 0x1fb0ad8
sigset = {__val = {0 <repeats 16 times>}}
psigset = 0x0
base = 9639759052
have_signals = 0
nevents = 0
count = 48
nfds = 1
fd = 80
op = 3
i = 0
__PRETTY_FUNCTION__ = "uv__io_poll"
#7 0x000000000074eb0d in uv_run (loop=0x19a6d40, mode=UV_RUN_DEFAULT) at ../deps/libuv/src/unix/core.c:373
timeout = 13
r = 1
ran_pending = 0
#8 0x00000000005b2300 in DepLibUV::RunLoop () at ../src/DepLibUV.cpp:52
__FUNCTION__ = "RunLoop"
#9 0x00000000005be519 in Worker::Worker (this=0x7fff06e9b5c0, channel=0x19a7130) at ../src/Worker.cpp:31
No locals.
#10 0x000000000074676c in main (argc=10, argv=0x7fff06e9b758) at ../src/main.cpp:123
worker = {<Channel::UnixStreamSocket::Listener> = {_vptr.Listener = 0xc9f858 <vtable for Worker+16>}, <SignalsHandler::Listener> = {_vptr.Listener = 0xc9f890 <vtable for Worker+72>}, channel = 0x19a7130, signalsHandler = 0x19d4510, mapRouters = std::unordered_map with 11 elements = {["84f59105-1ef0-45b1-8947-eb41dde102a4"] = 0x1d960d0, ["c88a9731-222e-4007-ae5f-6c58a2e806cd"] = 0x1cbcff0, ["aea4553f-fe73-4079-88a4-1a752694e392"] = 0x1cc5210, ["93e0ec06-decd-403a-a317-67e3ee31072b"] = 0x1a60e20, ["f1fb31a7-5e98-47fe-a6c2-1a1888ece408"] = 0x1d95e00, ["3e767c68-483f-4a98-a633-c54652ca35df"] = 0x19d7c00, ["f603a808-269e-4cc3-bc00-37478181bdf7"] = 0x1a7dc50, ["ce58622c-64f7-4a32-b25f-714bdb44e261"] = 0x1a62120, ["95dcd893-767c-4754-bdb4-ccda90eba3f1"] = 0x1bec500, ["9c00ed7a-b073-4c09-a7ec-360b9f8e218c"] = 0x1cca390, ["362d8836-c211-4663-8db5-2c44460a5980"] = 0x1a7cdf0}, closed = false}
__FUNCTION__ = "main"
version = "3.2.5"
channel = 0x19a7130
(gdb)
Thanks. So all those crashes hapen in TcpConnection.cpp. So nothing has changed.
But much more crashes compared to the master branch.
Right, but the reason is the same. It just that in “devel” branch we do more things and, if a deallocated TcpConnection is attempted to be used, there are more places in which it can crash.
BTW:
- Which OS and compiler are you using?
- I’m trying again and again to reproduce the TCP crash using the mediasoup-demo with no luck. I’ve tried everything. Maybe you can install the mediasoup-demo in your server (or a similar server) and let me test it?
Ubuntu 16.04.6 LTS with 16 core Intel(R) Xeon(R) CPU E5620 @ 2.40GHz and 128G RAM
$ dpkg --list | grep compiler
ii g++ 4:5.3.1-1ubuntu1 amd64 GNU C++ compiler
ii g++-5 5.5.0-12ubuntu1~16.04 amd64 GNU C++ compiler
ii gcc 4:5.3.1-1ubuntu1 amd64 GNU C compiler
ii gcc-5 5.5.0-12ubuntu1~16.04 amd64 GNU C compiler
I’m open for any cooperation but I think the key with our server is the amount of connections/streams. I think we need a tool to do heavy loading tests. Trying to debug on production server is not wise.
Are you properly increasing the “max open files” settings and so on? mediasoup should reject createTransport() with an error if it cannot open more UDP or TCP ports, so that should not be the reason of your crashes. But just in case…
Is this enough?
$ ulimit -n
64000
I’ve also opened UDP/TCP port range 10000:12000 in the firewall.
I assume that you are setting 10000 and 12000 as min and max port in Workers settings. Questions here:
- How many mediasoup Workers are you launching?
- Are you passing the very same port range (10000-12000) to all them?
Regarding the limits, honestly I cannot provide exact details. But in addition to what you did I also mean UDP and TCP kernel buffers:
- https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html
- https://www.cyberciti.biz/faq/linux-tcp-tuning/
I also do this:
/etc/security/limits.d# cat mediasoup.conf
* soft nofile 1000000
* hard nofile 1000000
As number of CPUs (16 workers here)
Yes exactly the same config/port range.
No, I’ve done nothing with buffers! I’ll check that links.
And here’s my limits.conf:
* soft nofile 64000
* hard nofile 64000
root soft nofile 64000
root hard nofile 64000
I can increase the limit but the number of streams on the server has never exceeded than 1000 so even less for number of connections.
The only think coming to my mind is that you are sharing the port range for all workers. It’s supposed to work fine even if the same port range is given to many workers, but may be there is a bug there and when two workers try to bind on the same TCP port at the same time there is conflict.
So, may be you could modify your app to distribute the port range? I would expand the total por range. 10000-12000 just allows 1000 peers in total, and having 16 workers it means 1000/16 = 62 peers per worker. Let’s assume you increase it to 10000-40000, so 30000 ports in total. Then each worker must have 30000/16 = 1875 ports, so:
- worker1: 10000-11874
- worker2: 11875-13750
- worker3: 13750-25625
and so on.
Of course, to see if this is the reason of the crash you must enable again the commented code in DtlsTransport.cpp destructor.
Since rooms are assigned to workers one after the other without any rule, there is a chance that some big rooms are assigned to the same worker. So your solution is a little risky by limiting the worker to 1/16 of available ports unless I could properly distribute and balance users between workers.
This can be easily tested by sharing a few ports between some workers and see if they could handle it.
Not sure if I was clear, but currently you can handle 62 peers per worker, so assuming a worker just handles a single room such a room can just hold 62 peers.
UPDATE: No, I am wrong obviously. Such a worker can use up to 2000 ports if there are no other workers running active rooms. Ok, let’s focus on the crash.
And I’ve done it and stressed it as much as I could (in my Mac) and was not able to reproduce any issue. I’ll repeat the same test in Linux next week.
I can reproduce the crash in Linux. Let’s please continue in GitHub:
@mkh have you ever seen this coredump in your server?
Your core dump is completely in different format! I’ve shared all the core dumps of master branch and two of devel branch here. I can check the others if it helps.
It’s OSX