It’s basically the same. So it seems to be related to a TcpConnection being used after deallocated. May I know a bit about your scenario? Also, can you reproduce it easily in localhost with just a few peers?
It has never happened in my localhost and I still don’t know in what scenario the crash is happening. I just found a suspicious error in the logs right before the crash:
mediasoup:ERROR:worker[pid:6013] (stderr) mediasoup-worker: ../deps/libuv/src/unix/stream.c:1400: uv_write2: Assertion `(stream->type == UV_TCP || stream->type == UV_NAMED_PIPE || stream->type == UV_TTY) && "uv_write (unix) does not yet support other types of streams"' failed.
mediasoup:Channel[pid:6013] Channel ended by the worker process
mediasoup:Worker close()
mediasoup:Channel[pid:6013] close()
mediasoup:Router workerClosed()
mediasoup:Channel[pid:6013] close()
...
mediasoup:ERROR:Worker worker process died unexpectedly [pid:6013, code:null, signal:SIGABRT]
Are you forcing RTP over TCP? I mean, when you test in localhost or local network, are you also forcing RTP over TCP? or does it just happen because your users in prod cannot use UDP due to their networking constraints?
Nope, both TCP and UDP are allowed and preferUdp is set to true, the same goes for my local config.
But as I mentioned Safari on iOS does not work with UDP.
All crashes above are about TCP, so can you test locally by forcing TCP in createWebRtcTransport() please? To be perfectly clear, if you connect via UDP it won’t crash.
I tested also in my local VM having TCP forced with no crash. I keep testing more and I hope it happens. But how is that supposed to help to find the issue? 
It’s just that in localhost it may be easier to connect 2 peers using RTP over TCP, disconnect & reconnect, and get a way to reproduce the issue.
For instance, when do you call transport.close() in server side? May it happen that you call router.close() or worker.close() before calling transport.close()? (note: that’s perfectly fine, just trying to figure out the exact steps that produce the problem).
Since the crash does not happen regularly (once a week or more) it’s probably hard to reproduce on localhost. I may be better to look for a pattern in the logs. Or add some additional log info into the possible related codes?
It’s not about codecs, it’s about TCP usage for sure. Please try to reply these questions:
-
When do you call
transport.close()in server side? -
May it happen that you call
router.close()orworker.close()before callingtransport.close()? (note: that’s perfectly fine, just trying to figure out the exact steps that produce the problem)
- When user stops mic/webcam/screen which closes related client and server transports accordingly.
- When user leaves the room and all related transports are closed.
Yes when a room is closed then I just close the router, not the transports since they will close on router close. I never close a worker.
Normal usage then, thanks.
BTW I’m afraid I’ve been testing the mediasoup-demo with `&forceTcp=true’ for 2 hours in both local and server environments and I cannot get any crash. I’ve also checked the whole TCP related C++ code in mediasoup-worker and I cannot spot anything suspicious, everything looks correct. Will try to do more research.
I’m thinking of a way to store connections stats on disk or database for later analysis and tracking issues. But that would be a huge amount of data while most of it is repetitious and useless. Do you have any idea on how to store only the useful part, for example the stats of connection changes?
No idea, sorry. However you could listen for the transport state change events and call transport.getStats() (and also transport.dump()) and store the returned async objects/arrays into some storage.
Please try this. Remove these lines:
- https://github.com/versatica/mediasoup/blob/v3/worker/src/RTC/DtlsTransport.cpp#L598
- https://github.com/versatica/mediasoup/blob/v3/worker/src/RTC/DtlsTransport.cpp#L607
- https://github.com/versatica/mediasoup/blob/v3/worker/src/RTC/DtlsTransport.cpp#L608
Those lines may produce a double free() if something goes wrong, since goto error goes to section in which those objects are also freed.
Also please add the following lines after this one:
MS_DUMP("--- 1 DtlsTransport::Reset() ---");
MS_ERROR("--- 2 DtlsTransport::Reset() ---");
MS_ERROR_STD("--- 3 DtlsTransport::Reset() ---");
Compile again and let me know if this fixes anything.
Consider it done!
The above changes are not gonna fix the issue, I think. Not 100% sure. BTW also enable “dtls” in worker’s logTags, and ensure logLevel is “warn” (or “debug”, which is too much). You can change those worker logging settings via API in runtime.
I applied the changes and there is a new core dump after that:
$ gdb /var/conference/server/node_modules/mediasoup/worker/out/Release/mediasoup-worker /tmp/cores/core.mediasoup-worke.sig11.17833
GNU gdb (Ubuntu 8.2-0ubuntu1~16.04.1) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/conference/server/node_modules/mediasoup/worker/out/Release/mediasoup-worker...done.
warning: core file may not match specified executable file.
[New LWP 17833]
[New LWP 17838]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/conference/server/node_modules/mediasoup/worker/out/Release/mediasoup-work'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000007507f9 in uv_try_write (stream=0x0, bufs=0x7ffe80c9cc80, nbufs=2) at ../deps/libuv/src/unix/stream.c:1512
1512 if (stream->connect_req != NULL || stream->write_queue_size != 0)
[Current thread is 1 (Thread 0x7f341d6af740 (LWP 17833))]
(gdb) bt full
#0 0x00000000007507f9 in uv_try_write (stream=0x0, bufs=0x7ffe80c9cc80, nbufs=2) at ../deps/libuv/src/unix/stream.c:1512
r = 0
has_pollout = 0
written = 4781986
req_size = 31
req = {data = 0x7ffe80ca07a0, type = 2160708496, reserved = {0x1f, 0x1f, 0x24436e0, 0x7ffe80c9cb90, 0x0,
0x48929a <bwrite_conv+26>}, cb = 0x4935200, send_handle = 0x48836b <BIO_write+171>, handle = 0x1f, queue = {
0x8b3b8108091d2100, 0x5afb4b0}, write_index = 95401136, bufs = 0x7f341d6af6b8, nbufs = 2160708856, error = 32766, bufsml = {{
base = 0x5afac50 "\375\376", len = 4266021}, {
base = 0x1000000000001 <error: Cannot access memory at address 0x1000000000001>, len = 31}, {
base = 0x7ffe80c9cc00 "\220\201\370\004", len = 2}, {
base = 0x1f00000015 <error: Cannot access memory at address 0x1f00000015>, len = 83394960}}}
__PRETTY_FUNCTION__ = "uv_try_write"
#1 0x00000000005d3b89 in TcpConnection::Write (this=0x2c92180, data1=0x7ffe80c9cce6 "", len1=2,
data2=0x4f88190 "\025\376", <incomplete sequence \375>, len2=31) at ../src/handles/TcpConnection.cpp:271
totalLen = 33
buffers = {{base = 0x7ffe80c9cce6 "", len = 2}, {base = 0x4f88190 "\025\376", <incomplete sequence \375>, len = 31}}
written = 0
err = 21
pendingLen = 140731059129600
writeData = 0x611096 <RTC::DtlsTransport::OnSslInfo(int, int)+290>
buffer = {base = 0x400800000100 <error: Cannot access memory at address 0x400800000100>, len = 95397088}
__FUNCTION__ = "Write"
#2 0x000000000069d5e1 in RTC::TcpConnection::Send (this=0x2c92180, data=0x4f88190 "\025\376", <incomplete sequence \375>, len=31)
at ../src/RTC/TcpConnection.cpp:169
frameLen = "\000\037"
#3 0x000000000062df82 in RTC::TransportTuple::Send (this=0x47c0f80, data=0x4f88190 "\025\376", <incomplete sequence \375>, len=31)
at ../include/RTC/TransportTuple.hpp:113
No locals.
#4 0x00000000006b934a in RTC::WebRtcTransport::OnDtlsTransportSendData (this=0x2a2cc90,
data=0x4f88190 "\025\376", <incomplete sequence \375>, len=31) at ../src/RTC/WebRtcTransport.cpp:1638
__FUNCTION__ = "OnDtlsTransportSendData"
#5 0x000000000060fbb7 in RTC::DtlsTransport::SendPendingOutgoingDtlsData (this=0x5afa4e0) at ../src/RTC/DtlsTransport.cpp:1012
read = 31
data = 0x4f88190 "\025\376", <incomplete sequence \375>
#6 0x000000000060d7f5 in RTC::DtlsTransport::~DtlsTransport (this=0x5afa4e0, __in_chrg=<optimized out>)
at ../src/RTC/DtlsTransport.cpp:649
No locals.
#7 0x000000000060d8b4 in RTC::DtlsTransport::~DtlsTransport (this=0x5afa4e0, __in_chrg=<optimized out>)
at ../src/RTC/DtlsTransport.cpp:663
No locals.
#8 0x00000000006afc80 in RTC::WebRtcTransport::~WebRtcTransport (this=0x2a2cc90, __in_chrg=<optimized out>)
at ../src/RTC/WebRtcTransport.cpp:266
kv = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--c
__for_end = <optimized out>
udpSocket = <optimized out>
kv = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
tcpServer = <optimized out>
#9 0x00000000006aff38 in RTC::WebRtcTransport::~WebRtcTransport (this=0x2a2cc90, __in_chrg=<optimized out>) at ../src/RTC/WebRtcTransport.cpp:294
kv = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
udpSocket = <optimized out>
kv = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
tcpServer = <optimized out>
#10 0x000000000064d340 in RTC::Router::HandleRequest (this=0x2a62490, request=0x1f2b2b0) at ../src/RTC/Router.cpp:283
transport = 0x2a2cc90
#11 0x00000000005be6c5 in Worker::OnChannelRequest (this=0x7ffe80ca0610, request=0x1f2b2b0) at ../src/Worker.cpp:188
router = 0x2a62490
#12 0x00000000005e0321 in Channel::UnixStreamSocket::UserOnUnixStreamRead (this=0x1abaff0) at ../src/Channel/UnixStreamSocket.cpp:265
jsonRequest = {m_type = nlohmann::detail::value_t::object, m_value = {object = 0x20b4940, array = 0x20b4940, string = 0x20b4940, boolean = 64, number_integer = 34294080, number_unsigned = 34294080, number_float = 1.6943526783731376e-316}}
request = 0x1f2b2b0
readLen = 159
jsonStart = 0x7f341b7c6d28 "{\"id\":4033,\"method\":\"transport.close\",\"internal\":{\"routerId\":\"ba3cf5b6-e05d-49f3-ad15-ba9978500ddc\",\"transportId\":\"2dfdf99d-6457-4a04-9b81-ce828ccb9544\"}},"
jsonLen = 154
nsRet = 0
__FUNCTION__ = "UserOnUnixStreamRead"
#13 0x00000000005d8cf4 in UnixStreamSocket::OnUvRead (this=0x1abaff0, nread=159) at ../src/handles/UnixStreamSocket.cpp:265
__FUNCTION__ = "OnUvRead"
#14 0x00000000005d82cb in onRead (handle=0x1abb040, nread=159, buf=0x7ffe80c9d1d0) at ../src/handles/UnixStreamSocket.cpp:35
socket = 0x1abaff0
#15 0x000000000074fe7c in uv__read (stream=0x1abb040) at ../deps/libuv/src/unix/stream.c:1234
buflen = 1934069
buf = {base = 0x7f341b7c6d24 "154:{\"id\":4033,\"method\":\"transport.close\",\"internal\":{\"routerId\":\"ba3cf5b6-e05d-49f3-ad15-ba9978500ddc\",\"transportId\":\"2dfdf99d-6457-4a04-9b81-ce828ccb9544\"}},", len = 1934069}
nread = 159
msg = {msg_name = 0x7ffe80c9d250, msg_namelen = 2160710272, msg_iov = 0x7ffe80c9d240, msg_iovlen = 6124928, msg_control = 0x7ffe80c9d240, msg_controllen = 6124847, msg_flags = 0}
cmsg_space = "\000\000\000\000\000\000\000\000@NJ\005", '\000' <repeats 12 times>, "৴\001\000\000\000\000p\323ɀ\376\177\000\000A1u\000\000\000\000\000\260\322ɀ\376\177\000\000@NJ\005\000\000\000\000p\202\256\001\000\000\000\000\000\000\000\000\037\000\000\000\320\322ɀ\376\177\000\000\377\377\377\377\377\377\377\377\200(\303\000\000\000\000\000\000\000\001\000\000\000\000\000\320\322ɀ\376\177\000\000\200\000\000\000\000\000\000\000\200\322ɀ\376\177\000\000\001", '\000' <repeats 31 times>, "\210\224\016\002\000\000\000\000\002\000\366Y\005\240\250\206\000\000\000\000\000\000\000\000t/]\000\000\000\000\000 \323ɀ\376\177\000\000*L"...
count = 31
err = 0
is_ipc = 0
__PRETTY_FUNCTION__ = "uv__read"
#16 0x000000000075015b in uv__stream_io (loop=0x1abac00, w=0x1abb0c8, events=1) at ../deps/libuv/src/unix/stream.c:1301
stream = 0x1abb040
__PRETTY_FUNCTION__ = "uv__stream_io"
#17 0x0000000000755440 in uv__io_poll (loop=0x1abac00, timeout=1) at ../deps/libuv/src/unix/linux-core.c:384
max_safe_timeout = 1789569
events = {{events = 1, data = {ptr = 0x3, fd = 3, u32 = 3, u64 = 3}}, {events = 1, data = {ptr = 0x34, fd = 52, u32 = 52, u64 = 52}}, {events = 1, data = {ptr = 0xb8, fd = 184, u32 = 184, u64 = 184}}, {events = 1, data = {ptr = 0x22, fd = 34, u32 = 34, u64 = 34}}, {events = 1, data = {ptr = 0x2a, fd = 42, u32 = 42, u64 = 42}}, {events = 1, data = {ptr = 0xf, fd = 15, u32 = 15, u64 = 15}}, {events = 1, data = {ptr = 0xf, fd = 15, u32 = 15, u64 = 15}}, {events = 1, data = {ptr = 0xf, fd = 15, u32 = 15, u64 = 15}}, {events = 1, data = {ptr = 0x22, fd = 34, u32 = 34, u64 = 34}}, {events = 1, data = {ptr = 0x86, fd = 134, u32 = 134, u64 = 134}}, {events = 1, data = {ptr = 0x142, fd = 322, u32 = 322, u64 = 322}}, {events = 1, data = {ptr = 0x12a, fd = 298, u32 = 298, u64 = 298}}, {events = 1, data = {ptr = 0x2e, fd = 46, u32 = 46, u64 = 46}}, {events = 1, data = {ptr = 0x7b, fd = 123, u32 = 123, u64 = 123}}, {events = 1, data = {ptr = 0xf, fd = 15, u32 = 15, u64 = 15}}, {events = 1, data = {ptr = 0x154, fd = 340, u32 = 340, u64 = 340}}, {events = 1, data = {ptr = 0x86, fd = 134, u32 = 134, u64 = 134}}, {events = 1, data = {ptr = 0xd, fd = 13, u32 = 13, u64 = 13}}, {events = 1, data = {ptr = 0x6f, fd = 111, u32 = 111, u64 = 111}}, {events = 1, data = {ptr = 0x28, fd = 40, u32 = 40, u64 = 40}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 654 times>, {events = 28162448, data = {ptr = 0x1adc60800000000, fd = 0, u32 = 0, u64 = 120970502670909440}}, {events = 0, data = {ptr = 0x497bc5 <BN_CTX_get+85>, fd = 4815813, u32 = 4815813, u64 = 4815813}}, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x8, fd = 8, u32 = 8, u64 = 8}}, {events = 28165616, data = {ptr = 0x1adb99000000000, fd = 0, u32 = 0, u64 = 120956793135300608}}, {events = 0, data = {ptr = 0x4a7df7 <bn_sqr_fixed_top+103>, fd = 4881911, u32 = 4881911, u64 = 4881911}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 16, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 14 times>, {events = 0, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x1adc5f0, fd = 28165616, u32 = 28165616, u64 = 28165616}}, {events = 28166616, data = {ptr = 0x800000000, fd = 0, u32 = 0, u64 = 34359738368}}, {events = 0, data = {ptr = 0x4a13df <bn_from_montgomery_word+735>, fd = 4854751, u32 = 4854751, u64 = 4854751}}, {events = 4294967295, data = {ptr = 0x1adc9e0ffffffff, fd = -1, u32 = 4294967295, u64 = 120974733213695999}}, {events = 0, data = {ptr = 0x1adc590, fd = 28165520, u32 = 28165520, u64 = 28165520}}, {events = 8, data = {ptr = 0x1adc5f000000000, fd = 0, u32 = 0, u64 = 120970399591694336}}, {events = 0, data = {ptr = 0x1adb990, fd = 28162448, u32 = 28162448, u64 = 28162448}}, {events = 28165616, data = {ptr = 0x80c9ffe000000000, fd = 0, u32 = 0, u64 = 9280229844711374848}}, {events = 32766, data = {ptr = 0x1adc590, fd = 28165520, u32 = 28165520, u64 = 28165520}}, {events = 28166992, data = {ptr = 0x80c9ffe000000000, fd = 0, u32 = 0, u64 = 9280229844711374848}}, {events = 32766, data = {ptr = 0x4a16d4 <bn_from_mont_fixed_top+84>, fd = 4855508, u32 = 4855508, u64 = 4855508}}, {events = 28165520, data = {ptr = 0x80ca005000000000, fd = 0, u32 = 0, u64 = 9280230325747712000}}, {events = 32766, data = {ptr = 0x7ffe80c9ffe0, fd = -2134245408, u32 = 2160721888, u64 = 140731059142624}}, {events = 28162448, data = {ptr = 0x80ca000000000000, fd = 0, u32 = 0, u64 = 9280229982150328320}}, {events = 32766, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 2160722000, data = {ptr = 0x80c9ffe000007ffe, fd = 32766, u32 = 32766, u64 = 9280229844711407614}}, {events = 32766, data = {ptr = 0x1adb990, fd = 28162448, u32 = 28162448, u64 = 28162448}}, {events = 4820873, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 79 times>, {events = 0, data = {ptr = 0x80c9ffa000000000, fd = 0, u32 = 0, u64 = 9280229569833467904}}, {events = 32766, data = {ptr = 0x7ffe80c9ff90, fd = -2134245488, u32 = 2160721808, u64 = 140731059142544}}, {events = 8739333, data = {ptr = 0x80ca00c800000000, fd = 0, u32 = 0, u64 = 9280230841143787520}}, {events = 32766, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 471787821, data = {ptr = 0x7f34, fd = 32564, u32 = 32564, u64 = 32564}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 117, data = {ptr = 0x80c9fb0800000000, fd = 0, u32 = 0, u64 = 9280224518951927808}}, {events = 32766, data = {ptr = 0x7ffe80c9faf8, fd = -2134246664, u32 = 2160720632, u64 = 140731059141368}}, {events = 0, data = {ptr = 0x7f34, fd = 32564, u32 = 32564, u64 = 32564}}, {events = 0, data = {ptr = 0x7ffe80c9fb10, fd = -2134246640, u32 = 2160720656, u64 = 140731059141392}}, {events = 0, data = {ptr = 0x1ccb1d803ffa4000, fd = 1073364992, u32 = 1073364992, u64 = 2074784490000629760}}, {events = 32564, data = {ptr = 0x7ffe80ca0020, fd = -2134245344, u32 = 2160721952, u64 = 140731059142688}}, {events = 2160721936, data = {ptr = 0x855a0500007ffe, fd = 32766, u32 = 32766, u64 = 37535149423886334}}, {events = 0, data = {ptr = 0x7ffe80ca0148, fd = -2134245048, u32 = 2160722248, u64 = 140731059142984}}, {events = 0, data = {ptr = 0x1c1ee92d00000000, fd = 0, u32 = 0, u64 = 2026313261846102016}}, {events = 32564, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x80c9ff7500000000, fd = 0, u32 = 0, u64 = 9280229385149874176}}, {events = 0, data = {ptr = 0x7ffe80c9fb88, fd = -2134246520, u32 = 2160720776, u64 = 140731059141512}}, {events = 2160720760, data = {ptr = 0x7ffe, fd = 32766, u32 = 32766, u64 = 32766}}, {events = 1, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 2160720784, data = {ptr = 0x7ffe, fd = 32766, u32 = 32766, u64 = 32766}}, {events = 1073397760, data = {ptr = 0xb00000000, fd = 0, u32 = 0, u64 = 47244640256}}, {events = 0, data = {ptr = 0x855a2000000000, fd = 0, u32 = 0, u64 = 37535265387970560}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x80c9fba03ffac000, fd = 1073397760, u32 = 1073397760, u64 = 9280225172860354560}}, {events = 32766, data = {ptr = 0x7ffe00000000, fd = 0, u32 = 0, u64 = 140728898420736}}, {events = 10, data = {ptr = 0x300000000, fd = 0, u32 = 0, u64 = 12884901888}}, {events = 0, data = {ptr = 0x7ffe80c9ffd5, fd = -2134245419, u32 = 2160721877, u64 = 140731059142613}}, {events = 3, data = {ptr = 0x855a0800000000, fd = 0, u32 = 0, u64 = 37535162308755456}}, {events = 0, data = {ptr = 0x100000000, fd = 0, u32 = 0, u64 = 4294967296}}, {events = 2160720880, data = {ptr = 0x7ffe, fd = 32766, u32 = 32766, u64 = 32766}}, {events = 0, data = {ptr = 0xffffffffffffffff, fd = -1, u32 = 4294967295, u64 = 18446744073709551615}}, {events = 0, data = {ptr = 0x80ca02500000000b, fd = 11, u32 = 11, u64 = 9280232524770967563}}, {events = 32766, data = {ptr = 0x855a09, fd = 8739337, u32 = 8739337, u64 = 8739337}}, {events = 27, data = {ptr = 0x80ca026000000000, fd = 0, u32 = 0, u64 = 9280232593490444288}}, {events = 32766, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 484340239, data = {ptr = 0x80c9ff8000007f34, fd = 32564, u32 = 32564, u64 = 9280229432394546996}}, {events = 32766, data = {ptr = 0x1b, fd = 27, u32 = 27, u64 = 27}}, {events = 9, data = {ptr = 0x1000000000, fd = 0, u32 = 0, u64 = 68719476736}}, {events = 48, data = {ptr = 0x7ffe80ca0220, fd = -2134244832, u32 = 2160722464, u64 = 140731059143200}}, {events = 2160722272, data = {ptr = 0x80c9fc4000007ffe, fd = 32766, u32 = 32766, u64 = 9280225858981756926}}, {events = 0, data = {ptr = 0x7ffe80ca0250, fd = -2134244784, u32 = 2160722512, u64 = 140731059143248}}, {events = 10, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 484345771, data = {ptr = 0x80c9fc2000007f34, fd = 32564, u32 = 32564, u64 = 9280225721542803252}}, {events = 32766, data = {ptr = 0xdffc771e6fc824e1, fd = 1875387617, u32 = 1875387617, u64 = 16139906137197126881}}, {events = 64, data = {ptr = 0x80c9fd9000000000, fd = 0, u32 = 0, u64 = 9280227302090735616}}, {events = 32766, data = {ptr = 0xd, fd = 13, u32 = 13, u64 = 13}}, {events = 0, data = {ptr = 0x1800000000, fd = 0, u32 = 0, u64 = 103079215104}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 1782228940, data = {ptr = 0x599bcd4, fd = 93961428, u32 = 93961428, u64 = 93961428}}, {events = 0, data = {ptr = 0x33c5337909dfc9c5, fd = 165661125, u32 = 165661125, u64 = 3730444461296044485}}, {events = 0, data = {ptr = 0x6ece5f9800000000, fd = 0, u32 = 0, u64 = 7984424295814135808}}, {events = 4181041113, data = {ptr = 0x1141bbd3034a9897, fd = 55220375, u32 = 55220375, u64 = 1243481487098681495}}, {events = 44176300, data = {ptr = 0x80c9fd800dce2fdc, fd = 231616476, u32 = 231616476, u64 = 9280227233602875356}}, {events = 32766, data = {ptr = 0x7ffe80c9fd80, fd = -2134246016, u32 = 2160721280, u64 = 140731059142016}}, {events = 64, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x1bdeb60, fd = 29223776, u32 = 29223776, u64 = 29223776}}, {events = 2160721088, data = {ptr = 0x91d210000007ffe, fd = 32766, u32 = 32766, u64 = 656717404549709822}}, {events = 2335932680, data = {ptr = 0x7ffe80c9ffca, fd = -2134245430, u32 = 2160721866, u64 = 140731059142602}}, {events = 29223776, data = {ptr = 0x1f4571000000000, fd = 0, u32 = 0, u64 = 140833214586421248}}, {events = 0, data = {ptr = 0x1bdeb60, fd = 29223776, u32 = 29223776, u64 = 29223776}}, {events = 32790288, data = {ptr = 0x80c9fd4000000000, fd = 0, u32 = 0, u64 = 9280226958493351936}}, {events = 32766, data = {ptr = 0x7ffe80c9fd10, fd = -2134246128, u32 = 2160721168, u64 = 140731059141904}}, {events = 5305299, data = {ptr = 0x2800000000, fd = 0, u32 = 0, u64 = 171798691840}}, {events = 0, data = {ptr = 0x1bdeb60, fd = 29223776, u32 = 29223776, u64 = 29223776}}, {events = 77177669, data = {ptr = 0x499a34500000000, fd = 0, u32 = 0, u64 = 331475564336513024}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 11 times>, {events = 2160721456, data = {ptr = 0x40bf1000007ffe, fd = 32766, u32 = 32766, u64 = 18224473949896702}}, {events = 0, data = {ptr = 0x7ffe80ca07a0, fd = -2134243424, u32 = 2160723872, u64 = 140731059144608}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x8b3b8108091d2100, fd = 152903936, u32 = 152903936, u64 = 10032754466410537216}}, {events = 2160723872, data = {ptr = 0x100007ffe, fd = 32766, u32 = 32766, u64 = 4295000062}}, {events = 0, data = {ptr = 0x6019830, fd = 100767792, u32 = 100767792, u64 = 100767792}}, {events = 2160721512, data = {ptr = 0x100007ffe, fd = 32766, u32 = 32766, u64 = 4295000062}}, {events = 0, data = {ptr = 0x1f45710, fd = 32790288, u32 = 32790288, u64 = 32790288}}, {events = 100773864, data = {ptr = 0x414cb100000000, fd = 0, u32 = 0, u64 = 18380196579115008}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 2160722029, data = {ptr = 0x100007ffe, fd = 32766, u32 = 32766, u64 = 4295000062}}, {events = 0, data = {ptr = 0x1000000001, fd = 1, u32 = 1, u64 = 68719476737}}, {events = 2160722576, data = {ptr = 0x80ca020f00007ffe, fd = 32766, u32 = 32766, u64 = 9280232245598126078}}, {events = 32766, data = {ptr = 0xc16d40 <chacha20_poly1305>, fd = 12676416, u32 = 12676416, u64 = 12676416}}, {events = 100773864, data = {ptr = 0x601afe000000000, fd = 0, u32 = 0, u64 = 432820415811813376}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 1, data = {ptr = 0x80c9fe6000000000, fd = 0, u32 = 0, u64 = 9280228195443933184}}, {events = 32766, data = {ptr = 0x28, fd = 40, u32 = 40, u64 = 40}}, {events = 6052907, data = {ptr = 0x4f5217000000000, fd = 0, u32 = 0, u64 = 357228510365876224}}, {events = 0, data = {ptr = 0x60c6f20, fd = 101478176, u32 = 101478176, u64 = 101478176}}, {events = 2160721696, data = {ptr = 0x40bf1000007ffe, fd = 32766, u32 = 32766, u64 = 18224473949896702}}, {events = 0, data = {ptr = 0x7ffe80ca07a0, fd = -2134243424, u32 = 2160723872, u64 = 140731059144608}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7f341c22553c <__GI___libc_free+76>, fd = 472012092, u32 = 472012092, u64 = 139861787039036}}, {events = 2160721616, data = {ptr = 0x5c528200007ffe, fd = 32766, u32 = 32766, u64 = 25986416156639230}}, {events = 0, data = {ptr = 0x7ffe80c9fef0, fd = -2134245648, u32 = 2160721648, u64 = 140731059142384}}, {events = 84243728, data = {ptr = 0x80c9fef000000000, fd = 0, u32 = 0, u64 = 9280228813919223808}}, {events = 32766, data = {ptr = 0x60c6f20, fd = 101478176, u32 = 101478176, u64 = 101478176}}, {events = 62259736, data = {ptr = 0x1abac0000000000, fd = 0, u32 = 0, u64 = 120378931055427584}}, {events = 0, data = {ptr = 0x7ffe80c9ff20, fd = -2134245600, u32 = 2160721696, u64 = 140731059142432}}, {events = 7662972, data = {ptr = 0x80ca010000000000, fd = 0, u32 = 0, u64 = 9280231081661956096}}, {events = 32766, data = {ptr = 0x7ffe80ca0100, fd = -2134245120, u32 = 2160722176, u64 = 140731059142912}}, {events = 2160722336, data = {ptr = 0x3b6019000007ffe, fd = 32766, u32 = 32766, u64 = 267402945862074366}}, {events = 0, data = {ptr = 0x7ffe80ca0020, fd = -2134245344, u32 = 2160721952, u64 = 140731059142688}}, {events = 7663963, data = {ptr = 0x248084800000000, fd = 0, u32 = 0, u64 = 164390491729690624}}, {events = 0, data = {ptr = 0x3b60190, fd = 62259600, u32 = 62259600, u64 = 62259600}}, {events = 17433576, data = {ptr = 0x200000000, fd = 0, u32 = 0, u64 = 8589934592}}, {events = 3009084107, data = {ptr = 0xb135918e00000400, fd = 1024, u32 = 1024, u64 = 12769272357550294016}}, {events = 72, data = {ptr = 0x80ca015800000000, fd = 0, u32 = 0, u64 = 9280231459619078144}}, {events = 32766, data = {ptr = 0x7ffe80ca0100, fd = -2134245120, u32 = 2160722176, u64 = 140731059142912}}, {events = 2160722304, data = {ptr = 0x2a7085000007ffe, fd = 32766, u32 = 32766, u64 = 191130648876974078}}, {events = 0, data = {ptr = 0x2a70770, fd = 44500848, u32 = 44500848, u64 = 44500848}}, {events = 44501120, data = {ptr = 0x80c9ffb000000000, fd = 0, u32 = 0, u64 = 9280229638552944640}}, {events = 6, data = {ptr = 0x10a03e8 <RTC::EncryptBuffer+8>, fd = 17433576, u32 = 17433576, u64 = 17433576}}, {events = 4580016, data = {ptr = 0x10a041800000000, fd = 0, u32 = 0, u64 = 74876844930760704}}, {events = 0, data = {ptr = 0x45e2b0 <AES_encrypt>, fd = 4580016, u32 = 4580016, u64 = 4580016}}...}
pe = 0x7ffe80c9d4b0
e = {events = 1, data = {ptr = 0x11c, fd = 284, u32 = 284, u64 = 284}}
real_timeout = 1
q = 0x3b7c2c0
w = 0x1abb0c8
sigset = {__val = {0 <repeats 16 times>}}
psigset = 0x0
base = 8513784721
have_signals = 0
nevents = 0
count = 48
nfds = 1
fd = 3
op = 3
i = 0
__PRETTY_FUNCTION__ = "uv__io_poll"
#18 0x0000000000745773 in uv_run (loop=0x1abac00, mode=UV_RUN_DEFAULT) at ../deps/libuv/src/unix/core.c:368
timeout = 1
r = 1
ran_pending = 1
#19 0x00000000005b0350 in DepLibUV::RunLoop () at ../src/DepLibUV.cpp:52
__FUNCTION__ = "RunLoop"
#20 0x00000000005bc545 in Worker::Worker (this=0x7ffe80ca0610, channel=0x1abaff0) at ../src/Worker.cpp:31
No locals.
#21 0x000000000073d520 in main (argc=10, argv=0x7ffe80ca07a8) at ../src/main.cpp:116
worker = {<Channel::UnixStreamSocket::Listener> = {_vptr.Listener = 0xc0df60 <vtable for Worker+16>}, <SignalsHandler::Listener> = {_vptr.Listener = 0xc0df98 <vtable for Worker+72>}, channel = 0x1abaff0, signalsHandler = 0x1ae8270, closed = false, mapRouters = std::unordered_map with 3 elements = {["8d87f4f3-47be-4d99-a093-b980a671ffaf"] = 0x1af86d0, ["ba3cf5b6-e05d-49f3-ad15-ba9978500ddc"] = 0x2a62490, ["47916218-f557-4109-a47b-116a9b473ddb"] = 0x2a92bb0}}
__FUNCTION__ = "main"
version = "3.2.5"
channel = 0x1abaff0
(gdb)
Of course.
BTW you can select a code block by holding down the Shift key.
Thanks, I read about it but did not remember that Shift should be used 