order-book lib sometimes core-dumps on ctl-c.. (deribit, cryptofeed deps) #12

New Issue

goodboy · 2024-11-21T18:51:36Z

goodboy commented

2024-11-21 18:51:36 +00:00

Just logging this so we don’t forget about it (since i just did the gdb walkthrough on it..)

i think the issue is reproduced by..

use of breakpoint() from an asyncio side task without having first enabled greenback inside the tractor runtime using,


  if debug_mode:
      from tractor.devx._debug import maybe_init_greenback
      await maybe_init_greenback()

which, must be called from a trio side task..

Not sure exactly how this eventually results in a epoll_wait() call crashing from inside the order-book code (utils/orderbook.c see below) ??

core-dump appears to be sourced inside the file `utils/orderbook.c` ??

(piker-py3.12)  >>> coredumpctl info
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
           PID: 93643 (python)
           UID: 1000 (goodboy)
           GID: 1001 (goodboy)
        Signal: 3 (QUIT)
     Timestamp: Thu 2024-11-21 12:23:47 EST (1h 25min ago)
  Command Line: /home/goodboy/repos/piker/.venv/bin/python -m tractor._child --uid $'(\'brokerd.deribit\', \'e8bdb3db>
    Executable: /usr/bin/python3.12
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (goodboy)
       Boot ID: 8cee065f4d564d7f82796cd313864421
    Machine ID: 9094c966c6ea4b74945a70b76072614a
      Hostname: DoggyBoi
       Storage: /var/lib/systemd/coredump/core.python.1000.8cee065f4d564d7f82796cd313864421.93643.1732209827000000.zs>
  Size on Disk: 25.3M
       Message: Process 93643 (python) of user 1000 dumped core.

                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libquadmath-96973f99.s>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libquadmath-96973f99.s>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libgfortran-040039e1.s>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libgfortran-040039e1.s>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpyt>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpyt>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpytho>
                Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpytho>
                Stack trace of thread 93643:
                #0  0x0000711c5cb288b2 epoll_wait (libc.so.6 + 0x1198b2)
                #1  0x0000711c5d341859 n/a (select.cpython-312-x86_64-linux-gnu.so + 0x2859)
                #2  0x0000711c5cd8e59d _PyEval_EvalFrameDefault (libpython3.12.so.1.0 + 0x18e59d)
                #3  0x0000711c5ce4de35 PyEval_EvalCode (libpython3.12.so.1.0 + 0x24de35)
                #4  0x0000711c5ce6a516 n/a (libpython3.12.so.1.0 + 0x26a516)
                #5  0x0000711c5cda6a0e n/a (libpython3.12.so.1.0 + 0x1a6a0e)
                #6  0x0000711c5cda68bd PyObject_Vectorcall (libpython3.12.so.1.0 + 0x1a68bd)
                #7  0x0000711c5cd8b5b7 _PyEval_EvalFrameDefault (libpython3.12.so.1.0 + 0x18b5b7)
                #8  0x0000711c5ce7f69f n/a (libpython3.12.so.1.0 + 0x27f69f)
                #9  0x0000711c5ce7ef46 Py_RunMain (libpython3.12.so.1.0 + 0x27ef46)
                #10 0x0000711c5ce3c5ec Py_BytesMain (libpython3.12.so.1.0 + 0x23c5ec)
                #11 0x0000711c5ca34e08 n/a (libc.so.6 + 0x25e08)
                #12 0x0000711c5ca34ecc __libc_start_main (libc.so.6 + 0x25ecc)
                #13 0x00005ccd362fd045 _start (python3.12 + 0x1045)

                Stack trace of thread 93649:
                #0  0x0000711c5ca9fa19 n/a (libc.so.6 + 0x90a19)
                #1  0x0000711c5caa2479 pthread_cond_wait (libc.so.6 + 0x93479)
                #2  0x0000711c521506fb n/a (/home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/l>
                #3  0x0000000000000002 n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64

with a coredumpctl debug and a bit of poking showing..


(gdb) bt
#0  0x0000711c5cb288b2 in epoll_wait () from /usr/lib/libc.so.6
#1  0x0000711c5d341859 in ?? () from /usr/lib/python3.12/lib-dynload/select.cpython-312-x86_64-linux-gnu.so
#2  0x0000711c5cd8e59d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.12.so.1.0
#3  0x0000711c5ce4de35 in PyEval_EvalCode () from /usr/lib/libpython3.12.so.1.0
#4  0x0000711c5ce6a516 in ?? () from /usr/lib/libpython3.12.so.1.0
#5  0x0000711c5cda6a0e in ?? () from /usr/lib/libpython3.12.so.1.0
#6  0x0000711c5cda68bd in PyObject_Vectorcall () from /usr/lib/libpython3.12.so.1.0
#7  0x0000711c5cd8b5b7 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.12.so.1.0
#8  0x0000711c5ce7f69f in ?? () from /usr/lib/libpython3.12.so.1.0
#9  0x0000711c5ce7ef46 in Py_RunMain () from /usr/lib/libpython3.12.so.1.0
#10 0x0000711c5ce3c5ec in Py_BytesMain () from /usr/lib/libpython3.12.so.1.0
#11 0x0000711c5ca34e08 in ?? () from /usr/lib/libc.so.6
#12 0x0000711c5ca34ecc in __libc_start_main () from /usr/lib/libc.so.6
#13 0x00005ccd362fd045 in _start ()
(gdb) l
warning: 1  orderbook/utils.c: No such file or directory

which turns out to be inside a sub-dep of `cryptofeed` by the same author:

issue in cryptofeed talking about same file: https://github.com/bmoscon/cryptofeed/issues/1022
underlying repo: https://github.com/bmoscon/orderbook

Just logging this so we don't forget about it (since i just did the `gdb` walkthrough on it..) --- #### *i think* the issue is reproduced by.. use of `breakpoint()` from an `asyncio` side task without having first enabled `greenback` inside the `tractor` runtime using, ```python if debug_mode: from tractor.devx._debug import maybe_init_greenback await maybe_init_greenback() ``` which, **must be called** from a `trio` side task.. Not sure exactly how this eventually results in a `epoll_wait()` call crashing from inside the `order-book` code (`utils/orderbook.c` see below) ?? --- #### core-dump appears to be sourced inside the file `utils/orderbook.c` ?? ```sh (piker-py3.12) >>> coredumpctl info Hint: You are currently not seeing messages from other users and the system. Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages. Pass -q to turn off this notice. PID: 93643 (python) UID: 1000 (goodboy) GID: 1001 (goodboy) Signal: 3 (QUIT) Timestamp: Thu 2024-11-21 12:23:47 EST (1h 25min ago) Command Line: /home/goodboy/repos/piker/.venv/bin/python -m tractor._child --uid $'(\'brokerd.deribit\', \'e8bdb3db> Executable: /usr/bin/python3.12 Control Group: /user.slice/user-1000.slice/session-1.scope Unit: session-1.scope Slice: user-1000.slice Session: 1 Owner UID: 1000 (goodboy) Boot ID: 8cee065f4d564d7f82796cd313864421 Machine ID: 9094c966c6ea4b74945a70b76072614a Hostname: DoggyBoi Storage: /var/lib/systemd/coredump/core.python.1000.8cee065f4d564d7f82796cd313864421.93643.1732209827000000.zs> Size on Disk: 25.3M Message: Process 93643 (python) of user 1000 dumped core. Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libquadmath-96973f99.s> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libquadmath-96973f99.s> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libgfortran-040039e1.s> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/libgfortran-040039e1.s> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpyt> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpyt> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpytho> Module /home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpytho> Stack trace of thread 93643: #0 0x0000711c5cb288b2 epoll_wait (libc.so.6 + 0x1198b2) #1 0x0000711c5d341859 n/a (select.cpython-312-x86_64-linux-gnu.so + 0x2859) #2 0x0000711c5cd8e59d _PyEval_EvalFrameDefault (libpython3.12.so.1.0 + 0x18e59d) #3 0x0000711c5ce4de35 PyEval_EvalCode (libpython3.12.so.1.0 + 0x24de35) #4 0x0000711c5ce6a516 n/a (libpython3.12.so.1.0 + 0x26a516) #5 0x0000711c5cda6a0e n/a (libpython3.12.so.1.0 + 0x1a6a0e) #6 0x0000711c5cda68bd PyObject_Vectorcall (libpython3.12.so.1.0 + 0x1a68bd) #7 0x0000711c5cd8b5b7 _PyEval_EvalFrameDefault (libpython3.12.so.1.0 + 0x18b5b7) #8 0x0000711c5ce7f69f n/a (libpython3.12.so.1.0 + 0x27f69f) #9 0x0000711c5ce7ef46 Py_RunMain (libpython3.12.so.1.0 + 0x27ef46) #10 0x0000711c5ce3c5ec Py_BytesMain (libpython3.12.so.1.0 + 0x23c5ec) #11 0x0000711c5ca34e08 n/a (libc.so.6 + 0x25e08) #12 0x0000711c5ca34ecc __libc_start_main (libc.so.6 + 0x25ecc) #13 0x00005ccd362fd045 _start (python3.12 + 0x1045) Stack trace of thread 93649: #0 0x0000711c5ca9fa19 n/a (libc.so.6 + 0x90a19) #1 0x0000711c5caa2479 pthread_cond_wait (libc.so.6 + 0x93479) #2 0x0000711c521506fb n/a (/home/goodboy/repos/piker/.venv/lib/python3.12/site-packages/numpy.libs/l> #3 0x0000000000000002 n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 ``` with a `coredumpctl debug` and a bit of poking showing.. ```sh (gdb) bt #0 0x0000711c5cb288b2 in epoll_wait () from /usr/lib/libc.so.6 #1 0x0000711c5d341859 in ?? () from /usr/lib/python3.12/lib-dynload/select.cpython-312-x86_64-linux-gnu.so #2 0x0000711c5cd8e59d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.12.so.1.0 #3 0x0000711c5ce4de35 in PyEval_EvalCode () from /usr/lib/libpython3.12.so.1.0 #4 0x0000711c5ce6a516 in ?? () from /usr/lib/libpython3.12.so.1.0 #5 0x0000711c5cda6a0e in ?? () from /usr/lib/libpython3.12.so.1.0 #6 0x0000711c5cda68bd in PyObject_Vectorcall () from /usr/lib/libpython3.12.so.1.0 #7 0x0000711c5cd8b5b7 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.12.so.1.0 #8 0x0000711c5ce7f69f in ?? () from /usr/lib/libpython3.12.so.1.0 #9 0x0000711c5ce7ef46 in Py_RunMain () from /usr/lib/libpython3.12.so.1.0 #10 0x0000711c5ce3c5ec in Py_BytesMain () from /usr/lib/libpython3.12.so.1.0 #11 0x0000711c5ca34e08 in ?? () from /usr/lib/libc.so.6 #12 0x0000711c5ca34ecc in __libc_start_main () from /usr/lib/libc.so.6 #13 0x00005ccd362fd045 in _start () (gdb) l warning: 1 orderbook/utils.c: No such file or directory ``` --- #### which turns out to be inside a sub-dep of `cryptofeed` by the same author: - issue in `cryptofeed` talking about same file: https://github.com/bmoscon/cryptofeed/issues/1022 - underlying repo: https://github.com/bmoscon/orderbook

Sign in to join this conversation.

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#12

order-book lib sometimes core-dumps on ctl-c.. (deribit, cryptofeed deps) #12

i think the issue is reproduced by..

core-dump appears to be sourced inside the file utils/orderbook.c ??

which turns out to be inside a sub-dep of cryptofeed by the same author:

`order-book` lib sometimes core-dumps on ctl-c.. (`deribit`, `cryptofeed` deps) #12

core-dump appears to be sourced inside the file `utils/orderbook.c` ??

which turns out to be inside a sub-dep of `cryptofeed` by the same author: