pikers
/
piker
3
1
Fork 0
Code Issues 9 Pull Requests 14 Packages Projects Releases Wiki Activity

ibrk: await chan.receive() shrouds ib-gw connection timeout? #46

New Issue
Open
opened 2025-07-21 22:48:28 +00:00 by goodboy · 0 comments
goodboy commented 2025-07-21 22:48:28 +00:00
Owner

The following traceback should explain it, with the only missing counterpart being a client piker chartcmd (like piker --pdb -l cancel chart mnq.cme.ib).

It seems we need an integration test between tractor.trionics.maybe_open_context() and tractor.to_asyncio.open_channel_from() since the log @ tractor _rpc.py:845 IPC context terminated.. only reports the to_asyncio (internals) tb and not the underlying,

ConnectionError: No ib APIs could be found scanning @:
[('mini.dover', 4000)]

which is the src exc?

Full traceback from pikerd

(piker313)  >>> pikerd --pdb -l info
Jul 21 18:29:22 (no actor_name context[no actor_uid context], 93453, no task context)) [WARNING] piker __init__.py:142
!!! YOU HAVE ENABLED DAEMON DEBUG MODE !!!
When a `piker` daemon crashes it will block the task-thread until resumed from console!


Jul 21 18:29:22 (no actor_name context[no actor_uid context], 93453, tractor._root.open_root_actor.<locals>.ping_tpt_socket[370608])) [INFO] tractor _root.py:367 No root-actor registry found @ TCPAddress[('127.0.0.1', 6116)]

Jul 21 18:29:22 (pikerd[508ed6], 93453, piker.cli.pikerd.<locals>.main[369728])) [INFO] tractor _root.py:477 Starting actor-runtime for 'pikerd@93453'
-> Opening new registry @ TCPAddress[('127.0.0.1', 6116)]

Jul 21 18:29:22 (pikerd[508ed6], 93453, tractor.ipc._server._serve_ipc_eps[371136])) [INFO] tractor.ipc _tcp.py:181 Listening on TCP socket
[> TCPAddress[('127.0.0.1', 6116)]

Jul 21 18:29:28 (pikerd[508ed6], 93453, spawn_brokerd[374304])) [INFO] piker.service _daemon.py:205 Spawning ib broker daemon
Jul 21 18:29:29 (brokerd.ib[3065ce], 93474, no task context)) [INFO] tractor _entry.py:131 Starting `trio` subactor from parent @ ('127.0.0.1', 6116)
>(
|_<Actor(
   |_id: 'brokerd.ib@93474'
   |_rpc: 0 active tasks
   |_state: 'running'
     task: <dne>
     loglevel: 'INFO'
     subactors_spawned: 0
     spawn_method: 'trio'
  )>

Jul 21 18:29:29 (brokerd.ib[3065ce], 93474, tractor._runtime.async_main[132576])) [ERROR] tractor.devx _stackscope.py:240 `stackscope` not installed for use in debug mode!
`Ignoring enable_stack_on_sig() call!

Jul 21 18:29:30 (brokerd.ib[3065ce], 93474, tractor.ipc._server._serve_ipc_eps[695552])) [INFO] tractor.ipc _uds.py:218 Attempting to bind UDS socket
>[
|_UDSAddress[(/run/user/1000/tractor, brokerd.ib::3065cebe-da3a-4da7-9d1e-78e2b17d10bc@93474.sock, <unknown-peer-pid>)]

Jul 21 18:29:30 (brokerd.ib[3065ce], 93474, tractor.ipc._server._serve_ipc_eps[695552])) [INFO] tractor.ipc _uds.py:237 Listening on UDS socket
[>
 |_UDSAddress[(/run/user/1000/tractor, brokerd.ib::3065cebe-da3a-4da7-9d1e-78e2b17d10bc@93474.sock, <unknown-peer-pid>)]

Jul 21 18:29:30 (pikerd[508ed6], 93453, piker.service._mngr.Services.start_service_task.<locals>.open_context_in_task[281312])) [INFO] piker.service _mngr.py:105 `pikerd` service brokerd.ib started with value None
Jul 21 18:29:30 (pikerd[508ed6], 93453, piker.service._mngr.Services.start_service_task.<locals>.open_context_in_task[281312])) [WARNING] tractor _context.py:1307 `Context.result()` is DEPRECATED!
Use `Context.[no]wait_for_result()` instead!

API connection failed: TimeoutError()
API connection failed: TimeoutError()
API connection failed: TimeoutError()
Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [WARNING] piker.brokers.ib api.py:1304 Failed to connect on mini.dover:4000 after 2 tries with
'API connection failed: TimeoutError()'

Retrying with a new client id..

Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [ERROR] tractor to_asyncio.py:400 `asyncio` task errored

Traceback (most recent call last):
  File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim
    next_send = outcome.Value((yield next_yield))
                               ^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients
    await ib.connectAsync(
    ...<8 lines>...
    )
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync
    await self.client.connectAsync(host, port, clientId, timeout)
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync
    await asyncio.wait_for(self.apiStart, timeout)
  File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result
    result: Any = await coro
                  ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio
    async with load_aio_clients(
               ~~~~~~~~~~~~~~~~^
        disconnect_on_exit=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as accts2clients:
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients
    raise ConnectionError(
    ...<3 lines>...
    ) from _err
ConnectionError: No ib APIs could be found scanning @:
[('mini.dover', 4000)]
Check your `brokers.toml` and/or network
Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [ERROR] tractor to_asyncio.py:617 `trio`-side reports that the `asyncio`-side errored
x)>
 |_<Task finished name='Task-2' coro=<greenback_shim() done, defined at /home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py:90> exception=ConnectionError("No ib APIs could be found scanning @:\n[('mini.dover', 4000)]\nCheck your `brokers.toml` and/or network")>

Traceback (most recent call last):
  File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim
    next_send = outcome.Value((yield next_yield))
                               ^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients
    await ib.connectAsync(
    ...<8 lines>...
    )
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync
    await self.client.connectAsync(host, port, clientId, timeout)
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync
    await asyncio.wait_for(self.apiStart, timeout)
  File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 530, in signal_trio_when_done
    res: Any = task.result()
               ~~~~~~~~~~~^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 116, in greenback_shim
    return await _greenback_shim(orig_coro, next_send)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 201, in _greenback_shim
    next_yield, resume_greenlet = resume_greenlet.switch(next_send)
                                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 81, in trampoline
    next_yield: Any = next_send.send(orig_coro)  # type: ignore
                      ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 231, in send
    return gen.throw(self.error)
           ~~~~~~~~~^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result
    result: Any = await coro
                  ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio
    async with load_aio_clients(
               ~~~~~~~~~~~~~~~~^
        disconnect_on_exit=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as accts2clients:
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients
    raise ConnectionError(
    ...<3 lines>...
    ) from _err
ConnectionError: No ib APIs could be found scanning @:
[('mini.dover', 4000)]
Check your `brokers.toml` and/or network
Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [CANCEL] tractor to_asyncio.py:647 Cancelling `trio` side due to aio-side src exc
No ib APIs could be found scanning @:
[('mini.dover', 4000)]
Check your `brokers.toml` and/or network

(c>
  |_<Task 'tractor.trionics._mngrs._Cache.run_ctx' at 0x7fa640195020>

Exception in callback _run_asyncio_task.<locals>.signal_trio_when_done() at /home/goodboy/repos/tractor/tractor/to_asyncio.py:495
handle: <Handle _run_asyncio_task.<locals>.signal_trio_when_done() at /home/goodboy/repos/tractor/tractor/to_asyncio.py:495>
Traceback (most recent call last):
  File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for
    return await fut
           ^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim
    next_send = outcome.Value((yield next_yield))
                               ^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients
    await ib.connectAsync(
    ...<8 lines>...
    )
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync
    await self.client.connectAsync(host, port, clientId, timeout)
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync
    await asyncio.wait_for(self.apiStart, timeout)
  File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for
    async with timeouts.timeout(timeout):
               ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 727, in signal_trio_when_done
    raise aio_err
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 530, in signal_trio_when_done
    res: Any = task.result()
               ~~~~~~~~~~~^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 116, in greenback_shim
    return await _greenback_shim(orig_coro, next_send)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 201, in _greenback_shim
    next_yield, resume_greenlet = resume_greenlet.switch(next_send)
                                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 81, in trampoline
    next_yield: Any = next_send.send(orig_coro)  # type: ignore
                      ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 231, in send
    return gen.throw(self.error)
           ~~~~~~~~~^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result
    result: Any = await coro
                  ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio
    async with load_aio_clients(
               ~~~~~~~~~~~~~~~~^
        disconnect_on_exit=False,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as accts2clients:
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients
    raise ConnectionError(
    ...<3 lines>...
    ) from _err
ConnectionError: No ib APIs could be found scanning @:
[('mini.dover', 4000)]
Check your `brokers.toml` and/or network
Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_symbol_search[698544])) [ERROR] tractor _rpc.py:845 IPC context terminated

after having 'errored'

EndOfChannel()
Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke
    raise scope_err
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke
    collapse_eg(),
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg
    raise exc
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke
    res: Any|PayloadT = await coro
                        ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/symbols.py", line 198, in open_symbol_search
    open_client_proxies() as (proxies, _),
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies
    tractor.trionics.maybe_open_context(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        acm_func=tractor.to_asyncio.open_channel_from,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...


    ) as (cache_hit, (clients, _)),
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 320, in maybe_open_context
    yielded: Any = await service_n.start(
                   ^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start
    raise exc.exceptions[0] from None
  File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 220, in run_ctx
    async with mng as value:
               ^^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 1199, in open_channel_from
    first = await chan.receive()
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 255, in receive
    raise err
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 244, in receive
    return await self._from_aio.receive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_channel.py", line 362, in receive
    return await trio.lowlevel.wait_task_rescheduled(abort_fn)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_traps.py", line 208, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 213, in unwrap
    raise captured_error
trio.EndOfChannel


scope_error:
EndOfChannel()


Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [ERROR] tractor _rpc.py:845 IPC context terminated

after having 'errored'

AssertionError("Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc'))")
Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke
    raise scope_err
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke
    collapse_eg(),
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg
    raise exc
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke
    res: Any|PayloadT = await coro
                        ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/data/feed.py", line 512, in open_feed_bus
    await bus.nursery.start(
    ...<13 lines>...
    )
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start
    raise exc.exceptions[0] from None
  File "/home/goodboy/repos/piker/piker/data/feed.py", line 291, in allocate_persistent_feed
    ) = await bus.nursery.start(
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
    )
    ^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start
    raise exc.exceptions[0] from None
  File "/home/goodboy/repos/piker/piker/brokers/ib/feed.py", line 913, in stream_quotes
    open_data_client() as proxy,
    ~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/feed.py", line 114, in open_data_client
    open_client_proxies() as (proxies, clients),
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies
    tractor.trionics.maybe_open_context(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        acm_func=tractor.to_asyncio.open_channel_from,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...


    ) as (cache_hit, (clients, _)),
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 315, in maybe_open_context
    assert not resources.get(ctx_key), f'Resource exists? {ctx_key}'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc'))


scope_error:
AssertionError("Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc'))")


Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [WARNING] tractor.devx _trace.py:416 Already an ongoing repl request?
|_<Task 'tractor.devx.debug._tty_lock.request_root_stdio_lock' at 0x7fa640195b20>

REPL task is
|_None


Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [WARNING] tractor.devx _trace.py:463 Local task already has active debug request
|_<Task 'open_feed_bus' at 0x7fa6401952e0>

Waiting for previous request to complete..

Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_symbol_search[698544])) [PDB] tractor.devx _post_mortem.py:172 Opening a pdb REPL in crashed actor
x>(
 |_ <Task 'open_symbol_search' at 0x7fa640194cb0> @ ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc')

Traceback (most recent call last):
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 285, in _errors_relayed_via_ipc
    yield  # run RPC invoke body
    ^^^^^
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke
    raise scope_err
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke
    collapse_eg(),
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__
    await self.gen.athrow(value)
  File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg
    raise exc
  File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke
    res: Any|PayloadT = await coro
                        ^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/symbols.py", line 198, in open_symbol_search
    open_client_proxies() as (proxies, _),
    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies
    tractor.trionics.maybe_open_context(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        acm_func=tractor.to_asyncio.open_channel_from,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...


    ) as (cache_hit, (clients, _)),
    ^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 320, in maybe_open_context
    yielded: Any = await service_n.start(
                   ^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start
    raise exc.exceptions[0] from None
  File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 220, in run_ctx
    async with mng as value:
               ^^^
  File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 1199, in open_channel_from
    first = await chan.receive()
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 255, in receive
    raise err
  File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 244, in receive
    return await self._from_aio.receive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_channel.py", line 362, in receive
    return await trio.lowlevel.wait_task_rescheduled(abort_fn)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_traps.py", line 208, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 213, in unwrap
    raise captured_error
trio.EndOfChannel

[22] > /home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py(213)unwrap()
-> raise captured_error
(Pdb+)
The following traceback should explain it, with the only missing counterpart being a client `piker chart `cmd (like `piker --pdb -l cancel chart mnq.cme.ib`). It seems we need an integration test between `tractor.trionics.maybe_open_context()` and `tractor.to_asyncio.open_channel_from()` since the log @ `tractor _rpc.py:845 IPC context terminated`.. only reports the `to_asyncio` (internals) tb and **not** the underlying, ```python ConnectionError: No ib APIs could be found scanning @: [('mini.dover', 4000)] ``` which is the src exc? --- Full traceback from `pikerd` ---- ```python (piker313) >>> pikerd --pdb -l info Jul 21 18:29:22 (no actor_name context[no actor_uid context], 93453, no task context)) [WARNING] piker __init__.py:142 !!! YOU HAVE ENABLED DAEMON DEBUG MODE !!! When a `piker` daemon crashes it will block the task-thread until resumed from console! Jul 21 18:29:22 (no actor_name context[no actor_uid context], 93453, tractor._root.open_root_actor.<locals>.ping_tpt_socket[370608])) [INFO] tractor _root.py:367 No root-actor registry found @ TCPAddress[('127.0.0.1', 6116)] Jul 21 18:29:22 (pikerd[508ed6], 93453, piker.cli.pikerd.<locals>.main[369728])) [INFO] tractor _root.py:477 Starting actor-runtime for 'pikerd@93453' -> Opening new registry @ TCPAddress[('127.0.0.1', 6116)] Jul 21 18:29:22 (pikerd[508ed6], 93453, tractor.ipc._server._serve_ipc_eps[371136])) [INFO] tractor.ipc _tcp.py:181 Listening on TCP socket [> TCPAddress[('127.0.0.1', 6116)] Jul 21 18:29:28 (pikerd[508ed6], 93453, spawn_brokerd[374304])) [INFO] piker.service _daemon.py:205 Spawning ib broker daemon Jul 21 18:29:29 (brokerd.ib[3065ce], 93474, no task context)) [INFO] tractor _entry.py:131 Starting `trio` subactor from parent @ ('127.0.0.1', 6116) >( |_<Actor( |_id: 'brokerd.ib@93474' |_rpc: 0 active tasks |_state: 'running' task: <dne> loglevel: 'INFO' subactors_spawned: 0 spawn_method: 'trio' )> Jul 21 18:29:29 (brokerd.ib[3065ce], 93474, tractor._runtime.async_main[132576])) [ERROR] tractor.devx _stackscope.py:240 `stackscope` not installed for use in debug mode! `Ignoring enable_stack_on_sig() call! Jul 21 18:29:30 (brokerd.ib[3065ce], 93474, tractor.ipc._server._serve_ipc_eps[695552])) [INFO] tractor.ipc _uds.py:218 Attempting to bind UDS socket >[ |_UDSAddress[(/run/user/1000/tractor, brokerd.ib::3065cebe-da3a-4da7-9d1e-78e2b17d10bc@93474.sock, <unknown-peer-pid>)] Jul 21 18:29:30 (brokerd.ib[3065ce], 93474, tractor.ipc._server._serve_ipc_eps[695552])) [INFO] tractor.ipc _uds.py:237 Listening on UDS socket [> |_UDSAddress[(/run/user/1000/tractor, brokerd.ib::3065cebe-da3a-4da7-9d1e-78e2b17d10bc@93474.sock, <unknown-peer-pid>)] Jul 21 18:29:30 (pikerd[508ed6], 93453, piker.service._mngr.Services.start_service_task.<locals>.open_context_in_task[281312])) [INFO] piker.service _mngr.py:105 `pikerd` service brokerd.ib started with value None Jul 21 18:29:30 (pikerd[508ed6], 93453, piker.service._mngr.Services.start_service_task.<locals>.open_context_in_task[281312])) [WARNING] tractor _context.py:1307 `Context.result()` is DEPRECATED! Use `Context.[no]wait_for_result()` instead! API connection failed: TimeoutError() API connection failed: TimeoutError() API connection failed: TimeoutError() Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [WARNING] piker.brokers.ib api.py:1304 Failed to connect on mini.dover:4000 after 2 tries with 'API connection failed: TimeoutError()' Retrying with a new client id.. Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [ERROR] tractor to_asyncio.py:400 `asyncio` task errored Traceback (most recent call last): File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for return await fut ^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim next_send = outcome.Value((yield next_yield)) ^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients await ib.connectAsync( ...<8 lines>... ) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync await self.client.connectAsync(host, port, clientId, timeout) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync await asyncio.wait_for(self.apiStart, timeout) File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for async with timeouts.timeout(timeout): ~~~~~~~~~~~~~~~~^^^^^^^^^ File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result result: Any = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio async with load_aio_clients( ~~~~~~~~~~~~~~~~^ disconnect_on_exit=False, ^^^^^^^^^^^^^^^^^^^^^^^^^ ) as accts2clients: ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients raise ConnectionError( ...<3 lines>... ) from _err ConnectionError: No ib APIs could be found scanning @: [('mini.dover', 4000)] Check your `brokers.toml` and/or network Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [ERROR] tractor to_asyncio.py:617 `trio`-side reports that the `asyncio`-side errored x)> |_<Task finished name='Task-2' coro=<greenback_shim() done, defined at /home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py:90> exception=ConnectionError("No ib APIs could be found scanning @:\n[('mini.dover', 4000)]\nCheck your `brokers.toml` and/or network")> Traceback (most recent call last): File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for return await fut ^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim next_send = outcome.Value((yield next_yield)) ^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients await ib.connectAsync( ...<8 lines>... ) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync await self.client.connectAsync(host, port, clientId, timeout) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync await asyncio.wait_for(self.apiStart, timeout) File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for async with timeouts.timeout(timeout): ~~~~~~~~~~~~~~~~^^^^^^^^^ File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 530, in signal_trio_when_done res: Any = task.result() ~~~~~~~~~~~^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 116, in greenback_shim return await _greenback_shim(orig_coro, next_send) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 201, in _greenback_shim next_yield, resume_greenlet = resume_greenlet.switch(next_send) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 81, in trampoline next_yield: Any = next_send.send(orig_coro) # type: ignore ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 231, in send return gen.throw(self.error) ~~~~~~~~~^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result result: Any = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio async with load_aio_clients( ~~~~~~~~~~~~~~~~^ disconnect_on_exit=False, ^^^^^^^^^^^^^^^^^^^^^^^^^ ) as accts2clients: ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients raise ConnectionError( ...<3 lines>... ) from _err ConnectionError: No ib APIs could be found scanning @: [('mini.dover', 4000)] Check your `brokers.toml` and/or network Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, no task context)) [CANCEL] tractor to_asyncio.py:647 Cancelling `trio` side due to aio-side src exc No ib APIs could be found scanning @: [('mini.dover', 4000)] Check your `brokers.toml` and/or network (c> |_<Task 'tractor.trionics._mngrs._Cache.run_ctx' at 0x7fa640195020> Exception in callback _run_asyncio_task.<locals>.signal_trio_when_done() at /home/goodboy/repos/tractor/tractor/to_asyncio.py:495 handle: <Handle _run_asyncio_task.<locals>.signal_trio_when_done() at /home/goodboy/repos/tractor/tractor/to_asyncio.py:495> Traceback (most recent call last): File "/usr/lib/python3.13/asyncio/tasks.py", line 507, in wait_for return await fut ^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 210, in _greenback_shim next_send = outcome.Value((yield next_yield)) ^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1264, in load_aio_clients await ib.connectAsync( ...<8 lines>... ) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/ib.py", line 1748, in connectAsync await self.client.connectAsync(host, port, clientId, timeout) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/ib_insync/client.py", line 217, in connectAsync await asyncio.wait_for(self.apiStart, timeout) File "/usr/lib/python3.13/asyncio/tasks.py", line 506, in wait_for async with timeouts.timeout(timeout): ~~~~~~~~~~~~~~~~^^^^^^^^^ File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3.13/asyncio/events.py", line 89, in _run self._context.run(self._callback, *self._args) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 727, in signal_trio_when_done raise aio_err File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 530, in signal_trio_when_done res: Any = task.result() ~~~~~~~~~~~^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 116, in greenback_shim return await _greenback_shim(orig_coro, next_send) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 201, in _greenback_shim next_yield, resume_greenlet = resume_greenlet.switch(next_send) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/greenback/_impl.py", line 81, in trampoline next_yield: Any = next_send.send(orig_coro) # type: ignore ~~~~~~~~~~~~~~^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 231, in send return gen.throw(self.error) ~~~~~~~~~^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 391, in wait_on_coro_final_result result: Any = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1381, in load_clients_for_trio async with load_aio_clients( ~~~~~~~~~~~~~~~~^ disconnect_on_exit=False, ^^^^^^^^^^^^^^^^^^^^^^^^^ ) as accts2clients: ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1351, in load_aio_clients raise ConnectionError( ...<3 lines>... ) from _err ConnectionError: No ib APIs could be found scanning @: [('mini.dover', 4000)] Check your `brokers.toml` and/or network Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_symbol_search[698544])) [ERROR] tractor _rpc.py:845 IPC context terminated after having 'errored' EndOfChannel() Traceback (most recent call last): File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke raise scope_err File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke collapse_eg(), ~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__ await self.gen.athrow(value) File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg raise exc File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke res: Any|PayloadT = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/symbols.py", line 198, in open_symbol_search open_client_proxies() as (proxies, _), ~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies tractor.trionics.maybe_open_context( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ acm_func=tractor.to_asyncio.open_channel_from, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... ) as (cache_hit, (clients, _)), ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 320, in maybe_open_context yielded: Any = await service_n.start( ^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start raise exc.exceptions[0] from None File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 220, in run_ctx async with mng as value: ^^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 1199, in open_channel_from first = await chan.receive() ^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 255, in receive raise err File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 244, in receive return await self._from_aio.receive() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_channel.py", line 362, in receive return await trio.lowlevel.wait_task_rescheduled(abort_fn) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_traps.py", line 208, in wait_task_rescheduled return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 213, in unwrap raise captured_error trio.EndOfChannel scope_error: EndOfChannel() Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [ERROR] tractor _rpc.py:845 IPC context terminated after having 'errored' AssertionError("Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc'))") Traceback (most recent call last): File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke raise scope_err File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke collapse_eg(), ~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__ await self.gen.athrow(value) File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg raise exc File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke res: Any|PayloadT = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/data/feed.py", line 512, in open_feed_bus await bus.nursery.start( ...<13 lines>... ) File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start raise exc.exceptions[0] from None File "/home/goodboy/repos/piker/piker/data/feed.py", line 291, in allocate_persistent_feed ) = await bus.nursery.start( ^^^^^^^^^^^^^^^^^^^^^^^^ ...<11 lines>... ) ^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start raise exc.exceptions[0] from None File "/home/goodboy/repos/piker/piker/brokers/ib/feed.py", line 913, in stream_quotes open_data_client() as proxy, ~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/feed.py", line 114, in open_data_client open_client_proxies() as (proxies, clients), ~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies tractor.trionics.maybe_open_context( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ acm_func=tractor.to_asyncio.open_channel_from, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... ) as (cache_hit, (clients, _)), ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 315, in maybe_open_context assert not resources.get(ctx_key), f'Resource exists? {ctx_key}' ^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc')) scope_error: AssertionError("Resource exists? (140352467083456, ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc'))") Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [WARNING] tractor.devx _trace.py:416 Already an ongoing repl request? |_<Task 'tractor.devx.debug._tty_lock.request_root_stdio_lock' at 0x7fa640195b20> REPL task is |_None Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_feed_bus[700128])) [WARNING] tractor.devx _trace.py:463 Local task already has active debug request |_<Task 'open_feed_bus' at 0x7fa6401952e0> Waiting for previous request to complete.. Jul 21 18:30:00 (brokerd.ib[3065ce], 93474, open_symbol_search[698544])) [PDB] tractor.devx _post_mortem.py:172 Opening a pdb REPL in crashed actor x>( |_ <Task 'open_symbol_search' at 0x7fa640194cb0> @ ('brokerd.ib', '3065cebe-da3a-4da7-9d1e-78e2b17d10bc') Traceback (most recent call last): File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 285, in _errors_relayed_via_ipc yield # run RPC invoke body ^^^^^ File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 804, in _invoke raise scope_err File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 645, in _invoke collapse_eg(), ~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 235, in __aexit__ await self.gen.athrow(value) File "/home/goodboy/repos/tractor/tractor/trionics/_beg.py", line 67, in collapse_eg raise exc File "/home/goodboy/repos/tractor/tractor/_rpc.py", line 666, in _invoke res: Any|PayloadT = await coro ^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/symbols.py", line 198, in open_symbol_search open_client_proxies() as (proxies, _), ~~~~~~~~~~~~~~~~~~~^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/piker/brokers/ib/api.py", line 1397, in open_client_proxies tractor.trionics.maybe_open_context( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ acm_func=tractor.to_asyncio.open_channel_from, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<5 lines>... ) as (cache_hit, (clients, _)), ^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 320, in maybe_open_context yielded: Any = await service_n.start( ^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_run.py", line 1377, in start raise exc.exceptions[0] from None File "/home/goodboy/repos/tractor/tractor/trionics/_mngrs.py", line 220, in run_ctx async with mng as value: ^^^ File "/usr/lib/python3.13/contextlib.py", line 214, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 1199, in open_channel_from first = await chan.receive() ^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 255, in receive raise err File "/home/goodboy/repos/tractor/tractor/to_asyncio.py", line 244, in receive return await self._from_aio.receive() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_channel.py", line 362, in receive return await trio.lowlevel.wait_task_rescheduled(abort_fn) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/trio/_core/_traps.py", line 208, in wait_task_rescheduled return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py", line 213, in unwrap raise captured_error trio.EndOfChannel [22] > /home/goodboy/repos/piker/py313/lib/python3.13/site-packages/outcome/_impl.py(213)unwrap() -> raise captured_error (Pdb+) ```
goodboy self-assigned this 2025-07-21 22:54:36 +00:00
goodboy changed title from ibrk: await chan.receive() shrouds connection timeout? to ibrk: `await chan.receive()` shrouds ib-gw connection timeout? 2025-07-21 22:54:57 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
alt_tpts_for_perf
port_to_latest_tractor
bump_polars
decimal_prices_thru_ems
mp_fomo_polish
max_pain_storage
max_pain_chart
max_pain_deribit
deribit_fix
brokers_refinery
accounting_refinery
add_visidata
no_symcache_no_problem
service_mng_to_tractor
ems_no_last_required
stop_is_oec
gitea_feats
jsonrpc_err_in_rent
tsp_gaps
max_pain_chart_backup
subsys_refinery
subsys_refinery_BACKUP
max_pain_deribit_backup
add_ruff_linter
deribit_fix_backup
kucoin_and_binance_fix
uv_migration
kucoin_and_binance_BACKUP
go_httpx_no_unrelated_binance_stuff
go_httpx
go_httpx_orig_BACKUP
ib_refinements
fix_deribit_hist_queries_NEW
fix_deribit_hist_queries
prev_deribit_fix
nix_shell_env_fix
cherry_to_storage_pr
fix_deribit_hist_queries_BACKUP
kucoin_and_binance_fix_goodboyBACKUP
nix-headless-fix
go_httpx_binance
fix-binance-venues
pyqt6
distribute_dis
runtime_to_msgspec
nix-monkeys-fix
ib_py311_fixes
poetry2nix
account_tests
py311_ib_fix
master
basic_buy_bot
marketstore_disable_snappy
rekt_pps
py311
kucoin_backend
small_kucoin_fixes
pre_overruns_ctxcancelled
emit_clear_ticks_only_on_ts_change
binance_ws_ep_update
log_linearized_curve_overlays
xdotool_fixes
deribit_updates
storage_middleware_layer
service_subpkg
pps_precision_hotfixes
service_subpkg_backup
storage_cli
xdo_and_you
decimalization_take_2
backward_compat_trans_with_symbolinfo
explicit_write_pps_on_exit
backend_spec
paper_trade_improvements_rebase
loglevel_to_testpikerd
overlays_interaction_latency_tuning
kraken_deposits_fixes
l1_compaction
epoch_indexing_and_dataviz_layer
multichartz
axis_sticky_api
multi_symbol_input
update_qt_screen_info_script
fsps_and_flumes
epoch_index
pregraphics_formatters
multichartz_backup
samplerd_service
misc_brokerd_backend_repairs
pre_viz_calls
dark_clearing_improvements
dark_clearing_repairs
epoch_index_backup
agg_feedz
kraken_limits_fields
cz_post_ftx
kraken_pair_status
ib_contract_updates
pre_multifeed_hotfix
kraken_pair_updates
daemon_sockaddr_config
no_signal_pi_overlays
pre_multi_feedz
pg_exts_fork
ib_1m_hist
even_moar_kraken_order_fixes
ci_fix_tractor_testing
clears_table_events
offline_dark_clearing
multi_client_order_mgt
pin_tractor_main
history_view
tractor_core_port
kraken_fill_bugs
paper_clear_logics_fix
kraken_nameerr_fix
livenpaper
size_in_shm_token
deribit
asycvnc_pin_bump
live_n_paper
open_order_loading
doin_the_splits
dict_differ
msgpack_zombie
the_ems_flattening
kraken_ws_orders
kraken_userref_hackzin
pydantic_zombie
ib_pps_upgrade
multisympaper
ppu_history
basic_pp_audit
historical_breakeven_pp_price
ib_native_data_hack
pptables
ahab_you_bad_boi
fix_forex
fsp_shm_caching
paper_eng_msg_fixes
drop_pydantic
krakenwsbackup
dpbackup
null_last_quote_fix
ahab_hardkill
tractor_typed_msg_hackin
kraken_ledger_pps
ib_rt_pp_update_hotfix
notokeninwswrapper
pps_postmortem
lifo_pps_ib
310_plus
mxmn_from_m4
contain_mkts
slic_fix_v2
uppx_slice_fix
update_last_datums_in_view
ib_subpkg
flexxin
ib_checker_hackz
incremental_update_paths
ib_dedicated_data_client
dockerize_ib_gw
pre_flow
drop_pandas
no_ib_pps
trimeter_dep
l1_precision_fix
marketstore
no_orderid_in_error
incr_update_backup
big_data_lines
m4_corrections
offline_history_loading
drop_arrow_add_predulum
marketstore_backup
broker_bumpz
no_git_prot_w_pip
kraken_editorder
fqsns
kraken_cleaning
kraken_orders
only_draw_iv_for_ohlc
mkts_backup
pp_bar_fixes
async_hist_loading
kraken_gb
windows_fixes_yo
ib_mkt_closed
py3.10_support
gb_kraken_orders
dark_vlm
overlayed_dvlm
fsp_ui_mod
vlm_plotz
plotitem_overlays
misc_backend_fixes
dolla_vlm
fspd_cluster
misc_ib_updates
single_display_update_loop
basic_vlm_display
pane_sizing_fixes
tractor_clustering
msgpack_no_sets_allowed
tinas_unite
simpler_quote_throttle_logic
fast_step_curve
win_fixes
fsp_hotfixes
teardown_guesmost_via_cs
windows_testing_volume
fsp_drunken_alignment
vlm_plotz_backup
fsp_feeds
chart_mod_breakup
pause_feeds_on_sym_switch
brokers_config
ordermodepps_backup
readme_bump_zone
ci_on_forks
asyncify_input_modes
minimal_brokerd_trade_dialogues
wait_on_daemon_portals
backup_asyncify_input_modes
naive_feed_throttling
window_cuckery
msgspec_fixes
status_bar
ems_hotfixes
ems_to_bidir_streaming
web_utils
symbol_search
binance_syminfo_and_mintick
update_throttling
syseng_tweaks
ems_tweaks
py3.9
binance_aggtrades_and_ohlc_parsing
binance_backend
tractor_open_stream_from
feed_fixes
order_mode_finesse
cached_feeds
readme_bumpz
supervise
basic_orders
kraken_trades_data
basic_alerts
kraken_history
y_zoom
chart_trader
graphics_pixel_buffer
vwap_fsp
to_qpainterpath_and_beyond
tina_free_vwap
vwap_backup
bar_select
ib_backend
unleash_the_kraken
facepalm
marketstore_integration
its_happening
relicense_as_agplv3
questrade_candles
use_tractor_logging
questrade_conns
kivy_mainline_and_py3.8
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
goodboy
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#46
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?