Add initial repl_fixture support, enter/exit hooks around the debugger sys! #28

goodboy · 2025-05-12T04:47:28Z

goodboy commented

2025-05-12 04:47:28 +00:00

It turns out to be fairly useful to allow hooking into a given actor’s entry-and-exit around .devx._debug._pause/._post_mortem() calls which engage the pdbp.Pdb REPL (really our ._debug.PdbREPL but yeah).

Some very handy use cases include,

swapping out-of-band (config) state that may otherwise halt the user’s app since the actor normally handles kb&mouse input, in thread, which means that the handler will be blocked while the REPL is in use.
(remotely) reporting actor-runtime state for monitoring purposes around crashes or pauses in normal operation.
allowing for crash-handling to be hard-disabled via ._state._runtime_vars say for when you never want a debugger to be entered in a production instance where you’re not-sure-if/don’t-want per-actor debug_mode: bool settings to always be unset, say bc you’re still debugging some edge cases that ow you’d normally want to REPL up.

Impl deats for `repl_fxture`,

add a new optional ._state._runtime_vars['repl_fixture'] field which for now can be manually set; i saw no reason for a formal API yet since we want to convert the dict to a struct anyway (first).
augment both .devx._debug._pause()/._post_mortem() with a new optional repl_fixture: AbstractContextManager[bool] kwarg which when provided is opened (effectively ) as with repl_fixture() around the low-level-sync PdbREPL interaction calls; if the enter-result, an expected bool, is False then the interaction is hard-bypassed, the repl_fixture.__exit__() is later invoked by DebugStatus.release() as per the impl deats:
- ~~a new @cm helper to contain the activation logic,~~ ~~.devx.debug._post_mortem._maybe_open_repl_fixture() which:~~ ~~- when the new runtime-var repl_fixture is overridden, (only~~ ~~possible manually rn) use it but only whenever the~~ ~~repl_fixture`-kwarg is left null.~~
- ~~for the ._pause() case the @cm is opened around the entire body~~ ~~of the embedded _enter_repl_sync() closure (for now though,~~ ~~since ideally longer term this entire routine is factored to~~ ~~be a lot less “nested”)~~
- ~~for _post_mortem() the entire fn body is embedded and now also~~ ~~excepts an optional boxed_maybe_exc: BoxedMaybeException; it is~~ ~~only passed in the open_crash_handler() caller case.~~
- all of the ^^ above was replaced by adding a new DebugStatus.maybe_enter_repl_fixture() which internally uses a ._fiture_stack: ExitStack that is closed from .release() allowing for extended support from the .pause_from_sync() API since the user’s repl_fixture.__exit__() can be invoked an arbitrary time later once PdbREPL interaction is complete!

add a BoxedMaybeException.pformat() = __repr__() which when a .value: Exception is set renders a more “objecty” repr of the exc.

Reorg of `.devx._debug` to new `.devx.debug` sub-pkg,

split up the current .devx._debug components into various .debug.* submods, like:
- [maybe_]open_crash_handler() and friends into a _post_mortem.
- all the pdbp specific customizations into a ._repl.
- all the root actor TTY locking into its own ._tty_lock.
- all pausing/pdb.set_trace()-like apis into a ._trace
- our shielding SIGINT handler into ._sigint
- various (unrefined and higher level) syncing apis which parts of the runtime use mostly to avoid REPL/TTY clobbering in a ._sync
move .devx.debug.hide_runtime_frames() -> .devx._frame_stack

Tests

as repl_fixture input to sync code’s use of [maybe_]_open_crash_handler() (which is currently implemented the same as the async APIs since async def post_mortem() just calls def _post_mortem())
- the same but using the global rtvars override
  - which we should add an pub api for.. question is now or later?
tests for all async uses which are already implemented via the augmentation of _pause()’s internal _enter_repl_sync() closure.
- from root actor.
- from any (nested) sub.
- from an infected aio task?
support for pause_from_sync()/breakpoint() ??
- we need to break up the provided repl_fixture().__enter__() and .__exit__() calls with the latter invoked as part of the PdbREPL per method calls to DebugaStatus.release() to ensure the fixture’s teardown isn’t invoked until after the user releases their repl interaction.
  - likely impl will be to add an DebugStatus._fixture_stack: ExitStack and have every _pause() call .enter_context() prior to interaction and then have DebugStatus/PdbREPL call .fixture_stack.close() from .set_continue/quit()?
support for an @acm impls of replx_fixture?
- obvi this can only be supported from already async APIs or we have to somehow schedule the async code in the trio/runtime thread, so hardest to do for sync-code crashes/pauses.

Surrounding changes,

fix the stackscope-using shield-pause test by adding typing_extenions sub-dep it needs.
maybe we can also include some (stringent) tests around similar usage of the undocumented Actor.lifetime_stack: ExitStack ??

It turns out to be fairly useful to allow hooking into a given actor's entry-and-exit around `.devx._debug._pause/._post_mortem()` calls which engage the `pdbp.Pdb` REPL (really our `._debug.PdbREPL` but yeah). Some very handy use cases include, - swapping out-of-band (config) state that may otherwise halt the user's app since the actor normally handles kb&mouse input, in thread, which means that the handler will be blocked while the REPL is in use. - (remotely) reporting actor-runtime state for monitoring purposes around crashes or pauses in normal operation. - allowing for crash-handling to be hard-disabled via `._state._runtime_vars` say for when you never want a debugger to be entered in a production instance where you're not-sure-if/don't-want per-actor `debug_mode: bool` settings to always be unset, say bc you're still debugging some edge cases that ow you'd normally want to REPL up. --- #### Impl deats for `repl_fxture`, - add a new optional `._state._runtime_vars['repl_fixture']` field which for now can be manually set; i saw no reason for a formal API yet since we want to convert the `dict` to a struct anyway (first). - [x] augment both `.devx._debug._pause()/._post_mortem()` with a new optional `repl_fixture: AbstractContextManager[bool]` kwarg which when provided is opened (effectively ) as `with repl_fixture()` around the low-level-sync `PdbREPL` interaction calls; if the enter-result, an expected `bool`, is `False` then the interaction is hard-bypassed, the `repl_fixture.__exit__()` is later invoked by `DebugStatus.release()` as per the impl deats: * ~~a new `@cm` helper to contain the activation logic,~~ ~~`.devx.debug._post_mortem._maybe_open_repl_fixture()` which:~~ ~~- when the new runtime-var `repl_fixture` is overridden, (only~~ ~~possible manually rn) use it but only whenever the~~ ~~repl_fixture`-kwarg is left null.~~ * ~~for the `._pause()` case the `@cm` is opened around the entire body~~ ~~of the embedded `_enter_repl_sync()` closure (for now though,~~ ~~since ideally longer term this entire routine is factored to~~ ~~be a lot less "nested")~~ * ~~for `_post_mortem()` the entire fn body is embedded and now also~~ ~~excepts an optional `boxed_maybe_exc: BoxedMaybeException`; it is~~ ~~only passed in the `open_crash_handler()` caller case.~~ * **all of the ^^ above was replaced by** adding a new `DebugStatus.maybe_enter_repl_fixture()` which internally uses a `._fiture_stack: ExitStack` that is closed from `.release()` allowing for extended support from the `.pause_from_sync()` API since the user's `repl_fixture.__exit__()` can be invoked an arbitrary time later once `PdbREPL` interaction is complete! #### Other related refinements, - add a `BoxedMaybeException.pformat() = __repr__()` which when a `.value: Exception` is set renders a more "objecty" repr of the exc. --- #### Reorg of `.devx._debug` to new `.devx.debug` sub-pkg, - [x] split up the current `.devx._debug` components into various `.debug.*` submods, like: - [x] `[maybe_]open_crash_handler()` and friends into a `_post_mortem`. - [x] all the `pdbp` specific customizations into a `._repl`. - [x] all the root actor TTY locking into its own `._tty_lock`. - [x] all pausing/`pdb.set_trace()`-like apis into a `._trace` - [x] our shielding SIGINT handler into `._sigint` - [x] various (unrefined and higher level) syncing apis which parts of the runtime use mostly to avoid REPL/TTY clobbering in a `._sync` - [x] move `.devx.debug.hide_runtime_frames()` -> `.devx._frame_stack` --- #### Tests - [ ] as `repl_fixture` input to sync code's use of `[maybe_]_open_crash_handler()` (which is currently implemented the same as the async APIs since `async def post_mortem()` just calls `def _post_mortem()`) - [ ] the same but using the global rtvars override - [ ] which we should add an pub api for.. question is now or later? - [ ] tests for all async uses which are already implemented via the augmentation of `_pause()`'s internal `_enter_repl_sync()` closure. - [ ] from root actor. - [ ] from any (nested) sub. - [ ] from an infected aio task? - support for `pause_from_sync()`/`breakpoint()` ?? - [ ] we need to break up the provided `repl_fixture().__enter__()` and `.__exit__()` calls with the latter invoked as part of the `PdbREPL` per method calls to `DebugaStatus.release()` to ensure the fixture's teardown isn't invoked until **after** the user releases their repl interaction. - [ ] likely impl will be to add an `DebugStatus._fixture_stack: ExitStack` and have every `_pause()` call `.enter_context()` prior to interaction and then have `DebugStatus`/`PdbREPL` call `.fixture_stack.close()` from `.set_continue/quit()`? - [ ] support for an `@acm` impls of `replx_fixture`? - obvi this can *only* be supported from already async APIs or we have to somehow schedule the async code in the trio/runtime thread, so hardest to do for sync-code crashes/pauses. Surrounding changes, - [x] fix the `stackscope`-using shield-pause test by adding `typing_extenions` sub-dep it needs. - [ ] maybe we can also include some (stringent) tests around similar usage of the undocumented `Actor.lifetime_stack: ExitStack` ??

goodboy added 1 commit 2025-05-12 04:47:29 +00:00

f604c8836d Add initial `repl_fixture` support B)

It turns out to be fairly useful to allow hooking into a given actor's
entry-and-exit around `.devx._debug._pause/._post_mortem()` calls which
engage the `pdbp.Pdb` REPL (really our `._debug.PdbREPL` but yeah).

Some very handy use cases include,
- swapping out-of-band (config) state that may otherwise halt the
  user's app since the actor normally handles kb&mouse input, in thread,
  which means that the handler will be blocked while the REPL is in use.
- (remotely) reporting actor-runtime state for monitoring purposes
  around crashes or pauses in normal operation.
- allowing for crash-handling to be hard-disabled via
  `._state._runtime_vars` say for when you never want a debugger to be
  entered in a production instance where you're not-sure-if/don't-want
  per-actor `debug_mode: bool` settings to always be unset, say bc
  you're still debugging some edge cases that ow you'd normally want to
  REPL up.

Impl details,
- add a new optional `._state._runtime_vars['repl_fixture']` field which
  for now can be manually set; i saw no reason for a formal API yet
  since we want to convert the `dict` to a struct anyway (first).
- augment both `.devx._debug._pause()/._post_mortem()` with a new
  optional `repl_fixture: AbstractContextManager[bool]` kwarg which
  when provided is `with repl_fixture()` opened around the lowlevel
  REPL interaction calls; if the enter-result, an expected `bool`, is
  `False` then the interaction is hard-bypassed.
  * for the `._pause()` case the `@cm` is opened around the entire body
    of the embedded `_enter_repl_sync()` closure (for now) though
    ideally longer term this entire routine is factored to be a lot less
    "nested" Bp
  * in `_post_mortem()` the entire previous body is wrapped similarly
    and also now excepts an optional `boxed_maybe_exc: BoxedMaybeException`
    only passed in the `open_crash_handler()` caller case.
- when the new runtime-var is overridden, (only manually atm) it is used
  instead but only whenever the above `repl_fixture` kwarg is left null.
- add a `BoxedMaybeException.pformat() = __repr__()` which when
  a `.value: Exception` is set renders a more "objecty" repr of the exc.

Obviously tests for all this should be coming soon!

goodboy referenced this issue from a commit

2025-05-12 17:13:54 +00:00

Impl a `repl_fixture` that reverts to uler-config

goodboy referenced this issue from a commit

2025-05-12 17:14:11 +00:00

Impl a `repl_fixture` that reverts to uler-config

goodboy added 1 commit 2025-05-13 00:43:10 +00:00

09a61dbd8a Add exc suppression to `open_crash_handler()`

By supporting a new optional param to `open_crash_handler()`,
`raise_on_exit: bool|Sequence[Type[BaseException]] = True` which
determines whether, after the REPL interaction completes, the handled
exception is raised upward. This is **very** handy for writing bits of
"debug-able but resilient code" as is the case in (many) dependent
projects/apps.

Impl,
- `raise_on_exit` can be a `bool` or (set) sequence of types which will
  always be raised.
- also add a `BoxedMaybeException.raise_on_exit` equiv which (for now)
  we check matches (in case down the road we want to offer dynamic ctls).
- rename both crash-handler cm's `tb_hide` -> `hide_tb`.

goodboy added 8 commits 2025-05-15 19:00:30 +00:00

18ae7b0048 Mk `.devx._debug` a sub-pkg `.devx.debug`

With plans for much factoring of the original module into sub-mods!
Adjust all imports and refs throughout to match.

712450869b Add `_maybe_open_repl_fixture()`

Factoring the (basically duplicate) content from both use spots into
a common `@cm` which delivers a `bool` signalling whether the REPL
should be engaged. Fixes a lingering bug with `nullcontext()` calling
btw..

cb31a330b3 Start splitting into `devx.debug.` sub-mods

From what was originall the `.devx._debug` monolith module, since that
file was way out of ctl in terms of LoC!

New modules so far include,
- ._repl: our `pdb[p]` ext type/lowlevel-APIs and `mk_pdb()` factory.
- ._sigint: just our REPL-interaction shield-handler.
- ._tty_lock: containing all the root-actor TTY mutex machinery
  including the `Lock`/`DebugStatus` primitives/APIs as well as the
  inter-tree IPC context eps:
  * the server-side `lock_stdio_for_peer()` which pairs with the,
  * client-(subactor)-side `request_root_stdio_lock()` via the,
  * pld-msg-spec of `LockStatus/LockRelease`.
  AND the `any_connected_locker_child()` predicate.

1012581a0b Enable new `tractor.devx.debug._tty_lock` in the root

006ee0752c Be explicit with `SpawnSpec` processing in subs

As per the outstanding TODO just above the redic `setattr()` loop in
`Actor._from_parent()`!!

Instead of all that risk-ay monkeying, add detailed comment-sections
around each explicit assignment of each `SpawnSpec` field, including
those that were already being explicitly set.

Those and other deats,
- ONLY enable the `.devx.debug._tty_lock` module from `Actor.__init__()`
  in the root actor.
- add a new `get_mod_nsps2fps()` to replace the loop in init and assign
  the initial `.enable_modules: dict[str, str]` from it.
- do `self.enable_modules.update(spawnspec.enable_modules)` instead of
  an overwrite and assert the table is by default empty in all
  subs.

5024e71d8e Return `Path` from `_get_mod_abspath()` helper fn

69267ae656 Mv `.hide_runtime_frames()` -> `.devx._frame_stack`

A much more relevant module for a call-stack-frame hider ;)

f1e9926b79 Reorg `.devx.debug` into sub-mods!

Which cleans out the pkg-mod to just the expected exports with (its
longstanding todo comment list) and thus a separation-of-concerns
and smaller mod-file sizes via the following new sub-mods:
- `._trace` for the `.pause()`/`breakpoint()`/`pdb.set_trace()`-style
  APIs including all sync-caller variants.
- `._post_mortem` to contain our async `.post_mortem()` and all other
  public crash handling APIs for use from sync callers.
- `._sync` for the high-level syncing helper-routines used throughout the
  runtime to avoid multi-proc TTY use collisions.

And also,
- remove `hide_runtime_frames()` since moved to `.devx._frame_stack`.

goodboy added 1 commit 2025-05-15 19:20:20 +00:00

4fca7317ea Add `typing_extensions`, it's a sub-dep of `stackscope`?

Oddly my env was borked bc this (apparently missed by `uv`?) sub-dep
wasn't installed and then `stackscope` was silently failing import and
caused the shield-pause test to also fail (since it couldn't match the
expected `log.devx()` on console). The import failure is not very
explanatory due to the `log.warning()`; change it to `.error()` level.

Also, explicitly import `_sync_pause_from_builtin` in
`examples/debugging/restore_builtin_breakpoint.py` to ensure the ref is
exported properly from `.devx.debug` (which it wasn't during dev of the
prior commit Bp).

goodboy force-pushed repl_fixture from 4fca7317ea to d716c57234

2025-05-16 04:25:36 +00:00

Compare

goodboy added 1 commit 2025-05-18 16:47:41 +00:00

2a37f6abdd Re-impl as `DebugStatus.maybe_enter_repl_fixture()`

Dropping the `_maybe_open_repl_fixture()` approach and instead using
a `DebugStatus._fixture_stack = ExitStack()` which provides for much
simpler support around both sync and async pausing APIs thanks to only
invoking `repl_fixture.__exit__()` on actual `PdbREPL` interaction being
complete!

Deats,
- all `repl_fixture` detection logic still happens in one place (the new
  method) but we aren't limited to closing it via an immediate post REPL
  `.__exit__()` call which instead is triggered by,
- `DebugStatus.release()` which now calls `._fixture_stack.close()` and
  thus only invokes `repl_fixture.__exit__()` when user REPL-ing is
  **actually complete** an arbitrary amount of debugging time later.
- include the notes for `@acm` support above the new method, though not
  sure if they're as relevant any more?

Benefits,
- we can drop the previously added indent levels from
  `_enter_repl_sync()` and `_post_mortem()`.
- now we automatically have support for the `.pause_from_sync()` API
  since `_enter_repl_sync()` doesn't close the prior
  `_maybe_open_repl_fixture()` immediately when `debug_func=None`; the
  user's `__exit__()` is only ever called once `.release()` is.

Other,
- add big 'CASE' comments around the various blocks in
  `.pause_from_sync()`, i was having trouble figuring out which i was
  using from a `breakpoint()` in a dependent app..

goodboy added 9 commits 2025-06-12 00:00:48 +00:00

a7bbcdbeb8 Log listener bind status for TCP as for UDS

082d373947 Decouple actor-state from low-level ipc-server

As much as is possible given we currently do some graceful
cancellation join-waiting on any connected sub-actors whenever an active
`local_nursery: AcrtorNursery` in the post-rpc teardown sequence of
`handle_stream_from_peer()` is detected. In such cases we try to allow
the higher level inter-actor (task) context(s) to fully cancelled-ack
before conducting IPC machinery shutdown.

The main immediate motivation for all this is to support unit testing
the `.ipc._server` APIs but in the future may be useful for anyone
wanting to use our modular IPC transport layer sin-"actors".

Impl deats,
- drop passing an `actor: Actor` ref from as many routines in
  `.ipc._server` as possible instead opting to use
  `._state.current_actor()` where abs needed; thus the fns dropping an
  `actor` input param are:
  - `open_ipc_server()`
  - `IPCServer.listen_on()`
  - `._serve_ipc_eps()`
  - `.handle_stream_from_peer()`
- factor the above mentioned graceful remote-cancel-ack waiting into
  a new `maybe_wait_on_canced_subs()` which is called from
  `handle_stream_from_peer()` and delivers a
  maybe-`local_nursery: ActorNursery` for downstream logic; it's this
  new fn which primarily still needs to call `current_actor()`.
- in `handle_stream_from_peer()` also use `current_actor()` to check if
  a handshake is needed (or if it was called as part of some
  actor-runtime-less operation like our unit test suite!).
- also don't pass an `actor` to `._rpc.process_messages()` see how-n-why
  below..

Surrounding ipc-server client/caller adjustments,
- `._rpc.process_messages()` no longer takes an `actor` input and
  now calls `current_actor()` instead.
- `._portal.open_portal()` is adjusted to ^.
- `._runtime.async_main()` is adjusted to the `.ipc._server`'s removal
  of `actor` ref passing.

Also,
- drop some server `log.info()`s to `.runtime()`

3d6a194669 Start a very basic ipc-server unit test suite

For now it just boots a server, parametrized over all tpt-protos, sin
any actor runtime bootup. Obvi the future todo is ensuring it all works
with a client connecting via the equivalent lowlevel
`.ipc._chan._connect_chan()` API(s).

87049b991f Type annot the `testdir` fixture

6fd08c6e32 Type alias our `pexpect.spawn()` closure fixture

Such that we can more easily annotate any consumer test's of our
`.tests.devx.conftest.spawn()` fixture which delivers a closure which, when
called in a test fn body, transitively sub-invokes:
`pytest.Pytester.spawn()` -> `pexpect.spawn()`

IMO Expecting `Callable[[str], pexpect.pty_spawn.spawn]]` to be used all
over is a bit too.. verbose?

37377e8220 Prevent `test_breakpoint_hook_restored` subproc hangs

If the underlying example script fails (say due to a console output
pattern-mismatch, `AssertionError`) the `pexpect` managed subproc with
a `debug_mode=True` crash-handling-REPL engaged will ofc *not terminate*
due to any SIGINT sent by the test harnesss (since we shield from it as
part of normal sub-actor debugger operation). So instead always send
a 'continue' cmd to the active `PdbREPL`'s stdin so it deactivates and
allows the py-script-process to raise and terminate, unblocking the
`pexpect.spawn`'s internal subproc joiner (which would otherwise hang
without manual intervention, blocking downstream tests..).

Also, use the new `PexpectSpawner` type alias after actually importing
future annots.. XD

304117b4e1 Disable `greenback` sync fn breakpointing by def

Opting for performance over broad multi-actor "debug-ability" from
sync-function-contexts when `debug_mode=True` is set;
IOW prefer no behind-the-scenes `greenlet` perf impact over being
able to use an actor-safe `breakpoint()` wherever as per,
https://greenback.readthedocs.io/en/latest/principle.html#performance

Adjust the breakpoint restore ex script to match.

b3635aaf6d Drop `.to_asyncio`s usage-of-`greenback`-reporting to `log.devx()`

99c5d05aa6 Drop "

" from tail of `BoxedMaybeException.pformat()`

goodboy added 1 commit 2025-06-12 00:39:32 +00:00

24d21e7463 Bump lock file for new 3.13 wheels/schema

Buncha either new AOTc lib whls and they added an `upload-time` field.

goodboy force-pushed repl_fixture from 24d21e7463 to 7b05547fcc

2025-06-12 02:09:21 +00:00

Compare

goodboy force-pushed repl_fixture from 7b05547fcc to 490a4cf8c8

2025-07-13 20:53:33 +00:00

Compare

goodboy commented

2025-07-15 20:35:15 +00:00

Poster

This was already landed on main branch (already synced here as well) on GH,

https://github.com/goodboy/tractor/pull/385

Follow testing todo in,

https://github.com/goodboy/tractor/issues/386

This was already landed on `main` branch (already synced here as well) on GH, https://github.com/goodboy/tractor/pull/385 Follow testing todo in, https://github.com/goodboy/tractor/issues/386

goodboy closed this pull request

2025-07-15 20:35:15 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

No reviewers

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: goodboy/tractor#28

Add initial repl_fixture support, enter/exit hooks around the debugger sys! #28

Impl deats for repl_fxture,

Other related refinements,

Reorg of .devx._debug to new .devx.debug sub-pkg,

Tests

Pull request closed

Add initial `repl_fixture` support, enter/exit hooks around the debugger sys! #28

Impl deats for `repl_fxture`,

Reorg of `.devx._debug` to new `.devx.debug` sub-pkg,