Compare commits
2 Commits
9201a2ed53
...
e617b498ec
| Author | SHA1 | Date |
|---|---|---|
|
|
e617b498ec | |
|
|
b14bc8e273 |
|
|
@ -229,6 +229,53 @@ Gate (trio backend, all 0-failure):
|
||||||
completing within an 800s bound — but split across the
|
completing within an 800s bound — but split across the
|
||||||
above three runs EVERY module passed under step B.
|
above three runs EVERY module passed under step B.
|
||||||
|
|
||||||
|
## Step-B2 outcome (2026-07-02, done in tree)
|
||||||
|
|
||||||
|
Step B committed as `9201a2ed` (code) + `d2e812fb` (docs), then
|
||||||
|
branched to `drop_ria_nursery`. Step B2 (the deferred
|
||||||
|
handler-merge) implemented on top (uncommitted):
|
||||||
|
|
||||||
|
- the two nested handlers in
|
||||||
|
`_open_and_supervise_one_cancels_all_nursery` collapse to
|
||||||
|
ONE `except BaseException as _scope_err` + the existing
|
||||||
|
`finally`. The `outer_err`/`inner_err` locals go away.
|
||||||
|
|
||||||
|
Why it's safe (trace, not hope): the OLD inner handler records
|
||||||
|
`errors[actor.aid.uid]` as its FIRST statement (before any
|
||||||
|
await). So whenever an error path runs, `errors` is non-empty.
|
||||||
|
The OLD outer handler was only reachable via leakage from the
|
||||||
|
inner handler (it catches `BaseException`, so nothing from the
|
||||||
|
`yield` scope bypasses it) — and by then `errors` is already
|
||||||
|
populated, so the `finally`'s `raise` from `errors` ALWAYS
|
||||||
|
superseded the outer handler's own `raise`. i.e. the outer
|
||||||
|
`raise` was dead. The outer handler's other effects
|
||||||
|
(`_scope_error`, a 2nd debugger-wait, child-cancel) are
|
||||||
|
redundant with the merged handler + `finally`. So one handler
|
||||||
|
+ `finally` is observably equivalent.
|
||||||
|
|
||||||
|
Residual nuance (accepted): in the rare "`trio.Cancelled`
|
||||||
|
delivered during the non-shielded `maybe_wait_for_debugger`"
|
||||||
|
path, the merged form may leave `_cancel_called` False (cancel
|
||||||
|
happens after the wait), so `open_nursery`'s tb-hiding guard
|
||||||
|
(`not cancel_called and _scope_error`) can show a tb it
|
||||||
|
previously hid. More informative, not less; no test asserts on
|
||||||
|
it.
|
||||||
|
|
||||||
|
Gate (box ran ~2.7x slow this session, load-induced
|
||||||
|
`TooSlowError` flakiness on timing tests — NOT code; see
|
||||||
|
[[env_cpu_throttle_masquerades_as_regression]]):
|
||||||
|
- baseline (pre-B2 tip `9201a2ed`) full suite
|
||||||
|
(`-k 'not dynamic_pub_sub'`) = 300 passed + 1
|
||||||
|
`test_ext_types_over_ipc` `TooSlowError` that passes 6/6 in
|
||||||
|
isolation (4.89s).
|
||||||
|
- B2 error/cancel gate (`test_cancellation remote_exc_relay
|
||||||
|
inter_peer_cancellation advanced_faults oob_cancellation
|
||||||
|
to_actor spawning local rpc`) = 71 passed, 1 xfailed
|
||||||
|
(125s).
|
||||||
|
- B2 full-suite run: see `b2_full.log` (result appended on
|
||||||
|
completion). RECOMMEND a clean full-suite run on a
|
||||||
|
normal-speed box before this merges.
|
||||||
|
|
||||||
## Verification gate
|
## Verification gate
|
||||||
|
|
||||||
- `tests/test_cancellation.py test_spawning.py test_local.py
|
- `tests/test_cancellation.py test_spawning.py test_local.py
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,68 @@
|
||||||
|
---
|
||||||
|
model: claude-opus-4-8
|
||||||
|
service: claude
|
||||||
|
session: c7860302-6a33-4d92-821c-0807d68be464
|
||||||
|
timestamp: 2026-07-02T22:25:44Z
|
||||||
|
git_ref: 9201a2ed
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260702T222544Z_9201a2ed_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continuing on branch `drop_ria_nursery` after step B
|
||||||
|
(`9201a2ed`). User green-lit the deferred error-handler merge:
|
||||||
|
|
||||||
|
> yes, i checked us out on a new branch for the ria-tn removal;
|
||||||
|
> work off the new wkt branch drop_ria_nursery.
|
||||||
|
|
||||||
|
and (re the migration) "nope, later we'll do this", plus:
|
||||||
|
|
||||||
|
> do ensure the test suite is green in the current state and
|
||||||
|
> those going forward ya?
|
||||||
|
|
||||||
|
and the standing reminder: no commits without a human ack.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Step B2 of the `._ria_nursery` removal (issue #477): merge the
|
||||||
|
two nested error handlers in
|
||||||
|
`_open_and_supervise_one_cancels_all_nursery` into one, now
|
||||||
|
that the 2ndary nursery (whose `__aexit__` the outer handler
|
||||||
|
originally guarded) is gone.
|
||||||
|
|
||||||
|
Trace-driven justification (see
|
||||||
|
`ai/conc-anal/ria_nursery_removal_plan.md` Step-B2 section):
|
||||||
|
the old inner handler records `errors[uid]` as its FIRST
|
||||||
|
action, so `errors` is always non-empty by the time control
|
||||||
|
could reach the outer handler — making the outer handler's
|
||||||
|
`raise` DEAD (always superseded by the `finally`'s
|
||||||
|
raise-from-`errors`). Collapsing to a single
|
||||||
|
`except BaseException as _scope_err` + the existing `finally`
|
||||||
|
is therefore behavior-preserving.
|
||||||
|
|
||||||
|
- removed the outer `try` + the
|
||||||
|
`except (Exception, BaseExceptionGroup, trio.Cancelled)`
|
||||||
|
handler; de-indented the inner handler one level.
|
||||||
|
- dropped the now-unused `outer_err`/`inner_err` locals
|
||||||
|
(single `_scope_err`).
|
||||||
|
- net ~30 lines lighter.
|
||||||
|
|
||||||
|
Gated hard: baseline (pre-B2 tip) = 300 passed + 1
|
||||||
|
load-induced `TooSlowError` flake (passes 6/6 in isolation);
|
||||||
|
B2 error/cancel gate (`test_cancellation remote_exc_relay
|
||||||
|
inter_peer_cancellation advanced_faults oob_cancellation
|
||||||
|
to_actor spawning local rpc`) = 71 passed, 1 xfailed; B2
|
||||||
|
full-suite run following.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `tractor/runtime/_supervise.py` — collapse the two handlers
|
||||||
|
into one; drop `outer_err`/`inner_err`
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None yet — committed via the drafted
|
||||||
|
`.claude/git_commit_msg_ria_b2.md` (user-driven
|
||||||
|
`git commit --edit`).
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
---
|
||||||
|
model: claude-opus-4-8
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-07-02T22:25:44Z
|
||||||
|
git_ref: 9201a2ed
|
||||||
|
diff_cmd: git diff 9201a2ed..drop_ria_nursery
|
||||||
|
---
|
||||||
|
|
||||||
|
# Raw AI output (diff-ref mode)
|
||||||
|
|
||||||
|
Step-B2 code lives on `drop_ria_nursery` after `9201a2ed`; per
|
||||||
|
diff-ref mode the verbatim content is reachable via the pointer
|
||||||
|
below.
|
||||||
|
|
||||||
|
## Generated files
|
||||||
|
|
||||||
|
> `git diff 9201a2ed..drop_ria_nursery -- tractor/runtime/_supervise.py`
|
||||||
|
|
||||||
|
`_open_and_supervise_one_cancels_all_nursery`:
|
||||||
|
- removed the outer `try:` wrapper and the
|
||||||
|
`except (Exception, BaseExceptionGroup, trio.Cancelled) as
|
||||||
|
_outer_err:` safety-net handler.
|
||||||
|
- the former inner `except BaseException` is now THE handler,
|
||||||
|
renamed local `_inner_err` -> `_scope_err`, de-indented one
|
||||||
|
level; it sets `an._scope_error`, records `errors[uid]`,
|
||||||
|
waits on the debugger, `_join_procs.set()`, then a shielded
|
||||||
|
classify/log + snapshot-ria + `an.cancel()` + 0.5s-bounded
|
||||||
|
`_reap_ria_portals()`. No re-raise (the `finally` raises
|
||||||
|
from `errors`).
|
||||||
|
- `finally` block unchanged.
|
||||||
|
- dropped the `outer_err`/`inner_err` local decls at fn top.
|
||||||
|
|
||||||
|
(The diff is large — ~119+/149- — because de-indenting the
|
||||||
|
handler body one level rewrites every line in the block; the
|
||||||
|
logic delta is just "two handlers -> one".)
|
||||||
|
|
||||||
|
## Test runs (verbatim)
|
||||||
|
|
||||||
|
```
|
||||||
|
baseline (pre-B2, step-B tip 9201a2ed), full suite
|
||||||
|
(dynamic_pub_sub deselected):
|
||||||
|
1 failed, 300 passed, 9 skipped, 2 deselected, 1 xfailed,
|
||||||
|
2 xpassed in 1499.49s
|
||||||
|
-> the 1 failure = test_ext_types_over_ipc[...] trio.TooSlowError
|
||||||
|
(load-induced; passes 6/6 in isolation in 4.89s)
|
||||||
|
|
||||||
|
B2 error/cancel gate:
|
||||||
|
tests/test_cancellation test_remote_exc_relay
|
||||||
|
test_inter_peer_cancellation test_advanced_faults
|
||||||
|
test_oob_cancellation test_to_actor test_spawning test_local
|
||||||
|
test_rpc
|
||||||
|
-> 71 passed, 1 xfailed in 125.26s
|
||||||
|
|
||||||
|
B2 full-suite run: see b2_full.log
|
||||||
|
```
|
||||||
|
|
@ -623,9 +623,6 @@ async def _open_and_supervise_one_cancels_all_nursery(
|
||||||
# normally don't need to show user by default
|
# normally don't need to show user by default
|
||||||
__tracebackhide__: bool = hide_tb
|
__tracebackhide__: bool = hide_tb
|
||||||
|
|
||||||
outer_err: BaseException|None = None
|
|
||||||
inner_err: BaseException|None = None
|
|
||||||
|
|
||||||
# the collection of errors retreived from spawned sub-actors
|
# the collection of errors retreived from spawned sub-actors
|
||||||
errors: dict[tuple[str, str], BaseException] = {}
|
errors: dict[tuple[str, str], BaseException] = {}
|
||||||
|
|
||||||
|
|
@ -645,156 +642,129 @@ async def _open_and_supervise_one_cancels_all_nursery(
|
||||||
errors
|
errors
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
try:
|
# spawning of actors happens in the caller's scope
|
||||||
# spawning of actors happens in the caller's scope
|
# after we yield upwards
|
||||||
# after we yield upwards
|
yield an
|
||||||
yield an
|
|
||||||
|
|
||||||
# When we didn't error in the caller's scope,
|
# When we didn't error in the caller's scope,
|
||||||
# signal all process-monitor-tasks to conduct
|
# signal all process-monitor-tasks to conduct
|
||||||
# the "hard join phase".
|
# the "hard join phase".
|
||||||
log.runtime(
|
log.runtime(
|
||||||
'Waiting on subactors to complete:\n'
|
'Waiting on subactors to complete:\n'
|
||||||
f'>}} {len(an._children)}\n'
|
f'>}} {len(an._children)}\n'
|
||||||
)
|
)
|
||||||
an._join_procs.set()
|
an._join_procs.set()
|
||||||
|
|
||||||
# collect results (and errors) from all
|
# collect results (and errors) from all
|
||||||
# `.run_in_actor()` children then cancel
|
# `.run_in_actor()` children then cancel
|
||||||
# each, one reaper task per child.
|
# each, one reaper task per child.
|
||||||
await _reap_ria_portals(an, errors)
|
await _reap_ria_portals(an, errors)
|
||||||
|
|
||||||
except BaseException as _inner_err:
|
# Single one-cancels-all handler for the (now single)
|
||||||
inner_err = _inner_err
|
# daemon nursery. Pre-#477 a 2ndary `._ria_nursery`
|
||||||
errors[actor.aid.uid] = inner_err
|
# required a separate *outer* handler to catch errors
|
||||||
|
# bubbling from its task-reaping `__aexit__`; with that
|
||||||
|
# nursery gone this lone handler covers every scope
|
||||||
|
# error. NB: we deliberately do NOT re-raise here — the
|
||||||
|
# `finally` below raises the collected `errors` (as a
|
||||||
|
# single exc or `BaseExceptionGroup`), which already
|
||||||
|
# superseded the old outer handler's `raise` anyway
|
||||||
|
# since `errors` is populated (below) before any await.
|
||||||
|
except BaseException as _scope_err:
|
||||||
|
an._scope_error = _scope_err
|
||||||
|
errors[actor.aid.uid] = _scope_err
|
||||||
|
|
||||||
# If we error in the root but the debugger is
|
# If we error in the root but the debugger is
|
||||||
# engaged we don't want to prematurely kill (and
|
# engaged we don't want to prematurely kill (and
|
||||||
# thus clobber access to) the local tty since it
|
# thus clobber access to) the local tty since it
|
||||||
# will make the pdb repl unusable.
|
# will make the pdb repl unusable.
|
||||||
# Instead try to wait for pdb to be released before
|
# Instead try to wait for pdb to be released before
|
||||||
# tearing down.
|
# tearing down.
|
||||||
await debug.maybe_wait_for_debugger(
|
|
||||||
child_in_debug=an._at_least_one_child_in_debug
|
|
||||||
)
|
|
||||||
|
|
||||||
# if the caller's scope errored then we activate our
|
|
||||||
# one-cancels-all supervisor strategy (don't
|
|
||||||
# worry more are coming).
|
|
||||||
an._join_procs.set()
|
|
||||||
|
|
||||||
# XXX NOTE XXX: hypothetically an error could
|
|
||||||
# be raised and then a cancel signal shows up
|
|
||||||
# slightly after in which case the `else:`
|
|
||||||
# block here might not complete? For now,
|
|
||||||
# shield both.
|
|
||||||
with trio.CancelScope(shield=True):
|
|
||||||
etype: type = type(inner_err)
|
|
||||||
if etype in (
|
|
||||||
trio.Cancelled,
|
|
||||||
KeyboardInterrupt,
|
|
||||||
) or (
|
|
||||||
is_multi_cancelled(inner_err)
|
|
||||||
):
|
|
||||||
log.cancel(
|
|
||||||
f'Actor-nursery cancelled by {etype}\n\n'
|
|
||||||
|
|
||||||
f'{current_actor().aid.uid}\n'
|
|
||||||
f' |_{an}\n\n'
|
|
||||||
|
|
||||||
# TODO: show tb str?
|
|
||||||
# f'{tb_str}'
|
|
||||||
)
|
|
||||||
elif etype in {
|
|
||||||
ContextCancelled,
|
|
||||||
}:
|
|
||||||
log.cancel(
|
|
||||||
'Actor-nursery caught remote cancellation\n'
|
|
||||||
'\n'
|
|
||||||
f'{inner_err.tb_str}'
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
log.exception(
|
|
||||||
'Nursery errored with:\n'
|
|
||||||
|
|
||||||
# TODO: same thing as in
|
|
||||||
# `._invoke()` to compute how to
|
|
||||||
# place this div-line in the
|
|
||||||
# middle of the above msg
|
|
||||||
# content..
|
|
||||||
# -[ ] prolly helper-func it too
|
|
||||||
# in our `.log` module..
|
|
||||||
# '------ - ------'
|
|
||||||
)
|
|
||||||
|
|
||||||
# snapshot `.run_in_actor()` children
|
|
||||||
# BEFORE cancelling: each backend
|
|
||||||
# spawn-task pops its `._children`
|
|
||||||
# entry as the proc gets reaped.
|
|
||||||
ria_children: list = [
|
|
||||||
(portal, subactor)
|
|
||||||
for subactor, _, portal
|
|
||||||
in an._children.values()
|
|
||||||
if portal in
|
|
||||||
an._cancel_after_result_on_exit
|
|
||||||
]
|
|
||||||
# cancel all subactors
|
|
||||||
await an.cancel()
|
|
||||||
|
|
||||||
# then collect any already-relayed
|
|
||||||
# results/errors from ria children.
|
|
||||||
# Tightly bounded: anything
|
|
||||||
# collectable is already queued in
|
|
||||||
# the local ctx (relayed BEFORE the
|
|
||||||
# cancel above); a child hard-killed
|
|
||||||
# without relaying just parks its
|
|
||||||
# reaper which then self-cleans (a
|
|
||||||
# `trio.Cancelled` result is never
|
|
||||||
# stashed), mirroring the old
|
|
||||||
# backend-side reaper-vs-`soft_kill`
|
|
||||||
# cancel race.
|
|
||||||
with trio.move_on_after(0.5):
|
|
||||||
await _reap_ria_portals(
|
|
||||||
an,
|
|
||||||
errors,
|
|
||||||
ria_children=ria_children,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Safety-net handler for anything escaping the inner
|
|
||||||
# `except BaseException` above (e.g. a `trio.Cancelled`
|
|
||||||
# during its non-shielded debugger-wait, or a failure
|
|
||||||
# inside the shielded teardown itself). Post-#477 the
|
|
||||||
# 2ndary `._ria_nursery` (and its dedicated handler) is
|
|
||||||
# gone; this now guards the single daemon nursery scope.
|
|
||||||
# TODO: can this be merged into the inner handler now that
|
|
||||||
# there's only one nursery? (higher-risk, own PR)
|
|
||||||
except (
|
|
||||||
Exception,
|
|
||||||
BaseExceptionGroup,
|
|
||||||
trio.Cancelled
|
|
||||||
) as _outer_err:
|
|
||||||
outer_err = _outer_err
|
|
||||||
|
|
||||||
an._scope_error = outer_err or inner_err
|
|
||||||
|
|
||||||
# XXX: yet another guard before allowing the cancel
|
|
||||||
# sequence in case a (single) child is in debug.
|
|
||||||
await debug.maybe_wait_for_debugger(
|
await debug.maybe_wait_for_debugger(
|
||||||
child_in_debug=an._at_least_one_child_in_debug
|
child_in_debug=an._at_least_one_child_in_debug
|
||||||
)
|
)
|
||||||
|
|
||||||
# If actor-local error was raised while waiting on
|
# if the caller's scope errored then we activate our
|
||||||
# ".run_in_actor()" actors then we also want to cancel all
|
# one-cancels-all supervisor strategy (don't
|
||||||
# remaining sub-actors (due to our lone strategy:
|
# worry more are coming).
|
||||||
# one-cancels-all).
|
an._join_procs.set()
|
||||||
if an._children:
|
|
||||||
log.cancel(
|
# XXX NOTE XXX: hypothetically an error could
|
||||||
'Actor-nursery cancelling due error type:\n'
|
# be raised and then a cancel signal shows up
|
||||||
f'{outer_err}\n'
|
# slightly after in which case the `else:`
|
||||||
)
|
# block here might not complete? For now,
|
||||||
with trio.CancelScope(shield=True):
|
# shield both.
|
||||||
await an.cancel()
|
with trio.CancelScope(shield=True):
|
||||||
raise
|
etype: type = type(_scope_err)
|
||||||
|
if etype in (
|
||||||
|
trio.Cancelled,
|
||||||
|
KeyboardInterrupt,
|
||||||
|
) or (
|
||||||
|
is_multi_cancelled(_scope_err)
|
||||||
|
):
|
||||||
|
log.cancel(
|
||||||
|
f'Actor-nursery cancelled by {etype}\n\n'
|
||||||
|
|
||||||
|
f'{current_actor().aid.uid}\n'
|
||||||
|
f' |_{an}\n\n'
|
||||||
|
|
||||||
|
# TODO: show tb str?
|
||||||
|
# f'{tb_str}'
|
||||||
|
)
|
||||||
|
elif etype in {
|
||||||
|
ContextCancelled,
|
||||||
|
}:
|
||||||
|
log.cancel(
|
||||||
|
'Actor-nursery caught remote cancellation\n'
|
||||||
|
'\n'
|
||||||
|
f'{_scope_err.tb_str}'
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
log.exception(
|
||||||
|
'Nursery errored with:\n'
|
||||||
|
|
||||||
|
# TODO: same thing as in
|
||||||
|
# `._invoke()` to compute how to
|
||||||
|
# place this div-line in the
|
||||||
|
# middle of the above msg
|
||||||
|
# content..
|
||||||
|
# -[ ] prolly helper-func it too
|
||||||
|
# in our `.log` module..
|
||||||
|
# '------ - ------'
|
||||||
|
)
|
||||||
|
|
||||||
|
# snapshot `.run_in_actor()` children
|
||||||
|
# BEFORE cancelling: each backend
|
||||||
|
# spawn-task pops its `._children`
|
||||||
|
# entry as the proc gets reaped.
|
||||||
|
ria_children: list = [
|
||||||
|
(portal, subactor)
|
||||||
|
for subactor, _, portal
|
||||||
|
in an._children.values()
|
||||||
|
if portal in
|
||||||
|
an._cancel_after_result_on_exit
|
||||||
|
]
|
||||||
|
# cancel all subactors
|
||||||
|
await an.cancel()
|
||||||
|
|
||||||
|
# then collect any already-relayed
|
||||||
|
# results/errors from ria children.
|
||||||
|
# Tightly bounded: anything
|
||||||
|
# collectable is already queued in
|
||||||
|
# the local ctx (relayed BEFORE the
|
||||||
|
# cancel above); a child hard-killed
|
||||||
|
# without relaying just parks its
|
||||||
|
# reaper which then self-cleans (a
|
||||||
|
# `trio.Cancelled` result is never
|
||||||
|
# stashed), mirroring the old
|
||||||
|
# backend-side reaper-vs-`soft_kill`
|
||||||
|
# cancel race.
|
||||||
|
with trio.move_on_after(0.5):
|
||||||
|
await _reap_ria_portals(
|
||||||
|
an,
|
||||||
|
errors,
|
||||||
|
ria_children=ria_children,
|
||||||
|
)
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
# No errors were raised while awaiting ".run_in_actor()"
|
# No errors were raised while awaiting ".run_in_actor()"
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue