Doc ria-reap hang fix + paused reaper re-scope
Append two sections to the ria-removal plan capturing the 2026-07-02 hang episode + the resulting design pivot. Regression writeup: the full-suite hang on `test_tractor_cancels_aio` root-caused to the step-A reaper hoist (`5cd190c5`), not the B2 handler merge. The happy-path `_reap_ria_portals()` parks unbounded on `wait_for_result()` after a user `portal.cancel_actor()`; the old spawn-backend reaper raced `soft_kill()`'s scope-cancel, the hoist dropped it. Records the `proc.poll()` death-watch fix + why poll (not the event `wait_func`) bc `soft_kill` already awaits `proc.sentinel` (a 2nd `wait_readable` -> `BusyResourceError`). Pause writeup: user's insight that the hoist landed in the wrong scope — result-waiting belongs in the `to_actor` one-shot scope (`_invoke_in_subactor()`), beside `an` + a local task-nursery + cancel-scope, where bounding the wait is trivial + the hang dissolves. So the poll fix is likely SUPERSEDED (flagged do-not-land); the anti-hang guard commit (`d1fb4a1a`) stays red-first per the failing-test convention. (this patch was generated in some part by `claude-code` using `claude-opus-4-8` (`anthropic`))drop_ria_nursery
parent
ba4af0ad1a
commit
ff3474728c
|
|
@ -276,6 +276,97 @@ Gate (box ran ~2.7x slow this session, load-induced
|
|||
completion). RECOMMEND a clean full-suite run on a
|
||||
normal-speed box before this merges.
|
||||
|
||||
## Regression + fix: ria-reap hang (2026-07-02)
|
||||
|
||||
Human hit a full-suite hang on
|
||||
`test_infected_asyncio.py::test_tractor_cancels_aio`. Bisected:
|
||||
passes at pre-ria `a34aaf98` (0.59s), hangs at B2 `e617b498`
|
||||
(90s+). Root-caused to the STEP-A reaper hoist (`5cd190c5`),
|
||||
NOT B2 (`_reap_ria_portals` is byte-identical A->B2).
|
||||
|
||||
Bug: the test does `run_in_actor(asyncio_actor)` then a USER
|
||||
`portal.cancel_actor()` and exits the block cleanly -> the
|
||||
happy path's `await _reap_ria_portals()`, which waits UNBOUNDED
|
||||
on `cancel_on_completion -> wait_for_result()`. The child was
|
||||
cancelled out-of-band so no final result is relayed -> parked
|
||||
forever. The OLD spawn-backend reaper was raced against
|
||||
`soft_kill()` (per-child nursery `cancel_scope.cancel()` on
|
||||
subproc death); the hoist dropped that race.
|
||||
|
||||
Fix: `_reap_ria_portals()` runs each `cancel_on_completion()`
|
||||
in a local nursery alongside a `proc.poll()` death-watch that
|
||||
cancels the parked reaper once the subproc exits — restoring
|
||||
the old race, backend-agnostic (guarded by
|
||||
`hasattr(proc, 'poll')` for a future `subint` handle).
|
||||
|
||||
Why POLL (`proc.poll()`) not the event-driven `wait_func`:
|
||||
the mp waiter (`_spawn.proc_waiter`) does
|
||||
`wait_readable(proc.sentinel)`, and `soft_kill()` is ALREADY
|
||||
awaiting that same fd concurrently in the daemon nursery — a
|
||||
2nd `wait_readable` on one fd raises `trio.BusyResourceError`.
|
||||
(`trio.Process.wait()` IS multi-waiter-safe, but mp has no
|
||||
async equivalent.) `proc.poll()` — the same liveness check
|
||||
`soft_kill` itself falls back to — is the conflict-free common
|
||||
denominator. Verified: poll-fix passes on BOTH trio and
|
||||
mp_spawn.
|
||||
|
||||
Also added a per-test anti-hang guard: wrapped
|
||||
`test_tractor_cancels_aio`'s `main()` in
|
||||
`with trio.fail_after(9 * cpu_perf_headroom())` — the blessed
|
||||
pattern (`pytest-timeout`'s global cap is intentionally off;
|
||||
breaks trio under fork backends, see `pyproject` NOTE). So a
|
||||
future recurrence FAILS FAST instead of hanging the suite.
|
||||
(Several other tests in the file are still guardless —
|
||||
`test_aio_simple_error`, `test_trio_error_cancels_intertask_chan`,
|
||||
`test_aio_errors_and_channel_propagates_and_closes` — candidate
|
||||
follow-up sweep.)
|
||||
|
||||
Lesson: the B2 focused gate OMITTED `test_infected_asyncio`
|
||||
(and the full runs were clipped/slow), so the step-A hang
|
||||
slipped through. Any future ria-touching change MUST gate
|
||||
`test_infected_asyncio` explicitly.
|
||||
|
||||
Gate: `test_tractor_cancels_aio` green (trio 1.53s, mp 3.98s);
|
||||
fix gate (`test_infected_asyncio test_cancellation test_to_actor
|
||||
test_spawning`) = 74 passed, 3 xfailed, 0 failures.
|
||||
|
||||
## PAUSED (2026-07-02): re-assess the reaper's SCOPE
|
||||
|
||||
User's insight (compelling — likely the real root cause of
|
||||
the hang, not just the missing proc-death race):
|
||||
|
||||
> the "hoisting" of 5cd190c5 was just not really done right
|
||||
> — the hoist should have been into the `to_actor` scope,
|
||||
> not `_supervise`.
|
||||
|
||||
The argument: `.run_in_actor()`'s result-waiting/reaping got
|
||||
hoisted into `_supervise._reap_ria_portals` (nursery-machinery
|
||||
scope), which has NO natural cancel-scope to bound a parked
|
||||
`wait_for_result()` — hence the awkward proc-death race +
|
||||
the poll-vs-`proc_waiter` dilemma. If the result-wait instead
|
||||
lived in the `to_actor` one-shot scope
|
||||
(`to_actor._invoke_in_subactor()`), it would sit right next to
|
||||
the caller's `an` + a local `trio` task-nursery + cancel-scope
|
||||
(the `trio.to_thread`-style model #477 actually wants) — so
|
||||
bounding/cancelling the wait is trivial and the hang
|
||||
dissolves from correct scoping rather than a bolt-on race.
|
||||
|
||||
Follow-on to re-evaluate on resume:
|
||||
- should `_reap_ria_portals` exist AT ALL, or should
|
||||
result-waiting move entirely into
|
||||
`to_actor._invoke_in_subactor()`?
|
||||
- reimplement legacy `run_in_actor()` on top of
|
||||
`to_actor.run()` so `_reap_ria_portals` +
|
||||
`_cancel_after_result_on_exit` can be DROPPED from
|
||||
`_supervise` entirely (the true #477 simplification)?
|
||||
- the poll-vs-event decision is MOOT under this re-scoping.
|
||||
|
||||
State at pause: `test_infected_asyncio` anti-hang guard
|
||||
COMMITTED (`d1fb4a1a`, intentionally red w/o the fix — the
|
||||
user's failing-test-first convention). The poll-based reap
|
||||
fix in `_supervise.py` is UNCOMMITTED and likely SUPERSEDED
|
||||
by the re-scoping — do NOT land it as-is.
|
||||
|
||||
## Verification gate
|
||||
|
||||
- `tests/test_cancellation.py test_spawning.py test_local.py
|
||||
|
|
|
|||
Loading…
Reference in New Issue