Merge pull request #462 from goodboy/subint_era_tooling

Subint era tooling

.claude/skills/run-tests/SKILL.md

@@ -528,3 +528,105 @@ filling log volume. Full post-mortem in
 `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
 Lesson codified here so future-me grep-finds the
 workaround before digging.
+
+## 10. Reaping zombie subactors (`tractor-reap`)
+
+**Symptom:** after a `pytest` run crashes, times out,
+or is `Ctrl+C`'d, subactor forks (esp. under
+`subint_forkserver`) can be reparented to `init`
+(PPid==1) and linger. They hold onto ports, inherit
+pytest's capture-pipe fds, and flakify later
+sessions.
+
+**Two layers of defense:**
+
+### a) Session-scoped auto-fixture (always on)
+
+`tractor/_testing/pytest.py::_reap_orphaned_subactors`
+runs at pytest session teardown. It walks `/proc` for
+direct descendants of the pytest pid, SIGINTs them,
+waits up to 3s, then SIGKILLs survivors. SC-polite:
+gives the subactor runtime a chance to run its trio
+cancel shield + IPC teardown before escalation.
+
+This is *autouse* and session-scoped — you don't need
+to do anything. It just runs.
+
+### b) `scripts/tractor-reap` CLI (manual reap)
+
+For the **pytest-died-mid-session** case (Ctrl+C, OOM
+kill, hung process you had to `kill -9`), the fixture
+never ran. Reach for the CLI:
+
+```sh
+# default: orphans (PPid==1, cwd==repo, cmd contains python)
+scripts/tractor-reap
+
+# descendant-mode: from a still-live supervisor
+scripts/tractor-reap --parent <pytest-pid>
+
+# see what would be reaped, don't signal
+scripts/tractor-reap -n
+
+# tune the SIGINT → SIGKILL grace window
+scripts/tractor-reap --grace 5
+```
+
+Exit code: `0` if everyone exited on SIGINT, `1` if
+SIGKILL had to escalate — so you can chain it in CI
+health-checks (`scripts/tractor-reap || <alert>`).
+
+**What it matches** (orphan-mode):
+- `PPid == 1` (reparented to init → definitely
+  orphaned, not just a currently-running child)
+- `cwd == <repo-root>` (keeps the sweep scoped; won't
+  touch unrelated init-children elsewhere)
+- `python` in cmdline
+
+**What it does not do:** kill anything whose PPid is
+still a live tractor parent. If the parent is alive
+it's not an orphan; use `--parent <pid>` if you need
+to force-reap under a still-live supervisor.
+
+**When NOT to run it:** while a pytest session is
+active in another terminal. It's safe (won't touch
+that session's live children in orphan-mode) but can
+race if the target session is mid-teardown.
+
+### c) `--shm` / `--shm-only`: orphan-segment sweep
+
+Because `tractor.ipc._mp_bs.disable_mantracker()`
+turns off `mp.resource_tracker` (see
+`ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`),
+a hard-crashing actor can leave `/dev/shm/<key>`
+segments behind that nothing else GCs.
+
+```sh
+# process reap THEN shm sweep
+scripts/tractor-reap --shm
+
+# shm sweep only (skip process phase)
+scripts/tractor-reap --shm-only
+
+# dry-run: list candidates, don't unlink
+scripts/tractor-reap --shm -n
+```
+
+**Match criteria** (very conservative — this is a
+shared-system path, can't be wrong):
+- segment is a regular file under `/dev/shm`,
+- owned by the **current uid** (`stat.st_uid`),
+- AND **no live process holds it open** —
+  enumerated by walking every readable
+  `/proc/<pid>/maps` (post-mmap mappings) AND
+  `/proc/<pid>/fd/*` (pre-mmap shm-opened fds).
+
+The "nobody has it open" check is the
+kernel-canonical "is this leaked?" test — same
+answer `lsof /dev/shm/<key>` would give. No
+reliance on tractor-specific naming, so it works
+for any tractor app. Critically, it WILL NOT touch
+segments held by other apps you have running
+(e.g. `piker`, `lttng-ust-*`, `aja-shm-*` —
+verified locally with 81 in-use segments correctly
+preserved).

ai/conc-anal/spawn_time_boot_death_dup_name_issue.md

@@ -0,0 +1,142 @@
+# Spawn-time boot-death (`rc=2`) under rapid same-name spawn against a registrar
+
+## Symptom
+
+Spawning N (≥4) sub-actors with the **same name** in tight
+succession against a daemon registrar surfaces as
+`ActorFailure: Sub-actor (...) died during boot (rc=2)
+before completing parent-handshake`.
+
+```
+tests/discovery/test_multi_program.py
+  ::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]
+```
+
+```
+tractor._exceptions.ActorFailure:
+  Sub-actor ('doggy', '<uuid>') died during boot (rc=2)
+  before completing parent-handshake.
+    proc: <_ForkedProc pid=<n> returncode=None>
+```
+
+The `proc` repr shows `returncode=None` because the repr is
+captured before `proc.wait()` returns; the actual
+`os.WEXITSTATUS == 2` is reported via `result['died']` in the
+race-helper.
+
+## When it surfaces
+
+- N=2 (`n_dups=2`): **always passes**.
+- N=4 (`n_dups=4`): **consistent fail** under both `tpt-proto=tcp`
+  and `tpt-proto=uds`, MTF backend.
+- N=8 (`n_dups=8`): **passes** (counter-intuitive — see "racing
+  windows").
+- Non-MTF backends: not yet exercised systematically.
+
+## What previously masked it
+
+Pre the spawn-time `wait_for_peer_or_proc_death` race-helper
+(in `tractor.spawn._spawn`), the parent's `start_actor` flow
+ended with a bare:
+
+```python
+event, chan = await ipc_server.wait_for_peer(uid)
+```
+
+That awaits an unsignalled `trio.Event` on `_peer_connected[uid]`.
+If the sub-actor process **dies during boot** (before its
+runtime executes the parent-callback handshake that sets the
+event), the wait parks forever. The dead proc becomes a zombie
+because no one ever calls `proc.wait()` to reap it.
+
+In test contexts the failure presented as a hang or a much
+later `trio.TooSlowError` from an outer `fail_after`. In
+production it'd present as a parent that never makes progress
+past `start_actor`. The death itself was silently masked.
+
+## What surfaces it now
+
+`tractor.spawn._spawn.wait_for_peer_or_proc_death` (used by
+`_main_thread_forkserver_proc`) races the handshake-wait
+against `proc.wait()`. The race-helper raises `ActorFailure`
+on death-first instead of parking, exposing the rc=2.
+
+## Hypothesis: registrar-side same-name contention
+
+The test spawns N actors with name `doggy` sequentially:
+
+```python
+for i in range(n_dups):
+    p: Portal = await an.start_actor('doggy')
+    portals.append(p)
+```
+
+Each spawned doggy:
+
+1. Forks via the forkserver.
+2. Boots its runtime in `_actor_child_main`.
+3. Connects back to the parent for handshake.
+4. Connects to the daemon registrar to call `register_actor`.
+5. Enters its RPC msg-loop.
+
+Step (4) is where the same-name contention lives. The
+registrar's `register_actor` (in
+`tractor.discovery._registry`) accepts duplicate names
+(stores `(name, uuid) -> addr`), but its internal bookkeeping
+may have a non-trivial check (e.g. `wait_for_actor` resolution,
+`_addrs2aids` map updates) that errors out under specific
+ordering between the existing entry and the incoming one.
+
+`rc=2 == os.WEXITSTATUS == 2` corresponds to `sys.exit(2)`
+in the doggy process — typically reached via an unhandled
+exception that's translated to exit code 2 by Python's top-
+level (e.g. `argparse` errors use 2; `SystemExit(2)` etc.).
+So the doggy is hitting an explicit exit path during
+`register_actor` or just-after.
+
+The non-monotonic shape (N=2 OK, N=4 BAD, N=8 OK) suggests a
+specific timing window — likely "the 3rd register-RPC arrives
+while the 1st-or-2nd is in some intermediate state". With
+N=8, the additional procs widen the registration spread
+enough that no two land in the conflicting window.
+
+## Where to dig next
+
+- Add per-actor logging in `_actor_child_main` and
+  `register_actor` to surface the actual exception that
+  triggers the rc=2 exit. Currently the doggy dies before
+  the parent ever sees its stderr (forkserver doesn't
+  marshal child stdio back).
+- Race-test the registrar's `register_actor` /
+  `unregister_actor` /  `wait_for_actor` against same-name
+  concurrent calls in isolation (no spawn).
+- Consider whether `register_actor` should be idempotent
+  under same-name re-register or should explicitly reject
+  same-name (and ideally with a clear `RemoteActorError`,
+  not `sys.exit(2)`).
+
+## Test-suite handling
+
+Currently:
+
+- `tests/discovery/test_multi_program.py
+  ::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]`
+  is `pytest.mark.xfail(strict=False, reason=...)` to keep
+  the suite green while this issue is investigated.
+- `n_dups=2` and `n_dups=8` continue to validate the
+  cancel-cascade hard-kill escalation.
+
+Once the underlying race is understood + fixed, drop the
+xfail.
+
+## Related work
+
+- The cancel-cascade fix that introduced this regression
+  test:
+  `tractor/_exceptions.py:ActorTooSlowError`,
+  `tractor/runtime/_supervise.py:_try_cancel_then_kill`,
+  `tractor/runtime/_portal.py:Portal.cancel_actor(
+   raise_on_timeout=...)`.
+- The spawn-time death-detection that exposed this:
+  `tractor/spawn/_spawn.py:wait_for_peer_or_proc_death`,
+  used by `tractor/spawn/_main_thread_forkserver.py`.

ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md

@@ -0,0 +1,102 @@
+# `trio` 0.29 -> 0.33 slows the depth=3 cancel-cascade
+
+## Symptom
+
+After locking to `trio==0.33.0` (commit `c7741bba`, was
+`0.29.0`), this test reliably trips its `fail_after`
+deadline on the **`trio`** backend:
+
+```
+FAILED tests/test_cancellation.py::test_nested_multierrors[start_method=trio-depth=3]
+  - AssertionError: assert False
+    where False = isinstance(
+      Cancelled(source='deadline', source_task=None, reason=None),
+      tractor.RemoteActorError,
+    )
+```
+
+A `fail_after_w_trace` hang-snapshot is captured for the
+test each run (deadline-injected `Cancelled` wrapped into
+the actor-nursery `BaseExceptionGroup`).
+
+## Root cause (immediate)
+
+The test budgets `fail_after(6)` for the `trio` backend.
+That 6s was chosen (commit `32955db0`, while `trio==0.29`)
+with the assertion that trio finishes "well under" 6s.
+The `trio` 0.29 -> 0.33 bump slowed the depth=3 cascade
+past that budget, so the 6s deadline now fires mid-cascade.
+
+trio 0.33 added **cancel-reason tracking** — every
+`Cancelled` now carries `(source=, reason=, source_task=)`.
+The injected exc is `Cancelled(source='deadline')`, i.e.
+trio itself naming our `fail_after(6)` scope as the cancel
+origin. When that `Cancelled` collapses one branch of the
+nursery BEG, the test's `isinstance(subexc,
+RemoteActorError)` assertion fails. The healthy outcome is
+`BEG = [RemoteActorError, RemoteActorError]`; the
+`Cancelled` is purely an artifact of the deadline cutting
+the cascade short.
+
+## Measurements (standalone, this machine)
+
+```
+depth=1  trio   ~3.15s   PASS  (keeps 6s budget)
+depth=3  trio   ~6.8-8.2s  FAIL @ 6s  (now bumped to 12s)
+```
+
+depth=1 still fits comfortably; only depth=3 (deeper
+recursive spawn-and-error tree => more actors to reap)
+exceeds the old budget. The ~2s/depth-level cost looks
+like serialized per-actor reap / `terminate_after` waits.
+
+## Mitigation applied
+
+`test_nested_multierrors` now splits the `trio` budget:
+
+```python
+case ('trio', 1):
+    timeout = 6
+case ('trio', 3):
+    timeout = 12   # was 6; see this doc
+```
+
+This stops the deadline from firing so the cascade
+completes naturally to `[RAE, RAE]`.
+
+## Also affected — same root cause, different test
+
+`test_echoserver_detailed_mechanics[trio-raise_error=KeyboardInterrupt]`
+(`tests/test_infected_asyncio.py`) tripped the *same*
+slowdown via its much tighter `trio` budget of `1s`. The
+single-aio-subactor teardown now takes ~1s, so the `1s`
+`fail_after` raced the deadline (PASS at 0.99s / FAIL at
+1.03s across back-to-back standalone runs). On a deadline-
+fire the injected `Cancelled(source='deadline')` wraps the
+mid-stream `KeyboardInterrupt` into a `BaseExceptionGroup`,
+which is NOT a `KeyboardInterrupt` so the bare
+`pytest.raises(KeyboardInterrupt)` fails. (The sibling
+`raise_error=Exception` variant only "passes" by accident:
+an `ExceptionGroup` *is-a* `Exception`, so its
+`pytest.raises(Exception)` still matches even when wrapped.)
+
+Mitigation: bump that `trio` budget `1 -> 4s` (matching the
+forking-spawner case). Without a deadline-fire the KBI
+propagates bare and the assertion passes.
+
+## Open follow-up (the actual regression)
+
+The budget bump is a band-aid — the underlying question is
+**why** the depth=3 `trio` cancel-cascade went from <6s to
+~7-8s across `trio` 0.29 -> 0.33. Candidate avenues:
+
+- which scope owns the per-actor `terminate_after` wait,
+  and are the tree's reaps concurrent or serialized?
+- did trio 0.33's abort/reschedule or cancel-reason
+  bookkeeping change checkpoint timing on the cancel path?
+
+If/when the cascade speeds back up under-budget, depth=3
+will start completing well under 12s — at which point the
+budget can be tightened back toward 6s as a regression
+tripwire. Related (different backend, same cascade class):
+`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.

ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md

@@ -0,0 +1,221 @@
+# trio `WakeupSocketpair.drain()` busy-loop in forked child (peer-closed missed-EOF)
+
+## Reproducer
+
+```bash
+./py313/bin/python -m pytest \
+  tests/test_multi_program.py::test_register_duplicate_name \
+  --tpt-proto=tcp \
+  --spawn-backend=main_thread_forkserver \
+  -v --capture=sys
+```
+
+Subactor pegs a CPU core indefinitely; parent test
+hangs waiting for the subactor.
+
+## Empirical evidence (caught alive)
+
+```
+$ sudo strace -p <subactor-pid>
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+... (no `epoll_wait`, no other syscalls, just this back-to-back)
+```
+
+Pattern: tight C-level `recvfrom` loop returning 0
+each call. No `epoll_wait` between iterations →
+**not trio's task scheduler**. Pure synchronous C
+loop.
+
+```
+$ sudo readlink /proc/<subactor-pid>/fd/6
+socket:[<inode>]
+
+$ sudo lsof -p <subactor-pid> | grep ' 6u'
+<cmd> <pid> goodboy 6u unix 0xffff... 0t0 <inode> type=STREAM (CONNECTED)
+```
+
+fd=6 is an **AF_UNIX socket** in CONNECTED state.
+Even though the test uses `--tpt-proto=tcp`, this fd
+is NOT a tractor IPC channel — it's an internal
+trio socketpair.
+
+## Root-cause: `WakeupSocketpair.drain()`
+
+`/site-packages/trio/_core/_wakeup_socketpair.py`:
+
+```python
+class WakeupSocketpair:
+    def __init__(self) -> None:
+        self.wakeup_sock, self.write_sock = socket.socketpair()
+        self.wakeup_sock.setblocking(False)
+        self.write_sock.setblocking(False)
+        ...
+
+    def drain(self) -> None:
+        try:
+            while True:
+                self.wakeup_sock.recv(2**16)
+        except BlockingIOError:
+            pass
+```
+
+`socket.socketpair()` on Linux defaults to AF_UNIX
+SOCK_STREAM. Both ends non-blocking. Normal flow:
+
+1. Signal/wake event → `write_sock.send(b'\x00')`
+   queues a byte.
+2. `wakeup_sock` becomes readable → trio's epoll
+   triggers.
+3. Trio calls `drain()` to flush the buffer.
+4. drain loops on `wakeup_sock.recv(64KB)`.
+5. Eventually buffer empty → non-blocking socket
+   raises `BlockingIOError` → except → break.
+
+**Bug surface — peer-closed missed-EOF**:
+
+Non-blocking socket semantics:
+- buffer has data → `recv` returns N>0 bytes (loop continues)
+- buffer empty → `recv` raises `BlockingIOError`
+- **peer FIN'd → `recv` returns 0 bytes (NEITHER exception NOR
+  break — infinite tight loop)**
+
+`drain()` does not handle the `b''` return-value
+(EOF) case. If `write_sock` has been closed (or the
+process holding it is gone), every iteration returns
+0 → infinite loop → 100% CPU on a single core.
+
+## Why this triggers under `main_thread_forkserver`
+
+Under `os.fork()` from the forkserver-worker thread:
+
+1. Parent has a `WakeupSocketpair` instance with
+   `wakeup_sock=fdN`, `write_sock=fdM`. Both fds
+   open in parent.
+2. Fork → child inherits BOTH fds (kernel-level fd
+   table dup).
+3. `_close_inherited_fds()` runs in child →
+   closes everything except stdio. `wakeup_sock` and
+   `write_sock` of the parent's `WakeupSocketpair`
+   ARE closed in child.
+4. Child's trio (running fresh) creates its OWN
+   `WakeupSocketpair` → NEW fd numbers (e.g. fd 6, 7).
+5. **In `infect_asyncio` mode** the asyncio loop is
+   the host; trio runs as guest via
+   `start_guest_run`. trio still creates its
+   `WakeupSocketpair` in the I/O manager but its
+   role is different.
+
+The race window: somewhere between (3) and (5), if a
+`WakeupSocketpair` Python object reference inherited
+via COW (from parent's pre-fork heap) survives long
+enough that `drain()` is called on it AFTER its fds
+were closed but BEFORE the child's NEW socketpair
+takes over the recycled fd numbers — the recycled fd
+will be one of the child's NEW socketpair ends, whose
+peer might be FIN-flagged (e.g. parent-process
+peer-end is closed).
+
+Or simpler: the `wait_for_actor`/`find_actor` discovery
+flow in `test_register_duplicate_name` triggers an
+unusual code path where a stale `WakeupSocketpair`
+gets `drain()`-called on a fd whose peer has already
+closed.
+
+## Why `drain()` shouldn't loop indefinitely on EOF
+(upstream trio bug)
+
+Even WITHOUT fork, `drain()` should treat `b''` as
+EOF and break. The current code is correct for the
+"buffer drained on a healthy socketpair" scenario but
+incorrect for the "peer is gone" scenario. It's a
+defensive-programming gap in trio.
+
+A one-line patch upstream:
+
+```python
+def drain(self) -> None:
+    try:
+        while True:
+            data = self.wakeup_sock.recv(2**16)
+            if not data:
+                break  # peer-closed; nothing more to drain
+    except BlockingIOError:
+        pass
+```
+
+## Workarounds (until the underlying issue lands)
+
+1. **Skip-mark on the fork backend**:
+   `tests/test_multi_program.py` →
+   `pytest.mark.skipon_spawn_backend('main_thread_forkserver',
+   reason='trio WakeupSocketpair.drain busy-loop, see ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md')`.
+
+2. **Defensive monkey-patch in tractor's
+   forkserver-child prelude** — wrap
+   `WakeupSocketpair.drain` to handle `b''`:
+
+   ```python
+   # in `_actor_child_main` or `_close_inherited_fds`'s
+   # post-fork prelude:
+   from trio._core._wakeup_socketpair import WakeupSocketpair
+   _orig_drain = WakeupSocketpair.drain
+   def _safe_drain(self):
+       try:
+           while True:
+               data = self.wakeup_sock.recv(2**16)
+               if not data:
+                   return  # peer closed
+       except BlockingIOError:
+           pass
+   WakeupSocketpair.drain = _safe_drain
+   ```
+
+   Tracks upstream — remove once trio fixes.
+
+3. **Upstream the fix**: 1-line PR to `python-trio/trio`
+   adding `if not data: break` to `drain()`.
+
+## Investigation next steps
+
+1. **Confirm via py-spy**: when caught alive, detach
+   strace first then
+   `sudo py-spy dump --pid <subactor> --locals`. The
+   busy thread should show `drain` from `WakeupSocketpair`
+   in the call chain.
+2. **Identify which write-end peer is closed**: from
+   the inode of fd 6, look up the matching peer
+   inode via `ss -xp` and see whose process it
+   was/is.
+3. **Verify the missed-EOF hypothesis**: hand-craft a
+   minimal `WakeupSocketpair` repro:
+
+   ```python
+   from trio._core._wakeup_socketpair import WakeupSocketpair
+   ws = WakeupSocketpair()
+   ws.write_sock.close()  # simulate peer-gone
+   ws.drain()             # should hang forever
+   ```
+
+## Sibling bug
+
+`tests/test_infected_asyncio.py::test_aio_simple_error`
+hangs under the same backend with a DIFFERENT
+fingerprint (Mode-A deadlock, both parties in
+`epoll_wait`, no busy-loop). Distinct root cause —
+see `infected_asyncio_under_main_thread_forkserver_hang_issue.md`.
+
+Both share the broader theme: **trio internal-state
+initialization isn't fully fork-safe under
+`main_thread_forkserver`** for the more exotic
+dispatch paths.
+
+## See also
+
+- [#379](https://github.com/goodboy/tractor/issues/379) — subint umbrella
+- python-trio/trio#1614 — trio + fork hazards
+- `trio._core._wakeup_socketpair.WakeupSocketpair`
+  source (the smoking gun)
+- `ai/conc-anal/fork_thread_semantics_execution_vs_memory.md`
+- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`

examples/debugging/multi_daemon_subactors.py

@@ -27,12 +27,9 @@ async def main():
     '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='cancel',
+    ) as an:
-        # loglevel='devx',
+        p0 = await an.start_actor('bp_forever', enable_modules=[__name__])
-    ) as n:
+        p1 = await an.start_actor('name_error', enable_modules=[__name__])
-
-        p0 = await n.start_actor('bp_forever', enable_modules=[__name__])
-        p1 = await n.start_actor('name_error', enable_modules=[__name__])
 
         # retreive results
         async with p0.open_stream_from(breakpoint_forever) as stream:

examples/debugging/multi_nested_subactors_error_up_through_nurseries.py

@@ -67,7 +67,7 @@ async def main():
     """
     async with tractor.open_nursery(
         debug_mode=True,
-        # loglevel='cancel',
+        loglevel='pdb',
     ) as n:
 
         # spawn both actors

examples/debugging/root_cancelled_but_child_is_in_tty_lock.py

@@ -39,8 +39,8 @@ async def main():
     '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='devx',
+        enable_transports=['uds'],  # TODO, apss this via osenv?
-        enable_transports=['uds'],
+        loglevel='devx',  # XXX, required for test!
     ) as n:
 
         # spawn both actors

examples/debugging/root_timeout_while_child_crashed.py

@@ -1,4 +1,3 @@
-
 import trio
 import tractor
 
@@ -9,16 +8,22 @@ async def key_error():
 
 
 async def main():
-    """Root dies 
+    '''
+    Root is fail-after-cancelled while blocking and child RPC fails
+    simultaneously.
 
-    """
+    '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='debug'
+        # loglevel='debug'  # ?XXX required?
     ) as n:
 
         # spawn both actors
         portal = await n.run_in_actor(key_error)
+        print(
+            f'Child is up @ {portal.chan.aid.reprol()}'
+        )
+
 
         # XXX: originally a bug caused by this is where root would enter
         # the debugger and clobber the tty used by the repl even though

examples/debugging/shield_hang_in_sub.py

@@ -49,9 +49,11 @@ async def main(
         tractor.open_nursery(
             debug_mode=True,
             enable_stack_on_sig=True,
-            # maybe_enable_greenback=False,
+            loglevel='devx',  # XXX REQUIRED log level!
-            loglevel='devx',
             enable_transports=[tpt],
+            # maybe_enable_greenback=True,
+            # ^TODO? maybe a "smarter" way todo all this is how
+            # `modden` does with a rtv serialized through the osenv?
         ) as an,
     ):
         ptl: tractor.Portal  = await an.start_actor(
@@ -63,7 +65,9 @@ async def main(
             start_n_shield_hang,
         ) as (ctx, cpid):
 
-            _, proc, _ = an._children[ptl.chan.uid]
+            _, proc, _ = an._children[
+                ptl.chan.aid.uid
+            ]
             assert cpid == proc.pid
 
             print(

examples/debugging/subactor_bp_in_ctx.py

@@ -36,6 +36,11 @@ async def just_bp(
 
 async def main():
 
+    # !TODO, parametrize the --tpt-proto={key} with osenv vars just
+    # like we do for loglevel/spawn-backend!
+    # - [ ] run on both tpts for all such debugger tests?
+    # - [ ] special skip for macos!
+    #
     if platform.system() != 'Darwin':
         tpt = 'uds'
     else:

examples/debugging/subactor_error.py

@@ -9,7 +9,6 @@ async def name_error():
 async def main():
     async with tractor.open_nursery(
         debug_mode=True,
-        # loglevel='transport',
     ) as an:
 
         # TODO: ideally the REPL arrives at this frame in the parent,

examples/debugging/sync_bp.py

@@ -1,9 +1,22 @@
 from functools import partial
+import os
 import time
 
+# ?TODO? how to make `pdbp` enforce this?
+# os.environ['PYTHON_COLORS'] = '0'
+# os.environ['NO_COLOR'] = '1'
+
 import trio
 import tractor
 
+# disable `pbdp` prompt colors
+# for prompt matching in test.
+def disable_pdbp_color():
+    if os.environ.get('PYTHON_COLORS') == '0':
+        from tractor.devx.debug import _repl
+        _repl.TractorConfig.use_pygments = False
+
+
 # TODO: only import these when not running from test harness?
 # can we detect `pexpect` usage maybe?
 # from tractor.devx.debug import (
@@ -42,6 +55,7 @@ async def start_n_sync_pause(
     ctx: tractor.Context,
 ):
     actor: tractor.Actor = tractor.current_actor()
+    disable_pdbp_color()
 
     # sync to parent-side task
     await ctx.started()
@@ -52,13 +66,15 @@ async def start_n_sync_pause(
 
 
 async def main() -> None:
+    disable_pdbp_color()
     async with (
         tractor.open_nursery(
             debug_mode=True,
             maybe_enable_greenback=True,
-            enable_stack_on_sig=True,
+
-            # loglevel='warning',
+            # XXX flags required for test pattern matching.
-            # loglevel='devx',
+            loglevel='pdb',
+            # enable_stack_on_sig=True,
         ) as an,
         trio.open_nursery() as tn,
     ):
@@ -68,8 +84,8 @@ async def main() -> None:
         p: tractor.Portal  = await an.start_actor(
             'subactor',
             enable_modules=[__name__],
-            # infect_asyncio=True,
             debug_mode=True,
+            # infect_asyncio=True,
         )
 
         # TODO: 3 sub-actor usage cases:

pyproject.toml

@@ -43,15 +43,20 @@ dependencies = [
   "tricycle>=0.4.1,<0.5",
   "wrapt>=1.16.0,<2",
   "colorlog>=6.8.2,<7",
-
   # built-in multi-actor `pdb` REPL
   "pdbp>=1.8.2,<2", # windows only (from `pdbp`)
-
   # typed IPC msging
   "msgspec>=0.20.0",
   "bidict>=0.23.1",
   "multiaddr>=0.2.0",
   "platformdirs>=4.4.0",
+  # per-actor `argv[0]` proc-title for OS-level diag tools
+  # (`ps`, `top`, `psutil`-backed tooling like `acli.pytree`).
+  # Optional at runtime — guarded by `try/except ImportError` in
+  # `tractor.devx._proctitle` — but listed here so default
+  # installs benefit from it. See tracking issue for follow-ups
+  # (e.g. richer formats, per-backend overrides).
+  "setproctitle>=1.3,<2",
 ]
 
 # ------ project ------
@@ -61,6 +66,7 @@ dev = [
   {include-group = 'devx'},
   {include-group = 'testing'},
   {include-group = 'repl'},
+  {include-group = 'sync_pause'},
 ]
 devx = [
   # `tractor.devx` tooling
@@ -84,6 +90,16 @@ testing = [
   # interactions stay predictable across dev installs).
   "pytest>=9.0.3",  # CVE-2025-71176 (insecure tmpdir) patched in 9.0.3
   "pexpect>=4.9.0,<5",
+  # per-test wall-clock bound (used via
+  # `@pytest.mark.timeout(..., method='thread')` on the
+  # known-hanging `subint`-backend audit tests; see
+  # `ai/conc-anal/subint_*_issue.md`).
+  "pytest-timeout>=2.3",
+  # used by `tractor._testing._reap` for the
+  # `tractor-reap` zombie-subactor + leaked-shm
+  # cleanup utility (xplatform `Process.memory_maps`,
+  # `Process.open_files`).
+  "psutil>=7.0.0",
 ]
 repl = [
   "pyperclip>=1.9.0",
@@ -234,15 +250,27 @@ testpaths = [
 addopts = [
   # TODO: figure out why this isn't working..
   '--rootdir=./tests',
-
   '--import-mode=importlib',
   # don't show frickin captured logs AGAIN in the report..
   '--show-capture=no',
 
+  # load builtin plugin since we need a boostrapping hook,
+  # `pytest_load_initial_conftests()` for `--capture=` per:
+  # https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
+  '-p tractor._testing.pytest',
+
   # disable `xonsh` plugin
   # https://docs.pytest.org/en/stable/how-to/plugins.html#disabling-plugins-from-autoloading
   # https://docs.pytest.org/en/stable/how-to/plugins.html#deactivating-unregistering-a-plugin-by-name
-  '-p no:xonsh'
+  '-p no:xonsh',
+
+  # XXX default on non-forking spawners
+  '--capture=fd',
+  # '--capture=sys',
+  # ^XXX NOTE^ ALWAYS SET THIS for `*_forkserver` spawner
+  # backends! see details @
+  # `tractor._testing.pytest.pytest_load_initial_conftests()`
+
 ]
 log_cli = false
 # TODO: maybe some of these layout choices?

scripts/tractor-reap

@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+#
+# SPDX-License-Identifier: AGPL-3.0-or-later
+'''
+`tractor-reap` — SC-polite zombie-subactor reaper +
+optional `/dev/shm/` orphan-segment sweep.
+
+Two cleanup phases (run in order when both are enabled):
+
+1. **process reap** — finds `tractor` subactor processes
+   left alive after a `pytest` (or any tractor-app) run
+   that failed to fully cancel its actor tree, then sends
+   SIGINT with a bounded grace window before escalating
+   to SIGKILL.
+
+2. **shm sweep** (`--shm` / `--shm-only`) — unlinks
+   `/dev/shm/<file>` entries owned by the current uid
+   that no live process has open (mmap'd or fd-held).
+   Needed because `tractor` disables
+   `mp.resource_tracker` (see `tractor.ipc._mp_bs`), so a
+   hard-crashing actor leaves leaked segments that
+   nothing else GCs.
+
+3. **UDS sweep** (`--uds` / `--uds-only`) — unlinks
+   `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` files
+   whose binder pid is dead (or the `1616` registry
+   sentinel). Needed because the IPC server's
+   `os.unlink()` cleanup lives in a `finally:` block
+   that doesn't always run on hard exits (SIGKILL,
+   escaped `KeyboardInterrupt`, etc.) — see issue #452.
+
+Process-reap detection modes (auto-selected):
+
+    --parent <pid>  : descendant-mode — kill procs whose
+                      PPid == <pid>. Use when a parent
+                      is still alive and you want to
+                      scope the sweep precisely (e.g.
+                      CI wrapper calling in from outside
+                      pytest).
+
+    (default)       : orphan-mode — kill procs with
+                      PPid==1 (init-reparented) whose
+                      cwd matches the repo root AND
+                      whose cmdline contains `python`.
+                      The cwd filter is what prevents
+                      sweeping unrelated init-children.
+
+Usage:
+
+    # process reap only (default)
+    scripts/tractor-reap
+
+    # process reap + shm sweep
+    scripts/tractor-reap --shm
+
+    # only the shm sweep, skip process reap
+    scripts/tractor-reap --shm-only
+
+    # process reap + shm + UDS sweep (the works)
+    scripts/tractor-reap --shm --uds
+
+    # only UDS sweep
+    scripts/tractor-reap --uds-only
+
+    # from inside a still-live supervisor
+    scripts/tractor-reap --parent 12345
+
+    # dry-run: list what would be reaped, don't act
+    scripts/tractor-reap -n
+    scripts/tractor-reap --shm --uds -n
+
+'''
+import argparse
+import pathlib
+import subprocess
+import sys
+
+
+def _repo_root() -> pathlib.Path:
+    '''
+    Use `git rev-parse --show-toplevel` when available;
+    fall back to the repo this script lives in.
+
+    '''
+    try:
+        out: str = subprocess.check_output(
+            ['git', 'rev-parse', '--show-toplevel'],
+            stderr=subprocess.DEVNULL,
+            text=True,
+        ).strip()
+        return pathlib.Path(out)
+    except (subprocess.CalledProcessError, FileNotFoundError):
+        return pathlib.Path(__file__).resolve().parent.parent
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        prog='tractor-reap',
+        description=__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        '--parent', '-p',
+        type=int,
+        default=None,
+        help='descendant-mode: reap procs with PPid==<pid>',
+    )
+    parser.add_argument(
+        '--grace', '-g',
+        type=float,
+        default=3.0,
+        help='SIGINT grace window in seconds (default 3.0)',
+    )
+    parser.add_argument(
+        '--dry-run', '-n',
+        action='store_true',
+        help='list matched pids/paths but do not signal/unlink',
+    )
+    parser.add_argument(
+        '--shm',
+        action='store_true',
+        help=(
+            'after process reap, also unlink orphaned '
+            '/dev/shm segments owned by the current user '
+            'that no live process is mapping or holding open'
+        ),
+    )
+    parser.add_argument(
+        '--shm-only',
+        action='store_true',
+        help='skip process reap; only do the shm sweep',
+    )
+    parser.add_argument(
+        '--uds',
+        action='store_true',
+        help=(
+            'after process reap, also unlink orphaned '
+            '${XDG_RUNTIME_DIR}/tractor/*.sock files '
+            'whose binder pid is dead (or the 1616 '
+            'registry sentinel). See issue #452.'
+        ),
+    )
+    parser.add_argument(
+        '--uds-only',
+        action='store_true',
+        help='skip process reap + shm; only do the UDS sweep',
+    )
+    args = parser.parse_args()
+    # any *-only flag also skips the process reap phase
+    skip_proc_reap: bool = (
+        args.shm_only
+        or
+        args.uds_only
+    )
+
+    # import lazily so `--help` doesn't require the tractor
+    # package to be importable (e.g. when running from a
+    # shell not inside a venv).
+    repo = _repo_root()
+    sys.path.insert(0, str(repo))
+    from tractor._testing._reap import (
+        find_descendants,
+        find_orphans,
+        find_orphaned_shm,
+        find_orphaned_uds,
+        reap,
+        reap_shm,
+        reap_uds,
+    )
+
+    rc: int = 0
+
+    # --- phase 1: process reap (skipped under --*-only) ---
+    if not skip_proc_reap:
+        if args.parent is not None:
+            pids: list[int] = find_descendants(args.parent)
+            mode: str = f'descendants of PPid={args.parent}'
+        else:
+            pids = find_orphans(repo)
+            mode = f'orphans (PPid=1, cwd={repo})'
+
+        if not pids:
+            print(f'[tractor-reap] no {mode} to reap')
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {mode}:\n  {pids}'
+            )
+        else:
+            _, survivors = reap(pids, grace=args.grace)
+            if survivors:
+                rc = 1
+
+    # --- phase 2: shm sweep (opt-in) ---
+    if args.shm or args.shm_only:
+        leaked: list[str] = find_orphaned_shm()
+        if not leaked:
+            print(
+                '[tractor-reap] no orphaned /dev/shm '
+                'segments to sweep'
+            )
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {len(leaked)} '
+                f'orphaned shm segment(s):\n  {leaked}'
+            )
+        else:
+            _, errors = reap_shm(leaked)
+            if errors:
+                rc = 1
+
+    # --- phase 3: UDS sweep (opt-in) ---
+    if args.uds or args.uds_only:
+        leaked_uds: list[str] = find_orphaned_uds()
+        if not leaked_uds:
+            print(
+                '[tractor-reap] no orphaned UDS sock-files '
+                'to sweep'
+            )
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {len(leaked_uds)} '
+                f'orphaned UDS sock-file(s):\n  {leaked_uds}'
+            )
+        else:
+            _, errors = reap_uds(leaked_uds)
+            if errors:
+                rc = 1
+
+    # exit 0 if everything cleaned cleanly, else 1 — useful
+    # for CI health-check chaining.
+    return rc
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

tests/conftest.py

@@ -22,7 +22,8 @@ from tractor._testing import (
 
 pytest_plugins: list[str] = [
     'pytester',
-    'tractor._testing.pytest',
+    # NOTE, now loaded in `pytest-ini` section of `pyproject.toml`
+    # 'tractor._testing.pytest',
 ]
 
 _ci_env: bool = os.environ.get('CI', False)
@@ -33,15 +34,10 @@ if platform.system() == 'Windows':
     _KILL_SIGNAL = signal.CTRL_BREAK_EVENT
     _INT_SIGNAL = signal.CTRL_C_EVENT
     _INT_RETURN_CODE = 3221225786
-    _PROC_SPAWN_WAIT = 2
 else:
     _KILL_SIGNAL = signal.SIGKILL
     _INT_SIGNAL = signal.SIGINT
     _INT_RETURN_CODE = 1 if sys.version_info < (3, 8) else -signal.SIGINT.value
-    _PROC_SPAWN_WAIT = (
-        2 if _ci_env
-        else 1
-    )
 
 
 no_windows = pytest.mark.skipif(
@@ -100,91 +96,51 @@ def cpu_scaling_factor() -> float:
     much to inflate time-limits when CPU-freq scaling is active on
     linux.
 
-    When no scaling info is available (non-linux, missing sysfs),
+    When no local scaling info is available (non-linux, missing
-    returns 1.0 (i.e. no headroom adjustment needed).
+    sysfs) the base factor is 1.0; a flat CI bump is then applied
+    on top (see below).
 
     '''
-    if _non_linux:
+    factor: float = 1.
-        return 1.
+    if not _non_linux:
-
         mx = get_cpu_state()
         cur = get_cpu_state(setting='scaling_max_freq')
-    if mx is None or cur is None:
+        if (
-        return 1.
+            mx is not None
-
+            and
+            cur is not None
+        ):
             _mx_pth, max_freq = mx
             _cur_pth, cur_freq = cur
             cpu_scaled: float = int(cur_freq) / int(max_freq)
-
             if cpu_scaled != 1.:
-        return 1. / (
+                factor = 1. / (
                     cpu_scaled * 2  # <- bc likely "dual threaded"
                 )
 
-    return 1.
+    # XXX, GH Actions (and most shared) CI runners are slow + noisy
+    # and — unlike a throttled local box — do NOT expose CPU-freq
+    # scaling via sysfs, so the probe above reads 1.0 and adds no
+    # headroom. Apply a flat CI bump so every timing-test deadline
+    # /assert that keys off this factor gets headroom on CI HW
+    # (compounds with any local-throttle factor).
+    #
+    # macOS runners are noticeably slower + noisier than the linux
+    # ones for our multi-actor cancel-cascade tests, so give them
+    # extra headroom (3x vs 2x).
+    if _ci_env:
+        factor *= 3 if _non_linux else 2
+
+    return factor
 
 
-def pytest_addoption(
+# NOTE, the `--ll`/`--tl` CLI flags + the `loglevel`, `test_log`
-    parser: pytest.Parser,
+# and `testing_pkg_name` fixtures have been factored into the
-):
+# `tractor._testing.pytest` plugin (loaded via the `-p` entry in
-    # ?TODO? should this be exposed from our `._testing.pytest`
+# `pyproject.toml`'s `[tool.pytest.ini_options]`) so downstream
-    # plugin or should we make it more explicit with `--tl` for
+# consuming projects (eg. `modden`) inherit them for free. The
-    # tractor logging like we do in other client projects?
+# plugin's `testing_pkg_name` fixture defaults to `'tractor'`, so
-    parser.addoption(
+# this suite keeps treating `--ll` as the runtime loglevel.
-        "--ll",
-        action="store",
-        dest='loglevel',
-        default='ERROR', help="logging level to set when testing"
-    )
-
-
-@pytest.fixture(scope='session', autouse=True)
-def loglevel(request) -> str:
-    import tractor
-    orig = tractor.log._default_loglevel
-    level = tractor.log._default_loglevel = request.config.option.loglevel
-    log = tractor.log.get_console_log(
-        level=level,
-        name='tractor',  # <- enable root logger
-    )
-    log.info(
-        f'Test-harness set runtime loglevel: {level!r}\n'
-    )
-    yield level
-    tractor.log._default_loglevel = orig
-
-
-@pytest.fixture(scope='function')
-def test_log(
-    request,
-    loglevel: str,
-) -> tractor.log.StackLevelAdapter:
-    '''
-    Deliver a per test-module-fn logger instance for reporting from
-    within actual test bodies/fixtures.
-
-    For example this can be handy to report certain error cases from
-    exception handlers using `test_log.exception()`.
-
-    '''
-    modname: str = request.function.__module__
-    log = tractor.log.get_logger(
-        name=modname,  # <- enable root logger
-        # pkg_name='tests',
-    )
-    _log = tractor.log.get_console_log(
-        level=loglevel,
-        logger=log,
-        name=modname,
-        # pkg_name='tests',
-    )
-    _log.debug(
-        f'In-test-logging requested\n'
-        f'test_log.name: {log.name!r}\n'
-        f'level: {loglevel!r}\n'
-
-    )
-    yield _log
 
 
 @pytest.fixture(scope='session')
@@ -236,107 +192,14 @@ def sig_prog(
     assert ret
 
 
-# TODO: factor into @cm and move to `._testing`?
+# NOTE, the `daemon` fixture (+ its `_wait_for_daemon_ready`
-@pytest.fixture
+# helper + the post-yield teardown drain logic) has been
-def daemon(
+# moved to `tests/discovery/conftest.py` since 100% of its
-    debug_mode: bool,
+# consumers are discovery-protocol tests now living under
-    loglevel: str,
+# that subdir. See:
-    testdir: pytest.Pytester,
+# - `tests/discovery/test_multi_program.py`
-    reg_addr: tuple[str, int],
+# - `tests/discovery/test_registrar.py`
-    tpt_proto: str,
+# - `tests/discovery/test_tpt_bind_addrs.py`
-    ci_env: bool,
-    test_log: tractor.log.StackLevelAdapter,
-
-) -> subprocess.Popen:
-    '''
-    Run a daemon root actor as a separate actor-process tree and
-    "remote registrar" for discovery-protocol related tests.
-
-    '''
-    if loglevel in ('trace', 'debug'):
-        # XXX: too much logging will lock up the subproc (smh)
-        loglevel: str = 'info'
-
-    code: str = (
-        "import tractor; "
-        "tractor.run_daemon([], "
-        "registry_addrs={reg_addrs}, "
-        "enable_transports={enable_tpts}, "
-        "debug_mode={debug_mode}, "
-        "loglevel={ll})"
-    ).format(
-        reg_addrs=str([reg_addr]),
-        enable_tpts=str([tpt_proto]),
-        ll="'{}'".format(loglevel) if loglevel else None,
-        debug_mode=debug_mode,
-    )
-    cmd: list[str] = [
-        sys.executable,
-        '-c', code,
-    ]
-    # breakpoint()
-    kwargs = {}
-    if platform.system() == 'Windows':
-        # without this, tests hang on windows forever
-        kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
-
-    proc: subprocess.Popen = testdir.popen(
-        cmd,
-        **kwargs,
-    )
-
-    # TODO! we should poll for the registry socket-bind to take place
-    # and only once that's done yield to the requester!
-    # -[ ] TCP: use the `._root.open_root_actor()`::`ping_tpt_socket()`
-    #      closure!
-    # -[ ] UDS: can we do something similar for 'pinging" the
-    #     file-socket?
-    #
-    global _PROC_SPAWN_WAIT
-    # UDS sockets are **really** fast to bind()/listen()/connect()
-    # so it's often required that we delay a bit more starting
-    # the first actor-tree..
-    if tpt_proto == 'uds':
-        _PROC_SPAWN_WAIT += 1.6
-
-    if _non_linux and ci_env:
-        _PROC_SPAWN_WAIT += 1
-
-    # XXX, allow time for the sub-py-proc to boot up.
-    # !TODO, see ping-polling ideas above!
-    time.sleep(_PROC_SPAWN_WAIT)
-
-    assert not proc.returncode
-    yield proc
-    sig_prog(proc, _INT_SIGNAL)
-
-    # XXX! yeah.. just be reaaal careful with this bc sometimes it
-    # can lock up on the `_io.BufferedReader` and hang..
-    stderr: str = proc.stderr.read().decode()
-    stdout: str = proc.stdout.read().decode()
-    if (
-        stderr
-        or
-        stdout
-    ):
-        print(
-            f'Daemon actor tree produced output:\n'
-            f'{proc.args}\n'
-            f'\n'
-            f'stderr: {stderr!r}\n'
-            f'stdout: {stdout!r}\n'
-        )
-
-    if (rc := proc.returncode) != -2:
-        msg: str = (
-            f'Daemon actor tree was not cancelled !?\n'
-            f'proc.args: {proc.args!r}\n'
-            f'proc.returncode: {rc!r}\n'
-        )
-        if rc < 0:
-            raise RuntimeError(msg)
-
-        test_log.error(msg)
 
 
 # @pytest.fixture(autouse=True)

tests/devx/conftest.py

@@ -4,6 +4,7 @@
 '''
 from __future__ import annotations
 import platform
+import os
 import signal
 import time
 from typing import (
@@ -56,6 +57,7 @@ type PexpectSpawner = Callable[
 @pytest.fixture
 def spawn(
     start_method: str,
+    loglevel: str,
     testdir: pytest.Pytester,
     reg_addr: tuple[str, int],
 
@@ -65,9 +67,19 @@ def spawn(
     run an `./examples/..` script by name.
 
     '''
-    if start_method != 'trio':
+    supported_spawners: set[str] = {
+        'trio',
+        # `examples/debugging/<script>.py` picks up the spawn
+        # backend via the `TRACTOR_SPAWN_METHOD` env-var which
+        # is honored inside `tractor._root.open_root_actor()`,
+        # so no per-script edits are required.
+        'main_thread_forkserver',
+        'subint_forkserver',
+    }
+    if start_method not in supported_spawners:
         pytest.skip(
-            '`pexpect` based tests only supported on `trio` backend'
+            f'`pexpect` based tests NOT supported on spawning-backend: {start_method!r}\n'
+            f'supported-spawners: {supported_spawners!r}'
         )
 
     def unset_colors():
@@ -79,21 +91,64 @@ def spawn(
         https://docs.python.org/3/using/cmdline.html#using-on-controlling-color
 
         '''
-        import os
         # disable colored tbs
         os.environ['PYTHON_COLORS'] = '0'
         # disable all ANSI color output
         # os.environ['NO_COLOR'] = '1'
+        # ?TODO, doesn't seem to disable prompt color
+        # for `pdbp`?
+
+    def set_spawn_method(
+        start_method: str,
+    ):
+        '''
+        Drive the actor-spawn backend inside the spawned
+        `examples/debugging/<script>.py` subproc via env-var
+        (consumed by `tractor._root.open_root_actor()`),
+        without requiring per-script CLI plumbing.
+
+        '''
+        os.environ['TRACTOR_SPAWN_METHOD'] = start_method
+
+    def set_loglevel(
+        loglevel: str|None,
+    ):
+        '''
+        Forward the test-suite parametrized `loglevel` into the
+        spawned `examples/debugging/<script>.py` subproc via
+        env-var (consumed by `tractor._root.open_root_actor()`),
+        so console verbosity can be cranked or silenced from
+        the test harness without per-script edits.
+
+        '''
+        if loglevel:
+            os.environ['TRACTOR_LOGLEVEL'] = loglevel
+        else:
+            os.environ.pop('TRACTOR_LOGLEVEL', None)
 
     spawned: PexpectSpawner|None = None
 
     def _spawn(
         cmd: str,
         expect_timeout: float = 4,
+        start_method: str = start_method,
+        loglevel: str|None = None,
         **mkcmd_kwargs,
     ) -> pty_spawn.spawn:
+        '''
+        Inner closure handed to consumer tests to invoke
+        `pytest.Pytester.spawn`
+
+        '''
         nonlocal spawned
         unset_colors()
+        set_spawn_method(start_method=start_method)
+        set_loglevel(
+            loglevel=loglevel,
+            # ?TODO^ when should this be set by `--ll <level>` ?
+            # by default we apply 'error' but there should be a diff
+            # vs. when the flag IS NOT passed?
+        )
         spawned = testdir.spawn(
             cmd=mk_cmd(
                 cmd,
@@ -137,6 +192,14 @@ def spawn(
         if ptyproc.isalive():
             ptyproc.kill(signal.SIGKILL)
 
+    # Scope our env-var mutations to this single fixture invocation
+    # — both `TRACTOR_SPAWN_METHOD` and `TRACTOR_LOGLEVEL` are
+    # honored by `tractor._root.open_root_actor()` so leaking them
+    # past this test could inadvertently re-route a later in-process
+    # tractor test's spawn-backend / loglevel.
+    os.environ.pop('TRACTOR_SPAWN_METHOD', None)
+    os.environ.pop('TRACTOR_LOGLEVEL', None)
+
     # TODO? ensure we've cleaned up any UDS-paths?
     # breakpoint()
 
@@ -146,24 +209,40 @@ def spawn(
     ids='ctl-c={}'.format,
 )
 def ctlc(
-    request,
+    request: pytest.FixtureRequest,
     ci_env: bool,
-
+    start_method: str,
 ) -> bool:
+    '''
+    Parametrize and optionally skip tests which handle
+    ctlc-in-`pdbp`-REPL testing scenarios; certain spawners and actor-tree depths
+    cope very poorly with this..
 
-    use_ctlc = request.param
+    In particular the spawning backends from `multiprocessing` are
+    fragile, as can be the default `trio` spawner under certain
+    conditions where SIGINT is relayed down the entire subproc tree.
 
+    '''
+    use_ctlc: bool = request.param
     node = request.node
     markers = node.own_markers
     for mark in markers:
-        if mark.name == 'has_nested_actors':
+        if (
+            mark.name == 'has_nested_actors'
+            and
+            start_method not in {
+                # TODO, any spawners we should try again?
+                # - [ ] 'trio' but WITHOUT the SIGINT handler setup
+                #      per subproc?
+                # 'main_thread_forkserver',
+            }
+        ):
             pytest.skip(
                 f'Test {node} has nested actors and fails with Ctrl-C.\n'
                 f'The test can sometimes run fine locally but until'
                 ' we solve' 'this issue this CI test will be xfail:\n'
                 'https://github.com/goodboy/tractor/issues/320'
             )
-
         if (
             mark.name == 'ctlcs_bish'
             and
@@ -190,13 +269,10 @@ def ctlc(
 
 def expect(
     child,
-
+    patt: str,  # often a `pdbp`-prompt
-    # normally a `pdb` prompt by default
-    patt: str,
-
     **kwargs,
 
-) -> None:
+) -> str:
     '''
     Expect wrapper that prints last seen console
     data before failing.
@@ -207,6 +283,8 @@ def expect(
             patt,
             **kwargs,
         )
+        before = str(child.before.decode())
+        return before
     except TIMEOUT:
         before = str(child.before.decode())
         print(before)
@@ -261,10 +339,13 @@ def in_prompt_msg(
 def assert_before(
     child: SpawnBase,
     patts: list[str],
-
     **kwargs,
+) -> str:
+    '''
+    Assert a patter is in `child.before.decode() -> str`,
+    return the full `.before` output on success.
 
-) -> None:
+    '''
     __tracebackhide__: bool = False
 
     assert in_prompt_msg(
@@ -275,7 +356,8 @@ def assert_before(
         err_on_false=True,
         **kwargs
     )
-    return str(child.before.decode())
+    before: str = str(child.before.decode())
+    return before
 
 
 def do_ctlc(

tests/devx/test_debugger.py

@@ -24,6 +24,7 @@ from pexpect.exceptions import (
     TIMEOUT,
     EOF,
 )
+import tractor
 
 from .conftest import (
     do_ctlc,
@@ -343,6 +344,7 @@ def test_subactor_breakpoint(
 def test_multi_subactors(
     spawn,
     ctlc: bool,
+    set_fork_aware_capture,
 ):
     '''
     Multiple subactors, both erroring and
@@ -487,11 +489,12 @@ def test_multi_subactors(
 def test_multi_daemon_subactors(
     spawn,
     loglevel: str,
-    ctlc: bool
+    ctlc: bool,
+    set_fork_aware_capture,
 ):
     '''
-    Multiple daemon subactors, both erroring and breakpointing within a
+    Multiple daemon subactors, both erroring and breakpointing within
-    stream.
+    a stream.
 
     '''
     non_linux = _non_linux
@@ -604,7 +607,10 @@ def test_multi_daemon_subactors(
             child,
             bp_forev_parts,
         )
-    except AssertionError:
+    except (
+        # AssertionError,  # TODO? rm since never raised?
+        ValueError,
+    ):
         before: str = assert_before(
             child,
             name_error_parts,
@@ -765,6 +771,8 @@ def test_multi_subactors_root_errors(
 def test_multi_nested_subactors_error_through_nurseries(
     ci_env: bool,
     spawn: PexpectSpawner,
+    is_forking_spawner: bool,
+    test_log: tractor.log.StackLevelAdapter,
 
     # TODO: address debugger issue for nested tree:
     # https://github.com/goodboy/tractor/issues/320
@@ -781,16 +789,17 @@ def test_multi_nested_subactors_error_through_nurseries(
     # A test (below) has now been added to explicitly verify this is
     # fixed.
 
-    child = spawn('multi_nested_subactors_error_up_through_nurseries')
+    child = spawn(
-
+        'multi_nested_subactors_error_up_through_nurseries',
-    # timed_out_early: bool = False
+        loglevel='pdb',
-
+    )
+    last_send_char: str|None = None
     for (
         i,
         send_char,
     ) in enumerate(itertools.cycle(['c', 'q'])):
 
-        timeout: float = -1
+        timeout: float = child.timeout
         if (
             _non_linux
             and
@@ -803,24 +812,36 @@ def test_multi_nested_subactors_error_through_nurseries(
         elif i == 0:
             timeout = 5
 
+        # XXX forking backends may take longer due to
+        # determinstic IPC cancellation.
+        if is_forking_spawner:
+            timeout += 4
+
         try:
             child.expect(
                 PROMPT,
                 timeout=timeout,
             )
+            delay: float = 0.1
+            test_log.info('Sleeping {delay!r} before next send-chart..')
+            time.sleep(delay)
+            last_send_char: str = send_char
             child.sendline(send_char)
-            time.sleep(0.01)
+            time.sleep(delay)
 
+        # script finally exited with tb on console.
         except EOF:
+            test_log.info(
+                f'Breaking from send-char loop'
+                f'last_send_char: {last_send_char!r}\n'
+            )
             break
 
-    assert_before(
+    # boxed source errors
-        child,
+    expect_patts: list[str] = [
-        [ # boxed source errors
         "NameError: name 'doggypants' is not defined",
         "tractor._exceptions.RemoteActorError:",
         "('name_error'",
-            "bdb.BdbQuit",
 
         # first level subtrees
         # "tractor._exceptions.RemoteActorError: ('spawner0'",
@@ -834,18 +855,39 @@ def test_multi_nested_subactors_error_through_nurseries(
         # "tractor._exceptions.RemoteActorError: ('spawn_until_2'",
         # ^-NOTE-^ old RAE repr, new one is below with a field
         # showing the src actor's uid.
-            "src_uid=('spawn_until_0'",
-            "relay_uid=('spawn_until_1'",
         "src_uid=('spawn_until_2'",
     ]
+    # XXX, I HAVE NO IDEA why these patts only show on the
+    # `trio`-spawner but it seems to have something to do with
+    # what gets dumped in prior-prompt latches somehow??
+    # TODO for claude, explain and or work through how this is
+    # happening but ONLY WHEN RUN FROM THE TEST, bc when i try to
+    # run the test script manually the correct output ALWAYS seems
+    # to be in the last `str(child.before.decode())` output !?!?
+    if (
+        not is_forking_spawner
+        and
+        last_send_char == 'q'
+    ):
+        expect_patts += [
+            # expect the pdb-quit exc.
+            "bdb.BdbQuit",
+            # BUT WHY these dude!?
+            "src_uid=('spawn_until_0'",
+            "relay_uid=('spawn_until_1'",
+        ]
+
+    assert_before(
+        child,
+        expect_patts,
     )
+    expect(child, EOF)
 
 
-@pytest.mark.timeout(15)
+# @pytest.mark.timeout(15)
 @has_nested_actors
 def test_root_nursery_cancels_before_child_releases_tty_lock(
     spawn,
-    start_method,
     ctlc: bool,
 ):
     '''
@@ -1187,7 +1229,11 @@ def test_ctxep_pauses_n_maybe_ipc_breaks(
     mashed and zombie reaper kills sub with no hangs.
 
     '''
-    child = spawn('subactor_bp_in_ctx')
+    child = spawn(
+        'subactor_bp_in_ctx',
+        loglevel='devx'
+        # ^XXX REQUIRED for below patt matching!
+    )
     child.expect(PROMPT)
 
     # 3 iters for the `gen()` pause-points
@@ -1277,7 +1323,11 @@ def test_crash_handling_within_cancelled_root_actor(
     call.
 
     '''
-    child = spawn('root_self_cancelled_w_error')
+    child = spawn(
+        'root_self_cancelled_w_error',
+        loglevel='cancel',
+        # ^XXX REQUIRED for below patt matching!
+    )
     child.expect(PROMPT)
 
     assert_before(

tests/devx/test_pause_from_non_trio.py

@@ -66,19 +66,28 @@ def test_pause_from_sync(
     # XXX required for `breakpoint()` overload and
     # thus`tractor.devx.pause_from_sync()`.
     pytest.importorskip('greenback')
-    child = spawn('sync_bp')
+    child = spawn(
+        'sync_bp',
+        loglevel='pdb',  # XXX pattern matching
+    )
 
     # first `sync_pause()` after nurseries open
     child.expect(PROMPT)
-    assert_before(
+    _before: str = assert_before(
         child,
         [
-            # pre-prompt line
+            # devx-loglevel
-            _pause_msg,
+            # "imported <module 'greenback' from",
-            "<Task '__main__.main'",
+            # "successfully scheduled `._pause()` in `trio` thread on behalf of <Task",
+
+            _pause_msg,  # pre-prompt line
             "('root'",
+            "<Task '__main__.main'",
+            "tractor.pause_from_sync()",
         ]
     )
+    # XXX `enable_stack_on_sig=False` in script
+    assert 'stackscope' not in _before
     if ctlc:
         do_ctlc(child)
         # ^NOTE^ subactor not spawned yet; don't need extra delay.
@@ -88,18 +97,18 @@ def test_pause_from_sync(
     # first `await tractor.pause()` inside `p.open_context()` body
     child.expect(PROMPT)
 
-    # XXX shouldn't see gb loaded message with PDB loglevel!
-    # assert not in_prompt_msg(
-    #     child,
-    #     ['`greenback` portal opened!'],
-    # )
     # should be same root task
     assert_before(
         child,
         [
+            # XXX should see gb loaded with devx-loglevel.
+            # "`greenback` portal opened!",
+            # "Activated `greenback` for `tractor.pause_from_sync()` support!",
+
             _pause_msg,
-            "<Task '__main__.main'",
             "('root'",
+            "<Task '__main__.main'",
+            "tractor.pause()",
         ]
     )
 
@@ -130,17 +139,17 @@ def test_pause_from_sync(
     # `Lock.acquire()`-ed
     # (NOT both, which will result in REPL clobbering!)
     attach_patts: dict[str, list[str]] = {
-        'subactor': [
+        "|_<Task 'start_n_sync_pause'": [
-            "'start_n_sync_pause'",
+            "|_('subactor'",
-            "('subactor'",
+            "tractor.pause_from_sync()",
         ],
-        'inline_root_bg_thread': [
+        "|_<Thread(inline_root_bg_thread": [
-            "<Thread(inline_root_bg_thread",
             "('root'",
+            "breakpoint(hide_tb=hide_tb)",
         ],
-        'start_soon_root_bg_thread': [
+        "|_<Thread(start_soon_root_bg_thread": [
-            "<Thread(start_soon_root_bg_thread",
+            "|_('root'",
-            "('root'",
+            "tractor.pause_from_sync()",
         ],
     }
     conts: int = 0  # for debugging below matching logic on failure

tests/devx/test_proctitle.py

@@ -0,0 +1,170 @@
+'''
+Tests for `tractor.devx._proctitle` (per-actor `setproctitle`)
+and the intrinsic-signal sub-actor detection in
+`tractor._testing._reap`.
+
+The proctitle is set in `tractor._child._actor_child_main()`
+after `Actor` construction, so any spawned sub-actor process
+should:
+
+  - have `argv[0]` (== `/proc/<pid>/cmdline`) start with
+    `tractor[<aid.reprol()>]`
+  - have `/proc/<pid>/comm` start with `tractor[` (kernel
+    truncates to ~15 bytes)
+  - be detected as a tractor sub-actor by
+    `_is_tractor_subactor(pid)` via the cmdline marker.
+
+`set_actor_proctitle()` itself is also unit-tested in-process
+to verify the format string.
+
+'''
+from __future__ import annotations
+import platform
+
+import psutil
+import pytest
+import trio
+import tractor
+
+from tractor.runtime._runtime import Actor
+from tractor.devx._proctitle import set_actor_proctitle
+from tractor._testing._reap import (
+    _is_tractor_subactor,
+    _read_cmdline,
+    _read_comm,
+)
+
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+def test_set_actor_proctitle_format():
+    '''
+    `set_actor_proctitle()` returns the canonical
+    `tractor[<aid.reprol()>]` form and actually mutates
+    the running proc's title.
+
+    '''
+    pytest.importorskip(
+        'setproctitle',
+        reason='`setproctitle` is an optional runtime dep',
+    )
+    import setproctitle
+
+    # save + restore so we don't pollute pytest's own title
+    saved: str = setproctitle.getproctitle()
+    try:
+        actor = Actor(
+            name='unit_test_actor',
+            uuid='1027301b-a0e3-430e-8806-a5279f21abe6',
+        )
+        title: str = set_actor_proctitle(actor)
+
+        # canonical wrapping: `tractor[<aid.reprol()>]`. We
+        # compare against the runtime-computed `reprol()`
+        # rather than a hard-coded value so the test stays
+        # decoupled from `Aid.reprol()`'s internal format
+        # (currently `<name>@<pid>`, but could evolve).
+        expected: str = f'tractor[{actor.aid.reprol()}]'
+        assert title == expected
+        # sanity: the actor's name must be in the title
+        # somewhere (so a future `reprol()` change that
+        # drops the name is also caught).
+        assert 'unit_test_actor' in title
+
+        # actually set on the running proc
+        assert setproctitle.getproctitle() == title
+
+    finally:
+        setproctitle.setproctitle(saved)
+
+
+@pytest.mark.skipif(
+    _non_linux,
+    reason=(
+        'detection helpers read `/proc/<pid>/{cmdline,comm}` '
+        'which is Linux-specific'
+    ),
+)
+def test_subactor_proctitle_visible_via_proc():
+    '''
+    Spawn a sub-actor and verify its proc-title is visible
+    via both `/proc/<pid>/cmdline` AND `/proc/<pid>/comm`,
+    AND that `_is_tractor_subactor()` correctly identifies
+    it.
+
+    '''
+    pytest.importorskip('setproctitle')
+
+    async def main() -> dict:
+        async with tractor.open_nursery() as an:
+            portal = await an.start_actor('proctitle_boi')
+            # let the child finish setproctitle in
+            # `_actor_child_main`
+            await trio.sleep(0.3)
+
+            # the sub-actor's pid is on the portal's chan
+            # repr; psutil-walk `me.children()` is simpler.
+            me = psutil.Process()
+            sub_pids: list[int] = [
+                p.pid for p in me.children(recursive=True)
+            ]
+            assert sub_pids, (
+                'expected at least one spawned sub-actor pid'
+            )
+
+            results: dict = {}
+            for pid in sub_pids:
+                results[pid] = {
+                    'cmdline': _read_cmdline(pid),
+                    'comm': _read_comm(pid),
+                    'is_tractor': _is_tractor_subactor(pid),
+                }
+
+            await portal.cancel_actor()
+            return results
+
+    found: dict = trio.run(main)
+
+    # at least one of the spawned procs should match the
+    # `proctitle_boi` actor we started; assert the proc-
+    # title shape on it specifically.
+    matched: list[tuple[int, dict]] = [
+        (pid, info)
+        for pid, info in found.items()
+        if 'proctitle_boi' in info['cmdline']
+    ]
+    assert matched, (
+        f'no sub-actor pid had a `proctitle_boi` cmdline; '
+        f'all={found}'
+    )
+
+    pid, info = matched[0]
+    # canonical proctitle prefix in cmdline (full form)
+    assert info['cmdline'].startswith('tractor[proctitle_boi@'), (
+        f'cmdline missing `tractor[proctitle_boi@…]` prefix: '
+        f'{info["cmdline"]!r}'
+    )
+    # comm is kernel-truncated to ~15 bytes — just check the
+    # `tractor[` prefix made it.
+    assert info['comm'].startswith('tractor['), (
+        f'comm missing `tractor[` prefix: {info["comm"]!r}'
+    )
+    # intrinsic-signal detector should match.
+    assert info['is_tractor'] is True
+
+
+@pytest.mark.skipif(
+    _non_linux,
+    reason='reads /proc/<pid>/{cmdline,comm}',
+)
+def test_is_tractor_subactor_negative():
+    '''
+    `_is_tractor_subactor()` returns False for non-tractor
+    procs (e.g. the pytest test-runner pid itself, which
+    is `python -m pytest …` — no `tractor[` proctitle, no
+    `tractor._child` cmdline).
+
+    '''
+    import os
+    assert _is_tractor_subactor(os.getpid()) is False

tests/devx/test_tooling.py

@@ -21,6 +21,7 @@ import os
 import signal
 import time
 from typing import (
+    Callable,
     TYPE_CHECKING,
 )
 
@@ -47,7 +48,12 @@ if TYPE_CHECKING:
 
 @no_macos
 def test_shield_pause(
-    spawn: PexpectSpawner,
+    spawn: Callable[
+        ...,
+        PexpectSpawner,
+    ],
+    start_method: str,
+    request: pytest.FixtureRequest,
 ):
     '''
     Verify the `tractor.pause()/.post_mortem()` API works inside an
@@ -55,8 +61,10 @@ def test_shield_pause(
     next checkpoint wherein the cancelled will get raised.
 
     '''
-    child = spawn(
+    child: PexpectSpawner = spawn(
-        'shield_hang_in_sub'
+        'shield_hang_in_sub',
+        loglevel='devx',
+        # ^XXX REQUIRED for below patt matching!
     )
     expect(
         child,
@@ -86,31 +94,74 @@ def test_shield_pause(
         # end-of-tree delimiter
         "end-of-\('root'",
     )
-    assert_before(
+    _before: str = assert_before(
         child,
         [
             # 'Srying to dump `stackscope` tree..',
             # 'Dumping `stackscope` tree for actor',
             "('root'",  # uid line
 
-            # TODO!? this used to show?
+            # TODO!? this in-task-code used to show??
             # -[ ] mk reproducable for @oremanj?
+            # => SOLVED? by our `trio_token.run_sync_soon()`
+            #    approach?
             #
             # parent block point (non-shielded)
             # 'await trio.sleep_forever()  # in root',
         ]
     )
+
+    # NOTE, hierarchical-ordering invariant restored by
+    # `_dump_then_relay` (co-scheduled dump+relay on the
+    # trio loop, see `tractor.devx._stackscope`): the
+    # parent's full task-tree prints BEFORE the 'Relaying
+    # `SIGUSR1`' log msg, which prints BEFORE any sub-
+    # actor receives the signal and dumps its own tree.
+    # So the relay log appears BETWEEN `end-of-('root'`
+    # (above) and `end-of-('hanger'` (below).
+    handle_out_of_order: bool = False
+
+    # XXX, when capfd is NOT used we don't expect to
+    # see the logging output from the subactor.
+    if (no_capfd := (start_method in [
+            'main_thread_forkserver',
+        ])
+    ):
+        opts = request.config.option
+        assert opts.spawn_backend == start_method
+        # ?XXX? i guess the `testdir` fixture "pretends to" reset
+        # this to the default 'fd'??
+        # assert opts.capture in [
+        #     'sys',
+        #     'no',
+        # ]
+
+    if (
+        handle_out_of_order
+        and
+        "end-of-('hanger'" in _before
+    ):
+         assert "('hanger'" in _before
+         assert 'Relaying `SIGUSR1`[10] to sub-actor' in _before
+
+    else:
+        _before = expect(
+            child,
+            'Relaying `SIGUSR1`\\[10\\] to sub-actor',
+        )
+        # _before: str = assert_before(
+        #     child,
+        #     ["('hanger'",]  # uid line
+        # )
+        if not no_capfd:
             expect(
                 child,
-        # end-of-tree delimiter
+                # end-of-subactor's-tree delimiter
                 "end-of-\('hanger'",
             )
-    assert_before(
+            _before: str = assert_before(
                 child,
                 [
-            # relay to the sub should be reported
-            'Relaying `SIGUSR1`[10] to sub-actor',
-
                     "('hanger'",  # uid line
 
                     # TODO!? SEE ABOVE
@@ -119,6 +170,7 @@ def test_shield_pause(
                 ]
             )
 
+
     # simulate the user sending a ctl-c to the hanging program.
     # this should result in the terminator kicking in since
     # the sub is shield blocking and can't respond to SIGINT.
@@ -133,14 +185,19 @@ def test_shield_pause(
         _shutdown_msg,
         timeout=6,
     )
-    assert_before(
+    expect_on_teardown: list[str] = [
-        child,
-        [
         'raise KeyboardInterrupt',
+        'Root actor terminated',
+    ]
+    if not no_capfd:
+        expect_on_teardown += [
             # 'Shutting down actor runtime',
             '#T-800 deployed to collect zombie B0',
             "'--uid', \"('hanger',",
         ]
+    assert_before(
+        child,
+        expect_on_teardown,
     )
 
 

tests/discovery/conftest.py

@@ -0,0 +1,223 @@
+'''
+Discovery-suite fixtures, including the `daemon`
+remote-registrar subprocess used by the multi-program
+discovery tests.
+
+Lives here (vs. the parent `tests/conftest.py`)
+because `daemon` is a discovery-protocol primitive —
+boots a separate `tractor.run_daemon()` process whose
+sole purpose is to serve as a registrar peer for
+discovery-roundtrip tests. Pytest fixtures inherit
+DOWNWARD through conftest hierarchy, so anything
+under `tests/discovery/` automatically picks this up.
+
+'''
+from __future__ import annotations
+import os
+import platform
+import socket
+import subprocess
+import sys
+import time
+
+import pytest
+import tractor
+
+from ..conftest import (
+    sig_prog,
+    _INT_SIGNAL,
+    _non_linux,
+)
+
+
+def _wait_for_daemon_ready(
+    reg_addr: tuple,
+    tpt_proto: str,
+    *,
+    deadline: float = 10.0,
+    poll_interval: float = 0.05,
+    proc: subprocess.Popen|None = None,
+) -> None:
+    '''
+    Active-poll the daemon's bind address until it
+    accepts a connection (proving it has called
+    `bind() + listen()` and is ready to handle IPC).
+
+    Replaces the historical blind `time.sleep()` in the
+    `daemon` fixture which was racy under load — see
+    `ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`.
+
+    Uses stdlib `socket` directly (no trio runtime
+    bootstrap cost) — sufficient because
+    `tractor.run_daemon()` doesn't return from
+    bootstrap until the runtime is fully ready to
+    accept IPC.
+
+    Raises `TimeoutError` on `deadline` exceeded. If
+    `proc` is given, ALSO raises early if the daemon
+    process exits non-zero before the deadline (catches
+    daemon-startup-crash that the blind sleep used to
+    silently mask).
+
+    '''
+    end: float = time.monotonic() + deadline
+    last_exc: Exception|None = None
+    while time.monotonic() < end:
+        # Daemon-died-during-startup early-exit. Without
+        # this, a crashed-on-import daemon would just
+        # eat the full deadline before raising opaque
+        # TimeoutError.
+        if proc is not None and proc.poll() is not None:
+            raise RuntimeError(
+                f'Daemon proc exited (rc={proc.returncode}) '
+                f'before becoming ready to accept on '
+                f'{reg_addr!r}'
+            )
+        try:
+            if tpt_proto == 'tcp':
+                # `socket.create_connection` does the
+                # `socket() + connect()` dance with a
+                # builtin timeout — perfect primitive
+                # for a one-shot probe.
+                with socket.create_connection(
+                    reg_addr,
+                    timeout=poll_interval,
+                ):
+                    return
+            else:
+                # UDS — `reg_addr` is a `(filedir, sockname)`
+                # tuple per `tractor.ipc._uds.UDSAddress.unwrap`.
+                sockpath: str = os.path.join(*reg_addr)
+                sock = socket.socket(socket.AF_UNIX)
+                try:
+                    sock.settimeout(poll_interval)
+                    sock.connect(sockpath)
+                    return
+                finally:
+                    sock.close()
+        except (
+            ConnectionRefusedError,
+            FileNotFoundError,
+            OSError,
+            socket.timeout,
+        ) as exc:
+            last_exc = exc
+            time.sleep(poll_interval)
+    raise TimeoutError(
+        f'Daemon never accepted on {reg_addr!r} within '
+        f'{deadline}s (last connect-attempt exc: '
+        f'{last_exc!r})'
+    )
+
+
+# TODO: factor into @cm and move to `._testing`?
+@pytest.fixture
+def daemon(
+    debug_mode: bool,
+    loglevel: str,
+    testdir: pytest.Pytester,
+    reg_addr: tuple[str, int],
+    tpt_proto: str,
+    ci_env: bool,
+    test_log: tractor.log.StackLevelAdapter,
+
+) -> subprocess.Popen:
+    '''
+    Run a daemon root actor as a separate actor-process
+    tree and "remote registrar" for discovery-protocol
+    related tests.
+
+    '''
+    # XXX: too much logging will lock up the subproc (smh)
+    if loglevel in ('trace', 'debug'):
+        test_log.warning(
+            f'Test harness log level is too verbose: {loglevel!r}\n'
+            f'Reducing to INFO level..'
+        )
+        loglevel: str = 'info'
+
+    code: str = (
+        "import tractor; "
+        "tractor.run_daemon([], "
+        "registry_addrs={reg_addrs}, "
+        "enable_transports={enable_tpts}, "
+        "debug_mode={debug_mode}, "
+        "loglevel={ll})"
+    ).format(
+        reg_addrs=str([reg_addr]),
+        enable_tpts=str([tpt_proto]),
+        ll="'{}'".format(loglevel) if loglevel else None,
+        debug_mode=debug_mode,
+    )
+    cmd: list[str] = [
+        sys.executable,
+        '-c', code,
+    ]
+    kwargs = {}
+    if platform.system() == 'Windows':
+        # without this, tests hang on windows forever
+        kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
+
+    proc: subprocess.Popen = testdir.popen(
+        cmd,
+        **kwargs,
+    )
+
+    # Active-poll the daemon's bind address until it's
+    # ready to accept connections — replaces the legacy
+    # blind `time.sleep(2.2)` which was racy under load
+    # (see
+    # `ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`).
+    #
+    # Per-test deadline scales with platform: macOS/CI
+    # gets extra headroom; Linux dev boxes need very
+    # little.
+    deadline: float = (
+        15.0 if (_non_linux and ci_env)
+        else 10.0
+    )
+    _wait_for_daemon_ready(
+        reg_addr=reg_addr,
+        tpt_proto=tpt_proto,
+        deadline=deadline,
+        proc=proc,
+    )
+
+    assert not proc.returncode
+    yield proc
+    sig_prog(proc, _INT_SIGNAL)
+
+    # XXX! yeah.. just be reaaal careful with this bc
+    # sometimes it can lock up on the `_io.BufferedReader`
+    # and hang..
+    #
+    # NB, drain happens at TEARDOWN (post-yield), so the
+    # test body has its chance to read `proc.stderr`
+    # FIRST. Reading here AFTER would silently swallow
+    # the daemon's stderr output and break tests that
+    # assert on it (e.g. `test_abort_on_sigint`).
+    stderr: str = proc.stderr.read().decode()
+    stdout: str = proc.stdout.read().decode()
+    if (
+        stderr
+        or
+        stdout
+    ):
+        print(
+            f'Daemon actor tree produced output:\n'
+            f'{proc.args}\n'
+            f'\n'
+            f'stderr: {stderr!r}\n'
+            f'stdout: {stdout!r}\n'
+        )
+
+    if (rc := proc.returncode) != -2:
+        msg: str = (
+            f'Daemon actor tree was not cancelled !?\n'
+            f'proc.args: {proc.args!r}\n'
+            f'proc.returncode: {rc!r}\n'
+        )
+        if rc < 0:
+            raise RuntimeError(msg)
+
+        test_log.error(msg)

tests/discovery/test_multi_program.py

@@ -0,0 +1,355 @@
+"""
+Multiple python programs invoking the runtime.
+"""
+from __future__ import annotations
+import platform
+import subprocess
+import time
+from typing import (
+    TYPE_CHECKING,
+)
+
+import pytest
+import trio
+import tractor
+from tractor._testing import (
+    tractor_test,
+)
+from tractor import (
+    current_actor,
+    Actor,
+    Context,
+    Portal,
+)
+from tractor.runtime import _state
+from ..conftest import (
+    sig_prog,
+    _INT_SIGNAL,
+    _INT_RETURN_CODE,
+)
+
+if TYPE_CHECKING:
+    from tractor.msg import Aid
+    from tractor.discovery._addr import (
+        UnwrappedAddress,
+    )
+
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+# NOTE, multi-program tests historically triggered both
+# UDS sock-file leaks (daemon-subproc SIGKILL paths) AND
+# trio `WakeupSocketpair.drain()` busy-loops
+# (`test_register_duplicate_name`). Track + detect
+# per-test as a regression net.
+pytestmark = pytest.mark.usefixtures(
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
+
+
+def test_abort_on_sigint(
+    daemon: subprocess.Popen,
+):
+    assert daemon.returncode is None
+    time.sleep(0.1)
+    sig_prog(daemon, _INT_SIGNAL)
+    assert daemon.returncode == _INT_RETURN_CODE
+
+    # XXX: oddly, couldn't get capfd.readouterr() to work here?
+    if platform.system() != 'Windows':
+        # don't check stderr on windows as its empty when sending CTRL_C_EVENT
+        assert "KeyboardInterrupt" in str(daemon.stderr.read())
+
+
+@tractor_test
+async def test_cancel_remote_registrar(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+):
+    assert not current_actor().is_registrar
+    async with tractor.get_registry(reg_addr) as portal:
+        await portal.cancel_actor()
+
+    time.sleep(0.1)
+    # the registrar channel server is cancelled but not its main task
+    assert daemon.returncode is None
+
+    # no registrar socket should exist
+    with pytest.raises(OSError):
+        async with tractor.get_registry(reg_addr) as portal:
+            pass
+
+
+def test_register_duplicate_name(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+):
+    # bug-class-3 breadcrumbs: the *last* `[CANCEL]` line that
+    # appears under `--ll cancel`/`TRACTOR_LOG_FILE=...` names the
+    # cancel-cascade boundary that's parked. Pair with
+    # `_trio_main` entry/exit breadcrumbs in
+    # `tractor/spawn/_entry.py` to triangulate the swallow point.
+    log = tractor.log.get_logger('tractor.tests.test_multi_program')
+
+    async def main():
+        log.cancel('test_register_duplicate_name: enter `main()`')
+        try:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'actor nursery opened'
+                )
+
+                assert not current_actor().is_registrar
+
+                p1 = await an.start_actor('doggy')
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'spawned doggy #1'
+                )
+                p2 = await an.start_actor('doggy')
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'spawned doggy #2'
+                )
+
+                async with tractor.wait_for_actor('doggy') as portal:
+                    log.cancel(
+                        'test_register_duplicate_name: '
+                        '`wait_for_actor` returned'
+                    )
+                    assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
+
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'ABOUT TO CALL `an.cancel()`'
+                )
+                await an.cancel()
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    '`an.cancel()` returned'
+                )
+        finally:
+            log.cancel(
+                'test_register_duplicate_name: '
+                '`open_nursery.__aexit__` returned, leaving `main()`'
+            )
+
+    # XXX, run manually since we want to start this root **after**
+    # the other "daemon" program with it's own root.
+    trio.run(main)
+
+
+# `n_dups` in {4, 8} both expose the SAME pre-existing race:
+# under rapid same-name spawning against a forkserver +
+# registrar, ONE of the spawned doggies `sys.exit(2)`s during
+# boot before completing parent-handshake. Surfaces now (post
+# the spawn-time `wait_for_peer_or_proc_death` fix) as
+# `ActorFailure rc=2`; previously it was silently masked by
+# the handshake-wait parking forever.
+#
+# Larger `n_dups` widens the race window so the boot-race
+# fires more often — n_dups=4 hits ~always, n_dups=8 hits
+# occasionally. Both xfail(strict=False) so the cancel-cascade
+# regression-check still passes when the boot-race happens
+# NOT to fire.
+#
+# Tracked separately in,
+# https://github.com/goodboy/tractor/issues/456
+_DOGGY_BOOT_RACE_XFAIL = pytest.mark.xfail(
+    strict=False,
+    reason=(
+        'doggy boot-race rc=2 under rapid same-name '
+        'spawn — separate bug from cancel-cascade'
+    ),
+)
+
+
+@pytest.mark.parametrize(
+    'n_dups',
+    [
+        2,
+        pytest.param(4, marks=_DOGGY_BOOT_RACE_XFAIL),
+        pytest.param(8, marks=_DOGGY_BOOT_RACE_XFAIL),
+    ],
+    ids=lambda n: f'n_dups={n}',
+)
+def test_dup_name_cancel_cascade_escalates_to_hard_kill(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+    n_dups: int,
+):
+    '''
+    Regression for the duplicate-name cancel-cascade hang under
+    `tcp+main_thread_forkserver`.
+
+    When N actors share a single name and the parent calls
+    `an.cancel()`, the daemon registrar gets N `register_actor` RPCs
+    in tight succession. Under TCP+MTF, kernel-level socket-buffer
+    contention can push at least one sub-actor's cancel-RPC ack past
+    `Portal.cancel_timeout` (default 0.5s).
+
+    Pre-fix, `Portal.cancel_actor()` silently returned `False` on
+    that timeout, the supervisor's outer `move_on_after(3)` never
+    fired (each per-portal task always returned ≤0.5s, never
+    exceeded 3s), and `soft_kill()`'s `await wait_func(proc)` parked
+    forever — deadlocking nursery `__aexit__`.
+
+    Post-fix, `Portal.cancel_actor()` raises `ActorTooSlowError` on
+    the bounded-wait timeout, and `ActorNursery.cancel()`'s
+    per-child wrapper escalates to `proc.terminate()` (hard-kill).
+    The full nursery teardown therefore stays bounded even under
+    pathological timing.
+
+    `n_dups` is parametrized to widen the race window — more
+    same-name siblings = more concurrent register-RPCs at the
+    daemon = higher probability of hitting the contention path.
+
+    '''
+    log = tractor.log.get_logger(
+        'tractor.tests.test_multi_program'
+    )
+
+    # outer hard ceiling: a regression should fail-fast, NOT hang
+    # the test session for minutes. Budget scales with `n_dups`
+    # since each extra same-name sibling adds ~spawn-cost +
+    # potential cancel-ack-timeout escalation latency under
+    # TCP+forkserver. ~5s/sibling + 15s baseline gives plenty of
+    # headroom while still failing-loud on a real hang.
+    fail_after_s: int = 15 + (5 * n_dups)
+
+    async def main():
+        log.cancel(
+            f'enter `main()` n_dups={n_dups}'
+        )
+        with trio.fail_after(fail_after_s):
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
+                portals: list[Portal] = []
+                for i in range(n_dups):
+                    p: Portal = await an.start_actor('doggy')
+                    portals.append(p)
+                    log.cancel(
+                        f'spawned doggy #{i + 1}/{n_dups}'
+                    )
+
+                # at least one of the N must be discoverable by
+                # name; doesn't matter which one (registrar will
+                # have last-wins semantics under same-name).
+                async with tractor.wait_for_actor('doggy') as portal:
+                    expected_uids = {p.channel.uid for p in portals}
+                    assert portal.channel.uid in expected_uids
+
+                # critical section: this MUST return within
+                # `fail_after_s` even when one or more cancel-RPC
+                # acks time out. Pre-fix, this hangs forever.
+                log.cancel('about to call `an.cancel()`')
+                await an.cancel()
+                log.cancel('`an.cancel()` returned')
+
+        # post-teardown sanity: every child proc must be reaped.
+        # If escalation worked, even timed-out cancel-RPCs would
+        # have triggered `proc.terminate()` and the procs are dead.
+        for p in portals:
+            # `Portal.channel.connected()` -> False once the
+            # underlying chan disconnected (clean exit OR
+            # hard-killed proc both produce disconnect).
+            assert not p.channel.connected(), (
+                f'Portal chan still connected post-teardown?\n'
+                f'{p.channel}'
+            )
+
+    trio.run(main)
+
+
+@tractor.context
+async def get_root_portal(
+    ctx: Context,
+):
+    '''
+    Connect back to the root actor manually (using `._discovery` API)
+    and ensure it's contact info is the same as our immediate parent.
+
+    '''
+    sub: Actor = current_actor()
+    rtvs: dict = _state._runtime_vars
+    raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
+
+    # await tractor.pause()
+    # XXX, in case the sub->root discovery breaks you might need
+    # this (i know i did Xp)!!
+    # from tractor.devx import mk_pdb
+    # mk_pdb().set_trace()
+
+    assert (
+        len(raddrs) == 1
+        and
+        list(sub._parent_chan.raddr.unwrap()) in raddrs
+    )
+
+    # connect back to our immediate parent which should also
+    # be the actor-tree's root.
+    from tractor.discovery._api import get_root
+    ptl: Portal
+    async with get_root() as ptl:
+        root_aid: Aid = ptl.chan.aid
+        parent_ptl: Portal = current_actor().get_parent()
+        assert (
+            root_aid.name == 'root'
+            and
+            parent_ptl.chan.aid == root_aid
+        )
+        await ctx.started()
+
+
+def test_non_registrar_spawns_child(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+    loglevel: str,
+    debug_mode: bool,
+    ci_env: bool,
+):
+    '''
+    Ensure a non-regristar (serving) root actor can spawn a sub and
+    that sub can connect back (manually) to it's rent that is the
+    root without issue.
+
+    More or less this audits the global contact info in
+    `._state._runtime_vars`.
+
+    '''
+    async def main():
+
+        # XXX, since apparently on macos in GH's CI it can be a race
+        # with the `daemon` registrar on grabbing the socket-addr..
+        if ci_env and _non_linux:
+            await trio.sleep(.5)
+
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+            loglevel=loglevel,
+            debug_mode=debug_mode,
+        ) as an:
+
+            actor: Actor = tractor.current_actor()
+            assert not actor.is_registrar
+            sub_ptl: Portal = await an.start_actor(
+                name='sub',
+                enable_modules=[__name__],
+            )
+
+            async with sub_ptl.open_context(
+                get_root_portal,
+            ) as (ctx, _):
+                print('Waiting for `sub` to connect back to us..')
+
+            await an.cancel()
+
+    # XXX, run manually since we want to start this root **after**
+    # the other "daemon" program with it's own root.
+    trio.run(main)

tests/discovery/test_registrar.py

@@ -14,6 +14,7 @@ import psutil
 import pytest
 import subprocess
 import tractor
+from tractor.devx import dump_on_hang
 from tractor.trionics import collapse_eg
 from tractor._testing import tractor_test
 from tractor.discovery._addr import wrap_address
@@ -21,6 +22,20 @@ from tractor.discovery._multiaddr import mk_maddr
 import trio
 
 
+pytestmark = pytest.mark.usefixtures(
+    'reap_subactors_per_test',
+    # NOTE, registrar tests stress the discovery
+    # roundtrip (find_actor / wait_for_actor) which
+    # historically left orphaned UDS sock-files when
+    # subactor `hard_kill` SIGKILL'd, and which
+    # exercises the same trio `WakeupSocketpair`
+    # peer-disconnect path that triggered the
+    # busy-loop bug class.
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
+
+
 @tractor_test
 async def test_reg_then_unreg(
     reg_addr: tuple,
@@ -105,19 +120,6 @@ async def hi():
     return the_line.format(tractor.current_actor().name)
 
 
-async def say_hello(
-    other_actor: str,
-    reg_addr: tuple[str, int],
-):
-    await trio.sleep(1)  # wait for other actor to spawn
-    async with tractor.find_actor(
-        other_actor,
-        registry_addrs=[reg_addr],
-    ) as portal:
-        assert portal is not None
-        return await portal.run(__name__, 'hi')
-
-
 async def say_hello_use_wait(
     other_actor: str,
     reg_addr: tuple[str, int],
@@ -131,14 +133,17 @@ async def say_hello_use_wait(
         return result
 
 
-@tractor_test
+@tractor_test(
+    timeout=7,
+)
 @pytest.mark.parametrize(
-    'func',
+    'ria_fn',
-    [say_hello,
+    [
-     say_hello_use_wait]
+        say_hello_use_wait,
+    ]
 )
 async def test_trynamic_trio(
-    func: Callable,
+    ria_fn: Callable,
     start_method: str,
     reg_addr: tuple,
 ):
@@ -151,13 +156,13 @@ async def test_trynamic_trio(
         print("Alright... Action!")
 
         donny = await n.run_in_actor(
-            func,
+            ria_fn,
             other_actor='gretchen',
             reg_addr=reg_addr,
             name='donny',
         )
         gretchen = await n.run_in_actor(
-            func,
+            ria_fn,
             other_actor='donny',
             reg_addr=reg_addr,
             name='gretchen',
@@ -319,6 +324,14 @@ async def spawn_and_check_registry(
                 assert actor.aid.uid in registry
 
 
+async def with_timeout(
+    main: Callable,
+    timeout: float = 6,
+):
+    with trio.fail_after(timeout):
+        await main()
+
+
 @pytest.mark.parametrize('use_signal', [False, True])
 @pytest.mark.parametrize('with_streaming', [False, True])
 def test_subactors_unregister_on_cancel(
@@ -335,6 +348,7 @@ def test_subactors_unregister_on_cancel(
     '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
+            # with_timeout,
             partial(
                 spawn_and_check_registry,
                 reg_addr,
@@ -364,6 +378,7 @@ def test_subactors_unregister_on_cancel_remote_daemon(
     '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
+            with_timeout,
             partial(
                 spawn_and_check_registry,
                 reg_addr,
@@ -515,12 +530,43 @@ async def kill_transport(
 
 
 
+# ?TODO, do a OSc style signalling test on this?
+# -[ ] doesn't work for fork backends
 # @pytest.mark.parametrize('use_signal', [False, True])
+#
+# Wall-clock bound via `pytest-timeout` (`method='thread'`).
+# Under `--spawn-backend=subint` this test can wedge in an
+# un-Ctrl-C-able state (abandoned-subint + shared-GIL
+# starvation → signal-wakeup-fd pipe fills → SIGINT silently
+# dropped; see `ai/conc-anal/subint_sigint_starvation_issue.md`).
+# `method='thread'` is specifically required because `signal`-
+# method SIGALRM suffers the same GIL-starvation path and
+# wouldn't fire the Python-level handler.
+# At timeout the plugin hard-kills the pytest process — that's
+# the intended behavior here; the alternative is an unattended
+# suite run that never returns.
+# @pytest.mark.timeout(
+#     30,
+#     # NOTE should be a 2.1s happy path.
+#     # XXX for `main_thread_forkserver` this is SUPER SENSITIVE
+#     # so keep it higher to avoid flaky runs..
+#     method='thread',
+# )
+@pytest.mark.skipon_spawn_backend(
+    'subint',
+    # 'main_thread_forkserver',
+    reason=(
+        'XXX SUBINT HANGING TEST XXX\n'
+        'See outstanding issue(s)\n'
+        # TODO, put issue link!
+    )
+)
 def test_stale_entry_is_deleted(
     debug_mode: bool,
     daemon: subprocess.Popen,
     start_method: str,
     reg_addr: tuple,
+    # set_fork_aware_capture,
 ):
     '''
     Ensure that when a stale entry is detected in the registrar's
@@ -529,7 +575,6 @@ def test_stale_entry_is_deleted(
 
     '''
     async def main():
-
         name: str = 'transport_fails_actor'
         _reg_ptl: tractor.Portal
         an: tractor.ActorNursery
@@ -562,4 +607,67 @@ def test_stale_entry_is_deleted(
                 await ptl.cancel_actor()
                 await an.cancel()
 
-    trio.run(main)
+    # XXX, for tracing if this starts being flaky again..
+    #
+    timeout: float = 4
+    async def _timeout_main():
+        with trio.move_on_after(timeout) as cs:
+            await main()
+
+        if (
+            cs.cancel_called
+            and
+            debug_mode
+        ):
+            await tractor.pause()
+
+    # TODO, remove once the `[subint]` variant no longer hangs.
+    #
+    # Status (as of Phase B hard-kill landing):
+    #
+    # - `[trio]`/`[mp_*]` variants: completes normally; `dump_on_hang`
+    #   is a no-op safety net here.
+    #
+    # - `[subint]` variant: hangs indefinitely AND is un-Ctrl-C-able.
+    #   `strace -p <pytest_pid>` while in the hang reveals a silently-
+    #   dropped SIGINT — the C signal handler tries to write the
+    #   signum byte to Python's signal-wakeup fd and gets `EAGAIN`,
+    #   meaning the pipe is full (nobody's draining it).
+    #
+    #   Root-cause chain: our hard-kill in `spawn._subint` abandoned
+    #   the driver OS-thread (which is `daemon=True`) after the soft-
+    #   kill timeout, but the *sub-interpreter* inside that thread is
+    #   still running `trio.run()` — `_interpreters.destroy()` can't
+    #   force-stop a running subint (raises `InterpreterError`), and
+    #   legacy-config subints share the main GIL. The abandoned subint
+    #   starves the parent's trio event loop from iterating often
+    #   enough to drain its wakeup pipe → SIGINT silently drops.
+    #
+    #   This is structurally a CPython-level limitation: there's no
+    #   public force-destroy primitive for a running subint. We
+    #   escape on the harness side via a SIGINT-loop in the `daemon`
+    #   fixture teardown (killing the bg registrar subproc closes its
+    #   end of the IPC, which eventually unblocks a recv in main trio,
+    #   which lets the loop drain the wakeup pipe). Long-term fix path:
+    #   msgspec PEP 684 support (jcrist/msgspec#563) → isolated-mode
+    #   subints with per-interp GIL.
+    #
+    #   Full analysis:
+    #   `ai/conc-anal/subint_sigint_starvation_issue.md`
+    #
+    #   See also the *sibling* hang class documented in
+    #   `ai/conc-anal/subint_cancel_delivery_hang_issue.md` — same
+    #   subint backend, different root cause (Ctrl-C-able hang, main
+    #   trio loop iterating fine; ours to fix, not CPython's).
+    #   Reproduced by `tests/test_subint_cancellation.py
+    #   ::test_subint_non_checkpointing_child`.
+    #
+    # Kept here (and not behind a `pytestmark.skip`) so we can still
+    # inspect the dump file if the hang ever returns after a refactor.
+    # `pytest`'s stderr capture eats `faulthandler` output otherwise,
+    # so we route `dump_on_hang` to a file.
+    with dump_on_hang(
+        seconds=timeout*2,
+        path=f'/tmp/test_stale_entry_is_deleted_{start_method}.dump',
+    ):
+        trio.run(_timeout_main)

tests/ipc/test_multi_tpt.py

@@ -59,15 +59,18 @@ async def chk_tpts(
 )
 def test_root_passes_tpt_to_sub(
     tpt_proto_key: str,
+    tpt_proto: str,
     reg_addr: tuple,
     debug_mode: bool,
 ):
-    # XXX NOTE, the `reg_addr` addr won't be the same type as the
+    # `reg_addr` is sourced from the CLI `--tpt-proto={tpt_proto}`,
-    # `tpt_proto_key` would deliver here unless you pass `--tpt-proto
+    # so when the parametrized `tpt_proto_key` differs, the test
-    # <tpt_proto_key>` on the CLI.
+    # asks the runtime to `enable_transports=[<other_proto>]` while
-    #
+    # pointing `registry_addrs` at a `reg_addr` of the wrong proto.
-    # if tpt_proto_key == 'uds':
+    # The layer-2 guard in `open_root_actor` is expected to fail
-    #     breakpoint()
+    # fast with `ValueError` on this mismatch (rather than the prior
+    # silent hang during the registrar handshake).
+    proto_mismatch: bool = (tpt_proto_key != tpt_proto)
 
     async def main():
         async with tractor.open_nursery(
@@ -99,4 +102,14 @@ def test_root_passes_tpt_to_sub(
             # shudown sub-actor(s)
             await an.cancel()
 
+    if proto_mismatch:
+        # mismatched proto must raise `ValueError` from the
+        # `open_root_actor` runtime guard before any subactor spawn.
+        with pytest.raises(ValueError) as excinfo:
+            trio.run(main)
+        msg: str = str(excinfo.value)
+        assert 'enable_transports' in msg
+        assert 'registry_addrs' in msg
+        assert tpt_proto_key in msg or tpt_proto in msg
+    else:
         trio.run(main)

tests/msg/test_ext_types_msgspec.py

@@ -57,6 +57,7 @@ from tractor.msg._ops import (
     limit_plds,
 )
 
+
 def enc_nsp(obj: Any) -> Any:
     actor: Actor = tractor.current_actor(
         err_on_no_runtime=False,
@@ -617,6 +618,17 @@ def test_ext_types_over_ipc(
     debug_mode: bool,
     pld_spec: Union[Type],
     add_hooks: bool,
+
+    set_fork_aware_capture,
+    # ^^XXX? for forking spawners
+
+    # capfd: pytest.CaptureFixture,
+    # ^^NOTE, super interesting that if
+    # we disable this below then the tpt-layer
+    # suffers as an "unclean EOF"??
+    # ?TODO, determine why/how that mks sense when addressing,
+    # https://github.com/pytest-dev/pytest/issues/14444
+    #
 ):
     '''
     Ensure we can support extension types coverted using
@@ -725,18 +737,26 @@ def test_ext_types_over_ipc(
 
             await p.cancel_actor()
 
+    async def fa_main():
+        with (
+            trio.fail_after(2),
+            # ?TODO, investigate? see NOTE above..
+            # capfd.disabled(),
+        ):
+            await main()
+
     if (
         NamespacePath in pld_types
         and
         add_hooks
     ):
-        trio.run(main)
+        trio.run(fa_main)
 
     else:
         with pytest.raises(
             expected_exception=tractor.RemoteActorError,
         ) as excinfo:
-            trio.run(main)
+            trio.run(fa_main)
 
         exc = excinfo.value
         # bc `.started(nsp: NamespacePath)` will raise

tests/msg/test_pldrx_limiting.py

@@ -55,12 +55,37 @@ async def maybe_expect_raises(
     raises: BaseException|None = None,
     ensure_in_message: list[str]|None = None,
     post_mortem: bool = False,
-    timeout: int = 3,
+    # NOTE, `None` selects a backend-aware default below —
+    # see `_BACKEND_TIMEOUT_DEFAULTS` for rationale. Caller
+    # can override with an explicit value to opt out.
+    timeout: int|None = None,
 ) -> None:
     '''
     Async wrapper for ensuring errors propagate from the inner scope.
 
     '''
+    if timeout is None:
+        # Pick a backend-aware default. Fork-based backends
+        # (`main_thread_forkserver`) need much more headroom
+        # because actor spawn + IPC ctx-exit + msg-validation
+        # error path takes longer than under `trio` backend
+        # — especially under cross-pytest-stream contention
+        # (#451). `test_basic_payload_spec` empirically:
+        #   - 3s flaked all-valid variant (`TooSlowError`)
+        #   - 8s flaked `invalid-return` variant
+        #     (`Cancelled` surfaced instead of `MsgTypeError`
+        #     because `fail_after` fired mid-error-path)
+        #   - 15s flaked under cross-stream contention
+        # 30s for fork-based gives plenty of headroom while
+        # still failing-loud on a genuine hang. Other
+        # backends keep the original 3s.
+        from tractor.spawn import _spawn as _spawn_mod
+        timeout = (
+            30
+            if _spawn_mod._spawn_method == 'main_thread_forkserver'
+            else 3
+        )
+
     if tractor.debug_mode():
         timeout += 999
 
@@ -259,6 +284,11 @@ def test_basic_payload_spec(
     return_value: str|None,
     started_value: int|PldMsg,
     pld_check_started_value: bool,
+
+    set_fork_aware_capture,
+    # ^XXX TODO? for forking spawners, seems to prevent hangs when
+    # --capture=sys not set, but only for a while then the problem
+    # accumulates?
 ):
     '''
     Validate the most basic `PldRx` msg-type-spec semantics around

tests/test_advanced_streaming.py

@@ -5,10 +5,15 @@ Advanced streaming patterns using bidirectional streams and contexts.
 from collections import Counter
 import itertools
 import platform
+from typing import Type
 
 import pytest
 import trio
 import tractor
+from tractor._testing.trace import (
+    AfkAlarmWTraceFactory,
+    FailAfterWTraceFactory,
+)
 
 
 def is_win():
@@ -76,9 +81,7 @@ async def subscribe(
 
 
 async def consumer(
-
     subs: list[str],
-
 ) -> None:
 
     uid = tractor.current_actor().uid
@@ -108,15 +111,109 @@ async def consumer(
                         print(f'{uid} got: {value}')
 
 
-def test_dynamic_pub_sub():
+# NOTE: deliberately NOT using `@pytest.mark.timeout(...)` —
+# both pytest-timeout enforcement modes break trio under
+# fork-based backends:
+#
+# - `method='signal'` (SIGALRM): the handler synchronously
+#   raises `Failed` in trio's main thread mid-`epoll.poll()`,
+#   leaves `GLOBAL_RUN_CONTEXT` half-installed ("Trio guest
+#   run got abandoned"), and EVERY subsequent `trio.run()`
+#   in the same pytest process bails with
+#   `RuntimeError: Attempted to call run() from inside a
+#   run()` — session-wide poison.
+#
+# - `method='thread'`: calls `_thread.interrupt_main()`
+#   raising `KeyboardInterrupt` into the main thread. Under
+#   fork-based backends with mid-cascade fd-juggling the KBI
+#   can escape trio's `KIManager` and bubble out of pytest
+#   itself — kills the WHOLE session.
+#
+# Instead we use `trio.fail_after()` INSIDE `main()` below:
+# trio's own `Cancelled`/`TooSlowError` machinery handles the
+# timeout, cleanly unwinds the actor nursery's cancel
+# cascade, and only fails the single test (no cross-test
+# state corruption either way).
+#
+# `pyproject.toml`'s default `timeout = 200` is still a
+# last-resort safety net.
+@pytest.mark.parametrize(
+    'expect_cancel_exc', [
+        KeyboardInterrupt,
+        trio.TooSlowError,
+    ],
+    ids=lambda item:
+        f'expect_user_exc_raised={item.__name__}'
+)
+def test_dynamic_pub_sub(
+    reg_addr: tuple,
+    debug_mode: bool,
+    test_log: tractor.log.StackLevelAdapter,
+    reap_subactors_per_test: int,
+    expect_cancel_exc: Type[BaseException],
+
+    is_forking_spawner: bool,
+    set_fork_aware_capture,
+
+    fail_after_w_trace: FailAfterWTraceFactory,
+    afk_alarm_w_trace: AfkAlarmWTraceFactory,
+):
+    failed_to_raise_report: str = (
+        f'Never got a {expect_cancel_exc!r} ??'
+    )
 
     global _registry
 
     from multiprocessing import cpu_count
     cpus = cpu_count()
 
+    # Hard safety cap via trio's own cancellation. NOTE see the
+    # module-level note on why we avoid `pytest-timeout` for this
+    # test. Picked backend-aware: under `trio` backend spawn is
+    # cheap (~1s for `cpus` actors) but fork-based backends pay
+    # a per-spawn cost (forkserver round-trip + IPC peer-handshake)
+    # that can stack up over `cpus - 1` sequential `n.run_in_actor()`
+    # calls — especially on UDS under cross-pytest contention
+    # (#451 / #452). 4s was flaking right at the edge under fork
+    # backends — bumped to 8s with diag-snapshot-on-timeout via
+    # `fail_after_w_trace` so a borderline run still fails loud
+    # but lands a ptree/wchan/py-spy dump in
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` for inspection.
+    #
+    # XXX caveat: this is an *inner* trio cancel — its `Cancelled`
+    # cannot reach a task parked in a shielded `await` (e.g. inside
+    # actor-nursery teardown). When the in-band cancel path is
+    # itself buggy (the bug-class-3 `raise KBI` swallow we're
+    # currently chasing) this guard does NOT fire and the test
+    # sits forever until external SIGINT. The `afk_alarm_w_trace`
+    # outer guard below is the AFK-safety counterpart (SIGALRM
+    # raises in the main thread regardless of trio scope state).
+    fail_after_s: int = (
+        8
+        if is_forking_spawner
+        else 20
+    )
+
     async def main():
-        async with tractor.open_nursery() as n:
+        # bug-class-3 breadcrumb: tag each level of the cancel path
+        # so when the run hangs and we capture cancel-level logs, the
+        # *last* breadcrumb that fired names the swallow point.
+        test_log.cancel('test_dynamic_pub_sub: enter main()')
+        try:
+            async with fail_after_w_trace(fail_after_s):
+                test_log.cancel(
+                    f'test_dynamic_pub_sub: '
+                    f'enter `fail_after_w_trace({fail_after_s})` scope'
+                )
+                try:
+                    async with tractor.open_nursery(
+                        registry_addrs=[reg_addr],
+                        debug_mode=debug_mode,
+                    ) as n:
+                        test_log.cancel(
+                            'test_dynamic_pub_sub: '
+                            'actor nursery opened'
+                        )
 
                         # name of this actor will be same as target func
                         await n.run_in_actor(publisher)
@@ -138,29 +235,69 @@ def test_dynamic_pub_sub():
                             subs=list(_registry.keys()),
                         )
 
-            # block until cancelled by user
+                        # block until "cancelled by user"
-            with trio.fail_after(3):
+                        await trio.sleep(3)
-                await trio.sleep_forever()
+                        test_log.warning(
+                            f'Raising user cancel exc: '
+                            f'{expect_cancel_exc!r}'
+                        )
+                        test_log.cancel(
+                            f'test_dynamic_pub_sub: '
+                            f'ABOUT TO RAISE {expect_cancel_exc!r}'
+                        )
+                        raise expect_cancel_exc('simulate user cancel!')
+                finally:
+                    test_log.cancel(
+                        'test_dynamic_pub_sub: '
+                        'actor nursery `__aexit__` returned'
+                    )
+            test_log.cancel(
+                'test_dynamic_pub_sub: `fail_after` scope exited'
+            )
+        finally:
+            test_log.cancel(
+                'test_dynamic_pub_sub: leaving `main()`'
+            )
 
+    def _run_and_match():
         try:
             trio.run(main)
-    except (
+            pytest.fail(failed_to_raise_report)
-        trio.TooSlowError,
+        except expect_cancel_exc:
-        ExceptionGroup,
+            # parent-side raised the user-cancel exc directly and
-    ) as err:
+            # it propagated unwrapped; clean path.
-        if isinstance(err, ExceptionGroup):
+            test_log.exception('Got user-cancel exc AS EXPECTED')
-            for suberr in err.exceptions:
+        except BaseExceptionGroup as err:
-                if isinstance(suberr, trio.TooSlowError):
+            # under fork-based backends the user-raised cancel
-                    break
+            # can race with subactor-side stream teardown
+            # (`trio.EndOfChannel` from a publisher's `send()`
+            # whose remote half got cut). The expected exc may
+            # then be nested deeper in the group rather than at
+            # the top level. `BaseExceptionGroup.split()` walks
+            # the exc tree recursively (Python 3.11+).
+            matched, _ = err.split(expect_cancel_exc)
+            if matched is None:
+                pytest.fail(failed_to_raise_report)
+
+            test_log.exception('Got user-cancel exc AS EXPECTED')
+
+    # outer SIGALRM-based guard — survives a shielded-await
+    # deadlock since `signal.alarm` raises in the main thread
+    # regardless of trio's scope state, AND captures a full diag
+    # snapshot to `$XDG_CACHE_HOME/tractor/hung-dumps/` before
+    # re-raising. ONLY armed under fork-based backends since the
+    # bug we're chasing is MTF-specific. Cap = `fail_after_s + 5`
+    # so the trio-native path always wins when it works.
+    if is_forking_spawner:
+        with afk_alarm_w_trace(fail_after_s + 5):
+            _run_and_match()
     else:
-                pytest.fail('Never got a `TooSlowError` ?')
+        _run_and_match()
 
 
 @tractor.context
 async def one_task_streams_and_one_handles_reqresp(
-
     ctx: tractor.Context,
-
 ) -> None:
 
     await ctx.started()
@@ -257,7 +394,8 @@ async def echo_ctx_stream(
 
 
 def test_sigint_both_stream_types():
-    '''Verify that running a bi-directional and recv only stream
+    '''
+    Verify that running a bi-directional and recv only stream
     side-by-side will cancel correctly from SIGINT.
 
     '''
@@ -287,9 +425,11 @@ def test_sigint_both_stream_types():
                             assert resp == msg
                             raise KeyboardInterrupt
 
+    # TODO, use pytest.raises() here instead?
+    # (why weren't we originally?)
     try:
         trio.run(main)
-        assert 0, "Didn't receive KBI!?"
+        pytest.fail("Didn't receive KBI!?")
     except KeyboardInterrupt:
         pass
 
@@ -356,7 +496,12 @@ async def inf_streamer(
     print('streamer exited .open_streamer() block')
 
 
+# @pytest.mark.timeout(
+#     6,
+#     method='signal',
+# )
 def test_local_task_fanout_from_stream(
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -421,4 +566,9 @@ def test_local_task_fanout_from_stream(
 
             await p.cancel_actor()
 
-    trio.run(main)
+    async def w_timeout():
+        with trio.fail_after(6):
+            await main()
+
+    # trio.run(main)
+    trio.run(w_timeout)

tests/test_cancellation.py

@@ -7,6 +7,7 @@ import signal
 import platform
 import time
 from itertools import repeat
+from typing  import Type
 
 import pytest
 import trio
@@ -14,6 +15,7 @@ import tractor
 from tractor._testing import (
     tractor_test,
 )
+from tractor._testing.trace import FailAfterWTraceFactory
 from .conftest import no_windows
 
 
@@ -21,6 +23,46 @@ _non_linux: bool = platform.system() != 'Linux'
 _friggin_windows: bool = platform.system() == 'Windows'
 
 
+pytestmark = [
+    # Multi-actor cancel cascades under
+    # `--spawn-backend=subint` trip the abandoned-subint
+    # GIL-hostage class — a stuck subint can starve the
+    # parent's trio loop and block cancel-delivery.
+    # Apply the skip module-wide rather than per-test
+    # since every test here exercises the same cascade.
+    pytest.mark.skipon_spawn_backend(
+        'subint',
+        reason=(
+            'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
+            'Cancel cascades under '
+            '`--spawn-backend=subint` trip the abandoned-subint '
+            'GIL-hostage class — see\n'
+            '  - `ai/conc-anal/subint_sigint_starvation_issue.md` '
+            '(GIL-hostage, SIGINT-unresponsive)\n'
+            '  - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
+            '(sibling: parent parks on dead chan)\n'
+            '  - https://github.com/goodboy/tractor/issues/379 '
+            '(subint umbrella)\n'
+        )
+    ),
+    pytest.mark.usefixtures(
+        'reap_subactors_per_test',
+        # NOTE, cancellation tests stress the SIGKILL
+        # `hard_kill` path which leaks UDS sock-files when
+        # the subactor's IPC server `finally:` cleanup
+        # doesn't run. Track per-test for blame attribution.
+        'track_orphaned_uds_per_test',
+        # NOTE, cancel-cascade timing races (see
+        # `test_nested_multierrors`) can also leave a
+        # subactor spinning at 100% CPU when its cancel
+        # signal got swallowed mid-handshake. Catches the
+        # runaway-loop class that doesn't leak UDS socks
+        # but burns the box.
+        'detect_runaway_subactors_per_test',
+    ),
+]
+
+
 async def assert_err(delay=0):
     await trio.sleep(delay)
     assert 0
@@ -45,7 +87,11 @@ async def do_nuthin():
     ],
     ids=['no_args', 'unexpected_args'],
 )
-def test_remote_error(reg_addr, args_err):
+def test_remote_error(
+    reg_addr: tuple,
+    args_err: tuple[dict, Type[Exception]],
+    set_fork_aware_capture,
+):
     '''
     Verify an error raised in a subactor that is propagated
     to the parent nursery, contains the underlying boxed builtin
@@ -112,6 +158,8 @@ def test_remote_error(reg_addr, args_err):
 
 def test_multierror(
     reg_addr: tuple[str, int],
+    start_method: str,  # parametrized
+    set_fork_aware_capture, #: Callable,
 ):
     '''
     Verify we raise a ``BaseExceptionGroup`` out of a nursery where
@@ -141,16 +189,39 @@ def test_multierror(
         trio.run(main)
 
 
-@pytest.mark.parametrize('delay', (0, 0.5))
 @pytest.mark.parametrize(
-    'num_subactors', range(25, 26),
+    'delay',
+    (0, 0.5),
+    ids='delays={}'.format,
 )
-def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
+@pytest.mark.parametrize(
-    """Verify we raise a ``BaseExceptionGroup`` out of a nursery where
+    'num_subactors',
+    range(25, 26),
+    ids= 'num_subs={}'.format,
+)
+def test_multierror_fast_nursery(
+    reg_addr: tuple,
+    start_method: str,
+    num_subactors: int,
+    delay: float,
+    set_fork_aware_capture,
+    fail_after_w_trace: FailAfterWTraceFactory,
+):
+    '''
+    Verify we raise a ``BaseExceptionGroup`` out of a nursery where
     more then one actor errors and also with a delay before failure
     to test failure during an ongoing spawning.
-    """
+
+    '''
     async def main():
+        # budget = 2× natural trio-backend cascade time for
+        # 25 errorer subactors (~14s observed). on-timeout
+        # diag snapshot → if the cancel cascade hangs
+        # (observed under MTF backend with N>=14 errorer
+        # subactors) we get a fresh ptree/wchan/py-spy dump
+        # on disk INSTEAD of an opaque pytest timeout-kill.
+        # See `tractor/_testing/trace.py` for the helper.
+        async with fail_after_w_trace(30.0):
             async with tractor.open_nursery(
                 registry_addrs=[reg_addr],
             ) as nursery:
@@ -163,9 +234,23 @@ def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
                     )
 
     # with pytest.raises(trio.MultiError) as exc_info:
-    with pytest.raises(BaseExceptionGroup) as exc_info:
+    # NOTE, `trio.TooSlowError` from `fail_after_w_trace`
+    # bubbles UN-wrapped if `open_nursery.__aexit__` never
+    # gets re-entered; wrapped inside a `BaseExceptionGroup`
+    # if it did. Accept both shapes so the matcher itself
+    # doesn't lie about *what* failed.
+    with pytest.raises(
+        (BaseExceptionGroup, trio.TooSlowError),
+    ) as exc_info:
         trio.run(main)
 
+    if isinstance(exc_info.value, trio.TooSlowError):
+        pytest.fail(
+            f'cancel cascade hung past 12s '
+            f'(num_subactors={num_subactors}, delay={delay}); '
+            f'see stderr for `fail_after_w_trace` snapshot path'
+        )
+
     assert exc_info.type == ExceptionGroup
     err = exc_info.value
     exceptions = err.exceptions
@@ -189,8 +274,15 @@ async def do_nothing():
     pass
 
 
-@pytest.mark.parametrize('mechanism', ['nursery_cancel', KeyboardInterrupt])
+@pytest.mark.parametrize(
-def test_cancel_single_subactor(reg_addr, mechanism):
+    'mechanism', [
+    'nursery_cancel',
+    KeyboardInterrupt,
+])
+def test_cancel_single_subactor(
+    reg_addr: tuple,
+    mechanism: str|KeyboardInterrupt,
+):
     '''
     Ensure a ``ActorNursery.start_actor()`` spawned subactor
     cancels when the nursery is cancelled.
@@ -232,9 +324,14 @@ async def stream_forever():
         await trio.sleep(0.01)
 
 
-@tractor_test
+@tractor_test(
-async def test_cancel_infinite_streamer(start_method):
+    timeout=6,
-
+)
+async def test_cancel_infinite_streamer(
+    reg_addr: tuple,
+    start_method: str,
+    set_fork_aware_capture,
+):
     # stream for at most 1 seconds
     with (
         trio.fail_after(4),
@@ -286,11 +383,15 @@ async def test_cancel_infinite_streamer(start_method):
         'no_daemon_actors_fail_all_run_in_actors_sleep_then_fail',
     ],
 )
-@tractor_test
+@tractor_test(
+    timeout=10,
+)
 async def test_some_cancels_all(
     num_actors_and_errs: tuple,
+    reg_addr: tuple,
     start_method: str,
     loglevel: str,
+    set_fork_aware_capture, #: Callable,
 ):
     '''
     Verify a subset of failed subactors causes all others in
@@ -370,7 +471,10 @@ async def test_some_cancels_all(
         pytest.fail("Should have gotten a remote assertion error?")
 
 
-async def spawn_and_error(breadth, depth) -> None:
+async def spawn_and_error(
+    breadth: int,
+    depth: int,
+) -> None:
     name = tractor.current_actor().name
     async with tractor.open_nursery() as nursery:
         for i in range(breadth):
@@ -395,28 +499,150 @@ async def spawn_and_error(breadth, depth) -> None:
             await nursery.run_in_actor(*args, **kwargs)
 
 
-@tractor_test
+# NOTE: `main_thread_forkserver` capture-fd hang class is no
-async def test_nested_multierrors(loglevel, start_method):
+# longer skipped here — `--capture=sys` (the new `pyproject.toml`
+# default) sidesteps the pipe-buffer-fill deadlock for
+# `test_nested_multierrors`. See
+# `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
+# / #449 for the post-mortem.
+# @pytest.mark.timeout(
+#     10,
+#     method='thread',
+# )
+@pytest.mark.parametrize(
+    'depth',
+    [1, 3],
+    ids='depth={}'.format,
+)
+@tractor_test(
+    # XXX this OUTER `trio.fail_after` wall MUST exceed the
+    # largest INNER `fail_after_w_trace()` budget set in the body
+    # below (max = the MTF depth=3 == 30s case, further scaled by
+    # `cpu_scaling_factor()` on CI/throttle). Otherwise it fires
+    # FIRST and pre-empts the inner snapshot-capturing deadline,
+    # turning a graceful `TooSlowError`+ptree-dump into an opaque
+    # outer timeout-kill (the prior `timeout=10` did exactly this
+    # — it was *smaller* than the 12s trio depth=3 budget, so the
+    # depth-3 case `FAILED` on slow CI instead of dumping).
+    # Trio backend is fast and won't notice the extra budget.
+    # See `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    timeout=40,
+)
+async def test_nested_multierrors(
+    reg_addr: tuple,
+    loglevel: str,
+    start_method: str,
+    set_fork_aware_capture,
+    fail_after_w_trace: FailAfterWTraceFactory,
+    request: pytest.FixtureRequest,
+    depth: int,
+):
     '''
-    Test that failed actor sets are wrapped in `BaseExceptionGroup`s. This
+    Test that failed actor sets are wrapped in `BaseExceptionGroup`s.
-    test goes only 2 nurseries deep but we should eventually have tests
+
-    for arbitrary n-depth actor trees.
+    Parametrized over recursion `depth ∈ {1, 3}`:
+
+      - `depth=1`: shallow tree (2 spawners × 2 errorers, 2
+        levels). Cascade completes well within budget on ALL
+        backends including MTF — regression-safety green case.
+
+      - `depth=3`: deep tree (2 spawners × recursive depth-3
+        spawn-and-error). On `main_thread_forkserver` this
+        trips the cancel-cascade shape-mismatch bug class
+        (see `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`)
+        — xfailed below.
 
     '''
-    if start_method == 'trio':
+    # XXX: `multiprocessing.forkserver` can't handle nested
-        depth = 3
+    # spawning at any depth — hangs / broken-pipes. Pre-existing
-        subactor_breadth = 2
+    # backend limitation, NOT depth-specific.
-    else:
-        # XXX: multiprocessing can't seem to handle any more then 2 depth
-        # process trees for whatever reason.
-        # Any more process levels then this and we see bugs that cause
-        # hangs and broken pipes all over the place...
     if start_method == 'forkserver':
         pytest.skip("Forksever sux hard at nested spawning...")
-        depth = 1  # means an additional actor tree of spawning (2 levels deep)
+
     subactor_breadth = 2
 
-    with trio.fail_after(120):
+    # MTF backend trips a probabilistic timing race in the
+    # cancel-cascade — NOT depth-gated; depth amplifies the
+    # variance so depth=3 misses nearly every run while
+    # depth=1 misses occasionally. Both get the xfail mark
+    # (with `strict=False`) since the bug class can fire at
+    # either depth.
+    #
+    # The scenario in detail:
+    #
+    #     T=0      spawn spawner_0 + spawner_1 in parallel
+    #     T=t1     spawner_0's child errors →
+    #              RemoteActorError reaches root nursery
+    #     T=t1+ε   root nursery starts cancelling
+    #              spawner_1's portal-wait
+    #     T=t2     spawner_1's child errors → tries to send
+    #              RemoteActorError back
+    #
+    #     if t2 < t1+ε:  BEG = [RAE, RAE]        ← clean (xpass)
+    #     if t2 > t1+ε:  BEG = [RAE, Cancelled]  ← race tripped (xfail)
+    #
+    # i.e. the assertion below (`isinstance(_, RemoteActorError)`)
+    # fails iff cancel-delivery beats the other tree's natural
+    # error-propagation. Depth amplifies `t2-t1` variance
+    # (longer per-tree paths = more skew); under MTF the
+    # fork-spawn jitter + UDS-contention widens both `t1` and
+    # `t2` further.
+    #
+    # With `strict=False` the clean-cascade cases (most
+    # depth=1 runs, rare depth=3 runs) report as `xpassed`
+    # while the race-tripped cases report as `xfailed` —
+    # neither flakes `--lf`. When MTF cancel-cascade
+    # eventually speeds up enough to close the race even at
+    # depth=3, BOTH variants will reliably `xpass` and
+    # pytest will yell — our signal to drop the marker. See
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    if start_method == 'main_thread_forkserver':
+        request.node.add_marker(
+            pytest.mark.xfail(
+                strict=False,
+                reason=(
+                    f'MTF cancel-cascade shape-mismatch at '
+                    f'depth={depth} (Cancelled races '
+                    f'RemoteActorError in BEG); see conc-anal/'
+                    'cancel_cascade_too_slow_under_main_thread_forkserver_issue.md'
+                ),
+            )
+        )
+
+    # Per-backend/-depth budgets: in the non-hang case the
+    # whole spawn + cancel-cascade should complete in well
+    # under these. On the borderline hang case the
+    # `fail_after_w_trace` fires `TooSlowError` AND captures a
+    # ptree/wchan/py-spy snapshot to
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` for offline
+    # inspection. See
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    #
+    # NOTE: the `trio` depth=3 budget was bumped 6 -> 12s after
+    # the `trio` 0.29 -> 0.33 lock bump (commit c7741bba) slowed
+    # the depth-3 cancel-cascade from <6s to ~7-8s; the 6s
+    # deadline was firing and its `Cancelled(source='deadline')`
+    # (trio 0.33 cancel-reason metadata) collapsed a BEG branch,
+    # breaking the `RemoteActorError` assertion below. depth=1
+    # still finishes in ~3s so keeps the 6s budget. See
+    # `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
+    match (start_method, depth):
+        case ('trio', 1):
+            timeout = 6
+        case ('trio', 3):
+            timeout = 12
+        case ('main_thread_forkserver', 1):
+            timeout = 16
+        case ('main_thread_forkserver', 3):
+            timeout = 30
+
+    # headroom for CPU-freq scaling AND/OR slow CI so the inner
+    # snapshot-capturing budget doesn't fire spuriously on a
+    # sluggish runner; see `cpu_scaling_factor()`.
+    from .conftest import cpu_scaling_factor
+    timeout *= cpu_scaling_factor()
+
+    async with fail_after_w_trace(timeout):
         try:
             async with tractor.open_nursery() as nursery:
                 for i in range(subactor_breadth):
@@ -483,20 +709,24 @@ async def test_nested_multierrors(loglevel, start_method):
 
 @no_windows
 def test_cancel_via_SIGINT(
-    loglevel,
+    reg_addr: tuple,
-    start_method,
+    loglevel: str,
-    spawn_backend,
+    start_method: str,
 ):
-    """Ensure that a control-C (SIGINT) signal cancels both the parent and
+    '''
+    Ensure that a control-C (SIGINT) signal cancels both the parent and
     child processes in trionic fashion
-    """
+
+    '''
     pid: int = os.getpid()
 
     async def main():
         with trio.fail_after(2):
-            async with tractor.open_nursery() as tn:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as tn:
                 await tn.start_actor('sucka')
-                if 'mp' in spawn_backend:
+                if 'mp' in start_method:
                     time.sleep(0.1)
                 os.kill(pid, signal.SIGINT)
                 await trio.sleep_forever()
@@ -507,6 +737,7 @@ def test_cancel_via_SIGINT(
 
 @no_windows
 def test_cancel_via_SIGINT_other_task(
+    reg_addr: tuple,
     loglevel: str,
     start_method: str,
     spawn_backend: str,
@@ -535,7 +766,9 @@ def test_cancel_via_SIGINT_other_task(
     async def spawn_and_sleep_forever(
         task_status=trio.TASK_STATUS_IGNORED
     ):
-        async with tractor.open_nursery() as tn:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as tn:
             for i in range(3):
                 await tn.run_in_actor(
                     sleep_forever,
@@ -599,7 +832,7 @@ async def spawn_sub_with_sync_blocking_task():
 def test_cancel_while_childs_child_in_sync_sleep(
     loglevel: str,
     start_method: str,
-    spawn_backend: str,
+    is_forking_spawner: bool,
     debug_mode: bool,
     reg_addr: tuple,
     man_cancel_outer: bool,
@@ -615,7 +848,10 @@ def test_cancel_while_childs_child_in_sync_sleep(
 
     '''
     if start_method == 'forkserver':
-        pytest.skip("Forksever sux hard at resuming from sync sleep...")
+        pytest.skip(
+            "`multiprocessing`'s forkserver sux hard at "
+            "resuming from sync sleep..."
+        )
 
     async def main():
         #
@@ -658,7 +894,11 @@ def test_cancel_while_childs_child_in_sync_sleep(
         # delay = 2  # is AssertionError in eg AND no TooSlowError !?
         # is AssertionError in eg AND no _cs cancellation.
         delay = (
-            6 if _non_linux
+            6 if (
+                _non_linux
+                or
+                is_forking_spawner
+            )
             else 4 
         )
 
@@ -694,7 +934,7 @@ def test_cancel_while_childs_child_in_sync_sleep(
 
 
 def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
-    start_method,
+    start_method: str,
 ):
     '''
     This is a very subtle test which demonstrates how cancellation
@@ -715,6 +955,12 @@ def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
     if _friggin_windows:  # smh
         timeout += 1
 
+    # CPU-scaling / CI latency headroom — macOS CI especially is
+    # slow for this graceful-vs-hard-reap timing race; see
+    # `cpu_scaling_factor()`.
+    from .conftest import cpu_scaling_factor
+    timeout *= cpu_scaling_factor()
+
     async def main():
         start = time.time()
         try:

tests/test_clustering.py

@@ -77,6 +77,7 @@ async def worker(
 @tractor_test
 async def test_streaming_to_actor_cluster(
     tpt_proto: str,
+    is_forking_spawner: bool,
 ):
     '''
     Open an actor "cluster" using the (experimental) `._clustering`
@@ -88,7 +89,11 @@ async def test_streaming_to_actor_cluster(
             f'Test currently fails with tpt-proto={tpt_proto!r}\n'
         )
 
-    with trio.fail_after(6):
+    delay: float = (
+        10 if is_forking_spawner
+        else 6
+    )
+    with trio.fail_after(delay):
         async with (
             open_actor_cluster(modules=[__name__]) as portals,
 

tests/test_context_stream_semantics.py

@@ -115,10 +115,12 @@ async def not_started_but_stream_opened(
 )
 def test_started_misuse(
     target: Callable,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -184,15 +186,24 @@ def test_simple_context(
     error_parent,
     child_blocks_forever,
     pointlessly_open_stream,
+    reg_addr: tuple,
     debug_mode: bool,
+    is_forking_spawner: bool,
 ):
 
-    timeout = 1.5 if not platform.system() == 'Windows' else 4
+    timeout: float = 1.5
+    # windows and forking-spawner both have "slower but more
+    # deterministic" cancel teardown.
+    if platform.system() == 'Windows':
+        timeout = 4
+    elif is_forking_spawner:
+        timeout = 3
 
     async def main():
 
         with trio.fail_after(timeout):
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
             ) as an:
                 portal = await an.start_actor(
@@ -278,6 +289,7 @@ def test_parent_cancels(
     cancel_method: str,
     chk_ctx_result_before_exit: bool,
     child_returns_early: bool,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -355,6 +367,7 @@ def test_parent_cancels(
     async def main():
 
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -931,6 +944,7 @@ async def keep_sending_from_child(
 )
 def test_one_end_stream_not_opened(
     overrun_by: tuple[str, int, Callable],
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -949,6 +963,7 @@ def test_one_end_stream_not_opened(
 
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -1113,6 +1128,7 @@ def test_maybe_allow_overruns_stream(
 
     # conftest wide
     loglevel: str,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -1133,6 +1149,7 @@ def test_maybe_allow_overruns_stream(
     '''
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -1249,6 +1266,7 @@ def test_maybe_allow_overruns_stream(
 
 def test_ctx_with_self_actor(
     loglevel: str,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -1263,6 +1281,7 @@ def test_ctx_with_self_actor(
     '''
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
             enable_modules=[__name__],
         ) as an:

tests/test_infected_asyncio.py

@@ -30,6 +30,32 @@ from tractor import (
 from tractor.runtime import _state
 from tractor.trionics import BroadcastReceiver
 from tractor._testing import expect_ctxc
+from tractor._testing.trace import (
+    AfkAlarmWTraceFactory,
+    FailAfterWTraceFactory,
+)
+
+
+# Per-test zombie-subactor reaper. Opt-in (NOT autouse) —
+# see `tractor._testing.pytest.reap_subactors_per_test`'s
+# docstring for the full rationale. This module specifically
+# needs it because tests like
+# `test_echoserver_detailed_mechanics[KeyboardInterrupt]`
+# and the `test_sigint_closes_lifetime_stack[*]` matrix have
+# been observed to hang past pytest's wall-clock under
+# `main_thread_forkserver`, leaving subactor forks that
+# squat on registrar resources and cascade-fail every
+# subsequent test (`test_inter_peer_cancellation`,
+# `test_legacy_one_way_streaming`, etc.).
+pytestmark = pytest.mark.usefixtures(
+    'reap_subactors_per_test',
+    # NOTE, asyncio cancel cascade has historically
+    # triggered both UDS sockfile leaks (SIGKILL path)
+    # AND the trio `WakeupSocketpair.drain()` busy-loop
+    # — see `test_aio_simple_error`'s history.
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
 
 
 @pytest.fixture(
@@ -183,6 +209,7 @@ def test_tractor_cancels_aio(
     async def main():
         async with tractor.open_nursery(
             debug_mode=debug_mode,
+            registry_addrs=[reg_addr],
         ) as an:
             portal = await an.run_in_actor(
                 asyncio_actor,
@@ -205,11 +232,11 @@ def test_trio_cancels_aio(
 
     '''
     async def main():
-
-        with trio.move_on_after(1):
         # cancel the nursery shortly after boot
-
+        with trio.move_on_after(1):
-            async with tractor.open_nursery() as tn:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as tn:
                 await tn.run_in_actor(
                     asyncio_actor,
                     target='aio_sleep_forever',
@@ -277,7 +304,9 @@ def test_context_spawns_aio_task_that_errors(
     '''
     async def main():
         with trio.fail_after(1 + delay):
-            async with tractor.open_nursery() as an:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
                 p = await an.start_actor(
                     'aio_daemon',
                     enable_modules=[__name__],
@@ -360,7 +389,9 @@ def test_aio_cancelled_from_aio_causes_trio_cancelled(
     async def main():
 
         an: tractor.ActorNursery
-        async with tractor.open_nursery() as an:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as an:
             p: tractor.Portal = await an.run_in_actor(
                 asyncio_actor,
                 target='aio_cancel',
@@ -569,7 +600,9 @@ def test_basic_interloop_channel_stream(
     async def main():
         # TODO, figure out min timeout here!
         with trio.fail_after(6):
-            async with tractor.open_nursery() as an:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
                 portal = await an.run_in_actor(
                     stream_from_aio,
                     infect_asyncio=True,
@@ -582,9 +615,13 @@ def test_basic_interloop_channel_stream(
 
 
 # TODO: parametrize the above test and avoid the duplication here?
-def test_trio_error_cancels_intertask_chan(reg_addr):
+def test_trio_error_cancels_intertask_chan(
+    reg_addr: tuple[str, int],
+):
     async def main():
-        async with tractor.open_nursery() as an:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as an:
             portal = await an.run_in_actor(
                 stream_from_aio,
                 trio_raise_err=True,
@@ -619,6 +656,7 @@ def test_trio_closes_early_causes_aio_checkpoint_raise(
             async with tractor.open_nursery(
                 debug_mode=debug_mode,
                 # enable_stack_on_sig=True,
+                registry_addrs=[reg_addr],
             ) as an:
                 portal = await an.run_in_actor(
                     stream_from_aio,
@@ -667,6 +705,7 @@ def test_aio_exits_early_relays_AsyncioTaskExited(
     async def main():
         with trio.fail_after(1 + delay):
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
                 # enable_stack_on_sig=True,
             ) as an:
@@ -707,6 +746,7 @@ def test_aio_errors_and_channel_propagates_and_closes(
 ):
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.run_in_actor(
@@ -796,16 +836,47 @@ async def trio_to_aio_echo_server(
 
 @pytest.mark.parametrize(
     'raise_error_mid_stream',
-    [False, Exception, KeyboardInterrupt],
+    [
+        False,
+        Exception,
+        KeyboardInterrupt,
+    ],
     ids='raise_error={}'.format,
 )
 def test_echoserver_detailed_mechanics(
     reg_addr: tuple[str, int],
     debug_mode: bool,
     raise_error_mid_stream,
+
+    is_forking_spawner: bool,
+    fail_after_w_trace: FailAfterWTraceFactory,
 ):
-    async def main():
+    # NOTE: under fork-based backends the cancel-cascade
+    # path is structurally slower than `trio`'s subproc-exec
+    # (per-spawn forkserver-handshake compounds during
+    # teardown). Bump the cap so cross-test contamination
+    # doesn't flake this — see
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    timeout: float = (
+        999 if tractor.debug_mode()
+        else 4 if is_forking_spawner
+        # was 1; the `trio` 0.29 -> 0.33 bump slowed the
+        # cancel-cascade so a 1s budget raced the ~1s teardown
+        # deadline. On a deadline-fire the injected
+        # `Cancelled(source='deadline')` wraps the mid-stream
+        # KBI in a `BaseExceptionGroup`, breaking the bare
+        # `pytest.raises(KeyboardInterrupt)` below. See
+        # `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
+        else 4
+    )
+
+    # body factored out so the `fail_after_w_trace`-wrapping
+    # `main()` stays a 2-liner — keeps the deep `open_nursery`
+    # /`open_context`/`open_stream` block at its natural indent
+    # level instead of pushing it under yet another `async with`.
+    async def _body():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             p = await an.start_actor(
@@ -849,6 +920,15 @@ def test_echoserver_detailed_mechanics(
             # is cancelled by kbi or out of task cancellation
             await p.cancel_actor()
 
+    async def main():
+        # on-timeout diag snapshot via `fail_after_w_trace`
+        # — when the cancel cascade hangs under MTF we get a
+        # fresh `ptree`/`wchan`/`py-spy` dump on disk INSTEAD
+        # of an opaque pytest timeout-kill. See
+        # `tractor/_testing/trace.py`.
+        async with fail_after_w_trace(timeout):
+            await _body()
+
     if raise_error_mid_stream:
         with pytest.raises(raise_error_mid_stream):
             trio.run(main)
@@ -984,7 +1064,7 @@ async def manage_file(
     ],
     ids=[
         'bg_aio_task',
-        'just_trio_slee',
+        'just_trio_sleep',
     ],
 )
 @pytest.mark.parametrize(
@@ -1000,11 +1080,15 @@ async def manage_file(
 )
 def test_sigint_closes_lifetime_stack(
     tmp_path: Path,
+    reg_addr: tuple,
+    debug_mode: bool,
+
     wait_for_ctx: bool,
     bg_aio_task: bool,
     trio_side_is_shielded: bool,
-    debug_mode: bool,
     send_sigint_to: str,
+    is_forking_spawner: bool,
+    afk_alarm_w_trace: AfkAlarmWTraceFactory,
 ):
     '''
     Ensure that an infected child can use the `Actor.lifetime_stack`
@@ -1014,12 +1098,30 @@ def test_sigint_closes_lifetime_stack(
     '''
     async def main():
 
-        delay = 999 if tractor.debug_mode() else 1
+        delay: float = (
+            999
+            if debug_mode
+            else 1
+        )
+        # pre-init so the `except (KeyboardInterrupt, ContextCancelled)`
+        # handler below doesn't `UnboundLocalError` if KBI fires BEFORE
+        # we ever enter the `as (ctx, first)` body (e.g. when
+        # `p.open_context().__aenter__` is hung waiting for the
+        # subactor's `StartAck` due to a fork-child IPC race —
+        # see `dynamic_pub_sub_spawn_time_transport_close_under_mtf_issue.md`).
+        tmp_file: Path|None = None
+        ctx: tractor.Context|None = None
         try:
             an: tractor.ActorNursery
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
             ) as an:
+
+                # sanity
+                if debug_mode:
+                    assert tractor.debug_mode()
+
                 p: tractor.Portal = await an.start_actor(
                     'file_mngr',
                     enable_modules=[__name__],
@@ -1034,7 +1136,7 @@ def test_sigint_closes_lifetime_stack(
                 ) as (ctx, first):
 
                     path_str, cpid = first
-                    tmp_file: Path = Path(path_str)
+                    tmp_file = Path(path_str)
                     assert tmp_file.exists()
 
                     # XXX originally to simulate what (hopefully)
@@ -1054,6 +1156,10 @@ def test_sigint_closes_lifetime_stack(
                         cpid if send_sigint_to == 'child'
                         else os.getpid()
                     )
+                    print(
+                        f'Sending SIGINT to {send_sigint_to!r}\n'
+                        f'pid: {pid!r}\n'
+                    )
                     os.kill(
                         pid,
                         signal.SIGINT,
@@ -1064,13 +1170,37 @@ def test_sigint_closes_lifetime_stack(
                     # timeout should trigger!
                     if wait_for_ctx:
                         print('waiting for ctx outcome in parent..')
+
+                        if debug_mode:
+                            assert delay == 999
+
                         try:
-                            with trio.fail_after(1 + delay):
+                            with trio.fail_after(
+                                1 + delay
+                            ):
                                 await ctx.wait_for_result()
                         except tractor.ContextCancelled as ctxc:
                             assert ctxc.canceller == ctx.chan.uid
                             raise
 
+                        except trio.TooSlowError:
+                            if (
+                                send_sigint_to == 'child'
+                                and
+                                is_forking_spawner
+                            ):
+                                pytest.xfail(
+                                    reason=(
+                                        'SIGINT delivery to fork-child subactor is known '
+                                        'to NOT SUCCEED, precisely bc we have not wired up a'
+                                        '"trio SIGINT mode" in the child pre-fork.\n'
+                                        'Also see `test_orphaned_subactor_sigint_cleanup_DRAFT` for'
+                                        'a dedicated suite demonstrating this expected limitation as '
+                                        'well as the detailed doc:\n'
+                                        '`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.\n'
+                                    ),
+                                )
+
                     # XXX CASE 2: this seems to be the source of the
                     # original issue which exhibited BEFORE we put
                     # a `Actor.cancel_soon()` inside
@@ -1084,6 +1214,21 @@ def test_sigint_closes_lifetime_stack(
             KeyboardInterrupt,
             ContextCancelled,
         ):
+            # If we got here BEFORE entering the ctx body (e.g.
+            # spawn-time IPC race hung `open_context.__aenter__` and
+            # the AFK-guard `signal.alarm` fired KBI from outside the
+            # trio loop), `tmp_file`/`ctx` are still `None` — surface
+            # that fact directly instead of `UnboundLocalError`.
+            if tmp_file is None:
+                pytest.fail(
+                    'KBI/ctxc fired BEFORE `p.open_context()` returned '
+                    "the child's `started` value — likely fork-child "
+                    'IPC race; see '
+                    '`ai/conc-anal/'
+                    'dynamic_pub_sub_spawn_time_transport_close_'
+                    'under_mtf_issue.md`'
+                )
+
             # XXX CASE 2: without the bug fixed, in the
             # KBI-raised-in-parent case, the actor teardown should
             # never get run (silently abaondoned by `asyncio`..) and
@@ -1091,6 +1236,25 @@ def test_sigint_closes_lifetime_stack(
             assert not tmp_file.exists()
             assert ctx.maybe_error
 
+    # outer hard wall-clock backstop via `afk_alarm_w_trace`:
+    # when the in-band trio cancel path doesn't fire (e.g.
+    # parent is parked in a shielded `await` inside actor-
+    # nursery teardown, or `open_context.__aenter__` hangs
+    # waiting for a child's `StartAck` that never comes), the
+    # `signal.alarm` inside the CM raises `AFKAlarmTimeout`
+    # in the main thread regardless of trio's scope state —
+    # AND captures a full diag snapshot to
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` before re-raising.
+    # Only armed under fork-based backends since this hang-
+    # class is MTF-specific.
+    if (
+        not debug_mode
+        and
+        is_forking_spawner
+    ):
+        with afk_alarm_w_trace(10):
+            trio.run(main)
+    else:
         trio.run(main)
 
 
@@ -1170,6 +1334,7 @@ def test_aio_side_raises_before_started(
         with trio.fail_after(3):
             an: tractor.ActorNursery
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
                 loglevel=loglevel,
             ) as an:

tests/test_inter_peer_cancellation.py

@@ -26,6 +26,31 @@ from tractor._testing import (
 
 from .conftest import cpu_scaling_factor
 
+pytestmark = [
+    pytest.mark.skipon_spawn_backend(
+        'subint',
+        reason=(
+            'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
+            'Inter-peer cancel cascades under '
+            '`--spawn-backend=subint` trip the abandoned-subint '
+            'GIL-hostage class — see\n'
+            '  - `ai/conc-anal/subint_sigint_starvation_issue.md` '
+            '(GIL-hostage, SIGINT-unresponsive)\n'
+            '  - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
+            '(sibling: parent parks on dead chan)\n'
+            '  - https://github.com/goodboy/tractor/issues/379 '
+            '(subint umbrella)\n'
+        )
+    ),
+    # NOTE, inter-peer cancellation tests stress the
+    # multi-actor cancel cascade which under SIGKILL
+    # leaves UDS sock-files orphaned. Track per-test
+    # for blame attribution.
+    pytest.mark.usefixtures(
+        'track_orphaned_uds_per_test',
+    ),
+]
+
 # XXX TODO cases:
 # - [x] WE cancelled the peer and thus should not see any raised
 #   `ContextCancelled` as it should be reaped silently?

tests/test_legacy_one_way_streaming.py

@@ -1,7 +1,7 @@
-"""
+'''
 Streaming via the, now legacy, "async-gen API".
 
-"""
+'''
 import time
 from functools import partial
 import platform
@@ -12,6 +12,11 @@ import tractor
 import pytest
 
 from tractor._testing import tractor_test
+from tractor._exceptions import ActorTooSlowError
+
+_non_linux: bool = (
+    _sys := platform.system()
+) != 'Linux'
 
 
 def test_must_define_ctx():
@@ -68,8 +73,10 @@ async def stream_from_single_subactor(
     start_method,
     stream_func,
 ):
-    """Verify we can spawn a daemon actor and retrieve streamed data.
+    '''
-    """
+    Verify we can spawn a daemon actor and retrieve streamed data.
+
+    '''
     # only one per host address, spawns an actor if None
 
     async with tractor.open_nursery(
@@ -242,14 +249,19 @@ async def a_quadruple_example() -> list[int]:
         start = time.time()
         # the portal call returns exactly what you'd expect
         # as if the remote "aggregate" function was called locally
-        result_stream = []
+        result_stream: list[int] = []
 
-        async with portal.open_stream_from(aggregate, seed=seed) as stream:
+        async with portal.open_stream_from(
+            aggregate,
+            seed=seed,
+        ) as stream:
             async for value in stream:
                 result_stream.append(value)
 
-        print(f"STREAM TIME = {time.time() - start}")
+        print(
-        print(f"STREAM + SPAWN TIME = {time.time() - pre_start}")
+            f"STREAM TIME = {time.time() - start}\n"
+            f"STREAM + SPAWN TIME = {time.time() - pre_start}\n"
+        )
         assert result_stream == list(range(seed))
         await portal.cancel_actor()
         return result_stream
@@ -258,13 +270,24 @@ async def a_quadruple_example() -> list[int]:
 async def cancel_after(
     wait: float,
     reg_addr: tuple,
+    expect_cancel: bool,
 ) -> list[int]:
 
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
     ):
-        with trio.move_on_after(wait):
+        res: list[int]|None = None
-            return await a_quadruple_example()
+        with trio.move_on_after(wait) as cs:
+            res: list[int] = await a_quadruple_example()
+            return res
+
+        if (
+            not expect_cancel
+            and
+            cs.cancelled_caught
+        ):
+            assert not res
+            raise ActorTooSlowError
 
 
 @pytest.fixture(scope='module')
@@ -272,9 +295,14 @@ def time_quad_ex(
     reg_addr: tuple,
     ci_env: bool,
     spawn_backend: str,
+    is_forking_spawner: bool,
+    tpt_proto: str,
 ):
-    non_linux: bool = (_sys := platform.system()) != 'Linux'
+    if (
-    if ci_env and non_linux:
+        ci_env
+        and
+        _non_linux
+    ):
         pytest.skip(f'Test is too flaky on {_sys!r} in CI')
 
     if spawn_backend == 'mp':
@@ -284,15 +312,42 @@ def time_quad_ex(
         '''
         pytest.skip("Test is too flaky on mp in CI")
 
-    timeout = 7 if non_linux else 4
+    timeout: float = (
-    start = time.time()
+        7 if _non_linux
-    results: list[int] = trio.run(
+        else 4
-        cancel_after,
-        timeout,
-        reg_addr,
     )
+
+    if (
+        is_forking_spawner
+        and
+        tpt_proto in [
+            'uds',
+        ]
+    ):
+        timeout += 1
+
+    # inflate the cancel-deadline for CPU-freq scaling AND/OR CI
+    # latency (see `cpu_scaling_factor()`) so the example isn't
+    # cancelled mid-stream on a throttled/CI runner.
+    from .conftest import cpu_scaling_factor
+    timeout *= cpu_scaling_factor()
+
+    start: float = time.time()
+    results: list[int] = trio.run(partial(
+        cancel_after,
+        wait=timeout,
+        reg_addr=reg_addr,
+        expect_cancel=True,
+    ))
     diff: float = time.time() - start
-    assert results
+    if results is None:
+        raise ActorTooSlowError(
+            f'Streaming example took longer then timeout ??\n'
+            f'timeout={timeout!r}\n'
+            f'diff={diff!r}\n'
+            f'results={results!r}\n'
+        )
+
     return results, diff
 
 
@@ -307,11 +362,10 @@ def test_a_quadruple_example(
     given past empirical eval of this suite.
 
     '''
-    non_linux: bool = (_sys := platform.system()) != 'Linux'
 
     this_fast_on_linux: float = 3
     this_fast = (
-        6 if non_linux
+        6 if _non_linux
         else this_fast_on_linux
     )
     # ^ XXX NOTE,
@@ -348,21 +402,26 @@ def test_not_fast_enough_quad(
     reg_addr: tuple,
     time_quad_ex: tuple[list[int], float],
     cancel_delay: float,
+
     ci_env: bool,
     spawn_backend: str,
+    is_forking_spawner: bool,
+    tpt_proto: str,
+    test_log: tractor.log.StackLevelAdapter,
 ):
     '''
-    Verify we can cancel midway through the quad example and all
+    Verify we can cancel midway through `a_quadruple_example()`, at
-    actors cancel gracefully.
+    various delays, and all subactors cancel gracefully.
 
     '''
     results, diff = time_quad_ex
     delay = max(diff - cancel_delay, 0)
-    results = trio.run(
+    results: list[int] = trio.run(partial(
         cancel_after,
-        delay,
+        wait=delay,
-        reg_addr,
+        reg_addr=reg_addr,
-    )
+        expect_cancel=True,
+    ))
     system: str = platform.system()
     if (
         system in ('Windows', 'Darwin')
@@ -373,6 +432,20 @@ def test_not_fast_enough_quad(
         # so just ignore these
         print(f'Woa there {system} caught your breath eh?')
     else:
+        if (
+            results
+            and
+            is_forking_spawner
+            and
+            tpt_proto in [
+                'uds',
+            ]
+        ):
+            pytest.xfail(
+                f'Spawning backend + tpt-proto is too fast XD\n'
+                f'{spawn_backend!r} + {tpt_proto!r}\n'
+            )
+
         # should be cancelled mid-streaming
         assert results is None
 

tests/test_log_sys.py

@@ -20,6 +20,16 @@ def test_root_pkg_not_duplicated_in_logger_name():
     a common `<root_name>.< >` prefix, ensure that it is not
     duplicated in the child's `StackLevelAdapter.name: str`.
 
+    Also pins the explicit-`name` contract: an explicitly passed
+    dotted `name` is treated as a *literal* sub-logger path and is
+    NOT leaf-collapsed. The leaf-module is only dropped when the
+    trailing token duplicates the *caller's own* `__name__` leaf (the
+    `{filename}` field) — see `test_implicit_mod_name_applied_for_child`
+    for that (auto-naming) path. This is what keeps a real (possibly
+    nested) sub-PACKAGE like `subpkg.mod` -> `devx.debug` addressable
+    by the `tractor.log` logging-spec, instead of collapsing to its
+    parent.
+
     '''
     project_name: str = 'pylib'
     pkg_path: str = 'pylib.subpkg.mod'
@@ -38,8 +48,13 @@ def test_root_pkg_not_duplicated_in_logger_name():
     )
 
     assert proj_log is not sublog
+    # the root pkg-name appears exactly once (no `pylib.pylib...`)
     assert sublog.name.count(proj_log.name) == 1
-    assert 'mod' not in sublog.name
+    # explicit dotted `name` is preserved literally (NOT collapsed);
+    # the trailing token survives since it's not the *caller's* own
+    # leaf-module (`test_log_sys`), so this is treated as a literal
+    # sub-pkg path.
+    assert sublog.name == f'{project_name}.subpkg.mod'
 
 
 def test_implicit_mod_name_applied_for_child(

tests/test_multi_program.py

@@ -1,183 +0,0 @@
-"""
-Multiple python programs invoking the runtime.
-"""
-from __future__ import annotations
-import platform
-import subprocess
-import time
-from typing import (
-    TYPE_CHECKING,
-)
-
-import pytest
-import trio
-import tractor
-from tractor._testing import (
-    tractor_test,
-)
-from tractor import (
-    current_actor,
-    Actor,
-    Context,
-    Portal,
-)
-from tractor.runtime import _state
-from .conftest import (
-    sig_prog,
-    _INT_SIGNAL,
-    _INT_RETURN_CODE,
-)
-
-if TYPE_CHECKING:
-    from tractor.msg import Aid
-    from tractor.discovery._addr import (
-        UnwrappedAddress,
-    )
-
-
-_non_linux: bool = platform.system() != 'Linux'
-
-
-def test_abort_on_sigint(
-    daemon: subprocess.Popen,
-):
-    assert daemon.returncode is None
-    time.sleep(0.1)
-    sig_prog(daemon, _INT_SIGNAL)
-    assert daemon.returncode == _INT_RETURN_CODE
-
-    # XXX: oddly, couldn't get capfd.readouterr() to work here?
-    if platform.system() != 'Windows':
-        # don't check stderr on windows as its empty when sending CTRL_C_EVENT
-        assert "KeyboardInterrupt" in str(daemon.stderr.read())
-
-
-@tractor_test
-async def test_cancel_remote_registrar(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-):
-    assert not current_actor().is_registrar
-    async with tractor.get_registry(reg_addr) as portal:
-        await portal.cancel_actor()
-
-    time.sleep(0.1)
-    # the registrar channel server is cancelled but not its main task
-    assert daemon.returncode is None
-
-    # no registrar socket should exist
-    with pytest.raises(OSError):
-        async with tractor.get_registry(reg_addr) as portal:
-            pass
-
-
-def test_register_duplicate_name(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-):
-    async def main():
-        async with tractor.open_nursery(
-            registry_addrs=[reg_addr],
-        ) as an:
-
-            assert not current_actor().is_registrar
-
-            p1 = await an.start_actor('doggy')
-            p2 = await an.start_actor('doggy')
-
-            async with tractor.wait_for_actor('doggy') as portal:
-                assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
-
-            await an.cancel()
-
-    # XXX, run manually since we want to start this root **after**
-    # the other "daemon" program with it's own root.
-    trio.run(main)
-
-
-@tractor.context
-async def get_root_portal(
-    ctx: Context,
-):
-    '''
-    Connect back to the root actor manually (using `._discovery` API)
-    and ensure it's contact info is the same as our immediate parent.
-
-    '''
-    sub: Actor = current_actor()
-    rtvs: dict = _state._runtime_vars
-    raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
-
-    # await tractor.pause()
-    # XXX, in case the sub->root discovery breaks you might need
-    # this (i know i did Xp)!!
-    # from tractor.devx import mk_pdb
-    # mk_pdb().set_trace()
-
-    assert (
-        len(raddrs) == 1
-        and
-        list(sub._parent_chan.raddr.unwrap()) in raddrs
-    )
-
-    # connect back to our immediate parent which should also
-    # be the actor-tree's root.
-    from tractor.discovery._api import get_root
-    ptl: Portal
-    async with get_root() as ptl:
-        root_aid: Aid = ptl.chan.aid
-        parent_ptl: Portal = current_actor().get_parent()
-        assert (
-            root_aid.name == 'root'
-            and
-            parent_ptl.chan.aid == root_aid
-        )
-        await ctx.started()
-
-
-def test_non_registrar_spawns_child(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-    loglevel: str,
-    debug_mode: bool,
-    ci_env: bool,
-):
-    '''
-    Ensure a non-regristar (serving) root actor can spawn a sub and
-    that sub can connect back (manually) to it's rent that is the
-    root without issue.
-
-    More or less this audits the global contact info in
-    `._state._runtime_vars`.
-
-    '''
-    async def main():
-
-        # XXX, since apparently on macos in GH's CI it can be a race
-        # with the `daemon` registrar on grabbing the socket-addr..
-        if ci_env and _non_linux:
-            await trio.sleep(.5)
-
-        async with tractor.open_nursery(
-            registry_addrs=[reg_addr],
-            loglevel=loglevel,
-            debug_mode=debug_mode,
-        ) as an:
-
-            actor: Actor = tractor.current_actor()
-            assert not actor.is_registrar
-            sub_ptl: Portal = await an.start_actor(
-                name='sub',
-                enable_modules=[__name__],
-            )
-
-            async with sub_ptl.open_context(
-                get_root_portal,
-            ) as (ctx, _):
-                print('Waiting for `sub` to connect back to us..')
-
-            await an.cancel()
-
-    # XXX, run manually since we want to start this root **after**
-    # the other "daemon" program with it's own root.
-    trio.run(main)

tests/test_pubsub.py

@@ -7,6 +7,14 @@ import tractor
 from tractor.experimental import msgpub
 from tractor._testing import tractor_test
 
+pytestmark = pytest.mark.skipon_spawn_backend(
+    'subint',
+    reason=(
+        'XXX SUBINT HANGING TEST XXX\n'
+        'See oustanding issue(s)\n'
+        # TODO, put issue link!
+    )
+)
 
 def test_type_checks():
 

tests/test_ringbuf.py

@@ -21,6 +21,9 @@ from tractor._testing.samples import (
 # XXX, in case you want to melt your cores, comment this skip line XD
 pytestmark = pytest.mark.skip
 
+# XXX `cffi` dun build on py3.14 yet..
+cffi = pytest.importorskip("cffi")
+
 
 @tractor.context
 async def child_read_shm(

tests/test_shm.py

@@ -14,6 +14,20 @@ from tractor.ipc._shm import (
     attach_shm_list,
 )
 
+pytestmark = pytest.mark.skipon_spawn_backend(
+    'subint',
+    # NOTE, `main_thread_forkserver` works for these tests
+    # via the `mp.SharedMemory(track=False)` +
+    # `mp.resource_tracker` monkey-patch in `.ipc._mp_bs`.
+    # Without that workaround the fork-inherited
+    # `resource_tracker` fd would EBADF on first shm op +
+    # cascade into `FileExistsError` across parametrize
+    # variants. Tracker doc:
+    # `ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`.
+    reason=(
+        'subint: GIL-contention hanging class.\n'
+    )
+)
 
 @tractor.context
 async def child_attach_shml_alot(

tests/test_spawning.py

@@ -194,9 +194,14 @@ def test_loglevel_propagated_to_subactor(
     reg_addr: tuple,
     level: str,
 ):
-    if start_method == 'mp_forkserver':
+    if start_method in ('mp_forkserver', 'main_thread_forkserver'):
         pytest.skip(
-            "a bug with `capfd` seems to make forkserver capture not work?"
+            "a bug with `capfd` seems to make forkserver capture not work? "
+            "(same class as the `mp_forkserver` pre-existing skip — fork-"
+            "based backends inherit pytest's capfd temp-file fds into the "
+            "subactor and the IPC handshake reads garbage (`unclean EOF "
+            "read only X/HUGE_NUMBER bytes`). Work around by using "
+            "`capsys` instead or skip entirely."
         )
 
     async def main():

tests/trionics/test_patches.py

@@ -0,0 +1,99 @@
+'''
+Regression tests for `tractor.trionics.patches` —
+defensive monkey-patches on upstream `trio` bugs.
+
+Each test asserts:
+
+1. The bug exists (or is gone — skip cleanly if
+   upstream shipped the fix and our `is_needed()` now
+   returns `False`).
+2. Our patch fixes it (post-`apply()` the `repro()`
+   returns cleanly within a tight wall-clock cap).
+
+Wall-clock caps are critical here — the bugs we patch
+are tight-loops or deadlocks, so a regression would
+HANG the test runner unless we hard-cap each
+`repro()` call.
+
+'''
+import signal
+
+import pytest
+
+from tractor.trionics import patches
+from tractor.trionics.patches import _wakeup_socketpair as wsp
+
+
+@pytest.fixture(autouse=True)
+def _alarm_cleanup():
+    '''
+    Ensure no leftover SIGALRM survives a test failure
+    or unexpected return.
+
+    '''
+    yield
+    signal.alarm(0)
+
+
+def test_wakeup_socketpair_drain_eof_patch_works():
+    '''
+    Without the patch, `WakeupSocketpair.drain()` on a
+    socketpair whose write-end has been closed spins
+    forever. With the patch applied, it returns
+    cleanly within milliseconds.
+
+    Wall-clock cap: 2s. If the patch regresses, SIGALRM
+    fires and the test hard-fails with a clear signal
+    instead of hanging CI indefinitely.
+
+    '''
+    if not wsp.is_needed():
+        pytest.skip(
+            'upstream trio shipped the fix — '
+            'patch no longer needed for trio '
+            '(see `is_needed()` for version gate)'
+        )
+
+    # Apply the patch.
+    applied: bool = wsp.apply()
+    # First call MUST return True; idempotent guard
+    # prevents False on subsequent calls within the
+    # same process.
+    assert isinstance(applied, bool)  # idempotent (order-dependent value)
+
+    # Cap wall-clock at 2s; SIGALRM raises in main
+    # thread which interrupts the C-level recv loop
+    # IF the patch regresses (since `signal.alarm`
+    # uses Python's signal-wakeup-fd which the patch
+    # itself relies on... but `repro()` runs OUTSIDE
+    # a trio.run, so it's plain stdlib semantics here
+    # — alarm WILL fire during `recv` syscall).
+    signal.alarm(2)
+    wsp.repro()
+    signal.alarm(0)
+
+
+def test_apply_all_idempotent():
+    '''
+    Calling `apply_all()` twice should not double-
+    apply: second call's dict has all-False values
+    (every patch reports "already applied").
+
+    '''
+    first: dict[str, bool] = patches.apply_all()
+    second: dict[str, bool] = patches.apply_all()
+
+    # Second call: every patch reports skipped.
+    assert all(v is False for v in second.values()), (
+        f'apply_all() not idempotent: {second}'
+    )
+
+    # First call: at least one patch was applied
+    # (or all are no-ops because `is_needed()` is
+    # False everywhere — the all-fixed-upstream future
+    # state which is also valid).
+    assert isinstance(first, dict)
+    for name, applied in first.items():
+        assert isinstance(applied, bool), (
+            f'patch {name!r} returned non-bool: {applied!r}'
+        )

tractor/_child.py

@@ -15,16 +15,23 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 """
-This is the "bootloader" for actors started using the native trio backend.
+The "bootloader" for sub-actors spawned via the native `trio`
+backend (the default `python -m tractor._child` CLI entry) and
+the in-process `subint` backend (`tractor.spawn._subint`).
 
 """
+from __future__ import annotations
 import argparse
-
 from ast import literal_eval
+from typing import TYPE_CHECKING
 
 from .runtime._runtime import Actor
 from .spawn._entry import _trio_main
 
+if TYPE_CHECKING:
+    from .discovery._addr import UnwrappedAddress
+    from .spawn._spawn import SpawnMethodKey
+
 
 def parse_uid(arg):
     name, uuid = literal_eval(arg)  # ensure 2 elements
@@ -39,6 +46,73 @@ def parse_ipaddr(arg):
         return arg
 
 
+def _actor_child_main(
+    uid: tuple[str, str],
+    loglevel: str | None,
+    parent_addr: UnwrappedAddress | None,
+    infect_asyncio: bool,
+    spawn_method: SpawnMethodKey = 'trio',
+
+) -> None:
+    '''
+    Construct the child `Actor` and dispatch to `_trio_main()`.
+
+    Shared entry shape used by both the `python -m tractor._child`
+    CLI (trio/mp subproc backends) and the `subint` backend, which
+    invokes this from inside a fresh `concurrent.interpreters`
+    sub-interpreter via `Interpreter.call()`.
+
+    '''
+    # Apply defensive monkey-patches for upstream `trio`
+    # bugs we've encountered while running tractor — see
+    # `tractor.trionics.patches` for the catalog +
+    # per-patch upstream-fix tracking. Must run BEFORE
+    # any trio runtime init.
+    from .trionics.patches import apply_all
+    apply_all()
+
+    subactor = Actor(
+        name=uid[0],
+        uuid=uid[1],
+        loglevel=loglevel,
+        spawn_method=spawn_method,
+    )
+
+    # XXX, set a stable OS-level proc-title BEFORE entering
+    # the trio runtime so `ps`/`top`/`acli.pytree` and
+    # orphan-reapers can identify this actor for its full
+    # lifetime — e.g.
+    #   `tractor[doggy@1027301b]`
+    # vs. the default uninformative
+    #   `python -m tractor._child --uid (...)`
+    #
+    # `setproctitle` mutates `argv[0]` (visible in
+    # `/proc/<pid>/cmdline`) AND the kernel `comm`
+    # (visible in `/proc/<pid>/comm`, kernel-truncated to
+    # ~15 bytes, but preserved through zombie state). Both
+    # surfaces are enough for `_testing._reap` /
+    # `acli.reap` orphan- and zombie-detection to identify
+    # tractor sub-actors via intrinsic signals — no cwd,
+    # venv path, or env-var coincidence-of-implementation
+    # matching needed.
+    #
+    # NB: an earlier draft also wrote `TRACTOR_AID` to
+    # `os.environ` here for `pgrep --env`-style discovery,
+    # but Linux snapshots `/proc/<pid>/environ` at exec/fork
+    # time, so post-fork runtime mutations don't propagate
+    # to the kernel-visible env. The proc-title path
+    # provides equivalent ergonomics
+    # (`pgrep -f 'tractor\['`) without that gotcha.
+    from .devx._proctitle import set_actor_proctitle
+    set_actor_proctitle(subactor)
+
+    _trio_main(
+        subactor,
+        parent_addr=parent_addr,
+        infect_asyncio=infect_asyncio,
+    )
+
+
 if __name__ == "__main__":
     __tracebackhide__: bool = True
 
@@ -49,15 +123,10 @@ if __name__ == "__main__":
     parser.add_argument("--asyncio", action='store_true')
     args = parser.parse_args()
 
-    subactor = Actor(
+    _actor_child_main(
-        name=args.uid[0],
+        uid=args.uid,
-        uuid=args.uid[1],
         loglevel=args.loglevel,
-        spawn_method="trio"
-    )
-
-    _trio_main(
-        subactor,
         parent_addr=args.parent_addr,
         infect_asyncio=args.asyncio,
+        spawn_method='trio',
     )

tractor/_exceptions.py

@@ -89,6 +89,28 @@ class ActorFailure(RuntimeFailure):
     '''
 
 
+class ActorTooSlowError(RuntimeFailure):
+    '''
+    A peer-`Actor` failed to ack an actor-runtime cancel-cascade
+    request (e.g. `Portal.cancel_actor()` -> `Actor.cancel()`)
+    within the bounded wait window.
+
+    Distinct exc-type (NOT a `trio.TooSlowError` subclass) so that
+    `except trio.TooSlowError:` blocks elsewhere in the test-suite
+    or `tractor` internals do NOT silently mask actor-cancel
+    timeouts — these MUST propagate so a supervisor can escalate
+    to `proc.terminate()` (hard-kill) per SC-discipline:
+
+      graceful cancel-req -> bounded wait -> hard-kill
+
+    Reason: see #subint_forkserver duplicate-name hang
+    diagnosis where `Portal.cancel_actor()` silently swallowed
+    the timeout and the supervisor never escalated, leaving
+    a same-named sibling subactor parked forever.
+
+    '''
+
+
 class InternalError(RuntimeError):
     '''
     Entirely unexpected internal machinery error indicating

tractor/_root.py

@@ -69,6 +69,20 @@ from ._exceptions import (
 logger = log.get_logger('tractor')
 
 
+# Spawn backends under which `debug_mode=True` is supported.
+# Requirement: the spawned subactor's root runtime must be
+# trio-native so `tractor.devx.debug._tty_lock` works. Matches
+# both the enable-site in `open_root_actor` and the cleanup-
+# site reset of `_runtime_vars['_debug_mode']` — keep them in
+# lockstep when adding backends.
+_DEBUG_COMPATIBLE_BACKENDS: tuple[str, ...] = (
+    'trio',
+    # forkserver children run `_trio_main` in their own OS
+    # process — same child-side runtime shape as `trio_proc`.
+    'main_thread_forkserver',
+)
+
+
 # TODO: stick this in a `@acm` defined in `devx.debug`?
 # -[ ] also maybe consider making this a `wrapt`-deco to
 #     save an indent level?
@@ -227,6 +241,7 @@ async def open_root_actor(
             f'_registry_addrs: {registry_addrs!r}\n'
         )
 
+    # debug.mk_pdb().set_trace()
     async with maybe_block_bp(
         debug_mode=debug_mode,
         maybe_enable_greenback=maybe_enable_greenback,
@@ -270,6 +285,82 @@ async def open_root_actor(
             )
             enable_modules.extend(rpc_module_paths)
 
+        # `TRACTOR_LOGLEVEL` env-var wins over any caller-passed
+        # `loglevel` so devs/test-runs can crank (or silence)
+        # console verbosity without touching application code.
+        env_ll_report: str = ''
+        if env_ll := os.environ.get('TRACTOR_LOGLEVEL'):
+            # capture the caller-passed value BEFORE the env-var
+            # clobbers it, else the override-notice below is dead
+            # code (the `!=` compare is always `False`).
+            caller_ll: str|None = loglevel
+            loglevel = env_ll
+            env_ll_report: str = (
+                f'Detected env-var setting,\n'
+                f'TRACTOR_LOGLEVEL={env_ll!r}\n'
+                f'\n'
+                f'Setting console loglevel per,\n'
+                f'loglevel={loglevel!r}\n'
+            )
+            if (
+                caller_ll
+                and
+                caller_ll.upper() != env_ll.upper()
+            ):
+                env_ll_report += (
+                    f'\n'
+                    f'NOTE env-var OVERRIDES caller-passed,\n'
+                    f'loglevel={caller_ll!r}\n'
+                )
+
+        loglevel: str = (
+            loglevel
+            or
+            log._default_loglevel
+        )
+        loglevel: str = loglevel.upper()
+
+        assert loglevel
+        _log = log.get_console_log(
+            level=loglevel,
+            name='tractor',
+            logger=logger,
+        )
+        assert _log
+        if env_ll_report:
+            _log.info(env_ll_report)
+
+        # `TRACTOR_SPAWN_METHOD` env-var wins over any caller-passed
+        # `start_method` so devs/test-runs can swap the actor spawn
+        # backend without touching application code (e.g. driving
+        # the `examples/debugging/<script>.py` suite under each
+        # backend from `tests/devx/conftest.py::spawn`).
+        if env_sm := os.environ.get('TRACTOR_SPAWN_METHOD'):
+            # capture the caller-passed value BEFORE the env-var
+            # clobbers it (else the override-notice is dead code).
+            caller_sm: str|None = start_method
+            start_method: str = env_sm
+            env_sm_report: str = (
+                f'Detected env-var setting,\n'
+                f'TRACTOR_SPAWN_METHOD={env_sm!r}\n'
+                f'\n'
+                f'Setting spawn backend as,\n'
+                f'start_method={env_sm!r}\n'
+            )
+            if (
+                caller_sm
+                and
+                caller_sm != env_sm
+            ):
+                _log.warning(
+                    env_sm_report
+                    +
+                    f'NOTE env-var OVERRIDES caller-passed,\n'
+                    f'`start_method={caller_sm!r}`\n'
+                )
+            else:
+                _log.info(env_sm_report)
+
         if start_method is not None:
             _spawn.try_set_start_method(start_method)
 
@@ -286,17 +377,44 @@ async def open_root_actor(
             wrap_address(uw_addr)
             for uw_addr in uw_reg_addrs
         ]
-        loglevel: str = (
-            loglevel
-            or
-            log._default_loglevel
-        )
-        loglevel: str = loglevel.upper()
 
+        # fail-fast on `enable_transports` / `registry_addrs` proto
+        # mismatch — historically this caused a silent indefinite
+        # hang during the registrar handshake (registry was reachable
+        # only via a transport not in `enable_transports`, so the
+        # actor could never connect to register/discover). See
+        # `tests/ipc/test_multi_tpt.py::test_root_passes_tpt_to_sub`
+        # for the foot-gun case + its layer-1 skip-guard.
+        bad_addrs: list[tuple[str, Address]] = [
+            (addr.proto_key, addr)
+            for addr in registry_addrs
+            if addr.proto_key not in enable_transports
+        ]
+        if bad_addrs:
+            mismatch_lines: str = '\n'.join(
+                f'  - proto_key={pk!r}  addr={a!r}'
+                for pk, a in bad_addrs
+            )
+            raise ValueError(
+                f'`registry_addrs` contains addr(s) whose proto is '
+                f'not in `enable_transports`!\n'
+                f'enable_transports: {enable_transports!r}\n'
+                f'mismatched_addrs:\n'
+                f'{mismatch_lines}\n'
+                f'\n'
+                f'Either add the missing proto to '
+                f'`enable_transports`, or remove the addr from '
+                f'`registry_addrs`.'
+            )
+
+        # Debug-mode is currently only supported for backends whose
+        # subactor root runtime is trio-native (so `tractor.devx.
+        # debug._tty_lock` works). See `_DEBUG_COMPATIBLE_BACKENDS`
+        # module-const for the list.
         if (
             debug_mode
             and
-            _spawn._spawn_method == 'trio'
+            _spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
         ):
             _state._runtime_vars['_debug_mode'] = True
 
@@ -318,22 +436,18 @@ async def open_root_actor(
 
         elif debug_mode:
             raise RuntimeError(
-                "Debug mode is only supported for the `trio` backend!"
+                f'Debug mode currently supported only for '
+                f'{_DEBUG_COMPATIBLE_BACKENDS!r} spawn backends, not '
+                f'{_spawn._spawn_method!r}.'
             )
 
-        assert loglevel
-        _log = log.get_console_log(
-            level=loglevel,
-            name='tractor',
-        )
-        assert _log
-
         # TODO: factor this into `.devx._stackscope`!!
-        if (
+        #
-            debug_mode
+        # NOTE, intentionally NOT gated on `debug_mode` so SIGUSR1
-            and
+        # task-tree dumps work in plain (non-pdb) runs too — esp.
-            enable_stack_on_sig
+        # in infected-`asyncio` root processes where the default
-        ):
+        # SIGUSR1 action would otherwise terminate the proc.
+        if enable_stack_on_sig:
             from .devx._stackscope import enable_stack_on_sig
             enable_stack_on_sig()
 
@@ -619,7 +733,7 @@ async def open_root_actor(
             if (
                 debug_mode
                 and
-                _spawn._spawn_method == 'trio'
+                _spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
             ):
                 _state._runtime_vars['_debug_mode'] = False
 

tractor/_testing/addr.py

@@ -22,6 +22,7 @@ Might be eventually useful to expose as a util set from
 our `tractor.discovery` subsys?
 
 '''
+import os
 import random
 from typing import (
     Type,
@@ -31,17 +32,28 @@ from tractor.discovery import _addr
 
 def get_rando_addr(
     tpt_proto: str,
-    *,
-
-    # choose random port at import time
-    _rando_port: str = random.randint(1000, 9999)
-
 ) -> tuple[str, str|int]:
     '''
     Used to globally override the runtime to the
     per-test-session-dynamic addr so that all tests never conflict
     with any other actor tree using the default.
 
+    Cross-process isolation: TCP-port picks salt
+    `random.randint()` with `os.getpid()` so two parallel
+    pytest sessions (e.g. one running `--tpt-proto=tcp` and
+    another `--tpt-proto=uds` concurrently) almost-never
+    collide on the same port. Without the salt, the prior
+    impl's import-time `random.randint(1000, 9999)` default
+    arg was effectively a process-singleton with a 1/9000
+    chance of cross-run collision per pair — and when it
+    happened EVERY `reg_addr`-using test in BOTH runs would
+    fight over the bind, cascading into a chain of
+    "Address already in use" failures.
+
+    For UDS this concern doesn't apply: `UDSAddress.get_random()`
+    already builds socket paths from `os.getpid()` so each
+    pytest process gets its own socket-path namespace.
+
     '''
     addr_type: Type[_addr.Addres] = _addr._address_types[tpt_proto]
     def_reg_addr: tuple[str, int] = _addr._default_lo_addrs[tpt_proto]
@@ -51,9 +63,21 @@ def get_rando_addr(
     testrun_reg_addr: tuple[str, int|str]
     match tpt_proto:
         case 'tcp':
+            # Per-call randomness mixed with `os.getpid()` —
+            # see the docstring above for the cross-process
+            # isolation rationale. The mix means:
+            # - within one pytest session, two calls return
+            #   distinct ports (good for tests that need a
+            #   second-different-reg-addr in one fn body, e.g.
+            #   `test_tpt_bind_addrs::bind-subset-reg`),
+            # - across parallel pytest sessions, the pid bias
+            #   makes coincident port choices unlikely.
+            port: int = 1000 + (
+                random.randint(0, 8999) + os.getpid()
+            ) % 9000
             testrun_reg_addr = (
                 addr_type.def_bindspace,
-                _rando_port,
+                port,
             )
 
         # NOTE, file-name uniqueness (no-collisions) will be based on

tractor/_testing/pytest.py

@@ -24,16 +24,192 @@ from functools import (
     wraps,
 )
 import inspect
+import os
 import platform
 from typing import (
     Callable,
     get_args,
+    TYPE_CHECKING,
 )
+import warnings
 
 import pytest
 import tractor
+from tractor.spawn._spawn import SpawnMethodKey
 import trio
 
+# Re-export `_testing.trace`'s pytest fixtures so they're
+# picked up by pytest's plugin-discovery (this module is
+# loaded via `pytest_plugins` from `pyproject.toml`). The
+# `noqa: F401` annotations make linters tolerate the
+# unused-looking imports — they're load-bearing for pytest
+# discovery. The fixtures share their `name=` kw with the
+# underlying CM functions; the python-level identifiers
+# below carry the `_fixture` suffix to avoid module-scope
+# collision (see `_testing/trace.py` for details).
+from .trace import (  # noqa: F401
+    afk_alarm_w_trace_fixture,
+    fail_after_w_trace_fixture,
+)
+
+# Spawn-backend keys which may appear in `skipon_spawn_backend`
+# marks ahead of the named backend actually being registered in
+# `tractor.spawn._spawn.SpawnMethodKey`; such marks are inert
+# (they can never match an active backend) but must not break
+# collection.
+_IN_DEV_SPAWN_BACKENDS: tuple[str, ...] = (
+    'subint',
+    'subint_forkserver',
+    'main_thread_forkserver',
+)
+
+# Sub-plugin: zombie-subactor + UDS sock-file + shm
+# reaping fixtures live in `tractor._testing._reap`
+# alongside the underlying detection/cleanup helpers.
+# Loading `_reap` as a sub-plugin here keeps reaping
+# concerns co-located + this module focused on tractor-
+# tooling-specific hooks (option/marker/parametrize,
+# `tractor_test` deco, transport / spawn-method
+# fixtures).
+pytest_plugins: tuple[str, ...] = (
+    'tractor._testing._reap',
+)
+
+if TYPE_CHECKING:
+    from argparse import Namespace
+
+
+_cap_sys_passed_as_flag: bool = False
+
+# Spawn backends that need `--capture=sys` to avoid the
+# fork-child×pytest-capture-fd deadlock. See the long
+# NOTE in `pytest_load_initial_conftests` below for the
+# full mechanism + tradeoff write-up.
+_CAPSYS_REQUIRED_SPAWNERS: frozenset[str] = frozenset({
+    'main_thread_forkserver',
+    # TODO future variant-2 'subint_forkserver' lands
+    # here too once the impl is unblocked.
+})
+
+
+# XXX REQUIRED in order to enforce `--capture=` flag
+# pre test session.
+# https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
+@pytest.hookimpl(tryfirst=True)
+def pytest_load_initial_conftests(
+    early_config: pytest.Config,
+    parser: pytest.Parser,
+    args: list[str],
+):
+    '''
+    Validate the `--capture=` × `--spawn-backend=`
+    combination at session-startup.
+
+    Background
+    ----------
+    `--capture=sys` is REQUIRED for fork-based spawn backends (e.g.
+    `main_thread_forkserver`): default `--capture=fd` redirects fd
+    1,2 to temp files, and fork children inherit those fds — opaque
+    deadlocks happen in the pytest-capture-machinery ↔ fork-child
+    stdio interaction. `--capture=sys` only redirects Python- level
+    `sys.stdout`/`sys.stderr`, leaving fd 1,2 alone.
+
+    Trade-off (vs. `--capture=fd`):
+
+    - LOST: per-test attribution of subactor *raw-fd* output (C-ext
+      writes, `os.write(2, ...)`, subproc stdout). Not zero — those
+      go to the terminal, captured by CI's terminal-level capture,
+      just not per-test-scoped in the pytest failure report.
+
+    - KEPT: Python-level `print()` + `logging` capture per-test
+      (tractor's logger uses `sys.stderr`, so tractor log output IS
+      still attributed per-test).
+
+    - KEPT: user `pytest -s` for debugging (unaffected).
+
+    Full post-mortem in
+    `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
+
+    Validation policy:
+    - **CI mode** (`CI` env-var set): fail-fast at
+      session start if a fork-spawn backend is requested
+      WITHOUT `--capture=sys`. CI must be explicit; no
+      auto-fallbacks. Forces every CI matrix-row's run
+      line to declare its capture mode plainly.
+    - **Local mode** (no `CI` env-var): emit a loud
+      warning + suggest `--capture=sys`, but allow the
+      run to proceed. Lets devs experiment with the bad
+      combo (e.g. to validate whether recent
+      fork-survival fixes have made `--capture=fd` work
+      after all).
+
+    '''
+    global _cap_sys_passed_as_flag
+    opts_w_args: Namespace = parser.parse_known_args(args)
+    spawner: str|None = getattr(
+        opts_w_args,
+        'spawn_backend',
+        None,
+    )
+    capture: str|None = getattr(
+        opts_w_args,
+        'capture',
+        None,
+    )
+    if '--capture=sys' in args:
+        _cap_sys_passed_as_flag = True
+        assert capture == 'sys'
+
+    in_ci: bool = bool(os.environ.get('CI'))
+
+    if (
+        spawner in _CAPSYS_REQUIRED_SPAWNERS
+        and
+        capture == 'fd'
+    ):
+        msg: str = (
+            f'\n'
+            f'XXX `--spawn-backend={spawner}` REQUIRES '
+            f'`--capture=sys` XXX\n'
+            f'fork-child × `--capture=fd` is a known '
+            f'deadlock pattern.\n'
+            f'See `tractor._testing.pytest`\'s '
+            f'`pytest_load_initial_conftests` docstring '
+            f'for the full mechanism.\n'
+            f'\n'
+            f'Re-invoke with `--capture=sys` (or run '
+            f'with `pytest -s` for no capture).\n'
+        )
+        # fail-fast: CI must declare capture explicitly for
+        # fork-spawn backends.
+        if in_ci:
+            pytest.exit(
+                f'{msg}\n'
+                f'FAIL-FAST: CI=1 detected; aborting session.\n',
+                returncode=2,
+            )
+
+        # local: loud warn but let the run proceed so devs can
+        # experiment.
+        else:
+            warnings.warn(
+                f'{msg}\n'
+                f'Local mode (no `CI` env var) — '
+                f'continuing. Expect potential hangs.\n',
+                category=UserWarning,
+                stacklevel=1,
+            )
+            # ??TODO?? is there a way to force the `--capture=sys` sin CLI ??
+            # - [x] ask pytest peeps in chat!
+            # - [x] pytest` issue,
+            #       https://github.com/pytest-dev/pytest/issues/14444
+
+    # TODO, set various `$TRACTOR_X*` osenv vars here!
+    print(
+        f'Applying `tractor`-specific `pytest` config,\n'
+        f'{opts_w_args!r}\n'
+    )
+
 
 def tractor_test(
     wrapped: Callable|None = None,
@@ -112,11 +288,17 @@ def tractor_test(
     # injection (via `__wrapped__`) without leaking the async
     # nature.
     @wraps(wrapped)
-    def wrapper(**kwargs):
+    def wrapper(
+        set_fork_aware_capture: pytest.CaptureFixture|None = None,
+        # ^NOTE when set, the decorated fn declared as fixture-param.
+
+        **kwargs,
+    ):
         __tracebackhide__: bool = hide_tb
 
         # NOTE, ensure we inject any test-fn declared fixture
         # names.
+        sig = inspect.signature(wrapped)
         for kw in [
             'reg_addr',
             'loglevel',
@@ -125,9 +307,13 @@ def tractor_test(
             'tpt_proto',
             'timeout',
         ]:
-            if kw in inspect.signature(wrapped).parameters:
+            if kw in sig.parameters:
                 assert kw in kwargs
 
+        if 'set_fork_aware_capture' in sig.parameters:
+            assert set_fork_aware_capture
+            kwargs['set_fork_aware_capture'] = set_fork_aware_capture
+
         # Extract runtime settings as locals for
         # `open_root_actor()`; these must NOT leak into
         # `kwargs` when the test fn doesn't declare them
@@ -170,7 +356,6 @@ def tractor_test(
                     # invoke test-fn body IN THIS task
                     await wrapped(**kwargs)
 
-        # invoke runtime via a root task.
         return trio.run(
             partial(
                 _main,
@@ -184,13 +369,6 @@ def tractor_test(
 def pytest_addoption(
     parser: pytest.Parser,
 ):
-    # parser.addoption(
-    #     "--ll",
-    #     action="store",
-    #     dest='loglevel',
-    #     default='ERROR', help="logging level to set when testing"
-    # )
-
     parser.addoption(
         "--spawn-backend",
         action="store",
@@ -212,6 +390,21 @@ def pytest_addoption(
         ),
     )
 
+    parser.addoption(
+        "--enable-stackscope",
+        action="store_true",
+        dest='enable_stackscope',
+        default=False,
+        help=(
+            'Install `stackscope` SIGUSR1 handler in pytest + '
+            'every spawned subactor for live trio task-tree '
+            'dumps during hang investigations. Lighter than '
+            '`--tpdb` (no pdb machinery / tty-lock contention) '
+            '— use when you only need stack visibility. To '
+            'capture: `kill -USR1 <pytest-or-subactor-pid>`.'
+        ),
+    )
+
     # provide which IPC transport protocols opting-in test suites
     # should accumulatively run against.
     parser.addoption(
@@ -223,11 +416,66 @@ def pytest_addoption(
         help="Transport protocol to use under the `tractor.ipc.Channel`",
     )
 
+    # console loglevel for the test-session, scoped to the
+    # consuming-project's OWN pkg-hierarchy (see the
+    # `testing_pkg_name` fixture). For `tractor` itself this IS the
+    # runtime loglevel; downstream projects use `--ll` for their own
+    # ("internal") app-logging and `--tl` for tractor-as-runtime.
+    parser.addoption(
+        "--ll",
+        "--loglevel",
+        action="store",
+        dest='loglevel',
+        default=None,
+        help=(
+            "console loglevel to set for the test session, scoped to "
+            "the consuming-project pkg (see `testing_pkg_name`). "
+            "Falls through as the `--tl` default."
+        ),
+    )
 
-def pytest_configure(config):
+    # tractor-as-runtime loglevel, DISTINCT from `--ll` so downstream
-    backend = config.option.spawn_backend
+    # projects can split their app-logs from the `tractor.*` runtime
+    # hierarchy. Accepts a `tractor.log` "logging-spec" (see
+    # `tractor.log.apply_logspec()`).
+    parser.addoption(
+        "--tl",
+        "--tractor-loglevel",
+        action="store",
+        dest='tractor_loglevel',
+        default=None,
+        help=(
+            "loglevel (or logging-spec) for `tractor`-as-runtime, "
+            "distinct from `--ll`. Accepts a bare level (eg. "
+            "'info', 'cancel') or a sub-logger filter-spec, "
+            "'<sublog>:<level>,...' (eg. "
+            "'devx:runtime,trionics:cancel'). Falls back to `--ll` "
+            "when unset. Mirrors the logging-spec grammar consumed "
+            "by `tractor.log.apply_logspec()` (see its sub-pkg "
+            "granularity caveat)."
+        ),
+    )
+
+
+def pytest_configure(
+    config: pytest.Config,
+):
+    # opts: Namespace = config.option
+    # print(
+    #     f'PYTEST_CONFIGURE\n'
+    #     f'capture={opts.capture!r}\n'
+    # )
+    # breakpoint()
+
+    backend: str = config.option.spawn_backend
     from tractor.spawn._spawn import try_set_start_method
+    try:
         try_set_start_method(backend)
+    except RuntimeError as err:
+        # e.g. `--spawn-backend=subint` on Python < 3.14 — turn the
+        # runtime gate error into a clean pytest usage error so the
+        # suite exits with a helpful banner instead of a traceback.
+        raise pytest.UsageError(str(err)) from err
 
     # register custom marks to avoid warnings see,
     # https://docs.pytest.org/en/stable/how-to/writing_plugins.html#registering-custom-markers
@@ -235,10 +483,139 @@ def pytest_configure(config):
         'markers',
         'no_tpt(proto_key): test will (likely) not behave with tpt backend'
     )
+    config.addinivalue_line(
+        'markers',
+        'skipon_spawn_backend(*start_methods, reason=None): '
+        'skip this test under any of the given `--spawn-backend` '
+        'values; useful for backend-specific known-hang / -borked '
+        'cases (e.g. the `subint` GIL-starvation class documented '
+        'in `ai/conc-anal/subint_sigint_starvation_issue.md`).'
+    )
+
+    # `--enable-stackscope`: install SIGUSR1 → trio task-tree
+    # dump in pytest itself + propagate to every subactor via
+    # an env var that fork-children inherit and the runtime
+    # gate honors. Lighter than `--tpdb` (no pdb machinery) —
+    # purely for hang-investigation stack visibility.
+    if getattr(
+        config.option,
+        'enable_stackscope',
+        False
+    ):
+        # Env var inherited via fork → subactor's runtime
+        # picks it up at `Actor.async_main` startup. See the
+        # gate in `tractor.runtime._runtime` matching this
+        # var name.
+        os.environ['TRACTOR_ENABLE_STACKSCOPE'] = '1'
+
+        # Install in pytest itself so `kill -USR1 <pytest>`
+        # dumps the parent trio task-tree (which is where
+        # most Mode-A-class hangs park).
+        try:
+            from tractor.devx._stackscope import (
+                enable_stack_on_sig,
+            )
+            enable_stack_on_sig()
+        except ImportError:
+            warnings.warn(
+                '`stackscope` not installed — '
+                '--enable-stackscope is a no-op. '
+                'Install via the `devx` dep group.'
+            )
+    else:
+        os.environ.pop('TRACTOR_ENABLE_STACKSCOPE', None)
+
+
+def pytest_collection_modifyitems(
+    config: pytest.Config,
+    items: list[pytest.Function],
+):
+    '''
+    Expand any `@pytest.mark.skipon_spawn_backend('<backend>'[,
+    ...], reason='...')` markers into concrete
+    `pytest.mark.skip(reason=...)` calls for tests whose
+    backend-arg set contains the active `--spawn-backend`.
+
+    Uses `item.iter_markers(name=...)` which walks function +
+    class + module-level marks in the correct scope order (and
+    handles both the single-`MarkDecorator` and `list[Mark]`
+    forms of a module-level `pytestmark`) — so the same marker
+    works at any level a user puts it.
+
+    '''
+    backend: str = config.option.spawn_backend
+    default_reason: str = f'Borked on --spawn-backend={backend!r}'
+    for item in items:
+        for mark in item.iter_markers(name='skipon_spawn_backend'):
+            skip_backends: tuple[str] = mark.args
+            for skip_backend in skip_backends:
+                assert (
+                    skip_backend in get_args(SpawnMethodKey)
+                    or
+                    skip_backend in _IN_DEV_SPAWN_BACKENDS
+                )
+            # ?TODO, run these through the try-set-backend checker to
+            # avoid typos?
+            if backend in skip_backends:
+                reason: str = mark.kwargs.get(
+                    'reason',
+                    default_reason,
+                )
+                item.add_marker(pytest.mark.skip(reason=reason))
+                # first matching mark wins; no value in stacking
+                # multiple `skip`s on the same item.
+                break
+
+
+@pytest.fixture(
+    scope="session",
+    autouse=True,
+)
+def alert_on_finish():
+    '''
+    Ring a terminal notification on full test session
+    completion to alert any would be human.
+
+    '''
+    # TODO, check attached to tty or skip!
+    yield  # run all tests
+    print("\a")  # trigger terminal bell
+    # ?TODO, any other nice-tricks/specific tuis we could try?
+    # - supposedly works in many terminals:
+    #   >> print("\033]5;Alert: Tests Finished\a")
+    # - sway/i3-nag?
+
+
+@pytest.fixture(autouse=True)
+def _reset_runtime_vars():
+    '''
+    Per-test isolation of the process-global
+    `tractor.runtime._state._runtime_vars`.
+
+    `open_root_actor()` writes `_enable_tpts` (and other runtime
+    vars) into this module-global dict, but nothing resets it on
+    actor teardown. Under the in-process `pytest` launchpad a
+    uds-using test therefore leaks `_enable_tpts=['uds']` into a
+    sibling tcp test, which then trips the
+    `registry_addrs`×`enable_transports` proto-guard in
+    `open_root_actor()` with a `ValueError`. Snapshot + restore
+    around every test so no runtime-var state crosses a test
+    boundary.
+
+    '''
+    from tractor.runtime import _state
+    snapshot: dict = dict(_state._runtime_vars)
+    try:
+        yield
+    finally:
+        _state._runtime_vars.clear()
+        _state._runtime_vars.update(snapshot)
 
 
 @pytest.fixture(scope='session')
-def debug_mode(request) -> bool:
+def debug_mode(
+    request: pytest.FixtureRequest,
+) -> bool:
     '''
     Flag state for whether `--tpdb` (for `tractor`-py-debugger)
     was passed to the test run.
@@ -252,12 +629,145 @@ def debug_mode(request) -> bool:
 
 
 @pytest.fixture(scope='session')
-def spawn_backend(request) -> str:
+def testing_pkg_name() -> str:
+    '''
+    Root pkg-name of the project consuming this plugin, used to
+    scope `--ll` "internal"/app-level console logging into that
+    project's OWN `tractor.log.get_logger(pkg_name=<.>)` hierarchy
+    — distinct from the `tractor.*` runtime hierarchy configured
+    via `--tl`.
+
+    Defaults to `'tractor'` (so tractor's own suite treats `--ll`
+    as the runtime level). Downstream projects override this from
+    their `conftest.py`, eg.
+
+    .. code:: python
+
+        @pytest.fixture(scope='session')
+        def testing_pkg_name() -> str:
+            return 'modden'
+
+    '''
+    return 'tractor'
+
+
+@pytest.fixture(
+    scope='session',
+    autouse=True,
+)
+def loglevel(
+    request: pytest.FixtureRequest,
+    testing_pkg_name: str,
+) -> str|None:
+    '''
+    Resolve + apply the test-session console loglevels and yield
+    the `tractor`-runtime level (also passed to
+    `open_root_actor(loglevel=<.>)` by `@tractor_test`).
+
+    - `--tl <logspec>`: tractor-runtime level (falls back to the
+      generic `--ll`); applied to the `tractor.*` logger hierarchy
+      and `tractor.log._default_loglevel` via
+      `tractor.log.apply_logspec()`.
+    - `--ll <level>`: the consuming-project's OWN console loglevel,
+      applied to its `testing_pkg_name` hierarchy when that isn't
+      `tractor` itself.
+
+    '''
+    import tractor
+    orig: str = tractor.log._default_loglevel
+
+    ll: str|None = request.config.option.loglevel
+    tl: str|None = request.config.option.tractor_loglevel
+
+    # tractor-runtime loglevel: explicit `--tl` wins, else fall
+    # back to the generic `--ll`, else leave the lib default.
+    logspec: str|None = tl if tl is not None else ll
+    tractor_level: str|None = None
+    if logspec is not None:
+        tractor_level, _ = tractor.log.apply_logspec(
+            logspec,
+            default_level=ll,
+            pkg_name='tractor',
+        )
+        if tractor_level is not None:
+            tractor.log._default_loglevel = tractor_level
+
+    # consuming-project ("internal") console logging at the generic
+    # `--ll` level, scoped to ITS OWN pkg-hierarchy (NOT `tractor.*`)
+    # so downstream projects can split app-logs from runtime-logs.
+    if (
+        ll is not None
+        and
+        testing_pkg_name
+        and
+        testing_pkg_name != 'tractor'
+    ):
+        tractor.log.get_console_log(
+            level=ll,
+            pkg_name=testing_pkg_name,
+            name=testing_pkg_name,
+        )
+
+    log = tractor.log.get_console_log(
+        level=tractor_level,
+        name='tractor',  # <- enable root logger
+    )
+    log.info(
+        f'Test-harness set session loglevels:\n'
+        f'tractor-runtime (`--tl`/`--ll`): {tractor_level!r}\n'
+        f'{testing_pkg_name!r} (`--ll`): {ll!r}\n'
+    )
+    yield tractor_level
+    tractor.log._default_loglevel = orig
+
+
+@pytest.fixture(scope='function')
+def test_log(
+    request: pytest.FixtureRequest,
+    loglevel: str,
+    testing_pkg_name: str,
+) -> tractor.log.StackLevelAdapter:
+    '''
+    Deliver a per test-module-fn logger instance for reporting from
+    within actual test bodies/fixtures.
+
+    For example this can be handy to report certain error cases from
+    exception handlers using `test_log.exception()`.
+
+    The logger is scoped to the consuming-project's
+    `testing_pkg_name` hierarchy so downstream suites' in-test logs
+    land under their own pkg, not `tractor.*`.
+
+    '''
+    modname: str = request.function.__module__
+    log = tractor.log.get_logger(
+        name=modname,
+        pkg_name=testing_pkg_name,
+    )
+    _log = tractor.log.get_console_log(
+        level=loglevel,
+        logger=log,
+        name=modname,
+    )
+    _log.debug(
+        f'In-test-logging requested\n'
+        f'test_log.name: {log.name!r}\n'
+        f'level: {loglevel!r}\n'
+    )
+    yield _log
+
+
+@pytest.fixture(scope='session')
+def spawn_backend(
+    request: pytest.FixtureRequest,
+) -> str:
     return request.config.option.spawn_backend
 
 
 @pytest.fixture(scope='session')
-def tpt_protos(request) -> list[str]:
+def tpt_protos(
+    request: pytest.FixtureRequest,
+) -> list[str]:
 
     # allow quoting on CLI
     proto_keys: list[str] = [
@@ -285,7 +795,7 @@ def tpt_protos(request) -> list[str]:
     autouse=True,
 )
 def tpt_proto(
-    request,
+    request: pytest.FixtureRequest,
     tpt_protos: list[str],
 ) -> str:
     proto_key: str = tpt_protos[0]
@@ -337,7 +847,6 @@ def pytest_generate_tests(
     metafunc: pytest.Metafunc,
 ):
     spawn_backend: str = metafunc.config.option.spawn_backend
-
     if not spawn_backend:
         # XXX some weird windows bug with `pytest`?
         spawn_backend = 'trio'
@@ -345,7 +854,6 @@ def pytest_generate_tests(
     # drive the valid-backend set from the canonical `Literal` so
     # adding a new spawn backend (e.g. `'subint'`) doesn't require
     # touching the harness.
-    from tractor.spawn._spawn import SpawnMethodKey
     assert spawn_backend in get_args(SpawnMethodKey)
 
     # NOTE: used-to-be-used-to dyanmically parametrize tests for when
@@ -356,7 +864,8 @@ def pytest_generate_tests(
         metafunc.parametrize(
             "start_method",
             [spawn_backend],
-            scope='module',
+            scope='session',
+            ids=lambda item: f'start_method={spawn_backend}',
         )
 
     # TODO, parametrize any `tpt_proto: str` declaring tests!
@@ -367,3 +876,136 @@ def pytest_generate_tests(
     #         proto_tpts,  # TODO, double check this list usage!
     #         scope='module',
     #     )
+
+
+def _is_forking_spawner(
+    start_method: str,
+) -> bool:
+    return start_method in [
+        'main_thread_forkserver',
+        'mp_forkserver',
+    ]
+
+
+@pytest.fixture(scope='session')
+def is_forking_spawner(
+    start_method: str,
+) -> bool:
+    '''
+    Is the `pytest` run using a `fork()`ing process spawning-backend?
+
+    '''
+    return _is_forking_spawner(start_method)
+
+
+def maybe_xfail_for_spawner(
+    request: pytest.FixtureRequest,
+    start_method: str,
+    is_forking_spawner: bool,
+) -> None:
+    '''
+    Fork based spawning backends cause issues with
+    `pytest`'s fd-capture mechanism and can cause various
+    suites to hang.
+
+    This helper allows skipping/xfailing from a test when
+    a fork-spawn backend is being used WITHOUT
+    `--capture=sys`.
+
+    '''
+    capture_mode: str = request.config.option.capture
+    # `tee-sys` is also sys-level capture (just additionally writes
+    # to the original `sys.__stdout__/__stderr__`); fork-safe like
+    # `sys`. Only `fd`-level capture is the deadlock pattern.
+    if (
+        capture_mode not in (
+            'sys',
+            'tee-sys',
+        )
+        and
+        is_forking_spawner
+    ):
+        pytest.skip(
+            f'Spawner {start_method!r} requires the flag,\n'
+            f'--capture=sys or --capture=tee-sys..\n'
+            f'(got --capture={capture_mode!r})\n'
+        )
+
+
+def maybe_override_capture(
+    request: pytest.FixtureRequest,
+    start_method: bool,
+) -> str:
+    if _is_forking_spawner(start_method):
+        request.getfixturevalue('capsys')
+        return 'sys'
+
+    return request.config.option.capture
+
+
+@pytest.fixture
+def set_fork_aware_capture(
+    request: pytest.FixtureRequest,
+    start_method: str,
+) -> pytest.CaptureFixture|str:
+    '''
+    Force `--capture=sys` method for tests using
+    a forking-spawner backend due to fd-copying issues
+    which can oddly make certain tests hang/fail.
+
+    '''
+    # Fast-path: user already passed sys-level capture
+    # (`sys` or `tee-sys`) at the CLI — no override needed.
+    if request.config.option.capture in (
+            'sys',
+            'tee-sys',
+    ):
+        return request.config.option.capture
+
+    capsys: pytest.CaptureFixture = maybe_override_capture(
+        request=request,
+        start_method=start_method,
+    )
+    return capsys
+    # XXX reset?
+    # with capsys.disabled():
+    #     pass
+    # return partial(
+    #     maybe_override_capture,
+    #     request=request,
+    #     start_method=start_method,
+    # )
+
+
+def pytest_terminal_summary(
+    terminalreporter,
+    exitstatus: int,
+    config: pytest.Config,
+) -> None:
+    '''
+    End-of-session summary: list all
+    `fail_after_w_trace`/`afk_alarm_w_trace` snapshot dirs
+    captured during the run so the human doesn't have to scroll
+    back through captured-stderr lines to find dump paths.
+
+    Reads from `tractor._testing.trace._SNAPSHOT_INDEX` which is
+    populated by `_do_capture_snapshot()` on each successful
+    snapshot capture.
+
+    No-op when zero snapshots were captured (most sessions).
+
+    '''
+    from .trace import _SNAPSHOT_INDEX
+
+    if not _SNAPSHOT_INDEX:
+        return
+
+    tr = terminalreporter
+    tr.write_sep('=', 'tractor hang-snapshot index')
+    tr.write_line(
+        f'{len(_SNAPSHOT_INDEX)} `fail_after_w_trace` / '
+        f'`afk_alarm_w_trace` snapshot(s) captured this session:'
+    )
+    for label, path in _SNAPSHOT_INDEX:
+        tr.write_line(f'  {label}')
+        tr.write_line(f'    → {path}')

tractor/devx/__init__.py

@@ -41,6 +41,11 @@ from .pformat import (
     pformat_caller_frame as pformat_caller_frame,
     pformat_boxed_tb as pformat_boxed_tb,
 )
+from ._debug_hangs import (
+    dump_on_hang as dump_on_hang,
+    track_resource_deltas as track_resource_deltas,
+    resource_delta_fixture as resource_delta_fixture,
+)
 
 
 # TODO, move this to a new `.devx._pdbp` mod?

tractor/devx/_debug_hangs.py

@@ -0,0 +1,227 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Hang-diagnostic helpers for concurrent / multi-interpreter code.
+
+Collected from the `subint` spawn backend bringup (issue #379)
+where silent test-suite hangs needed careful teardown
+instrumentation to diagnose. This module bottles up the
+techniques that actually worked so future hangs are faster
+to corner.
+
+Two primitives:
+
+1. `dump_on_hang()` — context manager wrapping
+   `faulthandler.dump_traceback_later()` with the critical
+   gotcha baked in: write the dump to a **file**, not
+   `sys.stderr`. Under `pytest` (and any other output
+   capturer) stderr gets swallowed and the dump is easy to
+   miss — burning hours convinced you're looking at the wrong
+   thing.
+
+2. `track_resource_deltas()` — context manager (+ optional
+   autouse-fixture factory) logging per-block deltas of
+   `threading.active_count()` and — if running on py3.13+ —
+   `len(_interpreters.list_all())`. Lets you quickly rule out
+   leak-accumulation theories when a suite hangs more
+   frequently as it progresses (if counts don't grow, it's
+   not a leak; look for a race on shared cleanup instead).
+
+See issue #379 / commit `26fb820` for the worked example.
+
+'''
+from __future__ import annotations
+import faulthandler
+import sys
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+from typing import (
+    Callable,
+    Iterator,
+)
+
+try:
+    import _interpreters  # type: ignore
+except ImportError:
+    _interpreters = None  # type: ignore
+
+
+__all__ = [
+    'dump_on_hang',
+    'track_resource_deltas',
+    'resource_delta_fixture',
+]
+
+
+@contextmanager
+def dump_on_hang(
+    seconds: float = 30.0,
+    *,
+    path: str | Path = '/tmp/tractor_hang.dump',
+    all_threads: bool = True,
+
+) -> Iterator[str]:
+    '''
+    Arm `faulthandler` to dump all-thread tracebacks to
+    `path` after `seconds` if the with-block hasn't exited.
+
+    *Writes to a file, not stderr* — `pytest`'s stderr
+    capture silently eats stderr-destined `faulthandler`
+    output, and the same happens under any framework that
+    redirects file-descriptors. Pointing the dump at a real
+    file sidesteps that.
+
+    Yields the resolved file path so it's easy to read back.
+
+    Example
+    -------
+    ::
+
+        from tractor.devx import dump_on_hang
+
+        def test_hang():
+            with dump_on_hang(
+                seconds=15,
+                path='/tmp/my_test_hang.dump',
+            ) as dump_path:
+                trio.run(main)
+            # if it hangs, inspect dump_path afterward
+
+    '''
+    dump_path = Path(path)
+    f = dump_path.open('w')
+    try:
+        faulthandler.dump_traceback_later(
+            seconds,
+            repeat=False,
+            file=f,
+            exit=False,
+        )
+        try:
+            yield str(dump_path)
+        finally:
+            faulthandler.cancel_dump_traceback_later()
+    finally:
+        f.close()
+
+
+def _snapshot() -> tuple[int, int]:
+    '''
+    Return `(thread_count, subint_count)`.
+
+    Subint count reported as `0` on pythons lacking the
+    private `_interpreters` stdlib module (i.e. py<3.13).
+
+    '''
+    threads: int = threading.active_count()
+    subints: int = (
+        len(_interpreters.list_all())
+        if _interpreters is not None
+        else 0
+    )
+    return threads, subints
+
+
+@contextmanager
+def track_resource_deltas(
+    label: str = '',
+    *,
+    writer: Callable[[str], None] | None = None,
+
+) -> Iterator[tuple[int, int]]:
+    '''
+    Log `(threads, subints)` deltas across the with-block.
+
+    `writer` defaults to `sys.stderr.write` (+ trailing
+    newline); pass a custom callable to route elsewhere
+    (e.g., a log handler or an append-to-file).
+
+    Yields the pre-entry snapshot so callers can assert
+    against the expected counts if they want.
+
+    Example
+    -------
+    ::
+
+        from tractor.devx import track_resource_deltas
+
+        async def test_foo():
+            with track_resource_deltas(label='test_foo'):
+                async with tractor.open_nursery() as an:
+                    ...
+
+        # Output:
+        #   test_foo: threads 2->2, subints 1->1
+
+    '''
+    before = _snapshot()
+    try:
+        yield before
+    finally:
+        after = _snapshot()
+        msg: str = (
+            f'{label}: '
+            f'threads {before[0]}->{after[0]}, '
+            f'subints {before[1]}->{after[1]}'
+        )
+        if writer is None:
+            sys.stderr.write(msg + '\n')
+            sys.stderr.flush()
+        else:
+            writer(msg)
+
+
+def resource_delta_fixture(
+    *,
+    autouse: bool = True,
+    writer: Callable[[str], None] | None = None,
+
+) -> Callable:
+    '''
+    Factory returning a `pytest` fixture that wraps each test
+    in `track_resource_deltas(label=<node.name>)`.
+
+    Usage in a `conftest.py`::
+
+        # tests/conftest.py
+        from tractor.devx import resource_delta_fixture
+
+        track_resources = resource_delta_fixture()
+
+    or opt-in per-test::
+
+        track_resources = resource_delta_fixture(autouse=False)
+
+        def test_foo(track_resources):
+            ...
+
+    Kept as a factory (not a bare fixture) so callers control
+    `autouse` / `writer` without having to subclass or patch.
+
+    '''
+    import pytest  # deferred: only needed when caller opts in
+
+    @pytest.fixture(autouse=autouse)
+    def _track_resources(request):
+        with track_resource_deltas(
+            label=request.node.name,
+            writer=writer,
+        ):
+            yield
+
+    return _track_resources

tractor/devx/_proctitle.py

@@ -0,0 +1,74 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Per-actor proc-title via `py-setproctitle`.
+
+Sets a stable, OS-level identifier for each `tractor` actor
+process so diag tools (`ps`, `top`, `htop`, `psutil`) and our
+own `acli.pytree`/`acli.hung_dump` can show "which actor is
+which" at a glance without needing to read full
+`/proc/<pid>/cmdline`.
+
+Format:
+  ``tractor[<aid.reprol()>]``    e.g. ``tractor[doggy@1027301b]``
+
+Uses the canonical `Aid.reprol()` form
+(``<name>@<uuid_short>``) so the proc-title matches the
+identifier shape used in tractor's logs, the `TRACTOR_AID`
+env-var, and orphan-reaper scans — one identity across
+all surfaces.
+
+Optional dep: silently no-op when `setproctitle` is missing.
+
+'''
+from __future__ import annotations
+from typing import TYPE_CHECKING
+
+
+if TYPE_CHECKING:
+    from tractor.runtime._runtime import Actor
+
+
+# `setproctitle` is an optional dep — tractor's runtime path
+# treats this as best-effort diag, so missing import is a
+# no-op rather than a hard error.
+try:
+    import setproctitle as _stp
+except ImportError:
+    _stp = None
+
+
+def set_actor_proctitle(actor: 'Actor') -> str | None:
+    '''
+    Set the calling process's proc-title to identify it as a
+    tractor sub-actor.
+
+    Returns the title string set, or `None` if `setproctitle`
+    isn't available.
+
+    Should be called early in the actor's process lifetime
+    (after `Actor` construction, before `_trio_main`) so the
+    new title is visible to OS-level tooling for the entire
+    runtime.
+
+    '''
+    if _stp is None:
+        return None
+
+    title: str = f'tractor[{actor.aid.reprol()}]'
+    _stp.setproctitle(title)
+    return title

tractor/devx/_stackscope.py

@@ -24,7 +24,7 @@ disjoint, parallel executing tasks in separate actors.
 
 '''
 from __future__ import annotations
-# from functools import partial
+from functools import partial
 from threading import (
     current_thread,
     Thread,
@@ -47,7 +47,9 @@ from typing import (
 import trio
 from tractor.runtime import _state
 from tractor import log as logmod
-from tractor.devx import debug
+from tractor.devx import (
+    debug,
+)
 
 log = logmod.get_logger()
 
@@ -61,12 +63,29 @@ if TYPE_CHECKING:
 
 
 @trio.lowlevel.disable_ki_protection
-def dump_task_tree() -> None:
+def dump_task_tree(
+    write_file: bool = False,
+    write_tty: bool = False,
+) -> None:
     '''
     Do a classic `stackscope.extract()` task-tree dump to console at
     `.devx()` level.
 
+    When `write_file`/`write_tty` are set, ALSO tee the rendered
+    tree to capture-bypassing sinks so SIGUSR1 dumps remain
+    visible when the parent process has captured stdio (e.g.
+    pytest's default `--capture=fd`); the SIGUSR1 handler passes
+    `write_file=True` for exactly this reason:
+
+    - `write_file` -> `/tmp/tractor-stackscope-<pid>.log`
+      (append-mode) — guaranteed-readable artifact even under CI
+      / `nohup` / no-tty conditions. `tail -f` to follow.
+    - `write_tty` -> `/dev/tty` if a controlling terminal is
+      attached — best-effort, ignored if the device is missing
+      or write fails. pytest never captures the tty.
+
     '''
+    import os
     import stackscope
     tree_str: str = str(
         stackscope.extract(
@@ -96,46 +115,158 @@ def dump_task_tree() -> None:
     # |_{Supervisor/Scope
     # |_[Storage/Memory/IPC-Stream/Data-Struct
 
-    log.devx(
+    fpath: str = f'/tmp/tractor-stackscope-{os.getpid()}.log'
+    from . import pformat
+    actor_repr: str = pformat.nest_from_op(
+        input_op='|_',
+        text=f'{actor}',
+        nest_prefix='|_',
+        nest_indent=3,
+    )
+    full_dump: str = (
         f'Dumping `stackscope` tree for actor\n'
-        f'(>: {actor.uid!r}\n'
+        f'(>: {actor.aid.uid!r}\n'
         f' |_{mp.current_process()}\n'
         f'   |_{thr}\n'
-        f'     |_{actor}\n'
+        # TODO, use the nest_from_op
+        f'{actor_repr}'
+        # f'     |_{actor}'
         f'\n'
         f'{sigint_handler_report}\n'
         f'signal.getsignal(SIGINT) -> {current_sigint_handler!r}\n'
-        # f'\n'
-        # start-of-trace-tree delimiter (mostly for testing)
-        # f'------ {actor.uid!r} ------\n'
         f'\n'
-        f'------ start-of-{actor.uid!r} ------\n'
+        f'capture-bypass tee: {fpath}\n'
+        f'(`tail -f {fpath}` to follow across signals)\n'
+        f'\n'
+        f'------ start-of-{actor.aid.uid!r} ------\n'
         f'|\n'
         f'{tree_str}'
-        # end-of-trace-tree delimiter (mostly for testing)
         f'|\n'
-        f'|_____ end-of-{actor.uid!r} ______\n'
+        f'|_____ end-of-{actor.aid.uid!r} ______\n'
     )
-    # TODO: can remove this right?
+    log.devx(full_dump)
-    # -[ ] was original code from author
+
-    #
+    # NOTE, capture-bypass sinks. Pytest's default
-    # print(
+    # `--capture=fd` swallows `log.devx()` above; the
-    #     'DUMPING FROM PRINT\n'
+    # following two writes guarantee the dump reaches the
-    #     +
+    # human even when stdio is captured.
-    #     content
+    if write_file:
-    # )
+        try:
-    # import logging
+            with open(fpath, 'a') as f:
-    # try:
+                f.write(full_dump + '\n')
-    #     with open("/dev/tty", "w") as tty:
+        except OSError:
-    #         tty.write(tree_str)
+            log.exception(
-    # except BaseException:
+                f'Failed to tee stackscope dump to {fpath!r}'
-    #     logging.getLogger(
+            )
-    #         "task_tree"
+
-    #     ).exception("Error printing task tree")
+    if write_tty:
+        try:
+            with open('/dev/tty', 'w') as tty:
+                tty.write(full_dump + '\n')
+        except OSError:
+            # no controlling tty (CI / nohup / detached) —
+            # silently fall through; the file sink covers it.
+            pass
 
 _handler_lock = RLock()
 _tree_dumped: bool = False
 
+# Captured at `enable_stack_on_sig()` time when running
+# inside a trio task. `dump_tree_on_sig` uses this to
+# schedule `dump_task_tree()` ON the trio loop via
+# `token.run_sync_soon` so stackscope sees a real current
+# task and can recurse into nursery children. Without
+# it (signal handler running in a non-trio stack frame),
+# `stackscope.extract` only walks the `<init>` task and
+# misses everything inside `async_main`'s nurseries.
+_trio_token: trio.lowlevel.TrioToken|None = None
+
+
+def _relay_sig_to_subactors(sig: int) -> None:
+    '''
+    Forward `sig` to every live sub-actor's underlying
+    process so each runs its own `dump_tree_on_sig`
+    handler.
+
+    Factored out of `dump_tree_on_sig` so the
+    `run_sync_soon`-deferred path can call it AFTER
+    the parent's `dump_task_tree()` completes — see
+    `_dump_then_relay` below for why ordering matters.
+
+    '''
+    an: ActorNursery
+    for an in _state.current_actor()._actoruid2nursery.values():
+        subproc: ProcessType
+        subactor: Actor
+        for (
+            subactor,
+            subproc,
+            _,
+        ) in an._children.values():
+            log.warning(
+                f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
+                f'{subactor}\n'
+                f' |_{subproc}\n'
+            )
+            # bc of course stdlib can't have a std API.. XD
+            match subproc:
+                case trio.Process():
+                    subproc.send_signal(sig)
+
+                case mp.Process():
+                    subproc._send_signal(sig)
+
+
+def _dump_then_relay(
+    sig: int|None,
+) -> None:
+    '''
+    `run_sync_soon`-friendly callback: dump THIS actor's
+    task tree first, THEN relay `sig` to subactors so
+    their dumps can't race ahead of ours.
+
+    Hierarchical-ordering preservation: the legacy
+    direct-call path (pre-`run_sync_soon`) ran the dump
+    synchronously inside the signal handler, then
+    relayed — guaranteeing parent-output-before-child
+    in the multiplexed pty stream. The pure-deferred
+    path (schedule dump only, relay sync from handler)
+    inverts that: relay fires while the parent's
+    dump is still queued, subs receive SIGUSR1 and
+    schedule their own dumps, all dumps then race in
+    arbitrary order through stdio.
+
+    Co-scheduling fixes that: by chaining relay AFTER
+    `dump_task_tree()` inside the same trio-loop
+    callback, parent output flushes before any sub
+    receives the signal, restoring the
+    parent → relay-log → sub-dump ordering humans
+    expect when reading hang-investigation traces.
+
+    Trio prints + crashes on uncaught exceptions in
+    scheduled callbacks; we swallow + log so the test
+    keeps running and the user can re-trigger.
+
+    '''
+    try:
+        dump_task_tree(write_file=True)
+    except BaseException:
+        log.exception(
+            '`dump_task_tree()` raised (scheduled via '
+            '`run_sync_soon`); continuing.\n'
+        )
+
+    if sig is None:
+        return
+
+    try:
+        _relay_sig_to_subactors(sig)
+    except BaseException:
+        log.exception(
+            f'`_relay_sig_to_subactors({sig})` raised '
+            f'(scheduled via `run_sync_soon`); continuing.\n'
+        )
+
 
 def dump_tree_on_sig(
     sig: int,
@@ -159,16 +290,32 @@ def dump_tree_on_sig(
             'Trying to dump `stackscope` tree..\n'
         )
         try:
-            dump_task_tree()
+            # Prefer scheduling on the trio loop — runs the
-            # await actor._service_n.start_soon(
+            # dump from a real trio-task context so
-            #     partial(
+            # `stackscope.extract(recurse_child_tasks=True)`
-            #         trio.to_thread.run_sync,
+            # walks every nursery child instead of seeing
-            #         dump_task_tree,
+            # only the `<init>` task. Falls back to a direct
-            #     )
+            # call when no token was captured (e.g. signal
-            # )
+            # delivered outside a trio.run).
-            # trio.lowlevel.current_trio_token().run_sync_soon(
+            #
-            #     dump_task_tree
+            # Co-schedule the relay-to-subs in the SAME
-            # )
+            # callback so parent's dump prints BEFORE any
+            # sub receives SIGUSR1 — see `_dump_then_relay`
+            # for the full hierarchical-ordering rationale.
+            if _trio_token is not None:
+                _trio_token.run_sync_soon(
+                    partial(
+                        _dump_then_relay,
+                        sig=sig if relay_to_subs else None,
+                    )
+                )
+                # NOTE, `_dump_then_relay` handles the relay
+                # internally; bail out before the
+                # direct-path relay below.
+                return
+
+            else:
+                dump_task_tree(write_file=True)
 
         except RuntimeError:
             log.exception(
@@ -188,27 +335,15 @@ def dump_tree_on_sig(
         #     'Supposedly we dumped just fine..?'
         # )
 
+    # Direct-path relay (only reached when `_trio_token`
+    # was None — the run_sync_soon path returned above
+    # to let `_dump_then_relay` handle the relay
+    # in-callback).
     if not relay_to_subs:
+        log.devx(f'Skipping {sig!r} relay to subactors..')
         return
 
-    an: ActorNursery
+    _relay_sig_to_subactors(sig)
-    for an in _state.current_actor()._actoruid2nursery.values():
-        subproc: ProcessType
-        subactor: Actor
-        for subactor, subproc, _ in an._children.values():
-            log.warning(
-                f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
-                f'{subactor}\n'
-                f' |_{subproc}\n'
-            )
-
-            # bc of course stdlib can't have a std API.. XD
-            match subproc:
-                case trio.Process():
-                    subproc.send_signal(sig)
-
-                case mp.Process():
-                    subproc._send_signal(sig)
 
 
 def enable_stack_on_sig(
@@ -233,19 +368,50 @@ def enable_stack_on_sig(
 
     '''
     try:
+        # NOTE, `stackscope._glue` does intentional async-gen type
+        # introspection at import-time which trips
+        # `RuntimeWarning: coroutine method 'asend'/'athrow' was
+        # never awaited`. Benign — they only want the wrapper
+        # type — but visible to users. Squelch the import-only
+        # warning so SIGUSR1 setup stays quiet.
+        import warnings
+        with warnings.catch_warnings():
+            warnings.filterwarnings(
+                'ignore',
+                category=RuntimeWarning,
+                message=r"coroutine method '(asend|athrow)' .* was never awaited",
+            )
             import stackscope
+            _state._runtime_vars['use_stackscope'] = True
     except ImportError:
         log.warning(
             'The `stackscope` lib is not installed!\n'
             '`Ignoring enable_stack_on_sig() call!\n'
         )
+        assert not _state._runtime_vars['use_stackscope']
         return None
 
+    # Capture the trio token if we're inside `trio.run`
+    # so SIGUSR1 dispatches the dump *onto* the trio loop
+    # (full task-tree visibility). When called outside trio
+    # (e.g. from `pytest_configure`), token capture fails
+    # silently and `dump_tree_on_sig` falls back to the
+    # direct-call path.
+    global _trio_token
+    try:
+        _trio_token = trio.lowlevel.current_trio_token()
+    except RuntimeError:
+        # not in a `trio.run` — leave None; runtime can
+        # re-call `enable_stack_on_sig()` later from
+        # inside `async_main` to capture it.
+        _trio_token = None
+
     handler: Callable|int = getsignal(sig)
     if handler is dump_tree_on_sig:
         log.devx(
             'A `SIGUSR1` handler already exists?\n'
             f'|_ {handler!r}\n'
+            f'(trio_token captured: {_trio_token is not None})\n'
         )
         return
 
@@ -259,5 +425,6 @@ def enable_stack_on_sig(
         f'{stackscope!r}\n\n'
         f'With `SIGUSR1` handler\n'
         f'|_{dump_tree_on_sig}\n'
+        f'(trio_token captured: {_trio_token is not None})\n'
     )
     return stackscope

tractor/devx/debug/_tty_lock.py

@@ -181,7 +181,7 @@ class Lock:
         return (
             f'<{cls.__name__}(\n'
             f'{body}'
-            ')>\n\n'
+            ')>\n'
         )
 
     @classmethod
@@ -282,7 +282,7 @@ class Lock:
             ):
                 message += (
                     '-> No new task holds the TTY lock!\n\n'
-                    f'{Lock.repr()}\n'
+                    f'{Lock.repr()}'
                 )
 
             elif (

tractor/ipc/_linux.py

@@ -17,10 +17,20 @@
 Linux specifics, for now we are only exposing EventFD
 
 '''
-import os
 import errno
+import os
+import sys
+
+try:
+    import cffi
+except ImportError as ie:
+    if sys.version_info >= (3, 14):
+        ie.add_note(
+            f'The `cffi` pkg has no 3.14 support yet.\n'
+        )
+
+    raise ie
 
-import cffi
 import trio
 
 ffi = cffi.FFI()

tractor/ipc/_mp_bs.py

@@ -17,7 +17,7 @@
 Utils to tame mp non-SC madeness
 
 '''
-import platform
+from functools import partial
 
 
 def disable_mantracker():
@@ -27,30 +27,13 @@ def disable_mantracker():
 
     '''
     from multiprocessing.shared_memory import SharedMemory
-
-
-    # 3.13+ only.. can pass `track=False` to disable
-    # all the resource tracker bs.
-    # https://docs.python.org/3/library/multiprocessing.shared_memory.html
-    if (_py_313 := (
-            platform.python_version_tuple()[:-1]
-            >=
-            ('3', '13')
-        )
-    ):
-        from functools import partial
-        return partial(
-            SharedMemory,
-            track=False,
-        )
-
-    # !TODO, once we drop 3.12- we can obvi remove all this!
-    else:
     from multiprocessing import (
         resource_tracker as mantracker,
     )
 
-        # Tell the "resource tracker" thing to fuck off.
+    # XXX ALWAYS disable the stdlib's "resource tracker"; it prevents
+    # fork backends and never was useful to us since we're SC
+    # lifetime managing all allocations.
     class ManTracker(mantracker.ResourceTracker):
         def register(self, name, rtype):
             pass
@@ -69,7 +52,12 @@ def disable_mantracker():
     mantracker.unregister = mantracker._resource_tracker.unregister
     mantracker.getfd = mantracker._resource_tracker.getfd
 
-        # use std type verbatim
+    # 3.13+ only.. can pass `track=False` to disable
-        shmT = SharedMemory
+    # all the resource tracker bs.
+    # https://docs.python.org/3/library/multiprocessing.shared_memory.html
+    shmT = partial(
+        SharedMemory,
+        track=False,
+    )
 
     return shmT

tractor/ipc/_server.py

@@ -1122,20 +1122,32 @@ async def _serve_ipc_eps(
             )
 
     finally:
+        # close every endpoint INDEPENDENTLY: a close raising
+        # mid-iter (e.g. UDS `os.unlink` racing concurrent reap) must
+        # not strand the rest of the eps + must not skip the
+        # `_shutdown.set()` below.
         if eps:
             addr: Address
             ep: Endpoint
-            for addr, ep in server.epsdict().items():
+            for addr, ep in list(server.epsdict().items()):
+                try:
                     ep.close_listener()
+                except Exception as ep_close_err:
+                    log.exception(
+                        f'Endpoint close raised, continuing teardown\n'
+                        f'  |_{ep!r}\n'
+                        f'  |_{ep_close_err!r}\n'
+                    )
+                finally:
+                    try:
                         server._endpoints.remove(ep)
+                    except ValueError:
+                        pass
 
-        # actor = _state.current_actor()
+        # always signal "shutdown" so `actor.cancel()` →
-        # if actor.is_arbiter:
+        # `ipc_server.wait_for_shutdown()` doesn't deadlock when an
-        #     import pdbp; pdbp.set_trace()
+        # endpoint close raised above.
-
+        if server._shutdown is not None:
-        # signal the server is "shutdown"/"terminated"
-        # since no more active endpoints are active.
-        if not server._endpoints:
             server._shutdown.set()
 
 @acm

tractor/ipc/_shm.py

@@ -929,15 +929,26 @@ def open_shm_list(
     # "close" attached shm on actor teardown
     try:
         actor = tractor.current_actor()
-
         actor.lifetime_stack.callback(shml.shm.close)
 
-        # XXX on 3.13+ we don't need to call this?
+        # >XXX NOTE< on 3.13+ we need to call this AS WELL AS pass
-        # -> bc we pass `track=False` for `SharedMemeory` orr?
+        # `track=False` for `mp.SharedMemeory` otherwise fork based
-        if (
+        # backends will error out due to long lived stdlib
-            platform.python_version_tuple()[:-1] < ('3', '13')
+        # limitations,
-        ):
+        # - https://bugs.python.org/issue38119
-            actor.lifetime_stack.callback(shml.shm.unlink)
+        # - https://bugs.python.org/issue45209
+        #
+        def try_unlink():
+            try:
+                shml.shm.unlink()
+            except FileNotFoundError as fne:
+                log.debug(
+                    f'ShmList already deallocated pre-actor-shutdown.\n'
+                    f'{fne!r}\n'
+                )
+
+        actor.lifetime_stack.callback(try_unlink)
+
     except RuntimeError:
         log.warning('tractor runtime not active, skipping teardown steps')
 

tractor/ipc/_uds.py

@@ -344,7 +344,18 @@ def close_listener(
 
     '''
     lstnr.socket.close()
+    # tolerate the sock-file being already gone — under concurrent
+    # pytest sessions sharing the bindspace dir, another session's
+    # reap path can unlink it first; raising here aborts the
+    # `_serve_ipc_eps` finally before `_shutdown.set()`, deadlocking
+    # `wait_for_shutdown()` on `actor.cancel()`.
+    try:
         os.unlink(addr.sockpath)
+    except FileNotFoundError:
+        log.warning(
+            f'UDS sock-file already unlinked, skipping\n'
+            f'  |_{addr.sockpath}\n'
+        )
 
 
 async def open_unix_socket_w_passcred(

tractor/log.py

@@ -543,21 +543,45 @@ def get_logger(
         #   only includes the first 2 sub-pkg name-tokens in the
         #   child-logger's name; the colored "pkg-namespace" header
         #   will then correctly show the same value as `name`.
+        #
+        # XXX, strip the trailing `pkg_path` token ONLY when it
+        # duplicates the caller's leaf-*module* name — which the
+        # console header already renders via its `{filename}` field.
+        # We compare against the caller module's `__name__`/
+        # `__package__` (rather than blindly dropping the last token)
+        # so genuine, possibly-*nested* sub-PACKAGE components stay
+        # addressable as their own sub-loggers:
+        #
+        # - `name='trionics._broadcast'` (a leaf-module, from a
+        #   `get_logger(__name__)`-style call) -> `tractor.trionics`
+        #   (leaf dropped; `_broadcast.py` is in the header).
+        # - `name='devx.debug'` (a real sub-PACKAGE, whether
+        #   auto-derived from a module's `__package__` or passed
+        #   explicitly by a logging-spec) -> `tractor.devx.debug`,
+        #   DISTINCT from a bare `devx` -> `tractor.devx`.
+        #
+        # The previous unconditional `pkg_path = subpkg_path` also ate
+        # the deepest sub-pkg, collapsing `devx.debug` -> `tractor.devx`
+        # and silently breaking per-sub-pkg level control via the
+        # logging-spec; see `tractor.log.LogSpec`/`apply_logspec()`.
+        caller_leaf_mod: str|None = None
+        if (caller_mod := get_caller_mod()):
+            cmod_name: str = getattr(caller_mod, '__name__', '') or ''
+            cmod_pkg: str = getattr(caller_mod, '__package__', '') or ''
+            # a leaf-*module* has `__name__ != __package__`; a package
+            # `__init__` has them equal (so its trailing token is a
+            # real sub-pkg, NOT a leaf-module-filename to strip).
+            if cmod_name and cmod_name != cmod_pkg:
+                caller_leaf_mod = cmod_name.rpartition('.')[2]
+
         if (
-            # XXX, TRY to remove duplication cases
-            # which get warn-logged on below!
-            (
-                # when, subpkg_path == pkg_path
             subpkg_path
             and
             rname == pkg_name
-            )
+            and
-            # ) or (
+            # only collapse when the trailing token IS the caller's
-            #     # when, pkg_path == leaf_mod
+            # leaf-module (i.e. the `{filename}` already shows it).
-            #     pkg_path
+            leaf_mod == caller_leaf_mod
-            #     and
-            #     leaf_mod == pkg_path
-            # )
         ):
             pkg_path = subpkg_path
 
@@ -711,6 +735,167 @@ def get_console_log(
     return log
 
 
+# A `tractor` "logging-spec": a compact, code-free way for a
+# consuming project's test-iface (or runtime) to dial-in console
+# loglevels across the lib's logger hierarchy. Mirrors the grammar
+# consumed by `modden.runtime.daemon.setup_tractor_logging()`.
+#
+# Accepted forms (`str|bool`),
+# - `True`              -> enable the `pkg_name` root-logger at
+#                          `default_level` (or 'cancel').
+# - `False`             -> disable (no-op, configure nothing).
+# - 'info'              -> a bare level for the root-logger.
+# - 'sub:info,x:cancel' -> per-sub-logger levels; each `<name>` is
+#                          RELATIVE to `pkg_name` (must NOT include
+#                          the `pkg_name` token itself), eg.
+#                          'devx.debug:runtime,trionics:cancel'.
+#
+# !GRANULARITY! sub-logger names match at the `pkg_name.<name>`
+# *logger* level — which (per `get_logger()`'s name-derivation) is
+# *sub-PACKAGE* granularity, addressable at ANY nesting depth:
+# - 'devx.debug' -> the `tractor.devx.debug` logger, DISTINCT from a
+#   bare 'devx' -> `tractor.devx` (its parent). Setting `devx` also
+#   gates `devx.debug` via normal stdlib level-inheritance unless the
+#   child sets its own level.
+# - leaf *modules* are intentionally NOT individually addressable:
+#   `get_logger()` drops the leaf module-name from the logger key
+#   since the console header already renders it via `{filename}`, so
+#   every module in a (sub-)pkg shares that pkg's logger. Per-leaf
+#   level control would need a record-filter (see follow-up notes:
+#   `ai/tooling-todos/logspec_leaf_module_granularity_route_b.md`).
+# - top-level lib modules (eg. `tractor.to_asyncio`) emit under the
+#   *root* `pkg_name` logger (their `__package__` IS `pkg_name`), so
+#   a 'to_asyncio:<level>' entry targets a phantom child that nothing
+#   emits to -> no-op. Use the bare-level/root form for those.
+LogSpec = str|bool
+
+
+def parse_logspec(
+    logspec: LogSpec,
+    default_level: str|None = None,
+    pkg_name: str = _proj_name,
+
+) -> dict[str|None, str]:
+    '''
+    Parse a `tractor` "logging-spec" (see `LogSpec`) into a
+    `{sublog_name|None: level}` mapping where a `None` key denotes
+    the `pkg_name` root-logger itself.
+
+    '''
+    match logspec:
+
+        # explicit disable -> configure nothing.
+        case False:
+            return {}
+
+        # enable the root-logger at the fallback level.
+        case True:
+            return {None: (default_level or 'cancel')}
+
+        case str(spec):
+            filters: list[str] = [
+                part.strip()
+                for part in spec.split(',')
+                if part.strip()
+            ]
+            # i. a bare level (no sub-logger filtering),
+            #    eg. 'info' | 'cancel'
+            if (
+                len(filters) == 1
+                and
+                ':' not in filters[0]
+            ):
+                return {None: filters[0]}
+
+            # ii. a per-sub-logger filter-spec of the form,
+            #     '<sublog_0>:<level>,<.. N-other-parts>'
+            #     eg. 'to_asyncio:cancel,devx._debug:runtime'
+            out: dict[str|None, str] = {}
+            for log_filter in filters:
+                name, sep, level = log_filter.partition(':')
+                if not sep:
+                    raise ValueError(
+                        f'Invalid `tractor` logging-spec part!\n'
+                        f'{log_filter!r}\n'
+                        f'\n'
+                        f'Mixed bare-level + sub-logger filters are '
+                        f'not supported; every comma-part must be '
+                        f'`<sublog>:<level>`.\n'
+                    )
+                # the sub-logger name is RELATIVE to `pkg_name`;
+                # duplicating the pkg-token is a user error since
+                # the root-logger already IS `pkg_name`.
+                if pkg_name in name.split('.'):
+                    raise ValueError(
+                        f'logging-spec sub-name should NOT include '
+                        f'the `pkg_name={pkg_name!r}` token!\n'
+                        f'got name={name!r}\n'
+                    )
+                out[name] = level
+            return out
+
+        case _:
+            raise ValueError(
+                f'Invalid `tractor` logging-spec!\n'
+                f'{logspec!r}\n'
+            )
+
+
+def apply_logspec(
+    logspec: LogSpec,
+    default_level: str|None = None,
+    pkg_name: str = _proj_name,
+
+) -> tuple[
+    str|None,
+    dict[str, StackLevelAdapter],
+]:
+    '''
+    Parse + apply a `tractor` "logging-spec" (see `parse_logspec()`):
+    enable a `colorlog` stderr console handler for each
+    (sub-)logger named in the spec at its requested level.
+
+    Returns a 2-tuple,
+    - the resolved "primary" runtime-level: the root-logger level if
+      the spec set one, else `default_level`; suitable for passing
+      to `open_root_actor(loglevel=<.>)`,
+    - a `{logger_name: StackLevelAdapter}` map of every logger the
+      spec touched.
+
+    '''
+    specs: dict[str|None, str] = parse_logspec(
+        logspec,
+        default_level=default_level,
+        pkg_name=pkg_name,
+    )
+    logs: dict[str, StackLevelAdapter] = {}
+    for sub_name, level in specs.items():
+        # NOTE, pass the RELATIVE sub-name (no `pkg_name.` prefix)
+        # to avoid `get_logger()`'s duplicate-pkg-token warning;
+        # it re-adds the pkg-name via `.getChild()` internally.
+        log: StackLevelAdapter = get_console_log(
+            level=level,
+            pkg_name=pkg_name,
+            name=(sub_name or pkg_name),
+        )
+        # XXX, a sub-logger filter is "authoritative" for its
+        # subtree: it gets its OWN stderr handler (added by
+        # `get_console_log()` above), so DON'T also let its records
+        # propagate up to a root `pkg_name`-logger handler — that
+        # would double-emit every line when a root-level console
+        # (eg. via `--ll`) is also active. The root-level form
+        # (`sub_name is None`) keeps default propagation.
+        if sub_name is not None:
+            log.logger.propagate = False
+        logs[log.name] = log
+
+    primary_level: str|None = specs.get(None, default_level)
+    return (
+        primary_level,
+        logs,
+    )
+
+
 def get_loglevel() -> str:
     return _default_loglevel
 

tractor/runtime/_portal.py

@@ -55,6 +55,7 @@ from ..msg import (
     Return,
 )
 from .._exceptions import (
+    ActorTooSlowError,
     NoResult,
     TransportClosed,
 )
@@ -268,6 +269,7 @@ class Portal:
     async def cancel_actor(
         self,
         timeout: float | None = None,
+        raise_on_timeout: bool = False,
 
     ) -> bool:
         '''
@@ -281,6 +283,17 @@ class Portal:
         `._context.Context.cancel()` which CAN be used for this
         purpose.
 
+        `raise_on_timeout` (default `False`):
+
+        - `False` (legacy): on bounded-wait expiry, log at DEBUG
+          and return `False`. Used by callers that issue cancel
+          fire-and-forget and have their own escalation
+          (e.g. `_spawn.soft_kill()` checks `proc.poll()` after).
+        - `True`: on bounded-wait expiry, raise `ActorTooSlowError`
+          so the caller MUST handle the failure explicitly.
+          `ActorNursery.cancel()` opts in so it can escalate via
+          `proc.terminate()` per SC-discipline.
+
         '''
         __runtimeframe__: int = 1  # noqa
 
@@ -301,15 +314,16 @@ class Portal:
 
         # XXX the one spot we set it?
         chan._cancel_called: bool = True
+        cancel_timeout: float = (
+            timeout
+            or
+            self.cancel_timeout
+        )
         try:
             # send cancel cmd - might not get response
             # XXX: sure would be nice to make this work with
             # a proper shield
-            with trio.move_on_after(
+            with trio.move_on_after(cancel_timeout) as cs:
-                timeout
-                or
-                self.cancel_timeout
-            ) as cs:
                 cs.shield: bool = True
                 await self.run_from_ns(
                     'self',
@@ -317,16 +331,32 @@ class Portal:
                 )
                 return True
 
-            if cs.cancelled_caught:
+            # `move_on_after` fired — peer didn't ack within
-                # may timeout and we never get an ack (obvi racy)
+            # bounded window. Behaviour depends on
-                # but that doesn't mean it wasn't cancelled.
+            # `raise_on_timeout`:
+            if (
+                cs.cancelled_caught
+                and
+                raise_on_timeout
+            ):
+                raise ActorTooSlowError(
+                    f'Peer {peer_id} did not ack its '
+                    f'`Actor.cancel()` RPC within bounded wait '
+                    f'of {cancel_timeout!r}s'
+                )
+
+            # legacy fire-and-forget path: log + return False so
+            # the caller can decide whether to escalate.
+            #
+            # NOTE, we also land here in the (unexpected) case where
+            # the shielded `move_on_after` block exits WITHOUT
+            # `return True` and WITHOUT the deadline firing — prefer
+            # a soft `False` over an `assert`-crash mid-teardown.
             log.debug(
                 f'May have failed to cancel peer?\n'
                 f'\n'
                 f'c)=?> {peer_id}\n'
             )
-
-            # if we get here some weird cancellation case happened
             return False
 
         except TransportClosed as tpt_err:

tractor/runtime/_runtime.py

@@ -870,7 +870,14 @@ class Actor:
 
             accept_addrs: list[UnwrappedAddress]|None = None
 
-            if self._spawn_method == "trio":
+            if self._spawn_method in (
+                'trio',
+                'subint',
+                # `subint_forkserver` parent-side sends a
+                # `SpawnSpec` over IPC just like the other two
+                # — fork child-side runtime is trio-native.
+                'subint_forkserver',
+            ):
 
                 # Receive post-spawn runtime state from our parent.
                 spawnspec: msgtypes.SpawnSpec = await chan.recv()
@@ -922,11 +929,29 @@ class Actor:
                 # => update process-wide globals
                 # TODO! -[ ] another `Struct` for rtvs..
                 rvs: dict[str, Any] = spawnspec._runtime_vars
-                if rvs['_debug_mode']:
+
-                    from ..devx import (
+                # `stackscope` SIGUSR1 handler: install when ANY of
-                        enable_stack_on_sig,
+                # `_debug_mode` / `use_stackscope` rt-vars OR the
-                        maybe_init_greenback,
+                # `TRACTOR_ENABLE_STACKSCOPE` env var is set (the
-                    )
+                # latter being a lighter test-time hang-debug path;
+                # see `tractor._testing.pytest`'s `--enable-stackscope`
+                # CLI flag — env var propagates via fork-inherited
+                # environ).
+                #
+                # NOTE, NOT *exclusively* gated on `_debug_mode` so
+                # SIGUSR1 task-tree dumps work in plain (non-pdb)
+                # runs too — but we DO still install under
+                # `_debug_mode` since otherwise the default SIGUSR1
+                # action would terminate the proc, esp. nasty in
+                # infected-`asyncio` sub-actors mid-REPL.
+                if (
+                    rvs.get('_debug_mode')
+                    or
+                    rvs.get('use_stackscope')
+                    or
+                    os.environ.get('TRACTOR_ENABLE_STACKSCOPE')
+                ):
+                    from ..devx import enable_stack_on_sig
                     try:
                         # TODO: maybe return some status msgs upward
                         # to that we can emit them in `con_status`
@@ -938,10 +963,13 @@ class Actor:
 
                     except ImportError:
                         log.warning(
-                            '`stackscope` not installed for use in debug mode!'
+                            '`stackscope` not installed for use in '
+                            'debug mode / `--enable-stackscope`!'
                         )
 
+                if rvs['_debug_mode']:
                     if rvs.get('use_greenback', False):
+                        from ..devx import maybe_init_greenback
                         maybe_mod: ModuleType|None = await maybe_init_greenback()
                         if maybe_mod:
                             log.devx(
@@ -1209,6 +1237,23 @@ class Actor:
                 ipc_server.cancel()
                 await ipc_server.wait_for_shutdown()
 
+            # Break the shield on the parent-channel
+            # `process_messages` loop (started with `shield=True`
+            # in `async_main` above). Required to avoid a
+            # deadlock during teardown of fork-spawned subactors:
+            # without this cancel, the loop parks waiting for
+            # EOF on the parent channel, but the parent is
+            # blocked on `os.waitpid()` for THIS actor's exit
+            # — mutual wait. For exec-spawn backends the EOF
+            # arrives naturally when the parent closes its
+            # handler-task socket during its own teardown, but
+            # in fork backends the shared-process-image makes
+            # that delivery racy / not guaranteed. Explicit
+            # cancel here gives us deterministic unwinding
+            # regardless of backend.
+            if self._parent_chan_cs is not None:
+                self._parent_chan_cs.cancel()
+
             # cancel all rpc tasks permanently
             if self._service_tn:
                 self._service_tn.cancel_scope.cancel()
@@ -1729,7 +1774,16 @@ async def async_main(
                 # start processing parent requests until our channel
                 # server is 100% up and running.
                 if actor._parent_chan:
-                    await root_tn.start(
+                    # Capture the shielded `loop_cs` for the
+                    # parent-channel `process_messages` task so
+                    # `Actor.cancel()` has a handle to break the
+                    # shield during teardown — without this, the
+                    # shielded loop would park on the parent chan
+                    # indefinitely waiting for EOF that only arrives
+                    # after the PARENT tears down, which under
+                    # fork-based backends (e.g. `main_thread_forkserver`)
+                    # it waits on THIS actor's exit — deadlock.
+                    actor._parent_chan_cs = await root_tn.start(
                         partial(
                             _rpc.process_messages,
                             chan=actor._parent_chan,
@@ -1940,7 +1994,25 @@ async def async_main(
                 f'   {pformat(ipc_server._peers)}'
             )
             log.runtime(teardown_report)
+            # NOTE: bound the peer-clear wait — otherwise if any
+            # peer-channel handler is stuck (e.g. never got its
+            # cancel propagated due to a runtime bug), this wait
+            # blocks forever and deadlocks the whole actor-tree
+            # teardown cascade. 3s is enough for any graceful
+            # cancel-ack round-trip; beyond that we're in bug
+            # territory and need to proceed with local teardown
+            # so the parent's `_ForkedProc.wait()` can unblock.
+            # See `ai/conc-anal/
+            # subint_forkserver_test_cancellation_leak_issue.md`
+            # for the full diagnosis.
+            with trio.move_on_after(3.0) as _peers_cs:
                 await ipc_server.wait_for_no_more_peers()
+            if _peers_cs.cancelled_caught:
+                teardown_report += (
+                    f'-> TIMED OUT waiting for peers to clear '
+                    f'({len(ipc_server._peers)} still connected)\n'
+                )
+                log.warning(teardown_report)
 
         teardown_report += (
             '-]> all peer channels are complete.\n'

tractor/runtime/_state.py

@@ -93,6 +93,7 @@ class RuntimeVars(Struct):
     repl_fixture: bool|Callable = False  # |AbstractContextManager[bool]
     # for `tractor.pause_from_sync()` & `breakpoint()` support
     use_greenback: bool = False
+    use_stackscope: bool = False
 
     # infected-`asyncio`-mode: `trio` running as guest.
     _is_infected_aio: bool = False
@@ -102,7 +103,6 @@ class RuntimeVars(Struct):
         key,
         val,
     ) -> None:
-        breakpoint()
         super().__setattr__(key, val)
 
     def update(
@@ -117,7 +117,14 @@ class RuntimeVars(Struct):
             )
 
 
-_runtime_vars: dict[str, Any] = {
+# The "fresh process" defaults — what `_runtime_vars` looks
+# like in a just-booted Python process that hasn't yet entered
+# `open_root_actor()` nor received a parent `SpawnSpec`. Kept
+# as a module-level constant so `get_runtime_vars(clear_values=
+# True)` can reset the live dict back to this baseline (see
+# `tractor.spawn._main_thread_forkserver` for the one current
+# caller that needs it).
+_RUNTIME_VARS_DEFAULTS: dict[str, Any] = {
     # root of actor-process tree info
     '_is_root': False,  # bool
     '_root_mailbox': (None, None),  # tuple[str|None, str|None]
@@ -132,16 +139,19 @@ _runtime_vars: dict[str, Any] = {
     # `debug_mode: bool` settings
     '_debug_mode': False,  # bool
     'repl_fixture': False,  # |AbstractContextManager[bool]
-    # for `tractor.pause_from_sync()` & `breakpoint()` support
+    
-    'use_greenback': False,
+    'use_greenback': False,  # `.pause_from_sync()`/`breakpoint()`
+    'use_stackscope': False,  # trio-task-stack dumps on SIGUSR1
 
     # infected-`asyncio`-mode: `trio` running as guest.
     '_is_infected_aio': False,
 }
+_runtime_vars: dict[str, Any] = dict(_RUNTIME_VARS_DEFAULTS)
 
 
 def get_runtime_vars(
     as_dict: bool = True,
+    clear_values: bool = False,
 ) -> dict:
     '''
     Deliver a **copy** of the current `Actor`'s "runtime variables".
@@ -150,11 +160,62 @@ def get_runtime_vars(
     form, but the `RuntimeVars` struct should be utilized as possible
     for future calls.
 
-    '''
+    Pure read — **never mutates** the module-level `_runtime_vars`.
-    if as_dict:
-        return dict(_runtime_vars)
 
-    return RuntimeVars(**_runtime_vars)
+    If `clear_values=True`, return a copy of the fresh-process
+    defaults (`_RUNTIME_VARS_DEFAULTS`) instead of the live
+    dict. Useful in combination with `set_runtime_vars()` to
+    reset process-global state back to "cold" — the main caller
+    today is the `main_thread_forkserver` spawn backend's post-fork
+    child prelude:
+
+        set_runtime_vars(get_runtime_vars(clear_values=True))
+
+    `os.fork()` inherits the parent's full memory image, so the
+    child sees the parent's populated `_runtime_vars` (e.g.
+    `_is_root=True`) which would trip the `assert not
+    self.enable_modules` gate in `Actor._from_parent()` on the
+    subsequent parent→child `SpawnSpec` handshake if left alone.
+
+    '''
+    src: dict = (
+        _RUNTIME_VARS_DEFAULTS
+        if clear_values
+        else _runtime_vars
+    )
+    snapshot: dict = dict(src)
+    if as_dict:
+        return snapshot
+    return RuntimeVars(**snapshot)
+
+
+def set_runtime_vars(
+    rtvars: dict | RuntimeVars,
+) -> None:
+    '''
+    Atomically replace the module-level `_runtime_vars` contents
+    with those of `rtvars` (via `.clear()` + `.update()` so
+    live references to the same dict object remain valid).
+
+    Accepts either the historical `dict` form or the `RuntimeVars`
+    `msgspec.Struct` form (the latter still mostly unused but
+    the blessed forward shape — see the struct's definition).
+
+    Paired with `get_runtime_vars()` as the explicit
+    write-half of the runtime-vars API — prefer this over
+    direct mutation of `_runtime_vars[...]` from new call sites.
+
+    '''
+    if isinstance(rtvars, RuntimeVars):
+        # `msgspec.Struct` → dict via its declared field set;
+        # avoids pulling in `msgspec.structs.asdict` just for
+        # this one call path.
+        rtvars = {
+            field_name: getattr(rtvars, field_name)
+            for field_name in rtvars.__struct_fields__
+        }
+    _runtime_vars.clear()
+    _runtime_vars.update(rtvars)
 
 
 def last_actor() -> Actor|None:

tractor/runtime/_supervise.py

@@ -38,8 +38,14 @@ from ..discovery._addr import (
     UnwrappedAddress,
     mk_uuid,
 )
-from ._state import current_actor, is_main_process
+from ._state import (
-from ..log import get_logger, get_loglevel
+    current_actor,
+    is_main_process,
+)
+from ..log import (
+    get_logger,
+    get_loglevel,
+)
 from ._runtime import Actor
 from ._portal import Portal
 from ..trionics import (
@@ -47,6 +53,7 @@ from ..trionics import (
     collapse_eg,
 )
 from .._exceptions import (
+    ActorTooSlowError,
     ContextCancelled,
 )
 from .._root import (
@@ -60,11 +67,106 @@ if TYPE_CHECKING:
     import multiprocessing as mp
     # from ..ipc._server import IPCServer
     from ..ipc import IPCServer
+    from ..spawn._spawn import ProcessType
 
 
 log = get_logger()
 
 
+async def _try_cancel_then_kill(
+    portal: Portal,
+    # `ProcessType` is `TYPE_CHECKING`-only (defined under that
+    # guard in `..spawn._spawn`) so we stringify here to avoid
+    # eager runtime eval of the annotation at function-def time
+    # (this module has no `from __future__ import annotations`).
+    proc: 'ProcessType',
+    subactor: Actor,
+    debug_mode_active: bool = False,
+) -> None:
+    '''
+    Per-child cancel-then-escalate helper used by
+    `ActorNursery.cancel()`.
+
+    Sends a graceful actor-runtime cancel-RPC via
+    `Portal.cancel_actor(raise_on_timeout=True)`. If the bounded-wait
+    expires before the peer ack's, `ActorTooSlowError` is raised and
+    we escalate via `proc.terminate()` (SIGTERM) per SC-discipline:
+
+      graceful cancel-req -> bounded wait -> hard-kill
+
+    Without this escalation, a same-name sibling subactor whose
+    cancel-RPC failed to ack within `Portal.cancel_timeout` (e.g.
+    under TCP+forkserver register-RPC contention) would park the
+    parent's `soft_kill()` watcher forever waiting on `proc.poll()`,
+    deadlocking nursery `__aexit__`. See `ActorTooSlowError` for
+    the wider write-up.
+
+    '''
+    # XXX, do NOT escalate to `proc.terminate()` while ANY of
+    # the following are true — SIGTERM-ing a sub would tear
+    # down its sub-tree including any descendant proxying
+    # stdio to/from a REPL-locked actor, clobbering the user's
+    # debug session:
+    #
+    # - `Lock.ctx_in_debug is not None`: most precise — some
+    #   actor in the tree is currently REPL-locked. Set in the
+    #   root actor for the lifetime of the lock. Raceable
+    #   (false negative if SIGINT arrives before lock-acquire
+    #   RPC completes).
+    #
+    # - `_runtime_vars['_debug_mode']`: root-actor was opened
+    #   with `debug_mode=True` (via `open_root_actor` /
+    #   `open_nursery`). Set once at root boot, never cleared.
+    #   Catches deep-descendant REPL sessions even when the
+    #   intermediate nurseries didn't pass `debug_mode=` per-
+    #   child.
+    #
+    # - `debug_mode_active`: this nursery has at least one
+    #   child started with an explicit `debug_mode=` arg
+    #   (`ActorNursery._at_least_one_child_in_debug`). Catches
+    #   the case where root is NOT in debug-mode but a
+    #   nursery-direct child opted in.
+    #
+    # Independent because root may NOT be in debug-mode even
+    # when a child is (only the child's `_runtime_vars` is
+    # mutated by per-child `debug_mode=True`). ORing covers
+    # every flavor without false-positively skipping
+    # legitimate hard-kill paths in non-debug trees.
+    if (
+        debug.Lock.ctx_in_debug is not None
+        or
+        _state._runtime_vars.get('_debug_mode', False)
+        or
+        debug_mode_active
+    ):
+        await portal.cancel_actor()
+        return
+
+    try:
+        await portal.cancel_actor(raise_on_timeout=True)
+    except ActorTooSlowError as too_slow:
+        log.error(
+            f'Cancel-ack TIMED OUT for sub-actor\n'
+            f'  uid: {subactor.aid.reprol()!r}\n'
+            f'  reason: {too_slow}\n'
+            f'-> escalating to `proc.terminate()` (hard-kill)\n'
+        )
+        # XXX, the `subint` backend stores an `int` interp-id in the
+        # `proc` slot (not a `Process`), so it has no `.terminate()`.
+        # Guard here so a cancel-ack timeout doesn't `AttributeError`
+        # once that backend lands; its hard-kill path is a TODO.
+        if hasattr(proc, 'terminate'):
+            proc.terminate()
+        else:
+            log.error(
+                f'Cannot hard-kill sub-actor — backend proc-handle '
+                f'{proc!r} ({type(proc).__name__!r}) has no '
+                f'`.terminate()`!\n'
+                f'  uid: {subactor.aid.reprol()!r}\n'
+                f'TODO: per-backend cancel-escalation.\n'
+            )
+
+
 class ActorNursery:
     '''
     The fundamental actor supervision construct: spawn and manage
@@ -428,10 +530,23 @@ class ActorNursery:
                                 else:  # there's no other choice left
                                     proc.terminate()
 
-                        # spawn cancel tasks for each sub-actor
+                        # spawn per-child cancel tasks; the helper
+                        # escalates to hard-kill on
+                        # `ActorTooSlowError` rather than silently
+                        # swallowing the cancel-ack timeout, EXCEPT
+                        # when this nursery has any debug-eligible
+                        # child (in which case we keep legacy
+                        # fire-and-forget semantics to avoid
+                        # clobbering an active REPL).
                         assert portal
                         if portal.channel.connected():
-                            tn.start_soon(portal.cancel_actor)
+                            tn.start_soon(
+                                _try_cancel_then_kill,
+                                portal,
+                                proc,
+                                subactor,
+                                self._at_least_one_child_in_debug,
+                            )
 
                 log.cancel(msg)
         # if we cancelled the cancel (we hung cancelling remote actors)

tractor/spawn/_reap.py

@@ -0,0 +1,183 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Post-mortem subactor cleanup primitives — things the parent
+runtime has to clean up because the dead-or-SIGKILL'd child
+couldn't.
+
+Sibling of `tractor._testing._reap` which is the test-harness
+equivalent (orphan-pid + leaked-shm + leaked-UDS-sock sweeper
+fixtures). This module is the spawn-layer counterpart, called
+inline from `hard_kill` and the broader subactor reap path.
+
+Today this is just `unlink_uds_bind_addrs()`. As future
+post-mortem cleanup needs surface (e.g. `/dev/shm` segment
+unlink for hard-crashed actors, leaked-pidfile cleanup), they
+land here too.
+
+Future-work TODO — authoritative UDS bind-addr tracking
+-------------------------------------------------------
+
+`unlink_uds_bind_addrs()` currently has two cleanup paths:
+
+1. Explicit `bind_addrs` (when parent set them at spawn time)
+2. **Convention-based reconstruction** —
+   `<XDG_RUNTIME_DIR>/tractor/<name>@<pid>.sock` — for the
+   common case where the subactor self-assigned a random sock
+   via `UDSAddress.get_random()`.
+
+Path (2) hardcodes the `<name>@<pid>.sock` convention from
+`tractor.ipc._uds.UDSAddress`. If that convention ever
+changes — or the subactor binds to a non-default
+`bindspace`/`filedir` — we'll silently fail to unlink.
+
+A more authoritative approach would be:
+
+- Subactors register their bound UDS sockpaths in a
+  per-process registry inside `tractor.ipc._uds` at
+  `start_listener()` time.
+- The subactor reports its bound sockpath(s) back to the
+  parent over IPC immediately post-bind (extension to
+  `SpawnSpec` reply / a new handshake msg).
+- Parent caches the subactor's authoritative sockpaths.
+- `unlink_uds_bind_addrs()` checks the cache FIRST, falls
+  back to convention-reconstruction if the subactor died
+  before reporting (which is the SIGKILL case this fn
+  primarily exists for).
+
+Tracked as future work in #454 (the parent UDS-leak
+issue this module addresses); a separate issue may be
+filed if/when the registry impl is scoped.
+
+See also #452 — the discovery-client `CLOSE_WAIT` TCP
+fd leak. Different bug class but same broader theme of
+"fork-spawn unmasked latent cleanup gaps".
+
+'''
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+import trio
+
+from tractor.discovery._addr import (
+    UnwrappedAddress,
+    wrap_address,
+)
+from tractor.ipc._uds import UDSAddress
+from tractor.log import get_logger
+
+
+if TYPE_CHECKING:
+    from tractor.runtime._runtime import Actor
+
+
+log = get_logger('tractor')
+
+
+def unlink_uds_bind_addrs(
+    proc: trio.Process,
+    *,
+    bind_addrs: list[UnwrappedAddress] | None = None,
+    subactor: Actor | None = None,
+) -> None:
+    '''
+    Best-effort post-mortem cleanup of any UDS sock-files
+    a hard-killed subactor was bound to.
+
+    SIGKILL bypasses Python execution → the subactor's
+    `_serve_ipc_eps` `finally:` block (which normally calls
+    `os.unlink(addr.sockpath)`) never runs. Without this
+    parent-side cleanup, the dead subactor's
+    `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` file
+    accumulates on the filesystem (see issue #454 + the
+    autouse `_track_orphaned_uds_per_test` fixture).
+
+    Two cleanup paths, in order:
+
+    1. **Explicit `bind_addrs`** — when the parent set the
+       subactor's bind addrs at spawn time, unlink each
+       UDS-flavored sockpath directly.
+    2. **Self-assigned reconstruction** — when
+       `bind_addrs` is empty (the common case: subactor
+       picked its own random sock via
+       `UDSAddress.get_random()`), reconstruct the path
+       from `(subactor.aid.name, proc.pid)` using the
+       same `<name>@<pid>.sock` convention. We can do this
+       because the subactor uses its OWN `os.getpid()` at
+       bind time, which equals `proc.pid` from the
+       parent's view.
+
+    Idempotent: `FileNotFoundError` (graceful exit
+    already-unlinked, or sock never bound under early-
+    spawn cancel) is silenced; other `OSError`s log a
+    warning but never raise. TCP / non-UDS bind addrs are
+    skipped.
+
+    '''
+    sockpaths: list[str] = []
+
+    # path 1: explicit bind_addrs set at spawn time
+    for unwrapped in (bind_addrs or ()):
+        try:
+            addr = wrap_address(unwrapped)
+        except Exception:
+            log.exception(
+                f'Failed to wrap addr for UDS post-kill cleanup '
+                f'— skipping {unwrapped!r}\n'
+            )
+            continue
+        if isinstance(addr, UDSAddress):
+            sockpaths.append(str(addr.sockpath))
+
+    # path 2: reconstruct from subactor name + proc pid
+    # for the random-self-assign case (bind_addrs=None)
+    #
+    # TODO authoritative tracking — see module docstring.
+    if (
+        not sockpaths
+        and subactor is not None
+        and proc.pid is not None
+    ):
+        sockname: str = f'{subactor.aid.name}@{proc.pid}.sock'
+        sockpath: str = str(
+            UDSAddress.def_bindspace / sockname
+        )
+        sockpaths.append(sockpath)
+
+    for sockpath in sockpaths:
+        try:
+            os.unlink(sockpath)
+            log.runtime(
+                f'Unlinked orphaned UDS sock-file post-SIGKILL\n'
+                f' |_{proc}\n'
+                f' |_{sockpath}\n'
+            )
+        except FileNotFoundError:
+            # raced — subactor cleaned up before SIGKILL,
+            # OR sockfile never bound (early-spawn cancel),
+            # OR transport wasn't UDS this run.
+            pass
+        except OSError as exc:
+            log.warning(
+                f'Failed to unlink subactor UDS sock-file '
+                f'post-SIGKILL\n'
+                f' |_{proc}\n'
+                f' |_{sockpath}\n'
+                f' |_{exc!r}\n'
+            )

tractor/spawn/_spawn.py

@@ -39,7 +39,10 @@ from tractor.runtime._state import (
     _runtime_vars,
 )
 from tractor.log import get_logger
-from tractor.discovery._addr import UnwrappedAddress
+from tractor.discovery._addr import (
+    UnwrappedAddress,
+)
+from ._reap import unlink_uds_bind_addrs
 from tractor.runtime._portal import Portal
 from tractor.runtime._runtime import Actor
 from tractor.msg import types as msgtypes
@@ -223,6 +226,16 @@ async def hard_kill(
     # whilst also hacking on it XD
     # terminate_after: int = 99999,
 
+    *,
+    # Subactor's bind addresses + subactor record, used
+    # for post-SIGKILL UDS sockpath cleanup. Optional for
+    # legacy callers; new call sites should pass at least
+    # `subactor` (which lets us reconstruct the sock path
+    # from `aid.name + proc.pid` when `bind_addrs` is
+    # empty/self-assigned). See `._reap.unlink_uds_bind_addrs()`.
+    bind_addrs: list[UnwrappedAddress] | None = None,
+    subactor: Actor | None = None,
+
 ) -> None:
     '''
     Un-gracefully terminate an OS level `trio.Process` after timeout.
@@ -310,6 +323,21 @@ async def hard_kill(
         )
         proc.kill()
 
+    # Post-mortem UDS sockpath cleanup. SIGKILL bypassed
+    # the subactor's normal `os.unlink(addr.sockpath)` in
+    # `_serve_ipc_eps`'s `finally:`; the parent has the
+    # bind addrs (or can reconstruct from name + pid) so
+    # we do it here. Runs UNCONDITIONALLY (graceful-exit
+    # case is a no-op via `FileNotFoundError` skip in the
+    # helper) so the cleanup also covers the "cancelled
+    # during spawn" path where the subactor never reached
+    # its IPC server finally block.
+    unlink_uds_bind_addrs(
+        proc,
+        bind_addrs=bind_addrs,
+        subactor=subactor,
+    )
+
 
 async def soft_kill(
     proc: ProcessType,

tractor/spawn/_trio.py

@@ -282,7 +282,23 @@ async def trio_proc(
 
                 if proc.poll() is None:
                     log.cancel(f"Attempting to hard kill {proc}")
-                    await hard_kill(proc)
+                    await hard_kill(
+                        proc,
+                        # NOTE, pass through so post-SIGKILL we
+                        # can `os.unlink()` the subactor's
+                        # orphaned UDS sock-file(s) — the
+                        # subactor's own
+                        # `_serve_ipc_eps`-`finally:` cleanup
+                        # never runs under SIGKILL. `subactor`
+                        # lets the helper reconstruct the
+                        # sock path via `aid.name + proc.pid`
+                        # when `bind_addrs` is the common
+                        # self-assigned-random case
+                        # (bind_addrs=None at spawn). See
+                        # `_unlink_uds_bind_addrs()` in `_spawn`.
+                        bind_addrs=bind_addrs,
+                        subactor=subactor,
+                    )
 
                 log.debug(f"Joined {proc}")
         else:

tractor/to_asyncio.py

@@ -27,6 +27,7 @@ from contextlib import asynccontextmanager as acm
 from dataclasses import dataclass
 import inspect
 import platform
+import sys
 import traceback
 from typing import (
     Any,
@@ -810,6 +811,151 @@ def _run_asyncio_task(
     return chan
 
 
+def maybe_signal_aio_task(
+    aio_task: asyncio.Task,
+    exc: BaseException,
+    *,
+    cause: BaseException|None = None,
+    pre_captured_fut: asyncio.Future|None = None,
+    allow_cancel_fallback: bool = False,
+
+) -> tuple[bool, str]:
+    '''
+    Best-effort delivery of `exc` to a still-running `aio_task`
+    via its `_fut_waiter` (the `asyncio.Future` the task is
+    currently `await`-ing on).
+
+    Returns `(delivered, report)` where `delivered=True` iff
+    either,
+      - `fut.set_exception(exc)` was successfully called on an
+        un-`done()` `_fut_waiter`, OR
+      - the cancel-fallback path fired (only when the caller
+        opted-in via `allow_cancel_fallback=True`).
+
+    Why `_fut_waiter.set_exception(exc)` and NOT
+    `aio_task.set_exception(exc)`:
+
+      On py3.13+ `asyncio.Task.set_exception()` ALWAYS raises
+      `RuntimeError("Task does not support set_exception
+      operation")` — so calling it as a relay mechanism is dead
+      code. The `_fut_waiter` is a plain `asyncio.Future` and
+      its `set_exception()` works on all Python versions; the
+      task's `_wakeup` callback then propagates the exc into
+      the coro on its next tick.
+
+    Why we PREFER NOT to call `aio_task.cancel()`:
+
+      `Task.cancel()` injects a `CancelledError` that races
+      any in-flight exception already queued on `_fut_waiter`
+      (e.g. via a prior `set_exception()` from a sibling
+      teardown path). The race can mask BOTH the original
+      trio-side error and any asyncio-side error the task was
+      mid-raising. See the
+      `test_trio_closes_early_and_channel_exits` hang TODO
+      around the `translate_aio_errors` finally for the
+      historical artifact.
+
+      However a caller may have NO OTHER way to terminate the
+      task — when `_fut_waiter is None` AND the task is busy
+      looping / runnable, neither `set_exception` nor a chan
+      close can poke it. In that narrow case `cancel()` is the
+      only available termination signal; opt-in via
+      `allow_cancel_fallback=True`. The fallback NEVER runs
+      when `_fut_waiter` carries an in-flight exc (the
+      `fut.done()` branch); only when there's truly no
+      `_fut_waiter` ref to poke.
+
+    Pre-checkpoint capture:
+
+      `asyncio.Task._wakeup` clears `_fut_waiter = None` as
+      part of the wakeup sequence. If the caller crosses a
+      trio checkpoint between fut-capture and this call,
+      re-reading `aio_task._fut_waiter` will see `None` even
+      though the exc is still in flight on the (now-`done()`)
+      original fut. Pass `pre_captured_fut` to use the
+      already-captured reference.
+
+    Causal chaining via `cause`:
+
+      Pass the underlying trio-side exc (the *reason* we're
+      poking the aio side) via `cause` and the helper sets
+      `exc.__cause__ = cause`. The chain travels with `exc`
+      through `_fut_waiter.set_exception()` → `Task._wakeup`
+      → coro raise → `wait_on_coro_final_result`'s except →
+      `signal_trio_when_done`'s `task.result()`-`raise
+      aio_err`. The final traceback then renders as
+      "<trio-side exc> -> (direct cause of) -> <relay exc>"
+      instead of an opaque, root-cause-detached relay.
+
+    See the "cross-loop cause-chain matrix" comment in
+    `translate_aio_errors()`'s final-raise block for how this
+    `cause` interacts with every `raise X [from Y]` exit path
+    (esp. the relay-echo guard which prevents a cause CYCLE).
+
+    '''
+    if cause is not None and exc.__cause__ is None:
+        exc.__cause__ = cause
+
+    if aio_task.done():
+        return False, (
+            f'aio-task already done; nothing to signal\n'
+            f' |_{aio_task!r}\n'
+        )
+
+    fut: asyncio.Future|None = (
+        pre_captured_fut
+        if pre_captured_fut is not None
+        else aio_task._fut_waiter
+    )
+
+    if fut and not fut.done():
+        fut.set_exception(exc)
+        return True, (
+            f'signalled aio-task via `_fut_waiter.set_exception()`\n'
+            f'exc: {exc!r}\n'
+            f' |_{aio_task!r}\n'
+        )
+
+    if fut and fut.done():
+        # NEVER cancel here even when `allow_cancel_fallback=True`
+        # — the in-flight exc on `fut` will terminate the task
+        # on its next tick; injecting `CancelledError` on top
+        # would race and mask the real exc.
+        return False, (
+            f'`_fut_waiter` already signalled with,\n'
+            f' |_{fut.exception()!r}\n'
+            f'aio-task will exit on next tick via the in-flight exc;\n'
+            f'SKIPPING re-signal (would race in-flight delivery).\n'
+            f' |_{aio_task!r}\n'
+        )
+
+    # fut is None — task is runnable (sitting in asyncio's
+    # ready queue), not parked on a future we can poke.
+    if allow_cancel_fallback:
+        cancel_msg: str = (
+            f'\n'
+            f'MANUALLY Cancelling `asyncio`-task: '
+            f'{aio_task.get_name()}!\n\n'
+            f'**THIS CAN SILENTLY SUPPRESS ERRORS FYI\n\n'
+        )
+        aio_task.cancel(msg=cancel_msg)
+        return True, (
+            f'aio-task has no `_fut_waiter`; FALLBACK cancel issued\n'
+            f'(caller opted-in via `allow_cancel_fallback=True`).\n'
+            f'{cancel_msg}'
+            f' |_{aio_task!r}\n'
+        )
+
+    return False, (
+        f'aio-task has no `_fut_waiter`; cannot signal without\n'
+        f'`aio_task.cancel()` which can mask errors.\n'
+        f'LEAVING AS-IS (caller did NOT opt-in to cancel fallback);\n'
+        f'task should exit via chan close / aio-loop teardown\n'
+        f'already in flight.\n'
+        f' |_{aio_task!r}\n'
+    )
+
+
 @acm
 async def translate_aio_errors(
     chan: LinkedTaskChannel,
@@ -985,38 +1131,25 @@ async def translate_aio_errors(
         # if isinstance(chan._aio_err, AsyncioTaskExited):
         #     await tractor.pause(shield=True)
 
-        # if aio side is still active cancel it due to the trio-side
+        # if aio side is still active relay the trio-side error
-        # error!
+        # to it via `_fut_waiter.set_exception()`.
         # ?TODO, mk `AsyncioCancelled[typeof(trio_err)]` embed the
         # current exc?
-        if (
-            # not aio_task.cancelled()
-            # and
-            not aio_task.done()  # TODO? only need this one?
-
-            # XXX LOL, so if it's not set it's an error !?
-            # yet another good jerb by `ascyncio`..
-            # and
-            # not aio_task.exception()
-        ):
         aio_taskc = TrioCancelled(
             f'The `trio`-side task crashed!\n'
             f'{trio_err}'
         )
-            # ??TODO? move this into the func that tries to use
+        delivered, report = maybe_signal_aio_task(
-            # `Task._fut_waiter: Future` instead??
+            aio_task,
-            #
+            aio_taskc,
-            # aio_task.set_exception(aio_taskc)
+            # so the relay carries a "<trio_err> -> caused ->
-            # wait_on_aio_task = False
+            # TrioCancelled" chain when it eventually re-raises
-            try:
+            # on the aio side.
-                aio_task.set_exception(aio_taskc)
+            cause=trio_err,
-            except (
+        )
-                asyncio.InvalidStateError,
+        if not delivered:
-                RuntimeError,
-                # ^XXX, uhh bc apparently we can't use `.set_exception()`
-                # any more XD .. ??
-            ):
             wait_on_aio_task = False
+        log.cancel(report)
 
     finally:
         # record wtv `trio`-side error transpired
@@ -1099,27 +1232,22 @@ async def translate_aio_errors(
                 if _py_313:
                     chan._to_aio.shutdown()
 
+                # XXX CRITICAL ordering: capture `_fut_waiter`
+                # BEFORE the checkpoint. `asyncio.Task._wakeup`
+                # clears `_fut_waiter = None` as part of wakeup,
+                # so re-reading after the checkpoint loses the
+                # ref even though the exc is still in-flight on
+                # the (now-`done()`) original fut. The helper
+                # uses `pre_captured_fut` to recover that.
+                pre_cp_fut: asyncio.Future|None = aio_task._fut_waiter
+
                 # pump this event-loop (well `Runner` but ya)
-                #
+                # so the aio side can error on next tick and we
-                # TODO? is this actually needed?
+                # sync task states from here onward.
-                # -[ ] theory is this let's the aio side error on
-                #     next tick and then we sync task states from
-                #     here onward?
                 await trio.lowlevel.checkpoint()
 
-                # TODO? factor the next 2 branches into a func like
-                # `try_terminate_aio_task()` and use it for the taskc
-                # case above as well?
-                fut: asyncio.Future|None = aio_task._fut_waiter
-                if (
-                    fut
-                    and
-                    not fut.done()
-                ):
-                    # await tractor.pause()
                 if graceful_trio_exit:
-                        fut.set_exception(
+                    relay_exc = TrioTaskExited(
-                            TrioTaskExited(
                         f'the `trio.Task` gracefully exited but '
                         f'its `asyncio` peer is not done?\n'
                         f')>\n'
@@ -1128,31 +1256,30 @@ async def translate_aio_errors(
                         f'>>\n'
                         f' |_{aio_task!r}\n'
                     )
-                        )
-
-                    # TODO? should this need to exist given the equiv
-                    # `TrioCancelled` equivalent in the be handler
-                    # above??
                 else:
-                        fut.set_exception(
+                    relay_exc = TrioTaskExited(
-                            TrioTaskExited(
                         f'The `trio`-side task crashed!\n'
                         f'{trio_err}'
                     )
+
+                delivered, signal_report = maybe_signal_aio_task(
+                    aio_task,
+                    relay_exc,
+                    pre_captured_fut=pre_cp_fut,
+                    # XXX historically this branch called
+                    # `aio_task.cancel()` when `_fut_waiter`
+                    # was None — required to actually terminate
+                    # aio tasks that aren't parked on a poke-able
+                    # future (e.g. the `aio_echo_server` loop in
+                    # `test_echoserver_detailed_mechanics`). Opt
+                    # into the fallback so we don't regress.
+                    allow_cancel_fallback=True,
+                    # carry the trio-side exc (if any) as the
+                    # cause so the aio-side relay shows the
+                    # real root-cause chain when re-raised.
+                    cause=trio_err,
                 )
-                else:
+                report += signal_report
-                    aio_taskc_warn: str = (
-                        f'\n'
-                        f'MANUALLY Cancelling `asyncio`-task: {aio_task.get_name()}!\n\n'
-                        f'**THIS CAN SILENTLY SUPPRESS ERRORS FYI\n\n'
-                    )
-                    # await tractor.pause()
-                    report += aio_taskc_warn
-                    # TODO XXX, figure out the case where calling this makes the
-                    # `test_infected_asyncio.py::test_trio_closes_early_and_channel_exits`
-                    # hang and then don't call it in that case!
-                    #
-                    aio_task.cancel(msg=aio_taskc_warn)
 
             log.warning(report)
 
@@ -1161,10 +1288,11 @@ async def translate_aio_errors(
         # `channel._aio_err/._trio_to_raise`) BEFORE calling
         # `maybe_raise_aio_side_err()` below!
         #
-        # XXX WARNING NOTE
+        # NOTE, `wait_on_aio_task` may have been flipped to `False`
-        # the `task.set_exception(aio_taskc)` call above MUST NOT
+        # by `maybe_signal_aio_task()` above when delivery
-        # EXCEPT or this WILL HANG!! SO, if you get a hang maybe step
+        # failed (e.g. `_fut_waiter is None`) — in that case we
-        # through and figure out why it erroed out up there!
+        # skip the wait since the aio task won't process our
+        # relay exc and `_aio_task_complete` may never set.
         #
         if wait_on_aio_task:
             await chan._aio_task_complete.wait()
@@ -1181,6 +1309,47 @@ async def translate_aio_errors(
           - `run_task()`
 
         '''
+        # ===== cross-loop cause-chain matrix =====
+        # How `(trio_err, aio_err, trio_to_raise)` resolve into ONE
+        # terminal `raise X [from Y]` (or an early `return`).
+        #
+        # legend (the possible `X` / `Y` operands):
+        # - trio_err        : `chan._trio_err`, the trio-side exc.
+        # - aio_err         : `chan._aio_err`, the aio-side exc.
+        # - trio_to_raise   : `chan._trio_to_raise`, a tractor-chosen
+        #                     relay exc (`AsyncioCancelled`/`AsyncioTaskExited`).
+        # - raise_from      : `trio_err if (aio_err is trio_to_raise)
+        #                      else aio_err` (the chosen `__cause__`).
+        # - relay-echo      : an `aio_err` that is one of OUR OWN
+        #                     `TrioTaskExited|TrioCancelled` signals,
+        #                     synth'd + delivered to the aio-side by
+        #                     `maybe_signal_aio_task()`; its `__cause__`
+        #                     is ALREADY `trio_err`.
+        # - "(bare)"        : raised with NO explicit `from` clause.
+        #
+        # this block (final-raise in `translate_aio_errors`):
+        #   condition                            =>  raises         from
+        #   -----------------------------------      -------------  -----------
+        #   not suppress_graceful_exits          =>  trio_to_raise  raise_from
+        #   AsyncioTaskExited + trio Cancelled/None  => return (aio-exit ignored)
+        #   AsyncioTaskExited + trio EoC         =>  trio_err       (bare)
+        #   AsyncioCancelled  + trio Cancelled   =>  return (co-cancel ignored)
+        #   trio_to_raise match catch-all        =>  trio_to_raise  raise_from
+        #   aio_err is relay-echo  ◄── the GUARD  =>  trio_err       (bare)
+        #   aio_err independent (real aio fail)  =>  trio_err       aio_err
+        #   aio_err independent, no trio_err     =>  aio_err        (bare)
+        #   only trio_err                        =>  trio_err       (bare)
+        #
+        # sibling block (`signal_trio_when_done()`, the aio done-cb):
+        #   AsyncioTaskExited relay-out          =>  trio_to_raise  aio_err
+        #   plain aio_err re-raise               =>  aio_err  (__cause__ preset)
+        #
+        # INVARIANT: a relay-echo must NEVER become `trio_err.__cause__`
+        #  (it's ALREADY caused-BY `trio_err`) → doing so would CYCLE
+        #  (`trio_err ◄─► relay`). So the guard raises the root
+        #  `trio_err` bare; the relay still keeps its own correct
+        #  "relay ◄ trio_err" chain for any aio-side inspection.
+        # ===== / cross-loop cause-chain matrix =====
         aio_err: BaseException|None = chan._aio_err
         trio_to_raise: (
             AsyncioCancelled|
@@ -1237,6 +1406,32 @@ async def translate_aio_errors(
             and
             type(aio_err) is not AsyncioCancelled
         ):
+            # XXX, if `aio_err` is one of OUR OWN relay-signals
+            # (`TrioTaskExited`/`TrioCancelled`) that we delivered
+            # to the aio-side via `maybe_signal_aio_task()`, AND
+            # its `__cause__` already points back at `trio_err`,
+            # then it's just a derivative ECHO of the trio-side
+            # error, NOT an independent asyncio failure.
+            #
+            # Raising `trio_err from aio_err` here would invert
+            # (and cyclically tangle) the cause chain since the
+            # relay was itself caused-by `trio_err`:
+            #
+            #   trio_err.__cause__ = aio_err   (from `raise .. from`)
+            #   aio_err.__cause__  = trio_err  (set in `maybe_signal_aio_task`)
+            #
+            # So raise the REAL root `trio_err` alone; the relay's
+            # own `__cause__` chain still correctly reads
+            # "TrioTaskExited <- trio_err" for aio-side inspection.
+            if (
+                trio_err is not None
+                and
+                isinstance(aio_err, (TrioTaskExited, TrioCancelled))
+                and
+                aio_err.__cause__ is trio_err
+            ):
+                raise trio_err
+
             # always raise from any captured asyncio error
             if trio_err:
                 raise trio_err from aio_err
@@ -1353,19 +1548,22 @@ async def open_channel_from(
                 # a `Return`-msg for IPC ctxs)
                 aio_task: asyncio.Task = chan._aio_task
                 if not aio_task.done():
-                    fut: asyncio.Future|None = aio_task._fut_waiter
+                    # capture the in-flight trio-side exc (if any)
-                    if fut:
+                    # so the relay's `__cause__` chain shows the
-                        fut.set_exception(
+                    # real root cause when the aio task re-raises.
+                    # `sys.exc_info()[1]` is non-`None` only when
+                    # the `try` body raised (graceful exit -> None).
+                    trio_exc: BaseException|None = sys.exc_info()[1]
+                    _, report = maybe_signal_aio_task(
+                        aio_task,
                         TrioTaskExited(
                             f'but the child `asyncio` task is still running?\n'
                             f'>>\n'
                             f' |_{aio_task!r}\n'
+                        ),
+                        cause=trio_exc,
                     )
-                        )
+                    log.cancel(report)
-                    else:
-                        # XXX SHOULD NEVER HAPPEN!
-                        log.error("SHOULD NEVER GET HERE !?!?")
-                        await tractor.pause(shield=True)
                 else:
                     chan._to_trio.close()
 
@@ -1602,6 +1800,7 @@ def run_as_asyncio_guest(
         fute_err: BaseException|None = None
         try:
             out: Outcome = await asyncio.shield(trio_done_fute)
+            # out: Outcome = await trio_done_fute
             # ^TODO still don't really understand why the `.shield()`
             # is required ... ??
             # https://docs.python.org/3/library/asyncio-task.html#asyncio.shield

tractor/trionics/patches/README.md

@@ -0,0 +1,95 @@
+# `tractor.trionics.patches`
+
+Defensive monkey-patches for bugs in `trio` itself.
+
+## What goes here
+
+- Bugs in upstream `trio` that we've encountered while
+  running `tractor` and need to work around until
+  upstream releases a fix.
+- Each patch fixes EXACTLY one trio internal — no
+  multi-bug omnibus patches.
+
+## What does NOT go here
+
+- Bugs in `tractor`'s own code (those get fixed
+  in-tree, in the offending tractor module).
+- Bugs in `asyncio`, `pytest`, the stdlib, etc. (file
+  separate `tractor.<lib>.patches` subpkgs as
+  needed).
+- Workarounds for behavior we *disagree* with but that
+  isn't a bug per se. If trio's API does what it says
+  on the tin, we don't override it here.
+
+## Per-patch contract
+
+Every `_<topic>.py` module in this directory MUST
+expose:
+
+- **`apply() -> bool`** — apply the patch. Idempotent
+  (safe to call multiple times). Version-gated — must
+  consult `is_needed()` and skip when False. Returns
+  `True` if patched this call, `False` if skipped.
+
+- **`is_needed() -> bool`** — does upstream still need
+  patching? Today most patches return `True`
+  unconditionally, but as upstream releases land each
+  should gate on `Version(trio.__version__) <
+  Version('X.Y.Z')`. When the gated version is
+  released, the patch can be DELETED entirely.
+
+- **`repro() -> None`** — minimal demonstration of the
+  bug. Used by the regression test suite to assert (a)
+  the upstream bug still exists, (b) our patch fixes
+  it. Should be tight enough that calling it post-
+  `apply()` returns cleanly within a few hundred
+  milliseconds — tests wrap it with a wall-clock cap.
+
+Each module's docstring MUST contain:
+
+- **Problem**: what trio does wrong + the trigger
+  conditions (e.g. "fork-spawn backend, peer-closed
+  socketpair, etc.")
+- **Fix**: the one-line (ideally) patch
+- **Repro**: the standalone snippet `repro()`
+  implements
+- **Upstream**: link to filed issue/PR (or
+  `TODO: file`)
+- **REMOVE WHEN**: `trio>=X.Y.Z` ships the upstream
+  fix
+
+## Adding a patch
+
+1. Create `_<topic>.py` with the `apply` /
+   `is_needed` / `repro` API.
+2. Register it in `__init__.py::_PATCHES`.
+3. Add a regression test in
+   `tests/trionics/test_patches.py` that uses
+   `repro()` to assert pre/post-patch behavior with a
+   wall-clock cap.
+4. File the upstream issue/PR. Add the link to your
+   module's `Upstream:` and `# REMOVE WHEN:` lines.
+
+## Removing a patch (when upstream releases the fix)
+
+1. Confirm the upstream-fixed `trio` version is the
+   minimum we depend on, OR keep the version-gate in
+   `is_needed()` if we still support older trio.
+2. If we've fully bumped past the broken versions:
+   - Delete `_<topic>.py`
+   - Remove the entry from `__init__.py::_PATCHES`
+   - Delete the corresponding test in
+     `tests/trionics/test_patches.py`
+   - Bump the conc-anal doc with a "FIXED" header
+
+## Calling
+
+```python
+from tractor.trionics.patches import apply_all
+apply_all()
+```
+
+Currently invoked from `tractor._child._actor_child_main`
+before `_trio_main` so every spawned subactor gets
+patched. The root actor's entry could opt in too if a
+patch turns out to bite the root (none do today).

tractor/trionics/patches/__init__.py

@@ -0,0 +1,84 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Defensive monkey-patches for `trio` internals.
+
+Every patch in this package fixes a bug in `trio` itself
+that we've encountered while running `tractor` — usually
+a fork-survival edge case that upstream `trio` hasn't
+filed/fixed yet. Each patch is:
+
+- **idempotent** — safe to call multiple times
+- **version-gated** — checks `trio.__version__` and skips
+  itself if upstream has shipped the fix
+- **scoped** — only modifies the specific trio internal
+  it's targeting; no broad side effects
+- **removable** — every patch carries a `# REMOVE WHEN:`
+  marker in its docstring pointing at the upstream PR
+  whose release allows us to drop it
+
+Add a new patch by:
+
+1. Create `tractor/trionics/patches/_<topic>.py` exposing
+   the `apply()` / `is_needed()` / `repro()` API
+   contract.
+2. Import it in this `__init__.py` and add an entry to
+   `_PATCHES`.
+3. Document upstream-fix-tracking in the module
+   docstring's `# REMOVE WHEN:` line.
+4. Add a regression test in
+   `tests/trionics/test_patches.py` that uses the
+   patch's `repro()` to assert the bug exists + the
+   patch fixes it.
+
+Calling `apply_all()` from a tractor entry point (e.g.
+`tractor._child._actor_child_main`) applies every
+registered patch + returns `{patch_name: applied?}` so
+callers can log/assert as needed.
+
+'''
+from typing import Callable
+
+from . import _wakeup_socketpair
+
+
+_PATCHES: list[tuple[str, Callable[[], bool]]] = [
+    (
+        'trio_wakeup_socketpair_drain_eof',
+        _wakeup_socketpair.apply,
+    ),
+]
+
+
+def apply_all() -> dict[str, bool]:
+    '''
+    Apply every registered patch. Idempotent — calling
+    twice is fine, second call's dict will be all
+    `False`.
+
+    Returns `{patch_name: applied?}`:
+
+    - `True` — patch was applied THIS call (inaugural
+      apply, or first-call-since-process-start).
+    - `False` — skipped (already applied OR upstream fix
+      detected via `is_needed() == False`).
+
+    '''
+    results: dict[str, bool] = {}
+    for name, applier in _PATCHES:
+        results[name] = applier()
+    return results

tractor/trionics/patches/_wakeup_socketpair.py

@@ -0,0 +1,171 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Patch `trio._core._wakeup_socketpair.WakeupSocketpair.drain()`
+to break on peer-closed EOF.
+
+Problem
+-------
+`drain()` loops on `self.wakeup_sock.recv(2**16)` and
+exits ONLY on `BlockingIOError` (buffer-empty on a
+non-blocking socket), NEVER on `recv() == b''`
+(peer-closed FIN). When the socketpair's write-end
+has been closed, `recv` returns 0 bytes each call →
+infinite C-level tight loop → 100% CPU, no Python
+checkpoints, no signal delivery, no progress.
+
+Most reliably triggered under fork-spawn backends —
+`os.fork()` + `_close_inherited_fds()` can leave a
+`WakeupSocketpair` instance whose `write_sock` was
+closed in the child (or whose peer-end is held by a
+process that has since exited).
+
+Repro
+-----
+```python
+from trio._core._wakeup_socketpair import WakeupSocketpair
+ws = WakeupSocketpair()
+ws.write_sock.close()
+ws.drain()  # spins forever pre-patch
+```
+
+Fix
+---
+One line: break the drain loop on `b''` EOF
+in addition to the existing `BlockingIOError` exit.
+
+```python
+def _safe_drain(self) -> None:
+    try:
+        while True:
+            data = self.wakeup_sock.recv(2**16)
+            if not data:  # ← peer-closed; nothing more to drain
+                return
+    except BlockingIOError:
+        pass
+```
+
+Upstream
+--------
+TODO: file at `python-trio/trio` — the standalone
+`repro()` below + this docstring is the issue body's
+evidence section.
+
+REMOVE WHEN: trio>=`<TBD>` ships the EOF-break in
+`_wakeup_socketpair.WakeupSocketpair.drain()`.
+
+See also
+--------
+- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
+- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
+  — sibling-bug analysis fixed by the same patch.
+
+'''
+from __future__ import annotations
+
+
+# Module-local sentinel — set True by `apply()` after the
+# first successful patch. Idempotency guard.
+_APPLIED: bool = False
+
+
+def is_needed() -> bool:
+    '''
+    True iff upstream `trio` is the broken version that
+    needs our patch.
+
+    Today: always True since no released `trio` has the
+    fix. When upstream lands it, gate on:
+
+    ```python
+    from packaging.version import Version
+    import trio
+    return Version(trio.__version__) < Version('<TBD>')
+    ```
+
+    '''
+    # TODO version-gate once upstream lands the fix.
+    return True
+
+
+def repro() -> None:
+    '''
+    Minimal hang demonstrator + regression test target.
+
+    Returns CLEANLY when `apply()` has been called
+    earlier in this process (the patched
+    `_safe_drain` breaks on EOF). Spins forever
+    UNPATCHED — caller should wrap with a wall-clock
+    cap (e.g. `signal.alarm(N)` or `trio.fail_after`)
+    to avoid hanging the test runner if regressing.
+
+    Used by `tests/trionics/test_patches.py` to assert
+    both:
+
+    1. The bug exists upstream (sanity check the
+       repro is real).
+    2. Our patch fixes it (post-`apply()` returns
+       cleanly).
+
+    '''
+    from trio._core._wakeup_socketpair import (
+        WakeupSocketpair,
+    )
+    ws = WakeupSocketpair()
+    ws.write_sock.close()
+    ws.drain()  # ← targeted operation
+
+
+def apply() -> bool:
+    '''
+    Apply the EOF-break patch to
+    `WakeupSocketpair.drain`. Idempotent + version-
+    gated.
+
+    Returns:
+
+    - `True` if patched THIS call (inaugural apply).
+    - `False` if skipped (already applied this process,
+      OR `is_needed() == False` because upstream fixed
+      it).
+
+    '''
+    global _APPLIED
+    if _APPLIED or not is_needed():
+        return False
+
+    from trio._core._wakeup_socketpair import (
+        WakeupSocketpair as _WSP,
+    )
+
+    def _safe_drain(self) -> None:
+        try:
+            while True:
+                data = self.wakeup_sock.recv(2**16)
+                # XXX patch — break on EOF instead of
+                # spinning. Upstream trio's `drain()`
+                # only handles the `BlockingIOError`
+                # (buffer-empty) case; missed the
+                # peer-closed (`recv == b''`) case.
+                if not data:
+                    return
+        except BlockingIOError:
+            pass
+
+    _WSP.drain = _safe_drain
+    _APPLIED = True
+    return True

uv.lock

@@ -573,6 +573,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/8b/5a/ba30a81239b909821b3153e303e7def45178bf353da4f72380e6c5e8793b/pytest-9.1.0-py3-none-any.whl", hash = "sha256:8ebb0e7888bdf2bdfc602ec51f8f62d50200af37356c74e503c79a94f5c81f32", size = 386453, upload-time = "2026-06-13T18:52:44.045Z" },
 ]
 
+[[package]]
+name = "pytest-timeout"
+version = "2.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pytest" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/82/4c9ecabab13363e72d880f2fb504c5f750433b2b6f16e99f4ec21ada284c/pytest_timeout-2.4.0.tar.gz", hash = "sha256:7e68e90b01f9eff71332b25001f85c75495fc4e3a836701876183c4bcfd0540a", size = 17973, upload-time = "2025-05-05T19:44:34.99Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/fa/b6/3127540ecdf1464a00e5a01ee60a1b09175f6913f0644ac748494d9c4b21/pytest_timeout-2.4.0-py3-none-any.whl", hash = "sha256:c42667e5cdadb151aeb5b26d114aff6bdf5a907f176a007a30b940d3d865b5c2", size = 14382, upload-time = "2025-05-05T19:44:33.502Z" },
+]
+
 [[package]]
 name = "python-baseconv"
 version = "1.2.2"
@@ -605,6 +617,54 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/9e/6a/40fee331a52339926a92e17ae748827270b288a35ef4a15c9c8f2ec54715/ruff-0.14.14-py3-none-win_arm64.whl", hash = "sha256:56e6981a98b13a32236a72a8da421d7839221fa308b223b9283312312e5ac76c", size = 10920448, upload-time = "2026-01-22T22:30:15.417Z" },
 ]
 
+[[package]]
+name = "setproctitle"
+version = "1.3.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8d/48/49393a96a2eef1ab418b17475fb92b8fcfad83d099e678751b05472e69de/setproctitle-1.3.7.tar.gz", hash = "sha256:bc2bc917691c1537d5b9bca1468437176809c7e11e5694ca79a9ca12345dcb9e", size = 27002, upload-time = "2025-09-05T12:51:25.278Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/5d/2f/fcedcade3b307a391b6e17c774c6261a7166aed641aee00ed2aad96c63ce/setproctitle-1.3.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c3736b2a423146b5e62230502e47e08e68282ff3b69bcfe08a322bee73407922", size = 18047, upload-time = "2025-09-05T12:49:50.271Z" },
+    { url = "https://files.pythonhosted.org/packages/23/ae/afc141ca9631350d0a80b8f287aac79a76f26b6af28fd8bf92dae70dc2c5/setproctitle-1.3.7-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3384e682b158d569e85a51cfbde2afd1ab57ecf93ea6651fe198d0ba451196ee", size = 13073, upload-time = "2025-09-05T12:49:51.46Z" },
+    { url = "https://files.pythonhosted.org/packages/87/ed/0a4f00315bc02510395b95eec3d4aa77c07192ee79f0baae77ea7b9603d8/setproctitle-1.3.7-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0564a936ea687cd24dffcea35903e2a20962aa6ac20e61dd3a207652401492dd", size = 33284, upload-time = "2025-09-05T12:49:52.741Z" },
+    { url = "https://files.pythonhosted.org/packages/fc/e4/adf3c4c0a2173cb7920dc9df710bcc67e9bcdbf377e243b7a962dc31a51a/setproctitle-1.3.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5d1cb3f81531f0eb40e13246b679a1bdb58762b170303463cb06ecc296f26d0", size = 34104, upload-time = "2025-09-05T12:49:54.416Z" },
+    { url = "https://files.pythonhosted.org/packages/52/4f/6daf66394152756664257180439d37047aa9a1cfaa5e4f5ed35e93d1dc06/setproctitle-1.3.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a7d159e7345f343b44330cbba9194169b8590cb13dae940da47aa36a72aa9929", size = 35982, upload-time = "2025-09-05T12:49:56.295Z" },
+    { url = "https://files.pythonhosted.org/packages/1b/62/f2c0595403cf915db031f346b0e3b2c0096050e90e0be658a64f44f4278a/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0b5074649797fd07c72ca1f6bff0406f4a42e1194faac03ecaab765ce605866f", size = 33150, upload-time = "2025-09-05T12:49:58.025Z" },
+    { url = "https://files.pythonhosted.org/packages/a0/29/10dd41cde849fb2f9b626c846b7ea30c99c81a18a5037a45cc4ba33c19a7/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:61e96febced3f61b766115381d97a21a6265a0f29188a791f6df7ed777aef698", size = 34463, upload-time = "2025-09-05T12:49:59.424Z" },
+    { url = "https://files.pythonhosted.org/packages/71/3c/cedd8eccfaf15fb73a2c20525b68c9477518917c9437737fa0fda91e378f/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:047138279f9463f06b858e579cc79580fbf7a04554d24e6bddf8fe5dddbe3d4c", size = 32848, upload-time = "2025-09-05T12:50:01.107Z" },
+    { url = "https://files.pythonhosted.org/packages/d1/3e/0a0e27d1c9926fecccfd1f91796c244416c70bf6bca448d988638faea81d/setproctitle-1.3.7-cp313-cp313-win32.whl", hash = "sha256:7f47accafac7fe6535ba8ba9efd59df9d84a6214565108d0ebb1199119c9cbbd", size = 12544, upload-time = "2025-09-05T12:50:15.81Z" },
+    { url = "https://files.pythonhosted.org/packages/36/1b/6bf4cb7acbbd5c846ede1c3f4d6b4ee52744d402e43546826da065ff2ab7/setproctitle-1.3.7-cp313-cp313-win_amd64.whl", hash = "sha256:fe5ca35aeec6dc50cabab9bf2d12fbc9067eede7ff4fe92b8f5b99d92e21263f", size = 13235, upload-time = "2025-09-05T12:50:16.89Z" },
+    { url = "https://files.pythonhosted.org/packages/e6/a4/d588d3497d4714750e3eaf269e9e8985449203d82b16b933c39bd3fc52a1/setproctitle-1.3.7-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:10e92915c4b3086b1586933a36faf4f92f903c5554f3c34102d18c7d3f5378e9", size = 18058, upload-time = "2025-09-05T12:50:02.501Z" },
+    { url = "https://files.pythonhosted.org/packages/05/77/7637f7682322a7244e07c373881c7e982567e2cb1dd2f31bd31481e45500/setproctitle-1.3.7-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:de879e9c2eab637f34b1a14c4da1e030c12658cdc69ee1b3e5be81b380163ce5", size = 13072, upload-time = "2025-09-05T12:50:03.601Z" },
+    { url = "https://files.pythonhosted.org/packages/52/09/f366eca0973cfbac1470068d1313fa3fe3de4a594683385204ec7f1c4101/setproctitle-1.3.7-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:c18246d88e227a5b16248687514f95642505000442165f4b7db354d39d0e4c29", size = 34490, upload-time = "2025-09-05T12:50:04.948Z" },
+    { url = "https://files.pythonhosted.org/packages/71/36/611fc2ed149fdea17c3677e1d0df30d8186eef9562acc248682b91312706/setproctitle-1.3.7-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7081f193dab22df2c36f9fc6d113f3793f83c27891af8fe30c64d89d9a37e152", size = 35267, upload-time = "2025-09-05T12:50:06.015Z" },
+    { url = "https://files.pythonhosted.org/packages/88/a4/64e77d0671446bd5a5554387b69e1efd915274686844bea733714c828813/setproctitle-1.3.7-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9cc9b901ce129350637426a89cfd650066a4adc6899e47822e2478a74023ff7c", size = 37376, upload-time = "2025-09-05T12:50:07.484Z" },
+    { url = "https://files.pythonhosted.org/packages/89/bc/ad9c664fe524fb4a4b2d3663661a5c63453ce851736171e454fa2cdec35c/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:80e177eff2d1ec172188d0d7fd9694f8e43d3aab76a6f5f929bee7bf7894e98b", size = 33963, upload-time = "2025-09-05T12:50:09.056Z" },
+    { url = "https://files.pythonhosted.org/packages/ab/01/a36de7caf2d90c4c28678da1466b47495cbbad43badb4e982d8db8167ed4/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:23e520776c445478a67ee71b2a3c1ffdafbe1f9f677239e03d7e2cc635954e18", size = 35550, upload-time = "2025-09-05T12:50:10.791Z" },
+    { url = "https://files.pythonhosted.org/packages/dd/68/17e8aea0ed5ebc17fbf03ed2562bfab277c280e3625850c38d92a7b5fcd9/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5fa1953126a3b9bd47049d58c51b9dac72e78ed120459bd3aceb1bacee72357c", size = 33727, upload-time = "2025-09-05T12:50:12.032Z" },
+    { url = "https://files.pythonhosted.org/packages/b2/33/90a3bf43fe3a2242b4618aa799c672270250b5780667898f30663fd94993/setproctitle-1.3.7-cp313-cp313t-win32.whl", hash = "sha256:4a5e212bf438a4dbeece763f4962ad472c6008ff6702e230b4f16a037e2f6f29", size = 12549, upload-time = "2025-09-05T12:50:13.074Z" },
+    { url = "https://files.pythonhosted.org/packages/0b/0e/50d1f07f3032e1f23d814ad6462bc0a138f369967c72494286b8a5228e40/setproctitle-1.3.7-cp313-cp313t-win_amd64.whl", hash = "sha256:cf2727b733e90b4f874bac53e3092aa0413fe1ea6d4f153f01207e6ce65034d9", size = 13243, upload-time = "2025-09-05T12:50:14.146Z" },
+    { url = "https://files.pythonhosted.org/packages/89/c7/43ac3a98414f91d1b86a276bc2f799ad0b4b010e08497a95750d5bc42803/setproctitle-1.3.7-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:80c36c6a87ff72eabf621d0c79b66f3bdd0ecc79e873c1e9f0651ee8bf215c63", size = 18052, upload-time = "2025-09-05T12:50:17.928Z" },
+    { url = "https://files.pythonhosted.org/packages/cd/2c/dc258600a25e1a1f04948073826bebc55e18dbd99dc65a576277a82146fa/setproctitle-1.3.7-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b53602371a52b91c80aaf578b5ada29d311d12b8a69c0c17fbc35b76a1fd4f2e", size = 13071, upload-time = "2025-09-05T12:50:19.061Z" },
+    { url = "https://files.pythonhosted.org/packages/ab/26/8e3bb082992f19823d831f3d62a89409deb6092e72fc6940962983ffc94f/setproctitle-1.3.7-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fcb966a6c57cf07cc9448321a08f3be6b11b7635be502669bc1d8745115d7e7f", size = 33180, upload-time = "2025-09-05T12:50:20.395Z" },
+    { url = "https://files.pythonhosted.org/packages/f1/af/ae692a20276d1159dd0cf77b0bcf92cbb954b965655eb4a69672099bb214/setproctitle-1.3.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:46178672599b940368d769474fe13ecef1b587d58bb438ea72b9987f74c56ea5", size = 34043, upload-time = "2025-09-05T12:50:22.454Z" },
+    { url = "https://files.pythonhosted.org/packages/34/b2/6a092076324dd4dac1a6d38482bedebbff5cf34ef29f58585ec76e47bc9d/setproctitle-1.3.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7f9e9e3ff135cbcc3edd2f4cf29b139f4aca040d931573102742db70ff428c17", size = 35892, upload-time = "2025-09-05T12:50:23.937Z" },
+    { url = "https://files.pythonhosted.org/packages/1c/1a/8836b9f28cee32859ac36c3df85aa03e1ff4598d23ea17ca2e96b5845a8f/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:14c7eba8d90c93b0e79c01f0bd92a37b61983c27d6d7d5a3b5defd599113d60e", size = 32898, upload-time = "2025-09-05T12:50:25.617Z" },
+    { url = "https://files.pythonhosted.org/packages/ef/22/8fabdc24baf42defb599714799d8445fe3ae987ec425a26ec8e80ea38f8e/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:9e64e98077fb30b6cf98073d6c439cd91deb8ebbf8fc62d9dbf52bd38b0c6ac0", size = 34308, upload-time = "2025-09-05T12:50:26.827Z" },
+    { url = "https://files.pythonhosted.org/packages/15/1b/b9bee9de6c8cdcb3b3a6cb0b3e773afdb86bbbc1665a3bfa424a4294fda2/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b91387cc0f02a00ac95dcd93f066242d3cca10ff9e6153de7ee07069c6f0f7c8", size = 32536, upload-time = "2025-09-05T12:50:28.5Z" },
+    { url = "https://files.pythonhosted.org/packages/37/0c/75e5f2685a5e3eda0b39a8b158d6d8895d6daf3ba86dec9e3ba021510272/setproctitle-1.3.7-cp314-cp314-win32.whl", hash = "sha256:52b054a61c99d1b72fba58b7f5486e04b20fefc6961cd76722b424c187f362ed", size = 12731, upload-time = "2025-09-05T12:50:43.955Z" },
+    { url = "https://files.pythonhosted.org/packages/d2/ae/acddbce90d1361e1786e1fb421bc25baeb0c22ef244ee5d0176511769ec8/setproctitle-1.3.7-cp314-cp314-win_amd64.whl", hash = "sha256:5818e4080ac04da1851b3ec71e8a0f64e3748bf9849045180566d8b736702416", size = 13464, upload-time = "2025-09-05T12:50:45.057Z" },
+    { url = "https://files.pythonhosted.org/packages/01/6d/20886c8ff2e6d85e3cabadab6aab9bb90acaf1a5cfcb04d633f8d61b2626/setproctitle-1.3.7-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:6fc87caf9e323ac426910306c3e5d3205cd9f8dcac06d233fcafe9337f0928a3", size = 18062, upload-time = "2025-09-05T12:50:29.78Z" },
+    { url = "https://files.pythonhosted.org/packages/9a/60/26dfc5f198715f1343b95c2f7a1c16ae9ffa45bd89ffd45a60ed258d24ea/setproctitle-1.3.7-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6134c63853d87a4897ba7d5cc0e16abfa687f6c66fc09f262bb70d67718f2309", size = 13075, upload-time = "2025-09-05T12:50:31.604Z" },
+    { url = "https://files.pythonhosted.org/packages/21/9c/980b01f50d51345dd513047e3ba9e96468134b9181319093e61db1c47188/setproctitle-1.3.7-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1403d2abfd32790b6369916e2313dffbe87d6b11dca5bbd898981bcde48e7a2b", size = 34744, upload-time = "2025-09-05T12:50:32.777Z" },
+    { url = "https://files.pythonhosted.org/packages/86/b4/82cd0c86e6d1c4538e1a7eb908c7517721513b801dff4ba3f98ef816a240/setproctitle-1.3.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e7c5bfe4228ea22373e3025965d1a4116097e555ee3436044f5c954a5e63ac45", size = 35589, upload-time = "2025-09-05T12:50:34.13Z" },
+    { url = "https://files.pythonhosted.org/packages/8a/4f/9f6b2a7417fd45673037554021c888b31247f7594ff4bd2239918c5cd6d0/setproctitle-1.3.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:585edf25e54e21a94ccb0fe81ad32b9196b69ebc4fc25f81da81fb8a50cca9e4", size = 37698, upload-time = "2025-09-05T12:50:35.524Z" },
+    { url = "https://files.pythonhosted.org/packages/20/92/927b7d4744aac214d149c892cb5fa6dc6f49cfa040cb2b0a844acd63dcaf/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:96c38cdeef9036eb2724c2210e8d0b93224e709af68c435d46a4733a3675fee1", size = 34201, upload-time = "2025-09-05T12:50:36.697Z" },
+    { url = "https://files.pythonhosted.org/packages/0a/0c/fd4901db5ba4b9d9013e62f61d9c18d52290497f956745cd3e91b0d80f90/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:45e3ef48350abb49cf937d0a8ba15e42cee1e5ae13ca41a77c66d1abc27a5070", size = 35801, upload-time = "2025-09-05T12:50:38.314Z" },
+    { url = "https://files.pythonhosted.org/packages/e7/e3/54b496ac724e60e61cc3447f02690105901ca6d90da0377dffe49ff99fc7/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1fae595d032b30dab4d659bece20debd202229fce12b55abab978b7f30783d73", size = 33958, upload-time = "2025-09-05T12:50:39.841Z" },
+    { url = "https://files.pythonhosted.org/packages/ea/a8/c84bb045ebf8c6fdc7f7532319e86f8380d14bbd3084e6348df56bdfe6fd/setproctitle-1.3.7-cp314-cp314t-win32.whl", hash = "sha256:02432f26f5d1329ab22279ff863c83589894977063f59e6c4b4845804a08f8c2", size = 12745, upload-time = "2025-09-05T12:50:41.377Z" },
+    { url = "https://files.pythonhosted.org/packages/08/b6/3a5a4f9952972791a9114ac01dfc123f0df79903577a3e0a7a404a695586/setproctitle-1.3.7-cp314-cp314t-win_amd64.whl", hash = "sha256:cbc388e3d86da1f766d8fc2e12682e446064c01cea9f88a88647cfe7c011de6a", size = 13469, upload-time = "2025-09-05T12:50:42.67Z" },
+]
+
 [[package]]
 name = "six"
 version = "1.17.0"
@@ -664,6 +724,7 @@ dependencies = [
     { name = "multiaddr" },
     { name = "pdbp" },
     { name = "platformdirs" },
+    { name = "setproctitle" },
     { name = "tricycle" },
     { name = "trio" },
     { name = "wrapt" },
@@ -671,11 +732,13 @@ dependencies = [
 
 [package.dev-dependencies]
 dev = [
+    { name = "greenback", marker = "python_full_version < '3.14'" },
     { name = "pexpect" },
     { name = "prompt-toolkit" },
     { name = "psutil" },
     { name = "pyperclip" },
     { name = "pytest" },
+    { name = "pytest-timeout" },
     { name = "stackscope" },
     { name = "typing-extensions" },
     { name = "xonsh" },
@@ -704,7 +767,9 @@ sync-pause = [
 ]
 testing = [
     { name = "pexpect" },
+    { name = "psutil" },
     { name = "pytest" },
+    { name = "pytest-timeout" },
 ]
 
 [package.metadata]
@@ -715,6 +780,7 @@ requires-dist = [
     { name = "multiaddr", specifier = ">=0.2.0" },
     { name = "pdbp", specifier = ">=1.8.2,<2" },
     { name = "platformdirs", specifier = ">=4.4.0" },
+    { name = "setproctitle", specifier = ">=1.3,<2" },
     { name = "tricycle", specifier = ">=0.4.1,<0.5" },
     { name = "trio", specifier = ">0.27" },
     { name = "wrapt", specifier = ">=1.16.0,<2" },
@@ -722,11 +788,13 @@ requires-dist = [
 
 [package.metadata.requires-dev]
 dev = [
+    { name = "greenback", marker = "python_full_version == '3.13.*'", specifier = ">=1.2.1,<2" },
     { name = "pexpect", specifier = ">=4.9.0,<5" },
     { name = "prompt-toolkit", specifier = ">=3.0.50" },
     { name = "psutil", specifier = ">=7.0.0" },
     { name = "pyperclip", specifier = ">=1.9.0" },
     { name = "pytest", specifier = ">=9.0.3" },
+    { name = "pytest-timeout", specifier = ">=2.3" },
     { name = "stackscope", specifier = ">=0.2.2,<0.3" },
     { name = "typing-extensions", specifier = ">=4.14.1" },
     { name = "xonsh", specifier = ">=0.23.0" },
@@ -747,7 +815,9 @@ subints = [{ name = "msgspec", marker = "python_full_version >= '3.14'", specifi
 sync-pause = [{ name = "greenback", marker = "python_full_version == '3.13.*'", specifier = ">=1.2.1,<2" }]
 testing = [
     { name = "pexpect", specifier = ">=4.9.0,<5" },
+    { name = "psutil", specifier = ">=7.0.0" },
     { name = "pytest", specifier = ">=9.0.3" },
+    { name = "pytest-timeout", specifier = ">=2.3" },
 ]
 
 [[package]]

xontrib/tractor_diag.xsh

@@ -0,0 +1,586 @@
+"""
+`xontrib_tractor_diag`: pytest/tractor diagnostic aliases.
+
+All aliases live under the `acli.` namespace so xonsh's
+prefix-completion treats them as a sub-cmd group — type
+`acli.<TAB>` to see the full set.
+
+Provides:
+  - `acli.ptree <pid|pgrep-pat>`        psutil-backed proc tree,
+                                        live + zombies split.
+  - `acli.hung_dump <pid|pat> [...]`    kernel `wchan`/`stack` +
+                                        `py-spy dump` (incl `--locals`)
+                                        for each pid in tree.
+  - `acli.bindspace_scan [<name>|<dir>]` find orphaned tractor UDS
+                                        sock files (no live owner pid).
+                                        bare name -> `$XDG_RUNTIME_DIR/<name>`
+                                        (e.g. `piker`, `tractor`);
+                                        path -> use as-is.
+                                        default: `$XDG_RUNTIME_DIR/tractor`.
+  - `acli.dump_all <pid> [--out-dir]    full snapshot bundle —
+                          [--label]`    ptree + hung_dump + bindspace
+                                        written to a timestamped dir
+                                        for sharing / AI introspection.
+  - `acli.reap [opts]`                  SC-polite zombie-subactor
+                                        reaper + optional `/dev/shm/`
+                                        + UDS sock-file sweeps.
+                                        alias for `scripts/tractor-reap`.
+  - `acli.watch [-n SEC] <alias-name>   run a callable alias in
+                        [alias-args]`   an alt-screen loop with
+                                        flicker-free repaint
+                                        (cursor-home + per-line
+                                        EL + post-draw erase-down).
+
+Loading from repo root:
+  xontrib load -p ./xontrib tractor_diag
+
+Or source directly:
+  source ./xontrib/tractor_diag.xsh
+
+Pipe-to-paste idiom (xonsh):
+  acli.hung_dump pytest |t /tmp/hung.log
+
+The diagnostic core lives in `tractor._testing.trace` so it
+can also be invoked from inside pytest tests (e.g. via
+`fail_after_w_trace` / `afk_alarm_w_trace` capture-on-hang
+helpers) — these aliases are just thin terminal wrappers.
+
+Requires `psutil` for full functionality (`ptree` and the
+`hung_dump` tree-walk). Falls back to `pgrep -P` recursion if
+missing.
+
+"""
+import os
+import sys
+import signal
+import time
+from typing import (
+    Callable,
+)
+
+
+from pathlib import Path
+
+from tractor._testing.trace import (
+    dump_all as _dump_all,
+    dump_hung_state,
+    dump_proc_tree,
+    resolve_pids,
+    scan_bindspace,
+)
+
+@aliases.unthreadable
+def watch(
+    args: list[str],
+) -> int:
+    '''
+    A per-term optimized `watch`-like alias for xonsh
+    that runs an arbitrary callable alias in a loop
+    inside the alt-screen buffer. Ctrl-C returns to a
+    pristine shell, SIGWINCH triggers a full redraw,
+    and the per-frame draw uses cursor-home + per-line
+    EL + post-draw erase-down so the loop is flicker-
+    free even when individual lines shrink or grow
+    between frames.
+
+    usage: acli.watch [-n SEC] <alias-name>
+                      [alias-args]...
+
+    Examples:
+
+      acli.watch acli.ptree pytest
+      acli.watch -n 1.0 acli.bindspace_scan piker
+      acli.watch acli.hung_dump pytest
+
+    Only callable aliases (Python functions registered
+    in `aliases`) are supported. Subprocess-style
+    aliases raise an error — wrap them in a thin
+    callable if you need watching.
+
+    Output capture: the watched alias's stdout is
+    redirected into a `StringIO` per frame so we can
+    post-process it (insert `\033[K` before each `\n`).
+    Aliases that write directly to `sys.stdout.buffer`
+    or `os.write(1, ...)` bypass capture; for those the
+    EL-fix won't apply but the loop still functions.
+
+    '''
+    import argparse, io
+    from contextlib import redirect_stdout
+
+    parser = argparse.ArgumentParser(
+        prog='acli.watch',
+        description=watch.__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        '-n', '--interval',
+        type=float,
+        default=0.3,
+        help='poll interval in seconds (default: 0.3)',
+    )
+    parser.add_argument(
+        'alias',
+        help='name of a registered xonsh callable alias',
+    )
+    parser.add_argument(
+        'alias_args',
+        nargs=argparse.REMAINDER,
+        help='args forwarded to the watched alias',
+    )
+
+    try:
+        ns = parser.parse_args(args)
+    except SystemExit as se:
+        return int(se.code) if se.code is not None else 0
+
+    raw = aliases.get(ns.alias)
+    if raw is None:
+        print(
+            f'[acli.watch] no such alias: {ns.alias!r}'
+        )
+        return 1
+
+    # xonsh stores callable aliases as a bare callable
+    # OR wraps them in `[fn, *preset_args]` (depending
+    # on registration path / version). Unwrap both.
+    fn: Callable|None = None
+    preset_args: list = []
+    if callable(raw):
+        fn = raw
+    elif (
+        isinstance(raw, list)
+        and raw
+        and callable(raw[0])
+    ):
+        fn = raw[0]
+        preset_args = list(raw[1:])
+
+    if fn is None:
+        kind: str = type(raw).__name__
+        print(
+            f'[acli.watch] alias {ns.alias!r} is not a '
+            f'callable alias (got {kind}); '
+            f'subprocess-style aliases not supported'
+        )
+        return 1
+
+    _FD: int = sys.stdout.fileno()
+    need_full_clear: bool = False
+
+    def _on_winch(signum, frame):
+        nonlocal need_full_clear
+        need_full_clear = True
+
+    prev_winch = signal.signal(
+        signal.SIGWINCH,
+        _on_winch,
+    )
+    prev_sigint = signal.signal(
+        signal.SIGINT,
+        signal.default_int_handler,
+    )
+
+    os.write(_FD, b'\033[?1049h\033[?25l')
+    try:
+        while True:
+            buf = io.StringIO()
+            with redirect_stdout(buf):
+                fn(preset_args + ns.alias_args)
+
+            if need_full_clear:
+                os.write(_FD, b'\033[H\033[2J')
+                need_full_clear = False
+            else:
+                os.write(_FD, b'\033[H')
+
+            # `\033[K` (EL) before each newline erases
+            # any stale tail chars left by a longer
+            # prior-frame version of the same line.
+            text: str = buf.getvalue()
+            painted: bytes = (
+                text.replace('\n', '\033[K\n').encode()
+            )
+            os.write(_FD, painted)
+            os.write(_FD, b'\033[J')
+            time.sleep(ns.interval)
+    except KeyboardInterrupt:
+        pass
+    finally:
+        os.write(_FD, b'\033[?25h\033[?1049l')
+        signal.signal(signal.SIGWINCH, prev_winch)
+        signal.signal(signal.SIGINT, prev_sigint)
+
+    return 0
+
+
+# --- ptree ----------------------------------------------------
+
+def _ptree(
+    args: list[str],
+):
+    '''
+    psutil-backed proc tree; per-proc classification into
+    severity-ordered buckets so leaked / defunct procs
+    don't hide in the noise of normal `live` rows.
+
+    usage: acli.ptree [--tree|-t] <pid|pgrep-pattern> [...]
+
+    See `tractor._testing.trace.dump_proc_tree()` for the
+    bucket semantics + classification details.
+
+    To watch this live with flicker-free repaint
+    (alt-screen, per-line EL, SIGWINCH-aware):
+
+    .. code-block:: xonsh
+
+        acli.watch acli.ptree pytest
+
+    '''
+    flag_tree: bool = False
+    pos_args: list = []
+    for a in args:
+        if a in ('--tree', '-t'):
+            flag_tree = True
+        else:
+            pos_args.append(a)
+
+    if not pos_args:
+        print('usage: acli.ptree [--tree|-t] <pid|pgrep-pattern> [...]')
+        return 1
+
+    roots: list = []
+    for a in pos_args:
+        roots.extend(resolve_pids(a))
+    roots = sorted(set(roots))
+    if not roots:
+        print(f'(no procs match: {pos_args})')
+        return 1
+
+    print(dump_proc_tree(roots, flag_tree=flag_tree), end='')
+
+
+# --- hung-dump -----------------------------------------------
+
+def _hung_dump(args):
+    '''
+    kernel + python state for a hung pytest/tractor tree.
+    walks all descendants of each `<pid|pgrep-pat>` arg.
+
+    usage: acli.hung_dump <pid|pgrep-pattern> [...]
+
+    note: `/proc/<pid>/stack` and `py-spy dump` typically
+    require CAP_SYS_PTRACE — invoked via `sudo -n`. If sudo
+    isn't cached this alias prompts (via `sudo -v`); for the
+    non-interactive equivalent see
+    `tractor._testing.trace.dump_hung_state(allow_sudo_prompt=False)`.
+
+    '''
+    if not args:
+        print('usage: acli.hung_dump <pid|pgrep-pattern> [...]')
+        return 1
+
+    roots: list = []
+    for a in args:
+        roots.extend(resolve_pids(a))
+    roots = sorted(set(roots))
+    if not roots:
+        print(f'(no procs match: {args})')
+        return 1
+
+    print(
+        dump_hung_state(roots, allow_sudo_prompt=True),
+        end='',
+    )
+
+
+# --- bindspace-scan ------------------------------------------
+
+def _bindspace_scan(args):
+    '''
+    Scan a tractor UDS bindspace dir for orphan sock files.
+
+    usage: acli.bindspace_scan [<name>|<dir>]
+
+    See `tractor._testing.trace.scan_bindspace()` for full arg
+    semantics + output-bucket details.
+
+    '''
+    arg: str | None = args[0] if args else None
+    print(scan_bindspace(arg), end='')
+
+
+# --- dump-all (snapshot bundle) ------------------------------
+
+def _dump_all_alias(args):
+    '''
+    Capture a full diag snapshot bundle for a hung proc-tree
+    into a timestamped directory for offline / AI inspection.
+
+    usage: acli.dump_all <pid|pgrep-pat>
+                        [--label <label>]
+                        [--out-dir <path>]
+
+    Writes:
+      <out_dir>/<label>__<ts>/{trace.txt, bindspace.txt, meta.json}
+
+    Defaults:
+      --label   = `manual`
+      --out-dir = `$XDG_CACHE_HOME/tractor/hung-dumps/`
+                  (fallback `~/.cache/tractor/hung-dumps/`)
+
+    '''
+    import argparse
+    parser = argparse.ArgumentParser(
+        prog='acli.dump_all',
+        description=_dump_all_alias.__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        'target',
+        help='pid or pgrep -f pattern',
+    )
+    parser.add_argument(
+        '--label', '-l',
+        default='manual',
+        help='snapshot dir label prefix (default: `manual`)',
+    )
+    parser.add_argument(
+        '--out-dir', '-o',
+        type=Path,
+        default=None,
+        help='snapshot root dir (default: '
+             '$XDG_CACHE_HOME/tractor/hung-dumps/)',
+    )
+    try:
+        ns = parser.parse_args(args)
+    except SystemExit as se:
+        return int(se.code) if se.code is not None else 0
+
+    pids: list = resolve_pids(ns.target)
+    if not pids:
+        print(f'(no procs match: {ns.target})')
+        return 1
+
+    # snapshot scoped to ONE root — pick the first matched
+    # pid. Multi-root snapshots can be done by invoking
+    # `acli.dump_all <pid>` per root.
+    root_pid: int = pids[0]
+    if len(pids) > 1:
+        print(
+            f'[acli.dump_all] {len(pids)} pids matched '
+            f'{ns.target!r}; snapshotting tree from {root_pid} '
+            f'(re-run per-pid for others: {pids[1:]})'
+        )
+
+    dump_dir = _dump_all(
+        root_pid,
+        out_dir=ns.out_dir,
+        label=ns.label,
+        allow_sudo_prompt=True,  # CLI: ok to prompt
+    )
+    print(f'[acli.dump_all] snapshot written to: {dump_dir}')
+
+
+# --- acli.reap ------------------------------------------------
+
+def _tractor_reap(args):
+    '''
+    SC-polite zombie-subactor reaper + optional `/dev/shm/`
+    orphan-segment sweep + optional UDS sock-file sweep.
+
+    usage: acli.reap [-h] [--parent PID] [--grace SEC]
+                    [--dry-run] [--shm | --shm-only]
+                    [--uds | --uds-only]
+
+    phases (run in order when enabled):
+
+      1. process reap — finds tractor subactor procs left
+         alive after a `pytest`/app run that failed to fully
+         cancel its tree. Default = orphan-mode (PPid==1
+         init-reparented procs whose cwd matches repo root
+         AND cmdline contains `python`). With `--parent`,
+         scopes to descendants of a specific live PID.
+         SIGINT first, then SIGKILL after `--grace` (default
+         3.0s).
+      2. shm sweep (`--shm`/`--shm-only`) — unlinks
+         `/dev/shm/<file>` entries owned by the current uid
+         that no live process has open. Needed because
+         `tractor` disables `mp.resource_tracker`.
+      3. UDS sweep (`--uds`/`--uds-only`) — unlinks
+         `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock`
+         files whose binder pid is dead (or the `1616`
+         registry sentinel). See issue #452.
+
+    Mirrors `scripts/tractor-reap` (use `-n`/`--dry-run`
+    first to see what would be touched).
+
+    '''
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        prog='acli.reap',
+        description=_tractor_reap.__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        '--parent', '-p',
+        type=int,
+        default=None,
+        help='descendant-mode: reap procs with PPid==<pid>',
+    )
+    parser.add_argument(
+        '--grace', '-g',
+        type=float,
+        default=3.0,
+        help='SIGINT grace window in seconds (default 3.0)',
+    )
+    parser.add_argument(
+        '--dry-run', '-n',
+        action='store_true',
+        help='list matched pids/paths but do not signal/unlink',
+    )
+    parser.add_argument(
+        '--shm',
+        action='store_true',
+        help='also unlink orphaned /dev/shm segments',
+    )
+    parser.add_argument(
+        '--shm-only',
+        action='store_true',
+        help='skip process reap; only do the shm sweep',
+    )
+    parser.add_argument(
+        '--uds',
+        action='store_true',
+        help='also unlink orphaned UDS sock-files',
+    )
+    parser.add_argument(
+        '--uds-only',
+        action='store_true',
+        help='skip process reap + shm; only do the UDS sweep',
+    )
+
+    try:
+        ns = parser.parse_args(args)
+    except SystemExit as se:
+        # `argparse` raises SystemExit on `-h`/bad-args; let
+        # xonsh treat it as a normal alias return code.
+        return int(se.code) if se.code is not None else 0
+
+    skip_proc_reap: bool = (
+        ns.shm_only
+        or
+        ns.uds_only
+    )
+
+    # `tractor` is assumed to be importable in the xonsh env
+    # this xontrib was sourced into (a venv with the package
+    # installed). The standalone `scripts/tractor-reap` does
+    # `git rev-parse --show-toplevel` + `sys.path.insert` for
+    # cold-shell usability — that overhead is unnecessary
+    # here since we're already inside the project's venv.
+    from tractor._testing._reap import (
+        find_descendants,
+        find_orphans,
+        find_orphaned_shm,
+        find_orphaned_uds,
+        reap,
+        reap_shm,
+        reap_uds,
+    )
+
+    rc: int = 0
+
+    # phase 1: process reap (skipped under `--*-only`)
+    if not skip_proc_reap:
+        if ns.parent is not None:
+            pids: list = find_descendants(ns.parent)
+            mode: str = f'descendants of PPid={ns.parent}'
+        else:
+            pids = find_orphans()
+            mode = (
+                'orphans (PPid==1, intrinsic '
+                'cmdline/comm match — `tractor[…]` or '
+                '`tractor._child`)'
+            )
+
+        if not pids:
+            print(f'[acli.reap] no {mode} to reap')
+        elif ns.dry_run:
+            print(
+                f'[acli.reap] dry-run — {mode}:\n  {pids}'
+            )
+        else:
+            _, survivors = reap(pids, grace=ns.grace)
+            if survivors:
+                rc = 1
+
+    # phase 2: shm sweep (opt-in)
+    if ns.shm or ns.shm_only:
+        leaked: list = find_orphaned_shm()
+        if not leaked:
+            print(
+                '[acli.reap] no orphaned /dev/shm '
+                'segments to sweep'
+            )
+        elif ns.dry_run:
+            print(
+                f'[acli.reap] dry-run — {len(leaked)} '
+                f'orphaned shm segment(s):\n  {leaked}'
+            )
+        else:
+            _, errors = reap_shm(leaked)
+            if errors:
+                rc = 1
+
+    # phase 3: UDS sweep (opt-in)
+    if ns.uds or ns.uds_only:
+        leaked_uds: list = find_orphaned_uds()
+        if not leaked_uds:
+            print(
+                '[acli.reap] no orphaned UDS sock-files '
+                'to sweep'
+            )
+        elif ns.dry_run:
+            print(
+                f'[acli.reap] dry-run — {len(leaked_uds)} '
+                f'orphaned UDS sock-file(s):\n  {leaked_uds}'
+            )
+        else:
+            _, errors = reap_uds(leaked_uds)
+            if errors:
+                rc = 1
+
+    return rc
+
+
+# --- registration ---------------------------------------------
+
+# all aliases under the `acli.` namespace so xonsh's prefix-
+# completion makes them feel like a sub-cmd group: type
+# `acli.<TAB>` and the full set is suggested. no parent
+# `acli` cmd exists — the dot is purely a naming convention.
+_TCLI_ALIASES: dict = {
+    'acli.ptree': _ptree,
+    'acli.hung_dump': _hung_dump,
+    'acli.bindspace_scan': _bindspace_scan,
+    'acli.dump_all': _dump_all_alias,
+    'acli.reap': _tractor_reap,
+    'acli.watch': watch,
+}
+
+for _name, _fn in _TCLI_ALIASES.items():
+    aliases[_name] = _fn
+
+
+# xontrib protocol hooks (for `xontrib load tractor_diag`).
+# also harmless when sourced directly.
+def _load_xontrib_(xsh, **_):
+    return {}
+
+
+def _unload_xontrib_(xsh, **_):
+    for name in _TCLI_ALIASES:
+        aliases.pop(name, None)
+    return {}