Add `cpu-perf-check` sustained-throttle gate script

Standalone CLI companion to `cpu_perf_headroom()` (20cb99ec): idle
freq snapshots LIE — every static knob (`governor`, EPP,
`platform_profile`, `scaling_max_freq`) can read "performance"
while a firmware/EC power cap (AMD PPT/STAPM + friends) clamps the
package to ~30% the moment a sustained multi-core load lands,
masquerading as a `trio`-backend deadline-miss "regression" on
byte-identical code.

Deats,
- burns every core for `CPU_PERF_SECS` (default 4s) and samples the
  ACHIEVED `scaling_cur_freq` steady-state (post boost-ramp) vs the
  package max ceiling,
- exits 0 when the sustained fraction clears
  `CPU_PERF_HEALTHY_FRAC` (default 0.45), 1 when throttled — so it
  gates a suite run: `scripts/cpu-perf-check && pytest tests/ ...`,
- prints the static knobs first (to show they all read fine) then
  the remediation list on failure (`platform_profile` bounce, USB-C
  PD replug, `ryzenadj`, reboot) w/ the key reminder: do NOT bump
  test budgets — the box is slow, not the code.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

.claude/notes/rt_vars_lift_plan.md

@@ -0,0 +1,125 @@
+# `RuntimeVars` env-var lift — design plan
+
+Status: **draft, awaiting user edits**
+
+## Goal
+
+Consolidate the sprawl of pytest CLI flags + ad-hoc env vars +
+hardcoded fixture defaults into a *single* env-var-encoded
+runtime-vars envelope, with a typed in-memory representation
+(`tractor.runtime._state.RuntimeVars`) as the sole source of
+truth.
+
+## Why now
+
+- `--tpt-proto`, `--spawn-backend`, `--diag-on-hang`,
+  `--diag-capture-delay` and (soon) `TRACTOR_REG_ADDR` etc. are
+  proliferating. Each adds a parsing seam.
+- `tests/devx/test_debugger.py` invokes example scripts as
+  separate subprocesses; they currently can't see the
+  fixture-allocated `reg_addr` at all (root cause of why
+  parametrizing devx scripts on `reg_addr` is on your TODO).
+- Concurrent pytest sessions on the same host collide on
+  shared defaults (the `registry@1616` race we just fixed is
+  one symptom; per-session unique addr is the structural
+  fix).
+- `tractor.runtime._state.RuntimeVars: Struct` is already
+  defined and **unused** — its docstring even says it
+  "should be utilized as possible for future calls."
+
+## Design
+
+### Module: `tractor/_testing/_rtvars.py`
+
+Lifted from `modden.runtime.env`, ~50 LOC, no new deps.
+
+```python
+_TRACTOR_RT_VARS_OSENV: str = '_TRACTOR_RT_VARS'
+
+def dump_rtvars(rtvars: RuntimeVars|dict) -> tuple[str, str]:
+    '''str-serialize via `str(dict)` — ast.literal_eval-able'''
+
+def load_rtvars(env: dict) -> RuntimeVars:
+    '''ast.literal_eval the env-var value, hydrate to struct'''
+
+def get_rtvars(proc: psutil.Process|None = None) -> RuntimeVars:
+    '''read the var from a target proc's env (or current)'''
+
+def update_rtvars(
+    rtvars: RuntimeVars|dict|None = None,
+    update_osenv: bool|dict = True,
+) -> tuple[str, str]:
+    '''mutate + re-encode + (optionally) write to os.environ'''
+```
+
+### Encoding choice: `str(dict)` + `ast.literal_eval`
+
+Pros:
+- stdlib only
+- handles all the types tractor's tests need: `str`, `int`,
+  `float`, `bool`, `None`, `list`, `tuple`, `dict`
+- human-readable in the env (greppable, inspectable via
+  `cat /proc/<pid>/environ | tr '\0' '\n'`)
+
+Cons:
+- non-stdlib types (msgspec Structs, `Path`, custom classes)
+  must be lowered first — fine for the test fixture set
+- not stable across Python versions for esoteric repr cases
+  (we don't hit any)
+
+Alternatives considered:
+- **msgpack**: adds a dep + binary form is ungreppable
+- **json**: doesn't preserve tuples (becomes lists), which is
+  a common type for `reg_addr`
+- **toml/yaml**: heavier deps, no real benefit
+
+### `RuntimeVars` becomes the single source of truth
+
+The legacy `_runtime_vars: dict[str, Any]` global in
+`runtime/_state.py` becomes a *cached view* of a
+`RuntimeVars` singleton instance:
+
+- `get_runtime_vars()` returns either the struct or a
+  `.to_dict()` view depending on caller's preference
+- `set_runtime_vars(...)` validates against the struct schema
+- spawn-time SpawnSpec sends the struct (already does
+  conceptually — just gets typed)
+- `__setattr__` `breakpoint()` debug instrumentation gets
+  removed (unrelated cleanup, mentioned in conversation)
+
+### Migration path
+
+**Phase 0** *(prep)*: strip the stray `breakpoint()` from
+`RuntimeVars.__setattr__`.
+
+**Phase 1**: land `_rtvars.py` as a leaf module, used only by
+test infra. Subprocess-spawned scripts in `tests/devx/`
+read `_TRACTOR_RT_VARS` on startup → reconstruct
+`RuntimeVars` → call `tractor.open_root_actor(**rtvars.as_kwargs())`.
+Concurrent runs become deterministic-isolated because each
+session writes a unique `_registry_addrs` into the env.
+
+**Phase 2**: migrate runtime callers (`_state.get_runtime_vars`,
+spawn `SpawnSpec`, `Actor.async_main`) to operate on the
+struct directly, with the dict as a compat view that gets
+deprecated.
+
+**Phase 3** *(structural)*: per-session bindspace subdir
+`/run/user/<uid>/tractor/<session_uuid>/` — encoded in the
+rt-vars envelope, picked up by every subactor automatically.
+Obsoletes the entire bindspace-leak warning class.
+
+## Open design questions (user input wanted)
+
+- (placeholder for your edits)
+- (placeholder)
+- (placeholder)
+
+## Out-of-scope for this lift
+
+- Anything in `modden.runtime.env` related to `Spawn`,
+  `WmCtl`, `Wks` — that's a workspace orchestration layer,
+  not an env-var helper. We only lift the four utility
+  functions + the var name constant.
+- Switching to msgpack/json — explicitly chosen against
+  above.

.claude/settings.local.json

@@ -1,8 +1,16 @@
 {
   "permissions": {
     "allow": [
-      "Bash(date *)",
       "Bash(cp .claude/*)",
+      "Read(.claude/**)",
+      "Read(.claude/skills/run-tests/**)",
+      "Write(.claude/**/*commit_msg*)",
+      "Write(.claude/git_commit_msg_LATEST.md)",
+      "Skill(run-tests)",
+      "Skill(close-wkt)",
+      "Skill(open-wkt)",
+      "Skill(prompt-io)",
+      "Bash(date *)",
       "Bash(git diff *)",
       "Bash(git log *)",
       "Bash(git status)",
@@ -23,14 +31,12 @@
       "Bash(UV_PROJECT_ENVIRONMENT=py* uv sync:*)",
       "Bash(UV_PROJECT_ENVIRONMENT=py* uv run:*)",
       "Bash(echo EXIT:$?:*)",
-      "Write(.claude/*commit_msg*)",
-      "Write(.claude/git_commit_msg_LATEST.md)",
-      "Skill(run-tests)",
-      "Skill(close-wkt)",
-      "Skill(open-wkt)",
-      "Skill(prompt-io)"
+      "Bash(echo \"EXIT=$?\")",
+      "Read(//tmp/**)"
     ],
     "deny": [],
     "ask": []
-  }
+  },
+  "prefersReducedMotion": false,
+  "outputStyle": "default"
 }

.claude/skills/conc-anal/SKILL.md

@@ -229,3 +229,69 @@ Unlike asyncio, trio allows checkpoints in
 that does `await` can itself be cancelled (e.g.
 by nursery shutdown). Watch for cleanup code that
 assumes it will run to completion.
+
+### Unbounded waits in cleanup paths
+
+Any `await <event>.wait()` in a teardown path is
+a latent deadlock unless the event's setter is
+GUARANTEED to fire. If the setter depends on
+external state (peer disconnects, child process
+exit, subsequent task completion) that itself
+depends on the current task's progress, you have
+a mutual wait.
+
+Rule: **bound every `await X.wait()` in cleanup
+paths with `trio.move_on_after()`** unless you
+can prove the setter is unconditionally reachable
+from the state at the await site. Concrete recent
+example: `ipc_server.wait_for_no_more_peers()` in
+`async_main`'s finally (see
+`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
+"probe iteration 3") — it was unbounded, and when
+one peer-handler was stuck the wait-for-no-more-
+peers event never fired, deadlocking the whole
+actor-tree teardown cascade.
+
+### The capture-pipe-fill hang pattern (grep this first)
+
+When investigating any hang in the test suite
+**especially under fork-based backends**, first
+check whether the hang reproduces under `pytest
+-s` (`--capture=no`). If `-s` makes it go away
+you're not looking at a trio concurrency bug —
+you're looking at a Linux pipe-buffer fill.
+
+Mechanism: pytest replaces fds 1,2 with pipe
+write-ends. Fork-child subactors inherit those
+fds. High-volume error-log tracebacks (cancel
+cascade spew) fill the 64KB pipe buffer. Child
+`write()` blocks. Child can't exit. Parent's
+`waitpid`/pidfd wait blocks. Deadlock cascades up
+the tree.
+
+Pre-existing guards in `tests/conftest.py` encode
+this knowledge — grep these BEFORE blaming
+concurrency:
+
+```python
+# tests/conftest.py:258
+if loglevel in ('trace', 'debug'):
+    # XXX: too much logging will lock up the subproc (smh)
+    loglevel: str = 'info'
+
+# tests/conftest.py:316
+# can lock up on the `_io.BufferedReader` and hang..
+stderr: str = proc.stderr.read().decode()
+```
+
+Full post-mortem +
+`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
+for the canonical reproduction. Cost several
+investigation sessions before catching it —
+because the capture-pipe symptom was masked by
+deeper cascade-deadlocks. Once the cascades were
+fixed, the tree tore down enough to generate
+pipe-filling log volume → capture-pipe finally
+surfaced. Grep-note for future-self: **if a
+multi-subproc tractor test hangs, `pytest -s`
+first, conc-anal second.**

.claude/skills/run-tests/SKILL.md

@@ -8,14 +8,19 @@ allowed-tools:
   - Bash(python -m pytest *)
   - Bash(python -c *)
   - Bash(python --version *)
-  - Bash(git rev-parse *)
+  - Bash(UV_PROJECT_ENVIRONMENT=py* uv run python *)
+  - Bash(UV_PROJECT_ENVIRONMENT=py* uv run pytest *)
   - Bash(UV_PROJECT_ENVIRONMENT=py* uv sync *)
+  - Bash(UV_PROJECT_ENVIRONMENT=py* uv pip show *)
+  - Bash(git rev-parse *)
   - Bash(ls *)
   - Bash(cat *)
+  - Bash(jq * .pytest_cache/*)
   - Read
   - Grep
   - Glob
   - Task
+  - AskUserQuestion
 ---
 
 Run the `tractor` test suite using `pytest`. Follow this
@@ -90,41 +95,104 @@ python -m pytest tests/ -x --tb=short --no-header --tpt-proto uds
 python -m pytest tests/ -x --tb=short --no-header -k "cancel and not slow"
 ```
 
-## 3. Pre-flight checks (before running tests)
+## 3. Pre-flight: venv detection (MANDATORY)
 
-### Worktree venv detection
+**Always verify a `uv` venv is active before running
+`python` or `pytest`.** This project uses
+`UV_PROJECT_ENVIRONMENT=py<MINOR>` naming (e.g.
+`py313`) — never `.venv`.
 
-If running inside a git worktree (`git rev-parse
---git-common-dir` differs from `--git-dir`), verify
-the Python being used is from the **worktree's own
-venv**, not the main repo's. Check:
+### Step 1: detect active venv
+
+Run this check first:
+
+```sh
+python -c "
+import sys, os
+venv = os.environ.get('VIRTUAL_ENV', '')
+prefix = sys.prefix
+print(f'VIRTUAL_ENV={venv}')
+print(f'sys.prefix={prefix}')
+print(f'executable={sys.executable}')
+"
+```
+
+### Step 2: interpret results
+
+**Case A — venv is active** (`VIRTUAL_ENV` is set
+and points to a `py<MINOR>/` dir under the project
+root or worktree):
+
+Use bare `python` / `python -m pytest` for all
+commands. This is the normal, fast path.
+
+**Case B — no venv active** (`VIRTUAL_ENV` is empty
+or `sys.prefix` points to a system Python):
+
+Use `AskUserQuestion` to ask the user:
+
+> "No uv venv is active. Should I activate one
+> via `UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync`,
+> or would you prefer to activate your shell venv
+> first?"
+
+Options:
+1. **"Create/sync venv"** — run
+   `UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync` where
+   `<MINOR>` is detected from `python --version`
+   (e.g. `313` for 3.13). Then use
+   `py<MINOR>/bin/python` for all subsequent
+   commands in this session.
+2. **"I'll activate it myself"** — stop and let the
+   user `source py<MINOR>/bin/activate` or similar.
+
+**Case C — inside a git worktree** (`git rev-parse
+--git-common-dir` differs from `--git-dir`):
+
+Verify Python resolves from the **worktree's own
+venv**, not the main repo's:
 
 ```sh
 python -c "import tractor; print(tractor.__file__)"
 ```
 
-If the path points outside the worktree (e.g. to
-the main repo), set up a local venv first:
+If the path points outside the worktree, create a
+worktree-local venv:
 
 ```sh
 UV_PROJECT_ENVIRONMENT=py<MINOR> uv sync
 ```
 
-where `<MINOR>` matches the active cpython minor
-version (detect via `python --version`, e.g.
-`py313` for 3.13, `py314` for 3.14). Then use
-`py<MINOR>/bin/python` for all subsequent commands.
+Then use `py<MINOR>/bin/python` for all commands.
 
-**Why this matters**: without a worktree-local venv,
-subprocesses spawned by tractor resolve modules from
-the main repo's editable install, causing spurious
-`AttributeError` / `ModuleNotFoundError` for code
-that only exists on the worktree's branch.
+**Why this matters**: without the correct venv,
+subprocesses spawned by tractor resolve modules
+from the wrong editable install, causing spurious
+`AttributeError` / `ModuleNotFoundError`.
 
-### Import + collection checks
+### Fallback: `uv run`
 
-Always run these, especially after refactors or
-module moves — they catch import errors instantly:
+If the user can't or won't activate a venv, all
+`python` and `pytest` commands can be prefixed
+with `UV_PROJECT_ENVIRONMENT=py<MINOR> uv run`:
+
+```sh
+# instead of: python -m pytest tests/ -x
+UV_PROJECT_ENVIRONMENT=py313 uv run pytest tests/ -x
+
+# instead of: python -c 'import tractor'
+UV_PROJECT_ENVIRONMENT=py313 uv run python -c 'import tractor'
+```
+
+`uv run` auto-discovers the project and venv,
+but is slower than a pre-activated venv due to
+lock-file resolution on each invocation. Prefer
+activating the venv when possible.
+
+### Step 3: import + collection checks
+
+After venv is confirmed, always run these
+(especially after refactors or module moves):
 
 ```sh
 # 1. package import smoke check
@@ -137,6 +205,101 @@ python -m pytest tests/ -x -q --co 2>&1 | tail -5
 If either fails, fix the import error before running
 any actual tests.
 
+### Step 4: zombie-actor / stale-registry check (MANDATORY)
+
+The tractor runtime's default registry address is
+**`127.0.0.1:1616`** (TCP) / `/tmp/registry@1616.sock`
+(UDS). Whenever any prior test run — especially one
+using a fork-based backend like `subint_forkserver` —
+leaks a child actor process, that zombie keeps the
+registry port bound and **every subsequent test
+session fails to bind**, often presenting as 50+
+unrelated failures ("all tests broken"!) across
+backends.
+
+**This has to be checked before the first run AND
+after any cancelled/SIGINT'd run** — signal failures
+in the middle of a test can leave orphan children.
+
+```sh
+# 1. TCP registry — any listener on :1616? (primary signal)
+ss -tlnp 2>/dev/null | grep ':1616' || echo 'TCP :1616 free'
+
+# 2. leftover actor/forkserver procs — scoped to THIS
+#    repo's python path, so we don't false-flag legit
+#    long-running tractor-using apps (e.g. `piker`,
+#    downstream projects that embed tractor).
+pgrep -af "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" \
+  | grep -v 'grep\|pgrep' \
+  || echo 'no leaked actor procs from this repo'
+
+# 3. stale UDS registry sockets
+ls -la /tmp/registry@*.sock 2>/dev/null \
+  || echo 'no leaked UDS registry sockets'
+```
+
+**Interpretation:**
+
+- **TCP :1616 free AND no stale sockets** → clean,
+  proceed. The actor-procs probe is secondary — false
+  positives are common (piker, any other tractor-
+  embedding app); only cleanup if `:1616` is bound or
+  sockets linger.
+- **TCP :1616 bound OR stale sockets present** →
+  surface PIDs + cmdlines to the user, offer cleanup:
+
+  ```sh
+  # 1. GRACEFUL FIRST (tractor is structured concurrent — it
+  #    catches SIGINT as an OS-cancel in `_trio_main` and
+  #    cascades Portal.cancel_actor via IPC to every descendant.
+  #    So always try SIGINT first with a bounded timeout; only
+  #    escalate to SIGKILL if graceful cleanup doesn't complete).
+  pkill -INT -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv"
+
+  # 2. bounded wait for graceful teardown (usually sub-second).
+  #    Loop until the processes exit, or timeout. Keep the
+  #    bound tight — hung/abrupt-killed descendants usually
+  #    hang forever, so don't wait more than a few seconds.
+  for i in $(seq 1 10); do
+    pgrep -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" >/dev/null || break
+    sleep 0.3
+  done
+
+  # 3. ESCALATE TO SIGKILL only if graceful didn't finish.
+  if pgrep -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv" >/dev/null; then
+    echo 'graceful teardown timed out — escalating to SIGKILL'
+    pkill -9 -f "$(pwd)/py[0-9]*/bin/python.*_actor_child_main|subint-forkserv"
+  fi
+
+  # 4. if a test zombie holds :1616 specifically and doesn't
+  #    match the above pattern, find its PID the hard way:
+  ss -tlnp 2>/dev/null | grep ':1616'   # prints `users:(("<name>",pid=NNNN,...))`
+  # then (same SIGINT-first ladder):
+  # kill -INT <NNNN>; sleep 1; kill -9 <NNNN> 2>/dev/null
+
+  # 5. remove stale UDS sockets
+  rm -f /tmp/registry@*.sock
+
+  # 6. re-verify
+  ss -tlnp 2>/dev/null | grep ':1616' || echo 'TCP :1616 now free'
+  ```
+
+**Never ignore stale registry state.** If you see the
+"all tests failing" pattern — especially
+`trio.TooSlowError` / connection refused / address in
+use on many unrelated tests — check registry **before**
+spelunking into test code. The failure signature will
+be identical across backends because they're all
+fighting for the same port.
+
+**False-positive warning for step 2:** a plain
+`pgrep -af '_actor_child_main'` will also match
+legit long-running tractor-embedding apps (e.g.
+`piker` at `~/repos/piker/py*/bin/python3 -m
+tractor._child ...`). Always scope to the current
+repo's python path, or only use step 1 (`:1616`) as
+the authoritative signal.
+
 ## 4. Run and report
 
 - Run the constructed command.
@@ -217,7 +380,48 @@ python -c 'import tractor' && python -m pytest tests/ -x -q --co 2>&1 | tail -3
 python -m pytest tests/test_local.py tests/test_rpc.py tests/test_spawning.py tests/discovery/test_registrar.py -x --tb=short --no-header
 ```
 
-### Re-run last failures only:
+### Inspect last failures (without re-running):
+
+When the user asks "what failed?", "show failures",
+or wants to check the last-failed set before
+re-running — read the pytest cache directly. This
+is instant and avoids test collection overhead.
+
+```sh
+python -c "
+import json, pathlib, sys
+p = pathlib.Path('.pytest_cache/v/cache/lastfailed')
+if not p.exists():
+    print('No lastfailed cache found.'); sys.exit()
+data = json.loads(p.read_text())
+# filter to real test node IDs (ignore junk
+# entries that can accumulate from system paths)
+tests = sorted(k for k in data if k.startswith('tests/'))
+if not tests:
+    print('No failures recorded.')
+else:
+    print(f'{len(tests)} last-failed test(s):')
+    for t in tests:
+        print(f'  {t}')
+"
+```
+
+**Why not `--cache-show` or `--co --lf`?**
+
+- `pytest --cache-show 'cache/lastfailed'` works
+  but dumps raw dict repr including junk entries
+  (stale system paths that leak into the cache).
+- `pytest --co --lf` actually *collects* tests which
+  triggers import resolution and is slow (~0.5s+).
+  Worse, when cached node IDs don't exactly match
+  current parametrize IDs (e.g. param names changed
+  between runs), pytest falls back to collecting
+  the *entire file*, giving false positives.
+- Reading the JSON directly is instant, filterable
+  to `tests/`-prefixed entries, and shows exactly
+  what pytest recorded — no interpretation.
+
+**After inspecting**, re-run the failures:
 ```sh
 python -m pytest --lf -x --tb=short --no-header
 ```
@@ -247,3 +451,175 @@ by your changes — note them and move on.
 **Rule of thumb**: if a test fails with `TooSlowError`,
 `trio.TooSlowError`, or `pexpect.TIMEOUT` and you didn't
 touch the relevant code path, it's flaky — skip it.
+
+## 9. The pytest-capture hang pattern (CHECK THIS FIRST)
+
+**Symptom:** a tractor test hangs indefinitely under
+default `pytest` but passes instantly when you add
+`-s` (`--capture=no`).
+
+**Cause:** tractor subactors (especially under fork-
+based backends) inherit pytest's stdout/stderr
+capture pipes via fds 1,2. Under high-volume error
+logging (e.g. multi-level cancel cascade, nested
+`run_in_actor` failures, anything triggering
+`RemoteActorError` + `ExceptionGroup` traceback
+spew), the **64KB Linux pipe buffer fills** faster
+than pytest drains it. Subactor writes block → can't
+finish exit → parent's `waitpid`/pidfd wait blocks →
+deadlock cascades up the tree.
+
+**Pre-existing guards in the tractor harness** that
+encode this same knowledge — grep these FIRST
+before spelunking:
+
+- `tests/conftest.py:258-260` (in the `daemon`
+  fixture): `# XXX: too much logging will lock up
+  the subproc (smh)` — downgrades `trace`/`debug`
+  loglevel to `info` to prevent the hang.
+- `tests/conftest.py:316`: `# can lock up on the
+  _io.BufferedReader and hang..` — noted on the
+  `proc.stderr.read()` post-SIGINT.
+
+**Debug recipe (in priority order):**
+
+1. **Try `-s` first.** If the hang disappears with
+   `pytest -s`, you've confirmed it's capture-pipe
+   fill. Skip spelunking.
+2. **Lower the loglevel.** Default `--ll=error` on
+   this project; if you've bumped it to `debug` /
+   `info`, try dropping back. Each log level
+   multiplies pipe-pressure under fault cascades.
+3. **If you MUST use default capture + high log
+   volume**, redirect subactor stdout/stderr in the
+   child prelude (e.g.
+   `tractor.spawn._subint_forkserver._child_target`
+   post-`_close_inherited_fds`) to `/dev/null` or a
+   file.
+
+**Signature tells you it's THIS bug (vs. a real
+code hang):**
+
+- Multi-actor test under fork-based backend
+  (`subint_forkserver`, eventually `trio_proc` too
+  under enough log volume).
+- Multiple `RemoteActorError` / `ExceptionGroup`
+  tracebacks in the error path.
+- Test passes with `-s` in the 5-10s range, hangs
+  past pytest-timeout (usually 30+ s) without `-s`.
+- Subactor processes visible via `pgrep -af
+  subint-forkserv` or similar after the hang —
+  they're alive but blocked on `write()` to an
+  inherited stdout fd.
+
+**Historical reference:** this deadlock cost a
+multi-session investigation (4 genuine cascade
+fixes landed along the way) that only surfaced the
+capture-pipe issue AFTER the deeper fixes let the
+tree actually tear down enough to produce pipe-
+filling log volume. Full post-mortem in
+`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
+Lesson codified here so future-me grep-finds the
+workaround before digging.
+
+## 10. Reaping zombie subactors (`tractor-reap`)
+
+**Symptom:** after a `pytest` run crashes, times out,
+or is `Ctrl+C`'d, subactor forks (esp. under
+`subint_forkserver`) can be reparented to `init`
+(PPid==1) and linger. They hold onto ports, inherit
+pytest's capture-pipe fds, and flakify later
+sessions.
+
+**Two layers of defense:**
+
+### a) Session-scoped auto-fixture (always on)
+
+`tractor/_testing/pytest.py::_reap_orphaned_subactors`
+runs at pytest session teardown. It walks `/proc` for
+direct descendants of the pytest pid, SIGINTs them,
+waits up to 3s, then SIGKILLs survivors. SC-polite:
+gives the subactor runtime a chance to run its trio
+cancel shield + IPC teardown before escalation.
+
+This is *autouse* and session-scoped — you don't need
+to do anything. It just runs.
+
+### b) `scripts/tractor-reap` CLI (manual reap)
+
+For the **pytest-died-mid-session** case (Ctrl+C, OOM
+kill, hung process you had to `kill -9`), the fixture
+never ran. Reach for the CLI:
+
+```sh
+# default: orphans (PPid==1, cwd==repo, cmd contains python)
+scripts/tractor-reap
+
+# descendant-mode: from a still-live supervisor
+scripts/tractor-reap --parent <pytest-pid>
+
+# see what would be reaped, don't signal
+scripts/tractor-reap -n
+
+# tune the SIGINT → SIGKILL grace window
+scripts/tractor-reap --grace 5
+```
+
+Exit code: `0` if everyone exited on SIGINT, `1` if
+SIGKILL had to escalate — so you can chain it in CI
+health-checks (`scripts/tractor-reap || <alert>`).
+
+**What it matches** (orphan-mode):
+- `PPid == 1` (reparented to init → definitely
+  orphaned, not just a currently-running child)
+- `cwd == <repo-root>` (keeps the sweep scoped; won't
+  touch unrelated init-children elsewhere)
+- `python` in cmdline
+
+**What it does not do:** kill anything whose PPid is
+still a live tractor parent. If the parent is alive
+it's not an orphan; use `--parent <pid>` if you need
+to force-reap under a still-live supervisor.
+
+**When NOT to run it:** while a pytest session is
+active in another terminal. It's safe (won't touch
+that session's live children in orphan-mode) but can
+race if the target session is mid-teardown.
+
+### c) `--shm` / `--shm-only`: orphan-segment sweep
+
+Because `tractor.ipc._mp_bs.disable_mantracker()`
+turns off `mp.resource_tracker` (see
+`ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`),
+a hard-crashing actor can leave `/dev/shm/<key>`
+segments behind that nothing else GCs.
+
+```sh
+# process reap THEN shm sweep
+scripts/tractor-reap --shm
+
+# shm sweep only (skip process phase)
+scripts/tractor-reap --shm-only
+
+# dry-run: list candidates, don't unlink
+scripts/tractor-reap --shm -n
+```
+
+**Match criteria** (very conservative — this is a
+shared-system path, can't be wrong):
+- segment is a regular file under `/dev/shm`,
+- owned by the **current uid** (`stat.st_uid`),
+- AND **no live process holds it open** —
+  enumerated by walking every readable
+  `/proc/<pid>/maps` (post-mmap mappings) AND
+  `/proc/<pid>/fd/*` (pre-mmap shm-opened fds).
+
+The "nobody has it open" check is the
+kernel-canonical "is this leaked?" test — same
+answer `lsof /dev/shm/<key>` would give. No
+reliance on tractor-specific naming, so it works
+for any tractor app. Critically, it WILL NOT touch
+segments held by other apps you have running
+(e.g. `piker`, `lttng-ust-*`, `aja-shm-*` —
+verified locally with 81 in-use segments correctly
+preserved).

.github/workflows/ci.yml

@@ -83,10 +83,27 @@ jobs:
 
 
   testing:
-    name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
-    timeout-minutes: 16
+    name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
+    timeout-minutes: 20
     runs-on: ${{ matrix.os }}
 
+    # NOTE on the matrix shape — the `capture=` mode follows
+    # `spawn_backend`:
+    #
+    # - `trio` / `mp_*` backends use `--capture=fd` (default)
+    #   for per-test attribution of subactor *raw-fd* output
+    #   in failure reports.
+    # - Fork-based backends (`main_thread_forkserver`,
+    #   `subint_forkserver`) REQUIRE `--capture=sys` because
+    #   fork-child × `--capture=fd` is a known deadlock
+    #   pattern. See the long NOTE in `tractor._testing.pytest`'s
+    #   `pytest_load_initial_conftests` for the mechanism +
+    #   tradeoff write-up.
+    #
+    # If a future matrix row adds a fork-spawn backend
+    # WITHOUT setting `capture: 'sys'`, the
+    # `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
+    # with a clear error msg. So the matrix is self-policing.
     strategy:
       fail-fast: false
       matrix:
@@ -113,6 +130,26 @@ jobs:
           'tcp',
           'uds',
         ]
+        capture: [
+          'fd',  # default for non-fork backends
+        ]
+
+        # Fork-based backends — added via `include:` so each
+        # cell carries its REQUIRED `capture: 'sys'` mode.
+        # Linux-only for now; macOS coverage TBD pending
+        # local validation.
+        include:
+          - os: ubuntu-latest
+            python-version: '3.13'
+            spawn_backend: 'main_thread_forkserver'
+            tpt_proto: 'tcp'
+            capture: 'sys'
+          - os: ubuntu-latest
+            python-version: '3.13'
+            spawn_backend: 'main_thread_forkserver'
+            tpt_proto: 'uds'
+            capture: 'sys'
+
         # https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
         exclude:
           # don't do UDS run on macOS (for now)
@@ -148,9 +185,16 @@ jobs:
       - name: Run tests
         run: >
           uv run
-          pytest tests/ -rsx
+          pytest
+          tests/
+          -rsx
           --spawn-backend=${{ matrix.spawn_backend }}
           --tpt-proto=${{ matrix.tpt_proto }}
+          --capture=${{ matrix.capture }}
+        # NOTE: capture mode is matrix-driven — `fd` for
+        # non-fork backends (per-test fd attribution),
+        # `sys` for fork-based (avoids fork-child x
+        # capture-fd deadlock). See matrix-NOTE above.
 
   # XXX legacy NOTE XXX
   #

.gitignore

@@ -106,46 +106,55 @@ venv.bak/
 # all files under
 .git/
 
-# any commit-msg gen tmp files
-.claude/skills/commit-msg/msgs/
-.claude/git_commit_msg_LATEST.md
-.claude/*_commit_*.md
-.claude/*_commit*.toml
-.claude/*_commit*.txt
-.claude/skills/commit-msg/msgs/*
+# require very explicit staging for anything we **really**
+# want put/kept in repo.
+notes_to_self/
+snippets/
 
-.claude/skills/pr-msg/msgs/*
-# XXX, for rn, so i can telescope this file.
-!/.claude/skills/pr-msg/pr_msg_LATEST.md
-
-# review-skill ephemeral ctx (per-PR, single-use)
-.claude/review_context.md
-.claude/review_regression.md
-
-# per-skill session/conf (machine-local)
-.claude/skills/*/conf.toml
-
-# ai.skillz symlinks (machine-local, deploy via deploy-skill.sh)
+# ------- AI shiz -------
+# `ai.skillz` symlinks,
+# (machine-local, deploy via deploy-skill.sh)
 .claude/skills/py-codestyle
-.claude/skills/code-review-changes
 .claude/skills/close-wkt
-.claude/skills/open-wkt
 .claude/skills/plan-io
 .claude/skills/prompt-io
 .claude/skills/resolve-conflicts
 .claude/skills/inter-skill-review
-.claude/skills/yt-url-lookup
 
-# hybrid skills — symlinked SKILL.md + references
-.claude/skills/commit-msg/SKILL.md
-.claude/skills/pr-msg/SKILL.md
-.claude/skills/pr-msg/references
+# /open-wkt specifics
+.claude/skills/open-wkt
+.claude/wkts/
+claude_wkts
+
+# /code-review-changes specifics
+.claude/skills/code-review-changes
+# review-skill ephemeral ctx (per-PR, single-use)
+.claude/review_context.md
+.claude/review_regression.md
+
+# /pr-msg specifics
+.claude/skills/pr-msg/*
+# repo-specific
+!.claude/skills/pr-msg/format-reference.md
+# XXX, so u can nvim-telescope this file.
+# !.claude/skills/pr-msg/pr_msg_LATEST.md
+
+# /commit-msg specifics
+# - any commit-msg gen tmp files
+.claude/*_commit_*.md
+.claude/*_commit*.txt
+.claude/skills/commit-msg/*
+!.claude/skills/commit-msg/style-duie-reference.md
+
+# use prompt-io instead?
+.claude/plans
 
 # nix develop --profile .nixdev
 .nixdev*
 
 # :Obsession .
 Session.vim
+
 # `gish` local `.md`-files
 # TODO? better all around automation!
 # -[ ] it'd be handy to also commit and sync with wtv git service?
@@ -159,7 +168,3 @@ gh/
 
 # LLM conversations that should remain private
 docs/conversations/
-
-# Claude worktrees
-.claude/wkts/
-claude_wkts

ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md

@@ -0,0 +1,314 @@
+# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
+
+## Symptom
+
+Running the full test suite under
+
+```bash
+./py313/bin/python -m pytest tests/ \
+  --tpt-proto=tcp \
+  --spawn-backend=main_thread_forkserver
+```
+
+surfaces a single, **rotating** `trio.TooSlowError`
+failure each run. The failure isn't deterministic on
+test identity — different test each run — but it
+ALWAYS looks like:
+
+```
+FAILED tests/<file>::test_<name> - trio.TooSlowError
+==== 1 failed, 373 passed, 17 skipped, 11–12 xfailed,
+       0–1 xpassed, ~550 warnings in ~6min ====
+```
+
+Pass rate: **~99.7%** (373 of 374 non-skip tests).
+Wall-clock per full run: 5–6 min.
+
+## Tests observed flaking so far
+
+Each row was the SOLE failure in a separate run:
+
+| run # | test |
+|---|---|
+| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
+| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
+
+Both share the same shape:
+
+- **Cancel cascade** of N subactors back to a parent root actor.
+- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
+  (it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
+- N ≈ 2 for `test_context_spawns_aio_task_that_errors` —
+  but each subactor is `infect_asyncio=True`, so each
+  cancel involves the trio↔asyncio guest-run unwind
+  which is structurally heavier than pure-trio.
+- Test wraps the cascade in `trio.fail_after(N seconds)`
+  and the cap fires before the cascade completes.
+
+The exact failing test rotates because each test is
+independently close to the cap; whichever happens to
+be unlucky in scheduling/CPU-contention on a given run
+is the one that times out.
+
+## Root-cause family
+
+`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
+the SC-graceful teardown ladder per subactor:
+
+1. `Portal.cancel_actor()` — graceful IPC cancel-req.
+2. Wait `terminate_after=1.6s` for sub to exit.
+3. If still alive: `proc.kill()` (SIGKILL).
+4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
+   sock-file cleanup for UDS leaks (issue #452 fix).
+
+For a cascade of N subactors, each pays steps 1–4. If
+graceful-cancel doesn't complete within 1.6s for ANY
+sub, that sub eats a full 1.6s of `move_on_after` plus
+the `proc.wait()` post-SIGKILL.
+
+Worst case under fork backend with N=cpus subs:
+- N × 1.6s = 16s+ on a 10-core box just for the
+  graceful timeout phase
+- Plus per-spawn fork-IPC handshake cost compounds
+  during teardown (each sub's IPC cleanup goes through
+  the same forkserver coordinator)
+- Plus the new autouse fixtures
+  (`_track_orphaned_uds_per_test`,
+  `_detect_runaway_subactors_per_test`,
+  `_reap_orphaned_subactors`) all run at test
+  teardown, adding small (10s of ms) but cumulative
+  overhead
+
+Current cap: 30s (`fail_after_s = 30 if
+is_forking_spawner else 12`). Empirically fits the
+median run but the tail breaks ~0.3% of the time.
+
+## NOT regressing
+
+To confirm this is a flake and not a regression:
+
+- Pre-`WakeupSocketpair`-patch baseline: tests
+  HUNG INDEFINITELY (busy-loop never released).
+- Post-patch: pass-or-fail-fast, ~99.7% pass, the
+  occasional cap-hit fails in bounded time (<60s for
+  the offending test).
+- Same test PASSES under `--spawn-backend=trio`
+  (no fork, no hard-kill compounding).
+
+So the suite is dramatically better than before; the
+remaining flake is a known-tolerable steady-state.
+
+## Possible mitigations (ranked)
+
+### A. Bump the cap further
+
+Cheapest. Change the per-test `fail_after_s` from 30
+to e.g. 60 for fork backends. Pros: trivial. Cons:
+masks any genuine slowness regression we'd want to
+catch.
+
+### B. CPU-count-aware cap
+
+For tests whose N scales with `cpu_count()`, scale
+the cap too:
+
+```python
+fail_after_s = (
+    max(30, cpu_count() * 3)  # 3s/actor floor
+    if is_forking_spawner
+    else 12
+)
+```
+
+Pros: scales with the actual cancel-cascade work.
+Cons: still arbitrary multiplier.
+
+### C. `pytest-rerunfailures` for these tests only
+
+Mark the known-flaky tests with
+`@pytest.mark.flaky(reruns=1)` (needs
+`pytest-rerunfailures` dep). Single retry hides
+genuine ~0.3% transient flakes.
+
+Pros: no cap change, surfaces persistent failures
+loudly. Cons: adds a dep, retries can mask real bugs
+if used widely.
+
+### D. Reduce `hard_kill`'s `terminate_after`
+
+Drop from 1.6s → 0.8s. Cuts the worst-case cascade
+time roughly in half. Risks: fewer subs get a chance
+to run their cleanup before SIGKILL → more orphaned
+state for the autouse reapers to handle (ironically,
+adds back overhead elsewhere).
+
+### E. Profile + targeted fix
+
+Add `log.devx()` markers in `hard_kill` to time each
+phase. Identify if any subactor is consistently
+hitting the 1.6s cap (vs. exiting in <0.1s). If so,
+that sub has a teardown bug worth fixing at source.
+Pros: actually fixes the underlying slowness. Cons:
+real investigation work, deferred from this round.
+
+## Recommendation
+
+Land this issue-doc as the tracker. Apply **(B)** as
+a small follow-up — cheap and proportional. If it
+still flakes, escalate to **(E)** with a `log.devx()`
+profile-pass.
+
+`(C)` is a backstop if `(B)` doesn't quite get there
+and we need green CI faster than (E) can deliver.
+
+## Verification protocol
+
+After applying any mitigation:
+
+```bash
+# Run the suite N times back-to-back, count failures.
+# A persistent failure on the SAME test == real bug.
+# Failures rotating across tests == still cap-related.
+
+for i in $(seq 1 5); do
+  ./py313/bin/python -m pytest tests/ \
+    --tpt-proto=tcp \
+    --spawn-backend=main_thread_forkserver \
+    -q 2>&1 | tail -2
+done
+```
+
+Target: 0 failures across 5 runs ⇒ ship. 1–2 failures
+still rotating ⇒ apply (C). Same test failing twice
+⇒ escalate to (E).
+
+## Snapshot evidence (2026-05-13)
+
+After landing the `fail_after_w_trace` /
+`afk_alarm_w_trace` capture-on-timeout helpers
+(`tractor._testing.trace`), `test_nested_multierrors`
+on the `main_thread_forkserver` backend produces
+**reproducible diag snapshots** at
+`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
+
+### Reproduction
+
+```bash
+pytest \
+  -v --verbose --durations=10 \
+  --spawn-backend=main_thread_forkserver \
+  --tpt-proto=uds \
+  --capture=sys --show-capture=stderr -rxX \
+  tests/test_cancellation.py::test_nested_multierrors
+```
+
+The test is `xfail(strict=False)` for MTF — it RUNS
+each invocation so snapshots accumulate, but doesn't
+break `--lf` workflow.
+
+### Consistent shape across runs
+
+5+ snapshots taken back-to-back show the SAME pattern:
+
+- **Timing:** ~10s wall-clock total. Inner
+  `fail_after_w_trace(10)` fires at exactly T=10s;
+  cascade's `nursery.__aexit__` takes ~0.6s more to
+  gather + propagate the resulting
+  `BaseExceptionGroup`. **Trio backend completes the
+  SAME test in <6s** — so the MTF cascade is ~2x
+  slower at minimum.
+
+- **`BaseExceptionGroup` shape:** mixed
+  `[RemoteActorError, Cancelled]`. The first
+  subactor's natural error-propagation (`assert 0`
+  raised → `RemoteActorError` portal-result)
+  arrives before T=10s; the OTHER subactor's
+  portal-wait is still in flight at T=10s, gets
+  cancelled by `fail_after_w_trace`'s scope-cancel
+  → returns `Cancelled` instead.
+
+- **Orphan-spawn skew:** snapshot's `orphans` bucket
+  (after the `_is_tractor_subactor` cgroup-slice
+  override fix) consistently shows 2-4 init-adopted
+  procs at `depth_3` and `depth_1` levels — these
+  are the leaves whose parent (`depth_2` spawner)
+  was killed mid-cascade but who hadn't yet seen
+  the cancel signal themselves.
+
+- **UDS sock-leak:** 2-6 dead-orphan socks per run
+  (varies with cascade timing). The
+  `track_orphaned_uds_per_test` fixture reaps them
+  post-test → contamination is isolated per-invocation.
+
+### Capture mechanism
+
+`fail_after_w_trace` covers two firing paths:
+
+1. **`trio.TooSlowError`** raised at scope-exit
+   (body returned cleanly past deadline) — direct
+   `except` handler captures.
+
+2. **Scope-cancel + body raises non-`Cancelled` exc**
+   (e.g. `nursery.__aexit__` wraps timeout-induced
+   `Cancelled` into a `BaseExceptionGroup` that
+   escapes before `trio.fail_after`'s exit-check
+   could fire `TooSlowError`) — body-raise `except`
+   handler checks `scope.cancel_called` and
+   captures if True. This path catches the
+   `test_nested_multierrors` shape specifically (see
+   "BaseExceptionGroup shape" above).
+
+The snapshot dir contains:
+- `trace.txt` — `ptree` + `hung_state` (kernel
+  `wchan`/`stack` + `py-spy dump --locals` when
+  sudo cached), with `include_strays=True`
+  surfacing any cross-test ghost subactor trees in
+  the `orphans` bucket.
+- `bindspace.txt` — UDS bindspace classification
+  (live-active / orphaned-alive / orphaned-dead).
+- `meta.json` — `{pid, label, captured_at, sudo_cached}`.
+
+The end-of-session `pytest_terminal_summary` hook
+in `tractor._testing.pytest` lists every snapshot
+dir from the run so you don't have to scroll back
+through captured-stderr lines:
+
+```
+========================= tractor hang-snapshot index ==========================
+N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
+  <test-id>
+    → /home/.../.cache/tractor/hung-dumps/<label>__<ts>
+```
+
+### Caveats
+
+The snapshot fires AFTER the body-raise (not at the
+exact moment of scope-cancel), so the parent's
+py-spy frames show `_do_capture_snapshot` itself
+running, NOT the cancel-cascade hang frame. To see
+the actual hang state, manual `acli.ptree` /
+`acli.hung_dump` from a second terminal at T=10s
+would be needed — **not currently possible**
+because per-test reaper fixtures clean up ~0.6s
+post-timeout. See follow-up TODO in
+`tractor/_testing/trace.py` for a
+`TRACTOR_TRACE_HOLD=1` env-var pause mode.
+
+## See also
+
+- [#452](https://github.com/goodboy/tractor/issues/452) —
+  UDS sock-file leak (related — `hard_kill`'s
+  cleanup phase contributes to cascade time)
+- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
+  — the upstream-trio fix that turned this from a
+  100% hang into a 0.3% flake
+- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
+  — the asyncio variant which contributes to one of
+  the rotating failures
+- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
+  cascade source
+- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
+  `_detect_runaway_subactors_per_test`,
+  `_reap_orphaned_subactors` — autouse cleanup
+  fixtures whose cumulative teardown overhead
+  contributes to the cascade time

ai/conc-anal/fork_thread_semantics_execution_vs_memory.md

@@ -0,0 +1,281 @@
+# `fork()` in a multi-threaded program — execution-side vs. memory-side of the same coin
+
+A reference doc for readers who've encountered one of two
+opposite-sounding framings of POSIX `fork()` semantics in a
+multi-threaded program and are confused by the other.
+
+This is a sibling to
+`subint_fork_blocked_by_cpython_post_fork_issue.md` — that
+doc covers a CPython-level refusal of fork-from-subint;
+this one covers the more general POSIX layer, since
+tractor's main-thread forkserver design rests on it.
+
+## TL;DR
+
+POSIX `fork()` only preserves the *calling* thread as a
+runnable thread in the child — every other thread in the
+parent simply never executes another instruction in the
+child. trio's docs call this "leaked"; tractor's
+`_main_thread_forkserver.py` docstring calls it "gone".
+Both are correct: "gone" is the *execution* side (no
+scheduler entry, no instructions retired), "leaked" is the
+*memory* side (the dead threads' stacks and per-thread
+heap structures still ride into the child's address space
+as orphaned COW pages with no owner and no cleanup hook).
+Same POSIX reality, two halves of the same coin.
+
+## The two framings
+
+[python-trio/trio#1614][trio-1614] (the canonical "trio +
+fork" hazards thread) puts it this way:
+
+> If you use `fork()` in a process with multiple threads,
+> all the other thread stacks are just leaked: there's
+> nothing else you can reasonably do with them.
+
+`tractor.spawn._main_thread_forkserver`'s module docstring
+(specifically the "What survives the fork? — POSIX
+semantics" section) puts it this way:
+
+> POSIX `fork()` only preserves the *calling* thread as a
+> runnable thread in the child. Every other thread in the
+> parent — trio's runner thread, any `to_thread` cache
+> threads, anything else — never executes another
+> instruction post-fork.
+
+A reader bouncing between the two can be forgiven for
+asking: well, *which* is it — leaked or gone?
+
+The answer is "yes". They're describing the same POSIX
+behavior from two different angles:
+
+- trio is talking about the **bytes** the dead threads
+  leave behind — stacks, TLS slots, per-thread arena
+  metadata — and the fact that nothing in the child can
+  drive them forward, free them, or even safely walk
+  them. That's a memory leak in the strict sense: held
+  but unreachable.
+- tractor is talking about the **execution** side
+  relevant to the forkserver design: which threads
+  retire instructions in the child? Exactly one — the
+  one that called `fork()`. Everything else, regardless
+  of the bytes left behind, is dead in a scheduler
+  sense.
+
+Neither framing is wrong; they're just answering
+different questions.
+
+## POSIX `fork()` in a multi-threaded program — what actually happens
+
+Per POSIX (and concretely on Linux glibc), the contract
+of `fork()` in a multi-threaded process is:
+
+1. The kernel creates a new process whose virtual
+   address space is a COW copy of the parent's. *All*
+   pages map across — code, heap, every thread's stack,
+   every malloc arena, every mmap region.
+2. Of the parent's N threads, exactly **one** is
+   reified in the child as a runnable kernel task: the
+   thread that called `fork()`. The other N-1 threads
+   have *no* corresponding task in the child kernel. They
+   were never scheduled, never `clone()`d for the child,
+   never exist as runnable entities.
+3. Their **memory artifacts** — pthread stacks, TLS,
+   `pthread_t` structures, glibc per-thread arena
+   bookkeeping — are still mapped in the child's address
+   space, because (1) duplicates *everything* page-wise.
+   They sit there as inert COW bytes.
+4. The kernel does not clean those bytes up. There is no
+   "phantom-thread cleanup" pass post-fork. The kernel
+   doesn't know which mapped pages "belonged to" which
+   thread — at the kernel level mappings are
+   process-scoped, not thread-scoped.
+5. The surviving thread (the caller of `fork()`) cannot
+   safely access those leaked bytes either. Any state
+   they encoded — held mutexes, in-flight syscalls,
+   half-updated invariants — is frozen at whatever
+   instant the parent's fork-syscall observed it. Some
+   of those mutexes may even still be locked from the
+   child's POV (the canonical "fork-in-multithreaded-
+   program-deadlocks" hazard; see `man pthread_atfork`).
+
+So: from the kernel's PoV, the child has one thread.
+From the address-space's PoV, the child has all the
+parent's bytes — including the corpses of the N-1 dead
+threads' stacks. Both true simultaneously.
+
+## Why trio says "leaked"
+
+trio's framing makes sense from the parent's
+PoV, looking at *what those threads were doing*. In a
+running `trio.run()` process you typically have:
+
+- The trio runner thread itself — owns the `selectors`
+  epoll fd, the signal-wakeup-fd, the run-queue.
+- Threadpool worker threads (`trio.to_thread`'s cache)
+  — blocked in `wait()` on the threadpool's work
+  condvar.
+- Whatever other ad-hoc threads the application
+  started.
+
+Each of those threads owns *real work-state*: epoll
+registrations, file descriptors held in
+soon-to-be-completed reads, half-released locks, posted
+but unconsumed wakeups. After fork, that state is still
+encoded in the child's memory. None of it is invalid in
+a well-formed-bytes sense. It's just that:
+
+- The thread that was driving it is gone.
+- Nothing else in the child knows the layout well
+  enough to take over.
+- Even if it did, the kernel objects backing the work
+  (epoll fd, signalfd) have separate post-fork
+  semantics that don't compose with userland trio
+  state.
+
+So the bytes are *held* (they're in the child's
+address space, they count against RSS, they survive
+until something clobbers them), and they're
+*unreachable* in any meaningful sense — no thread can
+safely drive them forward. That is the textbook
+definition of a leak.
+
+trio's quote is reminding the user that `fork()` from a
+multi-threaded process is a one-way memory hazard:
+whatever those threads were doing, that work-state is
+now garbage you happen to still be carrying.
+
+## Why tractor says "gone"
+
+tractor's `_main_thread_forkserver` framing is concerned
+with a different question: *which thread executes in the
+child, and is it safe?*
+
+The forkserver design rests on POSIX's "calling thread
+is the sole survivor" guarantee. We pick that calling
+thread very deliberately: a dedicated worker that has
+provably never entered trio. So the thread that *does*
+run in the child is one whose locals, TLS, and stack
+contain nothing trio-related. Trio's runner thread —
+the one that owned the epoll fd and the run-queue — is
+*gone* from the child in the execution sense. It will
+never run another instruction. The fact that its stack
+bytes still exist in the child's address space (the
+"leaked" view) is irrelevant to the forkserver, because
+nothing in the child reads or writes those pages.
+
+So when the docstring says "Every other thread … is
+gone the instant `fork()` returns in the child", it's
+being precise about the surface that matters for the
+backend: scheduler-level liveness. Nothing schedules
+those threads ever again. Whether their bytes are
+hanging around is a separate (and, for the design,
+non-load-bearing) fact.
+
+## Cross-table
+
+The same tabular layout the `_main_thread_forkserver`
+docstring uses, expanded with a fourth "what handles
+it" column:
+
+| thread              | parent    | child (executing) | child (memory)               | what handles it             |
+|---------------------|-----------|-------------------|------------------------------|-----------------------------|
+| forkserver worker   | continues | sole survivor     | live stack                   | runs the child's bootstrap  |
+| `trio.run()` thread | continues | not running       | leaked stack (zombie bytes)  | overwritten by child's fresh `trio.run()` |
+| any other thread    | continues | not running       | leaked stack (zombie bytes)  | overwritten / GC'd / clobbered by `exec()` if used |
+
+The "child (executing)" column is the *execution* side
+of the coin — what tractor cares about. The "child
+(memory)" column is the *memory* side — what trio
+cares about.
+
+The "what handles it" column is the deliberate punchline
+of the design: nothing has to handle the leaked bytes
+*explicitly*. They get clobbered by ordinary forward
+progress in the child:
+
+- The fresh `trio.run()` the child boots up allocates
+  its own stack, scheduler, and run-queue, which over
+  time overlaps and overwrites the inherited zombie
+  pages.
+- Python's GC walks live objects only; the dead-thread
+  Python frames aren't reachable from any
+  `PyThreadState`, so they get freed at the next
+  collection cycle.
+- If the child eventually `exec()`s, the entire address
+  space is replaced and the leak vanishes.
+
+## What this means for the forkserver design
+
+The crucial point is that **the design doesn't and
+*can't* prevent the leak**. There is no userland fix
+for COW thread stacks. The kernel hands the child a
+duplicated address space; that's what `fork()` *is*. No
+amount of pre-fork hookery, `pthread_atfork()`
+gymnastics, or post-fork cleanup can un-COW the dead
+threads' pages without unmapping them, and unmapping
+arbitrary regions of a duplicated address space is
+neither portable nor safe.
+
+What the design *does* ensure is the orthogonal
+property: the survivor thread is one that doesn't need
+any of that leaked state to function. Concretely:
+
+- Survivor is the forkserver worker thread.
+- That worker has provably never imported, called into,
+  or held any reference to `trio`. (Enforced by keeping
+  the worker's lifecycle entirely in
+  `_main_thread_forkserver.py` and never letting trio
+  task-state cross into it.)
+- So the leaked pages — trio runner stack, threadpool
+  caches, etc. — are inert relative to the survivor.
+  No code path in the child references them.
+- The child then boots its own fresh `trio.run()`,
+  which allocates new state in new pages. Over the
+  child's lifetime the COW'd zombie pages get
+  overwritten, GC'd, or (if the child eventually
+  `exec()`s) discarded wholesale.
+
+The "leak" is real but inert. It costs RSS until
+clobbered; it doesn't cost correctness. That's exactly
+the property the forkserver pattern is built on, and
+it's also why the design needs the "calling thread is
+trio-free" precondition to be airtight: if the survivor
+were a trio thread, it *would* try to drive the leaked
+trio state, and the leak would no longer be inert.
+
+## See also
+
+- `tractor/spawn/_main_thread_forkserver.py` — module
+  docstring's "What survives the fork? — POSIX
+  semantics" section is the in-tree, code-adjacent
+  prose this doc expands on. The cross-table here is a
+  fourth-column expansion of the table there.
+
+- [python-trio/trio#1614][trio-1614] — the trio issue
+  with the "leaked" framing, and the canonical thread
+  for trio + `fork()` hazards more broadly.
+
+- [`subint_fork_blocked_by_cpython_post_fork_issue.md`](./subint_fork_blocked_by_cpython_post_fork_issue.md)
+  — sibling analysis covering CPython's *post-fork*
+  hooks (`PyOS_AfterFork_Child`,
+  `_PyInterpreterState_DeleteExceptMain`) and why
+  fork-from-non-main-subint is a CPython-level hard
+  refusal. Complementary axis: this doc is about POSIX
+  semantics; that doc is about the CPython runtime
+  layer that runs *after* POSIX `fork()` returns in
+  the child.
+
+- `man pthread_atfork(3)` — canonical "fork in a
+  multithreaded process is dangerous" reference.
+  Especially the rationale section, which is the
+  closest thing to a normative statement of "the
+  surviving thread cannot safely use anything the dead
+  threads were touching."
+
+- `man fork(2)` (Linux) — "Other than [the calling
+  thread], … no other threads are replicated …"
+  paragraph is the kernel-side statement of the
+  execution-side framing this doc opens with.
+
+[trio-1614]: https://github.com/python-trio/trio/issues/1614

ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md

@@ -0,0 +1,378 @@
+# `infect_asyncio` × `main_thread_forkserver` Mode-A deadlock
+
+## Reproducer
+
+```bash
+./py313/bin/python -m pytest \
+  tests/test_infected_asyncio.py::test_aio_simple_error \
+  --tpt-proto=tcp \
+  --spawn-backend=main_thread_forkserver \
+  -v --capture=sys
+```
+
+Hangs indefinitely. Mode-A signature — both processes
+parked in `epoll_wait`, **neither burning CPU**.
+
+## Empirical observations (caught alive)
+
+### Outer pytest (parent)
+
+`py-spy dump` on the test runner pid shows the trio
+event loop parked at the bottom of `trio.run()`:
+
+```
+Thread <pid> (idle): "MainThread"
+    get_events (trio/_core/_io_epoll.py:245)
+        self: <EpollIOManager at 0x...>
+        timeout: 86400
+    run (trio/_core/_run.py:2415)
+        next_send: []
+        timeout: 86400
+    test_aio_simple_error (tests/test_infected_asyncio.py:175)
+```
+
+`timeout: 86400` is trio's "no scheduled work, just wait
+for I/O forever" sentinel. `next_send: []` confirms
+nothing is queued. The parent is stuck inside
+`tractor.open_nursery(...).run_in_actor(...)` waiting
+for `ipc_server.wait_for_peer(uid)` to fire — i.e.
+waiting for the spawned subactor to connect back.
+
+### Subactor (forked child)
+
+`/proc/<pid>/stack`:
+
+```
+do_epoll_wait+0x4c0/0x500
+__x64_sys_epoll_wait+0x70/0x120
+do_syscall_64+0xef/0x1540
+entry_SYSCALL_64_after_hwframe+0x77/0x7f
+```
+
+`strace -p <pid> -f`:
+
+```
+[pid <child-A>] epoll_wait(6 <unfinished ...>
+[pid <child-B>] epoll_wait(3
+```
+
+**Two threads**, both parked in `epoll_wait` on
+distinct epoll fds. Both blocked, neither making
+progress.
+
+### Subactor file-descriptor table
+
+```
+fd=0,1,2     stdio
+fd=3         eventpoll [watches fd 4]
+fd=4 ↔ fd=5  unix STREAM (CONNECTED) — internal pair
+fd=6         eventpoll [watches fds 7, 9]
+fd=7 ↔ fd=8  unix STREAM (CONNECTED) — internal pair
+fd=9 ↔ fd=10 unix STREAM (CONNECTED) — internal pair
+```
+
+Confirmed via `ss -xp` peer-inode lookup: **all 6 unix
+sockets are internal socketpairs** (peer in same pid).
+
+**Critical**: zero TCP/IPv4/IPv6 sockets, despite
+`--tpt-proto=tcp`:
+
+```
+$ sudo lsof -p <subactor> | grep -iE 'TCP|IPv'
+(empty)
+$ sudo ss -tnp | grep <subactor>
+(empty)
+```
+
+**The subactor never opened a TCP connection back to
+the parent.**
+
+## Diagnosis
+
+The subactor reaches `_actor_child_main` →
+`_trio_main(actor)` →
+`run_as_asyncio_guest(trio_main)`. Code path
+(`tractor.spawn._entry`):
+
+```python
+if infect_asyncio:
+    actor._infected_aio = True
+    run_as_asyncio_guest(trio_main)   # ←  this branch
+else:
+    trio.run(trio_main)
+```
+
+`run_as_asyncio_guest` (`tractor.to_asyncio`):
+
+```python
+def run_as_asyncio_guest(trio_main, ...):
+    async def aio_main(trio_main):
+        loop = asyncio.get_running_loop()
+        trio_done_fute = asyncio.Future()
+        ...
+        trio.lowlevel.start_guest_run(
+            trio_main,
+            run_sync_soon_threadsafe=loop.call_soon_threadsafe,
+            done_callback=trio_done_callback,
+        )
+        out = await asyncio.shield(trio_done_fute)
+        return out.unwrap()
+    ...
+    return asyncio.run(aio_main(trio_main))
+```
+
+Expected flow:
+1. `asyncio.run(aio_main(...))` — boots fresh asyncio
+   loop in calling thread.
+2. `aio_main` calls `trio.lowlevel.start_guest_run(...)`
+   — initializes trio's I/O manager, schedules first
+   trio slice via `loop.call_soon_threadsafe`.
+3. asyncio loop dispatches the callback → trio runs a
+   slice → yields back via `call_soon_threadsafe`.
+4. Trio's `async_main` (the user function) runs →
+   `Channel.from_addr(parent_addr)` → TCP connect to
+   parent.
+
+What we observe instead:
+- 2 threads in `epoll_wait` (one trio epoll, one
+  asyncio epoll, both inactive)
+- 6 unix-socket fds (3 socketpairs: trio
+  wakeup-fd-pair, asyncio wakeup-fd-pair, trio kicker
+  socketpair)
+- ZERO TCP — `Channel.from_addr` never ran
+
+Most likely cause: **trio's guest-run scheduling
+callback didn't get dispatched by asyncio's loop in
+the forked child**, so trio's `async_main` never
+executes past trio bootstrap, and the
+parent-IPC-connect step is never reached.
+
+## Fork-survival risk surface (hypothesis)
+
+`trio.lowlevel.start_guest_run` builds Python-level
+closures + signal handlers + wakeup-fd registrations
+that depend on:
+
+- The asyncio event loop's `call_soon_threadsafe`
+  thread-id matching the loop owner thread.
+- Process-wide signal-wakeup-fd state
+  (`signal.set_wakeup_fd`).
+- Trio's `KIManager` SIGINT handler.
+
+Under `main_thread_forkserver`, the fork happens from
+a worker thread that has **never entered trio**
+(intentional — trio-free launchpad). But the FORKED
+child then tries to bring up BOTH asyncio AND
+trio-as-guest fresh from this trio-free thread. The
+asyncio loop boots fine; trio's `start_guest_run`
+initializes BUT the cross-loop dispatch (asyncio
+queue → trio slice) appears to silently fail to wire
+up.
+
+Two more hypotheses worth probing:
+
+1. **Wakeup-fd contention**: asyncio installs
+   `signal.set_wakeup_fd(<own_pair>)`. trio's
+   guest-run also wants a wakeup-fd. Whoever installs
+   second wins; the loser's `epoll_wait` no longer
+   wakes on signals. Combined with the `asyncio.shield(
+   trio_done_fute)` + `asyncio.CancelledError`
+   handling in `run_as_asyncio_guest`, a missed signal
+   delivery could explain the indefinite park.
+
+2. **Trio kicker socketpair race**: trio's I/O manager
+   uses an internal `socket.socketpair()` to "kick"
+   itself out of `epoll_wait` when a non-IO task needs
+   scheduling. In guest mode, the kicker is still
+   present but is supposed to be triggered via the
+   asyncio dispatch. If the kicker write never gets
+   issued by asyncio's callback, trio's epoll never
+   wakes.
+
+## Confirmed via py-spy (live capture)
+
+After detaching `strace` (ptrace is exclusive — that's
+why `py-spy` returns EPERM if strace is attached):
+
+```
+Thread <pid> (idle): "main-thread-forkserver[asyncio_actor]"
+    select (selectors.py:452)                          # asyncio epoll
+    _run_once (asyncio/base_events.py:2012)
+    run_forever (asyncio/base_events.py:683)
+    run_until_complete (asyncio/base_events.py:712)
+    run (asyncio/runners.py:118)
+    run (asyncio/runners.py:195)
+    run_as_asyncio_guest (tractor/to_asyncio.py:1770)
+    _trio_main (tractor/spawn/_entry.py:160)
+    _actor_child_main (tractor/_child.py:72)
+    _child_target (tractor/spawn/_main_thread_forkserver.py:910)
+    _worker (tractor/spawn/_main_thread_forkserver.py:605)
+    [thread bootstrap]
+
+Thread <pid+1> (idle): "Trio thread 14"
+    get_events (trio/_core/_io_epoll.py:245)           # trio epoll
+    get_events (trio/_core/_run.py:1678)
+    capture (outcome/_impl.py:67)
+    _handle_job (trio/_core/_thread_cache.py:173)
+    _work (trio/_core/_thread_cache.py:196)
+    [thread bootstrap]
+```
+
+This data **rewrites the diagnosis**: trio guest-run
+isn't broken across the fork — it's working as designed.
+The two threads ARE the canonical guest-run architecture:
+
+1. **Asyncio main loop** runs in the lead thread. Parked
+   in `selectors.EpollSelector.select(timeout=-1)` —
+   waiting indefinitely for ANY callback to be queued.
+2. **Trio's I/O manager** offloads `get_events`
+   (`epoll_wait`) onto a `trio._core._thread_cache`
+   worker thread. The worker calls
+   `outcome.capture(get_events)` and parks in
+   `epoll_wait(timeout=86400)`.
+3. When trio I/O fires (or its kicker socketpair gets a
+   write), the worker returns from `epoll_wait`,
+   delivers the result via `_handle_job`'s `deliver`
+   callback, which schedules the next trio slice on
+   asyncio via `loop.call_soon_threadsafe`.
+
+The fact that the trio thread is *already* in
+`_thread_cache._handle_job` doing `capture(get_events)`
+means **trio's scheduler HAS started** — the bridge
+asyncio↔trio is wired correctly post-fork.
+
+So `async_main` DID run far enough to register some
+trio task that's now awaiting I/O. The question
+becomes: **what is `async_main` waiting on?**
+
+Process state confirms it's NOT waiting on the TCP
+connect to parent:
+
+```
+$ sudo lsof -p <subactor> | grep -iE 'TCP|IPv'
+(empty)
+$ sudo ss -tnp | grep <subactor>
+(empty)
+```
+
+`Channel.from_addr(parent_addr)` — the very first
+thing `async_main` does — was never reached, OR was
+reached but errored before `socket()` was called. The
+parent (running `ipc_server.wait_for_peer`) waits
+forever for the connection; it never comes.
+
+## Refined hypothesis
+
+`async_main` is stalled in some PRE-`Channel.from_addr`
+checkpoint. Candidates:
+
+1. **`get_console_log` / logger init** — called early in
+   `_trio_main` if `actor.loglevel is not None`. Logging
+   setup involves file/handler init that could block on
+   something fork-inherited (e.g. a stale lock).
+2. **`debug.maybe_init_greenback`** — `start_guest_run`
+   includes a check (`if debug_mode(): assert 0` —
+   currently asserts unsupported). For non-debug mode
+   this is bypassed but related machinery may run.
+3. **Stackscope SIGUSR1 handler install** — gated on
+   `_debug_mode` OR `TRACTOR_ENABLE_STACKSCOPE` env-var.
+   The `enable_stack_on_sig()` path captures a trio
+   token via `trio.lowlevel.current_trio_token()` —
+   could block under guest mode.
+4. **Initial `await trio.sleep(0)` / first checkpoint**
+   in `async_main` before reaching the
+   `Channel.from_addr` line. Under guest mode, if the
+   FIRST `call_soon_threadsafe` callback never gets
+   processed by asyncio, trio's first slice never
+   completes — but the worker thread WOULD still be in
+   `epoll_wait` having been started by trio's I/O
+   manager init.
+
+## Confirming `async_main`'s parked location
+
+Add temporary logging at the top of `Actor.async_main`:
+
+```python
+# tractor/runtime/_runtime.py around line 855
+async def async_main(self, parent_addr=None):
+    log.devx('async_main: ENTERED')                # marker A
+    try:
+        log.devx('async_main: pre-Channel.from_addr')  # marker B
+        chan = await Channel.from_addr(
+            addr=wrap_address(parent_addr)
+        )
+        log.devx('async_main: post-Channel.from_addr')  # marker C
+        ...
+```
+
+Re-run the test with `--ll=devx`. The last marker logged
+tells us exactly where `async_main` parked. If only A
+fires, the issue is between A and B (logger init,
+stackscope, etc.). If A and B fire but not C, it's in
+`Channel.from_addr` (DNS, socket creation, connect).
+
+## Related sibling bug
+
+`tests/test_multi_program.py::test_register_duplicate_name`
+hangs under the same backend with a DIFFERENT
+fingerprint:
+
+- Subactor at 100% CPU (busy-loop), not parked
+- `recvfrom(6, "", 65536, 0, NULL, NULL) = 0` repeating
+  with no `epoll_wait` in between
+- fd=6 is one of trio's internal AF_UNIX
+  socketpair fds (the kicker mechanism)
+
+Distinct root cause — possibly trio's kicker socketpair
+inheriting a half-closed state across the fork — but
+shares the broader theme: **trio internal-state
+initialization isn't fully fork-safe under
+`main_thread_forkserver`** for the more exotic
+dispatch paths.
+
+## Workarounds (until fix lands)
+
+1. **Skip-mark on the fork backend** — temporarily mark
+   `tests/test_infected_asyncio.py` with
+   `pytest.mark.skipon_spawn_backend('main_thread_forkserver',
+   reason='infect_asyncio + fork interaction broken,
+   see ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md')`.
+   Lets the rest of the test suite run green while
+   this is being fixed properly.
+
+2. **Run infected-asyncio tests under the `trio`
+   backend only** — they don't exercise fork
+   semantics, so they won't hit this bug.
+
+## Investigation next steps
+
+In rough priority:
+
+1. Catch the hang alive again, **detach strace**,
+   `py-spy --locals` the subactor — confirm trio
+   thread is NOT yet at `async_main`.
+2. Diff `start_guest_run` setup pre-fork vs post-fork
+   by adding `log.devx()` markers in
+   `tractor.to_asyncio.run_as_asyncio_guest::aio_main`
+   at:
+   - asyncio loop bringup
+   - immediately before `start_guest_run`
+   - immediately after `start_guest_run`
+   - inside the `trio_done_callback` registration
+3. Check whether the asyncio loop dispatches ANY
+   callbacks in the forked child — instrument
+   `loop.call_soon_threadsafe` (e.g. monkey-patch
+   `loop._call_soon` to log).
+4. If steps 1–3 confirm that asyncio's queue is
+   stuck, look at whether the asyncio event-loop
+   policy or selector is being inherited from a
+   pre-fork (parent-process) state in a way that
+   breaks the new loop.
+
+## See also
+
+- [#379](https://github.com/goodboy/tractor/issues/379) — subint umbrella
+- [#451](https://github.com/goodboy/tractor/issues/451) — Mode-A cancel-cascade hang
+- `ai/conc-anal/fork_thread_semantics_execution_vs_memory.md`
+- `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
+- python-trio/trio#1614 — trio + fork hazards

ai/conc-anal/spawn_time_boot_death_dup_name_issue.md

@@ -0,0 +1,142 @@
+# Spawn-time boot-death (`rc=2`) under rapid same-name spawn against a registrar
+
+## Symptom
+
+Spawning N (≥4) sub-actors with the **same name** in tight
+succession against a daemon registrar surfaces as
+`ActorFailure: Sub-actor (...) died during boot (rc=2)
+before completing parent-handshake`.
+
+```
+tests/discovery/test_multi_program.py
+  ::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]
+```
+
+```
+tractor._exceptions.ActorFailure:
+  Sub-actor ('doggy', '<uuid>') died during boot (rc=2)
+  before completing parent-handshake.
+    proc: <_ForkedProc pid=<n> returncode=None>
+```
+
+The `proc` repr shows `returncode=None` because the repr is
+captured before `proc.wait()` returns; the actual
+`os.WEXITSTATUS == 2` is reported via `result['died']` in the
+race-helper.
+
+## When it surfaces
+
+- N=2 (`n_dups=2`): **always passes**.
+- N=4 (`n_dups=4`): **consistent fail** under both `tpt-proto=tcp`
+  and `tpt-proto=uds`, MTF backend.
+- N=8 (`n_dups=8`): **passes** (counter-intuitive — see "racing
+  windows").
+- Non-MTF backends: not yet exercised systematically.
+
+## What previously masked it
+
+Pre the spawn-time `wait_for_peer_or_proc_death` race-helper
+(in `tractor.spawn._spawn`), the parent's `start_actor` flow
+ended with a bare:
+
+```python
+event, chan = await ipc_server.wait_for_peer(uid)
+```
+
+That awaits an unsignalled `trio.Event` on `_peer_connected[uid]`.
+If the sub-actor process **dies during boot** (before its
+runtime executes the parent-callback handshake that sets the
+event), the wait parks forever. The dead proc becomes a zombie
+because no one ever calls `proc.wait()` to reap it.
+
+In test contexts the failure presented as a hang or a much
+later `trio.TooSlowError` from an outer `fail_after`. In
+production it'd present as a parent that never makes progress
+past `start_actor`. The death itself was silently masked.
+
+## What surfaces it now
+
+`tractor.spawn._spawn.wait_for_peer_or_proc_death` (used by
+`_main_thread_forkserver_proc`) races the handshake-wait
+against `proc.wait()`. The race-helper raises `ActorFailure`
+on death-first instead of parking, exposing the rc=2.
+
+## Hypothesis: registrar-side same-name contention
+
+The test spawns N actors with name `doggy` sequentially:
+
+```python
+for i in range(n_dups):
+    p: Portal = await an.start_actor('doggy')
+    portals.append(p)
+```
+
+Each spawned doggy:
+
+1. Forks via the forkserver.
+2. Boots its runtime in `_actor_child_main`.
+3. Connects back to the parent for handshake.
+4. Connects to the daemon registrar to call `register_actor`.
+5. Enters its RPC msg-loop.
+
+Step (4) is where the same-name contention lives. The
+registrar's `register_actor` (in
+`tractor.discovery._registry`) accepts duplicate names
+(stores `(name, uuid) -> addr`), but its internal bookkeeping
+may have a non-trivial check (e.g. `wait_for_actor` resolution,
+`_addrs2aids` map updates) that errors out under specific
+ordering between the existing entry and the incoming one.
+
+`rc=2 == os.WEXITSTATUS == 2` corresponds to `sys.exit(2)`
+in the doggy process — typically reached via an unhandled
+exception that's translated to exit code 2 by Python's top-
+level (e.g. `argparse` errors use 2; `SystemExit(2)` etc.).
+So the doggy is hitting an explicit exit path during
+`register_actor` or just-after.
+
+The non-monotonic shape (N=2 OK, N=4 BAD, N=8 OK) suggests a
+specific timing window — likely "the 3rd register-RPC arrives
+while the 1st-or-2nd is in some intermediate state". With
+N=8, the additional procs widen the registration spread
+enough that no two land in the conflicting window.
+
+## Where to dig next
+
+- Add per-actor logging in `_actor_child_main` and
+  `register_actor` to surface the actual exception that
+  triggers the rc=2 exit. Currently the doggy dies before
+  the parent ever sees its stderr (forkserver doesn't
+  marshal child stdio back).
+- Race-test the registrar's `register_actor` /
+  `unregister_actor` /  `wait_for_actor` against same-name
+  concurrent calls in isolation (no spawn).
+- Consider whether `register_actor` should be idempotent
+  under same-name re-register or should explicitly reject
+  same-name (and ideally with a clear `RemoteActorError`,
+  not `sys.exit(2)`).
+
+## Test-suite handling
+
+Currently:
+
+- `tests/discovery/test_multi_program.py
+  ::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]`
+  is `pytest.mark.xfail(strict=False, reason=...)` to keep
+  the suite green while this issue is investigated.
+- `n_dups=2` and `n_dups=8` continue to validate the
+  cancel-cascade hard-kill escalation.
+
+Once the underlying race is understood + fixed, drop the
+xfail.
+
+## Related work
+
+- The cancel-cascade fix that introduced this regression
+  test:
+  `tractor/_exceptions.py:ActorTooSlowError`,
+  `tractor/runtime/_supervise.py:_try_cancel_then_kill`,
+  `tractor/runtime/_portal.py:Portal.cancel_actor(
+   raise_on_timeout=...)`.
+- The spawn-time death-detection that exposed this:
+  `tractor/spawn/_spawn.py:wait_for_peer_or_proc_death`,
+  used by `tractor/spawn/_main_thread_forkserver.py`.

ai/conc-anal/subint_cancel_delivery_hang_issue.md

@@ -0,0 +1,161 @@
+# `subint` backend: parent trio loop parks after subint teardown (Ctrl-C works; not a CPython-level issue)
+
+Follow-up to the Phase B subint spawn-backend PR (see
+`tractor.spawn._subint`, issue #379). Distinct from the
+`subint_sigint_starvation_issue.md` (SIGINT-unresponsive
+starvation hang): this one is **Ctrl-C-able**, which means
+it's *not* the shared-GIL-hostage class and is ours to fix
+from inside tractor rather than waiting on upstream CPython
+/ msgspec progress.
+
+## TL;DR
+
+After a stuck-subint subactor is torn down via the
+hard-kill path, a parent-side trio task parks on an
+*orphaned resource* (most likely a `chan.recv()` /
+`process_messages` loop on the now-dead subint's IPC
+channel) and waits forever for bytes that can't arrive —
+because the channel was torn down without emitting a clean
+EOF/`BrokenResourceError` to the waiting receiver.
+
+Unlike `subint_sigint_starvation_issue.md`, the main trio
+loop **is** iterating normally — SIGINT delivers cleanly
+and the test unhangs. But absent Ctrl-C, the test suite
+wedges indefinitely.
+
+## Symptom
+
+Running `test_subint_non_checkpointing_child` under
+`--spawn-backend=subint` (in
+`tests/test_subint_cancellation.py`):
+
+1. Test spawns a subactor whose main task runs
+   `threading.Event.wait(1.0)` in a loop — releases the
+   GIL but never inserts a trio checkpoint.
+2. Parent does `an.cancel_scope.cancel()`. Our
+   `subint_proc` cancel path fires: soft-kill sends
+   `Portal.cancel_actor()` over the live IPC channel →
+   subint's trio loop *should* process the cancel msg on
+   its IPC dispatcher task (since the GIL releases are
+   happening).
+3. Expected: subint's `trio.run()` unwinds, driver thread
+   exits naturally, parent returns.
+4. Actual: parent `trio.run()` never completes. Test
+   hangs past its `trio.fail_after()` deadline.
+
+## Evidence
+
+### `strace` on the hung pytest process during SIGINT
+
+```
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(17, "\2", 1)                      = 1
+```
+
+Contrast with the SIGINT-starvation hang (see
+`subint_sigint_starvation_issue.md`) where that same
+`write()` returned `EAGAIN`. Here the SIGINT byte is
+written successfully → Python's signal handler pipe is
+being drained → main trio loop **is** iterating → SIGINT
+gets turned into `trio.Cancelled` → the test unhangs (if
+the operator happens to be there to hit Ctrl-C).
+
+### Stack dump (via `tractor.devx.dump_on_hang`)
+
+Single main thread visible, parked in
+`trio._core._io_epoll.get_events` inside `trio.run` at the
+test's `trio.run(...)` call site. No subint driver thread
+(subint was destroyed successfully — this is *after* the
+hard-kill path, not during it).
+
+## Root cause hypothesis
+
+Most consistent with the evidence: a parent-side trio
+task is awaiting a `chan.recv()` / `process_messages` loop
+on the dead subint's IPC channel. The sequence:
+
+1. Soft-kill in `subint_proc` sends `Portal.cancel_actor()`
+   over the channel. The subint's trio dispatcher *may* or
+   may not have processed the cancel msg before the subint
+   was destroyed — timing-dependent.
+2. Hard-kill timeout fires (because the subint's main
+   task was in `threading.Event.wait()` with no trio
+   checkpoint — cancel-msg processing couldn't race the
+   timeout).
+3. Driver thread abandoned, `_interpreters.destroy()`
+   runs. Subint is gone.
+4. But the parent-side trio task holding a
+   `chan.recv()` / `process_messages` loop against that
+   channel was **not** explicitly cancelled. The channel's
+   underlying socket got torn down, but without a clean
+   EOF delivered to the waiting recv, the task parks
+   forever on `trio.lowlevel.wait_readable` (or similar).
+
+This matches the "main loop fine, task parked on
+orphaned I/O" signature.
+
+## Why this is ours to fix (not CPython's)
+
+- Main trio loop iterates normally → GIL isn't starved.
+- SIGINT is deliverable → not a signal-pipe-full /
+  wakeup-fd contention scenario.
+- The hang is in *our* supervision code, specifically in
+  how `subint_proc` tears down its side of the IPC when
+  the subint is abandoned/destroyed.
+
+## Possible fix directions
+
+1. **Explicit parent-side channel abort on subint
+   abandon.** In `subint_proc`'s teardown block, after the
+   hard-kill timeout fires, explicitly close the parent's
+   end of the IPC channel to the subint. Any waiting
+   `chan.recv()` / `process_messages` task sees
+   `BrokenResourceError` (or `ClosedResourceError`) and
+   unwinds.
+2. **Cancel parent-side RPC tasks tied to the dead
+   subint's channel.** The `Actor._rpc_tasks` / nursery
+   machinery should have a handle on any
+   `process_messages` loops bound to a specific peer
+   channel. Iterate those and cancel explicitly.
+3. **Bound the top-level `await actor_nursery
+   ._join_procs.wait()` shield in `subint_proc`** (same
+   pattern as the other bounded shields the hard-kill
+   patch added). If the nursery never sets `_join_procs`
+   because a child task is parked, the bound would at
+   least let the teardown proceed.
+
+Of these, (1) is the most surgical and directly addresses
+the root cause. (2) is a defense-in-depth companion. (3)
+is a band-aid but cheap to add.
+
+## Current workaround
+
+None in-tree. The test's `trio.fail_after()` bound
+currently fires and raises `TooSlowError`, so the test
+visibly **fails** rather than hangs — which is
+intentional (an unbounded cancellation-audit test would
+defeat itself). But in interactive test runs the operator
+has to hit Ctrl-C to move past the parked state before
+pytest reports the failure.
+
+## Reproducer
+
+```
+./py314/bin/python -m pytest \
+  tests/test_subint_cancellation.py::test_subint_non_checkpointing_child \
+  --spawn-backend=subint --tb=short --no-header -v
+```
+
+Expected: hangs until `trio.fail_after(15)` fires, or
+Ctrl-C unwedges it manually.
+
+## References
+
+- `tractor.spawn._subint.subint_proc` — current subint
+  teardown code; see the `_HARD_KILL_TIMEOUT` bounded
+  shields + `daemon=True` driver-thread abandonment
+  (commit `b025c982`).
+- `ai/conc-anal/subint_sigint_starvation_issue.md` — the
+  sibling CPython-level hang (GIL-starvation,
+  SIGINT-unresponsive) which is **not** this issue.
+- Phase B tracking: issue #379.

ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md

@@ -0,0 +1,337 @@
+# `os.fork()` from a non-main sub-interpreter aborts the child (CPython refuses post-fork cleanup)
+
+Third `subint`-class analysis in this project. Unlike its
+two siblings (`subint_sigint_starvation_issue.md`,
+`subint_cancel_delivery_hang_issue.md`), this one is not a
+hang — it's a **hard CPython-level refusal** of an
+experimental spawn strategy we wanted to try.
+
+## TL;DR
+
+An in-process sub-interpreter cannot be used as a
+"launchpad" for `os.fork()` on current CPython. The fork
+syscall succeeds in the parent, but the forked CHILD
+process is aborted immediately by CPython's post-fork
+cleanup with:
+
+```
+Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
+```
+
+This is enforced by a hard `PyStatus_ERR` gate in
+`Python/pystate.c`. The CPython devs acknowledge the
+fragility with an in-source comment (`// Ideally we could
+guarantee tstate is running main.`) but provide no
+mechanism to satisfy the precondition from user code.
+
+**Implication for tractor**: the `subint_fork` backend
+sketched in `tractor.spawn._subint_fork` is structurally
+dead on current CPython. The submodule is kept as
+documentation of the attempt; `--spawn-backend=subint_fork`
+raises `NotImplementedError` pointing here.
+
+## Context — why we tried this
+
+The motivation is issue #379's "Our own thoughts, ideas
+for `fork()`-workaround/hacks..." section. The existing
+trio-backend (`tractor.spawn._trio.trio_proc`) spawns
+subactors via `trio.lowlevel.open_process()` → ultimately
+`posix_spawn()` or `fork+exec`, from the parent's main
+interpreter that is currently running `trio.run()`. This
+brushes against a known-fragile interaction between
+`trio` and `fork()` tracked in
+[python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614)
+and siblings — mostly mitigated in `tractor`'s case only
+incidentally (we `exec()` immediately post-fork).
+
+The idea was:
+
+1. Create a subint that has *never* imported `trio`.
+2. From a worker thread in that subint, call `os.fork()`.
+3. In the child, `execv()` back into
+   `python -m tractor._child` — same as `trio_proc` does.
+4. The fork is from a trio-free context → trio+fork
+   hazards avoided regardless of downstream behavior.
+
+The parent-side orchestration (`ipc_server.wait_for_peer`,
+`SpawnSpec`, `Portal` yield) would reuse
+`trio_proc`'s flow verbatim, with only the subproc-spawn
+mechanics swapped.
+
+## Symptom
+
+Running the prototype (`tractor.spawn._subint_fork.subint_fork_proc`,
+see git history prior to the stub revert) on py3.14:
+
+```
+Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
+Python runtime state: initialized
+
+Current thread 0x00007f6b71a456c0 [subint-fork-lau] (most recent call first):
+  File "<script>", line 2 in <module>
+<script>:2: DeprecationWarning: This process (pid=802985) is multi-threaded, use of fork() may lead to deadlocks in the child.
+```
+
+Key clues:
+
+- The **`DeprecationWarning`** fires in the parent (before
+  fork completes) — fork *is* executing, we get that far.
+- The **`Fatal Python error`** comes from the child — it
+  aborts during CPython's post-fork C initialization
+  before any user Python runs in the child.
+- The thread name `subint-fork-lau[nchpad]` is ours —
+  confirms the fork is being called from the launchpad
+  subint's driver thread.
+
+## CPython source walkthrough
+
+### Call site — `Modules/posixmodule.c:728-793`
+
+The post-fork-child hook CPython runs in the child process:
+
+```c
+void
+PyOS_AfterFork_Child(void)
+{
+    PyStatus status;
+    _PyRuntimeState *runtime = &_PyRuntime;
+
+    // re-creates runtime->interpreters.mutex (HEAD_UNLOCK)
+    status = _PyRuntimeState_ReInitThreads(runtime);
+    ...
+
+    PyThreadState *tstate = _PyThreadState_GET();
+    _Py_EnsureTstateNotNULL(tstate);
+
+    ...
+
+    // Ideally we could guarantee tstate is running main.   ← !!!
+    _PyInterpreterState_ReinitRunningMain(tstate);
+
+    status = _PyEval_ReInitThreads(tstate);
+    ...
+
+    status = _PyInterpreterState_DeleteExceptMain(runtime);
+    if (_PyStatus_EXCEPTION(status)) {
+        goto fatal_error;
+    }
+    ...
+
+fatal_error:
+    Py_ExitStatusException(status);
+}
+```
+
+The `// Ideally we could guarantee tstate is running
+main.` comment is a flashing warning sign — the CPython
+devs *know* this path is fragile when fork is called from
+a non-main subint, but they've chosen to abort rather than
+silently corrupt state. Arguably the right call.
+
+### The refusal — `Python/pystate.c:1035-1075`
+
+```c
+/*
+ * Delete all interpreter states except the main interpreter.  If there
+ * is a current interpreter state, it *must* be the main interpreter.
+ */
+PyStatus
+_PyInterpreterState_DeleteExceptMain(_PyRuntimeState *runtime)
+{
+    struct pyinterpreters *interpreters = &runtime->interpreters;
+
+    PyThreadState *tstate = _PyThreadState_Swap(runtime, NULL);
+    if (tstate != NULL && tstate->interp != interpreters->main) {
+        return _PyStatus_ERR("not main interpreter");       ← our error
+    }
+
+    HEAD_LOCK(runtime);
+    PyInterpreterState *interp = interpreters->head;
+    interpreters->head = NULL;
+    while (interp != NULL) {
+        if (interp == interpreters->main) {
+            interpreters->main->next = NULL;
+            interpreters->head = interp;
+            interp = interp->next;
+            continue;
+        }
+
+        // XXX Won't this fail since PyInterpreterState_Clear() requires
+        // the "current" tstate to be set?
+        PyInterpreterState_Clear(interp);  // XXX must activate?
+        zapthreads(interp);
+        ...
+    }
+    ...
+}
+```
+
+The comment in the docstring (`If there is a current
+interpreter state, it *must* be the main interpreter.`) is
+the formal API contract. The `XXX` comments further in
+suggest the CPython team is already aware this function
+has latent issues even in the happy path.
+
+## Chain summary
+
+1. Our launchpad subint's driver OS-thread calls
+   `os.fork()`.
+2. `fork()` succeeds. Child wakes up with:
+   - The parent's full memory image (including all
+     subints).
+   - Only the *calling* thread alive (the driver thread).
+   - `_PyThreadState_GET()` on that thread returns the
+     **launchpad subint's tstate**, *not* main's.
+3. CPython runs `PyOS_AfterFork_Child()`.
+4. It reaches `_PyInterpreterState_DeleteExceptMain()`.
+5. Gate check fails: `tstate->interp != interpreters->main`.
+6. `PyStatus_ERR("not main interpreter")` → `fatal_error`
+   goto → `Py_ExitStatusException()` → child aborts.
+
+Parent-side consequence: `os.fork()` in the subint
+bootstrap returned successfully with the child's PID, but
+the child died before connecting back. Our parent's
+`ipc_server.wait_for_peer(uid)` would hang forever — the
+child never gets to `_actor_child_main`.
+
+## Definitive answer to "Open Question 1"
+
+From the (now-stub) `subint_fork_proc` docstring:
+
+> Does CPython allow `os.fork()` from a non-main
+> sub-interpreter under the legacy config?
+
+**No.** Not in a usable-by-user-code sense. The fork
+syscall is not blocked, but the child cannot survive
+CPython's post-fork initialization. This is enforced, not
+accidental, and the CPython devs have acknowledged the
+fragility in-source.
+
+## What we'd need from CPython to unblock
+
+Any one of these, from least-to-most invasive:
+
+1. **A pre-fork hook mechanism** that lets user code (or
+   tractor itself via `os.register_at_fork(before=...)`)
+   swap the current tstate to main before fork runs. The
+   swap would need to work across the subint→main
+   boundary, which is the actual hard part —
+   `_PyThreadState_Swap()` exists but is internal.
+
+2. **A `_PyInterpreterState_DeleteExceptFor(tstate->interp)`
+   variant** that cleans up all *other* subints while
+   preserving the calling subint's state. Lets the child
+   continue executing in the subint after fork; a
+   subsequent `execv()` clears everything at the OS
+   level anyway.
+
+3. **A cleaner error** than `Fatal Python error` aborting
+   the child. Even without fixing the underlying
+   capability, a raised Python-level exception in the
+   parent's `fork()` call (rather than a silent child
+   abort) would at least make the failure mode
+   debuggable.
+
+## Upstream-report draft (for CPython issue tracker)
+
+### Title
+
+> `os.fork()` from a non-main sub-interpreter aborts the
+> child with a fatal error in `PyOS_AfterFork_Child`; can
+> we at least make it a clean `RuntimeError` in the
+> parent?
+
+### Body
+
+> **Version**: Python 3.14.x
+>
+> **Summary**: Calling `os.fork()` from a thread currently
+> executing inside a sub-interpreter causes the forked
+> child process to abort during CPython's post-fork
+> cleanup, with the following output in the child:
+>
+> ```
+> Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
+> ```
+>
+> From the **parent's** point of view the fork succeeded
+> (returned a valid child PID). The failure is completely
+> opaque to parent-side Python code — unless the parent
+> does `os.waitpid()` it won't even notice the child
+> died.
+>
+> **Root cause** (as I understand it from reading sources):
+> `Modules/posixmodule.c::PyOS_AfterFork_Child()` calls
+> `_PyInterpreterState_DeleteExceptMain()` with a
+> precondition that `_PyThreadState_GET()->interp` be the
+> main interpreter. When `fork()` is called from a thread
+> executing inside a subinterpreter, the child wakes up
+> with its tstate still pointing at the subint, and the
+> gate in `Python/pystate.c:1044-1047` fails.
+>
+> A comment in the source
+> (`Modules/posixmodule.c:753` — `// Ideally we could
+> guarantee tstate is running main.`) suggests this is a
+> known-fragile path rather than an intentional
+> invariant.
+>
+> **Use case**: I was experimenting with using a
+> sub-interpreter as a "fork launchpad" — have a subint
+> that has never imported `trio`, call `os.fork()` from
+> that subint's thread, and in the child `execv()` back
+> into a fresh Python interpreter process. The goal was
+> to sidestep known issues with `trio` + `fork()`
+> interaction (see
+> [python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
+> by guaranteeing the forking context had never been
+> "contaminated" by trio's imports or globals. This
+> approach would allow `trio`-using applications to
+> combine `fork`-based subprocess spawning with
+> per-worker `trio.run()` runtimes — a fairly common
+> pattern that currently requires workarounds.
+>
+> **Request**:
+>
+> Ideally: make fork-from-subint work (e.g., by swapping
+> the caller's tstate to main in the pre-fork hook), or
+> provide a `_PyInterpreterState_DeleteExceptFor(interp)`
+> variant that permits the caller's subint to survive
+> post-fork so user code can subsequently `execv()`.
+>
+> Minimally: convert the fatal child-side abort into a
+> clean `RuntimeError` (or similar) raised in the
+> parent's `fork()` call. Even if the capability isn't
+> expanded, the failure mode should be debuggable by
+> user-code in the parent — right now it's a silent
+> child death with an error message buried in the
+> child's stderr that parent code can't programmatically
+> see.
+>
+> **Related**: PEP 684 (per-interpreter GIL), PEP 734
+> (`concurrent.interpreters` public API). The private
+> `_interpreters` module is what I used to create the
+> launchpad — behavior is the same whether using
+> `_interpreters.create('legacy')` or
+> `concurrent.interpreters.create()` (the latter was not
+> tested but the gate is identical).
+>
+> Happy to contribute a minimal reproducer + test case if
+> this is something the team wants to pursue.
+
+## References
+
+- `Modules/posixmodule.c:728` —
+  [`PyOS_AfterFork_Child`](https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L728)
+- `Python/pystate.c:1040` —
+  [`_PyInterpreterState_DeleteExceptMain`](https://github.com/python/cpython/blob/main/Python/pystate.c#L1040)
+- PEP 684 (per-interpreter GIL):
+  <https://peps.python.org/pep-0684/>
+- PEP 734 (`concurrent.interpreters` public API):
+  <https://peps.python.org/pep-0734/>
+- [python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614)
+  — the original motivation for the launchpad idea.
+- tractor issue #379 — "Our own thoughts, ideas for
+  `fork()`-workaround/hacks..." section where this was
+  first sketched.
+- `tractor.spawn._subint_fork` — in-tree stub preserving
+  the attempted impl's shape in git history.

ai/conc-anal/subint_fork_from_main_thread_smoketest.py

@@ -0,0 +1,375 @@
+#!/usr/bin/env python3
+'''
+Standalone CPython-level feasibility check for the "main-interp
+worker-thread forkserver + subint-hosted trio" architecture
+proposed as a workaround to the CPython-level refusal
+documented in
+`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
+
+Purpose
+-------
+Deliberately NOT a `tractor` test. Zero `tractor` imports.
+Uses `_interpreters` (private stdlib) + `os.fork()` directly so
+the signal is unambiguous — pass/fail here is a property of
+CPython alone, independent of our runtime.
+
+Run each scenario in isolation; the child's fate is observable
+only via `os.waitpid()` of the parent and the scenario's own
+status prints.
+
+Scenarios (pick one with `--scenario <name>`)
+---------------------------------------------
+
+- `control_subint_thread_fork` — the KNOWN-BROKEN case we
+  documented in `subint_fork_blocked_by_cpython_post_fork_issue.md`:
+  drive a subint from a thread, call `os.fork()` inside its
+  `_interpreters.exec()`, watch the child abort. **Included as
+  a control** — if this scenario DOESN'T abort the child, our
+  analysis is wrong and we should re-check everything.
+
+- `main_thread_fork` — baseline sanity. Call `os.fork()` from
+  the process's main thread. Must always succeed; if this
+  fails something much bigger is broken.
+
+- `worker_thread_fork` — the architectural assertion. Spawn a
+  regular `threading.Thread` (attached to main interp, NOT a
+  subint), have IT call `os.fork()`. Child should survive
+  post-fork cleanup.
+
+- `full_architecture` — end-to-end: main-interp worker thread
+  forks. In the child, fork-thread (still main-interp) creates
+  a subint, drives a second worker thread inside it that runs
+  a trivial `trio.run()`. Validates the "root runtime lives in
+  a subint in the child" piece of the proposed arch.
+
+All scenarios print a self-contained pass/fail banner. Exit
+code 0 on expected outcome (which for `control_*` means "child
+aborted", not "child succeeded"!).
+
+Requires Python 3.14+.
+
+Usage
+-----
+::
+
+    python subint_fork_from_main_thread_smoketest.py \\
+        --scenario main_thread_fork
+
+    python subint_fork_from_main_thread_smoketest.py \\
+        --scenario full_architecture
+
+'''
+from __future__ import annotations
+import argparse
+import os
+import sys
+import threading
+import time
+
+
+# Hard-require py3.14 for the public `concurrent.interpreters`
+# API (we still drop to `_interpreters` internally, same as
+# `tractor.spawn._subint`).
+try:
+    from concurrent import interpreters as _public_interpreters  # noqa: F401
+    import _interpreters  # type: ignore
+except ImportError:
+    print(
+        'FAIL (setup): requires Python 3.14+ '
+        '(missing `concurrent.interpreters`)',
+        file=sys.stderr,
+    )
+    sys.exit(2)
+
+
+# The actual primitives this script exercises live in
+# `tractor.spawn._subint_forkserver` — we re-import them here
+# rather than inlining so the module and the validation stay
+# in sync. (Early versions of this file had them inline for
+# the "zero tractor imports" isolation guarantee; now that
+# CPython-level feasibility is confirmed, the validated
+# primitives have moved into tractor proper.)
+from tractor.spawn._main_thread_forkserver import (
+    fork_from_worker_thread,
+    wait_child,
+)
+from tractor.spawn._subint_forkserver import (
+    run_subint_in_worker_thread,
+)
+
+
+# ----------------------------------------------------------------
+# small observability helpers (test-harness only)
+# ----------------------------------------------------------------
+
+
+def _banner(title: str) -> None:
+    line = '=' * 60
+    print(f'\n{line}\n{title}\n{line}', flush=True)
+
+
+def _report(
+    label: str,
+    *,
+    ok: bool,
+    status_str: str,
+    expect_exit_ok: bool,
+) -> None:
+    verdict: str = 'PASS' if ok else 'FAIL'
+    expected_str: str = (
+        'normal exit (rc=0)'
+        if expect_exit_ok
+        else 'abnormal death (signal or nonzero exit)'
+    )
+    print(
+        f'[{verdict}] {label}: '
+        f'expected {expected_str}; observed {status_str}',
+        flush=True,
+    )
+
+
+# ----------------------------------------------------------------
+# scenario: `control_subint_thread_fork` (known-broken)
+# ----------------------------------------------------------------
+
+
+def scenario_control_subint_thread_fork() -> int:
+    _banner(
+        '[control] fork from INSIDE a subint (expected: child aborts)'
+    )
+    interp_id = _interpreters.create('legacy')
+    print(f'  created subint {interp_id}', flush=True)
+
+    # Shared flag: child writes a sentinel file we can detect from
+    # the parent. If the child manages to write this, CPython's
+    # post-fork refusal is NOT happening → analysis is wrong.
+    sentinel = '/tmp/subint_fork_smoketest_control_child_ran'
+    try:
+        os.unlink(sentinel)
+    except FileNotFoundError:
+        pass
+
+    bootstrap = (
+        'import os\n'
+        'pid = os.fork()\n'
+        'if pid == 0:\n'
+        # child — if CPython's refusal fires this code never runs
+        f'    with open({sentinel!r}, "w") as f:\n'
+        '        f.write("ran")\n'
+        '    os._exit(0)\n'
+        'else:\n'
+        # parent side (inside the launchpad subint) — stash the
+        # forked PID on a shareable dict so we can waitpid()
+        # from the outer main interp. We can't just return it;
+        # _interpreters.exec() returns nothing useful.
+        '    import builtins\n'
+        '    builtins._forked_child_pid = pid\n'
+    )
+
+    # NOTE, we can't easily pull state back from the subint.
+    # For the CONTROL scenario we just time-bound the fork +
+    # check the sentinel. If sentinel exists → child ran →
+    # analysis wrong. If not → child aborted → analysis
+    # confirmed.
+    done = threading.Event()
+
+    def _drive() -> None:
+        try:
+            _interpreters.exec(interp_id, bootstrap)
+        except Exception as err:
+            print(
+                f'  subint bootstrap raised (expected on some '
+                f'CPython versions): {type(err).__name__}: {err}',
+                flush=True,
+            )
+        finally:
+            done.set()
+
+    t = threading.Thread(
+        target=_drive,
+        name='control-subint-fork-launchpad',
+        daemon=True,
+    )
+    t.start()
+    done.wait(timeout=5.0)
+    t.join(timeout=2.0)
+
+    # Give the (possibly-aborted) child a moment to die.
+    time.sleep(0.5)
+
+    sentinel_present = os.path.exists(sentinel)
+    verdict = (
+        # "PASS" for our analysis means sentinel NOT present.
+        'PASS' if not sentinel_present else 'FAIL (UNEXPECTED)'
+    )
+    print(
+        f'[{verdict}] control: sentinel present={sentinel_present} '
+        f'(analysis predicts False — child should abort before '
+        f'writing)',
+        flush=True,
+    )
+    if sentinel_present:
+        os.unlink(sentinel)
+
+    try:
+        _interpreters.destroy(interp_id)
+    except _interpreters.InterpreterError:
+        pass
+
+    return 0 if not sentinel_present else 1
+
+
+# ----------------------------------------------------------------
+# scenario: `main_thread_fork` (baseline sanity)
+# ----------------------------------------------------------------
+
+
+def scenario_main_thread_fork() -> int:
+    _banner(
+        '[baseline] fork from MAIN thread (expected: child exits normally)'
+    )
+
+    pid = os.fork()
+    if pid == 0:
+        os._exit(0)
+
+    return 0 if _wait_child(
+        pid,
+        label='main_thread_fork',
+        expect_exit_ok=True,
+    ) else 1
+
+
+# ----------------------------------------------------------------
+# scenario: `worker_thread_fork` (architectural assertion)
+# ----------------------------------------------------------------
+
+
+def _run_worker_thread_fork_scenario(
+    label: str,
+    *,
+    child_target=None,
+) -> int:
+    '''
+    Thin wrapper: delegate the actual fork to the
+    `tractor.spawn._subint_forkserver` primitive, then wait
+    on the child and render a pass/fail banner.
+
+    '''
+    try:
+        pid: int = fork_from_worker_thread(
+            child_target=child_target,
+            thread_name=f'worker-fork-thread[{label}]',
+        )
+    except RuntimeError as err:
+        print(f'[FAIL] {label}: {err}', flush=True)
+        return 1
+    print(f'  forked child pid={pid}', flush=True)
+    ok, status_str = wait_child(pid, expect_exit_ok=True)
+    _report(
+        label,
+        ok=ok,
+        status_str=status_str,
+        expect_exit_ok=True,
+    )
+    return 0 if ok else 1
+
+
+def scenario_worker_thread_fork() -> int:
+    _banner(
+        '[arch] fork from MAIN-INTERP WORKER thread '
+        '(expected: child exits normally — this is the one '
+        'that matters)'
+    )
+    return _run_worker_thread_fork_scenario(
+        'worker_thread_fork',
+    )
+
+
+# ----------------------------------------------------------------
+# scenario: `full_architecture`
+# ----------------------------------------------------------------
+
+
+_CHILD_TRIO_BOOTSTRAP: str = (
+    'import trio\n'
+    'async def _main():\n'
+    '    await trio.sleep(0.05)\n'
+    '    return 42\n'
+    'result = trio.run(_main)\n'
+    'assert result == 42, f"trio.run returned {result}"\n'
+    'print("  CHILD subint: trio.run OK, result=42", '
+    'flush=True)\n'
+)
+
+
+def _child_trio_in_subint() -> int:
+    '''
+    CHILD-side `child_target`: drive a trivial `trio.run()`
+    inside a fresh legacy-config subint on a worker thread,
+    using the `tractor.spawn._subint_forkserver.run_subint_in_worker_thread`
+    primitive. Returns 0 on success.
+
+    '''
+    try:
+        run_subint_in_worker_thread(
+            _CHILD_TRIO_BOOTSTRAP,
+            thread_name='child-subint-trio-thread',
+        )
+    except RuntimeError as err:
+        print(
+            f'  CHILD: run_subint_in_worker_thread timed out / thread '
+            f'never returned: {err}',
+            flush=True,
+        )
+        return 3
+    except BaseException as err:
+        print(
+            f'  CHILD: subint bootstrap raised: '
+            f'{type(err).__name__}: {err}',
+            flush=True,
+        )
+        return 4
+    return 0
+
+
+def scenario_full_architecture() -> int:
+    _banner(
+        '[arch-full] worker-thread fork + child runs trio in a '
+        'subint (end-to-end proposed arch)'
+    )
+    return _run_worker_thread_fork_scenario(
+        'full_architecture',
+        child_target=_child_trio_in_subint,
+    )
+
+
+# ----------------------------------------------------------------
+# main
+# ----------------------------------------------------------------
+
+
+SCENARIOS: dict[str, Callable[[], int]] = {
+    'control_subint_thread_fork': scenario_control_subint_thread_fork,
+    'main_thread_fork': scenario_main_thread_fork,
+    'worker_thread_fork': scenario_worker_thread_fork,
+    'full_architecture': scenario_full_architecture,
+}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(
+        description=__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    ap.add_argument(
+        '--scenario',
+        choices=sorted(SCENARIOS.keys()),
+        required=True,
+    )
+    args = ap.parse_args()
+    return SCENARIOS[args.scenario]()
+
+
+if __name__ == '__main__':
+    sys.exit(main())

ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md

@@ -0,0 +1,187 @@
+# `subint_forkserver` × `multiprocessing.SharedMemory`: fork-inherited `resource_tracker` fd
+
+Surfaced by `tests/test_shm.py` under
+`--spawn-backend=subint_forkserver`. Two distinct
+failure modes, one root cause:
+**`multiprocessing.resource_tracker` is fork-without-exec
+unsafe** (canonical CPython class — bpo-38119, bpo-45209).
+
+**Status: resolved by `tractor/ipc/_mp_bs.py` +
+`tractor/ipc/_shm.py` changes (see "Resolution" below).
+This doc kept as the
+post-mortem / decision record.**
+
+## TL;DR
+
+`mp.shared_memory.SharedMemory` registers each shm
+allocation with the per-process
+`multiprocessing.resource_tracker` singleton. The
+tracker is a daemon process started lazily; the
+parent owns a unix-pipe-fd to it. When the parent
+forks-without-execing into a `subint_forkserver`
+child, the child inherits that fd — but it refers to
+the *parent's* tracker, which the child has no
+business writing to.
+
+Two manifestations under the original (pre-fix) code:
+
+1. **`test_child_attaches_alot`** — child loops 1000×
+   `attach_shm_list()`. First `mp.SharedMemory` call
+   in the child triggers
+   `resource_tracker._ensure_running_and_write` →
+   `_teardown_dead_process` → `os.close(self._fd)` on
+   an fd the child should never have touched. Surfaces
+   as `OSError: [Errno 9] Bad file descriptor`
+   wrapped in `tractor.RemoteActorError`.
+
+2. **`test_parent_writer_child_reader[*]`** — first
+   parametrize variant "passes" (with
+   `resource_tracker: leaked shared_memory` warning)
+   because nobody ever cleans up `/shm_list`.
+   Subsequent variants then fail with
+   `FileExistsError: '/shm_list'` because the leak
+   persists across the parametrize loop and forkserver
+   children can't `shm_open(create=True)` an existing
+   key.
+
+Trio backend (`mp_spawn`-style) doesn't surface this:
+each subactor `exec`s a fresh interpreter →
+independent resource tracker per subactor → no
+inherited-fd issue, and the test's pre-existing leak
+gets masked by the per-process tracker reset.
+
+Under `subint_forkserver`, the child is `os.fork()`'d
+from a worker thread (no `exec`) → inherits parent's
+`mp.resource_tracker._resource_tracker._fd` → EBADF
+/ cross-talk on first `mp.SharedMemory` op.
+
+## Resolution
+
+We side-step the broken upstream machinery entirely
+rather than try to make it fork-safe. Two-part fix
+landed (commits to follow this doc):
+
+### 1. `tractor/ipc/_mp_bs.py::disable_mantracker()`
+   — unconditional disable
+
+The previous "3.13+ short-circuit" path used
+`partial(SharedMemory, track=False)` to opt-out of
+registration on 3.13+. The `track=False` switch is
+necessary but not sufficient under fork: the
+inherited tracker fd can still be touched indirectly
+(e.g. through `_ensure_running_and_write`'s
+self-check path).
+
+The fix takes both belts AND suspenders:
+
+- **Always** monkey-patch
+  `mp.resource_tracker._resource_tracker` to a
+  no-op `ManTracker` subclass whose
+  `register`/`unregister`/`ensure_running` are all
+  empty.
+- **Always** wrap `SharedMemory` with
+  `track=False`.
+
+Result: the inherited tracker fd in the fork child
+is still inherited (fd is a kernel object; we can't
+un-inherit it across fork) but **nothing in the
+shm code path will ever try to use it** — both the
+tracker singleton and the per-allocation registration
+are short-circuited.
+
+### 2. `tractor/ipc/_shm.py::open_shm_list()`
+   — own the cleanup
+
+Without `mp.resource_tracker`, nobody else will
+unlink leaked segments at process exit. tractor
+already controls actor lifecycle, so we register
+unlink on the actor's lifetime stack:
+
+```python
+def try_unlink():
+    try:
+        shml.shm.unlink()
+    except FileNotFoundError as fne:
+        log.exception(...)  # benign sibling-already-cleaned race
+
+actor.lifetime_stack.callback(try_unlink)
+```
+
+The `FileNotFoundError` swallow handles the case
+where a sibling actor already unlinked the same
+segment (legitimate race in shared-key setups).
+
+## Why this is the right call
+
+- **mp's tracker is widely criticized.** The
+  in-tree comment "non-SC madness" predates this
+  fix and matches CPython upstream's own discomfort
+  (e.g. the per-context tracker design rework
+  discussions in bpo-43475).
+- **tractor already owns process lifecycle.** We
+  have `actor.lifetime_stack`, `Portal.cancel_actor`,
+  and the IPC cancel cascade. Adding mp's tracker
+  on top buys nothing we can't do better ourselves.
+- **Backend-uniform.** No special-casing per spawn
+  backend. trio (`mp_spawn`-style), `subint_forkserver`,
+  and the future `subint` all behave identically
+  — register-time no-op, exit-time unlink-via-
+  lifetime-stack.
+
+## Trade-offs / known gaps
+
+- **Crash-leaked segments.** If an actor segfaults
+  or is `SIGKILL`'d before its lifetime stack runs,
+  `/dev/shm/<key>` will leak. Mitigation:
+  `scripts/tractor-reap --shm` walks `/dev/shm`,
+  filters to segments owned by the current uid that
+  no live process is mapping or holding open (via
+  `/proc/*/maps` + `/proc/*/fd/*`), and unlinks
+  them. The "nobody-has-it-open" filter is
+  kernel-canonical so it never touches in-flight
+  segments held by sibling apps (verified locally
+  against 81 piker/lttng/aja-held segments — all
+  preserved).
+  - Higher-level apps using shm should still pin a
+    UUID into the key (the `'shml_<uuid>'` pattern
+    in `test_child_attaches_alot`) so concurrent
+    sessions don't collide on the same key.
+- **Cross-actor unlink races.** Two actors holding
+  the same shm key racing on `unlink()` — handled
+  by the `FileNotFoundError` swallow.
+- **Crashes won't show up in mp's leak warning.**
+  We've turned off `resource_tracker`, so the usual
+  `resource_tracker: There appear to be N leaked
+  shared_memory objects to clean up at shutdown`
+  warning is gone too. If we ever want it back as
+  a crash-detection signal, we'd need our own
+  equivalent (walk the actor's `_shm_list_keys` set
+  at root teardown, log any unfreed).
+
+## Verification
+
+```sh
+# fixed under both backends:
+./py314/bin/python -m pytest tests/test_shm.py \
+    --spawn-backend=subint_forkserver
+# 7 passed
+
+./py314/bin/python -m pytest tests/test_shm.py \
+    --spawn-backend=trio
+# 7 passed (regression check)
+```
+
+## References
+
+- CPython upstream issues:
+  - https://bugs.python.org/issue38119 (fork
+    + resource_tracker fd inheritance)
+  - https://bugs.python.org/issue45209
+    (SharedMemory + resource_tracker)
+  - https://bugs.python.org/issue43475
+    (per-context tracker rework discussion)
+- Long-term alternative: migrate off
+  `multiprocessing.shared_memory` entirely to
+  `posix_ipc` (no tracker) or finish the
+  `hotbaud`-based ringbuf transport. Not blocked on
+  this fix — both are independently tracked.

ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md

@@ -0,0 +1,385 @@
+# `subint_forkserver` backend: orphaned-subactor SIGINT wedged in `epoll_wait`
+
+Follow-up to the Phase C `subint_forkserver` spawn-backend
+PR (see `tractor.spawn._subint_forkserver`, issue #379).
+Surfaced by the xfail'd
+`tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`.
+
+Related-but-distinct from
+`subint_cancel_delivery_hang_issue.md` (orphaned-channel
+park AFTER subint teardown) and
+`subint_sigint_starvation_issue.md` (GIL-starvation,
+SIGINT never delivered): here the SIGINT IS delivered,
+trio's handler IS installed, but trio's event loop never
+wakes — so the KBI-at-checkpoint → `_trio_main` catch path
+(which is the runtime's *intentional* OS-cancel design)
+never fires.
+
+## TL;DR
+
+When a `subint_forkserver`-spawned subactor is orphaned
+(parent `SIGKILL`'d, no IPC cancel path available) and then
+externally `SIGINT`'d, the subactor hangs in
+`trio/_core/_io_epoll.py::get_events` (epoll_wait)
+indefinitely — even though:
+
+1. `threading.current_thread() is threading.main_thread()`
+   post-fork (CPython 3.14 re-designates correctly).
+2. Trio's SIGINT handler IS installed in the subactor
+   (`signal.getsignal(SIGINT)` returns
+   `<function KIManager.install.<locals>.handler at 0x...>`).
+3. The kernel does deliver SIGINT — the signal arrives at
+   the only thread in the process (the fork-inherited
+   worker which IS now "main" per Python).
+
+Yet `epoll_wait` does not return. Trio's wakeup-fd mechanism
+— the machinery that turns SIGINT into an epoll-wake — is
+somehow not firing the wakeup. Until that's fixed, the
+intentional "KBI-as-OS-cancel" path in
+`tractor/spawn/_entry.py::_trio_main:164` is unreachable
+for forkserver-spawned subactors whose parent dies.
+
+## Symptom
+
+Test: `tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`
+(currently marked `@pytest.mark.xfail(strict=True)`).
+
+1. Harness subprocess brings up a tractor root actor +
+   one `run_in_actor(_sleep_forever)` subactor via
+   `try_set_start_method('subint_forkserver')`.
+2. Harness prints `CHILD_PID` (subactor) and
+   `PARENT_READY` (root actor) markers to stdout.
+3. Test `os.kill(parent_pid, SIGKILL)` + `proc.wait()`
+   to fully reap the root-actor harness.
+4. Child (now reparented to pid 1) is still alive.
+5. Test `os.kill(child_pid, SIGINT)` and polls
+   `os.kill(child_pid, 0)` for up to 10s.
+6. **Observed**: the child is still alive at deadline —
+   SIGINT did not unwedge the trio loop.
+
+## What the "intentional" cancel path IS
+
+`tractor/spawn/_entry.py::_trio_main:157-186` —
+
+```python
+try:
+    if infect_asyncio:
+        actor._infected_aio = True
+        run_as_asyncio_guest(trio_main)
+    else:
+        trio.run(trio_main)
+
+except KeyboardInterrupt:
+    logmeth = log.cancel
+    exit_status: str = (
+        'Actor received KBI (aka an OS-cancel)\n'
+        ...
+    )
+```
+
+The "KBI == OS-cancel" mapping IS the runtime's
+deliberate, documented design. An OS-level SIGINT should
+flow as: kernel → trio handler → KBI at trio checkpoint
+→ unwinds `async_main` → surfaces at `_trio_main`'s
+`except KeyboardInterrupt:` → `log.cancel` + clean `rc=0`.
+
+**So fixing this hang is not "add a new SIGINT behavior" —
+it's "make the existing designed behavior actually fire in
+this backend config".** That's why option (B) ("fix root
+cause") is aligned with existing design intent, not a
+scope expansion.
+
+## Evidence
+
+### Positive control: standalone fork-from-worker + `trio.run(sleep_forever)` + SIGINT WORKS
+
+```python
+import os, signal, time, trio
+from tractor.spawn._subint_forkserver import (
+    fork_from_worker_thread, wait_child,
+)
+
+def child_target() -> int:
+    async def _main():
+        try:
+            await trio.sleep_forever()
+        except KeyboardInterrupt:
+            print('CHILD: caught KBI — trio SIGINT works!')
+            return
+    trio.run(_main)
+    return 0
+
+pid = fork_from_worker_thread(child_target, thread_name='trio-sigint-test')
+time.sleep(1.0)
+os.kill(pid, signal.SIGINT)
+wait_child(pid)
+```
+
+Result: `CHILD: caught KBI — trio SIGINT works!` + clean
+exit. So the fork-child + trio signal plumbing IS healthy
+in isolation. The hang appears only with the full tractor
+subactor runtime on top.
+
+### Negative test: full tractor subactor + orphan-SIGINT
+
+Equivalent to the xfail test. Traceback dump via
+`faulthandler.register(SIGUSR1, all_threads=True)` at the
+stuck moment:
+
+```
+Current thread 0x00007... [subint-forkserv] (most recent call first):
+  File ".../trio/_core/_io_epoll.py", line 245 in get_events
+  File ".../trio/_core/_run.py", line 2415 in run
+  File "tractor/spawn/_entry.py", line 162 in _trio_main
+  File "tractor/_child.py", line 72 in _actor_child_main
+  File "tractor/spawn/_subint_forkserver.py", line 650 in _child_target
+  File "tractor/spawn/_subint_forkserver.py", line 308 in _worker
+  File ".../threading.py", line 1024 in run
+```
+
+### Thread + signal-mask inventory of the stuck subactor
+
+Single thread (`tid == pid`, comm `'subint-forkserv'`,
+which IS `threading.main_thread()` post-fork):
+
+```
+SigBlk:  0000000000000000  # nothing blocked
+SigIgn:  0000000001001000  # SIGPIPE etc (Python defaults)
+SigCgt:  0000000108000202  # bit 1 = SIGINT caught
+```
+
+Bit 1 set in `SigCgt` → SIGINT handler IS installed. So
+trio's handler IS in place at the kernel level — not a
+"handler missing" situation.
+
+### Handler identity
+
+Inside the subactor's RPC body, `signal.getsignal(SIGINT)`
+returns `<function KIManager.install.<locals>.handler at
+0x...>` — trio's own `KIManager` handler. tractor's only
+SIGINT touches are `signal.getsignal()` *reads* (to stash
+into `debug.DebugStatus._trio_handler`); nothing writes
+over trio's handler outside the debug-REPL shielding path
+(`devx/debug/_tty_lock.py::shield_sigint`) which isn't
+engaged here (no debug_mode).
+
+## Ruled out
+
+- **GIL starvation / signal-pipe-full** (class A,
+  `subint_sigint_starvation_issue.md`): subactor runs on
+  its own GIL (separate OS process), not sharing with the
+  parent → no cross-process GIL contention. And `strace`-
+  equivalent in the signal mask shows SIGINT IS caught,
+  not queued.
+- **Orphaned channel park** (`subint_cancel_delivery_hang_issue.md`):
+  different failure mode — that one has trio iterating
+  normally and getting wedged on an orphaned
+  `chan.recv()` AFTER teardown. Here trio's event loop
+  itself never wakes.
+- **Tractor explicitly catching + swallowing KBI**:
+  greppable — the one `except KeyboardInterrupt:` in the
+  runtime is the INTENTIONAL cancel-path catch at
+  `_trio_main:164`. `async_main` uses `except Exception`
+  (not BaseException), so KBI should propagate through
+  cleanly if it ever fires.
+- **Missing `signal.set_wakeup_fd` (main-thread
+  restriction)**: post-fork, the fork-worker thread IS
+  `threading.main_thread()`, so trio's main-thread check
+  passes and its wakeup-fd install should succeed.
+
+## Root cause hypothesis (unverified)
+
+The SIGINT handler fires but trio's wakeup-fd write does
+not wake `epoll_wait`. Candidate causes, ranked by
+plausibility:
+
+1. **Wakeup-fd lifecycle race around tractor IPC setup.**
+   `async_main` spins up an IPC server + `process_messages`
+   loops early. Somewhere in that path the wakeup-fd that
+   trio registered with its epoll instance may be
+   closed/replaced/clobbered, so subsequent SIGINT writes
+   land on an fd that's no longer in the epoll set.
+   Evidence needed: compare
+   `signal.set_wakeup_fd(-1)` return value inside a
+   post-tractor-bringup RPC body vs. a pre-bringup
+   equivalent. If they differ, that's it.
+2. **Shielded cancel scope around `process_messages`.**
+   The RPC message loop is likely wrapped in a trio cancel
+   scope; if that scope is `shield=True` at any outer
+   layer, KBI scheduled at a checkpoint could be absorbed
+   by the shield and never bubble out to `_trio_main`.
+3. **Pre-fork wakeup-fd inheritance.** trio in the PARENT
+   process registered a wakeup-fd with its own epoll. The
+   child inherits the fd number but not the parent's
+   epoll instance — if tractor/trio re-uses the parent's
+   stale fd number anywhere, writes would go to a no-op
+   fd. (This is the least likely — `trio.run()` on the
+   child calls `KIManager.install` which should install a
+   fresh wakeup-fd from scratch.)
+
+## Cross-backend scope question
+
+**Untested**: does the same orphan-SIGINT hang reproduce
+against the `trio_proc` backend (stock subprocess + exec)?
+If yes → pre-existing tractor bug, independent of
+`subint_forkserver`. If no → something specific to the
+fork-from-worker path (e.g. inherited fds, mid-epoll-setup
+interference).
+
+**Quick repro for trio_proc**:
+
+```python
+# save as /tmp/trio_proc_orphan_sigint_repro.py
+import os, sys, signal, time, glob
+import subprocess as sp
+
+SCRIPT = '''
+import os, sys, trio, tractor
+async def _sleep_forever():
+    print(f"CHILD_PID={os.getpid()}", flush=True)
+    await trio.sleep_forever()
+
+async def _main():
+    async with (
+        tractor.open_root_actor(registry_addrs=[("127.0.0.1", 12350)]),
+        tractor.open_nursery() as an,
+    ):
+        await an.run_in_actor(_sleep_forever, name="sf-child")
+        print(f"PARENT_READY={os.getpid()}", flush=True)
+        await trio.sleep_forever()
+
+trio.run(_main)
+'''
+
+proc = sp.Popen(
+    [sys.executable, '-c', SCRIPT],
+    stdout=sp.PIPE, stderr=sp.STDOUT,
+)
+# parse CHILD_PID + PARENT_READY off proc.stdout ...
+# SIGKILL parent, SIGINT child, poll.
+```
+
+If that hangs too, open a broader issue; if not, this is
+`subint_forkserver`-specific (likely fd-inheritance-related).
+
+## Why this is ours to fix (not CPython's)
+
+- Signal IS delivered (`SigCgt` bitmask confirms).
+- Handler IS installed (trio's `KIManager`).
+- Thread identity is correct post-fork.
+- `_trio_main` already has the intentional KBI→clean-exit
+  path waiting to fire.
+
+Every CPython-level precondition is met. Something in
+tractor's runtime or trio's integration with it is
+breaking the SIGINT→wakeup→event-loop-wake pipeline.
+
+## Possible fix directions
+
+1. **Audit the wakeup-fd across tractor's IPC bringup.**
+   Add a trio startup hook that captures
+   `signal.set_wakeup_fd(-1)` at `_trio_main` entry,
+   after `async_main` enters, and periodically — assert
+   it's unchanged. If it moves, track down the writer.
+2. **Explicit `signal.set_wakeup_fd` reset after IPC
+   setup.** Brute force: re-install a fresh wakeup-fd
+   mid-bringup. Band-aid, but fast to try.
+3. **Ensure no `shield=True` cancel scope envelopes the
+   RPC-message-loop / IPC-server task.** If one does,
+   KBI-at-checkpoint never escapes.
+4. **Once fixed, the `child_sigint='trio'` mode on
+   `subint_forkserver_proc`** becomes effectively a no-op
+   or a doc-only mode — trio's natural handler already
+   does the right thing. Might end up removing the flag
+   entirely if there's no behavioral difference between
+   modes.
+
+## Current workaround
+
+None; `child_sigint` defaults to `'ipc'` (IPC cancel is
+the only reliable cancel path today), and the xfail test
+documents the gap. Operators hitting orphan-SIGINT get a
+hung process that needs `SIGKILL`.
+
+## Reproducer
+
+Inline, standalone (no pytest):
+
+```python
+# save as /tmp/orphan_sigint_repro.py  (py3.14+)
+import os, sys, signal, time, glob, trio
+import tractor
+from tractor.spawn._subint_forkserver import (
+    fork_from_worker_thread,
+)
+
+async def _sleep_forever():
+    print(f'SUBACTOR[{os.getpid()}]', flush=True)
+    await trio.sleep_forever()
+
+async def _main():
+    async with (
+        tractor.open_root_actor(
+            registry_addrs=[('127.0.0.1', 12349)],
+        ),
+        tractor.open_nursery() as an,
+    ):
+        await an.run_in_actor(_sleep_forever, name='sf-child')
+        await trio.sleep_forever()
+
+def child_target() -> int:
+    from tractor.spawn._spawn import try_set_start_method
+    try_set_start_method('subint_forkserver')
+    trio.run(_main)
+    return 0
+
+pid = fork_from_worker_thread(child_target, thread_name='repro')
+time.sleep(3.0)
+
+# find the subactor pid via /proc
+children = []
+for path in glob.glob(f'/proc/{pid}/task/*/children'):
+    with open(path) as f:
+        children.extend(int(x) for x in f.read().split() if x)
+subactor_pid = children[0]
+
+# SIGKILL root → orphan the subactor
+os.kill(pid, signal.SIGKILL)
+os.waitpid(pid, 0)
+time.sleep(0.3)
+
+# SIGINT the orphan — should cause clean trio exit
+os.kill(subactor_pid, signal.SIGINT)
+
+# poll for exit
+for _ in range(100):
+    try:
+        os.kill(subactor_pid, 0)
+        time.sleep(0.1)
+    except ProcessLookupError:
+        print('HARNESS: subactor exited cleanly ✔')
+        sys.exit(0)
+os.kill(subactor_pid, signal.SIGKILL)
+print('HARNESS: subactor hung — reproduced')
+sys.exit(1)
+```
+
+Expected (current): `HARNESS: subactor hung — reproduced`.
+
+After fix: `HARNESS: subactor exited cleanly ✔`.
+
+## References
+
+- `tractor/spawn/_entry.py::_trio_main:157-186` — the
+  intentional KBI→clean-exit path this bug makes
+  unreachable.
+- `tractor/spawn/_subint_forkserver` — the backend whose
+  orphan cancel-robustness this blocks.
+- `tests/spawn/test_subint_forkserver.py::test_orphaned_subactor_sigint_cleanup_DRAFT`
+  — the xfail'd reproducer in the test suite.
+- `ai/conc-anal/subint_cancel_delivery_hang_issue.md` —
+  sibling "orphaned channel park" hang (different class).
+- `ai/conc-anal/subint_sigint_starvation_issue.md` —
+  sibling "GIL starvation SIGINT drop" hang (different
+  class).
+- tractor issue #379 — subint backend tracking.

ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md

@@ -0,0 +1,851 @@
+# `subint_forkserver` backend: `test_cancellation.py` multi-level cancel cascade hang
+
+> **Tracked at:** [#449](https://github.com/goodboy/tractor/issues/449)
+
+Follow-up tracker: surfaced while wiring the new
+`subint_forkserver` spawn backend into the full tractor
+test matrix (step 2 of the post-backend-lands plan).
+See also
+`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
+— sibling tracker for a different forkserver-teardown
+class which probably shares the same fundamental root
+cause (fork-FD-inheritance across nested spawns).
+
+## TL;DR
+
+`tests/test_cancellation.py::test_nested_multierrors[subint_forkserver]`
+hangs indefinitely under our new backend. The hang is
+**inside the graceful IPC cancel cascade** — every actor
+in the multi-level tree parks in `epoll_wait` waiting
+for IPC messages that never arrive. Not a hard-kill /
+tree-reap issue (we don't reach the hard-kill fallback
+path at all).
+
+Working hypothesis (unverified): **`os.fork()` from a
+subactor inherits the root parent's IPC listener socket
+FDs**. When a first-level subactor forkserver-spawns a
+grandchild, that grandchild inherits both its direct
+spawner's FDs AND the root's FDs — IPC message routing
+becomes ambiguous (or silently sends to the wrong
+channel), so the cancel cascade can't reach its target.
+
+## Corrected diagnosis vs. earlier draft
+
+An earlier version of this doc claimed the root cause
+was **"forkserver teardown doesn't tree-kill
+descendants"** (SIGKILL only reaches the direct child,
+grandchildren survive and hold TCP `:1616`). That
+diagnosis was **wrong**, caused by conflating two
+observations:
+
+1. *5-zombie leak holding :1616* — happened in my own
+   workflow when I aborted a bg pytest task with
+   `pkill` (SIGTERM/SIGKILL, not SIGINT). The abrupt
+   kill skipped the graceful `ActorNursery.__aexit__`
+   cancel cascade entirely, orphaning descendants to
+   init. **This was my cleanup bug, not a forkserver
+   teardown bug.** Codified the fix (SIGINT-first +
+   bounded wait before SIGKILL) in
+   `feedback_sc_graceful_cancel_first.md` +
+   `.claude/skills/run-tests/SKILL.md`.
+2. *`test_nested_multierrors` hangs indefinitely* —
+   the real, separate, forkserver-specific bug
+   captured by this doc.
+
+The two symptoms are unrelated. The tree-kill / setpgrp
+fix direction proposed earlier would not help (1) (SC-
+graceful-cleanup is the right answer there) and would
+not help (2) (the hang is in the cancel cascade, not
+in the hard-kill fallback).
+
+## Symptom
+
+Reproducer (py3.14, clean env):
+
+```sh
+# preflight: ensure clean env
+ss -tlnp 2>/dev/null | grep ':1616' && echo 'FOUL — cleanup first!' || echo 'clean'
+
+./py314/bin/python -m pytest --spawn-backend=subint_forkserver \
+  'tests/test_cancellation.py::test_nested_multierrors[subint_forkserver]' \
+  --timeout=30 --timeout-method=thread --tb=short -v
+```
+
+Expected: `pytest-timeout` fires at 30s with a thread-
+dump banner, but the process itself **remains alive
+after timeout** and doesn't unwedge on subsequent
+SIGINT. Requires SIGKILL to reap.
+
+## Evidence (tree structure at hang point)
+
+All 5 processes are kernel-level `S` (sleeping) in
+`do_epoll_wait` (trio's event loop waiting on I/O):
+
+```
+PID     PPID    THREADS  NAME             ROLE
+333986  1       2        subint-forkserv  pytest main (the test body)
+333993  333986  3        subint-forkserv  "child 1" spawner subactor
+  334003 333993 1        subint-forkserv  grandchild errorer under child-1
+  334014 333993 1        subint-forkserv  grandchild errorer under child-1
+333999  333986  1        subint-forkserv  "child 2" spawner subactor (NO grandchildren!)
+```
+
+### Asymmetric tree depth
+
+The test's `spawn_and_error(breadth=2, depth=3)` should
+have BOTH direct children spawning 2 grandchildren
+each, going 3 levels deep. Reality:
+
+- Child 1 (333993, 3 threads) DID spawn its two
+  grandchildren as expected — fully booted trio
+  runtime.
+- Child 2 (333999, 1 thread) did NOT spawn any
+  grandchildren — clearly never completed its
+  nursery's first `run_in_actor`. Its 1-thread state
+  suggests the runtime never fully booted (no trio
+  worker threads for `waitpid`/IPC).
+
+This asymmetry is the key clue: the two direct
+children started identically but diverged. Probably a
+race around fork-inherited state (listener FDs,
+subactor-nursery channel state) that happens to land
+differently depending on spawn ordering.
+
+### Parent-side state
+
+Thread-dump of pytest main (333986) at the hang:
+
+- Main trio thread — parked in
+  `trio._core._io_epoll.get_events` (epoll_wait on
+  its event loop). Waiting for IPC from children.
+- Two trio-cache worker threads — each parked in
+  `outcome.capture(sync_fn)` calling
+  `os.waitpid(child_pid, 0)`. These are our
+  `_ForkedProc.wait()` off-loads. They're waiting for
+  the direct children to exit — but children are
+  stuck in their own epoll_wait waiting for IPC from
+  the parent.
+
+**It's a deadlock, not a leak:** the parent is
+correctly running `soft_kill(proc, _ForkedProc.wait,
+portal)` (graceful IPC cancel via
+`Portal.cancel_actor()`), but the children never
+acknowledge the cancel message (or the message never
+reaches them through the tangled post-fork IPC).
+
+## What's NOT the cause (ruled out)
+
+- **`_ForkedProc.kill()` only SIGKILLs direct pid /
+  missing tree-kill**: doesn't apply — we never reach
+  the hard-kill path. The deadlock is in the graceful
+  cancel cascade.
+- **Port `:1616` contention**: ruled out after the
+  `reg_addr` fixture-wiring fix; each test session
+  gets a unique port now.
+- **GIL starvation / SIGINT pipe filling** (class-A,
+  `subint_sigint_starvation_issue.md`): doesn't apply
+  — each subactor is its own OS process with its own
+  GIL (not legacy-config subint).
+- **Child-side `_trio_main` absorbing KBI**: grep
+  confirmed; `_trio_main` only catches KBI at the
+  `trio.run()` callsite, which is reached only if the
+  trio loop exits normally. The children here never
+  exit trio.run() — they're wedged inside.
+
+## Hypothesis: FD inheritance across nested forks
+
+`subint_forkserver_proc` calls
+`fork_from_worker_thread()` which ultimately does
+`os.fork()` from a dedicated worker thread. Standard
+Linux/POSIX fork semantics: **the child inherits ALL
+open FDs from the parent**, including listener
+sockets, epoll fds, trio wakeup pipes, and the
+parent's IPC channel sockets.
+
+At root-actor fork-spawn time, the root's IPC server
+listener FDs are open in the parent. Those get
+inherited by child 1. Child 1 then forkserver-spawns
+its OWN subactor (grandchild). The grandchild
+inherits FDs from child 1 — but child 1's address
+space still contains **the root's IPC listener FDs
+too** (inherited at first fork). So the grandchild
+has THREE sets of FDs:
+
+1. Its own (created after becoming a subactor).
+2. Its direct parent child-1's.
+3. The ROOT's (grandparent's) — inherited transitively.
+
+IPC message routing may be ambiguous in this tangled
+state. Or a listener socket that the root thinks it
+owns is actually open in multiple processes, and
+messages sent to it go to an arbitrary one. That
+would exactly match the observed "graceful cancel
+never propagates".
+
+This hypothesis predicts the bug **scales with fork
+depth**: single-level forkserver spawn
+(`test_subint_forkserver_spawn_basic`) works
+perfectly, but any test that spawns a second level
+deadlocks. Matches observations so far.
+
+## Fix directions (to validate)
+
+### 1. `close_fds=True` equivalent in `fork_from_worker_thread()`
+
+`subprocess.Popen` / `trio.lowlevel.open_process` have
+`close_fds=True` by default on POSIX — they
+enumerate open FDs in the child post-fork and close
+everything except stdio + any explicitly-passed FDs.
+Our raw `os.fork()` doesn't. Adding the equivalent to
+our `_worker` prelude would isolate each fork
+generation's FD set.
+
+Implementation sketch in
+`tractor.spawn._subint_forkserver.fork_from_worker_thread._worker`:
+
+```python
+def _worker() -> None:
+    pid: int = os.fork()
+    if pid == 0:
+        # CHILD: close inherited FDs except stdio + the
+        # pid-pipe we just opened.
+        keep: set[int] = {0, 1, 2, rfd, wfd}
+        import resource
+        soft, _ = resource.getrlimit(resource.RLIMIT_NOFILE)
+        os.closerange(3, soft)  # blunt; or enumerate /proc/self/fd
+        # ... then child_target() as before
+```
+
+Problem: overly aggressive — closes FDs the
+grandchild might legitimately need (e.g. its parent's
+IPC channel for the spawn-spec handshake, if we rely
+on that). Needs thought about which FDs are
+"inheritable and safe" vs. "inherited by accident".
+
+### 2. Cloexec on tractor's own FDs
+
+Set `FD_CLOEXEC` on tractor-created sockets (listener
+sockets, IPC channel sockets, pipes). This flag
+causes automatic close on `execve`, but since we
+`fork()` without `exec()`, this alone doesn't help.
+BUT — combined with a child-side explicit close-
+non-cloexec loop, it gives us a way to mark "my
+private FDs" vs. "safe to inherit". Most robust, but
+requires tractor-wide audit.
+
+### 3. Explicit FD cleanup in `_ForkedProc`/`_child_target`
+
+Have `subint_forkserver_proc`'s `_child_target`
+closure explicitly close the parent-side IPC listener
+FDs before calling `_actor_child_main`. Requires
+being able to enumerate "the parent's listener FDs
+that the child shouldn't keep" — plausible via
+`Actor.ipc_server`'s socket objects.
+
+### 4. Use `os.posix_spawn` with explicit `file_actions`
+
+Instead of raw `os.fork()`, use `os.posix_spawn()`
+which supports explicit file-action specifications
+(close this FD, dup2 that FD). Cleaner semantics, but
+probably incompatible with our "no exec" requirement
+(subint_forkserver is a fork-without-exec design).
+
+**Likely correct answer: (3) — targeted FD cleanup
+via `actor.ipc_server` handle.** (1) is too blunt,
+(2) is too wide-ranging, (4) changes the spawn
+mechanism.
+
+## Reproducer (standalone, no pytest)
+
+```python
+# save as /tmp/forkserver_nested_hang_repro.py  (py3.14+)
+import trio, tractor
+
+async def assert_err():
+    assert 0
+
+async def spawn_and_error(breadth: int = 2, depth: int = 1):
+    async with tractor.open_nursery() as n:
+        for i in range(breadth):
+            if depth > 0:
+                await n.run_in_actor(
+                    spawn_and_error,
+                    breadth=breadth,
+                    depth=depth - 1,
+                    name=f'spawner_{i}_{depth}',
+                )
+            else:
+                await n.run_in_actor(
+                    assert_err,
+                    name=f'errorer_{i}',
+                )
+
+async def _main():
+    async with tractor.open_nursery() as n:
+        for i in range(2):
+            await n.run_in_actor(
+                spawn_and_error,
+                name=f'top_{i}',
+                breadth=2,
+                depth=1,
+            )
+
+if __name__ == '__main__':
+    from tractor.spawn._spawn import try_set_start_method
+    try_set_start_method('subint_forkserver')
+    with trio.fail_after(20):
+        trio.run(_main)
+```
+
+Expected (current): hangs on `trio.fail_after(20)`
+— children never ack the error-propagation cancel
+cascade. Pattern: top 2 direct children, 4
+grandchildren, 1 errorer deadlocks while trying to
+unwind through its parent chain.
+
+After fix: `trio.TooSlowError`-free completion; the
+root's `open_nursery` receives the
+`BaseExceptionGroup` containing the `AssertionError`
+from the errorer and unwinds cleanly.
+
+## Update — 2026-04-23: partial fix landed, deeper layer surfaced
+
+Three improvements landed as separate commits in the
+`subint_forkserver_backend` branch (see `git log`):
+
+1. **`_close_inherited_fds()` in fork-child prelude**
+   (`tractor/spawn/_subint_forkserver.py`). POSIX
+   close-fds-equivalent enumeration via
+   `/proc/self/fd` (or `RLIMIT_NOFILE` fallback), keep
+   only stdio. This is fix-direction (1) from the list
+   above — went with the blunt form rather than the
+   targeted enum-via-`actor.ipc_server` form, turns
+   out the aggressive close is safe because every
+   inheritable resource the fresh child needs
+   (IPC-channel socket, etc.) is opened AFTER the
+   fork anyway.
+2. **`_ForkedProc.wait()` via `os.pidfd_open()` +
+   `trio.lowlevel.wait_readable()`** — matches the
+   `trio.Process.wait` / `mp.Process.sentinel` pattern
+   used by `trio_proc` and `proc_waiter`. Gives us
+   fully trio-cancellable child-wait (prior impl
+   blocked a cache thread on a sync `os.waitpid` that
+   was NOT trio-cancellable due to
+   `abandon_on_cancel=False`).
+3. **`_parent_chan_cs` wiring** in
+   `tractor/runtime/_runtime.py`: capture the shielded
+   `loop_cs` for the parent-channel `process_messages`
+   task in `async_main`; explicitly cancel it in
+   `Actor.cancel()` teardown. This breaks the shield
+   during teardown so the parent-chan loop exits when
+   cancel is issued, instead of parking on a parent-
+   socket EOF that might never arrive under fork
+   semantics.
+
+**Concrete wins from (1):** the sibling
+`subint_forkserver_orphan_sigint_hang_issue.md` class
+is **now fixed** — `test_orphaned_subactor_sigint_cleanup_DRAFT`
+went from strict-xfail to pass. The xfail mark was
+removed; the test remains as a regression guard.
+
+**test_nested_multierrors STILL hangs** though.
+
+### Updated diagnosis (narrowed)
+
+DIAGDEBUG instrumentation of `process_messages` ENTER/
+EXIT pairs + `_parent_chan_cs.cancel()` call sites
+showed (captured during a 20s-timeout repro):
+
+- 80 `process_messages` ENTERs, 75 EXITs → 5 stuck.
+- **All 40 `shield=True` ENTERs matched EXIT** — every
+  shielded parent-chan loop exits cleanly. The
+  `_parent_chan_cs` wiring works as intended.
+- **The 5 stuck loops are all `shield=False`** — peer-
+  channel handlers (inbound connections handled by
+  `handle_stream_from_peer` in stream_handler_tn).
+- After our `_parent_chan_cs.cancel()` fires, NEW
+  shielded process_messages loops start (on the
+  session reg_addr port — probably discovery-layer
+  reconnection attempts). These don't block teardown
+  (they all exit) but indicate the cancel cascade has
+  more moving parts than expected.
+
+### Remaining unknown
+
+Why don't the 5 peer-channel loops exit when
+`service_tn.cancel_scope.cancel()` fires? They're in
+`stream_handler_tn` which IS `service_tn` in the
+current configuration (`open_ipc_server(parent_tn=
+service_tn, stream_handler_tn=service_tn)`). A
+standard nursery-scope-cancel should propagate through
+them — no shield, no special handler. Something
+specific to the fork-spawned configuration keeps them
+alive.
+
+Candidate follow-up experiments:
+
+- Dump the trio task tree at the hang point (via
+  `stackscope` or direct trio introspection) to see
+  what each stuck loop is awaiting. `chan.__anext__`
+  on a socket recv? An inner lock? A shielded sub-task?
+- Compare peer-channel handler lifecycle under
+  `trio_proc` vs `subint_forkserver` with equivalent
+  logging to spot the divergence.
+- Investigate whether the peer handler is caught in
+  the `except trio.Cancelled:` path at
+  `tractor/ipc/_server.py:448` that re-raises — but
+  re-raise means it should still exit. Unless
+  something higher up swallows it.
+
+### Attempted fix (DID NOT work) — hypothesis (3)
+
+Tried: in `_serve_ipc_eps` finally, after closing
+listeners, also iterate `server._peers` and
+sync-close each peer channel's underlying stream
+socket fd:
+
+```python
+for _uid, _chans in list(server._peers.items()):
+    for _chan in _chans:
+        try:
+            _stream = _chan._transport.stream if _chan._transport else None
+            if _stream is not None:
+                _stream.socket.close()  # sync fd close
+        except (AttributeError, OSError):
+            pass
+```
+
+Theory: closing the socket fd from outside the stuck
+recv task would make the recv see EBADF /
+ClosedResourceError and unblock.
+
+Result: `test_nested_multierrors[subint_forkserver]`
+still hangs identically. Either:
+- The sync `socket.close()` doesn't propagate into
+  trio's in-flight `recv_some()` the way I expected
+  (trio may hold an internal reference that keeps the
+  fd open even after an external close), or
+- The stuck recv isn't even the root blocker and the
+  peer handlers never reach the finally for some
+  reason I haven't understood yet.
+
+Either way, the sync-close hypothesis is **ruled
+out**. Reverted the experiment, restored the skip-
+mark on the test.
+
+### Aside: `-s` flag does NOT change `test_nested_multierrors` behavior
+
+Tested explicitly: both with and without `-s`, the
+test hangs identically. So the capture-pipe-fill
+hypothesis is **ruled out** for this test.
+
+The earlier `test_context_stream_semantics.py` `-s`
+observation was most likely caused by a competing
+pytest run in my session (confirmed via process list
+— my leftover pytest was alive at that time and
+could have been holding state on the default
+registry port).
+
+## Update — 2026-04-23 (late): cancel delivery ruled in, nursery-wait ruled BLOCKER
+
+**New diagnostic run** instrumented
+`handle_stream_from_peer` at ENTER / `except
+trio.Cancelled:` / finally, plus `Actor.cancel()`
+just before `self._parent_chan_cs.cancel()`. Result:
+
+- **40 `handle_stream_from_peer` ENTERs**.
+- **0 `except trio.Cancelled:` hits** — cancel
+  never fires on any peer-handler.
+- **35 finally hits** — those handlers exit via
+  peer-initiated EOF (normal return), NOT cancel.
+- **5 handlers never reach finally** — stuck forever.
+- **`Actor.cancel()` fired in 12 PIDs** — but the
+  PIDs with peer handlers that DIDN'T fire
+  Actor.cancel are exactly **root + 2 direct
+  spawners**. These 3 actors have peer handlers
+  (for their own subactors) that stay stuck because
+  **`Actor.cancel()` at these levels never runs**.
+
+### The actual deadlock shape
+
+`Actor.cancel()` lives in
+`open_root_actor.__aexit__` / `async_main` teardown.
+That only runs when the enclosing `async with
+tractor.open_nursery()` exits. The nursery's
+`__aexit__` calls the backend `*_proc` spawn target's
+teardown, which does `soft_kill() →
+_ForkedProc.wait()` on its child PID. That wait is
+trio-cancellable via pidfd now (good) — but nothing
+CANCELS it because the outer scope only cancels when
+`Actor.cancel()` runs, which only runs when the
+nursery completes, which waits on the child.
+
+It's a **multi-level mutual wait**:
+
+```
+root              blocks on spawner.wait()
+  spawner         blocks on grandchild.wait()
+    grandchild    blocks on errorer.wait()
+      errorer     Actor.cancel() ran, but process
+                  may not have fully exited yet
+                  (something in root_tn holding on?)
+```
+
+Each level waits for the level below. The bottom
+level (errorer) reaches Actor.cancel(), but its
+process may not fully exit — meaning its pidfd
+doesn't go readable, meaning the grandchild's
+waitpid doesn't return, meaning the grandchild's
+nursery doesn't unwind, etc. all the way up.
+
+### Refined question
+
+**Why does an errorer process not exit after its
+`Actor.cancel()` completes?**
+
+Possibilities:
+1. `_parent_chan_cs.cancel()` fires (shielded
+   parent-chan loop unshielded), but the task is
+   stuck INSIDE the shielded loop's recv in a way
+   that cancel still can't break.
+2. After `Actor.cancel()` returns, `async_main`
+   still has other tasks in `root_tn` waiting for
+   something that never arrives (e.g. outbound
+   IPC reply delivery).
+3. The `os._exit(rc)` in `_worker` (at
+   `_subint_forkserver.py`) doesn't run because
+   `_child_target` never returns.
+
+Next-session candidate probes (in priority order):
+
+1. **Instrument `_worker`'s fork-child branch** to
+   confirm whether `child_target()` returns (and
+   thus `os._exit(rc)` is reached) for errorer
+   PIDs. If yes → process should die; if no →
+   trace back into `_actor_child_main` /
+   `_trio_main` / `async_main` to find the stuck
+   spot.
+2. **Instrument `async_main`'s final unwind** to
+   see which await in the teardown doesn't
+   complete.
+3. **Compare under `trio_proc` backend** at the
+   same `_worker`-equivalent level to see where
+   the flows diverge.
+
+### Rule-out: NOT a stuck peer-chan recv
+
+Earlier hypothesis was that the 5 stuck peer-chan
+loops were blocked on a socket recv that cancel
+couldn't interrupt. This pass revealed the real
+cause: cancel **never reaches those tasks** because
+their owning actor's `Actor.cancel()` never runs.
+The recvs are fine — they're just parked because
+nothing is telling them to stop.
+
+## Update — 2026-04-23 (very late): leaves exit, middle actors stuck in `trio.run`
+
+Yet another instrumentation pass — this time
+printing at:
+
+- `_worker` child branch: `pre child_target()` /
+  `child_target RETURNED rc=N` / `about to
+  os._exit(rc)` 
+- `_trio_main`: `about to trio.run` /
+  `trio.run RETURNED NORMALLY` / `FINALLY`
+
+**Fresh-run results** (`test_nested_multierrors[
+subint_forkserver]`, depth=1/breadth=2, 1 root + 14
+forked = 15 actors total):
+
+- **9 processes completed the full flow** —
+  `trio.run RETURNED NORMALLY` → `child_target
+  RETURNED rc=0` → `about to os._exit(0)`. These
+  are the LEAVES of the tree (errorer actors) plus
+  their direct parents (depth-0 spawners). They
+  actually exit their processes.
+- **5 processes are stuck INSIDE `trio.run(trio_main)`**
+  — they hit "about to trio.run" but NEVER see
+  "trio.run RETURNED NORMALLY". These are root +
+  top-level spawners + one intermediate.
+
+**What this means:** `async_main` itself is the
+deadlock holder, not the peer-channel loops.
+Specifically, the outer `async with root_tn:` in
+`async_main` never exits for the 5 stuck actors.
+Their `trio.run` never returns → `_trio_main`
+catch/finally never runs → `_worker` never reaches
+`os._exit(rc)` → the PROCESS never dies → its
+parent's `_ForkedProc.wait()` blocks → parent's
+nursery hangs → parent's `async_main` hangs → ...
+
+### The new precise question
+
+**What task in the 5 stuck actors' `async_main`
+never completes?** Candidates:
+
+1. The shielded parent-chan `process_messages`
+   task in `root_tn` — but we explicitly cancel it
+   via `_parent_chan_cs.cancel()` in `Actor.cancel()`.
+   However, `Actor.cancel()` only runs during
+   `open_root_actor.__aexit__`, which itself runs
+   only after `async_main`'s outer unwind — which
+   doesn't happen. So the shield isn't broken.
+
+2. `await actor_nursery._join_procs.wait()` or
+   similar in the inline backend `*_proc` flow.
+
+3. `_ForkedProc.wait()` on a grandchild that
+   actually DID exit — but the pidfd_open watch
+   didn't fire for some reason (race between
+   pidfd_open and the child exiting?).
+
+The most specific next probe: **add DIAG around
+`_ForkedProc.wait()` enter/exit** to see whether
+the pidfd-based wait returns for every grandchild
+exit. If a stuck parent's `_ForkedProc.wait()`
+NEVER returns despite its child exiting, the
+pidfd mechanism has a race bug under nested
+forkserver.
+
+Alternative probe: instrument `async_main`'s outer
+nursery exits to find which nursery's `__aexit__`
+is stuck, drilling down from `trio.run` to the
+specific `async with` that never completes.
+
+### Cascade summary (updated tree view)
+
+```
+ROOT (pytest)                       STUCK in trio.run
+├── top_0 (spawner, d=1)            STUCK in trio.run
+│   ├── spawner_0_d1_0 (d=0)        exited (os._exit 0)
+│   │   ├── errorer_0_0             exited (os._exit 0)
+│   │   └── errorer_0_1             exited (os._exit 0)
+│   └── spawner_0_d1_1 (d=0)        exited (os._exit 0)
+│       ├── errorer_0_2             exited (os._exit 0)
+│       └── errorer_0_3             exited (os._exit 0)
+└── top_1 (spawner, d=1)            STUCK in trio.run
+    ├── spawner_1_d1_0 (d=0)        STUCK in trio.run (sibling race?)
+    │   ├── errorer_1_0             exited
+    │   └── errorer_1_1             exited
+    └── spawner_1_d1_1 (d=0)        STUCK in trio.run
+        ├── errorer_1_2             exited
+        └── errorer_1_3             exited
+```
+
+Grandchildren (d=0 spawners) exit OR stick —
+asymmetric. Not purely depth-determined. Some race
+condition in nursery teardown when multiple
+siblings error simultaneously.
+
+## Update — 2026-04-23 (late, probe iteration 3): hang pinpointed to `wait_for_no_more_peers()`
+
+Further DIAGDEBUG at every milestone in `async_main`
+(runtime UP / EXITED service_tn / EXITED root_tn /
+FINALLY ENTER / RETURNING) plus `_ForkedProc.wait`
+ENTER/RETURNED per-pidfd. Result:
+
+**Every stuck actor reaches `async_main: FINALLY
+ENTER` but NOT `async_main: RETURNING`.**
+
+That isolates the hang to a specific await in
+`async_main`'s finally block at
+`tractor/runtime/_runtime.py:1837+`. The suspect:
+
+```python
+# Ensure all peers (actors connected to us as clients) are finished
+if ipc_server := actor.ipc_server and ipc_server.has_peers(check_chans=True):
+    ...
+    await ipc_server.wait_for_no_more_peers()  # ← UNBOUNDED, blocks forever
+```
+
+`_no_more_peers` is an `Event` set only when
+`server._peers` empties (see
+`ipc/_server.py:526-530`). If ANY peer-handler is
+stuck (the 5 unclosed loops from the earlier pass),
+it keeps its channel in `server._peers`, so the
+event never fires, so the wait hangs.
+
+### Applied fix (partial, landed as defensive-in-depth)
+
+`tractor/runtime/_runtime.py:1981` —
+`wait_for_no_more_peers()` call now wrapped in
+`trio.move_on_after(3.0)` + a warning log when the
+timeout fires. Commented with the full rationale.
+
+**Verified:** with this fix, ALL 15 actors reach
+`async_main: RETURNING` cleanly (up from 10/15
+reaching end before).
+
+**Unfortunately:** the test still hangs past 45s
+total — meaning there's YET ANOTHER unbounded wait
+downstream of `async_main`. The bounded
+`wait_for_no_more_peers` unblocks one level, but
+the cascade has another level above it.
+
+### Candidates for the remaining hang
+
+1. `open_root_actor`'s own finally / post-
+   `async_main` flow in `_root.py` — specifically
+   `await actor.cancel(None)` which has its own
+   internal waits.
+2. The `trio.run()` itself doesn't return even
+   after the root task completes because trio's
+   nursery still has background tasks running.
+3. Maybe `_serve_ipc_eps`'s finally has an await
+   that blocks when peers aren't clearing.
+
+### Current stance
+
+- Defensive `wait_for_no_more_peers` bound landed
+  (good hygiene regardless). Revealing a real
+  deadlock-avoidance gap in tractor's cleanup.
+- Test still hangs → skip-mark restored on
+  `test_nested_multierrors[subint_forkserver]`.
+- The full chain of unbounded waits needs another
+  session of drilling, probably at
+  `open_root_actor` / `actor.cancel` level.
+
+### Summary of this investigation's wins
+
+1. **FD hygiene fix** (`_close_inherited_fds`) —
+   correct, closed orphan-SIGINT sibling issue.
+2. **pidfd-based `_ForkedProc.wait`** — cancellable,
+   matches trio_proc pattern.
+3. **`_parent_chan_cs` wiring** —
+   `Actor.cancel()` now breaks the shielded parent-
+   chan `process_messages` loop.
+4. **`wait_for_no_more_peers` bounded** —
+   prevents the actor-level finally hang.
+5. **Ruled-out hypotheses:** tree-kill missing
+   (wrong), stuck socket recv (wrong).
+6. **Pinpointed remaining unknown:** at least one
+   more unbounded wait in the teardown cascade
+   above `async_main`. Concrete candidates
+   enumerated above.
+
+## Update — 2026-04-23 (VERY late): pytest capture pipe IS the final gate
+
+After landing fixes 1-4 and instrumenting every
+layer down to `tractor_test`'s `trio.run(_main)`:
+
+**Empirical result: with `pytest -s` the test PASSES
+in 6.20s.** Without `-s` (default `--capture=fd`) it
+hangs forever.
+
+DIAG timeline for the root pytest PID (with `-s`
+implied from later verification):
+
+```
+tractor_test: about to trio.run(_main)
+open_root_actor: async_main task started, yielding to test body
+_main: about to await wrapped test fn
+_main: wrapped RETURNED cleanly        ← test body completed!
+open_root_actor: about to actor.cancel(None)
+Actor.cancel ENTER req_chan=False
+Actor.cancel RETURN
+open_root_actor: actor.cancel RETURNED
+open_root_actor: outer FINALLY
+open_root_actor: finally END (returning from ctxmgr)
+tractor_test: trio.run FINALLY (returned or raised)  ← trio.run fully returned!
+```
+
+`trio.run()` fully returns. The test body itself
+completes successfully (pytest.raises absorbed the
+expected `BaseExceptionGroup`). What blocks is
+**pytest's own stdout/stderr capture** — under
+`--capture=fd` default, pytest replaces the parent
+process's fd 1,2 with pipe write-ends it's reading
+from. Fork children inherit those pipe fds
+(because `_close_inherited_fds` correctly preserves
+stdio). High-volume subactor error-log tracebacks
+(7+ actors each logging multiple
+`RemoteActorError`/`ExceptionGroup` tracebacks on
+the error-propagation cascade) fill the 64KB Linux
+pipe buffer. Subactor writes block. Subactor can't
+progress. Process doesn't exit. Parent's
+`_ForkedProc.wait` (now pidfd-based and
+cancellable, but nothing's cancelling here since
+the test body already completed) keeps the pipe
+reader alive... but pytest isn't draining its end
+fast enough because test-teardown/fixture-cleanup
+is in progress.
+
+**Actually** the exact mechanism is slightly
+different: pytest's capture fixture MIGHT be
+actively reading, but faster-than-writer subactors
+overflow its internal buffer. Or pytest might be
+blocked itself on the finalization step.
+
+Either way, `-s` conclusively fixes it.
+
+### Why I ruled this out earlier (and shouldn't have)
+
+Earlier in this investigation I tested
+`test_nested_multierrors` with/without `-s` and
+both hung. That's because AT THAT TIME, fixes 1-4
+weren't all in place yet. The test was hanging at
+multiple deeper levels long before reaching the
+"generate lots of error-log output" phase. Once
+the cascade actually tore down cleanly, enough
+output was produced to hit the capture-pipe limit.
+
+**Classic order-of-operations mistake in
+debugging:** ruling something out too early based
+on a test that was actually failing for a
+different reason.
+
+### Fix direction (next session)
+
+Redirect subactor stdout/stderr to `/dev/null` (or
+a session-scoped log file) in the fork-child
+prelude, right after `_close_inherited_fds()`. This
+severs the inherited pytest-capture pipes and lets
+subactor output flow elsewhere. Under normal
+production use (non-pytest), stdout/stderr would
+be the TTY — we'd want to keep that. So the
+redirect should be conditional or opt-in via the
+`child_sigint`/proc_kwargs flag family.
+
+Alternative: document as a gotcha and recommend
+`pytest -s` for any tests using the
+`subint_forkserver` backend with multi-level actor
+trees. Simpler, user-visible, no code change.
+
+### Current state
+
+- Skip-mark on `test_nested_multierrors[subint_forkserver]`
+  restored with reason pointing here.
+- Test confirmed passing with `-s` after all 4
+  cascade fixes applied.
+- The 4 cascade fixes are NOT wasted — they're
+  correct hardening regardless of the capture-pipe
+  issue, AND without them we'd never reach the
+  "actually produces enough output to fill the
+  pipe" state.
+
+## Stopgap (landed)
+
+`test_nested_multierrors` skip-marked under
+`subint_forkserver` via
+`@pytest.mark.skipon_spawn_backend('subint_forkserver',
+reason='...')`, cross-referenced to this doc. Mark
+should be dropped once the peer-channel-loop exit
+issue is fixed.
+
+## References
+
+- `tractor/spawn/_subint_forkserver.py::fork_from_worker_thread`
+  — the primitive whose post-fork FD hygiene is
+  probably the culprit.
+- `tractor/spawn/_subint_forkserver.py::subint_forkserver_proc`
+  — the backend function that orchestrates the
+  graceful cancel path hitting this bug.
+- `tractor/spawn/_subint_forkserver.py::_ForkedProc`
+  — the `trio.Process`-compatible shim; NOT the
+  failing component (confirmed via thread-dump).
+- `tests/test_cancellation.py::test_nested_multierrors`
+  — the test that surfaced the hang.
+- `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
+  — sibling hang class; probably same underlying
+  fork-FD-inheritance root cause.
+- tractor issue #379 — subint backend tracking.

ai/conc-anal/subint_forkserver_thread_constraints_on_pep684_issue.md

@@ -0,0 +1,186 @@
+# Revisit `subint_forkserver` thread-cache constraints once msgspec PEP 684 support lands
+
+> **Tracked at:** [#450](https://github.com/goodboy/tractor/issues/450)
+
+Follow-up tracker for cleanup work gated on the msgspec
+PEP 684 adoption upstream ([jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
+
+Context — why this exists
+-------------------------
+
+The `tractor.spawn._subint_forkserver` submodule currently
+carries two "non-trio" thread-hygiene constraints whose
+necessity is tangled with issues that *should* dissolve
+under PEP 684 isolated-mode subinterpreters:
+
+1. `fork_from_worker_thread()` / `run_subint_in_worker_thread()`
+   internally allocate a **dedicated `threading.Thread`**
+   rather than using `trio.to_thread.run_sync()`.
+2. The test helper is named
+   `run_fork_in_non_trio_thread()` — the
+   `non_trio` qualifier is load-bearing today.
+
+This doc catalogs *why* those constraints exist, which of
+them isolated-mode would fix, and what the
+audit-and-cleanup path looks like once msgspec #563 is
+resolved.
+
+The three reasons the constraints exist
+---------------------------------------
+
+### 1. GIL-starvation class → fixed by PEP 684 isolated mode
+
+The class-A hang documented in
+`subint_sigint_starvation_issue.md` is entirely about
+legacy-config subints **sharing the main GIL**. Once
+msgspec #563 lands and tractor flips
+`tractor.spawn._subint` to
+`concurrent.interpreters.create()` (isolated config), each
+subint gets its own GIL. Abandoned subint threads can't
+contend for main's GIL → can't starve the main trio loop
+→ signal-wakeup-pipe drains normally → no SIGINT-drop.
+
+This class of hazard **dissolves entirely**. The
+non-trio-thread requirement for *this reason* disappears.
+
+### 2. Destroy race / tstate-recycling → orthogonal; unclear
+
+The `subint_proc` dedicated-thread fix (commit `26fb8206`)
+addressed a different issue: `_interpreters.destroy(interp_id)`
+was blocking on a trio-cache worker that had run an
+earlier `interp.exec()` for that subint. Working
+hypothesis at the time was "the cached thread retains the
+subint's tstate".
+
+But tstate-handling is **not specific to GIL mode** —
+`_PyXI_Enter` / `_PyXI_Exit` (the C-level machinery both
+configs use to enter/leave a subint from a thread) should
+restore the caller's tstate regardless of GIL config. So
+isolated mode **doesn't obviously fix this**. It might be:
+
+- A py3.13 bug fixed in later versions — we saw the race
+  first on 3.13 and never re-tested on 3.14 after moving
+  to dedicated threads.
+- A genuine CPython quirk around cached threads that
+  exec'd into a subint, persisting across GIL modes.
+- Something else we misdiagnosed — the empirical fix
+  (dedicated thread) worked but the analysis may have
+  been incomplete.
+
+Only way to know: once we're on isolated mode, empirically
+retry `trio.to_thread.run_sync(interp.exec, ...)` and see
+if `destroy()` still blocks. If it does, keep the
+dedicated thread; if not, one constraint relaxed.
+
+### 3. Fork-from-main-interp-tstate (the constraint in this module's helper names)
+
+The fork-from-main-interp-tstate invariant — CPython's
+`PyOS_AfterFork_Child` →
+`_PyInterpreterState_DeleteExceptMain` gate documented in
+`subint_fork_blocked_by_cpython_post_fork_issue.md` — is
+about the calling thread's **current** tstate at the
+moment `os.fork()` runs. If trio's cache threads never
+enter subints at all, their tstate is plain main-interp,
+and fork from them would be fine.
+
+The reason the smoke test +
+`run_fork_in_non_trio_thread` test helper
+currently use a dedicated `threading.Thread` is narrow:
+**we don't want to risk a trio cache thread that has
+previously been used as a subint driver being the one that
+picks up the fork job**. If cached tstate doesn't get
+cleared (back to reason #2), the fork's child-side
+post-init would see the wrong interp and abort.
+
+In an isolated-mode world where msgspec works:
+
+- `subint_proc` would use the public
+  `concurrent.interpreters.create()` + `Interpreter.exec()`
+  / `Interpreter.close()` — which *should* handle tstate
+  cleanly (they're the "blessed" API).
+- If so, trio's cache threads are safe to fork from
+  regardless of whether they've previously driven subints.
+- → the `non_trio` qualifier in
+  `run_fork_in_non_trio_thread` becomes
+  *overcautious* rather than load-bearing, and the
+  dedicated-thread primitives in `_subint_forkserver.py`
+  can likely be replaced with straight
+  `trio.to_thread.run_sync()` wrappers.
+
+TL;DR
+-----
+
+| constraint | fixed by isolated mode? |
+|---|---|
+| GIL-starvation (class A) | **yes** |
+| destroy race on cached worker | unclear — empirical test on py3.14 + isolated API required |
+| fork-from-main-tstate requirement on worker | **probably yes, conditional on the destroy-race question above** |
+
+If #2 also resolves on py3.14+ with isolated mode,
+tractor could drop the `non_trio` qualifier from the fork
+helper's name and just use `trio.to_thread.run_sync(...)`
+for everything. But **we shouldn't do that preemptively**
+— the current cautious design is cheap (one dedicated
+thread per fork / per subint-exec) and correct.
+
+Audit plan when msgspec #563 lands
+----------------------------------
+
+Assuming msgspec grows `Py_mod_multiple_interpreters`
+support:
+
+1. **Flip `tractor.spawn._subint` to isolated mode.** Drop
+   the `_interpreters.create('legacy')` call in favor of
+   the public API (`concurrent.interpreters.create()` +
+   `Interpreter.exec()` / `Interpreter.close()`). Run the
+   three `ai/conc-anal/subint_*_issue.md` reproducers —
+   class-A (`test_stale_entry_is_deleted` etc.) should
+   pass without the `skipon_spawn_backend('subint')` marks
+   (revisit the marker inventory).
+
+2. **Empirical destroy-race retest.** In `subint_proc`,
+   swap the dedicated `threading.Thread` back to
+   `trio.to_thread.run_sync(Interpreter.exec, ...,
+   abandon_on_cancel=False)` and run the full subint test
+   suite. If `Interpreter.close()` (or the backing
+   destroy) blocks the same way as the legacy version
+   did, revert and keep the dedicated thread.
+
+3. **If #2 clean**, audit `_subint_forkserver.py`:
+   - Rename `run_fork_in_non_trio_thread` → drop the
+     `_non_trio_` qualifier (e.g. `run_fork_in_thread`) or
+     inline the two-line `trio.to_thread.run_sync` call at
+     the call sites and drop the helper entirely.
+   - Consider whether `fork_from_worker_thread` +
+     `run_subint_in_worker_thread` still warrant being
+     separate module-level primitives or whether they
+     collapse into a compound
+     `trio.to_thread.run_sync`-driven pattern inside the
+     (future) `subint_forkserver_proc` backend.
+
+4. **Doc fallout.** `subint_sigint_starvation_issue.md`
+   and `subint_cancel_delivery_hang_issue.md` both cite
+   the legacy-GIL-sharing architecture as the root cause.
+   Close them with commit-refs to the isolated-mode
+   migration. This doc itself should get a closing
+   post-mortem section noting which of #1/#2/#3 actually
+   resolved vs persisted.
+
+References
+----------
+
+- `tractor.spawn._subint_forkserver` — the in-tree module
+  whose constraints this doc catalogs.
+- `ai/conc-anal/subint_sigint_starvation_issue.md` — the
+  GIL-starvation class.
+- `ai/conc-anal/subint_cancel_delivery_hang_issue.md` —
+  sibling Ctrl-C-able hang class.
+- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+  — why fork-from-subint is blocked (this drives the
+  forkserver-via-non-subint-thread workaround).
+- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
+  — empirical validation for the workaround.
+- [PEP 684 — per-interpreter GIL](https://peps.python.org/pep-0684/)
+- [PEP 734 — `concurrent.interpreters` public API](https://peps.python.org/pep-0734/)
+- [jcrist/msgspec#563 — PEP 684 support tracker](https://github.com/jcrist/msgspec/issues/563)
+- tractor issue #379 — subint backend tracking.

ai/conc-anal/subint_sigint_starvation_issue.md

@@ -0,0 +1,350 @@
+# `subint` backend: abandoned-subint thread can wedge main trio event loop (Ctrl-C unresponsive)
+
+Follow-up to the Phase B subint spawn-backend PR (see
+`tractor.spawn._subint`, issue #379). The hard-kill escape
+hatch we landed (`_HARD_KILL_TIMEOUT`, bounded shields,
+`daemon=True` driver-thread abandonment) handles *most*
+stuck-subint scenarios cleanly, but there's one class of
+hang that can't be fully escaped from within tractor: a
+still-running abandoned sub-interpreter can starve the
+**parent's** trio event loop to the point where **SIGINT is
+effectively dropped by the kernel ↔ Python boundary** —
+making the pytest process un-Ctrl-C-able.
+
+## Symptom
+
+Running `test_stale_entry_is_deleted[subint]` under
+`--spawn-backend=subint`:
+
+1. Test spawns a subactor (`transport_fails_actor`) which
+   kills its own IPC server and then
+   `trio.sleep_forever()`.
+2. Parent tries `Portal.cancel_actor()` → channel
+   disconnected → fast return.
+3. Nursery teardown triggers our `subint_proc` cancel path.
+   Portal-cancel fails (dead channel),
+   `_HARD_KILL_TIMEOUT` fires, driver thread is abandoned
+   (`daemon=True`), `_interpreters.destroy(interp_id)`
+   raises `InterpreterError` (because the subint is still
+   running).
+4. Test appears to hang indefinitely at the *outer*
+   `async with tractor.open_nursery() as an:` exit.
+5. `Ctrl-C` at the terminal does nothing. The pytest
+   process is un-interruptable.
+
+## Evidence
+
+### `strace` on the hung pytest process
+
+```
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(37, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
+rt_sigreturn({mask=[WINCH]}) = 140585542325792
+```
+
+Translated:
+
+- Kernel delivers `SIGINT` to pytest.
+- CPython's C-level signal handler fires and tries to
+  write the signal number byte (`0x02` = SIGINT) to fd 37
+  — the **Python signal-wakeup fd** (set via
+  `signal.set_wakeup_fd()`, which trio uses to wake its
+  event loop on signals).
+- Write returns `EAGAIN` — **the pipe is full**. Nothing
+  is draining it.
+- `rt_sigreturn` with the signal masked off — signal is
+  "handled" from the kernel's perspective but the actual
+  Python-level handler (and therefore trio's
+  `KeyboardInterrupt` delivery) never runs.
+
+### Stack dump (via `tractor.devx.dump_on_hang`)
+
+At 20s into the hang, only the **main thread** is visible:
+
+```
+Thread 0x...7fdca0191780 [python] (most recent call first):
+  File ".../trio/_core/_io_epoll.py", line 245 in get_events
+  File ".../trio/_core/_run.py", line 2415 in run
+  File ".../tests/discovery/test_registrar.py", line 575 in test_stale_entry_is_deleted
+  ...
+```
+
+No driver thread shows up. The abandoned-legacy-subint
+thread still exists from the OS's POV (it's still running
+inside `_interpreters.exec()` driving the subint's
+`trio.run()` on `trio.sleep_forever()`) but the **main
+interp's faulthandler can't see threads currently executing
+inside a sub-interpreter's tstate**. Concretely: the thread
+is alive, holding state we can't introspect from here.
+
+## Root cause analysis
+
+The most consistent explanation for both observations:
+
+1. **Legacy-config subinterpreters share the main GIL.**
+   PEP 734's public `concurrent.interpreters.create()`
+   defaults to `'isolated'` (per-interp GIL), but tractor
+   uses `_interpreters.create('legacy')` as a workaround
+   for C extensions that don't yet support PEP 684
+   (notably `msgspec`, see
+   [jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
+   Legacy-mode subints share process-global state
+   including the GIL.
+
+2. **Our abandoned subint thread never exits.** After our
+   hard-kill timeout, `driver_thread.join()` is abandoned
+   via `abandon_on_cancel=True` and the thread is
+   `daemon=True` so proc-exit won't block on it — but the
+   thread *itself* is still alive inside
+   `_interpreters.exec()`, driving a `trio.run()` that
+   will never return (the subint actor is in
+   `trio.sleep_forever()`).
+
+3. **`_interpreters.destroy()` cannot force-stop a running
+   subint.** It raises `InterpreterError` on any
+   still-running subinterpreter; there is no public
+   CPython API to force-destroy one.
+
+4. **Shared-GIL + non-terminating subint thread → main
+   trio loop starvation.** Under enough load (the subint's
+   trio event loop iterating in the background, IPC-layer
+   tasks still in the subint, etc.) the main trio event
+   loop can fail to iterate frequently enough to drain its
+   wakeup pipe. Once that pipe fills, `SIGINT` writes from
+   the C signal handler return `EAGAIN` and signals are
+   silently dropped — exactly what `strace` shows.
+
+The shielded
+`await actor_nursery._join_procs.wait()` at the top of
+`subint_proc` (inherited unchanged from the `trio_proc`
+pattern) is structurally involved too: if main trio *does*
+get a schedule slice, it'd find the `subint_proc` task
+parked on `_join_procs` under shield — which traps whatever
+`Cancelled` arrives. But that's a second-order effect; the
+signal-pipe-full condition is the primary "Ctrl-C doesn't
+work" cause.
+
+## Why we can't fix this from inside tractor
+
+- **No force-destroy API.** CPython provides neither a
+  `_interpreters.force_destroy()` nor a thread-
+  cancellation primitive (`pthread_cancel` is actively
+  discouraged and unavailable on Windows). A subint stuck
+  in pure-Python loops (or worse, C code that doesn't poll
+  for signals) is structurally unreachable from outside.
+- **Shared GIL is the root scheduling issue.** As long as
+  we're forced into legacy-mode subints for `msgspec`
+  compatibility, the abandoned-thread scenario is
+  fundamentally a process-global GIL-starvation window.
+- **`signal.set_wakeup_fd()` is process-global.** Even if
+  we wanted to put our own drainer on the wakeup pipe,
+  only one party owns it at a time.
+
+## Current workaround
+
+- **Fixture-side SIGINT loop on the `daemon` subproc** (in
+  this test's `daemon: subprocess.Popen` fixture in
+  `tests/conftest.py`). The daemon dying closes its end of
+  the registry IPC, which unblocks a pending recv in main
+  trio's IPC-server task, which lets the event loop
+  iterate, which drains the wakeup pipe, which finally
+  delivers the test-harness SIGINT.
+- **Module-level skip on py3.13**
+  (`pytest.importorskip('concurrent.interpreters')`) — the
+  private `_interpreters` C module exists on 3.13 but the
+  multi-trio-task interaction hangs silently there
+  independently of this issue.
+
+## Path forward
+
+1. **Primary**: upstream `msgspec` PEP 684 adoption
+   ([jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)).
+   Unlocks `concurrent.interpreters.create()` isolated
+   mode → per-interp GIL → abandoned subint threads no
+   longer starve the parent's main trio loop. At that
+   point we can flip `_subint.py` back to the public API
+   (`create()` / `Interpreter.exec()` / `Interpreter.close()`)
+   and drop the private `_interpreters` path.
+
+2. **Secondary**: watch CPython for a public
+   force-destroy primitive. If something like
+   `Interpreter.close(force=True)` lands, we can use it as
+   a hard-kill final stage and actually tear down
+   abandoned subints.
+
+3. **Harness-level**: document the fixture-side SIGINT
+   loop pattern as the "known workaround" for subint-
+   backend tests that can leave background state holding
+   the main event loop hostage.
+
+## References
+
+- PEP 734 (`concurrent.interpreters`):
+  <https://peps.python.org/pep-0734/>
+- PEP 684 (per-interpreter GIL):
+  <https://peps.python.org/pep-0684/>
+- `msgspec` PEP 684 tracker:
+  <https://github.com/jcrist/msgspec/issues/563>
+- CPython `_interpretersmodule.c` source:
+  <https://github.com/python/cpython/blob/main/Modules/_interpretersmodule.c>
+- `tractor.spawn._subint` module docstring (in-tree
+  explanation of the legacy-mode choice and its
+  tradeoffs).
+
+## Reproducer
+
+```
+./py314/bin/python -m pytest \
+  tests/discovery/test_registrar.py::test_stale_entry_is_deleted \
+  --spawn-backend=subint \
+  --tb=short --no-header -v
+```
+
+Hangs indefinitely without the fixture-side SIGINT loop;
+with the loop, the test completes (albeit with the
+abandoned-thread warning in logs).
+
+## Additional known-hanging tests (same class)
+
+All three tests below exhibit the same
+signal-wakeup-fd-starvation fingerprint (`write() → EAGAIN`
+on the wakeup pipe after enough SIGINT attempts) and
+share the same structural cause — abandoned legacy-subint
+driver threads contending with the main interpreter for
+the shared GIL until the main trio loop can no longer
+drain its wakeup pipe fast enough to deliver signals.
+
+They're listed separately because each exposes the class
+under a different load pattern worth documenting.
+
+### `tests/discovery/test_registrar.py::test_stale_entry_is_deleted[subint]`
+
+Original exemplar — see the **Symptom** and **Evidence**
+sections above. One abandoned subint
+(`transport_fails_actor`, stuck in `trio.sleep_forever()`
+after self-cancelling its IPC server) is sufficient to
+tip main into starvation once the harness's `daemon`
+fixture subproc keeps its half of the registry IPC alive.
+
+### `tests/test_cancellation.py::test_cancel_while_childs_child_in_sync_sleep[subint-False]`
+
+Cancel a grandchild that's in sync Python sleep from 2
+nurseries up. The test's own docstring declares the
+dependency: "its parent should issue a 'zombie reaper' to
+hard kill it after sufficient timeout" — which for
+`trio`/`mp_*` is an OS-level `SIGKILL` of the grandchild
+subproc. **Under `subint` there's no equivalent** (no
+public CPython API to force-destroy a running
+sub-interpreter), so the grandchild's sync-sleeping
+`trio.run()` persists inside its abandoned driver thread
+indefinitely. The nested actor-tree (parent → child →
+grandchild, all subints) means a single cancel triggers
+multiple concurrent hard-kill abandonments, each leaving
+a live driver thread.
+
+This test often only manifests the starvation under
+**full-suite runs** rather than solo execution —
+earlier-in-session subint tests also leave abandoned
+driver threads behind, and the combined population is
+what actually tips main trio into starvation. Solo runs
+may stay Ctrl-C-able with fewer abandoned threads in the
+mix.
+
+### `tests/test_cancellation.py::test_multierror_fast_nursery[subint-25-0.5]`
+
+Nursery-error-path throughput stress-test parametrized
+for **25 concurrent subactors**. When the multierror
+fires and the nursery cancels, every subactor goes
+through our `subint_proc` teardown. The bounded
+hard-kills run in parallel (all `subint_proc` tasks are
+sibling trio tasks), so the timeout budget is ~3s total
+rather than 3s × 25. After that, **25 abandoned
+`daemon=True` driver threads are simultaneously alive** —
+an extreme pressure multiplier on the same mechanism.
+
+The `strace` fingerprint is striking under this load: six
+or more **successful** `write(16, "\2", 1) = 1` calls
+(main trio getting brief GIL slices, each long enough to
+drain exactly one wakeup-pipe byte) before finally
+saturating with `EAGAIN`:
+
+```
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = 1
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = -1 EAGAIN (Resource temporarily unavailable)
+rt_sigreturn({mask=[WINCH]})            = 140141623162400
+```
+
+Those successful writes indicate CPython's
+`sys.getswitchinterval()`-based GIL round-robin *is*
+giving main brief slices — just never long enough to run
+the Python-level signal handler through to the point
+where trio converts the delivered SIGINT into a
+`Cancelled` on the appropriate scope. Once the
+accumulated write rate outpaces main's drain rate, the
+pipe saturates and subsequent signals are silently
+dropped.
+
+The `pstree` below (pid `530060` = hung `pytest`) shows
+the subint-driver thread population at the moment of
+capture. Even with fewer than the full 25 shown (pstree
+truncates thread names to `subint-driver[<interp_id>` —
+interpreters `3` and `4` visible across 16 thread
+entries), the GIL-contender count is more than enough to
+explain the starvation:
+
+```
+>>> pstree -snapt 530060
+systemd,1 --switched-root --system --deserialize=40
+  └─login,1545 --
+      └─bash,1872
+          └─sway,2012
+              └─alacritty,70471 -e xonsh
+                  └─xonsh,70487 .../bin/xonsh
+                      └─uv,70955 run xonsh
+                          └─xonsh,70959 .../py314/bin/xonsh
+                              └─python,530060 .../py314/bin/pytest -v tests/test_cancellation.py --spawn-backend=subint
+                                  ├─{subint-driver[3},531857
+                                  ├─{subint-driver[3},531860
+                                  ├─{subint-driver[3},531862
+                                  ├─{subint-driver[3},531866
+                                  ├─{subint-driver[3},531877
+                                  ├─{subint-driver[3},531882
+                                  ├─{subint-driver[3},531884
+                                  ├─{subint-driver[3},531945
+                                  ├─{subint-driver[3},531950
+                                  ├─{subint-driver[3},531952
+                                  ├─{subint-driver[4},531956
+                                  ├─{subint-driver[4},531959
+                                  ├─{subint-driver[4},531961
+                                  ├─{subint-driver[4},531965
+                                  ├─{subint-driver[4},531968
+                                  └─{subint-driver[4},531979
+```
+
+(`pstree` uses `{...}` to denote threads rather than
+processes — these are all the **driver OS-threads** our
+`subint_proc` creates with name
+`f'subint-driver[{interp_id}]'`. Every one of them is
+still alive, executing `_interpreters.exec()` inside a
+sub-interpreter our hard-kill has abandoned. At 16+
+abandoned driver threads competing for the main GIL, the
+main-interpreter trio loop gets starved and signal
+delivery stalls.)

ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md

@@ -0,0 +1,273 @@
+# `test_register_duplicate_name` racy connect-failure on `daemon` fixture readiness
+
+## Symptom
+
+`tests/test_multi_program.py::test_register_duplicate_name`
+fails intermittently under BOTH transports + ALL spawn
+backends with connect-refused errors:
+
+```
+# under --tpt-proto=uds
+FAILED tests/test_multi_program.py::test_register_duplicate_name
+- ConnectionRefusedError: [Errno 111] Connection refused
+( ^^^ this exc was collapsed from a group ^^^ )
+
+# under --tpt-proto=tcp
+FAILED tests/test_multi_program.py::test_register_duplicate_name
+- OSError: all attempts to connect to 127.0.0.1:36003 failed
+( ^^^ this exc was collapsed from a group ^^^ )
+```
+
+Distinct from the cancel-cascade `TooSlowError` flake
+class — see
+`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+This is a **connect-time race** before the daemon is
+fully ready to `accept()`, not a teardown-cascade
+slowness.
+
+## Root cause: blind `time.sleep()` in `daemon` fixture
+
+`tests/conftest.py::daemon` boots a sub-py-process via
+`subprocess.Popen([python, '-c', 'tractor.run_daemon(...)'])`,
+then **blindly sleeps** a fixed delay before yielding
+`proc` to the test:
+
+```python
+# excerpt from tests/conftest.py::daemon
+proc = subprocess.Popen([
+    sys.executable, '-c', code,
+])
+
+bg_daemon_spawn_delay: float = _PROC_SPAWN_WAIT  # 0.6
+if tpt_proto == 'uds':
+    bg_daemon_spawn_delay += 1.6
+if _non_linux and ci_env:
+    bg_daemon_spawn_delay += 1
+
+# XXX, allow time for the sub-py-proc to boot up.
+# !TODO, see ping-polling ideas above!
+time.sleep(bg_daemon_spawn_delay)
+
+assert not proc.returncode
+yield proc
+```
+
+Inherent fragility: the delay is "long enough on dev
+boxes most of the time" but has no actual
+synchronization with the daemon's `bind()` + `listen()`
+completion. Under any of:
+
+- Loaded box (CI parallelism, big rebuild in
+  background, low-cpu-freq)
+- Cold first-run (`importlib` cache miss, JIT warmup)
+- Higher-than-expected `tractor` import cost
+- Filesystem latency (UDS sockfile create, slow
+  tmpfs)
+
+...the sleep finishes BEFORE the daemon has bound its
+listen socket → first test client call to
+`tractor.find_actor()` / `wait_for_actor()` /
+`open_nursery(registry_addrs=[reg_addr])`'s implicit
+connect → `ConnectionRefusedError` (TCP) or
+`FileNotFoundError`/`ConnectionRefusedError` (UDS).
+
+## Reproducer
+
+Easiest: run the suite under load.
+
+```bash
+# create CPU pressure on another core in parallel
+stress-ng --cpu 2 --timeout 600s &
+
+./py313/bin/python -m pytest \
+  tests/test_multi_program.py::test_register_duplicate_name \
+  --spawn-backend=main_thread_forkserver \
+  --tpt-proto=tcp -v
+```
+
+Reproduces ~30-50% of the time on a dev laptop. On a
+quiet idle box, may need 5-10 runs to hit.
+
+## Why the existing `_PROC_SPAWN_WAIT` tuning is
+inadequate
+
+Recent `bg_daemon_spawn_delay` rename
+(de-monotonic-grow fix) just-shipped removed the
+*accumulation* bug where each invocation made the
+NEXT test's wait longer too. Net effect: every
+invocation now uses the SAME `0.6 + 1.6` (UDS) or
+`0.6` (TCP) sleep, no growth. Good — but does
+NOTHING for the underlying race. Each individual
+test still relies on a blind sleep that may or may
+not be sufficient.
+
+Bumping the constant higher pushes flake rate down
+but never to zero AND adds dead time to every
+non-flaking run. Not a fix, just a knob.
+
+## Side effects
+
+- **Inter-test cascade**: a single failure can cascade
+  via leaked subprocesses (the `daemon` fixture's
+  cleanup may not fully tear down a daemon that never
+  reached "ready"). The `_reap_orphaned_subactors`
+  session-end + `_track_orphaned_uds_per_test`
+  per-test fixtures handle most of this now, but the
+  affected test itself still fails.
+- **Worsens under fork-spawn backends**: the daemon
+  has more init work
+  (`_main_thread_forkserver`-coordinator-thread
+  startup, etc.) so the sleep has to cover MORE.
+
+## Fix design — replace blind sleep with active poll
+
+The right primitive is **poll the daemon's bind
+address until it accepts a connection or we time
+out**, with the timeout being a hard ceiling rather
+than a baseline. Two implementation paths:
+
+### Path A — TCP/UDS connect-poll loop
+
+Try `socket.connect(reg_addr)` in a tight loop with
+short backoff (~50ms), succeed on the first non-error
+return, fail-loud on a hard cap (e.g. 10s). Same
+primitive works for both transports because both use
+`socket.connect()` semantics.
+
+Rough shape:
+
+```python
+def _wait_for_daemon_ready(
+    reg_addr,
+    tpt_proto: str,
+    timeout: float = 10.0,
+    poll_interval: float = 0.05,
+) -> None:
+    deadline = time.monotonic() + timeout
+    while True:
+        if tpt_proto == 'tcp':
+            sock = socket.socket(socket.AF_INET)
+            target = reg_addr  # (host, port)
+        else:  # uds
+            sock = socket.socket(socket.AF_UNIX)
+            target = os.path.join(*reg_addr)
+        try:
+            sock.settimeout(poll_interval)
+            sock.connect(target)
+        except (
+            ConnectionRefusedError,
+            FileNotFoundError,
+            socket.timeout,
+        ) as exc:
+            if time.monotonic() >= deadline:
+                raise TimeoutError(
+                    f'Daemon never accepted on {target!r} '
+                    f'within {timeout}s'
+                ) from exc
+            time.sleep(poll_interval)
+        else:
+            sock.close()
+            return
+```
+
+Pros: trivial primitive, no tractor-runtime
+dependency, works pre-yield in the fixture body,
+fail-fast on truly-broken daemon.
+Cons: doesn't actually do an IPC handshake, just
+proves listen-side is up. A daemon that bound but
+hasn't initialized its registrar table yet would
+still race.
+
+### Path B — `tractor.find_actor()` poll
+
+Use the actual discovery API the test would call:
+
+```python
+async def _wait_for_daemon_ready_via_discovery(
+    reg_addr,
+    timeout: float = 10.0,
+    poll_interval: float = 0.05,
+):
+    deadline = trio.current_time() + timeout
+    async with tractor.open_root_actor(
+        registry_addrs=[reg_addr],
+        # ephemeral root just for the probe
+    ):
+        while True:
+            try:
+                async with tractor.find_actor(
+                    'registrar',  # daemon's own name
+                    registry_addrs=[reg_addr],
+                ) as portal:
+                    if portal is not None:
+                        return
+            except Exception:
+                pass
+            if trio.current_time() >= deadline:
+                raise TimeoutError(...)
+            await trio.sleep(poll_interval)
+```
+
+Pros: actually proves the discovery path works,
+handles the "bound but not ready" case naturally.
+Cons: requires booting an ephemeral root actor JUST
+for the probe (overhead), more code, and runs in trio
+which complicates the sync-fixture context. Need a
+`trio.run()` wrapper.
+
+### Recommended: Path A with optional handshake check
+
+Path A is much simpler + handles 95% of the bug
+class. If "bound-but-not-ready" turns out to still
+race (it shouldn't — `tractor.run_daemon` doesn't
+return from `bind()` until the registrar is
+fully populated), escalate to Path B as a focused
+follow-up.
+
+## Workarounds (until fix lands)
+
+1. **Bump `_PROC_SPAWN_WAIT`** higher (current: 0.6).
+   2.0–3.0 hides most flakes at the cost of adding
+   dead time to every test. Not a fix but reduces
+   blast radius while the proper poll lands.
+2. **`pytest-rerunfailures`** with `reruns=1` on the
+   `daemon` fixture's tests specifically. Hides the
+   flake but doesn't address it.
+3. **Mark known-affected tests as `xfail(strict=False)`**
+   under `--ci`. Lets CI go green at the cost of
+   silently hiding regressions.
+
+(Recommend skipping all three — implement the active
+poll instead.)
+
+## Investigation next steps
+
+1. Implement Path A as a `_wait_for_daemon_ready()`
+   helper in `tests/conftest.py`. Replace the
+   `time.sleep(bg_daemon_spawn_delay)` call with it.
+2. Drop the `_PROC_SPAWN_WAIT` constant entirely
+   (active poll obsoletes blind sleep).
+3. Run the suite 5-10 times to validate flake rate
+   drops to 0.
+4. If flakes persist, profile whether the daemon
+   process exits with non-zero before the poll's
+   deadline hits — that'd be a different bug
+   (daemon startup crash) that the blind sleep was
+   masking.
+5. Cross-check `tests/test_multi_program.py::test_*`
+   — multiple tests use the `daemon` fixture; all
+   should benefit from the same poll primitive.
+
+## Related
+
+- `tests/conftest.py::daemon` — the fixture under
+  fix
+- `tests/conftest.py::_PROC_SPAWN_WAIT` — the
+  constant to drop
+- `cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`
+  — distinct flake class (cancel-cascade
+  `TooSlowError` at teardown, not connect-time race)
+- `trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
+  — different bug entirely; this race was masked
+  pre-WakeupSocketpair-patch by the busy-loop
+  hangs.

ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md

@@ -0,0 +1,102 @@
+# `trio` 0.29 -> 0.33 slows the depth=3 cancel-cascade
+
+## Symptom
+
+After locking to `trio==0.33.0` (commit `c7741bba`, was
+`0.29.0`), this test reliably trips its `fail_after`
+deadline on the **`trio`** backend:
+
+```
+FAILED tests/test_cancellation.py::test_nested_multierrors[start_method=trio-depth=3]
+  - AssertionError: assert False
+    where False = isinstance(
+      Cancelled(source='deadline', source_task=None, reason=None),
+      tractor.RemoteActorError,
+    )
+```
+
+A `fail_after_w_trace` hang-snapshot is captured for the
+test each run (deadline-injected `Cancelled` wrapped into
+the actor-nursery `BaseExceptionGroup`).
+
+## Root cause (immediate)
+
+The test budgets `fail_after(6)` for the `trio` backend.
+That 6s was chosen (commit `32955db0`, while `trio==0.29`)
+with the assertion that trio finishes "well under" 6s.
+The `trio` 0.29 -> 0.33 bump slowed the depth=3 cascade
+past that budget, so the 6s deadline now fires mid-cascade.
+
+trio 0.33 added **cancel-reason tracking** — every
+`Cancelled` now carries `(source=, reason=, source_task=)`.
+The injected exc is `Cancelled(source='deadline')`, i.e.
+trio itself naming our `fail_after(6)` scope as the cancel
+origin. When that `Cancelled` collapses one branch of the
+nursery BEG, the test's `isinstance(subexc,
+RemoteActorError)` assertion fails. The healthy outcome is
+`BEG = [RemoteActorError, RemoteActorError]`; the
+`Cancelled` is purely an artifact of the deadline cutting
+the cascade short.
+
+## Measurements (standalone, this machine)
+
+```
+depth=1  trio   ~3.15s   PASS  (keeps 6s budget)
+depth=3  trio   ~6.8-8.2s  FAIL @ 6s  (now bumped to 12s)
+```
+
+depth=1 still fits comfortably; only depth=3 (deeper
+recursive spawn-and-error tree => more actors to reap)
+exceeds the old budget. The ~2s/depth-level cost looks
+like serialized per-actor reap / `terminate_after` waits.
+
+## Mitigation applied
+
+`test_nested_multierrors` now splits the `trio` budget:
+
+```python
+case ('trio', 1):
+    timeout = 6
+case ('trio', 3):
+    timeout = 12   # was 6; see this doc
+```
+
+This stops the deadline from firing so the cascade
+completes naturally to `[RAE, RAE]`.
+
+## Also affected — same root cause, different test
+
+`test_echoserver_detailed_mechanics[trio-raise_error=KeyboardInterrupt]`
+(`tests/test_infected_asyncio.py`) tripped the *same*
+slowdown via its much tighter `trio` budget of `1s`. The
+single-aio-subactor teardown now takes ~1s, so the `1s`
+`fail_after` raced the deadline (PASS at 0.99s / FAIL at
+1.03s across back-to-back standalone runs). On a deadline-
+fire the injected `Cancelled(source='deadline')` wraps the
+mid-stream `KeyboardInterrupt` into a `BaseExceptionGroup`,
+which is NOT a `KeyboardInterrupt` so the bare
+`pytest.raises(KeyboardInterrupt)` fails. (The sibling
+`raise_error=Exception` variant only "passes" by accident:
+an `ExceptionGroup` *is-a* `Exception`, so its
+`pytest.raises(Exception)` still matches even when wrapped.)
+
+Mitigation: bump that `trio` budget `1 -> 4s` (matching the
+forking-spawner case). Without a deadline-fire the KBI
+propagates bare and the assertion passes.
+
+## Open follow-up (the actual regression)
+
+The budget bump is a band-aid — the underlying question is
+**why** the depth=3 `trio` cancel-cascade went from <6s to
+~7-8s across `trio` 0.29 -> 0.33. Candidate avenues:
+
+- which scope owns the per-actor `terminate_after` wait,
+  and are the tree's reaps concurrent or serialized?
+- did trio 0.33's abort/reschedule or cancel-reason
+  bookkeeping change checkpoint timing on the cancel path?
+
+If/when the cascade speeds back up under-budget, depth=3
+will start completing well under 12s — at which point the
+budget can be tightened back toward 6s as a regression
+tripwire. Related (different backend, same cascade class):
+`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.

ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md

@@ -0,0 +1,221 @@
+# trio `WakeupSocketpair.drain()` busy-loop in forked child (peer-closed missed-EOF)
+
+## Reproducer
+
+```bash
+./py313/bin/python -m pytest \
+  tests/test_multi_program.py::test_register_duplicate_name \
+  --tpt-proto=tcp \
+  --spawn-backend=main_thread_forkserver \
+  -v --capture=sys
+```
+
+Subactor pegs a CPU core indefinitely; parent test
+hangs waiting for the subactor.
+
+## Empirical evidence (caught alive)
+
+```
+$ sudo strace -p <subactor-pid>
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+recvfrom(6, "", 65536, 0, NULL, NULL)   = 0
+... (no `epoll_wait`, no other syscalls, just this back-to-back)
+```
+
+Pattern: tight C-level `recvfrom` loop returning 0
+each call. No `epoll_wait` between iterations →
+**not trio's task scheduler**. Pure synchronous C
+loop.
+
+```
+$ sudo readlink /proc/<subactor-pid>/fd/6
+socket:[<inode>]
+
+$ sudo lsof -p <subactor-pid> | grep ' 6u'
+<cmd> <pid> goodboy 6u unix 0xffff... 0t0 <inode> type=STREAM (CONNECTED)
+```
+
+fd=6 is an **AF_UNIX socket** in CONNECTED state.
+Even though the test uses `--tpt-proto=tcp`, this fd
+is NOT a tractor IPC channel — it's an internal
+trio socketpair.
+
+## Root-cause: `WakeupSocketpair.drain()`
+
+`/site-packages/trio/_core/_wakeup_socketpair.py`:
+
+```python
+class WakeupSocketpair:
+    def __init__(self) -> None:
+        self.wakeup_sock, self.write_sock = socket.socketpair()
+        self.wakeup_sock.setblocking(False)
+        self.write_sock.setblocking(False)
+        ...
+
+    def drain(self) -> None:
+        try:
+            while True:
+                self.wakeup_sock.recv(2**16)
+        except BlockingIOError:
+            pass
+```
+
+`socket.socketpair()` on Linux defaults to AF_UNIX
+SOCK_STREAM. Both ends non-blocking. Normal flow:
+
+1. Signal/wake event → `write_sock.send(b'\x00')`
+   queues a byte.
+2. `wakeup_sock` becomes readable → trio's epoll
+   triggers.
+3. Trio calls `drain()` to flush the buffer.
+4. drain loops on `wakeup_sock.recv(64KB)`.
+5. Eventually buffer empty → non-blocking socket
+   raises `BlockingIOError` → except → break.
+
+**Bug surface — peer-closed missed-EOF**:
+
+Non-blocking socket semantics:
+- buffer has data → `recv` returns N>0 bytes (loop continues)
+- buffer empty → `recv` raises `BlockingIOError`
+- **peer FIN'd → `recv` returns 0 bytes (NEITHER exception NOR
+  break — infinite tight loop)**
+
+`drain()` does not handle the `b''` return-value
+(EOF) case. If `write_sock` has been closed (or the
+process holding it is gone), every iteration returns
+0 → infinite loop → 100% CPU on a single core.
+
+## Why this triggers under `main_thread_forkserver`
+
+Under `os.fork()` from the forkserver-worker thread:
+
+1. Parent has a `WakeupSocketpair` instance with
+   `wakeup_sock=fdN`, `write_sock=fdM`. Both fds
+   open in parent.
+2. Fork → child inherits BOTH fds (kernel-level fd
+   table dup).
+3. `_close_inherited_fds()` runs in child →
+   closes everything except stdio. `wakeup_sock` and
+   `write_sock` of the parent's `WakeupSocketpair`
+   ARE closed in child.
+4. Child's trio (running fresh) creates its OWN
+   `WakeupSocketpair` → NEW fd numbers (e.g. fd 6, 7).
+5. **In `infect_asyncio` mode** the asyncio loop is
+   the host; trio runs as guest via
+   `start_guest_run`. trio still creates its
+   `WakeupSocketpair` in the I/O manager but its
+   role is different.
+
+The race window: somewhere between (3) and (5), if a
+`WakeupSocketpair` Python object reference inherited
+via COW (from parent's pre-fork heap) survives long
+enough that `drain()` is called on it AFTER its fds
+were closed but BEFORE the child's NEW socketpair
+takes over the recycled fd numbers — the recycled fd
+will be one of the child's NEW socketpair ends, whose
+peer might be FIN-flagged (e.g. parent-process
+peer-end is closed).
+
+Or simpler: the `wait_for_actor`/`find_actor` discovery
+flow in `test_register_duplicate_name` triggers an
+unusual code path where a stale `WakeupSocketpair`
+gets `drain()`-called on a fd whose peer has already
+closed.
+
+## Why `drain()` shouldn't loop indefinitely on EOF
+(upstream trio bug)
+
+Even WITHOUT fork, `drain()` should treat `b''` as
+EOF and break. The current code is correct for the
+"buffer drained on a healthy socketpair" scenario but
+incorrect for the "peer is gone" scenario. It's a
+defensive-programming gap in trio.
+
+A one-line patch upstream:
+
+```python
+def drain(self) -> None:
+    try:
+        while True:
+            data = self.wakeup_sock.recv(2**16)
+            if not data:
+                break  # peer-closed; nothing more to drain
+    except BlockingIOError:
+        pass
+```
+
+## Workarounds (until the underlying issue lands)
+
+1. **Skip-mark on the fork backend**:
+   `tests/test_multi_program.py` →
+   `pytest.mark.skipon_spawn_backend('main_thread_forkserver',
+   reason='trio WakeupSocketpair.drain busy-loop, see ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md')`.
+
+2. **Defensive monkey-patch in tractor's
+   forkserver-child prelude** — wrap
+   `WakeupSocketpair.drain` to handle `b''`:
+
+   ```python
+   # in `_actor_child_main` or `_close_inherited_fds`'s
+   # post-fork prelude:
+   from trio._core._wakeup_socketpair import WakeupSocketpair
+   _orig_drain = WakeupSocketpair.drain
+   def _safe_drain(self):
+       try:
+           while True:
+               data = self.wakeup_sock.recv(2**16)
+               if not data:
+                   return  # peer closed
+       except BlockingIOError:
+           pass
+   WakeupSocketpair.drain = _safe_drain
+   ```
+
+   Tracks upstream — remove once trio fixes.
+
+3. **Upstream the fix**: 1-line PR to `python-trio/trio`
+   adding `if not data: break` to `drain()`.
+
+## Investigation next steps
+
+1. **Confirm via py-spy**: when caught alive, detach
+   strace first then
+   `sudo py-spy dump --pid <subactor> --locals`. The
+   busy thread should show `drain` from `WakeupSocketpair`
+   in the call chain.
+2. **Identify which write-end peer is closed**: from
+   the inode of fd 6, look up the matching peer
+   inode via `ss -xp` and see whose process it
+   was/is.
+3. **Verify the missed-EOF hypothesis**: hand-craft a
+   minimal `WakeupSocketpair` repro:
+
+   ```python
+   from trio._core._wakeup_socketpair import WakeupSocketpair
+   ws = WakeupSocketpair()
+   ws.write_sock.close()  # simulate peer-gone
+   ws.drain()             # should hang forever
+   ```
+
+## Sibling bug
+
+`tests/test_infected_asyncio.py::test_aio_simple_error`
+hangs under the same backend with a DIFFERENT
+fingerprint (Mode-A deadlock, both parties in
+`epoll_wait`, no busy-loop). Distinct root cause —
+see `infected_asyncio_under_main_thread_forkserver_hang_issue.md`.
+
+Both share the broader theme: **trio internal-state
+initialization isn't fully fork-safe under
+`main_thread_forkserver`** for the more exotic
+dispatch paths.
+
+## See also
+
+- [#379](https://github.com/goodboy/tractor/issues/379) — subint umbrella
+- python-trio/trio#1614 — trio + fork hazards
+- `trio._core._wakeup_socketpair.WakeupSocketpair`
+  source (the smoking gun)
+- `ai/conc-anal/fork_thread_semantics_execution_vs_memory.md`
+- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`

ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.md

@@ -0,0 +1,97 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subints-spawner-phase-b1-impl
+timestamp: 2026-04-17T12:44:37Z
+git_ref: 5cd6df5
+scope: code
+substantive: true
+raw_file: 20260417T124437Z_5cd6df5_prompt_io.raw.md
+---
+
+## Prompt
+
+Continuing the Phase B work from the design sessions
+logged in `20260417T034918Z_9703210_prompt_io.md` and
+the Phase A impl in `20260417T035800Z_61a73ba_...`.
+
+User direction at this step:
+- "ok now let's continue with the subints part of the
+  plan ya?" → kick off Phase B.
+- After I proposed a 4-sub-phase breakdown (B.1
+  scaffolding, B.2 min-viable spawn, B.3 lifecycle,
+  B.4 test enablement), user: "ok can we continue
+  with B" → approving the phasing and design
+  defaults (thread-per-subint, UDS default,
+  code-string bootstrap via `interp.exec()`).
+- User switched the worktree to
+  `subint_spawner_backend` (instead of opening a
+  fresh worktree as I'd proposed): "i already
+  switched branches in the wkt, you should work off
+  `subint_spawner_backend` now".
+
+Scope of this turn: **B.1 scaffolding only** —
+wire the `'subint'` name through the spawn-method
+registry + harness, add a feature-detected stub
+module, bump pyproject. No actual spawn flow.
+
+## Response summary
+
+Produced the Phase B.1 scaffolding patch, landed as
+commit `5cd6df58` (subject: `Add `'subint'` spawn
+backend scaffold (#379)`). Four files changed,
++124 / -2 lines.
+
+Key pieces (all generated by claude, reviewed by
+human before commit):
+- `tractor/spawn/_subint.py` — **new**; feature-
+  detects `concurrent.interpreters`; `subint_proc()`
+  stub raises `RuntimeError` on py<3.14 or
+  `NotImplementedError` with issue-#379 URL on
+  py≥3.14. Signature mirrors `trio_proc`/`mp_proc`
+  so B.2 can drop the impl in without touching
+  `_methods`.
+- `tractor/spawn/_spawn.py` — adds `'subint'` to
+  `SpawnMethodKey`, grows a `case 'subint'` arm in
+  `try_set_start_method()` with feature-gate, re-
+  imports `sys` for the gate-error msg, adds late
+  `from ._subint import subint_proc` import and
+  `_methods` entry.
+- `tractor/_testing/pytest.py` — converts the
+  gate-error into `pytest.UsageError` via a
+  `try/except` around `try_set_start_method()` so
+  `--spawn-backend=subint` on py<3.14 prints a
+  clean banner instead of a traceback.
+- `pyproject.toml` — pin `requires-python` `<3.14`
+  → `<3.15`, add `3.14` trove classifier.
+
+## Files changed
+
+See `git diff 5cd6df5~1..5cd6df5 --stat`:
+
+```
+ pyproject.toml             |   3 +-
+ tractor/_testing/pytest.py |   8 +++-
+ tractor/spawn/_spawn.py    |  15 ++++++
+ tractor/spawn/_subint.py   | 100 +++++++++++++++++++++++++
+ 4 files changed, 124 insertions(+), 2 deletions(-)
+```
+
+Validation on py3.13:
+- registry/gate probe — OK; `_has_subints = False`,
+  `subint_proc()` raises `RuntimeError` as expected.
+- `try_set_start_method('subint')` — raises cleanly.
+- `pytest --spawn-backend=subint` — exits with
+  `pytest.UsageError` banner.
+- Spawn-relevant test subset — 69 passed, 1 skipped.
+- Full suite on py3.13 deferred (scaffolding is
+  no-op on <3.14).
+- Full-suite on py3.14 pending user setting up the
+  venv.
+
+## Human edits
+
+None — committed as generated by claude. `uv.lock`
+was intentionally left unstaged by the user (pin-
+bump regenerated cp314 wheel entries but they were
+deferred to a follow-up).

ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.raw.md

@@ -0,0 +1,168 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-04-17T12:44:37Z
+git_ref: 5cd6df5
+diff_cmd: git diff 5cd6df5~1..5cd6df5
+---
+
+Code generated in this turn was committed verbatim as
+`5cd6df58` ("Add `'subint'` spawn backend scaffold
+(#379)"). Per diff-ref mode, per-file code is captured
+via the pointers below, each followed by a prose
+summary. Non-code output (sanity-check results,
+design rationale) is included verbatim.
+
+## Per-file generated content
+
+### `tractor/spawn/_subint.py` (new, 100 lines)
+
+> `git diff 5cd6df5~1..5cd6df5 -- tractor/spawn/_subint.py`
+
+New scaffolding module for the PEP 734 subinterpreter
+backend. Contents:
+- AGPL header + module docstring (describes backend
+  intent, 3.14+ availability gate, and explicit
+  "SCAFFOLDING STUB" status pointing at issue #379).
+- Top-level `try/except ImportError` wrapping
+  `from concurrent import interpreters as
+  _interpreters` → sets module-global
+  `_has_subints: bool`. This lets the registry stay
+  introspectable on py<3.14 while spawn-time still
+  fails cleanly.
+- `subint_proc()` coroutine with signature matching
+  `trio_proc`/`mp_proc` exactly (same param names,
+  defaults, and `TaskStatus[Portal]` typing) —
+  intentional so Phase B.2 can drop the impl in
+  without touching `_methods` or changing call-site
+  binding.
+- Body raises `RuntimeError` on py<3.14 (with
+  `sys.version` printed) or `NotImplementedError`
+  with issue-#379 URL on py≥3.14.
+
+### `tractor/spawn/_spawn.py` (modified, +15 LOC)
+
+> `git diff 5cd6df5~1..5cd6df5 -- tractor/spawn/_spawn.py`
+
+- `import sys` re-added (pruned during Phase A, now
+  needed again for the py-version string in the
+  `'subint'` gate-error).
+- `SpawnMethodKey = Literal[...]` grows `'subint'` as
+  the 4th member, with inline comment `# py3.14+ via
+  `concurrent.interpreters` (PEP 734)`.
+- `try_set_start_method()` match-block grows a new
+  `case 'subint':` arm that imports
+  `from ._subint import _has_subints` lazily and
+  raises `RuntimeError` with a multi-line gate msg
+  if unavailable.
+- Bottom-of-module late-import section grows
+  `from ._subint import subint_proc` alongside the
+  existing `_trio` / `_mp` imports.
+- `_methods` dict grows `'subint': subint_proc`.
+
+### `tractor/_testing/pytest.py` (modified, +8 LOC)
+
+> `git diff 5cd6df5~1..5cd6df5 -- tractor/_testing/pytest.py`
+
+`pytest_configure()` wraps the
+`try_set_start_method(backend)` call in a
+`try/except RuntimeError` that re-raises as
+`pytest.UsageError(str(err))`. Rationale: the gate
+error on py<3.14 is legitimately a configuration
+problem, not a test failure, so pytest's UsageError
+path (exit code 4) gives a clean single-line banner
+instead of a traceback.
+
+### `pyproject.toml` (modified, +2 / -1)
+
+> `git diff 5cd6df5~1..5cd6df5 -- pyproject.toml`
+
+- `requires-python` pin relaxed `>=3.12, <3.14` →
+  `>=3.12, <3.15` to admit 3.14 as a valid target.
+- Added `"Programming Language :: Python :: 3.14"`
+  to the trove classifiers.
+
+## Non-code output (verbatim)
+
+### Design choices captured in the prior turn
+
+(Previously confirmed by the user on April 17 in this
+session — relevant excerpts captured here for
+provenance since they drove the shape of the impl):
+
+1. **Python pin**: `<3.15` + feature-gate the subint
+   backend at spawn time — user answered option (a)
+   from the design triage.
+2. **Deferred `fork()`-via-subint hack** from issue
+   #379's "Our own thoughts" section.
+3. **Phase B phasing**: user approved the B.1 / B.2 /
+   B.3 / B.4 breakdown — this commit is strictly B.1
+   (scaffolding only, no spawn-flow impl).
+4. **Option (B) worktree strategy**: new worktree
+   branched from `wkt/spawn_modularize`. *(Amended by
+   user at runtime: user switched the existing
+   `spawn_modularize` worktree to the
+   `subint_spawner_backend` branch instead.)*
+
+### Sanity-check output (verbatim terminal excerpts)
+
+Registry / feature-gate verification on py3.13:
+```
+SpawnMethodKey values: ('trio', 'mp_spawn',
+                       'mp_forkserver', 'subint')
+_methods keys: ['trio', 'mp_spawn',
+                'mp_forkserver', 'subint']
+_has_subints: False (py version: (3, 13) )
+[expected] RuntimeError: The 'subint' spawn backend
+requires Python 3.14+ (stdlib
+`concurrent.interpreters`, PEP 734).
+```
+
+`try_set_start_method('subint')` gate on py3.13:
+```
+[expected] RuntimeError: Spawn method 'subint'
+requires Python 3.14+ (stdlib
+`concurrent.interpreters`, PEP 734).
+```
+
+Pytest `--spawn-backend=subint` on py3.13 (the new
+UsageError wrapper kicking in):
+```
+ERROR: Spawn method 'subint' requires Python 3.14+
+(stdlib `concurrent.interpreters`, PEP 734).
+Current runtime: 3.13.11 (main, Dec  5 2025,
+16:06:33) [GCC 15.2.0]
+```
+
+Collection probe: `404 tests collected in 0.18s`
+(no import errors from the new module).
+
+Spawn-relevant test subset (`tests/test_local.py
+test_rpc.py test_spawning.py test_multi_program.py
+tests/discovery/`):
+```
+69 passed, 1 skipped, 10 warnings in 61.38s
+```
+
+Full suite was **not** run on py3.13 for this commit
+— the scaffolding is no-op on <3.14 and full-suite
+validation under py3.14 is pending that venv being
+set up by the user.
+
+### Commit message
+
+Also AI-drafted (via `/commit-msg`, with the prose
+rewrapped through `/home/goodboy/.claude/skills/pr-msg/
+scripts/rewrap.py --width 67`) — the 33-line message
+on commit `5cd6df58` itself. Not reproduced here; see
+`git log -1 5cd6df58`.
+
+### Known follow-ups flagged to user
+
+- **`uv.lock` deferred**: pin-bump regenerated cp314
+  wheel entries in `uv.lock`, but the user chose to
+  not stage `uv.lock` for this commit. Warned
+  explicitly.
+- **Phase B.2 needs py3.14 venv** — running the
+  actual subint impl requires it; user said they'd
+  set it up separately.

ai/prompt-io/claude/20260418T042526Z_26fb820_prompt_io.md

@@ -0,0 +1,117 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subints-phase-b2-destroy-race-fix
+timestamp: 2026-04-18T04:25:26Z
+git_ref: 26fb820
+scope: code
+substantive: true
+raw_file: 20260418T042526Z_26fb820_prompt_io.raw.md
+---
+
+## Prompt
+
+Follow-up to Phase B.2 (`5cd6df58`) after the user
+observed intermittent mid-suite hangs when running
+the tractor test suite under `--spawn-backend=subint`
+on py3.14. The specific sequence of prompts over
+several turns:
+
+1. User pointed at the `test_context_stream_semantics.py`
+   suite as the first thing to make run clean under
+   `--spawn-backend=subint`.
+2. After a series of `timeout`-terminated runs that
+   gave no diagnostic info, user nudged me to stop
+   relying on `timeout` and get actual runtime
+   diagnostics ("the suite hangs indefinitely, so i
+   don't think this `timeout 30` is helping you at
+   all.."). Switched to
+   `faulthandler.dump_traceback_later(...)` and a
+   resource-tracker fixture to rule out leaks.
+3. Captured a stack pinning the hang on
+   `_interpreters.destroy(interp_id)` in the subint
+   teardown finally block.
+4. Proposed dedicated-OS-thread fix. User greenlit.
+5. Implemented + verified on-worktree; user needed
+   to be pointed at the *worktree*'s `./py313` venv
+   because bare `pytest` was picking up the main
+   repo's venv (running un-patched `_subint.py`) and
+   still hanging.
+
+Running theme over the whole exchange: this patch
+only closes the *destroy race*. The user and I also
+traced through the deeper cancellation story — SIGINT
+can't reach subints, legacy-mode shares the GIL,
+portal-cancel dies when the IPC channel is already
+broken — and agreed the next step is a bounded
+hard-kill in `subint_proc`'s teardown plus a
+dedicated cancellation test suite. Those land as
+separate commits.
+
+## Response summary
+
+Produced the `tractor/spawn/_subint.py` patch landed
+as commit `26fb8206` ("Fix subint destroy race via
+dedicated OS thread"). One file, +110/-84 LOC.
+
+Mechanism: swap `trio.to_thread.run_sync(_interpreters
+.exec, ...)` for a plain `threading.Thread(target=...
+, daemon=False)`. The trio thread cache recycles
+workers — so the OS thread that ran `_interpreters
+.exec()` remained alive in the cache holding a
+stale subint tstate, blocking
+`_interpreters.destroy()` in the finally indefinitely.
+A dedicated one-shot thread exits naturally after
+the sync target returns, releasing tstate and
+unblocking destroy.
+
+Coordination across the trio↔thread boundary:
+- `trio.lowlevel.current_trio_token()` captured at
+  `subint_proc` entry
+- driver thread signals `subint_exited.set()` back
+  to parent trio via `trio.from_thread.run_sync(...,
+  trio_token=token)` (synchronous from the thread's
+  POV; the call returns after trio has run `.set()`)
+- `trio.RunFinishedError` swallowed in that path for
+  the process-teardown case where parent trio already
+  exited
+- teardown `finally` off-loads the sync
+  `driver_thread.join()` via `to_thread.run_sync` (a
+  cache thread carries no subint tstate — safe)
+
+## Files changed
+
+See `git diff 26fb820~1..26fb820 --stat`:
+
+```
+ tractor/spawn/_subint.py | 194 +++++++++++++++++++------------
+ 1 file changed, 110 insertions(+), 84 deletions(-)
+```
+
+Validation:
+- `test_parent_cancels[chk_ctx_result_before_exit=True-
+  cancel_method=ctx-child_returns_early=False]`
+  (the specific test that was hanging for the user)
+  — passed in 1.06s.
+- Full `tests/test_context_stream_semantics.py` under
+  subint — 61 passed in 100.35s (clean-cache re-run:
+  100.82s).
+- Trio backend regression subset — 69 passed / 1
+  skipped / 89.19s — no regressions from this change.
+
+## Files changed
+
+Beyond the `_subint.py` patch, the raw log also
+records the cancellation-semantics research that
+spanned this conversation but did not ship as code
+in *this* commit. Preserving it inline under "Non-
+code output" because it directly informs the
+Phase B.3 hard-kill impl that will follow (and any
+upstream CPython bug reports we end up filing).
+
+## Human edits
+
+None — committed as generated. The commit message
+itself was also AI-drafted via `/commit-msg` and
+rewrapped via the project's `rewrap.py --width 67`
+tooling; user landed it without edits.

ai/prompt-io/claude/20260418T042526Z_26fb820_prompt_io.raw.md

@@ -0,0 +1,220 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-04-18T04:25:26Z
+git_ref: 26fb820
+diff_cmd: git diff 26fb820~1..26fb820
+---
+
+Code generated in this turn was committed as `26fb8206`
+("Fix subint destroy race via dedicated OS thread").
+Per diff-ref mode, per-file code is captured via the
+pointer below; the rest is verbatim non-code output
+— test-session diagnostics, CPython-internals
+reasoning, and the design synthesis that's worth
+keeping alongside the code since it informs the
+still-pending Phase B.3 hard-kill work.
+
+## Per-file generated content
+
+### `tractor/spawn/_subint.py` (modified, +110/-84 LOC)
+
+> `git diff 26fb820~1..26fb820 -- tractor/spawn/_subint.py`
+
+Rewrites the subint driver-thread strategy:
+- replaces `trio.to_thread.run_sync(_interpreters.exec, ...)`
+  with a plain `threading.Thread(target=_subint_target,
+  daemon=False)` so the OS thread truly exits after
+  `_interpreters.exec()` returns
+- captures a `trio.lowlevel.current_trio_token()` at
+  `subint_proc` entry; the driver thread signals
+  completion back via `trio.from_thread.run_sync(
+  subint_exited.set, trio_token=...)`
+- swallows `trio.RunFinishedError` in the signal path
+  for the case where the parent trio loop has already
+  exited (process teardown)
+- in the teardown `finally` off-loads the sync
+  `driver_thread.join()` call to `trio.to_thread.run_sync`
+  (a cache thread w/ no subint tstate — so no cache
+  conflict) to wait for the driver thread to fully
+  exit before calling `_interpreters.destroy()`
+
+## Non-code output (verbatim) — the CPython-internals research
+
+### What went wrong before this commit
+
+Under `--spawn-backend=subint` on py3.14, most single
+tests passed but longer runs hung intermittently. The
+position of the hang moved between runs (test #22 on
+one run, test #53 on another) suggesting a timing-
+dependent race rather than a deterministic bug.
+
+`faulthandler.dump_traceback_later()` eventually
+caught a stack with the main thread blocked in
+`_interpreters.destroy(interp_id)` at `_subint.py:293`.
+Only 2 threads were alive:
+- main thread waiting in `_interpreters.destroy()`
+- one idle trio thread-cache worker in
+  `trio._core._thread_cache._work`
+
+No subint was still running (`_interpreters.list_all()`
+showed only the main interp). A resource-tracker
+pytest fixture confirmed threads/subints did NOT
+accumulate across tests — this was not a leak but a
+specific "destroy blocks on cached thread w/ stale
+tstate" race.
+
+### Why the race exists
+
+`trio.to_thread.run_sync` uses a thread *cache* to
+avoid OS-thread creation overhead. When the sync
+callable returns, the OS thread is NOT terminated —
+it's parked in `_thread_cache._work` waiting for the
+next job. CPython's subinterpreter implementation
+attaches a **tstate** (thread-state object) to each
+OS thread that ever entered a subint via
+`_interpreters.exec()`. That tstate is released
+lazily — either when the thread picks up a new job
+(which re-attaches a new tstate, evicting the old
+one) or when the thread truly exits.
+
+`_interpreters.destroy(interp_id)` waits for *all*
+tstates associated w/ that subint to be released
+before it can proceed. If the cached worker is idle
+holding the stale tstate, destroy blocks indefinitely.
+Whether the race manifests depends on timing — if
+the cached thread happens to pick up another job
+quickly, destroy unblocks; if it sits idle, we hang.
+
+### Why a dedicated `threading.Thread` fixes it
+
+A plain `threading.Thread(target=_subint_target,
+daemon=False)` runs its target once and exits. When
+the target returns, OS-thread teardown (`_bootstrap_inner`
+→ `_bootstrap`) fires and CPython releases the
+tstate for that thread. `_interpreters.destroy()`
+then has no blocker.
+
+### Diagnostic tactics that actually helped
+
+1. `faulthandler.dump_traceback_later(n, repeat=False,
+   file=open(path, 'w'))` for captured stack dumps on
+   hang. Critically, pipe to a `file=` not stderr —
+   pytest captures stderr weirdly and the dump is
+   easy to miss.
+2. A resource-tracker autouse fixture printing
+   per-test `threading.active_count()` +
+   `len(_interpreters.list_all())` deltas → ruled out
+   leak-accumulation theories quickly.
+3. Running the hanging test *solo* vs in-suite —
+   when solo passes but in-suite hangs, you know
+   it's a cross-test state-transfer bug rather than
+   a test-internal bug.
+
+### Design synthesis — SIGINT + subints + SC
+
+The user and I walked through the cancellation
+semantics of PEP 684/734 subinterpreters in detail.
+Key findings we want to preserve:
+
+**Signal delivery in subints (stdlib limitation).**
+CPython's signal machinery only delivers signals
+(SIGINT included) to the *main thread of the main
+interpreter*. Subints cannot install signal handlers
+that will ever fire. This is an intentional design
+choice in PEP 684 and not expected to change. For
+tractor's subint actors, this means:
+
+- Ctrl-C never reaches a subint directly.
+- `trio.run()` running on a worker thread (as we do
+  for subints) already skips SIGINT handler install
+  because `signal.signal()` raises on non-main
+  threads.
+- The only cancellation surface into a subint is
+  our IPC `Portal.cancel_actor()`.
+
+**Legacy-mode subints share the main GIL** (which
+our impl uses since `msgspec` lacks PEP 684 support
+per `jcrist/msgspec#563`). This means a stuck subint
+thread can starve the parent's trio loop during
+cancellation — the parent can't even *start* its
+teardown handling until the subint yields the GIL.
+
+**Failure modes identified for Phase B.3 audit:**
+
+1. Portal cancel lands cleanly → subint unwinds →
+   thread exits → destroy succeeds. (Happy path.)
+2. IPC channel is already broken when we try to
+   send cancel (e.g., `test_ipc_channel_break_*`)
+   → cancel raises `BrokenResourceError` → subint
+   keeps running unaware → parent hangs waiting for
+   `subint_exited`. This is what breaks
+   `test_advanced_faults.py` under subint.
+3. Subint is stuck in non-checkpointing Python code
+   → portal-cancel msg queued but never processed.
+4. Subint is in a shielded cancel scope when cancel
+   arrives → delay until shield exits.
+
+**Current teardown has a shield-bug too:**
+`trio.CancelScope(shield=True)` wrapping the `finally`
+block absorbs Ctrl-C, so even when the user tries
+to break out they can't. This is the reason
+`test_ipc_channel_break_during_stream[break_parent-...
+no_msgstream_aclose]` locks up unkillable.
+
+**B.3 hard-kill fix plan (next commit):**
+
+1. Bound `driver_thread.join()` with
+   `trio.move_on_after(HARD_KILL_TIMEOUT)`.
+2. If it times out, log a warning naming the
+   `interp_id` and switch the driver thread to
+   `daemon=True` mode (not actually possible after
+   start — so instead create as daemon=True upfront
+   and accept the tradeoff of proc-exit not waiting
+   for a stuck subint).
+3. Best-effort `_interpreters.destroy()`; catch the
+   `InterpreterError` if the subint is still running.
+4. Document that the leak is real and the only
+   escape hatch we have without upstream cooperation.
+
+**Test plan for Phase B.3:**
+
+New `tests/test_subint_cancellation.py` covering:
+- SIGINT at spawn
+- SIGINT mid-portal-RPC
+- SIGINT during shielded section in subint
+- Dead-channel cancel (mirror of `test_ipc_channel_
+  break_during_stream` minimized)
+- Non-checkpointing subint (tight `while True` in
+  user code)
+- Per-test `pytest-timeout`-style bounds so the
+  tests visibly fail instead of wedging the runner
+
+### Sanity-check output (verbatim terminal excerpts)
+
+Post-fix single-test validation:
+```
+1 passed, 1 warning in 1.06s
+```
+(same test that was hanging pre-fix:
+`test_parent_cancels[...cancel_method=ctx-...False]`)
+
+Full `tests/test_context_stream_semantics.py`
+under subint:
+```
+61 passed, 1 warning in 100.35s (0:01:40)
+```
+and a clean-cache re-run:
+```
+61 passed, 1 warning in 100.82s (0:01:40)
+```
+
+No regressions on trio backend (same subset):
+```
+69 passed, 1 skipped, 3 warnings in 89.19s
+```
+
+### Commit msg
+
+Also AI-drafted via `/commit-msg` + `rewrap.py
+--width 67`. See `git log -1 26fb820`.

ai/prompt-io/claude/20260420T192739Z_5e8cd8b2_prompt_io.md

@@ -0,0 +1,111 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subint-phase-b-hang-classification
+timestamp: 2026-04-20T19:27:39Z
+git_ref: HEAD (pre-commit; on branch `subint_spawner_backend`)
+scope: docs
+substantive: true
+raw_file: 20260420T192739Z_5e8cd8b2_prompt_io.raw.md
+---
+
+## Prompt
+
+Collab between user (`goodboy`) and `claude` to
+disambiguate two distinct hang modes hit during
+Phase B subint-spawn-backend bringup (issue #379).
+The user ran the failing suites, captured `strace`
+evidence on hung pytest pids, and set the framing:
+
+> "These cannot be the same bug. Different fix
+> paths. Write them up separately or we'll keep
+> conflating them."
+
+Follow-on asks:
+- Cross-link each doc from its reproducer test so
+  a future triager lands on the analysis without
+  needing to grep `ai/conc-anal/`.
+- On `test_stale_entry_is_deleted`: arm
+  `dump_on_hang(seconds=20, path=...)` so a
+  regression captures a stack dump (pytest stderr
+  capture otherwise eats `faulthandler` output).
+- Keep `test_stale_entry_is_deleted` un-`skip`ped
+  so the dump file remains inspectable.
+
+Defer the actual fix for the class-2 hang
+(Ctrl-C-able, tractor-side bug) to a later commit
+per the incremental-commits preference.
+
+## Response summary
+
+Wrote two new `ai/conc-anal/` classification docs
+and cross-linked them from the reproducer tests:
+
+1. **`subint_sigint_starvation_issue.md`** —
+   class 1: abandoned-legacy-subint thread +
+   shared GIL starves main trio loop →
+   signal-wakeup-fd pipe fills → SIGINT silently
+   dropped (`write() = EAGAIN`). Pytest process
+   un-Ctrl-C-able. Structurally a CPython limit;
+   blocked on `msgspec` PEP 684 support
+   (jcrist/msgspec#563). Reproducer:
+   `test_stale_entry_is_deleted[subint]`.
+
+2. **`subint_cancel_delivery_hang_issue.md`** —
+   class 2: parent-side trio task parks on an
+   orphaned IPC channel after subint teardown;
+   no clean EOF delivered to waiting receiver.
+   Ctrl-C-able (main trio loop iterating fine).
+   OUR bug to fix. Candidate fix: explicit
+   parent-side channel abort in `subint_proc`'s
+   hard-kill teardown. Reproducer:
+   `test_subint_non_checkpointing_child`.
+
+Test-side cross-links:
+- `tests/discovery/test_registrar.py`:
+  `test_stale_entry_is_deleted` → `trio.run(main)`
+  wrapped in `dump_on_hang(seconds=20,
+  path=<per-method-tmp>)`; long inline comment
+  summarizes `strace` evidence + root-cause chain
+  and points at both docs.
+- `tests/test_subint_cancellation.py`:
+  `test_subint_non_checkpointing_child` docstring
+  extended with "KNOWN ISSUE (Ctrl-C-able hang)"
+  section pointing at the class-2 doc + noting
+  the class-1 doc is NOT what this test hits.
+
+## Files changed
+
+- `ai/conc-anal/subint_sigint_starvation_issue.md`
+  — new, 205 LOC
+- `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
+  — new, 161 LOC
+- `tests/discovery/test_registrar.py` — +52/-1
+  (arm `dump_on_hang`, inline-comment cross-link)
+- `tests/test_subint_cancellation.py` — +26
+  (docstring "KNOWN ISSUE" block)
+
+## Human edits
+
+Substantive collab — prose was jointly iterated:
+
+- User framed the two-doc split, set the
+  classification criteria (Ctrl-C-able vs not),
+  and provided the `strace` evidence.
+- User decided to keep `test_stale_entry_is_deleted`
+  un-`skip`ped (my initial suggestion was
+  `pytestmark.skipif(spawn_backend=='subint')`).
+- User chose the candidate fix ordering for
+  class 2 and marked "explicit parent-side channel
+  abort" as the surgical preferred fix.
+- User picked the file naming convention
+  (`subint_<hang-shape>_issue.md`) over my initial
+  `hang_class_{1,2}.md`.
+- Assistant drafted the prose, aggregated prior-
+  session root-cause findings from Phase B.2/B.3
+  bringup, and wrote the test-side cross-linking
+  comments.
+
+No further mechanical edits expected before
+commit; user may still rewrap via
+`scripts/rewrap.py` if preferred.

ai/prompt-io/claude/20260420T192739Z_5e8cd8b2_prompt_io.raw.md

@@ -0,0 +1,198 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-04-20T19:27:39Z
+git_ref: HEAD (pre-commit; will land on branch `subint_spawner_backend`)
+diff_cmd: git diff HEAD~1..HEAD
+---
+
+Collab between `goodboy` (user) and `claude` (this
+assistant) spanning multiple test-run iterations on
+branch `subint_spawner_backend`. The user ran the
+failing suites, captured `strace` evidence on the
+hung pytest pids, and set the direction ("these are
+two different hangs — write them up separately so
+we don't re-confuse ourselves later"). The assistant
+aggregated prior-session findings (Phase B.2/B.3
+bringup) into two classification docs + test-side
+cross-links. All prose was jointly iterated; the
+user had final say on framing and decided which
+candidate fix directions to list.
+
+## Per-file generated content
+
+### `ai/conc-anal/subint_sigint_starvation_issue.md` (new, 205 LOC)
+
+> `git diff HEAD~1..HEAD -- ai/conc-anal/subint_sigint_starvation_issue.md`
+
+Writes up the "abandoned-legacy-subint thread wedges
+the parent trio loop" class. Key sections:
+
+- **Symptom** — `test_stale_entry_is_deleted[subint]`
+  hangs indefinitely AND is un-Ctrl-C-able.
+- **Evidence** — annotated `strace` excerpt showing
+  SIGINT delivered to pytest, C-level signal handler
+  tries to write to the signal-wakeup-fd pipe, gets
+  `write() = -1 EAGAIN (Resource temporarily
+  unavailable)`. Pipe is full because main trio loop
+  isn't iterating often enough to drain it.
+- **Root-cause chain** — our hard-kill abandons the
+  `daemon=True` driver OS thread after
+  `_HARD_KILL_TIMEOUT`; the subint *inside* that
+  thread is still running `trio.run()`;
+  `_interpreters.destroy()` cannot force-stop a
+  running subint (raises `InterpreterError`); legacy
+  subints share the main GIL → abandoned subint
+  starves main trio loop → wakeup-fd fills → SIGINT
+  silently dropped.
+- **Why it's structurally a CPython limit** — no
+  public force-destroy primitive for a running
+  subint; the only escape is per-interpreter GIL
+  isolation, gated on msgspec PEP 684 adoption
+  (jcrist/msgspec#563).
+- **Current escape hatch** — harness-side SIGINT
+  loop in the `daemon` fixture teardown that kills
+  the bg registrar subproc, eventually unblocking
+  a parent-side recv enough for the main loop to
+  drain the wakeup pipe.
+
+### `ai/conc-anal/subint_cancel_delivery_hang_issue.md` (new, 161 LOC)
+
+> `git diff HEAD~1..HEAD -- ai/conc-anal/subint_cancel_delivery_hang_issue.md`
+
+Writes up the *sibling* hang class — same subint
+backend, distinct root cause:
+
+- **TL;DR** — Ctrl-C-able, so NOT the SIGINT-
+  starvation class; main trio loop iterates fine;
+  ours to fix.
+- **Symptom** — `test_subint_non_checkpointing_child`
+  hangs past the expected `_HARD_KILL_TIMEOUT`
+  budget even after the subint is torn down.
+- **Diagnosis** — a parent-side trio task (likely
+  a `chan.recv()` in `process_messages`) parks on
+  an orphaned IPC channel; channel was torn down
+  without emitting a clean EOF /
+  `BrokenResourceError` to the waiting receiver.
+- **Candidate fix directions** — listed in rough
+  order of preference:
+  1. Explicit parent-side channel abort in
+     `subint_proc`'s hard-kill teardown (surgical;
+     most likely).
+  2. Audit `process_messages` to add a timeout or
+     cancel-scope protection that catches the
+     orphaned-recv state.
+  3. Wrap subint IPC channel construction in a
+     sentinel that can force-close from the parent
+     side regardless of subint liveness.
+
+### `tests/discovery/test_registrar.py` (modified, +52/-1 LOC)
+
+> `git diff HEAD~1..HEAD -- tests/discovery/test_registrar.py`
+
+Wraps the `trio.run(main)` call at the bottom of
+`test_stale_entry_is_deleted` in
+`dump_on_hang(seconds=20, path=<per-method-tmp>)`.
+Adds a long inline comment that:
+- Enumerates variant-by-variant status
+  (`[trio]`/`[mp_*]` = clean; `[subint]` = hangs
+  + un-Ctrl-C-able)
+- Summarizes the `strace` evidence and root-cause
+  chain inline (so a future reader hitting this
+  test doesn't need to cross-ref the doc to
+  understand the hang shape)
+- Points at
+  `ai/conc-anal/subint_sigint_starvation_issue.md`
+  for full analysis
+- Cross-links to the *sibling*
+  `subint_cancel_delivery_hang_issue.md` so
+  readers can tell the two classes apart
+- Explains why it's kept un-`skip`ped: the dump
+  file is useful if the hang ever returns after
+  a refactor. pytest stderr capture would
+  otherwise eat `faulthandler` output, hence the
+  file path.
+
+### `tests/test_subint_cancellation.py` (modified, +26 LOC)
+
+> `git diff HEAD~1..HEAD -- tests/test_subint_cancellation.py`
+
+Extends the docstring of
+`test_subint_non_checkpointing_child` with a
+"KNOWN ISSUE (Ctrl-C-able hang)" block:
+- Describes the current hang: parent-side orphaned
+  IPC recv after hard-kill; distinct from the
+  SIGINT-starvation sibling class.
+- Cites `strace` distinguishing signal: wakeup-fd
+  `write() = 1` (not `EAGAIN`) — i.e. main loop
+  iterating.
+- Points at
+  `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
+  for full analysis + candidate fix directions.
+- Clarifies that the *other* sibling doc
+  (SIGINT-starvation) is NOT what this test hits.
+
+## Non-code output
+
+### Classification reasoning (why two docs, not one)
+
+The user and I converged on the two-doc split after
+running the suites and noticing two *qualitatively
+different* hang symptoms:
+
+1. `test_stale_entry_is_deleted[subint]` — pytest
+   process un-Ctrl-C-able. Ctrl-C at the terminal
+   does nothing. Must kill-9 from another shell.
+2. `test_subint_non_checkpointing_child` — pytest
+   process Ctrl-C-able. One Ctrl-C at the prompt
+   unblocks cleanly and the test reports a hang
+   via pytest-timeout.
+
+From the user: "These cannot be the same bug.
+Different fix paths. Write them up separately or
+we'll keep conflating them."
+
+`strace` on the `[subint]` hang gave the decisive
+signal for the first class:
+
+```
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(5, "\2", 1) = -1 EAGAIN (Resource temporarily unavailable)
+```
+
+fd 5 is Python's signal-wakeup-fd pipe. `EAGAIN`
+on a `write()` of 1 byte to a pipe means the pipe
+buffer is full → reader side (main Python thread
+inside `trio.run()`) isn't consuming. That's the
+GIL-hostage signature.
+
+The second class's `strace` showed `write(5, "\2",
+1) = 1` — clean drain — so the main trio loop was
+iterating and the hang had to be on the application
+side of things, not the kernel-↔-Python signal
+boundary.
+
+### Why the candidate fix for class 2 is "explicit parent-side channel abort"
+
+The second hang class has the trio loop alive. A
+parked `chan.recv()` that will never get bytes is
+fundamentally a tractor-side resource-lifetime bug
+— the IPC channel was torn down (subint destroyed)
+but no one explicitly raised
+`BrokenResourceError` at the parent-side receiver.
+The `subint_proc` hard-kill path is the natural
+place to add that notification, because it already
+knows the subint is unreachable at that point.
+
+Alternative fix paths (blanket timeouts on
+`process_messages`, sentinel-wrapped channels) are
+less surgical and risk masking unrelated bugs —
+hence the preference ordering in the doc.
+
+### Why we're not just patching the code now
+
+The user explicitly deferred the fix to a later
+commit: "Document both classes now, land the fix
+for class 2 separately so the diff reviews clean."
+This matches the incremental-commits preference
+from memory.

ai/prompt-io/claude/20260422T200723Z_797f57c_prompt_io.md

@@ -0,0 +1,155 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subints-phase-b-hardening-and-fork-block
+timestamp: 2026-04-22T20:07:23Z
+git_ref: 797f57c
+scope: code
+substantive: true
+raw_file: 20260422T200723Z_797f57c_prompt_io.raw.md
+---
+
+## Prompt
+
+Session-spanning work on the Phase B `subint` spawn-backend.
+Three distinct sub-phases in one log:
+
+1. **Py3.13 gate tightening** — diagnose a reproducible hang
+   of subint spawn flow under py3.13 (works on py3.14), trace
+   to a private `_interpreters` module vintage issue, tighten
+   our feature gate from "`_interpreters` present" to "public
+   `concurrent.interpreters` present" (i.e. py3.14+).
+2. **Test-harness hardening** — add `pytest-timeout` dep, put
+   `@pytest.mark.timeout(30, method='thread')` on the
+   three known-hanging subint tests cataloged in
+   `ai/conc-anal/subint_sigint_starvation_issue.md`. Separately,
+   code-review the user's in-flight `skipon_spawn_backend`
+   marker implementation; find four bugs; refactor to use
+   `item.iter_markers()`.
+3. **`subint_fork` prototype → CPython-block finding** — draft
+   a WIP `subint_fork_proc` backend using a sub-interpreter as
+   a launchpad for `os.fork()` (to sidestep trio#1614). User
+   tests on py3.14, hits
+   `Fatal Python error: _PyInterpreterState_DeleteExceptMain:
+   not main interpreter`. Walk CPython sources (local clone at
+   `~/repos/cpython/`) to pinpoint the refusal
+   (`Modules/posixmodule.c:728` → `Python/pystate.c:1040`).
+   Revert implementation to a `NotImplementedError` stub in a
+   new `_subint_fork.py` submodule, document the finding in a
+   third `conc-anal/` doc with an upstream-report draft for
+   the CPython issue tracker. Finally, discuss user's proposed
+   workaround architecture (main-interp worker-thread
+   forkserver) and draft a standalone smoke-test script for
+   feasibility validation.
+
+## Response summary
+
+All three sub-phases landed concrete artifacts:
+
+**Sub-phase 1** — `_subint.py` + `_spawn.py` gates + error
+messages updated to require py3.14+ via the public
+`concurrent.interpreters` module presence check. Module
+docstring revised to explain the empirical reason
+(py3.13's private `_interpreters` vintage wedges under
+multi-trio-task usage even though minimal standalone
+reproducers work fine there). Test-module
+`pytest.importorskip` likewise switched.
+
+**Sub-phase 2** — `pytest-timeout>=2.3` added to `testing`
+dep group. `@pytest.mark.timeout(30, method='thread')`
+applied on:
+- `tests/discovery/test_registrar.py::test_stale_entry_is_deleted`
+- `tests/test_cancellation.py::test_cancel_while_childs_child_in_sync_sleep`
+- `tests/test_cancellation.py::test_multierror_fast_nursery`
+- `tests/test_subint_cancellation.py::test_subint_non_checkpointing_child`
+
+`method='thread'` documented inline as load-bearing — the
+GIL-starvation path that drops `SIGINT` would equally drop
+`SIGALRM`, so only a watchdog-thread timeout can reliably
+escape.
+
+`skipon_spawn_backend` plugin refactored into a single
+`iter_markers`-driven loop in `pytest_collection_modifyitems`
+(~30 LOC replacing ~30 LOC of nested conditionals). Four
+bugs dissolved: wrong `.get()` key, module-level `pytestmark`
+suppressing per-test marks, unhandled `pytestmark = [list]`
+form, `pytest.Makr` typo. Marker help text updated to
+document the variadic backend-list + `reason=` kwarg
+surface.
+
+**Sub-phase 3** — Prototype drafted (then reverted):
+
+- `tractor/spawn/_subint_fork.py` — new dedicated submodule
+  housing the `subint_fork_proc` stub. Module docstring +
+  fn docstring explain the attempt, the CPython-level
+  block, and the reason for keeping the stub in-tree
+  (documentation of the attempt + starting point if CPython
+  ever lifts the restriction).
+- `tractor/spawn/_spawn.py` — `'subint_fork'` registered as a
+  `SpawnMethodKey` literal + in `_methods`, so
+  `--spawn-backend=subint_fork` routes to a clean
+  `NotImplementedError` pointing at the analysis doc rather
+  than an "invalid backend" error.
+- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md` —
+  third sibling conc-anal doc. Full annotated CPython
+  source walkthrough from user-visible
+  `Fatal Python error` → `Modules/posixmodule.c:728
+  PyOS_AfterFork_Child()` → `Python/pystate.c:1040
+  _PyInterpreterState_DeleteExceptMain()` gate. Includes a
+  copy-paste-ready upstream-report draft for the CPython
+  issue tracker with a two-tier ask (ideally "make it work",
+  minimally "cleaner error than `Fatal Python error`
+  aborting the child").
+- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py` —
+  standalone zero-tractor-import CPython-level smoke test
+  for the user's proposed workaround architecture
+  (forkserver on a main-interp worker thread). Four
+  argparse-driven scenarios: `control_subint_thread_fork`
+  (reproduces the known-broken case as a test-harness
+  sanity),  `main_thread_fork` (baseline), `worker_thread_fork`
+  (architectural assertion), `full_architecture`
+  (end-to-end trio-in-subint in forked child). User will
+  run on py3.14 next.
+
+## Files changed
+
+See `git log 26fb820..HEAD --stat` for the canonical list.
+New files this session:
+- `tractor/spawn/_subint_fork.py`
+- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
+
+Modified (diff pointers in raw log):
+- `tractor/spawn/_subint.py` (py3.14 gate)
+- `tractor/spawn/_spawn.py` (`subint_fork` registration)
+- `tractor/_testing/pytest.py` (`skipon_spawn_backend` refactor)
+- `pyproject.toml` (`pytest-timeout` dep)
+- `tests/discovery/test_registrar.py`,
+  `tests/test_cancellation.py`,
+  `tests/test_subint_cancellation.py` (timeout marks,
+  cross-refs to conc-anal docs)
+
+## Human edits
+
+Several back-and-forth iterations with user-driven
+adjustments during the session:
+
+- User corrected my initial mis-classification of
+  `test_cancel_while_childs_child_in_sync_sleep[subint-False]`
+  as Ctrl-C-able — second strace showed `EAGAIN`, putting
+  it squarely in class A (GIL-starvation). Re-analysis
+  preserved in the raw log.
+- User independently fixed the `.get(reason)` → `.get('reason', reason)`
+  bug in the marker plugin before my review; preserved their
+  fix.
+- User suggested moving the `subint_fork_proc` stub from
+  the bottom of `_subint.py` into its own
+  `_subint_fork.py` submodule — applied.
+- User asked to keep the forkserver-architecture
+  discussion as background for the smoke-test rather than
+  committing to a tractor-side refactor until the smoke
+  test validates the CPython-level assumptions.
+
+Commit messages in this range (b025c982 … 797f57c) were
+drafted via `/commit-msg` + `rewrap.py --width 67`; user
+landed them with the usual review.

ai/prompt-io/claude/20260422T200723Z_797f57c_prompt_io.raw.md

@@ -0,0 +1,343 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-04-22T20:07:23Z
+git_ref: 797f57c
+diff_cmd: git log 26fb820..HEAD  # all session commits since the destroy-race fix log
+---
+
+Session-spanning conversation covering the Phase B hardening
+of the `subint` spawn-backend and an investigation into a
+proposed `subint_fork` follow-up which turned out to be
+blocked at the CPython level. This log is a narrative capture
+of the substantive turns (not every message) and references
+the concrete code + docs the session produced. Per diff-ref
+mode the actual code diffs are pointed at via `git log` on
+each ref rather than duplicated inline.
+
+## Narrative of the substantive turns
+
+### Py3.13 hang / gate tightening
+
+Diagnosed a reproducible hang of the `subint` backend under
+py3.13 (test_spawning tests wedge after root-actor bringup).
+Root cause: py3.13's vintage of the private `_interpreters` C
+module has a latent thread/subint-interaction issue that
+`_interpreters.exec()` silently fails to progress under
+tractor's multi-trio usage pattern — even though a minimal
+standalone `threading.Thread` + `_interpreters.exec()`
+reproducer works fine on the same Python. Empirically
+py3.14 fixes it.
+
+Fix (from this session): tighten the `_has_subints` gate in
+`tractor.spawn._subint` from "private module importable" to
+"public `concurrent.interpreters` present" — which is 3.14+
+only. This leaves `subint_proc()` unchanged in behavior (we
+still call the *private* `_interpreters.create('legacy')`
+etc. under the hood) but refuses to engage on 3.13.
+
+Also tightened the matching gate in
+`tractor.spawn._spawn.try_set_start_method('subint')` and
+rev'd the corresponding error messages from "3.13+" to
+"3.14+" with a sentence explaining why. Test-module
+`pytest.importorskip` switched from `_interpreters` →
+`concurrent.interpreters` to match.
+
+### `pytest-timeout` dep + `skipon_spawn_backend` marker plumbing
+
+Added `pytest-timeout>=2.3` to the `testing` dep group with
+an inline comment pointing at the `ai/conc-anal/*.md` docs.
+Applied `@pytest.mark.timeout(30, method='thread')` (the
+`method='thread'` is load-bearing — `signal`-method
+`SIGALRM` suffers the same GIL-starvation path that drops
+`SIGINT` in the class-A hang pattern) to the three known-
+hanging subint tests cataloged in
+`subint_sigint_starvation_issue.md`.
+
+Separately code-reviewed the user's newly-staged
+`skipon_spawn_backend` pytest marker implementation in
+`tractor/_testing/pytest.py`. Found four bugs:
+
+1. `modmark.kwargs.get(reason)` called `.get()` with the
+   *variable* `reason` as the dict key instead of the string
+   `'reason'` — user-supplied `reason=` was never picked up.
+   (User had already fixed this locally via `.get('reason',
+   reason)` by the time my review happened — preserved that
+   fix.)
+2. The module-level `pytestmark` branch suppressed per-test
+   marker handling (the `else:` was an `else:` rather than
+   independent iteration).
+3. `mod_pytestmark.mark` assumed a single
+   `MarkDecorator` — broke on the valid-pytest `pytestmark =
+   [mark, mark]` list form.
+4. Typo: `pytest.Makr` → `pytest.Mark`.
+
+Refactored the hook to use `item.iter_markers(name=...)`
+which walks function + class + module scopes uniformly and
+handles both `pytestmark` forms natively. ~30 LOC replaced
+the original ~30 LOC of nested conditionals, all four bugs
+dissolved. Also updated the marker help string to reflect
+the variadic `*start_methods` + `reason=` surface.
+
+### `subint_fork_proc` prototype attempt
+
+User's hypothesis: the known trio+`fork()` issues
+(python-trio/trio#1614) could be sidestepped by using a
+sub-interpreter purely as a launchpad — `os.fork()` from a
+subint that has never imported trio → child is in a
+trio-free context. In the child `execv()` back into
+`python -m tractor._child` and the downstream handshake
+matches `trio_proc()` identically.
+
+Drafted the prototype at `tractor/spawn/_subint.py`'s bottom
+(originally — later moved to its own submod, see below):
+launchpad-subint creation, bootstrap code-string with
+`os.fork()` + `execv()`, driver-thread orchestration,
+parent-side `ipc_server.wait_for_peer()` dance. Registered
+`'subint_fork'` as a new `SpawnMethodKey` literal, added
+`case 'subint' | 'subint_fork':` feature-gate arm in
+`try_set_start_method()`, added entry in `_methods` dict.
+
+### CPython-level block discovered
+
+User tested on py3.14 and saw:
+
+```
+Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
+Python runtime state: initialized
+
+Current thread 0x00007f6b71a456c0 [subint-fork-lau] (most recent call first):
+  File "<script>", line 2 in <module>
+<script>:2: DeprecationWarning: This process (pid=802985) is multi-threaded, use of fork() may lead to deadlocks in the child.
+```
+
+Walked CPython sources (local clone at `~/repos/cpython/`):
+
+- **`Modules/posixmodule.c:728` `PyOS_AfterFork_Child()`** —
+  post-fork child-side cleanup. Calls
+  `_PyInterpreterState_DeleteExceptMain(runtime)` with
+  `goto fatal_error` on non-zero status. Has the
+  `// Ideally we could guarantee tstate is running main.`
+  self-acknowledging-fragile comment directly above.
+
+- **`Python/pystate.c:1040`
+  `_PyInterpreterState_DeleteExceptMain()`** — the
+  refusal. Hard `PyStatus_ERR("not main interpreter")` gate
+  when `tstate->interp != interpreters->main`. Docstring
+  formally declares the precondition ("If there is a
+  current interpreter state, it *must* be the main
+  interpreter"). `XXX` comments acknowledge further latent
+  issues within.
+
+Definitive answer to "Open Question 1" of the prototype
+docstring: **no, CPython does not support `os.fork()` from
+a non-main sub-interpreter**. Not because the fork syscall
+is blocked (it isn't — the parent returns a valid pid),
+but because the child cannot survive CPython's post-fork
+initialization. This is an enforced invariant, not an
+incidental limitation.
+
+### Revert: move to stub submod + doc the finding
+
+Per user request:
+
+1. Reverted the working `subint_fork_proc` body to a
+   `NotImplementedError` stub, MOVED to its own submod
+   `tractor/spawn/_subint_fork.py` (keeps `_subint.py`
+   focused on the working `subint_proc` backend).
+2. Updated `_spawn.py` to import the stub from the new
+   submod path; kept `'subint_fork'` in `SpawnMethodKey` +
+   `_methods` so `--spawn-backend=subint_fork` routes to a
+   clean `NotImplementedError` with pointer to the analysis
+   doc rather than an "invalid backend" error.
+3. Wrote
+   `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+   with the full annotated CPython walkthrough + an
+   upstream-report draft for the CPython issue tracker.
+   Draft has a two-tier ask: ideally "make it work"
+   (pre-fork tstate-swap hook or `DeleteExceptFor(interp)`
+   variant), minimally "give us a clean `RuntimeError` in
+   the parent instead of a `Fatal Python error` aborting
+   the child silently".
+
+### Design discussion — main-interp-thread forkserver workaround
+
+User proposed: set up a "subint forking server" that fork()s
+on behalf of subint callers. Core insight: the CPython gate
+is on `tstate->interp`, not thread identity, so **any thread
+whose tstate is main-interp** can fork cleanly. A worker
+thread attached to main-interp (never entering a subint)
+satisfies the precondition.
+
+Structurally this is `mp.forkserver` (which tractor already
+has as `mp_forkserver`) but **in-process**: instead of a
+separate Python subproc as the fork server, we'd put the
+forkserver on a thread in the tractor parent process. Pros:
+faster spawn (no IPC marshalling to external server + no
+separate Python startup), inherits already-imported modules
+for free. Cons: less crash isolation (forkserver failure
+takes the whole process).
+
+Required tractor-side refactor: move the root actor's
+`trio.run()` off main-interp-main-thread (so main-thread can
+run the forkserver loop). Nontrivial; approximately the same
+magnitude as "Phase C".
+
+The design would also not fully resolve the class-A
+GIL-starvation issue because child actors' trio still runs
+inside subints (legacy config, msgspec PEP 684 pending).
+Would mitigate SIGINT-starvation specifically if signal
+handling moves to the forkserver thread.
+
+Recommended pre-commitment: a standalone CPython-only smoke
+test validating the four assumptions the arch rests on,
+before any tractor-side work.
+
+### Smoke-test script drafted
+
+Wrote `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`:
+argparse-driven, four scenarios (`control_subint_thread_fork`
+reproducing the known-broken case, `main_thread_fork`
+baseline, `worker_thread_fork` the architectural assertion,
+`full_architecture` end-to-end with trio in a subint in the
+forked child). No `tractor` imports; pure CPython + `_interpreters`
++ `trio`. Bails cleanly on py<3.14. Pass/fail banners per
+scenario.
+
+User will validate on their py3.14 env next.
+
+## Per-code-artifact provenance
+
+### `tractor/spawn/_subint_fork.py` (new submod)
+
+> `git show 797f57c -- tractor/spawn/_subint_fork.py`
+
+NotImplementedError stub for the subint-fork backend. Module
+docstring + fn docstring explain the attempt, the CPython
+block, and why the stub is kept in-tree. No runtime behavior
+beyond raising with a pointer at the conc-anal doc.
+
+### `tractor/spawn/_spawn.py` (modified)
+
+> `git log 26fb820..HEAD -- tractor/spawn/_spawn.py`
+
+- Added `'subint_fork'` to `SpawnMethodKey` literal with a
+  block comment explaining the CPython-level block.
+- Generalized the `case 'subint':` arm to `case 'subint' |
+  'subint_fork':` since both use the same py3.14+ gate.
+- Registered `subint_fork_proc` in `_methods` with a
+  pointer-comment at the analysis doc.
+
+### `tractor/spawn/_subint.py` (modified across session)
+
+> `git log 26fb820..HEAD -- tractor/spawn/_subint.py`
+
+- Tightened `_has_subints` gate: dual-requires public
+  `concurrent.interpreters` + private `_interpreters`
+  (tests for py3.14-or-newer on the public-API presence,
+  then uses the private one for legacy-config subints
+  because `msgspec` still blocks the public isolated mode
+  per jcrist/msgspec#563).
+- Updated module docstring, `subint_proc()` docstring, and
+  gate-error messages to reflect the 3.14+ requirement and
+  the reason (py3.13 wedges under multi-trio usage even
+  though the private module exists there).
+
+### `tractor/_testing/pytest.py` (modified)
+
+> `git log 26fb820..HEAD -- tractor/_testing/pytest.py`
+
+- New `skipon_spawn_backend(*start_methods, reason=...)`
+  pytest marker expanded into `pytest.mark.skip(reason=...)`
+  at collection time via
+  `pytest_collection_modifyitems()`.
+- Implementation uses `item.iter_markers(name=...)` which
+  walks function + class + module scopes uniformly and
+  handles both `pytestmark = <single Mark>` and
+  `pytestmark = [mark, ...]` forms natively. ~30-LOC
+  single-loop refactor replacing a prior nested
+  conditional that had four bugs (see "Review" narrative
+  above).
+- Added `pytest.Config` / `pytest.Function` /
+  `pytest.FixtureRequest` type annotations on fixture
+  signatures while touching the file.
+
+### `pyproject.toml` (modified)
+
+> `git log 26fb820..HEAD -- pyproject.toml`
+
+Added `pytest-timeout>=2.3` to `testing` dep group with
+comment pointing at the `ai/conc-anal/` docs.
+
+### `tests/discovery/test_registrar.py`,
+`tests/test_subint_cancellation.py`,
+`tests/test_cancellation.py` (modified)
+
+> `git log 26fb820..HEAD -- tests/`
+
+Applied `@pytest.mark.timeout(30, method='thread')` on
+known-hanging subint tests. Extended comments to cross-
+reference the `ai/conc-anal/*.md` docs. `method='thread'`
+is documented inline as load-bearing (`signal`-method
+SIGALRM suffers the same GIL-starvation path that drops
+SIGINT).
+
+### `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md` (new)
+
+> `git show 797f57c -- ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+
+Third sibling doc under `conc-anal/`. Structure: TL;DR,
+context ("what we tried"), symptom (the user's exact
+`Fatal Python error` output), CPython source walkthrough
+with excerpted snippets from `posixmodule.c` +
+`pystate.c`, chain summary, definitive answer to Open
+Question 1, `## Upstream-report draft (for CPython issue
+tracker)` section with a two-tier ask, references.
+
+### `ai/conc-anal/subint_fork_from_main_thread_smoketest.py` (new, THIS turn)
+
+Zero-tractor-import smoke test for the proposed workaround
+architecture. Four argparse-driven scenarios covering the
+control case + baseline + arch-critical case + end-to-end.
+Pass/fail banners per scenario; clean `--help` output;
+py3.13 early-exit.
+
+## Non-code output (verbatim)
+
+### The `strace` signature that kicked off the CPython
+walkthrough
+
+```
+--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
+write(16, "\2", 1)                      = -1 EAGAIN (Resource temporarily unavailable)
+rt_sigreturn({mask=[WINCH]})            = 139801964688928
+```
+
+### Key user quotes framing the direction
+
+> ok actually we get this [fatal error] ... see if you can
+> take a look at what's going on, in particular wrt to
+> cpython's sources. pretty sure there's a local copy at
+> ~/repos/cpython/
+
+(Drove the CPython walkthrough that produced the
+definitive refusal chain.)
+
+> is there any reason we can't just sidestep this "must fork
+> from main thread in main subint" issue by simply ensuring
+> a "subint forking server" is always setup prior to
+> invoking trio in a non-main-thread subint ...
+
+(Drove the main-interp-thread-forkserver architectural
+discussion + smoke-test script design.)
+
+### CPython source tags for quick jump-back
+
+```
+Modules/posixmodule.c:728   PyOS_AfterFork_Child()
+Modules/posixmodule.c:753   // Ideally we could guarantee tstate is running main.
+Modules/posixmodule.c:778   status = _PyInterpreterState_DeleteExceptMain(runtime);
+
+Python/pystate.c:1040       _PyInterpreterState_DeleteExceptMain()
+Python/pystate.c:1044-1047  tstate->interp != main → PyStatus_ERR("not main interpreter")
+```

ai/prompt-io/claude/20260601T231429Z_0e3e008b_prompt_io.md

@@ -0,0 +1,146 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: trio-0.33-subproc-supervisor-retroactive
+timestamp: 2026-06-01T23:14:29Z
+git_ref: 0e3e008b
+scope: code
+substantive: true
+raw_file: 20260601T231429Z_0e3e008b_prompt_io.raw.md
+---
+
+## Prompt
+
+**RETROACTIVE LOG** — original session prompts not
+preserved; reconstructed from the staged work product.
+
+The work designs a `trio.Nursery.start()`-style wrapper
+around `trio.run_process()` for SC-friendly subprocess
+supervision. From the resulting code shape, the
+prompting intent was:
+
+1. Surface rc!=0 `CalledProcessError` DETERMINISTICALLY,
+   without the nursery-eg-wrapping that complicates
+   `collapse_eg()` usage and races the relay reader on
+   trio's `check=True`-driven cancel cascade.
+2. ALWAYS isolate the parent controlling-tty so a
+   spawned child can't emit terminal control-seqs onto
+   the launching tty (clobbering scrollback). Default
+   `stdin=DEVNULL`; default `stdout=DEVNULL` unless
+   explicitly relayed/overridden; distinguish "caller
+   passed nothing" from "caller passed `None` for
+   inherit".
+3. Optional live per-line relay of child std-streams to
+   the `tractor` log — STREAMED (not
+   buffered-until-exit) so long-lived daemon output is
+   visible during the run. Pick a custom log level that
+   shows at usual `info`/`devx` console levels but is
+   separately filterable.
+4. Concurrent pipe-drain reader MANDATORY when piping
+   without `capture_*` — without it the child blocks on
+   `write()` once the OS pipe buffer fills (~64KiB),
+   causing deadlocks on output bursts.
+5. Non-blocking `tn.start()` semantics: hand the live
+   `trio.Process` to the parent immediately;
+   supervise/relay run to completion in the supervisor
+   coro.
+6. Hermetic `trio`-only unit tests (no actor-runtime)
+   covering each of: per-line relay, tty isolation,
+   no-deadlock on >64KiB unnewlined output, CPE
+   rebuild w/ stderr relay, CPE rebuild on the silent
+   drain+capture path.
+
+## Response summary
+
+Adds `tractor/trionics/_subproc.py` (296 LOC) +
+`tests/trionics/test_subproc.py` (230 LOC) + a
+re-export in `tractor/trionics/__init__.py`.
+
+**`supervise_run_process()`** (public, re-exported)
+- `check=False` is forced to `trio.run_process`; the
+  rc-check runs in the supervisor coro AFTER `own_tn`
+  unwinds (both the child AND the relay readers have
+  hit EOF + fully drained). A BARE
+  `subprocess.CalledProcessError` is rebuilt + raised
+  from there, with `.stderr` bytes passed in the
+  constructor AND attached as an `add_note()`'d
+  `|_.stderr:` block for legible teardown logs.
+- `stdin=DEVNULL` always. `stdout` default chosen via a
+  `_UNSET` sentinel: `relay_stdout=True` → PIPE,
+  explicit `stdout=...` → as given, else `DEVNULL`.
+  `stderr` defaults to PIPE whenever we relay OR need
+  the CPE note (when `check=True`), else `DEVNULL`.
+- `relay_level='io'` (custom level 21; sorts just
+  above stdlib `INFO`=20 so it shows at usual
+  `info`/`devx` levels and stays separately
+  filterable). `runtime`=15 would silently filter at
+  default levels, so it's rejected as a default.
+- `task_status.started(trio_proc)` delivers the live
+  process immediately. The internal `own_tn`
+  supervises `trio.run_process` + any relay readers to
+  completion.
+- `**run_process_kwargs` forward verbatim;
+  `stdin/stdout/stderr/check` are MANAGED keys
+  (override on conflict).
+- Crash-handling deliberately NOT baked in — compose
+  `maybe_open_crash_handler()` on top at the call-site.
+
+**`_relay_stream_lines()`** (internal helper)
+- Three modes (combinable): `emit`-only (live per-line
+  relay), `accum`-only (silent drain+capture for a CPE
+  note), or both (live relay AND capture).
+- Per-line split handles cross-chunk residuals via a
+  rolling `residual` bytes buffer; flushes any trailing
+  un-newline-term'd line at EOF.
+- `async with stream:` ensures aclose at EOF/cancel
+  (mirrors trio's internal `_subprocess` drain idiom).
+
+**`_add_stderr_note()`** (internal helper)
+- `add_note()`s a `textwrap.indent(...)`'d
+  `|_.stderr:` block onto a `CalledProcessError` for
+  teardown logs.
+
+**Tests** (5 hermetic, trio-only) — `_capture_relay`
+fixture monkeypatches `_subproc.log.<level>` to a list:
+- `test_stdout_relayed_per_line`: per-line stdout
+  relay carries each `line=N` to the records.
+- `test_parent_tty_isolated`: `readlink /proc/self/fd/0`
+  and `fd/1` from the child show `pipe:` (fd1) +
+  `/dev/null` (fd0); NO `/dev/pts/*`.
+- `test_no_deadlock_on_big_unnewlined_output`: 200KiB
+  of `x` with no newlines completes inside
+  `fail_after(2)` — exercises the concurrent drain.
+- `test_stderr_relay_and_cpe_rebuild`: rc=3 with
+  `relay_stderr=True` raises bare CPE
+  (via `collapse_eg()`) with `b'boom' in cpe.stderr`,
+  the note attached, AND per-line live relay.
+- `test_nonrelay_cpe_note`: rc=7 with no relay still
+  produces CPE with `.stderr` + note via the silent
+  drain+capture path.
+
+## Files changed
+
+- `tractor/trionics/_subproc.py` — NEW. Public
+  `supervise_run_process()` + helpers
+  `_relay_stream_lines()` / `_add_stderr_note()` + the
+  `_UNSET` sentinel.
+- `tests/trionics/test_subproc.py` — NEW. 5 hermetic
+  trio-only tests + `_capture_relay` monkeypatch
+  fixture.
+- `tractor/trionics/__init__.py` — re-export
+  `supervise_run_process`.
+
+## Human edits
+
+**RETROACTIVE**: this log is being written from the
+staged diff, not from a live session. The code as
+staged is the canonical artifact; any human edits the
+user made during the originating design session are
+already integrated and cannot be separated post-hoc.
+The `.raw.md` sibling is a diff-pointer placeholder,
+NOT a pre-edit transcript.
+
+Future prompt-io entries for in-flight work should be
+written DURING the design session per the skill
+contract so the pre-edit `.raw.md` captures the
+unedited model output for genuine provenance.

ai/prompt-io/claude/20260601T231429Z_0e3e008b_prompt_io.raw.md

@@ -0,0 +1,106 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-06-01T23:14:29Z
+git_ref: 0e3e008b
+diff_cmd: git diff HEAD~1..HEAD
+---
+
+# RETROACTIVE — original model output not preserved
+
+This `.raw.md` would normally contain the verbatim
+pre-human-edit response from the design session that
+produced the staged `_subproc.py` module + tests. That
+session's transcript is not available, so this file
+serves as a diff-pointer placeholder + transparency
+note.
+
+## Authoritative artifact
+
+The committed code IS the artifact of record. Once the
+companion commit lands, the unified diff is:
+
+> `git diff HEAD~1..HEAD -- tractor/trionics/_subproc.py`
+> `git diff HEAD~1..HEAD -- tests/trionics/test_subproc.py`
+> `git diff HEAD~1..HEAD -- tractor/trionics/__init__.py`
+
+Before committing, substitute `--cached` for the
+pre-commit form.
+
+## What is NOT here
+
+Because this is retroactive:
+- No verbatim chain-of-thought / discussion prose from
+  the design session.
+- No rejected alternatives the model considered before
+  arriving at the final shape (e.g. whether the
+  rc-check should live inside `own_tn` vs after it; the
+  `_UNSET` sentinel vs a `None`-means-DEVNULL
+  convention; `io` vs `info` as the default relay
+  level).
+- No pre-edit code blocks as the model first emitted
+  them, separable from any user cleanup applied before
+  the diff was staged.
+
+## Inferred design choices visible in the final code
+
+(Documented here because they're the kind of decision
+detail an unedited raw transcript would have captured.)
+
+1. **Post-drain rc-check in the supervisor coro body,
+   AFTER `own_tn.__aexit__`.** Placing the
+   `CalledProcessError` raise here (not inside
+   `own_tn`) means the EG-unwrap happens at the OUTER
+   `tn.start()` boundary — callers do `collapse_eg()`
+   if they want bare. Doing the raise INSIDE `own_tn`
+   would cancel the still-draining relay reader
+   mid-flight and lose stderr lines.
+
+2. **`_UNSET` sentinel for `stdout`.** A plain default
+   of `None` couldn't distinguish "use the safe
+   `DEVNULL` default" from "caller explicitly passed
+   `None` (inherit, presumably knowingly)". The
+   sentinel keeps the SAFE default while letting power
+   users opt into inherit.
+
+3. **`relay_level='io'` (custom level 21).** Chosen to
+   sort just above stdlib `INFO`=20 so a default
+   `--ll info` shows the relay, but it remains a
+   distinct level so users can filter
+   `tractor.trionics:io` separately. Picking
+   `runtime`=15 would have made the relay invisible at
+   default verbosity (a footgun for daemon supervisors
+   whose whole point is "I want to see this output").
+
+4. **Reader is MANDATORY, not opt-in cosmetic.** With
+   `stdout=PIPE` / `stderr=PIPE` we OWN the drain
+   responsibility — there's no `trio.capture_*` running
+   under the hood here. The ~64KiB OS pipe buffer
+   means a child writing more than that without us
+   reading hangs at `write()` — a deadlock that won't
+   show up in small-output tests, which is why the
+   200KiB-no-newline test is in the suite.
+
+5. **`task_status.started(trio_proc)` BEFORE the
+   `own_tn` exits.** Without this, `tn.start()` would
+   block until the child exits — losing the "start a
+   long-lived daemon and continue with parent work"
+   use case. With it, the parent gets the live process
+   handle immediately and the supervise+relay tasks
+   run in the supervisor coro until the child exits.
+
+6. **`__notes__` via `add_note()` for the CPE
+   `.stderr`.** The `.stderr` attribute is what
+   `subprocess` callers expect; the `add_note()` is
+   what trio's exception-rendering shows. Both wired so
+   programmatic AND human consumers see the stderr at
+   teardown.
+
+## Honesty statement
+
+This file's content is RECONSTRUCTED from the staged
+code, not extracted from a verbatim model transcript.
+The prompt-io skill's intent is for the `.raw.md` to
+be a pre-edit fossil; that's not possible here. Future
+work should write the prompt-io entry DURING the
+design session.

ai/tooling-todos/logspec_leaf_module_granularity_route_b.md

@@ -0,0 +1,159 @@
+# Logging-spec leaf-module granularity — "Route B" (decouple
+# logger-*identity* from console-*display*)
+
+Follow-up notes recording the breaking-changes / costs of the
+deeper fix that would give the `tractor.log` logging-spec (see
+`LogSpec`/`apply_logspec()`) true **per-leaf-MODULE** level
+control — deliberately *not* taken (for now) in favour of the
+smaller sub-PACKAGE fix already landed.
+
+## Status / what already shipped
+
+The cheap, contained fix is **done**: `get_logger()`'s "strip
+#2" (`log.py`, the `pkg_path = subpkg_path` collapse) no longer
+eats a real sub-package component. It now strips the trailing
+token *only* when it duplicates the caller's leaf-*module*
+filename (which the header already shows via `{filename}`).
+
+Result:
+
+- `devx.debug` resolves to `tractor.devx.debug`, **distinct**
+  from a bare `devx` -> `tractor.devx` (its parent). So the
+  logging-spec can dial sub-package levels at any nesting depth
+  (`devx.debug:runtime` ≠ `devx:cancel`).
+- The `get_logger(__name__)` cosmetic ("don't repeat the leaf
+  module in `{name}` since `{filename}` shows it") is preserved.
+
+What is **still NOT addressable** after that fix:
+
+- **Per-leaf-MODULE** levels. Every module in a (sub-)pkg shares
+  that pkg's logger, because `get_logger()` drops the leaf
+  module-name from the logger key by design.
+- **Top-level lib modules** (eg. `tractor.to_asyncio`,
+  `__package__ == 'tractor'`) emit on the *root* `tractor`
+  logger, so a `to_asyncio:<lvl>` spec entry hits a phantom
+  child -> no-op.
+
+## What "Route B" is
+
+Make the logger's *identity* the **full dotted module path**
+(incl. the leaf module + top-level modules), eg.
+`tractor.devx.debug._tty_lock` and `tractor.to_asyncio`, and
+move the cosmetic leaf-trim out of logger-naming and into the
+**formatter's `{name}` rendering**.
+
+Net effect:
+
+- Real per-module `Logger` nodes exist in the hierarchy ->
+  the spec can target ANY module; stdlib level-inheritance and
+  propagation "just work" top-down.
+- Console headers stay clean because the formatter computes a
+  trimmed display string (drop the trailing token that equals
+  `{filename}`'s stem) instead of the logger doing it.
+
+## Why it's "broad" — breaking changes / costs
+
+The logger *name* is currently load-bearing well beyond
+display; changing it ripples:
+
+1. **Every logger name changes.**
+   Today (post sub-pkg fix) names collapse to the sub-package;
+   Route B = full module path. This touches:
+   - handler attachment points + the `getChild()` hierarchy,
+   - any `logging.getLogger('tractor.X')` string lookups,
+   - any name-based filtering,
+   - the dedup / `_strict_debug` warning logic *inside*
+     `get_logger()` itself — the `pkg_name in name`,
+     `leaf_mod in pkg_path`, "duplicate pkg-name" branches all
+     key off the *name shape* and would need re-derivation.
+
+2. **Formatter rewrite.**
+   `LOG_FORMAT` uses `{name}` == `record.name` (the full logger
+   name). To keep headers clean we must compute a *display*
+   name and inject it as a record attr (eg. `record.pkg_ns`)
+   via a `logging.Filter` or a `colorlog.ColoredFormatter`
+   subclass overriding `.format()`, then point `LOG_FORMAT` at
+   that field. The `{filename}` vs `{name}` de-dup intent has
+   to be re-implemented per-record rather than per-logger.
+
+3. **Propagation / double-emit surface grows.**
+   Full-depth loggers mean more intermediate nodes
+   (`...debug._tty_lock` -> `.debug` -> `.devx` -> `tractor`).
+   If more than one level carries a handler (spec sub-handlers
+   + a root console), records double-emit. The
+   `propagate=False` trick we already use for filter-targeted
+   sub-loggers (`apply_logspec()`) must be applied carefully
+   across a deeper tree — more levels == more places to leak a
+   dup.
+
+4. **Level-inheritance semantics shift.**
+   Today setting a level on `tractor.devx` gates *all* devx
+   emits (they share that logger). Post-Route-B,
+   `tractor.devx.debug._tty_lock` is its own `NOTSET` logger
+   that *inherits* the effective level from ancestors —
+   functionally similar via inheritance, BUT any code that does
+   `log.setLevel(...)` / reads `log.level` on a (previously
+   collapsed) logger now only affects that exact node. All
+   `setLevel`/`.level =` call sites need an audit (eg.
+   `get_logger()`'s own `log.level = rlog.level` line).
+
+5. **Downstream contract churn.**
+   `modden` / `piker` call `get_logger()` / `get_console_log()`
+   and may depend on current names — including
+   `modden.runtime.daemon.setup_tractor_logging()` which
+   asserts `'tractor' not in name` on spec parts. The header
+   `{name}` field is user-visible in everyone's logs + CI
+   output. Changing the canonical names is a public-ish
+   behavior change -> needs a version note + downstream
+   coordination (or a formatter trim that keeps the *displayed*
+   string byte-identical to today).
+
+6. **`get_logger()` refactor risk.**
+   The fn tangles two concerns: compute logger *identity* and
+   compute the *display* string. Route B forces splitting them
+   inside a ~300-line fn with multiple `_strict_debug`
+   branches, dup-warnings, and the `name=__name__` convenience.
+   High chance of subtle regressions without an exhaustive
+   name-derivation test matrix.
+
+## Migration / test plan (if pursued)
+
+- Extract a pure helper
+  `_mk_logger_name(pkg_name, mod_name, mod_pkg) -> (logger_name,
+  display_name)` and cover it with an exhaustive unit matrix:
+  auto vs explicit vs `__name__`; package-`__init__` vs leaf
+  module; nested vs flat; `pkg_name in name` vs not; top-level
+  module (`__package__ == pkg_name`).
+- Switch `get_logger()` to use it for *identity*; switch the
+  formatter to use `display_name` (via a record attr).
+- Re-run the full suite + golden-diff a sample of rendered log
+  headers to confirm zero cosmetic churn.
+- Coordinate the name change with `modden`/`piker`; bump +
+  CHANGES note.
+
+## Cheaper alternative — "Route A" (record-filter)
+
+If per-leaf control is wanted *before* committing to Route B:
+keep names collapsed, add a `logging.Filter` on the configured
+handler keyed on `record.module` / `record.pathname` that maps
+each record's source module -> its spec level. Set the base
+logger to the *minimum* level in the spec (so records aren't
+pre-dropped by the logger), and let the filter discriminate
+up/down within that floor.
+
+- Pros: no name churn, no formatter change, fully contained
+  next to `apply_logspec()`.
+- Cons: a filter can only discriminate *within* what the logger
+  admits -> base must be permissive, so `at_least_level()`
+  expensive-work guards over-admit; matching dotted spec names
+  to a `pathname` is fiddly; doesn't clean up the hierarchy
+  itself.
+
+## Recommendation
+
+- Defer Route B unless true per-module loggers are wanted as a
+  first-class feature.
+- If per-leaf control is needed soon, prefer **Route A**
+  (filter) — lower risk.
+- The shipped sub-PACKAGE fix already covers the common ask
+  (`devx.debug` vs `devx`).

examples/debugging/multi_daemon_subactors.py

@@ -27,12 +27,9 @@ async def main():
     '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='cancel',
-        # loglevel='devx',
-    ) as n:
-
-        p0 = await n.start_actor('bp_forever', enable_modules=[__name__])
-        p1 = await n.start_actor('name_error', enable_modules=[__name__])
+    ) as an:
+        p0 = await an.start_actor('bp_forever', enable_modules=[__name__])
+        p1 = await an.start_actor('name_error', enable_modules=[__name__])
 
         # retreive results
         async with p0.open_stream_from(breakpoint_forever) as stream:

examples/debugging/multi_nested_subactors_error_up_through_nurseries.py

@@ -67,7 +67,7 @@ async def main():
     """
     async with tractor.open_nursery(
         debug_mode=True,
-        # loglevel='cancel',
+        loglevel='pdb',
     ) as n:
 
         # spawn both actors

examples/debugging/root_cancelled_but_child_is_in_tty_lock.py

@@ -39,8 +39,8 @@ async def main():
     '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='devx',
-        enable_transports=['uds'],
+        enable_transports=['uds'],  # TODO, apss this via osenv?
+        loglevel='devx',  # XXX, required for test!
     ) as n:
 
         # spawn both actors

examples/debugging/root_timeout_while_child_crashed.py

@@ -1,4 +1,3 @@
-
 import trio
 import tractor
 
@@ -9,16 +8,22 @@ async def key_error():
 
 
 async def main():
-    """Root dies 
+    '''
+    Root is fail-after-cancelled while blocking and child RPC fails
+    simultaneously.
 
-    """
+    '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='debug'
+        # loglevel='debug'  # ?XXX required?
     ) as n:
 
         # spawn both actors
         portal = await n.run_in_actor(key_error)
+        print(
+            f'Child is up @ {portal.chan.aid.reprol()}'
+        )
+
 
         # XXX: originally a bug caused by this is where root would enter
         # the debugger and clobber the tty used by the repl even though

examples/debugging/shield_hang_in_sub.py

@@ -49,9 +49,11 @@ async def main(
         tractor.open_nursery(
             debug_mode=True,
             enable_stack_on_sig=True,
-            # maybe_enable_greenback=False,
-            loglevel='devx',
+            loglevel='devx',  # XXX REQUIRED log level!
             enable_transports=[tpt],
+            # maybe_enable_greenback=True,
+            # ^TODO? maybe a "smarter" way todo all this is how
+            # `modden` does with a rtv serialized through the osenv?
         ) as an,
     ):
         ptl: tractor.Portal  = await an.start_actor(
@@ -63,7 +65,9 @@ async def main(
             start_n_shield_hang,
         ) as (ctx, cpid):
 
-            _, proc, _ = an._children[ptl.chan.uid]
+            _, proc, _ = an._children[
+                ptl.chan.aid.uid
+            ]
             assert cpid == proc.pid
 
             print(

examples/debugging/subactor_bp_in_ctx.py

@@ -36,6 +36,11 @@ async def just_bp(
 
 async def main():
 
+    # !TODO, parametrize the --tpt-proto={key} with osenv vars just
+    # like we do for loglevel/spawn-backend!
+    # - [ ] run on both tpts for all such debugger tests?
+    # - [ ] special skip for macos!
+    #
     if platform.system() != 'Darwin':
         tpt = 'uds'
     else:

examples/debugging/subactor_error.py

@@ -9,7 +9,6 @@ async def name_error():
 async def main():
     async with tractor.open_nursery(
         debug_mode=True,
-        # loglevel='transport',
     ) as an:
 
         # TODO: ideally the REPL arrives at this frame in the parent,

examples/debugging/sync_bp.py

@@ -1,9 +1,22 @@
 from functools import partial
+import os
 import time
 
+# ?TODO? how to make `pdbp` enforce this?
+# os.environ['PYTHON_COLORS'] = '0'
+# os.environ['NO_COLOR'] = '1'
+
 import trio
 import tractor
 
+# disable `pbdp` prompt colors
+# for prompt matching in test.
+def disable_pdbp_color():
+    if os.environ['PYTHON_COLORS'] == '0':
+        from tractor.devx.debug import _repl
+        _repl.TractorConfig.use_pygments = False
+
+
 # TODO: only import these when not running from test harness?
 # can we detect `pexpect` usage maybe?
 # from tractor.devx.debug import (
@@ -42,6 +55,7 @@ async def start_n_sync_pause(
     ctx: tractor.Context,
 ):
     actor: tractor.Actor = tractor.current_actor()
+    disable_pdbp_color()
 
     # sync to parent-side task
     await ctx.started()
@@ -52,13 +66,15 @@ async def start_n_sync_pause(
 
 
 async def main() -> None:
+    disable_pdbp_color()
     async with (
         tractor.open_nursery(
             debug_mode=True,
             maybe_enable_greenback=True,
-            enable_stack_on_sig=True,
-            # loglevel='warning',
-            # loglevel='devx',
+
+            # XXX flags required for test pattern matching.
+            loglevel='pdb',
+            # enable_stack_on_sig=True,
         ) as an,
         trio.open_nursery() as tn,
     ):
@@ -68,8 +84,8 @@ async def main() -> None:
         p: tractor.Portal  = await an.start_actor(
             'subactor',
             enable_modules=[__name__],
-            # infect_asyncio=True,
             debug_mode=True,
+            # infect_asyncio=True,
         )
 
         # TODO: 3 sub-actor usage cases:

pyproject.toml

@@ -9,7 +9,7 @@ name = "tractor"
 version = "0.1.0a6dev0"
 description = 'structured concurrent `trio`-"actors"'
 authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
-requires-python = ">=3.12, <3.14"
+requires-python = ">=3.13, <3.15"
 readme = "docs/README.rst"
 license = "AGPL-3.0-or-later"
 keywords = [
@@ -31,6 +31,7 @@ classifiers = [
   "Programming Language :: Python :: 3 :: Only",
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
   "Topic :: System :: Distributed Computing",
 ]
 dependencies = [
@@ -46,11 +47,17 @@ dependencies = [
   # built-in multi-actor `pdb` REPL
   "pdbp>=1.8.2,<2", # windows only (from `pdbp`)
   # typed IPC msging
-  "msgspec>=0.21.0",
-  "cffi>=1.17.1",
+  "msgspec>=0.20.0",
   "bidict>=0.23.1",
   "multiaddr>=0.2.0",
   "platformdirs>=4.4.0",
+  # per-actor `argv[0]` proc-title for OS-level diag tools
+  # (`ps`, `top`, `psutil`-backed tooling like `acli.pytree`).
+  # Optional at runtime — guarded by `try/except ImportError` in
+  # `tractor.devx._proctitle` — but listed here so default
+  # installs benefit from it. See tracking issue for follow-ups
+  # (e.g. richer formats, per-backend overrides).
+  "setproctitle>=1.3,<2",
 ]
 
 # ------ project ------
@@ -60,30 +67,58 @@ dev = [
   {include-group = 'devx'},
   {include-group = 'testing'},
   {include-group = 'repl'},
+  {include-group = 'sync_pause'},
 ]
 devx = [
   # `tractor.devx` tooling
-  "greenback>=1.2.1,<2",
   "stackscope>=0.2.2,<0.3",
   # ^ requires this?
   "typing-extensions>=4.14.1",
+  # {include-group = 'sync_pause'},  # XXX, no 3.14 yet!
+]
+sync_pause = [
+  "greenback>=1.2.1,<2",  # TODO? 3.14 greenlet on nix?
 ]
 testing = [
   # test suite
   # TODO: maybe some of these layout choices?
   # https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
-  "pytest>=8.3.5",
+  # bumped 8.3.5 → 9.0 per upstream security advisory + our
+  # local-only reliance on the post-9.0 capture-machinery shape
+  # (the `sys.__stderr__`-bypass print in
+  # `tractor._testing.trace._do_capture_snapshot` works on 8.x
+  # too, but standardizing on 9.x here ensures `--show-capture`
+  # interactions stay predictable across dev installs).
+  "pytest>=9.0",
   "pexpect>=4.9.0,<5",
+  # per-test wall-clock bound (used via
+  # `@pytest.mark.timeout(..., method='thread')` on the
+  # known-hanging `subint`-backend audit tests; see
+  # `ai/conc-anal/subint_*_issue.md`).
+  "pytest-timeout>=2.3",
+  # used by `tractor._testing._reap` for the
+  # `tractor-reap` zombie-subactor + leaked-shm
+  # cleanup utility (xplatform `Process.memory_maps`,
+  # `Process.open_files`).
+  "psutil>=7.0.0",
 ]
 repl = [
   "pyperclip>=1.9.0",
   "prompt-toolkit>=3.0.50",
-  "xonsh>=0.22.2",
+  "xonsh>=0.23.8",
   "psutil>=7.0.0",
 ]
 lint = [
   "ruff>=0.9.6"
 ]
+# XXX, used for linux-only hi perf eventfd+shm channels
+# now mostly moved over to `hotbaud`.
+eventfd = [
+  "cffi>=1.17.1",
+]
+subints = [
+  "msgspec>=0.21.0",
+]
 # TODO, add these with sane versions; were originally in
 # `requirements-docs.txt`..
 # docs = [
@@ -92,10 +127,26 @@ lint = [
 # ]
 # ------ dependency-groups ------
 
+[tool.uv.dependency-groups]
+# for subints, we require 3.14+ due to 2 issues,
+# - hanging behaviour for various multi-task teardown cases (see
+#   "Availability" section in the `tractor.spawn._subints` doc string).
+# - `msgspec` support which is oustanding per PEP 684 upstream tracker:
+#   https://github.com/jcrist/msgspec/issues/563
+#
+# https://docs.astral.sh/uv/concepts/projects/dependencies/#group-requires-python
+subints = {requires-python = ">=3.14"}
+eventfd = {requires-python = ">=3.13, <3.14"}
+sync_pause = {requires-python = ">=3.13, <3.14"}
 
 [tool.uv.sources]
 # XXX NOTE, only for @goodboy's hacking on `pprint(sort_dicts=False)`
 # for the `pp` alias..
+# ------ gh upstream ------
+# xonsh = { git = 'https://github.com/anki-code/xonsh.git', branch = 'prompt_next_suggestion' }
+# ^ https://github.com/xonsh/xonsh/pull/6048
+# xonsh = { git = 'https://github.com/xonsh/xonsh.git', branch = 'main' }
+# xonsh = { path = "../xonsh", editable = true }
 
 # [tool.uv.sources.pdbp]
 # XXX, in case we need to tmp patch again.
@@ -164,6 +215,35 @@ all_bullets = true
 
 [tool.pytest.ini_options]
 minversion = '6.0'
+# NOTE: `pytest-timeout`'s global per-test cap is intentionally
+# NOT set — both of its enforcement methods break trio's
+# runtime under our fork-based spawn backends:
+#
+# - `method='signal'` (the default; SIGALRM) raises `Failed`
+#   synchronously from the signal handler in trio's main
+#   thread, which leaves `GLOBAL_RUN_CONTEXT` half-installed
+#   ("Trio guest run got abandoned"). EVERY subsequent
+#   `trio.run()` in the same pytest session then bails with
+#   `RuntimeError: Attempted to call run() from inside a
+#   run()` — full-session poison: a single 200s hang
+#   cascades into 30+ false-positive failures across
+#   downstream test files.
+#
+# - `method='thread'` calls `_thread.interrupt_main()` which
+#   can let the resulting `KeyboardInterrupt` escape trio's
+#   `KIManager` under fork-cascade teardown races, killing
+#   the whole pytest session.
+#
+# For tests that legitimately need a wall-clock cap, use
+# `with trio.fail_after(N):` INSIDE the test — trio's own
+# Cancelled machinery handles the timeout cleanly through
+# the actor nursery without disturbing global state. See
+# `tests/test_advanced_streaming.py::test_dynamic_pub_sub`'s
+# module-level NOTE for the canonical pattern.
+#
+# CI environments should rely on job-level wall-clock
+# timeouts (e.g. GitHub Actions `timeout-minutes`) for an
+# escape hatch on genuinely-stuck suites.
 # https://docs.pytest.org/en/stable/reference/reference.html#configuration-options
 testpaths = [
   'tests'
@@ -171,15 +251,27 @@ testpaths = [
 addopts = [
   # TODO: figure out why this isn't working..
   '--rootdir=./tests',
-
   '--import-mode=importlib',
   # don't show frickin captured logs AGAIN in the report..
   '--show-capture=no',
 
+  # load builtin plugin since we need a boostrapping hook,
+  # `pytest_load_initial_conftests()` for `--capture=` per:
+  # https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
+  '-p tractor._testing.pytest',
+
   # disable `xonsh` plugin
   # https://docs.pytest.org/en/stable/how-to/plugins.html#disabling-plugins-from-autoloading
   # https://docs.pytest.org/en/stable/how-to/plugins.html#deactivating-unregistering-a-plugin-by-name
-  '-p no:xonsh'
+  '-p no:xonsh',
+
+  # XXX default on non-forking spawners
+  '--capture=fd',
+  # '--capture=sys',
+  # ^XXX NOTE^ ALWAYS SET THIS for `*_forkserver` spawner
+  # backends! see details @
+  # `tractor._testing.pytest.pytest_load_initial_conftests()`
+
 ]
 log_cli = false
 # TODO: maybe some of these layout choices?

scripts/cpu-perf-check

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+# tractor: distributed structured concurrency.
+# Copyright 2018-eternity Tyler Goodlet.
+#
+# SPDX-License-Identifier: AGPL-3.0-or-later
+'''
+`cpu-perf-check` — sustained-load CPU throttle detector.
+
+Idle freq snapshots LIE. A laptop can read
+`governor=performance`, `EPP=performance`,
+`platform_profile=performance`, `scaling_max_freq=<full>`
+and momentarily clock a P-core at 5GHz — while a
+firmware/EC power cap (AMD PPT/STAPM and friends) clamps
+the whole package to ~1.5GHz the instant a sustained
+multi-core load lands. That throttle masquerades as a
+`trio`-backend test *regression*: a wave of `fail_after` /
+`TooSlowError` / `Cancelled(source='deadline')` deadline
+misses on spawn-heavy tests, on byte-identical code that
+was green yesterday.
+
+The existing `tests/conftest.py:cpu_scaling_factor()` only
+reads STATIC `scaling_max_freq` vs `*_pstate_max_freq`, so
+it returns `1.0` (no throttle) during exactly this failure
+— it can't see the cap. This script complements it by
+BURNING every core for a few seconds and sampling the
+ACHIEVED `scaling_cur_freq`, which is the only thing that
+exposes the clamp.
+
+Exit code: `0` if sustained perf looks restored, `1` if
+throttled — so it gates a test run:
+
+    py313/bin/python scripts/cpu-perf-check && pytest tests/ ...
+
+Tunables (env-overridable):
+    CPU_PERF_SECS        load duration       (default 4.0)
+    CPU_PERF_HEALTHY_FRAC sustained/max floor (default 0.45)
+
+'''
+from __future__ import annotations
+import glob
+import os
+import time
+import multiprocessing as mp
+
+
+def _read(path: str) -> str | None:
+    try:
+        with open(path) as f:
+            return f.read().strip()
+    except OSError:
+        return None
+
+
+def _cur_freqs_mhz() -> list[int]:
+    out: list[int] = []
+    for f in glob.glob(
+        '/sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq'
+    ):
+        if (v := _read(f)):
+            out.append(int(v) // 1000)
+    return out
+
+
+def _pkg_max_mhz() -> int:
+    '''
+    Highest per-core ceiling across the package — the
+    P-core max on hybrid parts.
+
+    '''
+    mxs: list[int] = []
+    for f in glob.glob(
+        '/sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_max_freq'
+    ):
+        if (v := _read(f)):
+            mxs.append(int(v) // 1000)
+    return max(mxs) if mxs else 0
+
+
+def _burn(stop: float) -> None:
+    x: int = 1
+    while time.perf_counter() < stop:
+        x += x * x ^ 0x5
+
+
+def main(
+    secs: float = float(os.environ.get('CPU_PERF_SECS', 4.0)),
+    # sustained aggregate must clear this fraction of the
+    # package max-freq ceiling. Throttled (~1.5GHz of 5.1GHz)
+    # ~= 0.29; a healthy all-core load easily clears 0.5.
+    healthy_frac: float = float(
+        os.environ.get('CPU_PERF_HEALTHY_FRAC', 0.45)
+    ),
+) -> int:
+    if not glob.glob('/sys/devices/system/cpu/cpu0/cpufreq'):
+        print('no cpufreq sysfs (non-linux?) — skipping, assume OK')
+        return 0
+
+    b: str = '/sys/devices/system/cpu/cpu0/cpufreq/'
+    pkg_max: int = _pkg_max_mhz()
+    print('=== static knobs (ALL can read fine while throttled) ===')
+    print(f'  governor          : {_read(b + "scaling_governor")}')
+    print(f'  EPP               : {_read(b + "energy_performance_preference")}')
+    print(f'  platform_profile  : '
+          f'{_read("/sys/firmware/acpi/platform_profile")}')
+    print(f'  pkg max freq      : {pkg_max} MHz')
+
+    ncpu: int = os.cpu_count() or 1
+    print(f'\n=== sustained {ncpu}-core load ({secs:.0f}s) — the real test ===')
+    stop: float = time.perf_counter() + secs
+    procs = [
+        mp.Process(target=_burn, args=(stop,))
+        for _ in range(ncpu)
+    ]
+    for p in procs:
+        p.start()
+
+    # skip the initial ~0.6s ramp, then sample steady-state
+    samples: list[int] = []
+    time.sleep(0.6)
+    while time.perf_counter() < stop - 0.2:
+        if (fr := _cur_freqs_mhz()):
+            samples.append(sum(fr) // len(fr))
+        time.sleep(0.3)
+    for p in procs:
+        p.join()
+
+    if not (samples and pkg_max):
+        print('  could not sample cur freq — assume OK')
+        return 0
+
+    sustained: int = sum(samples) // len(samples)
+    frac: float = sustained / pkg_max
+    print(f'  aggregate cur-freq samples: {samples}')
+    print(f'  sustained avg             : {sustained} MHz '
+          f'({frac * 100:.0f}% of {pkg_max} MHz max)')
+
+    if frac < healthy_frac:
+        print(
+            f'\n  ❌ THROTTLED — sustained {sustained}MHz is only '
+            f'{frac * 100:.0f}% of max (< {healthy_frac * 100:.0f}%).\n'
+            f'     Power cap (PPT/STAPM) still engaged. Fixes:\n'
+            f'       - bounce /sys/firmware/acpi/platform_profile\n'
+            f'         (balanced -> performance)\n'
+            f'       - unplug/replug USB-C to re-negotiate PD\n'
+            f'       - ryzenadj to lift STAPM/PPT\n'
+            f'       - else reboot\n'
+            f'     Do NOT bump test budgets — the box is slow, not the code.'
+        )
+        return 1
+
+    print(
+        f'\n  ✅ PERF OK — sustained {sustained}MHz holds '
+        f'{frac * 100:.0f}% of max. Cap looks lifted; safe to run tests.'
+    )
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

scripts/tractor-reap

@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+#
+# SPDX-License-Identifier: AGPL-3.0-or-later
+'''
+`tractor-reap` — SC-polite zombie-subactor reaper +
+optional `/dev/shm/` orphan-segment sweep.
+
+Two cleanup phases (run in order when both are enabled):
+
+1. **process reap** — finds `tractor` subactor processes
+   left alive after a `pytest` (or any tractor-app) run
+   that failed to fully cancel its actor tree, then sends
+   SIGINT with a bounded grace window before escalating
+   to SIGKILL.
+
+2. **shm sweep** (`--shm` / `--shm-only`) — unlinks
+   `/dev/shm/<file>` entries owned by the current uid
+   that no live process has open (mmap'd or fd-held).
+   Needed because `tractor` disables
+   `mp.resource_tracker` (see `tractor.ipc._mp_bs`), so a
+   hard-crashing actor leaves leaked segments that
+   nothing else GCs.
+
+3. **UDS sweep** (`--uds` / `--uds-only`) — unlinks
+   `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` files
+   whose binder pid is dead (or the `1616` registry
+   sentinel). Needed because the IPC server's
+   `os.unlink()` cleanup lives in a `finally:` block
+   that doesn't always run on hard exits (SIGKILL,
+   escaped `KeyboardInterrupt`, etc.) — see issue #452.
+
+Process-reap detection modes (auto-selected):
+
+    --parent <pid>  : descendant-mode — kill procs whose
+                      PPid == <pid>. Use when a parent
+                      is still alive and you want to
+                      scope the sweep precisely (e.g.
+                      CI wrapper calling in from outside
+                      pytest).
+
+    (default)       : orphan-mode — kill procs with
+                      PPid==1 (init-reparented) whose
+                      cwd matches the repo root AND
+                      whose cmdline contains `python`.
+                      The cwd filter is what prevents
+                      sweeping unrelated init-children.
+
+Usage:
+
+    # process reap only (default)
+    scripts/tractor-reap
+
+    # process reap + shm sweep
+    scripts/tractor-reap --shm
+
+    # only the shm sweep, skip process reap
+    scripts/tractor-reap --shm-only
+
+    # process reap + shm + UDS sweep (the works)
+    scripts/tractor-reap --shm --uds
+
+    # only UDS sweep
+    scripts/tractor-reap --uds-only
+
+    # from inside a still-live supervisor
+    scripts/tractor-reap --parent 12345
+
+    # dry-run: list what would be reaped, don't act
+    scripts/tractor-reap -n
+    scripts/tractor-reap --shm --uds -n
+
+'''
+import argparse
+import pathlib
+import subprocess
+import sys
+
+
+def _repo_root() -> pathlib.Path:
+    '''
+    Use `git rev-parse --show-toplevel` when available;
+    fall back to the repo this script lives in.
+
+    '''
+    try:
+        out: str = subprocess.check_output(
+            ['git', 'rev-parse', '--show-toplevel'],
+            stderr=subprocess.DEVNULL,
+            text=True,
+        ).strip()
+        return pathlib.Path(out)
+    except (subprocess.CalledProcessError, FileNotFoundError):
+        return pathlib.Path(__file__).resolve().parent.parent
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        prog='tractor-reap',
+        description=__doc__,
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        '--parent', '-p',
+        type=int,
+        default=None,
+        help='descendant-mode: reap procs with PPid==<pid>',
+    )
+    parser.add_argument(
+        '--grace', '-g',
+        type=float,
+        default=3.0,
+        help='SIGINT grace window in seconds (default 3.0)',
+    )
+    parser.add_argument(
+        '--dry-run', '-n',
+        action='store_true',
+        help='list matched pids/paths but do not signal/unlink',
+    )
+    parser.add_argument(
+        '--shm',
+        action='store_true',
+        help=(
+            'after process reap, also unlink orphaned '
+            '/dev/shm segments owned by the current user '
+            'that no live process is mapping or holding open'
+        ),
+    )
+    parser.add_argument(
+        '--shm-only',
+        action='store_true',
+        help='skip process reap; only do the shm sweep',
+    )
+    parser.add_argument(
+        '--uds',
+        action='store_true',
+        help=(
+            'after process reap, also unlink orphaned '
+            '${XDG_RUNTIME_DIR}/tractor/*.sock files '
+            'whose binder pid is dead (or the 1616 '
+            'registry sentinel). See issue #452.'
+        ),
+    )
+    parser.add_argument(
+        '--uds-only',
+        action='store_true',
+        help='skip process reap + shm; only do the UDS sweep',
+    )
+    args = parser.parse_args()
+    # any *-only flag also skips the process reap phase
+    skip_proc_reap: bool = (
+        args.shm_only
+        or
+        args.uds_only
+    )
+
+    # import lazily so `--help` doesn't require the tractor
+    # package to be importable (e.g. when running from a
+    # shell not inside a venv).
+    repo = _repo_root()
+    sys.path.insert(0, str(repo))
+    from tractor._testing._reap import (
+        find_descendants,
+        find_orphans,
+        find_orphaned_shm,
+        find_orphaned_uds,
+        reap,
+        reap_shm,
+        reap_uds,
+    )
+
+    rc: int = 0
+
+    # --- phase 1: process reap (skipped under --*-only) ---
+    if not skip_proc_reap:
+        if args.parent is not None:
+            pids: list[int] = find_descendants(args.parent)
+            mode: str = f'descendants of PPid={args.parent}'
+        else:
+            pids = find_orphans(repo)
+            mode = f'orphans (PPid=1, cwd={repo})'
+
+        if not pids:
+            print(f'[tractor-reap] no {mode} to reap')
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {mode}:\n  {pids}'
+            )
+        else:
+            _, survivors = reap(pids, grace=args.grace)
+            if survivors:
+                rc = 1
+
+    # --- phase 2: shm sweep (opt-in) ---
+    if args.shm or args.shm_only:
+        leaked: list[str] = find_orphaned_shm()
+        if not leaked:
+            print(
+                '[tractor-reap] no orphaned /dev/shm '
+                'segments to sweep'
+            )
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {len(leaked)} '
+                f'orphaned shm segment(s):\n  {leaked}'
+            )
+        else:
+            _, errors = reap_shm(leaked)
+            if errors:
+                rc = 1
+
+    # --- phase 3: UDS sweep (opt-in) ---
+    if args.uds or args.uds_only:
+        leaked_uds: list[str] = find_orphaned_uds()
+        if not leaked_uds:
+            print(
+                '[tractor-reap] no orphaned UDS sock-files '
+                'to sweep'
+            )
+        elif args.dry_run:
+            print(
+                f'[tractor-reap] dry-run — {len(leaked_uds)} '
+                f'orphaned UDS sock-file(s):\n  {leaked_uds}'
+            )
+        else:
+            _, errors = reap_uds(leaked_uds)
+            if errors:
+                rc = 1
+
+    # exit 0 if everything cleaned cleanly, else 1 — useful
+    # for CI health-check chaining.
+    return rc
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

tests/conftest.py

@@ -22,7 +22,8 @@ from tractor._testing import (
 
 pytest_plugins: list[str] = [
     'pytester',
-    'tractor._testing.pytest',
+    # NOTE, now loaded in `pytest-ini` section of `pyproject.toml`
+    # 'tractor._testing.pytest',
 ]
 
 _ci_env: bool = os.environ.get('CI', False)
@@ -33,15 +34,10 @@ if platform.system() == 'Windows':
     _KILL_SIGNAL = signal.CTRL_BREAK_EVENT
     _INT_SIGNAL = signal.CTRL_C_EVENT
     _INT_RETURN_CODE = 3221225786
-    _PROC_SPAWN_WAIT = 2
 else:
     _KILL_SIGNAL = signal.SIGKILL
     _INT_SIGNAL = signal.SIGINT
     _INT_RETURN_CODE = 1 if sys.version_info < (3, 8) else -signal.SIGINT.value
-    _PROC_SPAWN_WAIT = (
-        2 if _ci_env
-        else 1
-    )
 
 
 no_windows = pytest.mark.skipif(
@@ -124,67 +120,146 @@ def cpu_scaling_factor() -> float:
     return 1.
 
 
-def pytest_addoption(
-    parser: pytest.Parser,
-):
-    # ?TODO? should this be exposed from our `._testing.pytest`
-    # plugin or should we make it more explicit with `--tl` for
-    # tractor logging like we do in other client projects?
-    parser.addoption(
-        "--ll",
-        action="store",
-        dest='loglevel',
-        default='ERROR', help="logging level to set when testing"
-    )
+# session-cached sustained-load throttle multiplier — measured
+# once (lazily) on the first `cpu_perf_headroom()` call. `None`
+# = not-yet-measured.
+_sustained_headroom: float|None = None
 
 
-@pytest.fixture(scope='session', autouse=True)
-def loglevel(request) -> str:
-    import tractor
-    orig = tractor.log._default_loglevel
-    level = tractor.log._default_loglevel = request.config.option.loglevel
-    log = tractor.log.get_console_log(
-        level=level,
-        name='tractor',  # <- enable root logger
-    )
-    log.info(
-        f'Test-harness set runtime loglevel: {level!r}\n'
-    )
-    yield level
-    tractor.log._default_loglevel = orig
-
-
-@pytest.fixture(scope='function')
-def test_log(
-    request,
-    loglevel: str,
-) -> tractor.log.StackLevelAdapter:
+def _measure_sustained_headroom(
+    secs: float = 0.9,
+    # a healthy all-core sustained clock holds AT/ABOVE this
+    # fraction of the package single-core max ceiling (boost sags
+    # under full multi-core load even un-throttled, but not far);
+    # at/above it we assume no throttle and return 1.0.
+    throttle_gate: float = 0.6,
+    max_headroom: float = 4.,
+) -> float:
     '''
-    Deliver a per test-module-fn logger instance for reporting from
-    within actual test bodies/fixtures.
+    One-shot all-core burn returning a latency multiplier
+    (>= 1.0) that reflects *sustained-load* CPU throttle.
 
-    For example this can be handy to report certain error cases from
-    exception handlers using `test_log.exception()`.
+    Catches the firmware/EC power-cap clamp (AMD PPT/STAPM &
+    friends) that pins achieved `scaling_cur_freq` to a fraction
+    of the ceiling under multi-core load while EVERY static knob
+    (`governor`, `scaling_max_freq`, `EPP`, `platform_profile`)
+    still reads "full performance". That cap is INVISIBLE to
+    `cpu_scaling_factor()` and is the gremlin behind mass `trio`
+    deadline-miss failures on byte-identical code — see
+    `scripts/cpu-perf-check`.
+
+    Best-effort: returns 1.0 on non-linux / missing sysfs / any
+    error so it can never break a test run.
 
     '''
-    modname: str = request.function.__module__
-    log = tractor.log.get_logger(
-        name=modname,  # <- enable root logger
-        # pkg_name='tests',
-    )
-    _log = tractor.log.get_console_log(
-        level=loglevel,
-        logger=log,
-        name=modname,
-        # pkg_name='tests',
-    )
-    _log.debug(
-        f'In-test-logging requested\n'
-        f'test_log.name: {log.name!r}\n'
-        f'level: {loglevel!r}\n'
+    import glob
+    import multiprocessing as mp
 
-    )
-    yield _log
+    def _read_mhz(path: str) -> int|None:
+        try:
+            return int(open(path).read()) // 1000
+        except OSError:
+            return None
+
+    try:
+        maxs: list[int] = [
+            v for f in glob.glob(
+                '/sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_max_freq'
+            )
+            if (v := _read_mhz(f)) is not None
+        ]
+        pkg_max: int = max(maxs) if maxs else 0
+        if not pkg_max:
+            return 1.
+
+        def _burn(stop: float) -> None:
+            x: int = 1
+            while time.perf_counter() < stop:
+                x += x * x ^ 0x5
+
+        # explicit `fork` ctx so we're immune to whatever global
+        # mp start-method tractor/the suite may have set (`spawn`
+        # would re-exec + re-import 24x — slow and pointless here).
+        ctx = mp.get_context('fork')
+        ncpu: int = os.cpu_count() or 1
+        stop: float = time.perf_counter() + secs
+        procs = [
+            ctx.Process(target=_burn, args=(stop,), daemon=True)
+            for _ in range(ncpu)
+        ]
+        for p in procs:
+            p.start()
+
+        # skip the ~0.4s boost window so we sample the steady
+        # state AFTER any power-cap has engaged.
+        samples: list[int] = []
+        time.sleep(0.4)
+        while time.perf_counter() < stop - 0.1:
+            curs: list[int] = [
+                v for f in glob.glob(
+                    '/sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq'
+                )
+                if (v := _read_mhz(f)) is not None
+            ]
+            if curs:
+                samples.append(sum(curs) // len(curs))
+            time.sleep(0.15)
+        for p in procs:
+            p.join()
+
+        if not samples:
+            return 1.
+        frac: float = (sum(samples) // len(samples)) / pkg_max
+        # below the gate we read it as a power-cap throttle. The
+        # spawn/IPC/fork-bound work these budgets guard slows ~1:1
+        # with the achieved-vs-max freq ratio, so compensate by the
+        # FULL inverse fraction (a boost-discounted factor
+        # under-shoots and still trips the marginal cases).
+        if frac >= throttle_gate:
+            return 1.
+        return min(max_headroom, 1. / frac)
+
+    except Exception:
+        return 1.
+
+
+def cpu_perf_headroom() -> float:
+    '''
+    Latency-headroom multiplier (>= 1.0) covering BOTH cpu-perf
+    throttle classes — multiply a test's deadline by it, e.g.
+    `timeout *= cpu_perf_headroom()`:
+
+      - static cpu-freq scaling — via `cpu_scaling_factor()`
+        (governor/policy lowered the `scaling_max_freq` ceiling).
+
+      - sustained-load power-cap throttle — via
+        `_measure_sustained_headroom()` (firmware/EC PPT/STAPM
+        clamps achieved freq under load while every static knob
+        reads "performance"; INVISIBLE to the static check). This
+        is the gremlin behind mass `trio` deadline-miss failures
+        on unchanged code — see
+        `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
+
+    The sustained probe runs ONCE per session (cached); the cost
+    is a ~0.9s all-core burn on first call only.
+
+    '''
+    global _sustained_headroom
+    static: float = cpu_scaling_factor()
+    if _non_linux:
+        return static
+    if _sustained_headroom is None:
+        _sustained_headroom = _measure_sustained_headroom()
+    return max(static, _sustained_headroom)
+
+
+# NOTE, the `--ll`/`--tl` CLI flags + the `loglevel`, `test_log`
+# and `testing_pkg_name` fixtures have been factored into the
+# `tractor._testing.pytest` plugin (loaded via the `-p` entry in
+# `pyproject.toml`'s `[tool.pytest.ini_options]`) so downstream
+# consuming projects (eg. `modden`) inherit them for free. The
+# plugin's `testing_pkg_name` fixture defaults to `'tractor'`, so
+# this suite keeps treating `--ll` as the runtime loglevel.
 
 
 @pytest.fixture(scope='session')
@@ -236,107 +311,14 @@ def sig_prog(
     assert ret
 
 
-# TODO: factor into @cm and move to `._testing`?
-@pytest.fixture
-def daemon(
-    debug_mode: bool,
-    loglevel: str,
-    testdir: pytest.Pytester,
-    reg_addr: tuple[str, int],
-    tpt_proto: str,
-    ci_env: bool,
-    test_log: tractor.log.StackLevelAdapter,
-
-) -> subprocess.Popen:
-    '''
-    Run a daemon root actor as a separate actor-process tree and
-    "remote registrar" for discovery-protocol related tests.
-
-    '''
-    if loglevel in ('trace', 'debug'):
-        # XXX: too much logging will lock up the subproc (smh)
-        loglevel: str = 'info'
-
-    code: str = (
-        "import tractor; "
-        "tractor.run_daemon([], "
-        "registry_addrs={reg_addrs}, "
-        "enable_transports={enable_tpts}, "
-        "debug_mode={debug_mode}, "
-        "loglevel={ll})"
-    ).format(
-        reg_addrs=str([reg_addr]),
-        enable_tpts=str([tpt_proto]),
-        ll="'{}'".format(loglevel) if loglevel else None,
-        debug_mode=debug_mode,
-    )
-    cmd: list[str] = [
-        sys.executable,
-        '-c', code,
-    ]
-    # breakpoint()
-    kwargs = {}
-    if platform.system() == 'Windows':
-        # without this, tests hang on windows forever
-        kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
-
-    proc: subprocess.Popen = testdir.popen(
-        cmd,
-        **kwargs,
-    )
-
-    # TODO! we should poll for the registry socket-bind to take place
-    # and only once that's done yield to the requester!
-    # -[ ] TCP: use the `._root.open_root_actor()`::`ping_tpt_socket()`
-    #      closure!
-    # -[ ] UDS: can we do something similar for 'pinging" the
-    #     file-socket?
-    #
-    global _PROC_SPAWN_WAIT
-    # UDS sockets are **really** fast to bind()/listen()/connect()
-    # so it's often required that we delay a bit more starting
-    # the first actor-tree..
-    if tpt_proto == 'uds':
-        _PROC_SPAWN_WAIT += 1.6
-
-    if _non_linux and ci_env:
-        _PROC_SPAWN_WAIT += 1
-
-    # XXX, allow time for the sub-py-proc to boot up.
-    # !TODO, see ping-polling ideas above!
-    time.sleep(_PROC_SPAWN_WAIT)
-
-    assert not proc.returncode
-    yield proc
-    sig_prog(proc, _INT_SIGNAL)
-
-    # XXX! yeah.. just be reaaal careful with this bc sometimes it
-    # can lock up on the `_io.BufferedReader` and hang..
-    stderr: str = proc.stderr.read().decode()
-    stdout: str = proc.stdout.read().decode()
-    if (
-        stderr
-        or
-        stdout
-    ):
-        print(
-            f'Daemon actor tree produced output:\n'
-            f'{proc.args}\n'
-            f'\n'
-            f'stderr: {stderr!r}\n'
-            f'stdout: {stdout!r}\n'
-        )
-
-    if (rc := proc.returncode) != -2:
-        msg: str = (
-            f'Daemon actor tree was not cancelled !?\n'
-            f'proc.args: {proc.args!r}\n'
-            f'proc.returncode: {rc!r}\n'
-        )
-        if rc < 0:
-            raise RuntimeError(msg)
-
-        test_log.error(msg)
+# NOTE, the `daemon` fixture (+ its `_wait_for_daemon_ready`
+# helper + the post-yield teardown drain logic) has been
+# moved to `tests/discovery/conftest.py` since 100% of its
+# consumers are discovery-protocol tests now living under
+# that subdir. See:
+# - `tests/discovery/test_multi_program.py`
+# - `tests/discovery/test_registrar.py`
+# - `tests/discovery/test_tpt_bind_addrs.py`
 
 
 # @pytest.fixture(autouse=True)

tests/devx/conftest.py

@@ -4,6 +4,8 @@
 '''
 from __future__ import annotations
 import platform
+import os
+import re
 import signal
 import time
 from typing import (
@@ -56,6 +58,7 @@ type PexpectSpawner = Callable[
 @pytest.fixture
 def spawn(
     start_method: str,
+    loglevel: str,
     testdir: pytest.Pytester,
     reg_addr: tuple[str, int],
 
@@ -65,9 +68,19 @@ def spawn(
     run an `./examples/..` script by name.
 
     '''
-    if start_method != 'trio':
+    supported_spawners: set[str] = {
+        'trio',
+        # `examples/debugging/<script>.py` picks up the spawn
+        # backend via the `TRACTOR_SPAWN_METHOD` env-var which
+        # is honored inside `tractor._root.open_root_actor()`,
+        # so no per-script edits are required.
+        'main_thread_forkserver',
+        'subint_forkserver',
+    }
+    if start_method not in supported_spawners:
         pytest.skip(
-            '`pexpect` based tests only supported on `trio` backend'
+            f'`pexpect` based tests NOT supported on spawning-backend: {start_method!r}\n'
+            f'supported-spawners: {supported_spawners!r}'
         )
 
     def unset_colors():
@@ -79,21 +92,64 @@ def spawn(
         https://docs.python.org/3/using/cmdline.html#using-on-controlling-color
 
         '''
-        import os
         # disable colored tbs
         os.environ['PYTHON_COLORS'] = '0'
         # disable all ANSI color output
         # os.environ['NO_COLOR'] = '1'
+        # ?TODO, doesn't seem to disable prompt color
+        # for `pdbp`?
+
+    def set_spawn_method(
+        start_method: str,
+    ):
+        '''
+        Drive the actor-spawn backend inside the spawned
+        `examples/debugging/<script>.py` subproc via env-var
+        (consumed by `tractor._root.open_root_actor()`),
+        without requiring per-script CLI plumbing.
+
+        '''
+        os.environ['TRACTOR_SPAWN_METHOD'] = start_method
+
+    def set_loglevel(
+        loglevel: str|None,
+    ):
+        '''
+        Forward the test-suite parametrized `loglevel` into the
+        spawned `examples/debugging/<script>.py` subproc via
+        env-var (consumed by `tractor._root.open_root_actor()`),
+        so console verbosity can be cranked or silenced from
+        the test harness without per-script edits.
+
+        '''
+        if loglevel:
+            os.environ['TRACTOR_LOGLEVEL'] = loglevel
+        else:
+            os.environ.pop('TRACTOR_LOGLEVEL', None)
 
     spawned: PexpectSpawner|None = None
 
     def _spawn(
         cmd: str,
         expect_timeout: float = 4,
+        start_method: str = start_method,
+        loglevel: str|None = None,
         **mkcmd_kwargs,
     ) -> pty_spawn.spawn:
+        '''
+        Inner closure handed to consumer tests to invoke
+        `pytest.Pytester.spawn`
+
+        '''
         nonlocal spawned
         unset_colors()
+        set_spawn_method(start_method=start_method)
+        set_loglevel(
+            loglevel=loglevel,
+            # ?TODO^ when should this be set by `--ll <level>` ?
+            # by default we apply 'error' but there should be a diff
+            # vs. when the flag IS NOT passed?
+        )
         spawned = testdir.spawn(
             cmd=mk_cmd(
                 cmd,
@@ -137,6 +193,14 @@ def spawn(
         if ptyproc.isalive():
             ptyproc.kill(signal.SIGKILL)
 
+    # Scope our env-var mutations to this single fixture invocation
+    # — both `TRACTOR_SPAWN_METHOD` and `TRACTOR_LOGLEVEL` are
+    # honored by `tractor._root.open_root_actor()` so leaking them
+    # past this test could inadvertently re-route a later in-process
+    # tractor test's spawn-backend / loglevel.
+    os.environ.pop('TRACTOR_SPAWN_METHOD', None)
+    os.environ.pop('TRACTOR_LOGLEVEL', None)
+
     # TODO? ensure we've cleaned up any UDS-paths?
     # breakpoint()
 
@@ -146,24 +210,40 @@ def spawn(
     ids='ctl-c={}'.format,
 )
 def ctlc(
-    request,
+    request: pytest.FixtureRequest,
     ci_env: bool,
-
+    start_method: str,
 ) -> bool:
+    '''
+    Parametrize and optionally skip tests which handle
+    ctlc-in-`pdbp`-REPL testing scenarios; certain spawners and actor-tree depths
+    cope very poorly with this..
 
-    use_ctlc = request.param
+    In particular the spawning backends from `multiprocessing` are
+    fragile, as can be the default `trio` spawner under certain
+    conditions where SIGINT is relayed down the entire subproc tree.
 
+    '''
+    use_ctlc: bool = request.param
     node = request.node
     markers = node.own_markers
     for mark in markers:
-        if mark.name == 'has_nested_actors':
+        if (
+            mark.name == 'has_nested_actors'
+            and
+            start_method not in {
+                # TODO, any spawners we should try again?
+                # - [ ] 'trio' but WITHOUT the SIGINT handler setup
+                #      per subproc?
+                # 'main_thread_forkserver',
+            }
+        ):
             pytest.skip(
                 f'Test {node} has nested actors and fails with Ctrl-C.\n'
                 f'The test can sometimes run fine locally but until'
                 ' we solve' 'this issue this CI test will be xfail:\n'
                 'https://github.com/goodboy/tractor/issues/320'
             )
-
         if (
             mark.name == 'ctlcs_bish'
             and
@@ -190,13 +270,10 @@ def ctlc(
 
 def expect(
     child,
-
-    # normally a `pdb` prompt by default
-    patt: str,
-
+    patt: str,  # often a `pdbp`-prompt
     **kwargs,
 
-) -> None:
+) -> str:
     '''
     Expect wrapper that prints last seen console
     data before failing.
@@ -207,6 +284,8 @@ def expect(
             patt,
             **kwargs,
         )
+        before = str(child.before.decode())
+        return before
     except TIMEOUT:
         before = str(child.before.decode())
         print(before)
@@ -216,6 +295,26 @@ def expect(
 PROMPT = r"\(Pdb\+\)"
 
 
+# Strip terminal color / ANSI-VT100 escape sequences so
+# substring matching against REPL + traceback output stays
+# robust to color leakage — Python 3.13's colored tracebacks,
+# `pdbp`'s pygments highlighting, etc. — even when
+# `PYTHON_COLORS=0` (set in the `spawn` fixture) isn't honored
+# by every renderer in the spawned subproc.
+# Regex per https://stackoverflow.com/a/14693789
+_ansi_re: re.Pattern = re.compile(
+    r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])'
+)
+
+
+def ansi_strip(text: str) -> str:
+    '''
+    Remove ANSI/VT100 escape sequences from `text`.
+
+    '''
+    return _ansi_re.sub('', text)
+
+
 def in_prompt_msg(
     child: SpawnBase,
     parts: list[str],
@@ -235,7 +334,7 @@ def in_prompt_msg(
     '''
     __tracebackhide__: bool = False
 
-    before: str = str(child.before.decode())
+    before: str = ansi_strip(str(child.before.decode()))
     for part in parts:
         if part not in before:
             if pause_on_false:
@@ -255,16 +354,19 @@ def in_prompt_msg(
     return True
 
 
-# TODO: todo support terminal color-chars stripping so we can match
-# against call stack frame output from the the 'll' command the like!
-# -[ ] SO answer for stipping ANSI codes: https://stackoverflow.com/a/14693789
+# NB: color-char stripping (so we can match against call-stack
+# frame output from the `ll` command and the like) is handled by
+# `ansi_strip()` applied inside `in_prompt_msg()` + below.
 def assert_before(
     child: SpawnBase,
     patts: list[str],
-
     **kwargs,
+) -> str:
+    '''
+    Assert a patter is in `child.before.decode() -> str`,
+    return the full `.before` output on success.
 
-) -> None:
+    '''
     __tracebackhide__: bool = False
 
     assert in_prompt_msg(
@@ -275,7 +377,8 @@ def assert_before(
         err_on_false=True,
         **kwargs
     )
-    return str(child.before.decode())
+    before: str = ansi_strip(str(child.before.decode()))
+    return before
 
 
 def do_ctlc(

tests/devx/test_debugger.py

@@ -24,6 +24,7 @@ from pexpect.exceptions import (
     TIMEOUT,
     EOF,
 )
+import tractor
 
 from .conftest import (
     do_ctlc,
@@ -343,6 +344,7 @@ def test_subactor_breakpoint(
 def test_multi_subactors(
     spawn,
     ctlc: bool,
+    set_fork_aware_capture,
 ):
     '''
     Multiple subactors, both erroring and
@@ -487,11 +489,12 @@ def test_multi_subactors(
 def test_multi_daemon_subactors(
     spawn,
     loglevel: str,
-    ctlc: bool
+    ctlc: bool,
+    set_fork_aware_capture,
 ):
     '''
-    Multiple daemon subactors, both erroring and breakpointing within a
-    stream.
+    Multiple daemon subactors, both erroring and breakpointing within
+    a stream.
 
     '''
     non_linux = _non_linux
@@ -604,7 +607,10 @@ def test_multi_daemon_subactors(
             child,
             bp_forev_parts,
         )
-    except AssertionError:
+    except (
+        # AssertionError,  # TODO? rm since never raised?
+        ValueError,
+    ):
         before: str = assert_before(
             child,
             name_error_parts,
@@ -765,6 +771,8 @@ def test_multi_subactors_root_errors(
 def test_multi_nested_subactors_error_through_nurseries(
     ci_env: bool,
     spawn: PexpectSpawner,
+    is_forking_spawner: bool,
+    test_log: tractor.log.StackLevelAdapter,
 
     # TODO: address debugger issue for nested tree:
     # https://github.com/goodboy/tractor/issues/320
@@ -781,16 +789,25 @@ def test_multi_nested_subactors_error_through_nurseries(
     # A test (below) has now been added to explicitly verify this is
     # fixed.
 
-    child = spawn('multi_nested_subactors_error_up_through_nurseries')
+    child = spawn(
+        'multi_nested_subactors_error_up_through_nurseries',
+        loglevel='pdb',
+    )
+    last_send_char: str|None = None
 
-    # timed_out_early: bool = False
+    # inflate pexpect waits under CPU throttle — incl. the
+    # sustained-load power-cap invisible to static freq reads — so
+    # a slow-to-boot child REPL doesn't trip a false `TIMEOUT`.
+    # See `scripts/cpu-perf-check`.
+    from ..conftest import cpu_perf_headroom
+    headroom: float = cpu_perf_headroom()
 
     for (
         i,
         send_char,
     ) in enumerate(itertools.cycle(['c', 'q'])):
 
-        timeout: float = -1
+        timeout: float = child.timeout
         if (
             _non_linux
             and
@@ -803,49 +820,85 @@ def test_multi_nested_subactors_error_through_nurseries(
         elif i == 0:
             timeout = 5
 
+        # XXX forking backends may take longer due to
+        # determinstic IPC cancellation.
+        if is_forking_spawner:
+            timeout += 4
+
+        if headroom != 1.:
+            timeout *= headroom
+
         try:
             child.expect(
                 PROMPT,
                 timeout=timeout,
             )
+            delay: float = 0.1
+            test_log.info('Sleeping {delay!r} before next send-chart..')
+            time.sleep(delay)
+            last_send_char: str = send_char
             child.sendline(send_char)
-            time.sleep(0.01)
+            time.sleep(delay)
 
+        # script finally exited with tb on console.
         except EOF:
+            test_log.info(
+                f'Breaking from send-char loop'
+                f'last_send_char: {last_send_char!r}\n'
+            )
             break
 
+    # boxed source errors
+    expect_patts: list[str] = [
+        "NameError: name 'doggypants' is not defined",
+        "tractor._exceptions.RemoteActorError:",
+        "('name_error'",
+
+        # first level subtrees
+        # "tractor._exceptions.RemoteActorError: ('spawner0'",
+        "src_uid=('spawner0'",
+
+        # "tractor._exceptions.RemoteActorError: ('spawner1'",
+
+        # propagation of errors up through nested subtrees
+        # "tractor._exceptions.RemoteActorError: ('spawn_until_0'",
+        # "tractor._exceptions.RemoteActorError: ('spawn_until_1'",
+        # "tractor._exceptions.RemoteActorError: ('spawn_until_2'",
+        # ^-NOTE-^ old RAE repr, new one is below with a field
+        # showing the src actor's uid.
+        "src_uid=('spawn_until_2'",
+    ]
+    # XXX, I HAVE NO IDEA why these patts only show on the
+    # `trio`-spawner but it seems to have something to do with
+    # what gets dumped in prior-prompt latches somehow??
+    # TODO for claude, explain and or work through how this is
+    # happening but ONLY WHEN RUN FROM THE TEST, bc when i try to
+    # run the test script manually the correct output ALWAYS seems
+    # to be in the last `str(child.before.decode())` output !?!?
+    if (
+        not is_forking_spawner
+        and
+        last_send_char == 'q'
+    ):
+        expect_patts += [
+            # expect the pdb-quit exc.
+            "bdb.BdbQuit",
+            # BUT WHY these dude!?
+            "src_uid=('spawn_until_0'",
+            "relay_uid=('spawn_until_1'",
+        ]
+
     assert_before(
         child,
-        [ # boxed source errors
-            "NameError: name 'doggypants' is not defined",
-            "tractor._exceptions.RemoteActorError:",
-            "('name_error'",
-            "bdb.BdbQuit",
-
-            # first level subtrees
-            # "tractor._exceptions.RemoteActorError: ('spawner0'",
-            "src_uid=('spawner0'",
-
-            # "tractor._exceptions.RemoteActorError: ('spawner1'",
-
-            # propagation of errors up through nested subtrees
-            # "tractor._exceptions.RemoteActorError: ('spawn_until_0'",
-            # "tractor._exceptions.RemoteActorError: ('spawn_until_1'",
-            # "tractor._exceptions.RemoteActorError: ('spawn_until_2'",
-            # ^-NOTE-^ old RAE repr, new one is below with a field
-            # showing the src actor's uid.
-            "src_uid=('spawn_until_0'",
-            "relay_uid=('spawn_until_1'",
-            "src_uid=('spawn_until_2'",
-        ]
+        expect_patts,
     )
+    expect(child, EOF)
 
 
-@pytest.mark.timeout(15)
+# @pytest.mark.timeout(15)
 @has_nested_actors
 def test_root_nursery_cancels_before_child_releases_tty_lock(
     spawn,
-    start_method,
     ctlc: bool,
 ):
     '''
@@ -1144,7 +1197,12 @@ def test_shield_pause(
             "('cancelled_before_pause'",  # actor name
             _repl_fail_msg,
             "trio.Cancelled",
-            "raise Cancelled._create()",
+            # trio >=0.30 raises via a multi-line
+            # `raise Cancelled._create(source=.., reason=..,
+            #  source_task=..)` (cancel-reason metadata), so
+            # match the open-paren form only, NOT the legacy
+            # bare `()`.
+            "raise Cancelled._create(",
 
             # we should be handling a taskc inside
             # the first `.port_mortem()` sin-shield!
@@ -1162,7 +1220,12 @@ def test_shield_pause(
             "('root'",  # actor name
             _repl_fail_msg,
             "trio.Cancelled",
-            "raise Cancelled._create()",
+            # trio >=0.30 raises via a multi-line
+            # `raise Cancelled._create(source=.., reason=..,
+            #  source_task=..)` (cancel-reason metadata), so
+            # match the open-paren form only, NOT the legacy
+            # bare `()`.
+            "raise Cancelled._create(",
 
             # handling a taskc inside the first unshielded
             # `.port_mortem()`.
@@ -1187,7 +1250,11 @@ def test_ctxep_pauses_n_maybe_ipc_breaks(
     mashed and zombie reaper kills sub with no hangs.
 
     '''
-    child = spawn('subactor_bp_in_ctx')
+    child = spawn(
+        'subactor_bp_in_ctx',
+        loglevel='devx'
+        # ^XXX REQUIRED for below patt matching!
+    )
     child.expect(PROMPT)
 
     # 3 iters for the `gen()` pause-points
@@ -1277,7 +1344,11 @@ def test_crash_handling_within_cancelled_root_actor(
     call.
 
     '''
-    child = spawn('root_self_cancelled_w_error')
+    child = spawn(
+        'root_self_cancelled_w_error',
+        loglevel='cancel',
+        # ^XXX REQUIRED for below patt matching!
+    )
     child.expect(PROMPT)
 
     assert_before(

tests/devx/test_pause_from_non_trio.py

@@ -63,19 +63,31 @@ def test_pause_from_sync(
     `examples/debugging/sync_bp.py`
 
     '''
-    child = spawn('sync_bp')
+    # XXX required for `breakpoint()` overload and
+    # thus`tractor.devx.pause_from_sync()`.
+    pytest.importorskip('greenback')
+    child = spawn(
+        'sync_bp',
+        loglevel='pdb',  # XXX pattern matching
+    )
 
     # first `sync_pause()` after nurseries open
     child.expect(PROMPT)
-    assert_before(
+    _before: str = assert_before(
         child,
         [
-            # pre-prompt line
-            _pause_msg,
-            "<Task '__main__.main'",
+            # devx-loglevel
+            # "imported <module 'greenback' from",
+            # "successfully scheduled `._pause()` in `trio` thread on behalf of <Task",
+
+            _pause_msg,  # pre-prompt line
             "('root'",
+            "<Task '__main__.main'",
+            "tractor.pause_from_sync()",
         ]
     )
+    # XXX `enable_stack_on_sig=False` in script
+    assert 'stackscope' not in _before
     if ctlc:
         do_ctlc(child)
         # ^NOTE^ subactor not spawned yet; don't need extra delay.
@@ -85,18 +97,18 @@ def test_pause_from_sync(
     # first `await tractor.pause()` inside `p.open_context()` body
     child.expect(PROMPT)
 
-    # XXX shouldn't see gb loaded message with PDB loglevel!
-    # assert not in_prompt_msg(
-    #     child,
-    #     ['`greenback` portal opened!'],
-    # )
     # should be same root task
     assert_before(
         child,
         [
+            # XXX should see gb loaded with devx-loglevel.
+            # "`greenback` portal opened!",
+            # "Activated `greenback` for `tractor.pause_from_sync()` support!",
+
             _pause_msg,
-            "<Task '__main__.main'",
             "('root'",
+            "<Task '__main__.main'",
+            "tractor.pause()",
         ]
     )
 
@@ -127,17 +139,17 @@ def test_pause_from_sync(
     # `Lock.acquire()`-ed
     # (NOT both, which will result in REPL clobbering!)
     attach_patts: dict[str, list[str]] = {
-        'subactor': [
-            "'start_n_sync_pause'",
-            "('subactor'",
+        "|_<Task 'start_n_sync_pause'": [
+            "|_('subactor'",
+            "tractor.pause_from_sync()",
         ],
-        'inline_root_bg_thread': [
-            "<Thread(inline_root_bg_thread",
+        "|_<Thread(inline_root_bg_thread": [
             "('root'",
+            "breakpoint(hide_tb=hide_tb)",
         ],
-        'start_soon_root_bg_thread': [
-            "<Thread(start_soon_root_bg_thread",
-            "('root'",
+        "|_<Thread(start_soon_root_bg_thread": [
+            "|_('root'",
+            "tractor.pause_from_sync()",
         ],
     }
     conts: int = 0  # for debugging below matching logic on failure
@@ -260,6 +272,9 @@ def test_sync_pause_from_aio_task(
     `examples/debugging/asycio_bp.py`
 
     '''
+    # XXX required for `breakpoint()` overload and
+    # thus`tractor.devx.pause_from_sync()`.
+    pytest.importorskip('greenback')
     child = spawn('asyncio_bp')
 
     # RACE on whether trio/asyncio task bps first

tests/devx/test_proctitle.py

@@ -0,0 +1,178 @@
+'''
+Tests for `tractor.devx._proctitle` (per-actor `setproctitle`)
+and the intrinsic-signal sub-actor detection in
+`tractor._testing._reap`.
+
+The proctitle is set in `tractor._child._actor_child_main()`
+after `Actor` construction, so any spawned sub-actor process
+should:
+
+  - have `argv[0]` (== `/proc/<pid>/cmdline`) start with
+    `<_def_prefix>[<aid.reprol()>]` (currently `_subactor[…]`)
+  - have `/proc/<pid>/comm` start with `<_def_prefix>[`
+    (kernel truncates to ~15 bytes)
+  - be detected as a tractor sub-actor by
+    `_is_tractor_subactor(pid)` via the cmdline marker.
+
+`set_actor_proctitle()` itself is also unit-tested in-process
+to verify the format string.
+
+'''
+from __future__ import annotations
+import platform
+
+import psutil
+import pytest
+import trio
+import tractor
+
+from tractor.runtime._runtime import Actor
+from tractor.devx._proctitle import (
+    set_actor_proctitle,
+    _def_prefix,
+)
+from tractor._testing._reap import (
+    _is_tractor_subactor,
+    _read_cmdline,
+    _read_comm,
+)
+
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+def test_set_actor_proctitle_format():
+    '''
+    `set_actor_proctitle()` returns the canonical
+    `<_def_prefix>[<aid.reprol()>]` form (currently
+    `_subactor[…]`) and actually mutates the running
+    proc's title.
+
+    '''
+    pytest.importorskip(
+        'setproctitle',
+        reason='`setproctitle` is an optional runtime dep',
+    )
+    import setproctitle
+
+    # save + restore so we don't pollute pytest's own title
+    saved: str = setproctitle.getproctitle()
+    try:
+        actor = Actor(
+            name='unit_test_actor',
+            uuid='1027301b-a0e3-430e-8806-a5279f21abe6',
+        )
+        title: str = set_actor_proctitle(actor)
+
+        # canonical wrapping: `<_def_prefix>[<aid.reprol()>]`.
+        # We source BOTH the prefix (`_def_prefix`) and the
+        # runtime-computed `reprol()` rather than hard-coding,
+        # so the test stays decoupled from the prefix shape
+        # (flipped to `_subactor` in `3a45dbd5`) AND from
+        # `Aid.reprol()`'s internal format (currently
+        # `<name>@<pid>`, but could evolve).
+        expected: str = f'{_def_prefix}[{actor.aid.reprol()}]'
+        assert title == expected
+        # sanity: the actor's name must be in the title
+        # somewhere (so a future `reprol()` change that
+        # drops the name is also caught).
+        assert 'unit_test_actor' in title
+
+        # actually set on the running proc
+        assert setproctitle.getproctitle() == title
+
+    finally:
+        setproctitle.setproctitle(saved)
+
+
+@pytest.mark.skipif(
+    _non_linux,
+    reason=(
+        'detection helpers read `/proc/<pid>/{cmdline,comm}` '
+        'which is Linux-specific'
+    ),
+)
+def test_subactor_proctitle_visible_via_proc():
+    '''
+    Spawn a sub-actor and verify its proc-title is visible
+    via both `/proc/<pid>/cmdline` AND `/proc/<pid>/comm`,
+    AND that `_is_tractor_subactor()` correctly identifies
+    it.
+
+    '''
+    pytest.importorskip('setproctitle')
+
+    async def main() -> dict:
+        async with tractor.open_nursery() as an:
+            portal = await an.start_actor('proctitle_boi')
+            # let the child finish setproctitle in
+            # `_actor_child_main`
+            await trio.sleep(0.3)
+
+            # the sub-actor's pid is on the portal's chan
+            # repr; psutil-walk `me.children()` is simpler.
+            me = psutil.Process()
+            sub_pids: list[int] = [
+                p.pid for p in me.children(recursive=True)
+            ]
+            assert sub_pids, (
+                'expected at least one spawned sub-actor pid'
+            )
+
+            results: dict = {}
+            for pid in sub_pids:
+                results[pid] = {
+                    'cmdline': _read_cmdline(pid),
+                    'comm': _read_comm(pid),
+                    'is_tractor': _is_tractor_subactor(pid),
+                }
+
+            await portal.cancel_actor()
+            return results
+
+    found: dict = trio.run(main)
+
+    # at least one of the spawned procs should match the
+    # `proctitle_boi` actor we started; assert the proc-
+    # title shape on it specifically.
+    matched: list[tuple[int, dict]] = [
+        (pid, info)
+        for pid, info in found.items()
+        if 'proctitle_boi' in info['cmdline']
+    ]
+    assert matched, (
+        f'no sub-actor pid had a `proctitle_boi` cmdline; '
+        f'all={found}'
+    )
+
+    pid, info = matched[0]
+    # canonical proctitle prefix in cmdline (full form);
+    # prefix sourced from `_def_prefix` so it tracks the
+    # `3a45dbd5` flip (`tractor[` -> `_subactor[`).
+    assert info['cmdline'].startswith(f'{_def_prefix}[proctitle_boi@'), (
+        f'cmdline missing `{_def_prefix}[proctitle_boi@…]` prefix: '
+        f'{info["cmdline"]!r}'
+    )
+    # comm is kernel-truncated to ~15 bytes — just check the
+    # `<_def_prefix>[` prefix made it.
+    assert info['comm'].startswith(f'{_def_prefix}['), (
+        f'comm missing `{_def_prefix}[` prefix: {info["comm"]!r}'
+    )
+    # intrinsic-signal detector should match.
+    assert info['is_tractor'] is True
+
+
+@pytest.mark.skipif(
+    _non_linux,
+    reason='reads /proc/<pid>/{cmdline,comm}',
+)
+def test_is_tractor_subactor_negative():
+    '''
+    `_is_tractor_subactor()` returns False for non-tractor
+    procs (e.g. the pytest test-runner pid itself, which
+    is `python -m pytest …` — no `tractor[` proctitle, no
+    `tractor._child` cmdline).
+
+    '''
+    import os
+    assert _is_tractor_subactor(os.getpid()) is False

tests/devx/test_tooling.py

@@ -21,6 +21,7 @@ import os
 import signal
 import time
 from typing import (
+    Callable,
     TYPE_CHECKING,
 )
 
@@ -47,7 +48,12 @@ if TYPE_CHECKING:
 
 @no_macos
 def test_shield_pause(
-    spawn: PexpectSpawner,
+    spawn: Callable[
+        ...,
+        PexpectSpawner,
+    ],
+    start_method: str,
+    request: pytest.FixtureRequest,
 ):
     '''
     Verify the `tractor.pause()/.post_mortem()` API works inside an
@@ -55,8 +61,10 @@ def test_shield_pause(
     next checkpoint wherein the cancelled will get raised.
 
     '''
-    child = spawn(
-        'shield_hang_in_sub'
+    child: PexpectSpawner = spawn(
+        'shield_hang_in_sub',
+        loglevel='devx',
+        # ^XXX REQUIRED for below patt matching!
     )
     expect(
         child,
@@ -86,38 +94,82 @@ def test_shield_pause(
         # end-of-tree delimiter
         "end-of-\('root'",
     )
-    assert_before(
+    _before: str = assert_before(
         child,
         [
             # 'Srying to dump `stackscope` tree..',
             # 'Dumping `stackscope` tree for actor',
             "('root'",  # uid line
 
-            # TODO!? this used to show?
+            # TODO!? this in-task-code used to show??
             # -[ ] mk reproducable for @oremanj?
+            # => SOLVED? by our `trio_token.run_sync_soon()`
+            #    approach?
             #
             # parent block point (non-shielded)
             # 'await trio.sleep_forever()  # in root',
         ]
     )
-    expect(
-        child,
-        # end-of-tree delimiter
-        "end-of-\('hanger'",
-    )
-    assert_before(
-        child,
-        [
-            # relay to the sub should be reported
-            'Relaying `SIGUSR1`[10] to sub-actor',
 
-            "('hanger'",  # uid line
+    # NOTE, hierarchical-ordering invariant restored by
+    # `_dump_then_relay` (co-scheduled dump+relay on the
+    # trio loop, see `tractor.devx._stackscope`): the
+    # parent's full task-tree prints BEFORE the 'Relaying
+    # `SIGUSR1`' log msg, which prints BEFORE any sub-
+    # actor receives the signal and dumps its own tree.
+    # So the relay log appears BETWEEN `end-of-('root'`
+    # (above) and `end-of-('hanger'` (below).
+    handle_out_of_order: bool = False
+
+    # XXX, when capfd is NOT used we don't expect to
+    # see the logging output from the subactor.
+    if (no_capfd := (start_method in [
+            'main_thread_forkserver',
+        ])
+    ):
+        opts = request.config.option
+        assert opts.spawn_backend == start_method
+        # ?XXX? i guess the `testdir` fixture "pretends to" reset
+        # this to the default 'fd'??
+        # assert opts.capture in [
+        #     'sys',
+        #     'no',
+        # ]
+
+    if (
+        handle_out_of_order
+        and
+        "end-of-('hanger'" in _before
+    ):
+         assert "('hanger'" in _before
+         assert 'Relaying `SIGUSR1`[10] to sub-actor' in _before
+
+    else:
+        _before = expect(
+            child,
+            'Relaying `SIGUSR1`\\[10\\] to sub-actor',
+        )
+        # _before: str = assert_before(
+        #     child,
+        #     ["('hanger'",]  # uid line
+        # )
+        if not no_capfd:
+            expect(
+                child,
+                # end-of-subactor's-tree delimiter
+                "end-of-\('hanger'",
+            )
+            _before: str = assert_before(
+                child,
+                [
+                    "('hanger'",  # uid line
+
+                    # TODO!? SEE ABOVE
+                    # hanger LOC where it's shield-halted
+                    # 'await trio.sleep_forever()  # in subactor',
+                ]
+            )
 
-            # TODO!? SEE ABOVE
-            # hanger LOC where it's shield-halted
-            # 'await trio.sleep_forever()  # in subactor',
-        ]
-    )
 
     # simulate the user sending a ctl-c to the hanging program.
     # this should result in the terminator kicking in since
@@ -133,14 +185,19 @@ def test_shield_pause(
         _shutdown_msg,
         timeout=6,
     )
-    assert_before(
-        child,
-        [
-            'raise KeyboardInterrupt',
+    expect_on_teardown: list[str] = [
+        'raise KeyboardInterrupt',
+        'Root actor terminated',
+    ]
+    if not no_capfd:
+        expect_on_teardown += [
             # 'Shutting down actor runtime',
             '#T-800 deployed to collect zombie B0',
             "'--uid', \"('hanger',",
         ]
+    assert_before(
+        child,
+        expect_on_teardown,
     )
 
 
@@ -156,8 +213,10 @@ def test_breakpoint_hook_restored(
     calls used.
 
     '''
+    # XXX required for `breakpoint()` overload and
+    # thus`tractor.devx.pause_from_sync()`.
+    pytest.importorskip('greenback')
     child = spawn('restore_builtin_breakpoint')
-
     child.expect(PROMPT)
     try:
         assert_before(

tests/discovery/conftest.py

@@ -0,0 +1,223 @@
+'''
+Discovery-suite fixtures, including the `daemon`
+remote-registrar subprocess used by the multi-program
+discovery tests.
+
+Lives here (vs. the parent `tests/conftest.py`)
+because `daemon` is a discovery-protocol primitive —
+boots a separate `tractor.run_daemon()` process whose
+sole purpose is to serve as a registrar peer for
+discovery-roundtrip tests. Pytest fixtures inherit
+DOWNWARD through conftest hierarchy, so anything
+under `tests/discovery/` automatically picks this up.
+
+'''
+from __future__ import annotations
+import os
+import platform
+import socket
+import subprocess
+import sys
+import time
+
+import pytest
+import tractor
+
+from ..conftest import (
+    sig_prog,
+    _INT_SIGNAL,
+    _non_linux,
+)
+
+
+def _wait_for_daemon_ready(
+    reg_addr: tuple,
+    tpt_proto: str,
+    *,
+    deadline: float = 10.0,
+    poll_interval: float = 0.05,
+    proc: subprocess.Popen|None = None,
+) -> None:
+    '''
+    Active-poll the daemon's bind address until it
+    accepts a connection (proving it has called
+    `bind() + listen()` and is ready to handle IPC).
+
+    Replaces the historical blind `time.sleep()` in the
+    `daemon` fixture which was racy under load — see
+    `ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`.
+
+    Uses stdlib `socket` directly (no trio runtime
+    bootstrap cost) — sufficient because
+    `tractor.run_daemon()` doesn't return from
+    bootstrap until the runtime is fully ready to
+    accept IPC.
+
+    Raises `TimeoutError` on `deadline` exceeded. If
+    `proc` is given, ALSO raises early if the daemon
+    process exits non-zero before the deadline (catches
+    daemon-startup-crash that the blind sleep used to
+    silently mask).
+
+    '''
+    end: float = time.monotonic() + deadline
+    last_exc: Exception|None = None
+    while time.monotonic() < end:
+        # Daemon-died-during-startup early-exit. Without
+        # this, a crashed-on-import daemon would just
+        # eat the full deadline before raising opaque
+        # TimeoutError.
+        if proc is not None and proc.poll() is not None:
+            raise RuntimeError(
+                f'Daemon proc exited (rc={proc.returncode}) '
+                f'before becoming ready to accept on '
+                f'{reg_addr!r}'
+            )
+        try:
+            if tpt_proto == 'tcp':
+                # `socket.create_connection` does the
+                # `socket() + connect()` dance with a
+                # builtin timeout — perfect primitive
+                # for a one-shot probe.
+                with socket.create_connection(
+                    reg_addr,
+                    timeout=poll_interval,
+                ):
+                    return
+            else:
+                # UDS — `reg_addr` is a `(filedir, sockname)`
+                # tuple per `tractor.ipc._uds.UDSAddress.unwrap`.
+                sockpath: str = os.path.join(*reg_addr)
+                sock = socket.socket(socket.AF_UNIX)
+                try:
+                    sock.settimeout(poll_interval)
+                    sock.connect(sockpath)
+                    return
+                finally:
+                    sock.close()
+        except (
+            ConnectionRefusedError,
+            FileNotFoundError,
+            OSError,
+            socket.timeout,
+        ) as exc:
+            last_exc = exc
+            time.sleep(poll_interval)
+    raise TimeoutError(
+        f'Daemon never accepted on {reg_addr!r} within '
+        f'{deadline}s (last connect-attempt exc: '
+        f'{last_exc!r})'
+    )
+
+
+# TODO: factor into @cm and move to `._testing`?
+@pytest.fixture
+def daemon(
+    debug_mode: bool,
+    loglevel: str,
+    testdir: pytest.Pytester,
+    reg_addr: tuple[str, int],
+    tpt_proto: str,
+    ci_env: bool,
+    test_log: tractor.log.StackLevelAdapter,
+
+) -> subprocess.Popen:
+    '''
+    Run a daemon root actor as a separate actor-process
+    tree and "remote registrar" for discovery-protocol
+    related tests.
+
+    '''
+    # XXX: too much logging will lock up the subproc (smh)
+    if loglevel in ('trace', 'debug'):
+        test_log.warning(
+            f'Test harness log level is too verbose: {loglevel!r}\n'
+            f'Reducing to INFO level..'
+        )
+        loglevel: str = 'info'
+
+    code: str = (
+        "import tractor; "
+        "tractor.run_daemon([], "
+        "registry_addrs={reg_addrs}, "
+        "enable_transports={enable_tpts}, "
+        "debug_mode={debug_mode}, "
+        "loglevel={ll})"
+    ).format(
+        reg_addrs=str([reg_addr]),
+        enable_tpts=str([tpt_proto]),
+        ll="'{}'".format(loglevel) if loglevel else None,
+        debug_mode=debug_mode,
+    )
+    cmd: list[str] = [
+        sys.executable,
+        '-c', code,
+    ]
+    kwargs = {}
+    if platform.system() == 'Windows':
+        # without this, tests hang on windows forever
+        kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
+
+    proc: subprocess.Popen = testdir.popen(
+        cmd,
+        **kwargs,
+    )
+
+    # Active-poll the daemon's bind address until it's
+    # ready to accept connections — replaces the legacy
+    # blind `time.sleep(2.2)` which was racy under load
+    # (see
+    # `ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`).
+    #
+    # Per-test deadline scales with platform: macOS/CI
+    # gets extra headroom; Linux dev boxes need very
+    # little.
+    deadline: float = (
+        15.0 if (_non_linux and ci_env)
+        else 10.0
+    )
+    _wait_for_daemon_ready(
+        reg_addr=reg_addr,
+        tpt_proto=tpt_proto,
+        deadline=deadline,
+        proc=proc,
+    )
+
+    assert not proc.returncode
+    yield proc
+    sig_prog(proc, _INT_SIGNAL)
+
+    # XXX! yeah.. just be reaaal careful with this bc
+    # sometimes it can lock up on the `_io.BufferedReader`
+    # and hang..
+    #
+    # NB, drain happens at TEARDOWN (post-yield), so the
+    # test body has its chance to read `proc.stderr`
+    # FIRST. Reading here AFTER would silently swallow
+    # the daemon's stderr output and break tests that
+    # assert on it (e.g. `test_abort_on_sigint`).
+    stderr: str = proc.stderr.read().decode()
+    stdout: str = proc.stdout.read().decode()
+    if (
+        stderr
+        or
+        stdout
+    ):
+        print(
+            f'Daemon actor tree produced output:\n'
+            f'{proc.args}\n'
+            f'\n'
+            f'stderr: {stderr!r}\n'
+            f'stdout: {stdout!r}\n'
+        )
+
+    if (rc := proc.returncode) != -2:
+        msg: str = (
+            f'Daemon actor tree was not cancelled !?\n'
+            f'proc.args: {proc.args!r}\n'
+            f'proc.returncode: {rc!r}\n'
+        )
+        if rc < 0:
+            raise RuntimeError(msg)
+
+        test_log.error(msg)

tests/discovery/test_multi_program.py

@@ -0,0 +1,355 @@
+"""
+Multiple python programs invoking the runtime.
+"""
+from __future__ import annotations
+import platform
+import subprocess
+import time
+from typing import (
+    TYPE_CHECKING,
+)
+
+import pytest
+import trio
+import tractor
+from tractor._testing import (
+    tractor_test,
+)
+from tractor import (
+    current_actor,
+    Actor,
+    Context,
+    Portal,
+)
+from tractor.runtime import _state
+from ..conftest import (
+    sig_prog,
+    _INT_SIGNAL,
+    _INT_RETURN_CODE,
+)
+
+if TYPE_CHECKING:
+    from tractor.msg import Aid
+    from tractor.discovery._addr import (
+        UnwrappedAddress,
+    )
+
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+# NOTE, multi-program tests historically triggered both
+# UDS sock-file leaks (daemon-subproc SIGKILL paths) AND
+# trio `WakeupSocketpair.drain()` busy-loops
+# (`test_register_duplicate_name`). Track + detect
+# per-test as a regression net.
+pytestmark = pytest.mark.usefixtures(
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
+
+
+def test_abort_on_sigint(
+    daemon: subprocess.Popen,
+):
+    assert daemon.returncode is None
+    time.sleep(0.1)
+    sig_prog(daemon, _INT_SIGNAL)
+    assert daemon.returncode == _INT_RETURN_CODE
+
+    # XXX: oddly, couldn't get capfd.readouterr() to work here?
+    if platform.system() != 'Windows':
+        # don't check stderr on windows as its empty when sending CTRL_C_EVENT
+        assert "KeyboardInterrupt" in str(daemon.stderr.read())
+
+
+@tractor_test
+async def test_cancel_remote_registrar(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+):
+    assert not current_actor().is_registrar
+    async with tractor.get_registry(reg_addr) as portal:
+        await portal.cancel_actor()
+
+    time.sleep(0.1)
+    # the registrar channel server is cancelled but not its main task
+    assert daemon.returncode is None
+
+    # no registrar socket should exist
+    with pytest.raises(OSError):
+        async with tractor.get_registry(reg_addr) as portal:
+            pass
+
+
+def test_register_duplicate_name(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+):
+    # bug-class-3 breadcrumbs: the *last* `[CANCEL]` line that
+    # appears under `--ll cancel`/`TRACTOR_LOG_FILE=...` names the
+    # cancel-cascade boundary that's parked. Pair with
+    # `_trio_main` entry/exit breadcrumbs in
+    # `tractor/spawn/_entry.py` to triangulate the swallow point.
+    log = tractor.log.get_logger('tractor.tests.test_multi_program')
+
+    async def main():
+        log.cancel('test_register_duplicate_name: enter `main()`')
+        try:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'actor nursery opened'
+                )
+
+                assert not current_actor().is_registrar
+
+                p1 = await an.start_actor('doggy')
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'spawned doggy #1'
+                )
+                p2 = await an.start_actor('doggy')
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'spawned doggy #2'
+                )
+
+                async with tractor.wait_for_actor('doggy') as portal:
+                    log.cancel(
+                        'test_register_duplicate_name: '
+                        '`wait_for_actor` returned'
+                    )
+                    assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
+
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    'ABOUT TO CALL `an.cancel()`'
+                )
+                await an.cancel()
+                log.cancel(
+                    'test_register_duplicate_name: '
+                    '`an.cancel()` returned'
+                )
+        finally:
+            log.cancel(
+                'test_register_duplicate_name: '
+                '`open_nursery.__aexit__` returned, leaving `main()`'
+            )
+
+    # XXX, run manually since we want to start this root **after**
+    # the other "daemon" program with it's own root.
+    trio.run(main)
+
+
+# `n_dups` in {4, 8} both expose the SAME pre-existing race:
+# under rapid same-name spawning against a forkserver +
+# registrar, ONE of the spawned doggies `sys.exit(2)`s during
+# boot before completing parent-handshake. Surfaces now (post
+# the spawn-time `wait_for_peer_or_proc_death` fix) as
+# `ActorFailure rc=2`; previously it was silently masked by
+# the handshake-wait parking forever.
+#
+# Larger `n_dups` widens the race window so the boot-race
+# fires more often — n_dups=4 hits ~always, n_dups=8 hits
+# occasionally. Both xfail(strict=False) so the cancel-cascade
+# regression-check still passes when the boot-race happens
+# NOT to fire.
+#
+# Tracked separately in,
+# https://github.com/goodboy/tractor/issues/456
+_DOGGY_BOOT_RACE_XFAIL = pytest.mark.xfail(
+    strict=False,
+    reason=(
+        'doggy boot-race rc=2 under rapid same-name '
+        'spawn — separate bug from cancel-cascade'
+    ),
+)
+
+
+@pytest.mark.parametrize(
+    'n_dups',
+    [
+        2,
+        pytest.param(4, marks=_DOGGY_BOOT_RACE_XFAIL),
+        pytest.param(8, marks=_DOGGY_BOOT_RACE_XFAIL),
+    ],
+    ids=lambda n: f'n_dups={n}',
+)
+def test_dup_name_cancel_cascade_escalates_to_hard_kill(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+    n_dups: int,
+):
+    '''
+    Regression for the duplicate-name cancel-cascade hang under
+    `tcp+main_thread_forkserver`.
+
+    When N actors share a single name and the parent calls
+    `an.cancel()`, the daemon registrar gets N `register_actor` RPCs
+    in tight succession. Under TCP+MTF, kernel-level socket-buffer
+    contention can push at least one sub-actor's cancel-RPC ack past
+    `Portal.cancel_timeout` (default 0.5s).
+
+    Pre-fix, `Portal.cancel_actor()` silently returned `False` on
+    that timeout, the supervisor's outer `move_on_after(3)` never
+    fired (each per-portal task always returned ≤0.5s, never
+    exceeded 3s), and `soft_kill()`'s `await wait_func(proc)` parked
+    forever — deadlocking nursery `__aexit__`.
+
+    Post-fix, `Portal.cancel_actor()` raises `ActorTooSlowError` on
+    the bounded-wait timeout, and `ActorNursery.cancel()`'s
+    per-child wrapper escalates to `proc.terminate()` (hard-kill).
+    The full nursery teardown therefore stays bounded even under
+    pathological timing.
+
+    `n_dups` is parametrized to widen the race window — more
+    same-name siblings = more concurrent register-RPCs at the
+    daemon = higher probability of hitting the contention path.
+
+    '''
+    log = tractor.log.get_logger(
+        'tractor.tests.test_multi_program'
+    )
+
+    # outer hard ceiling: a regression should fail-fast, NOT hang
+    # the test session for minutes. Budget scales with `n_dups`
+    # since each extra same-name sibling adds ~spawn-cost +
+    # potential cancel-ack-timeout escalation latency under
+    # TCP+forkserver. ~5s/sibling + 15s baseline gives plenty of
+    # headroom while still failing-loud on a real hang.
+    fail_after_s: int = 15 + (5 * n_dups)
+
+    async def main():
+        log.cancel(
+            f'enter `main()` n_dups={n_dups}'
+        )
+        with trio.fail_after(fail_after_s):
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
+                portals: list[Portal] = []
+                for i in range(n_dups):
+                    p: Portal = await an.start_actor('doggy')
+                    portals.append(p)
+                    log.cancel(
+                        f'spawned doggy #{i + 1}/{n_dups}'
+                    )
+
+                # at least one of the N must be discoverable by
+                # name; doesn't matter which one (registrar will
+                # have last-wins semantics under same-name).
+                async with tractor.wait_for_actor('doggy') as portal:
+                    expected_uids = {p.channel.uid for p in portals}
+                    assert portal.channel.uid in expected_uids
+
+                # critical section: this MUST return within
+                # `fail_after_s` even when one or more cancel-RPC
+                # acks time out. Pre-fix, this hangs forever.
+                log.cancel('about to call `an.cancel()`')
+                await an.cancel()
+                log.cancel('`an.cancel()` returned')
+
+        # post-teardown sanity: every child proc must be reaped.
+        # If escalation worked, even timed-out cancel-RPCs would
+        # have triggered `proc.terminate()` and the procs are dead.
+        for p in portals:
+            # `Portal.channel.connected()` -> False once the
+            # underlying chan disconnected (clean exit OR
+            # hard-killed proc both produce disconnect).
+            assert not p.channel.connected(), (
+                f'Portal chan still connected post-teardown?\n'
+                f'{p.channel}'
+            )
+
+    trio.run(main)
+
+
+@tractor.context
+async def get_root_portal(
+    ctx: Context,
+):
+    '''
+    Connect back to the root actor manually (using `._discovery` API)
+    and ensure it's contact info is the same as our immediate parent.
+
+    '''
+    sub: Actor = current_actor()
+    rtvs: dict = _state._runtime_vars
+    raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
+
+    # await tractor.pause()
+    # XXX, in case the sub->root discovery breaks you might need
+    # this (i know i did Xp)!!
+    # from tractor.devx import mk_pdb
+    # mk_pdb().set_trace()
+
+    assert (
+        len(raddrs) == 1
+        and
+        list(sub._parent_chan.raddr.unwrap()) in raddrs
+    )
+
+    # connect back to our immediate parent which should also
+    # be the actor-tree's root.
+    from tractor.discovery._api import get_root
+    ptl: Portal
+    async with get_root() as ptl:
+        root_aid: Aid = ptl.chan.aid
+        parent_ptl: Portal = current_actor().get_parent()
+        assert (
+            root_aid.name == 'root'
+            and
+            parent_ptl.chan.aid == root_aid
+        )
+        await ctx.started()
+
+
+def test_non_registrar_spawns_child(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+    loglevel: str,
+    debug_mode: bool,
+    ci_env: bool,
+):
+    '''
+    Ensure a non-regristar (serving) root actor can spawn a sub and
+    that sub can connect back (manually) to it's rent that is the
+    root without issue.
+
+    More or less this audits the global contact info in
+    `._state._runtime_vars`.
+
+    '''
+    async def main():
+
+        # XXX, since apparently on macos in GH's CI it can be a race
+        # with the `daemon` registrar on grabbing the socket-addr..
+        if ci_env and _non_linux:
+            await trio.sleep(.5)
+
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+            loglevel=loglevel,
+            debug_mode=debug_mode,
+        ) as an:
+
+            actor: Actor = tractor.current_actor()
+            assert not actor.is_registrar
+            sub_ptl: Portal = await an.start_actor(
+                name='sub',
+                enable_modules=[__name__],
+            )
+
+            async with sub_ptl.open_context(
+                get_root_portal,
+            ) as (ctx, _):
+                print('Waiting for `sub` to connect back to us..')
+
+            await an.cancel()
+
+    # XXX, run manually since we want to start this root **after**
+    # the other "daemon" program with it's own root.
+    trio.run(main)

tests/discovery/test_registrar.py

@@ -14,6 +14,7 @@ import psutil
 import pytest
 import subprocess
 import tractor
+from tractor.devx import dump_on_hang
 from tractor.trionics import collapse_eg
 from tractor._testing import tractor_test
 from tractor.discovery._addr import wrap_address
@@ -21,6 +22,20 @@ from tractor.discovery._multiaddr import mk_maddr
 import trio
 
 
+pytestmark = pytest.mark.usefixtures(
+    'reap_subactors_per_test',
+    # NOTE, registrar tests stress the discovery
+    # roundtrip (find_actor / wait_for_actor) which
+    # historically left orphaned UDS sock-files when
+    # subactor `hard_kill` SIGKILL'd, and which
+    # exercises the same trio `WakeupSocketpair`
+    # peer-disconnect path that triggered the
+    # busy-loop bug class.
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
+
+
 @tractor_test
 async def test_reg_then_unreg(
     reg_addr: tuple,
@@ -105,19 +120,6 @@ async def hi():
     return the_line.format(tractor.current_actor().name)
 
 
-async def say_hello(
-    other_actor: str,
-    reg_addr: tuple[str, int],
-):
-    await trio.sleep(1)  # wait for other actor to spawn
-    async with tractor.find_actor(
-        other_actor,
-        registry_addrs=[reg_addr],
-    ) as portal:
-        assert portal is not None
-        return await portal.run(__name__, 'hi')
-
-
 async def say_hello_use_wait(
     other_actor: str,
     reg_addr: tuple[str, int],
@@ -131,14 +133,17 @@ async def say_hello_use_wait(
         return result
 
 
-@tractor_test
+@tractor_test(
+    timeout=7,
+)
 @pytest.mark.parametrize(
-    'func',
-    [say_hello,
-     say_hello_use_wait]
+    'ria_fn',
+    [
+        say_hello_use_wait,
+    ]
 )
 async def test_trynamic_trio(
-    func: Callable,
+    ria_fn: Callable,
     start_method: str,
     reg_addr: tuple,
 ):
@@ -151,13 +156,13 @@ async def test_trynamic_trio(
         print("Alright... Action!")
 
         donny = await n.run_in_actor(
-            func,
+            ria_fn,
             other_actor='gretchen',
             reg_addr=reg_addr,
             name='donny',
         )
         gretchen = await n.run_in_actor(
-            func,
+            ria_fn,
             other_actor='donny',
             reg_addr=reg_addr,
             name='gretchen',
@@ -319,6 +324,14 @@ async def spawn_and_check_registry(
                 assert actor.aid.uid in registry
 
 
+async def with_timeout(
+    main: Callable,
+    timeout: float = 6,
+):
+    with trio.fail_after(timeout):
+        await main()
+
+
 @pytest.mark.parametrize('use_signal', [False, True])
 @pytest.mark.parametrize('with_streaming', [False, True])
 def test_subactors_unregister_on_cancel(
@@ -335,6 +348,7 @@ def test_subactors_unregister_on_cancel(
     '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
+            # with_timeout,
             partial(
                 spawn_and_check_registry,
                 reg_addr,
@@ -364,6 +378,7 @@ def test_subactors_unregister_on_cancel_remote_daemon(
     '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
+            with_timeout,
             partial(
                 spawn_and_check_registry,
                 reg_addr,
@@ -515,12 +530,43 @@ async def kill_transport(
 
 
 
+# ?TODO, do a OSc style signalling test on this?
+# -[ ] doesn't work for fork backends
 # @pytest.mark.parametrize('use_signal', [False, True])
+#
+# Wall-clock bound via `pytest-timeout` (`method='thread'`).
+# Under `--spawn-backend=subint` this test can wedge in an
+# un-Ctrl-C-able state (abandoned-subint + shared-GIL
+# starvation → signal-wakeup-fd pipe fills → SIGINT silently
+# dropped; see `ai/conc-anal/subint_sigint_starvation_issue.md`).
+# `method='thread'` is specifically required because `signal`-
+# method SIGALRM suffers the same GIL-starvation path and
+# wouldn't fire the Python-level handler.
+# At timeout the plugin hard-kills the pytest process — that's
+# the intended behavior here; the alternative is an unattended
+# suite run that never returns.
+# @pytest.mark.timeout(
+#     30,
+#     # NOTE should be a 2.1s happy path.
+#     # XXX for `main_thread_forkserver` this is SUPER SENSITIVE
+#     # so keep it higher to avoid flaky runs..
+#     method='thread',
+# )
+@pytest.mark.skipon_spawn_backend(
+    'subint',
+    # 'main_thread_forkserver',
+    reason=(
+        'XXX SUBINT HANGING TEST XXX\n'
+        'See outstanding issue(s)\n'
+        # TODO, put issue link!
+    )
+)
 def test_stale_entry_is_deleted(
     debug_mode: bool,
     daemon: subprocess.Popen,
     start_method: str,
     reg_addr: tuple,
+    # set_fork_aware_capture,
 ):
     '''
     Ensure that when a stale entry is detected in the registrar's
@@ -529,7 +575,6 @@ def test_stale_entry_is_deleted(
 
     '''
     async def main():
-
         name: str = 'transport_fails_actor'
         _reg_ptl: tractor.Portal
         an: tractor.ActorNursery
@@ -562,4 +607,67 @@ def test_stale_entry_is_deleted(
                 await ptl.cancel_actor()
                 await an.cancel()
 
-    trio.run(main)
+    # XXX, for tracing if this starts being flaky again..
+    #
+    timeout: float = 4
+    async def _timeout_main():
+        with trio.move_on_after(timeout) as cs:
+            await main()
+
+        if (
+            cs.cancel_called
+            and
+            debug_mode
+        ):
+            await tractor.pause()
+
+    # TODO, remove once the `[subint]` variant no longer hangs.
+    #
+    # Status (as of Phase B hard-kill landing):
+    #
+    # - `[trio]`/`[mp_*]` variants: completes normally; `dump_on_hang`
+    #   is a no-op safety net here.
+    #
+    # - `[subint]` variant: hangs indefinitely AND is un-Ctrl-C-able.
+    #   `strace -p <pytest_pid>` while in the hang reveals a silently-
+    #   dropped SIGINT — the C signal handler tries to write the
+    #   signum byte to Python's signal-wakeup fd and gets `EAGAIN`,
+    #   meaning the pipe is full (nobody's draining it).
+    #
+    #   Root-cause chain: our hard-kill in `spawn._subint` abandoned
+    #   the driver OS-thread (which is `daemon=True`) after the soft-
+    #   kill timeout, but the *sub-interpreter* inside that thread is
+    #   still running `trio.run()` — `_interpreters.destroy()` can't
+    #   force-stop a running subint (raises `InterpreterError`), and
+    #   legacy-config subints share the main GIL. The abandoned subint
+    #   starves the parent's trio event loop from iterating often
+    #   enough to drain its wakeup pipe → SIGINT silently drops.
+    #
+    #   This is structurally a CPython-level limitation: there's no
+    #   public force-destroy primitive for a running subint. We
+    #   escape on the harness side via a SIGINT-loop in the `daemon`
+    #   fixture teardown (killing the bg registrar subproc closes its
+    #   end of the IPC, which eventually unblocks a recv in main trio,
+    #   which lets the loop drain the wakeup pipe). Long-term fix path:
+    #   msgspec PEP 684 support (jcrist/msgspec#563) → isolated-mode
+    #   subints with per-interp GIL.
+    #
+    #   Full analysis:
+    #   `ai/conc-anal/subint_sigint_starvation_issue.md`
+    #
+    #   See also the *sibling* hang class documented in
+    #   `ai/conc-anal/subint_cancel_delivery_hang_issue.md` — same
+    #   subint backend, different root cause (Ctrl-C-able hang, main
+    #   trio loop iterating fine; ours to fix, not CPython's).
+    #   Reproduced by `tests/test_subint_cancellation.py
+    #   ::test_subint_non_checkpointing_child`.
+    #
+    # Kept here (and not behind a `pytestmark.skip`) so we can still
+    # inspect the dump file if the hang ever returns after a refactor.
+    # `pytest`'s stderr capture eats `faulthandler` output otherwise,
+    # so we route `dump_on_hang` to a file.
+    with dump_on_hang(
+        seconds=timeout*2,
+        path=f'/tmp/test_stale_entry_is_deleted_{start_method}.dump',
+    ):
+        trio.run(_timeout_main)

tests/ipc/test_multi_tpt.py

@@ -59,15 +59,18 @@ async def chk_tpts(
 )
 def test_root_passes_tpt_to_sub(
     tpt_proto_key: str,
+    tpt_proto: str,
     reg_addr: tuple,
     debug_mode: bool,
 ):
-    # XXX NOTE, the `reg_addr` addr won't be the same type as the
-    # `tpt_proto_key` would deliver here unless you pass `--tpt-proto
-    # <tpt_proto_key>` on the CLI.
-    #
-    # if tpt_proto_key == 'uds':
-    #     breakpoint()
+    # `reg_addr` is sourced from the CLI `--tpt-proto={tpt_proto}`,
+    # so when the parametrized `tpt_proto_key` differs, the test
+    # asks the runtime to `enable_transports=[<other_proto>]` while
+    # pointing `registry_addrs` at a `reg_addr` of the wrong proto.
+    # The layer-2 guard in `open_root_actor` is expected to fail
+    # fast with `ValueError` on this mismatch (rather than the prior
+    # silent hang during the registrar handshake).
+    proto_mismatch: bool = (tpt_proto_key != tpt_proto)
 
     async def main():
         async with tractor.open_nursery(
@@ -99,4 +102,14 @@ def test_root_passes_tpt_to_sub(
             # shudown sub-actor(s)
             await an.cancel()
 
-    trio.run(main)
+    if proto_mismatch:
+        # mismatched proto must raise `ValueError` from the
+        # `open_root_actor` runtime guard before any subactor spawn.
+        with pytest.raises(ValueError) as excinfo:
+            trio.run(main)
+        msg: str = str(excinfo.value)
+        assert 'enable_transports' in msg
+        assert 'registry_addrs' in msg
+        assert tpt_proto_key in msg or tpt_proto in msg
+    else:
+        trio.run(main)

tests/msg/test_ext_types_msgspec.py

@@ -57,6 +57,7 @@ from tractor.msg._ops import (
     limit_plds,
 )
 
+
 def enc_nsp(obj: Any) -> Any:
     actor: Actor = tractor.current_actor(
         err_on_no_runtime=False,
@@ -617,6 +618,17 @@ def test_ext_types_over_ipc(
     debug_mode: bool,
     pld_spec: Union[Type],
     add_hooks: bool,
+
+    set_fork_aware_capture,
+    # ^^XXX? for forking spawners
+
+    # capfd: pytest.CaptureFixture,
+    # ^^NOTE, super interesting that if
+    # we disable this below then the tpt-layer
+    # suffers as an "unclean EOF"??
+    # ?TODO, determine why/how that mks sense when addressing,
+    # https://github.com/pytest-dev/pytest/issues/14444
+    #
 ):
     '''
     Ensure we can support extension types coverted using
@@ -725,18 +737,26 @@ def test_ext_types_over_ipc(
 
             await p.cancel_actor()
 
+    async def fa_main():
+        with (
+            trio.fail_after(2),
+            # ?TODO, investigate? see NOTE above..
+            # capfd.disabled(),
+        ):
+            await main()
+
     if (
         NamespacePath in pld_types
         and
         add_hooks
     ):
-        trio.run(main)
+        trio.run(fa_main)
 
     else:
         with pytest.raises(
             expected_exception=tractor.RemoteActorError,
         ) as excinfo:
-            trio.run(main)
+            trio.run(fa_main)
 
         exc = excinfo.value
         # bc `.started(nsp: NamespacePath)` will raise

tests/msg/test_pldrx_limiting.py

@@ -55,12 +55,37 @@ async def maybe_expect_raises(
     raises: BaseException|None = None,
     ensure_in_message: list[str]|None = None,
     post_mortem: bool = False,
-    timeout: int = 3,
+    # NOTE, `None` selects a backend-aware default below —
+    # see `_BACKEND_TIMEOUT_DEFAULTS` for rationale. Caller
+    # can override with an explicit value to opt out.
+    timeout: int|None = None,
 ) -> None:
     '''
     Async wrapper for ensuring errors propagate from the inner scope.
 
     '''
+    if timeout is None:
+        # Pick a backend-aware default. Fork-based backends
+        # (`main_thread_forkserver`) need much more headroom
+        # because actor spawn + IPC ctx-exit + msg-validation
+        # error path takes longer than under `trio` backend
+        # — especially under cross-pytest-stream contention
+        # (#451). `test_basic_payload_spec` empirically:
+        #   - 3s flaked all-valid variant (`TooSlowError`)
+        #   - 8s flaked `invalid-return` variant
+        #     (`Cancelled` surfaced instead of `MsgTypeError`
+        #     because `fail_after` fired mid-error-path)
+        #   - 15s flaked under cross-stream contention
+        # 30s for fork-based gives plenty of headroom while
+        # still failing-loud on a genuine hang. Other
+        # backends keep the original 3s.
+        from tractor.spawn import _spawn as _spawn_mod
+        timeout = (
+            30
+            if _spawn_mod._spawn_method == 'main_thread_forkserver'
+            else 3
+        )
+
     if tractor.debug_mode():
         timeout += 999
 
@@ -259,6 +284,11 @@ def test_basic_payload_spec(
     return_value: str|None,
     started_value: int|PldMsg,
     pld_check_started_value: bool,
+
+    set_fork_aware_capture,
+    # ^XXX TODO? for forking spawners, seems to prevent hangs when
+    # --capture=sys not set, but only for a while then the problem
+    # accumulates?
 ):
     '''
     Validate the most basic `PldRx` msg-type-spec semantics around

tests/spawn/test_main_thread_forkserver.py

@@ -0,0 +1,652 @@
+'''
+Integration exercises for the `tractor.spawn._main_thread_forkserver`
+submodule at three tiers:
+
+1. the low-level primitives
+   (`fork_from_worker_thread()` from `_main_thread_forkserver`
+   + `run_subint_in_worker_thread()` from
+   `_subint_forkserver`) driven from inside a real
+   `trio.run()` in the parent process,
+
+2. the full `main_thread_forkserver_proc` spawn backend wired
+   through tractor's normal actor-nursery + portal-RPC
+   machinery — i.e. `open_root_actor` + `open_nursery` +
+   `run_in_actor` against a subactor spawned via fork from a
+   main-interp worker thread.
+
+Background
+----------
+`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+establishes that `os.fork()` from a non-main sub-interpreter
+aborts the child at the CPython level. The sibling
+`subint_fork_from_main_thread_smoketest.py` proves the escape
+hatch: fork from a main-interp *worker thread* (one that has
+never entered a subint) works, and the forked child can then
+host its own `trio.run()` inside a fresh subint.
+
+Those smoke-test scenarios are standalone — no trio runtime
+in the *parent*. Tiers (1)+(2) here cover the primitives
+driven from inside `trio.run()` in the parent, and tier (3)
+(the `*_spawn_basic` test) drives the registered
+`main_thread_forkserver` spawn backend end-to-end against
+the tractor runtime.
+
+Gating
+------
+- py3.14+ (via `concurrent.interpreters` presence)
+- no `--spawn-backend` restriction — the backend-level test
+  flips `tractor.spawn._spawn._spawn_method` programmatically
+  (via `try_set_start_method('main_thread_forkserver')`) and
+  restores it on teardown, so these tests are independent of
+  the session-level CLI backend choice.
+
+'''
+from __future__ import annotations
+from functools import partial
+import os
+from pathlib import Path
+import platform
+import select
+import signal
+import subprocess
+import sys
+import time
+
+import pytest
+import trio
+
+import tractor
+from tractor.devx import dump_on_hang
+
+
+# Gate: subint forkserver primitives require py3.14+. Check
+# the public stdlib wrapper's presence (added in 3.14) rather
+# than `_interpreters` directly — see
+# `tractor.spawn._subint` for why.
+pytest.importorskip('concurrent.interpreters')
+
+from tractor.spawn._main_thread_forkserver import (  # noqa: E402
+    fork_from_worker_thread,
+    wait_child,
+)
+from tractor.spawn._subint_forkserver import (  # noqa: E402
+    run_subint_in_worker_thread,
+)
+from tractor.spawn import _spawn as _spawn_mod  # noqa: E402
+from tractor.spawn._spawn import try_set_start_method  # noqa: E402
+
+
+# ----------------------------------------------------------------
+# child-side callables (passed via `child_target=` across fork)
+# ----------------------------------------------------------------
+
+
+_CHILD_TRIO_BOOTSTRAP: str = (
+    'import trio\n'
+    'async def _main():\n'
+    '    await trio.sleep(0.05)\n'
+    '    return 42\n'
+    'result = trio.run(_main)\n'
+    'assert result == 42, f"trio.run returned {result}"\n'
+)
+
+
+def _child_trio_in_subint() -> int:
+    '''
+    `child_target` for the trio-in-child scenario: drive a
+    trivial `trio.run()` inside a fresh legacy-config subint
+    on a worker thread.
+
+    Returns an exit code suitable for `os._exit()`:
+    - 0: subint-hosted `trio.run()` succeeded
+    - 3: driver thread hang (timeout inside `run_subint_in_worker_thread`)
+    - 4: subint bootstrap raised some other exception
+
+    '''
+    try:
+        run_subint_in_worker_thread(
+            _CHILD_TRIO_BOOTSTRAP,
+            thread_name='child-subint-trio-thread',
+        )
+    except RuntimeError:
+        # timeout / thread-never-returned
+        return 3
+    except BaseException:
+        return 4
+    return 0
+
+
+# ----------------------------------------------------------------
+# parent-side harnesses (run inside `trio.run()`)
+# ----------------------------------------------------------------
+
+
+async def run_fork_in_non_trio_thread(
+    deadline: float,
+    *,
+    child_target=None,
+) -> int:
+    '''
+    From inside a parent `trio.run()`, off-load the
+    forkserver primitive to a main-interp worker thread via
+    `trio.to_thread.run_sync()` and return the forked child's
+    pid.
+
+    Then `wait_child()` on that pid (also off-loaded so we
+    don't block trio's event loop on `waitpid()`) and assert
+    the child exited cleanly.
+
+    '''
+    with trio.fail_after(deadline):
+        # NOTE: `fork_from_worker_thread` internally spawns its
+        # own dedicated `threading.Thread` (not from trio's
+        # cache) and joins it before returning — so we can
+        # safely off-load via `to_thread.run_sync` without
+        # worrying about the trio-thread-cache recycling the
+        # runner. Pass `abandon_on_cancel=False` for the
+        # same "bounded + clean" rationale we use in
+        # `_subint.subint_proc`.
+        pid: int = await trio.to_thread.run_sync(
+            partial(
+                fork_from_worker_thread,
+                child_target,
+                thread_name='test-subint-forkserver',
+            ),
+            abandon_on_cancel=False,
+        )
+        assert pid > 0
+
+        ok, status_str = await trio.to_thread.run_sync(
+            partial(
+                wait_child,
+                pid,
+                expect_exit_ok=True,
+            ),
+            abandon_on_cancel=False,
+        )
+        assert ok, (
+            f'forked child did not exit cleanly: '
+            f'{status_str}'
+        )
+        return pid
+
+
+# ----------------------------------------------------------------
+# tests
+# ----------------------------------------------------------------
+
+
+# Bounded wall-clock via `pytest-timeout` (`method='thread'`)
+# for the usual GIL-hostage safety reason documented in the
+# sibling `test_subint_cancellation.py` / the class-A
+# `subint_sigint_starvation_issue.md`. Each test also has an
+# inner `trio.fail_after()` so assertion failures fire fast
+# under normal conditions.
+# @pytest.mark.timeout(30, method='thread')
+def test_fork_from_worker_thread_via_trio(
+) -> None:
+    '''
+    Baseline: inside `trio.run()`, call
+    `fork_from_worker_thread()` via `trio.to_thread.run_sync()`,
+    get a child pid back, reap the child cleanly.
+
+    No trio-in-child. If this regresses we know the parent-
+    side trio↔worker-thread plumbing is broken independent
+    of any child-side subint machinery.
+
+    '''
+    deadline: float = 10.0
+    with dump_on_hang(
+        seconds=deadline,
+        path='/tmp/main_thread_forkserver_baseline.dump',
+    ):
+        pid: int = trio.run(
+            partial(run_fork_in_non_trio_thread, deadline),
+        )
+    # parent-side sanity — we got a real pid back.
+    assert isinstance(pid, int) and pid > 0
+    # by now the child has been waited on; it shouldn't be
+    # reap-able again.
+    with pytest.raises((ChildProcessError, OSError)):
+        os.waitpid(pid, os.WNOHANG)
+
+
+@pytest.mark.timeout(30, method='thread')
+def test_fork_and_run_trio_in_child() -> None:
+    '''
+    End-to-end: inside the parent's `trio.run()`, off-load
+    `fork_from_worker_thread()` to a worker thread, have the
+    forked child then create a fresh subint and run
+    `trio.run()` inside it on yet another worker thread.
+
+    This is the full "forkserver + trio-in-subint-in-child"
+    pattern the proposed `main_thread_forkserver` spawn backend
+    would rest on.
+
+    '''
+    deadline: float = 15.0
+    with dump_on_hang(
+        seconds=deadline,
+        path='/tmp/main_thread_forkserver_trio_in_child.dump',
+    ):
+        pid: int = trio.run(
+            partial(
+                run_fork_in_non_trio_thread,
+                deadline,
+                child_target=_child_trio_in_subint,
+            ),
+        )
+    assert isinstance(pid, int) and pid > 0
+
+
+# ----------------------------------------------------------------
+# tier-3 backend test: drive the registered `main_thread_forkserver`
+# spawn backend end-to-end through tractor's actor-nursery +
+# portal-RPC machinery.
+# ----------------------------------------------------------------
+
+
+async def _trivial_rpc() -> str:
+    '''
+    Minimal subactor-side RPC body: just return a sentinel
+    string the parent can assert on.
+
+    '''
+    return 'hello from subint-forkserver child'
+
+
+async def _happy_path_forkserver(
+    reg_addr: tuple[str, int | str],
+    deadline: float,
+) -> None:
+    '''
+    Parent-side harness: stand up a root actor, open an actor
+    nursery, spawn one subactor via the currently-selected
+    spawn backend (which this test will have flipped to
+    `main_thread_forkserver`), run a trivial RPC through its
+    portal, assert the round-trip result.
+
+    '''
+    with trio.fail_after(deadline):
+        async with (
+            tractor.open_root_actor(
+                registry_addrs=[reg_addr],
+            ),
+            tractor.open_nursery() as an,
+        ):
+            portal: tractor.Portal = await an.run_in_actor(
+                _trivial_rpc,
+                name='subint-forkserver-child',
+            )
+            result: str = await portal.wait_for_result()
+            assert result == 'hello from subint-forkserver child'
+
+
+@pytest.fixture
+def forkserver_spawn_method():
+    '''
+    Flip `tractor.spawn._spawn._spawn_method` to
+    `'main_thread_forkserver'` for the duration of a test,
+    then restore whatever was in place before (usually the
+    session-level CLI choice, typically `'trio'`).
+
+    Without this, other tests in the same session would
+    observe the global flip and start spawning via fork —
+    which is almost certainly NOT what their assertions were
+    written against.
+
+    '''
+    prev_method: str = _spawn_mod._spawn_method
+    prev_ctx = _spawn_mod._ctx
+    try_set_start_method('main_thread_forkserver')
+    try:
+        yield
+    finally:
+        _spawn_mod._spawn_method = prev_method
+        _spawn_mod._ctx = prev_ctx
+
+
+@pytest.mark.timeout(60, method='thread')
+def test_main_thread_forkserver_spawn_basic(
+    reg_addr: tuple[str, int | str],
+    forkserver_spawn_method,
+) -> None:
+    '''
+    Happy-path: spawn ONE subactor via the
+    `main_thread_forkserver` backend (parent-side fork from a
+    main-interp worker thread), do a trivial portal-RPC
+    round-trip, tear the nursery down cleanly.
+
+    If this passes, the "forkserver + tractor runtime" arch
+    is proven end-to-end: the registered
+    `main_thread_forkserver_proc` spawn target successfully
+    forks a child, the child runs `_actor_child_main()` +
+    completes IPC handshake + serves an RPC, and the parent
+    reaps via `_ForkedProc.wait()` without regressing any of
+    the normal nursery teardown invariants.
+
+    '''
+    deadline: float = 20.0
+    with dump_on_hang(
+        seconds=deadline,
+        path='/tmp/main_thread_forkserver_spawn_basic.dump',
+    ):
+        trio.run(
+            partial(
+                _happy_path_forkserver,
+                reg_addr,
+                deadline,
+            ),
+        )
+
+
+# ----------------------------------------------------------------
+# tier-4 DRAFT: orphaned-subactor SIGINT survivability
+#
+# Motivating question: with `main_thread_forkserver`, the child's
+# `trio.run()` lives on the fork-inherited worker thread which
+# is NOT `threading.main_thread()` — so trio cannot install its
+# `signal.set_wakeup_fd`-based SIGINT handler. If the parent
+# goes away via `SIGKILL` (no IPC `Portal.cancel_actor()`
+# possible), does SIGINT on the orphan child cleanly tear it
+# down via CPython's default `KeyboardInterrupt` delivery, or
+# does it hang?
+#
+# Working hypothesis (unverified pre-this-test): post-fork the
+# child is effectively single-threaded (only the fork-worker
+# tstate survived), so SIGINT → default handler → raises
+# `KeyboardInterrupt` on the only thread — which happens to be
+# the one driving trio's event loop — so trio observes it at
+# the next checkpoint. If so, we're "fine" on this backend
+# despite the missing trio SIGINT handler.
+#
+# Cross-backend generalization (decide after this passes):
+# - applicable to any backend whose subactors are separate OS
+#   processes: `trio`, `mp_spawn`, `mp_forkserver`,
+#   `main_thread_forkserver`.
+# - NOT applicable to plain `subint` (subactors are in-process
+#   subinterpreters, no orphan child process to SIGINT).
+# - move path: lift the harness script into
+#   `tests/_orphan_harness.py`, parametrize on the session's
+#   `_spawn_method`, add `skipif _spawn_method == 'subint'`.
+# ----------------------------------------------------------------
+
+
+_ORPHAN_HARNESS_SCRIPT: str = '''
+import os
+import sys
+import trio
+import tractor
+from tractor.spawn._spawn import try_set_start_method
+
+async def _sleep_forever() -> None:
+    print(f"CHILD_PID={os.getpid()}", flush=True)
+    await trio.sleep_forever()
+
+async def _main(reg_addr):
+    async with (
+        tractor.open_root_actor(registry_addrs=[reg_addr]),
+        tractor.open_nursery() as an,
+    ):
+        portal = await an.run_in_actor(
+            _sleep_forever,
+            name="orphan-test-child",
+        )
+        print(f"PARENT_READY={os.getpid()}", flush=True)
+        await trio.sleep_forever()
+
+if __name__ == "__main__":
+    backend = sys.argv[1]
+    host = sys.argv[2]
+    port = int(sys.argv[3])
+    try_set_start_method(backend)
+    trio.run(_main, (host, port))
+'''
+
+
+def _read_marker(
+    proc: subprocess.Popen,
+    marker: str,
+    timeout: float,
+    _buf: dict,
+) -> str:
+    '''
+    Block until `<marker>=<value>\\n` appears on `proc.stdout`
+    and return `<value>`. Uses a per-proc byte buffer (`_buf`)
+    to carry partial lines across calls.
+
+    '''
+    deadline: float = time.monotonic() + timeout
+    remainder: bytes = _buf.get('remainder', b'')
+    prefix: bytes = f'{marker}='.encode()
+    while time.monotonic() < deadline:
+        # drain any complete lines already buffered
+        while b'\n' in remainder:
+            line, remainder = remainder.split(b'\n', 1)
+            if line.startswith(prefix):
+                _buf['remainder'] = remainder
+                return line[len(prefix):].decode().strip()
+        ready, _, _ = select.select([proc.stdout], [], [], 0.2)
+        if not ready:
+            continue
+        chunk: bytes = os.read(proc.stdout.fileno(), 4096)
+        if not chunk:
+            break
+        remainder += chunk
+    _buf['remainder'] = remainder
+    raise TimeoutError(
+        f'Never observed marker {marker!r} on harness stdout '
+        f'within {timeout}s'
+    )
+
+
+def _process_alive(pid: int) -> bool:
+    '''Liveness probe for a pid we do NOT parent (post-orphan).'''
+    try:
+        os.kill(pid, 0)
+        return True
+    except ProcessLookupError:
+        return False
+
+
+# Known-gap test — `main_thread_forkserver` orphan-SIGINT
+# handling. See
+# `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
+# `strict=True` so if a future fix closes the gap the
+# XPASS surfaces as a FAIL and forces us to drop the
+# mark intentionally.
+@pytest.mark.xfail(
+    strict=True,
+    reason=(
+        'Orphan subactor SIGINT delivery: trio event loop '
+        'on non-main thread post-fork doesn\'t see the '
+        'external SIGINT → KBI path. See tracker doc.\n'
+        'ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md'
+    ),
+)
+@pytest.mark.timeout(
+    30,
+    method='thread',
+)
+def test_orphaned_subactor_sigint_cleanup_DRAFT(
+    reg_addr: tuple[str, int | str],
+    tmp_path: Path,
+) -> None:
+    '''
+    DRAFT — orphaned-subactor SIGINT survivability under the
+    `main_thread_forkserver` backend.
+
+    Sequence:
+      1. Spawn a harness subprocess that brings up a root
+         actor + one `sleep_forever` subactor via
+         `main_thread_forkserver`.
+      2. Read the harness's stdout for `PARENT_READY=<pid>`
+         and `CHILD_PID=<pid>` markers (confirms the
+         parent→child IPC handshake completed).
+      3. `SIGKILL` the parent (no IPC cancel possible — the
+         whole point of this test).
+      4. `SIGINT` the orphan child.
+      5. Poll `os.kill(child_pid, 0)` for up to 10s — assert
+         the child exits.
+
+    Empirical result (2026-04, py3.14): currently **FAILS** —
+    SIGINT on the orphan child doesn't unwind the trio loop,
+    despite trio's `KIManager` handler being correctly
+    installed in the subactor (the post-fork thread IS
+    `threading.main_thread()` on py3.14). `faulthandler` dump
+    shows the subactor wedged in `trio/_core/_io_epoll.py::
+    get_events` — the signal's supposed wakeup of the event
+    loop isn't firing. Full analysis + diagnostic evidence
+    in `ai/conc-anal/
+    subint_forkserver_orphan_sigint_hang_issue.md`.
+
+    The runtime's *intentional* "KBI-as-OS-cancel" path at
+    `tractor/spawn/_entry.py::_trio_main:164` is therefore
+    unreachable under this backend+config. Closing the gap is
+    aligned with existing design intent (make the already-
+    designed behavior actually fire), not a new feature.
+    Marked `xfail(strict=True)` so the
+    mark flips to XPASS→fail once the gap is closed and we'll
+    know to drop the mark.
+
+    '''
+    if platform.system() != 'Linux':
+        pytest.skip(
+            'orphan-reparenting semantics only exercised on Linux'
+        )
+
+    script_path = tmp_path / '_orphan_harness.py'
+    script_path.write_text(_ORPHAN_HARNESS_SCRIPT)
+
+    # Offset the port so we don't race the session reg_addr with
+    # any concurrently-running backend test's listener.
+    host: str = reg_addr[0]
+    port: int = int(reg_addr[1]) + 17
+
+    proc: subprocess.Popen = subprocess.Popen(
+        [
+            sys.executable,
+            str(script_path),
+            'main_thread_forkserver',
+            host,
+            str(port),
+        ],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+    parent_pid: int | None = None
+    child_pid: int | None = None
+    buf: dict = {}
+    try:
+        child_pid = int(_read_marker(proc, 'CHILD_PID', 15.0, buf))
+        parent_pid = int(_read_marker(proc, 'PARENT_READY', 15.0, buf))
+
+        # sanity: both alive before we start killing stuff
+        assert _process_alive(parent_pid), (
+            f'harness parent pid={parent_pid} gone before '
+            f'SIGKILL — test premise broken'
+        )
+        assert _process_alive(child_pid), (
+            f'orphan-candidate child pid={child_pid} gone '
+            f'before test started'
+        )
+
+        # step 3: kill parent — no IPC cancel arrives at child.
+        # `proc.wait()` reaps the zombie so it truly disappears
+        # from the process table (otherwise `os.kill(pid, 0)`
+        # keeps reporting it as alive).
+        os.kill(parent_pid, signal.SIGKILL)
+        try:
+            proc.wait(timeout=3.0)
+        except subprocess.TimeoutExpired:
+            pytest.fail(
+                f'harness parent pid={parent_pid} did not die '
+                f'after SIGKILL — test premise broken'
+            )
+        assert _process_alive(child_pid), (
+            f'child pid={child_pid} died along with parent — '
+            f'did the parent reap it before SIGKILL took? '
+            f'test premise requires an orphan.'
+        )
+
+        # step 4+5: SIGINT the orphan, poll for exit.
+        os.kill(child_pid, signal.SIGINT)
+        timeout: float = 6.0
+        cleanup_deadline: float = time.monotonic() + timeout
+        while time.monotonic() < cleanup_deadline:
+            if not _process_alive(child_pid):
+                return  # <- success path
+            time.sleep(0.1)
+
+        pytest.fail(
+            f'Orphan subactor (pid={child_pid}) did NOT exit '
+            f'within 10s of SIGINT under `main_thread_forkserver` '
+            f'→ trio on non-main thread did not observe the '
+            f'default CPython KeyboardInterrupt; backend needs '
+            f'explicit SIGINT plumbing.'
+        )
+    finally:
+        # best-effort cleanup to avoid leaking orphans across
+        # the test session regardless of outcome.
+        for pid in (parent_pid, child_pid):
+            if pid is None:
+                continue
+            try:
+                os.kill(pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+        try:
+            proc.kill()
+        except OSError:
+            pass
+        try:
+            proc.wait(timeout=2.0)
+        except subprocess.TimeoutExpired:
+            pass
+
+
+# ----------------------------------------------------------------
+# regression guard: variant-2 (`subint_forkserver`) placeholder
+# MUST raise `NotImplementedError` today — guards against future
+# commits accidentally re-aliasing the key to the variant-1
+# coroutine (which was a transient state during the rename).
+# ----------------------------------------------------------------
+def test_subint_forkserver_key_errors_cleanly() -> None:
+    '''
+    `--spawn-backend=subint_forkserver` is reserved for the
+    eventual variant-2 (subint-isolated child runtime)
+    backend, gated on jcrist/msgspec#1026 unblocking PEP 684
+    isolated-mode subints upstream.
+
+    Until that lands, the dispatch entry MUST raise
+    `NotImplementedError` immediately rather than silently
+    aliasing to `main_thread_forkserver_proc`. Verify the
+    error message also surfaces both the working-backend
+    pointer and the upstream-blocker ref so an operator
+    arriving at the error has somewhere to go.
+
+    '''
+    import asyncio
+    from tractor.spawn._spawn import _methods
+
+    proc = _methods['subint_forkserver']
+    with pytest.raises(NotImplementedError) as ei:
+        # signature args match `main_thread_forkserver_proc`'s
+        # — the stub raises before touching them so dummy
+        # values are fine.
+        asyncio.run(
+            proc(
+                'x', None, None, {}, [],
+                ('127.0.0.1', 0), {},
+            )
+        )
+
+    msg: str = str(ei.value)
+    assert 'main_thread_forkserver' in msg, (
+        f'stub error msg should redirect to the working '
+        f'variant-1 backend; got: {msg!r}'
+    )
+    assert 'msgspec#1026' in msg or '1026' in msg, (
+        f'stub error msg should reference the upstream '
+        f'blocker (jcrist/msgspec#1026); got: {msg!r}'
+    )

tests/spawn/test_subint_cancellation.py

@@ -0,0 +1,245 @@
+'''
+Cancellation + hard-kill semantics audit for the `subint` spawn
+backend.
+
+Exercises the escape-hatch machinery added to
+`tractor.spawn._subint` (module-level `_HARD_KILL_TIMEOUT`,
+bounded shields around the soft-kill / thread-join sites, daemon
+driver-thread abandonment) so that future stdlib regressions or
+our own refactors don't silently re-introduce the hangs first
+diagnosed during the Phase B.2/B.3 bringup (issue #379).
+
+Every test in this module:
+- is wrapped in `trio.fail_after()` for a deterministic per-test
+  wall-clock ceiling (the whole point of these tests is to fail
+  fast when our escape hatches regress; an unbounded test would
+  defeat itself),
+- arms `tractor.devx.dump_on_hang()` to capture a stack dump on
+  failure — without it, a hang here is opaque because pytest's
+  stderr capture swallows `faulthandler` output by default
+  (hard-won lesson from the original diagnosis),
+- skips on py<3.13 (no `_interpreters`) and on any
+  `--spawn-backend` other than `'subint'` (these tests are
+  subint-specific by design — they'd be nonsense under `trio` or
+  `mp_*`).
+
+'''
+from __future__ import annotations
+from functools import partial
+
+import pytest
+import trio
+import tractor
+from tractor.devx import dump_on_hang
+
+
+# Gate: the `subint` backend requires py3.14+. Check the
+# public stdlib wrapper's presence (added in 3.14) rather than
+# the private `_interpreters` module (which exists on 3.13 but
+# wedges under tractor's usage — see `tractor.spawn._subint`).
+pytest.importorskip('concurrent.interpreters')
+
+# Subint-only: read the spawn method that `pytest_configure`
+# committed via `try_set_start_method()`. By the time this module
+# imports, the CLI backend choice has been applied.
+from tractor.spawn._spawn import _spawn_method  # noqa: E402
+
+if _spawn_method != 'subint':
+    pytestmark = pytest.mark.skip(
+        reason=(
+            "subint-specific cancellation audit — "
+            "pass `--spawn-backend=subint` to run."
+        ),
+    )
+
+
+# ----------------------------------------------------------------
+# child-side task bodies (run inside the spawned subint)
+# ----------------------------------------------------------------
+
+
+async def _trivial_rpc() -> str:
+    '''
+    Minimal RPC body for the baseline happy-teardown test.
+    '''
+    return 'hello from subint'
+
+
+async def _spin_without_trio_checkpoints() -> None:
+    '''
+    Block the main task with NO trio-visible checkpoints so any
+    `Portal.cancel_actor()` arriving over IPC has nothing to hand
+    off to.
+
+    `threading.Event.wait(timeout)` releases the GIL (so other
+    threads — including trio's IO/RPC tasks — can progress) but
+    does NOT insert a trio checkpoint, so the subactor's main
+    task never notices cancellation.
+
+    This is the exact "stuck subint" scenario the hard-kill
+    shields exist to survive.
+    '''
+    import threading
+    never_set = threading.Event()
+    while not never_set.is_set():
+        # 1s re-check granularity; low enough not to waste CPU,
+        # high enough that even a pathologically slow
+        # `_HARD_KILL_TIMEOUT` won't accidentally align with a
+        # wake.
+        never_set.wait(timeout=1.0)
+
+
+# ----------------------------------------------------------------
+# parent-side harnesses (driven inside `trio.run(...)`)
+# ----------------------------------------------------------------
+
+
+async def _happy_path(
+    reg_addr: tuple[str, int|str],
+    deadline: float,
+) -> None:
+    with trio.fail_after(deadline):
+        async with (
+            tractor.open_root_actor(
+                registry_addrs=[reg_addr],
+            ),
+            tractor.open_nursery() as an,
+        ):
+            portal: tractor.Portal = await an.run_in_actor(
+                _trivial_rpc,
+                name='subint-happy',
+            )
+            result: str = await portal.wait_for_result()
+            assert result == 'hello from subint'
+
+
+async def _spawn_stuck_then_cancel(
+    reg_addr: tuple[str, int|str],
+    deadline: float,
+) -> None:
+    with trio.fail_after(deadline):
+        async with (
+            tractor.open_root_actor(
+                registry_addrs=[reg_addr],
+            ),
+            tractor.open_nursery() as an,
+        ):
+            await an.run_in_actor(
+                _spin_without_trio_checkpoints,
+                name='subint-stuck',
+            )
+            # Give the child time to reach its non-checkpointing
+            # loop before we cancel; the precise value doesn't
+            # matter as long as it's a handful of trio schedule
+            # ticks.
+            await trio.sleep(0.5)
+            an.cancel_scope.cancel()
+
+
+# ----------------------------------------------------------------
+# tests
+# ----------------------------------------------------------------
+
+
+def test_subint_happy_teardown(
+    reg_addr: tuple[str, int|str],
+) -> None:
+    '''
+    Baseline: spawn a subactor, do one portal RPC, close nursery
+    cleanly. No cancel, no faults.
+
+    If this regresses we know something's wrong at the
+    spawn/teardown layer unrelated to the hard-kill escape
+    hatches.
+
+    '''
+    deadline: float = 10.0
+    with dump_on_hang(
+        seconds=deadline,
+        path='/tmp/subint_cancellation_happy.dump',
+    ):
+        trio.run(partial(_happy_path, reg_addr, deadline))
+
+
+@pytest.mark.skipon_spawn_backend(
+    'subint',
+    reason=(
+        'XXX SUBINT HANGING TEST XXX\n'
+        'See oustanding issue(s)\n'
+        # TODO, put issue link!
+    )
+)
+# Wall-clock bound via `pytest-timeout` (`method='thread'`)
+# as defense-in-depth over the inner `trio.fail_after(15)`.
+# Under the orphaned-channel hang class described in
+# `ai/conc-anal/subint_cancel_delivery_hang_issue.md`, SIGINT
+# is still deliverable and this test *should* be unwedgeable
+# by the inner trio timeout — but sibling subint-backend
+# tests in this repo have also exhibited the
+# `subint_sigint_starvation_issue.md` GIL-starvation flavor,
+# so `method='thread'` keeps us safe in case ordering or
+# load shifts the failure mode.
+# @pytest.mark.timeout(
+#     3,  # NOTE never passes pre-3.14+ subints support.
+#     method='thread',
+# )
+def test_subint_non_checkpointing_child(
+    reg_addr: tuple[str, int|str],
+) -> None:
+    '''
+    Cancel a subactor whose main task is stuck in a non-
+    checkpointing Python loop.
+
+    `Portal.cancel_actor()` may be delivered over IPC but the
+    main task never checkpoints to observe the Cancelled —
+    so the subint's `trio.run()` can't exit gracefully.
+
+    The parent `subint_proc` bounded-shield + daemon-driver-
+    thread combo should abandon the thread after
+    `_HARD_KILL_TIMEOUT` and let the parent return cleanly.
+
+    Wall-clock budget:
+    - ~0.5s: settle time for child to enter the stuck loop
+    - ~3s: `_HARD_KILL_TIMEOUT` (soft-kill wait)
+    - ~3s: `_HARD_KILL_TIMEOUT` (thread-join wait)
+    - margin
+
+    KNOWN ISSUE (Ctrl-C-able hang):
+    -------------------------------
+    This test currently hangs past the hard-kill timeout for
+    reasons unrelated to the subint teardown itself — after
+    the subint is destroyed, a parent-side trio task appears
+    to park on an orphaned IPC channel (no clean EOF
+    delivered to a waiting receive). Unlike the
+    SIGINT-starvation sibling case in
+    `test_stale_entry_is_deleted`, this hang IS Ctrl-C-able
+    (`strace` shows SIGINT wakeup-fd `write() = 1`, not
+    `EAGAIN`) — i.e. the main trio loop is still iterating
+    normally. That makes this *our* bug to fix, not a
+    CPython-level limitation.
+
+    See `ai/conc-anal/subint_cancel_delivery_hang_issue.md`
+    for the full analysis + candidate fix directions
+    (explicit parent-side channel abort in `subint_proc`
+    teardown being the most likely surgical fix).
+
+    The sibling `ai/conc-anal/subint_sigint_starvation_issue.md`
+    documents the *other* hang class (abandoned-legacy-subint
+    thread + shared-GIL starvation → signal-wakeup-fd pipe
+    fills → SIGINT silently dropped) — that one is
+    structurally blocked on msgspec PEP 684 adoption and is
+    NOT what this test is hitting.
+
+    '''
+    deadline: float = 15.0
+    with dump_on_hang(
+        seconds=deadline,
+        path='/tmp/subint_cancellation_stuck.dump',
+    ):
+        trio.run(
+            partial(
+                _spawn_stuck_then_cancel,
+                reg_addr,
+                deadline,
+            ),
+        )

tests/test_advanced_streaming.py

@@ -5,10 +5,15 @@ Advanced streaming patterns using bidirectional streams and contexts.
 from collections import Counter
 import itertools
 import platform
+from typing import Type
 
 import pytest
 import trio
 import tractor
+from tractor._testing.trace import (
+    AfkAlarmWTraceFactory,
+    FailAfterWTraceFactory,
+)
 
 
 def is_win():
@@ -76,9 +81,7 @@ async def subscribe(
 
 
 async def consumer(
-
     subs: list[str],
-
 ) -> None:
 
     uid = tractor.current_actor().uid
@@ -108,59 +111,200 @@ async def consumer(
                         print(f'{uid} got: {value}')
 
 
-def test_dynamic_pub_sub():
+# NOTE: deliberately NOT using `@pytest.mark.timeout(...)` —
+# both pytest-timeout enforcement modes break trio under
+# fork-based backends:
+#
+# - `method='signal'` (SIGALRM): the handler synchronously
+#   raises `Failed` in trio's main thread mid-`epoll.poll()`,
+#   leaves `GLOBAL_RUN_CONTEXT` half-installed ("Trio guest
+#   run got abandoned"), and EVERY subsequent `trio.run()`
+#   in the same pytest process bails with
+#   `RuntimeError: Attempted to call run() from inside a
+#   run()` — session-wide poison.
+#
+# - `method='thread'`: calls `_thread.interrupt_main()`
+#   raising `KeyboardInterrupt` into the main thread. Under
+#   fork-based backends with mid-cascade fd-juggling the KBI
+#   can escape trio's `KIManager` and bubble out of pytest
+#   itself — kills the WHOLE session.
+#
+# Instead we use `trio.fail_after()` INSIDE `main()` below:
+# trio's own `Cancelled`/`TooSlowError` machinery handles the
+# timeout, cleanly unwinds the actor nursery's cancel
+# cascade, and only fails the single test (no cross-test
+# state corruption either way).
+#
+# `pyproject.toml`'s default `timeout = 200` is still a
+# last-resort safety net.
+@pytest.mark.parametrize(
+    'expect_cancel_exc', [
+        KeyboardInterrupt,
+        trio.TooSlowError,
+    ],
+    ids=lambda item:
+        f'expect_user_exc_raised={item.__name__}'
+)
+def test_dynamic_pub_sub(
+    reg_addr: tuple,
+    debug_mode: bool,
+    test_log: tractor.log.StackLevelAdapter,
+    reap_subactors_per_test: int,
+    expect_cancel_exc: Type[BaseException],
+
+    is_forking_spawner: bool,
+    set_fork_aware_capture,
+
+    fail_after_w_trace: FailAfterWTraceFactory,
+    afk_alarm_w_trace: AfkAlarmWTraceFactory,
+):
+    failed_to_raise_report: str = (
+        f'Never got a {expect_cancel_exc!r} ??'
+    )
 
     global _registry
 
     from multiprocessing import cpu_count
     cpus = cpu_count()
 
+    # Hard safety cap via trio's own cancellation. NOTE see the
+    # module-level note on why we avoid `pytest-timeout` for this
+    # test. Picked backend-aware: under `trio` backend spawn is
+    # cheap (~1s for `cpus` actors) but fork-based backends pay
+    # a per-spawn cost (forkserver round-trip + IPC peer-handshake)
+    # that can stack up over `cpus - 1` sequential `n.run_in_actor()`
+    # calls — especially on UDS under cross-pytest contention
+    # (#451 / #452). 4s was flaking right at the edge under fork
+    # backends — bumped to 8s with diag-snapshot-on-timeout via
+    # `fail_after_w_trace` so a borderline run still fails loud
+    # but lands a ptree/wchan/py-spy dump in
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` for inspection.
+    #
+    # XXX caveat: this is an *inner* trio cancel — its `Cancelled`
+    # cannot reach a task parked in a shielded `await` (e.g. inside
+    # actor-nursery teardown). When the in-band cancel path is
+    # itself buggy (the bug-class-3 `raise KBI` swallow we're
+    # currently chasing) this guard does NOT fire and the test
+    # sits forever until external SIGINT. The `afk_alarm_w_trace`
+    # outer guard below is the AFK-safety counterpart (SIGALRM
+    # raises in the main thread regardless of trio scope state).
+    fail_after_s: float = (
+        8
+        if is_forking_spawner
+        else 20
+    )
+    # inflate under CPU throttle — incl. the sustained-load
+    # power-cap invisible to static freq reads — so a slow box
+    # doesn't trip the deadline. See `scripts/cpu-perf-check`.
+    from .conftest import cpu_perf_headroom
+    headroom: float = cpu_perf_headroom()
+    if headroom != 1.:
+        fail_after_s *= headroom
+
     async def main():
-        async with tractor.open_nursery() as n:
-
-            # name of this actor will be same as target func
-            await n.run_in_actor(publisher)
-
-            for i, sub in zip(
-                range(cpus - 2),
-                itertools.cycle(_registry.keys())
-            ):
-                await n.run_in_actor(
-                    consumer,
-                    name=f'consumer_{sub}',
-                    subs=[sub],
+        # bug-class-3 breadcrumb: tag each level of the cancel path
+        # so when the run hangs and we capture cancel-level logs, the
+        # *last* breadcrumb that fired names the swallow point.
+        test_log.cancel('test_dynamic_pub_sub: enter main()')
+        try:
+            async with fail_after_w_trace(fail_after_s):
+                test_log.cancel(
+                    f'test_dynamic_pub_sub: '
+                    f'enter `fail_after_w_trace({fail_after_s})` scope'
                 )
+                try:
+                    async with tractor.open_nursery(
+                        registry_addrs=[reg_addr],
+                        debug_mode=debug_mode,
+                    ) as n:
+                        test_log.cancel(
+                            'test_dynamic_pub_sub: '
+                            'actor nursery opened'
+                        )
 
-            # make one dynamic subscriber
-            await n.run_in_actor(
-                consumer,
-                name='consumer_dynamic',
-                subs=list(_registry.keys()),
+                        # name of this actor will be same as target func
+                        await n.run_in_actor(publisher)
+
+                        for i, sub in zip(
+                            range(cpus - 2),
+                            itertools.cycle(_registry.keys())
+                        ):
+                            await n.run_in_actor(
+                                consumer,
+                                name=f'consumer_{sub}',
+                                subs=[sub],
+                            )
+
+                        # make one dynamic subscriber
+                        await n.run_in_actor(
+                            consumer,
+                            name='consumer_dynamic',
+                            subs=list(_registry.keys()),
+                        )
+
+                        # block until "cancelled by user"
+                        await trio.sleep(3)
+                        test_log.warning(
+                            f'Raising user cancel exc: '
+                            f'{expect_cancel_exc!r}'
+                        )
+                        test_log.cancel(
+                            f'test_dynamic_pub_sub: '
+                            f'ABOUT TO RAISE {expect_cancel_exc!r}'
+                        )
+                        raise expect_cancel_exc('simulate user cancel!')
+                finally:
+                    test_log.cancel(
+                        'test_dynamic_pub_sub: '
+                        'actor nursery `__aexit__` returned'
+                    )
+            test_log.cancel(
+                'test_dynamic_pub_sub: `fail_after` scope exited'
+            )
+        finally:
+            test_log.cancel(
+                'test_dynamic_pub_sub: leaving `main()`'
             )
 
-            # block until cancelled by user
-            with trio.fail_after(3):
-                await trio.sleep_forever()
+    def _run_and_match():
+        try:
+            trio.run(main)
+            pytest.fail(failed_to_raise_report)
+        except expect_cancel_exc:
+            # parent-side raised the user-cancel exc directly and
+            # it propagated unwrapped; clean path.
+            test_log.exception('Got user-cancel exc AS EXPECTED')
+        except BaseExceptionGroup as err:
+            # under fork-based backends the user-raised cancel
+            # can race with subactor-side stream teardown
+            # (`trio.EndOfChannel` from a publisher's `send()`
+            # whose remote half got cut). The expected exc may
+            # then be nested deeper in the group rather than at
+            # the top level. `BaseExceptionGroup.split()` walks
+            # the exc tree recursively (Python 3.11+).
+            matched, _ = err.split(expect_cancel_exc)
+            if matched is None:
+                pytest.fail(failed_to_raise_report)
 
-    try:
-        trio.run(main)
-    except (
-        trio.TooSlowError,
-        ExceptionGroup,
-    ) as err:
-        if isinstance(err, ExceptionGroup):
-            for suberr in err.exceptions:
-                if isinstance(suberr, trio.TooSlowError):
-                    break
-            else:
-                pytest.fail('Never got a `TooSlowError` ?')
+            test_log.exception('Got user-cancel exc AS EXPECTED')
+
+    # outer SIGALRM-based guard — survives a shielded-await
+    # deadlock since `signal.alarm` raises in the main thread
+    # regardless of trio's scope state, AND captures a full diag
+    # snapshot to `$XDG_CACHE_HOME/tractor/hung-dumps/` before
+    # re-raising. ONLY armed under fork-based backends since the
+    # bug we're chasing is MTF-specific. Cap = `fail_after_s + 5`
+    # so the trio-native path always wins when it works.
+    if is_forking_spawner:
+        with afk_alarm_w_trace(fail_after_s + 5):
+            _run_and_match()
+    else:
+        _run_and_match()
 
 
 @tractor.context
 async def one_task_streams_and_one_handles_reqresp(
-
     ctx: tractor.Context,
-
 ) -> None:
 
     await ctx.started()
@@ -257,7 +401,8 @@ async def echo_ctx_stream(
 
 
 def test_sigint_both_stream_types():
-    '''Verify that running a bi-directional and recv only stream
+    '''
+    Verify that running a bi-directional and recv only stream
     side-by-side will cancel correctly from SIGINT.
 
     '''
@@ -287,9 +432,11 @@ def test_sigint_both_stream_types():
                             assert resp == msg
                             raise KeyboardInterrupt
 
+    # TODO, use pytest.raises() here instead?
+    # (why weren't we originally?)
     try:
         trio.run(main)
-        assert 0, "Didn't receive KBI!?"
+        pytest.fail("Didn't receive KBI!?")
     except KeyboardInterrupt:
         pass
 
@@ -356,7 +503,12 @@ async def inf_streamer(
     print('streamer exited .open_streamer() block')
 
 
+# @pytest.mark.timeout(
+#     6,
+#     method='signal',
+# )
 def test_local_task_fanout_from_stream(
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -421,4 +573,9 @@ def test_local_task_fanout_from_stream(
 
             await p.cancel_actor()
 
-    trio.run(main)
+    async def w_timeout():
+        with trio.fail_after(6):
+            await main()
+
+    # trio.run(main)
+    trio.run(w_timeout)

tests/test_cancellation.py

@@ -7,6 +7,7 @@ import signal
 import platform
 import time
 from itertools import repeat
+from typing  import Type
 
 import pytest
 import trio
@@ -14,6 +15,7 @@ import tractor
 from tractor._testing import (
     tractor_test,
 )
+from tractor._testing.trace import FailAfterWTraceFactory
 from .conftest import no_windows
 
 
@@ -21,6 +23,46 @@ _non_linux: bool = platform.system() != 'Linux'
 _friggin_windows: bool = platform.system() == 'Windows'
 
 
+pytestmark = [
+    # Multi-actor cancel cascades under
+    # `--spawn-backend=subint` trip the abandoned-subint
+    # GIL-hostage class — a stuck subint can starve the
+    # parent's trio loop and block cancel-delivery.
+    # Apply the skip module-wide rather than per-test
+    # since every test here exercises the same cascade.
+    pytest.mark.skipon_spawn_backend(
+        'subint',
+        reason=(
+            'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
+            'Cancel cascades under '
+            '`--spawn-backend=subint` trip the abandoned-subint '
+            'GIL-hostage class — see\n'
+            '  - `ai/conc-anal/subint_sigint_starvation_issue.md` '
+            '(GIL-hostage, SIGINT-unresponsive)\n'
+            '  - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
+            '(sibling: parent parks on dead chan)\n'
+            '  - https://github.com/goodboy/tractor/issues/379 '
+            '(subint umbrella)\n'
+        )
+    ),
+    pytest.mark.usefixtures(
+        'reap_subactors_per_test',
+        # NOTE, cancellation tests stress the SIGKILL
+        # `hard_kill` path which leaks UDS sock-files when
+        # the subactor's IPC server `finally:` cleanup
+        # doesn't run. Track per-test for blame attribution.
+        'track_orphaned_uds_per_test',
+        # NOTE, cancel-cascade timing races (see
+        # `test_nested_multierrors`) can also leave a
+        # subactor spinning at 100% CPU when its cancel
+        # signal got swallowed mid-handshake. Catches the
+        # runaway-loop class that doesn't leak UDS socks
+        # but burns the box.
+        'detect_runaway_subactors_per_test',
+    ),
+]
+
+
 async def assert_err(delay=0):
     await trio.sleep(delay)
     assert 0
@@ -45,7 +87,11 @@ async def do_nuthin():
     ],
     ids=['no_args', 'unexpected_args'],
 )
-def test_remote_error(reg_addr, args_err):
+def test_remote_error(
+    reg_addr: tuple,
+    args_err: tuple[dict, Type[Exception]],
+    set_fork_aware_capture,
+):
     '''
     Verify an error raised in a subactor that is propagated
     to the parent nursery, contains the underlying boxed builtin
@@ -112,6 +158,8 @@ def test_remote_error(reg_addr, args_err):
 
 def test_multierror(
     reg_addr: tuple[str, int],
+    start_method: str,  # parametrized
+    set_fork_aware_capture, #: Callable,
 ):
     '''
     Verify we raise a ``BaseExceptionGroup`` out of a nursery where
@@ -141,31 +189,68 @@ def test_multierror(
         trio.run(main)
 
 
-@pytest.mark.parametrize('delay', (0, 0.5))
 @pytest.mark.parametrize(
-    'num_subactors', range(25, 26),
+    'delay',
+    (0, 0.5),
+    ids='delays={}'.format,
 )
-def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
-    """Verify we raise a ``BaseExceptionGroup`` out of a nursery where
+@pytest.mark.parametrize(
+    'num_subactors',
+    range(25, 26),
+    ids= 'num_subs={}'.format,
+)
+def test_multierror_fast_nursery(
+    reg_addr: tuple,
+    start_method: str,
+    num_subactors: int,
+    delay: float,
+    set_fork_aware_capture,
+    fail_after_w_trace: FailAfterWTraceFactory,
+):
+    '''
+    Verify we raise a ``BaseExceptionGroup`` out of a nursery where
     more then one actor errors and also with a delay before failure
     to test failure during an ongoing spawning.
-    """
-    async def main():
-        async with tractor.open_nursery(
-            registry_addrs=[reg_addr],
-        ) as nursery:
 
-            for i in range(num_subactors):
-                await nursery.run_in_actor(
-                    assert_err,
-                    name=f'errorer{i}',
-                    delay=delay
-                )
+    '''
+    async def main():
+        # budget = 2× natural trio-backend cascade time for
+        # 25 errorer subactors (~14s observed). on-timeout
+        # diag snapshot → if the cancel cascade hangs
+        # (observed under MTF backend with N>=14 errorer
+        # subactors) we get a fresh ptree/wchan/py-spy dump
+        # on disk INSTEAD of an opaque pytest timeout-kill.
+        # See `tractor/_testing/trace.py` for the helper.
+        async with fail_after_w_trace(30.0):
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as nursery:
+
+                for i in range(num_subactors):
+                    await nursery.run_in_actor(
+                        assert_err,
+                        name=f'errorer{i}',
+                        delay=delay
+                    )
 
     # with pytest.raises(trio.MultiError) as exc_info:
-    with pytest.raises(BaseExceptionGroup) as exc_info:
+    # NOTE, `trio.TooSlowError` from `fail_after_w_trace`
+    # bubbles UN-wrapped if `open_nursery.__aexit__` never
+    # gets re-entered; wrapped inside a `BaseExceptionGroup`
+    # if it did. Accept both shapes so the matcher itself
+    # doesn't lie about *what* failed.
+    with pytest.raises(
+        (BaseExceptionGroup, trio.TooSlowError),
+    ) as exc_info:
         trio.run(main)
 
+    if isinstance(exc_info.value, trio.TooSlowError):
+        pytest.fail(
+            f'cancel cascade hung past 12s '
+            f'(num_subactors={num_subactors}, delay={delay}); '
+            f'see stderr for `fail_after_w_trace` snapshot path'
+        )
+
     assert exc_info.type == ExceptionGroup
     err = exc_info.value
     exceptions = err.exceptions
@@ -189,8 +274,15 @@ async def do_nothing():
     pass
 
 
-@pytest.mark.parametrize('mechanism', ['nursery_cancel', KeyboardInterrupt])
-def test_cancel_single_subactor(reg_addr, mechanism):
+@pytest.mark.parametrize(
+    'mechanism', [
+    'nursery_cancel',
+    KeyboardInterrupt,
+])
+def test_cancel_single_subactor(
+    reg_addr: tuple,
+    mechanism: str|KeyboardInterrupt,
+):
     '''
     Ensure a ``ActorNursery.start_actor()`` spawned subactor
     cancels when the nursery is cancelled.
@@ -232,9 +324,14 @@ async def stream_forever():
         await trio.sleep(0.01)
 
 
-@tractor_test
-async def test_cancel_infinite_streamer(start_method):
-
+@tractor_test(
+    timeout=6,
+)
+async def test_cancel_infinite_streamer(
+    reg_addr: tuple,
+    start_method: str,
+    set_fork_aware_capture,
+):
     # stream for at most 1 seconds
     with (
         trio.fail_after(4),
@@ -286,11 +383,15 @@ async def test_cancel_infinite_streamer(start_method):
         'no_daemon_actors_fail_all_run_in_actors_sleep_then_fail',
     ],
 )
-@tractor_test
+@tractor_test(
+    timeout=10,
+)
 async def test_some_cancels_all(
     num_actors_and_errs: tuple,
+    reg_addr: tuple,
     start_method: str,
     loglevel: str,
+    set_fork_aware_capture, #: Callable,
 ):
     '''
     Verify a subset of failed subactors causes all others in
@@ -370,7 +471,10 @@ async def test_some_cancels_all(
         pytest.fail("Should have gotten a remote assertion error?")
 
 
-async def spawn_and_error(breadth, depth) -> None:
+async def spawn_and_error(
+    breadth: int,
+    depth: int,
+) -> None:
     name = tractor.current_actor().name
     async with tractor.open_nursery() as nursery:
         for i in range(breadth):
@@ -395,28 +499,182 @@ async def spawn_and_error(breadth, depth) -> None:
             await nursery.run_in_actor(*args, **kwargs)
 
 
-@tractor_test
-async def test_nested_multierrors(loglevel, start_method):
+# NOTE: `main_thread_forkserver` capture-fd hang class is no
+# longer skipped here — `--capture=sys` (the new `pyproject.toml`
+# default) sidesteps the pipe-buffer-fill deadlock for
+# `test_nested_multierrors`. See
+# `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
+# / #449 for the post-mortem.
+# @pytest.mark.timeout(
+#     10,
+#     method='thread',
+# )
+@pytest.mark.parametrize(
+    'depth',
+    [1, 3],
+    ids='depth={}'.format,
+)
+@tractor_test(
+    # bumped from the 30s default to cover fork-based
+    # cancel-cascade flakes; 2 spawners × 2 errorers × depth 1+
+    # cascade through 6 portal-wait_for_result paths each
+    # paying `terminate_after=1.6s` + UDS sock-unlink under
+    # MTF/UDS contention can easily blow past 30s.
+    # Trio backend is fast and won't notice the extra budget.
+    # See `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    timeout=10,
+)
+async def test_nested_multierrors(
+    reg_addr: tuple,
+    loglevel: str,
+    start_method: str,
+    set_fork_aware_capture,
+    fail_after_w_trace: FailAfterWTraceFactory,
+    request: pytest.FixtureRequest,
+    depth: int,
+):
     '''
-    Test that failed actor sets are wrapped in `BaseExceptionGroup`s. This
-    test goes only 2 nurseries deep but we should eventually have tests
-    for arbitrary n-depth actor trees.
+    Test that failed actor sets are wrapped in `BaseExceptionGroup`s.
+
+    Parametrized over recursion `depth ∈ {1, 3}`:
+
+      - `depth=1`: shallow tree (2 spawners × 2 errorers, 2
+        levels). Cascade completes well within budget on ALL
+        backends including MTF — regression-safety green case.
+
+      - `depth=3`: deep tree (2 spawners × recursive depth-3
+        spawn-and-error). On `main_thread_forkserver` this
+        trips the cancel-cascade shape-mismatch bug class
+        (see `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`)
+        — xfailed below.
 
     '''
-    if start_method == 'trio':
-        depth = 3
-        subactor_breadth = 2
-    else:
-        # XXX: multiprocessing can't seem to handle any more then 2 depth
-        # process trees for whatever reason.
-        # Any more process levels then this and we see bugs that cause
-        # hangs and broken pipes all over the place...
-        if start_method == 'forkserver':
-            pytest.skip("Forksever sux hard at nested spawning...")
-        depth = 1  # means an additional actor tree of spawning (2 levels deep)
-        subactor_breadth = 2
+    # XXX: `multiprocessing.forkserver` can't handle nested
+    # spawning at any depth — hangs / broken-pipes. Pre-existing
+    # backend limitation, NOT depth-specific.
+    if start_method == 'forkserver':
+        pytest.skip("Forksever sux hard at nested spawning...")
 
-    with trio.fail_after(120):
+    subactor_breadth = 2
+
+    # MTF backend trips a probabilistic timing race in the
+    # cancel-cascade — NOT depth-gated; depth amplifies the
+    # variance so depth=3 misses nearly every run while
+    # depth=1 misses occasionally. Both get the xfail mark
+    # (with `strict=False`) since the bug class can fire at
+    # either depth.
+    #
+    # The scenario in detail:
+    #
+    #     T=0      spawn spawner_0 + spawner_1 in parallel
+    #     T=t1     spawner_0's child errors →
+    #              RemoteActorError reaches root nursery
+    #     T=t1+ε   root nursery starts cancelling
+    #              spawner_1's portal-wait
+    #     T=t2     spawner_1's child errors → tries to send
+    #              RemoteActorError back
+    #
+    #     if t2 < t1+ε:  BEG = [RAE, RAE]        ← clean (xpass)
+    #     if t2 > t1+ε:  BEG = [RAE, Cancelled]  ← race tripped (xfail)
+    #
+    # i.e. the assertion below (`isinstance(_, RemoteActorError)`)
+    # fails iff cancel-delivery beats the other tree's natural
+    # error-propagation. Depth amplifies `t2-t1` variance
+    # (longer per-tree paths = more skew); under MTF the
+    # fork-spawn jitter + UDS-contention widens both `t1` and
+    # `t2` further.
+    #
+    # With `strict=False` the clean-cascade cases (most
+    # depth=1 runs, rare depth=3 runs) report as `xpassed`
+    # while the race-tripped cases report as `xfailed` —
+    # neither flakes `--lf`. When MTF cancel-cascade
+    # eventually speeds up enough to close the race even at
+    # depth=3, BOTH variants will reliably `xpass` and
+    # pytest will yell — our signal to drop the marker. See
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    #
+    # Probe CPU throttle ONCE up-front (folds in the sustained-load
+    # power-cap that static freq reads miss): used BOTH to inflate
+    # the deadline budget below AND to xfail depth=3, whose failure
+    # mode under throttle is a runtime-internal reap deadline — not
+    # a test-budget miss. See `scripts/cpu-perf-check`.
+    from .conftest import cpu_perf_headroom
+    headroom: float = cpu_perf_headroom()
+
+    if start_method == 'main_thread_forkserver':
+        request.node.add_marker(
+            pytest.mark.xfail(
+                strict=False,
+                reason=(
+                    f'MTF cancel-cascade shape-mismatch at '
+                    f'depth={depth} (Cancelled races '
+                    f'RemoteActorError in BEG); see conc-anal/'
+                    'cancel_cascade_too_slow_under_main_thread_forkserver_issue.md'
+                ),
+            )
+        )
+
+    # Under CPU throttle (incl. the sustained-load power-cap that
+    # static freq reads miss) the DEEP depth=3 tree trips tractor's
+    # INTERNAL reap deadlines (`soft_kill`/`hard_kill`
+    # `move_on_after`/`terminate_after=1.6`) before slow subprocs
+    # exit, injecting a `Cancelled(source='deadline')` into the BEG
+    # — the SAME shape-mismatch class as the MTF xfail above, and
+    # NOT fixable by inflating the test-level budget (the Cancelled
+    # is minted inside the runtime, not by our `fail_after`).
+    # xfail(strict=False) so it auto-clears the moment the box is
+    # un-throttled (`headroom == 1.`); depth=1's shallow tree stays
+    # under those internal deadlines so it just rides the budget
+    # inflation below. See `scripts/cpu-perf-check`.
+    elif (
+        depth == 3
+        and
+        headroom != 1.
+    ):
+        request.node.add_marker(
+            pytest.mark.xfail(
+                strict=False,
+                reason=(
+                    'CPU throttled — tractor reap deadline injects '
+                    'Cancelled into BEG; see conc-anal/'
+                    'trio_033_cancel_cascade_slowdown_depth3_issue.md'
+                ),
+            )
+        )
+
+    # Per-backend/-depth budgets: in the non-hang case the
+    # whole spawn + cancel-cascade should complete in well
+    # under these. On the borderline hang case the
+    # `fail_after_w_trace` fires `TooSlowError` AND captures a
+    # ptree/wchan/py-spy snapshot to
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` for offline
+    # inspection. See
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    #
+    # NOTE: the `trio` depth=3 budget was bumped 6 -> 12s after
+    # the `trio` 0.29 -> 0.33 lock bump (commit c7741bba) slowed
+    # the depth-3 cancel-cascade from <6s to ~7-8s; the 6s
+    # deadline was firing and its `Cancelled(source='deadline')`
+    # (trio 0.33 cancel-reason metadata) collapsed a BEG branch,
+    # breaking the `RemoteActorError` assertion below. depth=1
+    # still finishes in ~3s so keeps the 6s budget. See
+    # `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
+    match (start_method, depth):
+        case ('trio', 1):
+            timeout = 6
+        case ('trio', 3):
+            timeout = 12
+        case ('main_thread_forkserver', 1):
+            timeout = 16
+        case ('main_thread_forkserver', 3):
+            timeout = 30
+
+    # inflate the budget by the throttle headroom probed above so
+    # a slow box doesn't masquerade as a deadline regression.
+    if headroom != 1.:
+        timeout *= headroom
+
+    async with fail_after_w_trace(timeout):
         try:
             async with tractor.open_nursery() as nursery:
                 for i in range(subactor_breadth):
@@ -483,20 +741,24 @@ async def test_nested_multierrors(loglevel, start_method):
 
 @no_windows
 def test_cancel_via_SIGINT(
-    loglevel,
-    start_method,
-    spawn_backend,
+    reg_addr: tuple,
+    loglevel: str,
+    start_method: str,
 ):
-    """Ensure that a control-C (SIGINT) signal cancels both the parent and
+    '''
+    Ensure that a control-C (SIGINT) signal cancels both the parent and
     child processes in trionic fashion
-    """
+
+    '''
     pid: int = os.getpid()
 
     async def main():
         with trio.fail_after(2):
-            async with tractor.open_nursery() as tn:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as tn:
                 await tn.start_actor('sucka')
-                if 'mp' in spawn_backend:
+                if 'mp' in start_method:
                     time.sleep(0.1)
                 os.kill(pid, signal.SIGINT)
                 await trio.sleep_forever()
@@ -507,6 +769,7 @@ def test_cancel_via_SIGINT(
 
 @no_windows
 def test_cancel_via_SIGINT_other_task(
+    reg_addr: tuple,
     loglevel: str,
     start_method: str,
     spawn_backend: str,
@@ -535,7 +798,9 @@ def test_cancel_via_SIGINT_other_task(
     async def spawn_and_sleep_forever(
         task_status=trio.TASK_STATUS_IGNORED
     ):
-        async with tractor.open_nursery() as tn:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as tn:
             for i in range(3):
                 await tn.run_in_actor(
                     sleep_forever,
@@ -599,7 +864,7 @@ async def spawn_sub_with_sync_blocking_task():
 def test_cancel_while_childs_child_in_sync_sleep(
     loglevel: str,
     start_method: str,
-    spawn_backend: str,
+    is_forking_spawner: bool,
     debug_mode: bool,
     reg_addr: tuple,
     man_cancel_outer: bool,
@@ -615,7 +880,10 @@ def test_cancel_while_childs_child_in_sync_sleep(
 
     '''
     if start_method == 'forkserver':
-        pytest.skip("Forksever sux hard at resuming from sync sleep...")
+        pytest.skip(
+            "`multiprocessing`'s forkserver sux hard at "
+            "resuming from sync sleep..."
+        )
 
     async def main():
         #
@@ -658,7 +926,11 @@ def test_cancel_while_childs_child_in_sync_sleep(
         # delay = 2  # is AssertionError in eg AND no TooSlowError !?
         # is AssertionError in eg AND no _cs cancellation.
         delay = (
-            6 if _non_linux
+            6 if (
+                _non_linux
+                or
+                is_forking_spawner
+            )
             else 4 
         )
 
@@ -694,7 +966,7 @@ def test_cancel_while_childs_child_in_sync_sleep(
 
 
 def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
-    start_method,
+    start_method: str,
 ):
     '''
     This is a very subtle test which demonstrates how cancellation

tests/test_clustering.py

@@ -24,8 +24,14 @@ def test_empty_mngrs_input_raises(
             'actor-cluster teardown hangs intermittently on UDS'
         )
 
+    # inflate under CPU throttle — incl. the sustained-load
+    # power-cap invisible to static freq reads. See
+    # `scripts/cpu-perf-check`.
+    from .conftest import cpu_perf_headroom
+    fail_after_s: float = 3 * cpu_perf_headroom()
+
     async def main():
-        with trio.fail_after(3):
+        with trio.fail_after(fail_after_s):
             async with (
                 open_actor_cluster(
                     modules=[__name__],
@@ -77,6 +83,7 @@ async def worker(
 @tractor_test
 async def test_streaming_to_actor_cluster(
     tpt_proto: str,
+    is_forking_spawner: bool,
 ):
     '''
     Open an actor "cluster" using the (experimental) `._clustering`
@@ -88,7 +95,18 @@ async def test_streaming_to_actor_cluster(
             f'Test currently fails with tpt-proto={tpt_proto!r}\n'
         )
 
-    with trio.fail_after(6):
+    delay: float = (
+        10 if is_forking_spawner
+        else 6
+    )
+    # inflate under CPU throttle — incl. the sustained-load
+    # power-cap invisible to static freq reads. See
+    # `scripts/cpu-perf-check`.
+    from .conftest import cpu_perf_headroom
+    headroom: float = cpu_perf_headroom()
+    if headroom != 1.:
+        delay *= headroom
+    with trio.fail_after(delay):
         async with (
             open_actor_cluster(modules=[__name__]) as portals,
 

tests/test_context_stream_semantics.py

@@ -115,10 +115,12 @@ async def not_started_but_stream_opened(
 )
 def test_started_misuse(
     target: Callable,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -184,15 +186,24 @@ def test_simple_context(
     error_parent,
     child_blocks_forever,
     pointlessly_open_stream,
+    reg_addr: tuple,
     debug_mode: bool,
+    is_forking_spawner: bool,
 ):
 
-    timeout = 1.5 if not platform.system() == 'Windows' else 4
+    timeout: float = 1.5
+    # windows and forking-spawner both have "slower but more
+    # deterministic" cancel teardown.
+    if platform.system() == 'Windows':
+        timeout = 4
+    elif is_forking_spawner:
+        timeout = 3
 
     async def main():
 
         with trio.fail_after(timeout):
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
             ) as an:
                 portal = await an.start_actor(
@@ -278,6 +289,7 @@ def test_parent_cancels(
     cancel_method: str,
     chk_ctx_result_before_exit: bool,
     child_returns_early: bool,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -355,6 +367,7 @@ def test_parent_cancels(
     async def main():
 
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -931,6 +944,7 @@ async def keep_sending_from_child(
 )
 def test_one_end_stream_not_opened(
     overrun_by: tuple[str, int, Callable],
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -949,6 +963,7 @@ def test_one_end_stream_not_opened(
 
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -1113,6 +1128,7 @@ def test_maybe_allow_overruns_stream(
 
     # conftest wide
     loglevel: str,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -1133,6 +1149,7 @@ def test_maybe_allow_overruns_stream(
     '''
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
@@ -1249,6 +1266,7 @@ def test_maybe_allow_overruns_stream(
 
 def test_ctx_with_self_actor(
     loglevel: str,
+    reg_addr: tuple,
     debug_mode: bool,
 ):
     '''
@@ -1263,6 +1281,7 @@ def test_ctx_with_self_actor(
     '''
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
             enable_modules=[__name__],
         ) as an:

tests/test_infected_asyncio.py

@@ -30,6 +30,32 @@ from tractor import (
 from tractor.runtime import _state
 from tractor.trionics import BroadcastReceiver
 from tractor._testing import expect_ctxc
+from tractor._testing.trace import (
+    AfkAlarmWTraceFactory,
+    FailAfterWTraceFactory,
+)
+
+
+# Per-test zombie-subactor reaper. Opt-in (NOT autouse) —
+# see `tractor._testing.pytest.reap_subactors_per_test`'s
+# docstring for the full rationale. This module specifically
+# needs it because tests like
+# `test_echoserver_detailed_mechanics[KeyboardInterrupt]`
+# and the `test_sigint_closes_lifetime_stack[*]` matrix have
+# been observed to hang past pytest's wall-clock under
+# `main_thread_forkserver`, leaving subactor forks that
+# squat on registrar resources and cascade-fail every
+# subsequent test (`test_inter_peer_cancellation`,
+# `test_legacy_one_way_streaming`, etc.).
+pytestmark = pytest.mark.usefixtures(
+    'reap_subactors_per_test',
+    # NOTE, asyncio cancel cascade has historically
+    # triggered both UDS sockfile leaks (SIGKILL path)
+    # AND the trio `WakeupSocketpair.drain()` busy-loop
+    # — see `test_aio_simple_error`'s history.
+    'track_orphaned_uds_per_test',
+    'detect_runaway_subactors_per_test',
+)
 
 
 @pytest.fixture(
@@ -183,6 +209,7 @@ def test_tractor_cancels_aio(
     async def main():
         async with tractor.open_nursery(
             debug_mode=debug_mode,
+            registry_addrs=[reg_addr],
         ) as an:
             portal = await an.run_in_actor(
                 asyncio_actor,
@@ -205,11 +232,11 @@ def test_trio_cancels_aio(
 
     '''
     async def main():
-
+        # cancel the nursery shortly after boot
         with trio.move_on_after(1):
-            # cancel the nursery shortly after boot
-
-            async with tractor.open_nursery() as tn:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as tn:
                 await tn.run_in_actor(
                     asyncio_actor,
                     target='aio_sleep_forever',
@@ -277,7 +304,9 @@ def test_context_spawns_aio_task_that_errors(
     '''
     async def main():
         with trio.fail_after(1 + delay):
-            async with tractor.open_nursery() as an:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
                 p = await an.start_actor(
                     'aio_daemon',
                     enable_modules=[__name__],
@@ -360,7 +389,9 @@ def test_aio_cancelled_from_aio_causes_trio_cancelled(
     async def main():
 
         an: tractor.ActorNursery
-        async with tractor.open_nursery() as an:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as an:
             p: tractor.Portal = await an.run_in_actor(
                 asyncio_actor,
                 target='aio_cancel',
@@ -569,7 +600,9 @@ def test_basic_interloop_channel_stream(
     async def main():
         # TODO, figure out min timeout here!
         with trio.fail_after(6):
-            async with tractor.open_nursery() as an:
+            async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
+            ) as an:
                 portal = await an.run_in_actor(
                     stream_from_aio,
                     infect_asyncio=True,
@@ -582,9 +615,13 @@ def test_basic_interloop_channel_stream(
 
 
 # TODO: parametrize the above test and avoid the duplication here?
-def test_trio_error_cancels_intertask_chan(reg_addr):
+def test_trio_error_cancels_intertask_chan(
+    reg_addr: tuple[str, int],
+):
     async def main():
-        async with tractor.open_nursery() as an:
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+        ) as an:
             portal = await an.run_in_actor(
                 stream_from_aio,
                 trio_raise_err=True,
@@ -619,6 +656,7 @@ def test_trio_closes_early_causes_aio_checkpoint_raise(
             async with tractor.open_nursery(
                 debug_mode=debug_mode,
                 # enable_stack_on_sig=True,
+                registry_addrs=[reg_addr],
             ) as an:
                 portal = await an.run_in_actor(
                     stream_from_aio,
@@ -667,6 +705,7 @@ def test_aio_exits_early_relays_AsyncioTaskExited(
     async def main():
         with trio.fail_after(1 + delay):
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
                 # enable_stack_on_sig=True,
             ) as an:
@@ -707,6 +746,7 @@ def test_aio_errors_and_channel_propagates_and_closes(
 ):
     async def main():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             portal = await an.run_in_actor(
@@ -796,16 +836,47 @@ async def trio_to_aio_echo_server(
 
 @pytest.mark.parametrize(
     'raise_error_mid_stream',
-    [False, Exception, KeyboardInterrupt],
+    [
+        False,
+        Exception,
+        KeyboardInterrupt,
+    ],
     ids='raise_error={}'.format,
 )
 def test_echoserver_detailed_mechanics(
     reg_addr: tuple[str, int],
     debug_mode: bool,
     raise_error_mid_stream,
+
+    is_forking_spawner: bool,
+    fail_after_w_trace: FailAfterWTraceFactory,
 ):
-    async def main():
+    # NOTE: under fork-based backends the cancel-cascade
+    # path is structurally slower than `trio`'s subproc-exec
+    # (per-spawn forkserver-handshake compounds during
+    # teardown). Bump the cap so cross-test contamination
+    # doesn't flake this — see
+    # `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
+    timeout: float = (
+        999 if tractor.debug_mode()
+        else 4 if is_forking_spawner
+        # was 1; the `trio` 0.29 -> 0.33 bump slowed the
+        # cancel-cascade so a 1s budget raced the ~1s teardown
+        # deadline. On a deadline-fire the injected
+        # `Cancelled(source='deadline')` wraps the mid-stream
+        # KBI in a `BaseExceptionGroup`, breaking the bare
+        # `pytest.raises(KeyboardInterrupt)` below. See
+        # `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
+        else 4
+    )
+
+    # body factored out so the `fail_after_w_trace`-wrapping
+    # `main()` stays a 2-liner — keeps the deep `open_nursery`
+    # /`open_context`/`open_stream` block at its natural indent
+    # level instead of pushing it under yet another `async with`.
+    async def _body():
         async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
             debug_mode=debug_mode,
         ) as an:
             p = await an.start_actor(
@@ -849,6 +920,15 @@ def test_echoserver_detailed_mechanics(
             # is cancelled by kbi or out of task cancellation
             await p.cancel_actor()
 
+    async def main():
+        # on-timeout diag snapshot via `fail_after_w_trace`
+        # — when the cancel cascade hangs under MTF we get a
+        # fresh `ptree`/`wchan`/`py-spy` dump on disk INSTEAD
+        # of an opaque pytest timeout-kill. See
+        # `tractor/_testing/trace.py`.
+        async with fail_after_w_trace(timeout):
+            await _body()
+
     if raise_error_mid_stream:
         with pytest.raises(raise_error_mid_stream):
             trio.run(main)
@@ -984,7 +1064,7 @@ async def manage_file(
     ],
     ids=[
         'bg_aio_task',
-        'just_trio_slee',
+        'just_trio_sleep',
     ],
 )
 @pytest.mark.parametrize(
@@ -1000,11 +1080,15 @@ async def manage_file(
 )
 def test_sigint_closes_lifetime_stack(
     tmp_path: Path,
+    reg_addr: tuple,
+    debug_mode: bool,
+
     wait_for_ctx: bool,
     bg_aio_task: bool,
     trio_side_is_shielded: bool,
-    debug_mode: bool,
     send_sigint_to: str,
+    is_forking_spawner: bool,
+    afk_alarm_w_trace: AfkAlarmWTraceFactory,
 ):
     '''
     Ensure that an infected child can use the `Actor.lifetime_stack`
@@ -1014,12 +1098,30 @@ def test_sigint_closes_lifetime_stack(
     '''
     async def main():
 
-        delay = 999 if tractor.debug_mode() else 1
+        delay: float = (
+            999
+            if debug_mode
+            else 1
+        )
+        # pre-init so the `except (KeyboardInterrupt, ContextCancelled)`
+        # handler below doesn't `UnboundLocalError` if KBI fires BEFORE
+        # we ever enter the `as (ctx, first)` body (e.g. when
+        # `p.open_context().__aenter__` is hung waiting for the
+        # subactor's `StartAck` due to a fork-child IPC race —
+        # see `dynamic_pub_sub_spawn_time_transport_close_under_mtf_issue.md`).
+        tmp_file: Path|None = None
+        ctx: tractor.Context|None = None
         try:
             an: tractor.ActorNursery
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
             ) as an:
+
+                # sanity
+                if debug_mode:
+                    assert tractor.debug_mode()
+
                 p: tractor.Portal = await an.start_actor(
                     'file_mngr',
                     enable_modules=[__name__],
@@ -1034,7 +1136,7 @@ def test_sigint_closes_lifetime_stack(
                 ) as (ctx, first):
 
                     path_str, cpid = first
-                    tmp_file: Path = Path(path_str)
+                    tmp_file = Path(path_str)
                     assert tmp_file.exists()
 
                     # XXX originally to simulate what (hopefully)
@@ -1054,6 +1156,10 @@ def test_sigint_closes_lifetime_stack(
                         cpid if send_sigint_to == 'child'
                         else os.getpid()
                     )
+                    print(
+                        f'Sending SIGINT to {send_sigint_to!r}\n'
+                        f'pid: {pid!r}\n'
+                    )
                     os.kill(
                         pid,
                         signal.SIGINT,
@@ -1064,13 +1170,37 @@ def test_sigint_closes_lifetime_stack(
                     # timeout should trigger!
                     if wait_for_ctx:
                         print('waiting for ctx outcome in parent..')
+
+                        if debug_mode:
+                            assert delay == 999
+
                         try:
-                            with trio.fail_after(1 + delay):
+                            with trio.fail_after(
+                                1 + delay
+                            ):
                                 await ctx.wait_for_result()
                         except tractor.ContextCancelled as ctxc:
                             assert ctxc.canceller == ctx.chan.uid
                             raise
 
+                        except trio.TooSlowError:
+                            if (
+                                send_sigint_to == 'child'
+                                and
+                                is_forking_spawner
+                            ):
+                                pytest.xfail(
+                                    reason=(
+                                        'SIGINT delivery to fork-child subactor is known '
+                                        'to NOT SUCCEED, precisely bc we have not wired up a'
+                                        '"trio SIGINT mode" in the child pre-fork.\n'
+                                        'Also see `test_orphaned_subactor_sigint_cleanup_DRAFT` for'
+                                        'a dedicated suite demonstrating this expected limitation as '
+                                        'well as the detailed doc:\n'
+                                        '`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.\n'
+                                    ),
+                                )
+
                     # XXX CASE 2: this seems to be the source of the
                     # original issue which exhibited BEFORE we put
                     # a `Actor.cancel_soon()` inside
@@ -1084,6 +1214,21 @@ def test_sigint_closes_lifetime_stack(
             KeyboardInterrupt,
             ContextCancelled,
         ):
+            # If we got here BEFORE entering the ctx body (e.g.
+            # spawn-time IPC race hung `open_context.__aenter__` and
+            # the AFK-guard `signal.alarm` fired KBI from outside the
+            # trio loop), `tmp_file`/`ctx` are still `None` — surface
+            # that fact directly instead of `UnboundLocalError`.
+            if tmp_file is None:
+                pytest.fail(
+                    'KBI/ctxc fired BEFORE `p.open_context()` returned '
+                    "the child's `started` value — likely fork-child "
+                    'IPC race; see '
+                    '`ai/conc-anal/'
+                    'dynamic_pub_sub_spawn_time_transport_close_'
+                    'under_mtf_issue.md`'
+                )
+
             # XXX CASE 2: without the bug fixed, in the
             # KBI-raised-in-parent case, the actor teardown should
             # never get run (silently abaondoned by `asyncio`..) and
@@ -1091,7 +1236,26 @@ def test_sigint_closes_lifetime_stack(
             assert not tmp_file.exists()
             assert ctx.maybe_error
 
-    trio.run(main)
+    # outer hard wall-clock backstop via `afk_alarm_w_trace`:
+    # when the in-band trio cancel path doesn't fire (e.g.
+    # parent is parked in a shielded `await` inside actor-
+    # nursery teardown, or `open_context.__aenter__` hangs
+    # waiting for a child's `StartAck` that never comes), the
+    # `signal.alarm` inside the CM raises `AFKAlarmTimeout`
+    # in the main thread regardless of trio's scope state —
+    # AND captures a full diag snapshot to
+    # `$XDG_CACHE_HOME/tractor/hung-dumps/` before re-raising.
+    # Only armed under fork-based backends since this hang-
+    # class is MTF-specific.
+    if (
+        not debug_mode
+        and
+        is_forking_spawner
+    ):
+        with afk_alarm_w_trace(10):
+            trio.run(main)
+    else:
+        trio.run(main)
 
 
 
@@ -1170,6 +1334,7 @@ def test_aio_side_raises_before_started(
         with trio.fail_after(3):
             an: tractor.ActorNursery
             async with tractor.open_nursery(
+                registry_addrs=[reg_addr],
                 debug_mode=debug_mode,
                 loglevel=loglevel,
             ) as an:

tests/test_inter_peer_cancellation.py

@@ -26,6 +26,31 @@ from tractor._testing import (
 
 from .conftest import cpu_scaling_factor
 
+pytestmark = [
+    pytest.mark.skipon_spawn_backend(
+        'subint',
+        reason=(
+            'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
+            'Inter-peer cancel cascades under '
+            '`--spawn-backend=subint` trip the abandoned-subint '
+            'GIL-hostage class — see\n'
+            '  - `ai/conc-anal/subint_sigint_starvation_issue.md` '
+            '(GIL-hostage, SIGINT-unresponsive)\n'
+            '  - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
+            '(sibling: parent parks on dead chan)\n'
+            '  - https://github.com/goodboy/tractor/issues/379 '
+            '(subint umbrella)\n'
+        )
+    ),
+    # NOTE, inter-peer cancellation tests stress the
+    # multi-actor cancel cascade which under SIGKILL
+    # leaves UDS sock-files orphaned. Track per-test
+    # for blame attribution.
+    pytest.mark.usefixtures(
+        'track_orphaned_uds_per_test',
+    ),
+]
+
 # XXX TODO cases:
 # - [x] WE cancelled the peer and thus should not see any raised
 #   `ContextCancelled` as it should be reaped silently?

tests/test_legacy_one_way_streaming.py

@@ -1,7 +1,7 @@
-"""
+'''
 Streaming via the, now legacy, "async-gen API".
 
-"""
+'''
 import time
 from functools import partial
 import platform
@@ -12,6 +12,11 @@ import tractor
 import pytest
 
 from tractor._testing import tractor_test
+from tractor._exceptions import ActorTooSlowError
+
+_non_linux: bool = (
+    _sys := platform.system()
+) != 'Linux'
 
 
 def test_must_define_ctx():
@@ -68,8 +73,10 @@ async def stream_from_single_subactor(
     start_method,
     stream_func,
 ):
-    """Verify we can spawn a daemon actor and retrieve streamed data.
-    """
+    '''
+    Verify we can spawn a daemon actor and retrieve streamed data.
+
+    '''
     # only one per host address, spawns an actor if None
 
     async with tractor.open_nursery(
@@ -242,14 +249,19 @@ async def a_quadruple_example() -> list[int]:
         start = time.time()
         # the portal call returns exactly what you'd expect
         # as if the remote "aggregate" function was called locally
-        result_stream = []
+        result_stream: list[int] = []
 
-        async with portal.open_stream_from(aggregate, seed=seed) as stream:
+        async with portal.open_stream_from(
+            aggregate,
+            seed=seed,
+        ) as stream:
             async for value in stream:
                 result_stream.append(value)
 
-        print(f"STREAM TIME = {time.time() - start}")
-        print(f"STREAM + SPAWN TIME = {time.time() - pre_start}")
+        print(
+            f"STREAM TIME = {time.time() - start}\n"
+            f"STREAM + SPAWN TIME = {time.time() - pre_start}\n"
+        )
         assert result_stream == list(range(seed))
         await portal.cancel_actor()
         return result_stream
@@ -258,13 +270,24 @@ async def a_quadruple_example() -> list[int]:
 async def cancel_after(
     wait: float,
     reg_addr: tuple,
+    expect_cancel: bool,
 ) -> list[int]:
 
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
     ):
-        with trio.move_on_after(wait):
-            return await a_quadruple_example()
+        res: list[int]|None = None
+        with trio.move_on_after(wait) as cs:
+            res: list[int] = await a_quadruple_example()
+            return res
+
+        if (
+            not expect_cancel
+            and
+            cs.cancelled_caught
+        ):
+            assert not res
+            raise ActorTooSlowError
 
 
 @pytest.fixture(scope='module')
@@ -272,9 +295,14 @@ def time_quad_ex(
     reg_addr: tuple,
     ci_env: bool,
     spawn_backend: str,
+    is_forking_spawner: bool,
+    tpt_proto: str,
 ):
-    non_linux: bool = (_sys := platform.system()) != 'Linux'
-    if ci_env and non_linux:
+    if (
+        ci_env
+        and
+        _non_linux
+    ):
         pytest.skip(f'Test is too flaky on {_sys!r} in CI')
 
     if spawn_backend == 'mp':
@@ -284,15 +312,36 @@ def time_quad_ex(
         '''
         pytest.skip("Test is too flaky on mp in CI")
 
-    timeout = 7 if non_linux else 4
-    start = time.time()
-    results: list[int] = trio.run(
-        cancel_after,
-        timeout,
-        reg_addr,
+    timeout: float = (
+        7 if _non_linux
+        else 4
     )
+
+    if (
+        is_forking_spawner
+        and
+        tpt_proto in [
+            'uds',
+        ]
+    ):
+        timeout += 1
+
+    start: float = time.time()
+    results: list[int] = trio.run(partial(
+        cancel_after,
+        wait=timeout,
+        reg_addr=reg_addr,
+        expect_cancel=True,
+    ))
     diff: float = time.time() - start
-    assert results
+    if results is None:
+        raise ActorTooSlowError(
+            f'Streaming example took longer then timeout ??\n'
+            f'timeout={timeout!r}\n'
+            f'diff={diff!r}\n'
+            f'results={results!r}\n'
+        )
+
     return results, diff
 
 
@@ -307,11 +356,10 @@ def test_a_quadruple_example(
     given past empirical eval of this suite.
 
     '''
-    non_linux: bool = (_sys := platform.system()) != 'Linux'
 
     this_fast_on_linux: float = 3
     this_fast = (
-        6 if non_linux
+        6 if _non_linux
         else this_fast_on_linux
     )
     # ^ XXX NOTE,
@@ -348,21 +396,26 @@ def test_not_fast_enough_quad(
     reg_addr: tuple,
     time_quad_ex: tuple[list[int], float],
     cancel_delay: float,
+
     ci_env: bool,
     spawn_backend: str,
+    is_forking_spawner: bool,
+    tpt_proto: str,
+    test_log: tractor.log.StackLevelAdapter,
 ):
     '''
-    Verify we can cancel midway through the quad example and all
-    actors cancel gracefully.
+    Verify we can cancel midway through `a_quadruple_example()`, at
+    various delays, and all subactors cancel gracefully.
 
     '''
     results, diff = time_quad_ex
     delay = max(diff - cancel_delay, 0)
-    results = trio.run(
+    results: list[int] = trio.run(partial(
         cancel_after,
-        delay,
-        reg_addr,
-    )
+        wait=delay,
+        reg_addr=reg_addr,
+        expect_cancel=True,
+    ))
     system: str = platform.system()
     if (
         system in ('Windows', 'Darwin')
@@ -373,6 +426,20 @@ def test_not_fast_enough_quad(
         # so just ignore these
         print(f'Woa there {system} caught your breath eh?')
     else:
+        if (
+            results
+            and
+            is_forking_spawner
+            and
+            tpt_proto in [
+                'uds',
+            ]
+        ):
+            pytest.xfail(
+                f'Spawning backend + tpt-proto is too fast XD\n'
+                f'{spawn_backend!r} + {tpt_proto!r}\n'
+            )
+
         # should be cancelled mid-streaming
         assert results is None
 

tests/test_local.py

@@ -10,18 +10,22 @@ import tractor
 from tractor._testing import tractor_test
 
 
-@pytest.mark.trio
-async def test_no_runtime():
-    """A registrar must be established before any nurseries
+def test_no_runtime():
+    '''
+    A registrar must be established before any nurseries
     can be created.
 
     (In other words ``tractor.open_root_actor()`` must be
     engaged at some point?)
-    """
-    with pytest.raises(RuntimeError) :
+
+    '''
+    async def main():
         async with tractor.find_actor('doggy'):
             pass
 
+    with pytest.raises(tractor._exceptions.NoRuntime) :
+        trio.run(main)
+
 
 @tractor_test
 async def test_self_is_registered(reg_addr):

tests/test_log_sys.py

@@ -20,6 +20,16 @@ def test_root_pkg_not_duplicated_in_logger_name():
     a common `<root_name>.< >` prefix, ensure that it is not
     duplicated in the child's `StackLevelAdapter.name: str`.
 
+    Also pins the explicit-`name` contract: an explicitly passed
+    dotted `name` is treated as a *literal* sub-logger path and is
+    NOT leaf-collapsed. The leaf-module is only dropped when the
+    trailing token duplicates the *caller's own* `__name__` leaf (the
+    `{filename}` field) — see `test_implicit_mod_name_applied_for_child`
+    for that (auto-naming) path. This is what keeps a real (possibly
+    nested) sub-PACKAGE like `subpkg.mod` -> `devx.debug` addressable
+    by the `tractor.log` logging-spec, instead of collapsing to its
+    parent.
+
     '''
     project_name: str = 'pylib'
     pkg_path: str = 'pylib.subpkg.mod'
@@ -38,8 +48,13 @@ def test_root_pkg_not_duplicated_in_logger_name():
     )
 
     assert proj_log is not sublog
+    # the root pkg-name appears exactly once (no `pylib.pylib...`)
     assert sublog.name.count(proj_log.name) == 1
-    assert 'mod' not in sublog.name
+    # explicit dotted `name` is preserved literally (NOT collapsed);
+    # the trailing token survives since it's not the *caller's* own
+    # leaf-module (`test_log_sys`), so this is treated as a literal
+    # sub-pkg path.
+    assert sublog.name == f'{project_name}.subpkg.mod'
 
 
 def test_implicit_mod_name_applied_for_child(
@@ -147,6 +162,66 @@ def test_implicit_mod_name_applied_for_child(
     assert submod.log.logger in sub_logs
 
 
+def test_io_custom_level_registered():
+    '''
+    The `IO`(21) level (registered via `add_log_level()` at
+    import, for `tractor.trionics._subproc`'s std-stream relay)
+    is fully wired and SHOWN BY DEFAULT at `info`-level consoles
+    since `21 >= INFO(20)`.
+
+    '''
+    import logging
+    assert log.CUSTOM_LEVELS.get('IO') == 21
+    assert logging.getLevelName(21) == 'IO'
+    assert log.STD_PALETTE.get('IO')
+    assert log.BOLD_PALETTE['bold'].get('IO')
+
+    iolog = log.get_logger('io_lvl_test')
+    assert callable(getattr(iolog, 'io', None))
+    # emit must not raise
+    iolog.io('hello from the IO level')
+
+    # 21 >= INFO(20) -> shown when console set to `info`
+    assert 21 >= logging.INFO
+
+
+def test_add_log_level_pluggable():
+    '''
+    `add_log_level()` is the single pluggable entry-point: one
+    call wires `CUSTOM_LEVELS` + `addLevelName` + both palettes +
+    a same-named `StackLevelAdapter` emit method (so
+    `get_logger()`'s per-level audit passes).
+
+    '''
+    import logging
+    name: str = 'XLVL'
+    val: int = 19
+    try:
+        log.add_log_level(name, val, 'cyan')
+
+        assert log.CUSTOM_LEVELS[name] == val
+        assert logging.getLevelName(val) == name
+        assert log.STD_PALETTE[name] == 'cyan'
+        assert log.BOLD_PALETTE['bold'][name] == 'bold_cyan'
+
+        # the audit in `get_logger()` (asserts a method per
+        # `CUSTOM_LEVELS` entry) must still pass.
+        xlog = log.get_logger('xlvl_test')
+        emit = getattr(xlog, name.lower(), None)
+        assert callable(emit)
+        emit('hello from a plugged-in level')
+
+    finally:
+        # best-effort cleanup of our module-global mutations so
+        # later `get_logger()` audits don't see a half-removed
+        # level.
+        log.CUSTOM_LEVELS.pop(name, None)
+        log.STD_PALETTE.pop(name, None)
+        log.BOLD_PALETTE['bold'].pop(name, None)
+        if hasattr(log.StackLevelAdapter, name.lower()):
+            delattr(log.StackLevelAdapter, name.lower())
+
+
 # TODO, moar tests against existing feats:
 # ------ - ------
 # - [ ] color settings?

tests/test_multi_program.py

@@ -1,183 +0,0 @@
-"""
-Multiple python programs invoking the runtime.
-"""
-from __future__ import annotations
-import platform
-import subprocess
-import time
-from typing import (
-    TYPE_CHECKING,
-)
-
-import pytest
-import trio
-import tractor
-from tractor._testing import (
-    tractor_test,
-)
-from tractor import (
-    current_actor,
-    Actor,
-    Context,
-    Portal,
-)
-from tractor.runtime import _state
-from .conftest import (
-    sig_prog,
-    _INT_SIGNAL,
-    _INT_RETURN_CODE,
-)
-
-if TYPE_CHECKING:
-    from tractor.msg import Aid
-    from tractor.discovery._addr import (
-        UnwrappedAddress,
-    )
-
-
-_non_linux: bool = platform.system() != 'Linux'
-
-
-def test_abort_on_sigint(
-    daemon: subprocess.Popen,
-):
-    assert daemon.returncode is None
-    time.sleep(0.1)
-    sig_prog(daemon, _INT_SIGNAL)
-    assert daemon.returncode == _INT_RETURN_CODE
-
-    # XXX: oddly, couldn't get capfd.readouterr() to work here?
-    if platform.system() != 'Windows':
-        # don't check stderr on windows as its empty when sending CTRL_C_EVENT
-        assert "KeyboardInterrupt" in str(daemon.stderr.read())
-
-
-@tractor_test
-async def test_cancel_remote_registrar(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-):
-    assert not current_actor().is_registrar
-    async with tractor.get_registry(reg_addr) as portal:
-        await portal.cancel_actor()
-
-    time.sleep(0.1)
-    # the registrar channel server is cancelled but not its main task
-    assert daemon.returncode is None
-
-    # no registrar socket should exist
-    with pytest.raises(OSError):
-        async with tractor.get_registry(reg_addr) as portal:
-            pass
-
-
-def test_register_duplicate_name(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-):
-    async def main():
-        async with tractor.open_nursery(
-            registry_addrs=[reg_addr],
-        ) as an:
-
-            assert not current_actor().is_registrar
-
-            p1 = await an.start_actor('doggy')
-            p2 = await an.start_actor('doggy')
-
-            async with tractor.wait_for_actor('doggy') as portal:
-                assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
-
-            await an.cancel()
-
-    # XXX, run manually since we want to start this root **after**
-    # the other "daemon" program with it's own root.
-    trio.run(main)
-
-
-@tractor.context
-async def get_root_portal(
-    ctx: Context,
-):
-    '''
-    Connect back to the root actor manually (using `._discovery` API)
-    and ensure it's contact info is the same as our immediate parent.
-
-    '''
-    sub: Actor = current_actor()
-    rtvs: dict = _state._runtime_vars
-    raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
-
-    # await tractor.pause()
-    # XXX, in case the sub->root discovery breaks you might need
-    # this (i know i did Xp)!!
-    # from tractor.devx import mk_pdb
-    # mk_pdb().set_trace()
-
-    assert (
-        len(raddrs) == 1
-        and
-        list(sub._parent_chan.raddr.unwrap()) in raddrs
-    )
-
-    # connect back to our immediate parent which should also
-    # be the actor-tree's root.
-    from tractor.discovery._api import get_root
-    ptl: Portal
-    async with get_root() as ptl:
-        root_aid: Aid = ptl.chan.aid
-        parent_ptl: Portal = current_actor().get_parent()
-        assert (
-            root_aid.name == 'root'
-            and
-            parent_ptl.chan.aid == root_aid
-        )
-        await ctx.started()
-
-
-def test_non_registrar_spawns_child(
-    daemon: subprocess.Popen,
-    reg_addr: UnwrappedAddress,
-    loglevel: str,
-    debug_mode: bool,
-    ci_env: bool,
-):
-    '''
-    Ensure a non-regristar (serving) root actor can spawn a sub and
-    that sub can connect back (manually) to it's rent that is the
-    root without issue.
-
-    More or less this audits the global contact info in
-    `._state._runtime_vars`.
-
-    '''
-    async def main():
-
-        # XXX, since apparently on macos in GH's CI it can be a race
-        # with the `daemon` registrar on grabbing the socket-addr..
-        if ci_env and _non_linux:
-            await trio.sleep(.5)
-
-        async with tractor.open_nursery(
-            registry_addrs=[reg_addr],
-            loglevel=loglevel,
-            debug_mode=debug_mode,
-        ) as an:
-
-            actor: Actor = tractor.current_actor()
-            assert not actor.is_registrar
-            sub_ptl: Portal = await an.start_actor(
-                name='sub',
-                enable_modules=[__name__],
-            )
-
-            async with sub_ptl.open_context(
-                get_root_portal,
-            ) as (ctx, _):
-                print('Waiting for `sub` to connect back to us..')
-
-            await an.cancel()
-
-    # XXX, run manually since we want to start this root **after**
-    # the other "daemon" program with it's own root.
-    trio.run(main)

tests/test_pubsub.py

@@ -7,6 +7,14 @@ import tractor
 from tractor.experimental import msgpub
 from tractor._testing import tractor_test
 
+pytestmark = pytest.mark.skipon_spawn_backend(
+    'subint',
+    reason=(
+        'XXX SUBINT HANGING TEST XXX\n'
+        'See oustanding issue(s)\n'
+        # TODO, put issue link!
+    )
+)
 
 def test_type_checks():
 

tests/test_ringbuf.py

@@ -4,6 +4,10 @@ import trio
 import pytest
 
 import tractor
+
+# XXX `cffi` dun build on py3.14 yet..
+cffi = pytest.importorskip("cffi")
+
 from tractor.ipc._ringbuf import (
     open_ringbuf,
     RBToken,
@@ -14,7 +18,7 @@ from tractor._testing.samples import (
     generate_sample_messages,
 )
 
-# in case you don't want to melt your cores, uncomment dis!
+# XXX, in case you want to melt your cores, comment this skip line XD
 pytestmark = pytest.mark.skip
 
 

tests/test_shm.py

@@ -14,6 +14,20 @@ from tractor.ipc._shm import (
     attach_shm_list,
 )
 
+pytestmark = pytest.mark.skipon_spawn_backend(
+    'subint',
+    # NOTE, `main_thread_forkserver` works for these tests
+    # via the `mp.SharedMemory(track=False)` +
+    # `mp.resource_tracker` monkey-patch in `.ipc._mp_bs`.
+    # Without that workaround the fork-inherited
+    # `resource_tracker` fd would EBADF on first shm op +
+    # cascade into `FileExistsError` across parametrize
+    # variants. Tracker doc:
+    # `ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`.
+    reason=(
+        'subint: GIL-contention hanging class.\n'
+    )
+)
 
 @tractor.context
 async def child_attach_shml_alot(

tests/test_spawning.py

@@ -194,9 +194,14 @@ def test_loglevel_propagated_to_subactor(
     reg_addr: tuple,
     level: str,
 ):
-    if start_method == 'mp_forkserver':
+    if start_method in ('mp_forkserver', 'main_thread_forkserver'):
         pytest.skip(
-            "a bug with `capfd` seems to make forkserver capture not work?"
+            "a bug with `capfd` seems to make forkserver capture not work? "
+            "(same class as the `mp_forkserver` pre-existing skip — fork-"
+            "based backends inherit pytest's capfd temp-file fds into the "
+            "subactor and the IPC handshake reads garbage (`unclean EOF "
+            "read only X/HUGE_NUMBER bytes`). Work around by using "
+            "`capsys` instead or skip entirely."
         )
 
     async def main():

tests/trionics/test_patches.py

@@ -0,0 +1,99 @@
+'''
+Regression tests for `tractor.trionics.patches` —
+defensive monkey-patches on upstream `trio` bugs.
+
+Each test asserts:
+
+1. The bug exists (or is gone — skip cleanly if
+   upstream shipped the fix and our `is_needed()` now
+   returns `False`).
+2. Our patch fixes it (post-`apply()` the `repro()`
+   returns cleanly within a tight wall-clock cap).
+
+Wall-clock caps are critical here — the bugs we patch
+are tight-loops or deadlocks, so a regression would
+HANG the test runner unless we hard-cap each
+`repro()` call.
+
+'''
+import signal
+
+import pytest
+
+from tractor.trionics import patches
+from tractor.trionics.patches import _wakeup_socketpair as wsp
+
+
+@pytest.fixture(autouse=True)
+def _alarm_cleanup():
+    '''
+    Ensure no leftover SIGALRM survives a test failure
+    or unexpected return.
+
+    '''
+    yield
+    signal.alarm(0)
+
+
+def test_wakeup_socketpair_drain_eof_patch_works():
+    '''
+    Without the patch, `WakeupSocketpair.drain()` on a
+    socketpair whose write-end has been closed spins
+    forever. With the patch applied, it returns
+    cleanly within milliseconds.
+
+    Wall-clock cap: 2s. If the patch regresses, SIGALRM
+    fires and the test hard-fails with a clear signal
+    instead of hanging CI indefinitely.
+
+    '''
+    if not wsp.is_needed():
+        pytest.skip(
+            'upstream trio shipped the fix — '
+            'patch no longer needed for trio '
+            '(see `is_needed()` for version gate)'
+        )
+
+    # Apply the patch.
+    applied: bool = wsp.apply()
+    # First call MUST return True; idempotent guard
+    # prevents False on subsequent calls within the
+    # same process.
+    assert applied is True or applied is False  # idempotent
+
+    # Cap wall-clock at 2s; SIGALRM raises in main
+    # thread which interrupts the C-level recv loop
+    # IF the patch regresses (since `signal.alarm`
+    # uses Python's signal-wakeup-fd which the patch
+    # itself relies on... but `repro()` runs OUTSIDE
+    # a trio.run, so it's plain stdlib semantics here
+    # — alarm WILL fire during `recv` syscall).
+    signal.alarm(2)
+    wsp.repro()
+    signal.alarm(0)
+
+
+def test_apply_all_idempotent():
+    '''
+    Calling `apply_all()` twice should not double-
+    apply: second call's dict has all-False values
+    (every patch reports "already applied").
+
+    '''
+    first: dict[str, bool] = patches.apply_all()
+    second: dict[str, bool] = patches.apply_all()
+
+    # Second call: every patch reports skipped.
+    assert all(v is False for v in second.values()), (
+        f'apply_all() not idempotent: {second}'
+    )
+
+    # First call: at least one patch was applied
+    # (or all are no-ops because `is_needed()` is
+    # False everywhere — the all-fixed-upstream future
+    # state which is also valid).
+    assert isinstance(first, dict)
+    for name, applied in first.items():
+        assert isinstance(applied, bool), (
+            f'patch {name!r} returned non-bool: {applied!r}'
+        )

tests/trionics/test_subproc.py

@@ -0,0 +1,230 @@
+'''
+Unit tests for `tractor.trionics.supervise_run_process` (in
+`tractor.trionics._subproc`) and its per-line std-stream relay.
+
+Hermetic `trio`-only coverage (no actor-runtime needed):
+
+- per-line stdout relay -> `log.io`
+- parent controlling-tty isolation (child fd1 is a pipe, fd0
+  `/dev/null` — never the parent `/dev/pts/*`)
+- mandatory concurrent pipe-drain (no deadlock on >64KiB
+  no-newline output)
+- live stderr relay + `CalledProcessError` rebuild (rc!=0 note)
+- legacy capture-stderr CPE note path
+
+'''
+from functools import partial
+import subprocess
+
+import pytest
+import trio
+
+from tractor.trionics import (
+    _subproc,
+    collapse_eg,
+    supervise_run_process,
+)
+
+
+def _capture_relay(monkeypatch, level: str = 'io') -> list[str]:
+    '''
+    Redirect `_subproc.log.<level>` (the relay's emit method —
+    `io` by default, see `supervise_run_process(relay_level=...)`)
+    into a list so tests can assert on the relayed lines.
+
+    '''
+    records: list[str] = []
+    monkeypatch.setattr(
+        _subproc.log,
+        level,
+        lambda msg, *a, **k: records.append(msg),
+    )
+    return records
+
+
+def test_stdout_relayed_per_line(monkeypatch):
+    records = _capture_relay(monkeypatch)
+
+    cmd = [
+        'sh', '-c',
+        'for i in 1 2 3; do echo line=$i; done',
+    ]
+
+    async def main():
+        async with trio.open_nursery() as tn:
+            await tn.start(
+                partial(
+                    supervise_run_process,
+                    cmd,
+                    label='t-out',
+                    relay_stdout=True,
+                )
+            )
+
+    trio.run(main)
+
+    out_lines = [r for r in records if '[t-out:out]' in r]
+    assert any('line=1' in r for r in out_lines)
+    assert any('line=2' in r for r in out_lines)
+    assert any('line=3' in r for r in out_lines)
+
+
+def test_parent_tty_isolated(monkeypatch):
+    records = _capture_relay(monkeypatch)
+
+    cmd = [
+        'sh', '-c',
+        'readlink /proc/self/fd/0; readlink /proc/self/fd/1',
+    ]
+
+    async def main():
+        async with trio.open_nursery() as tn:
+            await tn.start(
+                partial(
+                    supervise_run_process,
+                    cmd,
+                    label='t-tty',
+                    relay_stdout=True,
+                )
+            )
+
+    trio.run(main)
+
+    relayed = '\n'.join(records)
+    # fd1 (stdout) must be OUR pipe, never a controlling tty.
+    assert 'pipe:' in relayed
+    assert '/dev/pts/' not in relayed
+    # fd0 (stdin) is pinned to DEVNULL.
+    assert '/dev/null' in relayed
+
+
+def test_no_deadlock_on_big_unnewlined_output(monkeypatch):
+    '''
+    >64KiB of output with NO newline: only completes because the
+    relay reader concurrently drains the pipe (else the child
+    blocks on `write()` when the OS pipe buffer fills).
+
+    '''
+    records = _capture_relay(monkeypatch)
+
+    cmd = [
+        'sh', '-c',
+        'head -c 200000 /dev/zero | tr "\\0" x',
+    ]
+
+    async def main():
+        # generous vs the ~ms real runtime, but bounded so a
+        # genuine pipe-fill deadlock fails fast.
+        with trio.fail_after(2):
+            async with trio.open_nursery() as tn:
+                await tn.start(
+                    partial(
+                        supervise_run_process,
+                        cmd,
+                        label='t-big',
+                        relay_stdout=True,
+                    )
+                )
+
+    trio.run(main)
+
+    big = ''.join(
+        r.split('] ', 1)[-1]
+        for r in records
+        if '[t-big:out]' in r
+    )
+    assert len(big) == 200_000
+
+
+def test_stderr_relay_and_cpe_rebuild(monkeypatch):
+    '''
+    `relay_stderr=True` PIPEs stderr ourselves (mutually
+    exclusive with trio's `capture_stderr`), so on rc!=0 the
+    wrapper rebuilds a `CalledProcessError` from the live
+    accumulator and `.add_note()`s its `.stderr` — AND the
+    stderr is relayed per-line live.
+
+    '''
+    records = _capture_relay(monkeypatch)
+
+    cmd = [
+        'sh', '-c',
+        'echo boom 1>&2; exit 3',
+    ]
+
+    async def main():
+        # `collapse_eg()` unwraps the parent-nursery's single-exc
+        # eg so the bare CPE bubbles straight out (mirrors real
+        # caller usage).
+        async with (
+            collapse_eg(),
+            trio.open_nursery() as tn,
+        ):
+            await tn.start(
+                partial(
+                    supervise_run_process,
+                    cmd,
+                    label='t-err',
+                    relay_stderr=True,
+                    check=True,
+                )
+            )
+
+    with pytest.raises(subprocess.CalledProcessError) as ei:
+        trio.run(main)
+
+    cpe = ei.value
+    assert cpe.returncode == 3
+    # rebuilt `.stderr` (trio did NOT capture since we PIPE'd it).
+    assert b'boom' in (cpe.stderr or b'')
+    # note attached for legible teardown reporting.
+    assert any(
+        'boom' in n
+        for n in getattr(cpe, '__notes__', [])
+    )
+    # AND it was relayed live per-line.
+    assert any(
+        '[t-err:err]' in r and 'boom' in r
+        for r in records
+    )
+
+
+def test_nonrelay_cpe_note(monkeypatch):
+    '''
+    No live relay: stderr is silently drained + captured (NOT
+    emitted), and on rc!=0 the wrapper rebuilds the
+    `CalledProcessError` from that accumulator with a `.stderr`
+    note — same deterministic post-drain path as the relay case.
+
+    '''
+    cmd = [
+        'sh', '-c',
+        'echo nope 1>&2; exit 7',
+    ]
+
+    async def main():
+        async with (
+            collapse_eg(),
+            trio.open_nursery() as tn,
+        ):
+            await tn.start(
+                partial(
+                    supervise_run_process,
+                    cmd,
+                    label='t-legacy',
+                    check=True,
+                    # relay_* default False -> silent
+                    # drain+capture for the CPE note.
+                )
+            )
+
+    with pytest.raises(subprocess.CalledProcessError) as ei:
+        trio.run(main)
+
+    cpe = ei.value
+    assert cpe.returncode == 7
+    assert b'nope' in (cpe.stderr or b'')
+    assert any(
+        'nope' in n
+        for n in getattr(cpe, '__notes__', [])
+    )

tractor/_child.py

@@ -15,16 +15,23 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 """
-This is the "bootloader" for actors started using the native trio backend.
+The "bootloader" for sub-actors spawned via the native `trio`
+backend (the default `python -m tractor._child` CLI entry) and
+the in-process `subint` backend (`tractor.spawn._subint`).
 
 """
+from __future__ import annotations
 import argparse
-
 from ast import literal_eval
+from typing import TYPE_CHECKING
 
 from .runtime._runtime import Actor
 from .spawn._entry import _trio_main
 
+if TYPE_CHECKING:
+    from .discovery._addr import UnwrappedAddress
+    from .spawn._spawn import SpawnMethodKey
+
 
 def parse_uid(arg):
     name, uuid = literal_eval(arg)  # ensure 2 elements
@@ -39,6 +46,73 @@ def parse_ipaddr(arg):
         return arg
 
 
+def _actor_child_main(
+    uid: tuple[str, str],
+    loglevel: str | None,
+    parent_addr: UnwrappedAddress | None,
+    infect_asyncio: bool,
+    spawn_method: SpawnMethodKey = 'trio',
+
+) -> None:
+    '''
+    Construct the child `Actor` and dispatch to `_trio_main()`.
+
+    Shared entry shape used by both the `python -m tractor._child`
+    CLI (trio/mp subproc backends) and the `subint` backend, which
+    invokes this from inside a fresh `concurrent.interpreters`
+    sub-interpreter via `Interpreter.call()`.
+
+    '''
+    # Apply defensive monkey-patches for upstream `trio`
+    # bugs we've encountered while running tractor — see
+    # `tractor.trionics.patches` for the catalog +
+    # per-patch upstream-fix tracking. Must run BEFORE
+    # any trio runtime init.
+    from .trionics.patches import apply_all
+    apply_all()
+
+    subactor = Actor(
+        name=uid[0],
+        uuid=uid[1],
+        loglevel=loglevel,
+        spawn_method=spawn_method,
+    )
+
+    # XXX, set a stable OS-level proc-title BEFORE entering
+    # the trio runtime so `ps`/`top`/`acli.pytree` and
+    # orphan-reapers can identify this actor for its full
+    # lifetime — e.g.
+    #   `tractor[doggy@1027301b]`
+    # vs. the default uninformative
+    #   `python -m tractor._child --uid (...)`
+    #
+    # `setproctitle` mutates `argv[0]` (visible in
+    # `/proc/<pid>/cmdline`) AND the kernel `comm`
+    # (visible in `/proc/<pid>/comm`, kernel-truncated to
+    # ~15 bytes, but preserved through zombie state). Both
+    # surfaces are enough for `_testing._reap` /
+    # `acli.reap` orphan- and zombie-detection to identify
+    # tractor sub-actors via intrinsic signals — no cwd,
+    # venv path, or env-var coincidence-of-implementation
+    # matching needed.
+    #
+    # NB: an earlier draft also wrote `TRACTOR_AID` to
+    # `os.environ` here for `pgrep --env`-style discovery,
+    # but Linux snapshots `/proc/<pid>/environ` at exec/fork
+    # time, so post-fork runtime mutations don't propagate
+    # to the kernel-visible env. The proc-title path
+    # provides equivalent ergonomics
+    # (`pgrep -f 'tractor\['`) without that gotcha.
+    from .devx._proctitle import set_actor_proctitle
+    set_actor_proctitle(subactor)
+
+    _trio_main(
+        subactor,
+        parent_addr=parent_addr,
+        infect_asyncio=infect_asyncio,
+    )
+
+
 if __name__ == "__main__":
     __tracebackhide__: bool = True
 
@@ -49,15 +123,10 @@ if __name__ == "__main__":
     parser.add_argument("--asyncio", action='store_true')
     args = parser.parse_args()
 
-    subactor = Actor(
-        name=args.uid[0],
-        uuid=args.uid[1],
+    _actor_child_main(
+        uid=args.uid,
         loglevel=args.loglevel,
-        spawn_method="trio"
-    )
-
-    _trio_main(
-        subactor,
         parent_addr=args.parent_addr,
         infect_asyncio=args.asyncio,
+        spawn_method='trio',
     )

tractor/_exceptions.py

@@ -89,6 +89,28 @@ class ActorFailure(RuntimeFailure):
     '''
 
 
+class ActorTooSlowError(RuntimeFailure):
+    '''
+    A peer-`Actor` failed to ack an actor-runtime cancel-cascade
+    request (e.g. `Portal.cancel_actor()` -> `Actor.cancel()`)
+    within the bounded wait window.
+
+    Distinct exc-type (NOT a `trio.TooSlowError` subclass) so that
+    `except trio.TooSlowError:` blocks elsewhere in the test-suite
+    or `tractor` internals do NOT silently mask actor-cancel
+    timeouts — these MUST propagate so a supervisor can escalate
+    to `proc.terminate()` (hard-kill) per SC-discipline:
+
+      graceful cancel-req -> bounded wait -> hard-kill
+
+    Reason: see #subint_forkserver duplicate-name hang
+    diagnosis where `Portal.cancel_actor()` silently swallowed
+    the timeout and the supervisor never escalated, leaving
+    a same-named sibling subactor parked forever.
+
+    '''
+
+
 class InternalError(RuntimeError):
     '''
     Entirely unexpected internal machinery error indicating

tractor/_root.py

@@ -69,6 +69,20 @@ from ._exceptions import (
 logger = log.get_logger('tractor')
 
 
+# Spawn backends under which `debug_mode=True` is supported.
+# Requirement: the spawned subactor's root runtime must be
+# trio-native so `tractor.devx.debug._tty_lock` works. Matches
+# both the enable-site in `open_root_actor` and the cleanup-
+# site reset of `_runtime_vars['_debug_mode']` — keep them in
+# lockstep when adding backends.
+_DEBUG_COMPATIBLE_BACKENDS: tuple[str, ...] = (
+    'trio',
+    # forkserver children run `_trio_main` in their own OS
+    # process — same child-side runtime shape as `trio_proc`.
+    'main_thread_forkserver',
+)
+
+
 # TODO: stick this in a `@acm` defined in `devx.debug`?
 # -[ ] also maybe consider making this a `wrapt`-deco to
 #     save an indent level?
@@ -141,7 +155,6 @@ async def maybe_block_bp(
             os.environ.pop('PYTHONBREAKPOINT', None)
 
 
-
 @acm
 async def open_root_actor(
     *,
@@ -172,6 +185,7 @@ async def open_root_actor(
     # enables the multi-process debugger support
     debug_mode: bool = False,
     maybe_enable_greenback: bool = False,  # `.pause_from_sync()/breakpoint()` support
+
     # ^XXX NOTE^ the perf implications of use,
     # https://greenback.readthedocs.io/en/latest/principle.html#performance
     enable_stack_on_sig: bool = False,
@@ -227,6 +241,7 @@ async def open_root_actor(
             f'_registry_addrs: {registry_addrs!r}\n'
         )
 
+    # debug.mk_pdb().set_trace()
     async with maybe_block_bp(
         debug_mode=debug_mode,
         maybe_enable_greenback=maybe_enable_greenback,
@@ -270,6 +285,75 @@ async def open_root_actor(
             )
             enable_modules.extend(rpc_module_paths)
 
+        # `TRACTOR_LOGLEVEL` env-var wins over any caller-passed
+        # `loglevel` so devs/test-runs can crank (or silence)
+        # console verbosity without touching application code.
+        env_ll_report: str = ''
+        if env_ll := os.environ.get('TRACTOR_LOGLEVEL'):
+            loglevel = env_ll
+            env_ll_report: str = (
+                f'Detected env-var setting,\n'
+                f'TRACTOR_LOGLEVEL={env_ll!r}\n'
+                f'\n'
+                f'Setting console loglevel per,\n'
+                f'loglevel={loglevel!r}\n'
+            )
+            if (
+                loglevel
+                and
+                loglevel.upper() != env_ll.upper()
+            ):
+                env_ll_report += (
+                    f'\n'
+                    f'NOTE env-var OVERRIDES caller-passed,\n'
+                    f'loglevel={loglevel!r}\n'
+                )
+
+        loglevel: str = (
+            loglevel
+            or
+            log._default_loglevel
+        )
+        loglevel: str = loglevel.upper()
+
+        assert loglevel
+        _log = log.get_console_log(
+            level=loglevel,
+            name='tractor',
+            logger=logger,
+        )
+        assert _log
+        if env_ll_report:
+            _log.info(env_ll_report)
+
+        # `TRACTOR_SPAWN_METHOD` env-var wins over any caller-passed
+        # `start_method` so devs/test-runs can swap the actor spawn
+        # backend without touching application code (e.g. driving
+        # the `examples/debugging/<script>.py` suite under each
+        # backend from `tests/devx/conftest.py::spawn`).
+        if env_sm := os.environ.get('TRACTOR_SPAWN_METHOD'):
+            start_method: str = env_sm
+            env_sm_report: str = (
+                f'Detected env-var setting,\n'
+                f'TRACTOR_SPAWN_METHOD={env_sm!r}\n'
+                f'\n'
+                f'Setting spawn backend as,\n'
+                f'start_method={env_sm!r}\n'
+            )
+            if (
+                start_method
+                and
+                start_method != env_sm
+            ):
+                _log.warning(
+                    env_sm_report
+                    +
+                    f'NOTE env-var OVERRIDES caller-passed,\n'
+                    f'`start_method={start_method!r}`\n'
+                )
+            else:
+                _log.info(env_sm_report)
+
         if start_method is not None:
             _spawn.try_set_start_method(start_method)
 
@@ -286,17 +370,44 @@ async def open_root_actor(
             wrap_address(uw_addr)
             for uw_addr in uw_reg_addrs
         ]
-        loglevel: str = (
-            loglevel
-            or
-            log._default_loglevel
-        )
-        loglevel: str = loglevel.upper()
 
+        # fail-fast on `enable_transports` / `registry_addrs` proto
+        # mismatch — historically this caused a silent indefinite
+        # hang during the registrar handshake (registry was reachable
+        # only via a transport not in `enable_transports`, so the
+        # actor could never connect to register/discover). See
+        # `tests/ipc/test_multi_tpt.py::test_root_passes_tpt_to_sub`
+        # for the foot-gun case + its layer-1 skip-guard.
+        bad_addrs: list[tuple[str, Address]] = [
+            (addr.proto_key, addr)
+            for addr in registry_addrs
+            if addr.proto_key not in enable_transports
+        ]
+        if bad_addrs:
+            mismatch_lines: str = '\n'.join(
+                f'  - proto_key={pk!r}  addr={a!r}'
+                for pk, a in bad_addrs
+            )
+            raise ValueError(
+                f'`registry_addrs` contains addr(s) whose proto is '
+                f'not in `enable_transports`!\n'
+                f'enable_transports: {enable_transports!r}\n'
+                f'mismatched_addrs:\n'
+                f'{mismatch_lines}\n'
+                f'\n'
+                f'Either add the missing proto to '
+                f'`enable_transports`, or remove the addr from '
+                f'`registry_addrs`.'
+            )
+
+        # Debug-mode is currently only supported for backends whose
+        # subactor root runtime is trio-native (so `tractor.devx.
+        # debug._tty_lock` works). See `_DEBUG_COMPATIBLE_BACKENDS`
+        # module-const for the list.
         if (
             debug_mode
             and
-            _spawn._spawn_method == 'trio'
+            _spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
         ):
             _state._runtime_vars['_debug_mode'] = True
 
@@ -318,22 +429,18 @@ async def open_root_actor(
 
         elif debug_mode:
             raise RuntimeError(
-                "Debug mode is only supported for the `trio` backend!"
+                f'Debug mode currently supported only for '
+                f'{_DEBUG_COMPATIBLE_BACKENDS!r} spawn backends, not '
+                f'{_spawn._spawn_method!r}.'
             )
 
-        assert loglevel
-        _log = log.get_console_log(
-            level=loglevel,
-            name='tractor',
-        )
-        assert _log
-
         # TODO: factor this into `.devx._stackscope`!!
-        if (
-            debug_mode
-            and
-            enable_stack_on_sig
-        ):
+        #
+        # NOTE, intentionally NOT gated on `debug_mode` so SIGUSR1
+        # task-tree dumps work in plain (non-pdb) runs too — esp.
+        # in infected-`asyncio` root processes where the default
+        # SIGUSR1 action would otherwise terminate the proc.
+        if enable_stack_on_sig:
             from .devx._stackscope import enable_stack_on_sig
             enable_stack_on_sig()
 
@@ -619,7 +726,7 @@ async def open_root_actor(
             if (
                 debug_mode
                 and
-                _spawn._spawn_method == 'trio'
+                _spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
             ):
                 _state._runtime_vars['_debug_mode'] = False
 

tractor/_testing/addr.py

@@ -22,6 +22,7 @@ Might be eventually useful to expose as a util set from
 our `tractor.discovery` subsys?
 
 '''
+import os
 import random
 from typing import (
     Type,
@@ -31,17 +32,28 @@ from tractor.discovery import _addr
 
 def get_rando_addr(
     tpt_proto: str,
-    *,
-
-    # choose random port at import time
-    _rando_port: str = random.randint(1000, 9999)
-
 ) -> tuple[str, str|int]:
     '''
     Used to globally override the runtime to the
     per-test-session-dynamic addr so that all tests never conflict
     with any other actor tree using the default.
 
+    Cross-process isolation: TCP-port picks salt
+    `random.randint()` with `os.getpid()` so two parallel
+    pytest sessions (e.g. one running `--tpt-proto=tcp` and
+    another `--tpt-proto=uds` concurrently) almost-never
+    collide on the same port. Without the salt, the prior
+    impl's import-time `random.randint(1000, 9999)` default
+    arg was effectively a process-singleton with a 1/9000
+    chance of cross-run collision per pair — and when it
+    happened EVERY `reg_addr`-using test in BOTH runs would
+    fight over the bind, cascading into a chain of
+    "Address already in use" failures.
+
+    For UDS this concern doesn't apply: `UDSAddress.get_random()`
+    already builds socket paths from `os.getpid()` so each
+    pytest process gets its own socket-path namespace.
+
     '''
     addr_type: Type[_addr.Addres] = _addr._address_types[tpt_proto]
     def_reg_addr: tuple[str, int] = _addr._default_lo_addrs[tpt_proto]
@@ -51,9 +63,21 @@ def get_rando_addr(
     testrun_reg_addr: tuple[str, int|str]
     match tpt_proto:
         case 'tcp':
+            # Per-call randomness mixed with `os.getpid()` —
+            # see the docstring above for the cross-process
+            # isolation rationale. The mix means:
+            # - within one pytest session, two calls return
+            #   distinct ports (good for tests that need a
+            #   second-different-reg-addr in one fn body, e.g.
+            #   `test_tpt_bind_addrs::bind-subset-reg`),
+            # - across parallel pytest sessions, the pid bias
+            #   makes coincident port choices unlikely.
+            port: int = 1000 + (
+                random.randint(0, 8999) + os.getpid()
+            ) % 9000
             testrun_reg_addr = (
                 addr_type.def_bindspace,
-                _rando_port,
+                port,
             )
 
         # NOTE, file-name uniqueness (no-collisions) will be based on

tractor/_testing/pytest.py

@@ -24,16 +24,181 @@ from functools import (
     wraps,
 )
 import inspect
+import os
 import platform
 from typing import (
     Callable,
     get_args,
+    TYPE_CHECKING,
 )
+import warnings
 
 import pytest
 import tractor
+from tractor.spawn._spawn import SpawnMethodKey
 import trio
 
+# Re-export `_testing.trace`'s pytest fixtures so they're
+# picked up by pytest's plugin-discovery (this module is
+# loaded via `pytest_plugins` from `pyproject.toml`). The
+# `noqa: F401` annotations make linters tolerate the
+# unused-looking imports — they're load-bearing for pytest
+# discovery. The fixtures share their `name=` kw with the
+# underlying CM functions; the python-level identifiers
+# below carry the `_fixture` suffix to avoid module-scope
+# collision (see `_testing/trace.py` for details).
+from .trace import (  # noqa: F401
+    afk_alarm_w_trace_fixture,
+    fail_after_w_trace_fixture,
+)
+
+# Sub-plugin: zombie-subactor + UDS sock-file + shm
+# reaping fixtures live in `tractor._testing._reap`
+# alongside the underlying detection/cleanup helpers.
+# Loading `_reap` as a sub-plugin here keeps reaping
+# concerns co-located + this module focused on tractor-
+# tooling-specific hooks (option/marker/parametrize,
+# `tractor_test` deco, transport / spawn-method
+# fixtures).
+pytest_plugins: tuple[str, ...] = (
+    'tractor._testing._reap',
+)
+
+if TYPE_CHECKING:
+    from argparse import Namespace
+
+
+_cap_sys_passed_as_flag: bool = False
+
+# Spawn backends that need `--capture=sys` to avoid the
+# fork-child×pytest-capture-fd deadlock. See the long
+# NOTE in `pytest_load_initial_conftests` below for the
+# full mechanism + tradeoff write-up.
+_CAPSYS_REQUIRED_SPAWNERS: frozenset[str] = frozenset({
+    'main_thread_forkserver',
+    # TODO future variant-2 'subint_forkserver' lands
+    # here too once the impl is unblocked.
+})
+
+
+# XXX REQUIRED in order to enforce `--capture=` flag
+# pre test session.
+# https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
+@pytest.hookimpl(tryfirst=True)
+def pytest_load_initial_conftests(
+    early_config: pytest.Config,
+    parser: pytest.Parser,
+    args: list[str],
+):
+    '''
+    Validate the `--capture=` × `--spawn-backend=`
+    combination at session-startup.
+
+    Background
+    ----------
+    `--capture=sys` is REQUIRED for fork-based spawn backends (e.g.
+    `main_thread_forkserver`): default `--capture=fd` redirects fd
+    1,2 to temp files, and fork children inherit those fds — opaque
+    deadlocks happen in the pytest-capture-machinery ↔ fork-child
+    stdio interaction. `--capture=sys` only redirects Python- level
+    `sys.stdout`/`sys.stderr`, leaving fd 1,2 alone.
+
+    Trade-off (vs. `--capture=fd`):
+
+    - LOST: per-test attribution of subactor *raw-fd* output (C-ext
+      writes, `os.write(2, ...)`, subproc stdout). Not zero — those
+      go to the terminal, captured by CI's terminal-level capture,
+      just not per-test-scoped in the pytest failure report.
+
+    - KEPT: Python-level `print()` + `logging` capture per-test
+      (tractor's logger uses `sys.stderr`, so tractor log output IS
+      still attributed per-test).
+
+    - KEPT: user `pytest -s` for debugging (unaffected).
+
+    Full post-mortem in
+    `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
+
+    Validation policy:
+    - **CI mode** (`CI` env-var set): fail-fast at
+      session start if a fork-spawn backend is requested
+      WITHOUT `--capture=sys`. CI must be explicit; no
+      auto-fallbacks. Forces every CI matrix-row's run
+      line to declare its capture mode plainly.
+    - **Local mode** (no `CI` env-var): emit a loud
+      warning + suggest `--capture=sys`, but allow the
+      run to proceed. Lets devs experiment with the bad
+      combo (e.g. to validate whether recent
+      fork-survival fixes have made `--capture=fd` work
+      after all).
+
+    '''
+    global _cap_sys_passed_as_flag
+    opts_w_args: Namespace = parser.parse_known_args(args)
+    spawner: str|None = getattr(
+        opts_w_args,
+        'spawn_backend',
+        None,
+    )
+    capture: str|None = getattr(
+        opts_w_args,
+        'capture',
+        None,
+    )
+    if '--capture=sys' in args:
+        _cap_sys_passed_as_flag = True
+        assert capture == 'sys'
+
+    in_ci: bool = bool(os.environ.get('CI'))
+
+    if (
+        spawner in _CAPSYS_REQUIRED_SPAWNERS
+        and
+        capture == 'fd'
+    ):
+        msg: str = (
+            f'\n'
+            f'XXX `--spawn-backend={spawner}` REQUIRES '
+            f'`--capture=sys` XXX\n'
+            f'fork-child × `--capture=fd` is a known '
+            f'deadlock pattern.\n'
+            f'See `tractor._testing.pytest`\'s '
+            f'`pytest_load_initial_conftests` docstring '
+            f'for the full mechanism.\n'
+            f'\n'
+            f'Re-invoke with `--capture=sys` (or run '
+            f'with `pytest -s` for no capture).\n'
+        )
+        # fail-fast: CI must declare capture explicitly for
+        # fork-spawn backends.
+        if in_ci:
+            pytest.exit(
+                f'{msg}\n'
+                f'FAIL-FAST: CI=1 detected; aborting session.\n',
+                returncode=2,
+            )
+
+        # local: loud warn but let the run proceed so devs can
+        # experiment.
+        else:
+            warnings.warn(
+                f'{msg}\n'
+                f'Local mode (no `CI` env var) — '
+                f'continuing. Expect potential hangs.\n',
+                category=UserWarning,
+                stacklevel=1,
+            )
+            # ??TODO?? is there a way to force the `--capture=sys` sin CLI ??
+            # - [x] ask pytest peeps in chat!
+            # - [x] pytest` issue,
+            #       https://github.com/pytest-dev/pytest/issues/14444
+
+    # TODO, set various `$TRACTOR_X*` osenv vars here!
+    print(
+        f'Applying `tractor`-specific `pytest` config,\n'
+        f'{opts_w_args!r}\n'
+    )
+
 
 def tractor_test(
     wrapped: Callable|None = None,
@@ -112,11 +277,17 @@ def tractor_test(
     # injection (via `__wrapped__`) without leaking the async
     # nature.
     @wraps(wrapped)
-    def wrapper(**kwargs):
+    def wrapper(
+        set_fork_aware_capture: pytest.CaptureFixture|None = None,
+        # ^NOTE when set, the decorated fn declared as fixture-param.
+
+        **kwargs,
+    ):
         __tracebackhide__: bool = hide_tb
 
         # NOTE, ensure we inject any test-fn declared fixture
         # names.
+        sig = inspect.signature(wrapped)
         for kw in [
             'reg_addr',
             'loglevel',
@@ -125,9 +296,13 @@ def tractor_test(
             'tpt_proto',
             'timeout',
         ]:
-            if kw in inspect.signature(wrapped).parameters:
+            if kw in sig.parameters:
                 assert kw in kwargs
 
+        if 'set_fork_aware_capture' in sig.parameters:
+            assert set_fork_aware_capture
+            kwargs['set_fork_aware_capture'] = set_fork_aware_capture
+
         # Extract runtime settings as locals for
         # `open_root_actor()`; these must NOT leak into
         # `kwargs` when the test fn doesn't declare them
@@ -170,7 +345,6 @@ def tractor_test(
                     # invoke test-fn body IN THIS task
                     await wrapped(**kwargs)
 
-        # invoke runtime via a root task.
         return trio.run(
             partial(
                 _main,
@@ -184,13 +358,6 @@ def tractor_test(
 def pytest_addoption(
     parser: pytest.Parser,
 ):
-    # parser.addoption(
-    #     "--ll",
-    #     action="store",
-    #     dest='loglevel',
-    #     default='ERROR', help="logging level to set when testing"
-    # )
-
     parser.addoption(
         "--spawn-backend",
         action="store",
@@ -212,6 +379,21 @@ def pytest_addoption(
         ),
     )
 
+    parser.addoption(
+        "--enable-stackscope",
+        action="store_true",
+        dest='enable_stackscope',
+        default=False,
+        help=(
+            'Install `stackscope` SIGUSR1 handler in pytest + '
+            'every spawned subactor for live trio task-tree '
+            'dumps during hang investigations. Lighter than '
+            '`--tpdb` (no pdb machinery / tty-lock contention) '
+            '— use when you only need stack visibility. To '
+            'capture: `kill -USR1 <pytest-or-subactor-pid>`.'
+        ),
+    )
+
     # provide which IPC transport protocols opting-in test suites
     # should accumulatively run against.
     parser.addoption(
@@ -223,11 +405,66 @@ def pytest_addoption(
         help="Transport protocol to use under the `tractor.ipc.Channel`",
     )
 
+    # console loglevel for the test-session, scoped to the
+    # consuming-project's OWN pkg-hierarchy (see the
+    # `testing_pkg_name` fixture). For `tractor` itself this IS the
+    # runtime loglevel; downstream projects use `--ll` for their own
+    # ("internal") app-logging and `--tl` for tractor-as-runtime.
+    parser.addoption(
+        "--ll",
+        "--loglevel",
+        action="store",
+        dest='loglevel',
+        default=None,
+        help=(
+            "console loglevel to set for the test session, scoped to "
+            "the consuming-project pkg (see `testing_pkg_name`). "
+            "Falls through as the `--tl` default."
+        ),
+    )
 
-def pytest_configure(config):
-    backend = config.option.spawn_backend
+    # tractor-as-runtime loglevel, DISTINCT from `--ll` so downstream
+    # projects can split their app-logs from the `tractor.*` runtime
+    # hierarchy. Accepts a `tractor.log` "logging-spec" (see
+    # `tractor.log.apply_logspec()`).
+    parser.addoption(
+        "--tl",
+        "--tractor-loglevel",
+        action="store",
+        dest='tractor_loglevel',
+        default=None,
+        help=(
+            "loglevel (or logging-spec) for `tractor`-as-runtime, "
+            "distinct from `--ll`. Accepts a bare level (eg. "
+            "'info', 'cancel') or a sub-logger filter-spec, "
+            "'<sublog>:<level>,...' (eg. "
+            "'devx:runtime,trionics:cancel'). Falls back to `--ll` "
+            "when unset. Mirrors the logging-spec grammar consumed "
+            "by `tractor.log.apply_logspec()` (see its sub-pkg "
+            "granularity caveat)."
+        ),
+    )
+
+
+def pytest_configure(
+    config: pytest.Config,
+):
+    # opts: Namespace = config.option
+    # print(
+    #     f'PYTEST_CONFIGURE\n'
+    #     f'capture={opts.capture!r}\n'
+    # )
+    # breakpoint()
+
+    backend: str = config.option.spawn_backend
     from tractor.spawn._spawn import try_set_start_method
-    try_set_start_method(backend)
+    try:
+        try_set_start_method(backend)
+    except RuntimeError as err:
+        # e.g. `--spawn-backend=subint` on Python < 3.14 — turn the
+        # runtime gate error into a clean pytest usage error so the
+        # suite exits with a helpful banner instead of a traceback.
+        raise pytest.UsageError(str(err)) from err
 
     # register custom marks to avoid warnings see,
     # https://docs.pytest.org/en/stable/how-to/writing_plugins.html#registering-custom-markers
@@ -235,10 +472,109 @@ def pytest_configure(config):
         'markers',
         'no_tpt(proto_key): test will (likely) not behave with tpt backend'
     )
+    config.addinivalue_line(
+        'markers',
+        'skipon_spawn_backend(*start_methods, reason=None): '
+        'skip this test under any of the given `--spawn-backend` '
+        'values; useful for backend-specific known-hang / -borked '
+        'cases (e.g. the `subint` GIL-starvation class documented '
+        'in `ai/conc-anal/subint_sigint_starvation_issue.md`).'
+    )
+
+    # `--enable-stackscope`: install SIGUSR1 → trio task-tree
+    # dump in pytest itself + propagate to every subactor via
+    # an env var that fork-children inherit and the runtime
+    # gate honors. Lighter than `--tpdb` (no pdb machinery) —
+    # purely for hang-investigation stack visibility.
+    if getattr(
+        config.option,
+        'enable_stackscope',
+        False
+    ):
+        # Env var inherited via fork → subactor's runtime
+        # picks it up at `Actor.async_main` startup. See the
+        # gate in `tractor.runtime._runtime` matching this
+        # var name.
+        os.environ['TRACTOR_ENABLE_STACKSCOPE'] = '1'
+
+        # Install in pytest itself so `kill -USR1 <pytest>`
+        # dumps the parent trio task-tree (which is where
+        # most Mode-A-class hangs park).
+        try:
+            from tractor.devx._stackscope import (
+                enable_stack_on_sig,
+            )
+            enable_stack_on_sig()
+        except ImportError:
+            warnings.warn(
+                '`stackscope` not installed — '
+                '--enable-stackscope is a no-op. '
+                'Install via the `devx` dep group.'
+            )
+    else:
+        os.environ.pop('TRACTOR_ENABLE_STACKSCOPE', None)
+
+
+def pytest_collection_modifyitems(
+    config: pytest.Config,
+    items: list[pytest.Function],
+):
+    '''
+    Expand any `@pytest.mark.skipon_spawn_backend('<backend>'[,
+    ...], reason='...')` markers into concrete
+    `pytest.mark.skip(reason=...)` calls for tests whose
+    backend-arg set contains the active `--spawn-backend`.
+
+    Uses `item.iter_markers(name=...)` which walks function +
+    class + module-level marks in the correct scope order (and
+    handles both the single-`MarkDecorator` and `list[Mark]`
+    forms of a module-level `pytestmark`) — so the same marker
+    works at any level a user puts it.
+
+    '''
+    backend: str = config.option.spawn_backend
+    default_reason: str = f'Borked on --spawn-backend={backend!r}'
+    for item in items:
+        for mark in item.iter_markers(name='skipon_spawn_backend'):
+            skip_backends: tuple[str] = mark.args
+            for skip_backend in skip_backends:
+                assert skip_backend in get_args(SpawnMethodKey)
+            # ?TODO, run these through the try-set-backend checker to
+            # avoid typos?
+            if backend in skip_backends:
+                reason: str = mark.kwargs.get(
+                    'reason',
+                    default_reason,
+                )
+                item.add_marker(pytest.mark.skip(reason=reason))
+                # first matching mark wins; no value in stacking
+                # multiple `skip`s on the same item.
+                break
+
+
+@pytest.fixture(
+    scope="session",
+    autouse=True,
+)
+def alert_on_finish():
+    '''
+    Ring a terminal notification on full test session
+    completion to alert any would be human.
+
+    '''
+    # TODO, check attached to tty or skip!
+    yield  # run all tests
+    print("\a")  # trigger terminal bell
+    # ?TODO, any other nice-tricks/specific tuis we could try?
+    # - supposedly works in many terminals:
+    #   >> print("\033]5;Alert: Tests Finished\a")
+    # - sway/i3-nag?
 
 
 @pytest.fixture(scope='session')
-def debug_mode(request) -> bool:
+def debug_mode(
+    request: pytest.FixtureRequest,
+) -> bool:
     '''
     Flag state for whether `--tpdb` (for `tractor`-py-debugger)
     was passed to the test run.
@@ -252,12 +588,145 @@ def debug_mode(request) -> bool:
 
 
 @pytest.fixture(scope='session')
-def spawn_backend(request) -> str:
+def testing_pkg_name() -> str:
+    '''
+    Root pkg-name of the project consuming this plugin, used to
+    scope `--ll` "internal"/app-level console logging into that
+    project's OWN `tractor.log.get_logger(pkg_name=<.>)` hierarchy
+    — distinct from the `tractor.*` runtime hierarchy configured
+    via `--tl`.
+
+    Defaults to `'tractor'` (so tractor's own suite treats `--ll`
+    as the runtime level). Downstream projects override this from
+    their `conftest.py`, eg.
+
+    .. code:: python
+
+        @pytest.fixture(scope='session')
+        def testing_pkg_name() -> str:
+            return 'modden'
+
+    '''
+    return 'tractor'
+
+
+@pytest.fixture(
+    scope='session',
+    autouse=True,
+)
+def loglevel(
+    request: pytest.FixtureRequest,
+    testing_pkg_name: str,
+) -> str|None:
+    '''
+    Resolve + apply the test-session console loglevels and yield
+    the `tractor`-runtime level (also passed to
+    `open_root_actor(loglevel=<.>)` by `@tractor_test`).
+
+    - `--tl <logspec>`: tractor-runtime level (falls back to the
+      generic `--ll`); applied to the `tractor.*` logger hierarchy
+      and `tractor.log._default_loglevel` via
+      `tractor.log.apply_logspec()`.
+    - `--ll <level>`: the consuming-project's OWN console loglevel,
+      applied to its `testing_pkg_name` hierarchy when that isn't
+      `tractor` itself.
+
+    '''
+    import tractor
+    orig: str = tractor.log._default_loglevel
+
+    ll: str|None = request.config.option.loglevel
+    tl: str|None = request.config.option.tractor_loglevel
+
+    # tractor-runtime loglevel: explicit `--tl` wins, else fall
+    # back to the generic `--ll`, else leave the lib default.
+    logspec: str|None = tl if tl is not None else ll
+    tractor_level: str|None = None
+    if logspec is not None:
+        tractor_level, _ = tractor.log.apply_logspec(
+            logspec,
+            default_level=ll,
+            pkg_name='tractor',
+        )
+        if tractor_level is not None:
+            tractor.log._default_loglevel = tractor_level
+
+    # consuming-project ("internal") console logging at the generic
+    # `--ll` level, scoped to ITS OWN pkg-hierarchy (NOT `tractor.*`)
+    # so downstream projects can split app-logs from runtime-logs.
+    if (
+        ll is not None
+        and
+        testing_pkg_name
+        and
+        testing_pkg_name != 'tractor'
+    ):
+        tractor.log.get_console_log(
+            level=ll,
+            pkg_name=testing_pkg_name,
+            name=testing_pkg_name,
+        )
+
+    log = tractor.log.get_console_log(
+        level=tractor_level,
+        name='tractor',  # <- enable root logger
+    )
+    log.info(
+        f'Test-harness set session loglevels:\n'
+        f'tractor-runtime (`--tl`/`--ll`): {tractor_level!r}\n'
+        f'{testing_pkg_name!r} (`--ll`): {ll!r}\n'
+    )
+    yield tractor_level
+    tractor.log._default_loglevel = orig
+
+
+@pytest.fixture(scope='function')
+def test_log(
+    request: pytest.FixtureRequest,
+    loglevel: str,
+    testing_pkg_name: str,
+) -> tractor.log.StackLevelAdapter:
+    '''
+    Deliver a per test-module-fn logger instance for reporting from
+    within actual test bodies/fixtures.
+
+    For example this can be handy to report certain error cases from
+    exception handlers using `test_log.exception()`.
+
+    The logger is scoped to the consuming-project's
+    `testing_pkg_name` hierarchy so downstream suites' in-test logs
+    land under their own pkg, not `tractor.*`.
+
+    '''
+    modname: str = request.function.__module__
+    log = tractor.log.get_logger(
+        name=modname,
+        pkg_name=testing_pkg_name,
+    )
+    _log = tractor.log.get_console_log(
+        level=loglevel,
+        logger=log,
+        name=modname,
+    )
+    _log.debug(
+        f'In-test-logging requested\n'
+        f'test_log.name: {log.name!r}\n'
+        f'level: {loglevel!r}\n'
+    )
+    yield _log
+
+
+@pytest.fixture(scope='session')
+def spawn_backend(
+    request: pytest.FixtureRequest,
+) -> str:
     return request.config.option.spawn_backend
 
 
 @pytest.fixture(scope='session')
-def tpt_protos(request) -> list[str]:
+def tpt_protos(
+    request: pytest.FixtureRequest,
+) -> list[str]:
 
     # allow quoting on CLI
     proto_keys: list[str] = [
@@ -285,7 +754,7 @@ def tpt_protos(request) -> list[str]:
     autouse=True,
 )
 def tpt_proto(
-    request,
+    request: pytest.FixtureRequest,
     tpt_protos: list[str],
 ) -> str:
     proto_key: str = tpt_protos[0]
@@ -337,7 +806,6 @@ def pytest_generate_tests(
     metafunc: pytest.Metafunc,
 ):
     spawn_backend: str = metafunc.config.option.spawn_backend
-
     if not spawn_backend:
         # XXX some weird windows bug with `pytest`?
         spawn_backend = 'trio'
@@ -345,7 +813,6 @@ def pytest_generate_tests(
     # drive the valid-backend set from the canonical `Literal` so
     # adding a new spawn backend (e.g. `'subint'`) doesn't require
     # touching the harness.
-    from tractor.spawn._spawn import SpawnMethodKey
     assert spawn_backend in get_args(SpawnMethodKey)
 
     # NOTE: used-to-be-used-to dyanmically parametrize tests for when
@@ -356,7 +823,8 @@ def pytest_generate_tests(
         metafunc.parametrize(
             "start_method",
             [spawn_backend],
-            scope='module',
+            scope='session',
+            ids=lambda item: f'start_method={spawn_backend}',
         )
 
     # TODO, parametrize any `tpt_proto: str` declaring tests!
@@ -367,3 +835,136 @@ def pytest_generate_tests(
     #         proto_tpts,  # TODO, double check this list usage!
     #         scope='module',
     #     )
+
+
+def _is_forking_spawner(
+    start_method: str,
+) -> bool:
+    return start_method in [
+        'main_thread_forkserver',
+        'mp_forkserver',
+    ]
+
+
+@pytest.fixture(scope='session')
+def is_forking_spawner(
+    start_method: str,
+) -> bool:
+    '''
+    Is the `pytest` run using a `fork()`ing process spawning-backend?
+
+    '''
+    return _is_forking_spawner(start_method)
+
+
+def maybe_xfail_for_spawner(
+    request: pytest.FixtureRequest,
+    start_method: str,
+    is_forking_spawner: bool,
+) -> None:
+    '''
+    Fork based spawning backends cause issues with
+    `pytest`'s fd-capture mechanism and can cause various
+    suites to hang.
+
+    This helper allows skipping/xfailing from a test when
+    a fork-spawn backend is being used WITHOUT
+    `--capture=sys`.
+
+    '''
+    capture_mode: str = request.config.option.capture
+    # `tee-sys` is also sys-level capture (just additionally writes
+    # to the original `sys.__stdout__/__stderr__`); fork-safe like
+    # `sys`. Only `fd`-level capture is the deadlock pattern.
+    if (
+        capture_mode not in (
+            'sys',
+            'tee-sys',
+        )
+        and
+        is_forking_spawner
+    ):
+        pytest.skip(
+            f'Spawner {start_method!r} requires the flag,\n'
+            f'--capture=sys or --capture=tee-sys..\n'
+            f'(got --capture={capture_mode!r})\n'
+        )
+
+
+def maybe_override_capture(
+    request: pytest.FixtureRequest,
+    start_method: bool,
+) -> str:
+    if _is_forking_spawner(start_method):
+        request.getfixturevalue('capsys')
+        return 'sys'
+
+    return request.config.option.capture
+
+
+@pytest.fixture
+def set_fork_aware_capture(
+    request: pytest.FixtureRequest,
+    start_method: str,
+) -> pytest.CaptureFixture|str:
+    '''
+    Force `--capture=sys` method for tests using
+    a forking-spawner backend due to fd-copying issues
+    which can oddly make certain tests hang/fail.
+
+    '''
+    # Fast-path: user already passed sys-level capture
+    # (`sys` or `tee-sys`) at the CLI — no override needed.
+    if request.config.option.capture in (
+            'sys',
+            'tee-sys',
+    ):
+        return request.config.option.capture
+
+    capsys: pytest.CaptureFixture = maybe_override_capture(
+        request=request,
+        start_method=start_method,
+    )
+    return capsys
+    # XXX reset?
+    # with capsys.disabled():
+    #     pass
+    # return partial(
+    #     maybe_override_capture,
+    #     request=request,
+    #     start_method=start_method,
+    # )
+
+
+def pytest_terminal_summary(
+    terminalreporter,
+    exitstatus: int,
+    config: pytest.Config,
+) -> None:
+    '''
+    End-of-session summary: list all
+    `fail_after_w_trace`/`afk_alarm_w_trace` snapshot dirs
+    captured during the run so the human doesn't have to scroll
+    back through captured-stderr lines to find dump paths.
+
+    Reads from `tractor._testing.trace._SNAPSHOT_INDEX` which is
+    populated by `_do_capture_snapshot()` on each successful
+    snapshot capture.
+
+    No-op when zero snapshots were captured (most sessions).
+
+    '''
+    from .trace import _SNAPSHOT_INDEX
+
+    if not _SNAPSHOT_INDEX:
+        return
+
+    tr = terminalreporter
+    tr.write_sep('=', 'tractor hang-snapshot index')
+    tr.write_line(
+        f'{len(_SNAPSHOT_INDEX)} `fail_after_w_trace` / '
+        f'`afk_alarm_w_trace` snapshot(s) captured this session:'
+    )
+    for label, path in _SNAPSHOT_INDEX:
+        tr.write_line(f'  {label}')
+        tr.write_line(f'    → {path}')

tractor/devx/__init__.py

@@ -41,6 +41,11 @@ from .pformat import (
     pformat_caller_frame as pformat_caller_frame,
     pformat_boxed_tb as pformat_boxed_tb,
 )
+from ._debug_hangs import (
+    dump_on_hang as dump_on_hang,
+    track_resource_deltas as track_resource_deltas,
+    resource_delta_fixture as resource_delta_fixture,
+)
 
 
 # TODO, move this to a new `.devx._pdbp` mod?

tractor/devx/_debug_hangs.py

@@ -0,0 +1,227 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Hang-diagnostic helpers for concurrent / multi-interpreter code.
+
+Collected from the `subint` spawn backend bringup (issue #379)
+where silent test-suite hangs needed careful teardown
+instrumentation to diagnose. This module bottles up the
+techniques that actually worked so future hangs are faster
+to corner.
+
+Two primitives:
+
+1. `dump_on_hang()` — context manager wrapping
+   `faulthandler.dump_traceback_later()` with the critical
+   gotcha baked in: write the dump to a **file**, not
+   `sys.stderr`. Under `pytest` (and any other output
+   capturer) stderr gets swallowed and the dump is easy to
+   miss — burning hours convinced you're looking at the wrong
+   thing.
+
+2. `track_resource_deltas()` — context manager (+ optional
+   autouse-fixture factory) logging per-block deltas of
+   `threading.active_count()` and — if running on py3.13+ —
+   `len(_interpreters.list_all())`. Lets you quickly rule out
+   leak-accumulation theories when a suite hangs more
+   frequently as it progresses (if counts don't grow, it's
+   not a leak; look for a race on shared cleanup instead).
+
+See issue #379 / commit `26fb820` for the worked example.
+
+'''
+from __future__ import annotations
+import faulthandler
+import sys
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+from typing import (
+    Callable,
+    Iterator,
+)
+
+try:
+    import _interpreters  # type: ignore
+except ImportError:
+    _interpreters = None  # type: ignore
+
+
+__all__ = [
+    'dump_on_hang',
+    'track_resource_deltas',
+    'resource_delta_fixture',
+]
+
+
+@contextmanager
+def dump_on_hang(
+    seconds: float = 30.0,
+    *,
+    path: str | Path = '/tmp/tractor_hang.dump',
+    all_threads: bool = True,
+
+) -> Iterator[str]:
+    '''
+    Arm `faulthandler` to dump all-thread tracebacks to
+    `path` after `seconds` if the with-block hasn't exited.
+
+    *Writes to a file, not stderr* — `pytest`'s stderr
+    capture silently eats stderr-destined `faulthandler`
+    output, and the same happens under any framework that
+    redirects file-descriptors. Pointing the dump at a real
+    file sidesteps that.
+
+    Yields the resolved file path so it's easy to read back.
+
+    Example
+    -------
+    ::
+
+        from tractor.devx import dump_on_hang
+
+        def test_hang():
+            with dump_on_hang(
+                seconds=15,
+                path='/tmp/my_test_hang.dump',
+            ) as dump_path:
+                trio.run(main)
+            # if it hangs, inspect dump_path afterward
+
+    '''
+    dump_path = Path(path)
+    f = dump_path.open('w')
+    try:
+        faulthandler.dump_traceback_later(
+            seconds,
+            repeat=False,
+            file=f,
+            exit=False,
+        )
+        try:
+            yield str(dump_path)
+        finally:
+            faulthandler.cancel_dump_traceback_later()
+    finally:
+        f.close()
+
+
+def _snapshot() -> tuple[int, int]:
+    '''
+    Return `(thread_count, subint_count)`.
+
+    Subint count reported as `0` on pythons lacking the
+    private `_interpreters` stdlib module (i.e. py<3.13).
+
+    '''
+    threads: int = threading.active_count()
+    subints: int = (
+        len(_interpreters.list_all())
+        if _interpreters is not None
+        else 0
+    )
+    return threads, subints
+
+
+@contextmanager
+def track_resource_deltas(
+    label: str = '',
+    *,
+    writer: Callable[[str], None] | None = None,
+
+) -> Iterator[tuple[int, int]]:
+    '''
+    Log `(threads, subints)` deltas across the with-block.
+
+    `writer` defaults to `sys.stderr.write` (+ trailing
+    newline); pass a custom callable to route elsewhere
+    (e.g., a log handler or an append-to-file).
+
+    Yields the pre-entry snapshot so callers can assert
+    against the expected counts if they want.
+
+    Example
+    -------
+    ::
+
+        from tractor.devx import track_resource_deltas
+
+        async def test_foo():
+            with track_resource_deltas(label='test_foo'):
+                async with tractor.open_nursery() as an:
+                    ...
+
+        # Output:
+        #   test_foo: threads 2->2, subints 1->1
+
+    '''
+    before = _snapshot()
+    try:
+        yield before
+    finally:
+        after = _snapshot()
+        msg: str = (
+            f'{label}: '
+            f'threads {before[0]}->{after[0]}, '
+            f'subints {before[1]}->{after[1]}'
+        )
+        if writer is None:
+            sys.stderr.write(msg + '\n')
+            sys.stderr.flush()
+        else:
+            writer(msg)
+
+
+def resource_delta_fixture(
+    *,
+    autouse: bool = True,
+    writer: Callable[[str], None] | None = None,
+
+) -> Callable:
+    '''
+    Factory returning a `pytest` fixture that wraps each test
+    in `track_resource_deltas(label=<node.name>)`.
+
+    Usage in a `conftest.py`::
+
+        # tests/conftest.py
+        from tractor.devx import resource_delta_fixture
+
+        track_resources = resource_delta_fixture()
+
+    or opt-in per-test::
+
+        track_resources = resource_delta_fixture(autouse=False)
+
+        def test_foo(track_resources):
+            ...
+
+    Kept as a factory (not a bare fixture) so callers control
+    `autouse` / `writer` without having to subclass or patch.
+
+    '''
+    import pytest  # deferred: only needed when caller opts in
+
+    @pytest.fixture(autouse=autouse)
+    def _track_resources(request):
+        with track_resource_deltas(
+            label=request.node.name,
+            writer=writer,
+        ):
+            yield
+
+    return _track_resources

tractor/devx/_proctitle.py

@@ -0,0 +1,83 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Per-actor proc-title via `py-setproctitle`.
+
+Sets a stable, OS-level identifier for each `tractor` actor
+process so diag tools (`ps`, `top`, `htop`, `psutil`) and our
+own `acli.pytree`/`acli.hung_dump` can show "which actor is
+which" at a glance without needing to read full
+`/proc/<pid>/cmdline`.
+
+Format:
+  ``<_def_prefix>[<aid.reprol()>]``  e.g. ``_subactor[doggy@1027301b]``
+(prefix from the `_def_prefix` const, flipped `tractor` ->
+`_subactor` so sub-actor procs are visually distinct from the
+root in `ps`/`htop` and the reap-recognition markers.)
+
+Uses the canonical `Aid.reprol()` form
+(``<name>@<uuid_short>``) so the proc-title matches the
+identifier shape used in tractor's logs, the `TRACTOR_AID`
+env-var, and orphan-reaper scans — one identity across
+all surfaces.
+
+Optional dep: silently no-op when `setproctitle` is missing.
+
+'''
+from __future__ import annotations
+from typing import TYPE_CHECKING
+
+
+if TYPE_CHECKING:
+    from tractor.runtime._runtime import Actor
+
+
+# `setproctitle` is an optional dep — tractor's runtime path
+# treats this as best-effort diag, so missing import is a
+# no-op rather than a hard error.
+try:
+    import setproctitle as _stp
+except ImportError:
+    _stp = None
+
+
+_def_prefix: str = '_subactor'
+
+
+def set_actor_proctitle(
+    actor: 'Actor',
+    prefix: str = _def_prefix,
+) -> str | None:
+    '''
+    Set the calling process's proc-title to identify it as a
+    tractor sub-actor.
+
+    Returns the title string set, or `None` if `setproctitle`
+    isn't available.
+
+    Should be called early in the actor's process lifetime
+    (after `Actor` construction, before `_trio_main`) so the
+    new title is visible to OS-level tooling for the entire
+    runtime.
+
+    '''
+    if _stp is None:
+        return None
+
+    title: str = f'{prefix}[{actor.aid.reprol()}]'
+    _stp.setproctitle(title)
+    return title

tractor/devx/_stackscope.py

@@ -24,7 +24,7 @@ disjoint, parallel executing tasks in separate actors.
 
 '''
 from __future__ import annotations
-# from functools import partial
+from functools import partial
 from threading import (
     current_thread,
     Thread,
@@ -47,7 +47,9 @@ from typing import (
 import trio
 from tractor.runtime import _state
 from tractor import log as logmod
-from tractor.devx import debug
+from tractor.devx import (
+    debug,
+)
 
 log = logmod.get_logger()
 
@@ -61,12 +63,28 @@ if TYPE_CHECKING:
 
 
 @trio.lowlevel.disable_ki_protection
-def dump_task_tree() -> None:
+def dump_task_tree(
+    write_file: bool = False,
+    write_tty: bool = False,
+) -> None:
     '''
     Do a classic `stackscope.extract()` task-tree dump to console at
     `.devx()` level.
 
+    Also unconditionally tee the rendered tree to two
+    capture-bypassing sinks so SIGUSR1 dumps remain visible
+    when the parent process has captured stdio (e.g. pytest's
+    default `--capture=fd`):
+
+    - `/tmp/tractor-stackscope-<pid>.log` (append-mode, always
+      written) — guaranteed-readable artifact even under CI
+      / `nohup` / no-tty conditions. `tail -f` to follow.
+    - `/dev/tty` if a controlling terminal is attached —
+      best-effort, ignored if the device is missing or write
+      fails. pytest never captures the tty.
+
     '''
+    import os
     import stackscope
     tree_str: str = str(
         stackscope.extract(
@@ -96,46 +114,158 @@ def dump_task_tree() -> None:
     # |_{Supervisor/Scope
     # |_[Storage/Memory/IPC-Stream/Data-Struct
 
-    log.devx(
+    fpath: str = f'/tmp/tractor-stackscope-{os.getpid()}.log'
+    from . import pformat
+    actor_repr: str = pformat.nest_from_op(
+        input_op='|_',
+        text=f'{actor}',
+        nest_prefix='|_',
+        nest_indent=3,
+    )
+    full_dump: str = (
         f'Dumping `stackscope` tree for actor\n'
-        f'(>: {actor.uid!r}\n'
+        f'(>: {actor.aid.uid!r}\n'
         f' |_{mp.current_process()}\n'
         f'   |_{thr}\n'
-        f'     |_{actor}\n'
+        # TODO, use the nest_from_op
+        f'{actor_repr}'
+        # f'     |_{actor}'
         f'\n'
         f'{sigint_handler_report}\n'
         f'signal.getsignal(SIGINT) -> {current_sigint_handler!r}\n'
-        # f'\n'
-        # start-of-trace-tree delimiter (mostly for testing)
-        # f'------ {actor.uid!r} ------\n'
         f'\n'
-        f'------ start-of-{actor.uid!r} ------\n'
+        f'capture-bypass tee: {fpath}\n'
+        f'(`tail -f {fpath}` to follow across signals)\n'
+        f'\n'
+        f'------ start-of-{actor.aid.uid!r} ------\n'
         f'|\n'
         f'{tree_str}'
-        # end-of-trace-tree delimiter (mostly for testing)
         f'|\n'
-        f'|_____ end-of-{actor.uid!r} ______\n'
+        f'|_____ end-of-{actor.aid.uid!r} ______\n'
     )
-    # TODO: can remove this right?
-    # -[ ] was original code from author
-    #
-    # print(
-    #     'DUMPING FROM PRINT\n'
-    #     +
-    #     content
-    # )
-    # import logging
-    # try:
-    #     with open("/dev/tty", "w") as tty:
-    #         tty.write(tree_str)
-    # except BaseException:
-    #     logging.getLogger(
-    #         "task_tree"
-    #     ).exception("Error printing task tree")
+    log.devx(full_dump)
+
+    # NOTE, capture-bypass sinks. Pytest's default
+    # `--capture=fd` swallows `log.devx()` above; the
+    # following two writes guarantee the dump reaches the
+    # human even when stdio is captured.
+    if write_file:
+        try:
+            with open(fpath, 'a') as f:
+                f.write(full_dump + '\n')
+        except OSError:
+            log.exception(
+                f'Failed to tee stackscope dump to {fpath!r}'
+            )
+
+    if write_tty:
+        try:
+            with open('/dev/tty', 'w') as tty:
+                tty.write(full_dump + '\n')
+        except OSError:
+            # no controlling tty (CI / nohup / detached) —
+            # silently fall through; the file sink covers it.
+            pass
 
 _handler_lock = RLock()
 _tree_dumped: bool = False
 
+# Captured at `enable_stack_on_sig()` time when running
+# inside a trio task. `dump_tree_on_sig` uses this to
+# schedule `dump_task_tree()` ON the trio loop via
+# `token.run_sync_soon` so stackscope sees a real current
+# task and can recurse into nursery children. Without
+# it (signal handler running in a non-trio stack frame),
+# `stackscope.extract` only walks the `<init>` task and
+# misses everything inside `async_main`'s nurseries.
+_trio_token: trio.lowlevel.TrioToken|None = None
+
+
+def _relay_sig_to_subactors(sig: int) -> None:
+    '''
+    Forward `sig` to every live sub-actor's underlying
+    process so each runs its own `dump_tree_on_sig`
+    handler.
+
+    Factored out of `dump_tree_on_sig` so the
+    `run_sync_soon`-deferred path can call it AFTER
+    the parent's `dump_task_tree()` completes — see
+    `_dump_then_relay` below for why ordering matters.
+
+    '''
+    an: ActorNursery
+    for an in _state.current_actor()._actoruid2nursery.values():
+        subproc: ProcessType
+        subactor: Actor
+        for (
+            subactor,
+            subproc,
+            _,
+        ) in an._children.values():
+            log.warning(
+                f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
+                f'{subactor}\n'
+                f' |_{subproc}\n'
+            )
+            # bc of course stdlib can't have a std API.. XD
+            match subproc:
+                case trio.Process():
+                    subproc.send_signal(sig)
+
+                case mp.Process():
+                    subproc._send_signal(sig)
+
+
+def _dump_then_relay(
+    sig: int|None,
+) -> None:
+    '''
+    `run_sync_soon`-friendly callback: dump THIS actor's
+    task tree first, THEN relay `sig` to subactors so
+    their dumps can't race ahead of ours.
+
+    Hierarchical-ordering preservation: the legacy
+    direct-call path (pre-`run_sync_soon`) ran the dump
+    synchronously inside the signal handler, then
+    relayed — guaranteeing parent-output-before-child
+    in the multiplexed pty stream. The pure-deferred
+    path (schedule dump only, relay sync from handler)
+    inverts that: relay fires while the parent's
+    dump is still queued, subs receive SIGUSR1 and
+    schedule their own dumps, all dumps then race in
+    arbitrary order through stdio.
+
+    Co-scheduling fixes that: by chaining relay AFTER
+    `dump_task_tree()` inside the same trio-loop
+    callback, parent output flushes before any sub
+    receives the signal, restoring the
+    parent → relay-log → sub-dump ordering humans
+    expect when reading hang-investigation traces.
+
+    Trio prints + crashes on uncaught exceptions in
+    scheduled callbacks; we swallow + log so the test
+    keeps running and the user can re-trigger.
+
+    '''
+    try:
+        dump_task_tree()
+    except BaseException:
+        log.exception(
+            '`dump_task_tree()` raised (scheduled via '
+            '`run_sync_soon`); continuing.\n'
+        )
+
+    if sig is None:
+        return
+
+    try:
+        _relay_sig_to_subactors(sig)
+    except BaseException:
+        log.exception(
+            f'`_relay_sig_to_subactors({sig})` raised '
+            f'(scheduled via `run_sync_soon`); continuing.\n'
+        )
+
 
 def dump_tree_on_sig(
     sig: int,
@@ -159,16 +289,32 @@ def dump_tree_on_sig(
             'Trying to dump `stackscope` tree..\n'
         )
         try:
-            dump_task_tree()
-            # await actor._service_n.start_soon(
-            #     partial(
-            #         trio.to_thread.run_sync,
-            #         dump_task_tree,
-            #     )
-            # )
-            # trio.lowlevel.current_trio_token().run_sync_soon(
-            #     dump_task_tree
-            # )
+            # Prefer scheduling on the trio loop — runs the
+            # dump from a real trio-task context so
+            # `stackscope.extract(recurse_child_tasks=True)`
+            # walks every nursery child instead of seeing
+            # only the `<init>` task. Falls back to a direct
+            # call when no token was captured (e.g. signal
+            # delivered outside a trio.run).
+            #
+            # Co-schedule the relay-to-subs in the SAME
+            # callback so parent's dump prints BEFORE any
+            # sub receives SIGUSR1 — see `_dump_then_relay`
+            # for the full hierarchical-ordering rationale.
+            if _trio_token is not None:
+                _trio_token.run_sync_soon(
+                    partial(
+                        _dump_then_relay,
+                        sig=sig if relay_to_subs else None,
+                    )
+                )
+                # NOTE, `_dump_then_relay` handles the relay
+                # internally; bail out before the
+                # direct-path relay below.
+                return
+
+            else:
+                dump_task_tree()
 
         except RuntimeError:
             log.exception(
@@ -188,27 +334,15 @@ def dump_tree_on_sig(
         #     'Supposedly we dumped just fine..?'
         # )
 
+    # Direct-path relay (only reached when `_trio_token`
+    # was None — the run_sync_soon path returned above
+    # to let `_dump_then_relay` handle the relay
+    # in-callback).
     if not relay_to_subs:
+        log.devx(f'Skipping {sig!r} relay to subactors..')
         return
 
-    an: ActorNursery
-    for an in _state.current_actor()._actoruid2nursery.values():
-        subproc: ProcessType
-        subactor: Actor
-        for subactor, subproc, _ in an._children.values():
-            log.warning(
-                f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
-                f'{subactor}\n'
-                f' |_{subproc}\n'
-            )
-
-            # bc of course stdlib can't have a std API.. XD
-            match subproc:
-                case trio.Process():
-                    subproc.send_signal(sig)
-
-                case mp.Process():
-                    subproc._send_signal(sig)
+    _relay_sig_to_subactors(sig)
 
 
 def enable_stack_on_sig(
@@ -233,19 +367,50 @@ def enable_stack_on_sig(
 
     '''
     try:
-        import stackscope
+        # NOTE, `stackscope._glue` does intentional async-gen type
+        # introspection at import-time which trips
+        # `RuntimeWarning: coroutine method 'asend'/'athrow' was
+        # never awaited`. Benign — they only want the wrapper
+        # type — but visible to users. Squelch the import-only
+        # warning so SIGUSR1 setup stays quiet.
+        import warnings
+        with warnings.catch_warnings():
+            warnings.filterwarnings(
+                'ignore',
+                category=RuntimeWarning,
+                message=r"coroutine method '(asend|athrow)' .* was never awaited",
+            )
+            import stackscope
+            _state._runtime_vars['use_stackscope'] = True
     except ImportError:
         log.warning(
             'The `stackscope` lib is not installed!\n'
             '`Ignoring enable_stack_on_sig() call!\n'
         )
+        assert not _state._runtime_vars['use_stackscope']
         return None
 
+    # Capture the trio token if we're inside `trio.run`
+    # so SIGUSR1 dispatches the dump *onto* the trio loop
+    # (full task-tree visibility). When called outside trio
+    # (e.g. from `pytest_configure`), token capture fails
+    # silently and `dump_tree_on_sig` falls back to the
+    # direct-call path.
+    global _trio_token
+    try:
+        _trio_token = trio.lowlevel.current_trio_token()
+    except RuntimeError:
+        # not in a `trio.run` — leave None; runtime can
+        # re-call `enable_stack_on_sig()` later from
+        # inside `async_main` to capture it.
+        _trio_token = None
+
     handler: Callable|int = getsignal(sig)
     if handler is dump_tree_on_sig:
         log.devx(
             'A `SIGUSR1` handler already exists?\n'
             f'|_ {handler!r}\n'
+            f'(trio_token captured: {_trio_token is not None})\n'
         )
         return
 
@@ -259,5 +424,6 @@ def enable_stack_on_sig(
         f'{stackscope!r}\n\n'
         f'With `SIGUSR1` handler\n'
         f'|_{dump_tree_on_sig}\n'
+        f'(trio_token captured: {_trio_token is not None})\n'
     )
     return stackscope

tractor/devx/debug/_tty_lock.py

@@ -181,7 +181,7 @@ class Lock:
         return (
             f'<{cls.__name__}(\n'
             f'{body}'
-            ')>\n\n'
+            ')>\n'
         )
 
     @classmethod
@@ -282,7 +282,7 @@ class Lock:
             ):
                 message += (
                     '-> No new task holds the TTY lock!\n\n'
-                    f'{Lock.repr()}\n'
+                    f'{Lock.repr()}'
                 )
 
             elif (

tractor/ipc/_linux.py

@@ -17,10 +17,20 @@
 Linux specifics, for now we are only exposing EventFD
 
 '''
-import os
 import errno
+import os
+import sys
+
+try:
+    import cffi
+except ImportError as ie:
+    if sys.version_info < (3, 14):
+        ie.add_note(
+            f'The `cffi` pkg has no 3.14 support yet.\n'
+        )
+
+    raise ie
 
-import cffi
 import trio
 
 ffi = cffi.FFI()

tractor/ipc/_mp_bs.py

@@ -17,7 +17,7 @@
 Utils to tame mp non-SC madeness
 
 '''
-import platform
+from functools import partial
 
 
 def disable_mantracker():
@@ -27,49 +27,37 @@ def disable_mantracker():
 
     '''
     from multiprocessing.shared_memory import SharedMemory
+    from multiprocessing import (
+        resource_tracker as mantracker,
+    )
 
+    # XXX ALWAYS disable the stdlib's "resource tracker"; it prevents
+    # fork backends and never was useful to us since we're SC
+    # lifetime managing all allocations.
+    class ManTracker(mantracker.ResourceTracker):
+        def register(self, name, rtype):
+            pass
+
+        def unregister(self, name, rtype):
+            pass
+
+        def ensure_running(self):
+            pass
+
+    # "know your land and know your prey"
+    # https://www.dailymotion.com/video/x6ozzco
+    mantracker._resource_tracker = ManTracker()
+    mantracker.register = mantracker._resource_tracker.register
+    mantracker.ensure_running = mantracker._resource_tracker.ensure_running
+    mantracker.unregister = mantracker._resource_tracker.unregister
+    mantracker.getfd = mantracker._resource_tracker.getfd
 
     # 3.13+ only.. can pass `track=False` to disable
     # all the resource tracker bs.
     # https://docs.python.org/3/library/multiprocessing.shared_memory.html
-    if (_py_313 := (
-            platform.python_version_tuple()[:-1]
-            >=
-            ('3', '13')
-        )
-    ):
-        from functools import partial
-        return partial(
-            SharedMemory,
-            track=False,
-        )
-
-    # !TODO, once we drop 3.12- we can obvi remove all this!
-    else:
-        from multiprocessing import (
-            resource_tracker as mantracker,
-        )
-
-        # Tell the "resource tracker" thing to fuck off.
-        class ManTracker(mantracker.ResourceTracker):
-            def register(self, name, rtype):
-                pass
-
-            def unregister(self, name, rtype):
-                pass
-
-            def ensure_running(self):
-                pass
-
-        # "know your land and know your prey"
-        # https://www.dailymotion.com/video/x6ozzco
-        mantracker._resource_tracker = ManTracker()
-        mantracker.register = mantracker._resource_tracker.register
-        mantracker.ensure_running = mantracker._resource_tracker.ensure_running
-        mantracker.unregister = mantracker._resource_tracker.unregister
-        mantracker.getfd = mantracker._resource_tracker.getfd
-
-        # use std type verbatim
-        shmT = SharedMemory
+    shmT = partial(
+        SharedMemory,
+        track=False,
+    )
 
     return shmT

tractor/ipc/_server.py

@@ -398,7 +398,7 @@ async def handle_stream_from_peer(
         uid,
         None,
     )
-    if event:
+    if event is not None:
         con_status_steps += (
             ' -> Waking subactor spawn waiters: '
             f'{event.statistics().tasks_waiting}\n'
@@ -1122,20 +1122,32 @@ async def _serve_ipc_eps(
             )
 
     finally:
+        # close every endpoint INDEPENDENTLY: a close raising
+        # mid-iter (e.g. UDS `os.unlink` racing concurrent reap) must
+        # not strand the rest of the eps + must not skip the
+        # `_shutdown.set()` below.
         if eps:
             addr: Address
             ep: Endpoint
-            for addr, ep in server.epsdict().items():
-                ep.close_listener()
-                server._endpoints.remove(ep)
+            for addr, ep in list(server.epsdict().items()):
+                try:
+                    ep.close_listener()
+                except Exception as ep_close_err:
+                    log.exception(
+                        f'Endpoint close raised, continuing teardown\n'
+                        f'  |_{ep!r}\n'
+                        f'  |_{ep_close_err!r}\n'
+                    )
+                finally:
+                    try:
+                        server._endpoints.remove(ep)
+                    except ValueError:
+                        pass
 
-        # actor = _state.current_actor()
-        # if actor.is_arbiter:
-        #     import pdbp; pdbp.set_trace()
-
-        # signal the server is "shutdown"/"terminated"
-        # since no more active endpoints are active.
-        if not server._endpoints:
+        # always signal "shutdown" so `actor.cancel()` →
+        # `ipc_server.wait_for_shutdown()` doesn't deadlock when an
+        # endpoint close raised above.
+        if server._shutdown is not None:
             server._shutdown.set()
 
 @acm

tractor/ipc/_shm.py

@@ -929,15 +929,26 @@ def open_shm_list(
     # "close" attached shm on actor teardown
     try:
         actor = tractor.current_actor()
-
         actor.lifetime_stack.callback(shml.shm.close)
 
-        # XXX on 3.13+ we don't need to call this?
-        # -> bc we pass `track=False` for `SharedMemeory` orr?
-        if (
-            platform.python_version_tuple()[:-1] < ('3', '13')
-        ):
-            actor.lifetime_stack.callback(shml.shm.unlink)
+        # >XXX NOTE< on 3.13+ we need to call this AS WELL AS pass
+        # `track=False` for `mp.SharedMemeory` otherwise fork based
+        # backends will error out due to long lived stdlib
+        # limitations,
+        # - https://bugs.python.org/issue38119
+        # - https://bugs.python.org/issue45209
+        #
+        def try_unlink():
+            try:
+                shml.shm.unlink()
+            except FileNotFoundError as fne:
+                log.debug(
+                    f'ShmList already deallocated pre-actor-shutdown.\n'
+                    f'{fne!r}\n'
+                )
+
+        actor.lifetime_stack.callback(try_unlink)
+
     except RuntimeError:
         log.warning('tractor runtime not active, skipping teardown steps')
 

tractor/ipc/_uds.py

@@ -344,7 +344,18 @@ def close_listener(
 
     '''
     lstnr.socket.close()
-    os.unlink(addr.sockpath)
+    # tolerate the sock-file being already gone — under concurrent
+    # pytest sessions sharing the bindspace dir, another session's
+    # reap path can unlink it first; raising here aborts the
+    # `_serve_ipc_eps` finally before `_shutdown.set()`, deadlocking
+    # `wait_for_shutdown()` on `actor.cancel()`.
+    try:
+        os.unlink(addr.sockpath)
+    except FileNotFoundError:
+        log.warning(
+            f'UDS sock-file already unlinked, skipping\n'
+            f'  |_{addr.sockpath}\n'
+        )
 
 
 async def open_unix_socket_w_passcred(

tractor/log.py

@@ -262,6 +262,63 @@ class StackLevelAdapter(LoggerAdapter):
         )
 
 
+def add_log_level(
+    name: str,
+    value: int,
+    color: str = 'white',
+) -> None:
+    '''
+    Register a new custom log level with `tractor`'s logging
+    machinery in ONE call — the single pluggable entry-point that
+    keeps the (otherwise hand-synced) pieces consistent:
+
+    - `CUSTOM_LEVELS[name]` (drives the `stacklevel` bump in
+      `StackLevelAdapter.log()` + the `get_logger()` audit).
+    - `logging.addLevelName()` registration.
+    - `STD_PALETTE`/`BOLD_PALETTE` color entries (consumed when
+      `get_console_log()` builds its `ColoredFormatter`).
+    - a same-named (lowercase) emit method bound on
+      `StackLevelAdapter` so `log.<name>('msg')` works (and so
+      `get_logger()`'s per-level method audit passes).
+
+    Idempotent: re-registering an existing name is a no-op-ish
+    refresh (won't clobber an already-bound method).
+
+    '''
+    name_up: str = name.upper()
+    name_lo: str = name.lower()
+
+    CUSTOM_LEVELS[name_up] = value
+    logging.addLevelName(value, name_up)
+    STD_PALETTE[name_up] = color
+    BOLD_PALETTE['bold'][name_up] = f'bold_{color}'
+
+    if not hasattr(StackLevelAdapter, name_lo):
+        # bind via default-arg so `value` is captured (not
+        # late-bound); delegates to `.log()` exactly like the
+        # hand-written level methods above.
+        def _emit(
+            self,
+            msg: str,
+            *,
+            _level: int = value,
+        ) -> None:
+            return self.log(_level, msg)
+
+        _emit.__name__ = name_lo
+        _emit.__qualname__ = f'StackLevelAdapter.{name_lo}'
+        setattr(StackLevelAdapter, name_lo, _emit)
+
+
+# `IO`: child-subproc std-stream relay (see
+# `tractor.trionics._subproc`). Value 21 sits just ABOVE
+# `INFO`(20) so it's SHOWN BY DEFAULT at the usual `info`/`devx`
+# console levels (a `runtime`(15) relay would be silently
+# filtered) yet still distinctly labelled/colored + separately
+# filterable.
+add_log_level('IO', 21, 'purple')
+
+
 # TODO IDEAs:
 # -[ ] move to `.devx.pformat`?
 # -[ ] do per task-name and actor-name color coding
@@ -543,21 +600,45 @@ def get_logger(
         #   only includes the first 2 sub-pkg name-tokens in the
         #   child-logger's name; the colored "pkg-namespace" header
         #   will then correctly show the same value as `name`.
+        #
+        # XXX, strip the trailing `pkg_path` token ONLY when it
+        # duplicates the caller's leaf-*module* name — which the
+        # console header already renders via its `{filename}` field.
+        # We compare against the caller module's `__name__`/
+        # `__package__` (rather than blindly dropping the last token)
+        # so genuine, possibly-*nested* sub-PACKAGE components stay
+        # addressable as their own sub-loggers:
+        #
+        # - `name='trionics._broadcast'` (a leaf-module, from a
+        #   `get_logger(__name__)`-style call) -> `tractor.trionics`
+        #   (leaf dropped; `_broadcast.py` is in the header).
+        # - `name='devx.debug'` (a real sub-PACKAGE, whether
+        #   auto-derived from a module's `__package__` or passed
+        #   explicitly by a logging-spec) -> `tractor.devx.debug`,
+        #   DISTINCT from a bare `devx` -> `tractor.devx`.
+        #
+        # The previous unconditional `pkg_path = subpkg_path` also ate
+        # the deepest sub-pkg, collapsing `devx.debug` -> `tractor.devx`
+        # and silently breaking per-sub-pkg level control via the
+        # logging-spec; see `tractor.log.LogSpec`/`apply_logspec()`.
+        caller_leaf_mod: str|None = None
+        if (caller_mod := get_caller_mod()):
+            cmod_name: str = getattr(caller_mod, '__name__', '') or ''
+            cmod_pkg: str = getattr(caller_mod, '__package__', '') or ''
+            # a leaf-*module* has `__name__ != __package__`; a package
+            # `__init__` has them equal (so its trailing token is a
+            # real sub-pkg, NOT a leaf-module-filename to strip).
+            if cmod_name and cmod_name != cmod_pkg:
+                caller_leaf_mod = cmod_name.rpartition('.')[2]
+
         if (
-            # XXX, TRY to remove duplication cases
-            # which get warn-logged on below!
-            (
-                # when, subpkg_path == pkg_path
-                subpkg_path
-                and
-                rname == pkg_name
-            )
-            # ) or (
-            #     # when, pkg_path == leaf_mod
-            #     pkg_path
-            #     and
-            #     leaf_mod == pkg_path
-            # )
+            subpkg_path
+            and
+            rname == pkg_name
+            and
+            # only collapse when the trailing token IS the caller's
+            # leaf-module (i.e. the `{filename}` already shows it).
+            leaf_mod == caller_leaf_mod
         ):
             pkg_path = subpkg_path
 
@@ -711,6 +792,167 @@ def get_console_log(
     return log
 
 
+# A `tractor` "logging-spec": a compact, code-free way for a
+# consuming project's test-iface (or runtime) to dial-in console
+# loglevels across the lib's logger hierarchy. Mirrors the grammar
+# consumed by `modden.runtime.daemon.setup_tractor_logging()`.
+#
+# Accepted forms (`str|bool`),
+# - `True`              -> enable the `pkg_name` root-logger at
+#                          `default_level` (or 'cancel').
+# - `False`             -> disable (no-op, configure nothing).
+# - 'info'              -> a bare level for the root-logger.
+# - 'sub:info,x:cancel' -> per-sub-logger levels; each `<name>` is
+#                          RELATIVE to `pkg_name` (must NOT include
+#                          the `pkg_name` token itself), eg.
+#                          'devx.debug:runtime,trionics:cancel'.
+#
+# !GRANULARITY! sub-logger names match at the `pkg_name.<name>`
+# *logger* level — which (per `get_logger()`'s name-derivation) is
+# *sub-PACKAGE* granularity, addressable at ANY nesting depth:
+# - 'devx.debug' -> the `tractor.devx.debug` logger, DISTINCT from a
+#   bare 'devx' -> `tractor.devx` (its parent). Setting `devx` also
+#   gates `devx.debug` via normal stdlib level-inheritance unless the
+#   child sets its own level.
+# - leaf *modules* are intentionally NOT individually addressable:
+#   `get_logger()` drops the leaf module-name from the logger key
+#   since the console header already renders it via `{filename}`, so
+#   every module in a (sub-)pkg shares that pkg's logger. Per-leaf
+#   level control would need a record-filter (see follow-up notes:
+#   `ai/tooling-todos/logspec_leaf_module_granularity_route_b.md`).
+# - top-level lib modules (eg. `tractor.to_asyncio`) emit under the
+#   *root* `pkg_name` logger (their `__package__` IS `pkg_name`), so
+#   a 'to_asyncio:<level>' entry targets a phantom child that nothing
+#   emits to -> no-op. Use the bare-level/root form for those.
+LogSpec = str|bool
+
+
+def parse_logspec(
+    logspec: LogSpec,
+    default_level: str|None = None,
+    pkg_name: str = _proj_name,
+
+) -> dict[str|None, str]:
+    '''
+    Parse a `tractor` "logging-spec" (see `LogSpec`) into a
+    `{sublog_name|None: level}` mapping where a `None` key denotes
+    the `pkg_name` root-logger itself.
+
+    '''
+    match logspec:
+
+        # explicit disable -> configure nothing.
+        case False:
+            return {}
+
+        # enable the root-logger at the fallback level.
+        case True:
+            return {None: (default_level or 'cancel')}
+
+        case str(spec):
+            filters: list[str] = [
+                part.strip()
+                for part in spec.split(',')
+                if part.strip()
+            ]
+            # i. a bare level (no sub-logger filtering),
+            #    eg. 'info' | 'cancel'
+            if (
+                len(filters) == 1
+                and
+                ':' not in filters[0]
+            ):
+                return {None: filters[0]}
+
+            # ii. a per-sub-logger filter-spec of the form,
+            #     '<sublog_0>:<level>,<.. N-other-parts>'
+            #     eg. 'to_asyncio:cancel,devx._debug:runtime'
+            out: dict[str|None, str] = {}
+            for log_filter in filters:
+                name, sep, level = log_filter.partition(':')
+                if not sep:
+                    raise ValueError(
+                        f'Invalid `tractor` logging-spec part!\n'
+                        f'{log_filter!r}\n'
+                        f'\n'
+                        f'Mixed bare-level + sub-logger filters are '
+                        f'not supported; every comma-part must be '
+                        f'`<sublog>:<level>`.\n'
+                    )
+                # the sub-logger name is RELATIVE to `pkg_name`;
+                # duplicating the pkg-token is a user error since
+                # the root-logger already IS `pkg_name`.
+                if pkg_name in name.split('.'):
+                    raise ValueError(
+                        f'logging-spec sub-name should NOT include '
+                        f'the `pkg_name={pkg_name!r}` token!\n'
+                        f'got name={name!r}\n'
+                    )
+                out[name] = level
+            return out
+
+        case _:
+            raise ValueError(
+                f'Invalid `tractor` logging-spec!\n'
+                f'{logspec!r}\n'
+            )
+
+
+def apply_logspec(
+    logspec: LogSpec,
+    default_level: str|None = None,
+    pkg_name: str = _proj_name,
+
+) -> tuple[
+    str|None,
+    dict[str, StackLevelAdapter],
+]:
+    '''
+    Parse + apply a `tractor` "logging-spec" (see `parse_logspec()`):
+    enable a `colorlog` stderr console handler for each
+    (sub-)logger named in the spec at its requested level.
+
+    Returns a 2-tuple,
+    - the resolved "primary" runtime-level: the root-logger level if
+      the spec set one, else `default_level`; suitable for passing
+      to `open_root_actor(loglevel=<.>)`,
+    - a `{logger_name: StackLevelAdapter}` map of every logger the
+      spec touched.
+
+    '''
+    specs: dict[str|None, str] = parse_logspec(
+        logspec,
+        default_level=default_level,
+        pkg_name=pkg_name,
+    )
+    logs: dict[str, StackLevelAdapter] = {}
+    for sub_name, level in specs.items():
+        # NOTE, pass the RELATIVE sub-name (no `pkg_name.` prefix)
+        # to avoid `get_logger()`'s duplicate-pkg-token warning;
+        # it re-adds the pkg-name via `.getChild()` internally.
+        log: StackLevelAdapter = get_console_log(
+            level=level,
+            pkg_name=pkg_name,
+            name=(sub_name or pkg_name),
+        )
+        # XXX, a sub-logger filter is "authoritative" for its
+        # subtree: it gets its OWN stderr handler (added by
+        # `get_console_log()` above), so DON'T also let its records
+        # propagate up to a root `pkg_name`-logger handler — that
+        # would double-emit every line when a root-level console
+        # (eg. via `--ll`) is also active. The root-level form
+        # (`sub_name is None`) keeps default propagation.
+        if sub_name is not None:
+            log.logger.propagate = False
+        logs[log.name] = log
+
+    primary_level: str|None = specs.get(None, default_level)
+    return (
+        primary_level,
+        logs,
+    )
+
+
 def get_loglevel() -> str:
     return _default_loglevel
 

tractor/runtime/_portal.py

@@ -55,6 +55,7 @@ from ..msg import (
     Return,
 )
 from .._exceptions import (
+    ActorTooSlowError,
     NoResult,
     TransportClosed,
 )
@@ -268,6 +269,7 @@ class Portal:
     async def cancel_actor(
         self,
         timeout: float | None = None,
+        raise_on_timeout: bool = False,
 
     ) -> bool:
         '''
@@ -281,6 +283,17 @@ class Portal:
         `._context.Context.cancel()` which CAN be used for this
         purpose.
 
+        `raise_on_timeout` (default `False`):
+
+        - `False` (legacy): on bounded-wait expiry, log at DEBUG
+          and return `False`. Used by callers that issue cancel
+          fire-and-forget and have their own escalation
+          (e.g. `_spawn.soft_kill()` checks `proc.poll()` after).
+        - `True`: on bounded-wait expiry, raise `ActorTooSlowError`
+          so the caller MUST handle the failure explicitly.
+          `ActorNursery.cancel()` opts in so it can escalate via
+          `proc.terminate()` per SC-discipline.
+
         '''
         __runtimeframe__: int = 1  # noqa
 
@@ -301,15 +314,16 @@ class Portal:
 
         # XXX the one spot we set it?
         chan._cancel_called: bool = True
+        cancel_timeout: float = (
+            timeout
+            or
+            self.cancel_timeout
+        )
         try:
             # send cancel cmd - might not get response
             # XXX: sure would be nice to make this work with
             # a proper shield
-            with trio.move_on_after(
-                timeout
-                or
-                self.cancel_timeout
-            ) as cs:
+            with trio.move_on_after(cancel_timeout) as cs:
                 cs.shield: bool = True
                 await self.run_from_ns(
                     'self',
@@ -317,16 +331,24 @@ class Portal:
                 )
                 return True
 
-            if cs.cancelled_caught:
-                # may timeout and we never get an ack (obvi racy)
-                # but that doesn't mean it wasn't cancelled.
-                log.debug(
-                    f'May have failed to cancel peer?\n'
-                    f'\n'
-                    f'c)=?> {peer_id}\n'
+            # `move_on_after` fired — peer didn't ack within
+            # bounded window. Behaviour depends on
+            # `raise_on_timeout`:
+            assert cs.cancelled_caught
+            if raise_on_timeout:
+                raise ActorTooSlowError(
+                    f'Peer {peer_id} did not ack `Actor.cancel()`'
+                    f'-RPC within bounded wait of '
+                    f'{cancel_timeout!r}s'
                 )
 
-            # if we get here some weird cancellation case happened
+            # legacy fire-and-forget path: log + return False so
+            # the caller can decide whether to escalate.
+            log.debug(
+                f'May have failed to cancel peer?\n'
+                f'\n'
+                f'c)=?> {peer_id}\n'
+            )
             return False
 
         except TransportClosed as tpt_err:

tractor/runtime/_runtime.py

@@ -870,7 +870,17 @@ class Actor:
 
             accept_addrs: list[UnwrappedAddress]|None = None
 
-            if self._spawn_method == "trio":
+            if self._spawn_method in (
+                'trio',
+                'subint',
+                # `main_thread_forkserver` (and the future
+                # variant-2 `subint_forkserver`) parent-side
+                # sends a `SpawnSpec` over IPC just like the
+                # other two — fork child-side runtime is
+                # trio-native.
+                'main_thread_forkserver',
+                'subint_forkserver',
+            ):
 
                 # Receive post-spawn runtime state from our parent.
                 spawnspec: msgtypes.SpawnSpec = await chan.recv()
@@ -922,11 +932,26 @@ class Actor:
                 # => update process-wide globals
                 # TODO! -[ ] another `Struct` for rtvs..
                 rvs: dict[str, Any] = spawnspec._runtime_vars
-                if rvs['_debug_mode']:
-                    from ..devx import (
-                        enable_stack_on_sig,
-                        maybe_init_greenback,
-                    )
+
+                # `stackscope` SIGUSR1 handler: install when EITHER
+                # `use_stackscope` is set in rt-vars OR the
+                # `TRACTOR_ENABLE_STACKSCOPE` env var is set (lighter
+                # test-time hang-debug path; see
+                # `tractor._testing.pytest`'s `--enable-stackscope`
+                # CLI flag — env var propagates via fork-inherited
+                # environ).
+                #
+                # NOTE, intentionally NOT gated on `_debug_mode` so
+                # SIGUSR1 task-tree dumps work in plain (non-pdb)
+                # runs too — esp. in infected-`asyncio` sub-actors
+                # where the default SIGUSR1 action would otherwise
+                # terminate the proc.
+                if (
+                    rvs.get('use_stackscope')
+                    or
+                    os.environ.get('TRACTOR_ENABLE_STACKSCOPE')
+                ):
+                    from ..devx import enable_stack_on_sig
                     try:
                         # TODO: maybe return some status msgs upward
                         # to that we can emit them in `con_status`
@@ -938,10 +963,13 @@ class Actor:
 
                     except ImportError:
                         log.warning(
-                            '`stackscope` not installed for use in debug mode!'
+                            '`stackscope` not installed for use in '
+                            'debug mode / `--enable-stackscope`!'
                         )
 
+                if rvs['_debug_mode']:
                     if rvs.get('use_greenback', False):
+                        from ..devx import maybe_init_greenback
                         maybe_mod: ModuleType|None = await maybe_init_greenback()
                         if maybe_mod:
                             log.devx(
@@ -1209,6 +1237,23 @@ class Actor:
                 ipc_server.cancel()
                 await ipc_server.wait_for_shutdown()
 
+            # Break the shield on the parent-channel
+            # `process_messages` loop (started with `shield=True`
+            # in `async_main` above). Required to avoid a
+            # deadlock during teardown of fork-spawned subactors:
+            # without this cancel, the loop parks waiting for
+            # EOF on the parent channel, but the parent is
+            # blocked on `os.waitpid()` for THIS actor's exit
+            # — mutual wait. For exec-spawn backends the EOF
+            # arrives naturally when the parent closes its
+            # handler-task socket during its own teardown, but
+            # in fork backends the shared-process-image makes
+            # that delivery racy / not guaranteed. Explicit
+            # cancel here gives us deterministic unwinding
+            # regardless of backend.
+            if self._parent_chan_cs is not None:
+                self._parent_chan_cs.cancel()
+
             # cancel all rpc tasks permanently
             if self._service_tn:
                 self._service_tn.cancel_scope.cancel()
@@ -1729,7 +1774,16 @@ async def async_main(
                 # start processing parent requests until our channel
                 # server is 100% up and running.
                 if actor._parent_chan:
-                    await root_tn.start(
+                    # Capture the shielded `loop_cs` for the
+                    # parent-channel `process_messages` task so
+                    # `Actor.cancel()` has a handle to break the
+                    # shield during teardown — without this, the
+                    # shielded loop would park on the parent chan
+                    # indefinitely waiting for EOF that only arrives
+                    # after the PARENT tears down, which under
+                    # fork-based backends (e.g. `main_thread_forkserver`)
+                    # it waits on THIS actor's exit — deadlock.
+                    actor._parent_chan_cs = await root_tn.start(
                         partial(
                             _rpc.process_messages,
                             chan=actor._parent_chan,
@@ -1940,7 +1994,25 @@ async def async_main(
                 f'   {pformat(ipc_server._peers)}'
             )
             log.runtime(teardown_report)
-            await ipc_server.wait_for_no_more_peers()
+            # NOTE: bound the peer-clear wait — otherwise if any
+            # peer-channel handler is stuck (e.g. never got its
+            # cancel propagated due to a runtime bug), this wait
+            # blocks forever and deadlocks the whole actor-tree
+            # teardown cascade. 3s is enough for any graceful
+            # cancel-ack round-trip; beyond that we're in bug
+            # territory and need to proceed with local teardown
+            # so the parent's `_ForkedProc.wait()` can unblock.
+            # See `ai/conc-anal/
+            # subint_forkserver_test_cancellation_leak_issue.md`
+            # for the full diagnosis.
+            with trio.move_on_after(3.0) as _peers_cs:
+                await ipc_server.wait_for_no_more_peers()
+            if _peers_cs.cancelled_caught:
+                teardown_report += (
+                    f'-> TIMED OUT waiting for peers to clear '
+                    f'({len(ipc_server._peers)} still connected)\n'
+                )
+                log.warning(teardown_report)
 
         teardown_report += (
             '-]> all peer channels are complete.\n'

tractor/runtime/_state.py

@@ -93,6 +93,7 @@ class RuntimeVars(Struct):
     repl_fixture: bool|Callable = False  # |AbstractContextManager[bool]
     # for `tractor.pause_from_sync()` & `breakpoint()` support
     use_greenback: bool = False
+    use_stackscope: bool = False
 
     # infected-`asyncio`-mode: `trio` running as guest.
     _is_infected_aio: bool = False
@@ -117,7 +118,14 @@ class RuntimeVars(Struct):
             )
 
 
-_runtime_vars: dict[str, Any] = {
+# The "fresh process" defaults — what `_runtime_vars` looks
+# like in a just-booted Python process that hasn't yet entered
+# `open_root_actor()` nor received a parent `SpawnSpec`. Kept
+# as a module-level constant so `get_runtime_vars(clear_values=
+# True)` can reset the live dict back to this baseline (see
+# `tractor.spawn._main_thread_forkserver` for the one current
+# caller that needs it).
+_RUNTIME_VARS_DEFAULTS: dict[str, Any] = {
     # root of actor-process tree info
     '_is_root': False,  # bool
     '_root_mailbox': (None, None),  # tuple[str|None, str|None]
@@ -132,16 +140,19 @@ _runtime_vars: dict[str, Any] = {
     # `debug_mode: bool` settings
     '_debug_mode': False,  # bool
     'repl_fixture': False,  # |AbstractContextManager[bool]
-    # for `tractor.pause_from_sync()` & `breakpoint()` support
-    'use_greenback': False,
+
+    'use_greenback': False,  # `.pause_from_sync()`/`breakpoint()`
+    'use_stackscope': False,  # trio-task-stack dumps on SIGUSR1
 
     # infected-`asyncio`-mode: `trio` running as guest.
     '_is_infected_aio': False,
 }
+_runtime_vars: dict[str, Any] = dict(_RUNTIME_VARS_DEFAULTS)
 
 
 def get_runtime_vars(
     as_dict: bool = True,
+    clear_values: bool = False,
 ) -> dict:
     '''
     Deliver a **copy** of the current `Actor`'s "runtime variables".
@@ -150,11 +161,62 @@ def get_runtime_vars(
     form, but the `RuntimeVars` struct should be utilized as possible
     for future calls.
 
-    '''
-    if as_dict:
-        return dict(_runtime_vars)
+    Pure read — **never mutates** the module-level `_runtime_vars`.
 
-    return RuntimeVars(**_runtime_vars)
+    If `clear_values=True`, return a copy of the fresh-process
+    defaults (`_RUNTIME_VARS_DEFAULTS`) instead of the live
+    dict. Useful in combination with `set_runtime_vars()` to
+    reset process-global state back to "cold" — the main caller
+    today is the `main_thread_forkserver` spawn backend's post-fork
+    child prelude:
+
+        set_runtime_vars(get_runtime_vars(clear_values=True))
+
+    `os.fork()` inherits the parent's full memory image, so the
+    child sees the parent's populated `_runtime_vars` (e.g.
+    `_is_root=True`) which would trip the `assert not
+    self.enable_modules` gate in `Actor._from_parent()` on the
+    subsequent parent→child `SpawnSpec` handshake if left alone.
+
+    '''
+    src: dict = (
+        _RUNTIME_VARS_DEFAULTS
+        if clear_values
+        else _runtime_vars
+    )
+    snapshot: dict = dict(src)
+    if as_dict:
+        return snapshot
+    return RuntimeVars(**snapshot)
+
+
+def set_runtime_vars(
+    rtvars: dict | RuntimeVars,
+) -> None:
+    '''
+    Atomically replace the module-level `_runtime_vars` contents
+    with those of `rtvars` (via `.clear()` + `.update()` so
+    live references to the same dict object remain valid).
+
+    Accepts either the historical `dict` form or the `RuntimeVars`
+    `msgspec.Struct` form (the latter still mostly unused but
+    the blessed forward shape — see the struct's definition).
+
+    Paired with `get_runtime_vars()` as the explicit
+    write-half of the runtime-vars API — prefer this over
+    direct mutation of `_runtime_vars[...]` from new call sites.
+
+    '''
+    if isinstance(rtvars, RuntimeVars):
+        # `msgspec.Struct` → dict via its declared field set;
+        # avoids pulling in `msgspec.structs.asdict` just for
+        # this one call path.
+        rtvars = {
+            field_name: getattr(rtvars, field_name)
+            for field_name in rtvars.__struct_fields__
+        }
+    _runtime_vars.clear()
+    _runtime_vars.update(rtvars)
 
 
 def last_actor() -> Actor|None:

tractor/runtime/_supervise.py

@@ -38,8 +38,14 @@ from ..discovery._addr import (
     UnwrappedAddress,
     mk_uuid,
 )
-from ._state import current_actor, is_main_process
-from ..log import get_logger, get_loglevel
+from ._state import (
+    current_actor,
+    is_main_process,
+)
+from ..log import (
+    get_logger,
+    get_loglevel,
+)
 from ._runtime import Actor
 from ._portal import Portal
 from ..trionics import (
@@ -47,6 +53,7 @@ from ..trionics import (
     collapse_eg,
 )
 from .._exceptions import (
+    ActorTooSlowError,
     ContextCancelled,
 )
 from .._root import (
@@ -60,11 +67,93 @@ if TYPE_CHECKING:
     import multiprocessing as mp
     # from ..ipc._server import IPCServer
     from ..ipc import IPCServer
+    from ..spawn._spawn import ProcessType
 
 
 log = get_logger()
 
 
+async def _try_cancel_then_kill(
+    portal: Portal,
+    # `ProcessType` is `TYPE_CHECKING`-only (defined under that
+    # guard in `..spawn._spawn`) so we stringify here to avoid
+    # eager runtime eval of the annotation at function-def time
+    # (this module has no `from __future__ import annotations`).
+    proc: 'ProcessType',
+    subactor: Actor,
+    debug_mode_active: bool = False,
+) -> None:
+    '''
+    Per-child cancel-then-escalate helper used by
+    `ActorNursery.cancel()`.
+
+    Sends a graceful actor-runtime cancel-RPC via
+    `Portal.cancel_actor(raise_on_timeout=True)`. If the bounded-wait
+    expires before the peer ack's, `ActorTooSlowError` is raised and
+    we escalate via `proc.terminate()` (SIGTERM) per SC-discipline:
+
+      graceful cancel-req -> bounded wait -> hard-kill
+
+    Without this escalation, a same-name sibling subactor whose
+    cancel-RPC failed to ack within `Portal.cancel_timeout` (e.g.
+    under TCP+forkserver register-RPC contention) would park the
+    parent's `soft_kill()` watcher forever waiting on `proc.poll()`,
+    deadlocking nursery `__aexit__`. See `ActorTooSlowError` for
+    the wider write-up.
+
+    '''
+    # XXX, do NOT escalate to `proc.terminate()` while ANY of
+    # the following are true — SIGTERM-ing a sub would tear
+    # down its sub-tree including any descendant proxying
+    # stdio to/from a REPL-locked actor, clobbering the user's
+    # debug session:
+    #
+    # - `Lock.ctx_in_debug is not None`: most precise — some
+    #   actor in the tree is currently REPL-locked. Set in the
+    #   root actor for the lifetime of the lock. Raceable
+    #   (false negative if SIGINT arrives before lock-acquire
+    #   RPC completes).
+    #
+    # - `_runtime_vars['_debug_mode']`: root-actor was opened
+    #   with `debug_mode=True` (via `open_root_actor` /
+    #   `open_nursery`). Set once at root boot, never cleared.
+    #   Catches deep-descendant REPL sessions even when the
+    #   intermediate nurseries didn't pass `debug_mode=` per-
+    #   child.
+    #
+    # - `debug_mode_active`: this nursery has at least one
+    #   child started with an explicit `debug_mode=` arg
+    #   (`ActorNursery._at_least_one_child_in_debug`). Catches
+    #   the case where root is NOT in debug-mode but a
+    #   nursery-direct child opted in.
+    #
+    # Independent because root may NOT be in debug-mode even
+    # when a child is (only the child's `_runtime_vars` is
+    # mutated by per-child `debug_mode=True`). ORing covers
+    # every flavor without false-positively skipping
+    # legitimate hard-kill paths in non-debug trees.
+    if (
+        debug.Lock.ctx_in_debug is not None
+        or
+        _state._runtime_vars.get('_debug_mode', False)
+        or
+        debug_mode_active
+    ):
+        await portal.cancel_actor()
+        return
+
+    try:
+        await portal.cancel_actor(raise_on_timeout=True)
+    except ActorTooSlowError as too_slow:
+        log.error(
+            f'Cancel-ack TIMED OUT for sub-actor\n'
+            f'  uid: {subactor.aid.reprol()!r}\n'
+            f'  reason: {too_slow}\n'
+            f'-> escalating to `proc.terminate()` (hard-kill)\n'
+        )
+        proc.terminate()
+
+
 class ActorNursery:
     '''
     The fundamental actor supervision construct: spawn and manage
@@ -428,10 +517,23 @@ class ActorNursery:
                                 else:  # there's no other choice left
                                     proc.terminate()
 
-                        # spawn cancel tasks for each sub-actor
+                        # spawn per-child cancel tasks; the helper
+                        # escalates to hard-kill on
+                        # `ActorTooSlowError` rather than silently
+                        # swallowing the cancel-ack timeout, EXCEPT
+                        # when this nursery has any debug-eligible
+                        # child (in which case we keep legacy
+                        # fire-and-forget semantics to avoid
+                        # clobbering an active REPL).
                         assert portal
                         if portal.channel.connected():
-                            tn.start_soon(portal.cancel_actor)
+                            tn.start_soon(
+                                _try_cancel_then_kill,
+                                portal,
+                                proc,
+                                subactor,
+                                self._at_least_one_child_in_debug,
+                            )
 
                 log.cancel(msg)
         # if we cancelled the cancel (we hung cancelling remote actors)

tractor/spawn/_reap.py

@@ -0,0 +1,183 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Post-mortem subactor cleanup primitives — things the parent
+runtime has to clean up because the dead-or-SIGKILL'd child
+couldn't.
+
+Sibling of `tractor._testing._reap` which is the test-harness
+equivalent (orphan-pid + leaked-shm + leaked-UDS-sock sweeper
+fixtures). This module is the spawn-layer counterpart, called
+inline from `hard_kill` and the broader subactor reap path.
+
+Today this is just `unlink_uds_bind_addrs()`. As future
+post-mortem cleanup needs surface (e.g. `/dev/shm` segment
+unlink for hard-crashed actors, leaked-pidfile cleanup), they
+land here too.
+
+Future-work TODO — authoritative UDS bind-addr tracking
+-------------------------------------------------------
+
+`unlink_uds_bind_addrs()` currently has two cleanup paths:
+
+1. Explicit `bind_addrs` (when parent set them at spawn time)
+2. **Convention-based reconstruction** —
+   `<XDG_RUNTIME_DIR>/tractor/<name>@<pid>.sock` — for the
+   common case where the subactor self-assigned a random sock
+   via `UDSAddress.get_random()`.
+
+Path (2) hardcodes the `<name>@<pid>.sock` convention from
+`tractor.ipc._uds.UDSAddress`. If that convention ever
+changes — or the subactor binds to a non-default
+`bindspace`/`filedir` — we'll silently fail to unlink.
+
+A more authoritative approach would be:
+
+- Subactors register their bound UDS sockpaths in a
+  per-process registry inside `tractor.ipc._uds` at
+  `start_listener()` time.
+- The subactor reports its bound sockpath(s) back to the
+  parent over IPC immediately post-bind (extension to
+  `SpawnSpec` reply / a new handshake msg).
+- Parent caches the subactor's authoritative sockpaths.
+- `unlink_uds_bind_addrs()` checks the cache FIRST, falls
+  back to convention-reconstruction if the subactor died
+  before reporting (which is the SIGKILL case this fn
+  primarily exists for).
+
+Tracked as future work in #454 (the parent UDS-leak
+issue this module addresses); a separate issue may be
+filed if/when the registry impl is scoped.
+
+See also #452 — the discovery-client `CLOSE_WAIT` TCP
+fd leak. Different bug class but same broader theme of
+"fork-spawn unmasked latent cleanup gaps".
+
+'''
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING
+
+import trio
+
+from tractor.discovery._addr import (
+    UnwrappedAddress,
+    wrap_address,
+)
+from tractor.ipc._uds import UDSAddress
+from tractor.log import get_logger
+
+
+if TYPE_CHECKING:
+    from tractor.runtime._runtime import Actor
+
+
+log = get_logger('tractor')
+
+
+def unlink_uds_bind_addrs(
+    proc: trio.Process,
+    *,
+    bind_addrs: list[UnwrappedAddress] | None = None,
+    subactor: Actor | None = None,
+) -> None:
+    '''
+    Best-effort post-mortem cleanup of any UDS sock-files
+    a hard-killed subactor was bound to.
+
+    SIGKILL bypasses Python execution → the subactor's
+    `_serve_ipc_eps` `finally:` block (which normally calls
+    `os.unlink(addr.sockpath)`) never runs. Without this
+    parent-side cleanup, the dead subactor's
+    `${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` file
+    accumulates on the filesystem (see issue #454 + the
+    autouse `_track_orphaned_uds_per_test` fixture).
+
+    Two cleanup paths, in order:
+
+    1. **Explicit `bind_addrs`** — when the parent set the
+       subactor's bind addrs at spawn time, unlink each
+       UDS-flavored sockpath directly.
+    2. **Self-assigned reconstruction** — when
+       `bind_addrs` is empty (the common case: subactor
+       picked its own random sock via
+       `UDSAddress.get_random()`), reconstruct the path
+       from `(subactor.aid.name, proc.pid)` using the
+       same `<name>@<pid>.sock` convention. We can do this
+       because the subactor uses its OWN `os.getpid()` at
+       bind time, which equals `proc.pid` from the
+       parent's view.
+
+    Idempotent: `FileNotFoundError` (graceful exit
+    already-unlinked, or sock never bound under early-
+    spawn cancel) is silenced; other `OSError`s log a
+    warning but never raise. TCP / non-UDS bind addrs are
+    skipped.
+
+    '''
+    sockpaths: list[str] = []
+
+    # path 1: explicit bind_addrs set at spawn time
+    for unwrapped in (bind_addrs or ()):
+        try:
+            addr = wrap_address(unwrapped)
+        except Exception:
+            log.exception(
+                f'Failed to wrap addr for UDS post-kill cleanup '
+                f'— skipping {unwrapped!r}\n'
+            )
+            continue
+        if isinstance(addr, UDSAddress):
+            sockpaths.append(str(addr.sockpath))
+
+    # path 2: reconstruct from subactor name + proc pid
+    # for the random-self-assign case (bind_addrs=None)
+    #
+    # TODO authoritative tracking — see module docstring.
+    if (
+        not sockpaths
+        and subactor is not None
+        and proc.pid is not None
+    ):
+        sockname: str = f'{subactor.aid.name}@{proc.pid}.sock'
+        sockpath: str = str(
+            UDSAddress.def_bindspace / sockname
+        )
+        sockpaths.append(sockpath)
+
+    for sockpath in sockpaths:
+        try:
+            os.unlink(sockpath)
+            log.runtime(
+                f'Unlinked orphaned UDS sock-file post-SIGKILL\n'
+                f' |_{proc}\n'
+                f' |_{sockpath}\n'
+            )
+        except FileNotFoundError:
+            # raced — subactor cleaned up before SIGKILL,
+            # OR sockfile never bound (early-spawn cancel),
+            # OR transport wasn't UDS this run.
+            pass
+        except OSError as exc:
+            log.warning(
+                f'Failed to unlink subactor UDS sock-file '
+                f'post-SIGKILL\n'
+                f' |_{proc}\n'
+                f' |_{sockpath}\n'
+                f' |_{exc!r}\n'
+            )

tractor/spawn/_spawn.py

@@ -22,6 +22,7 @@ over multiple backends.
 from __future__ import annotations
 import multiprocessing as mp
 import platform
+import sys
 from typing import (
     Any,
     Awaitable,
@@ -39,7 +40,11 @@ from tractor.runtime._state import (
     _runtime_vars,
 )
 from tractor.log import get_logger
-from tractor.discovery._addr import UnwrappedAddress
+from tractor.discovery._addr import (
+    UnwrappedAddress,
+)
+from .._exceptions import ActorFailure
+from ._reap import unlink_uds_bind_addrs
 from tractor.runtime._portal import Portal
 from tractor.runtime._runtime import Actor
 from tractor.msg import types as msgtypes
@@ -61,6 +66,34 @@ SpawnMethodKey = Literal[
     'trio',  # supported on all platforms
     'mp_spawn',
     'mp_forkserver',  # posix only
+    'subint',  # py3.14+ via `concurrent.interpreters` (PEP 734)
+    # EXPERIMENTAL — blocked at the CPython level. The
+    # design goal was a `trio+fork`-safe subproc spawn via
+    # `os.fork()` from a trio-free launchpad sub-interpreter,
+    # but CPython's `PyOS_AfterFork_Child` → `_PyInterpreterState_DeleteExceptMain`
+    # requires fork come from the main interp. See
+    # `tractor.spawn._subint_fork` +
+    # `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
+    # + issue #379 for the full analysis.
+    'subint_fork',
+    # EXPERIMENTAL — the `subint_fork` workaround. `os.fork()`
+    # from a non-trio worker thread (never entered a subint)
+    # is CPython-legal and works cleanly; forked child runs
+    # `tractor._child._actor_child_main()` against a trio
+    # runtime, exactly like `trio_proc` but via fork instead
+    # of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
+    'main_thread_forkserver',
+    # Variant-2: same fork machinery as `main_thread_forkserver`
+    # but the child enters a sub-interpreter to host its
+    # `trio.run()`. Gated on jcrist/msgspec#1026 unblocking
+    # PEP 684 isolated-mode subints upstream — until then
+    # `subint_forkserver_proc` is a clean `NotImplementedError`
+    # stub pointing at variant-1 (`main_thread_forkserver`) +
+    # the upstream blocker. The key is reserved here (not just
+    # aliased to variant-1) so once upstream lands the impl can
+    # flip in-place without API churn. See
+    # `tractor.spawn._subint_forkserver`.
+    'subint_forkserver',
 ]
 _spawn_method: SpawnMethodKey = 'trio'
 
@@ -78,6 +111,71 @@ else:
         await trio.lowlevel.wait_readable(proc.sentinel)
 
 
+async def wait_for_peer_or_proc_death(
+    ipc_server,
+    uid: tuple[str, str],
+    # TODO? not not types?
+    proc_wait: 'Callable[[], Awaitable]',
+    proc_repr: str = '',
+
+) -> 'tuple[trio.Event, Channel]':
+    '''
+    Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
+    own `.wait()` coroutine. Whichever completes first cancels
+    the other.
+
+    Used by every spawn-backend to detect a sub-actor that
+    *dies during boot* before completing the parent-handshake-
+    callback (e.g. crashed on import, exec'd-out, kernel-killed
+    pre-`_actor_child_main`). Without this race, the
+    handshake-wait — backed by an unsignalled `trio.Event` —
+    parks the spawning task forever and leaves the dead child
+    as a zombie since nobody calls `proc.wait()` to reap.
+
+    On normal handshake-complete: returns `(event, chan)`
+    identical to a bare `wait_for_peer`.
+
+    On proc-death-first: raises `ActorFailure` carrying the
+    proc's exit code, allowing the supervisor to surface a
+    clean error rather than hanging indefinitely.
+
+    `proc_wait` is a 0-arg async callable returning the proc's
+    exit-status — kept generic so each backend can pass its
+    own (`trio.Process.wait`, `_ForkedProc.wait`,
+    `proc_waiter(mp.Process)`, etc.).
+
+    `proc_repr` is an optional string used in the
+    `ActorFailure` message for diag.
+
+    '''
+    result: dict = {}
+
+    async def _await_handshake():
+        event, chan = await ipc_server.wait_for_peer(uid)
+        result['handshake'] = (event, chan)
+        boot_n.cancel_scope.cancel()
+
+    async def _await_death():
+        rc = await proc_wait()
+        result['died'] = rc
+        boot_n.cancel_scope.cancel()
+
+    async with trio.open_nursery() as boot_n:
+        boot_n.start_soon(_await_handshake)
+        boot_n.start_soon(_await_death)
+
+    if 'handshake' in result:
+        return result['handshake']
+
+    # only reached if proc-death won the race
+    raise ActorFailure(
+        f'Sub-actor {uid!r} died during boot '
+        f'(rc={result.get("died")!r}) before completing '
+        f'parent-handshake.\n'
+        f'  proc: {proc_repr}'
+    )
+
+
 def try_set_start_method(
     key: SpawnMethodKey
 
@@ -110,7 +208,38 @@ def try_set_start_method(
         case 'mp_spawn':
             _ctx = mp.get_context('spawn')
 
-        case 'trio':
+        case (
+            'trio'
+            | 'main_thread_forkserver'
+        ):
+            _ctx = None
+
+        case (
+            'subint'
+            | 'subint_fork'
+            | 'subint_forkserver'
+        ):
+            # All subint-family backends need no `mp.context`;
+            # all four feature-gate on the py3.14 public
+            # `concurrent.interpreters` wrapper (PEP 734). See
+            # `tractor.spawn._subint` for the detailed
+            # reasoning. `subint_fork` is blocked at the
+            # CPython level (raises `NotImplementedError`);
+            # `main_thread_forkserver` is the working
+            # variant-1 backend; `subint_forkserver` aliases
+            # to it today, reserved for the future variant-2
+            # subint-isolated-child runtime once upstream
+            # msgspec#1026 unblocks.
+            from ._subint import _has_subints
+            if not _has_subints:
+                raise RuntimeError(
+                    f'Spawn method {key!r} requires Python 3.14+.\n'
+                    f'(On py3.13 the private `_interpreters` C '
+                    f'module exists but tractor\'s spawn flow '
+                    f'wedges — see `tractor.spawn._subint` '
+                    f'docstring for details.)\n'
+                    f'Current runtime: {sys.version}'
+                )
             _ctx = None
 
         case _:
@@ -223,6 +352,16 @@ async def hard_kill(
     # whilst also hacking on it XD
     # terminate_after: int = 99999,
 
+    *,
+    # Subactor's bind addresses + subactor record, used
+    # for post-SIGKILL UDS sockpath cleanup. Optional for
+    # legacy callers; new call sites should pass at least
+    # `subactor` (which lets us reconstruct the sock path
+    # from `aid.name + proc.pid` when `bind_addrs` is
+    # empty/self-assigned). See `._reap.unlink_uds_bind_addrs()`.
+    bind_addrs: list[UnwrappedAddress] | None = None,
+    subactor: Actor | None = None,
+
 ) -> None:
     '''
     Un-gracefully terminate an OS level `trio.Process` after timeout.
@@ -310,6 +449,21 @@ async def hard_kill(
         )
         proc.kill()
 
+    # Post-mortem UDS sockpath cleanup. SIGKILL bypassed
+    # the subactor's normal `os.unlink(addr.sockpath)` in
+    # `_serve_ipc_eps`'s `finally:`; the parent has the
+    # bind addrs (or can reconstruct from name + pid) so
+    # we do it here. Runs UNCONDITIONALLY (graceful-exit
+    # case is a no-op via `FileNotFoundError` skip in the
+    # helper) so the cleanup also covers the "cancelled
+    # during spawn" path where the subactor never reached
+    # its IPC server finally block.
+    unlink_uds_bind_addrs(
+        proc,
+        bind_addrs=bind_addrs,
+        subactor=subactor,
+    )
+
 
 async def soft_kill(
     proc: ProcessType,
@@ -437,6 +591,10 @@ async def new_proc(
 # `hard_kill`/`proc_waiter` from this module.
 from ._trio import trio_proc
 from ._mp import mp_proc
+from ._subint import subint_proc
+from ._subint_fork import subint_fork_proc
+from ._main_thread_forkserver import main_thread_forkserver_proc
+from ._subint_forkserver import subint_forkserver_proc
 
 
 # proc spawning backend target map
@@ -444,4 +602,25 @@ _methods: dict[SpawnMethodKey, Callable] = {
     'trio': trio_proc,
     'mp_spawn': mp_proc,
     'mp_forkserver': mp_proc,
+    'subint': subint_proc,
+    # blocked at CPython level — see `_subint_fork.py` +
+    # `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
+    # Kept here so `--spawn-backend=subint_fork` routes to a
+    # clean `NotImplementedError` with pointer to the analysis,
+    # rather than an "invalid backend" error.
+    'subint_fork': subint_fork_proc,
+    # Variant-1 (working today): fork from a regular main-interp
+    # worker thread, child runs trio on its own main interp.
+    # Validated by
+    # `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`.
+    # See `tractor.spawn._main_thread_forkserver`.
+    'main_thread_forkserver': main_thread_forkserver_proc,
+    # Variant-2 (future, reserved): same fork machinery but
+    # child enters a sub-interpreter to host its `trio.run()`
+    # — gated on jcrist/msgspec#1026 unblocking PEP 684
+    # isolated-mode subints. Today the stub raises
+    # `NotImplementedError` pointing at the variant-1 backend
+    # + upstream blocker. See
+    # `tractor.spawn._subint_forkserver`.
+    'subint_forkserver': subint_forkserver_proc,
 }