Add snapshot evidence to cancel-cascade MTF issue doc

Append "Snapshot evidence (2026-05-13)" section to
`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`
documenting `fail_after_w_trace` diag capture results for
`test_nested_multierrors` under the MTF backend — reproduction cmd,
ptree analysis, observed hang signature, and updated triage plan.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 5372fd178a)

.github/workflows/ci.yml

@@ -88,10 +88,27 @@ jobs:
 
 
   testing:
-    name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
+    name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
-    timeout-minutes: 16
+    timeout-minutes: 20
     runs-on: ${{ matrix.os }}
 
+    # NOTE on the matrix shape — the `capture=` mode follows
+    # `spawn_backend`:
+    #
+    # - `trio` / `mp_*` backends use `--capture=fd` (default)
+    #   for per-test attribution of subactor *raw-fd* output
+    #   in failure reports.
+    # - Fork-based backends (`main_thread_forkserver`,
+    #   `subint_forkserver`) REQUIRE `--capture=sys` because
+    #   fork-child × `--capture=fd` is a known deadlock
+    #   pattern. See the long NOTE in `tractor._testing.pytest`'s
+    #   `pytest_load_initial_conftests` for the mechanism +
+    #   tradeoff write-up.
+    #
+    # If a future matrix row adds a fork-spawn backend
+    # WITHOUT setting `capture: 'sys'`, the
+    # `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
+    # with a clear error msg. So the matrix is self-policing.
     strategy:
       fail-fast: false
       matrix:
@@ -118,6 +135,26 @@ jobs:
           'tcp',
           'uds',
         ]
+        capture: [
+          'fd',  # default for non-fork backends
+        ]
+
+        # Fork-based backends — added via `include:` so each
+        # cell carries its REQUIRED `capture: 'sys'` mode.
+        # Linux-only for now; macOS coverage TBD pending
+        # local validation.
+        include:
+          - os: ubuntu-latest
+            python-version: '3.13'
+            spawn_backend: 'main_thread_forkserver'
+            tpt_proto: 'tcp'
+            capture: 'sys'
+          - os: ubuntu-latest
+            python-version: '3.13'
+            spawn_backend: 'main_thread_forkserver'
+            tpt_proto: 'uds'
+            capture: 'sys'
+
         # https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
         exclude:
           # don't do UDS run on macOS (for now)
@@ -158,7 +195,11 @@ jobs:
           -rsx
           --spawn-backend=${{ matrix.spawn_backend }}
           --tpt-proto=${{ matrix.tpt_proto }}
-          --capture=fd
+          --capture=${{ matrix.capture }}
+        # NOTE: capture mode is matrix-driven — `fd` for
+        # non-fork backends (per-test fd attribution),
+        # `sys` for fork-based (avoids fork-child x
+        # capture-fd deadlock). See matrix-NOTE above.
 
   # XXX legacy NOTE XXX
   #

ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md

@@ -0,0 +1,314 @@
+# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
+
+## Symptom
+
+Running the full test suite under
+
+```bash
+./py313/bin/python -m pytest tests/ \
+  --tpt-proto=tcp \
+  --spawn-backend=main_thread_forkserver
+```
+
+surfaces a single, **rotating** `trio.TooSlowError`
+failure each run. The failure isn't deterministic on
+test identity — different test each run — but it
+ALWAYS looks like:
+
+```
+FAILED tests/<file>::test_<name> - trio.TooSlowError
+==== 1 failed, 373 passed, 17 skipped, 11–12 xfailed,
+       0–1 xpassed, ~550 warnings in ~6min ====
+```
+
+Pass rate: **~99.7%** (373 of 374 non-skip tests).
+Wall-clock per full run: 5–6 min.
+
+## Tests observed flaking so far
+
+Each row was the SOLE failure in a separate run:
+
+| run # | test |
+|---|---|
+| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
+| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
+
+Both share the same shape:
+
+- **Cancel cascade** of N subactors back to a parent root actor.
+- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
+  (it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
+- N ≈ 2 for `test_context_spawns_aio_task_that_errors` —
+  but each subactor is `infect_asyncio=True`, so each
+  cancel involves the trio↔asyncio guest-run unwind
+  which is structurally heavier than pure-trio.
+- Test wraps the cascade in `trio.fail_after(N seconds)`
+  and the cap fires before the cascade completes.
+
+The exact failing test rotates because each test is
+independently close to the cap; whichever happens to
+be unlucky in scheduling/CPU-contention on a given run
+is the one that times out.
+
+## Root-cause family
+
+`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
+the SC-graceful teardown ladder per subactor:
+
+1. `Portal.cancel_actor()` — graceful IPC cancel-req.
+2. Wait `terminate_after=1.6s` for sub to exit.
+3. If still alive: `proc.kill()` (SIGKILL).
+4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
+   sock-file cleanup for UDS leaks (issue #452 fix).
+
+For a cascade of N subactors, each pays steps 1–4. If
+graceful-cancel doesn't complete within 1.6s for ANY
+sub, that sub eats a full 1.6s of `move_on_after` plus
+the `proc.wait()` post-SIGKILL.
+
+Worst case under fork backend with N=cpus subs:
+- N × 1.6s = 16s+ on a 10-core box just for the
+  graceful timeout phase
+- Plus per-spawn fork-IPC handshake cost compounds
+  during teardown (each sub's IPC cleanup goes through
+  the same forkserver coordinator)
+- Plus the new autouse fixtures
+  (`_track_orphaned_uds_per_test`,
+  `_detect_runaway_subactors_per_test`,
+  `_reap_orphaned_subactors`) all run at test
+  teardown, adding small (10s of ms) but cumulative
+  overhead
+
+Current cap: 30s (`fail_after_s = 30 if
+is_forking_spawner else 12`). Empirically fits the
+median run but the tail breaks ~0.3% of the time.
+
+## NOT regressing
+
+To confirm this is a flake and not a regression:
+
+- Pre-`WakeupSocketpair`-patch baseline: tests
+  HUNG INDEFINITELY (busy-loop never released).
+- Post-patch: pass-or-fail-fast, ~99.7% pass, the
+  occasional cap-hit fails in bounded time (<60s for
+  the offending test).
+- Same test PASSES under `--spawn-backend=trio`
+  (no fork, no hard-kill compounding).
+
+So the suite is dramatically better than before; the
+remaining flake is a known-tolerable steady-state.
+
+## Possible mitigations (ranked)
+
+### A. Bump the cap further
+
+Cheapest. Change the per-test `fail_after_s` from 30
+to e.g. 60 for fork backends. Pros: trivial. Cons:
+masks any genuine slowness regression we'd want to
+catch.
+
+### B. CPU-count-aware cap
+
+For tests whose N scales with `cpu_count()`, scale
+the cap too:
+
+```python
+fail_after_s = (
+    max(30, cpu_count() * 3)  # 3s/actor floor
+    if is_forking_spawner
+    else 12
+)
+```
+
+Pros: scales with the actual cancel-cascade work.
+Cons: still arbitrary multiplier.
+
+### C. `pytest-rerunfailures` for these tests only
+
+Mark the known-flaky tests with
+`@pytest.mark.flaky(reruns=1)` (needs
+`pytest-rerunfailures` dep). Single retry hides
+genuine ~0.3% transient flakes.
+
+Pros: no cap change, surfaces persistent failures
+loudly. Cons: adds a dep, retries can mask real bugs
+if used widely.
+
+### D. Reduce `hard_kill`'s `terminate_after`
+
+Drop from 1.6s → 0.8s. Cuts the worst-case cascade
+time roughly in half. Risks: fewer subs get a chance
+to run their cleanup before SIGKILL → more orphaned
+state for the autouse reapers to handle (ironically,
+adds back overhead elsewhere).
+
+### E. Profile + targeted fix
+
+Add `log.devx()` markers in `hard_kill` to time each
+phase. Identify if any subactor is consistently
+hitting the 1.6s cap (vs. exiting in <0.1s). If so,
+that sub has a teardown bug worth fixing at source.
+Pros: actually fixes the underlying slowness. Cons:
+real investigation work, deferred from this round.
+
+## Recommendation
+
+Land this issue-doc as the tracker. Apply **(B)** as
+a small follow-up — cheap and proportional. If it
+still flakes, escalate to **(E)** with a `log.devx()`
+profile-pass.
+
+`(C)` is a backstop if `(B)` doesn't quite get there
+and we need green CI faster than (E) can deliver.
+
+## Verification protocol
+
+After applying any mitigation:
+
+```bash
+# Run the suite N times back-to-back, count failures.
+# A persistent failure on the SAME test == real bug.
+# Failures rotating across tests == still cap-related.
+
+for i in $(seq 1 5); do
+  ./py313/bin/python -m pytest tests/ \
+    --tpt-proto=tcp \
+    --spawn-backend=main_thread_forkserver \
+    -q 2>&1 | tail -2
+done
+```
+
+Target: 0 failures across 5 runs ⇒ ship. 1–2 failures
+still rotating ⇒ apply (C). Same test failing twice
+⇒ escalate to (E).
+
+## Snapshot evidence (2026-05-13)
+
+After landing the `fail_after_w_trace` /
+`afk_alarm_w_trace` capture-on-timeout helpers
+(`tractor._testing.trace`), `test_nested_multierrors`
+on the `main_thread_forkserver` backend produces
+**reproducible diag snapshots** at
+`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
+
+### Reproduction
+
+```bash
+pytest \
+  -v --verbose --durations=10 \
+  --spawn-backend=main_thread_forkserver \
+  --tpt-proto=uds \
+  --capture=sys --show-capture=stderr -rxX \
+  tests/test_cancellation.py::test_nested_multierrors
+```
+
+The test is `xfail(strict=False)` for MTF — it RUNS
+each invocation so snapshots accumulate, but doesn't
+break `--lf` workflow.
+
+### Consistent shape across runs
+
+5+ snapshots taken back-to-back show the SAME pattern:
+
+- **Timing:** ~10s wall-clock total. Inner
+  `fail_after_w_trace(10)` fires at exactly T=10s;
+  cascade's `nursery.__aexit__` takes ~0.6s more to
+  gather + propagate the resulting
+  `BaseExceptionGroup`. **Trio backend completes the
+  SAME test in <6s** — so the MTF cascade is ~2x
+  slower at minimum.
+
+- **`BaseExceptionGroup` shape:** mixed
+  `[RemoteActorError, Cancelled]`. The first
+  subactor's natural error-propagation (`assert 0`
+  raised → `RemoteActorError` portal-result)
+  arrives before T=10s; the OTHER subactor's
+  portal-wait is still in flight at T=10s, gets
+  cancelled by `fail_after_w_trace`'s scope-cancel
+  → returns `Cancelled` instead.
+
+- **Orphan-spawn skew:** snapshot's `orphans` bucket
+  (after the `_is_tractor_subactor` cgroup-slice
+  override fix) consistently shows 2-4 init-adopted
+  procs at `depth_3` and `depth_1` levels — these
+  are the leaves whose parent (`depth_2` spawner)
+  was killed mid-cascade but who hadn't yet seen
+  the cancel signal themselves.
+
+- **UDS sock-leak:** 2-6 dead-orphan socks per run
+  (varies with cascade timing). The
+  `track_orphaned_uds_per_test` fixture reaps them
+  post-test → contamination is isolated per-invocation.
+
+### Capture mechanism
+
+`fail_after_w_trace` covers two firing paths:
+
+1. **`trio.TooSlowError`** raised at scope-exit
+   (body returned cleanly past deadline) — direct
+   `except` handler captures.
+
+2. **Scope-cancel + body raises non-`Cancelled` exc**
+   (e.g. `nursery.__aexit__` wraps timeout-induced
+   `Cancelled` into a `BaseExceptionGroup` that
+   escapes before `trio.fail_after`'s exit-check
+   could fire `TooSlowError`) — body-raise `except`
+   handler checks `scope.cancel_called` and
+   captures if True. This path catches the
+   `test_nested_multierrors` shape specifically (see
+   "BaseExceptionGroup shape" above).
+
+The snapshot dir contains:
+- `trace.txt` — `ptree` + `hung_state` (kernel
+  `wchan`/`stack` + `py-spy dump --locals` when
+  sudo cached), with `include_strays=True`
+  surfacing any cross-test ghost subactor trees in
+  the `orphans` bucket.
+- `bindspace.txt` — UDS bindspace classification
+  (live-active / orphaned-alive / orphaned-dead).
+- `meta.json` — `{pid, label, captured_at, sudo_cached}`.
+
+The end-of-session `pytest_terminal_summary` hook
+in `tractor._testing.pytest` lists every snapshot
+dir from the run so you don't have to scroll back
+through captured-stderr lines:
+
+```
+========================= tractor hang-snapshot index ==========================
+N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
+  <test-id>
+    → /home/.../.cache/tractor/hung-dumps/<label>__<ts>
+```
+
+### Caveats
+
+The snapshot fires AFTER the body-raise (not at the
+exact moment of scope-cancel), so the parent's
+py-spy frames show `_do_capture_snapshot` itself
+running, NOT the cancel-cascade hang frame. To see
+the actual hang state, manual `acli.ptree` /
+`acli.hung_dump` from a second terminal at T=10s
+would be needed — **not currently possible**
+because per-test reaper fixtures clean up ~0.6s
+post-timeout. See follow-up TODO in
+`tractor/_testing/trace.py` for a
+`TRACTOR_TRACE_HOLD=1` env-var pause mode.
+
+## See also
+
+- [#452](https://github.com/goodboy/tractor/issues/452) —
+  UDS sock-file leak (related — `hard_kill`'s
+  cleanup phase contributes to cascade time)
+- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
+  — the upstream-trio fix that turned this from a
+  100% hang into a 0.3% flake
+- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
+  — the asyncio variant which contributes to one of
+  the rotating failures
+- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
+  cascade source
+- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
+  `_detect_runaway_subactors_per_test`,
+  `_reap_orphaned_subactors` — autouse cleanup
+  fixtures whose cumulative teardown overhead
+  contributes to the cascade time

ai/conc-anal/subint_fork_from_main_thread_smoketest.py

@@ -89,11 +89,13 @@ except ImportError:
 # the "zero tractor imports" isolation guarantee; now that
 # CPython-level feasibility is confirmed, the validated
 # primitives have moved into tractor proper.)
-from tractor.spawn._subint_forkserver import (
+from tractor.spawn._main_thread_forkserver import (
     fork_from_worker_thread,
-    run_subint_in_worker_thread,
     wait_child,
 )
+from tractor.spawn._subint_forkserver import (
+    run_subint_in_worker_thread,
+)
 
 
 # ----------------------------------------------------------------

tests/spawn/test_main_thread_forkserver.py

@@ -1,13 +1,14 @@
 '''
-Integration exercises for the `tractor.spawn._subint_forkserver`
+Integration exercises for the `tractor.spawn._main_thread_forkserver`
 submodule at three tiers:
 
 1. the low-level primitives
-   (`fork_from_worker_thread()` +
+   (`fork_from_worker_thread()` from `_main_thread_forkserver`
-   `run_subint_in_worker_thread()`) driven from inside a real
+   + `run_subint_in_worker_thread()` from
+   `_subint_forkserver`) driven from inside a real
    `trio.run()` in the parent process,
 
-2. the full `subint_forkserver_proc` spawn backend wired
+2. the full `main_thread_forkserver_proc` spawn backend wired
    through tractor's normal actor-nursery + portal-RPC
    machinery — i.e. `open_root_actor` + `open_nursery` +
    `run_in_actor` against a subactor spawned via fork from a
@@ -27,15 +28,15 @@ Those smoke-test scenarios are standalone — no trio runtime
 in the *parent*. Tiers (1)+(2) here cover the primitives
 driven from inside `trio.run()` in the parent, and tier (3)
 (the `*_spawn_basic` test) drives the registered
-`subint_forkserver` spawn backend end-to-end against the
+`main_thread_forkserver` spawn backend end-to-end against
-tractor runtime.
+the tractor runtime.
 
 Gating
 ------
 - py3.14+ (via `concurrent.interpreters` presence)
 - no `--spawn-backend` restriction — the backend-level test
   flips `tractor.spawn._spawn._spawn_method` programmatically
-  (via `try_set_start_method('subint_forkserver')`) and
+  (via `try_set_start_method('main_thread_forkserver')`) and
   restores it on teardown, so these tests are independent of
   the session-level CLI backend choice.
 
@@ -64,11 +65,13 @@ from tractor.devx import dump_on_hang
 # `tractor.spawn._subint` for why.
 pytest.importorskip('concurrent.interpreters')
 
-from tractor.spawn._subint_forkserver import (  # noqa: E402
+from tractor.spawn._main_thread_forkserver import (  # noqa: E402
     fork_from_worker_thread,
-    run_subint_in_worker_thread,
     wait_child,
 )
+from tractor.spawn._subint_forkserver import (  # noqa: E402
+    run_subint_in_worker_thread,
+)
 from tractor.spawn import _spawn as _spawn_mod  # noqa: E402
 from tractor.spawn._spawn import try_set_start_method  # noqa: E402
 
@@ -195,7 +198,7 @@ def test_fork_from_worker_thread_via_trio(
     deadline: float = 10.0
     with dump_on_hang(
         seconds=deadline,
-        path='/tmp/subint_forkserver_baseline.dump',
+        path='/tmp/main_thread_forkserver_baseline.dump',
     ):
         pid: int = trio.run(
             partial(run_fork_in_non_trio_thread, deadline),
@@ -217,14 +220,14 @@ def test_fork_and_run_trio_in_child() -> None:
     `trio.run()` inside it on yet another worker thread.
 
     This is the full "forkserver + trio-in-subint-in-child"
-    pattern the proposed `subint_forkserver` spawn backend
+    pattern the proposed `main_thread_forkserver` spawn backend
     would rest on.
 
     '''
     deadline: float = 15.0
     with dump_on_hang(
         seconds=deadline,
-        path='/tmp/subint_forkserver_trio_in_child.dump',
+        path='/tmp/main_thread_forkserver_trio_in_child.dump',
     ):
         pid: int = trio.run(
             partial(
@@ -237,7 +240,7 @@ def test_fork_and_run_trio_in_child() -> None:
 
 
 # ----------------------------------------------------------------
-# tier-3 backend test: drive the registered `subint_forkserver`
+# tier-3 backend test: drive the registered `main_thread_forkserver`
 # spawn backend end-to-end through tractor's actor-nursery +
 # portal-RPC machinery.
 # ----------------------------------------------------------------
@@ -260,7 +263,7 @@ async def _happy_path_forkserver(
     Parent-side harness: stand up a root actor, open an actor
     nursery, spawn one subactor via the currently-selected
     spawn backend (which this test will have flipped to
-    `subint_forkserver`), run a trivial RPC through its
+    `main_thread_forkserver`), run a trivial RPC through its
     portal, assert the round-trip result.
 
     '''
@@ -304,19 +307,19 @@ def forkserver_spawn_method():
 
 
 @pytest.mark.timeout(60, method='thread')
-def test_subint_forkserver_spawn_basic(
+def test_main_thread_forkserver_spawn_basic(
     reg_addr: tuple[str, int | str],
     forkserver_spawn_method,
 ) -> None:
     '''
     Happy-path: spawn ONE subactor via the
-    `subint_forkserver` backend (parent-side fork from a
+    `main_thread_forkserver` backend (parent-side fork from a
     main-interp worker thread), do a trivial portal-RPC
     round-trip, tear the nursery down cleanly.
 
     If this passes, the "forkserver + tractor runtime" arch
     is proven end-to-end: the registered
-    `subint_forkserver_proc` spawn target successfully
+    `main_thread_forkserver_proc` spawn target successfully
     forks a child, the child runs `_actor_child_main()` +
     completes IPC handshake + serves an RPC, and the parent
     reaps via `_ForkedProc.wait()` without regressing any of
@@ -326,7 +329,7 @@ def test_subint_forkserver_spawn_basic(
     deadline: float = 20.0
     with dump_on_hang(
         seconds=deadline,
-        path='/tmp/subint_forkserver_spawn_basic.dump',
+        path='/tmp/main_thread_forkserver_spawn_basic.dump',
     ):
         trio.run(
             partial(
@@ -340,7 +343,7 @@ def test_subint_forkserver_spawn_basic(
 # ----------------------------------------------------------------
 # tier-4 DRAFT: orphaned-subactor SIGINT survivability
 #
-# Motivating question: with `subint_forkserver`, the child's
+# Motivating question: with `main_thread_forkserver`, the child's
 # `trio.run()` lives on the fork-inherited worker thread which
 # is NOT `threading.main_thread()` — so trio cannot install its
 # `signal.set_wakeup_fd`-based SIGINT handler. If the parent
@@ -360,7 +363,7 @@ def test_subint_forkserver_spawn_basic(
 # Cross-backend generalization (decide after this passes):
 # - applicable to any backend whose subactors are separate OS
 #   processes: `trio`, `mp_spawn`, `mp_forkserver`,
-#   `subint_forkserver`.
+#   `main_thread_forkserver`.
 # - NOT applicable to plain `subint` (subactors are in-process
 #   subinterpreters, no orphan child process to SIGINT).
 # - move path: lift the harness script into
@@ -446,7 +449,7 @@ def _process_alive(pid: int) -> bool:
         return False
 
 
-# Known-gap test — `subint_forkserver` orphan-SIGINT
+# Known-gap test — `main_thread_forkserver` orphan-SIGINT
 # handling. See
 # `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
 # `strict=True` so if a future fix closes the gap the
@@ -471,12 +474,12 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
 ) -> None:
     '''
     DRAFT — orphaned-subactor SIGINT survivability under the
-    `subint_forkserver` backend.
+    `main_thread_forkserver` backend.
 
     Sequence:
       1. Spawn a harness subprocess that brings up a root
          actor + one `sleep_forever` subactor via
-         `subint_forkserver`.
+         `main_thread_forkserver`.
       2. Read the harness's stdout for `PARENT_READY=<pid>`
          and `CHILD_PID=<pid>` markers (confirms the
          parent→child IPC handshake completed).
@@ -524,7 +527,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
         [
             sys.executable,
             str(script_path),
-            'subint_forkserver',
+            'main_thread_forkserver',
             host,
             str(port),
         ],
@@ -577,7 +580,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
 
         pytest.fail(
             f'Orphan subactor (pid={child_pid}) did NOT exit '
-            f'within 10s of SIGINT under `subint_forkserver` '
+            f'within 10s of SIGINT under `main_thread_forkserver` '
             f'→ trio on non-main thread did not observe the '
             f'default CPython KeyboardInterrupt; backend needs '
             f'explicit SIGINT plumbing.'
@@ -600,3 +603,50 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
             proc.wait(timeout=2.0)
         except subprocess.TimeoutExpired:
             pass
+
+
+# ----------------------------------------------------------------
+# regression guard: variant-2 (`subint_forkserver`) placeholder
+# MUST raise `NotImplementedError` today — guards against future
+# commits accidentally re-aliasing the key to the variant-1
+# coroutine (which was a transient state during the rename).
+# ----------------------------------------------------------------
+def test_subint_forkserver_key_errors_cleanly() -> None:
+    '''
+    `--spawn-backend=subint_forkserver` is reserved for the
+    eventual variant-2 (subint-isolated child runtime)
+    backend, gated on jcrist/msgspec#1026 unblocking PEP 684
+    isolated-mode subints upstream.
+
+    Until that lands, the dispatch entry MUST raise
+    `NotImplementedError` immediately rather than silently
+    aliasing to `main_thread_forkserver_proc`. Verify the
+    error message also surfaces both the working-backend
+    pointer and the upstream-blocker ref so an operator
+    arriving at the error has somewhere to go.
+
+    '''
+    import asyncio
+    from tractor.spawn._spawn import _methods
+
+    proc = _methods['subint_forkserver']
+    with pytest.raises(NotImplementedError) as ei:
+        # signature args match `main_thread_forkserver_proc`'s
+        # — the stub raises before touching them so dummy
+        # values are fine.
+        asyncio.run(
+            proc(
+                'x', None, None, {}, [],
+                ('127.0.0.1', 0), {},
+            )
+        )
+
+    msg: str = str(ei.value)
+    assert 'main_thread_forkserver' in msg, (
+        f'stub error msg should redirect to the working '
+        f'variant-1 backend; got: {msg!r}'
+    )
+    assert 'msgspec#1026' in msg or '1026' in msg, (
+        f'stub error msg should reference the upstream '
+        f'blocker (jcrist/msgspec#1026); got: {msg!r}'
+    )

tractor/spawn/_main_thread_forkserver.py

@@ -38,6 +38,7 @@ Two empirical CPython properties drive the design:
    the forked child otherwise (`Fatal Python error: not main
    interpreter`). Full source-level walkthrough:
    `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
+
 2. **`os.fork()` from a regular `threading.Thread` attached to
    the *main* interpreter — i.e. a worker thread that has never
    entered a subint — works cleanly.** Empirically validated
@@ -86,9 +87,11 @@ costs:
 
 - **Sidecar lifecycle**: a second long-lived process per
   parent, with its own start/stop/health-check semantics.
+
 - **IPC overhead per spawn**: every actor-spawn round-trips
   an `mp` request message through a unix socket before any
   child code runs.
+
 - **State isolation by process boundary**: the sidecar can't
   share parent state at all — every spawn is a "cold" child
   re-importing modules from disk.
@@ -106,6 +109,7 @@ For the full variant-2 picture see
 1) we already get costs 1 + 2 collapsed; cost 3 will land
 when msgspec#1026 unblocks isolated-mode subints.
 
+
 What survives the fork? — POSIX semantics
 -----------------------------------------
 
@@ -113,33 +117,58 @@ A natural worry when forking from a parent that's running
 `trio.run()` on another thread: does that trio thread (and
 any other threads in the parent) keep running in the child?
 
-**No.** POSIX `fork()` only preserves the *calling* thread
+**No** — but with a precise meaning that's worth pinning
-in the child. Every other thread in the parent — trio's
+down, since the canonical trio framing
-runner thread, any `to_thread` cache threads, anything else
+([python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
-— is gone the instant `fork()` returns in the child.
+puts it the opposite-sounding way:
+
+> If you use `fork()` in a process with multiple threads,
+> all the other thread stacks are just leaked: there's
+> nothing else you can reasonably do with them.
+
+Both statements describe the same POSIX reality from
+opposite sides:
+
+- **Execution-side ("gone")**: POSIX `fork()` only
+  preserves the *calling* thread as a runnable thread in
+  the child. Every other thread in the parent — trio's
+  runner thread, any `to_thread` cache threads, anything
+  else — never executes another instruction post-fork.
+
+- **Memory-side ("leaked")**: those non-running threads'
+  *stacks* and per-thread heap structures are still
+  COW-inherited into the child's address space. They
+  persist as orphaned bytes with no owning thread, no
+  scheduler entry, and no way for the child to clean
+  them up — hence trio's word "leaked".
 
 Concretely, after the forkserver worker calls `os.fork()`:
 
-| thread                | parent    | child         |
+| thread              | parent    | child (executing) | child (memory)              |
-|-----------------------|-----------|---------------|
+|---------------------|-----------|-------------------|-----------------------------|
-| forkserver worker     | continues | sole survivor |
+| forkserver worker   | continues | sole survivor     | live stack                  |
-| `trio.run()` thread   | continues | gone          |
+| `trio.run()` thread | continues | not running       | leaked stack (zombie bytes) |
-| any other thread      | continues | gone          |
+| any other thread    | continues | not running       | leaked stack (zombie bytes) |
 
 The forkserver worker becomes the new "main" execution
 context in the child; `trio.run()` and every other parent
-thread never executes a single instruction post-fork in the
+thread never executes a single instruction post-fork.
-child.
+Their stack memory rides along as inert COW pages until
+the child's fresh `trio.run()` boots and overwrites/GCs
+it (or until the child `exec()`s and discards the entire
+image).
 
 This is exactly *why* `os.fork()` is delegated to a
 dedicated worker thread that has provably never entered
 trio: we want that trio-free thread to be the surviving
-one in the child.
+*executing* thread in the child, with the leaked trio
+stack reduced to inert COW pages we don't touch.
 
-That said, dead-thread *artifacts* still cross the fork
+The leaked-stack residue is one slice of the broader
-boundary (canonical "fork in a multithreaded program is
+"fork in a multithreaded program is dangerous" hazard
-dangerous" — see `man pthread_atfork`). What persists, and
+class (see `man pthread_atfork`). Other dead-thread
-how we handle each:
+artifacts that cross the fork boundary, and how we handle
+each:
 
 - **Inherited file descriptors** — the dead trio thread's
   epoll fd, signal-wakeup-fd, eventfds, sockets, IPC
@@ -148,16 +177,20 @@ how we handle each:
   `_close_inherited_fds()` in the child prelude — walks
   `/proc/self/fd` and closes everything except stdio +
   the channel pipe to the forkserver.
+
 - **Memory image** — trio's internal data structures
   (scheduler, task queues, runner state) sit in COW
-  memory but nobody's executing them. Get GC'd /
+  memory alongside the leaked stacks above. Nobody's
-  overwritten when the child's fresh `trio.run()` boots.
+  executing them; they get GC'd / overwritten when the
+  child's fresh `trio.run()` boots.
+
 - **Python thread state** — handled automatically by
   CPython. `PyOS_AfterFork_Child()` calls
   `_PyThreadState_DeleteExceptCurrent()`, so dead
   `PyThreadState` objects are cleaned and
   `threading.enumerate()` returns just the surviving
   thread.
+
 - **User-level locks (`threading.Lock`)** —
   held-by-dead-thread state is the canonical fork hazard.
   Not an issue in practice for tractor: trio doesn't hold
@@ -166,6 +199,7 @@ how we handle each:
   either direction). CPython's GIL is auto-reset by the
   fork callback.
 
+
 FYI: how this dodges the `trio.run()` × `fork()` hazards
 --------------------------------------------------------
 
@@ -183,13 +217,16 @@ design dodges each class explicitly:
   reader. *Dodge*: the inherited wakeup-fd is closed by
   `_close_inherited_fds()`, then the child's own
   `trio.run()` installs a fresh one.
+
 - **`epoll`/`kqueue` instance**: trio's I/O backend holds
   one. Inherited as a dead fd; same fix as above.
+
 - **Threadpool cache threads** (`trio.to_thread`): worker
   threads with cached tstate. Don't exist in the child
   (POSIX); cache state is meaningless garbage that gets
   reset when the child's trio.run() initializes its own
   thread cache.
+
 - **Cancel scopes / nurseries / open `trio.Process` /
   open sockets**: these are trio-runtime objects, not
   kernel objects. The runtime that owns them is gone in
@@ -197,9 +234,11 @@ design dodges each class explicitly:
   in COW memory and get overwritten as the child runs.
   Inherited *kernel* fds those objects wrapped (sockets,
   proc pipes) are caught by `_close_inherited_fds()`.
+
 - **`atexit` handlers**: trio doesn't register any that
   would mis-fire post-fork; trio's lifetime-stack is
   all `with`-block-scoped and dies with the runner.
+
 - **Foreign-language I/O state** (libcurl, OpenSSL session
   caches, etc.): out of scope — same hazard as any
   fork-without-exec; users layering those on top of
@@ -211,6 +250,7 @@ isolation + `_close_inherited_fds()` cleanup gives the
 forked child a clean trio environment. Everything else
 falls under the standard fork-without-exec disclaimer.
 
+
 Implementation status
 ---------------------
 
@@ -231,10 +271,11 @@ follow-up) including the
 
 Still-open work (tracked on tractor #379):
 
-- no cancellation / hard-kill stress coverage yet
+- [ ] no cancellation / hard-kill stress coverage yet
   (counterpart to `tests/test_subint_cancellation.py` for
   the plain `subint` backend),
-- `child_sigint='trio'` mode (flag scaffolded below; default
+
+- [ ] `child_sigint='trio'` mode (flag scaffolded below; default
   is `'ipc'`). Originally intended as a manual SIGINT →
   trio-cancel bridge, but investigation showed trio's
   handler IS already correctly installed in the fork-child
@@ -287,21 +328,24 @@ See also
 - `tractor.spawn._subint_forkserver` — variant-2 placeholder
   module; reserved for the future subint-isolated-child
   runtime once jcrist/msgspec#1026 unblocks.
+
 - `tractor.spawn._subint_fork` — the stub for the
   fork-from-non-main-subint strategy that DIDN'T work (kept
   in-tree as documentation of the attempt + the CPython-level
   block).
+
 - `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
   — CPython source walkthrough of why fork-from-subint is dead.
+
 - `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
   — standalone feasibility check (delegates to this module
   for the primitives it exercises).
 
 '''
 from __future__ import annotations
+import errno
 import os
 import signal
-import sys
 import threading
 from functools import partial
 from typing import (
@@ -324,8 +368,8 @@ from tractor.runtime._portal import Portal
 from ._spawn import (
     cancel_on_completion,
     soft_kill,
+    wait_for_peer_or_proc_death,
 )
-from ._subint import _has_subints
 
 if TYPE_CHECKING:
     from tractor.discovery._addr import UnwrappedAddress
@@ -423,9 +467,24 @@ def _close_inherited_fds(
         try:
             os.close(fd)
             closed += 1
-        except OSError:
+        except OSError as oserr:
-            # fd was already closed (race with listdir) or otherwise
+            # `EBADF` is the benign-and-expected case: the
-            # unclosable — either is fine.
+            # `os.listdir('/proc/self/fd')` call above itself
+            # opens a transient dirfd that ends up in
+            # `candidates`, then auto-closes before this loop
+            # reaches it. Same for any fd whose Python wrapper
+            # was GC'd between `listdir` and `os.close`.
+            # Suppress at debug-level — surfacing every
+            # EBADF as a full traceback (prior `log.exception`
+            # behavior) drowned the post-fork log channel.
+            if oserr.errno == errno.EBADF:
+                log.debug(
+                    f'Skip already-closed inherited fd {fd!r} '
+                    f'(EBADF, benign race with listdir)\n'
+                )
+                continue
+            # Other errnos (EIO / EPERM / EINTR / ...) are
+            # genuinely unexpected — keep the loud surface.
             log.exception(
                 f'Failed to close inherited fd in child ??\n'
                 f'{fd!r}\n'
@@ -702,6 +761,26 @@ class _ForkedProc:
         self._pidfd = -1
         return self._returncode
 
+    def terminate(self) -> None:
+        '''
+        OS-level `SIGTERM` to the child. Swallows
+        `ProcessLookupError` (already dead).
+
+        Mirrors `trio.Process.terminate()` /
+        `multiprocessing.Process.terminate()` — sends SIGTERM
+        (graceful, allows the child a chance to clean up via
+        signal-handlers) rather than SIGKILL. Used by
+        `ActorNursery.cancel()`'s per-child escalation when
+        `Portal.cancel_actor()` raises `ActorTooSlowError`,
+        and by the legacy `hard_kill=True` branch on the same
+        path.
+
+        '''
+        try:
+            os.kill(self.pid, signal.SIGTERM)
+        except ProcessLookupError:
+            pass
+
     def kill(self) -> None:
         '''
         OS-level `SIGKILL` to the child. Swallows
@@ -772,13 +851,6 @@ async def main_thread_forkserver_proc(
     thread instead of `trio.lowlevel.open_process()`.
 
     '''
-    if not _has_subints:
-        raise RuntimeError(
-            f'The {"main_thread_forkserver"!r} spawn backend '
-            f'requires Python 3.14+.\n'
-            f'Current runtime: {sys.version}'
-        )
-
     # Backend-scoped config pulled from `proc_kwargs`. Using
     # `proc_kwargs` (vs a first-class kwarg on this function)
     # matches how other backends expose per-spawn tuning
@@ -897,7 +969,18 @@ async def main_thread_forkserver_proc(
                 f' |_{proc}\n'
             )
 
-            event, chan = await ipc_server.wait_for_peer(uid)
+            # race the handshake-wait against proc-death so a
+            # sub that dies during boot (e.g. crashed on import
+            # before reaching `_actor_child_main`, leaving a
+            # zombie + no cmdline) surfaces as `ActorFailure`
+            # instead of parking the spawning task forever on
+            # an unsignalled `_peer_connected[uid]` event.
+            event, chan = await wait_for_peer_or_proc_death(
+                ipc_server,
+                uid,
+                proc_wait=proc.wait,
+                proc_repr=repr(proc),
+            )
 
         except trio.Cancelled:
             cancelled_during_spawn = True

tractor/spawn/_spawn.py

@@ -43,6 +43,7 @@ from tractor.log import get_logger
 from tractor.discovery._addr import (
     UnwrappedAddress,
 )
+from .._exceptions import ActorFailure
 from ._reap import unlink_uds_bind_addrs
 from tractor.runtime._portal import Portal
 from tractor.runtime._runtime import Actor
@@ -82,11 +83,15 @@ SpawnMethodKey = Literal[
     # runtime, exactly like `trio_proc` but via fork instead
     # of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
     'main_thread_forkserver',
-    # RESERVED for the future variant-2 subint-isolated-child
+    # Variant-2: same fork machinery as `main_thread_forkserver`
-    # runtime — gated on jcrist/msgspec#1026 + PEP 684. Today
+    # but the child enters a sub-interpreter to host its
-    # this key aliases to `main_thread_forkserver_proc`; once
+    # `trio.run()`. Gated on jcrist/msgspec#1026 unblocking
-    # the upstream unblocks land it'll dispatch to the
+    # PEP 684 isolated-mode subints upstream — until then
-    # subint-hosted-trio impl. See
+    # `subint_forkserver_proc` is a clean `NotImplementedError`
+    # stub pointing at variant-1 (`main_thread_forkserver`) +
+    # the upstream blocker. The key is reserved here (not just
+    # aliased to variant-1) so once upstream lands the impl can
+    # flip in-place without API churn. See
     # `tractor.spawn._subint_forkserver`.
     'subint_forkserver',
 ]
@@ -106,6 +111,71 @@ else:
         await trio.lowlevel.wait_readable(proc.sentinel)
 
 
+async def wait_for_peer_or_proc_death(
+    ipc_server,
+    uid: tuple[str, str],
+    # TODO? not not types?
+    proc_wait: 'Callable[[], Awaitable]',
+    proc_repr: str = '',
+
+) -> 'tuple[trio.Event, Channel]':
+    '''
+    Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
+    own `.wait()` coroutine. Whichever completes first cancels
+    the other.
+
+    Used by every spawn-backend to detect a sub-actor that
+    *dies during boot* before completing the parent-handshake-
+    callback (e.g. crashed on import, exec'd-out, kernel-killed
+    pre-`_actor_child_main`). Without this race, the
+    handshake-wait — backed by an unsignalled `trio.Event` —
+    parks the spawning task forever and leaves the dead child
+    as a zombie since nobody calls `proc.wait()` to reap.
+
+    On normal handshake-complete: returns `(event, chan)`
+    identical to a bare `wait_for_peer`.
+
+    On proc-death-first: raises `ActorFailure` carrying the
+    proc's exit code, allowing the supervisor to surface a
+    clean error rather than hanging indefinitely.
+
+    `proc_wait` is a 0-arg async callable returning the proc's
+    exit-status — kept generic so each backend can pass its
+    own (`trio.Process.wait`, `_ForkedProc.wait`,
+    `proc_waiter(mp.Process)`, etc.).
+
+    `proc_repr` is an optional string used in the
+    `ActorFailure` message for diag.
+
+    '''
+    result: dict = {}
+
+    async def _await_handshake():
+        event, chan = await ipc_server.wait_for_peer(uid)
+        result['handshake'] = (event, chan)
+        boot_n.cancel_scope.cancel()
+
+    async def _await_death():
+        rc = await proc_wait()
+        result['died'] = rc
+        boot_n.cancel_scope.cancel()
+
+    async with trio.open_nursery() as boot_n:
+        boot_n.start_soon(_await_handshake)
+        boot_n.start_soon(_await_death)
+
+    if 'handshake' in result:
+        return result['handshake']
+
+    # only reached if proc-death won the race
+    raise ActorFailure(
+        f'Sub-actor {uid!r} died during boot '
+        f'(rc={result.get("died")!r}) before completing '
+        f'parent-handshake.\n'
+        f'  proc: {proc_repr}'
+    )
+
+
 def try_set_start_method(
     key: SpawnMethodKey
 
@@ -138,13 +208,15 @@ def try_set_start_method(
         case 'mp_spawn':
             _ctx = mp.get_context('spawn')
 
-        case 'trio':
+        case (
+            'trio'
+            | 'main_thread_forkserver'
+        ):
             _ctx = None
 
         case (
             'subint'
             | 'subint_fork'
-            | 'main_thread_forkserver'
             | 'subint_forkserver'
         ):
             # All subint-family backends need no `mp.context`;

tractor/spawn/_subint_forkserver.py

@@ -18,15 +18,18 @@
 Variant-2 (future) "subint forkserver" placeholder — reserved
 for the eventual subint-isolated-child runtime variant.
 
-> **Status:** placeholder. Today
+> **Status:** reserved key, stub impl. Today
-> `--spawn-backend=subint_forkserver` aliases to
+> `--spawn-backend=subint_forkserver` raises a clean
-> `main_thread_forkserver_proc` (variant 1, see
+> `NotImplementedError` from `subint_forkserver_proc()`
-> `tractor.spawn._main_thread_forkserver`). A follow-up commit
+> below, pointing at variant-1
-> in this PR series flips the alias to a `NotImplementedError`
+> (`--spawn-backend=main_thread_forkserver`, see
-> stub reserving the `'subint_forkserver'` key for the literal
+> `tractor.spawn._main_thread_forkserver`) and the upstream
-> subint-hosted-child variant once
+> blocker
-> [jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)
+> ([jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)).
-> unblocks PEP 684 isolated-mode subints upstream.
+> The key is reserved here (not aliased to variant-1) so the
+> literal subint-hosted-child impl can flip in-place once
+> msgspec#1026 unblocks PEP 684 isolated-mode subints
+> upstream — no API churn at the call site.
 
 Future arch — what subints would buy us
 ---------------------------------------
@@ -154,21 +157,6 @@ from trio import TaskStatus
 from tractor.log import get_logger
 from ._subint import _has_subints
 
-# Backward-compat re-exports of the fork primitives whose
-# canonical home is now `_main_thread_forkserver`. Kept here
-# transiently so existing
-# `from tractor.spawn._subint_forkserver import ...` callsites
-# in the tests + the conc-anal smoketest keep resolving;
-# dropped once a follow-up commit migrates those imports to
-# the new module.
-from ._main_thread_forkserver import (
-    _close_inherited_fds as _close_inherited_fds,
-    _format_child_exit as _format_child_exit,
-    fork_from_worker_thread as fork_from_worker_thread,
-    wait_child as wait_child,
-    _ForkedProc as _ForkedProc,
-)
-
 if TYPE_CHECKING:
     from tractor.discovery._addr import UnwrappedAddress
     from tractor.runtime._portal import Portal