Compare commits

..

No commits in common. "fbaf933ccbcbdbfe001a6c031a3c595c9b453a40" and "3818fe863853d85345d8c3a302d24e3d7641f6e1" have entirely different histories.

7 changed files with 94 additions and 644 deletions

View File

@ -88,27 +88,10 @@ jobs:
testing:
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
timeout-minutes: 20
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
timeout-minutes: 16
runs-on: ${{ matrix.os }}
# NOTE on the matrix shape — the `capture=` mode follows
# `spawn_backend`:
#
# - `trio` / `mp_*` backends use `--capture=fd` (default)
# for per-test attribution of subactor *raw-fd* output
# in failure reports.
# - Fork-based backends (`main_thread_forkserver`,
# `subint_forkserver`) REQUIRE `--capture=sys` because
# fork-child × `--capture=fd` is a known deadlock
# pattern. See the long NOTE in `tractor._testing.pytest`'s
# `pytest_load_initial_conftests` for the mechanism +
# tradeoff write-up.
#
# If a future matrix row adds a fork-spawn backend
# WITHOUT setting `capture: 'sys'`, the
# `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
# with a clear error msg. So the matrix is self-policing.
strategy:
fail-fast: false
matrix:
@ -135,26 +118,6 @@ jobs:
'tcp',
'uds',
]
capture: [
'fd', # default for non-fork backends
]
# Fork-based backends — added via `include:` so each
# cell carries its REQUIRED `capture: 'sys'` mode.
# Linux-only for now; macOS coverage TBD pending
# local validation.
include:
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'tcp'
capture: 'sys'
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'uds'
capture: 'sys'
# https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
exclude:
# don't do UDS run on macOS (for now)
@ -195,11 +158,7 @@ jobs:
-rsx
--spawn-backend=${{ matrix.spawn_backend }}
--tpt-proto=${{ matrix.tpt_proto }}
--capture=${{ matrix.capture }}
# NOTE: capture mode is matrix-driven — `fd` for
# non-fork backends (per-test fd attribution),
# `sys` for fork-based (avoids fork-child x
# capture-fd deadlock). See matrix-NOTE above.
--capture=fd
# XXX legacy NOTE XXX
#

View File

@ -1,314 +0,0 @@
# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
## Symptom
Running the full test suite under
```bash
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver
```
surfaces a single, **rotating** `trio.TooSlowError`
failure each run. The failure isn't deterministic on
test identity — different test each run — but it
ALWAYS looks like:
```
FAILED tests/<file>::test_<name> - trio.TooSlowError
==== 1 failed, 373 passed, 17 skipped, 11–12 xfailed,
0–1 xpassed, ~550 warnings in ~6min ====
```
Pass rate: **~99.7%** (373 of 374 non-skip tests).
Wall-clock per full run: 5–6 min.
## Tests observed flaking so far
Each row was the SOLE failure in a separate run:
| run # | test |
|---|---|
| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
Both share the same shape:
- **Cancel cascade** of N subactors back to a parent root actor.
- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
(it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
- N ≈ 2 for `test_context_spawns_aio_task_that_errors` —
but each subactor is `infect_asyncio=True`, so each
cancel involves the trio↔asyncio guest-run unwind
which is structurally heavier than pure-trio.
- Test wraps the cascade in `trio.fail_after(N seconds)`
and the cap fires before the cascade completes.
The exact failing test rotates because each test is
independently close to the cap; whichever happens to
be unlucky in scheduling/CPU-contention on a given run
is the one that times out.
## Root-cause family
`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
the SC-graceful teardown ladder per subactor:
1. `Portal.cancel_actor()` — graceful IPC cancel-req.
2. Wait `terminate_after=1.6s` for sub to exit.
3. If still alive: `proc.kill()` (SIGKILL).
4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
sock-file cleanup for UDS leaks (issue #452 fix).
For a cascade of N subactors, each pays steps 1–4. If
graceful-cancel doesn't complete within 1.6s for ANY
sub, that sub eats a full 1.6s of `move_on_after` plus
the `proc.wait()` post-SIGKILL.
Worst case under fork backend with N=cpus subs:
- N × 1.6s = 16s+ on a 10-core box just for the
graceful timeout phase
- Plus per-spawn fork-IPC handshake cost compounds
during teardown (each sub's IPC cleanup goes through
the same forkserver coordinator)
- Plus the new autouse fixtures
(`_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors`) all run at test
teardown, adding small (10s of ms) but cumulative
overhead
Current cap: 30s (`fail_after_s = 30 if
is_forking_spawner else 12`). Empirically fits the
median run but the tail breaks ~0.3% of the time.
## NOT regressing
To confirm this is a flake and not a regression:
- Pre-`WakeupSocketpair`-patch baseline: tests
HUNG INDEFINITELY (busy-loop never released).
- Post-patch: pass-or-fail-fast, ~99.7% pass, the
occasional cap-hit fails in bounded time (<60s for
the offending test).
- Same test PASSES under `--spawn-backend=trio`
(no fork, no hard-kill compounding).
So the suite is dramatically better than before; the
remaining flake is a known-tolerable steady-state.
## Possible mitigations (ranked)
### A. Bump the cap further
Cheapest. Change the per-test `fail_after_s` from 30
to e.g. 60 for fork backends. Pros: trivial. Cons:
masks any genuine slowness regression we'd want to
catch.
### B. CPU-count-aware cap
For tests whose N scales with `cpu_count()`, scale
the cap too:
```python
fail_after_s = (
max(30, cpu_count() * 3) # 3s/actor floor
if is_forking_spawner
else 12
)
```
Pros: scales with the actual cancel-cascade work.
Cons: still arbitrary multiplier.
### C. `pytest-rerunfailures` for these tests only
Mark the known-flaky tests with
`@pytest.mark.flaky(reruns=1)` (needs
`pytest-rerunfailures` dep). Single retry hides
genuine ~0.3% transient flakes.
Pros: no cap change, surfaces persistent failures
loudly. Cons: adds a dep, retries can mask real bugs
if used widely.
### D. Reduce `hard_kill`'s `terminate_after`
Drop from 1.6s → 0.8s. Cuts the worst-case cascade
time roughly in half. Risks: fewer subs get a chance
to run their cleanup before SIGKILL → more orphaned
state for the autouse reapers to handle (ironically,
adds back overhead elsewhere).
### E. Profile + targeted fix
Add `log.devx()` markers in `hard_kill` to time each
phase. Identify if any subactor is consistently
hitting the 1.6s cap (vs. exiting in <0.1s). If so,
that sub has a teardown bug worth fixing at source.
Pros: actually fixes the underlying slowness. Cons:
real investigation work, deferred from this round.
## Recommendation
Land this issue-doc as the tracker. Apply **(B)** as
a small follow-up — cheap and proportional. If it
still flakes, escalate to **(E)** with a `log.devx()`
profile-pass.
`(C)` is a backstop if `(B)` doesn't quite get there
and we need green CI faster than (E) can deliver.
## Verification protocol
After applying any mitigation:
```bash
# Run the suite N times back-to-back, count failures.
# A persistent failure on the SAME test == real bug.
# Failures rotating across tests == still cap-related.
for i in $(seq 1 5); do
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver \
-q 2>&1 | tail -2
done
```
Target: 0 failures across 5 runs ⇒ ship. 1–2 failures
still rotating ⇒ apply (C). Same test failing twice
⇒ escalate to (E).
## Snapshot evidence (2026-05-13)
After landing the `fail_after_w_trace` /
`afk_alarm_w_trace` capture-on-timeout helpers
(`tractor._testing.trace`), `test_nested_multierrors`
on the `main_thread_forkserver` backend produces
**reproducible diag snapshots** at
`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
### Reproduction
```bash
pytest \
-v --verbose --durations=10 \
--spawn-backend=main_thread_forkserver \
--tpt-proto=uds \
--capture=sys --show-capture=stderr -rxX \
tests/test_cancellation.py::test_nested_multierrors
```
The test is `xfail(strict=False)` for MTF — it RUNS
each invocation so snapshots accumulate, but doesn't
break `--lf` workflow.
### Consistent shape across runs
5+ snapshots taken back-to-back show the SAME pattern:
- **Timing:** ~10s wall-clock total. Inner
`fail_after_w_trace(10)` fires at exactly T=10s;
cascade's `nursery.__aexit__` takes ~0.6s more to
gather + propagate the resulting
`BaseExceptionGroup`. **Trio backend completes the
SAME test in <6s** — so the MTF cascade is ~2x
slower at minimum.
- **`BaseExceptionGroup` shape:** mixed
`[RemoteActorError, Cancelled]`. The first
subactor's natural error-propagation (`assert 0`
raised → `RemoteActorError` portal-result)
arrives before T=10s; the OTHER subactor's
portal-wait is still in flight at T=10s, gets
cancelled by `fail_after_w_trace`'s scope-cancel
→ returns `Cancelled` instead.
- **Orphan-spawn skew:** snapshot's `orphans` bucket
(after the `_is_tractor_subactor` cgroup-slice
override fix) consistently shows 2-4 init-adopted
procs at `depth_3` and `depth_1` levels — these
are the leaves whose parent (`depth_2` spawner)
was killed mid-cascade but who hadn't yet seen
the cancel signal themselves.
- **UDS sock-leak:** 2-6 dead-orphan socks per run
(varies with cascade timing). The
`track_orphaned_uds_per_test` fixture reaps them
post-test → contamination is isolated per-invocation.
### Capture mechanism
`fail_after_w_trace` covers two firing paths:
1. **`trio.TooSlowError`** raised at scope-exit
(body returned cleanly past deadline) — direct
`except` handler captures.
2. **Scope-cancel + body raises non-`Cancelled` exc**
(e.g. `nursery.__aexit__` wraps timeout-induced
`Cancelled` into a `BaseExceptionGroup` that
escapes before `trio.fail_after`'s exit-check
could fire `TooSlowError`) — body-raise `except`
handler checks `scope.cancel_called` and
captures if True. This path catches the
`test_nested_multierrors` shape specifically (see
"BaseExceptionGroup shape" above).
The snapshot dir contains:
- `trace.txt` — `ptree` + `hung_state` (kernel
`wchan`/`stack` + `py-spy dump --locals` when
sudo cached), with `include_strays=True`
surfacing any cross-test ghost subactor trees in
the `orphans` bucket.
- `bindspace.txt` — UDS bindspace classification
(live-active / orphaned-alive / orphaned-dead).
- `meta.json` — `{pid, label, captured_at, sudo_cached}`.
The end-of-session `pytest_terminal_summary` hook
in `tractor._testing.pytest` lists every snapshot
dir from the run so you don't have to scroll back
through captured-stderr lines:
```
========================= tractor hang-snapshot index ==========================
N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
<test-id>
→ /home/.../.cache/tractor/hung-dumps/<label>__<ts>
```
### Caveats
The snapshot fires AFTER the body-raise (not at the
exact moment of scope-cancel), so the parent's
py-spy frames show `_do_capture_snapshot` itself
running, NOT the cancel-cascade hang frame. To see
the actual hang state, manual `acli.ptree` /
`acli.hung_dump` from a second terminal at T=10s
would be needed — **not currently possible**
because per-test reaper fixtures clean up ~0.6s
post-timeout. See follow-up TODO in
`tractor/_testing/trace.py` for a
`TRACTOR_TRACE_HOLD=1` env-var pause mode.
## See also
- [#452](https://github.com/goodboy/tractor/issues/452) —
UDS sock-file leak (related — `hard_kill`'s
cleanup phase contributes to cascade time)
- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
— the upstream-trio fix that turned this from a
100% hang into a 0.3% flake
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
— the asyncio variant which contributes to one of
the rotating failures
- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
cascade source
- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors` — autouse cleanup
fixtures whose cumulative teardown overhead
contributes to the cascade time

View File

@ -89,12 +89,10 @@ except ImportError:
# the "zero tractor imports" isolation guarantee; now that
# CPython-level feasibility is confirmed, the validated
# primitives have moved into tractor proper.)
from tractor.spawn._main_thread_forkserver import (
fork_from_worker_thread,
wait_child,
)
from tractor.spawn._subint_forkserver import (
fork_from_worker_thread,
run_subint_in_worker_thread,
wait_child,
)

View File

@ -1,14 +1,13 @@
'''
Integration exercises for the `tractor.spawn._main_thread_forkserver`
Integration exercises for the `tractor.spawn._subint_forkserver`
submodule at three tiers:
1. the low-level primitives
(`fork_from_worker_thread()` from `_main_thread_forkserver`
+ `run_subint_in_worker_thread()` from
`_subint_forkserver`) driven from inside a real
(`fork_from_worker_thread()` +
`run_subint_in_worker_thread()`) driven from inside a real
`trio.run()` in the parent process,
2. the full `main_thread_forkserver_proc` spawn backend wired
2. the full `subint_forkserver_proc` spawn backend wired
through tractor's normal actor-nursery + portal-RPC
machinery — i.e. `open_root_actor` + `open_nursery` +
`run_in_actor` against a subactor spawned via fork from a
@ -28,15 +27,15 @@ Those smoke-test scenarios are standalone — no trio runtime
in the *parent*. Tiers (1)+(2) here cover the primitives
driven from inside `trio.run()` in the parent, and tier (3)
(the `*_spawn_basic` test) drives the registered
`main_thread_forkserver` spawn backend end-to-end against
the tractor runtime.
`subint_forkserver` spawn backend end-to-end against the
tractor runtime.
Gating
------
- py3.14+ (via `concurrent.interpreters` presence)
- no `--spawn-backend` restriction — the backend-level test
flips `tractor.spawn._spawn._spawn_method` programmatically
(via `try_set_start_method('main_thread_forkserver')`) and
(via `try_set_start_method('subint_forkserver')`) and
restores it on teardown, so these tests are independent of
the session-level CLI backend choice.
@ -65,12 +64,10 @@ from tractor.devx import dump_on_hang
# `tractor.spawn._subint` for why.
pytest.importorskip('concurrent.interpreters')
from tractor.spawn._main_thread_forkserver import ( # noqa: E402
fork_from_worker_thread,
wait_child,
)
from tractor.spawn._subint_forkserver import ( # noqa: E402
fork_from_worker_thread,
run_subint_in_worker_thread,
wait_child,
)
from tractor.spawn import _spawn as _spawn_mod # noqa: E402
from tractor.spawn._spawn import try_set_start_method # noqa: E402
@ -198,7 +195,7 @@ def test_fork_from_worker_thread_via_trio(
deadline: float = 10.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_baseline.dump',
path='/tmp/subint_forkserver_baseline.dump',
):
pid: int = trio.run(
partial(run_fork_in_non_trio_thread, deadline),
@ -220,14 +217,14 @@ def test_fork_and_run_trio_in_child() -> None:
`trio.run()` inside it on yet another worker thread.
This is the full "forkserver + trio-in-subint-in-child"
pattern the proposed `main_thread_forkserver` spawn backend
pattern the proposed `subint_forkserver` spawn backend
would rest on.
'''
deadline: float = 15.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_trio_in_child.dump',
path='/tmp/subint_forkserver_trio_in_child.dump',
):
pid: int = trio.run(
partial(
@ -240,7 +237,7 @@ def test_fork_and_run_trio_in_child() -> None:
# ----------------------------------------------------------------
# tier-3 backend test: drive the registered `main_thread_forkserver`
# tier-3 backend test: drive the registered `subint_forkserver`
# spawn backend end-to-end through tractor's actor-nursery +
# portal-RPC machinery.
# ----------------------------------------------------------------
@ -263,7 +260,7 @@ async def _happy_path_forkserver(
Parent-side harness: stand up a root actor, open an actor
nursery, spawn one subactor via the currently-selected
spawn backend (which this test will have flipped to
`main_thread_forkserver`), run a trivial RPC through its
`subint_forkserver`), run a trivial RPC through its
portal, assert the round-trip result.
'''
@ -307,19 +304,19 @@ def forkserver_spawn_method():
@pytest.mark.timeout(60, method='thread')
def test_main_thread_forkserver_spawn_basic(
def test_subint_forkserver_spawn_basic(
reg_addr: tuple[str, int | str],
forkserver_spawn_method,
) -> None:
'''
Happy-path: spawn ONE subactor via the
`main_thread_forkserver` backend (parent-side fork from a
`subint_forkserver` backend (parent-side fork from a
main-interp worker thread), do a trivial portal-RPC
round-trip, tear the nursery down cleanly.
If this passes, the "forkserver + tractor runtime" arch
is proven end-to-end: the registered
`main_thread_forkserver_proc` spawn target successfully
`subint_forkserver_proc` spawn target successfully
forks a child, the child runs `_actor_child_main()` +
completes IPC handshake + serves an RPC, and the parent
reaps via `_ForkedProc.wait()` without regressing any of
@ -329,7 +326,7 @@ def test_main_thread_forkserver_spawn_basic(
deadline: float = 20.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_spawn_basic.dump',
path='/tmp/subint_forkserver_spawn_basic.dump',
):
trio.run(
partial(
@ -343,7 +340,7 @@ def test_main_thread_forkserver_spawn_basic(
# ----------------------------------------------------------------
# tier-4 DRAFT: orphaned-subactor SIGINT survivability
#
# Motivating question: with `main_thread_forkserver`, the child's
# Motivating question: with `subint_forkserver`, the child's
# `trio.run()` lives on the fork-inherited worker thread which
# is NOT `threading.main_thread()` — so trio cannot install its
# `signal.set_wakeup_fd`-based SIGINT handler. If the parent
@ -363,7 +360,7 @@ def test_main_thread_forkserver_spawn_basic(
# Cross-backend generalization (decide after this passes):
# - applicable to any backend whose subactors are separate OS
# processes: `trio`, `mp_spawn`, `mp_forkserver`,
# `main_thread_forkserver`.
# `subint_forkserver`.
# - NOT applicable to plain `subint` (subactors are in-process
# subinterpreters, no orphan child process to SIGINT).
# - move path: lift the harness script into
@ -449,7 +446,7 @@ def _process_alive(pid: int) -> bool:
return False
# Known-gap test — `main_thread_forkserver` orphan-SIGINT
# Known-gap test — `subint_forkserver` orphan-SIGINT
# handling. See
# `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
# `strict=True` so if a future fix closes the gap the
@ -474,12 +471,12 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
) -> None:
'''
DRAFT — orphaned-subactor SIGINT survivability under the
`main_thread_forkserver` backend.
`subint_forkserver` backend.
Sequence:
1. Spawn a harness subprocess that brings up a root
actor + one `sleep_forever` subactor via
`main_thread_forkserver`.
`subint_forkserver`.
2. Read the harness's stdout for `PARENT_READY=<pid>`
and `CHILD_PID=<pid>` markers (confirms the
parent→child IPC handshake completed).
@ -527,7 +524,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
[
sys.executable,
str(script_path),
'main_thread_forkserver',
'subint_forkserver',
host,
str(port),
],
@ -580,7 +577,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
pytest.fail(
f'Orphan subactor (pid={child_pid}) did NOT exit '
f'within 10s of SIGINT under `main_thread_forkserver` '
f'within 10s of SIGINT under `subint_forkserver` '
f'→ trio on non-main thread did not observe the '
f'default CPython KeyboardInterrupt; backend needs '
f'explicit SIGINT plumbing.'
@ -603,50 +600,3 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
pass
# ----------------------------------------------------------------
# regression guard: variant-2 (`subint_forkserver`) placeholder
# MUST raise `NotImplementedError` today — guards against future
# commits accidentally re-aliasing the key to the variant-1
# coroutine (which was a transient state during the rename).
# ----------------------------------------------------------------
def test_subint_forkserver_key_errors_cleanly() -> None:
'''
`--spawn-backend=subint_forkserver` is reserved for the
eventual variant-2 (subint-isolated child runtime)
backend, gated on jcrist/msgspec#1026 unblocking PEP 684
isolated-mode subints upstream.
Until that lands, the dispatch entry MUST raise
`NotImplementedError` immediately rather than silently
aliasing to `main_thread_forkserver_proc`. Verify the
error message also surfaces both the working-backend
pointer and the upstream-blocker ref so an operator
arriving at the error has somewhere to go.
'''
import asyncio
from tractor.spawn._spawn import _methods
proc = _methods['subint_forkserver']
with pytest.raises(NotImplementedError) as ei:
# signature args match `main_thread_forkserver_proc`'s
# — the stub raises before touching them so dummy
# values are fine.
asyncio.run(
proc(
'x', None, None, {}, [],
('127.0.0.1', 0), {},
)
)
msg: str = str(ei.value)
assert 'main_thread_forkserver' in msg, (
f'stub error msg should redirect to the working '
f'variant-1 backend; got: {msg!r}'
)
assert 'msgspec#1026' in msg or '1026' in msg, (
f'stub error msg should reference the upstream '
f'blocker (jcrist/msgspec#1026); got: {msg!r}'
)

View File

@ -38,7 +38,6 @@ Two empirical CPython properties drive the design:
the forked child otherwise (`Fatal Python error: not main
interpreter`). Full source-level walkthrough:
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
2. **`os.fork()` from a regular `threading.Thread` attached to
the *main* interpreter — i.e. a worker thread that has never
entered a subint — works cleanly.** Empirically validated
@ -87,11 +86,9 @@ costs:
- **Sidecar lifecycle**: a second long-lived process per
parent, with its own start/stop/health-check semantics.
- **IPC overhead per spawn**: every actor-spawn round-trips
an `mp` request message through a unix socket before any
child code runs.
- **State isolation by process boundary**: the sidecar can't
share parent state at all — every spawn is a "cold" child
re-importing modules from disk.
@ -109,7 +106,6 @@ For the full variant-2 picture see
1) we already get costs 1 + 2 collapsed; cost 3 will land
when msgspec#1026 unblocks isolated-mode subints.
What survives the fork? — POSIX semantics
-----------------------------------------
@ -117,58 +113,33 @@ A natural worry when forking from a parent that's running
`trio.run()` on another thread: does that trio thread (and
any other threads in the parent) keep running in the child?
**No** — but with a precise meaning that's worth pinning
down, since the canonical trio framing
([python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
puts it the opposite-sounding way:
> If you use `fork()` in a process with multiple threads,
> all the other thread stacks are just leaked: there's
> nothing else you can reasonably do with them.
Both statements describe the same POSIX reality from
opposite sides:
- **Execution-side ("gone")**: POSIX `fork()` only
preserves the *calling* thread as a runnable thread in
the child. Every other thread in the parent — trio's
runner thread, any `to_thread` cache threads, anything
else — never executes another instruction post-fork.
- **Memory-side ("leaked")**: those non-running threads'
*stacks* and per-thread heap structures are still
COW-inherited into the child's address space. They
persist as orphaned bytes with no owning thread, no
scheduler entry, and no way for the child to clean
them up — hence trio's word "leaked".
**No.** POSIX `fork()` only preserves the *calling* thread
in the child. Every other thread in the parent — trio's
runner thread, any `to_thread` cache threads, anything else
— is gone the instant `fork()` returns in the child.
Concretely, after the forkserver worker calls `os.fork()`:
| thread | parent | child (executing) | child (memory) |
|---------------------|-----------|-------------------|-----------------------------|
| forkserver worker | continues | sole survivor | live stack |
| `trio.run()` thread | continues | not running | leaked stack (zombie bytes) |
| any other thread | continues | not running | leaked stack (zombie bytes) |
| thread | parent | child |
|-----------------------|-----------|---------------|
| forkserver worker | continues | sole survivor |
| `trio.run()` thread | continues | gone |
| any other thread | continues | gone |
The forkserver worker becomes the new "main" execution
context in the child; `trio.run()` and every other parent
thread never executes a single instruction post-fork.
Their stack memory rides along as inert COW pages until
the child's fresh `trio.run()` boots and overwrites/GCs
it (or until the child `exec()`s and discards the entire
image).
thread never executes a single instruction post-fork in the
child.
This is exactly *why* `os.fork()` is delegated to a
dedicated worker thread that has provably never entered
trio: we want that trio-free thread to be the surviving
*executing* thread in the child, with the leaked trio
stack reduced to inert COW pages we don't touch.
one in the child.
The leaked-stack residue is one slice of the broader
"fork in a multithreaded program is dangerous" hazard
class (see `man pthread_atfork`). Other dead-thread
artifacts that cross the fork boundary, and how we handle
each:
That said, dead-thread *artifacts* still cross the fork
boundary (canonical "fork in a multithreaded program is
dangerous" — see `man pthread_atfork`). What persists, and
how we handle each:
- **Inherited file descriptors** — the dead trio thread's
epoll fd, signal-wakeup-fd, eventfds, sockets, IPC
@ -177,20 +148,16 @@ each:
`_close_inherited_fds()` in the child prelude — walks
`/proc/self/fd` and closes everything except stdio +
the channel pipe to the forkserver.
- **Memory image** — trio's internal data structures
(scheduler, task queues, runner state) sit in COW
memory alongside the leaked stacks above. Nobody's
executing them; they get GC'd / overwritten when the
child's fresh `trio.run()` boots.
memory but nobody's executing them. Get GC'd /
overwritten when the child's fresh `trio.run()` boots.
- **Python thread state** — handled automatically by
CPython. `PyOS_AfterFork_Child()` calls
`_PyThreadState_DeleteExceptCurrent()`, so dead
`PyThreadState` objects are cleaned and
`threading.enumerate()` returns just the surviving
thread.
- **User-level locks (`threading.Lock`)** —
held-by-dead-thread state is the canonical fork hazard.
Not an issue in practice for tractor: trio doesn't hold
@ -199,7 +166,6 @@ each:
either direction). CPython's GIL is auto-reset by the
fork callback.
FYI: how this dodges the `trio.run()` × `fork()` hazards
--------------------------------------------------------
@ -217,16 +183,13 @@ design dodges each class explicitly:
reader. *Dodge*: the inherited wakeup-fd is closed by
`_close_inherited_fds()`, then the child's own
`trio.run()` installs a fresh one.
- **`epoll`/`kqueue` instance**: trio's I/O backend holds
one. Inherited as a dead fd; same fix as above.
- **Threadpool cache threads** (`trio.to_thread`): worker
threads with cached tstate. Don't exist in the child
(POSIX); cache state is meaningless garbage that gets
reset when the child's trio.run() initializes its own
thread cache.
- **Cancel scopes / nurseries / open `trio.Process` /
open sockets**: these are trio-runtime objects, not
kernel objects. The runtime that owns them is gone in
@ -234,11 +197,9 @@ design dodges each class explicitly:
in COW memory and get overwritten as the child runs.
Inherited *kernel* fds those objects wrapped (sockets,
proc pipes) are caught by `_close_inherited_fds()`.
- **`atexit` handlers**: trio doesn't register any that
would mis-fire post-fork; trio's lifetime-stack is
all `with`-block-scoped and dies with the runner.
- **Foreign-language I/O state** (libcurl, OpenSSL session
caches, etc.): out of scope — same hazard as any
fork-without-exec; users layering those on top of
@ -250,7 +211,6 @@ isolation + `_close_inherited_fds()` cleanup gives the
forked child a clean trio environment. Everything else
falls under the standard fork-without-exec disclaimer.
Implementation status
---------------------
@ -271,11 +231,10 @@ follow-up) including the
Still-open work (tracked on tractor #379):
- [ ] no cancellation / hard-kill stress coverage yet
- no cancellation / hard-kill stress coverage yet
(counterpart to `tests/test_subint_cancellation.py` for
the plain `subint` backend),
- [ ] `child_sigint='trio'` mode (flag scaffolded below; default
- `child_sigint='trio'` mode (flag scaffolded below; default
is `'ipc'`). Originally intended as a manual SIGINT →
trio-cancel bridge, but investigation showed trio's
handler IS already correctly installed in the fork-child
@ -328,24 +287,21 @@ See also
- `tractor.spawn._subint_forkserver` — variant-2 placeholder
module; reserved for the future subint-isolated-child
runtime once jcrist/msgspec#1026 unblocks.
- `tractor.spawn._subint_fork` — the stub for the
fork-from-non-main-subint strategy that DIDN'T work (kept
in-tree as documentation of the attempt + the CPython-level
block).
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
— CPython source walkthrough of why fork-from-subint is dead.
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
— standalone feasibility check (delegates to this module
for the primitives it exercises).
'''
from __future__ import annotations
import errno
import os
import signal
import sys
import threading
from functools import partial
from typing import (
@ -368,8 +324,8 @@ from tractor.runtime._portal import Portal
from ._spawn import (
cancel_on_completion,
soft_kill,
wait_for_peer_or_proc_death,
)
from ._subint import _has_subints
if TYPE_CHECKING:
from tractor.discovery._addr import UnwrappedAddress
@ -467,24 +423,9 @@ def _close_inherited_fds(
try:
os.close(fd)
closed += 1
except OSError as oserr:
# `EBADF` is the benign-and-expected case: the
# `os.listdir('/proc/self/fd')` call above itself
# opens a transient dirfd that ends up in
# `candidates`, then auto-closes before this loop
# reaches it. Same for any fd whose Python wrapper
# was GC'd between `listdir` and `os.close`.
# Suppress at debug-level — surfacing every
# EBADF as a full traceback (prior `log.exception`
# behavior) drowned the post-fork log channel.
if oserr.errno == errno.EBADF:
log.debug(
f'Skip already-closed inherited fd {fd!r} '
f'(EBADF, benign race with listdir)\n'
)
continue
# Other errnos (EIO / EPERM / EINTR / ...) are
# genuinely unexpected — keep the loud surface.
except OSError:
# fd was already closed (race with listdir) or otherwise
# unclosable — either is fine.
log.exception(
f'Failed to close inherited fd in child ??\n'
f'{fd!r}\n'
@ -761,26 +702,6 @@ class _ForkedProc:
self._pidfd = -1
return self._returncode
def terminate(self) -> None:
'''
OS-level `SIGTERM` to the child. Swallows
`ProcessLookupError` (already dead).
Mirrors `trio.Process.terminate()` /
`multiprocessing.Process.terminate()` — sends SIGTERM
(graceful, allows the child a chance to clean up via
signal-handlers) rather than SIGKILL. Used by
`ActorNursery.cancel()`'s per-child escalation when
`Portal.cancel_actor()` raises `ActorTooSlowError`,
and by the legacy `hard_kill=True` branch on the same
path.
'''
try:
os.kill(self.pid, signal.SIGTERM)
except ProcessLookupError:
pass
def kill(self) -> None:
'''
OS-level `SIGKILL` to the child. Swallows
@ -851,6 +772,13 @@ async def main_thread_forkserver_proc(
thread instead of `trio.lowlevel.open_process()`.
'''
if not _has_subints:
raise RuntimeError(
f'The {"main_thread_forkserver"!r} spawn backend '
f'requires Python 3.14+.\n'
f'Current runtime: {sys.version}'
)
# Backend-scoped config pulled from `proc_kwargs`. Using
# `proc_kwargs` (vs a first-class kwarg on this function)
# matches how other backends expose per-spawn tuning
@ -969,18 +897,7 @@ async def main_thread_forkserver_proc(
f' |_{proc}\n'
)
# race the handshake-wait against proc-death so a
# sub that dies during boot (e.g. crashed on import
# before reaching `_actor_child_main`, leaving a
# zombie + no cmdline) surfaces as `ActorFailure`
# instead of parking the spawning task forever on
# an unsignalled `_peer_connected[uid]` event.
event, chan = await wait_for_peer_or_proc_death(
ipc_server,
uid,
proc_wait=proc.wait,
proc_repr=repr(proc),
)
event, chan = await ipc_server.wait_for_peer(uid)
except trio.Cancelled:
cancelled_during_spawn = True

View File

@ -43,7 +43,6 @@ from tractor.log import get_logger
from tractor.discovery._addr import (
UnwrappedAddress,
)
from .._exceptions import ActorFailure
from ._reap import unlink_uds_bind_addrs
from tractor.runtime._portal import Portal
from tractor.runtime._runtime import Actor
@ -83,15 +82,11 @@ SpawnMethodKey = Literal[
# runtime, exactly like `trio_proc` but via fork instead
# of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
'main_thread_forkserver',
# Variant-2: same fork machinery as `main_thread_forkserver`
# but the child enters a sub-interpreter to host its
# `trio.run()`. Gated on jcrist/msgspec#1026 unblocking
# PEP 684 isolated-mode subints upstream — until then
# `subint_forkserver_proc` is a clean `NotImplementedError`
# stub pointing at variant-1 (`main_thread_forkserver`) +
# the upstream blocker. The key is reserved here (not just
# aliased to variant-1) so once upstream lands the impl can
# flip in-place without API churn. See
# RESERVED for the future variant-2 subint-isolated-child
# runtime — gated on jcrist/msgspec#1026 + PEP 684. Today
# this key aliases to `main_thread_forkserver_proc`; once
# the upstream unblocks land it'll dispatch to the
# subint-hosted-trio impl. See
# `tractor.spawn._subint_forkserver`.
'subint_forkserver',
]
@ -111,71 +106,6 @@ else:
await trio.lowlevel.wait_readable(proc.sentinel)
async def wait_for_peer_or_proc_death(
ipc_server,
uid: tuple[str, str],
# TODO? not not types?
proc_wait: 'Callable[[], Awaitable]',
proc_repr: str = '',
) -> 'tuple[trio.Event, Channel]':
'''
Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
own `.wait()` coroutine. Whichever completes first cancels
the other.
Used by every spawn-backend to detect a sub-actor that
*dies during boot* before completing the parent-handshake-
callback (e.g. crashed on import, exec'd-out, kernel-killed
pre-`_actor_child_main`). Without this race, the
handshake-wait — backed by an unsignalled `trio.Event` —
parks the spawning task forever and leaves the dead child
as a zombie since nobody calls `proc.wait()` to reap.
On normal handshake-complete: returns `(event, chan)`
identical to a bare `wait_for_peer`.
On proc-death-first: raises `ActorFailure` carrying the
proc's exit code, allowing the supervisor to surface a
clean error rather than hanging indefinitely.
`proc_wait` is a 0-arg async callable returning the proc's
exit-status — kept generic so each backend can pass its
own (`trio.Process.wait`, `_ForkedProc.wait`,
`proc_waiter(mp.Process)`, etc.).
`proc_repr` is an optional string used in the
`ActorFailure` message for diag.
'''
result: dict = {}
async def _await_handshake():
event, chan = await ipc_server.wait_for_peer(uid)
result['handshake'] = (event, chan)
boot_n.cancel_scope.cancel()
async def _await_death():
rc = await proc_wait()
result['died'] = rc
boot_n.cancel_scope.cancel()
async with trio.open_nursery() as boot_n:
boot_n.start_soon(_await_handshake)
boot_n.start_soon(_await_death)
if 'handshake' in result:
return result['handshake']
# only reached if proc-death won the race
raise ActorFailure(
f'Sub-actor {uid!r} died during boot '
f'(rc={result.get("died")!r}) before completing '
f'parent-handshake.\n'
f' proc: {proc_repr}'
)
def try_set_start_method(
key: SpawnMethodKey
@ -208,15 +138,13 @@ def try_set_start_method(
case 'mp_spawn':
_ctx = mp.get_context('spawn')
case (
'trio'
| 'main_thread_forkserver'
):
case 'trio':
_ctx = None
case (
'subint'
| 'subint_fork'
| 'main_thread_forkserver'
| 'subint_forkserver'
):
# All subint-family backends need no `mp.context`;

View File

@ -18,18 +18,15 @@
Variant-2 (future) "subint forkserver" placeholder — reserved
for the eventual subint-isolated-child runtime variant.
> **Status:** reserved key, stub impl. Today
> `--spawn-backend=subint_forkserver` raises a clean
> `NotImplementedError` from `subint_forkserver_proc()`
> below, pointing at variant-1
> (`--spawn-backend=main_thread_forkserver`, see
> `tractor.spawn._main_thread_forkserver`) and the upstream
> blocker
> ([jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)).
> The key is reserved here (not aliased to variant-1) so the
> literal subint-hosted-child impl can flip in-place once
> msgspec#1026 unblocks PEP 684 isolated-mode subints
> upstream — no API churn at the call site.
> **Status:** placeholder. Today
> `--spawn-backend=subint_forkserver` aliases to
> `main_thread_forkserver_proc` (variant 1, see
> `tractor.spawn._main_thread_forkserver`). A follow-up commit
> in this PR series flips the alias to a `NotImplementedError`
> stub reserving the `'subint_forkserver'` key for the literal
> subint-hosted-child variant once
> [jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)
> unblocks PEP 684 isolated-mode subints upstream.
Future arch — what subints would buy us
---------------------------------------
@ -157,6 +154,21 @@ from trio import TaskStatus
from tractor.log import get_logger
from ._subint import _has_subints
# Backward-compat re-exports of the fork primitives whose
# canonical home is now `_main_thread_forkserver`. Kept here
# transiently so existing
# `from tractor.spawn._subint_forkserver import ...` callsites
# in the tests + the conc-anal smoketest keep resolving;
# dropped once a follow-up commit migrates those imports to
# the new module.
from ._main_thread_forkserver import (
_close_inherited_fds as _close_inherited_fds,
_format_child_exit as _format_child_exit,
fork_from_worker_thread as fork_from_worker_thread,
wait_child as wait_child,
_ForkedProc as _ForkedProc,
)
if TYPE_CHECKING:
from tractor.discovery._addr import UnwrappedAddress
from tractor.runtime._portal import Portal