Compare commits

..

No commits in common. "fbaf933ccbcbdbfe001a6c031a3c595c9b453a40" and "3818fe863853d85345d8c3a302d24e3d7641f6e1" have entirely different histories.

7 changed files with 94 additions and 644 deletions

View File

@ -88,27 +88,10 @@ jobs:
testing:
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
timeout-minutes: 20
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
timeout-minutes: 16
runs-on: ${{ matrix.os }}
# NOTE on the matrix shape — the `capture=` mode follows
# `spawn_backend`:
#
# - `trio` / `mp_*` backends use `--capture=fd` (default)
# for per-test attribution of subactor *raw-fd* output
# in failure reports.
# - Fork-based backends (`main_thread_forkserver`,
# `subint_forkserver`) REQUIRE `--capture=sys` because
# fork-child × `--capture=fd` is a known deadlock
# pattern. See the long NOTE in `tractor._testing.pytest`'s
# `pytest_load_initial_conftests` for the mechanism +
# tradeoff write-up.
#
# If a future matrix row adds a fork-spawn backend
# WITHOUT setting `capture: 'sys'`, the
# `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
# with a clear error msg. So the matrix is self-policing.
strategy:
fail-fast: false
matrix:
@ -135,26 +118,6 @@ jobs:
'tcp',
'uds',
]
capture: [
'fd', # default for non-fork backends
]
# Fork-based backends — added via `include:` so each
# cell carries its REQUIRED `capture: 'sys'` mode.
# Linux-only for now; macOS coverage TBD pending
# local validation.
include:
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'tcp'
capture: 'sys'
- os: ubuntu-latest
python-version: '3.13'
spawn_backend: 'main_thread_forkserver'
tpt_proto: 'uds'
capture: 'sys'
# https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
exclude:
# don't do UDS run on macOS (for now)
@ -195,11 +158,7 @@ jobs:
-rsx
--spawn-backend=${{ matrix.spawn_backend }}
--tpt-proto=${{ matrix.tpt_proto }}
--capture=${{ matrix.capture }}
# NOTE: capture mode is matrix-driven — `fd` for
# non-fork backends (per-test fd attribution),
# `sys` for fork-based (avoids fork-child x
# capture-fd deadlock). See matrix-NOTE above.
--capture=fd
# XXX legacy NOTE XXX
#

View File

@ -1,314 +0,0 @@
# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
## Symptom
Running the full test suite under
```bash
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver
```
surfaces a single, **rotating** `trio.TooSlowError`
failure each run. The failure isn't deterministic on
test identity — different test each run — but it
ALWAYS looks like:
```
FAILED tests/<file>::test_<name> - trio.TooSlowError
==== 1 failed, 373 passed, 17 skipped, 1112 xfailed,
01 xpassed, ~550 warnings in ~6min ====
```
Pass rate: **~99.7%** (373 of 374 non-skip tests).
Wall-clock per full run: 56 min.
## Tests observed flaking so far
Each row was the SOLE failure in a separate run:
| run # | test |
|---|---|
| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
Both share the same shape:
- **Cancel cascade** of N subactors back to a parent root actor.
- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
(it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
- N ≈ 2 for `test_context_spawns_aio_task_that_errors`
but each subactor is `infect_asyncio=True`, so each
cancel involves the trio↔asyncio guest-run unwind
which is structurally heavier than pure-trio.
- Test wraps the cascade in `trio.fail_after(N seconds)`
and the cap fires before the cascade completes.
The exact failing test rotates because each test is
independently close to the cap; whichever happens to
be unlucky in scheduling/CPU-contention on a given run
is the one that times out.
## Root-cause family
`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
the SC-graceful teardown ladder per subactor:
1. `Portal.cancel_actor()` — graceful IPC cancel-req.
2. Wait `terminate_after=1.6s` for sub to exit.
3. If still alive: `proc.kill()` (SIGKILL).
4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
sock-file cleanup for UDS leaks (issue #452 fix).
For a cascade of N subactors, each pays steps 14. If
graceful-cancel doesn't complete within 1.6s for ANY
sub, that sub eats a full 1.6s of `move_on_after` plus
the `proc.wait()` post-SIGKILL.
Worst case under fork backend with N=cpus subs:
- N × 1.6s = 16s+ on a 10-core box just for the
graceful timeout phase
- Plus per-spawn fork-IPC handshake cost compounds
during teardown (each sub's IPC cleanup goes through
the same forkserver coordinator)
- Plus the new autouse fixtures
(`_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors`) all run at test
teardown, adding small (10s of ms) but cumulative
overhead
Current cap: 30s (`fail_after_s = 30 if
is_forking_spawner else 12`). Empirically fits the
median run but the tail breaks ~0.3% of the time.
## NOT regressing
To confirm this is a flake and not a regression:
- Pre-`WakeupSocketpair`-patch baseline: tests
HUNG INDEFINITELY (busy-loop never released).
- Post-patch: pass-or-fail-fast, ~99.7% pass, the
occasional cap-hit fails in bounded time (<60s for
the offending test).
- Same test PASSES under `--spawn-backend=trio`
(no fork, no hard-kill compounding).
So the suite is dramatically better than before; the
remaining flake is a known-tolerable steady-state.
## Possible mitigations (ranked)
### A. Bump the cap further
Cheapest. Change the per-test `fail_after_s` from 30
to e.g. 60 for fork backends. Pros: trivial. Cons:
masks any genuine slowness regression we'd want to
catch.
### B. CPU-count-aware cap
For tests whose N scales with `cpu_count()`, scale
the cap too:
```python
fail_after_s = (
max(30, cpu_count() * 3) # 3s/actor floor
if is_forking_spawner
else 12
)
```
Pros: scales with the actual cancel-cascade work.
Cons: still arbitrary multiplier.
### C. `pytest-rerunfailures` for these tests only
Mark the known-flaky tests with
`@pytest.mark.flaky(reruns=1)` (needs
`pytest-rerunfailures` dep). Single retry hides
genuine ~0.3% transient flakes.
Pros: no cap change, surfaces persistent failures
loudly. Cons: adds a dep, retries can mask real bugs
if used widely.
### D. Reduce `hard_kill`'s `terminate_after`
Drop from 1.6s → 0.8s. Cuts the worst-case cascade
time roughly in half. Risks: fewer subs get a chance
to run their cleanup before SIGKILL → more orphaned
state for the autouse reapers to handle (ironically,
adds back overhead elsewhere).
### E. Profile + targeted fix
Add `log.devx()` markers in `hard_kill` to time each
phase. Identify if any subactor is consistently
hitting the 1.6s cap (vs. exiting in <0.1s). If so,
that sub has a teardown bug worth fixing at source.
Pros: actually fixes the underlying slowness. Cons:
real investigation work, deferred from this round.
## Recommendation
Land this issue-doc as the tracker. Apply **(B)** as
a small follow-up — cheap and proportional. If it
still flakes, escalate to **(E)** with a `log.devx()`
profile-pass.
`(C)` is a backstop if `(B)` doesn't quite get there
and we need green CI faster than (E) can deliver.
## Verification protocol
After applying any mitigation:
```bash
# Run the suite N times back-to-back, count failures.
# A persistent failure on the SAME test == real bug.
# Failures rotating across tests == still cap-related.
for i in $(seq 1 5); do
./py313/bin/python -m pytest tests/ \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver \
-q 2>&1 | tail -2
done
```
Target: 0 failures across 5 runs ⇒ ship. 12 failures
still rotating ⇒ apply (C). Same test failing twice
⇒ escalate to (E).
## Snapshot evidence (2026-05-13)
After landing the `fail_after_w_trace` /
`afk_alarm_w_trace` capture-on-timeout helpers
(`tractor._testing.trace`), `test_nested_multierrors`
on the `main_thread_forkserver` backend produces
**reproducible diag snapshots** at
`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
### Reproduction
```bash
pytest \
-v --verbose --durations=10 \
--spawn-backend=main_thread_forkserver \
--tpt-proto=uds \
--capture=sys --show-capture=stderr -rxX \
tests/test_cancellation.py::test_nested_multierrors
```
The test is `xfail(strict=False)` for MTF — it RUNS
each invocation so snapshots accumulate, but doesn't
break `--lf` workflow.
### Consistent shape across runs
5+ snapshots taken back-to-back show the SAME pattern:
- **Timing:** ~10s wall-clock total. Inner
`fail_after_w_trace(10)` fires at exactly T=10s;
cascade's `nursery.__aexit__` takes ~0.6s more to
gather + propagate the resulting
`BaseExceptionGroup`. **Trio backend completes the
SAME test in <6s** so the MTF cascade is ~2x
slower at minimum.
- **`BaseExceptionGroup` shape:** mixed
`[RemoteActorError, Cancelled]`. The first
subactor's natural error-propagation (`assert 0`
raised → `RemoteActorError` portal-result)
arrives before T=10s; the OTHER subactor's
portal-wait is still in flight at T=10s, gets
cancelled by `fail_after_w_trace`'s scope-cancel
→ returns `Cancelled` instead.
- **Orphan-spawn skew:** snapshot's `orphans` bucket
(after the `_is_tractor_subactor` cgroup-slice
override fix) consistently shows 2-4 init-adopted
procs at `depth_3` and `depth_1` levels — these
are the leaves whose parent (`depth_2` spawner)
was killed mid-cascade but who hadn't yet seen
the cancel signal themselves.
- **UDS sock-leak:** 2-6 dead-orphan socks per run
(varies with cascade timing). The
`track_orphaned_uds_per_test` fixture reaps them
post-test → contamination is isolated per-invocation.
### Capture mechanism
`fail_after_w_trace` covers two firing paths:
1. **`trio.TooSlowError`** raised at scope-exit
(body returned cleanly past deadline) — direct
`except` handler captures.
2. **Scope-cancel + body raises non-`Cancelled` exc**
(e.g. `nursery.__aexit__` wraps timeout-induced
`Cancelled` into a `BaseExceptionGroup` that
escapes before `trio.fail_after`'s exit-check
could fire `TooSlowError`) — body-raise `except`
handler checks `scope.cancel_called` and
captures if True. This path catches the
`test_nested_multierrors` shape specifically (see
"BaseExceptionGroup shape" above).
The snapshot dir contains:
- `trace.txt``ptree` + `hung_state` (kernel
`wchan`/`stack` + `py-spy dump --locals` when
sudo cached), with `include_strays=True`
surfacing any cross-test ghost subactor trees in
the `orphans` bucket.
- `bindspace.txt` — UDS bindspace classification
(live-active / orphaned-alive / orphaned-dead).
- `meta.json``{pid, label, captured_at, sudo_cached}`.
The end-of-session `pytest_terminal_summary` hook
in `tractor._testing.pytest` lists every snapshot
dir from the run so you don't have to scroll back
through captured-stderr lines:
```
========================= tractor hang-snapshot index ==========================
N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
<test-id>
→ /home/.../.cache/tractor/hung-dumps/<label>__<ts>
```
### Caveats
The snapshot fires AFTER the body-raise (not at the
exact moment of scope-cancel), so the parent's
py-spy frames show `_do_capture_snapshot` itself
running, NOT the cancel-cascade hang frame. To see
the actual hang state, manual `acli.ptree` /
`acli.hung_dump` from a second terminal at T=10s
would be needed — **not currently possible**
because per-test reaper fixtures clean up ~0.6s
post-timeout. See follow-up TODO in
`tractor/_testing/trace.py` for a
`TRACTOR_TRACE_HOLD=1` env-var pause mode.
## See also
- [#452](https://github.com/goodboy/tractor/issues/452) —
UDS sock-file leak (related — `hard_kill`'s
cleanup phase contributes to cascade time)
- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
— the upstream-trio fix that turned this from a
100% hang into a 0.3% flake
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
— the asyncio variant which contributes to one of
the rotating failures
- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
cascade source
- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
`_detect_runaway_subactors_per_test`,
`_reap_orphaned_subactors` — autouse cleanup
fixtures whose cumulative teardown overhead
contributes to the cascade time

View File

@ -89,12 +89,10 @@ except ImportError:
# the "zero tractor imports" isolation guarantee; now that
# CPython-level feasibility is confirmed, the validated
# primitives have moved into tractor proper.)
from tractor.spawn._main_thread_forkserver import (
fork_from_worker_thread,
wait_child,
)
from tractor.spawn._subint_forkserver import (
fork_from_worker_thread,
run_subint_in_worker_thread,
wait_child,
)

View File

@ -1,14 +1,13 @@
'''
Integration exercises for the `tractor.spawn._main_thread_forkserver`
Integration exercises for the `tractor.spawn._subint_forkserver`
submodule at three tiers:
1. the low-level primitives
(`fork_from_worker_thread()` from `_main_thread_forkserver`
+ `run_subint_in_worker_thread()` from
`_subint_forkserver`) driven from inside a real
(`fork_from_worker_thread()` +
`run_subint_in_worker_thread()`) driven from inside a real
`trio.run()` in the parent process,
2. the full `main_thread_forkserver_proc` spawn backend wired
2. the full `subint_forkserver_proc` spawn backend wired
through tractor's normal actor-nursery + portal-RPC
machinery i.e. `open_root_actor` + `open_nursery` +
`run_in_actor` against a subactor spawned via fork from a
@ -28,15 +27,15 @@ Those smoke-test scenarios are standalone — no trio runtime
in the *parent*. Tiers (1)+(2) here cover the primitives
driven from inside `trio.run()` in the parent, and tier (3)
(the `*_spawn_basic` test) drives the registered
`main_thread_forkserver` spawn backend end-to-end against
the tractor runtime.
`subint_forkserver` spawn backend end-to-end against the
tractor runtime.
Gating
------
- py3.14+ (via `concurrent.interpreters` presence)
- no `--spawn-backend` restriction the backend-level test
flips `tractor.spawn._spawn._spawn_method` programmatically
(via `try_set_start_method('main_thread_forkserver')`) and
(via `try_set_start_method('subint_forkserver')`) and
restores it on teardown, so these tests are independent of
the session-level CLI backend choice.
@ -65,12 +64,10 @@ from tractor.devx import dump_on_hang
# `tractor.spawn._subint` for why.
pytest.importorskip('concurrent.interpreters')
from tractor.spawn._main_thread_forkserver import ( # noqa: E402
fork_from_worker_thread,
wait_child,
)
from tractor.spawn._subint_forkserver import ( # noqa: E402
fork_from_worker_thread,
run_subint_in_worker_thread,
wait_child,
)
from tractor.spawn import _spawn as _spawn_mod # noqa: E402
from tractor.spawn._spawn import try_set_start_method # noqa: E402
@ -198,7 +195,7 @@ def test_fork_from_worker_thread_via_trio(
deadline: float = 10.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_baseline.dump',
path='/tmp/subint_forkserver_baseline.dump',
):
pid: int = trio.run(
partial(run_fork_in_non_trio_thread, deadline),
@ -220,14 +217,14 @@ def test_fork_and_run_trio_in_child() -> None:
`trio.run()` inside it on yet another worker thread.
This is the full "forkserver + trio-in-subint-in-child"
pattern the proposed `main_thread_forkserver` spawn backend
pattern the proposed `subint_forkserver` spawn backend
would rest on.
'''
deadline: float = 15.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_trio_in_child.dump',
path='/tmp/subint_forkserver_trio_in_child.dump',
):
pid: int = trio.run(
partial(
@ -240,7 +237,7 @@ def test_fork_and_run_trio_in_child() -> None:
# ----------------------------------------------------------------
# tier-3 backend test: drive the registered `main_thread_forkserver`
# tier-3 backend test: drive the registered `subint_forkserver`
# spawn backend end-to-end through tractor's actor-nursery +
# portal-RPC machinery.
# ----------------------------------------------------------------
@ -263,7 +260,7 @@ async def _happy_path_forkserver(
Parent-side harness: stand up a root actor, open an actor
nursery, spawn one subactor via the currently-selected
spawn backend (which this test will have flipped to
`main_thread_forkserver`), run a trivial RPC through its
`subint_forkserver`), run a trivial RPC through its
portal, assert the round-trip result.
'''
@ -307,19 +304,19 @@ def forkserver_spawn_method():
@pytest.mark.timeout(60, method='thread')
def test_main_thread_forkserver_spawn_basic(
def test_subint_forkserver_spawn_basic(
reg_addr: tuple[str, int | str],
forkserver_spawn_method,
) -> None:
'''
Happy-path: spawn ONE subactor via the
`main_thread_forkserver` backend (parent-side fork from a
`subint_forkserver` backend (parent-side fork from a
main-interp worker thread), do a trivial portal-RPC
round-trip, tear the nursery down cleanly.
If this passes, the "forkserver + tractor runtime" arch
is proven end-to-end: the registered
`main_thread_forkserver_proc` spawn target successfully
`subint_forkserver_proc` spawn target successfully
forks a child, the child runs `_actor_child_main()` +
completes IPC handshake + serves an RPC, and the parent
reaps via `_ForkedProc.wait()` without regressing any of
@ -329,7 +326,7 @@ def test_main_thread_forkserver_spawn_basic(
deadline: float = 20.0
with dump_on_hang(
seconds=deadline,
path='/tmp/main_thread_forkserver_spawn_basic.dump',
path='/tmp/subint_forkserver_spawn_basic.dump',
):
trio.run(
partial(
@ -343,7 +340,7 @@ def test_main_thread_forkserver_spawn_basic(
# ----------------------------------------------------------------
# tier-4 DRAFT: orphaned-subactor SIGINT survivability
#
# Motivating question: with `main_thread_forkserver`, the child's
# Motivating question: with `subint_forkserver`, the child's
# `trio.run()` lives on the fork-inherited worker thread which
# is NOT `threading.main_thread()` — so trio cannot install its
# `signal.set_wakeup_fd`-based SIGINT handler. If the parent
@ -363,7 +360,7 @@ def test_main_thread_forkserver_spawn_basic(
# Cross-backend generalization (decide after this passes):
# - applicable to any backend whose subactors are separate OS
# processes: `trio`, `mp_spawn`, `mp_forkserver`,
# `main_thread_forkserver`.
# `subint_forkserver`.
# - NOT applicable to plain `subint` (subactors are in-process
# subinterpreters, no orphan child process to SIGINT).
# - move path: lift the harness script into
@ -449,7 +446,7 @@ def _process_alive(pid: int) -> bool:
return False
# Known-gap test — `main_thread_forkserver` orphan-SIGINT
# Known-gap test — `subint_forkserver` orphan-SIGINT
# handling. See
# `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
# `strict=True` so if a future fix closes the gap the
@ -474,12 +471,12 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
) -> None:
'''
DRAFT orphaned-subactor SIGINT survivability under the
`main_thread_forkserver` backend.
`subint_forkserver` backend.
Sequence:
1. Spawn a harness subprocess that brings up a root
actor + one `sleep_forever` subactor via
`main_thread_forkserver`.
`subint_forkserver`.
2. Read the harness's stdout for `PARENT_READY=<pid>`
and `CHILD_PID=<pid>` markers (confirms the
parentchild IPC handshake completed).
@ -527,7 +524,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
[
sys.executable,
str(script_path),
'main_thread_forkserver',
'subint_forkserver',
host,
str(port),
],
@ -580,7 +577,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
pytest.fail(
f'Orphan subactor (pid={child_pid}) did NOT exit '
f'within 10s of SIGINT under `main_thread_forkserver` '
f'within 10s of SIGINT under `subint_forkserver` '
f'→ trio on non-main thread did not observe the '
f'default CPython KeyboardInterrupt; backend needs '
f'explicit SIGINT plumbing.'
@ -603,50 +600,3 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
pass
# ----------------------------------------------------------------
# regression guard: variant-2 (`subint_forkserver`) placeholder
# MUST raise `NotImplementedError` today — guards against future
# commits accidentally re-aliasing the key to the variant-1
# coroutine (which was a transient state during the rename).
# ----------------------------------------------------------------
def test_subint_forkserver_key_errors_cleanly() -> None:
'''
`--spawn-backend=subint_forkserver` is reserved for the
eventual variant-2 (subint-isolated child runtime)
backend, gated on jcrist/msgspec#1026 unblocking PEP 684
isolated-mode subints upstream.
Until that lands, the dispatch entry MUST raise
`NotImplementedError` immediately rather than silently
aliasing to `main_thread_forkserver_proc`. Verify the
error message also surfaces both the working-backend
pointer and the upstream-blocker ref so an operator
arriving at the error has somewhere to go.
'''
import asyncio
from tractor.spawn._spawn import _methods
proc = _methods['subint_forkserver']
with pytest.raises(NotImplementedError) as ei:
# signature args match `main_thread_forkserver_proc`'s
# — the stub raises before touching them so dummy
# values are fine.
asyncio.run(
proc(
'x', None, None, {}, [],
('127.0.0.1', 0), {},
)
)
msg: str = str(ei.value)
assert 'main_thread_forkserver' in msg, (
f'stub error msg should redirect to the working '
f'variant-1 backend; got: {msg!r}'
)
assert 'msgspec#1026' in msg or '1026' in msg, (
f'stub error msg should reference the upstream '
f'blocker (jcrist/msgspec#1026); got: {msg!r}'
)

View File

@ -38,7 +38,6 @@ Two empirical CPython properties drive the design:
the forked child otherwise (`Fatal Python error: not main
interpreter`). Full source-level walkthrough:
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
2. **`os.fork()` from a regular `threading.Thread` attached to
the *main* interpreter i.e. a worker thread that has never
entered a subint works cleanly.** Empirically validated
@ -87,11 +86,9 @@ costs:
- **Sidecar lifecycle**: a second long-lived process per
parent, with its own start/stop/health-check semantics.
- **IPC overhead per spawn**: every actor-spawn round-trips
an `mp` request message through a unix socket before any
child code runs.
- **State isolation by process boundary**: the sidecar can't
share parent state at all every spawn is a "cold" child
re-importing modules from disk.
@ -109,7 +106,6 @@ For the full variant-2 picture see
1) we already get costs 1 + 2 collapsed; cost 3 will land
when msgspec#1026 unblocks isolated-mode subints.
What survives the fork? POSIX semantics
-----------------------------------------
@ -117,58 +113,33 @@ A natural worry when forking from a parent that's running
`trio.run()` on another thread: does that trio thread (and
any other threads in the parent) keep running in the child?
**No** but with a precise meaning that's worth pinning
down, since the canonical trio framing
([python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
puts it the opposite-sounding way:
> If you use `fork()` in a process with multiple threads,
> all the other thread stacks are just leaked: there's
> nothing else you can reasonably do with them.
Both statements describe the same POSIX reality from
opposite sides:
- **Execution-side ("gone")**: POSIX `fork()` only
preserves the *calling* thread as a runnable thread in
the child. Every other thread in the parent trio's
runner thread, any `to_thread` cache threads, anything
else never executes another instruction post-fork.
- **Memory-side ("leaked")**: those non-running threads'
*stacks* and per-thread heap structures are still
COW-inherited into the child's address space. They
persist as orphaned bytes with no owning thread, no
scheduler entry, and no way for the child to clean
them up hence trio's word "leaked".
**No.** POSIX `fork()` only preserves the *calling* thread
in the child. Every other thread in the parent trio's
runner thread, any `to_thread` cache threads, anything else
is gone the instant `fork()` returns in the child.
Concretely, after the forkserver worker calls `os.fork()`:
| thread | parent | child (executing) | child (memory) |
|---------------------|-----------|-------------------|-----------------------------|
| forkserver worker | continues | sole survivor | live stack |
| `trio.run()` thread | continues | not running | leaked stack (zombie bytes) |
| any other thread | continues | not running | leaked stack (zombie bytes) |
| thread | parent | child |
|-----------------------|-----------|---------------|
| forkserver worker | continues | sole survivor |
| `trio.run()` thread | continues | gone |
| any other thread | continues | gone |
The forkserver worker becomes the new "main" execution
context in the child; `trio.run()` and every other parent
thread never executes a single instruction post-fork.
Their stack memory rides along as inert COW pages until
the child's fresh `trio.run()` boots and overwrites/GCs
it (or until the child `exec()`s and discards the entire
image).
thread never executes a single instruction post-fork in the
child.
This is exactly *why* `os.fork()` is delegated to a
dedicated worker thread that has provably never entered
trio: we want that trio-free thread to be the surviving
*executing* thread in the child, with the leaked trio
stack reduced to inert COW pages we don't touch.
one in the child.
The leaked-stack residue is one slice of the broader
"fork in a multithreaded program is dangerous" hazard
class (see `man pthread_atfork`). Other dead-thread
artifacts that cross the fork boundary, and how we handle
each:
That said, dead-thread *artifacts* still cross the fork
boundary (canonical "fork in a multithreaded program is
dangerous" — see `man pthread_atfork`). What persists, and
how we handle each:
- **Inherited file descriptors** the dead trio thread's
epoll fd, signal-wakeup-fd, eventfds, sockets, IPC
@ -177,20 +148,16 @@ each:
`_close_inherited_fds()` in the child prelude walks
`/proc/self/fd` and closes everything except stdio +
the channel pipe to the forkserver.
- **Memory image** trio's internal data structures
(scheduler, task queues, runner state) sit in COW
memory alongside the leaked stacks above. Nobody's
executing them; they get GC'd / overwritten when the
child's fresh `trio.run()` boots.
memory but nobody's executing them. Get GC'd /
overwritten when the child's fresh `trio.run()` boots.
- **Python thread state** handled automatically by
CPython. `PyOS_AfterFork_Child()` calls
`_PyThreadState_DeleteExceptCurrent()`, so dead
`PyThreadState` objects are cleaned and
`threading.enumerate()` returns just the surviving
thread.
- **User-level locks (`threading.Lock`)**
held-by-dead-thread state is the canonical fork hazard.
Not an issue in practice for tractor: trio doesn't hold
@ -199,7 +166,6 @@ each:
either direction). CPython's GIL is auto-reset by the
fork callback.
FYI: how this dodges the `trio.run()` × `fork()` hazards
--------------------------------------------------------
@ -217,16 +183,13 @@ design dodges each class explicitly:
reader. *Dodge*: the inherited wakeup-fd is closed by
`_close_inherited_fds()`, then the child's own
`trio.run()` installs a fresh one.
- **`epoll`/`kqueue` instance**: trio's I/O backend holds
one. Inherited as a dead fd; same fix as above.
- **Threadpool cache threads** (`trio.to_thread`): worker
threads with cached tstate. Don't exist in the child
(POSIX); cache state is meaningless garbage that gets
reset when the child's trio.run() initializes its own
thread cache.
- **Cancel scopes / nurseries / open `trio.Process` /
open sockets**: these are trio-runtime objects, not
kernel objects. The runtime that owns them is gone in
@ -234,11 +197,9 @@ design dodges each class explicitly:
in COW memory and get overwritten as the child runs.
Inherited *kernel* fds those objects wrapped (sockets,
proc pipes) are caught by `_close_inherited_fds()`.
- **`atexit` handlers**: trio doesn't register any that
would mis-fire post-fork; trio's lifetime-stack is
all `with`-block-scoped and dies with the runner.
- **Foreign-language I/O state** (libcurl, OpenSSL session
caches, etc.): out of scope same hazard as any
fork-without-exec; users layering those on top of
@ -250,7 +211,6 @@ isolation + `_close_inherited_fds()` cleanup gives the
forked child a clean trio environment. Everything else
falls under the standard fork-without-exec disclaimer.
Implementation status
---------------------
@ -271,11 +231,10 @@ follow-up) including the
Still-open work (tracked on tractor #379):
- [ ] no cancellation / hard-kill stress coverage yet
- no cancellation / hard-kill stress coverage yet
(counterpart to `tests/test_subint_cancellation.py` for
the plain `subint` backend),
- [ ] `child_sigint='trio'` mode (flag scaffolded below; default
- `child_sigint='trio'` mode (flag scaffolded below; default
is `'ipc'`). Originally intended as a manual SIGINT
trio-cancel bridge, but investigation showed trio's
handler IS already correctly installed in the fork-child
@ -328,24 +287,21 @@ See also
- `tractor.spawn._subint_forkserver` variant-2 placeholder
module; reserved for the future subint-isolated-child
runtime once jcrist/msgspec#1026 unblocks.
- `tractor.spawn._subint_fork` the stub for the
fork-from-non-main-subint strategy that DIDN'T work (kept
in-tree as documentation of the attempt + the CPython-level
block).
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
CPython source walkthrough of why fork-from-subint is dead.
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
standalone feasibility check (delegates to this module
for the primitives it exercises).
'''
from __future__ import annotations
import errno
import os
import signal
import sys
import threading
from functools import partial
from typing import (
@ -368,8 +324,8 @@ from tractor.runtime._portal import Portal
from ._spawn import (
cancel_on_completion,
soft_kill,
wait_for_peer_or_proc_death,
)
from ._subint import _has_subints
if TYPE_CHECKING:
from tractor.discovery._addr import UnwrappedAddress
@ -467,24 +423,9 @@ def _close_inherited_fds(
try:
os.close(fd)
closed += 1
except OSError as oserr:
# `EBADF` is the benign-and-expected case: the
# `os.listdir('/proc/self/fd')` call above itself
# opens a transient dirfd that ends up in
# `candidates`, then auto-closes before this loop
# reaches it. Same for any fd whose Python wrapper
# was GC'd between `listdir` and `os.close`.
# Suppress at debug-level — surfacing every
# EBADF as a full traceback (prior `log.exception`
# behavior) drowned the post-fork log channel.
if oserr.errno == errno.EBADF:
log.debug(
f'Skip already-closed inherited fd {fd!r} '
f'(EBADF, benign race with listdir)\n'
)
continue
# Other errnos (EIO / EPERM / EINTR / ...) are
# genuinely unexpected — keep the loud surface.
except OSError:
# fd was already closed (race with listdir) or otherwise
# unclosable — either is fine.
log.exception(
f'Failed to close inherited fd in child ??\n'
f'{fd!r}\n'
@ -761,26 +702,6 @@ class _ForkedProc:
self._pidfd = -1
return self._returncode
def terminate(self) -> None:
'''
OS-level `SIGTERM` to the child. Swallows
`ProcessLookupError` (already dead).
Mirrors `trio.Process.terminate()` /
`multiprocessing.Process.terminate()` sends SIGTERM
(graceful, allows the child a chance to clean up via
signal-handlers) rather than SIGKILL. Used by
`ActorNursery.cancel()`'s per-child escalation when
`Portal.cancel_actor()` raises `ActorTooSlowError`,
and by the legacy `hard_kill=True` branch on the same
path.
'''
try:
os.kill(self.pid, signal.SIGTERM)
except ProcessLookupError:
pass
def kill(self) -> None:
'''
OS-level `SIGKILL` to the child. Swallows
@ -851,6 +772,13 @@ async def main_thread_forkserver_proc(
thread instead of `trio.lowlevel.open_process()`.
'''
if not _has_subints:
raise RuntimeError(
f'The {"main_thread_forkserver"!r} spawn backend '
f'requires Python 3.14+.\n'
f'Current runtime: {sys.version}'
)
# Backend-scoped config pulled from `proc_kwargs`. Using
# `proc_kwargs` (vs a first-class kwarg on this function)
# matches how other backends expose per-spawn tuning
@ -969,18 +897,7 @@ async def main_thread_forkserver_proc(
f' |_{proc}\n'
)
# race the handshake-wait against proc-death so a
# sub that dies during boot (e.g. crashed on import
# before reaching `_actor_child_main`, leaving a
# zombie + no cmdline) surfaces as `ActorFailure`
# instead of parking the spawning task forever on
# an unsignalled `_peer_connected[uid]` event.
event, chan = await wait_for_peer_or_proc_death(
ipc_server,
uid,
proc_wait=proc.wait,
proc_repr=repr(proc),
)
event, chan = await ipc_server.wait_for_peer(uid)
except trio.Cancelled:
cancelled_during_spawn = True

View File

@ -43,7 +43,6 @@ from tractor.log import get_logger
from tractor.discovery._addr import (
UnwrappedAddress,
)
from .._exceptions import ActorFailure
from ._reap import unlink_uds_bind_addrs
from tractor.runtime._portal import Portal
from tractor.runtime._runtime import Actor
@ -83,15 +82,11 @@ SpawnMethodKey = Literal[
# runtime, exactly like `trio_proc` but via fork instead
# of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
'main_thread_forkserver',
# Variant-2: same fork machinery as `main_thread_forkserver`
# but the child enters a sub-interpreter to host its
# `trio.run()`. Gated on jcrist/msgspec#1026 unblocking
# PEP 684 isolated-mode subints upstream — until then
# `subint_forkserver_proc` is a clean `NotImplementedError`
# stub pointing at variant-1 (`main_thread_forkserver`) +
# the upstream blocker. The key is reserved here (not just
# aliased to variant-1) so once upstream lands the impl can
# flip in-place without API churn. See
# RESERVED for the future variant-2 subint-isolated-child
# runtime — gated on jcrist/msgspec#1026 + PEP 684. Today
# this key aliases to `main_thread_forkserver_proc`; once
# the upstream unblocks land it'll dispatch to the
# subint-hosted-trio impl. See
# `tractor.spawn._subint_forkserver`.
'subint_forkserver',
]
@ -111,71 +106,6 @@ else:
await trio.lowlevel.wait_readable(proc.sentinel)
async def wait_for_peer_or_proc_death(
ipc_server,
uid: tuple[str, str],
# TODO? not not types?
proc_wait: 'Callable[[], Awaitable]',
proc_repr: str = '',
) -> 'tuple[trio.Event, Channel]':
'''
Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
own `.wait()` coroutine. Whichever completes first cancels
the other.
Used by every spawn-backend to detect a sub-actor that
*dies during boot* before completing the parent-handshake-
callback (e.g. crashed on import, exec'd-out, kernel-killed
pre-`_actor_child_main`). Without this race, the
handshake-wait backed by an unsignalled `trio.Event`
parks the spawning task forever and leaves the dead child
as a zombie since nobody calls `proc.wait()` to reap.
On normal handshake-complete: returns `(event, chan)`
identical to a bare `wait_for_peer`.
On proc-death-first: raises `ActorFailure` carrying the
proc's exit code, allowing the supervisor to surface a
clean error rather than hanging indefinitely.
`proc_wait` is a 0-arg async callable returning the proc's
exit-status kept generic so each backend can pass its
own (`trio.Process.wait`, `_ForkedProc.wait`,
`proc_waiter(mp.Process)`, etc.).
`proc_repr` is an optional string used in the
`ActorFailure` message for diag.
'''
result: dict = {}
async def _await_handshake():
event, chan = await ipc_server.wait_for_peer(uid)
result['handshake'] = (event, chan)
boot_n.cancel_scope.cancel()
async def _await_death():
rc = await proc_wait()
result['died'] = rc
boot_n.cancel_scope.cancel()
async with trio.open_nursery() as boot_n:
boot_n.start_soon(_await_handshake)
boot_n.start_soon(_await_death)
if 'handshake' in result:
return result['handshake']
# only reached if proc-death won the race
raise ActorFailure(
f'Sub-actor {uid!r} died during boot '
f'(rc={result.get("died")!r}) before completing '
f'parent-handshake.\n'
f' proc: {proc_repr}'
)
def try_set_start_method(
key: SpawnMethodKey
@ -208,15 +138,13 @@ def try_set_start_method(
case 'mp_spawn':
_ctx = mp.get_context('spawn')
case (
'trio'
| 'main_thread_forkserver'
):
case 'trio':
_ctx = None
case (
'subint'
| 'subint_fork'
| 'main_thread_forkserver'
| 'subint_forkserver'
):
# All subint-family backends need no `mp.context`;

View File

@ -18,18 +18,15 @@
Variant-2 (future) "subint forkserver" placeholder reserved
for the eventual subint-isolated-child runtime variant.
> **Status:** reserved key, stub impl. Today
> `--spawn-backend=subint_forkserver` raises a clean
> `NotImplementedError` from `subint_forkserver_proc()`
> below, pointing at variant-1
> (`--spawn-backend=main_thread_forkserver`, see
> `tractor.spawn._main_thread_forkserver`) and the upstream
> blocker
> ([jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)).
> The key is reserved here (not aliased to variant-1) so the
> literal subint-hosted-child impl can flip in-place once
> msgspec#1026 unblocks PEP 684 isolated-mode subints
> upstream no API churn at the call site.
> **Status:** placeholder. Today
> `--spawn-backend=subint_forkserver` aliases to
> `main_thread_forkserver_proc` (variant 1, see
> `tractor.spawn._main_thread_forkserver`). A follow-up commit
> in this PR series flips the alias to a `NotImplementedError`
> stub reserving the `'subint_forkserver'` key for the literal
> subint-hosted-child variant once
> [jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)
> unblocks PEP 684 isolated-mode subints upstream.
Future arch what subints would buy us
---------------------------------------
@ -157,6 +154,21 @@ from trio import TaskStatus
from tractor.log import get_logger
from ._subint import _has_subints
# Backward-compat re-exports of the fork primitives whose
# canonical home is now `_main_thread_forkserver`. Kept here
# transiently so existing
# `from tractor.spawn._subint_forkserver import ...` callsites
# in the tests + the conc-anal smoketest keep resolving;
# dropped once a follow-up commit migrates those imports to
# the new module.
from ._main_thread_forkserver import (
_close_inherited_fds as _close_inherited_fds,
_format_child_exit as _format_child_exit,
fork_from_worker_thread as fork_from_worker_thread,
wait_child as wait_child,
_ForkedProc as _ForkedProc,
)
if TYPE_CHECKING:
from tractor.discovery._addr import UnwrappedAddress
from tractor.runtime._portal import Portal