Compare commits
No commits in common. "fbaf933ccbcbdbfe001a6c031a3c595c9b453a40" and "3818fe863853d85345d8c3a302d24e3d7641f6e1" have entirely different histories.
fbaf933ccb
...
3818fe8638
|
|
@ -88,27 +88,10 @@ jobs:
|
||||||
|
|
||||||
|
|
||||||
testing:
|
testing:
|
||||||
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }} capture=${{ matrix.capture }}'
|
name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
|
||||||
timeout-minutes: 20
|
timeout-minutes: 16
|
||||||
runs-on: ${{ matrix.os }}
|
runs-on: ${{ matrix.os }}
|
||||||
|
|
||||||
# NOTE on the matrix shape — the `capture=` mode follows
|
|
||||||
# `spawn_backend`:
|
|
||||||
#
|
|
||||||
# - `trio` / `mp_*` backends use `--capture=fd` (default)
|
|
||||||
# for per-test attribution of subactor *raw-fd* output
|
|
||||||
# in failure reports.
|
|
||||||
# - Fork-based backends (`main_thread_forkserver`,
|
|
||||||
# `subint_forkserver`) REQUIRE `--capture=sys` because
|
|
||||||
# fork-child × `--capture=fd` is a known deadlock
|
|
||||||
# pattern. See the long NOTE in `tractor._testing.pytest`'s
|
|
||||||
# `pytest_load_initial_conftests` for the mechanism +
|
|
||||||
# tradeoff write-up.
|
|
||||||
#
|
|
||||||
# If a future matrix row adds a fork-spawn backend
|
|
||||||
# WITHOUT setting `capture: 'sys'`, the
|
|
||||||
# `pytest_load_initial_conftests` hook fail-fasts on `CI=1`
|
|
||||||
# with a clear error msg. So the matrix is self-policing.
|
|
||||||
strategy:
|
strategy:
|
||||||
fail-fast: false
|
fail-fast: false
|
||||||
matrix:
|
matrix:
|
||||||
|
|
@ -135,26 +118,6 @@ jobs:
|
||||||
'tcp',
|
'tcp',
|
||||||
'uds',
|
'uds',
|
||||||
]
|
]
|
||||||
capture: [
|
|
||||||
'fd', # default for non-fork backends
|
|
||||||
]
|
|
||||||
|
|
||||||
# Fork-based backends — added via `include:` so each
|
|
||||||
# cell carries its REQUIRED `capture: 'sys'` mode.
|
|
||||||
# Linux-only for now; macOS coverage TBD pending
|
|
||||||
# local validation.
|
|
||||||
include:
|
|
||||||
- os: ubuntu-latest
|
|
||||||
python-version: '3.13'
|
|
||||||
spawn_backend: 'main_thread_forkserver'
|
|
||||||
tpt_proto: 'tcp'
|
|
||||||
capture: 'sys'
|
|
||||||
- os: ubuntu-latest
|
|
||||||
python-version: '3.13'
|
|
||||||
spawn_backend: 'main_thread_forkserver'
|
|
||||||
tpt_proto: 'uds'
|
|
||||||
capture: 'sys'
|
|
||||||
|
|
||||||
# https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
|
# https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
|
||||||
exclude:
|
exclude:
|
||||||
# don't do UDS run on macOS (for now)
|
# don't do UDS run on macOS (for now)
|
||||||
|
|
@ -195,11 +158,7 @@ jobs:
|
||||||
-rsx
|
-rsx
|
||||||
--spawn-backend=${{ matrix.spawn_backend }}
|
--spawn-backend=${{ matrix.spawn_backend }}
|
||||||
--tpt-proto=${{ matrix.tpt_proto }}
|
--tpt-proto=${{ matrix.tpt_proto }}
|
||||||
--capture=${{ matrix.capture }}
|
--capture=fd
|
||||||
# NOTE: capture mode is matrix-driven — `fd` for
|
|
||||||
# non-fork backends (per-test fd attribution),
|
|
||||||
# `sys` for fork-based (avoids fork-child x
|
|
||||||
# capture-fd deadlock). See matrix-NOTE above.
|
|
||||||
|
|
||||||
# XXX legacy NOTE XXX
|
# XXX legacy NOTE XXX
|
||||||
#
|
#
|
||||||
|
|
|
||||||
|
|
@ -1,314 +0,0 @@
|
||||||
# Cancel-cascade `trio.TooSlowError` flakes under `main_thread_forkserver`
|
|
||||||
|
|
||||||
## Symptom
|
|
||||||
|
|
||||||
Running the full test suite under
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./py313/bin/python -m pytest tests/ \
|
|
||||||
--tpt-proto=tcp \
|
|
||||||
--spawn-backend=main_thread_forkserver
|
|
||||||
```
|
|
||||||
|
|
||||||
surfaces a single, **rotating** `trio.TooSlowError`
|
|
||||||
failure each run. The failure isn't deterministic on
|
|
||||||
test identity — different test each run — but it
|
|
||||||
ALWAYS looks like:
|
|
||||||
|
|
||||||
```
|
|
||||||
FAILED tests/<file>::test_<name> - trio.TooSlowError
|
|
||||||
==== 1 failed, 373 passed, 17 skipped, 11–12 xfailed,
|
|
||||||
0–1 xpassed, ~550 warnings in ~6min ====
|
|
||||||
```
|
|
||||||
|
|
||||||
Pass rate: **~99.7%** (373 of 374 non-skip tests).
|
|
||||||
Wall-clock per full run: 5–6 min.
|
|
||||||
|
|
||||||
## Tests observed flaking so far
|
|
||||||
|
|
||||||
Each row was the SOLE failure in a separate run:
|
|
||||||
|
|
||||||
| run # | test |
|
|
||||||
|---|---|
|
|
||||||
| 1 | `tests/test_advanced_streaming.py::test_dynamic_pub_sub[KeyboardInterrupt]` |
|
|
||||||
| 2 | `tests/test_infected_asyncio.py::test_context_spawns_aio_task_that_errors[parent_actor_cancels_child=False]` |
|
|
||||||
|
|
||||||
Both share the same shape:
|
|
||||||
|
|
||||||
- **Cancel cascade** of N subactors back to a parent root actor.
|
|
||||||
- N ≥ `multiprocessing.cpu_count()` for `test_dynamic_pub_sub`
|
|
||||||
(it spawns `cpus - 1` consumers + publisher + dynamic-consumer).
|
|
||||||
- N ≈ 2 for `test_context_spawns_aio_task_that_errors` —
|
|
||||||
but each subactor is `infect_asyncio=True`, so each
|
|
||||||
cancel involves the trio↔asyncio guest-run unwind
|
|
||||||
which is structurally heavier than pure-trio.
|
|
||||||
- Test wraps the cascade in `trio.fail_after(N seconds)`
|
|
||||||
and the cap fires before the cascade completes.
|
|
||||||
|
|
||||||
The exact failing test rotates because each test is
|
|
||||||
independently close to the cap; whichever happens to
|
|
||||||
be unlucky in scheduling/CPU-contention on a given run
|
|
||||||
is the one that times out.
|
|
||||||
|
|
||||||
## Root-cause family
|
|
||||||
|
|
||||||
`hard_kill` (`tractor/spawn/_spawn.py:hard_kill`) runs
|
|
||||||
the SC-graceful teardown ladder per subactor:
|
|
||||||
|
|
||||||
1. `Portal.cancel_actor()` — graceful IPC cancel-req.
|
|
||||||
2. Wait `terminate_after=1.6s` for sub to exit.
|
|
||||||
3. If still alive: `proc.kill()` (SIGKILL).
|
|
||||||
4. (NEW) `_unlink_uds_bind_addrs()` — post-mortem
|
|
||||||
sock-file cleanup for UDS leaks (issue #452 fix).
|
|
||||||
|
|
||||||
For a cascade of N subactors, each pays steps 1–4. If
|
|
||||||
graceful-cancel doesn't complete within 1.6s for ANY
|
|
||||||
sub, that sub eats a full 1.6s of `move_on_after` plus
|
|
||||||
the `proc.wait()` post-SIGKILL.
|
|
||||||
|
|
||||||
Worst case under fork backend with N=cpus subs:
|
|
||||||
- N × 1.6s = 16s+ on a 10-core box just for the
|
|
||||||
graceful timeout phase
|
|
||||||
- Plus per-spawn fork-IPC handshake cost compounds
|
|
||||||
during teardown (each sub's IPC cleanup goes through
|
|
||||||
the same forkserver coordinator)
|
|
||||||
- Plus the new autouse fixtures
|
|
||||||
(`_track_orphaned_uds_per_test`,
|
|
||||||
`_detect_runaway_subactors_per_test`,
|
|
||||||
`_reap_orphaned_subactors`) all run at test
|
|
||||||
teardown, adding small (10s of ms) but cumulative
|
|
||||||
overhead
|
|
||||||
|
|
||||||
Current cap: 30s (`fail_after_s = 30 if
|
|
||||||
is_forking_spawner else 12`). Empirically fits the
|
|
||||||
median run but the tail breaks ~0.3% of the time.
|
|
||||||
|
|
||||||
## NOT regressing
|
|
||||||
|
|
||||||
To confirm this is a flake and not a regression:
|
|
||||||
|
|
||||||
- Pre-`WakeupSocketpair`-patch baseline: tests
|
|
||||||
HUNG INDEFINITELY (busy-loop never released).
|
|
||||||
- Post-patch: pass-or-fail-fast, ~99.7% pass, the
|
|
||||||
occasional cap-hit fails in bounded time (<60s for
|
|
||||||
the offending test).
|
|
||||||
- Same test PASSES under `--spawn-backend=trio`
|
|
||||||
(no fork, no hard-kill compounding).
|
|
||||||
|
|
||||||
So the suite is dramatically better than before; the
|
|
||||||
remaining flake is a known-tolerable steady-state.
|
|
||||||
|
|
||||||
## Possible mitigations (ranked)
|
|
||||||
|
|
||||||
### A. Bump the cap further
|
|
||||||
|
|
||||||
Cheapest. Change the per-test `fail_after_s` from 30
|
|
||||||
to e.g. 60 for fork backends. Pros: trivial. Cons:
|
|
||||||
masks any genuine slowness regression we'd want to
|
|
||||||
catch.
|
|
||||||
|
|
||||||
### B. CPU-count-aware cap
|
|
||||||
|
|
||||||
For tests whose N scales with `cpu_count()`, scale
|
|
||||||
the cap too:
|
|
||||||
|
|
||||||
```python
|
|
||||||
fail_after_s = (
|
|
||||||
max(30, cpu_count() * 3) # 3s/actor floor
|
|
||||||
if is_forking_spawner
|
|
||||||
else 12
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
Pros: scales with the actual cancel-cascade work.
|
|
||||||
Cons: still arbitrary multiplier.
|
|
||||||
|
|
||||||
### C. `pytest-rerunfailures` for these tests only
|
|
||||||
|
|
||||||
Mark the known-flaky tests with
|
|
||||||
`@pytest.mark.flaky(reruns=1)` (needs
|
|
||||||
`pytest-rerunfailures` dep). Single retry hides
|
|
||||||
genuine ~0.3% transient flakes.
|
|
||||||
|
|
||||||
Pros: no cap change, surfaces persistent failures
|
|
||||||
loudly. Cons: adds a dep, retries can mask real bugs
|
|
||||||
if used widely.
|
|
||||||
|
|
||||||
### D. Reduce `hard_kill`'s `terminate_after`
|
|
||||||
|
|
||||||
Drop from 1.6s → 0.8s. Cuts the worst-case cascade
|
|
||||||
time roughly in half. Risks: fewer subs get a chance
|
|
||||||
to run their cleanup before SIGKILL → more orphaned
|
|
||||||
state for the autouse reapers to handle (ironically,
|
|
||||||
adds back overhead elsewhere).
|
|
||||||
|
|
||||||
### E. Profile + targeted fix
|
|
||||||
|
|
||||||
Add `log.devx()` markers in `hard_kill` to time each
|
|
||||||
phase. Identify if any subactor is consistently
|
|
||||||
hitting the 1.6s cap (vs. exiting in <0.1s). If so,
|
|
||||||
that sub has a teardown bug worth fixing at source.
|
|
||||||
Pros: actually fixes the underlying slowness. Cons:
|
|
||||||
real investigation work, deferred from this round.
|
|
||||||
|
|
||||||
## Recommendation
|
|
||||||
|
|
||||||
Land this issue-doc as the tracker. Apply **(B)** as
|
|
||||||
a small follow-up — cheap and proportional. If it
|
|
||||||
still flakes, escalate to **(E)** with a `log.devx()`
|
|
||||||
profile-pass.
|
|
||||||
|
|
||||||
`(C)` is a backstop if `(B)` doesn't quite get there
|
|
||||||
and we need green CI faster than (E) can deliver.
|
|
||||||
|
|
||||||
## Verification protocol
|
|
||||||
|
|
||||||
After applying any mitigation:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run the suite N times back-to-back, count failures.
|
|
||||||
# A persistent failure on the SAME test == real bug.
|
|
||||||
# Failures rotating across tests == still cap-related.
|
|
||||||
|
|
||||||
for i in $(seq 1 5); do
|
|
||||||
./py313/bin/python -m pytest tests/ \
|
|
||||||
--tpt-proto=tcp \
|
|
||||||
--spawn-backend=main_thread_forkserver \
|
|
||||||
-q 2>&1 | tail -2
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
Target: 0 failures across 5 runs ⇒ ship. 1–2 failures
|
|
||||||
still rotating ⇒ apply (C). Same test failing twice
|
|
||||||
⇒ escalate to (E).
|
|
||||||
|
|
||||||
## Snapshot evidence (2026-05-13)
|
|
||||||
|
|
||||||
After landing the `fail_after_w_trace` /
|
|
||||||
`afk_alarm_w_trace` capture-on-timeout helpers
|
|
||||||
(`tractor._testing.trace`), `test_nested_multierrors`
|
|
||||||
on the `main_thread_forkserver` backend produces
|
|
||||||
**reproducible diag snapshots** at
|
|
||||||
`$XDG_CACHE_HOME/tractor/hung-dumps/test_nested_multierrors_start_method_main_thread_forkserver__<iso-ts>/`.
|
|
||||||
|
|
||||||
### Reproduction
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pytest \
|
|
||||||
-v --verbose --durations=10 \
|
|
||||||
--spawn-backend=main_thread_forkserver \
|
|
||||||
--tpt-proto=uds \
|
|
||||||
--capture=sys --show-capture=stderr -rxX \
|
|
||||||
tests/test_cancellation.py::test_nested_multierrors
|
|
||||||
```
|
|
||||||
|
|
||||||
The test is `xfail(strict=False)` for MTF — it RUNS
|
|
||||||
each invocation so snapshots accumulate, but doesn't
|
|
||||||
break `--lf` workflow.
|
|
||||||
|
|
||||||
### Consistent shape across runs
|
|
||||||
|
|
||||||
5+ snapshots taken back-to-back show the SAME pattern:
|
|
||||||
|
|
||||||
- **Timing:** ~10s wall-clock total. Inner
|
|
||||||
`fail_after_w_trace(10)` fires at exactly T=10s;
|
|
||||||
cascade's `nursery.__aexit__` takes ~0.6s more to
|
|
||||||
gather + propagate the resulting
|
|
||||||
`BaseExceptionGroup`. **Trio backend completes the
|
|
||||||
SAME test in <6s** — so the MTF cascade is ~2x
|
|
||||||
slower at minimum.
|
|
||||||
|
|
||||||
- **`BaseExceptionGroup` shape:** mixed
|
|
||||||
`[RemoteActorError, Cancelled]`. The first
|
|
||||||
subactor's natural error-propagation (`assert 0`
|
|
||||||
raised → `RemoteActorError` portal-result)
|
|
||||||
arrives before T=10s; the OTHER subactor's
|
|
||||||
portal-wait is still in flight at T=10s, gets
|
|
||||||
cancelled by `fail_after_w_trace`'s scope-cancel
|
|
||||||
→ returns `Cancelled` instead.
|
|
||||||
|
|
||||||
- **Orphan-spawn skew:** snapshot's `orphans` bucket
|
|
||||||
(after the `_is_tractor_subactor` cgroup-slice
|
|
||||||
override fix) consistently shows 2-4 init-adopted
|
|
||||||
procs at `depth_3` and `depth_1` levels — these
|
|
||||||
are the leaves whose parent (`depth_2` spawner)
|
|
||||||
was killed mid-cascade but who hadn't yet seen
|
|
||||||
the cancel signal themselves.
|
|
||||||
|
|
||||||
- **UDS sock-leak:** 2-6 dead-orphan socks per run
|
|
||||||
(varies with cascade timing). The
|
|
||||||
`track_orphaned_uds_per_test` fixture reaps them
|
|
||||||
post-test → contamination is isolated per-invocation.
|
|
||||||
|
|
||||||
### Capture mechanism
|
|
||||||
|
|
||||||
`fail_after_w_trace` covers two firing paths:
|
|
||||||
|
|
||||||
1. **`trio.TooSlowError`** raised at scope-exit
|
|
||||||
(body returned cleanly past deadline) — direct
|
|
||||||
`except` handler captures.
|
|
||||||
|
|
||||||
2. **Scope-cancel + body raises non-`Cancelled` exc**
|
|
||||||
(e.g. `nursery.__aexit__` wraps timeout-induced
|
|
||||||
`Cancelled` into a `BaseExceptionGroup` that
|
|
||||||
escapes before `trio.fail_after`'s exit-check
|
|
||||||
could fire `TooSlowError`) — body-raise `except`
|
|
||||||
handler checks `scope.cancel_called` and
|
|
||||||
captures if True. This path catches the
|
|
||||||
`test_nested_multierrors` shape specifically (see
|
|
||||||
"BaseExceptionGroup shape" above).
|
|
||||||
|
|
||||||
The snapshot dir contains:
|
|
||||||
- `trace.txt` — `ptree` + `hung_state` (kernel
|
|
||||||
`wchan`/`stack` + `py-spy dump --locals` when
|
|
||||||
sudo cached), with `include_strays=True`
|
|
||||||
surfacing any cross-test ghost subactor trees in
|
|
||||||
the `orphans` bucket.
|
|
||||||
- `bindspace.txt` — UDS bindspace classification
|
|
||||||
(live-active / orphaned-alive / orphaned-dead).
|
|
||||||
- `meta.json` — `{pid, label, captured_at, sudo_cached}`.
|
|
||||||
|
|
||||||
The end-of-session `pytest_terminal_summary` hook
|
|
||||||
in `tractor._testing.pytest` lists every snapshot
|
|
||||||
dir from the run so you don't have to scroll back
|
|
||||||
through captured-stderr lines:
|
|
||||||
|
|
||||||
```
|
|
||||||
========================= tractor hang-snapshot index ==========================
|
|
||||||
N `fail_after_w_trace` / `afk_alarm_w_trace` snapshot(s) captured this session:
|
|
||||||
<test-id>
|
|
||||||
→ /home/.../.cache/tractor/hung-dumps/<label>__<ts>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Caveats
|
|
||||||
|
|
||||||
The snapshot fires AFTER the body-raise (not at the
|
|
||||||
exact moment of scope-cancel), so the parent's
|
|
||||||
py-spy frames show `_do_capture_snapshot` itself
|
|
||||||
running, NOT the cancel-cascade hang frame. To see
|
|
||||||
the actual hang state, manual `acli.ptree` /
|
|
||||||
`acli.hung_dump` from a second terminal at T=10s
|
|
||||||
would be needed — **not currently possible**
|
|
||||||
because per-test reaper fixtures clean up ~0.6s
|
|
||||||
post-timeout. See follow-up TODO in
|
|
||||||
`tractor/_testing/trace.py` for a
|
|
||||||
`TRACTOR_TRACE_HOLD=1` env-var pause mode.
|
|
||||||
|
|
||||||
## See also
|
|
||||||
|
|
||||||
- [#452](https://github.com/goodboy/tractor/issues/452) —
|
|
||||||
UDS sock-file leak (related — `hard_kill`'s
|
|
||||||
cleanup phase contributes to cascade time)
|
|
||||||
- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
|
|
||||||
— the upstream-trio fix that turned this from a
|
|
||||||
100% hang into a 0.3% flake
|
|
||||||
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
|
|
||||||
— the asyncio variant which contributes to one of
|
|
||||||
the rotating failures
|
|
||||||
- `tractor/spawn/_spawn.py::hard_kill` — the SIGKILL
|
|
||||||
cascade source
|
|
||||||
- `tractor/_testing/_reap.py::_track_orphaned_uds_per_test`,
|
|
||||||
`_detect_runaway_subactors_per_test`,
|
|
||||||
`_reap_orphaned_subactors` — autouse cleanup
|
|
||||||
fixtures whose cumulative teardown overhead
|
|
||||||
contributes to the cascade time
|
|
||||||
|
|
@ -89,12 +89,10 @@ except ImportError:
|
||||||
# the "zero tractor imports" isolation guarantee; now that
|
# the "zero tractor imports" isolation guarantee; now that
|
||||||
# CPython-level feasibility is confirmed, the validated
|
# CPython-level feasibility is confirmed, the validated
|
||||||
# primitives have moved into tractor proper.)
|
# primitives have moved into tractor proper.)
|
||||||
from tractor.spawn._main_thread_forkserver import (
|
|
||||||
fork_from_worker_thread,
|
|
||||||
wait_child,
|
|
||||||
)
|
|
||||||
from tractor.spawn._subint_forkserver import (
|
from tractor.spawn._subint_forkserver import (
|
||||||
|
fork_from_worker_thread,
|
||||||
run_subint_in_worker_thread,
|
run_subint_in_worker_thread,
|
||||||
|
wait_child,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,13 @@
|
||||||
'''
|
'''
|
||||||
Integration exercises for the `tractor.spawn._main_thread_forkserver`
|
Integration exercises for the `tractor.spawn._subint_forkserver`
|
||||||
submodule at three tiers:
|
submodule at three tiers:
|
||||||
|
|
||||||
1. the low-level primitives
|
1. the low-level primitives
|
||||||
(`fork_from_worker_thread()` from `_main_thread_forkserver`
|
(`fork_from_worker_thread()` +
|
||||||
+ `run_subint_in_worker_thread()` from
|
`run_subint_in_worker_thread()`) driven from inside a real
|
||||||
`_subint_forkserver`) driven from inside a real
|
|
||||||
`trio.run()` in the parent process,
|
`trio.run()` in the parent process,
|
||||||
|
|
||||||
2. the full `main_thread_forkserver_proc` spawn backend wired
|
2. the full `subint_forkserver_proc` spawn backend wired
|
||||||
through tractor's normal actor-nursery + portal-RPC
|
through tractor's normal actor-nursery + portal-RPC
|
||||||
machinery — i.e. `open_root_actor` + `open_nursery` +
|
machinery — i.e. `open_root_actor` + `open_nursery` +
|
||||||
`run_in_actor` against a subactor spawned via fork from a
|
`run_in_actor` against a subactor spawned via fork from a
|
||||||
|
|
@ -28,15 +27,15 @@ Those smoke-test scenarios are standalone — no trio runtime
|
||||||
in the *parent*. Tiers (1)+(2) here cover the primitives
|
in the *parent*. Tiers (1)+(2) here cover the primitives
|
||||||
driven from inside `trio.run()` in the parent, and tier (3)
|
driven from inside `trio.run()` in the parent, and tier (3)
|
||||||
(the `*_spawn_basic` test) drives the registered
|
(the `*_spawn_basic` test) drives the registered
|
||||||
`main_thread_forkserver` spawn backend end-to-end against
|
`subint_forkserver` spawn backend end-to-end against the
|
||||||
the tractor runtime.
|
tractor runtime.
|
||||||
|
|
||||||
Gating
|
Gating
|
||||||
------
|
------
|
||||||
- py3.14+ (via `concurrent.interpreters` presence)
|
- py3.14+ (via `concurrent.interpreters` presence)
|
||||||
- no `--spawn-backend` restriction — the backend-level test
|
- no `--spawn-backend` restriction — the backend-level test
|
||||||
flips `tractor.spawn._spawn._spawn_method` programmatically
|
flips `tractor.spawn._spawn._spawn_method` programmatically
|
||||||
(via `try_set_start_method('main_thread_forkserver')`) and
|
(via `try_set_start_method('subint_forkserver')`) and
|
||||||
restores it on teardown, so these tests are independent of
|
restores it on teardown, so these tests are independent of
|
||||||
the session-level CLI backend choice.
|
the session-level CLI backend choice.
|
||||||
|
|
||||||
|
|
@ -65,12 +64,10 @@ from tractor.devx import dump_on_hang
|
||||||
# `tractor.spawn._subint` for why.
|
# `tractor.spawn._subint` for why.
|
||||||
pytest.importorskip('concurrent.interpreters')
|
pytest.importorskip('concurrent.interpreters')
|
||||||
|
|
||||||
from tractor.spawn._main_thread_forkserver import ( # noqa: E402
|
|
||||||
fork_from_worker_thread,
|
|
||||||
wait_child,
|
|
||||||
)
|
|
||||||
from tractor.spawn._subint_forkserver import ( # noqa: E402
|
from tractor.spawn._subint_forkserver import ( # noqa: E402
|
||||||
|
fork_from_worker_thread,
|
||||||
run_subint_in_worker_thread,
|
run_subint_in_worker_thread,
|
||||||
|
wait_child,
|
||||||
)
|
)
|
||||||
from tractor.spawn import _spawn as _spawn_mod # noqa: E402
|
from tractor.spawn import _spawn as _spawn_mod # noqa: E402
|
||||||
from tractor.spawn._spawn import try_set_start_method # noqa: E402
|
from tractor.spawn._spawn import try_set_start_method # noqa: E402
|
||||||
|
|
@ -198,7 +195,7 @@ def test_fork_from_worker_thread_via_trio(
|
||||||
deadline: float = 10.0
|
deadline: float = 10.0
|
||||||
with dump_on_hang(
|
with dump_on_hang(
|
||||||
seconds=deadline,
|
seconds=deadline,
|
||||||
path='/tmp/main_thread_forkserver_baseline.dump',
|
path='/tmp/subint_forkserver_baseline.dump',
|
||||||
):
|
):
|
||||||
pid: int = trio.run(
|
pid: int = trio.run(
|
||||||
partial(run_fork_in_non_trio_thread, deadline),
|
partial(run_fork_in_non_trio_thread, deadline),
|
||||||
|
|
@ -220,14 +217,14 @@ def test_fork_and_run_trio_in_child() -> None:
|
||||||
`trio.run()` inside it on yet another worker thread.
|
`trio.run()` inside it on yet another worker thread.
|
||||||
|
|
||||||
This is the full "forkserver + trio-in-subint-in-child"
|
This is the full "forkserver + trio-in-subint-in-child"
|
||||||
pattern the proposed `main_thread_forkserver` spawn backend
|
pattern the proposed `subint_forkserver` spawn backend
|
||||||
would rest on.
|
would rest on.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
deadline: float = 15.0
|
deadline: float = 15.0
|
||||||
with dump_on_hang(
|
with dump_on_hang(
|
||||||
seconds=deadline,
|
seconds=deadline,
|
||||||
path='/tmp/main_thread_forkserver_trio_in_child.dump',
|
path='/tmp/subint_forkserver_trio_in_child.dump',
|
||||||
):
|
):
|
||||||
pid: int = trio.run(
|
pid: int = trio.run(
|
||||||
partial(
|
partial(
|
||||||
|
|
@ -240,7 +237,7 @@ def test_fork_and_run_trio_in_child() -> None:
|
||||||
|
|
||||||
|
|
||||||
# ----------------------------------------------------------------
|
# ----------------------------------------------------------------
|
||||||
# tier-3 backend test: drive the registered `main_thread_forkserver`
|
# tier-3 backend test: drive the registered `subint_forkserver`
|
||||||
# spawn backend end-to-end through tractor's actor-nursery +
|
# spawn backend end-to-end through tractor's actor-nursery +
|
||||||
# portal-RPC machinery.
|
# portal-RPC machinery.
|
||||||
# ----------------------------------------------------------------
|
# ----------------------------------------------------------------
|
||||||
|
|
@ -263,7 +260,7 @@ async def _happy_path_forkserver(
|
||||||
Parent-side harness: stand up a root actor, open an actor
|
Parent-side harness: stand up a root actor, open an actor
|
||||||
nursery, spawn one subactor via the currently-selected
|
nursery, spawn one subactor via the currently-selected
|
||||||
spawn backend (which this test will have flipped to
|
spawn backend (which this test will have flipped to
|
||||||
`main_thread_forkserver`), run a trivial RPC through its
|
`subint_forkserver`), run a trivial RPC through its
|
||||||
portal, assert the round-trip result.
|
portal, assert the round-trip result.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
|
@ -307,19 +304,19 @@ def forkserver_spawn_method():
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.timeout(60, method='thread')
|
@pytest.mark.timeout(60, method='thread')
|
||||||
def test_main_thread_forkserver_spawn_basic(
|
def test_subint_forkserver_spawn_basic(
|
||||||
reg_addr: tuple[str, int | str],
|
reg_addr: tuple[str, int | str],
|
||||||
forkserver_spawn_method,
|
forkserver_spawn_method,
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Happy-path: spawn ONE subactor via the
|
Happy-path: spawn ONE subactor via the
|
||||||
`main_thread_forkserver` backend (parent-side fork from a
|
`subint_forkserver` backend (parent-side fork from a
|
||||||
main-interp worker thread), do a trivial portal-RPC
|
main-interp worker thread), do a trivial portal-RPC
|
||||||
round-trip, tear the nursery down cleanly.
|
round-trip, tear the nursery down cleanly.
|
||||||
|
|
||||||
If this passes, the "forkserver + tractor runtime" arch
|
If this passes, the "forkserver + tractor runtime" arch
|
||||||
is proven end-to-end: the registered
|
is proven end-to-end: the registered
|
||||||
`main_thread_forkserver_proc` spawn target successfully
|
`subint_forkserver_proc` spawn target successfully
|
||||||
forks a child, the child runs `_actor_child_main()` +
|
forks a child, the child runs `_actor_child_main()` +
|
||||||
completes IPC handshake + serves an RPC, and the parent
|
completes IPC handshake + serves an RPC, and the parent
|
||||||
reaps via `_ForkedProc.wait()` without regressing any of
|
reaps via `_ForkedProc.wait()` without regressing any of
|
||||||
|
|
@ -329,7 +326,7 @@ def test_main_thread_forkserver_spawn_basic(
|
||||||
deadline: float = 20.0
|
deadline: float = 20.0
|
||||||
with dump_on_hang(
|
with dump_on_hang(
|
||||||
seconds=deadline,
|
seconds=deadline,
|
||||||
path='/tmp/main_thread_forkserver_spawn_basic.dump',
|
path='/tmp/subint_forkserver_spawn_basic.dump',
|
||||||
):
|
):
|
||||||
trio.run(
|
trio.run(
|
||||||
partial(
|
partial(
|
||||||
|
|
@ -343,7 +340,7 @@ def test_main_thread_forkserver_spawn_basic(
|
||||||
# ----------------------------------------------------------------
|
# ----------------------------------------------------------------
|
||||||
# tier-4 DRAFT: orphaned-subactor SIGINT survivability
|
# tier-4 DRAFT: orphaned-subactor SIGINT survivability
|
||||||
#
|
#
|
||||||
# Motivating question: with `main_thread_forkserver`, the child's
|
# Motivating question: with `subint_forkserver`, the child's
|
||||||
# `trio.run()` lives on the fork-inherited worker thread which
|
# `trio.run()` lives on the fork-inherited worker thread which
|
||||||
# is NOT `threading.main_thread()` — so trio cannot install its
|
# is NOT `threading.main_thread()` — so trio cannot install its
|
||||||
# `signal.set_wakeup_fd`-based SIGINT handler. If the parent
|
# `signal.set_wakeup_fd`-based SIGINT handler. If the parent
|
||||||
|
|
@ -363,7 +360,7 @@ def test_main_thread_forkserver_spawn_basic(
|
||||||
# Cross-backend generalization (decide after this passes):
|
# Cross-backend generalization (decide after this passes):
|
||||||
# - applicable to any backend whose subactors are separate OS
|
# - applicable to any backend whose subactors are separate OS
|
||||||
# processes: `trio`, `mp_spawn`, `mp_forkserver`,
|
# processes: `trio`, `mp_spawn`, `mp_forkserver`,
|
||||||
# `main_thread_forkserver`.
|
# `subint_forkserver`.
|
||||||
# - NOT applicable to plain `subint` (subactors are in-process
|
# - NOT applicable to plain `subint` (subactors are in-process
|
||||||
# subinterpreters, no orphan child process to SIGINT).
|
# subinterpreters, no orphan child process to SIGINT).
|
||||||
# - move path: lift the harness script into
|
# - move path: lift the harness script into
|
||||||
|
|
@ -449,7 +446,7 @@ def _process_alive(pid: int) -> bool:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
# Known-gap test — `main_thread_forkserver` orphan-SIGINT
|
# Known-gap test — `subint_forkserver` orphan-SIGINT
|
||||||
# handling. See
|
# handling. See
|
||||||
# `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
|
# `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.
|
||||||
# `strict=True` so if a future fix closes the gap the
|
# `strict=True` so if a future fix closes the gap the
|
||||||
|
|
@ -474,12 +471,12 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
DRAFT — orphaned-subactor SIGINT survivability under the
|
DRAFT — orphaned-subactor SIGINT survivability under the
|
||||||
`main_thread_forkserver` backend.
|
`subint_forkserver` backend.
|
||||||
|
|
||||||
Sequence:
|
Sequence:
|
||||||
1. Spawn a harness subprocess that brings up a root
|
1. Spawn a harness subprocess that brings up a root
|
||||||
actor + one `sleep_forever` subactor via
|
actor + one `sleep_forever` subactor via
|
||||||
`main_thread_forkserver`.
|
`subint_forkserver`.
|
||||||
2. Read the harness's stdout for `PARENT_READY=<pid>`
|
2. Read the harness's stdout for `PARENT_READY=<pid>`
|
||||||
and `CHILD_PID=<pid>` markers (confirms the
|
and `CHILD_PID=<pid>` markers (confirms the
|
||||||
parent→child IPC handshake completed).
|
parent→child IPC handshake completed).
|
||||||
|
|
@ -527,7 +524,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
|
||||||
[
|
[
|
||||||
sys.executable,
|
sys.executable,
|
||||||
str(script_path),
|
str(script_path),
|
||||||
'main_thread_forkserver',
|
'subint_forkserver',
|
||||||
host,
|
host,
|
||||||
str(port),
|
str(port),
|
||||||
],
|
],
|
||||||
|
|
@ -580,7 +577,7 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
|
||||||
|
|
||||||
pytest.fail(
|
pytest.fail(
|
||||||
f'Orphan subactor (pid={child_pid}) did NOT exit '
|
f'Orphan subactor (pid={child_pid}) did NOT exit '
|
||||||
f'within 10s of SIGINT under `main_thread_forkserver` '
|
f'within 10s of SIGINT under `subint_forkserver` '
|
||||||
f'→ trio on non-main thread did not observe the '
|
f'→ trio on non-main thread did not observe the '
|
||||||
f'default CPython KeyboardInterrupt; backend needs '
|
f'default CPython KeyboardInterrupt; backend needs '
|
||||||
f'explicit SIGINT plumbing.'
|
f'explicit SIGINT plumbing.'
|
||||||
|
|
@ -603,50 +600,3 @@ def test_orphaned_subactor_sigint_cleanup_DRAFT(
|
||||||
proc.wait(timeout=2.0)
|
proc.wait(timeout=2.0)
|
||||||
except subprocess.TimeoutExpired:
|
except subprocess.TimeoutExpired:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
# ----------------------------------------------------------------
|
|
||||||
# regression guard: variant-2 (`subint_forkserver`) placeholder
|
|
||||||
# MUST raise `NotImplementedError` today — guards against future
|
|
||||||
# commits accidentally re-aliasing the key to the variant-1
|
|
||||||
# coroutine (which was a transient state during the rename).
|
|
||||||
# ----------------------------------------------------------------
|
|
||||||
def test_subint_forkserver_key_errors_cleanly() -> None:
|
|
||||||
'''
|
|
||||||
`--spawn-backend=subint_forkserver` is reserved for the
|
|
||||||
eventual variant-2 (subint-isolated child runtime)
|
|
||||||
backend, gated on jcrist/msgspec#1026 unblocking PEP 684
|
|
||||||
isolated-mode subints upstream.
|
|
||||||
|
|
||||||
Until that lands, the dispatch entry MUST raise
|
|
||||||
`NotImplementedError` immediately rather than silently
|
|
||||||
aliasing to `main_thread_forkserver_proc`. Verify the
|
|
||||||
error message also surfaces both the working-backend
|
|
||||||
pointer and the upstream-blocker ref so an operator
|
|
||||||
arriving at the error has somewhere to go.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import asyncio
|
|
||||||
from tractor.spawn._spawn import _methods
|
|
||||||
|
|
||||||
proc = _methods['subint_forkserver']
|
|
||||||
with pytest.raises(NotImplementedError) as ei:
|
|
||||||
# signature args match `main_thread_forkserver_proc`'s
|
|
||||||
# — the stub raises before touching them so dummy
|
|
||||||
# values are fine.
|
|
||||||
asyncio.run(
|
|
||||||
proc(
|
|
||||||
'x', None, None, {}, [],
|
|
||||||
('127.0.0.1', 0), {},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
msg: str = str(ei.value)
|
|
||||||
assert 'main_thread_forkserver' in msg, (
|
|
||||||
f'stub error msg should redirect to the working '
|
|
||||||
f'variant-1 backend; got: {msg!r}'
|
|
||||||
)
|
|
||||||
assert 'msgspec#1026' in msg or '1026' in msg, (
|
|
||||||
f'stub error msg should reference the upstream '
|
|
||||||
f'blocker (jcrist/msgspec#1026); got: {msg!r}'
|
|
||||||
)
|
|
||||||
|
|
@ -38,7 +38,6 @@ Two empirical CPython properties drive the design:
|
||||||
the forked child otherwise (`Fatal Python error: not main
|
the forked child otherwise (`Fatal Python error: not main
|
||||||
interpreter`). Full source-level walkthrough:
|
interpreter`). Full source-level walkthrough:
|
||||||
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
|
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`.
|
||||||
|
|
||||||
2. **`os.fork()` from a regular `threading.Thread` attached to
|
2. **`os.fork()` from a regular `threading.Thread` attached to
|
||||||
the *main* interpreter — i.e. a worker thread that has never
|
the *main* interpreter — i.e. a worker thread that has never
|
||||||
entered a subint — works cleanly.** Empirically validated
|
entered a subint — works cleanly.** Empirically validated
|
||||||
|
|
@ -87,11 +86,9 @@ costs:
|
||||||
|
|
||||||
- **Sidecar lifecycle**: a second long-lived process per
|
- **Sidecar lifecycle**: a second long-lived process per
|
||||||
parent, with its own start/stop/health-check semantics.
|
parent, with its own start/stop/health-check semantics.
|
||||||
|
|
||||||
- **IPC overhead per spawn**: every actor-spawn round-trips
|
- **IPC overhead per spawn**: every actor-spawn round-trips
|
||||||
an `mp` request message through a unix socket before any
|
an `mp` request message through a unix socket before any
|
||||||
child code runs.
|
child code runs.
|
||||||
|
|
||||||
- **State isolation by process boundary**: the sidecar can't
|
- **State isolation by process boundary**: the sidecar can't
|
||||||
share parent state at all — every spawn is a "cold" child
|
share parent state at all — every spawn is a "cold" child
|
||||||
re-importing modules from disk.
|
re-importing modules from disk.
|
||||||
|
|
@ -109,7 +106,6 @@ For the full variant-2 picture see
|
||||||
1) we already get costs 1 + 2 collapsed; cost 3 will land
|
1) we already get costs 1 + 2 collapsed; cost 3 will land
|
||||||
when msgspec#1026 unblocks isolated-mode subints.
|
when msgspec#1026 unblocks isolated-mode subints.
|
||||||
|
|
||||||
|
|
||||||
What survives the fork? — POSIX semantics
|
What survives the fork? — POSIX semantics
|
||||||
-----------------------------------------
|
-----------------------------------------
|
||||||
|
|
||||||
|
|
@ -117,58 +113,33 @@ A natural worry when forking from a parent that's running
|
||||||
`trio.run()` on another thread: does that trio thread (and
|
`trio.run()` on another thread: does that trio thread (and
|
||||||
any other threads in the parent) keep running in the child?
|
any other threads in the parent) keep running in the child?
|
||||||
|
|
||||||
**No** — but with a precise meaning that's worth pinning
|
**No.** POSIX `fork()` only preserves the *calling* thread
|
||||||
down, since the canonical trio framing
|
in the child. Every other thread in the parent — trio's
|
||||||
([python-trio/trio#1614](https://github.com/python-trio/trio/issues/1614))
|
runner thread, any `to_thread` cache threads, anything else
|
||||||
puts it the opposite-sounding way:
|
— is gone the instant `fork()` returns in the child.
|
||||||
|
|
||||||
> If you use `fork()` in a process with multiple threads,
|
|
||||||
> all the other thread stacks are just leaked: there's
|
|
||||||
> nothing else you can reasonably do with them.
|
|
||||||
|
|
||||||
Both statements describe the same POSIX reality from
|
|
||||||
opposite sides:
|
|
||||||
|
|
||||||
- **Execution-side ("gone")**: POSIX `fork()` only
|
|
||||||
preserves the *calling* thread as a runnable thread in
|
|
||||||
the child. Every other thread in the parent — trio's
|
|
||||||
runner thread, any `to_thread` cache threads, anything
|
|
||||||
else — never executes another instruction post-fork.
|
|
||||||
|
|
||||||
- **Memory-side ("leaked")**: those non-running threads'
|
|
||||||
*stacks* and per-thread heap structures are still
|
|
||||||
COW-inherited into the child's address space. They
|
|
||||||
persist as orphaned bytes with no owning thread, no
|
|
||||||
scheduler entry, and no way for the child to clean
|
|
||||||
them up — hence trio's word "leaked".
|
|
||||||
|
|
||||||
Concretely, after the forkserver worker calls `os.fork()`:
|
Concretely, after the forkserver worker calls `os.fork()`:
|
||||||
|
|
||||||
| thread | parent | child (executing) | child (memory) |
|
| thread | parent | child |
|
||||||
|---------------------|-----------|-------------------|-----------------------------|
|
|-----------------------|-----------|---------------|
|
||||||
| forkserver worker | continues | sole survivor | live stack |
|
| forkserver worker | continues | sole survivor |
|
||||||
| `trio.run()` thread | continues | not running | leaked stack (zombie bytes) |
|
| `trio.run()` thread | continues | gone |
|
||||||
| any other thread | continues | not running | leaked stack (zombie bytes) |
|
| any other thread | continues | gone |
|
||||||
|
|
||||||
The forkserver worker becomes the new "main" execution
|
The forkserver worker becomes the new "main" execution
|
||||||
context in the child; `trio.run()` and every other parent
|
context in the child; `trio.run()` and every other parent
|
||||||
thread never executes a single instruction post-fork.
|
thread never executes a single instruction post-fork in the
|
||||||
Their stack memory rides along as inert COW pages until
|
child.
|
||||||
the child's fresh `trio.run()` boots and overwrites/GCs
|
|
||||||
it (or until the child `exec()`s and discards the entire
|
|
||||||
image).
|
|
||||||
|
|
||||||
This is exactly *why* `os.fork()` is delegated to a
|
This is exactly *why* `os.fork()` is delegated to a
|
||||||
dedicated worker thread that has provably never entered
|
dedicated worker thread that has provably never entered
|
||||||
trio: we want that trio-free thread to be the surviving
|
trio: we want that trio-free thread to be the surviving
|
||||||
*executing* thread in the child, with the leaked trio
|
one in the child.
|
||||||
stack reduced to inert COW pages we don't touch.
|
|
||||||
|
|
||||||
The leaked-stack residue is one slice of the broader
|
That said, dead-thread *artifacts* still cross the fork
|
||||||
"fork in a multithreaded program is dangerous" hazard
|
boundary (canonical "fork in a multithreaded program is
|
||||||
class (see `man pthread_atfork`). Other dead-thread
|
dangerous" — see `man pthread_atfork`). What persists, and
|
||||||
artifacts that cross the fork boundary, and how we handle
|
how we handle each:
|
||||||
each:
|
|
||||||
|
|
||||||
- **Inherited file descriptors** — the dead trio thread's
|
- **Inherited file descriptors** — the dead trio thread's
|
||||||
epoll fd, signal-wakeup-fd, eventfds, sockets, IPC
|
epoll fd, signal-wakeup-fd, eventfds, sockets, IPC
|
||||||
|
|
@ -177,20 +148,16 @@ each:
|
||||||
`_close_inherited_fds()` in the child prelude — walks
|
`_close_inherited_fds()` in the child prelude — walks
|
||||||
`/proc/self/fd` and closes everything except stdio +
|
`/proc/self/fd` and closes everything except stdio +
|
||||||
the channel pipe to the forkserver.
|
the channel pipe to the forkserver.
|
||||||
|
|
||||||
- **Memory image** — trio's internal data structures
|
- **Memory image** — trio's internal data structures
|
||||||
(scheduler, task queues, runner state) sit in COW
|
(scheduler, task queues, runner state) sit in COW
|
||||||
memory alongside the leaked stacks above. Nobody's
|
memory but nobody's executing them. Get GC'd /
|
||||||
executing them; they get GC'd / overwritten when the
|
overwritten when the child's fresh `trio.run()` boots.
|
||||||
child's fresh `trio.run()` boots.
|
|
||||||
|
|
||||||
- **Python thread state** — handled automatically by
|
- **Python thread state** — handled automatically by
|
||||||
CPython. `PyOS_AfterFork_Child()` calls
|
CPython. `PyOS_AfterFork_Child()` calls
|
||||||
`_PyThreadState_DeleteExceptCurrent()`, so dead
|
`_PyThreadState_DeleteExceptCurrent()`, so dead
|
||||||
`PyThreadState` objects are cleaned and
|
`PyThreadState` objects are cleaned and
|
||||||
`threading.enumerate()` returns just the surviving
|
`threading.enumerate()` returns just the surviving
|
||||||
thread.
|
thread.
|
||||||
|
|
||||||
- **User-level locks (`threading.Lock`)** —
|
- **User-level locks (`threading.Lock`)** —
|
||||||
held-by-dead-thread state is the canonical fork hazard.
|
held-by-dead-thread state is the canonical fork hazard.
|
||||||
Not an issue in practice for tractor: trio doesn't hold
|
Not an issue in practice for tractor: trio doesn't hold
|
||||||
|
|
@ -199,7 +166,6 @@ each:
|
||||||
either direction). CPython's GIL is auto-reset by the
|
either direction). CPython's GIL is auto-reset by the
|
||||||
fork callback.
|
fork callback.
|
||||||
|
|
||||||
|
|
||||||
FYI: how this dodges the `trio.run()` × `fork()` hazards
|
FYI: how this dodges the `trio.run()` × `fork()` hazards
|
||||||
--------------------------------------------------------
|
--------------------------------------------------------
|
||||||
|
|
||||||
|
|
@ -217,16 +183,13 @@ design dodges each class explicitly:
|
||||||
reader. *Dodge*: the inherited wakeup-fd is closed by
|
reader. *Dodge*: the inherited wakeup-fd is closed by
|
||||||
`_close_inherited_fds()`, then the child's own
|
`_close_inherited_fds()`, then the child's own
|
||||||
`trio.run()` installs a fresh one.
|
`trio.run()` installs a fresh one.
|
||||||
|
|
||||||
- **`epoll`/`kqueue` instance**: trio's I/O backend holds
|
- **`epoll`/`kqueue` instance**: trio's I/O backend holds
|
||||||
one. Inherited as a dead fd; same fix as above.
|
one. Inherited as a dead fd; same fix as above.
|
||||||
|
|
||||||
- **Threadpool cache threads** (`trio.to_thread`): worker
|
- **Threadpool cache threads** (`trio.to_thread`): worker
|
||||||
threads with cached tstate. Don't exist in the child
|
threads with cached tstate. Don't exist in the child
|
||||||
(POSIX); cache state is meaningless garbage that gets
|
(POSIX); cache state is meaningless garbage that gets
|
||||||
reset when the child's trio.run() initializes its own
|
reset when the child's trio.run() initializes its own
|
||||||
thread cache.
|
thread cache.
|
||||||
|
|
||||||
- **Cancel scopes / nurseries / open `trio.Process` /
|
- **Cancel scopes / nurseries / open `trio.Process` /
|
||||||
open sockets**: these are trio-runtime objects, not
|
open sockets**: these are trio-runtime objects, not
|
||||||
kernel objects. The runtime that owns them is gone in
|
kernel objects. The runtime that owns them is gone in
|
||||||
|
|
@ -234,11 +197,9 @@ design dodges each class explicitly:
|
||||||
in COW memory and get overwritten as the child runs.
|
in COW memory and get overwritten as the child runs.
|
||||||
Inherited *kernel* fds those objects wrapped (sockets,
|
Inherited *kernel* fds those objects wrapped (sockets,
|
||||||
proc pipes) are caught by `_close_inherited_fds()`.
|
proc pipes) are caught by `_close_inherited_fds()`.
|
||||||
|
|
||||||
- **`atexit` handlers**: trio doesn't register any that
|
- **`atexit` handlers**: trio doesn't register any that
|
||||||
would mis-fire post-fork; trio's lifetime-stack is
|
would mis-fire post-fork; trio's lifetime-stack is
|
||||||
all `with`-block-scoped and dies with the runner.
|
all `with`-block-scoped and dies with the runner.
|
||||||
|
|
||||||
- **Foreign-language I/O state** (libcurl, OpenSSL session
|
- **Foreign-language I/O state** (libcurl, OpenSSL session
|
||||||
caches, etc.): out of scope — same hazard as any
|
caches, etc.): out of scope — same hazard as any
|
||||||
fork-without-exec; users layering those on top of
|
fork-without-exec; users layering those on top of
|
||||||
|
|
@ -250,7 +211,6 @@ isolation + `_close_inherited_fds()` cleanup gives the
|
||||||
forked child a clean trio environment. Everything else
|
forked child a clean trio environment. Everything else
|
||||||
falls under the standard fork-without-exec disclaimer.
|
falls under the standard fork-without-exec disclaimer.
|
||||||
|
|
||||||
|
|
||||||
Implementation status
|
Implementation status
|
||||||
---------------------
|
---------------------
|
||||||
|
|
||||||
|
|
@ -271,11 +231,10 @@ follow-up) including the
|
||||||
|
|
||||||
Still-open work (tracked on tractor #379):
|
Still-open work (tracked on tractor #379):
|
||||||
|
|
||||||
- [ ] no cancellation / hard-kill stress coverage yet
|
- no cancellation / hard-kill stress coverage yet
|
||||||
(counterpart to `tests/test_subint_cancellation.py` for
|
(counterpart to `tests/test_subint_cancellation.py` for
|
||||||
the plain `subint` backend),
|
the plain `subint` backend),
|
||||||
|
- `child_sigint='trio'` mode (flag scaffolded below; default
|
||||||
- [ ] `child_sigint='trio'` mode (flag scaffolded below; default
|
|
||||||
is `'ipc'`). Originally intended as a manual SIGINT →
|
is `'ipc'`). Originally intended as a manual SIGINT →
|
||||||
trio-cancel bridge, but investigation showed trio's
|
trio-cancel bridge, but investigation showed trio's
|
||||||
handler IS already correctly installed in the fork-child
|
handler IS already correctly installed in the fork-child
|
||||||
|
|
@ -328,24 +287,21 @@ See also
|
||||||
- `tractor.spawn._subint_forkserver` — variant-2 placeholder
|
- `tractor.spawn._subint_forkserver` — variant-2 placeholder
|
||||||
module; reserved for the future subint-isolated-child
|
module; reserved for the future subint-isolated-child
|
||||||
runtime once jcrist/msgspec#1026 unblocks.
|
runtime once jcrist/msgspec#1026 unblocks.
|
||||||
|
|
||||||
- `tractor.spawn._subint_fork` — the stub for the
|
- `tractor.spawn._subint_fork` — the stub for the
|
||||||
fork-from-non-main-subint strategy that DIDN'T work (kept
|
fork-from-non-main-subint strategy that DIDN'T work (kept
|
||||||
in-tree as documentation of the attempt + the CPython-level
|
in-tree as documentation of the attempt + the CPython-level
|
||||||
block).
|
block).
|
||||||
|
|
||||||
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
|
- `ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
|
||||||
— CPython source walkthrough of why fork-from-subint is dead.
|
— CPython source walkthrough of why fork-from-subint is dead.
|
||||||
|
|
||||||
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
|
- `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`
|
||||||
— standalone feasibility check (delegates to this module
|
— standalone feasibility check (delegates to this module
|
||||||
for the primitives it exercises).
|
for the primitives it exercises).
|
||||||
|
|
||||||
'''
|
'''
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
import errno
|
|
||||||
import os
|
import os
|
||||||
import signal
|
import signal
|
||||||
|
import sys
|
||||||
import threading
|
import threading
|
||||||
from functools import partial
|
from functools import partial
|
||||||
from typing import (
|
from typing import (
|
||||||
|
|
@ -368,8 +324,8 @@ from tractor.runtime._portal import Portal
|
||||||
from ._spawn import (
|
from ._spawn import (
|
||||||
cancel_on_completion,
|
cancel_on_completion,
|
||||||
soft_kill,
|
soft_kill,
|
||||||
wait_for_peer_or_proc_death,
|
|
||||||
)
|
)
|
||||||
|
from ._subint import _has_subints
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from tractor.discovery._addr import UnwrappedAddress
|
from tractor.discovery._addr import UnwrappedAddress
|
||||||
|
|
@ -467,24 +423,9 @@ def _close_inherited_fds(
|
||||||
try:
|
try:
|
||||||
os.close(fd)
|
os.close(fd)
|
||||||
closed += 1
|
closed += 1
|
||||||
except OSError as oserr:
|
except OSError:
|
||||||
# `EBADF` is the benign-and-expected case: the
|
# fd was already closed (race with listdir) or otherwise
|
||||||
# `os.listdir('/proc/self/fd')` call above itself
|
# unclosable — either is fine.
|
||||||
# opens a transient dirfd that ends up in
|
|
||||||
# `candidates`, then auto-closes before this loop
|
|
||||||
# reaches it. Same for any fd whose Python wrapper
|
|
||||||
# was GC'd between `listdir` and `os.close`.
|
|
||||||
# Suppress at debug-level — surfacing every
|
|
||||||
# EBADF as a full traceback (prior `log.exception`
|
|
||||||
# behavior) drowned the post-fork log channel.
|
|
||||||
if oserr.errno == errno.EBADF:
|
|
||||||
log.debug(
|
|
||||||
f'Skip already-closed inherited fd {fd!r} '
|
|
||||||
f'(EBADF, benign race with listdir)\n'
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
# Other errnos (EIO / EPERM / EINTR / ...) are
|
|
||||||
# genuinely unexpected — keep the loud surface.
|
|
||||||
log.exception(
|
log.exception(
|
||||||
f'Failed to close inherited fd in child ??\n'
|
f'Failed to close inherited fd in child ??\n'
|
||||||
f'{fd!r}\n'
|
f'{fd!r}\n'
|
||||||
|
|
@ -761,26 +702,6 @@ class _ForkedProc:
|
||||||
self._pidfd = -1
|
self._pidfd = -1
|
||||||
return self._returncode
|
return self._returncode
|
||||||
|
|
||||||
def terminate(self) -> None:
|
|
||||||
'''
|
|
||||||
OS-level `SIGTERM` to the child. Swallows
|
|
||||||
`ProcessLookupError` (already dead).
|
|
||||||
|
|
||||||
Mirrors `trio.Process.terminate()` /
|
|
||||||
`multiprocessing.Process.terminate()` — sends SIGTERM
|
|
||||||
(graceful, allows the child a chance to clean up via
|
|
||||||
signal-handlers) rather than SIGKILL. Used by
|
|
||||||
`ActorNursery.cancel()`'s per-child escalation when
|
|
||||||
`Portal.cancel_actor()` raises `ActorTooSlowError`,
|
|
||||||
and by the legacy `hard_kill=True` branch on the same
|
|
||||||
path.
|
|
||||||
|
|
||||||
'''
|
|
||||||
try:
|
|
||||||
os.kill(self.pid, signal.SIGTERM)
|
|
||||||
except ProcessLookupError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
def kill(self) -> None:
|
def kill(self) -> None:
|
||||||
'''
|
'''
|
||||||
OS-level `SIGKILL` to the child. Swallows
|
OS-level `SIGKILL` to the child. Swallows
|
||||||
|
|
@ -851,6 +772,13 @@ async def main_thread_forkserver_proc(
|
||||||
thread instead of `trio.lowlevel.open_process()`.
|
thread instead of `trio.lowlevel.open_process()`.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
if not _has_subints:
|
||||||
|
raise RuntimeError(
|
||||||
|
f'The {"main_thread_forkserver"!r} spawn backend '
|
||||||
|
f'requires Python 3.14+.\n'
|
||||||
|
f'Current runtime: {sys.version}'
|
||||||
|
)
|
||||||
|
|
||||||
# Backend-scoped config pulled from `proc_kwargs`. Using
|
# Backend-scoped config pulled from `proc_kwargs`. Using
|
||||||
# `proc_kwargs` (vs a first-class kwarg on this function)
|
# `proc_kwargs` (vs a first-class kwarg on this function)
|
||||||
# matches how other backends expose per-spawn tuning
|
# matches how other backends expose per-spawn tuning
|
||||||
|
|
@ -969,18 +897,7 @@ async def main_thread_forkserver_proc(
|
||||||
f' |_{proc}\n'
|
f' |_{proc}\n'
|
||||||
)
|
)
|
||||||
|
|
||||||
# race the handshake-wait against proc-death so a
|
event, chan = await ipc_server.wait_for_peer(uid)
|
||||||
# sub that dies during boot (e.g. crashed on import
|
|
||||||
# before reaching `_actor_child_main`, leaving a
|
|
||||||
# zombie + no cmdline) surfaces as `ActorFailure`
|
|
||||||
# instead of parking the spawning task forever on
|
|
||||||
# an unsignalled `_peer_connected[uid]` event.
|
|
||||||
event, chan = await wait_for_peer_or_proc_death(
|
|
||||||
ipc_server,
|
|
||||||
uid,
|
|
||||||
proc_wait=proc.wait,
|
|
||||||
proc_repr=repr(proc),
|
|
||||||
)
|
|
||||||
|
|
||||||
except trio.Cancelled:
|
except trio.Cancelled:
|
||||||
cancelled_during_spawn = True
|
cancelled_during_spawn = True
|
||||||
|
|
|
||||||
|
|
@ -43,7 +43,6 @@ from tractor.log import get_logger
|
||||||
from tractor.discovery._addr import (
|
from tractor.discovery._addr import (
|
||||||
UnwrappedAddress,
|
UnwrappedAddress,
|
||||||
)
|
)
|
||||||
from .._exceptions import ActorFailure
|
|
||||||
from ._reap import unlink_uds_bind_addrs
|
from ._reap import unlink_uds_bind_addrs
|
||||||
from tractor.runtime._portal import Portal
|
from tractor.runtime._portal import Portal
|
||||||
from tractor.runtime._runtime import Actor
|
from tractor.runtime._runtime import Actor
|
||||||
|
|
@ -83,15 +82,11 @@ SpawnMethodKey = Literal[
|
||||||
# runtime, exactly like `trio_proc` but via fork instead
|
# runtime, exactly like `trio_proc` but via fork instead
|
||||||
# of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
|
# of subproc-exec. See `tractor.spawn._main_thread_forkserver`.
|
||||||
'main_thread_forkserver',
|
'main_thread_forkserver',
|
||||||
# Variant-2: same fork machinery as `main_thread_forkserver`
|
# RESERVED for the future variant-2 subint-isolated-child
|
||||||
# but the child enters a sub-interpreter to host its
|
# runtime — gated on jcrist/msgspec#1026 + PEP 684. Today
|
||||||
# `trio.run()`. Gated on jcrist/msgspec#1026 unblocking
|
# this key aliases to `main_thread_forkserver_proc`; once
|
||||||
# PEP 684 isolated-mode subints upstream — until then
|
# the upstream unblocks land it'll dispatch to the
|
||||||
# `subint_forkserver_proc` is a clean `NotImplementedError`
|
# subint-hosted-trio impl. See
|
||||||
# stub pointing at variant-1 (`main_thread_forkserver`) +
|
|
||||||
# the upstream blocker. The key is reserved here (not just
|
|
||||||
# aliased to variant-1) so once upstream lands the impl can
|
|
||||||
# flip in-place without API churn. See
|
|
||||||
# `tractor.spawn._subint_forkserver`.
|
# `tractor.spawn._subint_forkserver`.
|
||||||
'subint_forkserver',
|
'subint_forkserver',
|
||||||
]
|
]
|
||||||
|
|
@ -111,71 +106,6 @@ else:
|
||||||
await trio.lowlevel.wait_readable(proc.sentinel)
|
await trio.lowlevel.wait_readable(proc.sentinel)
|
||||||
|
|
||||||
|
|
||||||
async def wait_for_peer_or_proc_death(
|
|
||||||
ipc_server,
|
|
||||||
uid: tuple[str, str],
|
|
||||||
# TODO? not not types?
|
|
||||||
proc_wait: 'Callable[[], Awaitable]',
|
|
||||||
proc_repr: str = '',
|
|
||||||
|
|
||||||
) -> 'tuple[trio.Event, Channel]':
|
|
||||||
'''
|
|
||||||
Race `IPCServer.wait_for_peer(uid)` against the sub-proc's
|
|
||||||
own `.wait()` coroutine. Whichever completes first cancels
|
|
||||||
the other.
|
|
||||||
|
|
||||||
Used by every spawn-backend to detect a sub-actor that
|
|
||||||
*dies during boot* before completing the parent-handshake-
|
|
||||||
callback (e.g. crashed on import, exec'd-out, kernel-killed
|
|
||||||
pre-`_actor_child_main`). Without this race, the
|
|
||||||
handshake-wait — backed by an unsignalled `trio.Event` —
|
|
||||||
parks the spawning task forever and leaves the dead child
|
|
||||||
as a zombie since nobody calls `proc.wait()` to reap.
|
|
||||||
|
|
||||||
On normal handshake-complete: returns `(event, chan)`
|
|
||||||
identical to a bare `wait_for_peer`.
|
|
||||||
|
|
||||||
On proc-death-first: raises `ActorFailure` carrying the
|
|
||||||
proc's exit code, allowing the supervisor to surface a
|
|
||||||
clean error rather than hanging indefinitely.
|
|
||||||
|
|
||||||
`proc_wait` is a 0-arg async callable returning the proc's
|
|
||||||
exit-status — kept generic so each backend can pass its
|
|
||||||
own (`trio.Process.wait`, `_ForkedProc.wait`,
|
|
||||||
`proc_waiter(mp.Process)`, etc.).
|
|
||||||
|
|
||||||
`proc_repr` is an optional string used in the
|
|
||||||
`ActorFailure` message for diag.
|
|
||||||
|
|
||||||
'''
|
|
||||||
result: dict = {}
|
|
||||||
|
|
||||||
async def _await_handshake():
|
|
||||||
event, chan = await ipc_server.wait_for_peer(uid)
|
|
||||||
result['handshake'] = (event, chan)
|
|
||||||
boot_n.cancel_scope.cancel()
|
|
||||||
|
|
||||||
async def _await_death():
|
|
||||||
rc = await proc_wait()
|
|
||||||
result['died'] = rc
|
|
||||||
boot_n.cancel_scope.cancel()
|
|
||||||
|
|
||||||
async with trio.open_nursery() as boot_n:
|
|
||||||
boot_n.start_soon(_await_handshake)
|
|
||||||
boot_n.start_soon(_await_death)
|
|
||||||
|
|
||||||
if 'handshake' in result:
|
|
||||||
return result['handshake']
|
|
||||||
|
|
||||||
# only reached if proc-death won the race
|
|
||||||
raise ActorFailure(
|
|
||||||
f'Sub-actor {uid!r} died during boot '
|
|
||||||
f'(rc={result.get("died")!r}) before completing '
|
|
||||||
f'parent-handshake.\n'
|
|
||||||
f' proc: {proc_repr}'
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def try_set_start_method(
|
def try_set_start_method(
|
||||||
key: SpawnMethodKey
|
key: SpawnMethodKey
|
||||||
|
|
||||||
|
|
@ -208,15 +138,13 @@ def try_set_start_method(
|
||||||
case 'mp_spawn':
|
case 'mp_spawn':
|
||||||
_ctx = mp.get_context('spawn')
|
_ctx = mp.get_context('spawn')
|
||||||
|
|
||||||
case (
|
case 'trio':
|
||||||
'trio'
|
|
||||||
| 'main_thread_forkserver'
|
|
||||||
):
|
|
||||||
_ctx = None
|
_ctx = None
|
||||||
|
|
||||||
case (
|
case (
|
||||||
'subint'
|
'subint'
|
||||||
| 'subint_fork'
|
| 'subint_fork'
|
||||||
|
| 'main_thread_forkserver'
|
||||||
| 'subint_forkserver'
|
| 'subint_forkserver'
|
||||||
):
|
):
|
||||||
# All subint-family backends need no `mp.context`;
|
# All subint-family backends need no `mp.context`;
|
||||||
|
|
|
||||||
|
|
@ -18,18 +18,15 @@
|
||||||
Variant-2 (future) "subint forkserver" placeholder — reserved
|
Variant-2 (future) "subint forkserver" placeholder — reserved
|
||||||
for the eventual subint-isolated-child runtime variant.
|
for the eventual subint-isolated-child runtime variant.
|
||||||
|
|
||||||
> **Status:** reserved key, stub impl. Today
|
> **Status:** placeholder. Today
|
||||||
> `--spawn-backend=subint_forkserver` raises a clean
|
> `--spawn-backend=subint_forkserver` aliases to
|
||||||
> `NotImplementedError` from `subint_forkserver_proc()`
|
> `main_thread_forkserver_proc` (variant 1, see
|
||||||
> below, pointing at variant-1
|
> `tractor.spawn._main_thread_forkserver`). A follow-up commit
|
||||||
> (`--spawn-backend=main_thread_forkserver`, see
|
> in this PR series flips the alias to a `NotImplementedError`
|
||||||
> `tractor.spawn._main_thread_forkserver`) and the upstream
|
> stub reserving the `'subint_forkserver'` key for the literal
|
||||||
> blocker
|
> subint-hosted-child variant once
|
||||||
> ([jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)).
|
> [jcrist/msgspec#1026](https://github.com/jcrist/msgspec/issues/1026)
|
||||||
> The key is reserved here (not aliased to variant-1) so the
|
> unblocks PEP 684 isolated-mode subints upstream.
|
||||||
> literal subint-hosted-child impl can flip in-place once
|
|
||||||
> msgspec#1026 unblocks PEP 684 isolated-mode subints
|
|
||||||
> upstream — no API churn at the call site.
|
|
||||||
|
|
||||||
Future arch — what subints would buy us
|
Future arch — what subints would buy us
|
||||||
---------------------------------------
|
---------------------------------------
|
||||||
|
|
@ -157,6 +154,21 @@ from trio import TaskStatus
|
||||||
from tractor.log import get_logger
|
from tractor.log import get_logger
|
||||||
from ._subint import _has_subints
|
from ._subint import _has_subints
|
||||||
|
|
||||||
|
# Backward-compat re-exports of the fork primitives whose
|
||||||
|
# canonical home is now `_main_thread_forkserver`. Kept here
|
||||||
|
# transiently so existing
|
||||||
|
# `from tractor.spawn._subint_forkserver import ...` callsites
|
||||||
|
# in the tests + the conc-anal smoketest keep resolving;
|
||||||
|
# dropped once a follow-up commit migrates those imports to
|
||||||
|
# the new module.
|
||||||
|
from ._main_thread_forkserver import (
|
||||||
|
_close_inherited_fds as _close_inherited_fds,
|
||||||
|
_format_child_exit as _format_child_exit,
|
||||||
|
fork_from_worker_thread as fork_from_worker_thread,
|
||||||
|
wait_child as wait_child,
|
||||||
|
_ForkedProc as _ForkedProc,
|
||||||
|
)
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from tractor.discovery._addr import UnwrappedAddress
|
from tractor.discovery._addr import UnwrappedAddress
|
||||||
from tractor.runtime._portal import Portal
|
from tractor.runtime._portal import Portal
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue