Experimental third spawn backend: use a fresh
sub-interpreter purely as a trio-free launchpad from
which to `os.fork()` + exec back into
`python -m tractor._child`. Per issue #379's
"fork()-workaround/hacks" thread.
Intent is to sidestep both,
- the trio+fork hazards hitting `trio_proc` (python- trio/trio#1614 et
al.), since the forking interp is guaranteed trio-free.
- the shared-GIL abandoned-thread hazards hitting `subint_proc`
(`ai/conc-anal/subint_sigint_starvation_issue.md`), since we don't
*stay* in the subint — it only lives long enough to call `os.fork()`
Downstream of the fork+exec, all the existing `trio_proc` plumbing is
reused verbatim: `ipc_server.wait_for_peer()`, `SpawnSpec`, `Portal`
yield, soft-kill.
Status: NOT wired up beyond scaffolding. The fn raises
`NotImplementedError` immediately; the `bootstrap` fork/exec string
builder and the `# TODO: orchestrate driver thread` block are kept
in-tree as deliberate dead code so the next iteration starts from
a concrete shape rather than a blank page.
Docstring calls out three open questions that need
empirical validation before wiring this up:
1. Does CPython permit `os.fork()` from a non-main
legacy subint?
2. Can the child stay fork-without-exec and
`trio.run()` directly from within the launchpad
subint?
3. How do `signal.set_wakeup_fd()` handlers and other
process-global state interact when the forking
thread is inside a subint?
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit eee79a0357)
The private `_interpreters` C module ships since 3.13, but that vintage
wedges under our `threading.Thread` + multi-trio usage pattern
—> `_interpreters.exec()` silently never makes progress. 3.14 fixes it.
So gate on the presence of the public `concurrent.interpreters` wrapper
(3.14+ only) even tho we still call into the private module at runtime.
Deats,
- `try_set_start_method('subint')` error msg + `_subint` module
docstring/comments rewritten to document the 3.14 floor and why 3.13
can't work.
- `_subint._has_subints` gate now imports `concurrent.interpreters` (not
`_interpreters`) as the version sentinel.
Also, reshuffle `pyproject.toml` deps into
per-python-version `[tool.uv.dependency-groups]`:
- `subints` group: `msgspec>=0.21.0`, py>=3.14
- `eventfd` group: `cffi>=1.17.1`, py>=3.13,<3.14
- `sync_pause` group: `greenback`, py>=3.13,<3.14
(was in `devx`; moved out bc no 3.14 yet)
Bump top-level `msgspec>=0.20.0` too.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 34d9d482e4)
(MTF-only portion: kept tractor/spawn/_spawn.py tractor/spawn/_subint.py)
Unbounded `trio.CancelScope(shield=True)` at the
soft-kill and thread-join sites can wedge the parent
trio loop indefinitely when a stuck subint ignores
portal-cancel (e.g. bc the IPC channel is already
broken).
Deats,
- add `_HARD_KILL_TIMEOUT` (3s) module-level const
- wrap both shield sites with
`trio.move_on_after()` so we abandon a stuck
subint after the deadline
- flip driver thread to `daemon=True` so proc-exit
also isn't blocked by a wedged subint
- pass `abandon_on_cancel=True` to
`trio.to_thread.run_sync(driver_thread.join)`
— load-bearing for `move_on_after` to actually
fire
- log warnings when either timeout triggers
- improve `InterpreterError` log msg to explain
the abandoned-thread scenario
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 99541feec7)
`trio.to_thread.run_sync(_interpreters.exec, ...)` runs `exec()` on
a cached worker thread — and when that thread is returned to the
cache after the subint's `trio.run()` exits, CPython still keeps
the subint's tstate attached to the (now idle) worker. Result: the
teardown `_interpreters.destroy(interp_id)` in the `finally` block
can block the parent's trio loop indefinitely, waiting for a tstate
release that only happens when the worker either picks up a new job
or exits.
Manifested as intermittent mid-suite hangs under
`--spawn-backend=subint` — caught by a
`faulthandler.dump_traceback_later()` showing the main thread stuck
in `_interpreters.destroy()` at `_subint.py:293` with only an idle
trio-cache worker as the other live thread.
Deats,
- drive the subint on a plain `threading.Thread` (not
`trio.to_thread`) so the OS thread truly exits after
`_interpreters.exec()` returns, releasing tstate and unblocking
destroy
- signal `subint_exited.set()` back to the parent trio loop from
the driver thread via `trio.from_thread.run_sync(...,
trio_token=...)` — capture the token at `subint_proc` entry
- swallow `trio.RunFinishedError` in that signal path for the case
where parent trio has already exited (proc teardown)
- in the teardown `finally`, off-load the sync
`driver_thread.join()` to `trio.to_thread.run_sync` (cache thread
w/ no subint tstate → safe) so we actually wait for the driver to
exit before `_interpreters.destroy()`
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 31cbd11a5b)
Expand the comment block above the `_interpreters`
import explaining *why* we use the private C mod
over `concurrent.interpreters`: the public API only
exposes PEP 734's `'isolated'` config which breaks
`msgspec` (missing PEP 684 slot). Add reference
links to PEP 734, PEP 684, cpython sources, and
the msgspec upstream tracker (jcrist/msgspec#563).
Also,
- update error msgs in both `_spawn.py` and
`_subint.py` to say "3.13+" (matching the actual
`_interpreters` availability) instead of "3.14+".
- tweak the mod docstring to reflect py3.13+
availability via the private C module.
Review: PR #444 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/444
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 8a8d01e076)
Replace the B.1 scaffold stub w/ a working spawn
flow driving PEP 734 sub-interpreters on dedicated
OS threads.
Deats,
- use private `_interpreters` C mod (not the public
`concurrent.interpreters` API) to get `'legacy'`
subint config — avoids PEP 684 C-ext compat
issues w/ `msgspec` and other deps missing the
`Py_mod_multiple_interpreters` slot
- bootstrap subint via code-string calling new
`_actor_child_main()` from `_child.py` (shared
entry for both CLI and subint backends)
- drive subint lifetime on an OS thread using
`trio.to_thread.run_sync(_interpreters.exec, ..)`
- full supervision lifecycle mirrors `trio_proc`:
`ipc_server.wait_for_peer()` → send `SpawnSpec`
→ yield `Portal` via `task_status.started()`
- graceful shutdown awaits the subint's inner
`trio.run()` completing; cancel path sends
`portal.cancel_actor()` then waits for thread
join before `_interpreters.destroy()`
Also,
- extract `_actor_child_main()` from `_child.py`
`__main__` block as callable entry shape bc the
subint needs it for code-string bootstrap
- add `"subint"` to the `_runtime.py` spawn-method
check so child accepts `SpawnSpec` over IPC
Prompt-IO: ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.md
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit b8f243e98d)
(MTF-only portion: kept ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.md ai/prompt-io/claude/20260417T124437Z_5cd6df5_prompt_io.raw.md tractor/spawn/_subint.py)
Land the scaffolding for a future sub-interpreter (PEP 734
`concurrent.interpreters`) actor spawn backend per issue #379. The
spawn flow itself is not yet implemented; `subint_proc()` raises a
placeholder `NotImplementedError` pointing at the tracking issue —
this commit only wires up the registry, the py-version gate, and
the harness.
Deats,
- bump `pyproject.toml` `requires-python` to `>=3.12, <3.15` and
list the `3.14` classifier — the new stdlib
`concurrent.interpreters` module only ships on 3.14
- extend `SpawnMethodKey = Literal[..., 'subint']`
- `try_set_start_method('subint')` grows a new `match` arm that
feature-detects the stdlib module and raises `RuntimeError` with
a clear banner on py<3.14
- `_methods` registers the new `subint_proc()` via the same
bottom-of-module late-import pattern used for `._trio` / `._mp`
Also,
- new `tractor/spawn/_subint.py` — top-level `try: from concurrent
import interpreters` guards `_has_subints: bool`; `subint_proc()`
signature mirrors `trio_proc`/`mp_proc` so the Phase B.2 impl can
drop in without touching the registry
- re-add `import sys` to `_spawn.py` (needed for the py-version msg
in the gate-error)
- `_testing.pytest.pytest_configure` wraps `try_set_start_method()`
in a `pytest.UsageError` handler so `--spawn-backend=subint` on
py<3.14 prints a clean banner instead of a traceback
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit d318f1f8f4)
(MTF-only portion: kept tractor/spawn/_spawn.py tractor/spawn/_subint.py)
Gud Boi