Sixth and final diagnostic pass — after all 4
cascade fixes landed (FD hygiene, pidfd wait,
`_parent_chan_cs` wiring, bounded peer-clear), the
actual last gate on
`test_nested_multierrors[subint_forkserver]`
turned out to be **pytest's default
`--capture=fd` stdout/stderr capture**, not
anything in the runtime cascade.
Empirical result: `pytest -s` → test PASSES in
6.20s. Default `--capture=fd` → hangs forever.
Mechanism: pytest replaces the parent's fds 1,2
with pipe write-ends it reads from. Fork children
inherit those pipes (since `_close_inherited_fds`
correctly preserves stdio). The error-propagation
cascade in a multi-level cancel test generates
7+ actors each logging multiple `RemoteActorError`
/ `ExceptionGroup` tracebacks — enough output to
fill Linux's 64KB pipe buffer. Writes block,
subactors can't progress, processes don't exit,
`_ForkedProc.wait` hangs.
Self-critical aside: I earlier tested w/ and w/o
`-s` and both hung, concluding "capture-pipe
ruled out". That was wrong — at that time fixes
1-4 weren't all in place, so the test was
failing at deeper levels long before reaching
the "produce lots of output" phase. Once the
cascade could actually tear down cleanly, enough
output flowed to hit the pipe limit. Order-of-
operations mistake: ruling something out based
on a test that was failing for a different
reason.
Deats,
- `subint_forkserver_test_cancellation_leak_issue
.md`: new section "Update — VERY late: pytest
capture pipe IS the final gate" w/ DIAG timeline
showing `trio.run` fully returns, diagnosis of
pipe-fill mechanism, retrospective on the
earlier wrong ruling-out, and fix direction
(redirect subactor stdout/stderr to `/dev/null`
in fork-child prelude, conditional on
pytest-detection or opt-in flag)
- `tests/test_cancellation.py`: skip-mark reason
rewritten to describe the capture-pipe gate
specifically; cross-refs the new doc section
- `tests/spawn/test_subint_forkserver.py`: the
orphan-SIGINT test regresses back to xfail.
Previously passed after the FD-hygiene fix,
but the new `wait_for_no_more_peers(
move_on_after=3.0)` bound in `async_main`'s
teardown added up to 3s latency, pushing
orphan-subactor exit past the test's 10s poll
window. Real fix: faster orphan-side teardown
OR extend poll window to 15s
No runtime code changes in this commit — just
test-mark adjustments + doc wrap-up.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit eceed29d4a)
(MTF-only portion: kept ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md tests/spawn/test_subint_forkserver.py)
Two coordinated improvements to the `subint_forkserver` backend:
1. Replace `trio.to_thread.run_sync(os.waitpid, ...,
abandon_on_cancel=False)` in `_ForkedProc.wait()`
with `trio.lowlevel.wait_readable(pidfd)`. The
prior version blocked a trio cache thread on a
sync syscall — outer cancel scopes couldn't
unwedge it when something downstream got stuck.
Same pattern `trio.Process.wait()` and
`proc_waiter` (the mp backend) already use.
2. Drop the `@pytest.mark.xfail(strict=True)` from
`test_orphaned_subactor_sigint_cleanup_DRAFT` —
the test now PASSES after 0cd0b633 (fork-child
FD scrub). Same root cause as the nested-cancel
hang: inherited IPC/trio FDs were poisoning the
child's event loop. Closing them lets SIGINT
propagation work as designed.
Deats,
- `_ForkedProc.__init__` opens a pidfd via
`os.pidfd_open(pid)` (Linux 5.3+, Python 3.9+)
- `wait()` parks on `trio.lowlevel.wait_readable()`,
then non-blocking `waitpid(WNOHANG)` to collect
the exit status (correct since the pidfd signal
IS the child-exit notification)
- `ChildProcessError` swallow handles the rare race
where someone else reaps first
- pidfd closed after `wait()` completes (one-shot
semantics) + `__del__` belt-and-braces for
unexpected-teardown paths
- test docstring's `@xfail` block replaced with a
`# NOTE` comment explaining the historical
context + cross-ref to the conc-anal doc; test
remains in place as a regression guard
The two changes are interdependent — the
cancellable `wait()` matters for the same nested-
cancel scenarios the FD scrub fixes, since the
original deadlock had trio cache workers wedged in
`os.waitpid` swallowing the outer cancel.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit c20b05e181)
Empirical follow-up to the xfail'd orphan-SIGINT test:
the hang is **not** "trio can't install a handler on a
non-main thread" (the original hypothesis from the
`child_sigint` scaffold commit). On py3.14:
- `threading.current_thread() is threading.main_thread()`
IS True post-fork — CPython re-designates the
fork-inheriting thread as "main" correctly
- trio's `KIManager` SIGINT handler IS installed in the
subactor (`signal.getsignal(SIGINT)` confirms)
- the kernel DOES deliver SIGINT to the thread
But `faulthandler` dumps show the subactor wedged in
`trio/_core/_io_epoll.py::get_events` — trio's
wakeup-fd mechanism (which turns SIGINT into an epoll-wake)
isn't firing. So the `except KeyboardInterrupt` at
`tractor/spawn/_entry.py::_trio_main:164` — the runtime's
intentional "KBI-as-OS-cancel" path — never fires.
Deats,
- new `ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`
(+385 LOC): full writeup — TL;DR, symptom reproducer,
the "intentional cancel path" the bug defeats,
diagnostic evidence (`faulthandler` output +
`getsignal` probe), ruled-out hypotheses
(non-main-thread issue, wakeup-fd inheritance,
KBI-as-trio-check-exception), and fix directions
- `test_orphaned_subactor_sigint_cleanup_DRAFT` xfail
`reason` + test docstring rewritten to match the
refined understanding — old wording blamed the
non-main-thread path, new wording points at the
`epoll_wait` wedge + cross-refs the new conc-anal doc
- `_subint_forkserver` module docstring's
`child_sigint='trio'` bullet updated: now notes trio's
handler is already correctly installed, so the flag may
end up a no-op / doc-only mode once the real root cause
is fixed
Closing the gap aligns with existing design intent (make
the already-designed "KBI-as-OS-cancel" behavior actually
fire), not a new feature.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit a72deef709)
Tier-4 test `test_orphaned_subactor_sigint_cleanup_DRAFT`
documents an empirical SIGINT-delivery gap in the
`subint_forkserver` backend: when the parent dies via
`SIGKILL` (no IPC `Portal.cancel_actor()` possible) and
`SIGINT` is sent to the orphan child, the child DOES NOT
unwind — CPython's default `KeyboardInterrupt` is delivered
to `threading.main_thread()`, whose tstate is dead in the
post-fork child bc fork inherited the worker thread, not
main. Trio running on the fork-inherited worker thread
therefore never observes the signal. Marked
`xfail(strict=True)` so the mark flips to XPASS→fail once
the backend grows explicit SIGINT plumbing.
Deats,
- harness runs the failure-mode sequence out-of-process:
1. harness subprocess runs a fresh Python script
that calls `try_set_start_method('subint_forkserver')`
then opens a root actor + one `sleep_forever` subactor
2. parse `PARENT_READY=<pid>` + `CHILD_PID=<pid>` markers
off harness `stdout` to confirm IPC handshake
completed
3. `SIGKILL` the parent, `proc.wait()` to reap the
zombie (otherwise `os.kill(pid, 0)` keeps reporting
it alive)
4. assert the child survived the parent-reap (i.e. was
actually orphaned, not reaped too) before moving on
5. `SIGINT` the orphan child, poll `os.kill(child_pid, 0)`
every 100ms for up to 10s
- supporting helpers: `_read_marker()` with per-proc
bytes-buffer to carry partial lines across calls,
`_process_alive()` liveness probe via `kill(pid, 0)`
- Linux-only via `platform.system() != 'Linux'` skip —
orphan-reparenting semantics don't generalize to
other platforms
- port offset (`reg_addr[1] + 17`) so the harness listener
doesn't race concurrently-running backend tests
- best-effort `finally:` cleanup: `SIGKILL` any still-alive
pids + `proc.kill()` + bounded `proc.wait()` to avoid
leaking orphans across the session
Also, tier-4 header comment documents the cross-backend
generalization path: applicable to any multi-process
backend (`trio`, `mp_spawn`, `mp_forkserver`,
`subint_forkserver`), NOT to plain `subint` (in-process
subints have no orphan OS-child). Move path: lift
harness into `tests/_orphan_harness.py`, parametrize on
session `_spawn_method`, add
`skipif _spawn_method == 'subint'`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 76605d5609)
Promote `_subint_forkserver` from primitives-only into a
registered spawn backend: `'subint_forkserver'` is now a
`SpawnMethodKey` literal, dispatched via `_methods` to
the new `subint_forkserver_proc()` target, feature-gated
under the existing `subint`-family py3.14+ case, and
selectable via `--spawn-backend=subint_forkserver`.
Deats,
- new `subint_forkserver_proc()` spawn target in
`_subint_forkserver`:
- mirrors `trio_proc()`'s supervision model — real OS
subprocess so `Portal.cancel_actor()` + `soft_kill()`
on graceful teardown, `os.kill(SIGKILL)` on hard-reap
(no `_interpreters.destroy()` race to fuss over bc the
child lives in its own process)
- only real diff from `trio_proc` is the spawn mechanism:
fork from a main-interp worker thread via
`fork_from_worker_thread()` (off-loaded to trio's
thread pool) instead of `trio.lowlevel.open_process()`
- child-side `_child_target` closure runs
`tractor._child._actor_child_main()` with
`spawn_method='trio'` — the child is a regular trio
actor, "subint_forkserver" names how the parent
spawned, not what the child runs
- new `_ForkedProc` class — thin `trio.Process`-compatible
shim around a raw OS pid: `.poll()` via
`waitpid(WNOHANG)`, async `.wait()` off-loaded to a trio
cache thread, `.kill()` via `SIGKILL`, `.returncode`
cached for repeat calls. `.stdin`/`.stdout`/`.stderr`
are `None` (fork-w/o-exec inherits parent FDs; we don't
marshal them) which matches `soft_kill()`'s `is not None`
guards
Also, new backend-tier test
`test_subint_forkserver_spawn_basic` drives the registered
backend end-to-end via `open_root_actor` + `open_nursery` +
`run_in_actor` w/ a trivial portal-RPC round-trip. Uses a
`forkserver_spawn_method` fixture to flip
`_spawn_method`/`_ctx` for the test's duration + restore on
teardown (so other session-level tests don't observe the
global flip). Test module docstring reworked to describe
the three tiers now covered: (1) primitive-level, (2)
parent-trio-driven primitives, (3) full registered backend.
Status: still-open work (tracked on `tractor#379`) doc'd
inline in the module docstring — no cancel/hard-kill stress
coverage yet, child-side subint-hosted root runtime still
future (gated on `msgspec#563`), thread-hygiene audit
pending the same unblock.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 26914fde75)
New pytest module `tests/spawn/test_subint_forkserver.py`
drives the forkserver primitives from inside a real
`trio.run()` in the parent — the runtime shape tractor will
actually use when we wire up a `subint_forkserver` spawn
backend proper. Complements the standalone no-trio-in-parent
`ai/conc-anal/subint_fork_from_main_thread_smoketest.py`.
Deats,
- new test pkg `tests/spawn/` (+ empty `__init__.py`)
- two tests, both `@pytest.mark.timeout(30, method='thread')`
for the GIL-hostage safety reason doc'd in
`ai/conc-anal/subint_sigint_starvation_issue.md`:
- `test_fork_from_worker_thread_via_trio` — parent-side
plumbing baseline. `trio.run()` off-loads forkserver
prims via `trio.to_thread.run_sync()` + asserts the
child reaps cleanly
- `test_fork_and_run_trio_in_child` — end-to-end: forked
child calls `run_subint_in_worker_thread()` with a
bootstrap str that does `trio.run()` in a fresh subint
- both tests wrap the inner `trio.run()` in a
`dump_on_hang()` for post-mortem if the outer
`pytest-timeout` fires
- intentionally NOT using `--spawn-backend` — the tests
drive the primitives directly rather than going through
tractor's spawn-method registry (which the forkserver
isn't plugged into yet)
Also, rename `run_trio_in_subint()` →
`run_subint_in_worker_thread()` for naming consistency with
the sibling `fork_from_worker_thread()`. The action is really
"host a subint on a worker thread", not specifically "run
trio" — trio just happens to be the typical payload.
Propagate the rename to the smoketest.
Further, add a "TODO — cleanup gated on msgspec PEP 684
support" section to the `_subint_forkserver` module
docstring: flags the dedicated-`threading.Thread` design as
potentially-revisable once isolated-mode subints are viable
in tractor. Cross-refs `msgspec#563` + `tractor#379` and
points at an audit-plan conc-anal doc we'll add next.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 25e400d526)
Add a hard process-level wall-clock bound on the two
known-hanging subint-backend tests so an unattended
suite run can't wedge indefinitely in either of the
hang classes doc'd in `ai/conc-anal/`.
Deats,
- New `testing` dep: `pytest-timeout>=2.3`.
- `test_stale_entry_is_deleted`:
`@pytest.mark.timeout(3, method='thread')`. The
`method='thread'` choice is deliberate —
`method='signal'` routes via `SIGALRM` which is
starved by the same GIL-hostage path that drops
`SIGINT` (see `subint_sigint_starvation_issue.md`),
so it'd never actually fire in the starvation case.
- `test_subint_non_checkpointing_child`: same
decorator, same reasoning (defense-in-depth over
the inner `trio.fail_after(15)`).
At timeout, `pytest-timeout` hard-kills the pytest
process itself — that's the intended behavior here;
the alternative is the suite never returning.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 189f4e3ffc)
(MTF-only portion: kept tests/test_subint_cancellation.py)
Classify and write up the two distinct hang modes hit during Phase
B subint bringup (issue #379) so future triage doesn't re-derive them
from scratch.
Deats, two new `ai/conc-anal/` docs,
- `subint_sigint_starvation_issue.md`: abandoned legacy-subint thread
+ shared GIL → main trio loop starves → signal-wakeup-fd pipe fills
→ `SIGINT` silently dropped (`strace` shows `write() = EAGAIN` on the
wakeup-fd). Un- Ctrl-C-able. Structurally a CPython limit; blocked on
`msgspec` PEP 684 (jcrist/msgspec#563)
- `subint_cancel_delivery_hang_issue.md`: parent-side trio task parks on
an orphaned IPC channel after subint teardown — no clean EOF delivered
to the waiting receive. Ctrl-C-able (main loop iterates fine); OUR bug
to fix. Candidate fix: explicit parent-side channel abort in
`subint_proc`'s hard-kill teardown
Cross-link the docs from their test reproducers,
- `test_stale_entry_is_deleted` (→ starvation class): wrap
`trio.run(main)` in `dump_on_hang(seconds=20)` so a future regression
captures a stack dump. Kept un- skipped so the dump file is
inspectable
- `test_subint_non_checkpointing_child` (→ delivery class): extend
docstring with a "KNOWN ISSUE" block pointing at the analysis
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 4a3254583b)
(MTF-only portion: kept ai/conc-anal/subint_cancel_delivery_hang_issue.md ai/conc-anal/subint_sigint_starvation_issue.md tests/test_subint_cancellation.py)
Lock in the escape-hatch machinery added to `tractor.spawn._subint`
during the Phase B.2/B.3 bringup (issue #379) so future stdlib
regressions or our own refactors don't silently re-introduce the
mid-suite hangs.
Deats,
- `test_subint_happy_teardown`: baseline — spawn a subactor, one portal
RPC, clean teardown. If this breaks, something's wrong unrelated to
the hard-kill shields.
- `test_subint_non_checkpointing_child`: cancel a subactor stuck in
a non-checkpointing Python loop (`threading.Event.wait()` releases the
GIL but never inserts a trio checkpoint). Validates the bounded-shield
+ daemon-driver-thread combo abandons the thread after
`_HARD_KILL_TIMEOUT`.
Every test is wrapped in `trio.fail_after()` for a deterministic
per-test wall-clock ceiling (an unbounded audit would defeat itself) and
arms `tractor.devx.dump_on_hang()` so a hang captures a stack dump
— pytest's stderr capture swallows `faulthandler` output by default.
Gated via `pytest.importorskip('concurrent.interpreters')` and
a module-level skip when `--spawn-backend` isn't `'subint'`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 2ed5e6a6e8)
Copilot's 2nd-pass review (PR #468) caught a regression I landed in
the uds fix: `UDSAddress.get_random()` put the per-call token AFTER
`@{pid}` (`no_runtime_*@{pid}.{token}.sock`), which breaks the
`tractor._testing._reap` matcher
`^(?P<name>.+)@(?P<pid>\d+)\.sock$` — so no-runtime orphan socks
stopped matching and never got reaped/attributed. Move the token
INTO the name (`{prefix}.{token}@{pid}.sock`) so the canonical
`@{pid}.sock` suffix stays intact for both that regex and the
`spawn._reap` reconstruction; also bump the token to 8 hex chars.
Also two robustness nits from the same review,
- `tests.conftest._measure_sustained_headroom()`: guard `frac <= 0`
before `1./frac` — a 0/parked-core freq read would
`ZeroDivisionError`, get swallowed by the broad `except` into a
1.0 (no-throttle), defeating the probe on the exact broken box it
should flag; read 0 as max throttle.
- `scripts/cpu-perf-check`: mark the burn procs `daemon=True` and
wrap sampling in `try/finally` so a Ctrl-C / error reaps them
instead of leaving stray CPU hogs.
Regressed-by: 09c50f49 (uds no-runtime token placed after `@pid`)
Found-via: Copilot review #4595803812 (`_testing._reap` regex)
Review: PR #468 (Copilot)
https://github.com/goodboy/tractor/pull/468#pullrequestreview-4595803812
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Finish TODO #2 from PR #468: the last raw `cpu_scaling_factor()`
call-sites still used the static-only check (blind to the
sustained-load power-cap). Point them at the `cpu_perf_headroom()`
SUPERSET — it calls `cpu_scaling_factor()` internally + the
session-cached throttle probe, so it's always >= the old factor
(never LESS headroom, so CI stays green).
Migrated,
- `test_spawning`: the `fail_after(1 * ...)` smoke deadline
(#465-added).
- `test_inter_peer_cancellation`: `this_fast` budget.
- `test_docs_examples`: example-run `timeout`.
- `test_resource_cache`: fan-out `timeout`.
`test_cancellation` already used `cpu_perf_headroom()`; only its
explanatory comments still name the static helper.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Both `_burn()` impls (the `cpu-perf-check` script + the
`_measure_sustained_headroom()` probe in `tests.conftest`) grew `x`
~x**2 per iter, so the loop quickly went bigint alloc/mul-bound — a
noisy CPU load + needless memory across N procs. Mask each step to
64-bit for a steady, fixed-width ALU burn (still pegs every core,
which is all the freq probe needs).
Also, `_read_mhz()` opened the sysfs freq files without a ctx-mgr;
`with open(...)` so the FD closes deterministically (matches the
script's own `_read()`).
Review: PR #468 (Copilot)
https://github.com/goodboy/tractor/pull/468
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
`cpu_perf_headroom()` is a strict SUPERSET of `cpu_scaling_factor()`
— it folds in static cpu-freq scaling + the slow-CI bump AND the
sustained-load power-cap throttle probe. Point the timing-deadline
call sites at it so they're robust to sustained throttling too (the
gremlin behind mass `trio` deadline-miss flakes),
- `test_legacy_one_way_streaming`: `time_quad_ex` cancel-deadline +
`test_a_quadruple_example`'s `this_fast` smoke bound.
- `test_cancellation`: `test_cancel_via_SIGINT_other_task` +
`test_fast_graceful_cancel_*` deadlines.
`cpu_perf_headroom >= cpu_scaling_factor` always, so never less
headroom (CI stays green); `cpu_scaling_factor()` remains the
internal static component.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Mass `trio` deadline-miss failures on byte-identical code turned
out to be a firmware/EC power-cap (AMD PPT/STAPM) clamping the
all-core sustained clock while every static knob (`governor`,
`scaling_max_freq`, EPP, platform-profile) still read "performance"
— invisible to the existing `cpu_scaling_factor()` check. See
`scripts/cpu-perf-check` + the
`ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`
notes.
Deats,
- add `_measure_sustained_headroom()` to `tests/conftest.py`: a
one-shot ~0.9s all-core burn (explicit `fork`-ctx `mp` procs)
sampling achieved-vs-max freq AFTER the boost window; under a 0.6
gate it returns the full inverse fraction (capped 4x), else 1.0;
best-effort 1.0 on non-linux or any error,
- add `cpu_perf_headroom()`: `max()` of the static scaling factor
and the (session-cached) sustained probe,
- inflate deadline budgets by it in `test_dynamic_pub_sub`, both
`test_clustering` cases, the
`test_multi_nested_subactors_error_through_nurseries` pexpect
waits + `test_nested_multierrors`,
- `xfail(strict=False)` `test_nested_multierrors` depth=3 under
throttle: the deep tree trips tractor's INTERNAL reap deadlines
(`soft_kill`/`hard_kill` `terminate_after=1.6`) minting a
`Cancelled` inside the runtime — not fixable by test-budget
inflation; auto-clears once the box un-throttles.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
`uds_transport_actor_tree.py` (new in 759d46dd) reliably fails on
macOS CI with a bare `returncode 1`. The example forces
`enable_transports=['uds']`, and UDS-on-macOS is otherwise
un-exercised by the matrix (there's no `tpt_proto=uds` macOS job),
so it's the first thing to surface a macOS UDS-path issue.
It passes cleanly on Linux, and the peer-cred paths are already
platform-guarded + graceful (`get_peer_info`/SO_PEERCRED on linux
vs `get_peer_pid`/LOCAL_PEERPID elsewhere) - so the crash is deeper
in the macOS UDS stack. The test swallows the subproc stderr (only
re-raises when the last line has "Error"), so the exact failure
can't be pinned from CI logs; needs a macOS box to root-cause.
Skip it on `_friggin_macos and ci_env` (same shape as the existing
`full_fledged_streaming_service` skip) so #460 lands green; the
macOS UDS root-cause is a follow-up.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
The `.uid`->`.aid` migration (prior commits) re-assembled the
legacy uid pair by hand as `(x.aid.name, x.aid.uuid)`.
`Aid.uid` (`msg/types.py`) is the non-deprecated canonical
property returning exactly that pair, so swap to `x.aid.uid`
everywhere — byte-identical, just DRY + consistent.
Covers the 5 review-flagged source sites (`msg._ops`,
`devx.debug._sync`, `experimental._pubsub`, `_exceptions`)
plus the ~22 unflagged identical sites (`_streaming` f-string
logs + 6 test modules); also tightens the `_exceptions`
`our_uid` annotation to `tuple[str, str]`.
Review: PR #469 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/469#pullrequestreview-4575979601
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
`test_cancel_via_SIGINT_other_task` opened its nursery with the
now-deprecated `strict_exception_groups=False`. Under strict EGs
(trio's default since 0.25) the SIGINT-induced `KeyboardInterrupt`
surfaces wrapped in a `BaseExceptionGroup` — alongside the child
tasks' `Cancelled`s, so it's MULTI-exc and `collapse_eg()` can't
fold it back to a bare KI (the `# why no work!?` the author hit).
Drop the deprecated flag, use a default (strict) nursery, and
assert the result is a bare `KeyboardInterrupt` OR a
`BaseExceptionGroup` whose `.subgroup(KeyboardInterrupt)` is
non-empty.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Mirror the core swap on the test side: every `Channel`/`Actor`
`.uid` access now reads the non-deprecated `.aid` struct's
`(.name, .uuid)` (or `.aid.name` for a `.uid[0]`), keeping the
compared / printed value byte-identical while silencing the
self-inflicted `DeprecationWarning`. Touches
`test_context_stream_semantics`, `test_spawning`, `test_local`,
`test_advanced_streaming`, `msg.test_ext_types_msgspec`,
`discovery.test_multi_program`, `test_infected_asyncio`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
`tests.test_cancellation` asserted on `ActorNursery.cancelled`,
which now warns + redirects to `.cancel_called`. Swap to the
non-deprecated property so the suite stops tripping its own
`DeprecationWarning`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
`tests.devx.test_tooling`'s end-of-tree `expect()` patterns use
a literal `\(` (pexpect regex), which Python 3.12+ flags as an
invalid escape sequence. Make them raw strings so the `\(` is
preserved verbatim and no `SyntaxWarning` fires.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Under `trio>=0.33`'s strict exception-groups a lone
`error_parent` arrives wrapped in a 1-exc `ExceptionGroup`
instead of collapsing to the bare exc, so it skips the
`except error_parent` arm and lands in the group arm — where
`is_multi_cancelled()` is (correctly) `False`, tripping the
assert. Also accept `beg.subgroup(error_parent) is not None`.
Review: PR #465 (copilot + self CI-triage)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Flat `trio.fail_after(1)` wraps a whole actor spawn + IPC
round-trip in `test_most_beautiful_word` — sub-second on a
warm box, but slow/noisy CI runners (esp. macOS) blow the
1s deadline with a `trio.TooSlowError`. Scale the budget by
`cpu_scaling_factor()`: a strict `1s` locally (factor
`1.0`), CI/CPU-throttle headroom (~3x) on macOS.
Review: PR #465 (copilot + self CI-triage)
https://github.com/goodboy/tractor/pull/465#pullrequestreview-4548191641
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Two version-compat fixes for the `devx` debugger
test-suite, both about matching upstream output that
got more verbose w/ recent lib releases.
ANSI stripping (`tests/devx/conftest.py`),
- Add `ansi_strip(text)` helper + `_ansi_re` pattern
(regex per https://stackoverflow.com/a/14693789).
- Apply inside `in_prompt_msg()` + `assert_before()` so
substring matches against REPL/traceback output stay
robust to color leakage.
- Motivated by py3.13's colored tracebacks +
`pdbp`/pygments highlighting leaking ANSI even when
`PYTHON_COLORS=0` is set in the `spawn` fixture (not
every renderer in the spawned subproc honors it).
- Replaces the longstanding inline TODO that linked
the SO answer w/o impl'ing.
trio 0.30+ `Cancelled._create(` match (`test_debugger`),
- In `test_shield_pause` swap the two
`"raise Cancelled._create()"` assertion patterns →
`"raise Cancelled._create("` (open-paren form, no
closing).
- trio >=0.30 raises a multi-line
`raise Cancelled._create(source=.., reason=..,
source_task=..)` w/ cancel-reason metadata, so the
legacy bare-`()` form no longer matches. Inline
comment documents the trio-version pivot.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 3854cf5ecb)
(cherry picked from commit c07cf2546b)
A `trio.Nursery.start()`-style wrapper around
`trio.run_process()` that surfaces rc!=0 errors
deterministically, ALWAYS isolates the parent
controlling-tty, and optionally live-relays the child's
std-streams to `log.<level>` per-line. Suits both
short-lived test-runners + long-lived daemons.
`supervise_run_process()`,
- Deterministic rc!=0: pass `check=False` to `trio`
and do our OWN post-drain rc-check from the
supervisor coro body AFTER `own_tn.__aexit__` — NOT
inside the internal nursery, since that would
race-cancel the still-draining relay reader and lose
stderr lines. (Re)build + raise a BARE
`subprocess.CalledProcessError`: `.stderr=` for
programmatic callers + an `add_note()`'d
`|_.stderr:` block for human teardown logs. No
nursery-eg-wrapped CPE to `collapse_eg` around.
- Parent controlling-tty isolation: `stdin=DEVNULL`
always, `stdout=DEVNULL` unless relayed/overridden
(via `stdout=` kwarg w/ `_UNSET` sentinel so explicit
`None` = inherit still works). Prevents a spawned
program from clobbering the launching tty's scrollback
w/ control-seqs.
- Live per-line relay: `relay_stdout=True`/
`relay_stderr=True` → relayed to `log.<relay_level>`
(default `'io'`, our custom level 21). Picked to sort
just above stdlib `INFO`=20 so it shows at usual
`info`/`devx` levels yet stays separately filterable;
`runtime`=15 was REJECTED as a default since it'd be
silently filtered at usual verbosity — footgun for
daemon supervisors whose whole point is visibility.
STREAMED, not buffered-until-exit.
- Non-blocking `tn.start()` semantics: live
`trio.Process` handed up via
`task_status.started()` immediately (else
`tn.start()` would block till child exit, losing
the long-lived-daemon use case). Supervise/relay bg
tasks run to completion in this coro.
- `**run_process_kwargs` forwarded verbatim (env, shell,
cwd, start_new_session, executable, ...); MANAGED keys
(`stdin`/`stdout`/`stderr`/`check`) win on conflict.
- Crash-handling layer intentionally NOT baked in —
compose `maybe_open_crash_handler()` ON TOP at the
call-site.
`_relay_stream_lines()` helper,
- Concurrent pipe-drain reader. MANDATORY whenever piping
w/o `capture_*` since nothing else drains the OS pipe —
child blocks on `write()` once kernel buf (~64KiB) fills
→ deadlock.
- Modes (combine freely): `emit`-only live relay,
`accum`-only silent drain+capture (for the CPE note),
or both. Per-line splitting handles cross-chunk
residuals + flushes any trailing un-newline-term'd line
at EOF.
`_add_stderr_note()` helper,
- Attaches an indented `|_.stderr:` note to a CPE via
`add_note()` for legible rc!=0 reporting at teardown.
Tests (`tests/trionics/test_subproc.py`),
- Hermetic `trio`-only (no actor-runtime).
- `test_stdout_relayed_per_line`: per-line stdout relay.
- `test_parent_tty_isolated`: child fd1 is OUR pipe (no
`/dev/pts/*`), fd0 pinned to `/dev/null`.
- `test_no_deadlock_on_big_unnewlined_output`: 200KiB
no-newline output completes under `fail_after(2)` —
exercises the concurrent drain (without it, the child
blocks at ~64KiB).
- `test_stderr_relay_and_cpe_rebuild`: rc!=0 w/
`relay_stderr=True` → bare `CalledProcessError` w/ the
`.stderr` note + per-line live relay.
- `test_nonrelay_cpe_note`: rc!=0 w/o relay → same
deterministic post-drain CPE w/ `.stderr` note (silent
drain+capture path).
Re-export `supervise_run_process` from `tractor.trionics`.
Prompt-IO: ai/prompt-io/claude/20260601T231429Z_0e3e008b_prompt_io.md
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit f595acc76c)
(cherry picked from commit 6df9ee11bc)
Make the sub-actor proc-title prefix a single
authoritative constant (`_proctitle._def_prefix`) so
the reap-recognition markers and `xontrib` banner pick
it up automatically — one place to flip the prefix
shape going fwd.
Deats,
- `_proctitle._def_prefix: str = '_subactor'`. New
module-level const consumed by everything that needs
to know the prefix.
- `set_actor_proctitle(actor, prefix=_def_prefix)`:
takes an explicit `prefix` arg (default = the const)
so callers can override per-spawn if they want.
- Default proc-title format:
`'tractor[<reprol>]'` → `f'{prefix}[<reprol>]'`
i.e. `_subactor[<reprol>]` by default.
- `_testing/_reap.py`: cmdline + comm markers source
the prefix from `_proctitle._def_prefix` instead of
the hardcoded `'tractor['`. So
`_is_tractor_subactor()` tracks the const
automatically.
- `xontrib/tractor_diag.xsh`: `acli.reap` orphan-mode
banner now interpolates the
`_TRACTOR_PROC_CMDLINE_MARKERS` tuple directly so
the human-readable mode line stays in sync if the
prefix shape changes again.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 3a45dbd503)
(cherry picked from commit fd8d39c0ce)
Follow-up to f595acc7 (`supervise_run_process`) which
called `log.io(...)` for std-stream relay assuming an
`IO=21` level existed. Add the registration via a new
factory + tests covering both the factory and the new
level.
`add_log_level()` factory,
- One call wires the four (otherwise hand-synced) pieces:
- `CUSTOM_LEVELS[NAME]` — drives the `stacklevel` bump
in `StackLevelAdapter.log()` + `get_logger()`'s
per-level audit.
- `logging.addLevelName()` — stdlib name registration.
- `STD_PALETTE[NAME]` + `BOLD_PALETTE['bold'][NAME]` —
color entries consumed by `get_console_log()`'s
`ColoredFormatter` build.
- Same-named (lowercase) emit method bound on
`StackLevelAdapter` so `log.<name>('msg')` works +
`get_logger()`'s per-level method audit passes.
- Idempotent: re-registering an existing name is a
no-op-ish refresh that won't clobber an already-bound
method.
- Method binding uses a default-arg `_level=value` so
the level int is captured (not late-bound across
multiple registrations).
`IO=21` level (first user),
- Purple. Used by `tractor.trionics._subproc`'s
std-stream relay (see f595acc7).
- Value 21 picked to sit just ABOVE stdlib `INFO`=20 so
it's SHOWN BY DEFAULT at usual `info`/`devx` console
levels — a `runtime`=15 relay would be silently
filtered (footgun for daemon supervisors whose whole
point is visibility). Still distinctly labeled +
filterable.
Tests (`tests/test_log_sys.py`),
- `test_io_custom_level_registered`: validates the IO
level is fully wired (`CUSTOM_LEVELS`, `addLevelName`,
both palettes, `StackLevelAdapter.io()` callable);
emits a record + sanity-asserts `21 >= INFO(20)`.
- `test_add_log_level_pluggable`: registers a fresh
`XLVL=19` (cyan) via `add_log_level()`, asserts all
four wires + the bound `xlog.xlvl()` emit, then
try/finally cleans up the module-global mutations so
later `get_logger()` audits don't trip on a
half-removed level.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 7bd7dd50c7)
(cherry picked from commit 93558fe3c9)
The `cpu_scaling_factor()` CI bump (2x) wasn't enough for the
macOS runners — slower + noisier than linux for our multi-actor
cancel-cascade timing — so `test_nested_multierrors[depth=3]` and
`test_fast_graceful_cancel_*` flaked there (linux + sdist green),
- make the CI headroom platform-aware: 3x on macOS, 2x on linux
(keeps the proven linux budget; depth=3 inner 12*3=36s still
fits under the 40s outer wall).
- give `test_fast_graceful_cancel_*` the `cpu_scaling_factor()`
headroom it was missing entirely (was a bare `timeout=2.9`).
Verified locally w/ `CI=1` (2x linux path; 3x macOS path confirmed
via forced `_non_linux`).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
All 5 flagged items were valid (4 real bugs + 1 dead assert),
- fix an inverted `sys.version_info < (3, 14)` guard in
`ipc._linux` — the "`cffi` has no 3.14 support" import note now
fires on 3.14+ (where it applies) instead of on older pys.
- use `os.environ.get('PYTHON_COLORS')` in the `sync_bp` example
so it doesn't `KeyError` when run outside the test harness.
- correct `dump_task_tree()`'s docstring: the `/tmp` + `/dev/tty`
tee is gated on `write_file`/`write_tty`, not "unconditional".
- tidy the `ActorTooSlowError` message spacing in `cancel_actor`.
- replace a tautological `applied is True or applied is False` in
`test_patches` with `isinstance(applied, bool)` (the value is
order-dependent across the module).
Review: PR #462 (copilot-pull-request-reviewer)
https://github.com/goodboy/tractor/pull/462#pullrequestreview-4527179852
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
GH Actions (and most shared) CI runners are slow + noisy and —
unlike a throttled local box — don't expose CPU-freq scaling via
sysfs, so `cpu_scaling_factor()` read `1.0` and the timing-
sensitive deadlines/asserts that key off it got NO headroom on
CI (a class of `TooSlowError` / `assert diff < this_fast` flakes),
- add a flat `_ci_env` x2 bump inside `cpu_scaling_factor()` so
every test already using it (quad streaming, SIGINT-cancel,
docs examples, ...) gets CI headroom for free — compounds with
any local-throttle factor.
- route the `time_quad_ex` cancel-deadline through it instead of
a bespoke per-test `ci_env` bump.
- fix a real bug in `test_nested_multierrors`: its OUTER
`@tractor_test(timeout=10)` was *smaller* than the inner
`fail_after_w_trace` budget (trio depth=3 = 12s), so the outer
wall fired first and pre-empted the snapshot-capturing inner
deadline -> `FAILED` instead of dumping. Bump the outer to `40`
(> max inner budget) and scale the inner budgets by
`cpu_scaling_factor()` too.
Verified locally with `CI=1` (quad + both `test_nested_multierrors`
depths + `test_cancel_via_SIGINT_other_task` green).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Same trio 0.29 → 0.33 cancel-cascade slowdown that hit
`test_nested_multierrors` (ea67f1b6) — bumps the
`trio`-backend (non-debug, non-forking) budget in
`test_echoserver_detailed_mechanics` from 1s → 4s.
- The 1s budget raced the ~1s teardown deadline. On a
deadline-fire trio 0.33 injects
`Cancelled(source='deadline')` (cancel-reason
metadata) that wraps the mid-stream KBI in a
`BaseExceptionGroup`, breaking the bare
`pytest.raises(KeyboardInterrupt)` below.
- Bump matches the forking-spawner branch (4s).
- Inline NOTE references the tracking issue
`ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit d7da502d93)
(cherry picked from commit d0144e52cb)
trio 0.29 → 0.33 lock bump (c7741bba) slowed the
depth=3 cancel-cascade in `test_nested_multierrors`
from <6s to ~7-8s; the 6s deadline was firing and its
`Cancelled(source='deadline')` (trio 0.33's new
cancel-reason metadata) collapsed a BEG branch,
breaking the `RemoteActorError` assertion downstream.
- Split the `('trio', _)` case-match into per-depth
arms: `('trio', 1)` keeps 6s (still finishes in
~3s); `('trio', 3)` → 12s.
- Updated inline NOTE explains the version pivot +
links the tracking issue
`ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
- Existing MTF/`subint_forkserver` budgets unchanged.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit ea67f1b67b)
(cherry picked from commit 57b3ea59ea)
Strip the trailing `pkg_path` token ONLY when it duplicates the
caller's leaf-*module* name (which the console header already
shows via `{filename}`), instead of blindly dropping the last
token. This keeps genuine, possibly-*nested* sub-PACKAGE parts
addressable as their own sub-loggers.
- detect a true leaf-mod by comparing the caller's `__name__`
vs `__package__` (a pkg `__init__` has them equal -> its
trailing token is a real sub-pkg, NOT a leaf to strip).
- `name='devx.debug'` now -> `tractor.devx.debug`, DISTINCT
from a bare `devx` -> `tractor.devx`; the old unconditional
`pkg_path = subpkg_path` collapsed both to `tractor.devx` and
silently broke per-sub-pkg level control via the logging-spec.
- `get_logger(__name__)` leaf-strip still works (cosmetic, bc
the leaf-mod is in the `{filename}` header field).
Also,
- update the `LogSpec` caveat: sub-PACKAGE granularity now
addressable at ANY depth; leaf *modules* intentionally aren't
(they're the `{filename}`); top-level mods (eg. `to_asyncio`)
still emit on the root logger.
- adjust `test_root_pkg_not_duplicated_in_logger_name` to the
new literal explicit-`name` contract (no leaf-collapse).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 9c36363b01)
Two coupled changes that let downstream projects (eg. `modden`) inherit
the test-harness loglevel plumbing for free via
`tractor._testing.pytest`:
Plugin lift (`tests/conftest.py` → `_testing/pytest.py`),
- mv `pytest_addoption(--ll)`, the `loglevel` autouse
fixture, and `test_log` fixture out of the test-suite-
local conftest into the reusable plugin.
- add `--tl`/`--tractor-loglevel` as a DISTINCT flag from
`--ll`: `--ll` is the consuming-project's OWN app
loglevel (scoped to its pkg-hierarchy), `--tl` is the
`tractor.*` runtime loglevel. `--tl` falls back to
`--ll` when unset (preserves current `tractor`-suite
behavior).
- add `testing_pkg_name` session fixture (default
`'tractor'`) — downstream projects override to e.g.
`'modden'` so `--ll` scopes to their own hierarchy
instead of `tractor.*`.
- `loglevel` fixture now yields the resolved
tractor-runtime level (passed to
`open_root_actor(loglevel=<.>)` by `@tractor_test`)
AND separately applies `--ll` to the
`testing_pkg_name` hierarchy when that isn't
`tractor`. `test_log` scopes the per-test logger to
`testing_pkg_name`.
`tractor.log` "logging-spec" mini-DSL,
- `LogSpec = str|bool`. Accepted forms:
- `True` → enable `pkg_name` root at `default_level`
(fallback `'cancel'`).
- `False` → no-op.
- bare level eg. `'info'` → root-logger at that level.
- `'sub:info,x:cancel'` → per-sub-logger filter-spec;
each `<name>` is RELATIVE to `pkg_name` (must NOT
include the pkg-token).
- `parse_logspec()` → `{sublog|None: level}` mapping.
`None` key = root-logger. Mixed bare-level + filters
in one spec is rejected w/ a helpful err msg; so is
embedding the `pkg_name` token in a sub-name.
- `apply_logspec()` → `(primary_level, {name: log})`:
parses then enables a `colorlog` stderr handler per
named (sub)logger. Authoritative sub-logger filters
get `propagate=False` so they don't double-emit
through a parallel root-level handler.
- !GRANULARITY CAVEAT! sub-logger names match at
sub-pkg granularity, not leaf-module — so `devx.debug`
collapses to the same `tractor.devx` logger as a bare
`devx`, and top-level lib modules (eg.
`tractor.to_asyncio`) emit under the *root* logger
rather than a phantom `to_asyncio` child. Documented
inline on `LogSpec`.
Other,
- `tests/conftest.py` keeps a NOTE pointing to the
plugin for future-debugging clarity (don't remove
silently — the lift is the relevant signal).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 19a77708ba)
Adopt the `_testing.trace` CM helpers in two MTF-hang-prone
tests so on-timeout we get a fresh
`ptree`/`wchan`/`py-spy` diag snapshot on disk instead of
opaque pytest timeout-kills. Same shape as bd07a95d for
`test_dynamic_pub_sub`.
Deats,
- `test_echoserver_detailed_mechanics`:
* inner `trio.fail_after` → `fail_after_w_trace`. Adds
`fail_after_w_trace: FailAfterWTraceFactory` fixture
param.
* mv per-backend `timeout` calc to top of test body (was
interleaved w/ helper defs).
* factor deep
`open_nursery`/`open_context`/`open_stream` body into
`_body()` so the wrapping `main()` stays a 2-liner —
keeps the nested-CM block at its natural indent level
instead of pushing it under yet another `async with`.
* drop `with_timeout: bool` knob + `fa_main()` helper
(knob was hard-coded `True`).
- `test_sigint_closes_lifetime_stack`:
* outer `signal.alarm`/`try`/`finally` → single
`afk_alarm_w_trace(10)` CM. Adds
`afk_alarm_w_trace: AfkAlarmWTraceFactory` fixture
param.
* drop `_AFK_CAP_S` + `armed_alarm` vars (CM owns both).
* explanatory comment refreshed to mention
`AFKAlarmTimeout` + the disk-snapshot side effect.
Other,
- Drop debug `return 1e3` short-circuit from `delay()`
fixture — snuck in as a scratch line, was clobbering the
proper `debug_mode`-branched return.
- Top-level import: `FailAfterWTraceFactory`,
`AfkAlarmWTraceFactory` from `tractor._testing.trace`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 1cafaecf52)
Replace inline `trio.fail_after` + manual `signal.alarm` guard with the
`_testing.trace` CM helpers that auto-capture a full ptree/wchan/py-spy
diag snapshot to disk on timeout.
Deats,
- inner guard: `trio.fail_after` → `fail_after_w_trace` (async CM,
captures on `TooSlowError`).
- outer AFK guard: raw `signal.alarm` → `afk_alarm_w_trace` (sync
CM, captures on `SIGALRM`), only armed under fork backends.
Extracts `_run_and_match()` helper to keep branching clean.
- bump `fail_after_s` from 4/12 → 8/20 to stop borderline flakes
while diag harness accumulates evidence.
- drop `_DIAG_CAP_S` var + manual signal import (now internal to
`afk_alarm_w_trace`).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit bd07a95d80)
Deats,
- `pytestmark`: enrich `skipon_spawn_backend('subint')` reason with
conc-anal doc refs + GH#379 link, add `reap_subactors_per_test`,
`track_orphaned_uds_per_test`,
`detect_runaway_subactors_per_test` fixtures
- `test_nested_multierrors`: parametrize over `depth` `{1, 3}`, add
MTF `xfail(strict=False)` with detailed race-window comment
explaining the BEG shape mismatch, wrap body in
`fail_after_w_trace` with per-backend timeout budget, bump
`@tractor_test(timeout=10)`, drop old multiprocessing depth
special-casing
- `test_multierror_fast_nursery`: wrap in
`fail_after_w_trace(30.0)`, accept `TooSlowError` in
`pytest.raises`, surface explicit `pytest.fail` on hang
- `test_cancel_while_childs_child_in_sync_sleep`: swap
`spawn_backend` param for `is_forking_spawner`, widen
`fail_after` delay for fork-based spawners
- `test_remote_error`, `test_multierror`,
`test_cancel_infinite_streamer`, `test_some_cancels_all`: add
`set_fork_aware_capture` fixture param
- Drop commented-out per-test `skipon_spawn_backend` blocks (now
covered by module-level `pytestmark`)
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 32955db02e)
Deats,
- `test_echoserver_detailed_mechanics`: add `is_forking_spawner`
param, wrap `main()` in `fa_main()` with per-backend
`trio.fail_after` (4s fork / 1s trio) to cap cancel-cascade
teardown that compounds under forkserver.
- `test_sigint_closes_lifetime_stack`: swap `start_method` param
for `is_forking_spawner`, pre-init `tmp_file`/`ctx` to `None` so
KBI firing before `open_context` body doesn't `UnboundLocalError`,
add `pytest.fail` guard for the spawn-time IPC race case, arm
`signal.alarm` AFK-safety cap (10s) under fork backends
Also,
- `pytestmark`: add `track_orphaned_uds_per_test` +
`detect_runaway_subactors_per_test` fixtures.
- `delay()`: hardcode `return 1e3` at top (debug override still in
place).
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 7ee0dc2e8f)
For forking spawner backends that is.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit b10011a36e)
- `skipon_spawn_backend('subint')`: expand reason with specific
analysis doc refs + GH issue #379 umbrella link.
- add `track_orphaned_uds_per_test` fixture via `usefixtures` to
blame-attribute UDS sock-file orphans left by SIGKILL cancel
cascades.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 7d0a53d205)
- `test_ext_types_over_ipc`: wrap `main()` in `fa_main()` with
`trio.fail_after(2)` + commented `capfd.disabled()` investigation
(pytest#14444).
- `test_basic_payload_spec`: add fixture param with note on fork-spawner
hang prevention.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 8aa07a7932)
Outer `signal.alarm` cap that fires even when trio's
`fail_after` is blocked by a shielded-await deadlock
(the bug-class-3 hang under MTF backends). Only armed
for fork-based spawners where the bug lives.
Deats,
- `_DIAG_CAP_S = fail_after_s + 5` — slightly larger than the
trio-native guard so it always loses when the in-band path works.
- `test_log.cancel()` breadcrumbs at each cancel-scope boundary so the
last-fired breadcrumb names the swallow point on hang.
- try/finally wrapping around each scope level for deterministic
breadcrumb emission.
- add `is_forking_spawner`, `set_fork_aware_capture` fixture params.
- rework `fail_after_s`: 4s for fork, 12s for trio (was 30/12).
Also,
- `test_sigint_both_stream_types`: `assert 0` -> `pytest.fail()`, add
TODO re `pytest.raises()`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 10db117864)
New `ai/conc-anal/spawn_time_boot_death_dup_name_issue.md`
documenting the spawn-time rc=2 race under rapid
same-name spawning against a forkserver + registrar
— the `wait_for_peer_or_proc_death` helper now surfaces
the death instead of parking forever on the handshake
wait.
Also,
- extract inline `xfail` into module-level
`_DOGGY_BOOT_RACE_XFAIL` marker.
- apply it to `n_dups=8` too (previously bare) bc
larger N widens the race window enough to fire
occasionally.
- link to tracking issue #456.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit 92443dc4ef)
Forking spawner + UDS transport has different timing
vs `trio_proc` — streaming example completes faster
in some cases, slower in others depending on fork
overhead + sock setup.
Deats,
- add `expect_cancel` param to `cancel_after()`, raise
`ActorTooSlowError` when cancel scope fires unexpectedly instead of
silently returning `None`.
- `time_quad_ex` fixture: bump timeout +1 for forking+UDS, explicit
`ActorTooSlowError` on `None` result instead of bare `assert results`.
- `test_not_fast_enough_quad`: `xfail` for forking+UDS being "too fast"
(cancel doesn't fire bc streaming finishes before delay).
- add `is_forking_spawner`, `tpt_proto` fixture params throughout.
Also,
- `_testing/pytest.py`: widen `start_method` parametrize and
`is_forking_spawner` fixture to `scope='session'`.
- `"""` -> `'''` docstring style throughout.
- hoist `_non_linux` to module scope (was redefined locally in two
places).
- type hints, kwarg-style `partial()` calls.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit d3cbc92751)
Add module-level `pytestmark` applying per-test
`reap_subactors_per_test`, `track_orphaned_uds_per_test`, and
`detect_runaway_subactors_per_test` fixtures — registrar tests stress
discovery roundtrips that historically left orphaned UDS sock-files.
Deats,
- drop unused `say_hello()` fn, keep only `say_hello_use_wait`;
rename param `func` -> `ria_fn`.
- use `@tractor_test(timeout=7)` instead of separate
`@pytest.mark.timeout(7, method='thread')` decorator.
- add `with_timeout()` helper, wire into
`test_subactors_unregister_on_cancel_remote_daemon`.
- uncomment `_timeout_main()` in `test_stale_entry_is_deleted`, use
configurable `timeout` var + `debug_mode` guard for `tractor.pause()`
on cancel.
- `dump_on_hang(seconds=timeout*2)` instead of hardcoded `20`.
- fix typo "oustanding" -> "outstanding".
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit abd3950ba6)
New `tractor.devx._proctitle` mod sets each
sub-actor's `argv[0]` (and kernel `comm`) to
`tractor[<aid.reprol()>]` — e.g.
`tractor[doggy@1027301b]` — so `ps`/`top`/`htop`
and `acli.pytree`/reaper tooling can identify
actors at a glance without parsing full cmdlines.
Deats,
- `set_actor_proctitle()` wraps the `setproctitle`
pkg with `ImportError` guard; optional at runtime
but listed in `pyproject.toml` so default installs
benefit.
- called early in `_child._actor_child_main()` after
`Actor` construction, before `_trio_main()` entry.
- tests in `tests/devx/test_proctitle.py`: format
unit test, `/proc/{cmdline,comm}` integration
test, negative detection test.
Resolves#457
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit d60245777e)
Extend `test_register_duplicate_name` w/ cancel-level log
breadcrumbs and `try/finally` for better diag on the cancel-cascade
hang.
Add `test_dup_name_cancel_cascade_escalates_to_hard_kill` as a
regression test for the TCP+MTF duplicate-name cancel-cascade
deadlock. Spawns N same-name actors, calls `an.cancel()`, and
asserts teardown completes within a `trio.fail_after()` budget that
scales w/ `n_dups`.
Deats,
- parametrize `n_dups` (2, 4, 8) to widen the race window for
concurrent `register_actor` RPCs.
- `n_dups=4` xfail'd — exposes a separate boot-race bug (doggy
`rc=2` under rapid same-name spawn), tracked in #456.
- post-teardown asserts all `Portal` chans disconnect, verifying
hard-kill escalation worked.
Relates to https://github.com/goodboy/tractor/issues/456
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(cherry picked from commit caebf60f4e)
Gud Boi