goodboy
/
tractor
1
0
Fork 2
Code Issues 5 Pull Requests 6 Packages Projects Releases Wiki Activity

Merge pull request #462 from goodboy/subint_era_tooling 

Subint era tooling
trionics_start_or_cancel
Bd 2026-06-19 08:58:12 -04:00 committed by GitHub
parent eb3e2b3574 084f0fc404
commit 2fdf3901cb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
72 changed files with 9299 additions and 988 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

102
.claude/skills/run-tests/SKILL.md
View File

@ -528,3 +528,105 @@ filling log volume. Full post-mortem in
`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
Lesson codified here so future-me grep-finds the
workaround before digging.
## 10. Reaping zombie subactors (`tractor-reap`)
**Symptom:** after a `pytest` run crashes, times out,
or is `Ctrl+C`'d, subactor forks (esp. under
`subint_forkserver`) can be reparented to `init`
(PPid==1) and linger. They hold onto ports, inherit
pytest's capture-pipe fds, and flakify later
sessions.
**Two layers of defense:**
### a) Session-scoped auto-fixture (always on)
`tractor/_testing/pytest.py::_reap_orphaned_subactors`
runs at pytest session teardown. It walks `/proc` for
direct descendants of the pytest pid, SIGINTs them,
waits up to 3s, then SIGKILLs survivors. SC-polite:
gives the subactor runtime a chance to run its trio
cancel shield + IPC teardown before escalation.
This is *autouse* and session-scoped — you don't need
to do anything. It just runs.
### b) `scripts/tractor-reap` CLI (manual reap)
For the **pytest-died-mid-session** case (Ctrl+C, OOM
kill, hung process you had to `kill -9`), the fixture
never ran. Reach for the CLI:
```sh
# default: orphans (PPid==1, cwd==repo, cmd contains python)
scripts/tractor-reap
# descendant-mode: from a still-live supervisor
scripts/tractor-reap --parent <pytest-pid>
# see what would be reaped, don't signal
scripts/tractor-reap -n
# tune the SIGINT → SIGKILL grace window
scripts/tractor-reap --grace 5
```
Exit code: `0` if everyone exited on SIGINT, `1` if
SIGKILL had to escalate — so you can chain it in CI
health-checks (`scripts/tractor-reap || <alert>`).
**What it matches** (orphan-mode):
- `PPid == 1` (reparented to init → definitely
orphaned, not just a currently-running child)
- `cwd == <repo-root>` (keeps the sweep scoped; won't
touch unrelated init-children elsewhere)
- `python` in cmdline
**What it does not do:** kill anything whose PPid is
still a live tractor parent. If the parent is alive
it's not an orphan; use `--parent <pid>` if you need
to force-reap under a still-live supervisor.
**When NOT to run it:** while a pytest session is
active in another terminal. It's safe (won't touch
that session's live children in orphan-mode) but can
race if the target session is mid-teardown.
### c) `--shm` / `--shm-only`: orphan-segment sweep
Because `tractor.ipc._mp_bs.disable_mantracker()`
turns off `mp.resource_tracker` (see
`ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`),
a hard-crashing actor can leave `/dev/shm/<key>`
segments behind that nothing else GCs.
```sh
# process reap THEN shm sweep
scripts/tractor-reap --shm
# shm sweep only (skip process phase)
scripts/tractor-reap --shm-only
# dry-run: list candidates, don't unlink
scripts/tractor-reap --shm -n
```
**Match criteria** (very conservative — this is a
shared-system path, can't be wrong):
- segment is a regular file under `/dev/shm`,
- owned by the **current uid** (`stat.st_uid`),
- AND **no live process holds it open** —
enumerated by walking every readable
`/proc/<pid>/maps` (post-mmap mappings) AND
`/proc/<pid>/fd/*` (pre-mmap shm-opened fds).
The "nobody has it open" check is the
kernel-canonical "is this leaked?" test — same
answer `lsof /dev/shm/<key>` would give. No
reliance on tractor-specific naming, so it works
for any tractor app. Critically, it WILL NOT touch
segments held by other apps you have running
(e.g. `piker`, `lttng-ust-*`, `aja-shm-*` —
verified locally with 81 in-use segments correctly
preserved).

142
ai/conc-anal/spawn_time_boot_death_dup_name_issue.md 100644
View File

@ -0,0 +1,142 @@
# Spawn-time boot-death (`rc=2`) under rapid same-name spawn against a registrar
## Symptom
Spawning N (≥4) sub-actors with the **same name** in tight
succession against a daemon registrar surfaces as
`ActorFailure: Sub-actor (...) died during boot (rc=2)
before completing parent-handshake`.
```
tests/discovery/test_multi_program.py
::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]
```
```
tractor._exceptions.ActorFailure:
Sub-actor ('doggy', '<uuid>') died during boot (rc=2)
before completing parent-handshake.
proc: <_ForkedProc pid=<n> returncode=None>
```
The `proc` repr shows `returncode=None` because the repr is
captured before `proc.wait()` returns; the actual
`os.WEXITSTATUS == 2` is reported via `result['died']` in the
race-helper.
## When it surfaces
- N=2 (`n_dups=2`): **always passes**.
- N=4 (`n_dups=4`): **consistent fail** under both `tpt-proto=tcp`
and `tpt-proto=uds`, MTF backend.
- N=8 (`n_dups=8`): **passes** (counter-intuitive — see "racing
windows").
- Non-MTF backends: not yet exercised systematically.
## What previously masked it
Pre the spawn-time `wait_for_peer_or_proc_death` race-helper
(in `tractor.spawn._spawn`), the parent's `start_actor` flow
ended with a bare:
```python
event, chan = await ipc_server.wait_for_peer(uid)
```
That awaits an unsignalled `trio.Event` on `_peer_connected[uid]`.
If the sub-actor process **dies during boot** (before its
runtime executes the parent-callback handshake that sets the
event), the wait parks forever. The dead proc becomes a zombie
because no one ever calls `proc.wait()` to reap it.
In test contexts the failure presented as a hang or a much
later `trio.TooSlowError` from an outer `fail_after`. In
production it'd present as a parent that never makes progress
past `start_actor`. The death itself was silently masked.
## What surfaces it now
`tractor.spawn._spawn.wait_for_peer_or_proc_death` (used by
`_main_thread_forkserver_proc`) races the handshake-wait
against `proc.wait()`. The race-helper raises `ActorFailure`
on death-first instead of parking, exposing the rc=2.
## Hypothesis: registrar-side same-name contention
The test spawns N actors with name `doggy` sequentially:
```python
for i in range(n_dups):
p: Portal = await an.start_actor('doggy')
portals.append(p)
```
Each spawned doggy:
1. Forks via the forkserver.
2. Boots its runtime in `_actor_child_main`.
3. Connects back to the parent for handshake.
4. Connects to the daemon registrar to call `register_actor`.
5. Enters its RPC msg-loop.
Step (4) is where the same-name contention lives. The
registrar's `register_actor` (in
`tractor.discovery._registry`) accepts duplicate names
(stores `(name, uuid) -> addr`), but its internal bookkeeping
may have a non-trivial check (e.g. `wait_for_actor` resolution,
`_addrs2aids` map updates) that errors out under specific
ordering between the existing entry and the incoming one.
`rc=2 == os.WEXITSTATUS == 2` corresponds to `sys.exit(2)`
in the doggy process — typically reached via an unhandled
exception that's translated to exit code 2 by Python's top-
level (e.g. `argparse` errors use 2; `SystemExit(2)` etc.).
So the doggy is hitting an explicit exit path during
`register_actor` or just-after.
The non-monotonic shape (N=2 OK, N=4 BAD, N=8 OK) suggests a
specific timing window — likely "the 3rd register-RPC arrives
while the 1st-or-2nd is in some intermediate state". With
N=8, the additional procs widen the registration spread
enough that no two land in the conflicting window.
## Where to dig next
- Add per-actor logging in `_actor_child_main` and
`register_actor` to surface the actual exception that
triggers the rc=2 exit. Currently the doggy dies before
the parent ever sees its stderr (forkserver doesn't
marshal child stdio back).
- Race-test the registrar's `register_actor` /
`unregister_actor` / `wait_for_actor` against same-name
concurrent calls in isolation (no spawn).
- Consider whether `register_actor` should be idempotent
under same-name re-register or should explicitly reject
same-name (and ideally with a clear `RemoteActorError`,
not `sys.exit(2)`).
## Test-suite handling
Currently:
- `tests/discovery/test_multi_program.py
::test_dup_name_cancel_cascade_escalates_to_hard_kill[n_dups=4]`
is `pytest.mark.xfail(strict=False, reason=...)` to keep
the suite green while this issue is investigated.
- `n_dups=2` and `n_dups=8` continue to validate the
cancel-cascade hard-kill escalation.
Once the underlying race is understood + fixed, drop the
xfail.
## Related work
- The cancel-cascade fix that introduced this regression
test:
`tractor/_exceptions.py:ActorTooSlowError`,
`tractor/runtime/_supervise.py:_try_cancel_then_kill`,
`tractor/runtime/_portal.py:Portal.cancel_actor(
raise_on_timeout=...)`.
- The spawn-time death-detection that exposed this:
`tractor/spawn/_spawn.py:wait_for_peer_or_proc_death`,
used by `tractor/spawn/_main_thread_forkserver.py`.

102
ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md 100644
View File

@ -0,0 +1,102 @@
# `trio` 0.29 -> 0.33 slows the depth=3 cancel-cascade
## Symptom
After locking to `trio==0.33.0` (commit `c7741bba`, was
`0.29.0`), this test reliably trips its `fail_after`
deadline on the **`trio`** backend:
```
FAILED tests/test_cancellation.py::test_nested_multierrors[start_method=trio-depth=3]
- AssertionError: assert False
where False = isinstance(
Cancelled(source='deadline', source_task=None, reason=None),
tractor.RemoteActorError,
)
```
A `fail_after_w_trace` hang-snapshot is captured for the
test each run (deadline-injected `Cancelled` wrapped into
the actor-nursery `BaseExceptionGroup`).
## Root cause (immediate)
The test budgets `fail_after(6)` for the `trio` backend.
That 6s was chosen (commit `32955db0`, while `trio==0.29`)
with the assertion that trio finishes "well under" 6s.
The `trio` 0.29 -> 0.33 bump slowed the depth=3 cascade
past that budget, so the 6s deadline now fires mid-cascade.
trio 0.33 added **cancel-reason tracking** — every
`Cancelled` now carries `(source=, reason=, source_task=)`.
The injected exc is `Cancelled(source='deadline')`, i.e.
trio itself naming our `fail_after(6)` scope as the cancel
origin. When that `Cancelled` collapses one branch of the
nursery BEG, the test's `isinstance(subexc,
RemoteActorError)` assertion fails. The healthy outcome is
`BEG = [RemoteActorError, RemoteActorError]`; the
`Cancelled` is purely an artifact of the deadline cutting
the cascade short.
## Measurements (standalone, this machine)
```
depth=1 trio ~3.15s PASS (keeps 6s budget)
depth=3 trio ~6.8-8.2s FAIL @ 6s (now bumped to 12s)
```
depth=1 still fits comfortably; only depth=3 (deeper
recursive spawn-and-error tree => more actors to reap)
exceeds the old budget. The ~2s/depth-level cost looks
like serialized per-actor reap / `terminate_after` waits.
## Mitigation applied
`test_nested_multierrors` now splits the `trio` budget:
```python
case ('trio', 1):
timeout = 6
case ('trio', 3):
timeout = 12 # was 6; see this doc
```
This stops the deadline from firing so the cascade
completes naturally to `[RAE, RAE]`.
## Also affected — same root cause, different test
`test_echoserver_detailed_mechanics[trio-raise_error=KeyboardInterrupt]`
(`tests/test_infected_asyncio.py`) tripped the *same*
slowdown via its much tighter `trio` budget of `1s`. The
single-aio-subactor teardown now takes ~1s, so the `1s`
`fail_after` raced the deadline (PASS at 0.99s / FAIL at
1.03s across back-to-back standalone runs). On a deadline-
fire the injected `Cancelled(source='deadline')` wraps the
mid-stream `KeyboardInterrupt` into a `BaseExceptionGroup`,
which is NOT a `KeyboardInterrupt` so the bare
`pytest.raises(KeyboardInterrupt)` fails. (The sibling
`raise_error=Exception` variant only "passes" by accident:
an `ExceptionGroup` *is-a* `Exception`, so its
`pytest.raises(Exception)` still matches even when wrapped.)
Mitigation: bump that `trio` budget `1 -> 4s` (matching the
forking-spawner case). Without a deadline-fire the KBI
propagates bare and the assertion passes.
## Open follow-up (the actual regression)
The budget bump is a band-aid — the underlying question is
**why** the depth=3 `trio` cancel-cascade went from <6s to
~7-8s across `trio` 0.29 -> 0.33. Candidate avenues:
- which scope owns the per-actor `terminate_after` wait,
and are the tree's reaps concurrent or serialized?
- did trio 0.33's abort/reschedule or cancel-reason
bookkeeping change checkpoint timing on the cancel path?
If/when the cascade speeds back up under-budget, depth=3
will start completing well under 12s — at which point the
budget can be tightened back toward 6s as a regression
tripwire. Related (different backend, same cascade class):
`cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.

221
ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md 100644
View File

@ -0,0 +1,221 @@
# trio `WakeupSocketpair.drain()` busy-loop in forked child (peer-closed missed-EOF)
## Reproducer
```bash
./py313/bin/python -m pytest \
tests/test_multi_program.py::test_register_duplicate_name \
--tpt-proto=tcp \
--spawn-backend=main_thread_forkserver \
-v --capture=sys
```
Subactor pegs a CPU core indefinitely; parent test
hangs waiting for the subactor.
## Empirical evidence (caught alive)
```
$ sudo strace -p <subactor-pid>
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
recvfrom(6, "", 65536, 0, NULL, NULL) = 0
... (no `epoll_wait`, no other syscalls, just this back-to-back)
```
Pattern: tight C-level `recvfrom` loop returning 0
each call. No `epoll_wait` between iterations →
**not trio's task scheduler**. Pure synchronous C
loop.
```
$ sudo readlink /proc/<subactor-pid>/fd/6
socket:[<inode>]
$ sudo lsof -p <subactor-pid> | grep ' 6u'
<cmd> <pid> goodboy 6u unix 0xffff... 0t0 <inode> type=STREAM (CONNECTED)
```
fd=6 is an **AF_UNIX socket** in CONNECTED state.
Even though the test uses `--tpt-proto=tcp`, this fd
is NOT a tractor IPC channel — it's an internal
trio socketpair.
## Root-cause: `WakeupSocketpair.drain()`
`/site-packages/trio/_core/_wakeup_socketpair.py`:
```python
class WakeupSocketpair:
def __init__(self) -> None:
self.wakeup_sock, self.write_sock = socket.socketpair()
self.wakeup_sock.setblocking(False)
self.write_sock.setblocking(False)
...
def drain(self) -> None:
try:
while True:
self.wakeup_sock.recv(2**16)
except BlockingIOError:
pass
```
`socket.socketpair()` on Linux defaults to AF_UNIX
SOCK_STREAM. Both ends non-blocking. Normal flow:
1. Signal/wake event → `write_sock.send(b'\x00')`
queues a byte.
2. `wakeup_sock` becomes readable → trio's epoll
triggers.
3. Trio calls `drain()` to flush the buffer.
4. drain loops on `wakeup_sock.recv(64KB)`.
5. Eventually buffer empty → non-blocking socket
raises `BlockingIOError` → except → break.
**Bug surface — peer-closed missed-EOF**:
Non-blocking socket semantics:
- buffer has data → `recv` returns N>0 bytes (loop continues)
- buffer empty → `recv` raises `BlockingIOError`
- **peer FIN'd → `recv` returns 0 bytes (NEITHER exception NOR
break — infinite tight loop)**
`drain()` does not handle the `b''` return-value
(EOF) case. If `write_sock` has been closed (or the
process holding it is gone), every iteration returns
0 → infinite loop → 100% CPU on a single core.
## Why this triggers under `main_thread_forkserver`
Under `os.fork()` from the forkserver-worker thread:
1. Parent has a `WakeupSocketpair` instance with
`wakeup_sock=fdN`, `write_sock=fdM`. Both fds
open in parent.
2. Fork → child inherits BOTH fds (kernel-level fd
table dup).
3. `_close_inherited_fds()` runs in child →
closes everything except stdio. `wakeup_sock` and
`write_sock` of the parent's `WakeupSocketpair`
ARE closed in child.
4. Child's trio (running fresh) creates its OWN
`WakeupSocketpair` → NEW fd numbers (e.g. fd 6, 7).
5. **In `infect_asyncio` mode** the asyncio loop is
the host; trio runs as guest via
`start_guest_run`. trio still creates its
`WakeupSocketpair` in the I/O manager but its
role is different.
The race window: somewhere between (3) and (5), if a
`WakeupSocketpair` Python object reference inherited
via COW (from parent's pre-fork heap) survives long
enough that `drain()` is called on it AFTER its fds
were closed but BEFORE the child's NEW socketpair
takes over the recycled fd numbers — the recycled fd
will be one of the child's NEW socketpair ends, whose
peer might be FIN-flagged (e.g. parent-process
peer-end is closed).
Or simpler: the `wait_for_actor`/`find_actor` discovery
flow in `test_register_duplicate_name` triggers an
unusual code path where a stale `WakeupSocketpair`
gets `drain()`-called on a fd whose peer has already
closed.
## Why `drain()` shouldn't loop indefinitely on EOF
(upstream trio bug)
Even WITHOUT fork, `drain()` should treat `b''` as
EOF and break. The current code is correct for the
"buffer drained on a healthy socketpair" scenario but
incorrect for the "peer is gone" scenario. It's a
defensive-programming gap in trio.
A one-line patch upstream:
```python
def drain(self) -> None:
try:
while True:
data = self.wakeup_sock.recv(2**16)
if not data:
break # peer-closed; nothing more to drain
except BlockingIOError:
pass
```
## Workarounds (until the underlying issue lands)
1. **Skip-mark on the fork backend**:
`tests/test_multi_program.py` →
`pytest.mark.skipon_spawn_backend('main_thread_forkserver',
reason='trio WakeupSocketpair.drain busy-loop, see ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md')`.
2. **Defensive monkey-patch in tractor's
forkserver-child prelude** — wrap
`WakeupSocketpair.drain` to handle `b''`:
```python
# in `_actor_child_main` or `_close_inherited_fds`'s
# post-fork prelude:
from trio._core._wakeup_socketpair import WakeupSocketpair
_orig_drain = WakeupSocketpair.drain
def _safe_drain(self):
try:
while True:
data = self.wakeup_sock.recv(2**16)
if not data:
return # peer closed
except BlockingIOError:
pass
WakeupSocketpair.drain = _safe_drain
```
Tracks upstream — remove once trio fixes.
3. **Upstream the fix**: 1-line PR to `python-trio/trio`
adding `if not data: break` to `drain()`.
## Investigation next steps
1. **Confirm via py-spy**: when caught alive, detach
strace first then
`sudo py-spy dump --pid <subactor> --locals`. The
busy thread should show `drain` from `WakeupSocketpair`
in the call chain.
2. **Identify which write-end peer is closed**: from
the inode of fd 6, look up the matching peer
inode via `ss -xp` and see whose process it
was/is.
3. **Verify the missed-EOF hypothesis**: hand-craft a
minimal `WakeupSocketpair` repro:
```python
from trio._core._wakeup_socketpair import WakeupSocketpair
ws = WakeupSocketpair()
ws.write_sock.close() # simulate peer-gone
ws.drain() # should hang forever
```
## Sibling bug
`tests/test_infected_asyncio.py::test_aio_simple_error`
hangs under the same backend with a DIFFERENT
fingerprint (Mode-A deadlock, both parties in
`epoll_wait`, no busy-loop). Distinct root cause —
see `infected_asyncio_under_main_thread_forkserver_hang_issue.md`.
Both share the broader theme: **trio internal-state
initialization isn't fully fork-safe under
`main_thread_forkserver`** for the more exotic
dispatch paths.
## See also
- [#379](https://github.com/goodboy/tractor/issues/379) — subint umbrella
- python-trio/trio#1614 — trio + fork hazards
- `trio._core._wakeup_socketpair.WakeupSocketpair`
source (the smoking gun)
- `ai/conc-anal/fork_thread_semantics_execution_vs_memory.md`
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`

9
examples/debugging/multi_daemon_subactors.py
View File

@ -27,12 +27,9 @@ async def main():
'''
async with tractor.open_nursery(
debug_mode=True,
loglevel='cancel',
# loglevel='devx',
) as n:
p0 = await n.start_actor('bp_forever', enable_modules=[__name__])
p1 = await n.start_actor('name_error', enable_modules=[__name__])
) as an:
p0 = await an.start_actor('bp_forever', enable_modules=[__name__])
p1 = await an.start_actor('name_error', enable_modules=[__name__])
# retreive results
async with p0.open_stream_from(breakpoint_forever) as stream:

2
examples/debugging/multi_nested_subactors_error_up_through_nurseries.py
View File

@ -67,7 +67,7 @@ async def main():
"""
async with tractor.open_nursery(
debug_mode=True,
# loglevel='cancel',
loglevel='pdb',
) as n:
# spawn both actors

4
examples/debugging/root_cancelled_but_child_is_in_tty_lock.py
View File

@ -39,8 +39,8 @@ async def main():
'''
async with tractor.open_nursery(
debug_mode=True,
loglevel='devx',
enable_transports=['uds'],
enable_transports=['uds'], # TODO, apss this via osenv?
loglevel='devx', # XXX, required for test!
) as n:
# spawn both actors

13
examples/debugging/root_timeout_while_child_crashed.py
View File

@ -1,4 +1,3 @@
import trio
import tractor
@ -9,16 +8,22 @@ async def key_error():
async def main():
"""Root dies
'''
Root is fail-after-cancelled while blocking and child RPC fails
simultaneously.
"""
'''
async with tractor.open_nursery(
debug_mode=True,
loglevel='debug'
# loglevel='debug' # ?XXX required?
) as n:
# spawn both actors
portal = await n.run_in_actor(key_error)
print(
f'Child is up @ {portal.chan.aid.reprol()}'
)
# XXX: originally a bug caused by this is where root would enter
# the debugger and clobber the tty used by the repl even though

10
examples/debugging/shield_hang_in_sub.py
View File

@ -49,9 +49,11 @@ async def main(
tractor.open_nursery(
debug_mode=True,
enable_stack_on_sig=True,
# maybe_enable_greenback=False,
loglevel='devx',
loglevel='devx', # XXX REQUIRED log level!
enable_transports=[tpt],
# maybe_enable_greenback=True,
# ^TODO? maybe a "smarter" way todo all this is how
# `modden` does with a rtv serialized through the osenv?
) as an,
):
ptl: tractor.Portal = await an.start_actor(
@ -63,7 +65,9 @@ async def main(
start_n_shield_hang,
) as (ctx, cpid):
_, proc, _ = an._children[ptl.chan.uid]
_, proc, _ = an._children[
ptl.chan.aid.uid
]
assert cpid == proc.pid
print(

5
examples/debugging/subactor_bp_in_ctx.py
View File

@ -36,6 +36,11 @@ async def just_bp(
async def main():
# !TODO, parametrize the --tpt-proto={key} with osenv vars just
# like we do for loglevel/spawn-backend!
# - [ ] run on both tpts for all such debugger tests?
# - [ ] special skip for macos!
#
if platform.system() != 'Darwin':
tpt = 'uds'
else:

1
examples/debugging/subactor_error.py
View File

@ -9,7 +9,6 @@ async def name_error():
async def main():
async with tractor.open_nursery(
debug_mode=True,
# loglevel='transport',
) as an:
# TODO: ideally the REPL arrives at this frame in the parent,

24
examples/debugging/sync_bp.py
View File

@ -1,9 +1,22 @@
from functools import partial
import os
import time
# ?TODO? how to make `pdbp` enforce this?
# os.environ['PYTHON_COLORS'] = '0'
# os.environ['NO_COLOR'] = '1'
import trio
import tractor
# disable `pbdp` prompt colors
# for prompt matching in test.
def disable_pdbp_color():
if os.environ.get('PYTHON_COLORS') == '0':
from tractor.devx.debug import _repl
_repl.TractorConfig.use_pygments = False
# TODO: only import these when not running from test harness?
# can we detect `pexpect` usage maybe?
# from tractor.devx.debug import (
@ -42,6 +55,7 @@ async def start_n_sync_pause(
ctx: tractor.Context,
):
actor: tractor.Actor = tractor.current_actor()
disable_pdbp_color()
# sync to parent-side task
await ctx.started()
@ -52,13 +66,15 @@ async def start_n_sync_pause(
async def main() -> None:
disable_pdbp_color()
async with (
tractor.open_nursery(
debug_mode=True,
maybe_enable_greenback=True,
enable_stack_on_sig=True,
# loglevel='warning',
# loglevel='devx',
# XXX flags required for test pattern matching.
loglevel='pdb',
# enable_stack_on_sig=True,
) as an,
trio.open_nursery() as tn,
):
@ -68,8 +84,8 @@ async def main() -> None:
p: tractor.Portal = await an.start_actor(
'subactor',
enable_modules=[__name__],
# infect_asyncio=True,
debug_mode=True,
# infect_asyncio=True,
)
# TODO: 3 sub-actor usage cases:

36
pyproject.toml
View File

@ -43,15 +43,20 @@ dependencies = [
"tricycle>=0.4.1,<0.5",
"wrapt>=1.16.0,<2",
"colorlog>=6.8.2,<7",
# built-in multi-actor `pdb` REPL
"pdbp>=1.8.2,<2", # windows only (from `pdbp`)
# typed IPC msging
"msgspec>=0.20.0",
"bidict>=0.23.1",
"multiaddr>=0.2.0",
"platformdirs>=4.4.0",
# per-actor `argv[0]` proc-title for OS-level diag tools
# (`ps`, `top`, `psutil`-backed tooling like `acli.pytree`).
# Optional at runtime — guarded by `try/except ImportError` in
# `tractor.devx._proctitle` — but listed here so default
# installs benefit from it. See tracking issue for follow-ups
# (e.g. richer formats, per-backend overrides).
"setproctitle>=1.3,<2",
]
# ------ project ------
@ -61,6 +66,7 @@ dev = [
{include-group = 'devx'},
{include-group = 'testing'},
{include-group = 'repl'},
{include-group = 'sync_pause'},
]
devx = [
# `tractor.devx` tooling
@ -84,6 +90,16 @@ testing = [
# interactions stay predictable across dev installs).
"pytest>=9.0.3", # CVE-2025-71176 (insecure tmpdir) patched in 9.0.3
"pexpect>=4.9.0,<5",
# per-test wall-clock bound (used via
# `@pytest.mark.timeout(..., method='thread')` on the
# known-hanging `subint`-backend audit tests; see
# `ai/conc-anal/subint_*_issue.md`).
"pytest-timeout>=2.3",
# used by `tractor._testing._reap` for the
# `tractor-reap` zombie-subactor + leaked-shm
# cleanup utility (xplatform `Process.memory_maps`,
# `Process.open_files`).
"psutil>=7.0.0",
]
repl = [
"pyperclip>=1.9.0",
@ -234,15 +250,27 @@ testpaths = [
addopts = [
# TODO: figure out why this isn't working..
'--rootdir=./tests',
'--import-mode=importlib',
# don't show frickin captured logs AGAIN in the report..
'--show-capture=no',
# load builtin plugin since we need a boostrapping hook,
# `pytest_load_initial_conftests()` for `--capture=` per:
# https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
'-p tractor._testing.pytest',
# disable `xonsh` plugin
# https://docs.pytest.org/en/stable/how-to/plugins.html#disabling-plugins-from-autoloading
# https://docs.pytest.org/en/stable/how-to/plugins.html#deactivating-unregistering-a-plugin-by-name
'-p no:xonsh'
'-p no:xonsh',
# XXX default on non-forking spawners
'--capture=fd',
# '--capture=sys',
# ^XXX NOTE^ ALWAYS SET THIS for `*_forkserver` spawner
# backends! see details @
# `tractor._testing.pytest.pytest_load_initial_conftests()`
]
log_cli = false
# TODO: maybe some of these layout choices?

237
scripts/tractor-reap 100755
View File

@ -0,0 +1,237 @@
#!/usr/bin/env python3
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
#
# SPDX-License-Identifier: AGPL-3.0-or-later
'''
`tractor-reap` — SC-polite zombie-subactor reaper +
optional `/dev/shm/` orphan-segment sweep.
Two cleanup phases (run in order when both are enabled):
1. **process reap** — finds `tractor` subactor processes
left alive after a `pytest` (or any tractor-app) run
that failed to fully cancel its actor tree, then sends
SIGINT with a bounded grace window before escalating
to SIGKILL.
2. **shm sweep** (`--shm` / `--shm-only`) — unlinks
`/dev/shm/<file>` entries owned by the current uid
that no live process has open (mmap'd or fd-held).
Needed because `tractor` disables
`mp.resource_tracker` (see `tractor.ipc._mp_bs`), so a
hard-crashing actor leaves leaked segments that
nothing else GCs.
3. **UDS sweep** (`--uds` / `--uds-only`) — unlinks
`${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` files
whose binder pid is dead (or the `1616` registry
sentinel). Needed because the IPC server's
`os.unlink()` cleanup lives in a `finally:` block
that doesn't always run on hard exits (SIGKILL,
escaped `KeyboardInterrupt`, etc.) — see issue #452.
Process-reap detection modes (auto-selected):
--parent <pid> : descendant-mode — kill procs whose
PPid == <pid>. Use when a parent
is still alive and you want to
scope the sweep precisely (e.g.
CI wrapper calling in from outside
pytest).
(default) : orphan-mode — kill procs with
PPid==1 (init-reparented) whose
cwd matches the repo root AND
whose cmdline contains `python`.
The cwd filter is what prevents
sweeping unrelated init-children.
Usage:
# process reap only (default)
scripts/tractor-reap
# process reap + shm sweep
scripts/tractor-reap --shm
# only the shm sweep, skip process reap
scripts/tractor-reap --shm-only
# process reap + shm + UDS sweep (the works)
scripts/tractor-reap --shm --uds
# only UDS sweep
scripts/tractor-reap --uds-only
# from inside a still-live supervisor
scripts/tractor-reap --parent 12345
# dry-run: list what would be reaped, don't act
scripts/tractor-reap -n
scripts/tractor-reap --shm --uds -n
'''
import argparse
import pathlib
import subprocess
import sys
def _repo_root() -> pathlib.Path:
'''
Use `git rev-parse --show-toplevel` when available;
fall back to the repo this script lives in.
'''
try:
out: str = subprocess.check_output(
['git', 'rev-parse', '--show-toplevel'],
stderr=subprocess.DEVNULL,
text=True,
).strip()
return pathlib.Path(out)
except (subprocess.CalledProcessError, FileNotFoundError):
return pathlib.Path(__file__).resolve().parent.parent
def main() -> int:
parser = argparse.ArgumentParser(
prog='tractor-reap',
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
'--parent', '-p',
type=int,
default=None,
help='descendant-mode: reap procs with PPid==<pid>',
)
parser.add_argument(
'--grace', '-g',
type=float,
default=3.0,
help='SIGINT grace window in seconds (default 3.0)',
)
parser.add_argument(
'--dry-run', '-n',
action='store_true',
help='list matched pids/paths but do not signal/unlink',
)
parser.add_argument(
'--shm',
action='store_true',
help=(
'after process reap, also unlink orphaned '
'/dev/shm segments owned by the current user '
'that no live process is mapping or holding open'
),
)
parser.add_argument(
'--shm-only',
action='store_true',
help='skip process reap; only do the shm sweep',
)
parser.add_argument(
'--uds',
action='store_true',
help=(
'after process reap, also unlink orphaned '
'${XDG_RUNTIME_DIR}/tractor/*.sock files '
'whose binder pid is dead (or the 1616 '
'registry sentinel). See issue #452.'
),
)
parser.add_argument(
'--uds-only',
action='store_true',
help='skip process reap + shm; only do the UDS sweep',
)
args = parser.parse_args()
# any *-only flag also skips the process reap phase
skip_proc_reap: bool = (
args.shm_only
or
args.uds_only
)
# import lazily so `--help` doesn't require the tractor
# package to be importable (e.g. when running from a
# shell not inside a venv).
repo = _repo_root()
sys.path.insert(0, str(repo))
from tractor._testing._reap import (
find_descendants,
find_orphans,
find_orphaned_shm,
find_orphaned_uds,
reap,
reap_shm,
reap_uds,
)
rc: int = 0
# --- phase 1: process reap (skipped under --*-only) ---
if not skip_proc_reap:
if args.parent is not None:
pids: list[int] = find_descendants(args.parent)
mode: str = f'descendants of PPid={args.parent}'
else:
pids = find_orphans(repo)
mode = f'orphans (PPid=1, cwd={repo})'
if not pids:
print(f'[tractor-reap] no {mode} to reap')
elif args.dry_run:
print(
f'[tractor-reap] dry-run — {mode}:\n {pids}'
)
else:
_, survivors = reap(pids, grace=args.grace)
if survivors:
rc = 1
# --- phase 2: shm sweep (opt-in) ---
if args.shm or args.shm_only:
leaked: list[str] = find_orphaned_shm()
if not leaked:
print(
'[tractor-reap] no orphaned /dev/shm '
'segments to sweep'
)
elif args.dry_run:
print(
f'[tractor-reap] dry-run — {len(leaked)} '
f'orphaned shm segment(s):\n {leaked}'
)
else:
_, errors = reap_shm(leaked)
if errors:
rc = 1
# --- phase 3: UDS sweep (opt-in) ---
if args.uds or args.uds_only:
leaked_uds: list[str] = find_orphaned_uds()
if not leaked_uds:
print(
'[tractor-reap] no orphaned UDS sock-files '
'to sweep'
)
elif args.dry_run:
print(
f'[tractor-reap] dry-run — {len(leaked_uds)} '
f'orphaned UDS sock-file(s):\n {leaked_uds}'
)
else:
_, errors = reap_uds(leaked_uds)
if errors:
rc = 1
# exit 0 if everything cleaned cleanly, else 1 — useful
# for CI health-check chaining.
return rc
if __name__ == '__main__':
raise SystemExit(main())

221
tests/conftest.py
View File

@ -22,7 +22,8 @@ from tractor._testing import (
pytest_plugins: list[str] = [
'pytester',
'tractor._testing.pytest',
# NOTE, now loaded in `pytest-ini` section of `pyproject.toml`
# 'tractor._testing.pytest',
]
_ci_env: bool = os.environ.get('CI', False)
@ -33,15 +34,10 @@ if platform.system() == 'Windows':
_KILL_SIGNAL = signal.CTRL_BREAK_EVENT
_INT_SIGNAL = signal.CTRL_C_EVENT
_INT_RETURN_CODE = 3221225786
_PROC_SPAWN_WAIT = 2
else:
_KILL_SIGNAL = signal.SIGKILL
_INT_SIGNAL = signal.SIGINT
_INT_RETURN_CODE = 1 if sys.version_info < (3, 8) else -signal.SIGINT.value
_PROC_SPAWN_WAIT = (
2 if _ci_env
else 1
)
no_windows = pytest.mark.skipif(
@ -100,91 +96,51 @@ def cpu_scaling_factor() -> float:
much to inflate time-limits when CPU-freq scaling is active on
linux.
When no scaling info is available (non-linux, missing sysfs),
returns 1.0 (i.e. no headroom adjustment needed).
When no local scaling info is available (non-linux, missing
sysfs) the base factor is 1.0; a flat CI bump is then applied
on top (see below).
'''
if _non_linux:
return 1.
factor: float = 1.
if not _non_linux:
mx = get_cpu_state()
cur = get_cpu_state(setting='scaling_max_freq')
if mx is None or cur is None:
return 1.
if (
mx is not None
and
cur is not None
):
_mx_pth, max_freq = mx
_cur_pth, cur_freq = cur
cpu_scaled: float = int(cur_freq) / int(max_freq)
if cpu_scaled != 1.:
return 1. / (
factor = 1. / (
cpu_scaled * 2 # <- bc likely "dual threaded"
)
return 1.
# XXX, GH Actions (and most shared) CI runners are slow + noisy
# and — unlike a throttled local box — do NOT expose CPU-freq
# scaling via sysfs, so the probe above reads 1.0 and adds no
# headroom. Apply a flat CI bump so every timing-test deadline
# /assert that keys off this factor gets headroom on CI HW
# (compounds with any local-throttle factor).
#
# macOS runners are noticeably slower + noisier than the linux
# ones for our multi-actor cancel-cascade tests, so give them
# extra headroom (3x vs 2x).
if _ci_env:
factor *= 3 if _non_linux else 2
return factor
def pytest_addoption(
parser: pytest.Parser,
):
# ?TODO? should this be exposed from our `._testing.pytest`
# plugin or should we make it more explicit with `--tl` for
# tractor logging like we do in other client projects?
parser.addoption(
"--ll",
action="store",
dest='loglevel',
default='ERROR', help="logging level to set when testing"
)
@pytest.fixture(scope='session', autouse=True)
def loglevel(request) -> str:
import tractor
orig = tractor.log._default_loglevel
level = tractor.log._default_loglevel = request.config.option.loglevel
log = tractor.log.get_console_log(
level=level,
name='tractor', # <- enable root logger
)
log.info(
f'Test-harness set runtime loglevel: {level!r}\n'
)
yield level
tractor.log._default_loglevel = orig
@pytest.fixture(scope='function')
def test_log(
request,
loglevel: str,
) -> tractor.log.StackLevelAdapter:
'''
Deliver a per test-module-fn logger instance for reporting from
within actual test bodies/fixtures.
For example this can be handy to report certain error cases from
exception handlers using `test_log.exception()`.
'''
modname: str = request.function.__module__
log = tractor.log.get_logger(
name=modname, # <- enable root logger
# pkg_name='tests',
)
_log = tractor.log.get_console_log(
level=loglevel,
logger=log,
name=modname,
# pkg_name='tests',
)
_log.debug(
f'In-test-logging requested\n'
f'test_log.name: {log.name!r}\n'
f'level: {loglevel!r}\n'
)
yield _log
# NOTE, the `--ll`/`--tl` CLI flags + the `loglevel`, `test_log`
# and `testing_pkg_name` fixtures have been factored into the
# `tractor._testing.pytest` plugin (loaded via the `-p` entry in
# `pyproject.toml`'s `[tool.pytest.ini_options]`) so downstream
# consuming projects (eg. `modden`) inherit them for free. The
# plugin's `testing_pkg_name` fixture defaults to `'tractor'`, so
# this suite keeps treating `--ll` as the runtime loglevel.
@pytest.fixture(scope='session')
@ -236,107 +192,14 @@ def sig_prog(
assert ret
# TODO: factor into @cm and move to `._testing`?
@pytest.fixture
def daemon(
debug_mode: bool,
loglevel: str,
testdir: pytest.Pytester,
reg_addr: tuple[str, int],
tpt_proto: str,
ci_env: bool,
test_log: tractor.log.StackLevelAdapter,
) -> subprocess.Popen:
'''
Run a daemon root actor as a separate actor-process tree and
"remote registrar" for discovery-protocol related tests.
'''
if loglevel in ('trace', 'debug'):
# XXX: too much logging will lock up the subproc (smh)
loglevel: str = 'info'
code: str = (
"import tractor; "
"tractor.run_daemon([], "
"registry_addrs={reg_addrs}, "
"enable_transports={enable_tpts}, "
"debug_mode={debug_mode}, "
"loglevel={ll})"
).format(
reg_addrs=str([reg_addr]),
enable_tpts=str([tpt_proto]),
ll="'{}'".format(loglevel) if loglevel else None,
debug_mode=debug_mode,
)
cmd: list[str] = [
sys.executable,
'-c', code,
]
# breakpoint()
kwargs = {}
if platform.system() == 'Windows':
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
proc: subprocess.Popen = testdir.popen(
cmd,
**kwargs,
)
# TODO! we should poll for the registry socket-bind to take place
# and only once that's done yield to the requester!
# -[ ] TCP: use the `._root.open_root_actor()`::`ping_tpt_socket()`
# closure!
# -[ ] UDS: can we do something similar for 'pinging" the
# file-socket?
#
global _PROC_SPAWN_WAIT
# UDS sockets are **really** fast to bind()/listen()/connect()
# so it's often required that we delay a bit more starting
# the first actor-tree..
if tpt_proto == 'uds':
_PROC_SPAWN_WAIT += 1.6
if _non_linux and ci_env:
_PROC_SPAWN_WAIT += 1
# XXX, allow time for the sub-py-proc to boot up.
# !TODO, see ping-polling ideas above!
time.sleep(_PROC_SPAWN_WAIT)
assert not proc.returncode
yield proc
sig_prog(proc, _INT_SIGNAL)
# XXX! yeah.. just be reaaal careful with this bc sometimes it
# can lock up on the `_io.BufferedReader` and hang..
stderr: str = proc.stderr.read().decode()
stdout: str = proc.stdout.read().decode()
if (
stderr
or
stdout
):
print(
f'Daemon actor tree produced output:\n'
f'{proc.args}\n'
f'\n'
f'stderr: {stderr!r}\n'
f'stdout: {stdout!r}\n'
)
if (rc := proc.returncode) != -2:
msg: str = (
f'Daemon actor tree was not cancelled !?\n'
f'proc.args: {proc.args!r}\n'
f'proc.returncode: {rc!r}\n'
)
if rc < 0:
raise RuntimeError(msg)
test_log.error(msg)
# NOTE, the `daemon` fixture (+ its `_wait_for_daemon_ready`
# helper + the post-yield teardown drain logic) has been
# moved to `tests/discovery/conftest.py` since 100% of its
# consumers are discovery-protocol tests now living under
# that subdir. See:
# - `tests/discovery/test_multi_program.py`
# - `tests/discovery/test_registrar.py`
# - `tests/discovery/test_tpt_bind_addrs.py`
# @pytest.fixture(autouse=True)

114
tests/devx/conftest.py
View File

@ -4,6 +4,7 @@
'''
from __future__ import annotations
import platform
import os
import signal
import time
from typing import (
@ -56,6 +57,7 @@ type PexpectSpawner = Callable[
@pytest.fixture
def spawn(
start_method: str,
loglevel: str,
testdir: pytest.Pytester,
reg_addr: tuple[str, int],
@ -65,9 +67,19 @@ def spawn(
run an `./examples/..` script by name.
'''
if start_method != 'trio':
supported_spawners: set[str] = {
'trio',
# `examples/debugging/<script>.py` picks up the spawn
# backend via the `TRACTOR_SPAWN_METHOD` env-var which
# is honored inside `tractor._root.open_root_actor()`,
# so no per-script edits are required.
'main_thread_forkserver',
'subint_forkserver',
}
if start_method not in supported_spawners:
pytest.skip(
'`pexpect` based tests only supported on `trio` backend'
f'`pexpect` based tests NOT supported on spawning-backend: {start_method!r}\n'
f'supported-spawners: {supported_spawners!r}'
)
def unset_colors():
@ -79,21 +91,64 @@ def spawn(
https://docs.python.org/3/using/cmdline.html#using-on-controlling-color
'''
import os
# disable colored tbs
os.environ['PYTHON_COLORS'] = '0'
# disable all ANSI color output
# os.environ['NO_COLOR'] = '1'
# ?TODO, doesn't seem to disable prompt color
# for `pdbp`?
def set_spawn_method(
start_method: str,
):
'''
Drive the actor-spawn backend inside the spawned
`examples/debugging/<script>.py` subproc via env-var
(consumed by `tractor._root.open_root_actor()`),
without requiring per-script CLI plumbing.
'''
os.environ['TRACTOR_SPAWN_METHOD'] = start_method
def set_loglevel(
loglevel: str|None,
):
'''
Forward the test-suite parametrized `loglevel` into the
spawned `examples/debugging/<script>.py` subproc via
env-var (consumed by `tractor._root.open_root_actor()`),
so console verbosity can be cranked or silenced from
the test harness without per-script edits.
'''
if loglevel:
os.environ['TRACTOR_LOGLEVEL'] = loglevel
else:
os.environ.pop('TRACTOR_LOGLEVEL', None)
spawned: PexpectSpawner|None = None
def _spawn(
cmd: str,
expect_timeout: float = 4,
start_method: str = start_method,
loglevel: str|None = None,
**mkcmd_kwargs,
) -> pty_spawn.spawn:
'''
Inner closure handed to consumer tests to invoke
`pytest.Pytester.spawn`
'''
nonlocal spawned
unset_colors()
set_spawn_method(start_method=start_method)
set_loglevel(
loglevel=loglevel,
# ?TODO^ when should this be set by `--ll <level>` ?
# by default we apply 'error' but there should be a diff
# vs. when the flag IS NOT passed?
)
spawned = testdir.spawn(
cmd=mk_cmd(
cmd,
@ -137,6 +192,14 @@ def spawn(
if ptyproc.isalive():
ptyproc.kill(signal.SIGKILL)
# Scope our env-var mutations to this single fixture invocation
# — both `TRACTOR_SPAWN_METHOD` and `TRACTOR_LOGLEVEL` are
# honored by `tractor._root.open_root_actor()` so leaking them
# past this test could inadvertently re-route a later in-process
# tractor test's spawn-backend / loglevel.
os.environ.pop('TRACTOR_SPAWN_METHOD', None)
os.environ.pop('TRACTOR_LOGLEVEL', None)
# TODO? ensure we've cleaned up any UDS-paths?
# breakpoint()
@ -146,24 +209,40 @@ def spawn(
ids='ctl-c={}'.format,
)
def ctlc(
request,
request: pytest.FixtureRequest,
ci_env: bool,
start_method: str,
) -> bool:
'''
Parametrize and optionally skip tests which handle
ctlc-in-`pdbp`-REPL testing scenarios; certain spawners and actor-tree depths
cope very poorly with this..
use_ctlc = request.param
In particular the spawning backends from `multiprocessing` are
fragile, as can be the default `trio` spawner under certain
conditions where SIGINT is relayed down the entire subproc tree.
'''
use_ctlc: bool = request.param
node = request.node
markers = node.own_markers
for mark in markers:
if mark.name == 'has_nested_actors':
if (
mark.name == 'has_nested_actors'
and
start_method not in {
# TODO, any spawners we should try again?
# - [ ] 'trio' but WITHOUT the SIGINT handler setup
# per subproc?
# 'main_thread_forkserver',
}
):
pytest.skip(
f'Test {node} has nested actors and fails with Ctrl-C.\n'
f'The test can sometimes run fine locally but until'
' we solve' 'this issue this CI test will be xfail:\n'
'https://github.com/goodboy/tractor/issues/320'
)
if (
mark.name == 'ctlcs_bish'
and
@ -190,13 +269,10 @@ def ctlc(
def expect(
child,
# normally a `pdb` prompt by default
patt: str,
patt: str, # often a `pdbp`-prompt
**kwargs,
) -> None:
) -> str:
'''
Expect wrapper that prints last seen console
data before failing.
@ -207,6 +283,8 @@ def expect(
patt,
**kwargs,
)
before = str(child.before.decode())
return before
except TIMEOUT:
before = str(child.before.decode())
print(before)
@ -261,10 +339,13 @@ def in_prompt_msg(
def assert_before(
child: SpawnBase,
patts: list[str],
**kwargs,
) -> str:
'''
Assert a patter is in `child.before.decode() -> str`,
return the full `.before` output on success.
) -> None:
'''
__tracebackhide__: bool = False
assert in_prompt_msg(
@ -275,7 +356,8 @@ def assert_before(
err_on_false=True,
**kwargs
)
return str(child.before.decode())
before: str = str(child.before.decode())
return before
def do_ctlc(

90
tests/devx/test_debugger.py
View File

@ -24,6 +24,7 @@ from pexpect.exceptions import (
TIMEOUT,
EOF,
)
import tractor
from .conftest import (
do_ctlc,
@ -343,6 +344,7 @@ def test_subactor_breakpoint(
def test_multi_subactors(
spawn,
ctlc: bool,
set_fork_aware_capture,
):
'''
Multiple subactors, both erroring and
@ -487,11 +489,12 @@ def test_multi_subactors(
def test_multi_daemon_subactors(
spawn,
loglevel: str,
ctlc: bool
ctlc: bool,
set_fork_aware_capture,
):
'''
Multiple daemon subactors, both erroring and breakpointing within a
stream.
Multiple daemon subactors, both erroring and breakpointing within
a stream.
'''
non_linux = _non_linux
@ -604,7 +607,10 @@ def test_multi_daemon_subactors(
child,
bp_forev_parts,
)
except AssertionError:
except (
# AssertionError, # TODO? rm since never raised?
ValueError,
):
before: str = assert_before(
child,
name_error_parts,
@ -765,6 +771,8 @@ def test_multi_subactors_root_errors(
def test_multi_nested_subactors_error_through_nurseries(
ci_env: bool,
spawn: PexpectSpawner,
is_forking_spawner: bool,
test_log: tractor.log.StackLevelAdapter,
# TODO: address debugger issue for nested tree:
# https://github.com/goodboy/tractor/issues/320
@ -781,16 +789,17 @@ def test_multi_nested_subactors_error_through_nurseries(
# A test (below) has now been added to explicitly verify this is
# fixed.
child = spawn('multi_nested_subactors_error_up_through_nurseries')
# timed_out_early: bool = False
child = spawn(
'multi_nested_subactors_error_up_through_nurseries',
loglevel='pdb',
)
last_send_char: str|None = None
for (
i,
send_char,
) in enumerate(itertools.cycle(['c', 'q'])):
timeout: float = -1
timeout: float = child.timeout
if (
_non_linux
and
@ -803,24 +812,36 @@ def test_multi_nested_subactors_error_through_nurseries(
elif i == 0:
timeout = 5
# XXX forking backends may take longer due to
# determinstic IPC cancellation.
if is_forking_spawner:
timeout += 4
try:
child.expect(
PROMPT,
timeout=timeout,
)
delay: float = 0.1
test_log.info('Sleeping {delay!r} before next send-chart..')
time.sleep(delay)
last_send_char: str = send_char
child.sendline(send_char)
time.sleep(0.01)
time.sleep(delay)
# script finally exited with tb on console.
except EOF:
test_log.info(
f'Breaking from send-char loop'
f'last_send_char: {last_send_char!r}\n'
)
break
assert_before(
child,
[ # boxed source errors
# boxed source errors
expect_patts: list[str] = [
"NameError: name 'doggypants' is not defined",
"tractor._exceptions.RemoteActorError:",
"('name_error'",
"bdb.BdbQuit",
# first level subtrees
# "tractor._exceptions.RemoteActorError: ('spawner0'",
@ -834,18 +855,39 @@ def test_multi_nested_subactors_error_through_nurseries(
# "tractor._exceptions.RemoteActorError: ('spawn_until_2'",
# ^-NOTE-^ old RAE repr, new one is below with a field
# showing the src actor's uid.
"src_uid=('spawn_until_0'",
"relay_uid=('spawn_until_1'",
"src_uid=('spawn_until_2'",
]
# XXX, I HAVE NO IDEA why these patts only show on the
# `trio`-spawner but it seems to have something to do with
# what gets dumped in prior-prompt latches somehow??
# TODO for claude, explain and or work through how this is
# happening but ONLY WHEN RUN FROM THE TEST, bc when i try to
# run the test script manually the correct output ALWAYS seems
# to be in the last `str(child.before.decode())` output !?!?
if (
not is_forking_spawner
and
last_send_char == 'q'
):
expect_patts += [
# expect the pdb-quit exc.
"bdb.BdbQuit",
# BUT WHY these dude!?
"src_uid=('spawn_until_0'",
"relay_uid=('spawn_until_1'",
]
assert_before(
child,
expect_patts,
)
expect(child, EOF)
@pytest.mark.timeout(15)
# @pytest.mark.timeout(15)
@has_nested_actors
def test_root_nursery_cancels_before_child_releases_tty_lock(
spawn,
start_method,
ctlc: bool,
):
'''
@ -1187,7 +1229,11 @@ def test_ctxep_pauses_n_maybe_ipc_breaks(
mashed and zombie reaper kills sub with no hangs.
'''
child = spawn('subactor_bp_in_ctx')
child = spawn(
'subactor_bp_in_ctx',
loglevel='devx'
# ^XXX REQUIRED for below patt matching!
)
child.expect(PROMPT)
# 3 iters for the `gen()` pause-points
@ -1277,7 +1323,11 @@ def test_crash_handling_within_cancelled_root_actor(
call.
'''
child = spawn('root_self_cancelled_w_error')
child = spawn(
'root_self_cancelled_w_error',
loglevel='cancel',
# ^XXX REQUIRED for below patt matching!
)
child.expect(PROMPT)
assert_before(

47
tests/devx/test_pause_from_non_trio.py
View File

@ -66,19 +66,28 @@ def test_pause_from_sync(
# XXX required for `breakpoint()` overload and
# thus`tractor.devx.pause_from_sync()`.
pytest.importorskip('greenback')
child = spawn('sync_bp')
child = spawn(
'sync_bp',
loglevel='pdb', # XXX pattern matching
)
# first `sync_pause()` after nurseries open
child.expect(PROMPT)
assert_before(
_before: str = assert_before(
child,
[
# pre-prompt line
_pause_msg,
"<Task '__main__.main'",
# devx-loglevel
# "imported <module 'greenback' from",
# "successfully scheduled `._pause()` in `trio` thread on behalf of <Task",
_pause_msg, # pre-prompt line
"('root'",
"<Task '__main__.main'",
"tractor.pause_from_sync()",
]
)
# XXX `enable_stack_on_sig=False` in script
assert 'stackscope' not in _before
if ctlc:
do_ctlc(child)
# ^NOTE^ subactor not spawned yet; don't need extra delay.
@ -88,18 +97,18 @@ def test_pause_from_sync(
# first `await tractor.pause()` inside `p.open_context()` body
child.expect(PROMPT)
# XXX shouldn't see gb loaded message with PDB loglevel!
# assert not in_prompt_msg(
# child,
# ['`greenback` portal opened!'],
# )
# should be same root task
assert_before(
child,
[
# XXX should see gb loaded with devx-loglevel.
# "`greenback` portal opened!",
# "Activated `greenback` for `tractor.pause_from_sync()` support!",
_pause_msg,
"<Task '__main__.main'",
"('root'",
"<Task '__main__.main'",
"tractor.pause()",
]
)
@ -130,17 +139,17 @@ def test_pause_from_sync(
# `Lock.acquire()`-ed
# (NOT both, which will result in REPL clobbering!)
attach_patts: dict[str, list[str]] = {
'subactor': [
"'start_n_sync_pause'",
"('subactor'",
"|_<Task 'start_n_sync_pause'": [
"|_('subactor'",
"tractor.pause_from_sync()",
],
'inline_root_bg_thread': [
"<Thread(inline_root_bg_thread",
"|_<Thread(inline_root_bg_thread": [
"('root'",
"breakpoint(hide_tb=hide_tb)",
],
'start_soon_root_bg_thread': [
"<Thread(start_soon_root_bg_thread",
"('root'",
"|_<Thread(start_soon_root_bg_thread": [
"|_('root'",
"tractor.pause_from_sync()",
],
}
conts: int = 0 # for debugging below matching logic on failure

170
tests/devx/test_proctitle.py 100644
View File

@ -0,0 +1,170 @@
'''
Tests for `tractor.devx._proctitle` (per-actor `setproctitle`)
and the intrinsic-signal sub-actor detection in
`tractor._testing._reap`.
The proctitle is set in `tractor._child._actor_child_main()`
after `Actor` construction, so any spawned sub-actor process
should:
- have `argv[0]` (== `/proc/<pid>/cmdline`) start with
`tractor[<aid.reprol()>]`
- have `/proc/<pid>/comm` start with `tractor[` (kernel
truncates to ~15 bytes)
- be detected as a tractor sub-actor by
`_is_tractor_subactor(pid)` via the cmdline marker.
`set_actor_proctitle()` itself is also unit-tested in-process
to verify the format string.
'''
from __future__ import annotations
import platform
import psutil
import pytest
import trio
import tractor
from tractor.runtime._runtime import Actor
from tractor.devx._proctitle import set_actor_proctitle
from tractor._testing._reap import (
_is_tractor_subactor,
_read_cmdline,
_read_comm,
)
_non_linux: bool = platform.system() != 'Linux'
def test_set_actor_proctitle_format():
'''
`set_actor_proctitle()` returns the canonical
`tractor[<aid.reprol()>]` form and actually mutates
the running proc's title.
'''
pytest.importorskip(
'setproctitle',
reason='`setproctitle` is an optional runtime dep',
)
import setproctitle
# save + restore so we don't pollute pytest's own title
saved: str = setproctitle.getproctitle()
try:
actor = Actor(
name='unit_test_actor',
uuid='1027301b-a0e3-430e-8806-a5279f21abe6',
)
title: str = set_actor_proctitle(actor)
# canonical wrapping: `tractor[<aid.reprol()>]`. We
# compare against the runtime-computed `reprol()`
# rather than a hard-coded value so the test stays
# decoupled from `Aid.reprol()`'s internal format
# (currently `<name>@<pid>`, but could evolve).
expected: str = f'tractor[{actor.aid.reprol()}]'
assert title == expected
# sanity: the actor's name must be in the title
# somewhere (so a future `reprol()` change that
# drops the name is also caught).
assert 'unit_test_actor' in title
# actually set on the running proc
assert setproctitle.getproctitle() == title
finally:
setproctitle.setproctitle(saved)
@pytest.mark.skipif(
_non_linux,
reason=(
'detection helpers read `/proc/<pid>/{cmdline,comm}` '
'which is Linux-specific'
),
)
def test_subactor_proctitle_visible_via_proc():
'''
Spawn a sub-actor and verify its proc-title is visible
via both `/proc/<pid>/cmdline` AND `/proc/<pid>/comm`,
AND that `_is_tractor_subactor()` correctly identifies
it.
'''
pytest.importorskip('setproctitle')
async def main() -> dict:
async with tractor.open_nursery() as an:
portal = await an.start_actor('proctitle_boi')
# let the child finish setproctitle in
# `_actor_child_main`
await trio.sleep(0.3)
# the sub-actor's pid is on the portal's chan
# repr; psutil-walk `me.children()` is simpler.
me = psutil.Process()
sub_pids: list[int] = [
p.pid for p in me.children(recursive=True)
]
assert sub_pids, (
'expected at least one spawned sub-actor pid'
)
results: dict = {}
for pid in sub_pids:
results[pid] = {
'cmdline': _read_cmdline(pid),
'comm': _read_comm(pid),
'is_tractor': _is_tractor_subactor(pid),
}
await portal.cancel_actor()
return results
found: dict = trio.run(main)
# at least one of the spawned procs should match the
# `proctitle_boi` actor we started; assert the proc-
# title shape on it specifically.
matched: list[tuple[int, dict]] = [
(pid, info)
for pid, info in found.items()
if 'proctitle_boi' in info['cmdline']
]
assert matched, (
f'no sub-actor pid had a `proctitle_boi` cmdline; '
f'all={found}'
)
pid, info = matched[0]
# canonical proctitle prefix in cmdline (full form)
assert info['cmdline'].startswith('tractor[proctitle_boi@'), (
f'cmdline missing `tractor[proctitle_boi@…]` prefix: '
f'{info["cmdline"]!r}'
)
# comm is kernel-truncated to ~15 bytes — just check the
# `tractor[` prefix made it.
assert info['comm'].startswith('tractor['), (
f'comm missing `tractor[` prefix: {info["comm"]!r}'
)
# intrinsic-signal detector should match.
assert info['is_tractor'] is True
@pytest.mark.skipif(
_non_linux,
reason='reads /proc/<pid>/{cmdline,comm}',
)
def test_is_tractor_subactor_negative():
'''
`_is_tractor_subactor()` returns False for non-tractor
procs (e.g. the pytest test-runner pid itself, which
is `python -m pytest …` — no `tractor[` proctitle, no
`tractor._child` cmdline).
'''
import os
assert _is_tractor_subactor(os.getpid()) is False

83
tests/devx/test_tooling.py
View File

@ -21,6 +21,7 @@ import os
import signal
import time
from typing import (
Callable,
TYPE_CHECKING,
)
@ -47,7 +48,12 @@ if TYPE_CHECKING:
@no_macos
def test_shield_pause(
spawn: PexpectSpawner,
spawn: Callable[
...,
PexpectSpawner,
],
start_method: str,
request: pytest.FixtureRequest,
):
'''
Verify the `tractor.pause()/.post_mortem()` API works inside an
@ -55,8 +61,10 @@ def test_shield_pause(
next checkpoint wherein the cancelled will get raised.
'''
child = spawn(
'shield_hang_in_sub'
child: PexpectSpawner = spawn(
'shield_hang_in_sub',
loglevel='devx',
# ^XXX REQUIRED for below patt matching!
)
expect(
child,
@ -86,31 +94,74 @@ def test_shield_pause(
# end-of-tree delimiter
"end-of-\('root'",
)
assert_before(
_before: str = assert_before(
child,
[
# 'Srying to dump `stackscope` tree..',
# 'Dumping `stackscope` tree for actor',
"('root'", # uid line
# TODO!? this used to show?
# TODO!? this in-task-code used to show??
# -[ ] mk reproducable for @oremanj?
# => SOLVED? by our `trio_token.run_sync_soon()`
# approach?
#
# parent block point (non-shielded)
# 'await trio.sleep_forever() # in root',
]
)
# NOTE, hierarchical-ordering invariant restored by
# `_dump_then_relay` (co-scheduled dump+relay on the
# trio loop, see `tractor.devx._stackscope`): the
# parent's full task-tree prints BEFORE the 'Relaying
# `SIGUSR1`' log msg, which prints BEFORE any sub-
# actor receives the signal and dumps its own tree.
# So the relay log appears BETWEEN `end-of-('root'`
# (above) and `end-of-('hanger'` (below).
handle_out_of_order: bool = False
# XXX, when capfd is NOT used we don't expect to
# see the logging output from the subactor.
if (no_capfd := (start_method in [
'main_thread_forkserver',
])
):
opts = request.config.option
assert opts.spawn_backend == start_method
# ?XXX? i guess the `testdir` fixture "pretends to" reset
# this to the default 'fd'??
# assert opts.capture in [
# 'sys',
# 'no',
# ]
if (
handle_out_of_order
and
"end-of-('hanger'" in _before
):
assert "('hanger'" in _before
assert 'Relaying `SIGUSR1`[10] to sub-actor' in _before
else:
_before = expect(
child,
'Relaying `SIGUSR1`\\[10\\] to sub-actor',
)
# _before: str = assert_before(
# child,
# ["('hanger'",] # uid line
# )
if not no_capfd:
expect(
child,
# end-of-tree delimiter
# end-of-subactor's-tree delimiter
"end-of-\('hanger'",
)
assert_before(
_before: str = assert_before(
child,
[
# relay to the sub should be reported
'Relaying `SIGUSR1`[10] to sub-actor',
"('hanger'", # uid line
# TODO!? SEE ABOVE
@ -119,6 +170,7 @@ def test_shield_pause(
]
)
# simulate the user sending a ctl-c to the hanging program.
# this should result in the terminator kicking in since
# the sub is shield blocking and can't respond to SIGINT.
@ -133,14 +185,19 @@ def test_shield_pause(
_shutdown_msg,
timeout=6,
)
assert_before(
child,
[
expect_on_teardown: list[str] = [
'raise KeyboardInterrupt',
'Root actor terminated',
]
if not no_capfd:
expect_on_teardown += [
# 'Shutting down actor runtime',
'#T-800 deployed to collect zombie B0',
"'--uid', \"('hanger',",
]
assert_before(
child,
expect_on_teardown,
)

223
tests/discovery/conftest.py 100644
View File

@ -0,0 +1,223 @@
'''
Discovery-suite fixtures, including the `daemon`
remote-registrar subprocess used by the multi-program
discovery tests.
Lives here (vs. the parent `tests/conftest.py`)
because `daemon` is a discovery-protocol primitive —
boots a separate `tractor.run_daemon()` process whose
sole purpose is to serve as a registrar peer for
discovery-roundtrip tests. Pytest fixtures inherit
DOWNWARD through conftest hierarchy, so anything
under `tests/discovery/` automatically picks this up.
'''
from __future__ import annotations
import os
import platform
import socket
import subprocess
import sys
import time
import pytest
import tractor
from ..conftest import (
sig_prog,
_INT_SIGNAL,
_non_linux,
)
def _wait_for_daemon_ready(
reg_addr: tuple,
tpt_proto: str,
*,
deadline: float = 10.0,
poll_interval: float = 0.05,
proc: subprocess.Popen|None = None,
) -> None:
'''
Active-poll the daemon's bind address until it
accepts a connection (proving it has called
`bind() + listen()` and is ready to handle IPC).
Replaces the historical blind `time.sleep()` in the
`daemon` fixture which was racy under load — see
`ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`.
Uses stdlib `socket` directly (no trio runtime
bootstrap cost) — sufficient because
`tractor.run_daemon()` doesn't return from
bootstrap until the runtime is fully ready to
accept IPC.
Raises `TimeoutError` on `deadline` exceeded. If
`proc` is given, ALSO raises early if the daemon
process exits non-zero before the deadline (catches
daemon-startup-crash that the blind sleep used to
silently mask).
'''
end: float = time.monotonic() + deadline
last_exc: Exception|None = None
while time.monotonic() < end:
# Daemon-died-during-startup early-exit. Without
# this, a crashed-on-import daemon would just
# eat the full deadline before raising opaque
# TimeoutError.
if proc is not None and proc.poll() is not None:
raise RuntimeError(
f'Daemon proc exited (rc={proc.returncode}) '
f'before becoming ready to accept on '
f'{reg_addr!r}'
)
try:
if tpt_proto == 'tcp':
# `socket.create_connection` does the
# `socket() + connect()` dance with a
# builtin timeout — perfect primitive
# for a one-shot probe.
with socket.create_connection(
reg_addr,
timeout=poll_interval,
):
return
else:
# UDS — `reg_addr` is a `(filedir, sockname)`
# tuple per `tractor.ipc._uds.UDSAddress.unwrap`.
sockpath: str = os.path.join(*reg_addr)
sock = socket.socket(socket.AF_UNIX)
try:
sock.settimeout(poll_interval)
sock.connect(sockpath)
return
finally:
sock.close()
except (
ConnectionRefusedError,
FileNotFoundError,
OSError,
socket.timeout,
) as exc:
last_exc = exc
time.sleep(poll_interval)
raise TimeoutError(
f'Daemon never accepted on {reg_addr!r} within '
f'{deadline}s (last connect-attempt exc: '
f'{last_exc!r})'
)
# TODO: factor into @cm and move to `._testing`?
@pytest.fixture
def daemon(
debug_mode: bool,
loglevel: str,
testdir: pytest.Pytester,
reg_addr: tuple[str, int],
tpt_proto: str,
ci_env: bool,
test_log: tractor.log.StackLevelAdapter,
) -> subprocess.Popen:
'''
Run a daemon root actor as a separate actor-process
tree and "remote registrar" for discovery-protocol
related tests.
'''
# XXX: too much logging will lock up the subproc (smh)
if loglevel in ('trace', 'debug'):
test_log.warning(
f'Test harness log level is too verbose: {loglevel!r}\n'
f'Reducing to INFO level..'
)
loglevel: str = 'info'
code: str = (
"import tractor; "
"tractor.run_daemon([], "
"registry_addrs={reg_addrs}, "
"enable_transports={enable_tpts}, "
"debug_mode={debug_mode}, "
"loglevel={ll})"
).format(
reg_addrs=str([reg_addr]),
enable_tpts=str([tpt_proto]),
ll="'{}'".format(loglevel) if loglevel else None,
debug_mode=debug_mode,
)
cmd: list[str] = [
sys.executable,
'-c', code,
]
kwargs = {}
if platform.system() == 'Windows':
# without this, tests hang on windows forever
kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
proc: subprocess.Popen = testdir.popen(
cmd,
**kwargs,
)
# Active-poll the daemon's bind address until it's
# ready to accept connections — replaces the legacy
# blind `time.sleep(2.2)` which was racy under load
# (see
# `ai/conc-anal/test_register_duplicate_name_daemon_connect_race_issue.md`).
#
# Per-test deadline scales with platform: macOS/CI
# gets extra headroom; Linux dev boxes need very
# little.
deadline: float = (
15.0 if (_non_linux and ci_env)
else 10.0
)
_wait_for_daemon_ready(
reg_addr=reg_addr,
tpt_proto=tpt_proto,
deadline=deadline,
proc=proc,
)
assert not proc.returncode
yield proc
sig_prog(proc, _INT_SIGNAL)
# XXX! yeah.. just be reaaal careful with this bc
# sometimes it can lock up on the `_io.BufferedReader`
# and hang..
#
# NB, drain happens at TEARDOWN (post-yield), so the
# test body has its chance to read `proc.stderr`
# FIRST. Reading here AFTER would silently swallow
# the daemon's stderr output and break tests that
# assert on it (e.g. `test_abort_on_sigint`).
stderr: str = proc.stderr.read().decode()
stdout: str = proc.stdout.read().decode()
if (
stderr
or
stdout
):
print(
f'Daemon actor tree produced output:\n'
f'{proc.args}\n'
f'\n'
f'stderr: {stderr!r}\n'
f'stdout: {stdout!r}\n'
)
if (rc := proc.returncode) != -2:
msg: str = (
f'Daemon actor tree was not cancelled !?\n'
f'proc.args: {proc.args!r}\n'
f'proc.returncode: {rc!r}\n'
)
if rc < 0:
raise RuntimeError(msg)
test_log.error(msg)

355
tests/discovery/test_multi_program.py 100644
View File

@ -0,0 +1,355 @@
"""
Multiple python programs invoking the runtime.
"""
from __future__ import annotations
import platform
import subprocess
import time
from typing import (
TYPE_CHECKING,
)
import pytest
import trio
import tractor
from tractor._testing import (
tractor_test,
)
from tractor import (
current_actor,
Actor,
Context,
Portal,
)
from tractor.runtime import _state
from ..conftest import (
sig_prog,
_INT_SIGNAL,
_INT_RETURN_CODE,
)
if TYPE_CHECKING:
from tractor.msg import Aid
from tractor.discovery._addr import (
UnwrappedAddress,
)
_non_linux: bool = platform.system() != 'Linux'
# NOTE, multi-program tests historically triggered both
# UDS sock-file leaks (daemon-subproc SIGKILL paths) AND
# trio `WakeupSocketpair.drain()` busy-loops
# (`test_register_duplicate_name`). Track + detect
# per-test as a regression net.
pytestmark = pytest.mark.usefixtures(
'track_orphaned_uds_per_test',
'detect_runaway_subactors_per_test',
)
def test_abort_on_sigint(
daemon: subprocess.Popen,
):
assert daemon.returncode is None
time.sleep(0.1)
sig_prog(daemon, _INT_SIGNAL)
assert daemon.returncode == _INT_RETURN_CODE
# XXX: oddly, couldn't get capfd.readouterr() to work here?
if platform.system() != 'Windows':
# don't check stderr on windows as its empty when sending CTRL_C_EVENT
assert "KeyboardInterrupt" in str(daemon.stderr.read())
@tractor_test
async def test_cancel_remote_registrar(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
):
assert not current_actor().is_registrar
async with tractor.get_registry(reg_addr) as portal:
await portal.cancel_actor()
time.sleep(0.1)
# the registrar channel server is cancelled but not its main task
assert daemon.returncode is None
# no registrar socket should exist
with pytest.raises(OSError):
async with tractor.get_registry(reg_addr) as portal:
pass
def test_register_duplicate_name(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
):
# bug-class-3 breadcrumbs: the *last* `[CANCEL]` line that
# appears under `--ll cancel`/`TRACTOR_LOG_FILE=...` names the
# cancel-cascade boundary that's parked. Pair with
# `_trio_main` entry/exit breadcrumbs in
# `tractor/spawn/_entry.py` to triangulate the swallow point.
log = tractor.log.get_logger('tractor.tests.test_multi_program')
async def main():
log.cancel('test_register_duplicate_name: enter `main()`')
try:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
log.cancel(
'test_register_duplicate_name: '
'actor nursery opened'
)
assert not current_actor().is_registrar
p1 = await an.start_actor('doggy')
log.cancel(
'test_register_duplicate_name: '
'spawned doggy #1'
)
p2 = await an.start_actor('doggy')
log.cancel(
'test_register_duplicate_name: '
'spawned doggy #2'
)
async with tractor.wait_for_actor('doggy') as portal:
log.cancel(
'test_register_duplicate_name: '
'`wait_for_actor` returned'
)
assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
log.cancel(
'test_register_duplicate_name: '
'ABOUT TO CALL `an.cancel()`'
)
await an.cancel()
log.cancel(
'test_register_duplicate_name: '
'`an.cancel()` returned'
)
finally:
log.cancel(
'test_register_duplicate_name: '
'`open_nursery.__aexit__` returned, leaving `main()`'
)
# XXX, run manually since we want to start this root **after**
# the other "daemon" program with it's own root.
trio.run(main)
# `n_dups` in {4, 8} both expose the SAME pre-existing race:
# under rapid same-name spawning against a forkserver +
# registrar, ONE of the spawned doggies `sys.exit(2)`s during
# boot before completing parent-handshake. Surfaces now (post
# the spawn-time `wait_for_peer_or_proc_death` fix) as
# `ActorFailure rc=2`; previously it was silently masked by
# the handshake-wait parking forever.
#
# Larger `n_dups` widens the race window so the boot-race
# fires more often — n_dups=4 hits ~always, n_dups=8 hits
# occasionally. Both xfail(strict=False) so the cancel-cascade
# regression-check still passes when the boot-race happens
# NOT to fire.
#
# Tracked separately in,
# https://github.com/goodboy/tractor/issues/456
_DOGGY_BOOT_RACE_XFAIL = pytest.mark.xfail(
strict=False,
reason=(
'doggy boot-race rc=2 under rapid same-name '
'spawn — separate bug from cancel-cascade'
),
)
@pytest.mark.parametrize(
'n_dups',
[
2,
pytest.param(4, marks=_DOGGY_BOOT_RACE_XFAIL),
pytest.param(8, marks=_DOGGY_BOOT_RACE_XFAIL),
],
ids=lambda n: f'n_dups={n}',
)
def test_dup_name_cancel_cascade_escalates_to_hard_kill(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
n_dups: int,
):
'''
Regression for the duplicate-name cancel-cascade hang under
`tcp+main_thread_forkserver`.
When N actors share a single name and the parent calls
`an.cancel()`, the daemon registrar gets N `register_actor` RPCs
in tight succession. Under TCP+MTF, kernel-level socket-buffer
contention can push at least one sub-actor's cancel-RPC ack past
`Portal.cancel_timeout` (default 0.5s).
Pre-fix, `Portal.cancel_actor()` silently returned `False` on
that timeout, the supervisor's outer `move_on_after(3)` never
fired (each per-portal task always returned ≤0.5s, never
exceeded 3s), and `soft_kill()`'s `await wait_func(proc)` parked
forever — deadlocking nursery `__aexit__`.
Post-fix, `Portal.cancel_actor()` raises `ActorTooSlowError` on
the bounded-wait timeout, and `ActorNursery.cancel()`'s
per-child wrapper escalates to `proc.terminate()` (hard-kill).
The full nursery teardown therefore stays bounded even under
pathological timing.
`n_dups` is parametrized to widen the race window — more
same-name siblings = more concurrent register-RPCs at the
daemon = higher probability of hitting the contention path.
'''
log = tractor.log.get_logger(
'tractor.tests.test_multi_program'
)
# outer hard ceiling: a regression should fail-fast, NOT hang
# the test session for minutes. Budget scales with `n_dups`
# since each extra same-name sibling adds ~spawn-cost +
# potential cancel-ack-timeout escalation latency under
# TCP+forkserver. ~5s/sibling + 15s baseline gives plenty of
# headroom while still failing-loud on a real hang.
fail_after_s: int = 15 + (5 * n_dups)
async def main():
log.cancel(
f'enter `main()` n_dups={n_dups}'
)
with trio.fail_after(fail_after_s):
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
portals: list[Portal] = []
for i in range(n_dups):
p: Portal = await an.start_actor('doggy')
portals.append(p)
log.cancel(
f'spawned doggy #{i + 1}/{n_dups}'
)
# at least one of the N must be discoverable by
# name; doesn't matter which one (registrar will
# have last-wins semantics under same-name).
async with tractor.wait_for_actor('doggy') as portal:
expected_uids = {p.channel.uid for p in portals}
assert portal.channel.uid in expected_uids
# critical section: this MUST return within
# `fail_after_s` even when one or more cancel-RPC
# acks time out. Pre-fix, this hangs forever.
log.cancel('about to call `an.cancel()`')
await an.cancel()
log.cancel('`an.cancel()` returned')
# post-teardown sanity: every child proc must be reaped.
# If escalation worked, even timed-out cancel-RPCs would
# have triggered `proc.terminate()` and the procs are dead.
for p in portals:
# `Portal.channel.connected()` -> False once the
# underlying chan disconnected (clean exit OR
# hard-killed proc both produce disconnect).
assert not p.channel.connected(), (
f'Portal chan still connected post-teardown?\n'
f'{p.channel}'
)
trio.run(main)
@tractor.context
async def get_root_portal(
ctx: Context,
):
'''
Connect back to the root actor manually (using `._discovery` API)
and ensure it's contact info is the same as our immediate parent.
'''
sub: Actor = current_actor()
rtvs: dict = _state._runtime_vars
raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
# await tractor.pause()
# XXX, in case the sub->root discovery breaks you might need
# this (i know i did Xp)!!
# from tractor.devx import mk_pdb
# mk_pdb().set_trace()
assert (
len(raddrs) == 1
and
list(sub._parent_chan.raddr.unwrap()) in raddrs
)
# connect back to our immediate parent which should also
# be the actor-tree's root.
from tractor.discovery._api import get_root
ptl: Portal
async with get_root() as ptl:
root_aid: Aid = ptl.chan.aid
parent_ptl: Portal = current_actor().get_parent()
assert (
root_aid.name == 'root'
and
parent_ptl.chan.aid == root_aid
)
await ctx.started()
def test_non_registrar_spawns_child(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
loglevel: str,
debug_mode: bool,
ci_env: bool,
):
'''
Ensure a non-regristar (serving) root actor can spawn a sub and
that sub can connect back (manually) to it's rent that is the
root without issue.
More or less this audits the global contact info in
`._state._runtime_vars`.
'''
async def main():
# XXX, since apparently on macos in GH's CI it can be a race
# with the `daemon` registrar on grabbing the socket-addr..
if ci_env and _non_linux:
await trio.sleep(.5)
async with tractor.open_nursery(
registry_addrs=[reg_addr],
loglevel=loglevel,
debug_mode=debug_mode,
) as an:
actor: Actor = tractor.current_actor()
assert not actor.is_registrar
sub_ptl: Portal = await an.start_actor(
name='sub',
enable_modules=[__name__],
)
async with sub_ptl.open_context(
get_root_portal,
) as (ctx, _):
print('Waiting for `sub` to connect back to us..')
await an.cancel()
# XXX, run manually since we want to start this root **after**
# the other "daemon" program with it's own root.
trio.run(main)

152
tests/discovery/test_registrar.py
View File

@ -14,6 +14,7 @@ import psutil
import pytest
import subprocess
import tractor
from tractor.devx import dump_on_hang
from tractor.trionics import collapse_eg
from tractor._testing import tractor_test
from tractor.discovery._addr import wrap_address
@ -21,6 +22,20 @@ from tractor.discovery._multiaddr import mk_maddr
import trio
pytestmark = pytest.mark.usefixtures(
'reap_subactors_per_test',
# NOTE, registrar tests stress the discovery
# roundtrip (find_actor / wait_for_actor) which
# historically left orphaned UDS sock-files when
# subactor `hard_kill` SIGKILL'd, and which
# exercises the same trio `WakeupSocketpair`
# peer-disconnect path that triggered the
# busy-loop bug class.
'track_orphaned_uds_per_test',
'detect_runaway_subactors_per_test',
)
@tractor_test
async def test_reg_then_unreg(
reg_addr: tuple,
@ -105,19 +120,6 @@ async def hi():
return the_line.format(tractor.current_actor().name)
async def say_hello(
other_actor: str,
reg_addr: tuple[str, int],
):
await trio.sleep(1) # wait for other actor to spawn
async with tractor.find_actor(
other_actor,
registry_addrs=[reg_addr],
) as portal:
assert portal is not None
return await portal.run(__name__, 'hi')
async def say_hello_use_wait(
other_actor: str,
reg_addr: tuple[str, int],
@ -131,14 +133,17 @@ async def say_hello_use_wait(
return result
@tractor_test
@tractor_test(
timeout=7,
)
@pytest.mark.parametrize(
'func',
[say_hello,
say_hello_use_wait]
'ria_fn',
[
say_hello_use_wait,
]
)
async def test_trynamic_trio(
func: Callable,
ria_fn: Callable,
start_method: str,
reg_addr: tuple,
):
@ -151,13 +156,13 @@ async def test_trynamic_trio(
print("Alright... Action!")
donny = await n.run_in_actor(
func,
ria_fn,
other_actor='gretchen',
reg_addr=reg_addr,
name='donny',
)
gretchen = await n.run_in_actor(
func,
ria_fn,
other_actor='donny',
reg_addr=reg_addr,
name='gretchen',
@ -319,6 +324,14 @@ async def spawn_and_check_registry(
assert actor.aid.uid in registry
async def with_timeout(
main: Callable,
timeout: float = 6,
):
with trio.fail_after(timeout):
await main()
@pytest.mark.parametrize('use_signal', [False, True])
@pytest.mark.parametrize('with_streaming', [False, True])
def test_subactors_unregister_on_cancel(
@ -335,6 +348,7 @@ def test_subactors_unregister_on_cancel(
'''
with pytest.raises(KeyboardInterrupt):
trio.run(
# with_timeout,
partial(
spawn_and_check_registry,
reg_addr,
@ -364,6 +378,7 @@ def test_subactors_unregister_on_cancel_remote_daemon(
'''
with pytest.raises(KeyboardInterrupt):
trio.run(
with_timeout,
partial(
spawn_and_check_registry,
reg_addr,
@ -515,12 +530,43 @@ async def kill_transport(
# ?TODO, do a OSc style signalling test on this?
# -[ ] doesn't work for fork backends
# @pytest.mark.parametrize('use_signal', [False, True])
#
# Wall-clock bound via `pytest-timeout` (`method='thread'`).
# Under `--spawn-backend=subint` this test can wedge in an
# un-Ctrl-C-able state (abandoned-subint + shared-GIL
# starvation → signal-wakeup-fd pipe fills → SIGINT silently
# dropped; see `ai/conc-anal/subint_sigint_starvation_issue.md`).
# `method='thread'` is specifically required because `signal`-
# method SIGALRM suffers the same GIL-starvation path and
# wouldn't fire the Python-level handler.
# At timeout the plugin hard-kills the pytest process — that's
# the intended behavior here; the alternative is an unattended
# suite run that never returns.
# @pytest.mark.timeout(
# 30,
# # NOTE should be a 2.1s happy path.
# # XXX for `main_thread_forkserver` this is SUPER SENSITIVE
# # so keep it higher to avoid flaky runs..
# method='thread',
# )
@pytest.mark.skipon_spawn_backend(
'subint',
# 'main_thread_forkserver',
reason=(
'XXX SUBINT HANGING TEST XXX\n'
'See outstanding issue(s)\n'
# TODO, put issue link!
)
)
def test_stale_entry_is_deleted(
debug_mode: bool,
daemon: subprocess.Popen,
start_method: str,
reg_addr: tuple,
# set_fork_aware_capture,
):
'''
Ensure that when a stale entry is detected in the registrar's
@ -529,7 +575,6 @@ def test_stale_entry_is_deleted(
'''
async def main():
name: str = 'transport_fails_actor'
_reg_ptl: tractor.Portal
an: tractor.ActorNursery
@ -562,4 +607,67 @@ def test_stale_entry_is_deleted(
await ptl.cancel_actor()
await an.cancel()
trio.run(main)
# XXX, for tracing if this starts being flaky again..
#
timeout: float = 4
async def _timeout_main():
with trio.move_on_after(timeout) as cs:
await main()
if (
cs.cancel_called
and
debug_mode
):
await tractor.pause()
# TODO, remove once the `[subint]` variant no longer hangs.
#
# Status (as of Phase B hard-kill landing):
#
# - `[trio]`/`[mp_*]` variants: completes normally; `dump_on_hang`
# is a no-op safety net here.
#
# - `[subint]` variant: hangs indefinitely AND is un-Ctrl-C-able.
# `strace -p <pytest_pid>` while in the hang reveals a silently-
# dropped SIGINT — the C signal handler tries to write the
# signum byte to Python's signal-wakeup fd and gets `EAGAIN`,
# meaning the pipe is full (nobody's draining it).
#
# Root-cause chain: our hard-kill in `spawn._subint` abandoned
# the driver OS-thread (which is `daemon=True`) after the soft-
# kill timeout, but the *sub-interpreter* inside that thread is
# still running `trio.run()` — `_interpreters.destroy()` can't
# force-stop a running subint (raises `InterpreterError`), and
# legacy-config subints share the main GIL. The abandoned subint
# starves the parent's trio event loop from iterating often
# enough to drain its wakeup pipe → SIGINT silently drops.
#
# This is structurally a CPython-level limitation: there's no
# public force-destroy primitive for a running subint. We
# escape on the harness side via a SIGINT-loop in the `daemon`
# fixture teardown (killing the bg registrar subproc closes its
# end of the IPC, which eventually unblocks a recv in main trio,
# which lets the loop drain the wakeup pipe). Long-term fix path:
# msgspec PEP 684 support (jcrist/msgspec#563) → isolated-mode
# subints with per-interp GIL.
#
# Full analysis:
# `ai/conc-anal/subint_sigint_starvation_issue.md`
#
# See also the *sibling* hang class documented in
# `ai/conc-anal/subint_cancel_delivery_hang_issue.md` — same
# subint backend, different root cause (Ctrl-C-able hang, main
# trio loop iterating fine; ours to fix, not CPython's).
# Reproduced by `tests/test_subint_cancellation.py
# ::test_subint_non_checkpointing_child`.
#
# Kept here (and not behind a `pytestmark.skip`) so we can still
# inspect the dump file if the hang ever returns after a refactor.
# `pytest`'s stderr capture eats `faulthandler` output otherwise,
# so we route `dump_on_hang` to a file.
with dump_on_hang(
seconds=timeout*2,
path=f'/tmp/test_stale_entry_is_deleted_{start_method}.dump',
):
trio.run(_timeout_main)

25
tests/ipc/test_multi_tpt.py
View File

@ -59,15 +59,18 @@ async def chk_tpts(
)
def test_root_passes_tpt_to_sub(
tpt_proto_key: str,
tpt_proto: str,
reg_addr: tuple,
debug_mode: bool,
):
# XXX NOTE, the `reg_addr` addr won't be the same type as the
# `tpt_proto_key` would deliver here unless you pass `--tpt-proto
# <tpt_proto_key>` on the CLI.
#
# if tpt_proto_key == 'uds':
# breakpoint()
# `reg_addr` is sourced from the CLI `--tpt-proto={tpt_proto}`,
# so when the parametrized `tpt_proto_key` differs, the test
# asks the runtime to `enable_transports=[<other_proto>]` while
# pointing `registry_addrs` at a `reg_addr` of the wrong proto.
# The layer-2 guard in `open_root_actor` is expected to fail
# fast with `ValueError` on this mismatch (rather than the prior
# silent hang during the registrar handshake).
proto_mismatch: bool = (tpt_proto_key != tpt_proto)
async def main():
async with tractor.open_nursery(
@ -99,4 +102,14 @@ def test_root_passes_tpt_to_sub(
# shudown sub-actor(s)
await an.cancel()
if proto_mismatch:
# mismatched proto must raise `ValueError` from the
# `open_root_actor` runtime guard before any subactor spawn.
with pytest.raises(ValueError) as excinfo:
trio.run(main)
msg: str = str(excinfo.value)
assert 'enable_transports' in msg
assert 'registry_addrs' in msg
assert tpt_proto_key in msg or tpt_proto in msg
else:
trio.run(main)

24
tests/msg/test_ext_types_msgspec.py
View File

@ -57,6 +57,7 @@ from tractor.msg._ops import (
limit_plds,
)
def enc_nsp(obj: Any) -> Any:
actor: Actor = tractor.current_actor(
err_on_no_runtime=False,
@ -617,6 +618,17 @@ def test_ext_types_over_ipc(
debug_mode: bool,
pld_spec: Union[Type],
add_hooks: bool,
set_fork_aware_capture,
# ^^XXX? for forking spawners
# capfd: pytest.CaptureFixture,
# ^^NOTE, super interesting that if
# we disable this below then the tpt-layer
# suffers as an "unclean EOF"??
# ?TODO, determine why/how that mks sense when addressing,
# https://github.com/pytest-dev/pytest/issues/14444
#
):
'''
Ensure we can support extension types coverted using
@ -725,18 +737,26 @@ def test_ext_types_over_ipc(
await p.cancel_actor()
async def fa_main():
with (
trio.fail_after(2),
# ?TODO, investigate? see NOTE above..
# capfd.disabled(),
):
await main()
if (
NamespacePath in pld_types
and
add_hooks
):
trio.run(main)
trio.run(fa_main)
else:
with pytest.raises(
expected_exception=tractor.RemoteActorError,
) as excinfo:
trio.run(main)
trio.run(fa_main)
exc = excinfo.value
# bc `.started(nsp: NamespacePath)` will raise

32
tests/msg/test_pldrx_limiting.py
View File

@ -55,12 +55,37 @@ async def maybe_expect_raises(
raises: BaseException|None = None,
ensure_in_message: list[str]|None = None,
post_mortem: bool = False,
timeout: int = 3,
# NOTE, `None` selects a backend-aware default below —
# see `_BACKEND_TIMEOUT_DEFAULTS` for rationale. Caller
# can override with an explicit value to opt out.
timeout: int|None = None,
) -> None:
'''
Async wrapper for ensuring errors propagate from the inner scope.
'''
if timeout is None:
# Pick a backend-aware default. Fork-based backends
# (`main_thread_forkserver`) need much more headroom
# because actor spawn + IPC ctx-exit + msg-validation
# error path takes longer than under `trio` backend
# — especially under cross-pytest-stream contention
# (#451). `test_basic_payload_spec` empirically:
# - 3s flaked all-valid variant (`TooSlowError`)
# - 8s flaked `invalid-return` variant
# (`Cancelled` surfaced instead of `MsgTypeError`
# because `fail_after` fired mid-error-path)
# - 15s flaked under cross-stream contention
# 30s for fork-based gives plenty of headroom while
# still failing-loud on a genuine hang. Other
# backends keep the original 3s.
from tractor.spawn import _spawn as _spawn_mod
timeout = (
30
if _spawn_mod._spawn_method == 'main_thread_forkserver'
else 3
)
if tractor.debug_mode():
timeout += 999
@ -259,6 +284,11 @@ def test_basic_payload_spec(
return_value: str|None,
started_value: int|PldMsg,
pld_check_started_value: bool,
set_fork_aware_capture,
# ^XXX TODO? for forking spawners, seems to prevent hangs when
# --capture=sys not set, but only for a while then the problem
# accumulates?
):
'''
Validate the most basic `PldRx` msg-type-spec semantics around

192
tests/test_advanced_streaming.py
View File

@ -5,10 +5,15 @@ Advanced streaming patterns using bidirectional streams and contexts.
from collections import Counter
import itertools
import platform
from typing import Type
import pytest
import trio
import tractor
from tractor._testing.trace import (
AfkAlarmWTraceFactory,
FailAfterWTraceFactory,
)
def is_win():
@ -76,9 +81,7 @@ async def subscribe(
async def consumer(
subs: list[str],
) -> None:
uid = tractor.current_actor().uid
@ -108,15 +111,109 @@ async def consumer(
print(f'{uid} got: {value}')
def test_dynamic_pub_sub():
# NOTE: deliberately NOT using `@pytest.mark.timeout(...)` —
# both pytest-timeout enforcement modes break trio under
# fork-based backends:
#
# - `method='signal'` (SIGALRM): the handler synchronously
# raises `Failed` in trio's main thread mid-`epoll.poll()`,
# leaves `GLOBAL_RUN_CONTEXT` half-installed ("Trio guest
# run got abandoned"), and EVERY subsequent `trio.run()`
# in the same pytest process bails with
# `RuntimeError: Attempted to call run() from inside a
# run()` — session-wide poison.
#
# - `method='thread'`: calls `_thread.interrupt_main()`
# raising `KeyboardInterrupt` into the main thread. Under
# fork-based backends with mid-cascade fd-juggling the KBI
# can escape trio's `KIManager` and bubble out of pytest
# itself — kills the WHOLE session.
#
# Instead we use `trio.fail_after()` INSIDE `main()` below:
# trio's own `Cancelled`/`TooSlowError` machinery handles the
# timeout, cleanly unwinds the actor nursery's cancel
# cascade, and only fails the single test (no cross-test
# state corruption either way).
#
# `pyproject.toml`'s default `timeout = 200` is still a
# last-resort safety net.
@pytest.mark.parametrize(
'expect_cancel_exc', [
KeyboardInterrupt,
trio.TooSlowError,
],
ids=lambda item:
f'expect_user_exc_raised={item.__name__}'
)
def test_dynamic_pub_sub(
reg_addr: tuple,
debug_mode: bool,
test_log: tractor.log.StackLevelAdapter,
reap_subactors_per_test: int,
expect_cancel_exc: Type[BaseException],
is_forking_spawner: bool,
set_fork_aware_capture,
fail_after_w_trace: FailAfterWTraceFactory,
afk_alarm_w_trace: AfkAlarmWTraceFactory,
):
failed_to_raise_report: str = (
f'Never got a {expect_cancel_exc!r} ??'
)
global _registry
from multiprocessing import cpu_count
cpus = cpu_count()
# Hard safety cap via trio's own cancellation. NOTE see the
# module-level note on why we avoid `pytest-timeout` for this
# test. Picked backend-aware: under `trio` backend spawn is
# cheap (~1s for `cpus` actors) but fork-based backends pay
# a per-spawn cost (forkserver round-trip + IPC peer-handshake)
# that can stack up over `cpus - 1` sequential `n.run_in_actor()`
# calls — especially on UDS under cross-pytest contention
# (#451 / #452). 4s was flaking right at the edge under fork
# backends — bumped to 8s with diag-snapshot-on-timeout via
# `fail_after_w_trace` so a borderline run still fails loud
# but lands a ptree/wchan/py-spy dump in
# `$XDG_CACHE_HOME/tractor/hung-dumps/` for inspection.
#
# XXX caveat: this is an *inner* trio cancel — its `Cancelled`
# cannot reach a task parked in a shielded `await` (e.g. inside
# actor-nursery teardown). When the in-band cancel path is
# itself buggy (the bug-class-3 `raise KBI` swallow we're
# currently chasing) this guard does NOT fire and the test
# sits forever until external SIGINT. The `afk_alarm_w_trace`
# outer guard below is the AFK-safety counterpart (SIGALRM
# raises in the main thread regardless of trio scope state).
fail_after_s: int = (
8
if is_forking_spawner
else 20
)
async def main():
async with tractor.open_nursery() as n:
# bug-class-3 breadcrumb: tag each level of the cancel path
# so when the run hangs and we capture cancel-level logs, the
# *last* breadcrumb that fired names the swallow point.
test_log.cancel('test_dynamic_pub_sub: enter main()')
try:
async with fail_after_w_trace(fail_after_s):
test_log.cancel(
f'test_dynamic_pub_sub: '
f'enter `fail_after_w_trace({fail_after_s})` scope'
)
try:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as n:
test_log.cancel(
'test_dynamic_pub_sub: '
'actor nursery opened'
)
# name of this actor will be same as target func
await n.run_in_actor(publisher)
@ -138,29 +235,69 @@ def test_dynamic_pub_sub():
subs=list(_registry.keys()),
)
# block until cancelled by user
with trio.fail_after(3):
await trio.sleep_forever()
# block until "cancelled by user"
await trio.sleep(3)
test_log.warning(
f'Raising user cancel exc: '
f'{expect_cancel_exc!r}'
)
test_log.cancel(
f'test_dynamic_pub_sub: '
f'ABOUT TO RAISE {expect_cancel_exc!r}'
)
raise expect_cancel_exc('simulate user cancel!')
finally:
test_log.cancel(
'test_dynamic_pub_sub: '
'actor nursery `__aexit__` returned'
)
test_log.cancel(
'test_dynamic_pub_sub: `fail_after` scope exited'
)
finally:
test_log.cancel(
'test_dynamic_pub_sub: leaving `main()`'
)
def _run_and_match():
try:
trio.run(main)
except (
trio.TooSlowError,
ExceptionGroup,
) as err:
if isinstance(err, ExceptionGroup):
for suberr in err.exceptions:
if isinstance(suberr, trio.TooSlowError):
break
pytest.fail(failed_to_raise_report)
except expect_cancel_exc:
# parent-side raised the user-cancel exc directly and
# it propagated unwrapped; clean path.
test_log.exception('Got user-cancel exc AS EXPECTED')
except BaseExceptionGroup as err:
# under fork-based backends the user-raised cancel
# can race with subactor-side stream teardown
# (`trio.EndOfChannel` from a publisher's `send()`
# whose remote half got cut). The expected exc may
# then be nested deeper in the group rather than at
# the top level. `BaseExceptionGroup.split()` walks
# the exc tree recursively (Python 3.11+).
matched, _ = err.split(expect_cancel_exc)
if matched is None:
pytest.fail(failed_to_raise_report)
test_log.exception('Got user-cancel exc AS EXPECTED')
# outer SIGALRM-based guard — survives a shielded-await
# deadlock since `signal.alarm` raises in the main thread
# regardless of trio's scope state, AND captures a full diag
# snapshot to `$XDG_CACHE_HOME/tractor/hung-dumps/` before
# re-raising. ONLY armed under fork-based backends since the
# bug we're chasing is MTF-specific. Cap = `fail_after_s + 5`
# so the trio-native path always wins when it works.
if is_forking_spawner:
with afk_alarm_w_trace(fail_after_s + 5):
_run_and_match()
else:
pytest.fail('Never got a `TooSlowError` ?')
_run_and_match()
@tractor.context
async def one_task_streams_and_one_handles_reqresp(
ctx: tractor.Context,
) -> None:
await ctx.started()
@ -257,7 +394,8 @@ async def echo_ctx_stream(
def test_sigint_both_stream_types():
'''Verify that running a bi-directional and recv only stream
'''
Verify that running a bi-directional and recv only stream
side-by-side will cancel correctly from SIGINT.
'''
@ -287,9 +425,11 @@ def test_sigint_both_stream_types():
assert resp == msg
raise KeyboardInterrupt
# TODO, use pytest.raises() here instead?
# (why weren't we originally?)
try:
trio.run(main)
assert 0, "Didn't receive KBI!?"
pytest.fail("Didn't receive KBI!?")
except KeyboardInterrupt:
pass
@ -356,7 +496,12 @@ async def inf_streamer(
print('streamer exited .open_streamer() block')
# @pytest.mark.timeout(
# 6,
# method='signal',
# )
def test_local_task_fanout_from_stream(
reg_addr: tuple,
debug_mode: bool,
):
'''
@ -421,4 +566,9 @@ def test_local_task_fanout_from_stream(
await p.cancel_actor()
trio.run(main)
async def w_timeout():
with trio.fail_after(6):
await main()
# trio.run(main)
trio.run(w_timeout)

328
tests/test_cancellation.py
View File

@ -7,6 +7,7 @@ import signal
import platform
import time
from itertools import repeat
from typing import Type
import pytest
import trio
@ -14,6 +15,7 @@ import tractor
from tractor._testing import (
tractor_test,
)
from tractor._testing.trace import FailAfterWTraceFactory
from .conftest import no_windows
@ -21,6 +23,46 @@ _non_linux: bool = platform.system() != 'Linux'
_friggin_windows: bool = platform.system() == 'Windows'
pytestmark = [
# Multi-actor cancel cascades under
# `--spawn-backend=subint` trip the abandoned-subint
# GIL-hostage class — a stuck subint can starve the
# parent's trio loop and block cancel-delivery.
# Apply the skip module-wide rather than per-test
# since every test here exercises the same cascade.
pytest.mark.skipon_spawn_backend(
'subint',
reason=(
'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
'Cancel cascades under '
'`--spawn-backend=subint` trip the abandoned-subint '
'GIL-hostage class — see\n'
' - `ai/conc-anal/subint_sigint_starvation_issue.md` '
'(GIL-hostage, SIGINT-unresponsive)\n'
' - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
'(sibling: parent parks on dead chan)\n'
' - https://github.com/goodboy/tractor/issues/379 '
'(subint umbrella)\n'
)
),
pytest.mark.usefixtures(
'reap_subactors_per_test',
# NOTE, cancellation tests stress the SIGKILL
# `hard_kill` path which leaks UDS sock-files when
# the subactor's IPC server `finally:` cleanup
# doesn't run. Track per-test for blame attribution.
'track_orphaned_uds_per_test',
# NOTE, cancel-cascade timing races (see
# `test_nested_multierrors`) can also leave a
# subactor spinning at 100% CPU when its cancel
# signal got swallowed mid-handshake. Catches the
# runaway-loop class that doesn't leak UDS socks
# but burns the box.
'detect_runaway_subactors_per_test',
),
]
async def assert_err(delay=0):
await trio.sleep(delay)
assert 0
@ -45,7 +87,11 @@ async def do_nuthin():
],
ids=['no_args', 'unexpected_args'],
)
def test_remote_error(reg_addr, args_err):
def test_remote_error(
reg_addr: tuple,
args_err: tuple[dict, Type[Exception]],
set_fork_aware_capture,
):
'''
Verify an error raised in a subactor that is propagated
to the parent nursery, contains the underlying boxed builtin
@ -112,6 +158,8 @@ def test_remote_error(reg_addr, args_err):
def test_multierror(
reg_addr: tuple[str, int],
start_method: str, # parametrized
set_fork_aware_capture, #: Callable,
):
'''
Verify we raise a ``BaseExceptionGroup`` out of a nursery where
@ -141,16 +189,39 @@ def test_multierror(
trio.run(main)
@pytest.mark.parametrize('delay', (0, 0.5))
@pytest.mark.parametrize(
'num_subactors', range(25, 26),
'delay',
(0, 0.5),
ids='delays={}'.format,
)
def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
"""Verify we raise a ``BaseExceptionGroup`` out of a nursery where
@pytest.mark.parametrize(
'num_subactors',
range(25, 26),
ids= 'num_subs={}'.format,
)
def test_multierror_fast_nursery(
reg_addr: tuple,
start_method: str,
num_subactors: int,
delay: float,
set_fork_aware_capture,
fail_after_w_trace: FailAfterWTraceFactory,
):
'''
Verify we raise a ``BaseExceptionGroup`` out of a nursery where
more then one actor errors and also with a delay before failure
to test failure during an ongoing spawning.
"""
'''
async def main():
# budget = 2× natural trio-backend cascade time for
# 25 errorer subactors (~14s observed). on-timeout
# diag snapshot → if the cancel cascade hangs
# (observed under MTF backend with N>=14 errorer
# subactors) we get a fresh ptree/wchan/py-spy dump
# on disk INSTEAD of an opaque pytest timeout-kill.
# See `tractor/_testing/trace.py` for the helper.
async with fail_after_w_trace(30.0):
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as nursery:
@ -163,9 +234,23 @@ def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
)
# with pytest.raises(trio.MultiError) as exc_info:
with pytest.raises(BaseExceptionGroup) as exc_info:
# NOTE, `trio.TooSlowError` from `fail_after_w_trace`
# bubbles UN-wrapped if `open_nursery.__aexit__` never
# gets re-entered; wrapped inside a `BaseExceptionGroup`
# if it did. Accept both shapes so the matcher itself
# doesn't lie about *what* failed.
with pytest.raises(
(BaseExceptionGroup, trio.TooSlowError),
) as exc_info:
trio.run(main)
if isinstance(exc_info.value, trio.TooSlowError):
pytest.fail(
f'cancel cascade hung past 12s '
f'(num_subactors={num_subactors}, delay={delay}); '
f'see stderr for `fail_after_w_trace` snapshot path'
)
assert exc_info.type == ExceptionGroup
err = exc_info.value
exceptions = err.exceptions
@ -189,8 +274,15 @@ async def do_nothing():
pass
@pytest.mark.parametrize('mechanism', ['nursery_cancel', KeyboardInterrupt])
def test_cancel_single_subactor(reg_addr, mechanism):
@pytest.mark.parametrize(
'mechanism', [
'nursery_cancel',
KeyboardInterrupt,
])
def test_cancel_single_subactor(
reg_addr: tuple,
mechanism: str|KeyboardInterrupt,
):
'''
Ensure a ``ActorNursery.start_actor()`` spawned subactor
cancels when the nursery is cancelled.
@ -232,9 +324,14 @@ async def stream_forever():
await trio.sleep(0.01)
@tractor_test
async def test_cancel_infinite_streamer(start_method):
@tractor_test(
timeout=6,
)
async def test_cancel_infinite_streamer(
reg_addr: tuple,
start_method: str,
set_fork_aware_capture,
):
# stream for at most 1 seconds
with (
trio.fail_after(4),
@ -286,11 +383,15 @@ async def test_cancel_infinite_streamer(start_method):
'no_daemon_actors_fail_all_run_in_actors_sleep_then_fail',
],
)
@tractor_test
@tractor_test(
timeout=10,
)
async def test_some_cancels_all(
num_actors_and_errs: tuple,
reg_addr: tuple,
start_method: str,
loglevel: str,
set_fork_aware_capture, #: Callable,
):
'''
Verify a subset of failed subactors causes all others in
@ -370,7 +471,10 @@ async def test_some_cancels_all(
pytest.fail("Should have gotten a remote assertion error?")
async def spawn_and_error(breadth, depth) -> None:
async def spawn_and_error(
breadth: int,
depth: int,
) -> None:
name = tractor.current_actor().name
async with tractor.open_nursery() as nursery:
for i in range(breadth):
@ -395,28 +499,150 @@ async def spawn_and_error(breadth, depth) -> None:
await nursery.run_in_actor(*args, **kwargs)
@tractor_test
async def test_nested_multierrors(loglevel, start_method):
# NOTE: `main_thread_forkserver` capture-fd hang class is no
# longer skipped here — `--capture=sys` (the new `pyproject.toml`
# default) sidesteps the pipe-buffer-fill deadlock for
# `test_nested_multierrors`. See
# `ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`
# / #449 for the post-mortem.
# @pytest.mark.timeout(
# 10,
# method='thread',
# )
@pytest.mark.parametrize(
'depth',
[1, 3],
ids='depth={}'.format,
)
@tractor_test(
# XXX this OUTER `trio.fail_after` wall MUST exceed the
# largest INNER `fail_after_w_trace()` budget set in the body
# below (max = the MTF depth=3 == 30s case, further scaled by
# `cpu_scaling_factor()` on CI/throttle). Otherwise it fires
# FIRST and pre-empts the inner snapshot-capturing deadline,
# turning a graceful `TooSlowError`+ptree-dump into an opaque
# outer timeout-kill (the prior `timeout=10` did exactly this
# — it was *smaller* than the 12s trio depth=3 budget, so the
# depth-3 case `FAILED` on slow CI instead of dumping).
# Trio backend is fast and won't notice the extra budget.
# See `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
timeout=40,
)
async def test_nested_multierrors(
reg_addr: tuple,
loglevel: str,
start_method: str,
set_fork_aware_capture,
fail_after_w_trace: FailAfterWTraceFactory,
request: pytest.FixtureRequest,
depth: int,
):
'''
Test that failed actor sets are wrapped in `BaseExceptionGroup`s. This
test goes only 2 nurseries deep but we should eventually have tests
for arbitrary n-depth actor trees.
Test that failed actor sets are wrapped in `BaseExceptionGroup`s.
Parametrized over recursion `depth ∈ {1, 3}`:
- `depth=1`: shallow tree (2 spawners × 2 errorers, 2
levels). Cascade completes well within budget on ALL
backends including MTF — regression-safety green case.
- `depth=3`: deep tree (2 spawners × recursive depth-3
spawn-and-error). On `main_thread_forkserver` this
trips the cancel-cascade shape-mismatch bug class
(see `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`)
— xfailed below.
'''
if start_method == 'trio':
depth = 3
subactor_breadth = 2
else:
# XXX: multiprocessing can't seem to handle any more then 2 depth
# process trees for whatever reason.
# Any more process levels then this and we see bugs that cause
# hangs and broken pipes all over the place...
# XXX: `multiprocessing.forkserver` can't handle nested
# spawning at any depth — hangs / broken-pipes. Pre-existing
# backend limitation, NOT depth-specific.
if start_method == 'forkserver':
pytest.skip("Forksever sux hard at nested spawning...")
depth = 1 # means an additional actor tree of spawning (2 levels deep)
subactor_breadth = 2
with trio.fail_after(120):
# MTF backend trips a probabilistic timing race in the
# cancel-cascade — NOT depth-gated; depth amplifies the
# variance so depth=3 misses nearly every run while
# depth=1 misses occasionally. Both get the xfail mark
# (with `strict=False`) since the bug class can fire at
# either depth.
#
# The scenario in detail:
#
# T=0 spawn spawner_0 + spawner_1 in parallel
# T=t1 spawner_0's child errors →
# RemoteActorError reaches root nursery
# T=t1+ε root nursery starts cancelling
# spawner_1's portal-wait
# T=t2 spawner_1's child errors → tries to send
# RemoteActorError back
#
# if t2 < t1+ε: BEG = [RAE, RAE] ← clean (xpass)
# if t2 > t1+ε: BEG = [RAE, Cancelled] ← race tripped (xfail)
#
# i.e. the assertion below (`isinstance(_, RemoteActorError)`)
# fails iff cancel-delivery beats the other tree's natural
# error-propagation. Depth amplifies `t2-t1` variance
# (longer per-tree paths = more skew); under MTF the
# fork-spawn jitter + UDS-contention widens both `t1` and
# `t2` further.
#
# With `strict=False` the clean-cascade cases (most
# depth=1 runs, rare depth=3 runs) report as `xpassed`
# while the race-tripped cases report as `xfailed` —
# neither flakes `--lf`. When MTF cancel-cascade
# eventually speeds up enough to close the race even at
# depth=3, BOTH variants will reliably `xpass` and
# pytest will yell — our signal to drop the marker. See
# `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
if start_method == 'main_thread_forkserver':
request.node.add_marker(
pytest.mark.xfail(
strict=False,
reason=(
f'MTF cancel-cascade shape-mismatch at '
f'depth={depth} (Cancelled races '
f'RemoteActorError in BEG); see conc-anal/'
'cancel_cascade_too_slow_under_main_thread_forkserver_issue.md'
),
)
)
# Per-backend/-depth budgets: in the non-hang case the
# whole spawn + cancel-cascade should complete in well
# under these. On the borderline hang case the
# `fail_after_w_trace` fires `TooSlowError` AND captures a
# ptree/wchan/py-spy snapshot to
# `$XDG_CACHE_HOME/tractor/hung-dumps/` for offline
# inspection. See
# `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
#
# NOTE: the `trio` depth=3 budget was bumped 6 -> 12s after
# the `trio` 0.29 -> 0.33 lock bump (commit c7741bba) slowed
# the depth-3 cancel-cascade from <6s to ~7-8s; the 6s
# deadline was firing and its `Cancelled(source='deadline')`
# (trio 0.33 cancel-reason metadata) collapsed a BEG branch,
# breaking the `RemoteActorError` assertion below. depth=1
# still finishes in ~3s so keeps the 6s budget. See
# `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
match (start_method, depth):
case ('trio', 1):
timeout = 6
case ('trio', 3):
timeout = 12
case ('main_thread_forkserver', 1):
timeout = 16
case ('main_thread_forkserver', 3):
timeout = 30
# headroom for CPU-freq scaling AND/OR slow CI so the inner
# snapshot-capturing budget doesn't fire spuriously on a
# sluggish runner; see `cpu_scaling_factor()`.
from .conftest import cpu_scaling_factor
timeout *= cpu_scaling_factor()
async with fail_after_w_trace(timeout):
try:
async with tractor.open_nursery() as nursery:
for i in range(subactor_breadth):
@ -483,20 +709,24 @@ async def test_nested_multierrors(loglevel, start_method):
@no_windows
def test_cancel_via_SIGINT(
loglevel,
start_method,
spawn_backend,
reg_addr: tuple,
loglevel: str,
start_method: str,
):
"""Ensure that a control-C (SIGINT) signal cancels both the parent and
'''
Ensure that a control-C (SIGINT) signal cancels both the parent and
child processes in trionic fashion
"""
'''
pid: int = os.getpid()
async def main():
with trio.fail_after(2):
async with tractor.open_nursery() as tn:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as tn:
await tn.start_actor('sucka')
if 'mp' in spawn_backend:
if 'mp' in start_method:
time.sleep(0.1)
os.kill(pid, signal.SIGINT)
await trio.sleep_forever()
@ -507,6 +737,7 @@ def test_cancel_via_SIGINT(
@no_windows
def test_cancel_via_SIGINT_other_task(
reg_addr: tuple,
loglevel: str,
start_method: str,
spawn_backend: str,
@ -535,7 +766,9 @@ def test_cancel_via_SIGINT_other_task(
async def spawn_and_sleep_forever(
task_status=trio.TASK_STATUS_IGNORED
):
async with tractor.open_nursery() as tn:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as tn:
for i in range(3):
await tn.run_in_actor(
sleep_forever,
@ -599,7 +832,7 @@ async def spawn_sub_with_sync_blocking_task():
def test_cancel_while_childs_child_in_sync_sleep(
loglevel: str,
start_method: str,
spawn_backend: str,
is_forking_spawner: bool,
debug_mode: bool,
reg_addr: tuple,
man_cancel_outer: bool,
@ -615,7 +848,10 @@ def test_cancel_while_childs_child_in_sync_sleep(
'''
if start_method == 'forkserver':
pytest.skip("Forksever sux hard at resuming from sync sleep...")
pytest.skip(
"`multiprocessing`'s forkserver sux hard at "
"resuming from sync sleep..."
)
async def main():
#
@ -658,7 +894,11 @@ def test_cancel_while_childs_child_in_sync_sleep(
# delay = 2 # is AssertionError in eg AND no TooSlowError !?
# is AssertionError in eg AND no _cs cancellation.
delay = (
6 if _non_linux
6 if (
_non_linux
or
is_forking_spawner
)
else 4
)
@ -694,7 +934,7 @@ def test_cancel_while_childs_child_in_sync_sleep(
def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
start_method,
start_method: str,
):
'''
This is a very subtle test which demonstrates how cancellation
@ -715,6 +955,12 @@ def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
if _friggin_windows: # smh
timeout += 1
# CPU-scaling / CI latency headroom — macOS CI especially is
# slow for this graceful-vs-hard-reap timing race; see
# `cpu_scaling_factor()`.
from .conftest import cpu_scaling_factor
timeout *= cpu_scaling_factor()
async def main():
start = time.time()
try:

7
tests/test_clustering.py
View File

@ -77,6 +77,7 @@ async def worker(
@tractor_test
async def test_streaming_to_actor_cluster(
tpt_proto: str,
is_forking_spawner: bool,
):
'''
Open an actor "cluster" using the (experimental) `._clustering`
@ -88,7 +89,11 @@ async def test_streaming_to_actor_cluster(
f'Test currently fails with tpt-proto={tpt_proto!r}\n'
)
with trio.fail_after(6):
delay: float = (
10 if is_forking_spawner
else 6
)
with trio.fail_after(delay):
async with (
open_actor_cluster(modules=[__name__]) as portals,

21
tests/test_context_stream_semantics.py
View File

@ -115,10 +115,12 @@ async def not_started_but_stream_opened(
)
def test_started_misuse(
target: Callable,
reg_addr: tuple,
debug_mode: bool,
):
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.start_actor(
@ -184,15 +186,24 @@ def test_simple_context(
error_parent,
child_blocks_forever,
pointlessly_open_stream,
reg_addr: tuple,
debug_mode: bool,
is_forking_spawner: bool,
):
timeout = 1.5 if not platform.system() == 'Windows' else 4
timeout: float = 1.5
# windows and forking-spawner both have "slower but more
# deterministic" cancel teardown.
if platform.system() == 'Windows':
timeout = 4
elif is_forking_spawner:
timeout = 3
async def main():
with trio.fail_after(timeout):
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.start_actor(
@ -278,6 +289,7 @@ def test_parent_cancels(
cancel_method: str,
chk_ctx_result_before_exit: bool,
child_returns_early: bool,
reg_addr: tuple,
debug_mode: bool,
):
'''
@ -355,6 +367,7 @@ def test_parent_cancels(
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.start_actor(
@ -931,6 +944,7 @@ async def keep_sending_from_child(
)
def test_one_end_stream_not_opened(
overrun_by: tuple[str, int, Callable],
reg_addr: tuple,
debug_mode: bool,
):
'''
@ -949,6 +963,7 @@ def test_one_end_stream_not_opened(
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.start_actor(
@ -1113,6 +1128,7 @@ def test_maybe_allow_overruns_stream(
# conftest wide
loglevel: str,
reg_addr: tuple,
debug_mode: bool,
):
'''
@ -1133,6 +1149,7 @@ def test_maybe_allow_overruns_stream(
'''
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.start_actor(
@ -1249,6 +1266,7 @@ def test_maybe_allow_overruns_stream(
def test_ctx_with_self_actor(
loglevel: str,
reg_addr: tuple,
debug_mode: bool,
):
'''
@ -1263,6 +1281,7 @@ def test_ctx_with_self_actor(
'''
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
enable_modules=[__name__],
) as an:

197
tests/test_infected_asyncio.py
View File

@ -30,6 +30,32 @@ from tractor import (
from tractor.runtime import _state
from tractor.trionics import BroadcastReceiver
from tractor._testing import expect_ctxc
from tractor._testing.trace import (
AfkAlarmWTraceFactory,
FailAfterWTraceFactory,
)
# Per-test zombie-subactor reaper. Opt-in (NOT autouse) —
# see `tractor._testing.pytest.reap_subactors_per_test`'s
# docstring for the full rationale. This module specifically
# needs it because tests like
# `test_echoserver_detailed_mechanics[KeyboardInterrupt]`
# and the `test_sigint_closes_lifetime_stack[*]` matrix have
# been observed to hang past pytest's wall-clock under
# `main_thread_forkserver`, leaving subactor forks that
# squat on registrar resources and cascade-fail every
# subsequent test (`test_inter_peer_cancellation`,
# `test_legacy_one_way_streaming`, etc.).
pytestmark = pytest.mark.usefixtures(
'reap_subactors_per_test',
# NOTE, asyncio cancel cascade has historically
# triggered both UDS sockfile leaks (SIGKILL path)
# AND the trio `WakeupSocketpair.drain()` busy-loop
# — see `test_aio_simple_error`'s history.
'track_orphaned_uds_per_test',
'detect_runaway_subactors_per_test',
)
@pytest.fixture(
@ -183,6 +209,7 @@ def test_tractor_cancels_aio(
async def main():
async with tractor.open_nursery(
debug_mode=debug_mode,
registry_addrs=[reg_addr],
) as an:
portal = await an.run_in_actor(
asyncio_actor,
@ -205,11 +232,11 @@ def test_trio_cancels_aio(
'''
async def main():
with trio.move_on_after(1):
# cancel the nursery shortly after boot
async with tractor.open_nursery() as tn:
with trio.move_on_after(1):
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as tn:
await tn.run_in_actor(
asyncio_actor,
target='aio_sleep_forever',
@ -277,7 +304,9 @@ def test_context_spawns_aio_task_that_errors(
'''
async def main():
with trio.fail_after(1 + delay):
async with tractor.open_nursery() as an:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
p = await an.start_actor(
'aio_daemon',
enable_modules=[__name__],
@ -360,7 +389,9 @@ def test_aio_cancelled_from_aio_causes_trio_cancelled(
async def main():
an: tractor.ActorNursery
async with tractor.open_nursery() as an:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
p: tractor.Portal = await an.run_in_actor(
asyncio_actor,
target='aio_cancel',
@ -569,7 +600,9 @@ def test_basic_interloop_channel_stream(
async def main():
# TODO, figure out min timeout here!
with trio.fail_after(6):
async with tractor.open_nursery() as an:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
portal = await an.run_in_actor(
stream_from_aio,
infect_asyncio=True,
@ -582,9 +615,13 @@ def test_basic_interloop_channel_stream(
# TODO: parametrize the above test and avoid the duplication here?
def test_trio_error_cancels_intertask_chan(reg_addr):
def test_trio_error_cancels_intertask_chan(
reg_addr: tuple[str, int],
):
async def main():
async with tractor.open_nursery() as an:
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
portal = await an.run_in_actor(
stream_from_aio,
trio_raise_err=True,
@ -619,6 +656,7 @@ def test_trio_closes_early_causes_aio_checkpoint_raise(
async with tractor.open_nursery(
debug_mode=debug_mode,
# enable_stack_on_sig=True,
registry_addrs=[reg_addr],
) as an:
portal = await an.run_in_actor(
stream_from_aio,
@ -667,6 +705,7 @@ def test_aio_exits_early_relays_AsyncioTaskExited(
async def main():
with trio.fail_after(1 + delay):
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
# enable_stack_on_sig=True,
) as an:
@ -707,6 +746,7 @@ def test_aio_errors_and_channel_propagates_and_closes(
):
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
portal = await an.run_in_actor(
@ -796,16 +836,47 @@ async def trio_to_aio_echo_server(
@pytest.mark.parametrize(
'raise_error_mid_stream',
[False, Exception, KeyboardInterrupt],
[
False,
Exception,
KeyboardInterrupt,
],
ids='raise_error={}'.format,
)
def test_echoserver_detailed_mechanics(
reg_addr: tuple[str, int],
debug_mode: bool,
raise_error_mid_stream,
is_forking_spawner: bool,
fail_after_w_trace: FailAfterWTraceFactory,
):
async def main():
# NOTE: under fork-based backends the cancel-cascade
# path is structurally slower than `trio`'s subproc-exec
# (per-spawn forkserver-handshake compounds during
# teardown). Bump the cap so cross-test contamination
# doesn't flake this — see
# `ai/conc-anal/cancel_cascade_too_slow_under_main_thread_forkserver_issue.md`.
timeout: float = (
999 if tractor.debug_mode()
else 4 if is_forking_spawner
# was 1; the `trio` 0.29 -> 0.33 bump slowed the
# cancel-cascade so a 1s budget raced the ~1s teardown
# deadline. On a deadline-fire the injected
# `Cancelled(source='deadline')` wraps the mid-stream
# KBI in a `BaseExceptionGroup`, breaking the bare
# `pytest.raises(KeyboardInterrupt)` below. See
# `ai/conc-anal/trio_033_cancel_cascade_slowdown_depth3_issue.md`.
else 4
)
# body factored out so the `fail_after_w_trace`-wrapping
# `main()` stays a 2-liner — keeps the deep `open_nursery`
# /`open_context`/`open_stream` block at its natural indent
# level instead of pushing it under yet another `async with`.
async def _body():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
p = await an.start_actor(
@ -849,6 +920,15 @@ def test_echoserver_detailed_mechanics(
# is cancelled by kbi or out of task cancellation
await p.cancel_actor()
async def main():
# on-timeout diag snapshot via `fail_after_w_trace`
# — when the cancel cascade hangs under MTF we get a
# fresh `ptree`/`wchan`/`py-spy` dump on disk INSTEAD
# of an opaque pytest timeout-kill. See
# `tractor/_testing/trace.py`.
async with fail_after_w_trace(timeout):
await _body()
if raise_error_mid_stream:
with pytest.raises(raise_error_mid_stream):
trio.run(main)
@ -984,7 +1064,7 @@ async def manage_file(
],
ids=[
'bg_aio_task',
'just_trio_slee',
'just_trio_sleep',
],
)
@pytest.mark.parametrize(
@ -1000,11 +1080,15 @@ async def manage_file(
)
def test_sigint_closes_lifetime_stack(
tmp_path: Path,
reg_addr: tuple,
debug_mode: bool,
wait_for_ctx: bool,
bg_aio_task: bool,
trio_side_is_shielded: bool,
debug_mode: bool,
send_sigint_to: str,
is_forking_spawner: bool,
afk_alarm_w_trace: AfkAlarmWTraceFactory,
):
'''
Ensure that an infected child can use the `Actor.lifetime_stack`
@ -1014,12 +1098,30 @@ def test_sigint_closes_lifetime_stack(
'''
async def main():
delay = 999 if tractor.debug_mode() else 1
delay: float = (
999
if debug_mode
else 1
)
# pre-init so the `except (KeyboardInterrupt, ContextCancelled)`
# handler below doesn't `UnboundLocalError` if KBI fires BEFORE
# we ever enter the `as (ctx, first)` body (e.g. when
# `p.open_context().__aenter__` is hung waiting for the
# subactor's `StartAck` due to a fork-child IPC race —
# see `dynamic_pub_sub_spawn_time_transport_close_under_mtf_issue.md`).
tmp_file: Path|None = None
ctx: tractor.Context|None = None
try:
an: tractor.ActorNursery
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
) as an:
# sanity
if debug_mode:
assert tractor.debug_mode()
p: tractor.Portal = await an.start_actor(
'file_mngr',
enable_modules=[__name__],
@ -1034,7 +1136,7 @@ def test_sigint_closes_lifetime_stack(
) as (ctx, first):
path_str, cpid = first
tmp_file: Path = Path(path_str)
tmp_file = Path(path_str)
assert tmp_file.exists()
# XXX originally to simulate what (hopefully)
@ -1054,6 +1156,10 @@ def test_sigint_closes_lifetime_stack(
cpid if send_sigint_to == 'child'
else os.getpid()
)
print(
f'Sending SIGINT to {send_sigint_to!r}\n'
f'pid: {pid!r}\n'
)
os.kill(
pid,
signal.SIGINT,
@ -1064,13 +1170,37 @@ def test_sigint_closes_lifetime_stack(
# timeout should trigger!
if wait_for_ctx:
print('waiting for ctx outcome in parent..')
if debug_mode:
assert delay == 999
try:
with trio.fail_after(1 + delay):
with trio.fail_after(
1 + delay
):
await ctx.wait_for_result()
except tractor.ContextCancelled as ctxc:
assert ctxc.canceller == ctx.chan.uid
raise
except trio.TooSlowError:
if (
send_sigint_to == 'child'
and
is_forking_spawner
):
pytest.xfail(
reason=(
'SIGINT delivery to fork-child subactor is known '
'to NOT SUCCEED, precisely bc we have not wired up a'
'"trio SIGINT mode" in the child pre-fork.\n'
'Also see `test_orphaned_subactor_sigint_cleanup_DRAFT` for'
'a dedicated suite demonstrating this expected limitation as '
'well as the detailed doc:\n'
'`ai/conc-anal/subint_forkserver_orphan_sigint_hang_issue.md`.\n'
),
)
# XXX CASE 2: this seems to be the source of the
# original issue which exhibited BEFORE we put
# a `Actor.cancel_soon()` inside
@ -1084,6 +1214,21 @@ def test_sigint_closes_lifetime_stack(
KeyboardInterrupt,
ContextCancelled,
):
# If we got here BEFORE entering the ctx body (e.g.
# spawn-time IPC race hung `open_context.__aenter__` and
# the AFK-guard `signal.alarm` fired KBI from outside the
# trio loop), `tmp_file`/`ctx` are still `None` — surface
# that fact directly instead of `UnboundLocalError`.
if tmp_file is None:
pytest.fail(
'KBI/ctxc fired BEFORE `p.open_context()` returned '
"the child's `started` value — likely fork-child "
'IPC race; see '
'`ai/conc-anal/'
'dynamic_pub_sub_spawn_time_transport_close_'
'under_mtf_issue.md`'
)
# XXX CASE 2: without the bug fixed, in the
# KBI-raised-in-parent case, the actor teardown should
# never get run (silently abaondoned by `asyncio`..) and
@ -1091,6 +1236,25 @@ def test_sigint_closes_lifetime_stack(
assert not tmp_file.exists()
assert ctx.maybe_error
# outer hard wall-clock backstop via `afk_alarm_w_trace`:
# when the in-band trio cancel path doesn't fire (e.g.
# parent is parked in a shielded `await` inside actor-
# nursery teardown, or `open_context.__aenter__` hangs
# waiting for a child's `StartAck` that never comes), the
# `signal.alarm` inside the CM raises `AFKAlarmTimeout`
# in the main thread regardless of trio's scope state —
# AND captures a full diag snapshot to
# `$XDG_CACHE_HOME/tractor/hung-dumps/` before re-raising.
# Only armed under fork-based backends since this hang-
# class is MTF-specific.
if (
not debug_mode
and
is_forking_spawner
):
with afk_alarm_w_trace(10):
trio.run(main)
else:
trio.run(main)
@ -1170,6 +1334,7 @@ def test_aio_side_raises_before_started(
with trio.fail_after(3):
an: tractor.ActorNursery
async with tractor.open_nursery(
registry_addrs=[reg_addr],
debug_mode=debug_mode,
loglevel=loglevel,
) as an:

25
tests/test_inter_peer_cancellation.py
View File

@ -26,6 +26,31 @@ from tractor._testing import (
from .conftest import cpu_scaling_factor
pytestmark = [
pytest.mark.skipon_spawn_backend(
'subint',
reason=(
'XXX SUBINT GIL-CONTENTION HANGING TEST XXX\n'
'Inter-peer cancel cascades under '
'`--spawn-backend=subint` trip the abandoned-subint '
'GIL-hostage class — see\n'
' - `ai/conc-anal/subint_sigint_starvation_issue.md` '
'(GIL-hostage, SIGINT-unresponsive)\n'
' - `ai/conc-anal/subint_cancel_delivery_hang_issue.md` '
'(sibling: parent parks on dead chan)\n'
' - https://github.com/goodboy/tractor/issues/379 '
'(subint umbrella)\n'
)
),
# NOTE, inter-peer cancellation tests stress the
# multi-actor cancel cascade which under SIGKILL
# leaves UDS sock-files orphaned. Track per-test
# for blame attribution.
pytest.mark.usefixtures(
'track_orphaned_uds_per_test',
),
]
# XXX TODO cases:
# - [x] WE cancelled the peer and thus should not see any raised
# `ContextCancelled` as it should be reaped silently?

127
tests/test_legacy_one_way_streaming.py
View File

@ -1,7 +1,7 @@
"""
'''
Streaming via the, now legacy, "async-gen API".
"""
'''
import time
from functools import partial
import platform
@ -12,6 +12,11 @@ import tractor
import pytest
from tractor._testing import tractor_test
from tractor._exceptions import ActorTooSlowError
_non_linux: bool = (
_sys := platform.system()
) != 'Linux'
def test_must_define_ctx():
@ -68,8 +73,10 @@ async def stream_from_single_subactor(
start_method,
stream_func,
):
"""Verify we can spawn a daemon actor and retrieve streamed data.
"""
'''
Verify we can spawn a daemon actor and retrieve streamed data.
'''
# only one per host address, spawns an actor if None
async with tractor.open_nursery(
@ -242,14 +249,19 @@ async def a_quadruple_example() -> list[int]:
start = time.time()
# the portal call returns exactly what you'd expect
# as if the remote "aggregate" function was called locally
result_stream = []
result_stream: list[int] = []
async with portal.open_stream_from(aggregate, seed=seed) as stream:
async with portal.open_stream_from(
aggregate,
seed=seed,
) as stream:
async for value in stream:
result_stream.append(value)
print(f"STREAM TIME = {time.time() - start}")
print(f"STREAM + SPAWN TIME = {time.time() - pre_start}")
print(
f"STREAM TIME = {time.time() - start}\n"
f"STREAM + SPAWN TIME = {time.time() - pre_start}\n"
)
assert result_stream == list(range(seed))
await portal.cancel_actor()
return result_stream
@ -258,13 +270,24 @@ async def a_quadruple_example() -> list[int]:
async def cancel_after(
wait: float,
reg_addr: tuple,
expect_cancel: bool,
) -> list[int]:
async with tractor.open_root_actor(
registry_addrs=[reg_addr],
):
with trio.move_on_after(wait):
return await a_quadruple_example()
res: list[int]|None = None
with trio.move_on_after(wait) as cs:
res: list[int] = await a_quadruple_example()
return res
if (
not expect_cancel
and
cs.cancelled_caught
):
assert not res
raise ActorTooSlowError
@pytest.fixture(scope='module')
@ -272,9 +295,14 @@ def time_quad_ex(
reg_addr: tuple,
ci_env: bool,
spawn_backend: str,
is_forking_spawner: bool,
tpt_proto: str,
):
if (
ci_env
and
_non_linux
):
non_linux: bool = (_sys := platform.system()) != 'Linux'
if ci_env and non_linux:
pytest.skip(f'Test is too flaky on {_sys!r} in CI')
if spawn_backend == 'mp':
@ -284,15 +312,42 @@ def time_quad_ex(
'''
pytest.skip("Test is too flaky on mp in CI")
timeout = 7 if non_linux else 4
start = time.time()
results: list[int] = trio.run(
cancel_after,
timeout,
reg_addr,
timeout: float = (
7 if _non_linux
else 4
)
if (
is_forking_spawner
and
tpt_proto in [
'uds',
]
):
timeout += 1
# inflate the cancel-deadline for CPU-freq scaling AND/OR CI
# latency (see `cpu_scaling_factor()`) so the example isn't
# cancelled mid-stream on a throttled/CI runner.
from .conftest import cpu_scaling_factor
timeout *= cpu_scaling_factor()
start: float = time.time()
results: list[int] = trio.run(partial(
cancel_after,
wait=timeout,
reg_addr=reg_addr,
expect_cancel=True,
))
diff: float = time.time() - start
assert results
if results is None:
raise ActorTooSlowError(
f'Streaming example took longer then timeout ??\n'
f'timeout={timeout!r}\n'
f'diff={diff!r}\n'
f'results={results!r}\n'
)
return results, diff
@ -307,11 +362,10 @@ def test_a_quadruple_example(
given past empirical eval of this suite.
'''
non_linux: bool = (_sys := platform.system()) != 'Linux'
this_fast_on_linux: float = 3
this_fast = (
6 if non_linux
6 if _non_linux
else this_fast_on_linux
)
# ^ XXX NOTE,
@ -348,21 +402,26 @@ def test_not_fast_enough_quad(
reg_addr: tuple,
time_quad_ex: tuple[list[int], float],
cancel_delay: float,
ci_env: bool,
spawn_backend: str,
is_forking_spawner: bool,
tpt_proto: str,
test_log: tractor.log.StackLevelAdapter,
):
'''
Verify we can cancel midway through the quad example and all
actors cancel gracefully.
Verify we can cancel midway through `a_quadruple_example()`, at
various delays, and all subactors cancel gracefully.
'''
results, diff = time_quad_ex
delay = max(diff - cancel_delay, 0)
results = trio.run(
results: list[int] = trio.run(partial(
cancel_after,
delay,
reg_addr,
)
wait=delay,
reg_addr=reg_addr,
expect_cancel=True,
))
system: str = platform.system()
if (
system in ('Windows', 'Darwin')
@ -373,6 +432,20 @@ def test_not_fast_enough_quad(
# so just ignore these
print(f'Woa there {system} caught your breath eh?')
else:
if (
results
and
is_forking_spawner
and
tpt_proto in [
'uds',
]
):
pytest.xfail(
f'Spawning backend + tpt-proto is too fast XD\n'
f'{spawn_backend!r} + {tpt_proto!r}\n'
)
# should be cancelled mid-streaming
assert results is None

17
tests/test_log_sys.py
View File

@ -20,6 +20,16 @@ def test_root_pkg_not_duplicated_in_logger_name():
a common `<root_name>.< >` prefix, ensure that it is not
duplicated in the child's `StackLevelAdapter.name: str`.
Also pins the explicit-`name` contract: an explicitly passed
dotted `name` is treated as a *literal* sub-logger path and is
NOT leaf-collapsed. The leaf-module is only dropped when the
trailing token duplicates the *caller's own* `__name__` leaf (the
`{filename}` field) — see `test_implicit_mod_name_applied_for_child`
for that (auto-naming) path. This is what keeps a real (possibly
nested) sub-PACKAGE like `subpkg.mod` -> `devx.debug` addressable
by the `tractor.log` logging-spec, instead of collapsing to its
parent.
'''
project_name: str = 'pylib'
pkg_path: str = 'pylib.subpkg.mod'
@ -38,8 +48,13 @@ def test_root_pkg_not_duplicated_in_logger_name():
)
assert proj_log is not sublog
# the root pkg-name appears exactly once (no `pylib.pylib...`)
assert sublog.name.count(proj_log.name) == 1
assert 'mod' not in sublog.name
# explicit dotted `name` is preserved literally (NOT collapsed);
# the trailing token survives since it's not the *caller's* own
# leaf-module (`test_log_sys`), so this is treated as a literal
# sub-pkg path.
assert sublog.name == f'{project_name}.subpkg.mod'
def test_implicit_mod_name_applied_for_child(

183
tests/test_multi_program.py
View File

@ -1,183 +0,0 @@
"""
Multiple python programs invoking the runtime.
"""
from __future__ import annotations
import platform
import subprocess
import time
from typing import (
TYPE_CHECKING,
)
import pytest
import trio
import tractor
from tractor._testing import (
tractor_test,
)
from tractor import (
current_actor,
Actor,
Context,
Portal,
)
from tractor.runtime import _state
from .conftest import (
sig_prog,
_INT_SIGNAL,
_INT_RETURN_CODE,
)
if TYPE_CHECKING:
from tractor.msg import Aid
from tractor.discovery._addr import (
UnwrappedAddress,
)
_non_linux: bool = platform.system() != 'Linux'
def test_abort_on_sigint(
daemon: subprocess.Popen,
):
assert daemon.returncode is None
time.sleep(0.1)
sig_prog(daemon, _INT_SIGNAL)
assert daemon.returncode == _INT_RETURN_CODE
# XXX: oddly, couldn't get capfd.readouterr() to work here?
if platform.system() != 'Windows':
# don't check stderr on windows as its empty when sending CTRL_C_EVENT
assert "KeyboardInterrupt" in str(daemon.stderr.read())
@tractor_test
async def test_cancel_remote_registrar(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
):
assert not current_actor().is_registrar
async with tractor.get_registry(reg_addr) as portal:
await portal.cancel_actor()
time.sleep(0.1)
# the registrar channel server is cancelled but not its main task
assert daemon.returncode is None
# no registrar socket should exist
with pytest.raises(OSError):
async with tractor.get_registry(reg_addr) as portal:
pass
def test_register_duplicate_name(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
):
async def main():
async with tractor.open_nursery(
registry_addrs=[reg_addr],
) as an:
assert not current_actor().is_registrar
p1 = await an.start_actor('doggy')
p2 = await an.start_actor('doggy')
async with tractor.wait_for_actor('doggy') as portal:
assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
await an.cancel()
# XXX, run manually since we want to start this root **after**
# the other "daemon" program with it's own root.
trio.run(main)
@tractor.context
async def get_root_portal(
ctx: Context,
):
'''
Connect back to the root actor manually (using `._discovery` API)
and ensure it's contact info is the same as our immediate parent.
'''
sub: Actor = current_actor()
rtvs: dict = _state._runtime_vars
raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
# await tractor.pause()
# XXX, in case the sub->root discovery breaks you might need
# this (i know i did Xp)!!
# from tractor.devx import mk_pdb
# mk_pdb().set_trace()
assert (
len(raddrs) == 1
and
list(sub._parent_chan.raddr.unwrap()) in raddrs
)
# connect back to our immediate parent which should also
# be the actor-tree's root.
from tractor.discovery._api import get_root
ptl: Portal
async with get_root() as ptl:
root_aid: Aid = ptl.chan.aid
parent_ptl: Portal = current_actor().get_parent()
assert (
root_aid.name == 'root'
and
parent_ptl.chan.aid == root_aid
)
await ctx.started()
def test_non_registrar_spawns_child(
daemon: subprocess.Popen,
reg_addr: UnwrappedAddress,
loglevel: str,
debug_mode: bool,
ci_env: bool,
):
'''
Ensure a non-regristar (serving) root actor can spawn a sub and
that sub can connect back (manually) to it's rent that is the
root without issue.
More or less this audits the global contact info in
`._state._runtime_vars`.
'''
async def main():
# XXX, since apparently on macos in GH's CI it can be a race
# with the `daemon` registrar on grabbing the socket-addr..
if ci_env and _non_linux:
await trio.sleep(.5)
async with tractor.open_nursery(
registry_addrs=[reg_addr],
loglevel=loglevel,
debug_mode=debug_mode,
) as an:
actor: Actor = tractor.current_actor()
assert not actor.is_registrar
sub_ptl: Portal = await an.start_actor(
name='sub',
enable_modules=[__name__],
)
async with sub_ptl.open_context(
get_root_portal,
) as (ctx, _):
print('Waiting for `sub` to connect back to us..')
await an.cancel()
# XXX, run manually since we want to start this root **after**
# the other "daemon" program with it's own root.
trio.run(main)

8
tests/test_pubsub.py
View File

@ -7,6 +7,14 @@ import tractor
from tractor.experimental import msgpub
from tractor._testing import tractor_test
pytestmark = pytest.mark.skipon_spawn_backend(
'subint',
reason=(
'XXX SUBINT HANGING TEST XXX\n'
'See oustanding issue(s)\n'
# TODO, put issue link!
)
)
def test_type_checks():

3
tests/test_ringbuf.py
View File

@ -21,6 +21,9 @@ from tractor._testing.samples import (
# XXX, in case you want to melt your cores, comment this skip line XD
pytestmark = pytest.mark.skip
# XXX `cffi` dun build on py3.14 yet..
cffi = pytest.importorskip("cffi")
@tractor.context
async def child_read_shm(

14
tests/test_shm.py
View File

@ -14,6 +14,20 @@ from tractor.ipc._shm import (
attach_shm_list,
)
pytestmark = pytest.mark.skipon_spawn_backend(
'subint',
# NOTE, `main_thread_forkserver` works for these tests
# via the `mp.SharedMemory(track=False)` +
# `mp.resource_tracker` monkey-patch in `.ipc._mp_bs`.
# Without that workaround the fork-inherited
# `resource_tracker` fd would EBADF on first shm op +
# cascade into `FileExistsError` across parametrize
# variants. Tracker doc:
# `ai/conc-anal/subint_forkserver_mp_shared_memory_issue.md`.
reason=(
'subint: GIL-contention hanging class.\n'
)
)
@tractor.context
async def child_attach_shml_alot(

7
tests/test_spawning.py
View File

@ -194,9 +194,14 @@ def test_loglevel_propagated_to_subactor(
reg_addr: tuple,
level: str,
):
if start_method == 'mp_forkserver':
if start_method in ('mp_forkserver', 'main_thread_forkserver'):
pytest.skip(
"a bug with `capfd` seems to make forkserver capture not work? "
"(same class as the `mp_forkserver` pre-existing skip — fork-"
"based backends inherit pytest's capfd temp-file fds into the "
"subactor and the IPC handshake reads garbage (`unclean EOF "
"read only X/HUGE_NUMBER bytes`). Work around by using "
"`capsys` instead or skip entirely."
)
async def main():

0
tests/trionics/__init__.py 100644
View File

99
tests/trionics/test_patches.py 100644
View File

@ -0,0 +1,99 @@
'''
Regression tests for `tractor.trionics.patches` —
defensive monkey-patches on upstream `trio` bugs.
Each test asserts:
1. The bug exists (or is gone — skip cleanly if
upstream shipped the fix and our `is_needed()` now
returns `False`).
2. Our patch fixes it (post-`apply()` the `repro()`
returns cleanly within a tight wall-clock cap).
Wall-clock caps are critical here — the bugs we patch
are tight-loops or deadlocks, so a regression would
HANG the test runner unless we hard-cap each
`repro()` call.
'''
import signal
import pytest
from tractor.trionics import patches
from tractor.trionics.patches import _wakeup_socketpair as wsp
@pytest.fixture(autouse=True)
def _alarm_cleanup():
'''
Ensure no leftover SIGALRM survives a test failure
or unexpected return.
'''
yield
signal.alarm(0)
def test_wakeup_socketpair_drain_eof_patch_works():
'''
Without the patch, `WakeupSocketpair.drain()` on a
socketpair whose write-end has been closed spins
forever. With the patch applied, it returns
cleanly within milliseconds.
Wall-clock cap: 2s. If the patch regresses, SIGALRM
fires and the test hard-fails with a clear signal
instead of hanging CI indefinitely.
'''
if not wsp.is_needed():
pytest.skip(
'upstream trio shipped the fix — '
'patch no longer needed for trio '
'(see `is_needed()` for version gate)'
)
# Apply the patch.
applied: bool = wsp.apply()
# First call MUST return True; idempotent guard
# prevents False on subsequent calls within the
# same process.
assert isinstance(applied, bool) # idempotent (order-dependent value)
# Cap wall-clock at 2s; SIGALRM raises in main
# thread which interrupts the C-level recv loop
# IF the patch regresses (since `signal.alarm`
# uses Python's signal-wakeup-fd which the patch
# itself relies on... but `repro()` runs OUTSIDE
# a trio.run, so it's plain stdlib semantics here
# — alarm WILL fire during `recv` syscall).
signal.alarm(2)
wsp.repro()
signal.alarm(0)
def test_apply_all_idempotent():
'''
Calling `apply_all()` twice should not double-
apply: second call's dict has all-False values
(every patch reports "already applied").
'''
first: dict[str, bool] = patches.apply_all()
second: dict[str, bool] = patches.apply_all()
# Second call: every patch reports skipped.
assert all(v is False for v in second.values()), (
f'apply_all() not idempotent: {second}'
)
# First call: at least one patch was applied
# (or all are no-ops because `is_needed()` is
# False everywhere — the all-fixed-upstream future
# state which is also valid).
assert isinstance(first, dict)
for name, applied in first.items():
assert isinstance(applied, bool), (
f'patch {name!r} returned non-bool: {applied!r}'
)

89
tractor/_child.py
View File

@ -15,16 +15,23 @@
# along with this program. If not, see <https://www.gnu.org/licenses/>.
"""
This is the "bootloader" for actors started using the native trio backend.
The "bootloader" for sub-actors spawned via the native `trio`
backend (the default `python -m tractor._child` CLI entry) and
the in-process `subint` backend (`tractor.spawn._subint`).
"""
from __future__ import annotations
import argparse
from ast import literal_eval
from typing import TYPE_CHECKING
from .runtime._runtime import Actor
from .spawn._entry import _trio_main
if TYPE_CHECKING:
from .discovery._addr import UnwrappedAddress
from .spawn._spawn import SpawnMethodKey
def parse_uid(arg):
name, uuid = literal_eval(arg) # ensure 2 elements
@ -39,6 +46,73 @@ def parse_ipaddr(arg):
return arg
def _actor_child_main(
uid: tuple[str, str],
loglevel: str | None,
parent_addr: UnwrappedAddress | None,
infect_asyncio: bool,
spawn_method: SpawnMethodKey = 'trio',
) -> None:
'''
Construct the child `Actor` and dispatch to `_trio_main()`.
Shared entry shape used by both the `python -m tractor._child`
CLI (trio/mp subproc backends) and the `subint` backend, which
invokes this from inside a fresh `concurrent.interpreters`
sub-interpreter via `Interpreter.call()`.
'''
# Apply defensive monkey-patches for upstream `trio`
# bugs we've encountered while running tractor — see
# `tractor.trionics.patches` for the catalog +
# per-patch upstream-fix tracking. Must run BEFORE
# any trio runtime init.
from .trionics.patches import apply_all
apply_all()
subactor = Actor(
name=uid[0],
uuid=uid[1],
loglevel=loglevel,
spawn_method=spawn_method,
)
# XXX, set a stable OS-level proc-title BEFORE entering
# the trio runtime so `ps`/`top`/`acli.pytree` and
# orphan-reapers can identify this actor for its full
# lifetime — e.g.
# `tractor[doggy@1027301b]`
# vs. the default uninformative
# `python -m tractor._child --uid (...)`
#
# `setproctitle` mutates `argv[0]` (visible in
# `/proc/<pid>/cmdline`) AND the kernel `comm`
# (visible in `/proc/<pid>/comm`, kernel-truncated to
# ~15 bytes, but preserved through zombie state). Both
# surfaces are enough for `_testing._reap` /
# `acli.reap` orphan- and zombie-detection to identify
# tractor sub-actors via intrinsic signals — no cwd,
# venv path, or env-var coincidence-of-implementation
# matching needed.
#
# NB: an earlier draft also wrote `TRACTOR_AID` to
# `os.environ` here for `pgrep --env`-style discovery,
# but Linux snapshots `/proc/<pid>/environ` at exec/fork
# time, so post-fork runtime mutations don't propagate
# to the kernel-visible env. The proc-title path
# provides equivalent ergonomics
# (`pgrep -f 'tractor\['`) without that gotcha.
from .devx._proctitle import set_actor_proctitle
set_actor_proctitle(subactor)
_trio_main(
subactor,
parent_addr=parent_addr,
infect_asyncio=infect_asyncio,
)
if __name__ == "__main__":
__tracebackhide__: bool = True
@ -49,15 +123,10 @@ if __name__ == "__main__":
parser.add_argument("--asyncio", action='store_true')
args = parser.parse_args()
subactor = Actor(
name=args.uid[0],
uuid=args.uid[1],
_actor_child_main(
uid=args.uid,
loglevel=args.loglevel,
spawn_method="trio"
)
_trio_main(
subactor,
parent_addr=args.parent_addr,
infect_asyncio=args.asyncio,
spawn_method='trio',
)

22
tractor/_exceptions.py
View File

@ -89,6 +89,28 @@ class ActorFailure(RuntimeFailure):
'''
class ActorTooSlowError(RuntimeFailure):
'''
A peer-`Actor` failed to ack an actor-runtime cancel-cascade
request (e.g. `Portal.cancel_actor()` -> `Actor.cancel()`)
within the bounded wait window.
Distinct exc-type (NOT a `trio.TooSlowError` subclass) so that
`except trio.TooSlowError:` blocks elsewhere in the test-suite
or `tractor` internals do NOT silently mask actor-cancel
timeouts — these MUST propagate so a supervisor can escalate
to `proc.terminate()` (hard-kill) per SC-discipline:
graceful cancel-req -> bounded wait -> hard-kill
Reason: see #subint_forkserver duplicate-name hang
diagnosis where `Portal.cancel_actor()` silently swallowed
the timeout and the supervisor never escalated, leaving
a same-named sibling subactor parked forever.
'''
class InternalError(RuntimeError):
'''
Entirely unexpected internal machinery error indicating

156
tractor/_root.py
View File

@ -69,6 +69,20 @@ from ._exceptions import (
logger = log.get_logger('tractor')
# Spawn backends under which `debug_mode=True` is supported.
# Requirement: the spawned subactor's root runtime must be
# trio-native so `tractor.devx.debug._tty_lock` works. Matches
# both the enable-site in `open_root_actor` and the cleanup-
# site reset of `_runtime_vars['_debug_mode']` — keep them in
# lockstep when adding backends.
_DEBUG_COMPATIBLE_BACKENDS: tuple[str, ...] = (
'trio',
# forkserver children run `_trio_main` in their own OS
# process — same child-side runtime shape as `trio_proc`.
'main_thread_forkserver',
)
# TODO: stick this in a `@acm` defined in `devx.debug`?
# -[ ] also maybe consider making this a `wrapt`-deco to
# save an indent level?
@ -227,6 +241,7 @@ async def open_root_actor(
f'_registry_addrs: {registry_addrs!r}\n'
)
# debug.mk_pdb().set_trace()
async with maybe_block_bp(
debug_mode=debug_mode,
maybe_enable_greenback=maybe_enable_greenback,
@ -270,6 +285,82 @@ async def open_root_actor(
)
enable_modules.extend(rpc_module_paths)
# `TRACTOR_LOGLEVEL` env-var wins over any caller-passed
# `loglevel` so devs/test-runs can crank (or silence)
# console verbosity without touching application code.
env_ll_report: str = ''
if env_ll := os.environ.get('TRACTOR_LOGLEVEL'):
# capture the caller-passed value BEFORE the env-var
# clobbers it, else the override-notice below is dead
# code (the `!=` compare is always `False`).
caller_ll: str|None = loglevel
loglevel = env_ll
env_ll_report: str = (
f'Detected env-var setting,\n'
f'TRACTOR_LOGLEVEL={env_ll!r}\n'
f'\n'
f'Setting console loglevel per,\n'
f'loglevel={loglevel!r}\n'
)
if (
caller_ll
and
caller_ll.upper() != env_ll.upper()
):
env_ll_report += (
f'\n'
f'NOTE env-var OVERRIDES caller-passed,\n'
f'loglevel={caller_ll!r}\n'
)
loglevel: str = (
loglevel
or
log._default_loglevel
)
loglevel: str = loglevel.upper()
assert loglevel
_log = log.get_console_log(
level=loglevel,
name='tractor',
logger=logger,
)
assert _log
if env_ll_report:
_log.info(env_ll_report)
# `TRACTOR_SPAWN_METHOD` env-var wins over any caller-passed
# `start_method` so devs/test-runs can swap the actor spawn
# backend without touching application code (e.g. driving
# the `examples/debugging/<script>.py` suite under each
# backend from `tests/devx/conftest.py::spawn`).
if env_sm := os.environ.get('TRACTOR_SPAWN_METHOD'):
# capture the caller-passed value BEFORE the env-var
# clobbers it (else the override-notice is dead code).
caller_sm: str|None = start_method
start_method: str = env_sm
env_sm_report: str = (
f'Detected env-var setting,\n'
f'TRACTOR_SPAWN_METHOD={env_sm!r}\n'
f'\n'
f'Setting spawn backend as,\n'
f'start_method={env_sm!r}\n'
)
if (
caller_sm
and
caller_sm != env_sm
):
_log.warning(
env_sm_report
+
f'NOTE env-var OVERRIDES caller-passed,\n'
f'`start_method={caller_sm!r}`\n'
)
else:
_log.info(env_sm_report)
if start_method is not None:
_spawn.try_set_start_method(start_method)
@ -286,17 +377,44 @@ async def open_root_actor(
wrap_address(uw_addr)
for uw_addr in uw_reg_addrs
]
loglevel: str = (
loglevel
or
log._default_loglevel
)
loglevel: str = loglevel.upper()
# fail-fast on `enable_transports` / `registry_addrs` proto
# mismatch — historically this caused a silent indefinite
# hang during the registrar handshake (registry was reachable
# only via a transport not in `enable_transports`, so the
# actor could never connect to register/discover). See
# `tests/ipc/test_multi_tpt.py::test_root_passes_tpt_to_sub`
# for the foot-gun case + its layer-1 skip-guard.
bad_addrs: list[tuple[str, Address]] = [
(addr.proto_key, addr)
for addr in registry_addrs
if addr.proto_key not in enable_transports
]
if bad_addrs:
mismatch_lines: str = '\n'.join(
f' - proto_key={pk!r} addr={a!r}'
for pk, a in bad_addrs
)
raise ValueError(
f'`registry_addrs` contains addr(s) whose proto is '
f'not in `enable_transports`!\n'
f'enable_transports: {enable_transports!r}\n'
f'mismatched_addrs:\n'
f'{mismatch_lines}\n'
f'\n'
f'Either add the missing proto to '
f'`enable_transports`, or remove the addr from '
f'`registry_addrs`.'
)
# Debug-mode is currently only supported for backends whose
# subactor root runtime is trio-native (so `tractor.devx.
# debug._tty_lock` works). See `_DEBUG_COMPATIBLE_BACKENDS`
# module-const for the list.
if (
debug_mode
and
_spawn._spawn_method == 'trio'
_spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
):
_state._runtime_vars['_debug_mode'] = True
@ -318,22 +436,18 @@ async def open_root_actor(
elif debug_mode:
raise RuntimeError(
"Debug mode is only supported for the `trio` backend!"
f'Debug mode currently supported only for '
f'{_DEBUG_COMPATIBLE_BACKENDS!r} spawn backends, not '
f'{_spawn._spawn_method!r}.'
)
assert loglevel
_log = log.get_console_log(
level=loglevel,
name='tractor',
)
assert _log
# TODO: factor this into `.devx._stackscope`!!
if (
debug_mode
and
enable_stack_on_sig
):
#
# NOTE, intentionally NOT gated on `debug_mode` so SIGUSR1
# task-tree dumps work in plain (non-pdb) runs too — esp.
# in infected-`asyncio` root processes where the default
# SIGUSR1 action would otherwise terminate the proc.
if enable_stack_on_sig:
from .devx._stackscope import enable_stack_on_sig
enable_stack_on_sig()
@ -619,7 +733,7 @@ async def open_root_actor(
if (
debug_mode
and
_spawn._spawn_method == 'trio'
_spawn._spawn_method in _DEBUG_COMPATIBLE_BACKENDS
):
_state._runtime_vars['_debug_mode'] = False

1202
tractor/_testing/_reap.py 100644
View File

File diff suppressed because it is too large Load Diff

36
tractor/_testing/addr.py
View File

@ -22,6 +22,7 @@ Might be eventually useful to expose as a util set from
our `tractor.discovery` subsys?
'''
import os
import random
from typing import (
Type,
@ -31,17 +32,28 @@ from tractor.discovery import _addr
def get_rando_addr(
tpt_proto: str,
*,
# choose random port at import time
_rando_port: str = random.randint(1000, 9999)
) -> tuple[str, str|int]:
'''
Used to globally override the runtime to the
per-test-session-dynamic addr so that all tests never conflict
with any other actor tree using the default.
Cross-process isolation: TCP-port picks salt
`random.randint()` with `os.getpid()` so two parallel
pytest sessions (e.g. one running `--tpt-proto=tcp` and
another `--tpt-proto=uds` concurrently) almost-never
collide on the same port. Without the salt, the prior
impl's import-time `random.randint(1000, 9999)` default
arg was effectively a process-singleton with a 1/9000
chance of cross-run collision per pair — and when it
happened EVERY `reg_addr`-using test in BOTH runs would
fight over the bind, cascading into a chain of
"Address already in use" failures.
For UDS this concern doesn't apply: `UDSAddress.get_random()`
already builds socket paths from `os.getpid()` so each
pytest process gets its own socket-path namespace.
'''
addr_type: Type[_addr.Addres] = _addr._address_types[tpt_proto]
def_reg_addr: tuple[str, int] = _addr._default_lo_addrs[tpt_proto]
@ -51,9 +63,21 @@ def get_rando_addr(
testrun_reg_addr: tuple[str, int|str]
match tpt_proto:
case 'tcp':
# Per-call randomness mixed with `os.getpid()` —
# see the docstring above for the cross-process
# isolation rationale. The mix means:
# - within one pytest session, two calls return
# distinct ports (good for tests that need a
# second-different-reg-addr in one fn body, e.g.
# `test_tpt_bind_addrs::bind-subset-reg`),
# - across parallel pytest sessions, the pid bias
# makes coincident port choices unlikely.
port: int = 1000 + (
random.randint(0, 8999) + os.getpid()
) % 9000
testrun_reg_addr = (
addr_type.def_bindspace,
_rando_port,
port,
)
# NOTE, file-name uniqueness (no-collisions) will be based on

680
tractor/_testing/pytest.py
View File

@ -24,16 +24,192 @@ from functools import (
wraps,
)
import inspect
import os
import platform
from typing import (
Callable,
get_args,
TYPE_CHECKING,
)
import warnings
import pytest
import tractor
from tractor.spawn._spawn import SpawnMethodKey
import trio
# Re-export `_testing.trace`'s pytest fixtures so they're
# picked up by pytest's plugin-discovery (this module is
# loaded via `pytest_plugins` from `pyproject.toml`). The
# `noqa: F401` annotations make linters tolerate the
# unused-looking imports — they're load-bearing for pytest
# discovery. The fixtures share their `name=` kw with the
# underlying CM functions; the python-level identifiers
# below carry the `_fixture` suffix to avoid module-scope
# collision (see `_testing/trace.py` for details).
from .trace import ( # noqa: F401
afk_alarm_w_trace_fixture,
fail_after_w_trace_fixture,
)
# Spawn-backend keys which may appear in `skipon_spawn_backend`
# marks ahead of the named backend actually being registered in
# `tractor.spawn._spawn.SpawnMethodKey`; such marks are inert
# (they can never match an active backend) but must not break
# collection.
_IN_DEV_SPAWN_BACKENDS: tuple[str, ...] = (
'subint',
'subint_forkserver',
'main_thread_forkserver',
)
# Sub-plugin: zombie-subactor + UDS sock-file + shm
# reaping fixtures live in `tractor._testing._reap`
# alongside the underlying detection/cleanup helpers.
# Loading `_reap` as a sub-plugin here keeps reaping
# concerns co-located + this module focused on tractor-
# tooling-specific hooks (option/marker/parametrize,
# `tractor_test` deco, transport / spawn-method
# fixtures).
pytest_plugins: tuple[str, ...] = (
'tractor._testing._reap',
)
if TYPE_CHECKING:
from argparse import Namespace
_cap_sys_passed_as_flag: bool = False
# Spawn backends that need `--capture=sys` to avoid the
# fork-child×pytest-capture-fd deadlock. See the long
# NOTE in `pytest_load_initial_conftests` below for the
# full mechanism + tradeoff write-up.
_CAPSYS_REQUIRED_SPAWNERS: frozenset[str] = frozenset({
'main_thread_forkserver',
# TODO future variant-2 'subint_forkserver' lands
# here too once the impl is unblocked.
})
# XXX REQUIRED in order to enforce `--capture=` flag
# pre test session.
# https://docs.pytest.org/en/stable/reference/reference.html#bootstrapping-hooks
@pytest.hookimpl(tryfirst=True)
def pytest_load_initial_conftests(
early_config: pytest.Config,
parser: pytest.Parser,
args: list[str],
):
'''
Validate the `--capture=` × `--spawn-backend=`
combination at session-startup.
Background
----------
`--capture=sys` is REQUIRED for fork-based spawn backends (e.g.
`main_thread_forkserver`): default `--capture=fd` redirects fd
1,2 to temp files, and fork children inherit those fds — opaque
deadlocks happen in the pytest-capture-machinery ↔ fork-child
stdio interaction. `--capture=sys` only redirects Python- level
`sys.stdout`/`sys.stderr`, leaving fd 1,2 alone.
Trade-off (vs. `--capture=fd`):
- LOST: per-test attribution of subactor *raw-fd* output (C-ext
writes, `os.write(2, ...)`, subproc stdout). Not zero — those
go to the terminal, captured by CI's terminal-level capture,
just not per-test-scoped in the pytest failure report.
- KEPT: Python-level `print()` + `logging` capture per-test
(tractor's logger uses `sys.stderr`, so tractor log output IS
still attributed per-test).
- KEPT: user `pytest -s` for debugging (unaffected).
Full post-mortem in
`ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md`.
Validation policy:
- **CI mode** (`CI` env-var set): fail-fast at
session start if a fork-spawn backend is requested
WITHOUT `--capture=sys`. CI must be explicit; no
auto-fallbacks. Forces every CI matrix-row's run
line to declare its capture mode plainly.
- **Local mode** (no `CI` env-var): emit a loud
warning + suggest `--capture=sys`, but allow the
run to proceed. Lets devs experiment with the bad
combo (e.g. to validate whether recent
fork-survival fixes have made `--capture=fd` work
after all).
'''
global _cap_sys_passed_as_flag
opts_w_args: Namespace = parser.parse_known_args(args)
spawner: str|None = getattr(
opts_w_args,
'spawn_backend',
None,
)
capture: str|None = getattr(
opts_w_args,
'capture',
None,
)
if '--capture=sys' in args:
_cap_sys_passed_as_flag = True
assert capture == 'sys'
in_ci: bool = bool(os.environ.get('CI'))
if (
spawner in _CAPSYS_REQUIRED_SPAWNERS
and
capture == 'fd'
):
msg: str = (
f'\n'
f'XXX `--spawn-backend={spawner}` REQUIRES '
f'`--capture=sys` XXX\n'
f'fork-child × `--capture=fd` is a known '
f'deadlock pattern.\n'
f'See `tractor._testing.pytest`\'s '
f'`pytest_load_initial_conftests` docstring '
f'for the full mechanism.\n'
f'\n'
f'Re-invoke with `--capture=sys` (or run '
f'with `pytest -s` for no capture).\n'
)
# fail-fast: CI must declare capture explicitly for
# fork-spawn backends.
if in_ci:
pytest.exit(
f'{msg}\n'
f'FAIL-FAST: CI=1 detected; aborting session.\n',
returncode=2,
)
# local: loud warn but let the run proceed so devs can
# experiment.
else:
warnings.warn(
f'{msg}\n'
f'Local mode (no `CI` env var) — '
f'continuing. Expect potential hangs.\n',
category=UserWarning,
stacklevel=1,
)
# ??TODO?? is there a way to force the `--capture=sys` sin CLI ??
# - [x] ask pytest peeps in chat!
# - [x] pytest` issue,
# https://github.com/pytest-dev/pytest/issues/14444
# TODO, set various `$TRACTOR_X*` osenv vars here!
print(
f'Applying `tractor`-specific `pytest` config,\n'
f'{opts_w_args!r}\n'
)
def tractor_test(
wrapped: Callable|None = None,
@ -112,11 +288,17 @@ def tractor_test(
# injection (via `__wrapped__`) without leaking the async
# nature.
@wraps(wrapped)
def wrapper(**kwargs):
def wrapper(
set_fork_aware_capture: pytest.CaptureFixture|None = None,
# ^NOTE when set, the decorated fn declared as fixture-param.
**kwargs,
):
__tracebackhide__: bool = hide_tb
# NOTE, ensure we inject any test-fn declared fixture
# names.
sig = inspect.signature(wrapped)
for kw in [
'reg_addr',
'loglevel',
@ -125,9 +307,13 @@ def tractor_test(
'tpt_proto',
'timeout',
]:
if kw in inspect.signature(wrapped).parameters:
if kw in sig.parameters:
assert kw in kwargs
if 'set_fork_aware_capture' in sig.parameters:
assert set_fork_aware_capture
kwargs['set_fork_aware_capture'] = set_fork_aware_capture
# Extract runtime settings as locals for
# `open_root_actor()`; these must NOT leak into
# `kwargs` when the test fn doesn't declare them
@ -170,7 +356,6 @@ def tractor_test(
# invoke test-fn body IN THIS task
await wrapped(**kwargs)
# invoke runtime via a root task.
return trio.run(
partial(
_main,
@ -184,13 +369,6 @@ def tractor_test(
def pytest_addoption(
parser: pytest.Parser,
):
# parser.addoption(
# "--ll",
# action="store",
# dest='loglevel',
# default='ERROR', help="logging level to set when testing"
# )
parser.addoption(
"--spawn-backend",
action="store",
@ -212,6 +390,21 @@ def pytest_addoption(
),
)
parser.addoption(
"--enable-stackscope",
action="store_true",
dest='enable_stackscope',
default=False,
help=(
'Install `stackscope` SIGUSR1 handler in pytest + '
'every spawned subactor for live trio task-tree '
'dumps during hang investigations. Lighter than '
'`--tpdb` (no pdb machinery / tty-lock contention) '
'— use when you only need stack visibility. To '
'capture: `kill -USR1 <pytest-or-subactor-pid>`.'
),
)
# provide which IPC transport protocols opting-in test suites
# should accumulatively run against.
parser.addoption(
@ -223,11 +416,66 @@ def pytest_addoption(
help="Transport protocol to use under the `tractor.ipc.Channel`",
)
# console loglevel for the test-session, scoped to the
# consuming-project's OWN pkg-hierarchy (see the
# `testing_pkg_name` fixture). For `tractor` itself this IS the
# runtime loglevel; downstream projects use `--ll` for their own
# ("internal") app-logging and `--tl` for tractor-as-runtime.
parser.addoption(
"--ll",
"--loglevel",
action="store",
dest='loglevel',
default=None,
help=(
"console loglevel to set for the test session, scoped to "
"the consuming-project pkg (see `testing_pkg_name`). "
"Falls through as the `--tl` default."
),
)
def pytest_configure(config):
backend = config.option.spawn_backend
# tractor-as-runtime loglevel, DISTINCT from `--ll` so downstream
# projects can split their app-logs from the `tractor.*` runtime
# hierarchy. Accepts a `tractor.log` "logging-spec" (see
# `tractor.log.apply_logspec()`).
parser.addoption(
"--tl",
"--tractor-loglevel",
action="store",
dest='tractor_loglevel',
default=None,
help=(
"loglevel (or logging-spec) for `tractor`-as-runtime, "
"distinct from `--ll`. Accepts a bare level (eg. "
"'info', 'cancel') or a sub-logger filter-spec, "
"'<sublog>:<level>,...' (eg. "
"'devx:runtime,trionics:cancel'). Falls back to `--ll` "
"when unset. Mirrors the logging-spec grammar consumed "
"by `tractor.log.apply_logspec()` (see its sub-pkg "
"granularity caveat)."
),
)
def pytest_configure(
config: pytest.Config,
):
# opts: Namespace = config.option
# print(
# f'PYTEST_CONFIGURE\n'
# f'capture={opts.capture!r}\n'
# )
# breakpoint()
backend: str = config.option.spawn_backend
from tractor.spawn._spawn import try_set_start_method
try:
try_set_start_method(backend)
except RuntimeError as err:
# e.g. `--spawn-backend=subint` on Python < 3.14 — turn the
# runtime gate error into a clean pytest usage error so the
# suite exits with a helpful banner instead of a traceback.
raise pytest.UsageError(str(err)) from err
# register custom marks to avoid warnings see,
# https://docs.pytest.org/en/stable/how-to/writing_plugins.html#registering-custom-markers
@ -235,10 +483,139 @@ def pytest_configure(config):
'markers',
'no_tpt(proto_key): test will (likely) not behave with tpt backend'
)
config.addinivalue_line(
'markers',
'skipon_spawn_backend(*start_methods, reason=None): '
'skip this test under any of the given `--spawn-backend` '
'values; useful for backend-specific known-hang / -borked '
'cases (e.g. the `subint` GIL-starvation class documented '
'in `ai/conc-anal/subint_sigint_starvation_issue.md`).'
)
# `--enable-stackscope`: install SIGUSR1 → trio task-tree
# dump in pytest itself + propagate to every subactor via
# an env var that fork-children inherit and the runtime
# gate honors. Lighter than `--tpdb` (no pdb machinery) —
# purely for hang-investigation stack visibility.
if getattr(
config.option,
'enable_stackscope',
False
):
# Env var inherited via fork → subactor's runtime
# picks it up at `Actor.async_main` startup. See the
# gate in `tractor.runtime._runtime` matching this
# var name.
os.environ['TRACTOR_ENABLE_STACKSCOPE'] = '1'
# Install in pytest itself so `kill -USR1 <pytest>`
# dumps the parent trio task-tree (which is where
# most Mode-A-class hangs park).
try:
from tractor.devx._stackscope import (
enable_stack_on_sig,
)
enable_stack_on_sig()
except ImportError:
warnings.warn(
'`stackscope` not installed — '
'--enable-stackscope is a no-op. '
'Install via the `devx` dep group.'
)
else:
os.environ.pop('TRACTOR_ENABLE_STACKSCOPE', None)
def pytest_collection_modifyitems(
config: pytest.Config,
items: list[pytest.Function],
):
'''
Expand any `@pytest.mark.skipon_spawn_backend('<backend>'[,
...], reason='...')` markers into concrete
`pytest.mark.skip(reason=...)` calls for tests whose
backend-arg set contains the active `--spawn-backend`.
Uses `item.iter_markers(name=...)` which walks function +
class + module-level marks in the correct scope order (and
handles both the single-`MarkDecorator` and `list[Mark]`
forms of a module-level `pytestmark`) — so the same marker
works at any level a user puts it.
'''
backend: str = config.option.spawn_backend
default_reason: str = f'Borked on --spawn-backend={backend!r}'
for item in items:
for mark in item.iter_markers(name='skipon_spawn_backend'):
skip_backends: tuple[str] = mark.args
for skip_backend in skip_backends:
assert (
skip_backend in get_args(SpawnMethodKey)
or
skip_backend in _IN_DEV_SPAWN_BACKENDS
)
# ?TODO, run these through the try-set-backend checker to
# avoid typos?
if backend in skip_backends:
reason: str = mark.kwargs.get(
'reason',
default_reason,
)
item.add_marker(pytest.mark.skip(reason=reason))
# first matching mark wins; no value in stacking
# multiple `skip`s on the same item.
break
@pytest.fixture(
scope="session",
autouse=True,
)
def alert_on_finish():
'''
Ring a terminal notification on full test session
completion to alert any would be human.
'''
# TODO, check attached to tty or skip!
yield # run all tests
print("\a") # trigger terminal bell
# ?TODO, any other nice-tricks/specific tuis we could try?
# - supposedly works in many terminals:
# >> print("\033]5;Alert: Tests Finished\a")
# - sway/i3-nag?
@pytest.fixture(autouse=True)
def _reset_runtime_vars():
'''
Per-test isolation of the process-global
`tractor.runtime._state._runtime_vars`.
`open_root_actor()` writes `_enable_tpts` (and other runtime
vars) into this module-global dict, but nothing resets it on
actor teardown. Under the in-process `pytest` launchpad a
uds-using test therefore leaks `_enable_tpts=['uds']` into a
sibling tcp test, which then trips the
`registry_addrs`×`enable_transports` proto-guard in
`open_root_actor()` with a `ValueError`. Snapshot + restore
around every test so no runtime-var state crosses a test
boundary.
'''
from tractor.runtime import _state
snapshot: dict = dict(_state._runtime_vars)
try:
yield
finally:
_state._runtime_vars.clear()
_state._runtime_vars.update(snapshot)
@pytest.fixture(scope='session')
def debug_mode(request) -> bool:
def debug_mode(
request: pytest.FixtureRequest,
) -> bool:
'''
Flag state for whether `--tpdb` (for `tractor`-py-debugger)
was passed to the test run.
@ -252,12 +629,145 @@ def debug_mode(request) -> bool:
@pytest.fixture(scope='session')
def spawn_backend(request) -> str:
def testing_pkg_name() -> str:
'''
Root pkg-name of the project consuming this plugin, used to
scope `--ll` "internal"/app-level console logging into that
project's OWN `tractor.log.get_logger(pkg_name=<.>)` hierarchy
— distinct from the `tractor.*` runtime hierarchy configured
via `--tl`.
Defaults to `'tractor'` (so tractor's own suite treats `--ll`
as the runtime level). Downstream projects override this from
their `conftest.py`, eg.
.. code:: python
@pytest.fixture(scope='session')
def testing_pkg_name() -> str:
return 'modden'
'''
return 'tractor'
@pytest.fixture(
scope='session',
autouse=True,
)
def loglevel(
request: pytest.FixtureRequest,
testing_pkg_name: str,
) -> str|None:
'''
Resolve + apply the test-session console loglevels and yield
the `tractor`-runtime level (also passed to
`open_root_actor(loglevel=<.>)` by `@tractor_test`).
- `--tl <logspec>`: tractor-runtime level (falls back to the
generic `--ll`); applied to the `tractor.*` logger hierarchy
and `tractor.log._default_loglevel` via
`tractor.log.apply_logspec()`.
- `--ll <level>`: the consuming-project's OWN console loglevel,
applied to its `testing_pkg_name` hierarchy when that isn't
`tractor` itself.
'''
import tractor
orig: str = tractor.log._default_loglevel
ll: str|None = request.config.option.loglevel
tl: str|None = request.config.option.tractor_loglevel
# tractor-runtime loglevel: explicit `--tl` wins, else fall
# back to the generic `--ll`, else leave the lib default.
logspec: str|None = tl if tl is not None else ll
tractor_level: str|None = None
if logspec is not None:
tractor_level, _ = tractor.log.apply_logspec(
logspec,
default_level=ll,
pkg_name='tractor',
)
if tractor_level is not None:
tractor.log._default_loglevel = tractor_level
# consuming-project ("internal") console logging at the generic
# `--ll` level, scoped to ITS OWN pkg-hierarchy (NOT `tractor.*`)
# so downstream projects can split app-logs from runtime-logs.
if (
ll is not None
and
testing_pkg_name
and
testing_pkg_name != 'tractor'
):
tractor.log.get_console_log(
level=ll,
pkg_name=testing_pkg_name,
name=testing_pkg_name,
)
log = tractor.log.get_console_log(
level=tractor_level,
name='tractor', # <- enable root logger
)
log.info(
f'Test-harness set session loglevels:\n'
f'tractor-runtime (`--tl`/`--ll`): {tractor_level!r}\n'
f'{testing_pkg_name!r} (`--ll`): {ll!r}\n'
)
yield tractor_level
tractor.log._default_loglevel = orig
@pytest.fixture(scope='function')
def test_log(
request: pytest.FixtureRequest,
loglevel: str,
testing_pkg_name: str,
) -> tractor.log.StackLevelAdapter:
'''
Deliver a per test-module-fn logger instance for reporting from
within actual test bodies/fixtures.
For example this can be handy to report certain error cases from
exception handlers using `test_log.exception()`.
The logger is scoped to the consuming-project's
`testing_pkg_name` hierarchy so downstream suites' in-test logs
land under their own pkg, not `tractor.*`.
'''
modname: str = request.function.__module__
log = tractor.log.get_logger(
name=modname,
pkg_name=testing_pkg_name,
)
_log = tractor.log.get_console_log(
level=loglevel,
logger=log,
name=modname,
)
_log.debug(
f'In-test-logging requested\n'
f'test_log.name: {log.name!r}\n'
f'level: {loglevel!r}\n'
)
yield _log
@pytest.fixture(scope='session')
def spawn_backend(
request: pytest.FixtureRequest,
) -> str:
return request.config.option.spawn_backend
@pytest.fixture(scope='session')
def tpt_protos(request) -> list[str]:
def tpt_protos(
request: pytest.FixtureRequest,
) -> list[str]:
# allow quoting on CLI
proto_keys: list[str] = [
@ -285,7 +795,7 @@ def tpt_protos(request) -> list[str]:
autouse=True,
)
def tpt_proto(
request,
request: pytest.FixtureRequest,
tpt_protos: list[str],
) -> str:
proto_key: str = tpt_protos[0]
@ -337,7 +847,6 @@ def pytest_generate_tests(
metafunc: pytest.Metafunc,
):
spawn_backend: str = metafunc.config.option.spawn_backend
if not spawn_backend:
# XXX some weird windows bug with `pytest`?
spawn_backend = 'trio'
@ -345,7 +854,6 @@ def pytest_generate_tests(
# drive the valid-backend set from the canonical `Literal` so
# adding a new spawn backend (e.g. `'subint'`) doesn't require
# touching the harness.
from tractor.spawn._spawn import SpawnMethodKey
assert spawn_backend in get_args(SpawnMethodKey)
# NOTE: used-to-be-used-to dyanmically parametrize tests for when
@ -356,7 +864,8 @@ def pytest_generate_tests(
metafunc.parametrize(
"start_method",
[spawn_backend],
scope='module',
scope='session',
ids=lambda item: f'start_method={spawn_backend}',
)
# TODO, parametrize any `tpt_proto: str` declaring tests!
@ -367,3 +876,136 @@ def pytest_generate_tests(
# proto_tpts, # TODO, double check this list usage!
# scope='module',
# )
def _is_forking_spawner(
start_method: str,
) -> bool:
return start_method in [
'main_thread_forkserver',
'mp_forkserver',
]
@pytest.fixture(scope='session')
def is_forking_spawner(
start_method: str,
) -> bool:
'''
Is the `pytest` run using a `fork()`ing process spawning-backend?
'''
return _is_forking_spawner(start_method)
def maybe_xfail_for_spawner(
request: pytest.FixtureRequest,
start_method: str,
is_forking_spawner: bool,
) -> None:
'''
Fork based spawning backends cause issues with
`pytest`'s fd-capture mechanism and can cause various
suites to hang.
This helper allows skipping/xfailing from a test when
a fork-spawn backend is being used WITHOUT
`--capture=sys`.
'''
capture_mode: str = request.config.option.capture
# `tee-sys` is also sys-level capture (just additionally writes
# to the original `sys.__stdout__/__stderr__`); fork-safe like
# `sys`. Only `fd`-level capture is the deadlock pattern.
if (
capture_mode not in (
'sys',
'tee-sys',
)
and
is_forking_spawner
):
pytest.skip(
f'Spawner {start_method!r} requires the flag,\n'
f'--capture=sys or --capture=tee-sys..\n'
f'(got --capture={capture_mode!r})\n'
)
def maybe_override_capture(
request: pytest.FixtureRequest,
start_method: bool,
) -> str:
if _is_forking_spawner(start_method):
request.getfixturevalue('capsys')
return 'sys'
return request.config.option.capture
@pytest.fixture
def set_fork_aware_capture(
request: pytest.FixtureRequest,
start_method: str,
) -> pytest.CaptureFixture|str:
'''
Force `--capture=sys` method for tests using
a forking-spawner backend due to fd-copying issues
which can oddly make certain tests hang/fail.
'''
# Fast-path: user already passed sys-level capture
# (`sys` or `tee-sys`) at the CLI — no override needed.
if request.config.option.capture in (
'sys',
'tee-sys',
):
return request.config.option.capture
capsys: pytest.CaptureFixture = maybe_override_capture(
request=request,
start_method=start_method,
)
return capsys
# XXX reset?
# with capsys.disabled():
# pass
# return partial(
# maybe_override_capture,
# request=request,
# start_method=start_method,
# )
def pytest_terminal_summary(
terminalreporter,
exitstatus: int,
config: pytest.Config,
) -> None:
'''
End-of-session summary: list all
`fail_after_w_trace`/`afk_alarm_w_trace` snapshot dirs
captured during the run so the human doesn't have to scroll
back through captured-stderr lines to find dump paths.
Reads from `tractor._testing.trace._SNAPSHOT_INDEX` which is
populated by `_do_capture_snapshot()` on each successful
snapshot capture.
No-op when zero snapshots were captured (most sessions).
'''
from .trace import _SNAPSHOT_INDEX
if not _SNAPSHOT_INDEX:
return
tr = terminalreporter
tr.write_sep('=', 'tractor hang-snapshot index')
tr.write_line(
f'{len(_SNAPSHOT_INDEX)} `fail_after_w_trace` / '
f'`afk_alarm_w_trace` snapshot(s) captured this session:'
)
for label, path in _SNAPSHOT_INDEX:
tr.write_line(f' {label}')
tr.write_line(f' → {path}')

1356
tractor/_testing/trace.py 100644
View File

File diff suppressed because it is too large Load Diff

5
tractor/devx/__init__.py
View File

@ -41,6 +41,11 @@ from .pformat import (
pformat_caller_frame as pformat_caller_frame,
pformat_boxed_tb as pformat_boxed_tb,
)
from ._debug_hangs import (
dump_on_hang as dump_on_hang,
track_resource_deltas as track_resource_deltas,
resource_delta_fixture as resource_delta_fixture,
)
# TODO, move this to a new `.devx._pdbp` mod?

227
tractor/devx/_debug_hangs.py 100644
View File

@ -0,0 +1,227 @@
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
Hang-diagnostic helpers for concurrent / multi-interpreter code.
Collected from the `subint` spawn backend bringup (issue #379)
where silent test-suite hangs needed careful teardown
instrumentation to diagnose. This module bottles up the
techniques that actually worked so future hangs are faster
to corner.
Two primitives:
1. `dump_on_hang()` — context manager wrapping
`faulthandler.dump_traceback_later()` with the critical
gotcha baked in: write the dump to a **file**, not
`sys.stderr`. Under `pytest` (and any other output
capturer) stderr gets swallowed and the dump is easy to
miss — burning hours convinced you're looking at the wrong
thing.
2. `track_resource_deltas()` — context manager (+ optional
autouse-fixture factory) logging per-block deltas of
`threading.active_count()` and — if running on py3.13+ —
`len(_interpreters.list_all())`. Lets you quickly rule out
leak-accumulation theories when a suite hangs more
frequently as it progresses (if counts don't grow, it's
not a leak; look for a race on shared cleanup instead).
See issue #379 / commit `26fb820` for the worked example.
'''
from __future__ import annotations
import faulthandler
import sys
import threading
from contextlib import contextmanager
from pathlib import Path
from typing import (
Callable,
Iterator,
)
try:
import _interpreters # type: ignore
except ImportError:
_interpreters = None # type: ignore
__all__ = [
'dump_on_hang',
'track_resource_deltas',
'resource_delta_fixture',
]
@contextmanager
def dump_on_hang(
seconds: float = 30.0,
*,
path: str | Path = '/tmp/tractor_hang.dump',
all_threads: bool = True,
) -> Iterator[str]:
'''
Arm `faulthandler` to dump all-thread tracebacks to
`path` after `seconds` if the with-block hasn't exited.
*Writes to a file, not stderr* — `pytest`'s stderr
capture silently eats stderr-destined `faulthandler`
output, and the same happens under any framework that
redirects file-descriptors. Pointing the dump at a real
file sidesteps that.
Yields the resolved file path so it's easy to read back.
Example
-------
::
from tractor.devx import dump_on_hang
def test_hang():
with dump_on_hang(
seconds=15,
path='/tmp/my_test_hang.dump',
) as dump_path:
trio.run(main)
# if it hangs, inspect dump_path afterward
'''
dump_path = Path(path)
f = dump_path.open('w')
try:
faulthandler.dump_traceback_later(
seconds,
repeat=False,
file=f,
exit=False,
)
try:
yield str(dump_path)
finally:
faulthandler.cancel_dump_traceback_later()
finally:
f.close()
def _snapshot() -> tuple[int, int]:
'''
Return `(thread_count, subint_count)`.
Subint count reported as `0` on pythons lacking the
private `_interpreters` stdlib module (i.e. py<3.13).
'''
threads: int = threading.active_count()
subints: int = (
len(_interpreters.list_all())
if _interpreters is not None
else 0
)
return threads, subints
@contextmanager
def track_resource_deltas(
label: str = '',
*,
writer: Callable[[str], None] | None = None,
) -> Iterator[tuple[int, int]]:
'''
Log `(threads, subints)` deltas across the with-block.
`writer` defaults to `sys.stderr.write` (+ trailing
newline); pass a custom callable to route elsewhere
(e.g., a log handler or an append-to-file).
Yields the pre-entry snapshot so callers can assert
against the expected counts if they want.
Example
-------
::
from tractor.devx import track_resource_deltas
async def test_foo():
with track_resource_deltas(label='test_foo'):
async with tractor.open_nursery() as an:
...
# Output:
# test_foo: threads 2->2, subints 1->1
'''
before = _snapshot()
try:
yield before
finally:
after = _snapshot()
msg: str = (
f'{label}: '
f'threads {before[0]}->{after[0]}, '
f'subints {before[1]}->{after[1]}'
)
if writer is None:
sys.stderr.write(msg + '\n')
sys.stderr.flush()
else:
writer(msg)
def resource_delta_fixture(
*,
autouse: bool = True,
writer: Callable[[str], None] | None = None,
) -> Callable:
'''
Factory returning a `pytest` fixture that wraps each test
in `track_resource_deltas(label=<node.name>)`.
Usage in a `conftest.py`::
# tests/conftest.py
from tractor.devx import resource_delta_fixture
track_resources = resource_delta_fixture()
or opt-in per-test::
track_resources = resource_delta_fixture(autouse=False)
def test_foo(track_resources):
...
Kept as a factory (not a bare fixture) so callers control
`autouse` / `writer` without having to subclass or patch.
'''
import pytest # deferred: only needed when caller opts in
@pytest.fixture(autouse=autouse)
def _track_resources(request):
with track_resource_deltas(
label=request.node.name,
writer=writer,
):
yield
return _track_resources

74
tractor/devx/_proctitle.py 100644
View File

@ -0,0 +1,74 @@
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
Per-actor proc-title via `py-setproctitle`.
Sets a stable, OS-level identifier for each `tractor` actor
process so diag tools (`ps`, `top`, `htop`, `psutil`) and our
own `acli.pytree`/`acli.hung_dump` can show "which actor is
which" at a glance without needing to read full
`/proc/<pid>/cmdline`.
Format:
``tractor[<aid.reprol()>]`` e.g. ``tractor[doggy@1027301b]``
Uses the canonical `Aid.reprol()` form
(``<name>@<uuid_short>``) so the proc-title matches the
identifier shape used in tractor's logs, the `TRACTOR_AID`
env-var, and orphan-reaper scans — one identity across
all surfaces.
Optional dep: silently no-op when `setproctitle` is missing.
'''
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from tractor.runtime._runtime import Actor
# `setproctitle` is an optional dep — tractor's runtime path
# treats this as best-effort diag, so missing import is a
# no-op rather than a hard error.
try:
import setproctitle as _stp
except ImportError:
_stp = None
def set_actor_proctitle(actor: 'Actor') -> str | None:
'''
Set the calling process's proc-title to identify it as a
tractor sub-actor.
Returns the title string set, or `None` if `setproctitle`
isn't available.
Should be called early in the actor's process lifetime
(after `Actor` construction, before `_trio_main`) so the
new title is visible to OS-level tooling for the entire
runtime.
'''
if _stp is None:
return None
title: str = f'tractor[{actor.aid.reprol()}]'
_stp.setproctitle(title)
return title

279
tractor/devx/_stackscope.py
View File

@ -24,7 +24,7 @@ disjoint, parallel executing tasks in separate actors.
'''
from __future__ import annotations
# from functools import partial
from functools import partial
from threading import (
current_thread,
Thread,
@ -47,7 +47,9 @@ from typing import (
import trio
from tractor.runtime import _state
from tractor import log as logmod
from tractor.devx import debug
from tractor.devx import (
debug,
)
log = logmod.get_logger()
@ -61,12 +63,29 @@ if TYPE_CHECKING:
@trio.lowlevel.disable_ki_protection
def dump_task_tree() -> None:
def dump_task_tree(
write_file: bool = False,
write_tty: bool = False,
) -> None:
'''
Do a classic `stackscope.extract()` task-tree dump to console at
`.devx()` level.
When `write_file`/`write_tty` are set, ALSO tee the rendered
tree to capture-bypassing sinks so SIGUSR1 dumps remain
visible when the parent process has captured stdio (e.g.
pytest's default `--capture=fd`); the SIGUSR1 handler passes
`write_file=True` for exactly this reason:
- `write_file` -> `/tmp/tractor-stackscope-<pid>.log`
(append-mode) — guaranteed-readable artifact even under CI
/ `nohup` / no-tty conditions. `tail -f` to follow.
- `write_tty` -> `/dev/tty` if a controlling terminal is
attached — best-effort, ignored if the device is missing
or write fails. pytest never captures the tty.
'''
import os
import stackscope
tree_str: str = str(
stackscope.extract(
@ -96,46 +115,158 @@ def dump_task_tree() -> None:
# |_{Supervisor/Scope
# |_[Storage/Memory/IPC-Stream/Data-Struct
log.devx(
fpath: str = f'/tmp/tractor-stackscope-{os.getpid()}.log'
from . import pformat
actor_repr: str = pformat.nest_from_op(
input_op='|_',
text=f'{actor}',
nest_prefix='|_',
nest_indent=3,
)
full_dump: str = (
f'Dumping `stackscope` tree for actor\n'
f'(>: {actor.uid!r}\n'
f'(>: {actor.aid.uid!r}\n'
f' |_{mp.current_process()}\n'
f' |_{thr}\n'
f' |_{actor}\n'
# TODO, use the nest_from_op
f'{actor_repr}'
# f' |_{actor}'
f'\n'
f'{sigint_handler_report}\n'
f'signal.getsignal(SIGINT) -> {current_sigint_handler!r}\n'
# f'\n'
# start-of-trace-tree delimiter (mostly for testing)
# f'------ {actor.uid!r} ------\n'
f'\n'
f'------ start-of-{actor.uid!r} ------\n'
f'capture-bypass tee: {fpath}\n'
f'(`tail -f {fpath}` to follow across signals)\n'
f'\n'
f'------ start-of-{actor.aid.uid!r} ------\n'
f'|\n'
f'{tree_str}'
# end-of-trace-tree delimiter (mostly for testing)
f'|\n'
f'|_____ end-of-{actor.uid!r} ______\n'
f'|_____ end-of-{actor.aid.uid!r} ______\n'
)
# TODO: can remove this right?
# -[ ] was original code from author
#
# print(
# 'DUMPING FROM PRINT\n'
# +
# content
# )
# import logging
# try:
# with open("/dev/tty", "w") as tty:
# tty.write(tree_str)
# except BaseException:
# logging.getLogger(
# "task_tree"
# ).exception("Error printing task tree")
log.devx(full_dump)
# NOTE, capture-bypass sinks. Pytest's default
# `--capture=fd` swallows `log.devx()` above; the
# following two writes guarantee the dump reaches the
# human even when stdio is captured.
if write_file:
try:
with open(fpath, 'a') as f:
f.write(full_dump + '\n')
except OSError:
log.exception(
f'Failed to tee stackscope dump to {fpath!r}'
)
if write_tty:
try:
with open('/dev/tty', 'w') as tty:
tty.write(full_dump + '\n')
except OSError:
# no controlling tty (CI / nohup / detached) —
# silently fall through; the file sink covers it.
pass
_handler_lock = RLock()
_tree_dumped: bool = False
# Captured at `enable_stack_on_sig()` time when running
# inside a trio task. `dump_tree_on_sig` uses this to
# schedule `dump_task_tree()` ON the trio loop via
# `token.run_sync_soon` so stackscope sees a real current
# task and can recurse into nursery children. Without
# it (signal handler running in a non-trio stack frame),
# `stackscope.extract` only walks the `<init>` task and
# misses everything inside `async_main`'s nurseries.
_trio_token: trio.lowlevel.TrioToken|None = None
def _relay_sig_to_subactors(sig: int) -> None:
'''
Forward `sig` to every live sub-actor's underlying
process so each runs its own `dump_tree_on_sig`
handler.
Factored out of `dump_tree_on_sig` so the
`run_sync_soon`-deferred path can call it AFTER
the parent's `dump_task_tree()` completes — see
`_dump_then_relay` below for why ordering matters.
'''
an: ActorNursery
for an in _state.current_actor()._actoruid2nursery.values():
subproc: ProcessType
subactor: Actor
for (
subactor,
subproc,
_,
) in an._children.values():
log.warning(
f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
f'{subactor}\n'
f' |_{subproc}\n'
)
# bc of course stdlib can't have a std API.. XD
match subproc:
case trio.Process():
subproc.send_signal(sig)
case mp.Process():
subproc._send_signal(sig)
def _dump_then_relay(
sig: int|None,
) -> None:
'''
`run_sync_soon`-friendly callback: dump THIS actor's
task tree first, THEN relay `sig` to subactors so
their dumps can't race ahead of ours.
Hierarchical-ordering preservation: the legacy
direct-call path (pre-`run_sync_soon`) ran the dump
synchronously inside the signal handler, then
relayed — guaranteeing parent-output-before-child
in the multiplexed pty stream. The pure-deferred
path (schedule dump only, relay sync from handler)
inverts that: relay fires while the parent's
dump is still queued, subs receive SIGUSR1 and
schedule their own dumps, all dumps then race in
arbitrary order through stdio.
Co-scheduling fixes that: by chaining relay AFTER
`dump_task_tree()` inside the same trio-loop
callback, parent output flushes before any sub
receives the signal, restoring the
parent → relay-log → sub-dump ordering humans
expect when reading hang-investigation traces.
Trio prints + crashes on uncaught exceptions in
scheduled callbacks; we swallow + log so the test
keeps running and the user can re-trigger.
'''
try:
dump_task_tree(write_file=True)
except BaseException:
log.exception(
'`dump_task_tree()` raised (scheduled via '
'`run_sync_soon`); continuing.\n'
)
if sig is None:
return
try:
_relay_sig_to_subactors(sig)
except BaseException:
log.exception(
f'`_relay_sig_to_subactors({sig})` raised '
f'(scheduled via `run_sync_soon`); continuing.\n'
)
def dump_tree_on_sig(
sig: int,
@ -159,16 +290,32 @@ def dump_tree_on_sig(
'Trying to dump `stackscope` tree..\n'
)
try:
dump_task_tree()
# await actor._service_n.start_soon(
# partial(
# trio.to_thread.run_sync,
# dump_task_tree,
# )
# )
# trio.lowlevel.current_trio_token().run_sync_soon(
# dump_task_tree
# )
# Prefer scheduling on the trio loop — runs the
# dump from a real trio-task context so
# `stackscope.extract(recurse_child_tasks=True)`
# walks every nursery child instead of seeing
# only the `<init>` task. Falls back to a direct
# call when no token was captured (e.g. signal
# delivered outside a trio.run).
#
# Co-schedule the relay-to-subs in the SAME
# callback so parent's dump prints BEFORE any
# sub receives SIGUSR1 — see `_dump_then_relay`
# for the full hierarchical-ordering rationale.
if _trio_token is not None:
_trio_token.run_sync_soon(
partial(
_dump_then_relay,
sig=sig if relay_to_subs else None,
)
)
# NOTE, `_dump_then_relay` handles the relay
# internally; bail out before the
# direct-path relay below.
return
else:
dump_task_tree(write_file=True)
except RuntimeError:
log.exception(
@ -188,27 +335,15 @@ def dump_tree_on_sig(
# 'Supposedly we dumped just fine..?'
# )
# Direct-path relay (only reached when `_trio_token`
# was None — the run_sync_soon path returned above
# to let `_dump_then_relay` handle the relay
# in-callback).
if not relay_to_subs:
log.devx(f'Skipping {sig!r} relay to subactors..')
return
an: ActorNursery
for an in _state.current_actor()._actoruid2nursery.values():
subproc: ProcessType
subactor: Actor
for subactor, subproc, _ in an._children.values():
log.warning(
f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
f'{subactor}\n'
f' |_{subproc}\n'
)
# bc of course stdlib can't have a std API.. XD
match subproc:
case trio.Process():
subproc.send_signal(sig)
case mp.Process():
subproc._send_signal(sig)
_relay_sig_to_subactors(sig)
def enable_stack_on_sig(
@ -233,19 +368,50 @@ def enable_stack_on_sig(
'''
try:
# NOTE, `stackscope._glue` does intentional async-gen type
# introspection at import-time which trips
# `RuntimeWarning: coroutine method 'asend'/'athrow' was
# never awaited`. Benign — they only want the wrapper
# type — but visible to users. Squelch the import-only
# warning so SIGUSR1 setup stays quiet.
import warnings
with warnings.catch_warnings():
warnings.filterwarnings(
'ignore',
category=RuntimeWarning,
message=r"coroutine method '(asend|athrow)' .* was never awaited",
)
import stackscope
_state._runtime_vars['use_stackscope'] = True
except ImportError:
log.warning(
'The `stackscope` lib is not installed!\n'
'`Ignoring enable_stack_on_sig() call!\n'
)
assert not _state._runtime_vars['use_stackscope']
return None
# Capture the trio token if we're inside `trio.run`
# so SIGUSR1 dispatches the dump *onto* the trio loop
# (full task-tree visibility). When called outside trio
# (e.g. from `pytest_configure`), token capture fails
# silently and `dump_tree_on_sig` falls back to the
# direct-call path.
global _trio_token
try:
_trio_token = trio.lowlevel.current_trio_token()
except RuntimeError:
# not in a `trio.run` — leave None; runtime can
# re-call `enable_stack_on_sig()` later from
# inside `async_main` to capture it.
_trio_token = None
handler: Callable|int = getsignal(sig)
if handler is dump_tree_on_sig:
log.devx(
'A `SIGUSR1` handler already exists?\n'
f'|_ {handler!r}\n'
f'(trio_token captured: {_trio_token is not None})\n'
)
return
@ -259,5 +425,6 @@ def enable_stack_on_sig(
f'{stackscope!r}\n\n'
f'With `SIGUSR1` handler\n'
f'|_{dump_tree_on_sig}\n'
f'(trio_token captured: {_trio_token is not None})\n'
)
return stackscope

4
tractor/devx/debug/_tty_lock.py
View File

@ -181,7 +181,7 @@ class Lock:
return (
f'<{cls.__name__}(\n'
f'{body}'
')>\n\n'
')>\n'
)
@classmethod
@ -282,7 +282,7 @@ class Lock:
):
message += (
'-> No new task holds the TTY lock!\n\n'
f'{Lock.repr()}\n'
f'{Lock.repr()}'
)
elif (

12
tractor/ipc/_linux.py
View File

@ -17,10 +17,20 @@
Linux specifics, for now we are only exposing EventFD
'''
import os
import errno
import os
import sys
try:
import cffi
except ImportError as ie:
if sys.version_info >= (3, 14):
ie.add_note(
f'The `cffi` pkg has no 3.14 support yet.\n'
)
raise ie
import trio
ffi = cffi.FFI()

34
tractor/ipc/_mp_bs.py
View File

@ -17,7 +17,7 @@
Utils to tame mp non-SC madeness
'''
import platform
from functools import partial
def disable_mantracker():
@ -27,30 +27,13 @@ def disable_mantracker():
'''
from multiprocessing.shared_memory import SharedMemory
# 3.13+ only.. can pass `track=False` to disable
# all the resource tracker bs.
# https://docs.python.org/3/library/multiprocessing.shared_memory.html
if (_py_313 := (
platform.python_version_tuple()[:-1]
>=
('3', '13')
)
):
from functools import partial
return partial(
SharedMemory,
track=False,
)
# !TODO, once we drop 3.12- we can obvi remove all this!
else:
from multiprocessing import (
resource_tracker as mantracker,
)
# Tell the "resource tracker" thing to fuck off.
# XXX ALWAYS disable the stdlib's "resource tracker"; it prevents
# fork backends and never was useful to us since we're SC
# lifetime managing all allocations.
class ManTracker(mantracker.ResourceTracker):
def register(self, name, rtype):
pass
@ -69,7 +52,12 @@ def disable_mantracker():
mantracker.unregister = mantracker._resource_tracker.unregister
mantracker.getfd = mantracker._resource_tracker.getfd
# use std type verbatim
shmT = SharedMemory
# 3.13+ only.. can pass `track=False` to disable
# all the resource tracker bs.
# https://docs.python.org/3/library/multiprocessing.shared_memory.html
shmT = partial(
SharedMemory,
track=False,
)
return shmT

28
tractor/ipc/_server.py
View File

@ -1122,20 +1122,32 @@ async def _serve_ipc_eps(
)
finally:
# close every endpoint INDEPENDENTLY: a close raising
# mid-iter (e.g. UDS `os.unlink` racing concurrent reap) must
# not strand the rest of the eps + must not skip the
# `_shutdown.set()` below.
if eps:
addr: Address
ep: Endpoint
for addr, ep in server.epsdict().items():
for addr, ep in list(server.epsdict().items()):
try:
ep.close_listener()
except Exception as ep_close_err:
log.exception(
f'Endpoint close raised, continuing teardown\n'
f' |_{ep!r}\n'
f' |_{ep_close_err!r}\n'
)
finally:
try:
server._endpoints.remove(ep)
except ValueError:
pass
# actor = _state.current_actor()
# if actor.is_arbiter:
# import pdbp; pdbp.set_trace()
# signal the server is "shutdown"/"terminated"
# since no more active endpoints are active.
if not server._endpoints:
# always signal "shutdown" so `actor.cancel()` →
# `ipc_server.wait_for_shutdown()` doesn't deadlock when an
# endpoint close raised above.
if server._shutdown is not None:
server._shutdown.set()
@acm

25
tractor/ipc/_shm.py
View File

@ -929,15 +929,26 @@ def open_shm_list(
# "close" attached shm on actor teardown
try:
actor = tractor.current_actor()
actor.lifetime_stack.callback(shml.shm.close)
# XXX on 3.13+ we don't need to call this?
# -> bc we pass `track=False` for `SharedMemeory` orr?
if (
platform.python_version_tuple()[:-1] < ('3', '13')
):
actor.lifetime_stack.callback(shml.shm.unlink)
# >XXX NOTE< on 3.13+ we need to call this AS WELL AS pass
# `track=False` for `mp.SharedMemeory` otherwise fork based
# backends will error out due to long lived stdlib
# limitations,
# - https://bugs.python.org/issue38119
# - https://bugs.python.org/issue45209
#
def try_unlink():
try:
shml.shm.unlink()
except FileNotFoundError as fne:
log.debug(
f'ShmList already deallocated pre-actor-shutdown.\n'
f'{fne!r}\n'
)
actor.lifetime_stack.callback(try_unlink)
except RuntimeError:
log.warning('tractor runtime not active, skipping teardown steps')

11
tractor/ipc/_uds.py
View File

@ -344,7 +344,18 @@ def close_listener(
'''
lstnr.socket.close()
# tolerate the sock-file being already gone — under concurrent
# pytest sessions sharing the bindspace dir, another session's
# reap path can unlink it first; raising here aborts the
# `_serve_ipc_eps` finally before `_shutdown.set()`, deadlocking
# `wait_for_shutdown()` on `actor.cancel()`.
try:
os.unlink(addr.sockpath)
except FileNotFoundError:
log.warning(
f'UDS sock-file already unlinked, skipping\n'
f' |_{addr.sockpath}\n'
)
async def open_unix_socket_w_passcred(

207
tractor/log.py
View File

@ -543,21 +543,45 @@ def get_logger(
# only includes the first 2 sub-pkg name-tokens in the
# child-logger's name; the colored "pkg-namespace" header
# will then correctly show the same value as `name`.
#
# XXX, strip the trailing `pkg_path` token ONLY when it
# duplicates the caller's leaf-*module* name — which the
# console header already renders via its `{filename}` field.
# We compare against the caller module's `__name__`/
# `__package__` (rather than blindly dropping the last token)
# so genuine, possibly-*nested* sub-PACKAGE components stay
# addressable as their own sub-loggers:
#
# - `name='trionics._broadcast'` (a leaf-module, from a
# `get_logger(__name__)`-style call) -> `tractor.trionics`
# (leaf dropped; `_broadcast.py` is in the header).
# - `name='devx.debug'` (a real sub-PACKAGE, whether
# auto-derived from a module's `__package__` or passed
# explicitly by a logging-spec) -> `tractor.devx.debug`,
# DISTINCT from a bare `devx` -> `tractor.devx`.
#
# The previous unconditional `pkg_path = subpkg_path` also ate
# the deepest sub-pkg, collapsing `devx.debug` -> `tractor.devx`
# and silently breaking per-sub-pkg level control via the
# logging-spec; see `tractor.log.LogSpec`/`apply_logspec()`.
caller_leaf_mod: str|None = None
if (caller_mod := get_caller_mod()):
cmod_name: str = getattr(caller_mod, '__name__', '') or ''
cmod_pkg: str = getattr(caller_mod, '__package__', '') or ''
# a leaf-*module* has `__name__ != __package__`; a package
# `__init__` has them equal (so its trailing token is a
# real sub-pkg, NOT a leaf-module-filename to strip).
if cmod_name and cmod_name != cmod_pkg:
caller_leaf_mod = cmod_name.rpartition('.')[2]
if (
# XXX, TRY to remove duplication cases
# which get warn-logged on below!
(
# when, subpkg_path == pkg_path
subpkg_path
and
rname == pkg_name
)
# ) or (
# # when, pkg_path == leaf_mod
# pkg_path
# and
# leaf_mod == pkg_path
# )
and
# only collapse when the trailing token IS the caller's
# leaf-module (i.e. the `{filename}` already shows it).
leaf_mod == caller_leaf_mod
):
pkg_path = subpkg_path
@ -711,6 +735,167 @@ def get_console_log(
return log
# A `tractor` "logging-spec": a compact, code-free way for a
# consuming project's test-iface (or runtime) to dial-in console
# loglevels across the lib's logger hierarchy. Mirrors the grammar
# consumed by `modden.runtime.daemon.setup_tractor_logging()`.
#
# Accepted forms (`str|bool`),
# - `True` -> enable the `pkg_name` root-logger at
# `default_level` (or 'cancel').
# - `False` -> disable (no-op, configure nothing).
# - 'info' -> a bare level for the root-logger.
# - 'sub:info,x:cancel' -> per-sub-logger levels; each `<name>` is
# RELATIVE to `pkg_name` (must NOT include
# the `pkg_name` token itself), eg.
# 'devx.debug:runtime,trionics:cancel'.
#
# !GRANULARITY! sub-logger names match at the `pkg_name.<name>`
# *logger* level — which (per `get_logger()`'s name-derivation) is
# *sub-PACKAGE* granularity, addressable at ANY nesting depth:
# - 'devx.debug' -> the `tractor.devx.debug` logger, DISTINCT from a
# bare 'devx' -> `tractor.devx` (its parent). Setting `devx` also
# gates `devx.debug` via normal stdlib level-inheritance unless the
# child sets its own level.
# - leaf *modules* are intentionally NOT individually addressable:
# `get_logger()` drops the leaf module-name from the logger key
# since the console header already renders it via `{filename}`, so
# every module in a (sub-)pkg shares that pkg's logger. Per-leaf
# level control would need a record-filter (see follow-up notes:
# `ai/tooling-todos/logspec_leaf_module_granularity_route_b.md`).
# - top-level lib modules (eg. `tractor.to_asyncio`) emit under the
# *root* `pkg_name` logger (their `__package__` IS `pkg_name`), so
# a 'to_asyncio:<level>' entry targets a phantom child that nothing
# emits to -> no-op. Use the bare-level/root form for those.
LogSpec = str|bool
def parse_logspec(
logspec: LogSpec,
default_level: str|None = None,
pkg_name: str = _proj_name,
) -> dict[str|None, str]:
'''
Parse a `tractor` "logging-spec" (see `LogSpec`) into a
`{sublog_name|None: level}` mapping where a `None` key denotes
the `pkg_name` root-logger itself.
'''
match logspec:
# explicit disable -> configure nothing.
case False:
return {}
# enable the root-logger at the fallback level.
case True:
return {None: (default_level or 'cancel')}
case str(spec):
filters: list[str] = [
part.strip()
for part in spec.split(',')
if part.strip()
]
# i. a bare level (no sub-logger filtering),
# eg. 'info' | 'cancel'
if (
len(filters) == 1
and
':' not in filters[0]
):
return {None: filters[0]}
# ii. a per-sub-logger filter-spec of the form,
# '<sublog_0>:<level>,<.. N-other-parts>'
# eg. 'to_asyncio:cancel,devx._debug:runtime'
out: dict[str|None, str] = {}
for log_filter in filters:
name, sep, level = log_filter.partition(':')
if not sep:
raise ValueError(
f'Invalid `tractor` logging-spec part!\n'
f'{log_filter!r}\n'
f'\n'
f'Mixed bare-level + sub-logger filters are '
f'not supported; every comma-part must be '
f'`<sublog>:<level>`.\n'
)
# the sub-logger name is RELATIVE to `pkg_name`;
# duplicating the pkg-token is a user error since
# the root-logger already IS `pkg_name`.
if pkg_name in name.split('.'):
raise ValueError(
f'logging-spec sub-name should NOT include '
f'the `pkg_name={pkg_name!r}` token!\n'
f'got name={name!r}\n'
)
out[name] = level
return out
case _:
raise ValueError(
f'Invalid `tractor` logging-spec!\n'
f'{logspec!r}\n'
)
def apply_logspec(
logspec: LogSpec,
default_level: str|None = None,
pkg_name: str = _proj_name,
) -> tuple[
str|None,
dict[str, StackLevelAdapter],
]:
'''
Parse + apply a `tractor` "logging-spec" (see `parse_logspec()`):
enable a `colorlog` stderr console handler for each
(sub-)logger named in the spec at its requested level.
Returns a 2-tuple,
- the resolved "primary" runtime-level: the root-logger level if
the spec set one, else `default_level`; suitable for passing
to `open_root_actor(loglevel=<.>)`,
- a `{logger_name: StackLevelAdapter}` map of every logger the
spec touched.
'''
specs: dict[str|None, str] = parse_logspec(
logspec,
default_level=default_level,
pkg_name=pkg_name,
)
logs: dict[str, StackLevelAdapter] = {}
for sub_name, level in specs.items():
# NOTE, pass the RELATIVE sub-name (no `pkg_name.` prefix)
# to avoid `get_logger()`'s duplicate-pkg-token warning;
# it re-adds the pkg-name via `.getChild()` internally.
log: StackLevelAdapter = get_console_log(
level=level,
pkg_name=pkg_name,
name=(sub_name or pkg_name),
)
# XXX, a sub-logger filter is "authoritative" for its
# subtree: it gets its OWN stderr handler (added by
# `get_console_log()` above), so DON'T also let its records
# propagate up to a root `pkg_name`-logger handler — that
# would double-emit every line when a root-level console
# (eg. via `--ll`) is also active. The root-level form
# (`sub_name is None`) keeps default propagation.
if sub_name is not None:
log.logger.propagate = False
logs[log.name] = log
primary_level: str|None = specs.get(None, default_level)
return (
primary_level,
logs,
)
def get_loglevel() -> str:
return _default_loglevel

50
tractor/runtime/_portal.py
View File

@ -55,6 +55,7 @@ from ..msg import (
Return,
)
from .._exceptions import (
ActorTooSlowError,
NoResult,
TransportClosed,
)
@ -268,6 +269,7 @@ class Portal:
async def cancel_actor(
self,
timeout: float | None = None,
raise_on_timeout: bool = False,
) -> bool:
'''
@ -281,6 +283,17 @@ class Portal:
`._context.Context.cancel()` which CAN be used for this
purpose.
`raise_on_timeout` (default `False`):
- `False` (legacy): on bounded-wait expiry, log at DEBUG
and return `False`. Used by callers that issue cancel
fire-and-forget and have their own escalation
(e.g. `_spawn.soft_kill()` checks `proc.poll()` after).
- `True`: on bounded-wait expiry, raise `ActorTooSlowError`
so the caller MUST handle the failure explicitly.
`ActorNursery.cancel()` opts in so it can escalate via
`proc.terminate()` per SC-discipline.
'''
__runtimeframe__: int = 1 # noqa
@ -301,15 +314,16 @@ class Portal:
# XXX the one spot we set it?
chan._cancel_called: bool = True
cancel_timeout: float = (
timeout
or
self.cancel_timeout
)
try:
# send cancel cmd - might not get response
# XXX: sure would be nice to make this work with
# a proper shield
with trio.move_on_after(
timeout
or
self.cancel_timeout
) as cs:
with trio.move_on_after(cancel_timeout) as cs:
cs.shield: bool = True
await self.run_from_ns(
'self',
@ -317,16 +331,32 @@ class Portal:
)
return True
if cs.cancelled_caught:
# may timeout and we never get an ack (obvi racy)
# but that doesn't mean it wasn't cancelled.
# `move_on_after` fired — peer didn't ack within
# bounded window. Behaviour depends on
# `raise_on_timeout`:
if (
cs.cancelled_caught
and
raise_on_timeout
):
raise ActorTooSlowError(
f'Peer {peer_id} did not ack its '
f'`Actor.cancel()` RPC within bounded wait '
f'of {cancel_timeout!r}s'
)
# legacy fire-and-forget path: log + return False so
# the caller can decide whether to escalate.
#
# NOTE, we also land here in the (unexpected) case where
# the shielded `move_on_after` block exits WITHOUT
# `return True` and WITHOUT the deadline firing — prefer
# a soft `False` over an `assert`-crash mid-teardown.
log.debug(
f'May have failed to cancel peer?\n'
f'\n'
f'c)=?> {peer_id}\n'
)
# if we get here some weird cancellation case happened
return False
except TransportClosed as tpt_err:

88
tractor/runtime/_runtime.py
View File

@ -870,7 +870,14 @@ class Actor:
accept_addrs: list[UnwrappedAddress]|None = None
if self._spawn_method == "trio":
if self._spawn_method in (
'trio',
'subint',
# `subint_forkserver` parent-side sends a
# `SpawnSpec` over IPC just like the other two
# — fork child-side runtime is trio-native.
'subint_forkserver',
):
# Receive post-spawn runtime state from our parent.
spawnspec: msgtypes.SpawnSpec = await chan.recv()
@ -922,11 +929,29 @@ class Actor:
# => update process-wide globals
# TODO! -[ ] another `Struct` for rtvs..
rvs: dict[str, Any] = spawnspec._runtime_vars
if rvs['_debug_mode']:
from ..devx import (
enable_stack_on_sig,
maybe_init_greenback,
)
# `stackscope` SIGUSR1 handler: install when ANY of
# `_debug_mode` / `use_stackscope` rt-vars OR the
# `TRACTOR_ENABLE_STACKSCOPE` env var is set (the
# latter being a lighter test-time hang-debug path;
# see `tractor._testing.pytest`'s `--enable-stackscope`
# CLI flag — env var propagates via fork-inherited
# environ).
#
# NOTE, NOT *exclusively* gated on `_debug_mode` so
# SIGUSR1 task-tree dumps work in plain (non-pdb)
# runs too — but we DO still install under
# `_debug_mode` since otherwise the default SIGUSR1
# action would terminate the proc, esp. nasty in
# infected-`asyncio` sub-actors mid-REPL.
if (
rvs.get('_debug_mode')
or
rvs.get('use_stackscope')
or
os.environ.get('TRACTOR_ENABLE_STACKSCOPE')
):
from ..devx import enable_stack_on_sig
try:
# TODO: maybe return some status msgs upward
# to that we can emit them in `con_status`
@ -938,10 +963,13 @@ class Actor:
except ImportError:
log.warning(
'`stackscope` not installed for use in debug mode!'
'`stackscope` not installed for use in '
'debug mode / `--enable-stackscope`!'
)
if rvs['_debug_mode']:
if rvs.get('use_greenback', False):
from ..devx import maybe_init_greenback
maybe_mod: ModuleType|None = await maybe_init_greenback()
if maybe_mod:
log.devx(
@ -1209,6 +1237,23 @@ class Actor:
ipc_server.cancel()
await ipc_server.wait_for_shutdown()
# Break the shield on the parent-channel
# `process_messages` loop (started with `shield=True`
# in `async_main` above). Required to avoid a
# deadlock during teardown of fork-spawned subactors:
# without this cancel, the loop parks waiting for
# EOF on the parent channel, but the parent is
# blocked on `os.waitpid()` for THIS actor's exit
# — mutual wait. For exec-spawn backends the EOF
# arrives naturally when the parent closes its
# handler-task socket during its own teardown, but
# in fork backends the shared-process-image makes
# that delivery racy / not guaranteed. Explicit
# cancel here gives us deterministic unwinding
# regardless of backend.
if self._parent_chan_cs is not None:
self._parent_chan_cs.cancel()
# cancel all rpc tasks permanently
if self._service_tn:
self._service_tn.cancel_scope.cancel()
@ -1729,7 +1774,16 @@ async def async_main(
# start processing parent requests until our channel
# server is 100% up and running.
if actor._parent_chan:
await root_tn.start(
# Capture the shielded `loop_cs` for the
# parent-channel `process_messages` task so
# `Actor.cancel()` has a handle to break the
# shield during teardown — without this, the
# shielded loop would park on the parent chan
# indefinitely waiting for EOF that only arrives
# after the PARENT tears down, which under
# fork-based backends (e.g. `main_thread_forkserver`)
# it waits on THIS actor's exit — deadlock.
actor._parent_chan_cs = await root_tn.start(
partial(
_rpc.process_messages,
chan=actor._parent_chan,
@ -1940,7 +1994,25 @@ async def async_main(
f' {pformat(ipc_server._peers)}'
)
log.runtime(teardown_report)
# NOTE: bound the peer-clear wait — otherwise if any
# peer-channel handler is stuck (e.g. never got its
# cancel propagated due to a runtime bug), this wait
# blocks forever and deadlocks the whole actor-tree
# teardown cascade. 3s is enough for any graceful
# cancel-ack round-trip; beyond that we're in bug
# territory and need to proceed with local teardown
# so the parent's `_ForkedProc.wait()` can unblock.
# See `ai/conc-anal/
# subint_forkserver_test_cancellation_leak_issue.md`
# for the full diagnosis.
with trio.move_on_after(3.0) as _peers_cs:
await ipc_server.wait_for_no_more_peers()
if _peers_cs.cancelled_caught:
teardown_report += (
f'-> TIMED OUT waiting for peers to clear '
f'({len(ipc_server._peers)} still connected)\n'
)
log.warning(teardown_report)
teardown_report += (
'-]> all peer channels are complete.\n'

77
tractor/runtime/_state.py
View File

@ -93,6 +93,7 @@ class RuntimeVars(Struct):
repl_fixture: bool|Callable = False # |AbstractContextManager[bool]
# for `tractor.pause_from_sync()` & `breakpoint()` support
use_greenback: bool = False
use_stackscope: bool = False
# infected-`asyncio`-mode: `trio` running as guest.
_is_infected_aio: bool = False
@ -102,7 +103,6 @@ class RuntimeVars(Struct):
key,
val,
) -> None:
breakpoint()
super().__setattr__(key, val)
def update(
@ -117,7 +117,14 @@ class RuntimeVars(Struct):
)
_runtime_vars: dict[str, Any] = {
# The "fresh process" defaults — what `_runtime_vars` looks
# like in a just-booted Python process that hasn't yet entered
# `open_root_actor()` nor received a parent `SpawnSpec`. Kept
# as a module-level constant so `get_runtime_vars(clear_values=
# True)` can reset the live dict back to this baseline (see
# `tractor.spawn._main_thread_forkserver` for the one current
# caller that needs it).
_RUNTIME_VARS_DEFAULTS: dict[str, Any] = {
# root of actor-process tree info
'_is_root': False, # bool
'_root_mailbox': (None, None), # tuple[str|None, str|None]
@ -132,16 +139,19 @@ _runtime_vars: dict[str, Any] = {
# `debug_mode: bool` settings
'_debug_mode': False, # bool
'repl_fixture': False, # |AbstractContextManager[bool]
# for `tractor.pause_from_sync()` & `breakpoint()` support
'use_greenback': False,
'use_greenback': False, # `.pause_from_sync()`/`breakpoint()`
'use_stackscope': False, # trio-task-stack dumps on SIGUSR1
# infected-`asyncio`-mode: `trio` running as guest.
'_is_infected_aio': False,
}
_runtime_vars: dict[str, Any] = dict(_RUNTIME_VARS_DEFAULTS)
def get_runtime_vars(
as_dict: bool = True,
clear_values: bool = False,
) -> dict:
'''
Deliver a **copy** of the current `Actor`'s "runtime variables".
@ -150,11 +160,62 @@ def get_runtime_vars(
form, but the `RuntimeVars` struct should be utilized as possible
for future calls.
'''
if as_dict:
return dict(_runtime_vars)
Pure read — **never mutates** the module-level `_runtime_vars`.
return RuntimeVars(**_runtime_vars)
If `clear_values=True`, return a copy of the fresh-process
defaults (`_RUNTIME_VARS_DEFAULTS`) instead of the live
dict. Useful in combination with `set_runtime_vars()` to
reset process-global state back to "cold" — the main caller
today is the `main_thread_forkserver` spawn backend's post-fork
child prelude:
set_runtime_vars(get_runtime_vars(clear_values=True))
`os.fork()` inherits the parent's full memory image, so the
child sees the parent's populated `_runtime_vars` (e.g.
`_is_root=True`) which would trip the `assert not
self.enable_modules` gate in `Actor._from_parent()` on the
subsequent parent→child `SpawnSpec` handshake if left alone.
'''
src: dict = (
_RUNTIME_VARS_DEFAULTS
if clear_values
else _runtime_vars
)
snapshot: dict = dict(src)
if as_dict:
return snapshot
return RuntimeVars(**snapshot)
def set_runtime_vars(
rtvars: dict | RuntimeVars,
) -> None:
'''
Atomically replace the module-level `_runtime_vars` contents
with those of `rtvars` (via `.clear()` + `.update()` so
live references to the same dict object remain valid).
Accepts either the historical `dict` form or the `RuntimeVars`
`msgspec.Struct` form (the latter still mostly unused but
the blessed forward shape — see the struct's definition).
Paired with `get_runtime_vars()` as the explicit
write-half of the runtime-vars API — prefer this over
direct mutation of `_runtime_vars[...]` from new call sites.
'''
if isinstance(rtvars, RuntimeVars):
# `msgspec.Struct` → dict via its declared field set;
# avoids pulling in `msgspec.structs.asdict` just for
# this one call path.
rtvars = {
field_name: getattr(rtvars, field_name)
for field_name in rtvars.__struct_fields__
}
_runtime_vars.clear()
_runtime_vars.update(rtvars)
def last_actor() -> Actor|None:

123
tractor/runtime/_supervise.py
View File

@ -38,8 +38,14 @@ from ..discovery._addr import (
UnwrappedAddress,
mk_uuid,
)
from ._state import current_actor, is_main_process
from ..log import get_logger, get_loglevel
from ._state import (
current_actor,
is_main_process,
)
from ..log import (
get_logger,
get_loglevel,
)
from ._runtime import Actor
from ._portal import Portal
from ..trionics import (
@ -47,6 +53,7 @@ from ..trionics import (
collapse_eg,
)
from .._exceptions import (
ActorTooSlowError,
ContextCancelled,
)
from .._root import (
@ -60,11 +67,106 @@ if TYPE_CHECKING:
import multiprocessing as mp
# from ..ipc._server import IPCServer
from ..ipc import IPCServer
from ..spawn._spawn import ProcessType
log = get_logger()
async def _try_cancel_then_kill(
portal: Portal,
# `ProcessType` is `TYPE_CHECKING`-only (defined under that
# guard in `..spawn._spawn`) so we stringify here to avoid
# eager runtime eval of the annotation at function-def time
# (this module has no `from __future__ import annotations`).
proc: 'ProcessType',
subactor: Actor,
debug_mode_active: bool = False,
) -> None:
'''
Per-child cancel-then-escalate helper used by
`ActorNursery.cancel()`.
Sends a graceful actor-runtime cancel-RPC via
`Portal.cancel_actor(raise_on_timeout=True)`. If the bounded-wait
expires before the peer ack's, `ActorTooSlowError` is raised and
we escalate via `proc.terminate()` (SIGTERM) per SC-discipline:
graceful cancel-req -> bounded wait -> hard-kill
Without this escalation, a same-name sibling subactor whose
cancel-RPC failed to ack within `Portal.cancel_timeout` (e.g.
under TCP+forkserver register-RPC contention) would park the
parent's `soft_kill()` watcher forever waiting on `proc.poll()`,
deadlocking nursery `__aexit__`. See `ActorTooSlowError` for
the wider write-up.
'''
# XXX, do NOT escalate to `proc.terminate()` while ANY of
# the following are true — SIGTERM-ing a sub would tear
# down its sub-tree including any descendant proxying
# stdio to/from a REPL-locked actor, clobbering the user's
# debug session:
#
# - `Lock.ctx_in_debug is not None`: most precise — some
# actor in the tree is currently REPL-locked. Set in the
# root actor for the lifetime of the lock. Raceable
# (false negative if SIGINT arrives before lock-acquire
# RPC completes).
#
# - `_runtime_vars['_debug_mode']`: root-actor was opened
# with `debug_mode=True` (via `open_root_actor` /
# `open_nursery`). Set once at root boot, never cleared.
# Catches deep-descendant REPL sessions even when the
# intermediate nurseries didn't pass `debug_mode=` per-
# child.
#
# - `debug_mode_active`: this nursery has at least one
# child started with an explicit `debug_mode=` arg
# (`ActorNursery._at_least_one_child_in_debug`). Catches
# the case where root is NOT in debug-mode but a
# nursery-direct child opted in.
#
# Independent because root may NOT be in debug-mode even
# when a child is (only the child's `_runtime_vars` is
# mutated by per-child `debug_mode=True`). ORing covers
# every flavor without false-positively skipping
# legitimate hard-kill paths in non-debug trees.
if (
debug.Lock.ctx_in_debug is not None
or
_state._runtime_vars.get('_debug_mode', False)
or
debug_mode_active
):
await portal.cancel_actor()
return
try:
await portal.cancel_actor(raise_on_timeout=True)
except ActorTooSlowError as too_slow:
log.error(
f'Cancel-ack TIMED OUT for sub-actor\n'
f' uid: {subactor.aid.reprol()!r}\n'
f' reason: {too_slow}\n'
f'-> escalating to `proc.terminate()` (hard-kill)\n'
)
# XXX, the `subint` backend stores an `int` interp-id in the
# `proc` slot (not a `Process`), so it has no `.terminate()`.
# Guard here so a cancel-ack timeout doesn't `AttributeError`
# once that backend lands; its hard-kill path is a TODO.
if hasattr(proc, 'terminate'):
proc.terminate()
else:
log.error(
f'Cannot hard-kill sub-actor — backend proc-handle '
f'{proc!r} ({type(proc).__name__!r}) has no '
f'`.terminate()`!\n'
f' uid: {subactor.aid.reprol()!r}\n'
f'TODO: per-backend cancel-escalation.\n'
)
class ActorNursery:
'''
The fundamental actor supervision construct: spawn and manage
@ -428,10 +530,23 @@ class ActorNursery:
else: # there's no other choice left
proc.terminate()
# spawn cancel tasks for each sub-actor
# spawn per-child cancel tasks; the helper
# escalates to hard-kill on
# `ActorTooSlowError` rather than silently
# swallowing the cancel-ack timeout, EXCEPT
# when this nursery has any debug-eligible
# child (in which case we keep legacy
# fire-and-forget semantics to avoid
# clobbering an active REPL).
assert portal
if portal.channel.connected():
tn.start_soon(portal.cancel_actor)
tn.start_soon(
_try_cancel_then_kill,
portal,
proc,
subactor,
self._at_least_one_child_in_debug,
)
log.cancel(msg)
# if we cancelled the cancel (we hung cancelling remote actors)

183
tractor/spawn/_reap.py 100644
View File

@ -0,0 +1,183 @@
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
Post-mortem subactor cleanup primitives — things the parent
runtime has to clean up because the dead-or-SIGKILL'd child
couldn't.
Sibling of `tractor._testing._reap` which is the test-harness
equivalent (orphan-pid + leaked-shm + leaked-UDS-sock sweeper
fixtures). This module is the spawn-layer counterpart, called
inline from `hard_kill` and the broader subactor reap path.
Today this is just `unlink_uds_bind_addrs()`. As future
post-mortem cleanup needs surface (e.g. `/dev/shm` segment
unlink for hard-crashed actors, leaked-pidfile cleanup), they
land here too.
Future-work TODO — authoritative UDS bind-addr tracking
-------------------------------------------------------
`unlink_uds_bind_addrs()` currently has two cleanup paths:
1. Explicit `bind_addrs` (when parent set them at spawn time)
2. **Convention-based reconstruction** —
`<XDG_RUNTIME_DIR>/tractor/<name>@<pid>.sock` — for the
common case where the subactor self-assigned a random sock
via `UDSAddress.get_random()`.
Path (2) hardcodes the `<name>@<pid>.sock` convention from
`tractor.ipc._uds.UDSAddress`. If that convention ever
changes — or the subactor binds to a non-default
`bindspace`/`filedir` — we'll silently fail to unlink.
A more authoritative approach would be:
- Subactors register their bound UDS sockpaths in a
per-process registry inside `tractor.ipc._uds` at
`start_listener()` time.
- The subactor reports its bound sockpath(s) back to the
parent over IPC immediately post-bind (extension to
`SpawnSpec` reply / a new handshake msg).
- Parent caches the subactor's authoritative sockpaths.
- `unlink_uds_bind_addrs()` checks the cache FIRST, falls
back to convention-reconstruction if the subactor died
before reporting (which is the SIGKILL case this fn
primarily exists for).
Tracked as future work in #454 (the parent UDS-leak
issue this module addresses); a separate issue may be
filed if/when the registry impl is scoped.
See also #452 — the discovery-client `CLOSE_WAIT` TCP
fd leak. Different bug class but same broader theme of
"fork-spawn unmasked latent cleanup gaps".
'''
from __future__ import annotations
import os
from typing import TYPE_CHECKING
import trio
from tractor.discovery._addr import (
UnwrappedAddress,
wrap_address,
)
from tractor.ipc._uds import UDSAddress
from tractor.log import get_logger
if TYPE_CHECKING:
from tractor.runtime._runtime import Actor
log = get_logger('tractor')
def unlink_uds_bind_addrs(
proc: trio.Process,
*,
bind_addrs: list[UnwrappedAddress] | None = None,
subactor: Actor | None = None,
) -> None:
'''
Best-effort post-mortem cleanup of any UDS sock-files
a hard-killed subactor was bound to.
SIGKILL bypasses Python execution → the subactor's
`_serve_ipc_eps` `finally:` block (which normally calls
`os.unlink(addr.sockpath)`) never runs. Without this
parent-side cleanup, the dead subactor's
`${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock` file
accumulates on the filesystem (see issue #454 + the
autouse `_track_orphaned_uds_per_test` fixture).
Two cleanup paths, in order:
1. **Explicit `bind_addrs`** — when the parent set the
subactor's bind addrs at spawn time, unlink each
UDS-flavored sockpath directly.
2. **Self-assigned reconstruction** — when
`bind_addrs` is empty (the common case: subactor
picked its own random sock via
`UDSAddress.get_random()`), reconstruct the path
from `(subactor.aid.name, proc.pid)` using the
same `<name>@<pid>.sock` convention. We can do this
because the subactor uses its OWN `os.getpid()` at
bind time, which equals `proc.pid` from the
parent's view.
Idempotent: `FileNotFoundError` (graceful exit
already-unlinked, or sock never bound under early-
spawn cancel) is silenced; other `OSError`s log a
warning but never raise. TCP / non-UDS bind addrs are
skipped.
'''
sockpaths: list[str] = []
# path 1: explicit bind_addrs set at spawn time
for unwrapped in (bind_addrs or ()):
try:
addr = wrap_address(unwrapped)
except Exception:
log.exception(
f'Failed to wrap addr for UDS post-kill cleanup '
f'— skipping {unwrapped!r}\n'
)
continue
if isinstance(addr, UDSAddress):
sockpaths.append(str(addr.sockpath))
# path 2: reconstruct from subactor name + proc pid
# for the random-self-assign case (bind_addrs=None)
#
# TODO authoritative tracking — see module docstring.
if (
not sockpaths
and subactor is not None
and proc.pid is not None
):
sockname: str = f'{subactor.aid.name}@{proc.pid}.sock'
sockpath: str = str(
UDSAddress.def_bindspace / sockname
)
sockpaths.append(sockpath)
for sockpath in sockpaths:
try:
os.unlink(sockpath)
log.runtime(
f'Unlinked orphaned UDS sock-file post-SIGKILL\n'
f' |_{proc}\n'
f' |_{sockpath}\n'
)
except FileNotFoundError:
# raced — subactor cleaned up before SIGKILL,
# OR sockfile never bound (early-spawn cancel),
# OR transport wasn't UDS this run.
pass
except OSError as exc:
log.warning(
f'Failed to unlink subactor UDS sock-file '
f'post-SIGKILL\n'
f' |_{proc}\n'
f' |_{sockpath}\n'
f' |_{exc!r}\n'
)

30
tractor/spawn/_spawn.py
View File

@ -39,7 +39,10 @@ from tractor.runtime._state import (
_runtime_vars,
)
from tractor.log import get_logger
from tractor.discovery._addr import UnwrappedAddress
from tractor.discovery._addr import (
UnwrappedAddress,
)
from ._reap import unlink_uds_bind_addrs
from tractor.runtime._portal import Portal
from tractor.runtime._runtime import Actor
from tractor.msg import types as msgtypes
@ -223,6 +226,16 @@ async def hard_kill(
# whilst also hacking on it XD
# terminate_after: int = 99999,
*,
# Subactor's bind addresses + subactor record, used
# for post-SIGKILL UDS sockpath cleanup. Optional for
# legacy callers; new call sites should pass at least
# `subactor` (which lets us reconstruct the sock path
# from `aid.name + proc.pid` when `bind_addrs` is
# empty/self-assigned). See `._reap.unlink_uds_bind_addrs()`.
bind_addrs: list[UnwrappedAddress] | None = None,
subactor: Actor | None = None,
) -> None:
'''
Un-gracefully terminate an OS level `trio.Process` after timeout.
@ -310,6 +323,21 @@ async def hard_kill(
)
proc.kill()
# Post-mortem UDS sockpath cleanup. SIGKILL bypassed
# the subactor's normal `os.unlink(addr.sockpath)` in
# `_serve_ipc_eps`'s `finally:`; the parent has the
# bind addrs (or can reconstruct from name + pid) so
# we do it here. Runs UNCONDITIONALLY (graceful-exit
# case is a no-op via `FileNotFoundError` skip in the
# helper) so the cleanup also covers the "cancelled
# during spawn" path where the subactor never reached
# its IPC server finally block.
unlink_uds_bind_addrs(
proc,
bind_addrs=bind_addrs,
subactor=subactor,
)
async def soft_kill(
proc: ProcessType,

18
tractor/spawn/_trio.py
View File

@ -282,7 +282,23 @@ async def trio_proc(
if proc.poll() is None:
log.cancel(f"Attempting to hard kill {proc}")
await hard_kill(proc)
await hard_kill(
proc,
# NOTE, pass through so post-SIGKILL we
# can `os.unlink()` the subactor's
# orphaned UDS sock-file(s) — the
# subactor's own
# `_serve_ipc_eps`-`finally:` cleanup
# never runs under SIGKILL. `subactor`
# lets the helper reconstruct the
# sock path via `aid.name + proc.pid`
# when `bind_addrs` is the common
# self-assigned-random case
# (bind_addrs=None at spawn). See
# `_unlink_uds_bind_addrs()` in `_spawn`.
bind_addrs=bind_addrs,
subactor=subactor,
)
log.debug(f"Joined {proc}")
else:

347
tractor/to_asyncio.py
View File

@ -27,6 +27,7 @@ from contextlib import asynccontextmanager as acm
from dataclasses import dataclass
import inspect
import platform
import sys
import traceback
from typing import (
Any,
@ -810,6 +811,151 @@ def _run_asyncio_task(
return chan
def maybe_signal_aio_task(
aio_task: asyncio.Task,
exc: BaseException,
*,
cause: BaseException|None = None,
pre_captured_fut: asyncio.Future|None = None,
allow_cancel_fallback: bool = False,
) -> tuple[bool, str]:
'''
Best-effort delivery of `exc` to a still-running `aio_task`
via its `_fut_waiter` (the `asyncio.Future` the task is
currently `await`-ing on).
Returns `(delivered, report)` where `delivered=True` iff
either,
- `fut.set_exception(exc)` was successfully called on an
un-`done()` `_fut_waiter`, OR
- the cancel-fallback path fired (only when the caller
opted-in via `allow_cancel_fallback=True`).
Why `_fut_waiter.set_exception(exc)` and NOT
`aio_task.set_exception(exc)`:
On py3.13+ `asyncio.Task.set_exception()` ALWAYS raises
`RuntimeError("Task does not support set_exception
operation")` — so calling it as a relay mechanism is dead
code. The `_fut_waiter` is a plain `asyncio.Future` and
its `set_exception()` works on all Python versions; the
task's `_wakeup` callback then propagates the exc into
the coro on its next tick.
Why we PREFER NOT to call `aio_task.cancel()`:
`Task.cancel()` injects a `CancelledError` that races
any in-flight exception already queued on `_fut_waiter`
(e.g. via a prior `set_exception()` from a sibling
teardown path). The race can mask BOTH the original
trio-side error and any asyncio-side error the task was
mid-raising. See the
`test_trio_closes_early_and_channel_exits` hang TODO
around the `translate_aio_errors` finally for the
historical artifact.
However a caller may have NO OTHER way to terminate the
task — when `_fut_waiter is None` AND the task is busy
looping / runnable, neither `set_exception` nor a chan
close can poke it. In that narrow case `cancel()` is the
only available termination signal; opt-in via
`allow_cancel_fallback=True`. The fallback NEVER runs
when `_fut_waiter` carries an in-flight exc (the
`fut.done()` branch); only when there's truly no
`_fut_waiter` ref to poke.
Pre-checkpoint capture:
`asyncio.Task._wakeup` clears `_fut_waiter = None` as
part of the wakeup sequence. If the caller crosses a
trio checkpoint between fut-capture and this call,
re-reading `aio_task._fut_waiter` will see `None` even
though the exc is still in flight on the (now-`done()`)
original fut. Pass `pre_captured_fut` to use the
already-captured reference.
Causal chaining via `cause`:
Pass the underlying trio-side exc (the *reason* we're
poking the aio side) via `cause` and the helper sets
`exc.__cause__ = cause`. The chain travels with `exc`
through `_fut_waiter.set_exception()` → `Task._wakeup`
→ coro raise → `wait_on_coro_final_result`'s except →
`signal_trio_when_done`'s `task.result()`-`raise
aio_err`. The final traceback then renders as
"<trio-side exc> -> (direct cause of) -> <relay exc>"
instead of an opaque, root-cause-detached relay.
See the "cross-loop cause-chain matrix" comment in
`translate_aio_errors()`'s final-raise block for how this
`cause` interacts with every `raise X [from Y]` exit path
(esp. the relay-echo guard which prevents a cause CYCLE).
'''
if cause is not None and exc.__cause__ is None:
exc.__cause__ = cause
if aio_task.done():
return False, (
f'aio-task already done; nothing to signal\n'
f' |_{aio_task!r}\n'
)
fut: asyncio.Future|None = (
pre_captured_fut
if pre_captured_fut is not None
else aio_task._fut_waiter
)
if fut and not fut.done():
fut.set_exception(exc)
return True, (
f'signalled aio-task via `_fut_waiter.set_exception()`\n'
f'exc: {exc!r}\n'
f' |_{aio_task!r}\n'
)
if fut and fut.done():
# NEVER cancel here even when `allow_cancel_fallback=True`
# — the in-flight exc on `fut` will terminate the task
# on its next tick; injecting `CancelledError` on top
# would race and mask the real exc.
return False, (
f'`_fut_waiter` already signalled with,\n'
f' |_{fut.exception()!r}\n'
f'aio-task will exit on next tick via the in-flight exc;\n'
f'SKIPPING re-signal (would race in-flight delivery).\n'
f' |_{aio_task!r}\n'
)
# fut is None — task is runnable (sitting in asyncio's
# ready queue), not parked on a future we can poke.
if allow_cancel_fallback:
cancel_msg: str = (
f'\n'
f'MANUALLY Cancelling `asyncio`-task: '
f'{aio_task.get_name()}!\n\n'
f'**THIS CAN SILENTLY SUPPRESS ERRORS FYI\n\n'
)
aio_task.cancel(msg=cancel_msg)
return True, (
f'aio-task has no `_fut_waiter`; FALLBACK cancel issued\n'
f'(caller opted-in via `allow_cancel_fallback=True`).\n'
f'{cancel_msg}'
f' |_{aio_task!r}\n'
)
return False, (
f'aio-task has no `_fut_waiter`; cannot signal without\n'
f'`aio_task.cancel()` which can mask errors.\n'
f'LEAVING AS-IS (caller did NOT opt-in to cancel fallback);\n'
f'task should exit via chan close / aio-loop teardown\n'
f'already in flight.\n'
f' |_{aio_task!r}\n'
)
@acm
async def translate_aio_errors(
chan: LinkedTaskChannel,
@ -985,38 +1131,25 @@ async def translate_aio_errors(
# if isinstance(chan._aio_err, AsyncioTaskExited):
# await tractor.pause(shield=True)
# if aio side is still active cancel it due to the trio-side
# error!
# if aio side is still active relay the trio-side error
# to it via `_fut_waiter.set_exception()`.
# ?TODO, mk `AsyncioCancelled[typeof(trio_err)]` embed the
# current exc?
if (
# not aio_task.cancelled()
# and
not aio_task.done() # TODO? only need this one?
# XXX LOL, so if it's not set it's an error !?
# yet another good jerb by `ascyncio`..
# and
# not aio_task.exception()
):
aio_taskc = TrioCancelled(
f'The `trio`-side task crashed!\n'
f'{trio_err}'
)
# ??TODO? move this into the func that tries to use
# `Task._fut_waiter: Future` instead??
#
# aio_task.set_exception(aio_taskc)
# wait_on_aio_task = False
try:
aio_task.set_exception(aio_taskc)
except (
asyncio.InvalidStateError,
RuntimeError,
# ^XXX, uhh bc apparently we can't use `.set_exception()`
# any more XD .. ??
):
delivered, report = maybe_signal_aio_task(
aio_task,
aio_taskc,
# so the relay carries a "<trio_err> -> caused ->
# TrioCancelled" chain when it eventually re-raises
# on the aio side.
cause=trio_err,
)
if not delivered:
wait_on_aio_task = False
log.cancel(report)
finally:
# record wtv `trio`-side error transpired
@ -1099,27 +1232,22 @@ async def translate_aio_errors(
if _py_313:
chan._to_aio.shutdown()
# XXX CRITICAL ordering: capture `_fut_waiter`
# BEFORE the checkpoint. `asyncio.Task._wakeup`
# clears `_fut_waiter = None` as part of wakeup,
# so re-reading after the checkpoint loses the
# ref even though the exc is still in-flight on
# the (now-`done()`) original fut. The helper
# uses `pre_captured_fut` to recover that.
pre_cp_fut: asyncio.Future|None = aio_task._fut_waiter
# pump this event-loop (well `Runner` but ya)
#
# TODO? is this actually needed?
# -[ ] theory is this let's the aio side error on
# next tick and then we sync task states from
# here onward?
# so the aio side can error on next tick and we
# sync task states from here onward.
await trio.lowlevel.checkpoint()
# TODO? factor the next 2 branches into a func like
# `try_terminate_aio_task()` and use it for the taskc
# case above as well?
fut: asyncio.Future|None = aio_task._fut_waiter
if (
fut
and
not fut.done()
):
# await tractor.pause()
if graceful_trio_exit:
fut.set_exception(
TrioTaskExited(
relay_exc = TrioTaskExited(
f'the `trio.Task` gracefully exited but '
f'its `asyncio` peer is not done?\n'
f')>\n'
@ -1128,31 +1256,30 @@ async def translate_aio_errors(
f'>>\n'
f' |_{aio_task!r}\n'
)
)
# TODO? should this need to exist given the equiv
# `TrioCancelled` equivalent in the be handler
# above??
else:
fut.set_exception(
TrioTaskExited(
relay_exc = TrioTaskExited(
f'The `trio`-side task crashed!\n'
f'{trio_err}'
)
delivered, signal_report = maybe_signal_aio_task(
aio_task,
relay_exc,
pre_captured_fut=pre_cp_fut,
# XXX historically this branch called
# `aio_task.cancel()` when `_fut_waiter`
# was None — required to actually terminate
# aio tasks that aren't parked on a poke-able
# future (e.g. the `aio_echo_server` loop in
# `test_echoserver_detailed_mechanics`). Opt
# into the fallback so we don't regress.
allow_cancel_fallback=True,
# carry the trio-side exc (if any) as the
# cause so the aio-side relay shows the
# real root-cause chain when re-raised.
cause=trio_err,
)
else:
aio_taskc_warn: str = (
f'\n'
f'MANUALLY Cancelling `asyncio`-task: {aio_task.get_name()}!\n\n'
f'**THIS CAN SILENTLY SUPPRESS ERRORS FYI\n\n'
)
# await tractor.pause()
report += aio_taskc_warn
# TODO XXX, figure out the case where calling this makes the
# `test_infected_asyncio.py::test_trio_closes_early_and_channel_exits`
# hang and then don't call it in that case!
#
aio_task.cancel(msg=aio_taskc_warn)
report += signal_report
log.warning(report)
@ -1161,10 +1288,11 @@ async def translate_aio_errors(
# `channel._aio_err/._trio_to_raise`) BEFORE calling
# `maybe_raise_aio_side_err()` below!
#
# XXX WARNING NOTE
# the `task.set_exception(aio_taskc)` call above MUST NOT
# EXCEPT or this WILL HANG!! SO, if you get a hang maybe step
# through and figure out why it erroed out up there!
# NOTE, `wait_on_aio_task` may have been flipped to `False`
# by `maybe_signal_aio_task()` above when delivery
# failed (e.g. `_fut_waiter is None`) — in that case we
# skip the wait since the aio task won't process our
# relay exc and `_aio_task_complete` may never set.
#
if wait_on_aio_task:
await chan._aio_task_complete.wait()
@ -1181,6 +1309,47 @@ async def translate_aio_errors(
- `run_task()`
'''
# ===== cross-loop cause-chain matrix =====
# How `(trio_err, aio_err, trio_to_raise)` resolve into ONE
# terminal `raise X [from Y]` (or an early `return`).
#
# legend (the possible `X` / `Y` operands):
# - trio_err : `chan._trio_err`, the trio-side exc.
# - aio_err : `chan._aio_err`, the aio-side exc.
# - trio_to_raise : `chan._trio_to_raise`, a tractor-chosen
# relay exc (`AsyncioCancelled`/`AsyncioTaskExited`).
# - raise_from : `trio_err if (aio_err is trio_to_raise)
# else aio_err` (the chosen `__cause__`).
# - relay-echo : an `aio_err` that is one of OUR OWN
# `TrioTaskExited|TrioCancelled` signals,
# synth'd + delivered to the aio-side by
# `maybe_signal_aio_task()`; its `__cause__`
# is ALREADY `trio_err`.
# - "(bare)" : raised with NO explicit `from` clause.
#
# this block (final-raise in `translate_aio_errors`):
# condition => raises from
# ----------------------------------- ------------- -----------
# not suppress_graceful_exits => trio_to_raise raise_from
# AsyncioTaskExited + trio Cancelled/None => return (aio-exit ignored)
# AsyncioTaskExited + trio EoC => trio_err (bare)
# AsyncioCancelled + trio Cancelled => return (co-cancel ignored)
# trio_to_raise match catch-all => trio_to_raise raise_from
# aio_err is relay-echo ◄── the GUARD => trio_err (bare)
# aio_err independent (real aio fail) => trio_err aio_err
# aio_err independent, no trio_err => aio_err (bare)
# only trio_err => trio_err (bare)
#
# sibling block (`signal_trio_when_done()`, the aio done-cb):
# AsyncioTaskExited relay-out => trio_to_raise aio_err
# plain aio_err re-raise => aio_err (__cause__ preset)
#
# INVARIANT: a relay-echo must NEVER become `trio_err.__cause__`
# (it's ALREADY caused-BY `trio_err`) → doing so would CYCLE
# (`trio_err ◄─► relay`). So the guard raises the root
# `trio_err` bare; the relay still keeps its own correct
# "relay ◄ trio_err" chain for any aio-side inspection.
# ===== / cross-loop cause-chain matrix =====
aio_err: BaseException|None = chan._aio_err
trio_to_raise: (
AsyncioCancelled|
@ -1237,6 +1406,32 @@ async def translate_aio_errors(
and
type(aio_err) is not AsyncioCancelled
):
# XXX, if `aio_err` is one of OUR OWN relay-signals
# (`TrioTaskExited`/`TrioCancelled`) that we delivered
# to the aio-side via `maybe_signal_aio_task()`, AND
# its `__cause__` already points back at `trio_err`,
# then it's just a derivative ECHO of the trio-side
# error, NOT an independent asyncio failure.
#
# Raising `trio_err from aio_err` here would invert
# (and cyclically tangle) the cause chain since the
# relay was itself caused-by `trio_err`:
#
# trio_err.__cause__ = aio_err (from `raise .. from`)
# aio_err.__cause__ = trio_err (set in `maybe_signal_aio_task`)
#
# So raise the REAL root `trio_err` alone; the relay's
# own `__cause__` chain still correctly reads
# "TrioTaskExited <- trio_err" for aio-side inspection.
if (
trio_err is not None
and
isinstance(aio_err, (TrioTaskExited, TrioCancelled))
and
aio_err.__cause__ is trio_err
):
raise trio_err
# always raise from any captured asyncio error
if trio_err:
raise trio_err from aio_err
@ -1353,19 +1548,22 @@ async def open_channel_from(
# a `Return`-msg for IPC ctxs)
aio_task: asyncio.Task = chan._aio_task
if not aio_task.done():
fut: asyncio.Future|None = aio_task._fut_waiter
if fut:
fut.set_exception(
# capture the in-flight trio-side exc (if any)
# so the relay's `__cause__` chain shows the
# real root cause when the aio task re-raises.
# `sys.exc_info()[1]` is non-`None` only when
# the `try` body raised (graceful exit -> None).
trio_exc: BaseException|None = sys.exc_info()[1]
_, report = maybe_signal_aio_task(
aio_task,
TrioTaskExited(
f'but the child `asyncio` task is still running?\n'
f'>>\n'
f' |_{aio_task!r}\n'
),
cause=trio_exc,
)
)
else:
# XXX SHOULD NEVER HAPPEN!
log.error("SHOULD NEVER GET HERE !?!?")
await tractor.pause(shield=True)
log.cancel(report)
else:
chan._to_trio.close()
@ -1602,6 +1800,7 @@ def run_as_asyncio_guest(
fute_err: BaseException|None = None
try:
out: Outcome = await asyncio.shield(trio_done_fute)
# out: Outcome = await trio_done_fute
# ^TODO still don't really understand why the `.shield()`
# is required ... ??
# https://docs.python.org/3/library/asyncio-task.html#asyncio.shield

95
tractor/trionics/patches/README.md 100644
View File

@ -0,0 +1,95 @@
# `tractor.trionics.patches`
Defensive monkey-patches for bugs in `trio` itself.
## What goes here
- Bugs in upstream `trio` that we've encountered while
running `tractor` and need to work around until
upstream releases a fix.
- Each patch fixes EXACTLY one trio internal — no
multi-bug omnibus patches.
## What does NOT go here
- Bugs in `tractor`'s own code (those get fixed
in-tree, in the offending tractor module).
- Bugs in `asyncio`, `pytest`, the stdlib, etc. (file
separate `tractor.<lib>.patches` subpkgs as
needed).
- Workarounds for behavior we *disagree* with but that
isn't a bug per se. If trio's API does what it says
on the tin, we don't override it here.
## Per-patch contract
Every `_<topic>.py` module in this directory MUST
expose:
- **`apply() -> bool`** — apply the patch. Idempotent
(safe to call multiple times). Version-gated — must
consult `is_needed()` and skip when False. Returns
`True` if patched this call, `False` if skipped.
- **`is_needed() -> bool`** — does upstream still need
patching? Today most patches return `True`
unconditionally, but as upstream releases land each
should gate on `Version(trio.__version__) <
Version('X.Y.Z')`. When the gated version is
released, the patch can be DELETED entirely.
- **`repro() -> None`** — minimal demonstration of the
bug. Used by the regression test suite to assert (a)
the upstream bug still exists, (b) our patch fixes
it. Should be tight enough that calling it post-
`apply()` returns cleanly within a few hundred
milliseconds — tests wrap it with a wall-clock cap.
Each module's docstring MUST contain:
- **Problem**: what trio does wrong + the trigger
conditions (e.g. "fork-spawn backend, peer-closed
socketpair, etc.")
- **Fix**: the one-line (ideally) patch
- **Repro**: the standalone snippet `repro()`
implements
- **Upstream**: link to filed issue/PR (or
`TODO: file`)
- **REMOVE WHEN**: `trio>=X.Y.Z` ships the upstream
fix
## Adding a patch
1. Create `_<topic>.py` with the `apply` /
`is_needed` / `repro` API.
2. Register it in `__init__.py::_PATCHES`.
3. Add a regression test in
`tests/trionics/test_patches.py` that uses
`repro()` to assert pre/post-patch behavior with a
wall-clock cap.
4. File the upstream issue/PR. Add the link to your
module's `Upstream:` and `# REMOVE WHEN:` lines.
## Removing a patch (when upstream releases the fix)
1. Confirm the upstream-fixed `trio` version is the
minimum we depend on, OR keep the version-gate in
`is_needed()` if we still support older trio.
2. If we've fully bumped past the broken versions:
- Delete `_<topic>.py`
- Remove the entry from `__init__.py::_PATCHES`
- Delete the corresponding test in
`tests/trionics/test_patches.py`
- Bump the conc-anal doc with a "FIXED" header
## Calling
```python
from tractor.trionics.patches import apply_all
apply_all()
```
Currently invoked from `tractor._child._actor_child_main`
before `_trio_main` so every spawned subactor gets
patched. The root actor's entry could opt in too if a
patch turns out to bite the root (none do today).

84
tractor/trionics/patches/__init__.py 100644
View File

@ -0,0 +1,84 @@
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
Defensive monkey-patches for `trio` internals.
Every patch in this package fixes a bug in `trio` itself
that we've encountered while running `tractor` — usually
a fork-survival edge case that upstream `trio` hasn't
filed/fixed yet. Each patch is:
- **idempotent** — safe to call multiple times
- **version-gated** — checks `trio.__version__` and skips
itself if upstream has shipped the fix
- **scoped** — only modifies the specific trio internal
it's targeting; no broad side effects
- **removable** — every patch carries a `# REMOVE WHEN:`
marker in its docstring pointing at the upstream PR
whose release allows us to drop it
Add a new patch by:
1. Create `tractor/trionics/patches/_<topic>.py` exposing
the `apply()` / `is_needed()` / `repro()` API
contract.
2. Import it in this `__init__.py` and add an entry to
`_PATCHES`.
3. Document upstream-fix-tracking in the module
docstring's `# REMOVE WHEN:` line.
4. Add a regression test in
`tests/trionics/test_patches.py` that uses the
patch's `repro()` to assert the bug exists + the
patch fixes it.
Calling `apply_all()` from a tractor entry point (e.g.
`tractor._child._actor_child_main`) applies every
registered patch + returns `{patch_name: applied?}` so
callers can log/assert as needed.
'''
from typing import Callable
from . import _wakeup_socketpair
_PATCHES: list[tuple[str, Callable[[], bool]]] = [
(
'trio_wakeup_socketpair_drain_eof',
_wakeup_socketpair.apply,
),
]
def apply_all() -> dict[str, bool]:
'''
Apply every registered patch. Idempotent — calling
twice is fine, second call's dict will be all
`False`.
Returns `{patch_name: applied?}`:
- `True` — patch was applied THIS call (inaugural
apply, or first-call-since-process-start).
- `False` — skipped (already applied OR upstream fix
detected via `is_needed() == False`).
'''
results: dict[str, bool] = {}
for name, applier in _PATCHES:
results[name] = applier()
return results

171
tractor/trionics/patches/_wakeup_socketpair.py 100644
View File

@ -0,0 +1,171 @@
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
'''
Patch `trio._core._wakeup_socketpair.WakeupSocketpair.drain()`
to break on peer-closed EOF.
Problem
-------
`drain()` loops on `self.wakeup_sock.recv(2**16)` and
exits ONLY on `BlockingIOError` (buffer-empty on a
non-blocking socket), NEVER on `recv() == b''`
(peer-closed FIN). When the socketpair's write-end
has been closed, `recv` returns 0 bytes each call →
infinite C-level tight loop → 100% CPU, no Python
checkpoints, no signal delivery, no progress.
Most reliably triggered under fork-spawn backends —
`os.fork()` + `_close_inherited_fds()` can leave a
`WakeupSocketpair` instance whose `write_sock` was
closed in the child (or whose peer-end is held by a
process that has since exited).
Repro
-----
```python
from trio._core._wakeup_socketpair import WakeupSocketpair
ws = WakeupSocketpair()
ws.write_sock.close()
ws.drain() # spins forever pre-patch
```
Fix
---
One line: break the drain loop on `b''` EOF
in addition to the existing `BlockingIOError` exit.
```python
def _safe_drain(self) -> None:
try:
while True:
data = self.wakeup_sock.recv(2**16)
if not data: # ← peer-closed; nothing more to drain
return
except BlockingIOError:
pass
```
Upstream
--------
TODO: file at `python-trio/trio` — the standalone
`repro()` below + this docstring is the issue body's
evidence section.
REMOVE WHEN: trio>=`<TBD>` ships the EOF-break in
`_wakeup_socketpair.WakeupSocketpair.drain()`.
See also
--------
- `ai/conc-anal/trio_wakeup_socketpair_busy_loop_under_fork_issue.md`
- `ai/conc-anal/infected_asyncio_under_main_thread_forkserver_hang_issue.md`
— sibling-bug analysis fixed by the same patch.
'''
from __future__ import annotations
# Module-local sentinel — set True by `apply()` after the
# first successful patch. Idempotency guard.
_APPLIED: bool = False
def is_needed() -> bool:
'''
True iff upstream `trio` is the broken version that
needs our patch.
Today: always True since no released `trio` has the
fix. When upstream lands it, gate on:
```python
from packaging.version import Version
import trio
return Version(trio.__version__) < Version('<TBD>')
```
'''
# TODO version-gate once upstream lands the fix.
return True
def repro() -> None:
'''
Minimal hang demonstrator + regression test target.
Returns CLEANLY when `apply()` has been called
earlier in this process (the patched
`_safe_drain` breaks on EOF). Spins forever
UNPATCHED — caller should wrap with a wall-clock
cap (e.g. `signal.alarm(N)` or `trio.fail_after`)
to avoid hanging the test runner if regressing.
Used by `tests/trionics/test_patches.py` to assert
both:
1. The bug exists upstream (sanity check the
repro is real).
2. Our patch fixes it (post-`apply()` returns
cleanly).
'''
from trio._core._wakeup_socketpair import (
WakeupSocketpair,
)
ws = WakeupSocketpair()
ws.write_sock.close()
ws.drain() # ← targeted operation
def apply() -> bool:
'''
Apply the EOF-break patch to
`WakeupSocketpair.drain`. Idempotent + version-
gated.
Returns:
- `True` if patched THIS call (inaugural apply).
- `False` if skipped (already applied this process,
OR `is_needed() == False` because upstream fixed
it).
'''
global _APPLIED
if _APPLIED or not is_needed():
return False
from trio._core._wakeup_socketpair import (
WakeupSocketpair as _WSP,
)
def _safe_drain(self) -> None:
try:
while True:
data = self.wakeup_sock.recv(2**16)
# XXX patch — break on EOF instead of
# spinning. Upstream trio's `drain()`
# only handles the `BlockingIOError`
# (buffer-empty) case; missed the
# peer-closed (`recv == b''`) case.
if not data:
return
except BlockingIOError:
pass
_WSP.drain = _safe_drain
_APPLIED = True
return True

70
uv.lock
View File

@ -573,6 +573,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/8b/5a/ba30a81239b909821b3153e303e7def45178bf353da4f72380e6c5e8793b/pytest-9.1.0-py3-none-any.whl", hash = "sha256:8ebb0e7888bdf2bdfc602ec51f8f62d50200af37356c74e503c79a94f5c81f32", size = 386453, upload-time = "2026-06-13T18:52:44.045Z" },
]
[[package]]
name = "pytest-timeout"
version = "2.4.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pytest" },
]
sdist = { url = "https://files.pythonhosted.org/packages/ac/82/4c9ecabab13363e72d880f2fb504c5f750433b2b6f16e99f4ec21ada284c/pytest_timeout-2.4.0.tar.gz", hash = "sha256:7e68e90b01f9eff71332b25001f85c75495fc4e3a836701876183c4bcfd0540a", size = 17973, upload-time = "2025-05-05T19:44:34.99Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/b6/3127540ecdf1464a00e5a01ee60a1b09175f6913f0644ac748494d9c4b21/pytest_timeout-2.4.0-py3-none-any.whl", hash = "sha256:c42667e5cdadb151aeb5b26d114aff6bdf5a907f176a007a30b940d3d865b5c2", size = 14382, upload-time = "2025-05-05T19:44:33.502Z" },
]
[[package]]
name = "python-baseconv"
version = "1.2.2"
@ -605,6 +617,54 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9e/6a/40fee331a52339926a92e17ae748827270b288a35ef4a15c9c8f2ec54715/ruff-0.14.14-py3-none-win_arm64.whl", hash = "sha256:56e6981a98b13a32236a72a8da421d7839221fa308b223b9283312312e5ac76c", size = 10920448, upload-time = "2026-01-22T22:30:15.417Z" },
]
[[package]]
name = "setproctitle"
version = "1.3.7"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/8d/48/49393a96a2eef1ab418b17475fb92b8fcfad83d099e678751b05472e69de/setproctitle-1.3.7.tar.gz", hash = "sha256:bc2bc917691c1537d5b9bca1468437176809c7e11e5694ca79a9ca12345dcb9e", size = 27002, upload-time = "2025-09-05T12:51:25.278Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/5d/2f/fcedcade3b307a391b6e17c774c6261a7166aed641aee00ed2aad96c63ce/setproctitle-1.3.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c3736b2a423146b5e62230502e47e08e68282ff3b69bcfe08a322bee73407922", size = 18047, upload-time = "2025-09-05T12:49:50.271Z" },
{ url = "https://files.pythonhosted.org/packages/23/ae/afc141ca9631350d0a80b8f287aac79a76f26b6af28fd8bf92dae70dc2c5/setproctitle-1.3.7-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3384e682b158d569e85a51cfbde2afd1ab57ecf93ea6651fe198d0ba451196ee", size = 13073, upload-time = "2025-09-05T12:49:51.46Z" },
{ url = "https://files.pythonhosted.org/packages/87/ed/0a4f00315bc02510395b95eec3d4aa77c07192ee79f0baae77ea7b9603d8/setproctitle-1.3.7-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0564a936ea687cd24dffcea35903e2a20962aa6ac20e61dd3a207652401492dd", size = 33284, upload-time = "2025-09-05T12:49:52.741Z" },
{ url = "https://files.pythonhosted.org/packages/fc/e4/adf3c4c0a2173cb7920dc9df710bcc67e9bcdbf377e243b7a962dc31a51a/setproctitle-1.3.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5d1cb3f81531f0eb40e13246b679a1bdb58762b170303463cb06ecc296f26d0", size = 34104, upload-time = "2025-09-05T12:49:54.416Z" },
{ url = "https://files.pythonhosted.org/packages/52/4f/6daf66394152756664257180439d37047aa9a1cfaa5e4f5ed35e93d1dc06/setproctitle-1.3.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a7d159e7345f343b44330cbba9194169b8590cb13dae940da47aa36a72aa9929", size = 35982, upload-time = "2025-09-05T12:49:56.295Z" },
{ url = "https://files.pythonhosted.org/packages/1b/62/f2c0595403cf915db031f346b0e3b2c0096050e90e0be658a64f44f4278a/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0b5074649797fd07c72ca1f6bff0406f4a42e1194faac03ecaab765ce605866f", size = 33150, upload-time = "2025-09-05T12:49:58.025Z" },
{ url = "https://files.pythonhosted.org/packages/a0/29/10dd41cde849fb2f9b626c846b7ea30c99c81a18a5037a45cc4ba33c19a7/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:61e96febced3f61b766115381d97a21a6265a0f29188a791f6df7ed777aef698", size = 34463, upload-time = "2025-09-05T12:49:59.424Z" },
{ url = "https://files.pythonhosted.org/packages/71/3c/cedd8eccfaf15fb73a2c20525b68c9477518917c9437737fa0fda91e378f/setproctitle-1.3.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:047138279f9463f06b858e579cc79580fbf7a04554d24e6bddf8fe5dddbe3d4c", size = 32848, upload-time = "2025-09-05T12:50:01.107Z" },
{ url = "https://files.pythonhosted.org/packages/d1/3e/0a0e27d1c9926fecccfd1f91796c244416c70bf6bca448d988638faea81d/setproctitle-1.3.7-cp313-cp313-win32.whl", hash = "sha256:7f47accafac7fe6535ba8ba9efd59df9d84a6214565108d0ebb1199119c9cbbd", size = 12544, upload-time = "2025-09-05T12:50:15.81Z" },
{ url = "https://files.pythonhosted.org/packages/36/1b/6bf4cb7acbbd5c846ede1c3f4d6b4ee52744d402e43546826da065ff2ab7/setproctitle-1.3.7-cp313-cp313-win_amd64.whl", hash = "sha256:fe5ca35aeec6dc50cabab9bf2d12fbc9067eede7ff4fe92b8f5b99d92e21263f", size = 13235, upload-time = "2025-09-05T12:50:16.89Z" },
{ url = "https://files.pythonhosted.org/packages/e6/a4/d588d3497d4714750e3eaf269e9e8985449203d82b16b933c39bd3fc52a1/setproctitle-1.3.7-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:10e92915c4b3086b1586933a36faf4f92f903c5554f3c34102d18c7d3f5378e9", size = 18058, upload-time = "2025-09-05T12:50:02.501Z" },
{ url = "https://files.pythonhosted.org/packages/05/77/7637f7682322a7244e07c373881c7e982567e2cb1dd2f31bd31481e45500/setproctitle-1.3.7-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:de879e9c2eab637f34b1a14c4da1e030c12658cdc69ee1b3e5be81b380163ce5", size = 13072, upload-time = "2025-09-05T12:50:03.601Z" },
{ url = "https://files.pythonhosted.org/packages/52/09/f366eca0973cfbac1470068d1313fa3fe3de4a594683385204ec7f1c4101/setproctitle-1.3.7-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:c18246d88e227a5b16248687514f95642505000442165f4b7db354d39d0e4c29", size = 34490, upload-time = "2025-09-05T12:50:04.948Z" },
{ url = "https://files.pythonhosted.org/packages/71/36/611fc2ed149fdea17c3677e1d0df30d8186eef9562acc248682b91312706/setproctitle-1.3.7-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7081f193dab22df2c36f9fc6d113f3793f83c27891af8fe30c64d89d9a37e152", size = 35267, upload-time = "2025-09-05T12:50:06.015Z" },
{ url = "https://files.pythonhosted.org/packages/88/a4/64e77d0671446bd5a5554387b69e1efd915274686844bea733714c828813/setproctitle-1.3.7-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9cc9b901ce129350637426a89cfd650066a4adc6899e47822e2478a74023ff7c", size = 37376, upload-time = "2025-09-05T12:50:07.484Z" },
{ url = "https://files.pythonhosted.org/packages/89/bc/ad9c664fe524fb4a4b2d3663661a5c63453ce851736171e454fa2cdec35c/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:80e177eff2d1ec172188d0d7fd9694f8e43d3aab76a6f5f929bee7bf7894e98b", size = 33963, upload-time = "2025-09-05T12:50:09.056Z" },
{ url = "https://files.pythonhosted.org/packages/ab/01/a36de7caf2d90c4c28678da1466b47495cbbad43badb4e982d8db8167ed4/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:23e520776c445478a67ee71b2a3c1ffdafbe1f9f677239e03d7e2cc635954e18", size = 35550, upload-time = "2025-09-05T12:50:10.791Z" },
{ url = "https://files.pythonhosted.org/packages/dd/68/17e8aea0ed5ebc17fbf03ed2562bfab277c280e3625850c38d92a7b5fcd9/setproctitle-1.3.7-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5fa1953126a3b9bd47049d58c51b9dac72e78ed120459bd3aceb1bacee72357c", size = 33727, upload-time = "2025-09-05T12:50:12.032Z" },
{ url = "https://files.pythonhosted.org/packages/b2/33/90a3bf43fe3a2242b4618aa799c672270250b5780667898f30663fd94993/setproctitle-1.3.7-cp313-cp313t-win32.whl", hash = "sha256:4a5e212bf438a4dbeece763f4962ad472c6008ff6702e230b4f16a037e2f6f29", size = 12549, upload-time = "2025-09-05T12:50:13.074Z" },
{ url = "https://files.pythonhosted.org/packages/0b/0e/50d1f07f3032e1f23d814ad6462bc0a138f369967c72494286b8a5228e40/setproctitle-1.3.7-cp313-cp313t-win_amd64.whl", hash = "sha256:cf2727b733e90b4f874bac53e3092aa0413fe1ea6d4f153f01207e6ce65034d9", size = 13243, upload-time = "2025-09-05T12:50:14.146Z" },
{ url = "https://files.pythonhosted.org/packages/89/c7/43ac3a98414f91d1b86a276bc2f799ad0b4b010e08497a95750d5bc42803/setproctitle-1.3.7-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:80c36c6a87ff72eabf621d0c79b66f3bdd0ecc79e873c1e9f0651ee8bf215c63", size = 18052, upload-time = "2025-09-05T12:50:17.928Z" },
{ url = "https://files.pythonhosted.org/packages/cd/2c/dc258600a25e1a1f04948073826bebc55e18dbd99dc65a576277a82146fa/setproctitle-1.3.7-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b53602371a52b91c80aaf578b5ada29d311d12b8a69c0c17fbc35b76a1fd4f2e", size = 13071, upload-time = "2025-09-05T12:50:19.061Z" },
{ url = "https://files.pythonhosted.org/packages/ab/26/8e3bb082992f19823d831f3d62a89409deb6092e72fc6940962983ffc94f/setproctitle-1.3.7-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fcb966a6c57cf07cc9448321a08f3be6b11b7635be502669bc1d8745115d7e7f", size = 33180, upload-time = "2025-09-05T12:50:20.395Z" },
{ url = "https://files.pythonhosted.org/packages/f1/af/ae692a20276d1159dd0cf77b0bcf92cbb954b965655eb4a69672099bb214/setproctitle-1.3.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:46178672599b940368d769474fe13ecef1b587d58bb438ea72b9987f74c56ea5", size = 34043, upload-time = "2025-09-05T12:50:22.454Z" },
{ url = "https://files.pythonhosted.org/packages/34/b2/6a092076324dd4dac1a6d38482bedebbff5cf34ef29f58585ec76e47bc9d/setproctitle-1.3.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7f9e9e3ff135cbcc3edd2f4cf29b139f4aca040d931573102742db70ff428c17", size = 35892, upload-time = "2025-09-05T12:50:23.937Z" },
{ url = "https://files.pythonhosted.org/packages/1c/1a/8836b9f28cee32859ac36c3df85aa03e1ff4598d23ea17ca2e96b5845a8f/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:14c7eba8d90c93b0e79c01f0bd92a37b61983c27d6d7d5a3b5defd599113d60e", size = 32898, upload-time = "2025-09-05T12:50:25.617Z" },
{ url = "https://files.pythonhosted.org/packages/ef/22/8fabdc24baf42defb599714799d8445fe3ae987ec425a26ec8e80ea38f8e/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:9e64e98077fb30b6cf98073d6c439cd91deb8ebbf8fc62d9dbf52bd38b0c6ac0", size = 34308, upload-time = "2025-09-05T12:50:26.827Z" },
{ url = "https://files.pythonhosted.org/packages/15/1b/b9bee9de6c8cdcb3b3a6cb0b3e773afdb86bbbc1665a3bfa424a4294fda2/setproctitle-1.3.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b91387cc0f02a00ac95dcd93f066242d3cca10ff9e6153de7ee07069c6f0f7c8", size = 32536, upload-time = "2025-09-05T12:50:28.5Z" },
{ url = "https://files.pythonhosted.org/packages/37/0c/75e5f2685a5e3eda0b39a8b158d6d8895d6daf3ba86dec9e3ba021510272/setproctitle-1.3.7-cp314-cp314-win32.whl", hash = "sha256:52b054a61c99d1b72fba58b7f5486e04b20fefc6961cd76722b424c187f362ed", size = 12731, upload-time = "2025-09-05T12:50:43.955Z" },
{ url = "https://files.pythonhosted.org/packages/d2/ae/acddbce90d1361e1786e1fb421bc25baeb0c22ef244ee5d0176511769ec8/setproctitle-1.3.7-cp314-cp314-win_amd64.whl", hash = "sha256:5818e4080ac04da1851b3ec71e8a0f64e3748bf9849045180566d8b736702416", size = 13464, upload-time = "2025-09-05T12:50:45.057Z" },
{ url = "https://files.pythonhosted.org/packages/01/6d/20886c8ff2e6d85e3cabadab6aab9bb90acaf1a5cfcb04d633f8d61b2626/setproctitle-1.3.7-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:6fc87caf9e323ac426910306c3e5d3205cd9f8dcac06d233fcafe9337f0928a3", size = 18062, upload-time = "2025-09-05T12:50:29.78Z" },
{ url = "https://files.pythonhosted.org/packages/9a/60/26dfc5f198715f1343b95c2f7a1c16ae9ffa45bd89ffd45a60ed258d24ea/setproctitle-1.3.7-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6134c63853d87a4897ba7d5cc0e16abfa687f6c66fc09f262bb70d67718f2309", size = 13075, upload-time = "2025-09-05T12:50:31.604Z" },
{ url = "https://files.pythonhosted.org/packages/21/9c/980b01f50d51345dd513047e3ba9e96468134b9181319093e61db1c47188/setproctitle-1.3.7-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1403d2abfd32790b6369916e2313dffbe87d6b11dca5bbd898981bcde48e7a2b", size = 34744, upload-time = "2025-09-05T12:50:32.777Z" },
{ url = "https://files.pythonhosted.org/packages/86/b4/82cd0c86e6d1c4538e1a7eb908c7517721513b801dff4ba3f98ef816a240/setproctitle-1.3.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e7c5bfe4228ea22373e3025965d1a4116097e555ee3436044f5c954a5e63ac45", size = 35589, upload-time = "2025-09-05T12:50:34.13Z" },
{ url = "https://files.pythonhosted.org/packages/8a/4f/9f6b2a7417fd45673037554021c888b31247f7594ff4bd2239918c5cd6d0/setproctitle-1.3.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:585edf25e54e21a94ccb0fe81ad32b9196b69ebc4fc25f81da81fb8a50cca9e4", size = 37698, upload-time = "2025-09-05T12:50:35.524Z" },
{ url = "https://files.pythonhosted.org/packages/20/92/927b7d4744aac214d149c892cb5fa6dc6f49cfa040cb2b0a844acd63dcaf/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:96c38cdeef9036eb2724c2210e8d0b93224e709af68c435d46a4733a3675fee1", size = 34201, upload-time = "2025-09-05T12:50:36.697Z" },
{ url = "https://files.pythonhosted.org/packages/0a/0c/fd4901db5ba4b9d9013e62f61d9c18d52290497f956745cd3e91b0d80f90/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:45e3ef48350abb49cf937d0a8ba15e42cee1e5ae13ca41a77c66d1abc27a5070", size = 35801, upload-time = "2025-09-05T12:50:38.314Z" },
{ url = "https://files.pythonhosted.org/packages/e7/e3/54b496ac724e60e61cc3447f02690105901ca6d90da0377dffe49ff99fc7/setproctitle-1.3.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1fae595d032b30dab4d659bece20debd202229fce12b55abab978b7f30783d73", size = 33958, upload-time = "2025-09-05T12:50:39.841Z" },
{ url = "https://files.pythonhosted.org/packages/ea/a8/c84bb045ebf8c6fdc7f7532319e86f8380d14bbd3084e6348df56bdfe6fd/setproctitle-1.3.7-cp314-cp314t-win32.whl", hash = "sha256:02432f26f5d1329ab22279ff863c83589894977063f59e6c4b4845804a08f8c2", size = 12745, upload-time = "2025-09-05T12:50:41.377Z" },
{ url = "https://files.pythonhosted.org/packages/08/b6/3a5a4f9952972791a9114ac01dfc123f0df79903577a3e0a7a404a695586/setproctitle-1.3.7-cp314-cp314t-win_amd64.whl", hash = "sha256:cbc388e3d86da1f766d8fc2e12682e446064c01cea9f88a88647cfe7c011de6a", size = 13469, upload-time = "2025-09-05T12:50:42.67Z" },
]
[[package]]
name = "six"
version = "1.17.0"
@ -664,6 +724,7 @@ dependencies = [
{ name = "multiaddr" },
{ name = "pdbp" },
{ name = "platformdirs" },
{ name = "setproctitle" },
{ name = "tricycle" },
{ name = "trio" },
{ name = "wrapt" },
@ -671,11 +732,13 @@ dependencies = [
[package.dev-dependencies]
dev = [
{ name = "greenback", marker = "python_full_version < '3.14'" },
{ name = "pexpect" },
{ name = "prompt-toolkit" },
{ name = "psutil" },
{ name = "pyperclip" },
{ name = "pytest" },
{ name = "pytest-timeout" },
{ name = "stackscope" },
{ name = "typing-extensions" },
{ name = "xonsh" },
@ -704,7 +767,9 @@ sync-pause = [
]
testing = [
{ name = "pexpect" },
{ name = "psutil" },
{ name = "pytest" },
{ name = "pytest-timeout" },
]
[package.metadata]
@ -715,6 +780,7 @@ requires-dist = [
{ name = "multiaddr", specifier = ">=0.2.0" },
{ name = "pdbp", specifier = ">=1.8.2,<2" },
{ name = "platformdirs", specifier = ">=4.4.0" },
{ name = "setproctitle", specifier = ">=1.3,<2" },
{ name = "tricycle", specifier = ">=0.4.1,<0.5" },
{ name = "trio", specifier = ">0.27" },
{ name = "wrapt", specifier = ">=1.16.0,<2" },
@ -722,11 +788,13 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [
{ name = "greenback", marker = "python_full_version == '3.13.*'", specifier = ">=1.2.1,<2" },
{ name = "pexpect", specifier = ">=4.9.0,<5" },
{ name = "prompt-toolkit", specifier = ">=3.0.50" },
{ name = "psutil", specifier = ">=7.0.0" },
{ name = "pyperclip", specifier = ">=1.9.0" },
{ name = "pytest", specifier = ">=9.0.3" },
{ name = "pytest-timeout", specifier = ">=2.3" },
{ name = "stackscope", specifier = ">=0.2.2,<0.3" },
{ name = "typing-extensions", specifier = ">=4.14.1" },
{ name = "xonsh", specifier = ">=0.23.0" },
@ -747,7 +815,9 @@ subints = [{ name = "msgspec", marker = "python_full_version >= '3.14'", specifi
sync-pause = [{ name = "greenback", marker = "python_full_version == '3.13.*'", specifier = ">=1.2.1,<2" }]
testing = [
{ name = "pexpect", specifier = ">=4.9.0,<5" },
{ name = "psutil", specifier = ">=7.0.0" },
{ name = "pytest", specifier = ">=9.0.3" },
{ name = "pytest-timeout", specifier = ">=2.3" },
]
[[package]]

586
xontrib/tractor_diag.xsh 100644
View File

@ -0,0 +1,586 @@
"""
`xontrib_tractor_diag`: pytest/tractor diagnostic aliases.
All aliases live under the `acli.` namespace so xonsh's
prefix-completion treats them as a sub-cmd group — type
`acli.<TAB>` to see the full set.
Provides:
- `acli.ptree <pid|pgrep-pat>` psutil-backed proc tree,
live + zombies split.
- `acli.hung_dump <pid|pat> [...]` kernel `wchan`/`stack` +
`py-spy dump` (incl `--locals`)
for each pid in tree.
- `acli.bindspace_scan [<name>|<dir>]` find orphaned tractor UDS
sock files (no live owner pid).
bare name -> `$XDG_RUNTIME_DIR/<name>`
(e.g. `piker`, `tractor`);
path -> use as-is.
default: `$XDG_RUNTIME_DIR/tractor`.
- `acli.dump_all <pid> [--out-dir] full snapshot bundle —
[--label]` ptree + hung_dump + bindspace
written to a timestamped dir
for sharing / AI introspection.
- `acli.reap [opts]` SC-polite zombie-subactor
reaper + optional `/dev/shm/`
+ UDS sock-file sweeps.
alias for `scripts/tractor-reap`.
- `acli.watch [-n SEC] <alias-name> run a callable alias in
[alias-args]` an alt-screen loop with
flicker-free repaint
(cursor-home + per-line
EL + post-draw erase-down).
Loading from repo root:
xontrib load -p ./xontrib tractor_diag
Or source directly:
source ./xontrib/tractor_diag.xsh
Pipe-to-paste idiom (xonsh):
acli.hung_dump pytest |t /tmp/hung.log
The diagnostic core lives in `tractor._testing.trace` so it
can also be invoked from inside pytest tests (e.g. via
`fail_after_w_trace` / `afk_alarm_w_trace` capture-on-hang
helpers) — these aliases are just thin terminal wrappers.
Requires `psutil` for full functionality (`ptree` and the
`hung_dump` tree-walk). Falls back to `pgrep -P` recursion if
missing.
"""
import os
import sys
import signal
import time
from typing import (
Callable,
)
from pathlib import Path
from tractor._testing.trace import (
dump_all as _dump_all,
dump_hung_state,
dump_proc_tree,
resolve_pids,
scan_bindspace,
)
@aliases.unthreadable
def watch(
args: list[str],
) -> int:
'''
A per-term optimized `watch`-like alias for xonsh
that runs an arbitrary callable alias in a loop
inside the alt-screen buffer. Ctrl-C returns to a
pristine shell, SIGWINCH triggers a full redraw,
and the per-frame draw uses cursor-home + per-line
EL + post-draw erase-down so the loop is flicker-
free even when individual lines shrink or grow
between frames.
usage: acli.watch [-n SEC] <alias-name>
[alias-args]...
Examples:
acli.watch acli.ptree pytest
acli.watch -n 1.0 acli.bindspace_scan piker
acli.watch acli.hung_dump pytest
Only callable aliases (Python functions registered
in `aliases`) are supported. Subprocess-style
aliases raise an error — wrap them in a thin
callable if you need watching.
Output capture: the watched alias's stdout is
redirected into a `StringIO` per frame so we can
post-process it (insert `\033[K` before each `\n`).
Aliases that write directly to `sys.stdout.buffer`
or `os.write(1, ...)` bypass capture; for those the
EL-fix won't apply but the loop still functions.
'''
import argparse, io
from contextlib import redirect_stdout
parser = argparse.ArgumentParser(
prog='acli.watch',
description=watch.__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
'-n', '--interval',
type=float,
default=0.3,
help='poll interval in seconds (default: 0.3)',
)
parser.add_argument(
'alias',
help='name of a registered xonsh callable alias',
)
parser.add_argument(
'alias_args',
nargs=argparse.REMAINDER,
help='args forwarded to the watched alias',
)
try:
ns = parser.parse_args(args)
except SystemExit as se:
return int(se.code) if se.code is not None else 0
raw = aliases.get(ns.alias)
if raw is None:
print(
f'[acli.watch] no such alias: {ns.alias!r}'
)
return 1
# xonsh stores callable aliases as a bare callable
# OR wraps them in `[fn, *preset_args]` (depending
# on registration path / version). Unwrap both.
fn: Callable|None = None
preset_args: list = []
if callable(raw):
fn = raw
elif (
isinstance(raw, list)
and raw
and callable(raw[0])
):
fn = raw[0]
preset_args = list(raw[1:])
if fn is None:
kind: str = type(raw).__name__
print(
f'[acli.watch] alias {ns.alias!r} is not a '
f'callable alias (got {kind}); '
f'subprocess-style aliases not supported'
)
return 1
_FD: int = sys.stdout.fileno()
need_full_clear: bool = False
def _on_winch(signum, frame):
nonlocal need_full_clear
need_full_clear = True
prev_winch = signal.signal(
signal.SIGWINCH,
_on_winch,
)
prev_sigint = signal.signal(
signal.SIGINT,
signal.default_int_handler,
)
os.write(_FD, b'\033[?1049h\033[?25l')
try:
while True:
buf = io.StringIO()
with redirect_stdout(buf):
fn(preset_args + ns.alias_args)
if need_full_clear:
os.write(_FD, b'\033[H\033[2J')
need_full_clear = False
else:
os.write(_FD, b'\033[H')
# `\033[K` (EL) before each newline erases
# any stale tail chars left by a longer
# prior-frame version of the same line.
text: str = buf.getvalue()
painted: bytes = (
text.replace('\n', '\033[K\n').encode()
)
os.write(_FD, painted)
os.write(_FD, b'\033[J')
time.sleep(ns.interval)
except KeyboardInterrupt:
pass
finally:
os.write(_FD, b'\033[?25h\033[?1049l')
signal.signal(signal.SIGWINCH, prev_winch)
signal.signal(signal.SIGINT, prev_sigint)
return 0
# --- ptree ----------------------------------------------------
def _ptree(
args: list[str],
):
'''
psutil-backed proc tree; per-proc classification into
severity-ordered buckets so leaked / defunct procs
don't hide in the noise of normal `live` rows.
usage: acli.ptree [--tree|-t] <pid|pgrep-pattern> [...]
See `tractor._testing.trace.dump_proc_tree()` for the
bucket semantics + classification details.
To watch this live with flicker-free repaint
(alt-screen, per-line EL, SIGWINCH-aware):
.. code-block:: xonsh
acli.watch acli.ptree pytest
'''
flag_tree: bool = False
pos_args: list = []
for a in args:
if a in ('--tree', '-t'):
flag_tree = True
else:
pos_args.append(a)
if not pos_args:
print('usage: acli.ptree [--tree|-t] <pid|pgrep-pattern> [...]')
return 1
roots: list = []
for a in pos_args:
roots.extend(resolve_pids(a))
roots = sorted(set(roots))
if not roots:
print(f'(no procs match: {pos_args})')
return 1
print(dump_proc_tree(roots, flag_tree=flag_tree), end='')
# --- hung-dump -----------------------------------------------
def _hung_dump(args):
'''
kernel + python state for a hung pytest/tractor tree.
walks all descendants of each `<pid|pgrep-pat>` arg.
usage: acli.hung_dump <pid|pgrep-pattern> [...]
note: `/proc/<pid>/stack` and `py-spy dump` typically
require CAP_SYS_PTRACE — invoked via `sudo -n`. If sudo
isn't cached this alias prompts (via `sudo -v`); for the
non-interactive equivalent see
`tractor._testing.trace.dump_hung_state(allow_sudo_prompt=False)`.
'''
if not args:
print('usage: acli.hung_dump <pid|pgrep-pattern> [...]')
return 1
roots: list = []
for a in args:
roots.extend(resolve_pids(a))
roots = sorted(set(roots))
if not roots:
print(f'(no procs match: {args})')
return 1
print(
dump_hung_state(roots, allow_sudo_prompt=True),
end='',
)
# --- bindspace-scan ------------------------------------------
def _bindspace_scan(args):
'''
Scan a tractor UDS bindspace dir for orphan sock files.
usage: acli.bindspace_scan [<name>|<dir>]
See `tractor._testing.trace.scan_bindspace()` for full arg
semantics + output-bucket details.
'''
arg: str | None = args[0] if args else None
print(scan_bindspace(arg), end='')
# --- dump-all (snapshot bundle) ------------------------------
def _dump_all_alias(args):
'''
Capture a full diag snapshot bundle for a hung proc-tree
into a timestamped directory for offline / AI inspection.
usage: acli.dump_all <pid|pgrep-pat>
[--label <label>]
[--out-dir <path>]
Writes:
<out_dir>/<label>__<ts>/{trace.txt, bindspace.txt, meta.json}
Defaults:
--label = `manual`
--out-dir = `$XDG_CACHE_HOME/tractor/hung-dumps/`
(fallback `~/.cache/tractor/hung-dumps/`)
'''
import argparse
parser = argparse.ArgumentParser(
prog='acli.dump_all',
description=_dump_all_alias.__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
'target',
help='pid or pgrep -f pattern',
)
parser.add_argument(
'--label', '-l',
default='manual',
help='snapshot dir label prefix (default: `manual`)',
)
parser.add_argument(
'--out-dir', '-o',
type=Path,
default=None,
help='snapshot root dir (default: '
'$XDG_CACHE_HOME/tractor/hung-dumps/)',
)
try:
ns = parser.parse_args(args)
except SystemExit as se:
return int(se.code) if se.code is not None else 0
pids: list = resolve_pids(ns.target)
if not pids:
print(f'(no procs match: {ns.target})')
return 1
# snapshot scoped to ONE root — pick the first matched
# pid. Multi-root snapshots can be done by invoking
# `acli.dump_all <pid>` per root.
root_pid: int = pids[0]
if len(pids) > 1:
print(
f'[acli.dump_all] {len(pids)} pids matched '
f'{ns.target!r}; snapshotting tree from {root_pid} '
f'(re-run per-pid for others: {pids[1:]})'
)
dump_dir = _dump_all(
root_pid,
out_dir=ns.out_dir,
label=ns.label,
allow_sudo_prompt=True, # CLI: ok to prompt
)
print(f'[acli.dump_all] snapshot written to: {dump_dir}')
# --- acli.reap ------------------------------------------------
def _tractor_reap(args):
'''
SC-polite zombie-subactor reaper + optional `/dev/shm/`
orphan-segment sweep + optional UDS sock-file sweep.
usage: acli.reap [-h] [--parent PID] [--grace SEC]
[--dry-run] [--shm | --shm-only]
[--uds | --uds-only]
phases (run in order when enabled):
1. process reap — finds tractor subactor procs left
alive after a `pytest`/app run that failed to fully
cancel its tree. Default = orphan-mode (PPid==1
init-reparented procs whose cwd matches repo root
AND cmdline contains `python`). With `--parent`,
scopes to descendants of a specific live PID.
SIGINT first, then SIGKILL after `--grace` (default
3.0s).
2. shm sweep (`--shm`/`--shm-only`) — unlinks
`/dev/shm/<file>` entries owned by the current uid
that no live process has open. Needed because
`tractor` disables `mp.resource_tracker`.
3. UDS sweep (`--uds`/`--uds-only`) — unlinks
`${XDG_RUNTIME_DIR}/tractor/<name>@<pid>.sock`
files whose binder pid is dead (or the `1616`
registry sentinel). See issue #452.
Mirrors `scripts/tractor-reap` (use `-n`/`--dry-run`
first to see what would be touched).
'''
import argparse
parser = argparse.ArgumentParser(
prog='acli.reap',
description=_tractor_reap.__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
'--parent', '-p',
type=int,
default=None,
help='descendant-mode: reap procs with PPid==<pid>',
)
parser.add_argument(
'--grace', '-g',
type=float,
default=3.0,
help='SIGINT grace window in seconds (default 3.0)',
)
parser.add_argument(
'--dry-run', '-n',
action='store_true',
help='list matched pids/paths but do not signal/unlink',
)
parser.add_argument(
'--shm',
action='store_true',
help='also unlink orphaned /dev/shm segments',
)
parser.add_argument(
'--shm-only',
action='store_true',
help='skip process reap; only do the shm sweep',
)
parser.add_argument(
'--uds',
action='store_true',
help='also unlink orphaned UDS sock-files',
)
parser.add_argument(
'--uds-only',
action='store_true',
help='skip process reap + shm; only do the UDS sweep',
)
try:
ns = parser.parse_args(args)
except SystemExit as se:
# `argparse` raises SystemExit on `-h`/bad-args; let
# xonsh treat it as a normal alias return code.
return int(se.code) if se.code is not None else 0
skip_proc_reap: bool = (
ns.shm_only
or
ns.uds_only
)
# `tractor` is assumed to be importable in the xonsh env
# this xontrib was sourced into (a venv with the package
# installed). The standalone `scripts/tractor-reap` does
# `git rev-parse --show-toplevel` + `sys.path.insert` for
# cold-shell usability — that overhead is unnecessary
# here since we're already inside the project's venv.
from tractor._testing._reap import (
find_descendants,
find_orphans,
find_orphaned_shm,
find_orphaned_uds,
reap,
reap_shm,
reap_uds,
)
rc: int = 0
# phase 1: process reap (skipped under `--*-only`)
if not skip_proc_reap:
if ns.parent is not None:
pids: list = find_descendants(ns.parent)
mode: str = f'descendants of PPid={ns.parent}'
else:
pids = find_orphans()
mode = (
'orphans (PPid==1, intrinsic '
'cmdline/comm match — `tractor[…]` or '
'`tractor._child`)'
)
if not pids:
print(f'[acli.reap] no {mode} to reap')
elif ns.dry_run:
print(
f'[acli.reap] dry-run — {mode}:\n {pids}'
)
else:
_, survivors = reap(pids, grace=ns.grace)
if survivors:
rc = 1
# phase 2: shm sweep (opt-in)
if ns.shm or ns.shm_only:
leaked: list = find_orphaned_shm()
if not leaked:
print(
'[acli.reap] no orphaned /dev/shm '
'segments to sweep'
)
elif ns.dry_run:
print(
f'[acli.reap] dry-run — {len(leaked)} '
f'orphaned shm segment(s):\n {leaked}'
)
else:
_, errors = reap_shm(leaked)
if errors:
rc = 1
# phase 3: UDS sweep (opt-in)
if ns.uds or ns.uds_only:
leaked_uds: list = find_orphaned_uds()
if not leaked_uds:
print(
'[acli.reap] no orphaned UDS sock-files '
'to sweep'
)
elif ns.dry_run:
print(
f'[acli.reap] dry-run — {len(leaked_uds)} '
f'orphaned UDS sock-file(s):\n {leaked_uds}'
)
else:
_, errors = reap_uds(leaked_uds)
if errors:
rc = 1
return rc
# --- registration ---------------------------------------------
# all aliases under the `acli.` namespace so xonsh's prefix-
# completion makes them feel like a sub-cmd group: type
# `acli.<TAB>` and the full set is suggested. no parent
# `acli` cmd exists — the dot is purely a naming convention.
_TCLI_ALIASES: dict = {
'acli.ptree': _ptree,
'acli.hung_dump': _hung_dump,
'acli.bindspace_scan': _bindspace_scan,
'acli.dump_all': _dump_all_alias,
'acli.reap': _tractor_reap,
'acli.watch': watch,
}
for _name, _fn in _TCLI_ALIASES.items():
aliases[_name] = _fn
# xontrib protocol hooks (for `xontrib load tractor_diag`).
# also harmless when sourced directly.
def _load_xontrib_(xsh, **_):
return {}
def _unload_xontrib_(xsh, **_):
for name in _TCLI_ALIASES:
aliases.pop(name, None)
return {}