Test SIGINT teardown with stalled IPC transports
Add `examples.advanced_faults.ipc_stalls` and an external `RootMonitor` so SIGINT delivery and shutdown evidence stay independent of the stalled actor channels. Exercise six Linux TCP cases with the `trio` spawner: healthy IPC, context receive, send, both directions, context cancel send, and both directions plus the subactor `Portal` cancel send. Deats, - Split cancellable send gates from buffered receive gates, and require actual parked operations before publishing readiness. - Keep the OOB context `Channel` distinct from the child-control `Portal`; context-only faults retain cooperative child exit. - Require finite cancel-send expiry, runtime hard-reap logs, SIGKILL status and completed `waitpid` for the control fault. - Bound readiness and shutdown with an independent stdout watchdog; rescue cleanup cannot satisfy teardown assertions. - Use `get_rando_addr()` for unreserved registry port selection, raise `TimeoutError` on watchdog expiry, and retain structured shutdown evidence through `record_property`. These waits model cancellable runtime unresponsiveness, not kernel TCP loss or an uninterruptible syscall. Preserve human review decisions and historical working-tree verification in the prompt provenance; exact-tree plan checks remain pending. Prompt-IO: ai/prompt-io/opencode/20260923_ipc_fault_regressions.md (this commit msg was generated in some part by `opencode` using `gpt-6.1-sol` (`openai`))ipc_fault_regressions
parent
b3e07bd569
commit
13ffb13be5
|
|
@ -0,0 +1,84 @@
|
|||
---
|
||||
model: openai/gpt-6-astra
|
||||
service: opencode
|
||||
scope: code
|
||||
substantive: true
|
||||
git_ref: 9547c2f9
|
||||
raw_file: 20260923_ipc_fault_regressions.raw.md
|
||||
---
|
||||
|
||||
## Human instructions
|
||||
|
||||
Audit terminal transport-close context interruption and reconcile the
|
||||
advanced-fault suite. Preserve its original stalled-IPC cancellation
|
||||
invariant using controlled cancellable waits, rather than privileged
|
||||
kernel DROP tooling immediately. Write a detailed future network-outage
|
||||
handoff for TIPC/WireGuard and IB resiliency. Do not disturb live setups.
|
||||
|
||||
## Changes
|
||||
|
||||
- examples/advanced_faults/ipc_failure_during_stream.py: distinguish
|
||||
detected close from stalled I/O and use explicit test registry inputs.
|
||||
- tests/test_advanced_faults.py: validate transport outcomes, origin and
|
||||
child reaping instead of obsolete expected hangs/cancel acknowledgements.
|
||||
- examples/advanced_faults/ipc_stalls.py and tests/test_ipc_stalls.py:
|
||||
controlled per-channel cancellable waits, actual test-root SIGINT,
|
||||
finite-deadline cleanup synchronization, and stalled-subactor-Portal
|
||||
hard-reap coverage. The example has a discoverable console entrypoint.
|
||||
The review split the transport send wrapper from its buffered
|
||||
receive-stream wrapper, moved the runnable scenario to examples,
|
||||
and extracted external process monitoring into a context manager.
|
||||
- tractor/_context.py: raise unfinished recorded transport errors even
|
||||
after scope cancellation was absorbed and the channel is disconnected.
|
||||
- plans/opencode/network-outage-resiliency-handoff.md: separate,
|
||||
follow-up engineering handoff for kernel/network/application
|
||||
resiliency; it is not part of the stall-test commit boundary.
|
||||
|
||||
## Verification
|
||||
|
||||
Advanced faults: 11 TCP and 11 UDS passed, including all 22 child-exit
|
||||
checks. Existing EOF: 5 TCP and 5 UDS passed. Context-stream semantics:
|
||||
65 TCP and 65 UDS passed. Final stall suite: 6 TCP cases passed, including
|
||||
runtime hard reaping of a test-owned child whose control channel stalls.
|
||||
Passing tests do not rely on forced watchdog cleanup. Reaper disabled;
|
||||
legacy test registry defaults were overridden in memory for isolation.
|
||||
These are working-tree checks, not commit-plan attestations.
|
||||
After the review adjustments, the six stall cases passed again.
|
||||
The third review retained those six cases after moving the runnable
|
||||
scenario and isolating subprocess monitoring from assertions.
|
||||
The next local review's six renamed TCP cases passed in 11.98 seconds
|
||||
with the reaper plugin disabled; the example's CLI help also passed.
|
||||
|
||||
## Human contributions
|
||||
|
||||
The human identified the old close simulation and its intended Ctrl-C
|
||||
contract, explicitly distinguished that from the newly accepted EOF
|
||||
notification contract, and selected controlled runtime stalls as the
|
||||
bounded first step. They supplied the Gateway/roaming incident traces
|
||||
and required isolated worktrees rather than changes to live source.
|
||||
The original unstaged EOF mask remains untouched in its own checkout.
|
||||
During a second review, the human supplied example documentation,
|
||||
asked for precise type/field references, pointed out the mixed
|
||||
transport/receiver interface, and requested explicit parameter
|
||||
types, method contracts, and test-mode descriptions. Those decisions
|
||||
shape the final test helper and driver.
|
||||
The human's third review requested the example-module location,
|
||||
explicit parked-I/O documentation, code-quoted symbols and namespaces,
|
||||
and a context manager for the external process monitor. They chose to
|
||||
consider the longer-term network handoff in a follow-up commit.
|
||||
The next local review requested explicit context/subactor Portal
|
||||
distinctions, six descriptive mode names, structured case documentation
|
||||
and per-branch shutdown evidence in tests/test_ipc_stalls.py and
|
||||
examples/advanced_faults/ipc_stalls.py. It also requested TimeoutError
|
||||
for watchdog expiry, a configurable signal with SIGINT as default,
|
||||
reuse of tractor._testing.addr.get_rando_addr(), and follow-up TODOs
|
||||
for a shared monitor/scenario interface and discovery daemon reuse.
|
||||
The human explicitly authorized removal of the record_property TODO.
|
||||
|
||||
## Scope limits
|
||||
|
||||
No automatic IB reconnect, kernel DROP, suspend/resume monitor, native
|
||||
debugger/Qt crash fix, or new production transport API is implemented.
|
||||
Stall tests currently target Linux/Trio-spawner/TCP. The separate
|
||||
detected-close fix was committed as `192411c4`; this stall-test
|
||||
follow-up remains uncommitted.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: openai/gpt-6-astra
|
||||
service: opencode
|
||||
git_ref: 9547c2f9
|
||||
diff_cmd: git diff HEAD
|
||||
---
|
||||
|
||||
> git diff HEAD -- tractor/_context.py tests/test_advanced_faults.py examples/advanced_faults/ipc_failure_during_stream.py tests/test_ipc_stalls.py tests/_ipc_stalls.py
|
||||
|
||||
Separate terminal transport closure from unresponsive IPC in fault
|
||||
coverage. Preserve original fault causes and cancellation provenance,
|
||||
and test actual SIGINT only on externally controlled test-root processes.
|
||||
Add normally cancellable gates with parked-operation handshakes.
|
||||
An additional control-stall case requires runtime out-of-IPC hard reaping,
|
||||
not cooperative cancellation through an unaffected control connection.
|
||||
|
||||
The corrected matrix exposed a swallowed TransportClosed after the
|
||||
context nursery absorbed its own cancellation. Route that unfinished
|
||||
recorded outcome through wait_for_result even on a disconnected channel.
|
||||
This raises without waiting on IPC and preserves normal final outcomes.
|
||||
|
||||
Validation completed: 22 advanced-fault cases, 10 existing EOF cases,
|
||||
130 context-stream cases, and 6 final controlled-stall cases passed.
|
||||
No kernel network tools, user processes, live containers, or unrelated
|
||||
shm resources were modified.
|
||||
|
|
@ -0,0 +1,514 @@
|
|||
'''
|
||||
Test-owned IPC stall simulation and a separately signalled root.
|
||||
|
||||
These cancellable waits simulate runtime unresponsiveness, NOT
|
||||
kernel TCP half-open connections, loss, or a shielded syscall stall.
|
||||
Context-only faults wrap the OOB ``peer_ptl``'s separate ``Channel``
|
||||
to the child, preserving cancellation through the subactor
|
||||
``staller_ptl`` returned by ``an.start_actor()``. The mode
|
||||
``stalled_context_and_subactor_cancel_send`` parks both context
|
||||
directions and the subactor ``Portal``'s cancel send. Registry
|
||||
traffic and the independent stdout control pipe remain usable.
|
||||
|
||||
'''
|
||||
import argparse
|
||||
from collections.abc import AsyncIterator
|
||||
from functools import partial
|
||||
import json
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import trio
|
||||
import tractor
|
||||
from tricycle import BufferedReceiveStream
|
||||
from tractor._testing import assert_child_reaped
|
||||
from tractor._testing.addr import get_rando_addr
|
||||
from tractor.discovery._addr import UnwrappedAddress
|
||||
from tractor.ipc._chan import _connect_chan
|
||||
from tractor.ipc._transport import MsgpackTransport
|
||||
from tractor.msg.types import (
|
||||
Start,
|
||||
Yield,
|
||||
)
|
||||
from tractor.runtime._portal import open_portal
|
||||
|
||||
|
||||
class StallableReceiveStream:
|
||||
'''
|
||||
Simulate a stalled buffered read on one IPC transport.
|
||||
|
||||
``MsgpackTransport._iter_packets()`` reads each packet header
|
||||
via ``MsgpackTransport.recv_stream.receive_exactly(4)``.
|
||||
Replacing ``MsgpackTransport.recv_stream`` intercepts that call,
|
||||
including reads satisfied without another socket operation.
|
||||
The packet iterator reads this field for each new header and
|
||||
payload. If an old call is already waiting inside the original
|
||||
receiver, the peer's responses finish that call first; its
|
||||
next read then goes through the replacement.
|
||||
|
||||
'''
|
||||
def __init__(
|
||||
self,
|
||||
orig_recv: BufferedReceiveStream,
|
||||
) -> None:
|
||||
'''
|
||||
Retain the receiver and allocate cancellable stall events.
|
||||
|
||||
'''
|
||||
self.orig_recv: BufferedReceiveStream = orig_recv
|
||||
self.enabled: bool = False
|
||||
self.parked: trio.Event = trio.Event()
|
||||
self.release: trio.Event = trio.Event()
|
||||
self.receiving: int = 0
|
||||
|
||||
async def receive_exactly(
|
||||
self,
|
||||
size: int,
|
||||
) -> bytes:
|
||||
'''
|
||||
Pause a buffered read until the test releases or cancels it.
|
||||
|
||||
Count active calls and set
|
||||
``StallableReceiveStream.parked`` before waiting on
|
||||
``StallableReceiveStream.release``. Once released, read
|
||||
from the original ``BufferedReceiveStream``.
|
||||
The wait stays cancellable for the SIGINT teardown test.
|
||||
|
||||
'''
|
||||
if self.enabled:
|
||||
self.receiving += 1
|
||||
try:
|
||||
self.parked.set()
|
||||
await self.release.wait()
|
||||
finally:
|
||||
self.receiving -= 1
|
||||
return await self.orig_recv.receive_exactly(size)
|
||||
|
||||
|
||||
class StallableTransport:
|
||||
'''
|
||||
Simulate stalled IPC sends and receives on one ``Channel``.
|
||||
|
||||
Replace ``Channel._transport`` to pause sends before its
|
||||
production shield, and replace ``MsgpackTransport.recv_stream``
|
||||
with a ``StallableReceiveStream`` for buffered packet reads.
|
||||
Other transport operations delegate to the original
|
||||
``MsgpackTransport``.
|
||||
The test can inspect each paused operation independently while
|
||||
the registry and unrelated actor channels remain usable.
|
||||
|
||||
'''
|
||||
def __init__(
|
||||
self,
|
||||
channel: tractor.Channel,
|
||||
control_marker: str|None = None,
|
||||
) -> None:
|
||||
'''
|
||||
Wrap one ``Channel``'s transport and buffered receiver.
|
||||
|
||||
'''
|
||||
self.orig_tpt: MsgpackTransport = channel._transport
|
||||
self.recv_stream: StallableReceiveStream = (
|
||||
StallableReceiveStream(
|
||||
self.orig_tpt.recv_stream
|
||||
)
|
||||
)
|
||||
self.send_enabled: bool = False
|
||||
self.send_parked: trio.Event = trio.Event()
|
||||
self.release_send: trio.Event = trio.Event()
|
||||
self.sending: int = 0
|
||||
self.finite_send: bool = False
|
||||
self.cancel_request: bool = False
|
||||
self.control_marker: str|None = control_marker
|
||||
self.orig_tpt.recv_stream = self.recv_stream
|
||||
channel._transport = self
|
||||
|
||||
def __getattr__(
|
||||
self,
|
||||
name: str,
|
||||
) -> Any:
|
||||
'''
|
||||
Delegate other transport operations to the original.
|
||||
|
||||
'''
|
||||
return getattr(self.orig_tpt, name)
|
||||
|
||||
def __aiter__(self) -> AsyncIterator[dict]:
|
||||
'''
|
||||
Delegate packet iteration to ``StallableTransport.orig_tpt``.
|
||||
|
||||
Its packet reader uses the replaced ``recv_stream`` field.
|
||||
|
||||
'''
|
||||
return self.orig_tpt.__aiter__()
|
||||
|
||||
async def send(
|
||||
self,
|
||||
msg: Any,
|
||||
send_deadline: float = float('inf'),
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
'''
|
||||
Park a send before ``MsgpackTransport.send()`` enters its
|
||||
shield.
|
||||
|
||||
``StallableTransport.send_parked`` tells the controller
|
||||
when a real send is blocked. The cancellable wait respects
|
||||
``send_deadline``; a stalled cancel request can time out
|
||||
and trigger runtime hard-reaping.
|
||||
|
||||
'''
|
||||
if self.send_enabled:
|
||||
self.sending += 1
|
||||
try:
|
||||
self.finite_send = send_deadline != float('inf')
|
||||
self.cancel_request = (
|
||||
isinstance(msg, Start)
|
||||
and
|
||||
msg.ns == 'self'
|
||||
and
|
||||
msg.func == 'cancel'
|
||||
)
|
||||
self.send_parked.set()
|
||||
with trio.CancelScope(deadline=send_deadline) as cs:
|
||||
await self.release_send.wait()
|
||||
if cs.cancelled_caught:
|
||||
raise tractor.TransportClosed(
|
||||
'Test IPC send deadline expired'
|
||||
)
|
||||
finally:
|
||||
self.sending -= 1
|
||||
if self.control_marker:
|
||||
emit(
|
||||
marker=self.control_marker,
|
||||
event='control_send_exit',
|
||||
deadline_expired=(
|
||||
trio.current_time() >= send_deadline
|
||||
),
|
||||
)
|
||||
await self.orig_tpt.send(
|
||||
msg,
|
||||
send_deadline=send_deadline,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def snapshot(self) -> dict[str, bool]:
|
||||
'''
|
||||
Report which transport operations are still parked.
|
||||
|
||||
'''
|
||||
return {
|
||||
'receive': bool(self.recv_stream.receiving),
|
||||
'send': bool(self.sending),
|
||||
'finite_send': self.finite_send,
|
||||
}
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def probe_peer(
|
||||
ctx: tractor.Context,
|
||||
) -> None:
|
||||
'''
|
||||
Keep the peer context alive while the caller stalls IPC.
|
||||
|
||||
Two responses finish any pending read in the original
|
||||
receiver. ``MsgpackTransport._iter_packets()`` then reads
|
||||
again through the replaced ``recv_stream`` field, even when
|
||||
the previous response used buffered bytes. Only the external
|
||||
controller's SIGINT may terminate this test context.
|
||||
|
||||
'''
|
||||
await ctx.started(os.getpid())
|
||||
async with ctx.open_stream() as stream:
|
||||
assert await stream.receive() == 'probe'
|
||||
await stream.send('probe-1')
|
||||
await stream.send('probe-2')
|
||||
await trio.sleep_forever()
|
||||
raise AssertionError('Peer stream completed before cancellation')
|
||||
|
||||
|
||||
async def peer_address() -> UnwrappedAddress:
|
||||
'''
|
||||
Provide the peer address for a distinct OOB context connection.
|
||||
|
||||
'''
|
||||
return tractor.current_actor().accept_addrs[0]
|
||||
|
||||
|
||||
def emit(
|
||||
marker: str,
|
||||
event: str,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
'''
|
||||
Publish a JSON test event on the root's stdout pipe.
|
||||
|
||||
The controller in ``tests/test_ipc_stalls.py`` uses ``marker``
|
||||
to distinguish child identity, readiness and final results
|
||||
from actor logs.
|
||||
|
||||
'''
|
||||
message: str = marker + json.dumps({
|
||||
'event': event,
|
||||
**fields,
|
||||
})
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
async def root_main(
|
||||
# !TODO, explain each of these inputs with comments..
|
||||
mode: str, # ``test_stalled_channel_sigint`` case to simulate
|
||||
port: int, # isolated root actor registry port
|
||||
marker: str, # prefix for controller-visible JSON records
|
||||
state: dict[str, Any], # child and parked-I/O result evidence
|
||||
) -> None:
|
||||
'''
|
||||
Park selected IPC operations in a test-owned actor tree.
|
||||
|
||||
``mode`` selects ``healthy_ipc``, ``stalled_context_receive``,
|
||||
``stalled_context_send``, ``stalled_context_send_and_receive``,
|
||||
``stalled_context_cancel_send``, or
|
||||
``stalled_context_and_subactor_cancel_send``. The last case
|
||||
parks both context directions and the subactor ``Portal``'s
|
||||
cancel request. ``port`` selects this root's registry address;
|
||||
``marker`` frames stdout events; ``state`` retains child and
|
||||
parked-I/O evidence for the SIGINT result.
|
||||
|
||||
'''
|
||||
async with tractor.open_root_actor(
|
||||
name='ipc-stall-root',
|
||||
registry_addrs=[('127.0.0.1', port)],
|
||||
enable_transports=['tcp'],
|
||||
ensure_registry=True,
|
||||
start_method='trio',
|
||||
debug_mode=False,
|
||||
loglevel='error',
|
||||
):
|
||||
async with tractor.open_nursery() as an:
|
||||
staller_ptl = await an.start_actor(
|
||||
'ipc-stall-peer',
|
||||
enable_modules=[__name__],
|
||||
)
|
||||
proc: trio.Process = next(
|
||||
iter(an._children.values())
|
||||
)[1]
|
||||
state['child'] = proc
|
||||
emit(
|
||||
marker=marker,
|
||||
event='child',
|
||||
pid=proc.pid,
|
||||
)
|
||||
peer_addr: UnwrappedAddress = await staller_ptl.run(
|
||||
peer_address
|
||||
)
|
||||
|
||||
# Open an OOB context ``Channel`` via ``_connect_chan``
|
||||
# and ``open_portal`` as ``peer_ptl``. The subactor
|
||||
# ``Portal``'s ``staller_ptl.channel`` is distinct.
|
||||
#
|
||||
# Most modes leave subactor ``Portal`` cancel usable.
|
||||
# ``stalled_context_and_subactor_cancel_send`` also
|
||||
# stalls its cancel send AND both context directions.
|
||||
async with (
|
||||
_connect_chan(peer_addr) as peer_channel,
|
||||
open_portal(peer_channel) as peer_ptl,
|
||||
peer_ptl.open_context(probe_peer) as (ctx, pid),
|
||||
):
|
||||
# Match the peer PID from ``Context.started()``.
|
||||
assert pid == proc.pid
|
||||
# The OOB context connection must not silently
|
||||
# stall the subactor ``Portal``'s channel as well.
|
||||
assert ctx.chan is not staller_ptl.channel
|
||||
stall_tpt: StallableTransport = StallableTransport(
|
||||
ctx.chan
|
||||
)
|
||||
stall_tpt.recv_stream.enabled = mode in {
|
||||
# Suspend context receive only.
|
||||
'stalled_context_receive',
|
||||
# Suspend context send and receive.
|
||||
'stalled_context_send_and_receive',
|
||||
# Also stall the subactor ``Portal`` cancel send.
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
}
|
||||
await ctx.chan.send(
|
||||
Yield(cid=ctx.cid, pld='probe')
|
||||
)
|
||||
# Wait for the packet reader to park for each
|
||||
# receive-stalled mode, not just wrapper setup.
|
||||
# ?TODO, in what cases is this set vs. not?
|
||||
# sure would be handy to see that from this layer..
|
||||
# ``stalled_context_receive``,
|
||||
# ``stalled_context_send_and_receive``, and
|
||||
# ``stalled_context_and_subactor_cancel_send`` wait
|
||||
# here; other cases consume both probe responses.
|
||||
if stall_tpt.recv_stream.enabled:
|
||||
await stall_tpt.recv_stream.parked.wait()
|
||||
else:
|
||||
# Drain the peer's two responses in modes that
|
||||
# do not intentionally stall context reads.
|
||||
for expected in ('probe-1', 'probe-2'):
|
||||
pld: str = await ctx.pld_rx.recv_pld(
|
||||
ipc=ctx,
|
||||
expect_msg=Yield,
|
||||
)
|
||||
assert pld == expected
|
||||
|
||||
stall_tpt.send_enabled = mode in {
|
||||
# Suspend context send only.
|
||||
'stalled_context_send',
|
||||
# Suspend both context directions.
|
||||
'stalled_context_send_and_receive',
|
||||
# Suspend ``Context.cancel()``'s timed send.
|
||||
'stalled_context_cancel_send',
|
||||
# Also suspend the subactor ``Portal`` cancel.
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
}
|
||||
control_tpt: StallableTransport|None = None
|
||||
if mode == (
|
||||
'stalled_context_and_subactor_cancel_send'
|
||||
):
|
||||
control_tpt = StallableTransport(
|
||||
staller_ptl.channel,
|
||||
control_marker=marker,
|
||||
)
|
||||
control_tpt.send_enabled = True
|
||||
|
||||
async def ready() -> None:
|
||||
'''
|
||||
Report selected parked I/O to the controller.
|
||||
|
||||
Readiness requires real blocked sends/receives;
|
||||
simply installing a wrapper would not prove a
|
||||
stalled connection before SIGINT arrives.
|
||||
|
||||
'''
|
||||
if stall_tpt.send_enabled:
|
||||
await stall_tpt.send_parked.wait()
|
||||
if control_tpt is not None:
|
||||
await control_tpt.send_parked.wait()
|
||||
assert control_tpt.cancel_request
|
||||
assert control_tpt.finite_send
|
||||
assert control_tpt.sending
|
||||
state['control_fault_active'] = (
|
||||
control_tpt.snapshot()
|
||||
)
|
||||
snapshot: dict[str, bool] = stall_tpt.snapshot()
|
||||
assert snapshot['receive'] == (
|
||||
stall_tpt.recv_stream.enabled
|
||||
)
|
||||
assert snapshot['send'] == stall_tpt.send_enabled
|
||||
if mode == 'stalled_context_cancel_send':
|
||||
assert snapshot['finite_send']
|
||||
assert proc.returncode is None
|
||||
assert ctx.chan.connected()
|
||||
state['stalls_fault_active'] = snapshot
|
||||
emit(
|
||||
marker=marker,
|
||||
event='ready',
|
||||
stalls=snapshot,
|
||||
control=state.get(
|
||||
'control_fault_active'
|
||||
),
|
||||
)
|
||||
|
||||
async with trio.open_nursery() as tn:
|
||||
tn.start_soon(ready)
|
||||
if mode == 'stalled_context_cancel_send':
|
||||
try:
|
||||
pass
|
||||
finally:
|
||||
# Actual teardown publication, with its
|
||||
# normal finite deadline, not a sleep.
|
||||
await ctx.cancel(timeout=5)
|
||||
raise AssertionError(
|
||||
'Cleanup finished before SIGINT'
|
||||
)
|
||||
if stall_tpt.send_enabled:
|
||||
tn.start_soon(
|
||||
ctx.chan.send,
|
||||
Yield(cid=ctx.cid, pld='blocked'),
|
||||
)
|
||||
if control_tpt is not None:
|
||||
await staller_ptl.cancel_actor(timeout=5)
|
||||
raise AssertionError(
|
||||
'Control cancel finished before SIGINT'
|
||||
)
|
||||
await trio.sleep_forever()
|
||||
raise AssertionError('Context completed before SIGINT')
|
||||
|
||||
|
||||
def main() -> None:
|
||||
'''
|
||||
Run the root in a process separate from the test controller.
|
||||
|
||||
``tests/test_ipc_stalls.py`` passes ``mode``, ``port``, and
|
||||
``marker`` as positional CLI args to run this example in a
|
||||
separate process. It sends SIGINT without signalling ``pytest``
|
||||
and observes JSON events and process exit through stdout and
|
||||
its OS process handle, not a potentially stalled ``Channel``.
|
||||
Running this module directly defaults to
|
||||
``stalled_context_receive`` and selects an isolated registry
|
||||
port with ``tractor._testing.addr.get_rando_addr()``.
|
||||
|
||||
'''
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
'mode', nargs='?', default='stalled_context_receive',
|
||||
choices=(
|
||||
'healthy_ipc',
|
||||
'stalled_context_receive',
|
||||
'stalled_context_send',
|
||||
'stalled_context_send_and_receive',
|
||||
'stalled_context_cancel_send',
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
),
|
||||
)
|
||||
parser.add_argument('port', type=int, nargs='?')
|
||||
parser.add_argument(
|
||||
'marker', nargs='?', default='IPC-STALL:',
|
||||
)
|
||||
args = parser.parse_args()
|
||||
port: int|None = args.port
|
||||
if port is None:
|
||||
_, port = get_rando_addr('tcp')
|
||||
|
||||
state: dict[str, Any] = {}
|
||||
try:
|
||||
trio.run(
|
||||
partial(
|
||||
root_main,
|
||||
mode=args.mode,
|
||||
port=port,
|
||||
marker=args.marker,
|
||||
state=state,
|
||||
),
|
||||
restrict_keyboard_interrupt_to_checkpoints=True,
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
proc: trio.Process = state['child']
|
||||
assert_child_reaped(proc)
|
||||
# A returncode alone is insufficient: prove the runtime
|
||||
# waited for its child rather than leaving a zombie behind.
|
||||
try:
|
||||
waited = os.waitpid(proc.pid, os.WNOHANG)
|
||||
except ChildProcessError:
|
||||
reaped = True
|
||||
else:
|
||||
raise AssertionError(
|
||||
('Runtime did not reap child', waited)
|
||||
)
|
||||
emit(
|
||||
marker=args.marker,
|
||||
event='result',
|
||||
root_exit='KeyboardInterrupt',
|
||||
stalls_fault_active=state['stalls_fault_active'],
|
||||
child_reaped=reaped,
|
||||
child_returncode=proc.returncode,
|
||||
control_fault_active=state.get('control_fault_active'),
|
||||
)
|
||||
else:
|
||||
raise AssertionError('Root exited without KeyboardInterrupt')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -0,0 +1,402 @@
|
|||
'''
|
||||
Externally supervised tests for cancellable IPC unresponsiveness.
|
||||
|
||||
No kernel packet filtering: these waits model a task making no
|
||||
progress. EOF remains enabled in production and is never injected.
|
||||
The watchdog signals only its own root process handle.
|
||||
|
||||
'''
|
||||
from collections.abc import (
|
||||
Callable,
|
||||
Iterator,
|
||||
)
|
||||
from contextlib import (
|
||||
contextmanager as cm,
|
||||
suppress,
|
||||
)
|
||||
from dataclasses import (
|
||||
dataclass,
|
||||
field,
|
||||
)
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import selectors
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
import psutil
|
||||
import pytest
|
||||
from tractor._testing.addr import get_rando_addr
|
||||
|
||||
|
||||
@dataclass
|
||||
class RootMonitor:
|
||||
'''
|
||||
Observe one stalled-IPC root outside its actor/IPC tree.
|
||||
|
||||
``RootMonitor.proc`` is the test-owned root subprocess.
|
||||
``RootMonitor.transcript`` retains plain logs; only lines
|
||||
prefixed with ``RootMonitor.marker`` contain JSON records
|
||||
from ``examples.advanced_faults.ipc_stalls.emit()``.
|
||||
``RootMonitor.child`` is learned from the peer's PID event,
|
||||
so cleanup never selects a process by global name.
|
||||
|
||||
'''
|
||||
# TODO: reuse for the ``daemon`` remote registrar in
|
||||
# ``tests/discovery/conftest.py`` to verify logged discovery
|
||||
# transactions from outside the actor tree.
|
||||
# TODO: colocate scenario and monitor behind a reusable
|
||||
# ``tractor._testing`` interface for other suites.
|
||||
proc: subprocess.Popen[bytes]
|
||||
marker: str
|
||||
child: psutil.Process|None = None
|
||||
transcript: list[str] = field(default_factory=list)
|
||||
events: list[dict[str, Any]] = field(default_factory=list)
|
||||
deadline: float = field(
|
||||
default_factory=lambda: time.monotonic() + 20
|
||||
)
|
||||
|
||||
def iter_events(self) -> Iterator[dict[str, Any]]:
|
||||
'''
|
||||
Yield marker-prefixed records until root stdout closes.
|
||||
|
||||
Poll the independent pipe using ``RootMonitor.deadline``.
|
||||
Keep plain actor logs in ``RootMonitor.transcript`` and
|
||||
parse only ``emit()`` records into ``RootMonitor.events``.
|
||||
A root that cannot publish readiness or finish after
|
||||
SIGINT fails the external watchdog, regardless of IPC.
|
||||
|
||||
'''
|
||||
stdout = self.proc.stdout
|
||||
assert stdout is not None
|
||||
pending: bytes = b''
|
||||
with selectors.DefaultSelector() as poller:
|
||||
poller.register(stdout, selectors.EVENT_READ)
|
||||
while poller.get_map():
|
||||
remaining: float = (
|
||||
self.deadline - time.monotonic()
|
||||
)
|
||||
if remaining <= 0:
|
||||
raise TimeoutError('External watchdog expired')
|
||||
if not poller.select(remaining):
|
||||
raise TimeoutError(
|
||||
'External watchdog expired'
|
||||
)
|
||||
chunk: bytes = os.read(stdout.fileno(), 65536)
|
||||
if not chunk:
|
||||
poller.unregister(stdout)
|
||||
break
|
||||
pending += chunk
|
||||
|
||||
while b'\n' in pending:
|
||||
line, pending = pending.split(b'\n', 1)
|
||||
text: str = line.decode(errors='replace')
|
||||
self.transcript.append(text)
|
||||
if not text.startswith(self.marker):
|
||||
continue
|
||||
event: dict[str, Any] = json.loads(
|
||||
text[len(self.marker):]
|
||||
)
|
||||
self.events.append(event)
|
||||
yield event
|
||||
|
||||
def signal_root(
|
||||
self,
|
||||
sig: int = signal.SIGINT,
|
||||
) -> None:
|
||||
'''
|
||||
Signal the owned root; bound the ensuing shutdown.
|
||||
|
||||
Default to graceful SIGINT cancellation. ``sig`` also allows
|
||||
future scenarios to exercise non-graceful termination.
|
||||
|
||||
'''
|
||||
self.proc.send_signal(sig)
|
||||
self.deadline = time.monotonic() + 20
|
||||
|
||||
|
||||
@cm
|
||||
def open_root_monitor(
|
||||
mode: str,
|
||||
) -> Iterator[RootMonitor]:
|
||||
'''
|
||||
Own one isolated actor program and rescue only failed runs.
|
||||
|
||||
Launch ``examples.advanced_faults.ipc_stalls`` as a separate
|
||||
process with a per-test registry port and stdout control pipe.
|
||||
Keep ``pytest`` outside both potentially stalled IPC channels
|
||||
so it can signal, time out, and inspect the root's OS exit.
|
||||
Failure cleanup touches only handles created by this context;
|
||||
watchdog rescue never counts as runtime teardown success.
|
||||
|
||||
'''
|
||||
root: Path = Path(__file__).resolve().parents[1]
|
||||
marker: str = f'IPC-STALL-{uuid4().hex}:'
|
||||
# Pure address selection, without ``reg_addr`` runtime overrides.
|
||||
# ``ensure_registry`` refuses a collision instead of attaching.
|
||||
_, port = get_rando_addr('tcp')
|
||||
|
||||
env: dict[str, str] = dict(os.environ)
|
||||
env['PYTHONPATH'] = os.pathsep.join(filter(None, (
|
||||
str(root), env.get('PYTHONPATH'),
|
||||
)))
|
||||
env['PYTHONDONTWRITEBYTECODE'] = '1'
|
||||
# TODO: generalize the scenario entrypoint together with the
|
||||
# ``daemon`` fixture's spawner in
|
||||
# ``tests/discovery/conftest.py``.
|
||||
proc: subprocess.Popen[bytes] = subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
'-u',
|
||||
'-m', 'examples.advanced_faults.ipc_stalls',
|
||||
mode,
|
||||
str(port),
|
||||
marker,
|
||||
],
|
||||
cwd=root,
|
||||
env=env,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
monitor = RootMonitor(proc=proc, marker=marker)
|
||||
try:
|
||||
yield monitor
|
||||
except BaseException as exc:
|
||||
output: str = '\n'.join(monitor.transcript)
|
||||
# XXX i tweaked this for newlines
|
||||
exc.add_note(
|
||||
f'Controller events:\n'
|
||||
f'{monitor.events!r}\n'
|
||||
f'{output}'
|
||||
)
|
||||
raise
|
||||
finally:
|
||||
# Failure-only rescue: retained test-owned PIDs, never a
|
||||
# global reaper, process group, or unrelated resource sweep.
|
||||
owned: list[psutil.Process] = (
|
||||
[monitor.child] if monitor.child else []
|
||||
)
|
||||
if proc.poll() is None:
|
||||
# Include children if root stalled before its PID event.
|
||||
with suppress(psutil.NoSuchProcess):
|
||||
owned.extend(
|
||||
psutil.Process(proc.pid).children(
|
||||
recursive=True,
|
||||
)
|
||||
)
|
||||
|
||||
for descendant in set(owned):
|
||||
with suppress(psutil.NoSuchProcess):
|
||||
if descendant.is_running():
|
||||
descendant.kill()
|
||||
|
||||
if proc.poll() is None:
|
||||
proc.send_signal(signal.SIGINT)
|
||||
try:
|
||||
proc.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait(timeout=3)
|
||||
assert proc.stdout is not None
|
||||
proc.stdout.close()
|
||||
|
||||
_, survivors = psutil.wait_procs(owned, timeout=3)
|
||||
assert not survivors, (
|
||||
'Watchdog cleanup survivors', survivors,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
sys.platform != 'linux',
|
||||
reason='Linux watchdog',
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
'mode',
|
||||
[
|
||||
# Baseline: both context and subactor ``Portal`` IPC work.
|
||||
'healthy_ipc',
|
||||
# Stall directions on the separate OOB context ``Channel``.
|
||||
'stalled_context_receive',
|
||||
'stalled_context_send',
|
||||
'stalled_context_send_and_receive',
|
||||
# Park ``Context.cancel()`` on that OOB context ``Channel``.
|
||||
'stalled_context_cancel_send',
|
||||
# Both context directions plus subactor ``Portal`` cancel.
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
],
|
||||
)
|
||||
def test_stalled_channel_sigint(
|
||||
mode: str,
|
||||
# ``pytest`` provides ``record_property`` via _pytest.junitxml.
|
||||
record_property: Callable[[str, object], None],
|
||||
) -> None:
|
||||
'''
|
||||
SIGINT must tear down a root with stalled child IPC.
|
||||
|
||||
The example's ``StallableReceiveStream.receive_exactly()`` sets
|
||||
``StallableReceiveStream.parked`` before waiting on
|
||||
``StallableReceiveStream.release``. Peer responses complete any
|
||||
pending read inside the original ``BufferedReceiveStream``.
|
||||
The reader task's next header or payload read accesses the
|
||||
replaced ``MsgpackTransport.recv_stream`` and parks on that
|
||||
event; this does not flush buffered bytes.
|
||||
``StallableTransport.send()`` sets
|
||||
``StallableTransport.send_parked`` before waiting on
|
||||
``StallableTransport.release_send``.
|
||||
|
||||
In ``examples.advanced_faults.ipc_stalls.root_main()``, the root
|
||||
actor's ``ready()`` closure waits for selected parked-operation
|
||||
events and emits ready JSON on stdout. ``RootMonitor`` checks
|
||||
that record before delivering SIGINT; readiness does not depend
|
||||
on communicating over a root ``Portal``.
|
||||
|
||||
CASES: (parametrized by ``mode``)
|
||||
|
||||
- Subactor ``Portal`` usable: ``staller_ptl`` comes from
|
||||
``an.start_actor()``. The distinct OOB context ``peer_ptl``
|
||||
uses a ``Channel`` from ``_connect_chan()`` and
|
||||
``open_portal()``.
|
||||
These five cases allow cooperative child cancellation:
|
||||
|
||||
- ``healthy_ipc``: probe traffic, with no stalled operations.
|
||||
- ``stalled_context_receive``: park context packet reads.
|
||||
- ``stalled_context_send``: park a context send.
|
||||
- ``stalled_context_send_and_receive``: park both directions.
|
||||
- ``stalled_context_cancel_send``: park the finite-deadline
|
||||
``Context.cancel()`` send during actual teardown.
|
||||
|
||||
- Subactor ``Portal`` stalled:
|
||||
|
||||
- ``stalled_context_and_subactor_cancel_send``: park both
|
||||
context directions AND ``staller_ptl``'s finite-deadline
|
||||
``Start(self.cancel)`` send. Require deadline expiration,
|
||||
runtime hard-reap logs, SIGKILL exit and completed waitpid.
|
||||
|
||||
``open_root_monitor()`` observes OS lifetime outside the actor
|
||||
tree. A ``Portal`` within the root cannot independently enforce
|
||||
a watchdog deadline or prove OS exit over stalled IPC. Missing
|
||||
readiness, premature completion, root timeout or an unreaped
|
||||
child fails this test; watchdog rescue cannot count as success.
|
||||
Kernel TCP half-open/DROP scenarios remain follow-up work.
|
||||
|
||||
'''
|
||||
signalled: bool = False
|
||||
result: dict[str, Any]|None = None
|
||||
stalls: dict[str, bool]|None = None
|
||||
with open_root_monitor(mode=mode) as monitor:
|
||||
proc: subprocess.Popen[bytes] = monitor.proc
|
||||
for event in monitor.iter_events():
|
||||
if event['event'] == 'child':
|
||||
# Retain one OS identity and verify root ownership;
|
||||
# later exit checks must concern this exact child.
|
||||
assert monitor.child is None
|
||||
monitor.child = psutil.Process(event['pid'])
|
||||
assert monitor.child.ppid() == proc.pid
|
||||
|
||||
elif event['event'] == 'ready':
|
||||
# Prove the child is live and selected operations
|
||||
# really parked before sending the sole SIGINT.
|
||||
assert not signalled
|
||||
child = monitor.child
|
||||
assert child is not None
|
||||
assert child.is_running()
|
||||
stalls = event['stalls']
|
||||
assert stalls['receive'] == (
|
||||
mode in {
|
||||
'stalled_context_receive',
|
||||
'stalled_context_send_and_receive',
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
}
|
||||
)
|
||||
assert stalls['send'] == (
|
||||
mode in {
|
||||
'stalled_context_send',
|
||||
'stalled_context_send_and_receive',
|
||||
'stalled_context_cancel_send',
|
||||
'stalled_context_and_subactor_cancel_send',
|
||||
}
|
||||
)
|
||||
assert stalls['finite_send'] == (
|
||||
mode == 'stalled_context_cancel_send'
|
||||
)
|
||||
if mode == (
|
||||
'stalled_context_and_subactor_cancel_send'
|
||||
):
|
||||
assert event['control'] == {
|
||||
'receive': False,
|
||||
'send': True,
|
||||
'finite_send': True,
|
||||
}
|
||||
assert proc.poll() is None
|
||||
monitor.signal_root()
|
||||
signalled = True
|
||||
|
||||
elif event['event'] == 'result':
|
||||
# Accept exactly one shutdown result, only after
|
||||
# SIGINT; voluntary early completion is a failure.
|
||||
assert signalled
|
||||
assert result is None
|
||||
result = event
|
||||
|
||||
elif event['event'] == 'control_send_exit':
|
||||
# Only this case parks the subactor ``Portal`` send.
|
||||
# Its finite-budget exit must ultimately time out;
|
||||
# the final checks below require that evidence.
|
||||
assert mode == (
|
||||
'stalled_context_and_subactor_cancel_send'
|
||||
)
|
||||
|
||||
else:
|
||||
raise AssertionError(event)
|
||||
|
||||
# The root caught SIGINT and reported runtime waitpid proof;
|
||||
# these checks precede any failure-only watchdog rescue.
|
||||
assert proc.wait(timeout=2) == 0
|
||||
assert signalled, 'Root completed before signal readiness'
|
||||
assert result is not None
|
||||
assert stalls is not None
|
||||
child = monitor.child
|
||||
assert child is not None
|
||||
assert result['root_exit'] == 'KeyboardInterrupt'
|
||||
assert result['child_reaped'] is True
|
||||
control_expired: bool = any(
|
||||
event.get('deadline_expired', False)
|
||||
for event in monitor.events
|
||||
)
|
||||
if mode == 'stalled_context_and_subactor_cancel_send':
|
||||
# Both context directions and the subactor ``Portal``
|
||||
# cancel send stalled: require actual budget expiry,
|
||||
# SIGKILL status and the runtime's hard-reap log.
|
||||
assert control_expired
|
||||
assert result['child_returncode'] == -signal.SIGKILL
|
||||
assert result['control_fault_active'] == {
|
||||
'receive': False,
|
||||
'send': True,
|
||||
'finite_send': True,
|
||||
}
|
||||
assert any(
|
||||
'escalating to `proc.kill()` (hard-reap)' in line
|
||||
for line in monitor.transcript
|
||||
), 'Missing runtime hard-reap evidence'
|
||||
else:
|
||||
# The baseline and context-only faults retain a usable
|
||||
# subactor ``Portal``: cooperative child exit is clean.
|
||||
assert result['child_returncode'] == 0
|
||||
assert result['stalls_fault_active'] == stalls
|
||||
assert not child.is_running()
|
||||
record_property('ipc_stall_result', json.dumps({
|
||||
**result,
|
||||
'mode': mode,
|
||||
'root_returncode': proc.returncode,
|
||||
'watchdog_forced_cleanup': False,
|
||||
'runtime_hard_reap': mode == (
|
||||
'stalled_context_and_subactor_cancel_send'
|
||||
),
|
||||
'control_deadline_expired': control_expired,
|
||||
}))
|
||||
Loading…
Reference in New Issue