Test `trio`-side channel exit in an infected child
Exercise the exit-signal handling fixed in 4dfcf6f6 using the real
`to_asyncio.open_channel_from()` API in an infected child actor.
An expected `TrioTaskExited` must not escape graceful teardown or
be raised again as an unhandled asyncio done-callback exception.
Deats,
- run six cases with native and Greenback-wrapped asyncio tasks:
direct channel closure, last cached caller exit/cancellation,
strict TTE reporting, and distinct RuntimeError/TTE cleanup faults;
- use `block_until_trio_side_exit()` to record task identity,
complete startup, send one channel value and block until signalled;
- wrap the channel API in `maybe_open_context()` for cached cases,
proving overlapping `trio.Task`s receive the same channel;
- synchronize the second subscription with `tn.start()` and an
event, then join its task before checking asyncio is still alive;
- open two asyncio tasks sequentially, checking each finishes when
its channel closes and retains the expected exception identity;
- raise a NEW TTE during cleanup to reject blanket type-based
suppression, retaining the issued TTE as the new error's cause;
- drain a host-loop callback before checking unhandled exceptions
for expected shutdown, separately from actor/RPC usability;
- await two sequential `Portal.run()` calls in the same child,
then explicitly cancel the actor before leaving its nursery.
Keep the Piker reproduction as context rather than describing the
blocking asyncio task as a quote/feed implementation. Document case
selection, closure contracts and the assertions at their call sites.
Qualification: latest matrix 12 passed. Earlier combined asyncio
and task-formatter run: 41 passed / 18 deselected. This matrix does
not yet cover an infected root or an asyncio task suppressing TTE
and returning or remaining blocked; those are follow-up coverage.
(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
wkt/cached_aio_owner_exit
parent
4dfcf6f660
commit
540dd2bf73
|
|
@ -0,0 +1,405 @@
|
|||
'''
|
||||
Inter-loop channel teardown must handle the issued exit signal.
|
||||
|
||||
'''
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
import pytest
|
||||
import tractor
|
||||
from tractor import (
|
||||
to_asyncio,
|
||||
trionics,
|
||||
)
|
||||
import trio
|
||||
|
||||
|
||||
_num_tasks: int = 2
|
||||
_rpc_repetitions: int = 2
|
||||
_actor_test_timeout_s: float = 10
|
||||
|
||||
|
||||
async def exercise_trio_side_exit(
|
||||
cached: bool,
|
||||
cancelled: bool,
|
||||
suppress: bool,
|
||||
expect_aio_side_err_type: str|None,
|
||||
greenback_shim: bool,
|
||||
) -> None:
|
||||
'''
|
||||
Exit direct and cached inter-loop channels in an infected actor.
|
||||
|
||||
Failure mechanism:
|
||||
- The trio-side parent task exits with asyncio still running.
|
||||
- ``open_channel_from()`` signals the asyncio task to exit.
|
||||
- Previously, that signal escaped channel teardown and became
|
||||
an unhandled exception from the asyncio done callback.
|
||||
|
||||
Cases:
|
||||
- ``cached`` wraps the channel API in ``maybe_open_context()``.
|
||||
- ``cancelled`` cancels the last consumer at a Trio checkpoint.
|
||||
- ``suppress`` selects graceful or strict exit-signal reporting.
|
||||
- ``expect_aio_side_err_type`` is a class name or ``None``.
|
||||
Class names travel as RPC payloads; actual exception objects
|
||||
remain inside this actor for identity/cause assertions.
|
||||
- ``greenback_shim`` wraps the asyncio task without enabling PDB.
|
||||
|
||||
Ordering and proof:
|
||||
- ``tn.start()`` waits for a sibling subscription; ``release``
|
||||
lets it retire while the original subscriber remains alive.
|
||||
- The asyncio task must finish before channel-context exit ends.
|
||||
- Identity assertions distinguish our signal from cleanup faults.
|
||||
- Each of ``_num_tasks`` iterations opens a fresh asyncio task.
|
||||
- A host-loop callback sets ``drained`` before callback-failure
|
||||
inspection, independently of actor/RPC survival.
|
||||
|
||||
'''
|
||||
if expect_aio_side_err_type not in {
|
||||
None,
|
||||
'RuntimeError',
|
||||
'TrioTaskExited',
|
||||
}:
|
||||
raise ValueError(
|
||||
f'Unknown aio error case: {expect_aio_side_err_type!r}'
|
||||
)
|
||||
|
||||
# Keep the actual objects: equality/type checks cannot prove that
|
||||
# teardown distinguishes an issued signal from a new same-type
|
||||
# asyncio failure. Task/signal/failure entries follow iteration
|
||||
# order; callback_errors records host-loop exception contexts.
|
||||
callback_errors: list[dict] = []
|
||||
tasks: list[asyncio.Task] = []
|
||||
failures: list[Exception] = []
|
||||
signals: list[BaseException] = []
|
||||
loop: asyncio.AbstractEventLoop = asyncio.get_running_loop()
|
||||
previous_handler = loop.get_exception_handler()
|
||||
loop.set_exception_handler(
|
||||
lambda loop, context: callback_errors.append(context),
|
||||
)
|
||||
|
||||
async def block_until_trio_side_exit(
|
||||
chan: to_asyncio.LinkedTaskChannel,
|
||||
) -> None:
|
||||
'''
|
||||
Block until the channel API signals trio-side parent exit.
|
||||
|
||||
- Record task identity, finish startup, send a channel value,
|
||||
then await an event which is never set.
|
||||
- Record the received ``TrioTaskExited`` object and either
|
||||
re-raise it or raise the selected distinct cleanup failure.
|
||||
- ``tasks``, ``signals`` and ``failures`` retain the objects
|
||||
checked after teardown; ``greenback_shim`` selects whether
|
||||
this coroutine initializes a Greenback portal.
|
||||
|
||||
'''
|
||||
task: asyncio.Task|None = asyncio.current_task()
|
||||
assert task is not None
|
||||
tasks.append(task)
|
||||
if greenback_shim:
|
||||
import greenback
|
||||
await greenback.ensure_portal()
|
||||
|
||||
chan.started_nowait('ready')
|
||||
chan.send_nowait('tick')
|
||||
try:
|
||||
await asyncio.Event().wait()
|
||||
|
||||
except to_asyncio.TrioTaskExited as signal:
|
||||
signals.append(signal)
|
||||
|
||||
# Healthy cleanup re-raises the very signal issued by
|
||||
# open_channel_from(); suppression uses object identity.
|
||||
if expect_aio_side_err_type is None:
|
||||
raise
|
||||
|
||||
failure: RuntimeError|to_asyncio.TrioTaskExited
|
||||
if expect_aio_side_err_type == 'RuntimeError':
|
||||
# Fail during cleanup after receiving the channel
|
||||
# owner's exit notification.
|
||||
failure = RuntimeError(
|
||||
'asyncio cleanup failed after trio-side exit',
|
||||
)
|
||||
|
||||
elif expect_aio_side_err_type == 'TrioTaskExited':
|
||||
# Deliberately use the SAME TYPE as the issued signal
|
||||
# but a DIFFERENT INSTANCE. A blanket TrioTaskExited
|
||||
# handler would hide this distinct asyncio error.
|
||||
failure = to_asyncio.TrioTaskExited(
|
||||
'new asyncio TTE, not the issued exit signal',
|
||||
)
|
||||
|
||||
# Retain both objects: assertions below require the new
|
||||
# failure to propagate with the issued signal as cause.
|
||||
# "Independent" means distinct from the issued signal;
|
||||
# the cleanup failure still follows that notification.
|
||||
failures.append(failure)
|
||||
raise failure from signal
|
||||
|
||||
@acm
|
||||
async def open_raw_aio_chan() -> AsyncIterator[
|
||||
to_asyncio.LinkedTaskChannel
|
||||
]:
|
||||
'''
|
||||
Open the blocking asyncio task through open_channel_from().
|
||||
|
||||
- Run ``block_until_trio_side_exit`` using the actual
|
||||
``to_asyncio.open_channel_from()`` API.
|
||||
- Forward captured ``suppress`` unchanged so strict reporting
|
||||
and graceful suppression exercise the same ownership path.
|
||||
- Yield after the asyncio task's startup handshake. Context
|
||||
exit signals that task and waits for it to finish.
|
||||
|
||||
'''
|
||||
chan: to_asyncio.LinkedTaskChannel
|
||||
first: str
|
||||
async with to_asyncio.open_channel_from(
|
||||
block_until_trio_side_exit,
|
||||
suppress_graceful_exits=suppress,
|
||||
) as (chan, first):
|
||||
assert first == 'ready'
|
||||
yield chan
|
||||
|
||||
@acm
|
||||
async def open_maybe_cached_aio_chan() -> AsyncIterator[
|
||||
to_asyncio.LinkedTaskChannel
|
||||
]:
|
||||
'''
|
||||
Optionally cache the channel with ``maybe_open_context()``.
|
||||
|
||||
With captured ``cached=True``, overlapping calling trio.Tasks
|
||||
receive the same channel instance. Otherwise each call enters
|
||||
``open_raw_aio_chan()`` directly.
|
||||
|
||||
'''
|
||||
chan: to_asyncio.LinkedTaskChannel
|
||||
if cached:
|
||||
async with trionics.maybe_open_context(
|
||||
acm_func=open_raw_aio_chan,
|
||||
# Same factory/key means both calling trio.Tasks get
|
||||
# the same underlying channel. Reuse it on subsequent
|
||||
# iterations to check retirement and reallocation.
|
||||
key='aio_chan',
|
||||
) as (_, chan):
|
||||
yield chan
|
||||
|
||||
else:
|
||||
async with open_raw_aio_chan() as chan:
|
||||
yield chan
|
||||
|
||||
async def subscribe_cached_chan(
|
||||
expected: to_asyncio.LinkedTaskChannel,
|
||||
release: trio.Event,
|
||||
*,
|
||||
task_status: trio.TaskStatus[None] = (
|
||||
trio.TASK_STATUS_IGNORED
|
||||
),
|
||||
) -> None:
|
||||
'''
|
||||
Close one cached-channel subscription while another remains.
|
||||
|
||||
- Enter ``open_maybe_cached_aio_chan`` in another trio.Task.
|
||||
- Require ``expected`` identity to prove actual cache reuse.
|
||||
- Report started only once the broadcast subscription exists.
|
||||
- Wait for ``release`` before exiting; the caller joins this
|
||||
task before checking that the asyncio task is still alive.
|
||||
|
||||
'''
|
||||
other: to_asyncio.LinkedTaskChannel
|
||||
async with open_maybe_cached_aio_chan() as other:
|
||||
assert other is expected
|
||||
async with other.subscribe():
|
||||
task_status.started()
|
||||
await release.wait()
|
||||
|
||||
try:
|
||||
task_index: int
|
||||
for task_index in range(_num_tasks):
|
||||
caught: BaseException|None = None
|
||||
try:
|
||||
cs: trio.CancelScope
|
||||
chan: to_asyncio.LinkedTaskChannel
|
||||
bchan: trionics.BroadcastReceiver
|
||||
with trio.CancelScope() as cs:
|
||||
async with open_maybe_cached_aio_chan() as chan:
|
||||
async with chan.subscribe() as bchan:
|
||||
if cached:
|
||||
release: trio.Event = trio.Event()
|
||||
|
||||
tn: trio.Nursery
|
||||
async with (
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
await tn.start(
|
||||
subscribe_cached_chan,
|
||||
chan,
|
||||
release,
|
||||
)
|
||||
release.set()
|
||||
|
||||
# The sibling has closed its handles.
|
||||
# Our subscription still retains the
|
||||
# aio task; this is not final exit.
|
||||
assert not tasks[task_index].done()
|
||||
|
||||
# Consume a real broadcast value before
|
||||
# exit or last-consumer cancellation.
|
||||
assert await bchan.receive() == 'tick'
|
||||
if cancelled:
|
||||
cs.cancel()
|
||||
await trio.lowlevel.checkpoint()
|
||||
|
||||
except Exception as error:
|
||||
caught = error
|
||||
|
||||
assert tasks[task_index].done()
|
||||
assert len(signals) == task_index + 1, (
|
||||
caught,
|
||||
tasks[task_index].exception(),
|
||||
signals,
|
||||
)
|
||||
if expect_aio_side_err_type:
|
||||
# For TTE, matching the issued signal's type must NOT
|
||||
# suppress a distinct asyncio-side exception.
|
||||
# Keep its identity AND its original shutdown cause.
|
||||
assert caught is failures[task_index]
|
||||
assert caught.__cause__ is signals[task_index]
|
||||
|
||||
elif not suppress:
|
||||
# Strict reporting returns the exact issued signal,
|
||||
# not a new wrapper or an asyncio-created substitute.
|
||||
assert caught is signals[task_index]
|
||||
|
||||
else:
|
||||
# Default policy absorbs expected owner exit, even
|
||||
# when the cached resource's last consumer cancels.
|
||||
assert caught is None, repr(caught)
|
||||
|
||||
# Cross the host loop's ready queue before accepting callback
|
||||
# cleanliness, independently of actor survival.
|
||||
drained: trio.Event = trio.Event()
|
||||
loop.call_soon(drained.set)
|
||||
await drained.wait()
|
||||
if not expect_aio_side_err_type:
|
||||
# Expected shutdown must have no unhandled callbacks.
|
||||
# Fault cases above test propagation and causes, without
|
||||
# asserting a broader callback-cleanliness policy.
|
||||
assert not callback_errors, callback_errors
|
||||
finally:
|
||||
loop.set_exception_handler(previous_handler)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'greenback_shim',
|
||||
[False, True],
|
||||
ids=['native', 'greenback'],
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
'cached,cancelled,suppress,expect_aio_side_err_type',
|
||||
[
|
||||
# Expected TTE ack: direct use, last cached caller
|
||||
# exit, and cancellation of that last cached caller.
|
||||
pytest.param(
|
||||
False, False, True, None,
|
||||
id='direct-close',
|
||||
),
|
||||
pytest.param(
|
||||
True, False, True, None,
|
||||
id='last-cache-user',
|
||||
),
|
||||
pytest.param(
|
||||
True, True, True, None,
|
||||
id='cancelled-last-cache-user',
|
||||
),
|
||||
# Strict reporting must preserve the issued TTE's identity.
|
||||
pytest.param(
|
||||
True, False, False, None,
|
||||
id='explicit-exit-reporting',
|
||||
),
|
||||
# New asyncio-side cleanup failures must escape suppression,
|
||||
# even when the exception class matches the issued signal.
|
||||
pytest.param(
|
||||
True, False, True, 'RuntimeError',
|
||||
id='independent-cleanup-error',
|
||||
),
|
||||
pytest.param(
|
||||
True, False, True, 'TrioTaskExited',
|
||||
id='independent-same-type-error',
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_channel_owner_exit(
|
||||
reg_addr: tuple[str, int],
|
||||
cached: bool,
|
||||
cancelled: bool,
|
||||
suppress: bool,
|
||||
expect_aio_side_err_type: str|None,
|
||||
greenback_shim: bool,
|
||||
) -> None:
|
||||
'''
|
||||
Channel teardown must leave the infected child actor RPC-capable.
|
||||
|
||||
Regression:
|
||||
- ``open_channel_from()`` signalled the running asyncio task when
|
||||
its trio-side parent task left the channel context.
|
||||
- Raising that issued ``TrioTaskExited`` escaped default graceful
|
||||
suppression and was also raised from the asyncio done callback.
|
||||
|
||||
Proof boundary:
|
||||
- Real TCP RPC invokes ``exercise_trio_side_exit`` in an infected
|
||||
child, using the channel API and its broadcast subscriptions.
|
||||
Cached cases additionally wrap it in ``maybe_open_context()``.
|
||||
- Repeated RPCs in that same actor prove it remains usable after
|
||||
the endpoint's asyncio tasks have finished.
|
||||
- Strict reporting and independent faults retain their identity;
|
||||
broad exception-type suppression cannot satisfy the matrix.
|
||||
- The deadline bounds hangs; no live broker or debugger is used.
|
||||
|
||||
Downstream reproduction:
|
||||
Piker encountered this during quote-task restart after a farm
|
||||
reset, using a cached channel. This test needs neither that
|
||||
application nor caching in its direct-channel case.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
'''
|
||||
Own the infected child and bound the complete RPC exercise.
|
||||
|
||||
- Capture ``reg_addr`` for the test's isolated TCP registry.
|
||||
- Forward the selected case to each RPC in the same child.
|
||||
- A callback could fail after the RPC body finishes and then
|
||||
terminate the actor. A second RPC checks continued use;
|
||||
the callback-drain assertion inside the endpoint separately
|
||||
checks the failure directly, without relying on RPC timing.
|
||||
- Explicitly cancel the child after the probes: a successful
|
||||
RPC does not itself end the actor's service lifetime.
|
||||
|
||||
'''
|
||||
an: tractor.ActorNursery
|
||||
with trio.fail_after(_actor_test_timeout_s):
|
||||
async with tractor.open_nursery(
|
||||
registry_addrs=[reg_addr],
|
||||
) as an:
|
||||
portal: tractor.Portal = await an.start_actor(
|
||||
'aio_channel_owner',
|
||||
enable_modules=[__name__],
|
||||
infect_asyncio=True,
|
||||
)
|
||||
# portal.run() waits for this remote task's final
|
||||
# result. These RPCs are sequential, not concurrent;
|
||||
# only the endpoint's subscription tasks overlap.
|
||||
for _ in range(_rpc_repetitions):
|
||||
await portal.run(
|
||||
exercise_trio_side_exit,
|
||||
cached=cached,
|
||||
cancelled=cancelled,
|
||||
suppress=suppress,
|
||||
expect_aio_side_err_type=(
|
||||
expect_aio_side_err_type
|
||||
),
|
||||
greenback_shim=greenback_shim,
|
||||
)
|
||||
|
||||
await portal.cancel_actor()
|
||||
|
||||
trio.run(main)
|
||||
Loading…
Reference in New Issue