Reproduce `@context` final-`Return` disconnect cascade
A chart disconnect can leave its context endpoint returning over a closed channel. Exercise the real `_rpc._invoke()` dispatcher with loopback TCP and a sibling service to isolate the resulting `TransportClosed` nursery cancellation. Deats, - Order caller close, callee EOF and endpoint completion with events instead of timing sleeps or injected transport errors. - Keep an ordinary-RPC control for existing response hardening. - Capture the collateral task, actor/PID, nursery, cancel scope and cancellation source in a test-generated exception note. - Record the live IB diagnosis and generated-test provenance. The ordinary-RPC control passes; the context case remains red with `ClosedResourceError` wrapped as `TransportClosed`. This commit establishes the regression before runtime remediation. Prompt-IO: ai/prompt-io/20261019T140536Z_9a0f220f_final_return_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-6.1-sol` (`openai`))piker_pin
parent
9a0f220fd5
commit
0553fcf88c
|
|
@ -0,0 +1,71 @@
|
|||
---
|
||||
created_at: 2026-10-19T14:05:36Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 9a0f220f
|
||||
scope: tests
|
||||
substantive: true
|
||||
raw_file: 20261019T140536Z_9a0f220f_final_return_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported chart Ctrl-C causing search's final Return delivery
|
||||
to raise TransportClosed in datad.ib, cancelling data services and then
|
||||
pikerd's trading/sampler services. They asked whether the Piker branch
|
||||
ib_connectivity_harden already remediates it and authorized a minimal
|
||||
Tractor sibling-survival reproducer if no existing remediation applies:
|
||||
|
||||
> ok let's do it if we find no other redmediations per above.
|
||||
|
||||
The human explicitly selected capture of this test-only provenance.
|
||||
|
||||
## Response summary
|
||||
|
||||
Compared ib_connectivity_harden with the committed evaluation baseline:
|
||||
efa72980's IB socket-disconnect propagation is already present; the
|
||||
missing d126302f/c0dae89a changes concern idle arithmetic and sampler
|
||||
cancellation-state access. Tractor ebf2258b hardens ordinary RPC response
|
||||
sends, but the context final Return remains exposed. Inspected the
|
||||
additional ipc_fault_regressions branch: its EOF/context-status changes
|
||||
do not guard this final-result send.
|
||||
|
||||
Added a focused real-TCP, same-process dispatcher regression with an
|
||||
ordinary RPC control and a context case. Explicit peer-close/EOF events
|
||||
force the failure ordering. A sibling request/response checks service
|
||||
survival; the failure path confirms premature sibling cancellation.
|
||||
|
||||
## Files changed
|
||||
|
||||
- tests/test_context_final_return_disconnect.py: reproducer and control.
|
||||
- This provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
- Ordinary RPC control passed.
|
||||
- Context case failed with the reported TransportClosed caused by
|
||||
ClosedResourceError, confirming collateral nursery cancellation.
|
||||
- Final two-case run: 1 failed, 1 passed in 0.07s.
|
||||
- Ruff passed; git diff --check passed.
|
||||
- Test uses isolated local TCP, no broker, feed, Qt or asyncio bridge.
|
||||
- Root actor and channels exit through their context managers.
|
||||
- No production fix, staging, commit or ref changes.
|
||||
|
||||
## Human edits
|
||||
|
||||
None observed in the generated test at capture time.
|
||||
|
||||
## Follow-up: identify the collateral sibling
|
||||
|
||||
The human requested the task repr and runtime metadata behind the
|
||||
test-generated exception note. The cancellation handler now captures
|
||||
the sibling task, Actor.aid (including PID), parent nursery, nursery
|
||||
cancel scope and original trio.Cancelled repr while still unwinding.
|
||||
The test attaches that snapshot to the escaping TransportClosed;
|
||||
Tractor production reporting and exception handling are unchanged.
|
||||
|
||||
The updated two-case reproducer again yielded 1 passed, 1 failed in
|
||||
0.07s, with the sibling identity and source invoke task in the note.
|
||||
Ruff and git diff --check passed after the diagnostic change.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
created_at: 2026-10-19T14:05:36Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 9a0f220f
|
||||
diff_cmd: git diff --no-index /dev/null tests/test_context_final_return_disconnect.py
|
||||
---
|
||||
|
||||
## Captured output
|
||||
|
||||
`ib_connectivity_harden` has no fix for this path. Its IB socket
|
||||
disconnect handling is already in our evaluation; the two missing
|
||||
commits fix idle-timer arithmetic and sampler cancellation-state
|
||||
access.
|
||||
|
||||
I also checked Tractor’s related fault work. The existing
|
||||
response-send fix covers ordinary RPCs, but leaves the `@context`
|
||||
final-result send exposed. I’m building the reproducer in the
|
||||
evaluated Tractor worktree, with isolated local IPC.
|
||||
|
||||
The reproducer hits the same failure with real loopback TCP:
|
||||
**ordinary RPC passes; `@context` fails**, and the unrelated
|
||||
sibling is confirmed cancelled before it can answer a request.
|
||||
The two cases finish in 0.07 seconds, without IB, Qt or asyncio.
|
||||
|
||||
## Generated test
|
||||
|
||||
> `git diff --no-index /dev/null tests/test_context_final_return_disconnect.py`
|
||||
|
||||
The file is currently untracked. The pointer above displays the entire
|
||||
generated artifact without duplicating it in this record.
|
||||
|
||||
finish_after_disconnect() is a real Tractor context endpoint that
|
||||
publishes readiness and returns after an event. ordinary_result()
|
||||
provides a non-context RPC control. The parameterized test invokes the
|
||||
actual dispatcher with a real root actor and real loopback TCP channels.
|
||||
It drains startup messages, closes the caller, observes callee EOF,
|
||||
closes the callee channel, then releases the endpoint. A nursery sibling
|
||||
must answer a subsequent request. On the observed TransportClosed it
|
||||
also verifies collateral sibling cancellation and re-raises the fault.
|
||||
|
||||
The explicit event ordering isolates final-result delivery, without
|
||||
mocked exceptions, sleeps, provider connections or process supervision.
|
||||
|
||||
## Final verification
|
||||
|
||||
Command environment: evaluation Piker py313/bin/python (Python 3.13.14,
|
||||
pytest 9.0.2), importing Tractor from address_logging_levels at 9a0f220f.
|
||||
VIRTUAL_ENV was removed; TRACTOR_SPAWN_METHOD and TRACTOR_LOGLEVEL unset.
|
||||
LD_LIBRARY_PATH included the existing Nix GCC runtime library.
|
||||
|
||||
`python -m pytest tests/test_context_final_return_disconnect.py --tpt-proto=tcp --tb=short`
|
||||
|
||||
Result: 1 failed, 1 passed in 0.07s. Diagnostic --tb=short was used
|
||||
after the ordinary full-trace run confirmed the dispatcher failure.
|
||||
The context case raises TransportClosed caused by ClosedResourceError
|
||||
at MsgpackTransport.send(), with the note:
|
||||
The unrelated sibling was cancelled before it could answer the post-disconnect request.
|
||||
|
||||
`ruff check tests/test_context_final_return_disconnect.py`
|
||||
|
||||
All checks passed!
|
||||
|
||||
`git diff --check` passed. No production runtime edits were made.
|
||||
|
|
@ -0,0 +1,227 @@
|
|||
'''
|
||||
Isolate final-result delivery from sibling service lifetimes.
|
||||
|
||||
'''
|
||||
from uuid import uuid4
|
||||
|
||||
import msgspec
|
||||
import pytest
|
||||
import tractor
|
||||
import trio
|
||||
|
||||
from tractor.ipc import Channel
|
||||
from tractor.msg import Aid, StartAck, Started
|
||||
from tractor.runtime import _rpc
|
||||
|
||||
|
||||
@tractor.context
|
||||
async def finish_after_disconnect(
|
||||
ctx: tractor.Context,
|
||||
peer_closed: trio.Event,
|
||||
endpoint_finished: trio.Event,
|
||||
|
||||
) -> str:
|
||||
'''
|
||||
Finish successfully only after the caller connection is closed.
|
||||
|
||||
'''
|
||||
await ctx.started('ready')
|
||||
await peer_closed.wait()
|
||||
endpoint_finished.set()
|
||||
return 'finished'
|
||||
|
||||
|
||||
async def ordinary_result(
|
||||
peer_closed: trio.Event,
|
||||
endpoint_finished: trio.Event,
|
||||
|
||||
) -> str:
|
||||
'''
|
||||
Exercise the already-hardened ordinary RPC final-result path.
|
||||
|
||||
'''
|
||||
await peer_closed.wait()
|
||||
endpoint_finished.set()
|
||||
return 'finished'
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'context_endpoint',
|
||||
[False, True],
|
||||
ids=['ordinary-rpc', 'context'],
|
||||
)
|
||||
def test_final_return_disconnect_preserves_sibling(
|
||||
reg_addr: tuple,
|
||||
context_endpoint: bool,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
A departing caller must not cancel an unrelated service task.
|
||||
|
||||
Chart Ctrl-C made open_symbol_search finish before Tractor sent
|
||||
its final Return. TransportClosed from that send escaped _invoke
|
||||
and cancelled the shared service nursery, including IB feeds.
|
||||
|
||||
Use real loopback TCP channels and a real Actor/Context, invoking
|
||||
_invoke directly to control the precise response-send ordering.
|
||||
The caller consumes startup messages, closes its socket, and the
|
||||
callee observes EOF and closes its channel before releasing the
|
||||
successful endpoint. No transport exception is mocked or injected.
|
||||
|
||||
After _invoke completes, a sibling must still answer a request.
|
||||
The ordinary-RPC case is a control for the existing response-send
|
||||
hardening. The context case exposes the uncovered final Return.
|
||||
Capture the sibling's task, actor, nursery and cancellation reprs
|
||||
while unwinding so the failure identifies the collateral victim
|
||||
and the cancellation source even after the runtime has exited.
|
||||
This isolates response delivery; it does not exercise discovery,
|
||||
process supervision, the asyncio bridge, or Qt cancellation.
|
||||
|
||||
'''
|
||||
collateral_cancellations: list[tuple[trio.Cancelled, str]] = []
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
Order peer EOF before endpoint completion without sleeps.
|
||||
|
||||
'''
|
||||
peer_closed = trio.Event()
|
||||
endpoint_finished = trio.Event()
|
||||
invoke_finished = trio.Event()
|
||||
ping = trio.Event()
|
||||
pong = trio.Event()
|
||||
sibling_stopped = trio.Event()
|
||||
|
||||
async def sibling(
|
||||
task_status: trio.TaskStatus = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
'''
|
||||
Answer a post-disconnect request until explicitly stopped.
|
||||
|
||||
'''
|
||||
task_status.started()
|
||||
try:
|
||||
await ping.wait()
|
||||
pong.set()
|
||||
await trio.sleep_forever()
|
||||
except trio.Cancelled as exc:
|
||||
if not pong.is_set():
|
||||
assert endpoint_finished.is_set()
|
||||
task: trio.lowlevel.Task = (
|
||||
trio.lowlevel.current_task()
|
||||
)
|
||||
aid: Aid = actor.aid
|
||||
nursery: trio.Nursery = task.parent_nursery
|
||||
cancel_scope: trio.CancelScope = (
|
||||
nursery.cancel_scope
|
||||
)
|
||||
runtime_info: str = (
|
||||
f'\nSibling task: {task!r}\n'
|
||||
f'Actor identity: {aid!r}\n'
|
||||
f'Parent nursery: {nursery!r}\n'
|
||||
f'Parent cancel scope: {cancel_scope!r}\n'
|
||||
f'Cancellation: {exc!r}'
|
||||
)
|
||||
collateral_cancellations.append(
|
||||
(exc, runtime_info),
|
||||
)
|
||||
raise
|
||||
finally:
|
||||
sibling_stopped.set()
|
||||
|
||||
with trio.fail_after(10):
|
||||
async with tractor.open_root_actor(
|
||||
registry_addrs=[reg_addr],
|
||||
enable_transports=['tcp'],
|
||||
loglevel='error',
|
||||
) as actor:
|
||||
listeners = await trio.open_tcp_listeners(
|
||||
0,
|
||||
host='127.0.0.1',
|
||||
)
|
||||
[listener] = listeners
|
||||
async with listener:
|
||||
port: int = listener.socket.getsockname()[1]
|
||||
caller_stream = await trio.open_tcp_stream(
|
||||
'127.0.0.1',
|
||||
port,
|
||||
)
|
||||
callee_stream = await listener.accept()
|
||||
|
||||
async with caller_stream, callee_stream:
|
||||
caller = Channel.from_stream(caller_stream)
|
||||
callee = Channel.from_stream(callee_stream)
|
||||
caller._aid = actor.aid
|
||||
callee._aid = Aid(
|
||||
name='departing-chart',
|
||||
uuid=str(uuid4()),
|
||||
)
|
||||
cid: str = str(uuid4())
|
||||
endpoint = (
|
||||
finish_after_disconnect
|
||||
if context_endpoint else ordinary_result
|
||||
)
|
||||
kwargs = {
|
||||
'peer_closed': peer_closed,
|
||||
'endpoint_finished': endpoint_finished,
|
||||
}
|
||||
|
||||
async def invoke(
|
||||
task_status: trio.TaskStatus,
|
||||
) -> None:
|
||||
'''
|
||||
Run the actual RPC dispatcher to completion.
|
||||
|
||||
'''
|
||||
await _rpc._invoke(
|
||||
actor,
|
||||
cid,
|
||||
callee,
|
||||
endpoint,
|
||||
kwargs,
|
||||
task_status=task_status,
|
||||
)
|
||||
invoke_finished.set()
|
||||
|
||||
async with trio.open_nursery() as services:
|
||||
await services.start(sibling)
|
||||
ctx = await services.start(invoke)
|
||||
ack = await caller.recv()
|
||||
assert isinstance(ack, StartAck)
|
||||
assert ack.cid == cid
|
||||
if context_endpoint:
|
||||
started = await caller.recv()
|
||||
assert isinstance(started, Started)
|
||||
assert started.cid == cid
|
||||
assert msgspec.msgpack.decode(
|
||||
started.pld,
|
||||
) == 'ready'
|
||||
|
||||
await caller.aclose()
|
||||
with pytest.raises(tractor.TransportClosed):
|
||||
await callee.recv()
|
||||
await callee.aclose()
|
||||
peer_closed.set()
|
||||
|
||||
await invoke_finished.wait()
|
||||
assert endpoint_finished.is_set()
|
||||
assert not sibling_stopped.is_set()
|
||||
assert (callee, cid) not in actor._rpc_tasks
|
||||
ping.set()
|
||||
await pong.wait()
|
||||
if context_endpoint:
|
||||
assert ctx._result == 'finished'
|
||||
services.cancel_scope.cancel()
|
||||
|
||||
try:
|
||||
trio.run(main)
|
||||
except tractor.TransportClosed as exc:
|
||||
assert collateral_cancellations
|
||||
cancelled, runtime_info = collateral_cancellations[0]
|
||||
assert 'TransportClosed' in str(cancelled)
|
||||
exc.add_note(
|
||||
'The unrelated sibling was cancelled before it could '
|
||||
'answer the post-disconnect request.'
|
||||
+ runtime_info
|
||||
)
|
||||
raise
|
||||
Loading…
Reference in New Issue