Contain `@context` final-result disconnects
A caller can depart after its endpoint finishes but before the final `Return` reaches it. Letting that `TransportClosed` escape `_invoke()` cancels unrelated RPCs in the actor's shared service nursery. Guard only the final result send and report the actor, context, scope, channel and transport error using existing primitive renderers. Keep endpoint execution and nursery teardown outside that suppression. Extend the real-TCP reproducer with a genuine endpoint-raised `TransportClosed` identity control. Snapshot the endpoint, sibling and nursery owner through the shared task formatter when cancellation exposes a collateral failure. Prompt-IO: ai/prompt-io/20261019T145431Z_0553fcf8_ctx_return_fix_prompt_io.md Prompt-IO: ai/prompt-io/20261020T193619Z_0553fcf8_review_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-6.1-sol` (`openai`))piker_pin
parent
bf2bb1c9a6
commit
b64c89f05a
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
created_at: 2026-10-19T14:54:31Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 0553fcf8
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261019T145431Z_0553fcf8_ctx_return_fix_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
After committing the isolated regression on
|
||||
wkt/ctx_final_return_disconnect, the human asked:
|
||||
|
||||
> committed, ok now a patch?
|
||||
|
||||
Earlier guidance requested runtime diagnostic information at the
|
||||
production detection point, alongside the test's sibling metadata.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added a narrow final-result delivery guard in _rpc._invoke(). A
|
||||
TransportClosed while sending a successful context endpoint's final
|
||||
Return now produces a metadata-rich warning rather than cancelling
|
||||
the actor's shared service nursery. Actual endpoint transport errors
|
||||
and cancellation retain the existing propagation path.
|
||||
|
||||
Extended the reproducer with a genuine endpoint-raised TransportClosed
|
||||
control and an identity assertion, preventing broad suppression of
|
||||
application failures. The formerly red context delivery case passes.
|
||||
|
||||
## Files changed
|
||||
|
||||
- tractor/runtime/_rpc.py: guard final response delivery and report
|
||||
actor/task/context/nursery/channel metadata on peer disconnect.
|
||||
- tests/test_context_final_return_disconnect.py: endpoint-error control.
|
||||
- This provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
Ran from the Tractor worktree at committed base 0553fcf8 using
|
||||
/home/goodboy/repos/piker/wkts/ib_downstream_pin_eval/py313/bin/python.
|
||||
VIRTUAL_ENV was removed and LD_LIBRARY_PATH included the existing
|
||||
Nix GCC runtime library; tests select isolated TCP addresses.
|
||||
|
||||
- pytest tests/test_context_final_return_disconnect.py tests/test_rpc.py
|
||||
--tpt-proto=tcp: 10 passed in 0.93s.
|
||||
- pytest tests/test_context_stream_semantics.py --tpt-proto=tcp:
|
||||
65 passed in 59.80s.
|
||||
- ruff check tractor/runtime/_rpc.py
|
||||
tests/test_context_final_return_disconnect.py: passed.
|
||||
- git diff --check: passed.
|
||||
- No live broker/chart trial was run by the agent.
|
||||
- All patch changes remain unstaged; no commit or ref move.
|
||||
|
||||
## Human edits
|
||||
|
||||
None observed in this patch at capture time.
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
created_at: 2026-10-19T14:54:31Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 0553fcf8
|
||||
diff_cmd: git diff HEAD -- tractor/runtime/_rpc.py tests/test_context_final_return_disconnect.py
|
||||
---
|
||||
|
||||
## Captured response
|
||||
|
||||
I’ll patch the context final-result delivery path, then run the
|
||||
reproducer and the related RPC/context tests.
|
||||
|
||||
## Generated patch
|
||||
|
||||
> `git diff HEAD -- tractor/runtime/_rpc.py`
|
||||
|
||||
Guard only chan.send(return_msg) after the context endpoint returns.
|
||||
Catch normalized TransportClosed and warn with actor identity, task,
|
||||
RPC/context ID, nursery, cancel scope, channel and transport error.
|
||||
Keep endpoint execution and nursery-exit errors outside the guard.
|
||||
|
||||
> `git diff HEAD -- tests/test_context_final_return_disconnect.py`
|
||||
|
||||
Extend the real-TCP reproducer with an endpoint-raised TransportClosed
|
||||
control. Verify its identity survives propagation and that swallowing
|
||||
it would fail the test. Preserve the successful ordinary/context
|
||||
cases' sibling responsiveness and RPC task cleanup assertions.
|
||||
|
||||
## Captured verification response
|
||||
|
||||
The patch turns the reproducer green. It catches
|
||||
`TransportClosed` only around the final-result send and logs the
|
||||
actor, task, RPC/context ID, nursery, cancel scope and channel.
|
||||
|
||||
An added control confirms that an endpoint’s own `TransportClosed`
|
||||
still propagates with its original identity.
|
||||
|
||||
Checks passed: **10 RPC/reproducer cases + 65 context/stream
|
||||
cases**, plus Ruff and whitespace checks.
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
created_at: 2026-10-20T19:36:19Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 0553fcf8
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261020T193619Z_0553fcf8_review_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Address three selected local Tuicr comments from
|
||||
`address_logging_levels@wkt-ctx_final_return_disconnect/unstaged/0553fcf`.
|
||||
The human requested detailed endpoint/nursery-owner task diagnostics,
|
||||
an extra newline after the post-disconnect request prose, and warning
|
||||
output consistent with runtime primitive pformat conventions. The `-r`
|
||||
invocation authorizes local responses only; staging and commits remain
|
||||
human-controlled.
|
||||
|
||||
Selected parent comment IDs:
|
||||
- `337f29ca-73b2-4351-b150-d0160d67a2dc`
|
||||
- `2c522ded-3bf1-4827-af8f-c27822d7c48d`
|
||||
- `f00171c0-7a10-455f-a9e3-dd85bfa63df8`
|
||||
|
||||
## Response summary
|
||||
|
||||
The existing pformat_task_uid helper only names the current task. Kept
|
||||
detailed arbitrary-task formatting local to the regression for now, and
|
||||
reused existing primitive formatters in the runtime's warning path.
|
||||
Preserved the narrow final-send suppression boundary and genuine endpoint
|
||||
error propagation control. Captured a selected-path pre-review Git tree
|
||||
without changing the real index, plus original comment/review-state data.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tests/test_context_final_return_disconnect.py`: task snapshots and note.
|
||||
- `tractor/runtime/_rpc.py`: structured primitive warning reports.
|
||||
|
||||
## Verification
|
||||
|
||||
See the raw partner for the 10-pass modern-stack check, successful Ruff
|
||||
and whitespace checks, expected-red diagnostic probe, and the separately
|
||||
reproduced pre-existing Trio 0.29 cancellation-provenance failure.
|
||||
Neither existing environment meets every current harness version bound:
|
||||
the modern downstream interpreter has pytest 9.0.2; the worktree has an
|
||||
older Trio. No dependency setup or lock-file changes were authorized.
|
||||
|
||||
## Human edits
|
||||
|
||||
The captured pre-review work includes the runtime fix and endpoint-error
|
||||
test control already under review. Existing comments, TODOs and review
|
||||
markers were retained; this record describes only the new adjustments.
|
||||
|
||||
## Module-qualified call follow-up
|
||||
|
||||
After tooling commit `bf2bb1c9`, the human requested restoring the
|
||||
existing `_pformat` module reference instead of a direct function import.
|
||||
Removed that import and qualified the sibling, endpoint and owner task
|
||||
calls, wrapping assignments to retain the source line-length convention.
|
||||
The raw partner retains the original generation history. This formatting
|
||||
correction changes no task-selection or disconnect behavior.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
created_at: 2026-10-20T19:36:19Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 0553fcf8
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Generated review adjustments
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_context_final_return_disconnect.py`
|
||||
|
||||
Added a test-local arbitrary-task formatter and snapshots of the endpoint
|
||||
task and its nursery-owning task before the dispatcher wrapper exits.
|
||||
The collateral-cancellation report includes actor identity, sibling,
|
||||
endpoint and owner task fields, parent cancel-scope state and the original
|
||||
cancellation repr. Added the requested blank line after the request prose.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tractor/runtime/_rpc.py`
|
||||
|
||||
Replaced ad-hoc warning labels with Actor/Context/Channel pformat output,
|
||||
the existing cancel-scope formatter and a nested transport-error report.
|
||||
The TransportClosed guard still encloses only final-result transmission.
|
||||
|
||||
## Verification output
|
||||
|
||||
Modern downstream evaluation environment (Python 3.13.14, Trio 0.32.0,
|
||||
pytest 9.0.2): regression plus RPC suite, 10 passed in 1.22s.
|
||||
Standalone Ruff: All checks passed! Git whitespace check passed.
|
||||
|
||||
The worktree environment has Python 3.13.13, Trio 0.29.0 and pytest 9.1.0:
|
||||
9 passed, 1 failed because the pre-existing test expects TransportClosed
|
||||
in cancellation provenance, while this older Trio renders only Cancelled.
|
||||
Executing the captured pre-review test in memory reproduced that assertion
|
||||
failure; it was not caused by these review adjustments. No environment or
|
||||
dependency was changed. The modern environment's pytest is below the
|
||||
current harness reference's >=9.0.3 requirement; that limitation remains.
|
||||
|
||||
An in-memory probe removed only the final-send guard and reproduced the
|
||||
expected TransportClosed failure. It verified the enhanced exception note
|
||||
contains the blank separator, actor identity, three structured task
|
||||
reports, task ownership fields and cancellation source. Source files were
|
||||
unchanged by the probe. Initial scratch-probe validation errors were
|
||||
corrected before the successful expected-red checks; they were not
|
||||
repository test regressions.
|
||||
|
|
@ -10,6 +10,7 @@ import tractor
|
|||
import trio
|
||||
|
||||
from tractor.ipc import Channel
|
||||
from tractor.devx import pformat as _pformat
|
||||
from tractor.msg import Aid, StartAck, Started
|
||||
from tractor.runtime import _rpc
|
||||
|
||||
|
|
@ -19,15 +20,18 @@ async def finish_after_disconnect(
|
|||
ctx: tractor.Context,
|
||||
peer_closed: trio.Event,
|
||||
endpoint_finished: trio.Event,
|
||||
endpoint_error: tractor.TransportClosed|None = None,
|
||||
|
||||
) -> str:
|
||||
'''
|
||||
Finish successfully only after the caller connection is closed.
|
||||
Return or raise the supplied error after the caller disconnects.
|
||||
|
||||
'''
|
||||
await ctx.started('ready')
|
||||
await peer_closed.wait()
|
||||
endpoint_finished.set()
|
||||
if endpoint_error is not None:
|
||||
raise endpoint_error
|
||||
return 'finished'
|
||||
|
||||
|
||||
|
|
@ -46,13 +50,14 @@ async def ordinary_result(
|
|||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'context_endpoint',
|
||||
[False, True],
|
||||
ids=['ordinary-rpc', 'context'],
|
||||
'context_endpoint, endpoint_transport_error',
|
||||
[(False, False), (True, False), (True, True)],
|
||||
ids=['ordinary-rpc', 'context', 'endpoint-transport-error'],
|
||||
)
|
||||
def test_final_return_disconnect_preserves_sibling(
|
||||
reg_addr: tuple,
|
||||
context_endpoint: bool,
|
||||
endpoint_transport_error: bool,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -74,11 +79,20 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
Capture the sibling's task, actor, nursery and cancellation reprs
|
||||
while unwinding so the failure identifies the collateral victim
|
||||
and the cancellation source even after the runtime has exited.
|
||||
Snapshot the endpoint task and its nursery's owning task before
|
||||
they exit, retaining their coroutine and nursery ownership fields.
|
||||
An endpoint-raised TransportClosed is a separate control: it
|
||||
must retain its identity and current nursery propagation rather
|
||||
than being suppressed as a final-result delivery failure.
|
||||
This isolates response delivery; it does not exercise discovery,
|
||||
process supervision, the asyncio bridge, or Qt cancellation.
|
||||
|
||||
'''
|
||||
collateral_cancellations: list[tuple[trio.Cancelled, str]] = []
|
||||
endpoint_error: tractor.TransportClosed|None = (
|
||||
tractor.TransportClosed('endpoint transport failure')
|
||||
if endpoint_transport_error else None
|
||||
)
|
||||
|
||||
async def main() -> None:
|
||||
'''
|
||||
|
|
@ -91,6 +105,8 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
ping = trio.Event()
|
||||
pong = trio.Event()
|
||||
sibling_stopped = trio.Event()
|
||||
endpoint_task_info: str = ''
|
||||
endpoint_owner_info: str = ''
|
||||
|
||||
async def sibling(
|
||||
task_status: trio.TaskStatus = trio.TASK_STATUS_IGNORED,
|
||||
|
|
@ -115,12 +131,20 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
cancel_scope: trio.CancelScope = (
|
||||
nursery.cancel_scope
|
||||
)
|
||||
sibling_info: str = _pformat.pformat_task(task)
|
||||
scope_info: str = _pformat.pformat_cs(
|
||||
cancel_scope,
|
||||
var_name='services.cancel_scope',
|
||||
)
|
||||
runtime_info: str = (
|
||||
f'\nSibling task: {task!r}\n'
|
||||
f'Actor identity: {aid!r}\n'
|
||||
f'Parent nursery: {nursery!r}\n'
|
||||
f'Parent cancel scope: {cancel_scope!r}\n'
|
||||
f'Cancellation: {exc!r}'
|
||||
f'\n'
|
||||
f'Sibling task:\n{sibling_info}\n'
|
||||
f'Endpoint task:\n{endpoint_task_info}\n'
|
||||
f'Endpoint nursery owner:\n'
|
||||
f'{endpoint_owner_info}\n'
|
||||
f'{scope_info}\n'
|
||||
f'Cancellation: {exc!r}\n'
|
||||
)
|
||||
collateral_cancellations.append(
|
||||
(exc, runtime_info),
|
||||
|
|
@ -165,6 +189,8 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
'peer_closed': peer_closed,
|
||||
'endpoint_finished': endpoint_finished,
|
||||
}
|
||||
if context_endpoint:
|
||||
kwargs['endpoint_error'] = endpoint_error
|
||||
|
||||
async def invoke(
|
||||
task_status: trio.TaskStatus,
|
||||
|
|
@ -173,6 +199,9 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
Run the actual RPC dispatcher to completion.
|
||||
|
||||
'''
|
||||
nonlocal endpoint_task_info
|
||||
nonlocal endpoint_owner_info
|
||||
try:
|
||||
await _rpc._invoke(
|
||||
actor,
|
||||
cid,
|
||||
|
|
@ -181,6 +210,18 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
kwargs,
|
||||
task_status=task_status,
|
||||
)
|
||||
finally:
|
||||
endpoint_task: trio.lowlevel.Task = (
|
||||
trio.lowlevel.current_task()
|
||||
)
|
||||
endpoint_task_info = (
|
||||
_pformat.pformat_task(endpoint_task)
|
||||
)
|
||||
endpoint_owner_info = (
|
||||
_pformat.pformat_task(
|
||||
services.parent_task,
|
||||
)
|
||||
)
|
||||
invoke_finished.set()
|
||||
|
||||
async with trio.open_nursery() as services:
|
||||
|
|
@ -219,9 +260,14 @@ def test_final_return_disconnect_preserves_sibling(
|
|||
assert collateral_cancellations
|
||||
cancelled, runtime_info = collateral_cancellations[0]
|
||||
assert 'TransportClosed' in str(cancelled)
|
||||
if endpoint_transport_error:
|
||||
assert exc is endpoint_error
|
||||
return
|
||||
exc.add_note(
|
||||
'The unrelated sibling was cancelled before it could '
|
||||
'answer the post-disconnect request.'
|
||||
+ runtime_info
|
||||
'answer the post-disconnect request.\n'
|
||||
f'\n'
|
||||
f'{runtime_info}'
|
||||
)
|
||||
raise
|
||||
assert not endpoint_transport_error, 'endpoint error suppressed'
|
||||
|
|
|
|||
|
|
@ -743,7 +743,38 @@ async def _invoke(
|
|||
f'\n'
|
||||
f'{pretty_struct.pformat(return_msg)}\n'
|
||||
)
|
||||
try:
|
||||
await chan.send(return_msg)
|
||||
except TransportClosed as ipc_err:
|
||||
# The endpoint completed successfully. Failure to
|
||||
# deliver its Return to a departed peer must not
|
||||
# cancel sibling RPCs in Actor._service_tn.
|
||||
# Keep this guard outside `await coro` so an
|
||||
# endpoint's own transport errors still propagate.
|
||||
actor_info: str = actor.pformat(privates=True)
|
||||
ctx_info: str = ctx.pformat(
|
||||
extra_fields={'scope_nursery': tn},
|
||||
indent='',
|
||||
)
|
||||
scope_info: str = _pformat.pformat_cs(
|
||||
rpc_ctx_cs,
|
||||
var_name='ctx._scope',
|
||||
)
|
||||
chan_info: str = chan.pformat(privates=True)
|
||||
ipc_info: str = _pformat.nest_from_op(
|
||||
input_op='x=>',
|
||||
text=repr(ipc_err),
|
||||
)
|
||||
log.warning(
|
||||
'Final context result delivery failed: '
|
||||
'peer disconnected\n'
|
||||
f'\n'
|
||||
f'{actor_info}\n'
|
||||
f'{ctx_info}\n'
|
||||
f'{scope_info}\n'
|
||||
f'{chan_info}\n'
|
||||
f'{ipc_info}\n'
|
||||
)
|
||||
|
||||
# NOTE: this happens IFF `ctx._scope.cancel()` is
|
||||
# called by any of,
|
||||
|
|
|
|||
Loading…
Reference in New Issue