Prevent asyncio from abandoning guest-runs, .pause_from_sync() support via .to_asyncio #2
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "aio_abandons"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
On
asynciobeing super lovely and abandoning our guest-run..Took me a while to figure out what the heck was going on but, turns out
asynciochanged their SIGINT handling in 3.11 as per:https://docs.python.org/3/library/asyncio-runner.html#handling-keyboard-interruption
I’m not entirely sure if it’s the 3.11 changes or possibly wtv further updates were made in 3.12 but more or less due to the way our current main task was written the
trioguest-run was getting abandoned on SIGINTs sent from the OS to the infected child proc..Note that much of the bug and soln cases are layed out in very detailed comment-notes both in the new test and
run_as_asyncio_guest(), right above the final “fix” lines.The (seemingly working) “fix” required 2 lines of code to be run inside a
asyncio.CancelledErrorhandler around the call toawait trio_done_fut:Actor.cancel_soon()which schedules the actor runtime to cancel on the nexttriorunner cycle and results in a “self cancellation” of the actor.asyncioevent loop” with a non-0.sleep(0.1)XD |_ seems that a “shielded” pump with some actualdelay: float >= 0did the trick to getasyncioto allow thetriorunner/loop to fully complete its guest-run without abandonment.Much improved
asyncio-mode support driven by upcoming py3.13 support,better handling of simultaneous but “independent”
triovs.asyncio.Taskerrors such that they are raised in an eg per1ff79f86b7support for infected-asyncio-mode in a root actor via the
tractor.to_asyncio.run_as_asyncio_guest()entrypoint which can be now used over the stdtrio.run()from the first main/root process to open the runtime via either.open_root_actor()or.open_nursery()(delivered in commit4a195eef4c)New test suites/extensions introduced here,
a new
tests/devxsubpkg for all tooling affiliated testsofficially testing the proto-ed
stackscopeintegrationa new
examples/debugging/asyncio_bp.pyto verifyto_asynciobased support when usingtractor.pause_from_sync()/breakpoint()fromasyncio.Tasks.extensions/reworks to existing
test_infected_asynciosuite generally as part of72035a20d7,test_trio_prestarted_task_bubblestest_trio_closes_early_and_channel_exitstest_aio_cancelled_from_aio_causes_trio_cancelledadding a new dedicated
tests/test_root_infect_asynciosuite inf26d487000to pair with new root actor support (see below).a new
examples/debugging/restore_builtin_breakpoint.pyandtest_breakpoint_hook_restoredto verifybreakpoint()restoration in9af6271e99a356233b47but was able to catch that muck-up thanks to this very bullet list! XD now brought back to life ine8111e40f9History of outstandings from the original
.pause_from_sync()and debug REPL from
asyncio.Tasks effort:There chronology started on github in a WIP PR,
but then was further followed up by adding a new
tractor.devxsub-pkg again via github in PR,There was also an original PR proposed in (gitea) #1 but i (by old habbit/accident) landed it via a new GH PR,
TODO list from GH #374
To be solved here obvi!
trio.to_thread.run_sync()32e12c8b035cdd012417d9662d9b34and7443e387b5.asynciotasks when using our.to_asynciosubsys:.pause_from_sync()andbreakpoint()which are verified supported as of commita356233b47BUT ALSO shows (via test) that crashes inside anasyncio.Taskengage crash-handling correctly (at least mostly)!greenbackbootstrapping:uv install --devoptional depsbreakpoint()hook when it is installed?breakpoint()usage from code that both does not specdebug_mode=Truetoopen_root_actor()as well as if nogreenbackis avail! as of test in9af6271e99greenbackis not installed how should we guard againstdebug_mode=Trueusage from sync code?breakpoint()usage as per test from9af6271e99asyncio.Task” tries to call breakpoint as per1afef149d4b7aa72465da newtest_sync_pause_from_non_greenbacked_aio_taskfor maybe attempting to support this somehow in the future?devx._debug.maybe_init_greenback()?_rootand in_invoke()tasks supported from commit9811db9ac5which (will) land(s) in the upstream #7Unrelated improvements thrown in,
Stuff that was deemed (historically) necessary enough to land alongside all the above,
72fc6fce24Support for passing pre-conf-edLogger; super handy for gettingtractor-styled console pretty-formatting around an external sys/lib’sloggingusage/config.deliver a new a boxed-maybe-error from
open_crash_handler()for post crash introspection (often for testing) purposes ina60837550eexample “raise-from-
finally:” intrionursery test which demos a footgun2bd4cc9727with a potential “holster” solution for unmasking the underlying suppressed errors in such cases in1075ea3687Maybe to cherry from
py313_supportandext_type_pldsbranches?py313_support,8573cd3Tweak some test asserts to betterisstyle4de4897Unset$PYTHON_COLORSfor test debugger suite..1f951a9Anotherisfix..08fa266Add per-side graceful-exit/cancel excs-as-signals985c5a4Moredebug_modetest support, better nursery var names60eca81Be extra sure to re-raise EoCs from translator5ff2740Add a mark topytest.xfail()questionably conc py stuff (ur mam.xfail()s bish!)e313cb5Repair/updatestackscopetestext_type_plds,90287b9Fix anaio_errref bug3d54885Continue supporting py3.11+.to_asyncioanyway?a66caa2Dropasyncio-canc error from._exceptions47ec7e7Add equiv ofAsyncioCancelledfor aio side5ed30dec40to284fa0340e284fa0340etoa870df68c0Since it was all ad-hoc defined inside `._ipc.MsgpackTCPStream._iter_pkts()` more or less, this starts formalizing a way for particular transport backends to indicate whether a disconnect condition should be re-raised in the RPC msg loop and if not what log level to report it at (if any). Based on our lone transport currently we try to suppress any logging noise from ephemeral connections expected during normal actor interaction and discovery subsys ops: - any short lived discovery related TCP connects are only logged as `.transport()` level. - both `.error()` and raise on any underlying `trio.ClosedResource` cause since that normally means some task touched transport layer internals that it shouldn't have. - do a `.warning()` on anything else unexpected. Impl deats: - extend the `._exceptions.TransportClosed` to accept an input log level, raise-on-report toggle and custom reporting & raising via a new `.report_n_maybe_raise()` method. - construct the TCs with inputs per case in (the newly named) `._iter_pkts(). - call ^ this method from the `TransportClosed` handler block inside the RPC msg loop thus delegating reporting levels and/or raising to the backend's per-case TC instantiating. Related `._ipc` changes: - mask out all the `MsgpackTCPStream._codec` debug helper stuff and drop any lingering cruft from the initial proto-ing of msg-codecs. - rename some attrs/methods: |_`MsgpackTCPStream._iter_packets()` -> `._iter_pkts()` and `._agen` -> `_aiter_pkts`. |_`Channel._aiter_recv()` -> `._aiter_msgs()` and `._agen` -> `_aiter_msgs`. - add `hide_tb: bool` support to `Channel.send()` and only show the frame on non-MTEs.The final issue was making sure we do the same thing on ctl-c/SIGINT from the user. That is, if there's already a bg-thread in REPL, we `log.pdb()` about SIGINT shielding and re-draw the prompt; the same UX as normal actor-runtime-task behaviour. Reasons this wasn't workin.. and the fix: - `.pause_from_sync()` was overriding the local `repl` var with `None` delivered by (transitive) calls to `_pause(debug_func=None)`.. so remove all that and only assign it OAOO prior to thread-type case branching. - always call `DebugStatus.shield_sigint()` as needed from all requesting threads/tasks: - in `_pause_from_bg_root_thread()` BEFORE calling `._pause()` AND BEFORE yielding back to the bg-thread via `.started(out)` to ensure we're definitely overriding the handler in the `trio`-main-thread task before unblocking the requesting bg-thread. - from any requesting bg-thread in the root actor such that both its main-`trio`-thread scheduled task (as per above bullet) AND it are SIGINT shielded. - always call `.shield_sigint()` BEFORE any `greenback._await()` case don't entirely grok why yet, but it works)? - for `greenback._await()` case always set `bg_task` to the current one.. - tweaks to the `SIGINT` handler, now renamed `sigint_shield()` so as not to name-collide with the methods when editor-searching: - always try to `repr()` the REPL thread/task "owner" as well as the active `PdbREPL` instance. - add `.devx()` notes around the prompt flushing deats and comments for any root-actor-bg-thread edge cases. Related/supporting refinements: - add `get_lock()`/`get_debug_req()` factory funcs since the plan is to eventually implement both as `@singleton` instances per actor. - fix `acquire_debug_lock()`'s call-sig-bug for scheduling `request_root_stdio_lock()`.. - in `._pause()` only call `mk_pdb()` when `debug_func != None`. - add some todo/warning notes around the `cls.repl = None` in `DebugStatus.release()` `test_pause_from_sync()` tweaks: - don't use a `attach_patts.copy()`, since we always `break` on match. - do `pytest.fail()` on that ^ loop's fallthrough.. - pass `do_ctlc(child, patt=attach_key)` such that we always match the the current thread's name with the ctl-c triggered `.pdb()` emission. - oh yeah, return the last `before: str` from `do_ctlc()`. - in the script, flip `abandon_on_cancel=True` since when `False` it seems to cause `trio.run()` to hang on exit from the last bg-thread case?!?547b957bbftof7469442e3Mostly due to magic from @oremanj (a super-end-level-boss) where we slap in a little bit of `.from_asyncio`-type stuff to run a `trio`-task from `asyncio` code I'm not gonna go into tooo too much detail but basically the primary thing needed was a way to (blocking-ly) invoke a `trio.lowlevel.Task` from an `asyncio.Task` (which we now have with a new `run_trio_task_in_future()` thanks to the "aforementioned jefe") which we now invoke from a dedicated aio case-branch inside `.devx._debug.pause_from_sync()`. Further include a case inside `DebugStatus.release()` to handle using the same func to set the `repl_release: trio.Event` from the `asyncio` side when releasing the REPL. Prolly more refinements to come ;{oa69bc00593to71cf9e7bd371cf9e7bd3to3b39cce7419002f60howtorelease.md fileSuch that you can use, ```python tractor.to_asyncio.run_as_asyncio_guest( trio_main=_trio_main, ) ``` to boostrap the root actor (and thus main parent process) to embed the actor-rumtime into an `asyncio` loop. Prove it all works with an subactor-free version of the aio echo-server test suite B)Such that we can hook into 3rd-party-libs more easily to monkey them and use our (prettier/hipper) console logging with something like (an example from the client project `modden`), ```python connection_mod = i3ipc.connection tractor_style_i3ipc_logger: logging.LoggingAdapter = tractor.log.get_console_log( _root_name=connection_mod.__name__, logger=i3ipc.connection_mod.logger, level='info', ) # monkey the instance-ref in 3rd-party module connection_mod.logger = our_logger ``` Impl deats, - expose as `get_console_log(logger: logging.Logger)` and add default failover logic. - toss in more typing, also for mod-global instance.The (rare) condition is heavily detailed in new comments in the `cancel_trio()` callback but, more or less the idea here is to be extra pedantic in raising an `Exceptiongroup` of errors from each task (both `asyncio` and `trio`) whenever the 2 tasks raise "independently" - in the sense that it's not obviously one side's task causing an error (or cancellation) in the other. In this case we set the error for each side on the `LinkedTaskChannel` (via new attrs described later). As a synopsis, most of this work was refined out of supporting `infected_aio=True` mode in the **root actor** and in particular as part of getting that to work inside the `modden` daemon which at the time of writing was still using the `i3ipc` lib and thus `asyncio`. Impl deats, - extend the `LinkedTaskChannel` field/API set (and type it), - `._trio_task: trio.Task` for test/user introspection. - also "stage" some ideas for a more refined interface, - `.started()` to deliver the value yielded to the `trio.Task` parent. |_ also includes some todos for how to implement this design underneath. - `._aio_first: Any|None = None` to hold that value ^. - `.wait_aio_complete()` for syncing to the asyncio task. - some detailed logging around "asyncio cancelled trio" case. - Move `AsyncioCancelled` in this module. Styling changes, - generally more explicit var naming. - some todos for getting modern and fancy with typing.. NB, Let it be known this commit msg was written on a friday with the help of various "mr. white" solns.a1d75625e4tob7aa72465dHack `asyncio` to not abandon a guest-mode run?to Hack `asyncio` to not abandon a guest-mode run, `.pause_from_sync()` support via `.to_asyncio`Hack `asyncio` to not abandon a guest-mode run, `.pause_from_sync()` support via `.to_asyncio`to Prevent `asyncio` from abandoning guest-runs, `.pause_from_sync()` support via `.to_asyncio`Such that any combination of task terminations/exits can be explicitly handled and "dual side independent" crash cases re-raised in egs. The main error-or-exit impl changes include, - use of new per-side "signaling exceptions": - TrioTaskExited|TrioCancelled for signalling aio. - AsyncioTaskExited|AsyncioCancelled for signalling trio. - NOT overloading the `LinkedTaskChannel._trio/aio_err` fields for err-as-signal relay and instead add a new pair of `._trio/aio_to_raise` maybe-exc-attrs which allow each side's task to specify what it would want the other side to raise to signal its/a termination outcome: - `._trio_to_raise: AsyncioTaskExited|AsyncioCancelled` to signal, |_ the aio task having returned while the trio side was still reading from the `asyncio.Queue` or is just not `.done()`. |_ the aio task being self or trio-request cancelled where a `asyncio.CancelledError` is raised and caught but NOT relayed as is back to trio; instead signal a "more explicit" exc type. - `._aio_to_raise: TrioTaskExited|TrioCancelled` to signal, |_ the trio task having returned while the aio side was still reading from the mem chan and indicating that the trio side might not care any more about future streamed values (like the `Stop/EndOfChannel` equivs for ipc `Context`s). |_ when the trio task canceld we do a `asyncio.Future.set_exception(TrioTaskExited())` to indicate to the aio side verbosely that it should cancel due to the trio parent. - `_aio/trio_err` are now left to only capturing the **actual** per-side task excs for introspection / other side's handling logic. - supporting "graceful exits" depending on API in use from `translate_aio_errors()` such that if either side exits but the other side isn't expect to consume the final `return`ed value, we just exit silently, which required: - adding a `suppress_graceful_exits: bool` flag. - adjusting the `maybe_raise_aio_side_err()` logic to use that flag and suppress only on certain combos of `._trio_to_raise/._trio_err`. - prefer to raise `._trio_to_raise` when the aio-side is the src and vice versa. - filling out pedantic logging for cancellation cases indicating which side is the cause. - add a `LinkedTaskChannel._aio_result` modelled after our `Context._result` a a similar `.wait_for_result()` interface which allows maybe accessing the aio task's final return value if desired when using the `open_channel_from()` API. - rename `cancel_trio()` done handler -> `signal_trio_when_done()` Also some fairly major test suite updates, - add a `delay: int` producing fixture which delivers a much larger timeout whenever `debug_mode` is set so that the REPL can be used without a surrounding cancel firing. - add a new `test_aio_exits_early_relays_AsyncioTaskExited` including a paired `exit_early: bool` flag to `push_from_aio_task()`. - adjust `test_trio_closes_early_causes_aio_checkpoint_raise` to expect a `to_asyncio.TrioTaskExited`.e646ce5c0dto15f99c313eI mean anyone wanting to click approve (since they already built a buncha sh#! on top of this ;) would allow us to conduct the (normal) formal [boom, rhyme time] protocol to all things “community” and “foss”..
@guille
Prevent `asyncio` from abandoning guest-runs, `.pause_from_sync()` support via `.to_asyncio`to Prevent `asyncio` from abandoning guest-runs, `.pause_from_sync()` support via `.to_asyncio`010d75248etoc91373148a@ -82,6 +82,48 @@ class InternalError(RuntimeError):'''class AsyncioCancelled(Exception):This is the main error-translation-semantics that changed, more or less being more pedantic about which side errored/cancelled/exited-gracefully and whether it was independent of the other side.
@ -40,0 +39,4 @@from tractor._exceptions import (InternalError,is_multi_cancelled,TrioTaskExited,For back-lookers (from the future) these new excs drove the improved error translation semantics throughout the cancel and exit handling machinery.
This file’s diff consists of the bulk of the changes described by the PR description.
@ -73,0 +162,4 @@# self._final_result_is_set()# )async def wait_for_result(Replicating the same outcome waiting API as
Context.@ -442,0 +987,4 @@# TODO? factor the next 2 branches into a func like# `try_terminate_aio_task()` and use it for the taskc# case above as well?fut: asyncio.Future|None = aio_task._fut_waiterThis is one of the critical-yet-questionable changes;
asyncio.Task.cancel()seems to never work reliably and can often cause full guest-run abandonment, so instead we take the approach of touching any internalFuturefirst and hoping for the best (which seems to work in practise!).@ -516,0 +1218,4 @@# a `Return`-msg for IPC ctxs)aio_task: asyncio.Task = chan._aio_taskif not aio_task.done():fut: asyncio.Future|None = aio_task._fut_waiterSame as mentioned above; appears to be the best/most-reliable hack for the moment..
@ -516,0 +1242,4 @@'''def run_trio_task_in_future(Much thanks to @oremanj (from core
trioteam on GH) for this fn!@ -1,8 +1,16 @@'''This is a pretty important step forward for the debugger REPL tooling since now you can definitely get multi-actor safe pausing from infected-
asyncioactors including crash handling B)