goodboy
/
tractor
1
0
Fork 2
Code Issues 5 Pull Requests 6 Packages Projects Releases Wiki Activity

Python 3.13 support #18

Merged
goodboy merged 29 commits from py313_support into main 2025-03-27 19:50:43 +00:00
Conversation 1 Commits 29 Files Changed 40
goodboy commented 2025-03-24 21:59:41 +00:00
Owner

Bumping to latest minor release with some refinements and wart fixes along the way!

Main breaking/unexpected changes due to deps

Those requiring attention are detailed in #8 and i’ll briefly re-list here,

  1. trio’s release schedule requires we bump to at least trio>0.27 which flips on strict egs (ExceptionGroups) by default. This originally broke most of our test suite an generally internal RemoteActorError relay and propagation as per the surgery completed in #8..
  • required either of,
    • setting (the soon to be removed/already deprecated) strict_exception_groups=False in various core-runtime and test-suite trio.open_nursery() usage
      • de91ca7 Flip to strict_exception_groups=False in core tns
      • 3e37508 A couple more loose-egs flag flips
      • 0af83bd Go to loose egs in Actor root & service nurseries (for now..)
    • forcing a move in some spots to explicit except*: handlers for topical/manual eg “loosify-ing”.
      • 67c6834 Handle egs on failed request_root_stdio_lock()
      • 539ae7d Draft some eg collapsing helpers (this adds a new trionics._beg mod FYI)
  • various test suite adjustments include,
    • b6e4e58 Another loose-egs flag in test_child_manages_service_nursery
    • ea29a1c Various test tweaks related to 3.13 egs
    • 1a0e631 Another couple loose-ifies for discovery and advanced fault suites
    • 8027746 Fix docs tests with yet another loosie-goosie
    • 42389f8 Use collapse_eg() in broadcaster suite
    • 307e3cf Another loosie in the trioisms suite
  1. asyncio re-mucking its SIGINT machinery; all the miserable details of which can be found in #2..
  • #2 fixes all of this including some 3.13 specific augmentations,
    • 3.13’s asyncio.Queue provides for more explicit aio-side graceful teardown detection using the new asyncio.Queue.shutdown()/.QueueShutDown feats delivered in,
      • 680501a Add per-side graceful-exit/cancel excs-as-signals
      • dc19659 Continue supporting py3.11+
    • avoiding calls to asyncio.Task.cancel() and instead hackily checking for and doing,
      • 5a9a3a4 Hm, asyncio.Task._fut_waiter.set_exception()
  1. cpython deciding to render colored console output in tracebacks which completely rekts pattern matching in the tests/devx/test_debugger.py suite(s).
  • obvi we need to disable that for tests as per
    • 149b75 Unset $PYTHON_COLORS for test debugger suite.. which lands in #2, and
    • 934f453 Disable tb colors in ._testing.mk_cmd()
  1. the stdlib getting a readline.backend: str and seemingly cpython itself being more commonly built (definitely with our move to uv for pking) to use libedit meaning our use of pdbp gets REPL UX issues (tab complete, vi mode) without specially configuring.
    • backported and landed in #14,
      • 9324d82 Handle cpython builds with libedit for readline

Additional dep related changes include,

  • prefer the system python in the uv config, choose the linux distro’s cpython over those distributed by astral.
  • pinning to msgspec>=0.19.0 release which supports 3.13+
  • 9681406 Bump up to pytest>=8.3.5 to match “GH actions”

Miscellaneous refinements,

various minor .devx related repairs/adjustments,

  • 68784a8 Match maybe_open_crash_handler() to non-maybe version
  • fd6d250 Unpack errors from pdb.bdb
  • 0696e7d Fix roundtripped ref error in validate_payload_msg()
  • e42bc33 Move bp to-match-comments on same line for py3.13

RE frame-hiding on internal errors,

  • 0e76a2d Expose hide_tb: bool from .open_nursery()
  • d6ed798 Hide open_nursery() frame by def
  • 78028ca Show frames when decode is handed bad input
  • 4d865b8 Bind another _bexc for debuggin

changes in namespacing,

  • 030a9b5 Expose ._state.debug_mode() predicate at top level

To-cherry into #2

  • 34ce5fd Drop asyncio-canc error from ._exceptions
  • 66bc38f Add equiv of AsyncioCancelled for aio side
  • 38497fa Comment-tag pause points in asycnio_bp.py
Bumping to latest minor release with some refinements and wart fixes along the way! --- Main breaking/unexpected changes due to deps -------------------------------------------- Those requiring attention are detailed in #8 and i'll briefly re-list here, 1. `trio`'s release schedule requires we bump to at least `trio>0.27` which flips on strict egs (`ExceptionGroup`s) by default. This originally broke most of our test suite an generally internal `RemoteActorError` relay and propagation as per the surgery completed in #8.. - [x] required either of, - setting (the soon to be removed/already deprecated) `strict_exception_groups=False` in various core-runtime and test-suite `trio.open_nursery()` usage - de91ca7 Flip to `strict_exception_groups=False` in core tns - 3e37508 A couple more loose-egs flag flips - 0af83bd Go to loose egs in `Actor` root & service nurseries (for now..) - forcing a move in some spots to explicit `except*:` handlers for topical/manual eg "loosify-ing". - 67c6834 Handle egs on failed `request_root_stdio_lock()` - 539ae7d Draft some eg collapsing helpers (this adds a new `trionics._beg` mod FYI) - various test suite adjustments include, - b6e4e58 Another loose-egs flag in `test_child_manages_service_nursery` - ea29a1c Various test tweaks related to 3.13 egs - 1a0e631 Another couple loose-ifies for discovery and advanced fault suites - 8027746 Fix docs tests with yet another loosie-goosie - 42389f8 Use `collapse_eg()` in broadcaster suite - 307e3cf Another loosie in the trioisms suite 2. `asyncio` re-mucking its SIGINT machinery; all the miserable details of which can be found in #2.. - [x] https://pikers.dev/goodboy/tractor/pulls/2 fixes all of this including some 3.13 specific augmentations, - 3.13's `asyncio.Queue` provides for more explicit aio-side graceful teardown detection using the new `asyncio.Queue.shutdown()`/`.QueueShutDown` feats delivered in, - 680501a Add per-side graceful-exit/cancel excs-as-signals - dc19659 Continue supporting py3.11+ - avoiding calls to `asyncio.Task.cancel()` and instead hackily checking for and doing, - 5a9a3a4 Hm, `asyncio.Task._fut_waiter.set_exception()` 3. cpython deciding to render colored console output in tracebacks which completely rekts pattern matching in the `tests/devx/test_debugger.py` suite(s). - [x] obvi we need to disable that for tests as per - 149b75 Unset `$PYTHON_COLORS` for test debugger suite.. which lands in #2, and - 934f453 Disable tb colors in `._testing.mk_cmd()` 4. the stdlib getting a `readline.backend: str` and seemingly cpython itself being more commonly built (definitely with our move to `uv` for pking) to use `libedit` meaning our use of `pdbp` gets REPL UX issues (tab complete, vi mode) without specially configuring. - [x] backported and landed in #14, - 9324d82 Handle cpython builds with `libedit` for `readline` --- Additional dep related changes include, --------------------------------------- - prefer the system python in the `uv` config, choose the linux distro's `cpython` over those distributed by astral. - pinning to `msgspec>=0.19.0` release which supports 3.13+ - 9681406 Bump up to `pytest>=8.3.5` to match "GH actions" --- Miscellaneous refinements, -------------------------- various minor `.devx` related repairs/adjustments, - 68784a8 Match `maybe_open_crash_handler()` to non-maybe version - fd6d250 Unpack errors from `pdb.bdb` - 0696e7d Fix `roundtripped` ref error in `validate_payload_msg()` - e42bc33 Move bp to-match-comments on same line for py3.13 RE frame-hiding on internal errors, - 0e76a2d Expose `hide_tb: bool` from `.open_nursery()` - d6ed798 Hide `open_nursery()` frame by def - 78028ca Show frames when decode is handed bad input - 4d865b8 Bind another `_bexc` for debuggin changes in namespacing, - 030a9b5 Expose `._state.debug_mode()` predicate at top level --- To-cherry into #2 ----------------- - [x] 34ce5fd Drop `asyncio`-canc error from `._exceptions` - [x] 66bc38f Add equiv of `AsyncioCancelled` for aio side - [x] 38497fa Comment-tag pause points in `asycnio_bp.py`
goodboy added 30 commits 2025-03-24 21:59:41 +00:00
845a10b9b2 Bump various (dev) deps and prefer sys python 
Since it turns out there's a few gotchas moving to python 3.13,
- we need to pin to new(er) `trio` which now flips to strict exception
  groups (something to be handled in a follow up patch).
- since we're now using `uv` we should (at least for now) prefer the
  system `python` (over astral's distis) since they compile for
  `libedit` in terms of what the (new) `readline.backend: str` will read
  as; this will break our tab-completion and vi-mode settings in
  the `pdbp` REPL without a user configuring a `~/.editrc`
  appropriately.
- go back to using latest `pdbp` (not a local dev version) since it
  should work fine presuming the previous bullet is addressed.

Lock bumps,
- for now use latest `trio==0.29.0` (which i gotta feeling might have
  broken some existing attempts at strict-eg handling i've tried..)
- update to latest `xonsh`, `pdbp` and its dep `tabcompleter`

Other cleaning,
- put back in various deps "comments" from `poetry` content.
- drop the `xonsh-vox` and `xontrib-vox` dev deps; no `vox` support with
  `uv` rn anyway..
97ba5b595f Flip to `strict_exception_groups=False` in core tns 
Since it'll likely need a bit of detailing to get the test suite running
identically with strict egs (exception groups), i've opted to just flip
the switch on a few core nursery scopes for now until as such a time
i can focus enough to port the matching internals.. Xp
82369773c5 Expose `hide_tb: bool` from `.open_nursery()` 
Such that it gets passed through to `.open_root_actor()` in the
`implicit_runtime==True` case - useful for debugging cases where
`.devx._debug` APIs might be used to avoid REPL clobbering in subactors.
c7f21b9dd4 Log format tweaks for sclang reprs 
A space here, a newline there..
e7638b127c Disable tb colors in `._testing.mk_cmd()` 
Unset the appropriate cpython osenv var such that our `pexpect` script
runs in the test suite can maintain original matching logic.
91d5ce5458 A couple more loose-egs flag flips 
Namely inside,
- `ActorNursery.open_portal()` which uses
  `.trionics.maybe_open_nursery()` and is now adjusted to
  pass-through `**kwargs` for at least this flag.
- inside the `.trionics.gather_contexts()`.
c4419a0d7a Handle egs on failed `request_root_stdio_lock()` 
Namely when the subactor fails to lock the root, in which case we
try to be very verbose about how/what failed in logging as well
as ensure we cancel the employed IPC ctx.

Implement the outer `BaseException` handler to handle both styles,
- match on an eg (or the prior std cancel excs) only raising a lone
  sub-exc from for former.
- always `as _req_err:` and assign to a new func-global `req_err`
  to enable the above matching.

Other,
- raise `DebugStateError` on `status.subactor_uid != actor_uid`.
- fix a `_repl_fail_report` ref error due to making silly assumptions
  about the `_repl_fail_msg` global; now copy from global as default.
- various log-fmt and logic expression styling tweaks.
- ignore `trio.Cancelled` by default in `open_crash_handler()`.
07bc63be8c Add equiv of `AsyncioCancelled` for aio side 
Such that a `TrioCancelled` is raised in the aio task via
`.set_exception()` to explicitly indicate and allow that task to handle
a taskc request from the parent `trio.Task`.
c2f278eace Various test tweaks related to 3.13 egs 
Including changes like,
- loose eg flagging in various test emedded `trio.open_nursery()`s.
- changes to eg handling (like using `except*`).
- added `debug_mode` integration to tests that needed some REPLin
  in order to figure out appropriate updates.
d1903ac437 Fix docs tests with yet another loosie-goosie 
So the KBI propagates up to the actor nursery scope and also avoid
running any `examples/multihost/` subdir scripts.
be0d65b5ee Draft some eg collapsing helpers 
Inside a new `.trionics._beg` and exposed from the subpkg ns in
anticipation of the `strict_exception_groups=False` being removed by
`trio` in py 3.15.

Notes,
- mk an embedded single-exc "extractor" using a `BaseExceptionGroup.exceptions` length
  check, when 1 return the lone child.
- use the above in a new `@acm`, async bc it's most likely to be composed in an
  `async with` tuple-style sequence block, called `collapse_eg()` which
  acts a one line "absorber" for when the above mentioned flag is no
  logner supported by `trio.open_nursery()`.

All untested atm fwiw.. but soon to be used in our test suite(s) likely!
7d435fa33b Use `collapse_eg()` in broadcaster suite 
Around the test embedded `trio.open_nursery()` calls as expected. Also
tidy up the various nursery var names.
26138e29e5 Match `maybe_open_crash_handler()` to non-maybe version 
Such that it will deliver a `BoxedMaybeException` to the caller
regardless whether `pdb` is set, and proxy through all `**kwargs`.
b87ca855be Unpack errors from `pdb.bdb` 
Like any `bdb.BdbQuit` that might be relayed from a remote context after
a REPl exit with the `quit` cmd. This fixes various issues while
debugging where it may not be clear to the parent task that the child
was terminated with a purposefully unrecoverable error.
5297c0b88b Comment-tag pause points in `asycnio_bp.py` 
Thought i already did this but, obvi needed these to make the expect
matches pass in our test.
goodboy added 2 commits 2025-03-24 23:10:36 +00:00
c69877ea6f Bump up to `pytest>=8.3.5` to match "GH actions" 
Ensure it's only for the `--dev` optional deps.
d97e426bca Continue supporting py3.11+ 
Apparently the only thing needing a guard was use of
`asyncio.Queue.shutdown()` and the paired `QueueShutDown` exception?

Cool.
goodboy force-pushed py313_support from d97e426bca to 4a6bac2fb1 2025-03-25 16:08:39 +00:00 Compare
goodboy force-pushed py313_support from 4a6bac2fb1 to e42bc33bd6 2025-03-25 20:00:30 +00:00 Compare
goodboy requested review from guille 2025-03-27 17:39:22 +00:00
goodboy requested review from jc211 2025-03-27 17:39:22 +00:00
goodboy reviewed 2025-03-27 17:41:40 +00:00
tractor/_root.py Outdated
@ -81,3 +81,3 @@
# enables the multi-process debugger support
debug_mode: bool = False,
maybe_enable_greenback: bool = False, # `.pause_from_sync()/breakpoint()` support
maybe_enable_greenback: bool = True, # `.pause_from_sync()/breakpoint()` support
goodboy commented 2025-03-27 17:41:40 +00:00
Poster
Owner

Not sure if this is the right choice, seems like it can’t hurt since we consider greenback a --dev dep?

Not sure if this is the right choice, seems like it can't hurt since we consider `greenback` a `--dev` dep?
goodboy reviewed 2025-03-27 17:45:53 +00:00
tests/test_trioisms.py Outdated
@ -83,0 +100,4 @@
# TODO, ask ronny how to impl this .. XD
# ids='unmask_from_canc={0}, canc_from_finally={1}',#.format,
)
def test_acm_embedded_nursery_propagates_enter_err(
goodboy commented 2025-03-27 17:45:53 +00:00
Poster
Owner

This test demonstrates a footgun i recently discovered debugging silent process tree cancellation in modden.

Turns out if you raise trio.Cancelled from a finally it’ll suppress any underlying error raised from the cancel scope.. This is actually expected behavior but definitely unexpected IHMO.

I spoke to core trio about it over chat but noone seemed that concerned XD

So i’ve replicated the condition as well as attempted to provide a devx-y solution to unwrap the underlying error for the user if detected in any BaseException.__context__ and then use .add_note() to report that the cancellation’s suppression.

This test demonstrates a footgun i recently discovered debugging silent process tree cancellation in `modden`. Turns out if you raise `trio.Cancelled` from a finally it'll suppress any underlying error raised from the cancel scope.. This is actually expected behavior but **definitely unexpected** IHMO. I spoke to core `trio` about it over chat but noone seemed **that concerned** XD So i've replicated the condition as well as attempted to provide a devx-y solution to unwrap the underlying error for the user if detected in any `BaseException.__context__` and then use `.add_note()` to report that the cancellation's suppression.
goodboy reviewed 2025-03-27 17:48:46 +00:00
tractor/__init__.py Outdated
@ -66,3 +67,4 @@ from ._root import (
from ._ipc import Channel as Channel
from ._portal import Portal as Portal
from ._runtime import Actor as Actor
from . import hilevel as hilevel
goodboy commented 2025-03-27 17:48:46 +00:00
Poster
Owner

woof, this should be removed since that subpkg won’t land until #12 ?

woof, this should be removed since that subpkg won't land until #12 ?
goodboy reviewed 2025-03-27 17:51:53 +00:00
examples/advanced_faults/ipc_failure_during_stream.py Outdated
@ -64,2 +64,3 @@
ctx.open_stream() as stream,
trio.open_nursery() as n,
trio.open_nursery(
strict_exception_groups=False,
goodboy commented 2025-03-27 17:51:53 +00:00
Poster
Owner

You’ll find the bulk of the strict-eg-style “sidestepping” is just this for now..

XD

I know that it’ll set to be removed in an upcoming trio release, but we’re likely just going to have to delivery our own version for a variety of remote-error relaying cases, particularly pertaining to RPC semantics.

It’s something to be followed up in #22

You'll find the bulk of the strict-eg-style "sidestepping" is just this for now.. XD I know that it'll set to be removed in an upcoming `trio` release, but we're likely just going to have to delivery our own version for a variety of remote-error relaying cases, particularly pertaining to RPC semantics. It's something to be followed up in #22
goodboy reviewed 2025-03-27 17:59:19 +00:00
tests/test_infected_asyncio.py Outdated
@ -199,2 +255,2 @@
)
await trio.sleep_forever()
# TODO, factor this into a `trionics.collapse()`?
except* BaseException as beg:
goodboy commented 2025-03-27 17:59:19 +00:00
Poster
Owner

prolly coulda also used ._beg.collapse_eg() here too..

Ahh well someone will swap it eventually, at least whenever we have to solve #22

prolly coulda also used `._beg.collapse_eg()` here too.. Ahh well someone will swap it eventually, at least whenever we have to solve #22
goodboy reviewed 2025-03-27 18:00:10 +00:00
tests/test_infected_asyncio.py Outdated
@ -289,3 +351,3 @@
def test_aio_cancelled_from_aio_causes_trio_cancelled(reg_addr):
def test_aio_cancelled_from_aio_causes_trio_cancelled(
goodboy commented 2025-03-27 18:00:10 +00:00
Poster
Owner

Ahh i guess this was a aio related refinement of a test discovered while working through strict egs alongside using to_asyncio stuff.

Prolly woulda been better landed in #2 but, meh.

Nm, was all todo with egs again.

~~Ahh i guess this was a aio related refinement of a test discovered while working through strict egs alongside using `to_asyncio` stuff.~~ ~~Prolly woulda been better landed in #2 but, meh.~~ Nm, was all todo with egs again.
goodboy marked this conversation as resolved
goodboy changed title from Python 3.13 support to Python 3.13 support 2025-03-27 18:31:35 +00:00
goodboy changed target branch from aio_abandons to main 2025-03-27 18:31:36 +00:00
goodboy force-pushed py313_support from e42bc33bd6 to eda48c8021 2025-03-27 18:32:38 +00:00 Compare
guille approved these changes 2025-03-27 19:08:22 +00:00
guille left a comment
Collaborator

Looking forward to #22 in order to get rid of all the strict_exceptions=False warnings

Looking forward to #22 in order to get rid of all the `strict_exceptions=False` warnings
goodboy commented 2025-03-27 19:50:24 +00:00
Poster
Owner

Looking forward to #22 in order to get rid of all the strict_exceptions=False warnings

Indeed!

I’m just holding off on doing it now bc i’d like to take a look at their original implementation and then really think about where we want to keep to the original .trionics._beg.collapse_eg() style.

> Looking forward to #22 in order to get rid of all the `strict_exceptions=False` warnings Indeed! I'm just holding off on doing it now bc i'd like to take a look at their original implementation and then really think about where we want to keep to the original `.trionics._beg.collapse_eg()` style.
goodboy merged commit 1e86722357 into main 2025-03-27 19:50:43 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
jc211
guille
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: goodboy/tractor#18
There is no content yet.
Delete Branch "py313_support"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?