py3.13 test-suite surgery #8

Open
opened 2025-02-26 16:32:12 +00:00 by goodboy · 1 comment

With py 3.13 comes all sorts of “fun”, namely some fairly disruptive lib and UX changes that break the test suite and in particular the pdbp REPL experience.

This issue is mostly meant as a notes-to-self while i work through figuring out all the broken test suites and ideally using any hanging tests to drive a tighter timeout style going forward.


Summary of to-blame cpython, stdlib, and core runtime changes

  1. trio’s release schedule requires we bump to at least trio>0.27 which flips on strict egs by default.
    • breaks most of our test suite, including hanging some tests, and likely will require either setting some strict_exception_groups=False (like ca65aea090 to start) in various suites and/or just forcing the move to explicit except* handling throughout..
    • this needed to be done sooner then later anyway Bp
  2. asyncio re-mucking its SIGINT machinery.
    • alongside new support for infected_aio=True set in the root actor massively borks the to_asyncio oriented tests amongst other very strange SIGINT handler mgmt (that i’d call fairly questionable)..
    • see #2 for hopefully most fixes to all this..
    • also interplays with the new .pause_from_sync() stuff intro-ed in #1 since all that is commonly/intended to be used within (partial) asyncio.Tasks.
  3. cpython deciding to render colored console output in tracebacks which completely rekts pattern matching in the tests/devx/test_debugger.py suite(s).
    • obvi we need to disable that for tests as per 8fa538c40e
  4. the stdlib getting a readline.backend: str and seemingly cpython itself being more commonly built (definitely with our move to uv for pking) to use libedit meaning our use of pdbp gets REPL UX issues (tab complete, vi mode) without specially configuring.
    • not that we test for this yet, but maybe we should ??

Summary of problematic/failing tests of note..

after running the full set and ctl-c any hangers the report is something like,

  • 92 failed,
  • 176 passed,
  • 7 skipped,
  • 4 xfailed,
  • 771 warnings

all in 10658.31s (2:57:38).. yeah bc some got left hanging until i noticed and mashed the SIGINT..


1st-PASS hanging tests

These needed ctl-c (and thought would need timeout wrapping..) UPDATE: seems like for some weird reason certain tests were hanging due to debug_mode==True being set in the ._state._runtime_vars?? No idea why it stopped either 😂

gonna order these according to what i think is least annoying to deal with first and then link commits which provide timeout-ing and then with follow up patches to fix 🙏..

  • ‘tests/test_child_manages_service_nursery.py::test_actor_managed_trio_nursery_task_error_cancels_aio[asyncio_mode=False]’

  • ‘tests/test_rpc.py::test_rpc_errors[this_mod_bad_func]’

test_iter_peer_cancellation-suite,

  • ‘tests/test_inter_peer_cancellation.py::test_do_not_swallow_error_before_started_by_remote_contextcancelled’
  • ‘tests/test_inter_peer_cancellation.py::test_peer_canceller[rte_during_ctxerr=False’
  • ‘tests/test_inter_peer_cancellation.py::test_peer_canceller[rte_during_ctxerr=True]’
  • ‘tests/test_inter_peer_cancellation.py::test_peer_spawns_and_cancels_service_subactor[None-None]’
  • ‘tests/test_inter_peer_cancellation.py::test_peer_spawns_and_cancels_service_subactor[None-KeyboardInterrupt]’

infected_asyncio-suite,

  • ‘tests/test_infected_asyncio.py::test_aio_simple_error’
  • ‘tests/test_infected_asyncio.py::test_aio_cancelled_from_aio_causes_trio_cancelled’
  • ‘tests/test_infected_asyncio.py::test_trio_error_cancels_intertask_chan’
  • ‘tests/test_infected_asyncio.py::test_aio_errors_and_channel_propagates_and_closes’
  • ‘tests/test_infected_asyncio.py::test_echoserver_detailed_mechanics[raise_error=False]’
  • ‘tests/test_infected_asyncio.py::test_echoserver_detailed_mechanics[raise_error=<class ’Exception’>]’

Seemingly all fixed after 71562e0af7


2nd-PASS (28) failing tests

once there were no more hangs the soln patches seemed to have narrowed us down to the remaining fail set,

  • tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_parent-no_msgstream_aclose] - RuntimeError: generator raised StopIteration
  • tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_parent-pre_aclose_msgstream] - RuntimeError: generator raised StopIteration
  • tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_both_parent_first-no_msgstream_aclose] - RuntimeError: generator raised StopIteration
  • tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_both_parent_first-pre_aclose_msgstream] - RuntimeError: generator raised StopIteration

Solved by 375c00da6e

  • tests/test_discovery.py::test_close_channel_explicit[trio-False] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  • tests/test_discovery.py::test_close_channel_explicit[trio-True] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  • tests/test_discovery.py::test_close_channel_explicit_remote_arbiter[trio-False] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  • tests/test_discovery.py::test_close_channel_explicit_remote_arbiter[trio-True] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)

Solved by 375c00da6e

  • tests/test_docs_examples.py::test_example[quick_cluster.py]
    • AssertionError: assert 1 == 0
  • tests/test_docs_examples.py::test_example[server.cpython-312.pyc]
    • UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xcb in position 0: i…

Fixed by ad56d10c51

  • tests/test_infected_asyncio.py::test_basic_interloop_channel_stream[fan_out_w_chan_subscribe=False] - tractor._exceptions.RemoteActorError: remote task raised a ‘AsyncioTaskExited’
  • tests/test_infected_asyncio.py::test_basic_interloop_channel_stream[fan_out_w_chan_subscribe=True] - tractor._exceptions.RemoteActorError: remote task raised a ‘AsyncioTaskExited’

fixed by 7fac170f8d and e1bacaf4f4

  • tests/test_task_broadcasting.py::test_consumer_and_parent_maybe_lag[trio-task_delays1] - ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  • tests/test_task_broadcasting.py::test_no_raise_on_lag - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)

Fixed by 55809dac51


3rd-PASS (22) failing tests

no more hanging tests, just devx/debugger relateds, root-infected-aio and some typed-msging laggards.

= 22 failed, 242 passed, 7 skipped, 4 xfailed, 1220 warnings in 464.82s (0:07:44) =

  • tests/test_trioisms.py::test_stashed_child_nursery[False]
    • ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  • tests/test_trioisms.py::test_acm_embedded_nursery_propagates_enter_err[True-False]
    • AttributeError: ‘NoneType’ object has no attribute ‘value’
  • tests/test_trioisms.py::test_acm_embedded_nursery_propagates_enter_err[True-True]
    • AttributeError: ‘NoneType’ object has no attribute ‘value’

|_> fixed by d32dc4e889

  • tests/devx/test_debugger.py::test_subactor_breakpoint[ctl-c=False-trio]
    • pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.
  • tests/devx/test_debugger.py::test_subactor_breakpoint[ctl-c=True-trio]
    • pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.
  • tests/devx/test_debugger.py::test_multi_nested_subactors_error_through_nurseries[trio]
    • pexpect.exceptions.TIMEOUT: Timeout exceeded.

Mostly repaired by #18

4th-PASS (11 failing) -> back to where i was pre 3.13 XD
  • tests/devx/test_pause_from_non_trio.py::test_pause_from_sync[ctl-c=True-trio]

    • this is getting stuck in a really bad root actor state where SIGINT is being ingored if you look at the process in htop.. So, made a couple issues and linked to .xfail()-ed tests:
  • tests/devx/test_pause_from_non_trio.py::test_sync_pause_from_aio_task[ctl-c=True-trio]

    • this is exiting early due to KBI since our shield handler seems to get removed?
  • commit 81349f7c09 marks the tests

  • #9 is the issue to track and ideally fix and/or report upstream ;)

  • tests/devx/test_tooling.py::test_shield_pause[trio]

    • ValueError: Could not find pattern in before output?

All fixed by b1bcbf5693 and generally in #18

5th-n-FINAL? (4 failing)

So more or less back to where i was doing typed-msging unit tests?

Bo

  • tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-any_type]
    • tractor._exceptions.RemoteActorError: remote task raised a ‘TypeError’
  • tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-nsp_type]
    • tractor._exceptions.RemoteActorError: remote task raised a ‘ValidationError’
  • tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-maybe_nsp_type]
    • tractor._exceptions.RemoteActorError: remote task raised a ‘ValidationError’
  • tests/test_caps_based_msging.py::test_limit_msgspec
    • AssertionError: assert typing.Union[typing.Any, msgspec.Raw] is typing.Any

All changed and fixed after #19

With py 3.13 comes all sorts of "fun", namely some fairly disruptive lib and UX changes that break the test suite and in particular the `pdbp` REPL experience. This issue is mostly meant as a notes-to-self while i work through figuring out all the broken test suites and ideally using any hanging tests to drive a tighter timeout style going forward. --- #### Summary of to-blame cpython, stdlib, and core runtime changes 1. `trio`'s release schedule requires we bump to at least `trio>0.27` which flips on strict egs by default. - breaks most of our test suite, including hanging some tests, and likely will require either setting some `strict_exception_groups=False` (like ca65aea0902c9b94d4c504d8339dd7046d68f03a to start) in various suites and/or just forcing the move to explicit `except*` handling throughout.. - [ ] this needed to be done sooner then later anyway Bp 2. `asyncio` re-mucking its SIGINT machinery. - alongside new support for `infected_aio=True` set in the root actor massively borks the `to_asyncio` oriented tests amongst other very strange SIGINT handler mgmt (that i'd call fairly questionable).. - see https://pikers.dev/goodboy/tractor/pulls/2 for hopefully most fixes to all this.. - also interplays with the new `.pause_from_sync()` stuff intro-ed in #1 since all that is commonly/intended to be used within (partial) `asyncio.Task`s. 3. cpython deciding to render colored console output in tracebacks which completely rekts pattern matching in the `tests/devx/test_debugger.py` suite(s). - obvi we need to disable that for tests as per 8fa538c40e4631018a77c421be632eb889f50f1c 4. the stdlib getting a `readline.backend: str` and seemingly cpython itself being more commonly built (definitely with our move to `uv` for pking) to use `libedit` meaning our use of `pdbp` gets REPL UX issues (tab complete, vi mode) without specially configuring. - [ ] not that we test for this yet, but maybe we should ?? --- #### Summary of problematic/failing tests of note.. after running the full set and ctl-c any hangers the report is something like, - 92 failed, - 176 passed, - 7 skipped, - 4 xfailed, - 771 warnings all in 10658.31s (2:57:38).. yeah bc some got left hanging until i noticed and mashed the SIGINT.. --- ##### 1st-PASS hanging tests These needed ctl-c (and thought would need timeout wrapping..) UPDATE: seems like for some weird reason certain tests were hanging due to `debug_mode==True` being set in the `._state._runtime_vars`?? No idea why it stopped either :joy: gonna order these according to what i think is least annoying to deal with first and then link commits which provide timeout-ing and then with follow up patches to fix :pray:.. - 'tests/test_child_manages_service_nursery.py::test_actor_managed_trio_nursery_task_error_cancels_aio[asyncio_mode=False]' --- - 'tests/test_rpc.py::test_rpc_errors[this_mod_bad_func]' --- `test_iter_peer_cancellation`-suite, - 'tests/test_inter_peer_cancellation.py::test_do_not_swallow_error_before_started_by_remote_contextcancelled' - 'tests/test_inter_peer_cancellation.py::test_peer_canceller[rte_during_ctxerr=False' - 'tests/test_inter_peer_cancellation.py::test_peer_canceller[rte_during_ctxerr=True]' - 'tests/test_inter_peer_cancellation.py::test_peer_spawns_and_cancels_service_subactor[None-None]' - 'tests/test_inter_peer_cancellation.py::test_peer_spawns_and_cancels_service_subactor[None-KeyboardInterrupt]' --- `infected_asyncio`-suite, - 'tests/test_infected_asyncio.py::test_aio_simple_error' - 'tests/test_infected_asyncio.py::test_aio_cancelled_from_aio_causes_trio_cancelled' - 'tests/test_infected_asyncio.py::test_trio_error_cancels_intertask_chan' - 'tests/test_infected_asyncio.py::test_aio_errors_and_channel_propagates_and_closes' - 'tests/test_infected_asyncio.py::test_echoserver_detailed_mechanics[raise_error=False]' - 'tests/test_infected_asyncio.py::test_echoserver_detailed_mechanics[raise_error=<class 'Exception'>]' Seemingly all fixed after 71562e0af7e9d13f04f8069640307b42557cdfe3 --- ##### 2nd-PASS (28) failing tests once there were no more hangs the soln patches seemed to have narrowed us down to the remaining fail set, - tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_parent-no_msgstream_aclose] - RuntimeError: generator raised StopIteration - tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_parent-pre_aclose_msgstream] - RuntimeError: generator raised StopIteration - tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_both_parent_first-no_msgstream_aclose] - RuntimeError: generator raised StopIteration - tests/test_advanced_faults.py::test_ipc_channel_break_during_stream[break_both_parent_first-pre_aclose_msgstream] - RuntimeError: generator raised StopIteration Solved by 375c00da6e7595454b363c726766081e6b7d3a03 - tests/test_discovery.py::test_close_channel_explicit[trio-False] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception) - tests/test_discovery.py::test_close_channel_explicit[trio-True] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception) - tests/test_discovery.py::test_close_channel_explicit_remote_arbiter[trio-False] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception) - tests/test_discovery.py::test_close_channel_explicit_remote_arbiter[trio-True] - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception) Solved by 375c00da6e7595454b363c726766081e6b7d3a03 - tests/test_docs_examples.py::test_example[quick_cluster.py] - AssertionError: assert 1 == 0 - tests/test_docs_examples.py::test_example[server.cpython-312.pyc] - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcb in position 0: i... Fixed by ad56d10c51ec29ee9248f2e438e9b2451270ed8c - tests/test_infected_asyncio.py::test_basic_interloop_channel_stream[fan_out_w_chan_subscribe=False] - tractor._exceptions.RemoteActorError: remote task raised a 'AsyncioTaskExited' - tests/test_infected_asyncio.py::test_basic_interloop_channel_stream[fan_out_w_chan_subscribe=True] - tractor._exceptions.RemoteActorError: remote task raised a 'AsyncioTaskExited' fixed by 7fac170f8d271dd5513368ebd4d66999c1e8815d and e1bacaf4f4223d40e9b6869fdc9138ac0773f6f9 - tests/test_task_broadcasting.py::test_consumer_and_parent_maybe_lag[trio-task_delays1] - ExceptionGroup: Exceptions from Trio nursery (1 sub-exception) - tests/test_task_broadcasting.py::test_no_raise_on_lag - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception) Fixed by 55809dac51ca9f1f343468dfa4c4e8369fc7e6bf --- ##### 3rd-PASS (22) failing tests no more hanging tests, just devx/debugger relateds, root-infected-aio and some typed-msging laggards. `= 22 failed, 242 passed, 7 skipped, 4 xfailed, 1220 warnings in 464.82s (0:07:44) =` - tests/test_trioisms.py::test_stashed_child_nursery[False] - ExceptionGroup: Exceptions from Trio nursery (1 sub-exception) - tests/test_trioisms.py::test_acm_embedded_nursery_propagates_enter_err[True-False] - AttributeError: 'NoneType' object has no attribute 'value' - tests/test_trioisms.py::test_acm_embedded_nursery_propagates_enter_err[True-True] - AttributeError: 'NoneType' object has no attribute 'value' |_> fixed by d32dc4e889443d81a818438ea9db326cbc6eab6c - tests/devx/test_debugger.py::test_subactor_breakpoint[ctl-c=False-trio] - pexpect.exceptions.EOF: End Of File (EOF). Exception style platform. - tests/devx/test_debugger.py::test_subactor_breakpoint[ctl-c=True-trio] - pexpect.exceptions.EOF: End Of File (EOF). Exception style platform. - tests/devx/test_debugger.py::test_multi_nested_subactors_error_through_nurseries[trio] - pexpect.exceptions.TIMEOUT: Timeout exceeded. Mostly repaired by #18 ##### 4th-PASS (11 failing) -> back to where i was pre 3.13 XD - tests/devx/test_pause_from_non_trio.py::test_pause_from_sync[ctl-c=True-trio] - this is getting stuck in a really bad root actor state where SIGINT is being ingored if you look at the process in `htop`.. So, made a couple issues and linked to `.xfail()`-ed tests: - tests/devx/test_pause_from_non_trio.py::test_sync_pause_from_aio_task[ctl-c=True-trio] - this is exiting early due to KBI since our shield handler seems to get removed? - commit 81349f7c09acf9098b560f44e32cb27d91e41242 marks the tests - #9 is the issue to track and ideally fix and/or report upstream ;) - tests/devx/test_tooling.py::test_shield_pause[trio] - ValueError: Could not find pattern in `before` output? All fixed by b1bcbf569386398eba3fc65d8d943e2941cfcc77 and generally in #18 ##### 5th-n-FINAL? (4 failing) So more or less back to where i was doing typed-msging unit tests? Bo - tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-any_type] - tractor._exceptions.RemoteActorError: remote task raised a 'TypeError' - tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-nsp_type] - tractor._exceptions.RemoteActorError: remote task raised a 'ValidationError' - tests/test_caps_based_msging.py::test_codec_hooks_mod[use_codec_hooks-maybe_nsp_type] - tractor._exceptions.RemoteActorError: remote task raised a 'ValidationError' - tests/test_caps_based_msging.py::test_limit_msgspec - AssertionError: assert typing.Union[typing.Any, msgspec.Raw] is typing.Any All changed and fixed after #19
Poster
Owner

As an update we are entirely clean here as of #18 with some lingering typed-msging tests being reworked in a follow up downstream PR.

As an update we are entirely clean here as of #18 with some lingering typed-msging tests being reworked in a follow up downstream PR.
Sign in to join this conversation.
No Label
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: goodboy/tractor#8
There is no content yet.