Merge pull request 'Extension types support via msgspec.Encoder/Decoder hooks' (#19) from ext_type_plds into main

Reviewed(-ish)-on: goodboy/tractor#19

.github/workflows/ci.yml

@@ -20,7 +20,7 @@ jobs:
       - name: Setup python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - name: Install dependencies
         run: pip install -U . --upgrade-strategy eager -r requirements-test.txt
@@ -41,7 +41,7 @@ jobs:
       - name: Setup python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - name: Build sdist
         run: python setup.py sdist --formats=zip
@@ -59,7 +59,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest]
-        python: ['3.10']
+        python: ['3.11']
         spawn_backend: [
           'trio',
           'mp_spawn',

docs/README.rst

@@ -1,20 +1,23 @@
-|logo| ``tractor``: next-gen Python parallelism
+|logo| ``tractor``: distributed structurred concurrency
 
 |gh_actions|
 |docs|
 
-``tractor`` is a `structured concurrent`_, (optionally
-distributed_) multi-processing_ runtime built on trio_.
+``tractor`` is a `structured concurrency`_ (SC), multi-processing_ runtime built on trio_.
 
-Fundamentally, ``tractor`` gives you parallelism via
-``trio``-"*actors*": independent Python processes (aka
-non-shared-memory threads) which maintain structured
-concurrency (SC) *end-to-end* inside a *supervision tree*.
+Fundamentally, ``tractor`` provides parallelism via
+``trio``-"*actors*": independent Python **processes** (i.e.
+*non-shared-memory threads*) which can schedule ``trio`` tasks whilst
+maintaining *end-to-end SC* inside a *distributed supervision tree*.
 
 Cross-process (and thus cross-host) SC is accomplished through the
-combined use of our "actor nurseries_" and an "SC-transitive IPC
-protocol" constructed on top of multiple Pythons each running a ``trio``
-scheduled runtime - a call to ``trio.run()``.
+combined use of our,
+
+- "actor nurseries_" which provide for spawning multiple, and
+  possibly nested, Python processes each running a ``trio`` scheduled
+  runtime - a call to ``trio.run()``,
+- an "SC-transitive supervision protocol" enforced as an
+  IPC-message-spec encapsulating all RPC-dialogs.
 
 We believe the system adheres to the `3 axioms`_ of an "`actor model`_"
 but likely **does not** look like what **you** probably *think* an "actor
@@ -27,6 +30,7 @@ The first step to grok ``tractor`` is to get an intermediate
 knowledge of ``trio`` and **structured concurrency** B)
 
 Some great places to start are,
+
 - the seminal `blog post`_
 - obviously the `trio docs`_
 - wikipedia's nascent SC_ page
@@ -35,22 +39,84 @@ Some great places to start are,
 
 Features
 --------
-- **It's just** a ``trio`` API
-- *Infinitely nesteable* process trees
-- Builtin IPC streaming APIs with task fan-out broadcasting
-- A "native" multi-core debugger REPL using `pdbp`_ (a fork & fix of
-  `pdb++`_ thanks to @mdmintz!)
-- Support for a swappable, OS specific, process spawning layer
-- A modular transport stack, allowing for custom serialization (eg. with
-  `msgspec`_), communications protocols, and environment specific IPC
-  primitives
-- Support for spawning process-level-SC, inter-loop one-to-one-task oriented
-  ``asyncio`` actors via "infected ``asyncio``" mode
-- `structured chadcurrency`_ from the ground up
+- **It's just** a ``trio`` API!
+- *Infinitely nesteable* process trees running embedded ``trio`` tasks.
+- Swappable, OS-specific, process spawning via multiple backends.
+- Modular IPC stack, allowing for custom interchange formats (eg.
+  as offered from `msgspec`_), varied transport protocols (TCP, RUDP,
+  QUIC, wireguard), and OS-env specific higher-perf primitives (UDS,
+  shm-ring-buffers).
+- Optionally distributed_: all IPC and RPC APIs work over multi-host
+  transports the same as local.
+- Builtin high-level streaming API that enables your app to easily
+  leverage the benefits of a "`cheap or nasty`_" `(un)protocol`_.
+- A "native UX" around a multi-process safe debugger REPL using
+  `pdbp`_ (a fork & fix of `pdb++`_)
+- "Infected ``asyncio``" mode: support for starting an actor's
+  runtime as a `guest`_ on the ``asyncio`` loop allowing us to
+  provide stringent SC-style ``trio.Task``-supervision around any
+  ``asyncio.Task`` spawned via our ``tractor.to_asyncio`` APIs.
+- A **very naive** and still very much work-in-progress inter-actor
+  `discovery`_ sys with plans to support multiple `modern protocol`_
+  approaches.
+- Various ``trio`` extension APIs via ``tractor.trionics`` such as,
+  - task fan-out `broadcasting`_,
+  - multi-task-single-resource-caching and fan-out-to-multi
+    ``__aenter__()`` APIs for ``@acm`` functions,
+  - (WIP) a ``TaskMngr``: one-cancels-one style nursery supervisor.
+
+
+Install
+-------
+``tractor`` is still in a *alpha-near-beta-stage* for many
+of its subsystems, however we are very close to having a stable
+lowlevel runtime and API.
+
+As such, it's currently recommended that you clone and install the
+repo from source::
+
+    pip install git+git://github.com/goodboy/tractor.git
+
+
+We use the very hip `uv`_ for project mgmt::
+
+    git clone https://github.com/goodboy/tractor.git
+    cd tractor
+    uv sync --dev
+    uv run python examples/rpc_bidir_streaming.py
+
+Consider activating a virtual/project-env before starting to hack on
+the code base::
+
+    # you could use plain ol' venvs
+    # https://docs.astral.sh/uv/pip/environments/
+    uv venv tractor_py313 --python 3.13
+
+    # but @goodboy prefers the more explicit (and shell agnostic)
+    # https://docs.astral.sh/uv/configuration/environment/#uv_project_environment
+    UV_PROJECT_ENVIRONMENT="tractor_py313
+
+    # hint hint, enter @goodboy's fave shell B)
+    uv run --dev xonsh
+
+Alongside all this we ofc offer "releases" on PyPi::
+
+    pip install tractor
+
+Just note that YMMV since the main git branch is often much further
+ahead then any latest release.
+
+
+Example codez
+-------------
+In ``tractor``'s (very lacking) documention we prefer to point to
+example scripts in the repo over duplicating them in docs, but with
+that in mind here are some definitive snippets to try and hook you
+into digging deeper.
 
 
 Run a func in a process
------------------------
+***********************
 Use ``trio``'s style of focussing on *tasks as functions*:
 
 .. code:: python
@@ -108,7 +174,7 @@ might want to check out `trio-parallel`_.
 
 
 Zombie safe: self-destruct a process tree
------------------------------------------
+*****************************************
 ``tractor`` tries to protect you from zombies, no matter what.
 
 .. code:: python
@@ -164,7 +230,7 @@ it **is a bug**.
 
 
 "Native" multi-process debugging
---------------------------------
+********************************
 Using the magic of `pdbp`_ and our internal IPC, we've
 been able to create a native feeling debugging experience for
 any (sub-)process in your ``tractor`` tree.
@@ -219,7 +285,7 @@ We're hoping to add a respawn-from-repl system soon!
 
 
 SC compatible bi-directional streaming
---------------------------------------
+**************************************
 Yes, you saw it here first; we provide 2-way streams
 with reliable, transitive setup/teardown semantics.
 
@@ -311,7 +377,7 @@ hear your thoughts on!
 
 
 Worker poolz are easy peasy
----------------------------
+***************************
 The initial ask from most new users is *"how do I make a worker
 pool thing?"*.
 
@@ -333,10 +399,10 @@ This uses no extra threads, fancy semaphores or futures; all we need
 is ``tractor``'s IPC!
 
 "Infected ``asyncio``" mode
----------------------------
+***************************
 Have a bunch of ``asyncio`` code you want to force to be SC at the process level?
 
-Check out our experimental system for `guest-mode`_ controlled
+Check out our experimental system for `guest`_-mode controlled
 ``asyncio`` actors:
 
 .. code:: python
@@ -442,7 +508,7 @@ We need help refining the `asyncio`-side channel API to be more
 
 
 Higher level "cluster" APIs
----------------------------
+***************************
 To be extra terse the ``tractor`` devs have started hacking some "higher
 level" APIs for managing actor trees/clusters. These interfaces should
 generally be condsidered provisional for now but we encourage you to try
@@ -499,18 +565,6 @@ spawn a flat cluster:
 .. _full worker pool re-implementation: https://github.com/goodboy/tractor/blob/master/examples/parallelism/concurrent_actors_primes.py
 
 
-Install
--------
-From PyPi::
-
-    pip install tractor
-
-
-From git::
-
-    pip install git+git://github.com/goodboy/tractor.git
-
-
 Under the hood
 --------------
 ``tractor`` is an attempt to pair trionic_ `structured concurrency`_ with
@@ -614,21 +668,26 @@ channel`_!
 .. _adherance to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
 .. _trio gitter channel: https://gitter.im/python-trio/general
 .. _matrix channel: https://matrix.to/#/!tractor:matrix.org
+.. _broadcasting: https://github.com/goodboy/tractor/pull/229
+.. _modern procotol: https://en.wikipedia.org/wiki/Rendezvous_protocol
 .. _pdbp: https://github.com/mdmintz/pdbp
 .. _pdb++: https://github.com/pdbpp/pdbpp
-.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
+.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
+.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
+.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
+.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
 .. _messages: https://en.wikipedia.org/wiki/Message_passing
 .. _trio docs: https://trio.readthedocs.io/en/latest/
 .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
 .. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
-.. _structured chadcurrency: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony
 .. _async generators: https://www.python.org/dev/peps/pep-0525/
 .. _trio-parallel: https://github.com/richardsheridan/trio-parallel
+.. _uv: https://docs.astral.sh/uv/
 .. _msgspec: https://jcristharif.com/msgspec/
-.. _guest-mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
+.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
 
 
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square

examples/advanced_faults/ipc_failure_during_stream.py

@@ -62,7 +62,9 @@ async def recv_and_spawn_net_killers(
     await ctx.started()
     async with (
         ctx.open_stream() as stream,
-        trio.open_nursery() as n,
+        trio.open_nursery(
+            strict_exception_groups=False,
+        ) as tn,
     ):
         async for i in stream:
             print(f'child echoing {i}')
@@ -77,11 +79,11 @@ async def recv_and_spawn_net_killers(
                 i >= break_ipc_after
             ):
                 broke_ipc = True
-                n.start_soon(
+                tn.start_soon(
                     iter_ipc_stream,
                     stream,
                 )
-                n.start_soon(
+                tn.start_soon(
                     partial(
                         break_ipc_then_error,
                         stream=stream,

examples/debugging/asyncio_bp.py

@@ -1,8 +1,16 @@
+'''
+Examples of using the builtin `breakpoint()` from an `asyncio.Task`
+running in a subactor spawned with `infect_asyncio=True`.
+
+'''
 import asyncio
 
 import trio
 import tractor
-from tractor import to_asyncio
+from tractor import (
+    to_asyncio,
+    Portal,
+)
 
 
 async def aio_sleep_forever():
@@ -17,21 +25,21 @@ async def bp_then_error(
 
 ) -> None:
 
-    # sync with ``trio``-side (caller) task
+    # sync with `trio`-side (caller) task
     to_trio.send_nowait('start')
 
     # NOTE: what happens here inside the hook needs some refinement..
     # => seems like it's still `._debug._set_trace()` but
     #    we set `Lock.local_task_in_debug = 'sync'`, we probably want
-    #    some further, at least, meta-data about the task/actoq in debug
-    #    in terms of making it clear it's asyncio mucking about.
-    breakpoint()
+    #    some further, at least, meta-data about the task/actor in debug
+    #    in terms of making it clear it's `asyncio` mucking about.
+    breakpoint()  # asyncio-side
 
     # short checkpoint / delay
-    await asyncio.sleep(0.5)
+    await asyncio.sleep(0.5)  # asyncio-side
 
     if raise_after_bp:
-        raise ValueError('blah')
+        raise ValueError('asyncio side error!')
 
     # TODO: test case with this so that it gets cancelled?
     else:
@@ -49,23 +57,21 @@ async def trio_ctx(
     # this will block until the ``asyncio`` task sends a "first"
     # message, see first line in above func.
     async with (
-
         to_asyncio.open_channel_from(
             bp_then_error,
-            raise_after_bp=not bp_before_started,
+            # raise_after_bp=not bp_before_started,
         ) as (first, chan),
 
-        trio.open_nursery() as n,
+        trio.open_nursery() as tn,
     ):
-
         assert first == 'start'
 
         if bp_before_started:
-            await tractor.breakpoint()
+            await tractor.pause()  # trio-side
 
-        await ctx.started(first)
+        await ctx.started(first)  # trio-side
 
-        n.start_soon(
+        tn.start_soon(
             to_asyncio.run_task,
             aio_sleep_forever,
         )
@@ -73,39 +79,50 @@ async def trio_ctx(
 
 
 async def main(
-    bps_all_over: bool = False,
+    bps_all_over: bool = True,
+
+    # TODO, WHICH OF THESE HAZ BUGZ?
+    cancel_from_root: bool = False,
+    err_from_root: bool = False,
 
 ) -> None:
 
     async with tractor.open_nursery(
-        # debug_mode=True,
-    ) as n:
-
-        p = await n.start_actor(
+        debug_mode=True,
+        maybe_enable_greenback=True,
+        # loglevel='devx',
+    ) as an:
+        ptl: Portal = await an.start_actor(
             'aio_daemon',
             enable_modules=[__name__],
             infect_asyncio=True,
             debug_mode=True,
-            loglevel='cancel',
+            # loglevel='cancel',
         )
 
-        async with p.open_context(
+        async with ptl.open_context(
             trio_ctx,
             bp_before_started=bps_all_over,
         ) as (ctx, first):
 
             assert first == 'start'
 
-            if bps_all_over:
-                await tractor.breakpoint()
+            # pause in parent to ensure no cross-actor
+            # locking problems exist!
+            await tractor.pause()  # trio-root
+
+            if cancel_from_root:
+                await ctx.cancel()
+
+            if err_from_root:
+                assert 0
+            else:
+                await trio.sleep_forever()
 
-            # await trio.sleep_forever()
-            await ctx.cancel()
-            assert 0
 
         # TODO: case where we cancel from trio-side while asyncio task
         # has debugger lock?
-        # await p.cancel_actor()
+        # await ptl.cancel_actor()
 
 
 if __name__ == '__main__':

examples/debugging/fast_error_in_root_after_spawn.py

@@ -1,5 +1,5 @@
 '''
-Fast fail test with a context.
+Fast fail test with a `Context`.
 
 Ensure the partially initialized sub-actor process
 doesn't cause a hang on error/cancel of the parent

examples/debugging/multi_daemon_subactors.py

@@ -7,7 +7,7 @@ async def breakpoint_forever():
     try:
         while True:
             yield 'yo'
-            await tractor.breakpoint()
+            await tractor.pause()
     except BaseException:
         tractor.log.get_console_log().exception(
             'Cancelled while trying to enter pause point!'
@@ -21,11 +21,14 @@ async def name_error():
 
 
 async def main():
-    """Test breakpoint in a streaming actor.
-    """
+    '''
+    Test breakpoint in a streaming actor.
+
+    '''
     async with tractor.open_nursery(
         debug_mode=True,
         loglevel='cancel',
+        # loglevel='devx',
     ) as n:
 
         p0 = await n.start_actor('bp_forever', enable_modules=[__name__])

examples/debugging/multi_nested_subactors_error_up_through_nurseries.py

@@ -10,7 +10,7 @@ async def name_error():
 async def breakpoint_forever():
     "Indefinitely re-enter debugger in child actor."
     while True:
-        await tractor.breakpoint()
+        await tractor.pause()
 
         # NOTE: if the test never sent 'q'/'quit' commands
         # on the pdb repl, without this checkpoint line the

examples/debugging/multi_subactor_root_errors.py

@@ -40,7 +40,7 @@ async def main():
     """
     async with tractor.open_nursery(
         debug_mode=True,
-        # loglevel='cancel',
+        loglevel='devx',
     ) as n:
 
         # spawn both actors

examples/debugging/multi_subactors.py

@@ -6,7 +6,7 @@ async def breakpoint_forever():
     "Indefinitely re-enter debugger in child actor."
     while True:
         await trio.sleep(0.1)
-        await tractor.breakpoint()
+        await tractor.pause()
 
 
 async def name_error():

examples/debugging/restore_builtin_breakpoint.py

@@ -6,19 +6,44 @@ import tractor
 
 
 async def main() -> None:
-    async with tractor.open_nursery(debug_mode=True) as an:
 
-        assert os.environ['PYTHONBREAKPOINT'] == 'tractor._debug._set_trace'
+    # intially unset, no entry.
+    orig_pybp_var: int = os.environ.get('PYTHONBREAKPOINT')
+    assert orig_pybp_var in {None, "0"}
+
+    async with tractor.open_nursery(
+        debug_mode=True,
+    ) as an:
+        assert an
+        assert (
+            (pybp_var := os.environ['PYTHONBREAKPOINT'])
+            ==
+            'tractor.devx._debug._sync_pause_from_builtin'
+        )
 
         # TODO: an assert that verifies the hook has indeed been, hooked
         # XD
-        assert sys.breakpointhook is not tractor._debug._set_trace
+        assert (
+            (pybp_hook := sys.breakpointhook)
+            is not tractor.devx._debug._set_trace
+        )
 
-        breakpoint()
+        print(
+            f'$PYTHONOBREAKPOINT: {pybp_var!r}\n'
+            f'`sys.breakpointhook`: {pybp_hook!r}\n'
+        )
+        breakpoint()  # first bp, tractor hook set.
 
-    # TODO: an assert that verifies the hook is unhooked..
+    # XXX AFTER EXIT (of actor-runtime) verify the hook is unset..
+    #
+    # YES, this is weird but it's how stdlib docs say to do it..
+    # https://docs.python.org/3/library/sys.html#sys.breakpointhook
+    assert os.environ.get('PYTHONBREAKPOINT') is orig_pybp_var
     assert sys.breakpointhook
-    breakpoint()
+
+    # now ensure a regular builtin pause still works
+    breakpoint()  # last bp, stdlib hook restored
+
 
 if __name__ == '__main__':
     trio.run(main)

examples/debugging/root_actor_breakpoint.py

@@ -10,7 +10,7 @@ async def main():
 
         await trio.sleep(0.1)
 
-        await tractor.breakpoint()
+        await tractor.pause()
 
         await trio.sleep(0.1)
 

examples/debugging/root_actor_breakpoint_forever.py

@@ -11,7 +11,7 @@ async def main(
         # loglevel='runtime',
     ):
         while True:
-            await tractor.breakpoint()
+            await tractor.pause()
 
 
 if __name__ == '__main__':

examples/debugging/shield_hang_in_sub.py

@@ -0,0 +1,83 @@
+'''
+Verify we can dump a `stackscope` tree on a hang.
+
+'''
+import os
+import signal
+
+import trio
+import tractor
+
+@tractor.context
+async def start_n_shield_hang(
+    ctx: tractor.Context,
+):
+    # actor: tractor.Actor = tractor.current_actor()
+
+    # sync to parent-side task
+    await ctx.started(os.getpid())
+
+    print('Entering shield sleep..')
+    with trio.CancelScope(shield=True):
+        await trio.sleep_forever()  # in subactor
+
+    # XXX NOTE ^^^ since this shields, we expect
+    # the zombie reaper (aka T800) to engage on
+    # SIGINT from the user and eventually hard-kill
+    # this subprocess!
+
+
+async def main(
+    from_test: bool = False,
+) -> None:
+
+    async with (
+        tractor.open_nursery(
+            debug_mode=True,
+            enable_stack_on_sig=True,
+            # maybe_enable_greenback=False,
+            loglevel='devx',
+        ) as an,
+    ):
+        ptl: tractor.Portal  = await an.start_actor(
+            'hanger',
+            enable_modules=[__name__],
+            debug_mode=True,
+        )
+        async with ptl.open_context(
+            start_n_shield_hang,
+        ) as (ctx, cpid):
+
+            _, proc, _ = an._children[ptl.chan.uid]
+            assert cpid == proc.pid
+
+            print(
+                'Yo my child hanging..?\n'
+                # "i'm a user who wants to see a `stackscope` tree!\n"
+            )
+
+            # XXX simulate the wrapping test's "user actions"
+            # (i.e. if a human didn't run this manually but wants to
+            # know what they should do to reproduce test behaviour)
+            if from_test:
+                print(
+                    f'Sending SIGUSR1 to {cpid!r}!\n'
+                )
+                os.kill(
+                    cpid,
+                    signal.SIGUSR1,
+                )
+
+                # simulate user cancelling program
+                await trio.sleep(0.5)
+                os.kill(
+                    os.getpid(),
+                    signal.SIGINT,
+                )
+            else:
+                # actually let user send the ctl-c
+                await trio.sleep_forever()  # in root
+
+
+if __name__ == '__main__':
+    trio.run(main)

examples/debugging/subactor_bp_in_ctx.py

@@ -4,9 +4,9 @@ import trio
 
 async def gen():
     yield 'yo'
-    await tractor.breakpoint()
+    await tractor.pause()
     yield 'yo'
-    await tractor.breakpoint()
+    await tractor.pause()
 
 
 @tractor.context
@@ -15,7 +15,7 @@ async def just_bp(
 ) -> None:
 
     await ctx.started()
-    await tractor.breakpoint()
+    await tractor.pause()
 
     # TODO: bps and errors in this call..
     async for val in gen():

examples/debugging/sync_bp.py

@@ -4,6 +4,13 @@ import time
 import trio
 import tractor
 
+# TODO: only import these when not running from test harness?
+# can we detect `pexpect` usage maybe?
+# from tractor.devx._debug import (
+#     get_lock,
+#     get_debug_req,
+# )
+
 
 def sync_pause(
     use_builtin: bool = False,
@@ -18,7 +25,13 @@ def sync_pause(
         breakpoint(hide_tb=hide_tb)
 
     else:
+        # TODO: maybe for testing some kind of cm style interface
+        # where the `._set_trace()` call doesn't happen until block
+        # exit?
+        # assert get_lock().ctx_in_debug is None
+        # assert get_debug_req().repl is None
         tractor.pause_from_sync()
+        # assert get_debug_req().repl is None
 
     if error:
         raise RuntimeError('yoyo sync code error')
@@ -41,10 +54,11 @@ async def start_n_sync_pause(
 async def main() -> None:
     async with (
         tractor.open_nursery(
-            # NOTE: required for pausing from sync funcs
-            maybe_enable_greenback=True,
             debug_mode=True,
-            # loglevel='cancel',
+            maybe_enable_greenback=True,
+            enable_stack_on_sig=True,
+            # loglevel='warning',
+            # loglevel='devx',
         ) as an,
         trio.open_nursery() as tn,
     ):
@@ -138,7 +152,9 @@ async def main() -> None:
                     # the case 2. from above still exists!
                     use_builtin=True,
                 ),
-                abandon_on_cancel=False,
+                # TODO: with this `False` we can hang!??!
+                # abandon_on_cancel=False,
+                abandon_on_cancel=True,
                 thread_name='inline_root_bg_thread',
             )
 

examples/full_fledged_streaming_service.py

@@ -91,7 +91,7 @@ async def main() -> list[int]:
     an: ActorNursery
     async with tractor.open_nursery(
         loglevel='cancel',
-        debug_mode=True,
+        # debug_mode=True,
     ) as an:
 
         seed = int(1e3)

examples/quick_cluster.py

@@ -3,20 +3,18 @@ import trio
 import tractor
 
 
-async def sleepy_jane():
-    uid = tractor.current_actor().uid
+async def sleepy_jane() -> None:
+    uid: tuple = tractor.current_actor().uid
     print(f'Yo i am actor {uid}')
     await trio.sleep_forever()
 
 
 async def main():
     '''
-    Spawn a flat actor cluster, with one process per
-    detected core.
+    Spawn a flat actor cluster, with one process per detected core.
 
     '''
     portal_map: dict[str, tractor.Portal]
-    results: dict[str, str]
 
     # look at this hip new syntax!
     async with (
@@ -25,11 +23,16 @@ async def main():
             modules=[__name__]
         ) as portal_map,
 
-        trio.open_nursery() as n,
+        trio.open_nursery(
+            strict_exception_groups=False,
+        ) as tn,
     ):
 
         for (name, portal) in portal_map.items():
-            n.start_soon(portal.run, sleepy_jane)
+            tn.start_soon(
+                portal.run,
+                sleepy_jane,
+            )
 
         await trio.sleep(0.5)
 
@@ -41,4 +44,4 @@ if __name__ == '__main__':
     try:
         trio.run(main)
     except KeyboardInterrupt:
-        pass
+        print('trio cancelled by KBI')

examples/service_discovery.py

@@ -9,7 +9,7 @@ async def main(service_name):
     async with tractor.open_nursery() as an:
         await an.start_actor(service_name)
 
-        async with tractor.get_arbiter('127.0.0.1', 1616) as portal:
+        async with tractor.get_registry('127.0.0.1', 1616) as portal:
             print(f"Arbiter is listening on {portal.channel}")
 
         async with tractor.wait_for_actor(service_name) as sockaddr:

notes_to_self/howtorelease.md

@@ -0,0 +1,18 @@
+First generate a built disti:
+
+```
+python -m pip install --upgrade build
+python -m build --sdist --outdir dist/alpha5/
+```
+
+Then try a test ``pypi`` upload:
+
+```
+python -m twine upload --repository testpypi dist/alpha5/*
+```
+
+The push to `pypi` for realz.
+
+```
+python -m twine upload --repository testpypi dist/alpha5/*
+```

pyproject.toml

@@ -1,71 +1,110 @@
 [build-system]
-requires = ["poetry-core"]
-build-backend = "poetry.core.masonry.api"
+requires = ["hatchling"]
+build-backend = "hatchling.build"
 
-# ------ - ------
+# ------ build-system ------
 
-[tool.poetry]
+[project]
 name = "tractor"
 version = "0.1.0a6dev0"
-description='structured concurrent `trio`-"actors"'
-authors = ["Tyler Goodlet <goodboy_foss@protonmail.com>"]
-license = "AGPlv3"
+description = 'structured concurrent `trio`-"actors"'
+authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
+requires-python = ">= 3.11"
 readme = "docs/README.rst"
-
-# TODO: do we need this xontrib loader at all given pep420
-# and xonsh's xontrib global-autoload-via-setuptools?
-# https://xon.sh/tutorial_xontrib.html#authoring-xontribs
-packages = [
-  {include = 'tractor' },
-  # {include = 'tractor.experimental' },
-  # {include = 'tractor.trionics' },
-  # {include = 'tractor.msg' },
-  # {include = 'tractor.devx' },
+license = "AGPL-3.0-or-later"
+keywords = [
+  "trio",
+  "async",
+  "concurrency",
+  "structured concurrency",
+  "actor model",
+  "distributed",
+  "multiprocessing",
+]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Operating System :: POSIX :: Linux",
+  "Framework :: Trio",
+  "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
+  "Programming Language :: Python :: Implementation :: CPython",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.11",
+  "Topic :: System :: Distributed Computing",
+]
+dependencies = [
+  # trio runtime and friends
+  # (poetry) proper range specs,
+  # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
+  # TODO, for 3.13 we must go go `0.27` which means we have to
+  # disable strict egs or port to handling them internally!
+  "trio>0.27",
+  "tricycle>=0.4.1,<0.5",
+  "wrapt>=1.16.0,<2",
+  "colorlog>=6.8.2,<7",
+  # built-in multi-actor `pdb` REPL
+  "pdbp>=1.6,<2", # windows only (from `pdbp`)
+  # typed IPC msging
+  "msgspec>=0.19.0",
 ]
 
-# ------ - ------
+# ------ project ------
 
-[tool.poetry.dependencies]
-python = "^3.11"
+[dependency-groups]
+dev = [
+  # test suite
+  # TODO: maybe some of these layout choices?
+  # https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
+  "pytest>=8.3.5",
+  "pexpect>=4.9.0,<5",
+  # `tractor.devx` tooling
+  "greenback>=1.2.1,<2",
+  "stackscope>=0.2.2,<0.3",
+  "pyperclip>=1.9.0",
+  "prompt-toolkit>=3.0.50",
+  "xonsh>=0.19.2",
+]
+# TODO, add these with sane versions; were originally in
+# `requirements-docs.txt`..
+# docs = [
+#   "sphinx>="
+#   "sphinx_book_theme>="
+# ]
 
-# trio runtime related
-# proper range spec:
-# https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
-trio='^0.24'
-tricycle = "^0.4.1"
-trio-typing = "^0.10.0"
+# ------ dependency-groups ------
 
-msgspec='^0.18.5'  # interchange
-wrapt = "^1.16.0"  # decorators
-colorlog = "^6.8.2"  # logging
+# ------ dependency-groups ------
 
-# built-in multi-actor `pdb` REPL
-pdbp = "^1.5.0"
+[tool.uv.sources]
+# XXX NOTE, only for @goodboy's hacking on `pprint(sort_dicts=False)`
+# for the `pp` alias..
+# pdbp = { path = "../pdbp", editable = true }
 
-
-# TODO: distributed transport using
+# ------ tool.uv.sources ------
+# TODO, distributed (multi-host) extensions
 # linux kernel networking
 # 'pyroute2
 
-# ------ - ------
+# ------ tool.uv.sources ------
 
-[tool.poetry.group.dev]
-optional = false
-[tool.poetry.group.dev.dependencies]
-# testing
-pytest = "^8.2.0"
-pexpect = "^4.9.0"
+[tool.uv]
+# XXX NOTE, prefer the sys python bc apparently the distis from
+# `astral` are built in a way that breaks `pdbp`+`tabcompleter`'s
+# likely due to linking against `libedit` over `readline`..
+# |_https://docs.astral.sh/uv/concepts/python-versions/#managed-python-distributions
+# |_https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html#use-of-libedit-on-linux
+#
+# https://docs.astral.sh/uv/reference/settings/#python-preference
+python-preference = 'system'
 
-# .devx tooling
-greenback = "^1.2.1"
-stackscope = "^0.2.2"
+# ------ tool.uv ------
 
-# (light) xonsh usage/integration
-xontrib-vox = "^0.0.1"
-prompt-toolkit = "^3.0.43"
-xonsh-vox-tabcomplete = "^0.5"
+[tool.hatch.build.targets.sdist]
+include = ["tractor"]
 
-# ------ - ------
+[tool.hatch.build.targets.wheel]
+include = ["tractor"]
+
+# ------ tool.hatch ------
 
 [tool.towncrier]
 package = "tractor"
@@ -76,27 +115,27 @@ title_format = "tractor {version} ({project_date})"
 template = "nooz/_template.rst"
 all_bullets = true
 
-  [[tool.towncrier.type]]
+[[tool.towncrier.type]]
   directory = "feature"
   name = "Features"
   showcontent = true
 
-  [[tool.towncrier.type]]
+[[tool.towncrier.type]]
   directory = "bugfix"
   name = "Bug Fixes"
   showcontent = true
 
-  [[tool.towncrier.type]]
+[[tool.towncrier.type]]
   directory = "doc"
   name = "Improved Documentation"
   showcontent = true
 
-  [[tool.towncrier.type]]
+[[tool.towncrier.type]]
   directory = "trivial"
   name = "Trivial/Internal Changes"
   showcontent = true
 
-# ------ - ------
+# ------ tool.towncrier ------
 
 [tool.pytest.ini_options]
 minversion = '6.0'
@@ -112,31 +151,8 @@ addopts = [
   '--show-capture=no',
 ]
 log_cli = false
-
 # TODO: maybe some of these layout choices?
 # https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
 # pythonpath = "src"
 
-# ------ - ------
-
-[project]
-name = "tractor"
-keywords = [
-  'trio',
-  'async',
-  'concurrency',
-  'structured concurrency',
-  'actor model',
-  'distributed',
-  'multiprocessing'
-]
-classifiers = [
-  "Development Status :: 3 - Alpha",
-  "Operating System :: POSIX :: Linux",
-  "Framework :: Trio",
-  "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
-  "Programming Language :: Python :: Implementation :: CPython",
-  "Programming Language :: Python :: 3 :: Only",
-  "Programming Language :: Python :: 3.11",
-  "Topic :: System :: Distributed Computing",
-]
+# ------ tool.pytest ------

pytest.ini

@@ -0,0 +1,8 @@
+# vim: ft=ini
+# pytest.ini for tractor
+
+[pytest]
+# don't show frickin captured logs AGAIN in the report..
+addopts = --show-capture='no'
+log_cli = false
+; minversion = 6.0

requirements-docs.txt

@@ -1,2 +0,0 @@
-sphinx
-sphinx_book_theme

requirements-test.txt

@@ -1,9 +0,0 @@
-pytest
-pytest-trio
-pytest-timeout
-pdbp
-mypy
-trio_typing
-pexpect
-towncrier
-numpy

ruff.toml

@@ -0,0 +1,82 @@
+# from default `ruff.toml` @
+# https://docs.astral.sh/ruff/configuration/
+
+# Exclude a variety of commonly ignored directories.
+exclude = [
+    ".bzr",
+    ".direnv",
+    ".eggs",
+    ".git",
+    ".git-rewrite",
+    ".hg",
+    ".ipynb_checkpoints",
+    ".mypy_cache",
+    ".nox",
+    ".pants.d",
+    ".pyenv",
+    ".pytest_cache",
+    ".pytype",
+    ".ruff_cache",
+    ".svn",
+    ".tox",
+    ".venv",
+    ".vscode",
+    "__pypackages__",
+    "_build",
+    "buck-out",
+    "build",
+    "dist",
+    "node_modules",
+    "site-packages",
+    "venv",
+]
+
+# Same as Black.
+line-length = 88
+indent-width = 4
+
+# Assume Python 3.9
+target-version = "py311"
+
+[lint]
+# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`)  codes by default.
+# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
+# McCabe complexity (`C901`) by default.
+select = ["E4", "E7", "E9", "F"]
+ignore = [
+  'E402',  # https://docs.astral.sh/ruff/rules/module-import-not-at-top-of-file/
+]
+
+# Allow fix for all enabled rules (when `--fix`) is provided.
+fixable = ["ALL"]
+unfixable = []
+
+# Allow unused variables when underscore-prefixed.
+# dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
+
+[format]
+# Use single quotes in `ruff format`.
+quote-style = "single"
+
+# Like Black, indent with spaces, rather than tabs.
+indent-style = "space"
+
+# Like Black, respect magic trailing commas.
+skip-magic-trailing-comma = false
+
+# Like Black, automatically detect the appropriate line ending.
+line-ending = "auto"
+
+# Enable auto-formatting of code examples in docstrings. Markdown,
+# reStructuredText code/literal blocks and doctests are all supported.
+#
+# This is currently disabled by default, but it is planned for this
+# to be opt-out in the future.
+docstring-code-format = false
+
+# Set the line length limit used when formatting code snippets in
+# docstrings.
+#
+# This only has an effect when the `docstring-code-format` setting is
+# enabled.
+docstring-code-line-length = "dynamic"

setup.py

@@ -1,102 +0,0 @@
-#!/usr/bin/env python
-#
-# tractor: structured concurrent "actors".
-#
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-from setuptools import setup
-
-with open('docs/README.rst', encoding='utf-8') as f:
-    readme = f.read()
-
-
-setup(
-    name="tractor",
-    version='0.1.0a6dev0',  # alpha zone
-    description='structured concurrent `trio`-"actors"',
-    long_description=readme,
-    license='AGPLv3',
-    author='Tyler Goodlet',
-    maintainer='Tyler Goodlet',
-    maintainer_email='goodboy_foss@protonmail.com',
-    url='https://github.com/goodboy/tractor',
-    platforms=['linux', 'windows'],
-    packages=[
-        'tractor',
-        'tractor.experimental',  # wacky ideas
-        'tractor.trionics',  # trio extensions
-        'tractor.msg',  # lowlevel data types
-        'tractor.devx',  # "dev-experience"
-    ],
-    install_requires=[
-
-        # trio related
-        # proper range spec:
-        # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
-        'trio >= 0.24',
-
-        # 'async_generator',  # in stdlib mostly!
-        # 'trio_typing',  # trio==0.23.0 has type hints!
-        # 'exceptiongroup',  # in stdlib as of 3.11!
-
-        # tooling
-        'stackscope',
-        'tricycle',
-        'trio_typing',
-        'colorlog',
-        'wrapt',
-
-        # IPC serialization
-        'msgspec>=0.18.5',
-
-        # debug mode REPL
-        'pdbp',
-
-        # TODO: distributed transport using
-        # linux kernel networking
-        # 'pyroute2',
-
-        # pip ref docs on these specs:
-        # https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples
-        # and pep:
-        # https://peps.python.org/pep-0440/#version-specifiers
-
-    ],
-    tests_require=['pytest'],
-    python_requires=">=3.10",
-    keywords=[
-        'trio',
-        'async',
-        'concurrency',
-        'structured concurrency',
-        'actor model',
-        'distributed',
-        'multiprocessing'
-    ],
-    classifiers=[
-        "Development Status :: 3 - Alpha",
-        "Operating System :: POSIX :: Linux",
-        "Operating System :: Microsoft :: Windows",
-        "Framework :: Trio",
-        "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
-        "Programming Language :: Python :: Implementation :: CPython",
-        "Programming Language :: Python :: 3 :: Only",
-        "Programming Language :: Python :: 3.10",
-        "Intended Audience :: Science/Research",
-        "Intended Audience :: Developers",
-        "Topic :: System :: Distributed Computing",
-    ],
-)

tests/conftest.py

@@ -75,7 +75,10 @@ def pytest_configure(config):
 
 @pytest.fixture(scope='session')
 def debug_mode(request):
-    return request.config.option.tractor_debug_mode
+    debug_mode: bool = request.config.option.tractor_debug_mode
+    # if debug_mode:
+    #     breakpoint()
+    return debug_mode
 
 
 @pytest.fixture(scope='session', autouse=True)
@@ -92,6 +95,12 @@ def spawn_backend(request) -> str:
     return request.config.option.spawn_backend
 
 
+# @pytest.fixture(scope='function', autouse=True)
+# def debug_enabled(request) -> str:
+#     from tractor import _state
+#     if _state._runtime_vars['_debug_mode']:
+#         breakpoint()
+
 _ci_env: bool = os.environ.get('CI', False)
 
 
@@ -150,6 +159,18 @@ def pytest_generate_tests(metafunc):
         metafunc.parametrize("start_method", [spawn_backend], scope='module')
 
 
+# TODO: a way to let test scripts (like from `examples/`)
+# guarantee they won't registry addr collide!
+# @pytest.fixture
+# def open_test_runtime(
+#     reg_addr: tuple,
+# ) -> AsyncContextManager:
+#     return partial(
+#         tractor.open_nursery,
+#         registry_addrs=[reg_addr],
+#     )
+
+
 def sig_prog(proc, sig):
     "Kill the actor-process with ``sig``."
     proc.send_signal(sig)

tests/devx/conftest.py

@@ -0,0 +1,243 @@
+'''
+`tractor.devx.*` tooling sub-pkg test space.
+
+'''
+import time
+from typing import (
+    Callable,
+)
+
+import pytest
+from pexpect.exceptions import (
+    TIMEOUT,
+)
+from pexpect.spawnbase import SpawnBase
+
+from tractor._testing import (
+    mk_cmd,
+)
+from tractor.devx._debug import (
+    _pause_msg as _pause_msg,
+    _crash_msg as _crash_msg,
+    _repl_fail_msg as _repl_fail_msg,
+    _ctlc_ignore_header as _ctlc_ignore_header,
+)
+from ..conftest import (
+    _ci_env,
+)
+
+
+@pytest.fixture
+def spawn(
+    start_method,
+    testdir: pytest.Pytester,
+    reg_addr: tuple[str, int],
+
+) -> Callable[[str], None]:
+    '''
+    Use the `pexpect` module shipped via `testdir.spawn()` to
+    run an `./examples/..` script by name.
+
+    '''
+    if start_method != 'trio':
+        pytest.skip(
+            '`pexpect` based tests only supported on `trio` backend'
+        )
+
+    def unset_colors():
+        '''
+        Python 3.13 introduced colored tracebacks that break patt
+        matching,
+
+        https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_COLORS
+        https://docs.python.org/3/using/cmdline.html#using-on-controlling-color
+
+        '''
+        import os
+        os.environ['PYTHON_COLORS'] = '0'
+
+    def _spawn(
+        cmd: str,
+        **mkcmd_kwargs,
+    ):
+        unset_colors()
+        return testdir.spawn(
+            cmd=mk_cmd(
+                cmd,
+                **mkcmd_kwargs,
+            ),
+            expect_timeout=3,
+            # preexec_fn=unset_colors,
+            # ^TODO? get `pytest` core to expose underlying
+            # `pexpect.spawn()` stuff?
+        )
+
+    # such that test-dep can pass input script name.
+    return _spawn
+
+
+@pytest.fixture(
+    params=[False, True],
+    ids='ctl-c={}'.format,
+)
+def ctlc(
+    request,
+    ci_env: bool,
+
+) -> bool:
+
+    use_ctlc = request.param
+
+    node = request.node
+    markers = node.own_markers
+    for mark in markers:
+        if mark.name == 'has_nested_actors':
+            pytest.skip(
+                f'Test {node} has nested actors and fails with Ctrl-C.\n'
+                f'The test can sometimes run fine locally but until'
+                ' we solve' 'this issue this CI test will be xfail:\n'
+                'https://github.com/goodboy/tractor/issues/320'
+            )
+
+        if mark.name == 'ctlcs_bish':
+            pytest.skip(
+                f'Test {node} prolly uses something from the stdlib (namely `asyncio`..)\n'
+                f'The test and/or underlying example script can *sometimes* run fine '
+                f'locally but more then likely until the cpython peeps get their sh#$ together, '
+                f'this test will definitely not behave like `trio` under SIGINT..\n'
+            )
+
+    if use_ctlc:
+        # XXX: disable pygments highlighting for auto-tests
+        # since some envs (like actions CI) will struggle
+        # the the added color-char encoding..
+        from tractor.devx._debug import TractorConfig
+        TractorConfig.use_pygements = False
+
+    yield use_ctlc
+
+
+def expect(
+    child,
+
+    # normally a `pdb` prompt by default
+    patt: str,
+
+    **kwargs,
+
+) -> None:
+    '''
+    Expect wrapper that prints last seen console
+    data before failing.
+
+    '''
+    try:
+        child.expect(
+            patt,
+            **kwargs,
+        )
+    except TIMEOUT:
+        before = str(child.before.decode())
+        print(before)
+        raise
+
+
+PROMPT = r"\(Pdb\+\)"
+
+
+def in_prompt_msg(
+    child: SpawnBase,
+    parts: list[str],
+
+    pause_on_false: bool = False,
+    err_on_false: bool = False,
+    print_prompt_on_false: bool = True,
+
+) -> bool:
+    '''
+    Predicate check if (the prompt's) std-streams output has all
+    `str`-parts in it.
+
+    Can be used in test asserts for bulk matching expected
+    log/REPL output for a given `pdb` interact point.
+
+    '''
+    __tracebackhide__: bool = False
+
+    before: str = str(child.before.decode())
+    for part in parts:
+        if part not in before:
+            if pause_on_false:
+                import pdbp
+                pdbp.set_trace()
+
+            if print_prompt_on_false:
+                print(before)
+
+            if err_on_false:
+                raise ValueError(
+                    f'Could not find pattern in `before` output?\n'
+                    f'part: {part!r}\n'
+                )
+            return False
+
+    return True
+
+
+# TODO: todo support terminal color-chars stripping so we can match
+# against call stack frame output from the the 'll' command the like!
+# -[ ] SO answer for stipping ANSI codes: https://stackoverflow.com/a/14693789
+def assert_before(
+    child: SpawnBase,
+    patts: list[str],
+
+    **kwargs,
+
+) -> None:
+    __tracebackhide__: bool = False
+
+    assert in_prompt_msg(
+        child=child,
+        parts=patts,
+
+        # since this is an "assert" helper ;)
+        err_on_false=True,
+        **kwargs
+    )
+
+
+def do_ctlc(
+    child,
+    count: int = 3,
+    delay: float = 0.1,
+    patt: str|None = None,
+
+    # expect repl UX to reprint the prompt after every
+    # ctrl-c send.
+    # XXX: no idea but, in CI this never seems to work even on 3.10 so
+    # needs some further investigation potentially...
+    expect_prompt: bool = not _ci_env,
+
+) -> str|None:
+
+    before: str|None = None
+
+    # make sure ctl-c sends don't do anything but repeat output
+    for _ in range(count):
+        time.sleep(delay)
+        child.sendcontrol('c')
+
+        # TODO: figure out why this makes CI fail..
+        # if you run this test manually it works just fine..
+        if expect_prompt:
+            time.sleep(delay)
+            child.expect(PROMPT)
+            before = str(child.before.decode())
+            time.sleep(delay)
+
+            if patt:
+                # should see the last line on console
+                assert patt in before
+
+    # return the console content up to the final prompt
+    return before

tests/devx/test_debugger.py

@@ -13,26 +13,25 @@ TODO:
 from functools import partial
 import itertools
 import platform
-import pathlib
 import time
 
 import pytest
-import pexpect
 from pexpect.exceptions import (
     TIMEOUT,
     EOF,
 )
 
-from tractor.devx._debug import (
+from .conftest import (
+    do_ctlc,
+    PROMPT,
     _pause_msg,
     _crash_msg,
     _repl_fail_msg,
 )
-from tractor._testing import (
-    examples_dir,
-)
-from conftest import (
-    _ci_env,
+from .conftest import (
+    expect,
+    in_prompt_msg,
+    assert_before,
 )
 
 # TODO: The next great debugger audit could be done by you!
@@ -52,15 +51,6 @@ if platform.system() == 'Windows':
     )
 
 
-def mk_cmd(ex_name: str) -> str:
-    '''
-    Generate a command suitable to pass to ``pexpect.spawn()``.
-
-    '''
-    script_path: pathlib.Path = examples_dir() / 'debugging' / f'{ex_name}.py'
-    return ' '.join(['python', str(script_path)])
-
-
 # TODO: was trying to this xfail style but some weird bug i see in CI
 # that's happening at collect time.. pretty soon gonna dump actions i'm
 # thinkin...
@@ -79,142 +69,6 @@ has_nested_actors = pytest.mark.has_nested_actors
 # )
 
 
-@pytest.fixture
-def spawn(
-    start_method,
-    testdir,
-    reg_addr,
-) -> 'pexpect.spawn':
-
-    if start_method != 'trio':
-        pytest.skip(
-            "Debugger tests are only supported on the trio backend"
-        )
-
-    def _spawn(cmd):
-        return testdir.spawn(
-            cmd=mk_cmd(cmd),
-            expect_timeout=3,
-        )
-
-    return _spawn
-
-
-PROMPT = r"\(Pdb\+\)"
-
-
-def expect(
-    child,
-
-    # prompt by default
-    patt: str = PROMPT,
-
-    **kwargs,
-
-) -> None:
-    '''
-    Expect wrapper that prints last seen console
-    data before failing.
-
-    '''
-    try:
-        child.expect(
-            patt,
-            **kwargs,
-        )
-    except TIMEOUT:
-        before = str(child.before.decode())
-        print(before)
-        raise
-
-
-def in_prompt_msg(
-    prompt: str,
-    parts: list[str],
-
-    pause_on_false: bool = False,
-    print_prompt_on_false: bool = True,
-
-) -> bool:
-    '''
-    Predicate check if (the prompt's) std-streams output has all
-    `str`-parts in it.
-
-    Can be used in test asserts for bulk matching expected
-    log/REPL output for a given `pdb` interact point.
-
-    '''
-    __tracebackhide__: bool = False
-
-    for part in parts:
-        if part not in prompt:
-            if pause_on_false:
-                import pdbp
-                pdbp.set_trace()
-
-            if print_prompt_on_false:
-                print(prompt)
-
-            return False
-
-    return True
-
-
-# TODO: todo support terminal color-chars stripping so we can match
-# against call stack frame output from the the 'll' command the like!
-# -[ ] SO answer for stipping ANSI codes: https://stackoverflow.com/a/14693789
-def assert_before(
-    child,
-    patts: list[str],
-
-    **kwargs,
-
-) -> None:
-    __tracebackhide__: bool = False
-
-    # as in before the prompt end
-    before: str = str(child.before.decode())
-    assert in_prompt_msg(
-        prompt=before,
-        parts=patts,
-
-        **kwargs
-    )
-
-
-@pytest.fixture(
-    params=[False, True],
-    ids='ctl-c={}'.format,
-)
-def ctlc(
-    request,
-    ci_env: bool,
-
-) -> bool:
-
-    use_ctlc = request.param
-
-    node = request.node
-    markers = node.own_markers
-    for mark in markers:
-        if mark.name == 'has_nested_actors':
-            pytest.skip(
-                f'Test {node} has nested actors and fails with Ctrl-C.\n'
-                f'The test can sometimes run fine locally but until'
-                ' we solve' 'this issue this CI test will be xfail:\n'
-                'https://github.com/goodboy/tractor/issues/320'
-            )
-
-    if use_ctlc:
-        # XXX: disable pygments highlighting for auto-tests
-        # since some envs (like actions CI) will struggle
-        # the the added color-char encoding..
-        from tractor.devx._debug import TractorConfig
-        TractorConfig.use_pygements = False
-
-    yield use_ctlc
-
-
 @pytest.mark.parametrize(
     'user_in_out',
     [
@@ -238,14 +92,15 @@ def test_root_actor_error(
     # scan for the prompt
     expect(child, PROMPT)
 
-    before = str(child.before.decode())
-
     # make sure expected logging and error arrives
     assert in_prompt_msg(
-        before,
-        [_crash_msg, "('root'"]
+        child,
+        [
+            _crash_msg,
+            "('root'",
+            'AssertionError',
+        ]
     )
-    assert 'AssertionError' in before
 
     # send user command
     child.sendline(user_input)
@@ -264,8 +119,10 @@ def test_root_actor_error(
     ids=lambda item: f'{item[0]} -> {item[1]}',
 )
 def test_root_actor_bp(spawn, user_in_out):
-    """Demonstrate breakpoint from in root actor.
-    """
+    '''
+    Demonstrate breakpoint from in root actor.
+
+    '''
     user_input, expect_err_str = user_in_out
     child = spawn('root_actor_breakpoint')
 
@@ -279,7 +136,7 @@ def test_root_actor_bp(spawn, user_in_out):
     child.expect('\r\n')
 
     # process should exit
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
     if expect_err_str is None:
         assert 'Error' not in str(child.before)
@@ -287,38 +144,6 @@ def test_root_actor_bp(spawn, user_in_out):
         assert expect_err_str in str(child.before)
 
 
-def do_ctlc(
-    child,
-    count: int = 3,
-    delay: float = 0.1,
-    patt: str|None = None,
-
-    # expect repl UX to reprint the prompt after every
-    # ctrl-c send.
-    # XXX: no idea but, in CI this never seems to work even on 3.10 so
-    # needs some further investigation potentially...
-    expect_prompt: bool = not _ci_env,
-
-) -> None:
-
-    # make sure ctl-c sends don't do anything but repeat output
-    for _ in range(count):
-        time.sleep(delay)
-        child.sendcontrol('c')
-
-        # TODO: figure out why this makes CI fail..
-        # if you run this test manually it works just fine..
-        if expect_prompt:
-            before = str(child.before.decode())
-            time.sleep(delay)
-            child.expect(PROMPT)
-            time.sleep(delay)
-
-            if patt:
-                # should see the last line on console
-                assert patt in before
-
-
 def test_root_actor_bp_forever(
     spawn,
     ctlc: bool,
@@ -358,7 +183,7 @@ def test_root_actor_bp_forever(
 
     # quit out of the loop
     child.sendline('q')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
 @pytest.mark.parametrize(
@@ -380,10 +205,12 @@ def test_subactor_error(
     # scan for the prompt
     child.expect(PROMPT)
 
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
-        [_crash_msg, "('name_error'"]
+        child,
+        [
+            _crash_msg,
+            "('name_error'",
+        ]
     )
 
     if do_next:
@@ -402,17 +229,15 @@ def test_subactor_error(
         child.sendline('continue')
 
     child.expect(PROMPT)
-    before = str(child.before.decode())
-
-    # root actor gets debugger engaged
     assert in_prompt_msg(
-        before,
-        [_crash_msg, "('root'"]
-    )
-    # error is a remote error propagated from the subactor
-    assert in_prompt_msg(
-        before,
-        [_crash_msg, "('name_error'"]
+        child,
+        [
+            _crash_msg,
+            # root actor gets debugger engaged
+            "('root'",
+            # error is a remote error propagated from the subactor
+            "('name_error'",
+        ]
     )
 
     # another round
@@ -423,7 +248,7 @@ def test_subactor_error(
     child.expect('\r\n')
 
     # process should exit
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
 def test_subactor_breakpoint(
@@ -433,14 +258,11 @@ def test_subactor_breakpoint(
     "Single subactor with an infinite breakpoint loop"
 
     child = spawn('subactor_breakpoint')
-
-    # scan for the prompt
     child.expect(PROMPT)
-
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
-        [_pause_msg, "('breakpoint_forever'"]
+        child,
+        [_pause_msg,
+         "('breakpoint_forever'",]
     )
 
     # do some "next" commands to demonstrate recurrent breakpoint
@@ -456,9 +278,8 @@ def test_subactor_breakpoint(
     for _ in range(5):
         child.sendline('continue')
         child.expect(PROMPT)
-        before = str(child.before.decode())
         assert in_prompt_msg(
-            before,
+            child,
             [_pause_msg, "('breakpoint_forever'"]
         )
 
@@ -471,9 +292,8 @@ def test_subactor_breakpoint(
     # child process should exit but parent will capture pdb.BdbQuit
     child.expect(PROMPT)
 
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
+        child,
         ['RemoteActorError:',
          "('breakpoint_forever'",
          'bdb.BdbQuit',]
@@ -486,14 +306,16 @@ def test_subactor_breakpoint(
     child.sendline('c')
 
     # process should exit
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
-        ['RemoteActorError:',
+        child, [
+        'MessagingError:',
+        'RemoteActorError:',
          "('breakpoint_forever'",
-         'bdb.BdbQuit',]
+         'bdb.BdbQuit',
+        ],
+        pause_on_false=True,
     )
 
 
@@ -514,7 +336,7 @@ def test_multi_subactors(
 
     before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
+        child,
         [_pause_msg, "('breakpoint_forever'"]
     )
 
@@ -535,12 +357,14 @@ def test_multi_subactors(
 
     # first name_error failure
     child.expect(PROMPT)
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
-        [_crash_msg, "('name_error'"]
+        child,
+        [
+            _crash_msg,
+            "('name_error'",
+            "NameError",
+        ]
     )
-    assert "NameError" in before
 
     if ctlc:
         do_ctlc(child)
@@ -564,9 +388,8 @@ def test_multi_subactors(
     # breakpoint loop should re-engage
     child.sendline('c')
     child.expect(PROMPT)
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
+        child,
         [_pause_msg, "('breakpoint_forever'"]
     )
 
@@ -629,7 +452,7 @@ def test_multi_subactors(
 
     # process should exit
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
     # repeat of previous multierror for final output
     assert_before(child, [
@@ -659,25 +482,28 @@ def test_multi_daemon_subactors(
     # the root's tty lock first so anticipate either crash
     # message on the first entry.
 
-    bp_forev_parts = [_pause_msg, "('bp_forever'"]
+    bp_forev_parts = [
+        _pause_msg,
+        "('bp_forever'",
+    ]
     bp_forev_in_msg = partial(
         in_prompt_msg,
         parts=bp_forev_parts,
     )
 
-    name_error_msg = "NameError: name 'doggypants' is not defined"
-    name_error_parts = [name_error_msg]
+    name_error_msg: str = "NameError: name 'doggypants' is not defined"
+    name_error_parts: list[str] = [name_error_msg]
 
     before = str(child.before.decode())
 
-    if bp_forev_in_msg(prompt=before):
+    if bp_forev_in_msg(child=child):
         next_parts = name_error_parts
 
     elif name_error_msg in before:
         next_parts = bp_forev_parts
 
     else:
-        raise ValueError("Neither log msg was found !?")
+        raise ValueError('Neither log msg was found !?')
 
     if ctlc:
         do_ctlc(child)
@@ -746,14 +572,12 @@ def test_multi_daemon_subactors(
     # wait for final error in root
     # where it crashs with boxed error
     while True:
-        try:
-            child.sendline('c')
-            child.expect(PROMPT)
-            assert_before(
-                child,
-                bp_forev_parts
-            )
-        except AssertionError:
+        child.sendline('c')
+        child.expect(PROMPT)
+        if not in_prompt_msg(
+            child,
+            bp_forev_parts
+        ):
             break
 
     assert_before(
@@ -769,7 +593,7 @@ def test_multi_daemon_subactors(
     )
 
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
 @has_nested_actors
@@ -845,7 +669,7 @@ def test_multi_subactors_root_errors(
     ])
 
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
     assert_before(child, [
         # "Attaching to pdb in crashed actor: ('root'",
@@ -934,10 +758,13 @@ def test_root_nursery_cancels_before_child_releases_tty_lock(
     child = spawn('root_cancelled_but_child_is_in_tty_lock')
 
     child.expect(PROMPT)
-
-    before = str(child.before.decode())
-    assert "NameError: name 'doggypants' is not defined" in before
-    assert "tractor._exceptions.RemoteActorError: ('name_error'" not in before
+    assert_before(
+        child,
+        [
+            "NameError: name 'doggypants' is not defined",
+            "tractor._exceptions.RemoteActorError: ('name_error'",
+        ],
+    )
     time.sleep(0.5)
 
     if ctlc:
@@ -975,7 +802,7 @@ def test_root_nursery_cancels_before_child_releases_tty_lock(
 
     for i in range(3):
         try:
-            child.expect(pexpect.EOF, timeout=0.5)
+            child.expect(EOF, timeout=0.5)
             break
         except TIMEOUT:
             child.sendline('c')
@@ -1017,7 +844,7 @@ def test_root_cancels_child_context_during_startup(
         do_ctlc(child)
 
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
 def test_different_debug_mode_per_actor(
@@ -1028,9 +855,8 @@ def test_different_debug_mode_per_actor(
     child.expect(PROMPT)
 
     # only one actor should enter the debugger
-    before = str(child.before.decode())
     assert in_prompt_msg(
-        before,
+        child,
         [_crash_msg, "('debugged_boi'", "RuntimeError"],
     )
 
@@ -1038,9 +864,7 @@ def test_different_debug_mode_per_actor(
         do_ctlc(child)
 
     child.sendline('c')
-    child.expect(pexpect.EOF)
-
-    before = str(child.before.decode())
+    child.expect(EOF)
 
     # NOTE: this debugged actor error currently WON'T show up since the
     # root will actually cancel and terminate the nursery before the error
@@ -1059,103 +883,6 @@ def test_different_debug_mode_per_actor(
     )
 
 
-def test_pause_from_sync(
-    spawn,
-    ctlc: bool
-):
-    '''
-    Verify we can use the `pdbp` REPL from sync functions AND from
-    any thread spawned with `trio.to_thread.run_sync()`.
-
-    `examples/debugging/sync_bp.py`
-
-    '''
-    child = spawn('sync_bp')
-
-    # first `sync_pause()` after nurseries open
-    child.expect(PROMPT)
-    assert_before(
-        child,
-        [
-            # pre-prompt line
-            _pause_msg,
-            "<Task '__main__.main'",
-            "('root'",
-        ]
-    )
-    if ctlc:
-        do_ctlc(child)
-
-    child.sendline('c')
-
-
-    # first `await tractor.pause()` inside `p.open_context()` body
-    child.expect(PROMPT)
-
-    # XXX shouldn't see gb loaded message with PDB loglevel!
-    before = str(child.before.decode())
-    assert not in_prompt_msg(
-        before,
-        ['`greenback` portal opened!'],
-    )
-    # should be same root task
-    assert_before(
-        child,
-        [
-            _pause_msg,
-            "<Task '__main__.main'",
-            "('root'",
-        ]
-    )
-
-    if ctlc:
-        do_ctlc(child)
-
-    # one of the bg thread or subactor should have
-    # `Lock.acquire()`-ed
-    # (NOT both, which will result in REPL clobbering!)
-    attach_patts: dict[str, list[str]] = {
-        'subactor': [
-            "'start_n_sync_pause'",
-            "('subactor'",
-        ],
-        'inline_root_bg_thread': [
-            "<Thread(inline_root_bg_thread",
-            "('root'",
-        ],
-        'start_soon_root_bg_thread': [
-            "<Thread(start_soon_root_bg_thread",
-            "('root'",
-        ],
-    }
-    while attach_patts:
-        child.sendline('c')
-        child.expect(PROMPT)
-        before = str(child.before.decode())
-        for key in attach_patts.copy():
-            if key in before:
-                expected_patts: str = attach_patts.pop(key)
-                assert_before(
-                    child,
-                    [_pause_msg] + expected_patts
-                )
-                break
-
-        # ensure no other task/threads engaged a REPL
-        # at the same time as the one that was detected above.
-        for key, other_patts in attach_patts.items():
-            assert not in_prompt_msg(
-                before,
-                other_patts,
-            )
-
-        if ctlc:
-            do_ctlc(child)
-
-    child.sendline('c')
-    child.expect(pexpect.EOF)
-
-
 def test_post_mortem_api(
     spawn,
     ctlc: bool,
@@ -1258,7 +985,7 @@ def test_post_mortem_api(
     # )
 
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
 def test_shield_pause(
@@ -1333,9 +1060,26 @@ def test_shield_pause(
         ]
     )
     child.sendline('c')
-    child.expect(pexpect.EOF)
+    child.expect(EOF)
 
 
+# TODO: better error for "non-ideal" usage from the root actor.
+# -[ ] if called from an async scope emit a message that suggests
+#    using `await tractor.pause()` instead since it's less overhead
+#    (in terms of `greenback` and/or extra threads) and if it's from
+#    a sync scope suggest that usage must first call
+#    `ensure_portal()` in the (eventual parent) async calling scope?
+def test_sync_pause_from_bg_task_in_root_actor_():
+    '''
+    When used from the root actor, normally we can only implicitly
+    support `.pause_from_sync()` from the main-parent-task (that
+    opens the runtime via `open_root_actor()`) since `greenback`
+    requires a `.ensure_portal()` call per `trio.Task` where it is
+    used.
+
+    '''
+    ...
+
 # TODO: needs ANSI code stripping tho, see `assert_before()` # above!
 def test_correct_frames_below_hidden():
     '''

tests/devx/test_pause_from_non_trio.py

@@ -0,0 +1,381 @@
+'''
+That "foreign loop/thread" debug REPL support better ALSO WORK!
+
+Same as `test_native_pause.py`.
+All these tests can be understood (somewhat) by running the
+equivalent `examples/debugging/` scripts manually.
+
+'''
+from contextlib import (
+    contextmanager as cm,
+)
+# from functools import partial
+# import itertools
+import time
+# from typing import (
+#     Iterator,
+# )
+
+import pytest
+from pexpect.exceptions import (
+    TIMEOUT,
+    EOF,
+)
+
+from .conftest import (
+    # _ci_env,
+    do_ctlc,
+    PROMPT,
+    # expect,
+    in_prompt_msg,
+    assert_before,
+    _pause_msg,
+    _crash_msg,
+    _ctlc_ignore_header,
+    # _repl_fail_msg,
+)
+
+@cm
+def maybe_expect_timeout(
+    ctlc: bool = False,
+) -> None:
+    try:
+        yield
+    except TIMEOUT:
+        # breakpoint()
+        if ctlc:
+            pytest.xfail(
+                'Some kinda redic threading SIGINT bug i think?\n'
+                'See the notes in `examples/debugging/sync_bp.py`..\n'
+            )
+        raise
+
+
+@pytest.mark.ctlcs_bish
+def test_pause_from_sync(
+    spawn,
+    ctlc: bool,
+):
+    '''
+    Verify we can use the `pdbp` REPL from sync functions AND from
+    any thread spawned with `trio.to_thread.run_sync()`.
+
+    `examples/debugging/sync_bp.py`
+
+    '''
+    child = spawn('sync_bp')
+
+    # first `sync_pause()` after nurseries open
+    child.expect(PROMPT)
+    assert_before(
+        child,
+        [
+            # pre-prompt line
+            _pause_msg,
+            "<Task '__main__.main'",
+            "('root'",
+        ]
+    )
+    if ctlc:
+        do_ctlc(child)
+        # ^NOTE^ subactor not spawned yet; don't need extra delay.
+
+    child.sendline('c')
+
+    # first `await tractor.pause()` inside `p.open_context()` body
+    child.expect(PROMPT)
+
+    # XXX shouldn't see gb loaded message with PDB loglevel!
+    # assert not in_prompt_msg(
+    #     child,
+    #     ['`greenback` portal opened!'],
+    # )
+    # should be same root task
+    assert_before(
+        child,
+        [
+            _pause_msg,
+            "<Task '__main__.main'",
+            "('root'",
+        ]
+    )
+
+    if ctlc:
+        do_ctlc(
+            child,
+            # NOTE: setting this to 0 (or some other sufficient
+            # small val) can cause the test to fail since the
+            # `subactor` suffers a race where the root/parent
+            # sends an actor-cancel prior to it hitting its pause
+            # point; by def the value is 0.1
+            delay=0.4,
+        )
+
+    # XXX, fwiw without a brief sleep here the SIGINT might actually
+    # trigger "subactor" cancellation by its parent  before the
+    # shield-handler is engaged.
+    #
+    # => similar to the `delay` input to `do_ctlc()` below, setting
+    # this too low can cause the test to fail since the `subactor`
+    # suffers a race where the root/parent sends an actor-cancel
+    # prior to the context task hitting its pause point (and thus
+    # engaging the `sigint_shield()` handler in time); this value
+    # seems be good enuf?
+    time.sleep(0.6)
+
+    # one of the bg thread or subactor should have
+    # `Lock.acquire()`-ed
+    # (NOT both, which will result in REPL clobbering!)
+    attach_patts: dict[str, list[str]] = {
+        'subactor': [
+            "'start_n_sync_pause'",
+            "('subactor'",
+        ],
+        'inline_root_bg_thread': [
+            "<Thread(inline_root_bg_thread",
+            "('root'",
+        ],
+        'start_soon_root_bg_thread': [
+            "<Thread(start_soon_root_bg_thread",
+            "('root'",
+        ],
+    }
+    conts: int = 0  # for debugging below matching logic on failure
+    while attach_patts:
+        child.sendline('c')
+        conts += 1
+        child.expect(PROMPT)
+        before = str(child.before.decode())
+        for key in attach_patts:
+            if key in before:
+                attach_key: str = key
+                expected_patts: str = attach_patts.pop(key)
+                assert_before(
+                    child,
+                    [_pause_msg]
+                    +
+                    expected_patts
+                )
+                break
+        else:
+            pytest.fail(
+                f'No keys found?\n\n'
+                f'{attach_patts.keys()}\n\n'
+                f'{before}\n'
+            )
+
+        # ensure no other task/threads engaged a REPL
+        # at the same time as the one that was detected above.
+        for key, other_patts in attach_patts.copy().items():
+            assert not in_prompt_msg(
+                child,
+                other_patts,
+            )
+
+        if ctlc:
+            do_ctlc(
+                child,
+                patt=attach_key,
+                # NOTE same as comment above
+                delay=0.4,
+            )
+
+    child.sendline('c')
+
+    # XXX TODO, weird threading bug it seems despite the
+    # `abandon_on_cancel: bool` setting to
+    # `trio.to_thread.run_sync()`..
+    with maybe_expect_timeout(
+        ctlc=ctlc,
+    ):
+        child.expect(EOF)
+
+
+def expect_any_of(
+    attach_patts: dict[str, list[str]],
+    child,   # what type?
+    ctlc: bool = False,
+    prompt: str = _ctlc_ignore_header,
+    ctlc_delay: float = .4,
+
+) -> list[str]:
+    '''
+    Receive any of a `list[str]` of patterns provided in
+    `attach_patts`.
+
+    Used to test racing prompts from multiple actors and/or
+    tasks using a common root process' `pdbp` REPL.
+
+    '''
+    assert attach_patts
+
+    child.expect(PROMPT)
+    before = str(child.before.decode())
+
+    for attach_key in attach_patts:
+        if attach_key in before:
+            expected_patts: str = attach_patts.pop(attach_key)
+            assert_before(
+                child,
+                expected_patts
+            )
+            break  # from for
+    else:
+        pytest.fail(
+            f'No keys found?\n\n'
+            f'{attach_patts.keys()}\n\n'
+            f'{before}\n'
+        )
+
+    # ensure no other task/threads engaged a REPL
+    # at the same time as the one that was detected above.
+    for key, other_patts in attach_patts.copy().items():
+        assert not in_prompt_msg(
+            child,
+            other_patts,
+        )
+
+    if ctlc:
+        do_ctlc(
+            child,
+            patt=prompt,
+            # NOTE same as comment above
+            delay=ctlc_delay,
+        )
+
+    return expected_patts
+
+
+@pytest.mark.ctlcs_bish
+def test_sync_pause_from_aio_task(
+    spawn,
+
+    ctlc: bool
+    # ^TODO, fix for `asyncio`!!
+):
+    '''
+    Verify we can use the `pdbp` REPL from an `asyncio.Task` spawned using
+    APIs in `.to_asyncio`.
+
+    `examples/debugging/asycio_bp.py`
+
+    '''
+    child = spawn('asyncio_bp')
+
+    # RACE on whether trio/asyncio task bps first
+    attach_patts: dict[str, list[str]] = {
+
+        # first pause in guest-mode (aka "infecting")
+        # `trio.Task`.
+        'trio-side': [
+            _pause_msg,
+            "<Task 'trio_ctx'",
+            "('aio_daemon'",
+        ],
+
+        # `breakpoint()` from `asyncio.Task`.
+        'asyncio-side': [
+            _pause_msg,
+            "<Task pending name='Task-2' coro=<greenback_shim()",
+            "('aio_daemon'",
+        ],
+    }
+
+    while attach_patts:
+        expect_any_of(
+            attach_patts=attach_patts,
+            child=child,
+            ctlc=ctlc,
+        )
+        child.sendline('c')
+
+    # NOW in race order,
+    # - the asyncio-task will error
+    # - the root-actor parent task will pause
+    #
+    attach_patts: dict[str, list[str]] = {
+
+        # error raised in `asyncio.Task`
+        "raise ValueError('asyncio side error!')": [
+            _crash_msg,
+            "<Task 'trio_ctx'",
+            "@ ('aio_daemon'",
+            "ValueError: asyncio side error!",
+
+            # XXX, we no longer show this frame by default!
+            # 'return await chan.receive()',  # `.to_asyncio` impl internals in tb
+        ],
+
+        # parent-side propagation via actor-nursery/portal
+        # "tractor._exceptions.RemoteActorError: remote task raised a 'ValueError'": [
+        "remote task raised a 'ValueError'": [
+            _crash_msg,
+            "src_uid=('aio_daemon'",
+            "('aio_daemon'",
+        ],
+
+        # a final pause in root-actor
+        "<Task '__main__.main'": [
+            _pause_msg,
+            "<Task '__main__.main'",
+            "('root'",
+        ],
+    }
+    while attach_patts:
+        expect_any_of(
+            attach_patts=attach_patts,
+            child=child,
+            ctlc=ctlc,
+        )
+        child.sendline('c')
+
+    assert not attach_patts
+
+    # final boxed error propagates to root
+    assert_before(
+        child,
+        [
+            _crash_msg,
+            "<Task '__main__.main'",
+            "('root'",
+            "remote task raised a 'ValueError'",
+            "ValueError: asyncio side error!",
+        ]
+    )
+
+    if ctlc:
+        do_ctlc(
+            child,
+            # NOTE: setting this to 0 (or some other sufficient
+            # small val) can cause the test to fail since the
+            # `subactor` suffers a race where the root/parent
+            # sends an actor-cancel prior to it hitting its pause
+            # point; by def the value is 0.1
+            delay=0.4,
+        )
+
+    child.sendline('c')
+    # with maybe_expect_timeout():
+    child.expect(EOF)
+
+
+def test_sync_pause_from_non_greenbacked_aio_task():
+    '''
+    Where the `breakpoint()` caller task is NOT spawned by
+    `tractor.to_asyncio` and thus never activates
+    a `greenback.ensure_portal()` beforehand, presumably bc the task
+    was started by some lib/dep as in often seen in the field.
+
+    Ensure sync pausing works when the pause is in,
+
+    - the root actor running in infected-mode?
+      |_ since we don't need any IPC to acquire the debug lock?
+      |_ is there some way to handle this like the non-main-thread case?
+
+    All other cases need to error out appropriately right?
+
+    - for any subactor we can't avoid needing the repl lock..
+      |_ is there a way to hook into `asyncio.ensure_future(obj)`?
+
+    '''
+    pass

tests/devx/test_tooling.py

@@ -0,0 +1,172 @@
+'''
+That "native" runtime-hackin toolset better be dang useful!
+
+Verify the funtion of a variety of "developer-experience" tools we
+offer from the `.devx` sub-pkg:
+
+- use of the lovely `stackscope` for dumping actor `trio`-task trees
+  during operation and hangs.
+
+TODO:
+- demonstration of `CallerInfo` call stack frame filtering such that
+  for logging and REPL purposes a user sees exactly the layers needed
+  when debugging a problem inside the stack vs. in their app.
+
+'''
+import os
+import signal
+import time
+
+from .conftest import (
+    expect,
+    assert_before,
+    in_prompt_msg,
+    PROMPT,
+    _pause_msg,
+)
+from pexpect.exceptions import (
+    # TIMEOUT,
+    EOF,
+)
+
+
+def test_shield_pause(
+    spawn,
+):
+    '''
+    Verify the `tractor.pause()/.post_mortem()` API works inside an
+    already cancelled `trio.CancelScope` and that you can step to the
+    next checkpoint wherein the cancelled will get raised.
+
+    '''
+    child = spawn(
+        'shield_hang_in_sub'
+    )
+    expect(
+        child,
+        'Yo my child hanging..?',
+    )
+    assert_before(
+        child,
+        [
+            'Entering shield sleep..',
+            'Enabling trace-trees on `SIGUSR1` since `stackscope` is installed @',
+        ]
+    )
+
+    script_pid: int = child.pid
+    print(
+        f'Sending SIGUSR1 to {script_pid}\n'
+        f'(kill -s SIGUSR1 {script_pid})\n'
+    )
+    os.kill(
+        script_pid,
+        signal.SIGUSR1,
+    )
+    time.sleep(0.2)
+    expect(
+        child,
+        # end-of-tree delimiter
+        "end-of-\('root'",
+    )
+    assert_before(
+        child,
+        [
+            # 'Srying to dump `stackscope` tree..',
+            # 'Dumping `stackscope` tree for actor',
+            "('root'",  # uid line
+
+            # TODO!? this used to show?
+            # -[ ] mk reproducable for @oremanj?
+            #
+            # parent block point (non-shielded)
+            # 'await trio.sleep_forever()  # in root',
+        ]
+    )
+    expect(
+        child,
+        # end-of-tree delimiter
+        "end-of-\('hanger'",
+    )
+    assert_before(
+        child,
+        [
+            # relay to the sub should be reported
+            'Relaying `SIGUSR1`[10] to sub-actor',
+
+            "('hanger'",  # uid line
+
+            # TODO!? SEE ABOVE
+            # hanger LOC where it's shield-halted
+            # 'await trio.sleep_forever()  # in subactor',
+        ]
+    )
+
+    # simulate the user sending a ctl-c to the hanging program.
+    # this should result in the terminator kicking in since
+    # the sub is shield blocking and can't respond to SIGINT.
+    os.kill(
+        child.pid,
+        signal.SIGINT,
+    )
+    expect(
+        child,
+        'Shutting down actor runtime',
+        timeout=6,
+    )
+    assert_before(
+        child,
+        [
+            'raise KeyboardInterrupt',
+            # 'Shutting down actor runtime',
+            '#T-800 deployed to collect zombie B0',
+            "'--uid', \"('hanger',",
+        ]
+    )
+
+
+def test_breakpoint_hook_restored(
+    spawn,
+):
+    '''
+    Ensures our actor runtime sets a custom `breakpoint()` hook
+    on open then restores the stdlib's default on close.
+
+    The hook state validation is done via `assert`s inside the
+    invoked script with only `breakpoint()` (not `tractor.pause()`)
+    calls used.
+
+    '''
+    child = spawn('restore_builtin_breakpoint')
+
+    child.expect(PROMPT)
+    assert_before(
+        child,
+        [
+            _pause_msg,
+            "<Task '__main__.main'",
+            "('root'",
+            "first bp, tractor hook set",
+        ]
+    )
+    child.sendline('c')
+    child.expect(PROMPT)
+    assert_before(
+        child,
+        [
+            "last bp, stdlib hook restored",
+        ]
+    )
+
+    # since the stdlib hook was already restored there should be NO
+    # `tractor` `log.pdb()` content from console!
+    assert not in_prompt_msg(
+        child,
+        [
+            _pause_msg,
+            "<Task '__main__.main'",
+            "('root'",
+        ],
+    )
+    child.sendline('c')
+    child.expect(EOF)

tests/test_advanced_faults.py

@@ -3,7 +3,6 @@ Sketchy network blackoutz, ugly byzantine gens, puedes eschuchar la
 cancelacion?..
 
 '''
-import itertools
 from functools import partial
 from types import ModuleType
 
@@ -91,7 +90,8 @@ def test_ipc_channel_break_during_stream(
 
         # non-`trio` spawners should never hit the hang condition that
         # requires the user to do ctl-c to cancel the actor tree.
-        expect_final_exc = trio.ClosedResourceError
+        # expect_final_exc = trio.ClosedResourceError
+        expect_final_exc = tractor.TransportClosed
 
     mod: ModuleType = import_path(
         examples_dir() / 'advanced_faults'
@@ -157,7 +157,7 @@ def test_ipc_channel_break_during_stream(
         if pre_aclose_msgstream:
             expect_final_exc = KeyboardInterrupt
 
-    # NOTE when the parent IPC side dies (even if the child's does as well
+    # NOTE when the parent IPC side dies (even if the child does as well
     # but the child fails BEFORE the parent) we always expect the
     # IPC layer to raise a closed-resource, NEVER do we expect
     # a stop msg since the parent-side ctx apis will error out
@@ -169,7 +169,8 @@ def test_ipc_channel_break_during_stream(
         and
         ipc_break['break_child_ipc_after'] is False
     ):
-        expect_final_exc = trio.ClosedResourceError
+        # expect_final_exc = trio.ClosedResourceError
+        expect_final_exc = tractor.TransportClosed
 
     # BOTH but, PARENT breaks FIRST
     elif (
@@ -180,7 +181,8 @@ def test_ipc_channel_break_during_stream(
             ipc_break['break_parent_ipc_after']
         )
     ):
-        expect_final_exc = trio.ClosedResourceError
+        # expect_final_exc = trio.ClosedResourceError
+        expect_final_exc = tractor.TransportClosed
 
     with pytest.raises(
         expected_exception=(
@@ -199,8 +201,8 @@ def test_ipc_channel_break_during_stream(
                     **ipc_break,
                 )
             )
-        except KeyboardInterrupt as kbi:
-            _err = kbi
+        except KeyboardInterrupt as _kbi:
+            kbi = _kbi
             if expect_final_exc is not KeyboardInterrupt:
                 pytest.fail(
                     'Rxed unexpected KBI !?\n'
@@ -209,16 +211,28 @@ def test_ipc_channel_break_during_stream(
 
             raise
 
+        except tractor.TransportClosed as _tc:
+            tc = _tc
+            if expect_final_exc is KeyboardInterrupt:
+                pytest.fail(
+                    'Unexpected transport failure !?\n'
+                    f'{repr(tc)}'
+                )
+            cause: Exception = tc.__cause__
+            assert (
+                type(cause) is trio.ClosedResourceError
+                and
+                cause.args[0] == 'another task closed this fd'
+            )
+            raise
+
     # get raw instance from pytest wrapper
     value = excinfo.value
     if isinstance(value, ExceptionGroup):
-        value = next(
-            itertools.dropwhile(
-                lambda exc: not isinstance(exc, expect_final_exc),
-                value.exceptions,
-            )
-        )
-        assert value
+        excs = value.exceptions
+        assert len(excs) == 1
+        final_exc = excs[0]
+        assert isinstance(final_exc, expect_final_exc)
 
 
 @tractor.context
@@ -241,15 +255,16 @@ async def break_ipc_after_started(
 
 def test_stream_closed_right_after_ipc_break_and_zombie_lord_engages():
     '''
-    Verify that is a subactor's IPC goes down just after bringing up a stream
-    the parent can trigger a SIGINT and the child will be reaped out-of-IPC by
-    the localhost process supervision machinery: aka "zombie lord".
+    Verify that is a subactor's IPC goes down just after bringing up
+    a stream the parent can trigger a SIGINT and the child will be
+    reaped out-of-IPC by the localhost process supervision machinery:
+    aka "zombie lord".
 
     '''
     async def main():
         with trio.fail_after(3):
-            async with tractor.open_nursery() as n:
-                portal = await n.start_actor(
+            async with tractor.open_nursery() as an:
+                portal = await an.start_actor(
                     'ipc_breaker',
                     enable_modules=[__name__],
                 )

tests/test_advanced_streaming.py

@@ -307,7 +307,15 @@ async def inf_streamer(
 
     async with (
         ctx.open_stream() as stream,
-        trio.open_nursery() as tn,
+
+        # XXX TODO, INTERESTING CASE!!
+        # - if we don't collapse the eg then the embedded
+        # `trio.EndOfChannel` doesn't propagate directly to the above
+        # .open_stream() parent, resulting in it also raising instead
+        # of gracefully absorbing as normal.. so how to handle?
+        trio.open_nursery(
+            strict_exception_groups=False,
+        ) as tn,
     ):
         async def close_stream_on_sentinel():
             async for msg in stream:

tests/test_cancellation.py

@@ -14,7 +14,7 @@ import tractor
 from tractor._testing import (
     tractor_test,
 )
-from conftest import no_windows
+from .conftest import no_windows
 
 
 def is_win():
@@ -130,7 +130,7 @@ def test_multierror(
             try:
                 await portal2.result()
             except tractor.RemoteActorError as err:
-                assert err.boxed_type == AssertionError
+                assert err.boxed_type is AssertionError
                 print("Look Maa that first actor failed hard, hehh")
                 raise
 
@@ -182,7 +182,7 @@ def test_multierror_fast_nursery(reg_addr, start_method, num_subactors, delay):
 
     for exc in exceptions:
         assert isinstance(exc, tractor.RemoteActorError)
-        assert exc.boxed_type == AssertionError
+        assert exc.boxed_type is AssertionError
 
 
 async def do_nothing():
@@ -504,7 +504,9 @@ def test_cancel_via_SIGINT_other_task(
     if is_win():  # smh
         timeout += 1
 
-    async def spawn_and_sleep_forever(task_status=trio.TASK_STATUS_IGNORED):
+    async def spawn_and_sleep_forever(
+        task_status=trio.TASK_STATUS_IGNORED
+    ):
         async with tractor.open_nursery() as tn:
             for i in range(3):
                 await tn.run_in_actor(
@@ -517,7 +519,9 @@ def test_cancel_via_SIGINT_other_task(
     async def main():
         # should never timeout since SIGINT should cancel the current program
         with trio.fail_after(timeout):
-            async with trio.open_nursery() as n:
+            async with trio.open_nursery(
+                strict_exception_groups=False,
+            ) as n:
                 await n.start(spawn_and_sleep_forever)
                 if 'mp' in spawn_backend:
                     time.sleep(0.1)
@@ -610,6 +614,12 @@ def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
                     nurse.start_soon(delayed_kbi)
 
                     await p.run(do_nuthin)
+
+        # need to explicitly re-raise the lone kbi..now
+        except* KeyboardInterrupt as kbi_eg:
+            assert (len(excs := kbi_eg.exceptions) == 1)
+            raise excs[0]
+
         finally:
             duration = time.time() - start
             if duration > timeout:

tests/test_caps_based_msging.py

@@ -1,930 +0,0 @@
-'''
-Low-level functional audits for our
-"capability based messaging"-spec feats.
-
-B~)
-
-'''
-import typing
-from typing import (
-    Any,
-    Type,
-    Union,
-)
-from contextvars import (
-    Context,
-)
-
-from msgspec import (
-    structs,
-    msgpack,
-    Struct,
-    ValidationError,
-)
-import pytest
-
-import tractor
-from tractor import (
-    _state,
-    MsgTypeError,
-)
-from tractor.msg import (
-    _codec,
-    _ctxvar_MsgCodec,
-
-    NamespacePath,
-    MsgCodec,
-    mk_codec,
-    apply_codec,
-    current_codec,
-)
-from tractor.msg.types import (
-    _payload_msgs,
-    log,
-    Msg,
-    Started,
-    mk_msg_spec,
-)
-import trio
-
-
-def mk_custom_codec(
-    pld_spec: Union[Type]|Any,
-    add_hooks: bool,
-
-) -> MsgCodec:
-    '''
-    Create custom `msgpack` enc/dec-hooks and set a `Decoder`
-    which only loads `pld_spec` (like `NamespacePath`) types.
-
-    '''
-    uid: tuple[str, str] = tractor.current_actor().uid
-
-    # XXX NOTE XXX: despite defining `NamespacePath` as a type
-    # field on our `Msg.pld`, we still need a enc/dec_hook() pair
-    # to cast to/from that type on the wire. See the docs:
-    # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
-
-    def enc_nsp(obj: Any) -> Any:
-        print(f'{uid} ENC HOOK')
-        match obj:
-            case NamespacePath():
-                print(
-                    f'{uid}: `NamespacePath`-Only ENCODE?\n'
-                    f'obj-> `{obj}`: {type(obj)}\n'
-                )
-                # if type(obj) != NamespacePath:
-                #     breakpoint()
-                return str(obj)
-
-        print(
-            f'{uid}\n'
-            'CUSTOM ENCODE\n'
-            f'obj-arg-> `{obj}`: {type(obj)}\n'
-        )
-        logmsg: str = (
-            f'{uid}\n'
-            'FAILED ENCODE\n'
-            f'obj-> `{obj}: {type(obj)}`\n'
-        )
-        raise NotImplementedError(logmsg)
-
-    def dec_nsp(
-        obj_type: Type,
-        obj: Any,
-
-    ) -> Any:
-        print(
-            f'{uid}\n'
-            'CUSTOM DECODE\n'
-            f'type-arg-> {obj_type}\n'
-            f'obj-arg-> `{obj}`: {type(obj)}\n'
-        )
-        nsp = None
-
-        if (
-            obj_type is NamespacePath
-            and isinstance(obj, str)
-            and ':' in obj
-        ):
-            nsp = NamespacePath(obj)
-            # TODO: we could built a generic handler using
-            # JUST matching the obj_type part?
-            # nsp = obj_type(obj)
-
-        if nsp:
-            print(f'Returning NSP instance: {nsp}')
-            return nsp
-
-        logmsg: str = (
-            f'{uid}\n'
-            'FAILED DECODE\n'
-            f'type-> {obj_type}\n'
-            f'obj-arg-> `{obj}`: {type(obj)}\n\n'
-            f'current codec:\n'
-            f'{current_codec()}\n'
-        )
-        # TODO: figure out the ignore subsys for this!
-        # -[ ] option whether to defense-relay backc the msg
-        #   inside an `Invalid`/`Ignore`
-        # -[ ] how to make this handling pluggable such that a
-        #   `Channel`/`MsgTransport` can intercept and process
-        #   back msgs either via exception handling or some other
-        #   signal?
-        log.warning(logmsg)
-        # NOTE: this delivers the invalid
-        # value up to `msgspec`'s decoding
-        # machinery for error raising.
-        return obj
-        # raise NotImplementedError(logmsg)
-
-    nsp_codec: MsgCodec = mk_codec(
-        ipc_pld_spec=pld_spec,
-
-        # NOTE XXX: the encode hook MUST be used no matter what since
-        # our `NamespacePath` is not any of a `Any` native type nor
-        # a `msgspec.Struct` subtype - so `msgspec` has no way to know
-        # how to encode it unless we provide the custom hook.
-        #
-        # AGAIN that is, regardless of whether we spec an
-        # `Any`-decoded-pld the enc has no knowledge (by default)
-        # how to enc `NamespacePath` (nsp), so we add a custom
-        # hook to do that ALWAYS.
-        enc_hook=enc_nsp if add_hooks else None,
-
-        # XXX NOTE: pretty sure this is mutex with the `type=` to
-        # `Decoder`? so it won't work in tandem with the
-        # `ipc_pld_spec` passed above?
-        dec_hook=dec_nsp if add_hooks else None,
-    )
-    return nsp_codec
-
-
-def chk_codec_applied(
-    expect_codec: MsgCodec,
-    enter_value: MsgCodec|None = None,
-
-) -> MsgCodec:
-    '''
-    buncha sanity checks ensuring that the IPC channel's
-    context-vars are set to the expected codec and that are
-    ctx-var wrapper APIs match the same.
-
-    '''
-    # TODO: play with tricyle again, bc this is supposed to work
-    # the way we want?
-    #
-    # TreeVar
-    # task: trio.Task = trio.lowlevel.current_task()
-    # curr_codec = _ctxvar_MsgCodec.get_in(task)
-
-    # ContextVar
-    # task_ctx: Context = task.context
-    # assert _ctxvar_MsgCodec in task_ctx
-    # curr_codec: MsgCodec = task.context[_ctxvar_MsgCodec]
-
-    # NOTE: currently we use this!
-    # RunVar
-    curr_codec: MsgCodec = current_codec()
-    last_read_codec = _ctxvar_MsgCodec.get()
-    # assert curr_codec is last_read_codec
-
-    assert (
-        (same_codec := expect_codec) is
-        # returned from `mk_codec()`
-
-        # yielded value from `apply_codec()`
-
-        # read from current task's `contextvars.Context`
-        curr_codec is
-        last_read_codec
-
-        # the default `msgspec` settings
-        is not _codec._def_msgspec_codec
-        is not _codec._def_tractor_codec
-    )
-
-    if enter_value:
-        enter_value is same_codec
-
-
-def iter_maybe_sends(
-    send_items: dict[Union[Type], Any] | list[tuple],
-    ipc_pld_spec: Union[Type] | Any,
-    add_codec_hooks: bool,
-
-    codec: MsgCodec|None = None,
-
-) -> tuple[Any, bool]:
-
-    if isinstance(send_items, dict):
-        send_items = send_items.items()
-
-    for (
-        send_type_spec,
-        send_value,
-    ) in send_items:
-
-        expect_roundtrip: bool = False
-
-        # values-to-typespec santiy
-        send_type = type(send_value)
-        assert send_type == send_type_spec or (
-            (subtypes := getattr(send_type_spec, '__args__', None))
-            and send_type in subtypes
-        )
-
-        spec_subtypes: set[Union[Type]] = (
-             getattr(
-                 ipc_pld_spec,
-                 '__args__',
-                 {ipc_pld_spec,},
-             )
-        )
-        send_in_spec: bool = (
-            send_type == ipc_pld_spec
-            or (
-                ipc_pld_spec != Any
-                and  # presume `Union` of types
-                send_type in spec_subtypes
-            )
-            or (
-                ipc_pld_spec == Any
-                and
-                send_type != NamespacePath
-            )
-        )
-        expect_roundtrip = (
-            send_in_spec
-            # any spec should support all other
-            # builtin py values that we send
-            # except our custom nsp type which
-            # we should be able to send as long
-            # as we provide the custom codec hooks.
-            or (
-                ipc_pld_spec == Any
-                and
-                send_type == NamespacePath
-                and
-                add_codec_hooks
-            )
-        )
-
-        if codec is not None:
-            # XXX FIRST XXX ensure roundtripping works
-            # before touching any IPC primitives/APIs.
-            wire_bytes: bytes = codec.encode(
-                Started(
-                    cid='blahblah',
-                    pld=send_value,
-                )
-            )
-            # NOTE: demonstrates the decoder loading
-            # to via our native SCIPP msg-spec
-            # (structurred-conc-inter-proc-protocol)
-            # implemented as per,
-            try:
-                msg: Started = codec.decode(wire_bytes)
-                if not expect_roundtrip:
-                    pytest.fail(
-                        f'NOT-EXPECTED able to roundtrip value given spec:\n'
-                        f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                        f'value -> {send_value}: {send_type}\n'
-                    )
-
-                pld = msg.pld
-                assert pld == send_value
-
-            except ValidationError:
-                if expect_roundtrip:
-                    pytest.fail(
-                        f'EXPECTED to roundtrip value given spec:\n'
-                        f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                        f'value -> {send_value}: {send_type}\n'
-                    )
-
-        yield (
-            str(send_type),
-            send_value,
-            expect_roundtrip,
-        )
-
-
-def dec_type_union(
-    type_names: list[str],
-) -> Type:
-    '''
-    Look up types by name, compile into a list and then create and
-    return a `typing.Union` from the full set.
-
-    '''
-    import importlib
-    types: list[Type] = []
-    for type_name in type_names:
-        for ns in [
-            typing,
-            importlib.import_module(__name__),
-        ]:
-            if type_ref := getattr(
-                ns,
-                type_name,
-                False,
-            ):
-                types.append(type_ref)
-
-    # special case handling only..
-    # ipc_pld_spec: Union[Type] = eval(
-    #     pld_spec_str,
-    #     {},  # globals
-    #     {'typing': typing},  # locals
-    # )
-
-    return Union[*types]
-
-
-def enc_type_union(
-    union_or_type: Union[Type]|Type,
-) -> list[str]:
-    '''
-    Encode a type-union or single type to a list of type-name-strings
-    ready for IPC interchange.
-
-    '''
-    type_strs: list[str] = []
-    for typ in getattr(
-        union_or_type,
-        '__args__',
-        {union_or_type,},
-    ):
-        type_strs.append(typ.__qualname__)
-
-    return type_strs
-
-
-@tractor.context
-async def send_back_values(
-    ctx: Context,
-    expect_debug: bool,
-    pld_spec_type_strs: list[str],
-    add_hooks: bool,
-    started_msg_bytes: bytes,
-    expect_ipc_send: dict[str, tuple[Any, bool]],
-
-) -> None:
-    '''
-    Setup up a custom codec to load instances of `NamespacePath`
-    and ensure we can round trip a func ref with our parent.
-
-    '''
-    uid: tuple = tractor.current_actor().uid
-
-    # debug mode sanity check (prolly superfluous but, meh)
-    assert expect_debug == _state.debug_mode()
-
-    # init state in sub-actor should be default
-    chk_codec_applied(
-        expect_codec=_codec._def_tractor_codec,
-    )
-
-    # load pld spec from input str
-    ipc_pld_spec = dec_type_union(
-        pld_spec_type_strs,
-    )
-    pld_spec_str = str(ipc_pld_spec)
-
-    # same as on parent side config.
-    nsp_codec: MsgCodec = mk_custom_codec(
-        pld_spec=ipc_pld_spec,
-        add_hooks=add_hooks,
-    )
-    with (
-        apply_codec(nsp_codec) as codec,
-    ):
-        chk_codec_applied(
-            expect_codec=nsp_codec,
-            enter_value=codec,
-        )
-
-        print(
-            f'{uid}: attempting `Started`-bytes DECODE..\n'
-        )
-        try:
-            msg: Started = nsp_codec.decode(started_msg_bytes)
-            expected_pld_spec_str: str = msg.pld
-            assert pld_spec_str == expected_pld_spec_str
-
-        # TODO: maybe we should add our own wrapper error so as to
-        # be interchange-lib agnostic?
-        # -[ ] the error type is wtv is raised from the hook so we
-        #   could also require a type-class of errors for
-        #   indicating whether the hook-failure can be handled by
-        #   a nasty-dialog-unprot sub-sys?
-        except ValidationError:
-
-            # NOTE: only in the `Any` spec case do we expect this to
-            # work since otherwise no spec covers a plain-ol'
-            # `.pld: str`
-            if pld_spec_str == 'Any':
-                raise
-            else:
-                print(
-                    f'{uid}: (correctly) unable to DECODE `Started`-bytes\n'
-                    f'{started_msg_bytes}\n'
-                )
-
-        iter_send_val_items = iter(expect_ipc_send.values())
-        sent: list[Any] = []
-        for send_value, expect_send in iter_send_val_items:
-            try:
-                print(
-                    f'{uid}: attempting to `.started({send_value})`\n'
-                    f'=> expect_send: {expect_send}\n'
-                    f'SINCE, ipc_pld_spec: {ipc_pld_spec}\n'
-                    f'AND, codec: {codec}\n'
-                )
-                await ctx.started(send_value)
-                sent.append(send_value)
-                if not expect_send:
-
-                    # XXX NOTE XXX THIS WON'T WORK WITHOUT SPECIAL
-                    # `str` handling! or special debug mode IPC
-                    # msgs!
-                    await tractor.pause()
-
-                    raise RuntimeError(
-                        f'NOT-EXPECTED able to roundtrip value given spec:\n'
-                        f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                        f'value -> {send_value}: {type(send_value)}\n'
-                    )
-
-                break  # move on to streaming block..
-
-            except tractor.MsgTypeError:
-                await tractor.pause()
-
-                if expect_send:
-                    raise RuntimeError(
-                        f'EXPECTED to `.started()` value given spec:\n'
-                        f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                        f'value -> {send_value}: {type(send_value)}\n'
-                    )
-
-        async with ctx.open_stream() as ipc:
-            print(
-                f'{uid}: Entering streaming block to send remaining values..'
-            )
-
-            for send_value, expect_send in iter_send_val_items:
-                send_type: Type = type(send_value)
-                print(
-                    '------ - ------\n'
-                    f'{uid}: SENDING NEXT VALUE\n'
-                    f'ipc_pld_spec: {ipc_pld_spec}\n'
-                    f'expect_send: {expect_send}\n'
-                    f'val: {send_value}\n'
-                    '------ - ------\n'
-                )
-                try:
-                    await ipc.send(send_value)
-                    print(f'***\n{uid}-CHILD sent {send_value!r}\n***\n')
-                    sent.append(send_value)
-
-                    # NOTE: should only raise above on
-                    # `.started()` or a `Return`
-                    # if not expect_send:
-                    #     raise RuntimeError(
-                    #         f'NOT-EXPECTED able to roundtrip value given spec:\n'
-                    #         f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                    #         f'value -> {send_value}: {send_type}\n'
-                    #     )
-
-                except ValidationError:
-                    print(f'{uid} FAILED TO SEND {send_value}!')
-
-                    # await tractor.pause()
-                    if expect_send:
-                        raise RuntimeError(
-                            f'EXPECTED to roundtrip value given spec:\n'
-                            f'ipc_pld_spec -> {ipc_pld_spec}\n'
-                            f'value -> {send_value}: {send_type}\n'
-                        )
-                    # continue
-
-            else:
-                print(
-                    f'{uid}: finished sending all values\n'
-                    'Should be exiting stream block!\n'
-                )
-
-        print(f'{uid}: exited streaming block!')
-
-        # TODO: this won't be true bc in streaming phase we DO NOT
-        # msgspec check outbound msgs!
-        # -[ ] once we implement the receiver side `InvalidMsg`
-        #   then we can expect it here?
-        # assert (
-        #     len(sent)
-        #     ==
-        #     len([val
-        #          for val, expect in
-        #          expect_ipc_send.values()
-        #          if expect is True])
-        # )
-
-
-def ex_func(*args):
-    print(f'ex_func({args})')
-
-
-@pytest.mark.parametrize(
-    'ipc_pld_spec',
-    [
-        Any,
-        NamespacePath,
-        NamespacePath|None,  # the "maybe" spec Bo
-    ],
-    ids=[
-        'any_type',
-        'nsp_type',
-        'maybe_nsp_type',
-    ]
-)
-@pytest.mark.parametrize(
-    'add_codec_hooks',
-    [
-        True,
-        False,
-    ],
-    ids=['use_codec_hooks', 'no_codec_hooks'],
-)
-def test_codec_hooks_mod(
-    debug_mode: bool,
-    ipc_pld_spec: Union[Type]|Any,
-    # send_value: None|str|NamespacePath,
-    add_codec_hooks: bool,
-):
-    '''
-    Audit the `.msg.MsgCodec` override apis details given our impl
-    uses `contextvars` to accomplish per `trio` task codec
-    application around an inter-proc-task-comms context.
-
-    '''
-    async def main():
-        nsp = NamespacePath.from_ref(ex_func)
-        send_items: dict[Union, Any] = {
-            Union[None]: None,
-            Union[NamespacePath]: nsp,
-            Union[str]: str(nsp),
-        }
-
-        # init default state for actor
-        chk_codec_applied(
-            expect_codec=_codec._def_tractor_codec,
-        )
-
-        async with tractor.open_nursery(
-            debug_mode=debug_mode,
-        ) as an:
-            p: tractor.Portal = await an.start_actor(
-                'sub',
-                enable_modules=[__name__],
-            )
-
-            # TODO: 2 cases:
-            # - codec not modified -> decode nsp as `str`
-            # - codec modified with hooks -> decode nsp as
-            #   `NamespacePath`
-            nsp_codec: MsgCodec = mk_custom_codec(
-                pld_spec=ipc_pld_spec,
-                add_hooks=add_codec_hooks,
-            )
-            with apply_codec(nsp_codec) as codec:
-                chk_codec_applied(
-                    expect_codec=nsp_codec,
-                    enter_value=codec,
-                )
-
-                expect_ipc_send: dict[str, tuple[Any, bool]] = {}
-
-                report: str = (
-                    'Parent report on send values with\n'
-                    f'ipc_pld_spec: {ipc_pld_spec}\n'
-                    '       ------ - ------\n'
-                )
-                for val_type_str, val, expect_send in iter_maybe_sends(
-                    send_items,
-                    ipc_pld_spec,
-                    add_codec_hooks=add_codec_hooks,
-                ):
-                    report += (
-                        f'send_value: {val}: {type(val)} '
-                        f'=> expect_send: {expect_send}\n'
-                    )
-                    expect_ipc_send[val_type_str] = (val, expect_send)
-
-                print(
-                    report +
-                    '       ------ - ------\n'
-                )
-                assert len(expect_ipc_send) == len(send_items)
-                # now try over real IPC with a the subactor
-                # expect_ipc_rountrip: bool = True
-                expected_started = Started(
-                    cid='cid',
-                    pld=str(ipc_pld_spec),
-                )
-                # build list of values we expect to receive from
-                # the subactor.
-                expect_to_send: list[Any] = [
-                    val
-                    for val, expect_send in expect_ipc_send.values()
-                    if expect_send
-                ]
-
-                pld_spec_type_strs: list[str] = enc_type_union(ipc_pld_spec)
-
-                # XXX should raise an mte (`MsgTypeError`)
-                # when `add_codec_hooks == False` bc the input
-                # `expect_ipc_send` kwarg has a nsp which can't be
-                # serialized!
-                #
-                # TODO:can we ensure this happens from the
-                # `Return`-side (aka the sub) as well?
-                if not add_codec_hooks:
-                    try:
-                        async with p.open_context(
-                            send_back_values,
-                            expect_debug=debug_mode,
-                            pld_spec_type_strs=pld_spec_type_strs,
-                            add_hooks=add_codec_hooks,
-                            started_msg_bytes=nsp_codec.encode(expected_started),
-
-                            # XXX NOTE bc we send a `NamespacePath` in this kwarg
-                            expect_ipc_send=expect_ipc_send,
-
-                        ) as (ctx, first):
-                            pytest.fail('ctx should fail to open without custom enc_hook!?')
-
-                    # this test passes bc we can go no further!
-                    except MsgTypeError:
-                        # teardown nursery
-                        await p.cancel_actor()
-                        return
-
-                # TODO: send the original nsp here and
-                # test with `limit_msg_spec()` above?
-                # await tractor.pause()
-                print('PARENT opening IPC ctx!\n')
-                async with (
-
-                    # XXX should raise an mte (`MsgTypeError`)
-                    # when `add_codec_hooks == False`..
-                    p.open_context(
-                        send_back_values,
-                        expect_debug=debug_mode,
-                        pld_spec_type_strs=pld_spec_type_strs,
-                        add_hooks=add_codec_hooks,
-                        started_msg_bytes=nsp_codec.encode(expected_started),
-                        expect_ipc_send=expect_ipc_send,
-                    ) as (ctx, first),
-
-                    ctx.open_stream() as ipc,
-                ):
-                    # ensure codec is still applied across
-                    # `tractor.Context` + its embedded nursery.
-                    chk_codec_applied(
-                        expect_codec=nsp_codec,
-                        enter_value=codec,
-                    )
-                    print(
-                        'root: ENTERING CONTEXT BLOCK\n'
-                        f'type(first): {type(first)}\n'
-                        f'first: {first}\n'
-                    )
-                    expect_to_send.remove(first)
-
-                    # TODO: explicit values we expect depending on
-                    # codec config!
-                    # assert first == first_val
-                    # assert first == f'{__name__}:ex_func'
-
-                    async for next_sent in ipc:
-                        print(
-                            'Parent: child sent next value\n'
-                            f'{next_sent}: {type(next_sent)}\n'
-                        )
-                        if expect_to_send:
-                            expect_to_send.remove(next_sent)
-                        else:
-                            print('PARENT should terminate stream loop + block!')
-
-                    # all sent values should have arrived!
-                    assert not expect_to_send
-
-            await p.cancel_actor()
-
-    trio.run(main)
-
-
-def chk_pld_type(
-    payload_spec: Type[Struct]|Any,
-    pld: Any,
-
-    expect_roundtrip: bool|None = None,
-
-) -> bool:
-
-    pld_val_type: Type = type(pld)
-
-    # TODO: verify that the overridden subtypes
-    # DO NOT have modified type-annots from original!
-    # 'Start',  .pld: FuncSpec
-    # 'StartAck',  .pld: IpcCtxSpec
-    # 'Stop',  .pld: UNSEt
-    # 'Error',  .pld: ErrorData
-
-    codec: MsgCodec = mk_codec(
-        # NOTE: this ONLY accepts `Msg.pld` fields of a specified
-        # type union.
-        ipc_pld_spec=payload_spec,
-    )
-
-    # make a one-off dec to compare with our `MsgCodec` instance
-    # which does the below `mk_msg_spec()` call internally
-    ipc_msg_spec: Union[Type[Struct]]
-    msg_types: list[Msg[payload_spec]]
-    (
-        ipc_msg_spec,
-        msg_types,
-    ) = mk_msg_spec(
-        payload_type_union=payload_spec,
-    )
-    _enc = msgpack.Encoder()
-    _dec = msgpack.Decoder(
-        type=ipc_msg_spec or Any,  # like `Msg[Any]`
-    )
-
-    assert (
-        payload_spec
-        ==
-        codec.pld_spec
-    )
-
-    # assert codec.dec == dec
-    #
-    # ^-XXX-^ not sure why these aren't "equal" but when cast
-    # to `str` they seem to match ?? .. kk
-
-    assert (
-        str(ipc_msg_spec)
-        ==
-        str(codec.msg_spec)
-        ==
-        str(_dec.type)
-        ==
-        str(codec.dec.type)
-    )
-
-    # verify the boxed-type for all variable payload-type msgs.
-    if not msg_types:
-        breakpoint()
-
-    roundtrip: bool|None = None
-    pld_spec_msg_names: list[str] = [
-        td.__name__ for td in _payload_msgs
-    ]
-    for typedef in msg_types:
-
-        skip_runtime_msg: bool = typedef.__name__ not in pld_spec_msg_names
-        if skip_runtime_msg:
-            continue
-
-        pld_field = structs.fields(typedef)[1]
-        assert pld_field.type is payload_spec # TODO-^ does this need to work to get all subtypes to adhere?
-
-        kwargs: dict[str, Any] = {
-            'cid': '666',
-            'pld': pld,
-        }
-        enc_msg: Msg = typedef(**kwargs)
-
-        _wire_bytes: bytes = _enc.encode(enc_msg)
-        wire_bytes: bytes = codec.enc.encode(enc_msg)
-        assert _wire_bytes == wire_bytes
-
-        ve: ValidationError|None = None
-        try:
-            dec_msg = codec.dec.decode(wire_bytes)
-            _dec_msg = _dec.decode(wire_bytes)
-
-            # decoded msg and thus payload should be exactly same!
-            assert (roundtrip := (
-                _dec_msg
-                ==
-                dec_msg
-                ==
-                enc_msg
-            ))
-
-            if (
-                expect_roundtrip is not None
-                and expect_roundtrip != roundtrip
-            ):
-                breakpoint()
-
-            assert (
-                pld
-                ==
-                dec_msg.pld
-                ==
-                enc_msg.pld
-            )
-            # assert (roundtrip := (_dec_msg == enc_msg))
-
-        except ValidationError as _ve:
-            ve = _ve
-            roundtrip: bool = False
-            if pld_val_type is payload_spec:
-                raise ValueError(
-                   'Got `ValidationError` despite type-var match!?\n'
-                    f'pld_val_type: {pld_val_type}\n'
-                    f'payload_type: {payload_spec}\n'
-                ) from ve
-
-            else:
-                # ow we good cuz the pld spec mismatched.
-                print(
-                    'Got expected `ValidationError` since,\n'
-                    f'{pld_val_type} is not {payload_spec}\n'
-                )
-        else:
-            if (
-                payload_spec is not Any
-                and
-                pld_val_type is not payload_spec
-            ):
-                raise ValueError(
-                   'DID NOT `ValidationError` despite expected type match!?\n'
-                    f'pld_val_type: {pld_val_type}\n'
-                    f'payload_type: {payload_spec}\n'
-                )
-
-    # full code decode should always be attempted!
-    if roundtrip is None:
-        breakpoint()
-
-    return roundtrip
-
-
-def test_limit_msgspec():
-
-    async def main():
-        async with tractor.open_root_actor(
-            debug_mode=True
-        ):
-
-            # ensure we can round-trip a boxing `Msg`
-            assert chk_pld_type(
-                # Msg,
-                Any,
-                None,
-                expect_roundtrip=True,
-            )
-
-            # TODO: don't need this any more right since
-            # `msgspec>=0.15` has the nice generics stuff yah??
-            #
-            # manually override the type annot of the payload
-            # field and ensure it propagates to all msg-subtypes.
-            # Msg.__annotations__['pld'] = Any
-
-            # verify that a mis-typed payload value won't decode
-            assert not chk_pld_type(
-                # Msg,
-                int,
-                pld='doggy',
-            )
-
-            # parametrize the boxed `.pld` type as a custom-struct
-            # and ensure that parametrization propagates
-            # to all payload-msg-spec-able subtypes!
-            class CustomPayload(Struct):
-                name: str
-                value: Any
-
-            assert not chk_pld_type(
-                # Msg,
-                CustomPayload,
-                pld='doggy',
-            )
-
-            assert chk_pld_type(
-                # Msg,
-                CustomPayload,
-                pld=CustomPayload(name='doggy', value='urmom')
-            )
-
-            # uhh bc we can `.pause_from_sync()` now! :surfer:
-            # breakpoint()
-
-    trio.run(main)

tests/test_child_manages_service_nursery.py

@@ -95,8 +95,8 @@ async def trio_main(
 
     # stash a "service nursery" as "actor local" (aka a Python global)
     global _nursery
-    n = _nursery
-    assert n
+    tn = _nursery
+    assert tn
 
     async def consume_stream():
         async with wrapper_mngr() as stream:
@@ -104,10 +104,10 @@ async def trio_main(
                 print(msg)
 
     # run 2 tasks to ensure broadcaster chan use
-    n.start_soon(consume_stream)
-    n.start_soon(consume_stream)
+    tn.start_soon(consume_stream)
+    tn.start_soon(consume_stream)
 
-    n.start_soon(trio_sleep_and_err)
+    tn.start_soon(trio_sleep_and_err)
 
     await trio.sleep_forever()
 
@@ -117,8 +117,10 @@ async def open_actor_local_nursery(
     ctx: tractor.Context,
 ):
     global _nursery
-    async with trio.open_nursery() as n:
-        _nursery = n
+    async with trio.open_nursery(
+        strict_exception_groups=False,
+    ) as tn:
+        _nursery = tn
         await ctx.started()
         await trio.sleep(10)
         # await trio.sleep(1)
@@ -132,7 +134,7 @@ async def open_actor_local_nursery(
         # never yields back.. aka a scenario where the
         # ``tractor.context`` task IS NOT in the service n's cancel
         # scope.
-        n.cancel_scope.cancel()
+        tn.cancel_scope.cancel()
 
 
 @pytest.mark.parametrize(
@@ -157,7 +159,7 @@ def test_actor_managed_trio_nursery_task_error_cancels_aio(
         async with tractor.open_nursery() as n:
             p = await n.start_actor(
                 'nursery_mngr',
-                infect_asyncio=asyncio_mode,
+                infect_asyncio=asyncio_mode,  # TODO, is this enabling debug mode?
                 enable_modules=[__name__],
             )
             async with (

tests/test_context_stream_semantics.py

@@ -38,9 +38,9 @@ from tractor._testing import (
 # - standard setup/teardown:
 #   ``Portal.open_context()`` starts a new
 #   remote task context in another actor. The target actor's task must
-#   call ``Context.started()`` to unblock this entry on the caller side.
-#   the callee task executes until complete and returns a final value
-#   which is delivered to the caller side and retreived via
+#   call ``Context.started()`` to unblock this entry on the parent side.
+#   the child task executes until complete and returns a final value
+#   which is delivered to the parent side and retreived via
 #   ``Context.result()``.
 
 # - cancel termination:
@@ -170,9 +170,9 @@ async def assert_state(value: bool):
     [False, ValueError, KeyboardInterrupt],
 )
 @pytest.mark.parametrize(
-    'callee_blocks_forever',
+    'child_blocks_forever',
     [False, True],
-    ids=lambda item: f'callee_blocks_forever={item}'
+    ids=lambda item: f'child_blocks_forever={item}'
 )
 @pytest.mark.parametrize(
     'pointlessly_open_stream',
@@ -181,7 +181,7 @@ async def assert_state(value: bool):
 )
 def test_simple_context(
     error_parent,
-    callee_blocks_forever,
+    child_blocks_forever,
     pointlessly_open_stream,
     debug_mode: bool,
 ):
@@ -204,13 +204,13 @@ def test_simple_context(
                         portal.open_context(
                             simple_setup_teardown,
                             data=10,
-                            block_forever=callee_blocks_forever,
+                            block_forever=child_blocks_forever,
                         ) as (ctx, sent),
                     ):
                         assert current_ipc_ctx() is ctx
                         assert sent == 11
 
-                        if callee_blocks_forever:
+                        if child_blocks_forever:
                             await portal.run(assert_state, value=True)
                         else:
                             assert await ctx.result() == 'yo'
@@ -220,7 +220,7 @@ def test_simple_context(
                                 if error_parent:
                                     raise error_parent
 
-                                if callee_blocks_forever:
+                                if child_blocks_forever:
                                     await ctx.cancel()
                                 else:
                                     # in this case the stream will send a
@@ -259,9 +259,9 @@ def test_simple_context(
 
 
 @pytest.mark.parametrize(
-    'callee_returns_early',
+    'child_returns_early',
     [True, False],
-    ids=lambda item: f'callee_returns_early={item}'
+    ids=lambda item: f'child_returns_early={item}'
 )
 @pytest.mark.parametrize(
     'cancel_method',
@@ -273,14 +273,14 @@ def test_simple_context(
     [True, False],
     ids=lambda item: f'chk_ctx_result_before_exit={item}'
 )
-def test_caller_cancels(
+def test_parent_cancels(
     cancel_method: str,
     chk_ctx_result_before_exit: bool,
-    callee_returns_early: bool,
+    child_returns_early: bool,
     debug_mode: bool,
 ):
     '''
-    Verify that when the opening side of a context (aka the caller)
+    Verify that when the opening side of a context (aka the parent)
     cancels that context, the ctx does not raise a cancelled when
     either calling `.result()` or on context exit.
 
@@ -294,7 +294,7 @@ def test_caller_cancels(
 
         if (
             cancel_method == 'portal'
-            and not callee_returns_early
+            and not child_returns_early
         ):
             try:
                 res = await ctx.result()
@@ -318,7 +318,7 @@ def test_caller_cancels(
                 pytest.fail(f'should not have raised ctxc\n{ctxc}')
 
         # we actually get a result
-        if callee_returns_early:
+        if child_returns_early:
             assert res == 'yo'
             assert ctx.outcome is res
             assert ctx.maybe_error is None
@@ -362,14 +362,14 @@ def test_caller_cancels(
             )
             timeout: float = (
                 0.5
-                if not callee_returns_early
+                if not child_returns_early
                 else 2
             )
             with trio.fail_after(timeout):
                 async with (
                     expect_ctxc(
                         yay=(
-                            not callee_returns_early
+                            not child_returns_early
                             and cancel_method == 'portal'
                         )
                     ),
@@ -377,13 +377,13 @@ def test_caller_cancels(
                     portal.open_context(
                         simple_setup_teardown,
                         data=10,
-                        block_forever=not callee_returns_early,
+                        block_forever=not child_returns_early,
                     ) as (ctx, sent),
                 ):
 
-                    if callee_returns_early:
+                    if child_returns_early:
                         # ensure we block long enough before sending
-                        # a cancel such that the callee has already
+                        # a cancel such that the child has already
                         # returned it's result.
                         await trio.sleep(0.5)
 
@@ -421,7 +421,7 @@ def test_caller_cancels(
             #   which should in turn cause `ctx._scope` to
             # catch any cancellation?
             if (
-                not callee_returns_early
+                not child_returns_early
                 and cancel_method != 'portal'
             ):
                 assert not ctx._scope.cancelled_caught
@@ -430,11 +430,11 @@ def test_caller_cancels(
 
 
 # basic stream terminations:
-# - callee context closes without using stream
-# - caller context closes without using stream
-# - caller context calls `Context.cancel()` while streaming
-#   is ongoing resulting in callee being cancelled
-# - callee calls `Context.cancel()` while streaming and caller
+# - child context closes without using stream
+# - parent context closes without using stream
+# - parent context calls `Context.cancel()` while streaming
+#   is ongoing resulting in child being cancelled
+# - child calls `Context.cancel()` while streaming and parent
 #   sees stream terminated in `RemoteActorError`
 
 # TODO: future possible features
@@ -443,7 +443,6 @@ def test_caller_cancels(
 
 @tractor.context
 async def close_ctx_immediately(
-
     ctx: Context,
 
 ) -> None:
@@ -454,13 +453,24 @@ async def close_ctx_immediately(
     async with ctx.open_stream():
         pass
 
+    print('child returning!')
 
+
+@pytest.mark.parametrize(
+    'parent_send_before_receive',
+    [
+        False,
+        True,
+    ],
+    ids=lambda item: f'child_send_before_receive={item}'
+)
 @tractor_test
-async def test_callee_closes_ctx_after_stream_open(
+async def test_child_exits_ctx_after_stream_open(
     debug_mode: bool,
+    parent_send_before_receive: bool,
 ):
     '''
-    callee context closes without using stream.
+    child context closes without using stream.
 
     This should result in a msg sequence
     |_<root>_
@@ -474,6 +484,9 @@ async def test_callee_closes_ctx_after_stream_open(
     => {'stop': True, 'cid': <str>}
 
     '''
+    timeout: float = (
+        0.5 if not debug_mode else 999
+    )
     async with tractor.open_nursery(
         debug_mode=debug_mode,
     ) as an:
@@ -482,7 +495,7 @@ async def test_callee_closes_ctx_after_stream_open(
             enable_modules=[__name__],
         )
 
-        with trio.fail_after(0.5):
+        with trio.fail_after(timeout):
             async with portal.open_context(
                 close_ctx_immediately,
 
@@ -494,41 +507,56 @@ async def test_callee_closes_ctx_after_stream_open(
 
                 with trio.fail_after(0.4):
                     async with ctx.open_stream() as stream:
+                        if parent_send_before_receive:
+                            print('sending first msg from parent!')
+                            await stream.send('yo')
 
                         # should fall through since ``StopAsyncIteration``
                         # should be raised through translation of
                         # a ``trio.EndOfChannel`` by
                         # ``trio.abc.ReceiveChannel.__anext__()``
-                        async for _ in stream:
+                        msg = 10
+                        async for msg in stream:
                             # trigger failure if we DO NOT
                             # get an EOC!
                             assert 0
                         else:
+                            # never should get anythinig new from
+                            # the underlying stream
+                            assert msg == 10
 
                             # verify stream is now closed
                             try:
                                 with trio.fail_after(0.3):
+                                    print('parent trying to `.receive()` on EoC stream!')
                                     await stream.receive()
+                                    assert 0, 'should have raised eoc!?'
                             except trio.EndOfChannel:
+                                print('parent got EoC as expected!')
                                 pass
+                                # raise
 
                 # TODO: should be just raise the closed resource err
                 # directly here to enforce not allowing a re-open
                 # of a stream to the context (at least until a time of
                 # if/when we decide that's a good idea?)
                 try:
-                    with trio.fail_after(0.5):
+                    with trio.fail_after(timeout):
                         async with ctx.open_stream() as stream:
                             pass
                 except trio.ClosedResourceError:
                     pass
 
+                # if ctx._rx_chan._state.data:
+                #     await tractor.pause()
+
         await portal.cancel_actor()
 
 
 @tractor.context
 async def expect_cancelled(
     ctx: Context,
+    send_before_receive: bool = False,
 
 ) -> None:
     global _state
@@ -538,6 +566,10 @@ async def expect_cancelled(
 
     try:
         async with ctx.open_stream() as stream:
+
+            if send_before_receive:
+                await stream.send('yo')
+
             async for msg in stream:
                 await stream.send(msg)  # echo server
 
@@ -564,26 +596,49 @@ async def expect_cancelled(
         raise
 
     else:
-        assert 0, "callee wasn't cancelled !?"
+        assert 0, "child wasn't cancelled !?"
 
 
+@pytest.mark.parametrize(
+    'child_send_before_receive',
+    [
+        False,
+        True,
+    ],
+    ids=lambda item: f'child_send_before_receive={item}'
+)
+@pytest.mark.parametrize(
+    'rent_wait_for_msg',
+    [
+        False,
+        True,
+    ],
+    ids=lambda item: f'rent_wait_for_msg={item}'
+)
 @pytest.mark.parametrize(
     'use_ctx_cancel_method',
-    [False, True],
+    [
+        False,
+        'pre_stream',
+        'post_stream_open',
+        'post_stream_close',
+    ],
+    ids=lambda item: f'use_ctx_cancel_method={item}'
 )
 @tractor_test
-async def test_caller_closes_ctx_after_callee_opens_stream(
-    use_ctx_cancel_method: bool,
+async def test_parent_exits_ctx_after_child_enters_stream(
+    use_ctx_cancel_method: bool|str,
     debug_mode: bool,
+    rent_wait_for_msg: bool,
+    child_send_before_receive: bool,
 ):
     '''
-    caller context closes without using/opening stream
+    Parent-side of IPC context closes without sending on `MsgStream`.
 
     '''
     async with tractor.open_nursery(
         debug_mode=debug_mode,
     ) as an:
-
         root: Actor = current_actor()
         portal = await an.start_actor(
             'ctx_cancelled',
@@ -592,41 +647,52 @@ async def test_caller_closes_ctx_after_callee_opens_stream(
 
         async with portal.open_context(
             expect_cancelled,
+            send_before_receive=child_send_before_receive,
         ) as (ctx, sent):
             assert sent is None
 
             await portal.run(assert_state, value=True)
 
             # call `ctx.cancel()` explicitly
-            if use_ctx_cancel_method:
+            if use_ctx_cancel_method == 'pre_stream':
                 await ctx.cancel()
 
                 # NOTE: means the local side `ctx._scope` will
                 # have been cancelled by an ctxc ack and thus
                 # `._scope.cancelled_caught` should be set.
-                try:
+                async with (
+                    expect_ctxc(
+                        # XXX: the cause is US since we call
+                        # `Context.cancel()` just above!
+                        yay=True,
+
+                        # XXX: must be propagated to __aexit__
+                        # and should be silently absorbed there
+                        # since we called `.cancel()` just above ;)
+                        reraise=True,
+                    ) as maybe_ctxc,
+                ):
                     async with ctx.open_stream() as stream:
-                        async for msg in stream:
-                            pass
 
-                except tractor.ContextCancelled as ctxc:
-                    # XXX: the cause is US since we call
-                    # `Context.cancel()` just above!
-                    assert (
-                        ctxc.canceller
-                        ==
-                        current_actor().uid
-                        ==
-                        root.uid
-                    )
+                        if rent_wait_for_msg:
+                            async for msg in stream:
+                                print(f'PARENT rx: {msg!r}\n')
+                                break
 
-                    # XXX: must be propagated to __aexit__
-                    # and should be silently absorbed there
-                    # since we called `.cancel()` just above ;)
-                    raise
+                        if use_ctx_cancel_method == 'post_stream_open':
+                            await ctx.cancel()
 
-                else:
-                    assert 0, "Should have context cancelled?"
+                    if use_ctx_cancel_method == 'post_stream_close':
+                        await ctx.cancel()
+
+                ctxc: tractor.ContextCancelled = maybe_ctxc.value
+                assert (
+                    ctxc.canceller
+                    ==
+                    current_actor().uid
+                    ==
+                    root.uid
+                )
 
                 # channel should still be up
                 assert portal.channel.connected()
@@ -637,13 +703,20 @@ async def test_caller_closes_ctx_after_callee_opens_stream(
                     value=False,
                 )
 
+            # XXX CHILD-BLOCKS case, we SHOULD NOT exit from the
+            # `.open_context()` before the child has returned,
+            # errored or been cancelled!
             else:
                 try:
-                    with trio.fail_after(0.2):
-                        await ctx.result()
+                    with trio.fail_after(
+                        0.5  # if not debug_mode else 999
+                    ):
+                        res = await ctx.wait_for_result()
+                        assert res is not tractor._context.Unresolved
                         assert 0, "Callee should have blocked!?"
                 except trio.TooSlowError:
-                    # NO-OP -> since already called above
+                    # NO-OP -> since already triggered by
+                    # `trio.fail_after()` above!
                     await ctx.cancel()
 
         # NOTE: local scope should have absorbed the cancellation since
@@ -683,7 +756,7 @@ async def test_caller_closes_ctx_after_callee_opens_stream(
 
 
 @tractor_test
-async def test_multitask_caller_cancels_from_nonroot_task(
+async def test_multitask_parent_cancels_from_nonroot_task(
     debug_mode: bool,
 ):
     async with tractor.open_nursery(
@@ -735,7 +808,6 @@ async def test_multitask_caller_cancels_from_nonroot_task(
 
 @tractor.context
 async def cancel_self(
-
     ctx: Context,
 
 ) -> None:
@@ -775,11 +847,11 @@ async def cancel_self(
 
 
 @tractor_test
-async def test_callee_cancels_before_started(
+async def test_child_cancels_before_started(
     debug_mode: bool,
 ):
     '''
-    Callee calls `Context.cancel()` while streaming and caller
+    Callee calls `Context.cancel()` while streaming and parent
     sees stream terminated in `ContextCancelled`.
 
     '''
@@ -826,14 +898,13 @@ async def never_open_stream(
 
 
 @tractor.context
-async def keep_sending_from_callee(
-
+async def keep_sending_from_child(
     ctx:  Context,
     msg_buffer_size: int|None = None,
 
 ) -> None:
     '''
-    Send endlessly on the calleee stream.
+    Send endlessly on the child stream.
 
     '''
     await ctx.started()
@@ -841,7 +912,7 @@ async def keep_sending_from_callee(
         msg_buffer_size=msg_buffer_size,
     ) as stream:
         for msg in count():
-            print(f'callee sending {msg}')
+            print(f'child sending {msg}')
             await stream.send(msg)
             await trio.sleep(0.01)
 
@@ -849,12 +920,12 @@ async def keep_sending_from_callee(
 @pytest.mark.parametrize(
     'overrun_by',
     [
-        ('caller', 1, never_open_stream),
-        ('callee', 0, keep_sending_from_callee),
+        ('parent', 1, never_open_stream),
+        ('child', 0, keep_sending_from_child),
     ],
     ids=[
-         ('caller_1buf_never_open_stream'),
-         ('callee_0buf_keep_sending_from_callee'),
+         ('parent_1buf_never_open_stream'),
+         ('child_0buf_keep_sending_from_child'),
     ]
 )
 def test_one_end_stream_not_opened(
@@ -885,8 +956,7 @@ def test_one_end_stream_not_opened(
                 ) as (ctx, sent):
                     assert sent is None
 
-                    if 'caller' in overrunner:
-
+                    if 'parent' in overrunner:
                         async with ctx.open_stream() as stream:
 
                             # itersend +1 msg more then the buffer size
@@ -901,7 +971,7 @@ def test_one_end_stream_not_opened(
                                 await trio.sleep_forever()
 
                     else:
-                        # callee overruns caller case so we do nothing here
+                        # child overruns parent case so we do nothing here
                         await trio.sleep_forever()
 
             await portal.cancel_actor()
@@ -909,19 +979,19 @@ def test_one_end_stream_not_opened(
     # 2 overrun cases and the no overrun case (which pushes right up to
     # the msg limit)
     if (
-        overrunner == 'caller'
+        overrunner == 'parent'
     ):
         with pytest.raises(tractor.RemoteActorError) as excinfo:
             trio.run(main)
 
         assert excinfo.value.boxed_type == StreamOverrun
 
-    elif overrunner == 'callee':
+    elif overrunner == 'child':
         with pytest.raises(tractor.RemoteActorError) as excinfo:
             trio.run(main)
 
         # TODO: embedded remote errors so that we can verify the source
-        # error? the callee delivers an error which is an overrun
+        # error? the child delivers an error which is an overrun
         # wrapped in a remote actor error.
         assert excinfo.value.boxed_type == tractor.RemoteActorError
 
@@ -931,8 +1001,7 @@ def test_one_end_stream_not_opened(
 
 @tractor.context
 async def echo_back_sequence(
-
-    ctx:  Context,
+    ctx: Context,
     seq: list[int],
     wait_for_cancel: bool,
     allow_overruns_side: str,
@@ -941,12 +1010,12 @@ async def echo_back_sequence(
 
 ) -> None:
     '''
-    Send endlessly on the calleee stream using a small buffer size
+    Send endlessly on the child stream using a small buffer size
     setting on the contex to simulate backlogging that would normally
     cause overruns.
 
     '''
-    # NOTE: ensure that if the caller is expecting to cancel this task
+    # NOTE: ensure that if the parent is expecting to cancel this task
     # that we stay echoing much longer then they are so we don't
     # return early instead of receive the cancel msg.
     total_batches: int = (
@@ -955,7 +1024,7 @@ async def echo_back_sequence(
     )
 
     await ctx.started()
-    # await tractor.breakpoint()
+    # await tractor.pause()
     async with ctx.open_stream(
         msg_buffer_size=msg_buffer_size,
 
@@ -996,18 +1065,18 @@ async def echo_back_sequence(
                 if be_slow:
                     await trio.sleep(0.05)
 
-                print('callee waiting on next')
+                print('child waiting on next')
 
-            print(f'callee echoing back latest batch\n{batch}')
+            print(f'child echoing back latest batch\n{batch}')
             for msg in batch:
-                print(f'callee sending msg\n{msg}')
+                print(f'child sending msg\n{msg}')
                 await stream.send(msg)
 
     try:
         return 'yo'
     finally:
         print(
-            'exiting callee with context:\n'
+            'exiting child with context:\n'
             f'{pformat(ctx)}\n'
         )
 
@@ -1061,7 +1130,7 @@ def test_maybe_allow_overruns_stream(
             debug_mode=debug_mode,
         ) as an:
             portal = await an.start_actor(
-                'callee_sends_forever',
+                'child_sends_forever',
                 enable_modules=[__name__],
                 loglevel=loglevel,
                 debug_mode=debug_mode,

tests/test_discovery.py

@@ -26,7 +26,7 @@ async def test_reg_then_unreg(reg_addr):
         portal = await n.start_actor('actor', enable_modules=[__name__])
         uid = portal.channel.uid
 
-        async with tractor.get_arbiter(*reg_addr) as aportal:
+        async with tractor.get_registry(*reg_addr) as aportal:
             # this local actor should be the arbiter
             assert actor is aportal.actor
 
@@ -160,7 +160,7 @@ async def spawn_and_check_registry(
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
     ):
-        async with tractor.get_arbiter(*reg_addr) as portal:
+        async with tractor.get_registry(*reg_addr) as portal:
             # runtime needs to be up to call this
             actor = tractor.current_actor()
 
@@ -181,7 +181,9 @@ async def spawn_and_check_registry(
 
             try:
                 async with tractor.open_nursery() as n:
-                    async with trio.open_nursery() as trion:
+                    async with trio.open_nursery(
+                        strict_exception_groups=False,
+                    ) as trion:
 
                         portals = {}
                         for i in range(3):
@@ -298,7 +300,7 @@ async def close_chans_before_nursery(
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
     ):
-        async with tractor.get_arbiter(*reg_addr) as aportal:
+        async with tractor.get_registry(*reg_addr) as aportal:
             try:
                 get_reg = partial(unpack_reg, aportal)
 
@@ -316,7 +318,9 @@ async def close_chans_before_nursery(
                         async with portal2.open_stream_from(
                             stream_forever
                         ) as agen2:
-                            async with trio.open_nursery() as n:
+                            async with trio.open_nursery(
+                                strict_exception_groups=False,
+                            ) as n:
                                 n.start_soon(streamer, agen1)
                                 n.start_soon(cancel, use_signal, .5)
                                 try:

tests/test_docs_examples.py

@@ -19,7 +19,7 @@ from tractor._testing import (
 @pytest.fixture
 def run_example_in_subproc(
     loglevel: str,
-    testdir,
+    testdir: pytest.Pytester,
     reg_addr: tuple[str, int],
 ):
 
@@ -81,27 +81,36 @@ def run_example_in_subproc(
 
     # walk yields: (dirpath, dirnames, filenames)
     [
-        (p[0], f) for p in os.walk(examples_dir()) for f in p[2]
+        (p[0], f)
+        for p in os.walk(examples_dir())
+        for f in p[2]
 
-        if '__' not in f
-        and f[0] != '_'
-        and 'debugging' not in p[0]
-        and 'integration' not in p[0]
-        and 'advanced_faults' not in p[0]
+        if (
+            '__' not in f
+            and f[0] != '_'
+            and 'debugging' not in p[0]
+            and 'integration' not in p[0]
+            and 'advanced_faults' not in p[0]
+            and 'multihost' not in p[0]
+        )
     ],
-
     ids=lambda t: t[1],
 )
-def test_example(run_example_in_subproc, example_script):
-    """Load and run scripts from this repo's ``examples/`` dir as a user
+def test_example(
+    run_example_in_subproc,
+    example_script,
+):
+    '''
+    Load and run scripts from this repo's ``examples/`` dir as a user
     would copy and pasing them into their editor.
 
     On windows a little more "finessing" is done to make
     ``multiprocessing`` play nice: we copy the ``__main__.py`` into the
     test directory and invoke the script as a module with ``python -m
     test_example``.
-    """
-    ex_file = os.path.join(*example_script)
+
+    '''
+    ex_file: str = os.path.join(*example_script)
 
     if 'rpc_bidir_streaming' in ex_file and sys.version_info < (3, 9):
         pytest.skip("2-way streaming example requires py3.9 async with syntax")
@@ -127,7 +136,8 @@ def test_example(run_example_in_subproc, example_script):
                     # shouldn't eventually once we figure out what's
                     # a better way to be explicit about aio side
                     # cancels?
-                    and 'asyncio.exceptions.CancelledError' not in last_error
+                    and
+                    'asyncio.exceptions.CancelledError' not in last_error
                 ):
                     raise Exception(errmsg)
 

tests/test_ext_types_msgspec.py

@@ -0,0 +1,946 @@
+'''
+Low-level functional audits for our
+"capability based messaging"-spec feats.
+
+B~)
+
+'''
+from contextlib import (
+    contextmanager as cm,
+    # nullcontext,
+)
+import importlib
+from typing import (
+    Any,
+    Type,
+    Union,
+)
+
+from msgspec import (
+    # structs,
+    # msgpack,
+    Raw,
+    # Struct,
+    ValidationError,
+)
+import pytest
+import trio
+
+import tractor
+from tractor import (
+    Actor,
+    # _state,
+    MsgTypeError,
+    Context,
+)
+from tractor.msg import (
+    _codec,
+    _ctxvar_MsgCodec,
+    _exts,
+
+    NamespacePath,
+    MsgCodec,
+    MsgDec,
+    mk_codec,
+    mk_dec,
+    apply_codec,
+    current_codec,
+)
+from tractor.msg.types import (
+    log,
+    Started,
+    # _payload_msgs,
+    # PayloadMsg,
+    # mk_msg_spec,
+)
+from tractor.msg._ops import (
+    limit_plds,
+)
+
+def enc_nsp(obj: Any) -> Any:
+    actor: Actor = tractor.current_actor(
+        err_on_no_runtime=False,
+    )
+    uid: tuple[str, str]|None = None if not actor else actor.uid
+    print(f'{uid} ENC HOOK')
+
+    match obj:
+        # case NamespacePath()|str():
+        case NamespacePath():
+            encoded: str = str(obj)
+            print(
+                f'----- ENCODING `NamespacePath` as `str` ------\n'
+                f'|_obj:{type(obj)!r} = {obj!r}\n'
+                f'|_encoded: str = {encoded!r}\n'
+            )
+            # if type(obj) != NamespacePath:
+            #     breakpoint()
+            return encoded
+        case _:
+            logmsg: str = (
+                f'{uid}\n'
+                'FAILED ENCODE\n'
+                f'obj-> `{obj}: {type(obj)}`\n'
+            )
+            raise NotImplementedError(logmsg)
+
+
+def dec_nsp(
+    obj_type: Type,
+    obj: Any,
+
+) -> Any:
+    # breakpoint()
+    actor: Actor = tractor.current_actor(
+        err_on_no_runtime=False,
+    )
+    uid: tuple[str, str]|None = None if not actor else actor.uid
+    print(
+        f'{uid}\n'
+        'CUSTOM DECODE\n'
+        f'type-arg-> {obj_type}\n'
+        f'obj-arg-> `{obj}`: {type(obj)}\n'
+    )
+    nsp = None
+    # XXX, never happens right?
+    if obj_type is Raw:
+        breakpoint()
+
+    if (
+        obj_type is NamespacePath
+        and isinstance(obj, str)
+        and ':' in obj
+    ):
+        nsp = NamespacePath(obj)
+        # TODO: we could built a generic handler using
+        # JUST matching the obj_type part?
+        # nsp = obj_type(obj)
+
+    if nsp:
+        print(f'Returning NSP instance: {nsp}')
+        return nsp
+
+    logmsg: str = (
+        f'{uid}\n'
+        'FAILED DECODE\n'
+        f'type-> {obj_type}\n'
+        f'obj-arg-> `{obj}`: {type(obj)}\n\n'
+        f'current codec:\n'
+        f'{current_codec()}\n'
+    )
+    # TODO: figure out the ignore subsys for this!
+    # -[ ] option whether to defense-relay backc the msg
+    #   inside an `Invalid`/`Ignore`
+    # -[ ] how to make this handling pluggable such that a
+    #   `Channel`/`MsgTransport` can intercept and process
+    #   back msgs either via exception handling or some other
+    #   signal?
+    log.warning(logmsg)
+    # NOTE: this delivers the invalid
+    # value up to `msgspec`'s decoding
+    # machinery for error raising.
+    return obj
+    # raise NotImplementedError(logmsg)
+
+
+def ex_func(*args):
+    '''
+    A mod level func we can ref and load via our `NamespacePath`
+    python-object pointer `str` subtype.
+
+    '''
+    print(f'ex_func({args})')
+
+
+@pytest.mark.parametrize(
+    'add_codec_hooks',
+    [
+        True,
+        False,
+    ],
+    ids=['use_codec_hooks', 'no_codec_hooks'],
+)
+def test_custom_extension_types(
+    debug_mode: bool,
+    add_codec_hooks: bool
+):
+    '''
+    Verify that a `MsgCodec` (used for encoding all outbound IPC msgs
+    and decoding all inbound `PayloadMsg`s) and a paired `MsgDec`
+    (used for decoding the `PayloadMsg.pld: Raw` received within a given
+    task's ipc `Context` scope) can both send and receive "extension types"
+    as supported via custom converter hooks passed to `msgspec`.
+
+    '''
+    nsp_pld_dec: MsgDec = mk_dec(
+        spec=None,  # ONLY support the ext type
+        dec_hook=dec_nsp if add_codec_hooks else None,
+        ext_types=[NamespacePath],
+    )
+    nsp_codec: MsgCodec = mk_codec(
+        # ipc_pld_spec=Raw,  # default!
+
+        # NOTE XXX: the encode hook MUST be used no matter what since
+        # our `NamespacePath` is not any of a `Any` native type nor
+        # a `msgspec.Struct` subtype - so `msgspec` has no way to know
+        # how to encode it unless we provide the custom hook.
+        #
+        # AGAIN that is, regardless of whether we spec an
+        # `Any`-decoded-pld the enc has no knowledge (by default)
+        # how to enc `NamespacePath` (nsp), so we add a custom
+        # hook to do that ALWAYS.
+        enc_hook=enc_nsp if add_codec_hooks else None,
+
+        # XXX NOTE: pretty sure this is mutex with the `type=` to
+        # `Decoder`? so it won't work in tandem with the
+        # `ipc_pld_spec` passed above?
+        ext_types=[NamespacePath],
+
+        # TODO? is it useful to have the `.pld` decoded *prior* to
+        # the `PldRx`?? like perf or mem related?
+        # ext_dec=nsp_pld_dec,
+    )
+    if add_codec_hooks:
+        assert nsp_codec.dec.dec_hook is None
+
+        # TODO? if we pass `ext_dec` above?
+        # assert nsp_codec.dec.dec_hook is dec_nsp
+
+        assert nsp_codec.enc.enc_hook is enc_nsp
+
+    nsp = NamespacePath.from_ref(ex_func)
+
+    try:
+        nsp_bytes: bytes = nsp_codec.encode(nsp)
+        nsp_rt_sin_msg = nsp_pld_dec.decode(nsp_bytes)
+        nsp_rt_sin_msg.load_ref() is ex_func
+    except TypeError:
+        if not add_codec_hooks:
+            pass
+
+    try:
+        msg_bytes: bytes = nsp_codec.encode(
+            Started(
+                cid='cid',
+                pld=nsp,
+            )
+        )
+        # since the ext-type obj should also be set as the msg.pld
+        assert nsp_bytes in msg_bytes
+        started_rt: Started = nsp_codec.decode(msg_bytes)
+        pld: Raw = started_rt.pld
+        assert isinstance(pld, Raw)
+        nsp_rt: NamespacePath = nsp_pld_dec.decode(pld)
+        assert isinstance(nsp_rt, NamespacePath)
+        # in obj comparison terms they should be the same
+        assert nsp_rt == nsp
+        # ensure we've decoded to ext type!
+        assert nsp_rt.load_ref() is ex_func
+
+    except TypeError:
+        if not add_codec_hooks:
+            pass
+
+@tractor.context
+async def sleep_forever_in_sub(
+    ctx: Context,
+) -> None:
+    await trio.sleep_forever()
+
+
+def mk_custom_codec(
+    add_hooks: bool,
+
+) -> tuple[
+    MsgCodec,  # encode to send
+    MsgDec,  # pld receive-n-decode
+]:
+    '''
+    Create custom `msgpack` enc/dec-hooks and set a `Decoder`
+    which only loads `pld_spec` (like `NamespacePath`) types.
+
+    '''
+
+    # XXX NOTE XXX: despite defining `NamespacePath` as a type
+    # field on our `PayloadMsg.pld`, we still need a enc/dec_hook() pair
+    # to cast to/from that type on the wire. See the docs:
+    # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
+
+    # if pld_spec is Any:
+    #     pld_spec = Raw
+
+    nsp_codec: MsgCodec = mk_codec(
+        # ipc_pld_spec=Raw,  # default!
+
+        # NOTE XXX: the encode hook MUST be used no matter what since
+        # our `NamespacePath` is not any of a `Any` native type nor
+        # a `msgspec.Struct` subtype - so `msgspec` has no way to know
+        # how to encode it unless we provide the custom hook.
+        #
+        # AGAIN that is, regardless of whether we spec an
+        # `Any`-decoded-pld the enc has no knowledge (by default)
+        # how to enc `NamespacePath` (nsp), so we add a custom
+        # hook to do that ALWAYS.
+        enc_hook=enc_nsp if add_hooks else None,
+
+        # XXX NOTE: pretty sure this is mutex with the `type=` to
+        # `Decoder`? so it won't work in tandem with the
+        # `ipc_pld_spec` passed above?
+        ext_types=[NamespacePath],
+    )
+    # dec_hook=dec_nsp if add_hooks else None,
+    return nsp_codec
+
+
+@pytest.mark.parametrize(
+    'limit_plds_args',
+    [
+        (
+            {'dec_hook': None, 'ext_types': None},
+            None,
+        ),
+        (
+            {'dec_hook': dec_nsp, 'ext_types': None},
+            TypeError,
+        ),
+        (
+            {'dec_hook': dec_nsp, 'ext_types': [NamespacePath]},
+            None,
+        ),
+        (
+            {'dec_hook': dec_nsp, 'ext_types': [NamespacePath|None]},
+            None,
+        ),
+    ],
+    ids=[
+        'no_hook_no_ext_types',
+        'only_hook',
+        'hook_and_ext_types',
+        'hook_and_ext_types_w_null',
+    ]
+)
+def test_pld_limiting_usage(
+    limit_plds_args: tuple[dict, Exception|None],
+):
+    '''
+    Verify `dec_hook()` and `ext_types` need to either both be
+    provided or we raise a explanator type-error.
+
+    '''
+    kwargs, maybe_err = limit_plds_args
+    async def main():
+        async with tractor.open_nursery() as an:  # just to open runtime
+
+            # XXX SHOULD NEVER WORK outside an ipc ctx scope!
+            try:
+                with limit_plds(**kwargs):
+                    pass
+            except RuntimeError:
+                pass
+
+            p: tractor.Portal = await an.start_actor(
+                'sub',
+                enable_modules=[__name__],
+            )
+            async with (
+                p.open_context(
+                    sleep_forever_in_sub
+                ) as (ctx, first),
+            ):
+                try:
+                    with limit_plds(**kwargs):
+                        pass
+                except maybe_err as exc:
+                    assert type(exc) is maybe_err
+                    pass
+
+
+def chk_codec_applied(
+    expect_codec: MsgCodec|None,
+    enter_value: MsgCodec|None = None,
+
+) -> MsgCodec:
+    '''
+    buncha sanity checks ensuring that the IPC channel's
+    context-vars are set to the expected codec and that are
+    ctx-var wrapper APIs match the same.
+
+    '''
+    # TODO: play with tricyle again, bc this is supposed to work
+    # the way we want?
+    #
+    # TreeVar
+    # task: trio.Task = trio.lowlevel.current_task()
+    # curr_codec = _ctxvar_MsgCodec.get_in(task)
+
+    # ContextVar
+    # task_ctx: Context = task.context
+    # assert _ctxvar_MsgCodec in task_ctx
+    # curr_codec: MsgCodec = task.context[_ctxvar_MsgCodec]
+    if expect_codec is None:
+        assert enter_value is None
+        return
+
+    # NOTE: currently we use this!
+    # RunVar
+    curr_codec: MsgCodec = current_codec()
+    last_read_codec = _ctxvar_MsgCodec.get()
+    # assert curr_codec is last_read_codec
+
+    assert (
+        (same_codec := expect_codec) is
+        # returned from `mk_codec()`
+
+        # yielded value from `apply_codec()`
+
+        # read from current task's `contextvars.Context`
+        curr_codec is
+        last_read_codec
+
+        # the default `msgspec` settings
+        is not _codec._def_msgspec_codec
+        is not _codec._def_tractor_codec
+    )
+
+    if enter_value:
+        assert enter_value is same_codec
+
+
+@tractor.context
+async def send_back_values(
+    ctx: Context,
+    rent_pld_spec_type_strs: list[str],
+    add_hooks: bool,
+
+) -> None:
+    '''
+    Setup up a custom codec to load instances of `NamespacePath`
+    and ensure we can round trip a func ref with our parent.
+
+    '''
+    uid: tuple = tractor.current_actor().uid
+
+    # init state in sub-actor should be default
+    chk_codec_applied(
+        expect_codec=_codec._def_tractor_codec,
+    )
+
+    # load pld spec from input str
+    rent_pld_spec = _exts.dec_type_union(
+        rent_pld_spec_type_strs,
+        mods=[
+            importlib.import_module(__name__),
+        ],
+    )
+    rent_pld_spec_types: set[Type] = _codec.unpack_spec_types(
+        rent_pld_spec,
+    )
+
+    # ONLY add ext-hooks if the rent specified a non-std type!
+    add_hooks: bool = (
+        NamespacePath in rent_pld_spec_types
+        and
+        add_hooks
+    )
+
+    # same as on parent side config.
+    nsp_codec: MsgCodec|None = None
+    if add_hooks:
+        nsp_codec = mk_codec(
+            enc_hook=enc_nsp,
+            ext_types=[NamespacePath],
+        )
+
+    with (
+        maybe_apply_codec(nsp_codec) as codec,
+        limit_plds(
+            rent_pld_spec,
+            dec_hook=dec_nsp if add_hooks else None,
+            ext_types=[NamespacePath]  if add_hooks else None,
+        ) as pld_dec,
+    ):
+        # ?XXX? SHOULD WE NOT be swapping the global codec since it
+        # breaks `Context.started()` roundtripping checks??
+        chk_codec_applied(
+            expect_codec=nsp_codec,
+            enter_value=codec,
+        )
+
+        # ?TODO, mismatch case(s)?
+        #
+        # ensure pld spec matches on both sides
+        ctx_pld_dec: MsgDec = ctx._pld_rx._pld_dec
+        assert pld_dec is ctx_pld_dec
+        child_pld_spec: Type = pld_dec.spec
+        child_pld_spec_types: set[Type] = _codec.unpack_spec_types(
+            child_pld_spec,
+        )
+        assert (
+            child_pld_spec_types.issuperset(
+                rent_pld_spec_types
+            )
+        )
+
+        # ?TODO, try loop for each of the types in pld-superset?
+        #
+        # for send_value in [
+        #     nsp,
+        #     str(nsp),
+        #     None,
+        # ]:
+        nsp = NamespacePath.from_ref(ex_func)
+        try:
+            print(
+                f'{uid}: attempting to `.started({nsp})`\n'
+                f'\n'
+                f'rent_pld_spec: {rent_pld_spec}\n'
+                f'child_pld_spec: {child_pld_spec}\n'
+                f'codec: {codec}\n'
+            )
+            # await tractor.pause()
+            await ctx.started(nsp)
+
+        except tractor.MsgTypeError as _mte:
+            mte = _mte
+
+            # false -ve case
+            if add_hooks:
+                raise RuntimeError(
+                    f'EXPECTED to `.started()` value given spec ??\n\n'
+                    f'child_pld_spec -> {child_pld_spec}\n'
+                    f'value = {nsp}: {type(nsp)}\n'
+                )
+
+            # true -ve case
+            raise mte
+
+        # TODO: maybe we should add our own wrapper error so as to
+        # be interchange-lib agnostic?
+        # -[ ] the error type is wtv is raised from the hook so we
+        #   could also require a type-class of errors for
+        #   indicating whether the hook-failure can be handled by
+        #   a nasty-dialog-unprot sub-sys?
+        except TypeError as typerr:
+            # false -ve
+            if add_hooks:
+                raise RuntimeError('Should have been able to send `nsp`??')
+
+            # true -ve
+            print('Failed to send `nsp` due to no ext hooks set!')
+            raise typerr
+
+        # now try sending a set of valid and invalid plds to ensure
+        # the pld spec is respected.
+        sent: list[Any] = []
+        async with ctx.open_stream() as ipc:
+            print(
+                f'{uid}: streaming all pld types to rent..'
+            )
+
+            # for send_value, expect_send in iter_send_val_items:
+            for send_value in [
+                nsp,
+                str(nsp),
+                None,
+            ]:
+                send_type: Type = type(send_value)
+                print(
+                    f'{uid}: SENDING NEXT pld\n'
+                    f'send_type: {send_type}\n'
+                    f'send_value: {send_value}\n'
+                )
+                try:
+                    await ipc.send(send_value)
+                    sent.append(send_value)
+
+                except ValidationError as valerr:
+                    print(f'{uid} FAILED TO SEND {send_value}!')
+
+                    # false -ve
+                    if add_hooks:
+                        raise RuntimeError(
+                            f'EXPECTED to roundtrip value given spec:\n'
+                            f'rent_pld_spec -> {rent_pld_spec}\n'
+                            f'child_pld_spec -> {child_pld_spec}\n'
+                            f'value = {send_value}: {send_type}\n'
+                        )
+
+                    # true -ve
+                    raise valerr
+                    # continue
+
+            else:
+                print(
+                    f'{uid}: finished sending all values\n'
+                    'Should be exiting stream block!\n'
+                )
+
+        print(f'{uid}: exited streaming block!')
+
+
+
+@cm
+def maybe_apply_codec(codec: MsgCodec|None) -> MsgCodec|None:
+    if codec is None:
+        yield None
+        return
+
+    with apply_codec(codec) as codec:
+        yield codec
+
+
+@pytest.mark.parametrize(
+    'pld_spec',
+    [
+        Any,
+        NamespacePath,
+        NamespacePath|None,  # the "maybe" spec Bo
+    ],
+    ids=[
+        'any_type',
+        'only_nsp_ext',
+        'maybe_nsp_ext',
+    ]
+)
+@pytest.mark.parametrize(
+    'add_hooks',
+    [
+        True,
+        False,
+    ],
+    ids=[
+        'use_codec_hooks',
+        'no_codec_hooks',
+    ],
+)
+def test_ext_types_over_ipc(
+    debug_mode: bool,
+    pld_spec: Union[Type],
+    add_hooks: bool,
+):
+    '''
+    Ensure we can support extension types coverted using
+    `enc/dec_hook()`s passed to the `.msg.limit_plds()` API
+    and that sane errors happen when we try do the same without
+    the codec hooks.
+
+    '''
+    pld_types: set[Type] = _codec.unpack_spec_types(pld_spec)
+
+    async def main():
+
+        # sanity check the default pld-spec beforehand
+        chk_codec_applied(
+            expect_codec=_codec._def_tractor_codec,
+        )
+
+        # extension type we want to send as msg payload
+        nsp = NamespacePath.from_ref(ex_func)
+
+        # ^NOTE, 2 cases:
+        # - codec hooks noto added -> decode nsp as `str`
+        # - codec with hooks -> decode nsp as `NamespacePath`
+        nsp_codec: MsgCodec|None = None
+        if (
+            NamespacePath in pld_types
+            and
+            add_hooks
+        ):
+            nsp_codec = mk_codec(
+                enc_hook=enc_nsp,
+                ext_types=[NamespacePath],
+            )
+
+        async with tractor.open_nursery(
+            debug_mode=debug_mode,
+        ) as an:
+            p: tractor.Portal = await an.start_actor(
+                'sub',
+                enable_modules=[__name__],
+            )
+            with (
+                maybe_apply_codec(nsp_codec) as codec,
+            ):
+                chk_codec_applied(
+                    expect_codec=nsp_codec,
+                    enter_value=codec,
+                )
+                rent_pld_spec_type_strs: list[str] = _exts.enc_type_union(pld_spec)
+
+                # XXX should raise an mte (`MsgTypeError`)
+                # when `add_hooks == False` bc the input
+                # `expect_ipc_send` kwarg has a nsp which can't be
+                # serialized!
+                #
+                # TODO:can we ensure this happens from the
+                # `Return`-side (aka the sub) as well?
+                try:
+                    ctx: tractor.Context
+                    ipc: tractor.MsgStream
+                    async with (
+
+                        # XXX should raise an mte (`MsgTypeError`)
+                        # when `add_hooks == False`..
+                        p.open_context(
+                            send_back_values,
+                            # expect_debug=debug_mode,
+                            rent_pld_spec_type_strs=rent_pld_spec_type_strs,
+                            add_hooks=add_hooks,
+                            # expect_ipc_send=expect_ipc_send,
+                        ) as (ctx, first),
+
+                        ctx.open_stream() as ipc,
+                    ):
+                        with (
+                            limit_plds(
+                                pld_spec,
+                                dec_hook=dec_nsp if add_hooks else None,
+                                ext_types=[NamespacePath]  if add_hooks else None,
+                            ) as pld_dec,
+                        ):
+                            ctx_pld_dec: MsgDec = ctx._pld_rx._pld_dec
+                            assert pld_dec is ctx_pld_dec
+
+                            # if (
+                            #     not add_hooks
+                            #     and
+                            #     NamespacePath in 
+                            # ):
+                            #     pytest.fail('ctx should fail to open without custom enc_hook!?')
+
+                            await ipc.send(nsp)
+                            nsp_rt = await ipc.receive()
+
+                            assert nsp_rt == nsp
+                            assert nsp_rt.load_ref() is ex_func
+
+                # this test passes bc we can go no further!
+                except MsgTypeError as mte:
+                    # if not add_hooks:
+                    #     # teardown nursery
+                    #     await p.cancel_actor()
+                        # return
+
+                    raise mte
+
+            await p.cancel_actor()
+
+    if (
+        NamespacePath in pld_types
+        and
+        add_hooks
+    ):
+        trio.run(main)
+
+    else:
+        with pytest.raises(
+            expected_exception=tractor.RemoteActorError,
+        ) as excinfo:
+            trio.run(main)
+
+        exc = excinfo.value
+        # bc `.started(nsp: NamespacePath)` will raise
+        assert exc.boxed_type is TypeError
+
+
+# def chk_pld_type(
+#     payload_spec: Type[Struct]|Any,
+#     pld: Any,
+
+#     expect_roundtrip: bool|None = None,
+
+# ) -> bool:
+
+#     pld_val_type: Type = type(pld)
+
+#     # TODO: verify that the overridden subtypes
+#     # DO NOT have modified type-annots from original!
+#     # 'Start',  .pld: FuncSpec
+#     # 'StartAck',  .pld: IpcCtxSpec
+#     # 'Stop',  .pld: UNSEt
+#     # 'Error',  .pld: ErrorData
+
+#     codec: MsgCodec = mk_codec(
+#         # NOTE: this ONLY accepts `PayloadMsg.pld` fields of a specified
+#         # type union.
+#         ipc_pld_spec=payload_spec,
+#     )
+
+#     # make a one-off dec to compare with our `MsgCodec` instance
+#     # which does the below `mk_msg_spec()` call internally
+#     ipc_msg_spec: Union[Type[Struct]]
+#     msg_types: list[PayloadMsg[payload_spec]]
+#     (
+#         ipc_msg_spec,
+#         msg_types,
+#     ) = mk_msg_spec(
+#         payload_type_union=payload_spec,
+#     )
+#     _enc = msgpack.Encoder()
+#     _dec = msgpack.Decoder(
+#         type=ipc_msg_spec or Any,  # like `PayloadMsg[Any]`
+#     )
+
+#     assert (
+#         payload_spec
+#         ==
+#         codec.pld_spec
+#     )
+
+#     # assert codec.dec == dec
+#     #
+#     # ^-XXX-^ not sure why these aren't "equal" but when cast
+#     # to `str` they seem to match ?? .. kk
+
+#     assert (
+#         str(ipc_msg_spec)
+#         ==
+#         str(codec.msg_spec)
+#         ==
+#         str(_dec.type)
+#         ==
+#         str(codec.dec.type)
+#     )
+
+#     # verify the boxed-type for all variable payload-type msgs.
+#     if not msg_types:
+#         breakpoint()
+
+#     roundtrip: bool|None = None
+#     pld_spec_msg_names: list[str] = [
+#         td.__name__ for td in _payload_msgs
+#     ]
+#     for typedef in msg_types:
+
+#         skip_runtime_msg: bool = typedef.__name__ not in pld_spec_msg_names
+#         if skip_runtime_msg:
+#             continue
+
+#         pld_field = structs.fields(typedef)[1]
+#         assert pld_field.type is payload_spec # TODO-^ does this need to work to get all subtypes to adhere?
+
+#         kwargs: dict[str, Any] = {
+#             'cid': '666',
+#             'pld': pld,
+#         }
+#         enc_msg: PayloadMsg = typedef(**kwargs)
+
+#         _wire_bytes: bytes = _enc.encode(enc_msg)
+#         wire_bytes: bytes = codec.enc.encode(enc_msg)
+#         assert _wire_bytes == wire_bytes
+
+#         ve: ValidationError|None = None
+#         try:
+#             dec_msg = codec.dec.decode(wire_bytes)
+#             _dec_msg = _dec.decode(wire_bytes)
+
+#             # decoded msg and thus payload should be exactly same!
+#             assert (roundtrip := (
+#                 _dec_msg
+#                 ==
+#                 dec_msg
+#                 ==
+#                 enc_msg
+#             ))
+
+#             if (
+#                 expect_roundtrip is not None
+#                 and expect_roundtrip != roundtrip
+#             ):
+#                 breakpoint()
+
+#             assert (
+#                 pld
+#                 ==
+#                 dec_msg.pld
+#                 ==
+#                 enc_msg.pld
+#             )
+#             # assert (roundtrip := (_dec_msg == enc_msg))
+
+#         except ValidationError as _ve:
+#             ve = _ve
+#             roundtrip: bool = False
+#             if pld_val_type is payload_spec:
+#                 raise ValueError(
+#                    'Got `ValidationError` despite type-var match!?\n'
+#                     f'pld_val_type: {pld_val_type}\n'
+#                     f'payload_type: {payload_spec}\n'
+#                 ) from ve
+
+#             else:
+#                 # ow we good cuz the pld spec mismatched.
+#                 print(
+#                     'Got expected `ValidationError` since,\n'
+#                     f'{pld_val_type} is not {payload_spec}\n'
+#                 )
+#         else:
+#             if (
+#                 payload_spec is not Any
+#                 and
+#                 pld_val_type is not payload_spec
+#             ):
+#                 raise ValueError(
+#                    'DID NOT `ValidationError` despite expected type match!?\n'
+#                     f'pld_val_type: {pld_val_type}\n'
+#                     f'payload_type: {payload_spec}\n'
+#                 )
+
+#     # full code decode should always be attempted!
+#     if roundtrip is None:
+#         breakpoint()
+
+#     return roundtrip
+
+
+# ?TODO? maybe remove since covered in the newer `test_pldrx_limiting`
+# via end-2-end testing of all this?
+# -[ ] IOW do we really NEED this lowlevel unit testing?
+#
+# def test_limit_msgspec(
+#     debug_mode: bool,
+# ):
+#     '''
+#     Internals unit testing to verify that type-limiting an IPC ctx's
+#     msg spec with `Pldrx.limit_plds()` results in various
+#     encapsulated `msgspec` object settings and state.
+
+#     '''
+#     async def main():
+#         async with tractor.open_root_actor(
+#             debug_mode=debug_mode,
+#         ):
+#             # ensure we can round-trip a boxing `PayloadMsg`
+#             assert chk_pld_type(
+#                 payload_spec=Any,
+#                 pld=None,
+#                 expect_roundtrip=True,
+#             )
+
+#             # verify that a mis-typed payload value won't decode
+#             assert not chk_pld_type(
+#                 payload_spec=int,
+#                 pld='doggy',
+#             )
+
+#             # parametrize the boxed `.pld` type as a custom-struct
+#             # and ensure that parametrization propagates
+#             # to all payload-msg-spec-able subtypes!
+#             class CustomPayload(Struct):
+#                 name: str
+#                 value: Any
+
+#             assert not chk_pld_type(
+#                 payload_spec=CustomPayload,
+#                 pld='doggy',
+#             )
+
+#             assert chk_pld_type(
+#                 payload_spec=CustomPayload,
+#                 pld=CustomPayload(name='doggy', value='urmom')
+#             )
+
+#             # yah, we can `.pause_from_sync()` now!
+#             # breakpoint()
+
+#     trio.run(main)

tests/test_inter_peer_cancellation.py

@@ -170,7 +170,7 @@ def test_do_not_swallow_error_before_started_by_remote_contextcancelled(
         trio.run(main)
 
     rae = excinfo.value
-    assert rae.boxed_type == TypeError
+    assert rae.boxed_type is TypeError
 
 
 @tractor.context

tests/test_local.py

@@ -38,7 +38,7 @@ async def test_self_is_registered_localportal(reg_addr):
     "Verify waiting on the arbiter to register itself using a local portal."
     actor = tractor.current_actor()
     assert actor.is_arbiter
-    async with tractor.get_arbiter(*reg_addr) as portal:
+    async with tractor.get_registry(*reg_addr) as portal:
         assert isinstance(portal, tractor._portal.LocalPortal)
 
         with trio.fail_after(0.2):

tests/test_multi_program.py

@@ -10,7 +10,7 @@ import tractor
 from tractor._testing import (
     tractor_test,
 )
-from conftest import (
+from .conftest import (
     sig_prog,
     _INT_SIGNAL,
     _INT_RETURN_CODE,
@@ -32,7 +32,7 @@ def test_abort_on_sigint(daemon):
 @tractor_test
 async def test_cancel_remote_arbiter(daemon, reg_addr):
     assert not tractor.current_actor().is_arbiter
-    async with tractor.get_arbiter(*reg_addr) as portal:
+    async with tractor.get_registry(*reg_addr) as portal:
         await portal.cancel_actor()
 
     time.sleep(0.1)
@@ -41,7 +41,7 @@ async def test_cancel_remote_arbiter(daemon, reg_addr):
 
     # no arbiter socket should exist
     with pytest.raises(OSError):
-        async with tractor.get_arbiter(*reg_addr) as portal:
+        async with tractor.get_registry(*reg_addr) as portal:
             pass
 
 

tests/test_pldrx_limiting.py

@@ -285,14 +285,14 @@ def test_basic_payload_spec(
 
             if invalid_started:
                 msg_type_str: str = 'Started'
-                bad_value_str: str = '10'
+                bad_value: int = 10
             elif invalid_return:
                 msg_type_str: str = 'Return'
-                bad_value_str: str = "'yo'"
+                bad_value: str = 'yo'
             else:
                 # XXX but should never be used below then..
                 msg_type_str: str = ''
-                bad_value_str: str = ''
+                bad_value: str = ''
 
             maybe_mte: MsgTypeError|None = None
             should_raise: Exception|None = (
@@ -307,8 +307,10 @@ def test_basic_payload_spec(
                     raises=should_raise,
                     ensure_in_message=[
                         f"invalid `{msg_type_str}` msg payload",
-                        f"value: `{bad_value_str}` does not "
-                        f"match type-spec: `{msg_type_str}.pld: PldMsg|NoneType`",
+                        f'{bad_value}',
+                        f'has type {type(bad_value)!r}',
+                        'not match type-spec',
+                        f'`{msg_type_str}.pld: PldMsg|NoneType`',
                     ],
                     # only for debug
                     # post_mortem=True,

tests/test_root_infect_asyncio.py

@@ -0,0 +1,248 @@
+'''
+Special attention cases for using "infect `asyncio`" mode from a root
+actor; i.e. not using a std `trio.run()` bootstrap.
+
+'''
+import asyncio
+from functools import partial
+
+import pytest
+import trio
+import tractor
+from tractor import (
+    to_asyncio,
+)
+from tests.test_infected_asyncio import (
+    aio_echo_server,
+)
+
+
+@pytest.mark.parametrize(
+    'raise_error_mid_stream',
+    [
+        False,
+        Exception,
+        KeyboardInterrupt,
+    ],
+    ids='raise_error={}'.format,
+)
+def test_infected_root_actor(
+    raise_error_mid_stream: bool|Exception,
+
+    # conftest wide
+    loglevel: str,
+    debug_mode: bool,
+):
+    '''
+    Verify you can run the `tractor` runtime with `Actor.is_infected_aio() == True`
+    in the root actor.
+
+    '''
+    async def _trio_main():
+        with trio.fail_after(2 if not debug_mode else 999):
+            first: str
+            chan: to_asyncio.LinkedTaskChannel
+            async with (
+                tractor.open_root_actor(
+                    debug_mode=debug_mode,
+                    loglevel=loglevel,
+                ),
+                to_asyncio.open_channel_from(
+                    aio_echo_server,
+                ) as (first, chan),
+            ):
+                assert first == 'start'
+
+                for i in range(1000):
+                    await chan.send(i)
+                    out = await chan.receive()
+                    assert out == i
+                    print(f'asyncio echoing {i}')
+
+                    if (
+                        raise_error_mid_stream
+                        and
+                        i == 500
+                    ):
+                        raise raise_error_mid_stream
+
+                    if out is None:
+                        try:
+                            out = await chan.receive()
+                        except trio.EndOfChannel:
+                            break
+                        else:
+                            raise RuntimeError(
+                                'aio channel never stopped?'
+                            )
+
+    if raise_error_mid_stream:
+        with pytest.raises(raise_error_mid_stream):
+            tractor.to_asyncio.run_as_asyncio_guest(
+                trio_main=_trio_main,
+            )
+    else:
+        tractor.to_asyncio.run_as_asyncio_guest(
+            trio_main=_trio_main,
+        )
+
+
+
+async def sync_and_err(
+    # just signature placeholders for compat with
+    # ``to_asyncio.open_channel_from()``
+    to_trio: trio.MemorySendChannel,
+    from_trio: asyncio.Queue,
+    ev: asyncio.Event,
+
+):
+    if to_trio:
+        to_trio.send_nowait('start')
+
+    await ev.wait()
+    raise RuntimeError('asyncio-side')
+
+
+@pytest.mark.parametrize(
+    'aio_err_trigger',
+    [
+        'before_start_point',
+        'after_trio_task_starts',
+        'after_start_point',
+    ],
+    ids='aio_err_triggered={}'.format
+)
+def test_trio_prestarted_task_bubbles(
+    aio_err_trigger: str,
+
+    # conftest wide
+    loglevel: str,
+    debug_mode: bool,
+):
+    async def pre_started_err(
+        raise_err: bool = False,
+        pre_sleep: float|None = None,
+        aio_trigger: asyncio.Event|None = None,
+        task_status=trio.TASK_STATUS_IGNORED,
+    ):
+        '''
+        Maybe pre-started error then sleep.
+
+        '''
+        if pre_sleep is not None:
+            print(f'Sleeping from trio for {pre_sleep!r}s !')
+            await trio.sleep(pre_sleep)
+
+        # signal aio-task to raise JUST AFTER this task
+        # starts but has not yet `.started()`
+        if aio_trigger:
+            print('Signalling aio-task to raise from `trio`!!')
+            aio_trigger.set()
+
+        if raise_err:
+            print('Raising from trio!')
+            raise TypeError('trio-side')
+
+        task_status.started()
+        await trio.sleep_forever()
+
+    async def _trio_main():
+        # with trio.fail_after(2):
+        with trio.fail_after(999):
+            first: str
+            chan: to_asyncio.LinkedTaskChannel
+            aio_ev = asyncio.Event()
+
+            async with (
+                tractor.open_root_actor(
+                    debug_mode=False,
+                    loglevel=loglevel,
+                ),
+            ):
+                # TODO, tests for this with 3.13 egs?
+                # from tractor.devx import open_crash_handler
+                # with open_crash_handler():
+                async with (
+                    # where we'll start a sub-task that errors BEFORE
+                    # calling `.started()` such that the error should
+                    # bubble before the guest run terminates!
+                    trio.open_nursery() as tn,
+
+                    # THEN start an infect task which should error just
+                    # after the trio-side's task does.
+                    to_asyncio.open_channel_from(
+                        partial(
+                            sync_and_err,
+                            ev=aio_ev,
+                        )
+                    ) as (first, chan),
+                ):
+
+                    for i in range(5):
+                        pre_sleep: float|None = None
+                        last_iter: bool = (i == 4)
+
+                        # TODO, missing cases?
+                        # -[ ] error as well on
+                        #    'after_start_point' case as well for
+                        #    another case?
+                        raise_err: bool = False
+
+                        if last_iter:
+                            raise_err: bool = True
+
+                            # trigger aio task to error on next loop
+                            # tick/checkpoint
+                            if aio_err_trigger == 'before_start_point':
+                                aio_ev.set()
+
+                            pre_sleep: float = 0
+
+                        await tn.start(
+                            pre_started_err,
+                            raise_err,
+                            pre_sleep,
+                            (aio_ev if (
+                                    aio_err_trigger == 'after_trio_task_starts'
+                                    and
+                                    last_iter
+                                ) else None
+                            ),
+                        )
+
+                        if (
+                            aio_err_trigger == 'after_start_point'
+                            and
+                            last_iter
+                        ):
+                            aio_ev.set()
+
+    with pytest.raises(
+        expected_exception=ExceptionGroup,
+    ) as excinfo:
+        tractor.to_asyncio.run_as_asyncio_guest(
+            trio_main=_trio_main,
+        )
+
+    eg = excinfo.value
+    rte_eg, rest_eg = eg.split(RuntimeError)
+
+    # ensure the trio-task's error bubbled despite the aio-side
+    # having (maybe) errored first.
+    if aio_err_trigger in (
+        'after_trio_task_starts',
+        'after_start_point',
+    ):
+        assert len(errs := rest_eg.exceptions) == 1
+        typerr = errs[0]
+        assert (
+            type(typerr) is TypeError
+            and
+            'trio-side' in typerr.args
+        )
+
+    # when aio errors BEFORE (last) trio task is scheduled, we should
+    # never see anythinb but the aio-side.
+    else:
+        assert len(rtes := rte_eg.exceptions) == 1
+        assert 'asyncio-side' in rtes[0].args[0]

tests/test_shm.py

@@ -1,167 +0,0 @@
-"""
-Shared mem primitives and APIs.
-
-"""
-import uuid
-
-# import numpy
-import pytest
-import trio
-import tractor
-from tractor._shm import (
-    open_shm_list,
-    attach_shm_list,
-)
-
-
-@tractor.context
-async def child_attach_shml_alot(
-    ctx: tractor.Context,
-    shm_key: str,
-) -> None:
-
-    await ctx.started(shm_key)
-
-    # now try to attach a boatload of times in a loop..
-    for _ in range(1000):
-        shml = attach_shm_list(
-            key=shm_key,
-            readonly=False,
-        )
-        assert shml.shm.name == shm_key
-        await trio.sleep(0.001)
-
-
-def test_child_attaches_alot():
-    async def main():
-        async with tractor.open_nursery() as an:
-
-            # allocate writeable list in parent
-            key = f'shml_{uuid.uuid4()}'
-            shml = open_shm_list(
-                key=key,
-            )
-
-            portal = await an.start_actor(
-                'shm_attacher',
-                enable_modules=[__name__],
-            )
-
-            async with (
-                portal.open_context(
-                    child_attach_shml_alot,
-                    shm_key=shml.key,
-                ) as (ctx, start_val),
-            ):
-                assert start_val == key
-                await ctx.result()
-
-            await portal.cancel_actor()
-
-    trio.run(main)
-
-
-@tractor.context
-async def child_read_shm_list(
-    ctx: tractor.Context,
-    shm_key: str,
-    use_str: bool,
-    frame_size: int,
-) -> None:
-
-    # attach in child
-    shml = attach_shm_list(
-        key=shm_key,
-        # dtype=str if use_str else float,
-    )
-    await ctx.started(shml.key)
-
-    async with ctx.open_stream() as stream:
-        async for i in stream:
-            print(f'(child): reading shm list index: {i}')
-
-            if use_str:
-                expect = str(float(i))
-            else:
-                expect = float(i)
-
-            if frame_size == 1:
-                val = shml[i]
-                assert expect == val
-                print(f'(child): reading value: {val}')
-            else:
-                frame = shml[i - frame_size:i]
-                print(f'(child): reading frame: {frame}')
-
-
-@pytest.mark.parametrize(
-    'use_str',
-    [False, True],
-    ids=lambda i: f'use_str_values={i}',
-)
-@pytest.mark.parametrize(
-    'frame_size',
-    [1, 2**6, 2**10],
-    ids=lambda i: f'frame_size={i}',
-)
-def test_parent_writer_child_reader(
-    use_str: bool,
-    frame_size: int,
-):
-
-    async def main():
-        async with tractor.open_nursery(
-            # debug_mode=True,
-        ) as an:
-
-            portal = await an.start_actor(
-                'shm_reader',
-                enable_modules=[__name__],
-                debug_mode=True,
-            )
-
-            # allocate writeable list in parent
-            key = 'shm_list'
-            seq_size = int(2 * 2 ** 10)
-            shml = open_shm_list(
-                key=key,
-                size=seq_size,
-                dtype=str if use_str else float,
-                readonly=False,
-            )
-
-            async with (
-                portal.open_context(
-                    child_read_shm_list,
-                    shm_key=key,
-                    use_str=use_str,
-                    frame_size=frame_size,
-                ) as (ctx, sent),
-
-                ctx.open_stream() as stream,
-            ):
-
-                assert sent == key
-
-                for i in range(seq_size):
-
-                    val = float(i)
-                    if use_str:
-                        val = str(val)
-
-                    # print(f'(parent): writing {val}')
-                    shml[i] = val
-
-                    # only on frame fills do we
-                    # signal to the child that a frame's
-                    # worth is ready.
-                    if (i % frame_size) == 0:
-                        print(f'(parent): signalling frame full on {val}')
-                        await stream.send(i)
-                else:
-                    print(f'(parent): signalling final frame on {val}')
-                    await stream.send(i)
-
-            await portal.cancel_actor()
-
-    trio.run(main)

tests/test_task_broadcasting.py

@@ -2,7 +2,9 @@
 Broadcast channels for fan-out to local tasks.
 
 """
-from contextlib import asynccontextmanager
+from contextlib import (
+    asynccontextmanager as acm,
+)
 from functools import partial
 from itertools import cycle
 import time
@@ -15,6 +17,7 @@ import tractor
 from tractor.trionics import (
     broadcast_receiver,
     Lagged,
+    collapse_eg,
 )
 
 
@@ -62,7 +65,7 @@ async def ensure_sequence(
                 break
 
 
-@asynccontextmanager
+@acm
 async def open_sequence_streamer(
 
     sequence: list[int],
@@ -74,9 +77,9 @@ async def open_sequence_streamer(
     async with tractor.open_nursery(
         arbiter_addr=reg_addr,
         start_method=start_method,
-    ) as tn:
+    ) as an:
 
-        portal = await tn.start_actor(
+        portal = await an.start_actor(
             'sequence_echoer',
             enable_modules=[__name__],
         )
@@ -155,9 +158,12 @@ def test_consumer_and_parent_maybe_lag(
         ) as stream:
 
             try:
-                async with trio.open_nursery() as n:
+                async with (
+                    collapse_eg(),
+                    trio.open_nursery() as tn,
+                ):
 
-                    n.start_soon(
+                    tn.start_soon(
                         ensure_sequence,
                         stream,
                         sequence.copy(),
@@ -230,8 +236,8 @@ def test_faster_task_to_recv_is_cancelled_by_slower(
 
         ) as stream:
 
-            async with trio.open_nursery() as n:
-                n.start_soon(
+            async with trio.open_nursery() as tn:
+                tn.start_soon(
                     ensure_sequence,
                     stream,
                     sequence.copy(),
@@ -253,7 +259,7 @@ def test_faster_task_to_recv_is_cancelled_by_slower(
                         continue
 
                 print('cancelling faster subtask')
-                n.cancel_scope.cancel()
+                tn.cancel_scope.cancel()
 
             try:
                 value = await stream.receive()
@@ -271,7 +277,7 @@ def test_faster_task_to_recv_is_cancelled_by_slower(
                         # the faster subtask was cancelled
                         break
 
-                # await tractor.breakpoint()
+                # await tractor.pause()
                 # await stream.receive()
                 print(f'final value: {value}')
 
@@ -371,13 +377,13 @@ def test_ensure_slow_consumers_lag_out(
                                     f'on {lags}:{value}')
                                 return
 
-            async with trio.open_nursery() as nursery:
+            async with trio.open_nursery() as tn:
 
                 for i in range(1, num_laggers):
 
                     task_name = f'sub_{i}'
                     laggers[task_name] = 0
-                    nursery.start_soon(
+                    tn.start_soon(
                         partial(
                             sub_and_print,
                             delay=i*0.001,
@@ -497,6 +503,7 @@ def test_no_raise_on_lag():
                 # internals when the no raise flag is set.
                 loglevel='warning',
             ),
+            collapse_eg(),
             trio.open_nursery() as n,
         ):
             n.start_soon(slow)

tests/test_trioisms.py

@@ -3,6 +3,10 @@ Reminders for oddities in `trio` that we need to stay aware of and/or
 want to see changed.
 
 '''
+from contextlib import (
+    asynccontextmanager as acm,
+)
+
 import pytest
 import trio
 from trio import TaskStatus
@@ -60,7 +64,9 @@ def test_stashed_child_nursery(use_start_soon):
     async def main():
 
         async with (
-            trio.open_nursery() as pn,
+            trio.open_nursery(
+                strict_exception_groups=False,
+            ) as pn,
         ):
             cn = await pn.start(mk_child_nursery)
             assert cn
@@ -80,3 +86,118 @@ def test_stashed_child_nursery(use_start_soon):
 
     with pytest.raises(NameError):
         trio.run(main)
+
+
+@pytest.mark.parametrize(
+    ('unmask_from_canc', 'canc_from_finally'),
+    [
+        (True, False),
+        (True, True),
+        pytest.param(False, True,
+                     marks=pytest.mark.xfail(reason="never raises!")
+        ),
+    ],
+    # TODO, ask ronny how to impl this .. XD
+    # ids='unmask_from_canc={0}, canc_from_finally={1}',#.format,
+)
+def test_acm_embedded_nursery_propagates_enter_err(
+    canc_from_finally: bool,
+    unmask_from_canc: bool,
+    debug_mode: bool,
+):
+    '''
+    Demo how a masking `trio.Cancelled` could be handled by unmasking from the
+    `.__context__` field when a user (by accident) re-raises from a `finally:`.
+
+    '''
+    import tractor
+
+    @acm
+    async def maybe_raise_from_masking_exc(
+        tn: trio.Nursery,
+        unmask_from: BaseException|None = trio.Cancelled
+
+        # TODO, maybe offer a collection?
+        # unmask_from: set[BaseException] = {
+        #     trio.Cancelled,
+        # },
+    ):
+        if not unmask_from:
+            yield
+            return
+
+        try:
+            yield
+        except* unmask_from as be_eg:
+
+            # TODO, if we offer `unmask_from: set`
+            # for masker_exc_type in unmask_from:
+
+            matches, rest = be_eg.split(unmask_from)
+            if not matches:
+                raise
+
+            for exc_match in be_eg.exceptions:
+                if (
+                    (exc_ctx := exc_match.__context__)
+                    and
+                    type(exc_ctx) not in {
+                        # trio.Cancelled,  # always by default?
+                        unmask_from,
+                    }
+                ):
+                    exc_ctx.add_note(
+                        f'\n'
+                        f'WARNING: the above error was masked by a {unmask_from!r} !?!\n'
+                        f'Are you always cancelling? Say from a `finally:` ?\n\n'
+
+                        f'{tn!r}'
+                    )
+                    raise exc_ctx from exc_match
+
+
+    @acm
+    async def wraps_tn_that_always_cancels():
+        async with (
+            trio.open_nursery() as tn,
+            maybe_raise_from_masking_exc(
+                tn=tn,
+                unmask_from=(
+                    trio.Cancelled
+                    if unmask_from_canc
+                    else None
+                ),
+            )
+        ):
+            try:
+                yield tn
+            finally:
+                if canc_from_finally:
+                    tn.cancel_scope.cancel()
+                    await trio.lowlevel.checkpoint()
+
+    async def _main():
+        with tractor.devx.maybe_open_crash_handler(
+            pdb=debug_mode,
+        ) as bxerr:
+            assert not bxerr.value
+
+            async with (
+                wraps_tn_that_always_cancels() as tn,
+            ):
+                assert not tn.cancel_scope.cancel_called
+                assert 0
+
+        assert (
+            (err := bxerr.value)
+            and
+            type(err) is AssertionError
+        )
+
+    with pytest.raises(ExceptionGroup) as excinfo:
+        trio.run(_main)
+
+    eg: ExceptionGroup = excinfo.value
+    assert_eg, rest_eg = eg.split(AssertionError)
+
+    assert len(assert_eg.exceptions) == 1

tractor/__init__.py

@@ -18,6 +18,7 @@
 tractor: structured concurrent ``trio``-"actors".
 
 """
+
 from ._clustering import (
     open_actor_cluster as open_actor_cluster,
 )
@@ -30,7 +31,7 @@ from ._streaming import (
     stream as stream,
 )
 from ._discovery import (
-    get_arbiter as get_arbiter,
+    get_registry as get_registry,
     find_actor as find_actor,
     wait_for_actor as wait_for_actor,
     query_actor as query_actor,
@@ -43,12 +44,14 @@ from ._state import (
     current_actor as current_actor,
     is_root_process as is_root_process,
     current_ipc_ctx as current_ipc_ctx,
+    debug_mode as debug_mode
 )
 from ._exceptions import (
     ContextCancelled as ContextCancelled,
     ModuleNotExposed as ModuleNotExposed,
     MsgTypeError as MsgTypeError,
     RemoteActorError as RemoteActorError,
+    TransportClosed as TransportClosed,
 )
 from .devx import (
     breakpoint as breakpoint,
@@ -64,3 +67,4 @@ from ._root import (
 from ._ipc import Channel as Channel
 from ._portal import Portal as Portal
 from ._runtime import Actor as Actor
+# from . import hilevel as hilevel

tractor/_clustering.py

@@ -19,10 +19,13 @@ Actor cluster helpers.
 
 '''
 from __future__ import annotations
-
-from contextlib import asynccontextmanager as acm
+from contextlib import (
+    asynccontextmanager as acm,
+)
 from multiprocessing import cpu_count
-from typing import AsyncGenerator, Optional
+from typing import (
+    AsyncGenerator,
+)
 
 import trio
 import tractor

tractor/_context.py

@@ -38,6 +38,7 @@ from collections import deque
 from contextlib import (
     asynccontextmanager as acm,
 )
+from contextvars import Token
 from dataclasses import (
     dataclass,
     field,
@@ -45,6 +46,10 @@ from dataclasses import (
 from functools import partial
 import inspect
 from pprint import pformat
+import textwrap
+from types import (
+    UnionType,
+)
 from typing import (
     Any,
     AsyncGenerator,
@@ -77,6 +82,7 @@ from .msg import (
     MsgType,
     NamespacePath,
     PayloadT,
+    Return,
     Started,
     Stop,
     Yield,
@@ -121,10 +127,19 @@ class Unresolved:
 @dataclass
 class Context:
     '''
-    An inter-actor, SC transitive, `Task` communication context.
+    An inter-actor, SC transitive, `trio.Task` (pair)
+    communication context.
 
-    NB: This class should **never be instatiated directly**, it is allocated
-    by the runtime in 2 ways:
+    (We've also considered other names and ideas:
+     - "communicating tasks scope": cts
+     - "distributed task scope": dts
+     - "communicating tasks context": ctc
+
+     **Got a better idea for naming? Make an issue dawg!**
+    )
+
+    NB: This class should **never be instatiated directly**, it is
+    allocated by the runtime in 2 ways:
      - by entering `Portal.open_context()` which is the primary
        public API for any "parent" task or,
      - by the RPC machinery's `._rpc._invoke()` as a `ctx` arg
@@ -210,6 +225,16 @@ class Context:
     # more the the `Context` is needed?
     _portal: Portal | None = None
 
+    @property
+    def portal(self) -> Portal|None:
+        '''
+        Return any wrapping memory-`Portal` if this is
+        a 'parent'-side task which called `Portal.open_context()`,
+        otherwise `None`.
+
+        '''
+        return self._portal
+
     # NOTE: each side of the context has its own cancel scope
     # which is exactly the primitive that allows for
     # cross-actor-task-supervision and thus SC.
@@ -221,11 +246,13 @@ class Context:
     # a drain loop?
     # _res_scope: trio.CancelScope|None = None
 
+    _outcome_msg: Return|Error|ContextCancelled = Unresolved
+
     # on a clean exit there should be a final value
     # delivered from the far end "callee" task, so
     # this value is only set on one side.
     # _result: Any | int = None
-    _result: Any|Unresolved = Unresolved
+    _result: PayloadT|Unresolved = Unresolved
 
     # if the local "caller"  task errors this value is always set
     # to the error that was captured in the
@@ -299,6 +326,8 @@ class Context:
     # boxed exception. NOW, it's used for spawning overrun queuing
     # tasks when `.allow_overruns ==  True` !!!
     _scope_nursery: trio.Nursery|None = None
+    # ^-TODO-^ change name?
+    # -> `._scope_tn` "scope task nursery"
 
     # streaming overrun state tracking
     _in_overrun: bool = False
@@ -313,6 +342,7 @@ class Context:
         extra_fields: dict[str, Any]|None = None,
         # ^-TODO-^ some built-in extra state fields
         # we'll want in some devx specific cases?
+        indent: str|None = None,
 
     ) -> str:
         ds: str = '='
@@ -332,7 +362,6 @@ class Context:
             show_error_fields=True
         )
         fmtstr: str = (
-            f'<Context(\n'
             # f'\n'
             # f'   ---\n'
             f' |_ipc: {self.dst_maddr}\n'
@@ -379,11 +408,20 @@ class Context:
                     f'   {key}{ds}{val!r}\n'
                 )
 
+        if indent:
+            fmtstr = textwrap.indent(
+                fmtstr,
+                prefix=indent,
+            )
+
         return (
+            '<Context(\n'
+            +
             fmtstr
             +
-            ')>\n'
+            f'{indent})>\n'
         )
+
     # NOTE: making this return a value that can be passed to
     # `eval()` is entirely **optional** dawggg B)
     # https://docs.python.org/3/library/functions.html#repr
@@ -408,10 +446,23 @@ class Context:
         '''
         return self._cancel_called
 
+    @cancel_called.setter
+    def cancel_called(self, val: bool) -> None:
+        '''
+        Set the self-cancelled request `bool` value.
+
+        '''
+        # to debug who frickin sets it..
+        # if val:
+        #     from .devx import pause_from_sync
+        #     pause_from_sync()
+
+        self._cancel_called = val
+
     @property
     def canceller(self) -> tuple[str, str]|None:
         '''
-        ``Actor.uid: tuple[str, str]`` of the (remote)
+        `Actor.uid: tuple[str, str]` of the (remote)
         actor-process who's task was cancelled thus causing this
         (side of the) context to also be cancelled.
 
@@ -515,7 +566,7 @@ class Context:
 
             # the local scope was never cancelled
             # and instead likely we received a remote side
-            # # cancellation that was raised inside `.result()`
+            # # cancellation that was raised inside `.wait_for_result()`
             # or (
             #     (se := self._local_error)
             #     and se is re
@@ -585,6 +636,10 @@ class Context:
         self,
         error: BaseException,
 
+        # TODO: manual toggle for cases where we wouldn't normally
+        # mark ourselves cancelled but want to?
+        # set_cancel_called: bool = False,
+
     ) -> None:
         '''
         (Maybe) cancel this local scope due to a received remote
@@ -603,7 +658,7 @@ class Context:
         - `Portal.open_context()`
         - `Portal.result()`
         - `Context.open_stream()`
-        - `Context.result()`
+        - `Context.wait_for_result()`
 
         when called/closed by actor local task(s).
 
@@ -729,7 +784,7 @@ class Context:
 
         # Cancel the local `._scope`, catch that
         # `._scope.cancelled_caught` and re-raise any remote error
-        # once exiting (or manually calling `.result()`) the
+        # once exiting (or manually calling `.wait_for_result()`) the
         # `.open_context()`  block.
         cs: trio.CancelScope = self._scope
         if (
@@ -764,8 +819,9 @@ class Context:
                 # `trio.Cancelled` subtype here ;)
                 # https://github.com/goodboy/tractor/issues/368
                 message: str = 'Cancelling `Context._scope` !\n\n'
+                # from .devx import pause_from_sync
+                # pause_from_sync()
                 self._scope.cancel()
-
         else:
             message: str = 'NOT cancelling `Context._scope` !\n\n'
             # from .devx import mk_pdb
@@ -845,15 +901,15 @@ class Context:
 
     @property
     def repr_api(self) -> str:
+        return 'Portal.open_context()'
+
+        # TODO: use `.dev._frame_stack` scanning to find caller!
         # ci: CallerInfo|None = self._caller_info
         # if ci:
         #     return (
         #         f'{ci.api_nsp}()\n'
         #     )
 
-        # TODO: use `.dev._frame_stack` scanning to find caller!
-        return 'Portal.open_context()'
-
     async def cancel(
         self,
         timeout: float = 0.616,
@@ -889,14 +945,16 @@ class Context:
 
         '''
         side: str = self.side
-        self._cancel_called: bool = True
+        # XXX for debug via the `@.setter`
+        self.cancel_called = True
 
         header: str = (
-            f'Cancelling ctx with peer from {side.upper()} side\n\n'
+            f'Cancelling ctx from {side.upper()}-side\n'
         )
         reminfo: str = (
             # ' =>\n'
-            f'Context.cancel() => {self.chan.uid}\n'
+            # f'Context.cancel() => {self.chan.uid}\n'
+            f'c)=> {self.chan.uid}\n'
             # f'{self.chan.uid}\n'
             f'  |_ @{self.dst_maddr}\n'
             f'    >> {self.repr_rpc}\n'
@@ -912,7 +970,7 @@ class Context:
         # `._scope.cancel()` since we expect the eventual
         # `ContextCancelled` from the other side to trigger this
         # when the runtime finally receives it during teardown
-        # (normally in `.result()` called from
+        # (normally in `.wait_for_result()` called from
         # `Portal.open_context().__aexit__()`)
         if side == 'parent':
             if not self._portal:
@@ -951,7 +1009,8 @@ class Context:
                     )
                 else:
                     log.cancel(
-                        'Timed out on cancel request of remote task?\n'
+                        f'Timed out on cancel request of remote task?\n'
+                        f'\n'
                         f'{reminfo}'
                     )
 
@@ -1025,10 +1084,10 @@ class Context:
 
         '''
         __tracebackhide__: bool = hide_tb
-        our_uid: tuple = self.chan.uid
+        peer_uid: tuple = self.chan.uid
 
         # XXX NOTE XXX: `ContextCancelled`/`StreamOverrun` absorption
-        # for "graceful cancellation" case:
+        # for "graceful cancellation" case(s):
         #
         # Whenever a "side" of a context (a `Task` running in
         # an actor) **is** the side which requested ctx
@@ -1045,9 +1104,11 @@ class Context:
         # set to the `Actor.uid` of THIS task (i.e. the
         # cancellation requesting task's actor is the actor
         # checking whether it should absorb the ctxc).
+        self_ctxc: bool = self._is_self_cancelled(remote_error)
         if (
+            self_ctxc
+            and
             not raise_ctxc_from_self_call
-            and self._is_self_cancelled(remote_error)
 
             # TODO: ?potentially it is useful to emit certain
             # warning/cancel logs for the cases where the
@@ -1077,8 +1138,8 @@ class Context:
             and isinstance(remote_error, RemoteActorError)
             and remote_error.boxed_type is StreamOverrun
 
-            # and tuple(remote_error.msgdata['sender']) == our_uid
-            and tuple(remote_error.sender) == our_uid
+            # and tuple(remote_error.msgdata['sender']) == peer_uid
+            and tuple(remote_error.sender) == peer_uid
         ):
             # NOTE: we set the local scope error to any "self
             # cancellation" error-response thus "absorbing"
@@ -1140,10 +1201,12 @@ class Context:
         of the remote cancellation.
 
         '''
-        __tracebackhide__ = hide_tb
-        assert self._portal, (
-            "Context.result() can not be called from callee side!"
-        )
+        __tracebackhide__: bool = hide_tb
+        if not self._portal:
+            raise RuntimeError(
+                'Invalid usage of `Context.wait_for_result()`!\n'
+                'Not valid on child-side IPC ctx!\n'
+            )
         if self._final_result_is_set():
             return self._result
 
@@ -1164,6 +1227,8 @@ class Context:
             # since every message should be delivered via the normal
             # `._deliver_msg()` route which will appropriately set
             # any `.maybe_error`.
+            outcome_msg: Return|Error|ContextCancelled
+            drained_msgs: list[MsgType]
             (
                 outcome_msg,
                 drained_msgs,
@@ -1171,11 +1236,19 @@ class Context:
                 ctx=self,
                 hide_tb=hide_tb,
             )
-
             drained_status: str = (
                 'Ctx drained to final outcome msg\n\n'
                 f'{outcome_msg}\n'
             )
+
+            # ?XXX, should already be set in `._deliver_msg()` right?
+            if self._outcome_msg is not Unresolved:
+                # from .devx import _debug
+                # await _debug.pause()
+                assert self._outcome_msg is outcome_msg
+            else:
+                self._outcome_msg = outcome_msg
+
             if drained_msgs:
                 drained_status += (
                     '\n'
@@ -1197,10 +1270,11 @@ class Context:
                 # raising something we know might happen
                 # during cancellation ;)
                 (not self._cancel_called)
-            )
+            ),
+            hide_tb=hide_tb,
         )
         # TODO: eventually make `.outcome: Outcome` and thus return
-        # `self.outcome.unwrap()` here!
+        # `self.outcome.unwrap()` here?
         return self.outcome
 
     # TODO: switch this with above!
@@ -1223,6 +1297,12 @@ class Context:
 
     @property
     def maybe_error(self) -> BaseException|None:
+        '''
+        Return the (remote) error as outcome or `None`.
+
+        Remote errors take precedence over local ones.
+
+        '''
         le: BaseException|None = self._local_error
         re: RemoteActorError|ContextCancelled|None = self._remote_error
 
@@ -1284,17 +1364,24 @@ class Context:
         Any|
         RemoteActorError|
         ContextCancelled
+        # TODO: make this a `outcome.Outcome`!
     ):
         '''
-        The final "outcome" from an IPC context which can either be
-        some Value returned from the target `@context`-decorated
-        remote task-as-func, or an `Error` wrapping an exception
-        raised from an RPC task fault or cancellation.
+        Return the "final outcome" (state) of the far end peer task
+        non-blocking. If the remote task has not completed then this
+        field always resolves to the module defined `Unresolved`
+        handle.
 
-        Note that if the remote task has not terminated then this
-        field always resolves to the module defined `Unresolved` handle.
+        ------ - ------
+        TODO->( this is doc-driven-dev content not yet actual ;P )
 
-        TODO: implement this using `outcome.Outcome` types?
+        The final "outcome" from an IPC context which can be any of:
+        - some `outcome.Value` which boxes the returned output from the peer task's 
+          `@context`-decorated remote task-as-func, or
+        - an `outcome.Error` wrapping an exception raised that same RPC task
+          after a fault or cancellation, or
+        - an unresolved `outcome.Outcome` when the peer task is still
+          executing and has not yet completed.
 
         '''
         return (
@@ -1492,12 +1579,12 @@ class Context:
                     strict_pld_parity=strict_pld_parity,
                     hide_tb=hide_tb,
                 )
-            except BaseException as err:
+            except BaseException as _bexc:
+                err = _bexc
                 if not isinstance(err, MsgTypeError):
                     __tracebackhide__: bool = False
 
-                raise
-
+                raise err
 
         # TODO: maybe a flag to by-pass encode op if already done
         # here in caller?
@@ -1583,7 +1670,7 @@ class Context:
 
         - NEVER `return` early before delivering the msg!
           bc if the error is a ctxc and there is a task waiting on
-          `.result()` we need the msg to be
+          `.wait_for_result()` we need the msg to be
           `send_chan.send_nowait()`-ed over the `._rx_chan` so
           that the error is relayed to that waiter task and thus
           raised in user code!
@@ -1635,15 +1722,28 @@ class Context:
         # TODO: expose as mod func instead!
         structfmt = pretty_struct.Struct.pformat
         if self._in_overrun:
-            log.warning(
-                f'Queueing OVERRUN msg on caller task:\n\n'
-
+            report: str = (
                 f'{flow_body}'
-
                 f'{structfmt(msg)}\n'
             )
+            over_q: deque = self._overflow_q
             self._overflow_q.append(msg)
 
+            if len(over_q) == over_q.maxlen:
+                report = (
+                    'FAILED to queue OVERRUN msg, OVERAN the OVERRUN QUEUE !!\n\n'
+                    + report
+                )
+                # log.error(report)
+                log.debug(report)
+
+            else:
+                report = (
+                    'Queueing OVERRUN msg on caller task:\n\n'
+                    + report
+                )
+                log.debug(report)
+
             # XXX NOTE XXX
             # overrun is the ONLY case where returning early is fine!
             return False
@@ -1656,7 +1756,6 @@ class Context:
 
                 f'{structfmt(msg)}\n'
             )
-
             # NOTE: if an error is deteced we should always still
             # send it through the feeder-mem-chan and expect
             # it to be raised by any context (stream) consumer
@@ -1668,6 +1767,21 @@ class Context:
             # normally the task that should get cancelled/error
             # from some remote fault!
             send_chan.send_nowait(msg)
+            match msg:
+                case Stop():
+                    if (stream := self._stream):
+                        stream._stop_msg = msg
+
+                case Return():
+                    if not self._outcome_msg:
+                        log.warning(
+                            f'Setting final outcome msg AFTER '
+                            f'`._rx_chan.send()`??\n'
+                            f'\n'
+                            f'{msg}'
+                        )
+                        self._outcome_msg = msg
+
             return True
 
         except trio.BrokenResourceError:
@@ -1828,7 +1942,7 @@ async def open_context_from_portal(
     When the "callee" (side that is "called"/started by a call
     to *this* method) returns, the caller side (this) unblocks
     and any final value delivered from the other end can be
-    retrieved using the `Contex.result()` api.
+    retrieved using the `Contex.wait_for_result()` api.
 
     The yielded ``Context`` instance further allows for opening
     bidirectional streams, explicit cancellation and
@@ -1893,7 +2007,7 @@ async def open_context_from_portal(
     )
     assert ctx._remote_func_type == 'context'
     assert ctx._caller_info
-    _ctxvar_Context.set(ctx)
+    prior_ctx_tok: Token = _ctxvar_Context.set(ctx)
 
     # placeholder for any exception raised in the runtime
     # or by user tasks which cause this context's closure.
@@ -1901,7 +2015,10 @@ async def open_context_from_portal(
     ctxc_from_callee: ContextCancelled|None = None
     try:
         async with (
-            trio.open_nursery() as tn,
+            trio.open_nursery(
+                strict_exception_groups=False,
+            ) as tn,
+
             msgops.maybe_limit_plds(
                 ctx=ctx,
                 spec=ctx_meta.get('pld_spec'),
@@ -1921,7 +2038,7 @@ async def open_context_from_portal(
             # the dialog, the `Error` msg should be raised from the `msg`
             # handling block below.
             try:
-                started_msg, first = await ctx._pld_rx.recv_msg_w_pld(
+                started_msg, first = await ctx._pld_rx.recv_msg(
                     ipc=ctx,
                     expect_msg=Started,
                     passthrough_non_pld_msgs=False,
@@ -1965,14 +2082,14 @@ async def open_context_from_portal(
             yield ctx, first
 
             # ??TODO??: do we still want to consider this or is
-            # the `else:` block handling via a `.result()`
+            # the `else:` block handling via a `.wait_for_result()`
             # call below enough??
             #
-            # -[ ] pretty sure `.result()` internals do the
+            # -[ ] pretty sure `.wait_for_result()` internals do the
             # same as our ctxc handler below so it ended up
             # being same (repeated?) behaviour, but ideally we
             # wouldn't have that duplication either by somehow
-            # factoring the `.result()` handler impl in a way
+            # factoring the `.wait_for_result()` handler impl in a way
             # that we can re-use it around the `yield` ^ here
             # or vice versa?
             #
@@ -2110,7 +2227,7 @@ async def open_context_from_portal(
         #    AND a group-exc is only raised if there was > 1
         #    tasks started *here* in the "caller" / opener
         #    block. If any one of those tasks calls
-        #    `.result()` or `MsgStream.receive()`
+        #    `.wait_for_result()` or `MsgStream.receive()`
         #    `._maybe_raise_remote_err()` will be transitively
         #    called and the remote error raised causing all
         #    tasks to be cancelled.
@@ -2131,9 +2248,16 @@ async def open_context_from_portal(
         # handled in the block above ^^^ !!
         # await _debug.pause()
         # log.cancel(
-        log.exception(
-            f'{ctx.side}-side of `Context` terminated with '
-            f'.outcome => {ctx.repr_outcome()}\n'
+        match scope_err:
+            case trio.Cancelled:
+                logmeth = log.cancel
+
+            # XXX explicitly report on any non-graceful-taskc cases
+            case _:
+                logmeth = log.exception
+
+        logmeth(
+            f'ctx {ctx.side!r}-side exited with {ctx.repr_outcome()}\n'
         )
 
         if debug_mode():
@@ -2141,12 +2265,12 @@ async def open_context_from_portal(
             #     pass
             # TODO: factor ^ into below for non-root cases?
             #
-            from .devx import maybe_wait_for_debugger
+            from .devx._debug import maybe_wait_for_debugger
             was_acquired: bool = await maybe_wait_for_debugger(
-                header_msg=(
-                    'Delaying `ctx.cancel()` until debug lock '
-                    'acquired..\n'
-                ),
+                # header_msg=(
+                #     'Delaying `ctx.cancel()` until debug lock '
+                #     'acquired..\n'
+                # ),
             )
             if was_acquired:
                 log.pdb(
@@ -2180,7 +2304,7 @@ async def open_context_from_portal(
                 f'|_{ctx._task}\n'
             )
             # XXX NOTE XXX: the below call to
-            # `Context.result()` will ALWAYS raise
+            # `Context.wait_for_result()` will ALWAYS raise
             # a `ContextCancelled` (via an embedded call to
             # `Context._maybe_raise_remote_err()`) IFF
             # a `Context._remote_error` was set by the runtime
@@ -2190,10 +2314,10 @@ async def open_context_from_portal(
             # ALWAYS SET any time "callee" side fails and causes "caller
             # side" cancellation via a `ContextCancelled` here.
             try:
-                result_or_err: Exception|Any = await ctx.result()
+                result_or_err: Exception|Any = await ctx.wait_for_result()
             except BaseException as berr:
                 # on normal teardown, if we get some error
-                # raised in `Context.result()` we still want to
+                # raised in `Context.wait_for_result()` we still want to
                 # save that error on the ctx's state to
                 # determine things like `.cancelled_caught` for
                 # cases where there was remote cancellation but
@@ -2240,7 +2364,7 @@ async def open_context_from_portal(
         # where the root is waiting on the lock to clear but the
         # child has already cleared it and clobbered IPC.
         if debug_mode():
-            from .devx import maybe_wait_for_debugger
+            from .devx._debug import maybe_wait_for_debugger
             await maybe_wait_for_debugger()
 
         # though it should be impossible for any tasks
@@ -2279,7 +2403,8 @@ async def open_context_from_portal(
             # displaying `ContextCancelled` traces where the
             # cause of crash/exit IS due to something in
             # user/app code on either end of the context.
-            and not rxchan._closed
+            and
+            not rxchan._closed
         ):
             # XXX NOTE XXX: and again as per above, we mask any
             # `trio.Cancelled` raised here so as to NOT mask
@@ -2311,8 +2436,9 @@ async def open_context_from_portal(
                 and ctx.cancel_acked
             ):
                 log.cancel(
-                    f'Context cancelled by {ctx.side!r}-side task\n'
-                    f'|_{ctx._task}\n\n'
+                    f'Context cancelled by local {ctx.side!r}-side task\n'
+                    f'c)>\n'
+                    f' |_{ctx._task}\n\n'
                     f'{repr(scope_err)}\n'
                 )
 
@@ -2328,13 +2454,16 @@ async def open_context_from_portal(
                     # type_only=True,
                 )
                 log.cancel(
-                    f'Context terminated due to local {ctx.side!r}-side error:\n\n'
-                    f'{ctx.chan.uid} => {outcome_str}\n'
+                    f'Context terminated due to {ctx.side!r}-side\n\n'
+                    # TODO: do an x)> on err and c)> only for ctxc?
+                    f'c)> {outcome_str}\n'
+                    f' |_{ctx.repr_rpc}\n'
                 )
 
         # FINALLY, remove the context from runtime tracking and
         # exit!
         log.runtime(
+        # log.cancel(
             f'De-allocating IPC ctx opened with {ctx.side!r} peer \n'
             f'uid: {uid}\n'
             f'cid: {ctx.cid}\n'
@@ -2344,6 +2473,9 @@ async def open_context_from_portal(
             None,
         )
 
+        # XXX revert to prior IPC-task-ctx scope
+        _ctxvar_Context.reset(prior_ctx_tok)
+
 
 def mk_context(
     chan: Channel,
@@ -2387,7 +2519,6 @@ def mk_context(
         _caller_info=caller_info,
         **kwargs,
     )
-    pld_rx._ctx = ctx
     ctx._result = Unresolved
     return ctx
 
@@ -2450,7 +2581,14 @@ def context(
     name: str
     param: Type
     for name, param in annots.items():
-        if param is Context:
+        if (
+            param is Context
+            or (
+                isinstance(param, UnionType)
+                and
+                Context in param.__args__
+            )
+        ):
             ctx_var_name: str = name
             break
     else:

tractor/_discovery.py

@@ -26,8 +26,8 @@ from typing import (
     TYPE_CHECKING,
 )
 from contextlib import asynccontextmanager as acm
-import warnings
 
+from tractor.log import get_logger
 from .trionics import gather_contexts
 from ._ipc import _connect_chan, Channel
 from ._portal import (
@@ -40,11 +40,13 @@ from ._state import (
     _runtime_vars,
 )
 
-
 if TYPE_CHECKING:
     from ._runtime import Actor
 
 
+log = get_logger(__name__)
+
+
 @acm
 async def get_registry(
     host: str,
@@ -56,14 +58,12 @@ async def get_registry(
 ]:
     '''
     Return a portal instance connected to a local or remote
-    arbiter.
+    registry-service actor; if a connection already exists re-use it
+    (presumably to call a `.register_actor()` registry runtime RPC
+    ep).
 
     '''
-    actor = current_actor()
-
-    if not actor:
-        raise RuntimeError("No actor instance has been defined yet?")
-
+    actor: Actor = current_actor()
     if actor.is_registrar:
         # we're already the arbiter
         # (likely a re-entrant call from the arbiter actor)
@@ -72,6 +72,8 @@ async def get_registry(
             Channel((host, port))
         )
     else:
+        # TODO: try to look pre-existing connection from
+        # `Actor._peers` and use it instead?
         async with (
             _connect_chan(host, port) as chan,
             open_portal(chan) as regstr_ptl,
@@ -80,19 +82,6 @@ async def get_registry(
 
 
 
-# TODO: deprecate and this remove _arbiter form!
-@acm
-async def get_arbiter(*args, **kwargs):
-    warnings.warn(
-        '`tractor.get_arbiter()` is now deprecated!\n'
-        'Use `.get_registry()` instead!',
-        DeprecationWarning,
-        stacklevel=2,
-    )
-    async with get_registry(*args, **kwargs) as to_yield:
-        yield to_yield
-
-
 @acm
 async def get_root(
     **kwargs,
@@ -110,22 +99,53 @@ async def get_root(
         yield portal
 
 
+def get_peer_by_name(
+    name: str,
+    # uuid: str|None = None,
+
+) -> list[Channel]|None:  # at least 1
+    '''
+    Scan for an existing connection (set) to a named actor
+    and return any channels from `Actor._peers`.
+
+    This is an optimization method over querying the registrar for
+    the same info.
+
+    '''
+    actor: Actor = current_actor()
+    to_scan: dict[tuple, list[Channel]] = actor._peers.copy()
+    pchan: Channel|None = actor._parent_chan
+    if pchan:
+        to_scan[pchan.uid].append(pchan)
+
+    for aid, chans in to_scan.items():
+        _, peer_name = aid
+        if name == peer_name:
+            if not chans:
+                log.warning(
+                    'No IPC chans for matching peer {peer_name}\n'
+                )
+                continue
+            return chans
+
+    return None
+
+
 @acm
 async def query_actor(
     name: str,
-    arbiter_sockaddr: tuple[str, int] | None = None,
-    regaddr: tuple[str, int] | None = None,
+    regaddr: tuple[str, int]|None = None,
 
 ) -> AsyncGenerator[
-    tuple[str, int] | None,
+    tuple[str, int]|None,
     None,
 ]:
     '''
-    Make a transport address lookup for an actor name to a specific
-    registrar.
+    Lookup a transport address (by actor name) via querying a registrar
+    listening @ `regaddr`.
 
-    Returns the (socket) address or ``None`` if no entry under that
-    name exists for the given registrar listening @ `regaddr`.
+    Returns the transport protocol (socket) address or `None` if no
+    entry under that name exists.
 
     '''
     actor: Actor = current_actor()
@@ -137,14 +157,10 @@ async def query_actor(
             'The current actor IS the registry!?'
         )
 
-    if arbiter_sockaddr is not None:
-        warnings.warn(
-            '`tractor.query_actor(regaddr=<blah>)` is deprecated.\n'
-            'Use `registry_addrs: list[tuple]` instead!',
-            DeprecationWarning,
-            stacklevel=2,
-        )
-        regaddr: list[tuple[str, int]] = arbiter_sockaddr
+    maybe_peers: list[Channel]|None = get_peer_by_name(name)
+    if maybe_peers:
+        yield maybe_peers[0].raddr
+        return
 
     reg_portal: Portal
     regaddr: tuple[str, int] = regaddr or actor.reg_addrs[0]
@@ -159,10 +175,28 @@ async def query_actor(
         yield sockaddr
 
 
+@acm
+async def maybe_open_portal(
+    addr: tuple[str, int],
+    name: str,
+):
+    async with query_actor(
+        name=name,
+        regaddr=addr,
+    ) as sockaddr:
+        pass
+
+    if sockaddr:
+        async with _connect_chan(*sockaddr) as chan:
+            async with open_portal(chan) as portal:
+                yield portal
+    else:
+        yield None
+
+
 @acm
 async def find_actor(
     name: str,
-    arbiter_sockaddr: tuple[str, int]|None = None,
     registry_addrs: list[tuple[str, int]]|None = None,
 
     only_first: bool = True,
@@ -179,29 +213,12 @@ async def find_actor(
     known to the arbiter.
 
     '''
-    if arbiter_sockaddr is not None:
-        warnings.warn(
-            '`tractor.find_actor(arbiter_sockaddr=<blah>)` is deprecated.\n'
-            'Use `registry_addrs: list[tuple]` instead!',
-            DeprecationWarning,
-            stacklevel=2,
-        )
-        registry_addrs: list[tuple[str, int]] = [arbiter_sockaddr]
-
-    @acm
-    async def maybe_open_portal_from_reg_addr(
-        addr: tuple[str, int],
-    ):
-        async with query_actor(
-            name=name,
-            regaddr=addr,
-        ) as sockaddr:
-            if sockaddr:
-                async with _connect_chan(*sockaddr) as chan:
-                    async with open_portal(chan) as portal:
-                        yield portal
-            else:
-                yield None
+    # optimization path, use any pre-existing peer channel
+    maybe_peers: list[Channel]|None = get_peer_by_name(name)
+    if maybe_peers and only_first:
+        async with open_portal(maybe_peers[0]) as peer_portal:
+            yield peer_portal
+            return
 
     if not registry_addrs:
         # XXX NOTE: make sure to dynamically read the value on
@@ -217,10 +234,13 @@ async def find_actor(
     maybe_portals: list[
         AsyncContextManager[tuple[str, int]]
     ] = list(
-        maybe_open_portal_from_reg_addr(addr)
+        maybe_open_portal(
+            addr=addr,
+            name=name,
+        )
         for addr in registry_addrs
     )
-
+    portals: list[Portal]
     async with gather_contexts(
         mngrs=maybe_portals,
     ) as portals:
@@ -254,31 +274,31 @@ async def find_actor(
 @acm
 async def wait_for_actor(
     name: str,
-    arbiter_sockaddr: tuple[str, int] | None = None,
     registry_addr: tuple[str, int] | None = None,
 
 ) -> AsyncGenerator[Portal, None]:
     '''
-    Wait on an actor to register with the arbiter.
-
-    A portal to the first registered actor is returned.
+    Wait on at least one peer actor to register `name` with the
+    registrar, yield a `Portal to the first registree.
 
     '''
     actor: Actor = current_actor()
 
-    if arbiter_sockaddr is not None:
-        warnings.warn(
-            '`tractor.wait_for_actor(arbiter_sockaddr=<foo>)` is deprecated.\n'
-            'Use `registry_addr: tuple` instead!',
-            DeprecationWarning,
-            stacklevel=2,
-        )
-        registry_addr: tuple[str, int] = arbiter_sockaddr
+    # optimization path, use any pre-existing peer channel
+    maybe_peers: list[Channel]|None = get_peer_by_name(name)
+    if maybe_peers:
+        async with open_portal(maybe_peers[0]) as peer_portal:
+            yield peer_portal
+            return
 
+    regaddr: tuple[str, int] = (
+        registry_addr
+        or
+        actor.reg_addrs[0]
+    )
     # TODO: use `.trionics.gather_contexts()` like
     # above in `find_actor()` as well?
     reg_portal: Portal
-    regaddr: tuple[str, int] = registry_addr or actor.reg_addrs[0]
     async with get_registry(*regaddr) as reg_portal:
         sockaddrs = await reg_portal.run_from_ns(
             'self',

tractor/_entry.py

@@ -20,7 +20,9 @@ Sub-process entry points.
 """
 from __future__ import annotations
 from functools import partial
-# import textwrap
+import multiprocessing as mp
+import os
+import textwrap
 from typing import (
     Any,
     TYPE_CHECKING,
@@ -58,25 +60,27 @@ def _mp_main(
 
 ) -> None:
     '''
-    The routine called *after fork* which invokes a fresh ``trio.run``
+    The routine called *after fork* which invokes a fresh `trio.run()`
 
     '''
     actor._forkserver_info = forkserver_info
     from ._spawn import try_set_start_method
-    spawn_ctx = try_set_start_method(start_method)
+    spawn_ctx: mp.context.BaseContext = try_set_start_method(start_method)
+    assert spawn_ctx
 
     if actor.loglevel is not None:
         log.info(
-            f"Setting loglevel for {actor.uid} to {actor.loglevel}")
+            f'Setting loglevel for {actor.uid} to {actor.loglevel}'
+        )
         get_console_log(actor.loglevel)
 
-    assert spawn_ctx
+    # TODO: use scops headers like for `trio` below!
+    # (well after we libify it maybe..)
     log.info(
-        f"Started new {spawn_ctx.current_process()} for {actor.uid}")
-
-    _state._current_actor = actor
-
-    log.debug(f"parent_addr is {parent_addr}")
+        f'Started new {spawn_ctx.current_process()} for {actor.uid}'
+    #     f"parent_addr is {parent_addr}"
+    )
+    _state._current_actor: Actor = actor
     trio_main = partial(
         async_main,
         actor=actor,
@@ -93,7 +97,110 @@ def _mp_main(
         pass  # handle it the same way trio does?
 
     finally:
-        log.info(f"Subactor {actor.uid} terminated")
+        log.info(
+            f'`mp`-subactor {actor.uid} exited'
+        )
+
+
+# TODO: move this func to some kinda `.devx._conc_lang.py` eventually
+# as we work out our multi-domain state-flow-syntax!
+def nest_from_op(
+    input_op: str,
+    #
+    # ?TODO? an idea for a syntax to the state of concurrent systems
+    # as a "3-domain" (execution, scope, storage) model and using
+    # a minimal ascii/utf-8 operator-set.
+    #
+    # try not to take any of this seriously yet XD
+    #
+    # > is a "play operator" indicating (CPU bound)
+    #   exec/work/ops required at the "lowest level computing"
+    #
+    # execution primititves (tasks, threads, actors..) denote their
+    # lifetime with '(' and ')' since parentheses normally are used
+    # in many langs to denote function calls.
+    #
+    # starting = (
+    # >(  opening/starting; beginning of the thread-of-exec (toe?)
+    # (>  opened/started,  (finished spawning toe)
+    # |_<Task: blah blah..>  repr of toe, in py these look like <objs>
+    #
+    # >) closing/exiting/stopping,
+    # )> closed/exited/stopped,
+    # |_<Task: blah blah..>
+    #   [OR <), )< ?? ]
+    #
+    # ending = )
+    # >c) cancelling to close/exit
+    # c)> cancelled (caused close), OR?
+    #  |_<Actor: ..>
+    #   OR maybe "<c)" which better indicates the cancel being
+    #   "delivered/returned" / returned" to LHS?
+    #
+    # >x)  erroring to eventuall exit
+    # x)>  errored and terminated
+    #  |_<Actor: ...>
+    #
+    # scopes: supers/nurseries, IPC-ctxs, sessions, perms, etc.
+    # >{  opening
+    # {>  opened
+    # }>  closed
+    # >}  closing
+    #
+    # storage: like queues, shm-buffers, files, etc..
+    # >[  opening
+    # [>  opened
+    #  |_<FileObj: ..>
+    #
+    # >]  closing
+    # ]>  closed
+
+    # IPC ops: channels, transports, msging
+    # =>  req msg
+    # <=  resp msg
+    # <=> 2-way streaming (of msgs)
+    # <-  recv 1 msg
+    # ->  send 1 msg
+    #
+    # TODO: still not sure on R/L-HS approach..?
+    # =>(  send-req to exec start (task, actor, thread..)
+    # (<=  recv-req to ^
+    #
+    # (<=  recv-req ^
+    # <=(  recv-resp opened remote exec primitive
+    # <=)  recv-resp closed
+    #
+    # )<=c req to stop due to cancel
+    # c=>) req to stop due to cancel
+    #
+    # =>{  recv-req to open
+    # <={  send-status that it closed
+
+    tree_str: str,
+
+    # NOTE: so move back-from-the-left of the `input_op` by
+    # this amount.
+    back_from_op: int = 0,
+) -> str:
+    '''
+    Depth-increment the input (presumably hierarchy/supervision)
+    input "tree string" below the provided `input_op` execution
+    operator, so injecting a `"\n|_{input_op}\n"`and indenting the
+    `tree_str` to nest content aligned with the ops last char.
+
+    '''
+    return (
+        f'{input_op}\n'
+        +
+        textwrap.indent(
+            tree_str,
+            prefix=(
+                len(input_op)
+                -
+                (back_from_op + 1)
+            ) * ' ',
+        )
+    )
 
 
 def _trio_main(
@@ -107,7 +214,6 @@ def _trio_main(
     Entry point for a `trio_run_in_process` subactor.
 
     '''
-    # __tracebackhide__: bool = True
     _debug.hide_runtime_frames()
 
     _state._current_actor = actor
@@ -119,7 +225,6 @@ def _trio_main(
 
     if actor.loglevel is not None:
         get_console_log(actor.loglevel)
-        import os
         actor_info: str = (
             f'|_{actor}\n'
             f'  uid: {actor.uid}\n'
@@ -128,13 +233,24 @@ def _trio_main(
             f'  loglevel: {actor.loglevel}\n'
         )
         log.info(
-            'Started new trio subactor:\n'
+            'Starting new `trio` subactor:\n'
             +
-            '>\n'  # like a "started/play"-icon from super perspective
-            +
-            actor_info,
+            nest_from_op(
+                input_op='>(',  # see syntax ideas above
+                tree_str=actor_info,
+                back_from_op=2,  # since "complete"
+            )
         )
-
+    logmeth = log.info
+    exit_status: str = (
+        'Subactor exited\n'
+        +
+        nest_from_op(
+            input_op=')>',  # like a "closed-to-play"-icon from super perspective
+            tree_str=actor_info,
+            back_from_op=1,
+        )
+    )
     try:
         if infect_asyncio:
             actor._infected_aio = True
@@ -143,16 +259,28 @@ def _trio_main(
             trio.run(trio_main)
 
     except KeyboardInterrupt:
-        log.cancel(
-            'Actor received KBI\n'
+        logmeth = log.cancel
+        exit_status: str = (
+            'Actor received KBI (aka an OS-cancel)\n'
             +
-            actor_info
+            nest_from_op(
+                input_op='c)>',  # closed due to cancel (see above)
+                tree_str=actor_info,
+            )
         )
+    except BaseException as err:
+        logmeth = log.error
+        exit_status: str = (
+            'Main actor task exited due to crash?\n'
+            +
+            nest_from_op(
+                input_op='x)>',  # closed by error
+                tree_str=actor_info,
+            )
+        )
+        # NOTE since we raise a tb will already be shown on the
+        # console, thus we do NOT use `.exception()` above.
+        raise err
+
     finally:
-        log.info(
-            'Subactor terminated\n'
-            +
-            'x\n'  # like a "crossed-out/killed" from super perspective
-            +
-            actor_info
-        )
+        logmeth(exit_status)

tractor/_exceptions.py

@@ -22,6 +22,7 @@ from __future__ import annotations
 import builtins
 import importlib
 from pprint import pformat
+from pdb import bdb
 import sys
 from types import (
     TracebackType,
@@ -82,6 +83,48 @@ class InternalError(RuntimeError):
 
     '''
 
+class AsyncioCancelled(Exception):
+    '''
+    Asyncio cancelled translation (non-base) error
+    for use with the ``to_asyncio`` module
+    to be raised in the ``trio`` side task
+
+    NOTE: this should NOT inherit from `asyncio.CancelledError` or
+    tests should break!
+
+    '''
+
+
+class AsyncioTaskExited(Exception):
+    '''
+    asyncio.Task "exited" translation error for use with the
+    `to_asyncio` APIs to be raised in the `trio` side task indicating
+    on `.run_task()`/`.open_channel_from()` exit that the aio side
+    exited early/silently.
+
+    '''
+
+class TrioCancelled(Exception):
+    '''
+    Trio cancelled translation (non-base) error
+    for use with the `to_asyncio` module
+    to be raised in the `asyncio.Task` to indicate
+    that the `trio` side raised `Cancelled` or an error.
+
+    '''
+
+class TrioTaskExited(Exception):
+    '''
+    The `trio`-side task exited without explicitly cancelling the
+    `asyncio.Task` peer.
+
+    This is very similar to how `trio.ClosedResource` acts as
+    a "clean shutdown" signal to the consumer side of a mem-chan,
+
+    https://trio.readthedocs.io/en/stable/reference-core.html#clean-shutdown-with-channels
+
+    '''
+
 
 # NOTE: more or less should be close to these:
 # 'boxed_type',
@@ -127,8 +170,8 @@ _body_fields: list[str] = list(
 
 def get_err_type(type_name: str) -> BaseException|None:
     '''
-    Look up an exception type by name from the set of locally
-    known namespaces:
+    Look up an exception type by name from the set of locally known
+    namespaces:
 
     - `builtins`
     - `tractor._exceptions`
@@ -139,6 +182,7 @@ def get_err_type(type_name: str) -> BaseException|None:
         builtins,
         _this_mod,
         trio,
+        bdb,
     ]:
         if type_ref := getattr(
             ns,
@@ -358,6 +402,13 @@ class RemoteActorError(Exception):
                 self._ipc_msg.src_type_str
             )
 
+            if not self._src_type:
+                raise TypeError(
+                    f'Failed to lookup src error type with '
+                    f'`tractor._exceptions.get_err_type()` :\n'
+                    f'{self.src_type_str}'
+                )
+
         return self._src_type
 
     @property
@@ -366,6 +417,9 @@ class RemoteActorError(Exception):
         String-name of the (last hop's) boxed error type.
 
         '''
+        # TODO, maybe support also serializing the
+        # `ExceptionGroup.exeptions: list[BaseException]` set under
+        # certain conditions?
         bt: Type[BaseException] = self.boxed_type
         if bt:
             return str(bt.__name__)
@@ -378,9 +432,13 @@ class RemoteActorError(Exception):
         Error type boxed by last actor IPC hop.
 
         '''
-        if self._boxed_type is None:
+        if (
+            self._boxed_type is None
+            and
+            (ipc_msg := self._ipc_msg)
+        ):
             self._boxed_type = get_err_type(
-                self._ipc_msg.boxed_type_str
+                ipc_msg.boxed_type_str
             )
 
         return self._boxed_type
@@ -609,6 +667,7 @@ class RemoteActorError(Exception):
                 # just after <Type(
                 #             |___ ..
                 tb_body_indent=1,
+                boxer_header=self.relay_uid,
             )
 
         tail = ''
@@ -651,16 +710,10 @@ class RemoteActorError(Exception):
         failing actor's remote env.
 
         '''
-        src_type_ref: Type[BaseException] = self.src_type
-        if not src_type_ref:
-            raise TypeError(
-                'Failed to lookup src error type:\n'
-                f'{self.src_type_str}'
-            )
-
         # TODO: better tb insertion and all the fancier dunder
         # metadata stuff as per `.__context__` etc. and friends:
         # https://github.com/python-trio/trio/issues/611
+        src_type_ref: Type[BaseException] = self.src_type
         return src_type_ref(self.tb_str)
 
     # TODO: local recontruction of nested inception for a given
@@ -786,8 +839,11 @@ class MsgTypeError(
         '''
         if (
             (_bad_msg := self.msgdata.get('_bad_msg'))
-            and
-            isinstance(_bad_msg, PayloadMsg)
+            and (
+                isinstance(_bad_msg, PayloadMsg)
+                or
+                isinstance(_bad_msg, msgtypes.Start)
+            )
         ):
             return _bad_msg
 
@@ -906,8 +962,59 @@ class StreamOverrun(
     '''
 
 
-class TransportClosed(trio.ClosedResourceError):
-    "Underlying channel transport was closed prior to use"
+class TransportClosed(trio.BrokenResourceError):
+    '''
+    IPC transport (protocol) connection was closed or broke and
+    indicates that the wrapping communication `Channel` can no longer
+    be used to send/receive msgs from the remote peer.
+
+    '''
+    def __init__(
+        self,
+        message: str,
+        loglevel: str = 'transport',
+        cause: BaseException|None = None,
+        raise_on_report: bool = False,
+
+    ) -> None:
+        self.message: str = message
+        self._loglevel = loglevel
+        super().__init__(message)
+
+        if cause is not None:
+            self.__cause__ = cause
+
+        # flag to toggle whether the msg loop should raise
+        # the exc in its `TransportClosed` handler block.
+        self._raise_on_report = raise_on_report
+
+    def report_n_maybe_raise(
+        self,
+        message: str|None = None,
+
+    ) -> None:
+        '''
+        Using the init-specified log level emit a logging report
+        for this error.
+
+        '''
+        message: str = message or self.message
+        # when a cause is set, slap it onto the log emission.
+        if cause := self.__cause__:
+            cause_tb_str: str = ''.join(
+                traceback.format_tb(cause.__traceback__)
+            )
+            message += (
+                f'{cause_tb_str}\n'  # tb
+                f'    {cause}\n'  # exc repr
+            )
+
+        getattr(log, self._loglevel)(message)
+
+        # some errors we want to blow up from
+        # inside the RPC msg loop
+        if self._raise_on_report:
+            raise self from cause
 
 
 class NoResult(RuntimeError):
@@ -922,15 +1029,6 @@ class NoRuntime(RuntimeError):
     "The root actor has not been initialized yet"
 
 
-
-class AsyncioCancelled(Exception):
-    '''
-    Asyncio cancelled translation (non-base) error
-    for use with the ``to_asyncio`` module
-    to be raised in the ``trio`` side task
-
-    '''
-
 class MessagingError(Exception):
     '''
     IPC related msg (typing), transaction (ordering) or dialog
@@ -938,7 +1036,6 @@ class MessagingError(Exception):
 
     '''
 
-
 def pack_error(
     exc: BaseException|RemoteActorError,
 
@@ -1050,6 +1147,8 @@ def unpack_error(
     which is the responsibilitiy of the caller.
 
     '''
+    # XXX, apparently we pass all sorts of msgs here?
+    # kinda odd but seems like maybe they shouldn't be?
     if not isinstance(msg, Error):
         return None
 
@@ -1092,19 +1191,51 @@ def unpack_error(
 
 
 def is_multi_cancelled(
-    exc: BaseException|BaseExceptionGroup
-) -> bool:
+    exc: BaseException|BaseExceptionGroup,
+
+    ignore_nested: set[BaseException] = set(),
+
+) -> bool|BaseExceptionGroup:
     '''
-    Predicate to determine if a possible ``BaseExceptionGroup`` contains
-    only ``trio.Cancelled`` sub-exceptions (and is likely the result of
-    cancelling a collection of subtasks.
+    Predicate to determine if an `BaseExceptionGroup` only contains
+    some (maybe nested) set of sub-grouped exceptions (like only
+    `trio.Cancelled`s which get swallowed silently by default) and is
+    thus the result of "gracefully cancelling" a collection of
+    sub-tasks (or other conc primitives) and receiving a "cancelled
+    ACK" from each after termination.
+
+    Docs:
+    ----
+    - https://docs.python.org/3/library/exceptions.html#exception-groups
+    - https://docs.python.org/3/library/exceptions.html#BaseExceptionGroup.subgroup
 
     '''
+
+    if (
+        not ignore_nested
+        or
+        trio.Cancelled in ignore_nested
+        # XXX always count-in `trio`'s native signal
+    ):
+        ignore_nested.update({trio.Cancelled})
+
     if isinstance(exc, BaseExceptionGroup):
-        return exc.subgroup(
-            lambda exc: isinstance(exc, trio.Cancelled)
-        ) is not None
+        matched_exc: BaseExceptionGroup|None = exc.subgroup(
+            tuple(ignore_nested),
 
+            # TODO, complain about why not allowed XD
+            # condition=tuple(ignore_nested),
+        )
+        if matched_exc is not None:
+            return matched_exc
+
+    # NOTE, IFF no excs types match (throughout the error-tree)
+    # -> return `False`, OW return the matched sub-eg.
+    #
+    # IOW, for the inverse of ^ for the purpose of
+    # maybe-enter-REPL--logic: "only debug when the err-tree contains
+    # at least one exc-type NOT in `ignore_nested`" ; i.e. the case where
+    # we fallthrough and return `False` here.
     return False
 
 
@@ -1324,7 +1455,9 @@ def _mk_recv_mte(
         any_pld: Any = msgpack.decode(msg.pld)
         message: str = (
             f'invalid `{msg_type.__qualname__}` msg payload\n\n'
-            f'value: `{any_pld!r}` does not match type-spec: '
+            f'{any_pld!r}\n\n'
+            f'has type {type(any_pld)!r}\n\n'
+            f'and does not match type-spec '
             f'`{type(msg).__qualname__}.pld: {codec.pld_spec_str}`'
         )
         bad_msg = msg

tractor/_ipc.py

@@ -54,7 +54,7 @@ from tractor._exceptions import (
 )
 from tractor.msg import (
     _ctxvar_MsgCodec,
-    _codec,
+    # _codec,  XXX see `self._codec` sanity/debug checks
     MsgCodec,
     types as msgtypes,
     pretty_struct,
@@ -65,8 +65,18 @@ log = get_logger(__name__)
 _is_windows = platform.system() == 'Windows'
 
 
-def get_stream_addrs(stream: trio.SocketStream) -> tuple:
-    # should both be IP sockets
+def get_stream_addrs(
+    stream: trio.SocketStream
+) -> tuple[
+    tuple[str, int],  # local
+    tuple[str, int],  # remote
+]:
+    '''
+    Return the `trio` streaming transport prot's socket-addrs for
+    both the local and remote sides as a pair.
+
+    '''
+    # rn, should both be IP sockets
     lsockname = stream.socket.getsockname()
     rsockname = stream.socket.getpeername()
     return (
@@ -75,17 +85,22 @@ def get_stream_addrs(stream: trio.SocketStream) -> tuple:
     )
 
 
-# TODO: this should be our `Union[*msgtypes.__spec__]` now right?
-MsgType = TypeVar("MsgType")
-
-# TODO: consider using a generic def and indexing with our eventual
-# msg definition/types?
-# - https://docs.python.org/3/library/typing.html#typing.Protocol
-# - https://jcristharif.com/msgspec/usage.html#structs
+# from tractor.msg.types import MsgType
+# ?TODO? this should be our `Union[*msgtypes.__spec__]` alias now right..?
+# => BLEH, except can't bc prots must inherit typevar or param-spec
+#   vars..
+MsgType = TypeVar('MsgType')
 
 
+# TODO: break up this mod into a subpkg so we can start adding new
+# backends and move this type stuff into a dedicated file.. Bo
+#
 @runtime_checkable
 class MsgTransport(Protocol[MsgType]):
+#
+# ^-TODO-^ consider using a generic def and indexing with our
+# eventual msg definition/types?
+# - https://docs.python.org/3/library/typing.html#typing.Protocol
 
     stream: trio.SocketStream
     drained: list[MsgType]
@@ -120,9 +135,9 @@ class MsgTransport(Protocol[MsgType]):
         ...
 
 
-# TODO: not sure why we have to inherit here, but it seems to be an
-# issue with ``get_msg_transport()`` returning a ``Type[Protocol]``;
-# probably should make a `mypy` issue?
+# TODO: typing oddity.. not sure why we have to inherit here, but it
+# seems to be an issue with `get_msg_transport()` returning
+# a `Type[Protocol]`; probably should make a `mypy` issue?
 class MsgpackTCPStream(MsgTransport):
     '''
     A ``trio.SocketStream`` delivering ``msgpack`` formatted data
@@ -145,7 +160,7 @@ class MsgpackTCPStream(MsgTransport):
         # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
         #
         # TODO: define this as a `Codec` struct which can be
-        # overriden dynamically by the application/runtime.
+        # overriden dynamically by the application/runtime?
         codec: tuple[
             Callable[[Any], Any]|None,  # coder
             Callable[[type, Any], Any]|None,  # decoder
@@ -160,7 +175,7 @@ class MsgpackTCPStream(MsgTransport):
         self._laddr, self._raddr = get_stream_addrs(stream)
 
         # create read loop instance
-        self._agen = self._iter_packets()
+        self._aiter_pkts = self._iter_packets()
         self._send_lock = trio.StrictFIFOLock()
 
         # public i guess?
@@ -174,15 +189,12 @@ class MsgpackTCPStream(MsgTransport):
         # allow for custom IPC msg interchange format
         # dynamic override Bo
         self._task = trio.lowlevel.current_task()
-        self._codec: MsgCodec = (
-            codec
-            or
-            _codec._ctxvar_MsgCodec.get()
-        )
-        # TODO: mask out before release?
-        # log.runtime(
-        #     f'New {self} created with codec\n'
-        #     f'codec: {self._codec}\n'
+
+        # XXX for ctxvar debug only!
+        # self._codec: MsgCodec = (
+        #     codec
+        #     or
+        #     _codec._ctxvar_MsgCodec.get()
         # )
 
     async def _iter_packets(self) -> AsyncGenerator[dict, None]:
@@ -190,6 +202,11 @@ class MsgpackTCPStream(MsgTransport):
         Yield `bytes`-blob decoded packets from the underlying TCP
         stream using the current task's `MsgCodec`.
 
+        This is a streaming routine implemented as an async generator
+        func (which was the original design, but could be changed?)
+        and is allocated by a `.__call__()` inside `.__init__()` where
+        it is assigned to the `._aiter_pkts` attr.
+
         '''
         decodes_failed: int = 0
 
@@ -204,16 +221,82 @@ class MsgpackTCPStream(MsgTransport):
                 # seem to be getting racy failures here on
                 # arbiter/registry name subs..
                 trio.BrokenResourceError,
-            ):
-                raise TransportClosed(
-                    f'transport {self} was already closed prior ro read'
-                )
 
+            ) as trans_err:
+
+                loglevel = 'transport'
+                match trans_err:
+                    # case (
+                    #     ConnectionResetError()
+                    # ):
+                    #     loglevel = 'transport'
+
+                    # peer actor (graceful??) TCP EOF but `tricycle`
+                    # seems to raise a 0-bytes-read?
+                    case ValueError() if (
+                        'unclean EOF' in trans_err.args[0]
+                    ):
+                        pass
+
+                    # peer actor (task) prolly shutdown quickly due
+                    # to cancellation
+                    case trio.BrokenResourceError() if (
+                        'Connection reset by peer' in trans_err.args[0]
+                    ):
+                        pass
+
+                    # unless the disconnect condition falls under "a
+                    # normal operation breakage" we usualy console warn
+                    # about it.
+                    case _:
+                        loglevel: str = 'warning'
+
+
+                raise TransportClosed(
+                    message=(
+                        f'IPC transport already closed by peer\n'
+                        f'x]> {type(trans_err)}\n'
+                        f'  |_{self}\n'
+                    ),
+                    loglevel=loglevel,
+                ) from trans_err
+
+            # XXX definitely can happen if transport is closed
+            # manually by another `trio.lowlevel.Task` in the
+            # same actor; we use this in some simulated fault
+            # testing for ex, but generally should never happen
+            # under normal operation!
+            #
+            # NOTE: as such we always re-raise this error from the
+            #       RPC msg loop!
+            except trio.ClosedResourceError as closure_err:
+                raise TransportClosed(
+                    message=(
+                        f'IPC transport already manually closed locally?\n'
+                        f'x]> {type(closure_err)} \n'
+                        f'  |_{self}\n'
+                    ),
+                    loglevel='error',
+                    raise_on_report=(
+                        closure_err.args[0] == 'another task closed this fd'
+                        or
+                        closure_err.args[0] in ['another task closed this fd']
+                    ),
+                ) from closure_err
+
+            # graceful TCP EOF disconnect
             if header == b'':
                 raise TransportClosed(
-                    f'transport {self} was already closed prior ro read'
+                    message=(
+                        f'IPC transport already gracefully closed\n'
+                        f']>\n'
+                        f' |_{self}\n'
+                    ),
+                    loglevel='transport',
+                    # cause=???  # handy or no?
                 )
 
+            size: int
             size, = struct.unpack("<I", header)
 
             log.transport(f'received header {size}')  # type: ignore
@@ -225,33 +308,20 @@ class MsgpackTCPStream(MsgTransport):
                 # the current `MsgCodec`.
                 codec: MsgCodec = _ctxvar_MsgCodec.get()
 
-                # TODO: mask out before release?
-                if self._codec.pld_spec != codec.pld_spec:
-                    # assert (
-                    #     task := trio.lowlevel.current_task()
-                    # ) is not self._task
-                    # self._task = task
-                    self._codec = codec
-                    log.runtime(
-                        f'Using new codec in {self}.recv()\n'
-                        f'codec: {self._codec}\n\n'
-                        f'msg_bytes: {msg_bytes}\n'
-                    )
+                # XXX for ctxvar debug only!
+                # if self._codec.pld_spec != codec.pld_spec:
+                #     assert (
+                #         task := trio.lowlevel.current_task()
+                #     ) is not self._task
+                #     self._task = task
+                #     self._codec = codec
+                #     log.runtime(
+                #         f'Using new codec in {self}.recv()\n'
+                #         f'codec: {self._codec}\n\n'
+                #         f'msg_bytes: {msg_bytes}\n'
+                #     )
                 yield codec.decode(msg_bytes)
 
-                # TODO: remove, was only for orig draft impl
-                # testing.
-                #
-                # curr_codec: MsgCodec = _ctxvar_MsgCodec.get()
-                # obj = curr_codec.decode(msg_bytes)
-                # if (
-                #     curr_codec is not
-                #     _codec._def_msgspec_codec
-                # ):
-                #     print(f'OBJ: {obj}\n')
-                #
-                # yield obj
-
             # XXX NOTE: since the below error derives from
             # `DecodeError` we need to catch is specially
             # and always raise such that spec violations
@@ -295,7 +365,8 @@ class MsgpackTCPStream(MsgTransport):
         msg: msgtypes.MsgType,
 
         strict_types: bool = True,
-        # hide_tb: bool = False,
+        hide_tb: bool = False,
+
     ) -> None:
         '''
         Send a msgpack encoded py-object-blob-as-msg over TCP.
@@ -304,21 +375,24 @@ class MsgpackTCPStream(MsgTransport):
         invalid msg type
 
         '''
-        # __tracebackhide__: bool = hide_tb
+        __tracebackhide__: bool = hide_tb
+
+        # XXX see `trio._sync.AsyncContextManagerMixin` for details
+        # on the `.acquire()`/`.release()` sequencing..
         async with self._send_lock:
 
             # NOTE: lookup the `trio.Task.context`'s var for
             # the current `MsgCodec`.
             codec: MsgCodec = _ctxvar_MsgCodec.get()
 
-            # TODO: mask out before release?
-            if self._codec.pld_spec != codec.pld_spec:
-                self._codec = codec
-                log.runtime(
-                    f'Using new codec in {self}.send()\n'
-                    f'codec: {self._codec}\n\n'
-                    f'msg: {msg}\n'
-                )
+            # XXX for ctxvar debug only!
+            # if self._codec.pld_spec != codec.pld_spec:
+            #     self._codec = codec
+            #     log.runtime(
+            #         f'Using new codec in {self}.send()\n'
+            #         f'codec: {self._codec}\n\n'
+            #         f'msg: {msg}\n'
+            #     )
 
             if type(msg) not in msgtypes.__msg_types__:
                 if strict_types:
@@ -352,6 +426,16 @@ class MsgpackTCPStream(MsgTransport):
             size: bytes = struct.pack("<I", len(bytes_data))
             return await self.stream.send_all(size + bytes_data)
 
+        # ?TODO? does it help ever to dynamically show this
+        # frame?
+        # try:
+        #     <the-above_code>
+        # except BaseException as _err:
+        #     err = _err
+        #     if not isinstance(err, MsgTypeError):
+        #         __tracebackhide__: bool = False
+        #     raise
+
     @property
     def laddr(self) -> tuple[str, int]:
         return self._laddr
@@ -361,7 +445,7 @@ class MsgpackTCPStream(MsgTransport):
         return self._raddr
 
     async def recv(self) -> Any:
-        return await self._agen.asend(None)
+        return await self._aiter_pkts.asend(None)
 
     async def drain(self) -> AsyncIterator[dict]:
         '''
@@ -378,7 +462,7 @@ class MsgpackTCPStream(MsgTransport):
                 yield msg
 
     def __aiter__(self):
-        return self._agen
+        return self._aiter_pkts
 
     def connected(self) -> bool:
         return self.stream.socket.fileno() != -1
@@ -433,7 +517,7 @@ class Channel:
         # set after handshake - always uid of far end
         self.uid: tuple[str, str]|None = None
 
-        self._agen = self._aiter_recv()
+        self._aiter_msgs = self._iter_msgs()
         self._exc: Exception|None = None  # set if far end actor errors
         self._closed: bool = False
 
@@ -497,8 +581,6 @@ class Channel:
         )
         return self._transport
 
-    # TODO: something simliar at the IPC-`Context`
-    # level so as to support 
     @cm
     def apply_codec(
         self,
@@ -517,6 +599,7 @@ class Channel:
         finally:
             self._transport.codec = orig
 
+    # TODO: do a .src/.dst: str for maddrs?
     def __repr__(self) -> str:
         if not self._transport:
             return '<Channel with inactive transport?>'
@@ -560,27 +643,43 @@ class Channel:
         )
         return transport
 
+    # TODO: something like,
+    # `pdbp.hideframe_on(errors=[MsgTypeError])`
+    # instead of the `try/except` hack we have rn..
+    # seems like a pretty useful thing to have in general
+    # along with being able to filter certain stack frame(s / sets)
+    # possibly based on the current log-level?
     async def send(
         self,
         payload: Any,
 
-        # hide_tb: bool = False,
+        hide_tb: bool = False,
 
     ) -> None:
         '''
         Send a coded msg-blob over the transport.
 
         '''
-        # __tracebackhide__: bool = hide_tb
-        log.transport(
-            '=> send IPC msg:\n\n'
-            f'{pformat(payload)}\n'
-        )  # type: ignore
-        assert self._transport
-        await self._transport.send(
-            payload,
-            # hide_tb=hide_tb,
-        )
+        __tracebackhide__: bool = hide_tb
+        try:
+            log.transport(
+                '=> send IPC msg:\n\n'
+                f'{pformat(payload)}\n'
+            )
+            # assert self._transport  # but why typing?
+            await self._transport.send(
+                payload,
+                hide_tb=hide_tb,
+            )
+        except BaseException as _err:
+            err = _err  # bind for introspection
+            if not isinstance(_err, MsgTypeError):
+                # assert err
+                __tracebackhide__: bool = False
+            else:
+                assert err.cid
+
+            raise
 
     async def recv(self) -> Any:
         assert self._transport
@@ -617,8 +716,11 @@ class Channel:
         await self.aclose(*args)
 
     def __aiter__(self):
-        return self._agen
+        return self._aiter_msgs
 
+    # ?TODO? run any reconnection sequence?
+    # -[ ] prolly should be impl-ed as deco-API?
+    #
     # async def _reconnect(self) -> None:
     #     """Handle connection failures by polling until a reconnect can be
     #     established.
@@ -636,7 +738,6 @@ class Channel:
     #             else:
     #                 log.transport("Stream connection re-established!")
 
-    #                 # TODO: run any reconnection sequence
     #                 # on_recon = self._recon_seq
     #                 # if on_recon:
     #                 #     await on_recon(self)
@@ -650,11 +751,17 @@ class Channel:
     #                     " for re-establishment")
     #             await trio.sleep(1)
 
-    async def _aiter_recv(
+    async def _iter_msgs(
         self
     ) -> AsyncGenerator[Any, None]:
         '''
-        Async iterate items from underlying stream.
+        Yield `MsgType` IPC msgs decoded and deliverd from
+        an underlying `MsgTransport` protocol.
+
+        This is a streaming routine alo implemented as an async-gen
+        func (same a `MsgTransport._iter_pkts()`) gets allocated by
+        a `.__call__()` inside `.__init__()` where it is assigned to
+        the `._aiter_msgs` attr.
 
         '''
         assert self._transport
@@ -680,15 +787,6 @@ class Channel:
                         case _:
                             yield msg
 
-                    # TODO: if we were gonna do this it should be
-                    # done up at the `MsgStream` layer!
-                    #
-                    # sent = yield item
-                    # if sent is not None:
-                    #     # optimization, passing None through all the
-                    #     # time is pointless
-                    #     await self._transport.send(sent)
-
             except trio.BrokenResourceError:
 
                 # if not self._autorecon:

tractor/_portal.py

@@ -97,7 +97,7 @@ class Portal:
         channel: Channel,
     ) -> None:
 
-        self.chan = channel
+        self._chan: Channel = channel
         # during the portal's lifetime
         self._final_result_pld: Any|None = None
         self._final_result_msg: PayloadMsg|None = None
@@ -109,6 +109,10 @@ class Portal:
         self._streams: set[MsgStream] = set()
         self.actor: Actor = current_actor()
 
+    @property
+    def chan(self) -> Channel:
+        return self._chan
+
     @property
     def channel(self) -> Channel:
         '''
@@ -121,7 +125,8 @@ class Portal:
         )
         return self.chan
 
-    # TODO: factor this out into an `ActorNursery` wrapper
+    # TODO: factor this out into a `.highlevel` API-wrapper that uses
+    # a single `.open_context()` call underneath.
     async def _submit_for_result(
         self,
         ns: str,
@@ -141,13 +146,22 @@ class Portal:
             portal=self,
         )
 
+    # TODO: we should deprecate this API right? since if we remove
+    # `.run_in_actor()` (and instead move it to a `.highlevel`
+    # wrapper api (around a single `.open_context()` call) we don't
+    # really have any notion of a "main" remote task any more?
+    #
     # @api_frame
-    async def result(self) -> Any:
+    async def wait_for_result(
+        self,
+        hide_tb: bool = True,
+    ) -> Any:
         '''
-        Return the result(s) from the remote actor's "main" task.
+        Return the final result delivered by a `Return`-msg from the
+        remote peer actor's "main" task's `return` statement.
 
         '''
-        __tracebackhide__ = True
+        __tracebackhide__: bool = hide_tb
         # Check for non-rpc errors slapped on the
         # channel for which we always raise
         exc = self.channel._exc
@@ -170,7 +184,7 @@ class Portal:
                 (
                     self._final_result_msg,
                     self._final_result_pld,
-                ) = await self._expect_result_ctx._pld_rx.recv_msg_w_pld(
+                ) = await self._expect_result_ctx._pld_rx.recv_msg(
                     ipc=self._expect_result_ctx,
                     expect_msg=Return,
                 )
@@ -182,6 +196,23 @@ class Portal:
 
         return self._final_result_pld
 
+    # TODO: factor this out into a `.highlevel` API-wrapper that uses
+    # a single `.open_context()` call underneath.
+    async def result(
+        self,
+        *args,
+        **kwargs,
+    ) -> Any|Exception:
+        typname: str = type(self).__name__
+        log.warning(
+            f'`{typname}.result()` is DEPRECATED!\n'
+            f'Use `{typname}.wait_for_result()` instead!\n'
+        )
+        return await self.wait_for_result(
+            *args,
+            **kwargs,
+        )
+
     async def _cancel_streams(self):
         # terminate all locally running async generator
         # IPC calls
@@ -232,14 +263,15 @@ class Portal:
             return False
 
         reminfo: str = (
-            f'Portal.cancel_actor() => {self.channel.uid}\n'
-            f'|_{chan}\n'
+            f'c)=> {self.channel.uid}\n'
+            f'  |_{chan}\n'
         )
         log.cancel(
-            f'Requesting runtime cancel for peer\n\n'
+            f'Requesting actor-runtime cancel for peer\n\n'
             f'{reminfo}'
         )
 
+        # XXX the one spot we set it?
         self.channel._cancel_called: bool = True
         try:
             # send cancel cmd - might not get response
@@ -279,6 +311,8 @@ class Portal:
             )
             return False
 
+    # TODO: do we still need this for low level `Actor`-runtime
+    # method calls or can we also remove it?
     async def run_from_ns(
         self,
         namespace_path: str,
@@ -316,6 +350,8 @@ class Portal:
             expect_msg=Return,
         )
 
+    # TODO: factor this out into a `.highlevel` API-wrapper that uses
+    # a single `.open_context()` call underneath.
     async def run(
         self,
         func: str,
@@ -370,6 +406,8 @@ class Portal:
             expect_msg=Return,
         )
 
+    # TODO: factor this out into a `.highlevel` API-wrapper that uses
+    # a single `.open_context()` call underneath.
     @acm
     async def open_stream_from(
         self,
@@ -495,6 +533,10 @@ async def open_portal(
     async with maybe_open_nursery(
         tn,
         shield=shield,
+        strict_exception_groups=False,
+        # ^XXX^ TODO? soo roll our own then ??
+        # -> since we kinda want the "if only one `.exception` then
+        # just raise that" interface?
     ) as tn:
 
         if not channel.connected():

tractor/_root.py

@@ -21,6 +21,7 @@ Root actor runtime ignition(s).
 from contextlib import asynccontextmanager as acm
 from functools import partial
 import importlib
+import inspect
 import logging
 import os
 import signal
@@ -69,10 +70,7 @@ async def open_root_actor(
 
     # defaults are above
     arbiter_addr: tuple[str, int]|None = None,
-    
-    # binding addrs for the transport layer server
-    trans_bind_addrs: list[tuple[str, int]] = [(_default_host, _default_port)],
-    
+
     name: str|None = 'root',
 
     # either the `multiprocessing` start method:
@@ -82,7 +80,7 @@ async def open_root_actor(
 
     # enables the multi-process debugger support
     debug_mode: bool = False,
-    maybe_enable_greenback: bool = False,  # `.pause_from_sync()/breakpoint()` support
+    maybe_enable_greenback: bool = True,  # `.pause_from_sync()/breakpoint()` support
     enable_stack_on_sig: bool = False,
 
     # internal logging
@@ -97,13 +95,24 @@ async def open_root_actor(
 
     hide_tb: bool = True,
 
+    # XXX, proxied directly to `.devx._debug._maybe_enter_pm()`
+    # for REPL-entry logic.
+    debug_filter: Callable[
+        [BaseException|BaseExceptionGroup],
+        bool,
+    ] = lambda err: not is_multi_cancelled(err),
+
+    # TODO, a way for actors to augment passing derived
+    # read-only state to sublayers?
+    # extra_rt_vars: dict|None = None,
+
 ) -> Actor:
     '''
     Runtime init entry point for ``tractor``.
 
     '''
-    __tracebackhide__: bool = hide_tb
     _debug.hide_runtime_frames()
+    __tracebackhide__: bool = hide_tb
 
     # TODO: stick this in a `@cm` defined in `devx._debug`?
     #
@@ -118,10 +127,16 @@ async def open_root_actor(
     if (
         debug_mode
         and maybe_enable_greenback
-        and await _debug.maybe_init_greenback(
-            raise_not_found=False,
+        and (
+            maybe_mod := await _debug.maybe_init_greenback(
+                raise_not_found=False,
+            )
         )
     ):
+        logger.info(
+            f'Found `greenback` installed @ {maybe_mod}\n'
+            'Enabling `tractor.pause_from_sync()` support!\n'
+        )
         os.environ['PYTHONBREAKPOINT'] = (
             'tractor.devx._debug._sync_pause_from_builtin'
         )
@@ -186,8 +201,6 @@ async def open_root_actor(
         _default_lo_addrs
     )
     assert registry_addrs
-    
-    assert trans_bind_addrs
 
     loglevel = (
         loglevel
@@ -231,14 +244,8 @@ async def open_root_actor(
         and
         enable_stack_on_sig
     ):
-        try:
-            logger.info('Enabling `stackscope` traces on SIGUSR1')
-            from .devx import enable_stack_on_sig
-            enable_stack_on_sig()
-        except ImportError:
-            logger.warning(
-                '`stackscope` not installed for use in debug mode!'
-            )
+        from .devx._stackscope import enable_stack_on_sig
+        enable_stack_on_sig()
 
     # closed into below ping task-func
     ponged_addrs: list[tuple[str, int]] = []
@@ -269,7 +276,9 @@ async def open_root_actor(
 
         except OSError:
             # TODO: make this a "discovery" log level?
-            logger.warning(f'No actor registry found @ {addr}')
+            logger.info(
+                f'No actor registry found @ {addr}\n'
+            )
 
     async with trio.open_nursery() as tn:
         for addr in registry_addrs:
@@ -278,10 +287,11 @@ async def open_root_actor(
                 tuple(addr),  # TODO: just drop this requirement?
             )
 
+    trans_bind_addrs: list[tuple[str, int]] = []
+
     # Create a new local root-actor instance which IS NOT THE
     # REGISTRAR
     if ponged_addrs:
-
         if ensure_registry:
             raise RuntimeError(
                  f'Failed to open `{name}`@{ponged_addrs}: '
@@ -299,6 +309,11 @@ async def open_root_actor(
             loglevel=loglevel,
             enable_modules=enable_modules,
         )
+        # DO NOT use the registry_addrs as the transport server
+        # addrs for this new non-registar, root-actor.
+        for host, port in ponged_addrs:
+            # NOTE: zero triggers dynamic OS port allocation
+            trans_bind_addrs.append((host, 0))
 
     # Start this local actor as the "registrar", aka a regular
     # actor who manages the local registry of "mailboxes" of
@@ -326,6 +341,10 @@ async def open_root_actor(
             loglevel=loglevel,
             enable_modules=enable_modules,
         )
+        # XXX, in case the root actor runtime was actually run from
+        # `tractor.to_asyncio.run_as_asyncio_guest()` and NOt
+        # `.trio.run()`.
+        actor._infected_aio = _state._runtime_vars['_is_infected_aio']
 
     # Start up main task set via core actor-runtime nurseries.
     try:
@@ -343,7 +362,10 @@ async def open_root_actor(
         )
 
         # start the actor runtime in a new task
-        async with trio.open_nursery() as nursery:
+        async with trio.open_nursery(
+            strict_exception_groups=False,
+            # ^XXX^ TODO? instead unpack any RAE as per "loose" style?
+        ) as nursery:
 
             # ``_runtime.async_main()`` creates an internal nursery
             # and blocks here until any underlying actor(-process)
@@ -363,23 +385,36 @@ async def open_root_actor(
             )
             try:
                 yield actor
-
             except (
                 Exception,
                 BaseExceptionGroup,
             ) as err:
 
-                import inspect
+                # TODO, in beginning to handle the subsubactor with
+                # crashed grandparent cases..
+                #
+                # was_locked: bool = await _debug.maybe_wait_for_debugger(
+                #     child_in_debug=True,
+                # )
+                # XXX NOTE XXX see equiv note inside
+                # `._runtime.Actor._stream_handler()` where in the
+                # non-root or root-that-opened-this-mahually case we
+                # wait for the local actor-nursery to exit before
+                # exiting the transport channel handler.
                 entered: bool = await _debug._maybe_enter_pm(
                     err,
                     api_frame=inspect.currentframe(),
+                    debug_filter=debug_filter,
                 )
 
                 if (
                     not entered
-                    and not is_multi_cancelled(err)
+                    and
+                    not is_multi_cancelled(
+                        err,
+                    )
                 ):
-                    logger.exception('Root actor crashed:\n')
+                    logger.exception('Root actor crashed\n')
 
                 # ALWAYS re-raise any error bubbled up from the
                 # runtime!
@@ -431,12 +466,19 @@ def run_daemon(
 
     start_method: str | None = None,
     debug_mode: bool = False,
+
+    # TODO, support `infected_aio=True` mode by,
+    # - calling the appropriate entrypoint-func from `.to_asyncio`
+    # - maybe init-ing `greenback` as done above in
+    #   `open_root_actor()`.
+
     **kwargs
 
 ) -> None:
     '''
-    Spawn daemon actor which will respond to RPC; the main task simply
-    starts the runtime and then sleeps forever.
+    Spawn a root (daemon) actor which will respond to RPC; the main
+    task simply starts the runtime and then blocks via embedded
+    `trio.sleep_forever()`.
 
     This is a very minimal convenience wrapper around starting
     a "run-until-cancelled" root actor which can be started with a set
@@ -449,7 +491,6 @@ def run_daemon(
         importlib.import_module(path)
 
     async def _main():
-
         async with open_root_actor(
             registry_addrs=registry_addrs,
             name=name,

tractor/_rpc.py

@@ -57,7 +57,6 @@ from ._exceptions import (
     unpack_error,
 )
 from .devx import (
-    maybe_wait_for_debugger,
     _debug,
     add_div,
 )
@@ -68,7 +67,7 @@ from .msg import (
     MsgCodec,
     PayloadT,
     NamespacePath,
-    pretty_struct,
+    # pretty_struct,
     _ops as msgops,
 )
 from tractor.msg.types import (
@@ -89,6 +88,16 @@ if TYPE_CHECKING:
 log = get_logger('tractor')
 
 
+# ?TODO? move to a `tractor.lowlevel._rpc` with the below
+# func-type-cases implemented "on top of" `@context` defs:
+# -[ ] std async func helper decorated with `@rpc_func`?
+# -[ ] `Portal.open_stream_from()` with async-gens?
+#  |_ possibly a duplex form of this with a
+#    `sent_from_peer = yield send_to_peer` form, which would require
+#    syncing the send/recv side with possibly `.receive_nowait()`
+#    on each `yield`?
+# -[ ] some kinda `@rpc_acm` maybe that does a fixture style with
+#     user only defining a single-`yield` generator-func?
 async def _invoke_non_context(
     actor: Actor,
     cancel_scope: CancelScope,
@@ -108,8 +117,9 @@ async def _invoke_non_context(
     ] = trio.TASK_STATUS_IGNORED,
 ):
     __tracebackhide__: bool = True
+    cs: CancelScope|None = None  # ref when activated
 
-    # TODO: can we unify this with the `context=True` impl below?
+    # ?TODO? can we unify this with the `context=True` impl below?
     if inspect.isasyncgen(coro):
         await chan.send(
             StartAck(
@@ -160,10 +170,6 @@ async def _invoke_non_context(
                 functype='asyncgen',
             )
         )
-        # XXX: the async-func may spawn further tasks which push
-        # back values like an async-generator would but must
-        # manualy construct the response dict-packet-responses as
-        # above
         with cancel_scope as cs:
             ctx._scope = cs
             task_status.started(ctx)
@@ -175,15 +181,13 @@ async def _invoke_non_context(
             await chan.send(
                 Stop(cid=cid)
             )
+
+    # simplest function/method request-response pattern
+    # XXX: in the most minimally used case, just a scheduled internal runtime
+    # call to `Actor._cancel_task()` from the ctx-peer task since we
+    # don't (yet) have a dedicated IPC msg.
+    # ------ - ------
     else:
-        # regular async function/method
-        # XXX: possibly just a scheduled `Actor._cancel_task()`
-        # from a remote request to cancel some `Context`.
-        # ------ - ------
-        # TODO: ideally we unify this with the above `context=True`
-        # block such that for any remote invocation ftype, we
-        # always invoke the far end RPC task scheduling the same
-        # way: using the linked IPC context machinery.
         failed_resp: bool = False
         try:
             ack = StartAck(
@@ -354,8 +358,15 @@ async def _errors_relayed_via_ipc(
             # channel.
             task_status.started(err)
 
-        # always reraise KBIs so they propagate at the sys-process level.
-        if isinstance(err, KeyboardInterrupt):
+        # always propagate KBIs at the sys-process level.
+        if (
+            isinstance(err, KeyboardInterrupt)
+
+            # ?TODO? except when running in asyncio mode?
+            # |_ wut if you want to open a `@context` FROM an
+            # infected_aio task?
+            # and not actor.is_infected_aio()
+        ):
             raise
 
     # RPC task bookeeping.
@@ -458,7 +469,6 @@ async def _invoke(
     # tb: TracebackType = None
 
     cancel_scope = CancelScope()
-    cs: CancelScope|None = None  # ref when activated
     ctx = actor.get_context(
         chan=chan,
         cid=cid,
@@ -607,8 +617,14 @@ async def _invoke(
         #     `@context` marked RPC function.
         # - `._portal` is never set.
         try:
+            tn: trio.Nursery
+            rpc_ctx_cs: CancelScope
             async with (
-                trio.open_nursery() as tn,
+                trio.open_nursery(
+                    strict_exception_groups=False,
+                    # ^XXX^ TODO? instead unpack any RAE as per "loose" style?
+
+                ) as tn,
                 msgops.maybe_limit_plds(
                     ctx=ctx,
                     spec=ctx_meta.get('pld_spec'),
@@ -616,7 +632,7 @@ async def _invoke(
                 ),
             ):
                 ctx._scope_nursery = tn
-                ctx._scope = tn.cancel_scope
+                rpc_ctx_cs = ctx._scope = tn.cancel_scope
                 task_status.started(ctx)
 
                 # TODO: better `trionics` tooling:
@@ -633,6 +649,10 @@ async def _invoke(
                 )
                 # set and shuttle final result to "parent"-side task.
                 ctx._result = res
+                log.runtime(
+                    f'Sending result msg and exiting {ctx.side!r}\n'
+                    f'{return_msg}\n'
+                )
                 await chan.send(return_msg)
 
             # NOTE: this happens IFF `ctx._scope.cancel()` is
@@ -642,7 +662,7 @@ async def _invoke(
             #   itself calls `ctx._maybe_cancel_and_set_remote_error()`
             #   which cancels the scope presuming the input error
             #   is not a `.cancel_acked` pleaser.
-            if ctx._scope.cancelled_caught:
+            if rpc_ctx_cs.cancelled_caught:
                 our_uid: tuple = actor.uid
 
                 # first check for and raise any remote error
@@ -652,9 +672,7 @@ async def _invoke(
                 if re := ctx._remote_error:
                     ctx._maybe_raise_remote_err(re)
 
-                cs: CancelScope = ctx._scope
-
-                if cs.cancel_called:
+                if rpc_ctx_cs.cancel_called:
                     canceller: tuple = ctx.canceller
                     explain: str = f'{ctx.side!r}-side task was cancelled by '
 
@@ -680,9 +698,15 @@ async def _invoke(
                     elif canceller == ctx.chan.uid:
                         explain += f'its {ctx.peer_side!r}-side peer'
 
-                    else:
+                    elif canceller == our_uid:
+                        explain += 'itself'
+
+                    elif canceller:
                         explain += 'a remote peer'
 
+                    else:
+                        explain += 'an unknown cause?'
+
                     explain += (
                         add_div(message=explain)
                         +
@@ -717,8 +741,8 @@ async def _invoke(
         # XXX: do we ever trigger this block any more?
         except (
             BaseExceptionGroup,
-            trio.Cancelled,
             BaseException,
+            trio.Cancelled,
 
         ) as scope_error:
             if (
@@ -755,7 +779,7 @@ async def _invoke(
 
             # don't pop the local context until we know the
             # associated child isn't in debug any more
-            await maybe_wait_for_debugger()
+            await _debug.maybe_wait_for_debugger()
             ctx: Context = actor._contexts.pop((
                 chan.uid,
                 cid,
@@ -831,8 +855,8 @@ async def try_ship_error_to_remote(
             log.critical(
                 'IPC transport failure -> '
                 f'failed to ship error to {remote_descr}!\n\n'
-                f'X=> {channel.uid}\n\n'
-
+                f'{type(msg)!r}[{msg.boxed_type_str}] X=> {channel.uid}\n'
+                f'\n'
                 # TODO: use `.msg.preetty_struct` for this!
                 f'{msg}\n'
             )
@@ -911,7 +935,10 @@ async def process_messages(
                     f'IPC msg from peer\n'
                     f'<= {chan.uid}\n\n'
 
-                    # TODO: avoid fmting depending on loglevel for perf?
+                    # TODO: use of the pprinting of structs is
+                    # FRAGILE and should prolly not be
+                    #
+                    # avoid fmting depending on loglevel for perf?
                     # -[ ] specifically `pretty_struct.pformat()` sub-call..?
                     #   - how to only log-level-aware actually call this?
                     # -[ ] use `.msg.pretty_struct` here now instead!
@@ -1177,7 +1204,7 @@ async def process_messages(
                 parent_chan=chan,
             )
 
-    except TransportClosed:
+    except TransportClosed as tc:
         # channels "breaking" (for TCP streams by EOF or 104
         # connection-reset) is ok since we don't have a teardown
         # handshake for them (yet) and instead we simply bail out of
@@ -1185,12 +1212,20 @@ async def process_messages(
         # up..
         #
         # TODO: maybe add a teardown handshake? and,
-        # -[ ] don't show this msg if it's an ephemeral discovery ep call?
+        # -[x] don't show this msg if it's an ephemeral discovery ep call?
+        #  |_ see the below `.report_n_maybe_raise()` impl as well as
+        #     tc-exc input details in `MsgpackTCPStream._iter_pkts()`
+        #     for different read-failure cases.
         # -[ ] figure out how this will break with other transports?
-        log.runtime(
-            f'IPC channel closed abruptly\n'
-            f'<=x peer: {chan.uid}\n'
-            f'   |_{chan.raddr}\n'
+        tc.report_n_maybe_raise(
+            message=(
+                f'peer IPC channel closed abruptly?\n\n'
+                f'<=x {chan}\n'
+                f'  |_{chan.raddr}\n\n'
+            )
+            +
+            tc.message
+
         )
 
         # transport **WAS** disconnected
@@ -1238,7 +1273,7 @@ async def process_messages(
                 'Exiting IPC msg loop with final msg\n\n'
                 f'<= peer: {chan.uid}\n'
                 f'  |_{chan}\n\n'
-                f'{pretty_struct.pformat(msg)}'
+                # f'{pretty_struct.pformat(msg)}'
             )
 
         log.runtime(message)

tractor/_runtime.py

@@ -45,6 +45,7 @@ from functools import partial
 from itertools import chain
 import importlib
 import importlib.util
+import os
 from pprint import pformat
 import signal
 import sys
@@ -55,10 +56,10 @@ from typing import (
 )
 import uuid
 from types import ModuleType
-import os
 import warnings
 
 import trio
+from trio._core import _run as trio_runtime
 from trio import (
     CancelScope,
     Nursery,
@@ -66,10 +67,11 @@ from trio import (
 )
 
 from tractor.msg import (
-    pretty_struct,
-    NamespacePath,
-    types as msgtypes,
     MsgType,
+    NamespacePath,
+    Stop,
+    pretty_struct,
+    types as msgtypes,
 )
 from ._ipc import Channel
 from ._context import (
@@ -79,16 +81,13 @@ from ._context import (
 from .log import get_logger
 from ._exceptions import (
     ContextCancelled,
+    InternalError,
     ModuleNotExposed,
     MsgTypeError,
     unpack_error,
     TransportClosed,
 )
-from .devx import (
-    # pause,
-    maybe_wait_for_debugger,
-    _debug,
-)
+from .devx import _debug
 from ._discovery import get_registry
 from ._portal import Portal
 from . import _state
@@ -101,12 +100,13 @@ from ._rpc import (
 
 if TYPE_CHECKING:
     from ._supervise import ActorNursery
+    from trio._channel import MemoryChannelState
 
 
 log = get_logger('tractor')
 
 
-def _get_mod_abspath(module: ModuleType) -> str:
+def _get_mod_abspath(module):
     return os.path.abspath(module.__file__)
 
 
@@ -114,25 +114,26 @@ class Actor:
     '''
     The fundamental "runtime" concurrency primitive.
 
-    An *actor* is the combination of a regular Python process executing
-    a ``trio`` task tree, communicating with other actors through
-    "memory boundary portals" - which provide a native async API around
-    IPC transport "channels" which themselves encapsulate various
-    (swappable) network protocols.
+    An "actor" is the combination of a regular Python process
+    executing a `trio.run()` task tree, communicating with other
+    "actors" through "memory boundary portals": `Portal`, which
+    provide a high-level async API around IPC "channels" (`Channel`)
+    which themselves encapsulate various (swappable) network
+    transport protocols for sending msgs between said memory domains
+    (processes, hosts, non-GIL threads).
 
-
-    Each "actor" is ``trio.run()`` scheduled "runtime" composed of
-    many concurrent tasks in a single thread. The "runtime" tasks
-    conduct a slew of low(er) level functions to make it possible
-    for message passing between actors as well as the ability to
-    create new actors (aka new "runtimes" in new processes which
-    are supervised via a nursery construct). Each task which sends
-    messages to a task in a "peer" (not necessarily a parent-child,
+    Each "actor" is `trio.run()` scheduled "runtime" composed of many
+    concurrent tasks in a single thread. The "runtime" tasks conduct
+    a slew of low(er) level functions to make it possible for message
+    passing between actors as well as the ability to create new
+    actors (aka new "runtimes" in new processes which are supervised
+    via an "actor-nursery" construct). Each task which sends messages
+    to a task in a "peer" actor (not necessarily a parent-child,
     depth hierarchy) is able to do so via an "address", which maps
     IPC connections across memory boundaries, and a task request id
-    which allows for per-actor tasks to send and receive messages
-    to specific peer-actor tasks with which there is an ongoing
-    RPC/IPC dialog.
+    which allows for per-actor tasks to send and receive messages to
+    specific peer-actor tasks with which there is an ongoing RPC/IPC
+    dialog.
 
     '''
     # ugh, we need to get rid of this and replace with a "registry" sys
@@ -229,17 +230,20 @@ class Actor:
         # by the user (currently called the "arbiter")
         self._spawn_method: str = spawn_method
 
-        self._peers: defaultdict = defaultdict(list)
+        self._peers: defaultdict[
+            str,  # uaid
+            list[Channel],  # IPC conns from peer
+        ] = defaultdict(list)
         self._peer_connected: dict[tuple[str, str], trio.Event] = {}
         self._no_more_peers = trio.Event()
         self._no_more_peers.set()
+
+        # RPC state
         self._ongoing_rpc_tasks = trio.Event()
         self._ongoing_rpc_tasks.set()
-
-        # (chan, cid) -> (cancel_scope, func)
         self._rpc_tasks: dict[
-            tuple[Channel, str],
-            tuple[Context, Callable, trio.Event]
+            tuple[Channel, str],  # (chan, cid)
+            tuple[Context, Callable, trio.Event]  # (ctx=>, fn(), done?)
         ] = {}
 
         # map {actor uids -> Context}
@@ -316,7 +320,10 @@ class Actor:
         event = self._peer_connected.setdefault(uid, trio.Event())
         await event.wait()
         log.debug(f'{uid!r} successfully connected back to us')
-        return event, self._peers[uid][-1]
+        return (
+            event,
+            self._peers[uid][-1],
+        )
 
     def load_modules(
         self,
@@ -407,26 +414,11 @@ class Actor:
         '''
         self._no_more_peers = trio.Event()  # unset by making new
         chan = Channel.from_stream(stream)
-        their_uid: tuple[str, str]|None = chan.uid
-
-        con_status: str = ''
-
-        # TODO: remove this branch since can never happen?
-        # NOTE: `.uid` is only set after first contact
-        if their_uid:
-            con_status = (
-                'IPC Re-connection from already known peer?\n'
-            )
-        else:
-            con_status = (
-                'New inbound IPC connection <=\n'
-            )
-
-        con_status += (
+        con_status: str = (
+            'New inbound IPC connection <=\n'
             f'|_{chan}\n'
-            # f' |_@{chan.raddr}\n\n'
-            # ^-TODO-^ remove since alfready in chan.__repr__()?
         )
+
         # send/receive initial handshake response
         try:
             uid: tuple|None = await self._do_handshake(chan)
@@ -438,10 +430,10 @@ class Actor:
 
             TransportClosed,
         ):
-            # XXX: This may propagate up from ``Channel._aiter_recv()``
-            # and ``MsgpackStream._inter_packets()`` on a read from the
+            # XXX: This may propagate up from `Channel._aiter_recv()`
+            # and `MsgpackStream._inter_packets()` on a read from the
             # stream particularly when the runtime is first starting up
-            # inside ``open_root_actor()`` where there is a check for
+            # inside `open_root_actor()` where there is a check for
             # a bound listener on the "arbiter" addr.  the reset will be
             # because the handshake was never meant took place.
             log.runtime(
@@ -451,9 +443,25 @@ class Actor:
             )
             return
 
+        familiar: str = 'new-peer'
+        if _pre_chan := self._peers.get(uid):
+            familiar: str = 'pre-existing-peer'
+        uid_short: str = f'{uid[0]}[{uid[1][-6:]}]'
         con_status += (
-            f' -> Handshake with actor `{uid[0]}[{uid[1][-6:]}]` complete\n'
+            f' -> Handshake with {familiar} `{uid_short}` complete\n'
         )
+
+        if _pre_chan:
+            # con_status += (
+            # ^TODO^ swap once we minimize conn duplication
+            # -[ ] last thing might be reg/unreg runtime reqs?
+            # log.warning(
+            log.debug(
+                f'?Wait?\n'
+                f'We already have IPC with peer {uid_short!r}\n'
+                f'|_{_pre_chan}\n'
+            )
+
         # IPC connection tracking for both peers and new children:
         # - if this is a new channel to a locally spawned
         #   sub-actor there will be a spawn wait even registered
@@ -506,8 +514,9 @@ class Actor:
             )
         except trio.Cancelled:
             log.cancel(
-                'IPC transport msg loop was cancelled for \n'
-                f'|_{chan}\n'
+                'IPC transport msg loop was cancelled\n'
+                f'c)>\n'
+                f' |_{chan}\n'
             )
             raise
 
@@ -544,8 +553,9 @@ class Actor:
 
             ):
                 log.cancel(
-                    'Waiting on cancel request to peer\n'
-                    f'`Portal.cancel_actor()` => {chan.uid}\n'
+                    'Waiting on cancel request to peer..\n'
+                    f'c)=>\n'
+                    f'  |_{chan.uid}\n'
                 )
 
                 # XXX: this is a soft wait on the channel (and its
@@ -642,12 +652,18 @@ class Actor:
                         # and
                         an_exit_cs.cancelled_caught
                     ):
-                        log.warning(
+                        report: str = (
                             'Timed out waiting on local actor-nursery to exit?\n'
-                            f'{local_nursery}\n'
-                            f' |_{pformat(local_nursery._children)}\n'
+                            f'c)>\n'
+                            f' |_{local_nursery}\n'
                         )
-                        # await _debug.pause()
+                        if children := local_nursery._children:
+                            # indent from above local-nurse repr
+                            report += (
+                                f'   |_{pformat(children)}\n'
+                            )
+
+                        log.warning(report)
 
                 if disconnected:
                     # if the transport died and this actor is still
@@ -758,7 +774,7 @@ class Actor:
                                     f'last disconnected child uid: {uid}\n'
                                     f'locking child uid: {pdb_user_uid}\n'
                                 )
-                                await maybe_wait_for_debugger(
+                                await _debug.maybe_wait_for_debugger(
                                     child_in_debug=True
                                 )
 
@@ -819,14 +835,19 @@ class Actor:
                 # side,
             )]
         except KeyError:
-            log.warning(
-                'Ignoring invalid IPC ctx msg!\n\n'
-                f'<= sender: {uid}\n\n'
-                # XXX don't need right since it's always in msg?
-                # f'=> cid: {cid}\n\n'
-
-                f'{pretty_struct.pformat(msg)}\n'
+            report: str = (
+                'Ignoring invalid IPC msg!?\n'
+                f'Ctx seems to not/no-longer exist??\n'
+                f'\n'
+                f'<=? {uid}\n'
+                f'  |_{pretty_struct.pformat(msg)}\n'
             )
+            match msg:
+                case Stop():
+                    log.runtime(report)
+                case _:
+                    log.warning(report)
+
             return
 
         # if isinstance(msg, MsgTypeError):
@@ -880,11 +901,15 @@ class Actor:
                 f'peer: {chan.uid}\n'
                 f'cid:{cid}\n'
             )
-            ctx._allow_overruns = allow_overruns
+            ctx._allow_overruns: bool = allow_overruns
 
             # adjust buffer size if specified
-            state = ctx._send_chan._state  # type: ignore
-            if msg_buffer_size and state.max_buffer_size != msg_buffer_size:
+            state: MemoryChannelState  = ctx._send_chan._state  # type: ignore
+            if (
+                msg_buffer_size
+                and
+                state.max_buffer_size != msg_buffer_size
+            ):
                 state.max_buffer_size = msg_buffer_size
 
         except KeyError:
@@ -1046,6 +1071,10 @@ class Actor:
                 # TODO: another `Struct` for rtvs..
                 rvs: dict[str, Any] = spawnspec._runtime_vars
                 if rvs['_debug_mode']:
+                    from .devx import (
+                        enable_stack_on_sig,
+                        maybe_init_greenback,
+                    )
                     try:
                         # TODO: maybe return some status msgs upward
                         # to that we can emit them in `con_status`
@@ -1053,14 +1082,57 @@ class Actor:
                         log.devx(
                             'Enabling `stackscope` traces on SIGUSR1'
                         )
-                        from .devx import enable_stack_on_sig
                         enable_stack_on_sig()
+
                     except ImportError:
                         log.warning(
                             '`stackscope` not installed for use in debug mode!'
                         )
 
-                rvs['_is_root'] = False
+                    if rvs.get('use_greenback', False):
+                        maybe_mod: ModuleType|None = await maybe_init_greenback()
+                        if maybe_mod:
+                            log.devx(
+                                'Activated `greenback` '
+                                'for `tractor.pause_from_sync()` support!'
+                            )
+                        else:
+                            rvs['use_greenback'] = False
+                            log.warning(
+                                '`greenback` not installed for use in debug mode!\n'
+                                '`tractor.pause_from_sync()` not available!'
+                            )
+
+                # XXX ensure the "infected `asyncio` mode" setting
+                # passed down from our spawning parent is consistent
+                # with `trio`-runtime initialization:
+                # - during sub-proc boot, the entrypoint func
+                #   (`._entry.<spawn_backend>_main()`) should set
+                #   `._infected_aio = True` before calling
+                #   `run_as_asyncio_guest()`,
+                # - the value of `infect_asyncio: bool = True` as
+                #   passed to `ActorNursery.start_actor()` must be
+                #   the same as `_runtime_vars['_is_infected_aio']`
+                if (
+                    (aio_rtv := rvs['_is_infected_aio'])
+                    !=
+                    (aio_attr := self._infected_aio)
+                ):
+                    raise InternalError(
+                        'Parent sent runtime-vars that mismatch for the '
+                        '"infected `asyncio` mode" settings ?!?\n\n'
+
+                        f'rvs["_is_infected_aio"] = {aio_rtv}\n'
+                        f'self._infected_aio = {aio_attr}\n'
+                    )
+                if aio_rtv:
+                    assert trio_runtime.GLOBAL_RUN_CONTEXT.runner.is_guest
+                    # ^TODO^ possibly add a `sniffio` or
+                    # `trio` pub-API for `is_guest_mode()`?
+
+                rvs['_is_root'] = False  # obvi XD
+
+                # update process-wide globals
                 _state._runtime_vars.update(rvs)
 
                 # XXX: ``msgspec`` doesn't support serializing tuples
@@ -1211,8 +1283,10 @@ class Actor:
         # TODO: just use the new `Context.repr_rpc: str` (and
         # other) repr fields instead of doing this all manual..
         msg: str = (
-            f'Runtime cancel request from {requester_type}:\n\n'
-            f'<= .cancel(): {requesting_uid}\n\n'
+            f'Actor-runtime cancel request from {requester_type}\n\n'
+            f'<=c) {requesting_uid}\n'
+            f'  |_{self}\n'
+            f'\n'
         )
 
         # TODO: what happens here when we self-cancel tho?
@@ -1232,13 +1306,15 @@ class Actor:
                 lock_req_ctx.has_outcome
             ):
                 msg += (
-                    '-> Cancelling active debugger request..\n'
+                    f'\n'
+                    f'-> Cancelling active debugger request..\n'
                     f'|_{_debug.Lock.repr()}\n\n'
                     f'|_{lock_req_ctx}\n\n'
                 )
                 # lock_req_ctx._scope.cancel()
                 # TODO: wrap this in a method-API..
                 debug_req.req_cs.cancel()
+                # if lock_req_ctx:
 
             # self-cancel **all** ongoing RPC tasks
             await self.cancel_rpc_tasks(
@@ -1320,10 +1396,11 @@ class Actor:
             return True
 
         log.cancel(
-            'Cancel request for RPC task\n\n'
-            f'<= Actor._cancel_task(): {requesting_uid}\n\n'
-            f'=> {ctx._task}\n'
-            f'  |_ >> {ctx.repr_rpc}\n'
+            'Rxed cancel request for RPC task\n'
+            f'<=c) {requesting_uid}\n'
+            f' |_{ctx._task}\n'
+            f'    >> {ctx.repr_rpc}\n'
+            # f'=> {ctx._task}\n'
             # f'  >> Actor._cancel_task() => {ctx._task}\n'
             # f'  |_ {ctx._task}\n\n'
 
@@ -1439,17 +1516,17 @@ class Actor:
             "IPC channel's "
         )
         rent_chan_repr: str = (
-            f'   |_{parent_chan}\n\n'
+            f' |_{parent_chan}\n\n'
             if parent_chan
             else ''
         )
         log.cancel(
             f'Cancelling {descr} RPC tasks\n\n'
-            f'<= canceller: {req_uid}\n'
+            f'<=c) {req_uid} [canceller]\n'
             f'{rent_chan_repr}'
-            f'=> cancellee: {self.uid}\n'
-            f'  |_{self}.cancel_rpc_tasks()\n'
-            f'  |_tasks: {len(tasks)}\n'
+            f'c)=> {self.uid} [cancellee]\n'
+            f'  |_{self} [with {len(tasks)} tasks]\n'
+            # f'  |_tasks: {len(tasks)}\n'
             # f'{tasks_str}'
         )
         for (
@@ -1504,8 +1581,8 @@ class Actor:
     @property
     def accept_addrs(self) -> list[tuple[str, int]]:
         '''
-        All addresses to which the IPC-transport-channel server
-        binds and listens for new connections.
+        All addresses to which the transport-channel server binds
+        and listens for new connections.
 
         '''
         # throws OSError on failure
@@ -1518,7 +1595,7 @@ class Actor:
     def accept_addr(self) -> tuple[str, int]:
         '''
         Primary address to which the IPC transport server is
-        bound.
+        bound and listening for new connections.
 
         '''
         # throws OSError on failure
@@ -1535,6 +1612,7 @@ class Actor:
     def get_chans(
         self,
         uid: tuple[str, str],
+
     ) -> list[Channel]:
         '''
         Return all IPC channels to the actor with provided `uid`.
@@ -1645,11 +1723,15 @@ async def async_main(
         # parent is kept alive as a resilient service until
         # cancellation steps have (mostly) occurred in
         # a deterministic way.
-        async with trio.open_nursery() as root_nursery:
+        async with trio.open_nursery(
+            strict_exception_groups=False,
+        ) as root_nursery:
             actor._root_n = root_nursery
             assert actor._root_n
 
-            async with trio.open_nursery() as service_nursery:
+            async with trio.open_nursery(
+                strict_exception_groups=False,
+            ) as service_nursery:
                 # This nursery is used to handle all inbound
                 # connections to us such that if the TCP server
                 # is killed, connections can continue to process
@@ -1695,11 +1777,12 @@ async def async_main(
                     # tranport address bind errors - normally it's
                     # something silly like the wrong socket-address
                     # passed via a config or CLI Bo
-                    entered_debug = await _debug._maybe_enter_pm(
-                        oserr,
-                    )
-                    if entered_debug:
+                    entered_debug: bool = await _debug._maybe_enter_pm(oserr)
+                    if not entered_debug:
+                        log.exception('Failed to init IPC channel server !?\n')
+                    else:
                         log.runtime('Exited debug REPL..')
+
                     raise
 
                 accept_addrs: list[tuple[str, int]] = actor.accept_addrs
@@ -1717,8 +1800,8 @@ async def async_main(
 
                 # Register with the arbiter if we're told its addr
                 log.runtime(
-                    f'Registering `{actor.name}` ->\n'
-                    f'{pformat(accept_addrs)}'
+                    f'Registering `{actor.name}` => {pformat(accept_addrs)}\n'
+                    # ^-TODO-^ we should instead show the maddr here^^
                 )
 
                 # TODO: ideally we don't fan out to all registrars
@@ -1776,57 +1859,90 @@ async def async_main(
 
         # Blocks here as expected until the root nursery is
         # killed (i.e. this actor is cancelled or signalled by the parent)
-    except Exception as err:
-        log.runtime("Closing all actor lifetime contexts")
-        actor.lifetime_stack.close()
-
+    except Exception as internal_err:
         if not is_registered:
+            err_report: str = (
+                '\n'
+                "Actor runtime (internally) failed BEFORE contacting the registry?\n"
+                f'registrars -> {actor.reg_addrs} ?!?!\n\n'
+
+                '^^^ THIS IS PROBABLY AN INTERNAL `tractor` BUG! ^^^\n\n'
+                '\t>> CALMLY CANCEL YOUR CHILDREN AND CALL YOUR PARENTS <<\n\n'
+
+                '\tIf this is a sub-actor hopefully its parent will keep running '
+                'and cancel/reap this sub-process..\n'
+                '(well, presuming this error was propagated upward)\n\n'
+
+                '\t---------------------------------------------\n'
+                '\tPLEASE REPORT THIS TRACEBACK IN A BUG REPORT @ '  # oneline
+                'https://github.com/goodboy/tractor/issues\n'
+                '\t---------------------------------------------\n'
+            )
+
             # TODO: I guess we could try to connect back
             # to the parent through a channel and engage a debugger
             # once we have that all working with std streams locking?
-            log.exception(
-                f"Actor errored and failed to register with arbiter "
-                f"@ {actor.reg_addrs[0]}?")
-            log.error(
-                "\n\n\t^^^ THIS IS PROBABLY AN INTERNAL `tractor` BUG! ^^^\n\n"
-                "\t>> CALMLY CALL THE AUTHORITIES AND HIDE YOUR CHILDREN <<\n\n"
-                "\tIf this is a sub-actor hopefully its parent will keep running "
-                "correctly presuming this error was safely ignored..\n\n"
-                "\tPLEASE REPORT THIS TRACEBACK IN A BUG REPORT: "
-                "https://github.com/goodboy/tractor/issues\n"
-            )
+            log.exception(err_report)
 
         if actor._parent_chan:
             await try_ship_error_to_remote(
                 actor._parent_chan,
-                err,
+                internal_err,
             )
 
         # always!
-        match err:
+        match internal_err:
             case ContextCancelled():
                 log.cancel(
                     f'Actor: {actor.uid} was task-context-cancelled with,\n'
-                    f'str(err)'
+                    f'str(internal_err)'
                 )
             case _:
-                log.exception("Actor errored:")
-        raise
+                log.exception(
+                    'Main actor-runtime task errored\n'
+                    f'<x)\n'
+                    f' |_{actor}\n'
+                )
+
+        raise internal_err
 
     finally:
-        log.runtime(
-            'Runtime nursery complete'
-            '-> Closing all actor lifetime contexts..'
+        teardown_report: str = (
+            'Main actor-runtime task completed\n'
         )
-        # tear down all lifetime contexts if not in guest mode
-        # XXX: should this just be in the entrypoint?
-        actor.lifetime_stack.close()
 
-        # TODO: we can't actually do this bc the debugger
-        # uses the _service_n to spawn the lock task, BUT,
-        # in theory if we had the root nursery surround this finally
-        # block it might be actually possible to debug THIS
-        # machinery in the same way as user task code?
+        # ?TODO? should this be in `._entry`/`._root` mods instead?
+        #
+        # teardown any actor-lifetime-bound contexts
+        ls: ExitStack = actor.lifetime_stack
+        # only report if there are any registered
+        cbs: list[Callable] = [
+            repr(tup[1].__wrapped__)
+            for tup in ls._exit_callbacks
+        ]
+        if cbs:
+            cbs_str: str = '\n'.join(cbs)
+            teardown_report += (
+                '-> Closing actor-lifetime-bound callbacks\n\n'
+                f'}}>\n'
+                f' |_{ls}\n'
+                f'   |_{cbs_str}\n'
+            )
+            # XXX NOTE XXX this will cause an error which
+            # prevents any `infected_aio` actor from continuing
+            # and any callbacks in the `ls` here WILL NOT be
+            # called!!
+            # await _debug.pause(shield=True)
+
+        ls.close()
+
+        # XXX TODO but hard XXX
+        # we can't actually do this bc the debugger uses the
+        # _service_n to spawn the lock task, BUT, in theory if we had
+        # the root nursery surround this finally block it might be
+        # actually possible to debug THIS machinery in the same way
+        # as user task code?
+        #
         # if actor.name == 'brokerd.ib':
         #     with CancelScope(shield=True):
         #         await _debug.breakpoint()
@@ -1856,9 +1972,9 @@ async def async_main(
                     failed = True
 
                 if failed:
-                    log.warning(
-                        f'Failed to unregister {actor.name} from '
-                        f'registar @ {addr}'
+                    teardown_report += (
+                        f'-> Failed to unregister {actor.name} from '
+                        f'registar @ {addr}\n'
                     )
 
         # Ensure all peers (actors connected to us as clients) are finished
@@ -1866,13 +1982,23 @@ async def async_main(
             if any(
                 chan.connected() for chan in chain(*actor._peers.values())
             ):
-                log.runtime(
-                    f"Waiting for remaining peers {actor._peers} to clear")
+                teardown_report += (
+                    f'-> Waiting for remaining peers {actor._peers} to clear..\n'
+                )
+                log.runtime(teardown_report)
                 with CancelScope(shield=True):
                     await actor._no_more_peers.wait()
-        log.runtime("All peer channels are complete")
 
-    log.runtime("Runtime completed")
+        teardown_report += (
+            '-> All peer channels are complete\n'
+        )
+
+    teardown_report += (
+        'Actor runtime exiting\n'
+        f'>)\n'
+        f'|_{actor}\n'
+    )
+    log.info(teardown_report)
 
 
 # TODO: rename to `Registry` and move to `._discovery`!

tractor/_shm.py

@@ -1,833 +0,0 @@
-# tractor: structured concurrent "actors".
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-SC friendly shared memory management geared at real-time
-processing.
-
-Support for ``numpy`` compatible array-buffers is provided but is
-considered optional within the context of this runtime-library.
-
-"""
-from __future__ import annotations
-from sys import byteorder
-import time
-from typing import Optional
-from multiprocessing import shared_memory as shm
-from multiprocessing.shared_memory import (
-    SharedMemory,
-    ShareableList,
-)
-
-from msgspec import Struct
-import tractor
-
-from .log import get_logger
-
-
-_USE_POSIX = getattr(shm, '_USE_POSIX', False)
-if _USE_POSIX:
-    from _posixshmem import shm_unlink
-
-
-try:
-    import numpy as np
-    from numpy.lib import recfunctions as rfn
-    # import nptyping
-except ImportError:
-    pass
-
-
-log = get_logger(__name__)
-
-
-def disable_mantracker():
-    '''
-    Disable all ``multiprocessing``` "resource tracking" machinery since
-    it's an absolute multi-threaded mess of non-SC madness.
-
-    '''
-    from multiprocessing import resource_tracker as mantracker
-
-    # Tell the "resource tracker" thing to fuck off.
-    class ManTracker(mantracker.ResourceTracker):
-        def register(self, name, rtype):
-            pass
-
-        def unregister(self, name, rtype):
-            pass
-
-        def ensure_running(self):
-            pass
-
-    # "know your land and know your prey"
-    # https://www.dailymotion.com/video/x6ozzco
-    mantracker._resource_tracker = ManTracker()
-    mantracker.register = mantracker._resource_tracker.register
-    mantracker.ensure_running = mantracker._resource_tracker.ensure_running
-    mantracker.unregister = mantracker._resource_tracker.unregister
-    mantracker.getfd = mantracker._resource_tracker.getfd
-
-
-disable_mantracker()
-
-
-class SharedInt:
-    '''
-    Wrapper around a single entry shared memory array which
-    holds an ``int`` value used as an index counter.
-
-    '''
-    def __init__(
-        self,
-        shm: SharedMemory,
-    ) -> None:
-        self._shm = shm
-
-    @property
-    def value(self) -> int:
-        return int.from_bytes(self._shm.buf, byteorder)
-
-    @value.setter
-    def value(self, value) -> None:
-        self._shm.buf[:] = value.to_bytes(self._shm.size, byteorder)
-
-    def destroy(self) -> None:
-        if _USE_POSIX:
-            # We manually unlink to bypass all the "resource tracker"
-            # nonsense meant for non-SC systems.
-            name = self._shm.name
-            try:
-                shm_unlink(name)
-            except FileNotFoundError:
-                # might be a teardown race here?
-                log.warning(f'Shm for {name} already unlinked?')
-
-
-class NDToken(Struct, frozen=True):
-    '''
-    Internal represenation of a shared memory ``numpy`` array "token"
-    which can be used to key and load a system (OS) wide shm entry
-    and correctly read the array by type signature.
-
-    This type is msg safe.
-
-    '''
-    shm_name: str  # this servers as a "key" value
-    shm_first_index_name: str
-    shm_last_index_name: str
-    dtype_descr: tuple
-    size: int  # in struct-array index / row terms
-
-    # TODO: use nptyping here on dtypes
-    @property
-    def dtype(self) -> list[tuple[str, str, tuple[int, ...]]]:
-        return np.dtype(
-            list(
-                map(tuple, self.dtype_descr)
-            )
-        ).descr
-
-    def as_msg(self):
-        return self.to_dict()
-
-    @classmethod
-    def from_msg(cls, msg: dict) -> NDToken:
-        if isinstance(msg, NDToken):
-            return msg
-
-        # TODO: native struct decoding
-        # return _token_dec.decode(msg)
-
-        msg['dtype_descr'] = tuple(map(tuple, msg['dtype_descr']))
-        return NDToken(**msg)
-
-
-# _token_dec = msgspec.msgpack.Decoder(NDToken)
-
-# TODO: this api?
-# _known_tokens = tractor.ActorVar('_shm_tokens', {})
-# _known_tokens = tractor.ContextStack('_known_tokens', )
-# _known_tokens = trio.RunVar('shms', {})
-
-# TODO: this should maybe be provided via
-# a `.trionics.maybe_open_context()` wrapper factory?
-# process-local store of keys to tokens
-_known_tokens: dict[str, NDToken] = {}
-
-
-def get_shm_token(key: str) -> NDToken | None:
-    '''
-    Convenience func to check if a token
-    for the provided key is known by this process.
-
-    Returns either the ``numpy`` token or a string for a shared list.
-
-    '''
-    return _known_tokens.get(key)
-
-
-def _make_token(
-    key: str,
-    size: int,
-    dtype: np.dtype,
-
-) -> NDToken:
-    '''
-    Create a serializable token that can be used
-    to access a shared array.
-
-    '''
-    return NDToken(
-        shm_name=key,
-        shm_first_index_name=key + "_first",
-        shm_last_index_name=key + "_last",
-        dtype_descr=tuple(np.dtype(dtype).descr),
-        size=size,
-    )
-
-
-class ShmArray:
-    '''
-    A shared memory ``numpy.ndarray`` API.
-
-    An underlying shared memory buffer is allocated based on
-    a user specified ``numpy.ndarray``. This fixed size array
-    can be read and written to by pushing data both onto the "front"
-    or "back" of a set index range. The indexes for the "first" and
-    "last" index are themselves stored in shared memory (accessed via
-    ``SharedInt`` interfaces) values such that multiple processes can
-    interact with the same array using a synchronized-index.
-
-    '''
-    def __init__(
-        self,
-        shmarr: np.ndarray,
-        first: SharedInt,
-        last: SharedInt,
-        shm: SharedMemory,
-        # readonly: bool = True,
-    ) -> None:
-        self._array = shmarr
-
-        # indexes for first and last indices corresponding
-        # to fille data
-        self._first = first
-        self._last = last
-
-        self._len = len(shmarr)
-        self._shm = shm
-        self._post_init: bool = False
-
-        # pushing data does not write the index (aka primary key)
-        self._write_fields: list[str] | None = None
-        dtype = shmarr.dtype
-        if dtype.fields:
-            self._write_fields = list(shmarr.dtype.fields.keys())[1:]
-
-    # TODO: ringbuf api?
-
-    @property
-    def _token(self) -> NDToken:
-        return NDToken(
-            shm_name=self._shm.name,
-            shm_first_index_name=self._first._shm.name,
-            shm_last_index_name=self._last._shm.name,
-            dtype_descr=tuple(self._array.dtype.descr),
-            size=self._len,
-        )
-
-    @property
-    def token(self) -> dict:
-        """Shared memory token that can be serialized and used by
-        another process to attach to this array.
-        """
-        return self._token.as_msg()
-
-    @property
-    def index(self) -> int:
-        return self._last.value % self._len
-
-    @property
-    def array(self) -> np.ndarray:
-        '''
-        Return an up-to-date ``np.ndarray`` view of the
-        so-far-written data to the underlying shm buffer.
-
-        '''
-        a = self._array[self._first.value:self._last.value]
-
-        # first, last = self._first.value, self._last.value
-        # a = self._array[first:last]
-
-        # TODO: eventually comment this once we've not seen it in the
-        # wild in a long time..
-        # XXX: race where first/last indexes cause a reader
-        # to load an empty array..
-        if len(a) == 0 and self._post_init:
-            raise RuntimeError('Empty array race condition hit!?')
-            # breakpoint()
-
-        return a
-
-    def ustruct(
-        self,
-        fields: Optional[list[str]] = None,
-
-        # type that all field values will be cast to
-        # in the returned view.
-        common_dtype: np.dtype = float,
-
-    ) -> np.ndarray:
-
-        array = self._array
-
-        if fields:
-            selection = array[fields]
-            # fcount = len(fields)
-        else:
-            selection = array
-            # fcount = len(array.dtype.fields)
-
-        # XXX: manual ``.view()`` attempt that also doesn't work.
-        # uview = selection.view(
-        #     dtype='<f16',
-        # ).reshape(-1, 4, order='A')
-
-        # assert len(selection) == len(uview)
-
-        u = rfn.structured_to_unstructured(
-            selection,
-            # dtype=float,
-            copy=True,
-        )
-
-        # unstruct = np.ndarray(u.shape, dtype=a.dtype, buffer=shm.buf)
-        # array[:] = a[:]
-        return u
-        # return ShmArray(
-        #     shmarr=u,
-        #     first=self._first,
-        #     last=self._last,
-        #     shm=self._shm
-        # )
-
-    def last(
-        self,
-        length: int = 1,
-
-    ) -> np.ndarray:
-        '''
-        Return the last ``length``'s worth of ("row") entries from the
-        array.
-
-        '''
-        return self.array[-length:]
-
-    def push(
-        self,
-        data: np.ndarray,
-
-        field_map: Optional[dict[str, str]] = None,
-        prepend: bool = False,
-        update_first: bool = True,
-        start: int | None = None,
-
-    ) -> int:
-        '''
-        Ring buffer like "push" to append data
-        into the buffer and return updated "last" index.
-
-        NB: no actual ring logic yet to give a "loop around" on overflow
-        condition, lel.
-
-        '''
-        length = len(data)
-
-        if prepend:
-            index = (start or self._first.value) - length
-
-            if index < 0:
-                raise ValueError(
-                    f'Array size of {self._len} was overrun during prepend.\n'
-                    f'You have passed {abs(index)} too many datums.'
-                )
-
-        else:
-            index = start if start is not None else self._last.value
-
-        end = index + length
-
-        if field_map:
-            src_names, dst_names = zip(*field_map.items())
-        else:
-            dst_names = src_names = self._write_fields
-
-        try:
-            self._array[
-                list(dst_names)
-            ][index:end] = data[list(src_names)][:]
-
-            # NOTE: there was a race here between updating
-            # the first and last indices and when the next reader
-            # tries to access ``.array`` (which due to the index
-            # overlap will be empty). Pretty sure we've fixed it now
-            # but leaving this here as a reminder.
-            if (
-                prepend
-                and update_first
-                and length
-            ):
-                assert index < self._first.value
-
-            if (
-                index < self._first.value
-                and update_first
-            ):
-                assert prepend, 'prepend=True not passed but index decreased?'
-                self._first.value = index
-
-            elif not prepend:
-                self._last.value = end
-
-            self._post_init = True
-            return end
-
-        except ValueError as err:
-            if field_map:
-                raise
-
-            # should raise if diff detected
-            self.diff_err_fields(data)
-            raise err
-
-    def diff_err_fields(
-        self,
-        data: np.ndarray,
-    ) -> None:
-        # reraise with any field discrepancy
-        our_fields, their_fields = (
-            set(self._array.dtype.fields),
-            set(data.dtype.fields),
-        )
-
-        only_in_ours = our_fields - their_fields
-        only_in_theirs = their_fields - our_fields
-
-        if only_in_ours:
-            raise TypeError(
-                f"Input array is missing field(s): {only_in_ours}"
-            )
-        elif only_in_theirs:
-            raise TypeError(
-                f"Input array has unknown field(s): {only_in_theirs}"
-            )
-
-    # TODO: support "silent" prepends that don't update ._first.value?
-    def prepend(
-        self,
-        data: np.ndarray,
-    ) -> int:
-        end = self.push(data, prepend=True)
-        assert end
-
-    def close(self) -> None:
-        self._first._shm.close()
-        self._last._shm.close()
-        self._shm.close()
-
-    def destroy(self) -> None:
-        if _USE_POSIX:
-            # We manually unlink to bypass all the "resource tracker"
-            # nonsense meant for non-SC systems.
-            shm_unlink(self._shm.name)
-
-        self._first.destroy()
-        self._last.destroy()
-
-    def flush(self) -> None:
-        # TODO: flush to storage backend like markestore?
-        ...
-
-
-def open_shm_ndarray(
-    size: int,
-    key: str | None = None,
-    dtype: np.dtype | None = None,
-    append_start_index: int | None = None,
-    readonly: bool = False,
-
-) -> ShmArray:
-    '''
-    Open a memory shared ``numpy`` using the standard library.
-
-    This call unlinks (aka permanently destroys) the buffer on teardown
-    and thus should be used from the parent-most accessor (process).
-
-    '''
-    # create new shared mem segment for which we
-    # have write permission
-    a = np.zeros(size, dtype=dtype)
-    a['index'] = np.arange(len(a))
-
-    shm = SharedMemory(
-        name=key,
-        create=True,
-        size=a.nbytes
-    )
-    array = np.ndarray(
-        a.shape,
-        dtype=a.dtype,
-        buffer=shm.buf
-    )
-    array[:] = a[:]
-    array.setflags(write=int(not readonly))
-
-    token = _make_token(
-        key=key,
-        size=size,
-        dtype=dtype,
-    )
-
-    # create single entry arrays for storing an first and last indices
-    first = SharedInt(
-        shm=SharedMemory(
-            name=token.shm_first_index_name,
-            create=True,
-            size=4,  # std int
-        )
-    )
-
-    last = SharedInt(
-        shm=SharedMemory(
-            name=token.shm_last_index_name,
-            create=True,
-            size=4,  # std int
-        )
-    )
-
-    # Start the "real-time" append-updated (or "pushed-to") section
-    # after some start index: ``append_start_index``. This allows appending
-    # from a start point in the array which isn't the 0 index and looks
-    # something like,
-    # -------------------------
-    # |              |        i
-    # _________________________
-    # <-------------> <------->
-    #  history         real-time
-    #
-    # Once fully "prepended", the history section will leave the
-    # ``ShmArray._start.value: int = 0`` and the yet-to-be written
-    # real-time section will start at ``ShmArray.index: int``.
-
-    # this sets the index to nearly 2/3rds into the the length of
-    # the buffer leaving at least a "days worth of second samples"
-    # for the real-time section.
-    if append_start_index is None:
-        append_start_index = round(size * 0.616)
-
-    last.value = first.value = append_start_index
-
-    shmarr = ShmArray(
-        array,
-        first,
-        last,
-        shm,
-    )
-
-    assert shmarr._token == token
-    _known_tokens[key] = shmarr.token
-
-    # "unlink" created shm on process teardown by
-    # pushing teardown calls onto actor context stack
-    stack = tractor.current_actor().lifetime_stack
-    stack.callback(shmarr.close)
-    stack.callback(shmarr.destroy)
-
-    return shmarr
-
-
-def attach_shm_ndarray(
-    token: tuple[str, str, tuple[str, str]],
-    readonly: bool = True,
-
-) -> ShmArray:
-    '''
-    Attach to an existing shared memory array previously
-    created by another process using ``open_shared_array``.
-
-    No new shared mem is allocated but wrapper types for read/write
-    access are constructed.
-
-    '''
-    token = NDToken.from_msg(token)
-    key = token.shm_name
-
-    if key in _known_tokens:
-        assert NDToken.from_msg(_known_tokens[key]) == token, "WTF"
-
-    # XXX: ugh, looks like due to the ``shm_open()`` C api we can't
-    # actually place files in a subdir, see discussion here:
-    # https://stackoverflow.com/a/11103289
-
-    # attach to array buffer and view as per dtype
-    _err: Optional[Exception] = None
-    for _ in range(3):
-        try:
-            shm = SharedMemory(
-                name=key,
-                create=False,
-            )
-            break
-        except OSError as oserr:
-            _err = oserr
-            time.sleep(0.1)
-    else:
-        if _err:
-            raise _err
-
-    shmarr = np.ndarray(
-        (token.size,),
-        dtype=token.dtype,
-        buffer=shm.buf
-    )
-    shmarr.setflags(write=int(not readonly))
-
-    first = SharedInt(
-        shm=SharedMemory(
-            name=token.shm_first_index_name,
-            create=False,
-            size=4,  # std int
-        ),
-    )
-    last = SharedInt(
-        shm=SharedMemory(
-            name=token.shm_last_index_name,
-            create=False,
-            size=4,  # std int
-        ),
-    )
-
-    # make sure we can read
-    first.value
-
-    sha = ShmArray(
-        shmarr,
-        first,
-        last,
-        shm,
-    )
-    # read test
-    sha.array
-
-    # Stash key -> token knowledge for future queries
-    # via `maybe_opepn_shm_array()` but only after we know
-    # we can attach.
-    if key not in _known_tokens:
-        _known_tokens[key] = token
-
-    # "close" attached shm on actor teardown
-    tractor.current_actor().lifetime_stack.callback(sha.close)
-
-    return sha
-
-
-def maybe_open_shm_ndarray(
-    key: str,  # unique identifier for segment
-    size: int,
-    dtype: np.dtype | None = None,
-    append_start_index: int = 0,
-    readonly: bool = True,
-
-) -> tuple[ShmArray, bool]:
-    '''
-    Attempt to attach to a shared memory block using a "key" lookup
-    to registered blocks in the users overall "system" registry
-    (presumes you don't have the block's explicit token).
-
-    This function is meant to solve the problem of discovering whether
-    a shared array token has been allocated or discovered by the actor
-    running in **this** process. Systems where multiple actors may seek
-    to access a common block can use this function to attempt to acquire
-    a token as discovered by the actors who have previously stored
-    a "key" -> ``NDToken`` map in an actor local (aka python global)
-    variable.
-
-    If you know the explicit ``NDToken`` for your memory segment instead
-    use ``attach_shm_array``.
-
-    '''
-    try:
-        # see if we already know this key
-        token = _known_tokens[key]
-        return (
-            attach_shm_ndarray(
-                token=token,
-                readonly=readonly,
-            ),
-            False,  # not newly opened
-        )
-    except KeyError:
-        log.warning(f"Could not find {key} in shms cache")
-        if dtype:
-            token = _make_token(
-                key,
-                size=size,
-                dtype=dtype,
-            )
-        else:
-
-            try:
-                return (
-                    attach_shm_ndarray(
-                        token=token,
-                        readonly=readonly,
-                    ),
-                    False,
-                )
-            except FileNotFoundError:
-                log.warning(f"Could not attach to shm with token {token}")
-
-        # This actor does not know about memory
-        # associated with the provided "key".
-        # Attempt to open a block and expect
-        # to fail if a block has been allocated
-        # on the OS by someone else.
-        return (
-            open_shm_ndarray(
-                key=key,
-                size=size,
-                dtype=dtype,
-                append_start_index=append_start_index,
-                readonly=readonly,
-            ),
-            True,
-        )
-
-
-class ShmList(ShareableList):
-    '''
-    Carbon copy of ``.shared_memory.ShareableList`` with a few
-    enhancements:
-
-    - readonly mode via instance var flag  `._readonly: bool`
-    - ``.__getitem__()`` accepts ``slice`` inputs
-    - exposes the underlying buffer "name" as a ``.key: str``
-
-    '''
-    def __init__(
-        self,
-        sequence: list | None = None,
-        *,
-        name: str | None = None,
-        readonly: bool = True
-
-    ) -> None:
-        self._readonly = readonly
-        self._key = name
-        return super().__init__(
-            sequence=sequence,
-            name=name,
-        )
-
-    @property
-    def key(self) -> str:
-        return self._key
-
-    @property
-    def readonly(self) -> bool:
-        return self._readonly
-
-    def __setitem__(
-        self,
-        position,
-        value,
-
-    ) -> None:
-
-        # mimick ``numpy`` error
-        if self._readonly:
-            raise ValueError('assignment destination is read-only')
-
-        return super().__setitem__(position, value)
-
-    def __getitem__(
-        self,
-        indexish,
-    ) -> list:
-
-        # NOTE: this is a non-writeable view (copy?) of the buffer
-        # in a new list instance.
-        if isinstance(indexish, slice):
-            return list(self)[indexish]
-
-        return super().__getitem__(indexish)
-
-    # TODO: should we offer a `.array` and `.push()` equivalent
-    # to the `ShmArray`?
-    # currently we have the following limitations:
-    # - can't write slices of input using traditional slice-assign
-    #   syntax due to the ``ShareableList.__setitem__()`` implementation.
-    # - ``list(shmlist)`` returns a non-mutable copy instead of
-    #   a writeable view which would be handier numpy-style ops.
-
-
-def open_shm_list(
-    key: str,
-    sequence: list | None = None,
-    size: int = int(2 ** 10),
-    dtype: float | int | bool | str | bytes | None = float,
-    readonly: bool = True,
-
-) -> ShmList:
-
-    if sequence is None:
-        default = {
-            float: 0.,
-            int: 0,
-            bool: True,
-            str: 'doggy',
-            None: None,
-        }[dtype]
-        sequence = [default] * size
-
-    shml = ShmList(
-        sequence=sequence,
-        name=key,
-        readonly=readonly,
-    )
-
-    # "close" attached shm on actor teardown
-    try:
-        actor = tractor.current_actor()
-        actor.lifetime_stack.callback(shml.shm.close)
-        actor.lifetime_stack.callback(shml.shm.unlink)
-    except RuntimeError:
-        log.warning('tractor runtime not active, skipping teardown steps')
-
-    return shml
-
-
-def attach_shm_list(
-    key: str,
-    readonly: bool = False,
-
-) -> ShmList:
-
-    return ShmList(
-        name=key,
-        readonly=readonly,
-    )

tractor/_spawn.py

@@ -34,7 +34,7 @@ from typing import (
 import trio
 from trio import TaskStatus
 
-from tractor.devx import (
+from .devx._debug import (
     maybe_wait_for_debugger,
     acquire_debug_lock,
 )
@@ -149,7 +149,7 @@ async def exhaust_portal(
 
         # XXX: streams should never be reaped here since they should
         # always be established and shutdown using a context manager api
-        final: Any = await portal.result()
+        final: Any = await portal.wait_for_result()
 
     except (
         Exception,
@@ -223,7 +223,11 @@ async def cancel_on_completion(
 
 async def hard_kill(
     proc: trio.Process,
+
     terminate_after: int = 1.6,
+    # NOTE: for mucking with `.pause()`-ing inside the runtime
+    # whilst also hacking on it XD
+    # terminate_after: int = 99999,
 
     # NOTE: for mucking with `.pause()`-ing inside the runtime
     # whilst also hacking on it XD
@@ -246,8 +250,9 @@ async def hard_kill(
 
     '''
     log.cancel(
-        'Terminating sub-proc:\n'
-        f'|_{proc}\n'
+        'Terminating sub-proc\n'
+        f'>x)\n'
+        f' |_{proc}\n'
     )
     # NOTE: this timeout used to do nothing since we were shielding
     # the ``.wait()`` inside ``new_proc()`` which will pretty much
@@ -293,8 +298,8 @@ async def hard_kill(
         log.critical(
             # 'Well, the #ZOMBIE_LORD_IS_HERE# to collect\n'
             '#T-800 deployed to collect zombie B0\n'
-            f'|\n'
-            f'|_{proc}\n'
+            f'>x)\n'
+            f' |_{proc}\n'
         )
         proc.kill()
 
@@ -322,8 +327,10 @@ async def soft_kill(
     uid: tuple[str, str] = portal.channel.uid
     try:
         log.cancel(
-            'Soft killing sub-actor via `Portal.cancel_actor()`\n'
-            f'|_{proc}\n'
+            f'Soft killing sub-actor via portal request\n'
+            f'\n'
+            f'(c=> {portal.chan.uid}\n'
+            f'  |_{proc}\n'
         )
         # wait on sub-proc to signal termination
         await wait_func(proc)
@@ -552,8 +559,9 @@ async def trio_proc(
             # cancel result waiter that may have been spawned in
             # tandem if not done already
             log.cancel(
-                'Cancelling existing result waiter task for '
-                f'{subactor.uid}'
+                'Cancelling portal result reaper task\n'
+                f'>c)\n'
+                f' |_{subactor.uid}\n'
             )
             nursery.cancel_scope.cancel()
 
@@ -562,7 +570,11 @@ async def trio_proc(
         # allowed! Do this **after** cancellation/teardown to avoid
         # killing the process too early.
         if proc:
-            log.cancel(f'Hard reap sequence starting for {subactor.uid}')
+            log.cancel(
+                f'Hard reap sequence starting for subactor\n'
+                f'>x)\n'
+                f' |_{subactor}@{subactor.uid}\n'
+            )
 
             with trio.CancelScope(shield=True):
                 # don't clobber an ongoing pdb

tractor/_state.py

@@ -44,7 +44,9 @@ _runtime_vars: dict[str, Any] = {
     '_root_mailbox': (None, None),
     '_registry_addrs': [],
 
-    # for `breakpoint()` support
+    '_is_infected_aio': False,
+
+    # for `tractor.pause_from_sync()` & `breakpoint()` support
     'use_greenback': False,
 }
 
@@ -70,7 +72,8 @@ def current_actor(
     '''
     if (
         err_on_no_runtime
-        and _current_actor is None
+        and
+        _current_actor is None
     ):
         msg: str = 'No local actor has been initialized yet?\n'
         from ._exceptions import NoRuntime
@@ -105,6 +108,7 @@ def is_main_process() -> bool:
     return mp.current_process().name == 'MainProcess'
 
 
+# TODO, more verby name?
 def debug_mode() -> bool:
     '''
     Bool determining if "debug mode" is on which enables

tractor/_streaming.py

@@ -36,8 +36,8 @@ import warnings
 import trio
 
 from ._exceptions import (
-    # _raise_from_no_key_in_msg,
     ContextCancelled,
+    RemoteActorError,
 )
 from .log import get_logger
 from .trionics import (
@@ -45,9 +45,11 @@ from .trionics import (
     BroadcastReceiver,
 )
 from tractor.msg import (
-    # Return,
-    # Stop,
+    Error,
+    Return,
+    Stop,
     MsgType,
+    PayloadT,
     Yield,
 )
 
@@ -70,8 +72,7 @@ class MsgStream(trio.abc.Channel):
     A bidirectional message stream for receiving logically sequenced
     values over an inter-actor IPC `Channel`.
 
-    This is the type returned to a local task which entered either
-    `Portal.open_stream_from()` or `Context.open_stream()`.
+
 
     Termination rules:
 
@@ -94,6 +95,9 @@ class MsgStream(trio.abc.Channel):
         self._rx_chan = rx_chan
         self._broadcaster = _broadcaster
 
+        # any actual IPC msg which is effectively an `EndOfStream`
+        self._stop_msg: bool|Stop = False
+
         # flag to denote end of stream
         self._eoc: bool|trio.EndOfChannel = False
         self._closed: bool|trio.ClosedResourceError = False
@@ -101,7 +105,7 @@ class MsgStream(trio.abc.Channel):
     @property
     def ctx(self) -> Context:
         '''
-        This stream's IPC `Context` ref.
+        A read-only ref to this stream's inter-actor-task `Context`.
 
         '''
         return self._ctx
@@ -125,16 +129,67 @@ class MsgStream(trio.abc.Channel):
     def receive_nowait(
         self,
         expect_msg: MsgType = Yield,
-    ):
+    ) -> PayloadT:
         ctx: Context = self._ctx
-        return ctx._pld_rx.recv_pld_nowait(
+        (
+            msg,
+            pld,
+        ) = ctx._pld_rx.recv_msg_nowait(
             ipc=self,
             expect_msg=expect_msg,
         )
 
+        # ?TODO, maybe factor this into a hyper-common `unwrap_pld()`
+        #
+        match msg:
+
+            # XXX, these never seems to ever hit? cool?
+            case Stop():
+                log.cancel(
+                    f'Msg-stream was ended via stop msg\n'
+                    f'{msg}'
+                )
+            case Error():
+                log.error(
+                    f'Msg-stream was ended via error msg\n'
+                    f'{msg}'
+                )
+
+            # XXX NOTE, always set any final result on the ctx to
+            # avoid teardown race conditions where previously this msg
+            # would be consumed silently (by `.aclose()` doing its
+            # own "msg drain loop" but WITHOUT those `drained: lists[MsgType]`
+            # being post-close-processed!
+            #
+            # !!TODO, see the equiv todo-comment in `.receive()`
+            # around the `if drained:` where we should prolly
+            # ACTUALLY be doing this post-close processing??
+            #
+            case Return(pld=pld):
+                log.warning(
+                    f'Msg-stream final result msg for IPC ctx?\n'
+                    f'{msg}'
+                )
+                # XXX TODO, this **should be covered** by higher
+                # scoped runtime-side method calls such as
+                # `Context._deliver_msg()`, so you should never
+                # really see the warning above or else something
+                # racy/out-of-order is likely going on between
+                # actor-runtime-side push tasks and the user-app-side
+                # consume tasks!
+                # -[ ] figure out that set of race cases and fix!
+                # -[ ] possibly return the `msg` given an input
+                #     arg-flag is set so we can process the `Return`
+                #     from the `.aclose()` caller?
+                #
+                # breakpoint()  # to debug this RACE CASE!
+                ctx._result = pld
+                ctx._outcome_msg = msg
+
+        return pld
+
     async def receive(
         self,
-
         hide_tb: bool = False,
     ):
         '''
@@ -145,9 +200,8 @@ class MsgStream(trio.abc.Channel):
         '''
         __tracebackhide__: bool = hide_tb
 
-        # NOTE: `trio.ReceiveChannel` implements
-        # EOC handling as follows (aka uses it
-        # to gracefully exit async for loops):
+        # NOTE FYI: `trio.ReceiveChannel` implements EOC handling as
+        # follows (aka uses it to gracefully exit async for loops):
         #
         # async def __anext__(self) -> ReceiveType:
         #     try:
@@ -155,7 +209,7 @@ class MsgStream(trio.abc.Channel):
         #     except trio.EndOfChannel:
         #         raise StopAsyncIteration
         #
-        # see ``.aclose()`` for notes on the old behaviour prior to
+        # see `.aclose()` for notes on the old behaviour prior to
         # introducing this
         if self._eoc:
             raise self._eoc
@@ -165,48 +219,33 @@ class MsgStream(trio.abc.Channel):
 
         src_err: Exception|None = None  # orig tb
         try:
-
             ctx: Context = self._ctx
-            return await ctx._pld_rx.recv_pld(ipc=self)
+            pld = await ctx._pld_rx.recv_pld(
+                ipc=self,
+                expect_msg=Yield,
+            )
+            return pld
 
         # XXX: the stream terminates on either of:
-        # - via `self._rx_chan.receive()` raising  after manual closure
-        #   by the rpc-runtime OR,
-        # - via a received `{'stop': ...}` msg from remote side.
-        #   |_ NOTE: previously this was triggered by calling
-        #   ``._rx_chan.aclose()`` on the send side of the channel inside
-        #   `Actor._deliver_ctx_payload()`, but now the 'stop' message handling
-        #   has been put just above inside `_raise_from_no_key_in_msg()`.
-        except (
-            trio.EndOfChannel,
-        ) as eoc:
-            src_err = eoc
+        # - `self._rx_chan.receive()` raising  after manual closure
+        #   by the rpc-runtime,
+        #   OR
+        # - via a `Stop`-msg received from remote peer task.
+        #   NOTE
+        #   |_ previously this was triggered by calling
+        #   `._rx_chan.aclose()` on the send side of the channel
+        #   inside `Actor._deliver_ctx_payload()`, but now the 'stop'
+        #   message handling gets delegated to `PldRFx.recv_pld()`
+        #   internals.
+        except trio.EndOfChannel as eoc:
+            # a graceful stream finished signal
             self._eoc = eoc
+            src_err = eoc
 
-            # TODO: Locally, we want to close this stream gracefully, by
-            # terminating any local consumers tasks deterministically.
-            # Once we have broadcast support, we **don't** want to be
-            # closing this stream and not flushing a final value to
-            # remaining (clone) consumers who may not have been
-            # scheduled to receive it yet.
-            # try:
-            #     maybe_err_msg_or_res: dict = self._rx_chan.receive_nowait()
-            #     if maybe_err_msg_or_res:
-            #         log.warning(
-            #             'Discarding un-processed msg:\n'
-            #             f'{maybe_err_msg_or_res}'
-            #         )
-            # except trio.WouldBlock:
-            #     # no queued msgs that might be another remote
-            #     # error, so just raise the original EoC
-            #     pass
-
-            # raise eoc
-
-        # a ``ClosedResourceError`` indicates that the internal
-        # feeder memory receive channel was closed likely by the
-        # runtime after the associated transport-channel
-        # disconnected or broke.
+        # a `ClosedResourceError` indicates that the internal feeder
+        # memory receive channel was closed likely by the runtime
+        # after the associated transport-channel disconnected or
+        # broke.
         except trio.ClosedResourceError as cre:  # by self._rx_chan.receive()
             src_err = cre
             log.warning(
@@ -218,14 +257,18 @@ class MsgStream(trio.abc.Channel):
         # terminated and signal this local iterator to stop
         drained: list[Exception|dict] = await self.aclose()
         if drained:
-            # from .devx import pause
-            # await pause()
+        #  ^^^^^^^^TODO? pass these to the `._ctx._drained_msgs:
+        #  deque` and then iterate them as part of any
+        #  `.wait_for_result()` call?
+        #
+        # -[ ] move the match-case processing from
+        #     `.receive_nowait()` instead to right here, use it from
+        #     a for msg in drained:` post-proc loop?
+        #
             log.warning(
-                'Drained context msgs during closure:\n'
+                'Drained context msgs during closure\n\n'
                 f'{drained}'
             )
-        # TODO: pass these to the `._ctx._drained_msgs: deque`
-        # and then iterate them as part of any `.result()` call?
 
         # NOTE XXX: if the context was cancelled or remote-errored
         # but we received the stream close msg first, we
@@ -238,28 +281,36 @@ class MsgStream(trio.abc.Channel):
             from_src_exc=src_err,
         )
 
-        # propagate any error but hide low-level frame details
-        # from the caller by default for debug noise reduction.
+        # propagate any error but hide low-level frame details from
+        # the caller by default for console/debug-REPL noise
+        # reduction.
         if (
             hide_tb
+            and (
 
-            # XXX NOTE XXX don't reraise on certain
-            # stream-specific internal error types like,
-            #
-            # - `trio.EoC` since we want to use the exact instance
-            #   to ensure that it is the error that bubbles upward
-            #   for silent absorption by `Context.open_stream()`.
-            and not self._eoc
+                # XXX NOTE special conditions: don't reraise on
+                # certain stream-specific internal error types like,
+                #
+                # - `trio.EoC` since we want to use the exact instance
+                #   to ensure that it is the error that bubbles upward
+                #   for silent absorption by `Context.open_stream()`.
+                not self._eoc
 
-            # - `RemoteActorError` (or `ContextCancelled`) if it gets
-            #   raised from `_raise_from_no_key_in_msg()` since we
-            #   want the same (as the above bullet) for any
-            #   `.open_context()` block bubbled error raised by
-            #   any nearby ctx API remote-failures.
-            # and not isinstance(src_err, RemoteActorError)
+                # - `RemoteActorError` (or subtypes like ctxc)
+                #    since we want to present the error as though it is
+                #    "sourced" directly from this `.receive()` call and
+                #    generally NOT include the stack frames raised from
+                #    inside the `PldRx` and/or the transport stack
+                #    layers.
+                or isinstance(src_err, RemoteActorError)
+            )
         ):
             raise type(src_err)(*src_err.args) from src_err
         else:
+            # for any non-graceful-EOC we want to NOT hide this frame
+            if not self._eoc:
+                __tracebackhide__: bool = False
+
             raise src_err
 
     async def aclose(self) -> list[Exception|dict]:
@@ -276,9 +327,6 @@ class MsgStream(trio.abc.Channel):
          - more or less we try to maintain adherance to trio's `.aclose()` semantics:
            https://trio.readthedocs.io/en/stable/reference-io.html#trio.abc.AsyncResource.aclose
         '''
-
-        # rx_chan = self._rx_chan
-
         # XXX NOTE XXX
         # it's SUPER IMPORTANT that we ensure we don't DOUBLE
         # DRAIN msgs on closure so avoid getting stuck handing on
@@ -290,15 +338,16 @@ class MsgStream(trio.abc.Channel):
             # this stream has already been closed so silently succeed as
             # per ``trio.AsyncResource`` semantics.
             # https://trio.readthedocs.io/en/stable/reference-io.html#trio.abc.AsyncResource.aclose
+            # import tractor
+            # await tractor.pause()
             return []
 
         ctx: Context = self._ctx
         drained: list[Exception|dict] = []
         while not drained:
             try:
-                maybe_final_msg = self.receive_nowait(
-                    # allow_msgs=[Yield, Return],
-                    expect_msg=Yield,
+                maybe_final_msg: Yield|Return = self.receive_nowait(
+                    expect_msg=Yield|Return,
                 )
                 if maybe_final_msg:
                     log.debug(
@@ -383,16 +432,30 @@ class MsgStream(trio.abc.Channel):
         #         await rx_chan.aclose()
 
         if not self._eoc:
+            this_side: str = self._ctx.side
+            peer_side: str = self._ctx.peer_side
             message: str = (
-                f'Stream self-closed by {self._ctx.side!r}-side before EoC\n'
-                f'|_{self}\n'
+                f'Stream self-closed by {this_side!r}-side before EoC from {peer_side!r}\n'
+                # } bc a stream is a "scope"/msging-phase inside an IPC
+                f'x}}>\n'
+                f'  |_{self}\n'
             )
             log.cancel(message)
             self._eoc = trio.EndOfChannel(message)
 
+            if (
+                (rx_chan := self._rx_chan)
+                and
+                (stats := rx_chan.statistics()).tasks_waiting_receive
+            ):
+                log.cancel(
+                    f'Msg-stream is closing but there is still reader tasks,\n'
+                    f'{stats}\n'
+                )
+
         # ?XXX WAIT, why do we not close the local mem chan `._rx_chan` XXX?
         # => NO, DEFINITELY NOT! <=
-        # if we're a bi-dir ``MsgStream`` BECAUSE this same
+        # if we're a bi-dir `MsgStream` BECAUSE this same
         # core-msg-loop mem recv-chan is used to deliver the
         # potential final result from the surrounding inter-actor
         # `Context` so we don't want to close it until that

tractor/_supervise.py

@@ -80,6 +80,7 @@ class ActorNursery:
     '''
     def __init__(
         self,
+        # TODO: maybe def these as fields of a struct looking type?
         actor: Actor,
         ria_nursery: trio.Nursery,
         da_nursery: trio.Nursery,
@@ -88,8 +89,10 @@ class ActorNursery:
     ) -> None:
         # self.supervisor = supervisor  # TODO
         self._actor: Actor = actor
-        self._ria_nursery = ria_nursery
+
+        # TODO: rename to `._tn` for our conventional "task-nursery"
         self._da_nursery = da_nursery
+
         self._children: dict[
             tuple[str, str],
             tuple[
@@ -98,15 +101,13 @@ class ActorNursery:
                 Portal | None,
             ]
         ] = {}
-        # portals spawned with ``run_in_actor()`` are
-        # cancelled when their "main" result arrives
-        self._cancel_after_result_on_exit: set = set()
+
         self.cancelled: bool = False
         self._join_procs = trio.Event()
         self._at_least_one_child_in_debug: bool = False
         self.errors = errors
-        self.exited = trio.Event()
         self._scope_error: BaseException|None = None
+        self.exited = trio.Event()
 
         # NOTE: when no explicit call is made to
         # `.open_root_actor()` by application code,
@@ -116,6 +117,13 @@ class ActorNursery:
         # and syncing purposes to any actor opened nurseries.
         self._implicit_runtime_started: bool = False
 
+        # TODO: remove the `.run_in_actor()` API and thus this 2ndary
+        # nursery when that API get's moved outside this primitive!
+        self._ria_nursery = ria_nursery
+        # portals spawned with ``run_in_actor()`` are
+        # cancelled when their "main" result arrives
+        self._cancel_after_result_on_exit: set = set()
+
     async def start_actor(
         self,
         name: str,
@@ -126,10 +134,14 @@ class ActorNursery:
         rpc_module_paths: list[str]|None = None,
         enable_modules: list[str]|None = None,
         loglevel: str|None = None,  # set log level per subactor
-        nursery: trio.Nursery|None = None,
         debug_mode: bool|None = None,
         infect_asyncio: bool = False,
 
+        # TODO: ideally we can rm this once we no longer have
+        # a `._ria_nursery` since the dependent APIs have been
+        # removed!
+        nursery: trio.Nursery|None = None,
+
     ) -> Portal:
         '''
         Start a (daemon) actor: an process that has no designated
@@ -146,6 +158,7 @@ class ActorNursery:
         # configure and pass runtime state
         _rtv = _state._runtime_vars.copy()
         _rtv['_is_root'] = False
+        _rtv['_is_infected_aio'] = infect_asyncio
 
         # allow setting debug policy per actor
         if debug_mode is not None:
@@ -200,6 +213,7 @@ class ActorNursery:
     #  |_ dynamic @context decoration on child side
     #  |_ implicit `Portal.open_context() as (ctx, first):`
     #    and `return first` on parent side.
+    #  |_ mention how it's similar to `trio-parallel` API?
     # -[ ] use @api_frame on the wrapper
     async def run_in_actor(
         self,
@@ -269,11 +283,14 @@ class ActorNursery:
 
     ) -> None:
         '''
-        Cancel this nursery by instructing each subactor to cancel
-        itself and wait for all subactors to terminate.
+        Cancel this actor-nursery by instructing each subactor's
+        runtime to cancel and wait for all underlying sub-processes
+        to terminate.
 
-        If ``hard_killl`` is set to ``True`` then kill the processes
-        directly without any far end graceful ``trio`` cancellation.
+        If `hard_kill` is set then kill the processes directly using
+        the spawning-backend's API/OS-machinery without any attempt
+        at (graceful) `trio`-style cancellation using our
+        `Actor.cancel()`.
 
         '''
         __runtimeframe__: int = 1  # noqa
@@ -357,11 +374,12 @@ class ActorNursery:
 @acm
 async def _open_and_supervise_one_cancels_all_nursery(
     actor: Actor,
+    tb_hide: bool = False,
 
 ) -> typing.AsyncGenerator[ActorNursery, None]:
 
     # normally don't need to show user by default
-    __tracebackhide__: bool = True
+    __tracebackhide__: bool = tb_hide
 
     outer_err: BaseException|None = None
     inner_err: BaseException|None = None
@@ -377,17 +395,23 @@ async def _open_and_supervise_one_cancels_all_nursery(
     # `ActorNursery.start_actor()`).
 
     # errors from this daemon actor nursery bubble up to caller
-    async with trio.open_nursery() as da_nursery:
+    async with trio.open_nursery(
+        strict_exception_groups=False,
+        # ^XXX^ TODO? instead unpack any RAE as per "loose" style?
+    ) as da_nursery:
         try:
             # This is the inner level "run in actor" nursery. It is
             # awaited first since actors spawned in this way (using
-            # ``ActorNusery.run_in_actor()``) are expected to only
+            # `ActorNusery.run_in_actor()`) are expected to only
             # return a single result and then complete (i.e. be canclled
             # gracefully). Errors collected from these actors are
             # immediately raised for handling by a supervisor strategy.
             # As such if the strategy propagates any error(s) upwards
             # the above "daemon actor" nursery will be notified.
-            async with trio.open_nursery() as ria_nursery:
+            async with trio.open_nursery(
+                strict_exception_groups=False,
+                # ^XXX^ TODO? instead unpack any RAE as per "loose" style?
+            ) as ria_nursery:
 
                 an = ActorNursery(
                     actor,
@@ -454,8 +478,8 @@ async def _open_and_supervise_one_cancels_all_nursery(
                             ContextCancelled,
                         }:
                             log.cancel(
-                                'Actor-nursery caught remote cancellation\n\n'
-
+                                'Actor-nursery caught remote cancellation\n'
+                                '\n'
                                 f'{inner_err.tb_str}'
                             )
                         else:
@@ -547,7 +571,9 @@ async def _open_and_supervise_one_cancels_all_nursery(
 @acm
 # @api_frame
 async def open_nursery(
+    hide_tb: bool = True,
     **kwargs,
+    # ^TODO, paramspec for `open_root_actor()`
 
 ) -> typing.AsyncGenerator[ActorNursery, None]:
     '''
@@ -565,7 +591,7 @@ async def open_nursery(
     which cancellation scopes correspond to each spawned subactor set.
 
     '''
-    __tracebackhide__: bool = True
+    __tracebackhide__: bool = hide_tb
     implicit_runtime: bool = False
     actor: Actor = current_actor(err_on_no_runtime=False)
     an: ActorNursery|None = None
@@ -581,7 +607,10 @@ async def open_nursery(
             # mark us for teardown on exit
             implicit_runtime: bool = True
 
-            async with open_root_actor(**kwargs) as actor:
+            async with open_root_actor(
+                hide_tb=hide_tb,
+                **kwargs,
+            ) as actor:
                 assert actor is current_actor()
 
                 try:
@@ -619,8 +648,10 @@ async def open_nursery(
         # show frame on any internal runtime-scope error
         if (
             an
-            and not an.cancelled
-            and an._scope_error
+            and
+            not an.cancelled
+            and
+            an._scope_error
         ):
             __tracebackhide__: bool = False
 
@@ -629,8 +660,12 @@ async def open_nursery(
             f'|_{an}\n'
         )
 
-        # shutdown runtime if it was started
         if implicit_runtime:
+            # shutdown runtime if it was started and report noisly
+            # that we're did so.
             msg += '=> Shutting down actor runtime <=\n'
+            log.info(msg)
 
-        log.info(msg)
+        else:
+            # keep noise low during std operation.
+            log.runtime(msg)

tractor/_testing/__init__.py

@@ -19,10 +19,16 @@ Various helpers/utils for auditing your `tractor` app and/or the
 core runtime.
 
 '''
-from contextlib import asynccontextmanager as acm
+from contextlib import (
+    asynccontextmanager as acm,
+)
+import os
 import pathlib
 
 import tractor
+from tractor.devx._debug import (
+    BoxedMaybeException,
+)
 from .pytest import (
     tractor_test as tractor_test
 )
@@ -54,6 +60,35 @@ def examples_dir() -> pathlib.Path:
     return repodir() / 'examples'
 
 
+def mk_cmd(
+    ex_name: str,
+    exs_subpath: str = 'debugging',
+) -> str:
+    '''
+    Generate a shell command suitable to pass to `pexpect.spawn()`
+    which runs the script as a python program's entrypoint.
+
+    In particular ensure we disable the new tb coloring via unsetting
+    `$PYTHON_COLORS` so that `pexpect` can pattern match without
+    color-escape-codes.
+
+    '''
+    script_path: pathlib.Path = (
+        examples_dir()
+        / exs_subpath
+        / f'{ex_name}.py'
+    )
+    py_cmd: str = ' '.join([
+        'python',
+        str(script_path)
+    ])
+    # XXX, required for py 3.13+
+    # https://docs.python.org/3/using/cmdline.html#using-on-controlling-color
+    # https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_COLORS
+    os.environ['PYTHON_COLORS'] = '0'
+    return py_cmd
+
+
 @acm
 async def expect_ctxc(
     yay: bool,
@@ -66,12 +101,13 @@ async def expect_ctxc(
     '''
     if yay:
         try:
-            yield
+            yield (maybe_exc := BoxedMaybeException())
             raise RuntimeError('Never raised ctxc?')
-        except tractor.ContextCancelled:
+        except tractor.ContextCancelled as ctxc:
+            maybe_exc.value = ctxc
             if reraise:
                 raise
             else:
                 return
     else:
-        yield
+        yield (maybe_exc := BoxedMaybeException())

tractor/devx/__init__.py

@@ -26,9 +26,10 @@ from ._debug import (
     breakpoint as breakpoint,
     pause as pause,
     pause_from_sync as pause_from_sync,
-    shield_sigint_handler as shield_sigint_handler,
+    sigint_shield as sigint_shield,
     open_crash_handler as open_crash_handler,
     maybe_open_crash_handler as maybe_open_crash_handler,
+    maybe_init_greenback as maybe_init_greenback,
     post_mortem as post_mortem,
     mk_pdb as mk_pdb,
 )
@@ -40,3 +41,38 @@ from .pformat import (
     pformat_caller_frame as pformat_caller_frame,
     pformat_boxed_tb as pformat_boxed_tb,
 )
+
+
+# TODO, move this to a new `.devx._pdbp` mod?
+def _enable_readline_feats() -> str:
+    '''
+    Handle `readline` when compiled with `libedit` to avoid breaking
+    tab completion in `pdbp` (and its dep `tabcompleter`)
+    particularly since `uv` cpython distis are compiled this way..
+
+    See docs for deats,
+    https://docs.python.org/3/library/readline.html#module-readline
+
+    Originally discovered soln via SO answer,
+    https://stackoverflow.com/q/49287102
+
+    '''
+    import readline
+    if (
+        # 3.13+ attr
+        # https://docs.python.org/3/library/readline.html#readline.backend
+        (getattr(readline, 'backend', False) == 'libedit')
+        or
+        'libedit' in readline.__doc__
+    ):
+        readline.parse_and_bind("python:bind -v")
+        readline.parse_and_bind("python:bind ^I rl_complete")
+        return 'libedit'
+    else:
+        readline.parse_and_bind("tab: complete")
+        readline.parse_and_bind("set editing-mode vi")
+        readline.parse_and_bind("set keymap vi")
+        return 'readline'
+
+
+_enable_readline_feats()

tractor/devx/_frame_stack.py

@@ -234,7 +234,7 @@ def find_caller_info(
 _frame2callerinfo_cache: dict[FrameType, CallerInfo] = {}
 
 
-# TODO: -[x] move all this into new `.devx._code`!
+# TODO: -[x] move all this into new `.devx._frame_stack`!
 # -[ ] consider rename to _callstack?
 # -[ ] prolly create a `@runtime_api` dec?
 #   |_ @api_frame seems better?
@@ -286,3 +286,18 @@ def api_frame(
     wrapped._call_infos: dict[FrameType, CallerInfo] = _frame2callerinfo_cache
     wrapped.__api_func__: bool = True
     return wrapper(wrapped)
+
+
+# TODO: something like this instead of the adhoc frame-unhiding
+# blocks all over the runtime!! XD
+# -[ ] ideally we can expect a certain error (set) and if something
+#     else is raised then all frames below the wrapped one will be
+#     un-hidden via `__tracebackhide__: bool = False`.
+# |_ might need to dynamically mutate the code objs like
+#    `pdbp.hideframe()` does?
+# -[ ] use this as a `@acm` decorator as introed in 3.10?
+# @acm
+# async def unhide_frame_when_not(
+#     error_set: set[BaseException],
+# ) -> TracebackType:
+#     ...

tractor/devx/_stackscope.py

@@ -24,19 +24,32 @@ disjoint, parallel executing tasks in separate actors.
 
 '''
 from __future__ import annotations
+# from functools import partial
+from threading import (
+    current_thread,
+    Thread,
+    RLock,
+)
 import multiprocessing as mp
 from signal import (
     signal,
+    getsignal,
     SIGUSR1,
+    SIGINT,
+)
+# import traceback
+from types import ModuleType
+from typing import (
+    Callable,
+    TYPE_CHECKING,
 )
-import traceback
-from typing import TYPE_CHECKING
 
 import trio
 from tractor import (
     _state,
     log as logmod,
 )
+from tractor.devx import _debug
 
 log = logmod.get_logger(__name__)
 
@@ -51,26 +64,68 @@ if TYPE_CHECKING:
 
 @trio.lowlevel.disable_ki_protection
 def dump_task_tree() -> None:
-    import stackscope
-    from tractor.log import get_console_log
+    '''
+    Do a classic `stackscope.extract()` task-tree dump to console at
+    `.devx()` level.
 
+    '''
+    import stackscope
     tree_str: str = str(
         stackscope.extract(
             trio.lowlevel.current_root_task(),
             recurse_child_tasks=True
         )
     )
-    log = get_console_log(
-        name=__name__,
-        level='cancel',
-    )
     actor: Actor = _state.current_actor()
+    thr: Thread = current_thread()
+    current_sigint_handler: Callable = getsignal(SIGINT)
+    if (
+        current_sigint_handler
+        is not
+        _debug.DebugStatus._trio_handler
+    ):
+        sigint_handler_report: str = (
+            'The default `trio` SIGINT handler was replaced?!'
+        )
+    else:
+        sigint_handler_report: str = (
+            'The default `trio` SIGINT handler is in use?!'
+        )
+
+    # sclang symbology
+    # |_<object>
+    # |_(Task/Thread/Process/Actor
+    # |_{Supervisor/Scope
+    # |_[Storage/Memory/IPC-Stream/Data-Struct
+
     log.devx(
         f'Dumping `stackscope` tree for actor\n'
-        f'{actor.name}: {actor}\n'
-        f' |_{mp.current_process()}\n\n'
-        f'{tree_str}\n'
+        f'(>: {actor.uid!r}\n'
+        f' |_{mp.current_process()}\n'
+        f'   |_{thr}\n'
+        f'     |_{actor}\n'
+        f'\n'
+        f'{sigint_handler_report}\n'
+        f'signal.getsignal(SIGINT) -> {current_sigint_handler!r}\n'
+        # f'\n'
+        # start-of-trace-tree delimiter (mostly for testing)
+        # f'------ {actor.uid!r} ------\n'
+        f'\n'
+        f'------ start-of-{actor.uid!r} ------\n'
+        f'|\n'
+        f'{tree_str}'
+        # end-of-trace-tree delimiter (mostly for testing)
+        f'|\n'
+        f'|_____ end-of-{actor.uid!r} ______\n'
     )
+    # TODO: can remove this right?
+    # -[ ] was original code from author
+    #
+    # print(
+    #     'DUMPING FROM PRINT\n'
+    #     +
+    #     content
+    # )
     # import logging
     # try:
     #     with open("/dev/tty", "w") as tty:
@@ -80,58 +135,130 @@ def dump_task_tree() -> None:
     #         "task_tree"
     #     ).exception("Error printing task tree")
 
+_handler_lock = RLock()
+_tree_dumped: bool = False
 
-def signal_handler(
+
+def dump_tree_on_sig(
     sig: int,
     frame: object,
 
     relay_to_subs: bool = True,
 
 ) -> None:
-    try:
-        trio.lowlevel.current_trio_token(
-        ).run_sync_soon(dump_task_tree)
-    except RuntimeError:
-        # not in async context -- print a normal traceback
-        traceback.print_stack()
+    global _tree_dumped, _handler_lock
+    with _handler_lock:
+        # if _tree_dumped:
+        #     log.warning(
+        #         'Already dumped for this actor...??'
+        #     )
+        #     return
+
+        _tree_dumped = True
+
+        # actor: Actor = _state.current_actor()
+        log.devx(
+            'Trying to dump `stackscope` tree..\n'
+        )
+        try:
+            dump_task_tree()
+            # await actor._service_n.start_soon(
+            #     partial(
+            #         trio.to_thread.run_sync,
+            #         dump_task_tree,
+            #     )
+            # )
+            # trio.lowlevel.current_trio_token().run_sync_soon(
+            #     dump_task_tree
+            # )
+
+        except RuntimeError:
+            log.exception(
+                'Failed to dump `stackscope` tree..\n'
+            )
+            # not in async context -- print a normal traceback
+            # traceback.print_stack()
+            raise
+
+        except BaseException:
+            log.exception(
+                'Failed to dump `stackscope` tree..\n'
+            )
+            raise
+
+        # log.devx(
+        #     'Supposedly we dumped just fine..?'
+        # )
 
     if not relay_to_subs:
         return
 
     an: ActorNursery
     for an in _state.current_actor()._actoruid2nursery.values():
-
         subproc: ProcessType
         subactor: Actor
         for subactor, subproc, _ in an._children.values():
-            log.devx(
+            log.warning(
                 f'Relaying `SIGUSR1`[{sig}] to sub-actor\n'
                 f'{subactor}\n'
                 f' |_{subproc}\n'
             )
 
-            if isinstance(subproc, trio.Process):
-                subproc.send_signal(sig)
+            # bc of course stdlib can't have a std API.. XD
+            match subproc:
+                case trio.Process():
+                    subproc.send_signal(sig)
 
-            elif isinstance(subproc, mp.Process):
-                subproc._send_signal(sig)
+                case mp.Process():
+                    subproc._send_signal(sig)
 
 
 def enable_stack_on_sig(
-    sig: int = SIGUSR1
-) -> None:
+    sig: int = SIGUSR1,
+) -> ModuleType:
     '''
     Enable `stackscope` tracing on reception of a signal; by
     default this is SIGUSR1.
 
+    HOT TIP: a task/ctx-tree dump can be triggered from a shell with
+    fancy cmds.
+
+    For ex. from `bash` using `pgrep` and cmd-sustitution
+    (https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution)
+    you could use:
+
+    >> kill -SIGUSR1 $(pgrep -f <part-of-cmd: str>)
+
+    OR without a sub-shell,
+
+    >> pkill --signal SIGUSR1 -f <part-of-cmd: str>
+
     '''
+    try:
+        import stackscope
+    except ImportError:
+        log.warning(
+            '`stackscope` not installed for use in debug mode!'
+        )
+        return None
+
+    handler: Callable|int = getsignal(sig)
+    if handler is dump_tree_on_sig:
+        log.devx(
+            'A `SIGUSR1` handler already exists?\n'
+            f'|_ {handler!r}\n'
+        )
+        return
+
     signal(
         sig,
-        signal_handler,
+        dump_tree_on_sig,
     )
-    # NOTE: not the above can be triggered from
-    # a (xonsh) shell using:
-    # kill -SIGUSR1 @$(pgrep -f '<cmd>')
-    #
-    # for example if you were looking to trace a `pytest` run
-    # kill -SIGUSR1 @$(pgrep -f 'pytest')
+    log.devx(
+        'Enabling trace-trees on `SIGUSR1` '
+        'since `stackscope` is installed @ \n'
+        f'{stackscope!r}\n\n'
+        f'With `SIGUSR1` handler\n'
+        f'|_{dump_tree_on_sig}\n'
+    )
+    return stackscope

tractor/devx/pformat.py

@@ -53,6 +53,7 @@ def pformat_boxed_tb(
 
     tb_box_indent: int|None = None,
     tb_body_indent: int = 1,
+    boxer_header: str = '-'
 
 ) -> str:
     '''
@@ -88,10 +89,10 @@ def pformat_boxed_tb(
 
     tb_box: str = (
         f'|\n'
-        f' ------ - ------\n'
+        f' ------ {boxer_header} ------\n'
         f'{tb_body}'
-        f' ------ - ------\n'
-        f'_|\n'
+        f' ------ {boxer_header}- ------\n'
+        f'_|'
     )
     tb_box_indent: str = (
         tb_box_indent

tractor/log.py

@@ -54,11 +54,12 @@ LOG_FORMAT = (
 DATE_FORMAT = '%b %d %H:%M:%S'
 
 # FYI, ERROR is 40
+# TODO: use a `bidict` to avoid the :155 check?
 CUSTOM_LEVELS: dict[str, int] = {
     'TRANSPORT': 5,
     'RUNTIME': 15,
     'DEVX': 17,
-    'CANCEL': 18,
+    'CANCEL': 22,
     'PDB': 500,
 }
 STD_PALETTE = {
@@ -147,6 +148,8 @@ class StackLevelAdapter(LoggerAdapter):
         Delegate a log call to the underlying logger, after adding
         contextual information from this adapter instance.
 
+        NOTE: all custom level methods (above) delegate to this!
+
         '''
         if self.isEnabledFor(level):
             stacklevel: int = 3
@@ -255,20 +258,28 @@ class ActorContextInfo(Mapping):
 
 
 def get_logger(
-
-    name: str | None = None,
+    name: str|None = None,
     _root_name: str = _proj_name,
 
+    logger: Logger|None = None,
+
+    # TODO, using `.config.dictConfig()` api?
+    # -[ ] SO answer with docs links
+    #  |_https://stackoverflow.com/questions/7507825/where-is-a-complete-example-of-logging-config-dictconfig
+    #  |_https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema
+    subsys_spec: str|None = None,
+
 ) -> StackLevelAdapter:
     '''Return the package log or a sub-logger for ``name`` if provided.
 
     '''
     log: Logger
-    log = rlog = logging.getLogger(_root_name)
+    log = rlog = logger or logging.getLogger(_root_name)
 
     if (
         name
-        and name != _proj_name
+        and
+        name != _proj_name
     ):
 
         # NOTE: for handling for modules that use ``get_logger(__name__)``
@@ -280,7 +291,7 @@ def get_logger(
         #   since in python the {filename} is always this same
         #   module-file.
 
-        sub_name: None | str = None
+        sub_name: None|str = None
         rname, _, sub_name = name.partition('.')
         pkgpath, _, modfilename = sub_name.rpartition('.')
 
@@ -303,7 +314,10 @@ def get_logger(
 
     # add our actor-task aware adapter which will dynamically look up
     # the actor and task names at each log emit
-    logger = StackLevelAdapter(log, ActorContextInfo())
+    logger = StackLevelAdapter(
+        log,
+        ActorContextInfo(),
+    )
 
     # additional levels
     for name, val in CUSTOM_LEVELS.items():
@@ -316,15 +330,25 @@ def get_logger(
 
 
 def get_console_log(
-    level: str | None = None,
+    level: str|None = None,
+    logger: Logger|None = None,
     **kwargs,
-) -> LoggerAdapter:
-    '''Get the package logger and enable a handler which writes to stderr.
 
-    Yeah yeah, i know we can use ``DictConfig``. You do it.
+) -> LoggerAdapter:
     '''
-    log = get_logger(**kwargs)  # our root logger
-    logger = log.logger
+    Get a `tractor`-style logging instance: a `Logger` wrapped in
+    a `StackLevelAdapter` which injects various concurrency-primitive
+    (process, thread, task) fields and enables a `StreamHandler` that
+    writes on stderr using `colorlog` formatting.
+
+    Yeah yeah, i know we can use `logging.config.dictConfig()`. You do it.
+
+    '''
+    log = get_logger(
+        logger=logger,
+        **kwargs
+    )  # set a root logger
+    logger: Logger = log.logger
 
     if not level:
         return log
@@ -343,9 +367,13 @@ def get_console_log(
             None,
         )
     ):
+        fmt = LOG_FORMAT
+        # if logger:
+        #     fmt = None
+
         handler = StreamHandler()
         formatter = colorlog.ColoredFormatter(
-            LOG_FORMAT,
+            fmt=fmt,
             datefmt=DATE_FORMAT,
             log_colors=STD_PALETTE,
             secondary_log_colors=BOLD_PALETTE,
@@ -362,7 +390,7 @@ def get_loglevel() -> str:
 
 
 # global module logger for tractor itself
-log = get_logger('tractor')
+log: StackLevelAdapter = get_logger('tractor')
 
 
 def at_least_level(

tractor/msg/__init__.py

@@ -33,6 +33,7 @@ from ._codec import (
 
     apply_codec as apply_codec,
     mk_codec as mk_codec,
+    mk_dec as mk_dec,
     MsgCodec as MsgCodec,
     MsgDec as MsgDec,
     current_codec as current_codec,

tractor/msg/_codec.py

@@ -41,8 +41,10 @@ import textwrap
 from typing import (
     Any,
     Callable,
+    Protocol,
     Type,
     TYPE_CHECKING,
+    TypeVar,
     Union,
 )
 from types import ModuleType
@@ -59,6 +61,7 @@ from tractor.msg.pretty_struct import Struct
 from tractor.msg.types import (
     mk_msg_spec,
     MsgType,
+    PayloadMsg,
 )
 from tractor.log import get_logger
 
@@ -78,6 +81,7 @@ class MsgDec(Struct):
 
     '''
     _dec: msgpack.Decoder
+    # _ext_types_box: Struct|None = None
 
     @property
     def dec(self) -> msgpack.Decoder:
@@ -177,19 +181,126 @@ class MsgDec(Struct):
 
 
 def mk_dec(
-    spec: Union[Type[Struct]]|Any = Any,
+    spec: Union[Type[Struct]]|Type|None,
+
+    # NOTE, required for ad-hoc type extensions to the underlying
+    # serialization proto (which is default `msgpack`),
+    # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
     dec_hook: Callable|None = None,
+    ext_types: list[Type]|None = None,
 
 ) -> MsgDec:
+    '''
+    Create an IPC msg decoder, a slightly higher level wrapper around
+    a `msgspec.msgpack.Decoder` which provides,
+
+    - easier introspection of the underlying type spec via
+      the `.spec` and `.spec_str` attrs,
+    - `.hook` access to the `Decoder.dec_hook()`,
+    - automatic custom extension-types decode support when
+      `dec_hook()` is provided such that any `PayloadMsg.pld` tagged
+      as a type from from `ext_types` (presuming the `MsgCodec.encode()` also used
+      a `.enc_hook()`) is processed and constructed by a `PldRx` implicitily.
+
+    NOTE, as mentioned a `MsgDec` is normally used for `PayloadMsg.pld: PayloadT` field
+    decoding inside an IPC-ctx-oriented `PldRx`.
+
+    '''
+    if (
+        spec is None
+        and
+        ext_types is None
+    ):
+        raise TypeError(
+            f'MIssing type-`spec` for msg decoder!\n'
+            f'\n'
+            f'`spec=None` is **only** permitted is if custom extension types '
+            f'are provided via `ext_types`, meaning it must be non-`None`.\n'
+            f'\n'
+            f'In this case it is presumed that only the `ext_types`, '
+            f'which much be handled by a paired `dec_hook()`, '
+            f'will be permitted within the payload type-`spec`!\n'
+            f'\n'
+            f'spec = {spec!r}\n'
+            f'dec_hook = {dec_hook!r}\n'
+            f'ext_types = {ext_types!r}\n'
+        )
+
+    if dec_hook:
+        if ext_types is None:
+            raise TypeError(
+                f'If extending the serializable types with a custom decode hook (`dec_hook()`), '
+                f'you must also provide the expected type set that the hook will handle '
+                f'via a `ext_types: Union[Type]|None = None` argument!\n'
+                f'\n'
+                f'dec_hook = {dec_hook!r}\n'
+                f'ext_types = {ext_types!r}\n'
+            )
+
+        # XXX, i *thought* we would require a boxing struct as per docs,
+        # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
+        # |_ see comment,
+        #  > Note that typed deserialization is required for
+        #  > successful roundtripping here, so we pass `MyMessage` to
+        #  > `Decoder`.
+        #
+        # BUT, turns out as long as you spec a union with `Raw` it
+        # will work? kk B)
+        #
+        # maybe_box_struct = mk_boxed_ext_struct(ext_types)
+        spec = Raw | Union[*ext_types]
 
     return MsgDec(
         _dec=msgpack.Decoder(
             type=spec,  # like `MsgType[Any]`
             dec_hook=dec_hook,
-        )
+        ),
     )
 
 
+# TODO? remove since didn't end up needing this?
+def mk_boxed_ext_struct(
+    ext_types: list[Type],
+) -> Struct:
+    # NOTE, originally was to wrap non-msgpack-supported "extension
+    # types" in a field-typed boxing struct, see notes around the
+    # `dec_hook()` branch in `mk_dec()`.
+    ext_types_union = Union[*ext_types]
+    repr_ext_types_union: str = (
+        str(ext_types_union)
+        or
+        "|".join(ext_types)
+    )
+    BoxedExtType = msgspec.defstruct(
+        f'BoxedExts[{repr_ext_types_union}]',
+        fields=[
+            ('boxed', ext_types_union),
+        ],
+    )
+    return BoxedExtType
+
+
+def unpack_spec_types(
+    spec: Union[Type]|Type,
+) -> set[Type]:
+    '''
+    Given an input type-`spec`, either a lone type
+    or a `Union` of types (like `str|int|MyThing`),
+    return a set of individual types.
+
+    When `spec` is not a type-union returns `{spec,}`.
+
+    '''
+    spec_subtypes: set[Union[Type]] = set(
+         getattr(
+             spec,
+             '__args__',
+             {spec,},
+         )
+    )
+    return spec_subtypes
+
+
 def mk_msgspec_table(
     dec: msgpack.Decoder,
     msg: MsgType|None = None,
@@ -227,6 +338,13 @@ def pformat_msgspec(
     join_char: str = '\n',
 
 ) -> str:
+    '''
+    Pretty `str` format the `msgspec.msgpack.Decoder.type` attribute
+    for display in (console) log messages as a nice (maybe multiline)
+    presentation of all supported `Struct`s (subtypes) available for
+    typed decoding.
+
+    '''
     dec: msgpack.Decoder = getattr(codec, 'dec', codec)
     return join_char.join(
         mk_msgspec_table(
@@ -260,6 +378,8 @@ class MsgCodec(Struct):
     _dec: msgpack.Decoder
     _pld_spec: Type[Struct]|Raw|Any
 
+    # _ext_types_box: Struct|None = None
+
     def __repr__(self) -> str:
         speclines: str = textwrap.indent(
             pformat_msgspec(codec=self),
@@ -326,12 +446,15 @@ class MsgCodec(Struct):
 
     def encode(
         self,
-        py_obj: Any,
+        py_obj: Any|PayloadMsg,
 
         use_buf: bool = False,
         # ^-XXX-^ uhh why am i getting this?
         # |_BufferError: Existing exports of data: object cannot be re-sized
 
+        as_ext_type: bool = False,
+        hide_tb: bool = True,
+
     ) -> bytes:
         '''
         Encode input python objects to `msgpack` bytes for
@@ -341,11 +464,46 @@ class MsgCodec(Struct):
         https://jcristharif.com/msgspec/perf-tips.html#reusing-an-output-buffer
 
         '''
+        __tracebackhide__: bool = hide_tb
         if use_buf:
             self._enc.encode_into(py_obj, self._buf)
             return self._buf
-        else:
-            return self._enc.encode(py_obj)
+
+        return self._enc.encode(py_obj)
+        # try:
+        #     return self._enc.encode(py_obj)
+        # except TypeError as typerr:
+        #     typerr.add_note(
+        #         '|_src error from `msgspec`'
+        #         # f'|_{self._enc.encode!r}'
+        #     )
+        #     raise typerr
+
+        # TODO! REMOVE once i'm confident we won't ever need it!
+        #
+        # box: Struct = self._ext_types_box
+        # if (
+        #     as_ext_type
+        #     or
+        #     (
+        #         # XXX NOTE, auto-detect if the input type
+        #         box
+        #         and
+        #         (ext_types := unpack_spec_types(
+        #             spec=box.__annotations__['boxed'])
+        #         )
+        #     )
+        # ):
+        #     match py_obj:
+        #         # case PayloadMsg(pld=pld) if (
+        #         #     type(pld) in ext_types
+        #         # ):
+        #         #     py_obj.pld = box(boxed=py_obj)
+        #         #     breakpoint()
+        #         case _ if (
+        #             type(py_obj) in ext_types
+        #         ):
+        #             py_obj = box(boxed=py_obj)
 
     @property
     def dec(self) -> msgpack.Decoder:
@@ -365,21 +523,30 @@ class MsgCodec(Struct):
         return self._dec.decode(msg)
 
 
-# [x] TODO: a sub-decoder system as well? => No!
+# ?TODO? time to remove this finally?
+#
+# -[x] TODO: a sub-decoder system as well?
+# => No! already re-architected to include a "payload-receiver"
+#   now found in `._ops`.
 #
 # -[x] do we still want to try and support the sub-decoder with
 # `.Raw` technique in the case that the `Generic` approach gives
 # future grief?
-# => NO, since we went with the `PldRx` approach instead B)
+# => well YES but NO, since we went with the `PldRx` approach
+#   instead!
 #
 # IF however you want to see the code that was staged for this
 # from wayyy back, see the pure removal commit.
 
 
 def mk_codec(
-    # struct type unions set for `Decoder`
-    # https://jcristharif.com/msgspec/structs.html#tagged-unions
-    ipc_pld_spec: Union[Type[Struct]]|Any = Any,
+    ipc_pld_spec: Union[Type[Struct]]|Any|Raw = Raw,
+    # tagged-struct-types-union set for `Decoder`ing of payloads, as
+    # per https://jcristharif.com/msgspec/structs.html#tagged-unions.
+    # NOTE that the default `Raw` here **is very intentional** since
+    # the `PldRx._pld_dec: MsgDec` is responsible for per ipc-ctx-task
+    # decoding of msg-specs defined by the user as part of **their**
+    # `tractor` "app's" type-limited IPC msg-spec.
 
     # TODO: offering a per-msg(-field) type-spec such that
     # the fields can be dynamically NOT decoded and left as `Raw`
@@ -392,13 +559,18 @@ def mk_codec(
 
     libname: str = 'msgspec',
 
-    # proxy as `Struct(**kwargs)` for ad-hoc type extensions
+    # settings for encoding-to-send extension-types,
     # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
-    # ------ - ------
-    dec_hook: Callable|None = None,
+    # dec_hook: Callable|None = None,
     enc_hook: Callable|None = None,
-    # ------ - ------
+    ext_types: list[Type]|None = None,
+
+    # optionally provided msg-decoder from which we pull its,
+    # |_.dec_hook()
+    # |_.type
+    ext_dec: MsgDec|None = None
     #
+    # ?TODO? other params we might want to support
     # Encoder:
     # write_buffer_size=write_buffer_size,
     #
@@ -412,26 +584,44 @@ def mk_codec(
     `msgspec` ;).
 
     '''
-    # (manually) generate a msg-payload-spec for all relevant
-    # god-boxing-msg subtypes, parameterizing the `PayloadMsg.pld: PayloadT`
-    # for the decoder such that all sub-type msgs in our SCIPP
-    # will automatically decode to a type-"limited" payload (`Struct`)
-    # object (set).
+    pld_spec = ipc_pld_spec
+    if enc_hook:
+        if not ext_types:
+            raise TypeError(
+                f'If extending the serializable types with a custom encode hook (`enc_hook()`), '
+                f'you must also provide the expected type set that the hook will handle '
+                f'via a `ext_types: Union[Type]|None = None` argument!\n'
+                f'\n'
+                f'enc_hook = {enc_hook!r}\n'
+                f'ext_types = {ext_types!r}\n'
+            )
+
+    dec_hook: Callable|None = None
+    if ext_dec:
+        dec: msgspec.Decoder = ext_dec.dec
+        dec_hook = dec.dec_hook
+        pld_spec |= dec.type
+        if ext_types:
+            pld_spec |= Union[*ext_types]
+
+    # (manually) generate a msg-spec (how appropes) for all relevant
+    # payload-boxing-struct-msg-types, parameterizing the
+    # `PayloadMsg.pld: PayloadT` for the decoder such that all msgs
+    # in our SC-RPC-protocol will automatically decode to
+    # a type-"limited" payload (`Struct`) object (set).
     (
         ipc_msg_spec,
         msg_types,
     ) = mk_msg_spec(
-        payload_type_union=ipc_pld_spec,
+        payload_type_union=pld_spec,
     )
-    assert len(ipc_msg_spec.__args__) == len(msg_types)
-    assert ipc_msg_spec
 
-    # TODO: use this shim instead?
-    # bc.. unification, err somethin?
-    # dec: MsgDec = mk_dec(
-    #     spec=ipc_msg_spec,
-    #     dec_hook=dec_hook,
-    # )
+    msg_spec_types: set[Type] = unpack_spec_types(ipc_msg_spec)
+    assert (
+        len(ipc_msg_spec.__args__) == len(msg_types)
+        and
+        len(msg_spec_types) == len(msg_types)
+    )
 
     dec = msgpack.Decoder(
         type=ipc_msg_spec,
@@ -440,22 +630,29 @@ def mk_codec(
     enc = msgpack.Encoder(
        enc_hook=enc_hook,
     )
-
     codec = MsgCodec(
         _enc=enc,
         _dec=dec,
-        _pld_spec=ipc_pld_spec,
+        _pld_spec=pld_spec,
     )
-
     # sanity on expected backend support
     assert codec.lib.__name__ == libname
-
     return codec
 
 
 # instance of the default `msgspec.msgpack` codec settings, i.e.
 # no custom structs, hooks or other special types.
-_def_msgspec_codec: MsgCodec = mk_codec(ipc_pld_spec=Any)
+#
+# XXX NOTE XXX, this will break our `Context.start()` call!
+#
+# * by default we roundtrip the started pld-`value` and if you apply
+#   this codec (globally anyway with `apply_codec()`) then the
+#   `roundtripped` value will include a non-`.pld: Raw` which will
+#   then type-error on the consequent `._ops.validte_payload_msg()`..
+#
+_def_msgspec_codec: MsgCodec = mk_codec(
+    ipc_pld_spec=Any,
+)
 
 # The built-in IPC `Msg` spec.
 # Our composing "shuttle" protocol which allows `tractor`-app code
@@ -463,13 +660,13 @@ _def_msgspec_codec: MsgCodec = mk_codec(ipc_pld_spec=Any)
 # https://jcristharif.com/msgspec/supported-types.html
 #
 _def_tractor_codec: MsgCodec = mk_codec(
-    # TODO: use this for debug mode locking prot?
-    # ipc_pld_spec=Any,
-    ipc_pld_spec=Raw,
+    ipc_pld_spec=Raw,  # XXX should be default righ!?
 )
-# TODO: IDEALLY provides for per-`trio.Task` specificity of the
+
+# -[x] TODO, IDEALLY provides for per-`trio.Task` specificity of the
 # IPC msging codec used by the transport layer when doing
 # `Channel.send()/.recv()` of wire data.
+# => impled as our `PldRx` which is `Context` scoped B)
 
 # ContextVar-TODO: DIDN'T WORK, kept resetting in every new task to default!?
 # _ctxvar_MsgCodec: ContextVar[MsgCodec] = ContextVar(
@@ -546,17 +743,6 @@ def apply_codec(
     )
     token: Token = var.set(codec)
 
-    # ?TODO? for TreeVar approach which copies from the
-    # cancel-scope of the prior value, NOT the prior task
-    # See the docs:
-    # - https://tricycle.readthedocs.io/en/latest/reference.html#tree-variables
-    # - https://github.com/oremanj/tricycle/blob/master/tricycle/_tests/test_tree_var.py
-    #   ^- see docs for @cm `.being()` API
-    # with _ctxvar_MsgCodec.being(codec):
-    #     new = _ctxvar_MsgCodec.get()
-    #     assert new is codec
-    #     yield codec
-
     try:
         yield var.get()
     finally:
@@ -567,6 +753,19 @@ def apply_codec(
         )
         assert var.get() is orig
 
+    # ?TODO? for TreeVar approach which copies from the
+    # cancel-scope of the prior value, NOT the prior task
+    #
+    # See the docs:
+    # - https://tricycle.readthedocs.io/en/latest/reference.html#tree-variables
+    # - https://github.com/oremanj/tricycle/blob/master/tricycle/_tests/test_tree_var.py
+    #   ^- see docs for @cm `.being()` API
+    #
+    # with _ctxvar_MsgCodec.being(codec):
+    #     new = _ctxvar_MsgCodec.get()
+    #     assert new is codec
+    #     yield codec
+
 
 def current_codec() -> MsgCodec:
     '''
@@ -586,6 +785,7 @@ def limit_msg_spec(
     # -> related to the `MsgCodec._payload_decs` stuff above..
     # tagged_structs: list[Struct]|None = None,
 
+    hide_tb: bool = True,
     **codec_kwargs,
 
 ) -> MsgCodec:
@@ -596,7 +796,7 @@ def limit_msg_spec(
     for all IPC contexts in use by the current `trio.Task`.
 
     '''
-    __tracebackhide__: bool = True
+    __tracebackhide__: bool = hide_tb
     curr_codec: MsgCodec = current_codec()
     msgspec_codec: MsgCodec = mk_codec(
         ipc_pld_spec=payload_spec,
@@ -630,31 +830,57 @@ def limit_msg_spec(
 #         # import pdbp; pdbp.set_trace()
 #         assert ext_codec.pld_spec == extended_spec
 #         yield ext_codec
+#
+# ^-TODO-^ is it impossible to make something like this orr!?
+
+# TODO: make an auto-custom hook generator from a set of input custom
+# types?
+# -[ ] below is a proto design using a `TypeCodec` idea?
+#
+# type var for the expected interchange-lib's
+# IPC-transport type when not available as a built-in
+# serialization output.
+WireT = TypeVar('WireT')
 
 
-# TODO: make something similar to this inside `._codec` such that
-# user can just pass a type table of some sort?
-# -[ ] we would need to decode all msgs to `pretty_struct.Struct`
-#     and then call `.to_dict()` on them?
-# -[x] we're going to need to re-impl all the stuff changed in the
-#    runtime port such that it can handle dicts or `Msg`s?
-#
-# def mk_dict_msg_codec_hooks() -> tuple[Callable, Callable]:
-#     '''
-#     Deliver a `enc_hook()`/`dec_hook()` pair which does
-#     manual convertion from our above native `Msg` set
-#     to `dict` equivalent (wire msgs) in order to keep legacy compat
-#     with the original runtime implementation.
-#
-#     Note: this is is/was primarly used while moving the core
-#     runtime over to using native `Msg`-struct types wherein we
-#     start with the send side emitting without loading
-#     a typed-decoder and then later flipping the switch over to
-#     load to the native struct types once all runtime usage has
-#     been adjusted appropriately.
-#
-#     '''
-#     return (
-#         # enc_to_dict,
-#         dec_from_dict,
-#     )
+# TODO: some kinda (decorator) API for built-in subtypes
+# that builds this implicitly by inspecting the `mro()`?
+class TypeCodec(Protocol):
+    '''
+    A per-custom-type wire-transport serialization translator
+    description type.
+
+    '''
+    src_type: Type
+    wire_type: WireT
+
+    def encode(obj: Type) -> WireT:
+        ...
+
+    def decode(
+        obj_type: Type[WireT],
+        obj: WireT,
+    ) -> Type:
+        ...
+
+
+class MsgpackTypeCodec(TypeCodec):
+    ...
+
+
+def mk_codec_hooks(
+    type_codecs: list[TypeCodec],
+
+) -> tuple[Callable, Callable]:
+    '''
+    Deliver a `enc_hook()`/`dec_hook()` pair which handle
+    manual convertion from an input `Type` set such that whenever
+    the `TypeCodec.filter()` predicate matches the
+    `TypeCodec.decode()` is called on the input native object by
+    the `dec_hook()` and whenever the
+    `isiinstance(obj, TypeCodec.type)` matches against an
+    `enc_hook(obj=obj)` the return value is taken from a
+    `TypeCodec.encode(obj)` callback.
+
+    '''
+    ...

tractor/msg/_exts.py

@@ -0,0 +1,94 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Type-extension-utils for codec-ing (python) objects not
+covered by the `msgspec.msgpack` protocol.
+
+See the various API docs from `msgspec`.
+
+extending from native types,
+- https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
+
+converters,
+- https://jcristharif.com/msgspec/converters.html
+- https://jcristharif.com/msgspec/api.html#msgspec.convert
+
+`Raw` fields,
+- https://jcristharif.com/msgspec/api.html#raw
+- support for `.convert()` and `Raw`,
+  |_ https://jcristharif.com/msgspec/changelog.html
+
+'''
+from types import (
+    ModuleType,
+)
+import typing
+from typing import (
+    Type,
+    Union,
+)
+
+def dec_type_union(
+    type_names: list[str],
+    mods: list[ModuleType] = []
+) -> Type|Union[Type]:
+    '''
+    Look up types by name, compile into a list and then create and
+    return a `typing.Union` from the full set.
+
+    '''
+    # import importlib
+    types: list[Type] = []
+    for type_name in type_names:
+        for mod in [
+            typing,
+            # importlib.import_module(__name__),
+        ] + mods:
+            if type_ref := getattr(
+                mod,
+                type_name,
+                False,
+            ):
+                types.append(type_ref)
+
+    # special case handling only..
+    # ipc_pld_spec: Union[Type] = eval(
+    #     pld_spec_str,
+    #     {},  # globals
+    #     {'typing': typing},  # locals
+    # )
+
+    return Union[*types]
+
+
+def enc_type_union(
+    union_or_type: Union[Type]|Type,
+) -> list[str]:
+    '''
+    Encode a type-union or single type to a list of type-name-strings
+    ready for IPC interchange.
+
+    '''
+    type_strs: list[str] = []
+    for typ in getattr(
+        union_or_type,
+        '__args__',
+        {union_or_type,},
+    ):
+        type_strs.append(typ.__qualname__)
+
+    return type_strs

tractor/msg/_ops.py

@@ -50,7 +50,9 @@ from tractor._exceptions import (
     _mk_recv_mte,
     pack_error,
 )
-from tractor._state import current_ipc_ctx
+from tractor._state import (
+    current_ipc_ctx,
+)
 from ._codec import (
     mk_dec,
     MsgDec,
@@ -78,7 +80,7 @@ if TYPE_CHECKING:
 log = get_logger(__name__)
 
 
-_def_any_pldec: MsgDec[Any] = mk_dec()
+_def_any_pldec: MsgDec[Any] = mk_dec(spec=Any)
 
 
 class PldRx(Struct):
@@ -108,33 +110,11 @@ class PldRx(Struct):
     # TODO: better to bind it here?
     # _rx_mc: trio.MemoryReceiveChannel
     _pld_dec: MsgDec
-    _ctx: Context|None = None
-    _ipc: Context|MsgStream|None = None
 
     @property
     def pld_dec(self) -> MsgDec:
         return self._pld_dec
 
-    # TODO: a better name?
-    # -[ ] when would this be used as it avoids needingn to pass the
-    #   ipc prim to every method
-    @cm
-    def wraps_ipc(
-        self,
-        ipc_prim: Context|MsgStream,
-
-    ) -> PldRx:
-        '''
-        Apply this payload receiver to an IPC primitive type, one
-        of `Context` or `MsgStream`.
-
-        '''
-        self._ipc = ipc_prim
-        try:
-            yield self
-        finally:
-            self._ipc = None
-
     @cm
     def limit_plds(
         self,
@@ -148,6 +128,10 @@ class PldRx(Struct):
         exit.
 
         '''
+        # TODO, ensure we pull the current `MsgCodec`'s custom
+        # dec/enc_hook settings as well ?
+        # -[ ] see `._codec.mk_codec()` inputs
+        #
         orig_dec: MsgDec = self._pld_dec
         limit_dec: MsgDec = mk_dec(
             spec=spec,
@@ -163,7 +147,7 @@ class PldRx(Struct):
     def dec(self) -> msgpack.Decoder:
         return self._pld_dec.dec
 
-    def recv_pld_nowait(
+    def recv_msg_nowait(
         self,
         # TODO: make this `MsgStream` compat as well, see above^
         # ipc_prim: Context|MsgStream,
@@ -174,34 +158,95 @@ class PldRx(Struct):
         hide_tb: bool = False,
         **dec_pld_kwargs,
 
-    ) -> Any|Raw:
+    ) -> tuple[
+        MsgType[PayloadT],
+        PayloadT,
+    ]:
+        '''
+        Attempt to non-blocking receive a message from the `._rx_chan` and
+        unwrap it's payload delivering the pair to the caller.
+
+        '''
         __tracebackhide__: bool = hide_tb
 
         msg: MsgType = (
             ipc_msg
             or
-
             # sync-rx msg from underlying IPC feeder (mem-)chan
             ipc._rx_chan.receive_nowait()
         )
-        return self.decode_pld(
+        pld: PayloadT = self.decode_pld(
             msg,
             ipc=ipc,
             expect_msg=expect_msg,
             hide_tb=hide_tb,
             **dec_pld_kwargs,
         )
+        return (
+            msg,
+            pld,
+        )
+
+    async def recv_msg(
+        self,
+        ipc: Context|MsgStream,
+        expect_msg: MsgType,
+
+        # NOTE: ONLY for handling `Stop`-msgs that arrive during
+        # a call to `drain_to_final_msg()` above!
+        passthrough_non_pld_msgs: bool = True,
+        hide_tb: bool = True,
+
+        **decode_pld_kwargs,
+
+    ) -> tuple[MsgType, PayloadT]:
+        '''
+        Retrieve the next avail IPC msg, decode its payload, and
+        return the (msg, pld) pair.
+
+        '''
+        __tracebackhide__: bool = hide_tb
+        msg: MsgType = await ipc._rx_chan.receive()
+        match msg:
+            case Return()|Error():
+                log.runtime(
+                    f'Rxed final outcome msg\n'
+                    f'{msg}\n'
+                )
+            case Stop():
+                log.runtime(
+                    f'Rxed stream stopped msg\n'
+                    f'{msg}\n'
+                )
+                if passthrough_non_pld_msgs:
+                    return msg, None
+
+        # TODO: is there some way we can inject the decoded
+        # payload into an existing output buffer for the original
+        # msg instance?
+        pld: PayloadT = self.decode_pld(
+            msg,
+            ipc=ipc,
+            expect_msg=expect_msg,
+            hide_tb=hide_tb,
+
+            **decode_pld_kwargs,
+        )
+        return (
+            msg,
+            pld,
+        )
 
     async def recv_pld(
         self,
         ipc: Context|MsgStream,
-        ipc_msg: MsgType|None = None,
+        ipc_msg: MsgType[PayloadT]|None = None,
         expect_msg: Type[MsgType]|None = None,
         hide_tb: bool = True,
 
         **dec_pld_kwargs,
 
-    ) -> Any|Raw:
+    ) -> PayloadT:
         '''
         Receive a `MsgType`, then decode and return its `.pld` field.
 
@@ -213,6 +258,13 @@ class PldRx(Struct):
             # async-rx msg from underlying IPC feeder (mem-)chan
             await ipc._rx_chan.receive()
         )
+        if (
+            type(msg) is Return
+        ):
+            log.info(
+                f'Rxed final result msg\n'
+                f'{msg}\n'
+            )
         return self.decode_pld(
             msg=msg,
             ipc=ipc,
@@ -258,6 +310,9 @@ class PldRx(Struct):
                         f'|_pld={pld!r}\n'
                     )
                     return pld
+                except TypeError as typerr:
+                    __tracebackhide__: bool = False
+                    raise typerr
 
                 # XXX pld-value type failure
                 except ValidationError as valerr:
@@ -374,7 +429,7 @@ class PldRx(Struct):
 
             case _:
                 src_err = InternalError(
-                    'Unknown IPC msg ??\n\n'
+                    'Invalid IPC msg ??\n\n'
                     f'{msg}\n'
                 )
 
@@ -398,45 +453,6 @@ class PldRx(Struct):
             __tracebackhide__: bool = False
             raise
 
-    dec_msg = decode_pld
-
-    async def recv_msg_w_pld(
-        self,
-        ipc: Context|MsgStream,
-        expect_msg: MsgType,
-
-        # NOTE: generally speaking only for handling `Stop`-msgs that
-        # arrive during a call to `drain_to_final_msg()` above!
-        passthrough_non_pld_msgs: bool = True,
-        hide_tb: bool = True,
-        **kwargs,
-
-    ) -> tuple[MsgType, PayloadT]:
-        '''
-        Retrieve the next avail IPC msg, decode it's payload, and return
-        the pair of refs.
-
-        '''
-        __tracebackhide__: bool = hide_tb
-        msg: MsgType = await ipc._rx_chan.receive()
-
-        if passthrough_non_pld_msgs:
-            match msg:
-                case Stop():
-                    return msg, None
-
-        # TODO: is there some way we can inject the decoded
-        # payload into an existing output buffer for the original
-        # msg instance?
-        pld: PayloadT = self.decode_pld(
-            msg,
-            ipc=ipc,
-            expect_msg=expect_msg,
-            hide_tb=hide_tb,
-            **kwargs,
-        )
-        return msg, pld
-
 
 @cm
 def limit_plds(
@@ -452,11 +468,16 @@ def limit_plds(
 
     '''
     __tracebackhide__: bool = True
+    curr_ctx: Context|None = current_ipc_ctx()
+    if curr_ctx is None:
+        raise RuntimeError(
+            'No IPC `Context` is active !?\n'
+            'Did you open `limit_plds()` from outside '
+            'a `Portal.open_context()` scope-block?'
+        )
     try:
-        curr_ctx: Context = current_ipc_ctx()
         rx: PldRx = curr_ctx._pld_rx
         orig_pldec: MsgDec = rx.pld_dec
-
         with rx.limit_plds(
             spec=spec,
             **dec_kwargs,
@@ -466,6 +487,11 @@ def limit_plds(
                 f'{pldec}\n'
             )
             yield pldec
+
+    except BaseException:
+        __tracebackhide__: bool = False
+        raise
+
     finally:
         log.runtime(
             'Reverted to previous payload-decoder\n\n'
@@ -499,7 +525,7 @@ async def maybe_limit_plds(
         yield None
         return
 
-    # sanity on scoping
+    # sanity check on IPC scoping
     curr_ctx: Context = current_ipc_ctx()
     assert ctx is curr_ctx
 
@@ -510,6 +536,8 @@ async def maybe_limit_plds(
     ) as msgdec:
         yield msgdec
 
+    # when the applied spec is unwound/removed, the same IPC-ctx
+    # should still be in scope.
     curr_ctx: Context = current_ipc_ctx()
     assert ctx is curr_ctx
 
@@ -517,28 +545,38 @@ async def maybe_limit_plds(
 async def drain_to_final_msg(
     ctx: Context,
 
-    hide_tb: bool = True,
     msg_limit: int = 6,
+    hide_tb: bool = True,
 
 ) -> tuple[
     Return|None,
     list[MsgType]
 ]:
     '''
-    Drain IPC msgs delivered to the underlying IPC primitive's
-    rx-mem-chan (eg. `Context._rx_chan`) from the runtime in
-    search for a final result or error.
+    Drain IPC msgs delivered to the underlying IPC context's
+    rx-mem-chan (i.e. from `Context._rx_chan`) in search for a final
+    `Return` or `Error` msg.
 
-    The motivation here is to ideally capture errors during ctxc
-    conditions where a canc-request/or local error is sent but the
-    local task also excepts and enters the
-    `Portal.open_context().__aexit__()` block wherein we prefer to
-    capture and raise any remote error or ctxc-ack as part of the
-    `ctx.result()` cleanup and teardown sequence.
+    Deliver the `Return` + preceding drained msgs (`list[MsgType]`)
+    as a pair unless an `Error` is found, in which unpack and raise
+    it.
+
+    The motivation here is to always capture any remote error relayed
+    by the remote peer task during a ctxc condition.
+
+    For eg. a ctxc-request may be sent to the peer as part of the
+    local task's (request for) cancellation but then that same task
+    **also errors** before executing the teardown in the
+    `Portal.open_context().__aexit__()` block. In such error-on-exit
+    cases we want to always capture and raise any delivered remote
+    error (like an expected ctxc-ACK) as part of the final
+    `ctx.wait_for_result()` teardown sequence such that the
+    `Context.outcome` related state always reflect what transpired
+    even after ctx closure and the `.open_context()` block exit.
 
     '''
-    __tracebackhide__: bool = hide_tb
     raise_overrun: bool = not ctx._allow_overruns
+    parent_never_opened_stream: bool = ctx._stream is None
 
     # wait for a final context result by collecting (but
     # basically ignoring) any bi-dir-stream msgs still in transit
@@ -547,13 +585,14 @@ async def drain_to_final_msg(
     result_msg: Return|Error|None = None
     while not (
         ctx.maybe_error
-        and not ctx._final_result_is_set()
+        and
+        not ctx._final_result_is_set()
     ):
         try:
             # receive all msgs, scanning for either a final result
             # or error; the underlying call should never raise any
             # remote error directly!
-            msg, pld = await ctx._pld_rx.recv_msg_w_pld(
+            msg, pld = await ctx._pld_rx.recv_msg(
                 ipc=ctx,
                 expect_msg=Return,
                 raise_error=False,
@@ -572,22 +611,47 @@ async def drain_to_final_msg(
             # |_from tractor.devx._debug import pause
             #   await pause()
 
-
         # NOTE: we get here if the far end was
         # `ContextCancelled` in 2 cases:
         # 1. we requested the cancellation and thus
         #    SHOULD NOT raise that far end error,
         # 2. WE DID NOT REQUEST that cancel and thus
         #    SHOULD RAISE HERE!
-        except trio.Cancelled as taskc:
+        except trio.Cancelled as _taskc:
+            taskc: trio.Cancelled = _taskc
 
+            # report when the cancellation wasn't (ostensibly) due to
+            # RPC operation, some surrounding parent cancel-scope.
+            if not ctx._scope.cancel_called:
+                task: trio.lowlevel.Task = trio.lowlevel.current_task()
+                rent_n: trio.Nursery = task.parent_nursery
+                if (
+                    (local_cs := rent_n.cancel_scope).cancel_called
+                ):
+                    log.cancel(
+                        'RPC-ctx cancelled by local-parent scope during drain!\n\n'
+                        f'c}}>\n'
+                        f' |_{rent_n}\n'
+                        f'   |_.cancel_scope = {local_cs}\n'
+                        f'   |_>c}}\n'
+                        f'      |_{ctx.pformat(indent=" "*9)}'
+                        # ^TODO, some (other) simpler repr here?
+                    )
+                    __tracebackhide__: bool = False
+
+            else:
+                log.cancel(
+                    f'IPC ctx cancelled externally during result drain ?\n'
+                    f'{ctx}'
+                )
             # CASE 2: mask the local cancelled-error(s)
             # only when we are sure the remote error is
             # the source cause of this local task's
             # cancellation.
             ctx.maybe_raise(
-                # TODO: when use this/
-                # from_src_exc=taskc,
+                hide_tb=hide_tb,
+                from_src_exc=taskc,
+                # ?TODO? when *should* we use this?
             )
 
             # CASE 1: we DID request the cancel we simply
@@ -611,17 +675,24 @@ async def drain_to_final_msg(
             case Yield():
                 pre_result_drained.append(msg)
                 if (
-                    (ctx._stream.closed
-                     and (reason := 'stream was already closed')
-                    )
-                    or (ctx.cancel_acked
-                        and (reason := 'ctx cancelled other side')
-                    )
-                    or (ctx._cancel_called
-                        and (reason := 'ctx called `.cancel()`')
-                    )
-                    or (len(pre_result_drained) > msg_limit
-                        and (reason := f'"yield" limit={msg_limit}')
+                    not parent_never_opened_stream
+                    and (
+                        (ctx._stream.closed
+                         and
+                         (reason := 'stream was already closed')
+                        ) or
+                        (ctx.cancel_acked
+                            and
+                            (reason := 'ctx cancelled other side')
+                        )
+                        or (ctx._cancel_called
+                            and
+                            (reason := 'ctx called `.cancel()`')
+                        )
+                        or (len(pre_result_drained) > msg_limit
+                            and
+                            (reason := f'"yield" limit={msg_limit}')
+                        )
                     )
                 ):
                     log.cancel(
@@ -639,7 +710,7 @@ async def drain_to_final_msg(
                 # drain up to the `msg_limit` hoping to get
                 # a final result or error/ctxc.
                 else:
-                    log.warning(
+                    report: str = (
                         'Ignoring "yield" msg during `ctx.result()` drain..\n'
                         f'<= {ctx.chan.uid}\n'
                         f'  |_{ctx._nsf}()\n\n'
@@ -648,6 +719,14 @@ async def drain_to_final_msg(
 
                         f'{pretty_struct.pformat(msg)}\n'
                     )
+                    if parent_never_opened_stream:
+                        report = (
+                            f'IPC ctx never opened stream on {ctx.side!r}-side!\n'
+                            f'\n'
+                            # f'{ctx}\n'
+                        ) + report
+
+                    log.warning(report)
                     continue
 
             # stream terminated, but no result yet..
@@ -659,7 +738,7 @@ async def drain_to_final_msg(
             # Stop()
             case Stop():
                 pre_result_drained.append(msg)
-                log.cancel(
+                log.runtime(  # normal/expected shutdown transaction
                     'Remote stream terminated due to "stop" msg:\n\n'
                     f'{pretty_struct.pformat(msg)}\n'
                 )
@@ -719,13 +798,19 @@ async def drain_to_final_msg(
                 pre_result_drained.append(msg)
                 # It's definitely an internal error if any other
                 # msg type without a`'cid'` field arrives here!
+                report: str = (
+                    f'Invalid or unknown msg type {type(msg)!r}!?\n'
+                )
                 if not msg.cid:
-                    raise InternalError(
-                        'Unexpected cid-missing msg?\n\n'
-                        f'{msg}\n'
+                    report += (
+                        '\nWhich also has no `.cid` field?\n'
                     )
 
-                raise RuntimeError('Unknown msg type: {msg}')
+                raise MessagingError(
+                    report
+                    +
+                    f'\n{msg}\n'
+                )
 
     else:
         log.cancel(
@@ -733,6 +818,7 @@ async def drain_to_final_msg(
             f'{ctx.outcome}\n'
         )
 
+    __tracebackhide__: bool = hide_tb
     return (
         result_msg,
         pre_result_drained,
@@ -758,8 +844,14 @@ def validate_payload_msg(
     __tracebackhide__: bool = hide_tb
     codec: MsgCodec = current_codec()
     msg_bytes: bytes = codec.encode(pld_msg)
+    roundtripped: Started|None = None
     try:
         roundtripped: Started = codec.decode(msg_bytes)
+    except TypeError as typerr:
+        __tracebackhide__: bool = False
+        raise typerr
+
+    try:
         ctx: Context = getattr(ipc, 'ctx', ipc)
         pld: PayloadT = ctx.pld_rx.decode_pld(
             msg=roundtripped,
@@ -784,6 +876,11 @@ def validate_payload_msg(
             )
             raise ValidationError(complaint)
 
+    # usually due to `.decode()` input type
+    except TypeError as typerr:
+        __tracebackhide__: bool = False
+        raise typerr
+
     # raise any msg type error NO MATTER WHAT!
     except ValidationError as verr:
         try:
@@ -794,9 +891,13 @@ def validate_payload_msg(
                 verb_header='Trying to send ',
                 is_invalid_payload=True,
             )
-        except BaseException:
+        except BaseException as _be:
+            if not roundtripped:
+                raise verr
+
+            be = _be
             __tracebackhide__: bool = False
-            raise
+            raise be
 
         if not raise_mte:
             return mte

tractor/msg/pretty_struct.py

@@ -30,10 +30,13 @@ from msgspec import (
     Struct as _Struct,
     structs,
 )
-from pprint import (
-    saferepr,
-)
+# from pprint import (
+#     saferepr,
+# )
 
+from tractor.log import get_logger
+
+log = get_logger()
 # TODO: auto-gen type sig for input func both for
 # type-msgs and logging of RPC tasks?
 # taken and modified from:
@@ -72,8 +75,8 @@ class DiffDump(UserList):
         for k, left, right in self:
             repstr += (
                 f'({k},\n'
-                f'\t{repr(left)},\n'
-                f'\t{repr(right)},\n'
+                f' |_{repr(left)},\n'
+                f' |_{repr(right)},\n'
                 ')\n'
             )
         repstr += ']\n'
@@ -141,9 +144,22 @@ def pformat(
                 field_indent=indent + field_indent,
             )
 
-        else:  # the `pprint` recursion-safe format:
+        else:
+            val_str: str = repr(v)
+
+            # XXX LOL, below just seems to be f#$%in causing
+            # recursion errs..
+            #
+            # the `pprint` recursion-safe format:
             # https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
-            val_str: str = saferepr(v)
+            # try:
+            #     val_str: str = saferepr(v)
+            # except Exception:
+            #     log.exception(
+            #         'Failed to `saferepr({type(struct)})` !?\n'
+            #     )
+                # raise
+                # return _Struct.__repr__(struct)
 
         # TODO: LOLOL use `textwrap.indent()` instead dawwwwwg!
         obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
@@ -194,12 +210,22 @@ class Struct(
         return sin_props
 
     pformat = pformat
+
+    def __repr__(self) -> str:
+        try:
+            return pformat(self)
+        except Exception:
+            log.exception(
+                f'Failed to `pformat({type(self)})` !?\n'
+            )
+            return _Struct.__repr__(self)
+
+    # __repr__ = pformat
     # __str__ = __repr__ = pformat
     # TODO: use a pprint.PrettyPrinter instance around ONLY rendering
     # inside a known tty?
     # def __repr__(self) -> str:
     #     ...
-    __repr__ = pformat
 
     def copy(
         self,
@@ -250,13 +276,15 @@ class Struct(
                 fi.type(getattr(self, fi.name)),
             )
 
+    # TODO: make a mod func instead and just point to it here for
+    # method impl?
     def __sub__(
         self,
         other: Struct,
 
     ) -> DiffDump[tuple[str, Any, Any]]:
         '''
-        Compare fields/items key-wise and return a ``DiffDump``
+        Compare fields/items key-wise and return a `DiffDump`
         for easy visual REPL comparison B)
 
         '''
@@ -273,3 +301,42 @@ class Struct(
                 ))
 
         return diffs
+
+    @classmethod
+    def fields_diff(
+        cls,
+        other: dict|Struct,
+
+    ) -> DiffDump[tuple[str, Any, Any]]:
+        '''
+        Very similar to `PrettyStruct.__sub__()` except accepts an
+        input `other: dict` (presumably that would normally be called
+        like `Struct(**other)`) which returns a `DiffDump` of the
+        fields of the struct and the `dict`'s fields.
+
+        '''
+        nullish = object()
+        consumed: dict = other.copy()
+        diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
+        for fi in structs.fields(cls):
+            field_name: str = fi.name
+            # ours: Any = getattr(self, field_name)
+            theirs: Any = consumed.pop(field_name, nullish)
+            if theirs is nullish:
+                diffs.append((
+                    field_name,
+                    f'{fi.type!r}',
+                    'NOT-DEFINED in `other: dict`',
+                ))
+
+        # when there are lingering fields in `other` that this struct
+        # DOES NOT define we also append those.
+        if consumed:
+            for k, v in consumed.items():
+                diffs.append((
+                    k,
+                    f'NOT-DEFINED for `{cls.__name__}`',
+                    f'`other: dict` has value = {v!r}',
+                ))
+
+        return diffs

tractor/msg/types.py

@@ -599,15 +599,15 @@ def mk_msg_spec(
         Msg[payload_type_union],
         Generic[PayloadT],
     )
-    defstruct_bases: tuple = (
-        Msg, # [payload_type_union],
-        # Generic[PayloadT],
-        # ^-XXX-^: not allowed? lul..
-    )
+    # defstruct_bases: tuple = (
+    #     Msg, # [payload_type_union],
+    #     # Generic[PayloadT],
+    #     # ^-XXX-^: not allowed? lul..
+    # )
     ipc_msg_types: list[Msg] = []
 
     idx_msg_types: list[Msg] = []
-    defs_msg_types: list[Msg] = []
+    # defs_msg_types: list[Msg] = []
     nc_msg_types: list[Msg] = []
 
     for msgtype in __msg_types__:
@@ -625,7 +625,7 @@ def mk_msg_spec(
         # TODO: wait why do we need the dynamic version here?
         # XXX ANSWER XXX -> BC INHERITANCE.. don't work w generics..
         #
-        # NOTE previously bc msgtypes WERE NOT inheritting
+        # NOTE previously bc msgtypes WERE NOT inheriting
         # directly the `Generic[PayloadT]` type, the manual method
         # of generic-paraming with `.__class_getitem__()` wasn't
         # working..
@@ -662,38 +662,35 @@ def mk_msg_spec(
 
         # with `msgspec.structs.defstruct`
         # XXX ALSO DOESN'T WORK
-        defstruct_msgtype = defstruct(
-            name=msgtype.__name__,
-            fields=[
-                ('cid', str),
+        # defstruct_msgtype = defstruct(
+        #     name=msgtype.__name__,
+        #     fields=[
+        #         ('cid', str),
 
-                # XXX doesn't seem to work..
-                # ('pld', PayloadT),
-
-                ('pld', payload_type_union),
-            ],
-            bases=defstruct_bases,
-        )
-        defs_msg_types.append(defstruct_msgtype)
+        #         # XXX doesn't seem to work..
+        #         # ('pld', PayloadT),
 
+        #         ('pld', payload_type_union),
+        #     ],
+        #     bases=defstruct_bases,
+        # )
+        # defs_msg_types.append(defstruct_msgtype)
         # assert index_paramed_msg_type == manual_paramed_msg_subtype
-
         # paramed_msg_type = manual_paramed_msg_subtype
-
         # ipc_payload_msgs_type_union |= index_paramed_msg_type
 
     idx_spec: Union[Type[Msg]] = Union[*idx_msg_types]
-    def_spec: Union[Type[Msg]] = Union[*defs_msg_types]
+    # def_spec: Union[Type[Msg]] = Union[*defs_msg_types]
     nc_spec: Union[Type[Msg]] = Union[*nc_msg_types]
 
     specs: dict[str, Union[Type[Msg]]] = {
         'indexed_generics': idx_spec,
-        'defstruct': def_spec,
+        # 'defstruct': def_spec,
         'types_new_class': nc_spec,
     }
     msgtypes_table: dict[str, list[Msg]] = {
         'indexed_generics': idx_msg_types,
-        'defstruct': defs_msg_types,
+        # 'defstruct': defs_msg_types,
         'types_new_class': nc_msg_types,
     }
 

tractor/trionics/__init__.py

@@ -29,3 +29,6 @@ from ._broadcast import (
     BroadcastReceiver as BroadcastReceiver,
     Lagged as Lagged,
 )
+from ._beg import (
+    collapse_eg as collapse_eg,
+)

tractor/trionics/_beg.py

@@ -0,0 +1,58 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+`BaseExceptionGroup` related utils and helpers pertaining to
+first-class-`trio` from a historical perspective B)
+
+'''
+from contextlib import (
+    asynccontextmanager as acm,
+)
+
+
+def maybe_collapse_eg(
+    beg: BaseExceptionGroup,
+) -> BaseException:
+    '''
+    If the input beg can collapse to a single non-eg sub-exception,
+    return it instead.
+
+    '''
+    if len(excs := beg.exceptions) == 1:
+        return excs[0]
+
+    return beg
+
+
+@acm
+async def collapse_eg():
+    '''
+    If `BaseExceptionGroup` raised in the body scope is
+    "collapse-able" (in the same way that
+    `trio.open_nursery(strict_exception_groups=False)` works) then
+    only raise the lone emedded non-eg in in place.
+
+    '''
+    try:
+        yield
+    except* BaseException as beg:
+        if (
+            exc := maybe_collapse_eg(beg)
+        ) is not beg:
+            raise exc
+
+        raise beg

tractor/trionics/_broadcast.py

@@ -15,7 +15,7 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-``tokio`` style broadcast channel.
+`tokio` style broadcast channel.
 https://docs.rs/tokio/1.11.0/tokio/sync/broadcast/index.html
 
 '''
@@ -156,11 +156,12 @@ class BroadcastState(Struct):
 
 class BroadcastReceiver(ReceiveChannel):
     '''
-    A memory receive channel broadcaster which is non-lossy for the
-    fastest consumer.
+    A memory receive channel broadcaster which is non-lossy for
+    the fastest consumer.
 
-    Additional consumer tasks can receive all produced values by registering
-    with ``.subscribe()`` and receiving from the new instance it delivers.
+    Additional consumer tasks can receive all produced values by
+    registering with ``.subscribe()`` and receiving from the new
+    instance it delivers.
 
     '''
     def __init__(
@@ -381,7 +382,7 @@ class BroadcastReceiver(ReceiveChannel):
                         # likely it makes sense to unwind back to the
                         # underlying?
                         # import tractor
-                        # await tractor.breakpoint()
+                        # await tractor.pause()
                         log.warning(
                             f'Only one sub left for {self}?\n'
                             'We can probably unwind from breceiver?'

tractor/trionics/_mngrs.py

@@ -18,8 +18,12 @@
 Async context manager primitives with hard ``trio``-aware semantics
 
 '''
-from contextlib import asynccontextmanager as acm
+from __future__ import annotations
+from contextlib import (
+    asynccontextmanager as acm,
+)
 import inspect
+from types import ModuleType
 from typing import (
     Any,
     AsyncContextManager,
@@ -30,13 +34,16 @@ from typing import (
     Optional,
     Sequence,
     TypeVar,
+    TYPE_CHECKING,
 )
 
 import trio
-
 from tractor._state import current_actor
 from tractor.log import get_logger
 
+if TYPE_CHECKING:
+    from tractor import ActorNursery
+
 
 log = get_logger(__name__)
 
@@ -46,8 +53,12 @@ T = TypeVar("T")
 
 @acm
 async def maybe_open_nursery(
-    nursery: trio.Nursery | None = None,
+    nursery: trio.Nursery|ActorNursery|None = None,
     shield: bool = False,
+    lib: ModuleType = trio,
+
+    **kwargs,  # proxy thru
+
 ) -> AsyncGenerator[trio.Nursery, Any]:
     '''
     Create a new nursery if None provided.
@@ -58,13 +69,12 @@ async def maybe_open_nursery(
     if nursery is not None:
         yield nursery
     else:
-        async with trio.open_nursery() as nursery:
+        async with lib.open_nursery(**kwargs) as nursery:
             nursery.cancel_scope.shield = shield
             yield nursery
 
 
 async def _enter_and_wait(
-
     mngr: AsyncContextManager[T],
     unwrapped: dict[int, T],
     all_entered: trio.Event,
@@ -91,7 +101,6 @@ async def _enter_and_wait(
 
 @acm
 async def gather_contexts(
-
     mngrs: Sequence[AsyncContextManager[T]],
 
 ) -> AsyncGenerator[
@@ -102,15 +111,17 @@ async def gather_contexts(
     None,
 ]:
     '''
-    Concurrently enter a sequence of async context managers, each in
-    a separate ``trio`` task and deliver the unwrapped values in the
-    same order once all managers have entered. On exit all contexts are
-    subsequently and concurrently exited.
+    Concurrently enter a sequence of async context managers (acms),
+    each from a separate `trio` task and deliver the unwrapped
+    `yield`-ed values in the same order once all managers have entered.
 
-    This function is somewhat similar to common usage of
-    ``contextlib.AsyncExitStack.enter_async_context()`` (in a loop) in
-    combo with ``asyncio.gather()`` except the managers are concurrently
-    entered and exited, and cancellation just works.
+    On exit, all acms are subsequently and concurrently exited.
+
+    This function is somewhat similar to a batch of non-blocking
+    calls to `contextlib.AsyncExitStack.enter_async_context()`
+    (inside a loop) *in combo with* a `asyncio.gather()` to get the
+    `.__aenter__()`-ed values, except the managers are both
+    concurrently entered and exited and *cancellation just works*(R).
 
     '''
     seed: int = id(mngrs)
@@ -134,9 +145,14 @@ async def gather_contexts(
             'Use a non-lazy iterator or sequence type intead!'
         )
 
-    async with trio.open_nursery() as n:
+    async with trio.open_nursery(
+        strict_exception_groups=False,
+        # ^XXX^ TODO? soo roll our own then ??
+        # -> since we kinda want the "if only one `.exception` then
+        # just raise that" interface?
+    ) as tn:
         for mngr in mngrs:
-            n.start_soon(
+            tn.start_soon(
                 _enter_and_wait,
                 mngr,
                 unwrapped,
@@ -210,9 +226,10 @@ async def maybe_open_context(
 
 ) -> AsyncIterator[tuple[bool, T]]:
     '''
-    Maybe open a context manager if there is not already a _Cached
-    version for the provided ``key`` for *this* actor. Return the
-    _Cached instance on a _Cache hit.
+    Maybe open an async-context-manager (acm) if there is not already
+    a `_Cached` version for the provided (input) `key` for *this* actor.
+
+    Return the `_Cached` instance on a _Cache hit.
 
     '''
     fid = id(acm_func)
@@ -273,8 +290,13 @@ async def maybe_open_context(
     else:
         _Cache.users += 1
         log.runtime(
-            f'Reusing resource for `_Cache` user {_Cache.users}\n\n'
-            f'{ctx_key!r} -> {yielded!r}\n'
+            f'Re-using cached resource for user {_Cache.users}\n\n'
+            f'{ctx_key!r} -> {type(yielded)}\n'
+
+            # TODO: make this work with values but without
+            # `msgspec.Struct` causing frickin crashes on field-type
+            # lookups..
+            # f'{ctx_key!r} -> {yielded!r}\n'
         )
         lock.release()
         yield True, yielded

uv.lock

@@ -0,0 +1,466 @@
+version = 1
+revision = 1
+requires-python = ">=3.11"
+
+[[package]]
+name = "attrs"
+version = "24.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/48/c8/6260f8ccc11f0917360fc0da435c5c9c7504e3db174d5a12a1494887b045/attrs-24.3.0.tar.gz", hash = "sha256:8f5c07333d543103541ba7be0e2ce16eeee8130cb0b3f9238ab904ce1e85baff", size = 805984 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/89/aa/ab0f7891a01eeb2d2e338ae8fecbe57fcebea1a24dbb64d45801bfab481d/attrs-24.3.0-py3-none-any.whl", hash = "sha256:ac96cd038792094f438ad1f6ff80837353805ac950cd2aa0e0625ef19850c308", size = 63397 },
+]
+
+[[package]]
+name = "cffi"
+version = "1.17.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pycparser" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fc/97/c783634659c2920c3fc70419e3af40972dbaf758daa229a7d6ea6135c90d/cffi-1.17.1.tar.gz", hash = "sha256:1c39c6016c32bc48dd54561950ebd6836e1670f2ae46128f67cf49e789c52824", size = 516621 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/34/33/e1b8a1ba29025adbdcda5fb3a36f94c03d771c1b7b12f726ff7fef2ebe36/cffi-1.17.1-cp311-cp311-win32.whl", hash = "sha256:85a950a4ac9c359340d5963966e3e0a94a676bd6245a4b55bc43949eee26a655", size = 171727 },
+    { url = "https://files.pythonhosted.org/packages/3d/97/50228be003bb2802627d28ec0627837ac0bf35c90cf769812056f235b2d1/cffi-1.17.1-cp311-cp311-win_amd64.whl", hash = "sha256:caaf0640ef5f5517f49bc275eca1406b0ffa6aa184892812030f04c2abf589a0", size = 181400 },
+    { url = "https://files.pythonhosted.org/packages/86/c5/28b2d6f799ec0bdecf44dced2ec5ed43e0eb63097b0f58c293583b406582/cffi-1.17.1-cp312-cp312-win32.whl", hash = "sha256:a08d7e755f8ed21095a310a693525137cfe756ce62d066e53f502a83dc550f65", size = 172448 },
+    { url = "https://files.pythonhosted.org/packages/50/b9/db34c4755a7bd1cb2d1603ac3863f22bcecbd1ba29e5ee841a4bc510b294/cffi-1.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:51392eae71afec0d0c8fb1a53b204dbb3bcabcb3c9b807eedf3e1e6ccf2de903", size = 181976 },
+    { url = "https://files.pythonhosted.org/packages/bf/ee/f94057fa6426481d663b88637a9a10e859e492c73d0384514a17d78ee205/cffi-1.17.1-cp313-cp313-win32.whl", hash = "sha256:e03eab0a8677fa80d646b5ddece1cbeaf556c313dcfac435ba11f107ba117b5d", size = 172475 },
+    { url = "https://files.pythonhosted.org/packages/7c/fc/6a8cb64e5f0324877d503c854da15d76c1e50eb722e320b15345c4d0c6de/cffi-1.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:f6a16c31041f09ead72d69f583767292f750d24913dadacf5756b966aacb3f1a", size = 182009 },
+]
+
+[[package]]
+name = "colorama"
+version = "0.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335 },
+]
+
+[[package]]
+name = "colorlog"
+version = "6.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d3/7a/359f4d5df2353f26172b3cc39ea32daa39af8de522205f512f458923e677/colorlog-6.9.0.tar.gz", hash = "sha256:bfba54a1b93b94f54e1f4fe48395725a3d92fd2a4af702f6bd70946bdc0c6ac2", size = 16624 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e3/51/9b208e85196941db2f0654ad0357ca6388ab3ed67efdbfc799f35d1f83aa/colorlog-6.9.0-py3-none-any.whl", hash = "sha256:5906e71acd67cb07a71e779c47c4bcb45fb8c2993eebe9e5adcd6a6f1b283eff", size = 11424 },
+]
+
+[[package]]
+name = "greenback"
+version = "1.2.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "greenlet" },
+    { name = "outcome" },
+    { name = "sniffio" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/dc/c1/ab3a42c0f3ed56df9cd33de1539b3198d98c6ccbaf88a73d6be0b72d85e0/greenback-1.2.1.tar.gz", hash = "sha256:de3ca656885c03b96dab36079f3de74bb5ba061da9bfe3bb69dccc866ef95ea3", size = 42597 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/71/d0/b8dc79d5ecfffacad9c844b6ae76b9c6259935796d3c561deccbf8fa421d/greenback-1.2.1-py3-none-any.whl", hash = "sha256:98768edbbe4340091a9730cf64a683fcbaa3f2cb81e4ac41d7ed28d3b6f74b79", size = 28062 },
+]
+
+[[package]]
+name = "greenlet"
+version = "3.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2f/ff/df5fede753cc10f6a5be0931204ea30c35fa2f2ea7a35b25bdaf4fe40e46/greenlet-3.1.1.tar.gz", hash = "sha256:4ce3ac6cdb6adf7946475d7ef31777c26d94bccc377e070a7986bd2d5c515467", size = 186022 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/28/62/1c2665558618553c42922ed47a4e6d6527e2fa3516a8256c2f431c5d0441/greenlet-3.1.1-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:e4d333e558953648ca09d64f13e6d8f0523fa705f51cae3f03b5983489958c70", size = 272479 },
+    { url = "https://files.pythonhosted.org/packages/76/9d/421e2d5f07285b6e4e3a676b016ca781f63cfe4a0cd8eaecf3fd6f7a71ae/greenlet-3.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:09fc016b73c94e98e29af67ab7b9a879c307c6731a2c9da0db5a7d9b7edd1159", size = 640404 },
+    { url = "https://files.pythonhosted.org/packages/e5/de/6e05f5c59262a584e502dd3d261bbdd2c97ab5416cc9c0b91ea38932a901/greenlet-3.1.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d5e975ca70269d66d17dd995dafc06f1b06e8cb1ec1e9ed54c1d1e4a7c4cf26e", size = 652813 },
+    { url = "https://files.pythonhosted.org/packages/49/93/d5f93c84241acdea15a8fd329362c2c71c79e1a507c3f142a5d67ea435ae/greenlet-3.1.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3b2813dc3de8c1ee3f924e4d4227999285fd335d1bcc0d2be6dc3f1f6a318ec1", size = 648517 },
+    { url = "https://files.pythonhosted.org/packages/15/85/72f77fc02d00470c86a5c982b8daafdf65d38aefbbe441cebff3bf7037fc/greenlet-3.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e347b3bfcf985a05e8c0b7d462ba6f15b1ee1c909e2dcad795e49e91b152c383", size = 647831 },
+    { url = "https://files.pythonhosted.org/packages/f7/4b/1c9695aa24f808e156c8f4813f685d975ca73c000c2a5056c514c64980f6/greenlet-3.1.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9e8f8c9cb53cdac7ba9793c276acd90168f416b9ce36799b9b885790f8ad6c0a", size = 602413 },
+    { url = "https://files.pythonhosted.org/packages/76/70/ad6e5b31ef330f03b12559d19fda2606a522d3849cde46b24f223d6d1619/greenlet-3.1.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:62ee94988d6b4722ce0028644418d93a52429e977d742ca2ccbe1c4f4a792511", size = 1129619 },
+    { url = "https://files.pythonhosted.org/packages/f4/fb/201e1b932e584066e0f0658b538e73c459b34d44b4bd4034f682423bc801/greenlet-3.1.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:1776fd7f989fc6b8d8c8cb8da1f6b82c5814957264d1f6cf818d475ec2bf6395", size = 1155198 },
+    { url = "https://files.pythonhosted.org/packages/12/da/b9ed5e310bb8b89661b80cbcd4db5a067903bbcd7fc854923f5ebb4144f0/greenlet-3.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:48ca08c771c268a768087b408658e216133aecd835c0ded47ce955381105ba39", size = 298930 },
+    { url = "https://files.pythonhosted.org/packages/7d/ec/bad1ac26764d26aa1353216fcbfa4670050f66d445448aafa227f8b16e80/greenlet-3.1.1-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:4afe7ea89de619adc868e087b4d2359282058479d7cfb94970adf4b55284574d", size = 274260 },
+    { url = "https://files.pythonhosted.org/packages/66/d4/c8c04958870f482459ab5956c2942c4ec35cac7fe245527f1039837c17a9/greenlet-3.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f406b22b7c9a9b4f8aa9d2ab13d6ae0ac3e85c9a809bd590ad53fed2bf70dc79", size = 649064 },
+    { url = "https://files.pythonhosted.org/packages/51/41/467b12a8c7c1303d20abcca145db2be4e6cd50a951fa30af48b6ec607581/greenlet-3.1.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c3a701fe5a9695b238503ce5bbe8218e03c3bcccf7e204e455e7462d770268aa", size = 663420 },
+    { url = "https://files.pythonhosted.org/packages/27/8f/2a93cd9b1e7107d5c7b3b7816eeadcac2ebcaf6d6513df9abaf0334777f6/greenlet-3.1.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2846930c65b47d70b9d178e89c7e1a69c95c1f68ea5aa0a58646b7a96df12441", size = 658035 },
+    { url = "https://files.pythonhosted.org/packages/57/5c/7c6f50cb12be092e1dccb2599be5a942c3416dbcfb76efcf54b3f8be4d8d/greenlet-3.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:99cfaa2110534e2cf3ba31a7abcac9d328d1d9f1b95beede58294a60348fba36", size = 660105 },
+    { url = "https://files.pythonhosted.org/packages/f1/66/033e58a50fd9ec9df00a8671c74f1f3a320564c6415a4ed82a1c651654ba/greenlet-3.1.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1443279c19fca463fc33e65ef2a935a5b09bb90f978beab37729e1c3c6c25fe9", size = 613077 },
+    { url = "https://files.pythonhosted.org/packages/19/c5/36384a06f748044d06bdd8776e231fadf92fc896bd12cb1c9f5a1bda9578/greenlet-3.1.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b7cede291382a78f7bb5f04a529cb18e068dd29e0fb27376074b6d0317bf4dd0", size = 1135975 },
+    { url = "https://files.pythonhosted.org/packages/38/f9/c0a0eb61bdf808d23266ecf1d63309f0e1471f284300ce6dac0ae1231881/greenlet-3.1.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:23f20bb60ae298d7d8656c6ec6db134bca379ecefadb0b19ce6f19d1f232a942", size = 1163955 },
+    { url = "https://files.pythonhosted.org/packages/43/21/a5d9df1d21514883333fc86584c07c2b49ba7c602e670b174bd73cfc9c7f/greenlet-3.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:7124e16b4c55d417577c2077be379514321916d5790fa287c9ed6f23bd2ffd01", size = 299655 },
+    { url = "https://files.pythonhosted.org/packages/f3/57/0db4940cd7bb461365ca8d6fd53e68254c9dbbcc2b452e69d0d41f10a85e/greenlet-3.1.1-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:05175c27cb459dcfc05d026c4232f9de8913ed006d42713cb8a5137bd49375f1", size = 272990 },
+    { url = "https://files.pythonhosted.org/packages/1c/ec/423d113c9f74e5e402e175b157203e9102feeb7088cee844d735b28ef963/greenlet-3.1.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:935e943ec47c4afab8965954bf49bfa639c05d4ccf9ef6e924188f762145c0ff", size = 649175 },
+    { url = "https://files.pythonhosted.org/packages/a9/46/ddbd2db9ff209186b7b7c621d1432e2f21714adc988703dbdd0e65155c77/greenlet-3.1.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:667a9706c970cb552ede35aee17339a18e8f2a87a51fba2ed39ceeeb1004798a", size = 663425 },
+    { url = "https://files.pythonhosted.org/packages/bc/f9/9c82d6b2b04aa37e38e74f0c429aece5eeb02bab6e3b98e7db89b23d94c6/greenlet-3.1.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b8a678974d1f3aa55f6cc34dc480169d58f2e6d8958895d68845fa4ab566509e", size = 657736 },
+    { url = "https://files.pythonhosted.org/packages/d9/42/b87bc2a81e3a62c3de2b0d550bf91a86939442b7ff85abb94eec3fc0e6aa/greenlet-3.1.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efc0f674aa41b92da8c49e0346318c6075d734994c3c4e4430b1c3f853e498e4", size = 660347 },
+    { url = "https://files.pythonhosted.org/packages/37/fa/71599c3fd06336cdc3eac52e6871cfebab4d9d70674a9a9e7a482c318e99/greenlet-3.1.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0153404a4bb921f0ff1abeb5ce8a5131da56b953eda6e14b88dc6bbc04d2049e", size = 615583 },
+    { url = "https://files.pythonhosted.org/packages/4e/96/e9ef85de031703ee7a4483489b40cf307f93c1824a02e903106f2ea315fe/greenlet-3.1.1-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:275f72decf9932639c1c6dd1013a1bc266438eb32710016a1c742df5da6e60a1", size = 1133039 },
+    { url = "https://files.pythonhosted.org/packages/87/76/b2b6362accd69f2d1889db61a18c94bc743e961e3cab344c2effaa4b4a25/greenlet-3.1.1-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:c4aab7f6381f38a4b42f269057aee279ab0fc7bf2e929e3d4abfae97b682a12c", size = 1160716 },
+    { url = "https://files.pythonhosted.org/packages/1f/1b/54336d876186920e185066d8c3024ad55f21d7cc3683c856127ddb7b13ce/greenlet-3.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:b42703b1cf69f2aa1df7d1030b9d77d3e584a70755674d60e710f0af570f3761", size = 299490 },
+    { url = "https://files.pythonhosted.org/packages/5f/17/bea55bf36990e1638a2af5ba10c1640273ef20f627962cf97107f1e5d637/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f1695e76146579f8c06c1509c7ce4dfe0706f49c6831a817ac04eebb2fd02011", size = 643731 },
+    { url = "https://files.pythonhosted.org/packages/78/d2/aa3d2157f9ab742a08e0fd8f77d4699f37c22adfbfeb0c610a186b5f75e0/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7876452af029456b3f3549b696bb36a06db7c90747740c5302f74a9e9fa14b13", size = 649304 },
+    { url = "https://files.pythonhosted.org/packages/f1/8e/d0aeffe69e53ccff5a28fa86f07ad1d2d2d6537a9506229431a2a02e2f15/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4ead44c85f8ab905852d3de8d86f6f8baf77109f9da589cb4fa142bd3b57b475", size = 646537 },
+    { url = "https://files.pythonhosted.org/packages/05/79/e15408220bbb989469c8871062c97c6c9136770657ba779711b90870d867/greenlet-3.1.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8320f64b777d00dd7ccdade271eaf0cad6636343293a25074cc5566160e4de7b", size = 642506 },
+    { url = "https://files.pythonhosted.org/packages/18/87/470e01a940307796f1d25f8167b551a968540fbe0551c0ebb853cb527dd6/greenlet-3.1.1-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6510bf84a6b643dabba74d3049ead221257603a253d0a9873f55f6a59a65f822", size = 602753 },
+    { url = "https://files.pythonhosted.org/packages/e2/72/576815ba674eddc3c25028238f74d7b8068902b3968cbe456771b166455e/greenlet-3.1.1-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:04b013dc07c96f83134b1e99888e7a79979f1a247e2a9f59697fa14b5862ed01", size = 1122731 },
+    { url = "https://files.pythonhosted.org/packages/ac/38/08cc303ddddc4b3d7c628c3039a61a3aae36c241ed01393d00c2fd663473/greenlet-3.1.1-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:411f015496fec93c1c8cd4e5238da364e1da7a124bcb293f085bf2860c32c6f6", size = 1142112 },
+]
+
+[[package]]
+name = "idna"
+version = "3.10"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f1/70/7703c29685631f5a7590aa73f1f1d3fa9a380e654b86af429e0934a32f7d/idna-3.10.tar.gz", hash = "sha256:12f65c9b470abda6dc35cf8e63cc574b1c52b11df2c86030af0ac09b01b13ea9", size = 190490 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442 },
+]
+
+[[package]]
+name = "iniconfig"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/4b/cbd8e699e64a6f16ca3a8220661b5f83792b3017d0f79807cb8708d33913/iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3", size = 4646 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374", size = 5892 },
+]
+
+[[package]]
+name = "msgspec"
+version = "0.19.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cf/9b/95d8ce458462b8b71b8a70fa94563b2498b89933689f3a7b8911edfae3d7/msgspec-0.19.0.tar.gz", hash = "sha256:604037e7cd475345848116e89c553aa9a233259733ab51986ac924ab1b976f8e", size = 216934 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/24/d4/2ec2567ac30dab072cce3e91fb17803c52f0a37aab6b0c24375d2b20a581/msgspec-0.19.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:aa77046904db764b0462036bc63ef71f02b75b8f72e9c9dd4c447d6da1ed8f8e", size = 187939 },
+    { url = "https://files.pythonhosted.org/packages/2b/c0/18226e4328897f4f19875cb62bb9259fe47e901eade9d9376ab5f251a929/msgspec-0.19.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:047cfa8675eb3bad68722cfe95c60e7afabf84d1bd8938979dd2b92e9e4a9551", size = 182202 },
+    { url = "https://files.pythonhosted.org/packages/81/25/3a4b24d468203d8af90d1d351b77ea3cffb96b29492855cf83078f16bfe4/msgspec-0.19.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e78f46ff39a427e10b4a61614a2777ad69559cc8d603a7c05681f5a595ea98f7", size = 209029 },
+    { url = "https://files.pythonhosted.org/packages/85/2e/db7e189b57901955239f7689b5dcd6ae9458637a9c66747326726c650523/msgspec-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c7adf191e4bd3be0e9231c3b6dc20cf1199ada2af523885efc2ed218eafd011", size = 210682 },
+    { url = "https://files.pythonhosted.org/packages/03/97/7c8895c9074a97052d7e4a1cc1230b7b6e2ca2486714eb12c3f08bb9d284/msgspec-0.19.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:f04cad4385e20be7c7176bb8ae3dca54a08e9756cfc97bcdb4f18560c3042063", size = 214003 },
+    { url = "https://files.pythonhosted.org/packages/61/61/e892997bcaa289559b4d5869f066a8021b79f4bf8e955f831b095f47a4cd/msgspec-0.19.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:45c8fb410670b3b7eb884d44a75589377c341ec1392b778311acdbfa55187716", size = 216833 },
+    { url = "https://files.pythonhosted.org/packages/ce/3d/71b2dffd3a1c743ffe13296ff701ee503feaebc3f04d0e75613b6563c374/msgspec-0.19.0-cp311-cp311-win_amd64.whl", hash = "sha256:70eaef4934b87193a27d802534dc466778ad8d536e296ae2f9334e182ac27b6c", size = 186184 },
+    { url = "https://files.pythonhosted.org/packages/b2/5f/a70c24f075e3e7af2fae5414c7048b0e11389685b7f717bb55ba282a34a7/msgspec-0.19.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f98bd8962ad549c27d63845b50af3f53ec468b6318400c9f1adfe8b092d7b62f", size = 190485 },
+    { url = "https://files.pythonhosted.org/packages/89/b0/1b9763938cfae12acf14b682fcf05c92855974d921a5a985ecc197d1c672/msgspec-0.19.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:43bbb237feab761b815ed9df43b266114203f53596f9b6e6f00ebd79d178cdf2", size = 183910 },
+    { url = "https://files.pythonhosted.org/packages/87/81/0c8c93f0b92c97e326b279795f9c5b956c5a97af28ca0fbb9fd86c83737a/msgspec-0.19.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4cfc033c02c3e0aec52b71710d7f84cb3ca5eb407ab2ad23d75631153fdb1f12", size = 210633 },
+    { url = "https://files.pythonhosted.org/packages/d0/ef/c5422ce8af73928d194a6606f8ae36e93a52fd5e8df5abd366903a5ca8da/msgspec-0.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d911c442571605e17658ca2b416fd8579c5050ac9adc5e00c2cb3126c97f73bc", size = 213594 },
+    { url = "https://files.pythonhosted.org/packages/19/2b/4137bc2ed45660444842d042be2cf5b18aa06efd2cda107cff18253b9653/msgspec-0.19.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:757b501fa57e24896cf40a831442b19a864f56d253679f34f260dcb002524a6c", size = 214053 },
+    { url = "https://files.pythonhosted.org/packages/9d/e6/8ad51bdc806aac1dc501e8fe43f759f9ed7284043d722b53323ea421c360/msgspec-0.19.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5f0f65f29b45e2816d8bded36e6b837a4bf5fb60ec4bc3c625fa2c6da4124537", size = 219081 },
+    { url = "https://files.pythonhosted.org/packages/b1/ef/27dd35a7049c9a4f4211c6cd6a8c9db0a50647546f003a5867827ec45391/msgspec-0.19.0-cp312-cp312-win_amd64.whl", hash = "sha256:067f0de1c33cfa0b6a8206562efdf6be5985b988b53dd244a8e06f993f27c8c0", size = 187467 },
+    { url = "https://files.pythonhosted.org/packages/3c/cb/2842c312bbe618d8fefc8b9cedce37f773cdc8fa453306546dba2c21fd98/msgspec-0.19.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f12d30dd6266557aaaf0aa0f9580a9a8fbeadfa83699c487713e355ec5f0bd86", size = 190498 },
+    { url = "https://files.pythonhosted.org/packages/58/95/c40b01b93465e1a5f3b6c7d91b10fb574818163740cc3acbe722d1e0e7e4/msgspec-0.19.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:82b2c42c1b9ebc89e822e7e13bbe9d17ede0c23c187469fdd9505afd5a481314", size = 183950 },
+    { url = "https://files.pythonhosted.org/packages/e8/f0/5b764e066ce9aba4b70d1db8b087ea66098c7c27d59b9dd8a3532774d48f/msgspec-0.19.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:19746b50be214a54239aab822964f2ac81e38b0055cca94808359d779338c10e", size = 210647 },
+    { url = "https://files.pythonhosted.org/packages/9d/87/bc14f49bc95c4cb0dd0a8c56028a67c014ee7e6818ccdce74a4862af259b/msgspec-0.19.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:60ef4bdb0ec8e4ad62e5a1f95230c08efb1f64f32e6e8dd2ced685bcc73858b5", size = 213563 },
+    { url = "https://files.pythonhosted.org/packages/53/2f/2b1c2b056894fbaa975f68f81e3014bb447516a8b010f1bed3fb0e016ed7/msgspec-0.19.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ac7f7c377c122b649f7545810c6cd1b47586e3aa3059126ce3516ac7ccc6a6a9", size = 213996 },
+    { url = "https://files.pythonhosted.org/packages/aa/5a/4cd408d90d1417e8d2ce6a22b98a6853c1b4d7cb7669153e4424d60087f6/msgspec-0.19.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a5bc1472223a643f5ffb5bf46ccdede7f9795078194f14edd69e3aab7020d327", size = 219087 },
+    { url = "https://files.pythonhosted.org/packages/23/d8/f15b40611c2d5753d1abb0ca0da0c75348daf1252220e5dda2867bd81062/msgspec-0.19.0-cp313-cp313-win_amd64.whl", hash = "sha256:317050bc0f7739cb30d257ff09152ca309bf5a369854bbf1e57dffc310c1f20f", size = 187432 },
+]
+
+[[package]]
+name = "outcome"
+version = "1.3.0.post0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "attrs" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/98/df/77698abfac98571e65ffeb0c1fba8ffd692ab8458d617a0eed7d9a8d38f2/outcome-1.3.0.post0.tar.gz", hash = "sha256:9dcf02e65f2971b80047b377468e72a268e15c0af3cf1238e6ff14f7f91143b8", size = 21060 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/55/8b/5ab7257531a5d830fc8000c476e63c935488d74609b50f9384a643ec0a62/outcome-1.3.0.post0-py2.py3-none-any.whl", hash = "sha256:e771c5ce06d1415e356078d3bdd68523f284b4ce5419828922b6871e65eda82b", size = 10692 },
+]
+
+[[package]]
+name = "packaging"
+version = "24.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/63/68dbb6eb2de9cb10ee4c9c14a0148804425e13c4fb20d61cce69f53106da/packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f", size = 163950 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/88/ef/eb23f262cca3c0c4eb7ab1933c3b1f03d021f2c48f54763065b6f0e321be/packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759", size = 65451 },
+]
+
+[[package]]
+name = "pdbp"
+version = "1.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "colorama", marker = "sys_platform == 'win32'" },
+    { name = "pygments" },
+    { name = "tabcompleter" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/69/13/80da03638f62facbee76312ca9ee5941c017b080f2e4c6919fd4e87e16e3/pdbp-1.6.1.tar.gz", hash = "sha256:f4041642952a05df89664e166d5bd379607a0866ddd753c06874f65552bdf40b", size = 25322 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/29/93/d56fb9ba5569dc29d8263c72e46d21a2fd38741339ebf03f54cf7561828c/pdbp-1.6.1-py3-none-any.whl", hash = "sha256:f10bad2ee044c0e5c168cb0825abfdbdc01c50013e9755df5261b060bdd35c22", size = 21495 },
+]
+
+[[package]]
+name = "pexpect"
+version = "4.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "ptyprocess" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523", size = 63772 },
+]
+
+[[package]]
+name = "pluggy"
+version = "1.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669", size = 20556 },
+]
+
+[[package]]
+name = "prompt-toolkit"
+version = "3.0.50"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "wcwidth" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/e1/bd15cb8ffdcfeeb2bdc215de3c3cffca11408d829e4b8416dcfe71ba8854/prompt_toolkit-3.0.50.tar.gz", hash = "sha256:544748f3860a2623ca5cd6d2795e7a14f3d0e1c3c9728359013f79877fc89bab", size = 429087 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e4/ea/d836f008d33151c7a1f62caf3d8dd782e4d15f6a43897f64480c2b8de2ad/prompt_toolkit-3.0.50-py3-none-any.whl", hash = "sha256:9b6427eb19e479d98acff65196a307c555eb567989e6d88ebbb1b509d9779198", size = 387816 },
+]
+
+[[package]]
+name = "ptyprocess"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/e5/16ff212c1e452235a90aeb09066144d0c5a6a8c0834397e03f5224495c4e/ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220", size = 70762 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35", size = 13993 },
+]
+
+[[package]]
+name = "pycparser"
+version = "2.22"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1d/b2/31537cf4b1ca988837256c910a668b553fceb8f069bedc4b1c826024b52c/pycparser-2.22.tar.gz", hash = "sha256:491c8be9c040f5390f5bf44a5b07752bd07f56edf992381b05c701439eec10f6", size = 172736 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/13/a3/a812df4e2dd5696d1f351d58b8fe16a405b234ad2886a0dab9183fb78109/pycparser-2.22-py3-none-any.whl", hash = "sha256:c3702b6d3dd8c7abc1afa565d7e63d53a1d0bd86cdc24edd75470f4de499cfcc", size = 117552 },
+]
+
+[[package]]
+name = "pygments"
+version = "2.19.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7c/2d/c3338d48ea6cc0feb8446d8e6937e1408088a72a39937982cc6111d17f84/pygments-2.19.1.tar.gz", hash = "sha256:61c16d2a8576dc0649d9f39e089b5f02bcd27fba10d8fb4dcc28173f7a45151f", size = 4968581 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/8a/0b/9fcc47d19c48b59121088dd6da2488a49d5f72dacf8262e2790a1d2c7d15/pygments-2.19.1-py3-none-any.whl", hash = "sha256:9ea1544ad55cecf4b8242fab6dd35a93bbce657034b0611ee383099054ab6d8c", size = 1225293 },
+]
+
+[[package]]
+name = "pyperclip"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/30/23/2f0a3efc4d6a32f3b63cdff36cd398d9701d26cda58e3ab97ac79fb5e60d/pyperclip-1.9.0.tar.gz", hash = "sha256:b7de0142ddc81bfc5c7507eea19da920b92252b548b96186caf94a5e2527d310", size = 20961 }
+
+[[package]]
+name = "pyreadline3"
+version = "3.5.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0f/49/4cea918a08f02817aabae639e3d0ac046fef9f9180518a3ad394e22da148/pyreadline3-3.5.4.tar.gz", hash = "sha256:8d57d53039a1c75adba8e50dd3d992b28143480816187ea5efbd5c78e6c885b7", size = 99839 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/5a/dc/491b7661614ab97483abf2056be1deee4dc2490ecbf7bff9ab5cdbac86e1/pyreadline3-3.5.4-py3-none-any.whl", hash = "sha256:eaf8e6cc3c49bcccf145fc6067ba8643d1df34d604a1ec0eccbf7a18e6d3fae6", size = 83178 },
+]
+
+[[package]]
+name = "pytest"
+version = "8.3.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "colorama", marker = "sys_platform == 'win32'" },
+    { name = "iniconfig" },
+    { name = "packaging" },
+    { name = "pluggy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/30/3d/64ad57c803f1fa1e963a7946b6e0fea4a70df53c1a7fed304586539c2bac/pytest-8.3.5-py3-none-any.whl", hash = "sha256:c69214aa47deac29fad6c2a4f590b9c4a9fdb16a403176fe154b79c0b4d4d820", size = 343634 },
+]
+
+[[package]]
+name = "sniffio"
+version = "1.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235 },
+]
+
+[[package]]
+name = "sortedcontainers"
+version = "2.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e8/c4/ba2f8066cceb6f23394729afe52f3bf7adec04bf9ed2c820b39e19299111/sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88", size = 30594 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0", size = 29575 },
+]
+
+[[package]]
+name = "stackscope"
+version = "0.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4a/fc/20dbb993353f31230138f3c63f3f0c881d1853e70d7a30cd68d2ba4cf1e2/stackscope-0.2.2.tar.gz", hash = "sha256:f508c93eb4861ada466dd3ff613ca203962ceb7587ad013759f15394e6a4e619", size = 90479 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/f1/5f/0a674fcafa03528089badb46419413f342537b5b57d2fefc9900fb8ee4e4/stackscope-0.2.2-py3-none-any.whl", hash = "sha256:c199b0cda738d39c993ee04eb01961b06b7e9aeb43ebf9fd6226cdd72ea9faf6", size = 80807 },
+]
+
+[[package]]
+name = "tabcompleter"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "pyreadline3", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/73/1a/ed3544579628c5709bae6fae2255e94c6982a9ff77d42d8ba59fd2f3b21a/tabcompleter-1.4.0.tar.gz", hash = "sha256:7562a9938e62f8e7c3be612c3ac4e14c5ec4307b58ba9031c148260e866e8814", size = 10431 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/65/44/bb509c3d2c0b5a87e7a5af1d5917a402a32ff026f777a6d7cb6990746cbb/tabcompleter-1.4.0-py3-none-any.whl", hash = "sha256:d744aa735b49c0a6cc2fb8fcd40077fec47425e4388301010b14e6ce3311368b", size = 6725 },
+]
+
+[[package]]
+name = "tractor"
+version = "0.1.0a6.dev0"
+source = { editable = "." }
+dependencies = [
+    { name = "colorlog" },
+    { name = "msgspec" },
+    { name = "pdbp" },
+    { name = "tricycle" },
+    { name = "trio" },
+    { name = "wrapt" },
+]
+
+[package.dev-dependencies]
+dev = [
+    { name = "greenback" },
+    { name = "pexpect" },
+    { name = "prompt-toolkit" },
+    { name = "pyperclip" },
+    { name = "pytest" },
+    { name = "stackscope" },
+    { name = "xonsh" },
+]
+
+[package.metadata]
+requires-dist = [
+    { name = "colorlog", specifier = ">=6.8.2,<7" },
+    { name = "msgspec", specifier = ">=0.19.0" },
+    { name = "pdbp", specifier = ">=1.6,<2" },
+    { name = "tricycle", specifier = ">=0.4.1,<0.5" },
+    { name = "trio", specifier = ">0.27" },
+    { name = "wrapt", specifier = ">=1.16.0,<2" },
+]
+
+[package.metadata.requires-dev]
+dev = [
+    { name = "greenback", specifier = ">=1.2.1,<2" },
+    { name = "pexpect", specifier = ">=4.9.0,<5" },
+    { name = "prompt-toolkit", specifier = ">=3.0.50" },
+    { name = "pyperclip", specifier = ">=1.9.0" },
+    { name = "pytest", specifier = ">=8.3.5" },
+    { name = "stackscope", specifier = ">=0.2.2,<0.3" },
+    { name = "xonsh", specifier = ">=0.19.2" },
+]
+
+[[package]]
+name = "tricycle"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "trio" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f8/8e/fdd7bc467b40eedd0a5f2ed36b0d692c6e6f2473be00c8160e2e9f53adc1/tricycle-0.4.1.tar.gz", hash = "sha256:f56edb4b3e1bed3e2552b1b499b24a2dab47741e92e9b4d806acc5c35c9e6066", size = 41551 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/d7/c6/7cc05d60e21c683df99167db071ce5d848f5063c2a63971a8443466f603e/tricycle-0.4.1-py3-none-any.whl", hash = "sha256:67900995a73e7445e2c70250cdca04a778d9c3923dd960a97ad4569085e0fb3f", size = 35316 },
+]
+
+[[package]]
+name = "trio"
+version = "0.29.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "attrs" },
+    { name = "cffi", marker = "implementation_name != 'pypy' and os_name == 'nt'" },
+    { name = "idna" },
+    { name = "outcome" },
+    { name = "sniffio" },
+    { name = "sortedcontainers" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/47/f62e62a1a6f37909aed0bf8f5d5411e06fa03846cfcb64540cd1180ccc9f/trio-0.29.0.tar.gz", hash = "sha256:ea0d3967159fc130acb6939a0be0e558e364fee26b5deeecc893a6b08c361bdf", size = 588952 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/c9/55/c4d9bea8b3d7937901958f65124123512419ab0eb73695e5f382521abbfb/trio-0.29.0-py3-none-any.whl", hash = "sha256:d8c463f1a9cc776ff63e331aba44c125f423a5a13c684307e828d930e625ba66", size = 492920 },
+]
+
+[[package]]
+name = "wcwidth"
+version = "0.2.13"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6c/63/53559446a878410fc5a5974feb13d31d78d752eb18aeba59c7fef1af7598/wcwidth-0.2.13.tar.gz", hash = "sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5", size = 101301 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/fd/84/fd2ba7aafacbad3c4201d395674fc6348826569da3c0937e75505ead3528/wcwidth-0.2.13-py2.py3-none-any.whl", hash = "sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859", size = 34166 },
+]
+
+[[package]]
+name = "wrapt"
+version = "1.17.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/fc/e91cc220803d7bc4db93fb02facd8461c37364151b8494762cc88b0fbcef/wrapt-1.17.2.tar.gz", hash = "sha256:41388e9d4d1522446fe79d3213196bd9e3b301a336965b9e27ca2788ebd122f3", size = 55531 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/cd/f7/a2aab2cbc7a665efab072344a8949a71081eed1d2f451f7f7d2b966594a2/wrapt-1.17.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ff04ef6eec3eee8a5efef2401495967a916feaa353643defcc03fc74fe213b58", size = 53308 },
+    { url = "https://files.pythonhosted.org/packages/50/ff/149aba8365fdacef52b31a258c4dc1c57c79759c335eff0b3316a2664a64/wrapt-1.17.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4db983e7bca53819efdbd64590ee96c9213894272c776966ca6306b73e4affda", size = 38488 },
+    { url = "https://files.pythonhosted.org/packages/65/46/5a917ce85b5c3b490d35c02bf71aedaa9f2f63f2d15d9949cc4ba56e8ba9/wrapt-1.17.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9abc77a4ce4c6f2a3168ff34b1da9b0f311a8f1cfd694ec96b0603dff1c79438", size = 38776 },
+    { url = "https://files.pythonhosted.org/packages/ca/74/336c918d2915a4943501c77566db41d1bd6e9f4dbc317f356b9a244dfe83/wrapt-1.17.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0b929ac182f5ace000d459c59c2c9c33047e20e935f8e39371fa6e3b85d56f4a", size = 83776 },
+    { url = "https://files.pythonhosted.org/packages/09/99/c0c844a5ccde0fe5761d4305485297f91d67cf2a1a824c5f282e661ec7ff/wrapt-1.17.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f09b286faeff3c750a879d336fb6d8713206fc97af3adc14def0cdd349df6000", size = 75420 },
+    { url = "https://files.pythonhosted.org/packages/b4/b0/9fc566b0fe08b282c850063591a756057c3247b2362b9286429ec5bf1721/wrapt-1.17.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1a7ed2d9d039bd41e889f6fb9364554052ca21ce823580f6a07c4ec245c1f5d6", size = 83199 },
+    { url = "https://files.pythonhosted.org/packages/9d/4b/71996e62d543b0a0bd95dda485219856def3347e3e9380cc0d6cf10cfb2f/wrapt-1.17.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:129a150f5c445165ff941fc02ee27df65940fcb8a22a61828b1853c98763a64b", size = 82307 },
+    { url = "https://files.pythonhosted.org/packages/39/35/0282c0d8789c0dc9bcc738911776c762a701f95cfe113fb8f0b40e45c2b9/wrapt-1.17.2-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:1fb5699e4464afe5c7e65fa51d4f99e0b2eadcc176e4aa33600a3df7801d6662", size = 75025 },
+    { url = "https://files.pythonhosted.org/packages/4f/6d/90c9fd2c3c6fee181feecb620d95105370198b6b98a0770cba090441a828/wrapt-1.17.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9a2bce789a5ea90e51a02dfcc39e31b7f1e662bc3317979aa7e5538e3a034f72", size = 81879 },
+    { url = "https://files.pythonhosted.org/packages/8f/fa/9fb6e594f2ce03ef03eddbdb5f4f90acb1452221a5351116c7c4708ac865/wrapt-1.17.2-cp311-cp311-win32.whl", hash = "sha256:4afd5814270fdf6380616b321fd31435a462019d834f83c8611a0ce7484c7317", size = 36419 },
+    { url = "https://files.pythonhosted.org/packages/47/f8/fb1773491a253cbc123c5d5dc15c86041f746ed30416535f2a8df1f4a392/wrapt-1.17.2-cp311-cp311-win_amd64.whl", hash = "sha256:acc130bc0375999da18e3d19e5a86403667ac0c4042a094fefb7eec8ebac7cf3", size = 38773 },
+    { url = "https://files.pythonhosted.org/packages/a1/bd/ab55f849fd1f9a58ed7ea47f5559ff09741b25f00c191231f9f059c83949/wrapt-1.17.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:d5e2439eecc762cd85e7bd37161d4714aa03a33c5ba884e26c81559817ca0925", size = 53799 },
+    { url = "https://files.pythonhosted.org/packages/53/18/75ddc64c3f63988f5a1d7e10fb204ffe5762bc663f8023f18ecaf31a332e/wrapt-1.17.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fc7cb4c1c744f8c05cd5f9438a3caa6ab94ce8344e952d7c45a8ed59dd88392", size = 38821 },
+    { url = "https://files.pythonhosted.org/packages/48/2a/97928387d6ed1c1ebbfd4efc4133a0633546bec8481a2dd5ec961313a1c7/wrapt-1.17.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8fdbdb757d5390f7c675e558fd3186d590973244fab0c5fe63d373ade3e99d40", size = 38919 },
+    { url = "https://files.pythonhosted.org/packages/73/54/3bfe5a1febbbccb7a2f77de47b989c0b85ed3a6a41614b104204a788c20e/wrapt-1.17.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5bb1d0dbf99411f3d871deb6faa9aabb9d4e744d67dcaaa05399af89d847a91d", size = 88721 },
+    { url = "https://files.pythonhosted.org/packages/25/cb/7262bc1b0300b4b64af50c2720ef958c2c1917525238d661c3e9a2b71b7b/wrapt-1.17.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d18a4865f46b8579d44e4fe1e2bcbc6472ad83d98e22a26c963d46e4c125ef0b", size = 80899 },
+    { url = "https://files.pythonhosted.org/packages/2a/5a/04cde32b07a7431d4ed0553a76fdb7a61270e78c5fd5a603e190ac389f14/wrapt-1.17.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc570b5f14a79734437cb7b0500376b6b791153314986074486e0b0fa8d71d98", size = 89222 },
+    { url = "https://files.pythonhosted.org/packages/09/28/2e45a4f4771fcfb109e244d5dbe54259e970362a311b67a965555ba65026/wrapt-1.17.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6d9187b01bebc3875bac9b087948a2bccefe464a7d8f627cf6e48b1bbae30f82", size = 86707 },
+    { url = "https://files.pythonhosted.org/packages/c6/d2/dcb56bf5f32fcd4bd9aacc77b50a539abdd5b6536872413fd3f428b21bed/wrapt-1.17.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:9e8659775f1adf02eb1e6f109751268e493c73716ca5761f8acb695e52a756ae", size = 79685 },
+    { url = "https://files.pythonhosted.org/packages/80/4e/eb8b353e36711347893f502ce91c770b0b0929f8f0bed2670a6856e667a9/wrapt-1.17.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e8b2816ebef96d83657b56306152a93909a83f23994f4b30ad4573b00bd11bb9", size = 87567 },
+    { url = "https://files.pythonhosted.org/packages/17/27/4fe749a54e7fae6e7146f1c7d914d28ef599dacd4416566c055564080fe2/wrapt-1.17.2-cp312-cp312-win32.whl", hash = "sha256:468090021f391fe0056ad3e807e3d9034e0fd01adcd3bdfba977b6fdf4213ea9", size = 36672 },
+    { url = "https://files.pythonhosted.org/packages/15/06/1dbf478ea45c03e78a6a8c4be4fdc3c3bddea5c8de8a93bc971415e47f0f/wrapt-1.17.2-cp312-cp312-win_amd64.whl", hash = "sha256:ec89ed91f2fa8e3f52ae53cd3cf640d6feff92ba90d62236a81e4e563ac0e991", size = 38865 },
+    { url = "https://files.pythonhosted.org/packages/ce/b9/0ffd557a92f3b11d4c5d5e0c5e4ad057bd9eb8586615cdaf901409920b14/wrapt-1.17.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:6ed6ffac43aecfe6d86ec5b74b06a5be33d5bb9243d055141e8cabb12aa08125", size = 53800 },
+    { url = "https://files.pythonhosted.org/packages/c0/ef/8be90a0b7e73c32e550c73cfb2fa09db62234227ece47b0e80a05073b375/wrapt-1.17.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:35621ae4c00e056adb0009f8e86e28eb4a41a4bfa8f9bfa9fca7d343fe94f998", size = 38824 },
+    { url = "https://files.pythonhosted.org/packages/36/89/0aae34c10fe524cce30fe5fc433210376bce94cf74d05b0d68344c8ba46e/wrapt-1.17.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a604bf7a053f8362d27eb9fefd2097f82600b856d5abe996d623babd067b1ab5", size = 38920 },
+    { url = "https://files.pythonhosted.org/packages/3b/24/11c4510de906d77e0cfb5197f1b1445d4fec42c9a39ea853d482698ac681/wrapt-1.17.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5cbabee4f083b6b4cd282f5b817a867cf0b1028c54d445b7ec7cfe6505057cf8", size = 88690 },
+    { url = "https://files.pythonhosted.org/packages/71/d7/cfcf842291267bf455b3e266c0c29dcb675b5540ee8b50ba1699abf3af45/wrapt-1.17.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:49703ce2ddc220df165bd2962f8e03b84c89fee2d65e1c24a7defff6f988f4d6", size = 80861 },
+    { url = "https://files.pythonhosted.org/packages/d5/66/5d973e9f3e7370fd686fb47a9af3319418ed925c27d72ce16b791231576d/wrapt-1.17.2-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8112e52c5822fc4253f3901b676c55ddf288614dc7011634e2719718eaa187dc", size = 89174 },
+    { url = "https://files.pythonhosted.org/packages/a7/d3/8e17bb70f6ae25dabc1aaf990f86824e4fd98ee9cadf197054e068500d27/wrapt-1.17.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9fee687dce376205d9a494e9c121e27183b2a3df18037f89d69bd7b35bcf59e2", size = 86721 },
+    { url = "https://files.pythonhosted.org/packages/6f/54/f170dfb278fe1c30d0ff864513cff526d624ab8de3254b20abb9cffedc24/wrapt-1.17.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:18983c537e04d11cf027fbb60a1e8dfd5190e2b60cc27bc0808e653e7b218d1b", size = 79763 },
+    { url = "https://files.pythonhosted.org/packages/4a/98/de07243751f1c4a9b15c76019250210dd3486ce098c3d80d5f729cba029c/wrapt-1.17.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:703919b1633412ab54bcf920ab388735832fdcb9f9a00ae49387f0fe67dad504", size = 87585 },
+    { url = "https://files.pythonhosted.org/packages/f9/f0/13925f4bd6548013038cdeb11ee2cbd4e37c30f8bfd5db9e5a2a370d6e20/wrapt-1.17.2-cp313-cp313-win32.whl", hash = "sha256:abbb9e76177c35d4e8568e58650aa6926040d6a9f6f03435b7a522bf1c487f9a", size = 36676 },
+    { url = "https://files.pythonhosted.org/packages/bf/ae/743f16ef8c2e3628df3ddfd652b7d4c555d12c84b53f3d8218498f4ade9b/wrapt-1.17.2-cp313-cp313-win_amd64.whl", hash = "sha256:69606d7bb691b50a4240ce6b22ebb319c1cfb164e5f6569835058196e0f3a845", size = 38871 },
+    { url = "https://files.pythonhosted.org/packages/3d/bc/30f903f891a82d402ffb5fda27ec1d621cc97cb74c16fea0b6141f1d4e87/wrapt-1.17.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:4a721d3c943dae44f8e243b380cb645a709ba5bd35d3ad27bc2ed947e9c68192", size = 56312 },
+    { url = "https://files.pythonhosted.org/packages/8a/04/c97273eb491b5f1c918857cd26f314b74fc9b29224521f5b83f872253725/wrapt-1.17.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:766d8bbefcb9e00c3ac3b000d9acc51f1b399513f44d77dfe0eb026ad7c9a19b", size = 40062 },
+    { url = "https://files.pythonhosted.org/packages/4e/ca/3b7afa1eae3a9e7fefe499db9b96813f41828b9fdb016ee836c4c379dadb/wrapt-1.17.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e496a8ce2c256da1eb98bd15803a79bee00fc351f5dfb9ea82594a3f058309e0", size = 40155 },
+    { url = "https://files.pythonhosted.org/packages/89/be/7c1baed43290775cb9030c774bc53c860db140397047cc49aedaf0a15477/wrapt-1.17.2-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:40d615e4fe22f4ad3528448c193b218e077656ca9ccb22ce2cb20db730f8d306", size = 113471 },
+    { url = "https://files.pythonhosted.org/packages/32/98/4ed894cf012b6d6aae5f5cc974006bdeb92f0241775addad3f8cd6ab71c8/wrapt-1.17.2-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a5aaeff38654462bc4b09023918b7f21790efb807f54c000a39d41d69cf552cb", size = 101208 },
+    { url = "https://files.pythonhosted.org/packages/ea/fd/0c30f2301ca94e655e5e057012e83284ce8c545df7661a78d8bfca2fac7a/wrapt-1.17.2-cp313-cp313t-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9a7d15bbd2bc99e92e39f49a04653062ee6085c0e18b3b7512a4f2fe91f2d681", size = 109339 },
+    { url = "https://files.pythonhosted.org/packages/75/56/05d000de894c4cfcb84bcd6b1df6214297b8089a7bd324c21a4765e49b14/wrapt-1.17.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e3890b508a23299083e065f435a492b5435eba6e304a7114d2f919d400888cc6", size = 110232 },
+    { url = "https://files.pythonhosted.org/packages/53/f8/c3f6b2cf9b9277fb0813418e1503e68414cd036b3b099c823379c9575e6d/wrapt-1.17.2-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:8c8b293cd65ad716d13d8dd3624e42e5a19cc2a2f1acc74b30c2c13f15cb61a6", size = 100476 },
+    { url = "https://files.pythonhosted.org/packages/a7/b1/0bb11e29aa5139d90b770ebbfa167267b1fc548d2302c30c8f7572851738/wrapt-1.17.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4c82b8785d98cdd9fed4cac84d765d234ed3251bd6afe34cb7ac523cb93e8b4f", size = 106377 },
+    { url = "https://files.pythonhosted.org/packages/6a/e1/0122853035b40b3f333bbb25f1939fc1045e21dd518f7f0922b60c156f7c/wrapt-1.17.2-cp313-cp313t-win32.whl", hash = "sha256:13e6afb7fe71fe7485a4550a8844cc9ffbe263c0f1a1eea569bc7091d4898555", size = 37986 },
+    { url = "https://files.pythonhosted.org/packages/09/5e/1655cf481e079c1f22d0cabdd4e51733679932718dc23bf2db175f329b76/wrapt-1.17.2-cp313-cp313t-win_amd64.whl", hash = "sha256:eaf675418ed6b3b31c7a989fd007fa7c3be66ce14e5c3b27336383604c9da85c", size = 40750 },
+    { url = "https://files.pythonhosted.org/packages/2d/82/f56956041adef78f849db6b289b282e72b55ab8045a75abad81898c28d19/wrapt-1.17.2-py3-none-any.whl", hash = "sha256:b18f2d1533a71f069c7f82d524a52599053d4c7166e9dd374ae2136b7f40f7c8", size = 23594 },
+]
+
+[[package]]
+name = "xonsh"
+version = "0.19.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/68/4e/56e95a5e607eb3b0da37396f87cde70588efc8ef819ab16f02d5b8378dc4/xonsh-0.19.2.tar.gz", hash = "sha256:cfdd0680d954a2c3aefd6caddcc7143a3d06aa417ed18365a08219bb71b960b0", size = 799960 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/6c/13/281094759df87b23b3c02dc4a16603ab08ea54d7f6acfeb69f3341137c7a/xonsh-0.19.2-py310-none-any.whl", hash = "sha256:ec7f163fd3a4943782aa34069d4e72793328c916a5975949dbec8536cbfc089b", size = 642301 },
+    { url = "https://files.pythonhosted.org/packages/29/41/a51e4c3918fe9a293b150cb949b1b8c6d45eb17dfed480dcb76ea43df4e7/xonsh-0.19.2-py311-none-any.whl", hash = "sha256:53c45f7a767901f2f518f9b8dd60fc653e0498e56e89825e1710bb0859985049", size = 642286 },
+    { url = "https://files.pythonhosted.org/packages/0a/93/9a77b731f492fac27c577dea2afb5a2bcc2a6a1c79be0c86c95498060270/xonsh-0.19.2-py312-none-any.whl", hash = "sha256:b24c619aa52b59eae4d35c4195dba9b19a2c548fb5c42c6f85f2b8ccb96807b5", size = 642386 },
+    { url = "https://files.pythonhosted.org/packages/be/75/070324769c1ff88d971ce040f4f486339be98e0a365c8dd9991eb654265b/xonsh-0.19.2-py313-none-any.whl", hash = "sha256:c53ef6c19f781fbc399ed1b382b5c2aac2125010679a3b61d643978273c27df0", size = 642873 },
+    { url = "https://files.pythonhosted.org/packages/fa/cb/2c7ccec54f5b0e73fdf7650e8336582ff0347d9001c5ef8271dc00c034fe/xonsh-0.19.2-py39-none-any.whl", hash = "sha256:bcc0225dc3847f1ed2f175dac6122fbcc54cea67d9c2dc2753d9615e2a5ff284", size = 634602 },
+]