.claude/settings.local.json

@@ -0,0 +1,15 @@
+{
+  "permissions": {
+    "allow": [
+      "Write(.claude/*commit_msg*)",
+      "Write(.claude/git_commit_msg_LATEST.md)",
+      "Bash(date *)",
+      "Bash(cp .claude/*)",
+      "Bash(git diff *)",
+      "Bash(git log *)",
+      "Bash(git status)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}

.claude/skills/commit-msg/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: commit-msg
+description: >
+  Generate git commit messages following project style. Use when user
+  wants to create a commit or asks for a commit message.
+argument-hint: "[optional-scope-or-description]"
+disable-model-invocation: true
+allowed-tools:
+  - Bash(git *)
+  - Bash(date *)
+  - Bash(cp *)
+  - Read
+  - Grep
+  - Glob
+  - Write
+---
+
+When generating commit messages, always follow this process:
+
+0. **Check for staged changes**: if `git diff --staged` is
+   empty, STOP and tell the user "nothing is staged!" with
+   a reminder to `git add` before invoking this skill.
+
+1. **Gather context** from the staged diff and recent
+   history:
+
+   - Staged changes: !`git diff --staged --stat`
+   - Recent commits: !`git log --oneline -5`
+
+2. **Analyze the diff**: understand what files changed and
+   the nature of the changes (new feature, bug fix, refactor,
+   etc.)
+
+3. **Write the commit message** following these rules:
+
+**Use the accompanying style guide:**
+- See [style-guide-reference.md](style-guide-reference.md)
+  for detailed analysis of past commits in this repo.
+
+**Subject Line Format:**
+- Present tense verbs: Add, Drop, Fix, Use, Move, Adjust, etc.
+- Target 50 chars (hard max: 67)
+- Backticks around ALL code elements (classes, functions, modules, vars)
+- Specific about what changed
+
+**Body Format (optional - keep simple if warranted):**
+- Max 67 char line length
+- Use `-` bullets for lists
+- Section markers: `Also,` `Deats,` `Other,` `Further,`
+- Abbreviations: msg, bg, ctx, impl, mod, obvi, tn, fn, bc, var, prolly
+- Casual yet technically precise tone
+- Never write lines with only whitespace
+
+**Common Opening Patterns:**
+- New features: "Add `feature` to `module`"
+- Removals: "Drop `attr` from `class`"
+- Bug fixes: "Fix `issue` in `function`"
+- Code moves: "Move `thing` to `new_location`"
+- Adoption: "Use `new_tool` for `task`"
+- Minor tweaks: "Adjust `behavior` in `component`"
+
+4. **Write to TWO files**:
+   - `.claude/<timestamp>_<hash>_commit_msg.md`
+     * with `<timestamp>` from `date -u +%Y%m%dT%H%M%SZ` or similar
+       filesystem-safe format.
+     * and `<hash>` from `git log -1 --format=%h` first 7 chars.
+   - `.claude/git_commit_msg_LATEST.md` (overwrite)
+
+5. **Always include credit footer**:
+
+When no part of the patch was written by `claude`,
+```
+(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
+[claude-code-gh]: https://github.com/anthropics/claude-code
+```
+
+when some or all of the patch was written by `claude`
+instead use,
+
+```
+(this patch was generated in some part by [`claude-code`][claude-code-gh])
+[claude-code-gh]: https://github.com/anthropics/claude-code
+```
+
+Keep it concise. Match the tone of recent commits. For simple
+changes, use subject line only.

.claude/skills/commit-msg/style-guide-reference.md

@@ -0,0 +1,225 @@
+# Commit Message Style Guide for `tractor`
+
+Analysis based on 500 recent commits from the `tractor` repository.
+
+## Core Principles
+
+Write commit messages that are technically precise yet casual in
+tone. Use abbreviations and informal language while maintaining
+clarity about what changed and why.
+
+## Subject Line Format
+
+### Length and Structure
+- Target: ~50 chars with a hard-max of 67.
+- Use backticks around code elements (72.2% of commits)
+- Rarely use colons (5.2%), except for file prefixes
+- End with '?' for uncertain changes (rare: 0.8%)
+- End with '!' for important changes (rare: 2.0%)
+
+### Opening Verbs (Present Tense)
+
+Most common verbs from analysis:
+- `Add` (14.4%) - wholly new features/functionality
+- `Use` (4.4%) - adopt new approach/tool
+- `Drop` (3.6%) - remove code/feature
+- `Fix` (2.4%) - bug fixes
+- `Move`/`Mv` (3.6%) - relocate code
+- `Adjust` (2.0%) - minor tweaks
+- `Update` (1.6%) - enhance existing feature
+- `Bump` (1.2%) - dependency updates
+- `Rename` (1.2%) - identifier changes
+- `Set` (1.2%) - configuration changes
+- `Handle` (1.0%) - add handling logic
+- `Raise` (1.0%) - add error raising
+- `Pass` (0.8%) - pass parameters/values
+- `Support` (0.8%) - add support for something
+- `Hide` (1.4%) - make private/internal
+- `Always` (1.4%) - enforce consistent behavior
+- `Mk` (1.4%) - make/create (abbreviated)
+- `Start` (1.0%) - begin implementation
+
+Other frequent verbs: `More`, `Change`, `Extend`, `Disable`, `Log`,
+`Enable`, `Ensure`, `Expose`, `Allow`
+
+### Backtick Usage
+
+Always use backticks for:
+- Module names: `trio`, `asyncio`, `msgspec`, `greenback`, `stackscope`
+- Class names: `Context`, `Actor`, `Address`, `PldRx`, `SpawnSpec`
+- Method names: `.pause_from_sync()`, `._pause()`, `.cancel()`
+- Function names: `breakpoint()`, `collapse_eg()`, `open_root_actor()`
+- Decorators: `@acm`, `@context`
+- Exceptions: `Cancelled`, `TransportClosed`, `MsgTypeError`
+- Keywords: `finally`, `None`, `False`
+- Variable names: `tn`, `debug_mode`
+- Complex expressions: `trio.Cancelled`, `asyncio.Task`
+
+Most backticked terms in tractor:
+`trio`, `asyncio`, `Context`, `.pause_from_sync()`, `tn`,
+`._pause()`, `breakpoint()`, `collapse_eg()`, `Actor`, `@acm`,
+`.cancel()`, `Cancelled`, `open_root_actor()`, `greenback`
+
+### Examples
+
+Good subject lines:
+```
+Add `uds` to `._multiaddr`, tweak typing
+Drop `DebugStatus.shield` attr, add `.req_finished`
+Use `stackscope` for all actor-tree rendered "views"
+Fix `.to_asyncio` inter-task-cancellation!
+Bump `ruff.toml` to target py313
+Mv `load_module_from_path()` to new `._code_load` submod
+Always use `tuple`-cast for singleton parent addrs
+```
+
+## Body Format
+
+### General Structure
+- 43.2% of commits have no body (simple changes)
+- Use blank line after subject
+- Max line length: 67 chars
+- Use `-` bullets for lists (28.0% of commits)
+- Rarely use `*` bullets (2.4%)
+
+### Section Markers
+
+Use these markers to organize longer commit bodies:
+- `Also,` (most common: 26 occurrences)
+- `Other,` (13 occurrences)
+- `Deats,` (11 occurrences) - for implementation details
+- `Further,` (7 occurrences)
+- `TODO,` (3 occurrences)
+- `Impl details,` (2 occurrences)
+- `Notes,` (1 occurrence)
+
+### Common Abbreviations
+
+Use these freely (sorted by frequency):
+- `msg` (63) - message
+- `bg` (37) - background
+- `ctx` (30) - context
+- `impl` (27) - implementation
+- `mod` (26) - module
+- `obvi` (17) - obviously
+- `tn` (16) - task name
+- `fn` (15) - function
+- `vs` (15) - versus
+- `bc` (14) - because
+- `var` (14) - variable
+- `prolly` (9) - probably
+- `ep` (6) - entry point
+- `OW` (5) - otherwise
+- `rn` (4) - right now
+- `sig` (4) - signal/signature
+- `deps` (3) - dependencies
+- `iface` (2) - interface
+- `subproc` (2) - subprocess
+- `tho` (2) - though
+- `ofc` (2) - of course
+
+### Tone and Style
+
+- Casual but technical (use `XD` for humor: 23 times)
+- Use `..` for trailing thoughts (108 occurrences)
+- Use `Woops,` to acknowledge mistakes (4 subject lines)
+- Don't be afraid to show personality while being precise
+
+### Example Bodies
+
+Simple with bullets:
+```
+Add `multiaddr` and bump up some deps
+
+Since we're planning to use it for (discovery)
+addressing, allowing replacement of the hacky (pretend)
+attempt in `tractor._multiaddr` Bp
+
+Also pin some deps,
+- make us py312+
+- use `pdbp` with my frame indexing fix.
+- mv to latest `xonsh` for fancy cmd/suggestion injections.
+
+Bump lock file to match obvi!
+```
+
+With section markers:
+```
+Use `stackscope` for all actor-tree rendered "views"
+
+Instead of the (much more) limited and hacky `.devx._code`
+impls, move to using the new `.devx._stackscope` API which
+wraps the `stackscope` project.
+
+Deats,
+- make new `stackscope.extract_stack()` wrapper
+- port over frame-descing to `_stackscope.pformat_stack()`
+- move `PdbREPL` to use `stackscope` render approach
+- update tests for new stack output format
+
+Also,
+- tweak log formatting for consistency
+- add typing hints throughout
+```
+
+## Special Patterns
+
+### WIP Commits
+Rare (0.2%) - avoid committing WIP if possible
+
+### Merge Commits
+Auto-generated (4.4%), don't worry about style
+
+### File References
+- Use `module.py` or `.submodule` style
+- Rarely use `file.py:line` references (0 in analysis)
+
+### Links
+- GitHub links used sparingly (3 total)
+- Prefer code references over external links
+
+## Footer
+
+The default footer should credit `claude` (you) for helping generate
+the commit msg content:
+
+```
+(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
+[claude-code-gh]: https://github.com/anthropics/claude-code
+```
+
+Further, if the patch was solely or in part written
+by `claude`, instead add:
+
+```
+(this patch was generated in some part by [`claude-code`][claude-code-gh])
+[claude-code-gh]: https://github.com/anthropics/claude-code
+```
+
+## Summary Checklist
+
+Before committing, verify:
+- [ ] Subject line uses present tense verb
+- [ ] Subject line ~50 chars (hard max 67)
+- [ ] Code elements wrapped in backticks
+- [ ] Body lines ≤67 chars
+- [ ] Abbreviations used where natural
+- [ ] Casual yet precise tone
+- [ ] Section markers if body >3 paragraphs
+- [ ] Technical accuracy maintained
+
+## Analysis Metadata
+
+```
+Source: tractor repository
+Commits analyzed: 500
+Date range: 2019-2025
+Analysis date: 2026-02-08
+```
+
+---
+
+(this style guide was generated by [`claude-code`][claude-code-gh]
+analyzing commit history)
+
+[claude-code-gh]: https://github.com/anthropics/claude-code

.github/workflows/ci.yml

@@ -8,83 +8,144 @@ on:
   workflow_dispatch:
 
 jobs:
-
+  # ------ sdist ------
-  mypy:
-    name: 'MyPy'
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-
-      - name: Setup python
-        uses: actions/setup-python@v2
-        with:
-          python-version: '3.11'
-
-      - name: Install dependencies
-        run: pip install -U . --upgrade-strategy eager -r requirements-test.txt
-
-      - name: Run MyPy check
-        run: mypy tractor/ --ignore-missing-imports --show-traceback
-
   # test that we can generate a software distribution and install it
   # thus avoid missing file issues after packaging.
+  #
+  # -[x] produce sdist with uv
+  # ------ - ------
   sdist-linux:
     name: 'sdist'
     runs-on: ubuntu-latest
 
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
 
-      - name: Setup python
+      - name: Install latest uv
-        uses: actions/setup-python@v2
+        uses: astral-sh/setup-uv@v6
-        with:
-          python-version: '3.11'
 
-      - name: Build sdist
+      - name: Build sdist as tar.gz
-        run: python setup.py sdist --formats=zip
+        run: uv build --sdist --python=3.13
 
-      - name: Install sdist from .zips
+      - name: Install sdist from .tar.gz
-        run: python -m pip install dist/*.zip
+        run: python -m pip install dist/*.tar.gz
+
+  # ------ type-check ------
+  # mypy:
+  #   name: 'MyPy'
+  #   runs-on: ubuntu-latest
+
+  #   steps:
+  #     - name: Checkout
+  #       uses: actions/checkout@v4
+
+  #     - name: Install latest uv
+  #       uses: astral-sh/setup-uv@v6
+
+  #     # faster due to server caching?
+  #     # https://docs.astral.sh/uv/guides/integration/github/#setting-up-python
+  #     - name: "Set up Python"
+  #       uses: actions/setup-python@v6
+  #       with:
+  #         python-version-file: "pyproject.toml"
+
+  #     # w uv
+  #     # - name: Set up Python
+  #     #   run: uv python install
+
+  #     - name: Setup uv venv
+  #       run: uv venv .venv --python=3.13
+
+  #     - name: Install
+  #       run: uv sync --dev
+
+  #     # TODO, ty cmd over repo
+  #     # - name: type check with ty
+  #     #   run: ty ./tractor/
+
+  #     # - uses: actions/cache@v3
+  #     #     name: Cache uv virtenv as default .venv
+  #     #     with:
+  #     #       path: ./.venv
+  #     #       key: venv-${{ hashFiles('uv.lock') }}
+
+  #     - name: Run MyPy check
+  #       run: mypy tractor/ --ignore-missing-imports --show-traceback
 
 
-  testing-linux:
+  testing:
-    name: '${{ matrix.os }} Python ${{ matrix.python }} - ${{ matrix.spawn_backend }}'
+    name: '${{ matrix.os }} Python${{ matrix.python-version }} spawn_backend=${{ matrix.spawn_backend }} tpt_proto=${{ matrix.tpt_proto }}'
-    timeout-minutes: 10
+    timeout-minutes: 16
     runs-on: ${{ matrix.os }}
 
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest]
+        os: [
-        python: ['3.11']
+          ubuntu-latest,
+          macos-latest,
+        ]
+        python-version: [
+          '3.13',
+          # '3.14',
+        ]
         spawn_backend: [
           'trio',
-          'mp_spawn',
+          # 'mp_spawn',
-          'mp_forkserver',
+          # 'mp_forkserver',
+          # ?TODO^ is it worth it to get these running again?
+          #
+          # - [ ] next-gen backends, on 3.13+
+          #   https://github.com/goodboy/tractor/issues/379
+          # 'subinterpreter',
+          # 'subint',
         ]
+        tpt_proto: [
+          'tcp',
+          'uds',
+        ]
+        # https://github.com/orgs/community/discussions/26253#discussioncomment-3250989
+        exclude:
+          # don't do UDS run on macOS (for now)
+          - os: macos-latest
+            tpt_proto: 'uds'
 
     steps:
+      - uses: actions/checkout@v4
 
-      - name: Checkout
+      - name: 'Install uv + py-${{ matrix.python-version }}'
-        uses: actions/checkout@v2
+        uses: astral-sh/setup-uv@v6
-
-      - name: Setup python
-        uses: actions/setup-python@v2
         with:
-          python-version: '${{ matrix.python }}'
+          python-version: ${{ matrix.python-version }}
 
-      - name: Install dependencies
+      # GH way.. faster?
-        run: pip install -U . -r requirements-test.txt -r requirements-docs.txt --upgrade-strategy eager
+      # - name: setup-python@v6
+      #   uses: actions/setup-python@v6
+      #   with:
+      #     python-version: '${{ matrix.python-version }}'
 
-      - name: List dependencies
+      # consider caching for speedups?
-        run: pip list
+      # https://docs.astral.sh/uv/guides/integration/github/#caching
+
+      - name: Install the project w uv
+        run: uv sync --all-extras --dev
+
+      # - name: Install dependencies
+      #   run: pip install -U . -r requirements-test.txt -r requirements-docs.txt --upgrade-strategy eager
+
+      - name: List deps tree
+        run: uv tree
 
       - name: Run tests
-        run: pytest tests/ --spawn-backend=${{ matrix.spawn_backend }} -rsx
+        run: >
+          uv run
+          pytest tests/ -rsx
+          --spawn-backend=${{ matrix.spawn_backend }}
+          --tpt-proto=${{ matrix.tpt_proto }}
 
+  # XXX legacy NOTE XXX
+  #
   # We skip 3.10 on windows for now due to not having any collabs to
   # debug the CI failures. Anyone wanting to hack and solve them is very
   # welcome, but our primary user base is not using that OS.

.gitignore

@@ -102,3 +102,30 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+
+# all files under
+.git/
+
+# any commit-msg gen tmp files
+.claude/*_commit_*.md
+.claude/*_commit*.toml
+.claude/*_commit*.txt
+
+# nix develop --profile .nixdev
+.nixdev*
+
+# :Obsession .
+Session.vim
+# `gish` local `.md`-files
+# TODO? better all around automation!
+# -[ ] it'd be handy to also commit and sync with wtv git service?
+# -[ ] everything should be put under a `.gish/` no?
+gitea/
+gh/
+
+# ------ macOS ------
+# Finder metadata
+**/.DS_Store
+
+# LLM conversations that should remain private
+docs/conversations/

default.nix

@@ -0,0 +1,19 @@
+{ pkgs ? import <nixpkgs> {} }:
+let
+  nativeBuildInputs = with pkgs; [
+    stdenv.cc.cc.lib
+    uv
+  ];
+
+in
+pkgs.mkShell {
+  inherit nativeBuildInputs;
+
+  LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath nativeBuildInputs;
+  TMPDIR = "/tmp";
+
+  shellHook = ''
+    set -e
+    uv venv .venv --python=3.12
+  '';
+}

docs/README.rst

@@ -1,8 +1,5 @@
 |logo| ``tractor``: distributed structurred concurrency
 
-|gh_actions|
-|docs|
-
 ``tractor`` is a `structured concurrency`_ (SC), multi-processing_ runtime built on trio_.
 
 Fundamentally, ``tractor`` provides parallelism via
@@ -66,6 +63,13 @@ Features
   - (WIP) a ``TaskMngr``: one-cancels-one style nursery supervisor.
 
 
+Status of `main` / infra
+------------------------
+
+- |gh_actions|
+- |docs|
+
+
 Install
 -------
 ``tractor`` is still in a *alpha-near-beta-stage* for many
@@ -416,20 +420,17 @@ Check out our experimental system for `guest`_-mode controlled
 
 
     async def aio_echo_server(
-        to_trio: trio.MemorySendChannel,
+        chan: tractor.to_asyncio.LinkedTaskChannel,
-        from_trio: asyncio.Queue,
     ) -> None:
 
         # a first message must be sent **from** this ``asyncio``
         # task or the ``trio`` side will never unblock from
         # ``tractor.to_asyncio.open_channel_from():``
-        to_trio.send_nowait('start')
+        chan.started_nowait('start')
 
-        # XXX: this uses an ``from_trio: asyncio.Queue`` currently but we
-        # should probably offer something better.
         while True:
             # echo the msg back
-            to_trio.send_nowait(await from_trio.get())
+            chan.send_nowait(await chan.get())
             await asyncio.sleep(0)
 
 
@@ -441,7 +442,7 @@ Check out our experimental system for `guest`_-mode controlled
         # message.
         async with tractor.to_asyncio.open_channel_from(
             aio_echo_server,
-        ) as (first, chan):
+        ) as (chan, first):
 
             assert first == 'start'
             await ctx.started(first)
@@ -500,8 +501,10 @@ Yes, we spawn a python process, run ``asyncio``, start ``trio`` on the
 ``asyncio`` loop, then send commands to the ``trio`` scheduled tasks to
 tell ``asyncio`` tasks what to do XD
 
-We need help refining the `asyncio`-side channel API to be more
+The ``asyncio``-side task receives a single
-`trio`-like. Feel free to sling your opinion in `#273`_!
+``chan: LinkedTaskChannel`` handle providing a ``trio``-like
+API: ``.started_nowait()``, ``.send_nowait()``, ``.get()``
+and more. Feel free to sling your opinion in `#273`_!
 
 
 .. _#273: https://github.com/goodboy/tractor/issues/273
@@ -637,13 +640,15 @@ Help us push toward the future of distributed `Python`.
 - Typed capability-based (dialog) protocols ( see `#196
   <https://github.com/goodboy/tractor/issues/196>`_ with draft work
   started in `#311 <https://github.com/goodboy/tractor/pull/311>`_)
-- We **recently disabled CI-testing on windows** and need help getting
+- **macOS is now officially supported** and tested in CI
-  it running again! (see `#327
+  alongside Linux!
-  <https://github.com/goodboy/tractor/pull/327>`_). **We do have windows
+- We **recently disabled CI-testing on windows** and need
-  support** (and have for quite a while) but since no active hacker
+  help getting it running again! (see `#327
-  exists in the user-base to help test on that OS, for now we're not
+  <https://github.com/goodboy/tractor/pull/327>`_). **We do
-  actively maintaining testing due to the added hassle and general
+  have windows support** (and have for quite a while) but
-  latency..
+  since no active hacker exists in the user-base to help
+  test on that OS, for now we're not actively maintaining
+  testing due to the added hassle and general latency..
 
 
 Feel like saying hi?
@@ -689,9 +694,11 @@ channel`_!
 .. _msgspec: https://jcristharif.com/msgspec/
 .. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
 
-
+..
-.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square
+   NOTE, on generating badge links from the UI
-    :target: https://actions-badge.atrox.dev/goodboy/tractor/goto
+   https://docs.github.com/en/actions/how-tos/monitoring-and-troubleshooting-workflows/monitoring-workflows/adding-a-workflow-status-badge?ref=gitguardian-blog-automated-secrets-detection#using-the-ui
+.. |gh_actions| image:: https://github.com/goodboy/tractor/actions/workflows/ci.yml/badge.svg?branch=main
+    :target: https://github.com/goodboy/tractor/actions/workflows/ci.yml
 
 .. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest
     :target: https://tractor.readthedocs.io/en/latest/?badge=latest

examples/advanced_faults/ipc_failure_during_stream.py

@@ -16,6 +16,8 @@ from tractor import (
     ContextCancelled,
     MsgStream,
     _testing,
+    trionics,
+    TransportClosed,
 )
 import trio
 import pytest
@@ -62,9 +64,8 @@ async def recv_and_spawn_net_killers(
     await ctx.started()
     async with (
         ctx.open_stream() as stream,
-        trio.open_nursery(
+        trionics.collapse_eg(),
-            strict_exception_groups=False,
+        trio.open_nursery() as tn,
-        ) as tn,
     ):
         async for i in stream:
             print(f'child echoing {i}')
@@ -120,6 +121,7 @@ async def main(
     break_parent_ipc_after: int|bool = False,
     break_child_ipc_after: int|bool = False,
     pre_close: bool = False,
+    tpt_proto: str = 'tcp',
 
 ) -> None:
 
@@ -131,6 +133,7 @@ async def main(
             # a hang since it never engages due to broken IPC
             debug_mode=debug_mode,
             loglevel=loglevel,
+            enable_transports=[tpt_proto],
 
         ) as an,
     ):
@@ -145,7 +148,8 @@ async def main(
             _testing.expect_ctxc(
                 yay=(
                     break_parent_ipc_after
-                    or break_child_ipc_after
+                    or
+                    break_child_ipc_after
                 ),
                 # TODO: we CAN'T remove this right?
                 # since we need the ctxc to bubble up from either
@@ -205,11 +209,15 @@ async def main(
                         # TODO: is this needed or no?
                         raise
 
-                    except trio.ClosedResourceError:
+                    except (
+                        trio.ClosedResourceError,
+                        TransportClosed,
+                    ) as _tpt_err:
                         # NOTE: don't send if we already broke the
                         # connection to avoid raising a closed-error
                         # such that we drop through to the ctl-c
                         # mashing by user.
+                        with trio.CancelScope(shield=True):
                             await trio.sleep(0.01)
 
                     # timeout: int = 1
@@ -244,6 +252,7 @@ async def main(
                     await stream.send(i)
                     pytest.fail('stream not closed?')
                 except (
+                    TransportClosed,
                     trio.ClosedResourceError,
                     trio.EndOfChannel,
                 ) as send_err:

examples/debugging/asyncio_bp.py

@@ -18,18 +18,17 @@ async def aio_sleep_forever():
 
 
 async def bp_then_error(
-    to_trio: trio.MemorySendChannel,
+    chan: to_asyncio.LinkedTaskChannel,
-    from_trio: asyncio.Queue,
 
     raise_after_bp: bool = True,
 
 ) -> None:
 
     # sync with `trio`-side (caller) task
-    to_trio.send_nowait('start')
+    chan.started_nowait('start')
 
     # NOTE: what happens here inside the hook needs some refinement..
-    # => seems like it's still `._debug._set_trace()` but
+    # => seems like it's still `.debug._set_trace()` but
     #    we set `Lock.local_task_in_debug = 'sync'`, we probably want
     #    some further, at least, meta-data about the task/actor in debug
     #    in terms of making it clear it's `asyncio` mucking about.
@@ -60,7 +59,7 @@ async def trio_ctx(
         to_asyncio.open_channel_from(
             bp_then_error,
             # raise_after_bp=not bp_before_started,
-        ) as (first, chan),
+        ) as (chan, first),
 
         trio.open_nursery() as tn,
     ):

examples/debugging/fast_error_in_root_after_spawn.py

@@ -20,7 +20,7 @@ async def sleep(
 
 
 async def open_ctx(
-    n: tractor._supervise.ActorNursery
+    n: tractor.runtime._supervise.ActorNursery
 ):
 
     # spawn both actors

examples/debugging/restore_builtin_breakpoint.py

@@ -4,6 +4,11 @@ import sys
 import trio
 import tractor
 
+# ensure mod-path is correct!
+from tractor.devx.debug import (
+    _sync_pause_from_builtin as _sync_pause_from_builtin,
+)
+
 
 async def main() -> None:
 
@@ -13,19 +18,23 @@ async def main() -> None:
 
     async with tractor.open_nursery(
         debug_mode=True,
-    ) as an:
+        loglevel='devx',
-        assert an
+        maybe_enable_greenback=True,
+        # ^XXX REQUIRED to enable `breakpoint()` support (from sync
+        # fns) and thus required here to avoid an assertion err
+        # on the next line
+    ):
         assert (
             (pybp_var := os.environ['PYTHONBREAKPOINT'])
             ==
-            'tractor.devx._debug._sync_pause_from_builtin'
+            'tractor.devx.debug._sync_pause_from_builtin'
         )
 
         # TODO: an assert that verifies the hook has indeed been, hooked
         # XD
         assert (
             (pybp_hook := sys.breakpointhook)
-            is not tractor.devx._debug._set_trace
+            is not tractor.devx.debug._set_trace
         )
 
         print(

examples/debugging/root_cancelled_but_child_is_in_tty_lock.py

@@ -24,10 +24,9 @@ async def spawn_until(depth=0):
 
 
 async def main():
-    """The main ``tractor`` routine.
+    '''
-
+    The process tree should look as approximately as follows when the
-    The process tree should look as approximately as follows when the debugger
+    debugger first engages:
-    first engages:
 
     python examples/debugging/multi_nested_subactors_bp_forever.py
     ├─ python -m tractor._child --uid ('spawner1', '7eab8462 ...)
@@ -37,10 +36,11 @@ async def main():
     └─ python -m tractor._child --uid ('spawner0', '1d42012b ...)
        └─ python -m tractor._child --uid ('name_error', '6c2733b8 ...)
 
-    """
+    '''
     async with tractor.open_nursery(
         debug_mode=True,
-        loglevel='warning'
+        loglevel='devx',
+        enable_transports=['uds'],
     ) as n:
 
         # spawn both actors

examples/debugging/root_self_cancelled_w_error.py

@@ -0,0 +1,35 @@
+import trio
+import tractor
+
+
+async def main():
+    async with tractor.open_root_actor(
+        debug_mode=True,
+        loglevel='cancel',
+    ) as _root:
+
+        # manually trigger self-cancellation and wait
+        # for it to fully trigger.
+        _root.cancel_soon()
+        await _root._cancel_complete.wait()
+        print('root cancelled')
+
+        # now ensure we can still use the REPL
+        try:
+            await tractor.pause()
+        except trio.Cancelled as _taskc:
+            assert (root_cs := _root._root_tn.cancel_scope).cancel_called
+            # NOTE^^ above logic but inside `open_root_actor()` and
+            # passed to the `shield=` expression is effectively what
+            # we're testing here!
+            await tractor.pause(shield=root_cs.cancel_called)
+
+        # XXX, if shield logic *is wrong* inside `open_root_actor()`'s
+        # crash-handler block this should never be interacted,
+        # instead `trio.Cancelled` would be bubbled up: the original
+        # BUG.
+        assert 0
+
+
+if __name__ == '__main__':
+    trio.run(main)

examples/debugging/shield_hang_in_sub.py

@@ -3,6 +3,7 @@ Verify we can dump a `stackscope` tree on a hang.
 
 '''
 import os
+import platform
 import signal
 
 import trio
@@ -31,12 +32,26 @@ async def main(
     from_test: bool = False,
 ) -> None:
 
+    if platform.system() != 'Darwin':
+        tpt = 'uds'
+    else:
+        # XXX, precisely we can't use pytest's tmp-path generation
+        # for tests.. apparently because:
+        #
+        # > The OSError: AF_UNIX path too long in macOS Python occurs
+        # > because the path to the Unix domain socket exceeds the
+        # > operating system's maximum path length limit (around 104
+        #
+        # WHICH IS just, wtf hillarious XD
+        tpt = 'tcp'
+
     async with (
         tractor.open_nursery(
             debug_mode=True,
             enable_stack_on_sig=True,
             # maybe_enable_greenback=False,
             loglevel='devx',
+            enable_transports=[tpt],
         ) as an,
     ):
         ptl: tractor.Portal  = await an.start_actor(

examples/debugging/subactor_bp_in_ctx.py

@@ -1,3 +1,5 @@
+import platform
+
 import tractor
 import trio
 
@@ -33,8 +35,24 @@ async def just_bp(
 
 
 async def main():
+
+    if platform.system() != 'Darwin':
+        tpt = 'uds'
+    else:
+        # XXX, precisely we can't use pytest's tmp-path generation
+        # for tests.. apparently because:
+        #
+        # > The OSError: AF_UNIX path too long in macOS Python occurs
+        # > because the path to the Unix domain socket exceeds the
+        # > operating system's maximum path length limit (around 104
+        #
+        # WHICH IS just, wtf hillarious XD
+        tpt = 'tcp'
+
     async with tractor.open_nursery(
         debug_mode=True,
+        enable_transports=[tpt],
+        loglevel='devx',
     ) as n:
         p = await n.start_actor(
             'bp_boi',

examples/debugging/sync_bp.py

@@ -6,7 +6,7 @@ import tractor
 
 # TODO: only import these when not running from test harness?
 # can we detect `pexpect` usage maybe?
-# from tractor.devx._debug import (
+# from tractor.devx.debug import (
 #     get_lock,
 #     get_debug_req,
 # )

examples/full_fledged_streaming_service.py

@@ -90,7 +90,7 @@ async def main() -> list[int]:
     # yes, a nursery which spawns `trio`-"actors" B)
     an: ActorNursery
     async with tractor.open_nursery(
-        loglevel='cancel',
+        loglevel='error',
         # debug_mode=True,
     ) as an:
 
@@ -118,8 +118,10 @@ async def main() -> list[int]:
         cancelled: bool = await portal.cancel_actor()
         assert cancelled
 
-        print(f"STREAM TIME = {time.time() - start}")
+        print(
-        print(f"STREAM + SPAWN TIME = {time.time() - pre_start}")
+            f"STREAM TIME = {time.time() - start}\n"
+            f"STREAM + SPAWN TIME = {time.time() - pre_start}\n"
+        )
         assert result_stream == list(range(seed))
         return result_stream
 

examples/infected_asyncio_echo_server.py

@@ -11,21 +11,17 @@ import tractor
 
 
 async def aio_echo_server(
-    to_trio: trio.MemorySendChannel,
+    chan: tractor.to_asyncio.LinkedTaskChannel,
-    from_trio: asyncio.Queue,
-
 ) -> None:
 
     # a first message must be sent **from** this ``asyncio``
     # task or the ``trio`` side will never unblock from
     # ``tractor.to_asyncio.open_channel_from():``
-    to_trio.send_nowait('start')
+    chan.started_nowait('start')
 
-    # XXX: this uses an ``from_trio: asyncio.Queue`` currently but we
-    # should probably offer something better.
     while True:
         # echo the msg back
-        to_trio.send_nowait(await from_trio.get())
+        chan.send_nowait(await chan.get())
         await asyncio.sleep(0)
 
 
@@ -37,7 +33,7 @@ async def trio_to_aio_echo_server(
     # message.
     async with tractor.to_asyncio.open_channel_from(
         aio_echo_server,
-    ) as (first, chan):
+    ) as (chan, first):
 
         assert first == 'start'
         await ctx.started(first)

examples/quick_cluster.py

@@ -23,9 +23,8 @@ async def main():
             modules=[__name__]
         ) as portal_map,
 
-        trio.open_nursery(
+        tractor.trionics.collapse_eg(),
-            strict_exception_groups=False,
+        trio.open_nursery() as tn,
-        ) as tn,
     ):
 
         for (name, portal) in portal_map.items():

examples/service_discovery.py

@@ -9,8 +9,8 @@ async def main(service_name):
     async with tractor.open_nursery() as an:
         await an.start_actor(service_name)
 
-        async with tractor.get_registry('127.0.0.1', 1616) as portal:
+        async with tractor.get_registry() as portal:
-            print(f"Arbiter is listening on {portal.channel}")
+            print(f"Registrar is listening on {portal.channel}")
 
         async with tractor.wait_for_actor(service_name) as sockaddr:
             print(f"my_service is found at {sockaddr}")

examples/trio/lockacquire_not_unmasked.py

@@ -0,0 +1,85 @@
+from contextlib import (
+    asynccontextmanager as acm,
+)
+from functools import partial
+
+import tractor
+import trio
+
+
+log = tractor.log.get_logger(
+    name=__name__
+)
+
+_lock: trio.Lock|None = None
+
+
+@acm
+async def acquire_singleton_lock(
+) -> None:
+    global _lock
+    if _lock is None:
+        log.info('Allocating LOCK')
+        _lock = trio.Lock()
+
+    log.info('TRYING TO LOCK ACQUIRE')
+    async with _lock:
+        log.info('ACQUIRED')
+        yield _lock
+
+    log.info('RELEASED')
+
+
+
+async def hold_lock_forever(
+    task_status=trio.TASK_STATUS_IGNORED
+):
+    async with (
+        tractor.trionics.maybe_raise_from_masking_exc(),
+        acquire_singleton_lock() as lock,
+    ):
+        task_status.started(lock)
+        await trio.sleep_forever()
+
+
+async def main(
+    ignore_special_cases: bool,
+    loglevel: str = 'info',
+    debug_mode: bool = True,
+):
+    async with (
+        trio.open_nursery() as tn,
+
+        # tractor.trionics.maybe_raise_from_masking_exc()
+        # ^^^ XXX NOTE, interestingly putting the unmasker
+        # here does not exhibit the same behaviour ??
+    ):
+        if not ignore_special_cases:
+            from tractor.trionics import _taskc
+            _taskc._mask_cases.clear()
+
+        _lock = await tn.start(
+            hold_lock_forever,
+        )
+        with trio.move_on_after(0.2):
+            await tn.start(
+                hold_lock_forever,
+            )
+
+        tn.cancel_scope.cancel()
+
+
+# XXX, manual test as script
+if __name__ == '__main__':
+    tractor.log.get_console_log(level='info')
+    for case in [True, False]:
+        log.info(
+            f'\n'
+            f'------ RUNNING SCRIPT TRIAL ------\n'
+            f'ignore_special_cases: {case!r}\n'
+        )
+        trio.run(partial(
+            main,
+            ignore_special_cases=case,
+            loglevel='info',
+        ))

examples/trio/send_chan_aclose_masks_beg.py

@@ -0,0 +1,195 @@
+from contextlib import (
+    contextmanager as cm,
+    # TODO, any diff in async case(s)??
+    # asynccontextmanager as acm,
+)
+from functools import partial
+
+import tractor
+import trio
+
+
+log = tractor.log.get_logger(
+    name=__name__
+)
+
+
+@cm
+def teardown_on_exc(
+    raise_from_handler: bool = False,
+):
+    '''
+    You could also have a teardown handler which catches any exc and
+    does some required teardown. In this case the problem is
+    compounded UNLESS you ensure the handler's scope is OUTSIDE the
+    `ux.aclose()`.. that is in the caller's enclosing scope.
+
+    '''
+    try:
+        yield
+    except BaseException as _berr:
+        berr = _berr
+        log.exception(
+            f'Handling termination teardown in child due to,\n'
+            f'{berr!r}\n'
+        )
+        if raise_from_handler:
+            # XXX teardown ops XXX
+            # on termination these steps say need to be run to
+            # ensure wider system consistency (like the state of
+            # remote connections/services).
+            #
+            # HOWEVER, any bug in this teardown code is also
+            # masked by the `tx.aclose()`!
+            # this is also true if `_tn.cancel_scope` is
+            # `.cancel_called` by the parent in a graceful
+            # request case..
+
+            # simulate a bug in teardown handler.
+            raise RuntimeError(
+                'woopsie teardown bug!'
+            )
+
+        raise  # no teardown bug.
+
+
+async def finite_stream_to_rent(
+    tx: trio.abc.SendChannel,
+    child_errors_mid_stream: bool,
+    raise_unmasked: bool,
+
+    task_status: trio.TaskStatus[
+        trio.CancelScope,
+    ] = trio.TASK_STATUS_IGNORED,
+):
+    async with (
+        # XXX without this unmasker the mid-streaming RTE is never
+        # reported since it is masked by the `tx.aclose()`
+        # call which in turn raises `Cancelled`!
+        #
+        # NOTE, this is WITHOUT doing any exception handling
+        # inside the child  task!
+        #
+        # TODO, uncomment next LoC to see the supprsessed beg[RTE]!
+        tractor.trionics.maybe_raise_from_masking_exc(
+            raise_unmasked=raise_unmasked,
+        ),
+
+        tx as tx,  # .aclose() is the guilty masker chkpt!
+
+        # XXX, this ONLY matters in the
+        # `child_errors_mid_stream=False` case oddly!?
+        # THAT IS, if no tn is opened in that case then the
+        # test will not fail; it raises the RTE correctly?
+        #
+        # -> so it seems this new scope somehow affects the form of
+        #    eventual in the parent EG?
+        tractor.trionics.maybe_open_nursery(
+            nursery=(
+                None
+                if not child_errors_mid_stream
+                else True
+            ),
+        ) as _tn,
+    ):
+        # pass our scope back to parent for supervision\
+        # control.
+        cs: trio.CancelScope|None = (
+            None
+            if _tn is True
+            else _tn.cancel_scope
+        )
+        task_status.started(cs)
+
+        with teardown_on_exc(
+            raise_from_handler=not child_errors_mid_stream,
+        ):
+            for i in range(100):
+                log.debug(
+                    f'Child tx {i!r}\n'
+                )
+                if (
+                    child_errors_mid_stream
+                    and
+                    i == 66
+                ):
+                    # oh wait but WOOPS there's a bug
+                    # in that teardown code!?
+                    raise RuntimeError(
+                        'woopsie, a mid-streaming bug!?'
+                    )
+
+                await tx.send(i)
+
+
+async def main(
+    # TODO! toggle this for the 2 cases!
+    # 1. child errors mid-stream while parent is also requesting
+    #   (graceful) cancel of that child streamer.
+    #
+    # 2. child contains a teardown handler which contains a
+    #   bug and raises.
+    #
+    child_errors_mid_stream: bool,
+
+    raise_unmasked: bool = False,
+    loglevel: str = 'info',
+):
+    tractor.log.get_console_log(level=loglevel)
+
+    # the `.aclose()` being checkpoints on these
+    # is the source of the problem..
+    tx, rx = trio.open_memory_channel(1)
+
+    async with (
+        tractor.trionics.collapse_eg(),
+        trio.open_nursery() as tn,
+        rx as rx,
+    ):
+        _child_cs = await tn.start(
+            partial(
+                finite_stream_to_rent,
+                child_errors_mid_stream=child_errors_mid_stream,
+                raise_unmasked=raise_unmasked,
+                tx=tx,
+            )
+        )
+        async for msg in rx:
+            log.debug(
+                f'Rent rx {msg!r}\n'
+            )
+
+            # simulate some external cancellation
+            # request **JUST BEFORE** the child errors.
+            if msg == 65:
+                log.cancel(
+                    f'Cancelling parent on,\n'
+                    f'msg={msg}\n'
+                    f'\n'
+                    f'Simulates OOB cancel request!\n'
+                )
+                tn.cancel_scope.cancel()
+
+
+# XXX, manual test as script
+if __name__ == '__main__':
+    tractor.log.get_console_log(level='info')
+    for case in [True, False]:
+        log.info(
+            f'\n'
+            f'------ RUNNING SCRIPT TRIAL ------\n'
+            f'child_errors_midstream: {case!r}\n'
+        )
+        try:
+            trio.run(partial(
+                main,
+                child_errors_mid_stream=case,
+                # raise_unmasked=True,
+                loglevel='info',
+            ))
+        except Exception as _exc:
+            exc = _exc
+            log.exception(
+                'Should have raised an RTE or Cancelled?\n'
+            )
+            breakpoint()

flake.lock

@@ -0,0 +1,27 @@
+{
+  "nodes": {
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1769018530,
+        "narHash": "sha256-MJ27Cy2NtBEV5tsK+YraYr2g851f3Fl1LpNHDzDX15c=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "88d3861acdd3d2f0e361767018218e51810df8a1",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "nixos-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "nixpkgs": "nixpkgs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,70 @@
+# An "impure" template thx to `pyproject.nix`,
+# https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
+# https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
+{
+  description = "An impure overlay (w dev-shell) using `uv`";
+
+  inputs = {
+    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
+  };
+
+  outputs =
+    { nixpkgs, ... }:
+    let
+      inherit (nixpkgs) lib;
+      forAllSystems = lib.genAttrs lib.systems.flakeExposed;
+    in
+    {
+      devShells = forAllSystems (
+        system:
+        let
+          pkgs = nixpkgs.legacyPackages.${system};
+
+          # XXX NOTE XXX, for now we overlay specific pkgs via
+          # a major-version-pinned-`cpython`
+          cpython = "python313";
+          venv_dir = "py313";
+          pypkgs = pkgs."${cpython}Packages";
+        in
+        {
+          default = pkgs.mkShell {
+
+            packages = [
+              # XXX, ensure sh completions activate!
+              pkgs.bashInteractive
+              pkgs.bash-completion
+
+              # XXX, on nix(os), use pkgs version to avoid
+              # build/sys-sh-integration issues
+              pkgs.ruff
+
+              pkgs.uv
+              pkgs.${cpython}# ?TODO^ how to set from `cpython` above?
+            ];
+
+            shellHook = ''
+              # unmask to debug **this** dev-shell-hook
+              # set -e
+
+              # link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
+              LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
+
+              export LD_LIBRARY_PATH
+
+              # RUNTIME-SETTINGS
+              # ------ uv ------
+              # - always use the ./py313/ venv-subdir
+              # - sync env with all extras
+              export UV_PROJECT_ENVIRONMENT=${venv_dir}
+              uv sync --dev --all-extras
+
+              # ------ TIPS ------
+              # NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
+              # run the `nix develop` cmd with,
+              # >> nix develop -c uv run xonsh
+            '';
+          };
+        }
+      );
+    };
+}

pyproject.toml

@@ -9,7 +9,7 @@ name = "tractor"
 version = "0.1.0a6dev0"
 description = 'structured concurrent `trio`-"actors"'
 authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
-requires-python = ">= 3.11"
+requires-python = ">=3.12, <3.14"
 readme = "docs/README.rst"
 license = "AGPL-3.0-or-later"
 keywords = [
@@ -24,11 +24,13 @@ keywords = [
 classifiers = [
   "Development Status :: 3 - Alpha",
   "Operating System :: POSIX :: Linux",
+  "Operating System :: MacOS",
   "Framework :: Trio",
   "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: 3 :: Only",
-  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
   "Topic :: System :: Distributed Computing",
 ]
 dependencies = [
@@ -42,26 +44,44 @@ dependencies = [
   "wrapt>=1.16.0,<2",
   "colorlog>=6.8.2,<7",
   # built-in multi-actor `pdb` REPL
-  "pdbp>=1.6,<2", # windows only (from `pdbp`)
+  "pdbp>=1.8.2,<2", # windows only (from `pdbp`)
   # typed IPC msging
   "msgspec>=0.19.0",
+  "cffi>=1.17.1",
+  "bidict>=0.23.1",
+  "platformdirs>=4.4.0",
 ]
 
 # ------ project ------
 
 [dependency-groups]
 dev = [
+  {include-group = 'devx'},
+  {include-group = 'testing'},
+  {include-group = 'repl'},
+]
+devx = [
+  # `tractor.devx` tooling
+  "greenback>=1.2.1,<2",
+  "stackscope>=0.2.2,<0.3",
+  # ^ requires this?
+  "typing-extensions>=4.14.1",
+]
+testing = [
   # test suite
   # TODO: maybe some of these layout choices?
   # https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
   "pytest>=8.3.5",
   "pexpect>=4.9.0,<5",
-  # `tractor.devx` tooling
+]
-  "greenback>=1.2.1,<2",
+repl = [
-  "stackscope>=0.2.2,<0.3",
   "pyperclip>=1.9.0",
   "prompt-toolkit>=3.0.50",
-  "xonsh>=0.19.2",
+  "xonsh>=0.22.2",
+  "psutil>=7.0.0",
+]
+lint = [
+  "ruff>=0.9.6"
 ]
 # TODO, add these with sane versions; were originally in
 # `requirements-docs.txt`..
@@ -69,15 +89,19 @@ dev = [
 #   "sphinx>="
 #   "sphinx_book_theme>="
 # ]
-
 # ------ dependency-groups ------
 
-# ------ dependency-groups ------
 
 [tool.uv.sources]
 # XXX NOTE, only for @goodboy's hacking on `pprint(sort_dicts=False)`
 # for the `pp` alias..
-# pdbp = { path = "../pdbp", editable = true }
+
+# [tool.uv.sources.pdbp]
+# XXX, in case we need to tmp patch again.
+# git = "https://github.com/goodboy/pdbp.git"
+# branch ="repair_stack_trace_frame_indexing"
+# path = "../pdbp"
+# editable = true
 
 # ------ tool.uv.sources ------
 # TODO, distributed (multi-host) extensions
@@ -139,6 +163,7 @@ all_bullets = true
 
 [tool.pytest.ini_options]
 minversion = '6.0'
+# https://docs.pytest.org/en/stable/reference/reference.html#configuration-options
 testpaths = [
   'tests'
 ]
@@ -149,10 +174,17 @@ addopts = [
   '--import-mode=importlib',
   # don't show frickin captured logs AGAIN in the report..
   '--show-capture=no',
+
+  # disable `xonsh` plugin
+  # https://docs.pytest.org/en/stable/how-to/plugins.html#disabling-plugins-from-autoloading
+  # https://docs.pytest.org/en/stable/how-to/plugins.html#deactivating-unregistering-a-plugin-by-name
+  '-p no:xonsh'
 ]
 log_cli = false
 # TODO: maybe some of these layout choices?
 # https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
 # pythonpath = "src"
 
+# https://docs.pytest.org/en/stable/reference/reference.html#confval-console_output_style
+console_output_style = 'progress'
 # ------ tool.pytest ------

pytest.ini

@@ -1,8 +0,0 @@
-# vim: ft=ini
-# pytest.ini for tractor
-
-[pytest]
-# don't show frickin captured logs AGAIN in the report..
-addopts = --show-capture='no'
-log_cli = false
-; minversion = 6.0

ruff.toml

@@ -35,8 +35,8 @@ exclude = [
 line-length = 88
 indent-width = 4
 
-# Assume Python 3.9
+# assume latest minor cpython
-target-version = "py311"
+target-version = "py313"
 
 [lint]
 # Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`)  codes by default.

tests/conftest.py

@@ -1,13 +1,16 @@
 """
-``tractor`` testing!!
+Top level of the testing suites!
+
 """
+from __future__ import annotations
 import sys
 import subprocess
 import os
-import random
 import signal
 import platform
 import time
+from pathlib import Path
+from typing import Literal
 
 import pytest
 import tractor
@@ -17,8 +20,13 @@ from tractor._testing import (
     expect_ctxc as expect_ctxc,
 )
 
-# TODO: include wtv plugin(s) we build in `._testing.pytest`?
+pytest_plugins: list[str] = [
-pytest_plugins = ['pytester']
+    'pytester',
+    'tractor._testing.pytest',
+]
+
+_ci_env: bool = os.environ.get('CI', False)
+_non_linux: bool = platform.system() != 'Linux'
 
 # Sending signal.SIGINT on subprocess fails on windows. Use CTRL_* alternatives
 if platform.system() == 'Windows':
@@ -30,16 +38,98 @@ else:
     _KILL_SIGNAL = signal.SIGKILL
     _INT_SIGNAL = signal.SIGINT
     _INT_RETURN_CODE = 1 if sys.version_info < (3, 8) else -signal.SIGINT.value
-    _PROC_SPAWN_WAIT = 0.6 if sys.version_info < (3, 7) else 0.4
+    _PROC_SPAWN_WAIT = (
+        2 if _ci_env
+        else 1
+    )
 
 
 no_windows = pytest.mark.skipif(
     platform.system() == "Windows",
     reason="Test is unsupported on windows",
 )
+no_macos = pytest.mark.skipif(
+    platform.system() == "Darwin",
+    reason="Test is unsupported on MacOS",
+)
 
 
-def pytest_addoption(parser):
+def get_cpu_state(
+    icpu: int = 0,
+    setting: Literal[
+        'scaling_governor',
+        '*_pstate_max_freq',
+        'scaling_max_freq',
+        # 'scaling_cur_freq',
+    ] = '*_pstate_max_freq',
+) -> tuple[
+    Path,
+    str|int,
+]|None:
+    '''
+    Attempt to read the (first) CPU's setting according
+    to the set `setting` from under the file-sys,
+
+    /sys/devices/system/cpu/cpu0/cpufreq/{setting}
+
+    Useful to determine latency headroom for various perf affected
+    test suites.
+
+    '''
+    try:
+        # Read governor for core 0 (usually same for all)
+        setting_path: Path = list(
+            Path(f'/sys/devices/system/cpu/cpu{icpu}/cpufreq/')
+            .glob(f'{setting}')
+        )[0]  # <- XXX must be single match!
+        with open(
+            setting_path,
+            'r',
+        ) as f:
+            return (
+                setting_path,
+                f.read().strip(),
+            )
+    except (FileNotFoundError, IndexError):
+        return None
+
+
+def cpu_scaling_factor() -> float:
+    '''
+    Return a latency-headroom multiplier (>= 1.0) reflecting how
+    much to inflate time-limits when CPU-freq scaling is active on
+    linux.
+
+    When no scaling info is available (non-linux, missing sysfs),
+    returns 1.0 (i.e. no headroom adjustment needed).
+
+    '''
+    if _non_linux:
+        return 1.
+
+    mx = get_cpu_state()
+    cur = get_cpu_state(setting='scaling_max_freq')
+    if mx is None or cur is None:
+        return 1.
+
+    _mx_pth, max_freq = mx
+    _cur_pth, cur_freq = cur
+    cpu_scaled: float = int(cur_freq) / int(max_freq)
+
+    if cpu_scaled != 1.:
+        return 1. / (
+            cpu_scaled * 2  # <- bc likely "dual threaded"
+        )
+
+    return 1.
+
+
+def pytest_addoption(
+    parser: pytest.Parser,
+):
+    # ?TODO? should this be exposed from our `._testing.pytest`
+    # plugin or should we make it more explicit with `--tl` for
+    # tractor logging like we do in other client projects?
     parser.addoption(
         "--ll",
         action="store",
@@ -47,149 +137,93 @@ def pytest_addoption(parser):
         default='ERROR', help="logging level to set when testing"
     )
 
-    parser.addoption(
-        "--spawn-backend",
-        action="store",
-        dest='spawn_backend',
-        default='trio',
-        help="Processing spawning backend to use for test run",
-    )
-
-    parser.addoption(
-        "--tpdb", "--debug-mode",
-        action="store_true",
-        dest='tractor_debug_mode',
-        # default=False,
-        help=(
-            'Enable a flag that can be used by tests to to set the '
-            '`debug_mode: bool` for engaging the internal '
-            'multi-proc debugger sys.'
-        ),
-    )
-
-
-def pytest_configure(config):
-    backend = config.option.spawn_backend
-    tractor._spawn.try_set_start_method(backend)
-
-
-@pytest.fixture(scope='session')
-def debug_mode(request):
-    debug_mode: bool = request.config.option.tractor_debug_mode
-    # if debug_mode:
-    #     breakpoint()
-    return debug_mode
-
 
 @pytest.fixture(scope='session', autouse=True)
-def loglevel(request):
+def loglevel(request) -> str:
+    import tractor
     orig = tractor.log._default_loglevel
     level = tractor.log._default_loglevel = request.config.option.loglevel
-    tractor.log.get_console_log(level)
+    log = tractor.log.get_console_log(
+        level=level,
+        name='tractor',  # <- enable root logger
+    )
+    log.info(
+        f'Test-harness set runtime loglevel: {level!r}\n'
+    )
     yield level
     tractor.log._default_loglevel = orig
 
 
-@pytest.fixture(scope='session')
+@pytest.fixture(scope='function')
-def spawn_backend(request) -> str:
+def test_log(
-    return request.config.option.spawn_backend
+    request,
+    loglevel: str,
+) -> tractor.log.StackLevelAdapter:
+    '''
+    Deliver a per test-module-fn logger instance for reporting from
+    within actual test bodies/fixtures.
 
+    For example this can be handy to report certain error cases from
+    exception handlers using `test_log.exception()`.
 
-# @pytest.fixture(scope='function', autouse=True)
+    '''
-# def debug_enabled(request) -> str:
+    modname: str = request.function.__module__
-#     from tractor import _state
+    log = tractor.log.get_logger(
-#     if _state._runtime_vars['_debug_mode']:
+        name=modname,  # <- enable root logger
-#         breakpoint()
+        # pkg_name='tests',
+    )
+    _log = tractor.log.get_console_log(
+        level=loglevel,
+        logger=log,
+        name=modname,
+        # pkg_name='tests',
+    )
+    _log.debug(
+        f'In-test-logging requested\n'
+        f'test_log.name: {log.name!r}\n'
+        f'level: {loglevel!r}\n'
 
-_ci_env: bool = os.environ.get('CI', False)
+    )
+    yield _log
 
 
 @pytest.fixture(scope='session')
 def ci_env() -> bool:
     '''
-    Detect CI envoirment.
+    Detect CI environment.
 
     '''
     return _ci_env
 
 
-# TODO: also move this to `._testing` for now?
+def sig_prog(
-# -[ ] possibly generalize and re-use for multi-tree spawning
+    proc: subprocess.Popen,
-#    along with the new stuff for multi-addrs in distribute_dis
+    sig: int,
-#    branch?
+    canc_timeout: float = 0.1,
-#
+) -> int:
-# choose randomly at import time
-_reg_addr: tuple[str, int] = (
-    '127.0.0.1',
-    random.randint(1000, 9999),
-)
-
-
-@pytest.fixture(scope='session')
-def reg_addr() -> tuple[str, int]:
-
-    # globally override the runtime to the per-test-session-dynamic
-    # addr so that all tests never conflict with any other actor
-    # tree using the default.
-    from tractor import _root
-    _root._default_lo_addrs = [_reg_addr]
-
-    return _reg_addr
-
-
-def pytest_generate_tests(metafunc):
-    spawn_backend = metafunc.config.option.spawn_backend
-
-    if not spawn_backend:
-        # XXX some weird windows bug with `pytest`?
-        spawn_backend = 'trio'
-
-    # TODO: maybe just use the literal `._spawn.SpawnMethodKey`?
-    assert spawn_backend in (
-        'mp_spawn',
-        'mp_forkserver',
-        'trio',
-    )
-
-    # NOTE: used to be used to dyanmically parametrize tests for when
-    # you just passed --spawn-backend=`mp` on the cli, but now we expect
-    # that cli input to be manually specified, BUT, maybe we'll do
-    # something like this again in the future?
-    if 'start_method' in metafunc.fixturenames:
-        metafunc.parametrize("start_method", [spawn_backend], scope='module')
-
-
-# TODO: a way to let test scripts (like from `examples/`)
-# guarantee they won't registry addr collide!
-# @pytest.fixture
-# def open_test_runtime(
-#     reg_addr: tuple,
-# ) -> AsyncContextManager:
-#     return partial(
-#         tractor.open_nursery,
-#         registry_addrs=[reg_addr],
-#     )
-
-
-def sig_prog(proc, sig):
     "Kill the actor-process with ``sig``."
     proc.send_signal(sig)
-    time.sleep(0.1)
+    time.sleep(canc_timeout)
     if not proc.poll():
         # TODO: why sometimes does SIGINT not work on teardown?
         # seems to happen only when trace logging enabled?
         proc.send_signal(_KILL_SIGNAL)
-    ret = proc.wait()
+    ret: int = proc.wait()
     assert ret
 
 
 # TODO: factor into @cm and move to `._testing`?
 @pytest.fixture
 def daemon(
+    debug_mode: bool,
     loglevel: str,
-    testdir,
+    testdir: pytest.Pytester,
     reg_addr: tuple[str, int],
-):
+    tpt_proto: str,
+    ci_env: bool,
+    test_log: tractor.log.StackLevelAdapter,
+
+) -> subprocess.Popen:
     '''
     Run a daemon root actor as a separate actor-process tree and
     "remote registrar" for discovery-protocol related tests.
@@ -201,27 +235,125 @@ def daemon(
 
     code: str = (
         "import tractor; "
-            "tractor.run_daemon([], registry_addrs={reg_addrs}, loglevel={ll})"
+        "tractor.run_daemon([], "
+        "registry_addrs={reg_addrs}, "
+        "enable_transports={enable_tpts}, "
+        "debug_mode={debug_mode}, "
+        "loglevel={ll})"
     ).format(
         reg_addrs=str([reg_addr]),
+        enable_tpts=str([tpt_proto]),
         ll="'{}'".format(loglevel) if loglevel else None,
+        debug_mode=debug_mode,
     )
     cmd: list[str] = [
         sys.executable,
         '-c', code,
     ]
+    # breakpoint()
     kwargs = {}
     if platform.system() == 'Windows':
         # without this, tests hang on windows forever
         kwargs['creationflags'] = subprocess.CREATE_NEW_PROCESS_GROUP
 
-    proc = testdir.popen(
+    proc: subprocess.Popen = testdir.popen(
         cmd,
-        stdout=subprocess.PIPE,
-        stderr=subprocess.PIPE,
         **kwargs,
     )
-    assert not proc.returncode
+
+    # TODO! we should poll for the registry socket-bind to take place
+    # and only once that's done yield to the requester!
+    # -[ ] TCP: use the `._root.open_root_actor()`::`ping_tpt_socket()`
+    #      closure!
+    # -[ ] UDS: can we do something similar for 'pinging" the
+    #     file-socket?
+    #
+    global _PROC_SPAWN_WAIT
+    # UDS sockets are **really** fast to bind()/listen()/connect()
+    # so it's often required that we delay a bit more starting
+    # the first actor-tree..
+    if tpt_proto == 'uds':
+        _PROC_SPAWN_WAIT += 1.6
+
+    if _non_linux and ci_env:
+        _PROC_SPAWN_WAIT += 1
+
+    # XXX, allow time for the sub-py-proc to boot up.
+    # !TODO, see ping-polling ideas above!
     time.sleep(_PROC_SPAWN_WAIT)
+
+    assert not proc.returncode
     yield proc
     sig_prog(proc, _INT_SIGNAL)
+
+    # XXX! yeah.. just be reaaal careful with this bc sometimes it
+    # can lock up on the `_io.BufferedReader` and hang..
+    stderr: str = proc.stderr.read().decode()
+    stdout: str = proc.stdout.read().decode()
+    if (
+        stderr
+        or
+        stdout
+    ):
+        print(
+            f'Daemon actor tree produced output:\n'
+            f'{proc.args}\n'
+            f'\n'
+            f'stderr: {stderr!r}\n'
+            f'stdout: {stdout!r}\n'
+        )
+
+    if (rc := proc.returncode) != -2:
+        msg: str = (
+            f'Daemon actor tree was not cancelled !?\n'
+            f'proc.args: {proc.args!r}\n'
+            f'proc.returncode: {rc!r}\n'
+        )
+        if rc < 0:
+            raise RuntimeError(msg)
+
+        test_log.error(msg)
+
+
+# @pytest.fixture(autouse=True)
+# def shared_last_failed(pytestconfig):
+#     val = pytestconfig.cache.get("example/value", None)
+#     breakpoint()
+#     if val is None:
+#         pytestconfig.cache.set("example/value", val)
+#     return val
+
+
+# TODO: a way to let test scripts (like from `examples/`)
+# guarantee they won't `registry_addrs` collide!
+# -[ ] maybe use some kinda standard `def main()` arg-spec that
+#     we can introspect from a fixture that is called from the test
+#     body?
+# -[ ] test and figure out typing for below prototype! Bp
+#
+# @pytest.fixture
+# def set_script_runtime_args(
+#     reg_addr: tuple,
+# ) -> Callable[[...], None]:
+
+#     def import_n_partial_in_args_n_triorun(
+#         script: Path,  # under examples?
+#         **runtime_args,
+#     ) -> Callable[[], Any]:  # a `partial`-ed equiv of `trio.run()`
+
+#         # NOTE, below is taken from
+#         # `.test_advanced_faults.test_ipc_channel_break_during_stream`
+#         mod: ModuleType = import_path(
+#             examples_dir() / 'advanced_faults'
+#             / 'ipc_failure_during_stream.py',
+#             root=examples_dir(),
+#             consider_namespace_packages=False,
+#         )
+#         return partial(
+#             trio.run,
+#             partial(
+#                 mod.main,
+#                 **runtime_args,
+#             )
+#         )
+#     return import_n_partial_in_args_n_triorun

tests/devx/conftest.py

@@ -2,9 +2,13 @@
 `tractor.devx.*` tooling sub-pkg test space.
 
 '''
+from __future__ import annotations
+import platform
+import signal
 import time
 from typing import (
     Callable,
+    TYPE_CHECKING,
 )
 
 import pytest
@@ -16,7 +20,7 @@ from pexpect.spawnbase import SpawnBase
 from tractor._testing import (
     mk_cmd,
 )
-from tractor.devx._debug import (
+from tractor.devx.debug import (
     _pause_msg as _pause_msg,
     _crash_msg as _crash_msg,
     _repl_fail_msg as _repl_fail_msg,
@@ -26,14 +30,36 @@ from ..conftest import (
     _ci_env,
 )
 
+if TYPE_CHECKING:
+    from pexpect import pty_spawn
+
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+def pytest_configure(config):
+    # register custom marks to avoid warnings see,
+    # https://docs.pytest.org/en/stable/how-to/writing_plugins.html#registering-custom-markers
+    config.addinivalue_line(
+        'markers',
+        'ctlcs_bish: test will (likely) not behave under SIGINT..'
+    )
+
+# a fn that sub-instantiates a `pexpect.spawn()`
+# and returns it.
+type PexpectSpawner = Callable[
+    [str],
+    pty_spawn.spawn,
+]
+
 
 @pytest.fixture
 def spawn(
-    start_method,
+    start_method: str,
     testdir: pytest.Pytester,
     reg_addr: tuple[str, int],
 
-) -> Callable[[str], None]:
+) -> PexpectSpawner:
     '''
     Use the `pexpect` module shipped via `testdir.spawn()` to
     run an `./examples/..` script by name.
@@ -54,26 +80,65 @@ def spawn(
 
         '''
         import os
+        # disable colored tbs
         os.environ['PYTHON_COLORS'] = '0'
+        # disable all ANSI color output
+        # os.environ['NO_COLOR'] = '1'
+
+    spawned: PexpectSpawner|None = None
 
     def _spawn(
         cmd: str,
+        expect_timeout: float = 4,
         **mkcmd_kwargs,
-    ):
+    ) -> pty_spawn.spawn:
+        nonlocal spawned
         unset_colors()
-        return testdir.spawn(
+        spawned = testdir.spawn(
             cmd=mk_cmd(
                 cmd,
                 **mkcmd_kwargs,
             ),
-            expect_timeout=3,
+            expect_timeout=(timeout:=(
+                expect_timeout + 6
+                if _non_linux and _ci_env
+                else expect_timeout
+            )),
             # preexec_fn=unset_colors,
             # ^TODO? get `pytest` core to expose underlying
             # `pexpect.spawn()` stuff?
         )
+        # sanity
+        assert spawned.timeout == timeout
+        return spawned
 
     # such that test-dep can pass input script name.
-    return _spawn
+    yield _spawn  # the `PexpectSpawner`, type alias.
+
+    if (
+        spawned
+        and
+        (ptyproc := spawned.ptyproc)
+    ):
+        start: float = time.time()
+        timeout: float = 5
+        while (
+            ptyproc.isalive()
+            and
+            (
+                (_time_took := (time.time() - start))
+                 <
+                 timeout
+            )
+        ):
+            ptyproc.kill(signal.SIGINT)
+            time.sleep(0.01)
+
+        if ptyproc.isalive():
+            ptyproc.kill(signal.SIGKILL)
+
+    # TODO? ensure we've cleaned up any UDS-paths?
+    # breakpoint()
 
 
 @pytest.fixture(
@@ -99,7 +164,13 @@ def ctlc(
                 'https://github.com/goodboy/tractor/issues/320'
             )
 
-        if mark.name == 'ctlcs_bish':
+        if (
+            mark.name == 'ctlcs_bish'
+            and
+            use_ctlc
+            and
+            all(mark.args)
+        ):
             pytest.skip(
                 f'Test {node} prolly uses something from the stdlib (namely `asyncio`..)\n'
                 f'The test and/or underlying example script can *sometimes* run fine '
@@ -111,7 +182,7 @@ def ctlc(
         # XXX: disable pygments highlighting for auto-tests
         # since some envs (like actions CI) will struggle
         # the the added color-char encoding..
-        from tractor.devx._debug import TractorConfig
+        from tractor.devx.debug import TractorConfig
         TractorConfig.use_pygements = False
 
     yield use_ctlc
@@ -204,12 +275,13 @@ def assert_before(
         err_on_false=True,
         **kwargs
     )
+    return str(child.before.decode())
 
 
 def do_ctlc(
     child,
     count: int = 3,
-    delay: float = 0.1,
+    delay: float|None = None,
     patt: str|None = None,
 
     # expect repl UX to reprint the prompt after every
@@ -221,6 +293,7 @@ def do_ctlc(
 ) -> str|None:
 
     before: str|None = None
+    delay = delay or 0.1
 
     # make sure ctl-c sends don't do anything but repeat output
     for _ in range(count):
@@ -231,7 +304,10 @@ def do_ctlc(
         # if you run this test manually it works just fine..
         if expect_prompt:
             time.sleep(delay)
-            child.expect(PROMPT)
+            child.expect(
+                PROMPT,
+                timeout=(child.timeout * 2) if _ci_env else child.timeout,
+            )
             before = str(child.before.decode())
             time.sleep(delay)
 

tests/devx/test_debugger.py

@@ -1,8 +1,8 @@
 """
 That "native" debug mode better work!
 
-All these tests can be understood (somewhat) by running the equivalent
+All these tests can be understood (somewhat) by running the
-`examples/debugging/` scripts manually.
+equivalent `examples/debugging/` scripts manually.
 
 TODO:
   - none of these tests have been run successfully on windows yet but
@@ -10,10 +10,14 @@ TODO:
   - wonder if any of it'll work on OS X?
 
 """
+from __future__ import annotations
 from functools import partial
 import itertools
 import platform
 import time
+from typing import (
+    TYPE_CHECKING,
+)
 
 import pytest
 from pexpect.exceptions import (
@@ -33,6 +37,12 @@ from .conftest import (
     in_prompt_msg,
     assert_before,
 )
+from ..conftest import (
+    _ci_env,
+)
+
+if TYPE_CHECKING:
+    from ..conftest import PexpectSpawner
 
 # TODO: The next great debugger audit could be done by you!
 # - recurrent entry to breakpoint() from single actor *after* and an
@@ -44,13 +54,14 @@ from .conftest import (
 # - recurrent root errors
 
 
+_non_linux: bool = platform.system() != 'Linux'
+
 if platform.system() == 'Windows':
     pytest.skip(
         'Debugger tests have no windows support (yet)',
         allow_module_level=True,
     )
 
-
 # TODO: was trying to this xfail style but some weird bug i see in CI
 # that's happening at collect time.. pretty soon gonna dump actions i'm
 # thinkin...
@@ -186,6 +197,11 @@ def test_root_actor_bp_forever(
     child.expect(EOF)
 
 
+# skip on non-Linux CI
+@pytest.mark.ctlcs_bish(
+    _non_linux,
+    _ci_env,
+)
 @pytest.mark.parametrize(
     'do_next',
     (True, False),
@@ -251,6 +267,11 @@ def test_subactor_error(
     child.expect(EOF)
 
 
+# skip on non-Linux CI
+@pytest.mark.ctlcs_bish(
+    _non_linux,
+    _ci_env,
+)
 def test_subactor_breakpoint(
     spawn,
     ctlc: bool,
@@ -310,7 +331,6 @@ def test_subactor_breakpoint(
 
     assert in_prompt_msg(
         child, [
-        'MessagingError:',
         'RemoteActorError:',
          "('breakpoint_forever'",
          'bdb.BdbQuit',
@@ -474,8 +494,24 @@ def test_multi_daemon_subactors(
     stream.
 
     '''
-    child = spawn('multi_daemon_subactors')
+    non_linux = _non_linux
+    if non_linux and ctlc:
+        pytest.skip(
+            'Ctl-c + MacOS is too unreliable/racy for this test..\n'
+        )
+        # !TODO, if someone with more patience then i wants to muck
+        # with the timings on this please feel free to see all the
+        # `non_linux` branching logic i added on my first attempt
+        # below!
+        #
+        # my conclusion was that if i were to run the script
+        # manually, and thus as slowly as a human would, the test
+        # would and should pass as described in this test fn, however
+        # after fighting with it for >= 1hr. i decided more then
+        # likely the more extensive `linux` testing should cover most
+        # regressions.
 
+    child = spawn('multi_daemon_subactors')
     child.expect(PROMPT)
 
     # there can be a race for which subactor will acquire
@@ -505,8 +541,19 @@ def test_multi_daemon_subactors(
     else:
         raise ValueError('Neither log msg was found !?')
 
+    non_linux_delay: float = 0.3
     if ctlc:
-        do_ctlc(child)
+        do_ctlc(
+            child,
+            delay=(
+                non_linux_delay
+                if non_linux
+                else None
+            ),
+        )
+
+        if non_linux:
+            time.sleep(1)
 
     # NOTE: previously since we did not have clobber prevention
     # in the root actor this final resume could result in the debugger
@@ -528,7 +575,7 @@ def test_multi_daemon_subactors(
     # now the root actor won't clobber the bp_forever child
     # during it's first access to the debug lock, but will instead
     # wait for the lock to release, by the edge triggered
-    # ``devx._debug.Lock.no_remote_has_tty`` event before sending cancel messages
+    # ``devx.debug.Lock.no_remote_has_tty`` event before sending cancel messages
     # (via portals) to its underlings B)
 
     # at some point here there should have been some warning msg from
@@ -537,33 +584,66 @@ def test_multi_daemon_subactors(
     # assert "in use by child ('bp_forever'," in before
 
     if ctlc:
-        do_ctlc(child)
+        do_ctlc(
+            child,
+            delay=(
+                non_linux_delay
+                if non_linux
+                else None
+            ),
+        )
+
+        if non_linux:
+            time.sleep(1)
 
     # expect another breakpoint actor entry
     child.sendline('c')
     child.expect(PROMPT)
-
     try:
-        assert_before(
+        before: str = assert_before(
             child,
             bp_forev_parts,
         )
     except AssertionError:
-        assert_before(
+        before: str = assert_before(
             child,
             name_error_parts,
         )
 
     else:
         if ctlc:
-            do_ctlc(child)
+            before: str = do_ctlc(
+                child,
+                delay=(
+                    non_linux_delay
+                    if non_linux
+                    else None
+                ),
+            )
+
+            if non_linux:
+                time.sleep(1)
 
         # should crash with the 2nd name error (simulates
         # a retry) and then the root eventually (boxed) errors
         # after 1 or more further bp actor entries.
 
         child.sendline('c')
-        child.expect(PROMPT)
+        try:
+            child.expect(
+                PROMPT,
+                timeout=3,
+            )
+        except EOF:
+            before: str = child.before.decode()
+            print(
+                f'\n'
+                f'??? NEVER RXED `pdb` PROMPT ???\n'
+                f'\n'
+                f'{before}\n'
+            )
+            raise
+
         assert_before(
             child,
             name_error_parts,
@@ -683,7 +763,8 @@ def test_multi_subactors_root_errors(
 
 @has_nested_actors
 def test_multi_nested_subactors_error_through_nurseries(
-    spawn,
+    ci_env: bool,
+    spawn: PexpectSpawner,
 
     # TODO: address debugger issue for nested tree:
     # https://github.com/goodboy/tractor/issues/320
@@ -706,7 +787,16 @@ def test_multi_nested_subactors_error_through_nurseries(
 
     for send_char in itertools.cycle(['c', 'q']):
         try:
-            child.expect(PROMPT)
+            child.expect(
+                PROMPT,
+                timeout=(
+                    6 if (
+                        _non_linux
+                        and
+                        ci_env
+                    ) else -1
+                ),
+            )
             child.sendline(send_char)
             time.sleep(0.01)
 
@@ -883,6 +973,11 @@ def test_different_debug_mode_per_actor(
     )
 
 
+# skip on non-Linux CI
+@pytest.mark.ctlcs_bish(
+    _non_linux,
+    _ci_env,
+)
 def test_post_mortem_api(
     spawn,
     ctlc: bool,
@@ -919,6 +1014,7 @@ def test_post_mortem_api(
             "<Task 'name_error'",
             "NameError",
             "('child'",
+            'getattr(doggypants)',  # exc-LoC
         ]
     )
     if ctlc:
@@ -935,8 +1031,8 @@ def test_post_mortem_api(
             "<Task '__main__.main'",
             "('root'",
             "NameError",
-            "tractor.post_mortem()",
             "src_uid=('child'",
+            "tractor.post_mortem()",  # in `main()`-LoC
         ]
     )
     if ctlc:
@@ -954,6 +1050,10 @@ def test_post_mortem_api(
             "('root'",
             "NameError",
             "src_uid=('child'",
+
+            # raising line in `main()` but from crash-handling
+            # in `tractor.open_nursery()`.
+            'async with p.open_context(name_error) as (ctx, first):',
         ]
     )
     if ctlc:
@@ -1063,6 +1163,145 @@ def test_shield_pause(
     child.expect(EOF)
 
 
+@pytest.mark.parametrize(
+    'quit_early', [False, True]
+)
+def test_ctxep_pauses_n_maybe_ipc_breaks(
+    spawn: PexpectSpawner,
+    quit_early: bool,
+):
+    '''
+    Audit generator embedded `.pause()`es from within a `@context`
+    endpoint with a chan close at the end, requiring that ctl-c is
+    mashed and zombie reaper kills sub with no hangs.
+
+    '''
+    child = spawn('subactor_bp_in_ctx')
+    child.expect(PROMPT)
+
+    # 3 iters for the `gen()` pause-points
+    for i in range(3):
+        assert_before(
+            child,
+            [
+                _pause_msg,
+                "('bp_boi'",  # actor name
+                "<Task 'just_bp'",  # task name
+            ]
+        )
+        if (
+            i == 1
+            and
+            quit_early
+        ):
+            child.sendline('q')
+            child.expect(PROMPT)
+            assert_before(
+                child,
+                ["tractor._exceptions.RemoteActorError: remote task raised a 'BdbQuit'",
+                 "bdb.BdbQuit",
+                 "('bp_boi'",
+                ]
+            )
+            child.sendline('c')
+            child.expect(EOF)
+            assert_before(
+                child,
+                ["tractor._exceptions.RemoteActorError: remote task raised a 'BdbQuit'",
+                 "bdb.BdbQuit",
+                 "('bp_boi'",
+                ]
+            )
+            break  # end-of-test
+
+        child.sendline('c')
+        try:
+            child.expect(PROMPT)
+        except TIMEOUT:
+            # no prompt since we hang due to IPC chan purposely
+            # closed so verify we see error reporting as well as
+            # a failed crash-REPL request msg and can CTL-c our way
+            # out.
+
+            # ?TODO, match depending on `tpt_proto(s)`?
+            # - [ ] how can we pass it into the script tho?
+            tpt: str = 'UDS'
+            if _non_linux:
+                tpt: str = 'TCP'
+
+            assert_before(
+                child,
+                ['peer IPC channel closed abruptly?',
+                 'another task closed this fd',
+                 'Debug lock request was CANCELLED?',
+                 f"'Msgpack{tpt}Stream' was already closed locally?",
+                 f"TransportClosed: 'Msgpack{tpt}Stream' was already closed 'by peer'?",
+                ]
+
+                # XXX races on whether these show/hit?
+                 # 'Failed to REPl via `_pause()` You called `tractor.pause()` from an already cancelled scope!',
+                 # 'AssertionError',
+            )
+            # OSc(ancel) the hanging tree
+            do_ctlc(
+                child=child,
+                expect_prompt=False,
+            )
+            child.expect(EOF)
+            assert_before(
+                child,
+                ['KeyboardInterrupt'],
+            )
+
+
+def test_crash_handling_within_cancelled_root_actor(
+    spawn: PexpectSpawner,
+):
+    '''
+    Ensure that when only a root-actor is started via `open_root_actor()`
+    we can crash-handle in debug-mode despite self-cancellation.
+
+    More-or-less ensures we conditionally shield the pause in
+    `._root.open_root_actor()`'s `await debug._maybe_enter_pm()`
+    call.
+
+    '''
+    child = spawn('root_self_cancelled_w_error')
+    child.expect(PROMPT)
+
+    assert_before(
+        child,
+        [
+            "Actor.cancel_soon()` was called!",
+            "root cancelled",
+            _pause_msg,
+            "('root'",  # actor name
+        ]
+    )
+
+    child.sendline('c')
+    child.expect(PROMPT)
+    assert_before(
+        child,
+        [
+            _crash_msg,
+            "('root'",  # actor name
+            "AssertionError",
+            "assert 0",
+        ]
+    )
+
+    child.sendline('c')
+    child.expect(EOF)
+    assert_before(
+        child,
+        [
+            "AssertionError",
+            "assert 0",
+        ]
+    )
+
+
 # TODO: better error for "non-ideal" usage from the root actor.
 # -[ ] if called from an async scope emit a message that suggests
 #    using `await tractor.pause()` instead since it's less overhead

tests/devx/test_tooling.py

@@ -13,9 +13,16 @@ TODO:
   when debugging a problem inside the stack vs. in their app.
 
 '''
+from __future__ import annotations
+from contextlib import (
+    contextmanager as cm,
+)
 import os
 import signal
 import time
+from typing import (
+    TYPE_CHECKING,
+)
 
 from .conftest import (
     expect,
@@ -24,14 +31,23 @@ from .conftest import (
     PROMPT,
     _pause_msg,
 )
+from ..conftest import (
+    no_macos,
+)
+
+import pytest
 from pexpect.exceptions import (
     # TIMEOUT,
     EOF,
 )
 
+if TYPE_CHECKING:
+    from ..conftest import PexpectSpawner
 
+
+@no_macos
 def test_shield_pause(
-    spawn,
+    spawn: PexpectSpawner,
 ):
     '''
     Verify the `tractor.pause()/.post_mortem()` API works inside an
@@ -45,6 +61,7 @@ def test_shield_pause(
     expect(
         child,
         'Yo my child hanging..?',
+        timeout=3,
     )
     assert_before(
         child,
@@ -109,9 +126,11 @@ def test_shield_pause(
         child.pid,
         signal.SIGINT,
     )
+    from tractor.runtime._supervise import _shutdown_msg
     expect(
         child,
-        'Shutting down actor runtime',
+        # 'Shutting down actor runtime',
+        _shutdown_msg,
         timeout=6,
     )
     assert_before(
@@ -126,7 +145,7 @@ def test_shield_pause(
 
 
 def test_breakpoint_hook_restored(
-    spawn,
+    spawn: PexpectSpawner,
 ):
     '''
     Ensures our actor runtime sets a custom `breakpoint()` hook
@@ -140,6 +159,7 @@ def test_breakpoint_hook_restored(
     child = spawn('restore_builtin_breakpoint')
 
     child.expect(PROMPT)
+    try:
         assert_before(
             child,
             [
@@ -149,7 +169,12 @@ def test_breakpoint_hook_restored(
                 "first bp, tractor hook set",
             ]
         )
+    # XXX if the above raises `AssertionError`, without sending
+    # the final 'continue' cmd to the REPL-active sub-process,
+    # we'll hang waiting for that pexpect instance to terminate..
+    finally:
         child.sendline('c')
+
     child.expect(PROMPT)
     assert_before(
         child,
@@ -170,3 +195,117 @@ def test_breakpoint_hook_restored(
     )
     child.sendline('c')
     child.expect(EOF)
+
+
+
+_to_raise = Exception('Triggering a crash')
+
+
+@pytest.mark.parametrize(
+    'to_raise',
+    [
+        None,
+        _to_raise,
+        RuntimeError('Never crash handle this!'),
+    ],
+)
+@pytest.mark.parametrize(
+    'raise_on_exit',
+    [
+        True,
+        [type(_to_raise)],
+        False,
+    ]
+)
+def test_crash_handler_cms(
+    debug_mode: bool,
+    to_raise: Exception,
+    raise_on_exit: bool|list[Exception],
+):
+    '''
+    Verify the `.devx.open_crash_handler()` API(s) by also
+    (conveniently enough) tesing its `repl_fixture: ContextManager`
+    param support which for this suite allows use to avoid use of
+    a `pexpect`-style-test since we use the fixture to avoid actually
+    entering `PdbpREPL.iteract()` :smirk:
+
+    '''
+    import tractor
+    # import trio
+
+    # state flags
+    repl_acquired: bool = False
+    repl_released: bool = False
+
+    @cm
+    def block_repl_ux(
+        repl: tractor.devx.debug.PdbREPL,
+        maybe_bxerr: (
+            tractor.devx._debug.BoxedMaybeException
+            |None
+        ) = None,
+        enter_repl: bool = True,
+
+    ) -> bool:
+        '''
+        Set pre/post-REPL state vars and bypass actual conole
+        interaction.
+
+        '''
+        nonlocal repl_acquired, repl_released
+
+        # task: trio.Task = trio.lowlevel.current_task()
+        # print(f'pre-REPL active_task={task.name}')
+
+        print('pre-REPL')
+        repl_acquired = True
+        yield False  # never actually .interact()
+        print('post-REPL')
+        repl_released = True
+
+    try:
+        # TODO, with runtime's `debug_mode` setting
+        # -[ ] need to open runtime tho obvi..
+        #
+        # with tractor.devx.maybe_open_crash_handler(
+        #     pdb=True,
+
+        with tractor.devx.open_crash_handler(
+            raise_on_exit=raise_on_exit,
+            repl_fixture=block_repl_ux
+        ) as bxerr:
+            if to_raise is not None:
+                raise to_raise
+
+    except Exception as _exc:
+        exc = _exc
+        if (
+            raise_on_exit is True
+            or
+            type(to_raise) in raise_on_exit
+        ):
+            assert (
+                exc
+                is
+                to_raise
+                is
+                bxerr.value
+            )
+
+        else:
+            raise
+    else:
+        assert (
+            to_raise is None
+            or
+            not raise_on_exit
+            or
+            type(to_raise) not in raise_on_exit
+        )
+        assert bxerr.value is to_raise
+
+    assert bxerr.raise_on_exit == raise_on_exit
+
+    if to_raise is not None:
+        assert repl_acquired
+        assert repl_released

tests/ipc/__init__.py

@@ -0,0 +1,4 @@
+'''
+`tractor.ipc` subsystem(s)/unit testing suites.
+
+'''

tests/ipc/test_each_tpt.py

@@ -0,0 +1,113 @@
+'''
+Unit-ish tests for specific IPC transport protocol backends.
+
+'''
+from __future__ import annotations
+from pathlib import Path
+
+import pytest
+import trio
+import tractor
+from tractor import Actor
+from tractor.runtime import _state
+from tractor.discovery import _addr
+
+
+@pytest.fixture
+def bindspace_dir_str() -> str:
+
+    from tractor.runtime._state import get_rt_dir
+    rt_dir: Path = get_rt_dir()
+    bs_dir: Path = rt_dir / 'doggy'
+    bs_dir_str: str = str(bs_dir)
+    assert not bs_dir.is_dir()
+
+    yield bs_dir_str
+
+    # delete it on suite teardown.
+    # ?TODO? should we support this internally
+    # or is leaking it ok?
+    if bs_dir.is_dir():
+        bs_dir.rmdir()
+
+
+def test_uds_bindspace_created_implicitly(
+    debug_mode: bool,
+    bindspace_dir_str: str,
+):
+    registry_addr: tuple = (
+        f'{bindspace_dir_str}',
+        'registry@doggy.sock',
+    )
+    bs_dir_str: str = registry_addr[0]
+
+    # XXX, ensure bindspace-dir DNE beforehand!
+    assert not Path(bs_dir_str).is_dir()
+
+    async def main():
+        async with tractor.open_nursery(
+            enable_transports=['uds'],
+            registry_addrs=[registry_addr],
+            debug_mode=debug_mode,
+        ) as _an:
+
+            # XXX MUST be created implicitly by
+            # `.ipc._uds.start_listener()`!
+            assert Path(bs_dir_str).is_dir()
+
+            root: Actor = tractor.current_actor()
+            assert root.is_registrar
+
+            assert registry_addr in root.reg_addrs
+            assert (
+                registry_addr
+                in
+                _state._runtime_vars['_registry_addrs']
+            )
+            assert (
+                _addr.wrap_address(registry_addr)
+                in
+                root.registry_addrs
+            )
+
+    trio.run(main)
+
+
+def test_uds_double_listen_raises_connerr(
+    debug_mode: bool,
+    bindspace_dir_str: str,
+):
+    registry_addr: tuple = (
+        f'{bindspace_dir_str}',
+        'registry@doggy.sock',
+    )
+
+    async def main():
+        async with tractor.open_nursery(
+            enable_transports=['uds'],
+            registry_addrs=[registry_addr],
+            debug_mode=debug_mode,
+        ) as _an:
+
+            # runtime up
+            root: Actor = tractor.current_actor()
+
+            from tractor.ipc._uds import (
+                start_listener,
+                UDSAddress,
+            )
+            ya_bound_addr: UDSAddress = root.registry_addrs[0]
+            try:
+                await start_listener(
+                    addr=ya_bound_addr,
+                )
+            except ConnectionError as connerr:
+                assert type(src_exc := connerr.__context__) is OSError
+                assert 'Address already in use' in src_exc.args
+                # complete, exit test.
+
+            else:
+                pytest.fail('It dint raise a connerr !?')
+
+
+    trio.run(main)

tests/ipc/test_multi_tpt.py

@@ -0,0 +1,102 @@
+'''
+Verify the `enable_transports` param drives various
+per-root/sub-actor IPC endpoint/server settings.
+
+'''
+from __future__ import annotations
+
+import pytest
+import trio
+import tractor
+from tractor import (
+    Actor,
+    Portal,
+    ipc,
+    msg,
+)
+from tractor.runtime import _state
+from tractor.discovery import _addr
+
+@tractor.context
+async def chk_tpts(
+    ctx: tractor.Context,
+    tpt_proto_key: str,
+):
+    rtvars = _state._runtime_vars
+    assert (
+        tpt_proto_key
+        in
+        rtvars['_enable_tpts']
+    )
+    actor: Actor = tractor.current_actor()
+    spec: msg.types.SpawnSpec = actor._spawn_spec
+    assert spec._runtime_vars == rtvars
+
+    # ensure individual IPC ep-addr types
+    serv: ipc._server.Server = actor.ipc_server
+    addr: ipc._types.Address
+    for addr in serv.addrs:
+        assert addr.proto_key == tpt_proto_key
+
+    # Actor delegate-props enforcement
+    assert (
+        actor.accept_addrs
+        ==
+        serv.accept_addrs
+    )
+
+    await ctx.started(serv.accept_addrs)
+
+
+# TODO, parametrize over mis-matched-proto-typed `registry_addrs`
+# since i seems to work in `piker` but not exactly sure if both tcp
+# & uds are being deployed then?
+#
+@pytest.mark.parametrize(
+    'tpt_proto_key',
+    ['tcp', 'uds'],
+    ids=lambda item: f'ipc_tpt={item!r}'
+)
+def test_root_passes_tpt_to_sub(
+    tpt_proto_key: str,
+    reg_addr: tuple,
+    debug_mode: bool,
+):
+    # XXX NOTE, the `reg_addr` addr won't be the same type as the
+    # `tpt_proto_key` would deliver here unless you pass `--tpt-proto
+    # <tpt_proto_key>` on the CLI.
+    #
+    # if tpt_proto_key == 'uds':
+    #     breakpoint()
+
+    async def main():
+        async with tractor.open_nursery(
+            enable_transports=[tpt_proto_key],
+            registry_addrs=[reg_addr],
+            debug_mode=debug_mode,
+        ) as an:
+
+            assert (
+                tpt_proto_key
+                in
+                _state._runtime_vars['_enable_tpts']
+            )
+
+            ptl: Portal = await an.start_actor(
+                name='sub',
+                enable_modules=[__name__],
+            )
+            async with ptl.open_context(
+                chk_tpts,
+                tpt_proto_key=tpt_proto_key,
+            ) as (ctx, accept_addrs):
+
+                uw_addr: tuple
+                for uw_addr in accept_addrs:
+                    addr = _addr.wrap_address(uw_addr)
+                    assert addr.is_valid
+
+            # shudown sub-actor(s)
+            await an.cancel()
+
+    trio.run(main)

tests/ipc/test_server.py

@@ -0,0 +1,72 @@
+'''
+High-level `.ipc._server` unit tests.
+
+'''
+from __future__ import annotations
+
+import pytest
+import trio
+from tractor import (
+    devx,
+    ipc,
+    log,
+)
+from tractor._testing.addr import (
+    get_rando_addr,
+)
+# TODO, use/check-roundtripping with some of these wrapper types?
+#
+# from .._addr import Address
+# from ._chan import Channel
+# from ._transport import MsgTransport
+# from ._uds import UDSAddress
+# from ._tcp import TCPAddress
+
+
+@pytest.mark.parametrize(
+    '_tpt_proto',
+    ['uds', 'tcp']
+)
+def test_basic_ipc_server(
+    _tpt_proto: str,
+    debug_mode: bool,
+    loglevel: str,
+):
+
+    # so we see the socket-listener reporting on console
+    log.get_console_log("INFO")
+
+    rando_addr: tuple = get_rando_addr(
+        tpt_proto=_tpt_proto,
+    )
+    async def main():
+        async with ipc._server.open_ipc_server() as server:
+
+            assert (
+                server._parent_tn
+                and
+                server._parent_tn is server._stream_handler_tn
+            )
+            assert server._no_more_peers.is_set()
+
+            eps: list[ipc._server.Endpoint] = await server.listen_on(
+                accept_addrs=[rando_addr],
+                stream_handler_nursery=None,
+            )
+            assert (
+                len(eps) == 1
+                and
+                (ep := eps[0])._listener
+                and
+                not ep.peer_tpts
+            )
+
+            server._parent_tn.cancel_scope.cancel()
+
+        # !TODO! actually make a bg-task connection from a client
+        # using `ipc._chan._connect_chan()`
+
+    with devx.maybe_open_crash_handler(
+        pdb=debug_mode,
+    ):
+        trio.run(main)

tests/msg/__init__.py

@@ -0,0 +1,4 @@
+'''
+`tractor.msg.*` sub-sys test suite.
+
+'''

tests/msg/conftest.py

@@ -0,0 +1,4 @@
+'''
+`tractor.msg.*` test sub-pkg conf.
+
+'''

tests/msg/test_pldrx_limiting.py

@@ -61,7 +61,7 @@ async def maybe_expect_raises(
     Async wrapper for ensuring errors propagate from the inner scope.
 
     '''
-    if tractor._state.debug_mode():
+    if tractor.debug_mode():
         timeout += 999
 
     with trio.fail_after(timeout):

tests/msg/test_pretty_struct.py

@@ -0,0 +1,240 @@
+'''
+Unit tests for `tractor.msg.pretty_struct`
+private-field filtering in `pformat()`.
+
+'''
+import pytest
+
+from tractor.msg.pretty_struct import (
+    Struct,
+    pformat,
+    iter_struct_ppfmt_lines,
+)
+from tractor.msg._codec import (
+    MsgDec,
+    mk_dec,
+)
+
+
+# ------ test struct definitions ------ #
+
+class PublicOnly(Struct):
+    '''
+    All-public fields for baseline testing.
+
+    '''
+    name: str = 'alice'
+    age: int = 30
+
+
+class PrivateOnly(Struct):
+    '''
+    Only underscore-prefixed (private) fields.
+
+    '''
+    _secret: str = 'hidden'
+    _internal: int = 99
+
+
+class MixedFields(Struct):
+    '''
+    Mix of public and private fields.
+
+    '''
+    name: str = 'bob'
+    _hidden: int = 42
+    value: float = 3.14
+    _meta: str = 'internal'
+
+
+class Inner(
+    Struct,
+    frozen=True,
+):
+    '''
+    Frozen inner struct with a private field,
+    for nesting tests.
+
+    '''
+    x: int = 1
+    _secret: str = 'nope'
+
+
+class Outer(Struct):
+    '''
+    Outer struct nesting an `Inner`.
+
+    '''
+    label: str = 'outer'
+    inner: Inner = Inner()
+
+
+class EmptyStruct(Struct):
+    '''
+    Struct with zero fields.
+
+    '''
+    pass
+
+
+# ------ tests ------ #
+
+@pytest.mark.parametrize(
+    'struct_and_expected',
+    [
+        (
+            PublicOnly(),
+            {
+                'shown': ['name', 'age'],
+                'hidden': [],
+            },
+        ),
+        (
+            MixedFields(),
+            {
+                'shown': ['name', 'value'],
+                'hidden': ['_hidden', '_meta'],
+            },
+        ),
+        (
+            PrivateOnly(),
+            {
+                'shown': [],
+                'hidden': ['_secret', '_internal'],
+            },
+        ),
+    ],
+    ids=[
+        'all-public',
+        'mixed-pub-priv',
+        'all-private',
+    ],
+)
+def test_field_visibility_in_pformat(
+    struct_and_expected: tuple[
+        Struct,
+        dict[str, list[str]],
+    ],
+):
+    '''
+    Verify `pformat()` shows public fields
+    and hides `_`-prefixed private fields.
+
+    '''
+    (
+        struct,
+        expected,
+    ) = struct_and_expected
+    output: str = pformat(struct)
+
+    for field_name in expected['shown']:
+        assert field_name in output, (
+            f'{field_name!r} should appear in:\n'
+            f'{output}'
+        )
+
+    for field_name in expected['hidden']:
+        assert field_name not in output, (
+            f'{field_name!r} should NOT appear in:\n'
+            f'{output}'
+        )
+
+
+def test_iter_ppfmt_lines_skips_private():
+    '''
+    Directly verify `iter_struct_ppfmt_lines()`
+    never yields tuples with `_`-prefixed field
+    names.
+
+    '''
+    struct = MixedFields()
+    lines: list[tuple[str, str]] = list(
+        iter_struct_ppfmt_lines(
+            struct,
+            field_indent=2,
+        )
+    )
+    # should have lines for public fields only
+    assert len(lines) == 2
+
+    for _prefix, line_content in lines:
+        field_name: str = (
+            line_content.split(':')[0].strip()
+        )
+        assert not field_name.startswith('_'), (
+            f'private field leaked: {field_name!r}'
+        )
+
+
+def test_nested_struct_filters_inner_private():
+    '''
+    Verify that nested struct's private fields
+    are also filtered out during recursion.
+
+    '''
+    outer = Outer()
+    output: str = pformat(outer)
+
+    # outer's public field
+    assert 'label' in output
+
+    # inner's public field (recursed into)
+    assert 'x' in output
+
+    # inner's private field must be hidden
+    assert '_secret' not in output
+
+
+def test_empty_struct_pformat():
+    '''
+    An empty struct should produce a valid
+    `pformat()` result with no field lines.
+
+    '''
+    output: str = pformat(EmptyStruct())
+    assert 'EmptyStruct(' in output
+    assert output.rstrip().endswith(')')
+
+    # no field lines => only struct header+footer
+    lines: list[tuple[str, str]] = list(
+        iter_struct_ppfmt_lines(
+            EmptyStruct(),
+            field_indent=2,
+        )
+    )
+    assert lines == []
+
+
+def test_real_msgdec_pformat_hides_private():
+    '''
+    Verify `pformat()` on a real `MsgDec`
+    hides the `_dec` internal field.
+
+    NOTE: `MsgDec.__repr__` is custom and does
+    NOT call `pformat()`, so we call it directly.
+
+    '''
+    dec: MsgDec = mk_dec(spec=int)
+    output: str = pformat(dec)
+
+    # the private `_dec` field should be filtered
+    assert '_dec' not in output
+
+    # but the struct type name should be present
+    assert 'MsgDec(' in output
+
+
+def test_pformat_repr_integration():
+    '''
+    Verify that `Struct.__repr__()` (which calls
+    `pformat()`) also hides private fields for
+    custom structs that do NOT override `__repr__`.
+
+    '''
+    mixed = MixedFields()
+    output: str = repr(mixed)
+
+    assert 'name' in output
+    assert 'value' in output
+    assert '_hidden' not in output
+    assert '_meta' not in output

tests/test_2way.py

@@ -1,7 +1,12 @@
-"""
+'''
-Bidirectional streaming.
+Audit the simplest inter-actor bidirectional (streaming)
+msg patterns.
 
-"""
+'''
+from __future__ import annotations
+from typing import (
+    Callable,
+)
 import pytest
 import trio
 import tractor
@@ -9,10 +14,8 @@ import tractor
 
 @tractor.context
 async def simple_rpc(
-
     ctx: tractor.Context,
     data: int,
-
 ) -> None:
     '''
     Test a small ping-pong server.
@@ -39,15 +42,13 @@ async def simple_rpc(
 
 @tractor.context
 async def simple_rpc_with_forloop(
-
     ctx: tractor.Context,
     data: int,
-
 ) -> None:
-    """Same as previous test but using ``async for`` syntax/api.
+    '''
-
+    Same as previous test but using `async for` syntax/api.
-    """
 
+    '''
     # signal to parent that we're up
     await ctx.started(data + 1)
 
@@ -68,21 +69,37 @@ async def simple_rpc_with_forloop(
 
 @pytest.mark.parametrize(
     'use_async_for',
-    [True, False],
+    [
+        True,
+        False,
+    ],
+    ids='use_async_for={}'.format,
 )
 @pytest.mark.parametrize(
     'server_func',
-    [simple_rpc, simple_rpc_with_forloop],
+    [
+        simple_rpc,
+        simple_rpc_with_forloop,
+    ],
+    ids='server_func={}'.format,
 )
-def test_simple_rpc(server_func, use_async_for):
+def test_simple_rpc(
+    server_func: Callable,
+    use_async_for: bool,
+    loglevel: str,
+    debug_mode: bool,
+):
     '''
     The simplest request response pattern.
 
     '''
     async def main():
-        async with tractor.open_nursery() as n:
+        with trio.fail_after(6):
-
+            async with tractor.open_nursery(
-            portal = await n.start_actor(
+                loglevel=loglevel,
+                debug_mode=debug_mode,
+            ) as an:
+                portal: tractor.Portal = await an.start_actor(
                     'rpc_server',
                     enable_modules=[__name__],
                 )

tests/test_advanced_faults.py

@@ -10,6 +10,9 @@ import pytest
 from _pytest.pathlib import import_path
 import trio
 import tractor
+from tractor import (
+    TransportClosed,
+)
 from tractor._testing import (
     examples_dir,
     break_ipc,
@@ -74,6 +77,7 @@ def test_ipc_channel_break_during_stream(
     spawn_backend: str,
     ipc_break: dict|None,
     pre_aclose_msgstream: bool,
+    tpt_proto: str,
 ):
     '''
     Ensure we can have an IPC channel break its connection during
@@ -91,10 +95,11 @@ def test_ipc_channel_break_during_stream(
         # non-`trio` spawners should never hit the hang condition that
         # requires the user to do ctl-c to cancel the actor tree.
         # expect_final_exc = trio.ClosedResourceError
-        expect_final_exc = tractor.TransportClosed
+        expect_final_exc = TransportClosed
 
     mod: ModuleType = import_path(
-        examples_dir() / 'advanced_faults'
+        examples_dir()
+        / 'advanced_faults'
         / 'ipc_failure_during_stream.py',
         root=examples_dir(),
         consider_namespace_packages=False,
@@ -104,11 +109,14 @@ def test_ipc_channel_break_during_stream(
     # period" wherein the user eventually hits ctl-c to kill the
     # root-actor tree.
     expect_final_exc: BaseException = KeyboardInterrupt
+    expect_final_cause: BaseException|None = None
+
     if (
         # only expect EoC if trans is broken on the child side,
         ipc_break['break_child_ipc_after'] is not False
+        and
         # AND we tell the child to call `MsgStream.aclose()`.
-        and pre_aclose_msgstream
+        pre_aclose_msgstream
     ):
         # expect_final_exc = trio.EndOfChannel
         # ^XXX NOPE! XXX^ since now `.open_stream()` absorbs this
@@ -151,7 +159,8 @@ def test_ipc_channel_break_during_stream(
         ipc_break['break_child_ipc_after'] is not False
         and (
             ipc_break['break_parent_ipc_after']
-            > ipc_break['break_child_ipc_after']
+            >
+            ipc_break['break_child_ipc_after']
         )
     ):
         if pre_aclose_msgstream:
@@ -169,8 +178,8 @@ def test_ipc_channel_break_during_stream(
         and
         ipc_break['break_child_ipc_after'] is False
     ):
-        # expect_final_exc = trio.ClosedResourceError
         expect_final_exc = tractor.TransportClosed
+        expect_final_cause = trio.ClosedResourceError
 
     # BOTH but, PARENT breaks FIRST
     elif (
@@ -181,8 +190,8 @@ def test_ipc_channel_break_during_stream(
             ipc_break['break_parent_ipc_after']
         )
     ):
-        # expect_final_exc = trio.ClosedResourceError
         expect_final_exc = tractor.TransportClosed
+        expect_final_cause = trio.ClosedResourceError
 
     with pytest.raises(
         expected_exception=(
@@ -198,6 +207,7 @@ def test_ipc_channel_break_during_stream(
                     start_method=spawn_backend,
                     loglevel=loglevel,
                     pre_close=pre_aclose_msgstream,
+                    tpt_proto=tpt_proto,
                     **ipc_break,
                 )
             )
@@ -220,17 +230,29 @@ def test_ipc_channel_break_during_stream(
                 )
             cause: Exception = tc.__cause__
             assert (
-                type(cause) is trio.ClosedResourceError
+                # type(cause) is trio.ClosedResourceError
-                and
+                type(cause) is expect_final_cause
-                cause.args[0] == 'another task closed this fd'
+
+                # TODO, should we expect a certain exc-message (per
+                # tpt) as well??
+                # and
+                # cause.args[0] == 'another task closed this fd'
             )
+
             raise
 
     # get raw instance from pytest wrapper
     value = excinfo.value
     if isinstance(value, ExceptionGroup):
-        excs = value.exceptions
+        excs: tuple[Exception] = value.exceptions
-        assert len(excs) == 1
+        assert (
+            len(excs) <= 2
+            and
+            all(
+                isinstance(exc, TransportClosed)
+                for exc in excs
+            )
+        )
         final_exc = excs[0]
         assert isinstance(final_exc, expect_final_exc)
 

tests/test_advanced_streaming.py

@@ -313,9 +313,8 @@ async def inf_streamer(
         # `trio.EndOfChannel` doesn't propagate directly to the above
         # .open_stream() parent, resulting in it also raising instead
         # of gracefully absorbing as normal.. so how to handle?
-        trio.open_nursery(
+        tractor.trionics.collapse_eg(),
-            strict_exception_groups=False,
+        trio.open_nursery() as tn,
-        ) as tn,
     ):
         async def close_stream_on_sentinel():
             async for msg in stream:

tests/test_cancellation.py

@@ -17,8 +17,8 @@ from tractor._testing import (
 from .conftest import no_windows
 
 
-def is_win():
+_non_linux: bool = platform.system() != 'Linux'
-    return platform.system() == 'Windows'
+_friggin_windows: bool = platform.system() == 'Windows'
 
 
 async def assert_err(delay=0):
@@ -236,7 +236,10 @@ async def stream_forever():
 async def test_cancel_infinite_streamer(start_method):
 
     # stream for at most 1 seconds
-    with trio.move_on_after(1) as cancel_scope:
+    with (
+        trio.fail_after(4),
+        trio.move_on_after(1) as cancel_scope
+    ):
         async with tractor.open_nursery() as n:
             portal = await n.start_actor(
                 'donny',
@@ -284,20 +287,32 @@ async def test_cancel_infinite_streamer(start_method):
     ],
 )
 @tractor_test
-async def test_some_cancels_all(num_actors_and_errs, start_method, loglevel):
+async def test_some_cancels_all(
-    """Verify a subset of failed subactors causes all others in
+    num_actors_and_errs: tuple,
+    start_method: str,
+    loglevel: str,
+):
+    '''
+    Verify a subset of failed subactors causes all others in
     the nursery to be cancelled just like the strategy in trio.
 
     This is the first and only supervisory strategy at the moment.
-    """
+
-    num_actors, first_err, err_type, ria_func, da_func = num_actors_and_errs
+    '''
+    (
+        num_actors,
+        first_err,
+        err_type,
+        ria_func,
+        da_func,
+    ) = num_actors_and_errs
     try:
-        async with tractor.open_nursery() as n:
+        async with tractor.open_nursery() as an:
 
             # spawn the same number of deamon actors which should be cancelled
             dactor_portals = []
             for i in range(num_actors):
-                dactor_portals.append(await n.start_actor(
+                dactor_portals.append(await an.start_actor(
                     f'deamon_{i}',
                     enable_modules=[__name__],
                 ))
@@ -307,7 +322,7 @@ async def test_some_cancels_all(num_actors_and_errs, start_method, loglevel):
             for i in range(num_actors):
                 # start actor(s) that will fail immediately
                 riactor_portals.append(
-                    await n.run_in_actor(
+                    await an.run_in_actor(
                         func,
                         name=f'actor_{i}',
                         **kwargs
@@ -337,7 +352,8 @@ async def test_some_cancels_all(num_actors_and_errs, start_method, loglevel):
 
         # should error here with a ``RemoteActorError`` or ``MultiError``
 
-    except first_err as err:
+    except first_err as _err:
+        err = _err
         if isinstance(err, BaseExceptionGroup):
             assert len(err.exceptions) == num_actors
             for exc in err.exceptions:
@@ -348,8 +364,8 @@ async def test_some_cancels_all(num_actors_and_errs, start_method, loglevel):
         elif isinstance(err, tractor.RemoteActorError):
             assert err.boxed_type == err_type
 
-        assert n.cancelled is True
+        assert an.cancelled is True
-        assert not n._children
+        assert not an._children
     else:
         pytest.fail("Should have gotten a remote assertion error?")
 
@@ -415,7 +431,7 @@ async def test_nested_multierrors(loglevel, start_method):
             for subexc in err.exceptions:
 
                 # verify first level actor errors are wrapped as remote
-                if is_win():
+                if _friggin_windows:
 
                     # windows is often too slow and cancellation seems
                     # to happen before an actor is spawned
@@ -448,7 +464,7 @@ async def test_nested_multierrors(loglevel, start_method):
                     # XXX not sure what's up with this..
                     # on windows sometimes spawning is just too slow and
                     # we get back the (sent) cancel signal instead
-                    if is_win():
+                    if _friggin_windows:
                         if isinstance(subexc, tractor.RemoteActorError):
                             assert subexc.boxed_type in (
                                 BaseExceptionGroup,
@@ -474,7 +490,7 @@ def test_cancel_via_SIGINT(
     """Ensure that a control-C (SIGINT) signal cancels both the parent and
     child processes in trionic fashion
     """
-    pid = os.getpid()
+    pid: int = os.getpid()
 
     async def main():
         with trio.fail_after(2):
@@ -491,19 +507,31 @@ def test_cancel_via_SIGINT(
 
 @no_windows
 def test_cancel_via_SIGINT_other_task(
-    loglevel,
+    loglevel: str,
-    start_method,
+    start_method: str,
-    spawn_backend,
+    spawn_backend: str,
 ):
-    """Ensure that a control-C (SIGINT) signal cancels both the parent
+    '''
-    and child processes in trionic fashion even a subprocess is started
+    Ensure that a control-C (SIGINT) signal cancels both the parent
-    from a seperate ``trio`` child  task.
+    and child processes in trionic fashion even a subprocess is
-    """
+    started from a seperate ``trio`` child  task.
-    pid = os.getpid()
+
-    timeout: float = 2
+    '''
-    if is_win():  # smh
+    from .conftest import cpu_scaling_factor
+
+    pid: int = os.getpid()
+    timeout: float = (
+        4 if _non_linux
+        else 2
+    )
+    if _friggin_windows:  # smh
         timeout += 1
 
+    # add latency headroom for CPU freq scaling (auto-cpufreq et al.)
+    headroom: float = cpu_scaling_factor()
+    if headroom != 1.:
+        timeout *= headroom
+
     async def spawn_and_sleep_forever(
         task_status=trio.TASK_STATUS_IGNORED
     ):
@@ -519,10 +547,15 @@ def test_cancel_via_SIGINT_other_task(
     async def main():
         # should never timeout since SIGINT should cancel the current program
         with trio.fail_after(timeout):
-            async with trio.open_nursery(
+            async with (
+
+                # XXX ?TODO? why no work!?
+                # tractor.trionics.collapse_eg(),
+                trio.open_nursery(
                     strict_exception_groups=False,
-            ) as n:
+                ) as tn,
-                await n.start(spawn_and_sleep_forever)
+            ):
+                await tn.start(spawn_and_sleep_forever)
                 if 'mp' in spawn_backend:
                     time.sleep(0.1)
                 os.kill(pid, signal.SIGINT)
@@ -533,38 +566,127 @@ def test_cancel_via_SIGINT_other_task(
 
 async def spin_for(period=3):
     "Sync sleep."
+    print(f'sync sleeping in sub-sub for {period}\n')
     time.sleep(period)
 
 
-async def spawn():
+async def spawn_sub_with_sync_blocking_task():
-    async with tractor.open_nursery() as tn:
+    async with tractor.open_nursery() as an:
-        await tn.run_in_actor(
+        print('starting sync blocking subactor..\n')
+        await an.run_in_actor(
             spin_for,
             name='sleeper',
         )
+        print('exiting first subactor layer..\n')
 
 
+@pytest.mark.parametrize(
+    'man_cancel_outer',
+    [
+        False,  # passes if delay != 2
+
+        # always causes an unexpected eg-w-embedded-assert-err?
+        pytest.param(True,
+             marks=pytest.mark.xfail(
+                 reason=(
+                    'always causes an unexpected eg-w-embedded-assert-err?'
+                )
+            ),
+        ),
+    ],
+)
 @no_windows
 def test_cancel_while_childs_child_in_sync_sleep(
-    loglevel,
+    loglevel: str,
-    start_method,
+    start_method: str,
-    spawn_backend,
+    spawn_backend: str,
+    debug_mode: bool,
+    reg_addr: tuple,
+    man_cancel_outer: bool,
 ):
-    """Verify that a child cancelled while executing sync code is torn
+    '''
+    Verify that a child cancelled while executing sync code is torn
     down even when that cancellation is triggered by the parent
     2 nurseries "up".
-    """
+
+    Though the grandchild should stay blocking its actor runtime, its
+    parent should issue a "zombie reaper" to hard kill it after
+    sufficient timeout.
+
+    '''
     if start_method == 'forkserver':
         pytest.skip("Forksever sux hard at resuming from sync sleep...")
 
     async def main():
-        with trio.fail_after(2):
+        #
-            async with tractor.open_nursery() as tn:
+        # XXX BIG TODO NOTE XXX
-                await tn.run_in_actor(
+        #
-                    spawn,
+        # it seems there's a strange race that can happen
-                    name='spawn',
+        # where where the fail-after will trigger outer scope
+        # .cancel() which then causes the inner scope to raise,
+        #
+        # BaseExceptionGroup('Exceptions from Trio nursery', [
+        #   BaseExceptionGroup('Exceptions from Trio nursery',
+        #   [
+        #       Cancelled(),
+        #       Cancelled(),
+        #   ]
+        #   ),
+        #   AssertionError('assert 0')
+        # ])
+        #
+        # WHY THIS DOESN'T MAKE SENSE:
+        # ---------------------------
+        # - it should raise too-slow-error when too slow..
+        #  * verified that using simple-cs and manually cancelling
+        #    you get same outcome -> indicates that the fail-after
+        #    can have its TooSlowError overriden!
+        #  |_ to check this it's easy, simplly decrease the timeout
+        #     as per the var below.
+        #
+        # - when using the manual simple-cs the outcome is different
+        #   DESPITE the `assert 0` which means regardless of the
+        #   inner scope effectively failing in the same way, the
+        #   bubbling up **is NOT the same**.
+        #
+        # delays trigger diff outcomes..
+        # ---------------------------
+        # as seen by uncommenting various lines below there is from
+        # my POV an unexpected outcome due to the delay=2 case.
+        #
+        # delay = 1  # no AssertionError in eg, TooSlowError raised.
+        # delay = 2  # is AssertionError in eg AND no TooSlowError !?
+        # is AssertionError in eg AND no _cs cancellation.
+        delay = (
+            6 if _non_linux
+            else 4 
+        )
+
+        with trio.fail_after(delay) as _cs:
+        # with trio.CancelScope() as cs:
+        # ^XXX^ can be used instead to see same outcome.
+
+            async with (
+                # tractor.trionics.collapse_eg(),  # doesn't help
+                tractor.open_nursery(
+                    hide_tb=False,
+                    debug_mode=debug_mode,
+                    registry_addrs=[reg_addr],
+                ) as an,
+            ):
+                await an.run_in_actor(
+                    spawn_sub_with_sync_blocking_task,
+                    name='sync_blocking_sub',
                 )
                 await trio.sleep(1)
+
+                if man_cancel_outer:
+                    print('Cancelling manually in root')
+                    _cs.cancel()
+
+                # trigger exc-srced taskc down
+                # the actor tree.
+                print('RAISING IN ROOT')
                 assert 0
 
     with pytest.raises(AssertionError):
@@ -590,7 +712,7 @@ def test_fast_graceful_cancel_when_spawn_task_in_soft_proc_wait_for_daemon(
     kbi_delay = 0.5
     timeout: float = 2.9
 
-    if is_win():  # smh
+    if _friggin_windows:  # smh
         timeout += 1
 
     async def main():

tests/test_child_manages_service_nursery.py

@@ -18,16 +18,15 @@ from tractor import RemoteActorError
 
 
 async def aio_streamer(
-    from_trio: asyncio.Queue,
+    chan: tractor.to_asyncio.LinkedTaskChannel,
-    to_trio: trio.abc.SendChannel,
 ) -> trio.abc.ReceiveChannel:
 
     # required first msg to sync caller
-    to_trio.send_nowait(None)
+    chan.started_nowait(None)
 
     from itertools import cycle
     for i in cycle(range(10)):
-        to_trio.send_nowait(i)
+        chan.send_nowait(i)
         await asyncio.sleep(0.01)
 
 
@@ -69,7 +68,7 @@ async def wrapper_mngr(
         else:
             async with tractor.to_asyncio.open_channel_from(
                 aio_streamer,
-            ) as (first, from_aio):
+            ) as (from_aio, first):
                 assert not first
 
                 # cache it so next task uses broadcast receiver
@@ -117,9 +116,10 @@ async def open_actor_local_nursery(
     ctx: tractor.Context,
 ):
     global _nursery
-    async with trio.open_nursery(
+    async with (
-        strict_exception_groups=False,
+        tractor.trionics.collapse_eg(),
-    ) as tn:
+        trio.open_nursery() as tn
+    ):
         _nursery = tn
         await ctx.started()
         await trio.sleep(10)

tests/test_clustering.py

@@ -10,29 +10,39 @@ from tractor._testing import tractor_test
 MESSAGE = 'tractoring at full speed'
 
 
-def test_empty_mngrs_input_raises() -> None:
+def test_empty_mngrs_input_raises(
+    tpt_proto: str,
+) -> None:
+    # TODO, the `open_actor_cluster()` teardown hangs
+    # intermittently on UDS when `gather_contexts(mngrs=())`
+    # raises `ValueError` mid-setup; likely a race in the
+    # actor-nursery cleanup vs UDS socket shutdown. Needs
+    # a deeper look at `._clustering`/`._supervise` teardown
+    # paths with the UDS transport.
+    if tpt_proto == 'uds':
+        pytest.skip(
+            'actor-cluster teardown hangs intermittently on UDS'
+        )
 
     async def main():
-        with trio.fail_after(1):
+        with trio.fail_after(3):
             async with (
                 open_actor_cluster(
                     modules=[__name__],
 
                     # NOTE: ensure we can passthrough runtime opts
-                    loglevel='info',
+                    loglevel='cancel',
-                    # debug_mode=True,
+                    debug_mode=False,
 
                 ) as portals,
 
-                gather_contexts(
+                gather_contexts(mngrs=()),
-                    # NOTE: it's the use of inline-generator syntax
-                    # here that causes the empty input.
-                    mngrs=(
-                        p.open_context(worker) for p in portals.values()
-                    ),
-                ),
             ):
-                assert 0
+                # should fail before this?
+                assert portals
+
+                # test should fail if we mk it here!
+                assert 0, 'Should have raised val-err !?'
 
     with pytest.raises(ValueError):
         trio.run(main)
@@ -58,13 +68,27 @@ async def worker(
             print(msg)
             assert msg == MESSAGE
 
-        # TODO: does this ever cause a hang
+        # ?TODO, does this ever cause a hang?
         # assert 0
 
 
+# ?TODO, but needs a fn-scoped tpt_proto fixture..
+# @pytest.mark.no_tpt('uds')
 @tractor_test
-async def test_streaming_to_actor_cluster() -> None:
+async def test_streaming_to_actor_cluster(
+    tpt_proto: str,
+):
+    '''
+    Open an actor "cluster" using the (experimental) `._clustering`
+    API and conduct standard inter-task-ctx streaming.
 
+    '''
+    if tpt_proto == 'uds':
+        pytest.skip(
+            f'Test currently fails with tpt-proto={tpt_proto!r}\n'
+        )
+
+    with trio.fail_after(6):
         async with (
             open_actor_cluster(modules=[__name__]) as portals,
 

tests/test_context_stream_semantics.py

@@ -9,6 +9,7 @@ from itertools import count
 import math
 import platform
 from pprint import pformat
+import sys
 from typing import (
     Callable,
 )
@@ -25,7 +26,7 @@ from tractor._exceptions import (
     StreamOverrun,
     ContextCancelled,
 )
-from tractor._state import current_ipc_ctx
+from tractor.runtime._state import current_ipc_ctx
 
 from tractor._testing import (
     tractor_test,
@@ -252,7 +253,7 @@ def test_simple_context(
             pass
         except BaseExceptionGroup as beg:
             # XXX: on windows it seems we may have to expect the group error
-            from tractor._exceptions import is_multi_cancelled
+            from tractor.trionics import is_multi_cancelled
             assert is_multi_cancelled(beg)
     else:
         trio.run(main)
@@ -938,9 +939,14 @@ def test_one_end_stream_not_opened(
 
     '''
     overrunner, buf_size_increase, entrypoint = overrun_by
-    from tractor._runtime import Actor
+    from tractor.runtime._runtime import Actor
     buf_size = buf_size_increase + Actor.msg_buffer_size
 
+    timeout: float = (
+        1 if sys.platform == 'linux'
+        else 3
+    )
+
     async def main():
         async with tractor.open_nursery(
             debug_mode=debug_mode,
@@ -950,7 +956,7 @@ def test_one_end_stream_not_opened(
                 enable_modules=[__name__],
             )
 
-            with trio.fail_after(1):
+            with trio.fail_after(timeout):
                 async with portal.open_context(
                     entrypoint,
                 ) as (ctx, sent):

tests/test_discovery.py

@@ -1,22 +1,30 @@
-"""
+'''
-Actor "discovery" testing
+Discovery subsystem via a "registrar" actor scenarios.
-"""
+
+'''
 import os
 import signal
 import platform
 from functools import partial
 import itertools
+import time
+from typing import Callable
 
+import psutil
 import pytest
+import subprocess
 import tractor
+from tractor.trionics import collapse_eg
 from tractor._testing import tractor_test
 import trio
 
 
 @tractor_test
-async def test_reg_then_unreg(reg_addr):
+async def test_reg_then_unreg(
+    reg_addr: tuple,
+):
     actor = tractor.current_actor()
-    assert actor.is_arbiter
+    assert actor.is_registrar
     assert len(actor._registry) == 1  # only self is registered
 
     async with tractor.open_nursery(
@@ -24,10 +32,10 @@ async def test_reg_then_unreg(reg_addr):
     ) as n:
 
         portal = await n.start_actor('actor', enable_modules=[__name__])
-        uid = portal.channel.uid
+        uid = portal.channel.aid.uid
 
-        async with tractor.get_registry(*reg_addr) as aportal:
+        async with tractor.get_registry(reg_addr) as aportal:
-            # this local actor should be the arbiter
+            # this local actor should be the registrar
             assert actor is aportal.actor
 
             async with tractor.wait_for_actor('actor'):
@@ -79,11 +87,15 @@ async def say_hello_use_wait(
 
 
 @tractor_test
-@pytest.mark.parametrize('func', [say_hello, say_hello_use_wait])
+@pytest.mark.parametrize(
+    'func',
+    [say_hello,
+     say_hello_use_wait]
+)
 async def test_trynamic_trio(
-    func,
+    func: Callable,
-    start_method,
+    start_method: str,
-    reg_addr,
+    reg_addr: tuple,
 ):
     '''
     Root actor acting as the "director" and running one-shot-task-actors
@@ -116,7 +128,10 @@ async def stream_forever():
         await trio.sleep(0.01)
 
 
-async def cancel(use_signal, delay=0):
+async def cancel(
+    use_signal: bool,
+    delay: float = 0,
+):
     # hold on there sally
     await trio.sleep(delay)
 
@@ -129,15 +144,17 @@ async def cancel(use_signal, delay=0):
         raise KeyboardInterrupt
 
 
-async def stream_from(portal):
+async def stream_from(portal: tractor.Portal):
     async with portal.open_stream_from(stream_forever) as stream:
         async for value in stream:
             print(value)
 
 
-async def unpack_reg(actor_or_portal):
+async def unpack_reg(
+    actor_or_portal: tractor.Portal|tractor.Actor,
+):
     '''
-    Get and unpack a "registry" RPC request from the "arbiter" registry
+    Get and unpack a "registry" RPC request from the registrar
     system.
 
     '''
@@ -146,60 +163,75 @@ async def unpack_reg(actor_or_portal):
     else:
         msg = await actor_or_portal.run_from_ns('self', 'get_registry')
 
-    return {tuple(key.split('.')): val for key, val in msg.items()}
+    return {
+        tuple(key.split('.')): val
+        for key, val in msg.items()
+    }
 
 
 async def spawn_and_check_registry(
     reg_addr: tuple,
     use_signal: bool,
+    debug_mode: bool = False,
     remote_arbiter: bool = False,
     with_streaming: bool = False,
+    maybe_daemon: tuple[
+        subprocess.Popen,
+        psutil.Process,
+    ]|None = None,
 
 ) -> None:
 
+    if maybe_daemon:
+        popen, proc = maybe_daemon
+        # breakpoint()
+
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
+        debug_mode=debug_mode,
     ):
-        async with tractor.get_registry(*reg_addr) as portal:
+        async with tractor.get_registry(
+            addr=reg_addr,
+        ) as portal:
             # runtime needs to be up to call this
             actor = tractor.current_actor()
 
             if remote_arbiter:
-                assert not actor.is_arbiter
+                assert not actor.is_registrar
 
-            if actor.is_arbiter:
+            if actor.is_registrar:
-                extra = 1  # arbiter is local root actor
+                extra = 1  # registrar is local root actor
                 get_reg = partial(unpack_reg, actor)
 
             else:
                 get_reg = partial(unpack_reg, portal)
-                extra = 2  # local root actor + remote arbiter
+                extra = 2  # local root actor + remote registrar
 
             # ensure current actor is registered
-            registry = await get_reg()
+            registry: dict = await get_reg()
-            assert actor.uid in registry
+            assert actor.aid.uid in registry
 
             try:
-                async with tractor.open_nursery() as n:
+                async with tractor.open_nursery() as an:
-                    async with trio.open_nursery(
+                    async with (
-                        strict_exception_groups=False,
+                        collapse_eg(),
-                    ) as trion:
+                        trio.open_nursery() as trion,
-
+                    ):
                         portals = {}
                         for i in range(3):
                             name = f'a{i}'
                             if with_streaming:
-                                portals[name] = await n.start_actor(
+                                portals[name] = await an.start_actor(
                                     name=name, enable_modules=[__name__])
 
                             else:  # no streaming
-                                portals[name] = await n.run_in_actor(
+                                portals[name] = await an.run_in_actor(
                                     trio.sleep_forever, name=name)
 
                         # wait on last actor to come up
                         async with tractor.wait_for_actor(name):
                             registry = await get_reg()
-                            for uid in n._children:
+                            for uid in an._children:
                                 assert uid in registry
 
                         assert len(portals) + extra == len(registry)
@@ -225,21 +257,35 @@ async def spawn_and_check_registry(
 
                 # all subactors should have de-registered
                 registry = await get_reg()
+                start: float = time.time()
+                while (
+                    not (len(registry) == extra)
+                    and
+                    (time.time() - start) < 5
+                ):
+                    print(
+                        f'Waiting for remaining subs to dereg..\n'
+                        f'{registry!r}\n'
+                    )
+                    await trio.sleep(0.3)
+                else:
                     assert len(registry) == extra
-                assert actor.uid in registry
+
+                assert actor.aid.uid in registry
 
 
 @pytest.mark.parametrize('use_signal', [False, True])
 @pytest.mark.parametrize('with_streaming', [False, True])
 def test_subactors_unregister_on_cancel(
-    start_method,
+    debug_mode: bool,
-    use_signal,
+    start_method: str,
-    reg_addr,
+    use_signal: bool,
-    with_streaming,
+    reg_addr: tuple,
+    with_streaming: bool,
 ):
     '''
     Verify that cancelling a nursery results in all subactors
-    deregistering themselves with the arbiter.
+    deregistering themselves with the registrar.
 
     '''
     with pytest.raises(KeyboardInterrupt):
@@ -248,6 +294,7 @@ def test_subactors_unregister_on_cancel(
                 spawn_and_check_registry,
                 reg_addr,
                 use_signal,
+                debug_mode=debug_mode,
                 remote_arbiter=False,
                 with_streaming=with_streaming,
             ),
@@ -257,24 +304,32 @@ def test_subactors_unregister_on_cancel(
 @pytest.mark.parametrize('use_signal', [False, True])
 @pytest.mark.parametrize('with_streaming', [False, True])
 def test_subactors_unregister_on_cancel_remote_daemon(
-    daemon,
+    daemon: subprocess.Popen,
-    start_method,
+    debug_mode: bool,
-    use_signal,
+    start_method: str,
-    reg_addr,
+    use_signal: bool,
-    with_streaming,
+    reg_addr: tuple,
+    with_streaming: bool,
 ):
-    """Verify that cancelling a nursery results in all subactors
+    '''
-    deregistering themselves with a **remote** (not in the local process
+    Verify that cancelling a nursery results in all subactors
-    tree) arbiter.
+    deregistering themselves with a **remote** (not in the local
-    """
+    process tree) registrar.
+
+    '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
             partial(
                 spawn_and_check_registry,
                 reg_addr,
                 use_signal,
+                debug_mode=debug_mode,
                 remote_arbiter=True,
                 with_streaming=with_streaming,
+                maybe_daemon=(
+                    daemon,
+                    psutil.Process(daemon.pid)
+                ),
             ),
         )
 
@@ -300,34 +355,39 @@ async def close_chans_before_nursery(
     async with tractor.open_root_actor(
         registry_addrs=[reg_addr],
     ):
-        async with tractor.get_registry(*reg_addr) as aportal:
+        async with tractor.get_registry(reg_addr) as aportal:
             try:
                 get_reg = partial(unpack_reg, aportal)
 
-                async with tractor.open_nursery() as tn:
+                async with tractor.open_nursery() as an:
-                    portal1 = await tn.start_actor(
+                    portal1 = await an.start_actor(
-                        name='consumer1', enable_modules=[__name__])
+                        name='consumer1',
-                    portal2 = await tn.start_actor(
+                        enable_modules=[__name__],
-                        'consumer2', enable_modules=[__name__])
+                    )
+                    portal2 = await an.start_actor(
+                        'consumer2',
+                        enable_modules=[__name__],
+                    )
 
-                    # TODO: compact this back as was in last commit once
+                    async with (
-                    # 3.9+, see https://github.com/goodboy/tractor/issues/207
+                        portal1.open_stream_from(
-                    async with portal1.open_stream_from(
                             stream_forever
-                    ) as agen1:
+                        ) as agen1,
-                        async with portal2.open_stream_from(
+                        portal2.open_stream_from(
                             stream_forever
-                        ) as agen2:
+                        ) as agen2,
-                            async with trio.open_nursery(
+                    ):
-                                strict_exception_groups=False,
+                            async with (
-                            ) as n:
+                                collapse_eg(),
-                                n.start_soon(streamer, agen1)
+                                trio.open_nursery() as tn,
-                                n.start_soon(cancel, use_signal, .5)
+                            ):
+                                tn.start_soon(streamer, agen1)
+                                tn.start_soon(cancel, use_signal, .5)
                                 try:
                                     await streamer(agen2)
                                 finally:
                                     # Kill the root nursery thus resulting in
-                                    # normal arbiter channel ops to fail during
+                                    # normal registrar channel ops to fail during
                                     # teardown. It doesn't seem like this is
                                     # reliably triggered by an external SIGINT.
                                     # tractor.current_actor()._root_nursery.cancel_scope.cancel()
@@ -339,27 +399,30 @@ async def close_chans_before_nursery(
                                     # also kill off channels cuz why not
                                     await agen1.aclose()
                                     await agen2.aclose()
+
             finally:
                 with trio.CancelScope(shield=True):
                     await trio.sleep(1)
 
                     # all subactors should have de-registered
                     registry = await get_reg()
-                    assert portal1.channel.uid not in registry
+                    assert portal1.channel.aid.uid not in registry
-                    assert portal2.channel.uid not in registry
+                    assert portal2.channel.aid.uid not in registry
                     assert len(registry) == entries_at_end
 
 
 @pytest.mark.parametrize('use_signal', [False, True])
 def test_close_channel_explicit(
-    start_method,
+    start_method: str,
-    use_signal,
+    use_signal: bool,
-    reg_addr,
+    reg_addr: tuple,
 ):
-    """Verify that closing a stream explicitly and killing the actor's
+    '''
+    Verify that closing a stream explicitly and killing the actor's
     "root nursery" **before** the containing nursery tears down also
-    results in subactor(s) deregistering from the arbiter.
+    results in subactor(s) deregistering from the registrar.
-    """
+
+    '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
             partial(
@@ -372,16 +435,18 @@ def test_close_channel_explicit(
 
 
 @pytest.mark.parametrize('use_signal', [False, True])
-def test_close_channel_explicit_remote_arbiter(
+def test_close_channel_explicit_remote_registrar(
-    daemon,
+    daemon: subprocess.Popen,
-    start_method,
+    start_method: str,
-    use_signal,
+    use_signal: bool,
-    reg_addr,
+    reg_addr: tuple,
 ):
-    """Verify that closing a stream explicitly and killing the actor's
+    '''
+    Verify that closing a stream explicitly and killing the actor's
     "root nursery" **before** the containing nursery tears down also
-    results in subactor(s) deregistering from the arbiter.
+    results in subactor(s) deregistering from the registrar.
-    """
+
+    '''
     with pytest.raises(KeyboardInterrupt):
         trio.run(
             partial(
@@ -391,3 +456,65 @@ def test_close_channel_explicit_remote_arbiter(
                 remote_arbiter=True,
             ),
         )
+
+
+@tractor.context
+async def kill_transport(
+    ctx: tractor.Context,
+) -> None:
+
+    await ctx.started()
+    actor: tractor.Actor = tractor.current_actor()
+    actor.ipc_server.cancel()
+    await trio.sleep_forever()
+
+
+
+# @pytest.mark.parametrize('use_signal', [False, True])
+def test_stale_entry_is_deleted(
+    debug_mode: bool,
+    daemon: subprocess.Popen,
+    start_method: str,
+    reg_addr: tuple,
+):
+    '''
+    Ensure that when a stale entry is detected in the registrar's
+    table that the `find_actor()` API takes care of deleting the
+    stale entry and not delivering a bad portal.
+
+    '''
+    async def main():
+
+        name: str = 'transport_fails_actor'
+        _reg_ptl: tractor.Portal
+        an: tractor.ActorNursery
+        async with (
+            tractor.open_nursery(
+                debug_mode=debug_mode,
+                registry_addrs=[reg_addr],
+            ) as an,
+            tractor.get_registry(reg_addr) as _reg_ptl,
+        ):
+            ptl: tractor.Portal = await an.start_actor(
+                name,
+                enable_modules=[__name__],
+            )
+            async with ptl.open_context(
+                kill_transport,
+            ) as (first, ctx):
+                async with tractor.find_actor(
+                    name,
+                    registry_addrs=[reg_addr],
+                ) as maybe_portal:
+                    # because the transitive
+                    # `._discovery.maybe_open_portal()` call should
+                    # fail and implicitly call `.delete_addr()`
+                    assert maybe_portal is None
+                    registry: dict = await unpack_reg(_reg_ptl)
+                    assert ptl.chan.aid.uid not in registry
+
+                # should fail since we knocked out the IPC tpt XD
+                await ptl.cancel_actor()
+                await an.cancel()
+
+    trio.run(main)

tests/test_docs_examples.py

@@ -9,12 +9,17 @@ import sys
 import subprocess
 import platform
 import shutil
+from typing import Callable
 
 import pytest
+import tractor
 from tractor._testing import (
     examples_dir,
 )
 
+_non_linux: bool = platform.system() != 'Linux'
+_friggin_macos: bool = platform.system() == 'Darwin'
+
 
 @pytest.fixture
 def run_example_in_subproc(
@@ -66,6 +71,9 @@ def run_example_in_subproc(
         # due to backpressure!!!
         proc = testdir.popen(
             cmdargs,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
             **kwargs,
         )
         assert not proc.returncode
@@ -86,19 +94,24 @@ def run_example_in_subproc(
         for f in p[2]
 
         if (
-            '__' not in f
+            '__' not in f  # ignore any pkg-mods
-            and f[0] != '_'
+            # ignore any `__pycache__` subdir
+            and '__pycache__' not in str(p[0])
+            and f[0] != '_'  # ignore any WIP "examplel mods"
             and 'debugging' not in p[0]
             and 'integration' not in p[0]
             and 'advanced_faults' not in p[0]
             and 'multihost' not in p[0]
+            and 'trio' not in p[0]
         )
     ],
     ids=lambda t: t[1],
 )
 def test_example(
-    run_example_in_subproc,
+    run_example_in_subproc: Callable,
-    example_script,
+    example_script: str,
+    test_log: tractor.log.StackLevelAdapter,
+    ci_env: bool,
 ):
     '''
     Load and run scripts from this repo's ``examples/`` dir as a user
@@ -112,17 +125,54 @@ def test_example(
     '''
     ex_file: str = os.path.join(*example_script)
 
-    if 'rpc_bidir_streaming' in ex_file and sys.version_info < (3, 9):
+    if (
+        'rpc_bidir_streaming' in ex_file
+        and
+        sys.version_info < (3, 9)
+    ):
         pytest.skip("2-way streaming example requires py3.9 async with syntax")
 
+    if (
+        'full_fledged_streaming_service' in ex_file
+        and
+        _friggin_macos
+        and
+        ci_env
+    ):
+        pytest.skip(
+            'Streaming example is too flaky in CI\n'
+            'AND their competitor runs this CI service..\n'
+            'This test does run just fine "in person" however..'
+        )
+
+    from .conftest import cpu_scaling_factor
+
+    timeout: float = (
+        60
+        if ci_env and _non_linux
+        else 16
+    )
+
+    # add latency headroom for CPU freq scaling (auto-cpufreq et al.)
+    headroom: float = cpu_scaling_factor()
+    if headroom != 1.:
+        timeout *= headroom
+
     with open(ex_file, 'r') as ex:
         code = ex.read()
 
         with run_example_in_subproc(code) as proc:
-            proc.wait()
+            err = None
-            err, _ = proc.stderr.read(), proc.stdout.read()
+            try:
-            # print(f'STDERR: {err}')
+                if not proc.poll():
-            # print(f'STDOUT: {out}')
+                    _, err = proc.communicate(timeout=timeout)
+
+            except subprocess.TimeoutExpired as e:
+                test_log.exception(
+                    f'Example failed to finish within {timeout}s ??\n'
+                )
+                proc.kill()
+                err = e.stderr
 
             # if we get some gnarly output let's aggregate and raise
             if err:

tests/test_infected_asyncio.py

@@ -26,8 +26,8 @@ from tractor import (
     to_asyncio,
     RemoteActorError,
     ContextCancelled,
-    _state,
 )
+from tractor.runtime import _state
 from tractor.trionics import BroadcastReceiver
 from tractor._testing import expect_ctxc
 
@@ -47,12 +47,11 @@ async def sleep_and_err(
 
     # just signature placeholders for compat with
     # ``to_asyncio.open_channel_from()``
-    to_trio: trio.MemorySendChannel|None = None,
+    chan: to_asyncio.LinkedTaskChannel|None = None,
-    from_trio: asyncio.Queue|None = None,
 
 ):
-    if to_trio:
+    if chan:
-        to_trio.send_nowait('start')
+        chan.started_nowait('start')
 
     await asyncio.sleep(sleep_for)
     assert 0
@@ -234,13 +233,11 @@ async def trio_ctx(
     with trio.fail_after(1 + delay):
         try:
             async with (
-                trio.open_nursery(
+                tractor.trionics.collapse_eg(),
-                    # TODO, for new `trio` / py3.13
+                trio.open_nursery() as tn,
-                    # strict_exception_groups=False,
-                ) as tn,
                 tractor.to_asyncio.open_channel_from(
                     sleep_and_err,
-                ) as (first, chan),
+                ) as (chan, first),
             ):
 
                 assert first == 'start'
@@ -401,7 +398,7 @@ async def no_to_trio_in_args():
 
 async def push_from_aio_task(
     sequence: Iterable,
-    to_trio: trio.abc.SendChannel,
+    chan: to_asyncio.LinkedTaskChannel,
     expect_cancel: False,
     fail_early: bool,
     exit_early: bool,
@@ -409,15 +406,12 @@ async def push_from_aio_task(
 ) -> None:
 
     try:
-        # print('trying breakpoint')
-        # breakpoint()
-
         # sync caller ctx manager
-        to_trio.send_nowait(True)
+        chan.started_nowait(True)
 
         for i in sequence:
             print(f'asyncio sending {i}')
-            to_trio.send_nowait(i)
+            chan.send_nowait(i)
             await asyncio.sleep(0.001)
 
             if (
@@ -480,7 +474,7 @@ async def stream_from_aio(
                 trio_exit_early
             ))
 
-        ) as (first, chan):
+        ) as (chan, first):
 
             assert first is True
 
@@ -573,6 +567,8 @@ def test_basic_interloop_channel_stream(
     fan_out: bool,
 ):
     async def main():
+        # TODO, figure out min timeout here!
+        with trio.fail_after(6):
             async with tractor.open_nursery() as an:
                 portal = await an.run_in_actor(
                     stream_from_aio,
@@ -732,15 +728,21 @@ def test_aio_errors_and_channel_propagates_and_closes(
 
 
 async def aio_echo_server(
-    to_trio: trio.MemorySendChannel,
+    chan: to_asyncio.LinkedTaskChannel,
-    from_trio: asyncio.Queue,
 ) -> None:
+    '''
+    An IPC-msg "echo server" with msgs received and relayed by
+    a parent `trio.Task` into a child `asyncio.Task`
+    and then repeated back to that local parent (`trio.Task`)
+    and sent again back to the original calling remote actor.
 
-    to_trio.send_nowait('start')
+    '''
+    # same semantics as `trio.TaskStatus.started()`
+    chan.started_nowait('start')
 
     while True:
         try:
-            msg = await from_trio.get()
+            msg = await chan.get()
         except to_asyncio.TrioTaskExited:
             print(
                 'breaking aio echo loop due to `trio` exit!'
@@ -748,7 +750,7 @@ async def aio_echo_server(
             break
 
         # echo the msg back
-        to_trio.send_nowait(msg)
+        chan.send_nowait(msg)
 
         # if we get the terminate sentinel
         # break the echo loop
@@ -765,7 +767,10 @@ async def trio_to_aio_echo_server(
 ):
     async with to_asyncio.open_channel_from(
         aio_echo_server,
-    ) as (first, chan):
+    ) as (
+        chan,
+        first,  # value from `chan.started_nowait()` above
+    ):
         assert first == 'start'
 
         await ctx.started(first)
@@ -776,7 +781,8 @@ async def trio_to_aio_echo_server(
                 await chan.send(msg)
 
                 out = await chan.receive()
-                # echo back to parent actor-task
+
+                # echo back to parent-actor's remote parent-ctx-task!
                 await stream.send(out)
 
                 if out is None:
@@ -889,7 +895,7 @@ async def manage_file(
 
         # NOTE: turns out you don't even need to sched an aio task
         # since the original issue, even though seemingly was due to
-        # the guest-run being abandoned + a `._debug.pause()` inside
+        # the guest-run being abandoned + a `.debug.pause()` inside
         # `._runtime._async_main()` (which was originally trying to
         # debug the `.lifetime_stack` not closing), IS NOT actually
         # the core issue?
@@ -1088,6 +1094,105 @@ def test_sigint_closes_lifetime_stack(
     trio.run(main)
 
 
+
+# ?TODO asyncio.Task fn-deco?
+# -[ ] chan: to_asyncio.InterloopChannel ??
+# -[ ] do fn-sig checking at import time like @context?
+#  |_[ ] maybe name it @a(sync)io_task ??
+# @asyncio_task  <- not bad ??
+async def raise_before_started(
+    chan: to_asyncio.LinkedTaskChannel,
+) -> None:
+    '''
+    `asyncio.Task` entry point which RTEs before calling
+    `chan.started_nowait()`.
+
+    '''
+    await asyncio.sleep(0.2)
+    raise RuntimeError('Some shite went wrong before `.send_nowait()`!!')
+
+    chan.started_nowait('Uhh we shouldve RTE-d ^^ ??')
+    await asyncio.sleep(float('inf'))
+
+
+@tractor.context
+async def caching_ep(
+    ctx: tractor.Context,
+):
+
+    log = tractor.log.get_logger('caching_ep')
+    log.info('syncing via `ctx.started()`')
+    await ctx.started()
+
+    # XXX, allocate the `open_channel_from()` inside
+    # a `.trionics.maybe_open_context()`.
+    chan: to_asyncio.LinkedTaskChannel
+    async with (
+        tractor.trionics.maybe_open_context(
+            acm_func=tractor.to_asyncio.open_channel_from,
+            kwargs={
+                'target': raise_before_started,
+                # ^XXX, kwarg to `open_channel_from()`
+            },
+
+            # lock around current actor task access
+            key=tractor.current_actor().uid,
+
+        ) as (cache_hit, (clients, chan)),
+    ):
+        if cache_hit:
+            log.error(
+                'Re-using cached `.open_from_channel()` call!\n'
+            )
+
+        else:
+            log.info(
+                'Allocating SHOULD-FAIL `.open_from_channel()`\n'
+            )
+
+        await trio.sleep_forever()
+
+
+def test_aio_side_raises_before_started(
+    reg_addr: tuple[str, int],
+    debug_mode: bool,
+    loglevel: str,
+):
+    '''
+    Simulates connection-err from `piker.brokers.ib.api`..
+
+    Ensure any error raised by child-`asyncio.Task` BEFORE
+    `chan.started()`
+
+    '''
+    # delay = 999 if debug_mode else 1
+    async def main():
+        with trio.fail_after(3):
+            an: tractor.ActorNursery
+            async with tractor.open_nursery(
+                debug_mode=debug_mode,
+                loglevel=loglevel,
+            ) as an:
+                p: tractor.Portal = await an.start_actor(
+                    'lchan_cacher_that_raises_fast',
+                    enable_modules=[__name__],
+                    infect_asyncio=True,
+                )
+                async with p.open_context(
+                    caching_ep,
+                ) as (ctx, first):
+                    assert not first
+
+    with pytest.raises(
+        expected_exception=(RemoteActorError),
+    ) as excinfo:
+        trio.run(main)
+
+    # ensure `asyncio.Task` exception is bubbled
+    # allll the way erp!!
+    rae = excinfo.value
+    assert rae.boxed_type is RuntimeError
+
 # TODO: debug_mode tests once we get support for `asyncio`!
 #
 # -[ ] need tests to wrap both scripts:
@@ -1101,7 +1206,7 @@ def test_sigint_closes_lifetime_stack(
 #    => completed using `.bestow_portal(task)` inside
 #     `.to_asyncio._run_asyncio_task()` right?
 #   -[ ] translation func to get from `asyncio` task calling to 
-#     `._debug.wait_for_parent_stdin_hijack()` which does root
+#     `.debug.wait_for_parent_stdin_hijack()` which does root
 #     call to do TTY locking.
 #
 def test_sync_breakpoint():

tests/test_inter_peer_cancellation.py

@@ -11,12 +11,13 @@ import trio
 import tractor
 from tractor import (  # typing
     Actor,
-    current_actor,
-    open_nursery,
-    Portal,
     Context,
     ContextCancelled,
+    MsgStream,
+    Portal,
     RemoteActorError,
+    current_actor,
+    open_nursery,
 )
 from tractor._testing import (
     # tractor_test,
@@ -24,14 +25,10 @@ from tractor._testing import (
 )
 
 # XXX TODO cases:
-# - [ ] peer cancelled itself - so other peers should
-#   get errors reflecting that the peer was itself the .canceller?
-
 # - [x] WE cancelled the peer and thus should not see any raised
 #   `ContextCancelled` as it should be reaped silently?
 #   => pretty sure `test_context_stream_semantics::test_caller_cancels()`
 #      already covers this case?
-
 # - [x] INTER-PEER: some arbitrary remote peer cancels via
 #   Portal.cancel_actor().
 #   => all other connected peers should get that cancel requesting peer's
@@ -44,16 +41,6 @@ from tractor._testing import (
 #   that also spawned a remote task task in that same peer-parent.
 
 
-# def test_self_cancel():
-#     '''
-#     2 cases:
-#     - calls `Actor.cancel()` locally in some task
-#     - calls LocalPortal.cancel_actor()` ?
-
-#     '''
-#     ...
-
-
 @tractor.context
 async def open_stream_then_sleep_forever(
     ctx: Context,
@@ -214,7 +201,7 @@ async def stream_from_peer(
 ) -> None:
 
     # sanity
-    assert tractor._state.debug_mode() == debug_mode
+    assert tractor.debug_mode() == debug_mode
 
     peer: Portal
     try:
@@ -410,7 +397,6 @@ def test_peer_canceller(
     '''
     async def main():
         async with tractor.open_nursery(
-            # NOTE: to halt the peer tasks on ctxc, uncomment this.
             debug_mode=debug_mode,
         ) as an:
             canceller: Portal = await an.start_actor(
@@ -595,7 +581,7 @@ def test_peer_canceller(
                             assert (
                                 re.canceller
                                 ==
-                                root.uid
+                                root.aid.uid
                             )
 
                         else:  # the other 2 ctxs
@@ -604,7 +590,7 @@ def test_peer_canceller(
                                 and (
                                     re.canceller
                                     ==
-                                    canceller.channel.uid
+                                    canceller.channel.aid.uid
                                 )
                             )
 
@@ -759,7 +745,7 @@ def test_peer_canceller(
                         # -> each context should have received
                         # a silently absorbed context cancellation
                         # in its remote nursery scope.
-                        # assert ctx.chan.uid == ctx.canceller
+                        # assert ctx.chan.aid.uid == ctx.canceller
 
                     # NOTE: when an inter-peer cancellation
                     # occurred, we DO NOT expect this
@@ -807,16 +793,16 @@ async def basic_echo_server(
     ctx: Context,
     peer_name: str = 'wittle_bruv',
 
-    err_after: int|None = None,
+    err_after_imsg: int|None = None,
 
 ) -> None:
     '''
-    Just the simplest `MsgStream` echo server which resays what
+    Just the simplest `MsgStream` echo server which resays what you
-    you told it but with its uid in front ;)
+    told it but with its uid in front ;)
 
     '''
     actor: Actor = tractor.current_actor()
-    uid: tuple = actor.uid
+    uid: tuple = actor.aid.uid
     await ctx.started(uid)
     async with ctx.open_stream() as ipc:
         async for msg in ipc:
@@ -836,8 +822,9 @@ async def basic_echo_server(
             await ipc.send(resp)
 
             if (
-                err_after
+                err_after_imsg
-                and i > err_after
+                and
+                i > err_after_imsg
             ):
                 raise RuntimeError(
                     f'Simulated error in `{peer_name}`'
@@ -854,7 +841,7 @@ async def serve_subactors(
     async with open_nursery() as an:
 
         # sanity
-        assert tractor._state.debug_mode() == debug_mode
+        assert tractor.debug_mode() == debug_mode
 
         await ctx.started(peer_name)
         async with ctx.open_stream() as ipc:
@@ -870,8 +857,8 @@ async def serve_subactors(
                     f'|_{peer}\n'
                 )
                 await ipc.send((
-                    peer.chan.uid,
+                    peer.chan.aid.uid,
-                    peer.chan.raddr,
+                    peer.chan.raddr.unwrap(),
                 ))
 
         print('Spawner exiting spawn serve loop!')
@@ -893,7 +880,7 @@ async def client_req_subactor(
 ) -> None:
     # sanity
     if debug_mode:
-        assert tractor._state.debug_mode()
+        assert tractor.debug_mode()
 
     # TODO: other cases to do with sub lifetimes:
     # -[ ] test that we can have the server spawn a sub
@@ -979,9 +966,15 @@ async def tell_little_bro(
     actor_name: str,
 
     caller: str = '',
-    err_after: int|None = None,
+    err_after: float|None = None,
+    rng_seed: int = 100,
+    # NOTE, ensure ^ is large enough (on fast hw anyway)
+    # to ensure the peer cancel req arrives before the
+    # echoing dialog does itself Bp
 ):
     # contact target actor, do a stream dialog.
+    lb: Portal
+    echo_ipc: MsgStream
     async with (
         tractor.wait_for_actor(
             name=actor_name
@@ -990,19 +983,23 @@ async def tell_little_bro(
             basic_echo_server,
 
             # XXX proxy any delayed err condition
-            err_after=err_after,
+            err_after_imsg=(
+                err_after * rng_seed
+                if err_after is not None
+                else None
+            ),
         ) as (sub_ctx, first),
-
         sub_ctx.open_stream() as echo_ipc,
     ):
         actor: Actor = current_actor()
-        uid: tuple = actor.uid
+        uid: tuple = actor.aid.uid
-        for i in range(100):
+        for i in range(rng_seed):
             msg: tuple = (
                 uid,
                 i,
             )
             await echo_ipc.send(msg)
+            await trio.sleep(0.001)
             resp = await echo_ipc.receive()
             print(
                 f'{caller} => {actor_name}: {msg}\n'
@@ -1015,6 +1012,9 @@ async def tell_little_bro(
             assert sub_uid != uid
             assert _i == i
 
+    # XXX, usually should never get here!
+    # await tractor.pause()
+
 
 @pytest.mark.parametrize(
     'raise_client_error',
@@ -1022,13 +1022,16 @@ async def tell_little_bro(
 )
 @pytest.mark.parametrize(
     'raise_sub_spawn_error_after',
-    [None, 50],
+    [None, 0.5],
 )
 def test_peer_spawns_and_cancels_service_subactor(
     debug_mode: bool,
     raise_client_error: str,
     reg_addr: tuple[str, int],
-    raise_sub_spawn_error_after: int|None,
+    raise_sub_spawn_error_after: float|None,
+    loglevel: str,
+    # ^XXX, set to 'warning' to see masked-exc warnings
+    # that may transpire during actor-nursery teardown.
 ):
     # NOTE: this tests for the modden `mod wks open piker` bug
     # discovered as part of implementing workspace ctx
@@ -1042,6 +1045,7 @@ def test_peer_spawns_and_cancels_service_subactor(
     #   and the server's spawned child should cancel and terminate!
     peer_name: str = 'little_bro'
 
+
     def check_inner_rte(rae: RemoteActorError):
         '''
         Validate the little_bro's relayed inception!
@@ -1057,6 +1061,7 @@ def test_peer_spawns_and_cancels_service_subactor(
             # NOTE: to halt the peer tasks on ctxc, uncomment this.
             debug_mode=debug_mode,
             registry_addrs=[reg_addr],
+            loglevel=loglevel,
         ) as an:
             server: Portal = await an.start_actor(
                 (server_name := 'spawn_server'),
@@ -1092,7 +1097,7 @@ def test_peer_spawns_and_cancels_service_subactor(
                     ) as (client_ctx, client_says),
                 ):
                     root: Actor = current_actor()
-                    spawner_uid: tuple = spawn_ctx.chan.uid
+                    spawner_uid: tuple = spawn_ctx.chan.aid.uid
                     print(
                         f'Server says: {first}\n'
                         f'Client says: {client_says}\n'
@@ -1111,7 +1116,7 @@ def test_peer_spawns_and_cancels_service_subactor(
                     print(
                         'Sub-spawn came online\n'
                         f'portal: {sub}\n'
-                        f'.uid: {sub.actor.uid}\n'
+                        f'.uid: {sub.actor.aid.uid}\n'
                         f'chan.raddr: {sub.chan.raddr}\n'
                     )
 
@@ -1135,8 +1140,7 @@ def test_peer_spawns_and_cancels_service_subactor(
                         )
 
                     try:
-                        res = await client_ctx.result(hide_tb=False)
+                        res = await client_ctx.wait_for_result(hide_tb=False)
-
                         # in remote (relayed inception) error
                         # case, we should error on the line above!
                         if raise_sub_spawn_error_after:
@@ -1146,7 +1150,24 @@ def test_peer_spawns_and_cancels_service_subactor(
 
                         assert isinstance(res, ContextCancelled)
                         assert client_ctx.cancel_acked
-                        assert res.canceller == root.uid
+                        assert res.canceller == root.aid.uid
+                        assert not raise_sub_spawn_error_after
+
+                        # cancelling the spawner sub should
+                        # transitively cancel it's sub, the little
+                        # bruv.
+                        print('root cancelling server/client sub-actors')
+                        await spawn_ctx.cancel()
+                        async with tractor.find_actor(
+                            name=peer_name,
+                        ) as sub:
+                            assert not sub
+
+                    # XXX, only for tracing
+                    # except BaseException as _berr:
+                    #     berr = _berr
+                    #     await tractor.pause(shield=True)
+                    #     raise berr
 
                     except RemoteActorError as rae:
                         _err = rae
@@ -1163,8 +1184,8 @@ def test_peer_spawns_and_cancels_service_subactor(
                         # little_bro: a `RuntimeError`.
                         #
                         check_inner_rte(rae)
-                        assert rae.relay_uid == client.chan.uid
+                        assert rae.relay_uid == client.chan.aid.uid
-                        assert rae.src_uid == sub.chan.uid
+                        assert rae.src_uid == sub.chan.aid.uid
 
                         assert not client_ctx.cancel_acked
                         assert (
@@ -1175,19 +1196,8 @@ def test_peer_spawns_and_cancels_service_subactor(
                         raise
                         # await tractor.pause()
 
-                    else:
-                        assert not raise_sub_spawn_error_after
-
-                        # cancelling the spawner sub should
-                        # transitively cancel it's sub, the little
-                        # bruv.
-                        print('root cancelling server/client sub-actors')
-                        await spawn_ctx.cancel()
-                        async with tractor.find_actor(
-                            name=peer_name,
-                        ) as sub:
-                            assert not sub
 
+                    # await tractor.pause()
                     # await server.cancel_actor()
 
             except RemoteActorError as rae:
@@ -1200,16 +1210,16 @@ def test_peer_spawns_and_cancels_service_subactor(
 
             # since we called `.cancel_actor()`, `.cancel_ack`
             # will not be set on the ctx bc `ctx.cancel()` was not
-            # called directly fot this confext.
+            # called directly for this confext.
             except ContextCancelled as ctxc:
                 _ctxc = ctxc
                 print(
-                    f'{root.uid} caught ctxc from ctx with {client_ctx.chan.uid}\n'
+                    f'{root.aid.uid} caught ctxc from ctx with {client_ctx.chan.aid.uid}\n'
                     f'{repr(ctxc)}\n'
                 )
 
                 if not raise_sub_spawn_error_after:
-                    assert ctxc.canceller == root.uid
+                    assert ctxc.canceller == root.aid.uid
                 else:
                     assert ctxc.canceller == spawner_uid
 
@@ -1240,12 +1250,19 @@ def test_peer_spawns_and_cancels_service_subactor(
 
                 # assert spawn_ctx.cancelled_caught
 
+    async def _main():
+        with trio.fail_after(
+            3 if not debug_mode
+            else 999
+        ):
+            await main()
+
     if raise_sub_spawn_error_after:
         with pytest.raises(RemoteActorError) as excinfo:
-            trio.run(main)
+            trio.run(_main)
 
         rae: RemoteActorError = excinfo.value
         check_inner_rte(rae)
 
     else:
-        trio.run(main)
+        trio.run(_main)

tests/test_legacy_one_way_streaming.py

@@ -1,9 +1,11 @@
 """
-Streaming via async gen api
+Streaming via the, now legacy, "async-gen API".
+
 """
 import time
 from functools import partial
 import platform
+from typing import Callable
 
 import trio
 import tractor
@@ -19,7 +21,11 @@ def test_must_define_ctx():
         async def no_ctx():
             pass
 
-    assert "no_ctx must be `ctx: tractor.Context" in str(err.value)
+    assert (
+        "no_ctx must be `ctx: tractor.Context"
+        in
+        str(err.value)
+    )
 
     @tractor.stream
     async def has_ctx(ctx):
@@ -69,14 +75,14 @@ async def stream_from_single_subactor(
     async with tractor.open_nursery(
         registry_addrs=[reg_addr],
         start_method=start_method,
-    ) as nursery:
+    ) as an:
 
         async with tractor.find_actor('streamerd') as portals:
 
             if not portals:
 
                 # no brokerd actor found
-                portal = await nursery.start_actor(
+                portal = await an.start_actor(
                     'streamerd',
                     enable_modules=[__name__],
                 )
@@ -116,11 +122,22 @@ async def stream_from_single_subactor(
 
 
 @pytest.mark.parametrize(
-    'stream_func', [async_gen_stream, context_stream]
+    'stream_func',
+    [
+        async_gen_stream,
+        context_stream,
+    ],
+    ids='stream_func={}'.format
 )
-def test_stream_from_single_subactor(reg_addr, start_method, stream_func):
+def test_stream_from_single_subactor(
-    """Verify streaming from a spawned async generator.
+    reg_addr: tuple,
-    """
+    start_method: str,
+    stream_func: Callable,
+):
+    '''
+    Verify streaming from a spawned async generator.
+
+    '''
     trio.run(
         partial(
             stream_from_single_subactor,
@@ -132,10 +149,9 @@ def test_stream_from_single_subactor(reg_addr, start_method, stream_func):
 
 
 # this is the first 2 actors, streamer_1 and streamer_2
-async def stream_data(seed):
+async def stream_data(seed: int):
 
     for i in range(seed):
-
         yield i
 
         # trigger scheduler to simulate practical usage
@@ -143,15 +159,17 @@ async def stream_data(seed):
 
 
 # this is the third actor; the aggregator
-async def aggregate(seed):
+async def aggregate(seed: int):
-    """Ensure that the two streams we receive match but only stream
+    '''
+    Ensure that the two streams we receive match but only stream
     a single set of values to the parent.
-    """
+
-    async with tractor.open_nursery() as nursery:
+    '''
+    async with tractor.open_nursery() as an:
         portals = []
         for i in range(1, 3):
             # fork point
-            portal = await nursery.start_actor(
+            portal = await an.start_actor(
                 name=f'streamer_{i}',
                 enable_modules=[__name__],
             )
@@ -164,20 +182,28 @@ async def aggregate(seed):
             async with send_chan:
 
                 async with portal.open_stream_from(
-                    stream_data, seed=seed,
+                    stream_data,
+                    seed=seed,
                 ) as stream:
 
                     async for value in stream:
                         # leverage trio's built-in backpressure
                         await send_chan.send(value)
 
-            print(f"FINISHED ITERATING {portal.channel.uid}")
+            print(
+                f'FINISHED ITERATING!\n'
+                f'peer: {portal.channel.aid.uid}'
+            )
 
         # spawn 2 trio tasks to collect streams and push to a local queue
-        async with trio.open_nursery() as n:
+        async with trio.open_nursery() as tn:
 
             for portal in portals:
-                n.start_soon(push_to_chan, portal, send_chan.clone())
+                tn.start_soon(
+                    push_to_chan,
+                    portal,
+                    send_chan.clone(),
+                )
 
             # close this local task's reference to send side
             await send_chan.aclose()
@@ -194,20 +220,21 @@ async def aggregate(seed):
 
             print("FINISHED ITERATING in aggregator")
 
-        await nursery.cancel()
+        await an.cancel()
         print("WAITING on `ActorNursery` to finish")
     print("AGGREGATOR COMPLETE!")
 
 
-# this is the main actor and *arbiter*
+async def a_quadruple_example() -> list[int]:
-async def a_quadruple_example():
+    '''
-    # a nursery which spawns "actors"
+    Open the root-actor which is also a "registrar".
-    async with tractor.open_nursery() as nursery:
 
+    '''
+    async with tractor.open_nursery() as an:
         seed = int(1e3)
         pre_start = time.time()
 
-        portal = await nursery.start_actor(
+        portal = await an.start_actor(
             name='aggregator',
             enable_modules=[__name__],
         )
@@ -228,33 +255,88 @@ async def a_quadruple_example():
         return result_stream
 
 
-async def cancel_after(wait, reg_addr):
+async def cancel_after(
-    async with tractor.open_root_actor(registry_addrs=[reg_addr]):
+    wait: float,
+    reg_addr: tuple,
+) -> list[int]:
+
+    async with tractor.open_root_actor(
+        registry_addrs=[reg_addr],
+    ):
         with trio.move_on_after(wait):
             return await a_quadruple_example()
 
 
 @pytest.fixture(scope='module')
-def time_quad_ex(reg_addr, ci_env, spawn_backend):
+def time_quad_ex(
+    reg_addr: tuple,
+    ci_env: bool,
+    spawn_backend: str,
+):
+    non_linux: bool = (_sys := platform.system()) != 'Linux'
+    if ci_env and non_linux:
+        pytest.skip(f'Test is too flaky on {_sys!r} in CI')
+
     if spawn_backend == 'mp':
-        """no idea but the  mp *nix runs are flaking out here often...
+        '''
-        """
+        no idea but the  mp *nix runs are flaking out here often...
+
+        '''
         pytest.skip("Test is too flaky on mp in CI")
 
-    timeout = 7 if platform.system() in ('Windows', 'Darwin') else 4
+    timeout = 7 if non_linux else 4
     start = time.time()
-    results = trio.run(cancel_after, timeout, reg_addr)
+    results: list[int] = trio.run(
-    diff = time.time() - start
+        cancel_after,
+        timeout,
+        reg_addr,
+    )
+    diff: float = time.time() - start
     assert results
     return results, diff
 
 
-def test_a_quadruple_example(time_quad_ex, ci_env, spawn_backend):
+def test_a_quadruple_example(
-    """This also serves as a kind of "we'd like to be this fast test"."""
+    time_quad_ex: tuple[list[int], float],
+    ci_env: bool,
+    spawn_backend: str,
+    test_log: tractor.log.StackLevelAdapter,
+):
+    '''
+    This also serves as a "we'd like to be this fast" smoke test
+    given past empirical eval of this suite.
+
+    '''
+    non_linux: bool = (_sys := platform.system()) != 'Linux'
+
+    this_fast_on_linux: float = 3
+    this_fast = (
+        6 if non_linux
+        else this_fast_on_linux
+    )
+    # ^ XXX NOTE,
+    # i've noticed that tweaking the CPU governor setting
+    # to not "always" enable "turbo" mode can result in latency
+    # which causes this limit to be too little. Not sure if it'd
+    # be worth it to adjust the linux value based on reading the
+    # CPU conf from the sys?
+    #
+    # For ex, see the `auto-cpufreq` docs on such settings,
+    # https://github.com/AdnanHodzic/auto-cpufreq?tab=readme-ov-file#example-config-file-contents
+    #
+    # HENCE this below latency-headroom compensation logic..
+    from .conftest import cpu_scaling_factor
+    headroom: float = cpu_scaling_factor()
+    if headroom != 1.:
+        this_fast = this_fast_on_linux * headroom
+        test_log.warning(
+            f'Adding latency headroom on linux bc CPU scaling,\n'
+            f'headroom: {headroom}\n'
+            f'this_fast_on_linux: {this_fast_on_linux} -> {this_fast}\n'
+        )
 
     results, diff = time_quad_ex
     assert results
-    this_fast = 6 if platform.system() in ('Windows', 'Darwin') else 3
     assert diff < this_fast
 
 
@@ -263,43 +345,58 @@ def test_a_quadruple_example(time_quad_ex, ci_env, spawn_backend):
     list(map(lambda i: i/10, range(3, 9)))
 )
 def test_not_fast_enough_quad(
-    reg_addr, time_quad_ex, cancel_delay, ci_env, spawn_backend
+    reg_addr: tuple,
+    time_quad_ex: tuple[list[int], float],
+    cancel_delay: float,
+    ci_env: bool,
+    spawn_backend: str,
 ):
-    """Verify we can cancel midway through the quad example and all actors
+    '''
-    cancel gracefully.
+    Verify we can cancel midway through the quad example and all
-    """
+    actors cancel gracefully.
+
+    '''
     results, diff = time_quad_ex
     delay = max(diff - cancel_delay, 0)
-    results = trio.run(cancel_after, delay, reg_addr)
+    results = trio.run(
-    system = platform.system()
+        cancel_after,
-    if system in ('Windows', 'Darwin') and results is not None:
+        delay,
+        reg_addr,
+    )
+    system: str = platform.system()
+    if (
+        system in ('Windows', 'Darwin')
+        and
+        results is not None
+    ):
         # In CI envoirments it seems later runs are quicker then the first
         # so just ignore these
-        print(f"Woa there {system} caught your breath eh?")
+        print(f'Woa there {system} caught your breath eh?')
     else:
         # should be cancelled mid-streaming
         assert results is None
 
 
-@tractor_test
+@tractor_test(timeout=20)
 async def test_respawn_consumer_task(
-    reg_addr,
+    reg_addr: tuple,
-    spawn_backend,
+    spawn_backend: str,
-    loglevel,
+    loglevel: str,
 ):
-    """Verify that ``._portal.ReceiveStream.shield()``
+    '''
+    Verify that ``._portal.ReceiveStream.shield()``
     sucessfully protects the underlying IPC channel from being closed
     when cancelling and respawning a consumer task.
 
     This also serves to verify that all values from the stream can be
     received despite the respawns.
 
-    """
+    '''
     stream = None
 
-    async with tractor.open_nursery() as n:
+    async with tractor.open_nursery() as an:
 
-        portal = await n.start_actor(
+        portal = await an.start_actor(
             name='streamer',
             enable_modules=[__name__]
         )

tests/test_local.py

@@ -1,5 +1,5 @@
 """
-Arbiter and "local" actor api
+Registrar and "local" actor api
 """
 import time
 
@@ -12,11 +12,11 @@ from tractor._testing import tractor_test
 
 @pytest.mark.trio
 async def test_no_runtime():
-    """An arbitter must be established before any nurseries
+    """A registrar must be established before any nurseries
     can be created.
 
-    (In other words ``tractor.open_root_actor()`` must be engaged at
+    (In other words ``tractor.open_root_actor()`` must be
-    some point?)
+    engaged at some point?)
     """
     with pytest.raises(RuntimeError) :
         async with tractor.find_actor('doggy'):
@@ -25,9 +25,9 @@ async def test_no_runtime():
 
 @tractor_test
 async def test_self_is_registered(reg_addr):
-    "Verify waiting on the arbiter to register itself using the standard api."
+    "Verify waiting on the registrar to register itself using the standard api."
     actor = tractor.current_actor()
-    assert actor.is_arbiter
+    assert actor.is_registrar
     with trio.fail_after(0.2):
         async with tractor.wait_for_actor('root') as portal:
             assert portal.channel.uid[0] == 'root'
@@ -35,11 +35,11 @@ async def test_self_is_registered(reg_addr):
 
 @tractor_test
 async def test_self_is_registered_localportal(reg_addr):
-    "Verify waiting on the arbiter to register itself using a local portal."
+    "Verify waiting on the registrar to register itself using a local portal."
     actor = tractor.current_actor()
-    assert actor.is_arbiter
+    assert actor.is_registrar
-    async with tractor.get_registry(*reg_addr) as portal:
+    async with tractor.get_registry(reg_addr) as portal:
-        assert isinstance(portal, tractor._portal.LocalPortal)
+        assert isinstance(portal, tractor.runtime._portal.LocalPortal)
 
         with trio.fail_after(0.2):
             sockaddr = await portal.run_from_ns(
@@ -57,8 +57,8 @@ def test_local_actor_async_func(reg_addr):
         async with tractor.open_root_actor(
             registry_addrs=[reg_addr],
         ):
-            # arbiter is started in-proc if dne
+            # registrar is started in-proc if dne
-            assert tractor.current_actor().is_arbiter
+            assert tractor.current_actor().is_registrar
 
             for i in range(10):
                 nums.append(i)

tests/test_log_sys.py

@@ -0,0 +1,185 @@
+'''
+`tractor.log`-wrapping unit tests.
+
+'''
+from pathlib import Path
+import shutil
+from types import ModuleType
+
+import pytest
+import tractor
+from tractor import (
+    _code_load,
+    log,
+)
+
+
+def test_root_pkg_not_duplicated_in_logger_name():
+    '''
+    When both `pkg_name` and `name` are passed and they have
+    a common `<root_name>.< >` prefix, ensure that it is not
+    duplicated in the child's `StackLevelAdapter.name: str`.
+
+    '''
+    project_name: str = 'pylib'
+    pkg_path: str = 'pylib.subpkg.mod'
+
+    assert not tractor.current_actor(
+        err_on_no_runtime=False,
+    )
+    proj_log = log.get_logger(
+        pkg_name=project_name,
+        mk_sublog=False,
+    )
+
+    sublog = log.get_logger(
+        pkg_name=project_name,
+        name=pkg_path,
+    )
+
+    assert proj_log is not sublog
+    assert sublog.name.count(proj_log.name) == 1
+    assert 'mod' not in sublog.name
+
+
+def test_implicit_mod_name_applied_for_child(
+    testdir: pytest.Pytester,
+    loglevel: str,
+):
+    '''
+    Verify that when `.log.get_logger(pkg_name='pylib')` is called
+    from a given sub-mod from within the `pylib` pkg-path, we
+    implicitly set the equiv of `name=__name__` from the caller's
+    module.
+
+    '''
+    # tractor.log.get_console_log(level=loglevel)
+    proj_name: str = 'snakelib'
+    mod_code: str = (
+        f'import tractor\n'
+        f'\n'
+        # if you need to trace `testdir` stuff @ import-time..
+        # f'breakpoint()\n'
+        f'log = tractor.log.get_logger(pkg_name="{proj_name}")\n'
+    )
+
+    # create a sub-module for each pkg layer
+    _lib = testdir.mkpydir(proj_name)
+    pkg: Path = Path(_lib)
+    pkg_init_mod: Path = pkg / "__init__.py"
+    pkg_init_mod.write_text(mod_code)
+
+    subpkg: Path = pkg / 'subpkg'
+    subpkg.mkdir()
+    subpkgmod: Path = subpkg / "__init__.py"
+    subpkgmod.touch()
+    subpkgmod.write_text(mod_code)
+
+    _submod: Path = testdir.makepyfile(
+        _mod=mod_code,
+    )
+
+    pkg_submod = pkg / 'mod.py'
+    pkg_subpkg_submod = subpkg / 'submod.py'
+    shutil.copyfile(
+        _submod,
+        pkg_submod,
+    )
+    shutil.copyfile(
+        _submod,
+        pkg_subpkg_submod,
+    )
+    testdir.chdir()
+    # NOTE, to introspect the py-file-module-layout use (in .xsh
+    # syntax): `ranger @str(testdir)`
+
+    # XXX NOTE, once the "top level" pkg mod has been
+    # imported, we can then use `import` syntax to
+    # import it's sub-pkgs and modules.
+    subpkgmod: ModuleType = _code_load.load_module_from_path(
+        Path(pkg / '__init__.py'),
+        module_name=proj_name,
+    )
+
+    pkg_root_log = log.get_logger(
+        pkg_name=proj_name,
+        mk_sublog=False,
+    )
+    # the top level pkg-mod, created just now,
+    # by above API call.
+    assert pkg_root_log.name == proj_name
+    assert not pkg_root_log.logger.getChildren()
+    #
+    # ^TODO! test this same output but created via a `get_logger()`
+    # call in the `snakelib.__init__py`!!
+
+    # NOTE, the pkg-level "init mod" should of course
+    # have the same name as the package ns-path.
+    import snakelib as init_mod
+    assert init_mod.log.name == proj_name
+
+    # NOTE, a first-pkg-level sub-module should only
+    # use the package-name since the leaf-node-module
+    # will be included in log headers by default.
+    from snakelib import mod
+    assert mod.log.name == proj_name
+
+    from snakelib import subpkg
+    assert (
+        subpkg.log.name
+        ==
+        subpkg.__package__ 
+        ==
+        f'{proj_name}.subpkg'
+    )
+
+    from snakelib.subpkg import submod
+    assert (
+        submod.log.name
+        ==
+        submod.__package__ 
+        ==
+        f'{proj_name}.subpkg'
+    )
+
+    sub_logs = pkg_root_log.logger.getChildren()
+    assert len(sub_logs) == 1  # only one nested sub-pkg module
+    assert submod.log.logger in sub_logs
+
+
+# TODO, moar tests against existing feats:
+# ------ - ------
+# - [ ] color settings?
+# - [ ] header contents like,
+#   - actor + thread + task names from various conc-primitives,
+# - [ ] `StackLevelAdapter` extensions,
+#   - our custom levels/methods: `transport|runtime|cance|pdb|devx`
+# - [ ] custom-headers support?
+#
+
+# TODO, test driven dev of new-ideas/long-wanted feats,
+# ------ - ------
+# - [ ] https://github.com/goodboy/tractor/issues/244
+#  - [ ] @catern mentioned using a sync / deterministic sys
+#       and in particular `svlogd`?
+#       |_ https://smarden.org/runit/svlogd.8
+
+# - [ ] using adapter vs. filters?
+#    - https://stackoverflow.com/questions/60691759/add-information-to-every-log-message-in-python-logging/61830838#61830838
+
+# - [ ] `.at_least_level()` optimization which short circuits wtv
+#      `logging` is doing behind the scenes when the level filters
+#      the emission..?
+
+# - [ ] use of `.log.get_console_log()` in subactors and the
+#    subtleties of ensuring it actually emits from a subproc.
+
+# - [ ] this idea of activating per-subsys emissions with some
+#    kind of `.name` filter passed to the runtime or maybe configured
+#    via the root `StackLevelAdapter`?
+
+# - [ ] use of `logging.dict.dictConfig()` to simplify the impl
+#      of any of ^^ ??
+#    - https://stackoverflow.com/questions/7507825/where-is-a-complete-example-of-logging-config-dictconfig
+#    - https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema
+#    - https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig

tests/test_multi_program.py

@@ -1,8 +1,13 @@
 """
 Multiple python programs invoking the runtime.
 """
+from __future__ import annotations
 import platform
+import subprocess
 import time
+from typing import (
+    TYPE_CHECKING,
+)
 
 import pytest
 import trio
@@ -10,14 +15,32 @@ import tractor
 from tractor._testing import (
     tractor_test,
 )
+from tractor import (
+    current_actor,
+    Actor,
+    Context,
+    Portal,
+)
+from tractor.runtime import _state
 from .conftest import (
     sig_prog,
     _INT_SIGNAL,
     _INT_RETURN_CODE,
 )
 
+if TYPE_CHECKING:
+    from tractor.msg import Aid
+    from tractor.discovery._addr import (
+        UnwrappedAddress,
+    )
 
-def test_abort_on_sigint(daemon):
+
+_non_linux: bool = platform.system() != 'Linux'
+
+
+def test_abort_on_sigint(
+    daemon: subprocess.Popen,
+):
     assert daemon.returncode is None
     time.sleep(0.1)
     sig_prog(daemon, _INT_SIGNAL)
@@ -30,39 +53,131 @@ def test_abort_on_sigint(daemon):
 
 
 @tractor_test
-async def test_cancel_remote_arbiter(daemon, reg_addr):
+async def test_cancel_remote_registrar(
-    assert not tractor.current_actor().is_arbiter
+    daemon: subprocess.Popen,
-    async with tractor.get_registry(*reg_addr) as portal:
+    reg_addr: UnwrappedAddress,
+):
+    assert not current_actor().is_registrar
+    async with tractor.get_registry(reg_addr) as portal:
         await portal.cancel_actor()
 
     time.sleep(0.1)
-    # the arbiter channel server is cancelled but not its main task
+    # the registrar channel server is cancelled but not its main task
     assert daemon.returncode is None
 
-    # no arbiter socket should exist
+    # no registrar socket should exist
     with pytest.raises(OSError):
-        async with tractor.get_registry(*reg_addr) as portal:
+        async with tractor.get_registry(reg_addr) as portal:
             pass
 
 
-def test_register_duplicate_name(daemon, reg_addr):
+def test_register_duplicate_name(
-
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+):
     async def main():
-
         async with tractor.open_nursery(
             registry_addrs=[reg_addr],
-        ) as n:
+        ) as an:
 
-            assert not tractor.current_actor().is_arbiter
+            assert not current_actor().is_registrar
 
-            p1 = await n.start_actor('doggy')
+            p1 = await an.start_actor('doggy')
-            p2 = await n.start_actor('doggy')
+            p2 = await an.start_actor('doggy')
 
             async with tractor.wait_for_actor('doggy') as portal:
                 assert portal.channel.uid in (p2.channel.uid, p1.channel.uid)
 
-            await n.cancel()
+            await an.cancel()
 
-    # run it manually since we want to start **after**
+    # XXX, run manually since we want to start this root **after**
-    # the other "daemon" program
+    # the other "daemon" program with it's own root.
+    trio.run(main)
+
+
+@tractor.context
+async def get_root_portal(
+    ctx: Context,
+):
+    '''
+    Connect back to the root actor manually (using `._discovery` API)
+    and ensure it's contact info is the same as our immediate parent.
+
+    '''
+    sub: Actor = current_actor()
+    rtvs: dict = _state._runtime_vars
+    raddrs: list[UnwrappedAddress] = rtvs['_root_addrs']
+
+    # await tractor.pause()
+    # XXX, in case the sub->root discovery breaks you might need
+    # this (i know i did Xp)!!
+    # from tractor.devx import mk_pdb
+    # mk_pdb().set_trace()
+
+    assert (
+        len(raddrs) == 1
+        and
+        list(sub._parent_chan.raddr.unwrap()) in raddrs
+    )
+
+    # connect back to our immediate parent which should also
+    # be the actor-tree's root.
+    from tractor.discovery._discovery import get_root
+    ptl: Portal
+    async with get_root() as ptl:
+        root_aid: Aid = ptl.chan.aid
+        parent_ptl: Portal = current_actor().get_parent()
+        assert (
+            root_aid.name == 'root'
+            and
+            parent_ptl.chan.aid == root_aid
+        )
+        await ctx.started()
+
+
+def test_non_registrar_spawns_child(
+    daemon: subprocess.Popen,
+    reg_addr: UnwrappedAddress,
+    loglevel: str,
+    debug_mode: bool,
+    ci_env: bool,
+):
+    '''
+    Ensure a non-regristar (serving) root actor can spawn a sub and
+    that sub can connect back (manually) to it's rent that is the
+    root without issue.
+
+    More or less this audits the global contact info in
+    `._state._runtime_vars`.
+
+    '''
+    async def main():
+
+        # XXX, since apparently on macos in GH's CI it can be a race
+        # with the `daemon` registrar on grabbing the socket-addr..
+        if ci_env and _non_linux:
+            await trio.sleep(.5)
+
+        async with tractor.open_nursery(
+            registry_addrs=[reg_addr],
+            loglevel=loglevel,
+            debug_mode=debug_mode,
+        ) as an:
+
+            actor: Actor = tractor.current_actor()
+            assert not actor.is_registrar
+            sub_ptl: Portal = await an.start_actor(
+                name='sub',
+                enable_modules=[__name__],
+            )
+
+            async with sub_ptl.open_context(
+                get_root_portal,
+            ) as (ctx, _):
+                print('Waiting for `sub` to connect back to us..')
+
+            await an.cancel()
+
+    # XXX, run manually since we want to start this root **after**
+    # the other "daemon" program with it's own root.
     trio.run(main)

tests/test_oob_cancellation.py

@@ -0,0 +1,239 @@
+'''
+Define the details of inter-actor "out-of-band" (OoB) cancel
+semantics, that is how cancellation works when a cancel request comes
+from the different concurrency (primitive's) "layer" then where the
+eventual `trio.Task` actually raises a signal.
+
+'''
+from functools import partial
+# from contextlib import asynccontextmanager as acm
+# import itertools
+
+import pytest
+import trio
+import tractor
+from tractor import (  # typing
+    ActorNursery,
+    Portal,
+    Context,
+    # ContextCancelled,
+    # RemoteActorError,
+)
+# from tractor._testing import (
+#     tractor_test,
+#     expect_ctxc,
+# )
+
+# XXX TODO cases:
+# - [ ] peer cancelled itself - so other peers should
+#   get errors reflecting that the peer was itself the .canceller?
+
+# def test_self_cancel():
+#     '''
+#     2 cases:
+#     - calls `Actor.cancel()` locally in some task
+#     - calls LocalPortal.cancel_actor()` ?
+#
+# things to ensure!
+# -[ ] the ctxc raised in a child should ideally show the tb of the
+#     underlying `Cancelled` checkpoint, i.e.
+#     `raise scope_error from ctxc`?
+#
+# -[ ] a self-cancelled context, if not allowed to block on
+#     `ctx.result()` at some point will hang since the `ctx._scope`
+#     is never `.cancel_called`; cases for this include,
+#     - an `open_ctx()` which never starteds before being OoB actor
+#       cancelled.
+#       |_ parent task will be blocked in `.open_context()` for the
+#         `Started` msg, and when the OoB ctxc arrives `ctx._scope`
+#         will never have been signalled..
+
+#     '''
+#     ...
+
+# TODO, sanity test against the case in `/examples/trio/lockacquire_not_unmasked.py`
+# but with the `Lock.acquire()` from a `@context` to ensure the
+# implicit ignore-case-non-unmasking.
+#
+# @tractor.context
+# async def acquire_actor_global_lock(
+#     ctx: tractor.Context,
+#     ignore_special_cases: bool,
+# ):
+
+#     async with maybe_unmask_excs(
+#         ignore_special_cases=ignore_special_cases,
+#     ):
+#         await ctx.started('locked')
+
+#     # block til cancelled
+#     await trio.sleep_forever()
+
+
+@tractor.context
+async def sleep_forever(
+    ctx: tractor.Context,
+    # ignore_special_cases: bool,
+    do_started: bool,
+):
+
+    # async with maybe_unmask_excs(
+    #     ignore_special_cases=ignore_special_cases,
+    # ):
+    #     await ctx.started('locked')
+    if do_started:
+        await ctx.started()
+
+    # block til cancelled
+    print('sleepin on child-side..')
+    await trio.sleep_forever()
+
+
+@pytest.mark.parametrize(
+    'cancel_ctx',
+    [True, False],
+)
+def test_cancel_ctx_with_parent_side_entered_in_bg_task(
+    debug_mode: bool,
+    loglevel: str,
+    cancel_ctx: bool,
+):
+    '''
+    The most "basic" out-of-band-task self-cancellation case where
+    `Portal.open_context()` is entered in a bg task and the
+    parent-task (of the containing nursery) calls `Context.cancel()`
+    without the child knowing; the `Context._scope` should be
+    `.cancel_called` when the IPC ctx's child-side relays
+    a `ContextCancelled` with a `.canceller` set to the parent
+    actor('s task).
+
+    '''
+    async def main():
+        with trio.fail_after(
+            2 if not debug_mode else 999,
+        ):
+            an: ActorNursery
+            async with (
+                tractor.open_nursery(
+                    debug_mode=debug_mode,
+                    loglevel='devx',
+                    enable_stack_on_sig=True,
+                ) as an,
+                trio.open_nursery() as tn,
+            ):
+                ptl: Portal = await an.start_actor(
+                    'sub',
+                    enable_modules=[__name__],
+                )
+
+                async def _open_ctx_async(
+                    do_started: bool = True,
+                    task_status=trio.TASK_STATUS_IGNORED,
+                ):
+                    # do we expect to never enter the
+                    # `.open_context()` below.
+                    if not do_started:
+                        task_status.started()
+
+                    async with ptl.open_context(
+                        sleep_forever,
+                        do_started=do_started,
+                    ) as (ctx, first):
+                        task_status.started(ctx)
+                        await trio.sleep_forever()
+
+                # XXX, this is the key OoB part!
+                #
+                # - start the `.open_context()` in a bg task which
+                #   blocks inside the embedded scope-body,
+                #
+                # -  when we call `Context.cancel()` it **is
+                #   not** from the same task which eventually runs
+                #   `.__aexit__()`,
+                #
+                # - since the bg "opener" task will be in
+                #   a `trio.sleep_forever()`, it must be interrupted
+                #   by the `ContextCancelled` delivered from the
+                #   child-side; `Context._scope: CancelScope` MUST
+                #   be `.cancel_called`!
+                #
+                print('ASYNC opening IPC context in subtask..')
+                maybe_ctx: Context|None = await tn.start(partial(
+                    _open_ctx_async,
+                ))
+
+                if (
+                    maybe_ctx
+                    and
+                    cancel_ctx
+                ):
+                    print('cancelling first IPC ctx!')
+                    await maybe_ctx.cancel()
+
+                # XXX, note that despite `maybe_context.cancel()`
+                # being called above, it's the parent (bg) task
+                # which was originally never interrupted in
+                # the `ctx._scope` body due to missing case logic in
+                # `ctx._maybe_cancel_and_set_remote_error()`.
+                #
+                # It didn't matter that the subactor process was
+                # already terminated and reaped, nothing was
+                # cancelling the ctx-parent task's scope!
+                #
+                print('cancelling subactor!')
+                await ptl.cancel_actor()
+
+                if maybe_ctx:
+                    try:
+                        await maybe_ctx.wait_for_result()
+                    except tractor.ContextCancelled as ctxc:
+                        assert not cancel_ctx
+                        assert (
+                            ctxc.canceller
+                            ==
+                            tractor.current_actor().aid.uid
+                        )
+                        # don't re-raise since it'll trigger
+                        # an EG from the above tn.
+
+    if cancel_ctx:
+        # graceful self-cancel
+        trio.run(main)
+
+    else:
+        # ctx parent task should see OoB ctxc due to
+        # `ptl.cancel_actor()`.
+        with pytest.raises(tractor.ContextCancelled) as excinfo:
+            trio.run(main)
+
+        assert 'root' in excinfo.value.canceller[0]
+
+
+# def test_parent_actor_cancels_subactor_with_gt1_ctxs_open_to_it(
+#     debug_mode: bool,
+#     loglevel: str,
+# ):
+#     '''
+#     Demos OoB cancellation from the perspective of a ctx opened with
+#     a child subactor where the parent cancels the child at the "actor
+#     layer" using `Portal.cancel_actor()` and thus the
+#     `ContextCancelled.canceller` received by the ctx's parent-side
+#     task will appear to be a "self cancellation" even though that
+#     specific task itself was not cancelled and thus
+#     `Context.cancel_called ==False`.
+#     '''
+                # TODO, do we have an existing implied ctx
+                # cancel test like this?
+                # with trio.move_on_after(0.5):# as cs:
+                #     await _open_ctx_async(
+                #         do_started=False,
+                #     )
+
+
+                # in-line ctx scope should definitely raise
+                # a ctxc with `.canceller = 'root'`
+                # async with ptl.open_context(
+                #     sleep_forever,
+                #     do_started=True,
+                # ) as pair:
+

tests/test_reg_err_types.py

@@ -0,0 +1,333 @@
+'''
+Verify that externally registered remote actor error
+types are correctly relayed, boxed, and re-raised across
+IPC actor hops via `reg_err_types()`.
+
+Also ensure that when custom error types are NOT registered
+the framework indicates the lookup failure to the user.
+
+'''
+import pytest
+import trio
+import tractor
+from tractor import (
+    Context,
+    Portal,
+    RemoteActorError,
+)
+from tractor._exceptions import (
+    get_err_type,
+    reg_err_types,
+)
+
+
+# -- custom app-level errors for testing --
+class CustomAppError(Exception):
+    '''
+    A hypothetical user-app error that should be
+    boxed+relayed by `tractor` IPC when registered.
+
+    '''
+
+
+class AnotherAppError(Exception):
+    '''
+    A second custom error for multi-type registration.
+
+    '''
+
+
+class UnregisteredAppError(Exception):
+    '''
+    A custom error that is intentionally NEVER
+    registered via `reg_err_types()` so we can
+    verify the framework's failure indication.
+
+    '''
+
+
+# -- remote-task endpoints --
+@tractor.context
+async def raise_custom_err(
+    ctx: Context,
+) -> None:
+    '''
+    Remote ep that raises a `CustomAppError`
+    after sync-ing with the caller.
+
+    '''
+    await ctx.started()
+    raise CustomAppError(
+        'the app exploded remotely'
+    )
+
+
+@tractor.context
+async def raise_another_err(
+    ctx: Context,
+) -> None:
+    '''
+    Remote ep that raises `AnotherAppError`.
+
+    '''
+    await ctx.started()
+    raise AnotherAppError(
+        'another app-level kaboom'
+    )
+
+
+@tractor.context
+async def raise_unreg_err(
+    ctx: Context,
+) -> None:
+    '''
+    Remote ep that raises an `UnregisteredAppError`
+    which has NOT been `reg_err_types()`-registered.
+
+    '''
+    await ctx.started()
+    raise UnregisteredAppError(
+        'this error type is unknown to tractor'
+    )
+
+
+# -- unit tests for the type-registry plumbing --
+
+class TestRegErrTypesPlumbing:
+    '''
+    Low-level checks on `reg_err_types()` and
+    `get_err_type()` without requiring IPC.
+
+    '''
+
+    def test_unregistered_type_returns_none(self):
+        '''
+        An unregistered custom error name should yield
+        `None` from `get_err_type()`.
+
+        '''
+        result = get_err_type('CustomAppError')
+        assert result is None
+
+    def test_register_and_lookup(self):
+        '''
+        After `reg_err_types()`, the custom type should
+        be discoverable via `get_err_type()`.
+
+        '''
+        reg_err_types([CustomAppError])
+        result = get_err_type('CustomAppError')
+        assert result is CustomAppError
+
+    def test_register_multiple_types(self):
+        '''
+        Registering a list of types should make each
+        one individually resolvable.
+
+        '''
+        reg_err_types([
+            CustomAppError,
+            AnotherAppError,
+        ])
+        assert (
+            get_err_type('CustomAppError')
+            is CustomAppError
+        )
+        assert (
+            get_err_type('AnotherAppError')
+            is AnotherAppError
+        )
+
+    def test_builtin_types_always_resolve(self):
+        '''
+        Builtin error types like `RuntimeError` and
+        `ValueError` should always be found without
+        any prior registration.
+
+        '''
+        assert (
+            get_err_type('RuntimeError')
+            is RuntimeError
+        )
+        assert (
+            get_err_type('ValueError')
+            is ValueError
+        )
+
+    def test_tractor_native_types_resolve(self):
+        '''
+        `tractor`-internal exc types (e.g.
+        `ContextCancelled`) should always resolve.
+
+        '''
+        assert (
+            get_err_type('ContextCancelled')
+            is tractor.ContextCancelled
+        )
+
+    def test_boxed_type_str_without_ipc_msg(self):
+        '''
+        When a `RemoteActorError` is constructed
+        without an IPC msg (and no resolvable type),
+        `.boxed_type_str` should return `'<unknown>'`.
+
+        '''
+        rae = RemoteActorError('test')
+        assert rae.boxed_type_str == '<unknown>'
+
+
+# -- IPC-level integration tests --
+
+def test_registered_custom_err_relayed(
+    debug_mode: bool,
+    tpt_proto: str,
+):
+    '''
+    When a custom error type is registered via
+    `reg_err_types()` on BOTH sides of an IPC dialog,
+    the parent should receive a `RemoteActorError`
+    whose `.boxed_type` matches the original custom
+    error type.
+
+    '''
+    reg_err_types([CustomAppError])
+
+    async def main():
+        async with tractor.open_nursery(
+            debug_mode=debug_mode,
+            enable_transports=[tpt_proto],
+        ) as an:
+            ptl: Portal = await an.start_actor(
+                'custom-err-raiser',
+                enable_modules=[__name__],
+            )
+            async with ptl.open_context(
+                raise_custom_err,
+            ) as (ctx, sent):
+                assert not sent
+                try:
+                    await ctx.wait_for_result()
+                except RemoteActorError as rae:
+                    assert rae.boxed_type is CustomAppError
+                    assert rae.src_type is CustomAppError
+                    assert 'the app exploded remotely' in str(
+                        rae.tb_str
+                    )
+                    raise
+
+    with pytest.raises(RemoteActorError) as excinfo:
+        trio.run(main)
+
+    rae = excinfo.value
+    assert rae.boxed_type is CustomAppError
+
+
+def test_registered_another_err_relayed(
+    debug_mode: bool,
+    tpt_proto: str,
+):
+    '''
+    Same as above but for a different custom error
+    type to verify multi-type registration works
+    end-to-end over IPC.
+
+    '''
+    reg_err_types([AnotherAppError])
+
+    async def main():
+        async with tractor.open_nursery(
+            debug_mode=debug_mode,
+            enable_transports=[tpt_proto],
+        ) as an:
+            ptl: Portal = await an.start_actor(
+                'another-err-raiser',
+                enable_modules=[__name__],
+            )
+            async with ptl.open_context(
+                raise_another_err,
+            ) as (ctx, sent):
+                assert not sent
+                try:
+                    await ctx.wait_for_result()
+                except RemoteActorError as rae:
+                    assert (
+                        rae.boxed_type
+                        is AnotherAppError
+                    )
+                    raise
+
+            await an.cancel()
+
+    with pytest.raises(RemoteActorError) as excinfo:
+        trio.run(main)
+
+    rae = excinfo.value
+    assert rae.boxed_type is AnotherAppError
+
+
+def test_unregistered_err_still_relayed(
+    debug_mode: bool,
+    tpt_proto: str,
+):
+    '''
+    Verify that even when a custom error type is NOT registered via
+    `reg_err_types()`, the remote error is still relayed as
+    a `RemoteActorError` with all string-level info preserved
+    (traceback, type name, source actor uid).
+
+    The `.boxed_type` will be `None` (type obj can't be resolved) but
+    `.boxed_type_str` and `.src_type_str` still report the original
+    type name from the IPC msg.
+
+    This documents the expected limitation: without `reg_err_types()`
+    the `.boxed_type` property can NOT resolve to the original Python
+    type.
+
+    '''
+    # NOTE: intentionally do NOT call
+    # `reg_err_types([UnregisteredAppError])`
+
+    async def main():
+        async with tractor.open_nursery(
+            debug_mode=debug_mode,
+            enable_transports=[tpt_proto],
+        ) as an:
+            ptl: Portal = await an.start_actor(
+                'unreg-err-raiser',
+                enable_modules=[__name__],
+            )
+            async with ptl.open_context(
+                raise_unreg_err,
+            ) as (ctx, sent):
+                assert not sent
+                await ctx.wait_for_result()
+
+            await an.cancel()
+
+    with pytest.raises(RemoteActorError) as excinfo:
+        trio.run(main)
+
+    rae = excinfo.value
+
+    # the error IS relayed even without
+    # registration; type obj is unresolvable but
+    # all string-level info is preserved.
+    assert rae.boxed_type is None # NOT `UnregisteredAppError`
+    assert rae.src_type is None
+
+    # string names survive the IPC round-trip
+    # via the `Error` msg fields.
+    assert (
+        rae.src_type_str
+        ==
+        'UnregisteredAppError'
+    )
+    assert (
+        rae.boxed_type_str
+        ==
+        'UnregisteredAppError'
+    )
+
+    # original traceback content is preserved
+    assert 'this error type is unknown' in rae.tb_str
+    assert 'UnregisteredAppError' in rae.tb_str

tests/test_remote_exc_relay.py

@@ -0,0 +1,237 @@
+'''
+Special case testing for issues not (dis)covered in the primary
+`Context` related functional/scenario suites.
+
+**NOTE: this mod is a WIP** space for handling
+odd/rare/undiscovered/not-yet-revealed faults which either
+loudly (ideal case) breakl our supervision protocol
+or (worst case) result in distributed sys hangs.
+
+Suites here further try to clarify (if [partially] ill-defined) and
+verify our edge case semantics for inter-actor-relayed-exceptions
+including,
+
+- lowlevel: what remote obj-data is interchanged for IPC and what is
+  native-obj form is expected from unpacking in the the new
+  mem-domain.
+
+- which kinds of `RemoteActorError` (and its derivs) are expected by which
+  (types of) peers (parent, child, sibling, etc) with what
+  particular meta-data set such as,
+
+  - `.src_uid`: the original (maybe) peer who raised.
+  - `.relay_uid`: the next-hop-peer who sent it.
+  - `.relay_path`: the sequence of peer actor hops.
+  - `.is_inception`: a predicate that denotes multi-hop remote errors.
+
+- when should `ExceptionGroup`s be relayed from a particular
+  remote endpoint, they should never be caused by implicit `._rpc`
+  nursery machinery!
+
+- various special `trio` edge cases around its cancellation semantics
+  and how we (currently) leverage `trio.Cancelled` as a signal for
+  whether a `Context` task should raise `ContextCancelled` (ctx).
+
+'''
+import pytest
+import trio
+import tractor
+from tractor import (  # typing
+    ActorNursery,
+    Portal,
+    Context,
+    ContextCancelled,
+)
+
+
+@tractor.context
+async def sleep_n_chkpt_in_finally(
+    ctx: Context,
+    sleep_n_raise: bool,
+
+    chld_raise_delay: float,
+    chld_finally_delay: float,
+
+    rent_cancels: bool,
+    rent_ctxc_delay: float,
+
+    expect_exc: str|None = None,
+
+) -> None:
+    '''
+    Sync, open a tn, then wait for cancel, run a chkpt inside
+    the user's `finally:` teardown.
+
+    This covers a footgun case that `trio` core doesn't seem to care about
+    wherein an exc can be masked by a `trio.Cancelled` raised inside a tn emedded
+    `finally:`.
+
+    Also see `test_trioisms::test_acm_embedded_nursery_propagates_enter_err`
+    for the down and gritty details.
+
+    Since a `@context` endpoint fn can also contain code like this,
+    **and** bc we currently have no easy way other then
+    `trio.Cancelled` to signal cancellation on each side of an IPC `Context`,
+    the footgun issue can compound itself as demonstrated in this suite..
+
+    Here are some edge cases codified with our WIP "sclang" syntax
+    (note the parent(rent)/child(chld) naming here is just
+    pragmatism, generally these most of these cases can occurr
+    regardless of the distributed-task's supervision hiearchy),
+
+    - rent c)=> chld.raises-then-taskc-in-finally
+     |_ chld's body raises an `exc: BaseException`.
+      _ in its `finally:` block it runs a chkpoint
+        which raises a taskc (`trio.Cancelled`) which
+        masks `exc` instead raising taskc up to the first tn.
+      _ the embedded/chld tn captures the masking taskc and then
+        raises it up to the ._rpc-ep-tn instead of `exc`.
+      _ the rent thinks the child ctxc-ed instead of errored..
+
+    '''
+    await ctx.started()
+
+    if expect_exc:
+        expect_exc: BaseException = tractor._exceptions.get_err_type(
+            type_name=expect_exc,
+        )
+
+    berr: BaseException|None = None
+    try:
+        if not sleep_n_raise:
+            await trio.sleep_forever()
+        elif sleep_n_raise:
+
+            # XXX this sleep is less then the sleep the parent
+            # does before calling `ctx.cancel()`
+            await trio.sleep(chld_raise_delay)
+
+            # XXX this will be masked by a taskc raised in
+            # the `finally:` if this fn doesn't terminate
+            # before any ctxc-req arrives AND a checkpoint is hit
+            # in that `finally:`.
+            raise RuntimeError('my app krurshed..')
+
+    except BaseException as _berr:
+        berr = _berr
+
+        # TODO: it'd sure be nice to be able to inject our own
+        # `ContextCancelled` here instead of of `trio.Cancelled`
+        # so that our runtime can expect it and this "user code"
+        # would be able to tell the diff between a generic trio
+        # cancel and a tractor runtime-IPC cancel.
+        if expect_exc:
+            if not isinstance(
+                berr,
+                expect_exc,
+            ):
+                raise ValueError(
+                    f'Unexpected exc type ??\n'
+                    f'{berr!r}\n'
+                    f'\n'
+                    f'Expected a {expect_exc!r}\n'
+                )
+
+        raise berr
+
+    # simulate what user code might try even though
+    # it's a known boo-boo..
+    finally:
+        # maybe wait for rent ctxc to arrive
+        with trio.CancelScope(shield=True):
+            await trio.sleep(chld_finally_delay)
+
+        # !!XXX this will raise `trio.Cancelled` which
+        # will mask the RTE from above!!!
+        #
+        # YES, it's the same case as our extant
+        # `test_trioisms::test_acm_embedded_nursery_propagates_enter_err`
+        try:
+            await trio.lowlevel.checkpoint()
+        except trio.Cancelled as taskc:
+            if (scope_err := taskc.__context__):
+                print(
+                    f'XXX MASKED REMOTE ERROR XXX\n'
+                    f'ENDPOINT exception -> {scope_err!r}\n'
+                    f'will be masked by -> {taskc!r}\n'
+                )
+                # await tractor.pause(shield=True)
+
+            raise taskc
+
+
+@pytest.mark.parametrize(
+    'chld_callspec',
+    [
+        dict(
+            sleep_n_raise=None,
+            chld_raise_delay=0.1,
+            chld_finally_delay=0.1,
+            expect_exc='Cancelled',
+            rent_cancels=True,
+            rent_ctxc_delay=0.1,
+        ),
+        dict(
+            sleep_n_raise='RuntimeError',
+            chld_raise_delay=0.1,
+            chld_finally_delay=1,
+            expect_exc='RuntimeError',
+            rent_cancels=False,
+            rent_ctxc_delay=0.1,
+        ),
+    ],
+    ids=lambda item: f'chld_callspec={item!r}'
+)
+def test_unmasked_remote_exc(
+    debug_mode: bool,
+    chld_callspec: dict,
+    tpt_proto: str,
+):
+    expect_exc_str: str|None = chld_callspec['sleep_n_raise']
+    rent_ctxc_delay: float|None = chld_callspec['rent_ctxc_delay']
+    async def main():
+        an: ActorNursery
+        async with tractor.open_nursery(
+            debug_mode=debug_mode,
+            enable_transports=[tpt_proto],
+        ) as an:
+            ptl: Portal = await an.start_actor(
+                'cancellee',
+                enable_modules=[__name__],
+            )
+            ctx: Context
+            async with (
+                ptl.open_context(
+                    sleep_n_chkpt_in_finally,
+                    **chld_callspec,
+                ) as (ctx, sent),
+            ):
+                assert not sent
+                await trio.sleep(rent_ctxc_delay)
+                await ctx.cancel()
+
+                # recv error or result from chld
+                ctxc: ContextCancelled = await ctx.wait_for_result()
+                assert (
+                    ctxc is ctx.outcome
+                    and
+                    isinstance(ctxc, ContextCancelled)
+                )
+
+            # always graceful terminate the sub in non-error cases
+            await an.cancel()
+
+    if expect_exc_str:
+        expect_exc: BaseException = tractor._exceptions.get_err_type(
+            type_name=expect_exc_str,
+        )
+        with pytest.raises(
+            expected_exception=tractor.RemoteActorError,
+        ) as excinfo:
+            trio.run(main)
+
+        rae = excinfo.value
+        assert expect_exc == rae.boxed_type
+
+    else:
+        trio.run(main)

tests/test_resource_cache.py

@@ -1,5 +1,6 @@
 '''
-Async context manager cache api testing: ``trionics.maybe_open_context():``
+Suites for our `.trionics.maybe_open_context()` multi-task
+shared-cached `@acm` API.
 
 '''
 from contextlib import asynccontextmanager as acm
@@ -9,7 +10,15 @@ from typing import Awaitable
 import pytest
 import trio
 import tractor
+from tractor.trionics import (
+    maybe_open_context,
+)
+from tractor.log import (
+    get_console_log,
+    get_logger,
+)
 
+log = get_logger()
 
 _resource: int = 0
 
@@ -52,7 +61,7 @@ def test_resource_only_entered_once(key_on):
                 # different task names per task will be used
                 kwargs = {'task_name': name}
 
-            async with tractor.trionics.maybe_open_context(
+            async with maybe_open_context(
                 maybe_increment_counter,
                 kwargs=kwargs,
                 key=key,
@@ -72,11 +81,13 @@ def test_resource_only_entered_once(key_on):
         with trio.move_on_after(0.5):
             async with (
                 tractor.open_root_actor(),
-                trio.open_nursery() as n,
+                trio.open_nursery() as tn,
             ):
-
                 for i in range(10):
-                    n.start_soon(enter_cached_mngr, f'task_{i}')
+                    tn.start_soon(
+                        enter_cached_mngr,
+                        f'task_{i}',
+                    )
                     await trio.sleep(0.001)
 
     trio.run(main)
@@ -98,27 +109,55 @@ async def streamer(
 
 
 @acm
-async def open_stream() -> Awaitable[tractor.MsgStream]:
+async def open_stream() -> Awaitable[
-
+    tuple[
-    async with tractor.open_nursery() as tn:
+        tractor.ActorNursery,
-        portal = await tn.start_actor('streamer', enable_modules=[__name__])
+        tractor.MsgStream,
+    ]
+]:
+    try:
+        async with tractor.open_nursery() as an:
+            portal = await an.start_actor(
+                'streamer',
+                enable_modules=[__name__],
+            )
+            try:
                 async with (
                     portal.open_context(streamer) as (ctx, first),
                     ctx.open_stream() as stream,
                 ):
-            yield stream
+                    print('Entered open_stream() caller')
+                    yield an, stream
+                    print('Exited open_stream() caller')
 
+            finally:
+                print(
+                    'Cancelling streamer with,\n'
+                    '=> `Portal.cancel_actor()`'
+                )
                 await portal.cancel_actor()
-    print('CANCELLED STREAMER')
+                print('Cancelled streamer')
+
+    except Exception as err:
+        print(
+            f'`open_stream()` errored?\n'
+            f'{err!r}\n'
+        )
+        await tractor.pause(shield=True)
+        raise err
 
 
 @acm
 async def maybe_open_stream(taskname: str):
-    async with tractor.trionics.maybe_open_context(
+    async with maybe_open_context(
         # NOTE: all secondary tasks should cache hit on the same key
         acm_func=open_stream,
-    ) as (cache_hit, stream):
+    ) as (
-
+        cache_hit,
+        (an, stream)
+    ):
+        # when the actor + portal + ctx + stream has already been
+        # allocated we want to just bcast to this task.
         if cache_hit:
             print(f'{taskname} loaded from cache')
 
@@ -126,27 +165,77 @@ async def maybe_open_stream(taskname: str):
             # if this feed is already allocated by the first
             # task that entereed
             async with stream.subscribe() as bstream:
-                yield bstream
+                yield an, bstream
+                print(
+                    f'cached task exited\n'
+                    f')>\n'
+                    f' |_{taskname}\n'
+                )
+
+            # we should always unreg the "cloned" bcrc for this
+            # consumer-task
+            assert id(bstream) not in bstream._state.subs
+
         else:
             # yield the actual stream
-            yield stream
+            try:
+                yield an, stream
+            finally:
+                print(
+                    f'NON-cached task exited\n'
+                    f')>\n'
+                    f' |_{taskname}\n'
+                )
+
+        first_bstream = stream._broadcaster
+        bcrx_state = first_bstream._state
+        subs: dict[int, int] = bcrx_state.subs
+        if len(subs) == 1:
+            assert id(first_bstream) in subs
+            # ^^TODO! the bcrx should always de-allocate all subs,
+            # including the implicit first one allocated on entry
+            # by the first subscribing peer task, no?
+            #
+            # -[ ] adjust `MsgStream.subscribe()` to do this mgmt!
+            #  |_ allows reverting `MsgStream.receive()` to the
+            #    non-bcaster method.
+            #  |_ we can decide whether to reset `._broadcaster`?
+            #
+            # await tractor.pause(shield=True)
 
 
-def test_open_local_sub_to_stream():
+def test_open_local_sub_to_stream(
+    debug_mode: bool,
+):
     '''
     Verify a single inter-actor stream can can be fanned-out shared to
-    N local tasks using ``trionics.maybe_open_context():``.
+    N local tasks using `trionics.maybe_open_context()`.
 
     '''
-    timeout: float = 3.6 if platform.system() != "Windows" else 10
+    timeout: float = 3.6
+    if platform.system() == "Windows":
+        timeout: float = 10
+
+    if debug_mode:
+        timeout = 999
+        print(f'IN debug_mode, setting large timeout={timeout!r}..')
 
     async def main():
 
         full = list(range(1000))
+        an: tractor.ActorNursery|None = None
+        num_tasks: int = 10
 
         async def get_sub_and_pull(taskname: str):
+
+            nonlocal an
+
+            stream: tractor.MsgStream
             async with (
-                maybe_open_stream(taskname) as stream,
+                maybe_open_stream(taskname) as (
+                    an,
+                    stream,
+                ),
             ):
                 if '0' in taskname:
                     assert isinstance(stream, tractor.MsgStream)
@@ -158,24 +247,159 @@ def test_open_local_sub_to_stream():
 
                 first = await stream.receive()
                 print(f'{taskname} started with value {first}')
-                seq = []
+                seq: list[int] = []
                 async for msg in stream:
                     seq.append(msg)
 
                 assert set(seq).issubset(set(full))
+
+            # end of @acm block
             print(f'{taskname} finished')
 
-        with trio.fail_after(timeout):
+        root: tractor.Actor
+        with trio.fail_after(timeout) as cs:
             # TODO: turns out this isn't multi-task entrant XD
             # We probably need an indepotent entry semantic?
-            async with tractor.open_root_actor():
+            async with tractor.open_root_actor(
+                debug_mode=debug_mode,
+                # maybe_enable_greenback=True,
+                #
+                # ^TODO? doesn't seem to mk breakpoint() usage work
+                # bc each bg task needs to open a portal??
+                # - [ ] we should consider making this part of
+                #      our taskman defaults?
+                #   |_see https://github.com/goodboy/tractor/pull/363
+                #
+            ) as root:
+                assert root.is_registrar
+
                 async with (
-                    trio.open_nursery() as nurse,
+                    trio.open_nursery() as tn,
                 ):
-                    for i in range(10):
+                    for i in range(num_tasks):
-                        nurse.start_soon(get_sub_and_pull, f'task_{i}')
+                        tn.start_soon(
+                            get_sub_and_pull,
+                            f'task_{i}',
+                        )
                         await trio.sleep(0.001)
 
-                print('all consumer tasks finished')
+                print('all consumer tasks finished!')
+
+                # ?XXX, ensure actor-nursery is shutdown or we might
+                # hang here due to a minor task deadlock/race-condition?
+                #
+                # - seems that all we need is a checkpoint to ensure
+                #   the last suspended task, which is inside
+                #   `.maybe_open_context()`, can do the
+                #   `Portal.cancel_actor()` call?
+                #
+                # - if that bg task isn't resumed, then this blocks
+                #   timeout might hit before that?
+                #
+                if root.ipc_server.has_peers():
+                    await trio.lowlevel.checkpoint()
+
+                    # alt approach, cancel the entire `an`
+                    # await tractor.pause()
+                    # await an.cancel()
+
+            # end of runtime scope
+            print('root actor terminated.')
+
+        if cs.cancelled_caught:
+            pytest.fail(
+                'Should NOT time out in `open_root_actor()` ?'
+            )
+
+        print('exiting main.')
+
+    trio.run(main)
+
+
+
+@acm
+async def cancel_outer_cs(
+    cs: trio.CancelScope|None = None,
+    delay: float = 0,
+):
+    # on first task delay this enough to block
+    # the 2nd task but then cancel it mid sleep
+    # so that the tn.start() inside the key-err handler block
+    # is cancelled and would previously corrupt the
+    # mutext state.
+    log.info(f'task entering sleep({delay})')
+    await trio.sleep(delay)
+    if cs:
+        log.info('task calling cs.cancel()')
+        cs.cancel()
+    trio.lowlevel.checkpoint()
+    yield
+    await trio.sleep_forever()
+
+
+def test_lock_not_corrupted_on_fast_cancel(
+    debug_mode: bool,
+    loglevel: str,
+):
+    '''
+    Verify that if the caching-task (the first to enter
+    `maybe_open_context()`) is cancelled mid-cache-miss, the embedded
+    mutex can never be left in a corrupted state.
+
+    That is, the lock is always eventually released ensuring a peer
+    (cache-hitting) task will never,
+
+    - be left to inf-block/hang on the `lock.acquire()`.
+    - try to release the lock when still owned by the caching-task
+      due to it having erronously exited without calling
+      `lock.release()`.
+
+
+    '''
+    delay: float = 1.
+
+    async def use_moc(
+        cs: trio.CancelScope|None,
+        delay: float,
+    ):
+        log.info('task entering moc')
+        async with maybe_open_context(
+            cancel_outer_cs,
+            kwargs={
+                'cs': cs,
+                'delay': delay,
+            },
+        ) as (cache_hit, _null):
+            if cache_hit:
+                log.info('2nd task entered')
+            else:
+                log.info('1st task entered')
+
+            await trio.sleep_forever()
+
+    async def main():
+        with trio.fail_after(delay + 2):
+            async with (
+                tractor.open_root_actor(
+                    debug_mode=debug_mode,
+                    loglevel=loglevel,
+                ),
+                trio.open_nursery() as tn,
+            ):
+                get_console_log('info')
+                log.info('yo starting')
+                cs = tn.cancel_scope
+                tn.start_soon(
+                    use_moc,
+                    cs,
+                    delay,
+                    name='child',
+                )
+                with trio.CancelScope() as rent_cs:
+                    await use_moc(
+                        cs=rent_cs,
+                        delay=delay,
+                    )
+
 
     trio.run(main)

tests/test_ringbuf.py

@@ -0,0 +1,211 @@
+import time
+
+import trio
+import pytest
+
+import tractor
+from tractor.ipc._ringbuf import (
+    open_ringbuf,
+    RBToken,
+    RingBuffSender,
+    RingBuffReceiver
+)
+from tractor._testing.samples import (
+    generate_sample_messages,
+)
+
+# in case you don't want to melt your cores, uncomment dis!
+pytestmark = pytest.mark.skip
+
+
+@tractor.context
+async def child_read_shm(
+    ctx: tractor.Context,
+    msg_amount: int,
+    token: RBToken,
+    total_bytes: int,
+) -> None:
+    recvd_bytes = 0
+    await ctx.started()
+    start_ts = time.time()
+    async with RingBuffReceiver(token) as receiver:
+        while recvd_bytes < total_bytes:
+            msg = await receiver.receive_some()
+            recvd_bytes += len(msg)
+
+        # make sure we dont hold any memoryviews
+        # before the ctx manager aclose()
+        msg = None
+
+    end_ts = time.time()
+    elapsed = end_ts - start_ts
+    elapsed_ms = int(elapsed * 1000)
+
+    print(f'\n\telapsed ms: {elapsed_ms}')
+    print(f'\tmsg/sec: {int(msg_amount / elapsed):,}')
+    print(f'\tbytes/sec: {int(recvd_bytes / elapsed):,}')
+
+
+@tractor.context
+async def child_write_shm(
+    ctx: tractor.Context,
+    msg_amount: int,
+    rand_min: int,
+    rand_max: int,
+    token: RBToken,
+) -> None:
+    msgs, total_bytes = generate_sample_messages(
+        msg_amount,
+        rand_min=rand_min,
+        rand_max=rand_max,
+    )
+    await ctx.started(total_bytes)
+    async with RingBuffSender(token) as sender:
+        for msg in msgs:
+            await sender.send_all(msg)
+
+
+@pytest.mark.parametrize(
+    'msg_amount,rand_min,rand_max,buf_size',
+    [
+        # simple case, fixed payloads, large buffer
+        (100_000, 0, 0, 10 * 1024),
+
+        # guaranteed wrap around on every write
+        (100, 10 * 1024, 20 * 1024, 10 * 1024),
+
+        # large payload size, but large buffer
+        (10_000, 256 * 1024, 512 * 1024, 10 * 1024 * 1024)
+    ],
+    ids=[
+        'fixed_payloads_large_buffer',
+        'wrap_around_every_write',
+        'large_payloads_large_buffer',
+    ]
+)
+def test_ringbuf(
+    msg_amount: int,
+    rand_min: int,
+    rand_max: int,
+    buf_size: int
+):
+    async def main():
+        with open_ringbuf(
+            'test_ringbuf',
+            buf_size=buf_size
+        ) as token:
+            proc_kwargs = {
+                'pass_fds': (token.write_eventfd, token.wrap_eventfd)
+            }
+
+            common_kwargs = {
+                'msg_amount': msg_amount,
+                'token': token,
+            }
+            async with tractor.open_nursery() as an:
+                send_p = await an.start_actor(
+                    'ring_sender',
+                    enable_modules=[__name__],
+                    proc_kwargs=proc_kwargs
+                )
+                recv_p = await an.start_actor(
+                    'ring_receiver',
+                    enable_modules=[__name__],
+                    proc_kwargs=proc_kwargs
+                )
+                async with (
+                    send_p.open_context(
+                        child_write_shm,
+                        rand_min=rand_min,
+                        rand_max=rand_max,
+                        **common_kwargs
+                    ) as (sctx, total_bytes),
+                    recv_p.open_context(
+                        child_read_shm,
+                        **common_kwargs,
+                        total_bytes=total_bytes,
+                    ) as (sctx, _sent),
+                ):
+                    await recv_p.result()
+
+                await send_p.cancel_actor()
+                await recv_p.cancel_actor()
+
+
+    trio.run(main)
+
+
+@tractor.context
+async def child_blocked_receiver(
+    ctx: tractor.Context,
+    token: RBToken
+):
+    async with RingBuffReceiver(token) as receiver:
+        await ctx.started()
+        await receiver.receive_some()
+
+
+def test_ring_reader_cancel():
+    async def main():
+        with open_ringbuf('test_ring_cancel_reader') as token:
+            async with (
+                tractor.open_nursery() as an,
+                RingBuffSender(token) as _sender,
+            ):
+                recv_p = await an.start_actor(
+                    'ring_blocked_receiver',
+                    enable_modules=[__name__],
+                    proc_kwargs={
+                        'pass_fds': (token.write_eventfd, token.wrap_eventfd)
+                    }
+                )
+                async with (
+                    recv_p.open_context(
+                        child_blocked_receiver,
+                        token=token
+                    ) as (sctx, _sent),
+                ):
+                    await trio.sleep(1)
+                    await an.cancel()
+
+
+    with pytest.raises(tractor._exceptions.ContextCancelled):
+        trio.run(main)
+
+
+@tractor.context
+async def child_blocked_sender(
+    ctx: tractor.Context,
+    token: RBToken
+):
+    async with RingBuffSender(token) as sender:
+        await ctx.started()
+        await sender.send_all(b'this will wrap')
+
+
+def test_ring_sender_cancel():
+    async def main():
+        with open_ringbuf(
+            'test_ring_cancel_sender',
+            buf_size=1
+        ) as token:
+            async with tractor.open_nursery() as an:
+                recv_p = await an.start_actor(
+                    'ring_blocked_sender',
+                    enable_modules=[__name__],
+                    proc_kwargs={
+                        'pass_fds': (token.write_eventfd, token.wrap_eventfd)
+                    }
+                )
+                async with (
+                    recv_p.open_context(
+                        child_blocked_sender,
+                        token=token
+                    ) as (sctx, _sent),
+                ):
+                    await trio.sleep(1)
+                    await an.cancel()
+
+
+    with pytest.raises(tractor._exceptions.ContextCancelled):
+        trio.run(main)

tests/test_root_infect_asyncio.py

@@ -49,7 +49,7 @@ def test_infected_root_actor(
                 ),
                 to_asyncio.open_channel_from(
                     aio_echo_server,
-                ) as (first, chan),
+                ) as (chan, first),
             ):
                 assert first == 'start'
 
@@ -91,13 +91,12 @@ def test_infected_root_actor(
 async def sync_and_err(
     # just signature placeholders for compat with
     # ``to_asyncio.open_channel_from()``
-    to_trio: trio.MemorySendChannel,
+    chan: tractor.to_asyncio.LinkedTaskChannel,
-    from_trio: asyncio.Queue,
     ev: asyncio.Event,
 
 ):
-    if to_trio:
+    if chan:
-        to_trio.send_nowait('start')
+        chan.started_nowait('start')
 
     await ev.wait()
     raise RuntimeError('asyncio-side')
@@ -147,8 +146,7 @@ def test_trio_prestarted_task_bubbles(
         await trio.sleep_forever()
 
     async def _trio_main():
-        # with trio.fail_after(2):
+        with trio.fail_after(2 if not debug_mode else 999):
-        with trio.fail_after(999):
             first: str
             chan: to_asyncio.LinkedTaskChannel
             aio_ev = asyncio.Event()
@@ -175,7 +173,7 @@ def test_trio_prestarted_task_bubbles(
                             sync_and_err,
                             ev=aio_ev,
                         )
-                    ) as (first, chan),
+                    ) as (chan, first),
                 ):
 
                     for i in range(5):
@@ -217,32 +215,25 @@ def test_trio_prestarted_task_bubbles(
                         ):
                             aio_ev.set()
 
-    with pytest.raises(
-        expected_exception=ExceptionGroup,
-    ) as excinfo:
-        tractor.to_asyncio.run_as_asyncio_guest(
-            trio_main=_trio_main,
-        )
-
-    eg = excinfo.value
-    rte_eg, rest_eg = eg.split(RuntimeError)
-
     # ensure the trio-task's error bubbled despite the aio-side
     # having (maybe) errored first.
     if aio_err_trigger in (
         'after_trio_task_starts',
         'after_start_point',
     ):
-        assert len(errs := rest_eg.exceptions) == 1
+        patt: str = 'trio-side'
-        typerr = errs[0]
+        expect_exc = TypeError
-        assert (
-            type(typerr) is TypeError
-            and
-            'trio-side' in typerr.args
-        )
 
     # when aio errors BEFORE (last) trio task is scheduled, we should
     # never see anythinb but the aio-side.
     else:
-        assert len(rtes := rte_eg.exceptions) == 1
+        patt: str = 'asyncio-side'
-        assert 'asyncio-side' in rtes[0].args[0]
+        expect_exc = RuntimeError
+
+    with pytest.raises(expect_exc) as excinfo:
+        tractor.to_asyncio.run_as_asyncio_guest(
+            trio_main=_trio_main,
+        )
+
+    caught_exc = excinfo.value
+    assert patt in caught_exc.args

tests/test_root_runtime.py

@@ -0,0 +1,108 @@
+'''
+Runtime boot/init sanity.
+
+'''
+
+import pytest
+import trio
+
+import tractor
+from tractor._exceptions import RuntimeFailure
+
+
+@tractor.context
+async def open_new_root_in_sub(
+    ctx: tractor.Context,
+) -> None:
+
+    async with tractor.open_root_actor():
+        pass
+
+
+@pytest.mark.parametrize(
+    'open_root_in',
+    ['root', 'sub'],
+    ids='open_2nd_root_in={}'.format,
+)
+def test_only_one_root_actor(
+    open_root_in: str,
+    reg_addr: tuple,
+    debug_mode: bool
+):
+    '''
+    Verify we specially fail whenever more then one root actor
+    is attempted to be opened within an already opened tree.
+
+    '''
+    async def main():
+        async with tractor.open_nursery() as an:
+
+            if open_root_in == 'root':
+                async with tractor.open_root_actor(
+                    registry_addrs=[reg_addr],
+                ):
+                    pass
+
+            ptl: tractor.Portal = await an.start_actor(
+                name='bad_rooty_boi',
+                enable_modules=[__name__],
+            )
+
+            async with ptl.open_context(
+                open_new_root_in_sub,
+            ) as (ctx, first):
+                pass
+
+    if open_root_in == 'root':
+        with pytest.raises(
+            RuntimeFailure
+        ) as excinfo:
+            trio.run(main)
+
+    else:
+        with pytest.raises(
+            tractor.RemoteActorError,
+        ) as excinfo:
+            trio.run(main)
+
+        assert excinfo.value.boxed_type is RuntimeFailure
+
+
+def test_implicit_root_via_first_nursery(
+    reg_addr: tuple,
+    debug_mode: bool
+):
+    '''
+    The first `ActorNursery` open should implicitly call
+    `_root.open_root_actor()`.
+
+    '''
+    async def main():
+        async with tractor.open_nursery() as an:
+            assert an._implicit_runtime_started
+            assert tractor.current_actor().aid.name == 'root'
+
+    trio.run(main)
+
+
+def test_runtime_vars_unset(
+    reg_addr: tuple,
+    debug_mode: bool
+):
+    '''
+    Ensure any `._state._runtime_vars` are restored to default values
+    after the root actor-runtime exits!
+
+    '''
+    assert not tractor.runtime._state._runtime_vars['_debug_mode']
+    async def main():
+        assert not tractor.runtime._state._runtime_vars['_debug_mode']
+        async with tractor.open_nursery(
+            debug_mode=True,
+        ):
+            assert tractor.runtime._state._runtime_vars['_debug_mode']
+
+        # after runtime closure, should be reverted!
+        assert not tractor.runtime._state._runtime_vars['_debug_mode']
+
+    trio.run(main)

tests/test_rpc.py

@@ -110,7 +110,7 @@ def test_rpc_errors(
         ) as n:
 
             actor = tractor.current_actor()
-            assert actor.is_arbiter
+            assert actor.is_registrar
             await n.run_in_actor(
                 sleep_back_actor,
                 actor_name=subactor_requests_to,

tests/test_runtime.py

@@ -22,6 +22,10 @@ def unlink_file():
 async def crash_and_clean_tmpdir(
     tmp_file_path: str,
     error: bool = True,
+    rent_cancel: bool = True,
+
+    # XXX unused, but do we really need to test these cases?
+    self_cancel: bool = False,
 ):
     global _file_path
     _file_path = tmp_file_path
@@ -32,43 +36,75 @@ async def crash_and_clean_tmpdir(
     assert os.path.isfile(tmp_file_path)
     await trio.sleep(0.1)
     if error:
+        print('erroring in subactor!')
         assert 0
-    else:
+
+    elif self_cancel:
+        print('SELF-cancelling subactor!')
         actor.cancel_soon()
 
+    elif rent_cancel:
+        await trio.sleep_forever()
+
+    print('subactor exiting task!')
+
 
 @pytest.mark.parametrize(
     'error_in_child',
     [True, False],
+    ids='error_in_child={}'.format,
 )
 @tractor_test
 async def test_lifetime_stack_wipes_tmpfile(
     tmp_path,
     error_in_child: bool,
+    loglevel: str,
+    # log: tractor.log.StackLevelAdapter,
+    # ^TODO, once landed via macos support!
 ):
     child_tmp_file = tmp_path / "child.txt"
     child_tmp_file.touch()
     assert child_tmp_file.exists()
     path = str(child_tmp_file)
 
+    # NOTE, this is expected to cancel the sub
+    # in the `error_in_child=False` case!
+    timeout: float = (
+        1.6 if error_in_child
+        else 1
+    )
     try:
-        with trio.move_on_after(0.5):
+        with trio.move_on_after(timeout) as cs:
-            async with tractor.open_nursery() as n:
+            async with tractor.open_nursery(
-                    await (  # inlined portal
+                loglevel=loglevel,
-                        await n.run_in_actor(
+            ) as an:
+                await (  # inlined `tractor.Portal`
+                    await an.run_in_actor(
                         crash_and_clean_tmpdir,
                         tmp_file_path=path,
                         error=error_in_child,
                     )
                 ).result()
-
     except (
         tractor.RemoteActorError,
-        # tractor.BaseExceptionGroup,
         BaseExceptionGroup,
-    ):
+    ) as _exc:
-        pass
+        exc = _exc
+        from tractor.log import get_console_log
+        log = get_console_log(
+            level=loglevel,
+            name=__name__,
+        )
+        log.exception(
+            f'Subactor failed as expected with {type(exc)!r}\n'
+        )
 
     # tmp file should have been wiped by
     # teardown stack.
     assert not child_tmp_file.exists()
+
+    if error_in_child:
+        assert not cs.cancel_called
+    else:
+        # expect timeout in some cases?
+        assert cs.cancel_called

tests/test_shm.py

@@ -0,0 +1,179 @@
+"""
+Shared mem primitives and APIs.
+
+"""
+import platform
+import uuid
+
+# import numpy
+import pytest
+import trio
+import tractor
+from tractor.ipc._shm import (
+    open_shm_list,
+    attach_shm_list,
+)
+
+
+@tractor.context
+async def child_attach_shml_alot(
+    ctx: tractor.Context,
+    shm_key: str,
+) -> None:
+
+    await ctx.started(shm_key)
+
+    # now try to attach a boatload of times in a loop..
+    for _ in range(1000):
+        shml = attach_shm_list(
+            key=shm_key,
+            readonly=False,
+        )
+        assert shml.shm.name == shm_key
+        await trio.sleep(0.001)
+
+
+def test_child_attaches_alot():
+    async def main():
+        async with tractor.open_nursery() as an:
+
+            # allocate writeable list in parent
+            key = f'shml_{uuid.uuid4()}'
+            shml = open_shm_list(
+                key=key,
+            )
+
+            portal = await an.start_actor(
+                'shm_attacher',
+                enable_modules=[__name__],
+            )
+
+            async with (
+                portal.open_context(
+                    child_attach_shml_alot,
+                    shm_key=shml.key,
+                ) as (ctx, start_val),
+            ):
+                assert (_key := shml.key) == start_val
+
+                if platform.system() != 'Darwin':
+                    # XXX, macOS has a char limit..
+                    # see `ipc._shm._shorten_key_for_macos`
+                    assert (
+                        start_val
+                        ==
+                        key
+                        ==
+                        _key
+                    )
+                await ctx.result()
+
+            await portal.cancel_actor()
+
+    trio.run(main)
+
+
+@tractor.context
+async def child_read_shm_list(
+    ctx: tractor.Context,
+    shm_key: str,
+    use_str: bool,
+    frame_size: int,
+) -> None:
+
+    # attach in child
+    shml = attach_shm_list(
+        key=shm_key,
+        # dtype=str if use_str else float,
+    )
+    await ctx.started(shml.key)
+
+    async with ctx.open_stream() as stream:
+        async for i in stream:
+            print(f'(child): reading shm list index: {i}')
+
+            if use_str:
+                expect = str(float(i))
+            else:
+                expect = float(i)
+
+            if frame_size == 1:
+                val = shml[i]
+                assert expect == val
+                print(f'(child): reading value: {val}')
+            else:
+                frame = shml[i - frame_size:i]
+                print(f'(child): reading frame: {frame}')
+
+
+@pytest.mark.parametrize(
+    'use_str',
+    [False, True],
+    ids=lambda i: f'use_str_values={i}',
+)
+@pytest.mark.parametrize(
+    'frame_size',
+    [1, 2**6, 2**10],
+    ids=lambda i: f'frame_size={i}',
+)
+def test_parent_writer_child_reader(
+    use_str: bool,
+    frame_size: int,
+):
+
+    async def main():
+        async with tractor.open_nursery(
+            # debug_mode=True,
+        ) as an:
+
+            portal = await an.start_actor(
+                'shm_reader',
+                enable_modules=[__name__],
+                debug_mode=True,
+            )
+
+            # allocate writeable list in parent
+            key = 'shm_list'
+            seq_size = int(2 * 2 ** 10)
+            shml = open_shm_list(
+                key=key,
+                size=seq_size,
+                dtype=str if use_str else float,
+                readonly=False,
+            )
+
+            async with (
+                portal.open_context(
+                    child_read_shm_list,
+                    shm_key=key,
+                    use_str=use_str,
+                    frame_size=frame_size,
+                ) as (ctx, sent),
+
+                ctx.open_stream() as stream,
+            ):
+
+                assert sent == key
+
+                for i in range(seq_size):
+
+                    val = float(i)
+                    if use_str:
+                        val = str(val)
+
+                    # print(f'(parent): writing {val}')
+                    shml[i] = val
+
+                    # only on frame fills do we
+                    # signal to the child that a frame's
+                    # worth is ready.
+                    if (i % frame_size) == 0:
+                        print(f'(parent): signalling frame full on {val}')
+                        await stream.send(i)
+                else:
+                    print(f'(parent): signalling final frame on {val}')
+                    await stream.send(i)
+
+            await portal.cancel_actor()
+
+    trio.run(main)

tests/test_spawning.py

@@ -2,6 +2,7 @@
 Spawning basics
 
 """
+from functools import partial
 from typing import (
     Any,
 )
@@ -12,74 +13,99 @@ import tractor
 
 from tractor._testing import tractor_test
 
-data_to_pass_down = {'doggy': 10, 'kitty': 4}
+data_to_pass_down = {
+    'doggy': 10,
+    'kitty': 4,
+}
 
 
 async def spawn(
-    is_arbiter: bool,
+    should_be_root: bool,
     data: dict,
     reg_addr: tuple[str, int],
+
+    debug_mode: bool = False,
 ):
-    namespaces = [__name__]
-
     await trio.sleep(0.1)
+    actor = tractor.current_actor(err_on_no_runtime=False)
 
-    async with tractor.open_root_actor(
+    if should_be_root:
+        assert actor is None  # no runtime yet
+        async with (
+            tractor.open_root_actor(
                 arbiter_addr=reg_addr,
+            ),
+            tractor.open_nursery() as an,
         ):
-        actor = tractor.current_actor()
+            # now runtime exists
-        assert actor.is_arbiter == is_arbiter
+            actor: tractor.Actor = tractor.current_actor()
-        data = data_to_pass_down
+            assert actor.is_registrar == should_be_root
 
-        if actor.is_arbiter:
+            # spawns subproc here
-            async with tractor.open_nursery() as nursery:
+            portal: tractor.Portal = await an.run_in_actor(
+                fn=spawn,
 
-                # forks here
+                # spawning args
-                portal = await nursery.run_in_actor(
-                    spawn,
-                    is_arbiter=False,
                 name='sub-actor',
-                    data=data,
+                enable_modules=[__name__],
+
+                # passed to a subactor-recursive RPC invoke
+                # of this same `spawn()` fn.
+                should_be_root=False,
+                data=data_to_pass_down,
                 reg_addr=reg_addr,
-                    enable_modules=namespaces,
             )
 
-                assert len(nursery._children) == 1
+            assert len(an._children) == 1
-                assert portal.channel.uid in tractor.current_actor()._peers
+            assert (
-                # be sure we can still get the result
+                portal.channel.uid
+                in
+                tractor.current_actor().ipc_server._peers
+            )
+
+            # get result from child subactor
             result = await portal.result()
             assert result == 10
             return result
     else:
+        assert actor.is_registrar == should_be_root
         return 10
 
 
-def test_local_arbiter_subactor_global_state(
+def test_run_in_actor_same_func_in_child(
-    reg_addr,
+    reg_addr: tuple,
+    debug_mode: bool,
 ):
     result = trio.run(
+        partial(
             spawn,
-        True,
+            should_be_root=True,
-        data_to_pass_down,
+            data=data_to_pass_down,
-        reg_addr,
+            reg_addr=reg_addr,
+            debug_mode=debug_mode,
+        )
     )
     assert result == 10
 
 
 async def movie_theatre_question():
-    """A question asked in a dark theatre, in a tangent
+    '''
+    A question asked in a dark theatre, in a tangent
     (errr, I mean different) process.
-    """
+
+    '''
     return 'have you ever seen a portal?'
 
 
 @tractor_test
 async def test_movie_theatre_convo(start_method):
-    """The main ``tractor`` routine.
+    '''
-    """
+    The main ``tractor`` routine.
-    async with tractor.open_nursery() as n:
 
-        portal = await n.start_actor(
+    '''
+    async with tractor.open_nursery(debug_mode=True) as an:
+
+        portal = await an.start_actor(
             'frank',
             # enable the actor to run funcs from this current module
             enable_modules=[__name__],
@@ -118,8 +144,8 @@ async def test_most_beautiful_word(
     with trio.fail_after(1):
         async with tractor.open_nursery(
             debug_mode=debug_mode,
-        ) as n:
+        ) as an:
-            portal = await n.run_in_actor(
+            portal = await an.run_in_actor(
                 cellar_door,
                 return_value=return_value,
                 name='some_linguist',
@@ -155,7 +181,7 @@ def test_loglevel_propagated_to_subactor(
 
     async def main():
         async with tractor.open_nursery(
-            name='arbiter',
+            name='registrar',
             start_method=start_method,
             arbiter_addr=reg_addr,
 

tests/test_trioisms.py

@@ -6,10 +6,18 @@ want to see changed.
 from contextlib import (
     asynccontextmanager as acm,
 )
+from types import ModuleType
+
+from functools import partial
 
 import pytest
+from _pytest import pathlib
+from tractor.trionics import collapse_eg
 import trio
 from trio import TaskStatus
+from tractor._testing import (
+    examples_dir,
+)
 
 
 @pytest.mark.parametrize(
@@ -64,9 +72,8 @@ def test_stashed_child_nursery(use_start_soon):
     async def main():
 
         async with (
-            trio.open_nursery(
+            collapse_eg(),
-                strict_exception_groups=False,
+            trio.open_nursery() as pn,
-            ) as pn,
         ):
             cn = await pn.start(mk_child_nursery)
             assert cn
@@ -106,66 +113,21 @@ def test_acm_embedded_nursery_propagates_enter_err(
     debug_mode: bool,
 ):
     '''
-    Demo how a masking `trio.Cancelled` could be handled by unmasking from the
+    Demo how a masking `trio.Cancelled` could be handled by unmasking
-    `.__context__` field when a user (by accident) re-raises from a `finally:`.
+    from the `.__context__` field when a user (by accident) re-raises
+    from a `finally:`.
 
     '''
     import tractor
 
-    @acm
-    async def maybe_raise_from_masking_exc(
-        tn: trio.Nursery,
-        unmask_from: BaseException|None = trio.Cancelled
-
-        # TODO, maybe offer a collection?
-        # unmask_from: set[BaseException] = {
-        #     trio.Cancelled,
-        # },
-    ):
-        if not unmask_from:
-            yield
-            return
-
-        try:
-            yield
-        except* unmask_from as be_eg:
-
-            # TODO, if we offer `unmask_from: set`
-            # for masker_exc_type in unmask_from:
-
-            matches, rest = be_eg.split(unmask_from)
-            if not matches:
-                raise
-
-            for exc_match in be_eg.exceptions:
-                if (
-                    (exc_ctx := exc_match.__context__)
-                    and
-                    type(exc_ctx) not in {
-                        # trio.Cancelled,  # always by default?
-                        unmask_from,
-                    }
-                ):
-                    exc_ctx.add_note(
-                        f'\n'
-                        f'WARNING: the above error was masked by a {unmask_from!r} !?!\n'
-                        f'Are you always cancelling? Say from a `finally:` ?\n\n'
-
-                        f'{tn!r}'
-                    )
-                    raise exc_ctx from exc_match
-
-
     @acm
     async def wraps_tn_that_always_cancels():
         async with (
             trio.open_nursery() as tn,
-            maybe_raise_from_masking_exc(
+            tractor.trionics.maybe_raise_from_masking_exc(
-                tn=tn,
                 unmask_from=(
-                    trio.Cancelled
+                    (trio.Cancelled,) if unmask_from_canc
-                    if unmask_from_canc
+                    else ()
-                    else None
                 ),
             )
         ):
@@ -188,6 +150,7 @@ def test_acm_embedded_nursery_propagates_enter_err(
                 assert not tn.cancel_scope.cancel_called
                 assert 0
 
+        if debug_mode:
             assert (
                 (err := bxerr.value)
                 and
@@ -201,3 +164,139 @@ def test_acm_embedded_nursery_propagates_enter_err(
     assert_eg, rest_eg = eg.split(AssertionError)
 
     assert len(assert_eg.exceptions) == 1
+
+
+def test_gatherctxs_with_memchan_breaks_multicancelled(
+    debug_mode: bool,
+):
+    '''
+    Demo how a using an `async with sndchan` inside
+    a `.trionics.gather_contexts()` task will break a strict-eg-tn's
+    multi-cancelled absorption..
+
+    '''
+    from tractor import (
+        trionics,
+    )
+
+    @acm
+    async def open_memchan() -> trio.abc.ReceiveChannel:
+
+        task: trio.Task = trio.lowlevel.current_task()
+        print(
+            f'Opening {task!r}\n'
+        )
+
+        # 1 to force eager sending
+        send, recv = trio.open_memory_channel(16)
+
+        try:
+            async with send:
+                yield recv
+        finally:
+            print(
+                f'Closed {task!r}\n'
+            )
+
+    async def main():
+        async with (
+            # XXX should ensure ONLY the KBI
+            # is relayed upward
+            collapse_eg(),
+            trio.open_nursery(), # as tn,
+
+            trionics.gather_contexts([
+                open_memchan(),
+                open_memchan(),
+            ]) as recv_chans,
+        ):
+            assert len(recv_chans) == 2
+
+            await trio.sleep(1)
+            raise KeyboardInterrupt
+            # tn.cancel_scope.cancel()
+
+    with pytest.raises(KeyboardInterrupt):
+        trio.run(main)
+
+
+@pytest.mark.parametrize(
+    'raise_unmasked', [
+        True,
+        pytest.param(
+            False,
+            marks=pytest.mark.xfail(
+                reason="see examples/trio/send_chan_aclose_masks.py"
+            )
+        ),
+    ]
+)
+@pytest.mark.parametrize(
+    'child_errors_mid_stream',
+    [True, False],
+)
+def test_unmask_aclose_as_checkpoint_on_aexit(
+    raise_unmasked: bool,
+    child_errors_mid_stream: bool,
+    debug_mode: bool,
+):
+    '''
+    Verify that our unmasker util works over the common case where
+    a mem-chan's `.aclose()` is included in an `@acm` stack
+    and it being currently a checkpoint, can `trio.Cancelled`-mask an embedded
+    exception from user code resulting in a silent failure which
+    appears like graceful cancellation.
+
+    This test suite is mostly implemented as an example script so it
+    could more easily be shared with `trio`-core peeps as `tractor`-less
+    minimum reproducing example.
+
+    '''
+    mod: ModuleType = pathlib.import_path(
+        examples_dir()
+        / 'trio'
+        / 'send_chan_aclose_masks_beg.py',
+        root=examples_dir(),
+        consider_namespace_packages=False,
+    )
+    with pytest.raises(RuntimeError):
+        trio.run(partial(
+            mod.main,
+            raise_unmasked=raise_unmasked,
+            child_errors_mid_stream=child_errors_mid_stream,
+        ))
+
+
+
+@pytest.mark.parametrize(
+    'ignore_special_cases', [
+        True,
+        pytest.param(
+            False,
+            marks=pytest.mark.xfail(
+                reason="see examples/trio/lockacquire_not_umasked.py"
+            )
+        ),
+    ]
+)
+def test_cancelled_lockacquire_in_ipctx_not_unmasked(
+    ignore_special_cases: bool,
+    loglevel: str,
+    debug_mode: bool,
+):
+    mod: ModuleType = pathlib.import_path(
+        examples_dir()
+        / 'trio'
+        / 'lockacquire_not_unmasked.py',
+        root=examples_dir(),
+        consider_namespace_packages=False,
+    )
+    async def _main():
+        with trio.fail_after(2):
+            await mod.main(
+                ignore_special_cases=ignore_special_cases,
+                loglevel=loglevel,
+                debug_mode=debug_mode,
+            )
+
+    trio.run(_main)

tractor/__init__.py

@@ -30,21 +30,23 @@ from ._streaming import (
     MsgStream as MsgStream,
     stream as stream,
 )
-from ._discovery import (
+from .discovery._discovery import (
     get_registry as get_registry,
     find_actor as find_actor,
     wait_for_actor as wait_for_actor,
     query_actor as query_actor,
 )
-from ._supervise import (
+from .runtime._supervise import (
     open_nursery as open_nursery,
     ActorNursery as ActorNursery,
 )
-from ._state import (
+from .runtime._state import (
+    RuntimeVars as RuntimeVars,
     current_actor as current_actor,
-    is_root_process as is_root_process,
     current_ipc_ctx as current_ipc_ctx,
-    debug_mode as debug_mode
+    debug_mode as debug_mode,
+    get_runtime_vars as get_runtime_vars,
+    is_root_process as is_root_process,
 )
 from ._exceptions import (
     ContextCancelled as ContextCancelled,
@@ -64,7 +66,11 @@ from ._root import (
     run_daemon as run_daemon,
     open_root_actor as open_root_actor,
 )
-from ._ipc import Channel as Channel
+from .ipc import Channel as Channel
-from ._portal import Portal as Portal
+from .runtime._portal import Portal as Portal
-from ._runtime import Actor as Actor
+from .runtime._runtime import Actor as Actor
+from .discovery._registry import (
+    Registrar as Registrar,
+    Arbiter as Arbiter,
+)
 # from . import hilevel as hilevel

tractor/_child.py

@@ -22,8 +22,8 @@ import argparse
 
 from ast import literal_eval
 
-from ._runtime import Actor
+from .runtime._runtime import Actor
-from ._entry import _trio_main
+from .spawn._entry import _trio_main
 
 
 def parse_uid(arg):
@@ -31,8 +31,12 @@ def parse_uid(arg):
     return str(name), str(uuid)  # ensures str encoding
 
 def parse_ipaddr(arg):
-    host, port = literal_eval(arg)
+    try:
-    return (str(host), int(port))
+        return literal_eval(arg)
+
+    except (ValueError, SyntaxError):
+        # UDS: try to interpret as a straight up str
+        return arg
 
 
 if __name__ == "__main__":
@@ -46,8 +50,8 @@ if __name__ == "__main__":
     args = parser.parse_args()
 
     subactor = Actor(
-        args.uid[0],
+        name=args.uid[0],
-        uid=args.uid[1],
+        uuid=args.uid[1],
         loglevel=args.loglevel,
         spawn_method="trio"
     )

tractor/_clustering.py

@@ -55,11 +55,18 @@ async def open_actor_cluster(
         raise ValueError(
             'Number of names is {len(names)} but count it {count}')
 
-    async with tractor.open_nursery(
+    async with (
+        # tractor.trionics.collapse_eg(),
+        tractor.open_nursery(
             **runtime_kwargs,
-    ) as an:
+        ) as an
-        async with trio.open_nursery() as n:
+    ):
-            uid = tractor.current_actor().uid
+        async with (
+            # tractor.trionics.collapse_eg(),
+            trio.open_nursery() as tn,
+            tractor.trionics.maybe_raise_from_masking_exc()
+        ):
+            uid = tractor.current_actor().aid.uid
 
             async def _start(name: str) -> None:
                 name = f'{uid[0]}.{name}'
@@ -69,9 +76,8 @@ async def open_actor_cluster(
                 )
 
             for name in names:
-                n.start_soon(_start, name)
+                tn.start_soon(_start, name)
 
         assert len(portals) == count
         yield portals
-
         await an.cancel(hard_kill=hard_kill)

tractor/_code_load.py

@@ -0,0 +1,48 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+(Hot) coad (re-)load utils for python.
+
+'''
+import importlib
+from pathlib import Path
+import sys
+from types import ModuleType
+
+# ?TODO, move this into internal libs?
+# -[ ] we already use it in `modden.config._pymod` as well
+def load_module_from_path(
+    path: Path,
+    module_name: str|None = None,
+) -> ModuleType:
+    '''
+    Taken from SO,
+    https://stackoverflow.com/a/67208147
+
+    which is based on stdlib docs,
+    https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
+
+    '''
+    module_name = module_name or path.stem
+    spec = importlib.util.spec_from_file_location(
+        module_name,
+        str(path),
+    )
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)
+    return module

tractor/_context.py

@@ -70,6 +70,7 @@ from ._exceptions import (
     MsgTypeError,
     RemoteActorError,
     StreamOverrun,
+    TransportClosed,
     pack_from_raise,
     unpack_error,
 )
@@ -89,28 +90,31 @@ from .msg import (
     pretty_struct,
     _ops as msgops,
 )
-from ._ipc import (
+from .ipc import (
     Channel,
 )
 from ._streaming import (
     MsgStream,
     open_stream_from_ctx,
 )
-from ._state import (
+from .runtime._state import (
     current_actor,
     debug_mode,
     _ctxvar_Context,
 )
+from .trionics import (
+    collapse_eg,
+)
 # ------ - ------
 if TYPE_CHECKING:
-    from ._portal import Portal
+    from .runtime._portal import Portal
-    from ._runtime import Actor
+    from .runtime._runtime import Actor
-    from ._ipc import MsgTransport
+    from .ipc._transport import MsgTransport
     from .devx._frame_stack import (
         CallerInfo,
     )
 
-log = get_logger(__name__)
+log = get_logger()
 
 
 class Unresolved:
@@ -151,7 +155,7 @@ class Context:
     2 cancel-scope-linked, communicating and parallel executing
     `Task`s. Contexts are allocated on each side of any task
     RPC-linked msg dialog, i.e. for every request to a remote
-    actor from a `Portal`. On the "callee" side a context is
+    actor from a `Portal`. On the "child" side a context is
     always allocated inside `._rpc._invoke()`.
 
     TODO: more detailed writeup on cancellation, error and
@@ -219,8 +223,8 @@ class Context:
     # `._runtime.invoke()`.
     _remote_func_type: str | None = None
 
-    # NOTE: (for now) only set (a portal) on the caller side since
+    # NOTE: (for now) only set (a portal) on the parent side since
-    # the callee doesn't generally need a ref to one and should
+    # the child doesn't generally need a ref to one and should
     # normally need to explicitly ask for handle to its peer if
     # more the the `Context` is needed?
     _portal: Portal | None = None
@@ -249,12 +253,12 @@ class Context:
     _outcome_msg: Return|Error|ContextCancelled = Unresolved
 
     # on a clean exit there should be a final value
-    # delivered from the far end "callee" task, so
+    # delivered from the far end "child" task, so
     # this value is only set on one side.
     # _result: Any | int = None
     _result: PayloadT|Unresolved = Unresolved
 
-    # if the local "caller"  task errors this value is always set
+    # if the local "parent"  task errors this value is always set
     # to the error that was captured in the
     # `Portal.open_context().__aexit__()` teardown block OR, in
     # 2 special cases when an (maybe) expected remote error
@@ -290,9 +294,9 @@ class Context:
     # a `ContextCancelled` due to a call to `.cancel()` triggering
     # "graceful closure" on either side:
     # - `._runtime._invoke()` will check this flag before engaging
-    #   the crash handler REPL in such cases where the "callee"
+    #   the crash handler REPL in such cases where the "child"
     #   raises the cancellation,
-    # - `.devx._debug.lock_stdio_for_peer()` will set it to `False` if
+    # - `.devx.debug.lock_stdio_for_peer()` will set it to `False` if
     #   the global tty-lock has been configured to filter out some
     #   actors from being able to acquire the debugger lock.
     _enter_debugger_on_cancel: bool = True
@@ -304,8 +308,8 @@ class Context:
     _stream_opened: bool = False
     _stream: MsgStream|None = None
 
-    # caller of `Portal.open_context()` for
+    # the parent-task's calling-fn's frame-info, the frame above
-    # logging purposes mostly
+    # `Portal.open_context()`, for introspection/logging.
     _caller_info: CallerInfo|None = None
 
     # overrun handling machinery
@@ -366,7 +370,7 @@ class Context:
             # f'   ---\n'
             f' |_ipc: {self.dst_maddr}\n'
             # f'   dst_maddr{ds}{self.dst_maddr}\n'
-            f"   uid{ds}'{self.chan.uid}'\n"
+            f"   uid{ds}'{self.chan.aid}'\n"
             f"   cid{ds}'{self.cid}'\n"
             # f'   ---\n'
             f'\n'
@@ -439,30 +443,31 @@ class Context:
         '''
         Records whether cancellation has been requested for this context
         by a call to  `.cancel()` either due to,
-        - either an explicit call by some local task,
+        - an explicit call by some local task,
         - or an implicit call due to an error caught inside
-          the ``Portal.open_context()`` block.
+          the `Portal.open_context()` block.
 
         '''
         return self._cancel_called
 
-    @cancel_called.setter
+    # XXX, to debug who frickin sets it..
-    def cancel_called(self, val: bool) -> None:
+    # @cancel_called.setter
-        '''
+    # def cancel_called(self, val: bool) -> None:
-        Set the self-cancelled request `bool` value.
+    #     '''
+    #     Set the self-cancelled request `bool` value.
 
-        '''
+    #     '''
-        # to debug who frickin sets it..
     #     if val:
     #         from .devx import pause_from_sync
     #         pause_from_sync()
 
-        self._cancel_called = val
+    #     self._cancel_called = val
 
+    # TODO, use the `Actor.aid: Aid` instead!
     @property
     def canceller(self) -> tuple[str, str]|None:
         '''
-        `Actor.uid: tuple[str, str]` of the (remote)
+        `Actor.aid.uid: tuple[str, str]` of the (remote)
         actor-process who's task was cancelled thus causing this
         (side of the) context to also be cancelled.
 
@@ -495,12 +500,12 @@ class Context:
         if from_uid := re.src_uid:
             from_uid: tuple = tuple(from_uid)
 
-        our_uid: tuple = self._actor.uid
+        our_uid: tuple = self._actor.aid.uid
         our_canceller = self.canceller
 
         return bool(
             isinstance((ctxc := re), ContextCancelled)
-            and from_uid == self.chan.uid
+            and from_uid == self.chan.aid.uid
             and ctxc.canceller == our_uid
             and our_canceller == our_uid
         )
@@ -511,7 +516,7 @@ class Context:
         Records whether the task on the remote side of this IPC
         context acknowledged a cancel request via a relayed
         `ContextCancelled` with the `.canceller` attr set to the
-        `Actor.uid` of the local actor who's task entered
+        `Actor.aid.uid` of the local actor who's task entered
         `Portal.open_context()`.
 
         This will only be `True` when `.cancel()` is called and
@@ -526,11 +531,11 @@ class Context:
         '''
         Exactly the value of `self._scope.cancelled_caught`
         (delegation) and should only be (able to be read as)
-        `True` for a `.side == "caller"` ctx wherein the
+        `True` for a `.side == "parent"` ctx wherein the
         `Portal.open_context()` block was exited due to a call to
         `._scope.cancel()` - which should only ocurr in 2 cases:
 
-        - a caller side calls `.cancel()`, the far side cancels
+        - a parent side calls `.cancel()`, the far side cancels
           and delivers back a `ContextCancelled` (making
           `.cancel_acked == True`) and `._scope.cancel()` is
           called by `._maybe_cancel_and_set_remote_error()` which
@@ -539,20 +544,20 @@ class Context:
           => `._scope.cancelled_caught == True` by normal `trio`
           cs semantics.
 
-        - a caller side is delivered a `._remote_error:
+        - a parent side is delivered a `._remote_error:
           RemoteActorError` via `._deliver_msg()` and a transitive
           call to `_maybe_cancel_and_set_remote_error()` calls
           `._scope.cancel()` and that cancellation eventually
           results in `trio.Cancelled`(s) caught in the
           `.open_context()` handling around the @acm's `yield`.
 
-        Only as an FYI, in the "callee" side case it can also be
+        Only as an FYI, in the "child" side case it can also be
         set but never is readable by any task outside the RPC
         machinery in `._invoke()` since,:
-        - when a callee side calls `.cancel()`, `._scope.cancel()`
+        - when a child side calls `.cancel()`, `._scope.cancel()`
           is called immediately and handled specially inside
           `._invoke()` to raise a `ContextCancelled` which is then
-          sent to the caller side.
+          sent to the parent side.
 
           However, `._scope.cancelled_caught` can NEVER be
           accessed/read as `True` by any RPC invoked task since it
@@ -632,6 +637,71 @@ class Context:
         '''
         await self.chan.send(Stop(cid=self.cid))
 
+    @property
+    def parent_task(self) -> trio.Task:
+        '''
+        This IPC context's "owning task" which is a `trio.Task`
+        on one of the "sides" of the IPC.
+
+        Note that the "parent_" prefix here refers to the local
+        `trio` task tree using the same interface as
+        `trio.Nursery.parent_task` whereas for IPC contexts,
+        a different cross-actor task hierarchy exists:
+
+        - a "parent"-side which originally entered
+          `Portal.open_context()`,
+
+        - the "child"-side which was spawned and scheduled to invoke
+          a function decorated with `@tractor.context`.
+
+        This task is thus a handle to mem-domain-distinct/per-process
+        `Nursery.parent_task` depending on in which of the above
+        "sides" this context exists.
+
+        '''
+        return self._task
+
+    def _is_blocked_on_rx_chan(self) -> bool:
+        '''
+        Predicate to indicate whether the owner `._task: trio.Task` is
+        currently blocked (by `.receive()`-ing) on its underlying RPC
+        feeder `._rx_chan`.
+
+        This knowledge is highly useful when handling so called
+        "out-of-band" (OoB) cancellation conditions where a peer
+        actor's task transmitted some remote error/cancel-msg and we
+        must know whether to signal-via-cancel currently executing
+        "user-code" (user defined code embedded in `ctx._scope`) or
+        simply to forward the IPC-msg-as-error **without calling**
+        `._scope.cancel()`.
+
+        In the latter case it is presumed that if the owner task is
+        blocking for the next IPC msg, it will eventually receive,
+        process and raise the equivalent local error **without**
+        requiring `._scope.cancel()` to be explicitly called by the
+        *delivering OoB RPC-task* (via `_deliver_msg()`).
+
+        '''
+        # NOTE, see the mem-chan meth-impls for *why* this
+        # logic works,
+        # `trio._channel.MemoryReceiveChannel.receive[_nowait]()`
+        #
+        # XXX realize that this is NOT an
+        # official/will-be-loudly-deprecated API:
+        # - https://trio.readthedocs.io/en/stable/reference-lowlevel.html#trio.lowlevel.Task.custom_sleep_data
+        #  |_https://trio.readthedocs.io/en/stable/reference-lowlevel.html#trio.lowlevel.wait_task_rescheduled
+        #
+        # orig repo intro in the mem-chan change over patch:
+        # - https://github.com/python-trio/trio/pull/586#issuecomment-414039117
+        #  |_https://github.com/python-trio/trio/pull/616
+        #  |_https://github.com/njsmith/trio/commit/98c38cef6f62e731bf8c7190e8756976bface8f0
+        #
+        return (
+            self._task.custom_sleep_data
+            is
+            self._rx_chan
+        )
+
     def _maybe_cancel_and_set_remote_error(
         self,
         error: BaseException,
@@ -663,7 +733,7 @@ class Context:
         when called/closed by actor local task(s).
 
         NOTEs: 
-          - It is expected that the caller has previously unwrapped
+          - It is expected that the parent has previously unwrapped
             the remote error using a call to `unpack_error()` and
             provides that output exception value as the input
             `error` argument *here*.
@@ -673,7 +743,7 @@ class Context:
             `Portal.open_context()` (ideally) we want to interrupt
             any ongoing local tasks operating within that
             `Context`'s cancel-scope so as to be notified ASAP of
-            the remote error and engage any caller handling (eg.
+            the remote error and engage any parent handling (eg.
             for cross-process task supervision).
 
           - In some cases we may want to raise the remote error
@@ -720,8 +790,8 @@ class Context:
         # appropriately.
         log.runtime(
             'Setting remote error for ctx\n\n'
-            f'<= {self.peer_side!r}: {self.chan.uid}\n'
+            f'<= {self.peer_side!r}: {self.chan.aid.reprol()}\n'
-            f'=> {self.side!r}: {self._actor.uid}\n\n'
+            f'=> {self.side!r}: {self._actor.aid.reprol()}\n\n'
             f'{error!r}'
         )
         self._remote_error: BaseException = error
@@ -740,7 +810,9 @@ class Context:
             # cancelled, NOT their reported canceller. IOW in the
             # latter case we're cancelled by someone else getting
             # cancelled.
-            if (canc := error.canceller) == self._actor.uid:
+            #
+            # !TODO, switching to `Actor.aid` here!
+            if (canc := error.canceller) == self._actor.aid.uid:
                 whom: str = 'us'
                 self._canceller = canc
             else:
@@ -782,13 +854,27 @@ class Context:
         if self._canceller is None:
             log.error('Ctx has no canceller set!?')
 
+        cs: trio.CancelScope = self._scope
+
+        # ?TODO? see comment @ .start_remote_task()`
+        #
+        # if not cs:
+        #     from .devx import mk_pdb
+        #     mk_pdb().set_trace()
+        #     raise RuntimeError(
+        #         f'IPC ctx was not be opened prior to remote error delivery !?\n'
+        #         f'{self}\n'
+        #         f'\n'
+        #         f'`Portal.open_context()` must be entered (somewhere) beforehand!\n'
+        #     )
+
         # Cancel the local `._scope`, catch that
         # `._scope.cancelled_caught` and re-raise any remote error
         # once exiting (or manually calling `.wait_for_result()`) the
         # `.open_context()`  block.
-        cs: trio.CancelScope = self._scope
         if (
             cs
+            and not cs.cancel_called
 
             # XXX this is an expected cancel request response
             # message and we **don't need to raise it** in the
@@ -797,8 +883,7 @@ class Context:
             # if `._cancel_called` then `.cancel_acked and .cancel_called`
             # always should be set.
             and not self._is_self_cancelled()
-            and not cs.cancel_called
+            # and not cs.cancelled_caught
-            and not cs.cancelled_caught
         ):
             if (
                 msgerr
@@ -809,7 +894,7 @@ class Context:
                 not self._cancel_on_msgerr
             ):
                 message: str = (
-                    'NOT Cancelling `Context._scope` since,\n'
+                    f'NOT Cancelling `Context._scope` since,\n'
                     f'Context._cancel_on_msgerr = {self._cancel_on_msgerr}\n\n'
                     f'AND we got a msg-type-error!\n'
                     f'{error}\n'
@@ -819,13 +904,43 @@ class Context:
                 # `trio.Cancelled` subtype here ;)
                 # https://github.com/goodboy/tractor/issues/368
                 message: str = 'Cancelling `Context._scope` !\n\n'
-                # from .devx import pause_from_sync
+                cs.cancel()
-                # pause_from_sync()
+
-                self._scope.cancel()
+        # TODO, explicit condition for OoB (self-)cancellation?
-        else:
+        # - we called `Portal.cancel_actor()` from this actor
-            message: str = 'NOT cancelling `Context._scope` !\n\n'
+        #   and the peer ctx task delivered ctxc due to it.
+        # - currently `self._is_self_cancelled()` will be true
+        #   since the ctxc.canceller check will match us even though it
+        #   wasn't from this ctx specifically!
+        elif (
+            cs
+            and self._is_self_cancelled()
+            and not cs.cancel_called
+        ):
+            message: str = (
+                'Cancelling `ctx._scope` due to OoB self-cancel ?!\n'
+                '\n'
+            )
             # from .devx import mk_pdb
             # mk_pdb().set_trace()
+            # TODO XXX, required to fix timeout failure in
+            # `test_cancelled_lockacquire_in_ipctx_not_unmaskeed`
+            #
+
+            # XXX NOTE XXX, this is SUPER SUBTLE!
+            # we only want to cancel our embedded `._scope`
+            # if the ctx's current/using task is NOT blocked
+            # on `._rx_chan.receive()` and on some other
+            # `trio`-checkpoint since in the former case
+            # any `._remote_error` will be relayed through
+            # the rx-chan and appropriately raised by the owning
+            # `._task` directly. IF the owner task is however
+            # blocking elsewhere we need to interrupt it **now**.
+            if not self._is_blocked_on_rx_chan():
+                cs.cancel()
+        else:
+            # rx_stats = self._rx_chan.statistics()
+            message: str = 'NOT cancelling `Context._scope` !\n\n'
 
         fmt_str: str = 'No `self._scope: CancelScope` was set/used ?\n'
         if (
@@ -849,6 +964,7 @@ class Context:
                 +
                 cs_fmt
             )
+
         log.cancel(
             message
             +
@@ -859,19 +975,10 @@ class Context:
     @property
     def dst_maddr(self) -> str:
         chan: Channel = self.chan
-        dst_addr, dst_port = chan.raddr
         trans: MsgTransport = chan.transport
         # cid: str = self.cid
         # cid_head, cid_tail = cid[:6], cid[-6:]
-        return (
+        return trans.maddr
-            f'/ipv4/{dst_addr}'
-            f'/{trans.name_key}/{dst_port}'
-            # f'/{self.chan.uid[0]}'
-            # f'/{self.cid}'
-
-            # f'/cid={cid_head}..{cid_tail}'
-            # TODO: ? not use this ^ right ?
-        )
 
     dmaddr = dst_maddr
 
@@ -890,6 +997,11 @@ class Context:
 
     @property
     def repr_caller(self) -> str:
+        '''
+        Render a "namespace-path" style representation of the calling
+        task-fn.
+
+        '''
         ci: CallerInfo|None = self._caller_info
         if ci:
             return (
@@ -903,7 +1015,7 @@ class Context:
     def repr_api(self) -> str:
         return 'Portal.open_context()'
 
-        # TODO: use `.dev._frame_stack` scanning to find caller!
+        # TODO: use `.dev._frame_stack` scanning to find caller fn!
         # ci: CallerInfo|None = self._caller_info
         # if ci:
         #     return (
@@ -925,7 +1037,7 @@ class Context:
         ---------
         - after the far end cancels, the `.cancel()` calling side
           should receive a `ContextCancelled` with the
-          `.canceller: tuple` uid set to the current `Actor.uid`.
+          `.canceller: tuple` uid set to the current `Actor.aid.uid`.
 
         - timeout (quickly) on failure to rx this ACK error-msg in
           an attempt to sidestep 2-generals when the transport
@@ -938,25 +1050,26 @@ class Context:
         => That is, an IPC `Context` (this) **does not**
            have the same semantics as a `trio.CancelScope`.
 
-        If the caller (who entered the `Portal.open_context()`)
+        If the parent (who entered the `Portal.open_context()`)
         desires that the internal block's cancel-scope  be
         cancelled it should open its own `trio.CancelScope` and
         manage it as needed.
 
         '''
         side: str = self.side
-        # XXX for debug via the `@.setter`
+        self._cancel_called = True
-        self.cancel_called = True
+        # ^ XXX for debug via the `@.setter`
+        # self.cancel_called = True
 
         header: str = (
-            f'Cancelling ctx from {side.upper()}-side\n'
+            f'Cancelling ctx from {side!r}-side\n'
         )
         reminfo: str = (
             # ' =>\n'
-            # f'Context.cancel() => {self.chan.uid}\n'
+            # f'Context.cancel() => {self.chan.aid.uid}\n'
-            f'c)=> {self.chan.uid}\n'
+            f'\n'
-            # f'{self.chan.uid}\n'
+            f'c)=> {self.chan.aid.reprol()}\n'
-            f'  |_ @{self.dst_maddr}\n'
+            f'   |_[{self.dst_maddr}\n'
             f'     >> {self.repr_rpc}\n'
             # f'    >> {self._nsf}() -> {codec}[dict]:\n\n'
             # TODO: pull msg-type from spec re #320
@@ -1010,7 +1123,6 @@ class Context:
                 else:
                     log.cancel(
                         f'Timed out on cancel request of remote task?\n'
-                        f'\n'
                         f'{reminfo}'
                     )
 
@@ -1021,7 +1133,7 @@ class Context:
         # `_invoke()` RPC task.
         #
         # NOTE: on this side we ALWAYS cancel the local scope
-        # since the caller expects a `ContextCancelled` to be sent
+        # since the parent expects a `ContextCancelled` to be sent
         # from `._runtime._invoke()` back to the other side. The
         # logic for catching the result of the below
         # `._scope.cancel()` is inside the `._runtime._invoke()`
@@ -1078,13 +1190,29 @@ class Context:
         |RemoteActorError  # stream overrun caused and ignored by us
     ):
         '''
-        Maybe raise a remote error depending on the type of error
+        Maybe raise a remote error depending on the type of error and
-        and *who* (i.e. which task from which actor) requested
+        *who*, i.e. which side of the task pair across actors,
-        a  cancellation (if any).
+        requested a cancellation (if any).
+
+        Depending on the input config-params suppress raising
+        certain remote excs:
+
+        - if `remote_error: ContextCancelled` (ctxc) AND this side's
+          task is the "requester", it at somem point called
+          `Context.cancel()`, then the peer's ctxc is treated
+          as a "cancel ack".
+
+         |_ this behaves exactly like how `trio.Nursery.cancel_scope`
+            absorbs any `BaseExceptionGroup[trio.Cancelled]` wherein the
+            owning parent task never will raise a `trio.Cancelled`
+            if `CancelScope.cancel_called == True`.
+
+        - `remote_error: StreamOverrrun` (overrun) AND
+           `raise_overrun_from_self` is set.
 
         '''
         __tracebackhide__: bool = hide_tb
-        peer_uid: tuple = self.chan.uid
+        peer_uid: tuple = self.chan.aid.uid
 
         # XXX NOTE XXX: `ContextCancelled`/`StreamOverrun` absorption
         # for "graceful cancellation" case(s):
@@ -1101,7 +1229,7 @@ class Context:
         # (`ContextCancelled`) as an expected
         # error-msg-is-cancellation-ack IFF said
         # `remote_error: ContextCancelled` has `.canceller`
-        # set to the `Actor.uid` of THIS task (i.e. the
+        # set to the `Actor.aid.uid` of THIS task (i.e. the
         # cancellation requesting task's actor is the actor
         # checking whether it should absorb the ctxc).
         self_ctxc: bool = self._is_self_cancelled(remote_error)
@@ -1122,18 +1250,19 @@ class Context:
             # for this ^, NO right?
 
         ) or (
-            # NOTE: whenever this context is the cause of an
+            # NOTE: whenever this side is the cause of an
-            # overrun on the remote side (aka we sent msgs too
+            # overrun on the peer side, i.e. we sent msgs too
-            # fast that the remote task was overrun according
+            # fast and the peer task was overrun according
-            # to `MsgStream` buffer settings) AND the caller
+            # to `MsgStream` buffer settings, AND this was
-            # has requested to not raise overruns this side
+            # called with `raise_overrun_from_self=True` (the
-            # caused, we also silently absorb any remotely
+            # default), silently absorb any `StreamOverrun`.
-            # boxed `StreamOverrun`. This is mostly useful for
+            #
-            # supressing such faults during
+            # XXX, this is namely useful for supressing such faults
-            # cancellation/error/final-result handling inside
+            # during cancellation/error/final-result handling inside
-            # `msg._ops.drain_to_final_msg()` such that we do not
+            # `.msg._ops.drain_to_final_msg()` such that we do not
-            # raise such errors particularly in the case where
+            # raise during a cancellation-request, i.e. when
             # `._cancel_called == True`.
+            #
             not raise_overrun_from_self
             and isinstance(remote_error, RemoteActorError)
             and remote_error.boxed_type is StreamOverrun
@@ -1177,8 +1306,8 @@ class Context:
 
     ) -> Any|Exception:
         '''
-        From some (caller) side task, wait for and return the final
+        From some (parent) side task, wait for and return the final
-        result from the remote (callee) side's task.
+        result from the remote (child) side's task.
 
         This provides a mechanism for one task running in some actor to wait
         on another task at the other side, in some other actor, to terminate.
@@ -1243,8 +1372,8 @@ class Context:
 
             # ?XXX, should already be set in `._deliver_msg()` right?
             if self._outcome_msg is not Unresolved:
-                # from .devx import _debug
+                # from .devx import debug
-                # await _debug.pause()
+                # await debug.pause()
                 assert self._outcome_msg is outcome_msg
             else:
                 self._outcome_msg = outcome_msg
@@ -1474,6 +1603,12 @@ class Context:
                 ):
                     status = 'peer-cancelled'
 
+            case (
+                Unresolved,
+                trio.Cancelled(),  # any error-type
+            ) if self.canceller:
+                status = 'actor-cancelled'
+
             # (remote) error condition
             case (
                 Unresolved,
@@ -1545,7 +1680,7 @@ class Context:
 
         elif self._started_called:
             raise RuntimeError(
-                f'called `.started()` twice on context with {self.chan.uid}'
+                f'called `.started()` twice on context with {self.chan.aid.uid}'
             )
 
         started_msg = Started(
@@ -1587,7 +1722,7 @@ class Context:
                 raise err
 
         # TODO: maybe a flag to by-pass encode op if already done
-        # here in caller?
+        # here in parent?
         await self.chan.send(started_msg)
 
         # set msg-related internal runtime-state
@@ -1663,7 +1798,7 @@ class Context:
 
          XXX RULES XXX
         ------ - ------
-        - NEVER raise remote errors from this method; a runtime task caller.
+        - NEVER raise remote errors from this method; a calling runtime-task.
           An error "delivered" to a ctx should always be raised by
           the corresponding local task operating on the
           `Portal`/`Context` APIs.
@@ -1678,7 +1813,7 @@ class Context:
         '''
         cid: str = self.cid
         chan: Channel = self.chan
-        from_uid: tuple[str, str]  = chan.uid
+        from_uid: tuple[str, str]  = chan.aid.uid
         send_chan: trio.MemorySendChannel = self._send_chan
         nsf: NamespacePath = self._nsf
 
@@ -1739,7 +1874,7 @@ class Context:
 
             else:
                 report = (
-                    'Queueing OVERRUN msg on caller task:\n\n'
+                    'Queueing OVERRUN msg on parent task:\n\n'
                     + report
                 )
                 log.debug(report)
@@ -1819,20 +1954,22 @@ class Context:
             # overrun state and that msg isn't stuck in an
             # overflow queue what happens?!?
 
-            local_uid = self._actor.uid
+            local_aid = self._actor.aid
             txt: str = (
                 'on IPC context:\n'
 
                 f'<= sender: {from_uid}\n'
                 f'  |_ {self._nsf}()\n\n'
 
-                f'=> overrun: {local_uid}\n'
+                f'=> overrun: {local_aid.reprol()!r}\n'
                 f'  |_cid: {cid}\n'
                 f'  |_task: {self._task}\n'
             )
             if not self._stream_opened:
                 txt += (
-                    f'\n*** No stream open on `{local_uid[0]}` side! ***\n\n'
+                    f'\n'
+                    f'*** No stream open on `{local_aid.name}` side! ***\n'
+                    f'\n'
                     f'{msg}\n'
                 )
 
@@ -1935,12 +2072,12 @@ async def open_context_from_portal(
     IPC protocol.
 
     The yielded `tuple` is a pair delivering a `tractor.Context`
-    and any first value "sent" by the "callee" task via a call
+    and any first value "sent" by the "child" task via a call
     to `Context.started(<value: Any>)`; this side of the
-    context does not unblock until the "callee" task calls
+    context does not unblock until the "child" task calls
     `.started()` in similar style to `trio.Nursery.start()`.
-    When the "callee" (side that is "called"/started by a call
+    When the "child" (side that is "called"/started by a call
-    to *this* method) returns, the caller side (this) unblocks
+    to *this* method) returns, the parent side (this) unblocks
     and any final value delivered from the other end can be
     retrieved using the `Contex.wait_for_result()` api.
 
@@ -1953,7 +2090,7 @@ async def open_context_from_portal(
     __tracebackhide__: bool = hide_tb
 
     # denote this frame as a "runtime frame" for stack
-    # introspection where we report the caller code in logging
+    # introspection where we report the parent code in logging
     # and error message content.
     # NOTE: 2 bc of the wrapping `@acm`
     __runtimeframe__: int = 2  # noqa
@@ -1981,13 +2118,20 @@ async def open_context_from_portal(
     # XXX NOTE XXX: currenly we do NOT allow opening a contex
     # with "self" since the local feeder mem-chan processing
     # is not built for it.
-    if (uid := portal.channel.uid) == portal.actor.uid:
+    if (
+        (uid := portal.channel.aid.uid)
+        ==
+        portal.actor.aid.uid
+    ):
         raise RuntimeError(
             '** !! Invalid Operation !! **\n'
             'Can not open an IPC ctx with the local actor!\n'
             f'|_{portal.actor}\n'
         )
 
+    # ?TODO? could we move this to inside the `tn` block?
+    # -> would allow doing `ctx.parent_task = tn.parent_task` ?
+    # -> would allow a `if not ._scope: => raise RTE` ?
     ctx: Context = await portal.actor.start_remote_task(
         portal.channel,
         nsf=nsf,
@@ -2012,13 +2156,12 @@ async def open_context_from_portal(
     # placeholder for any exception raised in the runtime
     # or by user tasks which cause this context's closure.
     scope_err: BaseException|None = None
-    ctxc_from_callee: ContextCancelled|None = None
+    ctxc_from_child: ContextCancelled|None = None
     try:
+        # from .devx import pause
         async with (
-            trio.open_nursery(
+            collapse_eg(),
-                strict_exception_groups=False,
+            trio.open_nursery() as tn,
-            ) as tn,
-
             msgops.maybe_limit_plds(
                 ctx=ctx,
                 spec=ctx_meta.get('pld_spec'),
@@ -2038,6 +2181,10 @@ async def open_context_from_portal(
             # the dialog, the `Error` msg should be raised from the `msg`
             # handling block below.
             try:
+                log.runtime(
+                    f'IPC ctx parent waiting on Started msg..\n'
+                    f'ctx.cid: {ctx.cid!r}\n'
+                )
                 started_msg, first = await ctx._pld_rx.recv_msg(
                     ipc=ctx,
                     expect_msg=Started,
@@ -2046,16 +2193,16 @@ async def open_context_from_portal(
                 )
             except trio.Cancelled as taskc:
                 ctx_cs: trio.CancelScope = ctx._scope
+                log.cancel(
+                    f'IPC ctx was cancelled during "child" task sync due to\n\n'
+                    f'.cid: {ctx.cid!r}\n'
+                    f'.maybe_error: {ctx.maybe_error!r}\n'
+                )
+                # await pause(shield=True)
+
                 if not ctx_cs.cancel_called:
                     raise
 
-                # from .devx import pause
-                # await pause(shield=True)
-
-                log.cancel(
-                    'IPC ctx was cancelled during "child" task sync due to\n\n'
-                    f'{ctx.maybe_error}\n'
-                )
                 # OW if the ctx's scope was cancelled manually,
                 # likely the `Context` was cancelled via a call to
                 # `._maybe_cancel_and_set_remote_error()` so ensure
@@ -2093,7 +2240,7 @@ async def open_context_from_portal(
             # that we can re-use it around the `yield` ^ here
             # or vice versa?
             #
-            # maybe TODO NOTE: between the caller exiting and
+            # maybe TODO NOTE: between the parent exiting and
             # arriving here the far end may have sent a ctxc-msg or
             # other error, so the quetion is whether we should check
             # for it here immediately and maybe raise so as to engage
@@ -2159,16 +2306,16 @@ async def open_context_from_portal(
     #   request in which case we DO let the error bubble to the
     #   opener.
     #
-    # 2-THIS "caller" task somewhere invoked `Context.cancel()`
+    # 2-THIS "parent" task somewhere invoked `Context.cancel()`
-    #   and received a `ContextCanclled` from the "callee"
+    #   and received a `ContextCanclled` from the "child"
     #   task, in which case we mask the `ContextCancelled` from
-    #   bubbling to this "caller" (much like how `trio.Nursery`
+    #   bubbling to this "parent" (much like how `trio.Nursery`
     #   swallows any `trio.Cancelled` bubbled by a call to
     #   `Nursery.cancel_scope.cancel()`)
     except ContextCancelled as ctxc:
         scope_err = ctxc
         ctx._local_error: BaseException = scope_err
-        ctxc_from_callee = ctxc
+        ctxc_from_child = ctxc
 
         # XXX TODO XXX: FIX THIS debug_mode BUGGGG!!!
         # using this code and then resuming the REPL will
@@ -2179,7 +2326,7 @@ async def open_context_from_portal(
         #   debugging the tractor-runtime itself using it's
         #   own `.devx.` tooling!
         #
-        # await _debug.pause()
+        # await debug.pause()
 
         # CASE 2: context was cancelled by local task calling
         # `.cancel()`, we don't raise and the exit block should
@@ -2189,7 +2336,7 @@ async def open_context_from_portal(
             and
             ctxc is ctx._remote_error
             and
-            ctxc.canceller == portal.actor.uid
+            ctxc.canceller == portal.actor.aid.uid
         ):
             log.cancel(
                 f'Context (cid=[{ctx.cid[-6:]}..] cancelled gracefully with:\n'
@@ -2205,11 +2352,11 @@ async def open_context_from_portal(
     # the above `._scope` can be cancelled due to:
     # 1. an explicit self cancel via `Context.cancel()` or
     #    `Actor.cancel()`,
-    # 2. any "callee"-side remote error, possibly also a cancellation
+    # 2. any "child"-side remote error, possibly also a cancellation
     #    request by some peer,
-    # 3. any "caller" (aka THIS scope's) local error raised in the above `yield`
+    # 3. any "parent" (aka THIS scope's) local error raised in the above `yield`
     except (
-        # CASE 3: standard local error in this caller/yieldee
+        # CASE 3: standard local error in this parent/yieldee
         Exception,
 
         # CASES 1 & 2: can manifest as a `ctx._scope_nursery`
@@ -2223,9 +2370,9 @@ async def open_context_from_portal(
         #   any `Context._maybe_raise_remote_err()` call.
         #
         # 2.-`BaseExceptionGroup[ContextCancelled | RemoteActorError]`
-        #    from any error delivered from the "callee" side
+        #    from any error delivered from the "child" side
         #    AND a group-exc is only raised if there was > 1
-        #    tasks started *here* in the "caller" / opener
+        #    tasks started *here* in the "parent" / opener
         #    block. If any one of those tasks calls
         #    `.wait_for_result()` or `MsgStream.receive()`
         #    `._maybe_raise_remote_err()` will be transitively
@@ -2238,34 +2385,39 @@ async def open_context_from_portal(
         trio.Cancelled,  # NOTE: NOT from inside the ctx._scope
         KeyboardInterrupt,
 
-    ) as caller_err:
+    ) as rent_err:
-        scope_err = caller_err
+        scope_err = rent_err
         ctx._local_error: BaseException = scope_err
 
         # XXX: ALWAYS request the context to CANCEL ON any ERROR.
         # NOTE: `Context.cancel()` is conversely NEVER CALLED in
         # the `ContextCancelled` "self cancellation absorbed" case
         # handled in the block above ^^^ !!
-        # await _debug.pause()
+        # await debug.pause()
         # log.cancel(
         match scope_err:
-            case trio.Cancelled:
+            case trio.Cancelled():
                 logmeth = log.cancel
+                cause: str = 'cancelled'
+                msg: str = (
+                    f'ctx {ctx.side!r}-side {cause!r} with,\n'
+                    f'{ctx.repr_outcome()!r}\n'
+                )
 
             # XXX explicitly report on any non-graceful-taskc cases
             case _:
+                cause: str = 'errored'
                 logmeth = log.exception
+                msg: str = f'ctx {ctx.side!r}-side {cause!r} with,\n'
 
-        logmeth(
+        logmeth(msg)
-            f'ctx {ctx.side!r}-side exited with {ctx.repr_outcome()}\n'
-        )
 
         if debug_mode():
-            # async with _debug.acquire_debug_lock(portal.actor.uid):
+            # async with debug.acquire_debug_lock(portal.actor.aid.uid):
             #     pass
             # TODO: factor ^ into below for non-root cases?
             #
-            from .devx._debug import maybe_wait_for_debugger
+            from .devx.debug import maybe_wait_for_debugger
             was_acquired: bool = await maybe_wait_for_debugger(
                 # header_msg=(
                 #     'Delaying `ctx.cancel()` until debug lock '
@@ -2278,15 +2430,13 @@ async def open_context_from_portal(
                     'Calling `ctx.cancel()`!\n'
                 )
 
-        # we don't need to cancel the callee if it already
+        # we don't need to cancel the child if it already
         # told us it's cancelled ;p
-        if ctxc_from_callee is None:
+        if ctxc_from_child is None:
             try:
+                # await pause(shield=True)
                 await ctx.cancel()
-            except (
+            except TransportClosed:
-                trio.BrokenResourceError,
-                trio.ClosedResourceError,
-            ):
                 log.warning(
                     'IPC connection for context is broken?\n'
                     f'task: {ctx.cid}\n'
@@ -2311,8 +2461,8 @@ async def open_context_from_portal(
             # via a call to
             # `Context._maybe_cancel_and_set_remote_error()`.
             # As per `Context._deliver_msg()`, that error IS
-            # ALWAYS SET any time "callee" side fails and causes "caller
+            # ALWAYS SET any time "child" side fails and causes
-            # side" cancellation via a `ContextCancelled` here.
+            # "parent side" cancellation via a `ContextCancelled` here.
             try:
                 result_or_err: Exception|Any = await ctx.wait_for_result()
             except BaseException as berr:
@@ -2328,8 +2478,8 @@ async def open_context_from_portal(
                 raise
 
             # yes this worx!
-            # from .devx import _debug
+            # from .devx import debug
-            # await _debug.pause()
+            # await debug.pause()
 
             # an exception type boxed in a `RemoteActorError`
             # is returned (meaning it was obvi not raised)
@@ -2348,7 +2498,7 @@ async def open_context_from_portal(
                     )
                 case (None, _):
                     log.runtime(
-                        'Context returned final result from callee task:\n'
+                        'Context returned final result from child task:\n'
                         f'<= peer: {uid}\n'
                         f'  |_ {nsf}()\n\n'
 
@@ -2364,7 +2514,7 @@ async def open_context_from_portal(
         # where the root is waiting on the lock to clear but the
         # child has already cleared it and clobbered IPC.
         if debug_mode():
-            from .devx._debug import maybe_wait_for_debugger
+            from .devx.debug import maybe_wait_for_debugger
             await maybe_wait_for_debugger()
 
         # though it should be impossible for any tasks
@@ -2438,12 +2588,14 @@ async def open_context_from_portal(
                 log.cancel(
                     f'Context cancelled by local {ctx.side!r}-side task\n'
                     f'c)>\n'
-                    f' |_{ctx._task}\n\n'
+                    f'  |_{ctx.parent_task}\n'
-                    f'{repr(scope_err)}\n'
+                    f'   .cid={ctx.cid!r}\n'
+                    f'\n'
+                    f'{scope_err!r}\n'
                 )
 
             # TODO: should we add a `._cancel_req_received`
-            # flag to determine if the callee manually called
+            # flag to determine if the child manually called
             # `ctx.cancel()`?
             # -[ ] going to need a cid check no?
 
@@ -2499,7 +2651,7 @@ def mk_context(
     recv_chan: trio.MemoryReceiveChannel
     send_chan, recv_chan = trio.open_memory_channel(msg_buffer_size)
 
-    # TODO: only scan caller-info if log level so high!
+    # TODO: only scan parent-info if log level so high!
     from .devx._frame_stack import find_caller_info
     caller_info: CallerInfo|None = find_caller_info()
 

tractor/_entry.py

@@ -1,286 +0,0 @@
-# tractor: structured concurrent "actors".
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Sub-process entry points.
-
-"""
-from __future__ import annotations
-from functools import partial
-import multiprocessing as mp
-import os
-import textwrap
-from typing import (
-    Any,
-    TYPE_CHECKING,
-)
-
-import trio  # type: ignore
-
-from .log import (
-    get_console_log,
-    get_logger,
-)
-from . import _state
-from .devx import _debug
-from .to_asyncio import run_as_asyncio_guest
-from ._runtime import (
-    async_main,
-    Actor,
-)
-
-if TYPE_CHECKING:
-    from ._spawn import SpawnMethodKey
-
-
-log = get_logger(__name__)
-
-
-def _mp_main(
-
-    actor: Actor,
-    accept_addrs: list[tuple[str, int]],
-    forkserver_info: tuple[Any, Any, Any, Any, Any],
-    start_method: SpawnMethodKey,
-    parent_addr: tuple[str, int] | None = None,
-    infect_asyncio: bool = False,
-
-) -> None:
-    '''
-    The routine called *after fork* which invokes a fresh `trio.run()`
-
-    '''
-    actor._forkserver_info = forkserver_info
-    from ._spawn import try_set_start_method
-    spawn_ctx: mp.context.BaseContext = try_set_start_method(start_method)
-    assert spawn_ctx
-
-    if actor.loglevel is not None:
-        log.info(
-            f'Setting loglevel for {actor.uid} to {actor.loglevel}'
-        )
-        get_console_log(actor.loglevel)
-
-    # TODO: use scops headers like for `trio` below!
-    # (well after we libify it maybe..)
-    log.info(
-        f'Started new {spawn_ctx.current_process()} for {actor.uid}'
-    #     f"parent_addr is {parent_addr}"
-    )
-    _state._current_actor: Actor = actor
-    trio_main = partial(
-        async_main,
-        actor=actor,
-        accept_addrs=accept_addrs,
-        parent_addr=parent_addr
-    )
-    try:
-        if infect_asyncio:
-            actor._infected_aio = True
-            run_as_asyncio_guest(trio_main)
-        else:
-            trio.run(trio_main)
-    except KeyboardInterrupt:
-        pass  # handle it the same way trio does?
-
-    finally:
-        log.info(
-            f'`mp`-subactor {actor.uid} exited'
-        )
-
-
-# TODO: move this func to some kinda `.devx._conc_lang.py` eventually
-# as we work out our multi-domain state-flow-syntax!
-def nest_from_op(
-    input_op: str,
-    #
-    # ?TODO? an idea for a syntax to the state of concurrent systems
-    # as a "3-domain" (execution, scope, storage) model and using
-    # a minimal ascii/utf-8 operator-set.
-    #
-    # try not to take any of this seriously yet XD
-    #
-    # > is a "play operator" indicating (CPU bound)
-    #   exec/work/ops required at the "lowest level computing"
-    #
-    # execution primititves (tasks, threads, actors..) denote their
-    # lifetime with '(' and ')' since parentheses normally are used
-    # in many langs to denote function calls.
-    #
-    # starting = (
-    # >(  opening/starting; beginning of the thread-of-exec (toe?)
-    # (>  opened/started,  (finished spawning toe)
-    # |_<Task: blah blah..>  repr of toe, in py these look like <objs>
-    #
-    # >) closing/exiting/stopping,
-    # )> closed/exited/stopped,
-    # |_<Task: blah blah..>
-    #   [OR <), )< ?? ]
-    #
-    # ending = )
-    # >c) cancelling to close/exit
-    # c)> cancelled (caused close), OR?
-    #  |_<Actor: ..>
-    #   OR maybe "<c)" which better indicates the cancel being
-    #   "delivered/returned" / returned" to LHS?
-    #
-    # >x)  erroring to eventuall exit
-    # x)>  errored and terminated
-    #  |_<Actor: ...>
-    #
-    # scopes: supers/nurseries, IPC-ctxs, sessions, perms, etc.
-    # >{  opening
-    # {>  opened
-    # }>  closed
-    # >}  closing
-    #
-    # storage: like queues, shm-buffers, files, etc..
-    # >[  opening
-    # [>  opened
-    #  |_<FileObj: ..>
-    #
-    # >]  closing
-    # ]>  closed
-
-    # IPC ops: channels, transports, msging
-    # =>  req msg
-    # <=  resp msg
-    # <=> 2-way streaming (of msgs)
-    # <-  recv 1 msg
-    # ->  send 1 msg
-    #
-    # TODO: still not sure on R/L-HS approach..?
-    # =>(  send-req to exec start (task, actor, thread..)
-    # (<=  recv-req to ^
-    #
-    # (<=  recv-req ^
-    # <=(  recv-resp opened remote exec primitive
-    # <=)  recv-resp closed
-    #
-    # )<=c req to stop due to cancel
-    # c=>) req to stop due to cancel
-    #
-    # =>{  recv-req to open
-    # <={  send-status that it closed
-
-    tree_str: str,
-
-    # NOTE: so move back-from-the-left of the `input_op` by
-    # this amount.
-    back_from_op: int = 0,
-) -> str:
-    '''
-    Depth-increment the input (presumably hierarchy/supervision)
-    input "tree string" below the provided `input_op` execution
-    operator, so injecting a `"\n|_{input_op}\n"`and indenting the
-    `tree_str` to nest content aligned with the ops last char.
-
-    '''
-    return (
-        f'{input_op}\n'
-        +
-        textwrap.indent(
-            tree_str,
-            prefix=(
-                len(input_op)
-                -
-                (back_from_op + 1)
-            ) * ' ',
-        )
-    )
-
-
-def _trio_main(
-    actor: Actor,
-    *,
-    parent_addr: tuple[str, int] | None = None,
-    infect_asyncio: bool = False,
-
-) -> None:
-    '''
-    Entry point for a `trio_run_in_process` subactor.
-
-    '''
-    _debug.hide_runtime_frames()
-
-    _state._current_actor = actor
-    trio_main = partial(
-        async_main,
-        actor,
-        parent_addr=parent_addr
-    )
-
-    if actor.loglevel is not None:
-        get_console_log(actor.loglevel)
-        actor_info: str = (
-            f'|_{actor}\n'
-            f'  uid: {actor.uid}\n'
-            f'  pid: {os.getpid()}\n'
-            f'  parent_addr: {parent_addr}\n'
-            f'  loglevel: {actor.loglevel}\n'
-        )
-        log.info(
-            'Starting new `trio` subactor:\n'
-            +
-            nest_from_op(
-                input_op='>(',  # see syntax ideas above
-                tree_str=actor_info,
-                back_from_op=2,  # since "complete"
-            )
-        )
-    logmeth = log.info
-    exit_status: str = (
-        'Subactor exited\n'
-        +
-        nest_from_op(
-            input_op=')>',  # like a "closed-to-play"-icon from super perspective
-            tree_str=actor_info,
-            back_from_op=1,
-        )
-    )
-    try:
-        if infect_asyncio:
-            actor._infected_aio = True
-            run_as_asyncio_guest(trio_main)
-        else:
-            trio.run(trio_main)
-
-    except KeyboardInterrupt:
-        logmeth = log.cancel
-        exit_status: str = (
-            'Actor received KBI (aka an OS-cancel)\n'
-            +
-            nest_from_op(
-                input_op='c)>',  # closed due to cancel (see above)
-                tree_str=actor_info,
-            )
-        )
-    except BaseException as err:
-        logmeth = log.error
-        exit_status: str = (
-            'Main actor task exited due to crash?\n'
-            +
-            nest_from_op(
-                input_op='x)>',  # closed by error
-                tree_str=actor_info,
-            )
-        )
-        # NOTE since we raise a tb will already be shown on the
-        # console, thus we do NOT use `.exception()` above.
-        raise err
-
-    finally:
-        logmeth(exit_status)

tractor/_exceptions.py

@@ -23,7 +23,6 @@ import builtins
 import importlib
 from pprint import pformat
 from pdb import bdb
-import sys
 from types import (
     TracebackType,
 )
@@ -44,7 +43,7 @@ from msgspec import (
     ValidationError,
 )
 
-from tractor._state import current_actor
+from tractor.runtime._state import current_actor
 from tractor.log import get_logger
 from tractor.msg import (
     Error,
@@ -65,15 +64,29 @@ if TYPE_CHECKING:
     from ._context import Context
     from .log import StackLevelAdapter
     from ._stream import MsgStream
-    from ._ipc import Channel
+    from .ipc import Channel
 
 log = get_logger('tractor')
 
 _this_mod = importlib.import_module(__name__)
 
 
-class ActorFailure(Exception):
+class RuntimeFailure(RuntimeError):
-    "General actor failure"
+    '''
+    General `Actor`-runtime failure due to,
+
+    - a bad runtime-env,
+    - falied spawning (bad input to process),
+    -   API usage.
+
+    '''
+
+
+class ActorFailure(RuntimeFailure):
+    '''
+    `Actor` failed to boot before/after spawn
+
+    '''
 
 
 class InternalError(RuntimeError):
@@ -126,6 +139,12 @@ class TrioTaskExited(Exception):
     '''
 
 
+class DebugRequestError(RuntimeError):
+    '''
+    Failed to request stdio lock from root actor!
+
+    '''
+
 # NOTE: more or less should be close to these:
 # 'boxed_type',
 # 'src_type',
@@ -168,7 +187,31 @@ _body_fields: list[str] = list(
 )
 
 
-def get_err_type(type_name: str) -> BaseException|None:
+def reg_err_types(
+    exc_types: list[Type[Exception]],
+) -> None:
+    '''
+    Register custom exception types for local lookup.
+
+    Such that error types can be registered by an external
+    `tractor`-use-app code base which are expected to be raised
+    remotely; enables them being re-raised on the receiver side of
+    some inter-actor IPC dialog.
+
+    '''
+    for exc_type in exc_types:
+        log.debug(
+            f'Register custom exception,\n'
+            f'{exc_type!r}\n'
+        )
+        setattr(
+            _this_mod,
+            exc_type.__name__,
+            exc_type,
+        )
+
+
+def get_err_type(type_name: str) -> Type[BaseException]|None:
     '''
     Look up an exception type by name from the set of locally known
     namespaces:
@@ -191,6 +234,8 @@ def get_err_type(type_name: str) -> BaseException|None:
         ):
             return type_ref
 
+    return None
+
 
 def pack_from_raise(
     local_err: (
@@ -280,7 +325,8 @@ class RemoteActorError(Exception):
         # also pertains to our long long oustanding issue XD
         # https://github.com/goodboy/tractor/issues/5
         self._boxed_type: BaseException = boxed_type
-        self._src_type: BaseException|None = None
+        self._src_type: Type[BaseException]|None = None
+        self._src_type_resolved: bool = False
         self._ipc_msg: Error|None = ipc_msg
         self._extra_msgdata = extra_msgdata
 
@@ -389,24 +435,41 @@ class RemoteActorError(Exception):
         return self._ipc_msg.src_type_str
 
     @property
-    def src_type(self) -> str:
+    def src_type(self) -> Type[BaseException]|None:
         '''
-        Error type raised by original remote faulting actor.
+        Error type raised by original remote faulting
+        actor.
 
-        When the error has only been relayed a single actor-hop
+        When the error has only been relayed a single
-        this will be the same as the `.boxed_type`.
+        actor-hop this will be the same as
+        `.boxed_type`.
+
+        If the type can not be resolved locally (i.e.
+        it was not registered via `reg_err_types()`)
+        a warning is logged and `None` is returned;
+        all string-level error info (`.src_type_str`,
+        `.tb_str`, etc.) remains available.
 
         '''
-        if self._src_type is None:
+        if not self._src_type_resolved:
+            self._src_type_resolved = True
+
+            if self._ipc_msg is None:
+                return None
+
             self._src_type = get_err_type(
                 self._ipc_msg.src_type_str
             )
-
             if not self._src_type:
-                raise TypeError(
+                log.warning(
-                    f'Failed to lookup src error type with '
+                    f'Failed to lookup src error type via\n'
-                    f'`tractor._exceptions.get_err_type()` :\n'
+                    f'`tractor._exceptions.get_err_type()`:\n'
-                    f'{self.src_type_str}'
+                    f'\n'
+                    f'`{self._ipc_msg.src_type_str}`'
+                    f' is not registered!\n'
+                    f'\n'
+                    f'Call `reg_err_types()` to enable'
+                    f' full type reconstruction.\n'
                 )
 
         return self._src_type
@@ -414,20 +477,30 @@ class RemoteActorError(Exception):
     @property
     def boxed_type_str(self) -> str:
         '''
-        String-name of the (last hop's) boxed error type.
+        String-name of the (last hop's) boxed error
+        type.
+
+        Falls back to the IPC-msg-encoded type-name
+        str when the type can not be resolved locally
+        (e.g. unregistered custom errors).
 
         '''
         # TODO, maybe support also serializing the
-        # `ExceptionGroup.exeptions: list[BaseException]` set under
+        # `ExceptionGroup.exceptions: list[BaseException]`
-        # certain conditions?
+        # set under certain conditions?
         bt: Type[BaseException] = self.boxed_type
         if bt:
             return str(bt.__name__)
 
-        return ''
+        # fallback to the str name from the IPC msg
+        # when the type obj can't be resolved.
+        if self._ipc_msg:
+            return self._ipc_msg.boxed_type_str
+
+        return '<unknown>'
 
     @property
-    def boxed_type(self) -> Type[BaseException]:
+    def boxed_type(self) -> Type[BaseException]|None:
         '''
         Error type boxed by last actor IPC hop.
 
@@ -521,7 +594,6 @@ class RemoteActorError(Exception):
             if val:
                 _repr += f'{key}={val_str}{end_char}'
 
-
         return _repr
 
     def reprol(self) -> str:
@@ -600,56 +672,9 @@ class RemoteActorError(Exception):
             the type name is already implicitly shown by python).
 
         '''
-        header: str = ''
-        body: str = ''
-        message: str = ''
-
-        # XXX when the currently raised exception is this instance,
-        # we do not ever use the "type header" style repr.
-        is_being_raised: bool = False
-        if (
-            (exc := sys.exception())
-            and
-            exc is self
-        ):
-            is_being_raised: bool = True
-
-        with_type_header: bool = (
-            with_type_header
-            and
-            not is_being_raised
-        )
-
-        # <RemoteActorError( .. )> style
-        if with_type_header:
-            header: str = f'<{type(self).__name__}('
-
-        if message := self._message:
-
-            # split off the first line so, if needed, it isn't
-            # indented the same like the "boxed content" which
-            # since there is no `.tb_str` is just the `.message`.
-            lines: list[str] = message.splitlines()
-            first: str = lines[0]
-            message: str = message.removeprefix(first)
-
-            # with a type-style header we,
-            # - have no special message "first line" extraction/handling
-            # - place the message a space in from the header:
-            #  `MsgTypeError( <message> ..`
-            #                 ^-here
-            # - indent the `.message` inside the type body.
-            if with_type_header:
-                first = f' {first} )>'
-
-            message: str = textwrap.indent(
-                message,
-                prefix=' '*2,
-            )
-            message: str = first + message
-
         # IFF there is an embedded traceback-str we always
         # draw the ascii-box around it.
+        body: str = ''
         if tb_str := self.tb_str:
             fields: str = self._mk_fields_str(
                 _body_fields
@@ -670,21 +695,15 @@ class RemoteActorError(Exception):
                 boxer_header=self.relay_uid,
             )
 
-        tail = ''
+        # !TODO, it'd be nice to import these top level without
-        if (
+        # cycles!
-            with_type_header
+        from tractor.devx.pformat import (
-            and not message
+            pformat_exc,
-        ):
+        )
-            tail: str = '>'
+        return pformat_exc(
-
+            exc=self,
-        return (
+            with_type_header=with_type_header,
-            header
+            body=body,
-            +
-            message
-            +
-            f'{body}'
-            +
-            tail
         )
 
     __repr__ = pformat
@@ -710,10 +729,22 @@ class RemoteActorError(Exception):
         failing actor's remote env.
 
         '''
-        # TODO: better tb insertion and all the fancier dunder
+        # TODO: better tb insertion and all the fancier
-        # metadata stuff as per `.__context__` etc. and friends:
+        # dunder metadata stuff as per `.__context__`
+        # etc. and friends:
         # https://github.com/python-trio/trio/issues/611
-        src_type_ref: Type[BaseException] = self.src_type
+        src_type_ref: Type[BaseException]|None = (
+            self.src_type
+        )
+        if src_type_ref is None:
+            # unresolvable type: fall back to
+            # a `RuntimeError` preserving original
+            # traceback + type name.
+            return RuntimeError(
+                f'{self.src_type_str}: '
+                f'{self.tb_str}'
+            )
+
         return src_type_ref(self.tb_str)
 
     # TODO: local recontruction of nested inception for a given
@@ -962,7 +993,7 @@ class StreamOverrun(
     '''
 
 
-class TransportClosed(trio.BrokenResourceError):
+class TransportClosed(Exception):
     '''
     IPC transport (protocol) connection was closed or broke and
     indicates that the wrapping communication `Channel` can no longer
@@ -973,24 +1004,39 @@ class TransportClosed(trio.BrokenResourceError):
         self,
         message: str,
         loglevel: str = 'transport',
-        cause: BaseException|None = None,
+        src_exc: Exception|None = None,
         raise_on_report: bool = False,
 
     ) -> None:
         self.message: str = message
-        self._loglevel = loglevel
+        self._loglevel: str = loglevel
         super().__init__(message)
 
-        if cause is not None:
+        self._src_exc = src_exc
-            self.__cause__ = cause
+        # set the cause manually if not already set by python
+        if (
+            src_exc is not None
+            and
+            not self.__cause__
+        ):
+            self.__cause__ = src_exc
 
         # flag to toggle whether the msg loop should raise
         # the exc in its `TransportClosed` handler block.
         self._raise_on_report = raise_on_report
 
+    @property
+    def src_exc(self) -> Exception:
+        return (
+            self.__cause__
+            or
+            self._src_exc
+        )
+
     def report_n_maybe_raise(
         self,
         message: str|None = None,
+        hide_tb: bool = True,
 
     ) -> None:
         '''
@@ -998,24 +1044,99 @@ class TransportClosed(trio.BrokenResourceError):
         for this error.
 
         '''
+        __tracebackhide__: bool = hide_tb
         message: str = message or self.message
+
         # when a cause is set, slap it onto the log emission.
-        if cause := self.__cause__:
+        if cause := self.src_exc:
             cause_tb_str: str = ''.join(
                 traceback.format_tb(cause.__traceback__)
             )
             message += (
                 f'{cause_tb_str}\n'  # tb
-                f'    {cause}\n'  # exc repr
+                f'{cause!r}\n'  # exc repr
             )
 
-        getattr(log, self._loglevel)(message)
+        getattr(
+            log,
+            self._loglevel
+        )(message)
 
         # some errors we want to blow up from
         # inside the RPC msg loop
         if self._raise_on_report:
             raise self from cause
 
+    @classmethod
+    def repr_src_exc(
+        self,
+        src_exc: Exception|None = None,
+    ) -> str:
+
+        if src_exc is None:
+            return '<unknown>'
+
+        src_msg: tuple[str] = src_exc.args
+        src_exc_repr: str = (
+            f'{type(src_exc).__name__}[ {src_msg} ]'
+        )
+        return src_exc_repr
+
+    def pformat(self) -> str:
+        from tractor.devx.pformat import (
+            pformat_exc,
+        )
+        return pformat_exc(
+            exc=self,
+        )
+
+    # delegate to `str`-ified pformat
+    __repr__ = pformat
+
+    @classmethod
+    def from_src_exc(
+        cls,
+        src_exc: (
+            Exception|
+            trio.ClosedResource|
+            trio.BrokenResourceError
+        ),
+        message: str,
+        body: str = '',
+        **init_kws,
+    ) -> TransportClosed:
+        '''
+        Convenience constructor for creation from an underlying
+        `trio`-sourced async-resource/chan/stream error.
+
+        Embeds the original `src_exc`'s repr within the
+        `Exception.args` via a first-line-in-`.message`-put-in-header
+        pre-processing and allows inserting additional content beyond
+        the main message via a `body: str`.
+
+        '''
+        repr_src_exc: str = cls.repr_src_exc(
+            src_exc,
+        )
+        next_line: str = f'  src_exc: {repr_src_exc}\n'
+        if body:
+            body: str = textwrap.indent(
+                body,
+                prefix=' '*2,
+            )
+
+        return TransportClosed(
+            message=(
+                message
+                +
+                next_line
+                +
+                body
+            ),
+            src_exc=src_exc,
+            **init_kws,
+        )
+
 
 class NoResult(RuntimeError):
     "No final result is expected for this actor"
@@ -1152,14 +1273,31 @@ def unpack_error(
     if not isinstance(msg, Error):
         return None
 
-    # try to lookup a suitable error type from the local runtime
+    # try to lookup a suitable error type from the
-    # env then use it to construct a local instance.
+    # local runtime env then use it to construct a
-    # boxed_type_str: str = error_dict['boxed_type_str']
+    # local instance.
     boxed_type_str: str = msg.boxed_type_str
-    boxed_type: Type[BaseException] = get_err_type(boxed_type_str)
+    boxed_type: Type[BaseException]|None = get_err_type(
+        boxed_type_str
+    )
 
-    # retrieve the error's msg-encoded remotoe-env info
+    if boxed_type is None:
-    message: str = f'remote task raised a {msg.boxed_type_str!r}\n'
+        log.warning(
+            f'Failed to resolve remote error type\n'
+            f'`{boxed_type_str}` - boxing as\n'
+            f'`RemoteActorError` with original\n'
+            f'traceback preserved.\n'
+            f'\n'
+            f'Call `reg_err_types()` to enable\n'
+            f'full type reconstruction.\n'
+        )
+
+    # retrieve the error's msg-encoded remote-env
+    # info
+    message: str = (
+        f'remote task raised a '
+        f'{msg.boxed_type_str!r}\n'
+    )
 
     # TODO: do we even really need these checks for RAEs?
     if boxed_type_str in [
@@ -1190,55 +1328,6 @@ def unpack_error(
     return exc
 
 
-def is_multi_cancelled(
-    exc: BaseException|BaseExceptionGroup,
-
-    ignore_nested: set[BaseException] = set(),
-
-) -> bool|BaseExceptionGroup:
-    '''
-    Predicate to determine if an `BaseExceptionGroup` only contains
-    some (maybe nested) set of sub-grouped exceptions (like only
-    `trio.Cancelled`s which get swallowed silently by default) and is
-    thus the result of "gracefully cancelling" a collection of
-    sub-tasks (or other conc primitives) and receiving a "cancelled
-    ACK" from each after termination.
-
-    Docs:
-    ----
-    - https://docs.python.org/3/library/exceptions.html#exception-groups
-    - https://docs.python.org/3/library/exceptions.html#BaseExceptionGroup.subgroup
-
-    '''
-
-    if (
-        not ignore_nested
-        or
-        trio.Cancelled in ignore_nested
-        # XXX always count-in `trio`'s native signal
-    ):
-        ignore_nested.update({trio.Cancelled})
-
-    if isinstance(exc, BaseExceptionGroup):
-        matched_exc: BaseExceptionGroup|None = exc.subgroup(
-            tuple(ignore_nested),
-
-            # TODO, complain about why not allowed XD
-            # condition=tuple(ignore_nested),
-        )
-        if matched_exc is not None:
-            return matched_exc
-
-    # NOTE, IFF no excs types match (throughout the error-tree)
-    # -> return `False`, OW return the matched sub-eg.
-    #
-    # IOW, for the inverse of ^ for the purpose of
-    # maybe-enter-REPL--logic: "only debug when the err-tree contains
-    # at least one exc-type NOT in `ignore_nested`" ; i.e. the case where
-    # we fallthrough and return `False` here.
-    return False
-
-
 def _raise_from_unexpected_msg(
     ctx: Context,
     msg: MsgType,

tractor/_ipc.py

@@ -1,820 +0,0 @@
-# tractor: structured concurrent "actors".
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Inter-process comms abstractions
-
-"""
-from __future__ import annotations
-from collections.abc import (
-    AsyncGenerator,
-    AsyncIterator,
-)
-from contextlib import (
-    asynccontextmanager as acm,
-    contextmanager as cm,
-)
-import platform
-from pprint import pformat
-import struct
-import typing
-from typing import (
-    Any,
-    Callable,
-    runtime_checkable,
-    Protocol,
-    Type,
-    TypeVar,
-)
-
-import msgspec
-from tricycle import BufferedReceiveStream
-import trio
-
-from tractor.log import get_logger
-from tractor._exceptions import (
-    MsgTypeError,
-    pack_from_raise,
-    TransportClosed,
-    _mk_send_mte,
-    _mk_recv_mte,
-)
-from tractor.msg import (
-    _ctxvar_MsgCodec,
-    # _codec,  XXX see `self._codec` sanity/debug checks
-    MsgCodec,
-    types as msgtypes,
-    pretty_struct,
-)
-
-log = get_logger(__name__)
-
-_is_windows = platform.system() == 'Windows'
-
-
-def get_stream_addrs(
-    stream: trio.SocketStream
-) -> tuple[
-    tuple[str, int],  # local
-    tuple[str, int],  # remote
-]:
-    '''
-    Return the `trio` streaming transport prot's socket-addrs for
-    both the local and remote sides as a pair.
-
-    '''
-    # rn, should both be IP sockets
-    lsockname = stream.socket.getsockname()
-    rsockname = stream.socket.getpeername()
-    return (
-        tuple(lsockname[:2]),
-        tuple(rsockname[:2]),
-    )
-
-
-# from tractor.msg.types import MsgType
-# ?TODO? this should be our `Union[*msgtypes.__spec__]` alias now right..?
-# => BLEH, except can't bc prots must inherit typevar or param-spec
-#   vars..
-MsgType = TypeVar('MsgType')
-
-
-# TODO: break up this mod into a subpkg so we can start adding new
-# backends and move this type stuff into a dedicated file.. Bo
-#
-@runtime_checkable
-class MsgTransport(Protocol[MsgType]):
-#
-# ^-TODO-^ consider using a generic def and indexing with our
-# eventual msg definition/types?
-# - https://docs.python.org/3/library/typing.html#typing.Protocol
-
-    stream: trio.SocketStream
-    drained: list[MsgType]
-
-    def __init__(self, stream: trio.SocketStream) -> None:
-        ...
-
-    # XXX: should this instead be called `.sendall()`?
-    async def send(self, msg: MsgType) -> None:
-        ...
-
-    async def recv(self) -> MsgType:
-        ...
-
-    def __aiter__(self) -> MsgType:
-        ...
-
-    def connected(self) -> bool:
-        ...
-
-    # defining this sync otherwise it causes a mypy error because it
-    # can't figure out it's a generator i guess?..?
-    def drain(self) -> AsyncIterator[dict]:
-        ...
-
-    @property
-    def laddr(self) -> tuple[str, int]:
-        ...
-
-    @property
-    def raddr(self) -> tuple[str, int]:
-        ...
-
-
-# TODO: typing oddity.. not sure why we have to inherit here, but it
-# seems to be an issue with `get_msg_transport()` returning
-# a `Type[Protocol]`; probably should make a `mypy` issue?
-class MsgpackTCPStream(MsgTransport):
-    '''
-    A ``trio.SocketStream`` delivering ``msgpack`` formatted data
-    using the ``msgspec`` codec lib.
-
-    '''
-    layer_key: int = 4
-    name_key: str = 'tcp'
-
-    # TODO: better naming for this?
-    # -[ ] check how libp2p does naming for such things?
-    codec_key: str = 'msgpack'
-
-    def __init__(
-        self,
-        stream: trio.SocketStream,
-        prefix_size: int = 4,
-
-        # XXX optionally provided codec pair for `msgspec`:
-        # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
-        #
-        # TODO: define this as a `Codec` struct which can be
-        # overriden dynamically by the application/runtime?
-        codec: tuple[
-            Callable[[Any], Any]|None,  # coder
-            Callable[[type, Any], Any]|None,  # decoder
-        ]|None = None,
-
-    ) -> None:
-
-        self.stream = stream
-        assert self.stream.socket
-
-        # should both be IP sockets
-        self._laddr, self._raddr = get_stream_addrs(stream)
-
-        # create read loop instance
-        self._aiter_pkts = self._iter_packets()
-        self._send_lock = trio.StrictFIFOLock()
-
-        # public i guess?
-        self.drained: list[dict] = []
-
-        self.recv_stream = BufferedReceiveStream(
-            transport_stream=stream
-        )
-        self.prefix_size = prefix_size
-
-        # allow for custom IPC msg interchange format
-        # dynamic override Bo
-        self._task = trio.lowlevel.current_task()
-
-        # XXX for ctxvar debug only!
-        # self._codec: MsgCodec = (
-        #     codec
-        #     or
-        #     _codec._ctxvar_MsgCodec.get()
-        # )
-
-    async def _iter_packets(self) -> AsyncGenerator[dict, None]:
-        '''
-        Yield `bytes`-blob decoded packets from the underlying TCP
-        stream using the current task's `MsgCodec`.
-
-        This is a streaming routine implemented as an async generator
-        func (which was the original design, but could be changed?)
-        and is allocated by a `.__call__()` inside `.__init__()` where
-        it is assigned to the `._aiter_pkts` attr.
-
-        '''
-        decodes_failed: int = 0
-
-        while True:
-            try:
-                header: bytes = await self.recv_stream.receive_exactly(4)
-            except (
-                ValueError,
-                ConnectionResetError,
-
-                # not sure entirely why we need this but without it we
-                # seem to be getting racy failures here on
-                # arbiter/registry name subs..
-                trio.BrokenResourceError,
-
-            ) as trans_err:
-
-                loglevel = 'transport'
-                match trans_err:
-                    # case (
-                    #     ConnectionResetError()
-                    # ):
-                    #     loglevel = 'transport'
-
-                    # peer actor (graceful??) TCP EOF but `tricycle`
-                    # seems to raise a 0-bytes-read?
-                    case ValueError() if (
-                        'unclean EOF' in trans_err.args[0]
-                    ):
-                        pass
-
-                    # peer actor (task) prolly shutdown quickly due
-                    # to cancellation
-                    case trio.BrokenResourceError() if (
-                        'Connection reset by peer' in trans_err.args[0]
-                    ):
-                        pass
-
-                    # unless the disconnect condition falls under "a
-                    # normal operation breakage" we usualy console warn
-                    # about it.
-                    case _:
-                        loglevel: str = 'warning'
-
-
-                raise TransportClosed(
-                    message=(
-                        f'IPC transport already closed by peer\n'
-                        f'x]> {type(trans_err)}\n'
-                        f'  |_{self}\n'
-                    ),
-                    loglevel=loglevel,
-                ) from trans_err
-
-            # XXX definitely can happen if transport is closed
-            # manually by another `trio.lowlevel.Task` in the
-            # same actor; we use this in some simulated fault
-            # testing for ex, but generally should never happen
-            # under normal operation!
-            #
-            # NOTE: as such we always re-raise this error from the
-            #       RPC msg loop!
-            except trio.ClosedResourceError as closure_err:
-                raise TransportClosed(
-                    message=(
-                        f'IPC transport already manually closed locally?\n'
-                        f'x]> {type(closure_err)} \n'
-                        f'  |_{self}\n'
-                    ),
-                    loglevel='error',
-                    raise_on_report=(
-                        closure_err.args[0] == 'another task closed this fd'
-                        or
-                        closure_err.args[0] in ['another task closed this fd']
-                    ),
-                ) from closure_err
-
-            # graceful TCP EOF disconnect
-            if header == b'':
-                raise TransportClosed(
-                    message=(
-                        f'IPC transport already gracefully closed\n'
-                        f']>\n'
-                        f' |_{self}\n'
-                    ),
-                    loglevel='transport',
-                    # cause=???  # handy or no?
-                )
-
-            size: int
-            size, = struct.unpack("<I", header)
-
-            log.transport(f'received header {size}')  # type: ignore
-            msg_bytes: bytes = await self.recv_stream.receive_exactly(size)
-
-            log.transport(f"received {msg_bytes}")  # type: ignore
-            try:
-                # NOTE: lookup the `trio.Task.context`'s var for
-                # the current `MsgCodec`.
-                codec: MsgCodec = _ctxvar_MsgCodec.get()
-
-                # XXX for ctxvar debug only!
-                # if self._codec.pld_spec != codec.pld_spec:
-                #     assert (
-                #         task := trio.lowlevel.current_task()
-                #     ) is not self._task
-                #     self._task = task
-                #     self._codec = codec
-                #     log.runtime(
-                #         f'Using new codec in {self}.recv()\n'
-                #         f'codec: {self._codec}\n\n'
-                #         f'msg_bytes: {msg_bytes}\n'
-                #     )
-                yield codec.decode(msg_bytes)
-
-            # XXX NOTE: since the below error derives from
-            # `DecodeError` we need to catch is specially
-            # and always raise such that spec violations
-            # are never allowed to be caught silently!
-            except msgspec.ValidationError as verr:
-                msgtyperr: MsgTypeError = _mk_recv_mte(
-                    msg=msg_bytes,
-                    codec=codec,
-                    src_validation_error=verr,
-                )
-                # XXX deliver up to `Channel.recv()` where
-                # a re-raise and `Error`-pack can inject the far
-                # end actor `.uid`.
-                yield msgtyperr
-
-            except (
-                msgspec.DecodeError,
-                UnicodeDecodeError,
-            ):
-                if decodes_failed < 4:
-                    # ignore decoding errors for now and assume they have to
-                    # do with a channel drop - hope that receiving from the
-                    # channel will raise an expected error and bubble up.
-                    try:
-                        msg_str: str|bytes = msg_bytes.decode()
-                    except UnicodeDecodeError:
-                        msg_str = msg_bytes
-
-                    log.exception(
-                        'Failed to decode msg?\n'
-                        f'{codec}\n\n'
-                        'Rxed bytes from wire:\n\n'
-                        f'{msg_str!r}\n'
-                    )
-                    decodes_failed += 1
-                else:
-                    raise
-
-    async def send(
-        self,
-        msg: msgtypes.MsgType,
-
-        strict_types: bool = True,
-        hide_tb: bool = False,
-
-    ) -> None:
-        '''
-        Send a msgpack encoded py-object-blob-as-msg over TCP.
-
-        If `strict_types == True` then a `MsgTypeError` will be raised on any
-        invalid msg type
-
-        '''
-        __tracebackhide__: bool = hide_tb
-
-        # XXX see `trio._sync.AsyncContextManagerMixin` for details
-        # on the `.acquire()`/`.release()` sequencing..
-        async with self._send_lock:
-
-            # NOTE: lookup the `trio.Task.context`'s var for
-            # the current `MsgCodec`.
-            codec: MsgCodec = _ctxvar_MsgCodec.get()
-
-            # XXX for ctxvar debug only!
-            # if self._codec.pld_spec != codec.pld_spec:
-            #     self._codec = codec
-            #     log.runtime(
-            #         f'Using new codec in {self}.send()\n'
-            #         f'codec: {self._codec}\n\n'
-            #         f'msg: {msg}\n'
-            #     )
-
-            if type(msg) not in msgtypes.__msg_types__:
-                if strict_types:
-                    raise _mk_send_mte(
-                        msg,
-                        codec=codec,
-                    )
-                else:
-                    log.warning(
-                        'Sending non-`Msg`-spec msg?\n\n'
-                        f'{msg}\n'
-                    )
-
-            try:
-                bytes_data: bytes = codec.encode(msg)
-            except TypeError as _err:
-                typerr = _err
-                msgtyperr: MsgTypeError = _mk_send_mte(
-                    msg,
-                    codec=codec,
-                    message=(
-                        f'IPC-msg-spec violation in\n\n'
-                        f'{pretty_struct.Struct.pformat(msg)}'
-                    ),
-                    src_type_error=typerr,
-                )
-                raise msgtyperr from typerr
-
-            # supposedly the fastest says,
-            # https://stackoverflow.com/a/54027962
-            size: bytes = struct.pack("<I", len(bytes_data))
-            return await self.stream.send_all(size + bytes_data)
-
-        # ?TODO? does it help ever to dynamically show this
-        # frame?
-        # try:
-        #     <the-above_code>
-        # except BaseException as _err:
-        #     err = _err
-        #     if not isinstance(err, MsgTypeError):
-        #         __tracebackhide__: bool = False
-        #     raise
-
-    @property
-    def laddr(self) -> tuple[str, int]:
-        return self._laddr
-
-    @property
-    def raddr(self) -> tuple[str, int]:
-        return self._raddr
-
-    async def recv(self) -> Any:
-        return await self._aiter_pkts.asend(None)
-
-    async def drain(self) -> AsyncIterator[dict]:
-        '''
-        Drain the stream's remaining messages sent from
-        the far end until the connection is closed by
-        the peer.
-
-        '''
-        try:
-            async for msg in self._iter_packets():
-                self.drained.append(msg)
-        except TransportClosed:
-            for msg in self.drained:
-                yield msg
-
-    def __aiter__(self):
-        return self._aiter_pkts
-
-    def connected(self) -> bool:
-        return self.stream.socket.fileno() != -1
-
-
-def get_msg_transport(
-
-    key: tuple[str, str],
-
-) -> Type[MsgTransport]:
-
-    return {
-        ('msgpack', 'tcp'): MsgpackTCPStream,
-    }[key]
-
-
-class Channel:
-    '''
-    An inter-process channel for communication between (remote) actors.
-
-    Wraps a ``MsgStream``: transport + encoding IPC connection.
-
-    Currently we only support ``trio.SocketStream`` for transport
-    (aka TCP) and the ``msgpack`` interchange format via the ``msgspec``
-    codec libary.
-
-    '''
-    def __init__(
-
-        self,
-        destaddr: tuple[str, int]|None,
-
-        msg_transport_type_key: tuple[str, str] = ('msgpack', 'tcp'),
-
-        # TODO: optional reconnection support?
-        # auto_reconnect: bool = False,
-        # on_reconnect: typing.Callable[..., typing.Awaitable] = None,
-
-    ) -> None:
-
-        # self._recon_seq = on_reconnect
-        # self._autorecon = auto_reconnect
-
-        self._destaddr = destaddr
-        self._transport_key = msg_transport_type_key
-
-        # Either created in ``.connect()`` or passed in by
-        # user in ``.from_stream()``.
-        self._stream: trio.SocketStream|None = None
-        self._transport: MsgTransport|None = None
-
-        # set after handshake - always uid of far end
-        self.uid: tuple[str, str]|None = None
-
-        self._aiter_msgs = self._iter_msgs()
-        self._exc: Exception|None = None  # set if far end actor errors
-        self._closed: bool = False
-
-        # flag set by ``Portal.cancel_actor()`` indicating remote
-        # (possibly peer) cancellation of the far end actor
-        # runtime.
-        self._cancel_called: bool = False
-
-    @property
-    def msgstream(self) -> MsgTransport:
-        log.info(
-            '`Channel.msgstream` is an old name, use `._transport`'
-        )
-        return self._transport
-
-    @property
-    def transport(self) -> MsgTransport:
-        return self._transport
-
-    @classmethod
-    def from_stream(
-        cls,
-        stream: trio.SocketStream,
-        **kwargs,
-
-    ) -> Channel:
-
-        src, dst = get_stream_addrs(stream)
-        chan = Channel(
-            destaddr=dst,
-            **kwargs,
-        )
-
-        # set immediately here from provided instance
-        chan._stream: trio.SocketStream = stream
-        chan.set_msg_transport(stream)
-        return chan
-
-    def set_msg_transport(
-        self,
-        stream: trio.SocketStream,
-        type_key: tuple[str, str]|None = None,
-
-        # XXX optionally provided codec pair for `msgspec`:
-        # https://jcristharif.com/msgspec/extending.html#mapping-to-from-native-types
-        codec: MsgCodec|None = None,
-
-    ) -> MsgTransport:
-        type_key = (
-            type_key
-            or
-            self._transport_key
-        )
-        # get transport type, then
-        self._transport = get_msg_transport(
-            type_key
-        # instantiate an instance of the msg-transport
-        )(
-            stream,
-            codec=codec,
-        )
-        return self._transport
-
-    @cm
-    def apply_codec(
-        self,
-        codec: MsgCodec,
-
-    ) -> None:
-        '''
-        Temporarily override the underlying IPC msg codec for
-        dynamic enforcement of messaging schema.
-
-        '''
-        orig: MsgCodec = self._transport.codec
-        try:
-            self._transport.codec = codec
-            yield
-        finally:
-            self._transport.codec = orig
-
-    # TODO: do a .src/.dst: str for maddrs?
-    def __repr__(self) -> str:
-        if not self._transport:
-            return '<Channel with inactive transport?>'
-
-        return repr(
-            self._transport.stream.socket._sock
-        ).replace(  # type: ignore
-            "socket.socket",
-            "Channel",
-        )
-
-    @property
-    def laddr(self) -> tuple[str, int]|None:
-        return self._transport.laddr if self._transport else None
-
-    @property
-    def raddr(self) -> tuple[str, int]|None:
-        return self._transport.raddr if self._transport else None
-
-    async def connect(
-        self,
-        destaddr: tuple[Any, ...] | None = None,
-        **kwargs
-
-    ) -> MsgTransport:
-
-        if self.connected():
-            raise RuntimeError("channel is already connected?")
-
-        destaddr = destaddr or self._destaddr
-        assert isinstance(destaddr, tuple)
-
-        stream = await trio.open_tcp_stream(
-            *destaddr,
-            **kwargs
-        )
-        transport = self.set_msg_transport(stream)
-
-        log.transport(
-            f'Opened channel[{type(transport)}]: {self.laddr} -> {self.raddr}'
-        )
-        return transport
-
-    # TODO: something like,
-    # `pdbp.hideframe_on(errors=[MsgTypeError])`
-    # instead of the `try/except` hack we have rn..
-    # seems like a pretty useful thing to have in general
-    # along with being able to filter certain stack frame(s / sets)
-    # possibly based on the current log-level?
-    async def send(
-        self,
-        payload: Any,
-
-        hide_tb: bool = False,
-
-    ) -> None:
-        '''
-        Send a coded msg-blob over the transport.
-
-        '''
-        __tracebackhide__: bool = hide_tb
-        try:
-            log.transport(
-                '=> send IPC msg:\n\n'
-                f'{pformat(payload)}\n'
-            )
-            # assert self._transport  # but why typing?
-            await self._transport.send(
-                payload,
-                hide_tb=hide_tb,
-            )
-        except BaseException as _err:
-            err = _err  # bind for introspection
-            if not isinstance(_err, MsgTypeError):
-                # assert err
-                __tracebackhide__: bool = False
-            else:
-                assert err.cid
-
-            raise
-
-    async def recv(self) -> Any:
-        assert self._transport
-        return await self._transport.recv()
-
-        # TODO: auto-reconnect features like 0mq/nanomsg?
-        # -[ ] implement it manually with nods to SC prot
-        #      possibly on multiple transport backends?
-        #  -> seems like that might be re-inventing scalability
-        #     prots tho no?
-        # try:
-        #     return await self._transport.recv()
-        # except trio.BrokenResourceError:
-        #     if self._autorecon:
-        #         await self._reconnect()
-        #         return await self.recv()
-        #     raise
-
-    async def aclose(self) -> None:
-
-        log.transport(
-            f'Closing channel to {self.uid} '
-            f'{self.laddr} -> {self.raddr}'
-        )
-        assert self._transport
-        await self._transport.stream.aclose()
-        self._closed = True
-
-    async def __aenter__(self):
-        await self.connect()
-        return self
-
-    async def __aexit__(self, *args):
-        await self.aclose(*args)
-
-    def __aiter__(self):
-        return self._aiter_msgs
-
-    # ?TODO? run any reconnection sequence?
-    # -[ ] prolly should be impl-ed as deco-API?
-    #
-    # async def _reconnect(self) -> None:
-    #     """Handle connection failures by polling until a reconnect can be
-    #     established.
-    #     """
-    #     down = False
-    #     while True:
-    #         try:
-    #             with trio.move_on_after(3) as cancel_scope:
-    #                 await self.connect()
-    #             cancelled = cancel_scope.cancelled_caught
-    #             if cancelled:
-    #                 log.transport(
-    #                     "Reconnect timed out after 3 seconds, retrying...")
-    #                 continue
-    #             else:
-    #                 log.transport("Stream connection re-established!")
-
-    #                 # on_recon = self._recon_seq
-    #                 # if on_recon:
-    #                 #     await on_recon(self)
-
-    #                 break
-    #         except (OSError, ConnectionRefusedError):
-    #             if not down:
-    #                 down = True
-    #                 log.transport(
-    #                     f"Connection to {self.raddr} went down, waiting"
-    #                     " for re-establishment")
-    #             await trio.sleep(1)
-
-    async def _iter_msgs(
-        self
-    ) -> AsyncGenerator[Any, None]:
-        '''
-        Yield `MsgType` IPC msgs decoded and deliverd from
-        an underlying `MsgTransport` protocol.
-
-        This is a streaming routine alo implemented as an async-gen
-        func (same a `MsgTransport._iter_pkts()`) gets allocated by
-        a `.__call__()` inside `.__init__()` where it is assigned to
-        the `._aiter_msgs` attr.
-
-        '''
-        assert self._transport
-        while True:
-            try:
-                async for msg in self._transport:
-                    match msg:
-                        # NOTE: if transport/interchange delivers
-                        # a type error, we pack it with the far
-                        # end peer `Actor.uid` and relay the
-                        # `Error`-msg upward to the `._rpc` stack
-                        # for normal RAE handling.
-                        case MsgTypeError():
-                            yield pack_from_raise(
-                                local_err=msg,
-                                cid=msg.cid,
-
-                                # XXX we pack it here bc lower
-                                # layers have no notion of an
-                                # actor-id ;)
-                                src_uid=self.uid,
-                            )
-                        case _:
-                            yield msg
-
-            except trio.BrokenResourceError:
-
-                # if not self._autorecon:
-                raise
-
-            await self.aclose()
-
-            # if self._autorecon:  # attempt reconnect
-            #     await self._reconnect()
-            #     continue
-
-    def connected(self) -> bool:
-        return self._transport.connected() if self._transport else False
-
-
-@acm
-async def _connect_chan(
-    host: str,
-    port: int
-
-) -> typing.AsyncGenerator[Channel, None]:
-    '''
-    Create and connect a channel with disconnect on context manager
-    teardown.
-
-    '''
-    chan = Channel((host, port))
-    await chan.connect()
-    yield chan
-    with trio.CancelScope(shield=True):
-        await chan.aclose()

tractor/_root.py

@@ -18,7 +18,9 @@
 Root actor runtime ignition(s).
 
 '''
-from contextlib import asynccontextmanager as acm
+from contextlib import (
+    asynccontextmanager as acm,
+)
 from functools import partial
 import importlib
 import inspect
@@ -26,96 +28,56 @@ import logging
 import os
 import signal
 import sys
-from typing import Callable
+from typing import (
+    Any,
+    Callable,
+)
 import warnings
 
 
 import trio
 
-from ._runtime import (
+from .runtime import _runtime
-    Actor,
+from .discovery._registry import Registrar
-    Arbiter,
+from .devx import (
-    # TODO: rename and make a non-actor subtype?
+    debug,
-    # Arbiter as Registry,
+    _frame_stack,
-    async_main,
+    pformat as _pformat,
 )
-from .devx import _debug
+from .spawn import _spawn
-from . import _spawn
+from .runtime import _state
-from . import _state
 from . import log
-from ._ipc import _connect_chan
+from .ipc import (
-from ._exceptions import is_multi_cancelled
+    _connect_chan,
-
+)
-
+from .discovery._addr import (
-# set at startup and after forks
+    Address,
-_default_host: str = '127.0.0.1'
+    UnwrappedAddress,
-_default_port: int = 1616
+    default_lo_addrs,
-
+    mk_uuid,
-# default registry always on localhost
+    wrap_address,
-_default_lo_addrs: list[tuple[str, int]] = [(
+)
-    _default_host,
+from .trionics import (
-    _default_port,
+    is_multi_cancelled,
-)]
+    collapse_eg,
+)
+from ._exceptions import (
+    RuntimeFailure,
+)
 
 
 logger = log.get_logger('tractor')
 
 
+# TODO: stick this in a `@acm` defined in `devx.debug`?
+# -[ ] also maybe consider making this a `wrapt`-deco to
+#     save an indent level?
+#
 @acm
-async def open_root_actor(
+async def maybe_block_bp(
-
+    debug_mode: bool,
-    *,
+    maybe_enable_greenback: bool,
-    # defaults are above
+) -> bool:
-    registry_addrs: list[tuple[str, int]]|None = None,
-
-    # defaults are above
-    arbiter_addr: tuple[str, int]|None = None,
-
-    name: str|None = 'root',
-
-    # either the `multiprocessing` start method:
-    # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
-    # OR `trio` (the new default).
-    start_method: _spawn.SpawnMethodKey|None = None,
-
-    # enables the multi-process debugger support
-    debug_mode: bool = False,
-    maybe_enable_greenback: bool = True,  # `.pause_from_sync()/breakpoint()` support
-    enable_stack_on_sig: bool = False,
-
-    # internal logging
-    loglevel: str|None = None,
-
-    enable_modules: list|None = None,
-    rpc_module_paths: list|None = None,
-
-    # NOTE: allow caller to ensure that only one registry exists
-    # and that this call creates it.
-    ensure_registry: bool = False,
-
-    hide_tb: bool = True,
-
-    # XXX, proxied directly to `.devx._debug._maybe_enter_pm()`
-    # for REPL-entry logic.
-    debug_filter: Callable[
-        [BaseException|BaseExceptionGroup],
-        bool,
-    ] = lambda err: not is_multi_cancelled(err),
-
-    # TODO, a way for actors to augment passing derived
-    # read-only state to sublayers?
-    # extra_rt_vars: dict|None = None,
-
-) -> Actor:
-    '''
-    Runtime init entry point for ``tractor``.
-
-    '''
-    _debug.hide_runtime_frames()
-    __tracebackhide__: bool = hide_tb
-
-    # TODO: stick this in a `@cm` defined in `devx._debug`?
-    #
     # Override the global debugger hook to make it play nice with
     # ``trio``, see much discussion in:
     # https://github.com/python-trio/trio/issues/1155#issuecomment-742964018
@@ -124,23 +86,26 @@ async def open_root_actor(
         'PYTHONBREAKPOINT',
         None,
     )
+    bp_blocked: bool
     if (
         debug_mode
-        and maybe_enable_greenback
+        and
+        maybe_enable_greenback
         and (
-            maybe_mod := await _debug.maybe_init_greenback(
+            maybe_mod := await debug.maybe_init_greenback(
                 raise_not_found=False,
             )
         )
     ):
         logger.info(
             f'Found `greenback` installed @ {maybe_mod}\n'
-            'Enabling `tractor.pause_from_sync()` support!\n'
+            f'Enabling `tractor.pause_from_sync()` support!\n'
         )
         os.environ['PYTHONBREAKPOINT'] = (
-            'tractor.devx._debug._sync_pause_from_builtin'
+            'tractor.devx.debug._sync_pause_from_builtin'
         )
         _state._runtime_vars['use_greenback'] = True
+        bp_blocked = False
 
     else:
         # TODO: disable `breakpoint()` by default (without
@@ -159,10 +124,127 @@ async def open_root_actor(
         # lol ok,
         # https://docs.python.org/3/library/sys.html#sys.breakpointhook
         os.environ['PYTHONBREAKPOINT'] = "0"
+        bp_blocked = True
+
+    try:
+        yield bp_blocked
+    finally:
+        # restore any prior built-in `breakpoint()` hook state
+        if builtin_bp_handler is not None:
+            sys.breakpointhook = builtin_bp_handler
+
+        if orig_bp_path is not None:
+            os.environ['PYTHONBREAKPOINT'] = orig_bp_path
+
+        else:
+            # clear env back to having no entry
+            os.environ.pop('PYTHONBREAKPOINT', None)
+
+
+
+@acm
+async def open_root_actor(
+    *,
+    # defaults are above
+    registry_addrs: list[UnwrappedAddress]|None = None,
+
+    # defaults are above
+    arbiter_addr: tuple[UnwrappedAddress]|None = None,
+
+    enable_transports: list[
+        # TODO, this should eventually be the pairs as
+        # defined by (codec, proto) as on `MsgTransport.
+        _state.TransportProtocolKey,
+    ]|None = None,
+
+    name: str|None = 'root',
+
+    # either the `multiprocessing` start method:
+    # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
+    # OR `trio` (the new default).
+    start_method: _spawn.SpawnMethodKey|None = None,
+
+    # enables the multi-process debugger support
+    debug_mode: bool = False,
+    maybe_enable_greenback: bool = False,  # `.pause_from_sync()/breakpoint()` support
+    # ^XXX NOTE^ the perf implications of use,
+    # https://greenback.readthedocs.io/en/latest/principle.html#performance
+    enable_stack_on_sig: bool = False,
+
+    # internal logging
+    loglevel: str|None = None,
+
+    enable_modules: list|None = None,
+    rpc_module_paths: list|None = None,
+
+    # NOTE: allow caller to ensure that only one registry exists
+    # and that this call creates it.
+    ensure_registry: bool = False,
+
+    hide_tb: bool = True,
+
+    # XXX, proxied directly to `.devx.debug._maybe_enter_pm()`
+    # for REPL-entry logic.
+    debug_filter: Callable[
+        [BaseException|BaseExceptionGroup],
+        bool,
+    ] = lambda err: not is_multi_cancelled(err),
+
+    # TODO, a way for actors to augment passing derived
+    # read-only state to sublayers?
+    # extra_rt_vars: dict|None = None,
+
+) -> _runtime.Actor:
+    '''
+    Initialize the `tractor` runtime by starting a "root actor" in
+    a parent-most Python process.
+
+    All (disjoint) actor-process-trees-as-programs are created via
+    this entrypoint.
+
+    '''
+    # XXX NEVER allow nested actor-trees!
+    if already_actor := _state.current_actor(
+        err_on_no_runtime=False,
+    ):
+        rtvs: dict[str, Any] = _state._runtime_vars
+        root_mailbox: list[str, int] = rtvs['_root_mailbox']
+        registry_addrs: list[list[str, int]] = rtvs['_registry_addrs']
+        raise RuntimeFailure(
+            f'A current actor already exists !?\n'
+            f'({already_actor}\n'
+            f'\n'
+            f'You can NOT open a second root actor from within '
+            f'an existing tree and the current root of this '
+            f'already exists !!\n'
+            f'\n'
+            f'_root_mailbox: {root_mailbox!r}\n'
+            f'_registry_addrs: {registry_addrs!r}\n'
+        )
+
+    async with maybe_block_bp(
+        debug_mode=debug_mode,
+        maybe_enable_greenback=maybe_enable_greenback,
+    ):
+        if enable_transports is None:
+            enable_transports: list[str] = _state.current_ipc_protos()
+        else:
+            _state._runtime_vars['_enable_tpts'] = enable_transports
+
+        # TODO! support multi-tpts per actor!
+        # Bo
+        if not len(enable_transports) == 1:
+            raise RuntimeError(
+                f'No multi-tpt support yet!\n'
+                f'enable_transports={enable_transports!r}\n'
+            )
+
+        _frame_stack.hide_runtime_frames()
+        __tracebackhide__: bool = hide_tb
 
         # attempt to retreive ``trio``'s sigint handler and stash it
         # on our debugger lock state.
-    _debug.DebugStatus._trio_handler = signal.getsignal(signal.SIGINT)
+        debug.DebugStatus._trio_handler = signal.getsignal(signal.SIGINT)
 
         # mark top most level process as root actor
         _state._runtime_vars['_is_root'] = True
@@ -193,29 +275,38 @@ async def open_root_actor(
                 DeprecationWarning,
                 stacklevel=2,
             )
-        registry_addrs = [arbiter_addr]
+            uw_reg_addrs = [arbiter_addr]
 
-    registry_addrs: list[tuple[str, int]] = (
+        uw_reg_addrs = registry_addrs
-        registry_addrs
+        if not uw_reg_addrs:
-        or
+            uw_reg_addrs: list[UnwrappedAddress] = default_lo_addrs(
-        _default_lo_addrs
+                enable_transports
             )
-    assert registry_addrs
 
-    loglevel = (
+        # must exist by now since all below code is dependent
+        assert uw_reg_addrs
+        registry_addrs: list[Address] = [
+            wrap_address(uw_addr)
+            for uw_addr in uw_reg_addrs
+        ]
+
+        loglevel: str = (
             loglevel
-        or log._default_loglevel
+            or
-    ).upper()
+            log._default_loglevel
+        )
+        loglevel: str = loglevel.upper()
 
         if (
             debug_mode
-        and _spawn._spawn_method == 'trio'
+            and
+            _spawn._spawn_method == 'trio'
         ):
             _state._runtime_vars['_debug_mode'] = True
 
             # expose internal debug module to every actor allowing for
             # use of ``await tractor.pause()``
-        enable_modules.append('tractor.devx._debug')
+            enable_modules.append('tractor.devx.debug._tty_lock')
 
             # if debug mode get's enabled *at least* use that level of
             # logging for some informative console prompts.
@@ -235,7 +326,10 @@ async def open_root_actor(
             )
 
         assert loglevel
-    _log = log.get_console_log(loglevel)
+        _log = log.get_console_log(
+            level=loglevel,
+            name='tractor',
+        )
         assert _log
 
         # TODO: factor this into `.devx._stackscope`!!
@@ -248,10 +342,10 @@ async def open_root_actor(
             enable_stack_on_sig()
 
         # closed into below ping task-func
-    ponged_addrs: list[tuple[str, int]] = []
+        ponged_addrs: list[Address] = []
 
         async def ping_tpt_socket(
-        addr: tuple[str, int],
+            addr: Address,
             timeout: float = 1,
         ) -> None:
             '''
@@ -271,26 +365,34 @@ async def open_root_actor(
                 # be better to eventually have a "discovery" protocol
                 # with basic handshake instead?
                 with trio.move_on_after(timeout):
-                async with _connect_chan(*addr):
+                    async with _connect_chan(addr.unwrap()):
                         ponged_addrs.append(addr)
 
             except OSError:
-            # TODO: make this a "discovery" log level?
+                # ?TODO, make this a "discovery" log level?
                 logger.info(
-                f'No actor registry found @ {addr}\n'
+                    f'No root-actor registry found @ {addr!r}\n'
                 )
 
+        # !TODO, this is basically just another (abstract)
+        # happy-eyeballs, so we should try for formalize it somewhere
+        # in a `.[_]discovery` ya?
+        #
         async with trio.open_nursery() as tn:
-        for addr in registry_addrs:
+            for uw_addr in uw_reg_addrs:
+                addr: Address = wrap_address(uw_addr)
                 tn.start_soon(
                     ping_tpt_socket,
-                tuple(addr),  # TODO: just drop this requirement?
+                    addr,
                 )
 
-    trans_bind_addrs: list[tuple[str, int]] = []
+        tpt_bind_addrs: list[
+            Address  # `Address.get_random()` case
+            |UnwrappedAddress  # registrar case `= uw_reg_addrs`
+        ] = []
 
-    # Create a new local root-actor instance which IS NOT THE
+        # ------ NON-REGISTRAR ------
-    # REGISTRAR
+        # create a new root-actor instance.
         if ponged_addrs:
             if ensure_registry:
                 raise RuntimeError(
@@ -298,33 +400,49 @@ async def open_root_actor(
                     'registry socket(s) already bound'
                 )
 
-        # we were able to connect to an arbiter
+            # we were able to connect to a registrar
             logger.info(
                 f'Registry(s) seem(s) to exist @ {ponged_addrs}'
             )
 
-        actor = Actor(
+            actor = _runtime.Actor(
                 name=name or 'anonymous',
+                uuid=mk_uuid(),
                 registry_addrs=ponged_addrs,
                 loglevel=loglevel,
                 enable_modules=enable_modules,
             )
-        # DO NOT use the registry_addrs as the transport server
+            # **DO NOT** use the registry_addrs as the
-        # addrs for this new non-registar, root-actor.
+            # ipc-transport-server's bind-addrs as this is
-        for host, port in ponged_addrs:
+            # a new NON-registrar, ROOT-actor.
-            # NOTE: zero triggers dynamic OS port allocation
+            #
-            trans_bind_addrs.append((host, 0))
+            # XXX INSTEAD, bind random addrs using the same tpt
+            # proto.
+            for addr in ponged_addrs:
+                tpt_bind_addrs.append(
+                    # XXX, these are `Address` NOT `UnwrappedAddress`.
+                    #
+                    # NOTE, in the case of posix/berkley socket
+                    # protos we allocate port=0 such that the system
+                    # allocates a random value at bind time; this
+                    # happens in the `.ipc.*` stack's backend.
+                    addr.get_random(
+                        bindspace=addr.bindspace,
+                    )
+                )
 
+        # ------ REGISTRAR ------
+        # create a new "registry providing" root-actor instance.
+        #
         # Start this local actor as the "registrar", aka a regular
         # actor who manages the local registry of "mailboxes" of
         # other process-tree-local sub-actors.
         else:
-
             # NOTE that if the current actor IS THE REGISTAR, the
             # following init steps are taken:
-        # - the tranport layer server is bound to each (host, port)
+            # - the tranport layer server is bound to each addr
             #   pair defined in provided registry_addrs, or the default.
-        trans_bind_addrs = registry_addrs
+            tpt_bind_addrs = uw_reg_addrs
 
             # - it is normally desirable for any registrar to stay up
             #   indefinitely until either all registered (child/sub)
@@ -335,14 +453,15 @@ async def open_root_actor(
             # https://github.com/goodboy/tractor/pull/348
             # https://github.com/goodboy/tractor/issues/296
 
-        actor = Arbiter(
+            actor = Registrar(
-            name or 'registrar',
+                name=name or 'registrar',
+                uuid=mk_uuid(),
                 registry_addrs=registry_addrs,
                 loglevel=loglevel,
                 enable_modules=enable_modules,
             )
             # XXX, in case the root actor runtime was actually run from
-        # `tractor.to_asyncio.run_as_asyncio_guest()` and NOt
+            # `tractor.to_asyncio.run_as_asyncio_guest()` and NOT
             # `.trio.run()`.
             actor._infected_aio = _state._runtime_vars['_is_infected_aio']
 
@@ -353,21 +472,28 @@ async def open_root_actor(
 
             # start local channel-server and fake the portal API
             # NOTE: this won't block since we provide the nursery
-        ml_addrs_str: str = '\n'.join(
+            report: str = f'Starting actor-runtime for {actor.aid.reprol()!r}\n'
-            f'@{addr}' for addr in trans_bind_addrs
+            if reg_addrs := actor.registry_addrs:
+                report += (
+                    '-> Opening new registry @ '
+                    +
+                    '\n'.join(
+                        f'{addr}' for addr in reg_addrs
                     )
-        logger.info(
-            f'Starting local {actor.uid} on the following transport addrs:\n'
-            f'{ml_addrs_str}'
                 )
+            logger.info(f'{report}\n')
 
-        # start the actor runtime in a new task
+            # start runtime in a bg sub-task, yield to caller.
-        async with trio.open_nursery(
+            async with (
-            strict_exception_groups=False,
+                collapse_eg(),
-            # ^XXX^ TODO? instead unpack any RAE as per "loose" style?
+                trio.open_nursery() as root_tn,
-        ) as nursery:
 
-            # ``_runtime.async_main()`` creates an internal nursery
+                # ?TODO? finally-footgun below?
+                # -> see note on why shielding.
+                # maybe_raise_from_masking_exc(),
+            ):
+                actor._root_tn = root_tn
+                # `_runtime.async_main()` creates an internal nursery
                 # and blocks here until any underlying actor(-process)
                 # tree has terminated thereby conducting so called
                 # "end-to-end" structured concurrency throughout an
@@ -375,14 +501,39 @@ async def open_root_actor(
                 # "actor runtime" primitives are SC-compat and thus all
                 # transitively spawned actors/processes must be as
                 # well.
-            await nursery.start(
+                accept_addrs: list[UnwrappedAddress]
+                reg_addrs: list[UnwrappedAddress]
+                (
+                    accept_addrs,
+                    reg_addrs,
+                ) = await root_tn.start(
                     partial(
-                    async_main,
+                        _runtime.async_main,
                         actor,
-                    accept_addrs=trans_bind_addrs,
+                        accept_addrs=tpt_bind_addrs,
                         parent_addr=None
                     )
                 )
+                # NOTE, only set a local-host addr (i.e. like
+                # `lo`-loopback for TCP) for the process-tree-global
+                # "root"-process (its tree-wide "mailbox") since all
+                # sub-actors should be able to speak to their root
+                # actor over that channel.
+                #
+                # ?TODO, per-OS non-network-proto alt options?
+                # -[ ] on linux we should be able to always use UDS?
+                #
+                raddrs: list[UnwrappedAddress] = _state._runtime_vars['_root_addrs']
+                raddrs.extend(
+                    accept_addrs,
+                )
+                # TODO, remove once we have also removed all usage;
+                # eventually all (root-)registry apis should expect > 1 addr.
+                _state._runtime_vars['_root_mailbox'] = raddrs[0]
+                # if 'chart' in actor.aid.name:
+                #     from tractor.devx import mk_pdb
+                #     mk_pdb().set_trace()
+
                 try:
                     yield actor
                 except (
@@ -393,7 +544,7 @@ async def open_root_actor(
                     # TODO, in beginning to handle the subsubactor with
                     # crashed grandparent cases..
                     #
-                # was_locked: bool = await _debug.maybe_wait_for_debugger(
+                    # was_locked: bool = await debug.maybe_wait_for_debugger(
                     #     child_in_debug=True,
                     # )
                     # XXX NOTE XXX see equiv note inside
@@ -401,10 +552,15 @@ async def open_root_actor(
                     # non-root or root-that-opened-this-mahually case we
                     # wait for the local actor-nursery to exit before
                     # exiting the transport channel handler.
-                entered: bool = await _debug._maybe_enter_pm(
+                    entered: bool = await debug._maybe_enter_pm(
                         err,
                         api_frame=inspect.currentframe(),
                         debug_filter=debug_filter,
+
+                        # XXX NOTE, required to debug root-actor
+                        # crashes under cancellation conditions; so
+                        # most of them!
+                        shield=root_tn.cancel_scope.cancel_called,
                     )
 
                     if (
@@ -414,14 +570,18 @@ async def open_root_actor(
                             err,
                         )
                     ):
-                    logger.exception('Root actor crashed\n')
+                        logger.exception(
+                            'Root actor crashed\n'
+                            f'>x)\n'
+                            f' |_{actor}\n'
+                        )
 
                     # ALWAYS re-raise any error bubbled up from the
                     # runtime!
                     raise
 
                 finally:
-                # NOTE: not sure if we'll ever need this but it's
+                    # NOTE/TODO?, not sure if we'll ever need this but it's
                     # possibly better for even more determinism?
                     # logger.cancel(
                     #     f'Waiting on {len(nurseries)} nurseries in root..')
@@ -430,31 +590,52 @@ async def open_root_actor(
                     #     for an in nurseries:
                     #         tempn.start_soon(an.exited.wait)
 
-                logger.info(
+                    op_nested_actor_repr: str = _pformat.nest_from_op(
-                    'Closing down root actor'
+                        input_op='>) ',
+                        text=actor.pformat(),
+                        nest_prefix='|_',
                     )
+                    logger.info(
+                        f'Closing down root actor\n'
+                        f'{op_nested_actor_repr}'
+                    )
+                    # XXX, THIS IS A *finally-footgun*!
+                    # (also mentioned in with-block above)
+                    # -> though already shields iternally it can
+                    # taskc here and mask underlying errors raised in
+                    # the try-block above?
+                    with trio.CancelScope(shield=True):
                         await actor.cancel(None)  # self cancel
         finally:
-        _state._current_actor = None
+            # revert all process-global runtime state
-        _state._last_actor_terminated = actor
-
-        # restore built-in `breakpoint()` hook state
             if (
                 debug_mode
                 and
-            maybe_enable_greenback
+                _spawn._spawn_method == 'trio'
             ):
-            if builtin_bp_handler is not None:
+                _state._runtime_vars['_debug_mode'] = False
-                sys.breakpointhook = builtin_bp_handler
 
-            if orig_bp_path is not None:
+            # !XXX, clear ALL prior contact info state, this is MEGA
-                os.environ['PYTHONBREAKPOINT'] = orig_bp_path
+            # important if you are opening the runtime multiple times
+            # from the same parent process (like in our test
+            # harness)!
+            _state._runtime_vars['_root_addrs'].clear()
+            _state._runtime_vars['_root_mailbox'] = None
 
-            else:
+            _state._current_actor = None
-                # clear env back to having no entry
+            _state._last_actor_terminated = actor
-                os.environ.pop('PYTHONBREAKPOINT', None)
 
-        logger.runtime("Root actor terminated")
+            sclang_repr: str = _pformat.nest_from_op(
+                input_op=')>',
+                text=actor.pformat(),
+                nest_prefix='|_',
+                nest_indent=1,
+            )
+
+            logger.info(
+                f'Root actor terminated\n'
+                f'{sclang_repr}'
+            )
 
 
 def run_daemon(
@@ -462,7 +643,7 @@ def run_daemon(
 
     # runtime kwargs
     name: str | None = 'root',
-    registry_addrs: list[tuple[str, int]] = _default_lo_addrs,
+    registry_addrs: list[UnwrappedAddress]|None = None,
 
     start_method: str | None = None,
     debug_mode: bool = False,

tractor/_state.py

@@ -1,145 +0,0 @@
-# tractor: structured concurrent "actors".
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-"""
-Per process state
-
-"""
-from __future__ import annotations
-from contextvars import (
-    ContextVar,
-)
-from typing import (
-    Any,
-    TYPE_CHECKING,
-)
-
-from trio.lowlevel import current_task
-
-if TYPE_CHECKING:
-    from ._runtime import Actor
-    from ._context import Context
-
-
-_current_actor: Actor|None = None  # type: ignore # noqa
-_last_actor_terminated: Actor|None = None
-
-# TODO: mk this a `msgspec.Struct`!
-_runtime_vars: dict[str, Any] = {
-    '_debug_mode': False,
-    '_is_root': False,
-    '_root_mailbox': (None, None),
-    '_registry_addrs': [],
-
-    '_is_infected_aio': False,
-
-    # for `tractor.pause_from_sync()` & `breakpoint()` support
-    'use_greenback': False,
-}
-
-
-def last_actor() -> Actor|None:
-    '''
-    Try to return last active `Actor` singleton
-    for this process.
-
-    For case where runtime already exited but someone is asking
-    about the "last" actor probably to get its `.uid: tuple`.
-
-    '''
-    return _last_actor_terminated
-
-
-def current_actor(
-    err_on_no_runtime: bool = True,
-) -> Actor:
-    '''
-    Get the process-local actor instance.
-
-    '''
-    if (
-        err_on_no_runtime
-        and
-        _current_actor is None
-    ):
-        msg: str = 'No local actor has been initialized yet?\n'
-        from ._exceptions import NoRuntime
-
-        if last := last_actor():
-            msg += (
-                f'Apparently the lact active actor was\n'
-                f'|_{last}\n'
-                f'|_{last.uid}\n'
-            )
-        # no actor runtime has (as of yet) ever been started for
-        # this process.
-        else:
-            msg += (
-                # 'No last actor found?\n'
-                '\nDid you forget to call one of,\n'
-                '- `tractor.open_root_actor()`\n'
-                '- `tractor.open_nursery()`\n'
-            )
-
-        raise NoRuntime(msg)
-
-    return _current_actor
-
-
-def is_main_process() -> bool:
-    '''
-    Bool determining if this actor is running in the top-most process.
-
-    '''
-    import multiprocessing as mp
-    return mp.current_process().name == 'MainProcess'
-
-
-# TODO, more verby name?
-def debug_mode() -> bool:
-    '''
-    Bool determining if "debug mode" is on which enables
-    remote subactor pdb entry on crashes.
-
-    '''
-    return bool(_runtime_vars['_debug_mode'])
-
-
-def is_root_process() -> bool:
-    return _runtime_vars['_is_root']
-
-
-_ctxvar_Context: ContextVar[Context] = ContextVar(
-    'ipc_context',
-    default=None,
-)
-
-
-def current_ipc_ctx(
-    error_on_not_set: bool = False,
-) -> Context|None:
-    ctx: Context = _ctxvar_Context.get()
-
-    if (
-        not ctx
-        and error_on_not_set
-    ):
-        from ._exceptions import InternalError
-        raise InternalError(
-            'No IPC context has been allocated for this task yet?\n'
-            f'|_{current_task()}\n'
-        )
-    return ctx

tractor/_streaming.py

@@ -38,6 +38,7 @@ import trio
 from ._exceptions import (
     ContextCancelled,
     RemoteActorError,
+    TransportClosed,
 )
 from .log import get_logger
 from .trionics import (
@@ -54,12 +55,12 @@ from tractor.msg import (
 )
 
 if TYPE_CHECKING:
-    from ._runtime import Actor
+    from .runtime._runtime import Actor
     from ._context import Context
-    from ._ipc import Channel
+    from .ipc import Channel
 
 
-log = get_logger(__name__)
+log = get_logger()
 
 
 # TODO: the list
@@ -409,10 +410,8 @@ class MsgStream(trio.abc.Channel):
             # it).
             with trio.CancelScope(shield=True):
                 await self._ctx.send_stop()
-
         except (
-            trio.BrokenResourceError,
+            TransportClosed,
-            trio.ClosedResourceError
         ) as re:
             # the underlying channel may already have been pulled
             # in which case our stop message is meaningless since
@@ -426,8 +425,8 @@ class MsgStream(trio.abc.Channel):
             self._closed = re
 
         # if caught_eoc:
-        #     # from .devx import _debug
+        #     # from .devx import debug
-        #     # await _debug.pause()
+        #     # await debug.pause()
         #     with trio.CancelScope(shield=True):
         #         await rx_chan.aclose()
 
@@ -437,22 +436,23 @@ class MsgStream(trio.abc.Channel):
             message: str = (
                 f'Stream self-closed by {this_side!r}-side before EoC from {peer_side!r}\n'
                 # } bc a stream is a "scope"/msging-phase inside an IPC
-                f'x}}>\n'
+                f'c}}>\n'
                 f'  |_{self}\n'
             )
-            log.cancel(message)
-            self._eoc = trio.EndOfChannel(message)
-
             if (
                 (rx_chan := self._rx_chan)
                 and
                 (stats := rx_chan.statistics()).tasks_waiting_receive
             ):
-                log.cancel(
+                message += (
-                    f'Msg-stream is closing but there is still reader tasks,\n'
+                    f'AND there is still reader tasks,\n'
+                    f'\n'
                     f'{stats}\n'
                 )
 
+            log.cancel(message)
+            self._eoc = trio.EndOfChannel(message)
+
         # ?XXX WAIT, why do we not close the local mem chan `._rx_chan` XXX?
         # => NO, DEFINITELY NOT! <=
         # if we're a bi-dir `MsgStream` BECAUSE this same
@@ -592,11 +592,19 @@ class MsgStream(trio.abc.Channel):
                 ),
             )
         except (
-            trio.ClosedResourceError,
-            trio.BrokenResourceError,
             BrokenPipeError,
-        ) as trans_err:
+            TransportClosed,
-            if hide_tb:
+        ) as _trans_err:
+            trans_err = _trans_err
+            if (
+                hide_tb
+                and
+                self._ctx.chan._exc is trans_err
+                # ^XXX, IOW, only if the channel is marked errored
+                # for the same reason as whatever its underlying
+                # transport raised, do we keep the full low-level tb
+                # suppressed from the user.
+            ):
                 raise type(trans_err)(
                     *trans_err.args
                 ) from trans_err
@@ -802,13 +810,12 @@ async def open_stream_from_ctx(
                 # sanity, can remove?
                 assert eoc is stream._eoc
 
-                log.warning(
+                log.runtime(
                     'Stream was terminated by EoC\n\n'
                     # NOTE: won't show the error <Type> but
                     # does show txt followed by IPC msg.
                     f'{str(eoc)}\n'
                 )
-
         finally:
             if ctx._portal:
                 try:

tractor/_testing/__init__.py

@@ -26,7 +26,7 @@ import os
 import pathlib
 
 import tractor
-from tractor.devx._debug import (
+from tractor.devx.debug import (
     BoxedMaybeException,
 )
 from .pytest import (
@@ -37,6 +37,9 @@ from .fault_simulation import (
 )
 
 
+# TODO, use dulwhich for this instead?
+# -> we're going to likely need it (or something similar)
+#   for supporting hot-coad reload feats eventually anyway!
 def repodir() -> pathlib.Path:
     '''
     Return the abspath to the repo directory.

tractor/_testing/addr.py

@@ -0,0 +1,72 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Random IPC addr generation for isolating
+the discovery space between test sessions.
+
+Might be eventually useful to expose as a util set from
+our `tractor.discovery` subsys?
+
+'''
+import random
+from typing import (
+    Type,
+)
+from tractor.discovery import _addr
+
+
+def get_rando_addr(
+    tpt_proto: str,
+    *,
+
+    # choose random port at import time
+    _rando_port: str = random.randint(1000, 9999)
+
+) -> tuple[str, str|int]:
+    '''
+    Used to globally override the runtime to the
+    per-test-session-dynamic addr so that all tests never conflict
+    with any other actor tree using the default.
+
+    '''
+    addr_type: Type[_addr.Addres] = _addr._address_types[tpt_proto]
+    def_reg_addr: tuple[str, int] = _addr._default_lo_addrs[tpt_proto]
+
+    # this is the "unwrapped" form expected to be passed to
+    # `.open_root_actor()` by test body.
+    testrun_reg_addr: tuple[str, int|str]
+    match tpt_proto:
+        case 'tcp':
+            testrun_reg_addr = (
+                addr_type.def_bindspace,
+                _rando_port,
+            )
+
+        # NOTE, file-name uniqueness (no-collisions) will be based on
+        # the runtime-directory and root (pytest-proc's) pid.
+        case 'uds':
+            from tractor.ipc._uds import UDSAddress
+            addr: UDSAddress = addr_type.get_random()
+            assert addr.is_valid
+            assert addr.sockpath.resolve()
+            testrun_reg_addr = addr.unwrap()
+
+    # XXX, as sanity it should never the same as the default for the
+    # host-singleton registry actor.
+    assert def_reg_addr != testrun_reg_addr
+
+    return testrun_reg_addr

tractor/_testing/pytest.py

@@ -21,77 +21,135 @@ and applications.
 '''
 from functools import (
     partial,
-    wraps,
 )
 import inspect
 import platform
+from typing import (
+    Callable,
+    Type,
+)
 
+import pytest
 import tractor
 import trio
+import wrapt
 
 
-def tractor_test(fn):
+def tractor_test(
+    wrapped: Callable|None = None,
+    *,
+    # @tractor_test(<deco-params>)
+    timeout:float = 30,
+    hide_tb: bool = True,
+):
     '''
-    Decorator for async test funcs to present them as "native"
+    Decorator for async test fns to decorator-wrap them as "native"
-    looking sync funcs runnable by `pytest` using `trio.run()`.
+    looking sync funcs runnable by `pytest` and auto invoked with
+    `trio.run()` (much like the `pytest-trio` plugin's approach).
 
-    Use:
+    Further the test fn body will be invoked AFTER booting the actor
+    runtime, i.e. from inside a `tractor.open_root_actor()` block AND
+    with various runtime and tooling parameters implicitly passed as
+    requested by by the test session's config; see immediately below.
 
-    @tractor_test
+    Basic deco use:
-    async def test_whatever():
+    ---------------
+
+      @tractor_test(
+        timeout=10,
+      )
+      async def test_whatever(
+          # fixture param declarations
+          loglevel: str,
+          start_method: str,
+          reg_addr: tuple,
+          tpt_proto: str,
+          debug_mode: bool,
+      ):
+          # already inside a root-actor runtime `trio.Task`
           await ...
 
-    If fixtures:
 
-        - ``reg_addr`` (a socket addr tuple where arbiter is listening)
+    Runtime config via special fixtures:
-        - ``loglevel`` (logging level passed to tractor internals)
+    ------------------------------------
-        - ``start_method`` (subprocess spawning backend)
+    If any of the following fixture are requested by the wrapped test
+    fn (via normal func-args declaration),
+
+    - `reg_addr` (a socket addr tuple where registrar is listening)
+    - `loglevel` (logging level passed to tractor internals)
+    - `start_method` (subprocess spawning backend)
+
+    (TODO support)
+    - `tpt_proto` (IPC transport protocol key)
+
+    they will be automatically injected to each test as normally
+    expected as well as passed to the initial
+    `tractor.open_root_actor()` funcargs.
 
-    are defined in the `pytest` fixture space they will be automatically
-    injected to tests declaring these funcargs.
     '''
-    @wraps(fn)
+    __tracebackhide__: bool = hide_tb
+
+    # handle the decorator not called with () case.
+    # i.e. in `wrapt` support a deco-with-optional-args,
+    # https://wrapt.readthedocs.io/en/master/decorators.html#decorators-with-optional-arguments
+    if wrapped is None:
+        return wrapt.PartialCallableObjectProxy(
+            tractor_test,
+            timeout=timeout,
+            hide_tb=hide_tb
+        )
+
+    @wrapt.decorator
     def wrapper(
+        wrapped: Callable,
+        instance: object|Type|None,
+        args: tuple,
+        kwargs: dict,
+    ):
+        __tracebackhide__: bool = hide_tb
+
+        # NOTE, ensure we inject any test-fn declared fixture names.
+        for kw in [
+            'reg_addr',
+            'loglevel',
+            'start_method',
+            'debug_mode',
+            'tpt_proto',
+            'timeout',
+        ]:
+            if kw in inspect.signature(wrapped).parameters:
+                assert kw in kwargs
+
+        start_method = kwargs.get('start_method')
+        if platform.system() == "Windows":
+            if start_method is None:
+                kwargs['start_method'] = 'trio'
+            elif start_method != 'trio':
+                raise ValueError(
+                    'ONLY the `start_method="trio"` is supported on Windows.'
+                )
+
+        # open a root-actor, passing certain
+        # `tractor`-runtime-settings, then invoke the test-fn body as
+        # the root-most task.
+        #
+        # https://wrapt.readthedocs.io/en/master/decorators.html#processing-function-arguments
+        async def _main(
             *args,
-        loglevel=None,
+
-        reg_addr=None,
+            # runtime-settings
+            loglevel:str|None = None,
+            reg_addr:tuple|None = None,
             start_method: str|None = None,
             debug_mode: bool = False,
-        **kwargs
+            tpt_proto: str|None = None,
+
+            **kwargs,
         ):
-        # __tracebackhide__ = True
+            __tracebackhide__: bool = hide_tb
 
-        # NOTE: inject ant test func declared fixture
+            with trio.fail_after(timeout):
-        # names by manually checking!
-        if 'reg_addr' in inspect.signature(fn).parameters:
-            # injects test suite fixture value to test as well
-            # as `run()`
-            kwargs['reg_addr'] = reg_addr
-
-        if 'loglevel' in inspect.signature(fn).parameters:
-            # allows test suites to define a 'loglevel' fixture
-            # that activates the internal logging
-            kwargs['loglevel'] = loglevel
-
-        if start_method is None:
-            if platform.system() == "Windows":
-                start_method = 'trio'
-
-        if 'start_method' in inspect.signature(fn).parameters:
-            # set of subprocess spawning backends
-            kwargs['start_method'] = start_method
-
-        if 'debug_mode' in inspect.signature(fn).parameters:
-            # set of subprocess spawning backends
-            kwargs['debug_mode'] = debug_mode
-
-
-        if kwargs:
-
-            # use explicit root actor start
-            async def _main():
                 async with tractor.open_root_actor(
-                    # **kwargs,
                     registry_addrs=[reg_addr] if reg_addr else None,
                     loglevel=loglevel,
                     start_method=start_method,
@@ -100,14 +158,217 @@ def tractor_test(fn):
                     debug_mode=debug_mode,
 
                 ):
-                    await fn(*args, **kwargs)
+                    # invoke test-fn body IN THIS task
+                    await wrapped(
+                        *args,
+                        **kwargs,
+                    )
 
-            main = _main
+        funcname = wrapped.__name__
+        if not inspect.iscoroutinefunction(wrapped):
+            raise TypeError(
+                f"Test-fn {funcname!r} must be an async-function !!"
+            )
 
-        else:
+        # invoke runtime via a root task.
-            # use implicit root actor start
+        return trio.run(
-            main = partial(fn, *args, **kwargs)
+            partial(
+                _main,
+                *args,
+                **kwargs,
+            )
+        )
 
-        return trio.run(main)
 
-    return wrapper
+    return wrapper(
+        wrapped,
+    )
+
+
+def pytest_addoption(
+    parser: pytest.Parser,
+):
+    # parser.addoption(
+    #     "--ll",
+    #     action="store",
+    #     dest='loglevel',
+    #     default='ERROR', help="logging level to set when testing"
+    # )
+
+    parser.addoption(
+        "--spawn-backend",
+        action="store",
+        dest='spawn_backend',
+        default='trio',
+        help="Processing spawning backend to use for test run",
+    )
+
+    parser.addoption(
+        "--tpdb",
+        "--debug-mode",
+        action="store_true",
+        dest='tractor_debug_mode',
+        # default=False,
+        help=(
+            'Enable a flag that can be used by tests to to set the '
+            '`debug_mode: bool` for engaging the internal '
+            'multi-proc debugger sys.'
+        ),
+    )
+
+    # provide which IPC transport protocols opting-in test suites
+    # should accumulatively run against.
+    parser.addoption(
+        "--tpt-proto",
+        nargs='+',  # accumulate-multiple-args
+        action="store",
+        dest='tpt_protos',
+        default=['tcp'],
+        help="Transport protocol to use under the `tractor.ipc.Channel`",
+    )
+
+
+def pytest_configure(config):
+    backend = config.option.spawn_backend
+    from tractor.spawn._spawn import try_set_start_method
+    try_set_start_method(backend)
+
+    # register custom marks to avoid warnings see,
+    # https://docs.pytest.org/en/stable/how-to/writing_plugins.html#registering-custom-markers
+    config.addinivalue_line(
+        'markers',
+        'no_tpt(proto_key): test will (likely) not behave with tpt backend'
+    )
+
+
+@pytest.fixture(scope='session')
+def debug_mode(request) -> bool:
+    '''
+    Flag state for whether `--tpdb` (for `tractor`-py-debugger)
+    was passed to the test run.
+
+    Normally tests should pass this directly to `.open_root_actor()`
+    to allow the user to opt into suite-wide crash handling.
+
+    '''
+    debug_mode: bool = request.config.option.tractor_debug_mode
+    return debug_mode
+
+
+@pytest.fixture(scope='session')
+def spawn_backend(request) -> str:
+    return request.config.option.spawn_backend
+
+
+@pytest.fixture(scope='session')
+def tpt_protos(request) -> list[str]:
+
+    # allow quoting on CLI
+    proto_keys: list[str] = [
+        proto_key.replace('"', '').replace("'", "")
+        for proto_key in request.config.option.tpt_protos
+    ]
+
+    # ?TODO, eventually support multiple protos per test-sesh?
+    if len(proto_keys) > 1:
+        pytest.fail(
+            'We only support one `--tpt-proto <key>` atm!\n'
+        )
+
+    # XXX ensure we support the protocol by name via lookup!
+    for proto_key in proto_keys:
+        from tractor.discovery import _addr
+        addr_type = _addr._address_types[proto_key]
+        assert addr_type.proto_key == proto_key
+
+    yield proto_keys
+
+
+@pytest.fixture(
+    scope='session',
+    autouse=True,
+)
+def tpt_proto(
+    request,
+    tpt_protos: list[str],
+) -> str:
+    proto_key: str = tpt_protos[0]
+
+    # ?TODO, but needs a fn-scoped tpt_proto fixture..
+    # @pytest.mark.no_tpt('uds')
+    # node = request.node
+    # markers = node.own_markers
+    # for mark in markers:
+    #     if (
+    #         mark.name == 'no_tpt'
+    #         and
+    #         proto_key in mark.args
+    #     ):
+    #         pytest.skip(
+    #             f'Test {node} normally fails with '
+    #             f'tpt-proto={proto_key!r}\n'
+    #         )
+
+    from tractor.runtime import _state
+    if _state._def_tpt_proto != proto_key:
+        _state._def_tpt_proto = proto_key
+        _state._runtime_vars['_enable_tpts'] = [
+            proto_key,
+        ]
+
+    yield proto_key
+
+
+@pytest.fixture(scope='session')
+def reg_addr(
+    tpt_proto: str,
+) -> tuple[str, int|str]:
+    '''
+    Deliver a test-sesh unique registry address such
+    that each run's (tests which use this fixture) will
+    have no conflicts/cross-talk when running simultaneously
+    nor will interfere with other live `tractor` apps active
+    on the same network-host (namespace).
+
+    '''
+    from tractor._testing.addr import get_rando_addr
+    return get_rando_addr(
+        tpt_proto=tpt_proto,
+    )
+
+
+def pytest_generate_tests(
+    metafunc: pytest.Metafunc,
+):
+    spawn_backend: str = metafunc.config.option.spawn_backend
+
+    if not spawn_backend:
+        # XXX some weird windows bug with `pytest`?
+        spawn_backend = 'trio'
+
+    # TODO: maybe just use the literal `._spawn.SpawnMethodKey`?
+    assert spawn_backend in (
+        'mp_spawn',
+        'mp_forkserver',
+        'trio',
+    )
+
+    # NOTE: used-to-be-used-to dyanmically parametrize tests for when
+    # you just passed --spawn-backend=`mp` on the cli, but now we expect
+    # that cli input to be manually specified, BUT, maybe we'll do
+    # something like this again in the future?
+    if 'start_method' in metafunc.fixturenames:
+        metafunc.parametrize(
+            "start_method",
+            [spawn_backend],
+            scope='module',
+        )
+
+    # TODO, parametrize any `tpt_proto: str` declaring tests!
+    # proto_tpts: list[str] = metafunc.config.option.proto_tpts
+    # if 'tpt_proto' in metafunc.fixturenames:
+    #     metafunc.parametrize(
+    #         'tpt_proto',
+    #         proto_tpts,  # TODO, double check this list usage!
+    #         scope='module',
+    #     )

tractor/_testing/samples.py

@@ -0,0 +1,35 @@
+import os
+import random
+
+
+def generate_sample_messages(
+    amount: int,
+    rand_min: int = 0,
+    rand_max: int = 0,
+    silent: bool = False
+) -> tuple[list[bytes], int]:
+
+    msgs = []
+    size = 0
+
+    if not silent:
+        print(f'\ngenerating {amount} messages...')
+
+    for i in range(amount):
+        msg = f'[{i:08}]'.encode('utf-8')
+
+        if rand_max > 0:
+            msg += os.urandom(
+                random.randint(rand_min, rand_max))
+
+        size += len(msg)
+
+        msgs.append(msg)
+
+        if not silent and i and i % 10_000 == 0:
+            print(f'{i} generated')
+
+    if not silent:
+        print(f'done, {size:,} bytes in total')
+
+    return msgs, size

tractor/devx/__init__.py

@@ -20,7 +20,7 @@ Runtime "developer experience" utils and addons to aid our
 and working with/on the actor runtime.
 
 """
-from ._debug import (
+from .debug import (
     maybe_wait_for_debugger as maybe_wait_for_debugger,
     acquire_debug_lock as acquire_debug_lock,
     breakpoint as breakpoint,

tractor/devx/_frame_stack.py

@@ -20,13 +20,18 @@ as it pertains to improving the grok-ability of our runtime!
 
 '''
 from __future__ import annotations
+from contextlib import (
+    _GeneratorContextManager,
+    _AsyncGeneratorContextManager,
+)
 from functools import partial
 import inspect
+import textwrap
 from types import (
     FrameType,
     FunctionType,
     MethodType,
-    # CodeType,
+    CodeType,
 )
 from typing import (
     Any,
@@ -34,6 +39,9 @@ from typing import (
     Type,
 )
 
+import pdbp
+from tractor.log import get_logger
+import trio
 from tractor.msg import (
     pretty_struct,
     NamespacePath,
@@ -41,6 +49,8 @@ from tractor.msg import (
 import wrapt
 
 
+log = get_logger()
+
 # TODO: yeah, i don't love this and we should prolly just
 # write a decorator that actually keeps a stupid ref to the func
 # obj..
@@ -301,3 +311,70 @@ def api_frame(
 #     error_set: set[BaseException],
 # ) -> TracebackType:
 #     ...
+
+
+def hide_runtime_frames() -> dict[FunctionType, CodeType]:
+    '''
+    Hide call-stack frames for various std-lib and `trio`-API primitives
+    such that the tracebacks presented from our runtime are as minimized
+    as possible, particularly from inside a `PdbREPL`.
+
+    '''
+    # XXX HACKZONE XXX
+    #  hide exit stack frames on nurseries and cancel-scopes!
+    # |_ so avoid seeing it when the `pdbp` REPL is first engaged from
+    #    inside a `trio.open_nursery()` scope (with no line after it
+    #    in before the block end??).
+    #
+    # TODO: FINALLY got this workin originally with
+    #  `@pdbp.hideframe` around the `wrapper()` def embedded inside
+    #  `_ki_protection_decoratior()`.. which is in the module:
+    #  /home/goodboy/.virtualenvs/tractor311/lib/python3.11/site-packages/trio/_core/_ki.py
+    #
+    # -[ ] make an issue and patch for `trio` core? maybe linked
+    #    to the long outstanding `pdb` one below?
+    #   |_ it's funny that there's frame hiding throughout `._run.py`
+    #      but not where it matters on the below exit funcs..
+    #
+    # -[ ] provide a patchset for the lonstanding
+    #   |_ https://github.com/python-trio/trio/issues/1155
+    #
+    # -[ ] make a linked issue to ^ and propose allowing all the
+    #     `._core._run` code to have their `__tracebackhide__` value
+    #     configurable by a `RunVar` to allow getting scheduler frames
+    #     if desired through configuration?
+    #
+    # -[ ] maybe dig into the core `pdb` issue why the extra frame is shown
+    #      at all?
+    #
+    funcs: list[FunctionType] = [
+        trio._core._run.NurseryManager.__aexit__,
+        trio._core._run.CancelScope.__exit__,
+         _GeneratorContextManager.__exit__,
+         _AsyncGeneratorContextManager.__aexit__,
+         _AsyncGeneratorContextManager.__aenter__,
+         trio.Event.wait,
+    ]
+    func_list_str: str = textwrap.indent(
+        "\n".join(f.__qualname__ for f in funcs),
+        prefix=' |_ ',
+    )
+    log.devx(
+        'Hiding the following runtime frames by default:\n'
+        f'{func_list_str}\n'
+    )
+
+    codes: dict[FunctionType, CodeType] = {}
+    for ref in funcs:
+        # stash a pre-modified version of each ref's code-obj
+        # so it can be reverted later if needed.
+        codes[ref] = ref.__code__
+        pdbp.hideframe(ref)
+    #
+    # pdbp.hideframe(trio._core._run.NurseryManager.__aexit__)
+    # pdbp.hideframe(trio._core._run.CancelScope.__exit__)
+    # pdbp.hideframe(_GeneratorContextManager.__exit__)
+    # pdbp.hideframe(_AsyncGeneratorContextManager.__aexit__)
+    # pdbp.hideframe(_AsyncGeneratorContextManager.__aenter__)
+    # pdbp.hideframe(trio.Event.wait)
+    return codes

tractor/devx/_stackscope.py

@@ -45,17 +45,15 @@ from typing import (
 )
 
 import trio
-from tractor import (
+from tractor.runtime import _state
-    _state,
+from tractor import log as logmod
-    log as logmod,
+from tractor.devx import debug
-)
-from tractor.devx import _debug
 
-log = logmod.get_logger(__name__)
+log = logmod.get_logger()
 
 
 if TYPE_CHECKING:
-    from tractor._spawn import ProcessType
+    from tractor.spawn._spawn import ProcessType
     from tractor import (
         Actor,
         ActorNursery,
@@ -82,7 +80,7 @@ def dump_task_tree() -> None:
     if (
         current_sigint_handler
         is not
-        _debug.DebugStatus._trio_handler
+        debug.DebugStatus._trio_handler
     ):
         sigint_handler_report: str = (
             'The default `trio` SIGINT handler was replaced?!'
@@ -238,7 +236,8 @@ def enable_stack_on_sig(
         import stackscope
     except ImportError:
         log.warning(
-            '`stackscope` not installed for use in debug mode!'
+            'The `stackscope` lib is not installed!\n'
+            '`Ignoring enable_stack_on_sig() call!\n'
         )
         return None
 
@@ -255,8 +254,8 @@ def enable_stack_on_sig(
         dump_tree_on_sig,
     )
     log.devx(
-        'Enabling trace-trees on `SIGUSR1` '
+        f'Enabling trace-trees on `SIGUSR1` '
-        'since `stackscope` is installed @ \n'
+        f'since `stackscope` is installed @ \n'
         f'{stackscope!r}\n\n'
         f'With `SIGUSR1` handler\n'
         f'|_{dump_tree_on_sig}\n'

tractor/devx/debug/__init__.py

@@ -0,0 +1,100 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+Multi-actor debugging for da peeps!
+
+'''
+from __future__ import annotations
+from tractor.log import get_logger
+from ._repl import (
+    PdbREPL as PdbREPL,
+    mk_pdb as mk_pdb,
+    TractorConfig as TractorConfig,
+)
+from ._tty_lock import (
+    DebugStatus as DebugStatus,
+    DebugStateError as DebugStateError,
+)
+from ._trace import (
+    Lock as Lock,
+    _pause_msg as _pause_msg,
+    _repl_fail_msg as _repl_fail_msg,
+    _set_trace as _set_trace,
+    _sync_pause_from_builtin as _sync_pause_from_builtin,
+    breakpoint as breakpoint,
+    maybe_init_greenback as maybe_init_greenback,
+    maybe_import_greenback as maybe_import_greenback,
+    pause as pause,
+    pause_from_sync as pause_from_sync,
+)
+from ._post_mortem import (
+    BoxedMaybeException as BoxedMaybeException,
+    maybe_open_crash_handler as maybe_open_crash_handler,
+    open_crash_handler as open_crash_handler,
+    post_mortem as post_mortem,
+    _crash_msg as _crash_msg,
+    _maybe_enter_pm as _maybe_enter_pm,
+)
+from ._sync import (
+    maybe_wait_for_debugger as maybe_wait_for_debugger,
+    acquire_debug_lock as acquire_debug_lock,
+)
+from ._sigint import (
+    sigint_shield as sigint_shield,
+    _ctlc_ignore_header as _ctlc_ignore_header
+)
+
+log = get_logger()
+
+# ----------------
+# XXX PKG TODO XXX
+# ----------------
+# refine the internal impl and APIs!
+#
+# -[ ] rework `._pause()` and it's branch-cases for root vs.
+#     subactor:
+#  -[ ] `._pause_from_root()` + `_pause_from_subactor()`?
+#  -[ ]  do the de-factor based on bg-thread usage in
+#    `.pause_from_sync()` & `_pause_from_bg_root_thread()`.
+#  -[ ] drop `debug_func == None` case which is confusing af..
+#  -[ ]  factor out `_enter_repl_sync()` into a util func for calling
+#    the `_set_trace()` / `_post_mortem()` APIs?
+#
+# -[ ] figure out if we need `acquire_debug_lock()` and/or re-implement
+#    it as part of the `.pause_from_sync()` rework per above?
+#
+# -[ ] pair the `._pause_from_subactor()` impl with a "debug nursery"
+#   that's dynamically allocated inside the `._rpc` task thus
+#   avoiding the `._service_n.start()` usage for the IPC request?
+#  -[ ] see the TODO inside `._rpc._errors_relayed_via_ipc()`
+#
+# -[ ] impl a `open_debug_request()` which encaps all
+#   `request_root_stdio_lock()` task scheduling deats
+#   + `DebugStatus` state mgmt; which should prolly be re-branded as
+#   a `DebugRequest` type anyway AND with suppoort for bg-thread
+#   (from root actor) usage?
+#
+# -[ ] handle the `xonsh` case for bg-root-threads in the SIGINT
+#     handler!
+#   -[ ] do we need to do the same for subactors?
+#   -[ ] make the failing tests finally pass XD
+#
+# -[ ] simplify `maybe_wait_for_debugger()` to be a root-task only
+#     API?
+#   -[ ] currently it's implemented as that so might as well make it
+#     formal?

tractor/devx/debug/_post_mortem.py

@@ -0,0 +1,412 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+Post-mortem debugging APIs and surrounding machinery for both
+sync and async contexts.
+
+Generally we maintain the same semantics a `pdb.post.mortem()` but
+with actor-tree-wide sync/cooperation around any (sub)actor's use of
+the root's TTY.
+
+'''
+from __future__ import annotations
+import bdb
+from contextlib import (
+    AbstractContextManager,
+    contextmanager as cm,
+    nullcontext,
+)
+from functools import (
+    partial,
+)
+import inspect
+import sys
+import traceback
+from typing import (
+    Callable,
+    Sequence,
+    Type,
+    TYPE_CHECKING,
+)
+from types import (
+    TracebackType,
+    FrameType,
+)
+
+from msgspec import Struct
+import trio
+from tractor._exceptions import (
+    NoRuntime,
+)
+from tractor.runtime import _state
+from tractor.runtime._state import (
+    current_actor,
+    debug_mode,
+)
+from tractor.log import get_logger
+from tractor.trionics import (
+    is_multi_cancelled,
+)
+from ._trace import (
+    _pause,
+)
+from ._tty_lock import (
+    DebugStatus,
+)
+from ._repl import (
+    PdbREPL,
+    mk_pdb,
+    TractorConfig as TractorConfig,
+)
+
+if TYPE_CHECKING:
+    from trio.lowlevel import Task
+    from tractor.runtime._runtime import (
+        Actor,
+    )
+
+_crash_msg: str = (
+    'Opening a pdb REPL in crashed actor'
+)
+
+log = get_logger()
+
+
+class BoxedMaybeException(Struct):
+    '''
+    Box a maybe-exception for post-crash introspection usage
+    from the body of a `open_crash_handler()` scope.
+
+    '''
+    value: BaseException|None = None
+
+    # handler can suppress crashes dynamically
+    raise_on_exit: bool|Sequence[Type[BaseException]] = True
+
+    def pformat(self) -> str:
+        '''
+        Repr the boxed `.value` error in more-than-string
+        repr form.
+
+        '''
+        if not self.value:
+            return f'<{type(self).__name__}( .value=None )>'
+
+        return (
+            f'<{type(self.value).__name__}(\n'
+            f' |_.value = {self.value}\n'
+            f')>\n'
+        )
+
+    __repr__ = pformat
+
+
+def _post_mortem(
+    repl: PdbREPL,  # normally passed by `_pause()`
+
+    # XXX all `partial`-ed in by `post_mortem()` below!
+    tb: TracebackType,
+    api_frame: FrameType,
+
+    shield: bool = False,
+    hide_tb: bool = True,
+
+    # maybe pre/post REPL entry
+    repl_fixture: (
+        AbstractContextManager[bool]
+        |None
+    ) = None,
+
+    boxed_maybe_exc: BoxedMaybeException|None = None,
+
+) -> None:
+    '''
+    Enter the ``pdbpp`` port mortem entrypoint using our custom
+    debugger instance.
+
+    '''
+    __tracebackhide__: bool = hide_tb
+
+    # maybe enter any user fixture
+    enter_repl: bool = DebugStatus.maybe_enter_repl_fixture(
+        repl=repl,
+        repl_fixture=repl_fixture,
+        boxed_maybe_exc=boxed_maybe_exc,
+    )
+    try:
+        if not enter_repl:
+            # XXX, trigger `.release()` below immediately!
+            return
+        try:
+            actor: Actor = current_actor()
+            actor_repr: str = str(actor.uid)
+            # ^TODO, instead a nice runtime-info + maddr + uid?
+            # -[ ] impl a `Actor.__repr()__`??
+            #  |_ <task>:<thread> @ <actor>
+
+        except NoRuntime:
+            actor_repr: str = '<no-actor-runtime?>'
+
+        try:
+            task_repr: Task = trio.lowlevel.current_task()
+        except RuntimeError:
+            task_repr: str = '<unknown-Task>'
+
+        # TODO: print the actor supervion tree up to the root
+        # here! Bo
+        log.pdb(
+            f'{_crash_msg}\n'
+            f'x>(\n'
+            f' |_ {task_repr} @ {actor_repr}\n'
+
+        )
+
+        # XXX NOTE(s) on `pdbp.xpm()` version..
+        #
+        # - seems to lose the up-stack tb-info?
+        # - currently we're (only) replacing this from `pdbp.xpm()`
+        #   to add the `end=''` to the print XD
+        #
+        print(traceback.format_exc(), end='')
+        caller_frame: FrameType = api_frame.f_back
+
+        # NOTE, see the impl details of these in the lib to
+        # understand usage:
+        # - `pdbp.post_mortem()`
+        # - `pdbp.xps()`
+        # - `bdb.interaction()`
+        repl.reset()
+        repl.interaction(
+            frame=caller_frame,
+            # frame=None,
+            traceback=tb,
+        )
+    finally:
+        # XXX NOTE XXX: this is abs required to avoid hangs!
+        #
+        # Since we presume the post-mortem was enaged to
+        # a task-ending error, we MUST release the local REPL request
+        # so that not other local task nor the root remains blocked!
+        DebugStatus.release()
+
+
+async def post_mortem(
+    *,
+    tb: TracebackType|None = None,
+    api_frame: FrameType|None = None,
+    hide_tb: bool = False,
+
+    # TODO: support shield here just like in `pause()`?
+    # shield: bool = False,
+
+    **_pause_kwargs,
+
+) -> None:
+    '''
+    Our builtin async equivalient of `pdb.post_mortem()` which can be
+    used inside exception handlers.
+
+    It's also used for the crash handler when `debug_mode == True` ;)
+
+    '''
+    __tracebackhide__: bool = hide_tb
+
+    tb: TracebackType = tb or sys.exc_info()[2]
+
+    # TODO: do upward stack scan for highest @api_frame and
+    # use its parent frame as the expected user-app code
+    # interact point.
+    api_frame: FrameType = api_frame or inspect.currentframe()
+
+    # TODO, move to submod `._pausing` or ._api? _trace
+    await _pause(
+        debug_func=partial(
+            _post_mortem,
+            api_frame=api_frame,
+            tb=tb,
+        ),
+        hide_tb=hide_tb,
+        **_pause_kwargs
+    )
+
+
+async def _maybe_enter_pm(
+    err: BaseException,
+    *,
+    tb: TracebackType|None = None,
+    api_frame: FrameType|None = None,
+    hide_tb: bool = True,
+
+    # only enter debugger REPL when returns `True`
+    debug_filter: Callable[
+        [BaseException|BaseExceptionGroup],
+        bool,
+    ] = lambda err: not is_multi_cancelled(err),
+    **_pause_kws,
+):
+    if (
+        debug_mode()
+
+        # NOTE: don't enter debug mode recursively after quitting pdb
+        # Iow, don't re-enter the repl if the `quit` command was issued
+        # by the user.
+        and not isinstance(err, bdb.BdbQuit)
+
+        # XXX: if the error is the likely result of runtime-wide
+        # cancellation, we don't want to enter the debugger since
+        # there's races between when the parent actor has killed all
+        # comms and when the child tries to contact said parent to
+        # acquire the tty lock.
+
+        # Really we just want to mostly avoid catching KBIs here so there
+        # might be a simpler check we can do?
+        and
+        debug_filter(err)
+    ):
+        api_frame: FrameType = api_frame or inspect.currentframe()
+        tb: TracebackType = tb or sys.exc_info()[2]
+        await post_mortem(
+            api_frame=api_frame,
+            tb=tb,
+            **_pause_kws,
+        )
+        return True
+
+    else:
+        return False
+
+
+# TODO: better naming and what additionals?
+# - [ ] optional runtime plugging?
+# - [ ] detection for sync vs. async code?
+# - [ ] specialized REPL entry when in distributed mode?
+# -[x] hide tb by def
+# - [x] allow ignoring kbi Bo
+@cm
+def open_crash_handler(
+    catch: set[BaseException] = {
+        BaseException,
+    },
+    ignore: set[BaseException] = {
+        KeyboardInterrupt,
+        trio.Cancelled,
+    },
+    hide_tb: bool = True,
+
+    repl_fixture: (
+        AbstractContextManager[bool]  # pre/post REPL entry
+        |None
+    ) = None,
+    raise_on_exit: bool|Sequence[Type[BaseException]] = True,
+):
+    '''
+    Generic "post mortem" crash handler using `pdbp` REPL debugger.
+
+    We expose this as a CLI framework addon to both `click` and
+    `typer` users so they can quickly wrap cmd endpoints which get
+    automatically wrapped to use the runtime's `debug_mode: bool`
+    AND `pdbp.pm()` around any code that is PRE-runtime entry
+    - any sync code which runs BEFORE the main call to
+      `trio.run()`.
+
+    '''
+    __tracebackhide__: bool = hide_tb
+
+    # TODO, yield a `outcome.Error`-like boxed type?
+    # -[~] use `outcome.Value/Error` X-> frozen!
+    # -[x] write our own..?
+    # -[ ] consider just wtv is used by `pytest.raises()`?
+    #
+    boxed_maybe_exc = BoxedMaybeException(
+        raise_on_exit=raise_on_exit,
+    )
+    err: BaseException
+    try:
+        yield boxed_maybe_exc
+    except tuple(catch) as err:
+        boxed_maybe_exc.value = err
+        if (
+            type(err) not in ignore
+            and
+            not is_multi_cancelled(
+                err,
+                ignore_nested=ignore
+            )
+        ):
+            try:
+                # use our re-impl-ed version of `pdbp.xpm()`
+                _post_mortem(
+                    repl=mk_pdb(),
+                    tb=sys.exc_info()[2],
+                    api_frame=inspect.currentframe().f_back,
+                    hide_tb=hide_tb,
+
+                    repl_fixture=repl_fixture,
+                    boxed_maybe_exc=boxed_maybe_exc,
+                )
+            except bdb.BdbQuit:
+                __tracebackhide__: bool = False
+                raise err
+
+        if (
+            raise_on_exit is True
+            or (
+                raise_on_exit is not False
+                and (
+                    set(raise_on_exit)
+                    and
+                    type(err) in raise_on_exit
+                )
+            )
+            and
+            boxed_maybe_exc.raise_on_exit == raise_on_exit
+        ):
+            raise err
+
+
+@cm
+def maybe_open_crash_handler(
+    pdb: bool|None = None,
+    hide_tb: bool = True,
+
+    **kwargs,
+):
+    '''
+    Same as `open_crash_handler()` but with bool input flag
+    to allow conditional handling.
+
+    Normally this is used with CLI endpoints such that if the --pdb
+    flag is passed the pdb REPL is engaed on any crashes B)
+
+    '''
+    __tracebackhide__: bool = hide_tb
+
+    if pdb is None:
+        pdb: bool = _state.is_debug_mode()
+
+    rtctx = nullcontext(
+        enter_result=BoxedMaybeException()
+    )
+    if pdb:
+        rtctx = open_crash_handler(
+            hide_tb=hide_tb,
+            **kwargs,
+        )
+
+    with rtctx as boxed_maybe_exc:
+        yield boxed_maybe_exc

tractor/devx/debug/_repl.py

@@ -0,0 +1,207 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+`pdpp.Pdb` extentions/customization and other delegate usage.
+
+'''
+from functools import (
+    cached_property,
+)
+import os
+
+import pdbp
+from tractor.runtime._state import (
+    is_root_process,
+)
+
+from ._tty_lock import (
+    Lock,
+    DebugStatus,
+)
+
+
+class TractorConfig(pdbp.DefaultConfig):
+    '''
+    Custom `pdbp` config which tries to use the best tradeoff
+    between pretty and minimal.
+
+    '''
+    use_pygments: bool = True
+    sticky_by_default: bool = False
+    enable_hidden_frames: bool = True
+
+    # much thanks @mdmintz for the hot tip!
+    # fixes line spacing issue when resizing terminal B)
+    truncate_long_lines: bool = False
+
+    # ------ - ------
+    # our own custom config vars mostly
+    # for syncing with the actor tree's singleton
+    # TTY `Lock`.
+
+
+class PdbREPL(pdbp.Pdb):
+    '''
+    Add teardown hooks and local state describing any
+    ongoing TTY `Lock` request dialog.
+
+    '''
+    # override the pdbp config with our coolio one
+    # NOTE: this is only loaded when no `~/.pdbrc` exists
+    # so we should prolly pass it into the .__init__() instead?
+    # i dunno, see the `DefaultFactory` and `pdb.Pdb` impls.
+    DefaultConfig = TractorConfig
+
+    status = DebugStatus
+
+    # NOTE: see details in stdlib's `bdb.py`
+    # def user_exception(self, frame, exc_info):
+    #     '''
+    #     Called when we stop on an exception.
+    #     '''
+    #     log.warning(
+    #         'Exception during REPL sesh\n\n'
+    #         f'{frame}\n\n'
+    #         f'{exc_info}\n\n'
+    #     )
+
+    # NOTE: this actually hooks but i don't see anyway to detect
+    # if an error was caught.. this is why currently we just always
+    # call `DebugStatus.release` inside `_post_mortem()`.
+    # def preloop(self):
+    #     print('IN PRELOOP')
+    #     super().preloop()
+
+    # TODO: cleaner re-wrapping of all this?
+    # -[ ] figure out how to disallow recursive .set_trace() entry
+    #     since that'll cause deadlock for us.
+    # -[ ] maybe a `@cm` to call `super().<same_meth_name>()`?
+    # -[ ] look at hooking into the `pp` hook specially with our
+    #     own set of pretty-printers?
+    #    * `.pretty_struct.Struct.pformat()`
+    #    * `.pformat(MsgType.pld)`
+    #    * `.pformat(Error.tb_str)`?
+    #    * .. maybe more?
+    #
+    def set_continue(self):
+        try:
+            super().set_continue()
+        finally:
+            # NOTE: for subactors the stdio lock is released via the
+            # allocated RPC locker task, so for root we have to do it
+            # manually.
+            if (
+                is_root_process()
+                and
+                Lock._debug_lock.locked()
+                and
+                DebugStatus.is_main_trio_thread()
+            ):
+                # Lock.release(raise_on_thread=False)
+                Lock.release()
+
+            # XXX AFTER `Lock.release()` for root local repl usage
+            DebugStatus.release()
+
+    def set_quit(self):
+        try:
+            super().set_quit()
+        finally:
+            if (
+                is_root_process()
+                and
+                Lock._debug_lock.locked()
+                and
+                DebugStatus.is_main_trio_thread()
+            ):
+                # Lock.release(raise_on_thread=False)
+                Lock.release()
+
+            # XXX after `Lock.release()` for root local repl usage
+            DebugStatus.release()
+
+    # XXX NOTE: we only override this because apparently the stdlib pdb
+    # bois likes to touch the SIGINT handler as much as i like to touch
+    # my d$%&.
+    def _cmdloop(self):
+        self.cmdloop()
+
+    @cached_property
+    def shname(self) -> str | None:
+        '''
+        Attempt to return the login shell name with a special check for
+        the infamous `xonsh` since it seems to have some issues much
+        different from std shells when it comes to flushing the prompt?
+
+        '''
+        # SUPER HACKY and only really works if `xonsh` is not used
+        # before spawning further sub-shells..
+        shpath = os.getenv('SHELL', None)
+
+        if shpath:
+            if (
+                os.getenv('XONSH_LOGIN', default=False)
+                or 'xonsh' in shpath
+            ):
+                return 'xonsh'
+
+            return os.path.basename(shpath)
+
+        return None
+
+
+def mk_pdb() -> PdbREPL:
+    '''
+    Deliver a new `PdbREPL`: a multi-process safe `pdbp.Pdb`-variant
+    using the magic of `tractor`'s SC-safe IPC.
+
+    B)
+
+    Our `pdb.Pdb` subtype accomplishes multi-process safe debugging
+    by:
+
+    - mutexing access to the root process' std-streams (& thus parent
+      process TTY) via an IPC managed `Lock` singleton per
+      actor-process tree.
+
+    - temporarily overriding any subactor's SIGINT handler to shield
+      during live REPL sessions in sub-actors such that cancellation
+      is never (mistakenly) triggered by a ctrl-c and instead only by
+      explicit runtime API requests or after the
+      `pdb.Pdb.interaction()` call has returned.
+
+    FURTHER, the `pdbp.Pdb` instance is configured to be `trio`
+    "compatible" from a SIGINT handling perspective; we mask out
+    the default `pdb` handler and instead apply `trio`s default
+    which mostly addresses all issues described in:
+
+     - https://github.com/python-trio/trio/issues/1155
+
+    The instance returned from this factory should always be
+    preferred over the default `pdb[p].set_trace()` whenever using
+    a `pdb` REPL inside a `trio` based runtime.
+
+    '''
+    pdb = PdbREPL()
+
+    # XXX: These are the important flags mentioned in
+    # https://github.com/python-trio/trio/issues/1155
+    # which resolve the traceback spews to console.
+    pdb.allow_kbdint = True
+    pdb.nosigint = True
+    return pdb

tractor/devx/debug/_sigint.py

@@ -0,0 +1,340 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+A custom SIGINT handler which mainly shields actor (task)
+cancellation during REPL interaction.
+
+'''
+from __future__ import annotations
+import platform
+from typing import (
+    TYPE_CHECKING,
+)
+import trio
+from tractor.log import get_logger
+from tractor.runtime._state import (
+    current_actor,
+    is_root_process,
+)
+from ._repl import (
+    PdbREPL,
+)
+from ._tty_lock import (
+    any_connected_locker_child,
+    DebugStatus,
+    Lock,
+)
+
+if TYPE_CHECKING:
+    from tractor.ipc import (
+        Channel,
+    )
+    from tractor.runtime._runtime import (
+        Actor,
+    )
+
+log = get_logger()
+
+_is_macos: bool = platform.system() == 'Darwin'
+_ctlc_ignore_header: str = (
+    'Ignoring SIGINT while debug REPL in use'
+)
+
+
+def sigint_shield(
+    signum: int,
+    frame: 'frame',  # type: ignore # noqa
+    *args,
+
+) -> None:
+    '''
+    Specialized, debugger-aware SIGINT handler.
+
+    In childred we always ignore/shield for SIGINT to avoid
+    deadlocks since cancellation should always be managed by the
+    supervising parent actor. The root actor-proces is always
+    cancelled on ctrl-c.
+
+    '''
+    __tracebackhide__: bool = True
+    actor: Actor = current_actor()
+
+    def do_cancel():
+        # If we haven't tried to cancel the runtime then do that instead
+        # of raising a KBI (which may non-gracefully destroy
+        # a ``trio.run()``).
+        if not actor._cancel_called:
+            actor.cancel_soon()
+
+        # If the runtime is already cancelled it likely means the user
+        # hit ctrl-c again because teardown didn't fully take place in
+        # which case we do the "hard" raising of a local KBI.
+        else:
+            raise KeyboardInterrupt
+
+    # only set in the actor actually running the REPL
+    repl: PdbREPL|None = DebugStatus.repl
+
+    # TODO: maybe we should flatten out all these cases using
+    # a match/case?
+    #
+    # root actor branch that reports whether or not a child
+    # has locked debugger.
+    if is_root_process():
+        # log.warning(
+        log.devx(
+            'Handling SIGINT in root actor\n'
+            f'{Lock.repr()}'
+            f'{DebugStatus.repr()}\n'
+        )
+        # try to see if the supposed (sub)actor in debug still
+        # has an active connection to *this* actor, and if not
+        # it's likely they aren't using the TTY lock / debugger
+        # and we should propagate SIGINT normally.
+        any_connected: bool = any_connected_locker_child()
+
+        problem = (
+            f'root {actor.uid} handling SIGINT\n'
+            f'any_connected: {any_connected}\n\n'
+
+            f'{Lock.repr()}\n'
+        )
+
+        if (
+            (ctx := Lock.ctx_in_debug)
+            and
+            (uid_in_debug := ctx.chan.uid) # "someone" is (ostensibly) using debug `Lock`
+        ):
+            name_in_debug: str = uid_in_debug[0]
+            assert not repl
+            # if not repl:  # but it's NOT us, the root actor.
+            # sanity: since no repl ref is set, we def shouldn't
+            # be the lock owner!
+            assert name_in_debug != 'root'
+
+            # IDEAL CASE: child has REPL as expected
+            if any_connected:  # there are subactors we can contact
+                # XXX: only if there is an existing connection to the
+                # (sub-)actor in debug do we ignore SIGINT in this
+                # parent! Otherwise we may hang waiting for an actor
+                # which has already terminated to unlock.
+                #
+                # NOTE: don't emit this with `.pdb()` level in
+                # root without a higher level.
+                log.runtime(
+                    _ctlc_ignore_header
+                    +
+                    f' by child '
+                    f'{uid_in_debug}\n'
+                )
+                problem = None
+
+            else:
+                problem += (
+                    '\n'
+                    f'A `pdb` REPL is SUPPOSEDLY in use by child {uid_in_debug}\n'
+                    f'BUT, no child actors are IPC contactable!?!?\n'
+                )
+
+        # IDEAL CASE: root has REPL as expected
+        else:
+            # root actor still has this SIGINT handler active without
+            # an actor using the `Lock` (a bug state) ??
+            # => so immediately cancel any stale lock cs and revert
+            # the handler!
+            if not DebugStatus.repl:
+                # TODO: WHEN should we revert back to ``trio``
+                # handler if this one is stale?
+                # -[ ] maybe after a counts work of ctl-c mashes?
+                # -[ ] use a state var like `stale_handler: bool`?
+                problem += (
+                    'No subactor is using a `pdb` REPL according `Lock.ctx_in_debug`?\n'
+                    'BUT, the root should be using it, WHY this handler ??\n\n'
+                    'So either..\n'
+                    '- some root-thread is using it but has no `.repl` set?, OR\n'
+                    '- something else weird is going on outside the runtime!?\n'
+                )
+            else:
+                # NOTE: since we emit this msg on ctl-c, we should
+                # also always re-print the prompt the tail block!
+                log.pdb(
+                    _ctlc_ignore_header
+                    +
+                    f' by root actor..\n'
+                    f'{DebugStatus.repl_task}\n'
+                    f' |_{repl}\n'
+                )
+                problem = None
+
+        # XXX if one is set it means we ARE NOT operating an ideal
+        # case where a child subactor or us (the root) has the
+        # lock without any other detected problems.
+        if problem:
+
+            # detect, report and maybe clear a stale lock request
+            # cancel scope.
+            lock_cs: trio.CancelScope = Lock.get_locking_task_cs()
+            maybe_stale_lock_cs: bool = (
+                lock_cs is not None
+                and not lock_cs.cancel_called
+            )
+            if maybe_stale_lock_cs:
+                problem += (
+                    '\n'
+                    'Stale `Lock.ctx_in_debug._scope: CancelScope` detected?\n'
+                    f'{Lock.ctx_in_debug}\n\n'
+
+                    '-> Calling ctx._scope.cancel()!\n'
+                )
+                lock_cs.cancel()
+
+            # TODO: wen do we actually want/need this, see above.
+            # DebugStatus.unshield_sigint()
+            log.warning(problem)
+
+    # child actor that has locked the debugger
+    elif not is_root_process():
+        log.debug(
+            f'Subactor {actor.uid} handling SIGINT\n\n'
+            f'{Lock.repr()}\n'
+        )
+
+        rent_chan: Channel = actor._parent_chan
+        if (
+            rent_chan is None
+            or
+            not rent_chan.connected()
+        ):
+            log.warning(
+                'This sub-actor thinks it is debugging '
+                'but it has no connection to its parent ??\n'
+                f'{actor.uid}\n'
+                'Allowing SIGINT propagation..'
+            )
+            DebugStatus.unshield_sigint()
+
+        repl_task: str|None = DebugStatus.repl_task
+        req_task: str|None = DebugStatus.req_task
+        if (
+            repl_task
+            and
+            repl
+        ):
+            log.pdb(
+                _ctlc_ignore_header
+                +
+                f' by local task\n\n'
+                f'{repl_task}\n'
+                f' |_{repl}\n'
+            )
+        elif req_task:
+            log.debug(
+                _ctlc_ignore_header
+                +
+                f' by local request-task and either,\n'
+                f'- someone else is already REPL-in and has the `Lock`, or\n'
+                f'- some other local task already is replin?\n\n'
+                f'{req_task}\n'
+            )
+
+        # TODO can we remove this now?
+        # -[ ] does this path ever get hit any more?
+        else:
+            msg: str = (
+                'SIGINT shield handler still active BUT, \n\n'
+            )
+            if repl_task is None:
+                msg += (
+                    '- No local task claims to be in debug?\n'
+                )
+
+            if repl is None:
+                msg += (
+                    '- No local REPL is currently active?\n'
+                )
+
+            if req_task is None:
+                msg += (
+                    '- No debug request task is active?\n'
+                )
+
+            log.warning(
+                msg
+                +
+                'Reverting handler to `trio` default!\n'
+            )
+            DebugStatus.unshield_sigint()
+
+            # XXX ensure that the reverted-to-handler actually is
+            # able to rx what should have been **this** KBI ;)
+            do_cancel()
+
+        # TODO: how to handle the case of an intermediary-child actor
+        # that **is not** marked in debug mode? See oustanding issue:
+        # https://github.com/goodboy/tractor/issues/320
+        # elif debug_mode():
+
+    # maybe redraw/print last REPL output to console since
+    # we want to alert the user that more input is expect since
+    # nothing has been done dur to ignoring sigint.
+    if (
+        DebugStatus.repl  # only when current actor has a REPL engaged
+    ):
+        flush_status: str = (
+            'Flushing stdout to ensure new prompt line!\n'
+        )
+
+        # XXX: yah, mega hack, but how else do we catch this madness XD
+        if (
+            repl.shname == 'xonsh'
+            or (
+                repl.shname == 'bash'
+                and
+                _is_macos
+            )
+        ):
+            flush_status += (
+                '-> ALSO re-flushing due to `xonsh`..\n'
+            )
+            repl.stdout.write(repl.prompt)
+
+        # log.warning(
+        log.devx(
+            flush_status
+        )
+        repl.stdout.flush()
+
+        # TODO: better console UX to match the current "mode":
+        # -[ ] for example if in sticky mode where if there is output
+        #   detected as written to the tty we redraw this part underneath
+        #   and erase the past draw of this same bit above?
+        # repl.sticky = True
+        # repl._print_if_sticky()
+
+        # also see these links for an approach from `ptk`:
+        # https://github.com/goodboy/tractor/issues/130#issuecomment-663752040
+        # https://github.com/prompt-toolkit/python-prompt-toolkit/blob/c2c6af8a0308f9e5d7c0e28cb8a02963fe0ce07a/prompt_toolkit/patch_stdout.py
+    else:
+        log.devx(
+        # log.warning(
+            'Not flushing stdout since not needed?\n'
+            f'|_{repl}\n'
+        )
+
+    # XXX only for tracing this handler
+    log.devx('exiting SIGINT')

tractor/devx/debug/_sync.py

@@ -0,0 +1,220 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3 of
+# the License, or (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+Debugger synchronization APIs to ensure orderly access and
+non-TTY-clobbering graceful teardown.
+
+
+'''
+from __future__ import annotations
+from contextlib import (
+    asynccontextmanager as acm,
+)
+from functools import (
+    partial,
+)
+from typing import (
+    AsyncGenerator,
+    Callable,
+)
+
+from tractor.log import get_logger
+import trio
+from trio.lowlevel import (
+    current_task,
+    Task,
+)
+from tractor._context import Context
+from tractor.runtime._state import (
+    current_actor,
+    debug_mode,
+    is_root_process,
+)
+from ._repl import (
+    TractorConfig as TractorConfig,
+)
+from ._tty_lock import (
+    Lock,
+    request_root_stdio_lock,
+    any_connected_locker_child,
+)
+from ._sigint import (
+    sigint_shield as sigint_shield,
+    _ctlc_ignore_header as _ctlc_ignore_header
+)
+
+log = get_logger()
+
+
+async def maybe_wait_for_debugger(
+    poll_steps: int = 2,
+    poll_delay: float = 0.1,
+    child_in_debug: bool = False,
+
+    header_msg: str = '',
+    _ll: str = 'devx',
+
+) -> bool:  # was locked and we polled?
+
+    if (
+        not debug_mode()
+        and
+        not child_in_debug
+    ):
+        return False
+
+    logmeth: Callable = getattr(log, _ll)
+
+    msg: str = header_msg
+    if (
+        is_root_process()
+    ):
+        # If we error in the root but the debugger is
+        # engaged we don't want to prematurely kill (and
+        # thus clobber access to) the local tty since it
+        # will make the pdb repl unusable.
+        # Instead try to wait for pdb to be released before
+        # tearing down.
+        ctx_in_debug: Context|None = Lock.ctx_in_debug
+        in_debug: tuple[str, str]|None = (
+            ctx_in_debug.chan.uid
+            if ctx_in_debug
+            else None
+        )
+        if in_debug == current_actor().uid:
+            log.debug(
+                msg
+                +
+                'Root already owns the TTY LOCK'
+            )
+            return True
+
+        elif in_debug:
+            msg += (
+                f'Debug `Lock` in use by subactor\n|\n|_{in_debug}\n'
+            )
+            # TODO: could this make things more deterministic?
+            # wait to see if a sub-actor task will be
+            # scheduled and grab the tty lock on the next
+            # tick?
+            # XXX => but it doesn't seem to work..
+            # await trio.testing.wait_all_tasks_blocked(cushion=0)
+        else:
+            logmeth(
+                msg
+                +
+                'Root immediately acquired debug TTY LOCK'
+            )
+            return False
+
+        for istep in range(poll_steps):
+            if (
+                Lock.req_handler_finished is not None
+                and not Lock.req_handler_finished.is_set()
+                and in_debug is not None
+            ):
+                # caller_frame_info: str = pformat_caller_frame()
+                logmeth(
+                    msg
+                    +
+                    '\n^^ Root is waiting on tty lock release.. ^^\n'
+                    # f'{caller_frame_info}\n'
+                )
+
+                if not any_connected_locker_child():
+                    Lock.get_locking_task_cs().cancel()
+
+                with trio.CancelScope(shield=True):
+                    await Lock.req_handler_finished.wait()
+
+                log.devx(
+                    f'Subactor released debug lock\n'
+                    f'|_{in_debug}\n'
+                )
+                break
+
+            # is no subactor locking debugger currently?
+            if (
+                in_debug is None
+                and (
+                    Lock.req_handler_finished is None
+                    or Lock.req_handler_finished.is_set()
+                )
+            ):
+                logmeth(
+                    msg
+                    +
+                    'Root acquired tty lock!'
+                )
+                break
+
+            else:
+                logmeth(
+                    'Root polling for debug:\n'
+                    f'poll step: {istep}\n'
+                    f'poll delya: {poll_delay}\n\n'
+                    f'{Lock.repr()}\n'
+                )
+                with trio.CancelScope(shield=True):
+                    await trio.sleep(poll_delay)
+                    continue
+
+        return True
+
+    # else:
+    #     # TODO: non-root call for #320?
+    #     this_uid: tuple[str, str] = current_actor().uid
+    #     async with acquire_debug_lock(
+    #         subactor_uid=this_uid,
+    #     ):
+    #         pass
+    return False
+
+
+@acm
+async def acquire_debug_lock(
+    subactor_uid: tuple[str, str],
+) -> AsyncGenerator[
+    trio.CancelScope|None,
+    tuple,
+]:
+    '''
+    Request to acquire the TTY `Lock` in the root actor, release on
+    exit.
+
+    This helper is for actor's who don't actually need to acquired
+    the debugger but want to wait until the lock is free in the
+    process-tree root such that they don't clobber an ongoing pdb
+    REPL session in some peer or child!
+
+    '''
+    if not debug_mode():
+        yield None
+        return
+
+    task: Task = current_task()
+    async with trio.open_nursery() as n:
+        ctx: Context = await n.start(
+            partial(
+                request_root_stdio_lock,
+                actor_uid=subactor_uid,
+                task_uid=(task.name, id(task)),
+            )
+        )
+        yield ctx
+        ctx.cancel()

tractor/devx/pformat.py

@@ -15,10 +15,13 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 '''
-Pretty formatters for use throughout the code base.
+Pretty formatters for use throughout our internals.
-Mostly handy for logging and exception message content.
+
+Handy for logging and exception message content but also for `repr()`
+in REPL(s).
 
 '''
+import sys
 import textwrap
 import traceback
 
@@ -115,6 +118,85 @@ def pformat_boxed_tb(
     )
 
 
+def pformat_exc(
+    exc: Exception,
+    header: str = '',
+    message: str = '',
+    body: str = '',
+    with_type_header: bool = True,
+) -> str:
+
+    # XXX when the currently raised exception is this instance,
+    # we do not ever use the "type header" style repr.
+    is_being_raised: bool = False
+    if (
+        (curr_exc := sys.exception())
+        and
+        curr_exc is exc
+    ):
+        is_being_raised: bool = True
+
+    with_type_header: bool = (
+        with_type_header
+        and
+        not is_being_raised
+    )
+
+    # <RemoteActorError( .. )> style
+    if (
+        with_type_header
+        and
+        not header
+    ):
+        header: str = f'<{type(exc).__name__}('
+
+    message: str = (
+        message
+        or
+        exc.message
+    )
+    if message:
+        # split off the first line so, if needed, it isn't
+        # indented the same like the "boxed content" which
+        # since there is no `.tb_str` is just the `.message`.
+        lines: list[str] = message.splitlines()
+        first: str = lines[0]
+        message: str = message.removeprefix(first)
+
+        # with a type-style header we,
+        # - have no special message "first line" extraction/handling
+        # - place the message a space in from the header:
+        #  `MsgTypeError( <message> ..`
+        #                 ^-here
+        # - indent the `.message` inside the type body.
+        if with_type_header:
+            first = f' {first} )>'
+
+        message: str = textwrap.indent(
+            message,
+            prefix=' '*2,
+        )
+        message: str = first + message
+
+    tail: str = ''
+    if (
+        with_type_header
+        and
+        not message
+    ):
+        tail: str = '>'
+
+    return (
+        header
+        +
+        message
+        +
+        f'{body}'
+        +
+        tail
+    )
+
+
 def pformat_caller_frame(
     stack_limit: int = 1,
     box_tb: bool = True,
@@ -144,8 +226,8 @@ def pformat_cs(
     field_prefix: str = ' |_',
 ) -> str:
     '''
-    Pretty format info about a `trio.CancelScope` including most
+    Pretty format info about a `trio.CancelScope` including most of
-    of its public state and `._cancel_status`.
+    its public state and `._cancel_status`.
 
     The output can be modified to show a "var name" for the
     instance as a field prefix, just a simple str before each
@@ -167,3 +249,279 @@ def pformat_cs(
         +
         fields
     )
+
+
+def nest_from_op(
+    input_op: str,  # TODO, Literal of all op-"symbols" from below?
+    text: str,
+    prefix_op: bool = True,  # unset is to suffix the first line
+    # optionally suffix `text`, by def on a newline
+    op_suffix='\n',
+
+    nest_prefix: str = '|_',
+    nest_indent: int|None = None,
+    # XXX indent `next_prefix` "to-the-right-of" `input_op`
+    # by this count of whitespaces (' ').
+    rm_from_first_ln: str|None = None,
+
+) -> str:
+    '''
+    Depth-increment the input (presumably hierarchy/supervision)
+    input "tree string" below the provided `input_op` execution
+    operator, so injecting a `"\n|_{input_op}\n"`and indenting the
+    `tree_str` to nest content aligned with the ops last char.
+
+    '''
+    # `sclang` "structurred-concurrency-language": an ascii-encoded
+    # symbolic alphabet to describe concurrent systems.
+    #
+    # ?TODO? aa more fomal idea for a syntax to the state of
+    # concurrent systems as a "3-domain" (execution, scope, storage)
+    # model and using a minimal ascii/utf-8 operator-set.
+    #
+    # try not to take any of this seriously yet XD
+    #
+    # > is a "play operator" indicating (CPU bound)
+    #   exec/work/ops required at the "lowest level computing"
+    #
+    # execution primititves (tasks, threads, actors..) denote their
+    # lifetime with '(' and ')' since parentheses normally are used
+    # in many langs to denote function calls.
+    #
+    # starting = (
+    # >(  opening/starting; beginning of the thread-of-exec (toe?)
+    # (>  opened/started,  (finished spawning toe)
+    # |_<Task: blah blah..>  repr of toe, in py these look like <objs>
+    #
+    # >) closing/exiting/stopping,
+    # )> closed/exited/stopped,
+    # |_<Task: blah blah..>
+    #   [OR <), )< ?? ]
+    #
+    # ending = )
+    # >c) cancelling to close/exit
+    # c)> cancelled (caused close), OR?
+    #  |_<Actor: ..>
+    #   OR maybe "<c)" which better indicates the cancel being
+    #   "delivered/returned" / returned" to LHS?
+    #
+    # >x)  erroring to eventuall exit
+    # x)>  errored and terminated
+    #  |_<Actor: ...>
+    #
+    # scopes: supers/nurseries, IPC-ctxs, sessions, perms, etc.
+    # >{  opening
+    # {>  opened
+    # }>  closed
+    # >}  closing
+    #
+    # storage: like queues, shm-buffers, files, etc..
+    # >[  opening
+    # [>  opened
+    #  |_<FileObj: ..>
+    #
+    # >]  closing
+    # ]>  closed
+
+    # IPC ops: channels, transports, msging
+    # =>  req msg
+    # <=  resp msg
+    # <=> 2-way streaming (of msgs)
+    # <-  recv 1 msg
+    # ->  send 1 msg
+    #
+    # TODO: still not sure on R/L-HS approach..?
+    # =>(  send-req to exec start (task, actor, thread..)
+    # (<=  recv-req to ^
+    #
+    # (<=  recv-req ^
+    # <=(  recv-resp opened remote exec primitive
+    # <=)  recv-resp closed
+    #
+    # )<=c req to stop due to cancel
+    # c=>) req to stop due to cancel
+    #
+    # =>{  recv-req to open
+    # <={  send-status that it closed
+    #
+    if (
+        nest_prefix
+        and
+        nest_indent != 0
+    ):
+        if nest_indent is not None:
+            nest_prefix: str = textwrap.indent(
+                nest_prefix,
+                prefix=nest_indent*' ',
+            )
+        nest_indent: int = len(nest_prefix)
+
+    # determine body-text indent either by,
+    # - using wtv explicit indent value is provided,
+    # OR
+    # - auto-calcing the indent to embed `text` under
+    #   the `nest_prefix` if provided, **IFF** `nest_indent=None`.
+    tree_str_indent: int = 0
+    if nest_indent not in {0, None}:
+        tree_str_indent = nest_indent
+    elif (
+        nest_prefix
+        and
+        nest_indent != 0
+    ):
+        tree_str_indent = len(nest_prefix)
+
+    indented_tree_str: str = text
+    if tree_str_indent:
+        indented_tree_str: str = textwrap.indent(
+            text,
+            prefix=' '*tree_str_indent,
+        )
+
+    # inject any provided nesting-prefix chars
+    # into the head of the first line.
+    if nest_prefix:
+        indented_tree_str: str = (
+            f'{nest_prefix}{indented_tree_str[tree_str_indent:]}'
+        )
+
+    if (
+        not prefix_op
+        or
+        rm_from_first_ln
+    ):
+        tree_lns: list[str] = indented_tree_str.splitlines()
+        first: str = tree_lns[0]
+        if rm_from_first_ln:
+            first = first.strip().replace(
+                rm_from_first_ln,
+                '',
+            )
+        indented_tree_str: str = '\n'.join(tree_lns[1:])
+
+        if prefix_op:
+            indented_tree_str = (
+                f'{first}\n'
+                f'{indented_tree_str}'
+            )
+
+    if prefix_op:
+        return (
+            f'{input_op}{op_suffix}'
+            f'{indented_tree_str}'
+        )
+    else:
+        return (
+            f'{first}{input_op}{op_suffix}'
+            f'{indented_tree_str}'
+        )
+
+
+# ------ modden.repr ------
+# XXX originally taken verbaatim from `modden.repr`
+'''
+More "multi-line" representation then the stdlib's `pprint` equivs.
+
+'''
+from inspect import (
+    FrameInfo,
+    stack,
+)
+import pprint
+import reprlib
+from typing import (
+    Callable,
+)
+
+
+def mk_repr(
+    **repr_kws,
+) -> Callable[[str], str]:
+    '''
+    Allocate and deliver a `repr.Repr` instance with provided input
+    settings using the std-lib's `reprlib` mod,
+     * https://docs.python.org/3/library/reprlib.html
+
+    ------ Ex. ------
+    An up to 6-layer-nested `dict` as multi-line:
+    - https://stackoverflow.com/a/79102479
+    - https://docs.python.org/3/library/reprlib.html#reprlib.Repr.maxlevel
+
+    '''
+    def_kws: dict[str, int] = dict(
+        indent=3,  # indent used for repr of recursive objects
+        maxlevel=616,  # recursion levels
+        maxdict=616,  # max items shown for `dict`
+        maxlist=616,  # max items shown for `dict`
+        maxstring=616,  # match editor line-len limit
+        maxtuple=616,  # match editor line-len limit
+        maxother=616,  # match editor line-len limit
+    )
+    def_kws |= repr_kws
+    reprr = reprlib.Repr(**def_kws)
+    return reprr.repr
+
+
+def ppfmt(
+    obj: object,
+    do_print: bool = False,
+) -> str:
+    '''
+    The `pprint.pformat()` version of `pprint.pp()`, namely
+    a default `sort_dicts=False`.. (which i think should be
+    the normal default in the stdlib).
+
+    '''
+    pprepr: Callable = mk_repr()
+    repr_str: str = pprepr(obj)
+
+    if do_print:
+        return pprint.pp(repr_str)
+
+    return repr_str
+
+
+pformat = ppfmt
+
+
+def pfmt_frame_info(fi: FrameInfo) -> str:
+    '''
+    Like a std `inspect.FrameInfo.__repr__()` but multi-line..
+
+    '''
+    return (
+        'FrameInfo(\n'
+        '  frame={!r},\n'
+        '  filename={!r},\n'
+        '  lineno={!r},\n'
+        '  function={!r},\n'
+        '  code_context={!r},\n'
+        '  index={!r},\n'
+        '  positions={!r})'
+        ).format(
+            fi.frame,
+            fi.filename,
+            fi.lineno,
+            fi.function,
+            fi.code_context,
+            fi.index,
+            fi.positions
+        )
+
+
+def pfmt_callstack(frames: int = 1) -> str:
+    '''
+    Generate a string of nested `inspect.FrameInfo` objects returned
+    from a `inspect.stack()` call such that only the `.frame` field
+    for each  layer is pprinted.
+
+    '''
+    caller_frames: list[FrameInfo] =  stack()[1:1+frames]
+    frames_str: str = ''
+    for i, frame_info in enumerate(caller_frames):
+        frames_str += textwrap.indent(
+            f'{frame_info.frame!r}\n',
+            prefix=' '*i,
+
+        )
+    return frames_str

tractor/discovery/__init__.py

@@ -0,0 +1,26 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+'''
+Discovery (protocols) API for automatic addressing
+and location management of (service) actors.
+
+NOTE: to avoid circular imports, this ``__init__``
+does NOT eagerly import submodules. Use direct
+module paths like ``tractor.discovery._addr`` or
+``tractor.discovery._discovery`` instead.
+
+'''

tractor/discovery/_addr.py

@@ -0,0 +1,284 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+from __future__ import annotations
+from uuid import uuid4
+from typing import (
+    Protocol,
+    ClassVar,
+    Type,
+    TYPE_CHECKING,
+)
+
+from bidict import bidict
+from trio import (
+    SocketListener,
+)
+
+from ..log import get_logger
+from ..runtime._state import (
+    _def_tpt_proto,
+)
+from ..ipc._tcp import TCPAddress
+from ..ipc._uds import UDSAddress
+
+if TYPE_CHECKING:
+    from ..runtime._runtime import Actor
+
+log = get_logger()
+
+
+# TODO, maybe breakout the netns key to a struct?
+# class NetNs(Struct)[str, int]:
+#     ...
+
+# TODO, can't we just use a type alias
+# for this? namely just some `tuple[str, int, str, str]`?
+#
+# -[ ] would also just be simpler to keep this as SockAddr[tuple]
+#     or something, implying it's just a simple pair of values which can
+#     presumably be mapped to all transports?
+# -[ ] `pydoc socket.socket.getsockname()` delivers a 4-tuple for
+#     ipv6 `(hostaddr, port, flowinfo, scope_id)`.. so how should we
+#     handle that?
+# -[ ] as a further alternative to this wrap()/unwrap() approach we
+#     could just implement `enc/dec_hook()`s for the `Address`-types
+#     and just deal with our internal objs directly and always and
+#     leave it to the codec layer to figure out marshalling?
+#    |_ would mean only one spot to do the `.unwrap()` (which we may
+#       end up needing to call from the hook()s anyway?)
+# -[x] rename to `UnwrappedAddress[Descriptor]` ??
+#    seems like the right name as per,
+#    https://www.geeksforgeeks.org/introduction-to-address-descriptor/
+#
+UnwrappedAddress = (
+    # tcp/udp/uds
+    tuple[
+        str,  # host/domain(tcp), filesys-dir(uds)
+        int|str,  # port/path(uds)
+    ]
+    # ?TODO? should we also include another 2 fields from
+    # our `Aid` msg such that we include the runtime `Actor.uid`
+    # of `.name` and `.uuid`?
+    # - would ensure uniqueness across entire net?
+    # - allows for easier runtime-level filtering of "actors by
+    #   service name"
+)
+
+
+# TODO, maybe rename to `SocketAddress`?
+class Address(Protocol):
+    proto_key: ClassVar[str]
+    unwrapped_type: ClassVar[UnwrappedAddress]
+
+    # TODO, i feel like an `.is_bound()` is a better thing to
+    # support?
+    # Lke, what use does this have besides a noop and if it's not
+    # valid why aren't we erroring on creation/use?
+    @property
+    def is_valid(self) -> bool:
+        ...
+
+    # TODO, maybe `.netns` is a better name?
+    @property
+    def namespace(self) -> tuple[str, int]|None:
+        '''
+        The if-available, OS-specific "network namespace" key.
+
+        '''
+        ...
+
+    @property
+    def bindspace(self) -> str:
+        '''
+        Deliver the socket address' "bindable space" from
+        a `socket.socket.bind()` and thus from the perspective of
+        specific transport protocol domain.
+
+        I.e. for most (layer-4) network-socket protocols this is
+        normally the ipv4/6 address, for UDS this is normally
+        a filesystem (sub-directory).
+
+        For (distributed) network protocols this is normally the routing
+        layer's domain/(ip-)address, though it might also include a "network namespace"
+        key different then the default.
+
+        For local-host-only transports this is either an explicit
+        namespace (with types defined by the OS: netns, Cgroup, IPC,
+        pid, etc. on linux) or failing that the sub-directory in the
+        filesys in which socket/shm files are located *under*.
+
+        '''
+        ...
+
+    @classmethod
+    def from_addr(cls, addr: UnwrappedAddress) -> Address:
+        ...
+
+    def unwrap(self) -> UnwrappedAddress:
+        '''
+        Deliver the underying minimum field set in
+        a primitive python data type-structure.
+        '''
+        ...
+
+    @classmethod
+    def get_random(
+        cls,
+        current_actor: Actor,
+        bindspace: str|None = None,
+    ) -> Address:
+        ...
+
+    # TODO, this should be something like a `.get_def_registar_addr()`
+    # or similar since,
+    # - it should be a **host singleton** (not root/tree singleton)
+    # - we **only need this value** when one isn't provided to the
+    #   runtime at boot and we want to implicitly provide a host-wide
+    #   registrar.
+    # - each rooted-actor-tree should likely have its own
+    #   micro-registry (likely the root being it), also see
+    @classmethod
+    def get_root(cls) -> Address:
+        ...
+
+    def __repr__(self) -> str:
+        ...
+
+    def __eq__(self, other) -> bool:
+        ...
+
+    async def open_listener(
+        self,
+        **kwargs,
+    ) -> SocketListener:
+        ...
+
+    async def close_listener(self):
+        ...
+
+
+_address_types: bidict[str, Type[Address]] = {
+    'tcp': TCPAddress,
+    'uds': UDSAddress
+}
+
+
+# TODO! really these are discovery sys default addrs ONLY useful for
+# when none is provided to a root actor on first boot.
+_default_lo_addrs: dict[
+    str,
+    UnwrappedAddress
+] = {
+    'tcp': TCPAddress.get_root().unwrap(),
+    'uds': UDSAddress.get_root().unwrap(),
+}
+
+
+def get_address_cls(name: str) -> Type[Address]:
+    return _address_types[name]
+
+
+def is_wrapped_addr(addr: any) -> bool:
+    return type(addr) in _address_types.values()
+
+
+def mk_uuid() -> str:
+    '''
+    Encapsulate creation of a uuid4 as `str` as used
+    for creating `Actor.uid: tuple[str, str]` and/or
+    `.msg.types.Aid`.
+
+    '''
+    return str(uuid4())
+
+
+def wrap_address(
+    addr: UnwrappedAddress
+) -> Address:
+    '''
+    Wrap an `UnwrappedAddress` as an `Address`-type based
+    on matching builtin python data-structures which we adhoc
+    use for each.
+
+    XXX NOTE, careful care must be placed to ensure
+    `UnwrappedAddress` cases are **definitely unique** otherwise the
+    wrong transport backend may be loaded and will break many
+    low-level things in our runtime in a not-fun-to-debug way!
+
+    XD
+
+    '''
+    if is_wrapped_addr(addr):
+        return addr
+
+    cls: Type|None = None
+    # if 'sock' in addr[0]:
+    #     import pdbp; pdbp.set_trace()
+    match addr:
+
+        # classic network socket-address as tuple/list
+        case (
+            (str(), int())
+            |
+            [str(), int()]
+        ):
+            cls = TCPAddress
+
+        case (
+            # (str()|Path(), str()|Path()),
+            # ^TODO? uhh why doesn't this work!?
+
+            (_, filename)
+        ) if type(filename) is str:
+            cls = UDSAddress
+
+        # likely an unset UDS or TCP reg address as defaulted in
+        # `_state._runtime_vars['_root_mailbox']`
+        #
+        # TODO? figure out when/if we even need this?
+        case (
+            None
+            |
+            [None, None]
+        ):
+            cls: Type[Address] = get_address_cls(_def_tpt_proto)
+            addr: UnwrappedAddress = cls.get_root().unwrap()
+
+        case _:
+            # import pdbp; pdbp.set_trace()
+            # from tractor.devx import mk_pdb
+            # mk_pdb().set_trace()
+            raise TypeError(
+                f'Can not wrap unwrapped-address ??\n'
+                f'type(addr): {type(addr)!r}\n'
+                f'addr: {addr!r}\n'
+            )
+
+    return cls.from_addr(addr)
+
+
+def default_lo_addrs(
+    transports: list[str],
+) -> list[Type[Address]]:
+    '''
+    Return the default, host-singleton, registry address
+    for an input transport key set.
+
+    '''
+    return [
+        _default_lo_addrs[transport]
+        for transport in transports
+    ]

tractor/discovery/_discovery.py

@@ -28,32 +28,39 @@ from typing import (
 from contextlib import asynccontextmanager as acm
 
 from tractor.log import get_logger
-from .trionics import gather_contexts
+from ..trionics import (
-from ._ipc import _connect_chan, Channel
+    gather_contexts,
-from ._portal import (
+    collapse_eg,
+)
+from ..ipc import _connect_chan, Channel
+from ._addr import (
+    UnwrappedAddress,
+    Address,
+    wrap_address
+)
+from ..runtime._portal import (
     Portal,
     open_portal,
     LocalPortal,
 )
-from ._state import (
+from ..runtime._state import (
     current_actor,
     _runtime_vars,
+    _def_tpt_proto,
 )
 
 if TYPE_CHECKING:
-    from ._runtime import Actor
+    from ..runtime._runtime import Actor
 
 
-log = get_logger(__name__)
+log = get_logger()
 
 
 @acm
 async def get_registry(
-    host: str,
+    addr: UnwrappedAddress|None = None,
-    port: int,
-
 ) -> AsyncGenerator[
-    Portal | LocalPortal | None,
+    Portal|LocalPortal|None,
     None,
 ]:
     '''
@@ -65,35 +72,38 @@ async def get_registry(
     '''
     actor: Actor = current_actor()
     if actor.is_registrar:
-        # we're already the arbiter
+        # we're already the registrar
-        # (likely a re-entrant call from the arbiter actor)
+        # (likely a re-entrant call from the registrar actor)
         yield LocalPortal(
             actor,
-            Channel((host, port))
+            Channel(transport=None)
+            # ^XXX, we DO NOT actually provide nor connect an
+            # underlying transport since this is merely an API shim.
         )
     else:
         # TODO: try to look pre-existing connection from
-        # `Actor._peers` and use it instead?
+        # `Server._peers` and use it instead?
         async with (
-            _connect_chan(host, port) as chan,
+            _connect_chan(addr) as chan,
             open_portal(chan) as regstr_ptl,
         ):
             yield regstr_ptl
 
 
-
 @acm
-async def get_root(
+async def get_root(**kwargs) -> AsyncGenerator[Portal, None]:
-    **kwargs,
+    '''
-) -> AsyncGenerator[Portal, None]:
+    Deliver the current actor's "root process" actor (yes in actor
+    and proc tree terms) by delivering a `Portal` from the spawn-time
+    provided contact address.
 
+    '''
     # TODO: rename mailbox to `_root_maddr` when we finally
     # add and impl libp2p multi-addrs?
-    host, port = _runtime_vars['_root_mailbox']
+    addr = _runtime_vars['_root_mailbox']
-    assert host is not None
 
     async with (
-        _connect_chan(host, port) as chan,
+        _connect_chan(addr) as chan,
         open_portal(chan, **kwargs) as portal,
     ):
         yield portal
@@ -106,17 +116,23 @@ def get_peer_by_name(
 ) -> list[Channel]|None:  # at least 1
     '''
     Scan for an existing connection (set) to a named actor
-    and return any channels from `Actor._peers`.
+    and return any channels from `Server._peers: dict`.
 
     This is an optimization method over querying the registrar for
     the same info.
 
     '''
     actor: Actor = current_actor()
-    to_scan: dict[tuple, list[Channel]] = actor._peers.copy()
+    to_scan: dict[tuple, list[Channel]] = actor.ipc_server._peers.copy()
-    pchan: Channel|None = actor._parent_chan
+
-    if pchan:
+    # TODO: is this ever needed? creates a duplicate channel on actor._peers
-        to_scan[pchan.uid].append(pchan)
+    # when multiple find_actor calls are made to same actor from a single ctx
+    # which causes actor exit to hang waiting forever on
+    # `actor._no_more_peers.wait()` in `_runtime.async_main`
+
+    # pchan: Channel|None = actor._parent_chan
+    # if pchan and pchan.uid not in to_scan:
+    #     to_scan[pchan.uid].append(pchan)
 
     for aid, chans in to_scan.items():
         _, peer_name = aid
@@ -134,24 +150,30 @@ def get_peer_by_name(
 @acm
 async def query_actor(
     name: str,
-    regaddr: tuple[str, int]|None = None,
+    regaddr: UnwrappedAddress|None = None,
 
 ) -> AsyncGenerator[
-    tuple[str, int]|None,
+    tuple[UnwrappedAddress|None, Portal|LocalPortal|None],
     None,
 ]:
     '''
     Lookup a transport address (by actor name) via querying a registrar
     listening @ `regaddr`.
 
-    Returns the transport protocol (socket) address or `None` if no
+    Yields a `tuple` of `(addr, reg_portal)` where,
-    entry under that name exists.
+    - `addr` is the transport protocol (socket) address or `None` if
+      no entry under that name exists,
+    - `reg_portal` is the `Portal` (or `LocalPortal` when the
+      current actor is the registrar) used for the lookup (or
+      `None` when the peer was found locally via
+      `get_peer_by_name()`).
 
     '''
     actor: Actor = current_actor()
     if (
         name == 'registrar'
-        and actor.is_registrar
+        and
+        actor.is_registrar
     ):
         raise RuntimeError(
             'The current actor IS the registry!?'
@@ -159,45 +181,84 @@ async def query_actor(
 
     maybe_peers: list[Channel]|None = get_peer_by_name(name)
     if maybe_peers:
-        yield maybe_peers[0].raddr
+        yield maybe_peers[0].raddr, None
         return
 
-    reg_portal: Portal
+    reg_portal: Portal|LocalPortal
-    regaddr: tuple[str, int] = regaddr or actor.reg_addrs[0]
+    regaddr: Address = wrap_address(regaddr) or actor.reg_addrs[0]
-    async with get_registry(*regaddr) as reg_portal:
+    async with get_registry(regaddr) as reg_portal:
         # TODO: return portals to all available actors - for now
         # just the last one that registered
-        sockaddr: tuple[str, int] = await reg_portal.run_from_ns(
+        addr: UnwrappedAddress = await reg_portal.run_from_ns(
             'self',
             'find_actor',
             name=name,
         )
-        yield sockaddr
+        yield addr, reg_portal
-
 
 @acm
 async def maybe_open_portal(
-    addr: tuple[str, int],
+    addr: UnwrappedAddress,
     name: str,
 ):
+    '''
+    Open a `Portal` to the actor serving @ `addr` or `None` if no
+    peer can be contacted or found.
+
+    '''
     async with query_actor(
         name=name,
         regaddr=addr,
-    ) as sockaddr:
+    ) as (addr, reg_portal):
-        pass
+        if not addr:
+            yield None
+            return
 
-    if sockaddr:
+        try:
-        async with _connect_chan(*sockaddr) as chan:
+            async with _connect_chan(addr) as chan:
                 async with open_portal(chan) as portal:
                     yield portal
+
+        # most likely we were unable to connect the
+        # transport and there is likely a stale entry in
+        # the registry actor's table, thus we need to
+        # instruct it to clear that stale entry and then
+        # more silently (pretend there was no reason but
+        # to) indicate that the target actor can't be
+        # contacted at that addr.
+        except OSError:
+            # NOTE: ensure we delete the stale entry
+            # from the registrar actor when available.
+            if reg_portal is not None:
+                uid: tuple[str, str]|None = await reg_portal.run_from_ns(
+                    'self',
+                    'delete_addr',
+                    addr=addr,
+                )
+                if uid:
+                    log.warning(
+                        f'Deleted stale registry entry !\n'
+                        f'addr: {addr!r}\n'
+                        f'uid: {uid!r}\n'
+                    )
                 else:
+                    log.warning(
+                        f'No registry entry found for addr: {addr!r}'
+                    )
+            else:
+                log.warning(
+                    f'Connection to {addr!r} failed'
+                    f' and no registry portal available'
+                    f' to delete stale entry.'
+                )
             yield None
 
 
 @acm
 async def find_actor(
     name: str,
-    registry_addrs: list[tuple[str, int]]|None = None,
+    registry_addrs: list[UnwrappedAddress]|None = None,
+    enable_transports: list[str] = [_def_tpt_proto],
 
     only_first: bool = True,
     raise_on_none: bool = False,
@@ -207,10 +268,10 @@ async def find_actor(
     None,
 ]:
     '''
-    Ask the arbiter to find actor(s) by name.
+    Ask the registrar to find actor(s) by name.
 
-    Returns a connected portal to the last registered matching actor
+    Returns a connected portal to the last registered
-    known to the arbiter.
+    matching actor known to the registrar.
 
     '''
     # optimization path, use any pre-existing peer channel
@@ -224,15 +285,15 @@ async def find_actor(
         # XXX NOTE: make sure to dynamically read the value on
         # every call since something may change it globally (eg.
         # like in our discovery test suite)!
-        from . import _root
+        from ._addr import default_lo_addrs
         registry_addrs = (
             _runtime_vars['_registry_addrs']
             or
-            _root._default_lo_addrs
+            default_lo_addrs(enable_transports)
         )
 
     maybe_portals: list[
-        AsyncContextManager[tuple[str, int]]
+        AsyncContextManager[UnwrappedAddress]
     ] = list(
         maybe_open_portal(
             addr=addr,
@@ -241,9 +302,12 @@ async def find_actor(
         for addr in registry_addrs
     )
     portals: list[Portal]
-    async with gather_contexts(
+    async with (
+        collapse_eg(),
+        gather_contexts(
             mngrs=maybe_portals,
-    ) as portals:
+        ) as portals,
+    ):
         # log.runtime(
         #     'Gathered portals:\n'
         #     f'{portals}'
@@ -255,7 +319,7 @@ async def find_actor(
         if not any(portals):
             if raise_on_none:
                 raise RuntimeError(
-                    f'No actor "{name}" found registered @ {registry_addrs}'
+                    f'No actor {name!r} found registered @ {registry_addrs!r}'
                 )
             yield None
             return
@@ -274,7 +338,7 @@ async def find_actor(
 @acm
 async def wait_for_actor(
     name: str,
-    registry_addr: tuple[str, int] | None = None,
+    registry_addr: UnwrappedAddress | None = None,
 
 ) -> AsyncGenerator[Portal, None]:
     '''
@@ -291,7 +355,7 @@ async def wait_for_actor(
             yield peer_portal
             return
 
-    regaddr: tuple[str, int] = (
+    regaddr: UnwrappedAddress = (
         registry_addr
         or
         actor.reg_addrs[0]
@@ -299,8 +363,8 @@ async def wait_for_actor(
     # TODO: use `.trionics.gather_contexts()` like
     # above in `find_actor()` as well?
     reg_portal: Portal
-    async with get_registry(*regaddr) as reg_portal:
+    async with get_registry(regaddr) as reg_portal:
-        sockaddrs = await reg_portal.run_from_ns(
+        addrs = await reg_portal.run_from_ns(
             'self',
             'wait_for_actor',
             name=name,
@@ -308,8 +372,8 @@ async def wait_for_actor(
 
         # get latest registered addr by default?
         # TODO: offer multi-portal yields in multi-homed case?
-        sockaddr: tuple[str, int] = sockaddrs[-1]
+        addr: UnwrappedAddress = addrs[-1]
 
-        async with _connect_chan(*sockaddr) as chan:
+        async with _connect_chan(addr) as chan:
             async with open_portal(chan) as portal:
                 yield portal

tractor/discovery/_registry.py

@@ -0,0 +1,253 @@
+# tractor: structured concurrent "actors".
+# Copyright 2018-eternity Tyler Goodlet.
+
+# This program is free software: you can redistribute it and/or
+# modify it under the terms of the GNU Affero General Public
+# License as published by the Free Software Foundation, either
+# version 3 of the License, or (at your option) any later
+# version.
+
+# This program is distributed in the hope that it will be
+# useful, but WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+# PURPOSE. See the GNU Affero General Public License for more
+# details.
+
+# You should have received a copy of the GNU Affero General
+# Public License along with this program. If not, see
+# <https://www.gnu.org/licenses/>.
+
+'''
+Actor-registry for process-tree service discovery.
+
+The `Registrar` is a special `Actor` subtype that serves as
+the process-tree's name-registry, tracking actor
+name-to-address mappings so peers can discover each other.
+
+'''
+from __future__ import annotations
+
+from bidict import bidict
+import trio
+
+from ..runtime._runtime import Actor
+from ._addr import (
+    UnwrappedAddress,
+    Address,
+    wrap_address,
+)
+from ..devx import debug
+from ..log import get_logger
+
+
+log = get_logger('tractor')
+
+
+class Registrar(Actor):
+    '''
+    A special registrar `Actor` who can contact all other
+    actors within its immediate process tree and keeps
+    a registry of others meant to be discoverable in
+    a distributed application.
+
+    Normally the registrar is also the "root actor" and
+    thus always has access to the top-most-level actor
+    (process) nursery.
+
+    By default, the registrar is always initialized when
+    and if no other registrar socket addrs have been
+    specified to runtime init entry-points (such as
+    `open_root_actor()` or `open_nursery()`). Any time
+    a new main process is launched (and thus a new root
+    actor created) and, no existing registrar can be
+    contacted at the provided `registry_addr`, then
+    a new one is always created; however, if one can be
+    reached it is used.
+
+    Normally a distributed app requires at least one
+    registrar per logical host where for that given
+    "host space" (aka localhost IPC domain of addresses)
+    it is responsible for making all other host (local
+    address) bound actors *discoverable* to external
+    actor trees running on remote hosts.
+
+    '''
+    is_registrar = True
+
+    def is_registry(self) -> bool:
+        return self.is_registrar
+
+    def __init__(
+        self,
+        *args,
+        **kwargs,
+    ) -> None:
+
+        self._registry: bidict[
+            tuple[str, str],
+            UnwrappedAddress,
+        ] = bidict({})
+        self._waiters: dict[
+            str,
+            # either an event to sync to receiving an
+            # actor uid (which is filled in once the actor
+            # has sucessfully registered), or that uid
+            # after registry is complete.
+            list[trio.Event|tuple[str, str]]
+        ] = {}
+
+        super().__init__(*args, **kwargs)
+
+    async def find_actor(
+        self,
+        name: str,
+
+    ) -> UnwrappedAddress|None:
+
+        for uid, addr in self._registry.items():
+            if name in uid:
+                return addr
+
+        return None
+
+    async def get_registry(
+        self
+
+    ) -> dict[str, UnwrappedAddress]:
+        '''
+        Return current name registry.
+
+        This method is async to allow for cross-actor
+        invocation.
+
+        '''
+        # NOTE: requires ``strict_map_key=False`` to the
+        # msgpack unpacker since we have tuples as keys
+        # (note this makes the registrar suscetible to
+        # hashdos):
+        # https://github.com/msgpack/msgpack-python#major-breaking-changes-in-msgpack-10
+        return {
+            '.'.join(key): val
+            for key, val in self._registry.items()
+        }
+
+    async def wait_for_actor(
+        self,
+        name: str,
+
+    ) -> list[UnwrappedAddress]:
+        '''
+        Wait for a particular actor to register.
+
+        This is a blocking call if no actor by the
+        provided name is currently registered.
+
+        '''
+        addrs: list[UnwrappedAddress] = []
+        addr: UnwrappedAddress
+
+        mailbox_info: str = (
+            'Actor registry contact infos:\n'
+        )
+        for uid, addr in self._registry.items():
+            mailbox_info += (
+                f'|_uid: {uid}\n'
+                f'|_addr: {addr}\n\n'
+            )
+            if name == uid[0]:
+                addrs.append(addr)
+
+        if not addrs:
+            waiter = trio.Event()
+            self._waiters.setdefault(
+                name, []
+            ).append(waiter)
+            await waiter.wait()
+
+            for uid in self._waiters[name]:
+                if not isinstance(uid, trio.Event):
+                    addrs.append(
+                        self._registry[uid]
+                    )
+
+        log.runtime(mailbox_info)
+        return addrs
+
+    async def register_actor(
+        self,
+        uid: tuple[str, str],
+        addr: UnwrappedAddress
+    ) -> None:
+        uid = name, hash = (
+            str(uid[0]),
+            str(uid[1]),
+        )
+        waddr: Address = wrap_address(addr)
+        if not waddr.is_valid:
+            # should never be 0-dynamic-os-alloc
+            await debug.pause()
+
+        # XXX NOTE, value must also be hashable AND since
+        # `._registry` is a `bidict` values must be unique;
+        # use `.forceput()` to replace any prior (stale)
+        # entries that might map a different uid to the same
+        # addr (e.g. after an unclean shutdown or
+        # actor-restart reusing the same address).
+        self._registry.forceput(uid, tuple(addr))
+
+        # pop and signal all waiter events
+        events = self._waiters.pop(name, [])
+        self._waiters.setdefault(
+            name, []
+        ).append(uid)
+        for event in events:
+            if isinstance(event, trio.Event):
+                event.set()
+
+    async def unregister_actor(
+        self,
+        uid: tuple[str, str]
+
+    ) -> None:
+        uid = (str(uid[0]), str(uid[1]))
+        entry: tuple = self._registry.pop(
+            uid, None
+        )
+        if entry is None:
+            log.warning(
+                f'Request to de-register'
+                f' {uid!r} failed?'
+            )
+
+    async def delete_addr(
+        self,
+        addr: tuple[str, int|str]|list[str|int],
+    ) -> tuple[str, str]|None:
+        # NOTE: `addr` arrives as a `list` over IPC
+        # (msgpack deserializes tuples -> lists) so
+        # coerce to `tuple` for the bidict hash lookup.
+        uid: tuple[str, str]|None = (
+            self._registry.inverse.pop(
+                tuple(addr),
+                None,
+            )
+        )
+        if uid:
+            report: str = (
+                'Deleting registry-entry for,\n'
+            )
+        else:
+            report: str = (
+                'No registry entry for,\n'
+            )
+
+        log.warning(
+            report
+            +
+            f'{addr!r}@{uid!r}'
+        )
+        return uid
+
+
+# Backward compat alias
+Arbiter = Registrar

tractor/experimental/_pubsub.py

@@ -45,6 +45,8 @@ __all__ = ['pub']
 log = get_logger('messaging')
 
 
+# TODO! this needs to reworked to use the modern
+# `Context`/`MsgStream` APIs!!
 async def fan_out_to_ctxs(
     pub_async_gen_func: typing.Callable,  # it's an async gen ... gd mypy
     topics2ctxs: dict[str, list],
@@ -144,7 +146,7 @@ _pubtask2lock: dict[str, trio.StrictFIFOLock] = {}
 
 
 def pub(
-    wrapped: typing.Callable | None = None,
+    wrapped: typing.Callable|None = None,
     *,
     tasks: set[str] = set(),
 ):
@@ -242,8 +244,12 @@ def pub(
         task2lock[name] = trio.StrictFIFOLock()
 
     @wrapt.decorator
-    async def wrapper(agen, instance, args, kwargs):
+    async def wrapper(
-
+        agen,
+        instance,
+        args,
+        kwargs,
+    ):
         # XXX: this is used to extract arguments properly as per the
         # `wrapt` docs
         async def _execute(