goodboy
/
tractor
1
0
Fork 2
Code Issues 5 Pull Requests 6 Packages Projects Releases Wiki Activity

Write the big boi docs content tree 

Replace the ancient `docs/index.rst` (still teaching
`tractor.run()`, `@stream` + arbiters..) with a full ~32 page tree
teaching ONLY the current api (`.wait_for_result()`, registrar
naming, `@context` + `open_context()` as the core model),

- landing: hero example, feature cards + canon links,
- `start/`: install + a 4-example quickstart on-ramp,
- `explain/`: an "SC across processes" essay distilling the essence
  per #157's orig ask + a runtime architecture tour,
- `guide/`: 12 task-focused pages incl the flagship multi-process
  debugging walkthrough, `Context` + `MsgStream` deep-dives,
  cancellation semantics (self-vs-cross cancel rules), discovery,
  infected `asyncio`, typed msging + the #126 testing-tips page,
- `api/`: 10 curated autodoc pages (all targets import-verified vs
  the reorg'd subpkg tree),
- `project/`: changelog include, ported dev-tips (drops old
  `docs/dev_tips.rst`) + roadmap.

Every code block is a `literalinclude` from `examples/`
- zero duplication, all CI-run - w/ `d2` figs floated
into the RHS margin per the 3-col design. Build is green; the 24
remaining warnings all source from lib docstring rst-isms or legacy
`NEWS.rst` content.

Substantially resolves #157 (refine round pending); chips at #175 +
#126.

Prompt-IO: ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Gud Boi 2026-06-11 15:17:14 -04:00
parent 45a8f2d992
commit 3366b1a70d
36 changed files with 5766 additions and 656 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

93
ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md 100644
View File

@ -0,0 +1,93 @@
---
model: claude-fable-5[1m]
service: claude
session: 0780a862-e19a-4f0a-86cd-c8afc0997757
timestamp: 2026-06-11T17:51:52Z
git_ref: 8526985c
scope: docs+code+config
substantive: true
raw_file: 20260611T175152Z_8526985c_prompt_io.raw.md
---
## Prompt
> we need big boi docs; ours are way out of date and generally
> terrible. i'd like to use a sphinx theme in the vein and/or
> one-of/nearly-the-same-as used in one or all of the `msgspec`,
> `numpy`, `ray`, `xonsh`, `polars` projects.
>
> see the following oustanding but now very old issues to guide
> you: #175, #126, #123, #157
>
> we should try to target a 3 columned sphinx theme with,
> - document index-nav on left,
> - body content in middle,
> - diagrams (ideally in d2lang if possible) on RHS
>
> optimize for (if possible),
> - reusing examples code without duplication in docs,
> - generating new examples from todos throughout code base,
> - distilling the essence of SC across processes in the simplest
> and most friendly way possible.
>
> do this work in a /open-wkt and do you best without my guidance
> for the first major pass - we will refine asap once you are
> complete.
Session settings: `/effort max` + ultracode (multi-agent workflow
orchestration). Fully autonomous first pass; two orchestrated
agent fleets were used (6-agent recon survey, then 10-agent
content fan-out: 9 section writers + 1 examples smith), with the
orchestrator authoring the landing page, SC essay, sphinx
scaffold, d2 pipeline + diagram sources, CI workflow and all
integration/fix passes directly.
## Response summary
Complete sphinx docs revamp on branch `wkt/big_boi_docs`:
`pydata-sphinx-theme` 0.18 (3-column: left nav / content /
page-toc, with d2 diagrams + asides floated into the RHS margin
via custom CSS), sphinx 9.1, a local `.. d2::` directive
rendering `docs/diagrams/*.d2` sources with committed-SVG
fallback, a vendored `.. margin::` directive, ~25 new doc pages
(landing, start/, explain/, 12 guides, 10 api-ref pages,
project/), 5 new auto-tested examples + 3 modernized + 1
renamed, and a gh-pages deploy workflow (issue #123). All
example code is `literalinclude`d from `examples/` (zero
duplication, CI-verified). Build: green, 24 warnings all
pre-existing-docstring/NEWS sourced.
## Files changed
See the branch diff (uncommitted at entry-write time):
> `git diff test_cpu_throttling..wkt/big_boi_docs`
> `git -C <wkt> status --short` (pre-commit working tree)
- `docs/conf.py` — full rewrite for pydata theme + ext stack
- `docs/_ext/d2diagrams.py` — new `.. d2::` sphinx directive
- `docs/_ext/marginalia.py` — new `.. margin::` directive
- `docs/_static/css/custom.css` — b&w skin + RHS margin floats
- `docs/diagrams/*.d2` (7) — diagram sources (sketch/grayscale)
- `docs/_diagrams/*.svg` (7) — committed rendered fallbacks
- `docs/index.rst` — new landing (replaces dead-API doc)
- `docs/start/*.rst` (3), `docs/explain/*.rst` (3),
`docs/guide/*.rst` (13), `docs/api/*.rst` (10),
`docs/project/*.rst` (3) — new content tree
- `docs/dev_tips.rst` — removed (ported to project/dev-tips)
- `examples/{typed_payloads,nested_actor_tree,
service_daemon_discovery,uds_transport_actor_tree,
streaming_broadcast_fanout}.py` — new, smoke-tested
- `examples/{a_trynamic_first_scene,
actor_spawning_and_causality,parallelism/single_func}.py` —
`.result()` -> `.wait_for_result()` modernization
- `examples/parallelism/concurrent_futures_primes.py` — renamed
from leading-underscore + trio-runner shim added
- `pyproject.toml` — `docs` dependency-group filled in
- `uv.lock` — relock for docs group
- `.github/workflows/docs.yml` — build + gh-pages deploy
## Human edits
None yet — entry written pre-commit; the author reviews, stages
and commits manually (per repo workflow policy).

49
ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md 100644
View File

@ -0,0 +1,49 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-11T17:51:52Z
git_ref: 8526985c
diff_cmd: git diff test_cpu_throttling..wkt/big_boi_docs
---
# Raw output pointers (diff-ref mode)
All generated content is code/config/docs committed alongside
this entry on branch `wkt/big_boi_docs`; per the prompt-io
diff-ref decision rule each file's verbatim content is the diff
itself rather than a copy here:
> `git diff test_cpu_throttling..wkt/big_boi_docs -- docs/`
> `git diff test_cpu_throttling..wkt/big_boi_docs -- examples/`
> `git diff test_cpu_throttling..wkt/big_boi_docs -- pyproject.toml uv.lock .github/`
## Generation notes (non-code output summary)
- Theme research (web, agent-verified 2026-06-11): msgspec=furo,
xonsh=furo, numpy/ray/polars=pydata-sphinx-theme (ray migrated
off sphinx-book-theme); sphinx-book-theme 1.2.0 hard-pins
pydata 0.16.1 (stale) -> chose pydata 0.18 + sphinx 9.1.
- d2 ecosystem: no production-grade pypi extension exists
(sphinxcontrib-d2lang 0.0.5 ignores returncodes, uuid4 output
names; sphinx-d2 is an empty stub) -> wrote local
`docs/_ext/d2diagrams.py` (~230 LOC) with D2_BIN env
discovery, mtime caching, committed-SVG fallback and
literal-block last resort.
- Diagrams authored in d2 (theme-id 1 "Neutral Grey" + sketch
mode + ELK layout, validated by render + headless-firefox
screenshot loop): actor_tree, context_handshake (real
msg-spec names Start/StartAck/Started/Yield/Stop/Return),
streaming_pipeline, runtime_stack, debug_lock,
error_propagation, infected_aio.
- API truth enforced from a 6-agent recon pass over the
reorganized package tree (runtime/, discovery/, spawn/, ipc/,
msg/, devx/, trionics/): docs teach `.wait_for_result()`,
registrar (not arbiter) naming, `@tractor.context` +
`open_context()` as the core model, `run_in_actor()` as
convenience only.
- All ~30 literalincluded example scripts verified present; 9
touched/new example files smoke-run green (exit 0, <16s).
- Final build: `sphinx-build -b html` succeeded; 24 residual
warnings, every one sourced from pre-existing library
docstring rst-isms or legacy NEWS.rst content (left untouched
by design; flagged for a follow-up docstring lint pass).

134
docs/api/context.rst 100644
View File

@ -0,0 +1,134 @@
Contexts and streaming
======================
The modern core of ``tractor``: a :class:`Context` links a task in
one actor to a task in another as a *single* structured concurrency
(SC) scope stretched across the IPC boundary — errors, results and
cancellation flow between the pair `exactly like trio`_ tasks under
a common `nursery`_. Open one with ``Portal.open_context()`` (see
:func:`tractor._context.open_context_from_portal` in
:doc:`/api/core`), then optionally bridge a bidirectional
:class:`MsgStream` between the two tasks.
.. d2:: diagrams/context_handshake.d2
:caption: The ``open_context()`` <-> ``ctx.started()`` handshake.
:margin:
:alt: parent and child actor context handshake sequence
For the guided, example-driven tour see :doc:`/guide/context`; this
page is the precise API surface.
.. currentmodule:: tractor
The ``@context`` decorator
--------------------------
.. autofunction:: context
.. note::
The decorated function **must** declare a parameter annotated
``tractor.Context`` (any param name works); the runtime injects
the context instance there on each remote invocation. Pass
``pld_spec`` to type-restrict (and validate) the payloads this
endpoint may shuttle — violations raise
:class:`MsgTypeError`. See ``examples/typed_payloads.py``.
``Context``
-----------
.. autoclass:: Context
:members: started,
wait_for_result,
cancel,
cid,
chan,
side,
cancel_called,
cancelled_caught,
cancel_acked,
canceller,
maybe_error,
outcome
.. deprecated:: 0.1.0a6
``Context.result()`` warns; use :meth:`Context.wait_for_result`.
.. note::
A :class:`Context` is **not** a :class:`trio.CancelScope`:
:meth:`Context.cancel` requests cancellation of the *remote*
peer task and does not cancel the local scope. If *you*
requested the cancel, the resulting :class:`ContextCancelled`
is absorbed at ``open_context()`` exit; a cancel originating
anywhere else (the peer, or a third-party actor recorded in
:attr:`ContextCancelled.canceller`) *is* raised locally. This
self-vs-cross-cancel rule is the key to writing correct
inter-actor teardown logic — see :doc:`/guide/context`.
Bidirectional streaming
-----------------------
.. autofunction:: tractor._streaming.open_stream_from_ctx
.. note::
:func:`~tractor._streaming.open_stream_from_ctx` is bound as
the **method-alias** ``Context.open_stream()`` — call it as
``async with ctx.open_stream() as stream:``. Both sides of the
context must enter it for the dialog to be open.
.. autoclass:: MsgStream
:members: send,
receive,
receive_nowait,
aclose,
subscribe,
ctx,
closed
.. note::
A :class:`MsgStream` is one-shot use: once closed it can never
be "re-opened" — open a fresh :class:`Context` instead. Remote
end-of-stream surfaces as :class:`StopAsyncIteration` from
``async for``; un-consumed sends overrun the receiver and raise
:class:`tractor._exceptions.StreamOverrun` unless the context
was opened with ``allow_overruns=True``.
:meth:`MsgStream.subscribe` fans a single IPC stream out to
multiple *local* tasks via a
:class:`tractor.trionics.BroadcastReceiver` (see
:doc:`/api/trionics`); the underlying allocation is idempotent and
non-reversible for the stream's lifetime. See
``examples/streaming_broadcast_fanout.py`` for the pattern in
action.
Legacy one-way streaming
------------------------
.. autofunction:: stream
.. warning::
``@tractor.stream`` and ``Portal.open_stream_from()`` are the
*legacy* one-way streaming API kept for backward compat: a
plain async-generator function streamed parent-ward with no
child-side receive leg. New code should use
``@tractor.context`` + ``ctx.open_stream()`` (bidirectional,
SC-linked, typed). Note ``ctx`` is now a reserved param name
for ``@context`` endpoints — ``@stream`` functions must use
``stream`` instead, and ``ctx.send_yield()`` is deprecated in
favor of :meth:`MsgStream.send`.
.. seealso::
:doc:`/api/errors` for :class:`ContextCancelled` /
:class:`MsgTypeError` semantics, :doc:`/api/msg` for payload
typing via ``pld_spec`` and codecs, :doc:`/api/trionics` for
the broadcast fan-out machinery, and the guided tours in
:doc:`/guide/streaming` + :doc:`/guide/cancellation`.
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning

157
docs/api/core.rst 100644
View File

@ -0,0 +1,157 @@
Runtime and spawning
====================
The core lifecycle API: boot the runtime in your root process,
spawn trio-"actors" (processes running ``trio.run()`` task trees)
under a one-cancels-all supervisor, and talk to them through
portals. This is structured concurrency (SC) applied *transitively*:
every spawned process is owned by a nursery block and errors
`always propagate`_. If you can create zombies it **is a bug**.
.. currentmodule:: tractor
Booting the runtime
-------------------
.. autofunction:: open_root_actor
.. note::
The env vars ``TRACTOR_LOGLEVEL`` and ``TRACTOR_SPAWN_METHOD``
override the ``loglevel`` / ``start_method`` params so you can
crank verbosity or swap spawn backends without touching app
code. Exactly **one** IPC transport may be enabled per actor
(see ``enable_transports`` and :doc:`/api/ipc`).
.. autofunction:: run_daemon
Spawning actors
---------------
.. d2:: diagrams/actor_tree.d2
:caption: A supervised actor (process) tree.
:margin:
:alt: root actor supervising a tree of subactors
.. autofunction:: open_nursery
.. autoclass:: ActorNursery
:members: start_actor,
run_in_actor,
cancel,
cancel_called,
cancelled_caught
.. note::
:meth:`ActorNursery.start_actor` (daemon actor + portal) is the
blessed spawning primitive; pair it with
``Portal.open_context()`` for SC-linked remote tasks.
:meth:`ActorNursery.run_in_actor` is a *convenience* one-shot —
spawn, run a single task, auto-cancel after the result — slated
to be rebuilt as a high-level wrapper, so don't design around
it as the core model.
.. deprecated:: 0.1.0a6
``ActorNursery.cancelled`` warns; use
:attr:`ActorNursery.cancel_called` and
:attr:`ActorNursery.cancelled_caught`. The ``rpc_module_paths``
kwarg is likewise deprecated in favor of ``enable_modules``.
Portals
-------
A :class:`Portal` "opens a portal" into a peer actor's memory
domain: you call functions and start SC-linked tasks *over IPC* as
though they were local, with results, errors and cancellation
flowing back `exactly like trio`_.
.. autoclass:: Portal
:members: run,
run_from_ns,
open_stream_from,
wait_for_result,
cancel_actor,
chan
.. deprecated:: 0.1.0a6
``Portal.result()`` warns; use :meth:`Portal.wait_for_result`.
The str-form ``Portal.run('mod.path', 'fn_name')`` also warns;
pass a function *object* whose module is listed in the target's
``enable_modules``. ``Portal.channel`` is the legacy spelling
of :attr:`Portal.chan`.
.. autofunction:: tractor._context.open_context_from_portal
.. note::
:func:`~tractor._context.open_context_from_portal` is bound as
the **method-alias** ``Portal.open_context()`` — that's the
spelling you should actually call:
``portal.open_context(fn, **kwargs)``. See :doc:`/api/context`
for the full ``Context`` + ``MsgStream`` API it unlocks.
.. note::
:meth:`Portal.cancel_actor` cancels the *whole* remote runtime
and process (machine-level), not a single task — use
:meth:`Context.cancel` for task-level cancellation. Pass
``raise_on_timeout=True`` to get an ``ActorTooSlowError`` you
can escalate per SC discipline (see :doc:`/api/errors`).
Clusters
--------
.. autofunction:: open_actor_cluster
Spawn a *flat* cluster of ``count`` worker actors (default: one
per core) all serving the RPC ``modules`` list, yielding a
``dict[str, Portal]`` keyed by actor name. Handy for
embarrassingly parallel fan-out; see ``examples/quick_cluster.py``.
Runtime introspection
---------------------
.. autofunction:: current_actor
.. autoclass:: Actor
:members: aid,
name,
uid,
is_registrar,
is_infected_aio,
cancel_soon
.. note::
:class:`Actor` is the per-process runtime singleton (msg loop,
RPC scheduling, IPC server) — you never instantiate it yourself
and should normally only touch the identity/introspection
surface listed above. The canonical identity type is
:attr:`Actor.aid` (a ``tractor.msg.Aid`` struct);
:attr:`Actor.uid` is the legacy ``(name, uuid)`` 2-tuple which
is still pervasive in logs and error metadata.
.. deprecated:: 0.1.0a6
``Actor.is_arbiter`` warns; use :attr:`Actor.is_registrar`.
The ``arbiter_addr`` constructor kwarg is deprecated for
``registry_addrs``.
.. autofunction:: current_ipc_ctx
.. autofunction:: is_root_process
.. autofunction:: get_runtime_vars
.. seealso::
:doc:`/api/context` for the SC-linked remote task API,
:doc:`/api/discovery` for finding actors by name, and the
guided tours in :doc:`/guide/spawning`, :doc:`/guide/rpc` and
:doc:`/guide/context`.
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics

92
docs/api/devx.rst 100644
View File

@ -0,0 +1,92 @@
Debugging and devx: ``tractor.devx``
====================================
Multi-process debugging that actually works: boot the tree with
``open_root_actor(debug_mode=True)`` (or pass it to
``open_nursery()``) and any crash or explicit pause in *any* actor
acquires a tree-global TTY lock and drops you into a
`pdbp`_-powered REPL — one actor at a time, ``SIGINT`` shielded,
no garbled terminals. The top-level helpers below are the daily
drivers; the rest of the toolbox lives under ``tractor.devx``.
.. currentmodule:: tractor
Pausing and post-mortems
------------------------
.. autofunction:: pause
.. autofunction:: pause_from_sync
.. note::
:func:`pause_from_sync` needs the `greenback`_ portal: boot
with ``open_root_actor(maybe_enable_greenback=True)`` (mind
the `performance implications`_). With ``debug_mode`` on, the
built-in ``breakpoint()`` is also remapped to a
``tractor``-safe equivalent.
.. autofunction:: post_mortem
.. deprecated:: 0.1.0a6
``tractor.breakpoint()`` warns and simply calls :func:`pause`
— use :func:`tractor.pause` (async) or
:func:`tractor.pause_from_sync` in new code.
Crash handling for CLIs and sync entrypoints
--------------------------------------------
.. currentmodule:: tractor.devx
.. autofunction:: open_crash_handler
.. autofunction:: maybe_open_crash_handler
.. note::
Both are *sync* context managers usable before (or without)
``trio.run()`` — wrap your CLI ``main()`` to get a post-mortem
REPL on any uncaught exception instead of a bare traceback.
Runtime hang-hunting
--------------------
.. autofunction:: enable_stack_on_sig
With stackscope_ integration enabled (also via
``open_root_actor(enable_stack_on_sig=True)`` or the
``TRACTOR_ENABLE_STACKSCOPE`` env var) a ``SIGUSR1`` triggers a
full trio task-tree dump from every actor — works on live,
*non*-debug-mode trees too:
.. code:: sh
pkill --signal SIGUSR1 -f <part-of-your-cmd>
Dumps also tee to ``/tmp/tractor-stackscope-<pid>.log`` so you
still get output under captured/CI stdio.
Lower-level debug plumbing
--------------------------
.. autofunction:: mk_pdb
.. autofunction:: maybe_wait_for_debugger
:func:`maybe_wait_for_debugger` is mainly useful in runtime/test
code that must avoid tearing down a tree while a child still
holds the global debug lock.
.. seealso::
:doc:`/api/errors` for the boxed error types you'll inspect
from the REPL, :doc:`/api/core` for the ``debug_mode`` /
``maybe_enable_greenback`` / ``enable_stack_on_sig`` boot
flags on :func:`tractor.open_root_actor`, and
:doc:`/guide/debugging` for the guided multi-actor REPL tour.
.. _pdbp: https://github.com/mdmintz/pdbp
.. _greenback: https://greenback.readthedocs.io/en/latest/
.. _performance implications: https://greenback.readthedocs.io/en/latest/principle.html#performance
.. _stackscope: https://github.com/oremanj/stackscope

67
docs/api/discovery.rst 100644
View File

@ -0,0 +1,67 @@
Discovery and the registrar
===========================
Every actor registers its ``(name, uuid)`` and transport addresses
with a *registrar* actor — by default the root of the tree, or
whichever actor serves at the ``registry_addrs`` you boot with.
The discovery API lets any actor look up any other **by name** and
get back a connected :class:`~tractor.Portal`, giving you
service-discovery patterns (daemons, service trees, multi-host
meshes) without hard-coding addresses.
Lookups first scan already-connected peers before RPC-ing the
registrar, and multihomed results are ranked UDS > local TCP >
remote TCP. See ``examples/service_daemon_discovery.py`` for the
canonical daemon + lookup pattern.
.. currentmodule:: tractor
Lookup APIs
-----------
.. autofunction:: find_actor
.. autofunction:: wait_for_actor
.. autofunction:: query_actor
.. autofunction:: get_registry
.. note::
:func:`find_actor` yields ``None`` when nothing is registered
under the name (or raises with ``raise_on_none=True``);
:func:`wait_for_actor` blocks until the name appears;
:func:`query_actor` only *looks up* the address without
connecting to the target.
The ``Registrar``
-----------------
.. autoclass:: Registrar
:show-inheritance:
A :class:`Registrar` is just an :class:`~tractor.Actor` subtype
maintaining the name -> addresses table; you rarely touch it
directly beyond passing ``registry_addrs`` /
``ensure_registry=True`` to :func:`~tractor.open_root_actor`.
Check :attr:`Actor.is_registrar <tractor.Actor.is_registrar>` to
ask "am I it?".
Legacy ``Arbiter`` alias
------------------------
.. deprecated:: 0.1.0a6
``tractor.Arbiter`` survives only as a class alias of
:class:`Registrar` and all "arbiter" terminology is replaced
by "registrar"/"registry" across the API: ``get_arbiter()`` is
removed (use :func:`get_registry`) and the ``arbiter_addr``
kwarg is replaced by ``registry_addrs``.
.. seealso::
:doc:`/api/core` for booting a registrar via
``open_root_actor(registry_addrs=...)``, :doc:`/api/ipc` for
the transport/address model the registry stores, and
:doc:`/guide/discovery` for the worked walkthrough.

102
docs/api/errors.rst 100644
View File

@ -0,0 +1,102 @@
Errors and cancellation types
=============================
``tractor`` extends trio's "exceptions `always propagate`_" rule
across the process boundary: a crash in any actor is serialized as
an ``Error`` msg, shuttled over IPC, and re-raised in the linked
parent scope as a *boxed* :class:`RemoteActorError` — preserving
the original type, traceback text and source-actor identity, even
across multi-hop relays (a.k.a. "inceptions").
The most-used types below are importable from ``tractor``
directly; the remainder live in ``tractor._exceptions`` (not yet
re-exported at top level).
.. currentmodule:: tractor
Boxed remote errors
-------------------
.. autoexception:: RemoteActorError
:members: boxed_type,
src_uid,
relay_uid,
pformat
.. code:: python
try:
async with portal.open_context(ep_fn) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # remote task raised a `ValueError`
.. autoexception:: ContextCancelled
:show-inheritance:
:members: canceller
.. note::
Inspect :attr:`ContextCancelled.canceller` (the requesting
actor's uid) to distinguish a *self*-requested cancel (absorbed
at ``open_context()`` exit) from a *cross*-actor cancel (raised
locally) — the full rules live in :doc:`/api/context`.
Typed-messaging errors
----------------------
.. autoexception:: MsgTypeError
:show-inheritance:
:members: bad_msg,
expected_msg_type
An "IPC ``TypeError``": a message failed validation against the
active msg-spec / ``pld_spec`` (see :doc:`/api/msg`). Raised
sender-side for control msgs (``Started``/``Return``) and
receiver-side for stream ``Yield`` payloads.
.. autoexception:: tractor._exceptions.StreamOverrun
:show-inheritance:
The sender out-paced the receiver's buffer on a
:class:`~tractor.MsgStream` opened without
``allow_overruns=True``; subtypes :class:`trio.TooSlowError`.
Transport and runtime errors
----------------------------
.. autoexception:: TransportClosed
.. autoexception:: ModuleNotExposed
:show-inheritance:
Raised when an RPC requests a function from a module not listed
in the target actor's ``enable_modules`` allowlist —
capability-style access control, not an import bug on your end ;)
.. autoexception:: tractor._exceptions.NoRuntime
:show-inheritance:
Raised by :func:`tractor.current_actor` (and friends) when no
actor runtime is up in the current process.
.. autoexception:: tractor._exceptions.ActorTooSlowError
:show-inheritance:
A peer actor failed to ack a cancel request within the bounded
wait — the SC-sanctioned escalation signal from APIs like
``Portal.cancel_actor(raise_on_timeout=True)``. Catch it to
escalate (e.g. hard-kill via the supervising
:class:`~tractor.ActorNursery`); never just ignore it, that's how
zombies happen.
.. seealso::
:doc:`/api/context` for how cancellation and errors flow
through a :class:`~tractor.Context`, :doc:`/api/devx` for
crash-handling REPL tooling (``debug_mode``, post-mortems),
and :doc:`/guide/cancellation` for the full SC-cancellation
story.
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate

67
docs/api/index.rst 100644
View File

@ -0,0 +1,67 @@
API reference
=============
This is the curated reference for ``tractor``'s public surface: the
names you can import and lean on without reading runtime internals.
Everything below is re-exported at the top level (``import
tractor``) unless a page says otherwise; subsystems like
``tractor.msg``, ``tractor.trionics``, ``tractor.to_asyncio``,
``tractor.devx`` and ``tractor.log`` are importable as submodules.
``tractor`` is "just trio_" extended across processes: every API
here is designed to keep the structured concurrency (SC) rules you
already know from the `trio docs`_ intact across the process
boundary. If a name isn't documented here it's an internal — expect
it to change without notice B).
.. currentmodule:: tractor
Most-used names at a glance:
.. autosummary::
:nosignatures:
open_root_actor
open_nursery
run_daemon
ActorNursery
Portal
context
Context
MsgStream
open_actor_cluster
find_actor
wait_for_actor
get_registry
current_actor
current_ipc_ctx
is_root_process
get_runtime_vars
RemoteActorError
ContextCancelled
MsgTypeError
pause
post_mortem
Channel
.. toctree::
:maxdepth: 1
:caption: Reference pages
core
context
discovery
errors
msg
trionics
to_asyncio
devx
ipc
Where to next? If you're new, start with the runtime and spawning
APIs in :doc:`/api/core`, then graduate to the inter-actor task
linking model in :doc:`/api/context` — it's the heart of the whole
system.
.. _trio: https://github.com/python-trio/trio
.. _trio docs: https://trio.readthedocs.io/en/latest/

89
docs/api/ipc.rst 100644
View File

@ -0,0 +1,89 @@
IPC and logging
===============
Under every portal, context and stream sits a per-peer
:class:`~tractor.Channel`: a msgpack-typed messaging link wrapping
one OS transport connection. Transports are pluggable per actor
via ``enable_transports=['tcp' | 'uds']`` — TCP is the default,
UDS (unix domain sockets) gives you port-less, same-host IPC with
kernel-provided peer credentials for free — and exactly **one**
transport may currently be enabled per actor.
.. d2:: diagrams/runtime_stack.d2
:caption: Where ``Channel`` sits in the runtime stack.
:margin:
:alt: layered runtime stack from app code down to transports
Addresses are "unwrapped" tuples at the API edges:
``('host', port)`` for TCP, filesystem-path pairs for UDS. For
the full layering story — transport protocols, the IPC server,
address types and the msg loop — see
:doc:`/explain/architecture`.
.. currentmodule:: tractor
``Channel``
-----------
.. autoclass:: Channel
:members: from_addr,
send,
recv,
aclose,
connected,
apply_codec,
aid,
laddr,
raddr,
closed
.. deprecated:: 0.1.0a6
``Channel.uid`` warns; use :attr:`Channel.aid` which carries
richer (optional) identity fields beyond the legacy
``(name, uuid)`` pair.
.. note::
You rarely construct a :class:`Channel` yourself — the runtime
hands them out via :attr:`Portal.chan <tractor.Portal.chan>`
and :attr:`Context.chan <tractor.Context.chan>`. Treat the
send/recv surface as advanced API: normal apps should speak
:class:`~tractor.MsgStream` instead.
Choosing a transport
--------------------
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
:caption: examples/uds_transport_actor_tree.py
:language: python
Logging
-------
``tractor.log`` provides the structured, colorized console
logging used across the runtime — with actor-name + task-aware
record headers and extra log levels below :data:`logging.DEBUG`
(``'transport'``, ``'runtime'``, ``'cancel'``, ``'devx'``) for
spelunking the runtime itself. Use it for your app too: it's
already distributed-system aware.
.. currentmodule:: tractor.log
.. autofunction:: get_logger
.. autofunction:: get_console_log
.. note::
The ``TRACTOR_LOGLEVEL`` env var overrides any caller-passed
``loglevel`` (e.g. to ``open_root_actor()``) so you can crank
console verbosity without touching code; subactors inherit
the root's level by default.
.. seealso::
:doc:`/explain/architecture` for the transport/server
internals, :doc:`/api/discovery` for how channel addresses
get registered and found, and :doc:`/api/msg` for the codec
layer every channel speaks.

107
docs/api/msg.rst 100644
View File

@ -0,0 +1,107 @@
Typed messaging: ``tractor.msg``
================================
All inter-actor communication rides a small, strictly-typed
msgpack wire protocol built from :class:`msgspec.Struct` types —
the "SC-shuttle" protocol that powers contexts, streams, RPC and
cancellation. You normally never touch these msg types directly
(the :class:`~tractor.Context` API speaks them for you) but you
*do* use this subpackage to define **payload type contracts**: per
endpoint via ``@tractor.context(pld_spec=...)`` or per channel via
custom codecs.
Violations of an active msg-spec surface as
:class:`~tractor.MsgTypeError` (see :doc:`/api/errors`); the full
typed-payload workflow is shown in ``examples/typed_payloads.py``.
.. currentmodule:: tractor.msg
The protocol message set
------------------------
.. autosummary::
:nosignatures:
PayloadMsg
Aid
SpawnSpec
Start
StartAck
Started
Yield
Stop
Return
CancelAck
Error
``Aid`` (identity handshake) and ``SpawnSpec`` (parent -> child
init) run at connection setup; ``Start``/``StartAck`` initiate an
RPC task; ``Started``/``Yield``/``Stop``/``Return`` are the
:class:`~tractor.Context` dialog phases; ``CancelAck`` and
``Error`` close the loop on cancellation and (boxed) failure.
``Msg`` is a legacy alias of ``PayloadMsg``. The union of all of
the above is exported as ``MsgType`` (also ``__msg_spec__``).
.. automodule:: tractor.msg.types
:no-members:
.. currentmodule:: tractor.msg
Codec construction and override
-------------------------------
.. autofunction:: mk_codec
.. autoclass:: MsgCodec
:members: encode,
decode,
msg_spec
.. autofunction:: mk_dec
.. autoclass:: MsgDec
:members: decode,
spec
.. autofunction:: apply_codec
.. autofunction:: current_codec
.. note::
:func:`apply_codec` swaps the codec via a
:class:`contextvars.ContextVar` — the override only applies to
the *current task* (and tasks it starts), not sibling tasks
already running in the actor. Payload-decoding is layered: the
outer codec leaves ``.pld`` fields as ``msgspec.Raw`` and each
context's payload-receiver decodes them against *its* spec
(the "cheap-or-nasty" validation pattern).
Namespace pointers
------------------
.. autoclass:: NamespacePath
:members: from_ref,
load_ref,
to_tuple
The ``'module.path:obj_name'`` :class:`str`-subtype used to
address every RPC target function over the wire (same format as
:func:`pkgutil.resolve_name`).
Pretty structs
--------------
.. autoclass:: Struct
:show-inheritance:
A :class:`msgspec.Struct` subtype with a multi-line pretty
``__repr__`` — handy as a base for your own IPC payload types so
crash logs stay readable.
.. seealso::
:doc:`/api/context` for where ``pld_spec`` typing plugs into
the ``@context`` decorator, :doc:`/api/errors` for
:class:`~tractor.MsgTypeError` semantics, and
:doc:`/guide/msging` for the guided typed-messaging tour.

97
docs/api/to_asyncio.rst 100644
View File

@ -0,0 +1,97 @@
asyncio interop: ``tractor.to_asyncio``
=======================================
"Infected asyncio" mode: spawn an actor with
``start_actor(..., infect_asyncio=True)`` and its process runs
:mod:`trio` as a `guest`_ on top of the :mod:`asyncio` loop —
letting your trio task tree drive asyncio tasks *in the same
process* while the rest of the actor tree stays pure trio. Each
trio <-> asyncio task pair is linked with structured concurrency
(SC) semantics: error or cancellation on either side tears down
both, with the cause translated cross-loop.
.. d2:: diagrams/infected_aio.d2
:caption: A trio guest driving asyncio tasks in one actor.
:margin:
:alt: trio guest mode inside an asyncio-infected actor
See ``examples/infected_asyncio_echo_server.py`` for a complete
worked example.
.. currentmodule:: tractor.to_asyncio
Starting asyncio tasks from trio
--------------------------------
.. autofunction:: open_channel_from
.. autofunction:: run_task
.. note::
:func:`open_channel_from` mirrors the
``Portal.open_context()`` handshake: the asyncio side calls
``chan.started_nowait(value)`` and that value pops out as
``first`` on the trio side. :func:`run_task` is the one-shot
form — run a single asyncio-compatible coroutine fn and return
its result to trio.
The inter-loop channel
----------------------
.. autoclass:: LinkedTaskChannel
:members: send,
receive,
wait_for_result,
subscribe,
started_nowait,
send_nowait,
get,
cancel_asyncio_task,
closed
.. note::
The trio side uses the async API
(:meth:`LinkedTaskChannel.send` /
:meth:`LinkedTaskChannel.receive`); the asyncio side uses the
loop-safe sync/await mix
(:meth:`LinkedTaskChannel.send_nowait` /
:meth:`LinkedTaskChannel.get` /
:meth:`LinkedTaskChannel.started_nowait`).
Translated exception types
--------------------------
Cross-loop failures are re-raised on the *other* side as one of
these explicit translation types, so you always know which loop
actually died first:
.. autoexception:: tractor._exceptions.AsyncioCancelled
.. autoexception:: tractor._exceptions.AsyncioTaskExited
.. autoexception:: tractor._exceptions.TrioCancelled
.. autoexception:: tractor._exceptions.TrioTaskExited
.. autoexception:: AsyncioRuntimeTranslationError
:show-inheritance:
Guest-mode entrypoint
---------------------
``run_as_asyncio_guest()`` is the runtime-internal entrypoint that
boots trio in `guest`_ mode inside an infected actor — you get it
implicitly via ``infect_asyncio=True`` and shouldn't need to call
it yourself.
.. seealso::
:doc:`/api/core` for the ``infect_asyncio`` spawn flag,
:meth:`tractor.Actor.is_infected_aio` for runtime
introspection, :doc:`/api/devx` for using the debugger from
inside asyncio tasks, and :doc:`/guide/asyncio` for the
guided tour.
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops

76
docs/api/trionics.rst 100644
View File

@ -0,0 +1,76 @@
Trio patterns: ``tractor.trionics``
===================================
Sugary structured concurrency (SC) patterns for plain :mod:`trio`
code — **no actor runtime required**. These helpers grew out of
real distributed-system needs in ``tractor`` apps but every one of
them works in a single-process program too; import via
``from tractor import trionics``.
.. currentmodule:: tractor.trionics
Context-manager helpers
-----------------------
.. autofunction:: gather_contexts
.. autofunction:: maybe_open_context
.. autofunction:: maybe_open_nursery
.. note::
:func:`gather_contexts` is "a nursery for async context
managers": it enters N acms concurrently and yields their
values in input order. :func:`maybe_open_context` is the
actor-wide cache/multiplex layer on top — the first task pays
the acm setup cost, later callers get ``(cache_hit=True, ...)``
and share the same value until all users exit.
Broadcast fan-out
-----------------
.. autofunction:: broadcast_receiver
.. autoclass:: BroadcastReceiver
:members: receive,
subscribe,
aclose
.. autoexception:: Lagged
:show-inheritance:
A single-producer, many-consumer broadcast layer over any
``trio``-style receive channel: non-lossy for the *fastest*
consumer while slower consumers raise :class:`Lagged` (a
:class:`trio.TooSlowError` subtype) once they fall behind the
internal ring. This is exactly the machinery behind
:meth:`tractor.MsgStream.subscribe` — see
``examples/streaming_broadcast_fanout.py``.
ExceptionGroup helpers
----------------------
.. autofunction:: collapse_eg
.. autofunction:: maybe_raise_from_masking_exc
.. note::
:func:`collapse_eg` "un-nests" single-exception
:class:`ExceptionGroup` wrappers from strict-eg ``trio``
nurseries so your ``except`` clauses match the original error;
:func:`maybe_raise_from_masking_exc` surfaces real errors that
would otherwise be masked by :class:`trio.Cancelled` during
teardown.
.. seealso::
:doc:`/api/context` for the IPC-stream consumer of
:class:`BroadcastReceiver`, :doc:`/guide/streaming` for
fan-out in a worked pipeline, and the `trio docs`_ for the
underlying channel and `nursery`_ semantics these helpers
compose.
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning

51
docs/dev_tips.rst
View File

@ -1,51 +0,0 @@
Hot tips for ``tractor`` hackers
================================
This is a WIP guide for newcomers to the project mostly to do with
dev, testing, CI and release gotchas, reminders and best practises.
``tractor`` is a fairly novel project compared to most since it is
effectively a new way of doing distributed computing in Python and is
much closer to working with an "application level runtime" (like erlang
OTP or scala's akka project) then it is a traditional Python library.
As such, having an arsenal of tools and recipes for figuring out the
right way to debug problems when they do arise is somewhat of
a necessity.
Making a Release
----------------
We currently do nothing special here except the traditional
PyPa release recipe as in `documented by twine`_. I personally
create sub-dirs within the generated `dist/` with an explicit
release name such as `alpha3/` when there's been a sequence of
releases I've made, but it really is up to you how you like to
organize generated sdists locally.
The resulting build cmds are approximately:
.. code:: bash
python setup.py sdist -d ./dist/XXX.X/
twine upload -r testpypi dist/XXX.X/*
twine upload dist/XXX.X/*
.. _documented by twine: https://twine.readthedocs.io/en/latest/#using-twine
Debugging and monitoring actor trees
------------------------------------
TODO: but there are tips in the readme for some terminal commands
which can be used to see the process trees easily on Linux.
Using the log system to trace `trio` task flow
----------------------------------------------
TODO: the logging system is meant to be oriented around
stack "layers" of the runtime such that you can track
"logical abstraction layers" in the code such as errors, cancellation,
IPC and streaming, and the low level transport and wire protocols.

368
docs/explain/architecture.rst 100644
View File

@ -0,0 +1,368 @@
Anatomy of the runtime
======================
You can get a long way with ``tractor`` by treating it as "trio_
with nurseries that spawn processes". But once you start asking
*where does my msg actually go?*, *which process is that?* or
*who keeps the phonebook?*, it pays to know how the runtime hangs
together. This page walks the stack top to bottom.
.. d2:: diagrams/runtime_stack.d2
:caption: The four runtime layers inside *every* actor process.
:alt: layer cake of app tasks, tractor runtime, IPC, OS process
:width: 70%
The layer cake
--------------
Every actor process is the same four-layer sandwich:
- **your app**: plain ``trio`` tasks, nurseries and cancel
scopes; nothing special. ``tractor`` is a `structured
concurrency`_ (SC) multi-processing runtime built on trio_ and
the whole pitch is that this layer stays *just trio*: no
callbacks, no futures, no proxy objects.
- **the** ``tractor`` **runtime**: a per-process
:class:`tractor.Actor` running the msg loop and RPC task
scheduler, plus the user-facing primitives layered on it:
:class:`tractor.ActorNursery` (spawning + supervision),
:class:`tractor.Portal` (calling into a peer) and
:class:`tractor.Context` + :class:`tractor.MsgStream`
(SC-linked cross-actor task pairs and streaming).
- **IPC channels**: one :class:`tractor.Channel` per connected
peer, each wrapping a ``MsgTransport`` that ships
msgspec_-typed msgs over TCP or UDS.
- **the OS**: one process per actor, started by a swappable
spawn backend.
The property that holds it all together: SC composes *through*
the layers. A crash in a leaf actor's app task unwinds that
actor's trio tree, ships across its IPC channel as a typed
``Error`` msg, and unwinds the parent's trio tree in turn — the
"SC-transitive supervision protocol" from the README's pitch.
The whole tree cancels and errors like one big trio program; it
just happens to be spread across processes.
One actor, one process, one ``trio.run()``
------------------------------------------
A ``tractor`` "actor" is not a green thread, nor an object with
a mailbox, nor a coroutine: it's one OS process running one
:func:`trio.run` whose root task boots the runtime machinery —
msg loop, RPC task scheduler, IPC server — all embodied by a
single :class:`tractor.Actor` instance.
.. margin:: Shared nothing
Processes buy you a real `shared nothing architecture`_: no
accidentally-shared mutable state, no GIL contention, and
every actor can be inspected (or killed) like any other OS
process.
You rarely construct an :class:`~tractor.Actor` yourself; the
runtime makes exactly one per process and you grab it with
:func:`tractor.current_actor`:
.. code:: python
import tractor
actor = tractor.current_actor() # NoRuntime if none running
print(actor.aid.name) # str name, need not be unique
print(actor.aid.uuid) # uuid4 str, IS unique
print(actor.aid.pid) # the OS pid
print(actor.uid) # legacy (name, uuid) pair
Identity is carried by the ``Aid`` msg-struct (see
``tractor.msg.types``): a ``name``/``uuid``/``pid`` triple
exchanged in the very first "mailbox handshake" whenever two
actors connect. It's what the registrar stores and what shows up
in logs and proc-titles. The older ``.uid`` 2-tuple of
``(name, uuid)`` predates ``Aid`` and is still pervasive across
the codebase; treat it as the legacy spelling of the same
identity.
If this smells like the `actor model`_, sure — but as the
README warns, it probably doesn't look like what you *think* an
actor model looks like, and that's intentional. Here an "actor"
is purely a runtime-unit-of-abstraction: process +
``trio.run()`` + IPC machinery.
IPC: channels, transports, addresses
------------------------------------
Two connected actors talk through a :class:`tractor.Channel`: a
duplex, per-peer msg pipe. Each ``Channel`` wraps a
``MsgTransport`` instance which does the wire work: framing,
encode/decode and the socket itself. The encoding is msgpack
(via msgspec_) and *every* msg is an instance of one of the
runtime's tagged-union :class:`msgspec.Struct` types: the
``Aid`` handshake, ``Start``/``StartAck`` (RPC init),
``Started``/``Yield``/``Stop``/``Return`` (the ctx dialog
phases), ``Error``, etc. There is no raw-bytes mode; the
msg-spec *is* the protocol, which is exactly what lets payloads
be type-limited per-context (see ``pld_spec`` in
:doc:`/guide/context`).
Addresses come in two spellings:
- *unwrapped*: the plain-tuple form you pass to user APIs —
``('127.0.0.1', 1616)`` for tcp, or a
``(<filedir>, <filename>)`` path-pair for uds;
- *wrapped*: the internal ``TCPAddress``/``UDSAddress`` struct
types (plus libp2p-style multiaddr helpers over in
``tractor.discovery``).
You only ever need the tuple form; the runtime wraps and
unwraps at the boundaries.
TCP: the boring default
***********************
The default transport (``'tcp'``) binds each actor's IPC server
to loopback ``('127.0.0.1', <random port>)`` unless told
otherwise, and is the only choice when your tree spans hosts.
Nothing exotic: ``trio`` TCP streams + length-prefixed msgpack
framing.
UDS: same-host, creds included
******************************
Pass ``enable_transports=['uds']`` and actors instead talk over
unix-domain sockets, with socket files placed in the per-user
runtime dir (``$XDG_RUNTIME_DIR/tractor/`` on linux, the
``platformdirs`` equivalent elsewhere). Two perks over tcp on a
single host:
- no ports to fight over; addrs are just file paths,
- the kernel snitches on your peer for free: the listening side
reads the connector's ``pid`` (plus ``uid``/``gid`` on linux)
straight off the socket via ``SO_PEERCRED`` /
``LOCAL_PEERPID`` — no extra handshake msgs required B)
.. warning::
Socket-file lifetime == listening actor lifetime. On
listener teardown the runtime ``os.unlink()``\s the socket
file immediately, so any *late* connection attempt (say, a
sub-actor racing to deregister with a registrar that's
already shutting down) fails with ``FileNotFoundError``.
And ofc, UDS is same-host only.
Here's a full actor tree run entirely over uds:
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
:caption: examples/uds_transport_actor_tree.py
:language: python
Picking a transport
*******************
Transport choice is per-actor via the ``enable_transports``
kwarg accepted by :func:`tractor.open_root_actor` (and proxied
through ``open_nursery()`` when it implicitly boots the
runtime), plus per-child via
``ActorNursery.start_actor(enable_transports=...)``. Two rules
the runtime enforces today:
- exactly ONE transport per actor: multi-transport actors are
on the roadmap but currently raise ``RuntimeError``;
- your ``registry_addrs`` protos must all be in
``enable_transports``: mismatches fail fast with
``ValueError`` instead of (as in darker times) hanging the
registrar handshake forever.
Spawn backends
--------------
How does an actor actually *become* a process? Via a swappable
spawn backend, selected with the ``start_method`` kwarg to
:func:`tractor.open_root_actor`:
``'trio'`` (default)
The home-grown spawner: re-exec the child as
``python -m tractor._child`` using ``trio``'s subprocess
machinery, then bootstrap it over the first IPC exchange
(the parent ships a ``SpawnSpec`` msg carrying all init
state). Supported on all platforms and the most battle
tested choice by far.
``'mp_spawn'`` / ``'mp_forkserver'``
The stdlib :mod:`multiprocessing` start-methods of the same
names (forkserver is posix-only). Mostly interesting for
ecosystem compat and start-up-latency tuning.
``'subint'`` (experimental, py3.14+)
Each actor runs as a `PEP 734`_ sub-interpreter
(``concurrent.interpreters``) driven on its own OS thread
*inside the parent process*: interpreter-level
shared-nothing isolation with much faster start-up. Yes,
this bends the one-actor-one-process rule; the rest of the
model is unchanged.
The ``TRACTOR_SPAWN_METHOD`` env-var beats any caller-passed
``start_method``, so you can swap backends under an unmodified
app:
.. code:: bash
TRACTOR_SPAWN_METHOD=mp_forkserver python my_app.py
One current limitation worth knowing: ``debug_mode=True`` (the
crash-to-REPL machinery) is only supported on backends whose
child-side runtime is trio-native, e.g. the default ``'trio'``;
see :doc:`/guide/debugging` for the deats.
The registrar
-------------
Discovery needs a phonebook. Every actor, as part of boot,
registers its ``Aid`` and bind-addrs with the *registrar*: an
otherwise ordinary actor (a :class:`tractor.Registrar`, subtype
of :class:`~tractor.Actor`) that keeps the name -> addrs table
for the tree; on graceful exit each actor de-registers itself.
.. margin:: Default registry addrs
With no ``registry_addrs`` passed:
- tcp: ``('127.0.0.1', 1616)``
- uds: ``registry@1616.sock``
in the runtime dir
Who *is* the registrar? Decided at root boot, rendezvous style.
:func:`tractor.open_root_actor` probes each addr in
``registry_addrs`` with a quick connect-ping, then:
- **somebody answered**: this root is a plain actor; it
registers with the existing registrar and binds random
same-proto addrs for its own IPC server;
- **nobody answered**: this root *becomes* the registrar and
binds the registry addrs itself.
So single-program trees need zero config — the root quietly
self-appoints — while multi-program setups share a registrar by
pointing every program at the same ``registry_addrs``. Pass
``ensure_registry=True`` to demand that *this* call create the
registry; it raises if the addrs are already served.
The lookup APIs — :func:`tractor.find_actor`,
:func:`tractor.wait_for_actor`, :func:`tractor.query_actor` and
:func:`tractor.get_registry` — all consult it (after first
checking already-connected peers):
.. literalinclude:: ../../examples/service_daemon_discovery.py
:caption: examples/service_daemon_discovery.py
:language: python
If you bump into "arbiter" in old issues or posts: that's the
legacy name for the same thing, surviving in-code only as the
``Arbiter = Registrar`` class alias; all current terminology is
"registrar"/"registry". Fair warning per the README: this is
still a **very naive** discovery sys (no re-election, no gossip
protocol... yet) and a registrar is expected to outlive its
registrants.
Runtime env vars
----------------
A few env-vars let you re-tune a whole tree *without touching
app code*; each wins over its corresponding kwarg:
.. list-table::
:header-rows: 1
:widths: 30 44 26
* - env-var
- effect
- vs. kwarg
* - ``TRACTOR_LOGLEVEL``
- crank (or silence) console-log verbosity for every actor
in the tree
- beats ``loglevel``
* - ``TRACTOR_SPAWN_METHOD``
- swap the process spawn backend
- beats ``start_method``
* - ``TRACTOR_ENABLE_STACKSCOPE``
- install the ``SIGUSR1`` task-tree-dump handler in every
actor, even outside ``debug_mode`` (see
:doc:`/guide/debugging`)
- OR'd with ``enable_stack_on_sig``
Spotting actors from your shell
*******************************
Every sub-actor sets an OS-level proc-title of the form
``_subactor[<name>@<pid>]`` (via ``setproctitle``, silently
skipped when not installed) so ``ps``/``htop``/``pstree`` show
*which actor is which* at a glance. The README's signature
incantation — watch a tree build and self-destruct live:
.. code:: bash
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/nested_actor_tree.py \
&& kill $!
For scripting there are two stable cmdline markers:
.. code:: bash
pgrep -fa '_subactor\[' # live, titled sub-actors
pgrep -fa 'tractor._child' # 'trio'-backend children not
# yet (re)titled
The title also lands in the kernel ``comm`` (truncated to ~15
bytes) which survives into zombie state — that's what
``tractor``'s own test-harness reapers key off. To be crystal
clear about the contract though: you should never *need* a
reaper; if you can create zombie child processes (without using
a system signal) it **is a bug** — please report it!
Logging
-------
The runtime logs through a thin adapter over stdlib
:mod:`logging` that stamps every record with actor + task info.
Two calls get you going:
.. code:: python
from tractor.log import get_console_log, get_logger
log = get_logger(__name__) # actor/task-aware sub-logger
get_console_log('info') # attach console handler @ level
(or just pass ``loglevel='info'`` to
:func:`tractor.open_root_actor` and the console handler comes up
with the runtime).
``tractor`` adds custom levels — and matching logger methods —
that slot between the stdlib ones so you can dial in *which
runtime subsystem* you want to hear from: ``.transport()`` (5),
``.runtime()`` (15), ``.devx()`` (17), ``.cancel()`` (22), plus
a ``PDB`` (500) level for debugger chatter. E.g.
``loglevel='cancel'`` plays the whole cancellation chorus while
staying quiet about transport-layer noise. Beyond that
``tractor`` isn't opinionated about how you consume logs: it's
all stdlib ``logging`` underneath.
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` — the SC-linked cross-actor task API
that rides on every ``Channel``.
- :doc:`/guide/debugging` — ``debug_mode``, the multi-process
REPL and ``stackscope`` task-tree dumps.
- :doc:`/explain/sc-distributed` — *why*
one-actor-one-process, and what kind of "actor model" this
is (and isn't).
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _trio: https://github.com/python-trio/trio
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
.. _msgspec: https://jcristharif.com/msgspec/
.. _PEP 734: https://peps.python.org/pep-0734/

16
docs/explain/index.rst 100644
View File

@ -0,0 +1,16 @@
Big ideas
=========
The conceptual core of ``tractor``: what *structured
concurrency* (SC) means once your "tasks" are whole
processes, and how the runtime is layered to deliver
that without (much) magic.
If you only read one page in these docs make it
:doc:`sc-distributed`; if you read two, follow it with
:doc:`architecture`.
.. toctree::
:maxdepth: 2
sc-distributed
architecture

207
docs/explain/sc-distributed.rst 100644
View File

@ -0,0 +1,207 @@
Structured concurrency, across processes
=========================================
``tractor`` makes one bet: the discipline that made
``trio``'s concurrency *readable* — `structured
concurrency`_ (SC) — works just as well when the
"tasks" are whole OS processes talking over a wire.
This page distills what that means, from first
principles, with as little ceremony as possible.
.. margin:: The canon
If SC is new to you, the seminal `blog post`_ is
still the best hour you'll spend on concurrent
programming; the `trio docs`_, wikipedia's SC_
page and the diagrams over at libdill-docs_ round
it out nicely.
SC in one breath
----------------
Structured concurrency is the rule that **concurrency
gets a scope**: every task is spawned *inside* a block
(a ``trio`` *nursery*) and that block **cannot exit
until every task it spawned has finished** — returned,
errored, or been cancelled.
That one rule buys you the properties you already
rely on in sequential code,
- a function call is a *black box*: when it returns,
everything it started is **done** — no secret
background tasks leaking out the sides,
- an exception **always has somewhere to go**: up the
(task) tree to a parent which is, by construction,
still there waiting,
- cancellation has a well defined *shape*: cancel a
scope and it flows down to every task inside it,
and only those.
In short: your **runtime task tree matches your source
code's indentation**. Concurrency you can read.
The leap: process-shaped tasks
------------------------------
Now swap "task" for "process".
A ``tractor`` *actor* is just a Python process running
its own ``trio.run()`` — its own private task tree,
sharing **nothing** with its siblings. You spawn
actors from an :class:`tractor.ActorNursery`, which
behaves exactly the way the name implies,
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.start_actor(
'worker',
enable_modules=[__name__],
)
...
# ^ block exit == every spawned process has
# completed, errored or been cancelled, and
# been **reaped**. No exceptions, no zombies.
so the whole program becomes a *tree of process-trees*
— a `supervision tree`_ in erlang-speak — where every
arrow means "spawned by, **waited on by**, and
supervised by".
.. d2:: diagrams/actor_tree.d2
:caption: A ``tractor`` program: a process tree of
``trio`` task trees; every parent **must wait**
on its children.
:width: 85%
Causality: no process outlives its parent
-----------------------------------------
The stdlib's ``multiprocessing`` (and most "job
queue" systems) treat child processes as
fire-and-forget by default: orphans, zombies, lost
tracebacks and ``kill -9`` cleanup scripts are *your*
problem. ``tractor`` instead inherits ``trio``'s
`causality`_ discipline,
- **no spawning willy-nilly**: every actor is born
from a nursery block with a known parent,
- **lifetimes nest**: a sub-actor's entire process
tree lives strictly inside its parent's nursery
scope,
- **teardown is guaranteed**: when a scope exits (or
errors, or is cancelled) the runtime SIGINTs,
waits, and (only if it must) hard-kills + reaps
everything underneath.
We take the zombie thing personally: *if you can
create orphaned child processes without using a
system signal, it* **is a bug** — and there's a test
suite to back that sentence up.
Errors always propagate (yes, across the wire)
----------------------------------------------
In ``trio``, an exception in any task tears through
its nursery to a parent that must handle it —
`exceptions always propagate`_. ``tractor`` extends
the same guarantee across process boundaries: an
uncaught error in a remote task is
1. captured + serialized in the child,
2. shipped home over IPC as a typed ``Error`` msg,
3. re-raised in the parent **boxed** as a
:class:`tractor.RemoteActorError` carrying the
original type (``.boxed_type``), a rendered remote
traceback, and the erroring actor's id,
while the supervising nursery applies its (currently
*one-cancels-all*, just like ``trio``) strategy to any
sibling actors. A crash three processes deep arrives
at your shell as one coherent, causal traceback chain
— not a silent dead worker and a stuck queue.
Cancellation is a request, supervision is the rule
--------------------------------------------------
Cancellation likewise keeps ``trio``'s semantics
*verbatim*, just transported: cancelling an actor
nursery (or a single :class:`tractor.Context` between
two tasks in different processes) sends an explicit
cancel **request** over IPC which the remote runtime
translates into a real ``trio`` cancel-scope cancel —
then *acks back* so the requester can await
confirmation within a bounded time. Nothing is ever
"just killed" first; graceful always precedes brutal.
Because every cross-process dialog is a pair of
**linked tasks** — one on each side, each inside its
own cancel scope — SC stays *transitive*: supervision
doesn't stop at the process boundary, it tunnels
through every hop of the tree. The wire protocol that
enforces this (a small set of typed msgs:
``Start``/``Started``/``Yield``/``Stop``/``Return``/
``Error``) is detailed in :doc:`/guide/msging` and
:doc:`/guide/context`.
Hold up, is this an "actor model"?
----------------------------------
Let's stop and ask how many canon actor model papers
you've actually read ;)
From `the author's mouth`_, the **only** requirement
is `adherence to`_ the `3 axioms`_::
In response to a message, an actor may:
- send a finite number of new messages
- create a finite number of new actors
- designate a new behavior to process subsequent
messages
``tractor`` adheres — actors exchange msgs, spawn
actors, and swap behaviors — **with no extra API** to
learn. What we *don't* copy is the cultural baggage:
no visible mailboxes, no untyped fire-and-forget
``send()``, no "let it crash" without a supervisor
that actually hears about it, and definitely no
shared-reference *proxy objects* pretending the
network isn't there. If our "actors" don't look like
what you expected, that's **intentional**: being an
actor model is just one property of the system; being
*structured* is the point.
Why processes at all?
---------------------
Python has a GIL; an actor model by definition shares
no state; so the *process* is the natural runtime
unit — you get real multi-core parallelism and hard
memory isolation for free. But the deeper win is
uniformity: because actors only ever talk via msgs
over a :class:`tractor.Channel` (TCP, UDS, more to
come), the **same code** runs your laptop's worker
pool and a multi-host cluster; "distributed" is a
deployment detail, not an API.
It's just ``trio``
------------------
If you remember one framing, make it this: ``tractor``
**is just** ``trio`` — with nurseries that can spawn
processes and streams that can cross them. Same
nursery discipline, same cancellation semantics, same
"how was this not always the API?" feeling, one level
up the process tree.
.. seealso::
:doc:`/explain/architecture` for how the runtime
layers deliver all of the above, and
:doc:`/start/quickstart` to feel it in ~20 lines of
code.
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
.. _supervision tree: https://www.erlang.org/doc/design_principles/des_princ.html
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _exceptions always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _the author's mouth: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s
.. _adherence to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts

309
docs/guide/asyncio.rst 100644
View File

@ -0,0 +1,309 @@
Infected ``asyncio``
====================
``tractor`` is "just trio_", but the Python world is packed with
libraries that only speak ``asyncio``: websocket stacks, vendor
SDKs, that one exchange client you can't route around. Rather than
make you rewrite them, ``tractor`` lets you *quarantine* them inside
a dedicated subactor which runs both event loops at once, with full
`structured concurrency`_ (SC) guarantees maintained across the
loop boundary *and* the process tree.
In the project's own words:
Yes, we spawn a python process, run ``asyncio``, start ``trio``
on the ``asyncio`` loop, then send commands to the ``trio``
scheduled tasks to tell ``asyncio`` tasks what to do XD
We call this "infected ``asyncio``" mode: the subactor's stdlib
loop runs as the *host* with ``trio`` embedded on top in `guest`_
mode, and your ``trio`` tasks drive ``asyncio`` tasks through
a linked, SC-supervised, in-memory channel.
.. d2:: diagrams/infected_aio.d2
:caption: One process, two schedulers: ``trio`` rides the
``asyncio`` loop as a guest while the parent speaks plain
``tractor`` IPC, none the wiser.
:alt: parent actor connected over IPC to a subactor whose
asyncio loop hosts trio in guest mode, with a
LinkedTaskChannel pairing a trio task to an asyncio task
.. note::
Infected ``asyncio`` mode is **experimental**: it works (we
beat on it plenty) but parts of the API surface and some
edge-case semantics are still settling. Got opinions on the
interop design? Feel free to sling them in `#273`_!
How the infection takes hold
----------------------------
A normal subactor boots by running the ``tractor`` runtime's task
tree directly under ``trio.run()``. Pass ``infect_asyncio=True``
at spawn time and the child's entrypoint changes shape entirely:
1. the process starts the stdlib loop via ``asyncio.run()``,
2. the first ``asyncio`` task calls
``trio.lowlevel.start_guest_run()``, embedding the ``trio``
scheduler *inside* the already running ``asyncio`` loop (the
upstream `guest`_-mode feature),
3. the regular ``tractor`` runtime then boots on the guest
``trio`` side and connects back to its parent like any other
subactor.
.. margin:: Symptoms
Looks like your stdlib event loop has caught a case of "the
trios"! Don't worry, you'll barely notice; and if anything
gets too bad, your parents will know about it B)
Both schedulers interleave in a single thread, no GIL gymnastics
required. From the rest of the actor tree the infected child is
indistinguishable from any other actor: same IPC protocol, same
supervision and cancellation semantics, same zombie-safety
guarantees. The difference is purely internal: ``trio`` tasks in
that process can start and drive ``asyncio`` tasks through the
``tractor.to_asyncio`` API.
Spawning an infected subactor
-----------------------------
Just flip the flag on :meth:`tractor.ActorNursery.start_actor`:
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.start_actor(
'aio_side',
enable_modules=[__name__],
infect_asyncio=True,
)
The one-shot convenience ``ActorNursery.run_in_actor()`` accepts
the same flag. The ``to_asyncio`` APIs may **only** be called from
tasks inside an infected actor; calling them anywhere else raises
a loud ``RuntimeError``. You can introspect at runtime with
``tractor.current_actor().is_infected_aio()``.
Linking tasks with ``open_channel_from()``
------------------------------------------
The core primitive is :func:`tractor.to_asyncio.open_channel_from`,
an async context manager which starts your ``asyncio`` function as
a real ``asyncio.Task`` and yields a two-way channel linking it to
the calling ``trio`` task:
.. code:: python
from tractor import to_asyncio
async with to_asyncio.open_channel_from(
aio_main, # async def aio_main(chan, **kwargs)
period=0.5, # extra kwargs are passed through
) as (chan, first):
await chan.send('tick')
The semantics deliberately mirror the inter-actor ``Context``
handshake from :doc:`/guide/context`:
- the target fn must declare a parameter literally named ``chan``;
the runtime injects the shared
:class:`~tractor.to_asyncio.LinkedTaskChannel` by keyword.
- the ``trio`` side blocks at entry until the ``asyncio`` task
calls ``chan.started_nowait(value)``; that value is delivered as
``first``, exactly like the ``(ctx, first)`` pair you get from
``Portal.open_context()`` after the child calls
``ctx.started()``.
- a first value **must** be sent from the ``asyncio`` side or the
``trio`` side will never unblock.
- on block exit the pair is torn down *together*; neither task can
outlive the other (more on this below).
A full example: the echo server
-------------------------------
Here's the canonical demo, a round-trip echo service where the
``asyncio`` task is told what to do by a ``trio`` task which is in
turn driven over IPC by the root actor:
.. literalinclude:: ../../examples/infected_asyncio_echo_server.py
:caption: examples/infected_asyncio_echo_server.py
:language: python
What's going on?
- there are three task layers: the root actor's pure ``trio``
task, the infected child's ``trio``-side ``@tractor.context``
endpoint (``trio_to_aio_echo_server()``), and the child's
``asyncio`` task (``aio_echo_server()``).
- two ``started``-style handshakes compose: the aio task's
``chan.started_nowait('start')`` unblocks the child's
``open_channel_from()`` entry, then the child relays that same
value up via ``await ctx.started(first)`` which unblocks the
root's ``open_context()`` entry. Synchronization all the way
down, er, up.
- each round trip flows: root ``stream.send()`` -> IPC -> child
``async for msg in stream`` -> ``chan.send(msg)`` -> aio
``await chan.get()`` -> ``chan.send_nowait()`` -> child
``chan.receive()`` -> ``stream.send(out)`` -> IPC -> root.
- when the root breaks out of its stream loop and exits the
context block, the child's stream ends, its channel block exits,
and the ``asyncio`` task is reaped along with it; the final
``portal.cancel_actor()`` then tears down the whole process. No
orphaned ``asyncio`` tasks, no zombie procs; if you manage to
create either it **is a bug**.
``LinkedTaskChannel``: one channel, two sides
---------------------------------------------
The same channel object is shared by both tasks; which methods you
call depends on which loop schedules your task. The ``trio`` side
gets a standard ``trio.abc.Channel`` interface while the
``asyncio`` side gets queue-flavored, mostly-sync methods:
.. list-table::
:header-rows: 1
:widths: 14 36 50
* - side
- call
- what it does
* - ``trio``
- ``await chan.send(item)``
- ship ``item`` to the ``asyncio`` task (enqueues onto an
internal ``asyncio.Queue``).
* - ``trio``
- ``await chan.receive()``
- wait for the next value from the ``asyncio`` side; the
channel also supports ``async for``.
* - ``trio``
- ``await chan.wait_for_result()``
- block until the ``asyncio`` task completes; return its
final result or raise its (translated) error.
* - ``trio``
- ``chan.subscribe()``
- acm yielding a ``BroadcastReceiver`` so N local tasks can
each consume a copy of the inbound stream (see below).
* - ``trio``
- ``chan.cancel_asyncio_task()``
- explicitly request cancellation of the linked ``asyncio``
task.
* - ``asyncio``
- ``chan.started_nowait(value)``
- deliver the "first" value; unblocks the ``trio`` side's
``open_channel_from()`` entry (mirrors ``ctx.started()``).
* - ``asyncio``
- ``await chan.get()``
- wait for the next value sent from the ``trio`` side.
* - ``asyncio``
- ``chan.send_nowait(item)``
- push a value to the ``trio`` side without blocking.
Fan-out with ``.subscribe()``
*****************************
Just like :meth:`tractor.MsgStream.subscribe` does for IPC
streams, ``chan.subscribe()`` lets multiple local ``trio`` tasks
each receive *every* value sent from the single ``asyncio`` task:
.. code:: python
async with chan.subscribe() as bcast:
async for msg in bcast:
...
The underlying broadcast machinery is lazily allocated on first
use and is *not* reversible for the channel's remaining lifetime,
so only reach for it when you actually want the fan-out.
One-shot calls with ``run_task()``
----------------------------------
When you just want a single ``asyncio`` result and no streaming
dialog, skip the channel ceremony and use
:func:`tractor.to_asyncio.run_task`:
.. code:: python
import asyncio
from tractor import to_asyncio
async def aio_fetch(url: str) -> str:
await asyncio.sleep(0.3) # pretend-IO, aio style
return f'<html>sup {url}</html>'
# from any trio task inside the infected actor:
page = await to_asyncio.run_task(aio_fetch, url='https://x.io')
It schedules the fn as an ``asyncio.Task``, waits for completion
and hands the return value back to ``trio``; think of it as the
cross-loop sibling of ``ActorNursery.run_in_actor()``. Errors and
cancellation are translated exactly as for channels.
Cross-loop errors and cancellation
----------------------------------
The paired tasks are *SC linked*: exception and cancel handling
tears down **both** sides on any unexpected error or cancellation,
in either loop. There is no fire-and-forget mode; a
``LinkedTaskChannel`` is a supervision scope just like a
``Context`` is across processes.
Because each loop has its own (incompatible) cancellation and exit
machinery, boundary crossings are translated into dedicated
exception types, all importable from ``tractor.to_asyncio``:
.. list-table::
:header-rows: 1
:widths: 26 22 52
* - exception
- raised in
- meaning
* - ``AsyncioCancelled``
- the ``trio`` task
- the linked ``asyncio`` task was cancelled by itself or
a 3rd party (i.e. *not* by the ``trio`` side).
* - ``AsyncioTaskExited``
- the ``trio`` task
- the ``asyncio`` task returned/exited early while the
``trio`` side still held the link open.
* - ``TrioCancelled``
- the ``asyncio`` task
- the ``trio`` side was cancelled (or crashed) so the
``asyncio`` task is being torn down per SC rules.
* - ``TrioTaskExited``
- the ``asyncio`` task
- the ``trio`` side exited gracefully while the ``asyncio``
task was still running; a "clean shutdown" signal much
like closing a ``trio`` mem-chan.
By default ``open_channel_from(suppress_graceful_exits=True)``
absorbs the two ``*TaskExited`` signals so happy-path teardown
stays silent; pass ``False`` when your app wants to handle early
peer-exit explicitly.
Past the task pair, everything composes with the normal actor
story: an unhandled ``asyncio`` error is translated into the
``trio`` side, propagates out of your ``@tractor.context``
endpoint, and arrives at the parent boxed as
a :class:`tractor.RemoteActorError`. One SC discipline,
end-to-end, across loops *and* processes.
Breakpoints in ``asyncio`` tasks
--------------------------------
Yes, the multi-actor REPL works here too. With
``debug_mode=True`` enabled on your tree the ``trio`` side of an
infected actor can ``await tractor.pause()`` as usual, and with
greenback enabled (``maybe_enable_greenback=True``) even the
builtin ``breakpoint()`` works from *inside* ``asyncio`` tasks;
see ``examples/debugging/asyncio_bp.py`` for the full tour. The
root-TTY locking dance behind all this is covered in
:doc:`/guide/debugging`.
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` for the inter-actor handshake and
streaming APIs which this whole interop layer mirrors.
- :doc:`/guide/msging` for typing the payloads you shuttle
between actors (and loops).
- :doc:`/guide/debugging` for the multi-process REPL that
keeps working even when your loop has "the trios".
.. _trio: https://github.com/python-trio/trio
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
.. _#273: https://github.com/goodboy/tractor/issues/273

285
docs/guide/cancellation.rst 100644
View File

@ -0,0 +1,285 @@
Cancellation and error propagation
==================================
``tractor`` supports ``trio``'s cancellation_ system *verbatim*,
then extends it across process boundaries. If you know how to
cancel a task in ``trio`` you already know how to cancel an actor —
and its whole subtree — in ``tractor``; the runtime's job is making
that statement hold over IPC with every structured concurrency (SC)
guarantee intact.
The ground rules,
- a remote actor is **never** cancelled unless explicitly requested
(by a parent or peer), unless supervision demands it (an error
triggered one-cancels-all teardown), or unless there's a bug in
``tractor`` itself (please report it!),
- (remote) errors `always propagate`_ back to the parent
supervisor; nothing is silently dropped on the floor,
- every spawned process gets reaped no matter how it dies; if you
can create a zombie child process (without using a system signal)
it **is a bug**.
``trio`` cancellation, across the wire
--------------------------------------
Locally everything is bog-standard ``trio``: nurseries, cancel
scopes, timeouts. ``tractor`` adds exactly one twist: a cancel
scope can't physically reach into another process, so the runtime
*relays cancellation as messages*. Concretely,
- cancelling an *actor* means sending it a runtime-cancel request
msg; the target then runs its own graceful teardown — cancelling
RPC tasks, closing channels, exiting its :func:`trio.run` — and
acks the request back to the canceller,
- cancelling a single *cross-actor task* works through the
:class:`tractor.Context` layer: each ``ctx`` task-pair is
cancel-scope-linked over IPC such that either side erroring or
cancelling relays an equivalent error to the other side (see
:doc:`/guide/context` for the gory details),
- a cancel is therefore always a *request with an ack*: the
canceller does a **bounded wait** for confirmation and escalates
if the peer is unresponsive (see the teardown ladder below).
One-cancels-all supervision
---------------------------
An :class:`tractor.ActorNursery` supervises subactors `exactly like
trio`_ nurseries supervise tasks: when one child errors, the error
propagates to the supervising block and **all** sibling subactors
get cancelled before the error continues bubbling up the (process)
tree.
.. d2:: diagrams/error_propagation.d2
:caption: One-cancels-all: no zombies, no lost errors.
:alt: error propagation up a subactor tree
:width: 80%
.. literalinclude:: ../../examples/remote_error_propagation.py
:caption: examples/remote_error_propagation.py
:language: python
What's going on here?
- three healthy actors are spawned as daemons via
:meth:`tractor.ActorNursery.start_actor`; left alone they'd
happily idle forever,
- a fourth actor runs ``assert_err()`` via ``.run_in_actor()`` and
promptly trips its ``assert 0``,
- the resulting ``AssertionError`` ships back over IPC as a
serialized error msg and re-raises *boxed* inside the nursery
block as a :class:`tractor.RemoteActorError`,
- the nursery reacts like any ``trio`` nursery would: it cancels
the three healthy siblings (graceful runtime-cancel requests,
acks awaited), reaps all four processes, then re-raises,
- ``trio.run(main)`` sees that same ``RemoteActorError`` in the
parent-most process — propagation is end-to-end or bust.
This one-cancels-all style is currently the *only* supervision
strategy offered (it's the one ``trio`` gives you); more
`erlang strategies`_ are roadmap, see the bottom of this page.
The boxed-error bestiary
------------------------
All remote failures arrive locally as one of a small set of
exception types, each carrying enough metadata to work out *who*
failed, *where*, and *why*.
``RemoteActorError``
********************
The workhorse: a "boxed" exception relayed over IPC from another
actor. The original error's type, traceback string and msgdata are
preserved so you can pattern-match on what actually went wrong
remotely,
- ``.boxed_type``: the reconstructed **type** of the original
remote exception (``ValueError``, ``NameError``, what have you),
- ``.src_uid``: the ``(name, uuid)`` pair of the actor where the
error *originated*,
- ``.relay_uid`` / ``.relay_path``: when an error crosses more than
one actor boundary (grandchild -> child -> root) every relaying
actor is recorded; multi-hop boxings are lovingly referred to as
"inceptions" in the runtime internals,
- ``.pformat()``: a rich "tb box" rendering of the remote traceback
for your logs or REPL.
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # the remote task raised `ValueError`
``ContextCancelled``
********************
The cancel-ack for a cross-actor task pair: raised when a
:class:`tractor.Context` task is cancelled *by request*. Its
``.canceller`` attr is the uid of the actor which **requested** the
cancel, which powers the key rule,
- if **you** requested it (you called
:meth:`tractor.Context.cancel`) the resulting ctxc is *absorbed*
at ``open_context()`` exit: an expected outcome, not an error,
- if **anyone else** did — the peer task, or some third-party actor
— it *raises* locally so your code always hears about it.
The full self- vs. cross-cancel semantics are a core teaching point
of :doc:`/guide/context`; go read them there.
``MsgTypeError``
****************
An IPC-payload "type error": a msg violated the dialog's declared
payload spec. See :doc:`/guide/msging` for the typed-messaging
system which enforces it.
``TransportClosed``
*******************
The underlying IPC transport (TCP stream, UDS socket, ...) died or
closed out from under a channel. You'll normally only see this
surface when a peer hard-exits without any graceful runtime
teardown; the supervision machinery treats unexpected transport
loss on a busy channel as a failure and tears down accordingly.
Pick your blast radius
----------------------
Three cancel surfaces, three scopes of effect; choose the smallest
hammer that does the job.
.. list-table::
:header-rows: 1
:widths: 36 34 30
* - surface
- cancels
- typical use
* - :meth:`tractor.ActorNursery.cancel`
- every subactor in the nursery
- whole-tree teardown
* - :meth:`tractor.Portal.cancel_actor`
- one actor: full runtime + proc
- daemon teardown
* - :meth:`tractor.Context.cancel`
- exactly one remote task
- surgical task cancel
``ActorNursery.cancel()``
*************************
The big red button: gracefully cancel every subactor supervised by
the nursery, in parallel, with the escalation discipline below
applied per-child. It's invoked for you whenever an error hits the
nursery block (one-cancels-all); call it yourself for an orderly
early shutdown. Passing ``hard_kill=True`` skips the graceful phase
and goes straight to OS-level process termination — rarely what you
want outside tests.
``Portal.cancel_actor()``
*************************
Cancel one **whole actor**: its entire runtime, every task it's
scheduled, and (for subactors) the OS process, via a graceful
runtime-cancel request,
.. code:: python
await portal.cancel_actor() # bounded wait, bool result
await portal.cancel_actor(
raise_on_timeout=True, # no ack in time?
) # -> `ActorTooSlowError`
The wait for the peer's ack is *bounded* (default
``Portal.cancel_timeout = 0.5`` seconds, tunable per call via
``timeout=``). By default a missed ack just returns ``False``; with
``raise_on_timeout=True`` you instead get an ``ActorTooSlowError``
(from ``tractor._exceptions``) so *your* code can escalate per SC
discipline — exactly what the nursery's own teardown does
internally before resorting to OS-level signalling.
Note the granularity: this cancels an **actor**, not a task. For
one remote task use the ``Context`` layer instead.
``Context.cancel()``
********************
Request cancellation of exactly one remote task: the peer task of
an open :class:`tractor.Context`. Two things to keep straight,
- it cancels the **remote** side only; a ``Context`` is *not* a
:class:`trio.CancelScope` and your local task keeps running until
you exit the ``open_context()`` block,
- the resulting :class:`tractor.ContextCancelled` is absorbed
locally (you asked for it, after all) per the self- vs.
cross-cancel rule above.
Again, :doc:`/guide/context` covers this dance in depth.
Graceful first, hard as a last resort
-------------------------------------
.. margin:: REPL-safe by design
The hard-kill path is *skipped* whenever an actor in the tree
holds the debug-REPL lock (``debug_mode=True`` flavors):
SIGTERM raining down on a tree mid-``pdb`` session would
clobber your prompt. See :doc:`/guide/debugging`.
Every process teardown in ``tractor`` walks the same escalation
ladder, top rung first,
1. **graceful cancel request**: a runtime-cancel msg over IPC; the
target actor cancels its tasks, closes its channels and exits
its :func:`trio.run` cleanly,
2. **soft wait**: the parent waits (bounded) for the child process
to exit on its own,
3. **SIGTERM**: no ack within the bounded wait (internally an
``ActorTooSlowError``) escalates to ``proc.terminate()``,
4. **SIGKILL ultimatum**: still alive after the hard-kill timeout
(~1.6s)? The runtime logs that the "T-800" has been deployed to
collect the zombie and issues ``proc.kill()``. No survivors.
The result is the **no-zombies guarantee**: ``tractor`` tries to
protect you from zombies, no matter what. Quoting the project
manifesto,
If you can create zombie child processes (without using
a system signal) it **is a bug**.
Run the quickstart's self-destructing process-tree demo
(``examples/parallelism/we_are_processes.py``, walked through in
:doc:`/start/quickstart`) under a ``pstree`` watcher and try to
catch a
straggler; we'll wait B)
Roadmap: ``erlang``-style strategies
------------------------------------
One-cancels-all is ``trio``'s strategy and, for now, the only one
``tractor`` ships. Pluggable `erlang strategies`_ — one-for-one
restarts, rest-for-one, transient/permanent child specs and friends
(see the `supervision strategies`_ canon) — are a long-standing
roadmap item tracked in `#22`_. If supervisors are your jam that
issue is the place to sling opinions.
.. seealso::
- :doc:`/guide/context` — the cross-actor task layer where
per-task cancellation actually lives,
- :doc:`/guide/msging` — the typed msg layer that raises
:class:`tractor.MsgTypeError`,
- :doc:`/guide/debugging` — what cancellation does (and very
carefully does *not* do) while a REPL is up.
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _erlang strategies: https://learnyousomeerlang.com/supervisors
.. _supervision strategies: https://www.erlang.org/doc/system/sup_princ.html
.. _#22: https://github.com/goodboy/tractor/issues/22

127
docs/guide/clustering.rst 100644
View File

@ -0,0 +1,127 @@
Higher-level cluster APIs
=========================
Sometimes you don't want a hand-crafted supervision tree; you want
"a pile of workers, one per core, now please". For that there's
:func:`tractor.open_actor_cluster`: a convenience wrapper which
spawns a *flat* cluster of subactors and hands you back a portal to
each,
.. code:: python
@acm
async def open_actor_cluster(
modules: list[str], # RPC allowlist for workers
count: int = cpu_count(), # one per core by default
names: list[str]|None = None, # default: 'worker_{i}'
hard_kill: bool = False, # fwd to `an.cancel()`
**runtime_kwargs, # fwd to `open_root_actor()`
) -> AsyncGenerator[dict[str, tractor.Portal], None]:
A cluster in one block
----------------------
.. literalinclude:: ../../examples/quick_cluster.py
:caption: examples/quick_cluster.py
:language: python
Walkthrough,
- ``open_actor_cluster(modules=[__name__])`` concurrently spawns
one subactor per detected core (per
:func:`multiprocessing.cpu_count`); the ``modules`` list is the
usual ``enable_modules``-style capability allowlist so workers
may run functions defined in this module,
- it yields a ``dict[str, tractor.Portal]`` mapping worker name to
portal; note the keys get prefixed with the *spawning* actor's
name, so from the root you'll see ``'root.worker_0'``,
``'root.worker_1'``, etc.,
- a plain :class:`trio.Nursery` then fans out one
``portal.run(sleepy_jane)`` per worker; each prints its actor
``.uid`` from inside its own process then naps forever — what
runs *inside* each worker (and how many tasks you point at it)
is entirely yours to compose,
- ``tractor.trionics.collapse_eg()`` un-nests the strict
``ExceptionGroup`` wrapping so the demo's ``KeyboardInterrupt``
surfaces as itself instead of arriving eg-boxed,
- on block exit the whole fleet is torn down for you via
:meth:`tractor.ActorNursery.cancel`; pass ``hard_kill=True`` at
open time to skip straight to OS-level termination instead of
the graceful ladder described in :doc:`/guide/cancellation`.
Sizing, naming, fleet-wide options
----------------------------------
``count`` doesn't have to be core-count and the auto-generated
``'worker_{i}'`` names are just the default; pass your own (the
length must match ``count`` or you get a ``ValueError``). Any
extra ``**runtime_kwargs`` pass through verbatim to
:func:`tractor.open_root_actor`, so fleet-wide runtime options are
one kwarg away,
.. code:: python
async with tractor.open_actor_cluster(
modules=['mylib.workers'],
count=4,
names=['scout', 'miner', 'smelter', 'smith'],
debug_mode=True, # whole-fleet crash-to-REPL
) as portal_map:
...
From here the composition patterns are the usual ``tractor`` fare:
``portal.run()`` for one-shot calls (as in the demo), or — for a
persistent bidirectional dialog per worker — concurrently enter N
``portal.open_context()`` blocks with
``tractor.trionics.gather_contexts()``; see :doc:`/guide/context`
for that whole layer.
Clusters vs. nurseries
----------------------
.. d2:: diagrams/actor_tree.d2
:margin:
:caption: The general shape: arbitrary nesting. A cluster is
this, minus the nesting.
:alt: a nested supervision tree of subactors
``open_actor_cluster()`` is sugar, not a new primitive: under the
hood it's just :func:`tractor.open_nursery` plus N concurrent
``start_actor()`` calls plus a ``.cancel()`` on the way out. Reach
for it when,
- you want a *flat*, homogeneous fleet (classic worker-pool or
map-style fan-out shapes),
- "one per core" — or a fixed ``count`` — is the right sizing
story,
- every child can share the same spawn options.
Drop down to a raw :class:`tractor.ActorNursery` when the topology
gets any fancier: nested trees, heterogeneous children, per-child
``debug_mode``/transport/module options, daemons mixed with
one-shot workers, and so on (see :doc:`/guide/parallelism` for a
hand-rolled pool). Either way the supervision semantics are
identical: one-cancels-all error propagation and the no-zombies
guarantee from :doc:`/guide/cancellation` apply to clusters too.
Provisional, by design
----------------------
.. note::
APIs in this section are considered **provisional**: the
signature and semantics of :func:`tractor.open_actor_cluster`
may shift as higher-level supervision machinery lands. We
encourage you to try it and provide feedback — the
`matrix channel`_ is the place to say hi, and `#22`_ tracks the
broader supervisor-strategy roadmap.
.. seealso::
- :doc:`/guide/parallelism` — worker pools built "by hand" with
plain actor nurseries (and why that's easy peasy),
- :doc:`/guide/cancellation` — the teardown machinery a cluster
inherits for free.
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
.. _#22: https://github.com/goodboy/tractor/issues/22

344
docs/guide/context.rst 100644
View File

@ -0,0 +1,344 @@
The ``Context``: a cross-actor task pair
=========================================
If you've written any trio_ you already know the contract: every
task lives in a nursery, errors always propagate, cancellation is
scoped, and nothing leaks. ``tractor`` extends that exact contract
*across processes* — the same guarantees from the seminal
`blog post`_, just with the nursery split across two memory
domains. The primitive that does it is :class:`tractor.Context`: a
**linked pair of tasks**, one in each of two actors, supervised as
a single `structured concurrency`_ (SC) scope over IPC.
.. d2:: diagrams/context_handshake.d2
:caption: The SC-transitive supervision protocol, msg by msg.
:alt: sequence diagram of the context handshake msg flow
Pretty much everything else is (or is slated to be) built on this
one primitive: ``ActorNursery.run_in_actor()`` is a convenience
for "spawn, open a context, await the result, tear down"; plain
``Portal.run()`` RPC is planned to be re-implemented on top of it;
the multi-process debugger's tree-wide REPL lock rides one. Grok
this page and the rest of the library reads as convenience
wrappers B)
The endpoint contract
---------------------
A context endpoint is an async function decorated with
:func:`tractor.context` which declares **a param annotated**
``tractor.Context`` — any param name you like, the annotation is
what's required:
.. code:: python
@tractor.context
async def trainer(
ctx: tractor.Context,
model: str,
) -> str:
await ctx.started('ready')
return f'trained {model}'
.. margin:: Who am I talking to?
Inside any context task
:func:`tractor.current_ipc_ctx` returns the
``Context`` bound to the current task; handy
in helpers that don't take ``ctx`` explicitly.
The parent (aka "opener") side invokes it through a
:class:`tractor.Portal` using ``Portal.open_context()``, passing
any extra kwargs which are shipped over the wire as the remote
task's arguments. Since the target fn is referenced by module
path, that module must be listed in the peer actor's
``enable_modules`` allowlist — RPC capability is always opt-in.
The decorator also accepts a ``pld_spec``: a type (union) which
every payload in the dialog is validated against, upgrading your
msgs to a typed contract enforced via :exc:`tractor.MsgTypeError`.
Validation strictness follows the "`cheap or nasty`_"
`(un)protocol`_ pattern: the one-shot ``Started`` payload gets the
nasty treatment (stringently round-trip checked before it's even
sent) while high-rate stream payloads stay cheap (checked only
receiver side).
The handshake, on the wire
--------------------------
Every context runs one instance of ``tractor``'s "SC-transitive
supervision protocol": a tiny fixed grammar of msgspec_-typed msgs
encapsulating *all* RPC dialogs between actors. *Transitive*
because each IPC link obeys the same rules a local nursery does —
starts are acked, completion is awaited, errors and cancels always
relay — so chaining links across a process tree composes into one
tree-wide SC scope.
The figure up top shows a full dialog; in order:
``Start``
sent by ``Portal.open_context()``: "schedule a task running
this function with these kwargs".
``StartAck``
the peer runtime confirms the task is scheduled and that the
endpoint really is a context-style fn.
``Started``
emitted when the child task calls
:meth:`tractor.Context.started`; carries the first payload
and unblocks the parent's entry of ``open_context()``.
``Yield``
one per :meth:`tractor.MsgStream.send`, flowing in *either*
direction while a stream is open.
``Stop``
graceful end-of-stream: the far side's ``async for``
terminates cleanly.
``Return``
the child fn returned; its value becomes the context's final
result. If the child raised instead, an ``Error`` msg takes
this slot carrying the boxed traceback.
``ctx.started()``: just like ``task_status.started()``
*******************************************************
The startup phase is a deliberate clone of
:meth:`trio.Nursery.start` semantics: the child decides when it's
"up", optionally handing back a first value, and the parent stays
blocked until that moment:
.. code:: python
# trio, in-process
first = await nursery.start(child_fn)
# tractor, cross-process
async with portal.open_context(child_fn) as (ctx, first):
...
The ``as (ctx, first)`` tuple is exactly that pair: the
:class:`tractor.Context` handle plus whatever value the child
passed to ``await ctx.started(value)``. And readiness is not
optional — for instance opening a stream before ``.started()``
has been called raises a ``RuntimeError``; handshake first, then
dialog.
Bidirectional streaming over a context
--------------------------------------
The canonical ping-pong (design history: `#53`_, `#223`_) — a
full-duplex msg stream between a parent and its spawned peer:
.. literalinclude:: ../../examples/rpc_bidir_streaming.py
:caption: examples/rpc_bidir_streaming.py
:language: python
What's going on?
- ``start_actor()`` spawns the daemon-style subactor
``'rpc_server'`` with this very module in its allowlist.
- ``portal.open_context(simple_rpc, data=10)`` fires the
``Start`` msg then blocks until the child task calls
``await ctx.started(data + 1)`` — hence ``sent == 11``.
- both tasks enter ``ctx.open_stream()``: a stream dialog is only
fully open once *each* side has entered its block.
- the parent seeds the first ``'ping'``; each side then echoes
the other, one ``Yield`` msg per ``stream.send()``.
- after the 9th pong the parent ``break``\ s (10 pings sent in
total) and exits its stream block, which sends ``Stop``; the
child's ``async for`` completes gracefully and its ``else``
clause asserts all 10 pings arrived.
- the 10th in-transit pong? Discarded by the implicit drain at
``open_context()`` exit, which runs the dialog down to the
child's ``Return`` (here ``None``).
- daemon actors live until told otherwise:
``portal.cancel_actor()`` reaps the subactor explicitly.
Results: the ``Return`` leg
---------------------------
Every context resolves to a final outcome. Wait on it explicitly
from the parent side:
.. code:: python
async with portal.open_context(ep) as (ctx, first):
...
result = await ctx.wait_for_result()
or just exit the block — ``__aexit__`` implicitly drains the msg
flow until the ``Return`` (or ``Error``) arrives, discarding any
in-transit ``Yield``\ s on the way. Either way the rule of
`causality`_ holds exactly as in a local nursery: **the opener
never unblocks before the remote task is done**.
For post-hoc inspection (think supervision/restart logic) the ctx
also exposes ``Context.outcome``, ``.maybe_error`` and
``.has_outcome`` — where a "result" might well be the error the
dialog ended with.
Cancellation semantics
----------------------
The part you actually came for; read it twice B)
A context's two tasks are **cancel-scope-linked across the IPC
boundary**: whatever ends one side — error, cancellation, plain
old return — is relayed such that the other side ends
equivalently. No silent half-open dialogs, no orphaned remote
tasks, ever.
``ctx.cancel()`` cancels the *remote* task
*******************************************
:meth:`tractor.Context.cancel` requests cancellation of the
**remote** task only:
.. code:: python
async with portal.open_context(ep) as (ctx, first):
await accomplish_things(ctx)
await ctx.cancel() # remote task, NOT me
A :class:`tractor.Context` is **not** a :class:`trio.CancelScope`:
the call doesn't (and can't) cancel your local task. It sends the
cancel request and waits a bounded ``timeout`` for the peer
runtime's ``CancelAck``, then your code proceeds to the block exit
as normal.
Compare scopes here: ``Portal.cancel_actor()`` is the big hammer
which cancels the peer's **entire runtime** (and thus process);
``ctx.cancel()`` is the per-dialog scalpel.
``ContextCancelled`` and the absorption rule
*********************************************
When a context task gets cancelled *by request* the requestee's
runtime reports back with a :exc:`tractor.ContextCancelled`
("ctxc") whose ``.canceller`` field holds the uid of the actor
which asked. That one field decides what you observe:
**you requested it**
i.e. ``ctxc.canceller == tractor.current_actor().uid``: the
ctxc is **absorbed** at ``open_context()`` exit — nothing
raises in your block. You asked for a graceful stop and got
it; if you care, ``await ctx.wait_for_result()`` hands the
ctxc back as a plain *value* for inspection.
**anyone else requested it**
the peer cancelling itself, or some third actor cancelling it
from the side: the ctxc **is raised** in your block. From
your scope's perspective a task you depend on was killed out
from under you and SC demands you hear about it — exactly
like a sibling crash in a `nursery`_.
In code:
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.ContextCancelled as ctxc:
# can only be a peer- or third-party cancel;
# self-requested cancels are absorbed at exit.
assert ctxc.canceller != tractor.current_actor().uid
This self- vs cross-cancel split is what makes explicit teardown
*composable*: a supervisor cancels its dialogs without try/except
noise, while unexpected cancellation anywhere in the tree still
propagates loudly like any other failure.
.. warning::
Once ``ctx.cancel()`` has been called the dialog is done: a
subsequent ``ctx.open_stream()`` raises ``RuntimeError``.
For introspection the ctx exposes trio-flavored status props:
``.cancel_called`` (this side requested), ``.cancel_acked`` (peer
confirmed), ``.cancelled_caught`` and ``.canceller`` —
deliberately mirroring :class:`trio.CancelScope` naming.
Errors propagate, both ways
---------------------------
A crash on either end tears down the pair, SC style:
- **child raises**: the exception ships back as an ``Error`` msg
and re-raises in the parent block boxed as a
:exc:`tractor.RemoteActorError`; the original class rides along
as ``.boxed_type`` with ``.src_uid`` naming the crashed actor.
- **parent raises** (or is cancelled) inside the block: an
equivalent error/cancel is relayed to the child task so it can
never outlive the dialog.
.. code:: python
try:
async with portal.open_context(ep) as (ctx, first):
...
except tractor.RemoteActorError as rae:
if rae.boxed_type is ValueError:
... # remote ValueError, type preserved
Errors that hop through intermediary actors on their way up the
tree ("inceptions" XD) keep the full relay trail in
``.relay_uid`` / ``.relay_path``. Payloads violating your declared
``pld_spec`` surface as the IPC analog of a ``TypeError``:
:exc:`tractor.MsgTypeError`.
Overruns and backpressure
-------------------------
Stream msgs land in a bounded per-context buffer on the receiver
side. A sender that outpaces a non-consuming receiver *overruns*
it and the runtime raises :exc:`tractor.StreamOverrun` (also a
:exc:`trio.TooSlowError`) instead of buffering without bound — SC
discipline applies to memory too.
Your knobs:
- ``msg_buffer_size`` on ``ctx.open_stream()`` sizes the buffer.
- ``allow_overruns=True`` (on ``Portal.open_context()`` and/or
``ctx.open_stream()``) opts in to absorbing overflow instead of
erroring — reasonable for bursty telemetry-ish feeds, just know
you're trading the error for extra buffering.
One context, one stream
-----------------------
A ``MsgStream`` is strictly **one-shot use**: once it closes —
gracefully or not, from either side — it can never be re-opened
on the same ctx. Want another round with the same peer? Open a
fresh context; they're cheap. The full close-vs-cancel teardown
story lives in :doc:`/guide/streaming`.
.. rubric:: Where to next?
:doc:`/guide/streaming` covers the rest of the msg-moving story:
the legacy one-way API, multi-actor pipelines and in-actor
broadcast fan-out. For exhaustive API detail see
:class:`tractor.Context`, :class:`tractor.MsgStream` and
:exc:`tractor.ContextCancelled`.
.. _trio: https://github.com/python-trio/trio
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _msgspec: https://jcristharif.com/msgspec/
.. _#53: https://github.com/goodboy/tractor/issues/53
.. _#223: https://github.com/goodboy/tractor/issues/223

545
docs/guide/debugging.rst 100644
View File

@ -0,0 +1,545 @@
.. _debugging:
================================
"Native" multi-process debugging
================================
``tractor`` ships the thing every ``multiprocessing`` user has
wished for and quietly assumed was impossible: a multi-process
debugger that *just works*.
Drop ``await tractor.pause()`` — or, with `greenback`_ installed,
a plain builtin ``breakpoint()`` — anywhere in any actor: the
root, a child, a grandchild, a sync helper function, even an
``asyncio`` task inside an "infected" actor. A full-featured
`pdbp`_ REPL opens *in that process*, with syntax-highlighted
source listings, tab completion and sticky mode, attached to your
one terminal.
Under the hood every REPL entry acquires a tree-global tty mutex
via an IPC request to the root actor, so prompts from concurrent
pauses and crashes never interleave. ``ctrl-c`` is shielded while
any REPL is live, so a stray ``SIGINT`` can't vaporize the tree
out from under you. And in debug mode any uncaught error drops
you into a crash REPL *first in the failing child*, then again at
each parent as the boxed :class:`~tractor.RemoteActorError` climbs
the supervision tree.
No remote-pdb sockets, no ``set_trace()`` port juggling, no
``ptrace`` attach dance: the debugger semantics you already know,
transparently extended across an entire process tree. Because
``tractor`` is a `structured concurrency`_ (SC) runtime, the
debugger composes with supervision instead of fighting it — quit
a REPL and errors keep propagating exactly like `trio`_ taught
you, ending in clean, zombie-free teardown.
We're pretty sure it's the (first ever?) "native" debugging UX
for multi-process Python B)
Enabling debug mode
-------------------
Pass ``debug_mode=True`` to your runtime entrypoint, either
:func:`tractor.open_nursery` (which forwards it to the implicitly
opened root actor) or :func:`tractor.open_root_actor` directly:
.. code:: python
async with tractor.open_nursery(
debug_mode=True, # arm the whole actor tree
) as an:
...
This arms the debug machinery *tree-wide*:
- crash handling is enabled in every actor: uncaught errors enter
a REPL before they propagate,
- the internal tty-lock module is auto-exposed over RPC to every
subactor (this is what makes the one-terminal handoff work),
- console logging is bumped to include ``PDB``-level status msgs
so you can see REPL acquire/release events as they happen.
You can instead flip it on for just one child, letting its
siblings crash-and-burn the normal way:
.. code:: python
portal = await an.start_actor(
'sketchy_worker',
debug_mode=True, # OR-ed with the tree-wide flag
)
See ``examples/debugging/per_actor_debug.py`` for a runnable
proof of the selective style.
.. note::
Debug mode requires the child-side runtime to be
``trio``-native so that the tty-lock IPC dialog works; it's
currently supported on the ``'trio'`` (default) and
``'main_thread_forkserver'`` spawn backends and raises
``RuntimeError`` for any other ``start_method``.
Your first pause point
----------------------
:func:`tractor.pause` is the SC-aware, multi-process spelling of
the stdlib's ``breakpoint()``. In the root actor it looks almost
boring:
.. literalinclude:: ../../examples/debugging/root_actor_breakpoint.py
:caption: examples/debugging/root_actor_breakpoint.py
:language: python
Run it and you get a ``(Pdb+)`` prompt parked on the ``pause()``
line; type ``c`` (continue) and the program finishes normally.
The exact same call works from *any* subactor, no matter how deep
in the tree:
.. literalinclude:: ../../examples/debugging/subactor_breakpoint.py
:caption: examples/debugging/subactor_breakpoint.py
:language: python
Each loop iteration the child actor requests the terminal from
the root over IPC, REPLs you, then releases it on ``c``. Pause
points are re-entrant-safe: repeat calls from the same task are
no-op'd and other local tasks queue politely for the REPL.
When you get bored, type ``q`` (quit): the resulting
``bdb.BdbQuit`` is boxed and shipped to the parent like any other
remote error XD — causality is preserved even for your debugging
mistakes.
Crash REPLs: errors climb the tree
----------------------------------
Pause points are only half the story. With debug mode armed, any
*uncaught* error anywhere in the tree triggers what we call crash
handling mode:
.. literalinclude:: ../../examples/debugging/subactor_error.py
:caption: examples/debugging/subactor_error.py
:language: python
What happens when the child hits that (very intentional)
``NameError``:
1. a REPL opens **in the crashed child first** — you inspect the
raising frame, its locals, the works, right inside the failed
process,
2. when you quit, the error is boxed into a
:class:`~tractor.RemoteActorError` and relayed to the parent,
3. the parent (here the root) gets *its own* crash REPL with the
rendered remote traceback,
4. quit again and the nursery tears the tree down — errors keep
propagating per SC rules, no zombies left behind.
You debug the failure at every hop of the supervision tree, which
for multi-hop trees means you can chase an error from the leaf
that raised it all the way up to the root that supervises it.
Need to skip REPL entry for certain exceptions? Pass a predicate
via ``open_root_actor(debug_filter=...)``; by default
cancellation-only exception (groups) don't engage the REPL.
One terminal, many actors
-------------------------
So how do N processes share one tty without garbling it? The root
actor owns stdio for the whole tree and guards it with a FIFO
mutex; every subactor REPL entry is an IPC lock request to the
root. Exactly one actor-task in the entire tree can own the
terminal at a time, so prompts never interleave — ever.
.. d2:: diagrams/debug_lock.d2
:caption: Every REPL entry serializes through the root actor's
tty lock; ``continue``-ing one REPL hands the terminal to
the next waiter, FIFO style.
:alt: sequence diagram of two subactors serializing pdb REPL
access through the root actor's tty lock
The runtime's teardown paths cooperate too: a cancelling parent
always waits for any live REPL to release before reaping
children, so the debugger never gets yanked out from under you
mid-keystroke.
.. margin:: Watch the tree live
Run any of these examples with a process-tree watcher in a
second terminal and watch actors come and go::
watch -n 0.1 "pstree -a $$"
Here's the showpiece: one daemon child re-entering
``tractor.pause()`` forever inside a stream, while its sibling
repeatedly raises a ``NameError``:
.. literalinclude:: ../../examples/debugging/multi_daemon_subactors.py
:caption: examples/debugging/multi_daemon_subactors.py
:language: python
What you'll actually see
************************
Running it looks *roughly* like this (uids, tracebacks and source
listings elided; REPL order can vary with who wins the lock
race)::
$ python examples/debugging/multi_daemon_subactors.py
Opening a pdb REPL in paused actor: ('bp_forever', '<uuid>')
<highlighted source around the `await tractor.pause()` line>
(Pdb+) c
Opening a pdb REPL in crashed actor: ('name_error', '<uuid>')
<live traceback: NameError: name 'doggypants' is not defined>
(Pdb+) q
Opening a pdb REPL in crashed actor: ('root', '<uuid>')
<boxed RemoteActorError traceback relayed from 'name_error'>
(Pdb+) q
Two (then three) processes, one terminal, zero confusion:
``c``-ing out of the paused daemon's REPL releases the tty lock,
which immediately hands the prompt to the crashed sibling; quit
that and the error propagates as a fully-rendered
:class:`~tractor.RemoteActorError` to the parent where one final
crash REPL catches it before clean, zombie-free teardown.
For maximum drama run
``multi_nested_subactors_error_up_through_nurseries.py`` (under
``examples/debugging/``) which pulls the same trick across a
*three-deep* process tree — the tty lock keeps every prompt
orderly the whole way up.
Post-mortem, on demand
----------------------
Crash handling is automatic, but you can also enter a REPL on
a live exception *manually* with :func:`tractor.post_mortem` —
the actor-aware equivalent of ``pdb.post_mortem()`` — from inside
any ``except`` block in any actor (kwargs: ``tb=`` for an
explicit traceback, plus ``shield=`` and ``hide_tb=``):
.. literalinclude:: ../../examples/debugging/pm_in_subactor.py
:caption: examples/debugging/pm_in_subactor.py
:language: python
This example demos three REPL entries from one error:
- the child's manual ``post_mortem()`` inside its ``except``,
- the runtime's automatic crash handler in the same child once
the error re-raises out of the RPC task,
- a manual ``post_mortem()`` in the parent on the received
:class:`~tractor.RemoteActorError`, whose ``.boxed_type``
faithfully reports the original ``NameError``.
Pausing from sync code
----------------------
No ``await``? No problem. :func:`tractor.pause_from_sync` brings
the same tree-aware REPL to plain synchronous functions — handy
when the suspect code is three helpers deep and decidedly not
async.
It's powered by `greenback`_, which is optional, so you need to:
1. install it (it ships in ``tractor``'s ``sync_pause``
dependency group),
2. enable it at runtime entry:
.. code:: python
async with tractor.open_nursery(
debug_mode=True,
maybe_enable_greenback=True,
) as an:
...
With that armed, sync code can pause from three different caller
environments: the main ``trio`` thread, ``trio.to_thread`` bg
threads, and (see the next section) ``asyncio`` tasks in infected
actors. The greenback "portal" hops back into the ``trio`` loop
to do the lock/REPL dance on your behalf:
.. literalinclude:: ../../examples/debugging/sync_bp.py
:caption: examples/debugging/sync_bp.py (the sync fn, excerpt)
:language: python
:pyobject: sync_pause
.. literalinclude:: ../../examples/debugging/sync_bp.py
:caption: examples/debugging/sync_bp.py (called in a subactor,
excerpt)
:language: python
:pyobject: start_n_sync_pause
The full script also exercises the hairier root-actor bg-thread
cases (and documents their remaining sharp edges) if you want the
deep lore.
The builtin ``breakpoint()`` override
*************************************
When debug mode boots with greenback available, ``tractor`` wires
Python's `PEP 553`_ hook so the *builtin* ``breakpoint()`` becomes
the actor-aware sync pause, by exporting::
PYTHONBREAKPOINT=tractor.devx.debug._sync_pause_from_builtin
That means third-party and legacy code containing bare
``breakpoint()`` calls debugs correctly inside your actor tree
with zero edits (the override even forwards kwargs like
``hide_tb`` to the underlying pause machinery, as shown in the
excerpt above).
.. warning::
Without greenback (or with ``maybe_enable_greenback=False``,
the default), ``debug_mode=True`` instead *blocks* the builtin
``breakpoint()``: ``sys.breakpointhook`` is swapped for a
raiser and ``PYTHONBREAKPOINT=0`` is set. A naive
``breakpoint()`` from some random process would clobber the
shared tty, so we'd rather hand you a loud ``RuntimeError``
with install instructions.
Both the hook and the env var are restored to their prior values
on runtime exit — see
``examples/debugging/restore_builtin_breakpoint.py`` for the
proof.
Breakpoints inside ``asyncio`` tasks
------------------------------------
Yes, even "infected ``asyncio``" actors get the goods. Spawn a
child with ``infect_asyncio=True`` (``trio`` runs as a guest on
the ``asyncio`` loop inside it) and, with debug mode + greenback
armed, every ``asyncio`` task started via ``tractor.to_asyncio``
is automatically granted a greenback portal — so a plain builtin
``breakpoint()`` (or ``tractor.pause_from_sync()``) inside an
``asyncio.Task`` joins the same single-terminal, tree-locked REPL
flow:
.. literalinclude:: ../../examples/debugging/asyncio_bp.py
:caption: examples/debugging/asyncio_bp.py
:language: python
Note the interleave: a ``breakpoint()`` on the ``asyncio`` side,
``tractor.pause()`` on the ``trio`` side of the same actor, and
another pause up in the root — all serialized through the one tty
lock with no cross-actor (or cross-event-loop!) clobbering.
One catch: ``asyncio`` tasks spawned *out-of-band* — i.e. not via
``tractor.to_asyncio``, typically by some third-party aio lib —
have no portal bestowed, so a sync pause from one raises a loud
``RuntimeError`` telling you to ``greenback.ensure_portal()``
first. See :ref:`the caveats <debugging-caveats>` below.
Teardown debugging: the shielded pause
--------------------------------------
`Cancellation`_ is ``trio``'s bread and butter, which raises an
awkward question: how do you REPL inside an *already-cancelled*
scope, say while debugging some teardown sequence? A bare
``pause()`` would itself be cancelled at its next checkpoint.
The answer is ``await tractor.pause(shield=True)``, which wraps
the lock acquisition and REPL session in a shielded cancel scope
(``post_mortem(shield=True)`` works the same way):
.. literalinclude:: ../../examples/debugging/shielded_pause.py
:caption: examples/debugging/shielded_pause.py
:language: python
If you forget, ``tractor`` has your back: an unshielded
``pause()`` from a cancelled scope fails fast with a hint
suggesting ``await tractor.pause(shield=True)`` instead of
silently never REPL-ing.
Go ahead, mash ctrl-c
---------------------
While any REPL is live the runtime installs a custom ``SIGINT``
handler tree-wide so that a reflexive ``ctrl-c`` (or five) can't
nuke your debug session:
- the actor that owns the REPL ignores the interrupt and simply
re-flushes the prompt — keep mashing, it's fine,
- the root actor ignores ``SIGINT`` while a still-IPC-connected
child holds the tty lock, so the supervisor won't tear down the
tree out from under the debugger,
- if the lock state has gone *stale* — the locking child died or
its IPC channel dropped — the root cancels the stale lock scope
and restores ``trio``'s default handler, so ``ctrl-c`` works
again exactly when it should.
The handler is uninstalled and ``trio``'s own ``SIGINT``
semantics restored every time a REPL releases (on ``continue`` /
``quit``).
Live task-tree dumps
--------------------
Sometimes there's no error to catch — the tree is just *hung* and
you want to know where. For that ``tractor`` integrates
`stackscope`_: send a signal, get a full ``trio`` task-tree dump
from every actor in the tree.
Enable it any of three ways:
- ``open_root_actor(enable_stack_on_sig=True)`` (or via
``open_nursery()`` which forwards it),
- set ``TRACTOR_ENABLE_STACKSCOPE=1`` in the env — it's inherited
through the process tree so every (sub)actor arms the handler
at boot,
- call ``tractor.devx.enable_stack_on_sig()`` directly.
It's intentionally *not* gated on ``debug_mode`` so you can leave
it armed in plain runs. Then, when the hang strikes, signal the
tree with ``SIGUSR1``.
.. tip::
No need to hunt down pids — pattern-match the original cmdline
with ``pkill``::
$ pkill --signal SIGUSR1 -f "python example_script.py"
Each actor dumps its entire ``trio`` task tree (full nursery
recursion via ``stackscope.extract()``) to its tty *and* tees it
to ``/tmp/tractor-stackscope-<pid>.log`` — so the trace survives
even under captured-stdio harnesses — then relays the signal on
to its children, parent-before-child, until the whole tree has
reported in.
Try it yourself with the demo script, which deliberately hangs a
subactor in a shielded sleep:
.. literalinclude:: ../../examples/debugging/shield_hang_in_sub.py
:caption: examples/debugging/shield_hang_in_sub.py
:language: python
(That ``trio.CancelScope(shield=True)`` hang also shows off the
zombie reaper: ``ctrl-c`` the root and the un-cancellable child
still gets hard-reaped — if you can create a zombie it **is a
bug**.)
Crash handling for sync and CLI code
------------------------------------
All of the above rides on the actor runtime, but crashes don't
politely wait for ``trio.run()``. For plain sync code — think
``typer``/``click`` CLI endpoints, config parsing, anything
pre-runtime — there's a sync context manager that wraps the same
``pdbp`` post-mortem UX:
.. code:: python
from tractor.devx import open_crash_handler
def main(): # any sync code, no runtime required
with open_crash_handler() as boxed:
run_my_cli_thing()
By default any ``BaseException`` (minus an ``ignore`` set
defaulting to ``KeyboardInterrupt`` and ``trio.Cancelled``)
enters the REPL then re-raises on exit; pass
``raise_on_exit=False`` to suppress instead and introspect the
``boxed.value`` afterward. The ``catch``/``ignore`` sets and a
``repl_fixture`` are all tweakable.
For the classic ``--pdb`` CLI-flag pattern use the conditional
variant:
.. code:: python
from tractor.devx import maybe_open_crash_handler
@app.command() # a `typer` (or `click`) endpoint
def cmd(pdb: bool = False):
with maybe_open_crash_handler(pdb=pdb):
...
REPL niceties and hooks
-----------------------
Every REPL in this guide is a `pdbp`_ instance (the maintained
fork-and-fix of `pdb++`_) pre-configured by ``tractor``:
- pygments syntax highlighting in listings and tracebacks,
- tab completion — including an automatic fixup for
libedit-compiled CPythons (e.g. ``uv``-distributed pythons),
- sticky mode available via the ``sticky`` command (off by
default),
- no long-line truncation (terminal resizes behave),
- the ``(Pdb+)`` prompt, ``ll``, hidden-frames support and the
rest of the ``pdb++`` goodies you may already know.
Internal runtime frames are traceback-hidden so the REPL lands
exactly on *your* ``pause()``-call or crash frame, never on
``tractor`` guts.
Finally, if your app owns the terminal (TUIs, fullscreen
dashboards) pass ``repl_fixture=<your ctx mngr>`` to ``pause()``,
``post_mortem()`` or ``open_crash_handler()``: it's entered just
before the REPL engages (return ``False`` to skip entry entirely)
and exited on release — perfect for suspending and restoring your
screen around a debug session.
.. _debugging-caveats:
Caveats and platform notes
--------------------------
An honest list of the current rough edges:
- **Windows**: the debugger has no CI coverage on windows at all
(the entire test module is skipped there); manual testing has
shown it *can* work, but you're in uncharted territory —
reports welcome!
- **macOS**: supported but with rough edges: special-cased prompt
re-flushing for ``bash``-on-darwin, a few tooling tests skipped
on CI, and the AF_UNIX ~104-char socket-path limit forces some
examples (like the stackscope demo above) to fall back from
``'uds'`` to ``'tcp'`` transport. Wonder if all of it'll work
on OS X? So do we.
- **CPython 3.14**: ``greenback`` (via ``greenlet``) doesn't
support 3.14 yet, so ``pause_from_sync()`` and the builtin
``breakpoint()`` override are effectively 3.13-only for now.
The async APIs — ``pause()`` and ``post_mortem()`` — need no
greenback and work everywhere.
- **out-of-band** ``asyncio`` **tasks**: sync pauses from aio
tasks *not* spawned via ``tractor.to_asyncio`` raise a
``RuntimeError`` (no greenback portal was bestowed); run
``await greenback.ensure_portal()`` inside such a task first.
- **nested-tree ctrl-c edges**: ``SIGINT`` relay through
intermediary parents that aren't themselves in debug mode still
has known rough edges — see `#320`_.
- **captured stdio**: ``pytest``-style output capture can hang a
``pause()``; use a real terminal (or a pty à la ``pexpect``,
which is how ``tractor``'s own suite drives every one of these
examples).
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` — the SC-linked cross-actor task API
that all the crash-propagation semantics above ride on.
- :func:`tractor.pause`, :func:`tractor.post_mortem` and
:func:`tractor.pause_from_sync` in the API reference.
- ``examples/debugging/`` — 20-odd runnable scripts, nearly
every one exercised by the test suite through a real pty.
.. _structured concurrency:
https://en.wikipedia.org/wiki/Structured_concurrency
.. _trio: https://github.com/python-trio/trio
.. _cancellation: https://trio.readthedocs.io/en/latest/
reference-core.html#cancellation-and-timeouts
.. _pdbp: https://github.com/mdmintz/pdbp
.. _pdb++: https://github.com/pdbpp/pdbpp
.. _greenback: https://github.com/oremanj/greenback
.. _stackscope: https://github.com/oremanj/stackscope
.. _PEP 553: https://peps.python.org/pep-0553/
.. _#320: https://github.com/goodboy/tractor/issues/320

255
docs/guide/discovery.rst 100644
View File

@ -0,0 +1,255 @@
Actor discovery
===============
So you've spawned a tree of trio-"actors"; now their tasks need to
*find* each other to start a dialog. ``tractor`` ships a (self
admittedly) **very naive** discovery system which is nonetheless
mighty handy for wiring up service-style apps: a built-in
*registrar* actor plus a small set of lookup APIs that deliver
a live, connected ``Portal`` to whichever peer you're after.
.. d2:: diagrams/actor_tree.d2
:margin:
:caption: The root actor doubles as the *registrar* by
default; every spawned actor registers itself with it.
:alt: actor tree with root acting as registrar
Because ``tractor`` is built on structured concurrency (SC), the
discovery layer is *not* some external etcd/consul-shaped service
you have to babysit; it's just another actor — normally the root
of your tree — doing a bit of bookkeeping as part of the runtime.
Every actor phones home
-----------------------
On runtime boot **every** actor self-registers with the registrar:
it submits its unique ``(name, uuid)`` identity pair (aka its
``uid``) mapped to the list of transport addresses its IPC server
is bound to. On graceful teardown it likewise *un*-registers, so
the registry tracks the live tree as it grows and shrinks.
.. note::
Actor names are **not** enforced unique — the registry is keyed
by the full ``(name, uuid)`` pair. Name-based lookups simply
resolve to the *last* registered match, so if you boot five
actors all named ``'bob'``, you get the freshest ``'bob'`` B)
First boot: who's the registrar?
--------------------------------
By default the **root actor** *is* the registrar; subactors
inherit the tree's ``registry_addrs`` at spawn time so the whole
clan shares one registry with zero config on your part.
The bootstrap rule inside ``open_root_actor()`` is delightfully
simple:
- on boot, ping every socket addr in ``registry_addrs``; when none
are passed the per-transport defaults are used: for TCP the
loopback ``('127.0.0.1', 1616)``, for UDS a
``registry@1616.sock`` file,
- if a registrar answers, you boot as a plain (non-registrar) root
actor and register with the *existing* registry; your own IPC
server binds random same-transport addrs instead,
- if **nothing answers, congratulations: you just became the
registrar**. Your transport server binds the registry addrs
themselves and you start serving lookups for everyone else.
Pass ``ensure_registry=True`` when your program *requires* being
the one-and-only registrar; boot then fails loudly with a
``RuntimeError`` if some other process already bound the registry
socket(s).
Looking up actors
-----------------
All lookup APIs are async context managers, so the SC rule you
already know from the rest of ``tractor`` holds here too: any
delivered portal (and its underlying IPC channel) is scoped to
your ``async with`` block — no dangling connections.
``find_actor()``
****************
The workhorse: ask the registrar for ``name`` and connect a portal
to the match, or get ``None`` back when nobody's home:
.. code:: python
async with tractor.find_actor('data_feed') as portal:
if portal is None:
... # not registered anywhere; maybe spawn it?
else:
await portal.run(do_stuff)
Knobs worth knowing:
- ``registry_addrs=[...]``: query specific (possibly multiple,
possibly remote) registrars instead of your tree's default,
- ``only_first=False``: deliver a ``list[Portal]`` of *all*
matches found across the queried registrars instead of just the
first,
- ``raise_on_none=True``: raise a ``RuntimeError`` instead of
yielding ``None`` when no match is found — for when absence is
a hard error in your app.
``wait_for_actor()``
********************
Blocks until *someone* registers under ``name``, then yields a
portal to that registree. Perfect for "wait for my sibling service
to come up" sequencing:
.. code:: python
async with tractor.wait_for_actor('service') as portal:
await portal.run(some_fn)
``query_actor()``
*****************
A lookup *without* connecting to the target: yields an
``(addr, reg_portal)`` pair where ``addr`` is the peer's preferred
transport address, or ``None`` when nothing is registered under
that name. Use it for liveness peeks or to log where a service
lives without actually dialing it up.
``get_registry()``
******************
Yields a portal straight to the registrar actor itself — or a
``LocalPortal`` shim when the calling actor *is* the registrar
(no IPC required to talk to yourself, hopefully).
Fast paths and address preference
---------------------------------
Before doing any RPC to the registrar, every lookup first scans
the calling actor's *already-connected peers*: if you have a live
channel to an actor named ``name`` you get a portal over it
immediately, no registrar round-trip at all.
When a registry entry holds *multiple* addresses (a multihomed
actor) the "best" one is chosen by locality:
1. UDS — same-host guaranteed, lowest overhead,
2. local TCP — loopback or any of this host's own interface
addrs,
3. remote TCP — the only option when actually distributed.
Within a tier the most recently registered addr wins. Stale
entries (an addr that no longer accepts connections) are detected
on use and deleted from the registrar's table on your behalf.
Demo: register and find a service
---------------------------------
The simplest possible spin: start a subactor, ask the registrar
where it lives, and wait on its registration:
.. literalinclude:: ../../examples/service_discovery.py
:caption: examples/service_discovery.py
:language: python
The daemon-service pattern
--------------------------
The classic deployment shape: a long-lived daemon actor serves
RPC, later-running code discovers it by name, calls in, and
gracefully cancels it when the job is done:
.. literalinclude:: ../../examples/service_daemon_discovery.py
:caption: examples/service_daemon_discovery.py
:language: python
Note the teardown ordering — *graceful cancel* of the daemon via
its portal is part of the pattern; under SC a "service" is still
somebody's child and somebody is responsible for reaping it.
Joining an existing tree from outside
-------------------------------------
Discovery isn't limited to a single program: any standalone script
can join a running tree by booting its *own* root actor pointed at
the existing registrar:
.. code:: python
import trio
import tractor
async def main():
async with (
# contact the live tree's registrar
tractor.open_root_actor(
registry_addrs=[('127.0.0.1', 1616)],
),
tractor.find_actor('data_feed') as portal,
):
... # RPC away like you were born here
trio.run(main)
Per the bootstrap rules above, if the registrar at those addrs is
*not* reachable this process simply becomes its own (registrar)
root — so the same code works standalone and as a tree-joiner.
"Arbiter"? A legacy naming note
-------------------------------
In older releases (and many an old blog post or issue thread) the
registrar actor was called the *arbiter*, with matching APIs like
``get_arbiter()`` and an ``arbiter_addr`` argument. All of that
terminology is retired: it's *registrar*/*registry* everywhere now
(``registry_addrs``, ``get_registry()``, ...) and the
``tractor.Arbiter`` export survives only as a back-compat alias of
``tractor.Registrar``. If you see "arbiter" somewhere, mentally
substitute "registrar" and you're up to date.
.. note::
Multihoming nerds: ``tractor.discovery`` also ships
libp2p-style *multiaddr* helpers — ``mk_maddr()`` and
``parse_maddr()`` — for describing transport endpoints as
structured strings.
Very naive, very honest
-----------------------
To be clear, this is a **very naive** discovery system: one
process-tree-local registrar holding a dict, no replication, no
re-election when it dies, no cross-host propagation. That's
intentional (for now); it covers the "wire up my services on this
host" case without dragging in a consensus protocol.
On the roadmap (issue `#216`_ tracks a chunk of it):
- registrar high(er)-availability: staying up past tree teardown
and re-election,
- a `gossip protocol`_ for decentralized cross-host discovery (the
zguide's `discovery`_ chapter is the spiritual reference),
- `modern protocol`_ (rendezvous) style meet-up points.
If any of that scratches your itch, the issue tracker would love
to hear from you.
.. seealso::
- :doc:`/guide/testing` — watching live actor trees (and their
registrar) while the test suite or your app runs.
- API refs: :func:`tractor.find_actor`,
:func:`tractor.wait_for_actor`,
:func:`tractor.query_actor`,
:func:`tractor.get_registry`,
:class:`tractor.Registrar`.
.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol
.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
.. _#216: https://github.com/goodboy/tractor/issues/216

52
docs/guide/index.rst 100644
View File

@ -0,0 +1,52 @@
Guides
======
Task-focused walkthroughs of every major ``tractor``
subsystem, each built around real, *test-suite
verified* example scripts from the repo's
``examples/`` dir (we never copy-paste code into
docs; what you read is what CI runs).
Roughly in "first date to long term relationship"
order,
- :doc:`spawning` — actor nurseries, daemons +
one-shot workers, process lifetimes.
- :doc:`rpc` — portals: calling into another
process like it's a local ``await``.
- :doc:`context` — the cross-actor task-pair
primitive at the heart of modern ``tractor``.
- :doc:`streaming` — one-way and bidirectional
msg streams between actors.
- :doc:`cancellation` — supervision, error
boxing + propagation, teardown discipline.
- :doc:`debugging` — the multi-process native
REPL debugger; our flagship DX feature B)
- :doc:`discovery` — the registrar, finding
actors by name, service patterns.
- :doc:`clustering` — quick flat process
clusters via one ``async with``.
- :doc:`parallelism` — worker pools without
pools; a ``concurrent.futures`` re-think.
- :doc:`asyncio` — "infected asyncio" mode:
SC-supervise ``asyncio`` tasks from ``trio``.
- :doc:`msging` — typed IPC payloads, the wire
msg-spec and custom codecs.
- :doc:`testing` — running + monitoring the
test suite (and testing your own actor apps).
.. toctree::
:hidden:
:maxdepth: 1
spawning
rpc
context
streaming
cancellation
debugging
discovery
clustering
parallelism
asyncio
msging
testing

260
docs/guide/msging.rst 100644
View File

@ -0,0 +1,260 @@
Typed messaging
===============
Every value that crosses an actor boundary rides inside a typed
msg. ``tractor`` ships a small, fixed family of msg types, the
"SC-transitive supervision protocol", which encapsulates *all*
RPC dialogs in the tree such that `structured concurrency`_ (SC)
semantics -- parent-child task linkage, error propagation,
graceful cancellation -- hold across every process hop. On top of
that
protocol you can layer **your own** payload type contracts,
per-endpoint, and have them enforced at runtime by the codec.
.. note::
Older posts and readmes claim ``tractor`` "uses
``msgpack``(-python)" on the wire. The wire *encoding* is still
msgpack, but since ``0.1.0a5`` all codec work is done by
msgspec_ against a strictly-typed, tagged-union msg-spec;
neither ``msgpack-python`` nor ``u-msgpack`` are involved.
The wire format
---------------
Each protocol msg is a :class:`msgspec.Struct` subtype declared
with ``tag=True, tag_field='msg_type'``, so the full set decodes
as a `tagged union`__ with zero dispatch code of our own. The
payload-carrying msgs all inherit from ``PayloadMsg`` which boxes:
__ https://jcristharif.com/msgspec/structs.html#tagged-unions
- ``.cid`` -- the "context id" identifying which dialog (i.e.
which ``Context``) the msg belongs to,
- ``.pld`` -- the *payload*, aka your app's actual data.
Decoding is deliberately two-layered:
- the **transport codec** decodes only the protocol *envelope*,
intentionally leaving ``.pld`` as raw bytes
(:class:`msgspec.Raw`),
- a **per-context payload-receiver** (the internal ``PldRx``) then
decodes each ``.pld`` against *that* dialog's user-defined type
spec.
This split is what lets every ``Context`` carry its own msg-spec
without reconfiguring the shared transport, keeps the runtime's
own traffic immune to your app's spec choices, and makes any
validation failure attributable to exactly one dialog (and thus
one task pair) instead of nuking the whole channel.
The protocol family
-------------------
The entire msg-spec is ten types, all importable from
``tractor.msg`` (defined in ``tractor.msg.types``):
.. list-table::
:header-rows: 1
:widths: 18 82
* - msg type
- role
* - ``Aid``
- actor-identity handshake; the first thing two peers
exchange on connect (name, uuid, pid).
* - ``SpawnSpec``
- parent -> child runtime config sent right after ``Aid``:
enabled modules, registry/bind addrs, runtime vars.
* - ``Start``
- request to remotely schedule an RPC task: target
namespace + func name, kwargs and the caller's uid.
* - ``StartAck``
- the callee's ack declaring the endpoint's "functype":
``asyncfunc``, ``asyncgen`` or ``context``.
* - ``Started``
- the first value passed to ``ctx.started()``; completes
the context handshake.
* - ``Yield``
- one streamed value per ``MsgStream.send()`` call.
* - ``Stop``
- graceful stream termination; the IPC rendition of
``StopAsyncIteration``.
* - ``Return``
- the final return value of the remote task fn.
* - ``CancelAck``
- ``bool`` result of a runtime cancel-request; always
decodable so graceful cancellation can never be broken
by a custom msg-spec.
* - ``Error``
- a boxed remote exception (src uid, relay path, tb str,
..) relayed for local re-raise as
:class:`tractor.RemoteActorError`.
Squint and you'll see an SC task scope serialized onto the wire:
every dialog opens with ``Start``/``StartAck`` (plus ``Started``
for ``@tractor.context`` endpoints), optionally streams
``Yield``-s until a ``Stop``, and **always** terminates with
exactly one of ``Return``, ``Error`` or ``CancelAck``. That 1:1
mapping of msg sequence onto a cross-process task pair is why we
call the protocol *SC-transitive*: supervision semantics survive
every hop of the tree. In `(un)protocol`_ terms it's our "SC
dialog un-protocol".
For introspection the union alias ``tractor.msg.MsgType``, the
list ``__msg_types__`` and the spec alias ``__msg_spec__`` are
all exported.
Payload typing with ``pld_spec``
--------------------------------
By default ``.pld`` may be any msgspec-supported type, i.e. the
spec is ``Any``. To constrain a single endpoint's dialog, pass
a type (union) to the decorator:
``@tractor.context(pld_spec=MyStruct|None)``. The spec then
applies to all payload-carrying msgs of that dialog --
``Started``, ``Yield`` and ``Return`` -- on both sides of the
IPC. Pro tip: keep ``None`` in your union since most endpoints
implicitly ``return None`` and a bare ``ctx.started()`` ships
``None`` too.
.. literalinclude:: ../../examples/typed_payloads.py
:caption: examples/typed_payloads.py
:language: python
What's going on?
- the payload schema is just a :class:`msgspec.Struct` subtype;
anything msgspec can tag and decode works, including unions
of structs, builtins and containers.
- decorating with ``@tractor.context(pld_spec=...)`` attaches the
spec to the endpoint; both peers' payload-receivers now decode
this dialog's payloads against it. No spec sharing files, no
IDL compiler, the contract *is* the Python type.
- the happy path looks identical to untyped code: the child calls
``await ctx.started(<conforming value>)``, streams or returns
more conforming values, and the parent receives fully decoded
struct instances (not dicts!) on its side.
- the sad path is the point: shipping a value *outside* the spec
raises :class:`tractor.MsgTypeError`, which the example catches
to show off the failure mode; see the anatomy section below for
exactly where it gets raised.
Where validation happens: cheap-or-nasty
----------------------------------------
A naive impl would validate every payload on both send *and*
receive, doubling your codec bill exactly where throughput
matters most. Instead ``tractor`` follows the 0mq lords'
"`cheap or nasty`_" pattern: be **nasty** (strict, eager,
expensive) on the rare control msgs and **cheap** (lazy, fast) on
the high-rate stream path.
- ``Started`` is the *only* payload that gets the full nasty
treatment: ``ctx.started(value)`` stringently
**roundtrip-checks** the encoded msg against the dialog's spec
*before* sending, so a non-conforming first value raises
:class:`tractor.MsgTypeError` immediately in the child and
never even hits the wire. (You can opt out per-call with
``ctx.started(..., validate_pld_spec=False)`` if you measure
a real cost.)
- ``Yield`` payloads are **never** checked inside
``MsgStream.send()``; they're validated receiver-side on each
``MsgStream.receive()``. A violation raises a ``MsgTypeError``
in the receiver *and* relays an ``Error`` msg back so the
offending sender gets one raised too.
- the remaining control msgs (``Start``, ``Return``) are likewise
validated such that violations raise in the **sending** actor,
pointing the traceback at the code that actually goofed.
Anatomy of a ``MsgTypeError``
-----------------------------
:class:`tractor.MsgTypeError` is the IPC equivalent of a builtin
``TypeError``: a ``RemoteActorError`` subtype raised whenever
a msg fails to decode against the active spec. The useful bits:
- ``.bad_msg`` -- the offending msg instance (reconstructed from
its wire form when necessary) so you can inspect the actual
``.pld`` that broke the contract.
- ``.expected_msg_type`` -- the protocol msg type the bad msg was
(supposed to be) decoded as, e.g. ``Started[Point]``.
- plus the standard ``RemoteActorError`` goodies: ``.boxed_type``,
``.src_uid``, ``.ipc_msg`` and the fancy ``.pformat()`` tb-box
rendering.
Practical reading guide: a *sender-side* MTE (``Started``,
``Return``) points straight at your offending ``await
ctx.started()`` or ``return`` statement, while a *receiver-side*
MTE (``Yield``) surfaces from the consumer's ``receive()`` call
with the relay copy delivered back to the producer. Either way
the failure is scoped to that one dialog; sibling contexts on the
same channel keep right on trucking.
Custom wire types: ``mk_codec()`` and friends
---------------------------------------------
msgspec covers a wide set of `builtin types`__ natively; for
anything else you teach the codec via extension hooks. The
easiest path is per-endpoint: ``@tractor.context()`` accepts
``enc_hook``/``dec_hook`` params right alongside ``pld_spec``.
For full control build and apply a codec yourself; encode-side:
__ https://jcristharif.com/msgspec/supported-types.html
.. code:: python
from tractor.msg import mk_codec, apply_codec
codec = mk_codec(
enc_hook=nsp_to_str, # your-type -> wire-type
ext_types=[NamespacePath],
)
with apply_codec(codec): # ContextVar-scoped override
... # msgs sent by this task now encode NSPs
and decode-side, scoped to an open context (note the import from
``tractor.msg._ops``, not yet re-exported):
.. code:: python
from tractor.msg._ops import limit_plds
with limit_plds(
NamespacePath,
dec_hook=str_to_nsp, # wire-type -> your-type
ext_types=[NamespacePath],
):
... # this dialog's payloads decode as NSPs
``apply_codec()`` is ``ContextVar``-scoped: it overrides the
codec for the current task (and only that task), not the whole
process. For complete working flows, including hook pairing rules
and roundtrip cases, see ``tests/msg/test_ext_types_msgspec.py``
and ``tests/msg/test_pldrx_limiting.py``.
The runtime dogfoods this pattern with
:class:`tractor.msg.NamespacePath`: a ``str``-subtype shaped like
``'module.path:obj_name'`` used for every RPC target reference.
It ships over the wire as a plain string yet ``.load_ref()``-s
back to the actual object in the receiving actor's memory domain;
a minimal "pointer type" for shared-nothing systems.
Toward capability-based msging
------------------------------
The ``pld_spec`` + codec-hook layer is the foundation for the
long-game: **capability-based msging** where each dialog's
type contract doubles as a capability grant, negotiated as part
of the protocol itself. That work is tracked in `#196`_ (with the
original typed-proto epic in `#36`_); if strongly-typed
distributed systems get you going, we'd love your input.
Where to next?
--------------
.. seealso::
- :doc:`/guide/context` for the dialog API these msgs
implement: ``started()``, streams and results.
- :doc:`/guide/asyncio` for shuttling (typed) payloads into
``asyncio``-land via an infected subactor.
- the msgspec_ docs for everything your payload types can be.
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _msgspec: https://jcristharif.com/msgspec/
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _#196: https://github.com/goodboy/tractor/issues/196
.. _#36: https://github.com/goodboy/tractor/issues/36

176
docs/guide/parallelism.rst 100644
View File

@ -0,0 +1,176 @@
Parallelism and worker pools
============================
The initial ask is almost always the same: *"how do i make a worker
pool?"* — i.e. the thing :mod:`multiprocessing` and
:class:`concurrent.futures.ProcessPoolExecutor` get reached for
once the GIL becomes the enemy.
Here's the structured concurrency (SC) answer: ``tractor`` is built
to handle any SC process tree you can imagine; a "worker pool"
pattern is a trivial special case. So instead of shipping a pool
*class* with knobs bolted on, you compose one from the same two
ingredients used everywhere else in ``tractor``: an actor nursery
and some IPC.
The stdlib baseline
-------------------
For a fair comparison, start from the canonical
:class:`~concurrent.futures.ProcessPoolExecutor` primes example
straight out of the Python docs,
.. literalinclude:: ../../examples/parallelism/concurrent_futures_primes.py
:caption: examples/parallelism/concurrent_futures_primes.py
:language: python
Synchronous code, a hidden thread + IPC machine under the hood, and
an API surface (executors, futures, ``.map()``) invented to paper
over the fact that the pool isn't part of your program's task tree.
Keep an eye on three things for the rewrite: how work is submitted,
how results come back, and what happens when a worker dies.
The ``tractor`` way
-------------------
Now the same workload as a ``tractor`` program,
.. literalinclude:: ../../examples/parallelism/concurrent_actors_primes.py
:caption: examples/parallelism/concurrent_actors_primes.py
:language: python
What's different (and what isn't),
- ``worker_pool()`` is ~30 lines of *your* code: an actor nursery
spawning ``workers`` subactors — each a full process running its
own ``trio`` task tree — kept alive and ready for work until the
block exits; ``enable_modules=[__name__]`` is the capability
allowlist letting them run this module's functions,
- jobs are "submitted" by just... calling the function:
``portal.run(is_prime, n=value)`` runs ``is_prime()`` in a
worker and hands back its result like any local ``await``,
- results stream back through a plain
:func:`trio.open_memory_channel` *as they complete* — no futures
and no polling,
- teardown is one ``await tn.cancel()``
(:meth:`tractor.ActorNursery.cancel`), and any worker crash
triggers the one-cancels-all machinery from
:doc:`/guide/cancellation` — a dead worker can never strand the
pool.
This uses no extra threads, fancy semaphores or futures; all we
need is ``tractor``'s IPC! The full scorecard,
.. list-table::
:header-rows: 1
:widths: 50 50
* - ``concurrent.futures``
- ``tractor``
* - ``ProcessPoolExecutor()``
- ``worker_pool()`` — yours, ~30 lines
* - ``executor.map(is_prime, PRIMES)``
- ``actor_map(is_prime, PRIMES)`` async-gen
* - ``Future`` + internal result queue
- :func:`trio.open_memory_channel`
* - results in input order
- results as they complete
* - worker crash -> ``BrokenProcessPool``
- boxed :class:`tractor.RemoteActorError`
* - pool teardown on ``with`` exit
- one-cancels-all nursery teardown
.. margin:: How many workers?
Same calculus as any process pool: about core-count for
CPU-bound work (the default sizing in
:doc:`/guide/clustering`); more only if workers block on I/O —
though at that point you likely want plain ``trio`` tasks, not
processes.
And because the pool is just SC code, every variation — bounded
submission, per-worker state, streaming partial results (see
:doc:`/guide/streaming`), nested pools — is a local edit to your
pool, not a feature request against an executor class B)
An *async* pool, though?
************************
Yep: RPC targets must be async functions — the runtime rejects a
plain ``def`` with ``TypeError: ... must be an async function!``.
That's not zealotry, it's cancel-responsiveness: each worker is a
full ``trio`` runtime whose msg loop is what hears graceful cancel
requests, and a hot loop that never yields can't be (politely)
interrupted.
Two practical consequences,
- CPU-bound loops should checkpoint once in a while; note how
``burn_cpu()`` in the next example sprinkles ``await
trio.sleep()`` calls so the worker stays responsive while still
pegging a core,
- if some sync call blocks a worker anyway you're still covered:
an unresponsive actor just rides the graceful-then-hard teardown
ladder from :doc:`/guide/cancellation` instead of acking its
cancel — slower, but never a zombie.
Run a func in a process
-----------------------
Even a pool can be overkill; "run this one async func in a
subprocess and give me the result" is a one-liner via
:meth:`tractor.ActorNursery.run_in_actor`,
.. literalinclude:: ../../examples/parallelism/single_func.py
:caption: examples/parallelism/single_func.py
:language: python
``run_in_actor()`` is a *convenience wrapper* — spawn an actor, run
exactly one task in it, reap on result — not the core spawning
model (that's :meth:`tractor.ActorNursery.start_actor` plus
:meth:`tractor.Portal.open_context`; see :doc:`/guide/context`).
But for this fire-and-collect shape it's exactly the right amount
of typing.
As the module docstring suggests, run it under a process-tree
monitor to watch the child appear and get reaped,
.. code:: bash
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/single_func.py \
&& kill $!
You'll see a core get burned in both parent and child — real
parallelism, no GIL sharing, since these are processes (i.e.
*non-shared-memory threads*).
When all you have is sync code
------------------------------
Honesty corner: if your workload is purely *synchronous* functions
and you've zero need for IPC dialogs, streaming, daemons or
supervision trees — i.e. you really do just want
"``ProcessPoolExecutor`` but ``trio``-native" — the smaller,
focused `trio-parallel`_ project may serve you better. ``tractor``
happily covers the use case (as above) but brings a whole runtime
along for the ride. (And when blocking I/O — not the GIL — is the
actual problem, plain in-process :func:`trio.to_thread.run_sync`
may be all you ever needed.)
And to *see* that runtime's process-management story — a per-core
fleet self-destructing with zero zombies left behind — go run
``examples/parallelism/we_are_processes.py``, walked through in
the :doc:`/start/quickstart`.
.. seealso::
- :doc:`/guide/clustering` — the one-liner flat-cluster
convenience (``open_actor_cluster()``) for when even a
hand-rolled pool is too much typing,
- :doc:`/guide/cancellation` — why pool teardown is bulletproof
(graceful-then-hard escalation, no zombies),
- :doc:`/guide/context` — the core per-task API your pool
workers can graduate to.
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel

177
docs/guide/rpc.rst 100644
View File

@ -0,0 +1,177 @@
RPC: calling into other actors
==============================
Every spawn call from :doc:`/guide/spawning` hands you back
a :class:`~tractor.Portal`: a live handle for calling into
another actor's **memory domain**. The name is borrowed from
``trio``'s portal concept — an object you use to submit work
*into* a separate concurrency domain — except here that domain
is a whole other process.
.. d2:: diagrams/runtime_stack.d2
:margin:
:caption: The layers a ``portal.run()`` request rides through.
:alt: app, tractor runtime, IPC channel and OS process layers
There are **no proxy objects** and no special calling
conventions: you pass a plain function reference plus keyword
args, and Python's normal ``await``-able semantics apply. The
function just happens to *run somewhere else*; from the calling
task it looks as though it was called locally. And since this is
all structured concurrency (SC) under the hood, the remote task
runs inside the callee's supervised task tree while its result
— or its failure, as a boxed
:exc:`~tractor.RemoteActorError` — always comes back to *you*.
``Portal.run()``: pass the function, not a string
-------------------------------------------------
:meth:`~tractor.Portal.run` schedules an async function as
a **new task** in the remote actor and waits on its result:
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.start_actor(
'service',
enable_modules=[__name__],
)
answer = await portal.run(movie_theatre_question)
The rules of engagement:
- the target must be an **async function** and its defining
module must be in the callee's ``enable_modules`` allowlist,
else an :exc:`~tractor.ModuleNotExposed` error is relayed
back (see :doc:`/guide/spawning` for the capability-allowlist
story).
- arguments are passed **by keyword only**; they ride the IPC
layer as msgspec_-encoded msgs, so keep them serializable.
- every call schedules a *fresh* task remotely — call it twice
and the callee runs two tasks, each supervised in its own
right.
- remote exceptions re-raise locally as
:exc:`~tractor.RemoteActorError` with the original type
preserved via ``.boxed_type``.
.. note::
Passing dotted-path *strings* to ``run()`` is an ancient,
deprecated form; always pass the function reference. If you
really need name-based addressing use ``run_from_ns()``
below.
Namespaced daemons: ``run_from_ns()``
-------------------------------------
Sometimes the calling process can't (or shouldn't) import the
target function — think a long-running rpc-daemon serving
modules your client never loads. For that,
:meth:`~tractor.Portal.run_from_ns` takes the explicit
namespace path:
.. code:: python
await portal.run_from_ns('mypkg.service', 'ping')
This is literally how ``.run()`` works underneath: the pair is
encoded as a ``'mod.path:func'`` style msg and resolved against
the callee's enabled modules.
One special namespace exists: ``'self'`` resolves to the remote
:class:`~tractor.Actor` instance, i.e. the runtime itself. It's
how internal machinery (cancel requests, registry ops) travels;
don't build your app on it.
One-shot results: ``wait_for_result()``
---------------------------------------
A portal returned from
:meth:`~tractor.ActorNursery.run_in_actor` has exactly one
"main" task running remotely; that task's ``return`` value is
delivered as the portal's *final result*:
.. code:: python
portal = await an.run_in_actor(fib, n=10)
final = await portal.wait_for_result()
Semantics worth knowing:
- it blocks until the remote task returns, re-raising any
remote error in the usual boxed form.
- once resolved it's idempotent: later calls return the same
cached value.
- a *daemon* portal (from ``start_actor()``) has no main task,
so there's no final result to wait for: you'll get a warning
plus a ``NoResult`` sentinel. Results of individual daemon
calls come straight back from each ``await portal.run()``.
Pure RPC daemons: ``run_daemon()``
----------------------------------
When a process's *only* job is to sit at the root of its own
tree and serve RPC, skip the boilerplate with
:func:`tractor.run_daemon`:
.. code:: python
import tractor
tractor.run_daemon(
['mypkg.service'],
name='service',
)
It's a blocking convenience (it calls ``trio.run()`` for you):
boot a root actor with the given modules enabled for RPC, then
sleep until cancelled. Pair it with the discovery system —
:func:`tractor.find_actor` / :func:`tractor.wait_for_actor`
from a *separate* program — and you've got a tiny service
architecture with zero framework ceremony; see
``examples/service_daemon_discovery.py`` for the full pattern.
Fan-out: RPC through nested trees
---------------------------------
Portals compose. An RPC task is just a ``trio`` task, so it can
open its own :class:`~tractor.ActorNursery` and portal into
*its* children — one inbound call fanning out into a whole
sub-tree of work. The mid-tier function from the nested-tree
example:
.. literalinclude:: ../../examples/nested_actor_tree.py
:caption: examples/nested_actor_tree.py (supervisor fan-out)
:language: python
:pyobject: fan_out_squares
The root portals into the ``supervisor`` actor; the
supervisor's RPC task spawns the leaf workers, portals into
each, and returns the combined result back up. Failures at any
depth relay hop-by-hop as boxed errors, and cancelling the root
call tears down the entire sub-tree — SC, transitively.
When to graduate to ``Context``
-------------------------------
``portal.run()`` is great for one-shot, request-response calls.
Reach for :meth:`~tractor.Portal.open_context` with an
``@tractor.context`` endpoint as soon as you want:
- a long-lived dialog with state held on both sides,
- bidirectional streaming via ``ctx.open_stream()``,
- typed payload contracts (``pld_spec``) enforced at the msg
layer,
- or *task-scoped* cancellation: ``Context.cancel()`` cancels
just the linked remote task, whereas
:meth:`~tractor.Portal.cancel_actor` nukes the **entire**
remote runtime and its process.
In fact the source plans for ``Portal.run()`` itself to be
rebuilt on top of ``open_context()`` — contexts *are* the core
inter-actor protocol. Take the full tour in
:doc:`/guide/context`.
.. seealso::
- :doc:`/guide/spawning` — where portals come from and how
their actors are supervised.
- :doc:`/guide/context` — the structured cross-actor task
API: handshake, streaming, typed payloads.
- :doc:`/guide/cancellation` — what happens to in-flight RPC
when trees get torn down.
.. _msgspec: https://jcristharif.com/msgspec/

299
docs/guide/spawning.rst 100644
View File

@ -0,0 +1,299 @@
Spawning actors
===============
If you know trio_ you know the drill: you don't get to launch
a task off into the void, you open a nursery_, the nursery owns
the task, and the block can't exit until every child is done.
That discipline is `structured concurrency`_ (SC) — see the
seminal `blog post`_ if you haven't yet — and it's the whole
religion around here.
``tractor`` applies that exact discipline to **processes**: an
:class:`~tractor.ActorNursery` is a *process nursery*. Every
"task" it starts is a fresh Python process running its own
``trio.run()``-scheduled task tree; we call each one a
``trio``-"*actor*". Parents must wait on (and clean up after)
their children, transitively, all the way down the tree.
.. d2:: diagrams/actor_tree.d2
:caption: A process tree of ``trio``-task-trees.
:alt: a nested actor tree where every parent supervises its children
Though a "process nursery" differs in complexity (and slightly
in semantics) from a single-threaded task nursery, most of the
interface is the same. The main difference is that each spawned
child contains a full, *parallel-executing* ``trio`` task tree.
The following super powers ensue:
- tasks started in a child actor are completely independent of
tasks started in the current process; they execute in
**parallel** and are scheduled by their own actor's ``trio``
run loop.
- tasks scheduled in a remote process still maintain an SC
protocol *across memory boundaries* using a so called
"SC dialogue protocol" which keeps task-hierarchy lifetimes
linked across the IPC layer.
- a remote task can fail and have that failure relayed back to
the caller task (living in some other actor) as a serialized
:exc:`~tractor.RemoteActorError`; no spawned process or RPC
task can ever just go off on its own.
Opening a (process) nursery
---------------------------
:func:`tractor.open_nursery` is the entrypoint:
.. code:: python
async def main():
async with tractor.open_nursery() as an:
... # spawn some actors B)
trio.run(main)
Notice there's no runtime-boot ceremony: if no actor runtime is
up yet (i.e. you're in a plain old Python process),
``open_nursery()`` *implicitly* enters
:func:`tractor.open_root_actor` for you, making this process the
**root actor** of a new tree. Any extra keyword args you pass
are proxied straight through to ``open_root_actor()``, so the
runtime config lives wherever you open your first nursery:
.. code:: python
async with tractor.open_nursery(
loglevel='info',
debug_mode=True, # crash-to-REPL for the whole tree
) as an:
...
If you want the runtime up *without* spawning anything (or you
prefer the config to be loudly explicit) enter
``open_root_actor()`` yourself first; the nursery will detect
the running runtime and skip the implicit boot. Either way,
nesting a second root inside an existing tree is an error.
Inside a *subactor* the same call just works: any actor may open
nurseries of its own, which is how you get arbitrarily deep
trees (more on that below).
``start_actor()``: daemons that live until cancelled
----------------------------------------------------
:meth:`~tractor.ActorNursery.start_actor` is **the** core
spawning primitive. It starts a *daemon* actor: a process with
no designated "main task" besides the runtime itself. It boots,
registers with its parent, and then sits there serving RPC
requests until somebody cancels it. You get back a
:class:`~tractor.Portal` for doing exactly that kind of
somebody-ing:
.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py
:caption: examples/actor_spawning_and_causality_with_daemon.py
:language: python
What's going on here?
- ``start_actor('frank', enable_modules=[__name__])`` forks off
a new process, boots a ``tractor`` runtime inside it, and
allows it to serve functions from the current module (see the
allowlist section below).
- each ``await portal.run(...)`` schedules a *new* task in
frank's task tree and waits on its result — the full RPC story
lives in :doc:`/guide/rpc`.
- frank has no main task to complete, so without the final
``await portal.cancel_actor()`` the nursery block would wait
on him **forever**. Daemon lifetimes are *yours* to end; that
explicitness is the point.
``run_in_actor()``: quick one-shot parallelism
----------------------------------------------
:meth:`~tractor.ActorNursery.run_in_actor` is the convenience
wrapper: spawn an actor, run exactly one async function in it,
then reap the process as soon as the result arrives.
.. code:: python
async with tractor.open_nursery() as an:
portal = await an.run_in_actor(burn_cpu)
# burn rubber in the parent too...
await burn_cpu()
total = await portal.wait_for_result()
A few details worth knowing:
- the actor is named after the function unless you pass
``name='something_cuter'``.
- the function's module is auto-added to the child's
``enable_modules`` allowlist.
- extra ``**kwargs`` are forwarded to the function itself.
- the child is *auto-cancelled* once its "main" result lands;
at nursery exit these run-once children are always reaped
first (causality_ is paramount!).
.. note::
``run_in_actor()`` is a convenience, **not** the core model.
The source literally marks it for an eventual rebuild as
a thin "hilevel" wrapper on top of
:meth:`~tractor.Portal.open_context` (the modern inter-actor
task API). Teach your fingers to use it for quick
fire-and-collect parallelism — think a per-function
trio-parallel_ style one-shot — and reach for
``start_actor()`` + ``open_context()`` for anything
long-lived, stateful or streaming
(:doc:`/guide/context`).
Actor lifetimes and teardown order
----------------------------------
So we have two lifetime flavors:
- **run-once** (``run_in_actor()``): lives exactly as long as
its single task; reaped the moment its result (or error)
arrives.
- **daemon** (``start_actor()``): lives until *someone* cancels
it — an explicit ``await portal.cancel_actor()``, a bulk
``await an.cancel()``, or the one-cancels-all strategy kicking
in on error.
On a clean exit of the nursery block the teardown order is:
1. the nursery waits on every run-once actor's final result;
any errors from these are raised immediately so your code
(acting as supervisor) gets first crack at handling them.
2. then it waits on daemon actors — **indefinitely**. If you
spawned a daemon, you own its lifetime.
When a child *is* cancelled, teardown is graceful-first per SC
discipline: the runtime sends an IPC cancel request and gives
the child a bounded window to ack; only when a child is too
slow does the nursery escalate to an OS-level hard kill of the
process. There is no path where a child is silently left
running:
``tractor`` tries to protect you from zombies, no matter
what. If you can create zombie child processes (without
using a system signal) it **is a bug**.
Per-process cleanup hooks
*************************
Need something torn down when an actor's runtime exits, no
matter how it exits? Every actor carries
a process-global :class:`contextlib.ExitStack` at
``Actor.lifetime_stack`` which is closed at the very end of
runtime teardown:
.. code:: python
db = await connect_db()
tractor.current_actor().lifetime_stack.callback(db.close)
(A so-far under-advertised api — expect it to get more love.)
When things blow up: one-cancels-all
------------------------------------
The default (and currently only) supervision strategy is the
same one ``trio`` nurseries use: **one-cancels-all**. If your
nursery-block body errors, every child actor is cancelled. If
a child errors, the failure is relayed to the nursery as a
boxed :exc:`~tractor.RemoteActorError` (original type preserved
via ``.boxed_type``), all *other* children are cancelled, and
the error(s) re-raise locally — exactly like ``trio``, just
process-wide. Erlang-style alternative strategies are a long
standing roadmap item.
The full story — how cancel requests relay across the tree, who
``.canceller`` was, debugging mid-teardown — lives in
:doc:`/guide/cancellation`.
The module allowlist: ``enable_modules``
----------------------------------------
A subactor will only serve functions from modules its parent
*explicitly* enabled at spawn time:
.. code:: python
portal = await an.start_actor(
'service',
enable_modules=['mypkg.service'], # or [__name__]
)
At child boot the runtime imports each listed module so inbound
RPC requests can resolve function references against it. Ask
a peer to run something from any *other* module and you get an
:exc:`~tractor.ModuleNotExposed` error relayed back — the child
never even looks the function up.
Think of it as the first, deliberately coarse layer of
capability-style permissioning: if you don't hand an actor
a module, no peer can invoke anything inside it. (Finer-grained
capability-based messaging protocols are on the roadmap.)
The ``enable_modules=[__name__]`` idiom — "let the child run
functions from the *current* module" — is what you'll use in
most scripts; bigger apps tend to pass dedicated service-module
paths instead.
Per-child knobs
---------------
Both spawn methods accept per-child config so one weird child
doesn't have to drag the whole tree along:
- ``loglevel='cancel'`` — crank console logging for just this
subactor (the ``TRACTOR_LOGLEVEL`` env var overrides whatever
the *root* was passed, handy for test runs).
- ``debug_mode=True`` — arm the crash-handling REPL machinery
for just this child instead of tree-wide, i.e. the selective
flavor of ``open_nursery(debug_mode=True)``; see
:doc:`/guide/debugging` for the multi-process debugger tour.
- ``infect_asyncio=True`` — run the child with ``trio`` as an
``asyncio`` guest, aka "infected asyncio" mode.
- ``enable_transports=['uds']`` — pick the IPC transport this
child should listen on (default ``'tcp'``).
Trees all the way down
----------------------
Since any actor can open an ``ActorNursery``, supervision trees
compose to arbitrary depth: a subactor can be a supervisor of
*its own* subactors, with every level holding the same SC
guarantees — error relay up, cancellation down, no orphans.
.. literalinclude:: ../../examples/nested_actor_tree.py
:caption: examples/nested_actor_tree.py
:language: python
Here the root spawns a ``supervisor`` actor whose RPC task opens
its *own* nursery and spawns the leaf workers; one call from the
root fans out through the middle layer and the aggregate comes
back up. Teardown ripples in reverse: the leaves are reaped when
the supervisor's nursery exits, the supervisor when the root
cancels it.
Watching your tree grow
-----------------------
Actors are real processes, so your favorite system tools just
work. The house incantation runs any example beside a live
process-tree monitor::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/nested_actor_tree.py \
&& kill $!
Every subactor also sets its OS process title to a stable
``_subactor[<name>@<uuid-prefix>]`` marker, so ``htop``,
``ps`` and friends show *which actor is which* at a glance::
pgrep -af '_subactor\['
.. seealso::
- :doc:`/guide/rpc` — actually invoking functions through
all these portals you've been collecting.
- :doc:`/guide/context` — the structured, streaming-capable
inter-actor task API.
- :doc:`/guide/cancellation` — cross-actor cancellation and
error propagation semantics in depth.
.. _trio: https://github.com/python-trio/trio
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel

244
docs/guide/streaming.rst 100644
View File

@ -0,0 +1,244 @@
Cross-process streaming
=======================
Spawning processes is the boring half of ``tractor``: the **real
cool stuff** is the native support for cross-process *streaming*.
Yes, you saw it here first — 2-way msg streams with reliable,
transitive setup/teardown semantics, wired straight into the
runtime's `structured concurrency`_ (SC) supervision machinery so
that *how* a stream ends is part of the protocol.
No broker, no topic exchange, no IDL compiler. The IPC layer is a
deliberately "`cheap or nasty`_" `(un)protocol`_: a tiny set of
msgspec_-typed msgs over a transport (TCP or UDS today) with
payload typing opt-in per dialog — handshake msgs get the *nasty*
treatment (strict validation) while high-rate stream payloads
stay *cheap* (receiver-side checks only). See
:doc:`/guide/context` for the typed ``pld_spec`` contract bits.
Two ways to stream
------------------
.. margin:: It's a ``trio.abc.Channel``
:class:`tractor.MsgStream` implements
:class:`trio.abc.Channel` — ``send()``,
``receive()``, async-iteration, ``aclose()`` —
so trio-generic channel code drives an IPC
stream unchanged.
- **Bidirectional, context-based**: open a
:class:`tractor.Context` to a peer task then enter
``ctx.open_stream()`` for a full-duplex
:class:`tractor.MsgStream`. This is the modern core API, taught
end-to-end in :doc:`/guide/context`; we won't re-teach it here.
- **One-way, portal-based**: point
:meth:`tractor.Portal.open_stream_from` at a plain async
generator fn in the peer actor. Legacy, but perfectly fine for
simple produce/consume pipelines — and it powers the classic
examples below.
Rule of thumb: if the consumer ever needs to *talk back* — acks,
control msgs, a final result — use a context. If it's a pure
pipeline stage, either works and the one-way form is less typing.
One-way streaming from an async generator
-----------------------------------------
The OG api. Write an async generator in the target actor's
module; iterate its yields from the spawning side:
.. literalinclude:: ../../examples/asynchronous_generators.py
:caption: examples/asynchronous_generators.py
:language: python
Each ``yield`` crosses the process boundary as one msg and feeds
the parent's ``async for``. When the consumer ``break``\ s out
and exits the ``open_stream_from()`` block the far-end generator
task is cancelled for you: the producer's lifetime is *coupled to
the consumer's scope* so a one-way stream can never leak a remote
task.
Any extra kwargs (``stream_data, seed=100`` style) are forwarded
to the remote generator's call, and a non-async-gen target is
rejected up front with a ``TypeError``.
.. note::
No decorator required — any plain async-gen fn works. You may
still meet ``@tractor.stream`` in the wild; it's the legacy
marker for one-way endpoints and sticks around only for
compat (heads up: the param name ``ctx`` is reserved for
``@context`` endpoints nowadays, so legacy fns should call
theirs ``stream``). New code wanting anything fancier than a
one-way pipe should use :func:`tractor.context` +
``ctx.open_stream()``.
.. warning::
One-way means one way: there's no sending *to* the generator
side and no graceful consumer-to-producer stop msg — the
teardown above is cancel-based. Needing upstream control flow
is the sign you've outgrown this API.
A full-fledged streaming service
--------------------------------
Now let's get fancy: compose one-way streams through a nested
actor tree and you've got yourself a fan-in pipeline.
.. d2:: diagrams/streaming_pipeline.d2
:caption: Four actors, three streams, one deduped feed.
:alt: two streamer actors fan in to an aggregator then root
.. literalinclude:: ../../examples/full_fledged_streaming_service.py
:caption: examples/full_fledged_streaming_service.py
:language: python
What's going on?
- the root actor spawns ``'aggregator'`` which opens its *own*
actor nursery and spawns ``'streamer_1'`` + ``'streamer_2'``: 4
processes total, supervision nested two levels deep with zero
special casing.
- ``aggregate()`` opens a one-way stream from each streamer and
fans both into a single :func:`trio.open_memory_channel` via
one local trio task per portal — in-actor fan-in riding trio's
built-in backpressure end-to-end.
- duplicates get dropped via a ``set`` and the deduped sequence
is *re-yielded* upward: ``aggregate()`` is itself an async gen
being consumed over IPC by the root. Streams compose.
- when the seed runs out the streamer gens finish, the memory
channel drains closed, the aggregator's gen returns and the
root's ``async for`` ends; ``await an.cancel()`` then reaps the
subtree. Every exit is awaited — if you can produce a zombie
process from this, it **is a bug**.
Watch the tree breathe while it runs, using the README's
signature process-monitor incantation::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/full_fledged_streaming_service.py \
&& kill $!
No extra threads, no fancy semaphores, no futures; all we need is
``tractor``'s IPC.
Two streams, one portal
-----------------------
Every ``open_stream_from()`` call starts its *own* remote task —
even through the same portal — so two local consumer tasks can
independently stream the same generator fn concurrently, both
dialogs multiplexed over the single underlying IPC channel:
.. literalinclude:: ../../examples/multiple_streams_one_portal.py
:caption: examples/multiple_streams_one_portal.py
:language: python
The add-else-remove trick on the shared ``consumed`` list is the
proof: each value arrives in *both* streams, getting appended by
whichever task sees it first and removed by the other, so the
list always ends up empty. Two streams, same data, zero
interference.
This works because every dialog is keyed by its own context id
(``Context.cid``) — any number of concurrent streams, contexts
and one-shot RPCs share a single underlying
:class:`tractor.Channel` per peer pair.
Fan-out inside an actor: ``MsgStream.subscribe()``
--------------------------------------------------
The inverse pattern: *one* IPC stream feeding *many* local tasks.
Instead of paying for N redundant cross-process streams, call
:meth:`tractor.MsgStream.subscribe` to get a
``BroadcastReceiver`` — a tokio-style broadcast channel from
``tractor.trionics`` — which copies every received value to each
subscribed task:
.. literalinclude:: ../../examples/streaming_broadcast_fanout.py
:caption: examples/streaming_broadcast_fanout.py
:language: python
Each task entering ``stream.subscribe()`` receives its own copy
of everything sent from that point on. The underlying stream
keeps pace with the *fastest* subscriber; a task falling more
than the buffered window behind has its next receive raise
``tractor.trionics.Lagged`` to say it lost data.
The broadcast handle stays duplex btw: it proxies ``send()``
through to the underlying stream, so each subscriber task can
keep talking upstream while consuming its fan-out copy.
.. warning::
``.subscribe()`` is **idempotent and non-reversible**: the
first call permanently swaps the stream's receive machinery
over to the internally allocated broadcaster. There's no
un-subscribing back to the raw stream, so make sure you're ok
with the (theoretical) overhead before opting in.
Consuming: ``async for`` and friends
------------------------------------
``async for msg in stream:`` is just sugar over repeated
``await stream.receive()``. The receive-side surface:
- ``receive()`` — next msg, or raises :exc:`trio.EndOfChannel`
on a graceful far-end close (``async for`` translates that
into a clean loop exit for you).
- ``receive_nowait()`` — opportunistic, non-blocking drain.
- ``closed`` — property flagging an already-ended stream.
Send-side it's just ``await stream.send(data)`` — one ``Yield``
msg per call carrying any msgspec_-encodable payload (or
whatever your ``pld_spec`` permits, see :doc:`/guide/context`).
End-of-stream: close vs. cancel
-------------------------------
How a stream ends is part of the protocol; the runtime keeps the
polite case and the violent case distinct:
- **graceful close**: the far side exits its stream block, its
async gen returns, or it calls ``await stream.aclose()``. A
``Stop`` msg is sent so your ``async for`` simply ends
(``StopAsyncIteration``, via :exc:`trio.EndOfChannel` under the
hood). A normal, non-error ending — the dialog's result phase
proceeds as usual.
- **cancel or error**: no ``Stop`` is sent. Instead the
cancel/error itself is relayed so the far end *knows* the
dialog did not end on purpose and raises accordingly — a
:exc:`tractor.ContextCancelled`, a boxed
:exc:`tractor.RemoteActorError`, etc. See the cancellation
section of :doc:`/guide/context` for exactly who raises what.
Tying it together: every ``MsgStream`` is **one-shot use**. Both
endings are final — once closed a stream can't be re-opened and
the supported "retry" is opening a fresh :class:`tractor.Context`
(they're cheap).
.. seealso::
- :doc:`/guide/context` — the full ``Context`` lifecycle: the
handshake, results, cancellation semantics and the
overrun/backpressure knobs.
- :class:`tractor.MsgStream` and
:meth:`tractor.Portal.open_stream_from` API docs.
- The zguide chapters our wire philosophy is named after:
"`cheap or nasty`_" and `(un)protocol`_\ s.
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _msgspec: https://jcristharif.com/msgspec/

256
docs/guide/testing.rst 100644
View File

@ -0,0 +1,256 @@
Testing tips
============
``tractor``'s test suite is a different kind of beast than your
average single-proc pytest run: nearly every test spawns a real
**process tree**, hammers on cancellation under structured
concurrency (SC), and tears the whole thing down again — hundreds
of times per session. This page collects the tips, knobs and
one-liners that make hacking on (and with) the suite pleasant.
Running the suite
-----------------
This is a uv_-managed project, so after cloning it's just::
uv sync --dev
uv run pytest tests/
Expect a *lot* of process churn; the suite is effectively a
rolling chaos exercise for the runtime.
The classic fix-iterate loop when something breaks::
# stop at the first failure
uv run pytest tests/ -x
# then iterate on just the failures til green
uv run pytest --lf -x
``--lf`` (last-failed) re-runs only what failed previously, so
combined with ``-x`` you get a tight one-test-at-a-time repair
loop.
Suite-specific flags
********************
The repo auto-loads the bundled ``tractor._testing.pytest`` plugin
(via ``addopts`` in ``pyproject.toml``) which adds a few extra
flags:
- ``--spawn-backend <key>``: pick the process spawn backend for
the session (default ``'trio'``); same keys as the
``start_method`` runtime argument,
- ``--tpt-proto <key> [...]``: which IPC transport(s) opting-in
suites should run against, eg. ``--tpt-proto uds``,
- ``--tpdb`` / ``--debug-mode``: flip on the ``debug_mode``
fixture so debugger-aware tests boot their trees with the
crash-REPL enabled,
- ``--enable-stackscope``: install the ``SIGUSR1`` task-tree dump
handler in pytest *and* every spawned subactor — much lighter
than a full debug-mode run when you only need stack visibility
during a hang hunt,
- ``--ll <level>`` / ``--tl <level-or-spec>``: console loglevels;
``--tl`` targets the ``tractor``-as-runtime logger and accepts
a per-subsystem spec like ``'devx:runtime,trionics:cancel'``.
Watch the tree grow
-------------------
The single most useful trick while the suite (or any ``tractor``
app) runs: keep a live ``pstree`` view going in a side terminal::
watch -n 0.1 "pstree -a $(pgrep -f pytest)"
You'll see actor processes pop in and out of existence as each
test builds and reaps its tree. Launch it *after* pytest is up
(the pid is substituted once, at ``watch`` startup).
Every subactor also sets its OS process title (via
``setproctitle``) to ``_subactor[<name>@<uuid-prefix>]`` so the
tree view shows *which actor is which* at a glance — and targeted
greps stay easy::
pgrep -af '_subactor\['
For a single example script, the repo's signature incantation
spawns the watcher alongside your program and cleans it up after::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/single_func.py \
&& kill $!
Env-var knobs
-------------
Two env-vars override their corresponding runtime arguments
*globally* — no application (or test) code changes required:
``TRACTOR_SPAWN_METHOD``
Wins over any caller-passed ``start_method`` so you can drive
the whole suite (or any app) under a different spawn backend::
TRACTOR_SPAWN_METHOD=mp_spawn uv run pytest tests/ -x
``TRACTOR_LOGLEVEL``
Wins over any caller-passed ``loglevel``; crank (or silence)
runtime console verbosity wholesale::
TRACTOR_LOGLEVEL=cancel uv run pytest tests/ -x -s
``TRACTOR_ENABLE_STACKSCOPE``
Force-install the ``SIGUSR1`` task-tree dump handler in every
actor, debug-mode or not; then
``pkill --signal SIGUSR1 -f <part-of-cmd>`` dumps every
actor's live ``trio`` task tree.
Debug mode vs. pytest capture
-----------------------------
The tree-wide crash-to-REPL experience (``debug_mode=True`` plus
``await tractor.pause()``) requires a **real tty**, and pytest's
default output capturing swallows exactly that. When you want to
interact with the REPL from inside a test run, disable capture::
uv run pytest tests/test_foo.py -x -s
(``-s`` is shorthand for ``--capture=no``.)
Tests should request the ``debug_mode`` fixture (driven by the
``--tpdb`` flag) rather than hard-coding it, so that normal CI
runs stay non-interactive.
For *automated* REPL interaction — asserting on prompt output,
sending debugger commands — you can't just turn capture off;
instead do what ``tests/devx/`` does: drive a child Python program
through pexpect_ on a real pseudo-tty and pattern-match the
``(Pdb+)`` prompts. See ``tests/devx/test_debugger.py`` for many
worked patterns.
Examples *are* tests
--------------------
Every script under ``examples/`` is run as a subprocess by
``tests/test_docs_examples.py``; since these docs
``literalinclude`` those same scripts, the code you read here is
CI-verified on every push and can never silently rot B)
Conventions when adding a new example:
- make it a standalone runnable script with the usual guard::
if __name__ == '__main__':
trio.run(main)
- it must exit cleanly (returncode ``0``) within the per-example
timeout (~16s locally, with headroom auto-added in CI and under
cpu-freq scaling) — keep sleeps short,
- any stderr line containing ``Error`` fails the test, so silence
or assert-around expected error output,
- don't crank ``tractor`` logging inside an example: subprocess
pipe **backpressure can deadlock** the run (ask us how we
know..),
- filenames starting with ``_`` are skipped (the WIP convention),
as are the special subdirs (``debugging/``, ``integration/``,
``advanced_faults/``, ``trio/``) which are driven by their own
dedicated suites instead.
Drop your script in, run the example suite, profit::
uv run pytest tests/test_docs_examples.py -x
Zombie cleanup
--------------
First, the contract: ``tractor`` **always** reaps its children —
if you can create a zombie process (without resorting to
untrappable signals) it **is a bug**, please report it!
That said, while hacking on the *runtime itself* you can
definitely wedge things — a ``SIGKILL``-ed pytest, a half-broken
spawn backend — and strand subactor procs plus their shm segments
and UDS socket files. The repo ships a dedicated cleanup tool::
uv run scripts/tractor-reap --shm --uds
It's SC-polite even as a reaper: matched processes get ``SIGINT``
first with a bounded grace window — so actor runtimes can run
their ``trio`` teardown paths — escalating to ``SIGKILL`` only as
a last resort. The ``--shm`` sweep unlinks ``/dev/shm/`` segments
that no live process has open (it leans on psutil_, already in
your dev venv, to check live mappings and fds) and ``--uds``
clears socket files whose binder pid is dead.
Testing your own ``tractor`` app
--------------------------------
The same plugin the suite uses ships in the package, so your
project can load it too::
[tool.pytest.ini_options]
addopts = ['-p tractor._testing.pytest']
That buys you the CLI flags above plus a set of fixtures —
``loglevel``, ``debug_mode``, ``reg_addr`` (a session-unique
registrar address so concurrent runs and other live ``tractor``
apps on the host can't cross-talk) — and the ``@tractor_test``
decorator:
.. code:: python
import tractor
from tractor._testing import tractor_test
@tractor_test
async def test_my_service(
reg_addr: tuple,
loglevel: str,
):
# already inside a root actor's trio task!
async with tractor.open_nursery() as an:
...
The decorator boots a root actor around your (async) test fn,
wires any of the special fixtures you declare (``reg_addr``,
``loglevel``, ``start_method``, ``debug_mode``) into
``open_root_actor()``, and runs the body as the root-most task
under a wall-clock ``trio.fail_after()`` guard.
General advice that has served this suite well:
- bound waits with ``trio.fail_after()`` *inside* tests; global
pytest timeout plugins interact badly with multi-process
``trio`` teardown,
- use the ``reg_addr`` fixture (or otherwise randomize your
registry addrs) so leftover registrars from prior runs can't
contaminate lookups,
- assert on **structured outcomes** — eg.
``RemoteActorError.boxed_type`` or
``ContextCancelled.canceller`` — not on log text.
.. note::
``tractor._testing`` is still an underscore-internal namespace:
shipped and handy, but its API may shift between alpha
releases.
(This page exists thanks to the ask in `#126`_.)
.. seealso::
- :doc:`/guide/discovery` — how registrar wiring (the thing
``reg_addr`` randomizes) works in the runtime proper.
- :doc:`/project/dev-tips` — contributor-oriented extras:
releases, log-system tracing, tree-monitoring recipes.
.. _uv: https://docs.astral.sh/uv/
.. _pexpect: https://pexpect.readthedocs.io/en/stable/
.. _psutil: https://psutil.readthedocs.io/en/latest/
.. _#126: https://github.com/goodboy/tractor/issues/126

752
docs/index.rst
View File

@ -1,612 +1,154 @@
.. tractor documentation master file, created by .. image:: _static/tractor_logo_side.svg
sphinx-quickstart on Sun Feb 9 22:26:51 2020. :class: hero-logo
You can adapt this file completely to your liking, but it should at least :align: center
contain the root `toctree` directive. :alt: tractor
``tractor`` tractor
=========== =======
**distributed structured concurrency**: a
multi-processing runtime built on (and shaped
entirely like) trio_.
A `structured concurrent`_, async-native "`actor model`_" built on trio_ and multiprocessing_. ``tractor`` provides parallelism via ``trio``
*"actors"*: independent Python **processes** (ie.
*non-shared-memory threads*) each running a ``trio``
task tree, all composed into a *distributed
supervision tree* with end-to-end `structured
concurrency`_ (SC) — spawning, cancellation, error
propagation and teardown that work **across
processes** (and hosts) exactly the way they work
across tasks.
.. margin:: tl;dr
It's **just** ``trio``, but with nurseries that
spawn *processes* and streams that cross them. If
you can read a ``trio`` program you can read a
``tractor`` one — that's the whole pitch.
Sixty seconds of why
--------------------
Spawn one actor per core, crash the root on purpose,
and watch the runtime contain the blast: errors
propagate, *every* child is reaped, zero zombies —
guaranteed (it's a bug otherwise).
.. literalinclude:: ../examples/parallelism/we_are_processes.py
:caption: examples/parallelism/we_are_processes.py
:language: python
Like every snippet in these docs this file lives in
the repo's ``examples/`` dir and runs under CI — docs
code that can't rot.
Dig in
------
.. grid:: 1 2 2 3
:gutter: 3
.. grid-item-card:: Get started
:link: start/index
:link-type: doc
Install + your first actor tree in ~20 lines;
causality, daemons and the trynamic scene.
.. grid-item-card:: The big ideas
:link: explain/sc-distributed
:link-type: doc
SC across processes, distilled — then the
runtime architecture under it.
.. grid-item-card:: Debug like a local
:link: guide/debugging
:link-type: doc
``await tractor.pause()`` anywhere in the
tree: one terminal, every process, zero
socket-juggling.
.. grid-item-card:: Streaming + contexts
:link: guide/context
:link-type: doc
Bidirectional, cancellation-safe msg streams
between any two actors.
.. grid-item-card:: Guides
:link: guide/index
:link-type: doc
RPC, supervision, clustering, "infected
asyncio", typed msging + more.
.. grid-item-card:: API reference
:link: api/index
:link-type: doc
The curated public surface; everything
importable from ``tractor``.
Features
--------
- **It's just a** ``trio`` **API** — same nursery
discipline, same cancellation semantics, one level
up the process tree.
- *Infinitely nestable* process trees: sub-actors can
spawn sub-actors, supervision stays transitive.
- A "native UX" **multi-process debugger REPL**:
built on pdbp_ with tree-wide tty locking (see
:doc:`guide/debugging`).
- Built-in, cancellation-safe **bidirectional
streaming** via a `cheap or nasty`_ `(un)protocol`_.
- **Typed IPC**: `msgspec`_-backed wire msgs with
optional per-dialog payload specs
(:doc:`guide/msging`).
- Swappable process-spawn backends + modular IPC
transports (TCP today, UDS on same-host, more
planned).
- Optionally distributed_: the same APIs work over
multiple hosts as on multiple cores.
- "**Infected** ``asyncio``" mode: SC-supervise
``asyncio`` tasks from ``trio``
(:doc:`guide/asyncio`).
- ``trio`` extension goodies via ``tractor.trionics``
(acm gathering, single-resource caching, broadcast
channels).
Where do i start!?
------------------
The first step to grok ``tractor`` is to get an
intermediate knowledge of ``trio`` and **structured
concurrency** B)
Some great places to start are,
- the seminal `blog post`_,
- obviously the `trio docs`_,
- wikipedia's nascent SC_ page,
- the fancy diagrams @ libdill-docs_,
then come back and hit :doc:`start/quickstart`.
.. toctree:: .. toctree::
:maxdepth: 1 :hidden:
:caption: Contents: :maxdepth: 2
Get started <start/index>
Big ideas <explain/index>
Guides <guide/index>
API <api/index>
Project <project/index>
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _trio: https://github.com/python-trio/trio .. _trio: https://github.com/python-trio/trio
.. _multiprocessing: https://en.wikipedia.org/wiki/Multiprocessing .. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles .. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich
.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228
``tractor`` is an attempt to bring trionic_ `structured concurrency`_ to
distributed multi-core Python; it aims to be the Python multi-processing
framework *you always wanted*.
``tractor`` lets you spawn ``trio`` *"actors"*: processes which each run
a ``trio`` scheduled task tree (also known as an `async sandwich`_).
*Actors* communicate by exchanging asynchronous messages_ and avoid
sharing any state. This model allows for highly distributed software
architecture which works just as well on multiple cores as it does over
many hosts.
The first step to grok ``tractor`` is to get the basics of ``trio`` down.
A great place to start is the `trio docs`_ and this `blog post`_.
.. _messages: https://en.wikipedia.org/wiki/Message_passing
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ .. _trio docs: https://trio.readthedocs.io/en/latest/
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
.. _pdbp: https://github.com/mdmintz/pdbp
Install .. _msgspec: https://jcristharif.com/msgspec/
------- .. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
No PyPi release yet! .. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
.. _distributed: https://en.wikipedia.org/wiki/Distributed_computing
::
pip install git+git://github.com/goodboy/tractor.git
Feel like saying hi?
--------------------
This project is very much coupled to the ongoing development of
``trio`` (i.e. ``tractor`` gets all its ideas from that brilliant
community). If you want to help, have suggestions or just want to
say hi, please feel free to ping me on the `trio gitter channel`_!
.. _trio gitter channel: https://gitter.im/python-trio/general
Philosophy
----------
Our tenets non-comprehensively include:
- strict adherence to the `concept-in-progress`_ of *structured concurrency*
- no spawning of processes *willy-nilly*; causality_ is paramount!
- (remote) errors `always propagate`_ back to the parent supervisor
- verbatim support for ``trio``'s cancellation_ system
- `shared nothing architecture`_
- no use of *proxy* objects or shared references between processes
- an immersive debugging experience
- anti-fragility through `chaos engineering`_
``tractor`` is an actor-model-*like* system in the sense that it adheres
to the `3 axioms`_ but does not (yet) fulfil all "unrequirements_" in
practise. It is an experiment in applying `structured concurrency`_
constraints on a parallel processing system where multiple Python
processes exist over many hosts but no process can outlive its parent.
In `erlang` parlance, it is an architecture where every process has
a mandatory supervisor enforced by the type system. The API design is
almost exclusively inspired by trio_'s concepts and primitives (though
we often lag a little). As a distributed computing system `tractor`
attempts to place sophistication at the correct layer such that
concurrency primitives are powerful yet simple, making it easy to build
complex systems (you can build a "worker pool" architecture but it's
definitely not required). There is first class support for inter-actor
streaming using `async generators`_ and ongoing work toward a functional
reactive style for IPC.
.. warning:: ``tractor`` is in alpha-alpha and is expected to change rapidly!
Expect nothing to be set in stone. Your ideas about where it should go
are greatly appreciated!
.. _concept-in-progress: https://trio.discourse.group/t/structured-concurrency-kickoff/55
.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts
.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony
.. _async generators: https://www.python.org/dev/peps/pep-0525/
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
.. _channels: https://en.wikipedia.org/wiki/Channel_(programming)
.. _chaos engineering: http://principlesofchaos.org/
Examples
--------
Note, if you are on Windows please be sure to see the :ref:`gotchas
<windowsgotchas>` section before trying these.
A trynamic first scene
**********************
Let's direct a couple *actors* and have them run their lines for
the hip new film we're shooting:
.. literalinclude:: ../examples/a_trynamic_first_scene.py
We spawn two *actors*, *donny* and *gretchen*.
Each actor starts up and executes their *main task* defined by an
async function, ``say_hello()``. The function instructs each actor
to find their partner and say hello by calling their partner's
``hi()`` function using something called a *portal*. Each actor
receives a response and relays that back to the parent actor (in
this case our "director" executing ``main()``).
Actor spawning and causality
****************************
``tractor`` tries to take ``trio``'s concept of causal task lifetimes
to multi-process land. Accordingly, ``tractor``'s *actor nursery* behaves
similar to ``trio``'s nursery_. That is, ``tractor.open_nursery()``
opens an ``ActorNursery`` which **must** wait on spawned *actors* to complete
(or error) in the same causal_ way ``trio`` waits on spawned subtasks.
This includes errors from any one actor causing all other actors
spawned by the same nursery to be cancelled_.
To spawn an actor and run a function in it, open a *nursery block*
and use the ``run_in_actor()`` method:
.. literalinclude:: ../examples/actor_spawning_and_causality.py
What's going on?
- an initial *actor* is started with ``trio.run()`` and told to execute
its main task_: ``main()``
- inside ``main()`` an actor is *spawned* using an ``ActorNusery`` and is told
to run a single function: ``cellar_door()``
- a ``portal`` instance (we'll get to what it is shortly)
returned from ``nursery.run_in_actor()`` is used to communicate with
the newly spawned *sub-actor*
- the second actor, *some_linguist*, in a new *process* running a new ``trio`` task_
then executes ``cellar_door()`` and returns its result over a *channel* back
to the parent actor
- the parent actor retrieves the subactor's *final result* using ``portal.result()``
much like you'd expect from a future_.
This ``run_in_actor()`` API should look very familiar to users of
``asyncio``'s `run_in_executor()`_ which uses a ``concurrent.futures`` Executor_.
Since you might also want to spawn long running *worker* or *daemon*
actors, each actor's *lifetime* can be determined based on the spawn
method:
- if the actor is spawned using ``run_in_actor()`` it terminates when
its *main* task completes (i.e. when the (async) function submitted
to it *returns*). The ``with tractor.open_nursery()`` exits only once
all actors' main function/task complete (just like the nursery_ in ``trio``)
- actors can be spawned to *live forever* using the ``start_actor()``
method and act like an RPC daemon that runs indefinitely (the
``with tractor.open_nursery()`` won't exit) until cancelled_
Here is a similar example using the latter method:
.. literalinclude:: ../examples/actor_spawning_and_causality_with_daemon.py
The ``enable_modules`` `kwarg` above is a list of module path
strings that will be loaded and made accessible for execution in the
remote actor through a call to ``Portal.run()``. For now this is
a simple mechanism to restrict the functionality of the remote
(and possibly daemonized) actor and uses Python's module system to
limit the allowed remote function namespace(s).
``tractor`` is opinionated about the underlying threading model used for
each *actor*. Since Python has a GIL and an actor model by definition
shares no state between actors, it fits naturally to use a multiprocessing_
``Process``. This allows ``tractor`` programs to leverage not only multi-core
hardware but also distribute over many hardware hosts (each *actor* can talk
to all others with ease over standard network protocols).
.. _task: https://trio.readthedocs.io/en/latest/reference-core.html#tasks-let-you-do-multiple-things-at-once
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality
.. _cancelled: https://trio.readthedocs.io/en/latest/reference-core.html#child-tasks-and-cancellation
.. _run_in_executor(): https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor
.. _Executor: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor
Cancellation
************
``tractor`` supports ``trio``'s cancellation_ system verbatim.
Cancelling a nursery block cancels all actors spawned by it.
Eventually ``tractor`` plans to support different `supervision strategies`_ like ``erlang``.
.. _supervision strategies: http://erlang.org/doc/man/supervisor.html#sup_flags
Remote error propagation
************************
Any task invoked in a remote actor should ship any error(s) back to the calling
actor where it is raised and expected to be dealt with. This way remote actors
are never cancelled unless explicitly asked or there's a bug in ``tractor`` itself.
.. literalinclude:: ../examples/remote_error_propagation.py
You'll notice the nursery cancellation conducts a *one-cancels-all*
supervisory strategy `exactly like trio`_. The plan is to add more
`erlang strategies`_ in the near future by allowing nurseries to accept
a ``Supervisor`` type.
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _erlang strategies: http://learnyousomeerlang.com/supervisors
IPC using *portals*
*******************
``tractor`` introduces the concept of a *portal* which is an API
borrowed_ from ``trio``. A portal may seem similar to the idea of
a RPC future_ except a *portal* allows invoking remote *async* functions and
generators and intermittently blocking to receive responses. This allows
for fully async-native IPC between actors.
When you invoke another actor's routines using a *portal* it looks as though
it was called locally in the current actor. So when you see a call to
``await portal.run()`` what you get back is what you'd expect
to if you'd called the function directly in-process. This approach avoids
the need to add any special RPC *proxy* objects to the library by instead just
relying on the built-in (async) function calling semantics and protocols of Python.
Depending on the function type ``Portal.run()`` tries to
correctly interface exactly like a local version of the remote
built-in Python *function type*. Currently async functions, generators,
and regular functions are supported. Inspiration for this API comes
`remote function execution`_ but without the client code being
concerned about the underlying channels_ system or shipping code
over the network.
This *portal* approach turns out to be paricularly exciting with the
introduction of `asynchronous generators`_ in Python 3.6! It means that
actors can compose nicely in a data streaming pipeline.
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
Streaming
*********
By now you've figured out that ``tractor`` lets you spawn process based
*actors* that can invoke cross-process (async) functions and all with
structured concurrency built in. But the **real cool stuff** is the
native support for cross-process *streaming*.
Asynchronous generators
+++++++++++++++++++++++
The default streaming function is simply an async generator definition.
Every value *yielded* from the generator is delivered to the calling
portal exactly like if you had invoked the function in-process meaning
you can ``async for`` to receive each value on the calling side.
As an example here's a parent actor that streams for 1 second from a
spawned subactor:
.. literalinclude:: ../examples/asynchronous_generators.py
By default async generator functions are treated as inter-actor
*streams* when invoked via a portal (how else could you really interface
with them anyway) so no special syntax to denote the streaming *service*
is necessary.
Channels and Contexts
+++++++++++++++++++++
If you aren't fond of having to write an async generator to stream data
between actors (or need something more flexible) you can instead use
a ``Context``. A context wraps an actor-local spawned task and
a ``Channel`` so that tasks executing across multiple processes can
stream data to one another using a low level, request oriented API.
A ``Channel`` wraps an underlying *transport* and *interchange* format
to enable *inter-actor-communication*. In its present state ``tractor``
uses TCP and msgpack_.
As an example if you wanted to create a streaming server without writing
an async generator that *yields* values you instead define a decorated
async function:
.. code:: python
@tractor.stream
async def streamer(ctx: tractor.Context, rate: int = 2) -> None:
"""A simple web response streaming server.
"""
while True:
val = await web_request('http://data.feed.com')
# this is the same as ``yield`` in the async gen case
await ctx.send_yield(val)
await trio.sleep(1 / rate)
You must decorate the function with ``@tractor.stream`` and declare
a ``ctx`` argument as the first in your function signature and then
``tractor`` will treat the async function like an async generator - as
a stream from the calling/client side.
This turns out to be handy particularly if you have multiple tasks
pushing responses concurrently:
.. code:: python
async def streamer(
ctx: tractor.Context,
rate: int = 2
) -> None:
"""A simple web response streaming server.
"""
while True:
val = await web_request(url)
# this is the same as ``yield`` in the async gen case
await ctx.send_yield(val)
await trio.sleep(1 / rate)
@tractor.stream
async def stream_multiple_sources(
ctx: tractor.Context,
sources: List[str]
) -> None:
async with trio.open_nursery() as n:
for url in sources:
n.start_soon(streamer, ctx, url)
The context notion comes from the context_ in nanomsg_.
.. _context: https://nanomsg.github.io/nng/man/tip/nng_ctx.5
.. _msgpack: https://en.wikipedia.org/wiki/MessagePack
A full fledged streaming service
++++++++++++++++++++++++++++++++
Alright, let's get fancy.
Say you wanted to spawn two actors which each pull data feeds from
two different sources (and wanted this work spread across 2 cpus).
You also want to aggregate these feeds, do some processing on them and then
deliver the final result stream to a client (or in this case parent) actor
and print the results to your screen:
.. literalinclude:: ../examples/full_fledged_streaming_service.py
Here there's four actors running in separate processes (using all the
cores on you machine). Two are streaming by *yielding* values from the
``stream_data()`` async generator, one is aggregating values from
those two in ``aggregate()`` (also an async generator) and shipping the
single stream of unique values up the parent actor (the ``'MainProcess'``
as ``multiprocessing`` calls it) which is running ``main()``.
.. _future: https://en.wikipedia.org/wiki/Futures_and_promises
.. _borrowed:
https://trio.readthedocs.io/en/latest/reference-core.html#getting-back-into-the-trio-thread-from-another-thread
.. _asynchronous generators: https://www.python.org/dev/peps/pep-0525/
.. _remote function execution: https://codespeak.net/execnet/example/test_info.html#remote-exec-a-function-avoiding-inlined-source-part-i
Actor local (aka *process global*) variables
********************************************
Although ``tractor`` uses a *shared-nothing* architecture between
processes you can of course share state between tasks running *within*
an actor (since a `trio.run()` runtime is single threaded). ``trio``
tasks spawned via multiple RPC calls to an actor can modify
*process-global-state* defined using Python module attributes:
.. code:: python
# a per process cache
_actor_cache: dict[str, bool] = {}
def ping_endpoints(endpoints: List[str]):
"""Start a polling process which runs completely separate
from our root actor/process.
"""
# This runs in a new process so no changes # will propagate
# back to the parent actor
while True:
for ep in endpoints:
status = await check_endpoint_is_up(ep)
_actor_cache[ep] = status
await trio.sleep(0.5)
async def get_alive_endpoints():
nonlocal _actor_cache
return {key for key, value in _actor_cache.items() if value}
async def main():
async with tractor.open_nursery() as n:
portal = await n.run_in_actor(ping_endpoints)
# print the alive endpoints after 3 seconds
await trio.sleep(3)
# this is submitted to be run in our "ping_endpoints" actor
print(await portal.run(get_alive_endpoints))
You can pass any kind of (`msgpack`) serializable data between actors using
function call semantics but building out a state sharing system per-actor
is totally up to you.
Service Discovery
*****************
Though it will be built out much more in the near future, ``tractor``
currently keeps track of actors by ``(name: str, id: str)`` using a
special actor called the *arbiter*. Currently the *arbiter* must exist
on a host (or it will be created if one can't be found) and keeps a
simple ``dict`` of actor names to sockets for discovery by other actors.
Obviously this can be made more sophisticated (help me with it!) but for
now it does the trick.
To find the arbiter from the current actor use the ``get_arbiter()`` function and to
find an actor's socket address by name use the ``find_actor()`` function:
.. literalinclude:: ../examples/service_discovery.py
The ``name`` value you should pass to ``find_actor()`` is the one you passed as the
*first* argument to either ``trio.run()`` or ``ActorNursery.start_actor()``.
Running actors standalone
*************************
You don't have to spawn any actors using ``open_nursery()`` if you just
want to run a single actor that connects to an existing cluster.
All the comms and arbiter registration stuff still works. This can
somtimes turn out being handy when debugging mult-process apps when you
need to hop into a debugger. You just need to pass the existing
*arbiter*'s socket address you'd like to connect to:
.. code:: python
import trio
import tractor
async def main():
async with tractor.open_root_actor(
arbiter_addr=('192.168.0.10', 1616)
):
await trio.sleep_forever()
trio.run(main)
Choosing a process spawning backend
***********************************
``tractor`` is architected to support multiple actor (sub-process)
spawning backends. Specific defaults are chosen based on your system
but you can also explicitly select a backend of choice at startup
via a ``start_method`` kwarg to ``tractor.open_nursery()``.
Currently the options available are:
- ``trio``: a ``trio``-native spawner which is an async wrapper around ``subprocess``
- ``spawn``: one of the stdlib's ``multiprocessing`` `start methods`_
- ``forkserver``: a faster ``multiprocessing`` variant that is Unix only
.. _start methods: https://docs.python.org/3.8/library/multiprocessing.html#contexts-and-start-methods
``trio``
++++++++
The ``trio`` backend offers a lightweight async wrapper around ``subprocess`` from the standard library and takes advantage of the ``trio.`` `open_process`_ API.
.. _open_process: https://trio.readthedocs.io/en/stable/reference-io.html#spawning-subprocesses
``multiprocessing``
+++++++++++++++++++
There is support for the stdlib's ``multiprocessing`` `start methods`_.
Note that on Windows *spawn* it the only supported method and on \*nix
systems *forkserver* is the best method for speed but has the caveat
that it will break easily (hangs due to broken pipes) if spawning actors
using nested nurseries.
In general, the ``multiprocessing`` backend **has not proven reliable**
for handling errors from actors more then 2 nurseries *deep* (see `#89`_).
If you for some reason need this consider sticking with alternative
backends.
.. _#89: https://github.com/goodboy/tractor/issues/89
.. _windowsgotchas:
Windows "gotchas"
^^^^^^^^^^^^^^^^^
On Windows (which requires the use of the stdlib's `multiprocessing`
package) there are some gotchas. Namely, the need for calling
`freeze_support()`_ inside the ``__main__`` context. Additionally you
may need place you `tractor` program entry point in a seperate
`__main__.py` module in your package in order to avoid an error like the
following ::
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\tractor19030601\lib\site-packages\tractor\_actor.py", line 234, in _get_rpc_func
return getattr(self._mods[ns], funcname)
KeyError: '__mp_main__'
To avoid this, the following is the **only code** that should be in your
main python module of the program:
.. code:: python
# application/__main__.py
import trio
import tractor
import multiprocessing
from . import tractor_app
if __name__ == '__main__':
multiprocessing.freeze_support()
trio.run(tractor_app.main)
And execute as::
python -m application
As an example we use the following code to test all documented examples
in the test suite on windows:
.. literalinclude:: ../examples/__main__.py
See `#61`_ and `#79`_ for further details.
.. _freeze_support(): https://docs.python.org/3/library/multiprocessing.html#multiprocessing.freeze_support
.. _#61: https://github.com/goodboy/tractor/pull/61#issuecomment-470053512
.. _#79: https://github.com/goodboy/tractor/pull/79
Enabling logging
****************
Considering how complicated distributed software can become it helps to know
what exactly it's doing (even at the lowest levels). Luckily ``tractor`` has
tons of logging throughout the core. ``tractor`` isn't opinionated on
how you use this information and users are expected to consume log messages in
whichever way is appropriate for the system at hand. That being said, when hacking
on ``tractor`` there is a prettified console formatter which you can enable to
see what the heck is going on. Just put the following somewhere in your code:
.. code:: python
from tractor.log import get_console_log
log = get_console_log('trace')
What the future holds
---------------------
Stuff I'd like to see ``tractor`` do real soon:
- TLS_, duh.
- erlang-like supervisors_
- native support for `nanomsg`_ as a channel transport
- native `gossip protocol`_ support for service discovery and arbiter election
- a distributed log ledger for tracking cluster behaviour
- a slick multi-process aware debugger much like in celery_
but with better `pdb++`_ support
- an extensive `chaos engineering`_ test suite
- support for reactive programming primitives and native support for asyncitertools_ like libs
- introduction of a `capability-based security`_ model
.. _TLS: https://trio.readthedocs.io/en/latest/reference-io.html#ssl-tls-support
.. _supervisors: https://github.com/goodboy/tractor/issues/22
.. _nanomsg: https://nanomsg.github.io/nng/index.html
.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol
.. _celery: http://docs.celeryproject.org/en/latest/userguide/debugging.html
.. _asyncitertools: https://github.com/vodik/asyncitertools
.. _pdb++: https://github.com/antocuni/pdb
.. _capability-based security: https://en.wikipedia.org/wiki/Capability-based_security

1
docs/project/changelog.rst 100644
View File

@ -0,0 +1 @@
.. include:: ../../NEWS.rst

133
docs/project/dev-tips.rst 100644
View File

@ -0,0 +1,133 @@
Hot tips for ``tractor`` hackers
================================
This is a (perpetually WIP) guide for newcomers to the project,
mostly to do with dev, testing, CI and release gotchas, reminders
and best practises.
``tractor`` is a fairly novel project compared to most since it is
effectively a new way of doing distributed computing in Python and
is much closer to working with an "application level runtime"
(like erlang OTP or scala's akka project) than it is a traditional
Python library. As such, having an arsenal of tools and recipes
for figuring out the right way to debug problems when they do
arise is somewhat of a necessity.
Making a release
----------------
Nothing fancy: the traditional PyPA flow on the hatchling_ build
backend, with uv_ doing the driving and towncrier_ generating the
changelog.
1. collect news fragments: user-facing changes should land with a
small ``.rst`` snippet under ``nooz/`` (see ``nooz/HOWTO.rst``;
fragment types are ``feature``, ``bugfix``, ``doc`` and
``trivial``),
2. render them into ``NEWS.rst``::
uvx towncrier build --version <version>
3. build and upload (testpypi first if you're being careful)::
uv build
uvx twine upload -r testpypi dist/*
uvx twine upload dist/*
How you organize built artifacts under ``dist/`` locally (per
release sub-dirs and such) is entirely up to you.
Keep in mind that PyPi releases tend to lag the ``main`` branch
since we develop in the open — ``main`` is usually the thing to
run when you want the latest.
Debugging and monitoring actor trees
------------------------------------
Your "what is my tree doing right now?" toolbox, in escalation
order:
**Live process-tree view** — keep a ``watch``-ed ``pstree``
running in a side terminal; actor procs are recognizable by their
``_subactor[<name>@<uuid-prefix>]`` process titles. The exact
one-liners (plus the ``pgrep`` marker recipes) live in
:doc:`/guide/testing`.
**SIGUSR1 task-tree dumps** — boot any tree with
``enable_stack_on_sig=True`` (or export
``TRACTOR_ENABLE_STACKSCOPE=1``) and every actor installs a
stackscope_ signal handler. Then from any shell::
# dump every actor's live trio task tree:
pkill --signal SIGUSR1 -f <part-of-your-cmd>
# or for a single process:
kill -SIGUSR1 $(pgrep -f <part-of-your-cmd>)
Each dump is also tee'd (append-mode) to
``/tmp/tractor-stackscope-<pid>.log`` so you still get output
under pytest capture or in CI. This works *without* debug-mode
being enabled — it's the lightest-weight hang-investigation tool
in the box.
**The built-in multi-process debugger** — ``debug_mode=True``
plus :func:`tractor.pause` and friends: the heavyweight champ for
interactive, REPL-driven inspection of a whole tree (including
crash handling). Remember pytest capture interplay — see
:doc:`/guide/testing`.
**Post-mortem zombie sweeps** — ``scripts/tractor-reap`` for the
(should-be-rare!) cases where hacking on the runtime itself wedges
a tree: a SIGINT-first, structured concurrency (SC) polite
escalation, plus ``--shm`` and ``--uds`` leaked-resource sweeps.
Using the log system to trace ``trio`` task flow
------------------------------------------------
The logging system is oriented around the **stack "layers" of the
runtime**, letting you trace logical abstraction layers in the
code — errors, cancellation, IPC and streaming, the low level
transport and wire protocols — independently of one another.
Concretely, ``tractor.log.get_logger()`` returns a
``StackLevelAdapter`` sporting extra level-methods beyond the
stdlib set, including:
- ``.cancel()`` — cancellation-machinery flow,
- ``.runtime()`` — actor-runtime lifecycle chatter,
- ``.devx()`` — debugger/devx tooling internals,
- ``.transport()`` — wire-level msging events.
To get console output at any level from your own code::
from tractor.log import get_console_log
get_console_log('cancel')
or, runtime-wide without touching code, just export
``TRACTOR_LOGLEVEL=cancel`` (the env-var wins over caller-passed
levels; great for test runs).
When you want only *one subsystem* cranked, the suite's ``--tl``
flag (and ``tractor.log.apply_logspec()``) accept a per-sublogger
spec::
uv run pytest tests/... --tl 'devx:runtime,trionics:cancel'
Every record's header includes the emitting actor and task names,
so cross-process flows can be stitched back together by eyeball
(or grep).
.. seealso::
- :doc:`/guide/testing` — running the suite, watching trees
live, examples-as-tests conventions and the zombie-reaper.
- :doc:`/guide/discovery` — the registrar mechanics you'll
bump into when running multiple trees on one host.
.. _hatchling: https://hatch.pypa.io/latest/
.. _uv: https://docs.astral.sh/uv/
.. _towncrier: https://towncrier.readthedocs.io/en/stable/
.. _stackscope: https://github.com/oremanj/stackscope

60
docs/project/index.rst 100644
View File

@ -0,0 +1,60 @@
Project
=======
Everything meta: where the project's been (the
:doc:`changelog`), where it's going (the roadmap below),
how to hack on it (:doc:`dev-tips` and
:doc:`/guide/testing`) and where to find the humans.
.. toctree::
:maxdepth: 1
changelog
dev-tips
What the future holds
---------------------
Help us push toward the future of distributed Python!
Planned (or dreamed of) work non-comprehensively
includes,
- Erlang-style supervisors via composed context
managers (see `#22`_),
- typed capability-based (dialog) protocols, ie.
evolving our `msg-spec`_ system into per-dialog
contracts (see `#196`_ with draft work in `#311`_),
- a higher level "service manager" API for daemon
lifetime mgmt over actor trees (in the works on an
experimental branch as ``tractor.hilevel``),
- richer `discovery`_ via gossip and/or
`rendezvous protocol`_ approaches (today's registrar
is intentionally naive),
- more IPC transports: the current ``tcp`` | ``uds``
pair wants friends (QUIC, shm-ring-buffers, RUDP,
wireguard tunnels),
- an extensive `chaos engineering`_ test suite,
- a respawn-from-REPL system for crashed (sub-)actors.
Feel like saying hi?
--------------------
This project is very much coupled to the ongoing
development of ``trio`` (i.e. ``tractor`` gets most of
its ideas from that brilliant community). If you want
to help, have suggestions or just want to say hi,
please feel free to reach us in our `matrix channel`_.
If matrix seems too hip, we're also mostly all in the
`trio gitter channel`_!
Contributions of all kinds welcome: docs, examples,
bug reports, new transports, supervisor strategies,
philosophical debates about what an "actor model"
really is B)
.. _#22: https://github.com/goodboy/tractor/issues/22
.. _#196: https://github.com/goodboy/tractor/issues/196
.. _#311: https://github.com/goodboy/tractor/pull/311
.. _msg-spec: https://jcristharif.com/msgspec/
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
.. _rendezvous protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
.. _chaos engineering: https://principlesofchaos.org/
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
.. _trio gitter channel: https://gitter.im/python-trio/general

36
docs/start/index.rst 100644
View File

@ -0,0 +1,36 @@
Getting started
===============
Welcome aboard B)
Real talk before any code: the first step to grok ``tractor`` is
to get an intermediate knowledge of ``trio`` and **structured
concurrency** (SC). ``tractor`` **is just** ``trio`` - but with
nurseries for process management and cancel-able streaming IPC -
so every rule you already know about task lifetimes, cancellation
and error propagation keeps holding, just now across process (and
host!) boundaries. Some great places to start are,
- the seminal `blog post`_,
- obviously the `trio docs`_,
- wikipedia's nascent SC_ page,
- the fancy diagrams @ libdill-docs_.
Once you've taken in (some of) the canon, get installed and go
spawn your first actor tree:
.. toctree::
:maxdepth: 1
install
quickstart
.. seealso::
Already installed and itching? Jump straight to
:doc:`/start/quickstart`; once you're through the on-ramp the
:doc:`guide pages </guide/index>` take each subsystem deeper.
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
.. _trio docs: https://trio.readthedocs.io/en/latest/
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html

75
docs/start/install.rst 100644
View File

@ -0,0 +1,75 @@
Install
=======
``tractor`` is still in an *alpha-near-beta-stage* for many of
its subsystems, however we are very close to having a stable
lowlevel runtime and API. Expect the occasional rough edge (and
feel free to report it!).
Supported platforms
-------------------
- **python**: ``>=3.13,<3.15`` - yes, we ride near the front of
the release train; 3.14 additionally unlocks the experimental
PEP-734 sub-interpreter spawn backend,
- **linux**: the primary development and CI platform,
- **macos**: officially supported,
- **windows**: currently *untested* - we disabled CI-testing on
windows a while back and haven't had the cycles to revive it.
It mostly worked historically; if you're a windows person we'd
love a hand getting it back in the test matrix.
With ``uv`` (preferred)
-----------------------
We use the very hip uv_ for project mgmt and recommend you do
too. Add ``tractor`` to your project::
uv add tractor
or, since the git ``main`` branch is often much further ahead
than any latest release (see the PyPi note below), track ``main``
directly::
uv add "tractor @ git+https://github.com/goodboy/tractor.git"
From PyPi
---------
We ofc also offer "releases" on PyPi_::
pip install tractor
Just note that **YMMV** since ``main`` is usually well ahead of
the latest published alpha; when in doubt go with a git install
as per above (or hack from source as per below).
From source
-----------
To run the bundled examples in-tree, or to start hacking on the
code base, clone and sync a dev env::
git clone https://github.com/goodboy/tractor.git
cd tractor
uv sync --dev
uv run python examples/rpc_bidir_streaming.py
Consider activating a virtual/project-env before starting to hack
on the code base::
# you could use plain ol' venvs
# https://docs.astral.sh/uv/pip/environments/
uv venv tractor_py313 --python 3.13
# but @goodboy prefers the more explicit (and shell agnostic)
# https://docs.astral.sh/uv/configuration/environment/#uv_project_environment
UV_PROJECT_ENVIRONMENT="tractor_py313"
# hint hint, enter @goodboy's fave shell B)
uv run --dev xonsh
.. seealso::
All set? Go boot a process tree in :doc:`/start/quickstart`.
And if the install fights you, swing by the `matrix channel`_
and we'll sort it out.
.. _uv: https://docs.astral.sh/uv/
.. _PyPi: https://pypi.org/project/tractor/
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org

264
docs/start/quickstart.rst 100644
View File

@ -0,0 +1,264 @@
Quickstart
==========
Time to spawn something B)
If you take one thing from this page make it this: ``tractor``
**is just** ``trio`` - but with nurseries for process management
and cancel-able streaming IPC. Every "*actor*" you'll meet below
is a plain Python **process** running its own ``trio.run()``
scheduled task tree, linked back to its parent through an IPC
protocol which keeps the whole tree `structured concurrency`_
(SC) compliant end-to-end. If you know your nursery_ semantics
you already know most of ``tractor``; we just stretch them across
the process boundary.
.. d2:: diagrams/actor_tree.d2
:alt: a supervision tree of actor processes
:margin:
:caption: every arrow is a parent which **must wait** on its kids
Your first actor tree
---------------------
``trio`` takes the hard-line position that a parent task **must
wait** on the children it spawns; causality_ is paramount! So
does ``tractor``, one abstraction layer up:
``tractor.open_nursery()`` yields an ``ActorNursery`` which
**must** wait on its spawned *subactors* to complete (or error)
before the ``async with`` block exits, in the same causal_ way a
``trio`` nursery waits on its subtasks. That includes any one
child's crash cancelling all of its siblings: *one-cancels-all*
supervision, `exactly like trio`_.
Enough preamble, spawn a process:
.. literalinclude:: ../../examples/actor_spawning_and_causality.py
:caption: examples/actor_spawning_and_causality.py
:language: python
Run it::
$ python examples/actor_spawning_and_causality.py
Dang that's beautiful
What's going on here?
- ``trio.run(main)`` starts the **root actor**; the ``tractor``
runtime boots *implicitly* inside ``tractor.open_nursery()``
whenever it isn't already up. No special entrypoint, no
framework takeover - it's just a ``trio`` app,
- inside ``main()`` a *subactor* is spawned via
``ActorNursery.run_in_actor()`` and told to run exactly one
function: ``cellar_door()``,
- you get back a ``Portal``: your handle for invoking tasks in
the new process's (separate!) memory domain. We lean on it
much harder in the next section,
- the subactor, *some_linguist*, boots a fresh ``trio.run()`` in
a **new process** and executes ``cellar_door()`` as its *main
task* (note the child proving it is *not* the root with
``tractor.is_root_process()``), then ships the return value
back over IPC,
- the parent grabs that *final result* with
``await portal.wait_for_result()``, much like you'd expect
from a "future" - except causality is preserved: the nursery
block only exits once the child is *done*, dead, and reaped.
.. margin:: Just need a worker pool?
If all you want is to throw *sync* functions at your cores,
also check out trio-parallel_. ``tractor`` is aimed at
structured, (possibly) distributed *trees* of cooperating
``trio`` programs; a worker pool is a trivial special case.
.. note::
``run_in_actor()`` is the *convenience* wrapper: one-shot
spawn-run-reap semantics for when a subactor's entire job is
a single function call. The core primitives are
``ActorNursery.start_actor()`` (next up) paired with
``Portal.open_context()`` for full, SC-linked cross-actor
dialogs - see :doc:`/guide/context`.
Daemon actors and RPC
---------------------
A ``run_in_actor()``-spawned actor terminates when its main task
returns. But often you want long-lived *daemon* actors instead:
spawned once, then serving (allowlisted) RPC requests until told
otherwise. That's ``start_actor()``:
.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py
:caption: examples/actor_spawning_and_causality_with_daemon.py
:language: python
Two lifetime rules to internalize:
- a ``run_in_actor()`` actor lives exactly as long as its main
task; the nursery waits for that function (and thus the
process) to complete before unblocking,
- a ``start_actor()`` actor *lives forever* - an RPC daemon the
nursery will happily wait on **indefinitely** - until some
task explicitly cancels it via ``Portal.cancel_actor()`` (as
above), or its parent nursery is cancelled wholesale.
.. tip::
Want your *entire program* to just be a long-lived RPC
daemon? ``tractor.run_daemon()`` is the blocking shorthand:
it ``trio.run()``\s a root actor which serves requests until
cancelled.
The ``enable_modules=[__name__]`` kwarg is the other thing to
notice: it lists the module paths the subactor will load and
*expose* for remote invocation.
``await portal.run(movie_theatre_question)`` works because this
very module is in that allowlist (and note we call it twice; the
daemon happily serves repeat requests). Ask for a function from
any module *not* enabled and you're denied with a
``ModuleNotExposed`` error: a simple, capability-style
restriction mechanism built on Python's own module system.
We are *processes*
------------------
Why processes (and not, say, threads)? Python has a GIL and an
`actor model`_ by definition shares **nothing** between its
concurrent units, so real OS processes are the natural fit: you
get all your cores locally, and since actors only ever talk via
IPC, the exact same code distributes over multiple hosts without
modification.
Of course, the moment you hear "process trees" you should be
asking: *what about zombies?* Watch ``tractor`` eat one for
breakfast - run this while monitoring your process tree::
$TERM -e watch -n 0.1 "pstree -a $$" \
& python examples/parallelism/we_are_processes.py \
&& kill $!
.. literalinclude:: ../../examples/parallelism/we_are_processes.py
:caption: examples/parallelism/we_are_processes.py
:language: python
.. margin:: Who's who in ``pstree``?
Every subactor (best-effort, via the optional
``setproctitle`` dep) re-titles its OS process like
``_subactor[worker_0@<uuid>]``, so ``pstree``/``htop``/
``pgrep -f`` can tell your actors apart at a glance.
You'll see something like::
$ python examples/parallelism/we_are_processes.py
Yo, i'm 'worker_2' running in pid 1777246
Yo, i'm 'worker_0' running in pid 1777244
Yo, i'm 'worker_3' running in pid 1777247
Yo, i'm 'worker_1' running in pid 1777245
This process tree will self-destruct in 1 sec...
Zombies Contained
(The worker lines land in whatever order the OS schedules them;
they're separate *processes*, racing, and that's the point.)
An actor is spawned per core, each parks itself in
``trio.sleep_forever()``... and then the root *crashes on
purpose*. The ``ActorNursery`` responds with hard ``trio``
discipline: every child is cancelled, every process is reaped,
the error propagates to ``trio.run()``, and your terminal prints
``Zombies Contained``. No orphans, no ``kill -9`` archaeology in
``htop`` afterwards.
.. note::
**The zombie-safety guarantee**: ``tractor`` tries to protect
you from zombies, *no matter what*. If you can create zombie
child processes (without using a system signal) it **is a
bug** - please report it so we can hunt it down.
A trynamic first scene
----------------------
So far the root actor has done all the talking, but subactors
can just as well discover and call *each other*. Let's direct a
couple actors and have them run their lines for the hip new film
we're shooting:
.. literalinclude:: ../../examples/a_trynamic_first_scene.py
:caption: examples/a_trynamic_first_scene.py
:language: python
The script of the scene (runtime ``INFO`` log lines trimmed)::
$ python examples/a_trynamic_first_scene.py
Alright... Action!
Hi my name is gretchen
Hi my name is donny
CUTTTT CUUTT CUT!!! Donny!! You're supposed to say...
The new tricks in play:
- two subactors, *donny* and *gretchen*, are each told to run
``say_hello()`` targeting the *other* by name,
- ``tractor.wait_for_actor()`` blocks until the named peer has
registered with the tree's *registrar* (every actor announces
itself at boot), then yields a ``Portal`` connected
**directly** to that peer,
- each actor invokes its partner's ``hi()`` over that portal:
actor-to-actor RPC with the root merely *directing* - and both
final lines flow back to ``main()`` via
``await portal.wait_for_result()``,
- ``tractor.log.get_console_log("INFO")`` cranks up runtime
logging so you can watch the spawn/register/cancel machinery
narrate itself; remove it for a quiet set.
Cross-actor calls look just like (async) function calls; there
are no proxy objects and no shared references, only messages B)
Crash handling, native feeling
------------------------------
One last teaser before the guide proper. Flip exactly one
switch:
.. code:: python
async with tractor.open_nursery(
debug_mode=True,
) as an:
...
and any crash, in *any* actor at *any* depth of the tree, drops
your terminal into a multi-process-safe pdbp_ REPL at the
offending frame, with the rest of the tree held back from
clobbering the tty. ``await tractor.pause()`` likewise gives you
a breakpoint that *just works* inside subprocesses. We think it
might be the first native multi-process debugging UX for Python;
get the full tour in :doc:`/guide/debugging`.
Where to next?
--------------
You can now boot a runtime, spawn one-shot and daemon actors,
make cross-process RPC calls, and contain zombies: that's the
on-ramp done. The guide takes each subsystem deeper,
- :doc:`/explain/sc-distributed` - the structured concurrency
worldview and how ``tractor`` extends it across processes,
- :doc:`/guide/spawning` - everything ``ActorNursery``: spawn
kwargs, lifetimes and supervision semantics,
- :doc:`/guide/rpc` - the ``Portal`` in depth: calling into
another actor's memory domain,
- :doc:`/guide/context` - the core API: ``@tractor.context``
endpoints, the ``ctx.started()`` handshake, and SC-linked
cross-actor task pairs,
- :doc:`/guide/streaming` - bidirectional ``MsgStream`` dialogs
and fan-out broadcasting,
- :doc:`/guide/debugging` - the multi-process REPL, crash
handling mode, and ``tractor.pause()``,
- :doc:`/guide/asyncio` - "infected ``asyncio``" mode: SC
supervision wrapped around ``asyncio`` tasks,
- :doc:`/guide/discovery` - registries, service daemons, and
finding actors from anywhere in (or out of) the tree.
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel
.. _pdbp: https://github.com/mdmintz/pdbp