From 3366b1a70d31cbad5b10a5d6dab4c205bf0de104 Mon Sep 17 00:00:00 2001 From: goodboy Date: Thu, 11 Jun 2026 15:17:14 -0400 Subject: [PATCH] Write the big boi docs content tree Replace the ancient `docs/index.rst` (still teaching `tractor.run()`, `@stream` + arbiters..) with a full ~32 page tree teaching ONLY the current api (`.wait_for_result()`, registrar naming, `@context` + `open_context()` as the core model), - landing: hero example, feature cards + canon links, - `start/`: install + a 4-example quickstart on-ramp, - `explain/`: an "SC across processes" essay distilling the essence per #157's orig ask + a runtime architecture tour, - `guide/`: 12 task-focused pages incl the flagship multi-process debugging walkthrough, `Context` + `MsgStream` deep-dives, cancellation semantics (self-vs-cross cancel rules), discovery, infected `asyncio`, typed msging + the #126 testing-tips page, - `api/`: 10 curated autodoc pages (all targets import-verified vs the reorg'd subpkg tree), - `project/`: changelog include, ported dev-tips (drops old `docs/dev_tips.rst`) + roadmap. Every code block is a `literalinclude` from `examples/` - zero duplication, all CI-run - w/ `d2` figs floated into the RHS margin per the 3-col design. Build is green; the 24 remaining warnings all source from lib docstring rst-isms or legacy `NEWS.rst` content. Substantially resolves #157 (refine round pending); chips at #175 + #126. Prompt-IO: ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code --- .../20260611T175152Z_8526985c_prompt_io.md | 93 +++ ...20260611T175152Z_8526985c_prompt_io.raw.md | 49 ++ docs/api/context.rst | 134 ++++ docs/api/core.rst | 157 ++++ docs/api/devx.rst | 92 +++ docs/api/discovery.rst | 67 ++ docs/api/errors.rst | 102 +++ docs/api/index.rst | 67 ++ docs/api/ipc.rst | 89 +++ docs/api/msg.rst | 107 +++ docs/api/to_asyncio.rst | 97 +++ docs/api/trionics.rst | 76 ++ docs/dev_tips.rst | 51 -- docs/explain/architecture.rst | 368 +++++++++ docs/explain/index.rst | 16 + docs/explain/sc-distributed.rst | 207 +++++ docs/guide/asyncio.rst | 309 +++++++ docs/guide/cancellation.rst | 285 +++++++ docs/guide/clustering.rst | 127 +++ docs/guide/context.rst | 344 ++++++++ docs/guide/debugging.rst | 545 +++++++++++++ docs/guide/discovery.rst | 255 ++++++ docs/guide/index.rst | 52 ++ docs/guide/msging.rst | 260 ++++++ docs/guide/parallelism.rst | 176 ++++ docs/guide/rpc.rst | 177 +++++ docs/guide/spawning.rst | 299 +++++++ docs/guide/streaming.rst | 244 ++++++ docs/guide/testing.rst | 256 ++++++ docs/index.rst | 752 ++++-------------- docs/project/changelog.rst | 1 + docs/project/dev-tips.rst | 133 ++++ docs/project/index.rst | 60 ++ docs/start/index.rst | 36 + docs/start/install.rst | 75 ++ docs/start/quickstart.rst | 264 ++++++ 36 files changed, 5766 insertions(+), 656 deletions(-) create mode 100644 ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md create mode 100644 ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md create mode 100644 docs/api/context.rst create mode 100644 docs/api/core.rst create mode 100644 docs/api/devx.rst create mode 100644 docs/api/discovery.rst create mode 100644 docs/api/errors.rst create mode 100644 docs/api/index.rst create mode 100644 docs/api/ipc.rst create mode 100644 docs/api/msg.rst create mode 100644 docs/api/to_asyncio.rst create mode 100644 docs/api/trionics.rst delete mode 100644 docs/dev_tips.rst create mode 100644 docs/explain/architecture.rst create mode 100644 docs/explain/index.rst create mode 100644 docs/explain/sc-distributed.rst create mode 100644 docs/guide/asyncio.rst create mode 100644 docs/guide/cancellation.rst create mode 100644 docs/guide/clustering.rst create mode 100644 docs/guide/context.rst create mode 100644 docs/guide/debugging.rst create mode 100644 docs/guide/discovery.rst create mode 100644 docs/guide/index.rst create mode 100644 docs/guide/msging.rst create mode 100644 docs/guide/parallelism.rst create mode 100644 docs/guide/rpc.rst create mode 100644 docs/guide/spawning.rst create mode 100644 docs/guide/streaming.rst create mode 100644 docs/guide/testing.rst create mode 100644 docs/project/changelog.rst create mode 100644 docs/project/dev-tips.rst create mode 100644 docs/project/index.rst create mode 100644 docs/start/index.rst create mode 100644 docs/start/install.rst create mode 100644 docs/start/quickstart.rst diff --git a/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md b/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md new file mode 100644 index 00000000..3dfd43c4 --- /dev/null +++ b/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.md @@ -0,0 +1,93 @@ +--- +model: claude-fable-5[1m] +service: claude +session: 0780a862-e19a-4f0a-86cd-c8afc0997757 +timestamp: 2026-06-11T17:51:52Z +git_ref: 8526985c +scope: docs+code+config +substantive: true +raw_file: 20260611T175152Z_8526985c_prompt_io.raw.md +--- + +## Prompt + +> we need big boi docs; ours are way out of date and generally +> terrible. i'd like to use a sphinx theme in the vein and/or +> one-of/nearly-the-same-as used in one or all of the `msgspec`, +> `numpy`, `ray`, `xonsh`, `polars` projects. +> +> see the following oustanding but now very old issues to guide +> you: #175, #126, #123, #157 +> +> we should try to target a 3 columned sphinx theme with, +> - document index-nav on left, +> - body content in middle, +> - diagrams (ideally in d2lang if possible) on RHS +> +> optimize for (if possible), +> - reusing examples code without duplication in docs, +> - generating new examples from todos throughout code base, +> - distilling the essence of SC across processes in the simplest +> and most friendly way possible. +> +> do this work in a /open-wkt and do you best without my guidance +> for the first major pass - we will refine asap once you are +> complete. + +Session settings: `/effort max` + ultracode (multi-agent workflow +orchestration). Fully autonomous first pass; two orchestrated +agent fleets were used (6-agent recon survey, then 10-agent +content fan-out: 9 section writers + 1 examples smith), with the +orchestrator authoring the landing page, SC essay, sphinx +scaffold, d2 pipeline + diagram sources, CI workflow and all +integration/fix passes directly. + +## Response summary + +Complete sphinx docs revamp on branch `wkt/big_boi_docs`: +`pydata-sphinx-theme` 0.18 (3-column: left nav / content / +page-toc, with d2 diagrams + asides floated into the RHS margin +via custom CSS), sphinx 9.1, a local `.. d2::` directive +rendering `docs/diagrams/*.d2` sources with committed-SVG +fallback, a vendored `.. margin::` directive, ~25 new doc pages +(landing, start/, explain/, 12 guides, 10 api-ref pages, +project/), 5 new auto-tested examples + 3 modernized + 1 +renamed, and a gh-pages deploy workflow (issue #123). All +example code is `literalinclude`d from `examples/` (zero +duplication, CI-verified). Build: green, 24 warnings all +pre-existing-docstring/NEWS sourced. + +## Files changed + +See the branch diff (uncommitted at entry-write time): + +> `git diff test_cpu_throttling..wkt/big_boi_docs` +> `git -C status --short` (pre-commit working tree) + +- `docs/conf.py` — full rewrite for pydata theme + ext stack +- `docs/_ext/d2diagrams.py` — new `.. d2::` sphinx directive +- `docs/_ext/marginalia.py` — new `.. margin::` directive +- `docs/_static/css/custom.css` — b&w skin + RHS margin floats +- `docs/diagrams/*.d2` (7) — diagram sources (sketch/grayscale) +- `docs/_diagrams/*.svg` (7) — committed rendered fallbacks +- `docs/index.rst` — new landing (replaces dead-API doc) +- `docs/start/*.rst` (3), `docs/explain/*.rst` (3), + `docs/guide/*.rst` (13), `docs/api/*.rst` (10), + `docs/project/*.rst` (3) — new content tree +- `docs/dev_tips.rst` — removed (ported to project/dev-tips) +- `examples/{typed_payloads,nested_actor_tree, + service_daemon_discovery,uds_transport_actor_tree, + streaming_broadcast_fanout}.py` — new, smoke-tested +- `examples/{a_trynamic_first_scene, + actor_spawning_and_causality,parallelism/single_func}.py` — + `.result()` -> `.wait_for_result()` modernization +- `examples/parallelism/concurrent_futures_primes.py` — renamed + from leading-underscore + trio-runner shim added +- `pyproject.toml` — `docs` dependency-group filled in +- `uv.lock` — relock for docs group +- `.github/workflows/docs.yml` — build + gh-pages deploy + +## Human edits + +None yet — entry written pre-commit; the author reviews, stages +and commits manually (per repo workflow policy). diff --git a/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md b/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md new file mode 100644 index 00000000..90408f21 --- /dev/null +++ b/ai/prompt-io/claude/20260611T175152Z_8526985c_prompt_io.raw.md @@ -0,0 +1,49 @@ +--- +model: claude-fable-5[1m] +service: claude +timestamp: 2026-06-11T17:51:52Z +git_ref: 8526985c +diff_cmd: git diff test_cpu_throttling..wkt/big_boi_docs +--- + +# Raw output pointers (diff-ref mode) + +All generated content is code/config/docs committed alongside +this entry on branch `wkt/big_boi_docs`; per the prompt-io +diff-ref decision rule each file's verbatim content is the diff +itself rather than a copy here: + +> `git diff test_cpu_throttling..wkt/big_boi_docs -- docs/` +> `git diff test_cpu_throttling..wkt/big_boi_docs -- examples/` +> `git diff test_cpu_throttling..wkt/big_boi_docs -- pyproject.toml uv.lock .github/` + +## Generation notes (non-code output summary) + +- Theme research (web, agent-verified 2026-06-11): msgspec=furo, + xonsh=furo, numpy/ray/polars=pydata-sphinx-theme (ray migrated + off sphinx-book-theme); sphinx-book-theme 1.2.0 hard-pins + pydata 0.16.1 (stale) -> chose pydata 0.18 + sphinx 9.1. +- d2 ecosystem: no production-grade pypi extension exists + (sphinxcontrib-d2lang 0.0.5 ignores returncodes, uuid4 output + names; sphinx-d2 is an empty stub) -> wrote local + `docs/_ext/d2diagrams.py` (~230 LOC) with D2_BIN env + discovery, mtime caching, committed-SVG fallback and + literal-block last resort. +- Diagrams authored in d2 (theme-id 1 "Neutral Grey" + sketch + mode + ELK layout, validated by render + headless-firefox + screenshot loop): actor_tree, context_handshake (real + msg-spec names Start/StartAck/Started/Yield/Stop/Return), + streaming_pipeline, runtime_stack, debug_lock, + error_propagation, infected_aio. +- API truth enforced from a 6-agent recon pass over the + reorganized package tree (runtime/, discovery/, spawn/, ipc/, + msg/, devx/, trionics/): docs teach `.wait_for_result()`, + registrar (not arbiter) naming, `@tractor.context` + + `open_context()` as the core model, `run_in_actor()` as + convenience only. +- All ~30 literalincluded example scripts verified present; 9 + touched/new example files smoke-run green (exit 0, <16s). +- Final build: `sphinx-build -b html` succeeded; 24 residual + warnings, every one sourced from pre-existing library + docstring rst-isms or legacy NEWS.rst content (left untouched + by design; flagged for a follow-up docstring lint pass). diff --git a/docs/api/context.rst b/docs/api/context.rst new file mode 100644 index 00000000..ff024bd9 --- /dev/null +++ b/docs/api/context.rst @@ -0,0 +1,134 @@ +Contexts and streaming +====================== + +The modern core of ``tractor``: a :class:`Context` links a task in +one actor to a task in another as a *single* structured concurrency +(SC) scope stretched across the IPC boundary — errors, results and +cancellation flow between the pair `exactly like trio`_ tasks under +a common `nursery`_. Open one with ``Portal.open_context()`` (see +:func:`tractor._context.open_context_from_portal` in +:doc:`/api/core`), then optionally bridge a bidirectional +:class:`MsgStream` between the two tasks. + +.. d2:: diagrams/context_handshake.d2 + :caption: The ``open_context()`` <-> ``ctx.started()`` handshake. + :margin: + :alt: parent and child actor context handshake sequence + +For the guided, example-driven tour see :doc:`/guide/context`; this +page is the precise API surface. + +.. currentmodule:: tractor + +The ``@context`` decorator +-------------------------- + +.. autofunction:: context + +.. note:: + + The decorated function **must** declare a parameter annotated + ``tractor.Context`` (any param name works); the runtime injects + the context instance there on each remote invocation. Pass + ``pld_spec`` to type-restrict (and validate) the payloads this + endpoint may shuttle — violations raise + :class:`MsgTypeError`. See ``examples/typed_payloads.py``. + +``Context`` +----------- + +.. autoclass:: Context + :members: started, + wait_for_result, + cancel, + cid, + chan, + side, + cancel_called, + cancelled_caught, + cancel_acked, + canceller, + maybe_error, + outcome + +.. deprecated:: 0.1.0a6 + + ``Context.result()`` warns; use :meth:`Context.wait_for_result`. + +.. note:: + + A :class:`Context` is **not** a :class:`trio.CancelScope`: + :meth:`Context.cancel` requests cancellation of the *remote* + peer task and does not cancel the local scope. If *you* + requested the cancel, the resulting :class:`ContextCancelled` + is absorbed at ``open_context()`` exit; a cancel originating + anywhere else (the peer, or a third-party actor recorded in + :attr:`ContextCancelled.canceller`) *is* raised locally. This + self-vs-cross-cancel rule is the key to writing correct + inter-actor teardown logic — see :doc:`/guide/context`. + +Bidirectional streaming +----------------------- + +.. autofunction:: tractor._streaming.open_stream_from_ctx + +.. note:: + + :func:`~tractor._streaming.open_stream_from_ctx` is bound as + the **method-alias** ``Context.open_stream()`` — call it as + ``async with ctx.open_stream() as stream:``. Both sides of the + context must enter it for the dialog to be open. + +.. autoclass:: MsgStream + :members: send, + receive, + receive_nowait, + aclose, + subscribe, + ctx, + closed + +.. note:: + + A :class:`MsgStream` is one-shot use: once closed it can never + be "re-opened" — open a fresh :class:`Context` instead. Remote + end-of-stream surfaces as :class:`StopAsyncIteration` from + ``async for``; un-consumed sends overrun the receiver and raise + :class:`tractor._exceptions.StreamOverrun` unless the context + was opened with ``allow_overruns=True``. + +:meth:`MsgStream.subscribe` fans a single IPC stream out to +multiple *local* tasks via a +:class:`tractor.trionics.BroadcastReceiver` (see +:doc:`/api/trionics`); the underlying allocation is idempotent and +non-reversible for the stream's lifetime. See +``examples/streaming_broadcast_fanout.py`` for the pattern in +action. + +Legacy one-way streaming +------------------------ + +.. autofunction:: stream + +.. warning:: + + ``@tractor.stream`` and ``Portal.open_stream_from()`` are the + *legacy* one-way streaming API kept for backward compat: a + plain async-generator function streamed parent-ward with no + child-side receive leg. New code should use + ``@tractor.context`` + ``ctx.open_stream()`` (bidirectional, + SC-linked, typed). Note ``ctx`` is now a reserved param name + for ``@context`` endpoints — ``@stream`` functions must use + ``stream`` instead, and ``ctx.send_yield()`` is deprecated in + favor of :meth:`MsgStream.send`. + +.. seealso:: + + :doc:`/api/errors` for :class:`ContextCancelled` / + :class:`MsgTypeError` semantics, :doc:`/api/msg` for payload + typing via ``pld_spec`` and codecs, :doc:`/api/trionics` for + the broadcast fan-out machinery, and the guided tours in + :doc:`/guide/streaming` + :doc:`/guide/cancellation`. + +.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics +.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning diff --git a/docs/api/core.rst b/docs/api/core.rst new file mode 100644 index 00000000..196da2d0 --- /dev/null +++ b/docs/api/core.rst @@ -0,0 +1,157 @@ +Runtime and spawning +==================== + +The core lifecycle API: boot the runtime in your root process, +spawn trio-"actors" (processes running ``trio.run()`` task trees) +under a one-cancels-all supervisor, and talk to them through +portals. This is structured concurrency (SC) applied *transitively*: +every spawned process is owned by a nursery block and errors +`always propagate`_. If you can create zombies it **is a bug**. + +.. currentmodule:: tractor + +Booting the runtime +------------------- + +.. autofunction:: open_root_actor + +.. note:: + + The env vars ``TRACTOR_LOGLEVEL`` and ``TRACTOR_SPAWN_METHOD`` + override the ``loglevel`` / ``start_method`` params so you can + crank verbosity or swap spawn backends without touching app + code. Exactly **one** IPC transport may be enabled per actor + (see ``enable_transports`` and :doc:`/api/ipc`). + +.. autofunction:: run_daemon + +Spawning actors +--------------- + +.. d2:: diagrams/actor_tree.d2 + :caption: A supervised actor (process) tree. + :margin: + :alt: root actor supervising a tree of subactors + +.. autofunction:: open_nursery + +.. autoclass:: ActorNursery + :members: start_actor, + run_in_actor, + cancel, + cancel_called, + cancelled_caught + +.. note:: + + :meth:`ActorNursery.start_actor` (daemon actor + portal) is the + blessed spawning primitive; pair it with + ``Portal.open_context()`` for SC-linked remote tasks. + :meth:`ActorNursery.run_in_actor` is a *convenience* one-shot — + spawn, run a single task, auto-cancel after the result — slated + to be rebuilt as a high-level wrapper, so don't design around + it as the core model. + +.. deprecated:: 0.1.0a6 + + ``ActorNursery.cancelled`` warns; use + :attr:`ActorNursery.cancel_called` and + :attr:`ActorNursery.cancelled_caught`. The ``rpc_module_paths`` + kwarg is likewise deprecated in favor of ``enable_modules``. + +Portals +------- + +A :class:`Portal` "opens a portal" into a peer actor's memory +domain: you call functions and start SC-linked tasks *over IPC* as +though they were local, with results, errors and cancellation +flowing back `exactly like trio`_. + +.. autoclass:: Portal + :members: run, + run_from_ns, + open_stream_from, + wait_for_result, + cancel_actor, + chan + +.. deprecated:: 0.1.0a6 + + ``Portal.result()`` warns; use :meth:`Portal.wait_for_result`. + The str-form ``Portal.run('mod.path', 'fn_name')`` also warns; + pass a function *object* whose module is listed in the target's + ``enable_modules``. ``Portal.channel`` is the legacy spelling + of :attr:`Portal.chan`. + +.. autofunction:: tractor._context.open_context_from_portal + +.. note:: + + :func:`~tractor._context.open_context_from_portal` is bound as + the **method-alias** ``Portal.open_context()`` — that's the + spelling you should actually call: + ``portal.open_context(fn, **kwargs)``. See :doc:`/api/context` + for the full ``Context`` + ``MsgStream`` API it unlocks. + +.. note:: + + :meth:`Portal.cancel_actor` cancels the *whole* remote runtime + and process (machine-level), not a single task — use + :meth:`Context.cancel` for task-level cancellation. Pass + ``raise_on_timeout=True`` to get an ``ActorTooSlowError`` you + can escalate per SC discipline (see :doc:`/api/errors`). + +Clusters +-------- + +.. autofunction:: open_actor_cluster + +Spawn a *flat* cluster of ``count`` worker actors (default: one +per core) all serving the RPC ``modules`` list, yielding a +``dict[str, Portal]`` keyed by actor name. Handy for +embarrassingly parallel fan-out; see ``examples/quick_cluster.py``. + +Runtime introspection +--------------------- + +.. autofunction:: current_actor + +.. autoclass:: Actor + :members: aid, + name, + uid, + is_registrar, + is_infected_aio, + cancel_soon + +.. note:: + + :class:`Actor` is the per-process runtime singleton (msg loop, + RPC scheduling, IPC server) — you never instantiate it yourself + and should normally only touch the identity/introspection + surface listed above. The canonical identity type is + :attr:`Actor.aid` (a ``tractor.msg.Aid`` struct); + :attr:`Actor.uid` is the legacy ``(name, uuid)`` 2-tuple which + is still pervasive in logs and error metadata. + +.. deprecated:: 0.1.0a6 + + ``Actor.is_arbiter`` warns; use :attr:`Actor.is_registrar`. + The ``arbiter_addr`` constructor kwarg is deprecated for + ``registry_addrs``. + +.. autofunction:: current_ipc_ctx + +.. autofunction:: is_root_process + +.. autofunction:: get_runtime_vars + +.. seealso:: + + :doc:`/api/context` for the SC-linked remote task API, + :doc:`/api/discovery` for finding actors by name, and the + guided tours in :doc:`/guide/spawning`, :doc:`/guide/rpc` and + :doc:`/guide/context`. + +.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate +.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics diff --git a/docs/api/devx.rst b/docs/api/devx.rst new file mode 100644 index 00000000..b45bcb87 --- /dev/null +++ b/docs/api/devx.rst @@ -0,0 +1,92 @@ +Debugging and devx: ``tractor.devx`` +==================================== + +Multi-process debugging that actually works: boot the tree with +``open_root_actor(debug_mode=True)`` (or pass it to +``open_nursery()``) and any crash or explicit pause in *any* actor +acquires a tree-global TTY lock and drops you into a +`pdbp`_-powered REPL — one actor at a time, ``SIGINT`` shielded, +no garbled terminals. The top-level helpers below are the daily +drivers; the rest of the toolbox lives under ``tractor.devx``. + +.. currentmodule:: tractor + +Pausing and post-mortems +------------------------ + +.. autofunction:: pause + +.. autofunction:: pause_from_sync + +.. note:: + + :func:`pause_from_sync` needs the `greenback`_ portal: boot + with ``open_root_actor(maybe_enable_greenback=True)`` (mind + the `performance implications`_). With ``debug_mode`` on, the + built-in ``breakpoint()`` is also remapped to a + ``tractor``-safe equivalent. + +.. autofunction:: post_mortem + +.. deprecated:: 0.1.0a6 + + ``tractor.breakpoint()`` warns and simply calls :func:`pause` + — use :func:`tractor.pause` (async) or + :func:`tractor.pause_from_sync` in new code. + +Crash handling for CLIs and sync entrypoints +-------------------------------------------- + +.. currentmodule:: tractor.devx + +.. autofunction:: open_crash_handler + +.. autofunction:: maybe_open_crash_handler + +.. note:: + + Both are *sync* context managers usable before (or without) + ``trio.run()`` — wrap your CLI ``main()`` to get a post-mortem + REPL on any uncaught exception instead of a bare traceback. + +Runtime hang-hunting +-------------------- + +.. autofunction:: enable_stack_on_sig + +With stackscope_ integration enabled (also via +``open_root_actor(enable_stack_on_sig=True)`` or the +``TRACTOR_ENABLE_STACKSCOPE`` env var) a ``SIGUSR1`` triggers a +full trio task-tree dump from every actor — works on live, +*non*-debug-mode trees too: + +.. code:: sh + + pkill --signal SIGUSR1 -f + +Dumps also tee to ``/tmp/tractor-stackscope-.log`` so you +still get output under captured/CI stdio. + +Lower-level debug plumbing +-------------------------- + +.. autofunction:: mk_pdb + +.. autofunction:: maybe_wait_for_debugger + +:func:`maybe_wait_for_debugger` is mainly useful in runtime/test +code that must avoid tearing down a tree while a child still +holds the global debug lock. + +.. seealso:: + + :doc:`/api/errors` for the boxed error types you'll inspect + from the REPL, :doc:`/api/core` for the ``debug_mode`` / + ``maybe_enable_greenback`` / ``enable_stack_on_sig`` boot + flags on :func:`tractor.open_root_actor`, and + :doc:`/guide/debugging` for the guided multi-actor REPL tour. + +.. _pdbp: https://github.com/mdmintz/pdbp +.. _greenback: https://greenback.readthedocs.io/en/latest/ +.. _performance implications: https://greenback.readthedocs.io/en/latest/principle.html#performance +.. _stackscope: https://github.com/oremanj/stackscope diff --git a/docs/api/discovery.rst b/docs/api/discovery.rst new file mode 100644 index 00000000..b054c4d4 --- /dev/null +++ b/docs/api/discovery.rst @@ -0,0 +1,67 @@ +Discovery and the registrar +=========================== + +Every actor registers its ``(name, uuid)`` and transport addresses +with a *registrar* actor — by default the root of the tree, or +whichever actor serves at the ``registry_addrs`` you boot with. +The discovery API lets any actor look up any other **by name** and +get back a connected :class:`~tractor.Portal`, giving you +service-discovery patterns (daemons, service trees, multi-host +meshes) without hard-coding addresses. + +Lookups first scan already-connected peers before RPC-ing the +registrar, and multihomed results are ranked UDS > local TCP > +remote TCP. See ``examples/service_daemon_discovery.py`` for the +canonical daemon + lookup pattern. + +.. currentmodule:: tractor + +Lookup APIs +----------- + +.. autofunction:: find_actor + +.. autofunction:: wait_for_actor + +.. autofunction:: query_actor + +.. autofunction:: get_registry + +.. note:: + + :func:`find_actor` yields ``None`` when nothing is registered + under the name (or raises with ``raise_on_none=True``); + :func:`wait_for_actor` blocks until the name appears; + :func:`query_actor` only *looks up* the address without + connecting to the target. + +The ``Registrar`` +----------------- + +.. autoclass:: Registrar + :show-inheritance: + +A :class:`Registrar` is just an :class:`~tractor.Actor` subtype +maintaining the name -> addresses table; you rarely touch it +directly beyond passing ``registry_addrs`` / +``ensure_registry=True`` to :func:`~tractor.open_root_actor`. +Check :attr:`Actor.is_registrar ` to +ask "am I it?". + +Legacy ``Arbiter`` alias +------------------------ + +.. deprecated:: 0.1.0a6 + + ``tractor.Arbiter`` survives only as a class alias of + :class:`Registrar` and all "arbiter" terminology is replaced + by "registrar"/"registry" across the API: ``get_arbiter()`` is + removed (use :func:`get_registry`) and the ``arbiter_addr`` + kwarg is replaced by ``registry_addrs``. + +.. seealso:: + + :doc:`/api/core` for booting a registrar via + ``open_root_actor(registry_addrs=...)``, :doc:`/api/ipc` for + the transport/address model the registry stores, and + :doc:`/guide/discovery` for the worked walkthrough. diff --git a/docs/api/errors.rst b/docs/api/errors.rst new file mode 100644 index 00000000..abfb0966 --- /dev/null +++ b/docs/api/errors.rst @@ -0,0 +1,102 @@ +Errors and cancellation types +============================= + +``tractor`` extends trio's "exceptions `always propagate`_" rule +across the process boundary: a crash in any actor is serialized as +an ``Error`` msg, shuttled over IPC, and re-raised in the linked +parent scope as a *boxed* :class:`RemoteActorError` — preserving +the original type, traceback text and source-actor identity, even +across multi-hop relays (a.k.a. "inceptions"). + +The most-used types below are importable from ``tractor`` +directly; the remainder live in ``tractor._exceptions`` (not yet +re-exported at top level). + +.. currentmodule:: tractor + +Boxed remote errors +------------------- + +.. autoexception:: RemoteActorError + :members: boxed_type, + src_uid, + relay_uid, + pformat + +.. code:: python + + try: + async with portal.open_context(ep_fn) as (ctx, first): + ... + except tractor.RemoteActorError as rae: + if rae.boxed_type is ValueError: + ... # remote task raised a `ValueError` + +.. autoexception:: ContextCancelled + :show-inheritance: + :members: canceller + +.. note:: + + Inspect :attr:`ContextCancelled.canceller` (the requesting + actor's uid) to distinguish a *self*-requested cancel (absorbed + at ``open_context()`` exit) from a *cross*-actor cancel (raised + locally) — the full rules live in :doc:`/api/context`. + +Typed-messaging errors +---------------------- + +.. autoexception:: MsgTypeError + :show-inheritance: + :members: bad_msg, + expected_msg_type + +An "IPC ``TypeError``": a message failed validation against the +active msg-spec / ``pld_spec`` (see :doc:`/api/msg`). Raised +sender-side for control msgs (``Started``/``Return``) and +receiver-side for stream ``Yield`` payloads. + +.. autoexception:: tractor._exceptions.StreamOverrun + :show-inheritance: + +The sender out-paced the receiver's buffer on a +:class:`~tractor.MsgStream` opened without +``allow_overruns=True``; subtypes :class:`trio.TooSlowError`. + +Transport and runtime errors +---------------------------- + +.. autoexception:: TransportClosed + +.. autoexception:: ModuleNotExposed + :show-inheritance: + +Raised when an RPC requests a function from a module not listed +in the target actor's ``enable_modules`` allowlist — +capability-style access control, not an import bug on your end ;) + +.. autoexception:: tractor._exceptions.NoRuntime + :show-inheritance: + +Raised by :func:`tractor.current_actor` (and friends) when no +actor runtime is up in the current process. + +.. autoexception:: tractor._exceptions.ActorTooSlowError + :show-inheritance: + +A peer actor failed to ack a cancel request within the bounded +wait — the SC-sanctioned escalation signal from APIs like +``Portal.cancel_actor(raise_on_timeout=True)``. Catch it to +escalate (e.g. hard-kill via the supervising +:class:`~tractor.ActorNursery`); never just ignore it, that's how +zombies happen. + +.. seealso:: + + :doc:`/api/context` for how cancellation and errors flow + through a :class:`~tractor.Context`, :doc:`/api/devx` for + crash-handling REPL tooling (``debug_mode``, post-mortems), + and :doc:`/guide/cancellation` for the full SC-cancellation + story. + +.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate diff --git a/docs/api/index.rst b/docs/api/index.rst new file mode 100644 index 00000000..f2b90900 --- /dev/null +++ b/docs/api/index.rst @@ -0,0 +1,67 @@ +API reference +============= + +This is the curated reference for ``tractor``'s public surface: the +names you can import and lean on without reading runtime internals. +Everything below is re-exported at the top level (``import +tractor``) unless a page says otherwise; subsystems like +``tractor.msg``, ``tractor.trionics``, ``tractor.to_asyncio``, +``tractor.devx`` and ``tractor.log`` are importable as submodules. + +``tractor`` is "just trio_" extended across processes: every API +here is designed to keep the structured concurrency (SC) rules you +already know from the `trio docs`_ intact across the process +boundary. If a name isn't documented here it's an internal — expect +it to change without notice B). + +.. currentmodule:: tractor + +Most-used names at a glance: + +.. autosummary:: + :nosignatures: + + open_root_actor + open_nursery + run_daemon + ActorNursery + Portal + context + Context + MsgStream + open_actor_cluster + find_actor + wait_for_actor + get_registry + current_actor + current_ipc_ctx + is_root_process + get_runtime_vars + RemoteActorError + ContextCancelled + MsgTypeError + pause + post_mortem + Channel + +.. toctree:: + :maxdepth: 1 + :caption: Reference pages + + core + context + discovery + errors + msg + trionics + to_asyncio + devx + ipc + +Where to next? If you're new, start with the runtime and spawning +APIs in :doc:`/api/core`, then graduate to the inter-actor task +linking model in :doc:`/api/context` — it's the heart of the whole +system. + +.. _trio: https://github.com/python-trio/trio +.. _trio docs: https://trio.readthedocs.io/en/latest/ diff --git a/docs/api/ipc.rst b/docs/api/ipc.rst new file mode 100644 index 00000000..a9a36f80 --- /dev/null +++ b/docs/api/ipc.rst @@ -0,0 +1,89 @@ +IPC and logging +=============== + +Under every portal, context and stream sits a per-peer +:class:`~tractor.Channel`: a msgpack-typed messaging link wrapping +one OS transport connection. Transports are pluggable per actor +via ``enable_transports=['tcp' | 'uds']`` — TCP is the default, +UDS (unix domain sockets) gives you port-less, same-host IPC with +kernel-provided peer credentials for free — and exactly **one** +transport may currently be enabled per actor. + +.. d2:: diagrams/runtime_stack.d2 + :caption: Where ``Channel`` sits in the runtime stack. + :margin: + :alt: layered runtime stack from app code down to transports + +Addresses are "unwrapped" tuples at the API edges: +``('host', port)`` for TCP, filesystem-path pairs for UDS. For +the full layering story — transport protocols, the IPC server, +address types and the msg loop — see +:doc:`/explain/architecture`. + +.. currentmodule:: tractor + +``Channel`` +----------- + +.. autoclass:: Channel + :members: from_addr, + send, + recv, + aclose, + connected, + apply_codec, + aid, + laddr, + raddr, + closed + +.. deprecated:: 0.1.0a6 + + ``Channel.uid`` warns; use :attr:`Channel.aid` which carries + richer (optional) identity fields beyond the legacy + ``(name, uuid)`` pair. + +.. note:: + + You rarely construct a :class:`Channel` yourself — the runtime + hands them out via :attr:`Portal.chan ` + and :attr:`Context.chan `. Treat the + send/recv surface as advanced API: normal apps should speak + :class:`~tractor.MsgStream` instead. + +Choosing a transport +-------------------- + +.. literalinclude:: ../../examples/uds_transport_actor_tree.py + :caption: examples/uds_transport_actor_tree.py + :language: python + +Logging +------- + +``tractor.log`` provides the structured, colorized console +logging used across the runtime — with actor-name + task-aware +record headers and extra log levels below :data:`logging.DEBUG` +(``'transport'``, ``'runtime'``, ``'cancel'``, ``'devx'``) for +spelunking the runtime itself. Use it for your app too: it's +already distributed-system aware. + +.. currentmodule:: tractor.log + +.. autofunction:: get_logger + +.. autofunction:: get_console_log + +.. note:: + + The ``TRACTOR_LOGLEVEL`` env var overrides any caller-passed + ``loglevel`` (e.g. to ``open_root_actor()``) so you can crank + console verbosity without touching code; subactors inherit + the root's level by default. + +.. seealso:: + + :doc:`/explain/architecture` for the transport/server + internals, :doc:`/api/discovery` for how channel addresses + get registered and found, and :doc:`/api/msg` for the codec + layer every channel speaks. diff --git a/docs/api/msg.rst b/docs/api/msg.rst new file mode 100644 index 00000000..4062b702 --- /dev/null +++ b/docs/api/msg.rst @@ -0,0 +1,107 @@ +Typed messaging: ``tractor.msg`` +================================ + +All inter-actor communication rides a small, strictly-typed +msgpack wire protocol built from :class:`msgspec.Struct` types — +the "SC-shuttle" protocol that powers contexts, streams, RPC and +cancellation. You normally never touch these msg types directly +(the :class:`~tractor.Context` API speaks them for you) but you +*do* use this subpackage to define **payload type contracts**: per +endpoint via ``@tractor.context(pld_spec=...)`` or per channel via +custom codecs. + +Violations of an active msg-spec surface as +:class:`~tractor.MsgTypeError` (see :doc:`/api/errors`); the full +typed-payload workflow is shown in ``examples/typed_payloads.py``. + +.. currentmodule:: tractor.msg + +The protocol message set +------------------------ + +.. autosummary:: + :nosignatures: + + PayloadMsg + Aid + SpawnSpec + Start + StartAck + Started + Yield + Stop + Return + CancelAck + Error + +``Aid`` (identity handshake) and ``SpawnSpec`` (parent -> child +init) run at connection setup; ``Start``/``StartAck`` initiate an +RPC task; ``Started``/``Yield``/``Stop``/``Return`` are the +:class:`~tractor.Context` dialog phases; ``CancelAck`` and +``Error`` close the loop on cancellation and (boxed) failure. +``Msg`` is a legacy alias of ``PayloadMsg``. The union of all of +the above is exported as ``MsgType`` (also ``__msg_spec__``). + +.. automodule:: tractor.msg.types + :no-members: + +.. currentmodule:: tractor.msg + +Codec construction and override +------------------------------- + +.. autofunction:: mk_codec + +.. autoclass:: MsgCodec + :members: encode, + decode, + msg_spec + +.. autofunction:: mk_dec + +.. autoclass:: MsgDec + :members: decode, + spec + +.. autofunction:: apply_codec + +.. autofunction:: current_codec + +.. note:: + + :func:`apply_codec` swaps the codec via a + :class:`contextvars.ContextVar` — the override only applies to + the *current task* (and tasks it starts), not sibling tasks + already running in the actor. Payload-decoding is layered: the + outer codec leaves ``.pld`` fields as ``msgspec.Raw`` and each + context's payload-receiver decodes them against *its* spec + (the "cheap-or-nasty" validation pattern). + +Namespace pointers +------------------ + +.. autoclass:: NamespacePath + :members: from_ref, + load_ref, + to_tuple + +The ``'module.path:obj_name'`` :class:`str`-subtype used to +address every RPC target function over the wire (same format as +:func:`pkgutil.resolve_name`). + +Pretty structs +-------------- + +.. autoclass:: Struct + :show-inheritance: + +A :class:`msgspec.Struct` subtype with a multi-line pretty +``__repr__`` — handy as a base for your own IPC payload types so +crash logs stay readable. + +.. seealso:: + + :doc:`/api/context` for where ``pld_spec`` typing plugs into + the ``@context`` decorator, :doc:`/api/errors` for + :class:`~tractor.MsgTypeError` semantics, and + :doc:`/guide/msging` for the guided typed-messaging tour. diff --git a/docs/api/to_asyncio.rst b/docs/api/to_asyncio.rst new file mode 100644 index 00000000..05baad9a --- /dev/null +++ b/docs/api/to_asyncio.rst @@ -0,0 +1,97 @@ +asyncio interop: ``tractor.to_asyncio`` +======================================= + +"Infected asyncio" mode: spawn an actor with +``start_actor(..., infect_asyncio=True)`` and its process runs +:mod:`trio` as a `guest`_ on top of the :mod:`asyncio` loop — +letting your trio task tree drive asyncio tasks *in the same +process* while the rest of the actor tree stays pure trio. Each +trio <-> asyncio task pair is linked with structured concurrency +(SC) semantics: error or cancellation on either side tears down +both, with the cause translated cross-loop. + +.. d2:: diagrams/infected_aio.d2 + :caption: A trio guest driving asyncio tasks in one actor. + :margin: + :alt: trio guest mode inside an asyncio-infected actor + +See ``examples/infected_asyncio_echo_server.py`` for a complete +worked example. + +.. currentmodule:: tractor.to_asyncio + +Starting asyncio tasks from trio +-------------------------------- + +.. autofunction:: open_channel_from + +.. autofunction:: run_task + +.. note:: + + :func:`open_channel_from` mirrors the + ``Portal.open_context()`` handshake: the asyncio side calls + ``chan.started_nowait(value)`` and that value pops out as + ``first`` on the trio side. :func:`run_task` is the one-shot + form — run a single asyncio-compatible coroutine fn and return + its result to trio. + +The inter-loop channel +---------------------- + +.. autoclass:: LinkedTaskChannel + :members: send, + receive, + wait_for_result, + subscribe, + started_nowait, + send_nowait, + get, + cancel_asyncio_task, + closed + +.. note:: + + The trio side uses the async API + (:meth:`LinkedTaskChannel.send` / + :meth:`LinkedTaskChannel.receive`); the asyncio side uses the + loop-safe sync/await mix + (:meth:`LinkedTaskChannel.send_nowait` / + :meth:`LinkedTaskChannel.get` / + :meth:`LinkedTaskChannel.started_nowait`). + +Translated exception types +-------------------------- + +Cross-loop failures are re-raised on the *other* side as one of +these explicit translation types, so you always know which loop +actually died first: + +.. autoexception:: tractor._exceptions.AsyncioCancelled + +.. autoexception:: tractor._exceptions.AsyncioTaskExited + +.. autoexception:: tractor._exceptions.TrioCancelled + +.. autoexception:: tractor._exceptions.TrioTaskExited + +.. autoexception:: AsyncioRuntimeTranslationError + :show-inheritance: + +Guest-mode entrypoint +--------------------- + +``run_as_asyncio_guest()`` is the runtime-internal entrypoint that +boots trio in `guest`_ mode inside an infected actor — you get it +implicitly via ``infect_asyncio=True`` and shouldn't need to call +it yourself. + +.. seealso:: + + :doc:`/api/core` for the ``infect_asyncio`` spawn flag, + :meth:`tractor.Actor.is_infected_aio` for runtime + introspection, :doc:`/api/devx` for using the debugger from + inside asyncio tasks, and :doc:`/guide/asyncio` for the + guided tour. + +.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops diff --git a/docs/api/trionics.rst b/docs/api/trionics.rst new file mode 100644 index 00000000..a92a290a --- /dev/null +++ b/docs/api/trionics.rst @@ -0,0 +1,76 @@ +Trio patterns: ``tractor.trionics`` +=================================== + +Sugary structured concurrency (SC) patterns for plain :mod:`trio` +code — **no actor runtime required**. These helpers grew out of +real distributed-system needs in ``tractor`` apps but every one of +them works in a single-process program too; import via +``from tractor import trionics``. + +.. currentmodule:: tractor.trionics + +Context-manager helpers +----------------------- + +.. autofunction:: gather_contexts + +.. autofunction:: maybe_open_context + +.. autofunction:: maybe_open_nursery + +.. note:: + + :func:`gather_contexts` is "a nursery for async context + managers": it enters N acms concurrently and yields their + values in input order. :func:`maybe_open_context` is the + actor-wide cache/multiplex layer on top — the first task pays + the acm setup cost, later callers get ``(cache_hit=True, ...)`` + and share the same value until all users exit. + +Broadcast fan-out +----------------- + +.. autofunction:: broadcast_receiver + +.. autoclass:: BroadcastReceiver + :members: receive, + subscribe, + aclose + +.. autoexception:: Lagged + :show-inheritance: + +A single-producer, many-consumer broadcast layer over any +``trio``-style receive channel: non-lossy for the *fastest* +consumer while slower consumers raise :class:`Lagged` (a +:class:`trio.TooSlowError` subtype) once they fall behind the +internal ring. This is exactly the machinery behind +:meth:`tractor.MsgStream.subscribe` — see +``examples/streaming_broadcast_fanout.py``. + +ExceptionGroup helpers +---------------------- + +.. autofunction:: collapse_eg + +.. autofunction:: maybe_raise_from_masking_exc + +.. note:: + + :func:`collapse_eg` "un-nests" single-exception + :class:`ExceptionGroup` wrappers from strict-eg ``trio`` + nurseries so your ``except`` clauses match the original error; + :func:`maybe_raise_from_masking_exc` surfaces real errors that + would otherwise be masked by :class:`trio.Cancelled` during + teardown. + +.. seealso:: + + :doc:`/api/context` for the IPC-stream consumer of + :class:`BroadcastReceiver`, :doc:`/guide/streaming` for + fan-out in a worked pipeline, and the `trio docs`_ for the + underlying channel and `nursery`_ semantics these helpers + compose. + +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning diff --git a/docs/dev_tips.rst b/docs/dev_tips.rst deleted file mode 100644 index 93e812dc..00000000 --- a/docs/dev_tips.rst +++ /dev/null @@ -1,51 +0,0 @@ -Hot tips for ``tractor`` hackers -================================ - -This is a WIP guide for newcomers to the project mostly to do with -dev, testing, CI and release gotchas, reminders and best practises. - -``tractor`` is a fairly novel project compared to most since it is -effectively a new way of doing distributed computing in Python and is -much closer to working with an "application level runtime" (like erlang -OTP or scala's akka project) then it is a traditional Python library. -As such, having an arsenal of tools and recipes for figuring out the -right way to debug problems when they do arise is somewhat of -a necessity. - - -Making a Release ----------------- -We currently do nothing special here except the traditional -PyPa release recipe as in `documented by twine`_. I personally -create sub-dirs within the generated `dist/` with an explicit -release name such as `alpha3/` when there's been a sequence of -releases I've made, but it really is up to you how you like to -organize generated sdists locally. - -The resulting build cmds are approximately: - -.. code:: bash - - python setup.py sdist -d ./dist/XXX.X/ - - twine upload -r testpypi dist/XXX.X/* - - twine upload dist/XXX.X/* - - - -.. _documented by twine: https://twine.readthedocs.io/en/latest/#using-twine - - -Debugging and monitoring actor trees ------------------------------------- -TODO: but there are tips in the readme for some terminal commands -which can be used to see the process trees easily on Linux. - - -Using the log system to trace `trio` task flow ----------------------------------------------- -TODO: the logging system is meant to be oriented around -stack "layers" of the runtime such that you can track -"logical abstraction layers" in the code such as errors, cancellation, -IPC and streaming, and the low level transport and wire protocols. diff --git a/docs/explain/architecture.rst b/docs/explain/architecture.rst new file mode 100644 index 00000000..37a7f147 --- /dev/null +++ b/docs/explain/architecture.rst @@ -0,0 +1,368 @@ +Anatomy of the runtime +====================== + +You can get a long way with ``tractor`` by treating it as "trio_ +with nurseries that spawn processes". But once you start asking +*where does my msg actually go?*, *which process is that?* or +*who keeps the phonebook?*, it pays to know how the runtime hangs +together. This page walks the stack top to bottom. + +.. d2:: diagrams/runtime_stack.d2 + :caption: The four runtime layers inside *every* actor process. + :alt: layer cake of app tasks, tractor runtime, IPC, OS process + :width: 70% + +The layer cake +-------------- + +Every actor process is the same four-layer sandwich: + +- **your app**: plain ``trio`` tasks, nurseries and cancel + scopes; nothing special. ``tractor`` is a `structured + concurrency`_ (SC) multi-processing runtime built on trio_ and + the whole pitch is that this layer stays *just trio*: no + callbacks, no futures, no proxy objects. +- **the** ``tractor`` **runtime**: a per-process + :class:`tractor.Actor` running the msg loop and RPC task + scheduler, plus the user-facing primitives layered on it: + :class:`tractor.ActorNursery` (spawning + supervision), + :class:`tractor.Portal` (calling into a peer) and + :class:`tractor.Context` + :class:`tractor.MsgStream` + (SC-linked cross-actor task pairs and streaming). +- **IPC channels**: one :class:`tractor.Channel` per connected + peer, each wrapping a ``MsgTransport`` that ships + msgspec_-typed msgs over TCP or UDS. +- **the OS**: one process per actor, started by a swappable + spawn backend. + +The property that holds it all together: SC composes *through* +the layers. A crash in a leaf actor's app task unwinds that +actor's trio tree, ships across its IPC channel as a typed +``Error`` msg, and unwinds the parent's trio tree in turn — the +"SC-transitive supervision protocol" from the README's pitch. +The whole tree cancels and errors like one big trio program; it +just happens to be spread across processes. + +One actor, one process, one ``trio.run()`` +------------------------------------------ + +A ``tractor`` "actor" is not a green thread, nor an object with +a mailbox, nor a coroutine: it's one OS process running one +:func:`trio.run` whose root task boots the runtime machinery — +msg loop, RPC task scheduler, IPC server — all embodied by a +single :class:`tractor.Actor` instance. + +.. margin:: Shared nothing + + Processes buy you a real `shared nothing architecture`_: no + accidentally-shared mutable state, no GIL contention, and + every actor can be inspected (or killed) like any other OS + process. + +You rarely construct an :class:`~tractor.Actor` yourself; the +runtime makes exactly one per process and you grab it with +:func:`tractor.current_actor`: + +.. code:: python + + import tractor + + actor = tractor.current_actor() # NoRuntime if none running + print(actor.aid.name) # str name, need not be unique + print(actor.aid.uuid) # uuid4 str, IS unique + print(actor.aid.pid) # the OS pid + print(actor.uid) # legacy (name, uuid) pair + +Identity is carried by the ``Aid`` msg-struct (see +``tractor.msg.types``): a ``name``/``uuid``/``pid`` triple +exchanged in the very first "mailbox handshake" whenever two +actors connect. It's what the registrar stores and what shows up +in logs and proc-titles. The older ``.uid`` 2-tuple of +``(name, uuid)`` predates ``Aid`` and is still pervasive across +the codebase; treat it as the legacy spelling of the same +identity. + +If this smells like the `actor model`_, sure — but as the +README warns, it probably doesn't look like what you *think* an +actor model looks like, and that's intentional. Here an "actor" +is purely a runtime-unit-of-abstraction: process + +``trio.run()`` + IPC machinery. + +IPC: channels, transports, addresses +------------------------------------ + +Two connected actors talk through a :class:`tractor.Channel`: a +duplex, per-peer msg pipe. Each ``Channel`` wraps a +``MsgTransport`` instance which does the wire work: framing, +encode/decode and the socket itself. The encoding is msgpack +(via msgspec_) and *every* msg is an instance of one of the +runtime's tagged-union :class:`msgspec.Struct` types: the +``Aid`` handshake, ``Start``/``StartAck`` (RPC init), +``Started``/``Yield``/``Stop``/``Return`` (the ctx dialog +phases), ``Error``, etc. There is no raw-bytes mode; the +msg-spec *is* the protocol, which is exactly what lets payloads +be type-limited per-context (see ``pld_spec`` in +:doc:`/guide/context`). + +Addresses come in two spellings: + +- *unwrapped*: the plain-tuple form you pass to user APIs — + ``('127.0.0.1', 1616)`` for tcp, or a + ``(, )`` path-pair for uds; +- *wrapped*: the internal ``TCPAddress``/``UDSAddress`` struct + types (plus libp2p-style multiaddr helpers over in + ``tractor.discovery``). + +You only ever need the tuple form; the runtime wraps and +unwraps at the boundaries. + +TCP: the boring default +*********************** + +The default transport (``'tcp'``) binds each actor's IPC server +to loopback ``('127.0.0.1', )`` unless told +otherwise, and is the only choice when your tree spans hosts. +Nothing exotic: ``trio`` TCP streams + length-prefixed msgpack +framing. + +UDS: same-host, creds included +****************************** + +Pass ``enable_transports=['uds']`` and actors instead talk over +unix-domain sockets, with socket files placed in the per-user +runtime dir (``$XDG_RUNTIME_DIR/tractor/`` on linux, the +``platformdirs`` equivalent elsewhere). Two perks over tcp on a +single host: + +- no ports to fight over; addrs are just file paths, +- the kernel snitches on your peer for free: the listening side + reads the connector's ``pid`` (plus ``uid``/``gid`` on linux) + straight off the socket via ``SO_PEERCRED`` / + ``LOCAL_PEERPID`` — no extra handshake msgs required B) + +.. warning:: + + Socket-file lifetime == listening actor lifetime. On + listener teardown the runtime ``os.unlink()``\s the socket + file immediately, so any *late* connection attempt (say, a + sub-actor racing to deregister with a registrar that's + already shutting down) fails with ``FileNotFoundError``. + And ofc, UDS is same-host only. + +Here's a full actor tree run entirely over uds: + +.. literalinclude:: ../../examples/uds_transport_actor_tree.py + :caption: examples/uds_transport_actor_tree.py + :language: python + +Picking a transport +******************* + +Transport choice is per-actor via the ``enable_transports`` +kwarg accepted by :func:`tractor.open_root_actor` (and proxied +through ``open_nursery()`` when it implicitly boots the +runtime), plus per-child via +``ActorNursery.start_actor(enable_transports=...)``. Two rules +the runtime enforces today: + +- exactly ONE transport per actor: multi-transport actors are + on the roadmap but currently raise ``RuntimeError``; +- your ``registry_addrs`` protos must all be in + ``enable_transports``: mismatches fail fast with + ``ValueError`` instead of (as in darker times) hanging the + registrar handshake forever. + +Spawn backends +-------------- + +How does an actor actually *become* a process? Via a swappable +spawn backend, selected with the ``start_method`` kwarg to +:func:`tractor.open_root_actor`: + +``'trio'`` (default) + The home-grown spawner: re-exec the child as + ``python -m tractor._child`` using ``trio``'s subprocess + machinery, then bootstrap it over the first IPC exchange + (the parent ships a ``SpawnSpec`` msg carrying all init + state). Supported on all platforms and the most battle + tested choice by far. + +``'mp_spawn'`` / ``'mp_forkserver'`` + The stdlib :mod:`multiprocessing` start-methods of the same + names (forkserver is posix-only). Mostly interesting for + ecosystem compat and start-up-latency tuning. + +``'subint'`` (experimental, py3.14+) + Each actor runs as a `PEP 734`_ sub-interpreter + (``concurrent.interpreters``) driven on its own OS thread + *inside the parent process*: interpreter-level + shared-nothing isolation with much faster start-up. Yes, + this bends the one-actor-one-process rule; the rest of the + model is unchanged. + +The ``TRACTOR_SPAWN_METHOD`` env-var beats any caller-passed +``start_method``, so you can swap backends under an unmodified +app: + +.. code:: bash + + TRACTOR_SPAWN_METHOD=mp_forkserver python my_app.py + +One current limitation worth knowing: ``debug_mode=True`` (the +crash-to-REPL machinery) is only supported on backends whose +child-side runtime is trio-native, e.g. the default ``'trio'``; +see :doc:`/guide/debugging` for the deats. + +The registrar +------------- + +Discovery needs a phonebook. Every actor, as part of boot, +registers its ``Aid`` and bind-addrs with the *registrar*: an +otherwise ordinary actor (a :class:`tractor.Registrar`, subtype +of :class:`~tractor.Actor`) that keeps the name -> addrs table +for the tree; on graceful exit each actor de-registers itself. + +.. margin:: Default registry addrs + + With no ``registry_addrs`` passed: + + - tcp: ``('127.0.0.1', 1616)`` + - uds: ``registry@1616.sock`` + in the runtime dir + +Who *is* the registrar? Decided at root boot, rendezvous style. +:func:`tractor.open_root_actor` probes each addr in +``registry_addrs`` with a quick connect-ping, then: + +- **somebody answered**: this root is a plain actor; it + registers with the existing registrar and binds random + same-proto addrs for its own IPC server; +- **nobody answered**: this root *becomes* the registrar and + binds the registry addrs itself. + +So single-program trees need zero config — the root quietly +self-appoints — while multi-program setups share a registrar by +pointing every program at the same ``registry_addrs``. Pass +``ensure_registry=True`` to demand that *this* call create the +registry; it raises if the addrs are already served. + +The lookup APIs — :func:`tractor.find_actor`, +:func:`tractor.wait_for_actor`, :func:`tractor.query_actor` and +:func:`tractor.get_registry` — all consult it (after first +checking already-connected peers): + +.. literalinclude:: ../../examples/service_daemon_discovery.py + :caption: examples/service_daemon_discovery.py + :language: python + +If you bump into "arbiter" in old issues or posts: that's the +legacy name for the same thing, surviving in-code only as the +``Arbiter = Registrar`` class alias; all current terminology is +"registrar"/"registry". Fair warning per the README: this is +still a **very naive** discovery sys (no re-election, no gossip +protocol... yet) and a registrar is expected to outlive its +registrants. + +Runtime env vars +---------------- + +A few env-vars let you re-tune a whole tree *without touching +app code*; each wins over its corresponding kwarg: + +.. list-table:: + :header-rows: 1 + :widths: 30 44 26 + + * - env-var + - effect + - vs. kwarg + * - ``TRACTOR_LOGLEVEL`` + - crank (or silence) console-log verbosity for every actor + in the tree + - beats ``loglevel`` + * - ``TRACTOR_SPAWN_METHOD`` + - swap the process spawn backend + - beats ``start_method`` + * - ``TRACTOR_ENABLE_STACKSCOPE`` + - install the ``SIGUSR1`` task-tree-dump handler in every + actor, even outside ``debug_mode`` (see + :doc:`/guide/debugging`) + - OR'd with ``enable_stack_on_sig`` + +Spotting actors from your shell +******************************* + +Every sub-actor sets an OS-level proc-title of the form +``_subactor[@]`` (via ``setproctitle``, silently +skipped when not installed) so ``ps``/``htop``/``pstree`` show +*which actor is which* at a glance. The README's signature +incantation — watch a tree build and self-destruct live: + +.. code:: bash + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/nested_actor_tree.py \ + && kill $! + +For scripting there are two stable cmdline markers: + +.. code:: bash + + pgrep -fa '_subactor\[' # live, titled sub-actors + pgrep -fa 'tractor._child' # 'trio'-backend children not + # yet (re)titled + +The title also lands in the kernel ``comm`` (truncated to ~15 +bytes) which survives into zombie state — that's what +``tractor``'s own test-harness reapers key off. To be crystal +clear about the contract though: you should never *need* a +reaper; if you can create zombie child processes (without using +a system signal) it **is a bug** — please report it! + +Logging +------- + +The runtime logs through a thin adapter over stdlib +:mod:`logging` that stamps every record with actor + task info. +Two calls get you going: + +.. code:: python + + from tractor.log import get_console_log, get_logger + + log = get_logger(__name__) # actor/task-aware sub-logger + get_console_log('info') # attach console handler @ level + +(or just pass ``loglevel='info'`` to +:func:`tractor.open_root_actor` and the console handler comes up +with the runtime). + +``tractor`` adds custom levels — and matching logger methods — +that slot between the stdlib ones so you can dial in *which +runtime subsystem* you want to hear from: ``.transport()`` (5), +``.runtime()`` (15), ``.devx()`` (17), ``.cancel()`` (22), plus +a ``PDB`` (500) level for debugger chatter. E.g. +``loglevel='cancel'`` plays the whole cancellation chorus while +staying quiet about transport-layer noise. Beyond that +``tractor`` isn't opinionated about how you consume logs: it's +all stdlib ``logging`` underneath. + +Where to next? +-------------- + +.. seealso:: + + - :doc:`/guide/context` — the SC-linked cross-actor task API + that rides on every ``Channel``. + - :doc:`/guide/debugging` — ``debug_mode``, the multi-process + REPL and ``stackscope`` task-tree dumps. + - :doc:`/explain/sc-distributed` — *why* + one-actor-one-process, and what kind of "actor model" this + is (and isn't). + +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _trio: https://github.com/python-trio/trio +.. _actor model: https://en.wikipedia.org/wiki/Actor_model +.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture +.. _msgspec: https://jcristharif.com/msgspec/ +.. _PEP 734: https://peps.python.org/pep-0734/ diff --git a/docs/explain/index.rst b/docs/explain/index.rst new file mode 100644 index 00000000..11052988 --- /dev/null +++ b/docs/explain/index.rst @@ -0,0 +1,16 @@ +Big ideas +========= +The conceptual core of ``tractor``: what *structured +concurrency* (SC) means once your "tasks" are whole +processes, and how the runtime is layered to deliver +that without (much) magic. + +If you only read one page in these docs make it +:doc:`sc-distributed`; if you read two, follow it with +:doc:`architecture`. + +.. toctree:: + :maxdepth: 2 + + sc-distributed + architecture diff --git a/docs/explain/sc-distributed.rst b/docs/explain/sc-distributed.rst new file mode 100644 index 00000000..86ee1c0d --- /dev/null +++ b/docs/explain/sc-distributed.rst @@ -0,0 +1,207 @@ +Structured concurrency, across processes +========================================= +``tractor`` makes one bet: the discipline that made +``trio``'s concurrency *readable* — `structured +concurrency`_ (SC) — works just as well when the +"tasks" are whole OS processes talking over a wire. +This page distills what that means, from first +principles, with as little ceremony as possible. + +.. margin:: The canon + + If SC is new to you, the seminal `blog post`_ is + still the best hour you'll spend on concurrent + programming; the `trio docs`_, wikipedia's SC_ + page and the diagrams over at libdill-docs_ round + it out nicely. + +SC in one breath +---------------- +Structured concurrency is the rule that **concurrency +gets a scope**: every task is spawned *inside* a block +(a ``trio`` *nursery*) and that block **cannot exit +until every task it spawned has finished** — returned, +errored, or been cancelled. + +That one rule buys you the properties you already +rely on in sequential code, + +- a function call is a *black box*: when it returns, + everything it started is **done** — no secret + background tasks leaking out the sides, +- an exception **always has somewhere to go**: up the + (task) tree to a parent which is, by construction, + still there waiting, +- cancellation has a well defined *shape*: cancel a + scope and it flows down to every task inside it, + and only those. + +In short: your **runtime task tree matches your source +code's indentation**. Concurrency you can read. + +The leap: process-shaped tasks +------------------------------ +Now swap "task" for "process". + +A ``tractor`` *actor* is just a Python process running +its own ``trio.run()`` — its own private task tree, +sharing **nothing** with its siblings. You spawn +actors from an :class:`tractor.ActorNursery`, which +behaves exactly the way the name implies, + +.. code:: python + + async with tractor.open_nursery() as an: + portal = await an.start_actor( + 'worker', + enable_modules=[__name__], + ) + ... + # ^ block exit == every spawned process has + # completed, errored or been cancelled, and + # been **reaped**. No exceptions, no zombies. + +so the whole program becomes a *tree of process-trees* +— a `supervision tree`_ in erlang-speak — where every +arrow means "spawned by, **waited on by**, and +supervised by". + +.. d2:: diagrams/actor_tree.d2 + :caption: A ``tractor`` program: a process tree of + ``trio`` task trees; every parent **must wait** + on its children. + :width: 85% + +Causality: no process outlives its parent +----------------------------------------- +The stdlib's ``multiprocessing`` (and most "job +queue" systems) treat child processes as +fire-and-forget by default: orphans, zombies, lost +tracebacks and ``kill -9`` cleanup scripts are *your* +problem. ``tractor`` instead inherits ``trio``'s +`causality`_ discipline, + +- **no spawning willy-nilly**: every actor is born + from a nursery block with a known parent, +- **lifetimes nest**: a sub-actor's entire process + tree lives strictly inside its parent's nursery + scope, +- **teardown is guaranteed**: when a scope exits (or + errors, or is cancelled) the runtime SIGINTs, + waits, and (only if it must) hard-kills + reaps + everything underneath. + +We take the zombie thing personally: *if you can +create orphaned child processes without using a +system signal, it* **is a bug** — and there's a test +suite to back that sentence up. + +Errors always propagate (yes, across the wire) +---------------------------------------------- +In ``trio``, an exception in any task tears through +its nursery to a parent that must handle it — +`exceptions always propagate`_. ``tractor`` extends +the same guarantee across process boundaries: an +uncaught error in a remote task is + +1. captured + serialized in the child, +2. shipped home over IPC as a typed ``Error`` msg, +3. re-raised in the parent **boxed** as a + :class:`tractor.RemoteActorError` carrying the + original type (``.boxed_type``), a rendered remote + traceback, and the erroring actor's id, + +while the supervising nursery applies its (currently +*one-cancels-all*, just like ``trio``) strategy to any +sibling actors. A crash three processes deep arrives +at your shell as one coherent, causal traceback chain +— not a silent dead worker and a stuck queue. + +Cancellation is a request, supervision is the rule +-------------------------------------------------- +Cancellation likewise keeps ``trio``'s semantics +*verbatim*, just transported: cancelling an actor +nursery (or a single :class:`tractor.Context` between +two tasks in different processes) sends an explicit +cancel **request** over IPC which the remote runtime +translates into a real ``trio`` cancel-scope cancel — +then *acks back* so the requester can await +confirmation within a bounded time. Nothing is ever +"just killed" first; graceful always precedes brutal. + +Because every cross-process dialog is a pair of +**linked tasks** — one on each side, each inside its +own cancel scope — SC stays *transitive*: supervision +doesn't stop at the process boundary, it tunnels +through every hop of the tree. The wire protocol that +enforces this (a small set of typed msgs: +``Start``/``Started``/``Yield``/``Stop``/``Return``/ +``Error``) is detailed in :doc:`/guide/msging` and +:doc:`/guide/context`. + +Hold up, is this an "actor model"? +---------------------------------- +Let's stop and ask how many canon actor model papers +you've actually read ;) + +From `the author's mouth`_, the **only** requirement +is `adherence to`_ the `3 axioms`_:: + + In response to a message, an actor may: + + - send a finite number of new messages + - create a finite number of new actors + - designate a new behavior to process subsequent + messages + +``tractor`` adheres — actors exchange msgs, spawn +actors, and swap behaviors — **with no extra API** to +learn. What we *don't* copy is the cultural baggage: +no visible mailboxes, no untyped fire-and-forget +``send()``, no "let it crash" without a supervisor +that actually hears about it, and definitely no +shared-reference *proxy objects* pretending the +network isn't there. If our "actors" don't look like +what you expected, that's **intentional**: being an +actor model is just one property of the system; being +*structured* is the point. + +Why processes at all? +--------------------- +Python has a GIL; an actor model by definition shares +no state; so the *process* is the natural runtime +unit — you get real multi-core parallelism and hard +memory isolation for free. But the deeper win is +uniformity: because actors only ever talk via msgs +over a :class:`tractor.Channel` (TCP, UDS, more to +come), the **same code** runs your laptop's worker +pool and a multi-host cluster; "distributed" is a +deployment detail, not an API. + +It's just ``trio`` +------------------ +If you remember one framing, make it this: ``tractor`` +**is just** ``trio`` — with nurseries that can spawn +processes and streams that can cross them. Same +nursery discipline, same cancellation semantics, same +"how was this not always the API?" feeling, one level +up the process tree. + +.. seealso:: + + :doc:`/explain/architecture` for how the runtime + layers deliver all of the above, and + :doc:`/start/quickstart` to feel it in ~20 lines of + code. + +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html +.. _supervision tree: https://www.erlang.org/doc/design_principles/des_princ.html +.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker +.. _exceptions always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate +.. _the author's mouth: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s +.. _adherence to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s +.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts diff --git a/docs/guide/asyncio.rst b/docs/guide/asyncio.rst new file mode 100644 index 00000000..27e1d05f --- /dev/null +++ b/docs/guide/asyncio.rst @@ -0,0 +1,309 @@ +Infected ``asyncio`` +==================== +``tractor`` is "just trio_", but the Python world is packed with +libraries that only speak ``asyncio``: websocket stacks, vendor +SDKs, that one exchange client you can't route around. Rather than +make you rewrite them, ``tractor`` lets you *quarantine* them inside +a dedicated subactor which runs both event loops at once, with full +`structured concurrency`_ (SC) guarantees maintained across the +loop boundary *and* the process tree. + +In the project's own words: + + Yes, we spawn a python process, run ``asyncio``, start ``trio`` + on the ``asyncio`` loop, then send commands to the ``trio`` + scheduled tasks to tell ``asyncio`` tasks what to do XD + +We call this "infected ``asyncio``" mode: the subactor's stdlib +loop runs as the *host* with ``trio`` embedded on top in `guest`_ +mode, and your ``trio`` tasks drive ``asyncio`` tasks through +a linked, SC-supervised, in-memory channel. + +.. d2:: diagrams/infected_aio.d2 + :caption: One process, two schedulers: ``trio`` rides the + ``asyncio`` loop as a guest while the parent speaks plain + ``tractor`` IPC, none the wiser. + :alt: parent actor connected over IPC to a subactor whose + asyncio loop hosts trio in guest mode, with a + LinkedTaskChannel pairing a trio task to an asyncio task + +.. note:: + + Infected ``asyncio`` mode is **experimental**: it works (we + beat on it plenty) but parts of the API surface and some + edge-case semantics are still settling. Got opinions on the + interop design? Feel free to sling them in `#273`_! + +How the infection takes hold +---------------------------- +A normal subactor boots by running the ``tractor`` runtime's task +tree directly under ``trio.run()``. Pass ``infect_asyncio=True`` +at spawn time and the child's entrypoint changes shape entirely: + +1. the process starts the stdlib loop via ``asyncio.run()``, +2. the first ``asyncio`` task calls + ``trio.lowlevel.start_guest_run()``, embedding the ``trio`` + scheduler *inside* the already running ``asyncio`` loop (the + upstream `guest`_-mode feature), +3. the regular ``tractor`` runtime then boots on the guest + ``trio`` side and connects back to its parent like any other + subactor. + +.. margin:: Symptoms + + Looks like your stdlib event loop has caught a case of "the + trios"! Don't worry, you'll barely notice; and if anything + gets too bad, your parents will know about it B) + +Both schedulers interleave in a single thread, no GIL gymnastics +required. From the rest of the actor tree the infected child is +indistinguishable from any other actor: same IPC protocol, same +supervision and cancellation semantics, same zombie-safety +guarantees. The difference is purely internal: ``trio`` tasks in +that process can start and drive ``asyncio`` tasks through the +``tractor.to_asyncio`` API. + +Spawning an infected subactor +----------------------------- +Just flip the flag on :meth:`tractor.ActorNursery.start_actor`: + +.. code:: python + + async with tractor.open_nursery() as an: + portal = await an.start_actor( + 'aio_side', + enable_modules=[__name__], + infect_asyncio=True, + ) + +The one-shot convenience ``ActorNursery.run_in_actor()`` accepts +the same flag. The ``to_asyncio`` APIs may **only** be called from +tasks inside an infected actor; calling them anywhere else raises +a loud ``RuntimeError``. You can introspect at runtime with +``tractor.current_actor().is_infected_aio()``. + +Linking tasks with ``open_channel_from()`` +------------------------------------------ +The core primitive is :func:`tractor.to_asyncio.open_channel_from`, +an async context manager which starts your ``asyncio`` function as +a real ``asyncio.Task`` and yields a two-way channel linking it to +the calling ``trio`` task: + +.. code:: python + + from tractor import to_asyncio + + async with to_asyncio.open_channel_from( + aio_main, # async def aio_main(chan, **kwargs) + period=0.5, # extra kwargs are passed through + ) as (chan, first): + await chan.send('tick') + +The semantics deliberately mirror the inter-actor ``Context`` +handshake from :doc:`/guide/context`: + +- the target fn must declare a parameter literally named ``chan``; + the runtime injects the shared + :class:`~tractor.to_asyncio.LinkedTaskChannel` by keyword. +- the ``trio`` side blocks at entry until the ``asyncio`` task + calls ``chan.started_nowait(value)``; that value is delivered as + ``first``, exactly like the ``(ctx, first)`` pair you get from + ``Portal.open_context()`` after the child calls + ``ctx.started()``. +- a first value **must** be sent from the ``asyncio`` side or the + ``trio`` side will never unblock. +- on block exit the pair is torn down *together*; neither task can + outlive the other (more on this below). + +A full example: the echo server +------------------------------- +Here's the canonical demo, a round-trip echo service where the +``asyncio`` task is told what to do by a ``trio`` task which is in +turn driven over IPC by the root actor: + +.. literalinclude:: ../../examples/infected_asyncio_echo_server.py + :caption: examples/infected_asyncio_echo_server.py + :language: python + +What's going on? + +- there are three task layers: the root actor's pure ``trio`` + task, the infected child's ``trio``-side ``@tractor.context`` + endpoint (``trio_to_aio_echo_server()``), and the child's + ``asyncio`` task (``aio_echo_server()``). +- two ``started``-style handshakes compose: the aio task's + ``chan.started_nowait('start')`` unblocks the child's + ``open_channel_from()`` entry, then the child relays that same + value up via ``await ctx.started(first)`` which unblocks the + root's ``open_context()`` entry. Synchronization all the way + down, er, up. +- each round trip flows: root ``stream.send()`` -> IPC -> child + ``async for msg in stream`` -> ``chan.send(msg)`` -> aio + ``await chan.get()`` -> ``chan.send_nowait()`` -> child + ``chan.receive()`` -> ``stream.send(out)`` -> IPC -> root. +- when the root breaks out of its stream loop and exits the + context block, the child's stream ends, its channel block exits, + and the ``asyncio`` task is reaped along with it; the final + ``portal.cancel_actor()`` then tears down the whole process. No + orphaned ``asyncio`` tasks, no zombie procs; if you manage to + create either it **is a bug**. + +``LinkedTaskChannel``: one channel, two sides +--------------------------------------------- +The same channel object is shared by both tasks; which methods you +call depends on which loop schedules your task. The ``trio`` side +gets a standard ``trio.abc.Channel`` interface while the +``asyncio`` side gets queue-flavored, mostly-sync methods: + +.. list-table:: + :header-rows: 1 + :widths: 14 36 50 + + * - side + - call + - what it does + * - ``trio`` + - ``await chan.send(item)`` + - ship ``item`` to the ``asyncio`` task (enqueues onto an + internal ``asyncio.Queue``). + * - ``trio`` + - ``await chan.receive()`` + - wait for the next value from the ``asyncio`` side; the + channel also supports ``async for``. + * - ``trio`` + - ``await chan.wait_for_result()`` + - block until the ``asyncio`` task completes; return its + final result or raise its (translated) error. + * - ``trio`` + - ``chan.subscribe()`` + - acm yielding a ``BroadcastReceiver`` so N local tasks can + each consume a copy of the inbound stream (see below). + * - ``trio`` + - ``chan.cancel_asyncio_task()`` + - explicitly request cancellation of the linked ``asyncio`` + task. + * - ``asyncio`` + - ``chan.started_nowait(value)`` + - deliver the "first" value; unblocks the ``trio`` side's + ``open_channel_from()`` entry (mirrors ``ctx.started()``). + * - ``asyncio`` + - ``await chan.get()`` + - wait for the next value sent from the ``trio`` side. + * - ``asyncio`` + - ``chan.send_nowait(item)`` + - push a value to the ``trio`` side without blocking. + +Fan-out with ``.subscribe()`` +***************************** +Just like :meth:`tractor.MsgStream.subscribe` does for IPC +streams, ``chan.subscribe()`` lets multiple local ``trio`` tasks +each receive *every* value sent from the single ``asyncio`` task: + +.. code:: python + + async with chan.subscribe() as bcast: + async for msg in bcast: + ... + +The underlying broadcast machinery is lazily allocated on first +use and is *not* reversible for the channel's remaining lifetime, +so only reach for it when you actually want the fan-out. + +One-shot calls with ``run_task()`` +---------------------------------- +When you just want a single ``asyncio`` result and no streaming +dialog, skip the channel ceremony and use +:func:`tractor.to_asyncio.run_task`: + +.. code:: python + + import asyncio + from tractor import to_asyncio + + async def aio_fetch(url: str) -> str: + await asyncio.sleep(0.3) # pretend-IO, aio style + return f'sup {url}' + + # from any trio task inside the infected actor: + page = await to_asyncio.run_task(aio_fetch, url='https://x.io') + +It schedules the fn as an ``asyncio.Task``, waits for completion +and hands the return value back to ``trio``; think of it as the +cross-loop sibling of ``ActorNursery.run_in_actor()``. Errors and +cancellation are translated exactly as for channels. + +Cross-loop errors and cancellation +---------------------------------- +The paired tasks are *SC linked*: exception and cancel handling +tears down **both** sides on any unexpected error or cancellation, +in either loop. There is no fire-and-forget mode; a +``LinkedTaskChannel`` is a supervision scope just like a +``Context`` is across processes. + +Because each loop has its own (incompatible) cancellation and exit +machinery, boundary crossings are translated into dedicated +exception types, all importable from ``tractor.to_asyncio``: + +.. list-table:: + :header-rows: 1 + :widths: 26 22 52 + + * - exception + - raised in + - meaning + * - ``AsyncioCancelled`` + - the ``trio`` task + - the linked ``asyncio`` task was cancelled by itself or + a 3rd party (i.e. *not* by the ``trio`` side). + * - ``AsyncioTaskExited`` + - the ``trio`` task + - the ``asyncio`` task returned/exited early while the + ``trio`` side still held the link open. + * - ``TrioCancelled`` + - the ``asyncio`` task + - the ``trio`` side was cancelled (or crashed) so the + ``asyncio`` task is being torn down per SC rules. + * - ``TrioTaskExited`` + - the ``asyncio`` task + - the ``trio`` side exited gracefully while the ``asyncio`` + task was still running; a "clean shutdown" signal much + like closing a ``trio`` mem-chan. + +By default ``open_channel_from(suppress_graceful_exits=True)`` +absorbs the two ``*TaskExited`` signals so happy-path teardown +stays silent; pass ``False`` when your app wants to handle early +peer-exit explicitly. + +Past the task pair, everything composes with the normal actor +story: an unhandled ``asyncio`` error is translated into the +``trio`` side, propagates out of your ``@tractor.context`` +endpoint, and arrives at the parent boxed as +a :class:`tractor.RemoteActorError`. One SC discipline, +end-to-end, across loops *and* processes. + +Breakpoints in ``asyncio`` tasks +-------------------------------- +Yes, the multi-actor REPL works here too. With +``debug_mode=True`` enabled on your tree the ``trio`` side of an +infected actor can ``await tractor.pause()`` as usual, and with +greenback enabled (``maybe_enable_greenback=True``) even the +builtin ``breakpoint()`` works from *inside* ``asyncio`` tasks; +see ``examples/debugging/asyncio_bp.py`` for the full tour. The +root-TTY locking dance behind all this is covered in +:doc:`/guide/debugging`. + +Where to next? +-------------- +.. seealso:: + + - :doc:`/guide/context` for the inter-actor handshake and + streaming APIs which this whole interop layer mirrors. + - :doc:`/guide/msging` for typing the payloads you shuttle + between actors (and loops). + - :doc:`/guide/debugging` for the multi-process REPL that + keeps working even when your loop has "the trios". + +.. _trio: https://github.com/python-trio/trio +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops +.. _#273: https://github.com/goodboy/tractor/issues/273 diff --git a/docs/guide/cancellation.rst b/docs/guide/cancellation.rst new file mode 100644 index 00000000..56f0d6d1 --- /dev/null +++ b/docs/guide/cancellation.rst @@ -0,0 +1,285 @@ +Cancellation and error propagation +================================== + +``tractor`` supports ``trio``'s cancellation_ system *verbatim*, +then extends it across process boundaries. If you know how to +cancel a task in ``trio`` you already know how to cancel an actor — +and its whole subtree — in ``tractor``; the runtime's job is making +that statement hold over IPC with every structured concurrency (SC) +guarantee intact. + +The ground rules, + +- a remote actor is **never** cancelled unless explicitly requested + (by a parent or peer), unless supervision demands it (an error + triggered one-cancels-all teardown), or unless there's a bug in + ``tractor`` itself (please report it!), +- (remote) errors `always propagate`_ back to the parent + supervisor; nothing is silently dropped on the floor, +- every spawned process gets reaped no matter how it dies; if you + can create a zombie child process (without using a system signal) + it **is a bug**. + +``trio`` cancellation, across the wire +-------------------------------------- + +Locally everything is bog-standard ``trio``: nurseries, cancel +scopes, timeouts. ``tractor`` adds exactly one twist: a cancel +scope can't physically reach into another process, so the runtime +*relays cancellation as messages*. Concretely, + +- cancelling an *actor* means sending it a runtime-cancel request + msg; the target then runs its own graceful teardown — cancelling + RPC tasks, closing channels, exiting its :func:`trio.run` — and + acks the request back to the canceller, +- cancelling a single *cross-actor task* works through the + :class:`tractor.Context` layer: each ``ctx`` task-pair is + cancel-scope-linked over IPC such that either side erroring or + cancelling relays an equivalent error to the other side (see + :doc:`/guide/context` for the gory details), +- a cancel is therefore always a *request with an ack*: the + canceller does a **bounded wait** for confirmation and escalates + if the peer is unresponsive (see the teardown ladder below). + +One-cancels-all supervision +--------------------------- + +An :class:`tractor.ActorNursery` supervises subactors `exactly like +trio`_ nurseries supervise tasks: when one child errors, the error +propagates to the supervising block and **all** sibling subactors +get cancelled before the error continues bubbling up the (process) +tree. + +.. d2:: diagrams/error_propagation.d2 + :caption: One-cancels-all: no zombies, no lost errors. + :alt: error propagation up a subactor tree + :width: 80% + +.. literalinclude:: ../../examples/remote_error_propagation.py + :caption: examples/remote_error_propagation.py + :language: python + +What's going on here? + +- three healthy actors are spawned as daemons via + :meth:`tractor.ActorNursery.start_actor`; left alone they'd + happily idle forever, +- a fourth actor runs ``assert_err()`` via ``.run_in_actor()`` and + promptly trips its ``assert 0``, +- the resulting ``AssertionError`` ships back over IPC as a + serialized error msg and re-raises *boxed* inside the nursery + block as a :class:`tractor.RemoteActorError`, +- the nursery reacts like any ``trio`` nursery would: it cancels + the three healthy siblings (graceful runtime-cancel requests, + acks awaited), reaps all four processes, then re-raises, +- ``trio.run(main)`` sees that same ``RemoteActorError`` in the + parent-most process — propagation is end-to-end or bust. + +This one-cancels-all style is currently the *only* supervision +strategy offered (it's the one ``trio`` gives you); more +`erlang strategies`_ are roadmap, see the bottom of this page. + +The boxed-error bestiary +------------------------ + +All remote failures arrive locally as one of a small set of +exception types, each carrying enough metadata to work out *who* +failed, *where*, and *why*. + +``RemoteActorError`` +******************** + +The workhorse: a "boxed" exception relayed over IPC from another +actor. The original error's type, traceback string and msgdata are +preserved so you can pattern-match on what actually went wrong +remotely, + +- ``.boxed_type``: the reconstructed **type** of the original + remote exception (``ValueError``, ``NameError``, what have you), +- ``.src_uid``: the ``(name, uuid)`` pair of the actor where the + error *originated*, +- ``.relay_uid`` / ``.relay_path``: when an error crosses more than + one actor boundary (grandchild -> child -> root) every relaying + actor is recorded; multi-hop boxings are lovingly referred to as + "inceptions" in the runtime internals, +- ``.pformat()``: a rich "tb box" rendering of the remote traceback + for your logs or REPL. + +.. code:: python + + try: + async with portal.open_context(ep) as (ctx, first): + ... + except tractor.RemoteActorError as rae: + if rae.boxed_type is ValueError: + ... # the remote task raised `ValueError` + +``ContextCancelled`` +******************** + +The cancel-ack for a cross-actor task pair: raised when a +:class:`tractor.Context` task is cancelled *by request*. Its +``.canceller`` attr is the uid of the actor which **requested** the +cancel, which powers the key rule, + +- if **you** requested it (you called + :meth:`tractor.Context.cancel`) the resulting ctxc is *absorbed* + at ``open_context()`` exit: an expected outcome, not an error, +- if **anyone else** did — the peer task, or some third-party actor + — it *raises* locally so your code always hears about it. + +The full self- vs. cross-cancel semantics are a core teaching point +of :doc:`/guide/context`; go read them there. + +``MsgTypeError`` +**************** + +An IPC-payload "type error": a msg violated the dialog's declared +payload spec. See :doc:`/guide/msging` for the typed-messaging +system which enforces it. + +``TransportClosed`` +******************* + +The underlying IPC transport (TCP stream, UDS socket, ...) died or +closed out from under a channel. You'll normally only see this +surface when a peer hard-exits without any graceful runtime +teardown; the supervision machinery treats unexpected transport +loss on a busy channel as a failure and tears down accordingly. + +Pick your blast radius +---------------------- + +Three cancel surfaces, three scopes of effect; choose the smallest +hammer that does the job. + +.. list-table:: + :header-rows: 1 + :widths: 36 34 30 + + * - surface + - cancels + - typical use + * - :meth:`tractor.ActorNursery.cancel` + - every subactor in the nursery + - whole-tree teardown + * - :meth:`tractor.Portal.cancel_actor` + - one actor: full runtime + proc + - daemon teardown + * - :meth:`tractor.Context.cancel` + - exactly one remote task + - surgical task cancel + +``ActorNursery.cancel()`` +************************* + +The big red button: gracefully cancel every subactor supervised by +the nursery, in parallel, with the escalation discipline below +applied per-child. It's invoked for you whenever an error hits the +nursery block (one-cancels-all); call it yourself for an orderly +early shutdown. Passing ``hard_kill=True`` skips the graceful phase +and goes straight to OS-level process termination — rarely what you +want outside tests. + +``Portal.cancel_actor()`` +************************* + +Cancel one **whole actor**: its entire runtime, every task it's +scheduled, and (for subactors) the OS process, via a graceful +runtime-cancel request, + +.. code:: python + + await portal.cancel_actor() # bounded wait, bool result + await portal.cancel_actor( + raise_on_timeout=True, # no ack in time? + ) # -> `ActorTooSlowError` + +The wait for the peer's ack is *bounded* (default +``Portal.cancel_timeout = 0.5`` seconds, tunable per call via +``timeout=``). By default a missed ack just returns ``False``; with +``raise_on_timeout=True`` you instead get an ``ActorTooSlowError`` +(from ``tractor._exceptions``) so *your* code can escalate per SC +discipline — exactly what the nursery's own teardown does +internally before resorting to OS-level signalling. + +Note the granularity: this cancels an **actor**, not a task. For +one remote task use the ``Context`` layer instead. + +``Context.cancel()`` +******************** + +Request cancellation of exactly one remote task: the peer task of +an open :class:`tractor.Context`. Two things to keep straight, + +- it cancels the **remote** side only; a ``Context`` is *not* a + :class:`trio.CancelScope` and your local task keeps running until + you exit the ``open_context()`` block, +- the resulting :class:`tractor.ContextCancelled` is absorbed + locally (you asked for it, after all) per the self- vs. + cross-cancel rule above. + +Again, :doc:`/guide/context` covers this dance in depth. + +Graceful first, hard as a last resort +------------------------------------- + +.. margin:: REPL-safe by design + + The hard-kill path is *skipped* whenever an actor in the tree + holds the debug-REPL lock (``debug_mode=True`` flavors): + SIGTERM raining down on a tree mid-``pdb`` session would + clobber your prompt. See :doc:`/guide/debugging`. + +Every process teardown in ``tractor`` walks the same escalation +ladder, top rung first, + +1. **graceful cancel request**: a runtime-cancel msg over IPC; the + target actor cancels its tasks, closes its channels and exits + its :func:`trio.run` cleanly, +2. **soft wait**: the parent waits (bounded) for the child process + to exit on its own, +3. **SIGTERM**: no ack within the bounded wait (internally an + ``ActorTooSlowError``) escalates to ``proc.terminate()``, +4. **SIGKILL ultimatum**: still alive after the hard-kill timeout + (~1.6s)? The runtime logs that the "T-800" has been deployed to + collect the zombie and issues ``proc.kill()``. No survivors. + +The result is the **no-zombies guarantee**: ``tractor`` tries to +protect you from zombies, no matter what. Quoting the project +manifesto, + + If you can create zombie child processes (without using + a system signal) it **is a bug**. + +Run the quickstart's self-destructing process-tree demo +(``examples/parallelism/we_are_processes.py``, walked through in +:doc:`/start/quickstart`) under a ``pstree`` watcher and try to +catch a +straggler; we'll wait B) + +Roadmap: ``erlang``-style strategies +------------------------------------ + +One-cancels-all is ``trio``'s strategy and, for now, the only one +``tractor`` ships. Pluggable `erlang strategies`_ — one-for-one +restarts, rest-for-one, transient/permanent child specs and friends +(see the `supervision strategies`_ canon) — are a long-standing +roadmap item tracked in `#22`_. If supervisors are your jam that +issue is the place to sling opinions. + +.. seealso:: + + - :doc:`/guide/context` — the cross-actor task layer where + per-task cancellation actually lives, + - :doc:`/guide/msging` — the typed msg layer that raises + :class:`tractor.MsgTypeError`, + - :doc:`/guide/debugging` — what cancellation does (and very + carefully does *not* do) while a REPL is up. + +.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts +.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics +.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate +.. _erlang strategies: https://learnyousomeerlang.com/supervisors +.. _supervision strategies: https://www.erlang.org/doc/system/sup_princ.html +.. _#22: https://github.com/goodboy/tractor/issues/22 diff --git a/docs/guide/clustering.rst b/docs/guide/clustering.rst new file mode 100644 index 00000000..bc56b1af --- /dev/null +++ b/docs/guide/clustering.rst @@ -0,0 +1,127 @@ +Higher-level cluster APIs +========================= + +Sometimes you don't want a hand-crafted supervision tree; you want +"a pile of workers, one per core, now please". For that there's +:func:`tractor.open_actor_cluster`: a convenience wrapper which +spawns a *flat* cluster of subactors and hands you back a portal to +each, + +.. code:: python + + @acm + async def open_actor_cluster( + modules: list[str], # RPC allowlist for workers + count: int = cpu_count(), # one per core by default + names: list[str]|None = None, # default: 'worker_{i}' + hard_kill: bool = False, # fwd to `an.cancel()` + **runtime_kwargs, # fwd to `open_root_actor()` + ) -> AsyncGenerator[dict[str, tractor.Portal], None]: + +A cluster in one block +---------------------- + +.. literalinclude:: ../../examples/quick_cluster.py + :caption: examples/quick_cluster.py + :language: python + +Walkthrough, + +- ``open_actor_cluster(modules=[__name__])`` concurrently spawns + one subactor per detected core (per + :func:`multiprocessing.cpu_count`); the ``modules`` list is the + usual ``enable_modules``-style capability allowlist so workers + may run functions defined in this module, +- it yields a ``dict[str, tractor.Portal]`` mapping worker name to + portal; note the keys get prefixed with the *spawning* actor's + name, so from the root you'll see ``'root.worker_0'``, + ``'root.worker_1'``, etc., +- a plain :class:`trio.Nursery` then fans out one + ``portal.run(sleepy_jane)`` per worker; each prints its actor + ``.uid`` from inside its own process then naps forever — what + runs *inside* each worker (and how many tasks you point at it) + is entirely yours to compose, +- ``tractor.trionics.collapse_eg()`` un-nests the strict + ``ExceptionGroup`` wrapping so the demo's ``KeyboardInterrupt`` + surfaces as itself instead of arriving eg-boxed, +- on block exit the whole fleet is torn down for you via + :meth:`tractor.ActorNursery.cancel`; pass ``hard_kill=True`` at + open time to skip straight to OS-level termination instead of + the graceful ladder described in :doc:`/guide/cancellation`. + +Sizing, naming, fleet-wide options +---------------------------------- + +``count`` doesn't have to be core-count and the auto-generated +``'worker_{i}'`` names are just the default; pass your own (the +length must match ``count`` or you get a ``ValueError``). Any +extra ``**runtime_kwargs`` pass through verbatim to +:func:`tractor.open_root_actor`, so fleet-wide runtime options are +one kwarg away, + +.. code:: python + + async with tractor.open_actor_cluster( + modules=['mylib.workers'], + count=4, + names=['scout', 'miner', 'smelter', 'smith'], + debug_mode=True, # whole-fleet crash-to-REPL + ) as portal_map: + ... + +From here the composition patterns are the usual ``tractor`` fare: +``portal.run()`` for one-shot calls (as in the demo), or — for a +persistent bidirectional dialog per worker — concurrently enter N +``portal.open_context()`` blocks with +``tractor.trionics.gather_contexts()``; see :doc:`/guide/context` +for that whole layer. + +Clusters vs. nurseries +---------------------- + +.. d2:: diagrams/actor_tree.d2 + :margin: + :caption: The general shape: arbitrary nesting. A cluster is + this, minus the nesting. + :alt: a nested supervision tree of subactors + +``open_actor_cluster()`` is sugar, not a new primitive: under the +hood it's just :func:`tractor.open_nursery` plus N concurrent +``start_actor()`` calls plus a ``.cancel()`` on the way out. Reach +for it when, + +- you want a *flat*, homogeneous fleet (classic worker-pool or + map-style fan-out shapes), +- "one per core" — or a fixed ``count`` — is the right sizing + story, +- every child can share the same spawn options. + +Drop down to a raw :class:`tractor.ActorNursery` when the topology +gets any fancier: nested trees, heterogeneous children, per-child +``debug_mode``/transport/module options, daemons mixed with +one-shot workers, and so on (see :doc:`/guide/parallelism` for a +hand-rolled pool). Either way the supervision semantics are +identical: one-cancels-all error propagation and the no-zombies +guarantee from :doc:`/guide/cancellation` apply to clusters too. + +Provisional, by design +---------------------- + +.. note:: + + APIs in this section are considered **provisional**: the + signature and semantics of :func:`tractor.open_actor_cluster` + may shift as higher-level supervision machinery lands. We + encourage you to try it and provide feedback — the + `matrix channel`_ is the place to say hi, and `#22`_ tracks the + broader supervisor-strategy roadmap. + +.. seealso:: + + - :doc:`/guide/parallelism` — worker pools built "by hand" with + plain actor nurseries (and why that's easy peasy), + - :doc:`/guide/cancellation` — the teardown machinery a cluster + inherits for free. + +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _#22: https://github.com/goodboy/tractor/issues/22 diff --git a/docs/guide/context.rst b/docs/guide/context.rst new file mode 100644 index 00000000..9ecab554 --- /dev/null +++ b/docs/guide/context.rst @@ -0,0 +1,344 @@ +The ``Context``: a cross-actor task pair +========================================= + +If you've written any trio_ you already know the contract: every +task lives in a nursery, errors always propagate, cancellation is +scoped, and nothing leaks. ``tractor`` extends that exact contract +*across processes* — the same guarantees from the seminal +`blog post`_, just with the nursery split across two memory +domains. The primitive that does it is :class:`tractor.Context`: a +**linked pair of tasks**, one in each of two actors, supervised as +a single `structured concurrency`_ (SC) scope over IPC. + +.. d2:: diagrams/context_handshake.d2 + :caption: The SC-transitive supervision protocol, msg by msg. + :alt: sequence diagram of the context handshake msg flow + +Pretty much everything else is (or is slated to be) built on this +one primitive: ``ActorNursery.run_in_actor()`` is a convenience +for "spawn, open a context, await the result, tear down"; plain +``Portal.run()`` RPC is planned to be re-implemented on top of it; +the multi-process debugger's tree-wide REPL lock rides one. Grok +this page and the rest of the library reads as convenience +wrappers B) + +The endpoint contract +--------------------- + +A context endpoint is an async function decorated with +:func:`tractor.context` which declares **a param annotated** +``tractor.Context`` — any param name you like, the annotation is +what's required: + +.. code:: python + + @tractor.context + async def trainer( + ctx: tractor.Context, + model: str, + ) -> str: + await ctx.started('ready') + return f'trained {model}' + +.. margin:: Who am I talking to? + + Inside any context task + :func:`tractor.current_ipc_ctx` returns the + ``Context`` bound to the current task; handy + in helpers that don't take ``ctx`` explicitly. + +The parent (aka "opener") side invokes it through a +:class:`tractor.Portal` using ``Portal.open_context()``, passing +any extra kwargs which are shipped over the wire as the remote +task's arguments. Since the target fn is referenced by module +path, that module must be listed in the peer actor's +``enable_modules`` allowlist — RPC capability is always opt-in. + +The decorator also accepts a ``pld_spec``: a type (union) which +every payload in the dialog is validated against, upgrading your +msgs to a typed contract enforced via :exc:`tractor.MsgTypeError`. +Validation strictness follows the "`cheap or nasty`_" +`(un)protocol`_ pattern: the one-shot ``Started`` payload gets the +nasty treatment (stringently round-trip checked before it's even +sent) while high-rate stream payloads stay cheap (checked only +receiver side). + +The handshake, on the wire +-------------------------- + +Every context runs one instance of ``tractor``'s "SC-transitive +supervision protocol": a tiny fixed grammar of msgspec_-typed msgs +encapsulating *all* RPC dialogs between actors. *Transitive* +because each IPC link obeys the same rules a local nursery does — +starts are acked, completion is awaited, errors and cancels always +relay — so chaining links across a process tree composes into one +tree-wide SC scope. + +The figure up top shows a full dialog; in order: + +``Start`` + sent by ``Portal.open_context()``: "schedule a task running + this function with these kwargs". + +``StartAck`` + the peer runtime confirms the task is scheduled and that the + endpoint really is a context-style fn. + +``Started`` + emitted when the child task calls + :meth:`tractor.Context.started`; carries the first payload + and unblocks the parent's entry of ``open_context()``. + +``Yield`` + one per :meth:`tractor.MsgStream.send`, flowing in *either* + direction while a stream is open. + +``Stop`` + graceful end-of-stream: the far side's ``async for`` + terminates cleanly. + +``Return`` + the child fn returned; its value becomes the context's final + result. If the child raised instead, an ``Error`` msg takes + this slot carrying the boxed traceback. + +``ctx.started()``: just like ``task_status.started()`` +******************************************************* + +The startup phase is a deliberate clone of +:meth:`trio.Nursery.start` semantics: the child decides when it's +"up", optionally handing back a first value, and the parent stays +blocked until that moment: + +.. code:: python + + # trio, in-process + first = await nursery.start(child_fn) + + # tractor, cross-process + async with portal.open_context(child_fn) as (ctx, first): + ... + +The ``as (ctx, first)`` tuple is exactly that pair: the +:class:`tractor.Context` handle plus whatever value the child +passed to ``await ctx.started(value)``. And readiness is not +optional — for instance opening a stream before ``.started()`` +has been called raises a ``RuntimeError``; handshake first, then +dialog. + +Bidirectional streaming over a context +-------------------------------------- + +The canonical ping-pong (design history: `#53`_, `#223`_) — a +full-duplex msg stream between a parent and its spawned peer: + +.. literalinclude:: ../../examples/rpc_bidir_streaming.py + :caption: examples/rpc_bidir_streaming.py + :language: python + +What's going on? + +- ``start_actor()`` spawns the daemon-style subactor + ``'rpc_server'`` with this very module in its allowlist. + +- ``portal.open_context(simple_rpc, data=10)`` fires the + ``Start`` msg then blocks until the child task calls + ``await ctx.started(data + 1)`` — hence ``sent == 11``. + +- both tasks enter ``ctx.open_stream()``: a stream dialog is only + fully open once *each* side has entered its block. + +- the parent seeds the first ``'ping'``; each side then echoes + the other, one ``Yield`` msg per ``stream.send()``. + +- after the 9th pong the parent ``break``\ s (10 pings sent in + total) and exits its stream block, which sends ``Stop``; the + child's ``async for`` completes gracefully and its ``else`` + clause asserts all 10 pings arrived. + +- the 10th in-transit pong? Discarded by the implicit drain at + ``open_context()`` exit, which runs the dialog down to the + child's ``Return`` (here ``None``). + +- daemon actors live until told otherwise: + ``portal.cancel_actor()`` reaps the subactor explicitly. + +Results: the ``Return`` leg +--------------------------- + +Every context resolves to a final outcome. Wait on it explicitly +from the parent side: + +.. code:: python + + async with portal.open_context(ep) as (ctx, first): + ... + result = await ctx.wait_for_result() + +or just exit the block — ``__aexit__`` implicitly drains the msg +flow until the ``Return`` (or ``Error``) arrives, discarding any +in-transit ``Yield``\ s on the way. Either way the rule of +`causality`_ holds exactly as in a local nursery: **the opener +never unblocks before the remote task is done**. + +For post-hoc inspection (think supervision/restart logic) the ctx +also exposes ``Context.outcome``, ``.maybe_error`` and +``.has_outcome`` — where a "result" might well be the error the +dialog ended with. + +Cancellation semantics +---------------------- + +The part you actually came for; read it twice B) + +A context's two tasks are **cancel-scope-linked across the IPC +boundary**: whatever ends one side — error, cancellation, plain +old return — is relayed such that the other side ends +equivalently. No silent half-open dialogs, no orphaned remote +tasks, ever. + +``ctx.cancel()`` cancels the *remote* task +******************************************* + +:meth:`tractor.Context.cancel` requests cancellation of the +**remote** task only: + +.. code:: python + + async with portal.open_context(ep) as (ctx, first): + await accomplish_things(ctx) + await ctx.cancel() # remote task, NOT me + +A :class:`tractor.Context` is **not** a :class:`trio.CancelScope`: +the call doesn't (and can't) cancel your local task. It sends the +cancel request and waits a bounded ``timeout`` for the peer +runtime's ``CancelAck``, then your code proceeds to the block exit +as normal. + +Compare scopes here: ``Portal.cancel_actor()`` is the big hammer +which cancels the peer's **entire runtime** (and thus process); +``ctx.cancel()`` is the per-dialog scalpel. + +``ContextCancelled`` and the absorption rule +********************************************* + +When a context task gets cancelled *by request* the requestee's +runtime reports back with a :exc:`tractor.ContextCancelled` +("ctxc") whose ``.canceller`` field holds the uid of the actor +which asked. That one field decides what you observe: + +**you requested it** + i.e. ``ctxc.canceller == tractor.current_actor().uid``: the + ctxc is **absorbed** at ``open_context()`` exit — nothing + raises in your block. You asked for a graceful stop and got + it; if you care, ``await ctx.wait_for_result()`` hands the + ctxc back as a plain *value* for inspection. + +**anyone else requested it** + the peer cancelling itself, or some third actor cancelling it + from the side: the ctxc **is raised** in your block. From + your scope's perspective a task you depend on was killed out + from under you and SC demands you hear about it — exactly + like a sibling crash in a `nursery`_. + +In code: + +.. code:: python + + try: + async with portal.open_context(ep) as (ctx, first): + ... + except tractor.ContextCancelled as ctxc: + # can only be a peer- or third-party cancel; + # self-requested cancels are absorbed at exit. + assert ctxc.canceller != tractor.current_actor().uid + +This self- vs cross-cancel split is what makes explicit teardown +*composable*: a supervisor cancels its dialogs without try/except +noise, while unexpected cancellation anywhere in the tree still +propagates loudly like any other failure. + +.. warning:: + + Once ``ctx.cancel()`` has been called the dialog is done: a + subsequent ``ctx.open_stream()`` raises ``RuntimeError``. + +For introspection the ctx exposes trio-flavored status props: +``.cancel_called`` (this side requested), ``.cancel_acked`` (peer +confirmed), ``.cancelled_caught`` and ``.canceller`` — +deliberately mirroring :class:`trio.CancelScope` naming. + +Errors propagate, both ways +--------------------------- + +A crash on either end tears down the pair, SC style: + +- **child raises**: the exception ships back as an ``Error`` msg + and re-raises in the parent block boxed as a + :exc:`tractor.RemoteActorError`; the original class rides along + as ``.boxed_type`` with ``.src_uid`` naming the crashed actor. + +- **parent raises** (or is cancelled) inside the block: an + equivalent error/cancel is relayed to the child task so it can + never outlive the dialog. + +.. code:: python + + try: + async with portal.open_context(ep) as (ctx, first): + ... + except tractor.RemoteActorError as rae: + if rae.boxed_type is ValueError: + ... # remote ValueError, type preserved + +Errors that hop through intermediary actors on their way up the +tree ("inceptions" XD) keep the full relay trail in +``.relay_uid`` / ``.relay_path``. Payloads violating your declared +``pld_spec`` surface as the IPC analog of a ``TypeError``: +:exc:`tractor.MsgTypeError`. + +Overruns and backpressure +------------------------- + +Stream msgs land in a bounded per-context buffer on the receiver +side. A sender that outpaces a non-consuming receiver *overruns* +it and the runtime raises :exc:`tractor.StreamOverrun` (also a +:exc:`trio.TooSlowError`) instead of buffering without bound — SC +discipline applies to memory too. + +Your knobs: + +- ``msg_buffer_size`` on ``ctx.open_stream()`` sizes the buffer. + +- ``allow_overruns=True`` (on ``Portal.open_context()`` and/or + ``ctx.open_stream()``) opts in to absorbing overflow instead of + erroring — reasonable for bursty telemetry-ish feeds, just know + you're trading the error for extra buffering. + +One context, one stream +----------------------- + +A ``MsgStream`` is strictly **one-shot use**: once it closes — +gracefully or not, from either side — it can never be re-opened +on the same ctx. Want another round with the same peer? Open a +fresh context; they're cheap. The full close-vs-cancel teardown +story lives in :doc:`/guide/streaming`. + +.. rubric:: Where to next? + +:doc:`/guide/streaming` covers the rest of the msg-moving story: +the legacy one-way API, multi-actor pipelines and in-actor +broadcast fan-out. For exhaustive API detail see +:class:`tractor.Context`, :class:`tractor.MsgStream` and +:exc:`tractor.ContextCancelled`. + +.. _trio: https://github.com/python-trio/trio +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning +.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker +.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern +.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols +.. _msgspec: https://jcristharif.com/msgspec/ +.. _#53: https://github.com/goodboy/tractor/issues/53 +.. _#223: https://github.com/goodboy/tractor/issues/223 diff --git a/docs/guide/debugging.rst b/docs/guide/debugging.rst new file mode 100644 index 00000000..a4a7d3d4 --- /dev/null +++ b/docs/guide/debugging.rst @@ -0,0 +1,545 @@ +.. _debugging: + +================================ +"Native" multi-process debugging +================================ + +``tractor`` ships the thing every ``multiprocessing`` user has +wished for and quietly assumed was impossible: a multi-process +debugger that *just works*. + +Drop ``await tractor.pause()`` — or, with `greenback`_ installed, +a plain builtin ``breakpoint()`` — anywhere in any actor: the +root, a child, a grandchild, a sync helper function, even an +``asyncio`` task inside an "infected" actor. A full-featured +`pdbp`_ REPL opens *in that process*, with syntax-highlighted +source listings, tab completion and sticky mode, attached to your +one terminal. + +Under the hood every REPL entry acquires a tree-global tty mutex +via an IPC request to the root actor, so prompts from concurrent +pauses and crashes never interleave. ``ctrl-c`` is shielded while +any REPL is live, so a stray ``SIGINT`` can't vaporize the tree +out from under you. And in debug mode any uncaught error drops +you into a crash REPL *first in the failing child*, then again at +each parent as the boxed :class:`~tractor.RemoteActorError` climbs +the supervision tree. + +No remote-pdb sockets, no ``set_trace()`` port juggling, no +``ptrace`` attach dance: the debugger semantics you already know, +transparently extended across an entire process tree. Because +``tractor`` is a `structured concurrency`_ (SC) runtime, the +debugger composes with supervision instead of fighting it — quit +a REPL and errors keep propagating exactly like `trio`_ taught +you, ending in clean, zombie-free teardown. + +We're pretty sure it's the (first ever?) "native" debugging UX +for multi-process Python B) + +Enabling debug mode +------------------- + +Pass ``debug_mode=True`` to your runtime entrypoint, either +:func:`tractor.open_nursery` (which forwards it to the implicitly +opened root actor) or :func:`tractor.open_root_actor` directly: + +.. code:: python + + async with tractor.open_nursery( + debug_mode=True, # arm the whole actor tree + ) as an: + ... + +This arms the debug machinery *tree-wide*: + +- crash handling is enabled in every actor: uncaught errors enter + a REPL before they propagate, +- the internal tty-lock module is auto-exposed over RPC to every + subactor (this is what makes the one-terminal handoff work), +- console logging is bumped to include ``PDB``-level status msgs + so you can see REPL acquire/release events as they happen. + +You can instead flip it on for just one child, letting its +siblings crash-and-burn the normal way: + +.. code:: python + + portal = await an.start_actor( + 'sketchy_worker', + debug_mode=True, # OR-ed with the tree-wide flag + ) + +See ``examples/debugging/per_actor_debug.py`` for a runnable +proof of the selective style. + +.. note:: + + Debug mode requires the child-side runtime to be + ``trio``-native so that the tty-lock IPC dialog works; it's + currently supported on the ``'trio'`` (default) and + ``'main_thread_forkserver'`` spawn backends and raises + ``RuntimeError`` for any other ``start_method``. + +Your first pause point +---------------------- + +:func:`tractor.pause` is the SC-aware, multi-process spelling of +the stdlib's ``breakpoint()``. In the root actor it looks almost +boring: + +.. literalinclude:: ../../examples/debugging/root_actor_breakpoint.py + :caption: examples/debugging/root_actor_breakpoint.py + :language: python + +Run it and you get a ``(Pdb+)`` prompt parked on the ``pause()`` +line; type ``c`` (continue) and the program finishes normally. + +The exact same call works from *any* subactor, no matter how deep +in the tree: + +.. literalinclude:: ../../examples/debugging/subactor_breakpoint.py + :caption: examples/debugging/subactor_breakpoint.py + :language: python + +Each loop iteration the child actor requests the terminal from +the root over IPC, REPLs you, then releases it on ``c``. Pause +points are re-entrant-safe: repeat calls from the same task are +no-op'd and other local tasks queue politely for the REPL. + +When you get bored, type ``q`` (quit): the resulting +``bdb.BdbQuit`` is boxed and shipped to the parent like any other +remote error XD — causality is preserved even for your debugging +mistakes. + +Crash REPLs: errors climb the tree +---------------------------------- + +Pause points are only half the story. With debug mode armed, any +*uncaught* error anywhere in the tree triggers what we call crash +handling mode: + +.. literalinclude:: ../../examples/debugging/subactor_error.py + :caption: examples/debugging/subactor_error.py + :language: python + +What happens when the child hits that (very intentional) +``NameError``: + +1. a REPL opens **in the crashed child first** — you inspect the + raising frame, its locals, the works, right inside the failed + process, +2. when you quit, the error is boxed into a + :class:`~tractor.RemoteActorError` and relayed to the parent, +3. the parent (here the root) gets *its own* crash REPL with the + rendered remote traceback, +4. quit again and the nursery tears the tree down — errors keep + propagating per SC rules, no zombies left behind. + +You debug the failure at every hop of the supervision tree, which +for multi-hop trees means you can chase an error from the leaf +that raised it all the way up to the root that supervises it. + +Need to skip REPL entry for certain exceptions? Pass a predicate +via ``open_root_actor(debug_filter=...)``; by default +cancellation-only exception (groups) don't engage the REPL. + +One terminal, many actors +------------------------- + +So how do N processes share one tty without garbling it? The root +actor owns stdio for the whole tree and guards it with a FIFO +mutex; every subactor REPL entry is an IPC lock request to the +root. Exactly one actor-task in the entire tree can own the +terminal at a time, so prompts never interleave — ever. + +.. d2:: diagrams/debug_lock.d2 + :caption: Every REPL entry serializes through the root actor's + tty lock; ``continue``-ing one REPL hands the terminal to + the next waiter, FIFO style. + :alt: sequence diagram of two subactors serializing pdb REPL + access through the root actor's tty lock + +The runtime's teardown paths cooperate too: a cancelling parent +always waits for any live REPL to release before reaping +children, so the debugger never gets yanked out from under you +mid-keystroke. + +.. margin:: Watch the tree live + + Run any of these examples with a process-tree watcher in a + second terminal and watch actors come and go:: + + watch -n 0.1 "pstree -a $$" + +Here's the showpiece: one daemon child re-entering +``tractor.pause()`` forever inside a stream, while its sibling +repeatedly raises a ``NameError``: + +.. literalinclude:: ../../examples/debugging/multi_daemon_subactors.py + :caption: examples/debugging/multi_daemon_subactors.py + :language: python + +What you'll actually see +************************ + +Running it looks *roughly* like this (uids, tracebacks and source +listings elided; REPL order can vary with who wins the lock +race):: + + $ python examples/debugging/multi_daemon_subactors.py + + Opening a pdb REPL in paused actor: ('bp_forever', '') + + (Pdb+) c + + Opening a pdb REPL in crashed actor: ('name_error', '') + + (Pdb+) q + + Opening a pdb REPL in crashed actor: ('root', '') + + (Pdb+) q + +Two (then three) processes, one terminal, zero confusion: +``c``-ing out of the paused daemon's REPL releases the tty lock, +which immediately hands the prompt to the crashed sibling; quit +that and the error propagates as a fully-rendered +:class:`~tractor.RemoteActorError` to the parent where one final +crash REPL catches it before clean, zombie-free teardown. + +For maximum drama run +``multi_nested_subactors_error_up_through_nurseries.py`` (under +``examples/debugging/``) which pulls the same trick across a +*three-deep* process tree — the tty lock keeps every prompt +orderly the whole way up. + +Post-mortem, on demand +---------------------- + +Crash handling is automatic, but you can also enter a REPL on +a live exception *manually* with :func:`tractor.post_mortem` — +the actor-aware equivalent of ``pdb.post_mortem()`` — from inside +any ``except`` block in any actor (kwargs: ``tb=`` for an +explicit traceback, plus ``shield=`` and ``hide_tb=``): + +.. literalinclude:: ../../examples/debugging/pm_in_subactor.py + :caption: examples/debugging/pm_in_subactor.py + :language: python + +This example demos three REPL entries from one error: + +- the child's manual ``post_mortem()`` inside its ``except``, +- the runtime's automatic crash handler in the same child once + the error re-raises out of the RPC task, +- a manual ``post_mortem()`` in the parent on the received + :class:`~tractor.RemoteActorError`, whose ``.boxed_type`` + faithfully reports the original ``NameError``. + +Pausing from sync code +---------------------- + +No ``await``? No problem. :func:`tractor.pause_from_sync` brings +the same tree-aware REPL to plain synchronous functions — handy +when the suspect code is three helpers deep and decidedly not +async. + +It's powered by `greenback`_, which is optional, so you need to: + +1. install it (it ships in ``tractor``'s ``sync_pause`` + dependency group), +2. enable it at runtime entry: + +.. code:: python + + async with tractor.open_nursery( + debug_mode=True, + maybe_enable_greenback=True, + ) as an: + ... + +With that armed, sync code can pause from three different caller +environments: the main ``trio`` thread, ``trio.to_thread`` bg +threads, and (see the next section) ``asyncio`` tasks in infected +actors. The greenback "portal" hops back into the ``trio`` loop +to do the lock/REPL dance on your behalf: + +.. literalinclude:: ../../examples/debugging/sync_bp.py + :caption: examples/debugging/sync_bp.py (the sync fn, excerpt) + :language: python + :pyobject: sync_pause + +.. literalinclude:: ../../examples/debugging/sync_bp.py + :caption: examples/debugging/sync_bp.py (called in a subactor, + excerpt) + :language: python + :pyobject: start_n_sync_pause + +The full script also exercises the hairier root-actor bg-thread +cases (and documents their remaining sharp edges) if you want the +deep lore. + +The builtin ``breakpoint()`` override +************************************* + +When debug mode boots with greenback available, ``tractor`` wires +Python's `PEP 553`_ hook so the *builtin* ``breakpoint()`` becomes +the actor-aware sync pause, by exporting:: + + PYTHONBREAKPOINT=tractor.devx.debug._sync_pause_from_builtin + +That means third-party and legacy code containing bare +``breakpoint()`` calls debugs correctly inside your actor tree +with zero edits (the override even forwards kwargs like +``hide_tb`` to the underlying pause machinery, as shown in the +excerpt above). + +.. warning:: + + Without greenback (or with ``maybe_enable_greenback=False``, + the default), ``debug_mode=True`` instead *blocks* the builtin + ``breakpoint()``: ``sys.breakpointhook`` is swapped for a + raiser and ``PYTHONBREAKPOINT=0`` is set. A naive + ``breakpoint()`` from some random process would clobber the + shared tty, so we'd rather hand you a loud ``RuntimeError`` + with install instructions. + +Both the hook and the env var are restored to their prior values +on runtime exit — see +``examples/debugging/restore_builtin_breakpoint.py`` for the +proof. + +Breakpoints inside ``asyncio`` tasks +------------------------------------ + +Yes, even "infected ``asyncio``" actors get the goods. Spawn a +child with ``infect_asyncio=True`` (``trio`` runs as a guest on +the ``asyncio`` loop inside it) and, with debug mode + greenback +armed, every ``asyncio`` task started via ``tractor.to_asyncio`` +is automatically granted a greenback portal — so a plain builtin +``breakpoint()`` (or ``tractor.pause_from_sync()``) inside an +``asyncio.Task`` joins the same single-terminal, tree-locked REPL +flow: + +.. literalinclude:: ../../examples/debugging/asyncio_bp.py + :caption: examples/debugging/asyncio_bp.py + :language: python + +Note the interleave: a ``breakpoint()`` on the ``asyncio`` side, +``tractor.pause()`` on the ``trio`` side of the same actor, and +another pause up in the root — all serialized through the one tty +lock with no cross-actor (or cross-event-loop!) clobbering. + +One catch: ``asyncio`` tasks spawned *out-of-band* — i.e. not via +``tractor.to_asyncio``, typically by some third-party aio lib — +have no portal bestowed, so a sync pause from one raises a loud +``RuntimeError`` telling you to ``greenback.ensure_portal()`` +first. See :ref:`the caveats ` below. + +Teardown debugging: the shielded pause +-------------------------------------- + +`Cancellation`_ is ``trio``'s bread and butter, which raises an +awkward question: how do you REPL inside an *already-cancelled* +scope, say while debugging some teardown sequence? A bare +``pause()`` would itself be cancelled at its next checkpoint. + +The answer is ``await tractor.pause(shield=True)``, which wraps +the lock acquisition and REPL session in a shielded cancel scope +(``post_mortem(shield=True)`` works the same way): + +.. literalinclude:: ../../examples/debugging/shielded_pause.py + :caption: examples/debugging/shielded_pause.py + :language: python + +If you forget, ``tractor`` has your back: an unshielded +``pause()`` from a cancelled scope fails fast with a hint +suggesting ``await tractor.pause(shield=True)`` instead of +silently never REPL-ing. + +Go ahead, mash ctrl-c +--------------------- + +While any REPL is live the runtime installs a custom ``SIGINT`` +handler tree-wide so that a reflexive ``ctrl-c`` (or five) can't +nuke your debug session: + +- the actor that owns the REPL ignores the interrupt and simply + re-flushes the prompt — keep mashing, it's fine, +- the root actor ignores ``SIGINT`` while a still-IPC-connected + child holds the tty lock, so the supervisor won't tear down the + tree out from under the debugger, +- if the lock state has gone *stale* — the locking child died or + its IPC channel dropped — the root cancels the stale lock scope + and restores ``trio``'s default handler, so ``ctrl-c`` works + again exactly when it should. + +The handler is uninstalled and ``trio``'s own ``SIGINT`` +semantics restored every time a REPL releases (on ``continue`` / +``quit``). + +Live task-tree dumps +-------------------- + +Sometimes there's no error to catch — the tree is just *hung* and +you want to know where. For that ``tractor`` integrates +`stackscope`_: send a signal, get a full ``trio`` task-tree dump +from every actor in the tree. + +Enable it any of three ways: + +- ``open_root_actor(enable_stack_on_sig=True)`` (or via + ``open_nursery()`` which forwards it), +- set ``TRACTOR_ENABLE_STACKSCOPE=1`` in the env — it's inherited + through the process tree so every (sub)actor arms the handler + at boot, +- call ``tractor.devx.enable_stack_on_sig()`` directly. + +It's intentionally *not* gated on ``debug_mode`` so you can leave +it armed in plain runs. Then, when the hang strikes, signal the +tree with ``SIGUSR1``. + +.. tip:: + + No need to hunt down pids — pattern-match the original cmdline + with ``pkill``:: + + $ pkill --signal SIGUSR1 -f "python example_script.py" + +Each actor dumps its entire ``trio`` task tree (full nursery +recursion via ``stackscope.extract()``) to its tty *and* tees it +to ``/tmp/tractor-stackscope-.log`` — so the trace survives +even under captured-stdio harnesses — then relays the signal on +to its children, parent-before-child, until the whole tree has +reported in. + +Try it yourself with the demo script, which deliberately hangs a +subactor in a shielded sleep: + +.. literalinclude:: ../../examples/debugging/shield_hang_in_sub.py + :caption: examples/debugging/shield_hang_in_sub.py + :language: python + +(That ``trio.CancelScope(shield=True)`` hang also shows off the +zombie reaper: ``ctrl-c`` the root and the un-cancellable child +still gets hard-reaped — if you can create a zombie it **is a +bug**.) + +Crash handling for sync and CLI code +------------------------------------ + +All of the above rides on the actor runtime, but crashes don't +politely wait for ``trio.run()``. For plain sync code — think +``typer``/``click`` CLI endpoints, config parsing, anything +pre-runtime — there's a sync context manager that wraps the same +``pdbp`` post-mortem UX: + +.. code:: python + + from tractor.devx import open_crash_handler + + def main(): # any sync code, no runtime required + with open_crash_handler() as boxed: + run_my_cli_thing() + +By default any ``BaseException`` (minus an ``ignore`` set +defaulting to ``KeyboardInterrupt`` and ``trio.Cancelled``) +enters the REPL then re-raises on exit; pass +``raise_on_exit=False`` to suppress instead and introspect the +``boxed.value`` afterward. The ``catch``/``ignore`` sets and a +``repl_fixture`` are all tweakable. + +For the classic ``--pdb`` CLI-flag pattern use the conditional +variant: + +.. code:: python + + from tractor.devx import maybe_open_crash_handler + + @app.command() # a `typer` (or `click`) endpoint + def cmd(pdb: bool = False): + with maybe_open_crash_handler(pdb=pdb): + ... + +REPL niceties and hooks +----------------------- + +Every REPL in this guide is a `pdbp`_ instance (the maintained +fork-and-fix of `pdb++`_) pre-configured by ``tractor``: + +- pygments syntax highlighting in listings and tracebacks, +- tab completion — including an automatic fixup for + libedit-compiled CPythons (e.g. ``uv``-distributed pythons), +- sticky mode available via the ``sticky`` command (off by + default), +- no long-line truncation (terminal resizes behave), +- the ``(Pdb+)`` prompt, ``ll``, hidden-frames support and the + rest of the ``pdb++`` goodies you may already know. + +Internal runtime frames are traceback-hidden so the REPL lands +exactly on *your* ``pause()``-call or crash frame, never on +``tractor`` guts. + +Finally, if your app owns the terminal (TUIs, fullscreen +dashboards) pass ``repl_fixture=`` to ``pause()``, +``post_mortem()`` or ``open_crash_handler()``: it's entered just +before the REPL engages (return ``False`` to skip entry entirely) +and exited on release — perfect for suspending and restoring your +screen around a debug session. + +.. _debugging-caveats: + +Caveats and platform notes +-------------------------- + +An honest list of the current rough edges: + +- **Windows**: the debugger has no CI coverage on windows at all + (the entire test module is skipped there); manual testing has + shown it *can* work, but you're in uncharted territory — + reports welcome! +- **macOS**: supported but with rough edges: special-cased prompt + re-flushing for ``bash``-on-darwin, a few tooling tests skipped + on CI, and the AF_UNIX ~104-char socket-path limit forces some + examples (like the stackscope demo above) to fall back from + ``'uds'`` to ``'tcp'`` transport. Wonder if all of it'll work + on OS X? So do we. +- **CPython 3.14**: ``greenback`` (via ``greenlet``) doesn't + support 3.14 yet, so ``pause_from_sync()`` and the builtin + ``breakpoint()`` override are effectively 3.13-only for now. + The async APIs — ``pause()`` and ``post_mortem()`` — need no + greenback and work everywhere. +- **out-of-band** ``asyncio`` **tasks**: sync pauses from aio + tasks *not* spawned via ``tractor.to_asyncio`` raise a + ``RuntimeError`` (no greenback portal was bestowed); run + ``await greenback.ensure_portal()`` inside such a task first. +- **nested-tree ctrl-c edges**: ``SIGINT`` relay through + intermediary parents that aren't themselves in debug mode still + has known rough edges — see `#320`_. +- **captured stdio**: ``pytest``-style output capture can hang a + ``pause()``; use a real terminal (or a pty à la ``pexpect``, + which is how ``tractor``'s own suite drives every one of these + examples). + +Where to next? +-------------- + +.. seealso:: + + - :doc:`/guide/context` — the SC-linked cross-actor task API + that all the crash-propagation semantics above ride on. + - :func:`tractor.pause`, :func:`tractor.post_mortem` and + :func:`tractor.pause_from_sync` in the API reference. + - ``examples/debugging/`` — 20-odd runnable scripts, nearly + every one exercised by the test suite through a real pty. + +.. _structured concurrency: + https://en.wikipedia.org/wiki/Structured_concurrency +.. _trio: https://github.com/python-trio/trio +.. _cancellation: https://trio.readthedocs.io/en/latest/ + reference-core.html#cancellation-and-timeouts +.. _pdbp: https://github.com/mdmintz/pdbp +.. _pdb++: https://github.com/pdbpp/pdbpp +.. _greenback: https://github.com/oremanj/greenback +.. _stackscope: https://github.com/oremanj/stackscope +.. _PEP 553: https://peps.python.org/pep-0553/ +.. _#320: https://github.com/goodboy/tractor/issues/320 diff --git a/docs/guide/discovery.rst b/docs/guide/discovery.rst new file mode 100644 index 00000000..cc76a7f3 --- /dev/null +++ b/docs/guide/discovery.rst @@ -0,0 +1,255 @@ +Actor discovery +=============== + +So you've spawned a tree of trio-"actors"; now their tasks need to +*find* each other to start a dialog. ``tractor`` ships a (self +admittedly) **very naive** discovery system which is nonetheless +mighty handy for wiring up service-style apps: a built-in +*registrar* actor plus a small set of lookup APIs that deliver +a live, connected ``Portal`` to whichever peer you're after. + +.. d2:: diagrams/actor_tree.d2 + :margin: + :caption: The root actor doubles as the *registrar* by + default; every spawned actor registers itself with it. + :alt: actor tree with root acting as registrar + +Because ``tractor`` is built on structured concurrency (SC), the +discovery layer is *not* some external etcd/consul-shaped service +you have to babysit; it's just another actor — normally the root +of your tree — doing a bit of bookkeeping as part of the runtime. + +Every actor phones home +----------------------- + +On runtime boot **every** actor self-registers with the registrar: +it submits its unique ``(name, uuid)`` identity pair (aka its +``uid``) mapped to the list of transport addresses its IPC server +is bound to. On graceful teardown it likewise *un*-registers, so +the registry tracks the live tree as it grows and shrinks. + +.. note:: + Actor names are **not** enforced unique — the registry is keyed + by the full ``(name, uuid)`` pair. Name-based lookups simply + resolve to the *last* registered match, so if you boot five + actors all named ``'bob'``, you get the freshest ``'bob'`` B) + +First boot: who's the registrar? +-------------------------------- + +By default the **root actor** *is* the registrar; subactors +inherit the tree's ``registry_addrs`` at spawn time so the whole +clan shares one registry with zero config on your part. + +The bootstrap rule inside ``open_root_actor()`` is delightfully +simple: + +- on boot, ping every socket addr in ``registry_addrs``; when none + are passed the per-transport defaults are used: for TCP the + loopback ``('127.0.0.1', 1616)``, for UDS a + ``registry@1616.sock`` file, + +- if a registrar answers, you boot as a plain (non-registrar) root + actor and register with the *existing* registry; your own IPC + server binds random same-transport addrs instead, + +- if **nothing answers, congratulations: you just became the + registrar**. Your transport server binds the registry addrs + themselves and you start serving lookups for everyone else. + +Pass ``ensure_registry=True`` when your program *requires* being +the one-and-only registrar; boot then fails loudly with a +``RuntimeError`` if some other process already bound the registry +socket(s). + +Looking up actors +----------------- + +All lookup APIs are async context managers, so the SC rule you +already know from the rest of ``tractor`` holds here too: any +delivered portal (and its underlying IPC channel) is scoped to +your ``async with`` block — no dangling connections. + +``find_actor()`` +**************** + +The workhorse: ask the registrar for ``name`` and connect a portal +to the match, or get ``None`` back when nobody's home: + +.. code:: python + + async with tractor.find_actor('data_feed') as portal: + if portal is None: + ... # not registered anywhere; maybe spawn it? + else: + await portal.run(do_stuff) + +Knobs worth knowing: + +- ``registry_addrs=[...]``: query specific (possibly multiple, + possibly remote) registrars instead of your tree's default, + +- ``only_first=False``: deliver a ``list[Portal]`` of *all* + matches found across the queried registrars instead of just the + first, + +- ``raise_on_none=True``: raise a ``RuntimeError`` instead of + yielding ``None`` when no match is found — for when absence is + a hard error in your app. + +``wait_for_actor()`` +******************** + +Blocks until *someone* registers under ``name``, then yields a +portal to that registree. Perfect for "wait for my sibling service +to come up" sequencing: + +.. code:: python + + async with tractor.wait_for_actor('service') as portal: + await portal.run(some_fn) + +``query_actor()`` +***************** + +A lookup *without* connecting to the target: yields an +``(addr, reg_portal)`` pair where ``addr`` is the peer's preferred +transport address, or ``None`` when nothing is registered under +that name. Use it for liveness peeks or to log where a service +lives without actually dialing it up. + +``get_registry()`` +****************** + +Yields a portal straight to the registrar actor itself — or a +``LocalPortal`` shim when the calling actor *is* the registrar +(no IPC required to talk to yourself, hopefully). + +Fast paths and address preference +--------------------------------- + +Before doing any RPC to the registrar, every lookup first scans +the calling actor's *already-connected peers*: if you have a live +channel to an actor named ``name`` you get a portal over it +immediately, no registrar round-trip at all. + +When a registry entry holds *multiple* addresses (a multihomed +actor) the "best" one is chosen by locality: + +1. UDS — same-host guaranteed, lowest overhead, + +2. local TCP — loopback or any of this host's own interface + addrs, + +3. remote TCP — the only option when actually distributed. + +Within a tier the most recently registered addr wins. Stale +entries (an addr that no longer accepts connections) are detected +on use and deleted from the registrar's table on your behalf. + +Demo: register and find a service +--------------------------------- + +The simplest possible spin: start a subactor, ask the registrar +where it lives, and wait on its registration: + +.. literalinclude:: ../../examples/service_discovery.py + :caption: examples/service_discovery.py + :language: python + +The daemon-service pattern +-------------------------- + +The classic deployment shape: a long-lived daemon actor serves +RPC, later-running code discovers it by name, calls in, and +gracefully cancels it when the job is done: + +.. literalinclude:: ../../examples/service_daemon_discovery.py + :caption: examples/service_daemon_discovery.py + :language: python + +Note the teardown ordering — *graceful cancel* of the daemon via +its portal is part of the pattern; under SC a "service" is still +somebody's child and somebody is responsible for reaping it. + +Joining an existing tree from outside +------------------------------------- + +Discovery isn't limited to a single program: any standalone script +can join a running tree by booting its *own* root actor pointed at +the existing registrar: + +.. code:: python + + import trio + import tractor + + async def main(): + async with ( + # contact the live tree's registrar + tractor.open_root_actor( + registry_addrs=[('127.0.0.1', 1616)], + ), + tractor.find_actor('data_feed') as portal, + ): + ... # RPC away like you were born here + + trio.run(main) + +Per the bootstrap rules above, if the registrar at those addrs is +*not* reachable this process simply becomes its own (registrar) +root — so the same code works standalone and as a tree-joiner. + +"Arbiter"? A legacy naming note +------------------------------- + +In older releases (and many an old blog post or issue thread) the +registrar actor was called the *arbiter*, with matching APIs like +``get_arbiter()`` and an ``arbiter_addr`` argument. All of that +terminology is retired: it's *registrar*/*registry* everywhere now +(``registry_addrs``, ``get_registry()``, ...) and the +``tractor.Arbiter`` export survives only as a back-compat alias of +``tractor.Registrar``. If you see "arbiter" somewhere, mentally +substitute "registrar" and you're up to date. + +.. note:: + Multihoming nerds: ``tractor.discovery`` also ships + libp2p-style *multiaddr* helpers — ``mk_maddr()`` and + ``parse_maddr()`` — for describing transport endpoints as + structured strings. + +Very naive, very honest +----------------------- + +To be clear, this is a **very naive** discovery system: one +process-tree-local registrar holding a dict, no replication, no +re-election when it dies, no cross-host propagation. That's +intentional (for now); it covers the "wire up my services on this +host" case without dragging in a consensus protocol. + +On the roadmap (issue `#216`_ tracks a chunk of it): + +- registrar high(er)-availability: staying up past tree teardown + and re-election, + +- a `gossip protocol`_ for decentralized cross-host discovery (the + zguide's `discovery`_ chapter is the spiritual reference), + +- `modern protocol`_ (rendezvous) style meet-up points. + +If any of that scratches your itch, the issue tracker would love +to hear from you. + +.. seealso:: + - :doc:`/guide/testing` — watching live actor trees (and their + registrar) while the test suite or your app runs. + - API refs: :func:`tractor.find_actor`, + :func:`tractor.wait_for_actor`, + :func:`tractor.query_actor`, + :func:`tractor.get_registry`, + :class:`tractor.Registrar`. + +.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol +.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol +.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery +.. _#216: https://github.com/goodboy/tractor/issues/216 diff --git a/docs/guide/index.rst b/docs/guide/index.rst new file mode 100644 index 00000000..cbd4fe7b --- /dev/null +++ b/docs/guide/index.rst @@ -0,0 +1,52 @@ +Guides +====== +Task-focused walkthroughs of every major ``tractor`` +subsystem, each built around real, *test-suite +verified* example scripts from the repo's +``examples/`` dir (we never copy-paste code into +docs; what you read is what CI runs). + +Roughly in "first date to long term relationship" +order, + +- :doc:`spawning` — actor nurseries, daemons + + one-shot workers, process lifetimes. +- :doc:`rpc` — portals: calling into another + process like it's a local ``await``. +- :doc:`context` — the cross-actor task-pair + primitive at the heart of modern ``tractor``. +- :doc:`streaming` — one-way and bidirectional + msg streams between actors. +- :doc:`cancellation` — supervision, error + boxing + propagation, teardown discipline. +- :doc:`debugging` — the multi-process native + REPL debugger; our flagship DX feature B) +- :doc:`discovery` — the registrar, finding + actors by name, service patterns. +- :doc:`clustering` — quick flat process + clusters via one ``async with``. +- :doc:`parallelism` — worker pools without + pools; a ``concurrent.futures`` re-think. +- :doc:`asyncio` — "infected asyncio" mode: + SC-supervise ``asyncio`` tasks from ``trio``. +- :doc:`msging` — typed IPC payloads, the wire + msg-spec and custom codecs. +- :doc:`testing` — running + monitoring the + test suite (and testing your own actor apps). + +.. toctree:: + :hidden: + :maxdepth: 1 + + spawning + rpc + context + streaming + cancellation + debugging + discovery + clustering + parallelism + asyncio + msging + testing diff --git a/docs/guide/msging.rst b/docs/guide/msging.rst new file mode 100644 index 00000000..5c5a4d12 --- /dev/null +++ b/docs/guide/msging.rst @@ -0,0 +1,260 @@ +Typed messaging +=============== +Every value that crosses an actor boundary rides inside a typed +msg. ``tractor`` ships a small, fixed family of msg types, the +"SC-transitive supervision protocol", which encapsulates *all* +RPC dialogs in the tree such that `structured concurrency`_ (SC) +semantics -- parent-child task linkage, error propagation, +graceful cancellation -- hold across every process hop. On top of +that +protocol you can layer **your own** payload type contracts, +per-endpoint, and have them enforced at runtime by the codec. + +.. note:: + + Older posts and readmes claim ``tractor`` "uses + ``msgpack``(-python)" on the wire. The wire *encoding* is still + msgpack, but since ``0.1.0a5`` all codec work is done by + msgspec_ against a strictly-typed, tagged-union msg-spec; + neither ``msgpack-python`` nor ``u-msgpack`` are involved. + +The wire format +--------------- +Each protocol msg is a :class:`msgspec.Struct` subtype declared +with ``tag=True, tag_field='msg_type'``, so the full set decodes +as a `tagged union`__ with zero dispatch code of our own. The +payload-carrying msgs all inherit from ``PayloadMsg`` which boxes: + +__ https://jcristharif.com/msgspec/structs.html#tagged-unions + +- ``.cid`` -- the "context id" identifying which dialog (i.e. + which ``Context``) the msg belongs to, +- ``.pld`` -- the *payload*, aka your app's actual data. + +Decoding is deliberately two-layered: + +- the **transport codec** decodes only the protocol *envelope*, + intentionally leaving ``.pld`` as raw bytes + (:class:`msgspec.Raw`), +- a **per-context payload-receiver** (the internal ``PldRx``) then + decodes each ``.pld`` against *that* dialog's user-defined type + spec. + +This split is what lets every ``Context`` carry its own msg-spec +without reconfiguring the shared transport, keeps the runtime's +own traffic immune to your app's spec choices, and makes any +validation failure attributable to exactly one dialog (and thus +one task pair) instead of nuking the whole channel. + +The protocol family +------------------- +The entire msg-spec is ten types, all importable from +``tractor.msg`` (defined in ``tractor.msg.types``): + +.. list-table:: + :header-rows: 1 + :widths: 18 82 + + * - msg type + - role + * - ``Aid`` + - actor-identity handshake; the first thing two peers + exchange on connect (name, uuid, pid). + * - ``SpawnSpec`` + - parent -> child runtime config sent right after ``Aid``: + enabled modules, registry/bind addrs, runtime vars. + * - ``Start`` + - request to remotely schedule an RPC task: target + namespace + func name, kwargs and the caller's uid. + * - ``StartAck`` + - the callee's ack declaring the endpoint's "functype": + ``asyncfunc``, ``asyncgen`` or ``context``. + * - ``Started`` + - the first value passed to ``ctx.started()``; completes + the context handshake. + * - ``Yield`` + - one streamed value per ``MsgStream.send()`` call. + * - ``Stop`` + - graceful stream termination; the IPC rendition of + ``StopAsyncIteration``. + * - ``Return`` + - the final return value of the remote task fn. + * - ``CancelAck`` + - ``bool`` result of a runtime cancel-request; always + decodable so graceful cancellation can never be broken + by a custom msg-spec. + * - ``Error`` + - a boxed remote exception (src uid, relay path, tb str, + ..) relayed for local re-raise as + :class:`tractor.RemoteActorError`. + +Squint and you'll see an SC task scope serialized onto the wire: +every dialog opens with ``Start``/``StartAck`` (plus ``Started`` +for ``@tractor.context`` endpoints), optionally streams +``Yield``-s until a ``Stop``, and **always** terminates with +exactly one of ``Return``, ``Error`` or ``CancelAck``. That 1:1 +mapping of msg sequence onto a cross-process task pair is why we +call the protocol *SC-transitive*: supervision semantics survive +every hop of the tree. In `(un)protocol`_ terms it's our "SC +dialog un-protocol". + +For introspection the union alias ``tractor.msg.MsgType``, the +list ``__msg_types__`` and the spec alias ``__msg_spec__`` are +all exported. + +Payload typing with ``pld_spec`` +-------------------------------- +By default ``.pld`` may be any msgspec-supported type, i.e. the +spec is ``Any``. To constrain a single endpoint's dialog, pass +a type (union) to the decorator: +``@tractor.context(pld_spec=MyStruct|None)``. The spec then +applies to all payload-carrying msgs of that dialog -- +``Started``, ``Yield`` and ``Return`` -- on both sides of the +IPC. Pro tip: keep ``None`` in your union since most endpoints +implicitly ``return None`` and a bare ``ctx.started()`` ships +``None`` too. + +.. literalinclude:: ../../examples/typed_payloads.py + :caption: examples/typed_payloads.py + :language: python + +What's going on? + +- the payload schema is just a :class:`msgspec.Struct` subtype; + anything msgspec can tag and decode works, including unions + of structs, builtins and containers. +- decorating with ``@tractor.context(pld_spec=...)`` attaches the + spec to the endpoint; both peers' payload-receivers now decode + this dialog's payloads against it. No spec sharing files, no + IDL compiler, the contract *is* the Python type. +- the happy path looks identical to untyped code: the child calls + ``await ctx.started()``, streams or returns + more conforming values, and the parent receives fully decoded + struct instances (not dicts!) on its side. +- the sad path is the point: shipping a value *outside* the spec + raises :class:`tractor.MsgTypeError`, which the example catches + to show off the failure mode; see the anatomy section below for + exactly where it gets raised. + +Where validation happens: cheap-or-nasty +---------------------------------------- +A naive impl would validate every payload on both send *and* +receive, doubling your codec bill exactly where throughput +matters most. Instead ``tractor`` follows the 0mq lords' +"`cheap or nasty`_" pattern: be **nasty** (strict, eager, +expensive) on the rare control msgs and **cheap** (lazy, fast) on +the high-rate stream path. + +- ``Started`` is the *only* payload that gets the full nasty + treatment: ``ctx.started(value)`` stringently + **roundtrip-checks** the encoded msg against the dialog's spec + *before* sending, so a non-conforming first value raises + :class:`tractor.MsgTypeError` immediately in the child and + never even hits the wire. (You can opt out per-call with + ``ctx.started(..., validate_pld_spec=False)`` if you measure + a real cost.) +- ``Yield`` payloads are **never** checked inside + ``MsgStream.send()``; they're validated receiver-side on each + ``MsgStream.receive()``. A violation raises a ``MsgTypeError`` + in the receiver *and* relays an ``Error`` msg back so the + offending sender gets one raised too. +- the remaining control msgs (``Start``, ``Return``) are likewise + validated such that violations raise in the **sending** actor, + pointing the traceback at the code that actually goofed. + +Anatomy of a ``MsgTypeError`` +----------------------------- +:class:`tractor.MsgTypeError` is the IPC equivalent of a builtin +``TypeError``: a ``RemoteActorError`` subtype raised whenever +a msg fails to decode against the active spec. The useful bits: + +- ``.bad_msg`` -- the offending msg instance (reconstructed from + its wire form when necessary) so you can inspect the actual + ``.pld`` that broke the contract. +- ``.expected_msg_type`` -- the protocol msg type the bad msg was + (supposed to be) decoded as, e.g. ``Started[Point]``. +- plus the standard ``RemoteActorError`` goodies: ``.boxed_type``, + ``.src_uid``, ``.ipc_msg`` and the fancy ``.pformat()`` tb-box + rendering. + +Practical reading guide: a *sender-side* MTE (``Started``, +``Return``) points straight at your offending ``await +ctx.started()`` or ``return`` statement, while a *receiver-side* +MTE (``Yield``) surfaces from the consumer's ``receive()`` call +with the relay copy delivered back to the producer. Either way +the failure is scoped to that one dialog; sibling contexts on the +same channel keep right on trucking. + +Custom wire types: ``mk_codec()`` and friends +--------------------------------------------- +msgspec covers a wide set of `builtin types`__ natively; for +anything else you teach the codec via extension hooks. The +easiest path is per-endpoint: ``@tractor.context()`` accepts +``enc_hook``/``dec_hook`` params right alongside ``pld_spec``. +For full control build and apply a codec yourself; encode-side: + +__ https://jcristharif.com/msgspec/supported-types.html + +.. code:: python + + from tractor.msg import mk_codec, apply_codec + + codec = mk_codec( + enc_hook=nsp_to_str, # your-type -> wire-type + ext_types=[NamespacePath], + ) + with apply_codec(codec): # ContextVar-scoped override + ... # msgs sent by this task now encode NSPs + +and decode-side, scoped to an open context (note the import from +``tractor.msg._ops``, not yet re-exported): + +.. code:: python + + from tractor.msg._ops import limit_plds + + with limit_plds( + NamespacePath, + dec_hook=str_to_nsp, # wire-type -> your-type + ext_types=[NamespacePath], + ): + ... # this dialog's payloads decode as NSPs + +``apply_codec()`` is ``ContextVar``-scoped: it overrides the +codec for the current task (and only that task), not the whole +process. For complete working flows, including hook pairing rules +and roundtrip cases, see ``tests/msg/test_ext_types_msgspec.py`` +and ``tests/msg/test_pldrx_limiting.py``. + +The runtime dogfoods this pattern with +:class:`tractor.msg.NamespacePath`: a ``str``-subtype shaped like +``'module.path:obj_name'`` used for every RPC target reference. +It ships over the wire as a plain string yet ``.load_ref()``-s +back to the actual object in the receiving actor's memory domain; +a minimal "pointer type" for shared-nothing systems. + +Toward capability-based msging +------------------------------ +The ``pld_spec`` + codec-hook layer is the foundation for the +long-game: **capability-based msging** where each dialog's +type contract doubles as a capability grant, negotiated as part +of the protocol itself. That work is tracked in `#196`_ (with the +original typed-proto epic in `#36`_); if strongly-typed +distributed systems get you going, we'd love your input. + +Where to next? +-------------- +.. seealso:: + + - :doc:`/guide/context` for the dialog API these msgs + implement: ``started()``, streams and results. + - :doc:`/guide/asyncio` for shuttling (typed) payloads into + ``asyncio``-land via an infected subactor. + - the msgspec_ docs for everything your payload types can be. + +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _msgspec: https://jcristharif.com/msgspec/ +.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern +.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols +.. _#196: https://github.com/goodboy/tractor/issues/196 +.. _#36: https://github.com/goodboy/tractor/issues/36 diff --git a/docs/guide/parallelism.rst b/docs/guide/parallelism.rst new file mode 100644 index 00000000..7a6dc1c2 --- /dev/null +++ b/docs/guide/parallelism.rst @@ -0,0 +1,176 @@ +Parallelism and worker pools +============================ + +The initial ask is almost always the same: *"how do i make a worker +pool?"* — i.e. the thing :mod:`multiprocessing` and +:class:`concurrent.futures.ProcessPoolExecutor` get reached for +once the GIL becomes the enemy. + +Here's the structured concurrency (SC) answer: ``tractor`` is built +to handle any SC process tree you can imagine; a "worker pool" +pattern is a trivial special case. So instead of shipping a pool +*class* with knobs bolted on, you compose one from the same two +ingredients used everywhere else in ``tractor``: an actor nursery +and some IPC. + +The stdlib baseline +------------------- + +For a fair comparison, start from the canonical +:class:`~concurrent.futures.ProcessPoolExecutor` primes example +straight out of the Python docs, + +.. literalinclude:: ../../examples/parallelism/concurrent_futures_primes.py + :caption: examples/parallelism/concurrent_futures_primes.py + :language: python + +Synchronous code, a hidden thread + IPC machine under the hood, and +an API surface (executors, futures, ``.map()``) invented to paper +over the fact that the pool isn't part of your program's task tree. +Keep an eye on three things for the rewrite: how work is submitted, +how results come back, and what happens when a worker dies. + +The ``tractor`` way +------------------- + +Now the same workload as a ``tractor`` program, + +.. literalinclude:: ../../examples/parallelism/concurrent_actors_primes.py + :caption: examples/parallelism/concurrent_actors_primes.py + :language: python + +What's different (and what isn't), + +- ``worker_pool()`` is ~30 lines of *your* code: an actor nursery + spawning ``workers`` subactors — each a full process running its + own ``trio`` task tree — kept alive and ready for work until the + block exits; ``enable_modules=[__name__]`` is the capability + allowlist letting them run this module's functions, +- jobs are "submitted" by just... calling the function: + ``portal.run(is_prime, n=value)`` runs ``is_prime()`` in a + worker and hands back its result like any local ``await``, +- results stream back through a plain + :func:`trio.open_memory_channel` *as they complete* — no futures + and no polling, +- teardown is one ``await tn.cancel()`` + (:meth:`tractor.ActorNursery.cancel`), and any worker crash + triggers the one-cancels-all machinery from + :doc:`/guide/cancellation` — a dead worker can never strand the + pool. + +This uses no extra threads, fancy semaphores or futures; all we +need is ``tractor``'s IPC! The full scorecard, + +.. list-table:: + :header-rows: 1 + :widths: 50 50 + + * - ``concurrent.futures`` + - ``tractor`` + * - ``ProcessPoolExecutor()`` + - ``worker_pool()`` — yours, ~30 lines + * - ``executor.map(is_prime, PRIMES)`` + - ``actor_map(is_prime, PRIMES)`` async-gen + * - ``Future`` + internal result queue + - :func:`trio.open_memory_channel` + * - results in input order + - results as they complete + * - worker crash -> ``BrokenProcessPool`` + - boxed :class:`tractor.RemoteActorError` + * - pool teardown on ``with`` exit + - one-cancels-all nursery teardown + +.. margin:: How many workers? + + Same calculus as any process pool: about core-count for + CPU-bound work (the default sizing in + :doc:`/guide/clustering`); more only if workers block on I/O — + though at that point you likely want plain ``trio`` tasks, not + processes. + +And because the pool is just SC code, every variation — bounded +submission, per-worker state, streaming partial results (see +:doc:`/guide/streaming`), nested pools — is a local edit to your +pool, not a feature request against an executor class B) + +An *async* pool, though? +************************ + +Yep: RPC targets must be async functions — the runtime rejects a +plain ``def`` with ``TypeError: ... must be an async function!``. +That's not zealotry, it's cancel-responsiveness: each worker is a +full ``trio`` runtime whose msg loop is what hears graceful cancel +requests, and a hot loop that never yields can't be (politely) +interrupted. + +Two practical consequences, + +- CPU-bound loops should checkpoint once in a while; note how + ``burn_cpu()`` in the next example sprinkles ``await + trio.sleep()`` calls so the worker stays responsive while still + pegging a core, +- if some sync call blocks a worker anyway you're still covered: + an unresponsive actor just rides the graceful-then-hard teardown + ladder from :doc:`/guide/cancellation` instead of acking its + cancel — slower, but never a zombie. + +Run a func in a process +----------------------- + +Even a pool can be overkill; "run this one async func in a +subprocess and give me the result" is a one-liner via +:meth:`tractor.ActorNursery.run_in_actor`, + +.. literalinclude:: ../../examples/parallelism/single_func.py + :caption: examples/parallelism/single_func.py + :language: python + +``run_in_actor()`` is a *convenience wrapper* — spawn an actor, run +exactly one task in it, reap on result — not the core spawning +model (that's :meth:`tractor.ActorNursery.start_actor` plus +:meth:`tractor.Portal.open_context`; see :doc:`/guide/context`). +But for this fire-and-collect shape it's exactly the right amount +of typing. + +As the module docstring suggests, run it under a process-tree +monitor to watch the child appear and get reaped, + +.. code:: bash + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/single_func.py \ + && kill $! + +You'll see a core get burned in both parent and child — real +parallelism, no GIL sharing, since these are processes (i.e. +*non-shared-memory threads*). + +When all you have is sync code +------------------------------ + +Honesty corner: if your workload is purely *synchronous* functions +and you've zero need for IPC dialogs, streaming, daemons or +supervision trees — i.e. you really do just want +"``ProcessPoolExecutor`` but ``trio``-native" — the smaller, +focused `trio-parallel`_ project may serve you better. ``tractor`` +happily covers the use case (as above) but brings a whole runtime +along for the ride. (And when blocking I/O — not the GIL — is the +actual problem, plain in-process :func:`trio.to_thread.run_sync` +may be all you ever needed.) + +And to *see* that runtime's process-management story — a per-core +fleet self-destructing with zero zombies left behind — go run +``examples/parallelism/we_are_processes.py``, walked through in +the :doc:`/start/quickstart`. + +.. seealso:: + + - :doc:`/guide/clustering` — the one-liner flat-cluster + convenience (``open_actor_cluster()``) for when even a + hand-rolled pool is too much typing, + - :doc:`/guide/cancellation` — why pool teardown is bulletproof + (graceful-then-hard escalation, no zombies), + - :doc:`/guide/context` — the core per-task API your pool + workers can graduate to. + +.. _trio-parallel: https://github.com/richardsheridan/trio-parallel diff --git a/docs/guide/rpc.rst b/docs/guide/rpc.rst new file mode 100644 index 00000000..6c5d8fa0 --- /dev/null +++ b/docs/guide/rpc.rst @@ -0,0 +1,177 @@ +RPC: calling into other actors +============================== +Every spawn call from :doc:`/guide/spawning` hands you back +a :class:`~tractor.Portal`: a live handle for calling into +another actor's **memory domain**. The name is borrowed from +``trio``'s portal concept — an object you use to submit work +*into* a separate concurrency domain — except here that domain +is a whole other process. + +.. d2:: diagrams/runtime_stack.d2 + :margin: + :caption: The layers a ``portal.run()`` request rides through. + :alt: app, tractor runtime, IPC channel and OS process layers + +There are **no proxy objects** and no special calling +conventions: you pass a plain function reference plus keyword +args, and Python's normal ``await``-able semantics apply. The +function just happens to *run somewhere else*; from the calling +task it looks as though it was called locally. And since this is +all structured concurrency (SC) under the hood, the remote task +runs inside the callee's supervised task tree while its result +— or its failure, as a boxed +:exc:`~tractor.RemoteActorError` — always comes back to *you*. + +``Portal.run()``: pass the function, not a string +------------------------------------------------- +:meth:`~tractor.Portal.run` schedules an async function as +a **new task** in the remote actor and waits on its result: + +.. code:: python + + async with tractor.open_nursery() as an: + portal = await an.start_actor( + 'service', + enable_modules=[__name__], + ) + answer = await portal.run(movie_theatre_question) + +The rules of engagement: + +- the target must be an **async function** and its defining + module must be in the callee's ``enable_modules`` allowlist, + else an :exc:`~tractor.ModuleNotExposed` error is relayed + back (see :doc:`/guide/spawning` for the capability-allowlist + story). +- arguments are passed **by keyword only**; they ride the IPC + layer as msgspec_-encoded msgs, so keep them serializable. +- every call schedules a *fresh* task remotely — call it twice + and the callee runs two tasks, each supervised in its own + right. +- remote exceptions re-raise locally as + :exc:`~tractor.RemoteActorError` with the original type + preserved via ``.boxed_type``. + +.. note:: + + Passing dotted-path *strings* to ``run()`` is an ancient, + deprecated form; always pass the function reference. If you + really need name-based addressing use ``run_from_ns()`` + below. + +Namespaced daemons: ``run_from_ns()`` +------------------------------------- +Sometimes the calling process can't (or shouldn't) import the +target function — think a long-running rpc-daemon serving +modules your client never loads. For that, +:meth:`~tractor.Portal.run_from_ns` takes the explicit +namespace path: + +.. code:: python + + await portal.run_from_ns('mypkg.service', 'ping') + +This is literally how ``.run()`` works underneath: the pair is +encoded as a ``'mod.path:func'`` style msg and resolved against +the callee's enabled modules. + +One special namespace exists: ``'self'`` resolves to the remote +:class:`~tractor.Actor` instance, i.e. the runtime itself. It's +how internal machinery (cancel requests, registry ops) travels; +don't build your app on it. + +One-shot results: ``wait_for_result()`` +--------------------------------------- +A portal returned from +:meth:`~tractor.ActorNursery.run_in_actor` has exactly one +"main" task running remotely; that task's ``return`` value is +delivered as the portal's *final result*: + +.. code:: python + + portal = await an.run_in_actor(fib, n=10) + final = await portal.wait_for_result() + +Semantics worth knowing: + +- it blocks until the remote task returns, re-raising any + remote error in the usual boxed form. +- once resolved it's idempotent: later calls return the same + cached value. +- a *daemon* portal (from ``start_actor()``) has no main task, + so there's no final result to wait for: you'll get a warning + plus a ``NoResult`` sentinel. Results of individual daemon + calls come straight back from each ``await portal.run()``. + +Pure RPC daemons: ``run_daemon()`` +---------------------------------- +When a process's *only* job is to sit at the root of its own +tree and serve RPC, skip the boilerplate with +:func:`tractor.run_daemon`: + +.. code:: python + + import tractor + + tractor.run_daemon( + ['mypkg.service'], + name='service', + ) + +It's a blocking convenience (it calls ``trio.run()`` for you): +boot a root actor with the given modules enabled for RPC, then +sleep until cancelled. Pair it with the discovery system — +:func:`tractor.find_actor` / :func:`tractor.wait_for_actor` +from a *separate* program — and you've got a tiny service +architecture with zero framework ceremony; see +``examples/service_daemon_discovery.py`` for the full pattern. + +Fan-out: RPC through nested trees +--------------------------------- +Portals compose. An RPC task is just a ``trio`` task, so it can +open its own :class:`~tractor.ActorNursery` and portal into +*its* children — one inbound call fanning out into a whole +sub-tree of work. The mid-tier function from the nested-tree +example: + +.. literalinclude:: ../../examples/nested_actor_tree.py + :caption: examples/nested_actor_tree.py (supervisor fan-out) + :language: python + :pyobject: fan_out_squares + +The root portals into the ``supervisor`` actor; the +supervisor's RPC task spawns the leaf workers, portals into +each, and returns the combined result back up. Failures at any +depth relay hop-by-hop as boxed errors, and cancelling the root +call tears down the entire sub-tree — SC, transitively. + +When to graduate to ``Context`` +------------------------------- +``portal.run()`` is great for one-shot, request-response calls. +Reach for :meth:`~tractor.Portal.open_context` with an +``@tractor.context`` endpoint as soon as you want: + +- a long-lived dialog with state held on both sides, +- bidirectional streaming via ``ctx.open_stream()``, +- typed payload contracts (``pld_spec``) enforced at the msg + layer, +- or *task-scoped* cancellation: ``Context.cancel()`` cancels + just the linked remote task, whereas + :meth:`~tractor.Portal.cancel_actor` nukes the **entire** + remote runtime and its process. + +In fact the source plans for ``Portal.run()`` itself to be +rebuilt on top of ``open_context()`` — contexts *are* the core +inter-actor protocol. Take the full tour in +:doc:`/guide/context`. + +.. seealso:: + + - :doc:`/guide/spawning` — where portals come from and how + their actors are supervised. + - :doc:`/guide/context` — the structured cross-actor task + API: handshake, streaming, typed payloads. + - :doc:`/guide/cancellation` — what happens to in-flight RPC + when trees get torn down. + +.. _msgspec: https://jcristharif.com/msgspec/ diff --git a/docs/guide/spawning.rst b/docs/guide/spawning.rst new file mode 100644 index 00000000..f1bbf26f --- /dev/null +++ b/docs/guide/spawning.rst @@ -0,0 +1,299 @@ +Spawning actors +=============== +If you know trio_ you know the drill: you don't get to launch +a task off into the void, you open a nursery_, the nursery owns +the task, and the block can't exit until every child is done. +That discipline is `structured concurrency`_ (SC) — see the +seminal `blog post`_ if you haven't yet — and it's the whole +religion around here. + +``tractor`` applies that exact discipline to **processes**: an +:class:`~tractor.ActorNursery` is a *process nursery*. Every +"task" it starts is a fresh Python process running its own +``trio.run()``-scheduled task tree; we call each one a +``trio``-"*actor*". Parents must wait on (and clean up after) +their children, transitively, all the way down the tree. + +.. d2:: diagrams/actor_tree.d2 + :caption: A process tree of ``trio``-task-trees. + :alt: a nested actor tree where every parent supervises its children + +Though a "process nursery" differs in complexity (and slightly +in semantics) from a single-threaded task nursery, most of the +interface is the same. The main difference is that each spawned +child contains a full, *parallel-executing* ``trio`` task tree. +The following super powers ensue: + +- tasks started in a child actor are completely independent of + tasks started in the current process; they execute in + **parallel** and are scheduled by their own actor's ``trio`` + run loop. +- tasks scheduled in a remote process still maintain an SC + protocol *across memory boundaries* using a so called + "SC dialogue protocol" which keeps task-hierarchy lifetimes + linked across the IPC layer. +- a remote task can fail and have that failure relayed back to + the caller task (living in some other actor) as a serialized + :exc:`~tractor.RemoteActorError`; no spawned process or RPC + task can ever just go off on its own. + +Opening a (process) nursery +--------------------------- +:func:`tractor.open_nursery` is the entrypoint: + +.. code:: python + + async def main(): + async with tractor.open_nursery() as an: + ... # spawn some actors B) + + trio.run(main) + +Notice there's no runtime-boot ceremony: if no actor runtime is +up yet (i.e. you're in a plain old Python process), +``open_nursery()`` *implicitly* enters +:func:`tractor.open_root_actor` for you, making this process the +**root actor** of a new tree. Any extra keyword args you pass +are proxied straight through to ``open_root_actor()``, so the +runtime config lives wherever you open your first nursery: + +.. code:: python + + async with tractor.open_nursery( + loglevel='info', + debug_mode=True, # crash-to-REPL for the whole tree + ) as an: + ... + +If you want the runtime up *without* spawning anything (or you +prefer the config to be loudly explicit) enter +``open_root_actor()`` yourself first; the nursery will detect +the running runtime and skip the implicit boot. Either way, +nesting a second root inside an existing tree is an error. + +Inside a *subactor* the same call just works: any actor may open +nurseries of its own, which is how you get arbitrarily deep +trees (more on that below). + +``start_actor()``: daemons that live until cancelled +---------------------------------------------------- +:meth:`~tractor.ActorNursery.start_actor` is **the** core +spawning primitive. It starts a *daemon* actor: a process with +no designated "main task" besides the runtime itself. It boots, +registers with its parent, and then sits there serving RPC +requests until somebody cancels it. You get back a +:class:`~tractor.Portal` for doing exactly that kind of +somebody-ing: + +.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py + :caption: examples/actor_spawning_and_causality_with_daemon.py + :language: python + +What's going on here? + +- ``start_actor('frank', enable_modules=[__name__])`` forks off + a new process, boots a ``tractor`` runtime inside it, and + allows it to serve functions from the current module (see the + allowlist section below). +- each ``await portal.run(...)`` schedules a *new* task in + frank's task tree and waits on its result — the full RPC story + lives in :doc:`/guide/rpc`. +- frank has no main task to complete, so without the final + ``await portal.cancel_actor()`` the nursery block would wait + on him **forever**. Daemon lifetimes are *yours* to end; that + explicitness is the point. + +``run_in_actor()``: quick one-shot parallelism +---------------------------------------------- +:meth:`~tractor.ActorNursery.run_in_actor` is the convenience +wrapper: spawn an actor, run exactly one async function in it, +then reap the process as soon as the result arrives. + +.. code:: python + + async with tractor.open_nursery() as an: + portal = await an.run_in_actor(burn_cpu) + # burn rubber in the parent too... + await burn_cpu() + total = await portal.wait_for_result() + +A few details worth knowing: + +- the actor is named after the function unless you pass + ``name='something_cuter'``. +- the function's module is auto-added to the child's + ``enable_modules`` allowlist. +- extra ``**kwargs`` are forwarded to the function itself. +- the child is *auto-cancelled* once its "main" result lands; + at nursery exit these run-once children are always reaped + first (causality_ is paramount!). + +.. note:: + + ``run_in_actor()`` is a convenience, **not** the core model. + The source literally marks it for an eventual rebuild as + a thin "hilevel" wrapper on top of + :meth:`~tractor.Portal.open_context` (the modern inter-actor + task API). Teach your fingers to use it for quick + fire-and-collect parallelism — think a per-function + trio-parallel_ style one-shot — and reach for + ``start_actor()`` + ``open_context()`` for anything + long-lived, stateful or streaming + (:doc:`/guide/context`). + +Actor lifetimes and teardown order +---------------------------------- +So we have two lifetime flavors: + +- **run-once** (``run_in_actor()``): lives exactly as long as + its single task; reaped the moment its result (or error) + arrives. +- **daemon** (``start_actor()``): lives until *someone* cancels + it — an explicit ``await portal.cancel_actor()``, a bulk + ``await an.cancel()``, or the one-cancels-all strategy kicking + in on error. + +On a clean exit of the nursery block the teardown order is: + +1. the nursery waits on every run-once actor's final result; + any errors from these are raised immediately so your code + (acting as supervisor) gets first crack at handling them. +2. then it waits on daemon actors — **indefinitely**. If you + spawned a daemon, you own its lifetime. + +When a child *is* cancelled, teardown is graceful-first per SC +discipline: the runtime sends an IPC cancel request and gives +the child a bounded window to ack; only when a child is too +slow does the nursery escalate to an OS-level hard kill of the +process. There is no path where a child is silently left +running: + + ``tractor`` tries to protect you from zombies, no matter + what. If you can create zombie child processes (without + using a system signal) it **is a bug**. + +Per-process cleanup hooks +************************* +Need something torn down when an actor's runtime exits, no +matter how it exits? Every actor carries +a process-global :class:`contextlib.ExitStack` at +``Actor.lifetime_stack`` which is closed at the very end of +runtime teardown: + +.. code:: python + + db = await connect_db() + tractor.current_actor().lifetime_stack.callback(db.close) + +(A so-far under-advertised api — expect it to get more love.) + +When things blow up: one-cancels-all +------------------------------------ +The default (and currently only) supervision strategy is the +same one ``trio`` nurseries use: **one-cancels-all**. If your +nursery-block body errors, every child actor is cancelled. If +a child errors, the failure is relayed to the nursery as a +boxed :exc:`~tractor.RemoteActorError` (original type preserved +via ``.boxed_type``), all *other* children are cancelled, and +the error(s) re-raise locally — exactly like ``trio``, just +process-wide. Erlang-style alternative strategies are a long +standing roadmap item. + +The full story — how cancel requests relay across the tree, who +``.canceller`` was, debugging mid-teardown — lives in +:doc:`/guide/cancellation`. + +The module allowlist: ``enable_modules`` +---------------------------------------- +A subactor will only serve functions from modules its parent +*explicitly* enabled at spawn time: + +.. code:: python + + portal = await an.start_actor( + 'service', + enable_modules=['mypkg.service'], # or [__name__] + ) + +At child boot the runtime imports each listed module so inbound +RPC requests can resolve function references against it. Ask +a peer to run something from any *other* module and you get an +:exc:`~tractor.ModuleNotExposed` error relayed back — the child +never even looks the function up. + +Think of it as the first, deliberately coarse layer of +capability-style permissioning: if you don't hand an actor +a module, no peer can invoke anything inside it. (Finer-grained +capability-based messaging protocols are on the roadmap.) + +The ``enable_modules=[__name__]`` idiom — "let the child run +functions from the *current* module" — is what you'll use in +most scripts; bigger apps tend to pass dedicated service-module +paths instead. + +Per-child knobs +--------------- +Both spawn methods accept per-child config so one weird child +doesn't have to drag the whole tree along: + +- ``loglevel='cancel'`` — crank console logging for just this + subactor (the ``TRACTOR_LOGLEVEL`` env var overrides whatever + the *root* was passed, handy for test runs). +- ``debug_mode=True`` — arm the crash-handling REPL machinery + for just this child instead of tree-wide, i.e. the selective + flavor of ``open_nursery(debug_mode=True)``; see + :doc:`/guide/debugging` for the multi-process debugger tour. +- ``infect_asyncio=True`` — run the child with ``trio`` as an + ``asyncio`` guest, aka "infected asyncio" mode. +- ``enable_transports=['uds']`` — pick the IPC transport this + child should listen on (default ``'tcp'``). + +Trees all the way down +---------------------- +Since any actor can open an ``ActorNursery``, supervision trees +compose to arbitrary depth: a subactor can be a supervisor of +*its own* subactors, with every level holding the same SC +guarantees — error relay up, cancellation down, no orphans. + +.. literalinclude:: ../../examples/nested_actor_tree.py + :caption: examples/nested_actor_tree.py + :language: python + +Here the root spawns a ``supervisor`` actor whose RPC task opens +its *own* nursery and spawns the leaf workers; one call from the +root fans out through the middle layer and the aggregate comes +back up. Teardown ripples in reverse: the leaves are reaped when +the supervisor's nursery exits, the supervisor when the root +cancels it. + +Watching your tree grow +----------------------- +Actors are real processes, so your favorite system tools just +work. The house incantation runs any example beside a live +process-tree monitor:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/nested_actor_tree.py \ + && kill $! + +Every subactor also sets its OS process title to a stable +``_subactor[@]`` marker, so ``htop``, +``ps`` and friends show *which actor is which* at a glance:: + + pgrep -af '_subactor\[' + +.. seealso:: + + - :doc:`/guide/rpc` — actually invoking functions through + all these portals you've been collecting. + - :doc:`/guide/context` — the structured, streaming-capable + inter-actor task API. + - :doc:`/guide/cancellation` — cross-actor cancellation and + error propagation semantics in depth. + +.. _trio: https://github.com/python-trio/trio +.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker +.. _trio-parallel: https://github.com/richardsheridan/trio-parallel diff --git a/docs/guide/streaming.rst b/docs/guide/streaming.rst new file mode 100644 index 00000000..1c071268 --- /dev/null +++ b/docs/guide/streaming.rst @@ -0,0 +1,244 @@ +Cross-process streaming +======================= + +Spawning processes is the boring half of ``tractor``: the **real +cool stuff** is the native support for cross-process *streaming*. +Yes, you saw it here first — 2-way msg streams with reliable, +transitive setup/teardown semantics, wired straight into the +runtime's `structured concurrency`_ (SC) supervision machinery so +that *how* a stream ends is part of the protocol. + +No broker, no topic exchange, no IDL compiler. The IPC layer is a +deliberately "`cheap or nasty`_" `(un)protocol`_: a tiny set of +msgspec_-typed msgs over a transport (TCP or UDS today) with +payload typing opt-in per dialog — handshake msgs get the *nasty* +treatment (strict validation) while high-rate stream payloads +stay *cheap* (receiver-side checks only). See +:doc:`/guide/context` for the typed ``pld_spec`` contract bits. + +Two ways to stream +------------------ + +.. margin:: It's a ``trio.abc.Channel`` + + :class:`tractor.MsgStream` implements + :class:`trio.abc.Channel` — ``send()``, + ``receive()``, async-iteration, ``aclose()`` — + so trio-generic channel code drives an IPC + stream unchanged. + +- **Bidirectional, context-based**: open a + :class:`tractor.Context` to a peer task then enter + ``ctx.open_stream()`` for a full-duplex + :class:`tractor.MsgStream`. This is the modern core API, taught + end-to-end in :doc:`/guide/context`; we won't re-teach it here. + +- **One-way, portal-based**: point + :meth:`tractor.Portal.open_stream_from` at a plain async + generator fn in the peer actor. Legacy, but perfectly fine for + simple produce/consume pipelines — and it powers the classic + examples below. + +Rule of thumb: if the consumer ever needs to *talk back* — acks, +control msgs, a final result — use a context. If it's a pure +pipeline stage, either works and the one-way form is less typing. + +One-way streaming from an async generator +----------------------------------------- + +The OG api. Write an async generator in the target actor's +module; iterate its yields from the spawning side: + +.. literalinclude:: ../../examples/asynchronous_generators.py + :caption: examples/asynchronous_generators.py + :language: python + +Each ``yield`` crosses the process boundary as one msg and feeds +the parent's ``async for``. When the consumer ``break``\ s out +and exits the ``open_stream_from()`` block the far-end generator +task is cancelled for you: the producer's lifetime is *coupled to +the consumer's scope* so a one-way stream can never leak a remote +task. + +Any extra kwargs (``stream_data, seed=100`` style) are forwarded +to the remote generator's call, and a non-async-gen target is +rejected up front with a ``TypeError``. + +.. note:: + + No decorator required — any plain async-gen fn works. You may + still meet ``@tractor.stream`` in the wild; it's the legacy + marker for one-way endpoints and sticks around only for + compat (heads up: the param name ``ctx`` is reserved for + ``@context`` endpoints nowadays, so legacy fns should call + theirs ``stream``). New code wanting anything fancier than a + one-way pipe should use :func:`tractor.context` + + ``ctx.open_stream()``. + +.. warning:: + + One-way means one way: there's no sending *to* the generator + side and no graceful consumer-to-producer stop msg — the + teardown above is cancel-based. Needing upstream control flow + is the sign you've outgrown this API. + +A full-fledged streaming service +-------------------------------- + +Now let's get fancy: compose one-way streams through a nested +actor tree and you've got yourself a fan-in pipeline. + +.. d2:: diagrams/streaming_pipeline.d2 + :caption: Four actors, three streams, one deduped feed. + :alt: two streamer actors fan in to an aggregator then root + +.. literalinclude:: ../../examples/full_fledged_streaming_service.py + :caption: examples/full_fledged_streaming_service.py + :language: python + +What's going on? + +- the root actor spawns ``'aggregator'`` which opens its *own* + actor nursery and spawns ``'streamer_1'`` + ``'streamer_2'``: 4 + processes total, supervision nested two levels deep with zero + special casing. + +- ``aggregate()`` opens a one-way stream from each streamer and + fans both into a single :func:`trio.open_memory_channel` via + one local trio task per portal — in-actor fan-in riding trio's + built-in backpressure end-to-end. + +- duplicates get dropped via a ``set`` and the deduped sequence + is *re-yielded* upward: ``aggregate()`` is itself an async gen + being consumed over IPC by the root. Streams compose. + +- when the seed runs out the streamer gens finish, the memory + channel drains closed, the aggregator's gen returns and the + root's ``async for`` ends; ``await an.cancel()`` then reaps the + subtree. Every exit is awaited — if you can produce a zombie + process from this, it **is a bug**. + +Watch the tree breathe while it runs, using the README's +signature process-monitor incantation:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/full_fledged_streaming_service.py \ + && kill $! + +No extra threads, no fancy semaphores, no futures; all we need is +``tractor``'s IPC. + +Two streams, one portal +----------------------- + +Every ``open_stream_from()`` call starts its *own* remote task — +even through the same portal — so two local consumer tasks can +independently stream the same generator fn concurrently, both +dialogs multiplexed over the single underlying IPC channel: + +.. literalinclude:: ../../examples/multiple_streams_one_portal.py + :caption: examples/multiple_streams_one_portal.py + :language: python + +The add-else-remove trick on the shared ``consumed`` list is the +proof: each value arrives in *both* streams, getting appended by +whichever task sees it first and removed by the other, so the +list always ends up empty. Two streams, same data, zero +interference. + +This works because every dialog is keyed by its own context id +(``Context.cid``) — any number of concurrent streams, contexts +and one-shot RPCs share a single underlying +:class:`tractor.Channel` per peer pair. + +Fan-out inside an actor: ``MsgStream.subscribe()`` +-------------------------------------------------- + +The inverse pattern: *one* IPC stream feeding *many* local tasks. +Instead of paying for N redundant cross-process streams, call +:meth:`tractor.MsgStream.subscribe` to get a +``BroadcastReceiver`` — a tokio-style broadcast channel from +``tractor.trionics`` — which copies every received value to each +subscribed task: + +.. literalinclude:: ../../examples/streaming_broadcast_fanout.py + :caption: examples/streaming_broadcast_fanout.py + :language: python + +Each task entering ``stream.subscribe()`` receives its own copy +of everything sent from that point on. The underlying stream +keeps pace with the *fastest* subscriber; a task falling more +than the buffered window behind has its next receive raise +``tractor.trionics.Lagged`` to say it lost data. + +The broadcast handle stays duplex btw: it proxies ``send()`` +through to the underlying stream, so each subscriber task can +keep talking upstream while consuming its fan-out copy. + +.. warning:: + + ``.subscribe()`` is **idempotent and non-reversible**: the + first call permanently swaps the stream's receive machinery + over to the internally allocated broadcaster. There's no + un-subscribing back to the raw stream, so make sure you're ok + with the (theoretical) overhead before opting in. + +Consuming: ``async for`` and friends +------------------------------------ + +``async for msg in stream:`` is just sugar over repeated +``await stream.receive()``. The receive-side surface: + +- ``receive()`` — next msg, or raises :exc:`trio.EndOfChannel` + on a graceful far-end close (``async for`` translates that + into a clean loop exit for you). + +- ``receive_nowait()`` — opportunistic, non-blocking drain. + +- ``closed`` — property flagging an already-ended stream. + +Send-side it's just ``await stream.send(data)`` — one ``Yield`` +msg per call carrying any msgspec_-encodable payload (or +whatever your ``pld_spec`` permits, see :doc:`/guide/context`). + +End-of-stream: close vs. cancel +------------------------------- + +How a stream ends is part of the protocol; the runtime keeps the +polite case and the violent case distinct: + +- **graceful close**: the far side exits its stream block, its + async gen returns, or it calls ``await stream.aclose()``. A + ``Stop`` msg is sent so your ``async for`` simply ends + (``StopAsyncIteration``, via :exc:`trio.EndOfChannel` under the + hood). A normal, non-error ending — the dialog's result phase + proceeds as usual. + +- **cancel or error**: no ``Stop`` is sent. Instead the + cancel/error itself is relayed so the far end *knows* the + dialog did not end on purpose and raises accordingly — a + :exc:`tractor.ContextCancelled`, a boxed + :exc:`tractor.RemoteActorError`, etc. See the cancellation + section of :doc:`/guide/context` for exactly who raises what. + +Tying it together: every ``MsgStream`` is **one-shot use**. Both +endings are final — once closed a stream can't be re-opened and +the supported "retry" is opening a fresh :class:`tractor.Context` +(they're cheap). + +.. seealso:: + + - :doc:`/guide/context` — the full ``Context`` lifecycle: the + handshake, results, cancellation semantics and the + overrun/backpressure knobs. + + - :class:`tractor.MsgStream` and + :meth:`tractor.Portal.open_stream_from` API docs. + + - The zguide chapters our wire philosophy is named after: + "`cheap or nasty`_" and `(un)protocol`_\ s. + +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern +.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols +.. _msgspec: https://jcristharif.com/msgspec/ diff --git a/docs/guide/testing.rst b/docs/guide/testing.rst new file mode 100644 index 00000000..d07415b8 --- /dev/null +++ b/docs/guide/testing.rst @@ -0,0 +1,256 @@ +Testing tips +============ + +``tractor``'s test suite is a different kind of beast than your +average single-proc pytest run: nearly every test spawns a real +**process tree**, hammers on cancellation under structured +concurrency (SC), and tears the whole thing down again — hundreds +of times per session. This page collects the tips, knobs and +one-liners that make hacking on (and with) the suite pleasant. + +Running the suite +----------------- + +This is a uv_-managed project, so after cloning it's just:: + + uv sync --dev + uv run pytest tests/ + +Expect a *lot* of process churn; the suite is effectively a +rolling chaos exercise for the runtime. + +The classic fix-iterate loop when something breaks:: + + # stop at the first failure + uv run pytest tests/ -x + + # then iterate on just the failures til green + uv run pytest --lf -x + +``--lf`` (last-failed) re-runs only what failed previously, so +combined with ``-x`` you get a tight one-test-at-a-time repair +loop. + +Suite-specific flags +******************** + +The repo auto-loads the bundled ``tractor._testing.pytest`` plugin +(via ``addopts`` in ``pyproject.toml``) which adds a few extra +flags: + +- ``--spawn-backend ``: pick the process spawn backend for + the session (default ``'trio'``); same keys as the + ``start_method`` runtime argument, + +- ``--tpt-proto [...]``: which IPC transport(s) opting-in + suites should run against, eg. ``--tpt-proto uds``, + +- ``--tpdb`` / ``--debug-mode``: flip on the ``debug_mode`` + fixture so debugger-aware tests boot their trees with the + crash-REPL enabled, + +- ``--enable-stackscope``: install the ``SIGUSR1`` task-tree dump + handler in pytest *and* every spawned subactor — much lighter + than a full debug-mode run when you only need stack visibility + during a hang hunt, + +- ``--ll `` / ``--tl ``: console loglevels; + ``--tl`` targets the ``tractor``-as-runtime logger and accepts + a per-subsystem spec like ``'devx:runtime,trionics:cancel'``. + +Watch the tree grow +------------------- + +The single most useful trick while the suite (or any ``tractor`` +app) runs: keep a live ``pstree`` view going in a side terminal:: + + watch -n 0.1 "pstree -a $(pgrep -f pytest)" + +You'll see actor processes pop in and out of existence as each +test builds and reaps its tree. Launch it *after* pytest is up +(the pid is substituted once, at ``watch`` startup). + +Every subactor also sets its OS process title (via +``setproctitle``) to ``_subactor[@]`` so the +tree view shows *which actor is which* at a glance — and targeted +greps stay easy:: + + pgrep -af '_subactor\[' + +For a single example script, the repo's signature incantation +spawns the watcher alongside your program and cleans it up after:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/single_func.py \ + && kill $! + +Env-var knobs +------------- + +Two env-vars override their corresponding runtime arguments +*globally* — no application (or test) code changes required: + +``TRACTOR_SPAWN_METHOD`` + Wins over any caller-passed ``start_method`` so you can drive + the whole suite (or any app) under a different spawn backend:: + + TRACTOR_SPAWN_METHOD=mp_spawn uv run pytest tests/ -x + +``TRACTOR_LOGLEVEL`` + Wins over any caller-passed ``loglevel``; crank (or silence) + runtime console verbosity wholesale:: + + TRACTOR_LOGLEVEL=cancel uv run pytest tests/ -x -s + +``TRACTOR_ENABLE_STACKSCOPE`` + Force-install the ``SIGUSR1`` task-tree dump handler in every + actor, debug-mode or not; then + ``pkill --signal SIGUSR1 -f `` dumps every + actor's live ``trio`` task tree. + +Debug mode vs. pytest capture +----------------------------- + +The tree-wide crash-to-REPL experience (``debug_mode=True`` plus +``await tractor.pause()``) requires a **real tty**, and pytest's +default output capturing swallows exactly that. When you want to +interact with the REPL from inside a test run, disable capture:: + + uv run pytest tests/test_foo.py -x -s + +(``-s`` is shorthand for ``--capture=no``.) + +Tests should request the ``debug_mode`` fixture (driven by the +``--tpdb`` flag) rather than hard-coding it, so that normal CI +runs stay non-interactive. + +For *automated* REPL interaction — asserting on prompt output, +sending debugger commands — you can't just turn capture off; +instead do what ``tests/devx/`` does: drive a child Python program +through pexpect_ on a real pseudo-tty and pattern-match the +``(Pdb+)`` prompts. See ``tests/devx/test_debugger.py`` for many +worked patterns. + +Examples *are* tests +-------------------- + +Every script under ``examples/`` is run as a subprocess by +``tests/test_docs_examples.py``; since these docs +``literalinclude`` those same scripts, the code you read here is +CI-verified on every push and can never silently rot B) + +Conventions when adding a new example: + +- make it a standalone runnable script with the usual guard:: + + if __name__ == '__main__': + trio.run(main) + +- it must exit cleanly (returncode ``0``) within the per-example + timeout (~16s locally, with headroom auto-added in CI and under + cpu-freq scaling) — keep sleeps short, + +- any stderr line containing ``Error`` fails the test, so silence + or assert-around expected error output, + +- don't crank ``tractor`` logging inside an example: subprocess + pipe **backpressure can deadlock** the run (ask us how we + know..), + +- filenames starting with ``_`` are skipped (the WIP convention), + as are the special subdirs (``debugging/``, ``integration/``, + ``advanced_faults/``, ``trio/``) which are driven by their own + dedicated suites instead. + +Drop your script in, run the example suite, profit:: + + uv run pytest tests/test_docs_examples.py -x + +Zombie cleanup +-------------- + +First, the contract: ``tractor`` **always** reaps its children — +if you can create a zombie process (without resorting to +untrappable signals) it **is a bug**, please report it! + +That said, while hacking on the *runtime itself* you can +definitely wedge things — a ``SIGKILL``-ed pytest, a half-broken +spawn backend — and strand subactor procs plus their shm segments +and UDS socket files. The repo ships a dedicated cleanup tool:: + + uv run scripts/tractor-reap --shm --uds + +It's SC-polite even as a reaper: matched processes get ``SIGINT`` +first with a bounded grace window — so actor runtimes can run +their ``trio`` teardown paths — escalating to ``SIGKILL`` only as +a last resort. The ``--shm`` sweep unlinks ``/dev/shm/`` segments +that no live process has open (it leans on psutil_, already in +your dev venv, to check live mappings and fds) and ``--uds`` +clears socket files whose binder pid is dead. + +Testing your own ``tractor`` app +-------------------------------- + +The same plugin the suite uses ships in the package, so your +project can load it too:: + + [tool.pytest.ini_options] + addopts = ['-p tractor._testing.pytest'] + +That buys you the CLI flags above plus a set of fixtures — +``loglevel``, ``debug_mode``, ``reg_addr`` (a session-unique +registrar address so concurrent runs and other live ``tractor`` +apps on the host can't cross-talk) — and the ``@tractor_test`` +decorator: + +.. code:: python + + import tractor + from tractor._testing import tractor_test + + @tractor_test + async def test_my_service( + reg_addr: tuple, + loglevel: str, + ): + # already inside a root actor's trio task! + async with tractor.open_nursery() as an: + ... + +The decorator boots a root actor around your (async) test fn, +wires any of the special fixtures you declare (``reg_addr``, +``loglevel``, ``start_method``, ``debug_mode``) into +``open_root_actor()``, and runs the body as the root-most task +under a wall-clock ``trio.fail_after()`` guard. + +General advice that has served this suite well: + +- bound waits with ``trio.fail_after()`` *inside* tests; global + pytest timeout plugins interact badly with multi-process + ``trio`` teardown, + +- use the ``reg_addr`` fixture (or otherwise randomize your + registry addrs) so leftover registrars from prior runs can't + contaminate lookups, + +- assert on **structured outcomes** — eg. + ``RemoteActorError.boxed_type`` or + ``ContextCancelled.canceller`` — not on log text. + +.. note:: + ``tractor._testing`` is still an underscore-internal namespace: + shipped and handy, but its API may shift between alpha + releases. + +(This page exists thanks to the ask in `#126`_.) + +.. seealso:: + - :doc:`/guide/discovery` — how registrar wiring (the thing + ``reg_addr`` randomizes) works in the runtime proper. + - :doc:`/project/dev-tips` — contributor-oriented extras: + releases, log-system tracing, tree-monitoring recipes. + +.. _uv: https://docs.astral.sh/uv/ +.. _pexpect: https://pexpect.readthedocs.io/en/stable/ +.. _psutil: https://psutil.readthedocs.io/en/latest/ +.. _#126: https://github.com/goodboy/tractor/issues/126 diff --git a/docs/index.rst b/docs/index.rst index 4ff198f5..98759aea 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,612 +1,154 @@ -.. tractor documentation master file, created by - sphinx-quickstart on Sun Feb 9 22:26:51 2020. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. image:: _static/tractor_logo_side.svg + :class: hero-logo + :align: center + :alt: tractor -``tractor`` -=========== +tractor +======= +**distributed structured concurrency**: a +multi-processing runtime built on (and shaped +entirely like) trio_. -A `structured concurrent`_, async-native "`actor model`_" built on trio_ and multiprocessing_. +``tractor`` provides parallelism via ``trio`` +*"actors"*: independent Python **processes** (ie. +*non-shared-memory threads*) each running a ``trio`` +task tree, all composed into a *distributed +supervision tree* with end-to-end `structured +concurrency`_ (SC) — spawning, cancellation, error +propagation and teardown that work **across +processes** (and hosts) exactly the way they work +across tasks. + +.. margin:: tl;dr + + It's **just** ``trio``, but with nurseries that + spawn *processes* and streams that cross them. If + you can read a ``trio`` program you can read a + ``tractor`` one — that's the whole pitch. + +Sixty seconds of why +-------------------- +Spawn one actor per core, crash the root on purpose, +and watch the runtime contain the blast: errors +propagate, *every* child is reaped, zero zombies — +guaranteed (it's a bug otherwise). + +.. literalinclude:: ../examples/parallelism/we_are_processes.py + :caption: examples/parallelism/we_are_processes.py + :language: python + +Like every snippet in these docs this file lives in +the repo's ``examples/`` dir and runs under CI — docs +code that can't rot. + +Dig in +------ +.. grid:: 1 2 2 3 + :gutter: 3 + + .. grid-item-card:: Get started + :link: start/index + :link-type: doc + + Install + your first actor tree in ~20 lines; + causality, daemons and the trynamic scene. + + .. grid-item-card:: The big ideas + :link: explain/sc-distributed + :link-type: doc + + SC across processes, distilled — then the + runtime architecture under it. + + .. grid-item-card:: Debug like a local + :link: guide/debugging + :link-type: doc + + ``await tractor.pause()`` anywhere in the + tree: one terminal, every process, zero + socket-juggling. + + .. grid-item-card:: Streaming + contexts + :link: guide/context + :link-type: doc + + Bidirectional, cancellation-safe msg streams + between any two actors. + + .. grid-item-card:: Guides + :link: guide/index + :link-type: doc + + RPC, supervision, clustering, "infected + asyncio", typed msging + more. + + .. grid-item-card:: API reference + :link: api/index + :link-type: doc + + The curated public surface; everything + importable from ``tractor``. + +Features +-------- +- **It's just a** ``trio`` **API** — same nursery + discipline, same cancellation semantics, one level + up the process tree. +- *Infinitely nestable* process trees: sub-actors can + spawn sub-actors, supervision stays transitive. +- A "native UX" **multi-process debugger REPL**: + built on pdbp_ with tree-wide tty locking (see + :doc:`guide/debugging`). +- Built-in, cancellation-safe **bidirectional + streaming** via a `cheap or nasty`_ `(un)protocol`_. +- **Typed IPC**: `msgspec`_-backed wire msgs with + optional per-dialog payload specs + (:doc:`guide/msging`). +- Swappable process-spawn backends + modular IPC + transports (TCP today, UDS on same-host, more + planned). +- Optionally distributed_: the same APIs work over + multiple hosts as on multiple cores. +- "**Infected** ``asyncio``" mode: SC-supervise + ``asyncio`` tasks from ``trio`` + (:doc:`guide/asyncio`). +- ``trio`` extension goodies via ``tractor.trionics`` + (acm gathering, single-resource caching, broadcast + channels). + +Where do i start!? +------------------ +The first step to grok ``tractor`` is to get an +intermediate knowledge of ``trio`` and **structured +concurrency** B) + +Some great places to start are, + +- the seminal `blog post`_, +- obviously the `trio docs`_, +- wikipedia's nascent SC_ page, +- the fancy diagrams @ libdill-docs_, + +then come back and hit :doc:`start/quickstart`. .. toctree:: - :maxdepth: 1 - :caption: Contents: + :hidden: + :maxdepth: 2 + + Get started + Big ideas + Guides + API + Project -.. _actor model: https://en.wikipedia.org/wiki/Actor_model .. _trio: https://github.com/python-trio/trio -.. _multiprocessing: https://en.wikipedia.org/wiki/Multiprocessing -.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles -.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich -.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 - - -``tractor`` is an attempt to bring trionic_ `structured concurrency`_ to -distributed multi-core Python; it aims to be the Python multi-processing -framework *you always wanted*. - -``tractor`` lets you spawn ``trio`` *"actors"*: processes which each run -a ``trio`` scheduled task tree (also known as an `async sandwich`_). -*Actors* communicate by exchanging asynchronous messages_ and avoid -sharing any state. This model allows for highly distributed software -architecture which works just as well on multiple cores as it does over -many hosts. - -The first step to grok ``tractor`` is to get the basics of ``trio`` down. -A great place to start is the `trio docs`_ and this `blog post`_. - -.. _messages: https://en.wikipedia.org/wiki/Message_passing -.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ - - -Install -------- -No PyPi release yet! - -:: - - pip install git+git://github.com/goodboy/tractor.git - - -Feel like saying hi? --------------------- -This project is very much coupled to the ongoing development of -``trio`` (i.e. ``tractor`` gets all its ideas from that brilliant -community). If you want to help, have suggestions or just want to -say hi, please feel free to ping me on the `trio gitter channel`_! - -.. _trio gitter channel: https://gitter.im/python-trio/general - - - -Philosophy ----------- -Our tenets non-comprehensively include: - -- strict adherence to the `concept-in-progress`_ of *structured concurrency* -- no spawning of processes *willy-nilly*; causality_ is paramount! -- (remote) errors `always propagate`_ back to the parent supervisor -- verbatim support for ``trio``'s cancellation_ system -- `shared nothing architecture`_ -- no use of *proxy* objects or shared references between processes -- an immersive debugging experience -- anti-fragility through `chaos engineering`_ - -``tractor`` is an actor-model-*like* system in the sense that it adheres -to the `3 axioms`_ but does not (yet) fulfil all "unrequirements_" in -practise. It is an experiment in applying `structured concurrency`_ -constraints on a parallel processing system where multiple Python -processes exist over many hosts but no process can outlive its parent. -In `erlang` parlance, it is an architecture where every process has -a mandatory supervisor enforced by the type system. The API design is -almost exclusively inspired by trio_'s concepts and primitives (though -we often lag a little). As a distributed computing system `tractor` -attempts to place sophistication at the correct layer such that -concurrency primitives are powerful yet simple, making it easy to build -complex systems (you can build a "worker pool" architecture but it's -definitely not required). There is first class support for inter-actor -streaming using `async generators`_ and ongoing work toward a functional -reactive style for IPC. - -.. warning:: ``tractor`` is in alpha-alpha and is expected to change rapidly! - Expect nothing to be set in stone. Your ideas about where it should go - are greatly appreciated! - -.. _concept-in-progress: https://trio.discourse.group/t/structured-concurrency-kickoff/55 -.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts -.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony -.. _async generators: https://www.python.org/dev/peps/pep-0525/ -.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate -.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker -.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture -.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts -.. _channels: https://en.wikipedia.org/wiki/Channel_(programming) -.. _chaos engineering: http://principlesofchaos.org/ - - -Examples --------- -Note, if you are on Windows please be sure to see the :ref:`gotchas -` section before trying these. - - -A trynamic first scene -********************** -Let's direct a couple *actors* and have them run their lines for -the hip new film we're shooting: - -.. literalinclude:: ../examples/a_trynamic_first_scene.py - -We spawn two *actors*, *donny* and *gretchen*. -Each actor starts up and executes their *main task* defined by an -async function, ``say_hello()``. The function instructs each actor -to find their partner and say hello by calling their partner's -``hi()`` function using something called a *portal*. Each actor -receives a response and relays that back to the parent actor (in -this case our "director" executing ``main()``). - - -Actor spawning and causality -**************************** -``tractor`` tries to take ``trio``'s concept of causal task lifetimes -to multi-process land. Accordingly, ``tractor``'s *actor nursery* behaves -similar to ``trio``'s nursery_. That is, ``tractor.open_nursery()`` -opens an ``ActorNursery`` which **must** wait on spawned *actors* to complete -(or error) in the same causal_ way ``trio`` waits on spawned subtasks. -This includes errors from any one actor causing all other actors -spawned by the same nursery to be cancelled_. - -To spawn an actor and run a function in it, open a *nursery block* -and use the ``run_in_actor()`` method: - -.. literalinclude:: ../examples/actor_spawning_and_causality.py - -What's going on? - -- an initial *actor* is started with ``trio.run()`` and told to execute - its main task_: ``main()`` - -- inside ``main()`` an actor is *spawned* using an ``ActorNusery`` and is told - to run a single function: ``cellar_door()`` - -- a ``portal`` instance (we'll get to what it is shortly) - returned from ``nursery.run_in_actor()`` is used to communicate with - the newly spawned *sub-actor* - -- the second actor, *some_linguist*, in a new *process* running a new ``trio`` task_ - then executes ``cellar_door()`` and returns its result over a *channel* back - to the parent actor - -- the parent actor retrieves the subactor's *final result* using ``portal.result()`` - much like you'd expect from a future_. - -This ``run_in_actor()`` API should look very familiar to users of -``asyncio``'s `run_in_executor()`_ which uses a ``concurrent.futures`` Executor_. - -Since you might also want to spawn long running *worker* or *daemon* -actors, each actor's *lifetime* can be determined based on the spawn -method: - -- if the actor is spawned using ``run_in_actor()`` it terminates when - its *main* task completes (i.e. when the (async) function submitted - to it *returns*). The ``with tractor.open_nursery()`` exits only once - all actors' main function/task complete (just like the nursery_ in ``trio``) - -- actors can be spawned to *live forever* using the ``start_actor()`` - method and act like an RPC daemon that runs indefinitely (the - ``with tractor.open_nursery()`` won't exit) until cancelled_ - -Here is a similar example using the latter method: - -.. literalinclude:: ../examples/actor_spawning_and_causality_with_daemon.py - -The ``enable_modules`` `kwarg` above is a list of module path -strings that will be loaded and made accessible for execution in the -remote actor through a call to ``Portal.run()``. For now this is -a simple mechanism to restrict the functionality of the remote -(and possibly daemonized) actor and uses Python's module system to -limit the allowed remote function namespace(s). - -``tractor`` is opinionated about the underlying threading model used for -each *actor*. Since Python has a GIL and an actor model by definition -shares no state between actors, it fits naturally to use a multiprocessing_ -``Process``. This allows ``tractor`` programs to leverage not only multi-core -hardware but also distribute over many hardware hosts (each *actor* can talk -to all others with ease over standard network protocols). - -.. _task: https://trio.readthedocs.io/en/latest/reference-core.html#tasks-let-you-do-multiple-things-at-once -.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning -.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality -.. _cancelled: https://trio.readthedocs.io/en/latest/reference-core.html#child-tasks-and-cancellation -.. _run_in_executor(): https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor -.. _Executor: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor - - -Cancellation -************ -``tractor`` supports ``trio``'s cancellation_ system verbatim. -Cancelling a nursery block cancels all actors spawned by it. -Eventually ``tractor`` plans to support different `supervision strategies`_ like ``erlang``. - -.. _supervision strategies: http://erlang.org/doc/man/supervisor.html#sup_flags - - -Remote error propagation -************************ -Any task invoked in a remote actor should ship any error(s) back to the calling -actor where it is raised and expected to be dealt with. This way remote actors -are never cancelled unless explicitly asked or there's a bug in ``tractor`` itself. - -.. literalinclude:: ../examples/remote_error_propagation.py - - -You'll notice the nursery cancellation conducts a *one-cancels-all* -supervisory strategy `exactly like trio`_. The plan is to add more -`erlang strategies`_ in the near future by allowing nurseries to accept -a ``Supervisor`` type. - -.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics -.. _erlang strategies: http://learnyousomeerlang.com/supervisors - - -IPC using *portals* -******************* -``tractor`` introduces the concept of a *portal* which is an API -borrowed_ from ``trio``. A portal may seem similar to the idea of -a RPC future_ except a *portal* allows invoking remote *async* functions and -generators and intermittently blocking to receive responses. This allows -for fully async-native IPC between actors. - -When you invoke another actor's routines using a *portal* it looks as though -it was called locally in the current actor. So when you see a call to -``await portal.run()`` what you get back is what you'd expect -to if you'd called the function directly in-process. This approach avoids -the need to add any special RPC *proxy* objects to the library by instead just -relying on the built-in (async) function calling semantics and protocols of Python. - -Depending on the function type ``Portal.run()`` tries to -correctly interface exactly like a local version of the remote -built-in Python *function type*. Currently async functions, generators, -and regular functions are supported. Inspiration for this API comes -`remote function execution`_ but without the client code being -concerned about the underlying channels_ system or shipping code -over the network. - -This *portal* approach turns out to be paricularly exciting with the -introduction of `asynchronous generators`_ in Python 3.6! It means that -actors can compose nicely in a data streaming pipeline. - -.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics - -Streaming -********* -By now you've figured out that ``tractor`` lets you spawn process based -*actors* that can invoke cross-process (async) functions and all with -structured concurrency built in. But the **real cool stuff** is the -native support for cross-process *streaming*. - - -Asynchronous generators -+++++++++++++++++++++++ -The default streaming function is simply an async generator definition. -Every value *yielded* from the generator is delivered to the calling -portal exactly like if you had invoked the function in-process meaning -you can ``async for`` to receive each value on the calling side. - -As an example here's a parent actor that streams for 1 second from a -spawned subactor: - -.. literalinclude:: ../examples/asynchronous_generators.py - -By default async generator functions are treated as inter-actor -*streams* when invoked via a portal (how else could you really interface -with them anyway) so no special syntax to denote the streaming *service* -is necessary. - - -Channels and Contexts -+++++++++++++++++++++ -If you aren't fond of having to write an async generator to stream data -between actors (or need something more flexible) you can instead use -a ``Context``. A context wraps an actor-local spawned task and -a ``Channel`` so that tasks executing across multiple processes can -stream data to one another using a low level, request oriented API. - -A ``Channel`` wraps an underlying *transport* and *interchange* format -to enable *inter-actor-communication*. In its present state ``tractor`` -uses TCP and msgpack_. - -As an example if you wanted to create a streaming server without writing -an async generator that *yields* values you instead define a decorated -async function: - -.. code:: python - - @tractor.stream - async def streamer(ctx: tractor.Context, rate: int = 2) -> None: - """A simple web response streaming server. - """ - while True: - val = await web_request('http://data.feed.com') - - # this is the same as ``yield`` in the async gen case - await ctx.send_yield(val) - - await trio.sleep(1 / rate) - - -You must decorate the function with ``@tractor.stream`` and declare -a ``ctx`` argument as the first in your function signature and then -``tractor`` will treat the async function like an async generator - as -a stream from the calling/client side. - -This turns out to be handy particularly if you have multiple tasks -pushing responses concurrently: - -.. code:: python - - async def streamer( - ctx: tractor.Context, - rate: int = 2 - ) -> None: - """A simple web response streaming server. - """ - while True: - val = await web_request(url) - - # this is the same as ``yield`` in the async gen case - await ctx.send_yield(val) - - await trio.sleep(1 / rate) - - - @tractor.stream - async def stream_multiple_sources( - ctx: tractor.Context, - sources: List[str] - ) -> None: - async with trio.open_nursery() as n: - for url in sources: - n.start_soon(streamer, ctx, url) - - -The context notion comes from the context_ in nanomsg_. - -.. _context: https://nanomsg.github.io/nng/man/tip/nng_ctx.5 -.. _msgpack: https://en.wikipedia.org/wiki/MessagePack - - - -A full fledged streaming service -++++++++++++++++++++++++++++++++ -Alright, let's get fancy. - -Say you wanted to spawn two actors which each pull data feeds from -two different sources (and wanted this work spread across 2 cpus). -You also want to aggregate these feeds, do some processing on them and then -deliver the final result stream to a client (or in this case parent) actor -and print the results to your screen: - -.. literalinclude:: ../examples/full_fledged_streaming_service.py - -Here there's four actors running in separate processes (using all the -cores on you machine). Two are streaming by *yielding* values from the -``stream_data()`` async generator, one is aggregating values from -those two in ``aggregate()`` (also an async generator) and shipping the -single stream of unique values up the parent actor (the ``'MainProcess'`` -as ``multiprocessing`` calls it) which is running ``main()``. - -.. _future: https://en.wikipedia.org/wiki/Futures_and_promises -.. _borrowed: - https://trio.readthedocs.io/en/latest/reference-core.html#getting-back-into-the-trio-thread-from-another-thread -.. _asynchronous generators: https://www.python.org/dev/peps/pep-0525/ -.. _remote function execution: https://codespeak.net/execnet/example/test_info.html#remote-exec-a-function-avoiding-inlined-source-part-i - - -Actor local (aka *process global*) variables -******************************************** -Although ``tractor`` uses a *shared-nothing* architecture between -processes you can of course share state between tasks running *within* -an actor (since a `trio.run()` runtime is single threaded). ``trio`` -tasks spawned via multiple RPC calls to an actor can modify -*process-global-state* defined using Python module attributes: - -.. code:: python - - - # a per process cache - _actor_cache: dict[str, bool] = {} - - - def ping_endpoints(endpoints: List[str]): - """Start a polling process which runs completely separate - from our root actor/process. - - """ - - # This runs in a new process so no changes # will propagate - # back to the parent actor - while True: - - for ep in endpoints: - status = await check_endpoint_is_up(ep) - _actor_cache[ep] = status - - await trio.sleep(0.5) - - - async def get_alive_endpoints(): - - nonlocal _actor_cache - - return {key for key, value in _actor_cache.items() if value} - - - async def main(): - - async with tractor.open_nursery() as n: - - portal = await n.run_in_actor(ping_endpoints) - - # print the alive endpoints after 3 seconds - await trio.sleep(3) - - # this is submitted to be run in our "ping_endpoints" actor - print(await portal.run(get_alive_endpoints)) - - -You can pass any kind of (`msgpack`) serializable data between actors using -function call semantics but building out a state sharing system per-actor -is totally up to you. - - -Service Discovery -***************** -Though it will be built out much more in the near future, ``tractor`` -currently keeps track of actors by ``(name: str, id: str)`` using a -special actor called the *arbiter*. Currently the *arbiter* must exist -on a host (or it will be created if one can't be found) and keeps a -simple ``dict`` of actor names to sockets for discovery by other actors. -Obviously this can be made more sophisticated (help me with it!) but for -now it does the trick. - -To find the arbiter from the current actor use the ``get_arbiter()`` function and to -find an actor's socket address by name use the ``find_actor()`` function: - -.. literalinclude:: ../examples/service_discovery.py - -The ``name`` value you should pass to ``find_actor()`` is the one you passed as the -*first* argument to either ``trio.run()`` or ``ActorNursery.start_actor()``. - - -Running actors standalone -************************* -You don't have to spawn any actors using ``open_nursery()`` if you just -want to run a single actor that connects to an existing cluster. -All the comms and arbiter registration stuff still works. This can -somtimes turn out being handy when debugging mult-process apps when you -need to hop into a debugger. You just need to pass the existing -*arbiter*'s socket address you'd like to connect to: - -.. code:: python - - import trio - import tractor - - async def main(): - - async with tractor.open_root_actor( - arbiter_addr=('192.168.0.10', 1616) - ): - await trio.sleep_forever() - - trio.run(main) - - -Choosing a process spawning backend -*********************************** -``tractor`` is architected to support multiple actor (sub-process) -spawning backends. Specific defaults are chosen based on your system -but you can also explicitly select a backend of choice at startup -via a ``start_method`` kwarg to ``tractor.open_nursery()``. - -Currently the options available are: - -- ``trio``: a ``trio``-native spawner which is an async wrapper around ``subprocess`` -- ``spawn``: one of the stdlib's ``multiprocessing`` `start methods`_ -- ``forkserver``: a faster ``multiprocessing`` variant that is Unix only - -.. _start methods: https://docs.python.org/3.8/library/multiprocessing.html#contexts-and-start-methods - - -``trio`` -++++++++ -The ``trio`` backend offers a lightweight async wrapper around ``subprocess`` from the standard library and takes advantage of the ``trio.`` `open_process`_ API. - -.. _open_process: https://trio.readthedocs.io/en/stable/reference-io.html#spawning-subprocesses - - -``multiprocessing`` -+++++++++++++++++++ -There is support for the stdlib's ``multiprocessing`` `start methods`_. -Note that on Windows *spawn* it the only supported method and on \*nix -systems *forkserver* is the best method for speed but has the caveat -that it will break easily (hangs due to broken pipes) if spawning actors -using nested nurseries. - -In general, the ``multiprocessing`` backend **has not proven reliable** -for handling errors from actors more then 2 nurseries *deep* (see `#89`_). -If you for some reason need this consider sticking with alternative -backends. - -.. _#89: https://github.com/goodboy/tractor/issues/89 - -.. _windowsgotchas: - -Windows "gotchas" -^^^^^^^^^^^^^^^^^ -On Windows (which requires the use of the stdlib's `multiprocessing` -package) there are some gotchas. Namely, the need for calling -`freeze_support()`_ inside the ``__main__`` context. Additionally you -may need place you `tractor` program entry point in a seperate -`__main__.py` module in your package in order to avoid an error like the -following :: - - Traceback (most recent call last): - File "C:\ProgramData\Miniconda3\envs\tractor19030601\lib\site-packages\tractor\_actor.py", line 234, in _get_rpc_func - return getattr(self._mods[ns], funcname) - KeyError: '__mp_main__' - - -To avoid this, the following is the **only code** that should be in your -main python module of the program: - -.. code:: python - - # application/__main__.py - import trio - import tractor - import multiprocessing - from . import tractor_app - - if __name__ == '__main__': - multiprocessing.freeze_support() - trio.run(tractor_app.main) - -And execute as:: - - python -m application - - -As an example we use the following code to test all documented examples -in the test suite on windows: - -.. literalinclude:: ../examples/__main__.py - -See `#61`_ and `#79`_ for further details. - -.. _freeze_support(): https://docs.python.org/3/library/multiprocessing.html#multiprocessing.freeze_support -.. _#61: https://github.com/goodboy/tractor/pull/61#issuecomment-470053512 -.. _#79: https://github.com/goodboy/tractor/pull/79 - - -Enabling logging -**************** -Considering how complicated distributed software can become it helps to know -what exactly it's doing (even at the lowest levels). Luckily ``tractor`` has -tons of logging throughout the core. ``tractor`` isn't opinionated on -how you use this information and users are expected to consume log messages in -whichever way is appropriate for the system at hand. That being said, when hacking -on ``tractor`` there is a prettified console formatter which you can enable to -see what the heck is going on. Just put the following somewhere in your code: - -.. code:: python - - from tractor.log import get_console_log - log = get_console_log('trace') - - -What the future holds ---------------------- -Stuff I'd like to see ``tractor`` do real soon: - -- TLS_, duh. -- erlang-like supervisors_ -- native support for `nanomsg`_ as a channel transport -- native `gossip protocol`_ support for service discovery and arbiter election -- a distributed log ledger for tracking cluster behaviour -- a slick multi-process aware debugger much like in celery_ - but with better `pdb++`_ support -- an extensive `chaos engineering`_ test suite -- support for reactive programming primitives and native support for asyncitertools_ like libs -- introduction of a `capability-based security`_ model - -.. _TLS: https://trio.readthedocs.io/en/latest/reference-io.html#ssl-tls-support -.. _supervisors: https://github.com/goodboy/tractor/issues/22 -.. _nanomsg: https://nanomsg.github.io/nng/index.html -.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol -.. _celery: http://docs.celeryproject.org/en/latest/userguide/debugging.html -.. _asyncitertools: https://github.com/vodik/asyncitertools -.. _pdb++: https://github.com/antocuni/pdb -.. _capability-based security: https://en.wikipedia.org/wiki/Capability-based_security +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html +.. _pdbp: https://github.com/mdmintz/pdbp +.. _msgspec: https://jcristharif.com/msgspec/ +.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern +.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols +.. _distributed: https://en.wikipedia.org/wiki/Distributed_computing diff --git a/docs/project/changelog.rst b/docs/project/changelog.rst new file mode 100644 index 00000000..bb01c125 --- /dev/null +++ b/docs/project/changelog.rst @@ -0,0 +1 @@ +.. include:: ../../NEWS.rst diff --git a/docs/project/dev-tips.rst b/docs/project/dev-tips.rst new file mode 100644 index 00000000..531a28fc --- /dev/null +++ b/docs/project/dev-tips.rst @@ -0,0 +1,133 @@ +Hot tips for ``tractor`` hackers +================================ + +This is a (perpetually WIP) guide for newcomers to the project, +mostly to do with dev, testing, CI and release gotchas, reminders +and best practises. + +``tractor`` is a fairly novel project compared to most since it is +effectively a new way of doing distributed computing in Python and +is much closer to working with an "application level runtime" +(like erlang OTP or scala's akka project) than it is a traditional +Python library. As such, having an arsenal of tools and recipes +for figuring out the right way to debug problems when they do +arise is somewhat of a necessity. + +Making a release +---------------- + +Nothing fancy: the traditional PyPA flow on the hatchling_ build +backend, with uv_ doing the driving and towncrier_ generating the +changelog. + +1. collect news fragments: user-facing changes should land with a + small ``.rst`` snippet under ``nooz/`` (see ``nooz/HOWTO.rst``; + fragment types are ``feature``, ``bugfix``, ``doc`` and + ``trivial``), + +2. render them into ``NEWS.rst``:: + + uvx towncrier build --version + +3. build and upload (testpypi first if you're being careful):: + + uv build + uvx twine upload -r testpypi dist/* + uvx twine upload dist/* + +How you organize built artifacts under ``dist/`` locally (per +release sub-dirs and such) is entirely up to you. + +Keep in mind that PyPi releases tend to lag the ``main`` branch +since we develop in the open — ``main`` is usually the thing to +run when you want the latest. + +Debugging and monitoring actor trees +------------------------------------ + +Your "what is my tree doing right now?" toolbox, in escalation +order: + +**Live process-tree view** — keep a ``watch``-ed ``pstree`` +running in a side terminal; actor procs are recognizable by their +``_subactor[@]`` process titles. The exact +one-liners (plus the ``pgrep`` marker recipes) live in +:doc:`/guide/testing`. + +**SIGUSR1 task-tree dumps** — boot any tree with +``enable_stack_on_sig=True`` (or export +``TRACTOR_ENABLE_STACKSCOPE=1``) and every actor installs a +stackscope_ signal handler. Then from any shell:: + + # dump every actor's live trio task tree: + pkill --signal SIGUSR1 -f + + # or for a single process: + kill -SIGUSR1 $(pgrep -f ) + +Each dump is also tee'd (append-mode) to +``/tmp/tractor-stackscope-.log`` so you still get output +under pytest capture or in CI. This works *without* debug-mode +being enabled — it's the lightest-weight hang-investigation tool +in the box. + +**The built-in multi-process debugger** — ``debug_mode=True`` +plus :func:`tractor.pause` and friends: the heavyweight champ for +interactive, REPL-driven inspection of a whole tree (including +crash handling). Remember pytest capture interplay — see +:doc:`/guide/testing`. + +**Post-mortem zombie sweeps** — ``scripts/tractor-reap`` for the +(should-be-rare!) cases where hacking on the runtime itself wedges +a tree: a SIGINT-first, structured concurrency (SC) polite +escalation, plus ``--shm`` and ``--uds`` leaked-resource sweeps. + +Using the log system to trace ``trio`` task flow +------------------------------------------------ + +The logging system is oriented around the **stack "layers" of the +runtime**, letting you trace logical abstraction layers in the +code — errors, cancellation, IPC and streaming, the low level +transport and wire protocols — independently of one another. + +Concretely, ``tractor.log.get_logger()`` returns a +``StackLevelAdapter`` sporting extra level-methods beyond the +stdlib set, including: + +- ``.cancel()`` — cancellation-machinery flow, + +- ``.runtime()`` — actor-runtime lifecycle chatter, + +- ``.devx()`` — debugger/devx tooling internals, + +- ``.transport()`` — wire-level msging events. + +To get console output at any level from your own code:: + + from tractor.log import get_console_log + get_console_log('cancel') + +or, runtime-wide without touching code, just export +``TRACTOR_LOGLEVEL=cancel`` (the env-var wins over caller-passed +levels; great for test runs). + +When you want only *one subsystem* cranked, the suite's ``--tl`` +flag (and ``tractor.log.apply_logspec()``) accept a per-sublogger +spec:: + + uv run pytest tests/... --tl 'devx:runtime,trionics:cancel' + +Every record's header includes the emitting actor and task names, +so cross-process flows can be stitched back together by eyeball +(or grep). + +.. seealso:: + - :doc:`/guide/testing` — running the suite, watching trees + live, examples-as-tests conventions and the zombie-reaper. + - :doc:`/guide/discovery` — the registrar mechanics you'll + bump into when running multiple trees on one host. + +.. _hatchling: https://hatch.pypa.io/latest/ +.. _uv: https://docs.astral.sh/uv/ +.. _towncrier: https://towncrier.readthedocs.io/en/stable/ +.. _stackscope: https://github.com/oremanj/stackscope diff --git a/docs/project/index.rst b/docs/project/index.rst new file mode 100644 index 00000000..07fc1f7d --- /dev/null +++ b/docs/project/index.rst @@ -0,0 +1,60 @@ +Project +======= +Everything meta: where the project's been (the +:doc:`changelog`), where it's going (the roadmap below), +how to hack on it (:doc:`dev-tips` and +:doc:`/guide/testing`) and where to find the humans. + +.. toctree:: + :maxdepth: 1 + + changelog + dev-tips + +What the future holds +--------------------- +Help us push toward the future of distributed Python! +Planned (or dreamed of) work non-comprehensively +includes, + +- Erlang-style supervisors via composed context + managers (see `#22`_), +- typed capability-based (dialog) protocols, ie. + evolving our `msg-spec`_ system into per-dialog + contracts (see `#196`_ with draft work in `#311`_), +- a higher level "service manager" API for daemon + lifetime mgmt over actor trees (in the works on an + experimental branch as ``tractor.hilevel``), +- richer `discovery`_ via gossip and/or + `rendezvous protocol`_ approaches (today's registrar + is intentionally naive), +- more IPC transports: the current ``tcp`` | ``uds`` + pair wants friends (QUIC, shm-ring-buffers, RUDP, + wireguard tunnels), +- an extensive `chaos engineering`_ test suite, +- a respawn-from-REPL system for crashed (sub-)actors. + +Feel like saying hi? +-------------------- +This project is very much coupled to the ongoing +development of ``trio`` (i.e. ``tractor`` gets most of +its ideas from that brilliant community). If you want +to help, have suggestions or just want to say hi, +please feel free to reach us in our `matrix channel`_. +If matrix seems too hip, we're also mostly all in the +`trio gitter channel`_! + +Contributions of all kinds welcome: docs, examples, +bug reports, new transports, supervisor strategies, +philosophical debates about what an "actor model" +really is B) + +.. _#22: https://github.com/goodboy/tractor/issues/22 +.. _#196: https://github.com/goodboy/tractor/issues/196 +.. _#311: https://github.com/goodboy/tractor/pull/311 +.. _msg-spec: https://jcristharif.com/msgspec/ +.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery +.. _rendezvous protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol +.. _chaos engineering: https://principlesofchaos.org/ +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _trio gitter channel: https://gitter.im/python-trio/general diff --git a/docs/start/index.rst b/docs/start/index.rst new file mode 100644 index 00000000..9e868270 --- /dev/null +++ b/docs/start/index.rst @@ -0,0 +1,36 @@ +Getting started +=============== +Welcome aboard B) + +Real talk before any code: the first step to grok ``tractor`` is +to get an intermediate knowledge of ``trio`` and **structured +concurrency** (SC). ``tractor`` **is just** ``trio`` - but with +nurseries for process management and cancel-able streaming IPC - +so every rule you already know about task lifetimes, cancellation +and error propagation keeps holding, just now across process (and +host!) boundaries. Some great places to start are, + +- the seminal `blog post`_, +- obviously the `trio docs`_, +- wikipedia's nascent SC_ page, +- the fancy diagrams @ libdill-docs_. + +Once you've taken in (some of) the canon, get installed and go +spawn your first actor tree: + +.. toctree:: + :maxdepth: 1 + + install + quickstart + +.. seealso:: + + Already installed and itching? Jump straight to + :doc:`/start/quickstart`; once you're through the on-ramp the + :doc:`guide pages ` take each subsystem deeper. + +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency +.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html diff --git a/docs/start/install.rst b/docs/start/install.rst new file mode 100644 index 00000000..7fc0a852 --- /dev/null +++ b/docs/start/install.rst @@ -0,0 +1,75 @@ +Install +======= +``tractor`` is still in an *alpha-near-beta-stage* for many of +its subsystems, however we are very close to having a stable +lowlevel runtime and API. Expect the occasional rough edge (and +feel free to report it!). + +Supported platforms +------------------- +- **python**: ``>=3.13,<3.15`` - yes, we ride near the front of + the release train; 3.14 additionally unlocks the experimental + PEP-734 sub-interpreter spawn backend, +- **linux**: the primary development and CI platform, +- **macos**: officially supported, +- **windows**: currently *untested* - we disabled CI-testing on + windows a while back and haven't had the cycles to revive it. + It mostly worked historically; if you're a windows person we'd + love a hand getting it back in the test matrix. + +With ``uv`` (preferred) +----------------------- +We use the very hip uv_ for project mgmt and recommend you do +too. Add ``tractor`` to your project:: + + uv add tractor + +or, since the git ``main`` branch is often much further ahead +than any latest release (see the PyPi note below), track ``main`` +directly:: + + uv add "tractor @ git+https://github.com/goodboy/tractor.git" + +From PyPi +--------- +We ofc also offer "releases" on PyPi_:: + + pip install tractor + +Just note that **YMMV** since ``main`` is usually well ahead of +the latest published alpha; when in doubt go with a git install +as per above (or hack from source as per below). + +From source +----------- +To run the bundled examples in-tree, or to start hacking on the +code base, clone and sync a dev env:: + + git clone https://github.com/goodboy/tractor.git + cd tractor + uv sync --dev + uv run python examples/rpc_bidir_streaming.py + +Consider activating a virtual/project-env before starting to hack +on the code base:: + + # you could use plain ol' venvs + # https://docs.astral.sh/uv/pip/environments/ + uv venv tractor_py313 --python 3.13 + + # but @goodboy prefers the more explicit (and shell agnostic) + # https://docs.astral.sh/uv/configuration/environment/#uv_project_environment + UV_PROJECT_ENVIRONMENT="tractor_py313" + + # hint hint, enter @goodboy's fave shell B) + uv run --dev xonsh + +.. seealso:: + + All set? Go boot a process tree in :doc:`/start/quickstart`. + And if the install fights you, swing by the `matrix channel`_ + and we'll sort it out. + +.. _uv: https://docs.astral.sh/uv/ +.. _PyPi: https://pypi.org/project/tractor/ +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org diff --git a/docs/start/quickstart.rst b/docs/start/quickstart.rst new file mode 100644 index 00000000..75bbafb2 --- /dev/null +++ b/docs/start/quickstart.rst @@ -0,0 +1,264 @@ +Quickstart +========== +Time to spawn something B) + +If you take one thing from this page make it this: ``tractor`` +**is just** ``trio`` - but with nurseries for process management +and cancel-able streaming IPC. Every "*actor*" you'll meet below +is a plain Python **process** running its own ``trio.run()`` +scheduled task tree, linked back to its parent through an IPC +protocol which keeps the whole tree `structured concurrency`_ +(SC) compliant end-to-end. If you know your nursery_ semantics +you already know most of ``tractor``; we just stretch them across +the process boundary. + +.. d2:: diagrams/actor_tree.d2 + :alt: a supervision tree of actor processes + :margin: + :caption: every arrow is a parent which **must wait** on its kids + +Your first actor tree +--------------------- +``trio`` takes the hard-line position that a parent task **must +wait** on the children it spawns; causality_ is paramount! So +does ``tractor``, one abstraction layer up: +``tractor.open_nursery()`` yields an ``ActorNursery`` which +**must** wait on its spawned *subactors* to complete (or error) +before the ``async with`` block exits, in the same causal_ way a +``trio`` nursery waits on its subtasks. That includes any one +child's crash cancelling all of its siblings: *one-cancels-all* +supervision, `exactly like trio`_. + +Enough preamble, spawn a process: + +.. literalinclude:: ../../examples/actor_spawning_and_causality.py + :caption: examples/actor_spawning_and_causality.py + :language: python + +Run it:: + + $ python examples/actor_spawning_and_causality.py + Dang that's beautiful + +What's going on here? + +- ``trio.run(main)`` starts the **root actor**; the ``tractor`` + runtime boots *implicitly* inside ``tractor.open_nursery()`` + whenever it isn't already up. No special entrypoint, no + framework takeover - it's just a ``trio`` app, +- inside ``main()`` a *subactor* is spawned via + ``ActorNursery.run_in_actor()`` and told to run exactly one + function: ``cellar_door()``, +- you get back a ``Portal``: your handle for invoking tasks in + the new process's (separate!) memory domain. We lean on it + much harder in the next section, +- the subactor, *some_linguist*, boots a fresh ``trio.run()`` in + a **new process** and executes ``cellar_door()`` as its *main + task* (note the child proving it is *not* the root with + ``tractor.is_root_process()``), then ships the return value + back over IPC, +- the parent grabs that *final result* with + ``await portal.wait_for_result()``, much like you'd expect + from a "future" - except causality is preserved: the nursery + block only exits once the child is *done*, dead, and reaped. + +.. margin:: Just need a worker pool? + + If all you want is to throw *sync* functions at your cores, + also check out trio-parallel_. ``tractor`` is aimed at + structured, (possibly) distributed *trees* of cooperating + ``trio`` programs; a worker pool is a trivial special case. + +.. note:: + + ``run_in_actor()`` is the *convenience* wrapper: one-shot + spawn-run-reap semantics for when a subactor's entire job is + a single function call. The core primitives are + ``ActorNursery.start_actor()`` (next up) paired with + ``Portal.open_context()`` for full, SC-linked cross-actor + dialogs - see :doc:`/guide/context`. + +Daemon actors and RPC +--------------------- +A ``run_in_actor()``-spawned actor terminates when its main task +returns. But often you want long-lived *daemon* actors instead: +spawned once, then serving (allowlisted) RPC requests until told +otherwise. That's ``start_actor()``: + +.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py + :caption: examples/actor_spawning_and_causality_with_daemon.py + :language: python + +Two lifetime rules to internalize: + +- a ``run_in_actor()`` actor lives exactly as long as its main + task; the nursery waits for that function (and thus the + process) to complete before unblocking, +- a ``start_actor()`` actor *lives forever* - an RPC daemon the + nursery will happily wait on **indefinitely** - until some + task explicitly cancels it via ``Portal.cancel_actor()`` (as + above), or its parent nursery is cancelled wholesale. + +.. tip:: + + Want your *entire program* to just be a long-lived RPC + daemon? ``tractor.run_daemon()`` is the blocking shorthand: + it ``trio.run()``\s a root actor which serves requests until + cancelled. + +The ``enable_modules=[__name__]`` kwarg is the other thing to +notice: it lists the module paths the subactor will load and +*expose* for remote invocation. +``await portal.run(movie_theatre_question)`` works because this +very module is in that allowlist (and note we call it twice; the +daemon happily serves repeat requests). Ask for a function from +any module *not* enabled and you're denied with a +``ModuleNotExposed`` error: a simple, capability-style +restriction mechanism built on Python's own module system. + +We are *processes* +------------------ +Why processes (and not, say, threads)? Python has a GIL and an +`actor model`_ by definition shares **nothing** between its +concurrent units, so real OS processes are the natural fit: you +get all your cores locally, and since actors only ever talk via +IPC, the exact same code distributes over multiple hosts without +modification. + +Of course, the moment you hear "process trees" you should be +asking: *what about zombies?* Watch ``tractor`` eat one for +breakfast - run this while monitoring your process tree:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/we_are_processes.py \ + && kill $! + +.. literalinclude:: ../../examples/parallelism/we_are_processes.py + :caption: examples/parallelism/we_are_processes.py + :language: python + +.. margin:: Who's who in ``pstree``? + + Every subactor (best-effort, via the optional + ``setproctitle`` dep) re-titles its OS process like + ``_subactor[worker_0@]``, so ``pstree``/``htop``/ + ``pgrep -f`` can tell your actors apart at a glance. + +You'll see something like:: + + $ python examples/parallelism/we_are_processes.py + Yo, i'm 'worker_2' running in pid 1777246 + Yo, i'm 'worker_0' running in pid 1777244 + Yo, i'm 'worker_3' running in pid 1777247 + Yo, i'm 'worker_1' running in pid 1777245 + This process tree will self-destruct in 1 sec... + Zombies Contained + +(The worker lines land in whatever order the OS schedules them; +they're separate *processes*, racing, and that's the point.) + +An actor is spawned per core, each parks itself in +``trio.sleep_forever()``... and then the root *crashes on +purpose*. The ``ActorNursery`` responds with hard ``trio`` +discipline: every child is cancelled, every process is reaped, +the error propagates to ``trio.run()``, and your terminal prints +``Zombies Contained``. No orphans, no ``kill -9`` archaeology in +``htop`` afterwards. + +.. note:: + + **The zombie-safety guarantee**: ``tractor`` tries to protect + you from zombies, *no matter what*. If you can create zombie + child processes (without using a system signal) it **is a + bug** - please report it so we can hunt it down. + +A trynamic first scene +---------------------- +So far the root actor has done all the talking, but subactors +can just as well discover and call *each other*. Let's direct a +couple actors and have them run their lines for the hip new film +we're shooting: + +.. literalinclude:: ../../examples/a_trynamic_first_scene.py + :caption: examples/a_trynamic_first_scene.py + :language: python + +The script of the scene (runtime ``INFO`` log lines trimmed):: + + $ python examples/a_trynamic_first_scene.py + Alright... Action! + Hi my name is gretchen + Hi my name is donny + CUTTTT CUUTT CUT!!! Donny!! You're supposed to say... + +The new tricks in play: + +- two subactors, *donny* and *gretchen*, are each told to run + ``say_hello()`` targeting the *other* by name, +- ``tractor.wait_for_actor()`` blocks until the named peer has + registered with the tree's *registrar* (every actor announces + itself at boot), then yields a ``Portal`` connected + **directly** to that peer, +- each actor invokes its partner's ``hi()`` over that portal: + actor-to-actor RPC with the root merely *directing* - and both + final lines flow back to ``main()`` via + ``await portal.wait_for_result()``, +- ``tractor.log.get_console_log("INFO")`` cranks up runtime + logging so you can watch the spawn/register/cancel machinery + narrate itself; remove it for a quiet set. + +Cross-actor calls look just like (async) function calls; there +are no proxy objects and no shared references, only messages B) + +Crash handling, native feeling +------------------------------ +One last teaser before the guide proper. Flip exactly one +switch: + +.. code:: python + + async with tractor.open_nursery( + debug_mode=True, + ) as an: + ... + +and any crash, in *any* actor at *any* depth of the tree, drops +your terminal into a multi-process-safe pdbp_ REPL at the +offending frame, with the rest of the tree held back from +clobbering the tty. ``await tractor.pause()`` likewise gives you +a breakpoint that *just works* inside subprocesses. We think it +might be the first native multi-process debugging UX for Python; +get the full tour in :doc:`/guide/debugging`. + +Where to next? +-------------- +You can now boot a runtime, spawn one-shot and daemon actors, +make cross-process RPC calls, and contain zombies: that's the +on-ramp done. The guide takes each subsystem deeper, + +- :doc:`/explain/sc-distributed` - the structured concurrency + worldview and how ``tractor`` extends it across processes, +- :doc:`/guide/spawning` - everything ``ActorNursery``: spawn + kwargs, lifetimes and supervision semantics, +- :doc:`/guide/rpc` - the ``Portal`` in depth: calling into + another actor's memory domain, +- :doc:`/guide/context` - the core API: ``@tractor.context`` + endpoints, the ``ctx.started()`` handshake, and SC-linked + cross-actor task pairs, +- :doc:`/guide/streaming` - bidirectional ``MsgStream`` dialogs + and fan-out broadcasting, +- :doc:`/guide/debugging` - the multi-process REPL, crash + handling mode, and ``tractor.pause()``, +- :doc:`/guide/asyncio` - "infected ``asyncio``" mode: SC + supervision wrapped around ``asyncio`` tasks, +- :doc:`/guide/discovery` - registries, service daemons, and + finding actors from anywhere in (or out of) the tree. + +.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency +.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning +.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker +.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality +.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics +.. _actor model: https://en.wikipedia.org/wiki/Actor_model +.. _trio-parallel: https://github.com/richardsheridan/trio-parallel +.. _pdbp: https://github.com/mdmintz/pdbp