Compare commits
No commits in common. "25f5210f265cbaf7126b3edfd698a1700bd63e86" and "6a1149d4015a9ee46ea7ce64adc9ebf76f5d1859" have entirely different histories.
25f5210f26
...
6a1149d401
|
|
@ -1,62 +0,0 @@
|
||||||
name: docs
|
|
||||||
|
|
||||||
# build sphinx docs on every PR + push to main;
|
|
||||||
# deploy to gh-pages only from main pushes.
|
|
||||||
# (see goodboy/tractor#123 for the original ask)
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- main
|
|
||||||
pull_request:
|
|
||||||
# to run workflow manually from the "Actions" tab
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
# needed by actions/deploy-pages
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
pages: write
|
|
||||||
id-token: write
|
|
||||||
|
|
||||||
# never run >1 pages deploy at once
|
|
||||||
concurrency:
|
|
||||||
group: 'pages'
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build:
|
|
||||||
name: 'sphinx build'
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Install latest uv
|
|
||||||
uses: astral-sh/setup-uv@v6
|
|
||||||
|
|
||||||
# NOTE, no `d2` bin is installed in CI (yet) so
|
|
||||||
# the pre-rendered + committed SVGs under
|
|
||||||
# `docs/_diagrams/` are used as-is; see
|
|
||||||
# `docs/_ext/d2diagrams.py` for the fallback
|
|
||||||
# policy.
|
|
||||||
- name: Build html docs
|
|
||||||
run: |
|
|
||||||
uv sync --no-dev --group docs
|
|
||||||
uv run --no-dev --group docs make -C docs html
|
|
||||||
|
|
||||||
- name: Upload pages artifact
|
|
||||||
uses: actions/upload-pages-artifact@v3
|
|
||||||
with:
|
|
||||||
path: docs/_build/html
|
|
||||||
|
|
||||||
deploy:
|
|
||||||
name: 'deploy to gh-pages'
|
|
||||||
if: ${{ github.ref == 'refs/heads/main' && github.event_name == 'push' }}
|
|
||||||
needs: build
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
environment:
|
|
||||||
name: github-pages
|
|
||||||
url: ${{ steps.deployment.outputs.page_url }}
|
|
||||||
steps:
|
|
||||||
- name: Deploy
|
|
||||||
id: deployment
|
|
||||||
uses: actions/deploy-pages@v4
|
|
||||||
|
|
@ -1,93 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 0780a862-e19a-4f0a-86cd-c8afc0997757
|
|
||||||
timestamp: 2026-06-11T17:51:52Z
|
|
||||||
git_ref: 8526985c
|
|
||||||
scope: docs+code+config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260611T175152Z_8526985c_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
> we need big boi docs; ours are way out of date and generally
|
|
||||||
> terrible. i'd like to use a sphinx theme in the vein and/or
|
|
||||||
> one-of/nearly-the-same-as used in one or all of the `msgspec`,
|
|
||||||
> `numpy`, `ray`, `xonsh`, `polars` projects.
|
|
||||||
>
|
|
||||||
> see the following oustanding but now very old issues to guide
|
|
||||||
> you: #175, #126, #123, #157
|
|
||||||
>
|
|
||||||
> we should try to target a 3 columned sphinx theme with,
|
|
||||||
> - document index-nav on left,
|
|
||||||
> - body content in middle,
|
|
||||||
> - diagrams (ideally in d2lang if possible) on RHS
|
|
||||||
>
|
|
||||||
> optimize for (if possible),
|
|
||||||
> - reusing examples code without duplication in docs,
|
|
||||||
> - generating new examples from todos throughout code base,
|
|
||||||
> - distilling the essence of SC across processes in the simplest
|
|
||||||
> and most friendly way possible.
|
|
||||||
>
|
|
||||||
> do this work in a /open-wkt and do you best without my guidance
|
|
||||||
> for the first major pass - we will refine asap once you are
|
|
||||||
> complete.
|
|
||||||
|
|
||||||
Session settings: `/effort max` + ultracode (multi-agent workflow
|
|
||||||
orchestration). Fully autonomous first pass; two orchestrated
|
|
||||||
agent fleets were used (6-agent recon survey, then 10-agent
|
|
||||||
content fan-out: 9 section writers + 1 examples smith), with the
|
|
||||||
orchestrator authoring the landing page, SC essay, sphinx
|
|
||||||
scaffold, d2 pipeline + diagram sources, CI workflow and all
|
|
||||||
integration/fix passes directly.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Complete sphinx docs revamp on branch `wkt/big_boi_docs`:
|
|
||||||
`pydata-sphinx-theme` 0.18 (3-column: left nav / content /
|
|
||||||
page-toc, with d2 diagrams + asides floated into the RHS margin
|
|
||||||
via custom CSS), sphinx 9.1, a local `.. d2::` directive
|
|
||||||
rendering `docs/diagrams/*.d2` sources with committed-SVG
|
|
||||||
fallback, a vendored `.. margin::` directive, ~25 new doc pages
|
|
||||||
(landing, start/, explain/, 12 guides, 10 api-ref pages,
|
|
||||||
project/), 5 new auto-tested examples + 3 modernized + 1
|
|
||||||
renamed, and a gh-pages deploy workflow (issue #123). All
|
|
||||||
example code is `literalinclude`d from `examples/` (zero
|
|
||||||
duplication, CI-verified). Build: green, 24 warnings all
|
|
||||||
pre-existing-docstring/NEWS sourced.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
See the branch diff (uncommitted at entry-write time):
|
|
||||||
|
|
||||||
> `git diff test_cpu_throttling..wkt/big_boi_docs`
|
|
||||||
> `git -C <wkt> status --short` (pre-commit working tree)
|
|
||||||
|
|
||||||
- `docs/conf.py` — full rewrite for pydata theme + ext stack
|
|
||||||
- `docs/_ext/d2diagrams.py` — new `.. d2::` sphinx directive
|
|
||||||
- `docs/_ext/marginalia.py` — new `.. margin::` directive
|
|
||||||
- `docs/_static/css/custom.css` — b&w skin + RHS margin floats
|
|
||||||
- `docs/diagrams/*.d2` (7) — diagram sources (sketch/grayscale)
|
|
||||||
- `docs/_diagrams/*.svg` (7) — committed rendered fallbacks
|
|
||||||
- `docs/index.rst` — new landing (replaces dead-API doc)
|
|
||||||
- `docs/start/*.rst` (3), `docs/explain/*.rst` (3),
|
|
||||||
`docs/guide/*.rst` (13), `docs/api/*.rst` (10),
|
|
||||||
`docs/project/*.rst` (3) — new content tree
|
|
||||||
- `docs/dev_tips.rst` — removed (ported to project/dev-tips)
|
|
||||||
- `examples/{typed_payloads,nested_actor_tree,
|
|
||||||
service_daemon_discovery,uds_transport_actor_tree,
|
|
||||||
streaming_broadcast_fanout}.py` — new, smoke-tested
|
|
||||||
- `examples/{a_trynamic_first_scene,
|
|
||||||
actor_spawning_and_causality,parallelism/single_func}.py` —
|
|
||||||
`.result()` -> `.wait_for_result()` modernization
|
|
||||||
- `examples/parallelism/concurrent_futures_primes.py` — renamed
|
|
||||||
from leading-underscore + trio-runner shim added
|
|
||||||
- `pyproject.toml` — `docs` dependency-group filled in
|
|
||||||
- `uv.lock` — relock for docs group
|
|
||||||
- `.github/workflows/docs.yml` — build + gh-pages deploy
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None yet — entry written pre-commit; the author reviews, stages
|
|
||||||
and commits manually (per repo workflow policy).
|
|
||||||
|
|
@ -1,49 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-11T17:51:52Z
|
|
||||||
git_ref: 8526985c
|
|
||||||
diff_cmd: git diff test_cpu_throttling..wkt/big_boi_docs
|
|
||||||
---
|
|
||||||
|
|
||||||
# Raw output pointers (diff-ref mode)
|
|
||||||
|
|
||||||
All generated content is code/config/docs committed alongside
|
|
||||||
this entry on branch `wkt/big_boi_docs`; per the prompt-io
|
|
||||||
diff-ref decision rule each file's verbatim content is the diff
|
|
||||||
itself rather than a copy here:
|
|
||||||
|
|
||||||
> `git diff test_cpu_throttling..wkt/big_boi_docs -- docs/`
|
|
||||||
> `git diff test_cpu_throttling..wkt/big_boi_docs -- examples/`
|
|
||||||
> `git diff test_cpu_throttling..wkt/big_boi_docs -- pyproject.toml uv.lock .github/`
|
|
||||||
|
|
||||||
## Generation notes (non-code output summary)
|
|
||||||
|
|
||||||
- Theme research (web, agent-verified 2026-06-11): msgspec=furo,
|
|
||||||
xonsh=furo, numpy/ray/polars=pydata-sphinx-theme (ray migrated
|
|
||||||
off sphinx-book-theme); sphinx-book-theme 1.2.0 hard-pins
|
|
||||||
pydata 0.16.1 (stale) -> chose pydata 0.18 + sphinx 9.1.
|
|
||||||
- d2 ecosystem: no production-grade pypi extension exists
|
|
||||||
(sphinxcontrib-d2lang 0.0.5 ignores returncodes, uuid4 output
|
|
||||||
names; sphinx-d2 is an empty stub) -> wrote local
|
|
||||||
`docs/_ext/d2diagrams.py` (~230 LOC) with D2_BIN env
|
|
||||||
discovery, mtime caching, committed-SVG fallback and
|
|
||||||
literal-block last resort.
|
|
||||||
- Diagrams authored in d2 (theme-id 1 "Neutral Grey" + sketch
|
|
||||||
mode + ELK layout, validated by render + headless-firefox
|
|
||||||
screenshot loop): actor_tree, context_handshake (real
|
|
||||||
msg-spec names Start/StartAck/Started/Yield/Stop/Return),
|
|
||||||
streaming_pipeline, runtime_stack, debug_lock,
|
|
||||||
error_propagation, infected_aio.
|
|
||||||
- API truth enforced from a 6-agent recon pass over the
|
|
||||||
reorganized package tree (runtime/, discovery/, spawn/, ipc/,
|
|
||||||
msg/, devx/, trionics/): docs teach `.wait_for_result()`,
|
|
||||||
registrar (not arbiter) naming, `@tractor.context` +
|
|
||||||
`open_context()` as the core model, `run_in_actor()` as
|
|
||||||
convenience only.
|
|
||||||
- All ~30 literalincluded example scripts verified present; 9
|
|
||||||
touched/new example files smoke-run green (exit 0, <16s).
|
|
||||||
- Final build: `sphinx-build -b html` succeeded; 24 residual
|
|
||||||
warnings, every one sourced from pre-existing library
|
|
||||||
docstring rst-isms or legacy NEWS.rst content (left untouched
|
|
||||||
by design; flagged for a follow-up docstring lint pass).
|
|
||||||
|
Before Width: | Height: | Size: 98 KiB |
|
Before Width: | Height: | Size: 83 KiB |
|
Before Width: | Height: | Size: 80 KiB |
|
Before Width: | Height: | Size: 80 KiB |
|
Before Width: | Height: | Size: 98 KiB |
|
Before Width: | Height: | Size: 61 KiB |
|
Before Width: | Height: | Size: 90 KiB |
|
|
@ -1,235 +0,0 @@
|
||||||
# tractor: distributed structured concurrency.
|
|
||||||
'''
|
|
||||||
``.. d2::`` - embed d2lang_ diagrams in sphinx docs
|
|
||||||
with build-time rendering and a committed-SVG
|
|
||||||
fallback.
|
|
||||||
|
|
||||||
Rendering policy,
|
|
||||||
|
|
||||||
- when a ``d2`` binary is found (see discovery order
|
|
||||||
below) any out-of-date SVG is (re)rendered from its
|
|
||||||
``.d2`` source (normally kept in ``docs/diagrams/``)
|
|
||||||
into ``docs/_diagrams/``,
|
|
||||||
- otherwise any pre-rendered (and git committed) SVG
|
|
||||||
already in ``docs/_diagrams/`` is used as-is,
|
|
||||||
- when neither is possible the diagram *source* is
|
|
||||||
emitted as a literal block so no content is ever
|
|
||||||
silently dropped.
|
|
||||||
|
|
||||||
Binary discovery order,
|
|
||||||
|
|
||||||
- the ``D2_BIN`` env var; may contain args which are
|
|
||||||
split via `shlex`, eg.
|
|
||||||
``D2_BIN='nix run nixpkgs#d2 --'``,
|
|
||||||
- the ``d2_bin`` sphinx config value,
|
|
||||||
- ``shutil.which('d2')``.
|
|
||||||
|
|
||||||
Usage,
|
|
||||||
|
|
||||||
.. code:: rst
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:caption: A tree of ``trio``-task-trees.
|
|
||||||
:margin:
|
|
||||||
|
|
||||||
.. _d2lang: https://d2lang.com
|
|
||||||
|
|
||||||
'''
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
import shlex
|
|
||||||
import shutil
|
|
||||||
import subprocess as sp
|
|
||||||
|
|
||||||
from docutils import nodes
|
|
||||||
from docutils.parsers.rst import directives
|
|
||||||
from sphinx.application import Sphinx
|
|
||||||
from sphinx.util import logging
|
|
||||||
from sphinx.util.docutils import SphinxDirective
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# subdir (under the sphinx srcdir) holding rendered,
|
|
||||||
# git-committed, fallback SVG outputs.
|
|
||||||
_outdir: str = '_diagrams'
|
|
||||||
|
|
||||||
|
|
||||||
def find_d2(
|
|
||||||
app: Sphinx,
|
|
||||||
) -> list[str]|None:
|
|
||||||
'''
|
|
||||||
Resolve the d2 render command as an argv list or
|
|
||||||
`None` when no binary can be found.
|
|
||||||
|
|
||||||
'''
|
|
||||||
if env_bin := os.environ.get('D2_BIN'):
|
|
||||||
return shlex.split(env_bin)
|
|
||||||
if cfg_bin := app.config.d2_bin:
|
|
||||||
return shlex.split(cfg_bin)
|
|
||||||
if path_bin := shutil.which('d2'):
|
|
||||||
return [path_bin]
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def render_svg(
|
|
||||||
app: Sphinx,
|
|
||||||
src: Path,
|
|
||||||
out: Path,
|
|
||||||
) -> bool:
|
|
||||||
'''
|
|
||||||
Maybe (re)render `src` -> `out`, returning
|
|
||||||
`True` when an up-to-date SVG exists after the
|
|
||||||
call.
|
|
||||||
|
|
||||||
'''
|
|
||||||
stale: bool = (
|
|
||||||
not out.exists()
|
|
||||||
or
|
|
||||||
src.stat().st_mtime > out.stat().st_mtime
|
|
||||||
)
|
|
||||||
if not stale:
|
|
||||||
return True
|
|
||||||
d2cmd: list[str]|None = find_d2(app)
|
|
||||||
if d2cmd is None:
|
|
||||||
if out.exists():
|
|
||||||
log.info(
|
|
||||||
f'no d2 binary; using committed svg '
|
|
||||||
f'for {src.name}'
|
|
||||||
)
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
out.parent.mkdir(
|
|
||||||
parents=True,
|
|
||||||
exist_ok=True,
|
|
||||||
)
|
|
||||||
argv: list[str] = (
|
|
||||||
d2cmd
|
|
||||||
+ list(app.config.d2_args)
|
|
||||||
+ [str(src), str(out)]
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
proc = sp.run(
|
|
||||||
argv,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=120,
|
|
||||||
)
|
|
||||||
except (
|
|
||||||
OSError,
|
|
||||||
sp.TimeoutExpired,
|
|
||||||
) as err:
|
|
||||||
log.warning(
|
|
||||||
f'd2 invocation failed for {src.name}: '
|
|
||||||
f'{err}'
|
|
||||||
)
|
|
||||||
return out.exists()
|
|
||||||
if proc.returncode != 0:
|
|
||||||
log.warning(
|
|
||||||
f'd2 render error for {src.name}:\n'
|
|
||||||
f'{proc.stderr}'
|
|
||||||
)
|
|
||||||
return out.exists()
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
class D2Diagram(SphinxDirective):
|
|
||||||
'''
|
|
||||||
Render a ``.d2`` source file (path relative to
|
|
||||||
the sphinx srcdir) as an SVG figure.
|
|
||||||
|
|
||||||
'''
|
|
||||||
required_arguments = 1
|
|
||||||
has_content = False
|
|
||||||
option_spec = {
|
|
||||||
'caption': directives.unchanged,
|
|
||||||
'alt': directives.unchanged,
|
|
||||||
'width': (
|
|
||||||
directives.length_or_percentage_or_unitless
|
|
||||||
),
|
|
||||||
'margin': directives.flag,
|
|
||||||
'class': directives.class_option,
|
|
||||||
'name': directives.unchanged,
|
|
||||||
}
|
|
||||||
|
|
||||||
def run(self) -> list[nodes.Node]:
|
|
||||||
relsrc: str = self.arguments[0]
|
|
||||||
srcdir = Path(self.env.srcdir)
|
|
||||||
src: Path = srcdir / relsrc
|
|
||||||
self.env.note_dependency(relsrc)
|
|
||||||
if not src.exists():
|
|
||||||
err = self.state_machine.reporter.error(
|
|
||||||
f'd2 source not found: {relsrc}',
|
|
||||||
line=self.lineno,
|
|
||||||
)
|
|
||||||
return [err]
|
|
||||||
out: Path = (
|
|
||||||
srcdir
|
|
||||||
/ _outdir
|
|
||||||
/ f'{src.stem}.svg'
|
|
||||||
)
|
|
||||||
if not render_svg(self.env.app, src, out):
|
|
||||||
# last resort: emit the raw d2 source.
|
|
||||||
log.warning(
|
|
||||||
f'no svg available for {relsrc}; '
|
|
||||||
f'emitting d2 source as literal block'
|
|
||||||
)
|
|
||||||
literal = nodes.literal_block(
|
|
||||||
src.read_text(),
|
|
||||||
src.read_text(),
|
|
||||||
)
|
|
||||||
literal['language'] = 'text'
|
|
||||||
return [literal]
|
|
||||||
img = nodes.image(
|
|
||||||
uri=f'/{_outdir}/{out.name}',
|
|
||||||
alt=self.options.get(
|
|
||||||
'alt',
|
|
||||||
f'd2 diagram: {src.stem}',
|
|
||||||
),
|
|
||||||
)
|
|
||||||
if width := self.options.get('width'):
|
|
||||||
img['width'] = width
|
|
||||||
fig = nodes.figure()
|
|
||||||
fig += img
|
|
||||||
classes: list[str] = (
|
|
||||||
['d2-diagram']
|
|
||||||
+ self.options.get('class', [])
|
|
||||||
)
|
|
||||||
if 'margin' in self.options:
|
|
||||||
# NB: the bare 'margin' class is what
|
|
||||||
# book-style themes key off for
|
|
||||||
# right-margin placement; our custom css
|
|
||||||
# uses 'd2-margin'.
|
|
||||||
classes += [
|
|
||||||
'margin',
|
|
||||||
'd2-margin',
|
|
||||||
]
|
|
||||||
fig['classes'] += classes
|
|
||||||
if caption_txt := self.options.get('caption'):
|
|
||||||
(
|
|
||||||
inline_nodes,
|
|
||||||
_msgs,
|
|
||||||
) = self.state.inline_text(
|
|
||||||
caption_txt,
|
|
||||||
self.lineno,
|
|
||||||
)
|
|
||||||
caption = nodes.caption(
|
|
||||||
caption_txt,
|
|
||||||
'',
|
|
||||||
*inline_nodes,
|
|
||||||
)
|
|
||||||
fig += caption
|
|
||||||
self.add_name(fig)
|
|
||||||
return [fig]
|
|
||||||
|
|
||||||
|
|
||||||
def setup(app: Sphinx) -> dict:
|
|
||||||
app.add_config_value('d2_bin', None, 'env')
|
|
||||||
app.add_config_value('d2_args', [], 'env')
|
|
||||||
app.add_directive('d2', D2Diagram)
|
|
||||||
return {
|
|
||||||
'version': '0.1.0',
|
|
||||||
'parallel_read_safe': True,
|
|
||||||
'parallel_write_safe': True,
|
|
||||||
}
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
# tractor: distributed structured concurrency.
|
|
||||||
'''
|
|
||||||
``.. margin::`` - prose-anchored, right-margin asides.
|
|
||||||
|
|
||||||
A theme-agnostic vendoring of `sphinx_book_theme`'s
|
|
||||||
`Margin` directive: a `docutils` `Sidebar` subclass
|
|
||||||
which tags the node with a ``margin`` class; placement
|
|
||||||
is then pure CSS (see ``_static/css/custom.css``)
|
|
||||||
allowing use on any theme incl. our
|
|
||||||
`pydata_sphinx_theme`.
|
|
||||||
|
|
||||||
Usage,
|
|
||||||
|
|
||||||
.. code:: rst
|
|
||||||
|
|
||||||
.. margin:: An optional title
|
|
||||||
|
|
||||||
Aside content; text, figures, whatever.
|
|
||||||
|
|
||||||
'''
|
|
||||||
from docutils import nodes
|
|
||||||
from docutils.parsers.rst.directives.body import (
|
|
||||||
Sidebar,
|
|
||||||
)
|
|
||||||
from sphinx.application import Sphinx
|
|
||||||
|
|
||||||
|
|
||||||
class Margin(Sidebar):
|
|
||||||
'''
|
|
||||||
Notes/figures placed in the right margin, anchored
|
|
||||||
at the current point in the prose flow.
|
|
||||||
|
|
||||||
'''
|
|
||||||
required_arguments = 0
|
|
||||||
optional_arguments = 1
|
|
||||||
|
|
||||||
def run(self) -> list[nodes.Node]:
|
|
||||||
if not self.arguments:
|
|
||||||
self.arguments = ['']
|
|
||||||
out: list[nodes.Node] = super().run()
|
|
||||||
out[0].attributes['classes'].append('margin')
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def setup(app: Sphinx) -> dict:
|
|
||||||
app.add_directive('margin', Margin)
|
|
||||||
return {
|
|
||||||
'version': '0.1.0',
|
|
||||||
'parallel_read_safe': True,
|
|
||||||
'parallel_write_safe': True,
|
|
||||||
}
|
|
||||||
|
|
@ -1,75 +0,0 @@
|
||||||
/* tractor docs: a minimal black + white skin over
|
|
||||||
* `pydata-sphinx-theme` plus RHS-marginalia + d2
|
|
||||||
* diagram styling.
|
|
||||||
*/
|
|
||||||
html[data-theme="light"] {
|
|
||||||
--pst-color-primary: #000000;
|
|
||||||
--pst-color-secondary: #3d3d3d;
|
|
||||||
--pst-color-accent: #5a5a5a;
|
|
||||||
--pst-color-link: #000000;
|
|
||||||
--pst-color-link-hover: #5a5a5a;
|
|
||||||
--pst-color-inline-code: #1a1a1a;
|
|
||||||
--pst-color-inline-code-links: #000000;
|
|
||||||
}
|
|
||||||
html[data-theme="dark"] {
|
|
||||||
--pst-color-primary: #ffffff;
|
|
||||||
--pst-color-secondary: #c9c9c9;
|
|
||||||
--pst-color-accent: #a8a8a8;
|
|
||||||
--pst-color-link: #ffffff;
|
|
||||||
--pst-color-link-hover: #bdbdbd;
|
|
||||||
--pst-color-inline-code: #e8e8e8;
|
|
||||||
--pst-color-inline-code-links: #ffffff;
|
|
||||||
}
|
|
||||||
/* mono-chrome links: rely on underline for
|
|
||||||
* affordance instead of color.
|
|
||||||
*/
|
|
||||||
.bd-content a:not(.headerlink) {
|
|
||||||
text-decoration: underline;
|
|
||||||
text-underline-offset: 0.18em;
|
|
||||||
}
|
|
||||||
/* d2 diagram figures */
|
|
||||||
figure.d2-diagram {
|
|
||||||
text-align: center;
|
|
||||||
}
|
|
||||||
figure.d2-diagram img {
|
|
||||||
max-width: 100%;
|
|
||||||
height: auto;
|
|
||||||
}
|
|
||||||
/* keep light-rendered svgs legible in dark mode */
|
|
||||||
html[data-theme="dark"] figure.d2-diagram img {
|
|
||||||
background: #ffffff;
|
|
||||||
border-radius: 6px;
|
|
||||||
padding: 6px;
|
|
||||||
}
|
|
||||||
/* prose-anchored right-margin asides (tufte-ish):
|
|
||||||
* float right within the content column on wide
|
|
||||||
* screens, collapse inline on narrow ones.
|
|
||||||
*/
|
|
||||||
@media (min-width: 960px) {
|
|
||||||
aside.margin,
|
|
||||||
figure.margin,
|
|
||||||
div.margin,
|
|
||||||
figure.d2-margin {
|
|
||||||
float: right;
|
|
||||||
clear: right;
|
|
||||||
width: 44%;
|
|
||||||
margin: 0.2rem 0 1rem 1.4rem;
|
|
||||||
font-size: 0.85rem;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
/* strip docutils sidebar chrome from margin asides */
|
|
||||||
aside.sidebar.margin {
|
|
||||||
border: none;
|
|
||||||
background: transparent;
|
|
||||||
padding: 0;
|
|
||||||
}
|
|
||||||
aside.sidebar.margin > p.sidebar-title {
|
|
||||||
font-weight: 600;
|
|
||||||
font-size: 0.9rem;
|
|
||||||
margin-bottom: 0.3rem;
|
|
||||||
}
|
|
||||||
/* landing page hero logo sizing */
|
|
||||||
img.hero-logo {
|
|
||||||
max-width: 360px;
|
|
||||||
width: 60%;
|
|
||||||
}
|
|
||||||
|
|
@ -1,134 +0,0 @@
|
||||||
Contexts and streaming
|
|
||||||
======================
|
|
||||||
|
|
||||||
The modern core of ``tractor``: a :class:`Context` links a task in
|
|
||||||
one actor to a task in another as a *single* structured concurrency
|
|
||||||
(SC) scope stretched across the IPC boundary — errors, results and
|
|
||||||
cancellation flow between the pair `exactly like trio`_ tasks under
|
|
||||||
a common `nursery`_. Open one with ``Portal.open_context()`` (see
|
|
||||||
:func:`tractor._context.open_context_from_portal` in
|
|
||||||
:doc:`/api/core`), then optionally bridge a bidirectional
|
|
||||||
:class:`MsgStream` between the two tasks.
|
|
||||||
|
|
||||||
.. d2:: diagrams/context_handshake.d2
|
|
||||||
:caption: The ``open_context()`` <-> ``ctx.started()`` handshake.
|
|
||||||
:margin:
|
|
||||||
:alt: parent and child actor context handshake sequence
|
|
||||||
|
|
||||||
For the guided, example-driven tour see :doc:`/guide/context`; this
|
|
||||||
page is the precise API surface.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
The ``@context`` decorator
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
.. autofunction:: context
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
The decorated function **must** declare a parameter annotated
|
|
||||||
``tractor.Context`` (any param name works); the runtime injects
|
|
||||||
the context instance there on each remote invocation. Pass
|
|
||||||
``pld_spec`` to type-restrict (and validate) the payloads this
|
|
||||||
endpoint may shuttle — violations raise
|
|
||||||
:class:`MsgTypeError`. See ``examples/typed_payloads.py``.
|
|
||||||
|
|
||||||
``Context``
|
|
||||||
-----------
|
|
||||||
|
|
||||||
.. autoclass:: Context
|
|
||||||
:members: started,
|
|
||||||
wait_for_result,
|
|
||||||
cancel,
|
|
||||||
cid,
|
|
||||||
chan,
|
|
||||||
side,
|
|
||||||
cancel_called,
|
|
||||||
cancelled_caught,
|
|
||||||
cancel_acked,
|
|
||||||
canceller,
|
|
||||||
maybe_error,
|
|
||||||
outcome
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``Context.result()`` warns; use :meth:`Context.wait_for_result`.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
A :class:`Context` is **not** a :class:`trio.CancelScope`:
|
|
||||||
:meth:`Context.cancel` requests cancellation of the *remote*
|
|
||||||
peer task and does not cancel the local scope. If *you*
|
|
||||||
requested the cancel, the resulting :class:`ContextCancelled`
|
|
||||||
is absorbed at ``open_context()`` exit; a cancel originating
|
|
||||||
anywhere else (the peer, or a third-party actor recorded in
|
|
||||||
:attr:`ContextCancelled.canceller`) *is* raised locally. This
|
|
||||||
self-vs-cross-cancel rule is the key to writing correct
|
|
||||||
inter-actor teardown logic — see :doc:`/guide/context`.
|
|
||||||
|
|
||||||
Bidirectional streaming
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
.. autofunction:: tractor._streaming.open_stream_from_ctx
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`~tractor._streaming.open_stream_from_ctx` is bound as
|
|
||||||
the **method-alias** ``Context.open_stream()`` — call it as
|
|
||||||
``async with ctx.open_stream() as stream:``. Both sides of the
|
|
||||||
context must enter it for the dialog to be open.
|
|
||||||
|
|
||||||
.. autoclass:: MsgStream
|
|
||||||
:members: send,
|
|
||||||
receive,
|
|
||||||
receive_nowait,
|
|
||||||
aclose,
|
|
||||||
subscribe,
|
|
||||||
ctx,
|
|
||||||
closed
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
A :class:`MsgStream` is one-shot use: once closed it can never
|
|
||||||
be "re-opened" — open a fresh :class:`Context` instead. Remote
|
|
||||||
end-of-stream surfaces as :class:`StopAsyncIteration` from
|
|
||||||
``async for``; un-consumed sends overrun the receiver and raise
|
|
||||||
:class:`tractor._exceptions.StreamOverrun` unless the context
|
|
||||||
was opened with ``allow_overruns=True``.
|
|
||||||
|
|
||||||
:meth:`MsgStream.subscribe` fans a single IPC stream out to
|
|
||||||
multiple *local* tasks via a
|
|
||||||
:class:`tractor.trionics.BroadcastReceiver` (see
|
|
||||||
:doc:`/api/trionics`); the underlying allocation is idempotent and
|
|
||||||
non-reversible for the stream's lifetime. See
|
|
||||||
``examples/streaming_broadcast_fanout.py`` for the pattern in
|
|
||||||
action.
|
|
||||||
|
|
||||||
Legacy one-way streaming
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
.. autofunction:: stream
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
``@tractor.stream`` and ``Portal.open_stream_from()`` are the
|
|
||||||
*legacy* one-way streaming API kept for backward compat: a
|
|
||||||
plain async-generator function streamed parent-ward with no
|
|
||||||
child-side receive leg. New code should use
|
|
||||||
``@tractor.context`` + ``ctx.open_stream()`` (bidirectional,
|
|
||||||
SC-linked, typed). Note ``ctx`` is now a reserved param name
|
|
||||||
for ``@context`` endpoints — ``@stream`` functions must use
|
|
||||||
``stream`` instead, and ``ctx.send_yield()`` is deprecated in
|
|
||||||
favor of :meth:`MsgStream.send`.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/errors` for :class:`ContextCancelled` /
|
|
||||||
:class:`MsgTypeError` semantics, :doc:`/api/msg` for payload
|
|
||||||
typing via ``pld_spec`` and codecs, :doc:`/api/trionics` for
|
|
||||||
the broadcast fan-out machinery, and the guided tours in
|
|
||||||
:doc:`/guide/streaming` + :doc:`/guide/cancellation`.
|
|
||||||
|
|
||||||
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
|
||||||
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
|
||||||
|
|
@ -1,157 +0,0 @@
|
||||||
Runtime and spawning
|
|
||||||
====================
|
|
||||||
|
|
||||||
The core lifecycle API: boot the runtime in your root process,
|
|
||||||
spawn trio-"actors" (processes running ``trio.run()`` task trees)
|
|
||||||
under a one-cancels-all supervisor, and talk to them through
|
|
||||||
portals. This is structured concurrency (SC) applied *transitively*:
|
|
||||||
every spawned process is owned by a nursery block and errors
|
|
||||||
`always propagate`_. If you can create zombies it **is a bug**.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
Booting the runtime
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
.. autofunction:: open_root_actor
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
The env vars ``TRACTOR_LOGLEVEL`` and ``TRACTOR_SPAWN_METHOD``
|
|
||||||
override the ``loglevel`` / ``start_method`` params so you can
|
|
||||||
crank verbosity or swap spawn backends without touching app
|
|
||||||
code. Exactly **one** IPC transport may be enabled per actor
|
|
||||||
(see ``enable_transports`` and :doc:`/api/ipc`).
|
|
||||||
|
|
||||||
.. autofunction:: run_daemon
|
|
||||||
|
|
||||||
Spawning actors
|
|
||||||
---------------
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:caption: A supervised actor (process) tree.
|
|
||||||
:margin:
|
|
||||||
:alt: root actor supervising a tree of subactors
|
|
||||||
|
|
||||||
.. autofunction:: open_nursery
|
|
||||||
|
|
||||||
.. autoclass:: ActorNursery
|
|
||||||
:members: start_actor,
|
|
||||||
run_in_actor,
|
|
||||||
cancel,
|
|
||||||
cancel_called,
|
|
||||||
cancelled_caught
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:meth:`ActorNursery.start_actor` (daemon actor + portal) is the
|
|
||||||
blessed spawning primitive; pair it with
|
|
||||||
``Portal.open_context()`` for SC-linked remote tasks.
|
|
||||||
:meth:`ActorNursery.run_in_actor` is a *convenience* one-shot —
|
|
||||||
spawn, run a single task, auto-cancel after the result — slated
|
|
||||||
to be rebuilt as a high-level wrapper, so don't design around
|
|
||||||
it as the core model.
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``ActorNursery.cancelled`` warns; use
|
|
||||||
:attr:`ActorNursery.cancel_called` and
|
|
||||||
:attr:`ActorNursery.cancelled_caught`. The ``rpc_module_paths``
|
|
||||||
kwarg is likewise deprecated in favor of ``enable_modules``.
|
|
||||||
|
|
||||||
Portals
|
|
||||||
-------
|
|
||||||
|
|
||||||
A :class:`Portal` "opens a portal" into a peer actor's memory
|
|
||||||
domain: you call functions and start SC-linked tasks *over IPC* as
|
|
||||||
though they were local, with results, errors and cancellation
|
|
||||||
flowing back `exactly like trio`_.
|
|
||||||
|
|
||||||
.. autoclass:: Portal
|
|
||||||
:members: run,
|
|
||||||
run_from_ns,
|
|
||||||
open_stream_from,
|
|
||||||
wait_for_result,
|
|
||||||
cancel_actor,
|
|
||||||
chan
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``Portal.result()`` warns; use :meth:`Portal.wait_for_result`.
|
|
||||||
The str-form ``Portal.run('mod.path', 'fn_name')`` also warns;
|
|
||||||
pass a function *object* whose module is listed in the target's
|
|
||||||
``enable_modules``. ``Portal.channel`` is the legacy spelling
|
|
||||||
of :attr:`Portal.chan`.
|
|
||||||
|
|
||||||
.. autofunction:: tractor._context.open_context_from_portal
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`~tractor._context.open_context_from_portal` is bound as
|
|
||||||
the **method-alias** ``Portal.open_context()`` — that's the
|
|
||||||
spelling you should actually call:
|
|
||||||
``portal.open_context(fn, **kwargs)``. See :doc:`/api/context`
|
|
||||||
for the full ``Context`` + ``MsgStream`` API it unlocks.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:meth:`Portal.cancel_actor` cancels the *whole* remote runtime
|
|
||||||
and process (machine-level), not a single task — use
|
|
||||||
:meth:`Context.cancel` for task-level cancellation. Pass
|
|
||||||
``raise_on_timeout=True`` to get an ``ActorTooSlowError`` you
|
|
||||||
can escalate per SC discipline (see :doc:`/api/errors`).
|
|
||||||
|
|
||||||
Clusters
|
|
||||||
--------
|
|
||||||
|
|
||||||
.. autofunction:: open_actor_cluster
|
|
||||||
|
|
||||||
Spawn a *flat* cluster of ``count`` worker actors (default: one
|
|
||||||
per core) all serving the RPC ``modules`` list, yielding a
|
|
||||||
``dict[str, Portal]`` keyed by actor name. Handy for
|
|
||||||
embarrassingly parallel fan-out; see ``examples/quick_cluster.py``.
|
|
||||||
|
|
||||||
Runtime introspection
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
.. autofunction:: current_actor
|
|
||||||
|
|
||||||
.. autoclass:: Actor
|
|
||||||
:members: aid,
|
|
||||||
name,
|
|
||||||
uid,
|
|
||||||
is_registrar,
|
|
||||||
is_infected_aio,
|
|
||||||
cancel_soon
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:class:`Actor` is the per-process runtime singleton (msg loop,
|
|
||||||
RPC scheduling, IPC server) — you never instantiate it yourself
|
|
||||||
and should normally only touch the identity/introspection
|
|
||||||
surface listed above. The canonical identity type is
|
|
||||||
:attr:`Actor.aid` (a ``tractor.msg.Aid`` struct);
|
|
||||||
:attr:`Actor.uid` is the legacy ``(name, uuid)`` 2-tuple which
|
|
||||||
is still pervasive in logs and error metadata.
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``Actor.is_arbiter`` warns; use :attr:`Actor.is_registrar`.
|
|
||||||
The ``arbiter_addr`` constructor kwarg is deprecated for
|
|
||||||
``registry_addrs``.
|
|
||||||
|
|
||||||
.. autofunction:: current_ipc_ctx
|
|
||||||
|
|
||||||
.. autofunction:: is_root_process
|
|
||||||
|
|
||||||
.. autofunction:: get_runtime_vars
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/context` for the SC-linked remote task API,
|
|
||||||
:doc:`/api/discovery` for finding actors by name, and the
|
|
||||||
guided tours in :doc:`/guide/spawning`, :doc:`/guide/rpc` and
|
|
||||||
:doc:`/guide/context`.
|
|
||||||
|
|
||||||
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
|
|
||||||
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
|
||||||
|
|
@ -1,92 +0,0 @@
|
||||||
Debugging and devx: ``tractor.devx``
|
|
||||||
====================================
|
|
||||||
|
|
||||||
Multi-process debugging that actually works: boot the tree with
|
|
||||||
``open_root_actor(debug_mode=True)`` (or pass it to
|
|
||||||
``open_nursery()``) and any crash or explicit pause in *any* actor
|
|
||||||
acquires a tree-global TTY lock and drops you into a
|
|
||||||
`pdbp`_-powered REPL — one actor at a time, ``SIGINT`` shielded,
|
|
||||||
no garbled terminals. The top-level helpers below are the daily
|
|
||||||
drivers; the rest of the toolbox lives under ``tractor.devx``.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
Pausing and post-mortems
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
.. autofunction:: pause
|
|
||||||
|
|
||||||
.. autofunction:: pause_from_sync
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`pause_from_sync` needs the `greenback`_ portal: boot
|
|
||||||
with ``open_root_actor(maybe_enable_greenback=True)`` (mind
|
|
||||||
the `performance implications`_). With ``debug_mode`` on, the
|
|
||||||
built-in ``breakpoint()`` is also remapped to a
|
|
||||||
``tractor``-safe equivalent.
|
|
||||||
|
|
||||||
.. autofunction:: post_mortem
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``tractor.breakpoint()`` warns and simply calls :func:`pause`
|
|
||||||
— use :func:`tractor.pause` (async) or
|
|
||||||
:func:`tractor.pause_from_sync` in new code.
|
|
||||||
|
|
||||||
Crash handling for CLIs and sync entrypoints
|
|
||||||
--------------------------------------------
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.devx
|
|
||||||
|
|
||||||
.. autofunction:: open_crash_handler
|
|
||||||
|
|
||||||
.. autofunction:: maybe_open_crash_handler
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Both are *sync* context managers usable before (or without)
|
|
||||||
``trio.run()`` — wrap your CLI ``main()`` to get a post-mortem
|
|
||||||
REPL on any uncaught exception instead of a bare traceback.
|
|
||||||
|
|
||||||
Runtime hang-hunting
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
.. autofunction:: enable_stack_on_sig
|
|
||||||
|
|
||||||
With stackscope_ integration enabled (also via
|
|
||||||
``open_root_actor(enable_stack_on_sig=True)`` or the
|
|
||||||
``TRACTOR_ENABLE_STACKSCOPE`` env var) a ``SIGUSR1`` triggers a
|
|
||||||
full trio task-tree dump from every actor — works on live,
|
|
||||||
*non*-debug-mode trees too:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
pkill --signal SIGUSR1 -f <part-of-your-cmd>
|
|
||||||
|
|
||||||
Dumps also tee to ``/tmp/tractor-stackscope-<pid>.log`` so you
|
|
||||||
still get output under captured/CI stdio.
|
|
||||||
|
|
||||||
Lower-level debug plumbing
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
.. autofunction:: mk_pdb
|
|
||||||
|
|
||||||
.. autofunction:: maybe_wait_for_debugger
|
|
||||||
|
|
||||||
:func:`maybe_wait_for_debugger` is mainly useful in runtime/test
|
|
||||||
code that must avoid tearing down a tree while a child still
|
|
||||||
holds the global debug lock.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/errors` for the boxed error types you'll inspect
|
|
||||||
from the REPL, :doc:`/api/core` for the ``debug_mode`` /
|
|
||||||
``maybe_enable_greenback`` / ``enable_stack_on_sig`` boot
|
|
||||||
flags on :func:`tractor.open_root_actor`, and
|
|
||||||
:doc:`/guide/debugging` for the guided multi-actor REPL tour.
|
|
||||||
|
|
||||||
.. _pdbp: https://github.com/mdmintz/pdbp
|
|
||||||
.. _greenback: https://greenback.readthedocs.io/en/latest/
|
|
||||||
.. _performance implications: https://greenback.readthedocs.io/en/latest/principle.html#performance
|
|
||||||
.. _stackscope: https://github.com/oremanj/stackscope
|
|
||||||
|
|
@ -1,67 +0,0 @@
|
||||||
Discovery and the registrar
|
|
||||||
===========================
|
|
||||||
|
|
||||||
Every actor registers its ``(name, uuid)`` and transport addresses
|
|
||||||
with a *registrar* actor — by default the root of the tree, or
|
|
||||||
whichever actor serves at the ``registry_addrs`` you boot with.
|
|
||||||
The discovery API lets any actor look up any other **by name** and
|
|
||||||
get back a connected :class:`~tractor.Portal`, giving you
|
|
||||||
service-discovery patterns (daemons, service trees, multi-host
|
|
||||||
meshes) without hard-coding addresses.
|
|
||||||
|
|
||||||
Lookups first scan already-connected peers before RPC-ing the
|
|
||||||
registrar, and multihomed results are ranked UDS > local TCP >
|
|
||||||
remote TCP. See ``examples/service_daemon_discovery.py`` for the
|
|
||||||
canonical daemon + lookup pattern.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
Lookup APIs
|
|
||||||
-----------
|
|
||||||
|
|
||||||
.. autofunction:: find_actor
|
|
||||||
|
|
||||||
.. autofunction:: wait_for_actor
|
|
||||||
|
|
||||||
.. autofunction:: query_actor
|
|
||||||
|
|
||||||
.. autofunction:: get_registry
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`find_actor` yields ``None`` when nothing is registered
|
|
||||||
under the name (or raises with ``raise_on_none=True``);
|
|
||||||
:func:`wait_for_actor` blocks until the name appears;
|
|
||||||
:func:`query_actor` only *looks up* the address without
|
|
||||||
connecting to the target.
|
|
||||||
|
|
||||||
The ``Registrar``
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
.. autoclass:: Registrar
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
A :class:`Registrar` is just an :class:`~tractor.Actor` subtype
|
|
||||||
maintaining the name -> addresses table; you rarely touch it
|
|
||||||
directly beyond passing ``registry_addrs`` /
|
|
||||||
``ensure_registry=True`` to :func:`~tractor.open_root_actor`.
|
|
||||||
Check :attr:`Actor.is_registrar <tractor.Actor.is_registrar>` to
|
|
||||||
ask "am I it?".
|
|
||||||
|
|
||||||
Legacy ``Arbiter`` alias
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``tractor.Arbiter`` survives only as a class alias of
|
|
||||||
:class:`Registrar` and all "arbiter" terminology is replaced
|
|
||||||
by "registrar"/"registry" across the API: ``get_arbiter()`` is
|
|
||||||
removed (use :func:`get_registry`) and the ``arbiter_addr``
|
|
||||||
kwarg is replaced by ``registry_addrs``.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/core` for booting a registrar via
|
|
||||||
``open_root_actor(registry_addrs=...)``, :doc:`/api/ipc` for
|
|
||||||
the transport/address model the registry stores, and
|
|
||||||
:doc:`/guide/discovery` for the worked walkthrough.
|
|
||||||
|
|
@ -1,102 +0,0 @@
|
||||||
Errors and cancellation types
|
|
||||||
=============================
|
|
||||||
|
|
||||||
``tractor`` extends trio's "exceptions `always propagate`_" rule
|
|
||||||
across the process boundary: a crash in any actor is serialized as
|
|
||||||
an ``Error`` msg, shuttled over IPC, and re-raised in the linked
|
|
||||||
parent scope as a *boxed* :class:`RemoteActorError` — preserving
|
|
||||||
the original type, traceback text and source-actor identity, even
|
|
||||||
across multi-hop relays (a.k.a. "inceptions").
|
|
||||||
|
|
||||||
The most-used types below are importable from ``tractor``
|
|
||||||
directly; the remainder live in ``tractor._exceptions`` (not yet
|
|
||||||
re-exported at top level).
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
Boxed remote errors
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
.. autoexception:: RemoteActorError
|
|
||||||
:members: boxed_type,
|
|
||||||
src_uid,
|
|
||||||
relay_uid,
|
|
||||||
pformat
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with portal.open_context(ep_fn) as (ctx, first):
|
|
||||||
...
|
|
||||||
except tractor.RemoteActorError as rae:
|
|
||||||
if rae.boxed_type is ValueError:
|
|
||||||
... # remote task raised a `ValueError`
|
|
||||||
|
|
||||||
.. autoexception:: ContextCancelled
|
|
||||||
:show-inheritance:
|
|
||||||
:members: canceller
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Inspect :attr:`ContextCancelled.canceller` (the requesting
|
|
||||||
actor's uid) to distinguish a *self*-requested cancel (absorbed
|
|
||||||
at ``open_context()`` exit) from a *cross*-actor cancel (raised
|
|
||||||
locally) — the full rules live in :doc:`/api/context`.
|
|
||||||
|
|
||||||
Typed-messaging errors
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. autoexception:: MsgTypeError
|
|
||||||
:show-inheritance:
|
|
||||||
:members: bad_msg,
|
|
||||||
expected_msg_type
|
|
||||||
|
|
||||||
An "IPC ``TypeError``": a message failed validation against the
|
|
||||||
active msg-spec / ``pld_spec`` (see :doc:`/api/msg`). Raised
|
|
||||||
sender-side for control msgs (``Started``/``Return``) and
|
|
||||||
receiver-side for stream ``Yield`` payloads.
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.StreamOverrun
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
The sender out-paced the receiver's buffer on a
|
|
||||||
:class:`~tractor.MsgStream` opened without
|
|
||||||
``allow_overruns=True``; subtypes :class:`trio.TooSlowError`.
|
|
||||||
|
|
||||||
Transport and runtime errors
|
|
||||||
----------------------------
|
|
||||||
|
|
||||||
.. autoexception:: TransportClosed
|
|
||||||
|
|
||||||
.. autoexception:: ModuleNotExposed
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
Raised when an RPC requests a function from a module not listed
|
|
||||||
in the target actor's ``enable_modules`` allowlist —
|
|
||||||
capability-style access control, not an import bug on your end ;)
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.NoRuntime
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
Raised by :func:`tractor.current_actor` (and friends) when no
|
|
||||||
actor runtime is up in the current process.
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.ActorTooSlowError
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
A peer actor failed to ack a cancel request within the bounded
|
|
||||||
wait — the SC-sanctioned escalation signal from APIs like
|
|
||||||
``Portal.cancel_actor(raise_on_timeout=True)``. Catch it to
|
|
||||||
escalate (e.g. hard-kill via the supervising
|
|
||||||
:class:`~tractor.ActorNursery`); never just ignore it, that's how
|
|
||||||
zombies happen.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/context` for how cancellation and errors flow
|
|
||||||
through a :class:`~tractor.Context`, :doc:`/api/devx` for
|
|
||||||
crash-handling REPL tooling (``debug_mode``, post-mortems),
|
|
||||||
and :doc:`/guide/cancellation` for the full SC-cancellation
|
|
||||||
story.
|
|
||||||
|
|
||||||
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
|
|
||||||
|
|
@ -1,67 +0,0 @@
|
||||||
API reference
|
|
||||||
=============
|
|
||||||
|
|
||||||
This is the curated reference for ``tractor``'s public surface: the
|
|
||||||
names you can import and lean on without reading runtime internals.
|
|
||||||
Everything below is re-exported at the top level (``import
|
|
||||||
tractor``) unless a page says otherwise; subsystems like
|
|
||||||
``tractor.msg``, ``tractor.trionics``, ``tractor.to_asyncio``,
|
|
||||||
``tractor.devx`` and ``tractor.log`` are importable as submodules.
|
|
||||||
|
|
||||||
``tractor`` is "just trio_" extended across processes: every API
|
|
||||||
here is designed to keep the structured concurrency (SC) rules you
|
|
||||||
already know from the `trio docs`_ intact across the process
|
|
||||||
boundary. If a name isn't documented here it's an internal — expect
|
|
||||||
it to change without notice B).
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
Most-used names at a glance:
|
|
||||||
|
|
||||||
.. autosummary::
|
|
||||||
:nosignatures:
|
|
||||||
|
|
||||||
open_root_actor
|
|
||||||
open_nursery
|
|
||||||
run_daemon
|
|
||||||
ActorNursery
|
|
||||||
Portal
|
|
||||||
context
|
|
||||||
Context
|
|
||||||
MsgStream
|
|
||||||
open_actor_cluster
|
|
||||||
find_actor
|
|
||||||
wait_for_actor
|
|
||||||
get_registry
|
|
||||||
current_actor
|
|
||||||
current_ipc_ctx
|
|
||||||
is_root_process
|
|
||||||
get_runtime_vars
|
|
||||||
RemoteActorError
|
|
||||||
ContextCancelled
|
|
||||||
MsgTypeError
|
|
||||||
pause
|
|
||||||
post_mortem
|
|
||||||
Channel
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:maxdepth: 1
|
|
||||||
:caption: Reference pages
|
|
||||||
|
|
||||||
core
|
|
||||||
context
|
|
||||||
discovery
|
|
||||||
errors
|
|
||||||
msg
|
|
||||||
trionics
|
|
||||||
to_asyncio
|
|
||||||
devx
|
|
||||||
ipc
|
|
||||||
|
|
||||||
Where to next? If you're new, start with the runtime and spawning
|
|
||||||
APIs in :doc:`/api/core`, then graduate to the inter-actor task
|
|
||||||
linking model in :doc:`/api/context` — it's the heart of the whole
|
|
||||||
system.
|
|
||||||
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
|
||||||
|
|
@ -1,89 +0,0 @@
|
||||||
IPC and logging
|
|
||||||
===============
|
|
||||||
|
|
||||||
Under every portal, context and stream sits a per-peer
|
|
||||||
:class:`~tractor.Channel`: a msgpack-typed messaging link wrapping
|
|
||||||
one OS transport connection. Transports are pluggable per actor
|
|
||||||
via ``enable_transports=['tcp' | 'uds']`` — TCP is the default,
|
|
||||||
UDS (unix domain sockets) gives you port-less, same-host IPC with
|
|
||||||
kernel-provided peer credentials for free — and exactly **one**
|
|
||||||
transport may currently be enabled per actor.
|
|
||||||
|
|
||||||
.. d2:: diagrams/runtime_stack.d2
|
|
||||||
:caption: Where ``Channel`` sits in the runtime stack.
|
|
||||||
:margin:
|
|
||||||
:alt: layered runtime stack from app code down to transports
|
|
||||||
|
|
||||||
Addresses are "unwrapped" tuples at the API edges:
|
|
||||||
``('host', port)`` for TCP, filesystem-path pairs for UDS. For
|
|
||||||
the full layering story — transport protocols, the IPC server,
|
|
||||||
address types and the msg loop — see
|
|
||||||
:doc:`/explain/architecture`.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor
|
|
||||||
|
|
||||||
``Channel``
|
|
||||||
-----------
|
|
||||||
|
|
||||||
.. autoclass:: Channel
|
|
||||||
:members: from_addr,
|
|
||||||
send,
|
|
||||||
recv,
|
|
||||||
aclose,
|
|
||||||
connected,
|
|
||||||
apply_codec,
|
|
||||||
aid,
|
|
||||||
laddr,
|
|
||||||
raddr,
|
|
||||||
closed
|
|
||||||
|
|
||||||
.. deprecated:: 0.1.0a6
|
|
||||||
|
|
||||||
``Channel.uid`` warns; use :attr:`Channel.aid` which carries
|
|
||||||
richer (optional) identity fields beyond the legacy
|
|
||||||
``(name, uuid)`` pair.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
You rarely construct a :class:`Channel` yourself — the runtime
|
|
||||||
hands them out via :attr:`Portal.chan <tractor.Portal.chan>`
|
|
||||||
and :attr:`Context.chan <tractor.Context.chan>`. Treat the
|
|
||||||
send/recv surface as advanced API: normal apps should speak
|
|
||||||
:class:`~tractor.MsgStream` instead.
|
|
||||||
|
|
||||||
Choosing a transport
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
|
|
||||||
:caption: examples/uds_transport_actor_tree.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Logging
|
|
||||||
-------
|
|
||||||
|
|
||||||
``tractor.log`` provides the structured, colorized console
|
|
||||||
logging used across the runtime — with actor-name + task-aware
|
|
||||||
record headers and extra log levels below :data:`logging.DEBUG`
|
|
||||||
(``'transport'``, ``'runtime'``, ``'cancel'``, ``'devx'``) for
|
|
||||||
spelunking the runtime itself. Use it for your app too: it's
|
|
||||||
already distributed-system aware.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.log
|
|
||||||
|
|
||||||
.. autofunction:: get_logger
|
|
||||||
|
|
||||||
.. autofunction:: get_console_log
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
The ``TRACTOR_LOGLEVEL`` env var overrides any caller-passed
|
|
||||||
``loglevel`` (e.g. to ``open_root_actor()``) so you can crank
|
|
||||||
console verbosity without touching code; subactors inherit
|
|
||||||
the root's level by default.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/explain/architecture` for the transport/server
|
|
||||||
internals, :doc:`/api/discovery` for how channel addresses
|
|
||||||
get registered and found, and :doc:`/api/msg` for the codec
|
|
||||||
layer every channel speaks.
|
|
||||||
107
docs/api/msg.rst
|
|
@ -1,107 +0,0 @@
|
||||||
Typed messaging: ``tractor.msg``
|
|
||||||
================================
|
|
||||||
|
|
||||||
All inter-actor communication rides a small, strictly-typed
|
|
||||||
msgpack wire protocol built from :class:`msgspec.Struct` types —
|
|
||||||
the "SC-shuttle" protocol that powers contexts, streams, RPC and
|
|
||||||
cancellation. You normally never touch these msg types directly
|
|
||||||
(the :class:`~tractor.Context` API speaks them for you) but you
|
|
||||||
*do* use this subpackage to define **payload type contracts**: per
|
|
||||||
endpoint via ``@tractor.context(pld_spec=...)`` or per channel via
|
|
||||||
custom codecs.
|
|
||||||
|
|
||||||
Violations of an active msg-spec surface as
|
|
||||||
:class:`~tractor.MsgTypeError` (see :doc:`/api/errors`); the full
|
|
||||||
typed-payload workflow is shown in ``examples/typed_payloads.py``.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.msg
|
|
||||||
|
|
||||||
The protocol message set
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
.. autosummary::
|
|
||||||
:nosignatures:
|
|
||||||
|
|
||||||
PayloadMsg
|
|
||||||
Aid
|
|
||||||
SpawnSpec
|
|
||||||
Start
|
|
||||||
StartAck
|
|
||||||
Started
|
|
||||||
Yield
|
|
||||||
Stop
|
|
||||||
Return
|
|
||||||
CancelAck
|
|
||||||
Error
|
|
||||||
|
|
||||||
``Aid`` (identity handshake) and ``SpawnSpec`` (parent -> child
|
|
||||||
init) run at connection setup; ``Start``/``StartAck`` initiate an
|
|
||||||
RPC task; ``Started``/``Yield``/``Stop``/``Return`` are the
|
|
||||||
:class:`~tractor.Context` dialog phases; ``CancelAck`` and
|
|
||||||
``Error`` close the loop on cancellation and (boxed) failure.
|
|
||||||
``Msg`` is a legacy alias of ``PayloadMsg``. The union of all of
|
|
||||||
the above is exported as ``MsgType`` (also ``__msg_spec__``).
|
|
||||||
|
|
||||||
.. automodule:: tractor.msg.types
|
|
||||||
:no-members:
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.msg
|
|
||||||
|
|
||||||
Codec construction and override
|
|
||||||
-------------------------------
|
|
||||||
|
|
||||||
.. autofunction:: mk_codec
|
|
||||||
|
|
||||||
.. autoclass:: MsgCodec
|
|
||||||
:members: encode,
|
|
||||||
decode,
|
|
||||||
msg_spec
|
|
||||||
|
|
||||||
.. autofunction:: mk_dec
|
|
||||||
|
|
||||||
.. autoclass:: MsgDec
|
|
||||||
:members: decode,
|
|
||||||
spec
|
|
||||||
|
|
||||||
.. autofunction:: apply_codec
|
|
||||||
|
|
||||||
.. autofunction:: current_codec
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`apply_codec` swaps the codec via a
|
|
||||||
:class:`contextvars.ContextVar` — the override only applies to
|
|
||||||
the *current task* (and tasks it starts), not sibling tasks
|
|
||||||
already running in the actor. Payload-decoding is layered: the
|
|
||||||
outer codec leaves ``.pld`` fields as ``msgspec.Raw`` and each
|
|
||||||
context's payload-receiver decodes them against *its* spec
|
|
||||||
(the "cheap-or-nasty" validation pattern).
|
|
||||||
|
|
||||||
Namespace pointers
|
|
||||||
------------------
|
|
||||||
|
|
||||||
.. autoclass:: NamespacePath
|
|
||||||
:members: from_ref,
|
|
||||||
load_ref,
|
|
||||||
to_tuple
|
|
||||||
|
|
||||||
The ``'module.path:obj_name'`` :class:`str`-subtype used to
|
|
||||||
address every RPC target function over the wire (same format as
|
|
||||||
:func:`pkgutil.resolve_name`).
|
|
||||||
|
|
||||||
Pretty structs
|
|
||||||
--------------
|
|
||||||
|
|
||||||
.. autoclass:: Struct
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
A :class:`msgspec.Struct` subtype with a multi-line pretty
|
|
||||||
``__repr__`` — handy as a base for your own IPC payload types so
|
|
||||||
crash logs stay readable.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/context` for where ``pld_spec`` typing plugs into
|
|
||||||
the ``@context`` decorator, :doc:`/api/errors` for
|
|
||||||
:class:`~tractor.MsgTypeError` semantics, and
|
|
||||||
:doc:`/guide/msging` for the guided typed-messaging tour.
|
|
||||||
|
|
@ -1,97 +0,0 @@
|
||||||
asyncio interop: ``tractor.to_asyncio``
|
|
||||||
=======================================
|
|
||||||
|
|
||||||
"Infected asyncio" mode: spawn an actor with
|
|
||||||
``start_actor(..., infect_asyncio=True)`` and its process runs
|
|
||||||
:mod:`trio` as a `guest`_ on top of the :mod:`asyncio` loop —
|
|
||||||
letting your trio task tree drive asyncio tasks *in the same
|
|
||||||
process* while the rest of the actor tree stays pure trio. Each
|
|
||||||
trio <-> asyncio task pair is linked with structured concurrency
|
|
||||||
(SC) semantics: error or cancellation on either side tears down
|
|
||||||
both, with the cause translated cross-loop.
|
|
||||||
|
|
||||||
.. d2:: diagrams/infected_aio.d2
|
|
||||||
:caption: A trio guest driving asyncio tasks in one actor.
|
|
||||||
:margin:
|
|
||||||
:alt: trio guest mode inside an asyncio-infected actor
|
|
||||||
|
|
||||||
See ``examples/infected_asyncio_echo_server.py`` for a complete
|
|
||||||
worked example.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.to_asyncio
|
|
||||||
|
|
||||||
Starting asyncio tasks from trio
|
|
||||||
--------------------------------
|
|
||||||
|
|
||||||
.. autofunction:: open_channel_from
|
|
||||||
|
|
||||||
.. autofunction:: run_task
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`open_channel_from` mirrors the
|
|
||||||
``Portal.open_context()`` handshake: the asyncio side calls
|
|
||||||
``chan.started_nowait(value)`` and that value pops out as
|
|
||||||
``first`` on the trio side. :func:`run_task` is the one-shot
|
|
||||||
form — run a single asyncio-compatible coroutine fn and return
|
|
||||||
its result to trio.
|
|
||||||
|
|
||||||
The inter-loop channel
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. autoclass:: LinkedTaskChannel
|
|
||||||
:members: send,
|
|
||||||
receive,
|
|
||||||
wait_for_result,
|
|
||||||
subscribe,
|
|
||||||
started_nowait,
|
|
||||||
send_nowait,
|
|
||||||
get,
|
|
||||||
cancel_asyncio_task,
|
|
||||||
closed
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
The trio side uses the async API
|
|
||||||
(:meth:`LinkedTaskChannel.send` /
|
|
||||||
:meth:`LinkedTaskChannel.receive`); the asyncio side uses the
|
|
||||||
loop-safe sync/await mix
|
|
||||||
(:meth:`LinkedTaskChannel.send_nowait` /
|
|
||||||
:meth:`LinkedTaskChannel.get` /
|
|
||||||
:meth:`LinkedTaskChannel.started_nowait`).
|
|
||||||
|
|
||||||
Translated exception types
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
Cross-loop failures are re-raised on the *other* side as one of
|
|
||||||
these explicit translation types, so you always know which loop
|
|
||||||
actually died first:
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.AsyncioCancelled
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.AsyncioTaskExited
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.TrioCancelled
|
|
||||||
|
|
||||||
.. autoexception:: tractor._exceptions.TrioTaskExited
|
|
||||||
|
|
||||||
.. autoexception:: AsyncioRuntimeTranslationError
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
Guest-mode entrypoint
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
``run_as_asyncio_guest()`` is the runtime-internal entrypoint that
|
|
||||||
boots trio in `guest`_ mode inside an infected actor — you get it
|
|
||||||
implicitly via ``infect_asyncio=True`` and shouldn't need to call
|
|
||||||
it yourself.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/core` for the ``infect_asyncio`` spawn flag,
|
|
||||||
:meth:`tractor.Actor.is_infected_aio` for runtime
|
|
||||||
introspection, :doc:`/api/devx` for using the debugger from
|
|
||||||
inside asyncio tasks, and :doc:`/guide/asyncio` for the
|
|
||||||
guided tour.
|
|
||||||
|
|
||||||
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
|
|
||||||
|
|
@ -1,76 +0,0 @@
|
||||||
Trio patterns: ``tractor.trionics``
|
|
||||||
===================================
|
|
||||||
|
|
||||||
Sugary structured concurrency (SC) patterns for plain :mod:`trio`
|
|
||||||
code — **no actor runtime required**. These helpers grew out of
|
|
||||||
real distributed-system needs in ``tractor`` apps but every one of
|
|
||||||
them works in a single-process program too; import via
|
|
||||||
``from tractor import trionics``.
|
|
||||||
|
|
||||||
.. currentmodule:: tractor.trionics
|
|
||||||
|
|
||||||
Context-manager helpers
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
.. autofunction:: gather_contexts
|
|
||||||
|
|
||||||
.. autofunction:: maybe_open_context
|
|
||||||
|
|
||||||
.. autofunction:: maybe_open_nursery
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`gather_contexts` is "a nursery for async context
|
|
||||||
managers": it enters N acms concurrently and yields their
|
|
||||||
values in input order. :func:`maybe_open_context` is the
|
|
||||||
actor-wide cache/multiplex layer on top — the first task pays
|
|
||||||
the acm setup cost, later callers get ``(cache_hit=True, ...)``
|
|
||||||
and share the same value until all users exit.
|
|
||||||
|
|
||||||
Broadcast fan-out
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
.. autofunction:: broadcast_receiver
|
|
||||||
|
|
||||||
.. autoclass:: BroadcastReceiver
|
|
||||||
:members: receive,
|
|
||||||
subscribe,
|
|
||||||
aclose
|
|
||||||
|
|
||||||
.. autoexception:: Lagged
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
A single-producer, many-consumer broadcast layer over any
|
|
||||||
``trio``-style receive channel: non-lossy for the *fastest*
|
|
||||||
consumer while slower consumers raise :class:`Lagged` (a
|
|
||||||
:class:`trio.TooSlowError` subtype) once they fall behind the
|
|
||||||
internal ring. This is exactly the machinery behind
|
|
||||||
:meth:`tractor.MsgStream.subscribe` — see
|
|
||||||
``examples/streaming_broadcast_fanout.py``.
|
|
||||||
|
|
||||||
ExceptionGroup helpers
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. autofunction:: collapse_eg
|
|
||||||
|
|
||||||
.. autofunction:: maybe_raise_from_masking_exc
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
:func:`collapse_eg` "un-nests" single-exception
|
|
||||||
:class:`ExceptionGroup` wrappers from strict-eg ``trio``
|
|
||||||
nurseries so your ``except`` clauses match the original error;
|
|
||||||
:func:`maybe_raise_from_masking_exc` surfaces real errors that
|
|
||||||
would otherwise be masked by :class:`trio.Cancelled` during
|
|
||||||
teardown.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/api/context` for the IPC-stream consumer of
|
|
||||||
:class:`BroadcastReceiver`, :doc:`/guide/streaming` for
|
|
||||||
fan-out in a worked pipeline, and the `trio docs`_ for the
|
|
||||||
underlying channel and `nursery`_ semantics these helpers
|
|
||||||
compose.
|
|
||||||
|
|
||||||
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
|
||||||
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
|
||||||
198
docs/conf.py
|
|
@ -1,147 +1,105 @@
|
||||||
# tractor: distributed structured concurrency.
|
# Configuration file for the Sphinx documentation builder.
|
||||||
'''
|
#
|
||||||
Sphinx config for `tractor`'s documentation.
|
# This file only contains a selection of the most common options. For a full
|
||||||
|
# list see the documentation:
|
||||||
|
# https://www.sphinx-doc.org/en/master/usage/configuration.html
|
||||||
|
|
||||||
Theme-wise we ride the `pydata_sphinx_theme` (per the
|
# -- Path setup --------------------------------------------------------------
|
||||||
research + history in #157) skinned to a minimal
|
|
||||||
black + white look via `_static/css/custom.css`; see
|
|
||||||
the local extensions under `_ext/` for our `.. d2::`
|
|
||||||
diagram and `.. margin::` aside directives.
|
|
||||||
|
|
||||||
Build locally via,
|
# If extensions (or modules to document with autodoc) are in another directory,
|
||||||
|
# add these directories to sys.path here. If the directory is relative to the
|
||||||
|
# documentation root, use os.path.abspath to make it absolute, like shown here.
|
||||||
|
#
|
||||||
|
# import os
|
||||||
|
# import sys
|
||||||
|
# sys.path.insert(0, os.path.abspath('.'))
|
||||||
|
|
||||||
uv run --group docs make -C docs html
|
# Warn about all references to unknown targets
|
||||||
|
nitpicky = True
|
||||||
|
|
||||||
'''
|
# The master toctree document.
|
||||||
from importlib.metadata import version as get_version
|
master_doc = 'index'
|
||||||
from pathlib import Path
|
|
||||||
import sys
|
|
||||||
|
|
||||||
# local sphinx extensions live in `_ext/`:
|
# -- Project information -----------------------------------------------------
|
||||||
# - `d2diagrams`: `.. d2::` diagram rendering
|
|
||||||
# - `marginalia`: `.. margin::` RHS prose-asides
|
|
||||||
sys.path.insert(
|
|
||||||
0,
|
|
||||||
str((Path(__file__).parent / '_ext').resolve()),
|
|
||||||
)
|
|
||||||
|
|
||||||
# -- project info ---------------------------------
|
|
||||||
|
|
||||||
project = 'tractor'
|
project = 'tractor'
|
||||||
copyright = '2018-2026, Tyler Goodlet'
|
copyright = '2018, Tyler Goodlet'
|
||||||
author = 'Tyler Goodlet'
|
author = 'Tyler Goodlet'
|
||||||
release: str = get_version('tractor')
|
|
||||||
version: str = release
|
|
||||||
|
|
||||||
# -- general config -------------------------------
|
# The full version, including alpha/beta/rc tags
|
||||||
|
release = '0.0.0a0.dev0'
|
||||||
|
|
||||||
|
# -- General configuration ---------------------------------------------------
|
||||||
|
|
||||||
|
# Add any Sphinx extension module names here, as strings. They can be
|
||||||
|
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||||
|
# ones.
|
||||||
extensions = [
|
extensions = [
|
||||||
'sphinx.ext.autodoc',
|
'sphinx.ext.autodoc',
|
||||||
'sphinx.ext.autosummary',
|
|
||||||
'sphinx.ext.intersphinx',
|
'sphinx.ext.intersphinx',
|
||||||
'sphinx.ext.viewcode',
|
|
||||||
'sphinx.ext.todo',
|
'sphinx.ext.todo',
|
||||||
'sphinx_design',
|
|
||||||
'sphinx_copybutton',
|
|
||||||
'sphinxext.opengraph',
|
|
||||||
'sphinx_togglebutton',
|
|
||||||
'd2diagrams',
|
|
||||||
'marginalia',
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
# Add any paths that contain templates here, relative to this directory.
|
||||||
templates_path = ['_templates']
|
templates_path = ['_templates']
|
||||||
exclude_patterns = [
|
|
||||||
'_build',
|
|
||||||
# the pypi/gh readme + its standalone generator
|
|
||||||
# sub-project; NOT doc-tree pages.
|
|
||||||
'README.rst',
|
|
||||||
'github_readme',
|
|
||||||
'Thumbs.db',
|
|
||||||
'.DS_Store',
|
|
||||||
]
|
|
||||||
root_doc = 'index'
|
|
||||||
# TODO: flip this on + burn down the (many) warnings
|
|
||||||
# from our informal docstring style; see the autodoc
|
|
||||||
# readiness notes from the revamp's recon pass.
|
|
||||||
nitpicky = False
|
|
||||||
|
|
||||||
# -- autodoc/autosummary --------------------------
|
# List of patterns, relative to source directory, that match files and
|
||||||
|
# directories to ignore when looking for source files.
|
||||||
|
# This pattern also affects html_static_path and html_extra_path.
|
||||||
|
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
|
||||||
|
|
||||||
autodoc_member_order = 'bysource'
|
|
||||||
autodoc_typehints = 'description'
|
|
||||||
autosummary_generate = True
|
|
||||||
|
|
||||||
# -- intersphinx ----------------------------------
|
# -- Options for HTML output -------------------------------------------------
|
||||||
|
|
||||||
intersphinx_mapping = {
|
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||||
'python': (
|
# a list of builtin themes.
|
||||||
'https://docs.python.org/3',
|
#
|
||||||
None,
|
html_theme = 'sphinx_book_theme'
|
||||||
),
|
|
||||||
'trio': (
|
pygments_style = 'algol_nu'
|
||||||
'https://trio.readthedocs.io/en/stable',
|
|
||||||
None,
|
# Theme options are theme-specific and customize the look and feel of a theme
|
||||||
),
|
# further. For a list of options available for each theme, see the
|
||||||
# NOTE, msgspec's site doesn't publish an
|
# documentation.
|
||||||
# `objects.inv` (404s) so no intersphinx for it.
|
html_theme_options = {
|
||||||
'pytest': (
|
# 'logo': 'tractor_logo_side.svg',
|
||||||
'https://docs.pytest.org/en/stable',
|
# 'description': 'Structured concurrent "actors"',
|
||||||
None,
|
"repository_url": "https://github.com/goodboy/tractor",
|
||||||
),
|
"use_repository_button": True,
|
||||||
|
"home_page_in_toc": False,
|
||||||
|
"show_toc_level": 1,
|
||||||
|
"path_to_docs": "docs",
|
||||||
|
|
||||||
|
}
|
||||||
|
html_sidebars = {
|
||||||
|
"**": [
|
||||||
|
"sbt-sidebar-nav.html",
|
||||||
|
# "sidebar-search-bs.html",
|
||||||
|
# 'localtoc.html',
|
||||||
|
],
|
||||||
|
# 'logo.html',
|
||||||
|
# 'github.html',
|
||||||
|
# 'relations.html',
|
||||||
|
# 'searchbox.html'
|
||||||
|
# ]
|
||||||
}
|
}
|
||||||
|
|
||||||
# -- html output ----------------------------------
|
# doesn't seem to work?
|
||||||
|
# extra_navbar = "<p>nextttt-gennnnn</p>"
|
||||||
|
|
||||||
html_theme = 'pydata_sphinx_theme'
|
html_title = ''
|
||||||
html_title = 'tractor'
|
|
||||||
html_logo = '_static/tractor_logo_side.svg'
|
html_logo = '_static/tractor_logo_side.svg'
|
||||||
html_favicon = '_static/tractor_logo_side.svg'
|
html_favicon = '_static/tractor_logo_side.svg'
|
||||||
|
# show_navbar_depth = 1
|
||||||
|
|
||||||
|
# Add any paths that contain custom static files (such as style sheets) here,
|
||||||
|
# relative to this directory. They are copied after the builtin static files,
|
||||||
|
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||||
html_static_path = ['_static']
|
html_static_path = ['_static']
|
||||||
html_css_files = ['css/custom.css']
|
|
||||||
html_show_sourcelink = False
|
# Example configuration for intersphinx: refer to the Python standard library.
|
||||||
html_theme_options = {
|
intersphinx_mapping = {
|
||||||
'logo': {
|
"python": ("https://docs.python.org/3", None),
|
||||||
'image_light': '_static/tractor_logo_side.svg',
|
"pytest": ("https://docs.pytest.org/en/latest", None),
|
||||||
'image_dark': '_static/tractor_logo_side.svg',
|
"setuptools": ("https://setuptools.readthedocs.io/en/latest", None),
|
||||||
'alt_text': 'tractor',
|
|
||||||
},
|
|
||||||
'github_url': 'https://github.com/goodboy/tractor',
|
|
||||||
'navbar_align': 'content',
|
|
||||||
'show_toc_level': 2,
|
|
||||||
'secondary_sidebar_items': {
|
|
||||||
'**': ['page-toc'],
|
|
||||||
'index': [],
|
|
||||||
},
|
|
||||||
'use_edit_page_button': True,
|
|
||||||
'footer_start': ['copyright'],
|
|
||||||
'footer_end': ['theme-version'],
|
|
||||||
'pygments_light_style': 'algol_nu',
|
|
||||||
'pygments_dark_style': 'github-dark',
|
|
||||||
}
|
}
|
||||||
html_context = {
|
|
||||||
'github_user': 'goodboy',
|
|
||||||
'github_repo': 'tractor',
|
|
||||||
'github_version': 'main',
|
|
||||||
'doc_path': 'docs',
|
|
||||||
}
|
|
||||||
|
|
||||||
# -- ext: opengraph -------------------------------
|
|
||||||
|
|
||||||
ogp_site_url = 'https://goodboy.github.io/tractor/'
|
|
||||||
ogp_use_first_image = True
|
|
||||||
|
|
||||||
# -- ext: copybutton ------------------------------
|
|
||||||
|
|
||||||
copybutton_prompt_text = r'>>> |\.\.\. |\$ '
|
|
||||||
copybutton_prompt_is_regexp = True
|
|
||||||
|
|
||||||
# -- ext: todo ------------------------------------
|
|
||||||
|
|
||||||
todo_include_todos = True
|
|
||||||
|
|
||||||
# -- ext: d2diagrams (local) ----------------------
|
|
||||||
|
|
||||||
# normally resolved from PATH or the `D2_BIN` env
|
|
||||||
# var; when neither hits, the committed SVGs under
|
|
||||||
# `_diagrams/` are used as-is.
|
|
||||||
d2_bin = None
|
|
||||||
d2_args = []
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
Hot tips for ``tractor`` hackers
|
||||||
|
================================
|
||||||
|
|
||||||
|
This is a WIP guide for newcomers to the project mostly to do with
|
||||||
|
dev, testing, CI and release gotchas, reminders and best practises.
|
||||||
|
|
||||||
|
``tractor`` is a fairly novel project compared to most since it is
|
||||||
|
effectively a new way of doing distributed computing in Python and is
|
||||||
|
much closer to working with an "application level runtime" (like erlang
|
||||||
|
OTP or scala's akka project) then it is a traditional Python library.
|
||||||
|
As such, having an arsenal of tools and recipes for figuring out the
|
||||||
|
right way to debug problems when they do arise is somewhat of
|
||||||
|
a necessity.
|
||||||
|
|
||||||
|
|
||||||
|
Making a Release
|
||||||
|
----------------
|
||||||
|
We currently do nothing special here except the traditional
|
||||||
|
PyPa release recipe as in `documented by twine`_. I personally
|
||||||
|
create sub-dirs within the generated `dist/` with an explicit
|
||||||
|
release name such as `alpha3/` when there's been a sequence of
|
||||||
|
releases I've made, but it really is up to you how you like to
|
||||||
|
organize generated sdists locally.
|
||||||
|
|
||||||
|
The resulting build cmds are approximately:
|
||||||
|
|
||||||
|
.. code:: bash
|
||||||
|
|
||||||
|
python setup.py sdist -d ./dist/XXX.X/
|
||||||
|
|
||||||
|
twine upload -r testpypi dist/XXX.X/*
|
||||||
|
|
||||||
|
twine upload dist/XXX.X/*
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
.. _documented by twine: https://twine.readthedocs.io/en/latest/#using-twine
|
||||||
|
|
||||||
|
|
||||||
|
Debugging and monitoring actor trees
|
||||||
|
------------------------------------
|
||||||
|
TODO: but there are tips in the readme for some terminal commands
|
||||||
|
which can be used to see the process trees easily on Linux.
|
||||||
|
|
||||||
|
|
||||||
|
Using the log system to trace `trio` task flow
|
||||||
|
----------------------------------------------
|
||||||
|
TODO: the logging system is meant to be oriented around
|
||||||
|
stack "layers" of the runtime such that you can track
|
||||||
|
"logical abstraction layers" in the code such as errors, cancellation,
|
||||||
|
IPC and streaming, and the low level transport and wire protocols.
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
# tractor docs diagram: the hero supervision tree.
|
|
||||||
# A process tree of trio-task-trees; every arrow is
|
|
||||||
# a parent which *must wait* on its children.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
direction: down
|
|
||||||
root: "root actor" {
|
|
||||||
main: "main()"
|
|
||||||
an: "ActorNursery"
|
|
||||||
main -> an: opens
|
|
||||||
}
|
|
||||||
w0: "subactor\n'worker_0'" {
|
|
||||||
tasks: "trio task tree"
|
|
||||||
}
|
|
||||||
w1: "subactor\n'worker_1'" {
|
|
||||||
main: "serve()"
|
|
||||||
an: "ActorNursery"
|
|
||||||
main -> an: opens
|
|
||||||
}
|
|
||||||
gc: "subactor\n'deeper'" {
|
|
||||||
tasks: "trio task tree"
|
|
||||||
}
|
|
||||||
root.an -> w0: "spawns +\nsupervises"
|
|
||||||
root.an -> w1: "spawns +\nsupervises"
|
|
||||||
w1.an -> gc: "spawns +\nsupervises"
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
# tractor docs diagram: the inter-actor `Context`
|
|
||||||
# dialog; tractor's SC-transitive msg protocol as
|
|
||||||
# seen from both sides of `Portal.open_context()`.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
shape: sequence_diagram
|
|
||||||
parent: "parent task\n(Portal)"
|
|
||||||
child: "child actor task\n(@tractor.context fn)"
|
|
||||||
parent -> child: "Start: open_context(fn, **kwargs)"
|
|
||||||
child -> parent: "StartAck"
|
|
||||||
child -> parent: "Started: await ctx.started(value)"
|
|
||||||
parent -> child: "Yield: stream.send()"
|
|
||||||
child -> parent: "Yield: stream.send()"
|
|
||||||
parent -> child: "Stop: stream closed"
|
|
||||||
child -> parent: "Return: fn return value"
|
|
||||||
|
|
@ -1,19 +0,0 @@
|
||||||
# tractor docs diagram: multi-actor debugger REPL
|
|
||||||
# serialization via the root actor's tty lock.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
shape: sequence_diagram
|
|
||||||
mary: "subactor 'mary'"
|
|
||||||
root: "root actor\n(owns the tty)"
|
|
||||||
bob: "subactor 'bob'"
|
|
||||||
mary -> root: "tractor.pause(): acquire tty lock"
|
|
||||||
root -> mary: "granted: pdb REPL is yours"
|
|
||||||
bob -> root: "tractor.pause(): acquire tty lock"
|
|
||||||
mary -> root: "continue: release lock"
|
|
||||||
root -> bob: "granted: pdb REPL is yours"
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
# tractor docs diagram: one-cancels-all error
|
|
||||||
# propagation up a (sub-)actor tree; no zombies,
|
|
||||||
# no lost errors.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
direction: down
|
|
||||||
root: "root actor\n(ActorNursery)"
|
|
||||||
gertie: "subactor 'gertie'\n(healthy, gets cancelled)"
|
|
||||||
bobbie: "subactor 'bobbie'\nraises NameError"
|
|
||||||
root -> gertie: supervises
|
|
||||||
root -> bobbie: supervises
|
|
||||||
bobbie -> root: "RemoteActorError\n(boxed NameError)" {
|
|
||||||
style: {
|
|
||||||
stroke-dash: 3
|
|
||||||
}
|
|
||||||
}
|
|
||||||
root -> gertie: "cancel()" {
|
|
||||||
style: {
|
|
||||||
stroke-dash: 3
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
||||||
# tractor docs diagram: "infected asyncio" mode;
|
|
||||||
# trio runs as a guest on the asyncio loop with
|
|
||||||
# SC supervision linking tasks across both.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
direction: down
|
|
||||||
sub: "subactor process (infect_asyncio=True)" {
|
|
||||||
aio: "asyncio event loop" {
|
|
||||||
aiotask: "asyncio.Task\naio_echo_server()"
|
|
||||||
trio: "trio (guest mode)" {
|
|
||||||
triotask: "trio task\n@tractor.context fn"
|
|
||||||
}
|
|
||||||
trio.triotask <-> aiotask: "LinkedTaskChannel\n.send()/.receive()"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
parent: "parent actor (pure trio)"
|
|
||||||
parent <-> sub.aio.trio.triotask: "IPC: Context + MsgStream"
|
|
||||||
|
|
@ -1,16 +0,0 @@
|
||||||
# tractor docs diagram: the layered runtime view
|
|
||||||
# inside any single actor process.
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
grid-rows: 4
|
|
||||||
grid-gap: 0
|
|
||||||
app: "your app: plain trio tasks + nurseries"
|
|
||||||
tractor: "tractor runtime: actors, portals,\ncontexts + streams, RPC"
|
|
||||||
ipc: "IPC Channel: msgspec-typed msgs\nover TCP | UDS transports"
|
|
||||||
os: "OS: one process per actor"
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
# tractor docs diagram: multi-actor streaming
|
|
||||||
# pipeline topology from
|
|
||||||
# examples/full_fledged_streaming_service.py
|
|
||||||
vars: {
|
|
||||||
d2-config: {
|
|
||||||
sketch: true
|
|
||||||
theme-id: 1
|
|
||||||
pad: 16
|
|
||||||
layout-engine: elk
|
|
||||||
}
|
|
||||||
}
|
|
||||||
direction: right
|
|
||||||
s0: "actor 'streamer_0'" {
|
|
||||||
fn: "stream_data(0)"
|
|
||||||
}
|
|
||||||
s1: "actor 'streamer_1'" {
|
|
||||||
fn: "stream_data(1)"
|
|
||||||
}
|
|
||||||
agg: "actor 'aggregator'" {
|
|
||||||
fn: "aggregate()"
|
|
||||||
}
|
|
||||||
main: "root actor" {
|
|
||||||
fn: "main()"
|
|
||||||
}
|
|
||||||
s0 -> agg: "async for:\nyielded ints"
|
|
||||||
s1 -> agg: "async for:\nyielded ints"
|
|
||||||
agg -> main: "deduped\nstream"
|
|
||||||
|
|
@ -1,368 +0,0 @@
|
||||||
Anatomy of the runtime
|
|
||||||
======================
|
|
||||||
|
|
||||||
You can get a long way with ``tractor`` by treating it as "trio_
|
|
||||||
with nurseries that spawn processes". But once you start asking
|
|
||||||
*where does my msg actually go?*, *which process is that?* or
|
|
||||||
*who keeps the phonebook?*, it pays to know how the runtime hangs
|
|
||||||
together. This page walks the stack top to bottom.
|
|
||||||
|
|
||||||
.. d2:: diagrams/runtime_stack.d2
|
|
||||||
:caption: The four runtime layers inside *every* actor process.
|
|
||||||
:alt: layer cake of app tasks, tractor runtime, IPC, OS process
|
|
||||||
:width: 70%
|
|
||||||
|
|
||||||
The layer cake
|
|
||||||
--------------
|
|
||||||
|
|
||||||
Every actor process is the same four-layer sandwich:
|
|
||||||
|
|
||||||
- **your app**: plain ``trio`` tasks, nurseries and cancel
|
|
||||||
scopes; nothing special. ``tractor`` is a `structured
|
|
||||||
concurrency`_ (SC) multi-processing runtime built on trio_ and
|
|
||||||
the whole pitch is that this layer stays *just trio*: no
|
|
||||||
callbacks, no futures, no proxy objects.
|
|
||||||
- **the** ``tractor`` **runtime**: a per-process
|
|
||||||
:class:`tractor.Actor` running the msg loop and RPC task
|
|
||||||
scheduler, plus the user-facing primitives layered on it:
|
|
||||||
:class:`tractor.ActorNursery` (spawning + supervision),
|
|
||||||
:class:`tractor.Portal` (calling into a peer) and
|
|
||||||
:class:`tractor.Context` + :class:`tractor.MsgStream`
|
|
||||||
(SC-linked cross-actor task pairs and streaming).
|
|
||||||
- **IPC channels**: one :class:`tractor.Channel` per connected
|
|
||||||
peer, each wrapping a ``MsgTransport`` that ships
|
|
||||||
msgspec_-typed msgs over TCP or UDS.
|
|
||||||
- **the OS**: one process per actor, started by a swappable
|
|
||||||
spawn backend.
|
|
||||||
|
|
||||||
The property that holds it all together: SC composes *through*
|
|
||||||
the layers. A crash in a leaf actor's app task unwinds that
|
|
||||||
actor's trio tree, ships across its IPC channel as a typed
|
|
||||||
``Error`` msg, and unwinds the parent's trio tree in turn — the
|
|
||||||
"SC-transitive supervision protocol" from the README's pitch.
|
|
||||||
The whole tree cancels and errors like one big trio program; it
|
|
||||||
just happens to be spread across processes.
|
|
||||||
|
|
||||||
One actor, one process, one ``trio.run()``
|
|
||||||
------------------------------------------
|
|
||||||
|
|
||||||
A ``tractor`` "actor" is not a green thread, nor an object with
|
|
||||||
a mailbox, nor a coroutine: it's one OS process running one
|
|
||||||
:func:`trio.run` whose root task boots the runtime machinery —
|
|
||||||
msg loop, RPC task scheduler, IPC server — all embodied by a
|
|
||||||
single :class:`tractor.Actor` instance.
|
|
||||||
|
|
||||||
.. margin:: Shared nothing
|
|
||||||
|
|
||||||
Processes buy you a real `shared nothing architecture`_: no
|
|
||||||
accidentally-shared mutable state, no GIL contention, and
|
|
||||||
every actor can be inspected (or killed) like any other OS
|
|
||||||
process.
|
|
||||||
|
|
||||||
You rarely construct an :class:`~tractor.Actor` yourself; the
|
|
||||||
runtime makes exactly one per process and you grab it with
|
|
||||||
:func:`tractor.current_actor`:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
actor = tractor.current_actor() # NoRuntime if none running
|
|
||||||
print(actor.aid.name) # str name, need not be unique
|
|
||||||
print(actor.aid.uuid) # uuid4 str, IS unique
|
|
||||||
print(actor.aid.pid) # the OS pid
|
|
||||||
print(actor.uid) # legacy (name, uuid) pair
|
|
||||||
|
|
||||||
Identity is carried by the ``Aid`` msg-struct (see
|
|
||||||
``tractor.msg.types``): a ``name``/``uuid``/``pid`` triple
|
|
||||||
exchanged in the very first "mailbox handshake" whenever two
|
|
||||||
actors connect. It's what the registrar stores and what shows up
|
|
||||||
in logs and proc-titles. The older ``.uid`` 2-tuple of
|
|
||||||
``(name, uuid)`` predates ``Aid`` and is still pervasive across
|
|
||||||
the codebase; treat it as the legacy spelling of the same
|
|
||||||
identity.
|
|
||||||
|
|
||||||
If this smells like the `actor model`_, sure — but as the
|
|
||||||
README warns, it probably doesn't look like what you *think* an
|
|
||||||
actor model looks like, and that's intentional. Here an "actor"
|
|
||||||
is purely a runtime-unit-of-abstraction: process +
|
|
||||||
``trio.run()`` + IPC machinery.
|
|
||||||
|
|
||||||
IPC: channels, transports, addresses
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
Two connected actors talk through a :class:`tractor.Channel`: a
|
|
||||||
duplex, per-peer msg pipe. Each ``Channel`` wraps a
|
|
||||||
``MsgTransport`` instance which does the wire work: framing,
|
|
||||||
encode/decode and the socket itself. The encoding is msgpack
|
|
||||||
(via msgspec_) and *every* msg is an instance of one of the
|
|
||||||
runtime's tagged-union :class:`msgspec.Struct` types: the
|
|
||||||
``Aid`` handshake, ``Start``/``StartAck`` (RPC init),
|
|
||||||
``Started``/``Yield``/``Stop``/``Return`` (the ctx dialog
|
|
||||||
phases), ``Error``, etc. There is no raw-bytes mode; the
|
|
||||||
msg-spec *is* the protocol, which is exactly what lets payloads
|
|
||||||
be type-limited per-context (see ``pld_spec`` in
|
|
||||||
:doc:`/guide/context`).
|
|
||||||
|
|
||||||
Addresses come in two spellings:
|
|
||||||
|
|
||||||
- *unwrapped*: the plain-tuple form you pass to user APIs —
|
|
||||||
``('127.0.0.1', 1616)`` for tcp, or a
|
|
||||||
``(<filedir>, <filename>)`` path-pair for uds;
|
|
||||||
- *wrapped*: the internal ``TCPAddress``/``UDSAddress`` struct
|
|
||||||
types (plus libp2p-style multiaddr helpers over in
|
|
||||||
``tractor.discovery``).
|
|
||||||
|
|
||||||
You only ever need the tuple form; the runtime wraps and
|
|
||||||
unwraps at the boundaries.
|
|
||||||
|
|
||||||
TCP: the boring default
|
|
||||||
***********************
|
|
||||||
|
|
||||||
The default transport (``'tcp'``) binds each actor's IPC server
|
|
||||||
to loopback ``('127.0.0.1', <random port>)`` unless told
|
|
||||||
otherwise, and is the only choice when your tree spans hosts.
|
|
||||||
Nothing exotic: ``trio`` TCP streams + length-prefixed msgpack
|
|
||||||
framing.
|
|
||||||
|
|
||||||
UDS: same-host, creds included
|
|
||||||
******************************
|
|
||||||
|
|
||||||
Pass ``enable_transports=['uds']`` and actors instead talk over
|
|
||||||
unix-domain sockets, with socket files placed in the per-user
|
|
||||||
runtime dir (``$XDG_RUNTIME_DIR/tractor/`` on linux, the
|
|
||||||
``platformdirs`` equivalent elsewhere). Two perks over tcp on a
|
|
||||||
single host:
|
|
||||||
|
|
||||||
- no ports to fight over; addrs are just file paths,
|
|
||||||
- the kernel snitches on your peer for free: the listening side
|
|
||||||
reads the connector's ``pid`` (plus ``uid``/``gid`` on linux)
|
|
||||||
straight off the socket via ``SO_PEERCRED`` /
|
|
||||||
``LOCAL_PEERPID`` — no extra handshake msgs required B)
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
Socket-file lifetime == listening actor lifetime. On
|
|
||||||
listener teardown the runtime ``os.unlink()``\s the socket
|
|
||||||
file immediately, so any *late* connection attempt (say, a
|
|
||||||
sub-actor racing to deregister with a registrar that's
|
|
||||||
already shutting down) fails with ``FileNotFoundError``.
|
|
||||||
And ofc, UDS is same-host only.
|
|
||||||
|
|
||||||
Here's a full actor tree run entirely over uds:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/uds_transport_actor_tree.py
|
|
||||||
:caption: examples/uds_transport_actor_tree.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Picking a transport
|
|
||||||
*******************
|
|
||||||
|
|
||||||
Transport choice is per-actor via the ``enable_transports``
|
|
||||||
kwarg accepted by :func:`tractor.open_root_actor` (and proxied
|
|
||||||
through ``open_nursery()`` when it implicitly boots the
|
|
||||||
runtime), plus per-child via
|
|
||||||
``ActorNursery.start_actor(enable_transports=...)``. Two rules
|
|
||||||
the runtime enforces today:
|
|
||||||
|
|
||||||
- exactly ONE transport per actor: multi-transport actors are
|
|
||||||
on the roadmap but currently raise ``RuntimeError``;
|
|
||||||
- your ``registry_addrs`` protos must all be in
|
|
||||||
``enable_transports``: mismatches fail fast with
|
|
||||||
``ValueError`` instead of (as in darker times) hanging the
|
|
||||||
registrar handshake forever.
|
|
||||||
|
|
||||||
Spawn backends
|
|
||||||
--------------
|
|
||||||
|
|
||||||
How does an actor actually *become* a process? Via a swappable
|
|
||||||
spawn backend, selected with the ``start_method`` kwarg to
|
|
||||||
:func:`tractor.open_root_actor`:
|
|
||||||
|
|
||||||
``'trio'`` (default)
|
|
||||||
The home-grown spawner: re-exec the child as
|
|
||||||
``python -m tractor._child`` using ``trio``'s subprocess
|
|
||||||
machinery, then bootstrap it over the first IPC exchange
|
|
||||||
(the parent ships a ``SpawnSpec`` msg carrying all init
|
|
||||||
state). Supported on all platforms and the most battle
|
|
||||||
tested choice by far.
|
|
||||||
|
|
||||||
``'mp_spawn'`` / ``'mp_forkserver'``
|
|
||||||
The stdlib :mod:`multiprocessing` start-methods of the same
|
|
||||||
names (forkserver is posix-only). Mostly interesting for
|
|
||||||
ecosystem compat and start-up-latency tuning.
|
|
||||||
|
|
||||||
``'subint'`` (experimental, py3.14+)
|
|
||||||
Each actor runs as a `PEP 734`_ sub-interpreter
|
|
||||||
(``concurrent.interpreters``) driven on its own OS thread
|
|
||||||
*inside the parent process*: interpreter-level
|
|
||||||
shared-nothing isolation with much faster start-up. Yes,
|
|
||||||
this bends the one-actor-one-process rule; the rest of the
|
|
||||||
model is unchanged.
|
|
||||||
|
|
||||||
The ``TRACTOR_SPAWN_METHOD`` env-var beats any caller-passed
|
|
||||||
``start_method``, so you can swap backends under an unmodified
|
|
||||||
app:
|
|
||||||
|
|
||||||
.. code:: bash
|
|
||||||
|
|
||||||
TRACTOR_SPAWN_METHOD=mp_forkserver python my_app.py
|
|
||||||
|
|
||||||
One current limitation worth knowing: ``debug_mode=True`` (the
|
|
||||||
crash-to-REPL machinery) is only supported on backends whose
|
|
||||||
child-side runtime is trio-native, e.g. the default ``'trio'``;
|
|
||||||
see :doc:`/guide/debugging` for the deats.
|
|
||||||
|
|
||||||
The registrar
|
|
||||||
-------------
|
|
||||||
|
|
||||||
Discovery needs a phonebook. Every actor, as part of boot,
|
|
||||||
registers its ``Aid`` and bind-addrs with the *registrar*: an
|
|
||||||
otherwise ordinary actor (a :class:`tractor.Registrar`, subtype
|
|
||||||
of :class:`~tractor.Actor`) that keeps the name -> addrs table
|
|
||||||
for the tree; on graceful exit each actor de-registers itself.
|
|
||||||
|
|
||||||
.. margin:: Default registry addrs
|
|
||||||
|
|
||||||
With no ``registry_addrs`` passed:
|
|
||||||
|
|
||||||
- tcp: ``('127.0.0.1', 1616)``
|
|
||||||
- uds: ``registry@1616.sock``
|
|
||||||
in the runtime dir
|
|
||||||
|
|
||||||
Who *is* the registrar? Decided at root boot, rendezvous style.
|
|
||||||
:func:`tractor.open_root_actor` probes each addr in
|
|
||||||
``registry_addrs`` with a quick connect-ping, then:
|
|
||||||
|
|
||||||
- **somebody answered**: this root is a plain actor; it
|
|
||||||
registers with the existing registrar and binds random
|
|
||||||
same-proto addrs for its own IPC server;
|
|
||||||
- **nobody answered**: this root *becomes* the registrar and
|
|
||||||
binds the registry addrs itself.
|
|
||||||
|
|
||||||
So single-program trees need zero config — the root quietly
|
|
||||||
self-appoints — while multi-program setups share a registrar by
|
|
||||||
pointing every program at the same ``registry_addrs``. Pass
|
|
||||||
``ensure_registry=True`` to demand that *this* call create the
|
|
||||||
registry; it raises if the addrs are already served.
|
|
||||||
|
|
||||||
The lookup APIs — :func:`tractor.find_actor`,
|
|
||||||
:func:`tractor.wait_for_actor`, :func:`tractor.query_actor` and
|
|
||||||
:func:`tractor.get_registry` — all consult it (after first
|
|
||||||
checking already-connected peers):
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/service_daemon_discovery.py
|
|
||||||
:caption: examples/service_daemon_discovery.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
If you bump into "arbiter" in old issues or posts: that's the
|
|
||||||
legacy name for the same thing, surviving in-code only as the
|
|
||||||
``Arbiter = Registrar`` class alias; all current terminology is
|
|
||||||
"registrar"/"registry". Fair warning per the README: this is
|
|
||||||
still a **very naive** discovery sys (no re-election, no gossip
|
|
||||||
protocol... yet) and a registrar is expected to outlive its
|
|
||||||
registrants.
|
|
||||||
|
|
||||||
Runtime env vars
|
|
||||||
----------------
|
|
||||||
|
|
||||||
A few env-vars let you re-tune a whole tree *without touching
|
|
||||||
app code*; each wins over its corresponding kwarg:
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 30 44 26
|
|
||||||
|
|
||||||
* - env-var
|
|
||||||
- effect
|
|
||||||
- vs. kwarg
|
|
||||||
* - ``TRACTOR_LOGLEVEL``
|
|
||||||
- crank (or silence) console-log verbosity for every actor
|
|
||||||
in the tree
|
|
||||||
- beats ``loglevel``
|
|
||||||
* - ``TRACTOR_SPAWN_METHOD``
|
|
||||||
- swap the process spawn backend
|
|
||||||
- beats ``start_method``
|
|
||||||
* - ``TRACTOR_ENABLE_STACKSCOPE``
|
|
||||||
- install the ``SIGUSR1`` task-tree-dump handler in every
|
|
||||||
actor, even outside ``debug_mode`` (see
|
|
||||||
:doc:`/guide/debugging`)
|
|
||||||
- OR'd with ``enable_stack_on_sig``
|
|
||||||
|
|
||||||
Spotting actors from your shell
|
|
||||||
*******************************
|
|
||||||
|
|
||||||
Every sub-actor sets an OS-level proc-title of the form
|
|
||||||
``_subactor[<name>@<pid>]`` (via ``setproctitle``, silently
|
|
||||||
skipped when not installed) so ``ps``/``htop``/``pstree`` show
|
|
||||||
*which actor is which* at a glance. The README's signature
|
|
||||||
incantation — watch a tree build and self-destruct live:
|
|
||||||
|
|
||||||
.. code:: bash
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/nested_actor_tree.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
For scripting there are two stable cmdline markers:
|
|
||||||
|
|
||||||
.. code:: bash
|
|
||||||
|
|
||||||
pgrep -fa '_subactor\[' # live, titled sub-actors
|
|
||||||
pgrep -fa 'tractor._child' # 'trio'-backend children not
|
|
||||||
# yet (re)titled
|
|
||||||
|
|
||||||
The title also lands in the kernel ``comm`` (truncated to ~15
|
|
||||||
bytes) which survives into zombie state — that's what
|
|
||||||
``tractor``'s own test-harness reapers key off. To be crystal
|
|
||||||
clear about the contract though: you should never *need* a
|
|
||||||
reaper; if you can create zombie child processes (without using
|
|
||||||
a system signal) it **is a bug** — please report it!
|
|
||||||
|
|
||||||
Logging
|
|
||||||
-------
|
|
||||||
|
|
||||||
The runtime logs through a thin adapter over stdlib
|
|
||||||
:mod:`logging` that stamps every record with actor + task info.
|
|
||||||
Two calls get you going:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor.log import get_console_log, get_logger
|
|
||||||
|
|
||||||
log = get_logger(__name__) # actor/task-aware sub-logger
|
|
||||||
get_console_log('info') # attach console handler @ level
|
|
||||||
|
|
||||||
(or just pass ``loglevel='info'`` to
|
|
||||||
:func:`tractor.open_root_actor` and the console handler comes up
|
|
||||||
with the runtime).
|
|
||||||
|
|
||||||
``tractor`` adds custom levels — and matching logger methods —
|
|
||||||
that slot between the stdlib ones so you can dial in *which
|
|
||||||
runtime subsystem* you want to hear from: ``.transport()`` (5),
|
|
||||||
``.runtime()`` (15), ``.devx()`` (17), ``.cancel()`` (22), plus
|
|
||||||
a ``PDB`` (500) level for debugger chatter. E.g.
|
|
||||||
``loglevel='cancel'`` plays the whole cancellation chorus while
|
|
||||||
staying quiet about transport-layer noise. Beyond that
|
|
||||||
``tractor`` isn't opinionated about how you consume logs: it's
|
|
||||||
all stdlib ``logging`` underneath.
|
|
||||||
|
|
||||||
Where to next?
|
|
||||||
--------------
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` — the SC-linked cross-actor task API
|
|
||||||
that rides on every ``Channel``.
|
|
||||||
- :doc:`/guide/debugging` — ``debug_mode``, the multi-process
|
|
||||||
REPL and ``stackscope`` task-tree dumps.
|
|
||||||
- :doc:`/explain/sc-distributed` — *why*
|
|
||||||
one-actor-one-process, and what kind of "actor model" this
|
|
||||||
is (and isn't).
|
|
||||||
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
|
|
||||||
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
|
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
.. _PEP 734: https://peps.python.org/pep-0734/
|
|
||||||
|
|
@ -1,16 +0,0 @@
|
||||||
Big ideas
|
|
||||||
=========
|
|
||||||
The conceptual core of ``tractor``: what *structured
|
|
||||||
concurrency* (SC) means once your "tasks" are whole
|
|
||||||
processes, and how the runtime is layered to deliver
|
|
||||||
that without (much) magic.
|
|
||||||
|
|
||||||
If you only read one page in these docs make it
|
|
||||||
:doc:`sc-distributed`; if you read two, follow it with
|
|
||||||
:doc:`architecture`.
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:maxdepth: 2
|
|
||||||
|
|
||||||
sc-distributed
|
|
||||||
architecture
|
|
||||||
|
|
@ -1,207 +0,0 @@
|
||||||
Structured concurrency, across processes
|
|
||||||
=========================================
|
|
||||||
``tractor`` makes one bet: the discipline that made
|
|
||||||
``trio``'s concurrency *readable* — `structured
|
|
||||||
concurrency`_ (SC) — works just as well when the
|
|
||||||
"tasks" are whole OS processes talking over a wire.
|
|
||||||
This page distills what that means, from first
|
|
||||||
principles, with as little ceremony as possible.
|
|
||||||
|
|
||||||
.. margin:: The canon
|
|
||||||
|
|
||||||
If SC is new to you, the seminal `blog post`_ is
|
|
||||||
still the best hour you'll spend on concurrent
|
|
||||||
programming; the `trio docs`_, wikipedia's SC_
|
|
||||||
page and the diagrams over at libdill-docs_ round
|
|
||||||
it out nicely.
|
|
||||||
|
|
||||||
SC in one breath
|
|
||||||
----------------
|
|
||||||
Structured concurrency is the rule that **concurrency
|
|
||||||
gets a scope**: every task is spawned *inside* a block
|
|
||||||
(a ``trio`` *nursery*) and that block **cannot exit
|
|
||||||
until every task it spawned has finished** — returned,
|
|
||||||
errored, or been cancelled.
|
|
||||||
|
|
||||||
That one rule buys you the properties you already
|
|
||||||
rely on in sequential code,
|
|
||||||
|
|
||||||
- a function call is a *black box*: when it returns,
|
|
||||||
everything it started is **done** — no secret
|
|
||||||
background tasks leaking out the sides,
|
|
||||||
- an exception **always has somewhere to go**: up the
|
|
||||||
(task) tree to a parent which is, by construction,
|
|
||||||
still there waiting,
|
|
||||||
- cancellation has a well defined *shape*: cancel a
|
|
||||||
scope and it flows down to every task inside it,
|
|
||||||
and only those.
|
|
||||||
|
|
||||||
In short: your **runtime task tree matches your source
|
|
||||||
code's indentation**. Concurrency you can read.
|
|
||||||
|
|
||||||
The leap: process-shaped tasks
|
|
||||||
------------------------------
|
|
||||||
Now swap "task" for "process".
|
|
||||||
|
|
||||||
A ``tractor`` *actor* is just a Python process running
|
|
||||||
its own ``trio.run()`` — its own private task tree,
|
|
||||||
sharing **nothing** with its siblings. You spawn
|
|
||||||
actors from an :class:`tractor.ActorNursery`, which
|
|
||||||
behaves exactly the way the name implies,
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'worker',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
...
|
|
||||||
# ^ block exit == every spawned process has
|
|
||||||
# completed, errored or been cancelled, and
|
|
||||||
# been **reaped**. No exceptions, no zombies.
|
|
||||||
|
|
||||||
so the whole program becomes a *tree of process-trees*
|
|
||||||
— a `supervision tree`_ in erlang-speak — where every
|
|
||||||
arrow means "spawned by, **waited on by**, and
|
|
||||||
supervised by".
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:caption: A ``tractor`` program: a process tree of
|
|
||||||
``trio`` task trees; every parent **must wait**
|
|
||||||
on its children.
|
|
||||||
:width: 85%
|
|
||||||
|
|
||||||
Causality: no process outlives its parent
|
|
||||||
-----------------------------------------
|
|
||||||
The stdlib's ``multiprocessing`` (and most "job
|
|
||||||
queue" systems) treat child processes as
|
|
||||||
fire-and-forget by default: orphans, zombies, lost
|
|
||||||
tracebacks and ``kill -9`` cleanup scripts are *your*
|
|
||||||
problem. ``tractor`` instead inherits ``trio``'s
|
|
||||||
`causality`_ discipline,
|
|
||||||
|
|
||||||
- **no spawning willy-nilly**: every actor is born
|
|
||||||
from a nursery block with a known parent,
|
|
||||||
- **lifetimes nest**: a sub-actor's entire process
|
|
||||||
tree lives strictly inside its parent's nursery
|
|
||||||
scope,
|
|
||||||
- **teardown is guaranteed**: when a scope exits (or
|
|
||||||
errors, or is cancelled) the runtime SIGINTs,
|
|
||||||
waits, and (only if it must) hard-kills + reaps
|
|
||||||
everything underneath.
|
|
||||||
|
|
||||||
We take the zombie thing personally: *if you can
|
|
||||||
create orphaned child processes without using a
|
|
||||||
system signal, it* **is a bug** — and there's a test
|
|
||||||
suite to back that sentence up.
|
|
||||||
|
|
||||||
Errors always propagate (yes, across the wire)
|
|
||||||
----------------------------------------------
|
|
||||||
In ``trio``, an exception in any task tears through
|
|
||||||
its nursery to a parent that must handle it —
|
|
||||||
`exceptions always propagate`_. ``tractor`` extends
|
|
||||||
the same guarantee across process boundaries: an
|
|
||||||
uncaught error in a remote task is
|
|
||||||
|
|
||||||
1. captured + serialized in the child,
|
|
||||||
2. shipped home over IPC as a typed ``Error`` msg,
|
|
||||||
3. re-raised in the parent **boxed** as a
|
|
||||||
:class:`tractor.RemoteActorError` carrying the
|
|
||||||
original type (``.boxed_type``), a rendered remote
|
|
||||||
traceback, and the erroring actor's id,
|
|
||||||
|
|
||||||
while the supervising nursery applies its (currently
|
|
||||||
*one-cancels-all*, just like ``trio``) strategy to any
|
|
||||||
sibling actors. A crash three processes deep arrives
|
|
||||||
at your shell as one coherent, causal traceback chain
|
|
||||||
— not a silent dead worker and a stuck queue.
|
|
||||||
|
|
||||||
Cancellation is a request, supervision is the rule
|
|
||||||
--------------------------------------------------
|
|
||||||
Cancellation likewise keeps ``trio``'s semantics
|
|
||||||
*verbatim*, just transported: cancelling an actor
|
|
||||||
nursery (or a single :class:`tractor.Context` between
|
|
||||||
two tasks in different processes) sends an explicit
|
|
||||||
cancel **request** over IPC which the remote runtime
|
|
||||||
translates into a real ``trio`` cancel-scope cancel —
|
|
||||||
then *acks back* so the requester can await
|
|
||||||
confirmation within a bounded time. Nothing is ever
|
|
||||||
"just killed" first; graceful always precedes brutal.
|
|
||||||
|
|
||||||
Because every cross-process dialog is a pair of
|
|
||||||
**linked tasks** — one on each side, each inside its
|
|
||||||
own cancel scope — SC stays *transitive*: supervision
|
|
||||||
doesn't stop at the process boundary, it tunnels
|
|
||||||
through every hop of the tree. The wire protocol that
|
|
||||||
enforces this (a small set of typed msgs:
|
|
||||||
``Start``/``Started``/``Yield``/``Stop``/``Return``/
|
|
||||||
``Error``) is detailed in :doc:`/guide/msging` and
|
|
||||||
:doc:`/guide/context`.
|
|
||||||
|
|
||||||
Hold up, is this an "actor model"?
|
|
||||||
----------------------------------
|
|
||||||
Let's stop and ask how many canon actor model papers
|
|
||||||
you've actually read ;)
|
|
||||||
|
|
||||||
From `the author's mouth`_, the **only** requirement
|
|
||||||
is `adherence to`_ the `3 axioms`_::
|
|
||||||
|
|
||||||
In response to a message, an actor may:
|
|
||||||
|
|
||||||
- send a finite number of new messages
|
|
||||||
- create a finite number of new actors
|
|
||||||
- designate a new behavior to process subsequent
|
|
||||||
messages
|
|
||||||
|
|
||||||
``tractor`` adheres — actors exchange msgs, spawn
|
|
||||||
actors, and swap behaviors — **with no extra API** to
|
|
||||||
learn. What we *don't* copy is the cultural baggage:
|
|
||||||
no visible mailboxes, no untyped fire-and-forget
|
|
||||||
``send()``, no "let it crash" without a supervisor
|
|
||||||
that actually hears about it, and definitely no
|
|
||||||
shared-reference *proxy objects* pretending the
|
|
||||||
network isn't there. If our "actors" don't look like
|
|
||||||
what you expected, that's **intentional**: being an
|
|
||||||
actor model is just one property of the system; being
|
|
||||||
*structured* is the point.
|
|
||||||
|
|
||||||
Why processes at all?
|
|
||||||
---------------------
|
|
||||||
Python has a GIL; an actor model by definition shares
|
|
||||||
no state; so the *process* is the natural runtime
|
|
||||||
unit — you get real multi-core parallelism and hard
|
|
||||||
memory isolation for free. But the deeper win is
|
|
||||||
uniformity: because actors only ever talk via msgs
|
|
||||||
over a :class:`tractor.Channel` (TCP, UDS, more to
|
|
||||||
come), the **same code** runs your laptop's worker
|
|
||||||
pool and a multi-host cluster; "distributed" is a
|
|
||||||
deployment detail, not an API.
|
|
||||||
|
|
||||||
It's just ``trio``
|
|
||||||
------------------
|
|
||||||
If you remember one framing, make it this: ``tractor``
|
|
||||||
**is just** ``trio`` — with nurseries that can spawn
|
|
||||||
processes and streams that can cross them. Same
|
|
||||||
nursery discipline, same cancellation semantics, same
|
|
||||||
"how was this not always the API?" feeling, one level
|
|
||||||
up the process tree.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
:doc:`/explain/architecture` for how the runtime
|
|
||||||
layers deliver all of the above, and
|
|
||||||
:doc:`/start/quickstart` to feel it in ~20 lines of
|
|
||||||
code.
|
|
||||||
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
|
||||||
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
|
||||||
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
|
|
||||||
.. _supervision tree: https://www.erlang.org/doc/design_principles/des_princ.html
|
|
||||||
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
|
|
||||||
.. _exceptions always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
|
|
||||||
.. _the author's mouth: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s
|
|
||||||
.. _adherence to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
|
|
||||||
.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts
|
|
||||||
|
|
@ -1,309 +0,0 @@
|
||||||
Infected ``asyncio``
|
|
||||||
====================
|
|
||||||
``tractor`` is "just trio_", but the Python world is packed with
|
|
||||||
libraries that only speak ``asyncio``: websocket stacks, vendor
|
|
||||||
SDKs, that one exchange client you can't route around. Rather than
|
|
||||||
make you rewrite them, ``tractor`` lets you *quarantine* them inside
|
|
||||||
a dedicated subactor which runs both event loops at once, with full
|
|
||||||
`structured concurrency`_ (SC) guarantees maintained across the
|
|
||||||
loop boundary *and* the process tree.
|
|
||||||
|
|
||||||
In the project's own words:
|
|
||||||
|
|
||||||
Yes, we spawn a python process, run ``asyncio``, start ``trio``
|
|
||||||
on the ``asyncio`` loop, then send commands to the ``trio``
|
|
||||||
scheduled tasks to tell ``asyncio`` tasks what to do XD
|
|
||||||
|
|
||||||
We call this "infected ``asyncio``" mode: the subactor's stdlib
|
|
||||||
loop runs as the *host* with ``trio`` embedded on top in `guest`_
|
|
||||||
mode, and your ``trio`` tasks drive ``asyncio`` tasks through
|
|
||||||
a linked, SC-supervised, in-memory channel.
|
|
||||||
|
|
||||||
.. d2:: diagrams/infected_aio.d2
|
|
||||||
:caption: One process, two schedulers: ``trio`` rides the
|
|
||||||
``asyncio`` loop as a guest while the parent speaks plain
|
|
||||||
``tractor`` IPC, none the wiser.
|
|
||||||
:alt: parent actor connected over IPC to a subactor whose
|
|
||||||
asyncio loop hosts trio in guest mode, with a
|
|
||||||
LinkedTaskChannel pairing a trio task to an asyncio task
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Infected ``asyncio`` mode is **experimental**: it works (we
|
|
||||||
beat on it plenty) but parts of the API surface and some
|
|
||||||
edge-case semantics are still settling. Got opinions on the
|
|
||||||
interop design? Feel free to sling them in `#273`_!
|
|
||||||
|
|
||||||
How the infection takes hold
|
|
||||||
----------------------------
|
|
||||||
A normal subactor boots by running the ``tractor`` runtime's task
|
|
||||||
tree directly under ``trio.run()``. Pass ``infect_asyncio=True``
|
|
||||||
at spawn time and the child's entrypoint changes shape entirely:
|
|
||||||
|
|
||||||
1. the process starts the stdlib loop via ``asyncio.run()``,
|
|
||||||
2. the first ``asyncio`` task calls
|
|
||||||
``trio.lowlevel.start_guest_run()``, embedding the ``trio``
|
|
||||||
scheduler *inside* the already running ``asyncio`` loop (the
|
|
||||||
upstream `guest`_-mode feature),
|
|
||||||
3. the regular ``tractor`` runtime then boots on the guest
|
|
||||||
``trio`` side and connects back to its parent like any other
|
|
||||||
subactor.
|
|
||||||
|
|
||||||
.. margin:: Symptoms
|
|
||||||
|
|
||||||
Looks like your stdlib event loop has caught a case of "the
|
|
||||||
trios"! Don't worry, you'll barely notice; and if anything
|
|
||||||
gets too bad, your parents will know about it B)
|
|
||||||
|
|
||||||
Both schedulers interleave in a single thread, no GIL gymnastics
|
|
||||||
required. From the rest of the actor tree the infected child is
|
|
||||||
indistinguishable from any other actor: same IPC protocol, same
|
|
||||||
supervision and cancellation semantics, same zombie-safety
|
|
||||||
guarantees. The difference is purely internal: ``trio`` tasks in
|
|
||||||
that process can start and drive ``asyncio`` tasks through the
|
|
||||||
``tractor.to_asyncio`` API.
|
|
||||||
|
|
||||||
Spawning an infected subactor
|
|
||||||
-----------------------------
|
|
||||||
Just flip the flag on :meth:`tractor.ActorNursery.start_actor`:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'aio_side',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
infect_asyncio=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
The one-shot convenience ``ActorNursery.run_in_actor()`` accepts
|
|
||||||
the same flag. The ``to_asyncio`` APIs may **only** be called from
|
|
||||||
tasks inside an infected actor; calling them anywhere else raises
|
|
||||||
a loud ``RuntimeError``. You can introspect at runtime with
|
|
||||||
``tractor.current_actor().is_infected_aio()``.
|
|
||||||
|
|
||||||
Linking tasks with ``open_channel_from()``
|
|
||||||
------------------------------------------
|
|
||||||
The core primitive is :func:`tractor.to_asyncio.open_channel_from`,
|
|
||||||
an async context manager which starts your ``asyncio`` function as
|
|
||||||
a real ``asyncio.Task`` and yields a two-way channel linking it to
|
|
||||||
the calling ``trio`` task:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor import to_asyncio
|
|
||||||
|
|
||||||
async with to_asyncio.open_channel_from(
|
|
||||||
aio_main, # async def aio_main(chan, **kwargs)
|
|
||||||
period=0.5, # extra kwargs are passed through
|
|
||||||
) as (chan, first):
|
|
||||||
await chan.send('tick')
|
|
||||||
|
|
||||||
The semantics deliberately mirror the inter-actor ``Context``
|
|
||||||
handshake from :doc:`/guide/context`:
|
|
||||||
|
|
||||||
- the target fn must declare a parameter literally named ``chan``;
|
|
||||||
the runtime injects the shared
|
|
||||||
:class:`~tractor.to_asyncio.LinkedTaskChannel` by keyword.
|
|
||||||
- the ``trio`` side blocks at entry until the ``asyncio`` task
|
|
||||||
calls ``chan.started_nowait(value)``; that value is delivered as
|
|
||||||
``first``, exactly like the ``(ctx, first)`` pair you get from
|
|
||||||
``Portal.open_context()`` after the child calls
|
|
||||||
``ctx.started()``.
|
|
||||||
- a first value **must** be sent from the ``asyncio`` side or the
|
|
||||||
``trio`` side will never unblock.
|
|
||||||
- on block exit the pair is torn down *together*; neither task can
|
|
||||||
outlive the other (more on this below).
|
|
||||||
|
|
||||||
A full example: the echo server
|
|
||||||
-------------------------------
|
|
||||||
Here's the canonical demo, a round-trip echo service where the
|
|
||||||
``asyncio`` task is told what to do by a ``trio`` task which is in
|
|
||||||
turn driven over IPC by the root actor:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/infected_asyncio_echo_server.py
|
|
||||||
:caption: examples/infected_asyncio_echo_server.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on?
|
|
||||||
|
|
||||||
- there are three task layers: the root actor's pure ``trio``
|
|
||||||
task, the infected child's ``trio``-side ``@tractor.context``
|
|
||||||
endpoint (``trio_to_aio_echo_server()``), and the child's
|
|
||||||
``asyncio`` task (``aio_echo_server()``).
|
|
||||||
- two ``started``-style handshakes compose: the aio task's
|
|
||||||
``chan.started_nowait('start')`` unblocks the child's
|
|
||||||
``open_channel_from()`` entry, then the child relays that same
|
|
||||||
value up via ``await ctx.started(first)`` which unblocks the
|
|
||||||
root's ``open_context()`` entry. Synchronization all the way
|
|
||||||
down, er, up.
|
|
||||||
- each round trip flows: root ``stream.send()`` -> IPC -> child
|
|
||||||
``async for msg in stream`` -> ``chan.send(msg)`` -> aio
|
|
||||||
``await chan.get()`` -> ``chan.send_nowait()`` -> child
|
|
||||||
``chan.receive()`` -> ``stream.send(out)`` -> IPC -> root.
|
|
||||||
- when the root breaks out of its stream loop and exits the
|
|
||||||
context block, the child's stream ends, its channel block exits,
|
|
||||||
and the ``asyncio`` task is reaped along with it; the final
|
|
||||||
``portal.cancel_actor()`` then tears down the whole process. No
|
|
||||||
orphaned ``asyncio`` tasks, no zombie procs; if you manage to
|
|
||||||
create either it **is a bug**.
|
|
||||||
|
|
||||||
``LinkedTaskChannel``: one channel, two sides
|
|
||||||
---------------------------------------------
|
|
||||||
The same channel object is shared by both tasks; which methods you
|
|
||||||
call depends on which loop schedules your task. The ``trio`` side
|
|
||||||
gets a standard ``trio.abc.Channel`` interface while the
|
|
||||||
``asyncio`` side gets queue-flavored, mostly-sync methods:
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 14 36 50
|
|
||||||
|
|
||||||
* - side
|
|
||||||
- call
|
|
||||||
- what it does
|
|
||||||
* - ``trio``
|
|
||||||
- ``await chan.send(item)``
|
|
||||||
- ship ``item`` to the ``asyncio`` task (enqueues onto an
|
|
||||||
internal ``asyncio.Queue``).
|
|
||||||
* - ``trio``
|
|
||||||
- ``await chan.receive()``
|
|
||||||
- wait for the next value from the ``asyncio`` side; the
|
|
||||||
channel also supports ``async for``.
|
|
||||||
* - ``trio``
|
|
||||||
- ``await chan.wait_for_result()``
|
|
||||||
- block until the ``asyncio`` task completes; return its
|
|
||||||
final result or raise its (translated) error.
|
|
||||||
* - ``trio``
|
|
||||||
- ``chan.subscribe()``
|
|
||||||
- acm yielding a ``BroadcastReceiver`` so N local tasks can
|
|
||||||
each consume a copy of the inbound stream (see below).
|
|
||||||
* - ``trio``
|
|
||||||
- ``chan.cancel_asyncio_task()``
|
|
||||||
- explicitly request cancellation of the linked ``asyncio``
|
|
||||||
task.
|
|
||||||
* - ``asyncio``
|
|
||||||
- ``chan.started_nowait(value)``
|
|
||||||
- deliver the "first" value; unblocks the ``trio`` side's
|
|
||||||
``open_channel_from()`` entry (mirrors ``ctx.started()``).
|
|
||||||
* - ``asyncio``
|
|
||||||
- ``await chan.get()``
|
|
||||||
- wait for the next value sent from the ``trio`` side.
|
|
||||||
* - ``asyncio``
|
|
||||||
- ``chan.send_nowait(item)``
|
|
||||||
- push a value to the ``trio`` side without blocking.
|
|
||||||
|
|
||||||
Fan-out with ``.subscribe()``
|
|
||||||
*****************************
|
|
||||||
Just like :meth:`tractor.MsgStream.subscribe` does for IPC
|
|
||||||
streams, ``chan.subscribe()`` lets multiple local ``trio`` tasks
|
|
||||||
each receive *every* value sent from the single ``asyncio`` task:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with chan.subscribe() as bcast:
|
|
||||||
async for msg in bcast:
|
|
||||||
...
|
|
||||||
|
|
||||||
The underlying broadcast machinery is lazily allocated on first
|
|
||||||
use and is *not* reversible for the channel's remaining lifetime,
|
|
||||||
so only reach for it when you actually want the fan-out.
|
|
||||||
|
|
||||||
One-shot calls with ``run_task()``
|
|
||||||
----------------------------------
|
|
||||||
When you just want a single ``asyncio`` result and no streaming
|
|
||||||
dialog, skip the channel ceremony and use
|
|
||||||
:func:`tractor.to_asyncio.run_task`:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from tractor import to_asyncio
|
|
||||||
|
|
||||||
async def aio_fetch(url: str) -> str:
|
|
||||||
await asyncio.sleep(0.3) # pretend-IO, aio style
|
|
||||||
return f'<html>sup {url}</html>'
|
|
||||||
|
|
||||||
# from any trio task inside the infected actor:
|
|
||||||
page = await to_asyncio.run_task(aio_fetch, url='https://x.io')
|
|
||||||
|
|
||||||
It schedules the fn as an ``asyncio.Task``, waits for completion
|
|
||||||
and hands the return value back to ``trio``; think of it as the
|
|
||||||
cross-loop sibling of ``ActorNursery.run_in_actor()``. Errors and
|
|
||||||
cancellation are translated exactly as for channels.
|
|
||||||
|
|
||||||
Cross-loop errors and cancellation
|
|
||||||
----------------------------------
|
|
||||||
The paired tasks are *SC linked*: exception and cancel handling
|
|
||||||
tears down **both** sides on any unexpected error or cancellation,
|
|
||||||
in either loop. There is no fire-and-forget mode; a
|
|
||||||
``LinkedTaskChannel`` is a supervision scope just like a
|
|
||||||
``Context`` is across processes.
|
|
||||||
|
|
||||||
Because each loop has its own (incompatible) cancellation and exit
|
|
||||||
machinery, boundary crossings are translated into dedicated
|
|
||||||
exception types, all importable from ``tractor.to_asyncio``:
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 26 22 52
|
|
||||||
|
|
||||||
* - exception
|
|
||||||
- raised in
|
|
||||||
- meaning
|
|
||||||
* - ``AsyncioCancelled``
|
|
||||||
- the ``trio`` task
|
|
||||||
- the linked ``asyncio`` task was cancelled by itself or
|
|
||||||
a 3rd party (i.e. *not* by the ``trio`` side).
|
|
||||||
* - ``AsyncioTaskExited``
|
|
||||||
- the ``trio`` task
|
|
||||||
- the ``asyncio`` task returned/exited early while the
|
|
||||||
``trio`` side still held the link open.
|
|
||||||
* - ``TrioCancelled``
|
|
||||||
- the ``asyncio`` task
|
|
||||||
- the ``trio`` side was cancelled (or crashed) so the
|
|
||||||
``asyncio`` task is being torn down per SC rules.
|
|
||||||
* - ``TrioTaskExited``
|
|
||||||
- the ``asyncio`` task
|
|
||||||
- the ``trio`` side exited gracefully while the ``asyncio``
|
|
||||||
task was still running; a "clean shutdown" signal much
|
|
||||||
like closing a ``trio`` mem-chan.
|
|
||||||
|
|
||||||
By default ``open_channel_from(suppress_graceful_exits=True)``
|
|
||||||
absorbs the two ``*TaskExited`` signals so happy-path teardown
|
|
||||||
stays silent; pass ``False`` when your app wants to handle early
|
|
||||||
peer-exit explicitly.
|
|
||||||
|
|
||||||
Past the task pair, everything composes with the normal actor
|
|
||||||
story: an unhandled ``asyncio`` error is translated into the
|
|
||||||
``trio`` side, propagates out of your ``@tractor.context``
|
|
||||||
endpoint, and arrives at the parent boxed as
|
|
||||||
a :class:`tractor.RemoteActorError`. One SC discipline,
|
|
||||||
end-to-end, across loops *and* processes.
|
|
||||||
|
|
||||||
Breakpoints in ``asyncio`` tasks
|
|
||||||
--------------------------------
|
|
||||||
Yes, the multi-actor REPL works here too. With
|
|
||||||
``debug_mode=True`` enabled on your tree the ``trio`` side of an
|
|
||||||
infected actor can ``await tractor.pause()`` as usual, and with
|
|
||||||
greenback enabled (``maybe_enable_greenback=True``) even the
|
|
||||||
builtin ``breakpoint()`` works from *inside* ``asyncio`` tasks;
|
|
||||||
see ``examples/debugging/asyncio_bp.py`` for the full tour. The
|
|
||||||
root-TTY locking dance behind all this is covered in
|
|
||||||
:doc:`/guide/debugging`.
|
|
||||||
|
|
||||||
Where to next?
|
|
||||||
--------------
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` for the inter-actor handshake and
|
|
||||||
streaming APIs which this whole interop layer mirrors.
|
|
||||||
- :doc:`/guide/msging` for typing the payloads you shuttle
|
|
||||||
between actors (and loops).
|
|
||||||
- :doc:`/guide/debugging` for the multi-process REPL that
|
|
||||||
keeps working even when your loop has "the trios".
|
|
||||||
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
|
|
||||||
.. _#273: https://github.com/goodboy/tractor/issues/273
|
|
||||||
|
|
@ -1,285 +0,0 @@
|
||||||
Cancellation and error propagation
|
|
||||||
==================================
|
|
||||||
|
|
||||||
``tractor`` supports ``trio``'s cancellation_ system *verbatim*,
|
|
||||||
then extends it across process boundaries. If you know how to
|
|
||||||
cancel a task in ``trio`` you already know how to cancel an actor —
|
|
||||||
and its whole subtree — in ``tractor``; the runtime's job is making
|
|
||||||
that statement hold over IPC with every structured concurrency (SC)
|
|
||||||
guarantee intact.
|
|
||||||
|
|
||||||
The ground rules,
|
|
||||||
|
|
||||||
- a remote actor is **never** cancelled unless explicitly requested
|
|
||||||
(by a parent or peer), unless supervision demands it (an error
|
|
||||||
triggered one-cancels-all teardown), or unless there's a bug in
|
|
||||||
``tractor`` itself (please report it!),
|
|
||||||
- (remote) errors `always propagate`_ back to the parent
|
|
||||||
supervisor; nothing is silently dropped on the floor,
|
|
||||||
- every spawned process gets reaped no matter how it dies; if you
|
|
||||||
can create a zombie child process (without using a system signal)
|
|
||||||
it **is a bug**.
|
|
||||||
|
|
||||||
``trio`` cancellation, across the wire
|
|
||||||
--------------------------------------
|
|
||||||
|
|
||||||
Locally everything is bog-standard ``trio``: nurseries, cancel
|
|
||||||
scopes, timeouts. ``tractor`` adds exactly one twist: a cancel
|
|
||||||
scope can't physically reach into another process, so the runtime
|
|
||||||
*relays cancellation as messages*. Concretely,
|
|
||||||
|
|
||||||
- cancelling an *actor* means sending it a runtime-cancel request
|
|
||||||
msg; the target then runs its own graceful teardown — cancelling
|
|
||||||
RPC tasks, closing channels, exiting its :func:`trio.run` — and
|
|
||||||
acks the request back to the canceller,
|
|
||||||
- cancelling a single *cross-actor task* works through the
|
|
||||||
:class:`tractor.Context` layer: each ``ctx`` task-pair is
|
|
||||||
cancel-scope-linked over IPC such that either side erroring or
|
|
||||||
cancelling relays an equivalent error to the other side (see
|
|
||||||
:doc:`/guide/context` for the gory details),
|
|
||||||
- a cancel is therefore always a *request with an ack*: the
|
|
||||||
canceller does a **bounded wait** for confirmation and escalates
|
|
||||||
if the peer is unresponsive (see the teardown ladder below).
|
|
||||||
|
|
||||||
One-cancels-all supervision
|
|
||||||
---------------------------
|
|
||||||
|
|
||||||
An :class:`tractor.ActorNursery` supervises subactors `exactly like
|
|
||||||
trio`_ nurseries supervise tasks: when one child errors, the error
|
|
||||||
propagates to the supervising block and **all** sibling subactors
|
|
||||||
get cancelled before the error continues bubbling up the (process)
|
|
||||||
tree.
|
|
||||||
|
|
||||||
.. d2:: diagrams/error_propagation.d2
|
|
||||||
:caption: One-cancels-all: no zombies, no lost errors.
|
|
||||||
:alt: error propagation up a subactor tree
|
|
||||||
:width: 80%
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/remote_error_propagation.py
|
|
||||||
:caption: examples/remote_error_propagation.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on here?
|
|
||||||
|
|
||||||
- three healthy actors are spawned as daemons via
|
|
||||||
:meth:`tractor.ActorNursery.start_actor`; left alone they'd
|
|
||||||
happily idle forever,
|
|
||||||
- a fourth actor runs ``assert_err()`` via ``.run_in_actor()`` and
|
|
||||||
promptly trips its ``assert 0``,
|
|
||||||
- the resulting ``AssertionError`` ships back over IPC as a
|
|
||||||
serialized error msg and re-raises *boxed* inside the nursery
|
|
||||||
block as a :class:`tractor.RemoteActorError`,
|
|
||||||
- the nursery reacts like any ``trio`` nursery would: it cancels
|
|
||||||
the three healthy siblings (graceful runtime-cancel requests,
|
|
||||||
acks awaited), reaps all four processes, then re-raises,
|
|
||||||
- ``trio.run(main)`` sees that same ``RemoteActorError`` in the
|
|
||||||
parent-most process — propagation is end-to-end or bust.
|
|
||||||
|
|
||||||
This one-cancels-all style is currently the *only* supervision
|
|
||||||
strategy offered (it's the one ``trio`` gives you); more
|
|
||||||
`erlang strategies`_ are roadmap, see the bottom of this page.
|
|
||||||
|
|
||||||
The boxed-error bestiary
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
All remote failures arrive locally as one of a small set of
|
|
||||||
exception types, each carrying enough metadata to work out *who*
|
|
||||||
failed, *where*, and *why*.
|
|
||||||
|
|
||||||
``RemoteActorError``
|
|
||||||
********************
|
|
||||||
|
|
||||||
The workhorse: a "boxed" exception relayed over IPC from another
|
|
||||||
actor. The original error's type, traceback string and msgdata are
|
|
||||||
preserved so you can pattern-match on what actually went wrong
|
|
||||||
remotely,
|
|
||||||
|
|
||||||
- ``.boxed_type``: the reconstructed **type** of the original
|
|
||||||
remote exception (``ValueError``, ``NameError``, what have you),
|
|
||||||
- ``.src_uid``: the ``(name, uuid)`` pair of the actor where the
|
|
||||||
error *originated*,
|
|
||||||
- ``.relay_uid`` / ``.relay_path``: when an error crosses more than
|
|
||||||
one actor boundary (grandchild -> child -> root) every relaying
|
|
||||||
actor is recorded; multi-hop boxings are lovingly referred to as
|
|
||||||
"inceptions" in the runtime internals,
|
|
||||||
- ``.pformat()``: a rich "tb box" rendering of the remote traceback
|
|
||||||
for your logs or REPL.
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with portal.open_context(ep) as (ctx, first):
|
|
||||||
...
|
|
||||||
except tractor.RemoteActorError as rae:
|
|
||||||
if rae.boxed_type is ValueError:
|
|
||||||
... # the remote task raised `ValueError`
|
|
||||||
|
|
||||||
``ContextCancelled``
|
|
||||||
********************
|
|
||||||
|
|
||||||
The cancel-ack for a cross-actor task pair: raised when a
|
|
||||||
:class:`tractor.Context` task is cancelled *by request*. Its
|
|
||||||
``.canceller`` attr is the uid of the actor which **requested** the
|
|
||||||
cancel, which powers the key rule,
|
|
||||||
|
|
||||||
- if **you** requested it (you called
|
|
||||||
:meth:`tractor.Context.cancel`) the resulting ctxc is *absorbed*
|
|
||||||
at ``open_context()`` exit: an expected outcome, not an error,
|
|
||||||
- if **anyone else** did — the peer task, or some third-party actor
|
|
||||||
— it *raises* locally so your code always hears about it.
|
|
||||||
|
|
||||||
The full self- vs. cross-cancel semantics are a core teaching point
|
|
||||||
of :doc:`/guide/context`; go read them there.
|
|
||||||
|
|
||||||
``MsgTypeError``
|
|
||||||
****************
|
|
||||||
|
|
||||||
An IPC-payload "type error": a msg violated the dialog's declared
|
|
||||||
payload spec. See :doc:`/guide/msging` for the typed-messaging
|
|
||||||
system which enforces it.
|
|
||||||
|
|
||||||
``TransportClosed``
|
|
||||||
*******************
|
|
||||||
|
|
||||||
The underlying IPC transport (TCP stream, UDS socket, ...) died or
|
|
||||||
closed out from under a channel. You'll normally only see this
|
|
||||||
surface when a peer hard-exits without any graceful runtime
|
|
||||||
teardown; the supervision machinery treats unexpected transport
|
|
||||||
loss on a busy channel as a failure and tears down accordingly.
|
|
||||||
|
|
||||||
Pick your blast radius
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
Three cancel surfaces, three scopes of effect; choose the smallest
|
|
||||||
hammer that does the job.
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 36 34 30
|
|
||||||
|
|
||||||
* - surface
|
|
||||||
- cancels
|
|
||||||
- typical use
|
|
||||||
* - :meth:`tractor.ActorNursery.cancel`
|
|
||||||
- every subactor in the nursery
|
|
||||||
- whole-tree teardown
|
|
||||||
* - :meth:`tractor.Portal.cancel_actor`
|
|
||||||
- one actor: full runtime + proc
|
|
||||||
- daemon teardown
|
|
||||||
* - :meth:`tractor.Context.cancel`
|
|
||||||
- exactly one remote task
|
|
||||||
- surgical task cancel
|
|
||||||
|
|
||||||
``ActorNursery.cancel()``
|
|
||||||
*************************
|
|
||||||
|
|
||||||
The big red button: gracefully cancel every subactor supervised by
|
|
||||||
the nursery, in parallel, with the escalation discipline below
|
|
||||||
applied per-child. It's invoked for you whenever an error hits the
|
|
||||||
nursery block (one-cancels-all); call it yourself for an orderly
|
|
||||||
early shutdown. Passing ``hard_kill=True`` skips the graceful phase
|
|
||||||
and goes straight to OS-level process termination — rarely what you
|
|
||||||
want outside tests.
|
|
||||||
|
|
||||||
``Portal.cancel_actor()``
|
|
||||||
*************************
|
|
||||||
|
|
||||||
Cancel one **whole actor**: its entire runtime, every task it's
|
|
||||||
scheduled, and (for subactors) the OS process, via a graceful
|
|
||||||
runtime-cancel request,
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
await portal.cancel_actor() # bounded wait, bool result
|
|
||||||
await portal.cancel_actor(
|
|
||||||
raise_on_timeout=True, # no ack in time?
|
|
||||||
) # -> `ActorTooSlowError`
|
|
||||||
|
|
||||||
The wait for the peer's ack is *bounded* (default
|
|
||||||
``Portal.cancel_timeout = 0.5`` seconds, tunable per call via
|
|
||||||
``timeout=``). By default a missed ack just returns ``False``; with
|
|
||||||
``raise_on_timeout=True`` you instead get an ``ActorTooSlowError``
|
|
||||||
(from ``tractor._exceptions``) so *your* code can escalate per SC
|
|
||||||
discipline — exactly what the nursery's own teardown does
|
|
||||||
internally before resorting to OS-level signalling.
|
|
||||||
|
|
||||||
Note the granularity: this cancels an **actor**, not a task. For
|
|
||||||
one remote task use the ``Context`` layer instead.
|
|
||||||
|
|
||||||
``Context.cancel()``
|
|
||||||
********************
|
|
||||||
|
|
||||||
Request cancellation of exactly one remote task: the peer task of
|
|
||||||
an open :class:`tractor.Context`. Two things to keep straight,
|
|
||||||
|
|
||||||
- it cancels the **remote** side only; a ``Context`` is *not* a
|
|
||||||
:class:`trio.CancelScope` and your local task keeps running until
|
|
||||||
you exit the ``open_context()`` block,
|
|
||||||
- the resulting :class:`tractor.ContextCancelled` is absorbed
|
|
||||||
locally (you asked for it, after all) per the self- vs.
|
|
||||||
cross-cancel rule above.
|
|
||||||
|
|
||||||
Again, :doc:`/guide/context` covers this dance in depth.
|
|
||||||
|
|
||||||
Graceful first, hard as a last resort
|
|
||||||
-------------------------------------
|
|
||||||
|
|
||||||
.. margin:: REPL-safe by design
|
|
||||||
|
|
||||||
The hard-kill path is *skipped* whenever an actor in the tree
|
|
||||||
holds the debug-REPL lock (``debug_mode=True`` flavors):
|
|
||||||
SIGTERM raining down on a tree mid-``pdb`` session would
|
|
||||||
clobber your prompt. See :doc:`/guide/debugging`.
|
|
||||||
|
|
||||||
Every process teardown in ``tractor`` walks the same escalation
|
|
||||||
ladder, top rung first,
|
|
||||||
|
|
||||||
1. **graceful cancel request**: a runtime-cancel msg over IPC; the
|
|
||||||
target actor cancels its tasks, closes its channels and exits
|
|
||||||
its :func:`trio.run` cleanly,
|
|
||||||
2. **soft wait**: the parent waits (bounded) for the child process
|
|
||||||
to exit on its own,
|
|
||||||
3. **SIGTERM**: no ack within the bounded wait (internally an
|
|
||||||
``ActorTooSlowError``) escalates to ``proc.terminate()``,
|
|
||||||
4. **SIGKILL ultimatum**: still alive after the hard-kill timeout
|
|
||||||
(~1.6s)? The runtime logs that the "T-800" has been deployed to
|
|
||||||
collect the zombie and issues ``proc.kill()``. No survivors.
|
|
||||||
|
|
||||||
The result is the **no-zombies guarantee**: ``tractor`` tries to
|
|
||||||
protect you from zombies, no matter what. Quoting the project
|
|
||||||
manifesto,
|
|
||||||
|
|
||||||
If you can create zombie child processes (without using
|
|
||||||
a system signal) it **is a bug**.
|
|
||||||
|
|
||||||
Run the quickstart's self-destructing process-tree demo
|
|
||||||
(``examples/parallelism/we_are_processes.py``, walked through in
|
|
||||||
:doc:`/start/quickstart`) under a ``pstree`` watcher and try to
|
|
||||||
catch a
|
|
||||||
straggler; we'll wait B)
|
|
||||||
|
|
||||||
Roadmap: ``erlang``-style strategies
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
One-cancels-all is ``trio``'s strategy and, for now, the only one
|
|
||||||
``tractor`` ships. Pluggable `erlang strategies`_ — one-for-one
|
|
||||||
restarts, rest-for-one, transient/permanent child specs and friends
|
|
||||||
(see the `supervision strategies`_ canon) — are a long-standing
|
|
||||||
roadmap item tracked in `#22`_. If supervisors are your jam that
|
|
||||||
issue is the place to sling opinions.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` — the cross-actor task layer where
|
|
||||||
per-task cancellation actually lives,
|
|
||||||
- :doc:`/guide/msging` — the typed msg layer that raises
|
|
||||||
:class:`tractor.MsgTypeError`,
|
|
||||||
- :doc:`/guide/debugging` — what cancellation does (and very
|
|
||||||
carefully does *not* do) while a REPL is up.
|
|
||||||
|
|
||||||
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
|
|
||||||
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
|
||||||
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
|
|
||||||
.. _erlang strategies: https://learnyousomeerlang.com/supervisors
|
|
||||||
.. _supervision strategies: https://www.erlang.org/doc/system/sup_princ.html
|
|
||||||
.. _#22: https://github.com/goodboy/tractor/issues/22
|
|
||||||
|
|
@ -1,127 +0,0 @@
|
||||||
Higher-level cluster APIs
|
|
||||||
=========================
|
|
||||||
|
|
||||||
Sometimes you don't want a hand-crafted supervision tree; you want
|
|
||||||
"a pile of workers, one per core, now please". For that there's
|
|
||||||
:func:`tractor.open_actor_cluster`: a convenience wrapper which
|
|
||||||
spawns a *flat* cluster of subactors and hands you back a portal to
|
|
||||||
each,
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
@acm
|
|
||||||
async def open_actor_cluster(
|
|
||||||
modules: list[str], # RPC allowlist for workers
|
|
||||||
count: int = cpu_count(), # one per core by default
|
|
||||||
names: list[str]|None = None, # default: 'worker_{i}'
|
|
||||||
hard_kill: bool = False, # fwd to `an.cancel()`
|
|
||||||
**runtime_kwargs, # fwd to `open_root_actor()`
|
|
||||||
) -> AsyncGenerator[dict[str, tractor.Portal], None]:
|
|
||||||
|
|
||||||
A cluster in one block
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/quick_cluster.py
|
|
||||||
:caption: examples/quick_cluster.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Walkthrough,
|
|
||||||
|
|
||||||
- ``open_actor_cluster(modules=[__name__])`` concurrently spawns
|
|
||||||
one subactor per detected core (per
|
|
||||||
:func:`multiprocessing.cpu_count`); the ``modules`` list is the
|
|
||||||
usual ``enable_modules``-style capability allowlist so workers
|
|
||||||
may run functions defined in this module,
|
|
||||||
- it yields a ``dict[str, tractor.Portal]`` mapping worker name to
|
|
||||||
portal; note the keys get prefixed with the *spawning* actor's
|
|
||||||
name, so from the root you'll see ``'root.worker_0'``,
|
|
||||||
``'root.worker_1'``, etc.,
|
|
||||||
- a plain :class:`trio.Nursery` then fans out one
|
|
||||||
``portal.run(sleepy_jane)`` per worker; each prints its actor
|
|
||||||
``.uid`` from inside its own process then naps forever — what
|
|
||||||
runs *inside* each worker (and how many tasks you point at it)
|
|
||||||
is entirely yours to compose,
|
|
||||||
- ``tractor.trionics.collapse_eg()`` un-nests the strict
|
|
||||||
``ExceptionGroup`` wrapping so the demo's ``KeyboardInterrupt``
|
|
||||||
surfaces as itself instead of arriving eg-boxed,
|
|
||||||
- on block exit the whole fleet is torn down for you via
|
|
||||||
:meth:`tractor.ActorNursery.cancel`; pass ``hard_kill=True`` at
|
|
||||||
open time to skip straight to OS-level termination instead of
|
|
||||||
the graceful ladder described in :doc:`/guide/cancellation`.
|
|
||||||
|
|
||||||
Sizing, naming, fleet-wide options
|
|
||||||
----------------------------------
|
|
||||||
|
|
||||||
``count`` doesn't have to be core-count and the auto-generated
|
|
||||||
``'worker_{i}'`` names are just the default; pass your own (the
|
|
||||||
length must match ``count`` or you get a ``ValueError``). Any
|
|
||||||
extra ``**runtime_kwargs`` pass through verbatim to
|
|
||||||
:func:`tractor.open_root_actor`, so fleet-wide runtime options are
|
|
||||||
one kwarg away,
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_actor_cluster(
|
|
||||||
modules=['mylib.workers'],
|
|
||||||
count=4,
|
|
||||||
names=['scout', 'miner', 'smelter', 'smith'],
|
|
||||||
debug_mode=True, # whole-fleet crash-to-REPL
|
|
||||||
) as portal_map:
|
|
||||||
...
|
|
||||||
|
|
||||||
From here the composition patterns are the usual ``tractor`` fare:
|
|
||||||
``portal.run()`` for one-shot calls (as in the demo), or — for a
|
|
||||||
persistent bidirectional dialog per worker — concurrently enter N
|
|
||||||
``portal.open_context()`` blocks with
|
|
||||||
``tractor.trionics.gather_contexts()``; see :doc:`/guide/context`
|
|
||||||
for that whole layer.
|
|
||||||
|
|
||||||
Clusters vs. nurseries
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:margin:
|
|
||||||
:caption: The general shape: arbitrary nesting. A cluster is
|
|
||||||
this, minus the nesting.
|
|
||||||
:alt: a nested supervision tree of subactors
|
|
||||||
|
|
||||||
``open_actor_cluster()`` is sugar, not a new primitive: under the
|
|
||||||
hood it's just :func:`tractor.open_nursery` plus N concurrent
|
|
||||||
``start_actor()`` calls plus a ``.cancel()`` on the way out. Reach
|
|
||||||
for it when,
|
|
||||||
|
|
||||||
- you want a *flat*, homogeneous fleet (classic worker-pool or
|
|
||||||
map-style fan-out shapes),
|
|
||||||
- "one per core" — or a fixed ``count`` — is the right sizing
|
|
||||||
story,
|
|
||||||
- every child can share the same spawn options.
|
|
||||||
|
|
||||||
Drop down to a raw :class:`tractor.ActorNursery` when the topology
|
|
||||||
gets any fancier: nested trees, heterogeneous children, per-child
|
|
||||||
``debug_mode``/transport/module options, daemons mixed with
|
|
||||||
one-shot workers, and so on (see :doc:`/guide/parallelism` for a
|
|
||||||
hand-rolled pool). Either way the supervision semantics are
|
|
||||||
identical: one-cancels-all error propagation and the no-zombies
|
|
||||||
guarantee from :doc:`/guide/cancellation` apply to clusters too.
|
|
||||||
|
|
||||||
Provisional, by design
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
APIs in this section are considered **provisional**: the
|
|
||||||
signature and semantics of :func:`tractor.open_actor_cluster`
|
|
||||||
may shift as higher-level supervision machinery lands. We
|
|
||||||
encourage you to try it and provide feedback — the
|
|
||||||
`matrix channel`_ is the place to say hi, and `#22`_ tracks the
|
|
||||||
broader supervisor-strategy roadmap.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/parallelism` — worker pools built "by hand" with
|
|
||||||
plain actor nurseries (and why that's easy peasy),
|
|
||||||
- :doc:`/guide/cancellation` — the teardown machinery a cluster
|
|
||||||
inherits for free.
|
|
||||||
|
|
||||||
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
|
|
||||||
.. _#22: https://github.com/goodboy/tractor/issues/22
|
|
||||||
|
|
@ -1,344 +0,0 @@
|
||||||
The ``Context``: a cross-actor task pair
|
|
||||||
=========================================
|
|
||||||
|
|
||||||
If you've written any trio_ you already know the contract: every
|
|
||||||
task lives in a nursery, errors always propagate, cancellation is
|
|
||||||
scoped, and nothing leaks. ``tractor`` extends that exact contract
|
|
||||||
*across processes* — the same guarantees from the seminal
|
|
||||||
`blog post`_, just with the nursery split across two memory
|
|
||||||
domains. The primitive that does it is :class:`tractor.Context`: a
|
|
||||||
**linked pair of tasks**, one in each of two actors, supervised as
|
|
||||||
a single `structured concurrency`_ (SC) scope over IPC.
|
|
||||||
|
|
||||||
.. d2:: diagrams/context_handshake.d2
|
|
||||||
:caption: The SC-transitive supervision protocol, msg by msg.
|
|
||||||
:alt: sequence diagram of the context handshake msg flow
|
|
||||||
|
|
||||||
Pretty much everything else is (or is slated to be) built on this
|
|
||||||
one primitive: ``ActorNursery.run_in_actor()`` is a convenience
|
|
||||||
for "spawn, open a context, await the result, tear down"; plain
|
|
||||||
``Portal.run()`` RPC is planned to be re-implemented on top of it;
|
|
||||||
the multi-process debugger's tree-wide REPL lock rides one. Grok
|
|
||||||
this page and the rest of the library reads as convenience
|
|
||||||
wrappers B)
|
|
||||||
|
|
||||||
The endpoint contract
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
A context endpoint is an async function decorated with
|
|
||||||
:func:`tractor.context` which declares **a param annotated**
|
|
||||||
``tractor.Context`` — any param name you like, the annotation is
|
|
||||||
what's required:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
@tractor.context
|
|
||||||
async def trainer(
|
|
||||||
ctx: tractor.Context,
|
|
||||||
model: str,
|
|
||||||
) -> str:
|
|
||||||
await ctx.started('ready')
|
|
||||||
return f'trained {model}'
|
|
||||||
|
|
||||||
.. margin:: Who am I talking to?
|
|
||||||
|
|
||||||
Inside any context task
|
|
||||||
:func:`tractor.current_ipc_ctx` returns the
|
|
||||||
``Context`` bound to the current task; handy
|
|
||||||
in helpers that don't take ``ctx`` explicitly.
|
|
||||||
|
|
||||||
The parent (aka "opener") side invokes it through a
|
|
||||||
:class:`tractor.Portal` using ``Portal.open_context()``, passing
|
|
||||||
any extra kwargs which are shipped over the wire as the remote
|
|
||||||
task's arguments. Since the target fn is referenced by module
|
|
||||||
path, that module must be listed in the peer actor's
|
|
||||||
``enable_modules`` allowlist — RPC capability is always opt-in.
|
|
||||||
|
|
||||||
The decorator also accepts a ``pld_spec``: a type (union) which
|
|
||||||
every payload in the dialog is validated against, upgrading your
|
|
||||||
msgs to a typed contract enforced via :exc:`tractor.MsgTypeError`.
|
|
||||||
Validation strictness follows the "`cheap or nasty`_"
|
|
||||||
`(un)protocol`_ pattern: the one-shot ``Started`` payload gets the
|
|
||||||
nasty treatment (stringently round-trip checked before it's even
|
|
||||||
sent) while high-rate stream payloads stay cheap (checked only
|
|
||||||
receiver side).
|
|
||||||
|
|
||||||
The handshake, on the wire
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
Every context runs one instance of ``tractor``'s "SC-transitive
|
|
||||||
supervision protocol": a tiny fixed grammar of msgspec_-typed msgs
|
|
||||||
encapsulating *all* RPC dialogs between actors. *Transitive*
|
|
||||||
because each IPC link obeys the same rules a local nursery does —
|
|
||||||
starts are acked, completion is awaited, errors and cancels always
|
|
||||||
relay — so chaining links across a process tree composes into one
|
|
||||||
tree-wide SC scope.
|
|
||||||
|
|
||||||
The figure up top shows a full dialog; in order:
|
|
||||||
|
|
||||||
``Start``
|
|
||||||
sent by ``Portal.open_context()``: "schedule a task running
|
|
||||||
this function with these kwargs".
|
|
||||||
|
|
||||||
``StartAck``
|
|
||||||
the peer runtime confirms the task is scheduled and that the
|
|
||||||
endpoint really is a context-style fn.
|
|
||||||
|
|
||||||
``Started``
|
|
||||||
emitted when the child task calls
|
|
||||||
:meth:`tractor.Context.started`; carries the first payload
|
|
||||||
and unblocks the parent's entry of ``open_context()``.
|
|
||||||
|
|
||||||
``Yield``
|
|
||||||
one per :meth:`tractor.MsgStream.send`, flowing in *either*
|
|
||||||
direction while a stream is open.
|
|
||||||
|
|
||||||
``Stop``
|
|
||||||
graceful end-of-stream: the far side's ``async for``
|
|
||||||
terminates cleanly.
|
|
||||||
|
|
||||||
``Return``
|
|
||||||
the child fn returned; its value becomes the context's final
|
|
||||||
result. If the child raised instead, an ``Error`` msg takes
|
|
||||||
this slot carrying the boxed traceback.
|
|
||||||
|
|
||||||
``ctx.started()``: just like ``task_status.started()``
|
|
||||||
*******************************************************
|
|
||||||
|
|
||||||
The startup phase is a deliberate clone of
|
|
||||||
:meth:`trio.Nursery.start` semantics: the child decides when it's
|
|
||||||
"up", optionally handing back a first value, and the parent stays
|
|
||||||
blocked until that moment:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
# trio, in-process
|
|
||||||
first = await nursery.start(child_fn)
|
|
||||||
|
|
||||||
# tractor, cross-process
|
|
||||||
async with portal.open_context(child_fn) as (ctx, first):
|
|
||||||
...
|
|
||||||
|
|
||||||
The ``as (ctx, first)`` tuple is exactly that pair: the
|
|
||||||
:class:`tractor.Context` handle plus whatever value the child
|
|
||||||
passed to ``await ctx.started(value)``. And readiness is not
|
|
||||||
optional — for instance opening a stream before ``.started()``
|
|
||||||
has been called raises a ``RuntimeError``; handshake first, then
|
|
||||||
dialog.
|
|
||||||
|
|
||||||
Bidirectional streaming over a context
|
|
||||||
--------------------------------------
|
|
||||||
|
|
||||||
The canonical ping-pong (design history: `#53`_, `#223`_) — a
|
|
||||||
full-duplex msg stream between a parent and its spawned peer:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/rpc_bidir_streaming.py
|
|
||||||
:caption: examples/rpc_bidir_streaming.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on?
|
|
||||||
|
|
||||||
- ``start_actor()`` spawns the daemon-style subactor
|
|
||||||
``'rpc_server'`` with this very module in its allowlist.
|
|
||||||
|
|
||||||
- ``portal.open_context(simple_rpc, data=10)`` fires the
|
|
||||||
``Start`` msg then blocks until the child task calls
|
|
||||||
``await ctx.started(data + 1)`` — hence ``sent == 11``.
|
|
||||||
|
|
||||||
- both tasks enter ``ctx.open_stream()``: a stream dialog is only
|
|
||||||
fully open once *each* side has entered its block.
|
|
||||||
|
|
||||||
- the parent seeds the first ``'ping'``; each side then echoes
|
|
||||||
the other, one ``Yield`` msg per ``stream.send()``.
|
|
||||||
|
|
||||||
- after the 9th pong the parent ``break``\ s (10 pings sent in
|
|
||||||
total) and exits its stream block, which sends ``Stop``; the
|
|
||||||
child's ``async for`` completes gracefully and its ``else``
|
|
||||||
clause asserts all 10 pings arrived.
|
|
||||||
|
|
||||||
- the 10th in-transit pong? Discarded by the implicit drain at
|
|
||||||
``open_context()`` exit, which runs the dialog down to the
|
|
||||||
child's ``Return`` (here ``None``).
|
|
||||||
|
|
||||||
- daemon actors live until told otherwise:
|
|
||||||
``portal.cancel_actor()`` reaps the subactor explicitly.
|
|
||||||
|
|
||||||
Results: the ``Return`` leg
|
|
||||||
---------------------------
|
|
||||||
|
|
||||||
Every context resolves to a final outcome. Wait on it explicitly
|
|
||||||
from the parent side:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with portal.open_context(ep) as (ctx, first):
|
|
||||||
...
|
|
||||||
result = await ctx.wait_for_result()
|
|
||||||
|
|
||||||
or just exit the block — ``__aexit__`` implicitly drains the msg
|
|
||||||
flow until the ``Return`` (or ``Error``) arrives, discarding any
|
|
||||||
in-transit ``Yield``\ s on the way. Either way the rule of
|
|
||||||
`causality`_ holds exactly as in a local nursery: **the opener
|
|
||||||
never unblocks before the remote task is done**.
|
|
||||||
|
|
||||||
For post-hoc inspection (think supervision/restart logic) the ctx
|
|
||||||
also exposes ``Context.outcome``, ``.maybe_error`` and
|
|
||||||
``.has_outcome`` — where a "result" might well be the error the
|
|
||||||
dialog ended with.
|
|
||||||
|
|
||||||
Cancellation semantics
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
The part you actually came for; read it twice B)
|
|
||||||
|
|
||||||
A context's two tasks are **cancel-scope-linked across the IPC
|
|
||||||
boundary**: whatever ends one side — error, cancellation, plain
|
|
||||||
old return — is relayed such that the other side ends
|
|
||||||
equivalently. No silent half-open dialogs, no orphaned remote
|
|
||||||
tasks, ever.
|
|
||||||
|
|
||||||
``ctx.cancel()`` cancels the *remote* task
|
|
||||||
*******************************************
|
|
||||||
|
|
||||||
:meth:`tractor.Context.cancel` requests cancellation of the
|
|
||||||
**remote** task only:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with portal.open_context(ep) as (ctx, first):
|
|
||||||
await accomplish_things(ctx)
|
|
||||||
await ctx.cancel() # remote task, NOT me
|
|
||||||
|
|
||||||
A :class:`tractor.Context` is **not** a :class:`trio.CancelScope`:
|
|
||||||
the call doesn't (and can't) cancel your local task. It sends the
|
|
||||||
cancel request and waits a bounded ``timeout`` for the peer
|
|
||||||
runtime's ``CancelAck``, then your code proceeds to the block exit
|
|
||||||
as normal.
|
|
||||||
|
|
||||||
Compare scopes here: ``Portal.cancel_actor()`` is the big hammer
|
|
||||||
which cancels the peer's **entire runtime** (and thus process);
|
|
||||||
``ctx.cancel()`` is the per-dialog scalpel.
|
|
||||||
|
|
||||||
``ContextCancelled`` and the absorption rule
|
|
||||||
*********************************************
|
|
||||||
|
|
||||||
When a context task gets cancelled *by request* the requestee's
|
|
||||||
runtime reports back with a :exc:`tractor.ContextCancelled`
|
|
||||||
("ctxc") whose ``.canceller`` field holds the uid of the actor
|
|
||||||
which asked. That one field decides what you observe:
|
|
||||||
|
|
||||||
**you requested it**
|
|
||||||
i.e. ``ctxc.canceller == tractor.current_actor().uid``: the
|
|
||||||
ctxc is **absorbed** at ``open_context()`` exit — nothing
|
|
||||||
raises in your block. You asked for a graceful stop and got
|
|
||||||
it; if you care, ``await ctx.wait_for_result()`` hands the
|
|
||||||
ctxc back as a plain *value* for inspection.
|
|
||||||
|
|
||||||
**anyone else requested it**
|
|
||||||
the peer cancelling itself, or some third actor cancelling it
|
|
||||||
from the side: the ctxc **is raised** in your block. From
|
|
||||||
your scope's perspective a task you depend on was killed out
|
|
||||||
from under you and SC demands you hear about it — exactly
|
|
||||||
like a sibling crash in a `nursery`_.
|
|
||||||
|
|
||||||
In code:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with portal.open_context(ep) as (ctx, first):
|
|
||||||
...
|
|
||||||
except tractor.ContextCancelled as ctxc:
|
|
||||||
# can only be a peer- or third-party cancel;
|
|
||||||
# self-requested cancels are absorbed at exit.
|
|
||||||
assert ctxc.canceller != tractor.current_actor().uid
|
|
||||||
|
|
||||||
This self- vs cross-cancel split is what makes explicit teardown
|
|
||||||
*composable*: a supervisor cancels its dialogs without try/except
|
|
||||||
noise, while unexpected cancellation anywhere in the tree still
|
|
||||||
propagates loudly like any other failure.
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
Once ``ctx.cancel()`` has been called the dialog is done: a
|
|
||||||
subsequent ``ctx.open_stream()`` raises ``RuntimeError``.
|
|
||||||
|
|
||||||
For introspection the ctx exposes trio-flavored status props:
|
|
||||||
``.cancel_called`` (this side requested), ``.cancel_acked`` (peer
|
|
||||||
confirmed), ``.cancelled_caught`` and ``.canceller`` —
|
|
||||||
deliberately mirroring :class:`trio.CancelScope` naming.
|
|
||||||
|
|
||||||
Errors propagate, both ways
|
|
||||||
---------------------------
|
|
||||||
|
|
||||||
A crash on either end tears down the pair, SC style:
|
|
||||||
|
|
||||||
- **child raises**: the exception ships back as an ``Error`` msg
|
|
||||||
and re-raises in the parent block boxed as a
|
|
||||||
:exc:`tractor.RemoteActorError`; the original class rides along
|
|
||||||
as ``.boxed_type`` with ``.src_uid`` naming the crashed actor.
|
|
||||||
|
|
||||||
- **parent raises** (or is cancelled) inside the block: an
|
|
||||||
equivalent error/cancel is relayed to the child task so it can
|
|
||||||
never outlive the dialog.
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
try:
|
|
||||||
async with portal.open_context(ep) as (ctx, first):
|
|
||||||
...
|
|
||||||
except tractor.RemoteActorError as rae:
|
|
||||||
if rae.boxed_type is ValueError:
|
|
||||||
... # remote ValueError, type preserved
|
|
||||||
|
|
||||||
Errors that hop through intermediary actors on their way up the
|
|
||||||
tree ("inceptions" XD) keep the full relay trail in
|
|
||||||
``.relay_uid`` / ``.relay_path``. Payloads violating your declared
|
|
||||||
``pld_spec`` surface as the IPC analog of a ``TypeError``:
|
|
||||||
:exc:`tractor.MsgTypeError`.
|
|
||||||
|
|
||||||
Overruns and backpressure
|
|
||||||
-------------------------
|
|
||||||
|
|
||||||
Stream msgs land in a bounded per-context buffer on the receiver
|
|
||||||
side. A sender that outpaces a non-consuming receiver *overruns*
|
|
||||||
it and the runtime raises :exc:`tractor.StreamOverrun` (also a
|
|
||||||
:exc:`trio.TooSlowError`) instead of buffering without bound — SC
|
|
||||||
discipline applies to memory too.
|
|
||||||
|
|
||||||
Your knobs:
|
|
||||||
|
|
||||||
- ``msg_buffer_size`` on ``ctx.open_stream()`` sizes the buffer.
|
|
||||||
|
|
||||||
- ``allow_overruns=True`` (on ``Portal.open_context()`` and/or
|
|
||||||
``ctx.open_stream()``) opts in to absorbing overflow instead of
|
|
||||||
erroring — reasonable for bursty telemetry-ish feeds, just know
|
|
||||||
you're trading the error for extra buffering.
|
|
||||||
|
|
||||||
One context, one stream
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
A ``MsgStream`` is strictly **one-shot use**: once it closes —
|
|
||||||
gracefully or not, from either side — it can never be re-opened
|
|
||||||
on the same ctx. Want another round with the same peer? Open a
|
|
||||||
fresh context; they're cheap. The full close-vs-cancel teardown
|
|
||||||
story lives in :doc:`/guide/streaming`.
|
|
||||||
|
|
||||||
.. rubric:: Where to next?
|
|
||||||
|
|
||||||
:doc:`/guide/streaming` covers the rest of the msg-moving story:
|
|
||||||
the legacy one-way API, multi-actor pipelines and in-actor
|
|
||||||
broadcast fan-out. For exhaustive API detail see
|
|
||||||
:class:`tractor.Context`, :class:`tractor.MsgStream` and
|
|
||||||
:exc:`tractor.ContextCancelled`.
|
|
||||||
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
|
||||||
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
|
||||||
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
|
|
||||||
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
|
|
||||||
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
|
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
.. _#53: https://github.com/goodboy/tractor/issues/53
|
|
||||||
.. _#223: https://github.com/goodboy/tractor/issues/223
|
|
||||||
|
|
@ -1,545 +0,0 @@
|
||||||
.. _debugging:
|
|
||||||
|
|
||||||
================================
|
|
||||||
"Native" multi-process debugging
|
|
||||||
================================
|
|
||||||
|
|
||||||
``tractor`` ships the thing every ``multiprocessing`` user has
|
|
||||||
wished for and quietly assumed was impossible: a multi-process
|
|
||||||
debugger that *just works*.
|
|
||||||
|
|
||||||
Drop ``await tractor.pause()`` — or, with `greenback`_ installed,
|
|
||||||
a plain builtin ``breakpoint()`` — anywhere in any actor: the
|
|
||||||
root, a child, a grandchild, a sync helper function, even an
|
|
||||||
``asyncio`` task inside an "infected" actor. A full-featured
|
|
||||||
`pdbp`_ REPL opens *in that process*, with syntax-highlighted
|
|
||||||
source listings, tab completion and sticky mode, attached to your
|
|
||||||
one terminal.
|
|
||||||
|
|
||||||
Under the hood every REPL entry acquires a tree-global tty mutex
|
|
||||||
via an IPC request to the root actor, so prompts from concurrent
|
|
||||||
pauses and crashes never interleave. ``ctrl-c`` is shielded while
|
|
||||||
any REPL is live, so a stray ``SIGINT`` can't vaporize the tree
|
|
||||||
out from under you. And in debug mode any uncaught error drops
|
|
||||||
you into a crash REPL *first in the failing child*, then again at
|
|
||||||
each parent as the boxed :class:`~tractor.RemoteActorError` climbs
|
|
||||||
the supervision tree.
|
|
||||||
|
|
||||||
No remote-pdb sockets, no ``set_trace()`` port juggling, no
|
|
||||||
``ptrace`` attach dance: the debugger semantics you already know,
|
|
||||||
transparently extended across an entire process tree. Because
|
|
||||||
``tractor`` is a `structured concurrency`_ (SC) runtime, the
|
|
||||||
debugger composes with supervision instead of fighting it — quit
|
|
||||||
a REPL and errors keep propagating exactly like `trio`_ taught
|
|
||||||
you, ending in clean, zombie-free teardown.
|
|
||||||
|
|
||||||
We're pretty sure it's the (first ever?) "native" debugging UX
|
|
||||||
for multi-process Python B)
|
|
||||||
|
|
||||||
Enabling debug mode
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
Pass ``debug_mode=True`` to your runtime entrypoint, either
|
|
||||||
:func:`tractor.open_nursery` (which forwards it to the implicitly
|
|
||||||
opened root actor) or :func:`tractor.open_root_actor` directly:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery(
|
|
||||||
debug_mode=True, # arm the whole actor tree
|
|
||||||
) as an:
|
|
||||||
...
|
|
||||||
|
|
||||||
This arms the debug machinery *tree-wide*:
|
|
||||||
|
|
||||||
- crash handling is enabled in every actor: uncaught errors enter
|
|
||||||
a REPL before they propagate,
|
|
||||||
- the internal tty-lock module is auto-exposed over RPC to every
|
|
||||||
subactor (this is what makes the one-terminal handoff work),
|
|
||||||
- console logging is bumped to include ``PDB``-level status msgs
|
|
||||||
so you can see REPL acquire/release events as they happen.
|
|
||||||
|
|
||||||
You can instead flip it on for just one child, letting its
|
|
||||||
siblings crash-and-burn the normal way:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'sketchy_worker',
|
|
||||||
debug_mode=True, # OR-ed with the tree-wide flag
|
|
||||||
)
|
|
||||||
|
|
||||||
See ``examples/debugging/per_actor_debug.py`` for a runnable
|
|
||||||
proof of the selective style.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Debug mode requires the child-side runtime to be
|
|
||||||
``trio``-native so that the tty-lock IPC dialog works; it's
|
|
||||||
currently supported on the ``'trio'`` (default) and
|
|
||||||
``'main_thread_forkserver'`` spawn backends and raises
|
|
||||||
``RuntimeError`` for any other ``start_method``.
|
|
||||||
|
|
||||||
Your first pause point
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
:func:`tractor.pause` is the SC-aware, multi-process spelling of
|
|
||||||
the stdlib's ``breakpoint()``. In the root actor it looks almost
|
|
||||||
boring:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/root_actor_breakpoint.py
|
|
||||||
:caption: examples/debugging/root_actor_breakpoint.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Run it and you get a ``(Pdb+)`` prompt parked on the ``pause()``
|
|
||||||
line; type ``c`` (continue) and the program finishes normally.
|
|
||||||
|
|
||||||
The exact same call works from *any* subactor, no matter how deep
|
|
||||||
in the tree:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/subactor_breakpoint.py
|
|
||||||
:caption: examples/debugging/subactor_breakpoint.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Each loop iteration the child actor requests the terminal from
|
|
||||||
the root over IPC, REPLs you, then releases it on ``c``. Pause
|
|
||||||
points are re-entrant-safe: repeat calls from the same task are
|
|
||||||
no-op'd and other local tasks queue politely for the REPL.
|
|
||||||
|
|
||||||
When you get bored, type ``q`` (quit): the resulting
|
|
||||||
``bdb.BdbQuit`` is boxed and shipped to the parent like any other
|
|
||||||
remote error XD — causality is preserved even for your debugging
|
|
||||||
mistakes.
|
|
||||||
|
|
||||||
Crash REPLs: errors climb the tree
|
|
||||||
----------------------------------
|
|
||||||
|
|
||||||
Pause points are only half the story. With debug mode armed, any
|
|
||||||
*uncaught* error anywhere in the tree triggers what we call crash
|
|
||||||
handling mode:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/subactor_error.py
|
|
||||||
:caption: examples/debugging/subactor_error.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What happens when the child hits that (very intentional)
|
|
||||||
``NameError``:
|
|
||||||
|
|
||||||
1. a REPL opens **in the crashed child first** — you inspect the
|
|
||||||
raising frame, its locals, the works, right inside the failed
|
|
||||||
process,
|
|
||||||
2. when you quit, the error is boxed into a
|
|
||||||
:class:`~tractor.RemoteActorError` and relayed to the parent,
|
|
||||||
3. the parent (here the root) gets *its own* crash REPL with the
|
|
||||||
rendered remote traceback,
|
|
||||||
4. quit again and the nursery tears the tree down — errors keep
|
|
||||||
propagating per SC rules, no zombies left behind.
|
|
||||||
|
|
||||||
You debug the failure at every hop of the supervision tree, which
|
|
||||||
for multi-hop trees means you can chase an error from the leaf
|
|
||||||
that raised it all the way up to the root that supervises it.
|
|
||||||
|
|
||||||
Need to skip REPL entry for certain exceptions? Pass a predicate
|
|
||||||
via ``open_root_actor(debug_filter=...)``; by default
|
|
||||||
cancellation-only exception (groups) don't engage the REPL.
|
|
||||||
|
|
||||||
One terminal, many actors
|
|
||||||
-------------------------
|
|
||||||
|
|
||||||
So how do N processes share one tty without garbling it? The root
|
|
||||||
actor owns stdio for the whole tree and guards it with a FIFO
|
|
||||||
mutex; every subactor REPL entry is an IPC lock request to the
|
|
||||||
root. Exactly one actor-task in the entire tree can own the
|
|
||||||
terminal at a time, so prompts never interleave — ever.
|
|
||||||
|
|
||||||
.. d2:: diagrams/debug_lock.d2
|
|
||||||
:caption: Every REPL entry serializes through the root actor's
|
|
||||||
tty lock; ``continue``-ing one REPL hands the terminal to
|
|
||||||
the next waiter, FIFO style.
|
|
||||||
:alt: sequence diagram of two subactors serializing pdb REPL
|
|
||||||
access through the root actor's tty lock
|
|
||||||
|
|
||||||
The runtime's teardown paths cooperate too: a cancelling parent
|
|
||||||
always waits for any live REPL to release before reaping
|
|
||||||
children, so the debugger never gets yanked out from under you
|
|
||||||
mid-keystroke.
|
|
||||||
|
|
||||||
.. margin:: Watch the tree live
|
|
||||||
|
|
||||||
Run any of these examples with a process-tree watcher in a
|
|
||||||
second terminal and watch actors come and go::
|
|
||||||
|
|
||||||
watch -n 0.1 "pstree -a $$"
|
|
||||||
|
|
||||||
Here's the showpiece: one daemon child re-entering
|
|
||||||
``tractor.pause()`` forever inside a stream, while its sibling
|
|
||||||
repeatedly raises a ``NameError``:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/multi_daemon_subactors.py
|
|
||||||
:caption: examples/debugging/multi_daemon_subactors.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What you'll actually see
|
|
||||||
************************
|
|
||||||
|
|
||||||
Running it looks *roughly* like this (uids, tracebacks and source
|
|
||||||
listings elided; REPL order can vary with who wins the lock
|
|
||||||
race)::
|
|
||||||
|
|
||||||
$ python examples/debugging/multi_daemon_subactors.py
|
|
||||||
|
|
||||||
Opening a pdb REPL in paused actor: ('bp_forever', '<uuid>')
|
|
||||||
<highlighted source around the `await tractor.pause()` line>
|
|
||||||
(Pdb+) c
|
|
||||||
|
|
||||||
Opening a pdb REPL in crashed actor: ('name_error', '<uuid>')
|
|
||||||
<live traceback: NameError: name 'doggypants' is not defined>
|
|
||||||
(Pdb+) q
|
|
||||||
|
|
||||||
Opening a pdb REPL in crashed actor: ('root', '<uuid>')
|
|
||||||
<boxed RemoteActorError traceback relayed from 'name_error'>
|
|
||||||
(Pdb+) q
|
|
||||||
|
|
||||||
Two (then three) processes, one terminal, zero confusion:
|
|
||||||
``c``-ing out of the paused daemon's REPL releases the tty lock,
|
|
||||||
which immediately hands the prompt to the crashed sibling; quit
|
|
||||||
that and the error propagates as a fully-rendered
|
|
||||||
:class:`~tractor.RemoteActorError` to the parent where one final
|
|
||||||
crash REPL catches it before clean, zombie-free teardown.
|
|
||||||
|
|
||||||
For maximum drama run
|
|
||||||
``multi_nested_subactors_error_up_through_nurseries.py`` (under
|
|
||||||
``examples/debugging/``) which pulls the same trick across a
|
|
||||||
*three-deep* process tree — the tty lock keeps every prompt
|
|
||||||
orderly the whole way up.
|
|
||||||
|
|
||||||
Post-mortem, on demand
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
Crash handling is automatic, but you can also enter a REPL on
|
|
||||||
a live exception *manually* with :func:`tractor.post_mortem` —
|
|
||||||
the actor-aware equivalent of ``pdb.post_mortem()`` — from inside
|
|
||||||
any ``except`` block in any actor (kwargs: ``tb=`` for an
|
|
||||||
explicit traceback, plus ``shield=`` and ``hide_tb=``):
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/pm_in_subactor.py
|
|
||||||
:caption: examples/debugging/pm_in_subactor.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
This example demos three REPL entries from one error:
|
|
||||||
|
|
||||||
- the child's manual ``post_mortem()`` inside its ``except``,
|
|
||||||
- the runtime's automatic crash handler in the same child once
|
|
||||||
the error re-raises out of the RPC task,
|
|
||||||
- a manual ``post_mortem()`` in the parent on the received
|
|
||||||
:class:`~tractor.RemoteActorError`, whose ``.boxed_type``
|
|
||||||
faithfully reports the original ``NameError``.
|
|
||||||
|
|
||||||
Pausing from sync code
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
No ``await``? No problem. :func:`tractor.pause_from_sync` brings
|
|
||||||
the same tree-aware REPL to plain synchronous functions — handy
|
|
||||||
when the suspect code is three helpers deep and decidedly not
|
|
||||||
async.
|
|
||||||
|
|
||||||
It's powered by `greenback`_, which is optional, so you need to:
|
|
||||||
|
|
||||||
1. install it (it ships in ``tractor``'s ``sync_pause``
|
|
||||||
dependency group),
|
|
||||||
2. enable it at runtime entry:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery(
|
|
||||||
debug_mode=True,
|
|
||||||
maybe_enable_greenback=True,
|
|
||||||
) as an:
|
|
||||||
...
|
|
||||||
|
|
||||||
With that armed, sync code can pause from three different caller
|
|
||||||
environments: the main ``trio`` thread, ``trio.to_thread`` bg
|
|
||||||
threads, and (see the next section) ``asyncio`` tasks in infected
|
|
||||||
actors. The greenback "portal" hops back into the ``trio`` loop
|
|
||||||
to do the lock/REPL dance on your behalf:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/sync_bp.py
|
|
||||||
:caption: examples/debugging/sync_bp.py (the sync fn, excerpt)
|
|
||||||
:language: python
|
|
||||||
:pyobject: sync_pause
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/sync_bp.py
|
|
||||||
:caption: examples/debugging/sync_bp.py (called in a subactor,
|
|
||||||
excerpt)
|
|
||||||
:language: python
|
|
||||||
:pyobject: start_n_sync_pause
|
|
||||||
|
|
||||||
The full script also exercises the hairier root-actor bg-thread
|
|
||||||
cases (and documents their remaining sharp edges) if you want the
|
|
||||||
deep lore.
|
|
||||||
|
|
||||||
The builtin ``breakpoint()`` override
|
|
||||||
*************************************
|
|
||||||
|
|
||||||
When debug mode boots with greenback available, ``tractor`` wires
|
|
||||||
Python's `PEP 553`_ hook so the *builtin* ``breakpoint()`` becomes
|
|
||||||
the actor-aware sync pause, by exporting::
|
|
||||||
|
|
||||||
PYTHONBREAKPOINT=tractor.devx.debug._sync_pause_from_builtin
|
|
||||||
|
|
||||||
That means third-party and legacy code containing bare
|
|
||||||
``breakpoint()`` calls debugs correctly inside your actor tree
|
|
||||||
with zero edits (the override even forwards kwargs like
|
|
||||||
``hide_tb`` to the underlying pause machinery, as shown in the
|
|
||||||
excerpt above).
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
Without greenback (or with ``maybe_enable_greenback=False``,
|
|
||||||
the default), ``debug_mode=True`` instead *blocks* the builtin
|
|
||||||
``breakpoint()``: ``sys.breakpointhook`` is swapped for a
|
|
||||||
raiser and ``PYTHONBREAKPOINT=0`` is set. A naive
|
|
||||||
``breakpoint()`` from some random process would clobber the
|
|
||||||
shared tty, so we'd rather hand you a loud ``RuntimeError``
|
|
||||||
with install instructions.
|
|
||||||
|
|
||||||
Both the hook and the env var are restored to their prior values
|
|
||||||
on runtime exit — see
|
|
||||||
``examples/debugging/restore_builtin_breakpoint.py`` for the
|
|
||||||
proof.
|
|
||||||
|
|
||||||
Breakpoints inside ``asyncio`` tasks
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
Yes, even "infected ``asyncio``" actors get the goods. Spawn a
|
|
||||||
child with ``infect_asyncio=True`` (``trio`` runs as a guest on
|
|
||||||
the ``asyncio`` loop inside it) and, with debug mode + greenback
|
|
||||||
armed, every ``asyncio`` task started via ``tractor.to_asyncio``
|
|
||||||
is automatically granted a greenback portal — so a plain builtin
|
|
||||||
``breakpoint()`` (or ``tractor.pause_from_sync()``) inside an
|
|
||||||
``asyncio.Task`` joins the same single-terminal, tree-locked REPL
|
|
||||||
flow:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/asyncio_bp.py
|
|
||||||
:caption: examples/debugging/asyncio_bp.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Note the interleave: a ``breakpoint()`` on the ``asyncio`` side,
|
|
||||||
``tractor.pause()`` on the ``trio`` side of the same actor, and
|
|
||||||
another pause up in the root — all serialized through the one tty
|
|
||||||
lock with no cross-actor (or cross-event-loop!) clobbering.
|
|
||||||
|
|
||||||
One catch: ``asyncio`` tasks spawned *out-of-band* — i.e. not via
|
|
||||||
``tractor.to_asyncio``, typically by some third-party aio lib —
|
|
||||||
have no portal bestowed, so a sync pause from one raises a loud
|
|
||||||
``RuntimeError`` telling you to ``greenback.ensure_portal()``
|
|
||||||
first. See :ref:`the caveats <debugging-caveats>` below.
|
|
||||||
|
|
||||||
Teardown debugging: the shielded pause
|
|
||||||
--------------------------------------
|
|
||||||
|
|
||||||
`Cancellation`_ is ``trio``'s bread and butter, which raises an
|
|
||||||
awkward question: how do you REPL inside an *already-cancelled*
|
|
||||||
scope, say while debugging some teardown sequence? A bare
|
|
||||||
``pause()`` would itself be cancelled at its next checkpoint.
|
|
||||||
|
|
||||||
The answer is ``await tractor.pause(shield=True)``, which wraps
|
|
||||||
the lock acquisition and REPL session in a shielded cancel scope
|
|
||||||
(``post_mortem(shield=True)`` works the same way):
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/shielded_pause.py
|
|
||||||
:caption: examples/debugging/shielded_pause.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
If you forget, ``tractor`` has your back: an unshielded
|
|
||||||
``pause()`` from a cancelled scope fails fast with a hint
|
|
||||||
suggesting ``await tractor.pause(shield=True)`` instead of
|
|
||||||
silently never REPL-ing.
|
|
||||||
|
|
||||||
Go ahead, mash ctrl-c
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
While any REPL is live the runtime installs a custom ``SIGINT``
|
|
||||||
handler tree-wide so that a reflexive ``ctrl-c`` (or five) can't
|
|
||||||
nuke your debug session:
|
|
||||||
|
|
||||||
- the actor that owns the REPL ignores the interrupt and simply
|
|
||||||
re-flushes the prompt — keep mashing, it's fine,
|
|
||||||
- the root actor ignores ``SIGINT`` while a still-IPC-connected
|
|
||||||
child holds the tty lock, so the supervisor won't tear down the
|
|
||||||
tree out from under the debugger,
|
|
||||||
- if the lock state has gone *stale* — the locking child died or
|
|
||||||
its IPC channel dropped — the root cancels the stale lock scope
|
|
||||||
and restores ``trio``'s default handler, so ``ctrl-c`` works
|
|
||||||
again exactly when it should.
|
|
||||||
|
|
||||||
The handler is uninstalled and ``trio``'s own ``SIGINT``
|
|
||||||
semantics restored every time a REPL releases (on ``continue`` /
|
|
||||||
``quit``).
|
|
||||||
|
|
||||||
Live task-tree dumps
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
Sometimes there's no error to catch — the tree is just *hung* and
|
|
||||||
you want to know where. For that ``tractor`` integrates
|
|
||||||
`stackscope`_: send a signal, get a full ``trio`` task-tree dump
|
|
||||||
from every actor in the tree.
|
|
||||||
|
|
||||||
Enable it any of three ways:
|
|
||||||
|
|
||||||
- ``open_root_actor(enable_stack_on_sig=True)`` (or via
|
|
||||||
``open_nursery()`` which forwards it),
|
|
||||||
- set ``TRACTOR_ENABLE_STACKSCOPE=1`` in the env — it's inherited
|
|
||||||
through the process tree so every (sub)actor arms the handler
|
|
||||||
at boot,
|
|
||||||
- call ``tractor.devx.enable_stack_on_sig()`` directly.
|
|
||||||
|
|
||||||
It's intentionally *not* gated on ``debug_mode`` so you can leave
|
|
||||||
it armed in plain runs. Then, when the hang strikes, signal the
|
|
||||||
tree with ``SIGUSR1``.
|
|
||||||
|
|
||||||
.. tip::
|
|
||||||
|
|
||||||
No need to hunt down pids — pattern-match the original cmdline
|
|
||||||
with ``pkill``::
|
|
||||||
|
|
||||||
$ pkill --signal SIGUSR1 -f "python example_script.py"
|
|
||||||
|
|
||||||
Each actor dumps its entire ``trio`` task tree (full nursery
|
|
||||||
recursion via ``stackscope.extract()``) to its tty *and* tees it
|
|
||||||
to ``/tmp/tractor-stackscope-<pid>.log`` — so the trace survives
|
|
||||||
even under captured-stdio harnesses — then relays the signal on
|
|
||||||
to its children, parent-before-child, until the whole tree has
|
|
||||||
reported in.
|
|
||||||
|
|
||||||
Try it yourself with the demo script, which deliberately hangs a
|
|
||||||
subactor in a shielded sleep:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/debugging/shield_hang_in_sub.py
|
|
||||||
:caption: examples/debugging/shield_hang_in_sub.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
(That ``trio.CancelScope(shield=True)`` hang also shows off the
|
|
||||||
zombie reaper: ``ctrl-c`` the root and the un-cancellable child
|
|
||||||
still gets hard-reaped — if you can create a zombie it **is a
|
|
||||||
bug**.)
|
|
||||||
|
|
||||||
Crash handling for sync and CLI code
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
All of the above rides on the actor runtime, but crashes don't
|
|
||||||
politely wait for ``trio.run()``. For plain sync code — think
|
|
||||||
``typer``/``click`` CLI endpoints, config parsing, anything
|
|
||||||
pre-runtime — there's a sync context manager that wraps the same
|
|
||||||
``pdbp`` post-mortem UX:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor.devx import open_crash_handler
|
|
||||||
|
|
||||||
def main(): # any sync code, no runtime required
|
|
||||||
with open_crash_handler() as boxed:
|
|
||||||
run_my_cli_thing()
|
|
||||||
|
|
||||||
By default any ``BaseException`` (minus an ``ignore`` set
|
|
||||||
defaulting to ``KeyboardInterrupt`` and ``trio.Cancelled``)
|
|
||||||
enters the REPL then re-raises on exit; pass
|
|
||||||
``raise_on_exit=False`` to suppress instead and introspect the
|
|
||||||
``boxed.value`` afterward. The ``catch``/``ignore`` sets and a
|
|
||||||
``repl_fixture`` are all tweakable.
|
|
||||||
|
|
||||||
For the classic ``--pdb`` CLI-flag pattern use the conditional
|
|
||||||
variant:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor.devx import maybe_open_crash_handler
|
|
||||||
|
|
||||||
@app.command() # a `typer` (or `click`) endpoint
|
|
||||||
def cmd(pdb: bool = False):
|
|
||||||
with maybe_open_crash_handler(pdb=pdb):
|
|
||||||
...
|
|
||||||
|
|
||||||
REPL niceties and hooks
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
Every REPL in this guide is a `pdbp`_ instance (the maintained
|
|
||||||
fork-and-fix of `pdb++`_) pre-configured by ``tractor``:
|
|
||||||
|
|
||||||
- pygments syntax highlighting in listings and tracebacks,
|
|
||||||
- tab completion — including an automatic fixup for
|
|
||||||
libedit-compiled CPythons (e.g. ``uv``-distributed pythons),
|
|
||||||
- sticky mode available via the ``sticky`` command (off by
|
|
||||||
default),
|
|
||||||
- no long-line truncation (terminal resizes behave),
|
|
||||||
- the ``(Pdb+)`` prompt, ``ll``, hidden-frames support and the
|
|
||||||
rest of the ``pdb++`` goodies you may already know.
|
|
||||||
|
|
||||||
Internal runtime frames are traceback-hidden so the REPL lands
|
|
||||||
exactly on *your* ``pause()``-call or crash frame, never on
|
|
||||||
``tractor`` guts.
|
|
||||||
|
|
||||||
Finally, if your app owns the terminal (TUIs, fullscreen
|
|
||||||
dashboards) pass ``repl_fixture=<your ctx mngr>`` to ``pause()``,
|
|
||||||
``post_mortem()`` or ``open_crash_handler()``: it's entered just
|
|
||||||
before the REPL engages (return ``False`` to skip entry entirely)
|
|
||||||
and exited on release — perfect for suspending and restoring your
|
|
||||||
screen around a debug session.
|
|
||||||
|
|
||||||
.. _debugging-caveats:
|
|
||||||
|
|
||||||
Caveats and platform notes
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
An honest list of the current rough edges:
|
|
||||||
|
|
||||||
- **Windows**: the debugger has no CI coverage on windows at all
|
|
||||||
(the entire test module is skipped there); manual testing has
|
|
||||||
shown it *can* work, but you're in uncharted territory —
|
|
||||||
reports welcome!
|
|
||||||
- **macOS**: supported but with rough edges: special-cased prompt
|
|
||||||
re-flushing for ``bash``-on-darwin, a few tooling tests skipped
|
|
||||||
on CI, and the AF_UNIX ~104-char socket-path limit forces some
|
|
||||||
examples (like the stackscope demo above) to fall back from
|
|
||||||
``'uds'`` to ``'tcp'`` transport. Wonder if all of it'll work
|
|
||||||
on OS X? So do we.
|
|
||||||
- **CPython 3.14**: ``greenback`` (via ``greenlet``) doesn't
|
|
||||||
support 3.14 yet, so ``pause_from_sync()`` and the builtin
|
|
||||||
``breakpoint()`` override are effectively 3.13-only for now.
|
|
||||||
The async APIs — ``pause()`` and ``post_mortem()`` — need no
|
|
||||||
greenback and work everywhere.
|
|
||||||
- **out-of-band** ``asyncio`` **tasks**: sync pauses from aio
|
|
||||||
tasks *not* spawned via ``tractor.to_asyncio`` raise a
|
|
||||||
``RuntimeError`` (no greenback portal was bestowed); run
|
|
||||||
``await greenback.ensure_portal()`` inside such a task first.
|
|
||||||
- **nested-tree ctrl-c edges**: ``SIGINT`` relay through
|
|
||||||
intermediary parents that aren't themselves in debug mode still
|
|
||||||
has known rough edges — see `#320`_.
|
|
||||||
- **captured stdio**: ``pytest``-style output capture can hang a
|
|
||||||
``pause()``; use a real terminal (or a pty à la ``pexpect``,
|
|
||||||
which is how ``tractor``'s own suite drives every one of these
|
|
||||||
examples).
|
|
||||||
|
|
||||||
Where to next?
|
|
||||||
--------------
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` — the SC-linked cross-actor task API
|
|
||||||
that all the crash-propagation semantics above ride on.
|
|
||||||
- :func:`tractor.pause`, :func:`tractor.post_mortem` and
|
|
||||||
:func:`tractor.pause_from_sync` in the API reference.
|
|
||||||
- ``examples/debugging/`` — 20-odd runnable scripts, nearly
|
|
||||||
every one exercised by the test suite through a real pty.
|
|
||||||
|
|
||||||
.. _structured concurrency:
|
|
||||||
https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _cancellation: https://trio.readthedocs.io/en/latest/
|
|
||||||
reference-core.html#cancellation-and-timeouts
|
|
||||||
.. _pdbp: https://github.com/mdmintz/pdbp
|
|
||||||
.. _pdb++: https://github.com/pdbpp/pdbpp
|
|
||||||
.. _greenback: https://github.com/oremanj/greenback
|
|
||||||
.. _stackscope: https://github.com/oremanj/stackscope
|
|
||||||
.. _PEP 553: https://peps.python.org/pep-0553/
|
|
||||||
.. _#320: https://github.com/goodboy/tractor/issues/320
|
|
||||||
|
|
@ -1,255 +0,0 @@
|
||||||
Actor discovery
|
|
||||||
===============
|
|
||||||
|
|
||||||
So you've spawned a tree of trio-"actors"; now their tasks need to
|
|
||||||
*find* each other to start a dialog. ``tractor`` ships a (self
|
|
||||||
admittedly) **very naive** discovery system which is nonetheless
|
|
||||||
mighty handy for wiring up service-style apps: a built-in
|
|
||||||
*registrar* actor plus a small set of lookup APIs that deliver
|
|
||||||
a live, connected ``Portal`` to whichever peer you're after.
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:margin:
|
|
||||||
:caption: The root actor doubles as the *registrar* by
|
|
||||||
default; every spawned actor registers itself with it.
|
|
||||||
:alt: actor tree with root acting as registrar
|
|
||||||
|
|
||||||
Because ``tractor`` is built on structured concurrency (SC), the
|
|
||||||
discovery layer is *not* some external etcd/consul-shaped service
|
|
||||||
you have to babysit; it's just another actor — normally the root
|
|
||||||
of your tree — doing a bit of bookkeeping as part of the runtime.
|
|
||||||
|
|
||||||
Every actor phones home
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
On runtime boot **every** actor self-registers with the registrar:
|
|
||||||
it submits its unique ``(name, uuid)`` identity pair (aka its
|
|
||||||
``uid``) mapped to the list of transport addresses its IPC server
|
|
||||||
is bound to. On graceful teardown it likewise *un*-registers, so
|
|
||||||
the registry tracks the live tree as it grows and shrinks.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
Actor names are **not** enforced unique — the registry is keyed
|
|
||||||
by the full ``(name, uuid)`` pair. Name-based lookups simply
|
|
||||||
resolve to the *last* registered match, so if you boot five
|
|
||||||
actors all named ``'bob'``, you get the freshest ``'bob'`` B)
|
|
||||||
|
|
||||||
First boot: who's the registrar?
|
|
||||||
--------------------------------
|
|
||||||
|
|
||||||
By default the **root actor** *is* the registrar; subactors
|
|
||||||
inherit the tree's ``registry_addrs`` at spawn time so the whole
|
|
||||||
clan shares one registry with zero config on your part.
|
|
||||||
|
|
||||||
The bootstrap rule inside ``open_root_actor()`` is delightfully
|
|
||||||
simple:
|
|
||||||
|
|
||||||
- on boot, ping every socket addr in ``registry_addrs``; when none
|
|
||||||
are passed the per-transport defaults are used: for TCP the
|
|
||||||
loopback ``('127.0.0.1', 1616)``, for UDS a
|
|
||||||
``registry@1616.sock`` file,
|
|
||||||
|
|
||||||
- if a registrar answers, you boot as a plain (non-registrar) root
|
|
||||||
actor and register with the *existing* registry; your own IPC
|
|
||||||
server binds random same-transport addrs instead,
|
|
||||||
|
|
||||||
- if **nothing answers, congratulations: you just became the
|
|
||||||
registrar**. Your transport server binds the registry addrs
|
|
||||||
themselves and you start serving lookups for everyone else.
|
|
||||||
|
|
||||||
Pass ``ensure_registry=True`` when your program *requires* being
|
|
||||||
the one-and-only registrar; boot then fails loudly with a
|
|
||||||
``RuntimeError`` if some other process already bound the registry
|
|
||||||
socket(s).
|
|
||||||
|
|
||||||
Looking up actors
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
All lookup APIs are async context managers, so the SC rule you
|
|
||||||
already know from the rest of ``tractor`` holds here too: any
|
|
||||||
delivered portal (and its underlying IPC channel) is scoped to
|
|
||||||
your ``async with`` block — no dangling connections.
|
|
||||||
|
|
||||||
``find_actor()``
|
|
||||||
****************
|
|
||||||
|
|
||||||
The workhorse: ask the registrar for ``name`` and connect a portal
|
|
||||||
to the match, or get ``None`` back when nobody's home:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.find_actor('data_feed') as portal:
|
|
||||||
if portal is None:
|
|
||||||
... # not registered anywhere; maybe spawn it?
|
|
||||||
else:
|
|
||||||
await portal.run(do_stuff)
|
|
||||||
|
|
||||||
Knobs worth knowing:
|
|
||||||
|
|
||||||
- ``registry_addrs=[...]``: query specific (possibly multiple,
|
|
||||||
possibly remote) registrars instead of your tree's default,
|
|
||||||
|
|
||||||
- ``only_first=False``: deliver a ``list[Portal]`` of *all*
|
|
||||||
matches found across the queried registrars instead of just the
|
|
||||||
first,
|
|
||||||
|
|
||||||
- ``raise_on_none=True``: raise a ``RuntimeError`` instead of
|
|
||||||
yielding ``None`` when no match is found — for when absence is
|
|
||||||
a hard error in your app.
|
|
||||||
|
|
||||||
``wait_for_actor()``
|
|
||||||
********************
|
|
||||||
|
|
||||||
Blocks until *someone* registers under ``name``, then yields a
|
|
||||||
portal to that registree. Perfect for "wait for my sibling service
|
|
||||||
to come up" sequencing:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.wait_for_actor('service') as portal:
|
|
||||||
await portal.run(some_fn)
|
|
||||||
|
|
||||||
``query_actor()``
|
|
||||||
*****************
|
|
||||||
|
|
||||||
A lookup *without* connecting to the target: yields an
|
|
||||||
``(addr, reg_portal)`` pair where ``addr`` is the peer's preferred
|
|
||||||
transport address, or ``None`` when nothing is registered under
|
|
||||||
that name. Use it for liveness peeks or to log where a service
|
|
||||||
lives without actually dialing it up.
|
|
||||||
|
|
||||||
``get_registry()``
|
|
||||||
******************
|
|
||||||
|
|
||||||
Yields a portal straight to the registrar actor itself — or a
|
|
||||||
``LocalPortal`` shim when the calling actor *is* the registrar
|
|
||||||
(no IPC required to talk to yourself, hopefully).
|
|
||||||
|
|
||||||
Fast paths and address preference
|
|
||||||
---------------------------------
|
|
||||||
|
|
||||||
Before doing any RPC to the registrar, every lookup first scans
|
|
||||||
the calling actor's *already-connected peers*: if you have a live
|
|
||||||
channel to an actor named ``name`` you get a portal over it
|
|
||||||
immediately, no registrar round-trip at all.
|
|
||||||
|
|
||||||
When a registry entry holds *multiple* addresses (a multihomed
|
|
||||||
actor) the "best" one is chosen by locality:
|
|
||||||
|
|
||||||
1. UDS — same-host guaranteed, lowest overhead,
|
|
||||||
|
|
||||||
2. local TCP — loopback or any of this host's own interface
|
|
||||||
addrs,
|
|
||||||
|
|
||||||
3. remote TCP — the only option when actually distributed.
|
|
||||||
|
|
||||||
Within a tier the most recently registered addr wins. Stale
|
|
||||||
entries (an addr that no longer accepts connections) are detected
|
|
||||||
on use and deleted from the registrar's table on your behalf.
|
|
||||||
|
|
||||||
Demo: register and find a service
|
|
||||||
---------------------------------
|
|
||||||
|
|
||||||
The simplest possible spin: start a subactor, ask the registrar
|
|
||||||
where it lives, and wait on its registration:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/service_discovery.py
|
|
||||||
:caption: examples/service_discovery.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
The daemon-service pattern
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
The classic deployment shape: a long-lived daemon actor serves
|
|
||||||
RPC, later-running code discovers it by name, calls in, and
|
|
||||||
gracefully cancels it when the job is done:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/service_daemon_discovery.py
|
|
||||||
:caption: examples/service_daemon_discovery.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Note the teardown ordering — *graceful cancel* of the daemon via
|
|
||||||
its portal is part of the pattern; under SC a "service" is still
|
|
||||||
somebody's child and somebody is responsible for reaping it.
|
|
||||||
|
|
||||||
Joining an existing tree from outside
|
|
||||||
-------------------------------------
|
|
||||||
|
|
||||||
Discovery isn't limited to a single program: any standalone script
|
|
||||||
can join a running tree by booting its *own* root actor pointed at
|
|
||||||
the existing registrar:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
async def main():
|
|
||||||
async with (
|
|
||||||
# contact the live tree's registrar
|
|
||||||
tractor.open_root_actor(
|
|
||||||
registry_addrs=[('127.0.0.1', 1616)],
|
|
||||||
),
|
|
||||||
tractor.find_actor('data_feed') as portal,
|
|
||||||
):
|
|
||||||
... # RPC away like you were born here
|
|
||||||
|
|
||||||
trio.run(main)
|
|
||||||
|
|
||||||
Per the bootstrap rules above, if the registrar at those addrs is
|
|
||||||
*not* reachable this process simply becomes its own (registrar)
|
|
||||||
root — so the same code works standalone and as a tree-joiner.
|
|
||||||
|
|
||||||
"Arbiter"? A legacy naming note
|
|
||||||
-------------------------------
|
|
||||||
|
|
||||||
In older releases (and many an old blog post or issue thread) the
|
|
||||||
registrar actor was called the *arbiter*, with matching APIs like
|
|
||||||
``get_arbiter()`` and an ``arbiter_addr`` argument. All of that
|
|
||||||
terminology is retired: it's *registrar*/*registry* everywhere now
|
|
||||||
(``registry_addrs``, ``get_registry()``, ...) and the
|
|
||||||
``tractor.Arbiter`` export survives only as a back-compat alias of
|
|
||||||
``tractor.Registrar``. If you see "arbiter" somewhere, mentally
|
|
||||||
substitute "registrar" and you're up to date.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
Multihoming nerds: ``tractor.discovery`` also ships
|
|
||||||
libp2p-style *multiaddr* helpers — ``mk_maddr()`` and
|
|
||||||
``parse_maddr()`` — for describing transport endpoints as
|
|
||||||
structured strings.
|
|
||||||
|
|
||||||
Very naive, very honest
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
To be clear, this is a **very naive** discovery system: one
|
|
||||||
process-tree-local registrar holding a dict, no replication, no
|
|
||||||
re-election when it dies, no cross-host propagation. That's
|
|
||||||
intentional (for now); it covers the "wire up my services on this
|
|
||||||
host" case without dragging in a consensus protocol.
|
|
||||||
|
|
||||||
On the roadmap (issue `#216`_ tracks a chunk of it):
|
|
||||||
|
|
||||||
- registrar high(er)-availability: staying up past tree teardown
|
|
||||||
and re-election,
|
|
||||||
|
|
||||||
- a `gossip protocol`_ for decentralized cross-host discovery (the
|
|
||||||
zguide's `discovery`_ chapter is the spiritual reference),
|
|
||||||
|
|
||||||
- `modern protocol`_ (rendezvous) style meet-up points.
|
|
||||||
|
|
||||||
If any of that scratches your itch, the issue tracker would love
|
|
||||||
to hear from you.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
- :doc:`/guide/testing` — watching live actor trees (and their
|
|
||||||
registrar) while the test suite or your app runs.
|
|
||||||
- API refs: :func:`tractor.find_actor`,
|
|
||||||
:func:`tractor.wait_for_actor`,
|
|
||||||
:func:`tractor.query_actor`,
|
|
||||||
:func:`tractor.get_registry`,
|
|
||||||
:class:`tractor.Registrar`.
|
|
||||||
|
|
||||||
.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol
|
|
||||||
.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
|
|
||||||
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
|
|
||||||
.. _#216: https://github.com/goodboy/tractor/issues/216
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
Guides
|
|
||||||
======
|
|
||||||
Task-focused walkthroughs of every major ``tractor``
|
|
||||||
subsystem, each built around real, *test-suite
|
|
||||||
verified* example scripts from the repo's
|
|
||||||
``examples/`` dir (we never copy-paste code into
|
|
||||||
docs; what you read is what CI runs).
|
|
||||||
|
|
||||||
Roughly in "first date to long term relationship"
|
|
||||||
order,
|
|
||||||
|
|
||||||
- :doc:`spawning` — actor nurseries, daemons +
|
|
||||||
one-shot workers, process lifetimes.
|
|
||||||
- :doc:`rpc` — portals: calling into another
|
|
||||||
process like it's a local ``await``.
|
|
||||||
- :doc:`context` — the cross-actor task-pair
|
|
||||||
primitive at the heart of modern ``tractor``.
|
|
||||||
- :doc:`streaming` — one-way and bidirectional
|
|
||||||
msg streams between actors.
|
|
||||||
- :doc:`cancellation` — supervision, error
|
|
||||||
boxing + propagation, teardown discipline.
|
|
||||||
- :doc:`debugging` — the multi-process native
|
|
||||||
REPL debugger; our flagship DX feature B)
|
|
||||||
- :doc:`discovery` — the registrar, finding
|
|
||||||
actors by name, service patterns.
|
|
||||||
- :doc:`clustering` — quick flat process
|
|
||||||
clusters via one ``async with``.
|
|
||||||
- :doc:`parallelism` — worker pools without
|
|
||||||
pools; a ``concurrent.futures`` re-think.
|
|
||||||
- :doc:`asyncio` — "infected asyncio" mode:
|
|
||||||
SC-supervise ``asyncio`` tasks from ``trio``.
|
|
||||||
- :doc:`msging` — typed IPC payloads, the wire
|
|
||||||
msg-spec and custom codecs.
|
|
||||||
- :doc:`testing` — running + monitoring the
|
|
||||||
test suite (and testing your own actor apps).
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:hidden:
|
|
||||||
:maxdepth: 1
|
|
||||||
|
|
||||||
spawning
|
|
||||||
rpc
|
|
||||||
context
|
|
||||||
streaming
|
|
||||||
cancellation
|
|
||||||
debugging
|
|
||||||
discovery
|
|
||||||
clustering
|
|
||||||
parallelism
|
|
||||||
asyncio
|
|
||||||
msging
|
|
||||||
testing
|
|
||||||
|
|
@ -1,260 +0,0 @@
|
||||||
Typed messaging
|
|
||||||
===============
|
|
||||||
Every value that crosses an actor boundary rides inside a typed
|
|
||||||
msg. ``tractor`` ships a small, fixed family of msg types, the
|
|
||||||
"SC-transitive supervision protocol", which encapsulates *all*
|
|
||||||
RPC dialogs in the tree such that `structured concurrency`_ (SC)
|
|
||||||
semantics -- parent-child task linkage, error propagation,
|
|
||||||
graceful cancellation -- hold across every process hop. On top of
|
|
||||||
that
|
|
||||||
protocol you can layer **your own** payload type contracts,
|
|
||||||
per-endpoint, and have them enforced at runtime by the codec.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Older posts and readmes claim ``tractor`` "uses
|
|
||||||
``msgpack``(-python)" on the wire. The wire *encoding* is still
|
|
||||||
msgpack, but since ``0.1.0a5`` all codec work is done by
|
|
||||||
msgspec_ against a strictly-typed, tagged-union msg-spec;
|
|
||||||
neither ``msgpack-python`` nor ``u-msgpack`` are involved.
|
|
||||||
|
|
||||||
The wire format
|
|
||||||
---------------
|
|
||||||
Each protocol msg is a :class:`msgspec.Struct` subtype declared
|
|
||||||
with ``tag=True, tag_field='msg_type'``, so the full set decodes
|
|
||||||
as a `tagged union`__ with zero dispatch code of our own. The
|
|
||||||
payload-carrying msgs all inherit from ``PayloadMsg`` which boxes:
|
|
||||||
|
|
||||||
__ https://jcristharif.com/msgspec/structs.html#tagged-unions
|
|
||||||
|
|
||||||
- ``.cid`` -- the "context id" identifying which dialog (i.e.
|
|
||||||
which ``Context``) the msg belongs to,
|
|
||||||
- ``.pld`` -- the *payload*, aka your app's actual data.
|
|
||||||
|
|
||||||
Decoding is deliberately two-layered:
|
|
||||||
|
|
||||||
- the **transport codec** decodes only the protocol *envelope*,
|
|
||||||
intentionally leaving ``.pld`` as raw bytes
|
|
||||||
(:class:`msgspec.Raw`),
|
|
||||||
- a **per-context payload-receiver** (the internal ``PldRx``) then
|
|
||||||
decodes each ``.pld`` against *that* dialog's user-defined type
|
|
||||||
spec.
|
|
||||||
|
|
||||||
This split is what lets every ``Context`` carry its own msg-spec
|
|
||||||
without reconfiguring the shared transport, keeps the runtime's
|
|
||||||
own traffic immune to your app's spec choices, and makes any
|
|
||||||
validation failure attributable to exactly one dialog (and thus
|
|
||||||
one task pair) instead of nuking the whole channel.
|
|
||||||
|
|
||||||
The protocol family
|
|
||||||
-------------------
|
|
||||||
The entire msg-spec is ten types, all importable from
|
|
||||||
``tractor.msg`` (defined in ``tractor.msg.types``):
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 18 82
|
|
||||||
|
|
||||||
* - msg type
|
|
||||||
- role
|
|
||||||
* - ``Aid``
|
|
||||||
- actor-identity handshake; the first thing two peers
|
|
||||||
exchange on connect (name, uuid, pid).
|
|
||||||
* - ``SpawnSpec``
|
|
||||||
- parent -> child runtime config sent right after ``Aid``:
|
|
||||||
enabled modules, registry/bind addrs, runtime vars.
|
|
||||||
* - ``Start``
|
|
||||||
- request to remotely schedule an RPC task: target
|
|
||||||
namespace + func name, kwargs and the caller's uid.
|
|
||||||
* - ``StartAck``
|
|
||||||
- the callee's ack declaring the endpoint's "functype":
|
|
||||||
``asyncfunc``, ``asyncgen`` or ``context``.
|
|
||||||
* - ``Started``
|
|
||||||
- the first value passed to ``ctx.started()``; completes
|
|
||||||
the context handshake.
|
|
||||||
* - ``Yield``
|
|
||||||
- one streamed value per ``MsgStream.send()`` call.
|
|
||||||
* - ``Stop``
|
|
||||||
- graceful stream termination; the IPC rendition of
|
|
||||||
``StopAsyncIteration``.
|
|
||||||
* - ``Return``
|
|
||||||
- the final return value of the remote task fn.
|
|
||||||
* - ``CancelAck``
|
|
||||||
- ``bool`` result of a runtime cancel-request; always
|
|
||||||
decodable so graceful cancellation can never be broken
|
|
||||||
by a custom msg-spec.
|
|
||||||
* - ``Error``
|
|
||||||
- a boxed remote exception (src uid, relay path, tb str,
|
|
||||||
..) relayed for local re-raise as
|
|
||||||
:class:`tractor.RemoteActorError`.
|
|
||||||
|
|
||||||
Squint and you'll see an SC task scope serialized onto the wire:
|
|
||||||
every dialog opens with ``Start``/``StartAck`` (plus ``Started``
|
|
||||||
for ``@tractor.context`` endpoints), optionally streams
|
|
||||||
``Yield``-s until a ``Stop``, and **always** terminates with
|
|
||||||
exactly one of ``Return``, ``Error`` or ``CancelAck``. That 1:1
|
|
||||||
mapping of msg sequence onto a cross-process task pair is why we
|
|
||||||
call the protocol *SC-transitive*: supervision semantics survive
|
|
||||||
every hop of the tree. In `(un)protocol`_ terms it's our "SC
|
|
||||||
dialog un-protocol".
|
|
||||||
|
|
||||||
For introspection the union alias ``tractor.msg.MsgType``, the
|
|
||||||
list ``__msg_types__`` and the spec alias ``__msg_spec__`` are
|
|
||||||
all exported.
|
|
||||||
|
|
||||||
Payload typing with ``pld_spec``
|
|
||||||
--------------------------------
|
|
||||||
By default ``.pld`` may be any msgspec-supported type, i.e. the
|
|
||||||
spec is ``Any``. To constrain a single endpoint's dialog, pass
|
|
||||||
a type (union) to the decorator:
|
|
||||||
``@tractor.context(pld_spec=MyStruct|None)``. The spec then
|
|
||||||
applies to all payload-carrying msgs of that dialog --
|
|
||||||
``Started``, ``Yield`` and ``Return`` -- on both sides of the
|
|
||||||
IPC. Pro tip: keep ``None`` in your union since most endpoints
|
|
||||||
implicitly ``return None`` and a bare ``ctx.started()`` ships
|
|
||||||
``None`` too.
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/typed_payloads.py
|
|
||||||
:caption: examples/typed_payloads.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on?
|
|
||||||
|
|
||||||
- the payload schema is just a :class:`msgspec.Struct` subtype;
|
|
||||||
anything msgspec can tag and decode works, including unions
|
|
||||||
of structs, builtins and containers.
|
|
||||||
- decorating with ``@tractor.context(pld_spec=...)`` attaches the
|
|
||||||
spec to the endpoint; both peers' payload-receivers now decode
|
|
||||||
this dialog's payloads against it. No spec sharing files, no
|
|
||||||
IDL compiler, the contract *is* the Python type.
|
|
||||||
- the happy path looks identical to untyped code: the child calls
|
|
||||||
``await ctx.started(<conforming value>)``, streams or returns
|
|
||||||
more conforming values, and the parent receives fully decoded
|
|
||||||
struct instances (not dicts!) on its side.
|
|
||||||
- the sad path is the point: shipping a value *outside* the spec
|
|
||||||
raises :class:`tractor.MsgTypeError`, which the example catches
|
|
||||||
to show off the failure mode; see the anatomy section below for
|
|
||||||
exactly where it gets raised.
|
|
||||||
|
|
||||||
Where validation happens: cheap-or-nasty
|
|
||||||
----------------------------------------
|
|
||||||
A naive impl would validate every payload on both send *and*
|
|
||||||
receive, doubling your codec bill exactly where throughput
|
|
||||||
matters most. Instead ``tractor`` follows the 0mq lords'
|
|
||||||
"`cheap or nasty`_" pattern: be **nasty** (strict, eager,
|
|
||||||
expensive) on the rare control msgs and **cheap** (lazy, fast) on
|
|
||||||
the high-rate stream path.
|
|
||||||
|
|
||||||
- ``Started`` is the *only* payload that gets the full nasty
|
|
||||||
treatment: ``ctx.started(value)`` stringently
|
|
||||||
**roundtrip-checks** the encoded msg against the dialog's spec
|
|
||||||
*before* sending, so a non-conforming first value raises
|
|
||||||
:class:`tractor.MsgTypeError` immediately in the child and
|
|
||||||
never even hits the wire. (You can opt out per-call with
|
|
||||||
``ctx.started(..., validate_pld_spec=False)`` if you measure
|
|
||||||
a real cost.)
|
|
||||||
- ``Yield`` payloads are **never** checked inside
|
|
||||||
``MsgStream.send()``; they're validated receiver-side on each
|
|
||||||
``MsgStream.receive()``. A violation raises a ``MsgTypeError``
|
|
||||||
in the receiver *and* relays an ``Error`` msg back so the
|
|
||||||
offending sender gets one raised too.
|
|
||||||
- the remaining control msgs (``Start``, ``Return``) are likewise
|
|
||||||
validated such that violations raise in the **sending** actor,
|
|
||||||
pointing the traceback at the code that actually goofed.
|
|
||||||
|
|
||||||
Anatomy of a ``MsgTypeError``
|
|
||||||
-----------------------------
|
|
||||||
:class:`tractor.MsgTypeError` is the IPC equivalent of a builtin
|
|
||||||
``TypeError``: a ``RemoteActorError`` subtype raised whenever
|
|
||||||
a msg fails to decode against the active spec. The useful bits:
|
|
||||||
|
|
||||||
- ``.bad_msg`` -- the offending msg instance (reconstructed from
|
|
||||||
its wire form when necessary) so you can inspect the actual
|
|
||||||
``.pld`` that broke the contract.
|
|
||||||
- ``.expected_msg_type`` -- the protocol msg type the bad msg was
|
|
||||||
(supposed to be) decoded as, e.g. ``Started[Point]``.
|
|
||||||
- plus the standard ``RemoteActorError`` goodies: ``.boxed_type``,
|
|
||||||
``.src_uid``, ``.ipc_msg`` and the fancy ``.pformat()`` tb-box
|
|
||||||
rendering.
|
|
||||||
|
|
||||||
Practical reading guide: a *sender-side* MTE (``Started``,
|
|
||||||
``Return``) points straight at your offending ``await
|
|
||||||
ctx.started()`` or ``return`` statement, while a *receiver-side*
|
|
||||||
MTE (``Yield``) surfaces from the consumer's ``receive()`` call
|
|
||||||
with the relay copy delivered back to the producer. Either way
|
|
||||||
the failure is scoped to that one dialog; sibling contexts on the
|
|
||||||
same channel keep right on trucking.
|
|
||||||
|
|
||||||
Custom wire types: ``mk_codec()`` and friends
|
|
||||||
---------------------------------------------
|
|
||||||
msgspec covers a wide set of `builtin types`__ natively; for
|
|
||||||
anything else you teach the codec via extension hooks. The
|
|
||||||
easiest path is per-endpoint: ``@tractor.context()`` accepts
|
|
||||||
``enc_hook``/``dec_hook`` params right alongside ``pld_spec``.
|
|
||||||
For full control build and apply a codec yourself; encode-side:
|
|
||||||
|
|
||||||
__ https://jcristharif.com/msgspec/supported-types.html
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor.msg import mk_codec, apply_codec
|
|
||||||
|
|
||||||
codec = mk_codec(
|
|
||||||
enc_hook=nsp_to_str, # your-type -> wire-type
|
|
||||||
ext_types=[NamespacePath],
|
|
||||||
)
|
|
||||||
with apply_codec(codec): # ContextVar-scoped override
|
|
||||||
... # msgs sent by this task now encode NSPs
|
|
||||||
|
|
||||||
and decode-side, scoped to an open context (note the import from
|
|
||||||
``tractor.msg._ops``, not yet re-exported):
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
from tractor.msg._ops import limit_plds
|
|
||||||
|
|
||||||
with limit_plds(
|
|
||||||
NamespacePath,
|
|
||||||
dec_hook=str_to_nsp, # wire-type -> your-type
|
|
||||||
ext_types=[NamespacePath],
|
|
||||||
):
|
|
||||||
... # this dialog's payloads decode as NSPs
|
|
||||||
|
|
||||||
``apply_codec()`` is ``ContextVar``-scoped: it overrides the
|
|
||||||
codec for the current task (and only that task), not the whole
|
|
||||||
process. For complete working flows, including hook pairing rules
|
|
||||||
and roundtrip cases, see ``tests/msg/test_ext_types_msgspec.py``
|
|
||||||
and ``tests/msg/test_pldrx_limiting.py``.
|
|
||||||
|
|
||||||
The runtime dogfoods this pattern with
|
|
||||||
:class:`tractor.msg.NamespacePath`: a ``str``-subtype shaped like
|
|
||||||
``'module.path:obj_name'`` used for every RPC target reference.
|
|
||||||
It ships over the wire as a plain string yet ``.load_ref()``-s
|
|
||||||
back to the actual object in the receiving actor's memory domain;
|
|
||||||
a minimal "pointer type" for shared-nothing systems.
|
|
||||||
|
|
||||||
Toward capability-based msging
|
|
||||||
------------------------------
|
|
||||||
The ``pld_spec`` + codec-hook layer is the foundation for the
|
|
||||||
long-game: **capability-based msging** where each dialog's
|
|
||||||
type contract doubles as a capability grant, negotiated as part
|
|
||||||
of the protocol itself. That work is tracked in `#196`_ (with the
|
|
||||||
original typed-proto epic in `#36`_); if strongly-typed
|
|
||||||
distributed systems get you going, we'd love your input.
|
|
||||||
|
|
||||||
Where to next?
|
|
||||||
--------------
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` for the dialog API these msgs
|
|
||||||
implement: ``started()``, streams and results.
|
|
||||||
- :doc:`/guide/asyncio` for shuttling (typed) payloads into
|
|
||||||
``asyncio``-land via an infected subactor.
|
|
||||||
- the msgspec_ docs for everything your payload types can be.
|
|
||||||
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
|
|
||||||
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
|
|
||||||
.. _#196: https://github.com/goodboy/tractor/issues/196
|
|
||||||
.. _#36: https://github.com/goodboy/tractor/issues/36
|
|
||||||
|
|
@ -1,176 +0,0 @@
|
||||||
Parallelism and worker pools
|
|
||||||
============================
|
|
||||||
|
|
||||||
The initial ask is almost always the same: *"how do i make a worker
|
|
||||||
pool?"* — i.e. the thing :mod:`multiprocessing` and
|
|
||||||
:class:`concurrent.futures.ProcessPoolExecutor` get reached for
|
|
||||||
once the GIL becomes the enemy.
|
|
||||||
|
|
||||||
Here's the structured concurrency (SC) answer: ``tractor`` is built
|
|
||||||
to handle any SC process tree you can imagine; a "worker pool"
|
|
||||||
pattern is a trivial special case. So instead of shipping a pool
|
|
||||||
*class* with knobs bolted on, you compose one from the same two
|
|
||||||
ingredients used everywhere else in ``tractor``: an actor nursery
|
|
||||||
and some IPC.
|
|
||||||
|
|
||||||
The stdlib baseline
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
For a fair comparison, start from the canonical
|
|
||||||
:class:`~concurrent.futures.ProcessPoolExecutor` primes example
|
|
||||||
straight out of the Python docs,
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/parallelism/concurrent_futures_primes.py
|
|
||||||
:caption: examples/parallelism/concurrent_futures_primes.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Synchronous code, a hidden thread + IPC machine under the hood, and
|
|
||||||
an API surface (executors, futures, ``.map()``) invented to paper
|
|
||||||
over the fact that the pool isn't part of your program's task tree.
|
|
||||||
Keep an eye on three things for the rewrite: how work is submitted,
|
|
||||||
how results come back, and what happens when a worker dies.
|
|
||||||
|
|
||||||
The ``tractor`` way
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
Now the same workload as a ``tractor`` program,
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/parallelism/concurrent_actors_primes.py
|
|
||||||
:caption: examples/parallelism/concurrent_actors_primes.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's different (and what isn't),
|
|
||||||
|
|
||||||
- ``worker_pool()`` is ~30 lines of *your* code: an actor nursery
|
|
||||||
spawning ``workers`` subactors — each a full process running its
|
|
||||||
own ``trio`` task tree — kept alive and ready for work until the
|
|
||||||
block exits; ``enable_modules=[__name__]`` is the capability
|
|
||||||
allowlist letting them run this module's functions,
|
|
||||||
- jobs are "submitted" by just... calling the function:
|
|
||||||
``portal.run(is_prime, n=value)`` runs ``is_prime()`` in a
|
|
||||||
worker and hands back its result like any local ``await``,
|
|
||||||
- results stream back through a plain
|
|
||||||
:func:`trio.open_memory_channel` *as they complete* — no futures
|
|
||||||
and no polling,
|
|
||||||
- teardown is one ``await tn.cancel()``
|
|
||||||
(:meth:`tractor.ActorNursery.cancel`), and any worker crash
|
|
||||||
triggers the one-cancels-all machinery from
|
|
||||||
:doc:`/guide/cancellation` — a dead worker can never strand the
|
|
||||||
pool.
|
|
||||||
|
|
||||||
This uses no extra threads, fancy semaphores or futures; all we
|
|
||||||
need is ``tractor``'s IPC! The full scorecard,
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:header-rows: 1
|
|
||||||
:widths: 50 50
|
|
||||||
|
|
||||||
* - ``concurrent.futures``
|
|
||||||
- ``tractor``
|
|
||||||
* - ``ProcessPoolExecutor()``
|
|
||||||
- ``worker_pool()`` — yours, ~30 lines
|
|
||||||
* - ``executor.map(is_prime, PRIMES)``
|
|
||||||
- ``actor_map(is_prime, PRIMES)`` async-gen
|
|
||||||
* - ``Future`` + internal result queue
|
|
||||||
- :func:`trio.open_memory_channel`
|
|
||||||
* - results in input order
|
|
||||||
- results as they complete
|
|
||||||
* - worker crash -> ``BrokenProcessPool``
|
|
||||||
- boxed :class:`tractor.RemoteActorError`
|
|
||||||
* - pool teardown on ``with`` exit
|
|
||||||
- one-cancels-all nursery teardown
|
|
||||||
|
|
||||||
.. margin:: How many workers?
|
|
||||||
|
|
||||||
Same calculus as any process pool: about core-count for
|
|
||||||
CPU-bound work (the default sizing in
|
|
||||||
:doc:`/guide/clustering`); more only if workers block on I/O —
|
|
||||||
though at that point you likely want plain ``trio`` tasks, not
|
|
||||||
processes.
|
|
||||||
|
|
||||||
And because the pool is just SC code, every variation — bounded
|
|
||||||
submission, per-worker state, streaming partial results (see
|
|
||||||
:doc:`/guide/streaming`), nested pools — is a local edit to your
|
|
||||||
pool, not a feature request against an executor class B)
|
|
||||||
|
|
||||||
An *async* pool, though?
|
|
||||||
************************
|
|
||||||
|
|
||||||
Yep: RPC targets must be async functions — the runtime rejects a
|
|
||||||
plain ``def`` with ``TypeError: ... must be an async function!``.
|
|
||||||
That's not zealotry, it's cancel-responsiveness: each worker is a
|
|
||||||
full ``trio`` runtime whose msg loop is what hears graceful cancel
|
|
||||||
requests, and a hot loop that never yields can't be (politely)
|
|
||||||
interrupted.
|
|
||||||
|
|
||||||
Two practical consequences,
|
|
||||||
|
|
||||||
- CPU-bound loops should checkpoint once in a while; note how
|
|
||||||
``burn_cpu()`` in the next example sprinkles ``await
|
|
||||||
trio.sleep()`` calls so the worker stays responsive while still
|
|
||||||
pegging a core,
|
|
||||||
- if some sync call blocks a worker anyway you're still covered:
|
|
||||||
an unresponsive actor just rides the graceful-then-hard teardown
|
|
||||||
ladder from :doc:`/guide/cancellation` instead of acking its
|
|
||||||
cancel — slower, but never a zombie.
|
|
||||||
|
|
||||||
Run a func in a process
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
Even a pool can be overkill; "run this one async func in a
|
|
||||||
subprocess and give me the result" is a one-liner via
|
|
||||||
:meth:`tractor.ActorNursery.run_in_actor`,
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/parallelism/single_func.py
|
|
||||||
:caption: examples/parallelism/single_func.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
``run_in_actor()`` is a *convenience wrapper* — spawn an actor, run
|
|
||||||
exactly one task in it, reap on result — not the core spawning
|
|
||||||
model (that's :meth:`tractor.ActorNursery.start_actor` plus
|
|
||||||
:meth:`tractor.Portal.open_context`; see :doc:`/guide/context`).
|
|
||||||
But for this fire-and-collect shape it's exactly the right amount
|
|
||||||
of typing.
|
|
||||||
|
|
||||||
As the module docstring suggests, run it under a process-tree
|
|
||||||
monitor to watch the child appear and get reaped,
|
|
||||||
|
|
||||||
.. code:: bash
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/parallelism/single_func.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
You'll see a core get burned in both parent and child — real
|
|
||||||
parallelism, no GIL sharing, since these are processes (i.e.
|
|
||||||
*non-shared-memory threads*).
|
|
||||||
|
|
||||||
When all you have is sync code
|
|
||||||
------------------------------
|
|
||||||
|
|
||||||
Honesty corner: if your workload is purely *synchronous* functions
|
|
||||||
and you've zero need for IPC dialogs, streaming, daemons or
|
|
||||||
supervision trees — i.e. you really do just want
|
|
||||||
"``ProcessPoolExecutor`` but ``trio``-native" — the smaller,
|
|
||||||
focused `trio-parallel`_ project may serve you better. ``tractor``
|
|
||||||
happily covers the use case (as above) but brings a whole runtime
|
|
||||||
along for the ride. (And when blocking I/O — not the GIL — is the
|
|
||||||
actual problem, plain in-process :func:`trio.to_thread.run_sync`
|
|
||||||
may be all you ever needed.)
|
|
||||||
|
|
||||||
And to *see* that runtime's process-management story — a per-core
|
|
||||||
fleet self-destructing with zero zombies left behind — go run
|
|
||||||
``examples/parallelism/we_are_processes.py``, walked through in
|
|
||||||
the :doc:`/start/quickstart`.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/clustering` — the one-liner flat-cluster
|
|
||||||
convenience (``open_actor_cluster()``) for when even a
|
|
||||||
hand-rolled pool is too much typing,
|
|
||||||
- :doc:`/guide/cancellation` — why pool teardown is bulletproof
|
|
||||||
(graceful-then-hard escalation, no zombies),
|
|
||||||
- :doc:`/guide/context` — the core per-task API your pool
|
|
||||||
workers can graduate to.
|
|
||||||
|
|
||||||
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel
|
|
||||||
|
|
@ -1,177 +0,0 @@
|
||||||
RPC: calling into other actors
|
|
||||||
==============================
|
|
||||||
Every spawn call from :doc:`/guide/spawning` hands you back
|
|
||||||
a :class:`~tractor.Portal`: a live handle for calling into
|
|
||||||
another actor's **memory domain**. The name is borrowed from
|
|
||||||
``trio``'s portal concept — an object you use to submit work
|
|
||||||
*into* a separate concurrency domain — except here that domain
|
|
||||||
is a whole other process.
|
|
||||||
|
|
||||||
.. d2:: diagrams/runtime_stack.d2
|
|
||||||
:margin:
|
|
||||||
:caption: The layers a ``portal.run()`` request rides through.
|
|
||||||
:alt: app, tractor runtime, IPC channel and OS process layers
|
|
||||||
|
|
||||||
There are **no proxy objects** and no special calling
|
|
||||||
conventions: you pass a plain function reference plus keyword
|
|
||||||
args, and Python's normal ``await``-able semantics apply. The
|
|
||||||
function just happens to *run somewhere else*; from the calling
|
|
||||||
task it looks as though it was called locally. And since this is
|
|
||||||
all structured concurrency (SC) under the hood, the remote task
|
|
||||||
runs inside the callee's supervised task tree while its result
|
|
||||||
— or its failure, as a boxed
|
|
||||||
:exc:`~tractor.RemoteActorError` — always comes back to *you*.
|
|
||||||
|
|
||||||
``Portal.run()``: pass the function, not a string
|
|
||||||
-------------------------------------------------
|
|
||||||
:meth:`~tractor.Portal.run` schedules an async function as
|
|
||||||
a **new task** in the remote actor and waits on its result:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'service',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
answer = await portal.run(movie_theatre_question)
|
|
||||||
|
|
||||||
The rules of engagement:
|
|
||||||
|
|
||||||
- the target must be an **async function** and its defining
|
|
||||||
module must be in the callee's ``enable_modules`` allowlist,
|
|
||||||
else an :exc:`~tractor.ModuleNotExposed` error is relayed
|
|
||||||
back (see :doc:`/guide/spawning` for the capability-allowlist
|
|
||||||
story).
|
|
||||||
- arguments are passed **by keyword only**; they ride the IPC
|
|
||||||
layer as msgspec_-encoded msgs, so keep them serializable.
|
|
||||||
- every call schedules a *fresh* task remotely — call it twice
|
|
||||||
and the callee runs two tasks, each supervised in its own
|
|
||||||
right.
|
|
||||||
- remote exceptions re-raise locally as
|
|
||||||
:exc:`~tractor.RemoteActorError` with the original type
|
|
||||||
preserved via ``.boxed_type``.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
Passing dotted-path *strings* to ``run()`` is an ancient,
|
|
||||||
deprecated form; always pass the function reference. If you
|
|
||||||
really need name-based addressing use ``run_from_ns()``
|
|
||||||
below.
|
|
||||||
|
|
||||||
Namespaced daemons: ``run_from_ns()``
|
|
||||||
-------------------------------------
|
|
||||||
Sometimes the calling process can't (or shouldn't) import the
|
|
||||||
target function — think a long-running rpc-daemon serving
|
|
||||||
modules your client never loads. For that,
|
|
||||||
:meth:`~tractor.Portal.run_from_ns` takes the explicit
|
|
||||||
namespace path:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
await portal.run_from_ns('mypkg.service', 'ping')
|
|
||||||
|
|
||||||
This is literally how ``.run()`` works underneath: the pair is
|
|
||||||
encoded as a ``'mod.path:func'`` style msg and resolved against
|
|
||||||
the callee's enabled modules.
|
|
||||||
|
|
||||||
One special namespace exists: ``'self'`` resolves to the remote
|
|
||||||
:class:`~tractor.Actor` instance, i.e. the runtime itself. It's
|
|
||||||
how internal machinery (cancel requests, registry ops) travels;
|
|
||||||
don't build your app on it.
|
|
||||||
|
|
||||||
One-shot results: ``wait_for_result()``
|
|
||||||
---------------------------------------
|
|
||||||
A portal returned from
|
|
||||||
:meth:`~tractor.ActorNursery.run_in_actor` has exactly one
|
|
||||||
"main" task running remotely; that task's ``return`` value is
|
|
||||||
delivered as the portal's *final result*:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
portal = await an.run_in_actor(fib, n=10)
|
|
||||||
final = await portal.wait_for_result()
|
|
||||||
|
|
||||||
Semantics worth knowing:
|
|
||||||
|
|
||||||
- it blocks until the remote task returns, re-raising any
|
|
||||||
remote error in the usual boxed form.
|
|
||||||
- once resolved it's idempotent: later calls return the same
|
|
||||||
cached value.
|
|
||||||
- a *daemon* portal (from ``start_actor()``) has no main task,
|
|
||||||
so there's no final result to wait for: you'll get a warning
|
|
||||||
plus a ``NoResult`` sentinel. Results of individual daemon
|
|
||||||
calls come straight back from each ``await portal.run()``.
|
|
||||||
|
|
||||||
Pure RPC daemons: ``run_daemon()``
|
|
||||||
----------------------------------
|
|
||||||
When a process's *only* job is to sit at the root of its own
|
|
||||||
tree and serve RPC, skip the boilerplate with
|
|
||||||
:func:`tractor.run_daemon`:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
tractor.run_daemon(
|
|
||||||
['mypkg.service'],
|
|
||||||
name='service',
|
|
||||||
)
|
|
||||||
|
|
||||||
It's a blocking convenience (it calls ``trio.run()`` for you):
|
|
||||||
boot a root actor with the given modules enabled for RPC, then
|
|
||||||
sleep until cancelled. Pair it with the discovery system —
|
|
||||||
:func:`tractor.find_actor` / :func:`tractor.wait_for_actor`
|
|
||||||
from a *separate* program — and you've got a tiny service
|
|
||||||
architecture with zero framework ceremony; see
|
|
||||||
``examples/service_daemon_discovery.py`` for the full pattern.
|
|
||||||
|
|
||||||
Fan-out: RPC through nested trees
|
|
||||||
---------------------------------
|
|
||||||
Portals compose. An RPC task is just a ``trio`` task, so it can
|
|
||||||
open its own :class:`~tractor.ActorNursery` and portal into
|
|
||||||
*its* children — one inbound call fanning out into a whole
|
|
||||||
sub-tree of work. The mid-tier function from the nested-tree
|
|
||||||
example:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/nested_actor_tree.py
|
|
||||||
:caption: examples/nested_actor_tree.py (supervisor fan-out)
|
|
||||||
:language: python
|
|
||||||
:pyobject: fan_out_squares
|
|
||||||
|
|
||||||
The root portals into the ``supervisor`` actor; the
|
|
||||||
supervisor's RPC task spawns the leaf workers, portals into
|
|
||||||
each, and returns the combined result back up. Failures at any
|
|
||||||
depth relay hop-by-hop as boxed errors, and cancelling the root
|
|
||||||
call tears down the entire sub-tree — SC, transitively.
|
|
||||||
|
|
||||||
When to graduate to ``Context``
|
|
||||||
-------------------------------
|
|
||||||
``portal.run()`` is great for one-shot, request-response calls.
|
|
||||||
Reach for :meth:`~tractor.Portal.open_context` with an
|
|
||||||
``@tractor.context`` endpoint as soon as you want:
|
|
||||||
|
|
||||||
- a long-lived dialog with state held on both sides,
|
|
||||||
- bidirectional streaming via ``ctx.open_stream()``,
|
|
||||||
- typed payload contracts (``pld_spec``) enforced at the msg
|
|
||||||
layer,
|
|
||||||
- or *task-scoped* cancellation: ``Context.cancel()`` cancels
|
|
||||||
just the linked remote task, whereas
|
|
||||||
:meth:`~tractor.Portal.cancel_actor` nukes the **entire**
|
|
||||||
remote runtime and its process.
|
|
||||||
|
|
||||||
In fact the source plans for ``Portal.run()`` itself to be
|
|
||||||
rebuilt on top of ``open_context()`` — contexts *are* the core
|
|
||||||
inter-actor protocol. Take the full tour in
|
|
||||||
:doc:`/guide/context`.
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/spawning` — where portals come from and how
|
|
||||||
their actors are supervised.
|
|
||||||
- :doc:`/guide/context` — the structured cross-actor task
|
|
||||||
API: handshake, streaming, typed payloads.
|
|
||||||
- :doc:`/guide/cancellation` — what happens to in-flight RPC
|
|
||||||
when trees get torn down.
|
|
||||||
|
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
|
|
@ -1,299 +0,0 @@
|
||||||
Spawning actors
|
|
||||||
===============
|
|
||||||
If you know trio_ you know the drill: you don't get to launch
|
|
||||||
a task off into the void, you open a nursery_, the nursery owns
|
|
||||||
the task, and the block can't exit until every child is done.
|
|
||||||
That discipline is `structured concurrency`_ (SC) — see the
|
|
||||||
seminal `blog post`_ if you haven't yet — and it's the whole
|
|
||||||
religion around here.
|
|
||||||
|
|
||||||
``tractor`` applies that exact discipline to **processes**: an
|
|
||||||
:class:`~tractor.ActorNursery` is a *process nursery*. Every
|
|
||||||
"task" it starts is a fresh Python process running its own
|
|
||||||
``trio.run()``-scheduled task tree; we call each one a
|
|
||||||
``trio``-"*actor*". Parents must wait on (and clean up after)
|
|
||||||
their children, transitively, all the way down the tree.
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:caption: A process tree of ``trio``-task-trees.
|
|
||||||
:alt: a nested actor tree where every parent supervises its children
|
|
||||||
|
|
||||||
Though a "process nursery" differs in complexity (and slightly
|
|
||||||
in semantics) from a single-threaded task nursery, most of the
|
|
||||||
interface is the same. The main difference is that each spawned
|
|
||||||
child contains a full, *parallel-executing* ``trio`` task tree.
|
|
||||||
The following super powers ensue:
|
|
||||||
|
|
||||||
- tasks started in a child actor are completely independent of
|
|
||||||
tasks started in the current process; they execute in
|
|
||||||
**parallel** and are scheduled by their own actor's ``trio``
|
|
||||||
run loop.
|
|
||||||
- tasks scheduled in a remote process still maintain an SC
|
|
||||||
protocol *across memory boundaries* using a so called
|
|
||||||
"SC dialogue protocol" which keeps task-hierarchy lifetimes
|
|
||||||
linked across the IPC layer.
|
|
||||||
- a remote task can fail and have that failure relayed back to
|
|
||||||
the caller task (living in some other actor) as a serialized
|
|
||||||
:exc:`~tractor.RemoteActorError`; no spawned process or RPC
|
|
||||||
task can ever just go off on its own.
|
|
||||||
|
|
||||||
Opening a (process) nursery
|
|
||||||
---------------------------
|
|
||||||
:func:`tractor.open_nursery` is the entrypoint:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async def main():
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
... # spawn some actors B)
|
|
||||||
|
|
||||||
trio.run(main)
|
|
||||||
|
|
||||||
Notice there's no runtime-boot ceremony: if no actor runtime is
|
|
||||||
up yet (i.e. you're in a plain old Python process),
|
|
||||||
``open_nursery()`` *implicitly* enters
|
|
||||||
:func:`tractor.open_root_actor` for you, making this process the
|
|
||||||
**root actor** of a new tree. Any extra keyword args you pass
|
|
||||||
are proxied straight through to ``open_root_actor()``, so the
|
|
||||||
runtime config lives wherever you open your first nursery:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery(
|
|
||||||
loglevel='info',
|
|
||||||
debug_mode=True, # crash-to-REPL for the whole tree
|
|
||||||
) as an:
|
|
||||||
...
|
|
||||||
|
|
||||||
If you want the runtime up *without* spawning anything (or you
|
|
||||||
prefer the config to be loudly explicit) enter
|
|
||||||
``open_root_actor()`` yourself first; the nursery will detect
|
|
||||||
the running runtime and skip the implicit boot. Either way,
|
|
||||||
nesting a second root inside an existing tree is an error.
|
|
||||||
|
|
||||||
Inside a *subactor* the same call just works: any actor may open
|
|
||||||
nurseries of its own, which is how you get arbitrarily deep
|
|
||||||
trees (more on that below).
|
|
||||||
|
|
||||||
``start_actor()``: daemons that live until cancelled
|
|
||||||
----------------------------------------------------
|
|
||||||
:meth:`~tractor.ActorNursery.start_actor` is **the** core
|
|
||||||
spawning primitive. It starts a *daemon* actor: a process with
|
|
||||||
no designated "main task" besides the runtime itself. It boots,
|
|
||||||
registers with its parent, and then sits there serving RPC
|
|
||||||
requests until somebody cancels it. You get back a
|
|
||||||
:class:`~tractor.Portal` for doing exactly that kind of
|
|
||||||
somebody-ing:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py
|
|
||||||
:caption: examples/actor_spawning_and_causality_with_daemon.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on here?
|
|
||||||
|
|
||||||
- ``start_actor('frank', enable_modules=[__name__])`` forks off
|
|
||||||
a new process, boots a ``tractor`` runtime inside it, and
|
|
||||||
allows it to serve functions from the current module (see the
|
|
||||||
allowlist section below).
|
|
||||||
- each ``await portal.run(...)`` schedules a *new* task in
|
|
||||||
frank's task tree and waits on its result — the full RPC story
|
|
||||||
lives in :doc:`/guide/rpc`.
|
|
||||||
- frank has no main task to complete, so without the final
|
|
||||||
``await portal.cancel_actor()`` the nursery block would wait
|
|
||||||
on him **forever**. Daemon lifetimes are *yours* to end; that
|
|
||||||
explicitness is the point.
|
|
||||||
|
|
||||||
``run_in_actor()``: quick one-shot parallelism
|
|
||||||
----------------------------------------------
|
|
||||||
:meth:`~tractor.ActorNursery.run_in_actor` is the convenience
|
|
||||||
wrapper: spawn an actor, run exactly one async function in it,
|
|
||||||
then reap the process as soon as the result arrives.
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.run_in_actor(burn_cpu)
|
|
||||||
# burn rubber in the parent too...
|
|
||||||
await burn_cpu()
|
|
||||||
total = await portal.wait_for_result()
|
|
||||||
|
|
||||||
A few details worth knowing:
|
|
||||||
|
|
||||||
- the actor is named after the function unless you pass
|
|
||||||
``name='something_cuter'``.
|
|
||||||
- the function's module is auto-added to the child's
|
|
||||||
``enable_modules`` allowlist.
|
|
||||||
- extra ``**kwargs`` are forwarded to the function itself.
|
|
||||||
- the child is *auto-cancelled* once its "main" result lands;
|
|
||||||
at nursery exit these run-once children are always reaped
|
|
||||||
first (causality_ is paramount!).
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``run_in_actor()`` is a convenience, **not** the core model.
|
|
||||||
The source literally marks it for an eventual rebuild as
|
|
||||||
a thin "hilevel" wrapper on top of
|
|
||||||
:meth:`~tractor.Portal.open_context` (the modern inter-actor
|
|
||||||
task API). Teach your fingers to use it for quick
|
|
||||||
fire-and-collect parallelism — think a per-function
|
|
||||||
trio-parallel_ style one-shot — and reach for
|
|
||||||
``start_actor()`` + ``open_context()`` for anything
|
|
||||||
long-lived, stateful or streaming
|
|
||||||
(:doc:`/guide/context`).
|
|
||||||
|
|
||||||
Actor lifetimes and teardown order
|
|
||||||
----------------------------------
|
|
||||||
So we have two lifetime flavors:
|
|
||||||
|
|
||||||
- **run-once** (``run_in_actor()``): lives exactly as long as
|
|
||||||
its single task; reaped the moment its result (or error)
|
|
||||||
arrives.
|
|
||||||
- **daemon** (``start_actor()``): lives until *someone* cancels
|
|
||||||
it — an explicit ``await portal.cancel_actor()``, a bulk
|
|
||||||
``await an.cancel()``, or the one-cancels-all strategy kicking
|
|
||||||
in on error.
|
|
||||||
|
|
||||||
On a clean exit of the nursery block the teardown order is:
|
|
||||||
|
|
||||||
1. the nursery waits on every run-once actor's final result;
|
|
||||||
any errors from these are raised immediately so your code
|
|
||||||
(acting as supervisor) gets first crack at handling them.
|
|
||||||
2. then it waits on daemon actors — **indefinitely**. If you
|
|
||||||
spawned a daemon, you own its lifetime.
|
|
||||||
|
|
||||||
When a child *is* cancelled, teardown is graceful-first per SC
|
|
||||||
discipline: the runtime sends an IPC cancel request and gives
|
|
||||||
the child a bounded window to ack; only when a child is too
|
|
||||||
slow does the nursery escalate to an OS-level hard kill of the
|
|
||||||
process. There is no path where a child is silently left
|
|
||||||
running:
|
|
||||||
|
|
||||||
``tractor`` tries to protect you from zombies, no matter
|
|
||||||
what. If you can create zombie child processes (without
|
|
||||||
using a system signal) it **is a bug**.
|
|
||||||
|
|
||||||
Per-process cleanup hooks
|
|
||||||
*************************
|
|
||||||
Need something torn down when an actor's runtime exits, no
|
|
||||||
matter how it exits? Every actor carries
|
|
||||||
a process-global :class:`contextlib.ExitStack` at
|
|
||||||
``Actor.lifetime_stack`` which is closed at the very end of
|
|
||||||
runtime teardown:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
db = await connect_db()
|
|
||||||
tractor.current_actor().lifetime_stack.callback(db.close)
|
|
||||||
|
|
||||||
(A so-far under-advertised api — expect it to get more love.)
|
|
||||||
|
|
||||||
When things blow up: one-cancels-all
|
|
||||||
------------------------------------
|
|
||||||
The default (and currently only) supervision strategy is the
|
|
||||||
same one ``trio`` nurseries use: **one-cancels-all**. If your
|
|
||||||
nursery-block body errors, every child actor is cancelled. If
|
|
||||||
a child errors, the failure is relayed to the nursery as a
|
|
||||||
boxed :exc:`~tractor.RemoteActorError` (original type preserved
|
|
||||||
via ``.boxed_type``), all *other* children are cancelled, and
|
|
||||||
the error(s) re-raise locally — exactly like ``trio``, just
|
|
||||||
process-wide. Erlang-style alternative strategies are a long
|
|
||||||
standing roadmap item.
|
|
||||||
|
|
||||||
The full story — how cancel requests relay across the tree, who
|
|
||||||
``.canceller`` was, debugging mid-teardown — lives in
|
|
||||||
:doc:`/guide/cancellation`.
|
|
||||||
|
|
||||||
The module allowlist: ``enable_modules``
|
|
||||||
----------------------------------------
|
|
||||||
A subactor will only serve functions from modules its parent
|
|
||||||
*explicitly* enabled at spawn time:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'service',
|
|
||||||
enable_modules=['mypkg.service'], # or [__name__]
|
|
||||||
)
|
|
||||||
|
|
||||||
At child boot the runtime imports each listed module so inbound
|
|
||||||
RPC requests can resolve function references against it. Ask
|
|
||||||
a peer to run something from any *other* module and you get an
|
|
||||||
:exc:`~tractor.ModuleNotExposed` error relayed back — the child
|
|
||||||
never even looks the function up.
|
|
||||||
|
|
||||||
Think of it as the first, deliberately coarse layer of
|
|
||||||
capability-style permissioning: if you don't hand an actor
|
|
||||||
a module, no peer can invoke anything inside it. (Finer-grained
|
|
||||||
capability-based messaging protocols are on the roadmap.)
|
|
||||||
|
|
||||||
The ``enable_modules=[__name__]`` idiom — "let the child run
|
|
||||||
functions from the *current* module" — is what you'll use in
|
|
||||||
most scripts; bigger apps tend to pass dedicated service-module
|
|
||||||
paths instead.
|
|
||||||
|
|
||||||
Per-child knobs
|
|
||||||
---------------
|
|
||||||
Both spawn methods accept per-child config so one weird child
|
|
||||||
doesn't have to drag the whole tree along:
|
|
||||||
|
|
||||||
- ``loglevel='cancel'`` — crank console logging for just this
|
|
||||||
subactor (the ``TRACTOR_LOGLEVEL`` env var overrides whatever
|
|
||||||
the *root* was passed, handy for test runs).
|
|
||||||
- ``debug_mode=True`` — arm the crash-handling REPL machinery
|
|
||||||
for just this child instead of tree-wide, i.e. the selective
|
|
||||||
flavor of ``open_nursery(debug_mode=True)``; see
|
|
||||||
:doc:`/guide/debugging` for the multi-process debugger tour.
|
|
||||||
- ``infect_asyncio=True`` — run the child with ``trio`` as an
|
|
||||||
``asyncio`` guest, aka "infected asyncio" mode.
|
|
||||||
- ``enable_transports=['uds']`` — pick the IPC transport this
|
|
||||||
child should listen on (default ``'tcp'``).
|
|
||||||
|
|
||||||
Trees all the way down
|
|
||||||
----------------------
|
|
||||||
Since any actor can open an ``ActorNursery``, supervision trees
|
|
||||||
compose to arbitrary depth: a subactor can be a supervisor of
|
|
||||||
*its own* subactors, with every level holding the same SC
|
|
||||||
guarantees — error relay up, cancellation down, no orphans.
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/nested_actor_tree.py
|
|
||||||
:caption: examples/nested_actor_tree.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Here the root spawns a ``supervisor`` actor whose RPC task opens
|
|
||||||
its *own* nursery and spawns the leaf workers; one call from the
|
|
||||||
root fans out through the middle layer and the aggregate comes
|
|
||||||
back up. Teardown ripples in reverse: the leaves are reaped when
|
|
||||||
the supervisor's nursery exits, the supervisor when the root
|
|
||||||
cancels it.
|
|
||||||
|
|
||||||
Watching your tree grow
|
|
||||||
-----------------------
|
|
||||||
Actors are real processes, so your favorite system tools just
|
|
||||||
work. The house incantation runs any example beside a live
|
|
||||||
process-tree monitor::
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/nested_actor_tree.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
Every subactor also sets its OS process title to a stable
|
|
||||||
``_subactor[<name>@<uuid-prefix>]`` marker, so ``htop``,
|
|
||||||
``ps`` and friends show *which actor is which* at a glance::
|
|
||||||
|
|
||||||
pgrep -af '_subactor\['
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/rpc` — actually invoking functions through
|
|
||||||
all these portals you've been collecting.
|
|
||||||
- :doc:`/guide/context` — the structured, streaming-capable
|
|
||||||
inter-actor task API.
|
|
||||||
- :doc:`/guide/cancellation` — cross-actor cancellation and
|
|
||||||
error propagation semantics in depth.
|
|
||||||
|
|
||||||
.. _trio: https://github.com/python-trio/trio
|
|
||||||
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
|
||||||
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
|
|
||||||
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel
|
|
||||||
|
|
@ -1,244 +0,0 @@
|
||||||
Cross-process streaming
|
|
||||||
=======================
|
|
||||||
|
|
||||||
Spawning processes is the boring half of ``tractor``: the **real
|
|
||||||
cool stuff** is the native support for cross-process *streaming*.
|
|
||||||
Yes, you saw it here first — 2-way msg streams with reliable,
|
|
||||||
transitive setup/teardown semantics, wired straight into the
|
|
||||||
runtime's `structured concurrency`_ (SC) supervision machinery so
|
|
||||||
that *how* a stream ends is part of the protocol.
|
|
||||||
|
|
||||||
No broker, no topic exchange, no IDL compiler. The IPC layer is a
|
|
||||||
deliberately "`cheap or nasty`_" `(un)protocol`_: a tiny set of
|
|
||||||
msgspec_-typed msgs over a transport (TCP or UDS today) with
|
|
||||||
payload typing opt-in per dialog — handshake msgs get the *nasty*
|
|
||||||
treatment (strict validation) while high-rate stream payloads
|
|
||||||
stay *cheap* (receiver-side checks only). See
|
|
||||||
:doc:`/guide/context` for the typed ``pld_spec`` contract bits.
|
|
||||||
|
|
||||||
Two ways to stream
|
|
||||||
------------------
|
|
||||||
|
|
||||||
.. margin:: It's a ``trio.abc.Channel``
|
|
||||||
|
|
||||||
:class:`tractor.MsgStream` implements
|
|
||||||
:class:`trio.abc.Channel` — ``send()``,
|
|
||||||
``receive()``, async-iteration, ``aclose()`` —
|
|
||||||
so trio-generic channel code drives an IPC
|
|
||||||
stream unchanged.
|
|
||||||
|
|
||||||
- **Bidirectional, context-based**: open a
|
|
||||||
:class:`tractor.Context` to a peer task then enter
|
|
||||||
``ctx.open_stream()`` for a full-duplex
|
|
||||||
:class:`tractor.MsgStream`. This is the modern core API, taught
|
|
||||||
end-to-end in :doc:`/guide/context`; we won't re-teach it here.
|
|
||||||
|
|
||||||
- **One-way, portal-based**: point
|
|
||||||
:meth:`tractor.Portal.open_stream_from` at a plain async
|
|
||||||
generator fn in the peer actor. Legacy, but perfectly fine for
|
|
||||||
simple produce/consume pipelines — and it powers the classic
|
|
||||||
examples below.
|
|
||||||
|
|
||||||
Rule of thumb: if the consumer ever needs to *talk back* — acks,
|
|
||||||
control msgs, a final result — use a context. If it's a pure
|
|
||||||
pipeline stage, either works and the one-way form is less typing.
|
|
||||||
|
|
||||||
One-way streaming from an async generator
|
|
||||||
-----------------------------------------
|
|
||||||
|
|
||||||
The OG api. Write an async generator in the target actor's
|
|
||||||
module; iterate its yields from the spawning side:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/asynchronous_generators.py
|
|
||||||
:caption: examples/asynchronous_generators.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Each ``yield`` crosses the process boundary as one msg and feeds
|
|
||||||
the parent's ``async for``. When the consumer ``break``\ s out
|
|
||||||
and exits the ``open_stream_from()`` block the far-end generator
|
|
||||||
task is cancelled for you: the producer's lifetime is *coupled to
|
|
||||||
the consumer's scope* so a one-way stream can never leak a remote
|
|
||||||
task.
|
|
||||||
|
|
||||||
Any extra kwargs (``stream_data, seed=100`` style) are forwarded
|
|
||||||
to the remote generator's call, and a non-async-gen target is
|
|
||||||
rejected up front with a ``TypeError``.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
No decorator required — any plain async-gen fn works. You may
|
|
||||||
still meet ``@tractor.stream`` in the wild; it's the legacy
|
|
||||||
marker for one-way endpoints and sticks around only for
|
|
||||||
compat (heads up: the param name ``ctx`` is reserved for
|
|
||||||
``@context`` endpoints nowadays, so legacy fns should call
|
|
||||||
theirs ``stream``). New code wanting anything fancier than a
|
|
||||||
one-way pipe should use :func:`tractor.context` +
|
|
||||||
``ctx.open_stream()``.
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
One-way means one way: there's no sending *to* the generator
|
|
||||||
side and no graceful consumer-to-producer stop msg — the
|
|
||||||
teardown above is cancel-based. Needing upstream control flow
|
|
||||||
is the sign you've outgrown this API.
|
|
||||||
|
|
||||||
A full-fledged streaming service
|
|
||||||
--------------------------------
|
|
||||||
|
|
||||||
Now let's get fancy: compose one-way streams through a nested
|
|
||||||
actor tree and you've got yourself a fan-in pipeline.
|
|
||||||
|
|
||||||
.. d2:: diagrams/streaming_pipeline.d2
|
|
||||||
:caption: Four actors, three streams, one deduped feed.
|
|
||||||
:alt: two streamer actors fan in to an aggregator then root
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/full_fledged_streaming_service.py
|
|
||||||
:caption: examples/full_fledged_streaming_service.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
What's going on?
|
|
||||||
|
|
||||||
- the root actor spawns ``'aggregator'`` which opens its *own*
|
|
||||||
actor nursery and spawns ``'streamer_1'`` + ``'streamer_2'``: 4
|
|
||||||
processes total, supervision nested two levels deep with zero
|
|
||||||
special casing.
|
|
||||||
|
|
||||||
- ``aggregate()`` opens a one-way stream from each streamer and
|
|
||||||
fans both into a single :func:`trio.open_memory_channel` via
|
|
||||||
one local trio task per portal — in-actor fan-in riding trio's
|
|
||||||
built-in backpressure end-to-end.
|
|
||||||
|
|
||||||
- duplicates get dropped via a ``set`` and the deduped sequence
|
|
||||||
is *re-yielded* upward: ``aggregate()`` is itself an async gen
|
|
||||||
being consumed over IPC by the root. Streams compose.
|
|
||||||
|
|
||||||
- when the seed runs out the streamer gens finish, the memory
|
|
||||||
channel drains closed, the aggregator's gen returns and the
|
|
||||||
root's ``async for`` ends; ``await an.cancel()`` then reaps the
|
|
||||||
subtree. Every exit is awaited — if you can produce a zombie
|
|
||||||
process from this, it **is a bug**.
|
|
||||||
|
|
||||||
Watch the tree breathe while it runs, using the README's
|
|
||||||
signature process-monitor incantation::
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/full_fledged_streaming_service.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
No extra threads, no fancy semaphores, no futures; all we need is
|
|
||||||
``tractor``'s IPC.
|
|
||||||
|
|
||||||
Two streams, one portal
|
|
||||||
-----------------------
|
|
||||||
|
|
||||||
Every ``open_stream_from()`` call starts its *own* remote task —
|
|
||||||
even through the same portal — so two local consumer tasks can
|
|
||||||
independently stream the same generator fn concurrently, both
|
|
||||||
dialogs multiplexed over the single underlying IPC channel:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/multiple_streams_one_portal.py
|
|
||||||
:caption: examples/multiple_streams_one_portal.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
The add-else-remove trick on the shared ``consumed`` list is the
|
|
||||||
proof: each value arrives in *both* streams, getting appended by
|
|
||||||
whichever task sees it first and removed by the other, so the
|
|
||||||
list always ends up empty. Two streams, same data, zero
|
|
||||||
interference.
|
|
||||||
|
|
||||||
This works because every dialog is keyed by its own context id
|
|
||||||
(``Context.cid``) — any number of concurrent streams, contexts
|
|
||||||
and one-shot RPCs share a single underlying
|
|
||||||
:class:`tractor.Channel` per peer pair.
|
|
||||||
|
|
||||||
Fan-out inside an actor: ``MsgStream.subscribe()``
|
|
||||||
--------------------------------------------------
|
|
||||||
|
|
||||||
The inverse pattern: *one* IPC stream feeding *many* local tasks.
|
|
||||||
Instead of paying for N redundant cross-process streams, call
|
|
||||||
:meth:`tractor.MsgStream.subscribe` to get a
|
|
||||||
``BroadcastReceiver`` — a tokio-style broadcast channel from
|
|
||||||
``tractor.trionics`` — which copies every received value to each
|
|
||||||
subscribed task:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/streaming_broadcast_fanout.py
|
|
||||||
:caption: examples/streaming_broadcast_fanout.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Each task entering ``stream.subscribe()`` receives its own copy
|
|
||||||
of everything sent from that point on. The underlying stream
|
|
||||||
keeps pace with the *fastest* subscriber; a task falling more
|
|
||||||
than the buffered window behind has its next receive raise
|
|
||||||
``tractor.trionics.Lagged`` to say it lost data.
|
|
||||||
|
|
||||||
The broadcast handle stays duplex btw: it proxies ``send()``
|
|
||||||
through to the underlying stream, so each subscriber task can
|
|
||||||
keep talking upstream while consuming its fan-out copy.
|
|
||||||
|
|
||||||
.. warning::
|
|
||||||
|
|
||||||
``.subscribe()`` is **idempotent and non-reversible**: the
|
|
||||||
first call permanently swaps the stream's receive machinery
|
|
||||||
over to the internally allocated broadcaster. There's no
|
|
||||||
un-subscribing back to the raw stream, so make sure you're ok
|
|
||||||
with the (theoretical) overhead before opting in.
|
|
||||||
|
|
||||||
Consuming: ``async for`` and friends
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
``async for msg in stream:`` is just sugar over repeated
|
|
||||||
``await stream.receive()``. The receive-side surface:
|
|
||||||
|
|
||||||
- ``receive()`` — next msg, or raises :exc:`trio.EndOfChannel`
|
|
||||||
on a graceful far-end close (``async for`` translates that
|
|
||||||
into a clean loop exit for you).
|
|
||||||
|
|
||||||
- ``receive_nowait()`` — opportunistic, non-blocking drain.
|
|
||||||
|
|
||||||
- ``closed`` — property flagging an already-ended stream.
|
|
||||||
|
|
||||||
Send-side it's just ``await stream.send(data)`` — one ``Yield``
|
|
||||||
msg per call carrying any msgspec_-encodable payload (or
|
|
||||||
whatever your ``pld_spec`` permits, see :doc:`/guide/context`).
|
|
||||||
|
|
||||||
End-of-stream: close vs. cancel
|
|
||||||
-------------------------------
|
|
||||||
|
|
||||||
How a stream ends is part of the protocol; the runtime keeps the
|
|
||||||
polite case and the violent case distinct:
|
|
||||||
|
|
||||||
- **graceful close**: the far side exits its stream block, its
|
|
||||||
async gen returns, or it calls ``await stream.aclose()``. A
|
|
||||||
``Stop`` msg is sent so your ``async for`` simply ends
|
|
||||||
(``StopAsyncIteration``, via :exc:`trio.EndOfChannel` under the
|
|
||||||
hood). A normal, non-error ending — the dialog's result phase
|
|
||||||
proceeds as usual.
|
|
||||||
|
|
||||||
- **cancel or error**: no ``Stop`` is sent. Instead the
|
|
||||||
cancel/error itself is relayed so the far end *knows* the
|
|
||||||
dialog did not end on purpose and raises accordingly — a
|
|
||||||
:exc:`tractor.ContextCancelled`, a boxed
|
|
||||||
:exc:`tractor.RemoteActorError`, etc. See the cancellation
|
|
||||||
section of :doc:`/guide/context` for exactly who raises what.
|
|
||||||
|
|
||||||
Tying it together: every ``MsgStream`` is **one-shot use**. Both
|
|
||||||
endings are final — once closed a stream can't be re-opened and
|
|
||||||
the supported "retry" is opening a fresh :class:`tractor.Context`
|
|
||||||
(they're cheap).
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
- :doc:`/guide/context` — the full ``Context`` lifecycle: the
|
|
||||||
handshake, results, cancellation semantics and the
|
|
||||||
overrun/backpressure knobs.
|
|
||||||
|
|
||||||
- :class:`tractor.MsgStream` and
|
|
||||||
:meth:`tractor.Portal.open_stream_from` API docs.
|
|
||||||
|
|
||||||
- The zguide chapters our wire philosophy is named after:
|
|
||||||
"`cheap or nasty`_" and `(un)protocol`_\ s.
|
|
||||||
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
|
|
||||||
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
|
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
|
|
@ -1,256 +0,0 @@
|
||||||
Testing tips
|
|
||||||
============
|
|
||||||
|
|
||||||
``tractor``'s test suite is a different kind of beast than your
|
|
||||||
average single-proc pytest run: nearly every test spawns a real
|
|
||||||
**process tree**, hammers on cancellation under structured
|
|
||||||
concurrency (SC), and tears the whole thing down again — hundreds
|
|
||||||
of times per session. This page collects the tips, knobs and
|
|
||||||
one-liners that make hacking on (and with) the suite pleasant.
|
|
||||||
|
|
||||||
Running the suite
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
This is a uv_-managed project, so after cloning it's just::
|
|
||||||
|
|
||||||
uv sync --dev
|
|
||||||
uv run pytest tests/
|
|
||||||
|
|
||||||
Expect a *lot* of process churn; the suite is effectively a
|
|
||||||
rolling chaos exercise for the runtime.
|
|
||||||
|
|
||||||
The classic fix-iterate loop when something breaks::
|
|
||||||
|
|
||||||
# stop at the first failure
|
|
||||||
uv run pytest tests/ -x
|
|
||||||
|
|
||||||
# then iterate on just the failures til green
|
|
||||||
uv run pytest --lf -x
|
|
||||||
|
|
||||||
``--lf`` (last-failed) re-runs only what failed previously, so
|
|
||||||
combined with ``-x`` you get a tight one-test-at-a-time repair
|
|
||||||
loop.
|
|
||||||
|
|
||||||
Suite-specific flags
|
|
||||||
********************
|
|
||||||
|
|
||||||
The repo auto-loads the bundled ``tractor._testing.pytest`` plugin
|
|
||||||
(via ``addopts`` in ``pyproject.toml``) which adds a few extra
|
|
||||||
flags:
|
|
||||||
|
|
||||||
- ``--spawn-backend <key>``: pick the process spawn backend for
|
|
||||||
the session (default ``'trio'``); same keys as the
|
|
||||||
``start_method`` runtime argument,
|
|
||||||
|
|
||||||
- ``--tpt-proto <key> [...]``: which IPC transport(s) opting-in
|
|
||||||
suites should run against, eg. ``--tpt-proto uds``,
|
|
||||||
|
|
||||||
- ``--tpdb`` / ``--debug-mode``: flip on the ``debug_mode``
|
|
||||||
fixture so debugger-aware tests boot their trees with the
|
|
||||||
crash-REPL enabled,
|
|
||||||
|
|
||||||
- ``--enable-stackscope``: install the ``SIGUSR1`` task-tree dump
|
|
||||||
handler in pytest *and* every spawned subactor — much lighter
|
|
||||||
than a full debug-mode run when you only need stack visibility
|
|
||||||
during a hang hunt,
|
|
||||||
|
|
||||||
- ``--ll <level>`` / ``--tl <level-or-spec>``: console loglevels;
|
|
||||||
``--tl`` targets the ``tractor``-as-runtime logger and accepts
|
|
||||||
a per-subsystem spec like ``'devx:runtime,trionics:cancel'``.
|
|
||||||
|
|
||||||
Watch the tree grow
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
The single most useful trick while the suite (or any ``tractor``
|
|
||||||
app) runs: keep a live ``pstree`` view going in a side terminal::
|
|
||||||
|
|
||||||
watch -n 0.1 "pstree -a $(pgrep -f pytest)"
|
|
||||||
|
|
||||||
You'll see actor processes pop in and out of existence as each
|
|
||||||
test builds and reaps its tree. Launch it *after* pytest is up
|
|
||||||
(the pid is substituted once, at ``watch`` startup).
|
|
||||||
|
|
||||||
Every subactor also sets its OS process title (via
|
|
||||||
``setproctitle``) to ``_subactor[<name>@<uuid-prefix>]`` so the
|
|
||||||
tree view shows *which actor is which* at a glance — and targeted
|
|
||||||
greps stay easy::
|
|
||||||
|
|
||||||
pgrep -af '_subactor\['
|
|
||||||
|
|
||||||
For a single example script, the repo's signature incantation
|
|
||||||
spawns the watcher alongside your program and cleans it up after::
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/parallelism/single_func.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
Env-var knobs
|
|
||||||
-------------
|
|
||||||
|
|
||||||
Two env-vars override their corresponding runtime arguments
|
|
||||||
*globally* — no application (or test) code changes required:
|
|
||||||
|
|
||||||
``TRACTOR_SPAWN_METHOD``
|
|
||||||
Wins over any caller-passed ``start_method`` so you can drive
|
|
||||||
the whole suite (or any app) under a different spawn backend::
|
|
||||||
|
|
||||||
TRACTOR_SPAWN_METHOD=mp_spawn uv run pytest tests/ -x
|
|
||||||
|
|
||||||
``TRACTOR_LOGLEVEL``
|
|
||||||
Wins over any caller-passed ``loglevel``; crank (or silence)
|
|
||||||
runtime console verbosity wholesale::
|
|
||||||
|
|
||||||
TRACTOR_LOGLEVEL=cancel uv run pytest tests/ -x -s
|
|
||||||
|
|
||||||
``TRACTOR_ENABLE_STACKSCOPE``
|
|
||||||
Force-install the ``SIGUSR1`` task-tree dump handler in every
|
|
||||||
actor, debug-mode or not; then
|
|
||||||
``pkill --signal SIGUSR1 -f <part-of-cmd>`` dumps every
|
|
||||||
actor's live ``trio`` task tree.
|
|
||||||
|
|
||||||
Debug mode vs. pytest capture
|
|
||||||
-----------------------------
|
|
||||||
|
|
||||||
The tree-wide crash-to-REPL experience (``debug_mode=True`` plus
|
|
||||||
``await tractor.pause()``) requires a **real tty**, and pytest's
|
|
||||||
default output capturing swallows exactly that. When you want to
|
|
||||||
interact with the REPL from inside a test run, disable capture::
|
|
||||||
|
|
||||||
uv run pytest tests/test_foo.py -x -s
|
|
||||||
|
|
||||||
(``-s`` is shorthand for ``--capture=no``.)
|
|
||||||
|
|
||||||
Tests should request the ``debug_mode`` fixture (driven by the
|
|
||||||
``--tpdb`` flag) rather than hard-coding it, so that normal CI
|
|
||||||
runs stay non-interactive.
|
|
||||||
|
|
||||||
For *automated* REPL interaction — asserting on prompt output,
|
|
||||||
sending debugger commands — you can't just turn capture off;
|
|
||||||
instead do what ``tests/devx/`` does: drive a child Python program
|
|
||||||
through pexpect_ on a real pseudo-tty and pattern-match the
|
|
||||||
``(Pdb+)`` prompts. See ``tests/devx/test_debugger.py`` for many
|
|
||||||
worked patterns.
|
|
||||||
|
|
||||||
Examples *are* tests
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
Every script under ``examples/`` is run as a subprocess by
|
|
||||||
``tests/test_docs_examples.py``; since these docs
|
|
||||||
``literalinclude`` those same scripts, the code you read here is
|
|
||||||
CI-verified on every push and can never silently rot B)
|
|
||||||
|
|
||||||
Conventions when adding a new example:
|
|
||||||
|
|
||||||
- make it a standalone runnable script with the usual guard::
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
||||||
- it must exit cleanly (returncode ``0``) within the per-example
|
|
||||||
timeout (~16s locally, with headroom auto-added in CI and under
|
|
||||||
cpu-freq scaling) — keep sleeps short,
|
|
||||||
|
|
||||||
- any stderr line containing ``Error`` fails the test, so silence
|
|
||||||
or assert-around expected error output,
|
|
||||||
|
|
||||||
- don't crank ``tractor`` logging inside an example: subprocess
|
|
||||||
pipe **backpressure can deadlock** the run (ask us how we
|
|
||||||
know..),
|
|
||||||
|
|
||||||
- filenames starting with ``_`` are skipped (the WIP convention),
|
|
||||||
as are the special subdirs (``debugging/``, ``integration/``,
|
|
||||||
``advanced_faults/``, ``trio/``) which are driven by their own
|
|
||||||
dedicated suites instead.
|
|
||||||
|
|
||||||
Drop your script in, run the example suite, profit::
|
|
||||||
|
|
||||||
uv run pytest tests/test_docs_examples.py -x
|
|
||||||
|
|
||||||
Zombie cleanup
|
|
||||||
--------------
|
|
||||||
|
|
||||||
First, the contract: ``tractor`` **always** reaps its children —
|
|
||||||
if you can create a zombie process (without resorting to
|
|
||||||
untrappable signals) it **is a bug**, please report it!
|
|
||||||
|
|
||||||
That said, while hacking on the *runtime itself* you can
|
|
||||||
definitely wedge things — a ``SIGKILL``-ed pytest, a half-broken
|
|
||||||
spawn backend — and strand subactor procs plus their shm segments
|
|
||||||
and UDS socket files. The repo ships a dedicated cleanup tool::
|
|
||||||
|
|
||||||
uv run scripts/tractor-reap --shm --uds
|
|
||||||
|
|
||||||
It's SC-polite even as a reaper: matched processes get ``SIGINT``
|
|
||||||
first with a bounded grace window — so actor runtimes can run
|
|
||||||
their ``trio`` teardown paths — escalating to ``SIGKILL`` only as
|
|
||||||
a last resort. The ``--shm`` sweep unlinks ``/dev/shm/`` segments
|
|
||||||
that no live process has open (it leans on psutil_, already in
|
|
||||||
your dev venv, to check live mappings and fds) and ``--uds``
|
|
||||||
clears socket files whose binder pid is dead.
|
|
||||||
|
|
||||||
Testing your own ``tractor`` app
|
|
||||||
--------------------------------
|
|
||||||
|
|
||||||
The same plugin the suite uses ships in the package, so your
|
|
||||||
project can load it too::
|
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
|
||||||
addopts = ['-p tractor._testing.pytest']
|
|
||||||
|
|
||||||
That buys you the CLI flags above plus a set of fixtures —
|
|
||||||
``loglevel``, ``debug_mode``, ``reg_addr`` (a session-unique
|
|
||||||
registrar address so concurrent runs and other live ``tractor``
|
|
||||||
apps on the host can't cross-talk) — and the ``@tractor_test``
|
|
||||||
decorator:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
import tractor
|
|
||||||
from tractor._testing import tractor_test
|
|
||||||
|
|
||||||
@tractor_test
|
|
||||||
async def test_my_service(
|
|
||||||
reg_addr: tuple,
|
|
||||||
loglevel: str,
|
|
||||||
):
|
|
||||||
# already inside a root actor's trio task!
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
...
|
|
||||||
|
|
||||||
The decorator boots a root actor around your (async) test fn,
|
|
||||||
wires any of the special fixtures you declare (``reg_addr``,
|
|
||||||
``loglevel``, ``start_method``, ``debug_mode``) into
|
|
||||||
``open_root_actor()``, and runs the body as the root-most task
|
|
||||||
under a wall-clock ``trio.fail_after()`` guard.
|
|
||||||
|
|
||||||
General advice that has served this suite well:
|
|
||||||
|
|
||||||
- bound waits with ``trio.fail_after()`` *inside* tests; global
|
|
||||||
pytest timeout plugins interact badly with multi-process
|
|
||||||
``trio`` teardown,
|
|
||||||
|
|
||||||
- use the ``reg_addr`` fixture (or otherwise randomize your
|
|
||||||
registry addrs) so leftover registrars from prior runs can't
|
|
||||||
contaminate lookups,
|
|
||||||
|
|
||||||
- assert on **structured outcomes** — eg.
|
|
||||||
``RemoteActorError.boxed_type`` or
|
|
||||||
``ContextCancelled.canceller`` — not on log text.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
``tractor._testing`` is still an underscore-internal namespace:
|
|
||||||
shipped and handy, but its API may shift between alpha
|
|
||||||
releases.
|
|
||||||
|
|
||||||
(This page exists thanks to the ask in `#126`_.)
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
- :doc:`/guide/discovery` — how registrar wiring (the thing
|
|
||||||
``reg_addr`` randomizes) works in the runtime proper.
|
|
||||||
- :doc:`/project/dev-tips` — contributor-oriented extras:
|
|
||||||
releases, log-system tracing, tree-monitoring recipes.
|
|
||||||
|
|
||||||
.. _uv: https://docs.astral.sh/uv/
|
|
||||||
.. _pexpect: https://pexpect.readthedocs.io/en/stable/
|
|
||||||
.. _psutil: https://psutil.readthedocs.io/en/latest/
|
|
||||||
.. _#126: https://github.com/goodboy/tractor/issues/126
|
|
||||||
752
docs/index.rst
|
|
@ -1,154 +1,612 @@
|
||||||
.. image:: _static/tractor_logo_side.svg
|
.. tractor documentation master file, created by
|
||||||
:class: hero-logo
|
sphinx-quickstart on Sun Feb 9 22:26:51 2020.
|
||||||
:align: center
|
You can adapt this file completely to your liking, but it should at least
|
||||||
:alt: tractor
|
contain the root `toctree` directive.
|
||||||
|
|
||||||
tractor
|
``tractor``
|
||||||
=======
|
===========
|
||||||
**distributed structured concurrency**: a
|
|
||||||
multi-processing runtime built on (and shaped
|
|
||||||
entirely like) trio_.
|
|
||||||
|
|
||||||
``tractor`` provides parallelism via ``trio``
|
A `structured concurrent`_, async-native "`actor model`_" built on trio_ and multiprocessing_.
|
||||||
*"actors"*: independent Python **processes** (ie.
|
|
||||||
*non-shared-memory threads*) each running a ``trio``
|
|
||||||
task tree, all composed into a *distributed
|
|
||||||
supervision tree* with end-to-end `structured
|
|
||||||
concurrency`_ (SC) — spawning, cancellation, error
|
|
||||||
propagation and teardown that work **across
|
|
||||||
processes** (and hosts) exactly the way they work
|
|
||||||
across tasks.
|
|
||||||
|
|
||||||
.. margin:: tl;dr
|
|
||||||
|
|
||||||
It's **just** ``trio``, but with nurseries that
|
|
||||||
spawn *processes* and streams that cross them. If
|
|
||||||
you can read a ``trio`` program you can read a
|
|
||||||
``tractor`` one — that's the whole pitch.
|
|
||||||
|
|
||||||
Sixty seconds of why
|
|
||||||
--------------------
|
|
||||||
Spawn one actor per core, crash the root on purpose,
|
|
||||||
and watch the runtime contain the blast: errors
|
|
||||||
propagate, *every* child is reaped, zero zombies —
|
|
||||||
guaranteed (it's a bug otherwise).
|
|
||||||
|
|
||||||
.. literalinclude:: ../examples/parallelism/we_are_processes.py
|
|
||||||
:caption: examples/parallelism/we_are_processes.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Like every snippet in these docs this file lives in
|
|
||||||
the repo's ``examples/`` dir and runs under CI — docs
|
|
||||||
code that can't rot.
|
|
||||||
|
|
||||||
Dig in
|
|
||||||
------
|
|
||||||
.. grid:: 1 2 2 3
|
|
||||||
:gutter: 3
|
|
||||||
|
|
||||||
.. grid-item-card:: Get started
|
|
||||||
:link: start/index
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
Install + your first actor tree in ~20 lines;
|
|
||||||
causality, daemons and the trynamic scene.
|
|
||||||
|
|
||||||
.. grid-item-card:: The big ideas
|
|
||||||
:link: explain/sc-distributed
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
SC across processes, distilled — then the
|
|
||||||
runtime architecture under it.
|
|
||||||
|
|
||||||
.. grid-item-card:: Debug like a local
|
|
||||||
:link: guide/debugging
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
``await tractor.pause()`` anywhere in the
|
|
||||||
tree: one terminal, every process, zero
|
|
||||||
socket-juggling.
|
|
||||||
|
|
||||||
.. grid-item-card:: Streaming + contexts
|
|
||||||
:link: guide/context
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
Bidirectional, cancellation-safe msg streams
|
|
||||||
between any two actors.
|
|
||||||
|
|
||||||
.. grid-item-card:: Guides
|
|
||||||
:link: guide/index
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
RPC, supervision, clustering, "infected
|
|
||||||
asyncio", typed msging + more.
|
|
||||||
|
|
||||||
.. grid-item-card:: API reference
|
|
||||||
:link: api/index
|
|
||||||
:link-type: doc
|
|
||||||
|
|
||||||
The curated public surface; everything
|
|
||||||
importable from ``tractor``.
|
|
||||||
|
|
||||||
Features
|
|
||||||
--------
|
|
||||||
- **It's just a** ``trio`` **API** — same nursery
|
|
||||||
discipline, same cancellation semantics, one level
|
|
||||||
up the process tree.
|
|
||||||
- *Infinitely nestable* process trees: sub-actors can
|
|
||||||
spawn sub-actors, supervision stays transitive.
|
|
||||||
- A "native UX" **multi-process debugger REPL**:
|
|
||||||
built on pdbp_ with tree-wide tty locking (see
|
|
||||||
:doc:`guide/debugging`).
|
|
||||||
- Built-in, cancellation-safe **bidirectional
|
|
||||||
streaming** via a `cheap or nasty`_ `(un)protocol`_.
|
|
||||||
- **Typed IPC**: `msgspec`_-backed wire msgs with
|
|
||||||
optional per-dialog payload specs
|
|
||||||
(:doc:`guide/msging`).
|
|
||||||
- Swappable process-spawn backends + modular IPC
|
|
||||||
transports (TCP today, UDS on same-host, more
|
|
||||||
planned).
|
|
||||||
- Optionally distributed_: the same APIs work over
|
|
||||||
multiple hosts as on multiple cores.
|
|
||||||
- "**Infected** ``asyncio``" mode: SC-supervise
|
|
||||||
``asyncio`` tasks from ``trio``
|
|
||||||
(:doc:`guide/asyncio`).
|
|
||||||
- ``trio`` extension goodies via ``tractor.trionics``
|
|
||||||
(acm gathering, single-resource caching, broadcast
|
|
||||||
channels).
|
|
||||||
|
|
||||||
Where do i start!?
|
|
||||||
------------------
|
|
||||||
The first step to grok ``tractor`` is to get an
|
|
||||||
intermediate knowledge of ``trio`` and **structured
|
|
||||||
concurrency** B)
|
|
||||||
|
|
||||||
Some great places to start are,
|
|
||||||
|
|
||||||
- the seminal `blog post`_,
|
|
||||||
- obviously the `trio docs`_,
|
|
||||||
- wikipedia's nascent SC_ page,
|
|
||||||
- the fancy diagrams @ libdill-docs_,
|
|
||||||
|
|
||||||
then come back and hit :doc:`start/quickstart`.
|
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:hidden:
|
:maxdepth: 1
|
||||||
:maxdepth: 2
|
:caption: Contents:
|
||||||
|
|
||||||
Get started <start/index>
|
|
||||||
Big ideas <explain/index>
|
|
||||||
Guides <guide/index>
|
|
||||||
API <api/index>
|
|
||||||
Project <project/index>
|
|
||||||
|
|
||||||
|
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
|
||||||
.. _trio: https://github.com/python-trio/trio
|
.. _trio: https://github.com/python-trio/trio
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
.. _multiprocessing: https://en.wikipedia.org/wiki/Multiprocessing
|
||||||
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
|
.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles
|
||||||
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich
|
||||||
|
.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228
|
||||||
|
|
||||||
|
|
||||||
|
``tractor`` is an attempt to bring trionic_ `structured concurrency`_ to
|
||||||
|
distributed multi-core Python; it aims to be the Python multi-processing
|
||||||
|
framework *you always wanted*.
|
||||||
|
|
||||||
|
``tractor`` lets you spawn ``trio`` *"actors"*: processes which each run
|
||||||
|
a ``trio`` scheduled task tree (also known as an `async sandwich`_).
|
||||||
|
*Actors* communicate by exchanging asynchronous messages_ and avoid
|
||||||
|
sharing any state. This model allows for highly distributed software
|
||||||
|
architecture which works just as well on multiple cores as it does over
|
||||||
|
many hosts.
|
||||||
|
|
||||||
|
The first step to grok ``tractor`` is to get the basics of ``trio`` down.
|
||||||
|
A great place to start is the `trio docs`_ and this `blog post`_.
|
||||||
|
|
||||||
|
.. _messages: https://en.wikipedia.org/wiki/Message_passing
|
||||||
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
||||||
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
|
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
||||||
.. _pdbp: https://github.com/mdmintz/pdbp
|
.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
||||||
.. _msgspec: https://jcristharif.com/msgspec/
|
|
||||||
.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
|
|
||||||
.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
|
Install
|
||||||
.. _distributed: https://en.wikipedia.org/wiki/Distributed_computing
|
-------
|
||||||
|
No PyPi release yet!
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
pip install git+git://github.com/goodboy/tractor.git
|
||||||
|
|
||||||
|
|
||||||
|
Feel like saying hi?
|
||||||
|
--------------------
|
||||||
|
This project is very much coupled to the ongoing development of
|
||||||
|
``trio`` (i.e. ``tractor`` gets all its ideas from that brilliant
|
||||||
|
community). If you want to help, have suggestions or just want to
|
||||||
|
say hi, please feel free to ping me on the `trio gitter channel`_!
|
||||||
|
|
||||||
|
.. _trio gitter channel: https://gitter.im/python-trio/general
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Philosophy
|
||||||
|
----------
|
||||||
|
Our tenets non-comprehensively include:
|
||||||
|
|
||||||
|
- strict adherence to the `concept-in-progress`_ of *structured concurrency*
|
||||||
|
- no spawning of processes *willy-nilly*; causality_ is paramount!
|
||||||
|
- (remote) errors `always propagate`_ back to the parent supervisor
|
||||||
|
- verbatim support for ``trio``'s cancellation_ system
|
||||||
|
- `shared nothing architecture`_
|
||||||
|
- no use of *proxy* objects or shared references between processes
|
||||||
|
- an immersive debugging experience
|
||||||
|
- anti-fragility through `chaos engineering`_
|
||||||
|
|
||||||
|
``tractor`` is an actor-model-*like* system in the sense that it adheres
|
||||||
|
to the `3 axioms`_ but does not (yet) fulfil all "unrequirements_" in
|
||||||
|
practise. It is an experiment in applying `structured concurrency`_
|
||||||
|
constraints on a parallel processing system where multiple Python
|
||||||
|
processes exist over many hosts but no process can outlive its parent.
|
||||||
|
In `erlang` parlance, it is an architecture where every process has
|
||||||
|
a mandatory supervisor enforced by the type system. The API design is
|
||||||
|
almost exclusively inspired by trio_'s concepts and primitives (though
|
||||||
|
we often lag a little). As a distributed computing system `tractor`
|
||||||
|
attempts to place sophistication at the correct layer such that
|
||||||
|
concurrency primitives are powerful yet simple, making it easy to build
|
||||||
|
complex systems (you can build a "worker pool" architecture but it's
|
||||||
|
definitely not required). There is first class support for inter-actor
|
||||||
|
streaming using `async generators`_ and ongoing work toward a functional
|
||||||
|
reactive style for IPC.
|
||||||
|
|
||||||
|
.. warning:: ``tractor`` is in alpha-alpha and is expected to change rapidly!
|
||||||
|
Expect nothing to be set in stone. Your ideas about where it should go
|
||||||
|
are greatly appreciated!
|
||||||
|
|
||||||
|
.. _concept-in-progress: https://trio.discourse.group/t/structured-concurrency-kickoff/55
|
||||||
|
.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts
|
||||||
|
.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony
|
||||||
|
.. _async generators: https://www.python.org/dev/peps/pep-0525/
|
||||||
|
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
|
||||||
|
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
|
||||||
|
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
|
||||||
|
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
|
||||||
|
.. _channels: https://en.wikipedia.org/wiki/Channel_(programming)
|
||||||
|
.. _chaos engineering: http://principlesofchaos.org/
|
||||||
|
|
||||||
|
|
||||||
|
Examples
|
||||||
|
--------
|
||||||
|
Note, if you are on Windows please be sure to see the :ref:`gotchas
|
||||||
|
<windowsgotchas>` section before trying these.
|
||||||
|
|
||||||
|
|
||||||
|
A trynamic first scene
|
||||||
|
**********************
|
||||||
|
Let's direct a couple *actors* and have them run their lines for
|
||||||
|
the hip new film we're shooting:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/a_trynamic_first_scene.py
|
||||||
|
|
||||||
|
We spawn two *actors*, *donny* and *gretchen*.
|
||||||
|
Each actor starts up and executes their *main task* defined by an
|
||||||
|
async function, ``say_hello()``. The function instructs each actor
|
||||||
|
to find their partner and say hello by calling their partner's
|
||||||
|
``hi()`` function using something called a *portal*. Each actor
|
||||||
|
receives a response and relays that back to the parent actor (in
|
||||||
|
this case our "director" executing ``main()``).
|
||||||
|
|
||||||
|
|
||||||
|
Actor spawning and causality
|
||||||
|
****************************
|
||||||
|
``tractor`` tries to take ``trio``'s concept of causal task lifetimes
|
||||||
|
to multi-process land. Accordingly, ``tractor``'s *actor nursery* behaves
|
||||||
|
similar to ``trio``'s nursery_. That is, ``tractor.open_nursery()``
|
||||||
|
opens an ``ActorNursery`` which **must** wait on spawned *actors* to complete
|
||||||
|
(or error) in the same causal_ way ``trio`` waits on spawned subtasks.
|
||||||
|
This includes errors from any one actor causing all other actors
|
||||||
|
spawned by the same nursery to be cancelled_.
|
||||||
|
|
||||||
|
To spawn an actor and run a function in it, open a *nursery block*
|
||||||
|
and use the ``run_in_actor()`` method:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/actor_spawning_and_causality.py
|
||||||
|
|
||||||
|
What's going on?
|
||||||
|
|
||||||
|
- an initial *actor* is started with ``trio.run()`` and told to execute
|
||||||
|
its main task_: ``main()``
|
||||||
|
|
||||||
|
- inside ``main()`` an actor is *spawned* using an ``ActorNusery`` and is told
|
||||||
|
to run a single function: ``cellar_door()``
|
||||||
|
|
||||||
|
- a ``portal`` instance (we'll get to what it is shortly)
|
||||||
|
returned from ``nursery.run_in_actor()`` is used to communicate with
|
||||||
|
the newly spawned *sub-actor*
|
||||||
|
|
||||||
|
- the second actor, *some_linguist*, in a new *process* running a new ``trio`` task_
|
||||||
|
then executes ``cellar_door()`` and returns its result over a *channel* back
|
||||||
|
to the parent actor
|
||||||
|
|
||||||
|
- the parent actor retrieves the subactor's *final result* using ``portal.result()``
|
||||||
|
much like you'd expect from a future_.
|
||||||
|
|
||||||
|
This ``run_in_actor()`` API should look very familiar to users of
|
||||||
|
``asyncio``'s `run_in_executor()`_ which uses a ``concurrent.futures`` Executor_.
|
||||||
|
|
||||||
|
Since you might also want to spawn long running *worker* or *daemon*
|
||||||
|
actors, each actor's *lifetime* can be determined based on the spawn
|
||||||
|
method:
|
||||||
|
|
||||||
|
- if the actor is spawned using ``run_in_actor()`` it terminates when
|
||||||
|
its *main* task completes (i.e. when the (async) function submitted
|
||||||
|
to it *returns*). The ``with tractor.open_nursery()`` exits only once
|
||||||
|
all actors' main function/task complete (just like the nursery_ in ``trio``)
|
||||||
|
|
||||||
|
- actors can be spawned to *live forever* using the ``start_actor()``
|
||||||
|
method and act like an RPC daemon that runs indefinitely (the
|
||||||
|
``with tractor.open_nursery()`` won't exit) until cancelled_
|
||||||
|
|
||||||
|
Here is a similar example using the latter method:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/actor_spawning_and_causality_with_daemon.py
|
||||||
|
|
||||||
|
The ``enable_modules`` `kwarg` above is a list of module path
|
||||||
|
strings that will be loaded and made accessible for execution in the
|
||||||
|
remote actor through a call to ``Portal.run()``. For now this is
|
||||||
|
a simple mechanism to restrict the functionality of the remote
|
||||||
|
(and possibly daemonized) actor and uses Python's module system to
|
||||||
|
limit the allowed remote function namespace(s).
|
||||||
|
|
||||||
|
``tractor`` is opinionated about the underlying threading model used for
|
||||||
|
each *actor*. Since Python has a GIL and an actor model by definition
|
||||||
|
shares no state between actors, it fits naturally to use a multiprocessing_
|
||||||
|
``Process``. This allows ``tractor`` programs to leverage not only multi-core
|
||||||
|
hardware but also distribute over many hardware hosts (each *actor* can talk
|
||||||
|
to all others with ease over standard network protocols).
|
||||||
|
|
||||||
|
.. _task: https://trio.readthedocs.io/en/latest/reference-core.html#tasks-let-you-do-multiple-things-at-once
|
||||||
|
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
||||||
|
.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality
|
||||||
|
.. _cancelled: https://trio.readthedocs.io/en/latest/reference-core.html#child-tasks-and-cancellation
|
||||||
|
.. _run_in_executor(): https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor
|
||||||
|
.. _Executor: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor
|
||||||
|
|
||||||
|
|
||||||
|
Cancellation
|
||||||
|
************
|
||||||
|
``tractor`` supports ``trio``'s cancellation_ system verbatim.
|
||||||
|
Cancelling a nursery block cancels all actors spawned by it.
|
||||||
|
Eventually ``tractor`` plans to support different `supervision strategies`_ like ``erlang``.
|
||||||
|
|
||||||
|
.. _supervision strategies: http://erlang.org/doc/man/supervisor.html#sup_flags
|
||||||
|
|
||||||
|
|
||||||
|
Remote error propagation
|
||||||
|
************************
|
||||||
|
Any task invoked in a remote actor should ship any error(s) back to the calling
|
||||||
|
actor where it is raised and expected to be dealt with. This way remote actors
|
||||||
|
are never cancelled unless explicitly asked or there's a bug in ``tractor`` itself.
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/remote_error_propagation.py
|
||||||
|
|
||||||
|
|
||||||
|
You'll notice the nursery cancellation conducts a *one-cancels-all*
|
||||||
|
supervisory strategy `exactly like trio`_. The plan is to add more
|
||||||
|
`erlang strategies`_ in the near future by allowing nurseries to accept
|
||||||
|
a ``Supervisor`` type.
|
||||||
|
|
||||||
|
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
||||||
|
.. _erlang strategies: http://learnyousomeerlang.com/supervisors
|
||||||
|
|
||||||
|
|
||||||
|
IPC using *portals*
|
||||||
|
*******************
|
||||||
|
``tractor`` introduces the concept of a *portal* which is an API
|
||||||
|
borrowed_ from ``trio``. A portal may seem similar to the idea of
|
||||||
|
a RPC future_ except a *portal* allows invoking remote *async* functions and
|
||||||
|
generators and intermittently blocking to receive responses. This allows
|
||||||
|
for fully async-native IPC between actors.
|
||||||
|
|
||||||
|
When you invoke another actor's routines using a *portal* it looks as though
|
||||||
|
it was called locally in the current actor. So when you see a call to
|
||||||
|
``await portal.run()`` what you get back is what you'd expect
|
||||||
|
to if you'd called the function directly in-process. This approach avoids
|
||||||
|
the need to add any special RPC *proxy* objects to the library by instead just
|
||||||
|
relying on the built-in (async) function calling semantics and protocols of Python.
|
||||||
|
|
||||||
|
Depending on the function type ``Portal.run()`` tries to
|
||||||
|
correctly interface exactly like a local version of the remote
|
||||||
|
built-in Python *function type*. Currently async functions, generators,
|
||||||
|
and regular functions are supported. Inspiration for this API comes
|
||||||
|
`remote function execution`_ but without the client code being
|
||||||
|
concerned about the underlying channels_ system or shipping code
|
||||||
|
over the network.
|
||||||
|
|
||||||
|
This *portal* approach turns out to be paricularly exciting with the
|
||||||
|
introduction of `asynchronous generators`_ in Python 3.6! It means that
|
||||||
|
actors can compose nicely in a data streaming pipeline.
|
||||||
|
|
||||||
|
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
||||||
|
|
||||||
|
Streaming
|
||||||
|
*********
|
||||||
|
By now you've figured out that ``tractor`` lets you spawn process based
|
||||||
|
*actors* that can invoke cross-process (async) functions and all with
|
||||||
|
structured concurrency built in. But the **real cool stuff** is the
|
||||||
|
native support for cross-process *streaming*.
|
||||||
|
|
||||||
|
|
||||||
|
Asynchronous generators
|
||||||
|
+++++++++++++++++++++++
|
||||||
|
The default streaming function is simply an async generator definition.
|
||||||
|
Every value *yielded* from the generator is delivered to the calling
|
||||||
|
portal exactly like if you had invoked the function in-process meaning
|
||||||
|
you can ``async for`` to receive each value on the calling side.
|
||||||
|
|
||||||
|
As an example here's a parent actor that streams for 1 second from a
|
||||||
|
spawned subactor:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/asynchronous_generators.py
|
||||||
|
|
||||||
|
By default async generator functions are treated as inter-actor
|
||||||
|
*streams* when invoked via a portal (how else could you really interface
|
||||||
|
with them anyway) so no special syntax to denote the streaming *service*
|
||||||
|
is necessary.
|
||||||
|
|
||||||
|
|
||||||
|
Channels and Contexts
|
||||||
|
+++++++++++++++++++++
|
||||||
|
If you aren't fond of having to write an async generator to stream data
|
||||||
|
between actors (or need something more flexible) you can instead use
|
||||||
|
a ``Context``. A context wraps an actor-local spawned task and
|
||||||
|
a ``Channel`` so that tasks executing across multiple processes can
|
||||||
|
stream data to one another using a low level, request oriented API.
|
||||||
|
|
||||||
|
A ``Channel`` wraps an underlying *transport* and *interchange* format
|
||||||
|
to enable *inter-actor-communication*. In its present state ``tractor``
|
||||||
|
uses TCP and msgpack_.
|
||||||
|
|
||||||
|
As an example if you wanted to create a streaming server without writing
|
||||||
|
an async generator that *yields* values you instead define a decorated
|
||||||
|
async function:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
@tractor.stream
|
||||||
|
async def streamer(ctx: tractor.Context, rate: int = 2) -> None:
|
||||||
|
"""A simple web response streaming server.
|
||||||
|
"""
|
||||||
|
while True:
|
||||||
|
val = await web_request('http://data.feed.com')
|
||||||
|
|
||||||
|
# this is the same as ``yield`` in the async gen case
|
||||||
|
await ctx.send_yield(val)
|
||||||
|
|
||||||
|
await trio.sleep(1 / rate)
|
||||||
|
|
||||||
|
|
||||||
|
You must decorate the function with ``@tractor.stream`` and declare
|
||||||
|
a ``ctx`` argument as the first in your function signature and then
|
||||||
|
``tractor`` will treat the async function like an async generator - as
|
||||||
|
a stream from the calling/client side.
|
||||||
|
|
||||||
|
This turns out to be handy particularly if you have multiple tasks
|
||||||
|
pushing responses concurrently:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
async def streamer(
|
||||||
|
ctx: tractor.Context,
|
||||||
|
rate: int = 2
|
||||||
|
) -> None:
|
||||||
|
"""A simple web response streaming server.
|
||||||
|
"""
|
||||||
|
while True:
|
||||||
|
val = await web_request(url)
|
||||||
|
|
||||||
|
# this is the same as ``yield`` in the async gen case
|
||||||
|
await ctx.send_yield(val)
|
||||||
|
|
||||||
|
await trio.sleep(1 / rate)
|
||||||
|
|
||||||
|
|
||||||
|
@tractor.stream
|
||||||
|
async def stream_multiple_sources(
|
||||||
|
ctx: tractor.Context,
|
||||||
|
sources: List[str]
|
||||||
|
) -> None:
|
||||||
|
async with trio.open_nursery() as n:
|
||||||
|
for url in sources:
|
||||||
|
n.start_soon(streamer, ctx, url)
|
||||||
|
|
||||||
|
|
||||||
|
The context notion comes from the context_ in nanomsg_.
|
||||||
|
|
||||||
|
.. _context: https://nanomsg.github.io/nng/man/tip/nng_ctx.5
|
||||||
|
.. _msgpack: https://en.wikipedia.org/wiki/MessagePack
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
A full fledged streaming service
|
||||||
|
++++++++++++++++++++++++++++++++
|
||||||
|
Alright, let's get fancy.
|
||||||
|
|
||||||
|
Say you wanted to spawn two actors which each pull data feeds from
|
||||||
|
two different sources (and wanted this work spread across 2 cpus).
|
||||||
|
You also want to aggregate these feeds, do some processing on them and then
|
||||||
|
deliver the final result stream to a client (or in this case parent) actor
|
||||||
|
and print the results to your screen:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/full_fledged_streaming_service.py
|
||||||
|
|
||||||
|
Here there's four actors running in separate processes (using all the
|
||||||
|
cores on you machine). Two are streaming by *yielding* values from the
|
||||||
|
``stream_data()`` async generator, one is aggregating values from
|
||||||
|
those two in ``aggregate()`` (also an async generator) and shipping the
|
||||||
|
single stream of unique values up the parent actor (the ``'MainProcess'``
|
||||||
|
as ``multiprocessing`` calls it) which is running ``main()``.
|
||||||
|
|
||||||
|
.. _future: https://en.wikipedia.org/wiki/Futures_and_promises
|
||||||
|
.. _borrowed:
|
||||||
|
https://trio.readthedocs.io/en/latest/reference-core.html#getting-back-into-the-trio-thread-from-another-thread
|
||||||
|
.. _asynchronous generators: https://www.python.org/dev/peps/pep-0525/
|
||||||
|
.. _remote function execution: https://codespeak.net/execnet/example/test_info.html#remote-exec-a-function-avoiding-inlined-source-part-i
|
||||||
|
|
||||||
|
|
||||||
|
Actor local (aka *process global*) variables
|
||||||
|
********************************************
|
||||||
|
Although ``tractor`` uses a *shared-nothing* architecture between
|
||||||
|
processes you can of course share state between tasks running *within*
|
||||||
|
an actor (since a `trio.run()` runtime is single threaded). ``trio``
|
||||||
|
tasks spawned via multiple RPC calls to an actor can modify
|
||||||
|
*process-global-state* defined using Python module attributes:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
|
||||||
|
# a per process cache
|
||||||
|
_actor_cache: dict[str, bool] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def ping_endpoints(endpoints: List[str]):
|
||||||
|
"""Start a polling process which runs completely separate
|
||||||
|
from our root actor/process.
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
# This runs in a new process so no changes # will propagate
|
||||||
|
# back to the parent actor
|
||||||
|
while True:
|
||||||
|
|
||||||
|
for ep in endpoints:
|
||||||
|
status = await check_endpoint_is_up(ep)
|
||||||
|
_actor_cache[ep] = status
|
||||||
|
|
||||||
|
await trio.sleep(0.5)
|
||||||
|
|
||||||
|
|
||||||
|
async def get_alive_endpoints():
|
||||||
|
|
||||||
|
nonlocal _actor_cache
|
||||||
|
|
||||||
|
return {key for key, value in _actor_cache.items() if value}
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
|
||||||
|
async with tractor.open_nursery() as n:
|
||||||
|
|
||||||
|
portal = await n.run_in_actor(ping_endpoints)
|
||||||
|
|
||||||
|
# print the alive endpoints after 3 seconds
|
||||||
|
await trio.sleep(3)
|
||||||
|
|
||||||
|
# this is submitted to be run in our "ping_endpoints" actor
|
||||||
|
print(await portal.run(get_alive_endpoints))
|
||||||
|
|
||||||
|
|
||||||
|
You can pass any kind of (`msgpack`) serializable data between actors using
|
||||||
|
function call semantics but building out a state sharing system per-actor
|
||||||
|
is totally up to you.
|
||||||
|
|
||||||
|
|
||||||
|
Service Discovery
|
||||||
|
*****************
|
||||||
|
Though it will be built out much more in the near future, ``tractor``
|
||||||
|
currently keeps track of actors by ``(name: str, id: str)`` using a
|
||||||
|
special actor called the *arbiter*. Currently the *arbiter* must exist
|
||||||
|
on a host (or it will be created if one can't be found) and keeps a
|
||||||
|
simple ``dict`` of actor names to sockets for discovery by other actors.
|
||||||
|
Obviously this can be made more sophisticated (help me with it!) but for
|
||||||
|
now it does the trick.
|
||||||
|
|
||||||
|
To find the arbiter from the current actor use the ``get_arbiter()`` function and to
|
||||||
|
find an actor's socket address by name use the ``find_actor()`` function:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/service_discovery.py
|
||||||
|
|
||||||
|
The ``name`` value you should pass to ``find_actor()`` is the one you passed as the
|
||||||
|
*first* argument to either ``trio.run()`` or ``ActorNursery.start_actor()``.
|
||||||
|
|
||||||
|
|
||||||
|
Running actors standalone
|
||||||
|
*************************
|
||||||
|
You don't have to spawn any actors using ``open_nursery()`` if you just
|
||||||
|
want to run a single actor that connects to an existing cluster.
|
||||||
|
All the comms and arbiter registration stuff still works. This can
|
||||||
|
somtimes turn out being handy when debugging mult-process apps when you
|
||||||
|
need to hop into a debugger. You just need to pass the existing
|
||||||
|
*arbiter*'s socket address you'd like to connect to:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
import trio
|
||||||
|
import tractor
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
|
||||||
|
async with tractor.open_root_actor(
|
||||||
|
arbiter_addr=('192.168.0.10', 1616)
|
||||||
|
):
|
||||||
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
Choosing a process spawning backend
|
||||||
|
***********************************
|
||||||
|
``tractor`` is architected to support multiple actor (sub-process)
|
||||||
|
spawning backends. Specific defaults are chosen based on your system
|
||||||
|
but you can also explicitly select a backend of choice at startup
|
||||||
|
via a ``start_method`` kwarg to ``tractor.open_nursery()``.
|
||||||
|
|
||||||
|
Currently the options available are:
|
||||||
|
|
||||||
|
- ``trio``: a ``trio``-native spawner which is an async wrapper around ``subprocess``
|
||||||
|
- ``spawn``: one of the stdlib's ``multiprocessing`` `start methods`_
|
||||||
|
- ``forkserver``: a faster ``multiprocessing`` variant that is Unix only
|
||||||
|
|
||||||
|
.. _start methods: https://docs.python.org/3.8/library/multiprocessing.html#contexts-and-start-methods
|
||||||
|
|
||||||
|
|
||||||
|
``trio``
|
||||||
|
++++++++
|
||||||
|
The ``trio`` backend offers a lightweight async wrapper around ``subprocess`` from the standard library and takes advantage of the ``trio.`` `open_process`_ API.
|
||||||
|
|
||||||
|
.. _open_process: https://trio.readthedocs.io/en/stable/reference-io.html#spawning-subprocesses
|
||||||
|
|
||||||
|
|
||||||
|
``multiprocessing``
|
||||||
|
+++++++++++++++++++
|
||||||
|
There is support for the stdlib's ``multiprocessing`` `start methods`_.
|
||||||
|
Note that on Windows *spawn* it the only supported method and on \*nix
|
||||||
|
systems *forkserver* is the best method for speed but has the caveat
|
||||||
|
that it will break easily (hangs due to broken pipes) if spawning actors
|
||||||
|
using nested nurseries.
|
||||||
|
|
||||||
|
In general, the ``multiprocessing`` backend **has not proven reliable**
|
||||||
|
for handling errors from actors more then 2 nurseries *deep* (see `#89`_).
|
||||||
|
If you for some reason need this consider sticking with alternative
|
||||||
|
backends.
|
||||||
|
|
||||||
|
.. _#89: https://github.com/goodboy/tractor/issues/89
|
||||||
|
|
||||||
|
.. _windowsgotchas:
|
||||||
|
|
||||||
|
Windows "gotchas"
|
||||||
|
^^^^^^^^^^^^^^^^^
|
||||||
|
On Windows (which requires the use of the stdlib's `multiprocessing`
|
||||||
|
package) there are some gotchas. Namely, the need for calling
|
||||||
|
`freeze_support()`_ inside the ``__main__`` context. Additionally you
|
||||||
|
may need place you `tractor` program entry point in a seperate
|
||||||
|
`__main__.py` module in your package in order to avoid an error like the
|
||||||
|
following ::
|
||||||
|
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "C:\ProgramData\Miniconda3\envs\tractor19030601\lib\site-packages\tractor\_actor.py", line 234, in _get_rpc_func
|
||||||
|
return getattr(self._mods[ns], funcname)
|
||||||
|
KeyError: '__mp_main__'
|
||||||
|
|
||||||
|
|
||||||
|
To avoid this, the following is the **only code** that should be in your
|
||||||
|
main python module of the program:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
# application/__main__.py
|
||||||
|
import trio
|
||||||
|
import tractor
|
||||||
|
import multiprocessing
|
||||||
|
from . import tractor_app
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
multiprocessing.freeze_support()
|
||||||
|
trio.run(tractor_app.main)
|
||||||
|
|
||||||
|
And execute as::
|
||||||
|
|
||||||
|
python -m application
|
||||||
|
|
||||||
|
|
||||||
|
As an example we use the following code to test all documented examples
|
||||||
|
in the test suite on windows:
|
||||||
|
|
||||||
|
.. literalinclude:: ../examples/__main__.py
|
||||||
|
|
||||||
|
See `#61`_ and `#79`_ for further details.
|
||||||
|
|
||||||
|
.. _freeze_support(): https://docs.python.org/3/library/multiprocessing.html#multiprocessing.freeze_support
|
||||||
|
.. _#61: https://github.com/goodboy/tractor/pull/61#issuecomment-470053512
|
||||||
|
.. _#79: https://github.com/goodboy/tractor/pull/79
|
||||||
|
|
||||||
|
|
||||||
|
Enabling logging
|
||||||
|
****************
|
||||||
|
Considering how complicated distributed software can become it helps to know
|
||||||
|
what exactly it's doing (even at the lowest levels). Luckily ``tractor`` has
|
||||||
|
tons of logging throughout the core. ``tractor`` isn't opinionated on
|
||||||
|
how you use this information and users are expected to consume log messages in
|
||||||
|
whichever way is appropriate for the system at hand. That being said, when hacking
|
||||||
|
on ``tractor`` there is a prettified console formatter which you can enable to
|
||||||
|
see what the heck is going on. Just put the following somewhere in your code:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
from tractor.log import get_console_log
|
||||||
|
log = get_console_log('trace')
|
||||||
|
|
||||||
|
|
||||||
|
What the future holds
|
||||||
|
---------------------
|
||||||
|
Stuff I'd like to see ``tractor`` do real soon:
|
||||||
|
|
||||||
|
- TLS_, duh.
|
||||||
|
- erlang-like supervisors_
|
||||||
|
- native support for `nanomsg`_ as a channel transport
|
||||||
|
- native `gossip protocol`_ support for service discovery and arbiter election
|
||||||
|
- a distributed log ledger for tracking cluster behaviour
|
||||||
|
- a slick multi-process aware debugger much like in celery_
|
||||||
|
but with better `pdb++`_ support
|
||||||
|
- an extensive `chaos engineering`_ test suite
|
||||||
|
- support for reactive programming primitives and native support for asyncitertools_ like libs
|
||||||
|
- introduction of a `capability-based security`_ model
|
||||||
|
|
||||||
|
.. _TLS: https://trio.readthedocs.io/en/latest/reference-io.html#ssl-tls-support
|
||||||
|
.. _supervisors: https://github.com/goodboy/tractor/issues/22
|
||||||
|
.. _nanomsg: https://nanomsg.github.io/nng/index.html
|
||||||
|
.. _gossip protocol: https://en.wikipedia.org/wiki/Gossip_protocol
|
||||||
|
.. _celery: http://docs.celeryproject.org/en/latest/userguide/debugging.html
|
||||||
|
.. _asyncitertools: https://github.com/vodik/asyncitertools
|
||||||
|
.. _pdb++: https://github.com/antocuni/pdb
|
||||||
|
.. _capability-based security: https://en.wikipedia.org/wiki/Capability-based_security
|
||||||
|
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
.. include:: ../../NEWS.rst
|
|
||||||
|
|
@ -1,133 +0,0 @@
|
||||||
Hot tips for ``tractor`` hackers
|
|
||||||
================================
|
|
||||||
|
|
||||||
This is a (perpetually WIP) guide for newcomers to the project,
|
|
||||||
mostly to do with dev, testing, CI and release gotchas, reminders
|
|
||||||
and best practises.
|
|
||||||
|
|
||||||
``tractor`` is a fairly novel project compared to most since it is
|
|
||||||
effectively a new way of doing distributed computing in Python and
|
|
||||||
is much closer to working with an "application level runtime"
|
|
||||||
(like erlang OTP or scala's akka project) than it is a traditional
|
|
||||||
Python library. As such, having an arsenal of tools and recipes
|
|
||||||
for figuring out the right way to debug problems when they do
|
|
||||||
arise is somewhat of a necessity.
|
|
||||||
|
|
||||||
Making a release
|
|
||||||
----------------
|
|
||||||
|
|
||||||
Nothing fancy: the traditional PyPA flow on the hatchling_ build
|
|
||||||
backend, with uv_ doing the driving and towncrier_ generating the
|
|
||||||
changelog.
|
|
||||||
|
|
||||||
1. collect news fragments: user-facing changes should land with a
|
|
||||||
small ``.rst`` snippet under ``nooz/`` (see ``nooz/HOWTO.rst``;
|
|
||||||
fragment types are ``feature``, ``bugfix``, ``doc`` and
|
|
||||||
``trivial``),
|
|
||||||
|
|
||||||
2. render them into ``NEWS.rst``::
|
|
||||||
|
|
||||||
uvx towncrier build --version <version>
|
|
||||||
|
|
||||||
3. build and upload (testpypi first if you're being careful)::
|
|
||||||
|
|
||||||
uv build
|
|
||||||
uvx twine upload -r testpypi dist/*
|
|
||||||
uvx twine upload dist/*
|
|
||||||
|
|
||||||
How you organize built artifacts under ``dist/`` locally (per
|
|
||||||
release sub-dirs and such) is entirely up to you.
|
|
||||||
|
|
||||||
Keep in mind that PyPi releases tend to lag the ``main`` branch
|
|
||||||
since we develop in the open — ``main`` is usually the thing to
|
|
||||||
run when you want the latest.
|
|
||||||
|
|
||||||
Debugging and monitoring actor trees
|
|
||||||
------------------------------------
|
|
||||||
|
|
||||||
Your "what is my tree doing right now?" toolbox, in escalation
|
|
||||||
order:
|
|
||||||
|
|
||||||
**Live process-tree view** — keep a ``watch``-ed ``pstree``
|
|
||||||
running in a side terminal; actor procs are recognizable by their
|
|
||||||
``_subactor[<name>@<uuid-prefix>]`` process titles. The exact
|
|
||||||
one-liners (plus the ``pgrep`` marker recipes) live in
|
|
||||||
:doc:`/guide/testing`.
|
|
||||||
|
|
||||||
**SIGUSR1 task-tree dumps** — boot any tree with
|
|
||||||
``enable_stack_on_sig=True`` (or export
|
|
||||||
``TRACTOR_ENABLE_STACKSCOPE=1``) and every actor installs a
|
|
||||||
stackscope_ signal handler. Then from any shell::
|
|
||||||
|
|
||||||
# dump every actor's live trio task tree:
|
|
||||||
pkill --signal SIGUSR1 -f <part-of-your-cmd>
|
|
||||||
|
|
||||||
# or for a single process:
|
|
||||||
kill -SIGUSR1 $(pgrep -f <part-of-your-cmd>)
|
|
||||||
|
|
||||||
Each dump is also tee'd (append-mode) to
|
|
||||||
``/tmp/tractor-stackscope-<pid>.log`` so you still get output
|
|
||||||
under pytest capture or in CI. This works *without* debug-mode
|
|
||||||
being enabled — it's the lightest-weight hang-investigation tool
|
|
||||||
in the box.
|
|
||||||
|
|
||||||
**The built-in multi-process debugger** — ``debug_mode=True``
|
|
||||||
plus :func:`tractor.pause` and friends: the heavyweight champ for
|
|
||||||
interactive, REPL-driven inspection of a whole tree (including
|
|
||||||
crash handling). Remember pytest capture interplay — see
|
|
||||||
:doc:`/guide/testing`.
|
|
||||||
|
|
||||||
**Post-mortem zombie sweeps** — ``scripts/tractor-reap`` for the
|
|
||||||
(should-be-rare!) cases where hacking on the runtime itself wedges
|
|
||||||
a tree: a SIGINT-first, structured concurrency (SC) polite
|
|
||||||
escalation, plus ``--shm`` and ``--uds`` leaked-resource sweeps.
|
|
||||||
|
|
||||||
Using the log system to trace ``trio`` task flow
|
|
||||||
------------------------------------------------
|
|
||||||
|
|
||||||
The logging system is oriented around the **stack "layers" of the
|
|
||||||
runtime**, letting you trace logical abstraction layers in the
|
|
||||||
code — errors, cancellation, IPC and streaming, the low level
|
|
||||||
transport and wire protocols — independently of one another.
|
|
||||||
|
|
||||||
Concretely, ``tractor.log.get_logger()`` returns a
|
|
||||||
``StackLevelAdapter`` sporting extra level-methods beyond the
|
|
||||||
stdlib set, including:
|
|
||||||
|
|
||||||
- ``.cancel()`` — cancellation-machinery flow,
|
|
||||||
|
|
||||||
- ``.runtime()`` — actor-runtime lifecycle chatter,
|
|
||||||
|
|
||||||
- ``.devx()`` — debugger/devx tooling internals,
|
|
||||||
|
|
||||||
- ``.transport()`` — wire-level msging events.
|
|
||||||
|
|
||||||
To get console output at any level from your own code::
|
|
||||||
|
|
||||||
from tractor.log import get_console_log
|
|
||||||
get_console_log('cancel')
|
|
||||||
|
|
||||||
or, runtime-wide without touching code, just export
|
|
||||||
``TRACTOR_LOGLEVEL=cancel`` (the env-var wins over caller-passed
|
|
||||||
levels; great for test runs).
|
|
||||||
|
|
||||||
When you want only *one subsystem* cranked, the suite's ``--tl``
|
|
||||||
flag (and ``tractor.log.apply_logspec()``) accept a per-sublogger
|
|
||||||
spec::
|
|
||||||
|
|
||||||
uv run pytest tests/... --tl 'devx:runtime,trionics:cancel'
|
|
||||||
|
|
||||||
Every record's header includes the emitting actor and task names,
|
|
||||||
so cross-process flows can be stitched back together by eyeball
|
|
||||||
(or grep).
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
- :doc:`/guide/testing` — running the suite, watching trees
|
|
||||||
live, examples-as-tests conventions and the zombie-reaper.
|
|
||||||
- :doc:`/guide/discovery` — the registrar mechanics you'll
|
|
||||||
bump into when running multiple trees on one host.
|
|
||||||
|
|
||||||
.. _hatchling: https://hatch.pypa.io/latest/
|
|
||||||
.. _uv: https://docs.astral.sh/uv/
|
|
||||||
.. _towncrier: https://towncrier.readthedocs.io/en/stable/
|
|
||||||
.. _stackscope: https://github.com/oremanj/stackscope
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
Project
|
|
||||||
=======
|
|
||||||
Everything meta: where the project's been (the
|
|
||||||
:doc:`changelog`), where it's going (the roadmap below),
|
|
||||||
how to hack on it (:doc:`dev-tips` and
|
|
||||||
:doc:`/guide/testing`) and where to find the humans.
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:maxdepth: 1
|
|
||||||
|
|
||||||
changelog
|
|
||||||
dev-tips
|
|
||||||
|
|
||||||
What the future holds
|
|
||||||
---------------------
|
|
||||||
Help us push toward the future of distributed Python!
|
|
||||||
Planned (or dreamed of) work non-comprehensively
|
|
||||||
includes,
|
|
||||||
|
|
||||||
- Erlang-style supervisors via composed context
|
|
||||||
managers (see `#22`_),
|
|
||||||
- typed capability-based (dialog) protocols, ie.
|
|
||||||
evolving our `msg-spec`_ system into per-dialog
|
|
||||||
contracts (see `#196`_ with draft work in `#311`_),
|
|
||||||
- a higher level "service manager" API for daemon
|
|
||||||
lifetime mgmt over actor trees (in the works on an
|
|
||||||
experimental branch as ``tractor.hilevel``),
|
|
||||||
- richer `discovery`_ via gossip and/or
|
|
||||||
`rendezvous protocol`_ approaches (today's registrar
|
|
||||||
is intentionally naive),
|
|
||||||
- more IPC transports: the current ``tcp`` | ``uds``
|
|
||||||
pair wants friends (QUIC, shm-ring-buffers, RUDP,
|
|
||||||
wireguard tunnels),
|
|
||||||
- an extensive `chaos engineering`_ test suite,
|
|
||||||
- a respawn-from-REPL system for crashed (sub-)actors.
|
|
||||||
|
|
||||||
Feel like saying hi?
|
|
||||||
--------------------
|
|
||||||
This project is very much coupled to the ongoing
|
|
||||||
development of ``trio`` (i.e. ``tractor`` gets most of
|
|
||||||
its ideas from that brilliant community). If you want
|
|
||||||
to help, have suggestions or just want to say hi,
|
|
||||||
please feel free to reach us in our `matrix channel`_.
|
|
||||||
If matrix seems too hip, we're also mostly all in the
|
|
||||||
`trio gitter channel`_!
|
|
||||||
|
|
||||||
Contributions of all kinds welcome: docs, examples,
|
|
||||||
bug reports, new transports, supervisor strategies,
|
|
||||||
philosophical debates about what an "actor model"
|
|
||||||
really is B)
|
|
||||||
|
|
||||||
.. _#22: https://github.com/goodboy/tractor/issues/22
|
|
||||||
.. _#196: https://github.com/goodboy/tractor/issues/196
|
|
||||||
.. _#311: https://github.com/goodboy/tractor/pull/311
|
|
||||||
.. _msg-spec: https://jcristharif.com/msgspec/
|
|
||||||
.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
|
|
||||||
.. _rendezvous protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
|
|
||||||
.. _chaos engineering: https://principlesofchaos.org/
|
|
||||||
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
|
|
||||||
.. _trio gitter channel: https://gitter.im/python-trio/general
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
Getting started
|
|
||||||
===============
|
|
||||||
Welcome aboard B)
|
|
||||||
|
|
||||||
Real talk before any code: the first step to grok ``tractor`` is
|
|
||||||
to get an intermediate knowledge of ``trio`` and **structured
|
|
||||||
concurrency** (SC). ``tractor`` **is just** ``trio`` - but with
|
|
||||||
nurseries for process management and cancel-able streaming IPC -
|
|
||||||
so every rule you already know about task lifetimes, cancellation
|
|
||||||
and error propagation keeps holding, just now across process (and
|
|
||||||
host!) boundaries. Some great places to start are,
|
|
||||||
|
|
||||||
- the seminal `blog post`_,
|
|
||||||
- obviously the `trio docs`_,
|
|
||||||
- wikipedia's nascent SC_ page,
|
|
||||||
- the fancy diagrams @ libdill-docs_.
|
|
||||||
|
|
||||||
Once you've taken in (some of) the canon, get installed and go
|
|
||||||
spawn your first actor tree:
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:maxdepth: 1
|
|
||||||
|
|
||||||
install
|
|
||||||
quickstart
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
Already installed and itching? Jump straight to
|
|
||||||
:doc:`/start/quickstart`; once you're through the on-ramp the
|
|
||||||
:doc:`guide pages </guide/index>` take each subsystem deeper.
|
|
||||||
|
|
||||||
.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
|
|
||||||
.. _trio docs: https://trio.readthedocs.io/en/latest/
|
|
||||||
.. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
|
|
||||||
|
|
@ -1,75 +0,0 @@
|
||||||
Install
|
|
||||||
=======
|
|
||||||
``tractor`` is still in an *alpha-near-beta-stage* for many of
|
|
||||||
its subsystems, however we are very close to having a stable
|
|
||||||
lowlevel runtime and API. Expect the occasional rough edge (and
|
|
||||||
feel free to report it!).
|
|
||||||
|
|
||||||
Supported platforms
|
|
||||||
-------------------
|
|
||||||
- **python**: ``>=3.13,<3.15`` - yes, we ride near the front of
|
|
||||||
the release train; 3.14 additionally unlocks the experimental
|
|
||||||
PEP-734 sub-interpreter spawn backend,
|
|
||||||
- **linux**: the primary development and CI platform,
|
|
||||||
- **macos**: officially supported,
|
|
||||||
- **windows**: currently *untested* - we disabled CI-testing on
|
|
||||||
windows a while back and haven't had the cycles to revive it.
|
|
||||||
It mostly worked historically; if you're a windows person we'd
|
|
||||||
love a hand getting it back in the test matrix.
|
|
||||||
|
|
||||||
With ``uv`` (preferred)
|
|
||||||
-----------------------
|
|
||||||
We use the very hip uv_ for project mgmt and recommend you do
|
|
||||||
too. Add ``tractor`` to your project::
|
|
||||||
|
|
||||||
uv add tractor
|
|
||||||
|
|
||||||
or, since the git ``main`` branch is often much further ahead
|
|
||||||
than any latest release (see the PyPi note below), track ``main``
|
|
||||||
directly::
|
|
||||||
|
|
||||||
uv add "tractor @ git+https://github.com/goodboy/tractor.git"
|
|
||||||
|
|
||||||
From PyPi
|
|
||||||
---------
|
|
||||||
We ofc also offer "releases" on PyPi_::
|
|
||||||
|
|
||||||
pip install tractor
|
|
||||||
|
|
||||||
Just note that **YMMV** since ``main`` is usually well ahead of
|
|
||||||
the latest published alpha; when in doubt go with a git install
|
|
||||||
as per above (or hack from source as per below).
|
|
||||||
|
|
||||||
From source
|
|
||||||
-----------
|
|
||||||
To run the bundled examples in-tree, or to start hacking on the
|
|
||||||
code base, clone and sync a dev env::
|
|
||||||
|
|
||||||
git clone https://github.com/goodboy/tractor.git
|
|
||||||
cd tractor
|
|
||||||
uv sync --dev
|
|
||||||
uv run python examples/rpc_bidir_streaming.py
|
|
||||||
|
|
||||||
Consider activating a virtual/project-env before starting to hack
|
|
||||||
on the code base::
|
|
||||||
|
|
||||||
# you could use plain ol' venvs
|
|
||||||
# https://docs.astral.sh/uv/pip/environments/
|
|
||||||
uv venv tractor_py313 --python 3.13
|
|
||||||
|
|
||||||
# but @goodboy prefers the more explicit (and shell agnostic)
|
|
||||||
# https://docs.astral.sh/uv/configuration/environment/#uv_project_environment
|
|
||||||
UV_PROJECT_ENVIRONMENT="tractor_py313"
|
|
||||||
|
|
||||||
# hint hint, enter @goodboy's fave shell B)
|
|
||||||
uv run --dev xonsh
|
|
||||||
|
|
||||||
.. seealso::
|
|
||||||
|
|
||||||
All set? Go boot a process tree in :doc:`/start/quickstart`.
|
|
||||||
And if the install fights you, swing by the `matrix channel`_
|
|
||||||
and we'll sort it out.
|
|
||||||
|
|
||||||
.. _uv: https://docs.astral.sh/uv/
|
|
||||||
.. _PyPi: https://pypi.org/project/tractor/
|
|
||||||
.. _matrix channel: https://matrix.to/#/!tractor:matrix.org
|
|
||||||
|
|
@ -1,264 +0,0 @@
|
||||||
Quickstart
|
|
||||||
==========
|
|
||||||
Time to spawn something B)
|
|
||||||
|
|
||||||
If you take one thing from this page make it this: ``tractor``
|
|
||||||
**is just** ``trio`` - but with nurseries for process management
|
|
||||||
and cancel-able streaming IPC. Every "*actor*" you'll meet below
|
|
||||||
is a plain Python **process** running its own ``trio.run()``
|
|
||||||
scheduled task tree, linked back to its parent through an IPC
|
|
||||||
protocol which keeps the whole tree `structured concurrency`_
|
|
||||||
(SC) compliant end-to-end. If you know your nursery_ semantics
|
|
||||||
you already know most of ``tractor``; we just stretch them across
|
|
||||||
the process boundary.
|
|
||||||
|
|
||||||
.. d2:: diagrams/actor_tree.d2
|
|
||||||
:alt: a supervision tree of actor processes
|
|
||||||
:margin:
|
|
||||||
:caption: every arrow is a parent which **must wait** on its kids
|
|
||||||
|
|
||||||
Your first actor tree
|
|
||||||
---------------------
|
|
||||||
``trio`` takes the hard-line position that a parent task **must
|
|
||||||
wait** on the children it spawns; causality_ is paramount! So
|
|
||||||
does ``tractor``, one abstraction layer up:
|
|
||||||
``tractor.open_nursery()`` yields an ``ActorNursery`` which
|
|
||||||
**must** wait on its spawned *subactors* to complete (or error)
|
|
||||||
before the ``async with`` block exits, in the same causal_ way a
|
|
||||||
``trio`` nursery waits on its subtasks. That includes any one
|
|
||||||
child's crash cancelling all of its siblings: *one-cancels-all*
|
|
||||||
supervision, `exactly like trio`_.
|
|
||||||
|
|
||||||
Enough preamble, spawn a process:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/actor_spawning_and_causality.py
|
|
||||||
:caption: examples/actor_spawning_and_causality.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Run it::
|
|
||||||
|
|
||||||
$ python examples/actor_spawning_and_causality.py
|
|
||||||
Dang that's beautiful
|
|
||||||
|
|
||||||
What's going on here?
|
|
||||||
|
|
||||||
- ``trio.run(main)`` starts the **root actor**; the ``tractor``
|
|
||||||
runtime boots *implicitly* inside ``tractor.open_nursery()``
|
|
||||||
whenever it isn't already up. No special entrypoint, no
|
|
||||||
framework takeover - it's just a ``trio`` app,
|
|
||||||
- inside ``main()`` a *subactor* is spawned via
|
|
||||||
``ActorNursery.run_in_actor()`` and told to run exactly one
|
|
||||||
function: ``cellar_door()``,
|
|
||||||
- you get back a ``Portal``: your handle for invoking tasks in
|
|
||||||
the new process's (separate!) memory domain. We lean on it
|
|
||||||
much harder in the next section,
|
|
||||||
- the subactor, *some_linguist*, boots a fresh ``trio.run()`` in
|
|
||||||
a **new process** and executes ``cellar_door()`` as its *main
|
|
||||||
task* (note the child proving it is *not* the root with
|
|
||||||
``tractor.is_root_process()``), then ships the return value
|
|
||||||
back over IPC,
|
|
||||||
- the parent grabs that *final result* with
|
|
||||||
``await portal.wait_for_result()``, much like you'd expect
|
|
||||||
from a "future" - except causality is preserved: the nursery
|
|
||||||
block only exits once the child is *done*, dead, and reaped.
|
|
||||||
|
|
||||||
.. margin:: Just need a worker pool?
|
|
||||||
|
|
||||||
If all you want is to throw *sync* functions at your cores,
|
|
||||||
also check out trio-parallel_. ``tractor`` is aimed at
|
|
||||||
structured, (possibly) distributed *trees* of cooperating
|
|
||||||
``trio`` programs; a worker pool is a trivial special case.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``run_in_actor()`` is the *convenience* wrapper: one-shot
|
|
||||||
spawn-run-reap semantics for when a subactor's entire job is
|
|
||||||
a single function call. The core primitives are
|
|
||||||
``ActorNursery.start_actor()`` (next up) paired with
|
|
||||||
``Portal.open_context()`` for full, SC-linked cross-actor
|
|
||||||
dialogs - see :doc:`/guide/context`.
|
|
||||||
|
|
||||||
Daemon actors and RPC
|
|
||||||
---------------------
|
|
||||||
A ``run_in_actor()``-spawned actor terminates when its main task
|
|
||||||
returns. But often you want long-lived *daemon* actors instead:
|
|
||||||
spawned once, then serving (allowlisted) RPC requests until told
|
|
||||||
otherwise. That's ``start_actor()``:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/actor_spawning_and_causality_with_daemon.py
|
|
||||||
:caption: examples/actor_spawning_and_causality_with_daemon.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
Two lifetime rules to internalize:
|
|
||||||
|
|
||||||
- a ``run_in_actor()`` actor lives exactly as long as its main
|
|
||||||
task; the nursery waits for that function (and thus the
|
|
||||||
process) to complete before unblocking,
|
|
||||||
- a ``start_actor()`` actor *lives forever* - an RPC daemon the
|
|
||||||
nursery will happily wait on **indefinitely** - until some
|
|
||||||
task explicitly cancels it via ``Portal.cancel_actor()`` (as
|
|
||||||
above), or its parent nursery is cancelled wholesale.
|
|
||||||
|
|
||||||
.. tip::
|
|
||||||
|
|
||||||
Want your *entire program* to just be a long-lived RPC
|
|
||||||
daemon? ``tractor.run_daemon()`` is the blocking shorthand:
|
|
||||||
it ``trio.run()``\s a root actor which serves requests until
|
|
||||||
cancelled.
|
|
||||||
|
|
||||||
The ``enable_modules=[__name__]`` kwarg is the other thing to
|
|
||||||
notice: it lists the module paths the subactor will load and
|
|
||||||
*expose* for remote invocation.
|
|
||||||
``await portal.run(movie_theatre_question)`` works because this
|
|
||||||
very module is in that allowlist (and note we call it twice; the
|
|
||||||
daemon happily serves repeat requests). Ask for a function from
|
|
||||||
any module *not* enabled and you're denied with a
|
|
||||||
``ModuleNotExposed`` error: a simple, capability-style
|
|
||||||
restriction mechanism built on Python's own module system.
|
|
||||||
|
|
||||||
We are *processes*
|
|
||||||
------------------
|
|
||||||
Why processes (and not, say, threads)? Python has a GIL and an
|
|
||||||
`actor model`_ by definition shares **nothing** between its
|
|
||||||
concurrent units, so real OS processes are the natural fit: you
|
|
||||||
get all your cores locally, and since actors only ever talk via
|
|
||||||
IPC, the exact same code distributes over multiple hosts without
|
|
||||||
modification.
|
|
||||||
|
|
||||||
Of course, the moment you hear "process trees" you should be
|
|
||||||
asking: *what about zombies?* Watch ``tractor`` eat one for
|
|
||||||
breakfast - run this while monitoring your process tree::
|
|
||||||
|
|
||||||
$TERM -e watch -n 0.1 "pstree -a $$" \
|
|
||||||
& python examples/parallelism/we_are_processes.py \
|
|
||||||
&& kill $!
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/parallelism/we_are_processes.py
|
|
||||||
:caption: examples/parallelism/we_are_processes.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
.. margin:: Who's who in ``pstree``?
|
|
||||||
|
|
||||||
Every subactor (best-effort, via the optional
|
|
||||||
``setproctitle`` dep) re-titles its OS process like
|
|
||||||
``_subactor[worker_0@<uuid>]``, so ``pstree``/``htop``/
|
|
||||||
``pgrep -f`` can tell your actors apart at a glance.
|
|
||||||
|
|
||||||
You'll see something like::
|
|
||||||
|
|
||||||
$ python examples/parallelism/we_are_processes.py
|
|
||||||
Yo, i'm 'worker_2' running in pid 1777246
|
|
||||||
Yo, i'm 'worker_0' running in pid 1777244
|
|
||||||
Yo, i'm 'worker_3' running in pid 1777247
|
|
||||||
Yo, i'm 'worker_1' running in pid 1777245
|
|
||||||
This process tree will self-destruct in 1 sec...
|
|
||||||
Zombies Contained
|
|
||||||
|
|
||||||
(The worker lines land in whatever order the OS schedules them;
|
|
||||||
they're separate *processes*, racing, and that's the point.)
|
|
||||||
|
|
||||||
An actor is spawned per core, each parks itself in
|
|
||||||
``trio.sleep_forever()``... and then the root *crashes on
|
|
||||||
purpose*. The ``ActorNursery`` responds with hard ``trio``
|
|
||||||
discipline: every child is cancelled, every process is reaped,
|
|
||||||
the error propagates to ``trio.run()``, and your terminal prints
|
|
||||||
``Zombies Contained``. No orphans, no ``kill -9`` archaeology in
|
|
||||||
``htop`` afterwards.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
**The zombie-safety guarantee**: ``tractor`` tries to protect
|
|
||||||
you from zombies, *no matter what*. If you can create zombie
|
|
||||||
child processes (without using a system signal) it **is a
|
|
||||||
bug** - please report it so we can hunt it down.
|
|
||||||
|
|
||||||
A trynamic first scene
|
|
||||||
----------------------
|
|
||||||
So far the root actor has done all the talking, but subactors
|
|
||||||
can just as well discover and call *each other*. Let's direct a
|
|
||||||
couple actors and have them run their lines for the hip new film
|
|
||||||
we're shooting:
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/a_trynamic_first_scene.py
|
|
||||||
:caption: examples/a_trynamic_first_scene.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
The script of the scene (runtime ``INFO`` log lines trimmed)::
|
|
||||||
|
|
||||||
$ python examples/a_trynamic_first_scene.py
|
|
||||||
Alright... Action!
|
|
||||||
Hi my name is gretchen
|
|
||||||
Hi my name is donny
|
|
||||||
CUTTTT CUUTT CUT!!! Donny!! You're supposed to say...
|
|
||||||
|
|
||||||
The new tricks in play:
|
|
||||||
|
|
||||||
- two subactors, *donny* and *gretchen*, are each told to run
|
|
||||||
``say_hello()`` targeting the *other* by name,
|
|
||||||
- ``tractor.wait_for_actor()`` blocks until the named peer has
|
|
||||||
registered with the tree's *registrar* (every actor announces
|
|
||||||
itself at boot), then yields a ``Portal`` connected
|
|
||||||
**directly** to that peer,
|
|
||||||
- each actor invokes its partner's ``hi()`` over that portal:
|
|
||||||
actor-to-actor RPC with the root merely *directing* - and both
|
|
||||||
final lines flow back to ``main()`` via
|
|
||||||
``await portal.wait_for_result()``,
|
|
||||||
- ``tractor.log.get_console_log("INFO")`` cranks up runtime
|
|
||||||
logging so you can watch the spawn/register/cancel machinery
|
|
||||||
narrate itself; remove it for a quiet set.
|
|
||||||
|
|
||||||
Cross-actor calls look just like (async) function calls; there
|
|
||||||
are no proxy objects and no shared references, only messages B)
|
|
||||||
|
|
||||||
Crash handling, native feeling
|
|
||||||
------------------------------
|
|
||||||
One last teaser before the guide proper. Flip exactly one
|
|
||||||
switch:
|
|
||||||
|
|
||||||
.. code:: python
|
|
||||||
|
|
||||||
async with tractor.open_nursery(
|
|
||||||
debug_mode=True,
|
|
||||||
) as an:
|
|
||||||
...
|
|
||||||
|
|
||||||
and any crash, in *any* actor at *any* depth of the tree, drops
|
|
||||||
your terminal into a multi-process-safe pdbp_ REPL at the
|
|
||||||
offending frame, with the rest of the tree held back from
|
|
||||||
clobbering the tty. ``await tractor.pause()`` likewise gives you
|
|
||||||
a breakpoint that *just works* inside subprocesses. We think it
|
|
||||||
might be the first native multi-process debugging UX for Python;
|
|
||||||
get the full tour in :doc:`/guide/debugging`.
|
|
||||||
|
|
||||||
Where to next?
|
|
||||||
--------------
|
|
||||||
You can now boot a runtime, spawn one-shot and daemon actors,
|
|
||||||
make cross-process RPC calls, and contain zombies: that's the
|
|
||||||
on-ramp done. The guide takes each subsystem deeper,
|
|
||||||
|
|
||||||
- :doc:`/explain/sc-distributed` - the structured concurrency
|
|
||||||
worldview and how ``tractor`` extends it across processes,
|
|
||||||
- :doc:`/guide/spawning` - everything ``ActorNursery``: spawn
|
|
||||||
kwargs, lifetimes and supervision semantics,
|
|
||||||
- :doc:`/guide/rpc` - the ``Portal`` in depth: calling into
|
|
||||||
another actor's memory domain,
|
|
||||||
- :doc:`/guide/context` - the core API: ``@tractor.context``
|
|
||||||
endpoints, the ``ctx.started()`` handshake, and SC-linked
|
|
||||||
cross-actor task pairs,
|
|
||||||
- :doc:`/guide/streaming` - bidirectional ``MsgStream`` dialogs
|
|
||||||
and fan-out broadcasting,
|
|
||||||
- :doc:`/guide/debugging` - the multi-process REPL, crash
|
|
||||||
handling mode, and ``tractor.pause()``,
|
|
||||||
- :doc:`/guide/asyncio` - "infected ``asyncio``" mode: SC
|
|
||||||
supervision wrapped around ``asyncio`` tasks,
|
|
||||||
- :doc:`/guide/discovery` - registries, service daemons, and
|
|
||||||
finding actors from anywhere in (or out of) the tree.
|
|
||||||
|
|
||||||
.. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
|
|
||||||
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
|
|
||||||
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
|
|
||||||
.. _causal: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#causality
|
|
||||||
.. _exactly like trio: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-semantics
|
|
||||||
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
|
|
||||||
.. _trio-parallel: https://github.com/richardsheridan/trio-parallel
|
|
||||||
.. _pdbp: https://github.com/mdmintz/pdbp
|
|
||||||
|
|
@ -35,8 +35,8 @@ async def main():
|
||||||
name='gretchen',
|
name='gretchen',
|
||||||
other_actor='donny',
|
other_actor='donny',
|
||||||
)
|
)
|
||||||
print(await gretchen.wait_for_result())
|
print(await gretchen.result())
|
||||||
print(await donny.wait_for_result())
|
print(await donny.result())
|
||||||
print("CUTTTT CUUTT CUT!!! Donny!! You're supposed to say...")
|
print("CUTTTT CUUTT CUT!!! Donny!! You're supposed to say...")
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,7 @@ async def main():
|
||||||
# The ``async with`` will unblock here since the 'some_linguist'
|
# The ``async with`` will unblock here since the 'some_linguist'
|
||||||
# actor has completed its main task ``cellar_door``.
|
# actor has completed its main task ``cellar_door``.
|
||||||
|
|
||||||
print(await portal.wait_for_result())
|
print(await portal.result())
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|
|
||||||
|
|
@ -1,105 +0,0 @@
|
||||||
'''
|
|
||||||
Demonstrate a (3-level) nested actor tree where one RPC from
|
|
||||||
the root fans out through a mid-tier 'supervisor' actor to
|
|
||||||
2 'leaf' worker actors and an aggregate result is relayed
|
|
||||||
back up.
|
|
||||||
|
|
||||||
The process tree should look approximately like:
|
|
||||||
|
|
||||||
python examples/nested_actor_tree.py
|
|
||||||
`-python -m tractor._child --uid ('supervisor', '7c9b1039 ..)
|
|
||||||
|-python -m tractor._child --uid ('leaf_1', '92d62f50 ..)
|
|
||||||
`-python -m tractor._child --uid ('leaf_2', 'de91fdf5 ..)
|
|
||||||
|
|
||||||
Teardown runs inside-out: the supervisor cancels its leaves
|
|
||||||
first, then the root cancels the supervisor; watch the
|
|
||||||
prints to see the ordering.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
|
|
||||||
async def compute_square(x: int) -> int:
|
|
||||||
'''
|
|
||||||
Tiny "work unit" run inside a leaf actor.
|
|
||||||
|
|
||||||
'''
|
|
||||||
name: str = tractor.current_actor().name
|
|
||||||
print(f'{name}: squaring {x}')
|
|
||||||
return x * x
|
|
||||||
|
|
||||||
|
|
||||||
@tractor.context
|
|
||||||
async def fan_out_squares(
|
|
||||||
ctx: tractor.Context,
|
|
||||||
vals: list[int],
|
|
||||||
) -> list[int]:
|
|
||||||
'''
|
|
||||||
Spawn a (nested) pair of leaf actors, fan the input vals
|
|
||||||
out across them round-robin style, then return the
|
|
||||||
aggregated squares to our parent.
|
|
||||||
|
|
||||||
'''
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portals: list[tractor.Portal] = []
|
|
||||||
for i in (1, 2):
|
|
||||||
portals.append(
|
|
||||||
await an.start_actor(
|
|
||||||
f'leaf_{i}',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
)
|
|
||||||
# unblock the parent's `.open_context()` entry and
|
|
||||||
# report which leaves came up.
|
|
||||||
await ctx.started(
|
|
||||||
[p.chan.aid.name for p in portals]
|
|
||||||
)
|
|
||||||
squares: dict[int, int] = {}
|
|
||||||
|
|
||||||
async def run_in_leaf(
|
|
||||||
portal: tractor.Portal,
|
|
||||||
x: int,
|
|
||||||
) -> None:
|
|
||||||
squares[x] = await portal.run(
|
|
||||||
compute_square,
|
|
||||||
x=x,
|
|
||||||
)
|
|
||||||
|
|
||||||
# fan out one sub-RPC per input val, concurrently.
|
|
||||||
async with trio.open_nursery() as tn:
|
|
||||||
for i, x in enumerate(vals):
|
|
||||||
tn.start_soon(
|
|
||||||
run_in_leaf,
|
|
||||||
portals[i % len(portals)],
|
|
||||||
x,
|
|
||||||
)
|
|
||||||
# graceful inside-out teardown: leaves go first!
|
|
||||||
for portal in portals:
|
|
||||||
leaf_name: str = portal.chan.aid.name
|
|
||||||
print(f'supervisor: cancelling {leaf_name}')
|
|
||||||
await portal.cancel_actor()
|
|
||||||
return [squares[x] for x in vals]
|
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'supervisor',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
async with portal.open_context(
|
|
||||||
fan_out_squares,
|
|
||||||
vals=[1, 2, 3, 4],
|
|
||||||
) as (ctx, leaf_names):
|
|
||||||
print(f'root: supervisor spawned {leaf_names}')
|
|
||||||
squares: list[int] = await ctx.wait_for_result()
|
|
||||||
assert squares == [1, 4, 9, 16]
|
|
||||||
print(f'root: aggregate result {squares}')
|
|
||||||
print('root: cancelling supervisor')
|
|
||||||
await portal.cancel_actor()
|
|
||||||
print('root: tree torn down, what zombies?')
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
@ -1,19 +1,7 @@
|
||||||
'''
|
|
||||||
The pure-stdlib `concurrent.futures.ProcessPoolExecutor`
|
|
||||||
primes demo (from the std docs) verbatim; the baseline twin
|
|
||||||
of `concurrent_actors_primes.py`.
|
|
||||||
|
|
||||||
The `async def main()` + `trio.run()` shim at the bottom only
|
|
||||||
exists so the docs-example test runner can exercise this
|
|
||||||
script; the executor code itself is untouched stdlib fare.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import time
|
import time
|
||||||
import concurrent.futures
|
import concurrent.futures
|
||||||
import math
|
import math
|
||||||
|
|
||||||
import trio
|
|
||||||
|
|
||||||
PRIMES = [
|
PRIMES = [
|
||||||
112272535095293,
|
112272535095293,
|
||||||
112582705942171,
|
112582705942171,
|
||||||
|
|
@ -38,7 +26,7 @@ def is_prime(n):
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
def check_primes():
|
def main():
|
||||||
with concurrent.futures.ProcessPoolExecutor() as executor:
|
with concurrent.futures.ProcessPoolExecutor() as executor:
|
||||||
start = time.time()
|
start = time.time()
|
||||||
|
|
||||||
|
|
@ -48,14 +36,8 @@ def check_primes():
|
||||||
print(f'processing took {time.time() - start} seconds')
|
print(f'processing took {time.time() - start} seconds')
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
# thin shim: the pool blocks this (sole) trio task
|
|
||||||
# which is just fine for a one-shot baseline script.
|
|
||||||
check_primes()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|
||||||
start = time.time()
|
start = time.time()
|
||||||
trio.run(main)
|
main()
|
||||||
print(f'script took {time.time() - start} seconds')
|
print(f'script took {time.time() - start} seconds')
|
||||||
|
|
@ -33,7 +33,7 @@ async def main():
|
||||||
await burn_cpu()
|
await burn_cpu()
|
||||||
|
|
||||||
# wait on result from target function
|
# wait on result from target function
|
||||||
pid = await portal.wait_for_result()
|
pid = await portal.result()
|
||||||
|
|
||||||
# end of nursery block
|
# end of nursery block
|
||||||
print(f"Collected subproc {pid}")
|
print(f"Collected subproc {pid}")
|
||||||
|
|
|
||||||
|
|
@ -1,68 +0,0 @@
|
||||||
'''
|
|
||||||
Demonstrate the "service daemon" pattern: a named,
|
|
||||||
long-lived actor spawned via `ActorNursery.start_actor()`
|
|
||||||
which any other task can locate through the registrar using
|
|
||||||
`tractor.find_actor()` / `tractor.wait_for_actor()` - no
|
|
||||||
spawn-portal required - and RPC into directly.
|
|
||||||
|
|
||||||
Teardown is explicit and graceful via `portal.cancel_actor()`
|
|
||||||
once the clients are done.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
_quotes: dict[str, float] = {
|
|
||||||
'btcusdt': 66_000.5,
|
|
||||||
'ethusdt': 3_500.25,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
async def get_quote(sym: str) -> float:
|
|
||||||
'''
|
|
||||||
Look up the "current" quote for a symbol.
|
|
||||||
|
|
||||||
'''
|
|
||||||
name: str = tractor.current_actor().name
|
|
||||||
print(f'{name}: serving quote for {sym!r}')
|
|
||||||
return _quotes[sym]
|
|
||||||
|
|
||||||
|
|
||||||
async def client_task() -> None:
|
|
||||||
'''
|
|
||||||
Locate the quote service by name and RPC it; note no
|
|
||||||
spawn-nursery/portal reference is ever passed in here!
|
|
||||||
|
|
||||||
'''
|
|
||||||
# a lookup miss yields `None` (not an error).
|
|
||||||
async with tractor.find_actor('no_such_svc') as portal:
|
|
||||||
assert portal is None
|
|
||||||
print('client: "no_such_svc" is not registered')
|
|
||||||
# block until the service shows up in the registry,
|
|
||||||
# then call into it through the delivered portal.
|
|
||||||
async with tractor.wait_for_actor('quote_svc') as portal:
|
|
||||||
quote: float = await portal.run(
|
|
||||||
get_quote,
|
|
||||||
sym='btcusdt',
|
|
||||||
)
|
|
||||||
print(f'client: got btcusdt quote {quote}')
|
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'quote_svc',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
# run the client in a separate task which discovers
|
|
||||||
# the daemon purely by its registered name.
|
|
||||||
async with trio.open_nursery() as tn:
|
|
||||||
tn.start_soon(client_task)
|
|
||||||
# explicit graceful teardown of the daemon.
|
|
||||||
print('root: cancelling quote_svc')
|
|
||||||
await portal.cancel_actor()
|
|
||||||
print('root: service shut down cleanly')
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
@ -1,84 +0,0 @@
|
||||||
'''
|
|
||||||
Demonstrate fanning out ONE inter-actor `MsgStream` to N
|
|
||||||
local (parent-side) trio tasks using `MsgStream.subscribe()`:
|
|
||||||
each subscriber gets its own `BroadcastReceiver` copy of
|
|
||||||
every msg from the single underlying IPC stream.
|
|
||||||
|
|
||||||
The child waits for a 'go' msg so that all subscribers are
|
|
||||||
guaranteed-attached before the first tick is sent; when the
|
|
||||||
child's stream closes each subscriber's `async for` ends
|
|
||||||
cleanly.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
|
|
||||||
@tractor.context
|
|
||||||
async def tick_stream(
|
|
||||||
ctx: tractor.Context,
|
|
||||||
count: int,
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Send `count` "ticks" once the parent says go.
|
|
||||||
|
|
||||||
'''
|
|
||||||
await ctx.started(count)
|
|
||||||
async with ctx.open_stream() as stream:
|
|
||||||
# wait for the go-signal ensuring every parent-side
|
|
||||||
# subscriber is attached before any tick is sent.
|
|
||||||
assert await stream.receive() == 'go'
|
|
||||||
for i in range(count):
|
|
||||||
await stream.send(i)
|
|
||||||
# falling out gracefully closes our stream side;
|
|
||||||
# all parent-side subscribers see end-of-channel.
|
|
||||||
|
|
||||||
|
|
||||||
async def consume(
|
|
||||||
name: str,
|
|
||||||
stream: tractor.MsgStream,
|
|
||||||
task_status: trio.TaskStatus = trio.TASK_STATUS_IGNORED,
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Consume a private broadcast-copy of the IPC stream.
|
|
||||||
|
|
||||||
'''
|
|
||||||
async with stream.subscribe() as bcaster:
|
|
||||||
task_status.started()
|
|
||||||
ticks: list[int] = []
|
|
||||||
async for tick in bcaster:
|
|
||||||
print(f'{name}: rx {tick}')
|
|
||||||
ticks.append(tick)
|
|
||||||
# EVERY subscriber gets its own full copy B)
|
|
||||||
print(f'{name}: stream ended, got {ticks}')
|
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'ticker',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
async with (
|
|
||||||
portal.open_context(
|
|
||||||
tick_stream,
|
|
||||||
count=5,
|
|
||||||
) as (ctx, first),
|
|
||||||
ctx.open_stream() as stream,
|
|
||||||
):
|
|
||||||
assert first == 5
|
|
||||||
async with trio.open_nursery() as tn:
|
|
||||||
# use `.start()` so each consumer is known
|
|
||||||
# to be subscribed before the ticks flow.
|
|
||||||
for i in range(3):
|
|
||||||
await tn.start(
|
|
||||||
consume,
|
|
||||||
f'sub_{i}',
|
|
||||||
stream,
|
|
||||||
)
|
|
||||||
await stream.send('go')
|
|
||||||
await portal.cancel_actor()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
@ -1,85 +0,0 @@
|
||||||
'''
|
|
||||||
Demonstrate "typed messaging": applying a `msgspec.Struct`
|
|
||||||
payload-type-spec to an IPC context via
|
|
||||||
`@tractor.context(pld_spec=...)`.
|
|
||||||
|
|
||||||
The child's `ctx.started()` value is stringently (round-trip)
|
|
||||||
validated against the spec *on the send side*, so a mistyped
|
|
||||||
payload raises a `tractor.MsgTypeError` before it ever hits
|
|
||||||
the wire; stream payloads are checked on `receive()` and
|
|
||||||
decode natively to the struct type.
|
|
||||||
|
|
||||||
'''
|
|
||||||
from msgspec import Struct
|
|
||||||
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
|
|
||||||
class Point(Struct):
|
|
||||||
'''
|
|
||||||
A simple 2D-coordinate msg-payload type.
|
|
||||||
|
|
||||||
'''
|
|
||||||
x: int
|
|
||||||
y: int
|
|
||||||
|
|
||||||
|
|
||||||
@tractor.context(pld_spec=Point|None)
|
|
||||||
async def point_doubler(
|
|
||||||
ctx: tractor.Context,
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Stream back each received `Point` with doubled fields.
|
|
||||||
|
|
||||||
'''
|
|
||||||
# deliberately send a non-`Point` as our started-value;
|
|
||||||
# the send-side pld-spec validation catches it locally
|
|
||||||
# BEFORE anything is shipped over IPC.
|
|
||||||
try:
|
|
||||||
await ctx.started('this is no Point..')
|
|
||||||
except tractor.MsgTypeError:
|
|
||||||
print(
|
|
||||||
'child: just as planned, a `str` payload failed '
|
|
||||||
'the `Point|None` pld-spec B)'
|
|
||||||
)
|
|
||||||
# now do it right; the parent receives this as the 2nd
|
|
||||||
# element of its `.open_context()` entry tuple.
|
|
||||||
await ctx.started(Point(x=0, y=0))
|
|
||||||
async with ctx.open_stream() as stream:
|
|
||||||
async for pt in stream:
|
|
||||||
# natively decoded to our struct type!
|
|
||||||
assert type(pt) is Point
|
|
||||||
await stream.send(
|
|
||||||
Point(
|
|
||||||
x=pt.x * 2,
|
|
||||||
y=pt.y * 2,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
async with tractor.open_nursery() as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'point_doubler',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
async with (
|
|
||||||
portal.open_context(
|
|
||||||
point_doubler,
|
|
||||||
) as (ctx, first),
|
|
||||||
ctx.open_stream() as stream,
|
|
||||||
):
|
|
||||||
# the (validated) started-value from the child
|
|
||||||
assert first == Point(x=0, y=0)
|
|
||||||
for i in range(3):
|
|
||||||
await stream.send(Point(x=i, y=i))
|
|
||||||
doubled: Point = await stream.receive()
|
|
||||||
assert doubled == Point(x=i * 2, y=i * 2)
|
|
||||||
print(f'parent: rx {doubled}')
|
|
||||||
# explicitly teardown the daemon-actor
|
|
||||||
await portal.cancel_actor()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
@ -1,53 +0,0 @@
|
||||||
'''
|
|
||||||
Demonstrate an actor tree which talks over unix-domain-socket
|
|
||||||
(UDS) transport instead of the default TCP: pass
|
|
||||||
`enable_transports=['uds']` when opening the root and every
|
|
||||||
subactor inherits the preference.
|
|
||||||
|
|
||||||
The child's channel address is a filesystem socket path (no
|
|
||||||
TCP port in sight!) and, as a kernel-provided bonus, the
|
|
||||||
peer's pid is exchanged for free via `SO_PEERCRED`.
|
|
||||||
|
|
||||||
'''
|
|
||||||
import os
|
|
||||||
|
|
||||||
import trio
|
|
||||||
import tractor
|
|
||||||
|
|
||||||
|
|
||||||
async def report_addr() -> str:
|
|
||||||
'''
|
|
||||||
Return this actor's own accept (bind) addr + pid.
|
|
||||||
|
|
||||||
'''
|
|
||||||
actor = tractor.current_actor()
|
|
||||||
addr: tuple = actor.accept_addr
|
|
||||||
pid: int = os.getpid()
|
|
||||||
return f'{actor.name}@{addr} pid={pid}'
|
|
||||||
|
|
||||||
|
|
||||||
async def main() -> None:
|
|
||||||
async with tractor.open_nursery(
|
|
||||||
enable_transports=['uds'],
|
|
||||||
) as an:
|
|
||||||
portal = await an.start_actor(
|
|
||||||
'uds_child',
|
|
||||||
enable_modules=[__name__],
|
|
||||||
)
|
|
||||||
# the channel's remote addr is a `UDSAddress`: a
|
|
||||||
# filesystem socket path, NOT a (host, port) pair!
|
|
||||||
raddr = portal.chan.raddr
|
|
||||||
assert raddr.proto_key == 'uds'
|
|
||||||
print(
|
|
||||||
f'portal chan tpt proto: {raddr.proto_key!r}\n'
|
|
||||||
f'portal chan sock file: {raddr.sockpath}\n'
|
|
||||||
f'kernel-reported peer pid: {raddr.maybe_pid}\n'
|
|
||||||
)
|
|
||||||
# ask the child for its own bind addr: also a
|
|
||||||
# socket-file path under the runtime dir.
|
|
||||||
print(f'child says: {await portal.run(report_addr)}')
|
|
||||||
await portal.cancel_actor()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
trio.run(main)
|
|
||||||
|
|
@ -118,19 +118,12 @@ eventfd = [
|
||||||
subints = [
|
subints = [
|
||||||
"msgspec>=0.21.0",
|
"msgspec>=0.21.0",
|
||||||
]
|
]
|
||||||
# docs generation; build locally via,
|
# TODO, add these with sane versions; were originally in
|
||||||
# uv run --group docs make -C docs html
|
# `requirements-docs.txt`..
|
||||||
# diagrams re-render when a `d2` bin is found, see
|
# docs = [
|
||||||
# `docs/_ext/d2diagrams.py` (eg. via `nix run
|
# "sphinx>="
|
||||||
# nixpkgs#d2` and the `D2_BIN` env var).
|
# "sphinx_book_theme>="
|
||||||
docs = [
|
# ]
|
||||||
"sphinx>=9.1,<10",
|
|
||||||
"pydata-sphinx-theme>=0.18,<0.19",
|
|
||||||
"sphinx-design>=0.7,<0.8",
|
|
||||||
"sphinx-copybutton>=0.5.2,<0.6",
|
|
||||||
"sphinxext-opengraph>=0.13,<0.14",
|
|
||||||
"sphinx-togglebutton>=0.4.5,<0.5",
|
|
||||||
]
|
|
||||||
# ------ dependency-groups ------
|
# ------ dependency-groups ------
|
||||||
|
|
||||||
[tool.uv.dependency-groups]
|
[tool.uv.dependency-groups]
|
||||||
|
|
|
||||||