Structured concurrency, across processes ========================================= ``tractor`` makes one bet: the discipline that made ``trio``'s concurrency *readable* — `structured concurrency`_ (SC) — works just as well when the "tasks" are whole OS processes talking over a wire. This page distills what that means, from first principles, with as little ceremony as possible. .. margin:: The canon If SC is new to you, the seminal `blog post`_ is still the best hour you'll spend on concurrent programming; the `trio docs`_, wikipedia's SC_ page and the diagrams over at libdill-docs_ round it out nicely. SC in one breath ---------------- Structured concurrency is the rule that **concurrency gets a scope**: every task is spawned *inside* a block (a ``trio`` *nursery*) and that block **cannot exit until every task it spawned has finished** — returned, errored, or been cancelled. That one rule buys you the properties you already rely on in sequential code, - a function call is a *black box*: when it returns, everything it started is **done** — no secret background tasks leaking out the sides, - an exception **always has somewhere to go**: up the (task) tree to a parent which is, by construction, still there waiting, - cancellation has a well defined *shape*: cancel a scope and it flows down to every task inside it, and only those. In short: your **runtime task tree matches your source code's indentation**. Concurrency you can read. The leap: process-shaped tasks ------------------------------ Now swap "task" for "process". A ``tractor`` *actor* is just a Python process running its own ``trio.run()`` — its own private task tree, sharing **nothing** with its siblings. You spawn actors from an :class:`tractor.ActorNursery`, which behaves exactly the way the name implies, .. code:: python async with tractor.open_nursery() as an: portal = await an.start_actor( 'worker', enable_modules=[__name__], ) ... # ^ block exit == every spawned process has # completed, errored or been cancelled, and # been **reaped**. No exceptions, no zombies. so the whole program becomes a *tree of process-trees* — a `supervision tree`_ in erlang-speak — where every arrow means "spawned by, **waited on by**, and supervised by". .. d2:: diagrams/actor_tree.d2 :caption: A ``tractor`` program: a process tree of ``trio`` task trees; every parent **must wait** on its children. :width: 85% Causality: no process outlives its parent ----------------------------------------- The stdlib's ``multiprocessing`` (and most "job queue" systems) treat child processes as fire-and-forget by default: orphans, zombies, lost tracebacks and ``kill -9`` cleanup scripts are *your* problem. ``tractor`` instead inherits ``trio``'s `causality`_ discipline, - **no spawning willy-nilly**: every actor is born from a nursery block with a known parent, - **lifetimes nest**: a sub-actor's entire process tree lives strictly inside its parent's nursery scope, - **teardown is guaranteed**: when a scope exits (or errors, or is cancelled) the runtime SIGINTs, waits, and (only if it must) hard-kills + reaps everything underneath. We take the zombie thing personally: *if you can create orphaned child processes without using a system signal, it* **is a bug** — and there's a test suite to back that sentence up. Errors always propagate (yes, across the wire) ---------------------------------------------- In ``trio``, an exception in any task tears through its nursery to a parent that must handle it — `exceptions always propagate`_. ``tractor`` extends the same guarantee across process boundaries: an uncaught error in a remote task is 1. captured + serialized in the child, 2. shipped home over IPC as a typed ``Error`` msg, 3. re-raised in the parent **boxed** as a :class:`tractor.RemoteActorError` carrying the original type (``.boxed_type``), a rendered remote traceback, and the erroring actor's id, while the supervising nursery applies its (currently *one-cancels-all*, just like ``trio``) strategy to any sibling actors. A crash three processes deep arrives at your shell as one coherent, causal traceback chain — not a silent dead worker and a stuck queue. Cancellation is a request, supervision is the rule -------------------------------------------------- Cancellation likewise keeps ``trio``'s semantics *verbatim*, just transported: cancelling an actor nursery (or a single :class:`tractor.Context` between two tasks in different processes) sends an explicit cancel **request** over IPC which the remote runtime translates into a real ``trio`` cancel-scope cancel — then *acks back* so the requester can await confirmation within a bounded time. Nothing is ever "just killed" first; graceful always precedes brutal. Because every cross-process dialog is a pair of **linked tasks** — one on each side, each inside its own cancel scope — SC stays *transitive*: supervision doesn't stop at the process boundary, it tunnels through every hop of the tree. The wire protocol that enforces this (a small set of typed msgs: ``Start``/``Started``/``Yield``/``Stop``/``Return``/ ``Error``) is detailed in :doc:`/guide/msging` and :doc:`/guide/context`. Hold up, is this an "actor model"? ---------------------------------- Let's stop and ask how many canon actor model papers you've actually read ;) From our experience many "actor systems" aren't really "actor models" since they **don't adhere** to the `3 axioms`_ and pay even less attention to the problem of `unbounded non-determinism`_ — which was the whole point for creating the model in the first place. From `the author's mouth`_, the **only** requirement is `adherence to`_ the `3 axioms`_:: In response to a message, an actor may: - send a finite number of new messages - create a finite number of new actors - designate a new behavior to process subsequent messages Want to debate whether that makes us a "real" actor model? Chime in on our chat, or on one of these issues *after you've read everything in them* ;) - `#210 `_ - `#18 `_ ``tractor`` adheres — actors exchange msgs, spawn actors, and swap behaviors — **with no extra API** to learn. What we *don't* copy is the cultural baggage: no visible mailboxes, no untyped fire-and-forget ``send()``, no "let it crash" without a supervisor that actually hears about it, and definitely no shared-reference *proxy objects* pretending the network isn't there. If our "actors" don't look like what you expected, that's **intentional**: being an actor model is just one property of the system; being *structured* is the point. Why processes at all? --------------------- Python has a GIL; an actor model by definition shares no state; so the *process* is the natural runtime unit — you get real multi-core parallelism and hard memory isolation for free. But the deeper win is uniformity: because actors only ever talk via msgs over a :class:`tractor.Channel` (TCP, UDS, more to come), the **same code** runs your laptop's worker pool and a multi-host cluster; "distributed" is a deployment detail, not an API. It's just ``trio`` ------------------ If you remember one framing, make it this: ``tractor`` **is just** ``trio`` — with nurseries that can spawn processes and streams that can cross them. Same nursery discipline, same cancellation semantics, same "how was this not always the API?" feeling, one level up the process tree. .. seealso:: :doc:`/explain/architecture` for how the runtime layers deliver all of the above, and :doc:`/start/quickstart` to feel it in ~20 lines of code. .. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency .. _SC: https://en.wikipedia.org/wiki/Structured_concurrency .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ .. _trio docs: https://trio.readthedocs.io/en/latest/ .. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html .. _supervision tree: https://www.erlang.org/doc/design_principles/des_princ.html .. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker .. _exceptions always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate .. _the author's mouth: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=162s .. _adherence to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s .. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts .. _unbounded non-determinism: https://en.wikipedia.org/wiki/Unbounded_nondeterminism