docs/README.rst

@@ -1,20 +1,23 @@
-|logo| ``tractor``: next-gen Python parallelism
+|logo| ``tractor``: distributed structurred concurrency
 
 |gh_actions|
 |docs|
 
-``tractor`` is a `structured concurrent`_, (optionally
+``tractor`` is a `structured concurrency`_ (SC), multi-processing_ runtime built on trio_.
-distributed_) multi-processing_ runtime built on trio_.
 
-Fundamentally, ``tractor`` gives you parallelism via
+Fundamentally, ``tractor`` provides parallelism via
-``trio``-"*actors*": independent Python processes (aka
+``trio``-"*actors*": independent Python **processes** (i.e.
-non-shared-memory threads) which maintain structured
+*non-shared-memory threads*) which can schedule ``trio`` tasks whilst
-concurrency (SC) *end-to-end* inside a *supervision tree*.
+maintaining *end-to-end SC* inside a *distributed supervision tree*.
 
 Cross-process (and thus cross-host) SC is accomplished through the
-combined use of our "actor nurseries_" and an "SC-transitive IPC
+combined use of our,
-protocol" constructed on top of multiple Pythons each running a ``trio``
+
-scheduled runtime - a call to ``trio.run()``.
+- "actor nurseries_" which provide for spawning multiple, and
+  possibly nested, Python processes each running a ``trio`` scheduled
+  runtime - a call to ``trio.run()``,
+- an "SC-transitive supervision protocol" enforced as an
+  IPC-message-spec encapsulating all RPC-dialogs.
 
 We believe the system adheres to the `3 axioms`_ of an "`actor model`_"
 but likely **does not** look like what **you** probably *think* an "actor
@@ -27,6 +30,7 @@ The first step to grok ``tractor`` is to get an intermediate
 knowledge of ``trio`` and **structured concurrency** B)
 
 Some great places to start are,
+
 - the seminal `blog post`_
 - obviously the `trio docs`_
 - wikipedia's nascent SC_ page
@@ -35,22 +39,84 @@ Some great places to start are,
 
 Features
 --------
-- **It's just** a ``trio`` API
+- **It's just** a ``trio`` API!
-- *Infinitely nesteable* process trees
+- *Infinitely nesteable* process trees running embedded ``trio`` tasks.
-- Builtin IPC streaming APIs with task fan-out broadcasting
+- Swappable, OS-specific, process spawning via multiple backends.
-- A "native" multi-core debugger REPL using `pdbp`_ (a fork & fix of
+- Modular IPC stack, allowing for custom interchange formats (eg.
-  `pdb++`_ thanks to @mdmintz!)
+  as offered from `msgspec`_), varied transport protocols (TCP, RUDP,
-- Support for a swappable, OS specific, process spawning layer
+  QUIC, wireguard), and OS-env specific higher-perf primitives (UDS,
-- A modular transport stack, allowing for custom serialization (eg. with
+  shm-ring-buffers).
-  `msgspec`_), communications protocols, and environment specific IPC
+- Optionally distributed_: all IPC and RPC APIs work over multi-host
-  primitives
+  transports the same as local.
-- Support for spawning process-level-SC, inter-loop one-to-one-task oriented
+- Builtin high-level streaming API that enables your app to easily
-  ``asyncio`` actors via "infected ``asyncio``" mode
+  leverage the benefits of a "`cheap or nasty`_" `(un)protocol`_.
-- `structured chadcurrency`_ from the ground up
+- A "native UX" around a multi-process safe debugger REPL using
+  `pdbp`_ (a fork & fix of `pdb++`_)
+- "Infected ``asyncio``" mode: support for starting an actor's
+  runtime as a `guest`_ on the ``asyncio`` loop allowing us to
+  provide stringent SC-style ``trio.Task``-supervision around any
+  ``asyncio.Task`` spawned via our ``tractor.to_asyncio`` APIs.
+- A **very naive** and still very much work-in-progress inter-actor
+  `discovery`_ sys with plans to support multiple `modern protocol`_
+  approaches.
+- Various ``trio`` extension APIs via ``tractor.trionics`` such as,
+  - task fan-out `broadcasting`_,
+  - multi-task-single-resource-caching and fan-out-to-multi
+    ``__aenter__()`` APIs for ``@acm`` functions,
+  - (WIP) a ``TaskMngr``: one-cancels-one style nursery supervisor.
+
+
+Install
+-------
+``tractor`` is still in a *alpha-near-beta-stage* for many
+of its subsystems, however we are very close to having a stable
+lowlevel runtime and API.
+
+As such, it's currently recommended that you clone and install the
+repo from source::
+
+    pip install git+git://github.com/goodboy/tractor.git
+
+
+We use the very hip `uv`_ for project mgmt::
+
+    git clone https://github.com/goodboy/tractor.git
+    cd tractor
+    uv sync --dev
+    uv run python examples/rpc_bidir_streaming.py
+
+Consider activating a virtual/project-env before starting to hack on
+the code base::
+
+    # you could use plain ol' venvs
+    # https://docs.astral.sh/uv/pip/environments/
+    uv venv tractor_py313 --python 3.13
+
+    # but @goodboy prefers the more explicit (and shell agnostic)
+    # https://docs.astral.sh/uv/configuration/environment/#uv_project_environment
+    UV_PROJECT_ENVIRONMENT="tractor_py313
+
+    # hint hint, enter @goodboy's fave shell B)
+    uv run --dev xonsh
+
+Alongside all this we ofc offer "releases" on PyPi::
+
+    pip install tractor
+
+Just note that YMMV since the main git branch is often much further
+ahead then any latest release.
+
+
+Example codez
+-------------
+In ``tractor``'s (very lacking) documention we prefer to point to
+example scripts in the repo over duplicating them in docs, but with
+that in mind here are some definitive snippets to try and hook you
+into digging deeper.
 
 
 Run a func in a process
------------------------
+***********************
 Use ``trio``'s style of focussing on *tasks as functions*:
 
 .. code:: python
@@ -108,7 +174,7 @@ might want to check out `trio-parallel`_.
 
 
 Zombie safe: self-destruct a process tree
------------------------------------------
+*****************************************
 ``tractor`` tries to protect you from zombies, no matter what.
 
 .. code:: python
@@ -164,7 +230,7 @@ it **is a bug**.
 
 
 "Native" multi-process debugging
---------------------------------
+********************************
 Using the magic of `pdbp`_ and our internal IPC, we've
 been able to create a native feeling debugging experience for
 any (sub-)process in your ``tractor`` tree.
@@ -219,7 +285,7 @@ We're hoping to add a respawn-from-repl system soon!
 
 
 SC compatible bi-directional streaming
---------------------------------------
+**************************************
 Yes, you saw it here first; we provide 2-way streams
 with reliable, transitive setup/teardown semantics.
 
@@ -311,7 +377,7 @@ hear your thoughts on!
 
 
 Worker poolz are easy peasy
----------------------------
+***************************
 The initial ask from most new users is *"how do I make a worker
 pool thing?"*.
 
@@ -333,10 +399,10 @@ This uses no extra threads, fancy semaphores or futures; all we need
 is ``tractor``'s IPC!
 
 "Infected ``asyncio``" mode
----------------------------
+***************************
 Have a bunch of ``asyncio`` code you want to force to be SC at the process level?
 
-Check out our experimental system for `guest-mode`_ controlled
+Check out our experimental system for `guest`_-mode controlled
 ``asyncio`` actors:
 
 .. code:: python
@@ -442,7 +508,7 @@ We need help refining the `asyncio`-side channel API to be more
 
 
 Higher level "cluster" APIs
----------------------------
+***************************
 To be extra terse the ``tractor`` devs have started hacking some "higher
 level" APIs for managing actor trees/clusters. These interfaces should
 generally be condsidered provisional for now but we encourage you to try
@@ -499,18 +565,6 @@ spawn a flat cluster:
 .. _full worker pool re-implementation: https://github.com/goodboy/tractor/blob/master/examples/parallelism/concurrent_actors_primes.py
 
 
-Install
--------
-From PyPi::
-
-    pip install tractor
-
-
-From git::
-
-    pip install git+git://github.com/goodboy/tractor.git
-
-
 Under the hood
 --------------
 ``tractor`` is an attempt to pair trionic_ `structured concurrency`_ with
@@ -614,21 +668,26 @@ channel`_!
 .. _adherance to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s
 .. _trio gitter channel: https://gitter.im/python-trio/general
 .. _matrix channel: https://matrix.to/#/!tractor:matrix.org
+.. _broadcasting: https://github.com/goodboy/tractor/pull/229
+.. _modern procotol: https://en.wikipedia.org/wiki/Rendezvous_protocol
 .. _pdbp: https://github.com/mdmintz/pdbp
 .. _pdb++: https://github.com/pdbpp/pdbpp
-.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
+.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern
+.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols
+.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery
+.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol
 .. _messages: https://en.wikipedia.org/wiki/Message_passing
 .. _trio docs: https://trio.readthedocs.io/en/latest/
 .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
 .. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _SC: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html
-.. _structured chadcurrency: https://en.wikipedia.org/wiki/Structured_concurrency
 .. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony
 .. _async generators: https://www.python.org/dev/peps/pep-0525/
 .. _trio-parallel: https://github.com/richardsheridan/trio-parallel
+.. _uv: https://docs.astral.sh/uv/
 .. _msgspec: https://jcristharif.com/msgspec/
-.. _guest-mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
+.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops
 
 
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square

pyproject.toml

@@ -32,7 +32,7 @@ classifiers = [
   "Topic :: System :: Distributed Computing",
 ]
 dependencies = [
-# trio runtime and friends
+  # trio runtime and friends
   # (poetry) proper range specs,
   # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
   # TODO, for 3.13 we must go go `0.27` which means we have to
@@ -40,16 +40,12 @@ dependencies = [
   # trio='^0.27'
   "trio>=0.24,<0.25",
   "tricycle>=0.4.1,<0.5",
-  "trio-typing>=0.10.0,<0.11",
-
   "wrapt>=1.16.0,<2",
   "colorlog>=6.8.2,<7",
-
+  # built-in multi-actor `pdb` REPL
-# built-in multi-actor `pdb` REPL
   "pdbp>=1.5.0,<2",
-
+  # typed IPC msging
-# typed IPC msging
+  # TODO, get back on release once 3.13 support is out!
-# TODO, get back on release once 3.13 support is out!
   "msgspec",
 ]
 
@@ -73,6 +69,14 @@ dev = [
   "xonsh-vox-tabcomplete>=0.5,<0.6",
   "pyperclip>=1.9.0",
 ]
+# TODO, add these with sane versions; were originally in
+# `requirements-docs.txt`..
+# docs = [
+#   "sphinx>="
+#   "sphinx_book_theme>="
+# ]
+
+# ------ dependency-groups ------
 
 [tool.uv.sources]
 msgspec = { git = "https://github.com/jcrist/msgspec.git" }

requirements-docs.txt

@@ -1,2 +0,0 @@
-sphinx
-sphinx_book_theme

requirements-test.txt

@@ -1,8 +0,0 @@
-pytest
-pytest-trio
-pytest-timeout
-pdbp
-mypy
-trio_typing
-pexpect
-towncrier

setup.py

@@ -1,103 +0,0 @@
-#!/usr/bin/env python
-#
-# tractor: structured concurrent "actors".
-#
-# Copyright 2018-eternity Tyler Goodlet.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-
-# You should have received a copy of the GNU Affero General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-from setuptools import setup
-
-with open('docs/README.rst', encoding='utf-8') as f:
-    readme = f.read()
-
-
-setup(
-    name="tractor",
-    version='0.1.0a6dev0',  # alpha zone
-    description='structured concurrent `trio`-"actors"',
-    long_description=readme,
-    license='AGPLv3',
-    author='Tyler Goodlet',
-    maintainer='Tyler Goodlet',
-    maintainer_email='goodboy_foss@protonmail.com',
-    url='https://github.com/goodboy/tractor',
-    platforms=['linux', 'windows'],
-    packages=[
-        'tractor',
-        'tractor.experimental',  # wacky ideas
-        'tractor.trionics',  # trio extensions
-        'tractor.msg',  # lowlevel data types
-        'tractor._testing',  # internal cross-subsys suite utils
-        'tractor.devx',  # "dev-experience"
-    ],
-    install_requires=[
-
-        # trio related
-        # proper range spec:
-        # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
-        'trio == 0.24',
-
-        # 'async_generator',  # in stdlib mostly!
-        # 'trio_typing',  # trio==0.23.0 has type hints!
-        # 'exceptiongroup',  # in stdlib as of 3.11!
-
-        # tooling
-        'stackscope',
-        'tricycle',
-        'trio_typing',
-        'colorlog',
-        'wrapt',
-
-        # IPC serialization
-        'msgspec',
-
-        # debug mode REPL
-        'pdbp',
-
-        # TODO: distributed transport using
-        # linux kernel networking
-        # 'pyroute2',
-
-        # pip ref docs on these specs:
-        # https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples
-        # and pep:
-        # https://peps.python.org/pep-0440/#version-specifiers
-
-    ],
-    tests_require=['pytest'],
-    python_requires=">=3.11",
-    keywords=[
-        'trio',
-        'async',
-        'concurrency',
-        'structured concurrency',
-        'actor model',
-        'distributed',
-        'multiprocessing'
-    ],
-    classifiers=[
-        "Development Status :: 3 - Alpha",
-        "Operating System :: POSIX :: Linux",
-        "Operating System :: Microsoft :: Windows",
-        "Framework :: Trio",
-        "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
-        "Programming Language :: Python :: Implementation :: CPython",
-        "Programming Language :: Python :: 3 :: Only",
-        "Programming Language :: Python :: 3.10",
-        "Intended Audience :: Science/Research",
-        "Intended Audience :: Developers",
-        "Topic :: System :: Distributed Computing",
-    ],
-)

uv.lock

@@ -2,15 +2,6 @@ version = 1
 revision = 1
 requires-python = ">=3.11"
 
-[[package]]
-name = "async-generator"
-version = "1.10"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ce/b6/6fa6b3b598a03cba5e80f829e0dadbb49d7645f523d209b2fb7ea0bbb02a/async_generator-1.10.tar.gz", hash = "sha256:6ebb3d106c12920aaae42ccb6f787ef5eefdcdd166ea3d628fa8476abe712144", size = 29870 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/71/52/39d20e03abd0ac9159c162ec24b93fbcaa111e8400308f2465432495ca2b/async_generator-1.10-py3-none-any.whl", hash = "sha256:01c7bf666359b4967d2cda0000cc2e4af16a0ae098cbffcb8472fb9e8ad6585b", size = 18857 },
-]
-
 [[package]]
 name = "attrs"
 version = "24.3.0"
@@ -123,18 +114,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442 },
 ]
 
-[[package]]
-name = "importlib-metadata"
-version = "8.6.1"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "zipp" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/33/08/c1395a292bb23fd03bdf572a1357c5a733d3eecbab877641ceacab23db6e/importlib_metadata-8.6.1.tar.gz", hash = "sha256:310b41d755445d74569f993ccfc22838295d9fe005425094fad953d7f15c8580", size = 55767 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/79/9d/0fb148dc4d6fa4a7dd1d8378168d9b4cd8d4560a6fbf6f0121c5fc34eb68/importlib_metadata-8.6.1-py3-none-any.whl", hash = "sha256:02a89390c1e15fdfdc0d7c6b25cb3e62650d0494005c97d6f148bf5b9787525e", size = 26971 },
-]
-
 [[package]]
 name = "iniconfig"
 version = "2.0.0"
@@ -149,15 +128,6 @@ name = "msgspec"
 version = "0.19.0"
 source = { git = "https://github.com/jcrist/msgspec.git#dd965dce22e5278d4935bea923441ecde31b5325" }
 
-[[package]]
-name = "mypy-extensions"
-version = "1.0.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/98/a4/1ab47638b92648243faf97a5aeb6ea83059cc3624972ab6b8d2316078d3f/mypy_extensions-1.0.0.tar.gz", hash = "sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782", size = 4433 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl", hash = "sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d", size = 4695 },
-]
-
 [[package]]
 name = "outcome"
 version = "1.3.0.post0"
@@ -332,7 +302,6 @@ dependencies = [
     { name = "pdbp" },
     { name = "tricycle" },
     { name = "trio" },
-    { name = "trio-typing" },
     { name = "wrapt" },
 ]
 
@@ -356,7 +325,6 @@ requires-dist = [
     { name = "pdbp", specifier = ">=1.5.0,<2" },
     { name = "tricycle", specifier = ">=0.4.1,<0.5" },
     { name = "trio", specifier = ">=0.24,<0.25" },
-    { name = "trio-typing", specifier = ">=0.10.0,<0.11" },
     { name = "wrapt", specifier = ">=1.16.0,<2" },
 ]
 
@@ -402,32 +370,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/14/fb/9299cf74953f473a15accfdbe2c15218e766bae8c796f2567c83bae03e98/trio-0.24.0-py3-none-any.whl", hash = "sha256:c3bd3a4e3e3025cd9a2241eae75637c43fe0b9e88b4c97b9161a55b9e54cd72c", size = 460205 },
 ]
 
-[[package]]
-name = "trio-typing"
-version = "0.10.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "async-generator" },
-    { name = "importlib-metadata" },
-    { name = "mypy-extensions" },
-    { name = "packaging" },
-    { name = "trio" },
-    { name = "typing-extensions" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/b5/74/a87aafa40ec3a37089148b859892cbe2eef08d132c816d58a60459be5337/trio-typing-0.10.0.tar.gz", hash = "sha256:065ee684296d52a8ab0e2374666301aec36ee5747ac0e7a61f230250f8907ac3", size = 38747 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/89/ff/9bd795273eb14fac7f6a59d16cc8c4d0948a619a1193d375437c7f50f3eb/trio_typing-0.10.0-py3-none-any.whl", hash = "sha256:6d0e7ec9d837a2fe03591031a172533fbf4a1a95baf369edebfc51d5a49f0264", size = 42224 },
-]
-
-[[package]]
-name = "typing-extensions"
-version = "4.12.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/df/db/f35a00659bc03fec321ba8bce9420de607a1d37f8342eee1863174c69557/typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8", size = 85321 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/26/9f/ad63fc0248c5379346306f8668cda6e2e2e9c95e01216d2b8ffd9ff037d0/typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", size = 37438 },
-]
-
 [[package]]
 name = "wcwidth"
 version = "0.2.13"
@@ -522,12 +464,3 @@ sdist = { url = "https://files.pythonhosted.org/packages/6c/ac/a5db68a1f2e4036f7
 wheels = [
     { url = "https://files.pythonhosted.org/packages/23/58/dcdf11849c8340033da00669527ce75d8292a4e8d82605c082ed236a081a/xontrib_vox-0.0.1-py3-none-any.whl", hash = "sha256:df2bbb815832db5b04d46684f540eac967ee40ef265add2662a95d6947d04c70", size = 13467 },
 ]
-
-[[package]]
-name = "zipp"
-version = "3.21.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/3f/50/bad581df71744867e9468ebd0bcd6505de3b275e06f202c2cb016e3ff56f/zipp-3.21.0.tar.gz", hash = "sha256:2c9958f6430a2040341a52eb608ed6dd93ef4392e02ffe219417c1b28b5dd1f4", size = 24545 }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/b7/1a/7e4798e9339adc931158c9d69ecc34f5e6791489d469f5e50ec15e35f458/zipp-3.21.0-py3-none-any.whl", hash = "sha256:ac1bbe05fd2991f160ebce24ffbac5f6d11d83dc90891255885223d42b3cd931", size = 9630 },
-]