diff --git a/docs/README.rst b/docs/README.rst index 9dd7faf4..e3bd9f84 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,20 +1,23 @@ -|logo| ``tractor``: next-gen Python parallelism +|logo| ``tractor``: distributed structurred concurrency |gh_actions| |docs| -``tractor`` is a `structured concurrent`_, (optionally -distributed_) multi-processing_ runtime built on trio_. +``tractor`` is a `structured concurrency`_ (SC), multi-processing_ runtime built on trio_. -Fundamentally, ``tractor`` gives you parallelism via -``trio``-"*actors*": independent Python processes (aka -non-shared-memory threads) which maintain structured -concurrency (SC) *end-to-end* inside a *supervision tree*. +Fundamentally, ``tractor`` provides parallelism via +``trio``-"*actors*": independent Python **processes** (i.e. +*non-shared-memory threads*) which can schedule ``trio`` tasks whilst +maintaining *end-to-end SC* inside a *distributed supervision tree*. Cross-process (and thus cross-host) SC is accomplished through the -combined use of our "actor nurseries_" and an "SC-transitive IPC -protocol" constructed on top of multiple Pythons each running a ``trio`` -scheduled runtime - a call to ``trio.run()``. +combined use of our, + +- "actor nurseries_" which provide for spawning multiple, and + possibly nested, Python processes each running a ``trio`` scheduled + runtime - a call to ``trio.run()``, +- an "SC-transitive supervision protocol" enforced as an + IPC-message-spec encapsulating all RPC-dialogs. We believe the system adheres to the `3 axioms`_ of an "`actor model`_" but likely **does not** look like what **you** probably *think* an "actor @@ -27,6 +30,7 @@ The first step to grok ``tractor`` is to get an intermediate knowledge of ``trio`` and **structured concurrency** B) Some great places to start are, + - the seminal `blog post`_ - obviously the `trio docs`_ - wikipedia's nascent SC_ page @@ -35,22 +39,84 @@ Some great places to start are, Features -------- -- **It's just** a ``trio`` API -- *Infinitely nesteable* process trees -- Builtin IPC streaming APIs with task fan-out broadcasting -- A "native" multi-core debugger REPL using `pdbp`_ (a fork & fix of - `pdb++`_ thanks to @mdmintz!) -- Support for a swappable, OS specific, process spawning layer -- A modular transport stack, allowing for custom serialization (eg. with - `msgspec`_), communications protocols, and environment specific IPC - primitives -- Support for spawning process-level-SC, inter-loop one-to-one-task oriented - ``asyncio`` actors via "infected ``asyncio``" mode -- `structured chadcurrency`_ from the ground up +- **It's just** a ``trio`` API! +- *Infinitely nesteable* process trees running embedded ``trio`` tasks. +- Swappable, OS-specific, process spawning via multiple backends. +- Modular IPC stack, allowing for custom interchange formats (eg. + as offered from `msgspec`_), varied transport protocols (TCP, RUDP, + QUIC, wireguard), and OS-env specific higher-perf primitives (UDS, + shm-ring-buffers). +- Optionally distributed_: all IPC and RPC APIs work over multi-host + transports the same as local. +- Builtin high-level streaming API that enables your app to easily + leverage the benefits of a "`cheap or nasty`_" `(un)protocol`_. +- A "native UX" around a multi-process safe debugger REPL using + `pdbp`_ (a fork & fix of `pdb++`_) +- "Infected ``asyncio``" mode: support for starting an actor's + runtime as a `guest`_ on the ``asyncio`` loop allowing us to + provide stringent SC-style ``trio.Task``-supervision around any + ``asyncio.Task`` spawned via our ``tractor.to_asyncio`` APIs. +- A **very naive** and still very much work-in-progress inter-actor + `discovery`_ sys with plans to support multiple `modern protocol`_ + approaches. +- Various ``trio`` extension APIs via ``tractor.trionics`` such as, + - task fan-out `broadcasting`_, + - multi-task-single-resource-caching and fan-out-to-multi + ``__aenter__()`` APIs for ``@acm`` functions, + - (WIP) a ``TaskMngr``: one-cancels-one style nursery supervisor. + + +Install +------- +``tractor`` is still in a *alpha-near-beta-stage* for many +of its subsystems, however we are very close to having a stable +lowlevel runtime and API. + +As such, it's currently recommended that you clone and install the +repo from source:: + + pip install git+git://github.com/goodboy/tractor.git + + +We use the very hip `uv`_ for project mgmt:: + + git clone https://github.com/goodboy/tractor.git + cd tractor + uv sync --dev + uv run python examples/rpc_bidir_streaming.py + +Consider activating a virtual/project-env before starting to hack on +the code base:: + + # you could use plain ol' venvs + # https://docs.astral.sh/uv/pip/environments/ + uv venv tractor_py313 --python 3.13 + + # but @goodboy prefers the more explicit (and shell agnostic) + # https://docs.astral.sh/uv/configuration/environment/#uv_project_environment + UV_PROJECT_ENVIRONMENT="tractor_py313 + + # hint hint, enter @goodboy's fave shell B) + uv run --dev xonsh + +Alongside all this we ofc offer "releases" on PyPi:: + + pip install tractor + +Just note that YMMV since the main git branch is often much further +ahead then any latest release. + + +Example codez +------------- +In ``tractor``'s (very lacking) documention we prefer to point to +example scripts in the repo over duplicating them in docs, but with +that in mind here are some definitive snippets to try and hook you +into digging deeper. Run a func in a process ------------------------ +*********************** Use ``trio``'s style of focussing on *tasks as functions*: .. code:: python @@ -108,7 +174,7 @@ might want to check out `trio-parallel`_. Zombie safe: self-destruct a process tree ------------------------------------------ +***************************************** ``tractor`` tries to protect you from zombies, no matter what. .. code:: python @@ -164,7 +230,7 @@ it **is a bug**. "Native" multi-process debugging --------------------------------- +******************************** Using the magic of `pdbp`_ and our internal IPC, we've been able to create a native feeling debugging experience for any (sub-)process in your ``tractor`` tree. @@ -219,7 +285,7 @@ We're hoping to add a respawn-from-repl system soon! SC compatible bi-directional streaming --------------------------------------- +************************************** Yes, you saw it here first; we provide 2-way streams with reliable, transitive setup/teardown semantics. @@ -311,7 +377,7 @@ hear your thoughts on! Worker poolz are easy peasy ---------------------------- +*************************** The initial ask from most new users is *"how do I make a worker pool thing?"*. @@ -333,10 +399,10 @@ This uses no extra threads, fancy semaphores or futures; all we need is ``tractor``'s IPC! "Infected ``asyncio``" mode ---------------------------- +*************************** Have a bunch of ``asyncio`` code you want to force to be SC at the process level? -Check out our experimental system for `guest-mode`_ controlled +Check out our experimental system for `guest`_-mode controlled ``asyncio`` actors: .. code:: python @@ -442,7 +508,7 @@ We need help refining the `asyncio`-side channel API to be more Higher level "cluster" APIs ---------------------------- +*************************** To be extra terse the ``tractor`` devs have started hacking some "higher level" APIs for managing actor trees/clusters. These interfaces should generally be condsidered provisional for now but we encourage you to try @@ -499,18 +565,6 @@ spawn a flat cluster: .. _full worker pool re-implementation: https://github.com/goodboy/tractor/blob/master/examples/parallelism/concurrent_actors_primes.py -Install -------- -From PyPi:: - - pip install tractor - - -From git:: - - pip install git+git://github.com/goodboy/tractor.git - - Under the hood -------------- ``tractor`` is an attempt to pair trionic_ `structured concurrency`_ with @@ -614,21 +668,26 @@ channel`_! .. _adherance to: https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=1821s .. _trio gitter channel: https://gitter.im/python-trio/general .. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _broadcasting: https://github.com/goodboy/tractor/pull/229 +.. _modern procotol: https://en.wikipedia.org/wiki/Rendezvous_protocol .. _pdbp: https://github.com/mdmintz/pdbp .. _pdb++: https://github.com/pdbpp/pdbpp -.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops +.. _cheap or nasty: https://zguide.zeromq.org/docs/chapter7/#The-Cheap-or-Nasty-Pattern +.. _(un)protocol: https://zguide.zeromq.org/docs/chapter7/#Unprotocols +.. _discovery: https://zguide.zeromq.org/docs/chapter8/#Discovery +.. _modern protocol: https://en.wikipedia.org/wiki/Rendezvous_protocol .. _messages: https://en.wikipedia.org/wiki/Message_passing .. _trio docs: https://trio.readthedocs.io/en/latest/ .. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ .. _structured concurrency: https://en.wikipedia.org/wiki/Structured_concurrency .. _SC: https://en.wikipedia.org/wiki/Structured_concurrency .. _libdill-docs: https://sustrik.github.io/libdill/structured-concurrency.html -.. _structured chadcurrency: https://en.wikipedia.org/wiki/Structured_concurrency .. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony .. _async generators: https://www.python.org/dev/peps/pep-0525/ .. _trio-parallel: https://github.com/richardsheridan/trio-parallel +.. _uv: https://docs.astral.sh/uv/ .. _msgspec: https://jcristharif.com/msgspec/ -.. _guest-mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops +.. _guest: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square diff --git a/pyproject.toml b/pyproject.toml index 9372685e..fbacc415 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,7 +32,7 @@ classifiers = [ "Topic :: System :: Distributed Computing", ] dependencies = [ -# trio runtime and friends + # trio runtime and friends # (poetry) proper range specs, # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5 # TODO, for 3.13 we must go go `0.27` which means we have to @@ -40,16 +40,12 @@ dependencies = [ # trio='^0.27' "trio>=0.24,<0.25", "tricycle>=0.4.1,<0.5", - "trio-typing>=0.10.0,<0.11", - "wrapt>=1.16.0,<2", "colorlog>=6.8.2,<7", - -# built-in multi-actor `pdb` REPL + # built-in multi-actor `pdb` REPL "pdbp>=1.5.0,<2", - -# typed IPC msging -# TODO, get back on release once 3.13 support is out! + # typed IPC msging + # TODO, get back on release once 3.13 support is out! "msgspec", ] @@ -73,6 +69,14 @@ dev = [ "xonsh-vox-tabcomplete>=0.5,<0.6", "pyperclip>=1.9.0", ] +# TODO, add these with sane versions; were originally in +# `requirements-docs.txt`.. +# docs = [ +# "sphinx>=" +# "sphinx_book_theme>=" +# ] + +# ------ dependency-groups ------ [tool.uv.sources] msgspec = { git = "https://github.com/jcrist/msgspec.git" } diff --git a/requirements-docs.txt b/requirements-docs.txt deleted file mode 100644 index 2c88be7e..00000000 --- a/requirements-docs.txt +++ /dev/null @@ -1,2 +0,0 @@ -sphinx -sphinx_book_theme diff --git a/requirements-test.txt b/requirements-test.txt deleted file mode 100644 index 8070f2c7..00000000 --- a/requirements-test.txt +++ /dev/null @@ -1,8 +0,0 @@ -pytest -pytest-trio -pytest-timeout -pdbp -mypy -trio_typing -pexpect -towncrier diff --git a/setup.py b/setup.py deleted file mode 100755 index 66b2622d..00000000 --- a/setup.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python -# -# tractor: structured concurrent "actors". -# -# Copyright 2018-eternity Tyler Goodlet. - -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. - -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. - -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see . - -from setuptools import setup - -with open('docs/README.rst', encoding='utf-8') as f: - readme = f.read() - - -setup( - name="tractor", - version='0.1.0a6dev0', # alpha zone - description='structured concurrent `trio`-"actors"', - long_description=readme, - license='AGPLv3', - author='Tyler Goodlet', - maintainer='Tyler Goodlet', - maintainer_email='goodboy_foss@protonmail.com', - url='https://github.com/goodboy/tractor', - platforms=['linux', 'windows'], - packages=[ - 'tractor', - 'tractor.experimental', # wacky ideas - 'tractor.trionics', # trio extensions - 'tractor.msg', # lowlevel data types - 'tractor._testing', # internal cross-subsys suite utils - 'tractor.devx', # "dev-experience" - ], - install_requires=[ - - # trio related - # proper range spec: - # https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5 - 'trio == 0.24', - - # 'async_generator', # in stdlib mostly! - # 'trio_typing', # trio==0.23.0 has type hints! - # 'exceptiongroup', # in stdlib as of 3.11! - - # tooling - 'stackscope', - 'tricycle', - 'trio_typing', - 'colorlog', - 'wrapt', - - # IPC serialization - 'msgspec', - - # debug mode REPL - 'pdbp', - - # TODO: distributed transport using - # linux kernel networking - # 'pyroute2', - - # pip ref docs on these specs: - # https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples - # and pep: - # https://peps.python.org/pep-0440/#version-specifiers - - ], - tests_require=['pytest'], - python_requires=">=3.11", - keywords=[ - 'trio', - 'async', - 'concurrency', - 'structured concurrency', - 'actor model', - 'distributed', - 'multiprocessing' - ], - classifiers=[ - "Development Status :: 3 - Alpha", - "Operating System :: POSIX :: Linux", - "Operating System :: Microsoft :: Windows", - "Framework :: Trio", - "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", - "Programming Language :: Python :: Implementation :: CPython", - "Programming Language :: Python :: 3 :: Only", - "Programming Language :: Python :: 3.10", - "Intended Audience :: Science/Research", - "Intended Audience :: Developers", - "Topic :: System :: Distributed Computing", - ], -) diff --git a/uv.lock b/uv.lock index 97b2e166..d89cdfe2 100644 --- a/uv.lock +++ b/uv.lock @@ -2,15 +2,6 @@ version = 1 revision = 1 requires-python = ">=3.11" -[[package]] -name = "async-generator" -version = "1.10" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/ce/b6/6fa6b3b598a03cba5e80f829e0dadbb49d7645f523d209b2fb7ea0bbb02a/async_generator-1.10.tar.gz", hash = "sha256:6ebb3d106c12920aaae42ccb6f787ef5eefdcdd166ea3d628fa8476abe712144", size = 29870 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/71/52/39d20e03abd0ac9159c162ec24b93fbcaa111e8400308f2465432495ca2b/async_generator-1.10-py3-none-any.whl", hash = "sha256:01c7bf666359b4967d2cda0000cc2e4af16a0ae098cbffcb8472fb9e8ad6585b", size = 18857 }, -] - [[package]] name = "attrs" version = "24.3.0" @@ -123,18 +114,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442 }, ] -[[package]] -name = "importlib-metadata" -version = "8.6.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "zipp" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/33/08/c1395a292bb23fd03bdf572a1357c5a733d3eecbab877641ceacab23db6e/importlib_metadata-8.6.1.tar.gz", hash = "sha256:310b41d755445d74569f993ccfc22838295d9fe005425094fad953d7f15c8580", size = 55767 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/79/9d/0fb148dc4d6fa4a7dd1d8378168d9b4cd8d4560a6fbf6f0121c5fc34eb68/importlib_metadata-8.6.1-py3-none-any.whl", hash = "sha256:02a89390c1e15fdfdc0d7c6b25cb3e62650d0494005c97d6f148bf5b9787525e", size = 26971 }, -] - [[package]] name = "iniconfig" version = "2.0.0" @@ -149,15 +128,6 @@ name = "msgspec" version = "0.19.0" source = { git = "https://github.com/jcrist/msgspec.git#dd965dce22e5278d4935bea923441ecde31b5325" } -[[package]] -name = "mypy-extensions" -version = "1.0.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/98/a4/1ab47638b92648243faf97a5aeb6ea83059cc3624972ab6b8d2316078d3f/mypy_extensions-1.0.0.tar.gz", hash = "sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782", size = 4433 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl", hash = "sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d", size = 4695 }, -] - [[package]] name = "outcome" version = "1.3.0.post0" @@ -332,7 +302,6 @@ dependencies = [ { name = "pdbp" }, { name = "tricycle" }, { name = "trio" }, - { name = "trio-typing" }, { name = "wrapt" }, ] @@ -356,7 +325,6 @@ requires-dist = [ { name = "pdbp", specifier = ">=1.5.0,<2" }, { name = "tricycle", specifier = ">=0.4.1,<0.5" }, { name = "trio", specifier = ">=0.24,<0.25" }, - { name = "trio-typing", specifier = ">=0.10.0,<0.11" }, { name = "wrapt", specifier = ">=1.16.0,<2" }, ] @@ -402,32 +370,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/14/fb/9299cf74953f473a15accfdbe2c15218e766bae8c796f2567c83bae03e98/trio-0.24.0-py3-none-any.whl", hash = "sha256:c3bd3a4e3e3025cd9a2241eae75637c43fe0b9e88b4c97b9161a55b9e54cd72c", size = 460205 }, ] -[[package]] -name = "trio-typing" -version = "0.10.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "async-generator" }, - { name = "importlib-metadata" }, - { name = "mypy-extensions" }, - { name = "packaging" }, - { name = "trio" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/b5/74/a87aafa40ec3a37089148b859892cbe2eef08d132c816d58a60459be5337/trio-typing-0.10.0.tar.gz", hash = "sha256:065ee684296d52a8ab0e2374666301aec36ee5747ac0e7a61f230250f8907ac3", size = 38747 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/89/ff/9bd795273eb14fac7f6a59d16cc8c4d0948a619a1193d375437c7f50f3eb/trio_typing-0.10.0-py3-none-any.whl", hash = "sha256:6d0e7ec9d837a2fe03591031a172533fbf4a1a95baf369edebfc51d5a49f0264", size = 42224 }, -] - -[[package]] -name = "typing-extensions" -version = "4.12.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/df/db/f35a00659bc03fec321ba8bce9420de607a1d37f8342eee1863174c69557/typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8", size = 85321 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/26/9f/ad63fc0248c5379346306f8668cda6e2e2e9c95e01216d2b8ffd9ff037d0/typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d", size = 37438 }, -] - [[package]] name = "wcwidth" version = "0.2.13" @@ -522,12 +464,3 @@ sdist = { url = "https://files.pythonhosted.org/packages/6c/ac/a5db68a1f2e4036f7 wheels = [ { url = "https://files.pythonhosted.org/packages/23/58/dcdf11849c8340033da00669527ce75d8292a4e8d82605c082ed236a081a/xontrib_vox-0.0.1-py3-none-any.whl", hash = "sha256:df2bbb815832db5b04d46684f540eac967ee40ef265add2662a95d6947d04c70", size = 13467 }, ] - -[[package]] -name = "zipp" -version = "3.21.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3f/50/bad581df71744867e9468ebd0bcd6505de3b275e06f202c2cb016e3ff56f/zipp-3.21.0.tar.gz", hash = "sha256:2c9958f6430a2040341a52eb608ed6dd93ef4392e02ffe219417c1b28b5dd1f4", size = 24545 } -wheels = [ - { url = "https://files.pythonhosted.org/packages/b7/1a/7e4798e9339adc931158c9d69ecc34f5e6791489d469f5e50ec15e35f458/zipp-3.21.0-py3-none-any.whl", hash = "sha256:ac1bbe05fd2991f160ebce24ffbac5f6d11d83dc90891255885223d42b3cd931", size = 9630 }, -]