From 0a5a4d8487c4421186bfe9962caafcdb702d7fbc Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Wed, 14 Oct 2020 11:07:48 -0400 Subject: [PATCH 01/11] Readme rework draft --- docs/README.rst | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index e2e4dd3..ff58399 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,6 +1,7 @@ tractor ======= -A `structured concurrent`_, async-native "`actor model`_" built on trio_ and multiprocessing_. +The Python async-native multi-core system *you always wanted*. + |gh_actions| |docs| @@ -13,16 +14,22 @@ A `structured concurrent`_, async-native "`actor model`_" built on trio_ and mul .. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 -``tractor`` is an attempt to bring trionic_ `structured concurrency`_ to -distributed multi-core Python; it aims to be the Python multi-processing -framework *you always wanted*. +``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multiprocessing_. -``tractor`` lets you spawn ``trio`` *"actors"*: processes which each run -a ``trio`` scheduled task tree (also known as an `async sandwich`_). -*Actors* communicate by exchanging asynchronous messages_ and avoid -sharing any state. This model allows for highly distributed software -architecture which works just as well on multiple cores as it does over -many hosts. +It is an attempt to pair trionic_ `structured concurrency`_ with +distributed Python. You can think of it as a ``trio`` *-across-processes* +or simply as a replacement for the stdlib's `multiprocessing` but built +on async primitives for IPC. + +``tractor``'s nurseries lets you spawn ``trio`` *"actors"*: new Python +processes which each run a ``trio`` scheduled task tree (also known as +an `async sandwich`_). That is, each "*Actor*" is a new process plus +a `trio` runtime. + +Processes communicate by exchanging asynchronous messages_ and avoid +sharing state. The intention of this model is to allow for highly +distributed software that, through the adherence to *structured +concurrency*, results in systems which fail in predicatable ways. The first step to grok ``tractor`` is to get the basics of ``trio`` down. A great place to start is the `trio docs`_ and this `blog post`_. @@ -45,17 +52,28 @@ No PyPi release yet! pip install git+git://github.com/goodboy/tractor.git +Alluring Features +----------------- +- Infinitely nesteable process trees +- A built-in API for inter-process streaming +- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ +- (Soon to land) `asyncio` support allowing for "infected" actors where + `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" + + Feel like saying hi? -------------------- This project is very much coupled to the ongoing development of ``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant community). If you want to help, have suggestions or just want to -say hi, please feel free to reach us on in our `matrix channel`_. If +say hi, please feel free to reach us in our `matrix channel`_. If matrix seems too hip, we're also mostly all in the the `trio gitter channel`_! .. _trio gitter channel: https://gitter.im/python-trio/general .. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _pdb++: https://github.com/pdbpp/pdbpp +.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square From 90c987d0ae6ce8134aa9c68aff4ab9811d106164 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Wed, 9 Dec 2020 13:01:57 -0500 Subject: [PATCH 02/11] Further tweaks, add non-scary snippet --- docs/README.rst | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index ff58399..4300a84 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -8,28 +8,35 @@ The Python async-native multi-core system *you always wanted*. .. _actor model: https://en.wikipedia.org/wiki/Actor_model .. _trio: https://github.com/python-trio/trio -.. _multiprocessing: https://en.wikipedia.org/wiki/Multiprocessing +.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing .. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles .. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich .. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 -``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multiprocessing_. +``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. It is an attempt to pair trionic_ `structured concurrency`_ with -distributed Python. You can think of it as a ``trio`` *-across-processes* -or simply as a replacement for the stdlib's `multiprocessing` but built -on async primitives for IPC. +distributed Python. You can think of it as a ``trio`` +*-across-processes* or simply as an opinionated replacement for the +stdlib's ``multiprocessing`` but built on async programming primitives +from the ground up. -``tractor``'s nurseries lets you spawn ``trio`` *"actors"*: new Python +Don't be scared off by this description. ``tractor`` **is just ``trio``** +but with nurseries for process management and cancel-able IPC. +If you understand how to work with ``trio``, ``tractor`` will give you +the parallelism you've been missing. + +``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python processes which each run a ``trio`` scheduled task tree (also known as -an `async sandwich`_). That is, each "*Actor*" is a new process plus -a `trio` runtime. +an `async sandwich`_ - a call to ``trio.run()``). That is, each +"*Actor*" is a new process plus a ``trio`` runtime. -Processes communicate by exchanging asynchronous messages_ and avoid +"Actors" communicate by exchanging asynchronous messages_ and avoid sharing state. The intention of this model is to allow for highly distributed software that, through the adherence to *structured -concurrency*, results in systems which fail in predicatable ways. +concurrency*, results in systems which fail in predictable and +recoverable ways. The first step to grok ``tractor`` is to get the basics of ``trio`` down. A great place to start is the `trio docs`_ and this `blog post`_. @@ -54,13 +61,26 @@ No PyPi release yet! Alluring Features ----------------- +- **It's just** ``trio``, but with SC applied to processes (aka "actors") - Infinitely nesteable process trees -- A built-in API for inter-process streaming +- Built-in API for inter-process streaming - A (first ever?) "native" multi-core debugger for Python using `pdb++`_ -- (Soon to land) `asyncio` support allowing for "infected" actors where +- (Soon to land) ``asyncio`` support allowing for "infected" actors where `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" +The example you're probably after... +------------------------------------ +It seems the initial query from most new users is "how do I make a worker +pool thing?". + +``tractor`` is built to handle any SC process tree you can +imagine; the "worker pool" pattern is a trivial special case: + + +# TODO: workerpool example + + Feel like saying hi? -------------------- This project is very much coupled to the ongoing development of From 92f4b402ad2d45f108fc442a5edcd368938f24d3 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Sun, 21 Feb 2021 23:13:51 -0500 Subject: [PATCH 03/11] Draft use sphinx-restbuilder to gen readme --- docs/README.rst | 176 ++++++++++++++++---------- docs/_README.rst | 103 +++++++++++++++ docs/github_readme/_sphinx_readme.rst | 104 +++++++++++++++ docs/github_readme/conf.py | 51 ++++++++ docs/mk_gh_readme.sh | 4 + 5 files changed, 368 insertions(+), 70 deletions(-) create mode 100644 docs/_README.rst create mode 100644 docs/github_readme/_sphinx_readme.rst create mode 100644 docs/github_readme/conf.py create mode 100755 docs/mk_gh_readme.sh diff --git a/docs/README.rst b/docs/README.rst index 4300a84..150bbed 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,103 +1,139 @@ + tractor -======= +******* + The Python async-native multi-core system *you always wanted*. +https://actions-badge.atrox.dev/goodboy/tractor/goto +https://tractor.readthedocs.io/en/latest/?badge=latest -|gh_actions| -|docs| +``tractor`` is a `structured concurrent +`_ +“`actor model `_” built on +`trio `_ and `multi-processing +`_. -.. _actor model: https://en.wikipedia.org/wiki/Actor_model -.. _trio: https://github.com/python-trio/trio -.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing -.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles -.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich -.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 - - -``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. - -It is an attempt to pair trionic_ `structured concurrency`_ with -distributed Python. You can think of it as a ``trio`` +It is an attempt to pair `trionic +`_ +`structured concurrency +`_ +with distributed Python. You can think of it as a ``trio`` *-across-processes* or simply as an opinionated replacement for the -stdlib's ``multiprocessing`` but built on async programming primitives +stdlib’s ``multiprocessing`` but built on async programming primitives from the ground up. -Don't be scared off by this description. ``tractor`` **is just ``trio``** -but with nurseries for process management and cancel-able IPC. -If you understand how to work with ``trio``, ``tractor`` will give you -the parallelism you've been missing. +Don’t be scared off by this description. ``tractor`` **is just +``trio``** but with nurseries for process management and cancel-able +IPC. If you understand how to work with ``trio``, ``tractor`` will +give you the parallelism you’ve been missing. -``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python +``tractor``’s nurseries let you spawn ``trio`` *“actors”*: new Python processes which each run a ``trio`` scheduled task tree (also known as -an `async sandwich`_ - a call to ``trio.run()``). That is, each -"*Actor*" is a new process plus a ``trio`` runtime. +an `async sandwich +`_ +- a call to ``trio.run()``). That is, each “*Actor*” is a new process +plus a ``trio`` runtime. -"Actors" communicate by exchanging asynchronous messages_ and avoid -sharing state. The intention of this model is to allow for highly -distributed software that, through the adherence to *structured -concurrency*, results in systems which fail in predictable and -recoverable ways. +“Actors” communicate by exchanging asynchronous `messages +`_ and avoid sharing +state. The intention of this model is to allow for highly distributed +software that, through the adherence to *structured concurrency*, +results in systems which fail in predictable and recoverable ways. -The first step to grok ``tractor`` is to get the basics of ``trio`` down. -A great place to start is the `trio docs`_ and this `blog post`_. - -.. _messages: https://en.wikipedia.org/wiki/Message_passing -.. _trio docs: https://trio.readthedocs.io/en/latest/ -.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts -.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony -.. _async generators: https://www.python.org/dev/peps/pep-0525/ +The first step to grok ``tractor`` is to get the basics of ``trio`` +down. A great place to start is the `trio docs +`_ and this `blog post +`_. Install -------- +======= + No PyPi release yet! :: - pip install git+git://github.com/goodboy/tractor.git + pip install git+git://github.com/goodboy/tractor.git Alluring Features ------------------ -- **It's just** ``trio``, but with SC applied to processes (aka "actors") -- Infinitely nesteable process trees -- Built-in API for inter-process streaming -- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ -- (Soon to land) ``asyncio`` support allowing for "infected" actors where - `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" +================= + +* **It’s just** ``trio``, but with SC applied to processes (aka + “actors”) + +* Infinitely nesteable process trees + +* Built-in API for inter-process streaming + +* A (first ever?) “native” multi-core debugger for Python using + `pdb++ `_ + +* (Soon to land) ``asyncio`` support allowing for “infected” actors + where *trio* drives the *asyncio* scheduler via the astounding + “`guest mode + `_” -The example you're probably after... ------------------------------------- -It seems the initial query from most new users is "how do I make a worker -pool thing?". +The example you’re probably after… +================================== -``tractor`` is built to handle any SC process tree you can -imagine; the "worker pool" pattern is a trivial special case: +It seems the initial query from most new users is “how do I make a +worker pool thing?”. +``tractor`` is built to handle any SC process tree you can imagine; +the “worker pool” pattern is a trivial special case: # TODO: workerpool example +.. code:: + + """ + Run with a process monitor from a terminal using: + $TERM -e watch -n 0.1 "pstree -a $$" & python examples/parallelism/we_are_processes.py && kill $! + + """ + from multiprocessing import cpu_count + import os + + import tractor + import trio + + + async def target(): + print(f"Yo, i'm '{tractor.current_actor().name}' " + f"running in pid {os.getpid()}") + await trio.sleep_forever() + + + async def main(): + + async with tractor.open_nursery() as n: + + for i in range(cpu_count()): + await n.run_in_actor(target, name=f'worker_{i}') + + print('This process tree will self-destruct in 1 sec...') + await trio.sleep(1) + + # you could have done this yourself + raise Exception('Self Destructed') + + + if __name__ == '__main__': + try: + trio.run(main) + except Exception: + print('Zombies Contained') + Feel like saying hi? --------------------- +==================== + This project is very much coupled to the ongoing development of ``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant -community). If you want to help, have suggestions or just want to -say hi, please feel free to reach us in our `matrix channel`_. If -matrix seems too hip, we're also mostly all in the the `trio gitter -channel`_! - -.. _trio gitter channel: https://gitter.im/python-trio/general -.. _matrix channel: https://matrix.to/#/!tractor:matrix.org -.. _pdb++: https://github.com/pdbpp/pdbpp -.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops - - -.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square - :target: https://actions-badge.atrox.dev/goodboy/tractor/goto -.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest - :target: https://tractor.readthedocs.io/en/latest/?badge=latest - :alt: Documentation Status +community). If you want to help, have suggestions or just want to say +hi, please feel free to reach us in our `matrix channel +`_. If matrix seems too hip, +we’re also mostly all in the the `trio gitter channel +`_! diff --git a/docs/_README.rst b/docs/_README.rst new file mode 100644 index 0000000..4300a84 --- /dev/null +++ b/docs/_README.rst @@ -0,0 +1,103 @@ +tractor +======= +The Python async-native multi-core system *you always wanted*. + + +|gh_actions| +|docs| + +.. _actor model: https://en.wikipedia.org/wiki/Actor_model +.. _trio: https://github.com/python-trio/trio +.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing +.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles +.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich +.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 + + +``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. + +It is an attempt to pair trionic_ `structured concurrency`_ with +distributed Python. You can think of it as a ``trio`` +*-across-processes* or simply as an opinionated replacement for the +stdlib's ``multiprocessing`` but built on async programming primitives +from the ground up. + +Don't be scared off by this description. ``tractor`` **is just ``trio``** +but with nurseries for process management and cancel-able IPC. +If you understand how to work with ``trio``, ``tractor`` will give you +the parallelism you've been missing. + +``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python +processes which each run a ``trio`` scheduled task tree (also known as +an `async sandwich`_ - a call to ``trio.run()``). That is, each +"*Actor*" is a new process plus a ``trio`` runtime. + +"Actors" communicate by exchanging asynchronous messages_ and avoid +sharing state. The intention of this model is to allow for highly +distributed software that, through the adherence to *structured +concurrency*, results in systems which fail in predictable and +recoverable ways. + +The first step to grok ``tractor`` is to get the basics of ``trio`` down. +A great place to start is the `trio docs`_ and this `blog post`_. + +.. _messages: https://en.wikipedia.org/wiki/Message_passing +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts +.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony +.. _async generators: https://www.python.org/dev/peps/pep-0525/ + + +Install +------- +No PyPi release yet! + +:: + + pip install git+git://github.com/goodboy/tractor.git + + +Alluring Features +----------------- +- **It's just** ``trio``, but with SC applied to processes (aka "actors") +- Infinitely nesteable process trees +- Built-in API for inter-process streaming +- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ +- (Soon to land) ``asyncio`` support allowing for "infected" actors where + `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" + + +The example you're probably after... +------------------------------------ +It seems the initial query from most new users is "how do I make a worker +pool thing?". + +``tractor`` is built to handle any SC process tree you can +imagine; the "worker pool" pattern is a trivial special case: + + +# TODO: workerpool example + + +Feel like saying hi? +-------------------- +This project is very much coupled to the ongoing development of +``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant +community). If you want to help, have suggestions or just want to +say hi, please feel free to reach us in our `matrix channel`_. If +matrix seems too hip, we're also mostly all in the the `trio gitter +channel`_! + +.. _trio gitter channel: https://gitter.im/python-trio/general +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _pdb++: https://github.com/pdbpp/pdbpp +.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops + + +.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square + :target: https://actions-badge.atrox.dev/goodboy/tractor/goto +.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest + :target: https://tractor.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status diff --git a/docs/github_readme/_sphinx_readme.rst b/docs/github_readme/_sphinx_readme.rst new file mode 100644 index 0000000..03b09d5 --- /dev/null +++ b/docs/github_readme/_sphinx_readme.rst @@ -0,0 +1,104 @@ +tractor +======= +The Python async-native multi-core system *you always wanted*. + + +|gh_actions| +|docs| + +.. _actor model: https://en.wikipedia.org/wiki/Actor_model +.. _trio: https://github.com/python-trio/trio +.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing +.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles +.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich +.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 + + +``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. + +It is an attempt to pair trionic_ `structured concurrency`_ with +distributed Python. You can think of it as a ``trio`` +*-across-processes* or simply as an opinionated replacement for the +stdlib's ``multiprocessing`` but built on async programming primitives +from the ground up. + +Don't be scared off by this description. ``tractor`` **is just ``trio``** +but with nurseries for process management and cancel-able IPC. +If you understand how to work with ``trio``, ``tractor`` will give you +the parallelism you've been missing. + +``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python +processes which each run a ``trio`` scheduled task tree (also known as +an `async sandwich`_ - a call to ``trio.run()``). That is, each +"*Actor*" is a new process plus a ``trio`` runtime. + +"Actors" communicate by exchanging asynchronous messages_ and avoid +sharing state. The intention of this model is to allow for highly +distributed software that, through the adherence to *structured +concurrency*, results in systems which fail in predictable and +recoverable ways. + +The first step to grok ``tractor`` is to get the basics of ``trio`` down. +A great place to start is the `trio docs`_ and this `blog post`_. + +.. _messages: https://en.wikipedia.org/wiki/Message_passing +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts +.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony +.. _async generators: https://www.python.org/dev/peps/pep-0525/ + + +Install +------- +No PyPi release yet! + +:: + + pip install git+git://github.com/goodboy/tractor.git + + +Alluring Features +----------------- +- **It's just** ``trio``, but with SC applied to processes (aka "actors") +- Infinitely nesteable process trees +- Built-in API for inter-process streaming +- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ +- (Soon to land) ``asyncio`` support allowing for "infected" actors where + `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" + + +The example you're probably after... +------------------------------------ +It seems the initial query from most new users is "how do I make a worker +pool thing?". + +``tractor`` is built to handle any SC process tree you can +imagine; the "worker pool" pattern is a trivial special case: + +# TODO: workerpool example + +.. literalinclude:: ../../examples/parallelism/we_are_processes.py + + +Feel like saying hi? +-------------------- +This project is very much coupled to the ongoing development of +``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant +community). If you want to help, have suggestions or just want to +say hi, please feel free to reach us in our `matrix channel`_. If +matrix seems too hip, we're also mostly all in the the `trio gitter +channel`_! + +.. _trio gitter channel: https://gitter.im/python-trio/general +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _pdb++: https://github.com/pdbpp/pdbpp +.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops + + +.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square + :target: https://actions-badge.atrox.dev/goodboy/tractor/goto +.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest + :target: https://tractor.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status diff --git a/docs/github_readme/conf.py b/docs/github_readme/conf.py new file mode 100644 index 0000000..04d288e --- /dev/null +++ b/docs/github_readme/conf.py @@ -0,0 +1,51 @@ +# Configuration file for the Sphinx documentation builder. + +# this config is for the rst generation extension and thus +# requires only basic settings: +# https://github.com/sphinx-contrib/restbuilder + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + +# Warn about all references to unknown targets +nitpicky = True + +# The master toctree document. +master_doc = '_sphinx_readme' + +# -- Project information ----------------------------------------------------- + +project = 'tractor' +copyright = '2018, Tyler Goodlet' +author = 'Tyler Goodlet' + +# The full version, including alpha/beta/rc tags +release = '0.0.0a0.dev0' + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + # 'sphinx.ext.autodoc', + # 'sphinx.ext.intersphinx', + # 'sphinx.ext.todo', + 'sphinxcontrib.restbuilder', + +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] diff --git a/docs/mk_gh_readme.sh b/docs/mk_gh_readme.sh new file mode 100755 index 0000000..3836e2f --- /dev/null +++ b/docs/mk_gh_readme.sh @@ -0,0 +1,4 @@ +#!/bin/bash +sphinx-build -b rst ./github_readme ./ + +mv _sphinx_readme.rst README.rst From 0e7db46631bd56afaa1c3a34cee044053da9fafc Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Sun, 21 Feb 2021 23:52:41 -0500 Subject: [PATCH 04/11] Revert auto-gen readme and merge in auto-gen code blocks by hand for now --- docs/README.rst | 269 +++++++++++++++++++------- docs/_README.rst | 103 ---------- docs/github_readme/_sphinx_readme.rst | 11 +- docs/github_readme/conf.py | 6 +- docs/mk_gh_readme.sh | 2 +- 5 files changed, 210 insertions(+), 181 deletions(-) delete mode 100644 docs/_README.rst diff --git a/docs/README.rst b/docs/README.rst index 150bbed..5bf7952 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,92 +1,78 @@ - tractor -******* - +======= The Python async-native multi-core system *you always wanted*. -https://actions-badge.atrox.dev/goodboy/tractor/goto -https://tractor.readthedocs.io/en/latest/?badge=latest -``tractor`` is a `structured concurrent -`_ -“`actor model `_” built on -`trio `_ and `multi-processing -`_. +|gh_actions| +|docs| -It is an attempt to pair `trionic -`_ -`structured concurrency -`_ -with distributed Python. You can think of it as a ``trio`` +.. _actor model: https://en.wikipedia.org/wiki/Actor_model +.. _trio: https://github.com/python-trio/trio +.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing +.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles +.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich +.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 + + +``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. + +It is an attempt to pair trionic_ `structured concurrency`_ with +distributed Python. You can think of it as a ``trio`` *-across-processes* or simply as an opinionated replacement for the -stdlib’s ``multiprocessing`` but built on async programming primitives +stdlib's ``multiprocessing`` but built on async programming primitives from the ground up. -Don’t be scared off by this description. ``tractor`` **is just -``trio``** but with nurseries for process management and cancel-able -IPC. If you understand how to work with ``trio``, ``tractor`` will -give you the parallelism you’ve been missing. +Don't be scared off by this description. ``tractor`` **is just ``trio``** +but with nurseries for process management and cancel-able IPC. +If you understand how to work with ``trio``, ``tractor`` will give you +the parallelism you've been missing. -``tractor``’s nurseries let you spawn ``trio`` *“actors”*: new Python +``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python processes which each run a ``trio`` scheduled task tree (also known as -an `async sandwich -`_ -- a call to ``trio.run()``). That is, each “*Actor*” is a new process -plus a ``trio`` runtime. +an `async sandwich`_ - a call to ``trio.run()``). That is, each +"*Actor*" is a new process plus a ``trio`` runtime. -“Actors” communicate by exchanging asynchronous `messages -`_ and avoid sharing -state. The intention of this model is to allow for highly distributed -software that, through the adherence to *structured concurrency*, -results in systems which fail in predictable and recoverable ways. +"Actors" communicate by exchanging asynchronous messages_ and avoid +sharing state. The intention of this model is to allow for highly +distributed software that, through the adherence to *structured +concurrency*, results in systems which fail in predictable and +recoverable ways. -The first step to grok ``tractor`` is to get the basics of ``trio`` -down. A great place to start is the `trio docs -`_ and this `blog post -`_. +The first step to grok ``tractor`` is to get the basics of ``trio`` down. +A great place to start is the `trio docs`_ and this `blog post`_. + +.. _messages: https://en.wikipedia.org/wiki/Message_passing +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts +.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony +.. _async generators: https://www.python.org/dev/peps/pep-0525/ Install -======= - +------- No PyPi release yet! :: - pip install git+git://github.com/goodboy/tractor.git + pip install git+git://github.com/goodboy/tractor.git Alluring Features -================= - -* **It’s just** ``trio``, but with SC applied to processes (aka - “actors”) - -* Infinitely nesteable process trees - -* Built-in API for inter-process streaming - -* A (first ever?) “native” multi-core debugger for Python using - `pdb++ `_ - -* (Soon to land) ``asyncio`` support allowing for “infected” actors - where *trio* drives the *asyncio* scheduler via the astounding - “`guest mode - `_” +----------------- +- **It's just** ``trio``, but with SC applied to processes (aka "actors") +- Infinitely nesteable process trees +- Built-in API for inter-process streaming +- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ +- (Soon to land) ``asyncio`` support allowing for "infected" actors where + `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" -The example you’re probably after… -================================== +Example: self-destruct a process tree +------------------------------------- -It seems the initial query from most new users is “how do I make a -worker pool thing?”. - -``tractor`` is built to handle any SC process tree you can imagine; -the “worker pool” pattern is a trivial special case: - -# TODO: workerpool example - -.. code:: +.. code:: python """ Run with a process monitor from a terminal using: @@ -127,13 +113,154 @@ the “worker pool” pattern is a trivial special case: print('Zombies Contained') -Feel like saying hi? -==================== +The example you're probably after... +------------------------------------ +It seems the initial query from most new users is "how do I make a worker +pool thing?". +``tractor`` is built to handle any SC process tree you can +imagine; the "worker pool" pattern is a trivial special case: + +.. code:: python + + """ + Demonstration of the prime number detector example from the + ``concurrent.futures`` docs: + + https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example + + This uses no extra threads, fancy semaphores or futures; all we need + is ``tractor``'s channels. + + """ + from contextlib import asynccontextmanager + from typing import List, Callable + import itertools + import math + import time + + import tractor + import trio + from async_generator import aclosing + + + PRIMES = [ + 112272535095293, + 112582705942171, + 112272535095293, + 115280095190773, + 115797848077099, + 1099726899285419, + ] + + + def is_prime(n): + if n < 2: + return False + if n == 2: + return True + if n % 2 == 0: + return False + + sqrt_n = int(math.floor(math.sqrt(n))) + for i in range(3, sqrt_n + 1, 2): + if n % i == 0: + return False + return True + + + @asynccontextmanager + async def worker_pool(workers=4): + """Though it's a trivial special case for ``tractor``, the well + known "worker pool" seems to be the defacto "but, I want this + process pattern!" for most parallelism pilgrims. + + Yes, the workers stay alive (and ready for work) until you close + the context. + """ + async with tractor.open_nursery() as tn: + + portals = [] + snd_chan, recv_chan = trio.open_memory_channel(len(PRIMES)) + + for i in range(workers): + + # this starts a new sub-actor (process + trio runtime) and + # stores it's "portal" for later use to "submit jobs" (ugh). + portals.append( + await tn.start_actor( + f'worker_{i}', + enable_modules=[__name__], + ) + ) + + async def _map( + worker_func: Callable[[int], bool], + sequence: List[int] + ) -> List[bool]: + + # define an async (local) task to collect results from workers + async def send_result(func, value, portal): + await snd_chan.send((value, await portal.run(func, n=value))) + + async with trio.open_nursery() as n: + + for value, portal in zip(sequence, itertools.cycle(portals)): + n.start_soon( + send_result, + worker_func, + value, + portal + ) + + # deliver results as they arrive + for _ in range(len(sequence)): + yield await recv_chan.receive() + + # deliver the parallel "worker mapper" to user code + yield _map + + # tear down all "workers" on pool close + await tn.cancel() + + + async def main(): + + async with worker_pool() as actor_map: + + start = time.time() + + async with aclosing(actor_map(is_prime, PRIMES)) as results: + async for number, prime in results: + + print(f'{number} is prime: {prime}') + + print(f'processing took {time.time() - start} seconds') + + + if __name__ == '__main__': + start = time.time() + trio.run(main) + print(f'script took {time.time() - start} seconds') + + +Feel like saying hi? +-------------------- This project is very much coupled to the ongoing development of ``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant -community). If you want to help, have suggestions or just want to say -hi, please feel free to reach us in our `matrix channel -`_. If matrix seems too hip, -we’re also mostly all in the the `trio gitter channel -`_! +community). If you want to help, have suggestions or just want to +say hi, please feel free to reach us in our `matrix channel`_. If +matrix seems too hip, we're also mostly all in the the `trio gitter +channel`_! + +.. _trio gitter channel: https://gitter.im/python-trio/general +.. _matrix channel: https://matrix.to/#/!tractor:matrix.org +.. _pdb++: https://github.com/pdbpp/pdbpp +.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops + + +.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square + :target: https://actions-badge.atrox.dev/goodboy/tractor/goto +.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest + :target: https://tractor.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status diff --git a/docs/_README.rst b/docs/_README.rst deleted file mode 100644 index 4300a84..0000000 --- a/docs/_README.rst +++ /dev/null @@ -1,103 +0,0 @@ -tractor -======= -The Python async-native multi-core system *you always wanted*. - - -|gh_actions| -|docs| - -.. _actor model: https://en.wikipedia.org/wiki/Actor_model -.. _trio: https://github.com/python-trio/trio -.. _multi-processing: https://en.wikipedia.org/wiki/Multiprocessing -.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles -.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich -.. _structured concurrent: https://trio.discourse.group/t/concise-definition-of-structured-concurrency/228 - - -``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. - -It is an attempt to pair trionic_ `structured concurrency`_ with -distributed Python. You can think of it as a ``trio`` -*-across-processes* or simply as an opinionated replacement for the -stdlib's ``multiprocessing`` but built on async programming primitives -from the ground up. - -Don't be scared off by this description. ``tractor`` **is just ``trio``** -but with nurseries for process management and cancel-able IPC. -If you understand how to work with ``trio``, ``tractor`` will give you -the parallelism you've been missing. - -``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python -processes which each run a ``trio`` scheduled task tree (also known as -an `async sandwich`_ - a call to ``trio.run()``). That is, each -"*Actor*" is a new process plus a ``trio`` runtime. - -"Actors" communicate by exchanging asynchronous messages_ and avoid -sharing state. The intention of this model is to allow for highly -distributed software that, through the adherence to *structured -concurrency*, results in systems which fail in predictable and -recoverable ways. - -The first step to grok ``tractor`` is to get the basics of ``trio`` down. -A great place to start is the `trio docs`_ and this `blog post`_. - -.. _messages: https://en.wikipedia.org/wiki/Message_passing -.. _trio docs: https://trio.readthedocs.io/en/latest/ -.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts -.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony -.. _async generators: https://www.python.org/dev/peps/pep-0525/ - - -Install -------- -No PyPi release yet! - -:: - - pip install git+git://github.com/goodboy/tractor.git - - -Alluring Features ------------------ -- **It's just** ``trio``, but with SC applied to processes (aka "actors") -- Infinitely nesteable process trees -- Built-in API for inter-process streaming -- A (first ever?) "native" multi-core debugger for Python using `pdb++`_ -- (Soon to land) ``asyncio`` support allowing for "infected" actors where - `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" - - -The example you're probably after... ------------------------------------- -It seems the initial query from most new users is "how do I make a worker -pool thing?". - -``tractor`` is built to handle any SC process tree you can -imagine; the "worker pool" pattern is a trivial special case: - - -# TODO: workerpool example - - -Feel like saying hi? --------------------- -This project is very much coupled to the ongoing development of -``trio`` (i.e. ``tractor`` gets most of its ideas from that brilliant -community). If you want to help, have suggestions or just want to -say hi, please feel free to reach us in our `matrix channel`_. If -matrix seems too hip, we're also mostly all in the the `trio gitter -channel`_! - -.. _trio gitter channel: https://gitter.im/python-trio/general -.. _matrix channel: https://matrix.to/#/!tractor:matrix.org -.. _pdb++: https://github.com/pdbpp/pdbpp -.. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops - - -.. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square - :target: https://actions-badge.atrox.dev/goodboy/tractor/goto -.. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest - :target: https://tractor.readthedocs.io/en/latest/?badge=latest - :alt: Documentation Status diff --git a/docs/github_readme/_sphinx_readme.rst b/docs/github_readme/_sphinx_readme.rst index 03b09d5..f277af1 100644 --- a/docs/github_readme/_sphinx_readme.rst +++ b/docs/github_readme/_sphinx_readme.rst @@ -69,6 +69,12 @@ Alluring Features `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" +Example: self-destruct a process tree +------------------------------------- +.. literalinclude:: ../../examples/parallelism/we_are_processes.py + :language: python + + The example you're probably after... ------------------------------------ It seems the initial query from most new users is "how do I make a worker @@ -77,9 +83,8 @@ pool thing?". ``tractor`` is built to handle any SC process tree you can imagine; the "worker pool" pattern is a trivial special case: -# TODO: workerpool example - -.. literalinclude:: ../../examples/parallelism/we_are_processes.py +.. literalinclude:: ../../examples/parallelism/concurrent_actors_primes.py + :language: python Feel like saying hi? diff --git a/docs/github_readme/conf.py b/docs/github_readme/conf.py index 04d288e..2142785 100644 --- a/docs/github_readme/conf.py +++ b/docs/github_readme/conf.py @@ -35,9 +35,9 @@ release = '0.0.0a0.dev0' # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - # 'sphinx.ext.autodoc', - # 'sphinx.ext.intersphinx', - # 'sphinx.ext.todo', + 'sphinx.ext.autodoc', + 'sphinx.ext.intersphinx', + 'sphinx.ext.todo', 'sphinxcontrib.restbuilder', ] diff --git a/docs/mk_gh_readme.sh b/docs/mk_gh_readme.sh index 3836e2f..6f12dbe 100755 --- a/docs/mk_gh_readme.sh +++ b/docs/mk_gh_readme.sh @@ -1,4 +1,4 @@ #!/bin/bash sphinx-build -b rst ./github_readme ./ -mv _sphinx_readme.rst README.rst +mv _sphinx_readme.rst _README.rst From 4a512bc8799ea554bc800c5ff321c9066e80a5a0 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Mon, 22 Feb 2021 10:50:48 -0500 Subject: [PATCH 05/11] Compress terminal cmd line lens --- examples/parallelism/we_are_processes.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/examples/parallelism/we_are_processes.py b/examples/parallelism/we_are_processes.py index 8283b9c..9585ad2 100644 --- a/examples/parallelism/we_are_processes.py +++ b/examples/parallelism/we_are_processes.py @@ -1,6 +1,9 @@ """ -Run with a process monitor from a terminal using: -$TERM -e watch -n 0.1 "pstree -a $$" & python examples/parallelism/we_are_processes.py && kill $! +Run with a process monitor from a terminal using:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/we_are_processes.py \ + && kill $! """ from multiprocessing import cpu_count From 71a35aadef3682aff68ac43e5819c78d7d5e3da7 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Mon, 22 Feb 2021 13:35:22 -0500 Subject: [PATCH 06/11] Drop worker pool and add debug example --- docs/README.rst | 190 ++++++++++++++++++------------------------------ 1 file changed, 71 insertions(+), 119 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index 5bf7952..ba71a1c 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -71,28 +71,32 @@ Alluring Features Example: self-destruct a process tree ------------------------------------- +``tractor`` protects you from zombies, no matter what. .. code:: python - """ - Run with a process monitor from a terminal using: - $TERM -e watch -n 0.1 "pstree -a $$" & python examples/parallelism/we_are_processes.py && kill $! + """ + Run with a process monitor from a terminal using:: - """ - from multiprocessing import cpu_count - import os + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/we_are_processes.py \ + && kill $! - import tractor - import trio + """ + from multiprocessing import cpu_count + import os + + import tractor + import trio - async def target(): + async def target(): print(f"Yo, i'm '{tractor.current_actor().name}' " f"running in pid {os.getpid()}") await trio.sleep_forever() - async def main(): + async def main(): async with tractor.open_nursery() as n: @@ -106,7 +110,7 @@ Example: self-destruct a process tree raise Exception('Self Destructed') - if __name__ == '__main__': + if __name__ == '__main__': try: trio.run(main) except Exception: @@ -115,133 +119,81 @@ Example: self-destruct a process tree The example you're probably after... ------------------------------------ -It seems the initial query from most new users is "how do I make a worker +It seems the initial ask from most new users is "how do I make a worker pool thing?". ``tractor`` is built to handle any SC process tree you can -imagine; the "worker pool" pattern is a trivial special case: +imagine; the "worker pool" pattern is a trivial special case. + +We have a `full re-implementation `_ of the std-lib's +``concurrent.futures.ProcessPoolExecutor`` example for reference. + +You can run it like so (from this dir) to see the process tree in +real time:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/concurrent_actors_primes.py \ + && kill $! + +This uses no extra threads, fancy semaphores or futures; all we need +is ``tractor``'s IPC! + + +.. _concurrent_actors_primes: https://github.com/goodboy/tractor/blob/readme_pump/examples/parallelism/concurrent_actors_primes.py + + +"Native" sub-process debugging +------------------------------ +Using the magic of `pdb++`_ and some IPC tricks we've +been able to create a native feeling debugging experience for +any (sub)-process in your ``tractor`` tree. .. code:: python - """ - Demonstration of the prime number detector example from the - ``concurrent.futures`` docs: + from os import getpid - https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor-example - - This uses no extra threads, fancy semaphores or futures; all we need - is ``tractor``'s channels. - - """ - from contextlib import asynccontextmanager - from typing import List, Callable - import itertools - import math - import time - - import tractor - import trio - from async_generator import aclosing + import tractor + import trio - PRIMES = [ - 112272535095293, - 112582705942171, - 112272535095293, - 115280095190773, - 115797848077099, - 1099726899285419, - ] + async def breakpoint_forever(): + "Indefinitely re-enter debugger in child actor." + while True: + yield 'yo' + await tractor.breakpoint() - def is_prime(n): - if n < 2: - return False - if n == 2: - return True - if n % 2 == 0: - return False - - sqrt_n = int(math.floor(math.sqrt(n))) - for i in range(3, sqrt_n + 1, 2): - if n % i == 0: - return False - return True + async def name_error(): + "Raise a ``NameError``" + getattr(doggypants) - @asynccontextmanager - async def worker_pool(workers=4): - """Though it's a trivial special case for ``tractor``, the well - known "worker pool" seems to be the defacto "but, I want this - process pattern!" for most parallelism pilgrims. + async def main(): + """Test breakpoint in a streaming actor. + """ + async with tractor.open_nursery( + debug_mode=True, + loglevel='error', + ) as n: - Yes, the workers stay alive (and ready for work) until you close - the context. - """ - async with tractor.open_nursery() as tn: + p0 = await n.start_actor('bp_forever', enable_modules=[__name__]) + p1 = await n.start_actor('name_error', enable_modules=[__name__]) - portals = [] - snd_chan, recv_chan = trio.open_memory_channel(len(PRIMES)) - - for i in range(workers): - - # this starts a new sub-actor (process + trio runtime) and - # stores it's "portal" for later use to "submit jobs" (ugh). - portals.append( - await tn.start_actor( - f'worker_{i}', - enable_modules=[__name__], - ) - ) - - async def _map( - worker_func: Callable[[int], bool], - sequence: List[int] - ) -> List[bool]: - - # define an async (local) task to collect results from workers - async def send_result(func, value, portal): - await snd_chan.send((value, await portal.run(func, n=value))) - - async with trio.open_nursery() as n: - - for value, portal in zip(sequence, itertools.cycle(portals)): - n.start_soon( - send_result, - worker_func, - value, - portal - ) - - # deliver results as they arrive - for _ in range(len(sequence)): - yield await recv_chan.receive() - - # deliver the parallel "worker mapper" to user code - yield _map - - # tear down all "workers" on pool close - await tn.cancel() + # retreive results + stream = await p0.run(breakpoint_forever) + await p1.run(name_error) - async def main(): - - async with worker_pool() as actor_map: - - start = time.time() - - async with aclosing(actor_map(is_prime, PRIMES)) as results: - async for number, prime in results: - - print(f'{number} is prime: {prime}') - - print(f'processing took {time.time() - start} seconds') + if __name__ == '__main__': + trio.run(main) - if __name__ == '__main__': - start = time.time() - trio.run(main) - print(f'script took {time.time() - start} seconds') +You can run this with:: + + >>> python examples/debugging/multi_daemon_subactors.py + +And, yes, there's a built-in crash handling mode B) +We're hoping to add a respawn-from-repl system soon! Feel like saying hi? From 0c8f9dbce03472b8579f17732b43882e07e074cf Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Tue, 23 Feb 2021 06:48:32 -0500 Subject: [PATCH 07/11] Add comma --- docs/README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/README.rst b/docs/README.rst index ba71a1c..fbccd19 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -144,7 +144,7 @@ is ``tractor``'s IPC! "Native" sub-process debugging ------------------------------ -Using the magic of `pdb++`_ and some IPC tricks we've +Using the magic of `pdb++`_ and some IPC tricks, we've been able to create a native feeling debugging experience for any (sub)-process in your ``tractor`` tree. From f342c3a0c4c171f41771c99c5bba7188bb51086d Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Tue, 23 Feb 2021 22:23:31 -0500 Subject: [PATCH 08/11] Attempt to add logo --- docs/README.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index fbccd19..a6c724e 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,7 +1,4 @@ -tractor -======= -The Python async-native multi-core system *you always wanted*. - +|logo| ``tractor``: next-gen Python parallelism |gh_actions| |docs| @@ -216,3 +213,7 @@ channel`_! .. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest :target: https://tractor.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status + +.. |logo| image:: _static/tractor_logo_side.svg + :width: 200 + :align: middle From 8ee9007798b478ce8f71adfddb895dcabcc9ad64 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Tue, 23 Feb 2021 23:42:48 -0500 Subject: [PATCH 09/11] Reorg and rejig flow Thanks to @richardsheridan for many suggestions! --- docs/README.rst | 162 ++++++++++++----------- examples/parallelism/we_are_processes.py | 7 +- 2 files changed, 92 insertions(+), 77 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index a6c724e..12c4bd7 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -13,62 +13,27 @@ ``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. -It is an attempt to pair trionic_ `structured concurrency`_ with -distributed Python. You can think of it as a ``trio`` -*-across-processes* or simply as an opinionated replacement for the -stdlib's ``multiprocessing`` but built on async programming primitives -from the ground up. +We pair structured concurrency with true multi-core Python parallelism; +the aim is to be the multi-processing framework *you always wanted*. -Don't be scared off by this description. ``tractor`` **is just ``trio``** -but with nurseries for process management and cancel-able IPC. -If you understand how to work with ``trio``, ``tractor`` will give you -the parallelism you've been missing. - -``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python -processes which each run a ``trio`` scheduled task tree (also known as -an `async sandwich`_ - a call to ``trio.run()``). That is, each -"*Actor*" is a new process plus a ``trio`` runtime. - -"Actors" communicate by exchanging asynchronous messages_ and avoid -sharing state. The intention of this model is to allow for highly -distributed software that, through the adherence to *structured -concurrency*, results in systems which fail in predictable and -recoverable ways. - -The first step to grok ``tractor`` is to get the basics of ``trio`` down. +The first steps to grok ``tractor`` is to get the basics of ``trio`` down. A great place to start is the `trio docs`_ and this `blog post`_. -.. _messages: https://en.wikipedia.org/wiki/Message_passing -.. _trio docs: https://trio.readthedocs.io/en/latest/ -.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ -.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts -.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony -.. _async generators: https://www.python.org/dev/peps/pep-0525/ - -Install -------- -No PyPi release yet! - -:: - - pip install git+git://github.com/goodboy/tractor.git - - -Alluring Features ------------------ -- **It's just** ``trio``, but with SC applied to processes (aka "actors") +Features +-------- +- **It's just** a ``trio`` API - Infinitely nesteable process trees -- Built-in API for inter-process streaming +- Built-in APIs for inter-process streaming - A (first ever?) "native" multi-core debugger for Python using `pdb++`_ - (Soon to land) ``asyncio`` support allowing for "infected" actors where `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" +- (Soon to land) typed messaging protocols (ex. via ``msgspec``) -Example: self-destruct a process tree -------------------------------------- -``tractor`` protects you from zombies, no matter what. +Zombie safe: self-destruct a process tree +----------------------------------------- +``tractor`` (attempts to) protect from zombies, no matter what. .. code:: python @@ -88,8 +53,11 @@ Example: self-destruct a process tree async def target(): - print(f"Yo, i'm '{tractor.current_actor().name}' " - f"running in pid {os.getpid()}") + print( + f"Yo, i'm '{tractor.current_actor().name}' " + f"running in pid {os.getpid()}" + ) + await trio.sleep_forever() @@ -114,34 +82,13 @@ Example: self-destruct a process tree print('Zombies Contained') -The example you're probably after... ------------------------------------- -It seems the initial ask from most new users is "how do I make a worker -pool thing?". - -``tractor`` is built to handle any SC process tree you can -imagine; the "worker pool" pattern is a trivial special case. - -We have a `full re-implementation `_ of the std-lib's -``concurrent.futures.ProcessPoolExecutor`` example for reference. - -You can run it like so (from this dir) to see the process tree in -real time:: - - $TERM -e watch -n 0.1 "pstree -a $$" \ - & python examples/parallelism/concurrent_actors_primes.py \ - && kill $! - -This uses no extra threads, fancy semaphores or futures; all we need -is ``tractor``'s IPC! +If you can create zombie child processes (without using a system signal) +it **is a bug**. -.. _concurrent_actors_primes: https://github.com/goodboy/tractor/blob/readme_pump/examples/parallelism/concurrent_actors_primes.py - - -"Native" sub-process debugging ------------------------------- -Using the magic of `pdb++`_ and some IPC tricks, we've +"Native" multi-process debugging +-------------------------------- +Using the magic of `pdb++`_ and our internal IPC, we've been able to create a native feeling debugging experience for any (sub)-process in your ``tractor`` tree. @@ -190,9 +137,66 @@ You can run this with:: >>> python examples/debugging/multi_daemon_subactors.py And, yes, there's a built-in crash handling mode B) + We're hoping to add a respawn-from-repl system soon! +Worker poolz are easy peasy +--------------------------- +It seems the initial ask from most new users is "how do I make a worker +pool thing?". + +``tractor`` is built to handle any SC process tree you can +imagine; the "worker pool" pattern is a trivial special case. + +We have a `full re-implementation `_ of the std-lib's +``concurrent.futures.ProcessPoolExecutor`` example for reference. + +You can run it like so (from this dir) to see the process tree in +real time:: + + $TERM -e watch -n 0.1 "pstree -a $$" \ + & python examples/parallelism/concurrent_actors_primes.py \ + && kill $! + +This uses no extra threads, fancy semaphores or futures; all we need +is ``tractor``'s IPC! + + +.. _concurrent_actors_primes: https://github.com/goodboy/tractor/examples/parallelism/concurrent_actors_primes.py + +Install +------- +No PyPi release yet! + +:: + + pip install git+git://github.com/goodboy/tractor.git + + +Under the hood +-------------- +``tractor`` is an attempt to pair trionic_ `structured concurrency`_ with +distributed Python. You can think of it as a ``trio`` +*-across-processes* or simply as an opinionated replacement for the +stdlib's ``multiprocessing`` but built on async programming primitives +from the ground up. + +``tractor``'s nurseries let you spawn ``trio`` *"actors"*: new Python +processes which each run a ``trio`` scheduled runtime - a call to ``trio.run()``. + +Don't be scared off by this description. ``tractor`` **is just** ``trio`` +but with nurseries for process management and cancel-able streaming IPC. +If you understand how to work with ``trio``, ``tractor`` will give you +the parallelism you've been missing. + +"Actors" communicate by exchanging asynchronous messages_ and avoid +sharing state. The intention of this model is to allow for highly +distributed software that, through the adherence to *structured +concurrency*, results in systems which fail in predictable and +recoverable ways. + + Feel like saying hi? -------------------- This project is very much coupled to the ongoing development of @@ -206,14 +210,22 @@ channel`_! .. _matrix channel: https://matrix.to/#/!tractor:matrix.org .. _pdb++: https://github.com/pdbpp/pdbpp .. _guest mode: https://trio.readthedocs.io/en/stable/reference-lowlevel.html?highlight=guest%20mode#using-guest-mode-to-run-trio-on-top-of-other-event-loops +.. _messages: https://en.wikipedia.org/wiki/Message_passing +.. _trio docs: https://trio.readthedocs.io/en/latest/ +.. _blog post: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _structured concurrency: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ +.. _3 axioms: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concepts +.. _unrequirements: https://en.wikipedia.org/wiki/Actor_model#Direct_communication_and_asynchrony +.. _async generators: https://www.python.org/dev/peps/pep-0525/ .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fgoodboy%2Ftractor%2Fbadge&style=popout-square :target: https://actions-badge.atrox.dev/goodboy/tractor/goto + .. |docs| image:: https://readthedocs.org/projects/tractor/badge/?version=latest :target: https://tractor.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status .. |logo| image:: _static/tractor_logo_side.svg - :width: 200 + :width: 250 :align: middle diff --git a/examples/parallelism/we_are_processes.py b/examples/parallelism/we_are_processes.py index 9585ad2..755b89c 100644 --- a/examples/parallelism/we_are_processes.py +++ b/examples/parallelism/we_are_processes.py @@ -14,8 +14,11 @@ import trio async def target(): - print(f"Yo, i'm '{tractor.current_actor().name}' " - f"running in pid {os.getpid()}") + print( + f"Yo, i'm '{tractor.current_actor().name}' " + f"running in pid {os.getpid()}" + ) + await trio.sleep_forever() From ab192741cec958ffc6d275c6d86da958445a73ea Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Wed, 24 Feb 2021 09:12:43 -0500 Subject: [PATCH 10/11] Fix code indent and worker pool link --- docs/README.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index 12c4bd7..9ccb3ac 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -63,23 +63,23 @@ Zombie safe: self-destruct a process tree async def main(): - async with tractor.open_nursery() as n: + async with tractor.open_nursery() as n: - for i in range(cpu_count()): - await n.run_in_actor(target, name=f'worker_{i}') + for i in range(cpu_count()): + await n.run_in_actor(target, name=f'worker_{i}') - print('This process tree will self-destruct in 1 sec...') - await trio.sleep(1) + print('This process tree will self-destruct in 1 sec...') + await trio.sleep(1) - # you could have done this yourself - raise Exception('Self Destructed') + # you could have done this yourself + raise Exception('Self Destructed') if __name__ == '__main__': - try: - trio.run(main) - except Exception: - print('Zombies Contained') + try: + trio.run(main) + except Exception: + print('Zombies Contained') If you can create zombie child processes (without using a system signal) @@ -149,7 +149,7 @@ pool thing?". ``tractor`` is built to handle any SC process tree you can imagine; the "worker pool" pattern is a trivial special case. -We have a `full re-implementation `_ of the std-lib's +We have a `full re-implementation `_ of the std-lib's ``concurrent.futures.ProcessPoolExecutor`` example for reference. You can run it like so (from this dir) to see the process tree in @@ -163,7 +163,7 @@ This uses no extra threads, fancy semaphores or futures; all we need is ``tractor``'s IPC! -.. _concurrent_actors_primes: https://github.com/goodboy/tractor/examples/parallelism/concurrent_actors_primes.py +.. _worker_pool: https://github.com/goodboy/tractor/blob/master/examples/parallelism/concurrent_actors_primes.py Install ------- From 0679eb564663916c8506a471e0f34d363a217116 Mon Sep 17 00:00:00 2001 From: Tyler Goodlet Date: Wed, 24 Feb 2021 19:11:05 -0500 Subject: [PATCH 11/11] Further simplifications --- docs/README.rst | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/README.rst b/docs/README.rst index 9ccb3ac..ed19b83 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -13,10 +13,10 @@ ``tractor`` is a `structured concurrent`_ "`actor model`_" built on trio_ and multi-processing_. -We pair structured concurrency with true multi-core Python parallelism; -the aim is to be the multi-processing framework *you always wanted*. +We pair structured concurrency and true multi-core parallelism with +the aim of being the multi-processing framework *you always wanted*. -The first steps to grok ``tractor`` is to get the basics of ``trio`` down. +The first step to grok ``tractor`` is to get the basics of ``trio`` down. A great place to start is the `trio docs`_ and this `blog post`_. @@ -26,14 +26,14 @@ Features - Infinitely nesteable process trees - Built-in APIs for inter-process streaming - A (first ever?) "native" multi-core debugger for Python using `pdb++`_ -- (Soon to land) ``asyncio`` support allowing for "infected" actors where - `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" -- (Soon to land) typed messaging protocols (ex. via ``msgspec``) +- Support for multiple process spawning backends +- A modular transport layer, allowing for custom serialization, + communications protocols, and environment specific IPC primitives Zombie safe: self-destruct a process tree ----------------------------------------- -``tractor`` (attempts to) protect from zombies, no matter what. +``tractor`` tries to protect you from zombies, no matter what. .. code:: python @@ -90,7 +90,7 @@ it **is a bug**. -------------------------------- Using the magic of `pdb++`_ and our internal IPC, we've been able to create a native feeling debugging experience for -any (sub)-process in your ``tractor`` tree. +any (sub-)process in your ``tractor`` tree. .. code:: python @@ -143,11 +143,12 @@ We're hoping to add a respawn-from-repl system soon! Worker poolz are easy peasy --------------------------- -It seems the initial ask from most new users is "how do I make a worker -pool thing?". +The initial ask from most new users is *"how do I make a worker +pool thing?"*. -``tractor`` is built to handle any SC process tree you can -imagine; the "worker pool" pattern is a trivial special case. +``tractor`` is built to handle any SC (structured concurrent) process +tree you can imagine; a "worker pool" pattern is a trivial special +case. We have a `full re-implementation `_ of the std-lib's ``concurrent.futures.ProcessPoolExecutor`` example for reference. @@ -169,7 +170,7 @@ Install ------- No PyPi release yet! -:: +From git:: pip install git+git://github.com/goodboy/tractor.git @@ -197,6 +198,16 @@ concurrency*, results in systems which fail in predictable and recoverable ways. +What's on the TODO: +------------------- +Help us push toward the future. + +- (Soon to land) ``asyncio`` support allowing for "infected" actors where + `trio` drives the `asyncio` scheduler via the astounding "`guest mode`_" +- Typed messaging protocols (ex. via ``msgspec``) +- Erlang-style supervisors via composed context managers + + Feel like saying hi? -------------------- This project is very much coupled to the ongoing development of