forked from goodboy/tractor
1
0
Fork 0

Add simpler intro

As per a suggestion from @njsmith I've added a little less verbose intro
which mentions *actors* up front.

More,
- add install cmd
- reorg spawning and portal sections a wee bit
init_docs
Tyler Goodlet 2018-07-12 15:18:02 -04:00
parent 760ce362bf
commit 049513235d
1 changed files with 62 additions and 32 deletions

View File

@ -5,11 +5,12 @@ A minimalist `actor model`_ built on multiprocessing_ and trio_.
``tractor`` is an attempt to take trionic_ concurrency concepts and apply ``tractor`` is an attempt to take trionic_ concurrency concepts and apply
them to distributed-multicore Python. them to distributed-multicore Python.
``tractor`` lets you run or spawn Python processes which each internally ``tractor`` lets you run and spawn Python *actors*: separate processes which are internally
run a single ``trio`` task tree (also known as an `async sandwich`_) and running a ``trio`` scheduler and task tree (also known as an `async sandwich`_).
which can communicate with each other over channels_ using a transparent
async function calling API called *portals* (a name also borrowed_ Actors communicate with each other by sending *messages* over channels_, but the details of this
from ``trio``). in ``tractor`` is by default hidden and *actors* can instead easily invoke remote asynchronous
functions using *portals*.
``tractor``'s tenets non-comprehensively include: ``tractor``'s tenets non-comprehensively include:
@ -25,18 +26,25 @@ from ``trio``).
Expect nothing to be set in stone and your ideas about where it should go Expect nothing to be set in stone and your ideas about where it should go
to be greatly appreciated! to be greatly appreciated!
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _trio: https://github.com/python-trio/trio
.. _multiprocessing: https://docs.python.org/3/library/multiprocessing.html
.. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles .. _trionic: https://trio.readthedocs.io/en/latest/design.html#high-level-design-principles
.. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich .. _async sandwich: https://trio.readthedocs.io/en/latest/tutorial.html#async-sandwich
.. _actor model: https://en.wikipedia.org/wiki/Actor_model
.. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate .. _always propagate: https://trio.readthedocs.io/en/latest/design.html#exceptions-always-propagate
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
.. _multiprocessing: https://docs.python.org/3/library/multiprocessing.html
.. _trio: https://github.com/python-trio/trio
.. _channels: https://en.wikipedia.org/wiki/Channel_(programming)
.. _borrowed:
https://trio.readthedocs.io/en/latest/reference-core.html#getting-back-into-the-trio-thread-from-another-thread
.. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker .. _causality: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#c-c-c-c-causality-breaker
.. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture .. _shared nothing architecture: https://en.wikipedia.org/wiki/Shared-nothing_architecture
.. _cancellation: https://trio.readthedocs.io/en/latest/reference-core.html#cancellation-and-timeouts
.. _channels: https://en.wikipedia.org/wiki/Channel_(programming)
Install
-------
No PyPi release yet!
::
pip install git+git://github.com/tgoodlet/tractor.git
What's this? Spawning event loops in subprocesses? What's this? Spawning event loops in subprocesses?
@ -60,8 +68,7 @@ and of course does **not** use ``asyncio``, hence **no** event loops.
A trynamic first scene A trynamic first scene
---------------------- ----------------------
As a first example let's spawn a couple actors (in separate processes) As a first example let's spawn a couple *actors* and have them run their lines:
and have them run their lines:
.. code:: python .. code:: python
@ -129,8 +136,7 @@ similar to the nursery_ in ``trio``. That is, an ``ActorNursery``
created with ``tractor.open_nursery()`` waits on spawned sub-actors to created with ``tractor.open_nursery()`` waits on spawned sub-actors to
complete (or error) in the same causal_ way ``trio`` waits on spawned complete (or error) in the same causal_ way ``trio`` waits on spawned
subtasks. This includes errors from any one sub-actor causing all other subtasks. This includes errors from any one sub-actor causing all other
actors spawned by the nursery to be cancelled_. Eventually ``tractor`` actors spawned by the nursery to be cancelled_.
plans to support different `supervision strategies`_ like ``erlang``.
To spawn an actor open a *nursery block* and use the ``start_actor()`` To spawn an actor open a *nursery block* and use the ``start_actor()``
method: method:
@ -164,11 +170,8 @@ method:
# "outlive_main" actor it will never end until cancelled # "outlive_main" actor it will never end until cancelled
await portal.cancel_actor() await portal.cancel_actor()
Notice the ``rpc_module_paths`` `kwarg` here, it's a list of module path Notice the ``portal`` instance returned from ``nursery.start_actor()``,
strings that will be loaded and made accessible for execution in the we'll get to that shortly.
remote actor. For now this is a simple mechanism to restrict the
functionality of the remote actor and uses Python's module system to
define the allowed remote function namespace(s).
Spawned actor lifetimes can be configured in one of two ways: Spawned actor lifetimes can be configured in one of two ways:
@ -197,8 +200,26 @@ Had we wanted the former in our example it would have been much simpler:
print(await portal.result()) print(await portal.result())
Note that the main task's *final result(s)* is **always** accessed using Note that the main task's *final result(s)* (returned from the provided
``Portal.result()``. ``main`` function) is **always** accessed using ``Portal.result()`` much
like you'd expect from a future_.
The ``rpc_module_paths`` `kwarg` above is a list of module path
strings that will be loaded and made accessible for execution in the
remote actor through a call to ``Portal.run()``. For now this is
a simple mechanism to restrict the functionality of the remote
(daemonized) actor and uses Python's module system to limit the
allowed remote function namespace(s).
``tractor`` is opinionated about the underlying threading model used for
each actor. Since Python has a GIL and an actor model by definition
shares no state, there is no reason to use anything other then a
multiprocessing_ ``Process`` for execution. This makes ``tractor``
programs able leverage not only multi-core hardware but also distribute
over many hardware hosts (each *actor* can talk to all others with ease
over standard network protocols).
Eventually ``tractor`` plans to support different `supervision strategies`_ like ``erlang``.
.. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning .. _nursery: https://trio.readthedocs.io/en/latest/reference-core.html#nurseries-and-spawning
.. _supervision strategies: http://erlang.org/doc/man/supervisor.html#sup_flags .. _supervision strategies: http://erlang.org/doc/man/supervisor.html#sup_flags
@ -208,27 +229,32 @@ Note that the main task's *final result(s)* is **always** accessed using
Transparent function calling using *portals* Transparent function calling using *portals*
-------------------------------------------- --------------------------------------------
``tractor`` currently is experimenting with an *async-native* ``tractor`` introdces the concept of a *portal* which is an API
IPC API where routines that are invoked in remote *actors* are treated borrowed_ from ``trio``. A portal may seems similar to the idea of
as though they were invoked locally in the calling actor. So when you a RPC future_ except a *portal* allows invoking remote *async* functions and
see a call to ``await portal.run()`` what you get back is what you'd expect generators and intermittently blocking to receive responses. This allows
for fully async-native IPC between actors.
When you invoke another actor's routines using a *portal* it looks as though
it was called locally in the current actor. So when you see a call to
``await portal.run()`` what you get back is what you'd expect
to if you'd called the function directly in-process. This approach avoids to if you'd called the function directly in-process. This approach avoids
the need to add any special RPC *proxy* objects to the library by instead just the need to add any special RPC *proxy* objects to the library by instead just
relying on the built-in (async) function calling semantics and protocols of Python. relying on the built-in (async) function calling semantics and protocols of Python.
Depending on the function type ``Portal.run()`` tries to Depending on the function type ``Portal.run()`` tries to
correctly interface exactly like a local version of the remote correctly interface exactly like a local version of the remote
built-in Python function type. Currently async functions, generators, built-in Python *function type*. Currently async functions, generators,
and regular functions are supported. Inspiration for this API comes and regular functions are supported. Inspiration for this API comes
from the way execnet_ does `remote function execution`_ but without from the way execnet_ does `remote function execution`_ but without
the client code (necessarily) having to worry about the underlying the client code (necessarily) having to worry about the underlying
*channel* API. channels_ system or shipping code over the network.
This *portal* approach turns out to be paricularly exciting with the This *portal* approach turns out to be paricularly exciting with the
introduction of `asynchronous generators`_ in Python 3.6! It means that introduction of `asynchronous generators`_ in Python 3.6! It means that
actors can compose nicely in a data processing pipeline. actors can compose nicely in a data processing pipeline.
Say you wanted to spawn two actors which each pulled data feeds from Say you wanted to spawn two actors which each pulling data feeds from
two different sources (and wanted this work spread across 2 cpus). two different sources (and wanted this work spread across 2 cpus).
You also want to aggregate these feeds, do some processing on them and then You also want to aggregate these feeds, do some processing on them and then
deliver the final result stream to a client (or in this case parent) deliver the final result stream to a client (or in this case parent)
@ -335,11 +361,13 @@ aggregating values from those two in ``aggregate()`` and shipping the
single stream of unique values up the parent actor (the ``'MainProcess'`` single stream of unique values up the parent actor (the ``'MainProcess'``
as ``multiprocessing`` calls it) which is running ``main()``. as ``multiprocessing`` calls it) which is running ``main()``.
There has also been some discussion about adding support for reactive There has also been some discussion about adding support for reactive
programming primitives and native support for asyncitertools_ like libs - programming primitives and native support for asyncitertools_ like libs -
so keep an eye out for that! so keep an eye out for that!
.. _future: https://en.wikipedia.org/wiki/Futures_and_promises
.. _borrowed:
https://trio.readthedocs.io/en/latest/reference-core.html#getting-back-into-the-trio-thread-from-another-thread
.. _asynchronous generators: https://www.python.org/dev/peps/pep-0525/ .. _asynchronous generators: https://www.python.org/dev/peps/pep-0525/
.. _remote function execution: https://codespeak.net/execnet/example/test_info.html#remote-exec-a-function-avoiding-inlined-source-part-i .. _remote function execution: https://codespeak.net/execnet/example/test_info.html#remote-exec-a-function-avoiding-inlined-source-part-i
.. _asyncitertools: https://github.com/vodik/asyncitertools .. _asyncitertools: https://github.com/vodik/asyncitertools
@ -379,6 +407,8 @@ Cancellation
tractor.run(main) tractor.run(main)
Cancelling a nursery block cancels all actors spawned by it.
Remote error propagation Remote error propagation
------------------------ ------------------------
@ -517,7 +547,7 @@ what exactly it's doing (even at the lowest levels). Luckily ``tractor`` has
tons of logging throughout the core. ``tractor`` isn't opinionated on tons of logging throughout the core. ``tractor`` isn't opinionated on
how you use this information and users are expected to consume log messages in how you use this information and users are expected to consume log messages in
whichever way is appropriate for the system at hand. That being said, when hacking whichever way is appropriate for the system at hand. That being said, when hacking
on ``tractor`` there is a prettified console formatted which you can enable to on ``tractor`` there is a prettified console formatter which you can enable to
see what the heck is going on. Just put the following somewhere in your code: see what the heck is going on. Just put the following somewhere in your code:
.. code:: python .. code:: python