forked from goodboy/tractor
1
0
Fork 0
Commit Graph

713 Commits (b7b0b7d3fd6212d208d31372a4c2af3ffe086d52)

Author SHA1 Message Date
Tyler Goodlet 7017f68503 3.6.3 fix - missing older attr 2018-08-04 18:15:24 -04:00
Tyler Goodlet 50517c9488 Manage a `multiprocessing.forkserver` manually
Start a forkserver once in the main (parent-most) process
and pass ipc info (fds) to subprocesses manually such that embedded
calls to `multiprocessing.Process.start()` just work. Note that this
relies on our overridden version of the stdlib's
`multiprocessing.forkserver` module.

Resolves #6
2018-08-04 18:15:24 -04:00
Tyler Goodlet f46d5b2b62 Hackery to override the stdlib's forkserver
The stdlib insists on creating multiple forkservers and semaphore trackers
for each sub-sub-process launched. This isn't ideal since it costs each
`tractor` sub-actor an additional 2 more processes then necessary and is
confusing when viewed as a process tree (eg. via `pstree`).

The majority of the change is simply avoiding the call to
`forkserver.ensure_running()` and `semaphore_tracker.ensure_running()`
in `ForkServer.connect_new_process()` and instead treating the user like
an adult and expecting those calls to be made *once* in the parent most
process (i.e. what `multiprocessing` calls the `MainProcess`).

Really a proper patch should be made against cpython which allows for
similar manual management of the server along with a mechanism to communicate
forkserver and semaphore tracker fd info to sub-processes such that
further calls to `Process.start()` work as expected.

Relates to #6
2018-08-04 18:15:24 -04:00
Tyler Goodlet d6d7fea708 Use plain func for __aiter__() 2018-08-04 18:15:24 -04:00
goodboy 140956aedb
Merge pull request #25 from tgoodlet/drop_main_kwarg
Drop the "main" task via kwarg idea
2018-08-04 18:13:44 -04:00
Tyler Goodlet f7706074a2 Drop needless if check 2018-08-04 18:10:31 -04:00
Tyler Goodlet 1da69b1396 Allow daemonizing top level actor; don't require main func 2018-08-03 09:41:18 -04:00
Tyler Goodlet db08dbad3b Streaming is too fast, cancel sooner 2018-08-02 16:33:42 -04:00
Tyler Goodlet 41914f3118 Drop console logging 2018-08-02 16:29:01 -04:00
Tyler Goodlet dadea87451 Update test to new api 2018-08-02 15:27:09 -04:00
Tyler Goodlet bb13b79df5 Drop the "main" task via kwarg idea
Stop worrying about a "main task" in each actor and instead add an
additional `ActorNursery.run_in_actor()` method which wraps calls
to create an actor and run a lone RPC task inside it. Note this
adjusts the public API of `ActorNursery.start_actor()` to drop
its `main` kwarg.

The dirty deats of making this possible:
- each spawned RPC task is now tracked with a specific cancel scope such
  that when the actor is cancelled all ongoing responders are cancelled
  before any IPC/channel machinery is closed (turns out that spawning
  new actors from `outlive_main=True` actors was probably borked before
  finally getting this working).
- make each initial RPC response be a packet which describes the
  `functype` (eg. `{'functype': 'asyncfunction'}`) allowing for async
  calls/submissions by client actors (this was required to make
  `run_in_actor()` work - `Portal._submit()` is the new async method).
- hooray we can stop faking "main task" results for daemon actors
- add better handling/raising of internal errors caught in the bowels of
  the `Actor` itself.
- drop the rpc spawning nursery; just use the `Actor._root_nursery`
- only wait on `_no_more_peers` if there are existing peer channels that
  are actually still connected.
- an `ActorNursery.__aexit__()` now implicitly waits on `Portal.result()` on close
  for each `run_in_actor()` spawned actor.
- handle cancelling partial started actors which haven't yet connected
  back to the parent

Resolves #24
2018-08-02 15:24:28 -04:00
goodboy f726bd81da
Merge pull request #15 from tgoodlet/reorg
Reorg everything into private modules
2018-07-17 23:02:29 -04:00
Tyler Goodlet 9571f60a6d Expose channel in public api 2018-07-17 11:57:27 -04:00
Tyler Goodlet 64cbb922dc Reorg everything into private modules 2018-07-14 16:09:05 -04:00
goodboy f636bfdf83
Merge pull request #14 from tgoodlet/asyncgen_closing_fix
Asyncgen closing fix
2018-07-13 22:26:23 -04:00
Tyler Goodlet 1f85f71534 Use `async_generator`'s `aclosing()` helper
Take @njsmith's advice and properly close actor invoked async generators
using `async_generator.aclosing()` instead of hacking it (as previous)
with a shielded cancel scope.
2018-07-13 22:18:08 -04:00
Tyler Goodlet 2b7bbf32a1 One more super subtle cancellation fix
See python-trio/trio#455 for the deats...
2018-07-13 22:17:33 -04:00
goodboy c326a90484
Merge pull request #10 from tgoodlet/initial_tests_and_ci
Initial tests and ci
2018-07-12 00:15:58 -04:00
Tyler Goodlet 1b41b7b6b7 Add initial travis file 2018-07-11 22:32:54 -04:00
Tyler Goodlet a26d6f831f Add loglevel setting to test suite 2018-07-11 22:24:19 -04:00
Tyler Goodlet 590267ded2 Add a simpler cancel test 2018-07-11 22:24:19 -04:00
Tyler Goodlet d9aa6119e1 Set cancelled state in cancel method 2018-07-11 22:24:14 -04:00
Tyler Goodlet 1ade5c5fbb Add onc-cancels-all strategy to actor nursery 2018-07-11 22:24:06 -04:00
Tyler Goodlet 25852794a8 Move chan connect helper to ipc mod 2018-07-11 22:20:13 -04:00
Tyler Goodlet bb9309bdf5 Add a cancellation strategy test 2018-07-11 22:20:13 -04:00
Tyler Goodlet bb293905b9 Verify expected non-result under cancellation 2018-07-11 22:20:13 -04:00
Tyler Goodlet 209a6a2096 Add a separate cancel scope for the main task
Cancellation requires that each actor cancel it's spawned subactors
before cancelling its own root (nursery's) cancel scope to avoid breaking
channel connections before kill commands (`Actor.cancel()`) have been sent
off to peers. To solve this, ensure each main task is cancelled to
completion first (which will guarantee that all actor nurseries have
completed their cancellation steps) before cancelling the actor's "core"
tasks under the "root" scope.
2018-07-11 22:20:13 -04:00
Tyler Goodlet 1854471992 Add tests which verify the readme is correct
- steal from `trio` and add a `tractor_test` decorator
- use a random arbiter port to avoid conflicts with locally running
  systems
- add all the (obviously) hilarious readme tests
- add a complex cancellation test which works with
  `trio.move_on_after()`
2018-07-11 22:20:13 -04:00
Tyler Goodlet 49573c9a03 More fixes to do cancellation correctly
Here is a bunch of code tightening to make sure cancellation works even
if recently spawned actors haven't fully started up and the parent is
cancelled.

The fixes include:
- passing the arbiter socket address to each actor
- ensure all spawned actors respect the spawner's log level
- handle process versus final `portal.result()` teardown in multiple
  tasks such that if a proc dies before shipping a result we don't wait
- more detailed debug logging in teardown code paths
- don't store peer connected events in the same `dict` as the peer channels
- if necessary fake main task results on peer channel disconnect
- warn when a `trio.Cancelled` is what causes a nursery to bail
  otherwise error
- store the subactor portal in the nursery for teardown purposes
- add dedicated `Portal.cancel_actor()` which acts as a "hard cancel"
  and never blocks (indefinitely)
- add `Arbiter.unregister_actor()` it's more explicit what's being
  requested
2018-07-11 22:20:13 -04:00
Tyler Goodlet d94be22ef2 Add a "show me the code" test from the readme 2018-07-11 22:20:13 -04:00
Tyler Goodlet 77e34049b8 More fixes after unit testing
- Allow passing in a program-wide `loglevel`
- Add detailed debug logging particularly to do with channel msg processing
  and connection handling
- Don't daemonize subprocesses for now as it prevents use of
  sub-sub-actors (need to solve #6 first)
- Add a `Portal.close()` which just tells the remote actor to tear down
  the channel (for now)
- Add a message to signal the remote `StopAsyncIteration` from an async
  gen such that the client side terminates properly as well
- Make `Actor.cancel()` cancel the channel server first
- Actors *must* complete the arbiter registeration steps before moving
  on with their main taks and rpc handling
- When delivering rpc responses (using the local per caller queue) use
  the blocking interface (`trio.Queue.put()`) to get backpressure
- Properly detect an `partial` wrapped async generators in `_invoke`
2018-07-11 22:20:13 -04:00
Tyler Goodlet 10417303aa Get tests working again
Remove all the `piker` stuff and add some further checks including:
- main task result is returned correctly
- remote errors are raised locally
- remote async generator yields values locally
2018-07-11 22:20:13 -04:00
Tyler Goodlet 36fd75e217 Fix some bugs to get tests working
Fix quite a few little bugs:
- async gen func detection in `_invoke()`
- always cancel channel server on main task exit
- wait for remaining channel peers after unsub from arbiter
- return result from main task(s) all the way up to `tractor.run()`

Also add a `Portal.result()` for getting the final result(s) from the
actor's main task and fix up a bunch of docs.
2018-07-11 22:20:13 -04:00
Tyler Goodlet 2cc03965d8 Add trio plugin for testing 2018-07-11 22:20:13 -04:00
Tyler Goodlet 6163d4e9ea Don't create formatter if no log level set 2018-07-10 17:28:29 -04:00
Tyler Goodlet 9f5555ec21 Add trio framework classifier 2018-07-10 17:27:47 -04:00
Tyler Goodlet 0598c6ad58 Add setup script 2018-07-05 20:57:23 -04:00
Tyler Goodlet d3f01c29bf Add test deps 2018-07-05 20:55:38 -04:00
Tyler Goodlet c85752abd9 Steal piker's logging setup 2018-07-05 19:51:32 -04:00
Tyler Goodlet a2980d88c5 Fix import, but tests don't all work yet 2018-07-05 19:49:21 -04:00
Tyler Goodlet 8df706e535 Rename package dir to tractor 2018-07-05 19:40:36 -04:00
Tyler Goodlet f6080522f9 `tractor.run()` is required for testing now 2018-07-05 16:21:55 -04:00
Tyler Goodlet ae9ab81ff3 Don't bother unsetting the squeue; let errors propogate up 2018-07-05 16:21:55 -04:00
Tyler Goodlet b1ad909c54 Only cancel channel spawned rpc tasks when explicitly notified 2018-07-05 16:21:55 -04:00
Tyler Goodlet 56d3f6cffb Add a working arbiter registry system
Every actor now registers (and unregisters) with the arbiter at
startup/teardown. For now the registry is stored in a plain `dict` in
the arbiter's memory. This makes it possible to easily coordinate actors
started as plain Python processes or via `multiprocessing`.

A whole smörgåsbord of changes was required to accomplish this:
- factor handshake steps into a func
- track *every* channel connected to an actor including multiples to the
  same remote peer (may want to optimize this later)
- handle `trio.ClosedStreamError` gracefully in the message loop
- add an `open_portal` asynccontextmanager which handles channel
  creation, handshaking, and spawning a bg task for msg processing
- add a `start_actor()` for starting in-process actors directly
- add working `get_arbiter()` and `find_actor()` public routines
- `_main` now tries an anonymous channel connect to the stated
  arbiter sockaddr and uses that to determine whether to crown itself
2018-07-05 16:21:55 -04:00
Tyler Goodlet bf08310224 Add StreamQueue.connected() 2018-07-05 16:21:55 -04:00
Tyler Goodlet 82f22b76e5 Arbiter now supports non-empty statespace 2018-07-05 16:21:55 -04:00
Tyler Goodlet fa6f8185b6 Handle kb interrupt gracefully in sub-actors
Fail gracefully (by "aborting") the same way `trio` does. This avoids
ugly sub-proc tracebacks thrown at the console. Unset the local actor
when `tractor._main` completes. Cancel all tasks for a peer channel on
disconnect.
2018-07-05 16:21:55 -04:00
Tyler Goodlet 0aa49dcbdf Support re-entrant calls to `get_arbiter()`
It gets called more then once when using `tractor.run()` and
then `find_actor()`. Also, allow passing in the log level to each
new spawned subactor.
2018-07-05 16:21:55 -04:00
Tyler Goodlet 597546cf7b Drop console logging - messes with other tests 2018-07-05 16:21:55 -04:00