forked from goodboy/tractor
1
0
Fork 0
Commit Graph

877 Commits (03ae42fa1075a7dddf1870643baa2752631b5634)

Author SHA1 Message Date
Tyler Goodlet d4a6cbbc34 Add CI badge 2018-08-07 17:59:27 -04:00
Tyler Goodlet a7315f91ee Document per-func actors for clarity 2018-08-07 17:59:27 -04:00
Tyler Goodlet 8f342ad0dd Add simpler intro
As per a suggestion from @njsmith I've added a little less verbose intro
which mentions *actors* up front.

More,
- add install cmd
- reorg spawning and portal sections a wee bit
2018-08-07 17:59:27 -04:00
Tyler Goodlet 168ff190b3 Initial readme documenting most features
There is a slew of tests to match to verify everything documented thus
far. Hopefully it's a decent little start :)
2018-08-07 17:59:27 -04:00
goodboy a51fbbcf9f
Merge pull request #28 from tgoodlet/reg_with_uid
Reg with uid and drop aiter_recv()
2018-08-07 15:07:34 -04:00
Tyler Goodlet e4ef973be9 Add discovery testing
Add a new test to verify actors register with their `.uid` tuple as
per #7. Break off relevant "discovery" tests into a new test module.
2018-08-07 14:30:25 -04:00
Tyler Goodlet 1bd5582d8a Register each actor using its unique ID tuple
This allows for registering more then one actor with the same "name"
when you have multiple actors fulfilling the same role. Eventually
we'll need support for looking up all actors registered under a given
"service name" (or whatever we decide to call it).

Also, a fix to the arbiter such that each new instance refers to a
separate `_registry` dict (found an issue with duplicate names during
testing).

Resolves #7
2018-08-07 14:03:01 -04:00
Tyler Goodlet 758fbc6790 Drop `Channel.aiter_recv()`
Internalize the implementation of this and expect client code to
iterate the `Channel` directly.

Resolves #16
2018-08-07 14:03:01 -04:00
Tyler Goodlet bd14cbe082 Port to trio's new resource error 2018-08-07 14:03:01 -04:00
Tyler Goodlet 163f747afb Drop legacy write_unsigned() 2018-08-07 14:02:42 -04:00
goodboy 7f0f2e52a9
Merge pull request #21 from tgoodlet/forkserver_singleton
Forkserver singleton
2018-08-04 18:18:21 -04:00
Tyler Goodlet e7c7391497 Drop needless tuple unpack 2018-08-04 18:15:24 -04:00
Tyler Goodlet 4fc7edf466 Change class names, use errno constant 2018-08-04 18:15:24 -04:00
Tyler Goodlet 4b875f0ade Be more explicit with naming and stdlib override 2018-08-04 18:15:24 -04:00
Tyler Goodlet 7017f68503 3.6.3 fix - missing older attr 2018-08-04 18:15:24 -04:00
Tyler Goodlet 50517c9488 Manage a `multiprocessing.forkserver` manually
Start a forkserver once in the main (parent-most) process
and pass ipc info (fds) to subprocesses manually such that embedded
calls to `multiprocessing.Process.start()` just work. Note that this
relies on our overridden version of the stdlib's
`multiprocessing.forkserver` module.

Resolves #6
2018-08-04 18:15:24 -04:00
Tyler Goodlet f46d5b2b62 Hackery to override the stdlib's forkserver
The stdlib insists on creating multiple forkservers and semaphore trackers
for each sub-sub-process launched. This isn't ideal since it costs each
`tractor` sub-actor an additional 2 more processes then necessary and is
confusing when viewed as a process tree (eg. via `pstree`).

The majority of the change is simply avoiding the call to
`forkserver.ensure_running()` and `semaphore_tracker.ensure_running()`
in `ForkServer.connect_new_process()` and instead treating the user like
an adult and expecting those calls to be made *once* in the parent most
process (i.e. what `multiprocessing` calls the `MainProcess`).

Really a proper patch should be made against cpython which allows for
similar manual management of the server along with a mechanism to communicate
forkserver and semaphore tracker fd info to sub-processes such that
further calls to `Process.start()` work as expected.

Relates to #6
2018-08-04 18:15:24 -04:00
Tyler Goodlet d6d7fea708 Use plain func for __aiter__() 2018-08-04 18:15:24 -04:00
goodboy 140956aedb
Merge pull request #25 from tgoodlet/drop_main_kwarg
Drop the "main" task via kwarg idea
2018-08-04 18:13:44 -04:00
Tyler Goodlet f7706074a2 Drop needless if check 2018-08-04 18:10:31 -04:00
Tyler Goodlet 1da69b1396 Allow daemonizing top level actor; don't require main func 2018-08-03 09:41:18 -04:00
Tyler Goodlet db08dbad3b Streaming is too fast, cancel sooner 2018-08-02 16:33:42 -04:00
Tyler Goodlet 41914f3118 Drop console logging 2018-08-02 16:29:01 -04:00
Tyler Goodlet dadea87451 Update test to new api 2018-08-02 15:27:09 -04:00
Tyler Goodlet bb13b79df5 Drop the "main" task via kwarg idea
Stop worrying about a "main task" in each actor and instead add an
additional `ActorNursery.run_in_actor()` method which wraps calls
to create an actor and run a lone RPC task inside it. Note this
adjusts the public API of `ActorNursery.start_actor()` to drop
its `main` kwarg.

The dirty deats of making this possible:
- each spawned RPC task is now tracked with a specific cancel scope such
  that when the actor is cancelled all ongoing responders are cancelled
  before any IPC/channel machinery is closed (turns out that spawning
  new actors from `outlive_main=True` actors was probably borked before
  finally getting this working).
- make each initial RPC response be a packet which describes the
  `functype` (eg. `{'functype': 'asyncfunction'}`) allowing for async
  calls/submissions by client actors (this was required to make
  `run_in_actor()` work - `Portal._submit()` is the new async method).
- hooray we can stop faking "main task" results for daemon actors
- add better handling/raising of internal errors caught in the bowels of
  the `Actor` itself.
- drop the rpc spawning nursery; just use the `Actor._root_nursery`
- only wait on `_no_more_peers` if there are existing peer channels that
  are actually still connected.
- an `ActorNursery.__aexit__()` now implicitly waits on `Portal.result()` on close
  for each `run_in_actor()` spawned actor.
- handle cancelling partial started actors which haven't yet connected
  back to the parent

Resolves #24
2018-08-02 15:24:28 -04:00
goodboy f726bd81da
Merge pull request #15 from tgoodlet/reorg
Reorg everything into private modules
2018-07-17 23:02:29 -04:00
Tyler Goodlet 9571f60a6d Expose channel in public api 2018-07-17 11:57:27 -04:00
Tyler Goodlet 64cbb922dc Reorg everything into private modules 2018-07-14 16:09:05 -04:00
goodboy f636bfdf83
Merge pull request #14 from tgoodlet/asyncgen_closing_fix
Asyncgen closing fix
2018-07-13 22:26:23 -04:00
Tyler Goodlet 1f85f71534 Use `async_generator`'s `aclosing()` helper
Take @njsmith's advice and properly close actor invoked async generators
using `async_generator.aclosing()` instead of hacking it (as previous)
with a shielded cancel scope.
2018-07-13 22:18:08 -04:00
Tyler Goodlet 2b7bbf32a1 One more super subtle cancellation fix
See python-trio/trio#455 for the deats...
2018-07-13 22:17:33 -04:00
goodboy c326a90484
Merge pull request #10 from tgoodlet/initial_tests_and_ci
Initial tests and ci
2018-07-12 00:15:58 -04:00
Tyler Goodlet 1b41b7b6b7 Add initial travis file 2018-07-11 22:32:54 -04:00
Tyler Goodlet a26d6f831f Add loglevel setting to test suite 2018-07-11 22:24:19 -04:00
Tyler Goodlet 590267ded2 Add a simpler cancel test 2018-07-11 22:24:19 -04:00
Tyler Goodlet d9aa6119e1 Set cancelled state in cancel method 2018-07-11 22:24:14 -04:00
Tyler Goodlet 1ade5c5fbb Add onc-cancels-all strategy to actor nursery 2018-07-11 22:24:06 -04:00
Tyler Goodlet 25852794a8 Move chan connect helper to ipc mod 2018-07-11 22:20:13 -04:00
Tyler Goodlet bb9309bdf5 Add a cancellation strategy test 2018-07-11 22:20:13 -04:00
Tyler Goodlet bb293905b9 Verify expected non-result under cancellation 2018-07-11 22:20:13 -04:00
Tyler Goodlet 209a6a2096 Add a separate cancel scope for the main task
Cancellation requires that each actor cancel it's spawned subactors
before cancelling its own root (nursery's) cancel scope to avoid breaking
channel connections before kill commands (`Actor.cancel()`) have been sent
off to peers. To solve this, ensure each main task is cancelled to
completion first (which will guarantee that all actor nurseries have
completed their cancellation steps) before cancelling the actor's "core"
tasks under the "root" scope.
2018-07-11 22:20:13 -04:00
Tyler Goodlet 1854471992 Add tests which verify the readme is correct
- steal from `trio` and add a `tractor_test` decorator
- use a random arbiter port to avoid conflicts with locally running
  systems
- add all the (obviously) hilarious readme tests
- add a complex cancellation test which works with
  `trio.move_on_after()`
2018-07-11 22:20:13 -04:00
Tyler Goodlet 49573c9a03 More fixes to do cancellation correctly
Here is a bunch of code tightening to make sure cancellation works even
if recently spawned actors haven't fully started up and the parent is
cancelled.

The fixes include:
- passing the arbiter socket address to each actor
- ensure all spawned actors respect the spawner's log level
- handle process versus final `portal.result()` teardown in multiple
  tasks such that if a proc dies before shipping a result we don't wait
- more detailed debug logging in teardown code paths
- don't store peer connected events in the same `dict` as the peer channels
- if necessary fake main task results on peer channel disconnect
- warn when a `trio.Cancelled` is what causes a nursery to bail
  otherwise error
- store the subactor portal in the nursery for teardown purposes
- add dedicated `Portal.cancel_actor()` which acts as a "hard cancel"
  and never blocks (indefinitely)
- add `Arbiter.unregister_actor()` it's more explicit what's being
  requested
2018-07-11 22:20:13 -04:00
Tyler Goodlet d94be22ef2 Add a "show me the code" test from the readme 2018-07-11 22:20:13 -04:00
Tyler Goodlet 77e34049b8 More fixes after unit testing
- Allow passing in a program-wide `loglevel`
- Add detailed debug logging particularly to do with channel msg processing
  and connection handling
- Don't daemonize subprocesses for now as it prevents use of
  sub-sub-actors (need to solve #6 first)
- Add a `Portal.close()` which just tells the remote actor to tear down
  the channel (for now)
- Add a message to signal the remote `StopAsyncIteration` from an async
  gen such that the client side terminates properly as well
- Make `Actor.cancel()` cancel the channel server first
- Actors *must* complete the arbiter registeration steps before moving
  on with their main taks and rpc handling
- When delivering rpc responses (using the local per caller queue) use
  the blocking interface (`trio.Queue.put()`) to get backpressure
- Properly detect an `partial` wrapped async generators in `_invoke`
2018-07-11 22:20:13 -04:00
Tyler Goodlet 10417303aa Get tests working again
Remove all the `piker` stuff and add some further checks including:
- main task result is returned correctly
- remote errors are raised locally
- remote async generator yields values locally
2018-07-11 22:20:13 -04:00
Tyler Goodlet 36fd75e217 Fix some bugs to get tests working
Fix quite a few little bugs:
- async gen func detection in `_invoke()`
- always cancel channel server on main task exit
- wait for remaining channel peers after unsub from arbiter
- return result from main task(s) all the way up to `tractor.run()`

Also add a `Portal.result()` for getting the final result(s) from the
actor's main task and fix up a bunch of docs.
2018-07-11 22:20:13 -04:00
Tyler Goodlet 2cc03965d8 Add trio plugin for testing 2018-07-11 22:20:13 -04:00
Tyler Goodlet 6163d4e9ea Don't create formatter if no log level set 2018-07-10 17:28:29 -04:00
Tyler Goodlet 9f5555ec21 Add trio framework classifier 2018-07-10 17:27:47 -04:00