jc211
/
tractor
forked from goodboy/tractor
1
0
Fork 0
Code Pull Requests Activity
tractor/tractor/_entry.py

109 lines
2.9 KiB
Python
Raw Normal View History

Re-license code base for distribution under AGPLv4 This commit obviously denotes a re-license of all applicable parts of the code base (not including the minor patch setl not yet verified by contributors). From here henceforth all changes will be AGPLv4 licensed and distributed. This is purely an effort to maintain the same copy-left policy whilst closing the SaaS loophole the GPLv4 allows for. It is merely for this loophole, to avoid code hiding by any potential firms who are attempting to use the project to make a profit without either compensating the authors or re-distributing their changes. I thought quite a bit about this change and can't see a reason not to close the SaaS loophole in our current license. We still are (hard) copy-left and I plan to keep the code base this way for a couple reasons: - The code base produces income/profit through parent projects and is demonstrably of high value. - I believe firms should not get free lunch for the sake of "contributions from their employees" or "usage as a service" which I have found to be a dubious argument at best. - If a firm who intends to profit from the code base wants to use it they can propose a commercial license to purchase with the proceeds going to the project's authors under some form of well defined contract. - Many successful projects like Qt use this model; I see no reason it can't work in this case until such a time as the authors feel it should be loosened. There has been detailed discussion in #103 on licensing alternatives. The main point of this AGPL change is to protect the code base for the time being from exploitation and hijack as we move into the next phase of development which will include extending into the multi-host distributed software space.
2021-12-13 18:08:32 +00:00
# tractor: structured concurrent "actors".
# Copyright 2018-eternity Tyler Goodlet.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
"""
Re-org root actor startup into context manager This begins the move to dropping support for `tractor.run()` which we don't really need since the runtime is started (as it always has been) from a new sub-task / nursery. Instead this introduces starting the actor tree through a `open_root_actor()` async context manager which we'll likely implicitly call (from the root) on the first use of an actor nursery. Drop `_actor._start_actor()` and factor its contents into this new api. Make `run()` and `run_daemon()` use `open_root_actor()` until we decide to remove them. Relates to #168 and #177
2020-12-27 15:55:00 +00:00
Sub-process entry points.
Re-license code base for distribution under AGPLv4 This commit obviously denotes a re-license of all applicable parts of the code base (not including the minor patch setl not yet verified by contributors). From here henceforth all changes will be AGPLv4 licensed and distributed. This is purely an effort to maintain the same copy-left policy whilst closing the SaaS loophole the GPLv4 allows for. It is merely for this loophole, to avoid code hiding by any potential firms who are attempting to use the project to make a profit without either compensating the authors or re-distributing their changes. I thought quite a bit about this change and can't see a reason not to close the SaaS loophole in our current license. We still are (hard) copy-left and I plan to keep the code base this way for a couple reasons: - The code base produces income/profit through parent projects and is demonstrably of high value. - I believe firms should not get free lunch for the sake of "contributions from their employees" or "usage as a service" which I have found to be a dubious argument at best. - If a firm who intends to profit from the code base wants to use it they can propose a commercial license to purchase with the proceeds going to the project's authors under some form of well defined contract. - Many successful projects like Qt use this model; I see no reason it can't work in this case until such a time as the authors feel it should be loosened. There has been detailed discussion in #103 on licensing alternatives. The main point of this AGPL change is to protect the code base for the time being from exploitation and hijack as we move into the next phase of development which will include extending into the multi-host distributed software space.
2021-12-13 18:08:32 +00:00
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
"""
from functools import partial
Get entry points reorg without asyncio compat This is an edit to factor out changes needed for the `asyncio` in guest mode integration (which currently isn't tested well) so that later more pertinent changes (which are tested well) can be rebased off of this branch and merged into mainline sooner. The *infect_asyncio* branch will need to be rebased onto this branch as well before merge to mainline.
2020-07-24 20:55:34 +00:00
from typing import Tuple, Any
Disable SIGINT handling in child processes There seems to be no good reason not too since our cancellation machinery/protocol should do this work when the root receives the signal. This also (hopefully) helps with some debugging race condition stuff.
2020-09-28 13:24:36 +00:00
import signal
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
import trio # type: ignore
from .log import get_console_log, get_logger
from . import _state
log = get_logger(__name__)
def _mp_main(
Appease 3.8 mypy
2020-12-10 19:07:36 +00:00
actor: 'Actor', # type: ignore
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
accept_addr: Tuple[str, int],
forkserver_info: Tuple[Any, Any, Any, Any, Any],
start_method: str,
parent_addr: Tuple[str, int] = None,
) -> None:
"""The routine called *after fork* which invokes a fresh ``trio.run``
"""
actor._forkserver_info = forkserver_info
from ._spawn import try_set_start_method
spawn_ctx = try_set_start_method(start_method)
if actor.loglevel is not None:
log.info(
f"Setting loglevel for {actor.uid} to {actor.loglevel}")
get_console_log(actor.loglevel)
assert spawn_ctx
log.info(
f"Started new {spawn_ctx.current_process()} for {actor.uid}")
_state._current_actor = actor
log.debug(f"parent_addr is {parent_addr}")
trio_main = partial(
actor._async_main,
accept_addr,
parent_addr=parent_addr
)
try:
Get entry points reorg without asyncio compat This is an edit to factor out changes needed for the `asyncio` in guest mode integration (which currently isn't tested well) so that later more pertinent changes (which are tested well) can be rebased off of this branch and merged into mainline sooner. The *infect_asyncio* branch will need to be rebased onto this branch as well before merge to mainline.
2020-07-24 20:55:34 +00:00
trio.run(trio_main)
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
except KeyboardInterrupt:
pass # handle it the same way trio does?
End the `pdb` SIGINT handling madness Turns out this is a lower level issue in terms of the stdlib's default `pdb.Pdb` settings and how they conflict with `trio`s cancellation and KBI handling. The details are hashed out more thoroughly in python-trio/trio#1155. Maybe we can get a fix in trio so things are solved under our feet :)
2020-12-11 05:15:09 +00:00
finally:
log.info(f"Actor {actor.uid} terminated")
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
Now passing additional initialization parameters through channel early after handshake.
2020-07-27 17:55:37 +00:00
def _trio_main(
Appease 3.8 mypy
2020-12-10 19:07:36 +00:00
actor: 'Actor', # type: ignore
*,
parent_addr: Tuple[str, int] = None,
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
) -> None:
Add back subactor logging
2020-07-20 23:49:39 +00:00
"""Entry point for a `trio_run_in_process` subactor.
"""
Disable SIGINT handling in child processes There seems to be no good reason not too since our cancellation machinery/protocol should do this work when the root receives the signal. This also (hopefully) helps with some debugging race condition stuff.
2020-09-28 13:24:36 +00:00
# Disable sigint handling in children;
# we don't need it thanks to our cancellation machinery.
Try not masking SIGINT in child processes
2020-11-16 05:07:43 +00:00
# signal.signal(signal.SIGINT, signal.SIG_IGN)
Disable SIGINT handling in child processes There seems to be no good reason not too since our cancellation machinery/protocol should do this work when the root receives the signal. This also (hopefully) helps with some debugging race condition stuff.
2020-09-28 13:24:36 +00:00
Support sync code breakpointing via built-in Override `breakpoint()` for sync code making it work properly with `trio` as per: https://github.com/python-trio/trio/issues/1155#issuecomment-742964018 Relates to #193
2021-02-16 00:23:53 +00:00
log.info(f"Started new trio process for {actor.uid}")
Initial attempt at multi-actor debugging Allow entering and attaching to a `pdb` instance in a child process. The current hackery is to have the child make an rpc to the parent and ask it to hijack stdin, once complete the child enters a `pdb` blocking method. The parent then relays all stdin input to the child thus controlling the "remote" debugger. A few things were added to accomplish this: - tracking the mapping of subactors to their parent nurseries - in the root actor, cancelling all nurseries under the root `trio` task on cancellation (i.e. `Actor.cancel()`) - pass a "runtime vars" map down the actor tree for propagating global state
2020-07-23 17:23:55 +00:00
Add back subactor logging
2020-07-20 23:49:39 +00:00
if actor.loglevel is not None:
log.info(
f"Setting loglevel for {actor.uid} to {actor.loglevel}")
get_console_log(actor.loglevel)
Now passing additional initialization parameters through channel early after handshake.
2020-07-27 17:55:37 +00:00
log.info(
f"Started {actor.uid}")
Support "infected asyncio" actors This is an initial solution for #120. Allow spawning `asyncio` based actors which run `trio` in guest mode. This enables spawning `tractor` actors on top of the `asyncio` event loop whilst still leveraging the SC focused internal actor supervision machinery. Add a `tractor.to_syncio.run()` api to allow spawning tasks on the `asyncio` loop from an embedded (remote) `trio` task and return or stream results all the way back through the `tractor` IPC system using a very similar api to portals. One outstanding problem is getting SC around calls to `asyncio.create_task()`. Currently a task that crashes isn't able to easily relay the error to the embedded `trio` task without us fully enforcing the portals based message protocol (which seems superfluous given the error ref is in process). Further experiments using `anyio` task groups may alleviate this.
2020-06-28 17:10:02 +00:00
_state._current_actor = actor
First attempt at removing trip & updating hazmat -> lowlevel
2020-07-20 19:18:38 +00:00
Now passing additional initialization parameters through channel early after handshake.
2020-07-27 17:55:37 +00:00
log.debug(f"parent_addr is {parent_addr}")
trio_main = partial(
actor._async_main,
parent_addr=parent_addr
)
Log on KBI cancelled termination
2020-08-03 22:46:18 +00:00
Now passing additional initialization parameters through channel early after handshake.
2020-07-27 17:55:37 +00:00
try:
trio.run(trio_main)
except KeyboardInterrupt:
Log on KBI cancelled termination
2020-08-03 22:46:18 +00:00
log.warning(f"Actor {actor.uid} received KBI")
End the `pdb` SIGINT handling madness Turns out this is a lower level issue in terms of the stdlib's default `pdb.Pdb` settings and how they conflict with `trio`s cancellation and KBI handling. The details are hashed out more thoroughly in python-trio/trio#1155. Maybe we can get a fix in trio so things are solved under our feet :)
2020-12-11 05:15:09 +00:00
finally:
log.info(f"Actor {actor.uid} terminated")