goodboy
/
tractor
1
0
Fork 1
Code Issues 1 Pull Requests 4 Packages Projects Releases Wiki Activity
tractor/examples/advanced_faults/ipc_failure_during_stream.py

149 lines
4.4 KiB
Python
Raw Normal View History

Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
'''
Complex edge case where during real-time streaming the IPC tranport
channels are wiped out (purposely in this example though it could have
been an outage) and we want to ensure that despite being in debug mode
(or not) the user can sent SIGINT once they notice the hang and the
actor tree will eventually be cancelled without leaving any zombies.
'''
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
import trio
from tractor import (
open_nursery,
context,
Context,
MsgStream,
)
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
async def break_channel_silently_then_error(
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
stream: MsgStream,
):
async for msg in stream:
await stream.send(msg)
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
# XXX: close the channel right after an error is raised
# purposely breaking the IPC transport to make sure the parent
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
# doesn't get stuck in debug or hang on the connection join.
# this more or less simulates an infinite msg-receive hang on
# the other end.
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
await stream._ctx.chan.send(None)
assert 0
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
async def close_stream_and_error(
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
stream: MsgStream,
):
async for msg in stream:
await stream.send(msg)
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
# wipe out channel right before raising
await stream._ctx.chan.send(None)
await stream.aclose()
assert 0
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
@context
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
async def recv_and_spawn_net_killers(
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
ctx: Context,
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
break_ipc: bool = False,
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
**kwargs,
) -> None:
'''
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
Receive stream msgs and spawn some IPC killers mid-stream.
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
'''
await ctx.started()
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
async with (
ctx.open_stream() as stream,
trio.open_nursery() as n,
):
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
async for i in stream:
print(f'child echoing {i}')
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
await stream.send(i)
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
if (
break_ipc
and i > 500
):
'#################################\n'
'Simulating child-side IPC BREAK!\n'
'#################################'
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
n.start_soon(break_channel_silently_then_error, stream)
n.start_soon(close_stream_and_error, stream)
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
async def main(
Handle `mp` spawn method cases in test suite
2023-01-27 22:02:36 +00:00
debug_mode: bool = False,
start_method: str = 'trio',
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
break_parent_ipc: bool = False,
break_child_ipc: bool = False,
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
) -> None:
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
async with (
open_nursery(
start_method=start_method,
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
# NOTE: even debugger is used we shouldn't get
# a hang since it never engages due to broken IPC
debug_mode=debug_mode,
loglevel='warning',
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
) as an,
):
portal = await an.start_actor(
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
'chitty_hijo',
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
enable_modules=[__name__],
)
async with portal.open_context(
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
recv_and_spawn_net_killers,
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
break_ipc=break_child_ipc,
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
) as (ctx, sent):
async with ctx.open_stream() as stream:
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
for i in range(1000):
if (
break_parent_ipc
and i > 100
):
print(
'#################################\n'
'Simulating parent-side IPC BREAK!\n'
'#################################'
)
await stream._ctx.chan.send(None)
Handle `mp` spawn method cases in test suite
2023-01-27 22:02:36 +00:00
Skip `advanced_faults/` subset in docs examples tests
2023-01-27 22:17:17 +00:00
# it actually breaks right here in the
# mp_spawn/forkserver backends and thus the zombie
# reaper never even kicks in?
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
print(f'parent sending {i}')
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
await stream.send(i)
with trio.move_on_after(2) as cs:
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
# NOTE: in the parent side IPC failure case this
# will raise an ``EndOfChannel`` after the child
# is killed and sends a stop msg back to it's
# caller/this-parent.
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
rx = await stream.receive()
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
print(f"I'm a happy user and echoed to me is {rx}")
Refine example to drop IPC mid-stream Use a task nursery in the subactor to spawn tasks which cancel the IPC channel mid stream to simulate the most concurrent case we're likely to see. Make `main()` accept a `debug_mode: bool` for parametrization. Fill out detailed comments/docs on this example.
2023-01-27 21:27:25 +00:00
if cs.cancelled_caught:
# pretend to be a user seeing no streaming action
# thinking it's a hang, and then hitting ctl-c..
Add IPC breakage on both parent and child side With the new fancy `_pytest.pathlib.import_path()` we can do real parametrization of the example-script-module code and thus configure whether the child, parent, or both silently break the IPC connection. Parametrize the test for all the above mentioned cases as well as the case where the IPC never breaks but we still simulate the user hammering ctl-c / SIGINT to terminate the actor tree. Adjust expected errors based on each case and heavily document each of these.
2023-01-28 03:59:15 +00:00
print("YOO i'm a user anddd thingz hangin..")
print(
"YOO i'm mad send side dun but thingz hangin..\n"
'MASHING CTlR-C Ctl-c..'
)
raise KeyboardInterrupt
Add WIP example of silent IPC breaks while streaming
2023-01-26 22:48:08 +00:00
if __name__ == '__main__':
trio.run(main)