Commit Graph

2142 Commits (a553446619eec994f4cbb1bf814c0b15132597d7)

Author SHA1 Message Date
Guillermo Rodriguez a553446619
Pubsub topics, enc & decoders
Implicit aclose on all channels on ChannelManager aclose
Implicit nursery cancel on pubsub acms
Use long running actor portal for open_{pub,sub}_channel_at fns
Add optional encoder/decoder on pubsub
Add topic system for multiple pub or sub on same actor
Add wait fn for sub and pub channel register
2025-04-22 06:25:45 -03:00
Guillermo Rodriguez 8799cf3b78
Add optional msgpack encoder & decoder to ringbuf apis 2025-04-22 06:25:45 -03:00
Guillermo Rodriguez 86e09a80f4
Log warning instead of exception on pubsub cancelled 2025-04-22 06:25:45 -03:00
Guillermo Rodriguez 59521cd4db
Add fix for cases where sockname len > 100 2025-04-22 06:25:45 -03:00
Guillermo Rodriguez 06103d1f44
Disable parent channel append on get_peer_by_name to_scan 2025-04-22 06:25:45 -03:00
Guillermo Rodriguez 4ca1aaeaeb
Only set shield flag when trio nursery mode is used 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 9b16eeed2f
Fix chan manager close remove_channel call 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez d60a49a853
Check if fdshare module is enable on share_fds function 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez f5513ba005
Adapt ringbuf pubsub to new RBToken owner system 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 39dccbdde7
Add owner semantics to RBToken
Stop exporting `_ringbuf` on `tractor.ipc`
Use absolute imports on `_ringbuf` module
Add more comments and acm helpers for ringbuf allocation functions
Create generic FD sharing actor module in `tractor.linux._fdshare`
Include original allocator actor name as `owner` in RBToken
Auto share FDs of allocated ringbufs
On `attach_ringbuf_*` functions request fds from owner
Adapt all ringbuf tests to new system
2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 5d6fa643ba
Better APIs for ringd and pubsub
Pubsub:
Remove un-necesary ChannelManager locking mechanism
Make ChannelManager.close wait for all channel removals
Make publisher turn switch configurable with `msgs_per_turn` variable
Fix batch_size setter on publisher
Add broadcast to publisher
Add endpoints on pubsub for remote actors to dynamically add and remove channels

Ringd:
Add fifo lock and use it on methods that modify _rings state
Add comments
Break up ringd.open_ringbuf apis into attach_, open_ & maybe_open_
When attaching its no longer a long running context, only on opens
Adapt ringd test to new apis
2025-04-22 06:25:44 -03:00
Guillermo Rodriguez e4868ded54
Tweaks to make cancellation happen correctly on ringbuf receiver & fix test log msg 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez b2f6c298f5
Refactor generate_sample_messages to be a generator and use numpy 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 171545e4fb
Add trio resource semantics to ring pubsub 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 853aa740aa
RingBufferReceiveChannel fixes for the non clean eof case, add comments 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 8e1f95881c
Add trio resource semantics to eventfd 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 1451feb159
Adhere to trio semantics on channels for closed and busy resource cases 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez 3a1eda9d6d
Fix test docstring 2025-04-22 06:25:44 -03:00
Guillermo Rodriguez d942f073e0
Enable ordering assertion & simplify some parts of test 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez d8d01e8b3c
Add header to generic chan orderers 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 1dfc639e54
Fully test and fix bugs on _ringbuf._pubsub
Add generic channel orderer
2025-04-22 06:25:43 -03:00
Guillermo Rodriguez bebd327023
Improve ringd ringbuf lifecycle
Unlink sock after use in fdshare
2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 3568ba5d5d
Rename RingBuff -> RingBuffer
Combine RingBuffer stream and channel apis
Implement RingBufferReceiveChannel.receive_nowait
Make msg generator calculate hash
2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 95ea4647cc
Woops fix old typing Self stuff 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 4385d38bc4
Add header and fix white lines 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez b1e1187a19
Switch to using typing.Protocl instead of abc.ABC on ChannelManager, improve abstraction and add comments 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 4b9d6b9276
Improve error handling in fdshare functions, add comments 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 28b86cb880
Dont use relative import on ringd 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez e34b6519c7
recv_fds doesnt need to be an acm 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 6646deb7f4
Add LICENSE headers and module docstring on new files 2025-04-22 06:25:43 -03:00
Guillermo Rodriguez 1bb9918e2d
Add ringd test, which also tests fd share 2025-04-22 06:25:42 -03:00
Guillermo Rodriguez 9238c6b245
Linux top-level submodule and ipc._ringbuf submodule
Added ringd actor to broker ring buf resources dynamically
Added ring pubsub based on ringd
Created tractor.linux submodule and moved eventfd stuff there
Implemented linux file descriptor ipc share async helpers
2025-04-22 06:25:42 -03:00
Guillermo Rodriguez f0af419ab2
Important RingBuffBytesSender fix on non batched mode! & downgrade nix-shell python to lowest supported 2025-04-22 06:25:42 -03:00
Guillermo Rodriguez 3b5ade7118
Catch trio cancellation on RingBuffReceiver bg eof listener task, add batched mode to RingBuffBytesSender 2025-04-22 06:25:42 -03:00
Guillermo Rodriguez ce09c70a74
Add direct read method on EventFD
Type hint all ctx managers in _ringbuf.py
Remove unnecesary send lock on ring chan sender
Handle EOF on ring chan receiver
Rename ringbuf tests to make it less redundant
2025-04-22 06:25:42 -03:00
Guillermo Rodriguez 9f788e07d4
Add direct ctx managers for RB channels 2025-04-22 06:25:42 -03:00
Guillermo Rodriguez 69ceee09f2
Improve test_ringbuf test, drop MsgTransport ring buf impl for now in favour of a trio.abc.Channel[bytes] impl, add docstrings 2025-04-22 06:25:42 -03:00
Guillermo Rodriguez a7df2132fa
Switch `tractor.ipc.MsgTransport.stream` type to `trio.abc.Stream`
Add EOF signaling mechanism
Support proper `receive_some` end of stream semantics
Add StapledStream non-ipc test
Create MsgpackRBStream similar to MsgpackTCPStream for buffered whole-msg reads
Add EventFD.read cancellation on EventFD.close mechanism using cancel scope
Add test for eventfd cancellation
Improve and add docstrings
2025-04-22 06:25:42 -03:00
Guillermo Rodriguez dd1c0fa51d
Better encapsulate RingBuff ctx managment methods and support non ipc usage
Add trio.StrictFIFOLock on sender.send_all
Support max_bytes argument on receive_some, keep track of write_ptr on receiver
Add max_bytes receive test test_ringbuf_max_bytes
Add docstrings to all ringbuf tests
Remove EFD_NONBLOCK support, not necesary anymore since we can use abandon_on_cancel=True on trio.to_thread.run_sync
Close eventfd's after usage on open_ringbuf
2025-04-22 06:25:42 -03:00
Guillermo Rodriguez 6ee5e3e077
Refinements, fix dec_hook builtins and same type bug 2025-04-22 06:24:01 -03:00
Guillermo Rodriguez 470acd98cc
Fix typing on mk_boxed_ext_structs 2025-04-22 05:04:14 -03:00
Guillermo Rodriguez bb37c31a70
Update CI to use uv 2025-04-22 05:00:13 -03:00
Guillermo Rodriguez 99c383d3c1
Change test structs name to not get conflicts with pytest 2025-04-22 04:37:08 -03:00
Guillermo Rodriguez 51746a71ac
Re-add boxed struct type system on _codec & create enc/dec hook auto factory 2025-04-22 04:30:30 -03:00
Tyler Goodlet 112ed27cda Move peer-tracking attrs from `Actor` -> `IPCServer`
Namely transferring the `Actor` peer-`Channel` tracking attrs,
- `._peers` which maps the uids to client channels (with duplicates
  apparently..)
- the `._peer_connected: dict[tuple[str, str], trio.Event]` child-peer
  syncing table mostly used by parent actors to wait on sub's to connect
  back during spawn.
- the `._no_more_peers = trio.Event()` level triggered state signal.

Further we move over with some minor reworks,
- `.wait_for_peer()` verbatim (adjusting all dependants).
- factor the no-more-peers shielded wait branch-block out of
  the end of `async_main()` into 2 new server meths,
  * `.has_peers()` with optional chan-connected checking flag.
  * `.wait_for_no_more_peers()` which *just* does the
    maybe-shielded `._no_more_peers.wait()`
2025-04-11 18:11:35 -04:00
Tyler Goodlet 42cf9e11a4 Mv `Actor._stream_handler()` to `.ipc._server` func
Call it `handle_stream_from_peer()` and bind in the `actor: Actor` via
a `handler=partial()` to `trio.serve_listeners()`.

With this (minus the `Actor._peers/._peer_connected/._no_more_peers`
attrs ofc) we get nearly full separation of IPC-connection-processing
(concerns) from `Actor` state. Thus it's a first look at modularizing
the low-level runtime into isolated subsystems which will hopefully
improve the entire code base's grok-ability and ease any new feature
design discussions especially pertaining to introducing and/or
composing-together any new transport protocols.
2025-04-11 14:51:52 -04:00
Tyler Goodlet 1ccb14455d Passthrough `_pause()` kwargs from `_maybe_enter_pm()` 2025-04-11 01:16:46 -04:00
Tyler Goodlet d534f1491b Fix assert on `.devx.maybe_open_crash_handler()` delivered `bxerr` 2025-04-11 01:16:12 -04:00
Tyler Goodlet 0f8b299b4f Improve bit of tooling for `test_resource_cache.py`
Namely while what I was actually trying to solve was why
`TransportClosed` was getting raised from `Portal.cancel_actor()` but
still useful edge case auditing either way. Also opts into the
`debug_mode` fixture with apprope timeout adjustment B)
2025-04-11 01:12:34 -04:00
Tyler Goodlet 9807318e3d Never hide non-[msgtype/tpt-closed] error tbs in `Channel.send()` 2025-04-11 00:00:12 -04:00