Commit Graph

2078 Commits (ddfb1d0a7b466b0559ae00b5de71536497aedeeb)

Author SHA1 Message Date
Guillermo Rodriguez ddfb1d0a7b
Fix chan manager close remove_channel call 2025-04-13 14:02:41 -03:00
Guillermo Rodriguez 19ef8df9fd
Check if fdshare module is enable on share_fds function 2025-04-13 14:02:09 -03:00
Guillermo Rodriguez 73099fa1ba
Adapt ringbuf pubsub to new RBToken owner system 2025-04-13 13:36:02 -03:00
Guillermo Rodriguez b66ef12c2e
Add owner semantics to RBToken
Stop exporting `_ringbuf` on `tractor.ipc`
Use absolute imports on `_ringbuf` module
Add more comments and acm helpers for ringbuf allocation functions
Create generic FD sharing actor module in `tractor.linux._fdshare`
Include original allocator actor name as `owner` in RBToken
Auto share FDs of allocated ringbufs
On `attach_ringbuf_*` functions request fds from owner
Adapt all ringbuf tests to new system
2025-04-13 13:31:44 -03:00
Guillermo Rodriguez 8f400f6524
Better APIs for ringd and pubsub
Pubsub:
Remove un-necesary ChannelManager locking mechanism
Make ChannelManager.close wait for all channel removals
Make publisher turn switch configurable with `msgs_per_turn` variable
Fix batch_size setter on publisher
Add broadcast to publisher
Add endpoints on pubsub for remote actors to dynamically add and remove channels

Ringd:
Add fifo lock and use it on methods that modify _rings state
Add comments
Break up ringd.open_ringbuf apis into attach_, open_ & maybe_open_
When attaching its no longer a long running context, only on opens
Adapt ringd test to new apis
2025-04-10 13:13:08 -03:00
Guillermo Rodriguez 8f5d968078
Tweaks to make cancellation happen correctly on ringbuf receiver & fix test log msg 2025-04-10 11:27:52 -03:00
Guillermo Rodriguez 3327da8847
Refactor generate_sample_messages to be a generator and use numpy 2025-04-06 21:59:14 -03:00
Guillermo Rodriguez a15b852b18
Add trio resource semantics to ring pubsub 2025-04-06 21:19:39 -03:00
Guillermo Rodriguez 7b668c2f33
RingBufferReceiveChannel fixes for the non clean eof case, add comments 2025-04-06 21:16:55 -03:00
Guillermo Rodriguez 2bf8ce84cf
Add trio resource semantics to eventfd 2025-04-06 21:04:18 -03:00
Guillermo Rodriguez 0de70172fc
Adhere to trio semantics on channels for closed and busy resource cases 2025-04-06 17:02:15 -03:00
Guillermo Rodriguez 5fbd6aced7
Fix test docstring 2025-04-04 02:59:50 -03:00
Guillermo Rodriguez 1c5bebfca6
Enable ordering assertion & simplify some parts of test 2025-04-04 02:57:38 -03:00
Guillermo Rodriguez 53c98958b1
Add header to generic chan orderers 2025-04-04 02:47:27 -03:00
Guillermo Rodriguez 3c1873c68a
Fully test and fix bugs on _ringbuf._pubsub
Add generic channel orderer
2025-04-04 02:44:45 -03:00
Guillermo Rodriguez 0b9c2de3ad
Improve ringd ringbuf lifecycle
Unlink sock after use in fdshare
2025-04-04 02:41:50 -03:00
Guillermo Rodriguez eb20e5ea8d
Rename RingBuff -> RingBuffer
Combine RingBuffer stream and channel apis
Implement RingBufferReceiveChannel.receive_nowait
Make msg generator calculate hash
2025-04-04 02:36:59 -03:00
Guillermo Rodriguez 70d72fd173
Woops fix old typing Self stuff 2025-04-03 13:11:00 -03:00
Guillermo Rodriguez 26efa3b25f
Add header and fix white lines 2025-04-03 12:35:09 -03:00
Guillermo Rodriguez 20f7ba4c2c
Switch to using typing.Protocl instead of abc.ABC on ChannelManager, improve abstraction and add comments 2025-04-03 12:34:40 -03:00
Guillermo Rodriguez 57d48a3af7
Improve error handling in fdshare functions, add comments 2025-04-03 11:48:07 -03:00
Guillermo Rodriguez eef98bc826
Dont use relative import on ringd 2025-04-02 15:17:56 -03:00
Guillermo Rodriguez dfc0254995
recv_fds doesnt need to be an acm 2025-04-02 15:00:40 -03:00
Guillermo Rodriguez b8d1fd6978
Add LICENSE headers and module docstring on new files 2025-04-02 14:56:48 -03:00
Guillermo Rodriguez 739e728065
Add ringd test, which also tests fd share 2025-04-02 14:45:21 -03:00
Guillermo Rodriguez d01ad12b3b
Linux top-level submodule and ipc._ringbuf submodule
Added ringd actor to broker ring buf resources dynamically
Added ring pubsub based on ringd
Created tractor.linux submodule and moved eventfd stuff there
Implemented linux file descriptor ipc share async helpers
2025-04-02 14:39:02 -03:00
Guillermo Rodriguez bab265b2d8
Important RingBuffBytesSender fix on non batched mode! & downgrade nix-shell python to lowest supported 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 010874bed5
Catch trio cancellation on RingBuffReceiver bg eof listener task, add batched mode to RingBuffBytesSender 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez ea010ab46a
Add direct read method on EventFD
Type hint all ctx managers in _ringbuf.py
Remove unnecesary send lock on ring chan sender
Handle EOF on ring chan receiver
Rename ringbuf tests to make it less redundant
2025-03-27 20:36:46 -03:00
Guillermo Rodriguez be7fc89ae9
Add direct ctx managers for RB channels 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 2a9a78651b
Improve test_ringbuf test, drop MsgTransport ring buf impl for now in favour of a trio.abc.Channel[bytes] impl, add docstrings 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez be818a720a
Switch `tractor.ipc.MsgTransport.stream` type to `trio.abc.Stream`
Add EOF signaling mechanism
Support proper `receive_some` end of stream semantics
Add StapledStream non-ipc test
Create MsgpackRBStream similar to MsgpackTCPStream for buffered whole-msg reads
Add EventFD.read cancellation on EventFD.close mechanism using cancel scope
Add test for eventfd cancellation
Improve and add docstrings
2025-03-27 20:36:46 -03:00
Guillermo Rodriguez ba353bf46f
Better encapsulate RingBuff ctx managment methods and support non ipc usage
Add trio.StrictFIFOLock on sender.send_all
Support max_bytes argument on receive_some, keep track of write_ptr on receiver
Add max_bytes receive test test_ringbuf_max_bytes
Add docstrings to all ringbuf tests
Remove EFD_NONBLOCK support, not necesary anymore since we can use abandon_on_cancel=True on trio.to_thread.run_sync
Close eventfd's after usage on open_ringbuf
2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 9b2161506f
Break out transport protocol and tcp specifics into their own submodules under tractor.ipc 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 6b155849b7
Add buf_size to RBToken and add sender cancel test, move disable_mantracker to its own _mp_bs module 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 59c8c7bfe3
Make ring buf api use pickle-able RBToken 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 6ac6fd56c0
Address some of fomo\'s comments 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez f799e9ac51
Handle cancelation on EventFD.read 2025-03-27 20:36:46 -03:00
Guillermo Rodriguez 9980bb2bd0
Add module headers and fix spacing on tractor._ipc._linux 2025-03-27 20:36:45 -03:00
Guillermo Rodriguez 8de9ab291e
Move RingBuffSender|Receiver to its own tractor.ipc._ringbuf module 2025-03-27 20:36:45 -03:00
Guillermo Rodriguez 1a83626f26
Move linux specifics from tractor.ipc._shm into tractor.ipc._linux 2025-03-27 20:36:45 -03:00
Guillermo Rodriguez 6b4d08d030
Move tractor._shm to tractor.ipc._shm 2025-03-27 20:36:45 -03:00
Guillermo Rodriguez 7b8b9d6805
move tractor._ipc.py into tractor.ipc._chan.py 2025-03-27 20:36:45 -03:00
Guillermo Rodriguez 5afe0a0264
General improvements
EventFD class now expects the fd to already be init with open_eventfd
RingBuff Sender and Receiver fully manage SharedMemory and EventFD lifecycles, no aditional ctx mngrs needed
Separate ring buf tests into its own test bed
Add parametrization to test and cancellation
Add docstrings
Add simple testing data gen module .samples
2025-03-27 20:36:45 -03:00
Guillermo Rodriguez eeb9a7d61b
IPC ring bug impl with async read 2025-03-27 20:36:45 -03:00
Tyler Goodlet 5cee222353 Updates from latest `piker.data._sharedmem` changes 2025-03-27 17:54:04 -04:00
Tyler Goodlet 8ebb1f09de Pass `str` dtype for `use_str` case 2025-03-27 17:54:04 -04:00
Tyler Goodlet 2683a7f33a Allocate size-specced "empty" sequence from default values by type 2025-03-27 17:54:04 -04:00
Tyler Goodlet 255209f881 Mod define `_USE_POSIX`, add a of of todos 2025-03-27 17:54:04 -04:00
Tyler Goodlet 9a0d529b18 Parametrize rw test with variable frame sizes
Demonstrates fixed size frame-oriented reads by the child where the
parent only transmits a "read" stream msg on "frame fill events" such
that the child incrementally reads the shm list data (much like in
a real-time-buffered streaming system).
2025-03-27 17:54:04 -04:00