goodboy
/
tractor
1
0
Fork 2
Code Issues 4 Pull Requests 10 Packages Projects Releases Wiki Activity

Shared memory array API and optional tight integration with numpy #15

Open
goodboy wants to merge 12 commits from shm_apis into ext_type_plds
pull from: shm_apis
merge into: goodboy:ext_type_plds
Conversation 0 Commits 12 Files Changed 2
goodboy commented 2025-03-22 19:18:15 +00:00
Owner

Mega forward rebase of https://github.com/goodboy/tractor/pull/338 onto latest tip of dev/feats branches B)

Original motivating feature-issue is from GH,

https://github.com/goodboy/tractor/issues/339

Before landing here,

  • considering closing the PR on GH and doing the whole discussion/design/paired-testing here alongside the very exciting work from @guille in #10 !!
  • ensure CI runs clean on matching GH PR first
taken from GH todo list,
Mega forward rebase of https://github.com/goodboy/tractor/pull/338 onto latest tip of dev/feats branches B) Original motivating feature-issue is from GH, https://github.com/goodboy/tractor/issues/339 --- #### Before landing here, - [ ] considering closing the PR on GH and doing the whole discussion/design/paired-testing here alongside the very exciting work from @guille in https://pikers.dev/goodboy/tractor/pulls/10 !! - [ ] ensure CI runs clean on matching GH PR first - [ ] force push to GH once ready - https://github.com/goodboy/tractor/pull/338 ---- ##### taken from GH todo list, - [ ] test(s) which reproduces https://github.com/pikers/piker/issues/306 and https://github.com/pikers/piker/issues/332 - [ ] ideally we also can start running CI against OS X as in #255 - [x] naive support for a readonly capable shared list wrapper around [`ShareableList`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.ShareableList) - [ ] code adjustements for optional `numpy.ndarray` support - [ ] use [`nptyping`](https://github.com/ramonhagenaars/nptyping) to get `mypy` happy (as possible) - [x] first example of a readers-write lock style with both `numpy` and `ShmList` type - [x] include in test suite - [x] add a "frame size" parametrization - [ ] document this in the readme?
goodboy added 13 commits 2025-03-22 19:18:15 +00:00
25a333816f Initial module import from `piker.data._sharemem` 
More or less a verbatim copy-paste minus some edgy variable naming and
internal `piker` module imports. There is a bunch of OHLC related
defaults that need to be dropped and we need to adjust to an optional
dependence on `numpy` by supporting shared lists as per the mp docs.
c709a4ad72 Add `ShmList` wrapping the stdlib's `ShareableList` 
First attempt at getting `multiprocessing.shared_memory.ShareableList`
working; we wrap the stdlib type with a readonly attr and a `.key` for
cross-actor lookup. Also, rename all `numpy` specific routines to have
a `ndarray` suffix in the func names.
917e6fa2b2 Parametrize rw test with variable frame sizes 
Demonstrates fixed size frame-oriented reads by the child where the
parent only transmits a "read" stream msg on "frame fill events" such
that the child incrementally reads the shm list data (much like in
a real-time-buffered streaming system).
goodboy changed title from shm_apis to Shared memory array API and optional tight integration with `numpy` 2025-03-22 19:25:48 +00:00
goodboy force-pushed shm_apis from caa76eb30d to c64d5083cf 2025-03-24 18:30:53 +00:00 Compare
goodboy force-pushed shm_apis from c64d5083cf to c595a88069 2025-03-24 20:04:31 +00:00 Compare
goodboy force-pushed shm_apis from c595a88069 to 843fd96b5d 2025-03-25 16:09:06 +00:00 Compare
goodboy force-pushed shm_apis from 843fd96b5d to 8c6522bd12 2025-03-25 20:01:23 +00:00 Compare
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b shm_apis ext_type_plds
git pull origin shm_apis

Step 2:

Merge the changes and update on Gitea.
git checkout ext_type_plds
git merge --no-ff shm_apis
git push origin ext_type_plds
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: goodboy/tractor#15
There is no content yet.
Delete Branch "shm_apis"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?