pikers
/
piker
3
0
Fork 0
Code Issues 8 Pull Requests 12 Packages Projects Releases Wiki Activity

subsys_refinery #29

Open
goodboy wants to merge 22 commits from subsys_refinery into gitea_feats
pull from: subsys_refinery
merge into: pikers:gitea_feats
Conversation 0 Commits 22 Files Changed 21
goodboy commented 2025-02-12 16:34:45 +00:00
Owner

Originally there were some disparate subsys changes that were interleaved in the history of #9 (now landed) and various fixes/refinements added while (helping) dev(ing) #21, #13 and #23.

Some of these changes are required in order for a smooth landing of the above feature branches particularly ensuring the deribit backend is bug-free for options mkt history loading.

I plan to break up this PR into sub-PRs where possible each of the maximum-commits-per-subsys so we logically land each in sync with any upstream (github) issues that each (sub-PR) might potentially (partially) solve as well ensuring we don’t willy nilly merge in use of new feats/APIs from tractor without first ensuring we’re pinned to the appropriate dev-branch/alpha-release..

Additionally, the patches factored and distilled into this PR and should be mostly provisional fixes and/or non-critical refinements to various sub-systems utilized by data provider (backends).

Summary of per-subsy changes

(still need to reorder these bullets chronologically..)

  • now #35
    • piker.tsp: fixes for fault-less OHLCV time-series loads:
  • now factored to #36
    • .data._web_bs.open_jsonrpc_session() much needed refinements:
  • now in #38
    • an adjustment to .clearing._ems which permits using the last row of the Flume.rt_shm: ShmArray’s 'close' column when the received live-quote msg’s 'last' field is null.
  • now #39
    • .data._symcache: handling backends with no support..
  • now #49
    • miscelaneous piker.broker patches to various backends and some of daemon middleware layer.
what i’m hoping to at least factor out and land independently..

for landing alongside a proper synced tractor dev branch..

  • #37 for bringing in EndOfChannel handling to the Sampler subsys in sync with tractors new typed IPC msging.

  • both now included in history for #28

    • 18bf3484e0 delegates to the new tractor.msg.Struct originally factored from here which contains some pprint extensions around msgspec’s default type.
    • 7a912deb4a which enables the new .pause_from_sync() features available when greenback is installed throughout the (subactor) runtime.
Originally there were some disparate subsys changes that were interleaved in the history of #9 (now landed) and various fixes/refinements added while (helping) dev(ing) #21, #13 and #23. Some of these changes **are required** in order for a smooth landing of the above feature branches particularly ensuring the `deribit` backend is bug-free for options mkt history loading. I plan to break up this PR into sub-PRs where possible each of the maximum-commits-per-subsys so we logically land each in sync with any upstream (github) issues that each (sub-PR) might potentially (partially) solve as well ensuring we don't willy nilly merge in use of new feats/APIs from `tractor` without first ensuring we're pinned to the appropriate dev-branch/alpha-release.. Additionally, the patches factored and distilled into this PR and *should be* mostly provisional fixes and/or non-critical refinements to various sub-systems utilized by data provider (backends). --- ### Summary of per-subsy changes (still need to reorder these bullets chronologically..) - now #35 - `piker.tsp`: fixes for fault-less OHLCV time-series loads: - now factored to #36 - `.data._web_bs.open_jsonrpc_session()` much needed refinements: - now in #38 - an adjustment to `.clearing._ems` which permits using the last row of the `Flume.rt_shm: ShmArray`'s `'close'` column when the received live-quote msg's `'last'` field is null. - now #39 - `.data._symcache`: handling backends with no support.. - now #49 - miscelaneous `piker.broker` patches to various backends and some of `daemon` middleware layer. --- ##### what i'm hoping to at least factor out and land independently.. for landing alongside a proper synced `tractor` dev branch.. - #37 for bringing in `EndOfChannel` handling to the `Sampler` subsys in sync with `tractor`s new typed IPC msging. - both now included in history for #28 - https://pikers.dev/pikers/piker/commit/18bf3484e09865a1fdfe29cbe2034f62eea2266c delegates to the new `tractor.msg.Struct` originally factored from here which contains some pprint extensions around `msgspec`'s default type. - https://pikers.dev/pikers/piker/commit/7a912deb4a3b1f4c20cc46ba74a9b3eead5f0bae which enables the new `.pause_from_sync()` features available when `greenback` is installed throughout the (subactor) runtime.
goodboy added 5 commits 2025-02-12 16:34:46 +00:00
17e29715af Mask no-data pause, add perps to no-`/src`-in-fqme asset set 
Was orig for debugging an issue with `kucoin` i think but definitely
shouldn't be left in XD

Also add `'perpetual_future'` to the `.start_backfill()` input literal
set since we don't expect the 'btc/usd.perp.binance' for now.
cf9cb67b5d .clearing._ems: Don't require `first_quote['last']` 
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.

Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
492d673ec0 Allow ledger passes to ignore (symcache) unknown fqmes 
For example in the paper-eng, if you have a backend that doesn't fully
support a symcache (yet) it's handy to be able to ignore processing
other paper-eng txns when all you care about at the moment is the
simulated symbol.

NOTE, that currently this will still result in a key-error when you load
more then one mkt with the paper engine (for which the backend does not
have the symcache implemented) since no fqme ad-hoc query was made for
the 2nd symbol (and i'm not sure we should support that kinda hackery
over just encouraging the sym-cache being added?). Def needs a little
more thought depending on how many backends are never going to be able
to (easily) support caching..
07502aab1e Doc-n-clean `.data._web_bs.open_jsonrpc_session()` 
Add a doc-string reflecting recent refinements, drop all the old hook
params, rename `n: trio.Nursery` -> `tn` for "task nursery" fitting with
code base's naming style.
goodboy force-pushed subsys_refinery from 07502aab1e to f8e05effbc 2025-02-13 17:25:45 +00:00 Compare
goodboy added 3 commits 2025-02-14 01:01:53 +00:00
goodboy force-pushed subsys_refinery from 179a42ac78 to e4a68bdec9 2025-02-14 01:02:43 +00:00 Compare
goodboy force-pushed subsys_refinery from e4a68bdec9 to 6aedacd1b4 2025-02-14 02:20:44 +00:00 Compare
goodboy added 3 commits 2025-02-17 18:40:19 +00:00
3ab9a9b741 Only use `frame_types` if delivered during enter 
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..

As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
4c6c1029d6 Refine history gap/termination signalling 
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.

Deats,
- adjust `start_backfill()` to take a more explicit
  `def_frame_duration: Duration` expected to be unpacked from any
  backend hist init-config by the `tsdb_backfill()` caller which now
  also computes a value from the first received frame when the config
  section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
  input and always decrement the query range by this value whenever
  a `NoData` is raised by the provider-backend paired with an explicit
  `log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
  in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
  that received with much better humanized logging on the missing
  segment using `pendulum.Interval/Duration.in_words()` output.
bda23c25b9 Mk jsronrpc's underlying ws timeout `float('inf')` 
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.

Deats,
- expose and proxy through the `msg_recv_timeout` from
  `open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
  call.
goodboy force-pushed subsys_refinery from bda23c25b9 to d98fd54627 2025-02-17 19:56:49 +00:00 Compare
goodboy force-pushed subsys_refinery from d98fd54627 to 2836aa3e18 2025-02-17 19:57:42 +00:00 Compare
goodboy changed title from subsys_refinery to subsys_refinery 2025-02-18 16:02:48 +00:00
goodboy changed target branch from kucoin_and_binance_fix to gitea_feats 2025-02-18 16:02:49 +00:00
This pull request can be merged automatically.
You are not authorized to merge this pull request.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b subsys_refinery gitea_feats
git pull origin subsys_refinery

Step 2:

Merge the changes and update on Gitea.
git checkout gitea_feats
git merge --no-ff subsys_refinery
git push origin gitea_feats
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#29
There is no content yet.
Delete Branch "subsys_refinery"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?