pikers
/
piker
3
0
Fork 0
Code Issues 8 Pull Requests 11 Packages Projects Releases Wiki Activity

fix_deribit_hist_queries #10

Closed
goodboy wants to merge 77 commits from fix_deribit_hist_queries into deribit_fix
pull from: fix_deribit_hist_queries
merge into: pikers:deribit_fix
Conversation 2 Commits 77 Files Changed 41
goodboy commented 2024-11-19 22:54:23 +00:00
Owner

Fixes for some issues discussed in #8 (comment)

Please test and merge if satisfied B)

Fixes for some issues discussed in https://pikers.dev/pikers/piker/pulls/8#issuecomment-156 Please test and merge if satisfied B)
goodboy added 7 commits 2024-11-19 22:54:23 +00:00
f96bd51442 Only use `frame_types` if delivered during enter 
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..

As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
9232d09440 Refine history gap/termination signalling 
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.

Deats,
- adjust `start_backfill()` to take a more explicit
  `def_frame_duration: Duration` expected to be unpacked from any
  backend hist init-config by the `tsdb_backfill()` caller which now
  also computes a value from the first received frame when the config
  section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
  input and always decrement the query range by this value whenever
  a `NoData` is raised by the provider-backend paired with an explicit
  `log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
  in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
  that received with much better humanized logging on the missing
  segment using `pendulum.Interval/Duration.in_words()` output.
bdc3bc9219 Mk jsronrpc's underlying ws timeout `float('inf')` 
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.

Deats,
- expose and proxy through the `msg_recv_timeout` from
  `open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
  call.
dc2c379d86 `.deribit.api` bit of tidying/typing 
There were some imports missing or unused as well as a variety of spots
that had grokability issues due to missing type hints.

Other tweaks as part some more thorough manual testing:
- always raise when not `brokers.toml` section since the API can never
  work (no free data without keys).
- inline the `Asset.atype='crypto_currency` field despite it maybe not
  being the best value for `OptionPair` instruments..
- tossed in a now-masked pause block for debugging history queries in
  `Client.bars()`.
- commented out all the live order ctl (internal) endpoints for now
  since they're unused.
b734245183 Signal hist start using `OptionPair.creation_timestamp` 
Such that the `get_hist()` query func raises `DataUnavailable` with an
explicit message regarding the start of the (option) contract's
lifetime.

Other,
- mask some unused imports (for now?)
- drop a duplicate `tractor.get_console_log()` call which was causing
  duplicate console emits (it's already setup by brokerd init now).
- comment various unused code bits i found.
- add a info log around live quotes so we can see for the moment when
  they actually occur.. XD
goodboy added 5 commits 2024-11-20 02:17:32 +00:00
1616cc0e82 `.data._sampling`: warn about subscriber-less msgs 
Since it usually means the data-provider backend is keying the msgs
incorrectly (not using the equivalent `MktPair.bs_fqme` which as
would be rendered from the delivered `FeedInit.mkt` instance..) and
reporting the subs list should make it clear how the fqme matching is
off.

Deats,
- use the new `.log.mk_repr()` for a formatter.
- add a commented info emission that can be unmasked to help debug any
  such cases as mentioned in the summary ^^.
1705afb607 `.deribit.feed`: get live quotes workin (again) 
The quote-msg `'topic'` field was being set and sent as the
`OptionPair.symbol: str` value instead of as the `MktPair.bs_fqme: str`
as is required for matching on the `piker.data.feed` side. So change to
that and simplify the actual `.bs_fqme: str` value to NOT include the
ISO-format time (for now) since it's a big ugly and longer term we need
a `piker`-fqme friendly-on-ze-eyes format/style anyway..
goodboy commented 2024-11-20 02:21:58 +00:00
Poster
Owner

Just fixed the live quotes as well 🏄

This is prolly good enuff for now and i’ll report other issues on #8?

Just fixed the live quotes as well 🏄 This is prolly good enuff for now and i'll report other issues on #8?
goodboy added 2 commits 2024-11-20 02:39:26 +00:00
a0dcf14aba Mask `ruff` config and pin `websockets=0.12` 
- the `ruff` section in the `pyproject.toml` is somehow borked? (even
  though it def was working a while back..)
- `websockets` is completely broken in latest version since it's using
  old-ass `asyncio` APIs of some sort i think??
goodboy added 1 commit 2024-11-22 20:22:39 +00:00
5cefe8bcdb `deribit.feed`: fix "trade" event streaming 
The main change needed to make `piker.data.feed._FeedsBus` work was
to correctly format the `'trade'` msgs with the (new schema) expected
`'ticks': list[dict]` field which,
- we compute the `piker` quote-msg-`dict` from the (now directly proxied through)
  `cryptofeed.types.Trade`'s fields inside the body of `stream_quotes()`.
- similarly, move the `'l1'` msg processing, **out of** the `asyncio`-side
  `_l1()` callback (defined as a closure in `.api.aio_price_feed_relay()`
  and passed to the `cryptofeed.FeedHandler`) and instead mod the
  callback to simply pass through the `.types.L1Book` ref directly to
  the `piker`/`trio` side task for conversion.

In support of all that,
- mask-to-drop the alt-branch to wait on a first rt event when the
  `cryptofeed.LastTradesResult.trades: list[Trade]` is empty; doesn't
  seem like this ever even happens?
- add a buncha typing, comments and doc-strs to the routines in
  `.deribit.api` including notes on where we can choose to mod the
  `.bs_fqme` for our eventually preferred `piker` style format.
- simplify some nested `@acm` enters to the new single `async with
  <tuple>)` style.
- be particularly pedantic about typing
  `tractor.to_asyncio.LinkedTaskChannel`
- bit of pep8 line-spacing fixes in `.venues`.
ntorres force-pushed fix_deribit_hist_queries from 5cefe8bcdb to db2e72a53d 2025-01-29 04:13:09 +00:00 Compare
ntorres force-pushed fix_deribit_hist_queries from db2e72a53d to bf195e8796 2025-01-29 18:53:37 +00:00 Compare
ntorres force-pushed fix_deribit_hist_queries from bf195e8796 to ffbca17ba2 2025-01-29 22:37:39 +00:00 Compare
ntorres changed title from fix_deribit_hist_queries to fix_deribit_hist_queries 2025-01-29 22:44:23 +00:00
ntorres changed target branch from deribit_fix to deribit_fix_cleanup 2025-01-29 22:44:24 +00:00
ntorres added 1 commit 2025-01-29 23:24:07 +00:00
30fb7da233 Only use `frame_types` if delivered during enter 
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..

As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
ntorres added 1 commit 2025-01-29 23:24:36 +00:00
ed6a3bda39 Refine history gap/termination signalling 
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.

Deats,
- adjust `start_backfill()` to take a more explicit
  `def_frame_duration: Duration` expected to be unpacked from any
  backend hist init-config by the `tsdb_backfill()` caller which now
  also computes a value from the first received frame when the config
  section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
  input and always decrement the query range by this value whenever
  a `NoData` is raised by the provider-backend paired with an explicit
  `log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
  in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
  that received with much better humanized logging on the missing
  segment using `pendulum.Interval/Duration.in_words()` output.
ntorres added 1 commit 2025-01-29 23:25:43 +00:00
7b7fc17ff1 Mk jsronrpc's underlying ws timeout `float('inf')` 
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.

Deats,
- expose and proxy through the `msg_recv_timeout` from
  `open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
  call.
ntorres added 1 commit 2025-01-29 23:46:37 +00:00
ntorres added 1 commit 2025-01-29 23:47:53 +00:00
27c800f5c4 `.deribit.api` bit of tidying/typing 
There were some imports missing or unused as well as a variety of spots
that had grokability issues due to missing type hints.

Other tweaks as part some more thorough manual testing:
- always raise when not `brokers.toml` section since the API can never
  work (no free data without keys).
- inline the `Asset.atype='crypto_currency` field despite it maybe not
  being the best value for `OptionPair` instruments..
- tossed in a now-masked pause block for debugging history queries in
  `Client.bars()`.
- commented out all the live order ctl (internal) endpoints for now
  since they're unused.
ntorres added 1 commit 2025-01-29 23:48:14 +00:00
7d93acff6c Signal hist start using `OptionPair.creation_timestamp` 
Such that the `get_hist()` query func raises `DataUnavailable` with an
explicit message regarding the start of the (option) contract's
lifetime.

Other,
- mask some unused imports (for now?)
- drop a duplicate `tractor.get_console_log()` call which was causing
  duplicate console emits (it's already setup by brokerd init now).
- comment various unused code bits i found.
- add a info log around live quotes so we can see for the moment when
  they actually occur.. XD
ntorres added 1 commit 2025-01-29 23:48:35 +00:00
ntorres added 1 commit 2025-01-29 23:48:57 +00:00
ntorres added 1 commit 2025-01-29 23:49:12 +00:00
e4b670f6f0 `.data._sampling`: warn about subscriber-less msgs 
Since it usually means the data-provider backend is keying the msgs
incorrectly (not using the equivalent `MktPair.bs_fqme` which as
would be rendered from the delivered `FeedInit.mkt` instance..) and
reporting the subs list should make it clear how the fqme matching is
off.

Deats,
- use the new `.log.mk_repr()` for a formatter.
- add a commented info emission that can be unmasked to help debug any
  such cases as mentioned in the summary ^^.
ntorres added 1 commit 2025-01-29 23:49:31 +00:00
ntorres added 1 commit 2025-01-29 23:49:42 +00:00
ntorres added 1 commit 2025-01-29 23:49:57 +00:00
92cccf368a `.deribit.feed`: get live quotes workin (again) 
The quote-msg `'topic'` field was being set and sent as the
`OptionPair.symbol: str` value instead of as the `MktPair.bs_fqme: str`
as is required for matching on the `piker.data.feed` side. So change to
that and simplify the actual `.bs_fqme: str` value to NOT include the
ISO-format time (for now) since it's a big ugly and longer term we need
a `piker`-fqme friendly-on-ze-eyes format/style anyway..
ntorres added 1 commit 2025-01-29 23:50:12 +00:00
0f8a627867 Mask `ruff` config and pin `websockets=0.12` 
- the `ruff` section in the `pyproject.toml` is somehow borked? (even
  though it def was working a while back..)
- `websockets` is completely broken in latest version since it's using
  old-ass `asyncio` APIs of some sort i think??
ntorres added 1 commit 2025-01-29 23:50:30 +00:00
ntorres added 1 commit 2025-01-29 23:50:51 +00:00
376ea72dda `deribit.feed`: fix "trade" event streaming 
The main change needed to make `piker.data.feed._FeedsBus` work was
to correctly format the `'trade'` msgs with the (new schema) expected
`'ticks': list[dict]` field which,
- we compute the `piker` quote-msg-`dict` from the (now directly proxied through)
  `cryptofeed.types.Trade`'s fields inside the body of `stream_quotes()`.
- similarly, move the `'l1'` msg processing, **out of** the `asyncio`-side
  `_l1()` callback (defined as a closure in `.api.aio_price_feed_relay()`
  and passed to the `cryptofeed.FeedHandler`) and instead mod the
  callback to simply pass through the `.types.L1Book` ref directly to
  the `piker`/`trio` side task for conversion.

In support of all that,
- mask-to-drop the alt-branch to wait on a first rt event when the
  `cryptofeed.LastTradesResult.trades: list[Trade]` is empty; doesn't
  seem like this ever even happens?
- add a buncha typing, comments and doc-strs to the routines in
  `.deribit.api` including notes on where we can choose to mod the
  `.bs_fqme` for our eventually preferred `piker` style format.
- simplify some nested `@acm` enters to the new single `async with
  <tuple>)` style.
- be particularly pedantic about typing
  `tractor.to_asyncio.LinkedTaskChannel`
- bit of pep8 line-spacing fixes in `.venues`.
ntorres force-pushed fix_deribit_hist_queries from 376ea72dda to ae6e5f5994 2025-01-30 18:28:42 +00:00 Compare
ntorres changed title from fix_deribit_hist_queries to fix_deribit_hist_queries 2025-01-30 18:54:16 +00:00
ntorres changed target branch from deribit_fix_cleanup to deribit_fix 2025-01-30 18:54:17 +00:00
ntorres commented 2025-01-30 18:59:21 +00:00
Collaborator

Sooo, While rebasing all my branches, I lost all goodBoy’s commits in this branch, I recovered them but I cannot cherry-picked them, soy I “re-committed” again, @goodboy I think if you push force on top of this branch, i can recover those commits and fix this all mess, I know how to rebase now

Sooo, While rebasing all my branches, I lost all goodBoy's commits in this branch, I recovered them but I cannot cherry-picked them, soy I "re-committed" again, @goodboy I think if you push force on top of this branch, i can recover those commits and fix this all mess, I know how to rebase now
ntorres force-pushed fix_deribit_hist_queries from ae6e5f5994 to 5c1c0f1738 2025-02-03 21:57:54 +00:00 Compare
ntorres force-pushed fix_deribit_hist_queries from 5c1c0f1738 to 5ff1703130 2025-02-03 22:28:35 +00:00 Compare
ntorres force-pushed fix_deribit_hist_queries from 5ff1703130 to 022432cce7 2025-02-03 22:34:25 +00:00 Compare
goodboy force-pushed fix_deribit_hist_queries from 022432cce7 to b4d3bcf240 2025-02-11 16:09:21 +00:00 Compare
goodboy added 2 commits 2025-02-11 16:35:12 +00:00
3e5e704c2f Group bcast errors as `Sampler.bcast_errors` 
A new class var `tuple[Exception]` such that the err set can be reffed
externally as needed for catching other similar pub-sub/IPC failures in
other (related) real-time sub-systems.

Also added some now-masked logging for debugging live-feed stream reading
issues that should ONLY be used for debugging since they'll greatly
degrade HFT perf. Used the new `log.mk_repr()` stuff (that one day we
should prolly pull from `modden` as a dep) for pretty console emissions.
ntorres closed this pull request 2025-02-18 00:12:21 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: pikers/piker#10
There is no content yet.
Delete Branch "fix_deribit_hist_queries"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?