tsp_gaps: fixes for fault-less OHLCV time-series loads #35

goodboy · 2025-02-20T16:14:38Z

goodboy commented

2025-02-20 16:14:38 +00:00

Small patch set which avoids NoData caused crashes when a provider has an expected gap in historical data.

“automatic” history gap/end signalling (via NoData) for backends which provide no default frame size hint (via yielded config) where now the backfiller instead guesses based on the first received.
a tweak declaring perps as a no-/src-in-fqme-asset since we expect an fqme like btc.usdtm.perp.binance instead of btc/usd.usdtm.perp.binance atm.
ignore any non-.parquet extensioned files under the .config/piker/nativedb/ subdir to avoid a crash during indexing..
adds a new (but unused) detect_vlm_gaps() helper.
related to first bullet where we return early if backend does not define a get_mkt_pairs() mod ep.

Small patch set which avoids `NoData` caused crashes when a provider has an expected gap in historical data. - "automatic" history gap/end signalling (via `NoData`) for backends which provide no default frame size hint (via yielded config) where now the backfiller instead guesses based on the first received. - a tweak declaring perps as a no-`/src`-in-fqme-asset since we expect an fqme like `btc.usdtm.perp.binance` instead of `btc/usd.usdtm.perp.binance` atm. - ignore any non-`.parquet` extensioned files under the `.config/piker/nativedb/` subdir to avoid a crash during indexing.. - adds a new (but unused) `detect_vlm_gaps()` helper. - related to first bullet where we return early if backend does not define a `get_mkt_pairs()` mod ep.

goodboy added 6 commits 2025-02-20 16:14:38 +00:00

3caaa30b03 Mask no-data pause, add perps to no-`/src`-in-fqme asset set

Was orig for debugging an issue with `kucoin` i think but definitely
shouldn't be left in XD

Also add `'perpetual_future'` to the `.start_backfill()` input literal
set since we don't expect the 'btc/usd.perp.binance' for now.

5e4a6d61c7 Ignore any non-`.parquet` files under `.config/piker/nativedb/` subdir

c390e87536 `.storage.cli`: collect gap-markup-aids into `tf2aids: dict` prior to pause for introspection

d7179d47b0 `.tsp._anal`: add (unused) `detect_vlm_gaps()`

bf0ac93aa3 Only use `frame_types` if delivered during enter

The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..

As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.

d49608f74e Refine history gap/termination signalling

Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.

Deats,
- adjust `start_backfill()` to take a more explicit
  `def_frame_duration: Duration` expected to be unpacked from any
  backend hist init-config by the `tsdb_backfill()` caller which now
  also computes a value from the first received frame when the config
  section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
  input and always decrement the query range by this value whenever
  a `NoData` is raised by the provider-backend paired with an explicit
  `log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
  in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
  that received with much better humanized logging on the missing
  segment using `pendulum.Interval/Duration.in_words()` output.

goodboy referenced this pull request

2025-02-20 16:14:49 +00:00

subsys_refinery #29

goodboy merged commit 6c221bb348 into gitea_feats

2025-02-21 20:46:37 +00:00

goodboy referenced this issue from a commit

2025-02-21 20:46:39 +00:00

Merge pull request 'tsp_gaps: fixes for fault-less OHLCV time-series loads' (#35) from tsp_gaps into gitea_feats

Sign in to join this conversation.

No reviewers

No Label

No Milestone