xxx #20

Closed

ntorres wants to merge 15 commits from fix_deribit_hist_queries into deribit_fix_cleanup

Author	SHA1	Message	Date
Nelson Torres	376ea72dda	`deribit.feed`: fix "trade" event streaming The main change needed to make `piker.data.feed._FeedsBus` work was to correctly format the `'trade'` msgs with the (new schema) expected `'ticks': list[dict]` field which, - we compute the `piker` quote-msg-`dict` from the (now directly proxied through) `cryptofeed.types.Trade`'s fields inside the body of `stream_quotes()`. - similarly, move the `'l1'` msg processing, out of the `asyncio`-side `_l1()` callback (defined as a closure in `.api.aio_price_feed_relay()` and passed to the `cryptofeed.FeedHandler`) and instead mod the callback to simply pass through the `.types.L1Book` ref directly to the `piker`/`trio` side task for conversion. In support of all that, - mask-to-drop the alt-branch to wait on a first rt event when the `cryptofeed.LastTradesResult.trades: list[Trade]` is empty; doesn't seem like this ever even happens? - add a buncha typing, comments and doc-strs to the routines in `.deribit.api` including notes on where we can choose to mod the `.bs_fqme` for our eventually preferred `piker` style format. - simplify some nested `@acm` enters to the new single `async with <tuple>)` style. - be particularly pedantic about typing `tractor.to_asyncio.LinkedTaskChannel` - bit of pep8 line-spacing fixes in `.venues`.	2025-01-29 23:50:48 +00:00
Nelson Torres	ead1b76735	Ignore non-`.parquet` (suffixed) paths for now during tsdb fs-indexing	2025-01-29 23:50:27 +00:00
Nelson Torres	0f8a627867	Mask `ruff` config and pin `websockets=0.12` - the `ruff` section in the `pyproject.toml` is somehow borked? (even though it def was working a while back..) - `websockets` is completely broken in latest version since it's using old-ass `asyncio` APIs of some sort i think??	2025-01-29 23:50:09 +00:00
Nelson Torres	92cccf368a	`.deribit.feed`: get live quotes workin (again) The quote-msg `'topic'` field was being set and sent as the `OptionPair.symbol: str` value instead of as the `MktPair.bs_fqme: str` as is required for matching on the `piker.data.feed` side. So change to that and simplify the actual `.bs_fqme: str` value to NOT include the ISO-format time (for now) since it's a big ugly and longer term we need a `piker`-fqme friendly-on-ze-eyes format/style anyway..	2025-01-29 23:49:55 +00:00
Nelson Torres	e04e90a60e	Bit more `cryptofeed` adapter formatting and typing for clarity..	2025-01-29 23:49:41 +00:00
Nelson Torres	22912b260d	.deribit.venues: add todo for an ideal `OptionPair.expiry` fmt/value	2025-01-29 23:49:28 +00:00
Nelson Torres	e4b670f6f0	`.data._sampling`: warn about subscriber-less msgs Since it usually means the data-provider backend is keying the msgs incorrectly (not using the equivalent `MktPair.bs_fqme` which as would be rendered from the delivered `FeedInit.mkt` instance..) and reporting the subs list should make it clear how the fqme matching is off. Deats, - use the new `.log.mk_repr()` for a formatter. - add a commented info emission that can be unmasked to help debug any such cases as mentioned in the summary ^^.	2025-01-29 23:49:09 +00:00
Nelson Torres	fe422bfcb0	Add `.log.mk_repr()` to create `reprlib.Repr`s	2025-01-29 23:48:54 +00:00
Nelson Torres	5c19c2cec4	Report the closest (via fuzzy match) pairs on unmatched input	2025-01-29 23:48:32 +00:00
Nelson Torres	7d93acff6c	Signal hist start using `OptionPair.creation_timestamp` Such that the `get_hist()` query func raises `DataUnavailable` with an explicit message regarding the start of the (option) contract's lifetime. Other, - mask some unused imports (for now?) - drop a duplicate `tractor.get_console_log()` call which was causing duplicate console emits (it's already setup by brokerd init now). - comment various unused code bits i found. - add a info log around live quotes so we can see for the moment when they actually occur.. XD	2025-01-29 23:48:13 +00:00
Nelson Torres	27c800f5c4	`.deribit.api` bit of tidying/typing There were some imports missing or unused as well as a variety of spots that had grokability issues due to missing type hints. Other tweaks as part some more thorough manual testing: - always raise when not `brokers.toml` section since the API can never work (no free data without keys). - inline the `Asset.atype='crypto_currency` field despite it maybe not being the best value for `OptionPair` instruments.. - tossed in a now-masked pause block for debugging history queries in `Client.bars()`. - commented out all the live order ctl (internal) endpoints for now since they're unused.	2025-01-29 23:47:51 +00:00
Nelson Torres	d26a32abf2	'Fix `Optional` and use `'linear/reverse'` in `OptionPair.venue`'	2025-01-29 23:46:34 +00:00
Nelson Torres	7b7fc17ff1	Mk jsronrpc's underlying ws timeout `float('inf')` Since currently we're only using this IPC subsys for `deribit`, and generally speaking we're primarly supporting options markets (which are fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs` on timeout since doing so normally breaks the jsron-rpc IPC session. Without a proper `fixture` passed to `open_autorecon_ws()` (which we should eventually implement!!) relying on a timeout-to-reset more or less will just cause breakage issues - a proper reconnect sequence must be implemented before using that feature. Deats, - expose and proxy through the `msg_recv_timeout` from `open_jsonrpc_session()` into the underlying `open_autorecon_ws()` call.	2025-01-29 23:25:41 +00:00
Nelson Torres	ed6a3bda39	Refine history gap/termination signalling Namely handling backends which do not provide a default "frame size-duration" in their init-config by making the backfiller guess the value based on the first frame received. Deats, - adjust `start_backfill()` to take a more explicit `def_frame_duration: Duration` expected to be unpacked from any backend hist init-config by the `tsdb_backfill()` caller which now also computes a value from the first received frame when the config section isn't provided. - in `start_backfill()` we now always expect the `def_frame_duration` input and always decrement the query range by this value whenever a `NoData` is raised by the provider-backend paired with an explicit `log.warning()` about the handling. - also relay any `DataUnavailable.args[0]` message from the provider in the handler. - repair "gap reporting" which checks for expected frame duration vs. that received with much better humanized logging on the missing segment using `pendulum.Interval/Duration.in_words()` output.	2025-01-29 23:24:33 +00:00
Nelson Torres	30fb7da233	Only use `frame_types` if delivered during enter The `open_history_client()` provider endpoint can optionally deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its `config: dict[str, dict]` (as was implemented with the `ib` backend). This allows the `tsp` backfilling machinery to use this "recommended frame duration" to subtract from the `last_start_dt` any time a `NoData` gap is signalled by the `get_hist()` call allowing gaps to be ignored safely without missing history by knowing the next earliest dt we can query from using the `end_dt`. However, currently all crypto$ providers haven't implemented this feat yet.. As such only try to use the `frame_types` feature if provided when handling `NoData` conditions inside `tsp.start_backfill()` and otherwise raise as normal.	2025-01-29 23:24:04 +00:00