Compare commits

..

56 Commits

Author SHA1 Message Date
Tyler Goodlet db2e72a53d `deribit.feed`: fix "trade" event streaming
The main change needed to make `piker.data.feed._FeedsBus` work was
to correctly format the `'trade'` msgs with the (new schema) expected
`'ticks': list[dict]` field which,
- we compute the `piker` quote-msg-`dict` from the (now directly proxied through)
  `cryptofeed.types.Trade`'s fields inside the body of `stream_quotes()`.
- similarly, move the `'l1'` msg processing, **out of** the `asyncio`-side
  `_l1()` callback (defined as a closure in `.api.aio_price_feed_relay()`
  and passed to the `cryptofeed.FeedHandler`) and instead mod the
  callback to simply pass through the `.types.L1Book` ref directly to
  the `piker`/`trio` side task for conversion.

In support of all that,
- mask-to-drop the alt-branch to wait on a first rt event when the
  `cryptofeed.LastTradesResult.trades: list[Trade]` is empty; doesn't
  seem like this ever even happens?
- add a buncha typing, comments and doc-strs to the routines in
  `.deribit.api` including notes on where we can choose to mod the
  `.bs_fqme` for our eventually preferred `piker` style format.
- simplify some nested `@acm` enters to the new single `async with
  <tuple>)` style.
- be particularly pedantic about typing
  `tractor.to_asyncio.LinkedTaskChannel`
- bit of pep8 line-spacing fixes in `.venues`.
2025-01-29 01:11:28 -03:00
Tyler Goodlet 3f36be1010 Ignore non-`.parquet` (suffixed) paths for now during tsdb fs-indexing 2025-01-29 01:11:28 -03:00
Tyler Goodlet b915a10da4 Mask `ruff` config and pin `websockets=0.12`
- the `ruff` section in the `pyproject.toml` is somehow borked? (even
  though it def was working a while back..)
- `websockets` is completely broken in latest version since it's using
  old-ass `asyncio` APIs of some sort i think??
2025-01-29 01:11:23 -03:00
Tyler Goodlet 5c66a767d7 `.deribit.feed`: get live quotes workin (again)
The quote-msg `'topic'` field was being set and sent as the
`OptionPair.symbol: str` value instead of as the `MktPair.bs_fqme: str`
as is required for matching on the `piker.data.feed` side. So change to
that and simplify the actual `.bs_fqme: str` value to NOT include the
ISO-format time (for now) since it's a big ugly and longer term we need
a `piker`-fqme friendly-on-ze-eyes format/style anyway..
2025-01-29 01:10:58 -03:00
Tyler Goodlet 14b37975f3 Bit more `cryptofeed` adapter formatting and typing for clarity.. 2025-01-29 01:10:58 -03:00
Tyler Goodlet 678170b9f0 .deribit.venues: add todo for an ideal `OptionPair.expiry` fmt/value 2025-01-29 01:10:58 -03:00
Tyler Goodlet 82406f0ea6 `.data._sampling`: warn about subscriber-less msgs
Since it usually means the data-provider backend is keying the msgs
incorrectly (not using the equivalent `MktPair.bs_fqme` which as
would be rendered from the delivered `FeedInit.mkt` instance..) and
reporting the subs list should make it clear how the fqme matching is
off.

Deats,
- use the new `.log.mk_repr()` for a formatter.
- add a commented info emission that can be unmasked to help debug any
  such cases as mentioned in the summary ^^.
2025-01-29 01:10:58 -03:00
Tyler Goodlet 6b7600fe25 Add `.log.mk_repr()` to create `reprlib.Repr`s 2025-01-29 01:10:58 -03:00
Tyler Goodlet 3cc6a40cda Report the closest (via fuzzy match) pairs on unmatched input 2025-01-29 01:10:58 -03:00
Tyler Goodlet 2717e6adb2 Signal hist start using `OptionPair.creation_timestamp`
Such that the `get_hist()` query func raises `DataUnavailable` with an
explicit message regarding the start of the (option) contract's
lifetime.

Other,
- mask some unused imports (for now?)
- drop a duplicate `tractor.get_console_log()` call which was causing
  duplicate console emits (it's already setup by brokerd init now).
- comment various unused code bits i found.
- add a info log around live quotes so we can see for the moment when
  they actually occur.. XD
2025-01-29 01:10:58 -03:00
Tyler Goodlet d3930e3da9 `.deribit.api` bit of tidying/typing
There were some imports missing or unused as well as a variety of spots
that had grokability issues due to missing type hints.

Other tweaks as part some more thorough manual testing:
- always raise when not `brokers.toml` section since the API can never
  work (no free data without keys).
- inline the `Asset.atype='crypto_currency` field despite it maybe not
  being the best value for `OptionPair` instruments..
- tossed in a now-masked pause block for debugging history queries in
  `Client.bars()`.
- commented out all the live order ctl (internal) endpoints for now
  since they're unused.
2025-01-29 01:10:58 -03:00
Tyler Goodlet ad0ebecf3b 'Fix `Optional` and use `'linear/reverse'` in `OptionPair.venue`' 2025-01-29 01:10:58 -03:00
Tyler Goodlet 8b98296259 Mk jsronrpc's underlying ws timeout `float('inf')`
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.

Deats,
- expose and proxy through the `msg_recv_timeout` from
  `open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
  call.
2025-01-29 01:10:58 -03:00
Tyler Goodlet 16811acfd7 Refine history gap/termination signalling
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.

Deats,
- adjust `start_backfill()` to take a more explicit
  `def_frame_duration: Duration` expected to be unpacked from any
  backend hist init-config by the `tsdb_backfill()` caller which now
  also computes a value from the first received frame when the config
  section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
  input and always decrement the query range by this value whenever
  a `NoData` is raised by the provider-backend paired with an explicit
  `log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
  in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
  that received with much better humanized logging on the missing
  segment using `pendulum.Interval/Duration.in_words()` output.
2025-01-29 01:10:58 -03:00
Tyler Goodlet 457b6de8e2 Only use `frame_types` if delivered during enter
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..

As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
2025-01-29 01:10:58 -03:00
Nelson Torres dad339efad tractor branch updated, msgspec version upgraded, cython and greenback dependencies moved under dev group 2025-01-29 01:10:47 -03:00
Nelson Torres c8bb8d7893 config refactor
only one get_config method for api class and cryptofeed feed handler
2025-01-29 01:03:27 -03:00
Nelson Torres b4141d4208 move constants to venue 2025-01-29 01:03:27 -03:00
Nelson Torres 468b33cf60 refactor redundant code 2025-01-29 01:03:27 -03:00
Nelson Torres 2c2fa990ed name formatting fixes 2025-01-29 01:03:27 -03:00
Nelson Torres 7bc98f5055 get_mkt_info cleanup 2025-01-29 01:03:27 -03:00
Nelson Torres 8b4bcf134c cache_symbols refactor 2025-01-29 01:03:27 -03:00
Nelson Torres 3388bc89cc json_rpc_auth_wrapper 2025-01-29 01:03:27 -03:00
Nelson Torres b84880cce5 move object classes to venue 2025-01-29 01:03:27 -03:00
Nelson Torres 7f5e655720 Added options symbols to get_assets 2025-01-29 01:03:27 -03:00
Tyler Goodlet b8ad77e861 .clearing._ems: Don't require `first_quote['last']`
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.

Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
2025-01-29 01:03:27 -03:00
Tyler Goodlet dbafb64bc3 `kucoin`: repair live quotes streaming..
This must have broke at some point during the new `MktPair` and thus
`.fqme: str` updates; mas-o-menos the symbol key in the quote-msg-`dict`
was NOT set to the `MktPair.bs_fqme: str` value and thus wasn't being
processed by the downstream sampling and feed subsys.

So fix that as well as a few other refinements,
- set the `topic: mkt.bs_fqme` in quote msgs obvi.
- drop the "wait for first clearing vlm" quote poll loop; going to fix
  the sampler to handle a `first_quote` without a `'last'` key.
- add some typing around calls to `get_mkt_info()`.
- rename `stream_messages()` -> `iter_normed_quotes()`.
2025-01-29 01:03:27 -03:00
Nelson Torres 00fb5ceddd Deleted settlePlan field from binance FutesPair. 2025-01-29 01:03:27 -03:00
Nelson Torres e8d26f8b30 Added missing fields for kucoin.
feeCategory, makerFeeCoefficient, takerFeeCoefficient and st.
2025-01-29 01:03:27 -03:00
Nelson Torres fb76d1c9ed get_assets now uses public endpoint
It's better if the data is available through a public endpoint.
2025-01-29 01:03:27 -03:00
Nelson Torres 9688f4899a now using exch_info in search_symbols 2025-01-29 01:03:27 -03:00
Nelson Torres 7298374fe4 Fix bs_fqme using venue and expiry 2025-01-29 01:03:27 -03:00
Nelson Torres a7d8b5e1f4 Added expiry property for OptionPair 2025-01-29 01:03:27 -03:00
Nelson Torres 65e4bd80b2 No longer needed 2025-01-29 01:03:27 -03:00
Nelson Torres 660c580786 bs_mktid instead bs_fqme for deribits options 2025-01-29 01:03:27 -03:00
Nelson Torres a2651fd1de Fixed pair instrument name in search_symbols endpoint.
Fixed instrument in bars endpoint, for options in deribits bs_mktid instead bs_fqme.
Fixed the id is in msg.
2025-01-29 01:03:27 -03:00
Tyler Goodlet 12d557f12b data._web_bs: try to raise jsonrpc errors in parent task 2025-01-29 01:03:27 -03:00
Nelson Torres 8365b6d373 Add necessary classes in init file for deribit 2025-01-29 01:03:27 -03:00
Nelson Torres badb7342f7 Minor refactor in open_symbol_search 2025-01-29 01:03:27 -03:00
Nelson Torres b1bac1dc9f stream_quotes now using FeedInit 2025-01-29 01:03:27 -03:00
Nelson Torres be660f0275 symbol_info refactor 2025-01-29 01:03:27 -03:00
Nelson Torres 41e69af090 search_symbols output type fix 2025-01-29 01:03:27 -03:00
Nelson Torres 01e8fe18dd add get_mkt_pairs method 2025-01-29 01:03:27 -03:00
Nelson Torres b104e603db get_assets refactor 2025-01-29 01:03:27 -03:00
Nelson Torres 67424562ba formatting 2025-01-29 01:03:27 -03:00
Nelson Torres 0a92f54b59 created exch_info in api class 2025-01-29 01:03:27 -03:00
Nelson Torres bf1e2da1a7 modify self_pairs type to ChainMap 2025-01-29 01:03:27 -03:00
Nelson Torres d11e1d1ee4 Necessary imports 2025-01-29 01:03:27 -03:00
Nelson Torres cf2b507453 add get_market_info 2025-01-29 01:03:27 -03:00
Nelson Torres 61bde5a58d Necessary imports 2025-01-29 01:03:27 -03:00
Nelson Torres 43ec2e1897 minor fixes in venues 2025-01-29 01:03:27 -03:00
Nelson Torres d7669ba18b add class Pair in venues, PAIRTYPES for future uses 2025-01-29 01:03:27 -03:00
Nelson Torres 3821be082b fix syms for venues.
little refactor in get_config, and created get_fh_config for cryptofeed.
2025-01-29 01:03:27 -03:00
Nelson Torres 25d60d6bdd venues for deribit 2025-01-29 01:03:27 -03:00
Nelson Torres a9178b211d Added cryptofeed and pyarrow necessary for the feed, enable deribit
in the brokers init file, at this point the feed is working, to check
the tables use vd tool.
2025-01-29 01:03:27 -03:00
Nelson Torres ec2203168a `default.nix` is created with required nixos deps "wiring"
`pyproject.toml` update to use pikers.dev/goodboy/tractor.git, branch: `aio_abandons`.
2025-01-29 00:34:25 -03:00
2 changed files with 5 additions and 41 deletions

View File

@ -30,8 +30,7 @@ from types import ModuleType
from typing import (
Any,
Iterator,
Generator,
TYPE_CHECKING,
Generator
)
import pendulum
@ -60,10 +59,8 @@ from ..clearing._messages import (
BrokerdPosition,
)
from piker.types import Struct
from piker.log import get_logger
if TYPE_CHECKING:
from piker.data._symcache import SymbologyCache
from piker.data._symcache import SymbologyCache
from ..log import get_logger
log = get_logger(__name__)
@ -496,17 +493,6 @@ class Account(Struct):
_mktmap_table: dict[str, MktPair] | None = None,
only_require: list[str]|True = True,
# ^list of fqmes that are "required" to be processed from
# this ledger pass; we often don't care about others and
# definitely shouldn't always error in such cases.
# (eg. broker backend loaded that doesn't yet supsport the
# symcache but also, inside the paper engine we don't ad-hoc
# request `get_mkt_info()` for every symbol in the ledger,
# only the one for which we're simulating against).
# TODO, not sure if there's a better soln for this, ideally
# all backends get symcache support afap i guess..
) -> dict[str, Position]:
'''
Update the internal `.pps[str, Position]` table from input
@ -549,32 +535,11 @@ class Account(Struct):
if _mktmap_table is None:
raise
required: bool = (
only_require is True
or (
only_require is not True
and
fqme in only_require
)
)
# XXX: caller is allowed to provide a fallback
# mktmap table for the case where a new position is
# being added and the preloaded symcache didn't
# have this entry prior (eg. with frickin IB..)
if (
not (mkt := _mktmap_table.get(fqme))
and
required
):
raise
elif not required:
continue
else:
# should be an entry retreived somewhere
assert mkt
mkt = _mktmap_table[fqme]
if not (pos := pps.get(bs_mktid)):
@ -691,7 +656,7 @@ class Account(Struct):
def write_config(self) -> None:
'''
Write the current account state to the user's account TOML file, normally
something like `pps.toml`.
something like ``pps.toml``.
'''
# TODO: show diff output?

View File

@ -653,7 +653,6 @@ async def open_trade_dialog(
# in) use manually constructed table from calling
# the `.get_mkt_info()` provider EP above.
_mktmap_table=mkt_by_fqme,
only_require=list(mkt_by_fqme),
)
pp_msgs: list[BrokerdPosition] = []