fix_deribit_hist_queries #10
Open
goodboy
wants to merge 15 commits from
fix_deribit_hist_queries
into deribit_fix
pull from: fix_deribit_hist_queries
merge into: pikers:deribit_fix
pikers:max_pain_deribit
pikers:deribit_fix
pikers:jsonrpc_err_in_rent_task
pikers:kucoin_and_binance_fix
pikers:nix-qt6-fix
pikers:service_mng_to_tractor
pikers:go_httpx
pikers:nix-headless-fix
pikers:go_httpx_binance
pikers:fix-binance-venues
pikers:pyqt6
pikers:distribute_dis
pikers:runtime_to_msgspec
pikers:nix-monkeys-fix
pikers:ib_py311_fixes
pikers:poetry2nix
pikers:account_tests
pikers:py311_ib_fix
pikers:master
pikers:basic_buy_bot
pikers:marketstore_disable_snappy
pikers:rekt_pps
pikers:py311
pikers:kucoin_backend
pikers:small_kucoin_fixes
pikers:pre_overruns_ctxcancelled
pikers:emit_clear_ticks_only_on_ts_change
pikers:binance_ws_ep_update
pikers:log_linearized_curve_overlays
pikers:xdotool_fixes
pikers:deribit_updates
pikers:storage_middleware_layer
pikers:service_subpkg
pikers:pps_precision_hotfixes
pikers:service_subpkg_backup
pikers:storage_cli
pikers:xdo_and_you
pikers:decimalization_take_2
pikers:backward_compat_trans_with_symbolinfo
pikers:explicit_write_pps_on_exit
pikers:backend_spec
pikers:paper_trade_improvements_rebase
pikers:loglevel_to_testpikerd
pikers:overlays_interaction_latency_tuning
pikers:kraken_deposits_fixes
pikers:l1_compaction
pikers:epoch_indexing_and_dataviz_layer
pikers:multichartz
pikers:axis_sticky_api
pikers:multi_symbol_input
pikers:update_qt_screen_info_script
pikers:fsps_and_flumes
pikers:epoch_index
pikers:pregraphics_formatters
pikers:multichartz_backup
pikers:samplerd_service
pikers:misc_brokerd_backend_repairs
pikers:pre_viz_calls
pikers:dark_clearing_improvements
pikers:dark_clearing_repairs
pikers:epoch_index_backup
pikers:agg_feedz
pikers:kraken_limits_fields
pikers:cz_post_ftx
pikers:kraken_pair_status
pikers:ib_contract_updates
pikers:pre_multifeed_hotfix
pikers:kraken_pair_updates
pikers:daemon_sockaddr_config
pikers:no_signal_pi_overlays
pikers:pre_multi_feedz
pikers:pg_exts_fork
pikers:ib_1m_hist
pikers:even_moar_kraken_order_fixes
pikers:ci_fix_tractor_testing
pikers:clears_table_events
pikers:offline_dark_clearing
pikers:multi_client_order_mgt
pikers:pin_tractor_main
pikers:history_view
pikers:tractor_core_port
pikers:kraken_fill_bugs
pikers:paper_clear_logics_fix
pikers:kraken_nameerr_fix
pikers:livenpaper
pikers:size_in_shm_token
pikers:deribit
pikers:asycvnc_pin_bump
pikers:live_n_paper
pikers:open_order_loading
pikers:doin_the_splits
pikers:dict_differ
pikers:msgpack_zombie
pikers:the_ems_flattening
pikers:kraken_ws_orders
pikers:kraken_userref_hackzin
pikers:pydantic_zombie
pikers:ib_pps_upgrade
pikers:multisympaper
pikers:ppu_history
pikers:basic_pp_audit
pikers:historical_breakeven_pp_price
pikers:ib_native_data_hack
pikers:pptables
pikers:ahab_you_bad_boi
pikers:fix_forex
pikers:fsp_shm_caching
pikers:paper_eng_msg_fixes
pikers:drop_pydantic
pikers:krakenwsbackup
pikers:dpbackup
pikers:null_last_quote_fix
pikers:ahab_hardkill
pikers:tractor_typed_msg_hackin
pikers:kraken_ledger_pps
pikers:ib_rt_pp_update_hotfix
pikers:notokeninwswrapper
pikers:pps_postmortem
pikers:lifo_pps_ib
pikers:310_plus
pikers:mxmn_from_m4
pikers:contain_mkts
pikers:slic_fix_v2
pikers:uppx_slice_fix
pikers:update_last_datums_in_view
pikers:ib_subpkg
pikers:flexxin
pikers:ib_checker_hackz
pikers:incremental_update_paths
pikers:ib_dedicated_data_client
pikers:dockerize_ib_gw
pikers:pre_flow
pikers:drop_pandas
pikers:no_ib_pps
pikers:trimeter_dep
pikers:l1_precision_fix
pikers:marketstore
pikers:no_orderid_in_error
pikers:incr_update_backup
pikers:big_data_lines
pikers:m4_corrections
pikers:offline_history_loading
pikers:drop_arrow_add_predulum
pikers:marketstore_backup
pikers:broker_bumpz
pikers:no_git_prot_w_pip
pikers:kraken_editorder
pikers:fqsns
pikers:kraken_cleaning
pikers:kraken_orders
pikers:only_draw_iv_for_ohlc
pikers:mkts_backup
pikers:pp_bar_fixes
pikers:async_hist_loading
pikers:kraken_gb
pikers:windows_fixes_yo
pikers:ib_mkt_closed
pikers:py3.10_support
pikers:gb_kraken_orders
pikers:dark_vlm
pikers:overlayed_dvlm
pikers:fsp_ui_mod
pikers:vlm_plotz
pikers:plotitem_overlays
pikers:misc_backend_fixes
pikers:dolla_vlm
pikers:fspd_cluster
pikers:misc_ib_updates
pikers:single_display_update_loop
pikers:basic_vlm_display
pikers:pane_sizing_fixes
pikers:tractor_clustering
pikers:msgpack_no_sets_allowed
pikers:tinas_unite
pikers:simpler_quote_throttle_logic
pikers:fast_step_curve
pikers:win_fixes
pikers:fsp_hotfixes
pikers:teardown_guesmost_via_cs
pikers:windows_testing_volume
pikers:fsp_drunken_alignment
pikers:vlm_plotz_backup
pikers:fsp_feeds
pikers:chart_mod_breakup
pikers:pause_feeds_on_sym_switch
pikers:brokers_config
pikers:ordermodepps_backup
pikers:readme_bump_zone
pikers:ci_on_forks
pikers:asyncify_input_modes
pikers:minimal_brokerd_trade_dialogues
pikers:wait_on_daemon_portals
pikers:backup_asyncify_input_modes
pikers:naive_feed_throttling
pikers:window_cuckery
pikers:msgspec_fixes
pikers:status_bar
pikers:ems_hotfixes
pikers:ems_to_bidir_streaming
pikers:web_utils
pikers:symbol_search
pikers:binance_syminfo_and_mintick
pikers:update_throttling
pikers:syseng_tweaks
pikers:ems_tweaks
pikers:py3.9
pikers:binance_aggtrades_and_ohlc_parsing
pikers:binance_backend
pikers:tractor_open_stream_from
pikers:feed_fixes
pikers:order_mode_finesse
pikers:cached_feeds
pikers:readme_bumpz
pikers:supervise
pikers:basic_orders
pikers:kraken_trades_data
pikers:basic_alerts
pikers:kraken_history
pikers:y_zoom
pikers:chart_trader
pikers:graphics_pixel_buffer
pikers:vwap_fsp
pikers:to_qpainterpath_and_beyond
pikers:tina_free_vwap
pikers:vwap_backup
pikers:bar_select
pikers:ib_backend
pikers:unleash_the_kraken
pikers:facepalm
pikers:marketstore_integration
pikers:its_happening
pikers:relicense_as_agplv3
pikers:questrade_candles
pikers:use_tractor_logging
pikers:questrade_conns
pikers:kivy_mainline_and_py3.8
15 Commits (310_plus)
Author | SHA1 | Message | Date |
---|---|---|---|
Tyler Goodlet | 5cefe8bcdb |
`deribit.feed`: fix "trade" event streaming
The main change needed to make `piker.data.feed._FeedsBus` work was to correctly format the `'trade'` msgs with the (new schema) expected `'ticks': list[dict]` field which, - we compute the `piker` quote-msg-`dict` from the (now directly proxied through) `cryptofeed.types.Trade`'s fields inside the body of `stream_quotes()`. - similarly, move the `'l1'` msg processing, **out of** the `asyncio`-side `_l1()` callback (defined as a closure in `.api.aio_price_feed_relay()` and passed to the `cryptofeed.FeedHandler`) and instead mod the callback to simply pass through the `.types.L1Book` ref directly to the `piker`/`trio` side task for conversion. In support of all that, - mask-to-drop the alt-branch to wait on a first rt event when the `cryptofeed.LastTradesResult.trades: list[Trade]` is empty; doesn't seem like this ever even happens? - add a buncha typing, comments and doc-strs to the routines in `.deribit.api` including notes on where we can choose to mod the `.bs_fqme` for our eventually preferred `piker` style format. - simplify some nested `@acm` enters to the new single `async with <tuple>)` style. - be particularly pedantic about typing `tractor.to_asyncio.LinkedTaskChannel` - bit of pep8 line-spacing fixes in `.venues`. |
|
Tyler Goodlet | d96e9d4f11 | Ignore non-`.parquet` (suffixed) paths for now during tsdb fs-indexing | |
Tyler Goodlet | a0dcf14aba |
Mask `ruff` config and pin `websockets=0.12`
- the `ruff` section in the `pyproject.toml` is somehow borked? (even though it def was working a while back..) - `websockets` is completely broken in latest version since it's using old-ass `asyncio` APIs of some sort i think?? |
|
Tyler Goodlet | 1705afb607 |
`.deribit.feed`: get live quotes workin (again)
The quote-msg `'topic'` field was being set and sent as the `OptionPair.symbol: str` value instead of as the `MktPair.bs_fqme: str` as is required for matching on the `piker.data.feed` side. So change to that and simplify the actual `.bs_fqme: str` value to NOT include the ISO-format time (for now) since it's a big ugly and longer term we need a `piker`-fqme friendly-on-ze-eyes format/style anyway.. |
|
Tyler Goodlet | dafd5a3ca5 | Bit more `cryptofeed` adapter formatting and typing for clarity.. | |
Tyler Goodlet | b9dde98d1e | .deribit.venues: add todo for an ideal `OptionPair.expiry` fmt/value | |
Tyler Goodlet | 1616cc0e82 |
`.data._sampling`: warn about subscriber-less msgs
Since it usually means the data-provider backend is keying the msgs incorrectly (not using the equivalent `MktPair.bs_fqme` which as would be rendered from the delivered `FeedInit.mkt` instance..) and reporting the subs list should make it clear how the fqme matching is off. Deats, - use the new `.log.mk_repr()` for a formatter. - add a commented info emission that can be unmasked to help debug any such cases as mentioned in the summary ^^. |
|
Tyler Goodlet | 0a2ed195a7 | Add `.log.mk_repr()` to create `reprlib.Repr`s | |
Tyler Goodlet | 28e8628c61 | Report the closest (via fuzzy match) pairs on unmatched input | |
Tyler Goodlet | b734245183 |
Signal hist start using `OptionPair.creation_timestamp`
Such that the `get_hist()` query func raises `DataUnavailable` with an explicit message regarding the start of the (option) contract's lifetime. Other, - mask some unused imports (for now?) - drop a duplicate `tractor.get_console_log()` call which was causing duplicate console emits (it's already setup by brokerd init now). - comment various unused code bits i found. - add a info log around live quotes so we can see for the moment when they actually occur.. XD |
|
Tyler Goodlet | dc2c379d86 |
`.deribit.api` bit of tidying/typing
There were some imports missing or unused as well as a variety of spots that had grokability issues due to missing type hints. Other tweaks as part some more thorough manual testing: - always raise when not `brokers.toml` section since the API can never work (no free data without keys). - inline the `Asset.atype='crypto_currency` field despite it maybe not being the best value for `OptionPair` instruments.. - tossed in a now-masked pause block for debugging history queries in `Client.bars()`. - commented out all the live order ctl (internal) endpoints for now since they're unused. |
|
Tyler Goodlet | be84d0dae1 | 'Fix `Optional` and use `'linear/reverse'` in `OptionPair.venue`' | |
Tyler Goodlet | bdc3bc9219 |
Mk jsronrpc's underlying ws timeout `float('inf')`
Since currently we're only using this IPC subsys for `deribit`, and generally speaking we're primarly supporting options markets (which are fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs` on timeout since doing so normally breaks the jsron-rpc IPC session. Without a proper `fixture` passed to `open_autorecon_ws()` (which we should eventually implement!!) relying on a timeout-to-reset more or less will just cause breakage issues - a proper reconnect sequence must be implemented before using that feature. Deats, - expose and proxy through the `msg_recv_timeout` from `open_jsonrpc_session()` into the underlying `open_autorecon_ws()` call. |
|
Tyler Goodlet | 9232d09440 |
Refine history gap/termination signalling
Namely handling backends which do not provide a default "frame size-duration" in their init-config by making the backfiller guess the value based on the first frame received. Deats, - adjust `start_backfill()` to take a more explicit `def_frame_duration: Duration` expected to be unpacked from any backend hist init-config by the `tsdb_backfill()` caller which now also computes a value from the first received frame when the config section isn't provided. - in `start_backfill()` we now always expect the `def_frame_duration` input and always decrement the query range by this value whenever a `NoData` is raised by the provider-backend paired with an explicit `log.warning()` about the handling. - also relay any `DataUnavailable.args[0]` message from the provider in the handler. - repair "gap reporting" which checks for expected frame duration vs. that received with much better humanized logging on the missing segment using `pendulum.Interval/Duration.in_words()` output. |
|
Tyler Goodlet | f96bd51442 |
Only use `frame_types` if delivered during enter
The `open_history_client()` provider endpoint can *optionally* deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its `config: dict[str, dict]` (as was implemented with the `ib` backend). This allows the `tsp` backfilling machinery to use this "recommended frame duration" to subtract from the `last_start_dt` any time a `NoData` gap is signalled by the `get_hist()` call allowing gaps to be ignored safely without missing history by knowing the next earliest dt we can query from using the `end_dt`. However, currently all crypto$ providers haven't implemented this feat yet.. As such only try to use the `frame_types` feature if provided when handling `NoData` conditions inside `tsp.start_backfill()` and otherwise raise as normal. |