Since I was trying out the neat lookin `polars-fuzzy-match` (also added
for now as a core dep here) which requires the new plugin sys, plus it's
about time we synced with upstream!
Adjust some column syntax to the new `.name` sub-field-space and the
`uv` lock-file to match.
Other,
- add back `trio-typing` bc i guess something else needs it (debug
tooling stuff in new `tractor`?)
- flip back to the `tractor` pre-main pin since the new `main`-branch
requires new `trio` stuff we haven't ported yet..
For example in the paper-eng, if you have a backend that doesn't fully
support a symcache (yet) it's handy to be able to ignore processing
other paper-eng txns when all you care about at the moment is the
simulated symbol.
NOTE, that currently this will still result in a key-error when you load
more then one mkt with the paper engine (for which the backend does not
have the symcache implemented) since no fqme ad-hoc query was made for
the 2nd symbol (and i'm not sure we should support that kinda hackery
over just encouraging the sym-cache being added?). Def needs a little
more thought depending on how many backends are never going to be able
to (easily) support caching..
Trying to start organizing non-hard deps into groups with sensible
"domain names" as it were. I coulda sworn we originally had at least UI
libs setup this way.. musta got lost in prior nix(os) porting.
Specifics,
- move all Qt and `rapidfuzz` deps into the `uis` group.
- add a new `repl` group for all the `pdbp` (debugging utils) and
`xonsh` (@goodboy's shell pref) related console related extensions.
- add a `testing` group for the harness' needs.
- add a `de` for (as of rn) TWM specific libs.
- nest all the new ^ groups in the `dev` group as needed.
Since the linking will be borked if we pull the wheel using `uv`; we
need to instead delegate to the `nixpkgs` version in the dev-shell.
`pyrpoject` deats,
- add a new deps-group: 'lint' which contains `ruff`.
- drop `ruff` from std deps (not sure how it got there anyway).
- mv `elasticsearch` to a new `dbs` deps group (we don't really even
want to be using it in the near furure).
- mv `uis` group into dep-groups section from `project.optionals-deps`.
- add a `tool.uv.default-groups = ['uis', 'dev']` setting which then
will avoid install of any non-explicit extras.
- put `rapidfuzz` only in `uis` group.
`flake.nix` tweaks,
- include `ruff` and `pypkgs.ruff` in the overlay.
- pass `--no-group ruff` to the `uv sync` line of shell init.
Taking many tips from our `default.nix` (thanks @nt!) this seems to be
the minimal overlay required for a flake to get up and running with
`piker chart` B)
Notes,
- for now, we're pinning to a major `cpython` version (3.13)
- ensure we (can) build with `nixpkgs.qt6.qtwayland`
- add the minimal Qt ld-lib-path linkings including those for plugin
use (required for wayland mode).
- for now, hardcode "wayland" platform-mode and the linux standard
"xdg-shell" integration.
- leave some TODOs to better parameterize around py versions.
Particularly using their recommended "impure template",
- https://pyproject-nix.github.io/pyproject.nix/templates.html#impure
- code: https://github.com/pyproject-nix/pyproject.nix/blob/master/templates/impure/flake.nix
Note the `shellHook` now contains various `uv`-specific osenv settings
and cmds to get a dev-env setup the way i would do it by default, that
includes all dev and extra (group) deps. For now i've hard coded the
"virt-env subdir" used by `uv` to match the cpython version. We can
obviously parameterize this much better in the future.
For those who want to spawn a diff shell then bash see the commented
line, for ex. i personally use `nix develop -c uv run xonsh`.
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.
Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
A new class var `tuple[Exception]` such that the err set can be reffed
externally as needed for catching other similar pub-sub/IPC failures in
other (related) real-time sub-systems.
Also added some now-masked logging for debugging live-feed stream reading
issues that should ONLY be used for debugging since they'll greatly
degrade HFT perf. Used the new `log.mk_repr()` stuff (that one day we
should prolly pull from `modden` as a dep) for pretty console emissions.
Since now `tractor` will raise this native `trio`-exc translated from
a `Stop` msg when the peer gracefully terminates a `tractor.MsgStream`.
Just `info()` log in such cases versus continuing to warn for the
others.
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.
Deats,
- expose and proxy through the `msg_recv_timeout` from
`open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
call.
Add a doc-string reflecting recent refinements, drop all the old hook
params, rename `n: trio.Nursery` -> `tn` for "task nursery" fitting with
code base's naming style.
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.
Deats,
- adjust `start_backfill()` to take a more explicit
`def_frame_duration: Duration` expected to be unpacked from any
backend hist init-config by the `tsdb_backfill()` caller which now
also computes a value from the first received frame when the config
section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
input and always decrement the query range by this value whenever
a `NoData` is raised by the provider-backend paired with an explicit
`log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
that received with much better humanized logging on the missing
segment using `pendulum.Interval/Duration.in_words()` output.
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..
As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
Was orig for debugging an issue with `kucoin` i think but definitely
shouldn't be left in XD
Also add `'perpetual_future'` to the `.start_backfill()` input literal
set since we don't expect the 'btc/usd.perp.binance' for now.
Based on the default provided in their
[docs](https://docs.astral.sh/ruff/configuration/) and migrating
previous config from the prior `poetry`-verion of our `pyproject.toml`
Tyler Goodlet
goodboy
Nelson Torres