Such that we can avoid other (pretty unreliable) "alternative" checks to
determine whether a real-time quote should be waited on or (when venue
is closed) we should just signal that historical backfilling can
commence immediately.
This has been a todo for a very long time and it turned out to be much
easier to accomplish than anticipated..
Deats,
- add a new `is_current_time_in_range()` dt range checker to predicate
whether an input range contains `datetime.now(start_dt.tzinfo)`.
- in `.ib.feed.stream_quotes()` add a `venue_is_open: bool` which uses
all of the new ^^ to determine whether to branch for the
short-circuit-and-do-history-now-case or the std real-time-quotes
should-be-awaited-since-venue-is-open, case; drop all the old hacks
trying to workaround not figuring that venue state stuff..
Other,
- also add a gpt5 composed parser to `._util` for the
`ib_insync.ContractDetails.tradingHours: str` for before i realized
there was a `.tradingSessions` property XD
- in `.ib_feed`,
* add various EG-collapsings per recent tractor/trio updates.
* better logging / exc-handling around ticker quote pushes.
* stop clearing `Ticker.ticks` each quote iteration; not sure if this
is needed/correct tho?
* add masked `Ticker.ticks` poll loop that logs.
- fix some `str.format()` usage in `._util.try_xdo_manual()`
You'd think they could be bothered to make either a "log" or "warning"
msg type instead of a `type='error'`.. but alas, this attempts to detect
all such "warning"-errors and never proxy them to the clearing engine
thus avoiding the cancellation of any associated (by `reqid`)
pre-existing orders (control dialogs).
Also update all surrounding log messages to a more multiline style.
Topically, throughout various (seemingly) console-UX-affecting or benign
spots in the code base; nothing that required more intervention beyond
things superficial. A few spots also include `trio.Nursery` ref renames
(always to something with a `tn` in it) and log-level reductions to
quiet (benign) console noise oriented around issues meant to be solved
long..
Note there's still a couple spots i left with the loose-ify flag because
i haven't fully tested them without using the latest version of
`tractor.trionics.collapse_eg()`, but more then likely they should flip
over fine.
Commit this change separate from the (original) broader set applied to
the entire code base since the `.deribit.api` mod contained changes from
upstream max-pain work (from our very own @nt) which caused a noticeable
conflict and intros un-required changes from his work to re-enable
`deribit` support.
Note the original commit, "69eac7bb Spurious first-draft of EG
collapsing", applied similar changes through the rest of the code base.
AGAIN, this mod's change is only being broken out to minimize upstream
change conflicts due to updates to the `deribit` backend done earlier in
time-history.
That is inside embedded `.accounting.calc.dyn_parse_to_dt()` closure add
an optional `_invalid: list` param to where we can report
bad-timestamped records which we instead override and return as
`from_timestamp(0.)` (when the parser loop falls through) and report
later (in summary ) from the `.accounting.calc.iter_by_dt()` caller
. Add some logging and an optional debug block for future tracing.
Since apparently porting to the new docker container enforces using
a vnc password and `asyncvnc` seems to have a bug/mis-config whenever
i've tried a pw over a wg tunnel..?
Soo, this tries out the old `i3ipc`-win-focus + `xdo` click hack when
the above fails.
Deats,
- add a mod-level `try_xdo_manual()` to wrap calling
`i3ipc_xdotool_manual_click_hack()` with an oserr handler, ensure we
don't bother trying if `i3ipc` import fails beforehand tho.
- call ^ from both the orig case block and the failover from the
vnc-client case.
- factor the `+no_setup_msg: str` out to mod level and expect it to be
`.format()`-ed.
- refresh todo around `asyncvnc` pw ish..
- add a new `i3ipc_fin_wins_titled()` window-title scanner which
predicates input `titles` and delivers any matches alongside the orig
focused win at call time.
- tweak `i3ipc_xdotool_manual_click_hack()` to call ^ and remove prior
unfactored window scanning logic.
The root daemon, pikerd, needs to be adjusted to use diff default
registry addrs to also utilize non-TCP, but for now this gets us started
testing; so far so good B)
That is to use the new `tractor.msg.types.Aid` struct to pull the
`brokerd` info from the `tractor.Channel.aid: Aid` attr as well as more
generally handling the new `Channel.raddr.proto_key: str` and no longer
assuming a TCP IPC transport; this per the recent `tractor.ipc`
subsys which adds multi-IPC-transports!
Downstream tweaks to match,
- use an "opt-in" field set to display in the `brokerd` info pane in
`.ui._feedstatus.mk_feed_label()`.
|_ also add some todos and drop some seemingly unneeded form sizing
calcs?
- tweak `.ui._label` to allow not using markdown, though ended up not
doing that since it looked too plain..
Using `tractor.trionics.collapse_eg()` as needed to avoid, at the least,
crash-worthy (in debug-mode REPL-ing terms) nested cancellation egs that
exhibit on SIGINT/ctl-c of each "app" (chart & daemon).
Also a bit of renaming of all `trio.Nursery`s to `tn`, the new "task
nursery" shorthand-var-name being used in all our other `tractor`
related projects.
Luckily all core deps are already ported so this was pretty easy!
B)
I've opted (via `tool.uv` settings) to prefer the user's system
(installed) python distro and disable auto-download of astral's
distros for now since I recently hit some strange silent core dumping
(`brokerd` actors just disappearing..)
with their binaries; an introspect showed it seemingly todo with
p_threading in cpython internals? We can figure out how to
better accommodate users with the opposite pref later, presumably
non-opinionated-linux hackers?
Core pkg upgrades of note,
- manually re-pinned most numerics libs including `numpy`, `numba`,
`pyarrow`.
- for AOT ext-libs (thanks to `uv.lock` being so detailed), new
`cython`, `llvmlite`, `cffi`, `rapidfuzz`, `uvloop`, `wrapt` and
`PyQt6` wheels pulled in.
- `cryptofeed` did a required bump to `2.4.0` looks like which also
required the above (and notable?) `cffi` update.
Namely requiring a `trio` that supports py3.13, so "trio >=0.27".
Unfortunately this brings in strict egs and drops various `trio`-related
sub-deps we also import in `piker`, like `trio-typing`. So there's a few
"rough edges", mostly todo with the REPL activating on graceful cancels
(SIGINT) of `piker` CLIs atm - due to the new strict-egs in recent
`trio`, but nothing we can't work out pretty quickly i'd imagine with
the new `tractor.collapse_eg()` stacker.
Note that we're pinning to `tractor`'s main branch for the moment since
it should be "stable" vs. the `repl_fixture` i'm likely running local Bp
Since I was trying out the neat lookin `polars-fuzzy-match` (also added
for now as a core dep here) which requires the new plugin sys, plus it's
about time we synced with upstream!
Adjust some column syntax to the new `.name` sub-field-space and the
`uv` lock-file to match.
Other,
- add back `trio-typing` bc i guess something else needs it (debug
tooling stuff in new `tractor`?)
- flip back to the `tractor` pre-main pin since the new `main`-branch
requires new `trio` stuff we haven't ported yet..
Such that the next time i inevitably must debug the some order-request
error status or precision discrepancy, i have the mkt-symbol branch
ready to go. Also, switch to `'action': 'buy'|'sell' as action,` style
`case` matching instead of the post-`if` predicate style.
Since we're not quite yet using automatic typed msging from
`tractor`/`msgspec` (i.e. still manually decoding order ctl msgs from
built-in types..`dict`s still not `msgspec.Struct`) this adds the
appropriate typecasting ops to ensure the required precision is attained
prior to processing and/or submission to a brokerd backend service.
For the `.clearing._ems`,
- flip all `trigger_price` previously presumed to be `float` to just
the field-identical `price: Decimal` and ensure we cast to `float`
for any `trigger_price` usage, like before passing to `mk_check()`.
For `.ui.order_mode.OrderMode`,
- add a new `.curr_mkt: MktPair` convenience property to get the
chart-active value.
- ensure we always use the `.curr_mkt.quantize() -> Decimal` before
setting any IPC-msg's `.price` field!
- always cast `float(Order.price)` before use in setting line-levels.
- don't bother setting `Order.symbol` to a (now fully removed) `Symbol`
instance since it's not really required-for-use anywhere; leaving it
a `str` (per the type-annot) is fine for now?
By re-typing to a `.price: Decimal` field on both legs of the EMS.
It seems we must do it ourselves since,
- these msg's (fields) are relayed through the clearing engine to each
`brokerd` backend and,
- bc many (if not all) of those backends `.broker`-clients (nor their
encapsulated "brokerage services") **are not** doing any
precision-truncation themselves.
So, for now, instead we opt to expect rounding at the source. This means
we will explicitly require casting to/from `float` at the line-graphics
interface to the order-clearing-engine (as implemented throughout
`.ui.order_mode.OrderMode`); and this is coming shortly.
Instead just check for the field (which i'm not huge on the key-name for
anyway) and if not found get the "last price" from the real-time shm
buffer's latest 'close' sample.
Unrelatedly, use a `subs.copy()` in the `Router.client_broadcast()` loop
such that if a `client_stream` is popped on connection failure, we don't
RTE for the "size changed on iteration".
A new class var `tuple[Exception]` such that the err set can be reffed
externally as needed for catching other similar pub-sub/IPC failures in
other (related) real-time sub-systems.
Also added some now-masked logging for debugging live-feed stream reading
issues that should ONLY be used for debugging since they'll greatly
degrade HFT perf. Used the new `log.mk_repr()` stuff (that one day we
should prolly pull from `modden` as a dep) for pretty console emissions.
Since now `tractor` will raise this native `trio`-exc translated from
a `Stop` msg when the peer gracefully terminates a `tractor.MsgStream`.
Just `info()` log in such cases versus continuing to warn for the
others.
Since currently we're only using this IPC subsys for `deribit`, and
generally speaking we're primarly supporting options markets (which are
fairly "slow moving"), flip to a default of NOT resetting the `NoBsWs`
on timeout since doing so normally breaks the jsron-rpc IPC session.
Without a proper `fixture` passed to `open_autorecon_ws()` (which we
should eventually implement!!) relying on a timeout-to-reset more or
less will just cause breakage issues - a proper reconnect sequence must
be implemented before using that feature.
Deats,
- expose and proxy through the `msg_recv_timeout` from
`open_jsonrpc_session()` into the underlying `open_autorecon_ws()`
call.
Add a doc-string reflecting recent refinements, drop all the old hook
params, rename `n: trio.Nursery` -> `tn` for "task nursery" fitting with
code base's naming style.
Namely handling backends which do not provide a default "frame
size-duration" in their init-config by making the backfiller guess the
value based on the first frame received.
Deats,
- adjust `start_backfill()` to take a more explicit
`def_frame_duration: Duration` expected to be unpacked from any
backend hist init-config by the `tsdb_backfill()` caller which now
also computes a value from the first received frame when the config
section isn't provided.
- in `start_backfill()` we now always expect the `def_frame_duration`
input and always decrement the query range by this value whenever
a `NoData` is raised by the provider-backend paired with an explicit
`log.warning()` about the handling.
- also relay any `DataUnavailable.args[0]` message from the provider
in the handler.
- repair "gap reporting" which checks for expected frame duration vs.
that received with much better humanized logging on the missing
segment using `pendulum.Interval/Duration.in_words()` output.
The `open_history_client()` provider endpoint can *optionally*
deliver a `frame_types: dict[int, pendulum.Duration]` subsection in its
`config: dict[str, dict]` (as was implemented with the `ib` backend).
This allows the `tsp` backfilling machinery to use this "recommended
frame duration" to subtract from the `last_start_dt` any time a `NoData`
gap is signalled by the `get_hist()` call allowing gaps to be ignored
safely without missing history by knowing the next earliest dt we can
query from using the `end_dt`. However, currently all crypto$ providers
haven't implemented this feat yet..
As such only try to use the `frame_types` feature if provided when
handling `NoData` conditions inside `tsp.start_backfill()` and otherwise
raise as normal.
Tyler Goodlet
Gud Boi