Since we only ever want to do incremental y-range calcs based on the
price always skip any tick types emitted by the data daemon which aren't
defined in the fundamental set. Further, toss in a new `debug_n_trade:
bool` toggle which by default turns off all loggin and profiler calls;
if you want to do profiling this has to now be adjusted manually!
Not sure how this lasted so long without complaint (literally since we
added history 1m OHLC it seems; guess it means most backends are pretty
tolerant XD ) but we've been sending 2 cancels per order (dialog) due to
the mirrored lines on each chart: 1s and 1m. This fixes that by
reworking the `OrderMode` methods to be a bit more sane and less
conflated with the graphics (lines) layer.
Deatz:
- add new methods:
- `.oids_from_lines()` line -> oid extraction,
- `.cancel_orders()` which makes the order client cancel requests from
a `oids: list[str]`.
- re-impl `.cancel_all_orders()` and `.cancel_orders_under_cursor()` to
use the above methods thus fixing the original bug B)
Discovered due to originally having a history loading bug between
btcusdt futes display where the same time series was being loaded into
the graphics system, this avoids the issue where 2 (or more) curves are
measured to have the same dispersion and thus do not get added as unique
entries to the `overlay_table: dict[float, tuple]` during the scaling
phase..
Practically speaking this should never really be a problem if the curves
(and their backing timeseries) are indeed unique but keying the
overlay table by the dispersion and the `Viz` is a minimal performance
hit when looping the sorted table and is a lot nicer then you **do want
to show** duplicate curves then having one overlay just not be ranged
correctly at all XD
Also adjust sizing such that the history buffer will backfill the last
six years by default (in 1m OHLC) and the hft buffer will do only 3 days
worth. Also ensure the fsp layer passes the src shm's buffer size when
allocating since the size is now required by allocators in the shm apis.
Avoid unnecessarily re-rendering the wrong (1min OHLC history) chart
and/or other such charts with update tasks listening to the sampler
stream. Instead only redraw in tasks which are updating vizs which match
the actual details of the backfill event.
We can probably also eventually match against a range tuple (emitted in
the msg) and then have the task further only update the formatter layer
unless the range is actually in view?
It's no longer part of the default OHLCV array-buffer schema and just
generally we should be processing and managing **any** non source data
in the FSP subsystem(s) despite it maybe being provided as a default by
some backends.
Turns out no backend (including kraken) requires it and really this
kinda of measure should be implemented and recorded from our fsp layer
instead of (hackily) sometimes expecting it to be in "source data".
Was borked on linux if you didn't provide the setting in `conf.toml` due
to some logic errors. Fix that by rejigging `DpiAwareFont` internal
variables:
- add new `._font_size_calc_key: str` which was the old `._font_size`
and is only used when no explicit font size is set by the user in the
`conf.toml` config:
- this is the "key" that is used to lookup a calculation function
which attempts to compute a best fit font size given the measured
system displays DPI settings and dimensions.
- make the `._font_size: int` the **actual** font size integer that is
cached and passed to `Qt` to set the size.
- this is overridden by user config now if defined.
- change the input kwarg `font_size: str` to the constructor to better
change the input kwarg `font_size: str` to the constructor to better
named private `_font_size_key: str` which gets set to the new
`._font_size_calc_key`.
Also, adjust all client code which instantiates `DpiAwareFont` to use
the new `_font_size_key` kwarg input so nothing breaks XD
Was only really borked for higher-precision but lower priced assets
(like TLOS or peeneez) which have a `MktPair.price_tick_digits >= 2`.
The issue was using the wrong attr, the `size_tick_digits`..
Including changing to `LinkedSplits.mkt: MktPair` and adding an explicit
setter method for setting it and being sure that nothing breaks
in the display system init!
For this commit we leave in warning access to `LinkedSplits.symbol` but
will remove in following commit.
Stash it for now in the (now mutable by default) `.shm_write_opts` and
have the new `Flume._has_vlm: bool` (only set to false internally by
feed layer) which can be read via new public `.has_vlm()` predicate.
Move out the old `.ui/_fsp` helper logic to this flume method.
Order mode previously was just willy-nilly sending `float` prices
(particularly on order edits) which are generated from the associated
level line. This actually uses the `MktPair.price_tick: Decimal` to
ensure the value is rounded correctly before submission to the ems..
Also adjusts the order mode init to expect a table of tables of startup
position messages, with the inner table being keyed by fqme per msg.
We previously only offered a sync API (which was recently renamed to
`.<meth>_nowait()` style) since initially all order control was from our
`OrderMode` Qt driven UI/UX. This adds the equivalent async methods for
both testing as well as eventual auto-strat driven control B)
Also includes a bunch of renaming:
- `OrderBook` -> `OrderClient`.
- better internal renaming of the client's mem chan vars and add a ref
`._ems_stream: tractor.MsgStream`.
- drop `get_orders()` factory, just always check for the actor-global
instance and always set the ems stream on that client (in case old one
was closed).
Add a logic branch for now that switches on an instance check.
Generally swap over all `Position.symbol` and `Transaction.sym` refs to
`MktPair`. Do a wholesale rename of all `.bsuid` var names to
`.bs_mktid`.
Solve this by always scaling the y-range for the major/target curve
*before* the final overlay scaling loop; this implicitly always solve
the case where the major series is the only one in view.
Tidy up debug print formatting and add some loop-end demarcation comment
lines.
This is particularly more "good looking" when we boot with a pair that
doesn't have historical 1s OHLC and thus the fast chart is empty from
outset. In this case it's a lot nicer to be already zoomed to
a comfortable preset number of "datums in view" even when the history
isn't yet filled in.
Adjusts the chart display `Viz.default_view()` startup to explicitly
ensure this happens via the `do_min_bars=True` flag B)
For the purposes of eventually trying to resolve last-step indexing
synchronization (an intermittent but still existing) issue(s) that can
happen due to races during history frame query and shm writing during
startup. In fact, here we drop all `hist_viz` info queries from the main
display loop for now anticipating that this code will either be removed
or improved later.
Again, as per the signature change, never expect implicit time step
calcs from overlay processing/machinery code. Also, extend the debug
printing (yet again) to include better details around
"rescale-due-to-minor-range-out-of-view" cases and a detailed msg for
the transform/scaling calculation (inputs/outputs), particularly for the
cases when one of the curves has a lesser support.
As per the change to `slice_from_time()` this ensures this `Viz` always
passes its self-calculated time indexing step size to the time slicing
routine(s).
Further this contains a slight impl tweak to `.scalars_from_index()` to
slice the actual view range from `xref` to `Viz.ViewState.xrange[1]` and
then reading the corresponding `yref` from the first entry in that
array; this should be no slower in theory and makes way for further
caching of x-read-range to `ViewState` opportunities later.
When the target pinning curve (by default, the dispersion major) is
shorter then the pinned curve, we need to make sure we find still find
the x-intersect for computing returns scalars! Use `Viz.i_from_t()` to
accomplish this as well and, augment that method with a `return_y: bool`
to allow the caller to also retrieve the equivalent y-value at the
requested input time `t: float` for convenience.
Also tweak a few more internals around the 'loglin_ref_to_curve'
method:
- only solve / adjust for the above case when the major's xref is
detected as being "earlier" in time the current minor's.
- pop the major viz entry from the overlay table ahead of time to avoid
a needless iteration and simplify the transform calc phase loop to
avoid handling that needless cycle B)
- add much better "organized" debug printing with more clear headers
around which "phase"/loop the message pertains and well as more
explicit details in terms of x and y-range values on each cycle of
each loop.
Previously when very zoomed out and using the `'r'` hotkey the
interaction handler loop wouldn't trigger a re-(up)sampling to get
a more detailed curve graphic and instead the previous downsampled
(under-detailed) graphic would show. Fix that by ensuring we yield back
to the Qt event loop and do at least a couple render cycles with paired
`.interact_graphics_cycle()` calls.
Further this flips the `.start/signal_ic()` methods to use the new
`.reset_graphics_caches()` ctr-mngr method.
Instead delegate directly to `Viz.default_view()` throughout charting
startup and interaction handlers.
Also add a `ChartPlotWidget.reset_graphics_caches()` context mngr which
resets all managed graphics object's cacheing modes on enter and
restores them on exit for simplified use in interaction handling code.
This finally seems to mitigate all the "smearing" and "jitter" artifacts
when using Qt's "coordinate cache" graphics-mode:
- whenever we're in a mouse interaction (as per calls to
`ChartView.start/signal_ic()`) we simply disable the caching mode (set
`.NoCache` until the interaction is complete.
- only do this (for now) during a pan since it doesn't seem to be an
issue when zooming?
- ensure disabling all `Viz.graphics` and `.ds_graphics` to be agnostic
to any case where there's both a zoom and a pan simultaneously (not
that it's easy to do manually XD) as well as solving the problem
whenever an OHLC series is in traced-and-downsampled mode (during low
zoom).
Impl deatz:
- rename `ChartView._ic` -> `._in_interact: trio.Event`
- add `.ChartView._interact_stack: ExitStack` which we use to open.
and close the `FlowGraphics.reset_cache()` mngrs from mouse handlers.
- drop all the commented per-subtype overrides for `.cache_mode: int`.
- write up much better doc strings for `FlattenedOHLC` and `StepCurve`
including some very basic ASCII-art diagrams.
When the minor has the same scaling as the major in a given direction we
should still do back-scaling against the major-target and previous
minors to avoid strange edge cases where only the target-major might not
be shifted correctly to show an matched intersect point? More or less
this just meant making the y-mxmn checks interval-inclusive with
`>=`/`<=` operators.
Also adds a shite ton of detailed comments throughout the pin-to-target
method blocks and moves the final major y-range call outside the final
`scaled: dict` loop.
For the "pin to target major/target curve" overlay method, this finally
solves the longstanding issue of ensuring that any new minor curve,
which requires and increase in the major/target curve y-range, also
re-scales all previously scaled minor curves retroactively. Thus we now
guarantee that all minor curves are correctly "pinned" to their
target/major on their earliest available datum **and** are all kept in
view.
Yah yah, i know it's the same as before (the N > 2 curves case with
out-of-range-minor rescaling the previously scaled curves isn't fixed
yet...) but, this is a much better and optional implementation in less
code. Further we're now better leveraging various new cached properties
and methods on `Viz`.
We now handle different `overlay_technique: str` options using `match:`
syntax in the 2ndary scaling loop, stash the returns scalars per curve
in `overlay_table`, and store and iterate the curves by dispersion
measure sort order.
Further wrt "pin-to-target-curve" mode, which currently still pins to
the largest measured dispersion curve in the overlay set:
- pop major Ci overlay table entries at start for sub-calcs usage when
handling the "minor requires major rescale after pin" case.
- (finally) correctly rescale the major curve y-mxmn to whatever the
latest minor overlay curve by calcing the inverse transform from the
minor *at that point*:
- the intersect point being that which the minor has starts support on
the major's x-domain* using the new `Viz.scalars_from_index()` and,
- checking that the minor is not out of range (versus what the major's
transform calcs it to be, in which case,
- calc the inverse transform from the current out-of-range minor and
use it to project the new y-mxmn for the major/target based on the
same intersect-reference point in the x-domain used by the minor.
- always handle the target-major Ci specially by only setting the
`mx_ymn` / `mx_ymn` value when iterating that entry in the overlay
table.
- add todos around also doing the last sub-sub bullet for all previously
major-transform scaled minor overlays (this is coming next..i hope).
- add a final 3rd overlay loop which goes through a final `scaled: dict`
to apply all range values to each view; this is where we will
eventually solve that last edge case of an out-of-range minor's
scaling needing to be used to rescale already scaled minors XD
In an effort to make overlay calcs cleaner and leverage caching of view
range -> dispersion measures, this adds the following new methods:
- `._dispersion()` an lru cached returns scalar calculator given input
y-range and y-ref values.
- `.disp_from_range()` which calls the above method and returns variable
output depending on requested calc `method: str`.
- `.i_from_t()` a currently unused cached method for slicing the
in-view's array index from time stamp (though not working yet due to
needing to parameterize the cache by the input `.vs.xrange`).
Further refinements/adjustments:
- rename `.view_state: ViewState` -> `.vs`.
- drop the `.bars_range()` method as it's no longer used anywhere else
in the code base.
- always set the `ViewState.in_view: np.ndarray` inside `.read()`.
- return the start array index (from slice) and `yref` value @ `xref`
from `.scalars_from_index()` to aid with "pin to curve" rescaling
caused by out-of-range pinned-minor curves.
Adds a small struct which is used to track the most recently viewed
data's x/y ranges as well as the last `Viz.read()` "in view" array data
for fast access by chart related graphics processing code, namely view
mode overlay handling.
Also adds new `Viz` interfaces:
- `Viz.ds_yrange: tuple[float, float]' which replaces the previous
`.yrange` (now set by `.datums_range()` on manual y-range calcs) so
that the m4 downsampler can set this field specifically and then it
get used (when available) by `Viz.maxmin()`.
- `Viz.scalars_from_index()` a new returns-scalar generator which can be
used to calc the up and down returns values (used for scaling overlay
y-ranges) from an input `xref` x-domain index which maps to some
`Ci(xref) = yref`.
It was getting waayy to long to be jammed in a method XD
This moves all the chart-viz iteration and transform logic into a new
`piker.ui.view_mode.overlay_viewlists()` core routine which will make it
a lot nicer for,
- AOT compilation via `numba` / `cython` / `mypyc`.
- decoupling from the `pyqtgraph.ViewBox` APIs if we ever decide to get
crazy and go without another graphics engine.
- keeping your head clear when trying to rework the code B)
As part of solving a final bullet-issue in #455, which is specifically
a case:
- with N > 2 curves, one of which is the "major" dispersion curve" and
the others are "minors",
- we can run into a scenario where some minor curve which gets pinned to
the major (due to the original "pinning technique" -> "align to
major") at some `P(t)` which is *not* the major's minimum / maximum
due to the minor having a smaller/shorter support and thus,
- requires that in order to show then max/min on the minor curve we have
to expand the range of the major curve as well but,
- that also means any previously scaled (to the major) minor curves need
to be adjusted as well or they'll not be pinned to the major the same
way!
I originally was trying to avoid doing the recursive iteration back
through all previously scaled minor curves and instead decided to try
implementing the "per side" curve dispersion detection (as was
originally attempted when first starting this work). The idea is to
decide which curve's up or down "swing in % returns" would determine the
global y-range *on that side*. Turns out I stumbled on the "align to
first" technique in the process: "for each overlay curve we align its
earliest sample (in time) to the same level of the earliest such sample
for whatever is deemed the major (directionally disperse) curve in
view".
I decided (with help) that this "pin to first" approach/style is equally
as useful and maybe often more so when wanting to view support-disjoint
time series:
- instead of compressing the y-range on "longer series which have lesser
sigma" to make whatever "shorter but larger-sigma series" pin to it at
an intersect time step, this instead will expand the price ranges
based on the earliest time step in each series.
- the output global-returns-overlay-range for any N-set of series is equal to
the same in the previous "pin to intersect time" technique.
- the only time this technique seems less useful is for overlaying
market feeds which have the same destination asset but different
source assets (eg. btceur and btcusd on the same chart since if one
of the series is shorter it will always be aligned to the earliest
datum on the longer instead of more naturally to the intersect sample
level as was in the previous approach).
As such I'm going to keep this technique as discovered and will later
add back optional support for the "align to intersect" approach from
previous (which will again require detecting the highest dispersion
curve direction-agnostic) and pin all minors to the price level at which
they start on the major.
Further details of the implementation rework in
`.interact_graphics_cycle()` include:
- add `intersect_from_longer()` to detect and deliver a common datum
from 2 series which are different in length: the first time-index
sample in the longer.
- Rewrite the drafted `OverlayT` to only compute (inversed log-returns)
transforms for a single direction and use 2 instances, one for each
direction inside the `Viz`-overlay iteration loop.
- do all dispersion-per-side major curve detection in the first pass of
all `Viz`s on a plot, instead updating the `OverlayT` instances for
each side and compensating for any length mismatch and
rescale-to-minor cases in each loop cycle.
Previously we were aligning the child's `PlotItem` to the "root" (top
most) overlays `ViewBox`..smh. This is why there was a weird gap on the
LHS next to the 'left' price axes: something weird in the implied axes
offsets was getting jammed in that rect.
Also comments out "the-skipping-of" moving axes from the overlay's
`PlotItem.layout` to the root's linear layout(s) when an overlay's axis
is read as not visible; this isn't really necessary nor useful and if we
want to remove the axes entirely we should do it explicitly and/or
provide a way through the `ComposeGridLayout` API.
Tyler Goodlet
Ebisu