Commit Graph

3640 Commits (76cd5519b3a429c4aa7283d75a3244b8833e7f3c)

Author SHA1 Message Date
Tyler Goodlet 517c68f3ad Use `._pathops.slice_from_time()` for overlay intersects
It's way faster since it uses a uniform time arithmetic to narrow the
`numpy.searchsorted()` range before actually doing the index search B)
2023-03-10 18:20:22 -05:00
Tyler Goodlet ea84505682 Don't scale overlays on linked from display loop
In the (incrementally updated) display loop we have range logic that is
incrementally updated in real-time by streams, as such we don't really
need to update all linked chart's (for any given, currently updated
chart) y-ranges on calls of each separate (sub-)chart's
`ChartView.interact_graphics_cycle()`. In practise there are plenty of
cases where resizing in one chart (say the vlm fsps sub-plot) requires
a y-range re-calc but not in the OHLC price chart. Therefore
we always avoid doing more resizing then necessary despite it resulting
in potentially more method call overhead (which will later be justified
by better leveraging incrementally updated `Viz.maxmin()` and
`media_from_range()` calcs).
2023-03-10 18:20:22 -05:00
Tyler Goodlet 5eaca18ee0 Don't skip overlay scaling in disp-loop for now 2023-03-10 18:20:22 -05:00
Tyler Goodlet e06d4b405d Add linked charts guard-flag for use in display loop 2023-03-10 18:20:22 -05:00
Tyler Goodlet cf67c790e5 Use new cached median method in overlay scaling
Massively speeds up scaling transform cycles (duh).

Also includes a draft for an "overlay transform" type/api; obviously
still a WIP 🏄..
2023-03-10 18:20:22 -05:00
Tyler Goodlet ec8679ad74 Add `Viz.median_from_range()`
A super snappy `numpy.median()` calculator (per input range) which we
slap an `lru_cache` on thanks to handy dunder method hacks for such
things on mutable types XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 9418f53244 Speed up ranging in display loop
use the new `do_overlay_scaling: bool` since we know each feed will have
its own updates (cuz multiplexed by feed..) and we can avoid
ranging/scaling overlays that will make their own calls.

Also, pass in the last datum "brighter" color for ohlc curves as it was
originally (and now that we can pass that styling bit through).
2023-03-10 18:20:22 -05:00
Tyler Goodlet 497174c687 Add full profiling to `.interact_graphics_cycle()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 481f1b3d7e Fix intersect detection using time indexing
Facepalm, obviously absolute array indexes are not going to necessarily
align vs. time over multiple feeds/history. Instead use
`np.searchsorted()` on whatever curve has the smallest support and find
the appropriate index of intersection in time so that alignment always
starts at a sensible reference.

Also adds a `debug_print: bool` input arg which can enable all the
prints when working on this.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 776ffd2b1c Factor curve-dispersion sorting into primary loop
We can determine the major curve (in view) in the first pass of all
`Viz`s so drop the 2nd loop and thus the `mxmn_groups: dict`. Also
simplifies logic for the case of only one (the major) curve in view.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 896259d9e4 When only one curve is in view, skip group ranging 2023-03-10 18:20:22 -05:00
Tyler Goodlet 89e2e7fc54 Adjust `.update_graphics()` to expect `in_view: bool` in `_fsp.py` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 32f21dc06b Drop `update_graphics_from_flow()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet a0fb84f55b Just warn log on bad intersect indexing errors (for now) 2023-03-10 18:20:22 -05:00
Tyler Goodlet fc6ccc306c Only set the major curve's range once (per render cycle)
Turns out this is a limitation of the `ViewBox.setYRange()` api: you
can't call it more then once and expect anything but the first call to
be applied without letting a render cycle run. As such, we wait until
the end of the log-linear scaling loop to finally apply the major curves
y-mx/mn after all minor curves have been evaluated.

This also drops all the debug prints (for now) to get a feel for latency
in production mode.
2023-03-10 18:20:22 -05:00
Tyler Goodlet c2dd255e8a Only remove axis from scene when in one 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7e421ba57b Drop `.group_maxmin()`
We ended up doing groups maxmin sorting at the interaction layer (new
the view box) and thus this method is no longer needed, though it was
the reference for the code now in `ChartView.interact_graphics_cycle()`.

Further this adds a `remove_axes: bool` arg to `.insert_plotitem()`
which can be used to drop axis entries from the inserted pi (though it
doesn't seem like we really ever need that?) and does the removal in
a separate loop to avoid removing axes before they are registered in
`ComposedGridLayout._pi2axes`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 0591cb09f6 Clean up cross-curve intersect point indexing
When there are `N`-curves we need to consider the smallest
x-data-support subset when figuring out for each major-minor pair such
that the "shorter" series is always returns aligned to the longer one.

This makes the var naming more explicit with `major/minor_i_start` as
well as clarifies more stringently a bunch of other variables and
explicitly uses the `minor_y_intersect` y value in the scaling transform
calcs. Also fixes some debug prints.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 052ce65682 3rdz the charm: log-linearize minor y-ranges to a major
In very close manner to the original (gut instinct) attempt, this
properly (y-axis-vertically) aligns and scales overlaid curves according
to what we are calling a "log-linearized y-range multi-plot" B)

The basic idea is that a simple returns measure (eg. `R = (p1 - p0)
/ p0`) applied to all curves gives a constant output `R` no matter the
price co-domain in use and thus gives a constant returns over all assets
in view styled scaling; a intuitive visual of returns correlation. The
reference point is for now the left-most point in view (or highest
common index available to all curves), though we can make this
a parameter based on user needs.

A slew of debug `print()`s are left in for now until we iron out the
remaining edge cases to do with re-scaling a major (dispersion) curve
based on a minor now requiring a larger log-linear y-range from that
previous major' range.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 52ac1053aa 2nd try: dispersion normalize y-ranges around median
In the dispersion swing calcs, use the series median from the in-view
data to determine swing proportions to apply on each "minor curve"
(series with lesser dispersion the one with the greatest). Track the
major `Viz` as before by max dispersion. Apply the dispersion swing
proportions to each minor curve-series in a third loop/pass of all
overlay groups: this ensures all overlays are dispersion normalized in
their ranges but, minor curves are currently (vertically) centered (vs.
the major) via their medians.

There is a ton of commented code from attempts to try and vertically
align minor curves to the major via the "first datum" in-view/available.
This still needs work and we may want to offer it as optional.

Also adds logic to allow skipping margin adjustments in `._set_yrange()`
if you pass `range_margin=None`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet dfc35253ea First draft, group y-minmax transform algo
On overlaid ohlc vizs we compute the largest max/min spread and
apply that maxmimum "up and down swing" proportion to each `Viz`'s
viewbox in the group.

We obviously still need to clip to the shortest x-range so that
it doesn't look exactly the same as before XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8a5b9f4e8c Rename `.maybe_downsample_graphics()` -> `.interact_graphics_cycle()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet f89e11fc7d Right, handle y-ranging multiple paths per plot
We were hacking this before using the whole `ChartView._maxmin()`
setting stuff since in some cases you might want similarly ranged paths
on the same view, but of course you need to max/min them together..

This adds that group sorting by using a table of `dict[PlotItem,
tuple[float, float]` and taking the abs highest/lowest value for each
plot in the viz interaction update loop.

Also removes the now commented signal registry calls and thus
`._yranger`, drops the `set_range: bool` from `._set_yrange` and adds
and extra `.maybe_downsample_graphics()` to the mouse wheel handler to
avoid a weird slow debounce where ds-ing is delayed until a further
interaction.
2023-03-10 18:20:22 -05:00
Tyler Goodlet fc73becd5f Drop Qt interaction signal usage
It's kind of hard to understand with the C++ fan-out to multiple views
(imo a cluster-f#$*&) and seems honestly just plain faster to loop (in
python) through all the linked view handlers XD

Core adjustments:
- make the panning and wheel-scroll handlers just call
  `.maybe_downsample_graphics()` directly; drop all signal emissions.
- make `.maybe_downsample_graphics()` loop through all vizs per subchart
  and use the new pipeline-style call sequence of:
  - `Viz.update_graphics() -> <read_slc>: tuple`
  - `Viz.maxmin(i_read_range=<read_slc>) -> yrange: tuple`
  - `Viz.plot.vb._set_yrange(yrange=yrange)`
  which inlines all the necessary calls in the most efficient way whilst
  leveraging `.maxmin()` caching and ymxmn-from-m4-during-render to
  boot.
- drop registering `._set_yrange()` for handling `.sigRangeChangedManually`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 223e9d999c Add first-draft `PlotItemOverlay.group_maxmin()`
Computes the maxmin values for each underlying plot's in-view range as
well as the max up/down swing (in percentage terms) from the plot with
most dispersion and returns a all these values plus a `dict` of plots to
their ranges as part of output.
2023-03-10 18:20:22 -05:00
goodboy eb51033b18
Merge pull request #483 from pikers/service_subpkg
`.service` subpkg
2023-03-10 10:37:46 -05:00
Tyler Goodlet 12883c3c90 Don't double send `enable_modules` and `debug_mode` in kwargs..
This broke non-disti-mode actor tree spawn / runtime, seemingly because
the cli entrypoint for a `piker chart` also sends these values down
through the call stack independently? Pretty sure we don't need to send
the `enable_modules` from the chart actor anyway.
2023-03-10 10:30:26 -05:00
Tyler Goodlet 8ceaa27872 Add ES client polling to ensure eventual connectivity.. 2023-03-09 18:42:24 -05:00
Tyler Goodlet 97290fcb05 Never drop root perms in test harness 2023-03-09 18:42:24 -05:00
Tyler Goodlet 44a3115539 Expose `drop_root_perms_for_ahab` from `pikerd` factories to `ahabd` 2023-03-09 18:42:24 -05:00
Tyler Goodlet 0772b4a0fa Hard code version from our container, predicate renaming 2023-03-09 18:42:24 -05:00
Tyler Goodlet 15064d94cb `ahabd`: Harden cancellation teardown (again XD)
Needed to move the startup sequence inside the `try:` block to guarantee
we always do the (now shielded) `.cancel()` call if we get a cancel
during startup.

Also, support an optional `started_afunc` field in the config if
backends want to just provide a one-off blocking async func to sync
container startup. Add a `drop_root_perms: bool` to allow persisting
sudo perms for testing or dyanmic container spawning purposes.
2023-03-09 18:42:24 -05:00
Tyler Goodlet 9a00c45923 Add `log` fixture for easy test plugin 2023-03-09 18:42:24 -05:00
Tyler Goodlet 7cc9911565 Add connection poll loop to es test as well 2023-03-09 15:37:43 -05:00
Tyler Goodlet 79b0db4449 Pass a config `tmp_dir: Path` to the runtime when testing 2023-03-09 15:37:43 -05:00
Tyler Goodlet 5aaa7f47dc Pull testing config dir from `tractor` runtime vars
Provides a more correct solution (particularly for distributed testing)
to override the `piker` configuration directory by reading the path from
a specific `tractor._state._runtime_vars` entry that can be provided by
the test harness.

Also fix some typing and comments.
2023-03-09 15:37:43 -05:00
Tyler Goodlet aa36abf36e Support passing `tractor` "actor runtime vars" down the runtime 2023-03-09 15:37:43 -05:00
Tyler Goodlet 2014019b06 Add reconnect loop to `marketstore` startup test
Due to making ahabd supervisor init more async we need to be more
tolerant to mkts server startup: the grpc machinery needs to be up
otherwise a client which connects to early may just hang on requests..

Add a reconnect loop (which might end up getting factored into client
code too) so that we only block on requests once we know the client
connection is actually responsive.
2023-03-09 15:37:43 -05:00
Tyler Goodlet 75b7a8b56e `marketstore`: Pull default socket from server config 2023-03-09 15:37:43 -05:00
Tyler Goodlet 31392af427 Move enabled module defs to just above where used 2023-03-09 15:37:43 -05:00
Tyler Goodlet 6540c415c1 Lul, fix imports in elasticsearch block.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet fbc12b1b07 Add 10min timeout on CI job.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet cda7a54718 Fix final missed `marketstore` mod import
Thanks @esme! XD

Also, do a linter pass and remove a buncha unused references.
2023-03-09 15:37:43 -05:00
Tyler Goodlet 6f92c6b52d Don't crash on a `xdotool` timeout.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet 441243f83b Attempt to report `piker storage -d <fqsn>` errors
Not really sure there's much we can do besides dump Grpc stuff when we
detect an "error" `str` for the moment..

Either way leave a buncha complaints (como siempre) and do linting
fixups..
2023-03-09 15:37:43 -05:00
Tyler Goodlet cec2967071 Import `maybe_open_pikerd` at module level 2023-03-09 15:37:43 -05:00
Tyler Goodlet f95ea19b21 Move `pikerd` runtime boostrap to `.service._actor_runtime` 2023-03-09 15:37:43 -05:00
Tyler Goodlet eca048c0c5 Move daemon spawning endpoints to `service._daemon` module 2023-03-09 15:37:43 -05:00
Tyler Goodlet a2d40937a3 Move actor-discovery utils to `.service._registry 2023-03-09 15:37:42 -05:00
Tyler Goodlet 31f2b01c3e Move `Services` api to `.service._mngr` mod 2023-03-09 15:37:42 -05:00