Commit Graph

3112 Commits (5affad942ffe81f71e41bea5f20115d3e179d469)

Author SHA1 Message Date
Tyler Goodlet ee831baeb3 Display loop mega-cleanup
The most important changes include:
- iterating the new `Flow` type and updating graphics
- adding detailed profiling
- increasing the min uppx before graphics updates are throttled
- including the L1 spread in y-range calcs so that you never have the
  bid/ask go "out of view"..
- pass around `Flow`s instead of shms
- drop all the old prototyped downsampling code
2022-04-30 11:36:23 -04:00
Tyler Goodlet 7c615a403b Allow passing a `plotItem` to `.draw_curve()`
If manually managing an overlay you'll likely call `.overlay_plotitem()`
and then a plotting method so we need to accept a plot item input so
that the chart's pi doesn't get assigned incorrectly in the `Flow` entry
(though it is by default if no input is provided).

More,
- add a `Flow.graphics` field and set it to the `pg.GraphicsObject`.
- make `Flow.maxmin()` return `None` in the "can't calculate" cases.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b8374dbe9a Fsp UI initialization updates
- set shm refs on `Flow` entries.
- don't run a graphics cycle on 'update' msgs from the engine
  if the containing chart is hidden.
- drop `volume` from flows map and disable auto-yranging
  once $vlm comes up.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 454cd7920d Disconnect signals in `ChartView.disable_auto_yrange()`
Allows for removing resize callbacks for a flow/overlay that you wish to
remove from view (eg. unit volume after dollar volume is up) and thus
less general interaction callback overhead for any plot you don't wish
to show or resize.

Further,
- drop the `autoscale_linked_plots` block for now since with
  multi-view-box overlays each register their own vb resize slots
- pull the graphics object from the chart's `Flow` map inside
  `.maybe_downsample_graphics()`
2022-04-30 11:36:23 -04:00
Tyler Goodlet ca283660de Fix bug where if `yrange` was passed the mxmin callback was still used.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet d4eddbdb25 Guard against zero px width 2022-04-30 11:36:23 -04:00
Tyler Goodlet eec329a221 Add `Flow` type with a real chitty mxmn cacheing method
This new type wraps a shm data flow and will eventually include things
like incremental path-graphics updates and serialization + bg downsampling
techniques. The main immediate motivation was to get a cached y-range max/min
calc going since profiling revealed the `numpy` equivalents were
actually quite slow as the data set grows large. Likely we can use all
this to drive a streaming mx/mn routine that's always launched as part
of each on-host flow.

This is our official foray into use of `msgspec.Struct` B) and I have to
say, pretty impressed; we'll likely completely ditch `pydantic` from
here on out.
2022-04-30 11:36:23 -04:00
Tyler Goodlet a1de097a43 Loop for first graphic with xvec 2022-04-30 11:36:23 -04:00
Tyler Goodlet b5f2558cec Only `.maybe_downsample_graphics()` on manual changes
We don't need update graphics on every x-range change since that's what
the display loop does. Instead, only on manual changes do we make manual
calls into `.update_graphics_from_array()` and be sure to iterate all
linked subplots and all their embedded graphics.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 1a95712680 Don't return early on ds line render to avoid breaking profiling
The pg profiler seems to have trouble with early `return`s in function
calls (likely muckery with the GC/`.__delete__()`) so let's just try
to avoid it for now until we either fix it (probably by implementing as
a ctx mngr) or use diff one.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b20e9e58ee Use HL tracer by default, seems to be faster? 2022-04-30 11:36:23 -04:00
Tyler Goodlet 4bc2bbda69 Allow passing "ms slower then" value on cli to `--profile` 2022-04-30 11:36:23 -04:00
Tyler Goodlet b524929cb6 Only bail up pan updates if uppx > 16 2022-04-30 11:36:23 -04:00
Tyler Goodlet f95d22bfd3 Delegate `BarItems.x_uppx()` to internal ds curve 2022-04-30 11:36:23 -04:00
Tyler Goodlet 91de281b7e Downsample curves even less frequently 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2284e61eda Only pass vr for bars, allow source vb in autorange 2022-04-30 11:36:23 -04:00
Tyler Goodlet 082b02776c Drop the unit-volume chart once $vlm is fully drawn 2022-04-30 11:36:23 -04:00
Tyler Goodlet 27e3d0ef80 Ensure we update the volume array, not graphics
Ugh, turns out the wacky `ChartView.maxmin` callback stuff we did (for
determining y-range sizings) currently requires that the volume array
has a "bars in view" result.. so let's make that keep working without
rendering the graphics for the curve (since we're disabling them once
$vlm comes up).
2022-04-30 11:36:23 -04:00
Tyler Goodlet eeca9eb4c7 Add `.update_graphics_from_array()` flags for setting view-range use and graphics rendering 2022-04-30 11:36:23 -04:00
Tyler Goodlet 9bbfa4be02 Guard against zero px width 2022-04-30 11:36:23 -04:00
Tyler Goodlet ce85031ef2 Given in-view rendering, make bars downsample on uppx >= 8 2022-04-30 11:36:23 -04:00
Tyler Goodlet b6f852e0ad Make `FastAppendCurve` optionally view range aware
As with the `BarItems` graphics, this makes it possible to pass in a "in
view" range of array data that can be *only* rendered improving
performance for large(r) data sets. All the other normal behaviour is
kept (i.e a persistent, (pre/ap)pendable path can still be maintained)
if a ``view_range`` is not provided.

Further updates,
- drop the `.should_ds_or_redraw()` and `.maybe_downsample()` predicates
 instead moving all that logic inside `.update_from_array()`.
- disable the "cache flipping", which doesn't seem to be needed to avoid
  artifacts any more?
- handle all redraw/dowsampling logic in `.update_from_array()`.
- even more profiling.
- drop path `.reserve()` stuff until we better figure out how it's
  supposed to work.
2022-04-30 11:36:23 -04:00
Tyler Goodlet fdd5aa33d2 Fix view range array to include most recent (facepalm) 2022-04-30 11:36:23 -04:00
Tyler Goodlet 82732e3f17 TOQUASH: drop display loop old .update_ohlc_.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2c1daab990 Port to new `.update_graphics_from_array()`, pause quote updates on chart interaction 2022-04-30 11:36:23 -04:00
Tyler Goodlet a9e1c6c50e Make panning pause feeds, call into update method from downsampler cb loop 2022-04-30 11:36:23 -04:00
Tyler Goodlet ef03b8e987 Attempt only rendering ohlc bars in view and ds-ing otherwise 2022-04-30 11:36:23 -04:00
Tyler Goodlet 3b90b1f960 Unify into a single update method: `.update_graphics_from_array()` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1cf6ba789c Remove units vlm cuve once the $vlm one comes up 2022-04-30 11:36:23 -04:00
Tyler Goodlet 49c25eeef4 Index must be int bro.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5bcd6ac494 Move px width log scaling into `ds_m4()` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5da9f7fdb4 Add more frequent ds steps when zooming out; use profiler gt 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5128e4c304 Make `BarItems` use our line curve for downsampling
Drop all the logic originally in `.update_ds_line()` which is now done
internal to our `FastAppendCurve`. Add incremental update of the
flattened OHLC -> line curve (unfortunately using `np.concatenate()` for
the moment) and maintain a new `._ds_line_xy` arrays tuple which keeps
the internal state. Add `.maybe_downsample()` as per the new interaction
update method requirement. Draft out some fast path curve stuff like in
our line graphic. Short-circuit bars path updates when we downsample to
line. Oh, and add a ton more profiling in prep for getting
all this stuff faf.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 6410c68e2e Add global profile timeout var 2022-04-30 11:36:23 -04:00
Tyler Goodlet 947a514153 Add "native" downsampling to our `FastAppendCurve`
Build out an interface that makes it super easy to downsample curves
using the m4 algorithm while keeping our incremental `QPainterPath`
update feature. A lot of hard work and tinkering went into getting this
working all in-thread correctly and there are quite a few details..

New interface methods:
- `.x_uppx()` which returns the x-axis "view units per pixel"
- `.px_width()` which returns the total (rounded) x-axis pixels spanned
    by the curve in view.
- `.should_ds_or_redraw()` a predicate which checks internal state to
  see if either downsampling of the curve should take place, or the curve
  should have all downsampling removed and be redrawn with source array
  data.
- `.downsample()` the actual ds processing routine which delegates into
  the m4 algo impl.
- `.maybe_downsample()` a simple update method which can be called by
  the view box when the user changes the zoom level.

Implementation details/changes:

- make `.update_from_array()` check for downsample (or revert to source
  aka de-downsample) conditions exist and then downsample and re-draw
  path graphics accordingly.
- in order to even further speed up path appends (since our main
  bottleneck is measured to be `QPainter.drawPath()` calls with large
  paths which are frequently updates), add a secondary path `.fast_path`
  which is the path that is real-time updates by incremental appends and
  which is painted separately for speed in `.pain()`.
- drop all the `QPolyLine` stuff since it was tested to be much slower
  in general and especially so for append-updates.
- stop disabling the cache settings on updates since it doesn't seem to
  be required any more?
- more move toward deprecating and removing all lingering interface
  requirements from `pg.PlotCurveItem` (like `.xData`/`.yData`).
- adjust `.paint()` and `.boundingRect()` to compensate for the new
  `.fast_path`
- add a butt-load of profiling B)
2022-04-30 11:36:23 -04:00
Tyler Goodlet 8627f6f6c5 Add no-path guard now that we can use a poly 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5800c10901 First try, drop `FastAppendCurve` inheritance from `pg.PlotCurveItem` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 28bf8853aa Drop commented line from pq method copy/paste 2022-04-30 11:36:23 -04:00
Tyler Goodlet 86da64c2c2 Show baseline bars length on in view read < 6 2022-04-30 11:36:23 -04:00
Tyler Goodlet d59442e3b1 Bump up resolution log scaling a mag 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5e161aa251 Always clear previous downsample curve on switch
Pretty sure this was most of the cause of the stale (more downsampled)
curves showing when zooming in and out from bars mode quickly. All this
stuff needs to get factored out into a new abstraction anyway, but
i think this get's mostly correct functionality.

Only draw new ds curve on uppx steps >= 4 and stop adding/removing
graphics objects from the scene; doesn't seem to speed anything up
afaict. Add better reporting of ds scale changes.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 9b2ec871a0 Clear ds line graphics on switch back to bars 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2b12742992 More ems resiliency: discard broken client dialogs 2022-04-30 11:36:23 -04:00
Tyler Goodlet b262532fd4 Allocate m4 output arrays in `numba` code, avoid segfaults? 2022-04-30 11:36:23 -04:00
Tyler Goodlet 561d7e0349 Only clear/redraw curve on uppx diffs > 2
Only if the uppx increases by more then 2 we redraw the entire line
otherwise just ds with previous params and update the current curve.
This *should* avoid strange lower sample rate artefacts from showing on
updates.

Summary:
- stash both uppx and px width in `._dsi` (downsample info)
- use the new `ohlc_to_m4_line()` flags
- add notes about using `.reserve()` and friends
- always delete last `._array` ref prior to line updates
2022-04-30 11:36:23 -04:00
Tyler Goodlet 3a6c5a2fbd Try supporting reuse of path allocation 2022-04-30 11:36:23 -04:00
Tyler Goodlet 88a7314bd0 Add optional mxmn HL tracer support to m4 sampler 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1abe513ecb Add our own `FastAppendCurve.clear()`, try mem reso
In an effort to try and make `QPainterPath.reserve()` work, add internal
logic to use the same object without de-allocating memory from
a previous path write/creation.

Note this required the addition of a `._redraw` flag (to be used in
`.clear()` and a small patch to `pyqtgraph.functions.arrayToQPath` to
allow passing in an existing path (thus reusing the same underlying mem
alloc) which will likely be first pushed to our fork.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 44f3a08ef1 Add optional uppx log scaling to m4 sampler
We were previously ad-hoc scaling up the px count/width to get more
detail at lower uppx values. Add a log scaling sigmoid that range scales
between 1 < px_width < 16.

Add in a flag to use the mxmn OH tracer in `ohlc_flatten()` if desired.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 03e0e3e76b Delegate to m4 ohlc helper for curve, only ds on uppx steps > 2 2022-04-30 11:36:23 -04:00