Commit Graph

2776 Commits (8f764542d5b34fa3902763732d28b9c519ab4055)

Author SHA1 Message Date
Tyler Goodlet 8f764542d5 Make `Viz.incr_info()` sample rate agnostic
Mainly it was the global (should we )increment logic that needs to be
independent for the fast vs. slow chart such that the slow isn't
update-shifted by the fast and vice versa. We do this using a new
`'i_last_slow'` key in the `DisplayState.globalz: dict` which is
singleton for each sample-rate-specific chart and works for both time
and array indexing.

Also, we drop some old commented `graphics.draw_last_datum()` code that
never ended up being needed again inside the coordinate cache reset
bloc.
2023-01-30 11:49:06 -05:00
Tyler Goodlet 5892c79737 Use array-`int`-indexing on single feed
Might as well since it makes the chart look less gappy and we can easily
flip the index switch now B)

Also adds a new `'i_slow_last'` key to `DisplayState` for a singleton
across all slow charts and thus no more need for special case logic in
`viz.incr_info()`.
2023-01-30 11:49:06 -05:00
Tyler Goodlet 95e7c8648c Align step curves the same as OHLC bars 2023-01-30 11:49:06 -05:00
Tyler Goodlet 1b258d1bd6 Add `IncrementalFormatter.x_offset: np.ndarray`
Define the x-domain coords "offset" (determining the curve graphics
per-datum placement) for each formatter such that there's only on place
to change it when needed. Obviously each graphics type has it's own
dimensionality and this is reflected by the array shapes on each
subtype.
2023-01-30 11:49:06 -05:00
Tyler Goodlet f79efd4ca9 Adjust OHLC bar x-offsets to be time span matched
Previously we were drawing with the middle of the bar on each index with
arms to either side: +/- some arm length. Instead this changes so that
each bar is drawn *after* each index/timestamp such that in graphics
coords the bar span more correctly matches the time span in the
x-domain. This makes the linked region between slow and fast chart
directly match (without any transform) for epoch-time indexing such that
the last x-coord in view on the fast chart is no more then the
next time step in (downsampled) slow view.

Deats:
- adjust in `._pathops.path_arrays_from_ohlc()` and take an `bar_w` bar
  width input (normally taken from the data step size).
- change `.ui._ohlc.bar_from_ohlc_row()` and
  `BarItems.draw_last_datum()` to match.
2023-01-30 11:49:06 -05:00
Tyler Goodlet 6412b4ab5a `Viz._index_field` a `typing.Literal[str]` 2023-01-30 11:49:06 -05:00
Tyler Goodlet 045a1a6acc Set `path_arrays_from_ohlc(use_time_index=True)` on epoch indexing
Allows easily switching between normal array `int` indexing and time
indexing by just flipping the `Viz._index_field: str`.

Also, guard all the x-data audit breakpoints with a time indexing
condition.
2023-01-30 11:49:06 -05:00
Tyler Goodlet da3f74e28e Ugh, use `bool` flag to determine index field.. 2023-01-30 11:49:06 -05:00
Tyler Goodlet 9dd4945580 Make `LinearRegion` link using epoch-time index
Turned out to be super simple to get the first draft to work since the
fast and slow chart now use the same domain, however, it seems like
maybe there's an offset issue still where the fast may be a couple
minutes ahead of the slow?

Need to dig in a bit..
2023-01-30 11:49:06 -05:00
Tyler Goodlet cb78f0921c Add global `i_step` per overlay to `DisplayState`
Using a global "last index step" (via module var) obviously has problems
when working with multiple feed sets in a single global app instance:
any separate feed-set will be incremented according to an app-global
index-step and thus won't correctly calc per-feed-set-step update info.

Impl deatz:
- drop `DisplayState.incr_info()` (since previously moved to `Viz`) and
  call that method on each appropriate `Viz` instance where necessary;
  further ensure the appropriate `DisplayState` instance is passed in to
  each call and make sure to pass a `state: DisplayState`.
- add `DisplayState.hist_vars: dict` for history chart (sets) to
  determine the per-feed (not set) current slow chart (time) step.
- add `DisplayState.globalz: dict` to house a common per-feed-set state
  and use it inside the new `Viz.incr_info()` such that
  a `should_increment: bool` can be returned and used by the display
  loop to determine whether to x-shift the current chart.
2023-01-30 11:49:06 -05:00
Tyler Goodlet bd23d254da Move `DisplayState.incr_info()` -> `Viz` 2023-01-30 11:49:06 -05:00
Tyler Goodlet 89b6fde1ef Move `Viz` layer to new `.ui` mod 2023-01-30 11:49:06 -05:00
Tyler Goodlet 95325b22f0 Fix line -> bars on 6x UPPX
Read the `Viz.index_step()` directly to avoid always reading 1 on the
slow chart; this was completely broken before and resulting in not
rendering the bars graphic on the slow chart until at a true uppx of
1 which obviously doesn't work for 60 width bars XD

Further cleanups to `._render` module:
- drop `array` output from `Renderer.render()`, `read_from_key` input
  and fix type annot.
- drop `should_line`, `changed_to_line` and `render_kwargs` from
  `render_baritems()` outputs and instead calc `should_redraw` logic
  inside the func body and return as output.
2023-01-30 11:49:06 -05:00
Tyler Goodlet c9f2ca3f76 Drop unused `read_src_from_key: bool` to `.format_to_1d()` 2023-01-30 11:49:06 -05:00
Tyler Goodlet 9152babf2d Right, do index lookup for int-index as well.. 2023-01-30 11:49:06 -05:00
Tyler Goodlet b056e4bf17 Fix formatter xy ndarray first prepend case
First allocation vs. first "prepend" of source data to an xy `ndarray`
format **must be mutex** in order to avoid a double prepend.

Previously when both blocks were executed we'd end up with
a `.xy_nd_start` that was decremented (at least) twice as much as it
should be on the first `.format_to_1d()` call which is obviously
incorrect (and causes problems for m4 downsampling as discussed below).
Further, since the underlying `ShmArray` buffer indexing is managed
(i.e. write-updated) completely independently from the incremental
formatter updates and internal xy indexing, we can't use
`ShmArray._first.value` and instead need to use the particular `.diff()`
output's prepend length value to decrement the `.xy_nd_start` on updates
after initial alloc.

Problems this resolves with m4:
- m4 uses a x-domain diff to calculate the number of "frames" to
  downsample to, this is normally based on the ratio of pixel columns on
  screen vs. the size of the input xy data.
- previously using an int-index (not epoch time) the max diff between
  first and last index would be the size of the input buffer and thus
  would never cause a large mem allocation issue (though it may have
  been inefficient in terms of needed size).
- with an epoch time index this max diff could explode if you had some
  near-now epoch time stamp **minus** an x-allocation value: generally
  some value in `[0.5, -0.5]` which would result in a massive frames and
  thus internal `np.ndarray()` allocation causing either a crash in
  `numba` code or actual system mem over allocation.

Further, put in some more x value checks that trigger breakpoints if we
detect values that caused this issue - we'll remove em after this has
been tested enough.
2023-01-30 11:49:06 -05:00
Tyler Goodlet 49acaf12d1 Handle time-indexing for fill arrows
Call into a reworked `Flume.get_index()` for both the slow and fast
chart and do time index clipping to last datum where necessary.
2023-01-30 11:49:06 -05:00
Tyler Goodlet ff09625d83 Restore coord-cache resetting
Turns out we can't seem to avoid the artefacts when click-drag-scrolling
(results in weird repeated "smeared" curve segments) so just go back to
the original code.
2023-01-30 11:49:06 -05:00
Tyler Goodlet 28ca38ba87 Add some commented debug prints for default fmtr 2023-01-30 11:49:06 -05:00
Tyler Goodlet dc81d82d37 Slicec to an extra index around each timestamp input 2023-01-30 11:49:06 -05:00
Tyler Goodlet 8a295eba51 Drop passing `render_data` to `Curve.draw_last_datum()` 2023-01-30 11:49:06 -05:00
Tyler Goodlet 0516e7d6f2 Add back `.default_view()` slice logic for `int` indexing 2023-01-30 11:49:06 -05:00
Tyler Goodlet b9844f020f Block out `do_print` stuff inside `Viz.maxmin()` 2023-01-30 11:49:06 -05:00
Tyler Goodlet 44a3285367 Implement `stop_t` gap adjustments; the good lord said it is the problem 2023-01-30 11:49:05 -05:00
Tyler Goodlet bd5eae5636 Draw last datums on boot
Ensures that a "last datum" graphics object exists so that zooming can
read it using `.x_last()`. Also, disable the linked region stuff for now
since it's totally borked after flipping to the time indexing.
2023-01-30 11:49:05 -05:00
Tyler Goodlet 42ff039ce7 Use `Curve.x_last()` for zoom focal point 2023-01-30 11:49:05 -05:00
Tyler Goodlet 9d8617d04b Delegate to `Viz.default_view()` on chart
Also add a rage print to not forget about the global index
tracking/diffing in the display loop we still need to change.
2023-01-30 11:49:05 -05:00
Tyler Goodlet b4e4f914e9 Re-implement `.default_view()` on `Viz`
Since we don't really need it defined on the "chart widget" move it to
a viz method and rework it to hell:

- always discard the invalid view l > r case.
- use the graphic's UPPX to determine UI-to-scene coordinate scaling for
  the L1-label collision detection, if there is no L1 just offset by
  a few (index step scaled) datums; this allows us to drop the 2x
  x-range calls as was hacked previous.
- handle no-data-in-view cases explicitly and error if we get any
  ostensibly impossible cases.
- expect caller to trigger a graphics cycle if needed.

Further support this includes a rework a slew of other important
details:

- add `Viz.index_step`, an idempotent computed, index (presumably uniform)
  step value which is needed for variable sample rate graphics displayed
  on an epoch (second) time index.
- rework `Viz.datums_range()` to pass view x-endpoints as first and last
  elements in return `tuple`; tighten up snap-to-data edge case logic
  using `max()`/`min()` calls and better internal var naming.
- adjust all calls to `slice_from_time()` to not expect an "abs" slice.
- drop all `.yrange` resetting since we can just have the `Renderer` do
  it when necessary.
2023-01-30 11:49:05 -05:00
Tyler Goodlet 7cfb05f72a Add gap detection for `stop_t`, though only report atm 2023-01-30 11:49:05 -05:00
Tyler Goodlet c68a3a98f3 Add `.x_last()` meth to flow graphics 2023-01-30 11:49:05 -05:00
Tyler Goodlet 901e565dd1 Drop `Flume.view_data()` 2023-01-30 11:49:05 -05:00
Tyler Goodlet dcf774d1ff Drop old breakpoint 2023-01-30 11:49:05 -05:00
Tyler Goodlet 184a4d0730 Drop `_slice_from_time()` 2023-01-30 11:49:05 -05:00
Tyler Goodlet 1a45aa96c9 Use uniform step arithmetic in `slice_from_time()`
If we presume that time indexing using a uniform step we can calculate
the exact index (using `//`) for the input time presuming the data
set has zero gaps. This gives a massive speedup over `numpy` fancy
indexing and (naive) `numba` iteration. Further in the case where time
gaps are detected, we can use `numpy.searchsorted()` to binary search
for the nearest expected index at lower latency.

Deatz,
- comment-disable the call to the naive `numba` scan impl.
- add a optional `step: int` input (calced if not provided).
- add todos for caching binary search results in the gap detection
  cases.
- drop returning the "absolute buffer indexing" slice since the caller
  can always just use the read-relative slice to acquire it.
2023-01-30 11:49:05 -05:00
Tyler Goodlet 097f16f158 Make `.default_view()` time step aware
When we use an epoch index and any sample rate > 1s we need to scale the
"number of bars" to that step in order to place the view correctly in
x-domain terms. For now we're calcing the step in-method but likely,
longer run, we'll pull this from elsewhere (like a ``Viz`` attr).
2023-01-30 11:49:05 -05:00
Tyler Goodlet d0a0a4b4dd Flip over to epoch-time based x-domain indexing 2023-01-30 11:49:05 -05:00
Tyler Goodlet d895684235 Adjust all `slice_from_time()` calls to not expect mask 2023-01-30 11:49:05 -05:00
Tyler Goodlet 074ef078e8 Rewrite `slice_from_time()` using `numba`
Gives approx a 3-4x speedup using plain old iterate-with-for-loop style
though still not really happy with this .5 to 1 ms latency..

Move the core `@njit` part to a `_slice_from_time()` with a pure python
func with orig name around it. Also, drop the output `mask` array since
we can generally just use the slices in the caller to accomplish the
same input array slicing, duh..
2023-01-30 11:49:05 -05:00
Tyler Goodlet 0003e53ff5 Use index (time) step to calc OHLC bar/line uppx threshold 2023-01-30 11:49:05 -05:00
Tyler Goodlet 5648210710 Use step size to determine bar gaps 2023-01-30 11:49:05 -05:00
Tyler Goodlet 02ce03c191 Use step size to determine last datum bar gap 2023-01-30 11:49:05 -05:00
Tyler Goodlet a87f062a26 Move `Flume.slice_from_time()` to `.data._pathops` mod func 2023-01-30 11:49:05 -05:00
Tyler Goodlet 152c9e2c98 Drop `index_field` input to renders, add `.read()` profiling 2023-01-30 11:49:05 -05:00
Tyler Goodlet 35186ac3b1 Delegate formatter `.index_field` to the parent `Viz` 2023-01-30 11:49:05 -05:00
Tyler Goodlet 8b26473375 Facepalm**2: fix array-read-slice, like actually..
We need to subtract the first index in the array segment read, not the
first index value in the time-sliced output, to get the correct offset
into the non-absolute (`ShmArray.array` read) array..

Further we **do** need the `&` between the advance indexing conditions
and this adds profiling to see that it is indeed real slow (like 20ms
ish even when using `np.where()`).
2023-01-30 11:49:05 -05:00
Tyler Goodlet e1670cd45c Markup OHLC->path gen with `numba` issue # 2023-01-30 11:49:05 -05:00
Tyler Goodlet 69641b0679 Facepalm: put graphics cycle in `do_ds: bool` block.. 2023-01-30 11:49:05 -05:00
Tyler Goodlet 3ddb0f49e2 TOSQUASH: 552a8c298cd (return index for arrow..) 2023-01-30 11:49:05 -05:00
Tyler Goodlet 082cf2b1ea Facepalm: actually return latest index on time slice fail.. 2023-01-30 11:49:05 -05:00
Tyler Goodlet 6f65607296 Go with explicit `.data._m4` mod name
Since it's a notable and self-contained graphics compression algo, might
as well give it a dedicated module B)
2023-01-30 11:49:05 -05:00