Commit Graph

2170 Commits (bc2ebbc457850b20599df43b3b66bd7929e02c2f)

Author SHA1 Message Date
Tyler Goodlet bc2ebbc457 WIP get incremental step curve updates working
This took longer then i care to admit XD but it definitely adds a huge
speedup and with only a few outstanding correctness bugs:

- panning from left to right causes strange trailing artifacts in the
  flows fsp (vlm) sub-plot but only when some data is off-screen on the
  left but doesn't appear to be an issue if we keep the `._set_yrange()`
  handler hooked up to the `.sigXRangeChanged` signal (but we aren't
  going to because this makes panning way slower). i've got a feeling
  this is a bug todo with the device coordinate cache stuff and we may
  need to report to Qt core?
- factoring out the step curve logic from
  `FastAppendCurve.update_from_array()` (un)fortunately required some
  logic branch uncoupling but also meant we needed special input controls
  to avoid things like redraws and curve appends for special cases,
  this will hopefully all be better rectified in code when the core of
  this method is moved into a renderer type/implementation.
- the `tina_vwap` fsp curve now somehow causes hangs when doing erratic
  scrolling on downsampled graphics data. i have no idea why or how but
  disabling it makes the issue go away (ui will literally just freeze
  and gobble CPU on a `.paint()` call until you ctrl-c the hell out of
  it). my guess is that something in the logic for standard line curves
  and appends on large data sets is the issue?

Code related changes/hacks:
- drop use of `step_path_arrays_from_1d()`, it was always a bit hacky
  (being based on `pyqtgraph` internals) and was generally hard to
  understand since it returns 1d data instead of the more expected (N,2)
  array of "step levels"; instead this is now implemented (uglily) in
  the `Flow.update_graphics()` block for step curves (which will
  obviously get cleaned up and factored elsewhere).
- add a bunch of new flags to the update method on the fast append
  curve:  `draw_last: bool`, `slice_to_head: int`, `do_append: bool`,
  `should_redraw: bool` which are all controls to aid with previously
  mentioned issues specific to getting step curve updates working
  correctly.
- add a ton of commented tinkering related code (that we may end up
  using) to both the flow and append curve methods that was written as
  part of the effort to get this all working.
- implement all step curve updating inline in `Flow.update_graphics()`
  including prepend and append logic for pre-graphics incremental step
  data maintenance and in-view slicing as well as "last step" graphics
  updating.

Obviously clean up commits coming stat B)
2022-04-30 13:28:43 -04:00
Tyler Goodlet ecefc74d07 Drop cursor debounce delay, decrease rate limit 2022-04-30 13:28:43 -04:00
Tyler Goodlet 9505350ff0 Downsample on every uppx inrement since it's way faster 2022-04-30 13:28:43 -04:00
Tyler Goodlet 15fe46091b Drop log scaling support since uppx driven scaling seems way faster/better 2022-04-30 13:28:43 -04:00
Tyler Goodlet e726a3ebef Hipshot, use uppx to drive theoretical px w 2022-04-30 13:28:43 -04:00
Tyler Goodlet ba30b5d9bf Fix null match 2022-04-30 13:28:43 -04:00
Tyler Goodlet 447549e1af WIP incrementally update step array format 2022-04-30 13:28:43 -04:00
Tyler Goodlet 5ca0d1a19d Always maybe render graphics
Since we have in-view style rendering working for all curve types
(finally) we can avoid the guard for low uppx levels and without losing
interaction speed. Further don't delay the profiler so that the nested
method calls correctly report upward - which wasn't working likely due
to some kinda GC collection related issue.
2022-04-30 13:28:43 -04:00
Tyler Goodlet 2c3da825e3 Always set coords cache on curves 2022-04-30 13:28:43 -04:00
Tyler Goodlet a752a22a4b Handle null output case for vlm chart mxmn 2022-04-30 13:28:43 -04:00
Tyler Goodlet 55606cffbb Right, handle the case where the shm prepend history isn't full XD 2022-04-30 13:28:43 -04:00
Tyler Goodlet d3251a2922 Always use coord cache, add naive view range diffing logic 2022-04-30 13:28:43 -04:00
Tyler Goodlet dc4a0a6fd2 Put mxmn profile mapping at end of method 2022-04-30 13:28:43 -04:00
Tyler Goodlet 04c12a756c If a sample stream is already ded, just warn 2022-04-30 13:28:43 -04:00
Tyler Goodlet cde23361a4 `FastAppendCurve`: Only render in-view data if possible
More or less this improves update latency like mad. Only draw data in
view and avoid full path regen as much as possible within a given
(down)sampling setting. We now support append path updates with in-view
data and the *SPECIAL CAVEAT* is that we avoid redrawing the whole curve
**only when** we calc an `append_length <= 1` **even if the view range
changed**. XXX: this should change in the future probably such that the
caller graphics update code can pass a flag which says whether or not to
do a full redraw based on it knowing where it's an interaction based
view-range change or a flow update change which doesn't require a full
path re-render.
2022-04-30 13:28:43 -04:00
Tyler Goodlet 3c58847595 Remove `._set_yrange()` handler from x-range-change signal 2022-04-30 13:28:43 -04:00
Tyler Goodlet fae249dd2f Delegate graphics cycle max/min to chart/flows 2022-04-30 13:28:43 -04:00
Tyler Goodlet 6f5bb9cbe0 Incrementally update flattend OHLC data
After much effort (and exhaustion) but failure to get a view into
our `numpy` OHLC struct-array, this instead allocates an in-thread-memory
array which is updated with flattened data every flow update cycle.

I need to report what I think is a bug to `numpy` core about the whole
view thing not working but, more or less this gets the same behaviour
and minimizes work to flatten the sampled data for line-graphics
drawing thus improving refresh latency when drawing large downsampled
curves.

TL;DR:
- add `ShmArray.ustruct()` to return a **copy of** (since a view doesn't
  work..) the (field filtered) shm array which is the same index-length
  as the source data.
- update the OHLC ds curve with view aware data sliced out from the
  pre-allocated and incrementally updated data (we had to add a last
  index var `._iflat` to track appends - this should be moved into
  a renderer eventually?).
2022-04-30 13:28:43 -04:00
Tyler Goodlet b4c7d02fcb Add `FastAppendCurve.draw_last()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet b2697bfd13 Don't require data input to constructor 2022-04-30 13:28:43 -04:00
Tyler Goodlet 14037cd1dc More WIP, implement `BarItems` rendering in `Flow.update_graphics()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet a935245360 Add `BarItems.draw_last()` and disable `.update_from_array()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet 8a5538f490 WIP starting architecture doc str writeup.. 2022-04-30 13:28:43 -04:00
Tyler Goodlet 90b9c12d25 WIP incremental render apis 2022-04-30 13:28:43 -04:00
Tyler Goodlet aab1a3c565 Warn before return lul 2022-04-30 13:28:43 -04:00
Tyler Goodlet 6c136e82b4 Port view downsampling handler to new update apis 2022-04-30 13:28:43 -04:00
Tyler Goodlet 8388918af0 Port ui components to use flows, drop all late assignments of shm 2022-04-30 13:28:43 -04:00
Tyler Goodlet 82dbdd6148 Add new `ui._flows` module
This begins the removal of data processing / analysis methods from the
chart widget and instead moving them to our new `Flow` API (in the new
module introduce here) and delegating the old chart methods to the
respective internal flow. Most importantly is no longer storing the
"last read" of an array from shm in an internal chart table (was
`._arrays`) and instead the `ShmArray` instance is passed as input and
stored in the `Flow` instance. This greatly simplifies lookup logic such
that the display loop now doesn't have to worry about reading shm, it
can be done by internal graphics logic as desired. Generally speaking,
all previous `._arrays`/`._graphics` lookups are now delegated to the
entries in the chart's `._flows` table.

The new `Flow` methods are generally better factored and provide more
detailed output regarding data-stream <-> graphics inter-relations for
the future purpose of allowing much more efficient update calls in the
display loop as well as supporting low latency interaction UX.

The concept here is that we're introducing an intermediary layer that
ties together graphics and real-time data flows such that widget code is
oriented around plot layout and the flow apis are oriented around
real-time low latency updates and providing an efficient high level
metric layer for the UX.

The summary api transition is something like:
- `update_graphics_from_array()` -> `.update_graphics_from_flow()`
- `.bars_range()` -> `Flow.datums_range()`
- `.bars_range()` -> `Flow.datums_range()`
2022-04-30 13:28:43 -04:00
Tyler Goodlet 79eff13e76 Add back fqsn passthrough and feed opening 2022-04-30 13:28:24 -04:00
Tyler Goodlet 8e5f5b6be6 Implement `open_history_client()` correctly for `kraken` 2022-04-30 13:28:24 -04:00
Tyler Goodlet 8110c4c70d Add to signal broker won't deliver more data 2022-04-30 13:28:24 -04:00
Tyler Goodlet 6f64ff5842 Relay frame size in `NoData` due to null-result history 2022-04-30 13:28:24 -04:00
Tyler Goodlet bd23b6e8f2 Add , indicates hist size to decrement to storage logic 2022-04-30 13:28:24 -04:00
Tyler Goodlet 49f3e15a3c Truncate trade rate wma window sizes 2022-04-30 13:28:24 -04:00
Tyler Goodlet c985c01c62 More tolerance for "stream-ended-early" conditions in quote throttler 2022-04-30 13:28:24 -04:00
Tyler Goodlet 8e11d79712 Drop legacy back-filling logic
Use the new `open_history_client()` endpoint/API and expect backends to
provide a history "getter" routine that can be called to load historical
data into shm even when **not** using a tsdb. Add logic for filling in
data from the tsdb once the backend has provided data up to the last
recorded in the db. Add logic for avoiding overruns of the shm buffer
with more-then-necessary queries of tsdb data.
2022-04-30 13:28:24 -04:00
Tyler Goodlet 48cce42c77 Drop legacy backfilling, load a day's worth of data by default 2022-04-30 13:28:24 -04:00
Tyler Goodlet 1fd3513689 Add 16 fetch limit if no tsdb data found 2022-04-30 13:28:24 -04:00
Tyler Goodlet 9d375a0ce5 Don't require a symbol to subcmd 2022-04-30 13:28:24 -04:00
Tyler Goodlet c18795e454 Don't open a feed, write or read ohlc in for now 2022-04-30 13:28:24 -04:00
Tyler Goodlet 76287a7523 Don't run legacy backfill when isn't up 2022-04-30 13:28:24 -04:00
Tyler Goodlet b0f659a66b Start legacy backfill with partial too 2022-04-30 13:28:24 -04:00
Tyler Goodlet f7b3215aa4 Unify backfilling logic into common task-routine 2022-04-30 13:28:24 -04:00
Tyler Goodlet c5ed9b5955 Add `Storage.load()` and `.write_ohlcv()` 2022-04-30 13:28:23 -04:00
Tyler Goodlet ea9d76ffce A bit hacky but, broadcast index streams on each history prepend 2022-04-30 13:28:23 -04:00
Tyler Goodlet 16f2f6ff94 Factor subscription broadcasting into a func 2022-04-30 13:28:23 -04:00
Tyler Goodlet 4a383795bf Always update ohlc (main source chart) on `trigger_all=True` 2022-04-30 13:28:23 -04:00
Tyler Goodlet ef8e71f628 Make ib history client expect datetimes for input 2022-04-30 13:28:23 -04:00
Tyler Goodlet 21d0d551d3 Add binance history client support with datetime use throughout 2022-04-30 13:28:23 -04:00
Tyler Goodlet 32c3f63cfd Add `docker` as `tsdb` extras dep 2022-04-30 13:24:12 -04:00