Commit Graph

2197 Commits (incr_update_backup)

Author SHA1 Message Date
Tyler Goodlet 8c1905e35a Fix less-then-frame off by one slice, add db write toggle and disable 2022-05-06 13:09:30 -04:00
Tyler Goodlet d9e2666e80 More reliable `marketstored` + container supervision
It turns out (i guess not so shockingly?) that `marketstore` doesn't
always teardown "gracefully" under SIGINT (seems to hang if there are
open client connections which are also in the midst of teardown?) so
this instead first tries the SIGINT and then fails over to a SIGKILL
(destroy loop) which seems to be much more reliable to ensure shutdown
without any downside - in terms of a "hard kill".

Originally i was thinking the issue was root perms related (which get
relegated solely to the `marketstored` daemon actor after spawn) but
actually it was indeed the signalling / application layer causing the
hold-up/latency on teardown. There's a bunch of lingering (now
commented) code which tried to solve this non-problem as well as a bunch
logging/prints to help decipher the root of the issue - this will all
get cleaned out shortly.
2022-05-05 21:04:10 -04:00
Tyler Goodlet 1abe7d87a5 Include epoch timestamp in quote label for now 2022-05-05 17:09:17 -04:00
Tyler Goodlet 2f9d199a7f Handle ``iter_dts()`` already exhausted edge case 2022-05-03 19:07:32 -04:00
Tyler Goodlet 2f3418546f Label "humanized" sample period in window title-bar" 2022-05-03 17:27:38 -04:00
Tyler Goodlet 729c72a48f Add timeframe key to seconds map 2022-05-03 16:30:10 -04:00
Tyler Goodlet 324dcbbfb0 Always write newly pulled frames to tsdb 2022-05-03 16:22:01 -04:00
Tyler Goodlet 112cba43e5 Fix slice logic for less-then-frame tsdb overlap
When the tsdb has a last datum that is in the past less then a "frame's
worth" of sample steps we need to slice out only the data from the
latest frame that doesn't overlap; this fixes that slice logic..
Previously i dunno wth it was doing..
2022-05-03 16:01:02 -04:00
Tyler Goodlet 468cd3a381 Handle no sampler subs case on history broadcasts
When the market isn't open the feed layer won't create a subscriber
entry in the sampler broadcast loop and so if a manual call to
``broadcast()`` is made (like when trying to update a chart from
a history prepend) we need to handle that case and just broadcast
a random `-1` for now..BD
2022-05-03 13:52:23 -04:00
Tyler Goodlet e8aaf42cc6 Only assert if input array actually has a size 2022-05-03 13:51:29 -04:00
Tyler Goodlet 1967bc7973 Support async-batched ohlc queries in all backends
Expect each backend to deliver a `config: dict[str, Any]` which provides
concurrency controls to `trimeter`'s batch task scheduler such that
backends can define their own concurrency limits.

The dirty deats in this patch include handling history "gaps" where
a query returns a history-frame-result which spans more then the typical
frame size (in seconds). In such cases we reset the target frame index
(datetime index sequence implemented with a `pendulum.Period`) using
a generator protocol `.send()` such that the sequence can be dynamically
re-indexed starting at the new (possibly) pre-gap datetime. The new gap
logic also allows us to detect out of order frames easier and thus wait
for the next-in-order to arrive before making more requests.
2022-05-03 13:50:34 -04:00
Tyler Goodlet 303a5cc66c Support large ohlcv writes via slicing, add struct array keymap 2022-05-03 11:22:47 -04:00
Tyler Goodlet 2e6b7da4bc Add back mx/mn updates for L1-in-view, lost during rebase 2022-05-02 11:40:53 -04:00
Tyler Goodlet 69d0555180 Clean out legacy code from `Flow.update_graphics()` 2022-05-01 19:13:21 -04:00
Tyler Goodlet 8915f0c0c9 Terminate early on data unavailable errors 2022-05-01 19:12:48 -04:00
Tyler Goodlet 946d554716 Add first-draft `trimeter` based concurrent ohlc history fetching 2022-05-01 12:40:51 -04:00
Tyler Goodlet 8711465d96 Drop `bar_wap` curve for now, seems to also be causing hangs?! 2022-04-30 13:28:43 -04:00
Tyler Goodlet 0f53450c68 Add profiler passthrough type annot, comments about appends vs. uppx 2022-04-30 13:28:43 -04:00
Tyler Goodlet 1c91cf56de An absolute uppx diff of >= 1 seems more then fine 2022-04-30 13:28:43 -04:00
Tyler Goodlet 4129677ec5 Up the display throttle rate to 22Hz 2022-04-30 13:28:43 -04:00
Tyler Goodlet aa8f9c02f2 Add support for no `._first.value` update shm prepends 2022-04-30 13:28:43 -04:00
Tyler Goodlet c282555c2f Double up shm buffer size 2022-04-30 13:28:43 -04:00
Tyler Goodlet f4f6ffe819 Only do curve appends on low uppx levels 2022-04-30 13:28:43 -04:00
Tyler Goodlet 3b15f83e70 Startup up with 3k bars 2022-04-30 13:28:43 -04:00
Tyler Goodlet d968f4897f Drop vwap fsp for now; causes hangs.. 2022-04-30 13:28:43 -04:00
Tyler Goodlet ac57396acd Drop step routine import 2022-04-30 13:28:43 -04:00
Tyler Goodlet 9df96cd182 Drop uppx guard around downsamples on interaction
Since downsampling with the more correct version of m4 (uppx driven
windows sizing) is super fast now we don't need to avoid downsampling
on low uppx values. Further all graphics objects now support in-view
slicing so make sure to use it on interaction updates. Pass in the view
profiler to update method calls for more detailed measuring.

Even moar,
- Add a manual call to `.maybe_downsample_graphics()` inside the mouse
  wheel event handler since it seems that sometimes trailing events get
  lost from the `.sigRangeChangedManually` signal which can result in
  "non-downsampled-enough" graphics on chart given the scroll amount;
  this manual call seems to entirely fix this?
- drop "max zoom" guard since internals now support (near) infinite
  scroll out to graphics becoming a single pixel column line XD
- add back in commented xrange signal connect code for easy testing to
  verify against range updates not happening without it
2022-04-30 13:28:43 -04:00
Tyler Goodlet bc2ebbc457 WIP get incremental step curve updates working
This took longer then i care to admit XD but it definitely adds a huge
speedup and with only a few outstanding correctness bugs:

- panning from left to right causes strange trailing artifacts in the
  flows fsp (vlm) sub-plot but only when some data is off-screen on the
  left but doesn't appear to be an issue if we keep the `._set_yrange()`
  handler hooked up to the `.sigXRangeChanged` signal (but we aren't
  going to because this makes panning way slower). i've got a feeling
  this is a bug todo with the device coordinate cache stuff and we may
  need to report to Qt core?
- factoring out the step curve logic from
  `FastAppendCurve.update_from_array()` (un)fortunately required some
  logic branch uncoupling but also meant we needed special input controls
  to avoid things like redraws and curve appends for special cases,
  this will hopefully all be better rectified in code when the core of
  this method is moved into a renderer type/implementation.
- the `tina_vwap` fsp curve now somehow causes hangs when doing erratic
  scrolling on downsampled graphics data. i have no idea why or how but
  disabling it makes the issue go away (ui will literally just freeze
  and gobble CPU on a `.paint()` call until you ctrl-c the hell out of
  it). my guess is that something in the logic for standard line curves
  and appends on large data sets is the issue?

Code related changes/hacks:
- drop use of `step_path_arrays_from_1d()`, it was always a bit hacky
  (being based on `pyqtgraph` internals) and was generally hard to
  understand since it returns 1d data instead of the more expected (N,2)
  array of "step levels"; instead this is now implemented (uglily) in
  the `Flow.update_graphics()` block for step curves (which will
  obviously get cleaned up and factored elsewhere).
- add a bunch of new flags to the update method on the fast append
  curve:  `draw_last: bool`, `slice_to_head: int`, `do_append: bool`,
  `should_redraw: bool` which are all controls to aid with previously
  mentioned issues specific to getting step curve updates working
  correctly.
- add a ton of commented tinkering related code (that we may end up
  using) to both the flow and append curve methods that was written as
  part of the effort to get this all working.
- implement all step curve updating inline in `Flow.update_graphics()`
  including prepend and append logic for pre-graphics incremental step
  data maintenance and in-view slicing as well as "last step" graphics
  updating.

Obviously clean up commits coming stat B)
2022-04-30 13:28:43 -04:00
Tyler Goodlet ecefc74d07 Drop cursor debounce delay, decrease rate limit 2022-04-30 13:28:43 -04:00
Tyler Goodlet 9505350ff0 Downsample on every uppx inrement since it's way faster 2022-04-30 13:28:43 -04:00
Tyler Goodlet 15fe46091b Drop log scaling support since uppx driven scaling seems way faster/better 2022-04-30 13:28:43 -04:00
Tyler Goodlet e726a3ebef Hipshot, use uppx to drive theoretical px w 2022-04-30 13:28:43 -04:00
Tyler Goodlet ba30b5d9bf Fix null match 2022-04-30 13:28:43 -04:00
Tyler Goodlet 447549e1af WIP incrementally update step array format 2022-04-30 13:28:43 -04:00
Tyler Goodlet 5ca0d1a19d Always maybe render graphics
Since we have in-view style rendering working for all curve types
(finally) we can avoid the guard for low uppx levels and without losing
interaction speed. Further don't delay the profiler so that the nested
method calls correctly report upward - which wasn't working likely due
to some kinda GC collection related issue.
2022-04-30 13:28:43 -04:00
Tyler Goodlet 2c3da825e3 Always set coords cache on curves 2022-04-30 13:28:43 -04:00
Tyler Goodlet a752a22a4b Handle null output case for vlm chart mxmn 2022-04-30 13:28:43 -04:00
Tyler Goodlet 55606cffbb Right, handle the case where the shm prepend history isn't full XD 2022-04-30 13:28:43 -04:00
Tyler Goodlet d3251a2922 Always use coord cache, add naive view range diffing logic 2022-04-30 13:28:43 -04:00
Tyler Goodlet dc4a0a6fd2 Put mxmn profile mapping at end of method 2022-04-30 13:28:43 -04:00
Tyler Goodlet 04c12a756c If a sample stream is already ded, just warn 2022-04-30 13:28:43 -04:00
Tyler Goodlet cde23361a4 `FastAppendCurve`: Only render in-view data if possible
More or less this improves update latency like mad. Only draw data in
view and avoid full path regen as much as possible within a given
(down)sampling setting. We now support append path updates with in-view
data and the *SPECIAL CAVEAT* is that we avoid redrawing the whole curve
**only when** we calc an `append_length <= 1` **even if the view range
changed**. XXX: this should change in the future probably such that the
caller graphics update code can pass a flag which says whether or not to
do a full redraw based on it knowing where it's an interaction based
view-range change or a flow update change which doesn't require a full
path re-render.
2022-04-30 13:28:43 -04:00
Tyler Goodlet 3c58847595 Remove `._set_yrange()` handler from x-range-change signal 2022-04-30 13:28:43 -04:00
Tyler Goodlet fae249dd2f Delegate graphics cycle max/min to chart/flows 2022-04-30 13:28:43 -04:00
Tyler Goodlet 6f5bb9cbe0 Incrementally update flattend OHLC data
After much effort (and exhaustion) but failure to get a view into
our `numpy` OHLC struct-array, this instead allocates an in-thread-memory
array which is updated with flattened data every flow update cycle.

I need to report what I think is a bug to `numpy` core about the whole
view thing not working but, more or less this gets the same behaviour
and minimizes work to flatten the sampled data for line-graphics
drawing thus improving refresh latency when drawing large downsampled
curves.

TL;DR:
- add `ShmArray.ustruct()` to return a **copy of** (since a view doesn't
  work..) the (field filtered) shm array which is the same index-length
  as the source data.
- update the OHLC ds curve with view aware data sliced out from the
  pre-allocated and incrementally updated data (we had to add a last
  index var `._iflat` to track appends - this should be moved into
  a renderer eventually?).
2022-04-30 13:28:43 -04:00
Tyler Goodlet b4c7d02fcb Add `FastAppendCurve.draw_last()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet b2697bfd13 Don't require data input to constructor 2022-04-30 13:28:43 -04:00
Tyler Goodlet 14037cd1dc More WIP, implement `BarItems` rendering in `Flow.update_graphics()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet a935245360 Add `BarItems.draw_last()` and disable `.update_from_array()` 2022-04-30 13:28:43 -04:00
Tyler Goodlet 8a5538f490 WIP starting architecture doc str writeup.. 2022-04-30 13:28:43 -04:00