Commit Graph

2231 Commits (ba93f96c710cd618b533c9c8471a8b36fd31c724)

Author SHA1 Message Date
Tyler Goodlet d770867163 Drop width arg to bar lines factory 2022-06-05 22:13:36 -04:00
Tyler Goodlet c518553aa9 Add new curve doc string 2022-06-05 22:13:36 -04:00
Tyler Goodlet 4138cef512 Drop old state from `BarsItems` 2022-06-05 22:13:36 -04:00
Tyler Goodlet 0f4bfcdf22 Drop global pg settings 2022-06-05 22:13:36 -04:00
Tyler Goodlet 80835d4e04 More detailed rt feed drop logging 2022-06-05 22:13:36 -04:00
Tyler Goodlet e6d03ba97f Add missing f-str prefix 2022-06-05 22:13:36 -04:00
Tyler Goodlet b71e8c5e6d Guard against empty source history slice output 2022-06-05 22:13:36 -04:00
Tyler Goodlet 064d185395 Drop pointless geo call from `.pain()` 2022-06-05 22:13:36 -04:00
Tyler Goodlet 363ba8f9ae Only drop throttle feeds if channel disconnects? 2022-06-05 22:13:36 -04:00
Tyler Goodlet fc24f5efd1 Iterate 1s and 1m from tsdb series 2022-06-05 22:13:36 -04:00
Tyler Goodlet a7ff47158b Pass tsdb flag when db is up XD 2022-06-05 22:13:36 -04:00
Tyler Goodlet 57acc3bd29 Factor all per graphic `.draw_last()` methods into closures 2022-06-05 22:13:36 -04:00
Tyler Goodlet 8f1faf97ee Add todo for bars range reuse in interaction handler 2022-06-05 22:13:36 -04:00
Tyler Goodlet 3ab91deaec Drop all (old) unused state instance vars 2022-06-05 22:13:36 -04:00
Tyler Goodlet 6f00617bd3 Only do new "datum append" when visible in pixels
The basic logic is now this:
- when zooming out, uppx (units per pixel in x) can be >= 1
- if the uppx is `n` then the next pixel in view becomes occupied by
  a new datum-x-coordinate-value when the diff between the last
  datum step (since the last such update) is greater then the
  current uppx -> `datums_diff >= n`
- if we're less then some constant uppx we just always update (because
  it's not costly enough and we're not downsampling.

More or less this just avoids unnecessary real-time updates to flow
graphics until they would actually be noticeable via the next pixel
column on screen.
2022-06-05 22:13:36 -04:00
Tyler Goodlet 2c2c453932 Reset line graphics on downsample step..
This was a bit of a nightmare to figure out but, it seems that the
coordinate caching system will really be a dick (like the nickname for
richard for you serious types) about leaving stale graphics if we don't
reset the cache on downsample full-redraw updates...Sooo, instead we do
this manual reset to avoid such artifacts and consequently (for now)
return a `reset: bool` flag in the return tuple from `Renderer.render()`
to indicate as such.

Some further shite:
- move the step mode `.draw_last()` equivalent graphics updates down
  with the rest..
- drop some superfluous `should_redraw` logic from
  `Renderer.render()` and compound it in the full path redraw block.
2022-06-05 22:13:36 -04:00
Tyler Goodlet 360643b32f Fix optional input `bars_range` type to match `Flow.datums_range()` 2022-06-05 22:13:36 -04:00
Tyler Goodlet ab0def22c1 Change flag name to `autoscale_overlays` 2022-06-05 22:13:36 -04:00
Tyler Goodlet a9ec1a97dd Vlm "rate" fsps, change maxmin callback name to include `multi_` 2022-06-05 22:13:36 -04:00
Tyler Goodlet d61b636487 Auto-yrange overlays in interaction (downsampler) handler 2022-06-05 22:13:36 -04:00
Tyler Goodlet 88ac2fda52 Aggretate cache resetting into a single ctx mngr method 2022-06-05 22:13:36 -04:00
Tyler Goodlet 08c83afa90 Rejig config helpers for arbitrary named files 2022-06-05 22:13:36 -04:00
Tyler Goodlet 066b8df619 Implement OHLC downsampled curve via renderer, drop old bypass code 2022-06-05 22:13:36 -04:00
Tyler Goodlet d4f31f2b3c Move update-state-vars defaults above step mode block 2022-06-05 22:13:36 -04:00
Tyler Goodlet 04897fd402 Implement pre-graphics format incremental update
Adds a new pre-graphics data-format callback incremental update api to
our `Renderer`. `Renderer` instance can now overload these custom routines:

- `.update_xy()` a routine which accepts the latest [pre/a]pended data
  sliced out from shm and returns it in a format suitable to store in
  the optional `.[x/y]_data` arrays.
- `.allocate_xy()` which initially does the work of pre-allocating the
   `.[x/y]_data` arrays based on the source shm sizing such that new
   data can be filled in (to memory).
- `._xy_[first/last]: int` attrs to track index diffs between src shm
  and the xy format data updates.

Implement the step curve data format with 3 super simple routines:
- `.allocate_xy()` -> `._pathops.to_step_format()`
- `.update_xy()` -> `._flows.update_step_xy()`
- `.format_xy()` -> `._flows.step_to_xy()`

Further, adjust `._pathops.gen_ohlc_qpath()` to adhere to the new
call signature.
2022-06-05 22:13:36 -04:00
Tyler Goodlet 42572d3808 Add back linked plots/views y-range autoscaling 2022-06-05 22:13:36 -04:00
Tyler Goodlet 8ce7e99210 Drop prints 2022-06-05 22:13:36 -04:00
Tyler Goodlet 1b38628b09 Handle teardown race, add comment about shm subdirs 2022-06-05 22:13:36 -04:00
Tyler Goodlet bbe1ff19ef Don't kill all containers on teardown XD 2022-06-05 22:13:36 -04:00
Tyler Goodlet eca2401ab5 Lul, well that heigh did not work.. 2022-06-05 22:13:36 -04:00
Tyler Goodlet 5d91516b41 Drop step mode "last datum" graphics creation from `.draw_last()`
We're doing this in `Flow.update_graphics()` atm and probably are going
to in general want custom graphics objects for all the diff curve / path
types. The new flows work seems to fix the bounding rect width calcs to
not require the ad-hoc extra `+ 1` in the step mode case; before it was
always a bit hacky anyway. This also tries to add a more correct
bounding rect adjustment for the `._last_line` segment.
2022-06-05 22:13:36 -04:00
Tyler Goodlet b985b48eb3 Add `._last_bar_lines` guard to `.paint()` 2022-06-05 22:13:36 -04:00
Tyler Goodlet c256d3bdc0 Type annot name in put to log routine 2022-06-05 22:13:36 -04:00
Tyler Goodlet f5de361f49 Import directly from `tractor.trionics` 2022-06-05 22:13:35 -04:00
Tyler Goodlet 432d4545c2 Fix last values, must be pulled from source data in step mode 2022-06-05 22:13:08 -04:00
Tyler Goodlet fa30df36ba Simplify default xy formatter 2022-06-05 22:13:08 -04:00
Tyler Goodlet 17456d96e0 Drop tons of old cruft, move around some commented ideas 2022-06-05 22:13:08 -04:00
Tyler Goodlet 167ae96566 Move graphics update logic into `Renderer.render()`
Finally this gets us much closer to a generic incremental update system
for graphics wherein the input array diffing, pre-graphical format data
processing, downsampler activation and incremental update and storage of
any of these data flow stages can be managed in one modular sub-system
:surfer_boi:.

Dirty deatz:
- reorg and move all path logic into `Renderer.render()` and have it
  take in pretty much the same flags as the old
  `FastAppendCurve.update_from_array()` and instead storing all update
  state vars (even copies of the downsampler related ones) on the
  renderer instance:
    - new state vars: `._last_uppx, ._in_ds, ._vr, ._avr`
    - `.render()` input bools: `new_sample_rate, should_redraw,
      should_ds, showing_src_data`
    - add a hack-around for passing in incremental update data (for now)
    via a `input_data: tuple` of numpy arrays
    - a default `uppx: float = 1`

- add new render interface attrs:
 - `.format_xy()` which takes in the source data array and produces out
   x, y arrays (and maybe a `connect` array) that can be passed to
   `.draw_path()` (the default for this is just to slice out the index
   and `array_key: str` columns from the input struct array),
 - `.draw_path()` which takes in the x, y, connect arrays and generates
   a `QPainterPath`
 - `.fast_path`, for "appendable" updates like there was on the fast
   append curve
 - move redraw (aka `.clear()` calls) into `.draw_path()` and trigger
   via `redraw: bool` flag.

- our graphics objects no longer set their own `.path` state, it's done
  by the `Flow.update_graphics()` method using output from
  `Renderer.render()` (and it's state if necessary)
2022-06-05 22:13:08 -04:00
Tyler Goodlet aa0efe1523 Drop `BarItems.draw_from_data()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 664a208ae5 Drop path generation from `gen_ohlc_qpath()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 876add4fc2 Drop `.update()` call from `.draw_last()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 72e849c651 Drop commented cruft from update logic 2022-06-05 22:13:08 -04:00
Tyler Goodlet b3ae562e4f Fully drop `.update_from_array()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet b5b9ecf4b1 Treat paths like input/output vars 2022-06-05 22:13:08 -04:00
Tyler Goodlet 1dab77ca0b Rect wont show on step curves unless we avoid `.draw_last()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 4c7661fc23 Factor `.update_from_array()` into `Flow.update_graphics()`
A bit hacky to get all graphics types working but this is hopefully the
first step toward moving all the generic update logic into `Renderer`
types which can be themselves managed more compactly and cached per
uppx-m4 level.
2022-06-05 22:13:08 -04:00
Tyler Goodlet e258654c86 Just drop "line dot" updates for now.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet 81be0b4bd0 Dont pass `px_width` to m4, add some commented path cap tracking 2022-06-05 22:13:08 -04:00
Tyler Goodlet df1c89e811 Drop all "pixel width" refs (`px_width`) from m4 impl 2022-06-05 22:13:08 -04:00
Tyler Goodlet f67fd11a29 Little formattito 2022-06-05 22:13:08 -04:00
Tyler Goodlet 1f95ba4fd8 Drop input xy from constructor, only keep state for cursor stuff.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet 1dca7766d2 Add notes about how to do mkts "trimming"
Which is basically just "deleting" rows from a column series.
You can only use the trim command from the `.cmd` cli and only with a so
called `LocalClient` currently; it's also sketchy af and caused
a machine to hang due to mem usage..

Ideally we can patch in this functionality for use by the rpc api
and have it not hang like this XD

Pertains to https://github.com/alpacahq/marketstore/issues/264
2022-06-05 22:13:08 -04:00
Tyler Goodlet b236dc72e4 Make vlm a float; discrete is so 80s 2022-06-05 22:13:08 -04:00
Tyler Goodlet 27ee9fdc81 Drop old non-working flatten routine 2022-06-05 22:13:08 -04:00
Tyler Goodlet 5d294031f2 Factor step format data gen into `to_step_format()`
Yet another path ops routine which converts a 1d array into a data
format suitable for rendering a "step curve" graphics path (aka a "bar
graph" but implemented as a continuous line).

Also, factor the `BarItems` rendering logic (which determines whether to
render the literal bars lines or a downsampled curve) into a routine
`render_baritems()` until we figure out the right abstraction layer for
it.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 537b725bf3 Factor ohlc to line data conversion into `._pathops.ohlc_to_line()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet ca5a25f921 Drop commented `numba` imports 2022-06-05 22:13:08 -04:00
Tyler Goodlet 037300ced0 Move ohlc lines-curve generators into pathops mod 2022-06-05 22:13:08 -04:00
Tyler Goodlet 9c5bc6deda Add `.ui._pathops` module
Starts a module for grouping together all our `QPainterpath` related
generation and data format operations for creation of fast curve
graphics. To start, drops `FastAppendCurve.downsample()` and moves
it to a new `._pathops.xy_downsample()`.
2022-06-05 22:13:08 -04:00
Tyler Goodlet bc50db5925 Rename `._ohlc.gen_qpath()` -> `.gen_ohlc_qpath()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet e8e26232ea Drop `BarItems.update_from_array()`; moved into `Flow` 2022-06-05 22:13:08 -04:00
Tyler Goodlet f6909ae395 Drop legacy step mode data formatter 2022-06-05 22:13:08 -04:00
Tyler Goodlet b609f46d26 Always delay interaction update profiling 2022-06-05 22:13:08 -04:00
Tyler Goodlet 5d26609693 Add "no-tsdb-found" history load length defaults 2022-06-05 22:13:08 -04:00
Tyler Goodlet 09e988ec3e Use `ms_threshold` throughout remaining profilers 2022-06-05 22:13:08 -04:00
Tyler Goodlet 5e602214be Use new flag, add more marks through display loop 2022-06-05 22:13:08 -04:00
Tyler Goodlet cfc4198837 Use new profiler arg name, add more marks throughout flow update 2022-06-05 22:13:08 -04:00
Tyler Goodlet c455df7fa8 Drop legacy step path gen, always slice full data
Mostly just dropping old commented code for "step mode" format
generation. Always slice the tail part of the input data and move to the
new `ms_threshold` in the `pg` profiler'
2022-06-05 22:13:08 -04:00
Tyler Goodlet 47cf4aa4f7 Error log brokerd msgs that have `.reqid == None`
Relates to the bug discovered in #310, this should avoid out-of-order
msgs which do not have a `.reqid` set to be error logged to console.
Further, add `pformat()` to kraken logging of ems msging.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 4f36743f64 Only udpate prepended graphics when actually in view 2022-06-05 22:13:08 -04:00
Tyler Goodlet 1fcb9233b4 Add back mx/mn updates for L1-in-view, lost during rebase 2022-06-05 22:13:08 -04:00
Tyler Goodlet fb38265199 Clean out legacy code from `Flow.update_graphics()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet e163a7e336 Drop `bar_wap` curve for now, seems to also be causing hangs?! 2022-06-05 22:13:08 -04:00
Tyler Goodlet 36a10155bc Add profiler passthrough type annot, comments about appends vs. uppx 2022-06-05 22:13:08 -04:00
Tyler Goodlet 7a3437348d An absolute uppx diff of >= 1 seems more then fine 2022-06-05 22:13:08 -04:00
Tyler Goodlet 0744dd0415 Up the display throttle rate to 22Hz 2022-06-05 22:13:08 -04:00
Tyler Goodlet 0770a39125 Only do curve appends on low uppx levels 2022-06-05 22:13:08 -04:00
Tyler Goodlet 2b6041465c Startup up with 3k bars 2022-06-05 22:13:08 -04:00
Tyler Goodlet 859eaffa29 Drop vwap fsp for now; causes hangs.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet b12921678b Drop step routine import 2022-06-05 22:13:08 -04:00
Tyler Goodlet 186658ab09 Drop uppx guard around downsamples on interaction
Since downsampling with the more correct version of m4 (uppx driven
windows sizing) is super fast now we don't need to avoid downsampling
on low uppx values. Further all graphics objects now support in-view
slicing so make sure to use it on interaction updates. Pass in the view
profiler to update method calls for more detailed measuring.

Even moar,
- Add a manual call to `.maybe_downsample_graphics()` inside the mouse
  wheel event handler since it seems that sometimes trailing events get
  lost from the `.sigRangeChangedManually` signal which can result in
  "non-downsampled-enough" graphics on chart given the scroll amount;
  this manual call seems to entirely fix this?
- drop "max zoom" guard since internals now support (near) infinite
  scroll out to graphics becoming a single pixel column line XD
- add back in commented xrange signal connect code for easy testing to
  verify against range updates not happening without it
2022-06-05 22:13:08 -04:00
Tyler Goodlet 12d60e6d9c WIP get incremental step curve updates working
This took longer then i care to admit XD but it definitely adds a huge
speedup and with only a few outstanding correctness bugs:

- panning from left to right causes strange trailing artifacts in the
  flows fsp (vlm) sub-plot but only when some data is off-screen on the
  left but doesn't appear to be an issue if we keep the `._set_yrange()`
  handler hooked up to the `.sigXRangeChanged` signal (but we aren't
  going to because this makes panning way slower). i've got a feeling
  this is a bug todo with the device coordinate cache stuff and we may
  need to report to Qt core?
- factoring out the step curve logic from
  `FastAppendCurve.update_from_array()` (un)fortunately required some
  logic branch uncoupling but also meant we needed special input controls
  to avoid things like redraws and curve appends for special cases,
  this will hopefully all be better rectified in code when the core of
  this method is moved into a renderer type/implementation.
- the `tina_vwap` fsp curve now somehow causes hangs when doing erratic
  scrolling on downsampled graphics data. i have no idea why or how but
  disabling it makes the issue go away (ui will literally just freeze
  and gobble CPU on a `.paint()` call until you ctrl-c the hell out of
  it). my guess is that something in the logic for standard line curves
  and appends on large data sets is the issue?

Code related changes/hacks:
- drop use of `step_path_arrays_from_1d()`, it was always a bit hacky
  (being based on `pyqtgraph` internals) and was generally hard to
  understand since it returns 1d data instead of the more expected (N,2)
  array of "step levels"; instead this is now implemented (uglily) in
  the `Flow.update_graphics()` block for step curves (which will
  obviously get cleaned up and factored elsewhere).
- add a bunch of new flags to the update method on the fast append
  curve:  `draw_last: bool`, `slice_to_head: int`, `do_append: bool`,
  `should_redraw: bool` which are all controls to aid with previously
  mentioned issues specific to getting step curve updates working
  correctly.
- add a ton of commented tinkering related code (that we may end up
  using) to both the flow and append curve methods that was written as
  part of the effort to get this all working.
- implement all step curve updating inline in `Flow.update_graphics()`
  including prepend and append logic for pre-graphics incremental step
  data maintenance and in-view slicing as well as "last step" graphics
  updating.

Obviously clean up commits coming stat B)
2022-06-05 22:13:08 -04:00
Tyler Goodlet c5beecf8a1 Drop cursor debounce delay, decrease rate limit 2022-06-05 22:13:08 -04:00
Tyler Goodlet 629ea8ba9d Downsample on every uppx inrement since it's way faster 2022-06-05 22:13:08 -04:00
Tyler Goodlet ba0ba346ec Drop log scaling support since uppx driven scaling seems way faster/better 2022-06-05 22:13:08 -04:00
Tyler Goodlet 82b2d2ee3a Hipshot, use uppx to drive theoretical px w 2022-06-05 22:13:08 -04:00
Tyler Goodlet b2b31b8f84 WIP incrementally update step array format 2022-06-05 22:13:08 -04:00
Tyler Goodlet b97ec38baf Always maybe render graphics
Since we have in-view style rendering working for all curve types
(finally) we can avoid the guard for low uppx levels and without losing
interaction speed. Further don't delay the profiler so that the nested
method calls correctly report upward - which wasn't working likely due
to some kinda GC collection related issue.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 64c6287cd1 Always set coords cache on curves 2022-06-05 22:13:08 -04:00
Tyler Goodlet 69282a9924 Handle null output case for vlm chart mxmn 2022-06-05 22:13:08 -04:00
Tyler Goodlet aee44fed46 Right, handle the case where the shm prepend history isn't full XD 2022-06-05 22:13:08 -04:00
Tyler Goodlet db727910be Always use coord cache, add naive view range diffing logic 2022-06-05 22:13:08 -04:00
Tyler Goodlet 64206543cd Put mxmn profile mapping at end of method 2022-06-05 22:13:08 -04:00
Tyler Goodlet af6aad4e9c If a sample stream is already ded, just warn 2022-06-05 22:13:08 -04:00
Tyler Goodlet c94c53286b `FastAppendCurve`: Only render in-view data if possible
More or less this improves update latency like mad. Only draw data in
view and avoid full path regen as much as possible within a given
(down)sampling setting. We now support append path updates with in-view
data and the *SPECIAL CAVEAT* is that we avoid redrawing the whole curve
**only when** we calc an `append_length <= 1` **even if the view range
changed**. XXX: this should change in the future probably such that the
caller graphics update code can pass a flag which says whether or not to
do a full redraw based on it knowing where it's an interaction based
view-range change or a flow update change which doesn't require a full
path re-render.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 2af4050e5e Remove `._set_yrange()` handler from x-range-change signal 2022-06-05 22:13:08 -04:00
Tyler Goodlet df78e9ba96 Delegate graphics cycle max/min to chart/flows 2022-06-05 22:13:08 -04:00
Tyler Goodlet 7e1ec7b5a7 Incrementally update flattend OHLC data
After much effort (and exhaustion) but failure to get a view into our
`numpy` OHLC struct-array, this instead allocates an in-thread-memory
array which is updated with flattened data every flow update cycle.

I need to report what I think is a bug to `numpy` core about the whole
view thing not working but, more or less this gets the same behaviour
and minimizes work to flatten the sampled data for line-graphics drawing
thus improving refresh latency when drawing large downsampled curves.

Update the OHLC ds curve with view aware data sliced out from the
pre-allocated and incrementally updated data (we had to add a last index
var `._iflat` to track appends - this should be moved into a renderer
eventually?).
2022-06-05 22:13:08 -04:00
Tyler Goodlet 3dbce6f891 Add `FastAppendCurve.draw_last()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 239c9d701a Don't require data input to constructor 2022-06-05 22:13:08 -04:00
Tyler Goodlet 427a33654b More WIP, implement `BarItems` rendering in `Flow.update_graphics()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet f4dc0fbab8 Add `BarItems.draw_last()` and disable `.update_from_array()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet e0a72a2174 WIP starting architecture doc str writeup.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet 5a9bab0b69 WIP incremental render apis 2022-06-05 22:13:08 -04:00
Tyler Goodlet d0af280a59 Port view downsampling handler to new update apis 2022-06-05 22:13:08 -04:00
Tyler Goodlet 599c77ff84 Port ui components to use flows, drop all late assignments of shm 2022-06-05 22:13:08 -04:00
Tyler Goodlet c097016fd2 Add new `ui._flows` module
This begins the removal of data processing / analysis methods from the
chart widget and instead moving them to our new `Flow` API (in the new
module introduce here) and delegating the old chart methods to the
respective internal flow. Most importantly is no longer storing the
"last read" of an array from shm in an internal chart table (was
`._arrays`) and instead the `ShmArray` instance is passed as input and
stored in the `Flow` instance. This greatly simplifies lookup logic such
that the display loop now doesn't have to worry about reading shm, it
can be done by internal graphics logic as desired. Generally speaking,
all previous `._arrays`/`._graphics` lookups are now delegated to the
entries in the chart's `._flows` table.

The new `Flow` methods are generally better factored and provide more
detailed output regarding data-stream <-> graphics inter-relations for
the future purpose of allowing much more efficient update calls in the
display loop as well as supporting low latency interaction UX.

The concept here is that we're introducing an intermediary layer that
ties together graphics and real-time data flows such that widget code is
oriented around plot layout and the flow apis are oriented around
real-time low latency updates and providing an efficient high level
metric layer for the UX.

The summary api transition is something like:
- `update_graphics_from_array()` -> `.update_graphics_from_flow()`
- `.bars_range()` -> `Flow.datums_range()`
- `.bars_range()` -> `Flow.datums_range()`
2022-06-05 22:13:08 -04:00
Tyler Goodlet 051680e259 Fill data client sockaddr in feed status as `data_ep` field 2022-06-05 22:08:03 -04:00
Tyler Goodlet 88eccc1e15 Fill in label with pairs from `status` value of backend init msg 2022-06-05 22:08:00 -04:00
Tyler Goodlet 488506d8b8 Move feed status label generation into a new module 2022-06-05 22:07:13 -04:00
Tyler Goodlet 78b9333bcd Expect `list` of ports in `ib.ports` section
Given that naming the port map is mostly pointless, since accounts can
be detected once the client connects, just expect a `brokers.toml` to
define a simple sequence of port numbers. Toss in a warning for using
the old map/`dict` style.
2022-06-05 22:07:13 -04:00
Tyler Goodlet 7229a39f47 Drop data reset tries to 2 before connection reset 2022-06-04 20:44:43 -04:00
Tyler Goodlet d870a09a4b Increase timeouts, always connection reset after 3 tries 2022-06-04 20:44:03 -04:00
Tyler Goodlet 5d53ecb433 Switch vnc server to port 3003 2022-06-04 20:44:03 -04:00
Tyler Goodlet 06832b94d4 Add vnc password auth, connection reset logic
Now that we have working client auth thanks to:
https://github.com/barneygale/asyncvnc/pull/4 and related issue,
we can use a pw for the vnc server, though we should eventually
auto-generate a random one from a docker super obviously.

Add logic to the data reset hack loop to do a connection reset after
2 failed/timeout attempts at the regular data reset. We need to also add
this logic around reconnectionn events that are due to the host
network connection: aka roaming that's faster then timing logic
builtin to the gateway.
2022-06-04 20:44:03 -04:00
Tyler Goodlet 8d6c5b214e Add 6, 6s retries on feed resets 2022-06-04 20:44:03 -04:00
Tyler Goodlet a5389beccd Rejig scan loop for flaky TCP connects, better caching
`ib-gw` seems particularly fragile to connections from clients with the
same id (can result in weird connect hangs and even crashes) and
`ib_insync` doesn't handle intermittent tcp disconnects that
well..(especially on dockerized IBC setups). This adds a bunch of
changes to our client caching and scan loop as well a proper
task-locking-to-cache-proxies so that,

- `asyncio`-side clients aren't double-loaded/connected even when
  explicitly trying to reconnect repeatedly with a given client to work
  around the unreliability of the `asyncio.Transport` design in
  `ib_insync`.
- we can use `tractor.trionics.maybe_open_context()` to lock the `trio`
  side from loading more then one `Client` on the `asyncio` side and
  instead on cache hits only making a new `MethodProxy` around the
  reused `asyncio`-side client (since each `trio` task needs its own
  inter-task msg channel).
- a `finally:` block teardown on all clients loaded in the scan loop
  avoids stale connections.
- the connect params are now exposed as named args to
  `load_aio_clients()` can be easily controlled from caller code.

Oh, and we properly hooked up the internal `ib_insync` logging to our
own internal schema - makes it a lot easier to debug wtf is going on XD
2022-06-04 20:44:03 -04:00
Tyler Goodlet 26f47227d2 Fix `.ib` pattern match 2022-06-04 20:44:03 -04:00
Tyler Goodlet b357a120b9 Fix output unpack 2022-06-04 20:44:03 -04:00
Tyler Goodlet aba8b05a33 Fix null match 2022-06-04 20:44:03 -04:00
Tyler Goodlet c3142aec81 Drop `i3ipc + `xdotool` approach for feed hacks 2022-06-04 20:44:03 -04:00
Tyler Goodlet bff625725e Implement reset hacks via our patched `asyncvnc` client 2022-06-04 20:44:03 -04:00
Tyler Goodlet 6f172479eb Drop task-per-method `trio`-`asyncio` proxying
Use method proxies through the remaining endpoints and drop the old
spawn-a-task-per-method-call style helpers from module.
2022-06-04 20:44:03 -04:00
Tyler Goodlet a96f1dec3a Proxy heaven, choose one "preferred data client"
In order to expose more `asyncio` powered `Client` methods to endpoint
task-code this adds a more extensive and layered set of `MethodProxy`
loading routines, in dependency order these are:
- `load_clients_for_trio()` a `tractor.to_asyncio.open_channel_from()`
  entry-point factory for loading all scanned clients on the `asyncio` side
  and delivering them over the inter-task channel to a `trio`-side task.
- `get_preferred_data_client()` a simple client instance loading routine
  which reads from the users `brokers.toml -> `prefer_data_account:
  list[str]` which must list account names, in priority order, that are
  acceptable to be used as the main "data connection client" such that
  only one of the detected clients is used for data (whereas the rest
  are used only for order entry).
- `open_client_proxies()` which delivers the detected `Client` set
  wrapped each in a `MethodProxy`.
- `open_data_client()` which directly delivers the preferred data client
  as a proxy for `trio` tasks.
- update `open_client_method_proxy()` and `open_client_proxy` to require
  an input `Client` instance.

Further impl details:
- add `MethodProxy._aio_ns` to ref the original `asyncio` side proxied instance
- add `Client.trades()` to pull executions from the last day/session
- load proxies inside `trades_dialogue` and use the new `.trades()`
  method to try and pull a fill ledger for eventual correct pp price
  calcs (pertains to #307)..
2022-06-04 20:44:03 -04:00
dinkus 339fcda727 fix windows snap problem by removing maximum window size 2022-06-04 17:53:27 -04:00
Tyler Goodlet 09f2f32d5b Drop `pandas` timestamp for qt 2022-05-15 13:49:54 -04:00
Tyler Goodlet fb5df5ab5e Drop `pandas` usage throughout brokers cli 2022-05-15 13:49:50 -04:00
Tyler Goodlet 6e2e2fc03f Use `pendulum` for timestamp parsing 2022-05-15 13:45:44 -04:00
Tyler Goodlet a3b2ba9ae9 Use `numpy.datetime64` for x-axis tick strings 2022-05-15 13:45:37 -04:00
Tyler Goodlet de55565f60 We already collect account values/names in `load_io_clients()` 2022-05-12 14:21:31 -04:00
Tyler Goodlet d0530c4e26 Deliver accounts from query instead of just pps with `ib` 2022-05-12 13:22:51 -04:00
Tyler Goodlet fb91e27651 Well that was easy, convert tick/lot step sizes to `float` 2022-05-11 10:41:03 -04:00
Tyler Goodlet b3f9c4f93d Only assert if input array actually has a size 2022-05-10 17:59:24 -04:00
Tyler Goodlet 09431aad85 Add support for no `._first.value` update shm prepends 2022-05-10 17:59:16 -04:00
Tyler Goodlet 8219307bf5 Double up shm buffer size 2022-05-10 17:59:08 -04:00
Tyler Goodlet b910eceb3b Add `ShmArray.ustruct()`: return an unstructured array copy
We return a copy (since since a view doesn't seem to work..) of the
(field filtered) shm array contents which is the same index-length as
the source data.

Further, fence off the resource tracker disable-hack into a helper
routine.
2022-05-10 17:58:57 -04:00
Tyler Goodlet 1657f51edc Manually fetch missing out-of-order history frames
It seems once in a while a frame can get missed or dropped (at least
with binance?) so in those cases, when the request erlangs is already at
max, we just manually request the missing frame and presume things will
work out XD

Further, discard out of order frames that are "from the future" that
somehow end up in the async queue once in a while? Not sure why this
happens but it seems thus far just discarding them is nbd.
2022-05-10 17:25:20 -04:00
Tyler Goodlet b1246446c2 Raise error on 'fatal' and 'error' log levels 2022-05-10 17:25:20 -04:00
Tyler Goodlet 083a3296e7 Better formatted startup logging output 2022-05-10 14:55:52 -04:00
Tyler Goodlet 769e803695 Write `mkts.yml` from template if one dne 2022-05-10 14:55:52 -04:00
Tyler Goodlet e196e9d1a0 Factor `marketstore` container specifics into `piker.data.marketstore` 2022-05-10 14:55:52 -04:00
Tyler Goodlet 9ddfae44d2 Parametrize and deliver (relevant) mkts config in `start_ahab()` 2022-05-10 14:55:52 -04:00
Tyler Goodlet 277ca29018 Always write missing history frames to tsdb (again) 2022-05-10 14:55:52 -04:00
Tyler Goodlet 26fddae3c0 Fix earliest frame-end not-yet-pushed check
Bleh/🤦, the ``end_dt`` in scope is not the "earliest" frame's
`end_dt` in the async response queue.. Parse the queue's latest epoch
and use **that** to compare to the last last pushed datetime index..

Add more detailed logging to help debug any (un)expected datetime index
gaps.
2022-05-10 14:55:52 -04:00
Tyler Goodlet 4b6ecbfc79 Bring binance requests down to 3/sec; seems faster? 2022-05-10 14:55:52 -04:00
Tyler Goodlet 30ddf63ec0 Handle gaps greater then a frame within a frame 2022-05-09 11:15:14 -04:00
Tyler Goodlet 8e08fb7b23 Add comment about un-reffed vars meant for use in shell 2022-05-09 11:15:14 -04:00
Tyler Goodlet fb9b6990ae Drop unneeded/commented cancel-by-msg code; roots perms wasn't the problem 2022-05-09 11:15:14 -04:00
Tyler Goodlet 1676bceee1 Don't offset the start index by a step 2022-05-09 11:15:14 -04:00
Tyler Goodlet c9a621fc2a Fix less-then-frame off by one slice, add db write toggle and disable 2022-05-09 11:15:14 -04:00
Tyler Goodlet 0324404b03 Include epoch timestamp in quote label for now 2022-05-09 11:15:14 -04:00
Tyler Goodlet 61e9db3229 Handle ``iter_dts()`` already exhausted edge case 2022-05-09 11:15:14 -04:00
Tyler Goodlet 4a6f01747c Label "humanized" sample period in window title-bar" 2022-05-09 11:15:14 -04:00
Tyler Goodlet e4a900168d Add timeframe key to seconds map 2022-05-09 11:15:14 -04:00
Tyler Goodlet 40753ae93c Always write newly pulled frames to tsdb 2022-05-09 11:15:14 -04:00
Tyler Goodlet 969530ba19 Fix slice logic for less-then-frame tsdb overlap
When the tsdb has a last datum that is in the past less then a "frame's
worth" of sample steps we need to slice out only the data from the
latest frame that doesn't overlap; this fixes that slice logic..
Previously i dunno wth it was doing..
2022-05-09 11:15:14 -04:00
Tyler Goodlet 9b5f052597 Handle no sampler subs case on history broadcasts
When the market isn't open the feed layer won't create a subscriber
entry in the sampler broadcast loop and so if a manual call to
``broadcast()`` is made (like when trying to update a chart from
a history prepend) we need to handle that case and just broadcast
a random `-1` for now..BD
2022-05-09 11:15:14 -04:00
Tyler Goodlet b44786e5b7 Support async-batched ohlc queries in all backends
Expect each backend to deliver a `config: dict[str, Any]` which provides
concurrency controls to `trimeter`'s batch task scheduler such that
backends can define their own concurrency limits.

The dirty deats in this patch include handling history "gaps" where
a query returns a history-frame-result which spans more then the typical
frame size (in seconds). In such cases we reset the target frame index
(datetime index sequence implemented with a `pendulum.Period`) using
a generator protocol `.send()` such that the sequence can be dynamically
re-indexed starting at the new (possibly) pre-gap datetime. The new gap
logic also allows us to detect out of order frames easier and thus wait
for the next-in-order to arrive before making more requests.
2022-05-09 11:15:14 -04:00
Tyler Goodlet 7e951f17ca Support large ohlcv writes via slicing, add struct array keymap 2022-05-09 11:15:14 -04:00
Tyler Goodlet fcb85873de Terminate early on data unavailable errors 2022-05-09 11:15:14 -04:00
Tyler Goodlet 7b1c0939bd Add first-draft `trimeter` based concurrent ohlc history fetching 2022-05-09 11:15:14 -04:00
Tyler Goodlet d77cfa3587 Add back fqsn passthrough and feed opening 2022-05-09 11:15:14 -04:00
Tyler Goodlet 49509d55d2 Implement `open_history_client()` correctly for `kraken` 2022-05-09 11:15:14 -04:00
Tyler Goodlet 6ba3c15c4e Add to signal broker won't deliver more data 2022-05-09 11:15:14 -04:00
Tyler Goodlet a3db5d1bdc Relay frame size in `NoData` due to null-result history 2022-05-09 11:15:14 -04:00
Tyler Goodlet c672493998 Add , indicates hist size to decrement to storage logic 2022-05-09 11:15:14 -04:00
Tyler Goodlet 423af37389 Truncate trade rate wma window sizes 2022-05-09 11:15:14 -04:00
Tyler Goodlet 0061fabb56 More tolerance for "stream-ended-early" conditions in quote throttler 2022-05-09 11:15:14 -04:00
Tyler Goodlet 2f04a8c939 Drop legacy back-filling logic
Use the new `open_history_client()` endpoint/API and expect backends to
provide a history "getter" routine that can be called to load historical
data into shm even when **not** using a tsdb. Add logic for filling in
data from the tsdb once the backend has provided data up to the last
recorded in the db. Add logic for avoiding overruns of the shm buffer
with more-then-necessary queries of tsdb data.
2022-05-09 11:15:14 -04:00
Tyler Goodlet 8bf40ae299 Drop legacy backfilling, load a day's worth of data by default 2022-05-09 11:15:14 -04:00
Tyler Goodlet 0f683205f4 Add 16 fetch limit if no tsdb data found 2022-05-09 11:15:14 -04:00
Tyler Goodlet d244af69c9 Don't require a symbol to subcmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet b8b95f1081 Don't open a feed, write or read ohlc in for now 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3056bc3143 Don't run legacy backfill when isn't up 2022-05-09 11:15:13 -04:00
Tyler Goodlet d3824c8c0b Start legacy backfill with partial too 2022-05-09 11:15:13 -04:00
Tyler Goodlet 727d3cc027 Unify backfilling logic into common task-routine 2022-05-09 11:15:13 -04:00
Tyler Goodlet 46c23e90db Add `Storage.load()` and `.write_ohlcv()` 2022-05-09 11:15:13 -04:00
Tyler Goodlet bcf3be1fe4 A bit hacky but, broadcast index streams on each history prepend 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7d8cf3eaf8 Factor subscription broadcasting into a func 2022-05-09 11:15:13 -04:00
Tyler Goodlet d4e0d4463f Always update ohlc (main source chart) on `trigger_all=True` 2022-05-09 11:15:13 -04:00
Tyler Goodlet ab8629aa11 Make ib history client expect datetimes for input 2022-05-09 11:15:13 -04:00
Tyler Goodlet 2a07005c97 Add binance history client support with datetime use throughout 2022-05-09 11:15:13 -04:00
Tyler Goodlet 79160619bc Drop old type annot 2022-05-09 11:15:13 -04:00
Tyler Goodlet e1a88cb93c Only update y mxmn from L1 when last index in view 2022-05-09 11:15:13 -04:00
Tyler Goodlet a6c5902437 More reliable `marketstored` + container supervision
It turns out (i guess not so shockingly?) that `marketstore` doesn't
always teardown "gracefully" under SIGINT (seems to hang if there are
open client connections which are also in the midst of teardown?) so
this instead first tries the SIGINT and then fails over to a SIGKILL
(destroy loop) which seems to be much more reliable to ensure shutdown
without any downside - in terms of a "hard kill".

Originally i was thinking the issue was root perms related (which get
relegated solely to the `marketstored` daemon actor after spawn) but
actually it was indeed the signalling / application layer causing the
hold-up/latency on teardown. There's a bunch of lingering (now
commented) code which tried to solve this non-problem as well as a bunch
logging/prints to help decipher the root of the issue - this will all
get cleaned out shortly.
2022-05-09 11:15:13 -04:00
Tyler Goodlet 9fe5cd647a Handle non-fqsn for derivs and don't put brokername in 2022-05-09 11:15:13 -04:00
Tyler Goodlet 15630f465d Limit ohlc queries to 800k datums to avoid `purepc` size error 2022-05-09 11:15:13 -04:00
Tyler Goodlet ce3229df7d Get sync-to-marketstore-tsdb history retrieval workinnn 2022-05-09 11:15:13 -04:00
Tyler Goodlet 53ad5e6f65 Handle "fatal" level log msgs in docker super 2022-05-09 11:15:13 -04:00
Tyler Goodlet 41325ad418 Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-05-09 11:15:13 -04:00
Tyler Goodlet a971de2b67 Drop `ms-shell`, add `piker storesh` cmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet ca48577c60 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-05-09 11:15:13 -04:00
Tyler Goodlet 950cb03e07 Drop `pandas` to `numpy` converter 2022-05-09 11:15:13 -04:00
Tyler Goodlet 6cdd017cd6 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-05-09 11:15:13 -04:00
Tyler Goodlet 6dc6d00a9b Try downsampling mkts data 2022-05-09 11:15:13 -04:00
Tyler Goodlet 565573b609 Load any symbol-matching shm array if no `marketstored` found 2022-05-09 11:15:13 -04:00
Tyler Goodlet 6c6f2abd06 Use new `tractor.query_actor()` for service checking 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9138f376f7 Return all timeframe arrays if `timeframe` not passed as input 2022-05-09 11:15:13 -04:00
Tyler Goodlet f582af4c9f Make `pikerd` work again without `--tsdb` flag 2022-05-09 11:15:13 -04:00
Tyler Goodlet dd2edaeb3c Add a service checker predicate 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3d6d77364b Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8003878248 Proxy `marketstore` container log level to our own 2022-05-09 11:15:13 -04:00
Tyler Goodlet 706c8085f2 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-05-09 11:15:13 -04:00
Tyler Goodlet cbe74d126e Doc str formatting 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3dba456cf8 Add latency measures around diffs/writes to mkts 2022-05-09 11:15:13 -04:00
Tyler Goodlet 4555a1f279 Prototype out writing `1Sec` OHLCV data 2022-05-09 11:15:13 -04:00
Tyler Goodlet a2fe814857 Better doc string 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8c558d05d6 Persist backing `/data/` filesystem across container runs 2022-05-09 11:15:13 -04:00
Tyler Goodlet e1bbcff8e0 Get basic OHLCV writes working with `anyio` client 2022-05-09 11:15:13 -04:00
Tyler Goodlet ba82a18890 Pass in daemon name to `start_ahab()` 2022-05-09 11:15:13 -04:00
Tyler Goodlet d9773217e9 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-05-09 11:15:13 -04:00
Tyler Goodlet 2c51ad2a0d Revive `ms-shell` sub-cmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet 56fa759452 Add WIP backfiller from data feed helper 2022-05-09 11:15:13 -04:00
Tyler Goodlet 4bcc301c01 Better handle nested erros from docker client 2022-05-09 11:15:13 -04:00
Tyler Goodlet 445b82283d Add back in legacy write loop for reference 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8047714101 Add back in OHLCV dtype template and client side ws streamer 2022-05-09 11:15:13 -04:00
Tyler Goodlet 970393bb85 Drop ununsed `Services` ref 2022-05-09 11:15:13 -04:00
Tyler Goodlet ed5bae0e11 Py3.9+ type updates 2022-05-09 11:15:13 -04:00
Tyler Goodlet facc86f76e Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7395b56321 De-escalate sudo perms in `pikerd` once docker spawns 2022-05-09 11:15:13 -04:00
Tyler Goodlet aecc5973fa Handle the non-root perms case specifically too 2022-05-09 11:15:13 -04:00
Tyler Goodlet faa5a785cb Add explicit no-docker error and supervisor start task-func 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7d2e9bff46 Type annot updates 2022-05-09 11:15:13 -04:00
Tyler Goodlet ec413541d3 Drop old client instantiate line 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9203ebe044 Drop import, it's got madness with and SIGINT? 2022-05-09 11:15:13 -04:00
Tyler Goodlet fbd3d1e308 Add a super simple `marketstore` container supervisor 2022-05-09 11:15:13 -04:00
Tyler Goodlet 1cdb94374c Extract non-sudo user for config dir path 2022-05-09 11:15:13 -04:00
Tyler Goodlet aca3ca8aa6 Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 943b02573d Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 897a5cf2f6 Simplify and optimize tick format, similar to techtonicdb's 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 3c09bfba57 Add multi ingestor support and update to new feed API 2022-05-09 11:15:13 -04:00
Tyler Goodlet 34e6db6d98 Allow `None` for `BrokerdError.reqid`
Found this caused breakage on `kraken` orders which triggered the
"insufficient funds" error response. Makes sense since they won't
generate an order id if the order can't ever be submitted.
2022-05-09 10:56:47 -04:00
Tyler Goodlet 5921d18d66 Only update y-range from L1 mxmn when last index in view
We still have to always keep track of the last max and min
though.
2022-04-30 11:36:23 -04:00
Tyler Goodlet cdc882657a Drop old `pyqtgraph` downsample code 2022-04-30 11:36:23 -04:00
Tyler Goodlet 62d08eaf85 Tweak log-scaler for more detail 2022-04-30 11:36:23 -04:00
Tyler Goodlet f2f00dcc52 Drop `._ic` debugging prints 2022-04-30 11:36:23 -04:00
Tyler Goodlet ee831baeb3 Display loop mega-cleanup
The most important changes include:
- iterating the new `Flow` type and updating graphics
- adding detailed profiling
- increasing the min uppx before graphics updates are throttled
- including the L1 spread in y-range calcs so that you never have the
  bid/ask go "out of view"..
- pass around `Flow`s instead of shms
- drop all the old prototyped downsampling code
2022-04-30 11:36:23 -04:00
Tyler Goodlet 7c615a403b Allow passing a `plotItem` to `.draw_curve()`
If manually managing an overlay you'll likely call `.overlay_plotitem()`
and then a plotting method so we need to accept a plot item input so
that the chart's pi doesn't get assigned incorrectly in the `Flow` entry
(though it is by default if no input is provided).

More,
- add a `Flow.graphics` field and set it to the `pg.GraphicsObject`.
- make `Flow.maxmin()` return `None` in the "can't calculate" cases.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b8374dbe9a Fsp UI initialization updates
- set shm refs on `Flow` entries.
- don't run a graphics cycle on 'update' msgs from the engine
  if the containing chart is hidden.
- drop `volume` from flows map and disable auto-yranging
  once $vlm comes up.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 454cd7920d Disconnect signals in `ChartView.disable_auto_yrange()`
Allows for removing resize callbacks for a flow/overlay that you wish to
remove from view (eg. unit volume after dollar volume is up) and thus
less general interaction callback overhead for any plot you don't wish
to show or resize.

Further,
- drop the `autoscale_linked_plots` block for now since with
  multi-view-box overlays each register their own vb resize slots
- pull the graphics object from the chart's `Flow` map inside
  `.maybe_downsample_graphics()`
2022-04-30 11:36:23 -04:00
Tyler Goodlet ca283660de Fix bug where if `yrange` was passed the mxmin callback was still used.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet d4eddbdb25 Guard against zero px width 2022-04-30 11:36:23 -04:00
Tyler Goodlet eec329a221 Add `Flow` type with a real chitty mxmn cacheing method
This new type wraps a shm data flow and will eventually include things
like incremental path-graphics updates and serialization + bg downsampling
techniques. The main immediate motivation was to get a cached y-range max/min
calc going since profiling revealed the `numpy` equivalents were
actually quite slow as the data set grows large. Likely we can use all
this to drive a streaming mx/mn routine that's always launched as part
of each on-host flow.

This is our official foray into use of `msgspec.Struct` B) and I have to
say, pretty impressed; we'll likely completely ditch `pydantic` from
here on out.
2022-04-30 11:36:23 -04:00
Tyler Goodlet a1de097a43 Loop for first graphic with xvec 2022-04-30 11:36:23 -04:00
Tyler Goodlet b5f2558cec Only `.maybe_downsample_graphics()` on manual changes
We don't need update graphics on every x-range change since that's what
the display loop does. Instead, only on manual changes do we make manual
calls into `.update_graphics_from_array()` and be sure to iterate all
linked subplots and all their embedded graphics.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 1a95712680 Don't return early on ds line render to avoid breaking profiling
The pg profiler seems to have trouble with early `return`s in function
calls (likely muckery with the GC/`.__delete__()`) so let's just try
to avoid it for now until we either fix it (probably by implementing as
a ctx mngr) or use diff one.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b20e9e58ee Use HL tracer by default, seems to be faster? 2022-04-30 11:36:23 -04:00
Tyler Goodlet 4bc2bbda69 Allow passing "ms slower then" value on cli to `--profile` 2022-04-30 11:36:23 -04:00
Tyler Goodlet b524929cb6 Only bail up pan updates if uppx > 16 2022-04-30 11:36:23 -04:00
Tyler Goodlet f95d22bfd3 Delegate `BarItems.x_uppx()` to internal ds curve 2022-04-30 11:36:23 -04:00