Commit Graph

3310 Commits (8c568d1b2ae57d77348dccd685bdd86e80643679)

Author SHA1 Message Date
Tyler Goodlet cfc4198837 Use new profiler arg name, add more marks throughout flow update 2022-06-05 22:13:08 -04:00
Tyler Goodlet c455df7fa8 Drop legacy step path gen, always slice full data
Mostly just dropping old commented code for "step mode" format
generation. Always slice the tail part of the input data and move to the
new `ms_threshold` in the `pg` profiler'
2022-06-05 22:13:08 -04:00
Tyler Goodlet 47cf4aa4f7 Error log brokerd msgs that have `.reqid == None`
Relates to the bug discovered in #310, this should avoid out-of-order
msgs which do not have a `.reqid` set to be error logged to console.
Further, add `pformat()` to kraken logging of ems msging.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 4f36743f64 Only udpate prepended graphics when actually in view 2022-06-05 22:13:08 -04:00
Tyler Goodlet 1fcb9233b4 Add back mx/mn updates for L1-in-view, lost during rebase 2022-06-05 22:13:08 -04:00
Tyler Goodlet fb38265199 Clean out legacy code from `Flow.update_graphics()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet e163a7e336 Drop `bar_wap` curve for now, seems to also be causing hangs?! 2022-06-05 22:13:08 -04:00
Tyler Goodlet 36a10155bc Add profiler passthrough type annot, comments about appends vs. uppx 2022-06-05 22:13:08 -04:00
Tyler Goodlet 7a3437348d An absolute uppx diff of >= 1 seems more then fine 2022-06-05 22:13:08 -04:00
Tyler Goodlet 0744dd0415 Up the display throttle rate to 22Hz 2022-06-05 22:13:08 -04:00
Tyler Goodlet 0770a39125 Only do curve appends on low uppx levels 2022-06-05 22:13:08 -04:00
Tyler Goodlet 2b6041465c Startup up with 3k bars 2022-06-05 22:13:08 -04:00
Tyler Goodlet 859eaffa29 Drop vwap fsp for now; causes hangs.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet b12921678b Drop step routine import 2022-06-05 22:13:08 -04:00
Tyler Goodlet 186658ab09 Drop uppx guard around downsamples on interaction
Since downsampling with the more correct version of m4 (uppx driven
windows sizing) is super fast now we don't need to avoid downsampling
on low uppx values. Further all graphics objects now support in-view
slicing so make sure to use it on interaction updates. Pass in the view
profiler to update method calls for more detailed measuring.

Even moar,
- Add a manual call to `.maybe_downsample_graphics()` inside the mouse
  wheel event handler since it seems that sometimes trailing events get
  lost from the `.sigRangeChangedManually` signal which can result in
  "non-downsampled-enough" graphics on chart given the scroll amount;
  this manual call seems to entirely fix this?
- drop "max zoom" guard since internals now support (near) infinite
  scroll out to graphics becoming a single pixel column line XD
- add back in commented xrange signal connect code for easy testing to
  verify against range updates not happening without it
2022-06-05 22:13:08 -04:00
Tyler Goodlet 12d60e6d9c WIP get incremental step curve updates working
This took longer then i care to admit XD but it definitely adds a huge
speedup and with only a few outstanding correctness bugs:

- panning from left to right causes strange trailing artifacts in the
  flows fsp (vlm) sub-plot but only when some data is off-screen on the
  left but doesn't appear to be an issue if we keep the `._set_yrange()`
  handler hooked up to the `.sigXRangeChanged` signal (but we aren't
  going to because this makes panning way slower). i've got a feeling
  this is a bug todo with the device coordinate cache stuff and we may
  need to report to Qt core?
- factoring out the step curve logic from
  `FastAppendCurve.update_from_array()` (un)fortunately required some
  logic branch uncoupling but also meant we needed special input controls
  to avoid things like redraws and curve appends for special cases,
  this will hopefully all be better rectified in code when the core of
  this method is moved into a renderer type/implementation.
- the `tina_vwap` fsp curve now somehow causes hangs when doing erratic
  scrolling on downsampled graphics data. i have no idea why or how but
  disabling it makes the issue go away (ui will literally just freeze
  and gobble CPU on a `.paint()` call until you ctrl-c the hell out of
  it). my guess is that something in the logic for standard line curves
  and appends on large data sets is the issue?

Code related changes/hacks:
- drop use of `step_path_arrays_from_1d()`, it was always a bit hacky
  (being based on `pyqtgraph` internals) and was generally hard to
  understand since it returns 1d data instead of the more expected (N,2)
  array of "step levels"; instead this is now implemented (uglily) in
  the `Flow.update_graphics()` block for step curves (which will
  obviously get cleaned up and factored elsewhere).
- add a bunch of new flags to the update method on the fast append
  curve:  `draw_last: bool`, `slice_to_head: int`, `do_append: bool`,
  `should_redraw: bool` which are all controls to aid with previously
  mentioned issues specific to getting step curve updates working
  correctly.
- add a ton of commented tinkering related code (that we may end up
  using) to both the flow and append curve methods that was written as
  part of the effort to get this all working.
- implement all step curve updating inline in `Flow.update_graphics()`
  including prepend and append logic for pre-graphics incremental step
  data maintenance and in-view slicing as well as "last step" graphics
  updating.

Obviously clean up commits coming stat B)
2022-06-05 22:13:08 -04:00
Tyler Goodlet c5beecf8a1 Drop cursor debounce delay, decrease rate limit 2022-06-05 22:13:08 -04:00
Tyler Goodlet 629ea8ba9d Downsample on every uppx inrement since it's way faster 2022-06-05 22:13:08 -04:00
Tyler Goodlet ba0ba346ec Drop log scaling support since uppx driven scaling seems way faster/better 2022-06-05 22:13:08 -04:00
Tyler Goodlet 82b2d2ee3a Hipshot, use uppx to drive theoretical px w 2022-06-05 22:13:08 -04:00
Tyler Goodlet b2b31b8f84 WIP incrementally update step array format 2022-06-05 22:13:08 -04:00
Tyler Goodlet b97ec38baf Always maybe render graphics
Since we have in-view style rendering working for all curve types
(finally) we can avoid the guard for low uppx levels and without losing
interaction speed. Further don't delay the profiler so that the nested
method calls correctly report upward - which wasn't working likely due
to some kinda GC collection related issue.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 64c6287cd1 Always set coords cache on curves 2022-06-05 22:13:08 -04:00
Tyler Goodlet 69282a9924 Handle null output case for vlm chart mxmn 2022-06-05 22:13:08 -04:00
Tyler Goodlet aee44fed46 Right, handle the case where the shm prepend history isn't full XD 2022-06-05 22:13:08 -04:00
Tyler Goodlet db727910be Always use coord cache, add naive view range diffing logic 2022-06-05 22:13:08 -04:00
Tyler Goodlet 64206543cd Put mxmn profile mapping at end of method 2022-06-05 22:13:08 -04:00
Tyler Goodlet af6aad4e9c If a sample stream is already ded, just warn 2022-06-05 22:13:08 -04:00
Tyler Goodlet c94c53286b `FastAppendCurve`: Only render in-view data if possible
More or less this improves update latency like mad. Only draw data in
view and avoid full path regen as much as possible within a given
(down)sampling setting. We now support append path updates with in-view
data and the *SPECIAL CAVEAT* is that we avoid redrawing the whole curve
**only when** we calc an `append_length <= 1` **even if the view range
changed**. XXX: this should change in the future probably such that the
caller graphics update code can pass a flag which says whether or not to
do a full redraw based on it knowing where it's an interaction based
view-range change or a flow update change which doesn't require a full
path re-render.
2022-06-05 22:13:08 -04:00
Tyler Goodlet 2af4050e5e Remove `._set_yrange()` handler from x-range-change signal 2022-06-05 22:13:08 -04:00
Tyler Goodlet df78e9ba96 Delegate graphics cycle max/min to chart/flows 2022-06-05 22:13:08 -04:00
Tyler Goodlet 7e1ec7b5a7 Incrementally update flattend OHLC data
After much effort (and exhaustion) but failure to get a view into our
`numpy` OHLC struct-array, this instead allocates an in-thread-memory
array which is updated with flattened data every flow update cycle.

I need to report what I think is a bug to `numpy` core about the whole
view thing not working but, more or less this gets the same behaviour
and minimizes work to flatten the sampled data for line-graphics drawing
thus improving refresh latency when drawing large downsampled curves.

Update the OHLC ds curve with view aware data sliced out from the
pre-allocated and incrementally updated data (we had to add a last index
var `._iflat` to track appends - this should be moved into a renderer
eventually?).
2022-06-05 22:13:08 -04:00
Tyler Goodlet 3dbce6f891 Add `FastAppendCurve.draw_last()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet 239c9d701a Don't require data input to constructor 2022-06-05 22:13:08 -04:00
Tyler Goodlet 427a33654b More WIP, implement `BarItems` rendering in `Flow.update_graphics()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet f4dc0fbab8 Add `BarItems.draw_last()` and disable `.update_from_array()` 2022-06-05 22:13:08 -04:00
Tyler Goodlet e0a72a2174 WIP starting architecture doc str writeup.. 2022-06-05 22:13:08 -04:00
Tyler Goodlet 5a9bab0b69 WIP incremental render apis 2022-06-05 22:13:08 -04:00
Tyler Goodlet d0af280a59 Port view downsampling handler to new update apis 2022-06-05 22:13:08 -04:00
Tyler Goodlet 599c77ff84 Port ui components to use flows, drop all late assignments of shm 2022-06-05 22:13:08 -04:00
Tyler Goodlet c097016fd2 Add new `ui._flows` module
This begins the removal of data processing / analysis methods from the
chart widget and instead moving them to our new `Flow` API (in the new
module introduce here) and delegating the old chart methods to the
respective internal flow. Most importantly is no longer storing the
"last read" of an array from shm in an internal chart table (was
`._arrays`) and instead the `ShmArray` instance is passed as input and
stored in the `Flow` instance. This greatly simplifies lookup logic such
that the display loop now doesn't have to worry about reading shm, it
can be done by internal graphics logic as desired. Generally speaking,
all previous `._arrays`/`._graphics` lookups are now delegated to the
entries in the chart's `._flows` table.

The new `Flow` methods are generally better factored and provide more
detailed output regarding data-stream <-> graphics inter-relations for
the future purpose of allowing much more efficient update calls in the
display loop as well as supporting low latency interaction UX.

The concept here is that we're introducing an intermediary layer that
ties together graphics and real-time data flows such that widget code is
oriented around plot layout and the flow apis are oriented around
real-time low latency updates and providing an efficient high level
metric layer for the UX.

The summary api transition is something like:
- `update_graphics_from_array()` -> `.update_graphics_from_flow()`
- `.bars_range()` -> `Flow.datums_range()`
- `.bars_range()` -> `Flow.datums_range()`
2022-06-05 22:13:08 -04:00
goodboy be7c4e70f0
Merge pull request #321 from pikers/ib_dedicated_data_client
Ib dedicated data client
2022-06-05 22:12:46 -04:00
Tyler Goodlet 051680e259 Fill data client sockaddr in feed status as `data_ep` field 2022-06-05 22:08:03 -04:00
Tyler Goodlet 55a453a710 Update `ib` section in brokers config template 2022-06-05 22:08:03 -04:00
Tyler Goodlet 88eccc1e15 Fill in label with pairs from `status` value of backend init msg 2022-06-05 22:08:00 -04:00
Tyler Goodlet 488506d8b8 Move feed status label generation into a new module 2022-06-05 22:07:13 -04:00
Tyler Goodlet 78b9333bcd Expect `list` of ports in `ib.ports` section
Given that naming the port map is mostly pointless, since accounts can
be detected once the client connects, just expect a `brokers.toml` to
define a simple sequence of port numbers. Toss in a warning for using
the old map/`dict` style.
2022-06-05 22:07:13 -04:00
Tyler Goodlet 7229a39f47 Drop data reset tries to 2 before connection reset 2022-06-04 20:44:43 -04:00
Tyler Goodlet d870a09a4b Increase timeouts, always connection reset after 3 tries 2022-06-04 20:44:03 -04:00
Tyler Goodlet 5d53ecb433 Switch vnc server to port 3003 2022-06-04 20:44:03 -04:00