Since `ChartPlotWidget.update_graphics_from_flow()` is more or less just
a call to `Viz.update_graphics()` try to call that directly where
possible.
Changes include:
- calling the viz in the display state specific `maxmin()`.
- passing a viz instance to each `ChartView._set_yrange()` call (in prep
of explicit group auto-ranging); not that this input is unused in the
method for now.
- drop `bars_range` var passing since we don't use it.
Goes back to always adjusting the y-axis range to include the L1 spread
and clearing label in view whenever the last datum is also in view,
previously this was broken after reworking the display loop for
multi-feeds.
Drops a bunch of old commented tick looping cruft from before we started
using tick-type framing. Also adds more stringent guards for ignoring
but error logging quote values that are more then 25% out of range; it
seems particularly our `ib` feed has some issues with strange `price`
values that are way off here and there?
Turns out we were updating the wrong ``Viz``/``DisplayState`` inside the
closure style `increment_history_view()`` (probably due to looping
through the flumes and dynamically closing in that task-func).. Instead
define the history incrementer at module level and pass in the
`DisplayState` explicitly. Further rework the `DisplayState` attrs to be
more focused around the `Viz` associated with the fast and slow chart
and be sure to adjust output from each `Viz.incr_info()` call to latest
update. Oh, and just tweaked the line palette for the moment.
FYI "treading" here is referring to the x-shifting of the curve when
the last datum is in view such that on new sampled appends the "last"
datum is kept in the same x-location in UI terms.
Might as well since it makes the chart look less gappy and we can easily
flip the index switch now B)
Also adds a new `'i_slow_last'` key to `DisplayState` for a singleton
across all slow charts and thus no more need for special case logic in
`viz.incr_info()`.
Turned out to be super simple to get the first draft to work since the
fast and slow chart now use the same domain, however, it seems like
maybe there's an offset issue still where the fast may be a couple
minutes ahead of the slow?
Need to dig in a bit..
Using a global "last index step" (via module var) obviously has problems
when working with multiple feed sets in a single global app instance:
any separate feed-set will be incremented according to an app-global
index-step and thus won't correctly calc per-feed-set-step update info.
Impl deatz:
- drop `DisplayState.incr_info()` (since previously moved to `Viz`) and
call that method on each appropriate `Viz` instance where necessary;
further ensure the appropriate `DisplayState` instance is passed in to
each call and make sure to pass a `state: DisplayState`.
- add `DisplayState.hist_vars: dict` for history chart (sets) to
determine the per-feed (not set) current slow chart (time) step.
- add `DisplayState.globalz: dict` to house a common per-feed-set state
and use it inside the new `Viz.incr_info()` such that
a `should_increment: bool` can be returned and used by the display
loop to determine whether to x-shift the current chart.
Ensures that a "last datum" graphics object exists so that zooming can
read it using `.x_last()`. Also, disable the linked region stuff for now
since it's totally borked after flipping to the time indexing.
Again, to make epoch indexing a flip-of-switch for testing look up the
`Viz.index_field: str` value when updating labels.
Also, drops the legacy tick-type set tracking which we no longer use
thanks to the new throttler subsys and it's framing msgs.
Since higher level charting and fsp management need access to the
new `Flume` indexing apis this adjusts some func sigs to pass through
(and/or create) flume instances:
- `LinkedSplits.add_plot()` and dependents.
- `ChartPlotWidget.draw_curve()` and deps, and it now returns a `Flow`.
- `.ui._fsp.open_fsp_admin()` and `FspAdmin.open_fsp_ui()` related
methods => now we wrap the destination fsp shm in a flume on the admin
side and is returned from `.start_engine_method()`.
Drop a bunch of (unused) chart widget methods including some already
moved to flume methods: `.get_index()`, `.in_view()`,
`.last_bar_in_view()`, `.is_valid_index()`.
Move to expect and process new by-tick-event frames where the display
loop can now just iterate the most recent tick events by type instead of
the entire tick history sequence - thus we reduce iterations inside the
update loop.
Also, go back to use using the detected display's refresh rate (minus 6)
as the default feed requested throttle rate since we can now handle
much more bursty-ness in display updates thanks to the new framing
format B)
Factor out the chart widget creation since it's only executed once
during rendering of the first feed/flow whilst keeping plotitem overlay
creation inside the (flume oriented) init loop. Only create one vlm and
FSP chart/chain for now until we figure out if we want FSPs overlayed by
default or selected based on the "front" symbol in use. Add a default
color-palette set using shades of gray when plotting overlays. Presume
that the display loop's quote throttle rate should be uniformly
distributed over all input symbol-feeds for now. Restore feed pausing on
mouse interaction.
Initial support for real-time multi-symbol overlay charts using an
aggregate feed delivered by `Feed.open_multi_stream()`.
The setup steps for constructing the overlayed plot items is still very
very rough and will likely provide incentive for better refactoring high
level "charting APIs". For each fqsn passed into `display_symbol_data()`
we now synchronously,
- create a single call to `LinkedSplits.plot_ohlc_main() -> `ChartPlotWidget`
where we cache the chart in scope and for all other "sibling" fqsns
we,
- make a call to `ChartPlotWidget.overlay_plotitem()` -> `PlotItem`, hide its axes,
make another call with this plotitem input to
`ChartPlotWidget.draw_curve()`, set a sym-specific view box auto-yrange maxmin callback,
register the plotitem in a global `pis: dict[str, list[pgo.PlotItem, pgo.PlotItem]] = {}`
Once all plots have been created we then asynchronously for each symbol,
- maybe create a volume chart and register it in a similar task-global
table: `vlms: dict[str, ChartPlotWidget] = {}`
- start fsp displays for each symbol
Then common entrypoints are entered once for all symbols:
- a single `graphics_update_loop()` loop-task is started wherein
real-time graphics update components for each symbol are created,
* `L1Labels`
* y-axis last clearing price stickies
* `maxmin()` auto-ranger
* `DisplayState` (stored in a table `dss: dict[str, DisplayState] = {}`)
* an `increment_history_view()` task
and a single call to `Feed.open_multi_stream()` is used to create
a symbol-multiplexed quote stream which drives a single loop over all
symbols wherein for each quote the appropriate components are looked
up and passed to `graphics_update_cycle()`.
- a single call to `open_order_mode()` is made with the first symbol
provided as input, though eventually we want to support passing in the
entire list.
Further internal implementation details:
- special tweaks to the `pg.LinearRegionItem` setup wherein the region
is added with a zero opacity and *after* all plotitem overlays to
avoid and issue where overlays weren't being shown within the region
area in the history chart.
- all symbol-specific graphics oriented update calls are adjusted to
pass in the fqsn:
* `update_fsp_chart()`
* `ChartView._set_yrange()`
* ChartPlotWidget.update_graphics_from_flow()`
- avoid a double increment on sample step updates by not calling the
increment on any vlm chart since it seems the vlm-ohlc chart linking
already takes care of this now?
- use global counters for the last epoch time step to avoid incrementing
all views more then once per new time step given underlying shm array
buffers may be on different array-index values from one another.
This is to prep for multi-symbol feeds and charts so we accept
a sequence of fqsns to the top level entrypoints as well as the
`.data.feed.open_feed()` API (though we're not actually supporting true
multiplexed feeds nor shm lookups per fqsn yet).
Factor the multi-sample-rate region UI connecting into a new helper
`link_views_with_region()` which reads in the shm buffer offsets from
the `Feed` and appropriately connects the fast and slow chart handlers
for the linear region graphics. Add detailed comments writeup for the
inter-sampling transform algebra.
- Every time a symbol is switched on chart we need to wait until the
search bar sidepane has been added beside the slow chart before
determining the offset for the pp line's arrow/labels; trigger this in
`GodWidget.load_symbol()` -> required monkeypatching on a
`.mode: OrderMode` to the `.rt_linked` for now..
- Drop the search pane widget removal from the current linked chart,
seems faster?
- On the slow chart override the `LevelMarker.scene_x()` callback to
adjust for the case where no L1 labels are shown beside the y-axis.
Also adds a `GodWidget.resize_all()` helper method which resizes all
sub-widgets and charts to their default ratios and/or parent-widget
dependent defaults using the detected available space on screen. This is
a "default layout" config method that eventually we'll probably want
allow users to customize.
It ended up being what'd you expect, races on the accessing shm buffer
data by the UI during the whole "mega-async-startup-everything" phase XD
So we add the following list of ad-hoc startup steps:
- do `.default_view()` on the slow chart after the fast chart is mostly
fully spawned with the intention being to capture the state where the
historical buffer is mostly loaded before sizing the view to the
graphical form of the data.
- resize slow chart sidepanes from the fast chart just before sleeping
forever (and after order mode has booted).
The slow (history) chart requires it's own y-range checker logic which
needs to be run in 2 cases:
- the last datum is in view and goes outside the previous mx/mn in view
- the chart is incremented a step
Since we need this duplicate logic this patch also factors the incremental
graphics update info "reading" into a new `DisplayState.incr_info()`
method that can be configured to a chart and input state and returns all
relevant "graphics update measure" in a tuple (for now).
Use this method throughout the rest of the display loop for both fast
and slow chart checks and in the `increment_history_view()` slow chart
task.
Use the new `Feed.get_ds_info()` method in a poll loop to definitively
get the inter-chart sampling info and avoid races with shm buffer
backfilling.
Also, factor the history increment closure-task into
`graphics_update_loop()` which will make it clearer how to factor
all the "should we update" logic into some `DisplayState` API.
Add an update call to the display loop to consistently update the last
datum in the history view chart. Compute the inter-chart sampling ratio
and use it to sync the linear region.
Add a first draft of a working `pyqtgraph.LinearRegionItem` link between
a history view chart (+ data set) and the normal real-time "HFT" chart
set.
Add the history view (aka more downsampled data view) chart set to the
rt/hft set's splitter as it's "first widget". Hook up linear region
callbacks to enable syncing between charts including compenstating for
the downsampling rate ration (in this case hardcoded 60 since 1s to 1M,
but we'll actually compute it going forward obvs).
More to come dawgys..
This was introduced in #302 but after thorough testing was clear to be
not working XD. Adjust the display loop to update the last graphics
segment on both the OHLC and vlm charts (as well as all deriving fsp
flows) whenever the uppx >= 1 and there is no current path append
taking place (since more datums are needed to span an x-pixel in view).
Summary of tweaks:
- move vlm chart update code to be at the end of the cycle routine and
have that block include the tests for a "interpolated last datum in
view" line.
- make `do_append: bool` compare with a floor of the uppx value (i.e.
appends should happen when we're just fractionally over a pixel of
x units).
- never update the "volume" chart.
When using m4, we downsample to the max and min of each
pixel-column's-worth of data thus preserving range / dispersion details
whilst not drawing more graphics then can be displayed by the available
amount of horizontal pixels.
Take and apply this exact same concept to the "last datum" graphics
elements for any `Flow` that is reported as being in a downsampled
state:
- take the xy output from the `Curve.draw_last_datum()`,
- slice out all data that fits in the last pixel's worth of x-range
by using the uppx,
- compute the highest and lowest value from that data,
- draw a singe line segment which spans this yrange thus creating
a simple vertical set of pixels which are "filled in" and show the
entire y-range for the most recent data "contained with that pixel".
The basic logic is now this:
- when zooming out, uppx (units per pixel in x) can be >= 1
- if the uppx is `n` then the next pixel in view becomes occupied by
a new datum-x-coordinate-value when the diff between the last
datum step (since the last such update) is greater then the
current uppx -> `datums_diff >= n`
- if we're less then some constant uppx we just always update (because
it's not costly enough and we're not downsampling.
More or less this just avoids unnecessary real-time updates to flow
graphics until they would actually be noticeable via the next pixel
column on screen.
The most important changes include:
- iterating the new `Flow` type and updating graphics
- adding detailed profiling
- increasing the min uppx before graphics updates are throttled
- including the L1 spread in y-range calcs so that you never have the
bid/ask go "out of view"..
- pass around `Flow`s instead of shms
- drop all the old prototyped downsampling code
Ugh, turns out the wacky `ChartView.maxmin` callback stuff we did (for
determining y-range sizings) currently requires that the volume array
has a "bars in view" result.. so let's make that keep working without
rendering the graphics for the curve (since we're disabling them once
$vlm comes up).
Probably the best place to root the profiler since we can get a better
top down view of bottlenecks in the graphics stack.
More,
- add in draft M4 downsampling code (commented) after getting it mostly
working; next step is to move this processing into an FSP subactor.
- always update the vlm chart last y-axis sticky
- set call `.default_view()` just before inf sleep on startup
- the chart's uppx (units-per-pixel) is > 4 (i.e. zoomed out a lot)
- don't shift the chart (to keep the most recent step in view) if the
last datum isn't in view (aka the user is probably looking at history)
The graphics update loop is much easier to grok when all the UI
components which potentially need to be updated on a cycle are arranged
together in a high-level composite namespace, thus this new
`DisplayState` addition. Create and set this state on each
`LinkedSplits` chart set and add a new method `.graphics_cycle()` which
let's a caller trigger a graphics loop update manually. Use this method
in the fsp graphics manager such that a chain can update new history
output even if there is no real-time feed driving the display loop (eg.
when a market is "closed").
Since moving to a "god loop" for graphics, we don't really need to have
a dedicated task for updating graphics on new sample increments. The
only UX difference will be that curves won't be updated until an actual new
rt-quote-event triggers the graphics loop -> so we'll have the chart
"jump" to a new position and new curve segments generated only when new
data arrives. This is imo fine since it's just less "idle" updates
where the chart would sit printing the same (last) value every step.
Instead only update the view increment if a new index is detected by
reading shm.
If we ever want this dedicated task update again this commit can be
easily reverted B)
This is a huge commit which moves a bunch of code around in order to
simplify some of our UI modules as well as support our first official
mult-axis chart: overlaid volume and "dollar volume". A good deal of
this change set is to make startup fast such that volume data which is
often shipped alongside OHLC history is loaded and shown asap and FSPs
are loaded in an actor cluster with their graphics overlayed
concurrently as each responsible worker generates plottable output.
For everything to work this commit requires use of a draft `pyqtgraph`
PR: https://github.com/pyqtgraph/pyqtgraph/pull/2162
Change summary:
- move remaining FSP actor cluster helpers into `.ui._fsp` mod as well
as fsp specific UI managers (`maybe_open_vlm_display()`,
`start_fsp_displays()`).
- add an `FspAdmin` API for starting fsp chains on the cluster
concurrently allowing for future work toward reload/unloading.
- bring FSP config dict into `start_fsp_displays()` and `.started()`-deliver
both the fsp admin and any volume chart back up to the calling display
loop code.
ToDo:
- repair `ChartView` click-drag interactions
- auto-range on $ vlm needs to use `ChartPlotWidget._set_yrange()`
- a lot better styling for the $_vlm overlay XD
Tyler Goodlet
Guillermo Rodriguez