Instead of referencing the remote processing funcs by a `str` name start
embracing the new `@fsp`/`Fsp` API such that wrapped processing
functions are first class APIs.
Summary of the changeset:
- move and load the fsp built-in set in the new `.fsp._api` module
- handle processors ("fsps") which want to yield multiple keyed-values
(interleaved in time) by expecting both history that is keyed and
assigned to the appropriate struct-array field, *and* real-time
`yield`ed value in tuples of the form `tuple[str, float]` such that
any one (async) processing function can deliver multiple outputs from
the same base calculation.
- drop `maybe_mk_fsp_shm()` from UI module
- expect and manage `Fsp` instances (`@fsp` decorated funcs) throughout
the UI code, particularly the `FspAdmin` layer.
Since more curves costs more processing and since the vlm and $vlm
curves are normally very close to the same (graphically) we hide the
unit volume curve once the dollar volume is up (after the fsp daemon-task is
spawned) and just expect the user to understand the diff in axes units.
Also, use the new `title=` api to `.overlay_plotitem()`.
Use our internal `Label` with much better dpi based sizing of text and
placement below the y-axis ticks area for more minimalism and less
clutter.
Play around with `lru_cache` on axis label bounding rects and for now
just hack sizing by subtracting half the text height (not sure why) from
the width to avoid over-extension / overlap with any adjacent axis.
Allow passing in a formatter function for processing tick values on an
axis. This makes it easy to for example, `piker.calc.humanize()` dollar
volume on a subchart.
Factor `set_min_tick()` into the `PriceAxis` since it's not used on any
x-axis data thus far.
Adds `FspAdmin.open_fsp_chart()` which allows adding a real time graphics
display of an fsp's output with different options for where (which chart
or make a new one) to place it.
Further,
- change some method naming, namely the other fsp engine task methods to
`.open_chain()` and `.start_engine_task()`.
- make `run_fsp_ui()` a lone task function for now with the default
config parsing and chart setup logic (and it still includes a buncha
commented out stuff for doing graphics update which is now done in the
main loop to avoid task switching overhead).
- move all vlm related fsp config entries into the `open_vlm_displays()`
task for dedicated setup with the fsp admin api such as special
auto-yrange handling and graph overlays.
- `start_fsp_displays()` is now just a small loop through config entries
with synced startup status messages.
For wtv cucked reason all the viewbox/scene coordinate calcs do **not**
include a left axis in the geo (likely because it's a hacked in widget
+ layout thing managed by `PlotItem`). Detect if there's a left axis and
if so use it in the label placement scene coords calc. ToDo: probably
make this a non-move calc and only recompute any time the axis changes.
Other:
- rate limit mouse events down to the 60 (ish) Hz for now
- change one last lingering `'ohlc'` array lookup
- fix `.mouseMoved()` "event" type annot
This is a huge commit which moves a bunch of code around in order to
simplify some of our UI modules as well as support our first official
mult-axis chart: overlaid volume and "dollar volume". A good deal of
this change set is to make startup fast such that volume data which is
often shipped alongside OHLC history is loaded and shown asap and FSPs
are loaded in an actor cluster with their graphics overlayed
concurrently as each responsible worker generates plottable output.
For everything to work this commit requires use of a draft `pyqtgraph`
PR: https://github.com/pyqtgraph/pyqtgraph/pull/2162
Change summary:
- move remaining FSP actor cluster helpers into `.ui._fsp` mod as well
as fsp specific UI managers (`maybe_open_vlm_display()`,
`start_fsp_displays()`).
- add an `FspAdmin` API for starting fsp chains on the cluster
concurrently allowing for future work toward reload/unloading.
- bring FSP config dict into `start_fsp_displays()` and `.started()`-deliver
both the fsp admin and any volume chart back up to the calling display
loop code.
ToDo:
- repair `ChartView` click-drag interactions
- auto-range on $ vlm needs to use `ChartPlotWidget._set_yrange()`
- a lot better styling for the $_vlm overlay XD
As part of factoring `._set_yrange()` into the lower level view box,
move the y-range calculations into a new method. These calcs should
eventually be completely separate (as they are for the real-time version
in the graphics display update loop) and likely part of some kind of
graphics-related lower level management API. Draft such an API as an
`ArrayScene` (commented for now) as a sketch toward factoring array
tracking **out of** the chart widget. Drop the `'ohlc'` array name and
instead always use whatever `.name` was assigned to the chart widget
to lookup its "main" / source data array for now.
Enable auto-yranging on overlayed plotitems by enabling on its viewbox
and, for now, assign an ad-hoc `._maxmin()` since the widget version
from this commit has no easy way to know which internal array to use. If
an FSP (`dolla_vlm` in this case) is overlayed on an existing chart
without also having a full widget (which it doesn't in this case since
we're using an overlayed `PlotItem` instead of a full `ChartPlotWidget`)
we need some way to define the `.maxmin()` for the overlayed
data/graphics. This likely means the `.maxmin()` will eventually get
factored into wtv lowlevel `ArrayScene` API mentioned above.
Calculations for auto-yaxis ranging are both signalled and drawn by our
`ViewBox` so we might as well factor this handler down from the chart
widget into the view type. This makes it much easier (and clearer) that
`PlotItem` and other lower level overlayed `GraphicsObject`s can utilize
*size-to-data* style view modes easily without widget-level coupling.
Further changes,
- support a `._maxmin()` internal callable (temporarily) for allowing
a viewed graphics object to define it's own y-range max/min calc.
- add `._static_range` var (though usage hasn't been moved from the
chart plot widget yet
- drop y-axis click-drag zoom instead reverting back to default viewbox
behaviour with wheel-zoom and click-drag-pan on the axis.
This brings in the WIP components developed as part of
https://github.com/pyqtgraph/pyqtgraph/pull/2162.
Most of the history can be understood from that issue and effort but the
TL;DR is,
- add an event handler wrapper system which can be used to
wrap `ViewBox` methods such that multiple views can be overlayed and
a single event stream broadcast from one "main" view to others which
are overlaid with it.
- add in 2 relay `Signal` attrs to our `ViewBox` subtype (`Chartview`)
to accomplish per event `MouseEvent.emit()` style broadcasting to
multiple (sub-)views.
- Add a `PlotItemOverlay` api which does all the work of overlaying the
actual chart graphics and arranging multiple-axes without collision as
well as tying together all the event/signalling so that only a single
"focussed" view relays to all overlays.
Each `pyqtgraph.PlotItem` uses a `QGraphicsGridLayout` to place its view
box, axes and titles in the traditional graph format. With multiple
overlayed charts we need those axes to not collide with one another and
further allow for an "order" specified by the user. We accomplish this
by adding `QGraphicsLinearLayout`s for each axis "side": `{'left',
'right', 'top', 'bottom'}` such that plot axes can be inserted and moved
easily without having to constantly re-stack/order a grid layout (which
does not have a linked-list style API).
The new type is called `ComposedGridLayout` for now and offers a basic
list-like API with `.insert()`, `.append()`, and eventually a dict-style
`.pop()`. We probably want to also eventually offer a `.focus()` to
allow user switching of *which* main graphics object (aka chart) is "in
use".
This syncs with a dev branch in our `pyqtgraph` fork:
https://github.com/pyqtgraph/pyqtgraph/pull/2162
The main idea is to get mult-yaxis display fully functional with
multiple view boxes running in a "relay mode" where some focussed view
relays signals to overlaid views which may have independent axes. This
preps us for both displaying independent codomain-set FSP output as well
as so called "aggregate" feeds of multiple fins underlyings on the same
chart (eg. options and futures over top of ETFs and underlying stocks).
The eventual desired UX is to support fast switching of instruments for
order mode trading without requiring entirely separate charts as well as
simple real-time anal of associated instruments.
The first effort here is to display vlm and $_vlm alongside each other
as a built-in FSP subchart.
We can instead use the god widget's nursery to schedule all the feed
pause/resume requests and be even more concurrent during a view (of
symbols) switch.
Use `tractor.trionics.gather_contexts()` to start up the fsp and volume
chart-displays (for an additional conc speedup). Drop `dolla_vlm` again for
now until we figure out how we can display it *and* vlm on the same
sub-chart? It would be nice to avoid having to spawn an fsp process
before showing the volume curve.
Call the resize method only after all FSP subcharts have rendered
such that the main OHLC chart's final width is read.
Further tweaks:
- drop rsi by default
- drop the stream drain stuff
- fix failed-to-read shm logging
This fixes a weird re-render bug/slowdown/artifact that was introduced
with the order mode sidepane work. Prior to the sidepane addition, chart
switching was immediate with zero noticeable widget rendering steps.
The slow down was caused by 2 things:
- not yielding back to the Qt loop asap after re-showing/focussing
a linked split chart that was already in memory.
- pausing/resuming feeds only after a Qt loop render cycle has
completed.
This now restores the near zero latency UX.
There was a lingering issue where the fsp daemon would sync its shm
array with the source data and we'd set the start/end indices to the
same value. Under some races a reader would then read an empty `.array`
which it wasn't expecting. This fixes that as well as tidies up the
`ShmArray.push()` logic and adds a temporary check in `.array` for zero
length if the array hasn't been written yet.
We can now start removing read array length checks in consumer code
and hopefully no more races will show up.
Revert to old shm "last" meaning last row
It can now be declared inside an fsp config dict under the name
`dolla_vlm`. We still need to offer an engine control that zeros
the newest sample value instead of copying from the previous.
This also litters the engine code with `pyqtgraph` profiling to see if
we can improve startup times - likely it'll mean pre-allocating a small
fsp daemon cluster at startup.
Use a fixed worker count and don't respawn for every chart, instead
opting for a round-robin to tasks in a cluster and (for now) hoping for
the best in terms of trio scheduling, though we should obviously route
via symbol-locality next. This is currently a boon for chart spawning
startup times since actor creation is done AOT.
Additionally,
- use `zero_on_step` for dollar volume
- drop rsi on startup (again)
- add dollar volume (via fsp) along side unit volume
- litter more profiling to fsp chart startup sequence
- pre-define tick type classes for update loop
We are already packing framed ticks in extended lists from
the `.data._sampling.uniform_rate_send()` task so the natural solution
to avoid needless graphics cycles for HFT-ish feeds (like binance) is
to unpack those frames and for most cases only update graphics with the
"latest" data per loop iteration. Unpacking in this way also lessens
nested-iterations per tick type.
Btw, this also effectively solves all remaining issues of fast tick
feeds over-triggering the graphics loop renders as long as the original
quote stream is throttled appropriately, usually to the local display
rate.
Relates to #183, #192
Dirty deats:
- drop all per-tick rate checks, they were always somewhat pointless
when iterating a frame of ticks per render cycle XD.
- unpack tick frame into ticks per frame type, and last of each type;
the lasts are used to update each part of the UI/graphics by class.
- only skip the label update if we can't retrieve the last from from a
graphics source array; it seems `chart.update_curve_from_array()`
already does a `len` check internally.
- add some draft commented code for tick type classes and a possible
wire framed tick data structure.
- move `chart_maxmin()` range computer to module level, bind a chart to
it with a `partial.`
- only check rate limits in main quote loop thus reporting actual
overages
- add in commented logic for only updating the "last" cleared price from
the most recent framed value if we want to eventually (right now seems
like this is only relevant to ib and it's dark trades: `utrade`).
- rename `_clear_throttle_rate` -> `_quote_throttle_rate`, drop
`_book_throttle_rate`.
This is in prep toward doing fsp graphics updates from the main quotes
update loop (where OHLC and volume are done). Updating fsp output from
that task should, for the majority of cases, be fine presuming the
processing is derived from the quote stream as a source. Further,
calling an update function on each fsp subplot/overlay is of course
faster then a full task switch - which is how it currently works with
a separate stream for every fsp output. This also will let us delay
adding full `Feed` support around fsp streams for the moment while still
getting quote throttling dictated by the quote stream.
Going forward, We can still support a separate task/fsp stream for
updates as needed (ex. some kind of fast external data source that isn't
synced with price data) but it should be enabled as needed required by
the user.
The major change is moving the fsp "daemon" (more like wanna-be fspd)
endpoint to use the newer `tractor.Portal.open_context()` and
bi-directional streaming api.
There's a few other things in here too:
- make a helper for allocating single colume fsp shm arrays
- rename some some fsp related functions to be more explicit on their
purposes
Since our startup is very concurrent there is often races where widgets
have not fully spawned before python (re-)sizing code has a chance to
run sizing logic and thus incorrect dimensions are read. Instead ensure
the Qt render loop gets to run in between such checks.
Also add a `open_sidepane()` mngr for creating a minimal form widget for
FSP subchart sidepanes which can be configured from an input `dict`.
This should in theory result in increased burstiness since we remove
the plain `trio.sleep()` and instead always wait on the receive channel
as much as possible until the `trio.move_on_after()` (+ time diffing
calcs) times out and signals the next throttled send cycle. This also is
slightly easier to grok code-wise instead of the `try, except` and
another tight while loop until a `trio.WouldBlock`. The only simpler
way i can think to do it is with 2 tasks: 1 to collect ticks and the
other to read and send at the throttle rate.
Comment out the log msg for now to avoid latency and add much more
detailed comments. Add an overrun log msg to the main sample loop.
A `QRectF` is easier to make and draw (i think?) so use that and fill it
on volume events for decent sleek real-time look. Adjust the step array
generator to allow for an endpoints flag. Comment and/or clean out all
the old path filling calls that gave us perf issues..
Turns out the performance of updating and refilling step curves > 1k ish
points is super slow :sadkek:. Disabling the fill basically returns
normal performance, so it seems maybe we'll stick with unfilled volume
"bars" for now. The other tricky bit is getting the path to extend and
fill which is particularly slow if you use the `QPainterPath.united()`
(what `+` set op does) operation which seems to require an entire redraw
of the curve each paint iteration. Removing the pixel buffer cache makes
things that much worse too..
One technique i tried was only setting a `._fill` flag when so many
datums are in view (< 1k as determined by the chart widget), and this
helps, but under high load (trade rates) you still see more lag then
without the fill which makes me say screw it and let's stick with
unfilled bars for now. Trying go to get performant filled curves will be
an exercise for an aspiring graphics eng :P
In latest `pyqtgraph` it seems there's a discrepancy
since `function.arrayToQPath()` was reworked and now
we need to *not* connect the last point for each bar.
The prior PR for fixing fsp array misalignment also added
`tractor.Context` usage which wasn't reflected in the graphics update
loop (newer code added it but the prior PR was factored from path
dependent history) and thus was broken. Further in newer work we don't
have fsp actors actually stream value updates since the display loop can
already pull from the source feed and update graphics at a preferred
throttle rate. Re-enabled the fsp stream sending here by default until
that newer only-throttle-pull-from-source code is landed in the display
loop.
This should finally be correct fsp src-to-dst array syncing now..
There's a few edge cases but mostly we need to be sure we sync both
back-filled history diffs and avoid current step lag/leads. Use
a polling routine and the more stringent task re-spawn system to get
this right.
There was a lingering issue where the fsp daemon would sync its shm
array with the source data and we'd set the start/end indices to the
same value. Under some races a reader would then read an empty `.array`
which it wasn't expecting. This fixes that as well as tidies up the
`ShmArray.push()` logic and adds a temporary check in `.array` for zero
length if the array hasn't been written yet.
We can now start removing read array length checks in consumer code
and hopefully no more races will show up.
Litter the engine code with `pyqtgraph` profiling to see if we can
improve startup times - likely it'll mean pre-allocating a small fsp
daemon cluster at startup.
Split up the rather large `.ui._chart` module into its constituents:
- a `.ui._app` for the highlevel widget composition, qtractor entry
point and startup logic
- `.ui._display` for all the real-time graphics update tasks which
consume the `.ui._chart` widget apis
Must have run into some confusion with data structures in `brokerd` vs.
`emsd`. This fixes the ems `relay.positions` state tracking to be
composed maps, vs. messages from `brokerd` should just be a sequence.
This reverts commit 6fa8958acf.
We actually do need it since the selection widget of course won't tell
you its "key" that we assign and further we'd have to use a (value, key)
style invocation which isn't super pythonic.
The paper engine returns `"paper"` instead of `None` in the pp msgs so
expect that. Don't bother with fills tracking for now (since we'll need
either the account in the msg or a lookup table locally for oids to
accounts). Change the order line update handler to a local module function,
there was no reason for it to be a pane method.
Make a pp tracker per account and load on order mode boot.
Only show details on the pp tracker for the selected account.
Make the settings pane assign a `.current_pp` state on the order mode
instance (for the charted symbol) on account selection switches and no
longer keep a ref to a single pp tracker and allocator in the pane.
`SettingsPane.update_status_ui()` now expects an explicit tracker
reference as input. Still need to figure out the pnl update task logic
despite the intermittent account changes.
This adds full support for a single `brokerd` managing multiple API
endpoint clients in tandem. Get the client scan loop correct and load
accounts from all discovered clients as specified in a user's
`broker.toml`. We now just always re-scan for all clients and if there's
a cache hit just skip a creation/connection logic.
Route orders with an account name to the correct client in the
`handle_order_requests()` endpoint and spawn an event relay task per
client for transmitting trade events back to `emsd`.
Make the `handle_order_requests()` tasks now lookup the appropriate API
client for a given account (or error if it can't be found) and use it
for submission. Account names are loaded from the
`brokers.toml::accounts.ib` section both UI side and in the `brokerd`.
Change `_aio_get_client()` to a `load_aio_client()` which now tries to
scan and load api clients for all connections defined in the config as
well as deliver the client cache and account lookup tables.
Each backend broker may support multiple (types) of accounts; this patch
lets clients send order requests that pass through an `account` field in
certain `emsd` <-> `brokerd` transactions. This allows each provider to read
in and conduct logic based on what account value is passed via requests
to the `trades_dialogue()` endpoint as well as tie together positioning
updates with relevant account keys for display in UIs.
This also adds relay support for a `Status` msg with a `'broker_errored'`
status which for now will trigger the same logic as cancelled orders on
the client side and thus will remove order lines submitted on a chart.
Get rid of `PositionTracker.init_status_ui()` and instead make
a helper func `mk_allocator()` which takes in the alloc and adjusts
default settings on the allocator alone (which is expected to be
passed in). Expect a `Position` instance to be passed into the tracker
which will be looked up for UI updates. Move *update-from-position-msg*
ops into a `Position.update_from_msg()` method.
We weren't updating the LHS size labels on creation and we now use the
lot size digits to do so. Change `PositionTracker.update()` to
`.update_from_pp_msg()`.
Acts as a fix for lodpi and better sizing logic for the pp status bar.
Drop all the redundant passing of the form to its child layouts during
instantiating (since they're all added as layouts to the tree). Comment
out the feed status label for now since it's not hooked up to the
backend and we'll get it going in a new PR.
Down the road we probably want to do all the pp pane component-widget
sizing *after* the `pyqtgraph` chart is up; it's going to take some
reworking of the charting api tho.
We were re-implementing a few things order lines already support.
All we really needed was to not add a pp size label if one is provided.
Use `.hide_label()` in the mouse hover handler.
When exiting a pp toward net-zero, we may sometimes run into the issue
of having a "fractional slot" worth of units in allocator limit terms.
This is further nuanced by live orders which are submitted above the
current clearing price which get allocated a size (based on that staged
but non-cleared price) according to their limit size unit which can be
calculated to be less then the size that would have been allocated at
the actual clearing price. In the short term cope with this discrepancy
by simply using a "slot and a half" as the decision point of whether to
exit a slot's worth or the remaining pp's worth of units. In other words
if you can exit 1.5x a slot's worth or less, exit the remaining pp,
otherwise exit a slot's worth. This is a stop gap until we have a better
solution to limiting staged orders to (some range around) the currently
computed clear-able price.
We need a subtask to compute the current pp PnL in real-time but really
only if a pp exists - a spawnable subtask would be ideal for this. Stage
a tick streaming task using a stream bcaster; no actual pnl calc yet.
Since we're going to need subtasks anyway might as well stick the order
mode UI processing loop in a task as well and then just give the whole
thing a ctx mngr api. This'll probably be handy for when we have
auto-strats that need to dynamically use the mode's api as well.
Oh, and move the time -> index mapper to a chart method for now.
Use this method to go through writing all allocator parameters and then
reading all changes back into the order mode pane including updating the
limit and step labels by the fill bar.
Machinery changes:
- add `.limit()` and `.step_sizes()` methods to the allocator to
provide the appropriate data depending on the pp limit size unit (eg.
currency vs. units)
- humanize the label display text such that you have nice suffixes and
a fixed precision
- tweak the fill bar labels to be simpler since the values are now
humanized
- expect `.on_ui_settings_change()` to be called for every slots hotkey
tweak
Turned out to be pretty simple, on every pp update just recompute
the proportion of slots used based on the limit size units.
Don't assign the allocator callback method for alert lines since
there's no size to generate. Move from-existing-pp calculations
into the order pane itself.
Handling the edge cases in this was "fun", namely:
- entering with less then a slot's worth of units to purchase
before hitting the pp limit or, less then a slots worth when exiting
toward a net-zero position.
- round pp msg updates using the symbol tick and lot size digits to
avoid super small (1e-30 lel) positions lingering in the ems (happens
moreso with the paper engine).
- don't expect the next size method to be called for alert level changes
- pass label text and field widget key separately
- fix fill status bar slot sizing logic (once and for all) and
create a new type that allows generating / resizing the bar's
size / values with a `.set_slots()` method
- pull account names from allocator attr
- set `.fill_bar` as the fill status bar on the form for now
- make `GodWidget.load_symbol()` async
- track loaded feeds with a private `._feeds` dict
- add methods to pause/resume all feeds when chart is (un)focussed
- add some commented test code for 2nd feed consumer task and rsi2 fsp
- load async signal handler for view clicking
- generate lines from staged `Order` msgs
- apply level update callback to each order that dynamically
updates the order size from the allocator calcs
- pass order msg instances to the ems client for submission
- update order size on line moves
- add `Order` msg and `Symbol` refs to each dialog
In an effort to simplify line creation and management from an order
mode here's a slew of changes:
- use our new ``LevelMarker`` for order lines and fully drop usage
of the original marker implementation stuff from `pg.InfiniteLine`
- add a left side label which shows the instrument's "units" value
- the most fundamental unit for the "size" of the order
- allow passing in an optional `marker_size: str` so that `action: str`
doesn't necessarily have to be passed (eg. when copying from an
existing line)
- change a couple of internal line config options to be public attrs
which can now be configured dynamically in real-time (since they're
all `bool` anyway):
* `hl_on_hover` -> `highlight_on_hover`
* `_always_show_labels` -> `always_show_labels`
- `LevelLine.set_level()` now only sets the position if it was **not**
called from the position changed signal (which would be redundant)
Move all the ``pydantic`` finagling to an `_orm.py` and
just keep an `Allocator` as the backing model for our pp controls
in the position module. This all needs to be tied together in some sane
with with facility for multiple symbols/streams per chart for when we
get to charting-trading aggregate feeds.
Tyler Goodlet
wattygetlood