Use a fixed worker count and don't respawn for every chart, instead
opting for a round-robin to tasks in a cluster and (for now) hoping for
the best in terms of trio scheduling, though we should obviously route
via symbol-locality next. This is currently a boon for chart spawning
startup times since actor creation is done AOT.
Additionally,
- use `zero_on_step` for dollar volume
- drop rsi on startup (again)
- add dollar volume (via fsp) along side unit volume
- litter more profiling to fsp chart startup sequence
- pre-define tick type classes for update loop
We are already packing framed ticks in extended lists from
the `.data._sampling.uniform_rate_send()` task so the natural solution
to avoid needless graphics cycles for HFT-ish feeds (like binance) is
to unpack those frames and for most cases only update graphics with the
"latest" data per loop iteration. Unpacking in this way also lessens
nested-iterations per tick type.
Btw, this also effectively solves all remaining issues of fast tick
feeds over-triggering the graphics loop renders as long as the original
quote stream is throttled appropriately, usually to the local display
rate.
Relates to #183, #192
Dirty deats:
- drop all per-tick rate checks, they were always somewhat pointless
when iterating a frame of ticks per render cycle XD.
- unpack tick frame into ticks per frame type, and last of each type;
the lasts are used to update each part of the UI/graphics by class.
- only skip the label update if we can't retrieve the last from from a
graphics source array; it seems `chart.update_curve_from_array()`
already does a `len` check internally.
- add some draft commented code for tick type classes and a possible
wire framed tick data structure.
- move `chart_maxmin()` range computer to module level, bind a chart to
it with a `partial.`
- only check rate limits in main quote loop thus reporting actual
overages
- add in commented logic for only updating the "last" cleared price from
the most recent framed value if we want to eventually (right now seems
like this is only relevant to ib and it's dark trades: `utrade`).
- rename `_clear_throttle_rate` -> `_quote_throttle_rate`, drop
`_book_throttle_rate`.
This is in prep toward doing fsp graphics updates from the main quotes
update loop (where OHLC and volume are done). Updating fsp output from
that task should, for the majority of cases, be fine presuming the
processing is derived from the quote stream as a source. Further,
calling an update function on each fsp subplot/overlay is of course
faster then a full task switch - which is how it currently works with
a separate stream for every fsp output. This also will let us delay
adding full `Feed` support around fsp streams for the moment while still
getting quote throttling dictated by the quote stream.
Going forward, We can still support a separate task/fsp stream for
updates as needed (ex. some kind of fast external data source that isn't
synced with price data) but it should be enabled as needed required by
the user.
The major change is moving the fsp "daemon" (more like wanna-be fspd)
endpoint to use the newer `tractor.Portal.open_context()` and
bi-directional streaming api.
There's a few other things in here too:
- make a helper for allocating single colume fsp shm arrays
- rename some some fsp related functions to be more explicit on their
purposes
Since our startup is very concurrent there is often races where widgets
have not fully spawned before python (re-)sizing code has a chance to
run sizing logic and thus incorrect dimensions are read. Instead ensure
the Qt render loop gets to run in between such checks.
Also add a `open_sidepane()` mngr for creating a minimal form widget for
FSP subchart sidepanes which can be configured from an input `dict`.
A `QRectF` is easier to make and draw (i think?) so use that and fill it
on volume events for decent sleek real-time look. Adjust the step array
generator to allow for an endpoints flag. Comment and/or clean out all
the old path filling calls that gave us perf issues..
Turns out the performance of updating and refilling step curves > 1k ish
points is super slow :sadkek:. Disabling the fill basically returns
normal performance, so it seems maybe we'll stick with unfilled volume
"bars" for now. The other tricky bit is getting the path to extend and
fill which is particularly slow if you use the `QPainterPath.united()`
(what `+` set op does) operation which seems to require an entire redraw
of the curve each paint iteration. Removing the pixel buffer cache makes
things that much worse too..
One technique i tried was only setting a `._fill` flag when so many
datums are in view (< 1k as determined by the chart widget), and this
helps, but under high load (trade rates) you still see more lag then
without the fill which makes me say screw it and let's stick with
unfilled bars for now. Trying go to get performant filled curves will be
an exercise for an aspiring graphics eng :P
In latest `pyqtgraph` it seems there's a discrepancy
since `function.arrayToQPath()` was reworked and now
we need to *not* connect the last point for each bar.
The prior PR for fixing fsp array misalignment also added
`tractor.Context` usage which wasn't reflected in the graphics update
loop (newer code added it but the prior PR was factored from path
dependent history) and thus was broken. Further in newer work we don't
have fsp actors actually stream value updates since the display loop can
already pull from the source feed and update graphics at a preferred
throttle rate. Re-enabled the fsp stream sending here by default until
that newer only-throttle-pull-from-source code is landed in the display
loop.
Split up the rather large `.ui._chart` module into its constituents:
- a `.ui._app` for the highlevel widget composition, qtractor entry
point and startup logic
- `.ui._display` for all the real-time graphics update tasks which
consume the `.ui._chart` widget apis
This reverts commit 6fa8958acf.
We actually do need it since the selection widget of course won't tell
you its "key" that we assign and further we'd have to use a (value, key)
style invocation which isn't super pythonic.
The paper engine returns `"paper"` instead of `None` in the pp msgs so
expect that. Don't bother with fills tracking for now (since we'll need
either the account in the msg or a lookup table locally for oids to
accounts). Change the order line update handler to a local module function,
there was no reason for it to be a pane method.
Make a pp tracker per account and load on order mode boot.
Only show details on the pp tracker for the selected account.
Make the settings pane assign a `.current_pp` state on the order mode
instance (for the charted symbol) on account selection switches and no
longer keep a ref to a single pp tracker and allocator in the pane.
`SettingsPane.update_status_ui()` now expects an explicit tracker
reference as input. Still need to figure out the pnl update task logic
despite the intermittent account changes.
Each backend broker may support multiple (types) of accounts; this patch
lets clients send order requests that pass through an `account` field in
certain `emsd` <-> `brokerd` transactions. This allows each provider to read
in and conduct logic based on what account value is passed via requests
to the `trades_dialogue()` endpoint as well as tie together positioning
updates with relevant account keys for display in UIs.
This also adds relay support for a `Status` msg with a `'broker_errored'`
status which for now will trigger the same logic as cancelled orders on
the client side and thus will remove order lines submitted on a chart.
Get rid of `PositionTracker.init_status_ui()` and instead make
a helper func `mk_allocator()` which takes in the alloc and adjusts
default settings on the allocator alone (which is expected to be
passed in). Expect a `Position` instance to be passed into the tracker
which will be looked up for UI updates. Move *update-from-position-msg*
ops into a `Position.update_from_msg()` method.
We weren't updating the LHS size labels on creation and we now use the
lot size digits to do so. Change `PositionTracker.update()` to
`.update_from_pp_msg()`.
Acts as a fix for lodpi and better sizing logic for the pp status bar.
Drop all the redundant passing of the form to its child layouts during
instantiating (since they're all added as layouts to the tree). Comment
out the feed status label for now since it's not hooked up to the
backend and we'll get it going in a new PR.
Down the road we probably want to do all the pp pane component-widget
sizing *after* the `pyqtgraph` chart is up; it's going to take some
reworking of the charting api tho.
We were re-implementing a few things order lines already support.
All we really needed was to not add a pp size label if one is provided.
Use `.hide_label()` in the mouse hover handler.
When exiting a pp toward net-zero, we may sometimes run into the issue
of having a "fractional slot" worth of units in allocator limit terms.
This is further nuanced by live orders which are submitted above the
current clearing price which get allocated a size (based on that staged
but non-cleared price) according to their limit size unit which can be
calculated to be less then the size that would have been allocated at
the actual clearing price. In the short term cope with this discrepancy
by simply using a "slot and a half" as the decision point of whether to
exit a slot's worth or the remaining pp's worth of units. In other words
if you can exit 1.5x a slot's worth or less, exit the remaining pp,
otherwise exit a slot's worth. This is a stop gap until we have a better
solution to limiting staged orders to (some range around) the currently
computed clear-able price.
We need a subtask to compute the current pp PnL in real-time but really
only if a pp exists - a spawnable subtask would be ideal for this. Stage
a tick streaming task using a stream bcaster; no actual pnl calc yet.
Since we're going to need subtasks anyway might as well stick the order
mode UI processing loop in a task as well and then just give the whole
thing a ctx mngr api. This'll probably be handy for when we have
auto-strats that need to dynamically use the mode's api as well.
Oh, and move the time -> index mapper to a chart method for now.
Use this method to go through writing all allocator parameters and then
reading all changes back into the order mode pane including updating the
limit and step labels by the fill bar.
Machinery changes:
- add `.limit()` and `.step_sizes()` methods to the allocator to
provide the appropriate data depending on the pp limit size unit (eg.
currency vs. units)
- humanize the label display text such that you have nice suffixes and
a fixed precision
- tweak the fill bar labels to be simpler since the values are now
humanized
- expect `.on_ui_settings_change()` to be called for every slots hotkey
tweak
Turned out to be pretty simple, on every pp update just recompute
the proportion of slots used based on the limit size units.
Don't assign the allocator callback method for alert lines since
there's no size to generate. Move from-existing-pp calculations
into the order pane itself.
Handling the edge cases in this was "fun", namely:
- entering with less then a slot's worth of units to purchase
before hitting the pp limit or, less then a slots worth when exiting
toward a net-zero position.
- round pp msg updates using the symbol tick and lot size digits to
avoid super small (1e-30 lel) positions lingering in the ems (happens
moreso with the paper engine).
- don't expect the next size method to be called for alert level changes
- pass label text and field widget key separately
- fix fill status bar slot sizing logic (once and for all) and
create a new type that allows generating / resizing the bar's
size / values with a `.set_slots()` method
- pull account names from allocator attr
- set `.fill_bar` as the fill status bar on the form for now
- make `GodWidget.load_symbol()` async
- track loaded feeds with a private `._feeds` dict
- add methods to pause/resume all feeds when chart is (un)focussed
- add some commented test code for 2nd feed consumer task and rsi2 fsp
- load async signal handler for view clicking
- generate lines from staged `Order` msgs
- apply level update callback to each order that dynamically
updates the order size from the allocator calcs
- pass order msg instances to the ems client for submission
- update order size on line moves
- add `Order` msg and `Symbol` refs to each dialog
In an effort to simplify line creation and management from an order
mode here's a slew of changes:
- use our new ``LevelMarker`` for order lines and fully drop usage
of the original marker implementation stuff from `pg.InfiniteLine`
- add a left side label which shows the instrument's "units" value
- the most fundamental unit for the "size" of the order
- allow passing in an optional `marker_size: str` so that `action: str`
doesn't necessarily have to be passed (eg. when copying from an
existing line)
- change a couple of internal line config options to be public attrs
which can now be configured dynamically in real-time (since they're
all `bool` anyway):
* `hl_on_hover` -> `highlight_on_hover`
* `_always_show_labels` -> `always_show_labels`
- `LevelLine.set_level()` now only sets the position if it was **not**
called from the position changed signal (which would be redundant)
Move all the ``pydantic`` finagling to an `_orm.py` and
just keep an `Allocator` as the backing model for our pp controls
in the position module. This all needs to be tied together in some sane
with with facility for multiple symbols/streams per chart for when we
get to charting-trading aggregate feeds.
It was becoming too much with all the labels and markers and lines..
Might as well package it all together instead of cramming it in the
order mode loop, chief.
The techincal summary,
- move `_lines.position_line()` -> `PositionInfo.position_line()`.
- slap a `.pp` on the order mode instance which *is* a `PositionInfo`
- drop the position info info label for now (let's see what users want
eventually but for now let's keep it super minimal).
- add a `LevelMarker` type to replace the old `LevelLine` internal
marker system (includes ability to change the style and level on the
fly).
- change `_annotate.mk_marker()` -> `mk_maker_path()` and expect caller
to wrap in a `QGraphicsPathItem` if needed.
Add a new type/api to manage "contents labels" (labels that sit in
a view and display info about viewed data) since it's mostly used by
the linked charts cursor. Make `LinkedSplits.cursor` the new and only
instance var for the cursor such that charts can look it up from that
common class. Drop the `ChartPlotWidget._ohlc` array, just add
a `'ohlc'` entry to `._arrays`.
Orders in order mode should be chart oriented since there's a mode per
chart. If you want all orders just ask the ems or query all the charts
in a loop.
This fixes cancel-all-orders such that when 'cc' is tapped only the
orders on the *current* chart are cancelled, lel.
Generalize the methods for cancelling groups of orders (all or those
under cursor) and add new group status support such that statuses for
each cancel or order submission is displayed in the status bar. In the
"cancel-all-orders" case, use the new group status stuff.
Allows for submitting a top level "group status" associated with
a "group key" which eventually resolves once all sub-statuses associated
with that group key (and thus top level status) complete and are also
removed. Also add support for a "final message" for each status such
that once the status clear callback is called a final msg is placed on
the status bar that is then removed when the next status is set.
It's all a questionable bunch of closures/callbacks but it worx.
Instead of callbacks for key presses/releases convert our `ChartView`'s
kb input handling to async code using our event relaying-over-mem-chan
system. This is a first step toward a more async driven modal control
UX. Changed a bunch of "chart" component naming as part of this as well,
namely: `ChartSpace` -> `GodWidget` and `LinkedSplitCharts` ->
`LinkedSplits`. Engage the view boxe's async handler code as part of new
symbol data loading in `display_symbol_data()`. More re-orging to come!
Add an `open_handler()` ctx manager for wholesale handling event sets
with a passed in async func. Better document and implement the event
filtering core including adding support for key "auto repeat" filtering;
it turns out the events delivered when `trio` does its guest-most tick
are not the same (Qt has somehow consumed them or something) so we have
to do certain things (like getting the `.type()`, `.isAutoRepeat()`,
etc.) before shipping over the mem chan. The alt might be to copy the
event objects first but haven't tried it yet. For now just offer
auto-repeat filtering through a flag.
Avoids some cyclical and confusing import time stuff that we needed to get
DPI aware fonts configured from the active display. Move the main window
singleton into its own module and add a `main_window()` getter for it.
Make `current_screen()` a ``MainWindow` method to avoid so many module
variables.
This moves the entire clearing system to use typed messages using
`pydantic.BaseModel` such that the streamed request-response order
submission protocols can be explicitly viewed in terms of message
schema, flow, and sequencing. Using the explicit message formats we can
now dig into simplifying and normalizing across broker provider apis to
get the best uniformity and simplicity.
The order submission sequence is now fully async: an order request is
expected to be explicitly acked with a new message and if cancellation
is requested by the client before the ack arrives, the cancel message is
stashed and then later sent immediately on receipt of the order
submission's ack from the backend broker. Backend brokers are now
controlled using a 2-way request-response streaming dialogue which is
fully api agnostic of the clearing system's core processing; This
leverages the new bi-directional streaming apis from `tractor`. The
clearing core (emsd) was also simplified by moving the paper engine to
it's own sub-actor and making it api-symmetric with expected `brokerd`
endpoints.
A couple of the ems status messages were changed/added:
'dark_executed' -> 'dark_triggered'
added 'alert_triggered'
More cleaning of old code to come!
Makes it so we can move toward separate provider results fills in an
async way, on demand.
Also,
- add depth 1 iteration helper method
- add section finder helper method
- fix last selection loading to be mostly consistent
Some providers do well with a "longer" debounce period (like ib) since
searching them too frequently causes latency and stalls. By supporting
both a min and max debounce period on keyboard input we can only send
patterns to the slower engines when that period is triggered via
`trio.move_on_after()` and continue to relay to faster engines when the
measured period permits. Allow search routines to register their "min
period" such that they can choose to ignore patterns that arrive before
their heuristically known ideal wait.
This required a fsp task spawning logic rework which ended up being
cleaner just spawning tasks in a loop sequentially instead of trying
a 2-phase spawn-then-initialize approach.
This also includes changes from the symbol search branch hacked in.
Mostly it includes isolating the main chart startup-sequence to a
function that can be run in a new task every time a new symbol is
requested by the selector/searcher. The actual search functionality
obviously isn't in here yet but minor changes are included as part of
pulling out the `tractor` stream api patch from the symbol search dev
branch.
- break (custom) graphics item style marker drawing into separate func
but keep using it since it still seems oddly faster then the
QGraphicsPathItem thing..
- unfactor hover handler; it was uncessary
- make both the graphics path item and custom graphics items approaches
both work inside ``.paint()``
Add support for drawing ``QPathGraphicsItem`` markers but don't use them
since they seem to be shitting up when combined with the infinite line
(bounding rect?): weird artifacts and whatnot. The only way to avoid
said glitches seems to be to update inside the infinite line's
`.paint()` but that slows stuff down.. Instead stick with the manual
paint job use the same pin point: left of the L1 spread graphics - where
the lines now also extend to.
Further stuff:
- Pin the y-label to a line's value on hover.
- Disable x-dimension line moving
- Rework the labelling to be more minimal
Add a line which shows the current average price position with and arrow
marker denoting the direction (long or short). Required some further
rewriting of the infinite line from pyqtgraph including:
- adjusting marker (arrow) placement to be offset from axis + l1 labels
- fixing the hover event to not require the `.movable` attribute to be
set
This turned into a larger endeavour then intended but now we're using our
own label system on level lines to be able to display things nicely
**pinned wherever we want in the UI**. Keep the old ``LevelLabel`` for
now for the L1 graphics but we'll likely replace this as well since i'm
pretty sure the new label type (which wraps `QGraphicsTextItem`) is more
performant anyway.
For labels that want it add nice arrow paths that point just over the
respective axis. Couple label text offset from the axis line based on
parent 'tickTextOffset' setting. Drop `YSticky` it was not enough
meat to bother with.
The min tick size is the smallest step an instrument can move in value
(think the number of decimals places of precision the value can have).
We start leveraging this in a few places:
- make our internal "symbol" type expose it as part of it's api
so that it can be passed around by UI components
- in y-axis view box scaling, use it to keep the bid/ask spread (L1 UI)
always on screen even in the case where the spread has moved further
out of view then the last clearing price
- allows the EMS to determine dark order live order submission offsets
Our first major UI "mode" (yes kinda like the modes in emacs) that has
handles to a client side order book api, line and arrow editors, and
interacts with a spawned `emsd` (the EMS daemon actor).
Buncha cleaning and fixes in here for various thingers as well.
Since the "crosshair" is growing more and more UX implementation details
it probably makes sense to call it what it is; a python level mouse
abstraction. Add 2 internal sets: `_hovered` for allowing mouse hovered
objects to register themselves to other cursor aware components, and
`_trackers` for allowing scene items to "track" cursor movements via
a `on_tracked_source()` callback.
Support tracking the mouse cursor using a new `on_tracked_sources()`
callback method. Make hovered highlight a bit thicker and highlight when
click-dragged. Add a delete method for removing from the scene along
with label.
Leverages `QGraphicsItem.cacheMode` to speed up interactivity via
less `.paint()` calls (on mouse interaction) and redraws of the
underlying path when there are no transformations (other then a shift).
In order to keep the "flat bar on new time period" UX, a couple special
methods have to be triggered to get a redraw of the pixel buffer when
appending new data.
Use `QPainterPath.controlPointRect()` over `.boundingRect()` since
supposedly it's a lot faster. Drop all use of `QPicture` (since it seems
to conflict with the pixel buffer stuff?) and it doesn't give any
measurable speedup when drawing the "last bar" lines.
Oh, and add some profiling for now.
This is a bit hacky (what with array indexing semantics being relative
to the primary index's "start" value but it works. We'll likely want
to somehow wrap this index finagling into an API soon.
Failed at using either.
Quirks in numba's typing require specifying readonly arrays by
composing types manually.
The graphics item path thing, while it does take less time to write on
bar appends, seems to be slower in general in calculating the
``.boundingRect()`` value. Likely we'll just add manual max/min tracking
on array updates like ``pg.PlotCurveItem`` to squeeze some final juices
on this.
Pertains further to #109.
Instead of redrawing the entire `QPainterPath` every time there's
a historical bars update just use `.addPath()` to slap in latest
history. It seems to work and is fast. This also seems like it will be
a great strategy for filling in earlier data, woot!
This gives a massive speedup when viewing large bar sets (think a day's
worth of 5s bars) by using the `pg.functions.arrayToQPath()` "magic"
binary array writing that is also used in `PlotCurveItem`. We're using
this same (lower level) function directly to draw bars as part of one
large path and it seems to be painting 15k (ish) bars with around 3ms
`.paint()` latency. The only thing still a bit slow is the path array
generation despite doing it with `numba`. Likely, either having multiple
paths or, only regenerating the missing backing array elements should
speed this up further to avoid slight delays when incrementing the bar
step.
This is of course a first draft and more cleanups are coming.
This makes it so you don't have to ctrl-c kill apps.
Add in the experimental openGL support even though I'm pretty sure it's
not being used much for curve plotting (but could be wrong).
Tyler Goodlet
Conrad Steenberg
goodboy
wattygetlood