Commit Graph

2120 Commits (566a54ffb6d48c49854318f383006a463fbf990e)

Author SHA1 Message Date
Tyler Goodlet 2c51ad2a0d Revive `ms-shell` sub-cmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet 56fa759452 Add WIP backfiller from data feed helper 2022-05-09 11:15:13 -04:00
Tyler Goodlet 4bcc301c01 Better handle nested erros from docker client 2022-05-09 11:15:13 -04:00
Tyler Goodlet 445b82283d Add back in legacy write loop for reference 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8047714101 Add back in OHLCV dtype template and client side ws streamer 2022-05-09 11:15:13 -04:00
Tyler Goodlet 970393bb85 Drop ununsed `Services` ref 2022-05-09 11:15:13 -04:00
Tyler Goodlet ed5bae0e11 Py3.9+ type updates 2022-05-09 11:15:13 -04:00
Tyler Goodlet facc86f76e Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7395b56321 De-escalate sudo perms in `pikerd` once docker spawns 2022-05-09 11:15:13 -04:00
Tyler Goodlet aecc5973fa Handle the non-root perms case specifically too 2022-05-09 11:15:13 -04:00
Tyler Goodlet faa5a785cb Add explicit no-docker error and supervisor start task-func 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7d2e9bff46 Type annot updates 2022-05-09 11:15:13 -04:00
Tyler Goodlet ec413541d3 Drop old client instantiate line 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9203ebe044 Drop import, it's got madness with and SIGINT? 2022-05-09 11:15:13 -04:00
Tyler Goodlet fbd3d1e308 Add a super simple `marketstore` container supervisor 2022-05-09 11:15:13 -04:00
Tyler Goodlet 1cdb94374c Extract non-sudo user for config dir path 2022-05-09 11:15:13 -04:00
Tyler Goodlet aca3ca8aa6 Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 943b02573d Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 897a5cf2f6 Simplify and optimize tick format, similar to techtonicdb's 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 3c09bfba57 Add multi ingestor support and update to new feed API 2022-05-09 11:15:13 -04:00
Tyler Goodlet 34e6db6d98 Allow `None` for `BrokerdError.reqid`
Found this caused breakage on `kraken` orders which triggered the
"insufficient funds" error response. Makes sense since they won't
generate an order id if the order can't ever be submitted.
2022-05-09 10:56:47 -04:00
Tyler Goodlet 5921d18d66 Only update y-range from L1 mxmn when last index in view
We still have to always keep track of the last max and min
though.
2022-04-30 11:36:23 -04:00
Tyler Goodlet cdc882657a Drop old `pyqtgraph` downsample code 2022-04-30 11:36:23 -04:00
Tyler Goodlet 62d08eaf85 Tweak log-scaler for more detail 2022-04-30 11:36:23 -04:00
Tyler Goodlet f2f00dcc52 Drop `._ic` debugging prints 2022-04-30 11:36:23 -04:00
Tyler Goodlet ee831baeb3 Display loop mega-cleanup
The most important changes include:
- iterating the new `Flow` type and updating graphics
- adding detailed profiling
- increasing the min uppx before graphics updates are throttled
- including the L1 spread in y-range calcs so that you never have the
  bid/ask go "out of view"..
- pass around `Flow`s instead of shms
- drop all the old prototyped downsampling code
2022-04-30 11:36:23 -04:00
Tyler Goodlet 7c615a403b Allow passing a `plotItem` to `.draw_curve()`
If manually managing an overlay you'll likely call `.overlay_plotitem()`
and then a plotting method so we need to accept a plot item input so
that the chart's pi doesn't get assigned incorrectly in the `Flow` entry
(though it is by default if no input is provided).

More,
- add a `Flow.graphics` field and set it to the `pg.GraphicsObject`.
- make `Flow.maxmin()` return `None` in the "can't calculate" cases.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b8374dbe9a Fsp UI initialization updates
- set shm refs on `Flow` entries.
- don't run a graphics cycle on 'update' msgs from the engine
  if the containing chart is hidden.
- drop `volume` from flows map and disable auto-yranging
  once $vlm comes up.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 454cd7920d Disconnect signals in `ChartView.disable_auto_yrange()`
Allows for removing resize callbacks for a flow/overlay that you wish to
remove from view (eg. unit volume after dollar volume is up) and thus
less general interaction callback overhead for any plot you don't wish
to show or resize.

Further,
- drop the `autoscale_linked_plots` block for now since with
  multi-view-box overlays each register their own vb resize slots
- pull the graphics object from the chart's `Flow` map inside
  `.maybe_downsample_graphics()`
2022-04-30 11:36:23 -04:00
Tyler Goodlet ca283660de Fix bug where if `yrange` was passed the mxmin callback was still used.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet d4eddbdb25 Guard against zero px width 2022-04-30 11:36:23 -04:00
Tyler Goodlet eec329a221 Add `Flow` type with a real chitty mxmn cacheing method
This new type wraps a shm data flow and will eventually include things
like incremental path-graphics updates and serialization + bg downsampling
techniques. The main immediate motivation was to get a cached y-range max/min
calc going since profiling revealed the `numpy` equivalents were
actually quite slow as the data set grows large. Likely we can use all
this to drive a streaming mx/mn routine that's always launched as part
of each on-host flow.

This is our official foray into use of `msgspec.Struct` B) and I have to
say, pretty impressed; we'll likely completely ditch `pydantic` from
here on out.
2022-04-30 11:36:23 -04:00
Tyler Goodlet a1de097a43 Loop for first graphic with xvec 2022-04-30 11:36:23 -04:00
Tyler Goodlet b5f2558cec Only `.maybe_downsample_graphics()` on manual changes
We don't need update graphics on every x-range change since that's what
the display loop does. Instead, only on manual changes do we make manual
calls into `.update_graphics_from_array()` and be sure to iterate all
linked subplots and all their embedded graphics.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 1a95712680 Don't return early on ds line render to avoid breaking profiling
The pg profiler seems to have trouble with early `return`s in function
calls (likely muckery with the GC/`.__delete__()`) so let's just try
to avoid it for now until we either fix it (probably by implementing as
a ctx mngr) or use diff one.
2022-04-30 11:36:23 -04:00
Tyler Goodlet b20e9e58ee Use HL tracer by default, seems to be faster? 2022-04-30 11:36:23 -04:00
Tyler Goodlet 4bc2bbda69 Allow passing "ms slower then" value on cli to `--profile` 2022-04-30 11:36:23 -04:00
Tyler Goodlet b524929cb6 Only bail up pan updates if uppx > 16 2022-04-30 11:36:23 -04:00
Tyler Goodlet f95d22bfd3 Delegate `BarItems.x_uppx()` to internal ds curve 2022-04-30 11:36:23 -04:00
Tyler Goodlet 91de281b7e Downsample curves even less frequently 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2284e61eda Only pass vr for bars, allow source vb in autorange 2022-04-30 11:36:23 -04:00
Tyler Goodlet 082b02776c Drop the unit-volume chart once $vlm is fully drawn 2022-04-30 11:36:23 -04:00
Tyler Goodlet 27e3d0ef80 Ensure we update the volume array, not graphics
Ugh, turns out the wacky `ChartView.maxmin` callback stuff we did (for
determining y-range sizings) currently requires that the volume array
has a "bars in view" result.. so let's make that keep working without
rendering the graphics for the curve (since we're disabling them once
$vlm comes up).
2022-04-30 11:36:23 -04:00
Tyler Goodlet eeca9eb4c7 Add `.update_graphics_from_array()` flags for setting view-range use and graphics rendering 2022-04-30 11:36:23 -04:00
Tyler Goodlet 9bbfa4be02 Guard against zero px width 2022-04-30 11:36:23 -04:00
Tyler Goodlet ce85031ef2 Given in-view rendering, make bars downsample on uppx >= 8 2022-04-30 11:36:23 -04:00
Tyler Goodlet b6f852e0ad Make `FastAppendCurve` optionally view range aware
As with the `BarItems` graphics, this makes it possible to pass in a "in
view" range of array data that can be *only* rendered improving
performance for large(r) data sets. All the other normal behaviour is
kept (i.e a persistent, (pre/ap)pendable path can still be maintained)
if a ``view_range`` is not provided.

Further updates,
- drop the `.should_ds_or_redraw()` and `.maybe_downsample()` predicates
 instead moving all that logic inside `.update_from_array()`.
- disable the "cache flipping", which doesn't seem to be needed to avoid
  artifacts any more?
- handle all redraw/dowsampling logic in `.update_from_array()`.
- even more profiling.
- drop path `.reserve()` stuff until we better figure out how it's
  supposed to work.
2022-04-30 11:36:23 -04:00
Tyler Goodlet fdd5aa33d2 Fix view range array to include most recent (facepalm) 2022-04-30 11:36:23 -04:00
Tyler Goodlet 82732e3f17 TOQUASH: drop display loop old .update_ohlc_.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2c1daab990 Port to new `.update_graphics_from_array()`, pause quote updates on chart interaction 2022-04-30 11:36:23 -04:00
Tyler Goodlet a9e1c6c50e Make panning pause feeds, call into update method from downsampler cb loop 2022-04-30 11:36:23 -04:00
Tyler Goodlet ef03b8e987 Attempt only rendering ohlc bars in view and ds-ing otherwise 2022-04-30 11:36:23 -04:00
Tyler Goodlet 3b90b1f960 Unify into a single update method: `.update_graphics_from_array()` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1cf6ba789c Remove units vlm cuve once the $vlm one comes up 2022-04-30 11:36:23 -04:00
Tyler Goodlet 49c25eeef4 Index must be int bro.. 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5bcd6ac494 Move px width log scaling into `ds_m4()` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5da9f7fdb4 Add more frequent ds steps when zooming out; use profiler gt 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5128e4c304 Make `BarItems` use our line curve for downsampling
Drop all the logic originally in `.update_ds_line()` which is now done
internal to our `FastAppendCurve`. Add incremental update of the
flattened OHLC -> line curve (unfortunately using `np.concatenate()` for
the moment) and maintain a new `._ds_line_xy` arrays tuple which keeps
the internal state. Add `.maybe_downsample()` as per the new interaction
update method requirement. Draft out some fast path curve stuff like in
our line graphic. Short-circuit bars path updates when we downsample to
line. Oh, and add a ton more profiling in prep for getting
all this stuff faf.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 6410c68e2e Add global profile timeout var 2022-04-30 11:36:23 -04:00
Tyler Goodlet 947a514153 Add "native" downsampling to our `FastAppendCurve`
Build out an interface that makes it super easy to downsample curves
using the m4 algorithm while keeping our incremental `QPainterPath`
update feature. A lot of hard work and tinkering went into getting this
working all in-thread correctly and there are quite a few details..

New interface methods:
- `.x_uppx()` which returns the x-axis "view units per pixel"
- `.px_width()` which returns the total (rounded) x-axis pixels spanned
    by the curve in view.
- `.should_ds_or_redraw()` a predicate which checks internal state to
  see if either downsampling of the curve should take place, or the curve
  should have all downsampling removed and be redrawn with source array
  data.
- `.downsample()` the actual ds processing routine which delegates into
  the m4 algo impl.
- `.maybe_downsample()` a simple update method which can be called by
  the view box when the user changes the zoom level.

Implementation details/changes:

- make `.update_from_array()` check for downsample (or revert to source
  aka de-downsample) conditions exist and then downsample and re-draw
  path graphics accordingly.
- in order to even further speed up path appends (since our main
  bottleneck is measured to be `QPainter.drawPath()` calls with large
  paths which are frequently updates), add a secondary path `.fast_path`
  which is the path that is real-time updates by incremental appends and
  which is painted separately for speed in `.pain()`.
- drop all the `QPolyLine` stuff since it was tested to be much slower
  in general and especially so for append-updates.
- stop disabling the cache settings on updates since it doesn't seem to
  be required any more?
- more move toward deprecating and removing all lingering interface
  requirements from `pg.PlotCurveItem` (like `.xData`/`.yData`).
- adjust `.paint()` and `.boundingRect()` to compensate for the new
  `.fast_path`
- add a butt-load of profiling B)
2022-04-30 11:36:23 -04:00
Tyler Goodlet 8627f6f6c5 Add no-path guard now that we can use a poly 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5800c10901 First try, drop `FastAppendCurve` inheritance from `pg.PlotCurveItem` 2022-04-30 11:36:23 -04:00
Tyler Goodlet 28bf8853aa Drop commented line from pq method copy/paste 2022-04-30 11:36:23 -04:00
Tyler Goodlet 86da64c2c2 Show baseline bars length on in view read < 6 2022-04-30 11:36:23 -04:00
Tyler Goodlet d59442e3b1 Bump up resolution log scaling a mag 2022-04-30 11:36:23 -04:00
Tyler Goodlet 5e161aa251 Always clear previous downsample curve on switch
Pretty sure this was most of the cause of the stale (more downsampled)
curves showing when zooming in and out from bars mode quickly. All this
stuff needs to get factored out into a new abstraction anyway, but
i think this get's mostly correct functionality.

Only draw new ds curve on uppx steps >= 4 and stop adding/removing
graphics objects from the scene; doesn't seem to speed anything up
afaict. Add better reporting of ds scale changes.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 9b2ec871a0 Clear ds line graphics on switch back to bars 2022-04-30 11:36:23 -04:00
Tyler Goodlet 2b12742992 More ems resiliency: discard broken client dialogs 2022-04-30 11:36:23 -04:00
Tyler Goodlet b262532fd4 Allocate m4 output arrays in `numba` code, avoid segfaults? 2022-04-30 11:36:23 -04:00
Tyler Goodlet 561d7e0349 Only clear/redraw curve on uppx diffs > 2
Only if the uppx increases by more then 2 we redraw the entire line
otherwise just ds with previous params and update the current curve.
This *should* avoid strange lower sample rate artefacts from showing on
updates.

Summary:
- stash both uppx and px width in `._dsi` (downsample info)
- use the new `ohlc_to_m4_line()` flags
- add notes about using `.reserve()` and friends
- always delete last `._array` ref prior to line updates
2022-04-30 11:36:23 -04:00
Tyler Goodlet 3a6c5a2fbd Try supporting reuse of path allocation 2022-04-30 11:36:23 -04:00
Tyler Goodlet 88a7314bd0 Add optional mxmn HL tracer support to m4 sampler 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1abe513ecb Add our own `FastAppendCurve.clear()`, try mem reso
In an effort to try and make `QPainterPath.reserve()` work, add internal
logic to use the same object without de-allocating memory from
a previous path write/creation.

Note this required the addition of a `._redraw` flag (to be used in
`.clear()` and a small patch to `pyqtgraph.functions.arrayToQPath` to
allow passing in an existing path (thus reusing the same underlying mem
alloc) which will likely be first pushed to our fork.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 44f3a08ef1 Add optional uppx log scaling to m4 sampler
We were previously ad-hoc scaling up the px count/width to get more
detail at lower uppx values. Add a log scaling sigmoid that range scales
between 1 < px_width < 16.

Add in a flag to use the mxmn OH tracer in `ohlc_flatten()` if desired.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 03e0e3e76b Delegate to m4 ohlc helper for curve, only ds on uppx steps > 2 2022-04-30 11:36:23 -04:00
Tyler Goodlet 08f90c275c Add OHLC to m4 line converters
Helpers to quickly convert ohlc struct-array sequences into lines
for consumption by the m4 downsampler. Strip trailing zero entries
from the `ds_m4()` output if found (avoids lines back to origin).
2022-04-30 11:36:23 -04:00
Tyler Goodlet 7edfe68d4d M4 workin bishhhhh 2022-04-30 11:36:23 -04:00
Tyler Goodlet ff00993412 Call default view on symbol switch 2022-04-30 11:36:23 -04:00
Tyler Goodlet ed03d77e6e Make a derivs intrustment type table for alloc config checks 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1a0e89d07e Even more correct "default view" snap-to-pp-marker
This makes the `'r'` hotkey snap the last bar to the middle of the pp
line arrow marker no matter the zoom level. Now we also boot with
approximately the most number of x units on screen that keep the bars
graphics drawn in full (just before downsampling to a line).

Moved some internals around to get this all in place,
- drop `_anchors.marker_right_points()` and move it to a chart method.
- change `.pre_l1_x()` -> `.pre_l1_xs()` and just have it return the
  two view-mapped x values from the former method.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 56c163cdd7 Make `ChartPlotWidget.default_view()` pin to L1
Instead of using a guess about how many x-indexes to reset the last
datum in-view to, calculate and shift the latest index such that it's
just before any L1 spread labels on the y-axis. This makes the view
placement "widget aware" and gives a much more cross-display UX.

Summary:
- add `ChartPlotWidget.pre_l1_x()` which returns a `tuple` of
  x view-coord points for the absolute x-pos and length of any L1
  line/labels
- make `.default_view()` only shift to see the xlast just outside
  the l1 but keep whatever view range xfirst as the first datum in view
- drop `LevelLine.right_point()` since this is now just a
  `.pre_l1_x()` call and can be retrieved from the line's internal chart
  ref
- drop `._style.bars_from/to_..` vars since we aren't using hard coded
  offsets any more
2022-04-30 11:36:23 -04:00
Tyler Goodlet c4242acc21 Pass in fqsn from chart UI components 2022-04-30 11:36:23 -04:00
Tyler Goodlet 772f871272 Use units by default for continuous futes 2022-04-30 11:36:23 -04:00
Tyler Goodlet 1ad83e4556 WIP add non-working m4 ds code to ohlc graphic 2022-04-30 11:36:23 -04:00
Tyler Goodlet bedb55b79d Use service cancel method for graceful teardown 2022-04-30 11:36:23 -04:00
Tyler Goodlet 03a08b5f63 Add curve px width getter
`ChartPlotWidget.curve_width_pxs()` now can be used to get the total
horizontal (x) pixels on screen that are occupied by the current curve
graphics for a given chart. This will be used for downsampling large
data sets to the pixel domain using M4.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 8f26335aea Add display loop profiling
Probably the best place to root the profiler since we can get a better
top down view of bottlenecks in the graphics stack.

More,
- add in draft M4 downsampling code (commented) after getting it mostly
  working; next step is to move this processing into an FSP subactor.
- always update the vlm chart last y-axis sticky
- set call `.default_view()` just before inf sleep on startup
2022-04-30 11:36:23 -04:00
Tyler Goodlet f1f257d4a2 Profiler format, code stretch 2022-04-30 11:36:23 -04:00
Tyler Goodlet d02b1a17ad Fix x-range -> # of frames calculation
Obviously determining the x-range from indices was wrong and was the
reason for the incorrect (downsampled) output size XD. Instead correctly
determine the x range and start value from the *values of* the input
x-array. Pretty sure this makes the implementation nearly production
ready.

Relates to #109
2022-04-30 11:36:23 -04:00
Tyler Goodlet 4d4f745918 Add `ChartPlotWidget.in_view()` shm-compatible array slicer 2022-04-30 11:36:23 -04:00
Tyler Goodlet 39b7c9340d Add (ostensibly) working first attempt at M4 algo
All the refs are in the comments and original sample code from infinite
has been reworked to expect the input x/y arrays to already be sliced
(though we can later support passing in the start-end indexes if
desired).

The new routines are `ds_m4()` the python top level API and `_m4()` the
fast `numba` implementation.
2022-04-30 11:36:23 -04:00
Tyler Goodlet e7481b1634 Array diff lengths must be int 2022-04-30 11:36:23 -04:00
Tyler Goodlet 09d95157dc Limit real-time chart updates in "big data" cases
- the chart's uppx (units-per-pixel) is > 4 (i.e. zoomed out a lot)
- don't shift the chart (to keep the most recent step in view) if the
  last datum isn't in view (aka the user is probably looking at history)
2022-04-30 11:36:23 -04:00
Tyler Goodlet ea5b8f1dd0 Only trigger downsampling on manual changes, add a uppx method 2022-04-30 11:36:23 -04:00
Tyler Goodlet 7e49b7c033 Add for a `BarItems` to display a line on high uppx
When a bars graphic is zoomed out enough you get a high uppx, datum
units-per-pixel, and there is no point in drawing the 6-lines in each
bar element-graphic if you can't see them on the screen/display device.

Instead here we offer converting to a `FastAppendCurve` which traces
the high-low outline and instead display that when it's impossible to see the
details of bars - approximately when the uppx >= 2.

There is also some draft-commented code in here for downsampling the
outlines as zoom level increases but it's not fully working and should
likely be factored out into a higher level api anyway.
2022-04-30 11:36:23 -04:00
Tyler Goodlet e7dc1a036b Original index offset was right 2022-04-30 11:36:23 -04:00
Tyler Goodlet ab8ea41b93 Add an ohlcv high/low tracer with optional downsampling 2022-04-30 11:36:23 -04:00
Tyler Goodlet dbe55ad4d2 Pass linked charts into `BarItems` so that graphics can be cycled on downsample 2022-04-30 11:36:23 -04:00
Tyler Goodlet d7a9928293 Move graphics compression routines to new module 2022-04-30 11:36:23 -04:00
Tyler Goodlet 02300efb59 Use 12Hz as default fps throttle 2022-04-30 11:36:23 -04:00
Tyler Goodlet 7c4e55ed2c Add comment on how to enable `pyqtgraph` profiling 2022-04-30 11:36:23 -04:00
Tyler Goodlet 7811508307 Add basic optional polyline support, draft out downsampling routine 2022-04-30 11:36:23 -04:00
Tyler Goodlet 7e853fe345 Add a downsampled line-curve support to `BarItems`
In effort to start getting some graphics speedups as detailed in #109,
this adds a `FastAppendCurve`to every `BarItems` as a `._ds_line` which
is only displayed (instead of the normal mult-line bars curve) when the
"width" of a bar is indistinguishable on screen from a line -> so once
the view coordinates map to > 2 pixels on the display device.
`BarItems.maybe_paint_line()` takes care of this scaling detection logic and is
called by the associated view's `.sigXRangeChanged` signal handler.
2022-04-30 11:36:23 -04:00
Tyler Goodlet 11f8c4f350 Add detailed `.addItem()`` comment 2022-04-30 11:36:23 -04:00
Tyler Goodlet 7577443f95 Add guard for real-time-not-active last line is `None` case 2022-04-30 11:36:23 -04:00
goodboy 67cec4bc54
Merge pull request #304 from pikers/offline_history_loading
Offline history loading
2022-04-16 15:57:14 -04:00
Konstantine Tsafatinos 773ed5e7ad update to merge syntax in submit_limit, fix non_master push mistake 2022-04-16 15:01:31 -04:00
Konstantine Tsafatinos 59434b9a8a refactor submit
_limit and expore the 'paper' like feature
2022-04-16 14:54:25 -04:00
Konstantine Tsafatinos 250d9cbc03 fix kraken bug, allow for live order edits 2022-04-16 14:38:03 -04:00
Tyler Goodlet bcb4fe8c50 Indefinitely wait on feed hack for windows? 2022-04-16 13:25:14 -04:00
Tyler Goodlet d8db9233c9 Establish stream before `fsp_compute` so that backfill updates work again.. 2022-04-16 13:25:14 -04:00
Tyler Goodlet 82f2fa2d37 Pass in fqsn from chart UI components 2022-04-16 13:25:14 -04:00
Tyler Goodlet 8195fae289 Add a `trigger_all` arg to update cycle func; allows hard history updates 2022-04-16 13:25:14 -04:00
Tyler Goodlet 30656eda39 Use a `DisplayState` in the graphics update loop
The graphics update loop is much easier to grok when all the UI
components which potentially need to be updated on a cycle are arranged
together in a high-level composite namespace, thus this new
`DisplayState` addition. Create and set this state on each
`LinkedSplits` chart set and add a new method `.graphics_cycle()` which
let's a caller trigger a graphics loop update manually. Use this method
in the fsp graphics manager such that a chain can update new history
output even if there is no real-time feed driving the display loop (eg.
when a market is "closed").
2022-04-16 13:25:14 -04:00
Tyler Goodlet 2564acea1b Facepalm**2: only update on special "update" msg 2022-04-16 13:25:14 -04:00
Tyler Goodlet b3efa2874b Facepalm: display state must be linked charts specific 2022-04-16 13:25:14 -04:00
Tyler Goodlet ad1bbe74ad Manually trigger graphics loops updates on msgs from the fsp chain 2022-04-16 13:25:14 -04:00
Tyler Goodlet 761b823939 Always fire a "step/update message" on every fsp history update 2022-04-16 13:25:14 -04:00
Tyler Goodlet b75a3310fe Factor sync part of graphics update into func, add `trigger_update()`` 2022-04-16 13:25:14 -04:00
Tyler Goodlet 72ec34ffd2 Port to `pendulum` equivalent apis throughout 2022-04-16 13:23:42 -04:00
Tyler Goodlet d334e61b1f Drop 22s timeout on reset hack 2022-04-16 13:23:38 -04:00
Tyler Goodlet 4d23f6e4d7 Drop need for `ib_insync.IB.qualifyContractsAsync()' mod
As per https://github.com/erdewit/ib_insync/pull/454 the more correct
way to do this is with `.reqContractDetailsAsync()` which we wrap with
`Client.con_deats()` and which works just as well. Further drop all the
`dict`-ifying that was being done in that method and instead always
return `ContractDetails` object in an fqsn-like explicitly keyed `dict`.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 8b1c521ae9 Ignore symbol-not-found errors 2022-04-13 00:39:15 -04:00
Tyler Goodlet 7586e20ab4 Use new unpacker helper name 2022-04-13 00:39:15 -04:00
Tyler Goodlet 80d70216f7 Drop back down ohlc bars request count to not trigger feed hack 2022-04-13 00:39:15 -04:00
Tyler Goodlet d1f45b0883 Add `ShmArray.last()` docstr 2022-04-13 00:39:15 -04:00
Tyler Goodlet 00a7f20292 Up the shm size to 10d of 1s ohlc 2022-04-13 00:39:15 -04:00
Tyler Goodlet 0178fcd26f Increase shm size to days of 1s steps 2022-04-13 00:39:15 -04:00
Tyler Goodlet 24fa1b8ff7 Support an array field map to `ShmArray.push()`, start index 3days in 2022-04-13 00:39:15 -04:00
Tyler Goodlet 66ea74c6d5 Put back more bars iters in loop to handle no-data in range cases 2022-04-13 00:39:15 -04:00
Tyler Goodlet b579d4b1f5 Get ib data feed hackzorz workin
ib has a throttle limit for "hft" bars but contained in here is some
hackery using ``xdotool`` to reset data farms auto-magically B)

This copies the working script into the ib backend mod as a routine and
now uses `trio.run_process()` and calls into it from the `get_bars()`
history retriever and then waits for "data re-established" events to be
received from the client before making more history queries.

TL;DR summary of changes:
- relay ib's "system status" events (like for data farm statuses)
  as a new "event" msg that can be processed by registers of
  `Client.inline_errors()` (though we should probably make a new
  method for this).
- add `MethodProxy.status_event()` which allows a proxy user to register
  for a particular "system event" (as mentioned above), which puts
  a `trio.Event` entry in a small table can be set by an relay task if
  there are any detected waiters.
- start a "msg relay task" when opening the method proxy which does
  the event setting mentioned above in the background.
- drop the request error handling around the proxy creation, doesn't
  seem necessary any more now that we have better error propagation from
  `asyncio`.
- add event waiting logic around the data feed reset hackzorin.
- change the order relay task to only log system events for now (though
  we need to do some better parsing/logic to get tws-external order
  updates to work again..
2022-04-13 00:39:15 -04:00
Tyler Goodlet 874374af06 Drop `pandas` use in ib backend for history
Found an issue (that was predictably brushed aside XD) where the
`ib_insync.util.df()` helper was changing the timestamps on bars data to
be way off (probably a `pandas.Timestamp` timezone thing?).

Anyway, dropped all that (which will hopefully let us drop `pandas` as
a hard dep) and added a buncha timestamp checking as well as start/end
datetime return values using `pendulum` so that consumer code can know
which "slice" is output.

Also added some WIP code to work around "no history found" request
errors where instead now we try to increment backward another 200
seconds - not sure if this actually correct yet.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 62d073dc18 More IB repairs..
Make the throttle error propagate through to `trio` again by adding
`dict`-msg support between the two loops such that errors can be
re-raised on the `trio` side. This is all integrated into the
`MethoProxy` and accompanying result relay task.

Further fix a longer standing issue where sometimes the `ib_insync`
order entry method will raise a weird assertion error because it detects
some internal order-id state issue.. Just ignore those and make relay
back an error to the ems in such cases.

Add a bunch of notes for todos surrounding data feed reset hackery.
2022-04-13 00:39:15 -04:00
Tyler Goodlet 3e125625b1 Attempt to better handle history throttles using flag 2022-04-13 00:39:15 -04:00
Tyler Goodlet 8395a1fcfe IB: Comment on lowercase for the fqsn key 2022-04-13 00:39:15 -04:00
Tyler Goodlet 957686a9fe Comment exception debug in ib request error block 2022-04-13 00:39:15 -04:00
Tyler Goodlet 1e433ca4f4 Support "expiry" suffixes for derivatives with ib
To start we only have futes working but this allows both searching
and loading multiple expiries of the same instrument by specifying
different expiries with a `.<expiry>` suffix in the symbol key (eg.
`mnq.globex.20220617`). This also paves the way for options contracts
which will need something similar plus a strike property. This change
set also required a patch to `ib_insync` to allow retrieving multiple
"ambiguous" contracts from the `IB.reqContractDetailsAcync()` method,
see https://github.com/erdewit/ib_insync/pull/454 for further discussion
since the approach here might change.

This patch also includes a lot of serious reworking of some `trio`-`asyncio`
integration to use the newer `tractor.to_asyncio.open_channel_from()`
api and use it (with a relay task) to open a persistent connection with
an in-actor `ib_insync` `Client` mostly for history requests.

Deats,
- annot the module with a `_infect_asyncio: bool` for `tractor` spawning
- add a futes venu list
- support ambiguous futes contracts lookups so that all expiries will
  show in search
- support both continuous and specific expiry fute contract
  qualification
- allow searching with "fqsn" keys
- don't crash on "data not found" errors in history requests
- move all quotes msg "topic-key" generation (which should now be
  a broker-specific fqsn) and per-contract quote processing into
  `normalize()`
- set the fqsn key in the symbol info init msg
- use `open_client_proxy()` in bars backfiller endpoint
- include expiry suffix in position update keys
2022-04-13 00:39:15 -04:00
Tyler Goodlet 937406534c Maybe spawn `brokerd` in `asyncio` mode if declared in backend mod 2022-04-13 00:39:15 -04:00
Tyler Goodlet b26b66cc66 Add context-styled `asyncio` client proxy for ib
This adds a new client manager-factory: `open_client_proxy()` which uses
the newer `tractor.to_asyncio.open_channel_from()` (and thus the
inter-loop-task-channel style) a `aio_client_method_relay()` and
a re-implemented `MethodProxy` wrapper to allow transparently calling
`asyncio` client methods from `trio` tasks. Use this proxy in the
history backfiller task and add a new (prototype)
`open_history_client()` which will be used in the new storage management
layer. Drop `get_client()` which was the portal wrapping equivalent of
the same proxy but with a one-task-per-call approach. Oh, and
`Client.bars()` can take `datetime`, so let's use it B)
2022-04-13 00:39:15 -04:00
Tyler Goodlet 7936dcafbf Make linux timeout the same 2022-04-13 00:39:15 -04:00
Tyler Goodlet d32c26c5d7 Add flag to avoid logging json to console 2022-04-13 00:39:15 -04:00
Tyler Goodlet d2d3286fb8 Use `asyncio` in `Client.get_quote()` 2022-04-13 00:39:15 -04:00
Tyler Goodlet 4b0ca40b17 Document "fqsn" on `Symbol` method 2022-04-11 08:48:17 -04:00
Tyler Goodlet ebe2680355 Change `uncons_fqsn()` -> `unpack_fqsn()` 2022-04-11 01:01:36 -04:00
Tyler Goodlet e92632bd34 Remove old commented nan checking lines 2022-04-10 21:51:22 -04:00
Tyler Goodlet 32e316ebff Drop nl 2022-04-10 17:33:02 -04:00
Tyler Goodlet f604437897 Remove symbol key from first quote from ib feed 2022-04-10 17:33:02 -04:00
Tyler Goodlet c9e6c81459 Expect fqsn input to paper clearing engine 2022-04-10 17:33:02 -04:00
Tyler Goodlet ce7d630676 Pass in fqsn from fsp admin apis 2022-04-10 17:33:02 -04:00
Tyler Goodlet 6ac60fbe22 Expect fqsns through fsp machinery 2022-04-10 17:33:02 -04:00
Tyler Goodlet 998a5acd92 Crypto$ backend updates
- move to 3.9+ type annots
- add initial draft `open_history_client()` endpoints
- deliver `'fqsn'` keys in quote-stream init msgs
2022-04-10 17:33:00 -04:00
Tyler Goodlet 493e45e70a Strip broker name from symbol on pp msg updates 2022-04-10 17:30:02 -04:00
Tyler Goodlet c7f3e59105 Expect fqsn in ems and order mode
Use fqsn as input to the client-side EMS apis but strip broker-name
stuff before generating and sending `Brokerd*` msgs to each backend for
live order requests (since it's weird for a backend to expect it's own
name, though maybe that could be a sanity check?).

Summary of fqsn use vs. broker native keys:
- client side pps, order requests and general UX for order management
  use an fqsn for tracking
- brokerd side order dialogs use the broker-specific symbol which is
  usually nearly the same key minus the broker name
- internal dark book and quote feed lookups use the fqsn where possible
2022-04-10 17:30:02 -04:00
Tyler Goodlet d62a636bcc Pass concatted pre-fqsn directly to feed api 2022-04-10 17:30:02 -04:00
Tyler Goodlet d0205e726b Pass in fqsn from chart UI components 2022-04-10 17:30:02 -04:00
Tyler Goodlet 8df614465c Fix missing f-str prefix 2022-04-10 17:30:02 -04:00
Tyler Goodlet 81cd696ec8 Drop sampler consumers that overrun 6x 2022-04-10 17:30:02 -04:00
Tyler Goodlet a6e32e7530 Add `Symbol.tokens()` for grabbing separate strs 2022-04-10 17:30:02 -04:00
Tyler Goodlet 7bd5b42f9e Ensure we lower case the fqsn received from all backends before delivery 2022-04-10 17:30:02 -04:00
Tyler Goodlet 76f398bd9f Support no venue or suffix symbols (normally crypto$) 2022-04-10 17:30:02 -04:00
Tyler Goodlet 7f36e85815 Append broker name to symbols before quotes broadcast in sampler task 2022-04-10 17:30:02 -04:00
Tyler Goodlet 8462ea8a28 Make the data feed layer "fqsn" aware
In order to support instruments with lifetimes (aka derivatives) we need
generally need special symbol annotations which detail such meta data
(such as `MNQ.GLOBEX.20220717` for daq futes). Further there is really
no reason for the public api for this feed layer to care about getting
a special "brokername" field since generally the data is coming directly
from UIs (eg. search selection) so we might as well accept a fqsn (fully
qualified symbol name) which includes the broker name; for now a suffix
like `'.ib'`. We may change this schema (soon) but this at least gets us
to a point where we expect the full name including broker/provider.

An additional detail: for certain "generic" symbol names (like for
futes) we will pull a so called "front contract" and map this to
a specific fqsn underneath, so there is a double (cached) entry for that
entry such that other consumers can use it the same way if desired.

Some other machinery changes:
- expect the `stream_quotes()` endpoint to deliver it's `.started()` msg
  almost immediately since we now need it deliver any fqsn asap (yes
  this means the ep should no longer wait on a "live" first quote and
  instead deliver what quote data it can right away.
- expect the quotes ohlc sampler task to add in the broker name before
  broadcast to remote (actor) consumers since the backend isn't (yet)
  expected to do that add in itself.
- obviously we start using all the new fqsn related `Symbol` apis
2022-04-10 17:30:02 -04:00
Tyler Goodlet e9d64ffee8 Use fqsn in `.manage_history()`
Allocate and `.started()` return the `ShmArray` from here as well in
prep for tsdb integration.
2022-04-10 17:30:02 -04:00
Tyler Goodlet b16167b8f3 Add prelim fqsn support into our `Symbol` type 2022-04-10 17:30:02 -04:00
Tyler Goodlet 434c340cb8 Move factor helper to a classmethod 2022-04-10 17:30:02 -04:00
Tyler Goodlet 94e2103bf5 Be mega-tolerant to feed consumer disconnects 2022-04-10 17:30:02 -04:00
Tyler Goodlet cc026dfb1d Open feeds using `Portal.open_context()` 2022-04-10 17:30:02 -04:00
Tyler Goodlet 97c2a2da3e Convert `iter_ohlc_periods()` to a `@tractor.context` 2022-04-10 17:30:02 -04:00
Tyler Goodlet 58517295d2 Disable kraken orders due to #299 2022-04-10 17:27:15 -04:00
Tyler Goodlet c39fa825d0 More explicit order-cancel errors handling 2022-04-10 17:07:08 -04:00
Tyler Goodlet 88306a6c1e Drop invalid status msg, linting cleanups 2022-04-09 16:56:05 -04:00
Tyler Goodlet c034ea742f Fix comment: filled not executed is a valid status key 2022-04-09 16:46:25 -04:00
Konstantine Tsafatinos cb970cef46 dark order gui patch, add filled status message 2022-04-08 19:25:24 -04:00
Konstantine Tsafatinos c2e654aae2 change logic order for handling no config case 2022-04-07 13:03:53 -04:00
Konstantine Tsafatinos 2baa1b4605 fix hang when kraken is not in config 2022-03-28 18:28:19 -04:00
Konstantine Tsafatinos cb8e97a142 address latest comments, refactor the pack position function 2022-03-23 10:34:53 -04:00
Konstantine Tsafatinos 1525c645ce refactor get_positions into get_trades, and refactor pack_position with postion calc logic 2022-03-20 13:52:45 -04:00
Konstantine Tsafatinos fd0acd21fb refactory based on github comments, change doc string style 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 617bf3e0da fix typo and get rid of pprint of ws stream 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos a3345dbba2 cleaned up code and added loop to grab all trades for position calcs 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos ee0be13af1 repurpose ws code for ownTrades stream, get trade authentication going 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos b1bff1be85 remove ws support for orders, use rest api instead for easy oid association 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 46948e0a8b add order cancel support over websockets 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos d826a66c8c use a mapping from userref to oid for order ack 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 6c54c81f01 get stashed changes 2022-03-06 15:17:26 -05:00
Tyler Goodlet 0122669dd4 Factor out ws msg hearbeat and error handling
Move the core ws message handling into `stream_messages()` and call that
from 2 new stream processors: `process_data_feed_msgs()` and
`process_order_msgs()`. Add comments for hints on how to implement the
order msg parsing as well as `pprint` received msgs to console for now.
2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 0c905920e2 connect to krakens openOrders websocket 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 03d2eddce3 order submission and cancellation working 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 96dd5c632f basic order submission and cancelling with kraken 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos b21bbf5031 valdiate and ack order requests from ems 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 66da58525d mock orders validated from kraken 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos b55debbe95 get basic order request loop receiving msgs 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 1fe1f88806 added the bones for the handle_order_requests func 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 3d2be3674e save progress on kraken to test out unit_select_fixes 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 48c7b5262c get positions working for kraken 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos ef598444c4 get positions from trades 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 0285a847d8 Store changes for rebase, positions prototype 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 88061d8799 Add balance to the ledger 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos e12af8aa4c Add get_ledger function; parses raw ledger from kraken api 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos 184edb2a90 wrap api method calls with uri and nonce value 2022-03-06 15:17:26 -05:00
Konstantine Tsafatinos b88dd380a3 get kraken authentication and retrieve balances 2022-03-06 15:17:26 -05:00
Tyler Goodlet 01f5f2d015 Don't require a rt quote, increase client connect timeout 2022-03-03 17:49:21 -05:00
Tyler Goodlet af3d624281 Just give up on discretized pp bar for now 2022-03-03 17:15:55 -05:00
Tyler Goodlet 2c9612ebd8 Force exact pp bar size 2022-03-03 10:46:30 -05:00
Tyler Goodlet 16b9e39e11 Dis-allow an allocator limit less then the current pp size 2022-03-02 10:05:33 -05:00
Tyler Goodlet 6889a25926 Drop pp bar clipping, hopefully fix slot sizing 2022-03-02 10:05:33 -05:00
Tyler Goodlet f7d03489d8 Drop `marketstore` loading cruft (will come later) 2022-03-01 12:39:12 -05:00
Tyler Goodlet 09079b61fc Comment task canceller method prototype 2022-03-01 12:37:31 -05:00
Tyler Goodlet adccb687fe Fix `piker services` cmd 2022-03-01 12:36:32 -05:00
Tyler Goodlet c239faf4e5 Add a `._sampling.sampler` registry composite type 2022-03-01 12:36:32 -05:00
Tyler Goodlet 6f3d78b729 Handle "no data" case in ranger calcs and avoid crashes 2022-02-28 08:30:44 -05:00
Tyler Goodlet 3e7d4f8717 Detect and request sample period in fsp engine 2022-02-28 08:30:32 -05:00
Tyler Goodlet b1cce8f9cf Adjust and add notes for python-trio/trio#2258 2022-02-28 08:30:22 -05:00
Tyler Goodlet 89a98c4aa2 Fix portal result `await`, comment some unused code 2022-02-28 08:30:15 -05:00
Tyler Goodlet 7a943f0e1e Always transmit index event even when no shm is registered 2022-02-28 08:29:56 -05:00
Tyler Goodlet 786ffde4e6 Use 3.9+ annots 2022-02-28 08:27:59 -05:00
Tyler Goodlet 11d4ebd0b5 Just warn on double-remove of a sub 2022-02-28 08:27:37 -05:00
Tyler Goodlet 81f8b4e145 Don't zero clearing rates on sample steps 2022-02-28 08:26:48 -05:00
Tyler Goodlet cc55e1f4bb Drop task-driven sample step graphics updates
Since moving to a "god loop" for graphics, we don't really need to have
a dedicated task for updating graphics on new sample increments. The
only UX difference will be that curves won't be updated until an actual new
rt-quote-event triggers the graphics loop -> so we'll have the chart
"jump" to a new position and new curve segments generated only when new
data arrives. This is imo fine since it's just less "idle" updates
where the chart would sit printing the same (last) value every step.
Instead only update the view increment if a new index is detected by
reading shm.

If we ever want this dedicated task update again this commit can be
easily reverted B)
2022-02-28 08:26:26 -05:00
Tyler Goodlet 412c9ee6cf Support view increment with a steps size 2022-02-28 08:26:20 -05:00
Tyler Goodlet bf3b58e861 Async load data history, allow "offline" feed use
Break up real-time quote feed and history loading into 2 separate tasks
and deliver a client side `data.Feed` as soon as history is loaded
(instead of waiting for a rt quote - the previous logic). If
a symbol doesn't have history then likely the feed shouldn't be loaded
(since presumably client code will need at least "some" datums history
to do anything) and waiting on a real-time quote is dumb, since it'll
hang if the market isn't open XD. If a symbol doesn't have history we
can always write a zero/null array when we run into that case. This also
greatly speeds up feed loading when both history and quotes are available.

TL;DR summary:
- add a `_Feedsbus.start_task()` one-cancel-scope-per-task method for
  assisting with (re-)starting and stopping long running persistent
  feeds (basically a "one cancels one" style nursery API).
- add a `manage_history()` task which does all history loading (and
  eventually real-time writing) which has an independent signal and
  start it in a separate task.
- drop the "sample rate per symbol" stuff since client code doesn't really
  care when it can just inspect shm indexing/time-steps itself.
- run throttle tasks in the bus nursery thus avoiding cancelling the
  underlying sampler task on feed client disconnects.
- don't store a repeated ref the bus nursery's cancel scope..
2022-02-28 08:26:13 -05:00
Tyler Goodlet 1d3ed6c333 Add `mk_` prefix since assignments will use `fqsn` 2022-02-28 08:23:57 -05:00
Tyler Goodlet 832e4c97d2 Drop shm: ShmArray` to `stream_quotes()` endpoint 2022-02-28 08:23:16 -05:00
Tyler Goodlet 23aa7eb31c Stick time step in window header 2022-02-28 08:22:47 -05:00
Tyler Goodlet c2a13c474c Support no realtime stream sending with feed bus 2022-02-28 08:22:40 -05:00
Tyler Goodlet 7252094f90 Add `open_piker_runtime()` to setup actor runtime correctly from non-daemons 2022-02-28 08:16:30 -05:00
Tyler Goodlet b1dd24d1f7 Only throttle warn on rate >= display rate 2022-02-28 08:15:39 -05:00
Tyler Goodlet a073039b30 Drop dependence on `msgpack` and `msgpack_numpy` 2022-02-28 08:15:18 -05:00
Tyler Goodlet 5c343aa748 Misc curve doc strings 2022-02-28 08:14:11 -05:00
Tyler Goodlet 7b13124dd4 Keep clear loop price pedantically up to date
To avoid the "trigger finger" issue (darks execing before they should
due to a stale last price state, normally when generating a trigger
predicate..) always iterate the loop and update the last known book
price even when no execs/triggered orders are registered.
2022-02-11 10:30:30 -05:00
Tyler Goodlet ca1c1cf415 Annoying doc strings 2022-02-11 10:30:30 -05:00
Tyler Goodlet 92c63988bc Bleh, just fill the available window space 2022-02-11 10:07:43 -05:00
Tyler Goodlet 9ed153bcb6 Less gap below results view 2022-02-11 08:45:57 -05:00
Tyler Goodlet 412c34eba0 Drop width check logic; only do height 2022-02-11 08:32:28 -05:00
Tyler Goodlet 68e1db27f8 Drop old null window size 2022-02-10 14:35:28 -05:00
Tyler Goodlet 86b1316691 Handle no-rows-yet case 2022-02-10 14:35:11 -05:00
Tyler Goodlet 890ffc76cf Dynamically re-size the search results view 2022-02-10 14:22:46 -05:00
Tyler Goodlet 51d94a301a Support resize event relaying from the god widget 2022-02-10 14:21:17 -05:00
Tyler Goodlet c54c9ae3d3 Add doc string to DE sizing method 2022-02-10 14:20:15 -05:00
Tyler Goodlet 5a4c155798 Add detailed comment around DE scaling 2022-02-10 13:04:13 -05:00
wattygetlood a5ad24f770 Additionally apply DPI scaling to font size if detected 2022-02-10 10:26:52 -05:00
Tyler Goodlet a0034e2948 If the DE (like windohz) already scales DPI, just use that scale for font size 2022-02-10 10:26:52 -05:00
wattygetlood fc3c0741b8 Set isn't serializable on std msgpack 2022-02-10 10:26:52 -05:00
wattygetlood cc87508fd9 Only load 4 ib requests worth of bars on windows... 2022-02-10 10:26:52 -05:00
wattygetlood d069481f1d Hack search view on windows to 1/2 window height; needs a better solution 2022-02-10 10:26:52 -05:00
wattygetlood c411a244f6 Size the window to aproximately 1/3 the screen space 2022-02-10 10:26:52 -05:00
wattygetlood 15556e40f0 No support for notifications (yet) on windows 2022-02-10 10:26:52 -05:00
wattygetlood c0082e15bc Fix default `brokers.toml` copying since module move 2022-02-10 10:26:52 -05:00
wattygetlood 2ebdf008da Configure window size based on screen dims on windows 2022-02-10 10:26:52 -05:00
Tyler Goodlet 71f9b5c000 Don't enable curve coord cache unless in step mode
You can get a weird "last line segment" artifact if *only* that segment
is drawn and the cache is enabled, so just disable unless in step mode
at startup and re-flash as normal when new path data is appended. Add
a `.disable_cache()` method for the multi-use in the update method. Use
line style on the `._last_line: QLineF` segment as well.
2022-02-10 08:12:15 -05:00
Tyler Goodlet 228f21d7b0 Zero trade rates each step 2022-02-09 22:16:33 -05:00
Tyler Goodlet 45464a5465 Drop graphics throttle to 22Hz, add a `.maxmin` to our view box 2022-02-09 22:15:57 -05:00
Tyler Goodlet 723eef3fd6 🤦 assign `Flow` *after* type check... 2022-02-09 16:00:10 -05:00
Tyler Goodlet e0462f0a8c Type and formatting fixes 2022-02-08 15:57:32 -05:00
Tyler Goodlet 1c49f7f47f Tweak dash pattern to be less sparse 2022-02-08 15:57:02 -05:00
Tyler Goodlet ef04781a2b Expect new flow type through display and fsp UI code 2022-02-08 15:56:20 -05:00
Tyler Goodlet e3a3fd2d39 Add a `Flow` compound type for coupling graphics with backing data-streams 2022-02-08 15:52:50 -05:00
Tyler Goodlet 860ed99757 Drop dvlm "rates" curves from flows chart 2022-02-08 12:05:56 -05:00
Tyler Goodlet 326b2c089a Drop dvlm 'rates' (they're just means), add default params, period -> 6 2022-02-08 12:04:01 -05:00
Tyler Goodlet 8f467bf4f0 Factor batch curve plotting into helper func 2022-02-08 08:21:08 -05:00
Tyler Goodlet 4a7b2d835b Yield 0 initial values from `flow_rates` fsp 2022-02-08 07:46:36 -05:00
Tyler Goodlet 30cf54480d Add more appropriate default params 2022-02-07 13:59:26 -05:00
Tyler Goodlet ee4ad32d3b Fix `dvlm` to actually yield trade count, add instantaneous support 2022-02-07 12:53:30 -05:00
Tyler Goodlet e7516447df Better rate axis title? 2022-02-07 12:53:30 -05:00
Tyler Goodlet a006b87546 Exit `.maxmin()` early on non-yet-registered array lookup 2022-02-07 12:53:30 -05:00
Tyler Goodlet 9490129a74 Add overlays to end of layout grid (aka append) by default 2022-02-07 12:53:30 -05:00
Tyler Goodlet 2f2aef28dd Adjust x-axis label from summed left axes widths 2022-02-07 12:53:30 -05:00
Tyler Goodlet 0271841412 Add `PlotItemOverlay.get_axes()`
Enables retrieving all "named axes" on a particular "side" of the
overlayed plot items. This is useful for calculating how much space
needs to be allocated for the axes before the view box area starts.
2022-02-07 12:53:30 -05:00
Tyler Goodlet e8d7709358 Drop notification display time to piker seconds worth 2022-02-07 12:53:30 -05:00
Tyler Goodlet e3c46a5d4d Add draft, commented tickbytick for ib 2022-02-07 12:53:30 -05:00
Tyler Goodlet 8d432e1988 Shorter clear rate axis title 2022-02-07 12:53:30 -05:00
Tyler Goodlet 87653ddca2 Simplify to only needed one LHS axis for clearing rates 2022-02-07 12:53:30 -05:00
Tyler Goodlet 73faafcfc1 Add trade "rates" (i.e. trade counts) support B)
Though it's not per-tick accurate, accumulate the number of "trades"
(i.e. the "clearing rate" - maybe this is a better name?) per bar
inside the `dolla_vlm` fsp and average and report wmas of this in the
`flow_rates` fsp.
2022-02-07 12:53:30 -05:00
Tyler Goodlet e4244e96a9 Fix var name typo 2022-02-07 12:53:30 -05:00
Tyler Goodlet 5274eb538c Add 16 period dollar vlm rates, drop ib rates for now 2022-02-07 12:53:30 -05:00
Tyler Goodlet b358b8e874 Move `wma` fsp earlier in module 2022-02-07 12:53:30 -05:00
Tyler Goodlet 2d3c685e19 Typecast np dtype description to a tuple 2022-02-07 12:53:30 -05:00
Tyler Goodlet 4570b06c26 Handle no y-range maxmin output (seems like bug?) 2022-02-07 12:53:30 -05:00
Tyler Goodlet 26b0071471 Subscribe for rate calcs from IB in default tick set 2022-02-07 12:53:30 -05:00
Tyler Goodlet 1fc6429f75 Prep for manual rate calcs, handle non-ib backends XD 2022-02-07 12:53:30 -05:00
Tyler Goodlet ebf3e00438 Add `Fsp._flow_registry` as actor-local table
Define the flows table as a class var (thus making it a "global" and/or
actor-local state) which can be accessed by any in process task. Add
`Fsp.get_shm()` to allow accessing output streams by source-token + fsp
routine reference and thus providing inter-fsp low level access to
real-time flows.
2022-02-07 12:53:30 -05:00
Tyler Goodlet df6afe24a4 Define a flow registry on `FspAdmin`, use it to update fsp engine clusters 2022-02-07 12:53:30 -05:00
Tyler Goodlet d130f0449f Expect registry of fsp "flows" to each engine task
In order for fsp routines to be able to look up other "flows" in the
cascade, we need a small registry-table which gives access to a map of
a source stream + an fsp -> an output stream. Eventually we'll also
likely want a dependency (injection) mechanism so that any fsp demanded
can either be dynamically allocated or at the least waited upon before
a consumer tries to access it.
2022-02-07 12:53:30 -05:00
Tyler Goodlet efb743fd85 Flip to using `pydantic` for shm tokens 2022-02-07 12:53:30 -05:00
Tyler Goodlet 615bf3a55a Use solid line for vlm rate and dashed for trades rate 2022-02-07 12:53:30 -05:00
Tyler Goodlet d4f79a6245 Comment flow rates fsp prints 2022-02-07 12:53:30 -05:00
Tyler Goodlet 4b7d1fb35b Add line style via `str` style name to our fast curve 2022-02-07 12:53:30 -05:00
Tyler Goodlet 0b5250d5e3 Plot the vlm rate (per min) taken verbatim from ib 2022-02-07 12:53:30 -05:00
Tyler Goodlet 97c2f86092 TOSQUASH, fix separate vlm vs trade rate 2022-02-07 12:53:30 -05:00
Tyler Goodlet f3289c1977 Create source length zeroed arrays from yielded `None` fsp history 2022-02-07 12:53:30 -05:00
Tyler Goodlet 4e96dd09e3 Add a `.text_color` property to our axes types 2022-02-07 12:53:30 -05:00
Tyler Goodlet b81209e78e Ensure `sym` arg is a `str` 2022-02-07 12:53:30 -05:00
Tyler Goodlet dfe4473c9a Yield history `dict`s, add a `flow_rates` fsp 2022-02-07 12:53:30 -05:00
Tyler Goodlet 1aae40cdeb Expect multi-output fsps to yield a `dict` of history arrays 2022-02-07 12:53:30 -05:00
Tyler Goodlet 8118a57b9a Guard against no time field in some provider quotes 2022-02-07 12:53:30 -05:00
Tyler Goodlet 5952e7f538 Add dark vlm deduplication support via flag 2022-02-07 12:53:30 -05:00
Tyler Goodlet 16c04e11e4 Comment out nan-price assert previously for `ib` in ems clear tasks 2022-02-07 09:49:45 -05:00
Tyler Goodlet 9bfad86c29 Drop timeout-cancel block 2022-02-07 09:49:45 -05:00
Tyler Goodlet a9d42b374f ib: Allow history backfilling even when markets are closed 2022-02-07 09:49:45 -05:00
goodboy 43b39d3b6b
Merge pull request #275 from pikers/py3.10_support
Py3.10 support
2022-02-07 09:48:54 -05:00
Tyler Goodlet 00a90e7390 Change dpi log msg back to debug 2022-02-07 09:36:07 -05:00
Tyler Goodlet 1aaa382036 Avoid null index race-error during startup 2022-02-07 09:36:07 -05:00
Tyler Goodlet 999d3efdd7 Another `int` required 2022-02-07 09:36:07 -05:00
Tyler Goodlet f63a7c497d Latest `PyQt5` expects `ints` for widget sizings 2022-02-07 09:36:07 -05:00
Guillermo Rodriguez 82d1b85b09
Add default for multiplier var 2022-02-02 20:46:45 -03:00
Tyler Goodlet bb8fade16f Update `msgpub` import from `tractor.experimental` 2022-01-30 12:46:54 -05:00
Tyler Goodlet 296863348d Update imports to `tractor.msg.NamespacePath` 2022-01-30 12:46:20 -05:00
Tyler Goodlet 95b31cbc0f Drop references to deprecated `tractor.msg.pub` 2022-01-29 12:44:45 -05:00
Tyler Goodlet 6a0fba1eb3 Support maxmin over multiple arrays; Keep dark vlm in view 2022-01-28 11:45:47 -05:00
Tyler Goodlet 06934be047 Overlay dark $volume B) 2022-01-28 08:46:24 -05:00
Tyler Goodlet 28b5be0719 Accumulate dark vlm ticks independently per sample step 2022-01-28 08:46:04 -05:00
Tyler Goodlet 67de8f24b9 Init history output with `np.zeros()` 2022-01-28 08:45:28 -05:00
Tyler Goodlet b112f24e7e Drop old commented cruft, use `Fsp.name` 2022-01-28 07:51:13 -05:00
Tyler Goodlet bd2460846e Decorate momo routines 2022-01-28 07:43:49 -05:00
Tyler Goodlet be93ded0e5 Log fsp registy entries in `cascade` startup 2022-01-28 07:43:23 -05:00
Tyler Goodlet 9d9929fb89 Drop old `wrapt` cruft, add `Fsp.name` 2022-01-28 07:18:14 -05:00
Tyler Goodlet cc5390376c Use `Fsp` abstration layer through engine and UI
Instead of referencing the remote processing funcs by a `str` name start
embracing the new `@fsp`/`Fsp` API such that wrapped processing
functions are first class APIs.

Summary of the changeset:
- move and load the fsp built-in set in the new `.fsp._api` module
- handle processors ("fsps") which want to yield multiple keyed-values
  (interleaved in time) by expecting both history that is keyed and
  assigned to the appropriate struct-array field, *and* real-time
  `yield`ed value in tuples of the form `tuple[str, float]` such that
  any one (async) processing function can deliver multiple outputs from
  the same base calculation.
- drop `maybe_mk_fsp_shm()` from UI module
- expect and manage `Fsp` instances (`@fsp` decorated funcs) throughout
  the UI code, particularly the `FspAdmin` layer.
2022-01-27 18:57:16 -05:00
Tyler Goodlet 72f4474273 Use new decorator on volume fsp routines 2022-01-27 09:08:03 -05:00
Tyler Goodlet c6a3c66e7e WIP start a `@piker.fsp` API for registering processors 2022-01-26 14:38:49 -05:00
Tyler Goodlet 13b8807f1f Print dark trades to console for the moment 2022-01-26 13:48:21 -05:00
Tyler Goodlet 55cfe6082b Re-key ib's 'unreportable trades' (tick 48) as 2022-01-26 13:48:21 -05:00
Tyler Goodlet d351fe14a8 Annoying doc string(s) 2022-01-25 08:30:00 -05:00
Tyler Goodlet 4e884aec6c Fix bottom axis check logic for overlays, try out some px perfection 2022-01-25 08:30:00 -05:00
Tyler Goodlet 7b21ddd27f Allow passing in parent to `Label` 2022-01-25 08:30:00 -05:00
Tyler Goodlet fd31b843b9 Hide the unit vlm after the $vlm is up
Since more curves costs more processing and since the vlm and $vlm
curves are normally very close to the same (graphically) we hide the
unit volume curve once the dollar volume is up (after the fsp daemon-task is
spawned) and just expect the user to understand the diff in axes units.
Also, use the new `title=` api to `.overlay_plotitem()`.
2022-01-25 08:30:00 -05:00
Tyler Goodlet 7f4546b71f Use overlay api to access multi-axes by name 2022-01-25 08:30:00 -05:00
Tyler Goodlet f5eb34c4d7 Make axes labels more pixel perfect 2022-01-25 08:30:00 -05:00
Tyler Goodlet c7a588cf25 Pop vlm chart from subplots to avoid double render 2022-01-25 08:30:00 -05:00
Tyler Goodlet 5c2d3125b4 Add vlm axis titles and humanized $vlm y-range 2022-01-25 08:30:00 -05:00
Tyler Goodlet f011234285 Type annot and docs updates in anchors mod 2022-01-25 08:30:00 -05:00
Tyler Goodlet ce7c174059 Add `Axis.set_title()` for hipper labelling
Use our internal `Label` with much better dpi based sizing of text and
placement below the y-axis ticks area for more minimalism and less
clutter.

Play around with `lru_cache` on axis label bounding rects and for now
just hack sizing by subtracting half the text height (not sure why) from
the width to avoid over-extension / overlap with any adjacent axis.
2022-01-25 08:30:00 -05:00
Tyler Goodlet 94b6f370a9 Allow axis kwargs passthrough 2022-01-25 08:30:00 -05:00
Tyler Goodlet 349040dbf0 Revert cursor rate limit settings 2022-01-25 08:30:00 -05:00
Tyler Goodlet 80079105fc Add custom `.formatter` support to our `PriceAxis`
Allow passing in a formatter function for processing tick values on an
axis. This makes it easy to for example, `piker.calc.humanize()` dollar
volume on a subchart.

Factor `set_min_tick()` into the `PriceAxis` since it's not used on any
x-axis data thus far.
2022-01-25 08:30:00 -05:00
Tyler Goodlet 8911c3c8ed Add support for "humanized" axes tick values 2022-01-25 08:30:00 -05:00
Tyler Goodlet 9813cf4169 Add a symbol "front feed" helper 2022-01-25 08:24:55 -05:00
Tyler Goodlet c3c1e14cf4 Start vlm and other fsps as separate tasks 2022-01-25 08:24:55 -05:00
Tyler Goodlet 404f5d6d23 Factor (sub-)chart spawning into a admin method
Adds `FspAdmin.open_fsp_chart()` which allows adding a real time graphics
display of an fsp's output with different options for where (which chart
or make a new one) to place it.

Further,
- change some method naming, namely the other fsp engine task methods to
  `.open_chain()` and `.start_engine_task()`.
- make `run_fsp_ui()` a lone task function for now with the default
  config parsing and chart setup logic (and it still includes a buncha
  commented out stuff for doing graphics update which is now done in the
  main loop to avoid task switching overhead).
- move all vlm related fsp config entries into the `open_vlm_displays()`
  task for dedicated setup with the fsp admin api such as special
  auto-yrange handling and graph overlays.
- `start_fsp_displays()` is now just a small loop through config entries
  with synced startup status messages.
2022-01-25 08:24:55 -05:00
Tyler Goodlet e22a652852 Move plotitem overlaying into a `.overlay_plotitem()` 2022-01-25 08:24:55 -05:00
Tyler Goodlet 09fc901b0d Handle left axis case for x-axis label placement
For wtv cucked reason all the viewbox/scene coordinate calcs do **not**
include a left axis in the geo (likely because it's a hacked in widget
+ layout thing managed by `PlotItem`). Detect if there's a left axis and
if so use it in the label placement scene coords calc. ToDo: probably
make this a non-move calc and only recompute any time the axis changes.

Other:
- rate limit mouse events down to the 60 (ish) Hz for now
- change one last lingering `'ohlc'` array lookup
- fix `.mouseMoved()` "event" type annot
2022-01-25 08:24:55 -05:00
Tyler Goodlet e69af9e291 Show unit vlm on LHS for now 2022-01-25 08:24:55 -05:00
Tyler Goodlet 06fe2bd1be Support "volume" and "dollar volume" on same chart
This is a huge commit which moves a bunch of code around in order to
simplify some of our UI modules as well as support our first official
mult-axis chart: overlaid volume and "dollar volume". A good deal of
this change set is to make startup fast such that volume data which is
often shipped alongside OHLC history is loaded and shown asap and FSPs
are loaded in an actor cluster with their graphics overlayed
concurrently as each responsible worker generates plottable output.

For everything to work this commit requires use of a draft `pyqtgraph`
PR: https://github.com/pyqtgraph/pyqtgraph/pull/2162

Change summary:
- move remaining FSP actor cluster helpers into `.ui._fsp` mod as well
  as fsp specific UI managers (`maybe_open_vlm_display()`,
  `start_fsp_displays()`).
- add an `FspAdmin` API for starting fsp chains on the cluster
  concurrently allowing for future work toward reload/unloading.
- bring FSP config dict into `start_fsp_displays()` and `.started()`-deliver
  both the fsp admin and any volume chart back up to the calling display
  loop code.

ToDo:
- repair `ChartView` click-drag interactions
- auto-range on $ vlm needs to use `ChartPlotWidget._set_yrange()`
- a lot better styling for the $_vlm overlay XD
2022-01-25 08:24:55 -05:00
Tyler Goodlet 7a41c83f84 Move FSP related graphics management into new mod 2022-01-25 08:24:55 -05:00
Tyler Goodlet b7f27f201f Add `try_read()` to shm mod 2022-01-25 08:24:55 -05:00
Tyler Goodlet 0ae79d6418 Drop `pyqtgraph` import 2022-01-25 07:59:08 -05:00
Tyler Goodlet d170132eb5 Use view for auto-yrange in display loop 2022-01-25 07:59:08 -05:00
Tyler Goodlet ced310c194 Add `ChartPlotWidget.maxmin()` to calc in-view hi/lo y-values
As part of factoring `._set_yrange()` into the lower level view box,
move the y-range calculations into a new method. These calcs should
eventually be completely separate (as they are for the real-time version
in the graphics display update loop) and likely part of some kind of
graphics-related lower level management API. Draft such an API as an
`ArrayScene` (commented for now) as a sketch toward factoring array
tracking **out of** the chart widget. Drop the `'ohlc'` array name and
instead always use whatever `.name` was assigned to the chart widget
to lookup its "main" / source data array for now.

Enable auto-yranging on overlayed plotitems by enabling on its viewbox
and, for now, assign an ad-hoc `._maxmin()` since the widget version
from this commit has no easy way to know which internal array to use. If
an FSP (`dolla_vlm` in this case) is overlayed on an existing chart
without also having a full widget (which it doesn't in this case since
we're using an overlayed `PlotItem` instead of a full `ChartPlotWidget`)
we need some way to define the `.maxmin()` for the overlayed
data/graphics. This likely means the `.maxmin()` will eventually get
factored into wtv lowlevel `ArrayScene` API mentioned above.
2022-01-25 07:59:08 -05:00
Tyler Goodlet fbb765e1d8 Import overlay from new internal module 2022-01-25 07:59:08 -05:00
Tyler Goodlet 80d16886cb Add auto-yrange handler to our `ChartView`
Calculations for auto-yaxis ranging are both signalled and drawn by our
`ViewBox` so we might as well factor this handler down from the chart
widget into the view type. This makes it much easier (and clearer) that
`PlotItem` and other lower level overlayed `GraphicsObject`s can utilize
*size-to-data* style view modes easily without widget-level coupling.

Further changes,
- support a `._maxmin()` internal callable (temporarily) for allowing
  a viewed graphics object to define it's own y-range max/min calc.
- add `._static_range` var (though usage hasn't been moved from the
  chart plot widget yet
- drop y-axis click-drag zoom instead reverting back to default viewbox
  behaviour with wheel-zoom and click-drag-pan on the axis.
2022-01-25 07:59:08 -05:00
Tyler Goodlet e66b3792bb Add overlay relay signals and attr to our `ChartView` 2022-01-25 07:59:08 -05:00