Commit Graph

3588 Commits (482403c8878522627bc4713ce984bcdc8c7c4132)

Author SHA1 Message Date
jaredgoldman 9706803220 Refactor streaming logic to be less nested and readable 2023-04-16 10:11:17 -04:00
jaredgoldman 8403d8a482 Simplify numpy mapping logic 2023-04-15 21:05:25 -04:00
Tyler Goodlet a111819667 Few fixes after review to get running again B)
- use `Struct.copy()` for frozen type
- fix `BrokerConfig` delegation attr lookups
- bit of linting according to `flake8`
2023-04-14 19:05:19 -04:00
jaredgoldman 4f576b6f36 Fix typo with ts vars 2023-04-13 22:37:17 -04:00
jaredgoldman 672c01f13a Use trade_data_ts for trade message receival 2023-04-13 22:35:21 -04:00
jaredgoldman f67ffeb70f Remove extra Noen check on msg.get 2023-04-13 22:34:04 -04:00
jaredgoldman 1b1e35d32d Add comment explaining waiting for first trade quote 2023-04-13 22:28:44 -04:00
jaredgoldman 9f5dfe8501 Remove anext() comment 2023-04-13 22:27:56 -04:00
jaredgoldman 11bd2e2f65 Use datetime | none instead of Optional[datetime] in get_bars 2023-04-13 22:04:43 -04:00
jaredgoldman ebfd490a1a Cache instead of get pairs in symbol search 2023-04-13 22:02:13 -04:00
jaredgoldman 89bb124728 Remove old comments normalize arguents and improve pair fetching log 2023-04-13 22:00:41 -04:00
jaredgoldman 63e34cf595 Typecast config, add type hint to pair in init message creation and turn init msg vals into floats 2023-04-13 21:57:54 -04:00
jaredgoldman 92f372dcc8 Use proper value for init message 2023-04-13 21:52:40 -04:00
jaredgoldman b00abd0e51 Add a fail case ws token request 2023-04-13 21:48:17 -04:00
jaredgoldman 52a015d927 Remove typo in binance 2023-04-12 21:40:58 -04:00
jaredgoldman 2c82b2aba9 Remove breakpoint in binance 2023-04-12 20:43:28 -04:00
jaredgoldman ff0f8dfaca Improve client._get_ws_token docstring 2023-04-12 20:37:10 -04:00
jaredgoldman ace04af21a Use anext() in kucoin stream_quotes 2023-04-12 20:25:35 -04:00
jaredgoldman d2f3a79c09 Use pendulum for header timestamp,
type hint cleanup
2023-04-12 19:48:46 -04:00
jaredgoldman bedbbc3025 Only diff trade time 2023-04-12 19:48:46 -04:00
jaredgoldman 6e55f6706f Format condition for filtering and add link to docs explaining need for filtering in the first case 2023-04-12 19:48:46 -04:00
jaredgoldman d1b0608c88 Remove breakpoint 2023-04-12 19:48:46 -04:00
jaredgoldman 3bed3a64c3 Implement duplicate filtering at message level 2023-04-12 19:48:46 -04:00
jaredgoldman 93e7d54c5e Add api doc links to _get_bars def 2023-04-12 19:48:46 -04:00
jaredgoldman 9db84e8029 Remove norm_pairs method and do all normalization in initial _get_pairs call 2023-04-12 19:48:46 -04:00
jaredgoldman ea21656624 Don't cache pairs in _get_pairs call 2023-04-12 19:48:46 -04:00
jaredgoldman 5a0d29c774 Add ws token api doc link 2023-04-12 19:48:46 -04:00
jaredgoldman 13df3e70d5 Refactor sign gen into one line 2023-04-12 19:48:46 -04:00
jaredgoldman 208a8e5d7a Remove unecessary config vars 2023-04-12 19:48:46 -04:00
jaredgoldman ca937dff5e Add api doc links in structs 2023-04-12 19:48:46 -04:00
jaredgoldman c68fcf7e1c Remove extra line from docstrings 2023-04-12 19:48:46 -04:00
jaredgoldman 48c3b333b2 Format imports with parenthesis 2023-04-12 19:48:46 -04:00
jaredgoldman b71f6b6c67 Strip uneccesary data from ticks in l1 data feed 2023-04-12 19:48:46 -04:00
jaredgoldman 54cf648d74 Ensure sub logging dict attritbutes will be there 2023-04-12 19:48:46 -04:00
jaredgoldman 68d0327d41 Remove breakpoints, simplify backoff logic 2023-04-12 19:48:46 -04:00
jaredgoldman 68a06093e9 Format and ensure we're only grabbing the most closest bid and ask 2023-04-12 19:48:46 -04:00
jaredgoldman 52aadb374b Add L1 data feed and correct history issue 2023-04-12 19:48:46 -04:00
jaredgoldman dfd030a6aa Remove float conversion of key_id again 2023-04-12 19:48:46 -04:00
jaredgoldman 788e158d9f Stop still converting datetime to float 2023-04-12 19:48:46 -04:00
jaredgoldman 81890a39d9 Leave datetimes alone! 2023-04-12 19:48:46 -04:00
jaredgoldman ae170f2645 Add more informative logs on startup 2023-04-12 19:48:46 -04:00
jaredgoldman e2e5191ded Remove breaking useless condition for determining if res is list of ohlc values 2023-04-12 19:48:46 -04:00
jaredgoldman dcbb7fa64f Remove float conversion for config key id 2023-04-12 19:48:46 -04:00
jaredgoldman 32107d0ac3 Strengthen retry case and add comments 2023-04-12 19:48:46 -04:00
jaredgoldman 7bdebd47d1 Add exponential retry case for history client 2023-04-12 19:48:46 -04:00
jaredgoldman ac31bca181 Make broker creds/auth optional 2023-04-12 19:48:46 -04:00
jaredgoldman 52070c00f9 Remove typo 2023-04-12 19:48:46 -04:00
jaredgoldman 5ff0cc7905 Cast/validate streamed messages
Update comments

Minor formatting

Minor formatting
2023-04-12 19:48:46 -04:00
jaredgoldman 6ad1e3da38 Correct typo in license 2023-04-12 19:48:46 -04:00
jaredgoldman 9bf6f557ed Label private methods accordingly, remove cryptofeeds module 2023-04-12 19:48:46 -04:00
jaredgoldman 50e1070004 More cleanup, add comments re sub func 2023-04-12 19:48:46 -04:00
jaredgoldman 1c4c19b351 Clean up broker code,
Add typecasting for messages/rt-data and historcal user trades
ensure we're fetching all history
add multi-symbol support
'
2023-04-12 19:48:46 -04:00
jaredgoldman 199a70880c Spawn background ping task 2023-04-12 19:48:46 -04:00
jaredgoldman b14b323068 Remove breakpoint in web_bs,
ensure we only unsub if ws is connected
2023-04-12 19:48:46 -04:00
jaredgoldman a3c7bec576 Implement working message streaming 2023-04-12 19:48:46 -04:00
jaredgoldman ac34ca7cad Add sub method to flow
Stash for checkout of master
2023-04-12 19:48:46 -04:00
jaredgoldman ade2c32adb Succesfully connect to kucoin ws 2023-04-12 19:48:46 -04:00
jaredgoldman 109e7d7b43 Add back static API version in headers 2023-04-12 19:48:46 -04:00
jaredgoldman 1a655b7e39 Ensure we're passing the correct api version to the header builder,
make headers a default arg
2023-04-12 19:48:46 -04:00
jaredgoldman cda045f123 Abstract header gen to seperate function 2023-04-12 19:48:46 -04:00
jaredgoldman 7074ca7713 Implement Kucoin auth and last trades call 2023-04-12 19:48:46 -04:00
Tyler Goodlet 8e91e215b3 WIP - ensure `asyncio` pumps the event loop each send 2023-04-12 19:48:46 -04:00
jaredgoldman c751c36a8b Update trade message format 2023-04-12 19:48:46 -04:00
jaredgoldman ad9d645782 WIP - setup basic history and streaming client 2023-04-12 19:48:46 -04:00
jaredgoldman c96d4387c5 Start adding history client 2023-04-12 19:48:46 -04:00
jaredgoldman 5fdec8012d Add cryptofeeds data feed module,
Add Kucoin backend client
wip
2023-04-12 19:48:46 -04:00
Tyler Goodlet 609b91e848 Try out `@trio_util.async_generator` for streaming
Apparently it will likely fix our `trio`-cancel-scopes-corrupted crash
when we try to let our `._web_bs.NoBsWs` do reconnect logic around
the asyn-generator implemented data-feed streaming routines in `binance`
and `kraken`.  See the project docs for deatz; obvs we add the lib as
a dep.
2023-03-20 12:54:48 -04:00
Tyler Goodlet 78eb784091 Stick `try:` outside all `xdotool` subproc calls 2023-03-13 15:36:45 -04:00
Tyler Goodlet 973e4b5f44 `binance`: wrap streamer async-gen in `aclosing()` 2023-03-13 15:36:29 -04:00
Tyler Goodlet 9197e6decb `binance`: use built-in `anext()` add note about new ws ep URL 2023-03-13 15:36:29 -04:00
Tyler Goodlet 889e920796 Short-circuit rendering on no 1d-data; avoid m4 layer crash 2023-03-13 12:18:54 -04:00
Tyler Goodlet 1aab9f1f81 Actually yes, we need to handle empty in-view range.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5c697de58e Presume never handling not-in-view case for minor curves 2023-03-10 18:20:22 -05:00
Tyler Goodlet 3066b1541e Handle (shorter supported) minor-curve not-in-view
Solve this by always scaling the y-range for the major/target curve
*before* the final overlay scaling loop; this implicitly always solve
the case where the major series is the only one in view.

Tidy up debug print formatting and add some loop-end demarcation comment
lines.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 32339cb41a Always show a minimum bars during startup
This is particularly more "good looking" when we boot with a pair that
doesn't have historical 1s OHLC and thus the fast chart is empty from
outset. In this case it's a lot nicer to be already zoomed to
a comfortable preset number of "datums in view" even when the history
isn't yet filled in.

Adjusts the chart display `Viz.default_view()` startup to explicitly
ensure this happens via the `do_min_bars=True` flag B)
2023-03-10 18:20:22 -05:00
Tyler Goodlet 12e196a6f7 Catch `KeyError` on bcast errors which pop the sub
Not sure how i missed this (and left in handling of `list.remove()` and
it ever worked for that?) after the `samplerd` impl in 5ec1a72 but, this
adjusts the remove-broken-subscriber loop to catch the correct
`set.remove()` exception type on a missing (likely already removed)
subscription entry.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8a87e5f390 Remove leftover debug print in cache reset meth 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5958acebe1 Add (commented) draft 1min OHLC time index logging
For the purposes of eventually trying to resolve last-step indexing
synchronization (an intermittent but still existing) issue(s) that can
happen due to races during history frame query and shm writing during
startup. In fact, here we drop all `hist_viz` info queries from the main
display loop for now anticipating that this code will either be removed
or improved later.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8d1c713a5a Always pass step to `slice_from_time()` in view mode
Again, as per the signature change, never expect implicit time step
calcs from overlay processing/machinery code. Also, extend the debug
printing (yet again) to include better details around
"rescale-due-to-minor-range-out-of-view" cases and a detailed msg for
the transform/scaling calculation (inputs/outputs), particularly for the
cases when one of the curves has a lesser support.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 32926747c6 Always pass `step` to `slice_from_time()` in the `Viz`
As per the change to `slice_from_time()` this ensures this `Viz` always
passes its self-calculated time indexing step size to the time slicing
routine(s).

Further this contains a slight impl tweak to `.scalars_from_index()` to
slice the actual view range from `xref` to `Viz.ViewState.xrange[1]` and
then reading the corresponding `yref` from the first entry in that
array; this should be no slower in theory and makes way for further
caching of x-read-range to `ViewState` opportunities later.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 712f1a47a0 Require `step: float` input to `slice_from_time()`
There's been way too many issues when trying to calculate this
dynamically from the input array, so just expect the caller to know what
it's doing and don't bother with ever hitting the error case of
calculating and incorrect value internally.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 51f3733487 Handle "target-is-shorter-then-pinned" case
When the target pinning curve (by default, the dispersion major) is
shorter then the pinned curve, we need to make sure we find still find
the x-intersect for computing returns scalars! Use `Viz.i_from_t()` to
accomplish this as well and, augment that method with a `return_y: bool`
to allow the caller to also retrieve the equivalent y-value at the
requested input time `t: float` for convenience.

Also tweak a few more internals around the 'loglin_ref_to_curve'
method:
- only solve / adjust for the above case when the major's xref is
  detected as being "earlier" in time the current minor's.
- pop the major viz entry from the overlay table ahead of time to avoid
  a needless iteration and simplify the transform calc phase loop to
  avoid handling that needless cycle B)
- add much better "organized" debug printing with more clear headers
  around which "phase"/loop the message pertains and well as more
  explicit details in terms of x and y-range values on each cycle of
  each loop.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 4bb580ae60 Don't `@lru_cache` on `Viz.i_from_t()`, since view state.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 05aee4a311 Tweak debug printing to display y-mxmn per viz 2023-03-10 18:20:22 -05:00
Tyler Goodlet fc98d66ffc Fix curve up-sampling on `'r'` hotkey
Previously when very zoomed out and using the `'r'` hotkey the
interaction handler loop wouldn't trigger a re-(up)sampling to get
a more detailed curve graphic and instead the previous downsampled
(under-detailed) graphic would show. Fix that by ensuring we yield back
to the Qt event loop and do at least a couple render cycles with paired
`.interact_graphics_cycle()` calls.

Further this flips the `.start/signal_ic()` methods to use the new
`.reset_graphics_caches()` ctr-mngr method.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 57d56c4791 Facepalm: set `Viz.ViewState.yrange` even on cache hits.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7e6e04b7e2 Drop remaining usage of `ChartPlotWidget.default_view()`
Instead delegate directly to `Viz.default_view()` throughout charting
startup and interaction handlers.

Also add a `ChartPlotWidget.reset_graphics_caches()` context mngr which
resets all managed graphics object's cacheing modes on enter and
restores them on exit for simplified use in interaction handling code.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 12bee716c2 Add `do_min_bars: bool` flag to `Viz.default_view()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 6690bd4576 Drop remaining non-usage of `ChartPlotWidget.maxmin()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 9c8bd9b8ce Expand mxmn view y-margins back to 0.06 2023-03-10 18:20:22 -05:00
Tyler Goodlet eea850450a Handle yrange not set on view vase for vlm fsp plot 2023-03-10 18:20:22 -05:00
Tyler Goodlet f7dfe57090 Disable coordinate caching during interaction
This finally seems to mitigate all the "smearing" and "jitter" artifacts
when using Qt's "coordinate cache" graphics-mode:

- whenever we're in a mouse interaction (as per calls to
  `ChartView.start/signal_ic()`) we simply disable the caching mode (set
  `.NoCache` until the interaction is complete.
- only do this (for now) during a pan since it doesn't seem to be an
  issue when zooming?
- ensure disabling all `Viz.graphics` and `.ds_graphics` to be agnostic
  to any case where there's both a zoom and a pan simultaneously (not
  that it's easy to do manually XD) as well as solving the problem
  whenever an OHLC series is in traced-and-downsampled mode (during low
  zoom).

Impl deatz:
- rename `ChartView._ic` -> `._in_interact: trio.Event`
- add `.ChartView._interact_stack: ExitStack` which  we use to open.
  and close the `FlowGraphics.reset_cache()` mngrs from mouse handlers.
- drop all the commented per-subtype overrides for `.cache_mode: int`.
- write up much better doc strings for `FlattenedOHLC` and `StepCurve`
  including some very basic ASCII-art diagrams.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 9b960594aa Add per-chart `Viz`/overlay graphics iterator method 2023-03-10 18:20:22 -05:00
Tyler Goodlet 75642929e3 Move cache-reset ctx mngr to parent type: `FlowGraphics.reset_cache()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet eda283f059 Fix focal min calc after switching to `Viz.datums_range()`.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 77401a94fb Simplify `FlowGraphics.x_last()` logics 2023-03-10 18:20:22 -05:00
Tyler Goodlet 75807f4a96 Rename overlay technique var to `method` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 94f0ef13ef Repair x-label datetime labels when in array-index mode 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7579863068 Skip overlay handling when `N < 2` are detected 2023-03-10 18:20:22 -05:00
Tyler Goodlet 993bb47138 Drop passing overlay method from viewbox to view-mode handler 2023-03-10 18:20:22 -05:00
Tyler Goodlet 8c392fda60 Drop a bunch of commented/uneeded cruft 2023-03-10 18:20:22 -05:00
Tyler Goodlet 45e97dd4c8 Solve a final minor-should-rescale edge case
When the minor has the same scaling as the major in a given direction we
should still do back-scaling against the major-target and previous
minors to avoid strange edge cases where only the target-major might not
be shifted correctly to show an matched intersect point? More or less
this just meant making the y-mxmn checks interval-inclusive with
`>=`/`<=` operators.

Also adds a shite ton of detailed comments throughout the pin-to-target
method blocks and moves the final major y-range call outside the final
`scaled: dict` loop.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 01ea706644 Better doc string, use `Viz.vs: ViewState` 2023-03-10 18:20:22 -05:00
Tyler Goodlet c1ea8552ac Back-rescale previous (minor) curves from latest
For the "pin to target major/target curve" overlay method, this finally
solves the longstanding issue of ensuring that any new minor curve,
which requires and increase in the major/target curve y-range, also
re-scales all previously scaled minor curves retroactively. Thus we now
guarantee that all minor curves are correctly "pinned" to their
target/major on their earliest available datum **and** are all kept in
view.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 6601dea8cc Support "pin-to-target-curve" overlay method again
Yah yah, i know it's the same as before (the N > 2 curves case with
out-of-range-minor rescaling the previously scaled curves isn't fixed
yet...) but, this is a much better and optional implementation in less
code. Further we're now better leveraging various new cached properties
and methods on `Viz`.

We now handle different `overlay_technique: str` options using `match:`
syntax in the 2ndary scaling loop, stash the returns scalars per curve
in `overlay_table`, and store and iterate the curves by dispersion
measure sort order.

Further wrt "pin-to-target-curve" mode, which currently still pins to
the largest measured dispersion curve in the overlay set:

- pop major Ci overlay table entries at start for sub-calcs usage when
  handling the "minor requires major rescale after pin" case.
- (finally) correctly rescale the major curve y-mxmn to whatever the
  latest minor overlay curve by calcing the inverse transform from the
  minor *at that point*:
  - the intersect point being that which the minor has starts support on
    the major's x-domain* using the new `Viz.scalars_from_index()` and,
  - checking that the minor is not out of range (versus what the major's
    transform calcs it to be, in which case,
  - calc the inverse transform from the current out-of-range minor and
    use it to project the new y-mxmn for the major/target based on the
    same intersect-reference point in the x-domain used by the minor.
  - always handle the target-major Ci specially by only setting the
    `mx_ymn` / `mx_ymn` value when iterating that entry in the overlay
    table.
  - add todos around also doing the last sub-sub bullet for all previously
    major-transform scaled minor overlays (this is coming next..i hope).
- add a final 3rd overlay loop which goes through a final `scaled: dict`
  to apply all range values to each view; this is where we will
  eventually solve that last edge case of an out-of-range minor's
  scaling needing to be used to rescale already scaled minors XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 4d11c5c89c Add cached dispersion methods to `Viz`
In an effort to make overlay calcs cleaner and leverage caching of view
range -> dispersion measures, this adds the following new methods:

- `._dispersion()` an lru cached returns scalar calculator given input
  y-range and y-ref values.
- `.disp_from_range()` which calls the above method and returns variable
  output depending on requested calc `method: str`.
- `.i_from_t()` a currently unused cached method for slicing the
  in-view's array index from time stamp (though not working yet due to
  needing to parameterize the cache by the input `.vs.xrange`).

Further refinements/adjustments:
- rename `.view_state: ViewState` -> `.vs`.
- drop the `.bars_range()` method as it's no longer used anywhere else
  in the code base.
- always set the `ViewState.in_view: np.ndarray` inside `.read()`.
- return the start array index (from slice) and `yref` value @ `xref`
  from `.scalars_from_index()` to aid with "pin to curve" rescaling
  caused by out-of-range pinned-minor curves.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 29418e9655 Avoid index-from-time slicing including gaps
Not sure why this was ever allowed but, for slicing to the sample
*before* whatever target time stamp is passed in we should definitely
not return the prior index as for the slice start since that might
include a very large gap prior to whatever sample is scanned to have
the earliest matching time stamp.

This was essential to fixing overlay intersect points searching in our
``ui.view_mode`` machinery..
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8fd5c67f2a Drop last lingering usage of `Viz.bars_range()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 62e0889bf5 Add `Viz.view_state: ViewState`
Adds a small struct which is used to track the most recently viewed
data's x/y ranges as well as the last `Viz.read()` "in view" array data
for fast access by chart related graphics processing code, namely view
mode overlay handling.

Also adds new `Viz` interfaces:
- `Viz.ds_yrange: tuple[float, float]' which replaces the previous
  `.yrange` (now set by `.datums_range()` on manual y-range calcs) so
  that the m4 downsampler can set this field specifically and then it
  get used (when available) by `Viz.maxmin()`.
- `Viz.scalars_from_index()` a new returns-scalar generator which can be
  used to calc the up and down returns values (used for scaling overlay
  y-ranges) from an input `xref` x-domain index which maps to some
  `Ci(xref) = yref`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 26690b061b Make slow chart a teensie bit smaller 2023-03-10 18:20:22 -05:00
Tyler Goodlet 98b7d78476 Drop (now) unused major curve mx/mn variables 2023-03-10 18:20:22 -05:00
Tyler Goodlet 35c40e825a Move overlay transform logic to new `.ui.view_mode`
It was getting waayy to long to be jammed in a method XD

This moves all the chart-viz iteration and transform logic into a new
`piker.ui.view_mode.overlay_viewlists()` core routine which will make it
a lot nicer for,

- AOT compilation via `numba` / `cython` / `mypyc`.
- decoupling from the `pyqtgraph.ViewBox` APIs if we ever decide to get
  crazy and go without another graphics engine.
- keeping your head clear when trying to rework the code B)
2023-03-10 18:20:22 -05:00
Tyler Goodlet 753e991dae Adjust `.ui` modules to new set-style "optional" annots 2023-03-10 18:20:22 -05:00
Tyler Goodlet 54ecb0990f Remove vlm chart again, drop lotsa fsp cruft 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5f470d6122 Rework overlay pin technique: "align to first"
As part of solving a final bullet-issue in #455, which is specifically
a case:
- with N > 2 curves, one of which is the "major" dispersion curve" and
  the others are "minors",
- we can run into a scenario where some minor curve which gets pinned to
  the major (due to the original "pinning technique" -> "align to
  major") at some `P(t)` which is *not* the major's minimum / maximum
  due to the minor having a smaller/shorter support and thus,
- requires that in order to show then max/min on the minor curve we have
  to expand the range of the major curve as well but,
- that also means any previously scaled (to the major) minor curves need
  to be adjusted as well or they'll not be pinned to the major the same
  way!

I originally was trying to avoid doing the recursive iteration back
through all previously scaled minor curves and instead decided to try
implementing the "per side" curve dispersion detection (as was
originally attempted when first starting this work). The idea is to
decide which curve's up or down "swing in % returns" would determine the
global y-range *on that side*. Turns out I stumbled on the "align to
first" technique in the process: "for each overlay curve we align its
earliest sample (in time) to the same level of the earliest such sample
for whatever is deemed the major (directionally disperse) curve in
view".

I decided (with help) that this "pin to first" approach/style is equally
as useful and maybe often more so when wanting to view support-disjoint
time series:

- instead of compressing the y-range on "longer series which have lesser
  sigma" to make whatever "shorter but larger-sigma series" pin to it at
  an intersect time step, this instead will expand the price ranges
  based on the earliest time step in each series.
- the output global-returns-overlay-range for any N-set of series is equal to
  the same in the previous "pin to intersect time" technique.
- the only time this technique seems less useful is for overlaying
  market feeds which have the same destination asset but different
  source assets (eg. btceur and btcusd on the same chart since if one
  of the series is shorter it will always be aligned to the earliest
  datum on the longer instead of more naturally to the intersect sample
  level as was in the previous approach).

As such I'm going to keep this technique as discovered and will later
add back optional support for the "align to intersect" approach from
previous (which will again require detecting the highest dispersion
curve direction-agnostic) and pin all minors to the price level at which
they start on the major.

Further details of the implementation rework in
`.interact_graphics_cycle()` include:

- add `intersect_from_longer()` to detect and deliver a common datum
  from 2 series which are different in length: the first time-index
  sample in the longer.
- Rewrite the drafted `OverlayT` to only compute (inversed log-returns)
  transforms for a single direction and use 2 instances, one for each
  direction inside the `Viz`-overlay iteration loop.
- do all dispersion-per-side major curve detection in the first pass of
  all `Viz`s on a plot, instead updating the `OverlayT` instances for
  each side and compensating for any length mismatch and
  rescale-to-minor cases in each loop cycle.
2023-03-10 18:20:22 -05:00
Tyler Goodlet d5ba26cfaf Try to hide all axes even when removed 2023-03-10 18:20:22 -05:00
Tyler Goodlet cb5e2d48e2 Add hack-zone UI REPL access via `ctl-u` 2023-03-10 18:20:22 -05:00
Tyler Goodlet a6d1053c50 Facepalm, align overlay plot view exactly to parent
Previously we were aligning the child's `PlotItem` to the "root" (top
most) overlays `ViewBox`..smh. This is why there was a weird gap on the
LHS next to the 'left' price axes: something weird in the implied axes
offsets was getting jammed in that rect.

Also comments out "the-skipping-of" moving axes from the overlay's
`PlotItem.layout` to the root's linear layout(s) when an overlay's axis
is read as not visible; this isn't really necessary nor useful and if we
want to remove the axes entirely we should do it explicitly and/or
provide a way through the `ComposeGridLayout` API.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 3dc1f66ff6 Go back to caching on all curves
Despite there being artifacts when interacting, the speedups when
cross-hair-ing are just too good to ignore. We can always play with
disabling caches when interaction takes place much like we do with feed
pausing.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 091afccb72 Dynamically adjust y-range margin in display loop
When zoomed in alot, and thus a quote driven y-range resize takes place,
it makes more sense to increase the `range_margin: float` input to
`._set_yrange()` to ensure all L1 labels stay in view; generally the
more zoomed in,
- the smaller the y-range is and thus the larger the needed margin (on
  that range's dispersion diff) would be,
- it's more likely to get a last datum move outside the previous range.

Also, always do overlayT style scaling on the slow chart whenever it
treads.
2023-03-10 18:20:22 -05:00
Tyler Goodlet cda3bcc1f6 Expose `._set_yrange()` kwargs via `yrange_kwargs: dict`
Since it can be desirable to dynamically adjust inputs to the y-ranging
method (such as in the display loop when a chart is very zoomed in), this
adds such support through a new `yrange_kwargs: dict[Viz, dict]` which
replaces the `yrange` tuple we were passing through prior. Also, adjusts
the y-range margin back to the original 0.09 of the diff now that we can
support dynamic control.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 2d7359851f Go back to no-cache on OHLC downsample line 2023-03-10 18:20:22 -05:00
Tyler Goodlet db1e0a04f8 Only use last `ChartView._yrange` if set 2023-03-10 18:20:22 -05:00
Tyler Goodlet 972b723a5d Skip overlay transform calcs on common-pi curves
If there is a common `PlotItem` used for a set of `Viz`/curves (on
a given view) we don't need to do overlay scaling and thus can also
short circuit the viz iteration loop early.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 74c215d5b2 Lel, always meant to no-cache the step curve.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet c646b435bf Incrementally set vlm chart yrange per quote 2023-03-10 18:20:22 -05:00
Tyler Goodlet 0a939311fe Only set the specific view's yrange per quote
Somewhat of a facepalm but, for incremental update of the auto-yrange
from quotes in the display loop obviously we only want to update the
associated `Viz`/viewbox for *that* fqsn. Further we don't need to worry
about the whole "tick margin" stuff since `._set_yrange()` already adds
margin to the yrange by default; thus we remove all of that.
2023-03-10 18:20:22 -05:00
Tyler Goodlet a7db6adc2e Always set the `ChartView._viz` for each plot 2023-03-10 18:20:22 -05:00
Tyler Goodlet c57567ab0d No-overlays, y-ranging optimizations
When the caller passes `do_overlay_scaling=False` we skip the given
chart's `Viz` iteration loop, and set the yrange immediately, then
continue to the next chart (if `do_linked_charts` is set) instead of
a `continue` short circuit within the viz sub-loop.

Deats:
- add a `_maybe_calc_yrange()` helper which makes the `yranges`
  provided-or-not case logic more terse (factored).
- add a `do_linked_charts=False` short circuit.
- drop the legacy commented swing % calcs stuff.
- use the `ChartView._viz` when `do_overlay_scaling=False` thus
  presuming that we want to handle the viz mapped to *this* view box.
- add a `._yrange` "last set yrange" tracking var which keeps record of
  the last ymn/ymx value set in `._set_yrange()` BEFORE doing range
  margins; this will be used for incremental update in the display loop.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 3daee0caa9 Disable overlay scaling on per-symbol-feed updates
Since each symbol's feed is multiplexed by quote key (an fqsn), we can
avoid scaling overlay curves on any single update, presuming each quote
driven cycle will trigger **only** the specific symbol's curve.

Also disables fsp `.interact_graphics_cycle()` calls for now since it
seems they aren't really that critical to and we should be using the
same technique as above (doing incremental y-range checks/updates) for
FSPs as well.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 6ea64a7d2e Iterate all charts (widgets) when only one overlay
The reason (fsp) subcharts were not linked-updating correctly was
because of the early termination of the interact update loop when only
one "overlay" (aka no other overlays then the main curve) is detected.
Obviously in this case we still need to iterate all linked charts in the
set (presuming the user doesn't disable this).

Also tweaks a few internals:
- rename `start_datums: dict` -> `overlay_table`.
- compact all "single curve" checks to one logic block.
- don't collect curve info into the `overlay_table: dict` when
  `do_overlay_scaling=True`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 25cf8df367 Pass windowed y-mxmn to `.interact_graphics_cycle()` calls in display loop 2023-03-10 18:20:22 -05:00
Tyler Goodlet 91d41ebf76 Allow y-range input via a `yranges: dict[Viz, tuple[float, float]]` 2023-03-10 18:20:22 -05:00
Tyler Goodlet c690e141e1 Don't unset `Viz.render` for unit vlm
Such that we still y-range auto-sort inside
`ChartView.interact_graphics_cycle()` still runs on the unit vlm axis
and we always size such that the y-label stays in view.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 2ed43c0758 Fix profiler f-string 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7a83a7288c Update profile msgs to new apis 2023-03-10 18:20:22 -05:00
Tyler Goodlet 9930f25ad3 Move axis hiding into `.overlay_plotitem()`
Since we pretty much always want the 'bottom' and any side that is not
declared by the caller move the axis hides into this method. Lets us
drop the same calls in `.ui._fsp` and `._display`.

This also disables the auto-ranging back-linking for now since it
doesn't seem to be working quite yet?
2023-03-10 18:20:22 -05:00
Tyler Goodlet 5dd69b2295 Better handle dynamic registry sampler broadcasts
In situations where clients are (dynamically) subscribing *while*
broadcasts are starting to taking place we need to handle the
`set`-modified-during-iteration case. This scenario seems to be more
common during races on concurrent startup of multiple symbols. The
solution here is to use another set to take note of subscribers which
are successfully sent-to and then skipping them on re-try.

This also contains an attempt to exception-handle throttled stream
overruns caused by higher frequency feeds (like binance) pushing more
quotes then can be handled during (UI) client startup.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 246d07021e Drop old loop and wait on fsp engine tasks startups 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7ebcd6d734 Comment out all median usage, turns out it's uneeded.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5a8fd42c0c Lul, actually scaled main chart from linked set
This was a subtle logic error when building the `plots: dict` we weren't
adding the "main (ohlc or other source) chart" from the `LinkedSplits`
set when interacting with some sub-chart from `.subplots`..

Further this tries out bypassing `numpy.median()` altogether by just
using `median = (ymx - ymn) / 2` which should be nearly the same?
2023-03-10 18:20:22 -05:00
Tyler Goodlet 517c68f3ad Use `._pathops.slice_from_time()` for overlay intersects
It's way faster since it uses a uniform time arithmetic to narrow the
`numpy.searchsorted()` range before actually doing the index search B)
2023-03-10 18:20:22 -05:00
Tyler Goodlet ea84505682 Don't scale overlays on linked from display loop
In the (incrementally updated) display loop we have range logic that is
incrementally updated in real-time by streams, as such we don't really
need to update all linked chart's (for any given, currently updated
chart) y-ranges on calls of each separate (sub-)chart's
`ChartView.interact_graphics_cycle()`. In practise there are plenty of
cases where resizing in one chart (say the vlm fsps sub-plot) requires
a y-range re-calc but not in the OHLC price chart. Therefore
we always avoid doing more resizing then necessary despite it resulting
in potentially more method call overhead (which will later be justified
by better leveraging incrementally updated `Viz.maxmin()` and
`media_from_range()` calcs).
2023-03-10 18:20:22 -05:00
Tyler Goodlet 5eaca18ee0 Don't skip overlay scaling in disp-loop for now 2023-03-10 18:20:22 -05:00
Tyler Goodlet e06d4b405d Add linked charts guard-flag for use in display loop 2023-03-10 18:20:22 -05:00
Tyler Goodlet cf67c790e5 Use new cached median method in overlay scaling
Massively speeds up scaling transform cycles (duh).

Also includes a draft for an "overlay transform" type/api; obviously
still a WIP 🏄..
2023-03-10 18:20:22 -05:00
Tyler Goodlet ec8679ad74 Add `Viz.median_from_range()`
A super snappy `numpy.median()` calculator (per input range) which we
slap an `lru_cache` on thanks to handy dunder method hacks for such
things on mutable types XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 9418f53244 Speed up ranging in display loop
use the new `do_overlay_scaling: bool` since we know each feed will have
its own updates (cuz multiplexed by feed..) and we can avoid
ranging/scaling overlays that will make their own calls.

Also, pass in the last datum "brighter" color for ohlc curves as it was
originally (and now that we can pass that styling bit through).
2023-03-10 18:20:22 -05:00
Tyler Goodlet 497174c687 Add full profiling to `.interact_graphics_cycle()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 481f1b3d7e Fix intersect detection using time indexing
Facepalm, obviously absolute array indexes are not going to necessarily
align vs. time over multiple feeds/history. Instead use
`np.searchsorted()` on whatever curve has the smallest support and find
the appropriate index of intersection in time so that alignment always
starts at a sensible reference.

Also adds a `debug_print: bool` input arg which can enable all the
prints when working on this.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 776ffd2b1c Factor curve-dispersion sorting into primary loop
We can determine the major curve (in view) in the first pass of all
`Viz`s so drop the 2nd loop and thus the `mxmn_groups: dict`. Also
simplifies logic for the case of only one (the major) curve in view.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 896259d9e4 When only one curve is in view, skip group ranging 2023-03-10 18:20:22 -05:00
Tyler Goodlet 89e2e7fc54 Adjust `.update_graphics()` to expect `in_view: bool` in `_fsp.py` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 32f21dc06b Drop `update_graphics_from_flow()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet a0fb84f55b Just warn log on bad intersect indexing errors (for now) 2023-03-10 18:20:22 -05:00
Tyler Goodlet fc6ccc306c Only set the major curve's range once (per render cycle)
Turns out this is a limitation of the `ViewBox.setYRange()` api: you
can't call it more then once and expect anything but the first call to
be applied without letting a render cycle run. As such, we wait until
the end of the log-linear scaling loop to finally apply the major curves
y-mx/mn after all minor curves have been evaluated.

This also drops all the debug prints (for now) to get a feel for latency
in production mode.
2023-03-10 18:20:22 -05:00
Tyler Goodlet c2dd255e8a Only remove axis from scene when in one 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7e421ba57b Drop `.group_maxmin()`
We ended up doing groups maxmin sorting at the interaction layer (new
the view box) and thus this method is no longer needed, though it was
the reference for the code now in `ChartView.interact_graphics_cycle()`.

Further this adds a `remove_axes: bool` arg to `.insert_plotitem()`
which can be used to drop axis entries from the inserted pi (though it
doesn't seem like we really ever need that?) and does the removal in
a separate loop to avoid removing axes before they are registered in
`ComposedGridLayout._pi2axes`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 0591cb09f6 Clean up cross-curve intersect point indexing
When there are `N`-curves we need to consider the smallest
x-data-support subset when figuring out for each major-minor pair such
that the "shorter" series is always returns aligned to the longer one.

This makes the var naming more explicit with `major/minor_i_start` as
well as clarifies more stringently a bunch of other variables and
explicitly uses the `minor_y_intersect` y value in the scaling transform
calcs. Also fixes some debug prints.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 052ce65682 3rdz the charm: log-linearize minor y-ranges to a major
In very close manner to the original (gut instinct) attempt, this
properly (y-axis-vertically) aligns and scales overlaid curves according
to what we are calling a "log-linearized y-range multi-plot" B)

The basic idea is that a simple returns measure (eg. `R = (p1 - p0)
/ p0`) applied to all curves gives a constant output `R` no matter the
price co-domain in use and thus gives a constant returns over all assets
in view styled scaling; a intuitive visual of returns correlation. The
reference point is for now the left-most point in view (or highest
common index available to all curves), though we can make this
a parameter based on user needs.

A slew of debug `print()`s are left in for now until we iron out the
remaining edge cases to do with re-scaling a major (dispersion) curve
based on a minor now requiring a larger log-linear y-range from that
previous major' range.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 52ac1053aa 2nd try: dispersion normalize y-ranges around median
In the dispersion swing calcs, use the series median from the in-view
data to determine swing proportions to apply on each "minor curve"
(series with lesser dispersion the one with the greatest). Track the
major `Viz` as before by max dispersion. Apply the dispersion swing
proportions to each minor curve-series in a third loop/pass of all
overlay groups: this ensures all overlays are dispersion normalized in
their ranges but, minor curves are currently (vertically) centered (vs.
the major) via their medians.

There is a ton of commented code from attempts to try and vertically
align minor curves to the major via the "first datum" in-view/available.
This still needs work and we may want to offer it as optional.

Also adds logic to allow skipping margin adjustments in `._set_yrange()`
if you pass `range_margin=None`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet dfc35253ea First draft, group y-minmax transform algo
On overlaid ohlc vizs we compute the largest max/min spread and
apply that maxmimum "up and down swing" proportion to each `Viz`'s
viewbox in the group.

We obviously still need to clip to the shortest x-range so that
it doesn't look exactly the same as before XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8a5b9f4e8c Rename `.maybe_downsample_graphics()` -> `.interact_graphics_cycle()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet f89e11fc7d Right, handle y-ranging multiple paths per plot
We were hacking this before using the whole `ChartView._maxmin()`
setting stuff since in some cases you might want similarly ranged paths
on the same view, but of course you need to max/min them together..

This adds that group sorting by using a table of `dict[PlotItem,
tuple[float, float]` and taking the abs highest/lowest value for each
plot in the viz interaction update loop.

Also removes the now commented signal registry calls and thus
`._yranger`, drops the `set_range: bool` from `._set_yrange` and adds
and extra `.maybe_downsample_graphics()` to the mouse wheel handler to
avoid a weird slow debounce where ds-ing is delayed until a further
interaction.
2023-03-10 18:20:22 -05:00
Tyler Goodlet fc73becd5f Drop Qt interaction signal usage
It's kind of hard to understand with the C++ fan-out to multiple views
(imo a cluster-f#$*&) and seems honestly just plain faster to loop (in
python) through all the linked view handlers XD

Core adjustments:
- make the panning and wheel-scroll handlers just call
  `.maybe_downsample_graphics()` directly; drop all signal emissions.
- make `.maybe_downsample_graphics()` loop through all vizs per subchart
  and use the new pipeline-style call sequence of:
  - `Viz.update_graphics() -> <read_slc>: tuple`
  - `Viz.maxmin(i_read_range=<read_slc>) -> yrange: tuple`
  - `Viz.plot.vb._set_yrange(yrange=yrange)`
  which inlines all the necessary calls in the most efficient way whilst
  leveraging `.maxmin()` caching and ymxmn-from-m4-during-render to
  boot.
- drop registering `._set_yrange()` for handling `.sigRangeChangedManually`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 223e9d999c Add first-draft `PlotItemOverlay.group_maxmin()`
Computes the maxmin values for each underlying plot's in-view range as
well as the max up/down swing (in percentage terms) from the plot with
most dispersion and returns a all these values plus a `dict` of plots to
their ranges as part of output.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 12883c3c90 Don't double send `enable_modules` and `debug_mode` in kwargs..
This broke non-disti-mode actor tree spawn / runtime, seemingly because
the cli entrypoint for a `piker chart` also sends these values down
through the call stack independently? Pretty sure we don't need to send
the `enable_modules` from the chart actor anyway.
2023-03-10 10:30:26 -05:00
Tyler Goodlet 44a3115539 Expose `drop_root_perms_for_ahab` from `pikerd` factories to `ahabd` 2023-03-09 18:42:24 -05:00
Tyler Goodlet 0772b4a0fa Hard code version from our container, predicate renaming 2023-03-09 18:42:24 -05:00
Tyler Goodlet 15064d94cb `ahabd`: Harden cancellation teardown (again XD)
Needed to move the startup sequence inside the `try:` block to guarantee
we always do the (now shielded) `.cancel()` call if we get a cancel
during startup.

Also, support an optional `started_afunc` field in the config if
backends want to just provide a one-off blocking async func to sync
container startup. Add a `drop_root_perms: bool` to allow persisting
sudo perms for testing or dyanmic container spawning purposes.
2023-03-09 18:42:24 -05:00
Tyler Goodlet 79b0db4449 Pass a config `tmp_dir: Path` to the runtime when testing 2023-03-09 15:37:43 -05:00
Tyler Goodlet 5aaa7f47dc Pull testing config dir from `tractor` runtime vars
Provides a more correct solution (particularly for distributed testing)
to override the `piker` configuration directory by reading the path from
a specific `tractor._state._runtime_vars` entry that can be provided by
the test harness.

Also fix some typing and comments.
2023-03-09 15:37:43 -05:00
Tyler Goodlet aa36abf36e Support passing `tractor` "actor runtime vars" down the runtime 2023-03-09 15:37:43 -05:00
Tyler Goodlet 75b7a8b56e `marketstore`: Pull default socket from server config 2023-03-09 15:37:43 -05:00
Tyler Goodlet 31392af427 Move enabled module defs to just above where used 2023-03-09 15:37:43 -05:00
Tyler Goodlet 6540c415c1 Lul, fix imports in elasticsearch block.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet 6f92c6b52d Don't crash on a `xdotool` timeout.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet 441243f83b Attempt to report `piker storage -d <fqsn>` errors
Not really sure there's much we can do besides dump Grpc stuff when we
detect an "error" `str` for the moment..

Either way leave a buncha complaints (como siempre) and do linting
fixups..
2023-03-09 15:37:43 -05:00
Tyler Goodlet cec2967071 Import `maybe_open_pikerd` at module level 2023-03-09 15:37:43 -05:00
Tyler Goodlet f95ea19b21 Move `pikerd` runtime boostrap to `.service._actor_runtime` 2023-03-09 15:37:43 -05:00
Tyler Goodlet eca048c0c5 Move daemon spawning endpoints to `service._daemon` module 2023-03-09 15:37:43 -05:00
Tyler Goodlet a2d40937a3 Move actor-discovery utils to `.service._registry 2023-03-09 15:37:42 -05:00
Tyler Goodlet 31f2b01c3e Move `Services` api to `.service._mngr` mod 2023-03-09 15:37:42 -05:00
Tyler Goodlet b226b678e9 Fix missed `marketstore` mod imports 2023-03-09 15:37:42 -05:00
Tyler Goodlet dd87d1142e Bump mkts timeout to 2s 2023-03-09 15:37:42 -05:00
Tyler Goodlet afac553ea2 Move all docker and external db code to `piker.service` 2023-03-09 15:37:42 -05:00
Tyler Goodlet 93c81fa4d1 Start `piker.service` sub-package
For now just moves everything that was in `piker._daemon` to a subpkg
module but a reorg is coming pronto!
2023-03-09 15:37:42 -05:00
Tyler Goodlet bfe3ea1f59 Set explicit `marketstore` container startup timeout 2023-03-09 15:37:42 -05:00
Tyler Goodlet 56629b6b2e Hardcode `cancel` log level for `ahabd` for now 2023-03-09 15:37:42 -05:00
Tyler Goodlet bb723abc9d Always passthrough loglevel to `ahabd` supervisor 2023-03-09 15:37:42 -05:00
Tyler Goodlet 7694419e71 Background docker-container logs processing
Previously we would make the `ahabd` supervisor-actor sync to docker
container startup using pseudo-blocking log message processing.

This has issues,
- we're forced to do a hacky "yield back to `trio`" in order to be
  "fake async" when reading the log stream and further,
- blocking on a message is fragile and often slow.

Instead, run the log processor in a background task and in the parent
task poll for the container to be in the client list using a similar
pseudo-async poll pattern. This allows the super to `Context.started()`
sooner (when the container is actually registered as "up") and thus
unblock its (remote) caller faster whilst still doing full log msg
proxying!

Deatz:
- adds `Container.cuid: str` a unique container id for logging.
- correctly proxy through the `loglevel: str` from `pikerd` caller task.
- shield around `Container.cancel()` in the teardown block and use
  cancel level logging in that method.
2023-03-09 15:37:42 -05:00
Tyler Goodlet b078a06621 Doc string and types bump in loggin mod 2023-03-09 15:37:42 -05:00
Tyler Goodlet 05b67c27d0 Apply `Services` runtime state **immediately** inside starup block 2023-03-09 15:37:42 -05:00
Tyler Goodlet 8c66f066bd Deliver es specific ahab-super in endpoint startup config 2023-03-09 15:37:42 -05:00
Tyler Goodlet 959e423849 Add warning around detach flag to docker client 2023-03-09 15:37:42 -05:00
Tyler Goodlet 7b196b1b97 Support startup-config overrides to `ahabd` super
With the addition of a new `elastixsearch` docker support in
https://github.com/pikers/piker/pull/464, adjustments were made
to container startup sync logic (particularly the `trio` checkpoint
sleep period - which itself is a hack around a sync client api) which
caused a regression in upstream startup logic wherein container error
logs were not being bubbled up correctly causing a silent failure mode:

- `marketstore` container started with corrupt input config
- `ahabd` super code timed out on startup phase due to a larger log
  polling period, skipped processing startup logs from the container,
  and continued on as though the container was started
- history client fails on grpc connection with no clear error on why the
  connection failed.

Here we revert to the old poll period (1ms) to avoid any more silent
failures and further extend supervisor control through a configuration
override mechanism. To address the underlying design issue, this patch
adds support for container-endpoint-callbacks to override supervisor
startup configuration parameters via the 2nd value in their returned
tuple: the already delivered configuration `dict` value.

The current exposed values include:
    {
        'startup_timeout': 1.0,
        'startup_query_period': 0.001,
        'log_msg_key': 'msg',
    },

This allows for container specific control over the startup-sync query
period (the hack mentioned above)  as well as the expected log msg key
and of course the startup timeout.
2023-03-09 15:37:42 -05:00
Tyler Goodlet fe0695fb7b First draft storage layer cli
Adds a `piker storage` subcmd with a `-d` flag to wipe a particular
fqsn's time series (both 1s and 60s). Obviously this needs to be
extended much more but provides a start point.
2023-03-09 15:37:42 -05:00
Tyler Goodlet aba238e8b1 `kraken`: expect `Pair` in search results.. 2023-03-08 17:22:34 -05:00
Tyler Goodlet d3192bb8c2 Read `Symbol` tick precision fields when no entry in `.broker_info` 2023-03-08 17:22:34 -05:00
Tyler Goodlet daa6a5c80a `ib`: restore and (maybe) use `xdotool` + `i3ipc` reset method
Since apparently the container we were using is totally borked on new
kernels and/or latest jvm, this move our old manual local-X-desktop script
back for use in `brokerd` backend code.

Adds a new `.brokers.ib._util` which contains the 2 methods and fails
over to this one when we can't connect to a VNC server. Also adjusts the
original in `scripts/ib_data_reset.py` to import and run the module code
as a script-program.
2023-03-03 17:37:26 -05:00
Tyler Goodlet b4a1cc8f22 `kraken`: parse and load info `Transaction.sym: Symbol`
Also includes a retyping of `Client._pair: dict[str, Pair]` to look up
pair structs and map all alt-key-name-sets to each for easy precision
info lookup to set the `.sym` field for each transaction including for
on-chain transfers which kraken provides as an "asset decimals" field,
presumably pulled from the particular block-token's limitation info.
2023-03-02 19:25:43 -05:00
Tyler Goodlet 69b85aa7e5 `ib`: parse and load info for new `Transaction.sym: Symbol` field 2023-03-02 19:23:47 -05:00
Tyler Goodlet 3a4794e9d1 Backward-compat: don't require `'lot_tick_size'`
In order to support existing `pps.toml` files in the wild which don't
have the `asset_type, price_tick_size, lot_tick_size` fields, we need to
only optionally read them and instead expect that backends will write
the fields going forward (coming in follow patches).

Further this makes some small asset-size (vlm accounting) quantization
related adjustments:
- rename `Symbol.decimal_quant()` -> `.quantize_size()` since that is
  explicitly what this method is doing.
- and expect an input `size: float` which we cast to decimal instead of
  doing it inside the `.calc_size()` caller code.
- drop `Symbol.iterfqsns()` which wasn't being used anywhere at all..

Additionally, this drafts out a new replacement market-trading-pair data
type to eventually replace `.data._source.Symbol` -> `MktPair` which we
aren't using yet, but serves as the documentation-driven motivator ;)
and, it relates to https://github.com/pikers/piker/issues/467.
2023-03-02 19:22:19 -05:00
Guillermo Rodriguez 6be96a96aa
Drop symbol section on Position serialization 2023-03-01 21:06:52 -03:00
Guillermo Rodriguez d704b153ca
Fix mayor bug found by fomo, sym info getting stored incorrectly on pps.toml causing it to load pp wrong on second open, also fix header leak bug 2023-03-01 21:06:52 -03:00
Guillermo Rodriguez 20d91f5e06
Good catch by j, unnecesary kwarg on open_pps 2023-03-01 21:06:52 -03:00
Guillermo Rodriguez 6c23c79f2a
Minor fixes after fomo's review 2023-03-01 21:06:52 -03:00
Guillermo Rodriguez f5b8b9a14f
Add sym registry to PaperBoi as well as a sym ref on Transaction
Add decimal quantize API to Symbol to simplify by-broker truncation
Add symbol info to `pps.toml`
Move _assert call to outside the _async_main context manager
Minor indentation and styling changes, also convert a few prints to log calls
Fix multi write / race condition on open_pps call
Switch open_pps to not write by default
Fix integer math kraken syminfo _tick_size initialization
2023-03-01 21:06:48 -03:00
Guillermo Rodriguez dc78994dcf
Fixed float dust bug on zero position case 2023-03-01 21:05:37 -03:00
Tyler Goodlet 569df45d18 `kraken.`: drop trade history query limit 2023-03-01 17:40:36 -05:00
Tyler Goodlet f53f4df583 `ib/kraken`: adjust to new default of not-writing in `open_pps()` 2023-03-01 17:40:33 -05:00
jaredgoldman 41f81eb701 Make write on exit default false 2023-02-28 14:14:05 -05:00
jaredgoldman e4e368923d Add specific kwarg key to open_pps call when starting paperboi 2023-02-28 13:51:47 -05:00
jaredgoldman 342aec648b Skip zero test and change use Path when creating a config folder in marketstore 2023-02-28 13:51:47 -05:00
jaredgoldman 55253c8469 Remove whitespace and correct typo 2023-02-28 13:51:47 -05:00
jaredgoldman 4b72d3ba99 Add backpressure setting back as it wasn't altering test behaviour 2023-02-28 13:51:47 -05:00
jaredgoldman 61296bbdfc Minor formatting, removing whitespace 2023-02-28 13:51:47 -05:00
jaredgoldman 36f466fff8 Ensure tests are running and working up until asserting pps 2023-02-28 13:51:47 -05:00
jaredgoldman fcd8b8eb78 Remove breaking call to load pps from ledger 2023-02-25 18:59:40 -05:00
jaredgoldman 3e83764b5b Remove whitespace, uneeded comments 2023-02-25 18:59:40 -05:00
jaredgoldman 3a6fbabaf8 Minor formatting 2023-02-25 18:59:40 -05:00
jaredgoldman 4c2e776e01 Ensure to cleanup by passing fixture in paper_test signature 2023-02-25 18:59:40 -05:00
jaredgoldman 1e748f11ef Ensure config path is being updated with _testing correctly during testing 2023-02-25 18:59:40 -05:00
jaredgoldman 2d25d1f048 Push failing assert no pps test 2023-02-25 18:59:40 -05:00
jaredgoldman e54d928405 Reformat fake fill in paper engine,
Ensure tests pass, refactor test wrapper
2023-02-25 18:59:40 -05:00
jaredgoldman c99381216d Ensure actual pp is sent to ems
ensure not to write pp header on startup

Comment out pytest settings
Add comments explaining delete_testing_dir fixture
use nonlocal instead of global for test state

Add unpacking get_fqsn method
Format test_paper
Add comments explaining sync/async book.send calls
2023-02-25 18:59:40 -05:00
algorandpa db2e2ed78f Use constants value for test config dir path 2023-02-25 18:59:39 -05:00
algorandpa 3bc54e308f Use Path.mkdir instead of os.mkdir 2023-02-25 18:59:39 -05:00
algorandpa 8c9c165e0a Remove broken import 2023-02-25 18:59:39 -05:00
algorandpa 7bd8019876 Add back cleanup fixture 2023-02-25 18:59:39 -05:00
algorandpa 0dec2b9c89 Enable backpressure during data-feed layer startup to avoid overruns 2023-02-25 18:59:39 -05:00
algorandpa acc86ae6db more formatting 2023-02-25 18:59:39 -05:00
algorandpa 730906a072 Minor formatting 2023-02-25 18:59:39 -05:00
algorandpa e5cefeb44b Format to prep for PR 2023-02-25 18:59:39 -05:00
algorandpa 7142a6a7ca Add hacky cleanup solution for _testng data 2023-02-25 18:59:39 -05:00
algorandpa b180602a3e Make config grab _testing dir in pytest env,
- Remove print statements
2023-02-25 18:59:39 -05:00
algorandpa 95b9dacb7a Break test into steps 2023-02-25 18:59:39 -05:00
algorandpa 68a196218b force change branch name 2023-02-25 18:59:39 -05:00
algorandpa 84cd1e0059 initial commit on copy 2023-02-25 18:59:39 -05:00
algorandpa 86b4386522 minor changes, prepare for rebase of overlays branch 2023-02-25 18:59:39 -05:00
algorandpa 5bb93ccc5f change id to 'piker-paper' 2023-02-25 18:59:39 -05:00
algorandpa 3028a8b1f8 restore spacing 2023-02-25 18:59:39 -05:00
algorandpa 6126c4f438 restore spacing 2023-02-25 18:59:39 -05:00
algorandpa 41bb0445e0 remove unnecessary return 2023-02-25 18:59:39 -05:00
algorandpa 97627a4976 remove more logs 2023-02-25 18:59:39 -05:00
algorandpa 1b2fce430f remove logs, unused args 2023-02-25 18:59:39 -05:00
algorandpa 8cd2354d73 ensure that paper pps are pulled on open 2023-02-25 18:59:39 -05:00
algorandpa 9c28d7086e Add Generator as return type of open_trade_ledger 2023-02-25 18:59:39 -05:00
algorandpa a4bd51a01b change open_trade_ledger typing to return a Generator type 2023-02-25 18:59:39 -05:00
algorandpa b67d020e23 add basic func to load paper_trades file 2023-02-25 18:59:39 -05:00
Guillermo Rodriguez 85a1b858b4
Fix logging on emsd 2023-02-25 20:56:25 -03:00
Guillermo Rodriguez 47bf45f30e
Merge pull request #464 from pikers/elasticsearch_integration
Elasticsearch integration
2023-02-24 16:38:37 -03:00
Esmeralda Gallardo b96e2c314a
Minor style changes and removed unnecesary comments 2023-02-24 15:11:15 -03:00
Esmeralda Gallardo f96d6a04b6
Fixed UnboundLocalError on _ahab. Added test for marketstore's initialization 2023-02-22 13:28:07 -03:00
Guillermo Rodriguez acc6249d88
Remove unnesesary arguments to some pikerd functions, fix container init error
by switching from log reading to quering es health endpoint, fix install on ci
and add more logging.
2023-02-21 20:45:10 -03:00
Tyler Goodlet d0d1554d74 Expose `emsd` task loglevel through to clients 2023-02-21 12:02:01 -05:00
Esmeralda Gallardo b5cdf14036
Modified elasticsearch file name to 'elastic' to avoid name errors. Applied changes suggested in the pr. 2023-02-21 13:34:29 -03:00
Esmeralda Gallardo 3ce8bfa012
Moved database initialization code inside the open_pikerd context manager 2023-02-21 13:34:29 -03:00
Guillermo Rodriguez bf9ca4a4a8
Generalize ahab to support elasticsearch logs and init procedure 2023-02-21 13:34:29 -03:00
Guillermo Rodriguez 17a4fe4b2f
Trim unnecesary stuff left from marketstore copy, also fix elastic config name for docker build, add elasticsearch to dependencies 2023-02-21 13:34:28 -03:00
Esmeralda Gallardo 0dc24bd475
Added dockerfile, yaml file and script to statrt an elasticsearch's docker instance. 2023-02-21 13:34:26 -03:00
Tyler Goodlet 2bad692703 Fix up some test warnings (summary) spots 2023-02-21 10:54:18 -05:00
Tyler Goodlet e01220af14 Type annot tweaks to feeds mod 2023-02-21 10:54:18 -05:00
goodboy 139b8ba0f4
Merge pull request #453 from pikers/overlays_interaction_latency_tuning
Overlays interaction latency tuning
2023-02-14 13:48:12 -05:00
Guillermo Rodriguez ffd707db62
Add try catch for when notify-send is not present on system 2023-02-13 18:08:56 -03:00
Tyler Goodlet fefb0de51f Don't update overlays as fsps 2023-02-13 12:27:58 -05:00
Tyler Goodlet 59f34c94b0 Return fast on bad range in `.default_view()` 2023-02-13 12:27:58 -05:00
Tyler Goodlet ebf53e32bd Fix return type annot for `slice_from_time()` 2023-02-13 12:27:58 -05:00
Tyler Goodlet 9ce52033f0 Fix `do_px_step` output for epoch step sizing 2023-02-13 12:27:58 -05:00
Tyler Goodlet 9876f200c1 Support chart draw-api-kwargs-passthrough in lined plot meths 2023-02-13 12:27:58 -05:00
Tyler Goodlet 81b8cd5461 Use normal pen when last-datum color not provided 2023-02-13 12:27:58 -05:00
Tyler Goodlet 731eb91a58 Make profiler work when nested and not? 2023-02-13 12:27:58 -05:00
Tyler Goodlet 49ca743e6a Add back `.prepareGeometryChange()`, seems faster? 2023-02-13 12:27:58 -05:00
Tyler Goodlet a36d4b1dc6 Factor color and cache mode settings into `FlowGraphics`
Curve-path colouring and cache mode settings are used (and can thus be
factored out of) all child types; this moves them into the parent type's
`.__init__()` and adjusts all sub-types match:

- the bulk was moved out of the `Curve.__init__()` including all
  previous commentary around cache settings.
- adjust `BarItems` to use a `NoCache` mode and instead use the
  `last_step_pen: pg.Pen` and `._pen` inside it's `.pain()` instead of
  defining functionally duplicate vars.
- adjust all (transitive) calls to `BarItems` to use the new kwargs
  names.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 33df4f9927 Return `in_view: bool` from `Viz.update_graphics()`
Allows callers to know if they should care about a particular viz
rendering call by immediately knowing if the graphics are in view. This
turns out super useful particularly when doing dynamic y-ranging overlay
calcs.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 72a9af21ac Fix profiler f-strings 2023-02-13 12:27:58 -05:00
Tyler Goodlet 1a10514cad Disable coordinate caching on OHLC ds curves to avoid smearing 2023-02-13 12:27:58 -05:00
Tyler Goodlet 5d9b7c72b3 Fix `Viz.draw_last()` to divide by `.flat_index_ratio` for uppx index lookback 2023-02-13 12:27:58 -05:00
Tyler Goodlet efddd43760 Drop masked `._maxmin()` override code from fsp stuff 2023-02-13 12:27:58 -05:00
Tyler Goodlet 1606b3a9c3 Document `Viz.incr_info()` outputs 2023-02-13 12:27:58 -05:00
Tyler Goodlet 8b5b1c214b Rework display loop maxmin-ing with `Viz` pipelining
First, we rename what was `chart_maxmin()` -> `multi_maxmin()` and don't
`partial` it in to the `DisplayState`, just call it with correct `Viz`
ref inputs.

Second, as we've done with `ChartView.maybe_downsample_graphics()` use
the output from the main `Viz.update_graphics()` and feed it to the
`.maxmin()` calls for the ohlc and vlm chart but still deliver the same
output signature as prior. Also accept and use an optional profiler
input, drop `DisplayState.maxmin()` and add `.vlm_viz`.

Further perf related tweak to do with more efficient incremental
updates:
- only call `multi_maxmin()` if the main fast chart viz does a pixel
  column step.
- mask out hist viz and vlm viz and all linked fsp `._set_yrange()`
  calls for now until we figure out how to best optimize these updates
  when considering the new group-scaled-by-% style for multicharts.
- drop `.enable_auto_yrange()` calls during startup.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 9780263cfa Adjust vlm fsp code to new `Viz.update_graphics()` output sig 2023-02-13 12:27:58 -05:00
Tyler Goodlet e1e3afb495 Support read-slice input to `Viz.maxmin()`
Acts as short cut when pipe-lining from `Viz.update_graphics()` (which
now returns the needed in-view array-relative-read-slice as output) such
that `Viz.read()` and `.datums_range()` doesn't need to be called
internally multiple times. In this case where `i_read_range` is provided
we of course skip doing time index translations and consequently lookup
the appropriate (epoch-time) index indices for caching.
2023-02-13 12:27:58 -05:00
Tyler Goodlet f9eb880404 Backlink subchart views to "main chart" in `.add_plot()` 2023-02-13 12:27:58 -05:00
Tyler Goodlet a3bbbeda9d Drop `ChartView._maxmin()` usage in `.ui._fsp`
Removes the multi-maxmin usage as well as ensures appropriate `Viz` refs
are passed into the view methods now requiring it. Also drops the "back
linking" of the vlm chart view to the source OHLC chart since we're
going to add this as a default to the charting API.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 3ad7844fdf Drop `ChartView._maxmin()` idea, use `Viz.maxmin()`
The max min for a given data range is defined on the lowest level
through the `Viz` api intermingling it with the view is a layering
issue. Instead make `._set_yrange()` call the appropriate view's viz
(since they should be one-to-one) directly and thus avoid any callback
monkey patching nonsense.

Requires that we now make `._set_yrange()` require either one of an
explicit `yrange: tuple[float, float]` min/max pair or the `Viz` ref (so
that maxmin can be called) as input. Adjust
`enable/disable_auto_yrange()` to bind in a new `._yranger()` partial
that's (solely) needed for signal reg/unreg which binds in the now
required input `Viz` to these methods.

Comment the `autoscale_overlays` block in `.maybe_downsample_graphics()`
for now until we figure out the most sane way to auto-range all linked
overlays and subplots (with their own overlays).
2023-02-13 12:27:58 -05:00
Tyler Goodlet b71c61e23f More thoroughly profile the display loop 2023-02-13 12:27:58 -05:00
Tyler Goodlet 9650b32786 Use `Viz.draw_last()` inside `.update_graphics()`
In an effort to ensure uniform and uppx-optimized last datum graphics
updates call this method directly instead of the equivalent graphics
object thus ensuring we only update the last pixel column according with
the appropriate max/min computed from the last uppx's worth of data.

Fixes / improvements to enable `.draw_last()` usage include,
- change `Viz._render_table` -> `._alt_r: tuple[Renderer, pg.GraphicsItem] | None`
  which holds an alternative (usually downsampled) render and graphics
  obj.
- extend the `.draw_last()` signature to include:
  - `last_read` to allow passing in the already read data from
    `.update_graphics()`, if it isn't passed then a manual read is done
    internally.
  - `reset_cache: bool` which is passed through to the graphics obj.
- use the new `Formatter.flat_index_ratio: float` when indexing into xy
  1d data to compute the max/min for that px column.

Other,
- drop `bars_range` input from `maxmin()` since it's unused.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 433697cc4f Add cached refs to last 1d xy outputs
For the purposes of avoiding another full format call we can stash the
last rendered 1d xy pre-graphics formats as
`IncrementalFormatter.x/y_1d: np.ndarray`s and allow readers in the viz
and render machinery to use this data easily for things like "only
drawing the last uppx's worth of data as a line". Also add
a `.flat_index_ratio: float` which can be used similarly as a scalar
applied to indexes into the src array but instead when indexing
(flattened) 1d xy formatted outputs. Finally, this drops the way
overdone/noisy `.__repr__()` meth we had XD
2023-02-13 12:27:58 -05:00
Tyler Goodlet d622b4157c Only draw up to 2nd last datum for OHLC bars paths 2023-02-13 12:27:58 -05:00
Tyler Goodlet 1add591b2c Only update last datum graphic(s) on clear ticks
When a new tick comes in but no new time step / bar is yet needed (to be
appended) we can simply adjust **only** the last bar datum
lines-graphic(s) to avoid a redraw of the preceding `QPainterPath` on
every tick. Do this by calling `Viz.draw_last()` on the fast and slow
chart and adjusting the guards around calls to `Viz.update_graphics()`
(which *does* update paths) to only enter when there's a `do_px_step`
condition. We can stop calling `main_viz.plot.vb._set_yrange()` on view
treading cases since the range should have already been adjusted by the
clearing-tick processing mxmn updates.

Further this changes,
- the `chart_maxmin()` helper (which we should eventually just get rid
  of) to take bound in `Viz`s for the ohlc and vlm chart instead of the
  chart widget handles.
- extend the guard around hist viz yranging to only enter when not in
  "axis mode" - the same as for the fast viz.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 60440bc6b7 Ensure full hist OHLC path is drawn on tread
Since we removed the `Viz.update_graphics()` call from the main rt loop
we have to be sure to call it in the history chart incr-loop to avoid
a gap between the  last bar and prior history since startup. We only
need to update on tread since that should be the only time a full redraw
is ever necessary, ow only the last datum is needed.

Further this moves the graphics cycle func's profiler init to the top in
an effort to get more correct latency measures.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 4003729231 Use `Viz.update_graphics()` throughout remainder of graphics loop where possible 2023-02-13 12:27:58 -05:00
Tyler Goodlet 934b32c342 Use `Viz` over charts where possible in display loop
Since `ChartPlotWidget.update_graphics_from_flow()` is more or less just
a call to `Viz.update_graphics()` try to call that directly where
possible.

Changes include:
- calling the viz in the display state specific `maxmin()`.
- passing a viz instance to each `ChartView._set_yrange()` call (in prep
  of explicit group auto-ranging); not that this input is unused in the
  method for now.
- drop `bars_range` var passing since we don't use it.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 97bb3b48da Set a `PlotItem.viz` for interaction lookup
Inside `._interaction` routines we need access to `Viz` instances.
Instead of doing `CharPlotWidget._vizs: dict` lookups this ensures each
plot can lookup it's (parent) viz without error.

Also, adjusts `Viz.maxmin()` output parsing to new signature.
2023-02-13 12:27:58 -05:00
Tyler Goodlet da618e1d38 Always cache `read_slc` alongside y-mnmx values 2023-02-13 12:27:58 -05:00
Tyler Goodlet 23c03a0905 Add back coord-caching to ohlc graphic 2023-02-13 12:27:58 -05:00
Tyler Goodlet 07c8ed8a3a Use (modern) literal type annots in view code 2023-02-13 12:27:58 -05:00
Tyler Goodlet bcf2a9868d Drop x-range query from `ChartPlotWidget.maxmin()`
Move the `Viz.datums_range()` call into `Viz.maxmin()` itself thus
minimizing the chart `.maxmin()` method to an ultra light wrapper around
the viz call. Also move all profiling into the `Viz` method.

Adjust `Viz.maxmin()` to return both the (rounded) x-range values which
correspond to the range containing the y-domain min and max so that
it can be used for up and coming overlay group maxmin calcs.
2023-02-13 12:27:58 -05:00
Tyler Goodlet c09c3925a4 Drop multi mxmn from display mod 2023-02-13 12:27:58 -05:00
Tyler Goodlet 92ce1b3304 Only handle hist discrepancies when market is open
We obviously don't want to be debugging a sample-index issue if/when the
market for the asset is closed (since we'll be guaranteed to have
a mismatch, lul). Pass in the `feed_is_live: trio.Event` throughout the
backfilling routines to allow first checking for the live feed being active
so as to avoid breakpointing on false +ves. Also, add a detailed warning
log message for when *actually* investigating a mismatch.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 0fc06a98d4 Passthrough `tractor` kwargs directly 2023-02-13 12:27:58 -05:00
Tyler Goodlet 4ba99494f0 Fix `open_trade_ledger()` enter value type annot 2023-02-13 12:27:58 -05:00
Tyler Goodlet a8e1796a8b Comment bad x-range bp for now 2023-02-13 12:27:58 -05:00
Tyler Goodlet 5ced05aab0 Breakpoint bad (-ve or too large) x-ranges to m4
This should never really happen but when it does it appears to be a race
with writing startup pre-graphics-formatter array data where we get
`x_end` epoch value subtracting some really small offset value (like
`-/+0.5`) or the opposite where the `x_start` is epoch and `x_end` is
small.

This adds a warning msg and `breakpoint()` as well as guards around the
entire code downsampling code path so that when resumed the downsampling
cycle should just be skipped and avoid a crash.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 4a6339ffc2 Downthrottle to 16Hz on multi-feed charts 2023-02-13 12:27:58 -05:00
Tyler Goodlet efa4089920 Attempt to keep selected item highlighted
This attempt was unsuccessful since trying to (re)select the last
highlighted item on both an "enter" or "click" of that item causes
a hang and then segfault in `Qt`; no clue why..

Adds a `keep_current_item_selected: bool` flag to
`CompleterView.show_cache_entries()` but using it seems to always cause
a hang and crash; we keep all potential use spots commented for now
obviously to avoid this. Also included is a bunch of tidying to logic
blocks in the kb-control loop for readability.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 35cc37ddc1 Lol, pull hist chart from the display state 2023-02-13 12:27:58 -05:00
Tyler Goodlet 5ea4be1d4b Make (cache) search-results a `set` and avoid overlay duplicate entries 2023-02-13 12:27:58 -05:00
Tyler Goodlet 0c5b5a5aea Take outer-interval values in `Viz.datums_range()` 2023-02-13 12:27:58 -05:00
Tyler Goodlet 4027d683e9 Clean a buncha cruft from render mod 2023-02-13 12:27:58 -05:00
Tyler Goodlet 7afc9301ac Handle last-in-view time slicing edge case
Whenever the last datum is in view `slice_from_time()` need to always
spec the final array index (i.e. the len - 1 value we set as
`read_i_max`) to avoid a uniform-step arithmetic error where gaps in the
underlying time series causes an index that's too low to be returned.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 12c6d58c2a Drop bp blocks from formatters mod 2023-02-13 12:27:58 -05:00
Tyler Goodlet c5db7295e6 Fix query-mode cursor labels to work with epoch-indexing 2023-02-13 12:27:58 -05:00
Tyler Goodlet 02c3ea1743 Use `open_sample_stream()` in display loop 2023-02-13 12:27:58 -05:00
Tyler Goodlet 63f0567418 Drop `Flume.index_stream()`, `._sampling.open_sample_stream()` replaces it 2023-02-13 12:27:58 -05:00
Tyler Goodlet 3e17e52555 Add back another panes resize during startup 2023-02-13 12:27:58 -05:00
Tyler Goodlet 65dca16dc0 Always zero-on-step $vlm 2023-02-13 12:27:58 -05:00
Tyler Goodlet e742d18a6c Mouse interaction tweaks
- adjust zoom focal to be min of the view-right coord or the right-most
  point on the flow graphic in view and drop all the legacy l1-in-view
  focal point cruft.
- flip to not auto-scaling overlays by default.
- change the `._set_yrange()` margin to `0.09`.
- drop `use_vr: bool` usage.
2023-02-13 12:27:58 -05:00
Tyler Goodlet 7e29c36a24 Modernize optional path variable type annots 2023-02-13 12:27:58 -05:00
Tyler Goodlet 4d2b5c8f86 Use `Curve.x_last()` for zoom focal point 2023-02-13 12:27:58 -05:00
Tyler Goodlet fe932a96a9 Make `PlotItemOverlay` add items inwards->out
Before this axes were being stacked from the outside in (for `'right'`
and 'bottom'` axes) which is somewhat non-intuitive for an `.append()`
operation. As such this change makes a symbol list stack a set of
`'right'` axes from left-to-right.

Details:
- rename `ComposeGridLayout.items` -> `.pitems`
- return `(int, list[AxisItem])` pairs from `.insert/append_plotitem()`
  and the down stream `PlotItemOverlay.add_plotitem()`.
- drop `PlotItemOverlay.overlays` and add it back as `@property` around
  the underlying `.layout.pitems`.
2023-02-13 12:27:58 -05:00
Tyler Goodlet c1b7063e3c Drop the legacy `relayed_from` cruft from our view box 2023-02-13 12:27:58 -05:00
goodboy 42d2f9e461
Merge pull request #452 from pikers/l1_compaction
Compact L1 labels
2023-02-13 11:21:26 -05:00
Tyler Goodlet 1346c33f04 `kraken`: make pps work with arbitrary deposits
Factor and fix dst <- src pair parsing into a new func
`get_likely_pair()` and use throughout initial position loading; solves
a parsing bug for src asset balances which aren't only 3 chars long..
a terrible assumption.
2023-02-12 15:52:48 -05:00
Tyler Goodlet cee6321a9f Do full marker width after line 2023-02-12 15:38:43 -05:00
Tyler Goodlet 1abed2ad9e Fix indent level 2023-02-12 15:38:43 -05:00
Tyler Goodlet 5bd6fa3cbf Make $vlm axis color same as clears 2023-02-12 15:38:43 -05:00
Tyler Goodlet a82911d8a9 Correctly load order mode for first fqsn in overlay set 2023-02-12 15:38:43 -05:00
Tyler Goodlet dc88364253 Move $vlm y-axis to LHS 2023-02-12 15:38:43 -05:00
Tyler Goodlet 4c51a68691 Better index step value scanning by checking with our expected set 2023-02-12 15:38:43 -05:00
Tyler Goodlet 42d3537516 Repair auto-y-ranging to always include L1 spread
Goes back to always adjusting the y-axis range to include the L1 spread
and clearing label in view whenever the last datum is also in view,
previously this was broken after reworking the display loop for
multi-feeds.

Drops a bunch of old commented tick looping cruft from before we started
using tick-type framing. Also adds more stringent guards for ignoring
but error logging quote values that are more then 25% out of range; it
seems particularly our `ib` feed has some issues with strange `price`
values that are way off here and there?
2023-02-12 15:38:43 -05:00
Tyler Goodlet 3fd394d693 Use static `L1Label._x_br_offset` as l1 label length 2023-02-12 15:38:43 -05:00
Tyler Goodlet a7a08aced9 Drop l1 labels attr from chart widget 2023-02-12 15:38:43 -05:00
Tyler Goodlet 1d83fdb510 Handle empty `indexes` input edge case.. 2023-02-12 15:38:43 -05:00
Tyler Goodlet 924fcca463 TOSQUASH: 84f19308 (l1 rework) 2023-02-12 15:38:43 -05:00
Tyler Goodlet 26f497e2bb Set cursor label color to "bracket" 2023-02-12 15:38:43 -05:00
Tyler Goodlet e37e118a7e Don't set y-axis label colors to curve's, use the default from global scheme 2023-02-12 15:38:43 -05:00
Tyler Goodlet b2bb7f4923 Simplify L1 labels for multicharts
Instead of having the l1 lines be inside the view space, move them to be
inside their respective axis (with only a 16 unit portion inside the
view) such that the clear price label can overlay with them nicely
without obscuring; this is much better suited to multiple adjacent
y-axes and in general is simpler and less noisy.

Further `L1Labels` + `LevelLabel` style tweaks:
- adjust `.rect` positioning to be "right" (i.e. inside the parent
  y-axis) with a slight 16 unit shift toward the viewbox (using the new
  `._x_br_offset`) to allow seeing each level label's line even when the
  clearing price label is positioned at that same level.
- add a newline's worth of vertical space to each of the bid/ask labels
  so that L1 labels' text content isn't ever obscured by the clear price
  label.
- set a low (10) z-value to ensure l1 labels are always placed
  underneath the clear price label.
- always fill the label rect with the chosen background color.
- make labels fully opaque so as to always make them hide the parent
  axes' `.tickStrings()` contents.
- make default color the "default" from the global scheme.
- drop the "price" part from the l1 label text contents, just show the
  book-queue's amount (in dst asset's units, aka the potential clearing vlm).
2023-02-12 15:38:43 -05:00
Tyler Goodlet 97b03bbfbb Move old label sizing cruft to label mod 2023-02-12 15:38:43 -05:00
Tyler Goodlet 340045af77 Make `FlowGraphic.x_last()` be optionally `None`
In the case where the last-datum-graphic hasn't been created yet, simply
return a `None` from this method so the caller can choose to ignore the
output. Further, drop `.px_width()` since it makes more sense defined on
`Viz` as well as the previously commented `BarItems.x_uppx()` method.
Also, don't round the `.x_uppx()` output since it can then be used when
< 1 to do x-domain scaling during high zoom usage.
2023-02-12 13:55:26 -05:00
Tyler Goodlet c1988c4d8d Add a parent-type for graphics: `FlowGraphic`
Factor some common methods into the parent type:
- `.x_uppx()` for reading the horizontal units-per-pixel.
- `.x_last()` for reading the "closest to y-axis" last datum coordinate
  for zooming "around" during mouse interaction.
- `.px_width()` for computing the max width of any curve in view in
  pixels.

Adjust all previous derived `pg.GraphicsObject` child types to now
inherit from this new parent and in particular enable proper `.x_uppx()`
support to `BarItems`.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 6a0c36922e Drop `._index_step` from formatters and instead defer to `Viz.index_step()` 2023-02-12 13:55:26 -05:00
Tyler Goodlet 459cbfdbad Further fixes `Viz.default_view()` and `.index_step()`
Use proper uppx scaling when either of scaling the data to the x-domain
index-range or when the uppx is < 1 (now that we support it) such that
both the fast and slow chart always appropriately scale and offset to
the y-axis with the last datum graphic just adjacent to the order line
arrow markers.

Further this fixes the `.index_step()` calc to use the "earliest" 16
values to compute the expected sample step diff since the last set often
contained gaps due to start up race conditions and generated
unexpected/incorrect output.

Further this drops the `.curve_width_pxs()` method and replaces it with
`.px_width()`, taken from the graphics object API and instead returns
the pixel account for the whole view width instead of the
x-domain-data-range within the view.
2023-02-12 13:55:26 -05:00
Tyler Goodlet fc17187ff4 Drop edge case from `slice_from_time()`
Doesn't seem like we really need to handle the situation where the start
or stop input time stamps are outside the index range of the data since
the new binary search handling via `numpy.searchsorted()` covers this
case at minimal runtime cost and with an equally correct output. Allows
us to drop some other indexing endpoint internal variables as well.
2023-02-12 13:55:26 -05:00
Tyler Goodlet a7d78a3f40 Use left-style index search on RHS scan as well 2023-02-12 13:55:26 -05:00
Tyler Goodlet 7ce3f10e73 Just-offset-from-arrow-marker on slow chart
We want the fast and slow chart to behave the same on calls to
`Viz.default_view()` so adjust the offset calc to make both work:
- just offset by the line len regardless of step / uppx
- add back the `should_line: bool` output from `render_bar_items()` (and
  use it to set a new `ds_allowed: bool` guard variable) so that we can
  bypass calling the m4 downsampler unless the bars have been switched
  to the interpolation line graphic (which we normally required before
  any downsampling of OHLC graphics data).

Further, this drops use of the `use_vr: bool` flag from all rendering
since we pretty much always use it by default.
2023-02-12 13:55:26 -05:00
Tyler Goodlet bfc6014ad3 Fix history array name 2023-02-12 13:55:26 -05:00
Tyler Goodlet a5eed8fc1e Fix x-axis labelling when using an epoch domain
Previously with array-int indexing we had to map the input x-domain
"indexes" passed to `DynamicDateAxis._indexes_to_timestr()`. In the
epoch-time indexing case we obviously don't need to lookup time stamps
from the underlying shm array and can instead just cast to `int` and
relay the values verbatim.

Further, this patch includes some style adjustments to `AxisLabel` to
better enable multi-feed chart overlays by avoiding L1 label clutter
when multiple y-axes are stacked adjacent:
- adjust the `Axis` typical max string to include a couple spaces suffix
 providing for a bit more margin between side-by-side y-axes.
- make the default label (fill) color the "default" from the global
 color scheme and drop it's opacity to .9
- add some new label placement options and use them in the
 `.boundingRect()` method:
 * `._x/y_br_offset` for relatively shifting the overall label relative
   to it's parent axis.
 * `._y_txt_h_scaling` for increasing the bounding rect's height
   without including more whitespace in the label's text content.
- ensure labels have a high z-value such that by default they are always
 placed "on top" such that when we adjust the l1 labels they can be set
 to a lower value and thus never obscure the last-price label.
2023-02-12 13:55:26 -05:00
Tyler Goodlet cdec4782f0 Add commented append slice-len sanity check 2023-02-12 13:55:26 -05:00
Tyler Goodlet f30a48b82c Use `np.diff()` on last 16 samples instead of only last datum pair 2023-02-12 13:55:26 -05:00
Tyler Goodlet 98de22a740 Enable the experimental `QPrivatePath` functionality from latest `pyqtgraph` 2023-02-12 13:55:26 -05:00
Tyler Goodlet efbb8e86d4 Fix overlayed slow chart "treading"
Turns out we were updating the wrong ``Viz``/``DisplayState`` inside the
closure style `increment_history_view()`` (probably due to looping
through the flumes and dynamically closing in that task-func).. Instead
define the history incrementer at module level and pass in the
`DisplayState` explicitly. Further rework the `DisplayState` attrs to be
more focused around the `Viz` associated with the fast and slow chart
and be sure to adjust output from each `Viz.incr_info()` call to latest
update. Oh, and just tweaked the line palette for the moment.

FYI "treading" here is referring to  the x-shifting of the curve when
the last datum is in view such that on new sampled appends the "last"
datum is kept in the same x-location in UI terms.
2023-02-12 13:55:26 -05:00
Tyler Goodlet b6521498f4 Make `.increment_view()` take in a `datums: int` and always scale it by sample step size 2023-02-12 13:55:26 -05:00
Tyler Goodlet 06f1b94147 Make `Viz.incr_info()` do treading with time-index, and appending with array-index 2023-02-12 13:55:26 -05:00
Tyler Goodlet ffb57f0256 Rename `reset` -> `reset_cache` 2023-02-12 13:55:26 -05:00
Tyler Goodlet ed1f64cf43 Fix gap detection on RHS; always bin-search on overshot time range 2023-02-12 13:55:26 -05:00
Tyler Goodlet bf8ea33697 Add type annots to vars inside `Render.render()` 2023-02-12 13:55:26 -05:00
Tyler Goodlet bc17308de7 Drop coordinate cacheing from `BarItems`, causes weird jitter on pan 2023-02-12 13:55:26 -05:00
Tyler Goodlet 1ece704d6e Add `ChartPlotWidget.main_viz: Viz` convenience `@property` 2023-02-12 13:55:26 -05:00
Tyler Goodlet dea1c1c2d6 Make `Viz.incr_info()` sample rate agnostic
Mainly it was the global (should we )increment logic that needs to be
independent for the fast vs. slow chart such that the slow isn't
update-shifted by the fast and vice versa. We do this using a new
`'i_last_slow'` key in the `DisplayState.globalz: dict` which is
singleton for each sample-rate-specific chart and works for both time
and array indexing.

Also, we drop some old commented `graphics.draw_last_datum()` code that
never ended up being needed again inside the coordinate cache reset
bloc.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 3300a240c6 Use array-`int`-indexing on single feed
Might as well since it makes the chart look less gappy and we can easily
flip the index switch now B)

Also adds a new `'i_slow_last'` key to `DisplayState` for a singleton
across all slow charts and thus no more need for special case logic in
`viz.incr_info()`.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 50ef4efccb Align step curves the same as OHLC bars 2023-02-12 13:55:26 -05:00
Tyler Goodlet 51f2461e8b Add `IncrementalFormatter.x_offset: np.ndarray`
Define the x-domain coords "offset" (determining the curve graphics
per-datum placement) for each formatter such that there's only on place
to change it when needed. Obviously each graphics type has it's own
dimensionality and this is reflected by the array shapes on each
subtype.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 444768d30f Adjust OHLC bar x-offsets to be time span matched
Previously we were drawing with the middle of the bar on each index with
arms to either side: +/- some arm length. Instead this changes so that
each bar is drawn *after* each index/timestamp such that in graphics
coords the bar span more correctly matches the time span in the
x-domain. This makes the linked region between slow and fast chart
directly match (without any transform) for epoch-time indexing such that
the last x-coord in view on the fast chart is no more then the
next time step in (downsampled) slow view.

Deats:
- adjust in `._pathops.path_arrays_from_ohlc()` and take an `bar_w` bar
  width input (normally taken from the data step size).
- change `.ui._ohlc.bar_from_ohlc_row()` and
  `BarItems.draw_last_datum()` to match.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 0d0675ac7e `Viz._index_field` a `typing.Literal[str]` 2023-02-12 13:55:26 -05:00
Tyler Goodlet 24b384f3ef Set `path_arrays_from_ohlc(use_time_index=True)` on epoch indexing
Allows easily switching between normal array `int` indexing and time
indexing by just flipping the `Viz._index_field: str`.

Also, guard all the x-data audit breakpoints with a time indexing
condition.
2023-02-12 13:55:26 -05:00
Tyler Goodlet 93330954c2 Ugh, use `bool` flag to determine index field.. 2023-02-12 13:55:26 -05:00
Tyler Goodlet edf721f755 Make `LinearRegion` link using epoch-time index
Turned out to be super simple to get the first draft to work since the
fast and slow chart now use the same domain, however, it seems like
maybe there's an offset issue still where the fast may be a couple
minutes ahead of the slow?

Need to dig in a bit..
2023-02-12 13:55:26 -05:00
Tyler Goodlet 530b2731ba Add global `i_step` per overlay to `DisplayState`
Using a global "last index step" (via module var) obviously has problems
when working with multiple feed sets in a single global app instance:
any separate feed-set will be incremented according to an app-global
index-step and thus won't correctly calc per-feed-set-step update info.

Impl deatz:
- drop `DisplayState.incr_info()` (since previously moved to `Viz`) and
  call that method on each appropriate `Viz` instance where necessary;
  further ensure the appropriate `DisplayState` instance is passed in to
  each call and make sure to pass a `state: DisplayState`.
- add `DisplayState.hist_vars: dict` for history chart (sets) to
  determine the per-feed (not set) current slow chart (time) step.
- add `DisplayState.globalz: dict` to house a common per-feed-set state
  and use it inside the new `Viz.incr_info()` such that
  a `should_increment: bool` can be returned and used by the display
  loop to determine whether to x-shift the current chart.
2023-02-12 13:55:24 -05:00
Tyler Goodlet 14104185d2 Move `DisplayState.incr_info()` -> `Viz` 2023-02-12 13:41:18 -05:00
Tyler Goodlet 3019c35e30 Move `Viz` layer to new `.ui` mod 2023-02-12 13:41:18 -05:00
Tyler Goodlet 4d74bc29b4 Fix line -> bars on 6x UPPX
Read the `Viz.index_step()` directly to avoid always reading 1 on the
slow chart; this was completely broken before and resulting in not
rendering the bars graphic on the slow chart until at a true uppx of
1 which obviously doesn't work for 60 width bars XD

Further cleanups to `._render` module:
- drop `array` output from `Renderer.render()`, `read_from_key` input
  and fix type annot.
- drop `should_line`, `changed_to_line` and `render_kwargs` from
  `render_baritems()` outputs and instead calc `should_redraw` logic
  inside the func body and return as output.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 3638ae8d3e Drop unused `read_src_from_key: bool` to `.format_to_1d()` 2023-02-12 13:41:18 -05:00
Tyler Goodlet c5dd67e63c Right, do index lookup for int-index as well.. 2023-02-12 13:41:18 -05:00
Tyler Goodlet 0663880a6d Fix formatter xy ndarray first prepend case
First allocation vs. first "prepend" of source data to an xy `ndarray`
format **must be mutex** in order to avoid a double prepend.

Previously when both blocks were executed we'd end up with
a `.xy_nd_start` that was decremented (at least) twice as much as it
should be on the first `.format_to_1d()` call which is obviously
incorrect (and causes problems for m4 downsampling as discussed below).
Further, since the underlying `ShmArray` buffer indexing is managed
(i.e. write-updated) completely independently from the incremental
formatter updates and internal xy indexing, we can't use
`ShmArray._first.value` and instead need to use the particular `.diff()`
output's prepend length value to decrement the `.xy_nd_start` on updates
after initial alloc.

Problems this resolves with m4:
- m4 uses a x-domain diff to calculate the number of "frames" to
  downsample to, this is normally based on the ratio of pixel columns on
  screen vs. the size of the input xy data.
- previously using an int-index (not epoch time) the max diff between
  first and last index would be the size of the input buffer and thus
  would never cause a large mem allocation issue (though it may have
  been inefficient in terms of needed size).
- with an epoch time index this max diff could explode if you had some
  near-now epoch time stamp **minus** an x-allocation value: generally
  some value in `[0.5, -0.5]` which would result in a massive frames and
  thus internal `np.ndarray()` allocation causing either a crash in
  `numba` code or actual system mem over allocation.

Further, put in some more x value checks that trigger breakpoints if we
detect values that caused this issue - we'll remove em after this has
been tested enough.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 3bed142d15 Handle time-indexing for fill arrows
Call into a reworked `Flume.get_index()` for both the slow and fast
chart and do time index clipping to last datum where necessary.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 9fcc6f9c44 Restore coord-cache resetting
Turns out we can't seem to avoid the artefacts when click-drag-scrolling
(results in weird repeated "smeared" curve segments) so just go back to
the original code.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 7aef31701b Add some commented debug prints for default fmtr 2023-02-12 13:41:18 -05:00
Tyler Goodlet 135627e142 Slicec to an extra index around each timestamp input 2023-02-12 13:41:18 -05:00
Tyler Goodlet 5216a6b732 Drop passing `render_data` to `Curve.draw_last_datum()` 2023-02-12 13:41:18 -05:00
Tyler Goodlet 2a797d32dc Add back `.default_view()` slice logic for `int` indexing 2023-02-12 13:41:18 -05:00
Tyler Goodlet 35a16ded2d Block out `do_print` stuff inside `Viz.maxmin()` 2023-02-12 13:41:18 -05:00
Tyler Goodlet 44f50e3d0e Implement `stop_t` gap adjustments; the good lord said it is the problem 2023-02-12 13:41:18 -05:00
Tyler Goodlet 96b871c4d7 Draw last datums on boot
Ensures that a "last datum" graphics object exists so that zooming can
read it using `.x_last()`. Also, disable the linked region stuff for now
since it's totally borked after flipping to the time indexing.
2023-02-12 13:41:18 -05:00
Tyler Goodlet d2aad74dfc Delegate to `Viz.default_view()` on chart
Also add a rage print to not forget about the global index
tracking/diffing in the display loop we still need to change.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 50209752c3 Re-implement `.default_view()` on `Viz`
Since we don't really need it defined on the "chart widget" move it to
a viz method and rework it to hell:

- always discard the invalid view l > r case.
- use the graphic's UPPX to determine UI-to-scene coordinate scaling for
  the L1-label collision detection, if there is no L1 just offset by
  a few (index step scaled) datums; this allows us to drop the 2x
  x-range calls as was hacked previous.
- handle no-data-in-view cases explicitly and error if we get any
  ostensibly impossible cases.
- expect caller to trigger a graphics cycle if needed.

Further support this includes a rework a slew of other important
details:

- add `Viz.index_step`, an idempotent computed, index (presumably uniform)
  step value which is needed for variable sample rate graphics displayed
  on an epoch (second) time index.
- rework `Viz.datums_range()` to pass view x-endpoints as first and last
  elements in return `tuple`; tighten up snap-to-data edge case logic
  using `max()`/`min()` calls and better internal var naming.
- adjust all calls to `slice_from_time()` to not expect an "abs" slice.
- drop all `.yrange` resetting since we can just have the `Renderer` do
  it when necessary.
2023-02-12 13:41:18 -05:00
Tyler Goodlet 5ab4e5493e Add gap detection for `stop_t`, though only report atm 2023-02-12 13:41:18 -05:00
Tyler Goodlet e252f70253 Add `.x_last()` meth to flow graphics 2023-02-12 13:41:18 -05:00
Tyler Goodlet 98438e29ef Drop `Flume.view_data()` 2023-02-12 13:41:18 -05:00
Tyler Goodlet d649a7d1fa Drop old breakpoint 2023-02-12 13:41:18 -05:00
Tyler Goodlet 2669ced629 Drop `_slice_from_time()` 2023-02-12 13:41:18 -05:00
Tyler Goodlet f2c0987a04 Use uniform step arithmetic in `slice_from_time()`
If we presume that time indexing using a uniform step we can calculate
the exact index (using `//`) for the input time presuming the data
set has zero gaps. This gives a massive speedup over `numpy` fancy
indexing and (naive) `numba` iteration. Further in the case where time
gaps are detected, we can use `numpy.searchsorted()` to binary search
for the nearest expected index at lower latency.

Deatz,
- comment-disable the call to the naive `numba` scan impl.
- add a optional `step: int` input (calced if not provided).
- add todos for caching binary search results in the gap detection
  cases.
- drop returning the "absolute buffer indexing" slice since the caller
  can always just use the read-relative slice to acquire it.
2023-02-12 13:41:18 -05:00
Tyler Goodlet bb84715bf0 Make `.default_view()` time step aware
When we use an epoch index and any sample rate > 1s we need to scale the
"number of bars" to that step in order to place the view correctly in
x-domain terms. For now we're calcing the step in-method but likely,
longer run, we'll pull this from elsewhere (like a ``Viz`` attr).
2023-02-12 13:41:17 -05:00
Tyler Goodlet 0bdb7261d1 Flip over to epoch-time based x-domain indexing 2023-02-12 13:41:17 -05:00
Tyler Goodlet 12857a258b Adjust all `slice_from_time()` calls to not expect mask 2023-02-12 13:41:17 -05:00
Tyler Goodlet 46808fbb89 Rewrite `slice_from_time()` using `numba`
Gives approx a 3-4x speedup using plain old iterate-with-for-loop style
though still not really happy with this .5 to 1 ms latency..

Move the core `@njit` part to a `_slice_from_time()` with a pure python
func with orig name around it. Also, drop the output `mask` array since
we can generally just use the slices in the caller to accomplish the
same input array slicing, duh..
2023-02-12 13:41:17 -05:00
Tyler Goodlet 6ca8334253 Use index (time) step to calc OHLC bar/line uppx threshold 2023-02-12 13:41:17 -05:00
Tyler Goodlet a3844f9922 Use step size to determine bar gaps 2023-02-12 13:41:17 -05:00
Tyler Goodlet 58b36db2e5 Use step size to determine last datum bar gap 2023-02-12 13:41:17 -05:00
Tyler Goodlet a33f58a61a Move `Flume.slice_from_time()` to `.data._pathops` mod func 2023-02-12 13:41:17 -05:00
Tyler Goodlet a4392696a1 Drop `index_field` input to renders, add `.read()` profiling 2023-02-12 13:41:17 -05:00
Tyler Goodlet d5844ce8ff Delegate formatter `.index_field` to the parent `Viz` 2023-02-12 13:41:17 -05:00
Tyler Goodlet bf88b40a50 Facepalm**2: fix array-read-slice, like actually..
We need to subtract the first index in the array segment read, not the
first index value in the time-sliced output, to get the correct offset
into the non-absolute (`ShmArray.array` read) array..

Further we **do** need the `&` between the advance indexing conditions
and this adds profiling to see that it is indeed real slow (like 20ms
ish even when using `np.where()`).
2023-02-12 13:41:17 -05:00
Tyler Goodlet e4a0d4ecea Markup OHLC->path gen with `numba` issue # 2023-02-12 13:41:17 -05:00
Tyler Goodlet cca3417c57 Facepalm: put graphics cycle in `do_ds: bool` block.. 2023-02-12 13:41:17 -05:00
Tyler Goodlet 031d7967de Facepalm: actually return latest index on time slice fail.. 2023-02-12 13:41:17 -05:00
Tyler Goodlet 2e67e98b4d Go with explicit `.data._m4` mod name
Since it's a notable and self-contained graphics compression algo, might
as well give it a dedicated module B)
2023-02-12 13:41:17 -05:00
Tyler Goodlet 7124a131dd Move (unused) path gen routines to `.ui._pathops` 2023-02-12 13:41:17 -05:00
Tyler Goodlet 9052ed5ddf Move qpath-ops routines back to separate mod 2023-02-12 13:41:17 -05:00
Tyler Goodlet 7ec21c7f3b Rename `.ui._pathops.py` -> `.ui._formatters.py 2023-02-12 13:41:17 -05:00
Tyler Goodlet 309ae240cf Look up "index field" in display cycles
Again, to make epoch indexing a flip-of-switch for testing look up the
`Viz.index_field: str` value when updating labels.

Also, drops the legacy tick-type set tracking which we no longer use
thanks to the new throttler subsys and it's framing msgs.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 382a619a03 Fix from-time index slicing?
Apparently we want an `|` for the advanced indexing logic?
Also, fix `read_slc` start to not always be 0 XD
2023-02-12 13:41:17 -05:00
Tyler Goodlet 7f3f6f871a Move path ops routines to top of mod
Planning to put the formatters into a new mod and aggregate all path
gen/op helpers into this module.

Further tweak include:
- moving `path_arrays_from_ohlc()` back to module level
- slice out the last xy datum for `OHLCBarsAsCurveFmtr` 1d formatting
- always copy the new x-value from the source to `.x_nd`
2023-02-12 13:41:17 -05:00
Tyler Goodlet 6ea04f850d Drop diff state tracking in formatter
This was a major cause of error (particularly trying to get epoch
indexing working) and really isn't necessary; instead just have
`.diff()` always read from the underlying source array for current
index-step diffing and append/prepend slice construction.

Allows us to,
- drop `._last_read` state management and thus usage.
- better handle startup indexing by setting `.xy_nd_start/stop` to
  `None` initially so that the first update can be done in one large
  prepend.
- better understand and document the step curve "slice back to previous
  level" logic which is now heavily commented B)
- drop all the `slice_to_head` stuff from and instead allow each
  formatter to choose it's 1d segmenting.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 3d5695f40a Explicitly enable chart widget yranging in display init 2023-02-12 13:41:17 -05:00
Tyler Goodlet 5affad942f Enable/disable vlm chart yranging (TO SQUASH) 2023-02-12 13:41:17 -05:00
Tyler Goodlet eb9ab20646 Don't disable non-enabled vlm chart y-autoranging 2023-02-12 13:41:17 -05:00
Tyler Goodlet f3bab826f6 Comment out bps for time indexing 2023-02-12 13:41:17 -05:00
Tyler Goodlet 2b9ca5f805 Call `Viz.bars_range()` from display loop 2023-02-12 13:41:17 -05:00
Tyler Goodlet 25a75e5bec Fix `.default_view()` to view-left-of-data 2023-02-12 13:41:17 -05:00
Tyler Goodlet 702ae29a2c Add `Viz.index_field: str`, pass to graphics objs
In an effort to make it easy to override the indexing scheme.

Further, this repairs the `.datums_range()` special case to handle when
the view box is to-the-right-of the data set (i.e. l > datum_start).
2023-02-12 13:41:17 -05:00
Tyler Goodlet ac1f37a2c2 Expect `index_field: str` in all graphics objects 2023-02-12 13:41:17 -05:00
Tyler Goodlet 344d2eeb9e Facepalm: pass correct flume to each FSP chart group.. 2023-02-12 13:41:17 -05:00
Tyler Goodlet 9133103f8f Attempt to make `.default_view()` time-index ready
As in make the call to `Flume.slice_from_time()` to try and convert any
time index values from the view range to array-indices; all untested
atm.

Also drop some old/unused/moved methods:
- `._set_xlimits()`
- `.bars_range()`
- `.curve_width_pxs()`

and fix some `flow` -> `viz` var naming.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 166d14af69 Simplify formatter update methodology
Don't expect values (array + slice) to be returned and applied by
`.incr_update_xy_nd()` and instead presume this will implemented
internally in each (sub)formatter.

Attempt to simplify some incr-update routines, (particularly in the step
curve formatter, though most of it was reverted to just a simpler form
of the original implementation XD) including:
- dropping the need for the `slice_to_head: int` control.
- using the `xy_nd_start/stop` index counters over custom lookups.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 696c6f8897 First attempt, field-index agnostic formatting
Remove harcoded `'index'` field refs from all formatters in a first
attempt at moving towards epoch-time alignment (though don't actually
use it it yet).

Adjustments to the formatter interface:
- property for `.xy_nd` the x/y nd arrays.
- property for and `.xy_slice` the nd format array(s) start->stop index
  slice.

Internal routine tweaks:
- drop `read_src_from_key` and always pass full source array on updates
  and adjust handlers to expect to have to index the data field of
  interest.
- set `.last_read` right after update calls instead of after 1d
  conversion.
- drop `slice_to_head` array read slicing.
- add some debug points for testing 'time' indexing (though not used
  here yet).
- add `.x_nd` array update logic for when the `.index_field` is not
  'index' - i.e. when we begin to try and support epoch time.
- simplify some new y_nd updates to not require use of `np.broadcast()`
  where possible.
2023-02-12 13:41:17 -05:00
Tyler Goodlet be21f9829e Pepper render routines with time-slice calls 2023-02-12 13:41:17 -05:00
Tyler Goodlet 5a0673d66f Add `Viz.bars_range()` (moved from chart API)
Call it from view kb loop.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 6cacd7d18b Make `Viz.slice_from_time()` take input array
Probably means it doesn't need to be a `Flume` method but it's
convenient to expect the caller to pass in the `np.ndarray` with
a `'time'` field instead of a `timeframe: str` arg; also, return the
slice mask instead of the sliced array as output (again allowing the
caller to do any slicing). Also, handle the slice-outside-time-range
case by just returning the entire index range with a `None` mask.

Adjust `Viz.view_data()` to instead do timeframe (for rt vs. hist shm
array) lookup and equiv array slicing with the returned mask.
2023-02-12 13:41:17 -05:00
Tyler Goodlet 5b08e9cba3 Add breakpoint on -ve range for now 2023-02-12 13:41:17 -05:00
Tyler Goodlet d3f5ff1b4f Go back to hard-coded index field
Turns out https://github.com/numba/numba/issues/8622 is real
and the suggested `numba.literally` hack doesn't seem to work..
2023-02-12 13:41:16 -05:00
Tyler Goodlet e45bc4c619 Move `ui._compression`/`._pathops` to `.data` subpkg
Since these modules no longer contain Qt specific code we might
as well include them in the data sub-package.

Also, add `IncrementalFormatter.index_field` as single point to def the
indexing field that should be used for all x-domain graphics-data
rendering.
2023-02-12 13:39:10 -05:00
Tyler Goodlet baee86a2d6 Rename `.ui._flows.py` -> `.ui._render.py` 2023-02-12 13:39:10 -05:00
Tyler Goodlet 86d09d9305 Rename `Flow` -> `Viz`
The type is better described as a "data visualization":
https://en.wikipedia.org/wiki/Data_and_information_visualization

Add `ChartPlotWidget.get_viz()` to start working towards not accessing
the private table directly XD

We'll probably end up using the name `Flow` for a type that tracks
a collection of composed/cascaded `Flume`s:
https://en.wikipedia.org/wiki/Two-port_network#Cascade_connection
2023-02-12 13:39:10 -05:00
Tyler Goodlet 9ace053aaf Copy timestamps from source to FSP dest buffer 2023-02-12 13:39:10 -05:00
Guillermo Rodriguez 5ca45362c8
Add default factories for all required fields 2023-02-11 16:08:45 -03:00
Tyler Goodlet 1f2081911f Revert "Adjust chart call to graphics cycle to not pass quotes"
This reverts commit 50ad7370c7
which was originally applied due to missing API changes coming in
a future patchset..
2023-02-09 16:26:32 -05:00
goodboy a7d02ecec8
Merge pull request #449 from pikers/multi_symbol_input
Multi symbol input (support)
2023-02-09 16:20:34 -05:00
goodboy 11ba706797
Merge pull request #448 from pikers/axis_sticky_api
Axis sticky api, `PlotItem` is the new "chart"
2023-02-05 15:32:22 -05:00
Tyler Goodlet 50ad7370c7 Adjust chart call to graphics cycle to not pass quotes
Was breaking the `'r'` hotkey to reset the chart..
2023-02-05 15:27:12 -05:00
Tyler Goodlet af92602027 `ib`: make commodities search and feeds work again..
Was broken since the `_adhoc_futes_set` rework a while back. Removes the
cmdty symbols from that set into a new one and fixes the contract
case block to catch `Contract(secType='CMDTY')` case. Also makes
`Client.search_symbols()` return details `dict`s so that `piker search`
will work again..
2023-02-02 16:52:34 -05:00
Tyler Goodlet 07ab853d3d `Order.symbol` is a `str`.. 2023-02-02 15:05:26 -05:00
Tyler Goodlet 414866fc6b Assign pnl calc output for use when debugging 2023-02-02 15:05:26 -05:00
Tyler Goodlet bc7fe6114d Adjust order mode to use `Flume.get_index()` 2023-02-02 15:05:23 -05:00
Tyler Goodlet 8d592886fa Pass `Flume`s throughout FSP-ui and charting APIs
Since higher level charting and fsp management need access to the
new `Flume` indexing apis this adjusts some func sigs to pass through
(and/or create) flume instances:
- `LinkedSplits.add_plot()` and dependents.
- `ChartPlotWidget.draw_curve()` and deps, and it now returns a `Flow`.
- `.ui._fsp.open_fsp_admin()` and `FspAdmin.open_fsp_ui()` related
  methods => now we wrap the destination fsp shm in a flume on the admin
  side and is returned from `.start_engine_method()`.

Drop a bunch of (unused) chart widget methods including some already
moved to flume methods: `.get_index()`, `.in_view()`,
`.last_bar_in_view()`, `.is_valid_index()`.
2023-02-02 13:32:30 -05:00
Tyler Goodlet 69ea296a9b Max out per symbol throttle @ 22Hz 2023-02-02 13:32:30 -05:00
Tyler Goodlet 03821fdf6f Expect and update from by-type tick frames
Move to expect and process new by-tick-event frames where the display
loop can now just iterate the most recent tick events by type instead of
the entire tick history sequence - thus we reduce iterations inside the
update loop.

Also, go back to use using the detected display's refresh rate (minus 6)
as the default feed requested throttle rate since we can now handle
much more bursty-ness in display updates thanks to the new framing
format B)
2023-02-02 13:32:30 -05:00
Tyler Goodlet 1aa9ab03da Brighter last OHLC graphics datum by default 2023-02-02 13:32:20 -05:00
Tyler Goodlet 1d83b43efe Factor setup loop, 1 FSP chain, colors, throttling
Factor out the chart widget creation since it's only executed once
during rendering of the first feed/flow whilst keeping plotitem overlay
creation inside the (flume oriented) init loop. Only create one vlm and
FSP chart/chain for now until we figure out if we want FSPs overlayed by
default or selected based on the "front" symbol in use. Add a default
color-palette set using shades of gray when plotting overlays. Presume
that the display loop's quote throttle rate should be uniformly
distributed over all input symbol-feeds for now. Restore feed pausing on
mouse interaction.
2023-02-02 13:32:20 -05:00
Tyler Goodlet 6986be1b21 Define a single `ChartPlotWidget.feed: Feed` for pause/resume 2023-02-02 13:32:20 -05:00
Tyler Goodlet 92c50aa6a7 Drop tick frame builder loop for now 2023-02-02 13:32:20 -05:00
Tyler Goodlet eac79c5cdd Adjust FSP UI/mgmt apis to be `Flume` oriented 2023-02-02 13:32:20 -05:00
Tyler Goodlet 7aec238f5f Make graphics-update-loop multi-sym aware B)
Initial support for real-time multi-symbol overlay charts using an
aggregate feed delivered by `Feed.open_multi_stream()`.

The setup steps for constructing the overlayed plot items is still very
very rough and will likely provide incentive for better refactoring high
level "charting APIs". For each fqsn passed into `display_symbol_data()`
we now synchronously,
- create a single call to `LinkedSplits.plot_ohlc_main() -> `ChartPlotWidget`
  where we cache the chart in scope and for all other "sibling" fqsns
  we,
- make a call to `ChartPlotWidget.overlay_plotitem()` -> `PlotItem`, hide its axes,
  make another call with this plotitem input to
  `ChartPlotWidget.draw_curve()`, set a sym-specific view box auto-yrange maxmin callback,
  register the plotitem in a global `pis: dict[str, list[pgo.PlotItem, pgo.PlotItem]] = {}`

Once all plots have been created we then asynchronously for each symbol,
- maybe create a volume chart and register it in a similar task-global
  table: `vlms: dict[str, ChartPlotWidget] = {}`
- start fsp displays for each symbol

Then common entrypoints are entered once for all symbols:
- a single `graphics_update_loop()` loop-task is started wherein
  real-time graphics update components for each symbol are created,
      * `L1Labels`
      * y-axis last clearing price stickies
      * `maxmin()` auto-ranger
      * `DisplayState` (stored in a table `dss: dict[str, DisplayState] = {}`)
      * an `increment_history_view()` task
  and a single call to `Feed.open_multi_stream()` is used to create
  a symbol-multiplexed quote stream which drives a single loop over all
  symbols wherein for each quote the appropriate components are looked
  up and passed to `graphics_update_cycle()`.
- a single call to `open_order_mode()` is made with the first symbol
  provided as input, though eventually we want to support passing in the
  entire list.

Further internal implementation details:
- special tweaks to the `pg.LinearRegionItem` setup wherein the region
  is added with a zero opacity and *after* all plotitem overlays to
  avoid and issue where overlays weren't being shown within the region
  area in the history chart.
- all symbol-specific graphics oriented update calls are adjusted to
  pass in the fqsn:
  * `update_fsp_chart()`
  * `ChartView._set_yrange()`
  * ChartPlotWidget.update_graphics_from_flow()`
- avoid a double increment on sample step updates by not calling the
  increment on any vlm chart since it seems the vlm-ohlc chart linking
  already takes care of this now?
- use global counters for the last epoch time step to avoid incrementing
  all views more then once per new time step given underlying shm array
  buffers may be on different array-index values from one another.
2023-02-02 13:30:02 -05:00
Tyler Goodlet be3dc69290 Only update pnl label on quotes with an fqsn match 2023-02-02 13:30:02 -05:00
Tyler Goodlet 6100bd19c7 Adjust search to handle multi-sym results 2023-01-31 15:16:34 -05:00
Tyler Goodlet d57bc6c6d9 Adjust to using `PlotItem`s for axis sticky mgmt 2023-01-31 15:15:56 -05:00
Tyler Goodlet 58b42d629f Passthrough fqsns list directly to `.load_symbols()` 2023-01-31 14:54:19 -05:00
Tyler Goodlet 36a81cb2de Only add plot to cursor set if not an overlay 2023-01-31 14:27:39 -05:00
Tyler Goodlet ae0f3118f4 Pass plotitem to axis from cursor 2023-01-31 14:27:39 -05:00
Tyler Goodlet 727c7ce2b1 Adjust L1 labels to expect `.pi: PlotItem` 2023-01-31 14:27:39 -05:00
Tyler Goodlet a39c980266 Allocate our internal `Axis` subtype in our `PlotItem` override 2023-01-31 14:27:39 -05:00
Tyler Goodlet 00be100e71 Initial chart widget adjustments for agg feeds
Main "public" API change is to make `GodWidget.get/set_chart_symbol()`
accept and cache-on fqsn tuples to allow handling overlayed chart groups
and adjust method names to be plural to match.

Wrt `LinkedSplits`,
- create all chart widget axes with a `None` plotitem argument and set
  the `.pi` field after axis creation (since apparently we have another
  object reference causality dilemma..)
- set a monkeyed `PlotItem.chart_widget` for use in axes that still need
  the widget reference.
- drop feed pause/resume for now since it's leaking feed tasks on the
  `brokerd` side and we probably don't really need it any more, and if
  we still do it should be done on the feed not the flume.

Wrt `ChartPlotItem`,
- drop `._add_sticky()` and use the `Axis` method instead and add some
  overlay + axis sanity checks.
- refactor `.draw_ohlc()` to be a lighter wrapper around a call to
  `.add_plot()`.
2023-01-31 14:27:39 -05:00
Tyler Goodlet 9217610734 Simplify OHLC graphic color instance var name 2023-01-31 14:27:39 -05:00
Tyler Goodlet 31af7a2c99 Add `Axis.add_sticky()` for creating axis labels
We have this method on our `ChartPlotWidget` but it makes more sense to
directly associate axis-labels with, well, the label's parent axis XD.

We add `._stickies: dict[str, YAxisLabel]` to replace
`ChartPlotWidget._ysticks` and pass in the `pg.PlotItem` to each axis
instance, stored as `Axis.pi` instead of handing around linked split
references (which are way out of scope for a single axis).

More work needs to be done to remove dependence on `.chart:
ChartPlotWidget` references in the date axis type as per comments.
2023-01-31 14:27:39 -05:00
Tyler Goodlet 34fac364fd Add default YAxisLable.x_offset: int` 2023-01-31 14:27:39 -05:00
goodboy dcdfd2577a
Merge pull request #447 from pikers/pregraphics_formatters
Pregraphics formatters: `IncrementalFormatter`
2023-01-31 13:55:04 -05:00
goodboy 6733dc57af
Merge pull request #441 from pikers/dark_clearing_repairs
Dark clearing repairs
2023-01-30 14:21:23 -05:00
Tyler Goodlet 05c4b6afb9 Drop px-cache-resets, failed try at path appends
Comments out the pixel-cache resetting since it doesn't seem we need it
any more to avoid draw oddities?

For `.fast_path` appends, this nearly got it working except the new path
segments are either not being connected correctly (step curve) or not
being drawn in full since the history path (plain line).

Leaving the attempted code commented in for a retry in the future; my
best guesses are that maybe,
- `.connectPath()` call is being done with incorrect segment length
  and/or start point.
- the "appended" data: `appended = array[-append_len-1:slice_to_head]`
  (done inside the formatter) isn't correct (i.e. endpoint handling
  considering a path append) and needs special handling for different
  curve types?
2023-01-30 13:22:24 -05:00
Tyler Goodlet 4b22325ffc Mask profile points and drop rect `.united()` attempts 2023-01-30 13:22:14 -05:00
Tyler Goodlet 9d16299f60 Make curve graphics timeframe agnostic
Ensure `.boundingRect()` calcs and `.draw_last_datum()` do geo-sizing
based on source data instead of presuming some `1.0` unit steps in some
spots; we need this to support an epoch index as is needed for overlays.

Further, clean out a bunch of old bounding rect calc code and add some
commented code for trying out `QRectF.united()` on the path + last datum
curve segment. Turns out that approach is slower as per eyeballing the
added profiler points.
2023-01-30 13:21:43 -05:00
Tyler Goodlet ab1f15506d Add graphics incr-updated "formatter" subsys
After trying to hack epoch indexed time series and failing miserably,
decided to properly factor out all formatting routines into a common
subsystem API: ``IncrementalFormatter`` which provides the interface for
incrementally updating and tracking pre-path-graphics formatted data.

Previously this functionality was mangled into our `Renderer` (which
also does the work of `QPath` generation and update) but splitting it
out also preps for being able to do graphics-buffer downsampling and
caching on a remote host B)

The ``IncrementalFormatter`` (parent type) has the default behaviour of
tracking a single field-array on some source `ShmArray`, updating
a flattened `numpy.ndarray` in-mem allocation, and providing a default
1d conversion for pre-downsampling and path generation.

Changed out of `Renderer`,
- `.allocate_xy()`, `update_xy()` and `format_xy()` all are moved to
  more explicitly named formatter methods.
- all `.x/y_data` nd array management and update
- "last view range" tracking
- `.last_read`, `.diff()`
- now calls `IncrementalFormatter.format_to_1d()` inside `.render()`

The new API gets,
- `.diff()`, `.last_read`
- all view range diff tracking through `.track_inview_range()`.
- better nd format array names: `.x/y_nd`, `xy_nd_start/stop`.
- `.format_to_1d()` which renders pre-path formatted arrays ready for
  both m4 sampling and path gen.
- better explicit overloadable formatting method names:
  * `.allocate_xy()` -> `.allocate_xy_nd()`
  * `.update_xy()` -> `.incr_update_xy_nd()`
  * `.format_xy()` -> `.format_xy_nd_to_1d()`

Finally this implements per-graphics-type formatters which define
each set up related formatting routines:
- `OHLCBarsFmtr`: std multi-line style bars
- `OHLCBarsAsCurveFmtr`: draws an interpolated line for ohlc sampled data
- `StepCurveFmtr`: handles vlm style curves
2023-01-30 13:20:17 -05:00
Tyler Goodlet 0db5451e47 Move all pre-path formatting routines to `._pathops`, proto formatter type 2023-01-30 13:19:33 -05:00
goodboy 61218f30f5
Merge pull request #440 from pikers/samplerd_service
`samplerd` service
2023-01-30 11:48:07 -05:00
Tyler Goodlet fcfc0f31f0 Enable backpressure in an effort to prevent bootup overruns 2023-01-30 11:45:29 -05:00
Tyler Goodlet 61e20a86cc Fix clearing endpoint type annots, export `open_ems()` 2023-01-24 15:15:27 -05:00
Tyler Goodlet d9b73e1d08 Yield services (manager) from `maybe_open_pikerd()` 2023-01-24 15:15:27 -05:00
Tyler Goodlet 090d1ba524 `kraken`: catch value error not index on missing `src_fiat` in pair 2023-01-23 15:36:20 -05:00
Tyler Goodlet afc45a8e16 `binance`: same thing, only unsub when connected 2023-01-23 15:29:24 -05:00
Tyler Goodlet 844626f6dc Move `brokerd` service task to root `.data` mod 2023-01-13 13:21:49 -05:00
Tyler Goodlet 470079665f Use new tractor kwargs getter func 2023-01-13 13:21:49 -05:00
Tyler Goodlet 0cd87d9e54 Drop commented markestored spawner code 2023-01-13 13:21:49 -05:00
Tyler Goodlet 09711750bf Registry subsys rework
More or less a revamp (and possibly first draft for something similar in
`tractor` core) which ensures all actor trees attempt to discover the
`pikerd` registry actor.

Implementation improvements include:
- new `Registry` singleton which houses the `pikerd` discovery
  socket-address `Registry.addr` + a `open_registry()` manager which
  provides bootstrapped actor-local access.
- refine `open_piker_runtime()` to do the work of opening a root actor
  and call the new `open_registry()` depending on whether a runtime has
  yet been bootstrapped.
- rejig `[maybe_]open_pikerd()` in terms of the above.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 71ca4c8e1f Use actor uid in shm keys for rt quote buffers
Allows running simultaneous data feed services on the same (linux) host
by avoiding file-name collisions instead keying shm buffer sets by the
given `brokerd` instance. This allows, for example, either multiple dev
versions of the data layer to run side-by-side or for the test suite to
be seamlessly run alongside a production instance.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 9811dcf5f3 Match `services` subcmd to new reg addr module variables 2023-01-13 13:21:49 -05:00
Tyler Goodlet da659cf607 Facepalm: definitely do not short circuit discovery helpers.. 2023-01-13 13:21:49 -05:00
Tyler Goodlet 045b76bab5 Make `Flume.index_stream()` defer to new sampling api 2023-01-13 13:21:49 -05:00
Tyler Goodlet c8c641a038 Ensure all sub-services cancel on `pikerd` exit
Previously we were relying on implicit actor termination in
`maybe_spawn_daemon()` but really on `pikerd` teardown we should be sure
to tear down not only all service tasks in each actor but also the actor
runtimes. This adjusts `Services.cancel_service()` to only cancel the
service task scope and wait on the `complete` event and reworks the
`open_context_in_task()` inner closure body to,

- always cancel the service actor at exit.
- not call `.cancel_service()` (potentially causing recursion issues on
  cancellation).
- allocate a `complete: trio.Event` to signal full task + actor termination.
- pop the service task from the `.service_tasks` registry.

Further, add a `maybe_set_global_registry_sockaddr()` helper-cm to do
the work of checking whether a registry socket needs-to/has-been set
and use it for discovery calls to the `pikerd` service tree.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 75591dd7e9 Don't raise on quote feed lags to dark clearing loop 2023-01-13 13:21:49 -05:00
Tyler Goodlet d792fed099 Move sync log msg back to info 2023-01-13 13:21:49 -05:00
Tyler Goodlet d66fb49077 Don't deliver shms from `start_backfill()`, they're not used 2023-01-13 13:21:49 -05:00
Tyler Goodlet 78c7c8524c Breakpoint when bad 1m history offsets are detected 2023-01-13 13:21:49 -05:00
Tyler Goodlet a746258f99 `binance`: always request an extra 1min OHLC bar
Seems that by default their history indexing rounds down/back to the
previous time step, so make sure we add a minute inside `Client.bars()`
when the `end_dt=None`, indicating "get the latest bar". Add
a breakpoint block that should trigger whenever the latest bar vs. the
latest epoch time is mismatched; we'll remove this after some testing
verifying the history bars issue is resolved.

Further this drops the legacy `backfill_bars()` endpoint which has been
deprecated and unused for a while.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 5adb234a24 Don't receive sample-index msgs in feed layer 2023-01-13 13:21:49 -05:00
Tyler Goodlet 2778ee1401 Support not registering for sample-index msgs via `sub_for_broadcasts: bool` flag 2023-01-13 13:21:49 -05:00
Tyler Goodlet e0ca5d5200 Use `open_sample_stream()` to increment fsp buffers 2023-01-13 13:21:47 -05:00
Tyler Goodlet b3d1b1aa63 Port feed layer to use new `samplerd` APIs
Always use `open_sample_stream()` to register fast and slow quote feed
buffers and get a sampler stream which we use to trigger
`Sampler.broadcast_all()` calls on the service side after backfill
events.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 5ec1a72a3d Implement a `samplerd` singleton actor service
Now spawned under the `pikerd` tree as a singleton-daemon-actor we offer
a slew of new routines in support of this micro-service:

- `maybe_open_samplerd()` and `spawn_samplerd()` which provide the
  `._daemon.Services` integration to conduct service spawning.
- `open_sample_stream()` which is a client-side endpoint which does all
  the work of (lazily) starting the `samplerd` service (if dne) and
  registers shm buffers for update as well as connect a sample-index
  stream for iterator by the caller.
- `register_with_sampler()` which is the `samplerd`-side service task
  endpoint implementing all the shm buffer and index-stream registry
  details as well as logic to ensure a lone service task runs
  `Services.increment_ohlc_buffer()`; it increments at the shortest period
  registered which, for now, is the default 1s duration.

Further impl notes:
- fixes to `Services.broadcast()` to ensure broken streams get discarded
  gracefully.
- we use a `pikerd` side singleton mutex `trio.Lock()` to ensure
  one-and-only-one `samplerd` is ever spawned per `pikerd` actor tree.
2023-01-13 13:21:15 -05:00
Tyler Goodlet a342f7d2d4 Make `._daemon.Services` for use as singleton
Drop the `_services` module level ref and adjust all client code to
match. Drop struct inheritance and convert all methods to class level.
Move `Brokerd.locks` -> `Services.locks` and add sampling mod to pikerd
enabled set.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 2c76cee928 Begin formalizing `Sampler` singleton API
We're moving toward a single actor managing sampler work and distributed
independently of `brokerd` services such that a user can run samplers on
different hosts then real-time data feed infra. Most of the
implementation details include aggregating `.data._sampling` routines
into a new `Sampler` singleton type.

Move the following methods to class methods:
- `.increment_ohlc_buffer()` to allow a single task to increment all
  registered shm buffers.
- `.broadcast()` for IPC relay to all registered clients/shms.

Further add a new `maybe_open_global_sampler()` which allocates
a service nursery and assigns it to the `Sampler.service_nursery`; this
is prep for putting the step incrementer in a singleton service task
higher up the data-layer actor tree.
2023-01-13 13:21:15 -05:00