Commit Graph

3000 Commits (4020a198c43a596f8b1674f7b8b257701823653f)

Author SHA1 Message Date
Tyler Goodlet 4020a198c4 Type annot-declare fsp-engine data `Feed` 2023-01-10 11:09:49 -05:00
Tyler Goodlet d834dfac74 Define a single `ChartPlotWidget.feed: Feed` for pause/resume 2023-01-10 11:09:49 -05:00
Tyler Goodlet bdbc8de8c1 Assign pnl calc output for use when debugging 2023-01-10 11:09:49 -05:00
Tyler Goodlet 5e6ebca1e0 Rework `_FeedsBus` subscriptions mgmt using `set`
Allows using `set` ops for subscription management and guarantees no
duplicates per `brokerd` actor. New API is simpler for dynamic
pause/resume changes per `Feed`:
- `_FeedsBus.add_subs()`, `.get_subs()`, `.remove_subs()` all accept multi-sub
  `set` inputs.
- `Feed.pause()` / `.resume()` encapsulates management of *only* sending
  a msg on each unique underlying IPC msg stream.

Use new api in sampler task.
2023-01-10 11:09:49 -05:00
Tyler Goodlet 42e934b912 Init msg keys are always lower case 2023-01-10 11:09:49 -05:00
Tyler Goodlet da285d6275 Make `PlotItemOverlay` add items inwards->out
Before this axes were being stacked from the outside in (for `'right'`
and 'bottom'` axes) which is somewhat non-intuitive for an `.append()`
operation. As such this change makes a symbol list stack a set of
`'right'` axes from left-to-right.

Details:
- rename `ComposeGridLayout.items` -> `.pitems`
- return `(int, list[AxisItem])` pairs from `.insert/append_plotitem()`
  and the down stream `PlotItemOverlay.add_plotitem()`.
- drop `PlotItemOverlay.overlays` and add it back as `@property` around
  the underlying `.layout.pitems`.
2023-01-10 11:09:49 -05:00
Tyler Goodlet 396fb742bd Fix for empty tsdb query result case 2023-01-10 11:09:49 -05:00
Tyler Goodlet 8c6a18fdb7 Drop tick frame builder loop for now 2023-01-10 11:09:49 -05:00
Tyler Goodlet a6241a5a16 Adjust FSP UI/mgmt apis to be `Flume` oriented 2023-01-10 11:09:49 -05:00
Tyler Goodlet eb1650197b Make graphics-update-loop multi-sym aware B)
Initial support for real-time multi-symbol overlay charts using an
aggregate feed delivered by `Feed.open_multi_stream()`.

The setup steps for constructing the overlayed plot items is still very
very rough and will likely provide incentive for better refactoring high
level "charting APIs". For each fqsn passed into `display_symbol_data()`
we now synchronously,
- create a single call to `LinkedSplits.plot_ohlc_main() -> `ChartPlotWidget`
  where we cache the chart in scope and for all other "sibling" fqsns
  we,
- make a call to `ChartPlotWidget.overlay_plotitem()` -> `PlotItem`, hide its axes,
  make another call with this plotitem input to
  `ChartPlotWidget.draw_curve()`, set a sym-specific view box auto-yrange maxmin callback,
  register the plotitem in a global `pis: dict[str, list[pgo.PlotItem, pgo.PlotItem]] = {}`

Once all plots have been created we then asynchronously for each symbol,
- maybe create a volume chart and register it in a similar task-global
  table: `vlms: dict[str, ChartPlotWidget] = {}`
- start fsp displays for each symbol

Then common entrypoints are entered once for all symbols:
- a single `graphics_update_loop()` loop-task is started wherein
  real-time graphics update components for each symbol are created,
      * `L1Labels`
      * y-axis last clearing price stickies
      * `maxmin()` auto-ranger
      * `DisplayState` (stored in a table `dss: dict[str, DisplayState] = {}`)
      * an `increment_history_view()` task
  and a single call to `Feed.open_multi_stream()` is used to create
  a symbol-multiplexed quote stream which drives a single loop over all
  symbols wherein for each quote the appropriate components are looked
  up and passed to `graphics_update_cycle()`.
- a single call to `open_order_mode()` is made with the first symbol
  provided as input, though eventually we want to support passing in the
  entire list.

Further internal implementation details:
- special tweaks to the `pg.LinearRegionItem` setup wherein the region
  is added with a zero opacity and *after* all plotitem overlays to
  avoid and issue where overlays weren't being shown within the region
  area in the history chart.
- all symbol-specific graphics oriented update calls are adjusted to
  pass in the fqsn:
  * `update_fsp_chart()`
  * `ChartView._set_yrange()`
  * ChartPlotWidget.update_graphics_from_flow()`
- avoid a double increment on sample step updates by not calling the
  increment on any vlm chart since it seems the vlm-ohlc chart linking
  already takes care of this now?
- use global counters for the last epoch time step to avoid incrementing
  all views more then once per new time step given underlying shm array
  buffers may be on different array-index values from one another.
2023-01-10 11:09:49 -05:00
Tyler Goodlet 6b4614f735 Only add plot to cursor set if not an overlay 2023-01-10 11:09:49 -05:00
Tyler Goodlet fc067eb7a8 Adjust search to handle multi-sym results 2023-01-10 11:09:49 -05:00
Tyler Goodlet 0d657553f9 Drop the legacy `relayed_from` cruft from our view box 2023-01-10 11:09:49 -05:00
Tyler Goodlet b384cea706 Only update pnl label on quotes with an fqsn match 2023-01-10 11:09:49 -05:00
Tyler Goodlet 70b24795a6 Pass plotitem to axis from cursor 2023-01-10 11:09:49 -05:00
Tyler Goodlet dc4a2c8c2b Adjust L1 labels to expect `.pi: PlotItem` 2023-01-10 11:09:49 -05:00
Tyler Goodlet 322ab34200 Allocate our internal `Axis` subtype in our `PlotItem` override 2023-01-10 11:09:49 -05:00
Tyler Goodlet b768eb19ec Passthrough fqsns list directly to `.load_symbols()` 2023-01-10 11:09:49 -05:00
Tyler Goodlet c3e5162c30 Initial chart widget adjustments for agg feeds
Main "public" API change is to make `GodWidget.get/set_chart_symbol()`
accept and cache-on fqsn tuples to allow handling overlayed chart groups
and adjust method names to be plural to match.

Wrt `LinkedSplits`,
- create all chart widget axes with a `None` plotitem argument and set
  the `.pi` field after axis creation (since apparently we have another
  object reference causality dilemma..)
- set a monkeyed `PlotItem.chart_widget` for use in axes that still need
  the widget reference.
- drop feed pause/resume for now since it's leaking feed tasks on the
  `brokerd` side and we probably don't really need it any more, and if
  we still do it should be done on the feed not the flume.

Wrt `ChartPlotItem`,
- drop `._add_sticky()` and use the `Axis` method instead and add some
  overlay + axis sanity checks.
- refactor `.draw_ohlc()` to be a lighter wrapper around a call to
  `.add_plot()`.
2023-01-10 11:09:49 -05:00
Tyler Goodlet e677cb1ddb Simplify OHLC graphic color instance var name 2023-01-10 11:09:49 -05:00
Tyler Goodlet fc7c498c65 Add `Axis.add_sticky()` for creating axis labels
We have this method on our `ChartPlotWidget` but it makes more sense to
directly associate axis-labels with, well, the label's parent axis XD.

We add `._stickies: dict[str, YAxisLabel]` to replace
`ChartPlotWidget._ysticks` and pass in the `pg.PlotItem` to each axis
instance, stored as `Axis.pi` instead of handing around linked split
references (which are way out of scope for a single axis).

More work needs to be done to remove dependence on `.chart:
ChartPlotWidget` references in the date axis type as per comments.
2023-01-10 11:09:48 -05:00
Tyler Goodlet 6653ee8662 Add default YAxisLable.x_offset: int` 2023-01-10 11:09:48 -05:00
Tyler Goodlet 963e5bdd62 Go back to `Feed.pause/resume()`, new flume APIs coming later 2023-01-10 11:09:19 -05:00
Tyler Goodlet 55de9abc41 Adjust cli mod imports of daemon sockaddr vars 2023-01-10 11:09:19 -05:00
Tyler Goodlet 593db0ed0d Only run `kraken` feed tests in CI, use `open_test_pikerd()` 2023-01-10 11:09:19 -05:00
Tyler Goodlet 06622105cd Add a `open_test_pikerd()` acm fixture for easy booting of the service stack 2023-01-10 11:09:19 -05:00
Tyler Goodlet 008ae47e14 Reset `._registry_addr` to any passed in value from caller 2023-01-10 11:09:19 -05:00
Tyler Goodlet 81585d9e6e Set global registry addr after first entry point spawns `pikerd` 2023-01-10 11:09:19 -05:00
Tyler Goodlet f6b7057b0d `binance`: always request an extra 1min OHLC bar
Seems that by default their history indexing rounds down/back to the
previous time step, so make sure we add a minute inside `Client.bars()`
when the `end_dt=None`, indicating "get the latest bar". Add
a breakpoint block that should trigger whenever the latest bar vs. the
latest epoch time is mismatched; we'll remove this after some testing
verifying the history bars issue is resolved.

Further this drops the legacy `backfill_bars()` endpoint which has been
deprecated and unused for a while.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 76f920a16b Always force lowercase on `binance` symbol keys
Hopefully helps resolve #435
2023-01-10 11:09:19 -05:00
Tyler Goodlet f232d6d4ee Add `ci_env` detector fixture 2023-01-10 11:09:19 -05:00
Tyler Goodlet b7e1443618 Use ETH on kraken to ensure enough quotes 2023-01-10 11:09:19 -05:00
Tyler Goodlet 5d021ffb85 Bump up timeout on multi-feed test for CI 2023-01-10 11:09:19 -05:00
Tyler Goodlet 28fd795280 Only require `-b <brokername>` for filtering
Instead of requiring any `-b` try to import all built-in broker backend
python modules by default and only load those detected from the input symbol
list's fqsn values. In other words the `piker chart` cmd can be run sin
`-b` now and that flag is only required if you only want to load
a subset of the built-ins or are trying to load a specific
not-yet-builtin backend.
2023-01-10 11:09:19 -05:00
Tyler Goodlet c944db5f02 Revert "Fix `_main()` arg back to `sym: str`"
This reverts commit 02fbc0a0ed.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 967e28b7ac Adjust built-in backend list to known working 2023-01-10 11:09:19 -05:00
Tyler Goodlet 2a158aea2c Rework `_FeedsBus` subscriptions mgmt using `set`
Allows using `set` ops for subscription management and guarantees no
duplicates per `brokerd` actor. New API is simpler for dynamic
pause/resume changes per `Feed`:
- `_FeedsBus.add_subs()`, `.get_subs()`, `.remove_subs()` all accept multi-sub
  `set` inputs.
- `Feed.pause()` / `.resume()` encapsulates management of *only* sending
  a msg on each unique underlying IPC msg stream.

Use new api in sampler task.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 88870fdda7 Set `brokers: list[st]` from mods when not provided.. 2023-01-10 11:09:19 -05:00
Tyler Goodlet 326f153a47 Catch overruns on throttled feed subs too
Previously we would only detect overruns and drop subscriptions on
non-throttled feed subs, however you can get the same issue with
a wrapping throttler task:
- the intermediate mem chan can be blocked either by the throttler task
  being too slow, in which case we still want to warn about it
- the stream's IPC channel actually breaks and we still want to drop
  the connection and subscription so it doesn't be come a source of
  stale backpressure.
2023-01-10 11:09:19 -05:00
Tyler Goodlet f5cd63ad35 Ensure correct stream is set on each `Flume`
Set each quote-stream by matching the provider for each `Flume` and thus
results in some flumes mapping to the same (multiplexed) stream.
Monkey-patch the equivalent `tractor.MsgStream._ctx: tractor.Context` on
each broadcast-receiver subscription to allow use by feed bus methods as
well as other internals which need to reference IPC channel/portal info.

Start a `_FeedsBus` subscription management API:
- add `.get_subs()` which returns the list of tuples registered for the
  given key (normally the fqsn).
- add `.remove_sub()` which allows removing by key and tuple value and
  provides encapsulation for sampler task(s) which deal with dropped
  connections/subscribers.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 1e96ca32df Move `maybe_open_feed()` above for readability 2023-01-10 11:09:19 -05:00
Tyler Goodlet c088963cf2 Always touch config file dir if dne 2023-01-10 11:09:19 -05:00
Tyler Goodlet 79fcbcc281 Add an sdist job to CI 2023-01-10 11:09:19 -05:00
Tyler Goodlet ddbba76095 Use (a new) `piker_pin` branch in `tractor` (again) 2023-01-10 11:09:19 -05:00
Tyler Goodlet 0a959c1c74 Not all accounts will have API trade transactions this session.. 2023-01-10 11:09:19 -05:00
Tyler Goodlet e348968113 Add multi-broker streaming test using both `binance` and `kraken` 2023-01-10 11:09:19 -05:00
Tyler Goodlet 7bbe86d6fb Unpack broker mod and portal from fqsn for brokerd-trade-dialogs 2023-01-10 11:09:19 -05:00
Tyler Goodlet 7b9db86753 Multi-`broker` quotes with `Feed.open_multi_stream()`
Adds provider-list-filtered (quote) stream multiplexing support allowing
for merged real-time `tractor.MsgStream`s using an `@acm` interface.
Behind the scenes we are just doing a classic multi-task push to common
mem chan approach.

Details to make it work on `Feed`:
- add `Feed.mods: dict[str, Moduletype]` and
  `Feed.portals[ModuleType, tractor.Portal]` which are both populated
  during init in `open_feed()`
- drop `Feed.portal` and `Feed.name`

Also fix a final lingering tsdb history loading loop termination bug.
2023-01-10 11:09:19 -05:00
Tyler Goodlet 20a396270e `Storage.read_ohlcv()` now returns a `numpy` array 2023-01-10 11:09:19 -05:00
Tyler Goodlet 81516c5204 Finally fix tsdb -> shm backfill loading
A slight facepalm but, the main issue was a simple indexing logic error:
we need to slice with `tsdb_history[-shm._first.value:]` to push most
recent history not oldest.. This allows cleanup of tsdb backfill loop as
well.

Further, greatly simply `diff_history()` time slicing by using the
classic `numpy` conditional slice on the epoch field.
2023-01-10 11:09:19 -05:00