Commit Graph

3168 Commits (dea1c1c2d6a58a131138c0caacfedc2aef401668)

Author SHA1 Message Date
Tyler Goodlet 58b42d629f Passthrough fqsns list directly to `.load_symbols()` 2023-01-31 14:54:19 -05:00
Tyler Goodlet 36a81cb2de Only add plot to cursor set if not an overlay 2023-01-31 14:27:39 -05:00
Tyler Goodlet ae0f3118f4 Pass plotitem to axis from cursor 2023-01-31 14:27:39 -05:00
Tyler Goodlet 727c7ce2b1 Adjust L1 labels to expect `.pi: PlotItem` 2023-01-31 14:27:39 -05:00
Tyler Goodlet a39c980266 Allocate our internal `Axis` subtype in our `PlotItem` override 2023-01-31 14:27:39 -05:00
Tyler Goodlet 00be100e71 Initial chart widget adjustments for agg feeds
Main "public" API change is to make `GodWidget.get/set_chart_symbol()`
accept and cache-on fqsn tuples to allow handling overlayed chart groups
and adjust method names to be plural to match.

Wrt `LinkedSplits`,
- create all chart widget axes with a `None` plotitem argument and set
  the `.pi` field after axis creation (since apparently we have another
  object reference causality dilemma..)
- set a monkeyed `PlotItem.chart_widget` for use in axes that still need
  the widget reference.
- drop feed pause/resume for now since it's leaking feed tasks on the
  `brokerd` side and we probably don't really need it any more, and if
  we still do it should be done on the feed not the flume.

Wrt `ChartPlotItem`,
- drop `._add_sticky()` and use the `Axis` method instead and add some
  overlay + axis sanity checks.
- refactor `.draw_ohlc()` to be a lighter wrapper around a call to
  `.add_plot()`.
2023-01-31 14:27:39 -05:00
Tyler Goodlet 9217610734 Simplify OHLC graphic color instance var name 2023-01-31 14:27:39 -05:00
Tyler Goodlet 31af7a2c99 Add `Axis.add_sticky()` for creating axis labels
We have this method on our `ChartPlotWidget` but it makes more sense to
directly associate axis-labels with, well, the label's parent axis XD.

We add `._stickies: dict[str, YAxisLabel]` to replace
`ChartPlotWidget._ysticks` and pass in the `pg.PlotItem` to each axis
instance, stored as `Axis.pi` instead of handing around linked split
references (which are way out of scope for a single axis).

More work needs to be done to remove dependence on `.chart:
ChartPlotWidget` references in the date axis type as per comments.
2023-01-31 14:27:39 -05:00
Tyler Goodlet 34fac364fd Add default YAxisLable.x_offset: int` 2023-01-31 14:27:39 -05:00
goodboy dcdfd2577a
Merge pull request #447 from pikers/pregraphics_formatters
Pregraphics formatters: `IncrementalFormatter`
2023-01-31 13:55:04 -05:00
goodboy 6733dc57af
Merge pull request #441 from pikers/dark_clearing_repairs
Dark clearing repairs
2023-01-30 14:21:23 -05:00
Tyler Goodlet 05c4b6afb9 Drop px-cache-resets, failed try at path appends
Comments out the pixel-cache resetting since it doesn't seem we need it
any more to avoid draw oddities?

For `.fast_path` appends, this nearly got it working except the new path
segments are either not being connected correctly (step curve) or not
being drawn in full since the history path (plain line).

Leaving the attempted code commented in for a retry in the future; my
best guesses are that maybe,
- `.connectPath()` call is being done with incorrect segment length
  and/or start point.
- the "appended" data: `appended = array[-append_len-1:slice_to_head]`
  (done inside the formatter) isn't correct (i.e. endpoint handling
  considering a path append) and needs special handling for different
  curve types?
2023-01-30 13:22:24 -05:00
Tyler Goodlet 4b22325ffc Mask profile points and drop rect `.united()` attempts 2023-01-30 13:22:14 -05:00
Tyler Goodlet 9d16299f60 Make curve graphics timeframe agnostic
Ensure `.boundingRect()` calcs and `.draw_last_datum()` do geo-sizing
based on source data instead of presuming some `1.0` unit steps in some
spots; we need this to support an epoch index as is needed for overlays.

Further, clean out a bunch of old bounding rect calc code and add some
commented code for trying out `QRectF.united()` on the path + last datum
curve segment. Turns out that approach is slower as per eyeballing the
added profiler points.
2023-01-30 13:21:43 -05:00
Tyler Goodlet ab1f15506d Add graphics incr-updated "formatter" subsys
After trying to hack epoch indexed time series and failing miserably,
decided to properly factor out all formatting routines into a common
subsystem API: ``IncrementalFormatter`` which provides the interface for
incrementally updating and tracking pre-path-graphics formatted data.

Previously this functionality was mangled into our `Renderer` (which
also does the work of `QPath` generation and update) but splitting it
out also preps for being able to do graphics-buffer downsampling and
caching on a remote host B)

The ``IncrementalFormatter`` (parent type) has the default behaviour of
tracking a single field-array on some source `ShmArray`, updating
a flattened `numpy.ndarray` in-mem allocation, and providing a default
1d conversion for pre-downsampling and path generation.

Changed out of `Renderer`,
- `.allocate_xy()`, `update_xy()` and `format_xy()` all are moved to
  more explicitly named formatter methods.
- all `.x/y_data` nd array management and update
- "last view range" tracking
- `.last_read`, `.diff()`
- now calls `IncrementalFormatter.format_to_1d()` inside `.render()`

The new API gets,
- `.diff()`, `.last_read`
- all view range diff tracking through `.track_inview_range()`.
- better nd format array names: `.x/y_nd`, `xy_nd_start/stop`.
- `.format_to_1d()` which renders pre-path formatted arrays ready for
  both m4 sampling and path gen.
- better explicit overloadable formatting method names:
  * `.allocate_xy()` -> `.allocate_xy_nd()`
  * `.update_xy()` -> `.incr_update_xy_nd()`
  * `.format_xy()` -> `.format_xy_nd_to_1d()`

Finally this implements per-graphics-type formatters which define
each set up related formatting routines:
- `OHLCBarsFmtr`: std multi-line style bars
- `OHLCBarsAsCurveFmtr`: draws an interpolated line for ohlc sampled data
- `StepCurveFmtr`: handles vlm style curves
2023-01-30 13:20:17 -05:00
Tyler Goodlet 0db5451e47 Move all pre-path formatting routines to `._pathops`, proto formatter type 2023-01-30 13:19:33 -05:00
goodboy 61218f30f5
Merge pull request #440 from pikers/samplerd_service
`samplerd` service
2023-01-30 11:48:07 -05:00
Tyler Goodlet fcfc0f31f0 Enable backpressure in an effort to prevent bootup overruns 2023-01-30 11:45:29 -05:00
Tyler Goodlet 69074f4fa5 Bump up service tree spawn timeout a couple secs 2023-01-26 17:59:25 -05:00
Tyler Goodlet fe4fb37b58 Add service tree tests for data-feeds and the EMS 2023-01-24 15:15:27 -05:00
Tyler Goodlet 7cfd431a2b Yield `Services` in `open_test_pikerd()` fixture 2023-01-24 15:15:27 -05:00
Tyler Goodlet 61e20a86cc Fix clearing endpoint type annots, export `open_ems()` 2023-01-24 15:15:27 -05:00
Tyler Goodlet d9b73e1d08 Yield services (manager) from `maybe_open_pikerd()` 2023-01-24 15:15:27 -05:00
goodboy 4833d56ecb
Merge pull request #442 from pikers/misc_brokerd_backend_repairs
Misc brokerd backend repairs
2023-01-23 18:44:00 -05:00
Tyler Goodlet 090d1ba524 `kraken`: catch value error not index on missing `src_fiat` in pair 2023-01-23 15:36:20 -05:00
Tyler Goodlet afc45a8e16 `binance`: same thing, only unsub when connected 2023-01-23 15:29:24 -05:00
Tyler Goodlet 844626f6dc Move `brokerd` service task to root `.data` mod 2023-01-13 13:21:49 -05:00
Tyler Goodlet 470079665f Use new tractor kwargs getter func 2023-01-13 13:21:49 -05:00
Tyler Goodlet 0cd87d9e54 Drop commented markestored spawner code 2023-01-13 13:21:49 -05:00
Tyler Goodlet 09711750bf Registry subsys rework
More or less a revamp (and possibly first draft for something similar in
`tractor` core) which ensures all actor trees attempt to discover the
`pikerd` registry actor.

Implementation improvements include:
- new `Registry` singleton which houses the `pikerd` discovery
  socket-address `Registry.addr` + a `open_registry()` manager which
  provides bootstrapped actor-local access.
- refine `open_piker_runtime()` to do the work of opening a root actor
  and call the new `open_registry()` depending on whether a runtime has
  yet been bootstrapped.
- rejig `[maybe_]open_pikerd()` in terms of the above.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 71ca4c8e1f Use actor uid in shm keys for rt quote buffers
Allows running simultaneous data feed services on the same (linux) host
by avoiding file-name collisions instead keying shm buffer sets by the
given `brokerd` instance. This allows, for example, either multiple dev
versions of the data layer to run side-by-side or for the test suite to
be seamlessly run alongside a production instance.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 9811dcf5f3 Match `services` subcmd to new reg addr module variables 2023-01-13 13:21:49 -05:00
Tyler Goodlet da659cf607 Facepalm: definitely do not short circuit discovery helpers.. 2023-01-13 13:21:49 -05:00
Tyler Goodlet 37e0ec7b7d Assert fixture caller is `pikerd` 2023-01-13 13:21:49 -05:00
Tyler Goodlet 045b76bab5 Make `Flume.index_stream()` defer to new sampling api 2023-01-13 13:21:49 -05:00
Tyler Goodlet c8c641a038 Ensure all sub-services cancel on `pikerd` exit
Previously we were relying on implicit actor termination in
`maybe_spawn_daemon()` but really on `pikerd` teardown we should be sure
to tear down not only all service tasks in each actor but also the actor
runtimes. This adjusts `Services.cancel_service()` to only cancel the
service task scope and wait on the `complete` event and reworks the
`open_context_in_task()` inner closure body to,

- always cancel the service actor at exit.
- not call `.cancel_service()` (potentially causing recursion issues on
  cancellation).
- allocate a `complete: trio.Event` to signal full task + actor termination.
- pop the service task from the `.service_tasks` registry.

Further, add a `maybe_set_global_registry_sockaddr()` helper-cm to do
the work of checking whether a registry socket needs-to/has-been set
and use it for discovery calls to the `pikerd` service tree.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 6a1bb13feb Add base `pikerd` service tree custom check test 2023-01-13 13:21:49 -05:00
Tyler Goodlet 75591dd7e9 Don't raise on quote feed lags to dark clearing loop 2023-01-13 13:21:49 -05:00
Tyler Goodlet d792fed099 Move sync log msg back to info 2023-01-13 13:21:49 -05:00
Tyler Goodlet d66fb49077 Don't deliver shms from `start_backfill()`, they're not used 2023-01-13 13:21:49 -05:00
Tyler Goodlet 78c7c8524c Breakpoint when bad 1m history offsets are detected 2023-01-13 13:21:49 -05:00
Tyler Goodlet a746258f99 `binance`: always request an extra 1min OHLC bar
Seems that by default their history indexing rounds down/back to the
previous time step, so make sure we add a minute inside `Client.bars()`
when the `end_dt=None`, indicating "get the latest bar". Add
a breakpoint block that should trigger whenever the latest bar vs. the
latest epoch time is mismatched; we'll remove this after some testing
verifying the history bars issue is resolved.

Further this drops the legacy `backfill_bars()` endpoint which has been
deprecated and unused for a while.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 5adb234a24 Don't receive sample-index msgs in feed layer 2023-01-13 13:21:49 -05:00
Tyler Goodlet 2778ee1401 Support not registering for sample-index msgs via `sub_for_broadcasts: bool` flag 2023-01-13 13:21:49 -05:00
Tyler Goodlet e0ca5d5200 Use `open_sample_stream()` to increment fsp buffers 2023-01-13 13:21:47 -05:00
Tyler Goodlet b3d1b1aa63 Port feed layer to use new `samplerd` APIs
Always use `open_sample_stream()` to register fast and slow quote feed
buffers and get a sampler stream which we use to trigger
`Sampler.broadcast_all()` calls on the service side after backfill
events.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 5ec1a72a3d Implement a `samplerd` singleton actor service
Now spawned under the `pikerd` tree as a singleton-daemon-actor we offer
a slew of new routines in support of this micro-service:

- `maybe_open_samplerd()` and `spawn_samplerd()` which provide the
  `._daemon.Services` integration to conduct service spawning.
- `open_sample_stream()` which is a client-side endpoint which does all
  the work of (lazily) starting the `samplerd` service (if dne) and
  registers shm buffers for update as well as connect a sample-index
  stream for iterator by the caller.
- `register_with_sampler()` which is the `samplerd`-side service task
  endpoint implementing all the shm buffer and index-stream registry
  details as well as logic to ensure a lone service task runs
  `Services.increment_ohlc_buffer()`; it increments at the shortest period
  registered which, for now, is the default 1s duration.

Further impl notes:
- fixes to `Services.broadcast()` to ensure broken streams get discarded
  gracefully.
- we use a `pikerd` side singleton mutex `trio.Lock()` to ensure
  one-and-only-one `samplerd` is ever spawned per `pikerd` actor tree.
2023-01-13 13:21:15 -05:00
Tyler Goodlet a342f7d2d4 Make `._daemon.Services` for use as singleton
Drop the `_services` module level ref and adjust all client code to
match. Drop struct inheritance and convert all methods to class level.
Move `Brokerd.locks` -> `Services.locks` and add sampling mod to pikerd
enabled set.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 2c76cee928 Begin formalizing `Sampler` singleton API
We're moving toward a single actor managing sampler work and distributed
independently of `brokerd` services such that a user can run samplers on
different hosts then real-time data feed infra. Most of the
implementation details include aggregating `.data._sampling` routines
into a new `Sampler` singleton type.

Move the following methods to class methods:
- `.increment_ohlc_buffer()` to allow a single task to increment all
  registered shm buffers.
- `.broadcast()` for IPC relay to all registered clients/shms.

Further add a new `maybe_open_global_sampler()` which allocates
a service nursery and assigns it to the `Sampler.service_nursery`; this
is prep for putting the step incrementer in a singleton service task
higher up the data-layer actor tree.
2023-01-13 13:21:15 -05:00
Tyler Goodlet b5f2ff854c Drop meaning the clearing rate, use per step count 2023-01-13 13:21:15 -05:00