Commit Graph

3159 Commits (045a1a6acc93f86d2e22866599878cf8da4517f9)

Author SHA1 Message Date
Tyler Goodlet a8eee0f757 Adjust L1 labels to expect `.pi: PlotItem` 2023-01-30 11:48:51 -05:00
Tyler Goodlet d7166dd687 Allocate our internal `Axis` subtype in our `PlotItem` override 2023-01-30 11:48:51 -05:00
Tyler Goodlet d3dfebb965 Passthrough fqsns list directly to `.load_symbols()` 2023-01-30 11:48:51 -05:00
Tyler Goodlet ec48d2fbeb Initial chart widget adjustments for agg feeds
Main "public" API change is to make `GodWidget.get/set_chart_symbol()`
accept and cache-on fqsn tuples to allow handling overlayed chart groups
and adjust method names to be plural to match.

Wrt `LinkedSplits`,
- create all chart widget axes with a `None` plotitem argument and set
  the `.pi` field after axis creation (since apparently we have another
  object reference causality dilemma..)
- set a monkeyed `PlotItem.chart_widget` for use in axes that still need
  the widget reference.
- drop feed pause/resume for now since it's leaking feed tasks on the
  `brokerd` side and we probably don't really need it any more, and if
  we still do it should be done on the feed not the flume.

Wrt `ChartPlotItem`,
- drop `._add_sticky()` and use the `Axis` method instead and add some
  overlay + axis sanity checks.
- refactor `.draw_ohlc()` to be a lighter wrapper around a call to
  `.add_plot()`.
2023-01-30 11:48:51 -05:00
Tyler Goodlet 2bb2413391 Simplify OHLC graphic color instance var name 2023-01-30 11:48:51 -05:00
Tyler Goodlet a24acf48f8 Add `Axis.add_sticky()` for creating axis labels
We have this method on our `ChartPlotWidget` but it makes more sense to
directly associate axis-labels with, well, the label's parent axis XD.

We add `._stickies: dict[str, YAxisLabel]` to replace
`ChartPlotWidget._ysticks` and pass in the `pg.PlotItem` to each axis
instance, stored as `Axis.pi` instead of handing around linked split
references (which are way out of scope for a single axis).

More work needs to be done to remove dependence on `.chart:
ChartPlotWidget` references in the date axis type as per comments.
2023-01-30 11:48:51 -05:00
Tyler Goodlet e0f1520e0f Add default YAxisLable.x_offset: int` 2023-01-30 11:48:51 -05:00
Tyler Goodlet 05389bed12 Copy timestamps from source to FSP dest buffer 2023-01-30 11:48:51 -05:00
Tyler Goodlet 379679812c TOSQUASH? revert sym.lower() usage? 2023-01-30 11:48:51 -05:00
Tyler Goodlet bf0f7fafbb Init msg keys are always lower case 2023-01-30 11:48:51 -05:00
goodboy 61218f30f5
Merge pull request #440 from pikers/samplerd_service
`samplerd` service
2023-01-30 11:48:07 -05:00
Tyler Goodlet fcfc0f31f0 Enable backpressure in an effort to prevent bootup overruns 2023-01-30 11:45:29 -05:00
Tyler Goodlet 69074f4fa5 Bump up service tree spawn timeout a couple secs 2023-01-26 17:59:25 -05:00
Tyler Goodlet fe4fb37b58 Add service tree tests for data-feeds and the EMS 2023-01-24 15:15:27 -05:00
Tyler Goodlet 7cfd431a2b Yield `Services` in `open_test_pikerd()` fixture 2023-01-24 15:15:27 -05:00
Tyler Goodlet 61e20a86cc Fix clearing endpoint type annots, export `open_ems()` 2023-01-24 15:15:27 -05:00
Tyler Goodlet d9b73e1d08 Yield services (manager) from `maybe_open_pikerd()` 2023-01-24 15:15:27 -05:00
goodboy 4833d56ecb
Merge pull request #442 from pikers/misc_brokerd_backend_repairs
Misc brokerd backend repairs
2023-01-23 18:44:00 -05:00
Tyler Goodlet 090d1ba524 `kraken`: catch value error not index on missing `src_fiat` in pair 2023-01-23 15:36:20 -05:00
Tyler Goodlet afc45a8e16 `binance`: same thing, only unsub when connected 2023-01-23 15:29:24 -05:00
Tyler Goodlet 844626f6dc Move `brokerd` service task to root `.data` mod 2023-01-13 13:21:49 -05:00
Tyler Goodlet 470079665f Use new tractor kwargs getter func 2023-01-13 13:21:49 -05:00
Tyler Goodlet 0cd87d9e54 Drop commented markestored spawner code 2023-01-13 13:21:49 -05:00
Tyler Goodlet 09711750bf Registry subsys rework
More or less a revamp (and possibly first draft for something similar in
`tractor` core) which ensures all actor trees attempt to discover the
`pikerd` registry actor.

Implementation improvements include:
- new `Registry` singleton which houses the `pikerd` discovery
  socket-address `Registry.addr` + a `open_registry()` manager which
  provides bootstrapped actor-local access.
- refine `open_piker_runtime()` to do the work of opening a root actor
  and call the new `open_registry()` depending on whether a runtime has
  yet been bootstrapped.
- rejig `[maybe_]open_pikerd()` in terms of the above.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 71ca4c8e1f Use actor uid in shm keys for rt quote buffers
Allows running simultaneous data feed services on the same (linux) host
by avoiding file-name collisions instead keying shm buffer sets by the
given `brokerd` instance. This allows, for example, either multiple dev
versions of the data layer to run side-by-side or for the test suite to
be seamlessly run alongside a production instance.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 9811dcf5f3 Match `services` subcmd to new reg addr module variables 2023-01-13 13:21:49 -05:00
Tyler Goodlet da659cf607 Facepalm: definitely do not short circuit discovery helpers.. 2023-01-13 13:21:49 -05:00
Tyler Goodlet 37e0ec7b7d Assert fixture caller is `pikerd` 2023-01-13 13:21:49 -05:00
Tyler Goodlet 045b76bab5 Make `Flume.index_stream()` defer to new sampling api 2023-01-13 13:21:49 -05:00
Tyler Goodlet c8c641a038 Ensure all sub-services cancel on `pikerd` exit
Previously we were relying on implicit actor termination in
`maybe_spawn_daemon()` but really on `pikerd` teardown we should be sure
to tear down not only all service tasks in each actor but also the actor
runtimes. This adjusts `Services.cancel_service()` to only cancel the
service task scope and wait on the `complete` event and reworks the
`open_context_in_task()` inner closure body to,

- always cancel the service actor at exit.
- not call `.cancel_service()` (potentially causing recursion issues on
  cancellation).
- allocate a `complete: trio.Event` to signal full task + actor termination.
- pop the service task from the `.service_tasks` registry.

Further, add a `maybe_set_global_registry_sockaddr()` helper-cm to do
the work of checking whether a registry socket needs-to/has-been set
and use it for discovery calls to the `pikerd` service tree.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 6a1bb13feb Add base `pikerd` service tree custom check test 2023-01-13 13:21:49 -05:00
Tyler Goodlet 75591dd7e9 Don't raise on quote feed lags to dark clearing loop 2023-01-13 13:21:49 -05:00
Tyler Goodlet d792fed099 Move sync log msg back to info 2023-01-13 13:21:49 -05:00
Tyler Goodlet d66fb49077 Don't deliver shms from `start_backfill()`, they're not used 2023-01-13 13:21:49 -05:00
Tyler Goodlet 78c7c8524c Breakpoint when bad 1m history offsets are detected 2023-01-13 13:21:49 -05:00
Tyler Goodlet a746258f99 `binance`: always request an extra 1min OHLC bar
Seems that by default their history indexing rounds down/back to the
previous time step, so make sure we add a minute inside `Client.bars()`
when the `end_dt=None`, indicating "get the latest bar". Add
a breakpoint block that should trigger whenever the latest bar vs. the
latest epoch time is mismatched; we'll remove this after some testing
verifying the history bars issue is resolved.

Further this drops the legacy `backfill_bars()` endpoint which has been
deprecated and unused for a while.
2023-01-13 13:21:49 -05:00
Tyler Goodlet 5adb234a24 Don't receive sample-index msgs in feed layer 2023-01-13 13:21:49 -05:00
Tyler Goodlet 2778ee1401 Support not registering for sample-index msgs via `sub_for_broadcasts: bool` flag 2023-01-13 13:21:49 -05:00
Tyler Goodlet e0ca5d5200 Use `open_sample_stream()` to increment fsp buffers 2023-01-13 13:21:47 -05:00
Tyler Goodlet b3d1b1aa63 Port feed layer to use new `samplerd` APIs
Always use `open_sample_stream()` to register fast and slow quote feed
buffers and get a sampler stream which we use to trigger
`Sampler.broadcast_all()` calls on the service side after backfill
events.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 5ec1a72a3d Implement a `samplerd` singleton actor service
Now spawned under the `pikerd` tree as a singleton-daemon-actor we offer
a slew of new routines in support of this micro-service:

- `maybe_open_samplerd()` and `spawn_samplerd()` which provide the
  `._daemon.Services` integration to conduct service spawning.
- `open_sample_stream()` which is a client-side endpoint which does all
  the work of (lazily) starting the `samplerd` service (if dne) and
  registers shm buffers for update as well as connect a sample-index
  stream for iterator by the caller.
- `register_with_sampler()` which is the `samplerd`-side service task
  endpoint implementing all the shm buffer and index-stream registry
  details as well as logic to ensure a lone service task runs
  `Services.increment_ohlc_buffer()`; it increments at the shortest period
  registered which, for now, is the default 1s duration.

Further impl notes:
- fixes to `Services.broadcast()` to ensure broken streams get discarded
  gracefully.
- we use a `pikerd` side singleton mutex `trio.Lock()` to ensure
  one-and-only-one `samplerd` is ever spawned per `pikerd` actor tree.
2023-01-13 13:21:15 -05:00
Tyler Goodlet a342f7d2d4 Make `._daemon.Services` for use as singleton
Drop the `_services` module level ref and adjust all client code to
match. Drop struct inheritance and convert all methods to class level.
Move `Brokerd.locks` -> `Services.locks` and add sampling mod to pikerd
enabled set.
2023-01-13 13:21:15 -05:00
Tyler Goodlet 2c76cee928 Begin formalizing `Sampler` singleton API
We're moving toward a single actor managing sampler work and distributed
independently of `brokerd` services such that a user can run samplers on
different hosts then real-time data feed infra. Most of the
implementation details include aggregating `.data._sampling` routines
into a new `Sampler` singleton type.

Move the following methods to class methods:
- `.increment_ohlc_buffer()` to allow a single task to increment all
  registered shm buffers.
- `.broadcast()` for IPC relay to all registered clients/shms.

Further add a new `maybe_open_global_sampler()` which allocates
a service nursery and assigns it to the `Sampler.service_nursery`; this
is prep for putting the step incrementer in a singleton service task
higher up the data-layer actor tree.
2023-01-13 13:21:15 -05:00
Tyler Goodlet b5f2ff854c Drop meaning the clearing rate, use per step count 2023-01-13 13:21:15 -05:00
Tyler Goodlet 3efb0b5884 Sync 1s (or less) sampler steps using rounded now-epoch 2023-01-13 13:21:15 -05:00
Tyler Goodlet 009bbe456e Always `.error()` log unknown queries for `marketstore` 2023-01-13 13:21:15 -05:00
Tyler Goodlet daf7b3f4a5 Only accept 6 tries for the same duplicate hist frame
When we see multiple history frames that are duplicate to the request
set, bail re-trying after a number of tries (6 just cuz) and return
early from the tsdb backfill loop; presume that this many duplicates
means we've hit the beginning of history. Use a `collections.Counter`
for the duplicate counts. Make sure and warn log in such cases.
2023-01-13 13:21:15 -05:00
Tyler Goodlet b0a6dd46e4 Use recon set on stack closing during reconnect
Hopefully resolves https://github.com/pikers/piker/issues/434
2023-01-13 13:21:15 -05:00
Tyler Goodlet 1c5141f4c6 Fix f-str in duplicate frame msg print 2023-01-13 13:21:15 -05:00
Tyler Goodlet 4cdd2271b0 Drop `tractor` assert bug note 2023-01-13 13:21:15 -05:00