Commit Graph

3471 Commits (98b7d784760f5947263b18ac276d38717fd8132d)

Author SHA1 Message Date
Tyler Goodlet 52ac1053aa 2nd try: dispersion normalize y-ranges around median
In the dispersion swing calcs, use the series median from the in-view
data to determine swing proportions to apply on each "minor curve"
(series with lesser dispersion the one with the greatest). Track the
major `Viz` as before by max dispersion. Apply the dispersion swing
proportions to each minor curve-series in a third loop/pass of all
overlay groups: this ensures all overlays are dispersion normalized in
their ranges but, minor curves are currently (vertically) centered (vs.
the major) via their medians.

There is a ton of commented code from attempts to try and vertically
align minor curves to the major via the "first datum" in-view/available.
This still needs work and we may want to offer it as optional.

Also adds logic to allow skipping margin adjustments in `._set_yrange()`
if you pass `range_margin=None`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet dfc35253ea First draft, group y-minmax transform algo
On overlaid ohlc vizs we compute the largest max/min spread and
apply that maxmimum "up and down swing" proportion to each `Viz`'s
viewbox in the group.

We obviously still need to clip to the shortest x-range so that
it doesn't look exactly the same as before XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8a5b9f4e8c Rename `.maybe_downsample_graphics()` -> `.interact_graphics_cycle()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet f89e11fc7d Right, handle y-ranging multiple paths per plot
We were hacking this before using the whole `ChartView._maxmin()`
setting stuff since in some cases you might want similarly ranged paths
on the same view, but of course you need to max/min them together..

This adds that group sorting by using a table of `dict[PlotItem,
tuple[float, float]` and taking the abs highest/lowest value for each
plot in the viz interaction update loop.

Also removes the now commented signal registry calls and thus
`._yranger`, drops the `set_range: bool` from `._set_yrange` and adds
and extra `.maybe_downsample_graphics()` to the mouse wheel handler to
avoid a weird slow debounce where ds-ing is delayed until a further
interaction.
2023-03-10 18:20:22 -05:00
Tyler Goodlet fc73becd5f Drop Qt interaction signal usage
It's kind of hard to understand with the C++ fan-out to multiple views
(imo a cluster-f#$*&) and seems honestly just plain faster to loop (in
python) through all the linked view handlers XD

Core adjustments:
- make the panning and wheel-scroll handlers just call
  `.maybe_downsample_graphics()` directly; drop all signal emissions.
- make `.maybe_downsample_graphics()` loop through all vizs per subchart
  and use the new pipeline-style call sequence of:
  - `Viz.update_graphics() -> <read_slc>: tuple`
  - `Viz.maxmin(i_read_range=<read_slc>) -> yrange: tuple`
  - `Viz.plot.vb._set_yrange(yrange=yrange)`
  which inlines all the necessary calls in the most efficient way whilst
  leveraging `.maxmin()` caching and ymxmn-from-m4-during-render to
  boot.
- drop registering `._set_yrange()` for handling `.sigRangeChangedManually`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 223e9d999c Add first-draft `PlotItemOverlay.group_maxmin()`
Computes the maxmin values for each underlying plot's in-view range as
well as the max up/down swing (in percentage terms) from the plot with
most dispersion and returns a all these values plus a `dict` of plots to
their ranges as part of output.
2023-03-10 18:20:22 -05:00
goodboy eb51033b18
Merge pull request #483 from pikers/service_subpkg
`.service` subpkg
2023-03-10 10:37:46 -05:00
Tyler Goodlet 12883c3c90 Don't double send `enable_modules` and `debug_mode` in kwargs..
This broke non-disti-mode actor tree spawn / runtime, seemingly because
the cli entrypoint for a `piker chart` also sends these values down
through the call stack independently? Pretty sure we don't need to send
the `enable_modules` from the chart actor anyway.
2023-03-10 10:30:26 -05:00
Tyler Goodlet 8ceaa27872 Add ES client polling to ensure eventual connectivity.. 2023-03-09 18:42:24 -05:00
Tyler Goodlet 97290fcb05 Never drop root perms in test harness 2023-03-09 18:42:24 -05:00
Tyler Goodlet 44a3115539 Expose `drop_root_perms_for_ahab` from `pikerd` factories to `ahabd` 2023-03-09 18:42:24 -05:00
Tyler Goodlet 0772b4a0fa Hard code version from our container, predicate renaming 2023-03-09 18:42:24 -05:00
Tyler Goodlet 15064d94cb `ahabd`: Harden cancellation teardown (again XD)
Needed to move the startup sequence inside the `try:` block to guarantee
we always do the (now shielded) `.cancel()` call if we get a cancel
during startup.

Also, support an optional `started_afunc` field in the config if
backends want to just provide a one-off blocking async func to sync
container startup. Add a `drop_root_perms: bool` to allow persisting
sudo perms for testing or dyanmic container spawning purposes.
2023-03-09 18:42:24 -05:00
Tyler Goodlet 9a00c45923 Add `log` fixture for easy test plugin 2023-03-09 18:42:24 -05:00
Tyler Goodlet 7cc9911565 Add connection poll loop to es test as well 2023-03-09 15:37:43 -05:00
Tyler Goodlet 79b0db4449 Pass a config `tmp_dir: Path` to the runtime when testing 2023-03-09 15:37:43 -05:00
Tyler Goodlet 5aaa7f47dc Pull testing config dir from `tractor` runtime vars
Provides a more correct solution (particularly for distributed testing)
to override the `piker` configuration directory by reading the path from
a specific `tractor._state._runtime_vars` entry that can be provided by
the test harness.

Also fix some typing and comments.
2023-03-09 15:37:43 -05:00
Tyler Goodlet aa36abf36e Support passing `tractor` "actor runtime vars" down the runtime 2023-03-09 15:37:43 -05:00
Tyler Goodlet 2014019b06 Add reconnect loop to `marketstore` startup test
Due to making ahabd supervisor init more async we need to be more
tolerant to mkts server startup: the grpc machinery needs to be up
otherwise a client which connects to early may just hang on requests..

Add a reconnect loop (which might end up getting factored into client
code too) so that we only block on requests once we know the client
connection is actually responsive.
2023-03-09 15:37:43 -05:00
Tyler Goodlet 75b7a8b56e `marketstore`: Pull default socket from server config 2023-03-09 15:37:43 -05:00
Tyler Goodlet 31392af427 Move enabled module defs to just above where used 2023-03-09 15:37:43 -05:00
Tyler Goodlet 6540c415c1 Lul, fix imports in elasticsearch block.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet fbc12b1b07 Add 10min timeout on CI job.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet cda7a54718 Fix final missed `marketstore` mod import
Thanks @esme! XD

Also, do a linter pass and remove a buncha unused references.
2023-03-09 15:37:43 -05:00
Tyler Goodlet 6f92c6b52d Don't crash on a `xdotool` timeout.. 2023-03-09 15:37:43 -05:00
Tyler Goodlet 441243f83b Attempt to report `piker storage -d <fqsn>` errors
Not really sure there's much we can do besides dump Grpc stuff when we
detect an "error" `str` for the moment..

Either way leave a buncha complaints (como siempre) and do linting
fixups..
2023-03-09 15:37:43 -05:00
Tyler Goodlet cec2967071 Import `maybe_open_pikerd` at module level 2023-03-09 15:37:43 -05:00
Tyler Goodlet f95ea19b21 Move `pikerd` runtime boostrap to `.service._actor_runtime` 2023-03-09 15:37:43 -05:00
Tyler Goodlet eca048c0c5 Move daemon spawning endpoints to `service._daemon` module 2023-03-09 15:37:43 -05:00
Tyler Goodlet a2d40937a3 Move actor-discovery utils to `.service._registry 2023-03-09 15:37:42 -05:00
Tyler Goodlet 31f2b01c3e Move `Services` api to `.service._mngr` mod 2023-03-09 15:37:42 -05:00
Tyler Goodlet b226b678e9 Fix missed `marketstore` mod imports 2023-03-09 15:37:42 -05:00
Tyler Goodlet dd87d1142e Bump mkts timeout to 2s 2023-03-09 15:37:42 -05:00
Tyler Goodlet afac553ea2 Move all docker and external db code to `piker.service` 2023-03-09 15:37:42 -05:00
Tyler Goodlet 93c81fa4d1 Start `piker.service` sub-package
For now just moves everything that was in `piker._daemon` to a subpkg
module but a reorg is coming pronto!
2023-03-09 15:37:42 -05:00
Tyler Goodlet bfe3ea1f59 Set explicit `marketstore` container startup timeout 2023-03-09 15:37:42 -05:00
Tyler Goodlet 56629b6b2e Hardcode `cancel` log level for `ahabd` for now 2023-03-09 15:37:42 -05:00
Tyler Goodlet bb723abc9d Always passthrough loglevel to `ahabd` supervisor 2023-03-09 15:37:42 -05:00
Tyler Goodlet 7694419e71 Background docker-container logs processing
Previously we would make the `ahabd` supervisor-actor sync to docker
container startup using pseudo-blocking log message processing.

This has issues,
- we're forced to do a hacky "yield back to `trio`" in order to be
  "fake async" when reading the log stream and further,
- blocking on a message is fragile and often slow.

Instead, run the log processor in a background task and in the parent
task poll for the container to be in the client list using a similar
pseudo-async poll pattern. This allows the super to `Context.started()`
sooner (when the container is actually registered as "up") and thus
unblock its (remote) caller faster whilst still doing full log msg
proxying!

Deatz:
- adds `Container.cuid: str` a unique container id for logging.
- correctly proxy through the `loglevel: str` from `pikerd` caller task.
- shield around `Container.cancel()` in the teardown block and use
  cancel level logging in that method.
2023-03-09 15:37:42 -05:00
Tyler Goodlet b078a06621 Doc string and types bump in loggin mod 2023-03-09 15:37:42 -05:00
Tyler Goodlet 05b67c27d0 Apply `Services` runtime state **immediately** inside starup block 2023-03-09 15:37:42 -05:00
Tyler Goodlet 8c66f066bd Deliver es specific ahab-super in endpoint startup config 2023-03-09 15:37:42 -05:00
Tyler Goodlet 959e423849 Add warning around detach flag to docker client 2023-03-09 15:37:42 -05:00
Tyler Goodlet 7b196b1b97 Support startup-config overrides to `ahabd` super
With the addition of a new `elastixsearch` docker support in
https://github.com/pikers/piker/pull/464, adjustments were made
to container startup sync logic (particularly the `trio` checkpoint
sleep period - which itself is a hack around a sync client api) which
caused a regression in upstream startup logic wherein container error
logs were not being bubbled up correctly causing a silent failure mode:

- `marketstore` container started with corrupt input config
- `ahabd` super code timed out on startup phase due to a larger log
  polling period, skipped processing startup logs from the container,
  and continued on as though the container was started
- history client fails on grpc connection with no clear error on why the
  connection failed.

Here we revert to the old poll period (1ms) to avoid any more silent
failures and further extend supervisor control through a configuration
override mechanism. To address the underlying design issue, this patch
adds support for container-endpoint-callbacks to override supervisor
startup configuration parameters via the 2nd value in their returned
tuple: the already delivered configuration `dict` value.

The current exposed values include:
    {
        'startup_timeout': 1.0,
        'startup_query_period': 0.001,
        'log_msg_key': 'msg',
    },

This allows for container specific control over the startup-sync query
period (the hack mentioned above)  as well as the expected log msg key
and of course the startup timeout.
2023-03-09 15:37:42 -05:00
Tyler Goodlet fe0695fb7b First draft storage layer cli
Adds a `piker storage` subcmd with a `-d` flag to wipe a particular
fqsn's time series (both 1s and 60s). Obviously this needs to be
extended much more but provides a start point.
2023-03-09 15:37:42 -05:00
jaredgoldman dae8e59d26
Merge pull request #484 from pikers/pps_precision_hotfixes
Pps precision hotfixes
2023-03-08 19:50:09 -05:00
Tyler Goodlet aba238e8b1 `kraken`: expect `Pair` in search results.. 2023-03-08 17:22:34 -05:00
Tyler Goodlet d3192bb8c2 Read `Symbol` tick precision fields when no entry in `.broker_info` 2023-03-08 17:22:34 -05:00
goodboy 6cd18576aa
Merge pull request #474 from pikers/xdo_and_you
`ib`: restore and (maybe) use `xdotool` + `i3ipc` reset method
2023-03-03 17:42:29 -05:00
Tyler Goodlet daa6a5c80a `ib`: restore and (maybe) use `xdotool` + `i3ipc` reset method
Since apparently the container we were using is totally borked on new
kernels and/or latest jvm, this move our old manual local-X-desktop script
back for use in `brokerd` backend code.

Adds a new `.brokers.ib._util` which contains the 2 methods and fails
over to this one when we can't connect to a VNC server. Also adjusts the
original in `scripts/ib_data_reset.py` to import and run the module code
as a script-program.
2023-03-03 17:37:26 -05:00