Commit Graph

2095 Commits (8d77d177bfe6e066b092d3abb1bbf1570ead4c7d)

Author SHA1 Message Date
Tyler Goodlet 8d77d177bf Move ib data reset script into a new `scripts/` dir 2022-04-16 15:39:24 -04:00
Tyler Goodlet 36463dddbe Use new `tractor.query_actor()` for service checking 2022-04-16 15:39:24 -04:00
Tyler Goodlet e399b74b67 Return all timeframe arrays if `timeframe` not passed as input 2022-04-16 15:39:24 -04:00
Tyler Goodlet 15b609bf13 Make `pikerd` work again without `--tsdb` flag 2022-04-16 15:39:24 -04:00
Tyler Goodlet dfc3fb76fd Add a service checker predicate 2022-04-16 15:39:24 -04:00
Tyler Goodlet 881b1afc12 Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-04-16 15:39:24 -04:00
Tyler Goodlet 9878735ff3 Proxy `marketstore` container log level to our own 2022-04-16 15:39:24 -04:00
Tyler Goodlet 1389ee51f3 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-04-16 15:39:24 -04:00
Tyler Goodlet 6225e8a58e Doc str formatting 2022-04-16 15:39:24 -04:00
Tyler Goodlet 2b48943066 Add latency measures around diffs/writes to mkts 2022-04-16 15:39:24 -04:00
Tyler Goodlet 92d074f308 Prototype out writing `1Sec` OHLCV data 2022-04-16 15:39:24 -04:00
Tyler Goodlet 49a280ce14 Better doc string 2022-04-16 15:39:24 -04:00
Tyler Goodlet 42e5f48345 Persist backing `/data/` filesystem across container runs 2022-04-16 15:39:24 -04:00
Tyler Goodlet 18344603e6 Get basic OHLCV writes working with `anyio` client 2022-04-16 15:39:24 -04:00
Tyler Goodlet ae1d9efa97 Pass in daemon name to `start_ahab()` 2022-04-16 15:39:24 -04:00
Tyler Goodlet f1747749e7 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-04-16 15:39:24 -04:00
Tyler Goodlet cdd03759bf Revive `ms-shell` sub-cmd 2022-04-16 15:39:24 -04:00
Tyler Goodlet 00a36826c2 Add WIP backfiller from data feed helper 2022-04-16 15:39:24 -04:00
Tyler Goodlet 3aa8044aa1 Better handle nested erros from docker client 2022-04-16 15:39:24 -04:00
Tyler Goodlet f0ae7edb00 Add back in legacy write loop for reference 2022-04-16 15:39:24 -04:00
Tyler Goodlet 797ba15923 Add back in OHLCV dtype template and client side ws streamer 2022-04-16 15:39:24 -04:00
Tyler Goodlet b8f86d4599 Drop ununsed `Services` ref 2022-04-16 15:39:24 -04:00
Tyler Goodlet 908093264a Py3.9+ type updates 2022-04-16 15:39:24 -04:00
Tyler Goodlet 3ef68f78f7 Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-04-16 15:39:24 -04:00
Tyler Goodlet d095c21885 De-escalate sudo perms in `pikerd` once docker spawns 2022-04-16 15:39:24 -04:00
Tyler Goodlet 52dc0989ac Handle the non-root perms case specifically too 2022-04-16 15:39:24 -04:00
Tyler Goodlet 6f749c340d Add explicit no-docker error and supervisor start task-func 2022-04-16 15:39:24 -04:00
Tyler Goodlet dcff80131f Type annot updates 2022-04-16 15:39:24 -04:00
Tyler Goodlet 6cd829d1fd Drop old client instantiate line 2022-04-16 15:39:24 -04:00
Tyler Goodlet 76fff18b1a Drop import, it's got madness with and SIGINT? 2022-04-16 15:39:24 -04:00
Tyler Goodlet 88e1befffe Add a super simple `marketstore` container supervisor 2022-04-16 15:39:24 -04:00
Tyler Goodlet c1579b630e Extract non-sudo user for config dir path 2022-04-16 15:39:24 -04:00
Tyler Goodlet 7dfdd0f00b Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez 14a893a5e5 Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez c91599472d Simplify and optimize tick format, similar to techtonicdb's 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez 9e4dec98da Add multi ingestor support and update to new feed API 2022-04-16 15:39:24 -04:00
Tyler Goodlet 8800ee0661 Only update y mxmn from L1 when last index in view 2022-04-16 15:39:24 -04:00
Tyler Goodlet 885116ae46 Only update y-range from L1 mxmn when last index in view
We still have to always keep track of the last max and min
though.
2022-04-16 15:38:26 -04:00
Tyler Goodlet eab9127170 Drop old `pyqtgraph` downsample code 2022-04-16 15:38:26 -04:00
Tyler Goodlet c675773de4 Tweak log-scaler for more detail 2022-04-16 15:38:26 -04:00
Tyler Goodlet b20245398d Drop `._ic` debugging prints 2022-04-16 15:38:26 -04:00
Tyler Goodlet 2abb3ec84d Display loop mega-cleanup
The most important changes include:
- iterating the new `Flow` type and updating graphics
- adding detailed profiling
- increasing the min uppx before graphics updates are throttled
- including the L1 spread in y-range calcs so that you never have the
  bid/ask go "out of view"..
- pass around `Flow`s instead of shms
- drop all the old prototyped downsampling code
2022-04-16 15:38:26 -04:00
Tyler Goodlet 6843f9a515 Allow passing a `plotItem` to `.draw_curve()`
If manually managing an overlay you'll likely call `.overlay_plotitem()`
and then a plotting method so we need to accept a plot item input so
that the chart's pi doesn't get assigned incorrectly in the `Flow` entry
(though it is by default if no input is provided).

More,
- add a `Flow.graphics` field and set it to the `pg.GraphicsObject`.
- make `Flow.maxmin()` return `None` in the "can't calculate" cases.
2022-04-16 15:38:26 -04:00
Tyler Goodlet 5002e78b81 Fsp UI initialization updates
- set shm refs on `Flow` entries.
- don't run a graphics cycle on 'update' msgs from the engine
  if the containing chart is hidden.
- drop `volume` from flows map and disable auto-yranging
  once $vlm comes up.
2022-04-16 15:38:26 -04:00
Tyler Goodlet 5446cbd335 Disconnect signals in `ChartView.disable_auto_yrange()`
Allows for removing resize callbacks for a flow/overlay that you wish to
remove from view (eg. unit volume after dollar volume is up) and thus
less general interaction callback overhead for any plot you don't wish
to show or resize.

Further,
- drop the `autoscale_linked_plots` block for now since with
  multi-view-box overlays each register their own vb resize slots
- pull the graphics object from the chart's `Flow` map inside
  `.maybe_downsample_graphics()`
2022-04-16 15:38:26 -04:00
Tyler Goodlet 7870e759ff Fix bug where if `yrange` was passed the mxmin callback was still used.. 2022-04-16 15:38:26 -04:00
Tyler Goodlet dfed6cb8e9 Guard against zero px width 2022-04-16 15:38:26 -04:00
Tyler Goodlet b30b4bb555 Add `Flow` type with a real chitty mxmn cacheing method
This new type wraps a shm data flow and will eventually include things
like incremental path-graphics updates and serialization + bg downsampling
techniques. The main immediate motivation was to get a cached y-range max/min
calc going since profiling revealed the `numpy` equivalents were
actually quite slow as the data set grows large. Likely we can use all
this to drive a streaming mx/mn routine that's always launched as part
of each on-host flow.

This is our official foray into use of `msgspec.Struct` B) and I have to
say, pretty impressed; we'll likely completely ditch `pydantic` from
here on out.
2022-04-16 15:38:26 -04:00
Tyler Goodlet 514bbb1a98 Loop for first graphic with xvec 2022-04-16 15:38:26 -04:00
Tyler Goodlet 918789d1a3 Only `.maybe_downsample_graphics()` on manual changes
We don't need update graphics on every x-range change since that's what
the display loop does. Instead, only on manual changes do we make manual
calls into `.update_graphics_from_array()` and be sure to iterate all
linked subplots and all their embedded graphics.
2022-04-16 15:38:26 -04:00