Commit Graph

2122 Commits (marketstore)

Author SHA1 Message Date
Tyler Goodlet 381fa11458 More reliable `marketstored` + container supervision
It turns out (i guess not so shockingly?) that `marketstore` doesn't
always teardown "gracefully" under SIGINT (seems to hang if there are
open client connections which are also in the midst of teardown?) so
this instead first tries the SIGINT and then fails over to a SIGKILL
(destroy loop) which seems to be much more reliable to ensure shutdown
without any downside - in terms of a "hard kill".

Originally i was thinking the issue was root perms related (which get
relegated solely to the `marketstored` daemon actor after spawn) but
actually it was indeed the signalling / application layer causing the
hold-up/latency on teardown. There's a bunch of lingering (now
commented) code which tried to solve this non-problem as well as a bunch
logging/prints to help decipher the root of the issue - this will all
get cleaned out shortly.
2022-05-09 11:09:00 -04:00
Tyler Goodlet 7bb54f61b7 Add `docker` as `tsdb` extras dep 2022-05-09 11:09:00 -04:00
Tyler Goodlet c2c3516b34 Add `anyio-marketstore` client as dev dep 2022-05-09 11:09:00 -04:00
Tyler Goodlet 1f58f5daba Handle non-fqsn for derivs and don't put brokername in 2022-05-09 11:09:00 -04:00
Tyler Goodlet eb4b63a710 Limit ohlc queries to 800k datums to avoid `purepc` size error 2022-05-09 11:09:00 -04:00
Tyler Goodlet 6971eca46c Get sync-to-marketstore-tsdb history retrieval workinnn 2022-05-09 11:09:00 -04:00
Tyler Goodlet 033a32cff1 Handle "fatal" level log msgs in docker super 2022-05-09 11:09:00 -04:00
Tyler Goodlet df8054b7a4 Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-05-09 11:09:00 -04:00
Tyler Goodlet 288eda195f Drop `ms-shell`, add `piker storesh` cmd 2022-05-09 11:09:00 -04:00
Tyler Goodlet 17f469d619 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-05-09 11:09:00 -04:00
Tyler Goodlet 94cba54beb Drop `pandas` to `numpy` converter 2022-05-09 11:09:00 -04:00
Tyler Goodlet 1c6a46da5d Disable re-connect for now in ib script 2022-05-09 11:09:00 -04:00
Tyler Goodlet 54afa30759 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-05-09 11:09:00 -04:00
Tyler Goodlet 62ef081694 Try downsampling mkts data 2022-05-09 11:09:00 -04:00
Tyler Goodlet 279516959f Comment each special key combo 2022-05-09 11:09:00 -04:00
Tyler Goodlet 1c3457d829 Load any symbol-matching shm array if no `marketstored` found 2022-05-09 11:09:00 -04:00
Tyler Goodlet 97211a0a7e Get ib key hack script to work with reconnect 2022-05-09 11:09:00 -04:00
Tyler Goodlet 1115d06e33 Move ib data reset script into a new `scripts/` dir 2022-05-09 11:09:00 -04:00
Tyler Goodlet c874b6aca0 Use new `tractor.query_actor()` for service checking 2022-05-09 11:09:00 -04:00
Tyler Goodlet 531989a9e5 Return all timeframe arrays if `timeframe` not passed as input 2022-05-09 11:09:00 -04:00
Tyler Goodlet 7f5730d222 Make `pikerd` work again without `--tsdb` flag 2022-05-09 11:09:00 -04:00
Tyler Goodlet 38dc9cc99b Add a service checker predicate 2022-05-09 11:09:00 -04:00
Tyler Goodlet c63eb91f6f Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-05-09 11:09:00 -04:00
Tyler Goodlet 981735669c Proxy `marketstore` container log level to our own 2022-05-09 11:09:00 -04:00
Tyler Goodlet f1b73244f2 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-05-09 11:09:00 -04:00
Tyler Goodlet d1599c4f6c Doc str formatting 2022-05-09 11:09:00 -04:00
Tyler Goodlet 5a69ab800f Add latency measures around diffs/writes to mkts 2022-05-09 11:09:00 -04:00
Tyler Goodlet 1623d4ed83 Prototype out writing `1Sec` OHLCV data 2022-05-09 11:09:00 -04:00
Tyler Goodlet 96b7d55018 Better doc string 2022-05-09 11:09:00 -04:00
Tyler Goodlet b6282bc485 Persist backing `/data/` filesystem across container runs 2022-05-09 11:09:00 -04:00
Tyler Goodlet e81e83821e Get basic OHLCV writes working with `anyio` client 2022-05-09 11:09:00 -04:00
Tyler Goodlet 4af42ee1b5 Pass in daemon name to `start_ahab()` 2022-05-09 11:09:00 -04:00
Tyler Goodlet 3eeb1caf32 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-05-09 11:09:00 -04:00
Tyler Goodlet a5c1febd12 Revive `ms-shell` sub-cmd 2022-05-09 11:09:00 -04:00
Tyler Goodlet ab2d721be4 Add WIP backfiller from data feed helper 2022-05-09 11:09:00 -04:00
Tyler Goodlet 4c38d0246e Better handle nested erros from docker client 2022-05-09 11:09:00 -04:00
Tyler Goodlet ecc3613654 Add back in legacy write loop for reference 2022-05-09 11:08:59 -04:00
Tyler Goodlet 216ad65933 Add back in OHLCV dtype template and client side ws streamer 2022-05-09 11:08:59 -04:00
Tyler Goodlet ec501da681 Drop ununsed `Services` ref 2022-05-09 11:08:59 -04:00
Tyler Goodlet 218449a9ef Py3.9+ type updates 2022-05-09 11:08:59 -04:00
Tyler Goodlet 973f0e6180 Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-05-09 11:08:59 -04:00
Tyler Goodlet 5a592d5f42 De-escalate sudo perms in `pikerd` once docker spawns 2022-05-09 11:08:59 -04:00
Tyler Goodlet d6ffe1ba7a Handle the non-root perms case specifically too 2022-05-09 11:08:59 -04:00
Tyler Goodlet 8f22ccd5ef Add explicit no-docker error and supervisor start task-func 2022-05-09 11:08:59 -04:00
Tyler Goodlet 685ec489ef Type annot updates 2022-05-09 11:08:59 -04:00
Tyler Goodlet 48777d72e8 Drop old client instantiate line 2022-05-09 11:08:59 -04:00
Tyler Goodlet ac524dc78b Drop import, it's got madness with and SIGINT? 2022-05-09 11:08:59 -04:00
Tyler Goodlet 1693cc42c1 Add a super simple `marketstore` container supervisor 2022-05-09 11:08:59 -04:00
Tyler Goodlet 91d63f5957 Extract non-sudo user for config dir path 2022-05-09 11:08:59 -04:00
Tyler Goodlet c291c26a3e Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-05-09 11:08:59 -04:00