Commit Graph

2108 Commits (44b613024e43774a5318d38751c313c46ce36555)

Author SHA1 Message Date
Tyler Goodlet 44b613024e Get sync-to-marketstore-tsdb history retrieval workinnn 2022-04-16 15:39:24 -04:00
Tyler Goodlet 36b8253b66 Handle "fatal" level log msgs in docker super 2022-04-16 15:39:24 -04:00
Tyler Goodlet a16cd2fe6d Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-04-16 15:39:24 -04:00
Tyler Goodlet 245085d493 Drop `ms-shell`, add `piker storesh` cmd 2022-04-16 15:39:24 -04:00
Tyler Goodlet 0b68bf257e Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-04-16 15:39:24 -04:00
Tyler Goodlet 68ac759b3f Drop `pandas` to `numpy` converter 2022-04-16 15:39:24 -04:00
Tyler Goodlet 2679d6261e Disable re-connect for now in ib script 2022-04-16 15:39:24 -04:00
Tyler Goodlet 48c989d400 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-04-16 15:39:24 -04:00
Tyler Goodlet 831c23872c Try downsampling mkts data 2022-04-16 15:39:24 -04:00
Tyler Goodlet 008e153f36 Drop old type annot 2022-04-16 15:39:24 -04:00
Tyler Goodlet 38be93a009 Comment each special key combo 2022-04-16 15:39:24 -04:00
Tyler Goodlet 3bf3f450f4 Load any symbol-matching shm array if no `marketstored` found 2022-04-16 15:39:24 -04:00
Tyler Goodlet 4e1ab378f4 Get ib key hack script to work with reconnect 2022-04-16 15:39:24 -04:00
Tyler Goodlet 8d77d177bf Move ib data reset script into a new `scripts/` dir 2022-04-16 15:39:24 -04:00
Tyler Goodlet 36463dddbe Use new `tractor.query_actor()` for service checking 2022-04-16 15:39:24 -04:00
Tyler Goodlet e399b74b67 Return all timeframe arrays if `timeframe` not passed as input 2022-04-16 15:39:24 -04:00
Tyler Goodlet 15b609bf13 Make `pikerd` work again without `--tsdb` flag 2022-04-16 15:39:24 -04:00
Tyler Goodlet dfc3fb76fd Add a service checker predicate 2022-04-16 15:39:24 -04:00
Tyler Goodlet 881b1afc12 Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-04-16 15:39:24 -04:00
Tyler Goodlet 9878735ff3 Proxy `marketstore` container log level to our own 2022-04-16 15:39:24 -04:00
Tyler Goodlet 1389ee51f3 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-04-16 15:39:24 -04:00
Tyler Goodlet 6225e8a58e Doc str formatting 2022-04-16 15:39:24 -04:00
Tyler Goodlet 2b48943066 Add latency measures around diffs/writes to mkts 2022-04-16 15:39:24 -04:00
Tyler Goodlet 92d074f308 Prototype out writing `1Sec` OHLCV data 2022-04-16 15:39:24 -04:00
Tyler Goodlet 49a280ce14 Better doc string 2022-04-16 15:39:24 -04:00
Tyler Goodlet 42e5f48345 Persist backing `/data/` filesystem across container runs 2022-04-16 15:39:24 -04:00
Tyler Goodlet 18344603e6 Get basic OHLCV writes working with `anyio` client 2022-04-16 15:39:24 -04:00
Tyler Goodlet ae1d9efa97 Pass in daemon name to `start_ahab()` 2022-04-16 15:39:24 -04:00
Tyler Goodlet f1747749e7 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-04-16 15:39:24 -04:00
Tyler Goodlet cdd03759bf Revive `ms-shell` sub-cmd 2022-04-16 15:39:24 -04:00
Tyler Goodlet 00a36826c2 Add WIP backfiller from data feed helper 2022-04-16 15:39:24 -04:00
Tyler Goodlet 3aa8044aa1 Better handle nested erros from docker client 2022-04-16 15:39:24 -04:00
Tyler Goodlet f0ae7edb00 Add back in legacy write loop for reference 2022-04-16 15:39:24 -04:00
Tyler Goodlet 797ba15923 Add back in OHLCV dtype template and client side ws streamer 2022-04-16 15:39:24 -04:00
Tyler Goodlet b8f86d4599 Drop ununsed `Services` ref 2022-04-16 15:39:24 -04:00
Tyler Goodlet 908093264a Py3.9+ type updates 2022-04-16 15:39:24 -04:00
Tyler Goodlet 3ef68f78f7 Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-04-16 15:39:24 -04:00
Tyler Goodlet d095c21885 De-escalate sudo perms in `pikerd` once docker spawns 2022-04-16 15:39:24 -04:00
Tyler Goodlet 52dc0989ac Handle the non-root perms case specifically too 2022-04-16 15:39:24 -04:00
Tyler Goodlet 6f749c340d Add explicit no-docker error and supervisor start task-func 2022-04-16 15:39:24 -04:00
Tyler Goodlet dcff80131f Type annot updates 2022-04-16 15:39:24 -04:00
Tyler Goodlet 6cd829d1fd Drop old client instantiate line 2022-04-16 15:39:24 -04:00
Tyler Goodlet 76fff18b1a Drop import, it's got madness with and SIGINT? 2022-04-16 15:39:24 -04:00
Tyler Goodlet 88e1befffe Add a super simple `marketstore` container supervisor 2022-04-16 15:39:24 -04:00
Tyler Goodlet c1579b630e Extract non-sudo user for config dir path 2022-04-16 15:39:24 -04:00
Tyler Goodlet 7dfdd0f00b Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez 14a893a5e5 Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez c91599472d Simplify and optimize tick format, similar to techtonicdb's 2022-04-16 15:39:24 -04:00
Guillermo Rodriguez 9e4dec98da Add multi ingestor support and update to new feed API 2022-04-16 15:39:24 -04:00
Tyler Goodlet 8800ee0661 Only update y mxmn from L1 when last index in view 2022-04-16 15:39:24 -04:00