Commit Graph

2100 Commits (marketstore_backup)

Author SHA1 Message Date
Tyler Goodlet c3553de1f2 Handle non-fqsn for derivs and don't put brokername in 2022-04-13 08:11:57 -04:00
Tyler Goodlet 0790ae8235 Limit ohlc queries to 800k datums to avoid `purepc` size error 2022-04-13 08:11:57 -04:00
Tyler Goodlet a6b8c03e0e Get sync-to-marketstore-tsdb history retrieval workinnn 2022-04-13 08:11:57 -04:00
Tyler Goodlet ac22190b60 Handle "fatal" level log msgs in docker super 2022-04-13 08:11:57 -04:00
Tyler Goodlet 75d0bf3152 Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-04-13 08:11:57 -04:00
Tyler Goodlet 85e2602d2e Drop `ms-shell`, add `piker storesh` cmd 2022-04-13 08:11:57 -04:00
Tyler Goodlet edcac1e768 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-04-13 08:11:57 -04:00
Tyler Goodlet 73b8719984 Drop `pandas` to `numpy` converter 2022-04-13 08:11:57 -04:00
Tyler Goodlet 6b17370711 Disable re-connect for now in ib script 2022-04-13 08:11:57 -04:00
Tyler Goodlet cd3dbc9275 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-04-13 08:11:57 -04:00
Tyler Goodlet 87fab6c5b1 Try downsampling mkts data 2022-04-13 08:11:57 -04:00
Tyler Goodlet 2d719f88d2 Drop old type annot 2022-04-13 08:11:57 -04:00
Tyler Goodlet d3fb9f228d Comment each special key combo 2022-04-13 08:11:57 -04:00
Tyler Goodlet 58f6aa4308 Load any symbol-matching shm array if no `marketstored` found 2022-04-13 08:11:57 -04:00
Tyler Goodlet 1eb62e7678 Get ib key hack script to work with reconnect 2022-04-13 08:11:57 -04:00
Tyler Goodlet 965ea120e9 Move ib data reset script into a new `scripts/` dir 2022-04-13 08:11:57 -04:00
Tyler Goodlet fbdcbc9304 Use new `tractor.query_actor()` for service checking 2022-04-13 08:11:57 -04:00
Tyler Goodlet 7f87d09bf0 Return all timeframe arrays if `timeframe` not passed as input 2022-04-13 08:11:57 -04:00
Tyler Goodlet f7a0f117a0 Make `pikerd` work again without `--tsdb` flag 2022-04-13 08:11:57 -04:00
Tyler Goodlet ae986e9fcf Add a service checker predicate 2022-04-13 08:11:57 -04:00
Tyler Goodlet cad6508b28 Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-04-13 08:11:57 -04:00
Tyler Goodlet 46ad8557cd Proxy `marketstore` container log level to our own 2022-04-13 08:11:57 -04:00
Tyler Goodlet 85658a165c Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-04-13 08:11:57 -04:00
Tyler Goodlet 387cb51ca2 Doc str formatting 2022-04-13 08:11:57 -04:00
Tyler Goodlet 3c6f412a5b Add latency measures around diffs/writes to mkts 2022-04-13 08:11:57 -04:00
Tyler Goodlet 852f9e3e64 Prototype out writing `1Sec` OHLCV data 2022-04-13 08:11:57 -04:00
Tyler Goodlet 0af809849b Better doc string 2022-04-13 08:11:57 -04:00
Tyler Goodlet 5871aba04c Persist backing `/data/` filesystem across container runs 2022-04-13 08:11:57 -04:00
Tyler Goodlet 0904b48a23 Get basic OHLCV writes working with `anyio` client 2022-04-13 08:11:57 -04:00
Tyler Goodlet c2b5a3d225 Pass in daemon name to `start_ahab()` 2022-04-13 08:11:57 -04:00
Tyler Goodlet 4c5ea60035 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-04-13 08:11:57 -04:00
Tyler Goodlet abf7b14717 Revive `ms-shell` sub-cmd 2022-04-13 08:11:57 -04:00
Tyler Goodlet a6613161c3 Add WIP backfiller from data feed helper 2022-04-13 08:11:57 -04:00
Tyler Goodlet 6d38f3d0cb Better handle nested erros from docker client 2022-04-13 08:11:57 -04:00
Tyler Goodlet 0877d09bbf Add back in legacy write loop for reference 2022-04-13 08:11:57 -04:00
Tyler Goodlet 98ec2b71ae Add back in OHLCV dtype template and client side ws streamer 2022-04-13 08:11:57 -04:00
Tyler Goodlet 00cd77a0ad Drop ununsed `Services` ref 2022-04-13 08:11:57 -04:00
Tyler Goodlet cbf1dca0b1 Py3.9+ type updates 2022-04-13 08:11:57 -04:00
Tyler Goodlet 7cdf7f5d13 Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-04-13 08:11:57 -04:00
Tyler Goodlet 4130a1d2d6 De-escalate sudo perms in `pikerd` once docker spawns 2022-04-13 08:11:57 -04:00
Tyler Goodlet 17aa59b1f9 Handle the non-root perms case specifically too 2022-04-13 08:11:57 -04:00
Tyler Goodlet c27ce637c8 Add explicit no-docker error and supervisor start task-func 2022-04-13 08:11:56 -04:00
Tyler Goodlet 86b9c9a8f9 Type annot updates 2022-04-13 08:11:56 -04:00
Tyler Goodlet 54741ad9c6 Drop old client instantiate line 2022-04-13 08:11:56 -04:00
Tyler Goodlet 4670e66348 Drop import, it's got madness with and SIGINT? 2022-04-13 08:11:56 -04:00
Tyler Goodlet a3620a93ca Add a super simple `marketstore` container supervisor 2022-04-13 08:11:56 -04:00
Tyler Goodlet ddd92bb2ba Extract non-sudo user for config dir path 2022-04-13 08:11:56 -04:00
Tyler Goodlet 12a1074f0e Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-04-13 08:11:56 -04:00
Guillermo Rodriguez eb41ecb4db Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-04-13 08:11:56 -04:00
Guillermo Rodriguez bec76bfe59 Simplify and optimize tick format, similar to techtonicdb's 2022-04-13 08:11:56 -04:00