Commit Graph

2117 Commits (d3ddcc82066ff406477f40269ad7f67d89eb7a49)

Author SHA1 Message Date
Tyler Goodlet d3ddcc8206 Get sync-to-marketstore-tsdb history retrieval workinnn 2022-04-30 11:38:24 -04:00
Tyler Goodlet 9b1491efc9 Handle "fatal" level log msgs in docker super 2022-04-30 11:38:24 -04:00
Tyler Goodlet 7175901d0d Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-04-30 11:38:24 -04:00
Tyler Goodlet bb13f76375 Drop `ms-shell`, add `piker storesh` cmd 2022-04-30 11:38:24 -04:00
Tyler Goodlet a682887e63 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-04-30 11:38:24 -04:00
Tyler Goodlet 1837e467be Drop `pandas` to `numpy` converter 2022-04-30 11:38:24 -04:00
Tyler Goodlet 7df795435e Disable re-connect for now in ib script 2022-04-30 11:38:24 -04:00
Tyler Goodlet 8421422768 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-04-30 11:38:24 -04:00
Tyler Goodlet 23d386e0f5 Try downsampling mkts data 2022-04-30 11:38:24 -04:00
Tyler Goodlet f0c4261aa4 Drop old type annot 2022-04-30 11:38:24 -04:00
Tyler Goodlet 2be8f63487 Comment each special key combo 2022-04-30 11:38:24 -04:00
Tyler Goodlet 24a07fd5e5 Load any symbol-matching shm array if no `marketstored` found 2022-04-30 11:38:24 -04:00
Tyler Goodlet a893537ade Get ib key hack script to work with reconnect 2022-04-30 11:38:24 -04:00
Tyler Goodlet 15d15fdfbf Move ib data reset script into a new `scripts/` dir 2022-04-30 11:38:24 -04:00
Tyler Goodlet d1b05246f8 Use new `tractor.query_actor()` for service checking 2022-04-30 11:38:24 -04:00
Tyler Goodlet 96ec4ba28b Return all timeframe arrays if `timeframe` not passed as input 2022-04-30 11:38:24 -04:00
Tyler Goodlet fa0be47d66 Make `pikerd` work again without `--tsdb` flag 2022-04-30 11:38:24 -04:00
Tyler Goodlet 65609a35dc Add a service checker predicate 2022-04-30 11:38:24 -04:00
Tyler Goodlet d8d7757e88 Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-04-30 11:38:24 -04:00
Tyler Goodlet c439e99f8a Proxy `marketstore` container log level to our own 2022-04-30 11:38:24 -04:00
Tyler Goodlet 5c2b9a01e9 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-04-30 11:38:24 -04:00
Tyler Goodlet 15d3f99410 Doc str formatting 2022-04-30 11:38:24 -04:00
Tyler Goodlet ae8170204f Add latency measures around diffs/writes to mkts 2022-04-30 11:38:24 -04:00
Tyler Goodlet 7d628c4059 Prototype out writing `1Sec` OHLCV data 2022-04-30 11:38:24 -04:00
Tyler Goodlet 6dfe59cce6 Better doc string 2022-04-30 11:38:24 -04:00
Tyler Goodlet a465a11782 Persist backing `/data/` filesystem across container runs 2022-04-30 11:38:24 -04:00
Tyler Goodlet 807685d27e Get basic OHLCV writes working with `anyio` client 2022-04-30 11:38:24 -04:00
Tyler Goodlet aea42ccbd9 Pass in daemon name to `start_ahab()` 2022-04-30 11:38:24 -04:00
Tyler Goodlet 8fb9308e21 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-04-30 11:38:24 -04:00
Tyler Goodlet bbae8ad426 Revive `ms-shell` sub-cmd 2022-04-30 11:38:24 -04:00
Tyler Goodlet 893ac7a986 Add WIP backfiller from data feed helper 2022-04-30 11:38:24 -04:00
Tyler Goodlet 91856ddda8 Better handle nested erros from docker client 2022-04-30 11:38:24 -04:00
Tyler Goodlet 25e2e13bd7 Add back in legacy write loop for reference 2022-04-30 11:38:24 -04:00
Tyler Goodlet 2427c96336 Add back in OHLCV dtype template and client side ws streamer 2022-04-30 11:38:24 -04:00
Tyler Goodlet 8732b2bd5e Drop ununsed `Services` ref 2022-04-30 11:38:24 -04:00
Tyler Goodlet ff9208c15b Py3.9+ type updates 2022-04-30 11:38:24 -04:00
Tyler Goodlet a2547a548f Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-04-30 11:38:24 -04:00
Tyler Goodlet 3873b8619e De-escalate sudo perms in `pikerd` once docker spawns 2022-04-30 11:38:24 -04:00
Tyler Goodlet 326d05ac82 Handle the non-root perms case specifically too 2022-04-30 11:38:24 -04:00
Tyler Goodlet dcab99e3d2 Add explicit no-docker error and supervisor start task-func 2022-04-30 11:38:24 -04:00
Tyler Goodlet 5a5df21f94 Type annot updates 2022-04-30 11:38:24 -04:00
Tyler Goodlet d7c1286e5d Drop old client instantiate line 2022-04-30 11:38:24 -04:00
Tyler Goodlet 8acaa28df0 Drop import, it's got madness with and SIGINT? 2022-04-30 11:38:24 -04:00
Tyler Goodlet bb45100168 Add a super simple `marketstore` container supervisor 2022-04-30 11:38:24 -04:00
Tyler Goodlet d9ded54e10 Extract non-sudo user for config dir path 2022-04-30 11:38:24 -04:00
Tyler Goodlet 1bde86a7b2 Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-04-30 11:38:24 -04:00
Guillermo Rodriguez 5031892dcf Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-04-30 11:38:24 -04:00
Guillermo Rodriguez afe6f0b42b Simplify and optimize tick format, similar to techtonicdb's 2022-04-30 11:38:24 -04:00
Guillermo Rodriguez 5d539b7c49 Add multi ingestor support and update to new feed API 2022-04-30 11:38:24 -04:00
Tyler Goodlet e2ce341f93 Only update y mxmn from L1 when last index in view 2022-04-30 11:38:24 -04:00