Commit Graph

3544 Commits (788e158d9f762b5dbd94c5feceecbb1e42ad2b4d)

Author SHA1 Message Date
Tyler Goodlet 49509d55d2 Implement `open_history_client()` correctly for `kraken` 2022-05-09 11:15:14 -04:00
Tyler Goodlet 6ba3c15c4e Add to signal broker won't deliver more data 2022-05-09 11:15:14 -04:00
Tyler Goodlet a3db5d1bdc Relay frame size in `NoData` due to null-result history 2022-05-09 11:15:14 -04:00
Tyler Goodlet c672493998 Add , indicates hist size to decrement to storage logic 2022-05-09 11:15:14 -04:00
Tyler Goodlet 423af37389 Truncate trade rate wma window sizes 2022-05-09 11:15:14 -04:00
Tyler Goodlet 0061fabb56 More tolerance for "stream-ended-early" conditions in quote throttler 2022-05-09 11:15:14 -04:00
Tyler Goodlet 2f04a8c939 Drop legacy back-filling logic
Use the new `open_history_client()` endpoint/API and expect backends to
provide a history "getter" routine that can be called to load historical
data into shm even when **not** using a tsdb. Add logic for filling in
data from the tsdb once the backend has provided data up to the last
recorded in the db. Add logic for avoiding overruns of the shm buffer
with more-then-necessary queries of tsdb data.
2022-05-09 11:15:14 -04:00
Tyler Goodlet 8bf40ae299 Drop legacy backfilling, load a day's worth of data by default 2022-05-09 11:15:14 -04:00
Tyler Goodlet 0f683205f4 Add 16 fetch limit if no tsdb data found 2022-05-09 11:15:14 -04:00
Tyler Goodlet d244af69c9 Don't require a symbol to subcmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet b8b95f1081 Don't open a feed, write or read ohlc in for now 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3056bc3143 Don't run legacy backfill when isn't up 2022-05-09 11:15:13 -04:00
Tyler Goodlet d3824c8c0b Start legacy backfill with partial too 2022-05-09 11:15:13 -04:00
Tyler Goodlet 727d3cc027 Unify backfilling logic into common task-routine 2022-05-09 11:15:13 -04:00
Tyler Goodlet 46c23e90db Add `Storage.load()` and `.write_ohlcv()` 2022-05-09 11:15:13 -04:00
Tyler Goodlet bcf3be1fe4 A bit hacky but, broadcast index streams on each history prepend 2022-05-09 11:15:13 -04:00
Tyler Goodlet 7d8cf3eaf8 Factor subscription broadcasting into a func 2022-05-09 11:15:13 -04:00
Tyler Goodlet d4e0d4463f Always update ohlc (main source chart) on `trigger_all=True` 2022-05-09 11:15:13 -04:00
Tyler Goodlet ab8629aa11 Make ib history client expect datetimes for input 2022-05-09 11:15:13 -04:00
Tyler Goodlet 2a07005c97 Add binance history client support with datetime use throughout 2022-05-09 11:15:13 -04:00
Tyler Goodlet 79160619bc Drop old type annot 2022-05-09 11:15:13 -04:00
Tyler Goodlet e1a88cb93c Only update y mxmn from L1 when last index in view 2022-05-09 11:15:13 -04:00
Tyler Goodlet a6c5902437 More reliable `marketstored` + container supervision
It turns out (i guess not so shockingly?) that `marketstore` doesn't
always teardown "gracefully" under SIGINT (seems to hang if there are
open client connections which are also in the midst of teardown?) so
this instead first tries the SIGINT and then fails over to a SIGKILL
(destroy loop) which seems to be much more reliable to ensure shutdown
without any downside - in terms of a "hard kill".

Originally i was thinking the issue was root perms related (which get
relegated solely to the `marketstored` daemon actor after spawn) but
actually it was indeed the signalling / application layer causing the
hold-up/latency on teardown. There's a bunch of lingering (now
commented) code which tried to solve this non-problem as well as a bunch
logging/prints to help decipher the root of the issue - this will all
get cleaned out shortly.
2022-05-09 11:15:13 -04:00
Tyler Goodlet a10dc4fe77 Add `docker` as `tsdb` extras dep 2022-05-09 11:15:13 -04:00
Tyler Goodlet 71416f5752 Add `anyio-marketstore` client as dev dep 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9fe5cd647a Handle non-fqsn for derivs and don't put brokername in 2022-05-09 11:15:13 -04:00
Tyler Goodlet 15630f465d Limit ohlc queries to 800k datums to avoid `purepc` size error 2022-05-09 11:15:13 -04:00
Tyler Goodlet ce3229df7d Get sync-to-marketstore-tsdb history retrieval workinnn 2022-05-09 11:15:13 -04:00
Tyler Goodlet 53ad5e6f65 Handle "fatal" level log msgs in docker super 2022-05-09 11:15:13 -04:00
Tyler Goodlet 41325ad418 Add basic tsdb history loading
If `marketstore` is detected try to only load most recent missing data
from the data provider (broker) and the rest from the tsdb and push it
all to shm for display in the UI. If the provider/broker doesn't have
the history client endpoint, just use the old one for now so we can
start to incrementally add support. Don't start the ohlc step
incrementer task until the backend signals that the feed is live.
2022-05-09 11:15:13 -04:00
Tyler Goodlet a971de2b67 Drop `ms-shell`, add `piker storesh` cmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet ca48577c60 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-05-09 11:15:13 -04:00
Tyler Goodlet 950cb03e07 Drop `pandas` to `numpy` converter 2022-05-09 11:15:13 -04:00
Tyler Goodlet 907b7dd5c6 Disable re-connect for now in ib script 2022-05-09 11:15:13 -04:00
Tyler Goodlet 6cdd017cd6 Ensure bfqsn is lower cased for feed api consumers
Also, Start tinkering with `tractor.trionics.ipython_embed()`

In effort to get back to a usable REPL around the mkts client
this adds usage of the new `tractor` integration api as well as logic
for skipping backfilling if existing tsdb arrays are found.
2022-05-09 11:15:13 -04:00
Tyler Goodlet 6dc6d00a9b Try downsampling mkts data 2022-05-09 11:15:13 -04:00
Tyler Goodlet ba250c7197 Comment each special key combo 2022-05-09 11:15:13 -04:00
Tyler Goodlet 565573b609 Load any symbol-matching shm array if no `marketstored` found 2022-05-09 11:15:13 -04:00
Tyler Goodlet 39b4d2684a Get ib key hack script to work with reconnect 2022-05-09 11:15:13 -04:00
Tyler Goodlet 25dfe4115d Move ib data reset script into a new `scripts/` dir 2022-05-09 11:15:13 -04:00
Tyler Goodlet 6c6f2abd06 Use new `tractor.query_actor()` for service checking 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9138f376f7 Return all timeframe arrays if `timeframe` not passed as input 2022-05-09 11:15:13 -04:00
Tyler Goodlet f582af4c9f Make `pikerd` work again without `--tsdb` flag 2022-05-09 11:15:13 -04:00
Tyler Goodlet dd2edaeb3c Add a service checker predicate 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3d6d77364b Allow kill-child-proc-with-root-perms to fail silently in `tractor` reaping 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8003878248 Proxy `marketstore` container log level to our own 2022-05-09 11:15:13 -04:00
Tyler Goodlet 706c8085f2 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-05-09 11:15:13 -04:00
Tyler Goodlet cbe74d126e Doc str formatting 2022-05-09 11:15:13 -04:00
Tyler Goodlet 3dba456cf8 Add latency measures around diffs/writes to mkts 2022-05-09 11:15:13 -04:00
Tyler Goodlet 4555a1f279 Prototype out writing `1Sec` OHLCV data 2022-05-09 11:15:13 -04:00