Commit Graph

36 Commits (9bc7de0cb3d572cd30cb430cd6c31c371d87eb80)

Author SHA1 Message Date
Tyler Goodlet 9bc7de0cb3 Add notes about how to do mkts "trimming"
Which is basically just "deleting" rows from a column series.
You can only use the trim command from the `.cmd` cli and only with a so
called `LocalClient` currently; it's also sketchy af and caused
a machine to hang due to mem usage..

Ideally we can patch in this functionality for use by the rpc api
and have it not hang like this XD

Pertains to https://github.com/alpacahq/marketstore/issues/264
2022-05-16 14:31:23 -04:00
Tyler Goodlet 6e2e2fc03f Use `pendulum` for timestamp parsing 2022-05-15 13:45:44 -04:00
Tyler Goodlet 769e803695 Write `mkts.yml` from template if one dne 2022-05-10 14:55:52 -04:00
Tyler Goodlet e196e9d1a0 Factor `marketstore` container specifics into `piker.data.marketstore` 2022-05-10 14:55:52 -04:00
Tyler Goodlet 8e08fb7b23 Add comment about un-reffed vars meant for use in shell 2022-05-09 11:15:14 -04:00
Tyler Goodlet 7e951f17ca Support large ohlcv writes via slicing, add struct array keymap 2022-05-09 11:15:14 -04:00
Tyler Goodlet d77cfa3587 Add back fqsn passthrough and feed opening 2022-05-09 11:15:14 -04:00
Tyler Goodlet b8b95f1081 Don't open a feed, write or read ohlc in for now 2022-05-09 11:15:13 -04:00
Tyler Goodlet 46c23e90db Add `Storage.load()` and `.write_ohlcv()` 2022-05-09 11:15:13 -04:00
Tyler Goodlet 15630f465d Limit ohlc queries to 800k datums to avoid `purepc` size error 2022-05-09 11:15:13 -04:00
Tyler Goodlet a971de2b67 Drop `ms-shell`, add `piker storesh` cmd 2022-05-09 11:15:13 -04:00
Tyler Goodlet ca48577c60 Add diffing logic to `tsdb_history_update()`
Add some basic `numpy` epoch slice logic to generate append and prepend
arrays to write to the db.

Mooar cool things,
- add a `Storage.delete_ts()` method to wipe a column series from the db
  easily.
- don't attempt to read in any OHLC series by default on client load
- add some `pyqtgraph` profiling and drop manual latency measures
- if no db series for the fqsn exists write the entire shm array
2022-05-09 11:15:13 -04:00
Tyler Goodlet 6dc6d00a9b Try downsampling mkts data 2022-05-09 11:15:13 -04:00
Tyler Goodlet 9138f376f7 Return all timeframe arrays if `timeframe` not passed as input 2022-05-09 11:15:13 -04:00
Tyler Goodlet 706c8085f2 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.

Further:

- get `pikerd` working with and without `--tsdb` flag.
- support spawning `brokerd` with no real-time quotes.
- bring back in "fqsn" support that was originally not
  in this history before commits factoring.
2022-05-09 11:15:13 -04:00
Tyler Goodlet 3dba456cf8 Add latency measures around diffs/writes to mkts 2022-05-09 11:15:13 -04:00
Tyler Goodlet 4555a1f279 Prototype out writing `1Sec` OHLCV data 2022-05-09 11:15:13 -04:00
Tyler Goodlet e1bbcff8e0 Get basic OHLCV writes working with `anyio` client 2022-05-09 11:15:13 -04:00
Tyler Goodlet 56fa759452 Add WIP backfiller from data feed helper 2022-05-09 11:15:13 -04:00
Tyler Goodlet 445b82283d Add back in legacy write loop for reference 2022-05-09 11:15:13 -04:00
Tyler Goodlet 8047714101 Add back in OHLCV dtype template and client side ws streamer 2022-05-09 11:15:13 -04:00
Tyler Goodlet ed5bae0e11 Py3.9+ type updates 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 943b02573d Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 897a5cf2f6 Simplify and optimize tick format, similar to techtonicdb's 2022-05-09 11:15:13 -04:00
Guillermo Rodriguez 3c09bfba57 Add multi ingestor support and update to new feed API 2022-05-09 11:15:13 -04:00
Tyler Goodlet e6ea053d40 Get kivy/questrade shit working again 2021-02-21 12:32:40 -05:00
Tyler Goodlet c1109ee3fb Add license headers to pertinent files 2020-11-06 12:23:14 -05:00
Tyler Goodlet 1a1e768126 Port to new data apis 2020-10-02 12:13:28 -04:00
Tyler Goodlet 05d2985f5f Clarify some odd spots 2020-09-02 11:32:54 -04:00
Tyler Goodlet 316137fdf2 Begin to wrap marketstore as a data feed
Wrap the sync client in an async interface in anticipation of an actual
async client. This starts support for the `open_fee()`/`stream_quotes()`
api though the tick normalization isn't correct yet.
2020-09-02 00:40:35 -04:00
Tyler Goodlet 3c4699abef Pass broker name 2020-09-01 18:31:52 -04:00
Tyler Goodlet 78784a4bf3 Port to new data apis 2020-09-01 18:30:55 -04:00
Tyler Goodlet f6f6d98a95 Allow passing in tbk keys to query 2020-09-01 18:14:11 -04:00
Tyler Goodlet bc9af977a4 Update quote cache on each loop 2020-09-01 13:25:41 -04:00
Tyler Goodlet 436e4d2df4 Add tbk tick streaming with trio-websocket 2020-09-01 13:25:41 -04:00
Tyler Goodlet a6c692fb8b Add support for TICK ingest to marketstore 2020-09-01 13:25:09 -04:00