Commit Graph

1725 Commits (9fcb1d350161f46b1171f78e35856ff2f0cb2a9d)

Author SHA1 Message Date
Tyler Goodlet 8af76322c9 Prototype a high level `Storage` api
Starts a wrapper around the `marketstore` client to do basic ohlcv query
and retrieval and prototypes out write methods for ohlc and tick.
Try to connect to `marketstore` automatically (which will fail if not
started currently) but we will eventually first do a service query.
2022-03-11 19:50:58 -05:00
Tyler Goodlet eb5a4f7eeb Move factor helper to a classmethod 2022-03-11 19:50:58 -05:00
Tyler Goodlet e008f69505 Doc str formatting 2022-03-11 19:50:58 -05:00
Tyler Goodlet 6c8b79906b Make linux timeout the same 2022-03-11 19:50:58 -05:00
Tyler Goodlet 40e62c1a38 Add latency measures around diffs/writes to mkts 2022-03-11 19:50:58 -05:00
Tyler Goodlet bed47d3ae6 Add flag to avoid logging json to console 2022-03-11 19:50:58 -05:00
Tyler Goodlet f60d9dd79c Prototype out writing `1Sec` OHLCV data 2022-03-11 19:50:58 -05:00
Tyler Goodlet 4402b2dc73 Better doc string 2022-03-11 19:50:58 -05:00
Tyler Goodlet 6e37ab6bf9 Use `asyncio` in `Client.get_quote()` 2022-03-11 19:50:58 -05:00
Tyler Goodlet 88411a6a26 Persist backing `/data/` filesystem across container runs 2022-03-11 19:50:58 -05:00
Tyler Goodlet a0c3d5f32f Get basic OHLCV writes working with `anyio` client 2022-03-11 19:50:58 -05:00
Tyler Goodlet 236df4b6d6 Pass in daemon name to `start_ahab()` 2022-03-11 19:50:58 -05:00
Tyler Goodlet a3ec0c16c6 Map the grpc port and add graceful container teardown
Not sure how I missed mapping the 5995 grpc port 🤦; done now.
Also adds graceful teardown using SIGINT with included container
logging relayed to the piker console B).
2022-03-11 19:50:58 -05:00
Tyler Goodlet 51ced95962 Revive `ms-shell` sub-cmd 2022-03-11 19:50:58 -05:00
Tyler Goodlet 3487f76147 Add WIP backfiller from data feed helper 2022-03-11 19:50:58 -05:00
Tyler Goodlet fa69fca311 Better handle nested erros from docker client 2022-03-11 19:50:58 -05:00
Tyler Goodlet 57b3d2f7e4 Add back in legacy write loop for reference 2022-03-11 19:50:58 -05:00
Tyler Goodlet f9b799b53d Add back in OHLCV dtype template and client side ws streamer 2022-03-11 19:50:58 -05:00
Tyler Goodlet 35f7c3409a Drop ununsed `Services` ref 2022-03-11 19:50:58 -05:00
Tyler Goodlet 9c5f7a6bb9 Py3.9+ type updates 2022-03-11 19:50:58 -05:00
Tyler Goodlet 86337430d8 Add `--tsdb` flag to start `marketstore` with `pikerd` 2022-03-11 19:50:58 -05:00
Tyler Goodlet 8d09d63095 De-escalate sudo perms in `pikerd` once docker spawns 2022-03-11 19:50:58 -05:00
Tyler Goodlet df04ccb845 Handle the non-root perms case specifically too 2022-03-11 19:50:58 -05:00
Tyler Goodlet ad0ace2528 Add explicit no-docker error and supervisor start task-func 2022-03-11 19:50:58 -05:00
Tyler Goodlet edd273d5d8 Type annot updates 2022-03-11 19:50:58 -05:00
Tyler Goodlet cfc77a0a66 Drop old client instantiate line 2022-03-11 19:50:58 -05:00
Tyler Goodlet 69b3120444 Drop import, it's got madness with and SIGINT? 2022-03-11 19:50:58 -05:00
Tyler Goodlet 8662cde7ca Add a super simple `marketstore` container supervisor 2022-03-11 19:50:58 -05:00
Tyler Goodlet 73b3f7ead8 Extract non-sudo user for config dir path 2022-03-11 19:50:58 -05:00
Tyler Goodlet c3509e7f93 Basic module-script for spawning `marketstore`, needs correct bind mount usage 2022-03-11 19:50:58 -05:00
Guillermo Rodriguez d215a69049 Still WIP, switch to using new marketstore client, missing streaming from marketstore 2022-03-11 19:50:58 -05:00
Guillermo Rodriguez a11cee82d0 Simplify and optimize tick format, similar to techtonicdb's 2022-03-11 19:50:58 -05:00
Guillermo Rodriguez aba50515df Add multi ingestor support and update to new feed API 2022-03-11 19:50:58 -05:00
Tyler Goodlet 1b1bf07f54 Array diff lengths must be int 2022-03-11 16:49:58 -05:00
Tyler Goodlet 77a7b73260 Open feeds using `Portal.open_context()` 2022-03-11 16:49:37 -05:00
Tyler Goodlet 4ad06e4cc0 Limit real-time chart updates in "big data" cases
- the chart's uppx (units-per-pixel) is > 4 (i.e. zoomed out a lot)
- don't shift the chart (to keep the most recent step in view) if the
  last datum isn't in view (aka the user is probably looking at history)
2022-03-11 16:31:28 -05:00
Tyler Goodlet 3da081c67a Only trigger downsampling on manual changes, add a uppx method 2022-03-11 16:31:20 -05:00
Tyler Goodlet d56d1fc4c1 Add for a `BarItems` to display a line on high uppx
When a bars graphic is zoomed out enough you get a high uppx, datum
units-per-pixel, and there is no point in drawing the 6-lines in each
bar element-graphic if you can't see them on the screen/display device.

Instead here we offer converting to a `FastAppendCurve` which traces
the high-low outline and instead display that when it's impossible to see the
details of bars - approximately when the uppx >= 2.

There is also some draft-commented code in here for downsampling the
outlines as zoom level increases but it's not fully working and should
likely be factored out into a higher level api anyway.
2022-03-11 16:31:14 -05:00
Tyler Goodlet 544578c67d Original index offset was right 2022-03-11 16:31:07 -05:00
Tyler Goodlet 01ea2b3110 Add an ohlcv high/low tracer with optional downsampling 2022-03-11 16:30:52 -05:00
Tyler Goodlet 2f02f71610 Pass linked charts into `BarItems` so that graphics can be cycled on downsample 2022-03-11 16:24:43 -05:00
Tyler Goodlet b318ebc221 Move graphics compression routines to new module 2022-03-11 16:24:30 -05:00
Tyler Goodlet d737adb1b8 Use 12Hz as default fps throttle 2022-03-11 16:24:22 -05:00
Tyler Goodlet 75d7314493 Add comment on how to enable `pyqtgraph` profiling 2022-03-11 16:24:15 -05:00
Tyler Goodlet d7d824030d Add basic optional polyline support, draft out downsampling routine 2022-03-11 16:24:02 -05:00
Tyler Goodlet 28436bcb2b Add a downsampled line-curve support to `BarItems`
In effort to start getting some graphics speedups as detailed in #109,
this adds a `FastAppendCurve`to every `BarItems` as a `._ds_line` which
is only displayed (instead of the normal mult-line bars curve) when the
"width" of a bar is indistinguishable on screen from a line -> so once
the view coordinates map to > 2 pixels on the display device.
`BarItems.maybe_paint_line()` takes care of this scaling detection logic and is
called by the associated view's `.sigXRangeChanged` signal handler.
2022-03-11 16:23:55 -05:00
Tyler Goodlet 692e310a98 Use a `DisplayState` in the graphics update loop
The graphics update loop is much easier to grok when all the UI
components which potentially need to be updated on a cycle are arranged
together in a high-level composite namespace, thus this new
`DisplayState` addition. Create and set this state on each
`LinkedSplits` chart set and add a new method `.graphics_cycle()` which
let's a caller trigger a graphics loop update manually. Use this method
in the fsp graphics manager such that a chain can update new history
output even if there is no real-time feed driving the display loop (eg.
when a market is "closed").
2022-03-11 16:23:28 -05:00
Tyler Goodlet c60d523428 Support an array field map to `ShmArray.push()`, start index 3days in 2022-03-11 16:23:19 -05:00
Tyler Goodlet 00d7bb089f Convert `iter_ohlc_periods()` to a `@tractor.context` 2022-03-11 16:23:00 -05:00
Tyler Goodlet 3dc87e0426 Facepalm**2: only update on special "update" msg 2022-03-11 16:22:49 -05:00
Tyler Goodlet 49531a2da6 Facepalm: display state must be linked charts specific 2022-03-11 16:22:39 -05:00
Tyler Goodlet 53641abc4b Add detailed `.addItem()`` comment 2022-03-11 16:22:24 -05:00
Tyler Goodlet b0e236fadf Manually trigger graphics loops updates on msgs from the fsp chain 2022-03-11 16:21:06 -05:00
Tyler Goodlet ef0516a84b Always fire a "step/update message" on every fsp history update 2022-03-11 16:20:55 -05:00
Tyler Goodlet bcd0895a12 Factor sync part of graphics update into func, add `trigger_update()`` 2022-03-11 16:20:42 -05:00
Tyler Goodlet 81c69c54ec Add guard for real-time-not-active last line is `None` case 2022-03-11 16:13:13 -05:00
Tyler Goodlet 01f5f2d015 Don't require a rt quote, increase client connect timeout 2022-03-03 17:49:21 -05:00
Tyler Goodlet af3d624281 Just give up on discretized pp bar for now 2022-03-03 17:15:55 -05:00
Tyler Goodlet 2c9612ebd8 Force exact pp bar size 2022-03-03 10:46:30 -05:00
Tyler Goodlet 16b9e39e11 Dis-allow an allocator limit less then the current pp size 2022-03-02 10:05:33 -05:00
Tyler Goodlet 6889a25926 Drop pp bar clipping, hopefully fix slot sizing 2022-03-02 10:05:33 -05:00
Tyler Goodlet f7d03489d8 Drop `marketstore` loading cruft (will come later) 2022-03-01 12:39:12 -05:00
Tyler Goodlet 09079b61fc Comment task canceller method prototype 2022-03-01 12:37:31 -05:00
Tyler Goodlet adccb687fe Fix `piker services` cmd 2022-03-01 12:36:32 -05:00
Tyler Goodlet c239faf4e5 Add a `._sampling.sampler` registry composite type 2022-03-01 12:36:32 -05:00
Tyler Goodlet 6f3d78b729 Handle "no data" case in ranger calcs and avoid crashes 2022-02-28 08:30:44 -05:00
Tyler Goodlet 3e7d4f8717 Detect and request sample period in fsp engine 2022-02-28 08:30:32 -05:00
Tyler Goodlet b1cce8f9cf Adjust and add notes for python-trio/trio#2258 2022-02-28 08:30:22 -05:00
Tyler Goodlet 89a98c4aa2 Fix portal result `await`, comment some unused code 2022-02-28 08:30:15 -05:00
Tyler Goodlet 7a943f0e1e Always transmit index event even when no shm is registered 2022-02-28 08:29:56 -05:00
Tyler Goodlet 786ffde4e6 Use 3.9+ annots 2022-02-28 08:27:59 -05:00
Tyler Goodlet 11d4ebd0b5 Just warn on double-remove of a sub 2022-02-28 08:27:37 -05:00
Tyler Goodlet 81f8b4e145 Don't zero clearing rates on sample steps 2022-02-28 08:26:48 -05:00
Tyler Goodlet cc55e1f4bb Drop task-driven sample step graphics updates
Since moving to a "god loop" for graphics, we don't really need to have
a dedicated task for updating graphics on new sample increments. The
only UX difference will be that curves won't be updated until an actual new
rt-quote-event triggers the graphics loop -> so we'll have the chart
"jump" to a new position and new curve segments generated only when new
data arrives. This is imo fine since it's just less "idle" updates
where the chart would sit printing the same (last) value every step.
Instead only update the view increment if a new index is detected by
reading shm.

If we ever want this dedicated task update again this commit can be
easily reverted B)
2022-02-28 08:26:26 -05:00
Tyler Goodlet 412c9ee6cf Support view increment with a steps size 2022-02-28 08:26:20 -05:00
Tyler Goodlet bf3b58e861 Async load data history, allow "offline" feed use
Break up real-time quote feed and history loading into 2 separate tasks
and deliver a client side `data.Feed` as soon as history is loaded
(instead of waiting for a rt quote - the previous logic). If
a symbol doesn't have history then likely the feed shouldn't be loaded
(since presumably client code will need at least "some" datums history
to do anything) and waiting on a real-time quote is dumb, since it'll
hang if the market isn't open XD. If a symbol doesn't have history we
can always write a zero/null array when we run into that case. This also
greatly speeds up feed loading when both history and quotes are available.

TL;DR summary:
- add a `_Feedsbus.start_task()` one-cancel-scope-per-task method for
  assisting with (re-)starting and stopping long running persistent
  feeds (basically a "one cancels one" style nursery API).
- add a `manage_history()` task which does all history loading (and
  eventually real-time writing) which has an independent signal and
  start it in a separate task.
- drop the "sample rate per symbol" stuff since client code doesn't really
  care when it can just inspect shm indexing/time-steps itself.
- run throttle tasks in the bus nursery thus avoiding cancelling the
  underlying sampler task on feed client disconnects.
- don't store a repeated ref the bus nursery's cancel scope..
2022-02-28 08:26:13 -05:00
Tyler Goodlet 1d3ed6c333 Add `mk_` prefix since assignments will use `fqsn` 2022-02-28 08:23:57 -05:00
Tyler Goodlet 832e4c97d2 Drop shm: ShmArray` to `stream_quotes()` endpoint 2022-02-28 08:23:16 -05:00
Tyler Goodlet 23aa7eb31c Stick time step in window header 2022-02-28 08:22:47 -05:00
Tyler Goodlet c2a13c474c Support no realtime stream sending with feed bus 2022-02-28 08:22:40 -05:00
Tyler Goodlet 7252094f90 Add `open_piker_runtime()` to setup actor runtime correctly from non-daemons 2022-02-28 08:16:30 -05:00
Tyler Goodlet b1dd24d1f7 Only throttle warn on rate >= display rate 2022-02-28 08:15:39 -05:00
Tyler Goodlet a073039b30 Drop dependence on `msgpack` and `msgpack_numpy` 2022-02-28 08:15:18 -05:00
Tyler Goodlet 5c343aa748 Misc curve doc strings 2022-02-28 08:14:11 -05:00
Tyler Goodlet 7b13124dd4 Keep clear loop price pedantically up to date
To avoid the "trigger finger" issue (darks execing before they should
due to a stale last price state, normally when generating a trigger
predicate..) always iterate the loop and update the last known book
price even when no execs/triggered orders are registered.
2022-02-11 10:30:30 -05:00
Tyler Goodlet ca1c1cf415 Annoying doc strings 2022-02-11 10:30:30 -05:00
Tyler Goodlet 92c63988bc Bleh, just fill the available window space 2022-02-11 10:07:43 -05:00
Tyler Goodlet 9ed153bcb6 Less gap below results view 2022-02-11 08:45:57 -05:00
Tyler Goodlet 412c34eba0 Drop width check logic; only do height 2022-02-11 08:32:28 -05:00
Tyler Goodlet 68e1db27f8 Drop old null window size 2022-02-10 14:35:28 -05:00
Tyler Goodlet 86b1316691 Handle no-rows-yet case 2022-02-10 14:35:11 -05:00
Tyler Goodlet 890ffc76cf Dynamically re-size the search results view 2022-02-10 14:22:46 -05:00
Tyler Goodlet 51d94a301a Support resize event relaying from the god widget 2022-02-10 14:21:17 -05:00
Tyler Goodlet c54c9ae3d3 Add doc string to DE sizing method 2022-02-10 14:20:15 -05:00
Tyler Goodlet 5a4c155798 Add detailed comment around DE scaling 2022-02-10 13:04:13 -05:00
wattygetlood a5ad24f770 Additionally apply DPI scaling to font size if detected 2022-02-10 10:26:52 -05:00
Tyler Goodlet a0034e2948 If the DE (like windohz) already scales DPI, just use that scale for font size 2022-02-10 10:26:52 -05:00
wattygetlood fc3c0741b8 Set isn't serializable on std msgpack 2022-02-10 10:26:52 -05:00
wattygetlood cc87508fd9 Only load 4 ib requests worth of bars on windows... 2022-02-10 10:26:52 -05:00
wattygetlood d069481f1d Hack search view on windows to 1/2 window height; needs a better solution 2022-02-10 10:26:52 -05:00