Commit Graph

3544 Commits (788e158d9f762b5dbd94c5feceecbb1e42ad2b4d)

Author SHA1 Message Date
Tyler Goodlet 72ec9b1e10 Add `Pair.tick_size` to `kraken` schema 2022-11-10 13:16:21 -05:00
Tyler Goodlet 40c70ae6d8 Drop unecessary services var asserts? 2022-11-10 13:06:31 -05:00
Tyler Goodlet d3fefdeaff Expose registry sockaddr in `open_piker_runtime()` 2022-11-10 13:06:31 -05:00
Tyler Goodlet 8be005212f Expose `.open_feed()` and `open_piker_runtime()` eps at top level 2022-11-10 13:06:31 -05:00
Tyler Goodlet 5a2795e76b Passthrough registry sockaddr from chart cmd to daemon 2022-11-10 13:06:31 -05:00
Tyler Goodlet a987f0ab81 Add registry socket cli flags to all client cmds
Allows starting UI apps and passing the `pikerd` registry socket-addr
args via `--host` or `--port` such that a separate actor tree can be
started by selecting an unused port. This is handy when hacking new
features but while also wishing to run a more stable version of the code
for trading on the same host.
2022-11-10 13:06:31 -05:00
Tyler Goodlet d99b40317d Add a `pikerd -p <port_number>` flag 2022-11-10 13:06:31 -05:00
Tyler Goodlet 9ae519f6fa Re-work chart-overlay event broadcasting
Drop all attempts at rewiring `ViewBox` signals, monkey-patching
relayee handlers, and generally modifying event source public
attributes. Instead take a much simpler approach where the event source
graphics object simply has it's handler dynamically overridden by
a broadcaster function which relays to all consumers using a Python
loop.

The benefits of this much simplified approach include:
- avoiding the tedious and often complex (re)connection of signals between
  the source plot and the overlayed consumers.
- requiring zero modification of the public interface of any of the
  publisher or consumer `ViewBox`s, no decoration, extra signal
  definitions (eg. previous `mouseDragEventRelay` or the like).
- only a single dynamic method override on the event source graphics object
  (`ViewBox`) which does the broadcasting work and requires no
  modification to handler implementations.

Detailed `.ui._overlay` changes:
- drop `mk_relay_signal()`, `enable_relays()` which removes signal/slot
  hacking methodology.
- drop unused `ComposedGridLayout.grid` and `.reverse`, change some
  method names: `.insert()` -> `.insert_plotitem()`, `append()` ->
  `.append_plotitem()`.
- in `PlotOverlay`, again drop all signal/slot rewiring in
  `.add_plotitem()` and instead add our new closure based python-loop in
  `broadcast()` routine which is used to override the event-source
  object's handler.
- comment out all the auxiliary/want-to-have event source selection
  methods for now.
2022-11-10 11:45:49 -05:00
Tyler Goodlet 8f3fe8e542 Back link auto-y-ranging to ohlc chart from vlm overlay fsp 2022-11-10 11:45:49 -05:00
Tyler Goodlet 490d85aba5 Drop fast chart buffer to 2 days worth 2022-11-10 11:45:49 -05:00
goodboy ba2e1e04cd
Merge pull request #413 from pikers/pg_exts_fork
Pg exts fork
2022-11-08 12:47:32 -05:00
Tyler Goodlet 5d4929db9c Pin to our `pyqtgraph` fork's master branch 2022-10-31 15:00:38 -04:00
Tyler Goodlet c41400ae18 Use `.setRect()`; not sure how this was ever working? 2022-10-31 14:58:35 -04:00
Tyler Goodlet e71bd2cb1e Move axis-tick-values lru caching into our existing `Axis` 2022-10-31 14:23:29 -04:00
Tyler Goodlet be24473fb4 Adjust remaining chart internals to pg extensions
Mainly this involves instantiating our overriden `PlotItem` in a few
places and tweaking type annots. A further detail is that inside
the fsp sub-chart creation code we hide some axes for overlays in the
flows subchart; these were previously somehow hidden implicitly?
2022-10-31 14:13:02 -04:00
Tyler Goodlet b524ea5c22 Extract and fork `pyqtgraph` upstream submissions
Fork out our patch set submitted to upstream in multiple PRs (since they
aren't moving and/or aren't a priority to core) which can be seen in
full from the following diff:
https://github.com/pyqtgraph/pyqtgraph/compare/master...pikers:pyqtgraph:graphics_pin

Move these type extensions into the internal `.ui._pg_overrides` module.

The changes are related to both `pyqtgraph.PlotItem` and `.AxisItem` and
were driven for our need for multi-view overlays (overlaid charts with
optionally synced axis and interaction controls) as documented in the PR
to upstream: https://github.com/pyqtgraph/pyqtgraph/pull/2162

More specifically,
- wrt to `AxisItem` we added lru caching of tick values as per:
  https://github.com/pyqtgraph/pyqtgraph/pull/2160.
- wrt to `PlotItem` we adjusted some of the axis management code, namely
  adding a standalone `.removeAxis()` and modifying the `.setAxisItems()` logic
  to use it in: https://github.com/pyqtgraph/pyqtgraph/pull/2162
  as well as some tweaks to `.updateGrid()` to loop through all possible
  axes when grid setting.
2022-10-31 09:37:32 -04:00
Tyler Goodlet d46945cb09 Move profiler imports to internal version 2022-10-31 09:26:36 -04:00
Tyler Goodlet 1d4fc6f327 Fork our latency tune-able profiler from `pyqtgraph.debug`
Details of the original patch to upstream are in:
https://github.com/pyqtgraph/pyqtgraph/pull/2281

Instead of trying to land this we've opted to just copy out that version
of `.debug.Profiler` into our own internals (luckily the class is
entirely self-contained) until such a time when we choose to find
a better dependency as per https://github.com/pikers/piker/issues/337
2022-10-30 21:11:27 -04:00
Tyler Goodlet 5976acbe76 `PyQt5` + `pyqtgraph` import updates (`QtGui -> `QtWidgets`) 2022-10-30 21:11:14 -04:00
goodboy 11ecf9cb09
Merge pull request #401 from pikers/ib_1m_hist
Ib 1m hist
2022-10-29 13:14:53 -04:00
goodboy 2dac531729
Merge pull request #410 from pikers/even_moar_kraken_order_fixes
Even moar `kraken` order fixes
2022-10-28 19:52:20 -04:00
Tyler Goodlet 1fadf58ab7 Add todo for order duration setting `goodTillDuration` 2022-10-28 17:50:09 -04:00
Tyler Goodlet ceca0d9fb7 Order ledger entries by processed datetime
To make it easier to manually read/decipher long ledger files this adds
`dict` sorting based on record-type-specific (api vs. flex report)
datetime processing prior to ledger file write.

- break up parsers into separate routines for flex and api record
  processing.
- add `parse_flex_dt()` for special handling of the weird semicolon
  stamps in flex reports.
2022-10-28 16:17:27 -04:00
Tyler Goodlet df16726211 Just wipe wrong timeframe filled tsdb colseries for now 2022-10-28 16:17:14 -04:00
Tyler Goodlet fb4f1732b6 Drop key error again 2022-10-28 16:17:14 -04:00
Tyler Goodlet d5b357b69a Raise `DataUnavailable` on >= 6 no data error events 2022-10-28 16:17:14 -04:00
Tyler Goodlet 610fb5f7c6 Drop `NoData` handler, just let it bubble 2022-10-28 16:17:14 -04:00
Tyler Goodlet 2b231ba631 Lul, fix timeframe key when writing history
There never was any underlying db bug, it was a hardcoded timeframe in
the column series write key.. Now we always assert a matching timeframe
in results.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 286228c290 Only wait on backfill if provider supports timeframe 2022-10-28 16:17:14 -04:00
Tyler Goodlet a1a24da7b6 Make `binance` reject 1s OHLC history requests 2022-10-28 16:17:14 -04:00
Tyler Goodlet 553d0557b6 Raise `DataUnavailable` when a contract's 'earliest time' is hit 2022-10-28 16:17:14 -04:00
Tyler Goodlet 2f7b272d8c Make `ib` client's `.get_head_time()` (only) expect an fqsn 2022-10-28 16:17:14 -04:00
Tyler Goodlet dc1edeecda Do tsdb backloading to shm concurrently
Not only improves startup latency but also avoids a bug where the rt
buffer was being tsdb-history prepended *before* the backfilling of
recent data from the backend was complete resulting in our of order
frames in shm.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 4ca7817735 Use feed-shm offsets in fill-arrow indexing arithmetic 2022-10-28 16:17:14 -04:00
Tyler Goodlet 5b63585398 Pack multi-chart region linking into helper
Factor the multi-sample-rate region UI connecting into a new helper
`link_views_with_region()` which reads in the shm buffer offsets from
the `Feed` and appropriately connects the fast and slow chart handlers
for the linear region graphics. Add detailed comments writeup for the
inter-sampling transform algebra.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 0000d9a314 Handle backends with no 1s OHLC history
If a history manager raises a `DataUnavailable` just assume the sample
rate isn't supported and that no shm prepends will be done. Further seed
the shm array in such cases as before from the 1m history's last datum.

Also, fix tsdb -> shm back-loading, cancelling tsdb queries when either
no array-data is returned or a frame is delivered which has a start time
no lesser then the least last retrieved. Use strict timeframes for every
`Storage` API call.
2022-10-28 16:17:14 -04:00
Tyler Goodlet f7ec66362e Only get dbus user on sudo-user-present 2022-10-28 16:17:14 -04:00
Tyler Goodlet b7ef0596b9 Drop remaining timeframe scanning from `.read_ohlcv()` 2022-10-28 16:17:14 -04:00
Tyler Goodlet 143e86a80c Handle super annoying mkts query bug..
Turns out querying for a high freq timeframe (like 1sec) will still
return a lower freq timeframe (like 1Min) SMH, and no idea if it's the
server or the client's fault, so we have to explicitly check the sample
step size and discard lower freq series-results. Do this inside
`Storage.read_ohlcv()` and return an empty `dict` when the wrong time
step is detected from the query result.

Further enforcements,
- both `.load()` and `read_ohlcv()` now require an explicit `timeframe:
  int` input to guarantee the time step of the output array.
- drop all calls `.load()` with non-timeframe specific input.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 956c7d3435 Add concurrent multi-time-frame history loading
Our default sample periods are 60s (1m) for the history chart and 1s for
the fast chart. This patch adds concurrent loading of both (or more)
different sample period data sets using the existing loading code but
with new support for looping through a passed "timeframe" table which
points to each shm instance.

More detailed adjustments include:
- breaking the "basic" and tsdb loading into 2 new funcs:
  `basic_backfill()` and `tsdb_backfill()` the latter of which is run
  when the tsdb daemon is discovered.
- adjust the fast shm buffer to offset with one day's worth of 1s so
  that only up to a day is backfilled as history in the fast chart.
- adjust bus task starting in `manage_history()` to deliver back the
  offset indices for both fast and slow shms and set them on the
  `Feed` object as `.izero_hist/rt: int` values:
  - allows the chart-UI linked view region handlers to use the offsets
    in the view-linking-transform math to index-align the history and
    fast chart.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 330d16262e Add data-reset-task global state var
Allows keeping mutex state around data reset requests which (if more
then one are sent) can cause a throttling condition where ib's servers
will get slower and slower to conduct a reconnect. With this you can
have multiple ongoing contract requests without hitting that issue and
we can go back to having a nice 3s timeout on the history queries before
activating the hack.
2022-10-28 16:17:14 -04:00
Tyler Goodlet c7f57b940c Add back adhoc symbol lookup support, some exchs info is off 2022-10-28 16:17:14 -04:00
Tyler Goodlet 27bd3c07af Comment format tweak 2022-10-28 16:17:14 -04:00
Tyler Goodlet 55dc27a197 Subtract duration instead of passing to `.subtract()` (facepalm) 2022-10-28 16:17:14 -04:00
Tyler Goodlet a11f20fac2 Fix `piker services`; `tractor.run()` is done.. 2022-10-28 16:17:14 -04:00
Tyler Goodlet daebb78755 Re-request quote feed on data reset events
When a network outage or data feed connection is reset often the
`ib_insync` task will hang until some kind of (internal?) timeout takes
place or, in some (worst) cases it never re-establishes (the event
stream) and thus the backend needs to restart or the live feed will
never resume..

In order to avoid this issue once and for all this patch implements an
additional (extremely simple) task that is started with the  real-time
feed and simply waits for any market data reset events; when detected
restarts the `open_aio_quote_stream()` call in a loop using
a surrounding cancel scope.

Been meaning to implement this for ages and it's finally working!
2022-10-28 16:17:14 -04:00
Tyler Goodlet 90a395a069 Support no-disconnect on `open_aio_clients()` exit
Allows for easier restarts of certain `trio` side tasks without killing
the `asyncio`-side clients; support via flag.

Also fix a bug in `Client.bars()`: we need to return the duration on the
empty bars case..
2022-10-28 16:17:14 -04:00
Tyler Goodlet 23d0353934 Drop duplicate frame request
Must have gotten left in during refactor from the `trimeter` version?
Drop down to 6 years for 1m sampling.
2022-10-28 16:17:14 -04:00
Tyler Goodlet ede67ed184 Return history-frame duration from `.bars()`
This allows the history manager to know the decrement size for
`end_dt: datetime` on the next query if a no-data / gap case was
encountered; subtract this in `get_bars()` in such cases. Define the
expected `pendulum.Duration`s in the `.api._samplings` table.

Also add a bit of query latency profiling that we may use later to more
dynamically determine timeout driven data feed resets. Factor the `162`
error cases into a common exception handler block.
2022-10-28 16:17:14 -04:00
Tyler Goodlet 811d21e111 Explicit fast chart naming, auto-yrange the fast chart on increment 2022-10-28 16:17:14 -04:00