Commit Graph

2616 Commits (f2df32a673cd0c0ff5d78e43a4326fc004442265)

Author SHA1 Message Date
Tyler Goodlet 2fbfe583dd Drop the `Router.clients: set`, `.subscribers` is enough 2022-10-07 14:13:52 -04:00
Tyler Goodlet 525f805cdb Port order mode to new notify routine 2022-10-07 14:13:52 -04:00
Tyler Goodlet b65c02336d Don't short circuit relay loop when headless
If no clients are connected we now process as normal and try to fire
a desktop notification on linux.
2022-10-07 14:13:52 -04:00
Tyler Goodlet d3abfce540 Start notify mod, linux only 2022-10-07 14:13:52 -04:00
Tyler Goodlet 49433ea87d Run dark-clear-loop in daemon task
This enables "headless" dark order matching and clearing where an `emsd`
daemon subactor can be left running with active dark (or other
algorithmic) orders which will still trigger despite to attached-controlling
ems-client.

Impl details:
- rename/add `Router.maybe_open_trade_relays()` which now does all work
  of starting up ems-side long living clearing and relay tasks and the
  associated data feed; make is a `Nursery.start()`-able task instead of
  an `@acm`.
- drop `open_brokerd_trades_dialog()` and move/factor contents into the
  above method.
- add support for a `router.client_broadcast('all', msg)` to wholesale
  fan out a msg to all clients.
2022-10-07 14:13:52 -04:00
Tyler Goodlet 35871d0213 Support line update from `Order` msg in `.on_submit()` 2022-10-05 01:41:18 -04:00
Tyler Goodlet 4877af9bc3 Add pub-sub broadcasting
Establishes a more formalized subscription based fan out pattern to ems
clients who subscribe for order flow for a particular symbol (the fqsn
is the default subscription key for now).

Make `Router.client_broadcast()` take a `sub_key: str` value which
determines the set of clients to forward a message to and drop all such
manually defined broadcast loops from task (func) code. Also add
`.get_subs()` which (hackily) allows getting the set of clients for
a given sub key where any stream that is detected as "closed" is
discarded in the output. Further we simplify to `Router.dialogs:
defaultdict[str, set[tractor.MsgStream]]` and `.subscriptions` as maps
to sets of streams for much easier broadcast management/logic using set
operations inside `.client_broadcast()`.
2022-10-05 01:41:18 -04:00
Tyler Goodlet 909e068121 Support multi-client order-dialog management
This patch was originally to fix a bug where new clients who
re-connected to an `emsd` that was running a paper engine were not
getting updates from new fills and/or cancels. It turns out the solution
is more general: now, any client that creates a order dialog will be
subscribing to receive updates on the order flow set mapped for that
symbol/instrument as long as the client has registered for that
particular fqsn with the EMS. This means re-connecting clients as well
as "monitoring" clients can see the same orders, alerts, fills and
clears.

Impl details:
- change all var names spelled as `dialogues` -> `dialogs` to be
  murican.
- make `Router.dialogs: dict[str, defaultdict[str, list]]` so that each
  dialog id (oid) maps to a set of potential subscribing ems clients.
- add `Router.fqsn2dialogs: dict[str, list[str]]` a map of fqsn entries to
  sets of oids.
- adjust all core task code to make appropriate lookups into these 2 new
  tables instead of being handed specific client streams as input.
- start the `translate_and_relay_brokerd_events` task as a daemon task
  that lives with the particular `TradesRelay` such that dialogs cleared
  while no client is connected are still processed.
- rename `TradesRelay.brokerd_dialogue` -> `.brokerd_stream`
- broadcast all status msgs to all subscribed clients in the relay loop.
- always de-reg each client stream from the `Router.dialogs` table on close.
2022-10-05 01:41:18 -04:00
Tyler Goodlet cf835b97ca Add some info logs around paper fills 2022-10-05 01:41:18 -04:00
Tyler Goodlet 30bce42c0b Don't spin paper clear loop on non-clearing ticks
Not sure what exactly happened but it seemed clears weren't working in
some cases without this, also there's no point in spinning the simulated
clearing loop if we're handling a non-clearing tick type.
2022-10-05 01:41:18 -04:00
Tyler Goodlet 48ff4859e6 Update to new pair schema, adds `.cost_decimals` field 2022-10-05 01:41:18 -04:00
Tyler Goodlet 887583d27f Bleh, convert fill data to `float`s in kraken broker.. 2022-10-05 01:41:18 -04:00
Tyler Goodlet 45b97bf6c3 Make fill msg `.action: str` optional for `kraken` 2022-10-05 01:41:18 -04:00
Tyler Goodlet 91397b85a4 Fix missing f-str in ems msg sender err block 2022-10-05 01:41:18 -04:00
Tyler Goodlet 47f81b31af Kraken can cause status msg key error!? 2022-10-05 01:41:18 -04:00
Tyler Goodlet 41b0c11aaa Hide existing level line markers on startup 2022-09-23 17:17:32 -04:00
Tyler Goodlet cc67d23eee Drop old marker drawing code from `LevelLine.paint()`
We haven't been using it for a while and the supposed (remembered)
latency issue on interaction doesn't seem existing after applying the
cache mode. This allows dropping some internal state-logic and generally
simplifying the show-on-hover checks.

Further add `.show_markers()` and `.hide_markers()` as explicit methods
that can be called externally by UI business logic.
2022-09-23 17:17:32 -04:00
Tyler Goodlet 4818af1445 Add better doc string on marker factory 2022-09-21 15:43:35 -04:00
Tyler Goodlet 2cf1742999 Always apply at least the pos size as the limit 2022-09-21 15:43:35 -04:00
Tyler Goodlet 25ac6e6665 Soft pop lines, handle error-cancel races 2022-09-21 15:43:35 -04:00
Tyler Goodlet 90754f979b Tick the slow chart task on a 1sec index event 2022-09-19 17:39:26 -04:00
Tyler Goodlet c0d490ed63 Only show pos nav on non-zero size 2022-09-19 16:17:05 -04:00
Tyler Goodlet 7c6d12d982 Always set marker y-pos even if we're tracking its x-pos 2022-09-19 16:17:05 -04:00
Tyler Goodlet fd8c05e024 A lines entry should always exist or it's a bug 2022-09-19 16:17:05 -04:00
Tyler Goodlet 5d65c86c84 Don't delete pp lines or markers
Bit of a face palm but obviously `LevelLine.delete()` also removes any
`._marker` from the view which makes it disappear permanently when
moving from non-zero to zero to non-zero positions.. We don't really
need to delete the line since it can be re-used so just remove that
code.

Further this patch removes marker style setting logic from within the
`pp_line()` factory and instead expects the caller to set the correct
"direction" (for long / short) afterward.
2022-09-19 16:17:05 -04:00
Tyler Goodlet cf11e8d7d8 Update navs on all slow and fast charts, only default the fast chart on switch 2022-09-19 16:17:05 -04:00
Tyler Goodlet ed868f6246 Go back to origin slow chart split proportion 2022-09-19 16:17:05 -04:00
Tyler Goodlet 6897aed6b6 Don't call show on marker in `Nav.show()` 2022-09-14 16:02:07 -04:00
Tyler Goodlet a61a11f86b Add draft but commented "scale-to-fast-chart" logic 2022-09-14 10:11:43 -04:00
Tyler Goodlet 286f620f8e Use fqsn to key pnl tasks 2022-09-13 18:59:12 -04:00
Tyler Goodlet b7e60b9653 Hide labels, show markers for lines on slow chart 2022-09-13 18:31:21 -04:00
Tyler Goodlet df42e7acc4 Add `LevelLine.get_cursor()` to get any currently hovering mouse-cursor 2022-09-13 18:26:06 -04:00
Tyler Goodlet e492e9ca0c Fix pp arrow/label placement bugs
- Every time a symbol is switched on chart we need to wait until the
  search bar sidepane has been added beside the slow chart before
  determining the offset for the pp line's arrow/labels; trigger this in
  `GodWidget.load_symbol()` -> required monkeypatching on a
  `.mode: OrderMode` to the `.rt_linked` for now..
- Drop the search pane widget removal from the current linked chart,
  seems faster?
- On the slow chart override the `LevelMarker.scene_x()` callback to
  adjust for the case where no L1 labels are shown beside the y-axis.
2022-09-13 17:58:20 -04:00
Tyler Goodlet 44c6f6dfda Add level line flag to allow tracking its marker x-position 2022-09-13 17:43:04 -04:00
Tyler Goodlet ad2100fe3f Only don't pp arrow on startup 2022-09-13 16:21:49 -04:00
Tyler Goodlet ae64ac79a6 Doc str tweaks 2022-09-13 16:13:46 -04:00
Tyler Goodlet 20663dfa1c Add (more) order mode race guards to avoid crashes on "kitty-keys" 2022-09-12 20:25:15 -04:00
Tyler Goodlet 70f2241d22 Hide pp markers on startup 2022-09-12 20:25:15 -04:00
Tyler Goodlet b3fcc25e21 Add extra row count for header, drop prints 2022-09-12 20:25:15 -04:00
Tyler Goodlet 4f15ce346b Drop splitter resizes except for once at startup
Also adds a `GodWidget.resize_all()` helper method which resizes all
sub-widgets and charts to their default ratios and/or parent-widget
dependent defaults using the detected available space on screen. This is
a "default layout" config method that eventually we'll probably want
allow users to customize.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 445849337f Always resize to slow chart height, not just on changes 2022-09-12 20:25:15 -04:00
Tyler Goodlet 3fd7107e08 Scale view to measured results row count
In other words instead of some static view size previously determined by
the accompanying (slow) chart's height, (recursively) calculate the
number of displayed rows and compute the minimal height needed. This
still caps the view at the height of the chart such that the view will
switch to scroll bar mode when too many results are shown and can't all
be fit in the vertical space.

Deats:
- add a ``CompleterView.iter_df_rows()`` which recursively iterates all
  rows in depth-first order making it simple to compute the absolute
  number of result rows in view and thus the minimal number of pixels to
  show all results.
- always pass the height in the `.on_resize()` handler to ensure
  triggering the height logic when new results are generated in the
  search loop.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 73a02d54b7 Down size the slots bar by .9 2022-09-12 20:25:15 -04:00
Tyler Goodlet b734af6dd0 Only delete lines under cursor if not `None` 2022-09-12 20:25:15 -04:00
Tyler Goodlet f7c0ee930a Offset last (live) datum from y-axis by a 16th 2022-09-12 20:25:15 -04:00
Tyler Goodlet ead426abc4 More space to fast chart(s), less to slow chart 2022-09-12 20:25:15 -04:00
Tyler Goodlet bcd6bbb7ca Increase the `brokerd` mem-chan size
Intention is to hopefully minimize (as many) context switches when
processing (near-)HFT feeds - tho not sure if it's improving things that
much XD
2022-09-12 20:25:15 -04:00
Tyler Goodlet 80929d080f Add more detailed splitter of splitters comment 2022-09-12 20:25:15 -04:00
Tyler Goodlet eed47b3733 Add splitter move handler which calls search widget resizer method 2022-09-12 20:25:15 -04:00
Tyler Goodlet d5f0c59b57 Ignore resize events with the same height (for now) 2022-09-12 20:25:15 -04:00
Tyler Goodlet d11dc787a1 First working attempt of search results view scaling
Scales the "view" instance that holds search results to the size of the
accompanying "slow chart" for which the search pane is a "sidepane".
A lot of mucking about was required due to resizing of the view
seemingly feeding back into window resizing and further implementing the
sizing logic such that the parent `QSplitter` can be resized as the
user's whim as well.

Details,
- add a `CompleterView._init: bool` which is set once (and only once)
  after startup where the first display of the current symbol/feed is
  shown allowing and a single *width* padding applied once at startup
  to ensure we don't have an awkward line to the right of the longest
  result.
- in `.resize_to_results()` only apply a minimum height to the view
  using `.setMinimumHeight()` with a down-scaled (`0.91` for now) height
  value from input.
- re-implement `CompleterView.show_matches()` to accept and optional
  width, heigh tuple and when not supplied pull the slow chart's
  dimensions and pass as input to the resize method.
- Make `SearchWidget` x dim sizing policy "fixed".
- register the `SearchWidget` for resize events with god.
- add `.show_only_cache_entries()` for easy results clearing.
- add `.space_dims()` to retrieve slow linked-charts dimensions.
- implement `SearchWidget.on_resize()` which is the caller of all the
  previously mentioned resizing routines.
- do resizing and cache entry showing on search loop startup and be sure
  to clear to cache when the user selects a symbol-feed with Enter.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 1e81feee46 Finally get chart startup view-state kinda correct
It ended up being what'd you expect, races on the accessing shm buffer
data by the UI during the whole "mega-async-startup-everything" phase XD

So we add the following list of ad-hoc startup steps:
- do `.default_view()` on the slow chart after the fast chart is mostly
  fully spawned with the intention being to capture the state where the
  historical buffer is mostly loaded before sizing the view to the
  graphical form of the data.
- resize slow chart sidepanes from the fast chart just before sleeping
  forever (and after order mode has booted).
2022-09-12 20:25:15 -04:00
Tyler Goodlet 40a9761943 Actually support resize events..
Turns out god widget resizes aren't triggered implicitly by window
resizes, so instead, hook into the window by moving what was our useless
method to that class. Further we explicitly define and declare that our
window has a `.godwidget: GodWidget` and set it up in the bootstrap
phase - in `run_qutractor()` during `trio` guest mode configuration.

Further deatz:
- retype the runtime/bootstrap routines to take a qwidget "type" not an
  instance, and drop the whole implicit `.main_widget` stuff.
- delegate into the `GodWidget.on_win_resize()` for any window resize
  which then triggers all the custom resize callbacks we already had in
  place.
- privatize `ChartnPane.sidepane` so that it can't be mutated willy
  nilly without calling `.set_sidepane()`.
- always adjust splitter sizes inside `LinkeSplits.add_plot()`.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 256bcf36d3 Drop use `tractor.trionics.gather_contexts()` in `open_handlers()` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 9944277096 Handle null lines that were removed, don't error on bad $size 2022-09-12 20:25:15 -04:00
Tyler Goodlet f9dc5637fa Use rt buffer for last price on nan in ems 2022-09-12 20:25:15 -04:00
Tyler Goodlet addedc20f1 WIP search pane always shown.. 2022-09-12 20:25:15 -04:00
Tyler Goodlet 1fa6e8d9ba Only show slow chart xlabel when focussed 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2a06dc997f Use pixel caching on our level lines 2022-09-12 20:25:15 -04:00
Tyler Goodlet 6b93eedcda Port to new `._position.Nav` apis in order mode 2022-09-12 20:25:15 -04:00
Tyler Goodlet a786df65de Factor pos tracker UI element mgmt into new type
More or less moves all the UI related position "nav" logic and graphics
item management into a new `._position.Nav` composite type + api for
high level mgmt of position graphics indicators across multiple charts
(fast and slow).
2022-09-12 20:25:15 -04:00
Tyler Goodlet 8f2823d5f0 Stage line only on active cursor chart 2022-09-12 20:25:15 -04:00
Tyler Goodlet 58fe220fde Use ref annotations in position mod 2022-09-12 20:25:15 -04:00
Tyler Goodlet 161448c31a Support order staging from slow chart using `.get_cursor()` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 1c685189d1 Change to using real type annots 2022-09-12 20:25:15 -04:00
Tyler Goodlet ceac3f2ee4 Adjust corresponding fast/slow chart line level on edits 2022-09-12 20:25:15 -04:00
Tyler Goodlet a07367fae2 Fix div-by-zero split sizing bug 2022-09-12 20:25:15 -04:00
Tyler Goodlet 006190d227 Add fill arrow-mark support to history view 2022-09-12 20:25:15 -04:00
Tyler Goodlet 412197019e Make ArrowEditor.add()` expect a `PlotItem` as input for render 2022-09-12 20:25:15 -04:00
Tyler Goodlet 271e378ce3 Add `GodWidget.iter_linked()` interator over linked split charts 2022-09-12 20:25:15 -04:00
Tyler Goodlet 8e07fda88f Expose multi-chart-lines support through to order mode api 2022-09-12 20:25:15 -04:00
Tyler Goodlet a4935b8fa8 Make line editor multi-line aware, drop `dataclass` for `Struct` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2b76baeb10 Pass god widget to line editor and order mode instances 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2dfa8976a0 Make line editor expect god as input, use new .`get_cursor()` api 2022-09-12 20:25:15 -04:00
Tyler Goodlet d3402f715b Set godwidget active cursor from xhair callback 2022-09-12 20:25:15 -04:00
Tyler Goodlet f070f9a984 Add "active cursor" api to god widget 2022-09-12 20:25:15 -04:00
Tyler Goodlet 416270ee6c Refocus view on ctl-c from search 2022-09-12 20:25:15 -04:00
Tyler Goodlet 14bee778ec Hook up kb ctrls to hist chart, order mode not working yet 2022-09-12 20:25:15 -04:00
Tyler Goodlet 10c1944de5 Proper slow chart auto y-range support
The slow (history) chart requires it's own y-range checker logic which
needs to be run in 2 cases:
- the last datum is in view and goes outside the previous mx/mn in view
- the chart is incremented a step

Since we need this duplicate logic this patch also factors the incremental
graphics update info "reading" into a new `DisplayState.incr_info()`
method that can be configured to a chart and input state and returns all
relevant "graphics update measure" in a tuple (for now).

Use this method throughout the rest of the display loop for both fast
and slow chart checks and in the `increment_history_view()` slow chart
task.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 7958d8ad4f Up sample info poll loop iters 2022-09-12 20:25:15 -04:00
Tyler Goodlet 50c5dc255c Update history view y-sticky with last clear price 2022-09-12 20:25:15 -04:00
Tyler Goodlet 31735f26d3 Poll for sampling info at startup, tolerate races
Use the new `Feed.get_ds_info()` method in a poll loop to definitively
get the inter-chart sampling info and avoid races with shm buffer
backfilling.

Also, factor the history increment closure-task into
`graphics_update_loop()` which will make it clearer how to factor
all the "should we update" logic into some `DisplayState` API.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 2ef6460853 Add `Feed.get_ds_info()` to detect/compute sample rates 2022-09-12 20:25:15 -04:00
Tyler Goodlet 5e98a30537 Add simplified history incrementer consumer task 2022-09-12 20:25:15 -04:00
Tyler Goodlet dd03ef42ac Return empty search result on connection failure
If you spawn a brokerd set and no `ib` data feed was started (via our
`.data.feed.Feed` api) then there will be no active client loaded and
thus wont' be connected. So in these cases just return nothing, and
I guess we'll figure out real connection failures later?
2022-09-12 20:25:15 -04:00
Tyler Goodlet 59884d251e Update history "last" bar, compute ampling ratio
Add an update call to the display loop to consistently update the last
datum in the history view chart. Compute the inter-chart sampling ratio
and use it to sync the linear region.
2022-09-12 20:25:15 -04:00
Tyler Goodlet e06e257a81 Another history view splitter proportion tweak 2022-09-12 20:25:15 -04:00
Tyler Goodlet 6e574835c8 Update history shm buffer in ohlc sampler loop 2022-09-12 20:25:15 -04:00
Tyler Goodlet 49ccfdd673 Pass history shm "last index" in init msg, assign on feed 2022-09-12 20:25:15 -04:00
Tyler Goodlet 3a434f312b Add sidepane like color region styling 2022-09-12 20:25:15 -04:00
Tyler Goodlet bb4dc448b3 Add history chart and "linear region" for syncing
Add a first draft of a working `pyqtgraph.LinearRegionItem` link between
a history view chart (+ data set) and the normal real-time "HFT" chart
set.

Add the history view (aka more downsampled data view) chart set to the
rt/hft set's splitter as it's "first widget". Hook up linear region
callbacks to enable syncing between charts including compenstating for
the downsampling rate ration (in this case hardcoded 60 since 1s to 1M,
but we'll actually compute it going forward obvs).

More to come dawgys..
2022-09-12 20:25:15 -04:00
Tyler Goodlet 9846396df2 Add initial history (view) to charting sys
Adds an additional `GodWidget.hist_linked: LinkedSplits` alongside the
renamed `.rt_linked` to enable 2 sets of linked charts with different
sampled data sets/flows. The history set is added without "all the
fixins" for now (i.e. no order mode sidepane or search integration) such
that it is merely a top level chart which shows a much longer term
history and can be added to the UI via embedding the entire history
linked-splits instance into the real-time linked set's splitter.

Further impl deats:
- adjust the `GodWidget._chart_cache: dict[str, tuple]]` to store both
  linked split chart sets per symbol so that symbol switching will
  continue to work with the added history chart (set).
- rework `.load_symbol()` to operate on both the real-time (HFT) chart
  set and the history set.
- rework `LinkedSplits.set_split_sizes()` to compensate for the history
  chart and do more detailed height calcs arithmetic to make it appear
  by default as a minor sub-chart.
- adjust `LinkedSplits.add_plot()` and `ChartPlotWidget` internals to allow
  adding a plot without a sidepane and/or container `ChartnPane`
  composite widget by checking for a `sidepane == False` input.
- make `.default_view()` accept a manual y-axis offset kwarg.
- adjust search mode to provide history linked splits to
  `.set_chart_symbol()` call.
2022-09-12 20:25:15 -04:00
Tyler Goodlet f0d417ce42 Drop status msg var deleting from ns 2022-09-12 20:25:15 -04:00
Tyler Goodlet 55fc4114b4 Initial draft code working with `pg.LinearRegionItem` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 97b074365b Use rt buffer for close price pnl calcs 2022-09-12 20:25:15 -04:00
Tyler Goodlet f79c3617d6 Always load FSPs with the default (fast) sampling period 2022-09-12 20:25:15 -04:00
Tyler Goodlet 861fe791eb Allocate 2 shm buffers for history and real-time
As part of supporting a "history view" chart which shows downsampled
datums alongside our 1s (or higher) sampled OHLC we need a separate
buffer to store a the slower history from broker backends. This begins
that design by allocating 2 buffers:
- `rt_shm: ShmArray` which maps to a `/dev/shm/` file with `_rt` suffix
- `hist_shm: ShmArray` which maps to a file with `_hist` suffix

Deliver both of these shms back from both `manage_history()` and load
them as `Feed.rt_shm`/`.hist_shm` on the client side.

Impl deats:
- init the rt buffer with the first datum from loaded history and
  assign all OHLC values to that row's 'close' and the vlm to 0.
- pass the hist buffer to the backfiller task
- only spawn **one** global sampler array-row increment task per
  `brokerd` and pass in the 1s delay which we presume is our lowest
  OHLC sample rate for now.
- drop `open_sample_step_stream()` and just move its body contents into
  `Feed.index_stream()`
2022-09-12 20:25:15 -04:00
Tyler Goodlet 60052ff73a Presume shortest delay input to `increment_ohlc_buffer()`
Instead of worrying about the increment period per shm subscription,
just use the value passed as input and presume the caller knows that
only one task is necessary and that the wakeup (sampling) period should
be the shortest that is needed.

It's very unlikely we don't want at least a 1s sampling (both in terms
of task switching cost and general usage) which will eventually ship as
the default "real-time" feed "timeframe". Further, this "fast" increment
sampling task can handle all lower sampling periods (eg. 1m, 5m, 1H)
based on the current implementation just the same.

Also, add a global default sample period as `_defaul_delay_s` for use in
other internal modules.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 4d2708cd42 Force 1s sample step so crypto boiz can seee 2022-09-12 20:25:15 -04:00
Tyler Goodlet d1cc52dff5 Use new public lifetime-stack class attr 2022-09-12 20:24:56 -04:00
Tyler Goodlet 4fa901dbcb Port to new `tractor._runtime` mod 2022-09-12 20:24:56 -04:00
Tyler Goodlet 4a9c16d298 Fix stream type annot 2022-09-12 15:52:50 -04:00
Tyler Goodlet b9d5b904f4 Drop order entry removals on modify 2022-09-12 15:52:22 -04:00
Tyler Goodlet 0aef762d9a Bleh `kraken`, fix another ref error in fill block
Clearly, the linter didn't help us here.. but, just pass the
`brokerd` time for now in the `.broker_time` field; we can't get it from
the fill-case incremental updates in the `openOrders` sub. Add some
notes about this and how we might approach for backends with this
limitation.
2022-09-12 15:52:22 -04:00
goodboy c724117c1a
Merge pull request #398 from pikers/paper_clear_logics_fix
Oof, reverse clearing logic-routines in paper eng
2022-09-11 22:20:04 -04:00
Tyler Goodlet cc3bb85c66 Oof, reverse clearing logic-routines in paper eng 2022-09-10 16:35:31 -04:00
Tyler Goodlet 23d0b8a7ac Lul, fix name error on msg var name.. 2022-09-05 21:15:22 -04:00
Tyler Goodlet f9217570ab Add intiial `ib` backend readme 2022-08-31 17:38:24 -04:00
Tyler Goodlet 7f224f0342 Doc string typos 2022-08-31 17:22:15 -04:00
Tyler Goodlet de9f215c83 If more then one `ib` api client is available use next available for search 2022-08-31 17:22:15 -04:00
Tyler Goodlet 848e345364 POC using paper-in-docker gw for symbol search 2022-08-31 17:22:15 -04:00
Tyler Goodlet 38b190e598 Add `ib` `Crypto` contract support 2022-08-31 17:22:15 -04:00
Tyler Goodlet 7dfa4c3cde Better comment on the `size`'s purpose/units 2022-08-29 13:56:26 -04:00
Tyler Goodlet 7b653fe4f4 Store shm array size in token schema, use for loading 2022-08-29 13:46:41 -04:00
goodboy 77a687bced
Merge pull request #386 from pikers/paper_tolerance
Paper race tolerance
2022-08-29 13:28:38 -04:00
Tyler Goodlet d5c1cdd91d Configure allocator from pos msg on startup
This fixes a regression added after moving the msg parsing to later in
the order mode startup sequence. The `Allocator` needs to be configured
*to* the initial pos otherwise default settings will show in the UI..

Move the startup config logic from inside `mk_allocator()` to
`PositionTracker.update_from_pp()` and add a flag to allow setting the
`.startup_pp` from the current live one as is needed during initial
load.
2022-08-29 11:39:28 -04:00
Tyler Goodlet 46d3fe88ca Fix sub-slot-remains limiting for -ve sizes
In the short case (-ve size) we had a bug where the last sub-slots worth
of exit size would never be limited to zero once the allocator limit pos
size was hit (i.e. you could keep going more -ve on the pos,
exponentially per slot over the limit). It's a simple fix, just
a `max()` around the `l_sub_pp` var used in the next-step-size calc.

Resolves #392
2022-08-28 13:51:54 -04:00
Tyler Goodlet 5c8c5d8fbf Fix disti-mode paper pps relaying
Turns out we were putting too many brokername suffixes in the symbol
field and thus the order mode msg parser wasn't matching the current
asset to said msgs correctly and pps weren't being shown...

This repairs that plus simplifies the order mode initial pos msg loading
to just delegate into `process_trade_msg()` just as is done for
real-time msg updates.
2022-08-27 15:37:54 -04:00
Guillermo Rodriguez 0c323fdc0b
Minor style changes and warning on unexpected msg 2022-08-27 09:12:02 -03:00
Tyler Goodlet 02f53d0c13 Error on zero-size orders received by paper engine 2022-08-26 10:46:47 -04:00
Tyler Goodlet 8792c97de6 More stringent settings pane input handling
If a setting fails to apply try to log an error msg and revert to the
previous setting by not applying the UI read-update until after the new
`SettingsPane.apply_setting()` call. This prevents crashes when the user
tries to give bad inputs on editable allocator fields.
2022-08-26 10:46:47 -04:00
Tyler Goodlet 980815d075 Avoid handling account as numeric field in settings 2022-08-26 10:46:46 -04:00
Tyler Goodlet 4cedfedc21 Support clearing ticks ('last' & 'trade') fills
Previously we only simulated paper engine fills when the data feed
provide L1 queue-levels matched an execution. This patch add further
support for clear-level matches when there are real live clears on the
data feed that are faster/not synced with the L1 (aka usually during
periods of HFT).

The solution was to simply iterate the interleaved paper book entries on
both sides for said tick types and instead yield side-specific predicate
per entry.
2022-08-26 10:46:46 -04:00
Tyler Goodlet fe3d0c6fdd Handle too-fast-edits with `defaultdict[str, bidict[str, tuple]]`
Not entirely sure why this all of a sudden became a problem but it seems
price changes on order edits were sometimes resulting in key errors when
modifying paper book entries quickly. This changes the implementation to
not care about matching the last price when keying/popping old orders
and use `bidict`s to more easily pop cleared orders in the paper loop.
2022-08-26 10:46:46 -04:00
Tyler Goodlet 9200e8da57 Raw-dog-pop cancelled paper entries; old price dun matter 2022-08-26 10:46:46 -04:00
Tyler Goodlet 430d065da6 Handle paper-engine too-fast clearing race cases
When the paper engine is used it seems we can definitely hit races where
order ack msgs arrive close enough to status messages that `trio`
schedules the status processing before the acks. In such cases we want
to be tolerant and not crash but instead warn that we got an
unknown/out-of-order msg.
2022-08-26 10:46:46 -04:00
Tyler Goodlet ecd93cb05a Pass symbol with broker suffix to `.submit_limit()`; fix clearing 2022-08-26 10:46:46 -04:00
Guillermo Rodriguez 4facd161a9
Pull jsonrpc machinery out of deribit backend into piker.data._web_bs module and make it generic 2022-08-25 14:08:09 -03:00
Guillermo Rodriguez 0447612b34
Oneliner enable rpc modules on runtime open 2022-08-25 11:47:40 -03:00
Guillermo Rodriguez 00aabddfe8
Fix link 2022-08-25 09:22:15 -03:00
Guillermo Rodriguez 43fb720877
Do multiline imports 2022-08-25 09:20:41 -03:00
Guillermo Rodriguez 9626dbd7ac
Simplify rpc machinery, and switch refs to Dict and List to builtins, make brokercheck call public broker methods and get their results again 2022-08-25 09:18:52 -03:00
Guillermo Rodriguez f286c79a03
Woops enable backfill_bars in module __init__.py 2022-08-24 19:41:04 -03:00
Guillermo Rodriguez accb0eee6c
Add brokercheck guard on deribit.get_client && drop method running in brokercheck 2022-08-24 19:32:54 -03:00
Guillermo Rodriguez e97dd1cbdb
Stop using as much closures
Use a custom tractor branch that fixes a `maybe_open_context` re entrant related bug
2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 34fb497eb4
Add aiter api to NoBsWs and rework cryptofeed relay to not be OOPy 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 6669ba6590
Switch back to using async for and dont install signal handlers on cryptofeed 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez cb8099bb8c
Add README.rst and brokers.toml section in config example 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 80a1a58bfc
Refactor cryptofeed relay api and move it to client
Added submit_limit and submit_cancel
Cache syms correctly
Lowercase search results
2022-08-24 18:09:32 -03:00
Guillermo Rodriguez d60f222bb7
Add get_balances, and get_assets rpc to deribit.api.Client
Improve symbol_info search results
Expect cancellation on cryptofeeds asyncio task
Fix the no trades on instrument bug that we had on startup
2022-08-24 18:08:45 -03:00
Guillermo Rodriguez 2c2e43d8ac
Add comments and update cryptofeed fork url in requirements 2022-08-24 18:08:31 -03:00
Guillermo Rodriguez 212b3d620d
Tweaks on Client init to make api credentials optional 2022-08-24 18:08:29 -03:00
Guillermo Rodriguez 92090b01b8
Begin jsonrpc over ws refactor 2022-08-24 18:06:00 -03:00
Guillermo Rodriguez 9073fbc317
drop pydantic to match master 2022-08-23 15:18:45 -03:00
Guillermo Rodriguez f55f56a29f
Refactored deribit backend into new multi file format 2022-08-23 15:18:45 -03:00
Guillermo Rodriguez 28e025d02e
Finally get a chart going! lots of fixes to streaming machinery and custom cryptofeed fork with fixes 2022-08-23 15:18:43 -03:00
Guillermo Rodriguez e558e5837e
Introduce piker protocol in stream_messages 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez a0b415095a
Brokermod check output fixed and tweaks to deribit Client.bars function 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez 6df181c233
Add brokercheck test and got deribit to dump l1 and trades to console 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez 7acc4e3208
Initial deribit mock up 2022-08-23 15:17:18 -03:00
Tyler Goodlet b9dba48306 Show correct account label on loaded order lines
Quite a simple fix, we just assign the account-specific
`PositionTracker` to the level line's `._on_level_change()` handler
instead of whatever the current `OrderMode.current_pp` is set to.

Further this adds proper pane switching support such that when a user
modifies an order line from an account which is not the currently
selected one, the settings pane is changed to reflect the
account and thus corresponding position info for that account and
instrument B)
2022-08-18 16:04:44 -04:00
Tyler Goodlet 4d2e23b5ce Expose level line marker via property 2022-08-18 16:00:41 -04:00
Tyler Goodlet 973bf87e67 Don't log aboout unknown status msg if no oid 2022-08-18 11:51:12 -04:00
Tyler Goodlet 5861839783 Fix multi-account order loading..
We were overwriting the existing loaded orders list in the per client
loop (lul) so move the def above all that.

Comment out the "try-to-cancel-inactive-orders-via-task-after-timeout"
stuff pertaining to https://github.com/erdewit/ib_insync/issues/363 for
now since we don't have a mechanism in place to cancel the re-cancel
task once the order is cancelled - plus who knows if this is even the
best way to do it..
2022-08-18 11:51:12 -04:00
Tyler Goodlet 06845e5504 `kraken`: drop `make_sub()` and inline sub defs in `subscribe()` 2022-08-18 11:51:12 -04:00
Tyler Goodlet 43bdd4d022 Pass correct instrument symbol in position msgs 2022-08-18 11:51:12 -04:00
Tyler Goodlet bafd2cb44f Only relay fills if dialog still alive 2022-08-18 11:51:12 -04:00
Tyler Goodlet be8fd32e7d Only emit ems fill msgs for 'status' events from ib
Fills seems to be dual emitted from both the `status` and `fill` events
in `ib_insync` internals and more or less contain the same data nested
inside their `Trade` type. We started handling the 'fill' case to deal
with a race issue in commissions/cost report tracking but we don't
really want to leak that same race to incremental fills vs.
order-"closed" tracking.. So go back to only emitting the fill msgs
on statuses and a "closed" on `.remaining == 0`.
2022-08-18 11:51:12 -04:00
Tyler Goodlet ee8c00684b Add actor-global "broker client" for tracking reqids 2022-08-18 11:51:12 -04:00
Tyler Goodlet 7379dc03af The `ps1` check doesn't work for `pdb`.. 2022-08-18 11:51:12 -04:00
Tyler Goodlet a602c47d47 Support loading paper engine live orders 2022-08-18 11:51:12 -04:00
Tyler Goodlet 317610e00a Store positions globally and deliver on ctx connects 2022-08-18 11:51:12 -04:00
Tyler Goodlet c4af706d51 Make order-book-vars globals to persist across ems-dialog connections 2022-08-18 11:51:12 -04:00
Tyler Goodlet 665bb183f7 Unpack existing live order params in case statement 2022-08-18 11:51:12 -04:00
Tyler Goodlet f6ba95a6c7 Split existing live-open case into its own block 2022-08-18 11:51:12 -04:00
Tyler Goodlet e2cd8c4aef Add initial `kraken` live order loading 2022-08-18 11:51:12 -04:00
Tyler Goodlet c8bff81220 Add runtime guards around feed pausing during interaction 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2aec1c5f1d Only pprint our struct when we detect a py REPL 2022-08-18 11:51:12 -04:00
Tyler Goodlet bec32956a8 Move fill case-block earlier, log broker errors 2022-08-18 11:51:12 -04:00
Tyler Goodlet 91fdc7c5c7 Load boxed `.req` values as `Order`s in mode loop 2022-08-18 11:51:12 -04:00
Tyler Goodlet b59ed74bc1 'Only send `'closed'` on Filled events, lowercase all statues' 2022-08-18 11:51:12 -04:00
Tyler Goodlet 16012f6f02 Include both symbols in error msg when a mismatch 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2b61672723 Handle 'closed' vs. 'fill` race case..
`ib` is super good not being reliable with order event sequence order
and duplication of fill info. This adds some guards to try and avoid
popping the last status status too early if we end up receiving
a `'closed'` before the expected `'fill`' event(s). Further delete the
`status_msg` ref on each iteration to avoid stale reference lookups in
the relay task/loop.
2022-08-18 11:51:12 -04:00
Tyler Goodlet 176b230a46 Use modern `Union` pipe op syntax for msg fields 2022-08-18 11:51:12 -04:00
Tyler Goodlet 7fa9dbf869 Add full EMS order-dialog (re-)load support!
This includes darks, lives and alerts with all connecting clients
being broadcast all existing order-flow dialog states. Obviously
for now darks and alerts only live as long as the `emsd` actor lifetime
(though we will store these in local state eventually) and "live" orders
have lifetimes managed by their respective backend broker.

The details of this change-set is extensive, so here we go..

Messaging schema:
- change the messaging `Status` status-key set to:
  `resp: Literal['pending', 'open', 'dark_open', 'triggered',
                'closed',  'fill', 'canceled', 'error']`

  which better reflects the semantics of order lifetimes and was
  partially inspired by the status keys `kraken` provides for their
  order-entry API. The prior key set was based on `ib`'s horrible
  semantics which sound like they're right out of the 80s..
  Also, we reflect this same set in the `BrokerdStatus` msg and likely
  we'll just get rid of the separate brokerd-dialog side type
  eventually.
- use `Literal` type annots for statuses where applicable and as they
  are supported by `msgspec`.
- add additional optional `Status` fields:
  -`req: Order` to allow each status msg to optionally ref its
    commanding order-request msg allowing at least a request-response
    style implicit tracing in all response msgs.
  -`src: str` tag string to show the source of the msg.
  -`reqid: str | int` such that the ems can relay the `brokerd`
    request id both to the client side and have one spot to look
    up prior status msgs and
- draft a (unused/commented) `Dialog` type which can be eventually used
  at all EMS endpoints to track msg-flow states

EMS engine adjustments/rework:
- use the new status key set throughout and expect `BrokerdStatus` msgs
  to use the same new schema as `Status`.
- add a `_DarkBook._active: dict[str, Status]` table which is now used for
  all per-leg-dialog associations and order flow state tracking
  allowing for the both the brokerd-relay and client-request handler loops
  to read/write the same msg-table and provides for delivering
  the overall EMS-active-orders state to newly/re-connecting clients
  with minimal processing; this table replaces what the `._ems_entries`
  table from prior.
- add `Router.client_broadcast()` to send a msg to all currently
  connected peers.
- a variety of msg handler block logic tweaks including more `case:`
  blocks to be both flatter and improve explicitness:
  - for the relay loop move all `Status` msg update and sending to
    within each block instead of a fallthrough case plus hard-to-follow
    state logic.
  - add a specific case for unhandled backend status keys and just log
    them.
  - pop alerts from `._active` immediately once triggered.
  - where possible mutate status msgs fields over instantiating new
    ones.
- insert and expect `Order` instances in the dark clearing loop and
  adjust `case:` blocks accordingly.
- tag `dark_open` and `triggered` statuses as sourced from the ems.
- drop all the `ChainMap` stuff for now; we're going to make our own
  `Dialog` type for this purpose..

Order mode rework:
- always parse the `Status` msg and use match syntax cases with object
  patterns, hackily assign the `.req` in many blocks to work around not
  yet having proper on-the-wire decoding yet.
- make `.load_unknown_dialog_from_msg()` expect a `Status` with boxed
  `.req: Order` as input.
- change `OrderDialog` -> `Dialog` in prep for a general purpose type
  of the same name.

`ib` backend order loading support:
- do "closed" status detection inside the msg-relay loop instead
  of expecting the ems to do this..
- add an attempt to cancel inactive orders by scheduling cancel
  submissions continually (no idea if this works).
- add a status map to go from the 80s keys to our new set.
- deliver `Status` msgs with an embedded `Order` for existing live order
  loading and make sure to try an get the source exchange info (instead
  of SMART).

Paper engine ported to match:
- use new status keys in `BrokerdStatus` msgs
- use `match:` syntax in request handler loop
2022-08-18 11:51:12 -04:00
Tyler Goodlet 87ed9abefa WIP playing with a `ChainMap` of messages 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2548aae73d Deliver existing dialog (msgs) to every EMS client
Ideally every client that connects to the ems can know its state
(immediately) meaning relay all the order dialogs that are currently
active. This adds full (hacky WIP) support to receive those dialog
(msgs) from the `open_ems()` startup values via the `.started()` msg
from `_emsd_main()`.

Further this adds support to the order mode chart-UI to display existing
(live) orders on the chart during startup. Details include,

- add a `OrderMode.load_unknown_dialog_from_msg()` for processing and
  displaying a ``BrokerdStatus`` (for now) msg from the EMS that was not
  previously created by the current ems client and registering and
  displaying it on the chart.
- break out the ems msg processing into a new
  `order_mode.process_trade_msg()` func so that it can be called on the
  startup dialog-msg set as well as eventually used a more general low
  level auto-strat API (eg. when we get to displaying auto-strat and
  group trading automatically on an observing chart UI.
- hackyness around msg-processing for the dialogs delivery since we're
  technically delivering `BrokerdStatus` msgs when the client-side
  processing technically expects `Status` msgs.. we'll rectify this
  soon!
2022-08-18 11:51:12 -04:00
Tyler Goodlet 1cfa04927d Lol, handle failed-to-cancel statuses.. 2022-08-18 11:51:12 -04:00
Tyler Goodlet e34ea94f9f Start brokerd relay loop after opening client stream
In order to avoid missed existing order message emissions on startup we
need to be sure the client side stream is registered with the router
first. So break out the starting of the
`translate_and_relay_brokerd_events()` task until inside the client
stream block and start the task using the dark clearing loop nursery.

Also, ensure `oid` (and thus for `ib` the equivalent re-used `reqid`)
are cast to `str` before registering the dark book. Deliver the dark
book entries as part of the `_emsd_main()` context `.started()` values.
2022-08-18 11:51:12 -04:00
Tyler Goodlet 1510383738 Always cast ems `requid` values to `int` 2022-08-18 11:51:12 -04:00
Tyler Goodlet 016b669d63 Drop staged line runtime guard 2022-08-18 11:51:12 -04:00
Tyler Goodlet 682a0191ef First draft: relay open orders through ems and display on chart 2022-08-18 11:51:12 -04:00
Tyler Goodlet 9e36dbe47f Relay existing open orders from ib on startup 2022-08-18 11:51:12 -04:00
Tyler Goodlet 52febac6ae Facepalm: order-handler tasks are one-to-one with unique clients 2022-08-18 11:34:11 -04:00
Tyler Goodlet f202699c25 Fix scan loop: only stash clients that actually connect.. 2022-08-18 11:34:11 -04:00
Tyler Goodlet 0fb07670d2 Fix multi-account positioning and order tracking..
This seems to have been broken in refactoring from commit 279c899de5
which was never tested against multiple accounts/clients.

The fix is 2 part:
- position tables are now correctly loaded ahead of time and used by
  account for each connected client in processing of ledgers and
  existing positions.
- a task for each API client is started (as implemented prior) so that
  we actually get status updates for every client used for submissions.

Further we add a bit of code using `bisect.insort()` to normalize
ledgers to a datetime sorted list records (though pretty sure the `dict`
transform ruins it?) in an effort to avoid issues with ledger
transaction processing with previously minimized `Position.clears`
tables, which should (but might not?) avoid incorporating clear events
prior to the last "net-zero" positioning state.
2022-08-17 14:14:20 -04:00
Tyler Goodlet 73d2e7716f Pre-loop clients to build out pps tables, handle missing commission field 2022-08-17 10:23:01 -04:00
Tyler Goodlet 999ae5a1c6 Handle `Position.split_ratio` in state audits
This firstly changes `.audit_sizing()` => `.ensure_state()` and makes it
return `None` as well as only error when split ratio denoted (via
config) positions do not size as expected.

Further refinements,
- add an `.expired()` predicate method
- always return a size of zero from `.calc_size()` on expired assets
- load each `pps.toml` entry's clear tabe into `Transaction`s and use
  `.add_clear()` during from config init.
2022-08-17 10:06:58 -04:00
Tyler Goodlet 23ba0e5e69 Don't raise on missing position for now, just error log 2022-08-17 10:06:41 -04:00
Tyler Goodlet 941a2196b3 Get pos entry from table not `updated: dict` output 2022-08-17 10:06:37 -04:00
Tyler Goodlet 0cf4e07b84 Use `datetime` sorting on clears table appends
In order to avoid issues with reloading ledger and API trades after an
existing `pps.toml` exists we have to make sure we not only avoid
duplicate entries but also avoid re-adding entries that would have been
removed during a prior call to the `Position.minimize_clears()` filter.
The easiest way to do this is to sort on timestamps and avoid adding any
record that pre-existed the last net-zero position ledger event that
`.minimize_clears()` discarded. In order to implement this it means
parsing config file clears table's timestamps into datetime objects for
inequality checks and we add a `Position.first_clear_dt` attr for
storing this value when managing pps in object form but never store it
in the config (since it should be obviously from the sorted clear event
table).
2022-08-17 10:05:05 -04:00
Tyler Goodlet 7bec989eed First try mega-basic stock (reverse) split support with `ib` and `pps.toml` 2022-08-17 09:54:49 -04:00
Tyler Goodlet 6856ca207f Fix for TWS created position loading 2022-08-17 09:53:42 -04:00
Tyler Goodlet a83bd9c608 Drop `msgpack` from `marketstore` module 2022-08-11 14:21:36 -04:00
Tyler Goodlet 109b35f6eb Matchify paper clearing loop 2022-08-05 21:02:15 -04:00
Tyler Goodlet e28c1748fc Comment out "unknown msg" case for now 2022-08-05 21:02:15 -04:00
Tyler Goodlet 72889b4d1f Fix reference error 2022-08-05 21:02:15 -04:00
Tyler Goodlet ae001c3dd7 Matchify the dark trigger loop 2022-08-05 21:02:15 -04:00
Tyler Goodlet 2309e7ab05 Flatten the brokerd-dialog relay loop using `match:` 2022-08-05 21:02:15 -04:00
Tyler Goodlet 46c51b55f7 Flatten the client-request handler loop with `match:` 2022-08-05 21:02:15 -04:00