Commit Graph

2616 Commits (f2df32a673cd0c0ff5d78e43a4326fc004442265)

Author SHA1 Message Date
Tyler Goodlet d11dc787a1 First working attempt of search results view scaling
Scales the "view" instance that holds search results to the size of the
accompanying "slow chart" for which the search pane is a "sidepane".
A lot of mucking about was required due to resizing of the view
seemingly feeding back into window resizing and further implementing the
sizing logic such that the parent `QSplitter` can be resized as the
user's whim as well.

Details,
- add a `CompleterView._init: bool` which is set once (and only once)
  after startup where the first display of the current symbol/feed is
  shown allowing and a single *width* padding applied once at startup
  to ensure we don't have an awkward line to the right of the longest
  result.
- in `.resize_to_results()` only apply a minimum height to the view
  using `.setMinimumHeight()` with a down-scaled (`0.91` for now) height
  value from input.
- re-implement `CompleterView.show_matches()` to accept and optional
  width, heigh tuple and when not supplied pull the slow chart's
  dimensions and pass as input to the resize method.
- Make `SearchWidget` x dim sizing policy "fixed".
- register the `SearchWidget` for resize events with god.
- add `.show_only_cache_entries()` for easy results clearing.
- add `.space_dims()` to retrieve slow linked-charts dimensions.
- implement `SearchWidget.on_resize()` which is the caller of all the
  previously mentioned resizing routines.
- do resizing and cache entry showing on search loop startup and be sure
  to clear to cache when the user selects a symbol-feed with Enter.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 1e81feee46 Finally get chart startup view-state kinda correct
It ended up being what'd you expect, races on the accessing shm buffer
data by the UI during the whole "mega-async-startup-everything" phase XD

So we add the following list of ad-hoc startup steps:
- do `.default_view()` on the slow chart after the fast chart is mostly
  fully spawned with the intention being to capture the state where the
  historical buffer is mostly loaded before sizing the view to the
  graphical form of the data.
- resize slow chart sidepanes from the fast chart just before sleeping
  forever (and after order mode has booted).
2022-09-12 20:25:15 -04:00
Tyler Goodlet 40a9761943 Actually support resize events..
Turns out god widget resizes aren't triggered implicitly by window
resizes, so instead, hook into the window by moving what was our useless
method to that class. Further we explicitly define and declare that our
window has a `.godwidget: GodWidget` and set it up in the bootstrap
phase - in `run_qutractor()` during `trio` guest mode configuration.

Further deatz:
- retype the runtime/bootstrap routines to take a qwidget "type" not an
  instance, and drop the whole implicit `.main_widget` stuff.
- delegate into the `GodWidget.on_win_resize()` for any window resize
  which then triggers all the custom resize callbacks we already had in
  place.
- privatize `ChartnPane.sidepane` so that it can't be mutated willy
  nilly without calling `.set_sidepane()`.
- always adjust splitter sizes inside `LinkeSplits.add_plot()`.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 256bcf36d3 Drop use `tractor.trionics.gather_contexts()` in `open_handlers()` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 9944277096 Handle null lines that were removed, don't error on bad $size 2022-09-12 20:25:15 -04:00
Tyler Goodlet f9dc5637fa Use rt buffer for last price on nan in ems 2022-09-12 20:25:15 -04:00
Tyler Goodlet addedc20f1 WIP search pane always shown.. 2022-09-12 20:25:15 -04:00
Tyler Goodlet 1fa6e8d9ba Only show slow chart xlabel when focussed 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2a06dc997f Use pixel caching on our level lines 2022-09-12 20:25:15 -04:00
Tyler Goodlet 6b93eedcda Port to new `._position.Nav` apis in order mode 2022-09-12 20:25:15 -04:00
Tyler Goodlet a786df65de Factor pos tracker UI element mgmt into new type
More or less moves all the UI related position "nav" logic and graphics
item management into a new `._position.Nav` composite type + api for
high level mgmt of position graphics indicators across multiple charts
(fast and slow).
2022-09-12 20:25:15 -04:00
Tyler Goodlet 8f2823d5f0 Stage line only on active cursor chart 2022-09-12 20:25:15 -04:00
Tyler Goodlet 58fe220fde Use ref annotations in position mod 2022-09-12 20:25:15 -04:00
Tyler Goodlet 161448c31a Support order staging from slow chart using `.get_cursor()` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 1c685189d1 Change to using real type annots 2022-09-12 20:25:15 -04:00
Tyler Goodlet ceac3f2ee4 Adjust corresponding fast/slow chart line level on edits 2022-09-12 20:25:15 -04:00
Tyler Goodlet a07367fae2 Fix div-by-zero split sizing bug 2022-09-12 20:25:15 -04:00
Tyler Goodlet 006190d227 Add fill arrow-mark support to history view 2022-09-12 20:25:15 -04:00
Tyler Goodlet 412197019e Make ArrowEditor.add()` expect a `PlotItem` as input for render 2022-09-12 20:25:15 -04:00
Tyler Goodlet 271e378ce3 Add `GodWidget.iter_linked()` interator over linked split charts 2022-09-12 20:25:15 -04:00
Tyler Goodlet 8e07fda88f Expose multi-chart-lines support through to order mode api 2022-09-12 20:25:15 -04:00
Tyler Goodlet a4935b8fa8 Make line editor multi-line aware, drop `dataclass` for `Struct` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2b76baeb10 Pass god widget to line editor and order mode instances 2022-09-12 20:25:15 -04:00
Tyler Goodlet 2dfa8976a0 Make line editor expect god as input, use new .`get_cursor()` api 2022-09-12 20:25:15 -04:00
Tyler Goodlet d3402f715b Set godwidget active cursor from xhair callback 2022-09-12 20:25:15 -04:00
Tyler Goodlet f070f9a984 Add "active cursor" api to god widget 2022-09-12 20:25:15 -04:00
Tyler Goodlet 416270ee6c Refocus view on ctl-c from search 2022-09-12 20:25:15 -04:00
Tyler Goodlet 14bee778ec Hook up kb ctrls to hist chart, order mode not working yet 2022-09-12 20:25:15 -04:00
Tyler Goodlet 10c1944de5 Proper slow chart auto y-range support
The slow (history) chart requires it's own y-range checker logic which
needs to be run in 2 cases:
- the last datum is in view and goes outside the previous mx/mn in view
- the chart is incremented a step

Since we need this duplicate logic this patch also factors the incremental
graphics update info "reading" into a new `DisplayState.incr_info()`
method that can be configured to a chart and input state and returns all
relevant "graphics update measure" in a tuple (for now).

Use this method throughout the rest of the display loop for both fast
and slow chart checks and in the `increment_history_view()` slow chart
task.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 7958d8ad4f Up sample info poll loop iters 2022-09-12 20:25:15 -04:00
Tyler Goodlet 50c5dc255c Update history view y-sticky with last clear price 2022-09-12 20:25:15 -04:00
Tyler Goodlet 31735f26d3 Poll for sampling info at startup, tolerate races
Use the new `Feed.get_ds_info()` method in a poll loop to definitively
get the inter-chart sampling info and avoid races with shm buffer
backfilling.

Also, factor the history increment closure-task into
`graphics_update_loop()` which will make it clearer how to factor
all the "should we update" logic into some `DisplayState` API.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 2ef6460853 Add `Feed.get_ds_info()` to detect/compute sample rates 2022-09-12 20:25:15 -04:00
Tyler Goodlet 5e98a30537 Add simplified history incrementer consumer task 2022-09-12 20:25:15 -04:00
Tyler Goodlet dd03ef42ac Return empty search result on connection failure
If you spawn a brokerd set and no `ib` data feed was started (via our
`.data.feed.Feed` api) then there will be no active client loaded and
thus wont' be connected. So in these cases just return nothing, and
I guess we'll figure out real connection failures later?
2022-09-12 20:25:15 -04:00
Tyler Goodlet 59884d251e Update history "last" bar, compute ampling ratio
Add an update call to the display loop to consistently update the last
datum in the history view chart. Compute the inter-chart sampling ratio
and use it to sync the linear region.
2022-09-12 20:25:15 -04:00
Tyler Goodlet e06e257a81 Another history view splitter proportion tweak 2022-09-12 20:25:15 -04:00
Tyler Goodlet 6e574835c8 Update history shm buffer in ohlc sampler loop 2022-09-12 20:25:15 -04:00
Tyler Goodlet 49ccfdd673 Pass history shm "last index" in init msg, assign on feed 2022-09-12 20:25:15 -04:00
Tyler Goodlet 3a434f312b Add sidepane like color region styling 2022-09-12 20:25:15 -04:00
Tyler Goodlet bb4dc448b3 Add history chart and "linear region" for syncing
Add a first draft of a working `pyqtgraph.LinearRegionItem` link between
a history view chart (+ data set) and the normal real-time "HFT" chart
set.

Add the history view (aka more downsampled data view) chart set to the
rt/hft set's splitter as it's "first widget". Hook up linear region
callbacks to enable syncing between charts including compenstating for
the downsampling rate ration (in this case hardcoded 60 since 1s to 1M,
but we'll actually compute it going forward obvs).

More to come dawgys..
2022-09-12 20:25:15 -04:00
Tyler Goodlet 9846396df2 Add initial history (view) to charting sys
Adds an additional `GodWidget.hist_linked: LinkedSplits` alongside the
renamed `.rt_linked` to enable 2 sets of linked charts with different
sampled data sets/flows. The history set is added without "all the
fixins" for now (i.e. no order mode sidepane or search integration) such
that it is merely a top level chart which shows a much longer term
history and can be added to the UI via embedding the entire history
linked-splits instance into the real-time linked set's splitter.

Further impl deats:
- adjust the `GodWidget._chart_cache: dict[str, tuple]]` to store both
  linked split chart sets per symbol so that symbol switching will
  continue to work with the added history chart (set).
- rework `.load_symbol()` to operate on both the real-time (HFT) chart
  set and the history set.
- rework `LinkedSplits.set_split_sizes()` to compensate for the history
  chart and do more detailed height calcs arithmetic to make it appear
  by default as a minor sub-chart.
- adjust `LinkedSplits.add_plot()` and `ChartPlotWidget` internals to allow
  adding a plot without a sidepane and/or container `ChartnPane`
  composite widget by checking for a `sidepane == False` input.
- make `.default_view()` accept a manual y-axis offset kwarg.
- adjust search mode to provide history linked splits to
  `.set_chart_symbol()` call.
2022-09-12 20:25:15 -04:00
Tyler Goodlet f0d417ce42 Drop status msg var deleting from ns 2022-09-12 20:25:15 -04:00
Tyler Goodlet 55fc4114b4 Initial draft code working with `pg.LinearRegionItem` 2022-09-12 20:25:15 -04:00
Tyler Goodlet 97b074365b Use rt buffer for close price pnl calcs 2022-09-12 20:25:15 -04:00
Tyler Goodlet f79c3617d6 Always load FSPs with the default (fast) sampling period 2022-09-12 20:25:15 -04:00
Tyler Goodlet 861fe791eb Allocate 2 shm buffers for history and real-time
As part of supporting a "history view" chart which shows downsampled
datums alongside our 1s (or higher) sampled OHLC we need a separate
buffer to store a the slower history from broker backends. This begins
that design by allocating 2 buffers:
- `rt_shm: ShmArray` which maps to a `/dev/shm/` file with `_rt` suffix
- `hist_shm: ShmArray` which maps to a file with `_hist` suffix

Deliver both of these shms back from both `manage_history()` and load
them as `Feed.rt_shm`/`.hist_shm` on the client side.

Impl deats:
- init the rt buffer with the first datum from loaded history and
  assign all OHLC values to that row's 'close' and the vlm to 0.
- pass the hist buffer to the backfiller task
- only spawn **one** global sampler array-row increment task per
  `brokerd` and pass in the 1s delay which we presume is our lowest
  OHLC sample rate for now.
- drop `open_sample_step_stream()` and just move its body contents into
  `Feed.index_stream()`
2022-09-12 20:25:15 -04:00
Tyler Goodlet 60052ff73a Presume shortest delay input to `increment_ohlc_buffer()`
Instead of worrying about the increment period per shm subscription,
just use the value passed as input and presume the caller knows that
only one task is necessary and that the wakeup (sampling) period should
be the shortest that is needed.

It's very unlikely we don't want at least a 1s sampling (both in terms
of task switching cost and general usage) which will eventually ship as
the default "real-time" feed "timeframe". Further, this "fast" increment
sampling task can handle all lower sampling periods (eg. 1m, 5m, 1H)
based on the current implementation just the same.

Also, add a global default sample period as `_defaul_delay_s` for use in
other internal modules.
2022-09-12 20:25:15 -04:00
Tyler Goodlet 4d2708cd42 Force 1s sample step so crypto boiz can seee 2022-09-12 20:25:15 -04:00
Tyler Goodlet d1cc52dff5 Use new public lifetime-stack class attr 2022-09-12 20:24:56 -04:00
Tyler Goodlet 4fa901dbcb Port to new `tractor._runtime` mod 2022-09-12 20:24:56 -04:00
Tyler Goodlet 4a9c16d298 Fix stream type annot 2022-09-12 15:52:50 -04:00
Tyler Goodlet b9d5b904f4 Drop order entry removals on modify 2022-09-12 15:52:22 -04:00
Tyler Goodlet 0aef762d9a Bleh `kraken`, fix another ref error in fill block
Clearly, the linter didn't help us here.. but, just pass the
`brokerd` time for now in the `.broker_time` field; we can't get it from
the fill-case incremental updates in the `openOrders` sub. Add some
notes about this and how we might approach for backends with this
limitation.
2022-09-12 15:52:22 -04:00
goodboy c724117c1a
Merge pull request #398 from pikers/paper_clear_logics_fix
Oof, reverse clearing logic-routines in paper eng
2022-09-11 22:20:04 -04:00
Tyler Goodlet cc3bb85c66 Oof, reverse clearing logic-routines in paper eng 2022-09-10 16:35:31 -04:00
Tyler Goodlet 23d0b8a7ac Lul, fix name error on msg var name.. 2022-09-05 21:15:22 -04:00
Tyler Goodlet f9217570ab Add intiial `ib` backend readme 2022-08-31 17:38:24 -04:00
Tyler Goodlet 7f224f0342 Doc string typos 2022-08-31 17:22:15 -04:00
Tyler Goodlet de9f215c83 If more then one `ib` api client is available use next available for search 2022-08-31 17:22:15 -04:00
Tyler Goodlet 848e345364 POC using paper-in-docker gw for symbol search 2022-08-31 17:22:15 -04:00
Tyler Goodlet 38b190e598 Add `ib` `Crypto` contract support 2022-08-31 17:22:15 -04:00
Tyler Goodlet 7dfa4c3cde Better comment on the `size`'s purpose/units 2022-08-29 13:56:26 -04:00
Tyler Goodlet 7b653fe4f4 Store shm array size in token schema, use for loading 2022-08-29 13:46:41 -04:00
goodboy 77a687bced
Merge pull request #386 from pikers/paper_tolerance
Paper race tolerance
2022-08-29 13:28:38 -04:00
Tyler Goodlet d5c1cdd91d Configure allocator from pos msg on startup
This fixes a regression added after moving the msg parsing to later in
the order mode startup sequence. The `Allocator` needs to be configured
*to* the initial pos otherwise default settings will show in the UI..

Move the startup config logic from inside `mk_allocator()` to
`PositionTracker.update_from_pp()` and add a flag to allow setting the
`.startup_pp` from the current live one as is needed during initial
load.
2022-08-29 11:39:28 -04:00
Tyler Goodlet 46d3fe88ca Fix sub-slot-remains limiting for -ve sizes
In the short case (-ve size) we had a bug where the last sub-slots worth
of exit size would never be limited to zero once the allocator limit pos
size was hit (i.e. you could keep going more -ve on the pos,
exponentially per slot over the limit). It's a simple fix, just
a `max()` around the `l_sub_pp` var used in the next-step-size calc.

Resolves #392
2022-08-28 13:51:54 -04:00
Tyler Goodlet 5c8c5d8fbf Fix disti-mode paper pps relaying
Turns out we were putting too many brokername suffixes in the symbol
field and thus the order mode msg parser wasn't matching the current
asset to said msgs correctly and pps weren't being shown...

This repairs that plus simplifies the order mode initial pos msg loading
to just delegate into `process_trade_msg()` just as is done for
real-time msg updates.
2022-08-27 15:37:54 -04:00
Guillermo Rodriguez 0c323fdc0b
Minor style changes and warning on unexpected msg 2022-08-27 09:12:02 -03:00
Tyler Goodlet 02f53d0c13 Error on zero-size orders received by paper engine 2022-08-26 10:46:47 -04:00
Tyler Goodlet 8792c97de6 More stringent settings pane input handling
If a setting fails to apply try to log an error msg and revert to the
previous setting by not applying the UI read-update until after the new
`SettingsPane.apply_setting()` call. This prevents crashes when the user
tries to give bad inputs on editable allocator fields.
2022-08-26 10:46:47 -04:00
Tyler Goodlet 980815d075 Avoid handling account as numeric field in settings 2022-08-26 10:46:46 -04:00
Tyler Goodlet 4cedfedc21 Support clearing ticks ('last' & 'trade') fills
Previously we only simulated paper engine fills when the data feed
provide L1 queue-levels matched an execution. This patch add further
support for clear-level matches when there are real live clears on the
data feed that are faster/not synced with the L1 (aka usually during
periods of HFT).

The solution was to simply iterate the interleaved paper book entries on
both sides for said tick types and instead yield side-specific predicate
per entry.
2022-08-26 10:46:46 -04:00
Tyler Goodlet fe3d0c6fdd Handle too-fast-edits with `defaultdict[str, bidict[str, tuple]]`
Not entirely sure why this all of a sudden became a problem but it seems
price changes on order edits were sometimes resulting in key errors when
modifying paper book entries quickly. This changes the implementation to
not care about matching the last price when keying/popping old orders
and use `bidict`s to more easily pop cleared orders in the paper loop.
2022-08-26 10:46:46 -04:00
Tyler Goodlet 9200e8da57 Raw-dog-pop cancelled paper entries; old price dun matter 2022-08-26 10:46:46 -04:00
Tyler Goodlet 430d065da6 Handle paper-engine too-fast clearing race cases
When the paper engine is used it seems we can definitely hit races where
order ack msgs arrive close enough to status messages that `trio`
schedules the status processing before the acks. In such cases we want
to be tolerant and not crash but instead warn that we got an
unknown/out-of-order msg.
2022-08-26 10:46:46 -04:00
Tyler Goodlet ecd93cb05a Pass symbol with broker suffix to `.submit_limit()`; fix clearing 2022-08-26 10:46:46 -04:00
Guillermo Rodriguez 4facd161a9
Pull jsonrpc machinery out of deribit backend into piker.data._web_bs module and make it generic 2022-08-25 14:08:09 -03:00
Guillermo Rodriguez 0447612b34
Oneliner enable rpc modules on runtime open 2022-08-25 11:47:40 -03:00
Guillermo Rodriguez 00aabddfe8
Fix link 2022-08-25 09:22:15 -03:00
Guillermo Rodriguez 43fb720877
Do multiline imports 2022-08-25 09:20:41 -03:00
Guillermo Rodriguez 9626dbd7ac
Simplify rpc machinery, and switch refs to Dict and List to builtins, make brokercheck call public broker methods and get their results again 2022-08-25 09:18:52 -03:00
Guillermo Rodriguez f286c79a03
Woops enable backfill_bars in module __init__.py 2022-08-24 19:41:04 -03:00
Guillermo Rodriguez accb0eee6c
Add brokercheck guard on deribit.get_client && drop method running in brokercheck 2022-08-24 19:32:54 -03:00
Guillermo Rodriguez e97dd1cbdb
Stop using as much closures
Use a custom tractor branch that fixes a `maybe_open_context` re entrant related bug
2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 34fb497eb4
Add aiter api to NoBsWs and rework cryptofeed relay to not be OOPy 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 6669ba6590
Switch back to using async for and dont install signal handlers on cryptofeed 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez cb8099bb8c
Add README.rst and brokers.toml section in config example 2022-08-24 18:09:35 -03:00
Guillermo Rodriguez 80a1a58bfc
Refactor cryptofeed relay api and move it to client
Added submit_limit and submit_cancel
Cache syms correctly
Lowercase search results
2022-08-24 18:09:32 -03:00
Guillermo Rodriguez d60f222bb7
Add get_balances, and get_assets rpc to deribit.api.Client
Improve symbol_info search results
Expect cancellation on cryptofeeds asyncio task
Fix the no trades on instrument bug that we had on startup
2022-08-24 18:08:45 -03:00
Guillermo Rodriguez 2c2e43d8ac
Add comments and update cryptofeed fork url in requirements 2022-08-24 18:08:31 -03:00
Guillermo Rodriguez 212b3d620d
Tweaks on Client init to make api credentials optional 2022-08-24 18:08:29 -03:00
Guillermo Rodriguez 92090b01b8
Begin jsonrpc over ws refactor 2022-08-24 18:06:00 -03:00
Guillermo Rodriguez 9073fbc317
drop pydantic to match master 2022-08-23 15:18:45 -03:00
Guillermo Rodriguez f55f56a29f
Refactored deribit backend into new multi file format 2022-08-23 15:18:45 -03:00
Guillermo Rodriguez 28e025d02e
Finally get a chart going! lots of fixes to streaming machinery and custom cryptofeed fork with fixes 2022-08-23 15:18:43 -03:00
Guillermo Rodriguez e558e5837e
Introduce piker protocol in stream_messages 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez a0b415095a
Brokermod check output fixed and tweaks to deribit Client.bars function 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez 6df181c233
Add brokercheck test and got deribit to dump l1 and trades to console 2022-08-23 15:17:18 -03:00
Guillermo Rodriguez 7acc4e3208
Initial deribit mock up 2022-08-23 15:17:18 -03:00
Tyler Goodlet b9dba48306 Show correct account label on loaded order lines
Quite a simple fix, we just assign the account-specific
`PositionTracker` to the level line's `._on_level_change()` handler
instead of whatever the current `OrderMode.current_pp` is set to.

Further this adds proper pane switching support such that when a user
modifies an order line from an account which is not the currently
selected one, the settings pane is changed to reflect the
account and thus corresponding position info for that account and
instrument B)
2022-08-18 16:04:44 -04:00
Tyler Goodlet 4d2e23b5ce Expose level line marker via property 2022-08-18 16:00:41 -04:00
Tyler Goodlet 973bf87e67 Don't log aboout unknown status msg if no oid 2022-08-18 11:51:12 -04:00
Tyler Goodlet 5861839783 Fix multi-account order loading..
We were overwriting the existing loaded orders list in the per client
loop (lul) so move the def above all that.

Comment out the "try-to-cancel-inactive-orders-via-task-after-timeout"
stuff pertaining to https://github.com/erdewit/ib_insync/issues/363 for
now since we don't have a mechanism in place to cancel the re-cancel
task once the order is cancelled - plus who knows if this is even the
best way to do it..
2022-08-18 11:51:12 -04:00
Tyler Goodlet 06845e5504 `kraken`: drop `make_sub()` and inline sub defs in `subscribe()` 2022-08-18 11:51:12 -04:00
Tyler Goodlet 43bdd4d022 Pass correct instrument symbol in position msgs 2022-08-18 11:51:12 -04:00
Tyler Goodlet bafd2cb44f Only relay fills if dialog still alive 2022-08-18 11:51:12 -04:00
Tyler Goodlet be8fd32e7d Only emit ems fill msgs for 'status' events from ib
Fills seems to be dual emitted from both the `status` and `fill` events
in `ib_insync` internals and more or less contain the same data nested
inside their `Trade` type. We started handling the 'fill' case to deal
with a race issue in commissions/cost report tracking but we don't
really want to leak that same race to incremental fills vs.
order-"closed" tracking.. So go back to only emitting the fill msgs
on statuses and a "closed" on `.remaining == 0`.
2022-08-18 11:51:12 -04:00
Tyler Goodlet ee8c00684b Add actor-global "broker client" for tracking reqids 2022-08-18 11:51:12 -04:00
Tyler Goodlet 7379dc03af The `ps1` check doesn't work for `pdb`.. 2022-08-18 11:51:12 -04:00
Tyler Goodlet a602c47d47 Support loading paper engine live orders 2022-08-18 11:51:12 -04:00
Tyler Goodlet 317610e00a Store positions globally and deliver on ctx connects 2022-08-18 11:51:12 -04:00
Tyler Goodlet c4af706d51 Make order-book-vars globals to persist across ems-dialog connections 2022-08-18 11:51:12 -04:00
Tyler Goodlet 665bb183f7 Unpack existing live order params in case statement 2022-08-18 11:51:12 -04:00
Tyler Goodlet f6ba95a6c7 Split existing live-open case into its own block 2022-08-18 11:51:12 -04:00
Tyler Goodlet e2cd8c4aef Add initial `kraken` live order loading 2022-08-18 11:51:12 -04:00
Tyler Goodlet c8bff81220 Add runtime guards around feed pausing during interaction 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2aec1c5f1d Only pprint our struct when we detect a py REPL 2022-08-18 11:51:12 -04:00
Tyler Goodlet bec32956a8 Move fill case-block earlier, log broker errors 2022-08-18 11:51:12 -04:00
Tyler Goodlet 91fdc7c5c7 Load boxed `.req` values as `Order`s in mode loop 2022-08-18 11:51:12 -04:00
Tyler Goodlet b59ed74bc1 'Only send `'closed'` on Filled events, lowercase all statues' 2022-08-18 11:51:12 -04:00
Tyler Goodlet 16012f6f02 Include both symbols in error msg when a mismatch 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2b61672723 Handle 'closed' vs. 'fill` race case..
`ib` is super good not being reliable with order event sequence order
and duplication of fill info. This adds some guards to try and avoid
popping the last status status too early if we end up receiving
a `'closed'` before the expected `'fill`' event(s). Further delete the
`status_msg` ref on each iteration to avoid stale reference lookups in
the relay task/loop.
2022-08-18 11:51:12 -04:00
Tyler Goodlet 176b230a46 Use modern `Union` pipe op syntax for msg fields 2022-08-18 11:51:12 -04:00
Tyler Goodlet 7fa9dbf869 Add full EMS order-dialog (re-)load support!
This includes darks, lives and alerts with all connecting clients
being broadcast all existing order-flow dialog states. Obviously
for now darks and alerts only live as long as the `emsd` actor lifetime
(though we will store these in local state eventually) and "live" orders
have lifetimes managed by their respective backend broker.

The details of this change-set is extensive, so here we go..

Messaging schema:
- change the messaging `Status` status-key set to:
  `resp: Literal['pending', 'open', 'dark_open', 'triggered',
                'closed',  'fill', 'canceled', 'error']`

  which better reflects the semantics of order lifetimes and was
  partially inspired by the status keys `kraken` provides for their
  order-entry API. The prior key set was based on `ib`'s horrible
  semantics which sound like they're right out of the 80s..
  Also, we reflect this same set in the `BrokerdStatus` msg and likely
  we'll just get rid of the separate brokerd-dialog side type
  eventually.
- use `Literal` type annots for statuses where applicable and as they
  are supported by `msgspec`.
- add additional optional `Status` fields:
  -`req: Order` to allow each status msg to optionally ref its
    commanding order-request msg allowing at least a request-response
    style implicit tracing in all response msgs.
  -`src: str` tag string to show the source of the msg.
  -`reqid: str | int` such that the ems can relay the `brokerd`
    request id both to the client side and have one spot to look
    up prior status msgs and
- draft a (unused/commented) `Dialog` type which can be eventually used
  at all EMS endpoints to track msg-flow states

EMS engine adjustments/rework:
- use the new status key set throughout and expect `BrokerdStatus` msgs
  to use the same new schema as `Status`.
- add a `_DarkBook._active: dict[str, Status]` table which is now used for
  all per-leg-dialog associations and order flow state tracking
  allowing for the both the brokerd-relay and client-request handler loops
  to read/write the same msg-table and provides for delivering
  the overall EMS-active-orders state to newly/re-connecting clients
  with minimal processing; this table replaces what the `._ems_entries`
  table from prior.
- add `Router.client_broadcast()` to send a msg to all currently
  connected peers.
- a variety of msg handler block logic tweaks including more `case:`
  blocks to be both flatter and improve explicitness:
  - for the relay loop move all `Status` msg update and sending to
    within each block instead of a fallthrough case plus hard-to-follow
    state logic.
  - add a specific case for unhandled backend status keys and just log
    them.
  - pop alerts from `._active` immediately once triggered.
  - where possible mutate status msgs fields over instantiating new
    ones.
- insert and expect `Order` instances in the dark clearing loop and
  adjust `case:` blocks accordingly.
- tag `dark_open` and `triggered` statuses as sourced from the ems.
- drop all the `ChainMap` stuff for now; we're going to make our own
  `Dialog` type for this purpose..

Order mode rework:
- always parse the `Status` msg and use match syntax cases with object
  patterns, hackily assign the `.req` in many blocks to work around not
  yet having proper on-the-wire decoding yet.
- make `.load_unknown_dialog_from_msg()` expect a `Status` with boxed
  `.req: Order` as input.
- change `OrderDialog` -> `Dialog` in prep for a general purpose type
  of the same name.

`ib` backend order loading support:
- do "closed" status detection inside the msg-relay loop instead
  of expecting the ems to do this..
- add an attempt to cancel inactive orders by scheduling cancel
  submissions continually (no idea if this works).
- add a status map to go from the 80s keys to our new set.
- deliver `Status` msgs with an embedded `Order` for existing live order
  loading and make sure to try an get the source exchange info (instead
  of SMART).

Paper engine ported to match:
- use new status keys in `BrokerdStatus` msgs
- use `match:` syntax in request handler loop
2022-08-18 11:51:12 -04:00
Tyler Goodlet 87ed9abefa WIP playing with a `ChainMap` of messages 2022-08-18 11:51:12 -04:00
Tyler Goodlet 2548aae73d Deliver existing dialog (msgs) to every EMS client
Ideally every client that connects to the ems can know its state
(immediately) meaning relay all the order dialogs that are currently
active. This adds full (hacky WIP) support to receive those dialog
(msgs) from the `open_ems()` startup values via the `.started()` msg
from `_emsd_main()`.

Further this adds support to the order mode chart-UI to display existing
(live) orders on the chart during startup. Details include,

- add a `OrderMode.load_unknown_dialog_from_msg()` for processing and
  displaying a ``BrokerdStatus`` (for now) msg from the EMS that was not
  previously created by the current ems client and registering and
  displaying it on the chart.
- break out the ems msg processing into a new
  `order_mode.process_trade_msg()` func so that it can be called on the
  startup dialog-msg set as well as eventually used a more general low
  level auto-strat API (eg. when we get to displaying auto-strat and
  group trading automatically on an observing chart UI.
- hackyness around msg-processing for the dialogs delivery since we're
  technically delivering `BrokerdStatus` msgs when the client-side
  processing technically expects `Status` msgs.. we'll rectify this
  soon!
2022-08-18 11:51:12 -04:00
Tyler Goodlet 1cfa04927d Lol, handle failed-to-cancel statuses.. 2022-08-18 11:51:12 -04:00
Tyler Goodlet e34ea94f9f Start brokerd relay loop after opening client stream
In order to avoid missed existing order message emissions on startup we
need to be sure the client side stream is registered with the router
first. So break out the starting of the
`translate_and_relay_brokerd_events()` task until inside the client
stream block and start the task using the dark clearing loop nursery.

Also, ensure `oid` (and thus for `ib` the equivalent re-used `reqid`)
are cast to `str` before registering the dark book. Deliver the dark
book entries as part of the `_emsd_main()` context `.started()` values.
2022-08-18 11:51:12 -04:00
Tyler Goodlet 1510383738 Always cast ems `requid` values to `int` 2022-08-18 11:51:12 -04:00
Tyler Goodlet 016b669d63 Drop staged line runtime guard 2022-08-18 11:51:12 -04:00
Tyler Goodlet 682a0191ef First draft: relay open orders through ems and display on chart 2022-08-18 11:51:12 -04:00
Tyler Goodlet 9e36dbe47f Relay existing open orders from ib on startup 2022-08-18 11:51:12 -04:00
Tyler Goodlet 52febac6ae Facepalm: order-handler tasks are one-to-one with unique clients 2022-08-18 11:34:11 -04:00
Tyler Goodlet f202699c25 Fix scan loop: only stash clients that actually connect.. 2022-08-18 11:34:11 -04:00
Tyler Goodlet 0fb07670d2 Fix multi-account positioning and order tracking..
This seems to have been broken in refactoring from commit 279c899de5
which was never tested against multiple accounts/clients.

The fix is 2 part:
- position tables are now correctly loaded ahead of time and used by
  account for each connected client in processing of ledgers and
  existing positions.
- a task for each API client is started (as implemented prior) so that
  we actually get status updates for every client used for submissions.

Further we add a bit of code using `bisect.insort()` to normalize
ledgers to a datetime sorted list records (though pretty sure the `dict`
transform ruins it?) in an effort to avoid issues with ledger
transaction processing with previously minimized `Position.clears`
tables, which should (but might not?) avoid incorporating clear events
prior to the last "net-zero" positioning state.
2022-08-17 14:14:20 -04:00
Tyler Goodlet 73d2e7716f Pre-loop clients to build out pps tables, handle missing commission field 2022-08-17 10:23:01 -04:00
Tyler Goodlet 999ae5a1c6 Handle `Position.split_ratio` in state audits
This firstly changes `.audit_sizing()` => `.ensure_state()` and makes it
return `None` as well as only error when split ratio denoted (via
config) positions do not size as expected.

Further refinements,
- add an `.expired()` predicate method
- always return a size of zero from `.calc_size()` on expired assets
- load each `pps.toml` entry's clear tabe into `Transaction`s and use
  `.add_clear()` during from config init.
2022-08-17 10:06:58 -04:00
Tyler Goodlet 23ba0e5e69 Don't raise on missing position for now, just error log 2022-08-17 10:06:41 -04:00
Tyler Goodlet 941a2196b3 Get pos entry from table not `updated: dict` output 2022-08-17 10:06:37 -04:00
Tyler Goodlet 0cf4e07b84 Use `datetime` sorting on clears table appends
In order to avoid issues with reloading ledger and API trades after an
existing `pps.toml` exists we have to make sure we not only avoid
duplicate entries but also avoid re-adding entries that would have been
removed during a prior call to the `Position.minimize_clears()` filter.
The easiest way to do this is to sort on timestamps and avoid adding any
record that pre-existed the last net-zero position ledger event that
`.minimize_clears()` discarded. In order to implement this it means
parsing config file clears table's timestamps into datetime objects for
inequality checks and we add a `Position.first_clear_dt` attr for
storing this value when managing pps in object form but never store it
in the config (since it should be obviously from the sorted clear event
table).
2022-08-17 10:05:05 -04:00
Tyler Goodlet 7bec989eed First try mega-basic stock (reverse) split support with `ib` and `pps.toml` 2022-08-17 09:54:49 -04:00
Tyler Goodlet 6856ca207f Fix for TWS created position loading 2022-08-17 09:53:42 -04:00
Tyler Goodlet a83bd9c608 Drop `msgpack` from `marketstore` module 2022-08-11 14:21:36 -04:00
Tyler Goodlet 109b35f6eb Matchify paper clearing loop 2022-08-05 21:02:15 -04:00
Tyler Goodlet e28c1748fc Comment out "unknown msg" case for now 2022-08-05 21:02:15 -04:00
Tyler Goodlet 72889b4d1f Fix reference error 2022-08-05 21:02:15 -04:00
Tyler Goodlet ae001c3dd7 Matchify the dark trigger loop 2022-08-05 21:02:15 -04:00
Tyler Goodlet 2309e7ab05 Flatten the brokerd-dialog relay loop using `match:` 2022-08-05 21:02:15 -04:00
Tyler Goodlet 46c51b55f7 Flatten the client-request handler loop with `match:` 2022-08-05 21:02:15 -04:00
Tyler Goodlet 3a0987e0be Fix to-fast-edit guard case 2022-08-05 21:00:54 -04:00
Tyler Goodlet d280a592b1 Repair normalize method logic to only error on lookup failure 2022-08-05 16:14:19 -04:00
Tyler Goodlet 30bcfdcc83 Emit fills from `openOrders` block
The (partial) fills from this sub are most indicative of clears (also
says support) whereas the msgs in the `ownTrades` sub are only emitted
after the entire order request has completed - there is no size-vlm
remaining.

Further enhancements:
- this also includes proper subscription-syncing inside `subscribe()` with
  a small pre-msg-loop which waits on ack-msgs for each sub and raises any
  errors. This approach should probably be implemented for the data feed
  streams as well.
- configure the `ownTrades` sub to not bother sending historical data on
  startup.
- make the `openOrders` sub include rate limit counters.
- handle the rare case where the ems is trying to cancel an order which
  was just edited and hasn't yet had it's new `txid` registered.
2022-08-01 19:22:31 -04:00
Tyler Goodlet 1a291939c3 Drop subs ack handling from streamer 2022-08-01 16:55:04 -04:00
Tyler Goodlet 69e501764a Drop status event processing at large
Since we figured out how to pass through ems dialog ids to the
`openOrders` sub we don't really need to do much with status updates
other then error handling. This drops `process_status()` and moves the
error handling logic into a status handler sub-block; we now just
info-log status updates for troubleshooting purposes.
2022-08-01 14:08:45 -04:00
Tyler Goodlet 1cbf45b4c4 Use the ``newuserref`` field on order edits
Why we need so many fields to accomplish passing through a dialog key to
orders is beyond me but this is how they do it with edits..

Allows not having to handle `editOrderStatus` msgs to update the dialog
key table and instead just do it in the `openOrders` sub by checking the
canceled msg for a 'cancel_reason' of 'Order replaced', in which case we
just pop the txid and wait for the new order the kraken backend engine
will submit automatically, which will now have the correct 'userref'
value we passed in via the `newuserref`, and then we add that new `txid`
to our table.
2022-07-31 14:36:06 -04:00
Tyler Goodlet 227a80469e Use both `reqid` and `userref` in order requests
Turns out you can pass both thus making mapping an ems `oid` to
a brokerd-side `reqid` much more simple. This allows us to avoid keeping
as much local dialog state but with still the following caveats:

- ok `editOrder` msgs must update the reqid<->txid map
- only pop `reqids2txids` entries inside the `cancelOrderStatus` handler
2022-07-31 14:36:06 -04:00
Tyler Goodlet dc8072c6db WIP: use `userref` field over `reqid`... 2022-07-31 14:36:06 -04:00
Tyler Goodlet 808dbb12e6 Drop forgotten `pydantic` dataclass in binance backend.. 2022-07-31 14:35:25 -04:00
Tyler Goodlet 44e21b1de9 Drop field import 2022-07-30 17:34:40 -04:00
Tyler Goodlet b3058b8c78 Drop remaining `pydantic` usage, convert `OHLC` to our struct variant 2022-07-30 17:34:40 -04:00
Tyler Goodlet db564d7977 Add casting method to our struct variant 2022-07-30 17:34:40 -04:00
Tyler Goodlet e6a3e8b65a Add warning msg for `openOrders.userref` always being 0 2022-07-30 17:33:45 -04:00
Tyler Goodlet d43ba47ebe Renames to `ppu` 2022-07-30 17:33:45 -04:00
Tyler Goodlet 168c9863cb Look for transfers after ledger + api trans load
If we don't have a pos table built out already (in mem) we can't figure
out the likely dst asset (since there's no pair entry to guide us) that
we should use to search for withdrawal transactions; so move it later.

Further this ports to the new api changes in `piker.pp`` that will land
with #365.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 0fb31586fd Go back to using `Position.size` property in pp loading audits 2022-07-30 17:33:45 -04:00
Tyler Goodlet 8b609f531b Add transfers knowledge to positions validation 2022-07-30 17:33:45 -04:00
Tyler Goodlet d502274eb9 Add a `Client.get_xfers()` to retreive withdrawal transactions 2022-07-30 17:33:45 -04:00
Tyler Goodlet b1419c850d Update ledger from api immediately, cruft cleaning 2022-07-30 17:33:45 -04:00
Tyler Goodlet aa7f24b6db Drop old reversed order idea for rt-pp msg testing 2022-07-30 17:33:45 -04:00
Tyler Goodlet 319e68c855 TOSQUASH: revert to 22Hz display throttle 2022-07-30 17:33:45 -04:00
Tyler Goodlet 64f920d7e5 Accept direct fqsn matches on position msg updates 2022-07-30 17:33:45 -04:00
Tyler Goodlet 3b79743c7b Finally get real-time pp updates workin for `kraken`
This ended up driving the rework of the `piker.pp` apis to use context
manager + table style which resulted in a much easier to follow
state/update system B). Also added is a flag to do a manual simulation
of a "fill triggered rt pp msg" which requires the user to delete the
last ledgered trade entry from config files and then allowing that trade
to emit through the `openOrders` sub and update client shortly after
order mode boot; this is how the rt updates were verified to work
without doing even more live orders 😂.

Patch details:
- open both `open_trade_ledger()` and `open_pps()` inside the trade
  dialog startup and conduct a "pp state sync" logic phase where we now
  pull the account balances and incrementally load pp data (in order,
  from `pps.toml`, ledger, api) until we can generate the asset balance
  by reverse incrementing through trade history eventually erroring out
  if we can't reproduce the balance value.
- rework the `trade2pps()` to take in the `PpTable` and generate new
  ems msgs from table updates.
- return the new `dict[str, Transaction]` expected from
  `norm_trade_records()`
- only update pp config and ledger on dialog exit.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 54008a1976 Add balance and assets retreival methods, cache assets on startup
Pass config dict into client and assign to `.conf`.
2022-07-30 17:33:45 -04:00
Tyler Goodlet b96b7a8b9c Use `aclosing()` on all msg async-gens 2022-07-30 17:33:45 -04:00
Tyler Goodlet 0fca1b3e1a Also map the ws symbol set to the alt set 2022-07-30 17:33:45 -04:00
Tyler Goodlet 2386270cad Handle too-fast-edits, add `ChainMap` msg tracing
Since our ems doesn't actually do blocking style client-side submission
updates, thus resulting in the client being able to update an existing
order's state before knowing its current state, we can run into race
conditions where for some backends an order is updated using the wrong
order id. For kraken we manually implement detecting this race (lol, for
now anyway) such that when a new client side edit comes in before the
new `txid` is known, we simply expect the handler loop to cancel the
order. Further this adds cancellation on arbitrary status errors, like
rate limits.

Also this adds 2 leg (ems <-> brokerd <-> kraken) msg tracing using
a `collections.ChainMap` which is likely going to end up being the POC
for a more general data structure recommended for backends that need to
trace msg flow for translation with the ems.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 5b135fad61 Handle pre-existing open orders specifically by checking for null `oid` 2022-07-30 17:33:45 -04:00
Tyler Goodlet abb6854e74 Make all `.bsuid`s the normed symbol "altname"s 2022-07-30 17:33:45 -04:00
Tyler Goodlet 22f9b2552c Provide symbol norming via a classmethod + global table 2022-07-30 17:33:45 -04:00
Tyler Goodlet 57f2478dc7 Fixes for state updates and clears
Turns out the `openOrders` and `ownTrades` subs always return a `reqid`
value (the one brokerd sends to the kraken api in order requests) is
always set to zero, which seems to be a bug? So this includes patches to
work around that as well reliance on the `openOrders` sub to do most
`BrokerdStatus` updates since `XOrderStatus` events don't seem to have
much data in them at all (they almost look like pure ack events so maybe
they aren't affirmative of final state changes anyway..).

Other fixes:
- respond with a `BrokerdOrderAck` immediately after `requid` generation
  not after order submission to ensure the ems has a valid `requid`
  *before* kraken api events are relayed through.
- add a `reqids2txids: bidict[int, str]` which maps brokerd genned
  `requid`s to kraken-side `txid`s since (as mentioned above) the
  clearing and state endpoints don't relay back this value (it's always
  0...)
- add log messages for each sub so that (at least for now) we can see
  exact msg contents coming from kraken.
- drop `.remaining` calcs for now since we need to keep record of the
  order states manually in order to retreive the original submission
  vlm..
- fix the `openOrders` case for fills, in this case the message includes
  no `status` field and thus we must catch it in a block *after* the
  normal state handler to avoid masking.
- drop response msg generation from the cancel status case since we
  can do it again from the `openOrders` handler and sending a double
  status causes issues on the client side.
- add a shite ton of notes around all this missing `requid` stuff.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 5dc9a61ec4 Use cancel level logging for cancelled orders 2022-07-30 17:33:45 -04:00
Tyler Goodlet b0d3d9bb01 TOSQUASH: lingering `.dict()`s 2022-07-30 17:33:45 -04:00
Tyler Goodlet caecbaa231 Cancel any live orders found on connect
More or less just to avoid orders the user wasn't aware of from
persisting until we get "open order relaying" through the ems working.

Some further fixes which required a new `reqids2txids` map which keeps
track of which `kraken` "txid" is mapped to our `reqid: int`; mainly
this was needed for cancel requests which require knowing the underlying
`txid`s (since apparently kraken doesn't keep track of the "reqid"  we
pass it). Pass the ws instance into `handle_order_updates()` to enable
the cancelling orders on startup. Don't key error on unknown `reqid`
values (for eg. when receiving historical trade events on startup).
Handle cancel requests first in the ems side loop.
2022-07-30 17:33:45 -04:00
Tyler Goodlet a20a8d95d5 Use `aclosing()` around ws async gen 2022-07-30 17:33:45 -04:00
Tyler Goodlet ba93f96c71 Lol, gotta `float()` that vlm before `*` XD 2022-07-30 17:33:45 -04:00
Tyler Goodlet 804e9afdde Pass our manually mapped `reqid: int` to EMS
Since we seem to always be able to get back the `reqid`/`userref` value
we send to kraken ws endpoints, we can use this as our brokerd side
order id and avoid all race cases with getting the true `txid` value
that `kraken` assigns (and which changes when you do "edits"
:eyeroll:). This simplifies status updates by allowing our relay loop
just to pass back our generated `.reqid` verbatim and allows responding
with a `BrokerdOrderAck` immediately in the request handler task which
should guarantee there are no further race conditions with the relay
loop and mapping `txid`s from kraken.. and figuring out wtf to do when
they change, etc.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 89bcaed15e Add ledger and `pps.toml` snippets 2022-07-30 17:33:45 -04:00
Tyler Goodlet bb2f8e4304 Try out a backend readme 2022-07-30 17:33:45 -04:00
Tyler Goodlet 8ab8268edc Don't require an ems msg symbol on error statuses 2022-07-30 17:33:45 -04:00
Tyler Goodlet bbcc55b24c Update ledger *after* pps updates from new trades
Addressing same issue as in #350 where we need to compute position
updates using the *first read* from the ledger **before** we update it
to make sure `Position.lifo_update()` gets called and **not skipped**
because new trades were read as clears entries but haven't actually been
included in update calcs yet.. aka we call `Position.lifo_update()`.

Main change here is to convert `update_ledger()` into a context mngr so
that the ledger write is committed after pps updates using
`pp.update_pps_conf()`..

This is basically a hotfix to #346 as well.
2022-07-30 17:33:45 -04:00
Tyler Goodlet 9fa9c27e4d Factor status handling into a new `process_status()` helper 2022-07-30 17:33:45 -04:00
Tyler Goodlet d9b4c4a413 Factor msg loop into new func: `handle_order_updates()` 2022-07-30 17:33:45 -04:00
Tyler Goodlet 84cab1327d Drop uneeded count-sequencec verification 2022-07-30 17:33:45 -04:00
Tyler Goodlet df4cec930b Get order "editing" working fully
Turns out the EMS can support this as originally expected: you can
update a `brokerd`-side `.reqid` through a `BrokerdAck` msg and the ems
which update its cross-dialog (leg) tracking correctly! The issue was
a bug in the `editOrderStatus` msg handling and appropriate tracking
of the correct `.oid` (ems uid) on the kraken side. This unfortunately
required adding a `emsflow: dict[str, list[BrokerdOrder]]` msg flow
tracing table which means the broker daemon is tracking all the msg flow
with the ems, though I'm wondering now if this is just good practise
anyway and maybe we should offer a small primitive type from our msging
utils to aid with this? I've used such constructs in event handling
systems prior.

There's a lot more factoring that can be done after these changes as
well but the quick detailed summary is,
- rework the `handle_order_requests()` loop to use `match:` syntax and
  update the new `emsflow` table on every new request from the ems.
- fix the `editOrderStatus` case pattern to not include an error msg and
  thus actually be triggered to respond to the ems with a `BrokerdAck`
  containing the new `.reqid`, the new kraken side `txid`.
- skip any `openOrders` msgs which are detected as being kraken's
  internal order "edits" by matching on the `cancel_reason` field.
- update the `emsflow` table in all ws-stream msg handling blocks
  with responses sent to the ems.

Relates to #290
2022-07-30 17:33:45 -04:00
Tyler Goodlet ab08dc582d Make ems relay loop report on brokerd `.reqid` changes 2022-07-30 17:33:45 -04:00
Tyler Goodlet f79d9865a0 Use `match:` syntax in data feed subs processing 2022-07-30 17:33:45 -04:00
Tyler Goodlet 00378c330c First draft, working WS based order management
Move to using the websocket API for all order control ops and dropping
the sync rest api approach which resulted in a bunch of buggy races.
Further this gets us must faster (batch) order cancellation for free
and a simpler ems request handler loop. We now heavily leverage the new
py3.10 `match:` syntax for all kraken-side API msg parsing and
processing and handle both the `openOrders` and `ownTrades` subscription
streams.

We also block "order editing" (by immediate cancellation) for now since
the EMS isn't entirely yet equipped to handle brokerd side `.reqid`
changes (which is how kraken implements so called order "updates" or
"edits") for a given order-request dialog and we may want to even
consider just implementing "updates" ourselves via independent cancel
and submit requests? Definitely something to ponder. Alternatively we
can "masquerade" such updates behind the count-style `.oid` remapping we
had to implement anyway (kraken's limitation) and maybe everything will
just work?

Further details in this patch:
- create 2 tables for tracking the EMS's `.oid` (uui4) value to `int`s
  that kraken expects (for `reqid`s): `ids` and `reqmsgs` which enable
  local lookup of ems uids to piker-backend-client-side request ids and
  received order messages.
- add `openOrders` sub support which more or less directly relays to
  equivalent `BrokerdStatus` updates and calc the `.filled` and
  `.remaining` values based on cleared vlm updates.
- add handler blocks for `[add/edit/cancel]OrderStatus` events including
  error msg cases.
- don't do any order request response processing in
  `handle_order_requests()` since responses are always received via one
  (or both?) of the new ws subs: `ownTrades` and `openOrders` and thus
  such msgs are now handled in the response relay loop.

Relates to #290
Resolves #310, #296
2022-07-30 17:33:45 -04:00
Tyler Goodlet f0b3a4d5c0 Drop `pydantic.create_model()` usage for `msgspec.defstruct()` 2022-07-30 17:01:56 -04:00
Tyler Goodlet d950c78b81 Mention liquidation in error msg 2022-07-27 14:40:32 -04:00
Tyler Goodlet 7dbcbfdcd5 Write `pps.toml` shortly after broker startup 2022-07-27 14:40:32 -04:00
Tyler Goodlet 279c899de5 Port to new PpTable.dump_active()` output, move order event task to child nursery 2022-07-27 14:40:32 -04:00
Tyler Goodlet c7b84ab500 Port position calcs to new ctx mngr apis and drop multi-loop madness 2022-07-27 14:40:32 -04:00
Tyler Goodlet 9967adb371 Lol, drop unintented accound name key layer from ledger ledger 2022-07-27 14:40:32 -04:00
Tyler Goodlet 30ff793a22 Port `ib` broker machinery to new ctx mngr pp api
This drops the use of `pp.update_pps_conf()` (and friends) and instead
moves to using the context style `open_trade_ledger()` and `open_pps()`
managers for faster pp msg gen due to delayed file writing (which was
the main source update latency).

In order to make this work with potentially multiple accounts this also
uses an exit stack which loads each ledger / `pps.toml` into an account
id mapped `dict`; a POC for likely how we should implement some higher
level position manager api.
2022-07-27 12:29:53 -04:00
Tyler Goodlet 666587991a Avoid crash when no vnc server running 2022-07-27 12:29:53 -04:00
Tyler Goodlet 2766fad719 Fix #222 multi-symbol paper engine support 2022-07-27 12:18:59 -04:00
Tyler Goodlet ae71168216 Change name `be_price` -> `ppu` throughout codebase 2022-07-27 12:18:36 -04:00
Tyler Goodlet a0c238daa7 Adjust paper-engine to use `Transaction` for pps updates 2022-07-27 11:20:59 -04:00
Tyler Goodlet 7cbdc6a246 Move clears updates back into a method 2022-07-27 11:17:57 -04:00
Tyler Goodlet 2ff8be71aa Add `PpTable.write_config(), order `pps.toml` columns 2022-07-27 11:17:57 -04:00
Tyler Goodlet ddffaa952d Rework "breakeven" price as "price-per-uni": ppu
The original implementation of `.calc_be_price()` wasn't correct since
the real so called "price per unit" (ppu), is actually defined by
a recurrence relation (which is why the original state-updated
`.lifo_update()` approach worked well) and requires the previous ppu to
be weighted by the new accumulated position size when considering a new
clear event. The ppu is the price that above or below which the trader
takes a win or loss on transacting one unit of the trading asset and
thus it is the true "break even price" that determines making or losing
money per fill. This patches fixes the implementation to use trailing
windows of the accumulated size and ppu to compute the next ppu value
for any new clear event as well as handle rare cases where the
"direction" changes polarity (eg. long to short in a single order). The
new method is `Position.calc_ppu()` and further details of the relation
can be seen in the doc strings.

This patch also includes a wack-ton of clean ups and removals in an
effort to refine position management api for easier use in new backends:

- drop `updaate_pps_conf()`, `load_pps_from_toml()` and rename
  `load_trands_from_ledger()` -> `load_pps_from_ledger()`.
- extend `PpTable` to have a `.to_toml()` method which returns the
  active set of positions ready to be serialized to the `pps.toml` file
  which is collects from calling,
- `PpTable.dump_active()` which now returns double dicts of the
  open/closed pp object maps.
- make `Position.minimize_clears()` now iterate the clears table in
  chronological order (instead of reverse) and only drop fills prior
  to any zero-size state (the old reversed way can result incorrect
  history-size-retracement in cases where a position is lessened but
  not completely exited).
- drop `Position.add_clear()` and instead just manually add entries
  inside `.update_from_trans()` and also add a `accum_size` and `ppu`
  field to ever entry thus creating a position "history" sequence of
  the ppu and accum size for every position and prepares for being
  and to show "position lifetimes" in the UI.
- move fqsn getting into `Position.to_pretoml()`.
2022-07-26 12:09:59 -04:00
Tyler Goodlet 5520e9ef21 Minimize clears and audit sizing for all updates in `.update_from_trans()` 2022-07-26 12:09:59 -04:00
Tyler Goodlet 958e542f7d Drop `.lifo_upate()` add `.audit_sizing()`
Use the new `.calc_[be_price/size]()` methods when serializing to and
from the `pps.toml` format and add an audit method which will warn about
mismatched values and assign the clears table calculated values pre-write.

Drop the `.lifo_update()` method and instead allow both
`.size`/`.be_price` properties to exist (for non-ledger related uses of
`Position`) alongside the new calc methods and only get fussy about
*what* the properties are set to in the case of ledger audits.

Also changes `Position.update()` -> `.add_clear()`.
2022-07-25 12:06:52 -04:00
Tyler Goodlet 45bef0cea9 Add non-state-incremented calculation methods
Since we're going to need them anyway for desired features, add
2 new `Position` methods:
- `.calc_be_price()` which computes the breakeven cost basis price
  from the entries in the clears table.
- `.calc_size()` which just sums the clear sizes.

Add a `cost_scalar: float` control to the `.update_from_trans()` method
to allow manual adjustment of the cost weighting for the case where
a "non-symmetrical" model is wanted.

Go back to always trying to write the backing ledger files on exit, even
when there's an error (obvs without the `return` in the `finally:` block
f$#% up).
2022-07-23 19:39:47 -04:00
Tyler Goodlet 5684120c11 Wow, drop idiotic `return` inside `finally:`
Can't believe i missed this but any `return` inside a `finally` will
suppress the error from the `try:` part... XD

Thought i was losing my mind when the ledger was mutated and then
an error just after wasn't getting raised.. lul.

Never again...
2022-07-21 17:52:44 -04:00
Tyler Goodlet ddb195ed2c Add a flag to prevent writing `pps.toml` on exit 2022-07-21 17:52:44 -04:00
Tyler Goodlet 6747831677 Don't pop zero pps from table in `.dump_active()`
In order to avoid double transaction adds/updates and too-early-discard
of zero sized pps (like when trades are loaded from a backend broker but
were already added to a ledger or `pps.toml` prior) we now **don't** pop
such `Position` entries from the `.pps` table in order to keep each
position's clears table always in place. This avoids the edge case where
an entry was removed too early (due to zero size) but then duplicate
trade entries that were in that entrie's clears show up from the backend
and are entered into a new entry resulting in an incorrect size in a new
entry..We still only push non-net-zero entries to the `pps.toml`.

More fixes:
- return the updated set of `Positions` from `.lifo_update()`.
- return the full table set from `update_pps()`.
- use `PpTable.update_from_trans()` more throughout.
- always write the `pps.toml` on `open_pps()` exit.
- only return table from `load_pps_from_toml()`.
2022-07-21 17:52:44 -04:00
Tyler Goodlet 9326379b04 Add a `PpTable` type, give it the update methods
In an effort to begin allowing backends to have more granular control
over position updates, particular in the case where they need to be
reloaded from a trades ledger, this adds a new table API which can
be loaded using `open_pps()`.

- offer an `.update_trans()` method which takes in a `dict` of
  `Transactions` and updates the current table of `Positions` from it.
- add a `.dump_active()` which renders the active pp entries dict in
  a format ready for toml serialization and all closed positions since
  the last update (we might want to not drop these?)

All other module-function apis currently in use should remain working as
before for the moment.
2022-07-21 17:52:44 -04:00
Tyler Goodlet 09d9a7ea2b Expect `<brokermod>.norm_trade_records()` to return `dict` 2022-07-21 17:52:44 -04:00
Tyler Goodlet 45871d5846 Freeze transactions, add todo notes for incr update 2022-07-21 17:52:44 -04:00
goodboy bf7a49c19b
Merge pull request #358 from pikers/fix_forex
Fix forex
2022-07-21 17:52:08 -04:00
Tyler Goodlet d3130ca04c Revert to hard container kill on log error 2022-07-21 17:00:36 -04:00
Tyler Goodlet e30a3c5b54 Single chart requires view reset to size to data on startup 2022-07-21 11:39:10 -04:00
Tyler Goodlet 2393965e83 Fix bottom axis when no fsps/subplots 2022-07-21 11:39:04 -04:00
Tyler Goodlet fb39da19f4 Add option and adhoc meta-info support to `con2fqsn()` 2022-07-21 11:38:53 -04:00
Tyler Goodlet a27431c34f Unify contract->fqsn translation with new cached-helper 2022-07-21 11:38:42 -04:00
Tyler Goodlet 070b9f3dc1 Log msg tweak 2022-07-19 09:58:43 -04:00
Tyler Goodlet 0ef5da0881 Unbreak regular searches and stock lookups..
Change `.find_contract()` -> `.find_contracts()` to allow multi-search
for so called "ambiguous" contracts (like for `Future`s) such that the
method now returns a `list` of tracts and populates the contract cache
with all specific tracts retrieved. Let it take in an (unvalidated)
contract that will be fqsn-style-tokenized such that it can be called
from `.search_symbols()` (though we're not quite yet XD).

More stuff,

- add `Client.parse_patt2fqsn()` which is an fqsn to token unpacker
  built from the original logic in the old `.find_contract()`.
- handle fiat/forex pairs with the `'CASH'` sectype.
- add a flag to allow unqualified contracts to fail with a warning msg.
- populate the client's contract cache with all expiries of
  an ambiguous derivative.
- allow `.con_deats()` to warn msg instead of raise on def-not-found.
- add commented `assert 0` which was triggering a debugger deadlock in
  `tractor` which we still haven't been able to create a unit test for.
2022-07-19 09:42:01 -04:00
Tyler Goodlet 0580b204a3 A `size` field in ticks is optional 2022-07-19 09:41:37 -04:00
Tyler Goodlet 6ce699ae1f Repair display loop to work when no vlm chart is loaded 2022-07-19 09:41:37 -04:00
Tyler Goodlet 3aa72abacf Primary exchange can never be "smart" 2022-07-19 09:41:37 -04:00
Tyler Goodlet 04004525c1 Specifically denote no-vlm contracts in symbol info 2022-07-19 09:41:37 -04:00
Tyler Goodlet a7f0adf1cf Make forex rt feeds work again 2022-07-19 09:41:37 -04:00
Tyler Goodlet cef511092d Support `Forex` in the pp packer 2022-07-19 09:41:37 -04:00
Tyler Goodlet 4e5df973a9 Support `Forex` tracts in `normalize()` 2022-07-19 09:41:37 -04:00
Tyler Goodlet 6a1a62d8c0 Add (hacky) forex pair support to `Client.find_contract()` 2022-07-19 09:41:37 -04:00
Tyler Goodlet e0491cf2e7 Cache fsp ``ShmArrays`` where possible
Minimize calling `.data._shmarray.attach_shm_array()` as much as is
possible to avoid the crash from #332. This is the suggested hack from
issue #359.

Resolves https://github.com/pikers/piker/issues/359
2022-07-19 09:07:40 -04:00
Tyler Goodlet 90bc9b9730 Only 4k seconds of 1s ohlc when no tsdb 2022-07-19 09:07:27 -04:00
Tyler Goodlet fd22f45178 Oof, paper engine msg fixes after using `msgspec.Struct`.. 2022-07-11 13:04:07 -04:00
Tyler Goodlet 2a99f7a4d7 Drop remaining `BaseModel` api usage from rest of codebase 2022-07-09 12:38:17 -04:00
Tyler Goodlet b44e2d9ed9 Support `0` value `reqid`s 🤦 2022-07-09 12:10:23 -04:00
Tyler Goodlet 795d4d76f4 Add some todo-reminders for ``msgspec`` stuff 2022-07-09 12:09:50 -04:00
Tyler Goodlet c26acb1fa8 Add `Struct.copy()` which does a rountrip validate 2022-07-09 12:09:38 -04:00
Tyler Goodlet 11b6699a54 Change all clearing msgs over to `msgspec` 2022-07-09 12:09:38 -04:00
Tyler Goodlet f9bdd643cf Cast slots to `int` before range set 2022-07-09 12:09:38 -04:00
Tyler Goodlet 2baea21c7d Drop pydantic from allocator 2022-07-09 12:09:38 -04:00
Tyler Goodlet bea0111753 Add a custom `msgspec.Struct` with some humanizing 2022-07-09 12:09:38 -04:00
Tyler Goodlet c870665be0 Remove `BaseModel` use from all dataclass-like uses 2022-07-09 12:08:41 -04:00
Tyler Goodlet 4ff1090284 Use struct for shm tokens 2022-07-09 12:06:47 -04:00