Commit Graph

4373 Commits (nix-qt6-fix)

Author SHA1 Message Date
Tyler Goodlet f2fff5a5fa ib._ledger: move trades transaction processing helpers into new module 2023-06-27 15:47:05 -04:00
Tyler Goodlet c0d575c009 Change `Position.clears` -> `._clears[list[dict]]`
When you look at usage we don't end up really needing clear entries to
be keyed by their `Transaction.tid`, instead it's much more important to
ensure the time sorted order of trade-clearing transactions such that
position properties such as the size and ppu are calculated correctly.
Thus, this instead simplified the `.clears` table to a list of clear
dict entries making a bunch of things simpler:
- object form `Position._clears` compared to the offline TOML schema
  (saved in account files) is now data-structure-symmetrical.
- `Position.add_clear()` now uses `bisect.insort()` to
  datetime-field-sort-insert into the *list* which saves having to worry
  about sorting on every sequence *read*.

Further deats:
- adjust `.accounting._ledger.iter_by_dt()` to expect an input `list`.
- change `Position.iter_clears()` to iterate only the clearing entry
  dicts without yielding a key/tid; no more tuples.
- drop `Position.to_dict()` since parent `Struct` already implements it.
2023-06-27 15:47:05 -04:00
Tyler Goodlet 66d402b80e Load ledger records into `pl.DataFrame` for `disect`-tion 2023-06-27 15:47:05 -04:00
Tyler Goodlet ea270d3396 .data.ticktools: add reverse flag, better docs
Since it may be handy to get the latest ticks first, add a `reverse:
bool` to `iterticks()` and add some cleaner logic and a proper doc
string to `frame_ticks()`.
2023-06-27 15:47:05 -04:00
Tyler Goodlet 621634b5a2 Move `frame_ticks()` and tick-type defs into `.ticktools` 2023-06-27 15:47:05 -04:00
Tyler Goodlet eacc59226f rename `.data._normalize` -> `.ticktools` 2023-06-27 15:47:05 -04:00
Tyler Goodlet 7b4472e37e data._sampling.frame_ticks(): slight rework to generalize 2023-06-27 15:47:05 -04:00
Tyler Goodlet 4a8eafabb8 Never key error on bad flow pops.. 2023-06-27 13:48:03 -04:00
Tyler Goodlet e7e7919a43 Ensure paper engine logger is `piker.clearing` instance.. 2023-06-27 13:48:03 -04:00
Tyler Goodlet cdf9105d0d Export `Flume` and `Feed` from `piker.data` 2023-06-27 13:48:03 -04:00
Tyler Goodlet 49e67d5f36 Always add a paper (account) entry to order mode init
Allows for tracking paper engine orders despite the ems not necessarily
being opened by the current order mode instance (UI) in "paper"
execution mode; useful for tracking bots/strats running against the same
EMS daemon.
2023-06-27 13:48:03 -04:00
Tyler Goodlet 85fa87fe6f Update the `_emsd_main()` doc task tree layout 2023-06-27 13:48:03 -04:00
Tyler Goodlet 249b091c2f binance: better bad account in order request error msg 2023-06-27 13:48:03 -04:00
Tyler Goodlet 2d291bd2c3 ib: expose `.broker.norm_trade_records()` from pkg 2023-06-27 13:42:08 -04:00
Tyler Goodlet cf1f4bed75 Move `.accounting` related config loaders to subpkg
Like you'd think:
- `load_ledger()` -> ._ledger
- `load_accounrt()` -> ._pos

Also fixup the old `load_pps_from_ledger()` and expose it from a new
`.accounting.cli.disect` cli cmd for trying to figure out why pp calcs
are totally mucked on stupid ib..
2023-06-27 13:42:08 -04:00
Tyler Goodlet 032976b118 view_mode: add in one missing debug_print block.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet cbe364cb62 Add explicit `piker.cli` logger name for `pikerd` 2023-06-27 13:42:08 -04:00
Tyler Goodlet efd52e8ce3 kraken: always insert ticks `list`, only append if vlm 2023-06-27 13:42:08 -04:00
Tyler Goodlet 3be1d610e0 ib: expose trade EP as `open_trade_dialog()`
Should be the final production backend to switch this over B)

Also tidy up the `update_and_audit_msgs()` validator to log vs. raise
when `validate: bool` is set; turn it off by default to avoid raises
until we figure out wtf is up with ib ledger processing or wtv..
2023-06-27 13:42:08 -04:00
Tyler Goodlet b1ef549276 Move `broker_init()` into `brokers._daemon`
We might as well start standardizing on `brokerd` init such that it can
be used more generally in client code (such as the `.accounting.cli`
stuff).

Deats of `broker_init()` impl:
- loads appropriate py pkg module,
- reads any declared `__enable_modules__: listr[str]` which will be
  passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
- loads the `.brokers._daemon._setup_persistent_brokerd

As expected the `accounting.cli` tools can now import directly from this
new location and use the common daemon fixture definition.
2023-06-27 13:42:08 -04:00
Tyler Goodlet f7f76137ca kraken: handle `.spot.kraken` new-style FQMEs
After #520 we've moved to better supporting explicit venues for cex
backends which is important where a provider offers both spot and
derivatives markets (kraken, binance, kucoin) and we need to distinguish
which is being traded given a common asset pair (eg. BTC/USDT). So, make
this work for `kraken`'s brokerd such that requests and pre-existing
live order are (un)packed to/from EMS messaging form.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 3fcf44aa52 Skip marketstore docker tests, we're gonna drop it.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet d9708e28c8 kraken: drop `OHLC.ticks` field and just inject to quote before send 2023-06-27 13:42:08 -04:00
Tyler Goodlet 65f2549d90 binance: more explicit var naming in `OHLC` parse loop 2023-06-27 13:42:08 -04:00
Tyler Goodlet a4d16ec6ab Fix ems tests: add `.spot` venue token to fqme 2023-06-27 13:42:08 -04:00
Tyler Goodlet d82173dd50 Always use fully expanded FQME throughout `.clearing`
Since crypto backends now also may expand an FQME like `xbteur.kraken`
-> `xbteur.spot.kraken` (by filling in the venue token), we need to use
this identifier when looking up per-market order dialogs or submitting
new requests. The simple fix is to simply look up that expanded from
from the `Feed.flumes` table which is always keyed by the `MktPair.fqme:
str` - the expanded form.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 5d930175e4 kraken: use new `OrderDialogs` type, handle `.spot`
Drop the older `dict[str, ChainMap]` prototype we had since the new
`OrderDialogs` built-out while adding `binance` order support is more
refined and general. Also, handle new and now expect `.spot` venue token
in FQMEs since kraken too has futes markets that we'll likely want to
support eventually.
2023-06-27 13:42:08 -04:00
Tyler Goodlet e4c1003aba Hard code futes venue(s) for now in `brokerd`.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet 676b00592d Don't allow `Client.api()` testnet queries by default, require explicit flag set 2023-06-27 13:42:08 -04:00
Tyler Goodlet 9970fa89ee Drop per-venue request methods from `Client`
Use dynamic lookups instead by mapping to the correct http session and
endpoints path using the venue routing/mode key. This let's us simplify
from 3 methods down to a single `Client._api()` which either can be
passed the `venue: str` explicitly by the caller (as is needed in the
`._cache_pairs()` case) or falls back to the client's current
`.mkt_mode: str` setting B)

Deatz:
- add couple more tables to suffice all authed-endpoint use cases:
  - `.venue2configkey: dict[str, str]` which maps the venue key to the
    `brokers.toml` subsection which should be used for auth creds and
    testnet config.
  - `.confkey2venuekeys: dict[str, list[str]]` which maps each config
    subsection key to the list of venue name keys for doing config to
    venues lookup.
- always build out testnet sessions for spot and futes venues (though if
  not set the sessions obviously won't ever be used).
- add and use new `config.ConfigurationError` custom exceptions when api
  creds are missing.
- rename `action: str` to `method: str` in `._api()` since it's the
  proper ReST term and switch what was "method" to be `endpoint: str`.
- mask out `.get_positions()` since we can get that from a user stream
  wss request (and are doing that).
- (in theory) import and use spot testnet url as necessary.
2023-06-27 13:42:08 -04:00
Tyler Goodlet fe902c017b Drop `OrderedDict` usage, not necessary in modern python 2023-06-27 13:42:08 -04:00
Tyler Goodlet 77db2fa7c8 Support loading quarterly futes existing lives
Do parsing of the `'symbol'` and check for an `_<expiry>` suffix, in
which case we re-format in capitalized FQME style, do the
`Client._pairs[str, Pair]` lookup and then send the `Pair.bs_fqme` in
the `Order.fqme: str` field.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 7f39de59d4 Factor `OrderDialogs` into `.clearing._util`
It's finally a decent little design / interface and definitely can be
used in other backends like `kraken` which rolled something lower level
but more or less the same without a wrapper class.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 5c315ba163 Support live order loading (with caveats)
As you'd expect query and sync the EMS with existing live orders
reported by the market venue by packing them in `Status` msgs and
sending over the order dialog stream before starting the handler tasks.

XXX CAVEAT:
- there appears to be no way (at least on the usdtm market/venue) to
  distinguish between different contracts such as perps vs. the
  quarterlies?
- for now we just assume that the perp is being used since
  there's no indicator otherwise in the 'symbol' field?
- we should maybe open an issue with the futures-connector project to
  see how they'd recommend solving this discrepancy?
2023-06-27 13:42:08 -04:00
Tyler Goodlet dc3ac8de01 binance: support order "modifies" B)
Only a couple tweaks to make this work according to the docs:
https://binance-docs.github.io/apidocs/futures/en/#modify-order-trade

- use a PUT request.
- provide the original user id in a `'origClientOrderId'` msg field.
- don't expect the same oid in the PUT response.

Other broker-mode related details:
- don't call `OrderDialogs.add_msg()` until after the existing check
  since we want to check against the *last* msgs contents not the new
  request.
- ensure we pass the `modify=True` flag in the edit case.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 6eee6ead79 binance: add accounts def to `brokers.toml` template 2023-06-27 13:42:08 -04:00
Tyler Goodlet 572badb4d8 Add full real-time position update support B)
There was one trick which was that it seems that binance will often send
the account/position update event over the user stream *before* the
actual clearing (aka FILLED) order update event, so make sure we put an
entry in the `dialogs: OrderDialogs` as soon as an order request comes
in such that even if the account update arrives first the
`BrokerdPosition` msg can be relayed without delay / order event order
considerations.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 4eeb232248 kraken: add more type annots in broker codez 2023-06-27 13:42:08 -04:00
Tyler Goodlet 3f555b2f5a Fix user event matching
Was using the wrong key before from our old code (not sure how that
slipped back in.. prolly doing too many git stashes XD), so fix that to
properly match against order update events with 'ORDER_TRADE_UPDATE'.

Also, don't match on the types we want to *cast to*, that's not how
match syntax works (facepalm), so we have to typecast prior to EMS msg
creation / downstream logic.

Further,
- try not bothering with binance's own internal `'orderId'` field
  tracking since they seem to support just using your own user version
  for all ctl endpoints? (thus we only need to track the EMS `.oid`s B)
- log all event update msgs for now.
- pop order dialogs on 'closed' statuses.
- wrap cancel requests in an error handler block since it seems the EMS
  is double sending requests from the client?
2023-06-27 13:42:08 -04:00
Tyler Goodlet 09007cbf08 Do native symbology lookup in order methods, send user oid in cancel requests 2023-06-27 13:42:08 -04:00
Tyler Goodlet 8a06e4d073 Wrap dialog tracking in new `OrderDialogs` type, info log all user stream msgs 2023-06-27 13:42:08 -04:00
Tyler Goodlet 45ded4f2d1 binance: order submission "user id" is not the same as their internal `int` one.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet 60b0b721c5 Split out crypto$ derivs into separate type set
For crypto derivatives (at least futes), yes they are margined, but
generally not around a single unit of vlm (like equities or commodities
futes) so don't pre-set the order mode allocator to use a #unit limit,
$limit is fine.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 249d358737 Woops, fix wss_url lookup depending on venue.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet a9c016ba10 Use `Client._pairs` cross-venue table for orders
Since the request handler task will work concurrently across venues
(spot, futes, margin) we need to be sure that we look up the correct
venue to update the order dialog and this is naturally determined by the
FQME-style symbol in the `BrokerdOrder` msg; the best way to map that
symbol-key to the correct venue/`Pair` is by using said `._pairs:
ChainMap`.

Further, handle limit order errors by catching and relaying back an
error response to the EMS. Fix the "account name" to be `binance.usdtm`
so that we can eventually and explicitly support all venues by name.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 98f6d85b65 Make order request methods be venue aware 2023-06-27 13:42:08 -04:00
Tyler Goodlet f36061a149 binance: first draft live order ctl support B)
Untested fully but has ostensibly working position and balance loading
(by delegating entirely to binance's internals for that) and an MVP ems
order request handler; still need to fill out the order status update
task implementation..

Notes:
- uses user data stream for all per account balance and position tracking.
- no support yet for `piker.accounting` position tracking.
- no support yet for full order / position real-time update via user
  stream.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 43494e4994 Add note about expecting client side to cache search domain? 2023-06-27 13:42:08 -04:00
Tyler Goodlet c6d1007e66 Load `Asset`s during echange info queries
Since we need them for accounting and since we can get them directly
from the usdtm futes `exchangeInfo` ep, just preload all asset info that
we can during initial `Pair` caching. Cache the asset infos inside a new per venue
`Client._venues2assets: dict[str, dict[str, Asset | None]]` and mostly
be pedantic with the spot asset list for now since futes seems much
smaller and doesn't include transaction precision info.

Further:
- load a testnet http session if `binance.use_testnet.futes = true`.
- add testnet support for all non-data endpoints.
- hardcode user stream methods to work for usdtm futes for the moment.
- add logging around order request calls.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 1bb7c9a2e4 Handle pending futes, optional `.filters` add testnet urls 2023-06-27 13:42:08 -04:00