Commit Graph

4443 Commits (c4600a039261aab633dc01c94a8284716de1b777)

Author SHA1 Message Date
Tyler Goodlet fe902c017b Drop `OrderedDict` usage, not necessary in modern python 2023-06-27 13:42:08 -04:00
Tyler Goodlet 77db2fa7c8 Support loading quarterly futes existing lives
Do parsing of the `'symbol'` and check for an `_<expiry>` suffix, in
which case we re-format in capitalized FQME style, do the
`Client._pairs[str, Pair]` lookup and then send the `Pair.bs_fqme` in
the `Order.fqme: str` field.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 7f39de59d4 Factor `OrderDialogs` into `.clearing._util`
It's finally a decent little design / interface and definitely can be
used in other backends like `kraken` which rolled something lower level
but more or less the same without a wrapper class.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 5c315ba163 Support live order loading (with caveats)
As you'd expect query and sync the EMS with existing live orders
reported by the market venue by packing them in `Status` msgs and
sending over the order dialog stream before starting the handler tasks.

XXX CAVEAT:
- there appears to be no way (at least on the usdtm market/venue) to
  distinguish between different contracts such as perps vs. the
  quarterlies?
- for now we just assume that the perp is being used since
  there's no indicator otherwise in the 'symbol' field?
- we should maybe open an issue with the futures-connector project to
  see how they'd recommend solving this discrepancy?
2023-06-27 13:42:08 -04:00
Tyler Goodlet dc3ac8de01 binance: support order "modifies" B)
Only a couple tweaks to make this work according to the docs:
https://binance-docs.github.io/apidocs/futures/en/#modify-order-trade

- use a PUT request.
- provide the original user id in a `'origClientOrderId'` msg field.
- don't expect the same oid in the PUT response.

Other broker-mode related details:
- don't call `OrderDialogs.add_msg()` until after the existing check
  since we want to check against the *last* msgs contents not the new
  request.
- ensure we pass the `modify=True` flag in the edit case.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 6eee6ead79 binance: add accounts def to `brokers.toml` template 2023-06-27 13:42:08 -04:00
Tyler Goodlet 572badb4d8 Add full real-time position update support B)
There was one trick which was that it seems that binance will often send
the account/position update event over the user stream *before* the
actual clearing (aka FILLED) order update event, so make sure we put an
entry in the `dialogs: OrderDialogs` as soon as an order request comes
in such that even if the account update arrives first the
`BrokerdPosition` msg can be relayed without delay / order event order
considerations.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 4eeb232248 kraken: add more type annots in broker codez 2023-06-27 13:42:08 -04:00
Tyler Goodlet 3f555b2f5a Fix user event matching
Was using the wrong key before from our old code (not sure how that
slipped back in.. prolly doing too many git stashes XD), so fix that to
properly match against order update events with 'ORDER_TRADE_UPDATE'.

Also, don't match on the types we want to *cast to*, that's not how
match syntax works (facepalm), so we have to typecast prior to EMS msg
creation / downstream logic.

Further,
- try not bothering with binance's own internal `'orderId'` field
  tracking since they seem to support just using your own user version
  for all ctl endpoints? (thus we only need to track the EMS `.oid`s B)
- log all event update msgs for now.
- pop order dialogs on 'closed' statuses.
- wrap cancel requests in an error handler block since it seems the EMS
  is double sending requests from the client?
2023-06-27 13:42:08 -04:00
Tyler Goodlet 09007cbf08 Do native symbology lookup in order methods, send user oid in cancel requests 2023-06-27 13:42:08 -04:00
Tyler Goodlet 8a06e4d073 Wrap dialog tracking in new `OrderDialogs` type, info log all user stream msgs 2023-06-27 13:42:08 -04:00
Tyler Goodlet 45ded4f2d1 binance: order submission "user id" is not the same as their internal `int` one.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet 60b0b721c5 Split out crypto$ derivs into separate type set
For crypto derivatives (at least futes), yes they are margined, but
generally not around a single unit of vlm (like equities or commodities
futes) so don't pre-set the order mode allocator to use a #unit limit,
$limit is fine.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 249d358737 Woops, fix wss_url lookup depending on venue.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet a9c016ba10 Use `Client._pairs` cross-venue table for orders
Since the request handler task will work concurrently across venues
(spot, futes, margin) we need to be sure that we look up the correct
venue to update the order dialog and this is naturally determined by the
FQME-style symbol in the `BrokerdOrder` msg; the best way to map that
symbol-key to the correct venue/`Pair` is by using said `._pairs:
ChainMap`.

Further, handle limit order errors by catching and relaying back an
error response to the EMS. Fix the "account name" to be `binance.usdtm`
so that we can eventually and explicitly support all venues by name.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 98f6d85b65 Make order request methods be venue aware 2023-06-27 13:42:08 -04:00
Tyler Goodlet f36061a149 binance: first draft live order ctl support B)
Untested fully but has ostensibly working position and balance loading
(by delegating entirely to binance's internals for that) and an MVP ems
order request handler; still need to fill out the order status update
task implementation..

Notes:
- uses user data stream for all per account balance and position tracking.
- no support yet for `piker.accounting` position tracking.
- no support yet for full order / position real-time update via user
  stream.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 43494e4994 Add note about expecting client side to cache search domain? 2023-06-27 13:42:08 -04:00
Tyler Goodlet c6d1007e66 Load `Asset`s during echange info queries
Since we need them for accounting and since we can get them directly
from the usdtm futes `exchangeInfo` ep, just preload all asset info that
we can during initial `Pair` caching. Cache the asset infos inside a new per venue
`Client._venues2assets: dict[str, dict[str, Asset | None]]` and mostly
be pedantic with the spot asset list for now since futes seems much
smaller and doesn't include transaction precision info.

Further:
- load a testnet http session if `binance.use_testnet.futes = true`.
- add testnet support for all non-data endpoints.
- hardcode user stream methods to work for usdtm futes for the moment.
- add logging around order request calls.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 1bb7c9a2e4 Handle pending futes, optional `.filters` add testnet urls 2023-06-27 13:42:08 -04:00
Tyler Goodlet 2ee11f65f0 binance: facepalm, always lower case venue token.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet 0c74a67ee1 Move API urls to `.venues`
Also add a lookup helper for getting addrs by venue:
`get_api_eps()` which returns the rest and wss values.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 9972bd387a kraken: use new `open_trade_dialog()` ep name B) 2023-06-27 13:42:08 -04:00
Tyler Goodlet f792ecf3af binance: use new `open_trade_dialog()` endpoint name B) 2023-06-27 13:42:08 -04:00
Tyler Goodlet 3c89295efe Rename `.binance.schemas` -> `.venues` 2023-06-27 13:42:08 -04:00
Tyler Goodlet 9ff03ba00c kraken: add `<pair>.spot.kraken` fqme interpolation
As just added for binance move to using an explicit `.<venue>.kraken`
style for spot markets which makes the current spot symbology expand to
`<PAIR>.SPOT` from the new `Pair.bs_fqme: str`. Reasons for why are
laid out in the equivalent patch for binance. Obviously this also primes
for supporting kraken's futures venue APIs as well 🏄
https://docs.futures.kraken.com/#introduction

Detalles:
- add `.spot.kraken` parsing to `get_mkt_info()` so that if the venue
  token is not passed by caller we implicitly expand it in.
- change `normalize()` to only return the `quote: dict` not the topic
  key.
- rewrite live feed msg loop to use `match:` syntax B)
2023-06-27 13:42:08 -04:00
Tyler Goodlet 8e03212e40 Always expand FQMEs with .venue and .expiry values
Since there are indeed multiple futures (perp swaps) contracts including
a set with expiry, we need a way to distinguish through search and
`FutesPair` lookup which contract we're requesting. To solve this extend
the `FutesPair` and `SpotPair` to include a `.bs_fqme` field similar to
`MktPair` and key the `Client._pairs: ChainMap`'s backing tables with
these expanded fqmes. For example the perp swap now expands to
`btcusdt.usdtm.perp` which fills in the venue as `'usdtm'` (the
usd-margined fututes market) and the expiry as `'perp'` (as before).
This allows distinguishing explicitly from, for ex., coin-margined
contracts which could instead (since we haven't added the support yet)
fqmes of the sort `btcusdt.<coin>m.perp.binance` thus making it explicit
and obvious which contract is which B)

Further we interpolate the venue token to `spot` for spot markets going
forward, which again makes cex spot markets explicit in symbology; we'll
need to add this as well to other cex backends ;)

Other misc detalles:

- change USD-M futes `MarketType` key to `'usdtm_futes'`.

- add `Pair.bs_fqme: str` for all pair subtypes with particular
  special contract handling for futes including quarterlies, perps and
  the weird "DEFI" ones..

- drop `OHLC.bar_wap` since it's no longer in the default time-series
  schema and we weren't filling it in here anyway..

- `Client._pairs: ChainMap` is now a read-only fqme-re-keyed view into
  the underlying pairs tables (which themselves are ideally keyed
  identically cross-venue) which we populate inside `Client.exch_info()`
  which itself now does concurrent pairs info fetching via a new
  `._cache_pairs()` using a `trio` task per API-venue.

- support klines history query across all venues using same
  `Client.mkt_mode_req[Client.mkt_mode]` style as we're doing for
  `.exch_info()` B)
  - use the venue specific klines history query limits where documented.

- handle new FQME venue / expiry fields inside `get_mkt_info()` ep such
  that again the correct `Client.mkt_mode` is selected based on parsing
  the desired spot vs. derivative contract.

- do venue-specific-WSS-addr lookup based on output from
  `get_mkt_info()`; use usdtm venue WSS addr if a `FutesPair` is loaded.

- set `topic: str` to the `.bs_fqme` value in live feed quotes!

- use `Pair.bs_fqme: str` values for fuzzy-search input set.
2023-06-27 13:42:08 -04:00
Tyler Goodlet 4c4787ce58 Add a "perpetual_future" mkt info type 2023-06-27 13:42:08 -04:00
Tyler Goodlet e68c55e9bd Switch `Client.mkt_mode` to 'usd_futes' if 'perp' in fqme
The beginning of supporting multi-markets through a common API client.
Change to futes market mode in the client if `.perp.` is matched in the
fqme. Currently the exchange info and live feed ws impl will swap out
for their usd-margin futures market equivalent (endpoints).
2023-06-27 13:42:08 -04:00
Tyler Goodlet dc23f1c9bd binance: fix `FutesPair` to have `.filters`
Not sure why it seemed like futures pairs didn't have this field but add
it to the parent `Pair` type as well as drop the overridden
`.price/size_tick` fields instead doing the same as in spot as well.

Also moves the `MarketType: Literal` (for the `Client.mkt_mode: str`)
and adds a pair type lookup table for exchange info loading.
2023-06-27 13:42:08 -04:00
Tyler Goodlet d173d373cb kraken: raise `SymbolNotFound` on symbology query errors 2023-06-27 13:42:08 -04:00
Tyler Goodlet 8220bd152e Extend `MktPair` doc string to refer to binance pairs 2023-06-27 13:42:08 -04:00
Tyler Goodlet aa49c38d55 Add `binance` section to `brokers.toml` 2023-06-27 13:42:08 -04:00
Tyler Goodlet dac93dd8f8 Support USD-M futes live feeds and exchange info
Add the usd-futes "Pair" type and thus ability to load all exchange
(info for) contracts settled in USDT. Luckily we don't seem to have to
modify anything in the `Client` interface (yet) other then a new
`.mkt_mode: str` which determines which endpoint set to make requests.
Obviously data received from endpoints will likely need diff handling as
per below.

Deats:
- add a bunch more API and WSS top level domains to `.api` with comments
- start a `.binance.schemas` module to house the structs for loading
  different `Pair` subtypes depending on target market: `SpotPair`,
  `FutesPair`, .. etc. and implement required `MktPair` fields on the
  new futes type for compatibility with the clearing layer.
- add `Client.mkt_mode: str` and a method lookup for endpoint parent
  paths depending on market via `.mkt_req: dict`

Also related to live feeds,
- drop `Struct` typecasting instead opting for specific fields both for
  speed and simplicity atm.
- breakout `subscribe()` into module level acm from being embedded
  closure.
- for now swap over the ws feed to be strictly the futes ep (while
  testing) and set the `.mkt_mode = 'usd_futes'`.
- hack in `Client._pairs` to only load `FutesPair`s until we figure out
  whether we want separate `Client` instances per market or not..
2023-06-27 13:42:08 -04:00
Tyler Goodlet ae1c5a0db0 binance: breakout into `feed` and `broker` mods like other backends 2023-06-27 13:42:08 -04:00
Tyler Goodlet ed0c2555fc binance: make pkgmod expose endpoints from coming submods 2023-06-27 13:42:08 -04:00
Tyler Goodlet 26a8638836 binance: convert to subpkg module 2023-06-27 13:42:08 -04:00
Tyler Goodlet e035af2f42 Don't filter out clearing ticks XD 2023-06-27 13:42:08 -04:00
Tyler Goodlet 2dc8ee2b4e Don't bother casting `AggTrade` values for now, just floatify the price/quantity 2023-06-27 13:42:08 -04:00
Tyler Goodlet 06026ec661 Add `binance` section to broker conf template 2023-06-27 13:42:08 -04:00
Guillermo Rodriguez 7c00ca0254 binance: add deposits/withdrawals API support
From @guilledk,
- Drop Decimal quantize for now
- Minor tweaks to trades_dialogue proto
2023-06-27 13:42:08 -04:00
Tyler Goodlet eaaf6e4cc1 kraken: fix `trades2pps()` type sig 2023-06-27 13:42:08 -04:00
Guillermo Rodriguez ef544ba55a Add order status tracking 2023-06-27 13:42:08 -04:00
Tyler Goodlet e85e031df7 Use new config get/set API in `brokercnf` cmd? 2023-06-27 13:42:08 -04:00
Tyler Goodlet e03da40867 Add a config get/set API (from @guilledk) ? 2023-06-27 13:42:08 -04:00
Tyler Goodlet f8af13d010 binance: add `submit_cancel()` & listen key mgmt
Patch again originally from @guilledk and adds a sesh for futures
testnet as well as a order canceller method B)
2023-06-27 13:42:08 -04:00
Tyler Goodlet 1d9c195506 kraken: tidy up paper mode activation comments 2023-06-27 13:42:08 -04:00
Tyler Goodlet d3a504864a Add draft `brokercnf` CLI cmd from @guilledk 2023-06-27 13:42:08 -04:00
Tyler Goodlet f99e8fe7eb binance: dynamically choose the rest method
Instead of having a buncha logic branches for 'get', 'post', etc. just
pass the `method: str` and do a attr lookup on the `asks` sesh.

Also, adjust the `trades_dialogue()` ep to switch to paper mode when no
client API key is detected/loaded.
2023-06-27 13:42:08 -04:00
Guillermo Rodriguez bc4ded2662 binance: start drafting live order ctl endpoints
First draft originally by @guilledk but update by myself 2 years later
xD. Will crash at runtime but at least has the machinery to setup signed
requests for auth-ed endpoints B)

Also adds a generic `NoSignature` error for when credentials are not
present in `brokers.toml` but user is trying to access auth-ed eps with
the client.
2023-06-27 13:42:08 -04:00