Commit Graph

4188 Commits (5d86d336f2073ac07f64ad9b7c1a7eea0fac4442)

Author SHA1 Message Date
Tyler Goodlet b03564da2c binance: port to new `NoBsWs` api and drop `trio_util` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 59743b7b73 Rework `NoBsWs` to avoid agen/`trio` incompatibility
`trio`'s internals don't allow for async generator (and thus by
consequence dynamic reset of async exit stacks containing `@acm`s)
interleaving since doing so corrupts the cancel-scope stack. See details
in:
- https://github.com/python-trio/trio/issues/638
- https://trio-util.readthedocs.io/en/latest/#trio_util.trio_async_generator
- `trio._core._run.MISNESTING_ADVICE`

We originally tried to address this using
`@trio_util.trio_async_generator` in backend streaming code but for
whatever reason stopped working recently (at least for me) and it's more
or less implemented the same way as this patch but with more layers and
an extra dep. I also don't want us to have to address this problem again
if/when that lib isn't able to keep up to date with wtv `trio` is
doing..

So instead this is a complete rewrite of the conc design of our
auto-reconnect ws API to move all reset logic and msg relay into a bg
task which is respawned on reset-requiring events: user spec-ed msg recv
latency, network errors, roaming events.

Deatz:
- drop all usage of `AsyncExitStack` and no longer require client code
  to (hackily) call `NoBsWs._connect()` on msg latency conditions,
  intead this is all done behind the scenes and the user can instead
  pass in a `msg_recv_timeout: float`.
- massively simplify impl of `NoBsWs` and move all reset logic into a
  new `_reconnect_forever()` task.
- offer use of `reset_after: int` a count value that determines how many
  `msg_recv_timeout` events are allowed to occur before reconnecting the
  entire ws from scratch again.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 9d04accf2e Factor out all history mgmt-logic into a new `.data.history` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3cd853cb5d order_mode: revert switch to `MktPair` for pre-order loading 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4a0beda77e kraken: asyncify and use `get_mkt_info()` in `norm_trade_records()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d7288972b7 kraken: port to `FeedInit` and proper impl of `get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet 0d93871c88 kraken: drop `Client.cache_assets()`, simpler `.pair_info()`, drop `.mkt_info()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d0e01ff9b6 Fix `Symbol.from_fqme()` extra added symbols.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet af2f8756c5 binance: use `@async_lifo_cache` on `.get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet bcf355e2c8 Fix up `@async_lifo_cache` typing, add TODOs for move to `tractor` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 1b50bff625 Error test harness if `--pdb` passed without `-s` 2023-05-09 14:49:26 -04:00
Tyler Goodlet e317310ed3 binance: make `stream_quotes()` deliver new `list[FeedInit]` API 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4131ff1152 Rename `bs_mktid` -> `bs_fqme` and drop (some) `fqsn`s
Since we have made `MktPair.bs_mktid` mean something else now, change
all the feed setup var names to instead be more representative of the
actual value: `bs_fqme: str` and use the new `MktPair.bs_fqme` where
necessary.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 83802e932a Drop (missed) usage of `Symbol.from_fqsn()` in order mode 2023-05-09 14:49:26 -04:00
Tyler Goodlet 765b8f8e5c Support both input msg-sequence types
The legacy version was a `dict` of `dicts` vs. now we want to be handed
a `list[FeedInit]`; process both in a factored way.

Drop `FeedInit.bs_mktid` since it's already defined on `.mkt.bs_mktid`
and we don't really need it top level.
2023-05-09 14:49:26 -04:00
Tyler Goodlet b4f2f49001 ib: make `stream_quotes()` compat with new init msg bare-minimums 2023-05-09 14:49:26 -04:00
Tyler Goodlet d1cf90e2ae ib: finally convert ledger processing to use `MktPair` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 6008497b89 Use more "hierarchical" schema for fsp shm segment names 2023-05-09 14:49:26 -04:00
Tyler Goodlet adb62dc7b4 Port oustanding parts of codebase to `unpack_fqme()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4129d693be Add `.data.validate` checker for live feed layer
More or less a replacement for what @guilledk did with the initial
attempt at a "broker check" type script a while back except in this case
we're going to always run this validation routine and it now uses a new
`FeedInit` struct to ensure backends are delivering the right schema-ed
data during startup. Also allows us to stick deprecation warnings / and
or strict API compat errors all in one spot (at least for live feeds).

Factors out a bunch of `MktPair` related adapter-logic into a new
`.validate.valiate_backend()` which warns to the backend implementer via
log msgs all the problems outstanding. Ideally we do our backend module
endpoint scan-and-complain regarding missing feature support from here
as well (eg. search, broker/trade ctl, ledger processing, etc.).
2023-05-09 14:49:26 -04:00
Tyler Goodlet d48b2c5b57 `._paper_engine`: right, load `MktPair` in `fqme is not None` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 6f5a2654ab Port `.clearing` to new `unpack_fqme()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet afdbf8e10a `.accounting`: Use `_fqme()` throughout and export decimal converters 2023-05-09 14:49:26 -04:00
Tyler Goodlet d4c8ba19a2 `.accounting._mktinfo`: better fqme `MktPair` handling
It needed some work..

- Make `unpack_fqme()` always return a 4-tuple handling the venue and
  suffix parts more generally.
- add `Asset.Asset.guess_from_mkt_ep_key()` a like-it-sounds hack at
  trying to render a `.dst: Asset` for most most purposes throughout the
  stack.
- always try to preprocess the input `fqme: str` with `unpack_fqme()` in
  `MktPair.from_fqme()` and use the new `Asset` method (above) to make
  up a `.dst: Asset` pulling as much meta-info we can from the caller.
- add `MktPair.bs_fqme` to get the thing without the broker part..
- add an `'unknown'` value to the `_derivs` def.
- drop `Symbol.from_fqsn()` and `unpack_fqsn()` more generally (yes
  BREAKING).
2023-05-09 14:49:26 -04:00
Tyler Goodlet 53a41ba93d Add subsys log to new `.data._util` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 06b80ff9ed ARRG, disable `dunst` notifications for now in order mode 2023-05-09 14:49:26 -04:00
Tyler Goodlet fa88924f84 Do we need feed mod enabled? no right? 2023-05-09 14:49:26 -04:00
Tyler Goodlet 83f1922f6e `binance.get_mkt_info()`: bleh, right `@lru_cache` dun work for async.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4b7ac1d895 Port paper engine to latest `.accounting` sys fixes
- only preload necessary (one for clearing, all for ledger sync)
  `MktPair` info from the backend using `.get_mkt_info()`, build the
  `mkt_by_fqme: dict[str, MktPair]` and pass it to
  `TransactionLedger.iter_trans()`.
- use new `TransactionLedger.update_from_t()` method on clears.
- sanity check all `mkt_by_fqme` entries against `Flume.mkt` values
  when we open a data feed.
- rename `PaperBoi._syms` -> `._mkts`.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 7ee6f36e62 Actually, require `mkt_by_fqme` in `.iter_trans()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet f106472bcb Fix size quantization and closed position popping..
Turns out we actually had further pp entry bugs due to *not quantizing*
the size inside `.minimize_clears()` method calcs; fix that using
`Position.sys.mkt.quantize()` as is done in `Position.calc_size()`.

Fix `PpTable.write_config()` to drop from the TOML config any
`closed: dict[str, Position]` entries delivered by `.dump_active()`.

Add a more detailed doc string for our position type and a little todo
for the `.bep` B)
2023-05-09 14:49:26 -04:00
Tyler Goodlet bba1ee43ff Allow mkt info table input to `.iter_trans()`
Since ledger records are often provided (and thus stored) from most
backends *without* containing the info we normally need for accounting
defined by `MktPair`, this extends the ledger method to take in a table
that allows assigning the `Transaction.sys` from an fqme lookup. This
way client code (like the paper engine and new ledger mgmt tools) can
do the mkt info lookup before hand and then load both ledger
`Transactions` and positions via the `PpTable` and get correct
accounting calculations, always :fingers_crossed:

Also adds `TransactionLedger.update_from_t(t: Transaction)` to allow
updating directly from an existing tran instead of making the user cast
to a `dict` first. Includes fix to `.to_dict()` to always pop the `.sym`
again to avoid client code having to do so.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 0d2e713e9a `binance`: facepalm, swap price/size_tick methods..
Wow not sure how that happened, but we should probably use the correct
market precision info for the correct parameter..

Also, use `@lru_cache` on new `get_mkt_info()` ep, seems to work?
2023-05-09 14:49:26 -04:00
Tyler Goodlet 10a39ca42c More detailed dark-slap comments 2023-05-09 14:49:26 -04:00
Tyler Goodlet 0917b580c9 Flip `.feed` and `._sampling` over to new stuff
In `.feed` and `._sampling` move to using the new
`tractor.Context.open_stream(allow_overruns: bool)` (cough, A BREAKING
CHANGE).

Also set `Flume.mkt` during construction in `.feed.open_feed()`.
2023-05-09 14:49:26 -04:00
Tyler Goodlet a301fabd6c Change`.ui._fsp` to use `Flume.mkt` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 611d86d988 Change `Flume.symbol` -> `.mkt: MktPair`
Might as well try and flip it over to the new type; make appropriate
dict serialization changes in `.to_msg()`. Alias back to `.symbol:
Symbol` with a property.
2023-05-09 14:49:26 -04:00
Tyler Goodlet b1e162ebb4 Fix ._util import in questrade backend 2023-05-09 14:49:26 -04:00
Tyler Goodlet b810de3089 Rename fqsn -> fqme in feeds tests 2023-05-09 14:49:26 -04:00
Tyler Goodlet 48cae3c178 `ib`: rejects their own fractional size tick..
Frickin ib, they give you the `0.001` (or wtv) in the
`ContractDetails.minSize: float` but won't accept fractional sizes
through the API.. Either way, it's probably not sane to be supporting
fractional order sizes for legacy instruments by default especially
since it in theory affects a lot of the clearing outcomes by having ib
do wtv magical junk behind the scenes to make it work..
2023-05-09 14:49:26 -04:00
Tyler Goodlet 02eb966a87 Rename ems test mod 2023-05-09 14:49:26 -04:00
Tyler Goodlet 146e0993a9 More explicit test mod docstring 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2cf7daca30 Another fqsn -> fqme rename 2023-05-09 14:49:26 -04:00
Tyler Goodlet dedc51a939 Quantize order prices prior to `OrderClient.send()`
Order mode previously was just willy-nilly sending `float` prices
(particularly on order edits) which are generated from the associated
level line. This actually uses the `MktPair.price_tick: Decimal` to
ensure the value is rounded correctly before submission to the ems..

Also adjusts the order mode init to expect a table of tables of startup
position messages, with the inner table being keyed by fqme per msg.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 3b7579990b Link `tractor` debug mode to `pytest` --pdb flag 2023-05-09 14:49:26 -04:00
Tyler Goodlet 7de914d54c Fix bad-fqme test, adjust prices based on buy/sell 2023-05-09 14:49:26 -04:00
Tyler Goodlet 589232d12d Only flip size sign for seels if not already -ve 2023-05-09 14:49:26 -04:00
Tyler Goodlet 928765074f Fix zero-pp entry to toml case for new file-per-account format 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2ed9e40d5e Better EMS client-side msg formatting 2023-05-09 14:49:26 -04:00
Tyler Goodlet 30af91a82c Rewrite order ctl tests as a parametrization
More or less a complete rework which allows passing a detailed
clearing/fills input and allows for *not* rebooting the runtime / ems
between each position check.

Some further enhancements:
- use (unit) fractional sizes to simulate both the more realistic and
  more "complex position calculation" case; since this is crypto.
- add a no-fqme-found test.
- factor cross-session/offline pos storage (pps.toml) checks into
  a `load_and_check_pos()` helper which does all entry loading directly
  from a provided `BrokerdPosition` msg.
- use the new `OrderClient.send()` async api.
2023-05-09 14:49:26 -04:00