Commit Graph

3657 Commits (114e7660aa44397e09ab06876fa429187bf82588)

Author SHA1 Message Date
Tyler Goodlet 114e7660aa Fix bad-fqme test, adjust prices based on buy/sell 2023-04-14 16:00:40 -04:00
Tyler Goodlet d29e9eeb31 Only flip size sign for seels if not already -ve 2023-04-14 16:00:40 -04:00
Tyler Goodlet 69d6e9bb4e Fix zero-pp entry to toml case for new file-per-account format 2023-04-14 16:00:40 -04:00
Tyler Goodlet 81e9a990bf Better EMS client-side msg formatting 2023-04-14 16:00:40 -04:00
Tyler Goodlet be90fb458f Rewrite order ctl tests as a parametrization
More or less a complete rework which allows passing a detailed
clearing/fills input and allows for *not* rebooting the runtime / ems
between each position check.

Some further enhancements:
- use (unit) fractional sizes to simulate both the more realistic and
  more "complex position calculation" case; since this is crypto.
- add a no-fqme-found test.
- factor cross-session/offline pos storage (pps.toml) checks into
  a `load_and_check_pos()` helper which does all entry loading directly
  from a provided `BrokerdPosition` msg.
- use the new `OrderClient.send()` async api.
2023-04-14 16:00:40 -04:00
Tyler Goodlet 0c03434a15 `binance`: add startup caching info log msg 2023-04-14 16:00:40 -04:00
Tyler Goodlet 6163067b38 Pack startup pps into a table keyed by fqmes 2023-04-14 16:00:40 -04:00
Tyler Goodlet ac6eac046a `order_mode`: broad rename book -> client 2023-04-14 16:00:40 -04:00
Tyler Goodlet 13123b71ac Drop old blessings code, general cleanups 2023-04-14 16:00:40 -04:00
Tyler Goodlet 3937059cf0 paper-eng: close context and terminate actor on exit 2023-04-14 16:00:40 -04:00
Tyler Goodlet 49bd03c2b6 `ledger` cli: dump colored summary lines to console
Tried a couple libs and ended up sticking with `rich` (since it's the
sibling lib to `typer`) but also (initially) implemented a version with
`blessings` that I ended up commenting out (and will likely remove).

Adjusted the CLI I/O a slight bit as well:
- require a fully qualified account name of the form:
  `<brokername>.<accountname>` and error on non-matching input.
- dump positions summary lines as humanized size, ppu and cost basis
  values per line.
2023-04-14 16:00:40 -04:00
Tyler Goodlet 4aad380267 paper: on no input fqme, load all mktinfos from pos table 2023-04-14 16:00:40 -04:00
Tyler Goodlet 0251225951 `pprint.pformat()` IB position mismatch log msgs 2023-04-14 16:00:40 -04:00
Tyler Goodlet a75aa5e461 Use `force_mkt` override in paper pps updates
When processing paper trades ledgers we normally won't have specific
`MktPair` info for the backend market we're simulating, as such we
need to look up this info when updating pps.toml files such that we
get precision info correct (particularly in the case of cryptos!) and
can also run paper ledger processing without running the simulated
clearing loop. In order to make it happen we lookup any `get_mkt_info()`
ep on the backend and pass the output to the `force_mkt` input of the
`PpTable.update_from_trans()` method.
2023-04-14 16:00:40 -04:00
Tyler Goodlet a5748e1a67 `binance`: add `get_mkt_info()` ep 2023-04-14 16:00:40 -04:00
Tyler Goodlet 76222f85c8 `kraken`: add module level `get_mkt_info()`
This will (likely) act as a new backend query endpoint for other `piker`
(client) code to lookup `MktPair` info from each backend. To start it
also returns the backend-broker's local `Pair` (or wtv other type) as
well.

The main motivation for this is for our paper engine which can require
the mkt info when processing paper-trades ledgers which do not contain
appropriate info to compute position metrics.
2023-04-14 16:00:40 -04:00
Tyler Goodlet ba901e6ef5 kraken: drop console setup, now done during brokerd init 2023-04-14 16:00:40 -04:00
Tyler Goodlet d4af6589af kraken: rename `Client._atable` -> `_altnames` 2023-04-14 16:00:40 -04:00
Tyler Goodlet 89160e6a03 Only log about pps once in order mode code 2023-04-14 16:00:40 -04:00
Tyler Goodlet d5dacfc7bd Another `@acm` in `._cacheables` XD 2023-04-14 16:00:40 -04:00
Tyler Goodlet ffea96f1b5 Make default order size to decimal 2023-04-14 16:00:40 -04:00
Tyler Goodlet adb11fb778 Drop `"<broker>.<account>.."` from pps.toml entries
Add special blocks to handle removing the broker account levels from
both writing and reading routines.
2023-04-14 16:00:06 -04:00
Tyler Goodlet 1408ba80aa paper: always sync pps.toml state on startup 2023-04-14 16:00:06 -04:00
Tyler Goodlet e98ef3c512 Tweak ems msg-received log msg 2023-04-14 16:00:06 -04:00
Tyler Goodlet a05beab414 Drop `loglevel` from `spawn_args` inputs to `maybe_spawn_daemon()` 2023-04-14 16:00:06 -04:00
Tyler Goodlet e2adfae54c Use `--pdb` flag to config `brokerd` debug mode 2023-04-14 16:00:06 -04:00
Tyler Goodlet 64ca507196 `kraken`: handle ws connection startup status msgs 2023-04-14 16:00:06 -04:00
Tyler Goodlet 1e6e8ddf2e Drop masked `MktPair.size_tick_digits()` cruft 2023-04-14 16:00:06 -04:00
Tyler Goodlet 44ff2fd60f paper engine: use the `fqme` for the `bs_mktid`
Instead of stripping the broker part just use the full fqme for all
`Transaction.bs_mktid: str` values since it makes indexing the `PpTable`
much easier with less key mangling..
2023-04-14 16:00:06 -04:00
Tyler Goodlet c44627ab52 Cancel the `OrderClient` sync-method relay task on exit 2023-04-14 16:00:06 -04:00
Tyler Goodlet 9c5d6d2592 Set `emsd` log level and clearly report startup pps
Change the root-service-task entrypoint to accept the level and
setup a console log as is now expected for all sub-services. Cast all
backend delivered startup `BrokerdPosition` msgs and log them to
console.
2023-04-14 15:59:04 -04:00
Tyler Goodlet 5235cb5bfe Expect `loglevel: str` in brokerd root task ep
Set the level right after spawn and once for the lifetime of the daemon.
2023-04-14 15:59:04 -04:00
Tyler Goodlet 72c98af1d1 Always pass `loglevel: str` to daemon root task eps
If you want a sub-actor to write console logs (with the right level) the
`get_console_log()` call has to be made somewhere during service task
startup. Previously this wasn't well formalized nor used (depending on
daemon) so passing `loglevel` to the service's root-task-endpoint (eg.
`_setup_persistent_brokerd()`) encourages that the daemon's logging is
configured during init according to the spawner's requesting logging
config. The previous `get_console_log()` call happening inside
`maybe_spawn_daemon()` wasn't actually doing anything in the target
daemon XD, so obviously remove that and instead passthrough loglevel
to the ctx endpoints and service manager methods.
2023-04-14 15:59:04 -04:00
Tyler Goodlet 6e9d3fb66d Expose `piker.clearing.OrderClient` 2023-04-14 15:59:04 -04:00
Tyler Goodlet e0f502507d WIP complete rework of paper engine tests
More or less we need to be able to audit not only simple "make trades
check pps.toml files" tests (which btw were great to get started!).

We also need more sophisticated and granular order mgmt and service
config scenarios,

- full e2e EMS msg flow verification
- multi-client (dis)connection scenarios and/or monitoring
- dark order clearing and offline storage
- accounting schema and position calcs detailing

As such, this is the beginning to "modularlizingz" the components needed
in the test harness to this end by breaking up the `OrderClient` control
flows vs. position checking logic so as to allow for more flexible test
scenario cases and likely `pytest` parametrizations over different
transaction sequences.
2023-04-14 15:59:04 -04:00
Tyler Goodlet 0de2101eb4 Ensure we set the test config dir in the root actor..
Not sure how this worked before but we need to also override the
`piker._config_dir: Path` in the root actor when running in `pytest`; my
guess is something in the old test suite was masking this problem after
the change to passing the dir path down through the runtime vars via
`tractor`?

Also this drops the ems related fixtures/factories since they're
specific enough to define in the clearing engine tests directly.
2023-04-14 15:59:04 -04:00
Tyler Goodlet 426f3ff38b ib: lul, fix oil (cl) venue to correctly be nymex.. 2023-04-14 15:59:04 -04:00
Tyler Goodlet c26dc35308 Adjust tests to `.clearing._client.OrderClient` type 2023-04-14 15:59:04 -04:00
Tyler Goodlet 8efb9ece61 ib: maybe incr client id; can't catch api errors..
Turns out we don't hookup our eventkit handler until after the
`load_aio_clients()` is complete, which means we can't get
`ib_insync.Client.apiError` events unless inside the asyncio side task.
So I guess try to report any such errors during API scan (note the
duplicate client id case is a special one from ibis itself) even though
we're not going to catch them trio side. The hack to work around this is
to just increment the client id value with the `connect_retries` led `i`
value even though that will break on more then 3 clients attached to an
API endpoint lul ..

Further adjustments that were to the end of trying to fix this proper:
- add `remove_handler_on_err()` cm to disconnect a handler when the trio
  side of the channel closes.
- actually connect to client api erros in our `Client.inline_errors()`
- increase connect timeout to a sec.
- change the trio-asyncio proxy response-msg loop over to `match:`
  syntax and raise on unhandled msgs from eventkit handlers.
2023-04-14 15:59:04 -04:00
Tyler Goodlet ffa3e80503 Detail `pikerd` sock bind collision in error 2023-04-14 15:59:04 -04:00
Tyler Goodlet 1cc37673a4 `ib`: drop pp mismatch err block, we already do it in audit routine 2023-04-14 15:58:54 -04:00
Tyler Goodlet b8d7c05d74 Async-ify order client methods and some renaming
We previously only offered a sync API (which was recently renamed to
`.<meth>_nowait()` style) since initially all order control was from our
`OrderMode` Qt driven UI/UX. This adds the equivalent async methods for
both testing as well as eventual auto-strat driven control B)

Also includes a bunch of renaming:
- `OrderBook` -> `OrderClient`.
- better internal renaming of the client's mem chan vars and add a ref
  `._ems_stream: tractor.MsgStream`.
- drop `get_orders()` factory, just always check for the actor-global
  instance and always set the ems stream on that client (in case old one
  was closed).
2023-04-14 15:58:54 -04:00
Tyler Goodlet 5cb63a67e1 `kraken`: write ledger and pps files on startup 2023-04-14 15:58:54 -04:00
Tyler Goodlet fb90c2049f Rework paper engine for "offline" pp loading
This will end up being super handy for testing our accounting subsystems
as well as providing unified and simple cli utils for managing ledgers
and position tracking. Allows loading the paper boi without starting
a data feed and instead just trigger ledger and pps loading without
starting the entire clearing engine.

Deatz:
- only init `PaperBoi` and start clearing loop (tasks) if a non-`None`
  fqme is provided, ow just `Context.started()` the existing pps msgs
  as loaded from the ledger.
- always update both the ledger and pp table on startup and pass
  a single instance of each obj to the `PaperBoi` for reuse (without
  opening and closing backing config files since we now have
  `.write_config()`).
- drop the global `_positions` dict, it's not needed any more if we use
  a `PaperBoi.ppt: PpTable` which persists with the engine actor's
  lifetime.
2023-04-14 15:58:54 -04:00
Tyler Goodlet f46a5337d5 Convert `Flume.MktPair.size_tick` to float for dark clearing 2023-04-14 15:58:54 -04:00
Tyler Goodlet 5c61055411 Add paper engine "offline loading" support to the ledger cli 2023-04-14 15:58:54 -04:00
Tyler Goodlet 2f31e40d3b Formalize a ledger type + API: `TransactionLedger`
Add a new `class TransactionLedger(collections.UserDict)` for managing
ledger (files) from a `dict`-like API. The main motivations being easy
conversion between `dict` <-> `Transaction` obj forms as well as dynamic
(toml) file updates via a set of methods:

- `.write_config()` to render and write state to the local toml file.
- `.iter_trans()` to allow iterator style conversion to `Transaction`
  form for each entry.
- `.to_trans()` for the dict output from the above.

Some adjustments to `Transaction` namely making `.sym/.sys` optional for
now so that paper engine entries can be loaded (offline) without
connecting to the emulated broker backend. Move to using `pathlib.Path`
throughout for bootyful toml file mgmt B)
2023-04-14 15:58:54 -04:00
Tyler Goodlet 055025c64c Always use the "most resolved" `Position.symbol: MktPair`
When loading a `Position` from a pps file we might not have the entire
`MktPair` field-set loaded (though going forward that shouldn't really
ever happen except in the case of a legacy `pps.toml`), in which case we
can check if the `.fqme: str` value loaded from the transaction is
longer and use that instead - presuming it must have more mkt meta-data
filled out.

Also includes some more `fqsn` -> `fqme` renames.
2023-04-14 15:58:54 -04:00
Tyler Goodlet 4631af60e9 `ib`: keep broker name in `Transaction.fqsn` 2023-04-14 15:58:54 -04:00
Tyler Goodlet ba31f2dfb3 `ib`: move flex utils to new submod 2023-04-14 15:58:54 -04:00