Commit Graph

3924 Commits (084385e80844acf25b810b5dc272b9a86d5943d8)

Author SHA1 Message Date
Tyler Goodlet b572cd1b77 kucoin: store fqme -> mktids table
Instead of pre-converting and mapping piker style fqmes to
`KucoinMktPair`s make `Client._pairs` keyed by the kucoin native market
ids and instead also create a `._fqmes2mktids: bidict[str, str]` for
doing lookups to the native pair from the fqme.

Also, adjust any remaining `fqsn` naming to fqme.
2023-05-13 16:45:05 -04:00
Tyler Goodlet b288d7051a ib.broker: load account name map as a `bidict` (no `tomlkit` support) 2023-05-13 16:44:28 -04:00
Tyler Goodlet c349d50f2f Allow creation of empty account files 2023-05-13 16:12:18 -04:00
Tyler Goodlet 779c0b73c9 Make `.accounting._ledger` use `tomlkit`
So that styling is preserved on write but requires that we pop `None`
values (in this case any unset `.expiry` transactions) due to `tomkit`
having no support for writing them as values?
2023-05-13 16:07:17 -04:00
Tyler Goodlet 50a4c425d3 Add `touch_if_dne: bool` to `config.load()`
So that we aren't creating blank files for legacy configs (as we do name
changes or wtv). Further change `.get_conf_path()` to validate against
new `account.` prefix and a god `conf.toml` file.
2023-05-13 16:05:23 -04:00
Tyler Goodlet df96155057 Always allow overruns in sampler context
Requires https://github.com/goodboy/tractor/pull/357.
Avoid overruns when doing concurrent live feed init over multiple
brokers.
2023-05-13 14:06:27 -04:00
Tyler Goodlet a62283bae2 Drop final use of `toml` 3rd party lib
We moved to `tomlkit` as per #496 and this lets us drop the mess that
was the inline-table encoder in `.accounting._toml` XD

Relates to #496
2023-05-12 16:15:12 -04:00
Tyler Goodlet 2865f0efe9 `piker.config`: use `tomlkit` for accounting files
We still need to get some patches landed in order to resolve:
- https://github.com/sdispater/tomlkit/issues/288
- https://github.com/sdispater/tomlkit/issues/289
- https://github.com/sdispater/tomlkit/issues/290

But, this does work for style preservation and the inline-table style we
were previously hacking into the `toml` lib in `.accounting._toml`,
which we can pretty much just drop now B)

Relates to #496 (pretty much solves it near-term i think?)
2023-05-12 16:05:45 -04:00
Tyler Goodlet 5f79434b23 Use new `.config` helpers for `accounting._pos/._ledger` file loading 2023-05-12 13:02:29 -04:00
Tyler Goodlet 5278f8b560 Add `.config.load_ledger()` for transaction record files 2023-05-12 13:01:45 -04:00
Tyler Goodlet 488a0cd119 Add `.config.load_account()`
Allows for direct loading of an "account file configuration" contents
without having to pass the explicit config dir path. In this case we are
also rewriting the `pps.<brokername>.<accnt_name>.toml` file names to
instead have a `account.` prefix, but providing this helper function
allows such changes more easily in the future - since callers won't have
to use the lower level `.load()` input signature.

Also add some todo comments about moving to `tomlkit`.
2023-05-12 12:40:09 -04:00
Tyler Goodlet 957224bdc5 ib: support remote host vnc client connections
I figure we might as well support multiple types of distributed
multi-host setups; why not allow running the API (gateway) and thus vnc
server on a diff host and allowing clients to connect and do their thing
B)

Deatz:
- make `ib._util.data_reset_hack()` take in a `vnc_host` which gets
  proxied through to the `asyncvnc` client.
- pull `ib_insync.client.Client` host value and pass-through to data
  reset machinery, presuming the vnc server is running in the same
  container (and/or the same host).
- if no vnc connection **and** no i3ipc trick can be used, just report
  to the user that they need to remove the data throttle manually.
- fix `feed.get_bars()` to handle throttle cases the same based on error
  msg matching, not error the code and add a max `_failed_resets` count
  to trigger bailing on the query loop.
2023-05-12 09:48:31 -04:00
Tyler Goodlet 7ff8aa1ba0 ib: passthrough host arg to vnc client for click hack 2023-05-11 12:32:38 -04:00
Tyler Goodlet e06f9dc5c0 kucoin: port to new `NoBsWs` api semantics
No longer need to implement connection timeout logic in the streaming
code, instead we just `async for` that bby B)

Further refining:
- better `KucoinTrade` msg parsing and handling with object cases.
- make `subscribe()` do sub request in a loop wand wair for acks.
2023-05-10 16:22:09 -04:00
Tyler Goodlet c6e5368520 paperboi: fix fqme parsing to handle `bs_fqme` cases 2023-05-09 18:34:01 -04:00
Tyler Goodlet 769b292dca Allow `brokerd` runtime switch to paper mode
Previously you couldn't have a brokerd backend which defined
`.trades_dialogue()` but which could also indicate that the paper
clearing engine should be used. This adds that support by allowing the
endpoint task to return a simple `'paper'` string, in which case the ems
will boot a paperboi.

The obvious useful case for this is if you have a broker you want to use
but do not have actual broker credentials setup (yet) with that provider
in your `brokers.toml`; demonstrated here with the adjustment to
`kraken`'s startup to no longer raise a runtime error B)
2023-05-09 18:29:28 -04:00
Tyler Goodlet 361fc4645c Drop passing `loglevel` to `stream_quotes()`, level is set when actor spawns 2023-05-09 18:28:51 -04:00
Tyler Goodlet f1f2ba2e02 kucoin: deliver `FeedInit` msgs on feed startup
To fit with the rest of the new requirements added in `.data.validate`
this adds `FeedInit` init including `MktPair` and `Asset` loading for
all spot currencies provided by `kucoin`.

Deatz:
- add a `Currency` struct and accompanying `Client.get_currencies()` for
  storing all asset infos.
- implement `.get_mkt_info()` which loads all necessary accounting and
  mkt meta-data structs including adding `.price/size_tick` fields to
  the `KucoinMktPair`.
- on client boot, async spawn requests to cache both symbols and currencies.
- pass `subscribe()` as the `fixture` arg to `open_autorecon_ws()`
  instead of opening it manually.

Other:
- tweak `Client._request` to not expect the prefixed `'/'` for the
  `endpoint: str`.
- change the `api_v` arg to just be `api: str`.
2023-05-09 18:17:50 -04:00
Tyler Goodlet 80338e1ddd kucoin: WIP moving to FeedInit API 2023-05-09 14:49:46 -04:00
Tyler Goodlet f8c8f63e87 Drop `Optional` usage from marketstore module 2023-05-09 14:49:46 -04:00
Tyler Goodlet 96532ad38c ui._display: no downsampling on history chart default view call 2023-05-09 14:49:46 -04:00
Tyler Goodlet 88f3912b2d test_ems: doc out some remaining suites 2023-05-09 14:49:46 -04:00
Tyler Goodlet cb8833d430 ib: clear error events on every received? 2023-05-09 14:49:46 -04:00
Tyler Goodlet 038b20d13a wsbs: increase msg rx timeout to 16 secs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 05fb4a4014 kraken: drop recv timeout for recon ws 2023-05-09 14:49:46 -04:00
Tyler Goodlet c415bd1ee1 If backend does not provide `bs_mktid`, use the `bs_fqme` 2023-05-09 14:49:46 -04:00
Tyler Goodlet 226c3364c3 Smh, handle `fixture==None` case.. 2023-05-09 14:49:46 -04:00
Tyler Goodlet 685688d2b2 ib: add `mbt.cme` micro-btc futes to adhoc set 2023-05-09 14:49:46 -04:00
Tyler Goodlet 7a3bce3f33 .data._web_bs: add client module name to log msgs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 363a2bbcc6 binance: use new `int` sub-id for each request 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0a8dd7b6da Try to disable `snappy` compression on variables; it breaks everything XD 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0b43e0aa8c Try having `brokerd` eps defined in `.brokers._daemon`
Since it's a bit weird having service specific implementation details
inside the general service `._daemon` mod, and since i'd mentioned
trying this re-org; let's do it B)

Requires enabling the new mod in both `pikerd` and `brokerd` and
obviously a bit more runtime-loading of the service modules in the
`brokerd` service eps to avoid import cycles.

Also moved `_setup_persistent_brokerd()` into the new mod since the
naming would place it there even though the implementation really
wouldn't (longer run) since we want to split up `.data.feed` layer
backend-invoked eps into a separate actor eventually from the "actual"
`brokerd` which will be the actor running **only** the trade control eps
(eg. trades_dialogue()` and friends).
2023-05-09 14:49:26 -04:00
Tyler Goodlet ed434e284b Disable ems init order-dialog notifications by default 2023-05-09 14:49:26 -04:00
Tyler Goodlet af068c5c51 binance: port `stream_messages()` to use `match:` and a new `L1` struct 2023-05-09 14:49:26 -04:00
Tyler Goodlet f6cd08c6fa Attempt to guard against numercial "anomalies" in `Viz.maxmin()`, add cacheing flag 2023-05-09 14:49:26 -04:00
Tyler Goodlet 34ff5ff249 kraken: port to new `NoBsWs`, passing timeout (counts) during setup 2023-05-09 14:49:26 -04:00
Tyler Goodlet b03564da2c binance: port to new `NoBsWs` api and drop `trio_util` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 59743b7b73 Rework `NoBsWs` to avoid agen/`trio` incompatibility
`trio`'s internals don't allow for async generator (and thus by
consequence dynamic reset of async exit stacks containing `@acm`s)
interleaving since doing so corrupts the cancel-scope stack. See details
in:
- https://github.com/python-trio/trio/issues/638
- https://trio-util.readthedocs.io/en/latest/#trio_util.trio_async_generator
- `trio._core._run.MISNESTING_ADVICE`

We originally tried to address this using
`@trio_util.trio_async_generator` in backend streaming code but for
whatever reason stopped working recently (at least for me) and it's more
or less implemented the same way as this patch but with more layers and
an extra dep. I also don't want us to have to address this problem again
if/when that lib isn't able to keep up to date with wtv `trio` is
doing..

So instead this is a complete rewrite of the conc design of our
auto-reconnect ws API to move all reset logic and msg relay into a bg
task which is respawned on reset-requiring events: user spec-ed msg recv
latency, network errors, roaming events.

Deatz:
- drop all usage of `AsyncExitStack` and no longer require client code
  to (hackily) call `NoBsWs._connect()` on msg latency conditions,
  intead this is all done behind the scenes and the user can instead
  pass in a `msg_recv_timeout: float`.
- massively simplify impl of `NoBsWs` and move all reset logic into a
  new `_reconnect_forever()` task.
- offer use of `reset_after: int` a count value that determines how many
  `msg_recv_timeout` events are allowed to occur before reconnecting the
  entire ws from scratch again.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 9d04accf2e Factor out all history mgmt-logic into a new `.data.history` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3cd853cb5d order_mode: revert switch to `MktPair` for pre-order loading 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4a0beda77e kraken: asyncify and use `get_mkt_info()` in `norm_trade_records()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d7288972b7 kraken: port to `FeedInit` and proper impl of `get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet 0d93871c88 kraken: drop `Client.cache_assets()`, simpler `.pair_info()`, drop `.mkt_info()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d0e01ff9b6 Fix `Symbol.from_fqme()` extra added symbols.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet af2f8756c5 binance: use `@async_lifo_cache` on `.get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet bcf355e2c8 Fix up `@async_lifo_cache` typing, add TODOs for move to `tractor` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 1b50bff625 Error test harness if `--pdb` passed without `-s` 2023-05-09 14:49:26 -04:00
Tyler Goodlet e317310ed3 binance: make `stream_quotes()` deliver new `list[FeedInit]` API 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4131ff1152 Rename `bs_mktid` -> `bs_fqme` and drop (some) `fqsn`s
Since we have made `MktPair.bs_mktid` mean something else now, change
all the feed setup var names to instead be more representative of the
actual value: `bs_fqme: str` and use the new `MktPair.bs_fqme` where
necessary.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 83802e932a Drop (missed) usage of `Symbol.from_fqsn()` in order mode 2023-05-09 14:49:26 -04:00