Commit Graph

4086 Commits (87d611595486c0958c23f0a310614a2956d44985)

Author SHA1 Message Date
Tyler Goodlet 5c8a45c64a Fix `MktPair.bs_fqme` to properly strip broker suffix 2023-05-17 09:45:00 -04:00
Tyler Goodlet 07b7d1d229 ib: implement `FeedInit` style quote stream setup
As per the new market info packing schema this patch almost gets it
completely compatible and useful via implementing the `get_mkt_info()`
backend module endpoint B)

There's still some questions around `MktPair.src` since all the contract
search machinery in the ib api isn't expecting a fiat currency in the
symbol key: for ex. `mnq/usd.cme.20230616.ib` has no handling for the
`[/]usd` part. For now i'm just excluding the `.src` since it requires
extra parsing on quotes-feed requests even though this is also currently
breaking forex pairs (idealpro or wtv). I think ideally we do move to
a `dst/src.<venue>.<etc..>` style but it's going to require adjustments
to all the existing crypto backends..

This also allows dropping the old `mk_init_msgs()` closure.
2023-05-16 17:29:07 -04:00
Tyler Goodlet 147e1baee9 Remove typo-ed `sum_tick_vlm` config from all crypto backends 2023-05-16 17:00:15 -04:00
Tyler Goodlet b096ee3b7a Make `FeedInit.shm_write_opts` an empty dict by default 2023-05-16 16:30:30 -04:00
Tyler Goodlet f20e2d6ee2 ib.feed: start drafting out `get_mkt_info()` endpoint 2023-05-15 15:35:57 -04:00
Tyler Goodlet 1263835034 ib.api: make `get_sym_details()` and `get_quote()` mutex methods 2023-05-15 15:35:30 -04:00
Tyler Goodlet 1e1e64f7f9 ib: fix op error when `end_dt` is `None`: the first query 2023-05-15 13:30:34 -04:00
Tyler Goodlet 98c043815a Woops, implement `Symbol.fqme` same a `Mktpair`.. 2023-05-14 20:24:19 -04:00
Tyler Goodlet ebe351e2ee kucoin: raise `DataUnavailable` if we get empty time array at some point? 2023-05-14 15:13:14 -04:00
Tyler Goodlet cfb125beef `.data.feed`: finally solve startup overruns issue
We need to allow overruns during the async multi-broker context spawning
init bc some backends might take longer then others to setup (eg.
binance vs. kucoin) and result in some context (stream) being overrun by
the time we get to the `.open_stream()` phase. Ideally, we can maybe
adjust the concurrent setup to be more of a task-per-provider style to
avoid this in the future - which would also in theory result in
more-immediate per-provider setup in terms showing ready feeds asap.

Also, does a bunch of renaming from fqsn -> fqme and drops the lower
casing of input symbols instead expecting the caller to know what the
data backend it's requesting is going to be able to handle in terms of
symbology.
2023-05-13 17:35:46 -04:00
Tyler Goodlet 1f0db3103d ib.broker: always cast `asset_type` to `str` 2023-05-13 17:27:45 -04:00
Tyler Goodlet 2e8268b53e Allow passing `allow_overruns: bool` to `Services.start_service_task()` 2023-05-13 16:51:11 -04:00
Tyler Goodlet b572cd1b77 kucoin: store fqme -> mktids table
Instead of pre-converting and mapping piker style fqmes to
`KucoinMktPair`s make `Client._pairs` keyed by the kucoin native market
ids and instead also create a `._fqmes2mktids: bidict[str, str]` for
doing lookups to the native pair from the fqme.

Also, adjust any remaining `fqsn` naming to fqme.
2023-05-13 16:45:05 -04:00
Tyler Goodlet b288d7051a ib.broker: load account name map as a `bidict` (no `tomlkit` support) 2023-05-13 16:44:28 -04:00
Tyler Goodlet c349d50f2f Allow creation of empty account files 2023-05-13 16:12:18 -04:00
Tyler Goodlet 779c0b73c9 Make `.accounting._ledger` use `tomlkit`
So that styling is preserved on write but requires that we pop `None`
values (in this case any unset `.expiry` transactions) due to `tomkit`
having no support for writing them as values?
2023-05-13 16:07:17 -04:00
Tyler Goodlet 50a4c425d3 Add `touch_if_dne: bool` to `config.load()`
So that we aren't creating blank files for legacy configs (as we do name
changes or wtv). Further change `.get_conf_path()` to validate against
new `account.` prefix and a god `conf.toml` file.
2023-05-13 16:05:23 -04:00
Tyler Goodlet df96155057 Always allow overruns in sampler context
Requires https://github.com/goodboy/tractor/pull/357.
Avoid overruns when doing concurrent live feed init over multiple
brokers.
2023-05-13 14:06:27 -04:00
Tyler Goodlet a62283bae2 Drop final use of `toml` 3rd party lib
We moved to `tomlkit` as per #496 and this lets us drop the mess that
was the inline-table encoder in `.accounting._toml` XD

Relates to #496
2023-05-12 16:15:12 -04:00
Tyler Goodlet 2865f0efe9 `piker.config`: use `tomlkit` for accounting files
We still need to get some patches landed in order to resolve:
- https://github.com/sdispater/tomlkit/issues/288
- https://github.com/sdispater/tomlkit/issues/289
- https://github.com/sdispater/tomlkit/issues/290

But, this does work for style preservation and the inline-table style we
were previously hacking into the `toml` lib in `.accounting._toml`,
which we can pretty much just drop now B)

Relates to #496 (pretty much solves it near-term i think?)
2023-05-12 16:05:45 -04:00
Tyler Goodlet 5f79434b23 Use new `.config` helpers for `accounting._pos/._ledger` file loading 2023-05-12 13:02:29 -04:00
Tyler Goodlet 5278f8b560 Add `.config.load_ledger()` for transaction record files 2023-05-12 13:01:45 -04:00
Tyler Goodlet 488a0cd119 Add `.config.load_account()`
Allows for direct loading of an "account file configuration" contents
without having to pass the explicit config dir path. In this case we are
also rewriting the `pps.<brokername>.<accnt_name>.toml` file names to
instead have a `account.` prefix, but providing this helper function
allows such changes more easily in the future - since callers won't have
to use the lower level `.load()` input signature.

Also add some todo comments about moving to `tomlkit`.
2023-05-12 12:40:09 -04:00
Tyler Goodlet 957224bdc5 ib: support remote host vnc client connections
I figure we might as well support multiple types of distributed
multi-host setups; why not allow running the API (gateway) and thus vnc
server on a diff host and allowing clients to connect and do their thing
B)

Deatz:
- make `ib._util.data_reset_hack()` take in a `vnc_host` which gets
  proxied through to the `asyncvnc` client.
- pull `ib_insync.client.Client` host value and pass-through to data
  reset machinery, presuming the vnc server is running in the same
  container (and/or the same host).
- if no vnc connection **and** no i3ipc trick can be used, just report
  to the user that they need to remove the data throttle manually.
- fix `feed.get_bars()` to handle throttle cases the same based on error
  msg matching, not error the code and add a max `_failed_resets` count
  to trigger bailing on the query loop.
2023-05-12 09:48:31 -04:00
Tyler Goodlet 7ff8aa1ba0 ib: passthrough host arg to vnc client for click hack 2023-05-11 12:32:38 -04:00
Tyler Goodlet e06f9dc5c0 kucoin: port to new `NoBsWs` api semantics
No longer need to implement connection timeout logic in the streaming
code, instead we just `async for` that bby B)

Further refining:
- better `KucoinTrade` msg parsing and handling with object cases.
- make `subscribe()` do sub request in a loop wand wair for acks.
2023-05-10 16:22:09 -04:00
Tyler Goodlet c6e5368520 paperboi: fix fqme parsing to handle `bs_fqme` cases 2023-05-09 18:34:01 -04:00
Tyler Goodlet 769b292dca Allow `brokerd` runtime switch to paper mode
Previously you couldn't have a brokerd backend which defined
`.trades_dialogue()` but which could also indicate that the paper
clearing engine should be used. This adds that support by allowing the
endpoint task to return a simple `'paper'` string, in which case the ems
will boot a paperboi.

The obvious useful case for this is if you have a broker you want to use
but do not have actual broker credentials setup (yet) with that provider
in your `brokers.toml`; demonstrated here with the adjustment to
`kraken`'s startup to no longer raise a runtime error B)
2023-05-09 18:29:28 -04:00
Tyler Goodlet 361fc4645c Drop passing `loglevel` to `stream_quotes()`, level is set when actor spawns 2023-05-09 18:28:51 -04:00
Tyler Goodlet f1f2ba2e02 kucoin: deliver `FeedInit` msgs on feed startup
To fit with the rest of the new requirements added in `.data.validate`
this adds `FeedInit` init including `MktPair` and `Asset` loading for
all spot currencies provided by `kucoin`.

Deatz:
- add a `Currency` struct and accompanying `Client.get_currencies()` for
  storing all asset infos.
- implement `.get_mkt_info()` which loads all necessary accounting and
  mkt meta-data structs including adding `.price/size_tick` fields to
  the `KucoinMktPair`.
- on client boot, async spawn requests to cache both symbols and currencies.
- pass `subscribe()` as the `fixture` arg to `open_autorecon_ws()`
  instead of opening it manually.

Other:
- tweak `Client._request` to not expect the prefixed `'/'` for the
  `endpoint: str`.
- change the `api_v` arg to just be `api: str`.
2023-05-09 18:17:50 -04:00
Tyler Goodlet 80338e1ddd kucoin: WIP moving to FeedInit API 2023-05-09 14:49:46 -04:00
Tyler Goodlet f8c8f63e87 Drop `Optional` usage from marketstore module 2023-05-09 14:49:46 -04:00
Tyler Goodlet 96532ad38c ui._display: no downsampling on history chart default view call 2023-05-09 14:49:46 -04:00
Tyler Goodlet 88f3912b2d test_ems: doc out some remaining suites 2023-05-09 14:49:46 -04:00
Tyler Goodlet cb8833d430 ib: clear error events on every received? 2023-05-09 14:49:46 -04:00
Tyler Goodlet 038b20d13a wsbs: increase msg rx timeout to 16 secs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 05fb4a4014 kraken: drop recv timeout for recon ws 2023-05-09 14:49:46 -04:00
Tyler Goodlet c415bd1ee1 If backend does not provide `bs_mktid`, use the `bs_fqme` 2023-05-09 14:49:46 -04:00
Tyler Goodlet 226c3364c3 Smh, handle `fixture==None` case.. 2023-05-09 14:49:46 -04:00
Tyler Goodlet 685688d2b2 ib: add `mbt.cme` micro-btc futes to adhoc set 2023-05-09 14:49:46 -04:00
Tyler Goodlet 7a3bce3f33 .data._web_bs: add client module name to log msgs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 363a2bbcc6 binance: use new `int` sub-id for each request 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0a8dd7b6da Try to disable `snappy` compression on variables; it breaks everything XD 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0b43e0aa8c Try having `brokerd` eps defined in `.brokers._daemon`
Since it's a bit weird having service specific implementation details
inside the general service `._daemon` mod, and since i'd mentioned
trying this re-org; let's do it B)

Requires enabling the new mod in both `pikerd` and `brokerd` and
obviously a bit more runtime-loading of the service modules in the
`brokerd` service eps to avoid import cycles.

Also moved `_setup_persistent_brokerd()` into the new mod since the
naming would place it there even though the implementation really
wouldn't (longer run) since we want to split up `.data.feed` layer
backend-invoked eps into a separate actor eventually from the "actual"
`brokerd` which will be the actor running **only** the trade control eps
(eg. trades_dialogue()` and friends).
2023-05-09 14:49:26 -04:00
Tyler Goodlet ed434e284b Disable ems init order-dialog notifications by default 2023-05-09 14:49:26 -04:00
Tyler Goodlet af068c5c51 binance: port `stream_messages()` to use `match:` and a new `L1` struct 2023-05-09 14:49:26 -04:00
Tyler Goodlet f6cd08c6fa Attempt to guard against numercial "anomalies" in `Viz.maxmin()`, add cacheing flag 2023-05-09 14:49:26 -04:00
Tyler Goodlet 34ff5ff249 kraken: port to new `NoBsWs`, passing timeout (counts) during setup 2023-05-09 14:49:26 -04:00
Tyler Goodlet b03564da2c binance: port to new `NoBsWs` api and drop `trio_util` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 59743b7b73 Rework `NoBsWs` to avoid agen/`trio` incompatibility
`trio`'s internals don't allow for async generator (and thus by
consequence dynamic reset of async exit stacks containing `@acm`s)
interleaving since doing so corrupts the cancel-scope stack. See details
in:
- https://github.com/python-trio/trio/issues/638
- https://trio-util.readthedocs.io/en/latest/#trio_util.trio_async_generator
- `trio._core._run.MISNESTING_ADVICE`

We originally tried to address this using
`@trio_util.trio_async_generator` in backend streaming code but for
whatever reason stopped working recently (at least for me) and it's more
or less implemented the same way as this patch but with more layers and
an extra dep. I also don't want us to have to address this problem again
if/when that lib isn't able to keep up to date with wtv `trio` is
doing..

So instead this is a complete rewrite of the conc design of our
auto-reconnect ws API to move all reset logic and msg relay into a bg
task which is respawned on reset-requiring events: user spec-ed msg recv
latency, network errors, roaming events.

Deatz:
- drop all usage of `AsyncExitStack` and no longer require client code
  to (hackily) call `NoBsWs._connect()` on msg latency conditions,
  intead this is all done behind the scenes and the user can instead
  pass in a `msg_recv_timeout: float`.
- massively simplify impl of `NoBsWs` and move all reset logic into a
  new `_reconnect_forever()` task.
- offer use of `reset_after: int` a count value that determines how many
  `msg_recv_timeout` events are allowed to occur before reconnecting the
  entire ws from scratch again.
2023-05-09 14:49:26 -04:00