Commit Graph

3588 Commits (482403c8878522627bc4713ce984bcdc8c7c4132)

Author SHA1 Message Date
Tyler Goodlet af64152640 .data.history: update to new naming
-> `._source.def_iohlcv_fields`
-> `.storage.StorageClient`
2023-06-27 13:41:47 -04:00
Tyler Goodlet bf21d2e329 Rename default OHLCV `np.dtype` descriptions
Use `def_iohlcv_fields` for a name and instead of copying and inserting
the index field pop it for the non-index version. Drop creating
`np.dtype()` instances since `numpy`'s apis accept both input forms so
this is simpler on our end.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 848577488e Add public config dir getter 2023-06-27 13:41:47 -04:00
Tyler Goodlet e82538eded .data: export ohlc dtypes at top level 2023-06-27 13:41:47 -04:00
Tyler Goodlet 8ccb8b0744 kucoin: drop shm-array `numpy` dtype def, our default is the same 2023-06-27 13:41:47 -04:00
Tyler Goodlet e83de2906f Relegate old marketstore cli eps to masked module 2023-06-27 13:41:47 -04:00
Tyler Goodlet 33c464524b Lower the paper engine order-cancel latency 2023-06-27 13:41:47 -04:00
Tyler Goodlet cb774e5a5d Re-implement `piker store` CLI with `typer`
Turns out you can mix and match `click` with `typer` so this moves what
was the `.data.cli` stuff into `storage.cli` and uses the integration
api to make it all work B)

New subcmd: `piker store`
- add `piker store ls` which lists all fqme keyed time-series from backend.
- add `store delete` to remove any such key->time-series.
  - now uses a nursery for multi-timeframe concurrency B)

Mask out all the old `marketstore` specific subcmds for now (streaming,
ingest, storesh, etc..) in anticipation of moving them into
a subpkg-module and make sure to import the sub-cmd module in our top
level cli package.

Other `.storage` api tweaks:
- drop the reraising with custom error (for now).
- rename `Storage` -> `StorageClient` (or should it be API?).
2023-06-27 13:41:47 -04:00
Tyler Goodlet 1ec9b0565f Move `.data.cli` to `.storage.cli` 2023-06-27 13:41:47 -04:00
Tyler Goodlet 7ab97fb21d Add marketstore client as storage-backend module
To kick off our (tsdb) storage backends this adds our first implementing
a new `Storage(Protocol)` client interface. Going foward, the top level
`.storage` pkg-module will now expose backend agnostic APIs and helpers
whilst specific backend implementations will adhere to that middle-ware
layer.

Deats:
- add `.storage.marketstore.Storage` as the first client implementation,
  moving all needed (import) dependencies out from
  `.service.marketstore` as well as `.ohlc_key_map` and `get_client()`.
- move root `conf.toml` loading from `.data.history` into
  `.storage.__init__.open_storage_client()` which now takes in a `name:
  str` and does all the work of loading the correct backend module, its
  config, and determining if a service-instance can be contacted and
  a client loaded; in the case where this fails we raise a new
  `StorageConnectionError`.
- add a new `.storage.get_storagemod()` just like we have for brokers.
- make `open_storage_client()` also return the backend module such that
  the history-data layer can make backend specific calls as needed (eg.
  ohlc_key_map).
- fall back to a basic non-tsdb backfill when `open_storage_client()`
  raises the new connection error.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 29211b200d Start `piker.storage` subsys: cross-(ts)db middlewares
The plan is to offer multiple tsdb and other storage backends (for
a variety of use cases) and expose them similarly to how we do for
broker and data providers B)
2023-06-27 13:41:47 -04:00
Tyler Goodlet ae8358a5e7 Tidy up unused imports and doc string 2023-06-27 13:32:18 -04:00
Tyler Goodlet 00a51c0288 Use new `msgspec.structs` api for `.typecast()` 2023-06-27 13:26:52 -04:00
Tyler Goodlet 994564f923 Just warn-print when annots are str values? 2023-06-27 13:26:52 -04:00
Tyler Goodlet 12172cc5cd Make `.data.types.Struct.typecast()` work via type lookup from `builtins` 2023-06-27 13:26:52 -04:00
Tyler Goodlet 4b77de5e2d Fix reading font size from user config
Was borked on linux if you didn't provide the setting in `conf.toml` due
to some logic errors. Fix that by rejigging `DpiAwareFont` internal
variables:

- add new `._font_size_calc_key: str` which was the old `._font_size`
  and is only used when no explicit font size is set by the user in the
  `conf.toml` config:
  - this is the "key" that is used to lookup a calculation function
    which attempts to compute a best fit font size given the measured
    system displays DPI settings and dimensions.
- make the `._font_size: int` the **actual** font size integer that is
  cached and passed to `Qt` to set the size.
  - this is overridden by user config now if defined.
- change the input kwarg `font_size: str` to the constructor to better
  change the input kwarg `font_size: str` to the constructor to better
  named private `_font_size_key: str` which gets set to the new
  `._font_size_calc_key`.

Also, adjust all client code which instantiates `DpiAwareFont` to use
the new `_font_size_key` kwarg input so nothing breaks XD
2023-06-19 15:13:01 -04:00
Ebisu d660376206 get font style from main config 2023-06-19 00:10:37 +02:00
Tyler Goodlet c27da99e12 Fix price label precision as `MktPair.price_tick_digits`
Was only really borked for higher-precision but lower priced assets
(like TLOS or peeneez) which have a `MktPair.price_tick_digits >= 2`.

The issue was using the wrong attr, the `size_tick_digits`..
2023-05-31 10:36:20 -04:00
Tyler Goodlet f6549fcb62 Always allocate a new `OrderClient` per `open_ems()` call 2023-05-28 14:05:03 -04:00
Tyler Goodlet 41aa87f847 Fix `_digits` attr names in order mode.. 2023-05-28 13:13:43 -04:00
Tyler Goodlet d6331ce9e1 Add nonlocal annots to satisfy ruff 2023-05-28 12:41:14 -04:00
Tyler Goodlet 5e2107ff15 Adjust `config.load()` to handle CI git checkout dir, seems they changed it!? 2023-05-26 16:50:15 -04:00
Tyler Goodlet 3b5bd8f43e Ensure quote last price is a `float` 2023-05-26 14:42:35 -04:00
Tyler Goodlet 3d8c1a7b3c ib: don't log-emit ib pp msg when none exists.. 2023-05-26 14:05:32 -04:00
Tyler Goodlet 4a8e8a32f9 Fix account config loading logic discovered in new test XD 2023-05-25 17:56:14 -04:00
Tyler Goodlet 9c80969fd5 .data.validate: add missing endpoint warnings 2023-05-25 16:01:21 -04:00
Tyler Goodlet f0a346dcc3 Some linting fixes after trying out `ruff` 2023-05-24 17:25:23 -04:00
Tyler Goodlet 7381c361cd Strictly drop `LinkedSplits.symbol` B) 2023-05-24 15:42:14 -04:00
Tyler Goodlet 1b577eebf6 Change over the UI layer to use `MktPair`
Including changing to `LinkedSplits.mkt: MktPair` and adding an explicit
setter method for setting it and being sure that nothing breaks
in the display system init!

For this commit we leave in warning access to `LinkedSplits.symbol` but
will remove in following commit.
2023-05-24 15:30:17 -04:00
Tyler Goodlet 39af215d61 kraken: use new `Position.mkt` attr 2023-05-24 15:29:42 -04:00
Tyler Goodlet 35f0520cb0 Drop `Symbol` / `.symbol` support from `.accounting`
Only stuff left was the allocator stuff. Drop the top level subpkg
exports and finally kill off the awkwardly named
`Symbol.lot_size_digits` properties XD

Expose a bunch more util funcs at subpkg top level, do some typing in
allocator method internals.
2023-05-24 15:26:51 -04:00
Tyler Goodlet bd8e4760d5 Port everything strictly to `Position.mkt` and `Flume.mkt` 2023-05-24 12:16:28 -04:00
Tyler Goodlet 9a063ccb11 ib: Solve lingering bugs for non-vlm contracts
Contract matching in live setup was borked; switch to
`MktPair.dst.atype` matching, don't override the `cmdty` "venue" (a
weird special case) in `get_mkt_info()` otherwise lookup will fail..
2023-05-24 09:11:24 -04:00
Tyler Goodlet e8787d89c6 ib: unset vlm via new `FeedInit.shm_write_opts` field 2023-05-24 08:28:16 -04:00
Tyler Goodlet 8e97814c1f Add "no vlm" indication to `FeedInit`
Stash it for now in the (now mutable by default) `.shm_write_opts` and
have the new `Flume._has_vlm: bool` (only set to false internally by
feed layer) which can be read via new public `.has_vlm()` predicate.
Move out the old `.ui/_fsp` helper logic to this flume method.
2023-05-24 08:25:14 -04:00
Tyler Goodlet b44b0915ca ib: i guess only discard `MktPair.src: Asset` on non-forex XD 2023-05-23 19:11:40 -04:00
Tyler Goodlet ff74d47fd5 kucoin: fix fqme or search result key lookups 2023-05-23 16:46:21 -04:00
Tyler Goodlet 6ad8c603d5 More detailed `Position.events` todo 2023-05-23 16:45:58 -04:00
Tyler Goodlet d094625bd6 Activate docker daemons via flags using exit stack 2023-05-23 14:16:08 -04:00
Tyler Goodlet e7a172b656 Reimplement marketstore and elasticsearch daemons
Using the new `._ahab.start_ahab_service()` mngr of course, and now
support user config overrides (such that our defaults can be modified by
a keen user, say using a config file, or for testing). This is where the
functionality moved out of the `pikerd` init has been moved - instead of
being triggered by bool flag inputs to that factory.

For marketstore actually support overriding the entire yaml config via
runtime `_yaml_config_str: str` formatting with any passed user dict,
primarily focussing on supporting override of the sockaddrs for testing.
2023-05-23 14:16:02 -04:00
Tyler Goodlet bd919f9d66 _ahab: use `Services` api to spawn docker tasks
Allows for using the `Services.cancel_service()` api for explicit
cancellation in tests and eventually for remote teardown. Change
`.start_ahab()` to an `@acm` `start_ahab_service()` and just yield back
the same values we were returning prior. Also fix the logging (level) to
actually reflect what's passed in - we weren't using the correct name
/ instance from the `.sevice` subpkg..
2023-05-23 14:16:02 -04:00
Tyler Goodlet 611d1ee3fc Drop db flags from pikerd startup 2023-05-23 14:16:02 -04:00
Tyler Goodlet d3bafb0063 Always prefer a config template if found 2023-05-23 14:16:02 -04:00
Tyler Goodlet 7f246697b4 Remove remaining `fqsn` usage from code base minus backward compats 2023-05-23 14:16:02 -04:00
Tyler Goodlet dd10acbbf9 Replace `Transaction.fqsn` -> `.fqme`
Change over all client (broker) code which constructs transactions
and finally wipe required `.fqsn` usage from `.accounting` B)
2023-05-23 14:15:57 -04:00
Tyler Goodlet 31a00eca94 Rename fqsn -> fqme in ui mods 2023-05-22 12:13:00 -04:00
Tyler Goodlet 588770d034 ib: rename lingering fqsn -> fqme 2023-05-22 12:00:13 -04:00
Tyler Goodlet 2f2d612b5f Add todo to switch to `dst/src` delim 2023-05-22 11:57:37 -04:00
Tyler Goodlet 660a94d610 Don't expect `conf.toml`'s network section
For testing this is particularly true until we offer a template
with whatever (likely localhost) settings planned to ship.
2023-05-22 11:54:36 -04:00
Tyler Goodlet e4e4cacef3 .data.feed: Less stringency with fqme matching
`Flume.mkt.fqme` might not be exactly the same as the local
version now since we've had to add some hacks to certain backends
(cough ib) to handle `MktPair.src` not being set as an `Asset` (yet).
2023-05-22 11:52:36 -04:00
Tyler Goodlet 60a6f3269c ib: use flex report datetime sort
Since `open_trade_ledger()` now requires a sort we pass in a combo of
the std `pendulum.parse()` for API records and a custom flex parser for
flex entries pulled offline.

Add special handling for `MktPair.src` such that when it's a fiat (like
it should always be for most legacy assets) we try to get the fqme
without that `.src` token (i.e. not mnqusd) to avoid breaking
roundtripping of live feed requests (due to new symbology) as well as
the current tsdb table key set..

Do a wholesale renaming of fqsn -> fqme in most of the rest of the
backend modules.
2023-05-22 09:41:44 -04:00
Tyler Goodlet 53003618cb Add longer timeout on brokerd ctx cancel; seems to work? 2023-05-22 00:16:58 -04:00
Tyler Goodlet c6da09f3c6 Add fast(er), time-sorted ledger records
Turns out that reading **and** writing with `tomlkit` is just wayya slow
for large documents like ledger files so move to using the `tomli`
sibling pkg `tomli-w` which seems to much improve on the latency, though
obviously longer run we're likely going to want:
- a better algorithm for only back loading records using as little
  history as possible
- a different serialization format for production maybe something
  like apache parquet?

The only issue with using a non-style-preserving writer is that we don't
necessarily get TOML conf ordering for free (without first ordering it
ourselves), and thus this patch also adds much more general date-time
sorting machinery which is now **required** when using
`open_trades_ledger()` via a `tx_sort: Callable`. By default we now
provide `.accounting._ledger.iter_by_dt()` (exposed in the subpkg mod)
which conducts dynamic "datetime key detection" based parsing of records
based on a `parsers: dict[str, Callabe]` input table. The default should
handle most use cases including all currently supported live backends
(kraken, ib) as well as our paper engine ledger-records format.

Granulars:
- adjust `Position.iter_clears()` to use new `iter_by_dt(key=lambda ..)`
  signature.
- add `tomli-w` to setup and our `tomlkit` fork to requirements file.
- move `.write_config()` to bottom of class defn.
- fix closed pos popping to not error if pp was already popped..
2023-05-18 18:27:54 -04:00
Tyler Goodlet 89d24cfe33 Oof, fix closed position popping by fqme.. 2023-05-18 12:52:34 -04:00
Tyler Goodlet 8d7a9fa19e Make `MktPair.pair()` a meth, allow passing in a delim character 2023-05-18 12:01:30 -04:00
Tyler Goodlet a1a10676cd Go back to `tomllib` for ledger loading, it's wayy faster 2023-05-18 11:27:31 -04:00
Tyler Goodlet 97b2b25256 Avoid import cycle in clearing client 2023-05-18 01:25:04 -04:00
Tyler Goodlet b2bf0b06f2 ib.api: wholesale fqsn -> fqme renames 2023-05-17 16:56:04 -04:00
Tyler Goodlet 907eaa68cb Pass `mkt: MktPair` to `.open_history_client()`
Since porting all backends to the new `FeedInit` + `MktPair` + `Asset`
style init, we can now just directly pass a `MktPair` instance to the
history endpoint(s) since it's always called *after* the live feed
`.stream_quotes()` ep B)

This has a lot of benefits including allowing brokerd backends to have
more flexible, pre-processed market endpoint meta-data that piker has
already validated; makes handling special cases in much more straight
forward as well such as forex pairs from legacy brokers XD

First pass changes all crypto backends to expect this new input, ib will
come next after handling said special cases..
2023-05-17 16:52:15 -04:00
Tyler Goodlet 89e8a834bf Support fqme rendering *without* the src key
Since most (legacy) stock brokers design their symbology without
including the target exchange's source asset name - normally a fiat
currency like USD - this adds an option for rendering market endpoints
without that token for simpler use in backends for such brokers.

As an example IB doesn't expect a `mnq/usd.cme.ib` symbol and instead
presumes that since the CME lists all assets in USD then the source
asset is implied.

Impl details:
- add `MktPair.pair: str` which replaces `.key` as a better name.
- offer a `without_src: bool` to a new `.get_fqme()` getter method
  which will render everything the same minus the src token.
- expose the new flag through both the new `.get_fqme()` and
  `.get_bs_fqme()` methods and wrap those both under the original
  property names `.bs_fqme` and `.fqme`.
2023-05-17 16:47:15 -04:00
Tyler Goodlet 12bfabf056 Expose `.accounting.unpack_fqme()` 2023-05-17 16:43:31 -04:00
Tyler Goodlet a44e926c2f kucoin: handle ws welcome, subs-ack and pong msgs
Previously the subscription response handling was a bit sloppy what with
ignoring the welcome msg; this now correctly expects the correct startup
sequence. Also this avoids warn logging on pong messages by expecting
them in the msg loop and further drops the `KucoinMsg` struct and
instead changes the msg loop to expect `dict`s and only cast to structs
on live feed msgs that we actually process/relay.
2023-05-17 12:30:52 -04:00
Tyler Goodlet 3294defee1 `fqme` adjustments to marketstore module
Mostly renaming from the old acronym. This also contains necessary
conf.toml loading in order to call `open_storage_client()` which now
does not have default network contact info.
2023-05-17 10:46:32 -04:00
Tyler Goodlet ae049eb84f Pass and use `MktPair` throughout history routines
Previously we were passing the `fqme: str` which isn't as extensive nor
were we able to pass `MktPair` direct to backend history manager-loading
routines (which should be able to rely on always receiving it since
currently `stream_quotes()` is always called first for setup).

This also starts a slight bit of configuration oriented tsdb info
loading (via a new `conf.toml`) such that a user can decide to host
their (marketstore) db on a remote host and our container spawning and
client code will do the right startup automatically based on the config.
|-> Related to this I've added some comments about doing storage
backend module loading which should get actually written out as part of
patches coming in #486 (or something related).

Don't allow overruns again in history context since it seems it was
never a problem?
2023-05-17 10:19:14 -04:00
Tyler Goodlet 5c8a45c64a Fix `MktPair.bs_fqme` to properly strip broker suffix 2023-05-17 09:45:00 -04:00
Tyler Goodlet 07b7d1d229 ib: implement `FeedInit` style quote stream setup
As per the new market info packing schema this patch almost gets it
completely compatible and useful via implementing the `get_mkt_info()`
backend module endpoint B)

There's still some questions around `MktPair.src` since all the contract
search machinery in the ib api isn't expecting a fiat currency in the
symbol key: for ex. `mnq/usd.cme.20230616.ib` has no handling for the
`[/]usd` part. For now i'm just excluding the `.src` since it requires
extra parsing on quotes-feed requests even though this is also currently
breaking forex pairs (idealpro or wtv). I think ideally we do move to
a `dst/src.<venue>.<etc..>` style but it's going to require adjustments
to all the existing crypto backends..

This also allows dropping the old `mk_init_msgs()` closure.
2023-05-16 17:29:07 -04:00
Tyler Goodlet 147e1baee9 Remove typo-ed `sum_tick_vlm` config from all crypto backends 2023-05-16 17:00:15 -04:00
Tyler Goodlet b096ee3b7a Make `FeedInit.shm_write_opts` an empty dict by default 2023-05-16 16:30:30 -04:00
Tyler Goodlet f20e2d6ee2 ib.feed: start drafting out `get_mkt_info()` endpoint 2023-05-15 15:35:57 -04:00
Tyler Goodlet 1263835034 ib.api: make `get_sym_details()` and `get_quote()` mutex methods 2023-05-15 15:35:30 -04:00
Tyler Goodlet 1e1e64f7f9 ib: fix op error when `end_dt` is `None`: the first query 2023-05-15 13:30:34 -04:00
Tyler Goodlet 98c043815a Woops, implement `Symbol.fqme` same a `Mktpair`.. 2023-05-14 20:24:19 -04:00
Tyler Goodlet ebe351e2ee kucoin: raise `DataUnavailable` if we get empty time array at some point? 2023-05-14 15:13:14 -04:00
Tyler Goodlet cfb125beef `.data.feed`: finally solve startup overruns issue
We need to allow overruns during the async multi-broker context spawning
init bc some backends might take longer then others to setup (eg.
binance vs. kucoin) and result in some context (stream) being overrun by
the time we get to the `.open_stream()` phase. Ideally, we can maybe
adjust the concurrent setup to be more of a task-per-provider style to
avoid this in the future - which would also in theory result in
more-immediate per-provider setup in terms showing ready feeds asap.

Also, does a bunch of renaming from fqsn -> fqme and drops the lower
casing of input symbols instead expecting the caller to know what the
data backend it's requesting is going to be able to handle in terms of
symbology.
2023-05-13 17:35:46 -04:00
Tyler Goodlet 1f0db3103d ib.broker: always cast `asset_type` to `str` 2023-05-13 17:27:45 -04:00
Tyler Goodlet 2e8268b53e Allow passing `allow_overruns: bool` to `Services.start_service_task()` 2023-05-13 16:51:11 -04:00
Tyler Goodlet b572cd1b77 kucoin: store fqme -> mktids table
Instead of pre-converting and mapping piker style fqmes to
`KucoinMktPair`s make `Client._pairs` keyed by the kucoin native market
ids and instead also create a `._fqmes2mktids: bidict[str, str]` for
doing lookups to the native pair from the fqme.

Also, adjust any remaining `fqsn` naming to fqme.
2023-05-13 16:45:05 -04:00
Tyler Goodlet b288d7051a ib.broker: load account name map as a `bidict` (no `tomlkit` support) 2023-05-13 16:44:28 -04:00
Tyler Goodlet c349d50f2f Allow creation of empty account files 2023-05-13 16:12:18 -04:00
Tyler Goodlet 779c0b73c9 Make `.accounting._ledger` use `tomlkit`
So that styling is preserved on write but requires that we pop `None`
values (in this case any unset `.expiry` transactions) due to `tomkit`
having no support for writing them as values?
2023-05-13 16:07:17 -04:00
Tyler Goodlet 50a4c425d3 Add `touch_if_dne: bool` to `config.load()`
So that we aren't creating blank files for legacy configs (as we do name
changes or wtv). Further change `.get_conf_path()` to validate against
new `account.` prefix and a god `conf.toml` file.
2023-05-13 16:05:23 -04:00
Tyler Goodlet df96155057 Always allow overruns in sampler context
Requires https://github.com/goodboy/tractor/pull/357.
Avoid overruns when doing concurrent live feed init over multiple
brokers.
2023-05-13 14:06:27 -04:00
Tyler Goodlet a62283bae2 Drop final use of `toml` 3rd party lib
We moved to `tomlkit` as per #496 and this lets us drop the mess that
was the inline-table encoder in `.accounting._toml` XD

Relates to #496
2023-05-12 16:15:12 -04:00
Tyler Goodlet 2865f0efe9 `piker.config`: use `tomlkit` for accounting files
We still need to get some patches landed in order to resolve:
- https://github.com/sdispater/tomlkit/issues/288
- https://github.com/sdispater/tomlkit/issues/289
- https://github.com/sdispater/tomlkit/issues/290

But, this does work for style preservation and the inline-table style we
were previously hacking into the `toml` lib in `.accounting._toml`,
which we can pretty much just drop now B)

Relates to #496 (pretty much solves it near-term i think?)
2023-05-12 16:05:45 -04:00
Tyler Goodlet 5f79434b23 Use new `.config` helpers for `accounting._pos/._ledger` file loading 2023-05-12 13:02:29 -04:00
Tyler Goodlet 5278f8b560 Add `.config.load_ledger()` for transaction record files 2023-05-12 13:01:45 -04:00
Tyler Goodlet 488a0cd119 Add `.config.load_account()`
Allows for direct loading of an "account file configuration" contents
without having to pass the explicit config dir path. In this case we are
also rewriting the `pps.<brokername>.<accnt_name>.toml` file names to
instead have a `account.` prefix, but providing this helper function
allows such changes more easily in the future - since callers won't have
to use the lower level `.load()` input signature.

Also add some todo comments about moving to `tomlkit`.
2023-05-12 12:40:09 -04:00
Tyler Goodlet 957224bdc5 ib: support remote host vnc client connections
I figure we might as well support multiple types of distributed
multi-host setups; why not allow running the API (gateway) and thus vnc
server on a diff host and allowing clients to connect and do their thing
B)

Deatz:
- make `ib._util.data_reset_hack()` take in a `vnc_host` which gets
  proxied through to the `asyncvnc` client.
- pull `ib_insync.client.Client` host value and pass-through to data
  reset machinery, presuming the vnc server is running in the same
  container (and/or the same host).
- if no vnc connection **and** no i3ipc trick can be used, just report
  to the user that they need to remove the data throttle manually.
- fix `feed.get_bars()` to handle throttle cases the same based on error
  msg matching, not error the code and add a max `_failed_resets` count
  to trigger bailing on the query loop.
2023-05-12 09:48:31 -04:00
Tyler Goodlet 7ff8aa1ba0 ib: passthrough host arg to vnc client for click hack 2023-05-11 12:32:38 -04:00
Tyler Goodlet e06f9dc5c0 kucoin: port to new `NoBsWs` api semantics
No longer need to implement connection timeout logic in the streaming
code, instead we just `async for` that bby B)

Further refining:
- better `KucoinTrade` msg parsing and handling with object cases.
- make `subscribe()` do sub request in a loop wand wair for acks.
2023-05-10 16:22:09 -04:00
Tyler Goodlet c6e5368520 paperboi: fix fqme parsing to handle `bs_fqme` cases 2023-05-09 18:34:01 -04:00
Tyler Goodlet 769b292dca Allow `brokerd` runtime switch to paper mode
Previously you couldn't have a brokerd backend which defined
`.trades_dialogue()` but which could also indicate that the paper
clearing engine should be used. This adds that support by allowing the
endpoint task to return a simple `'paper'` string, in which case the ems
will boot a paperboi.

The obvious useful case for this is if you have a broker you want to use
but do not have actual broker credentials setup (yet) with that provider
in your `brokers.toml`; demonstrated here with the adjustment to
`kraken`'s startup to no longer raise a runtime error B)
2023-05-09 18:29:28 -04:00
Tyler Goodlet 361fc4645c Drop passing `loglevel` to `stream_quotes()`, level is set when actor spawns 2023-05-09 18:28:51 -04:00
Tyler Goodlet f1f2ba2e02 kucoin: deliver `FeedInit` msgs on feed startup
To fit with the rest of the new requirements added in `.data.validate`
this adds `FeedInit` init including `MktPair` and `Asset` loading for
all spot currencies provided by `kucoin`.

Deatz:
- add a `Currency` struct and accompanying `Client.get_currencies()` for
  storing all asset infos.
- implement `.get_mkt_info()` which loads all necessary accounting and
  mkt meta-data structs including adding `.price/size_tick` fields to
  the `KucoinMktPair`.
- on client boot, async spawn requests to cache both symbols and currencies.
- pass `subscribe()` as the `fixture` arg to `open_autorecon_ws()`
  instead of opening it manually.

Other:
- tweak `Client._request` to not expect the prefixed `'/'` for the
  `endpoint: str`.
- change the `api_v` arg to just be `api: str`.
2023-05-09 18:17:50 -04:00
Tyler Goodlet 80338e1ddd kucoin: WIP moving to FeedInit API 2023-05-09 14:49:46 -04:00
Tyler Goodlet f8c8f63e87 Drop `Optional` usage from marketstore module 2023-05-09 14:49:46 -04:00
Tyler Goodlet 96532ad38c ui._display: no downsampling on history chart default view call 2023-05-09 14:49:46 -04:00
Tyler Goodlet 88f3912b2d test_ems: doc out some remaining suites 2023-05-09 14:49:46 -04:00
Tyler Goodlet cb8833d430 ib: clear error events on every received? 2023-05-09 14:49:46 -04:00
Tyler Goodlet 038b20d13a wsbs: increase msg rx timeout to 16 secs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 05fb4a4014 kraken: drop recv timeout for recon ws 2023-05-09 14:49:46 -04:00
Tyler Goodlet c415bd1ee1 If backend does not provide `bs_mktid`, use the `bs_fqme` 2023-05-09 14:49:46 -04:00
Tyler Goodlet 226c3364c3 Smh, handle `fixture==None` case.. 2023-05-09 14:49:46 -04:00
Tyler Goodlet 685688d2b2 ib: add `mbt.cme` micro-btc futes to adhoc set 2023-05-09 14:49:46 -04:00
Tyler Goodlet 7a3bce3f33 .data._web_bs: add client module name to log msgs 2023-05-09 14:49:46 -04:00
Tyler Goodlet 363a2bbcc6 binance: use new `int` sub-id for each request 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0a8dd7b6da Try to disable `snappy` compression on variables; it breaks everything XD 2023-05-09 14:49:46 -04:00
Tyler Goodlet 0b43e0aa8c Try having `brokerd` eps defined in `.brokers._daemon`
Since it's a bit weird having service specific implementation details
inside the general service `._daemon` mod, and since i'd mentioned
trying this re-org; let's do it B)

Requires enabling the new mod in both `pikerd` and `brokerd` and
obviously a bit more runtime-loading of the service modules in the
`brokerd` service eps to avoid import cycles.

Also moved `_setup_persistent_brokerd()` into the new mod since the
naming would place it there even though the implementation really
wouldn't (longer run) since we want to split up `.data.feed` layer
backend-invoked eps into a separate actor eventually from the "actual"
`brokerd` which will be the actor running **only** the trade control eps
(eg. trades_dialogue()` and friends).
2023-05-09 14:49:26 -04:00
Tyler Goodlet ed434e284b Disable ems init order-dialog notifications by default 2023-05-09 14:49:26 -04:00
Tyler Goodlet af068c5c51 binance: port `stream_messages()` to use `match:` and a new `L1` struct 2023-05-09 14:49:26 -04:00
Tyler Goodlet f6cd08c6fa Attempt to guard against numercial "anomalies" in `Viz.maxmin()`, add cacheing flag 2023-05-09 14:49:26 -04:00
Tyler Goodlet 34ff5ff249 kraken: port to new `NoBsWs`, passing timeout (counts) during setup 2023-05-09 14:49:26 -04:00
Tyler Goodlet b03564da2c binance: port to new `NoBsWs` api and drop `trio_util` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 59743b7b73 Rework `NoBsWs` to avoid agen/`trio` incompatibility
`trio`'s internals don't allow for async generator (and thus by
consequence dynamic reset of async exit stacks containing `@acm`s)
interleaving since doing so corrupts the cancel-scope stack. See details
in:
- https://github.com/python-trio/trio/issues/638
- https://trio-util.readthedocs.io/en/latest/#trio_util.trio_async_generator
- `trio._core._run.MISNESTING_ADVICE`

We originally tried to address this using
`@trio_util.trio_async_generator` in backend streaming code but for
whatever reason stopped working recently (at least for me) and it's more
or less implemented the same way as this patch but with more layers and
an extra dep. I also don't want us to have to address this problem again
if/when that lib isn't able to keep up to date with wtv `trio` is
doing..

So instead this is a complete rewrite of the conc design of our
auto-reconnect ws API to move all reset logic and msg relay into a bg
task which is respawned on reset-requiring events: user spec-ed msg recv
latency, network errors, roaming events.

Deatz:
- drop all usage of `AsyncExitStack` and no longer require client code
  to (hackily) call `NoBsWs._connect()` on msg latency conditions,
  intead this is all done behind the scenes and the user can instead
  pass in a `msg_recv_timeout: float`.
- massively simplify impl of `NoBsWs` and move all reset logic into a
  new `_reconnect_forever()` task.
- offer use of `reset_after: int` a count value that determines how many
  `msg_recv_timeout` events are allowed to occur before reconnecting the
  entire ws from scratch again.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 9d04accf2e Factor out all history mgmt-logic into a new `.data.history` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3cd853cb5d order_mode: revert switch to `MktPair` for pre-order loading 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4a0beda77e kraken: asyncify and use `get_mkt_info()` in `norm_trade_records()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d7288972b7 kraken: port to `FeedInit` and proper impl of `get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet 0d93871c88 kraken: drop `Client.cache_assets()`, simpler `.pair_info()`, drop `.mkt_info()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet d0e01ff9b6 Fix `Symbol.from_fqme()` extra added symbols.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet af2f8756c5 binance: use `@async_lifo_cache` on `.get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet bcf355e2c8 Fix up `@async_lifo_cache` typing, add TODOs for move to `tractor` 2023-05-09 14:49:26 -04:00
Tyler Goodlet e317310ed3 binance: make `stream_quotes()` deliver new `list[FeedInit]` API 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4131ff1152 Rename `bs_mktid` -> `bs_fqme` and drop (some) `fqsn`s
Since we have made `MktPair.bs_mktid` mean something else now, change
all the feed setup var names to instead be more representative of the
actual value: `bs_fqme: str` and use the new `MktPair.bs_fqme` where
necessary.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 83802e932a Drop (missed) usage of `Symbol.from_fqsn()` in order mode 2023-05-09 14:49:26 -04:00
Tyler Goodlet 765b8f8e5c Support both input msg-sequence types
The legacy version was a `dict` of `dicts` vs. now we want to be handed
a `list[FeedInit]`; process both in a factored way.

Drop `FeedInit.bs_mktid` since it's already defined on `.mkt.bs_mktid`
and we don't really need it top level.
2023-05-09 14:49:26 -04:00
Tyler Goodlet b4f2f49001 ib: make `stream_quotes()` compat with new init msg bare-minimums 2023-05-09 14:49:26 -04:00
Tyler Goodlet d1cf90e2ae ib: finally convert ledger processing to use `MktPair` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 6008497b89 Use more "hierarchical" schema for fsp shm segment names 2023-05-09 14:49:26 -04:00
Tyler Goodlet adb62dc7b4 Port oustanding parts of codebase to `unpack_fqme()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4129d693be Add `.data.validate` checker for live feed layer
More or less a replacement for what @guilledk did with the initial
attempt at a "broker check" type script a while back except in this case
we're going to always run this validation routine and it now uses a new
`FeedInit` struct to ensure backends are delivering the right schema-ed
data during startup. Also allows us to stick deprecation warnings / and
or strict API compat errors all in one spot (at least for live feeds).

Factors out a bunch of `MktPair` related adapter-logic into a new
`.validate.valiate_backend()` which warns to the backend implementer via
log msgs all the problems outstanding. Ideally we do our backend module
endpoint scan-and-complain regarding missing feature support from here
as well (eg. search, broker/trade ctl, ledger processing, etc.).
2023-05-09 14:49:26 -04:00
Tyler Goodlet d48b2c5b57 `._paper_engine`: right, load `MktPair` in `fqme is not None` usage 2023-05-09 14:49:26 -04:00
Tyler Goodlet 6f5a2654ab Port `.clearing` to new `unpack_fqme()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet afdbf8e10a `.accounting`: Use `_fqme()` throughout and export decimal converters 2023-05-09 14:49:26 -04:00
Tyler Goodlet d4c8ba19a2 `.accounting._mktinfo`: better fqme `MktPair` handling
It needed some work..

- Make `unpack_fqme()` always return a 4-tuple handling the venue and
  suffix parts more generally.
- add `Asset.Asset.guess_from_mkt_ep_key()` a like-it-sounds hack at
  trying to render a `.dst: Asset` for most most purposes throughout the
  stack.
- always try to preprocess the input `fqme: str` with `unpack_fqme()` in
  `MktPair.from_fqme()` and use the new `Asset` method (above) to make
  up a `.dst: Asset` pulling as much meta-info we can from the caller.
- add `MktPair.bs_fqme` to get the thing without the broker part..
- add an `'unknown'` value to the `_derivs` def.
- drop `Symbol.from_fqsn()` and `unpack_fqsn()` more generally (yes
  BREAKING).
2023-05-09 14:49:26 -04:00
Tyler Goodlet 53a41ba93d Add subsys log to new `.data._util` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 06b80ff9ed ARRG, disable `dunst` notifications for now in order mode 2023-05-09 14:49:26 -04:00
Tyler Goodlet fa88924f84 Do we need feed mod enabled? no right? 2023-05-09 14:49:26 -04:00
Tyler Goodlet 83f1922f6e `binance.get_mkt_info()`: bleh, right `@lru_cache` dun work for async.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4b7ac1d895 Port paper engine to latest `.accounting` sys fixes
- only preload necessary (one for clearing, all for ledger sync)
  `MktPair` info from the backend using `.get_mkt_info()`, build the
  `mkt_by_fqme: dict[str, MktPair]` and pass it to
  `TransactionLedger.iter_trans()`.
- use new `TransactionLedger.update_from_t()` method on clears.
- sanity check all `mkt_by_fqme` entries against `Flume.mkt` values
  when we open a data feed.
- rename `PaperBoi._syms` -> `._mkts`.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 7ee6f36e62 Actually, require `mkt_by_fqme` in `.iter_trans()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet f106472bcb Fix size quantization and closed position popping..
Turns out we actually had further pp entry bugs due to *not quantizing*
the size inside `.minimize_clears()` method calcs; fix that using
`Position.sys.mkt.quantize()` as is done in `Position.calc_size()`.

Fix `PpTable.write_config()` to drop from the TOML config any
`closed: dict[str, Position]` entries delivered by `.dump_active()`.

Add a more detailed doc string for our position type and a little todo
for the `.bep` B)
2023-05-09 14:49:26 -04:00
Tyler Goodlet bba1ee43ff Allow mkt info table input to `.iter_trans()`
Since ledger records are often provided (and thus stored) from most
backends *without* containing the info we normally need for accounting
defined by `MktPair`, this extends the ledger method to take in a table
that allows assigning the `Transaction.sys` from an fqme lookup. This
way client code (like the paper engine and new ledger mgmt tools) can
do the mkt info lookup before hand and then load both ledger
`Transactions` and positions via the `PpTable` and get correct
accounting calculations, always :fingers_crossed:

Also adds `TransactionLedger.update_from_t(t: Transaction)` to allow
updating directly from an existing tran instead of making the user cast
to a `dict` first. Includes fix to `.to_dict()` to always pop the `.sym`
again to avoid client code having to do so.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 0d2e713e9a `binance`: facepalm, swap price/size_tick methods..
Wow not sure how that happened, but we should probably use the correct
market precision info for the correct parameter..

Also, use `@lru_cache` on new `get_mkt_info()` ep, seems to work?
2023-05-09 14:49:26 -04:00
Tyler Goodlet 10a39ca42c More detailed dark-slap comments 2023-05-09 14:49:26 -04:00
Tyler Goodlet 0917b580c9 Flip `.feed` and `._sampling` over to new stuff
In `.feed` and `._sampling` move to using the new
`tractor.Context.open_stream(allow_overruns: bool)` (cough, A BREAKING
CHANGE).

Also set `Flume.mkt` during construction in `.feed.open_feed()`.
2023-05-09 14:49:26 -04:00
Tyler Goodlet a301fabd6c Change`.ui._fsp` to use `Flume.mkt` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 611d86d988 Change `Flume.symbol` -> `.mkt: MktPair`
Might as well try and flip it over to the new type; make appropriate
dict serialization changes in `.to_msg()`. Alias back to `.symbol:
Symbol` with a property.
2023-05-09 14:49:26 -04:00
Tyler Goodlet b1e162ebb4 Fix ._util import in questrade backend 2023-05-09 14:49:26 -04:00
Tyler Goodlet 48cae3c178 `ib`: rejects their own fractional size tick..
Frickin ib, they give you the `0.001` (or wtv) in the
`ContractDetails.minSize: float` but won't accept fractional sizes
through the API.. Either way, it's probably not sane to be supporting
fractional order sizes for legacy instruments by default especially
since it in theory affects a lot of the clearing outcomes by having ib
do wtv magical junk behind the scenes to make it work..
2023-05-09 14:49:26 -04:00
Tyler Goodlet 2cf7daca30 Another fqsn -> fqme rename 2023-05-09 14:49:26 -04:00
Tyler Goodlet dedc51a939 Quantize order prices prior to `OrderClient.send()`
Order mode previously was just willy-nilly sending `float` prices
(particularly on order edits) which are generated from the associated
level line. This actually uses the `MktPair.price_tick: Decimal` to
ensure the value is rounded correctly before submission to the ems..

Also adjusts the order mode init to expect a table of tables of startup
position messages, with the inner table being keyed by fqme per msg.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 589232d12d Only flip size sign for seels if not already -ve 2023-05-09 14:49:26 -04:00
Tyler Goodlet 928765074f Fix zero-pp entry to toml case for new file-per-account format 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2ed9e40d5e Better EMS client-side msg formatting 2023-05-09 14:49:26 -04:00
Tyler Goodlet e524c6fe4f `binance`: add startup caching info log msg 2023-05-09 14:49:26 -04:00
Tyler Goodlet abbba1fa6e Pack startup pps into a table keyed by fqmes 2023-05-09 14:49:26 -04:00
Tyler Goodlet 484565988d `order_mode`: broad rename book -> client 2023-05-09 14:49:26 -04:00
Tyler Goodlet f92c289842 Drop old blessings code, general cleanups 2023-05-09 14:49:26 -04:00
Tyler Goodlet b7ddf9cb05 paper-eng: close context and terminate actor on exit 2023-05-09 14:49:26 -04:00
Tyler Goodlet 250e1c4c51 `ledger` cli: dump colored summary lines to console
Tried a couple libs and ended up sticking with `rich` (since it's the
sibling lib to `typer`) but also (initially) implemented a version with
`blessings` that I ended up commenting out (and will likely remove).

Adjusted the CLI I/O a slight bit as well:
- require a fully qualified account name of the form:
  `<brokername>.<accountname>` and error on non-matching input.
- dump positions summary lines as humanized size, ppu and cost basis
  values per line.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 62259880fd paper: on no input fqme, load all mktinfos from pos table 2023-05-09 14:49:26 -04:00
Tyler Goodlet f42bc2dbce `pprint.pformat()` IB position mismatch log msgs 2023-05-09 14:49:26 -04:00
Tyler Goodlet 55b4866d5e Use `force_mkt` override in paper pps updates
When processing paper trades ledgers we normally won't have specific
`MktPair` info for the backend market we're simulating, as such we
need to look up this info when updating pps.toml files such that we
get precision info correct (particularly in the case of cryptos!) and
can also run paper ledger processing without running the simulated
clearing loop. In order to make it happen we lookup any `get_mkt_info()`
ep on the backend and pass the output to the `force_mkt` input of the
`PpTable.update_from_trans()` method.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 83514b0e90 `binance`: add `get_mkt_info()` ep 2023-05-09 14:49:26 -04:00
Tyler Goodlet 21401853c4 `kraken`: add module level `get_mkt_info()`
This will (likely) act as a new backend query endpoint for other `piker`
(client) code to lookup `MktPair` info from each backend. To start it
also returns the backend-broker's local `Pair` (or wtv other type) as
well.

The main motivation for this is for our paper engine which can require
the mkt info when processing paper-trades ledgers which do not contain
appropriate info to compute position metrics.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 6decd4112a kraken: drop console setup, now done during brokerd init 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3f2f5edb28 kraken: rename `Client._atable` -> `_altnames` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 1d2d4b40a8 Only log about pps once in order mode code 2023-05-09 14:49:26 -04:00
Tyler Goodlet 5ee044e418 Another `@acm` in `._cacheables` XD 2023-05-09 14:49:26 -04:00
Tyler Goodlet b8a975a3fd Drop `"<broker>.<account>.."` from pps.toml entries
Add special blocks to handle removing the broker account levels from
both writing and reading routines.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 33a78366ff paper: always sync pps.toml state on startup 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2806a4c0e5 Tweak ems msg-received log msg 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2d609dceac Drop `loglevel` from `spawn_args` inputs to `maybe_spawn_daemon()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet b2a5f8698d Use `--pdb` flag to config `brokerd` debug mode 2023-05-09 14:49:26 -04:00
Tyler Goodlet 70efce1631 `kraken`: handle ws connection startup status msgs 2023-05-09 14:49:26 -04:00
Tyler Goodlet a63599828b Drop masked `MktPair.size_tick_digits()` cruft 2023-05-09 14:49:26 -04:00
Tyler Goodlet f51361435f paper engine: use the `fqme` for the `bs_mktid`
Instead of stripping the broker part just use the full fqme for all
`Transaction.bs_mktid: str` values since it makes indexing the `PpTable`
much easier with less key mangling..
2023-05-09 14:49:26 -04:00
Tyler Goodlet 9770a39d7b Cancel the `OrderClient` sync-method relay task on exit 2023-05-09 14:49:26 -04:00
Tyler Goodlet 97e3c06af8 Set `emsd` log level and clearly report startup pps
Change the root-service-task entrypoint to accept the level and
setup a console log as is now expected for all sub-services. Cast all
backend delivered startup `BrokerdPosition` msgs and log them to
console.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 4c1d174801 Expect `loglevel: str` in brokerd root task ep
Set the level right after spawn and once for the lifetime of the daemon.
2023-05-09 14:49:26 -04:00
Tyler Goodlet eb7a7462ad Always pass `loglevel: str` to daemon root task eps
If you want a sub-actor to write console logs (with the right level) the
`get_console_log()` call has to be made somewhere during service task
startup. Previously this wasn't well formalized nor used (depending on
daemon) so passing `loglevel` to the service's root-task-endpoint (eg.
`_setup_persistent_brokerd()`) encourages that the daemon's logging is
configured during init according to the spawner's requesting logging
config. The previous `get_console_log()` call happening inside
`maybe_spawn_daemon()` wasn't actually doing anything in the target
daemon XD, so obviously remove that and instead passthrough loglevel
to the ctx endpoints and service manager methods.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 1944f75ae8 Expose `piker.clearing.OrderClient` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 008bfed702 ib: lul, fix oil (cl) venue to correctly be nymex.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet 56cd15fa51 ib: maybe incr client id; can't catch api errors..
Turns out we don't hookup our eventkit handler until after the
`load_aio_clients()` is complete, which means we can't get
`ib_insync.Client.apiError` events unless inside the asyncio side task.
So I guess try to report any such errors during API scan (note the
duplicate client id case is a special one from ibis itself) even though
we're not going to catch them trio side. The hack to work around this is
to just increment the client id value with the `connect_retries` led `i`
value even though that will break on more then 3 clients attached to an
API endpoint lul ..

Further adjustments that were to the end of trying to fix this proper:
- add `remove_handler_on_err()` cm to disconnect a handler when the trio
  side of the channel closes.
- actually connect to client api erros in our `Client.inline_errors()`
- increase connect timeout to a sec.
- change the trio-asyncio proxy response-msg loop over to `match:`
  syntax and raise on unhandled msgs from eventkit handlers.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 879657cc75 Detail `pikerd` sock bind collision in error 2023-05-09 14:49:26 -04:00
Tyler Goodlet fb13c7cbf6 `ib`: drop pp mismatch err block, we already do it in audit routine 2023-05-09 14:49:26 -04:00
Tyler Goodlet 72abe98475 Async-ify order client methods and some renaming
We previously only offered a sync API (which was recently renamed to
`.<meth>_nowait()` style) since initially all order control was from our
`OrderMode` Qt driven UI/UX. This adds the equivalent async methods for
both testing as well as eventual auto-strat driven control B)

Also includes a bunch of renaming:
- `OrderBook` -> `OrderClient`.
- better internal renaming of the client's mem chan vars and add a ref
  `._ems_stream: tractor.MsgStream`.
- drop `get_orders()` factory, just always check for the actor-global
  instance and always set the ems stream on that client (in case old one
  was closed).
2023-05-09 14:49:26 -04:00
Tyler Goodlet 48f096995f `kraken`: write ledger and pps files on startup 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2cc77c21ba Rework paper engine for "offline" pp loading
This will end up being super handy for testing our accounting subsystems
as well as providing unified and simple cli utils for managing ledgers
and position tracking. Allows loading the paper boi without starting
a data feed and instead just trigger ledger and pps loading without
starting the entire clearing engine.

Deatz:
- only init `PaperBoi` and start clearing loop (tasks) if a non-`None`
  fqme is provided, ow just `Context.started()` the existing pps msgs
  as loaded from the ledger.
- always update both the ledger and pp table on startup and pass
  a single instance of each obj to the `PaperBoi` for reuse (without
  opening and closing backing config files since we now have
  `.write_config()`).
- drop the global `_positions` dict, it's not needed any more if we use
  a `PaperBoi.ppt: PpTable` which persists with the engine actor's
  lifetime.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 1560330acd Convert `Flume.MktPair.size_tick` to float for dark clearing 2023-05-09 14:49:26 -04:00
Tyler Goodlet a74caa9f77 Add paper engine "offline loading" support to the ledger cli 2023-05-09 14:49:26 -04:00
Tyler Goodlet 61fb783c4e Formalize a ledger type + API: `TransactionLedger`
Add a new `class TransactionLedger(collections.UserDict)` for managing
ledger (files) from a `dict`-like API. The main motivations being easy
conversion between `dict` <-> `Transaction` obj forms as well as dynamic
(toml) file updates via a set of methods:

- `.write_config()` to render and write state to the local toml file.
- `.iter_trans()` to allow iterator style conversion to `Transaction`
  form for each entry.
- `.to_trans()` for the dict output from the above.

Some adjustments to `Transaction` namely making `.sym/.sys` optional for
now so that paper engine entries can be loaded (offline) without
connecting to the emulated broker backend. Move to using `pathlib.Path`
throughout for bootyful toml file mgmt B)
2023-05-09 14:49:26 -04:00
Tyler Goodlet 9f7aa3d1ff Always use the "most resolved" `Position.symbol: MktPair`
When loading a `Position` from a pps file we might not have the entire
`MktPair` field-set loaded (though going forward that shouldn't really
ever happen except in the case of a legacy `pps.toml`), in which case we
can check if the `.fqme: str` value loaded from the transaction is
longer and use that instead - presuming it must have more mkt meta-data
filled out.

Also includes some more `fqsn` -> `fqme` renames.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 50be10a9bd `ib`: keep broker name in `Transaction.fqsn` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 29a5910b90 `ib`: move flex utils to new submod 2023-05-09 14:49:26 -04:00
Tyler Goodlet a336def65f `ib`: again, only *update* ledger records from API 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2cb59fe450 Flatter format for pos/ledger mngr statements 2023-05-09 14:49:26 -04:00
Tyler Goodlet 4494acbc01 Write a separate `pps.<brokername>.<accountname>.toml` file per account 2023-05-09 14:49:26 -04:00
Tyler Goodlet 7b3d724908 Rework `.config` routines to use `pathlib.Path`
Been meaning to do this port for a while and since it makes passing
around file handles (presumably alongside the in mem obj form) a lot
simpler/nicer and the implementations of all the config file handling
much more terse with less presumptions about the form of filename/dir
`str` values all over the place B)

moar technically, let's us:
- drop remaining `.config` usage of `os.path`.
- return `Path`s from most routines.
- adds a special case to `get_conf_path()` such that if the input name
  contains a `pps.` pattern, we avoid validating the name; this is going
  to be used by new `.accounting.open_pps()` code which will instead
  write a separate TOML file for each account B)
2023-05-09 14:49:26 -04:00