Commit Graph

3873 Commits (9a063ccb11620226bc7b8b237fe143675a5bb52a)

Author SHA1 Message Date
Tyler Goodlet 9a063ccb11 ib: Solve lingering bugs for non-vlm contracts
Contract matching in live setup was borked; switch to
`MktPair.dst.atype` matching, don't override the `cmdty` "venue" (a
weird special case) in `get_mkt_info()` otherwise lookup will fail..
2023-05-24 09:11:24 -04:00
Tyler Goodlet e8787d89c6 ib: unset vlm via new `FeedInit.shm_write_opts` field 2023-05-24 08:28:16 -04:00
Tyler Goodlet 8e97814c1f Add "no vlm" indication to `FeedInit`
Stash it for now in the (now mutable by default) `.shm_write_opts` and
have the new `Flume._has_vlm: bool` (only set to false internally by
feed layer) which can be read via new public `.has_vlm()` predicate.
Move out the old `.ui/_fsp` helper logic to this flume method.
2023-05-24 08:25:14 -04:00
Tyler Goodlet e82f7f9012 Skip elasticsearch test for now, container build seems borked? 2023-05-23 22:39:38 -04:00
Tyler Goodlet b44b0915ca ib: i guess only discard `MktPair.src: Asset` on non-forex XD 2023-05-23 19:11:40 -04:00
Tyler Goodlet ff74d47fd5 kucoin: fix fqme or search result key lookups 2023-05-23 16:46:21 -04:00
Tyler Goodlet 6ad8c603d5 More detailed `Position.events` todo 2023-05-23 16:45:58 -04:00
Tyler Goodlet cd55d027c4 Re-implement db tests using new ahab daemons
Avoids the really sloppy flag passing to `open_pikerd()` and allows for
separation of the individual docker daemon starts.

Also add a new `root_conf() -> Path` fixture which will open and load
the `dict` for the new root `conf.toml` file.
2023-05-23 14:16:08 -04:00
Tyler Goodlet d094625bd6 Activate docker daemons via flags using exit stack 2023-05-23 14:16:08 -04:00
Tyler Goodlet e7a172b656 Reimplement marketstore and elasticsearch daemons
Using the new `._ahab.start_ahab_service()` mngr of course, and now
support user config overrides (such that our defaults can be modified by
a keen user, say using a config file, or for testing). This is where the
functionality moved out of the `pikerd` init has been moved - instead of
being triggered by bool flag inputs to that factory.

For marketstore actually support overriding the entire yaml config via
runtime `_yaml_config_str: str` formatting with any passed user dict,
primarily focussing on supporting override of the sockaddrs for testing.
2023-05-23 14:16:02 -04:00
Tyler Goodlet bd919f9d66 _ahab: use `Services` api to spawn docker tasks
Allows for using the `Services.cancel_service()` api for explicit
cancellation in tests and eventually for remote teardown. Change
`.start_ahab()` to an `@acm` `start_ahab_service()` and just yield back
the same values we were returning prior. Also fix the logging (level) to
actually reflect what's passed in - we weren't using the correct name
/ instance from the `.sevice` subpkg..
2023-05-23 14:16:02 -04:00
Tyler Goodlet 611d1ee3fc Drop db flags from pikerd startup 2023-05-23 14:16:02 -04:00
Tyler Goodlet 56b23e1fcc Add docker and elasticsearch to test deps 2023-05-23 14:16:02 -04:00
Tyler Goodlet d3bafb0063 Always prefer a config template if found 2023-05-23 14:16:02 -04:00
Tyler Goodlet 7f246697b4 Remove remaining `fqsn` usage from code base minus backward compats 2023-05-23 14:16:02 -04:00
Tyler Goodlet dd10acbbf9 Replace `Transaction.fqsn` -> `.fqme`
Change over all client (broker) code which constructs transactions
and finally wipe required `.fqsn` usage from `.accounting` B)
2023-05-23 14:15:57 -04:00
Tyler Goodlet 31a00eca94 Rename fqsn -> fqme in ui mods 2023-05-22 12:13:00 -04:00
Tyler Goodlet c93d119873 Move tmpdir creation into separate fixture
Since `.config.load()` was changed to not touch conf files by default
(without explicitly setting `touch_if_dne: bool`), this ensures both the
global module value is set and the `brokers.toml` file exists before
every test.
2023-05-22 12:03:32 -04:00
Tyler Goodlet 588770d034 ib: rename lingering fqsn -> fqme 2023-05-22 12:00:13 -04:00
Tyler Goodlet 2f2d612b5f Add todo to switch to `dst/src` delim 2023-05-22 11:57:37 -04:00
Tyler Goodlet 660a94d610 Don't expect `conf.toml`'s network section
For testing this is particularly true until we offer a template
with whatever (likely localhost) settings planned to ship.
2023-05-22 11:54:36 -04:00
Tyler Goodlet e4e4cacef3 .data.feed: Less stringency with fqme matching
`Flume.mkt.fqme` might not be exactly the same as the local
version now since we've had to add some hacks to certain backends
(cough ib) to handle `MktPair.src` not being set as an `Asset` (yet).
2023-05-22 11:52:36 -04:00
Tyler Goodlet 60a6f3269c ib: use flex report datetime sort
Since `open_trade_ledger()` now requires a sort we pass in a combo of
the std `pendulum.parse()` for API records and a custom flex parser for
flex entries pulled offline.

Add special handling for `MktPair.src` such that when it's a fiat (like
it should always be for most legacy assets) we try to get the fqme
without that `.src` token (i.e. not mnqusd) to avoid breaking
roundtripping of live feed requests (due to new symbology) as well as
the current tsdb table key set..

Do a wholesale renaming of fqsn -> fqme in most of the rest of the
backend modules.
2023-05-22 09:41:44 -04:00
Tyler Goodlet 53003618cb Add longer timeout on brokerd ctx cancel; seems to work? 2023-05-22 00:16:58 -04:00
Tyler Goodlet c6da09f3c6 Add fast(er), time-sorted ledger records
Turns out that reading **and** writing with `tomlkit` is just wayya slow
for large documents like ledger files so move to using the `tomli`
sibling pkg `tomli-w` which seems to much improve on the latency, though
obviously longer run we're likely going to want:
- a better algorithm for only back loading records using as little
  history as possible
- a different serialization format for production maybe something
  like apache parquet?

The only issue with using a non-style-preserving writer is that we don't
necessarily get TOML conf ordering for free (without first ordering it
ourselves), and thus this patch also adds much more general date-time
sorting machinery which is now **required** when using
`open_trades_ledger()` via a `tx_sort: Callable`. By default we now
provide `.accounting._ledger.iter_by_dt()` (exposed in the subpkg mod)
which conducts dynamic "datetime key detection" based parsing of records
based on a `parsers: dict[str, Callabe]` input table. The default should
handle most use cases including all currently supported live backends
(kraken, ib) as well as our paper engine ledger-records format.

Granulars:
- adjust `Position.iter_clears()` to use new `iter_by_dt(key=lambda ..)`
  signature.
- add `tomli-w` to setup and our `tomlkit` fork to requirements file.
- move `.write_config()` to bottom of class defn.
- fix closed pos popping to not error if pp was already popped..
2023-05-18 18:27:54 -04:00
Tyler Goodlet 89d24cfe33 Oof, fix closed position popping by fqme.. 2023-05-18 12:52:34 -04:00
Tyler Goodlet 8d7a9fa19e Make `MktPair.pair()` a meth, allow passing in a delim character 2023-05-18 12:01:30 -04:00
Tyler Goodlet a1a10676cd Go back to `tomllib` for ledger loading, it's wayy faster 2023-05-18 11:27:31 -04:00
Tyler Goodlet 97b2b25256 Avoid import cycle in clearing client 2023-05-18 01:25:04 -04:00
Tyler Goodlet b2bf0b06f2 ib.api: wholesale fqsn -> fqme renames 2023-05-17 16:56:04 -04:00
Tyler Goodlet 907eaa68cb Pass `mkt: MktPair` to `.open_history_client()`
Since porting all backends to the new `FeedInit` + `MktPair` + `Asset`
style init, we can now just directly pass a `MktPair` instance to the
history endpoint(s) since it's always called *after* the live feed
`.stream_quotes()` ep B)

This has a lot of benefits including allowing brokerd backends to have
more flexible, pre-processed market endpoint meta-data that piker has
already validated; makes handling special cases in much more straight
forward as well such as forex pairs from legacy brokers XD

First pass changes all crypto backends to expect this new input, ib will
come next after handling said special cases..
2023-05-17 16:52:15 -04:00
Tyler Goodlet 89e8a834bf Support fqme rendering *without* the src key
Since most (legacy) stock brokers design their symbology without
including the target exchange's source asset name - normally a fiat
currency like USD - this adds an option for rendering market endpoints
without that token for simpler use in backends for such brokers.

As an example IB doesn't expect a `mnq/usd.cme.ib` symbol and instead
presumes that since the CME lists all assets in USD then the source
asset is implied.

Impl details:
- add `MktPair.pair: str` which replaces `.key` as a better name.
- offer a `without_src: bool` to a new `.get_fqme()` getter method
  which will render everything the same minus the src token.
- expose the new flag through both the new `.get_fqme()` and
  `.get_bs_fqme()` methods and wrap those both under the original
  property names `.bs_fqme` and `.fqme`.
2023-05-17 16:47:15 -04:00
Tyler Goodlet 12bfabf056 Expose `.accounting.unpack_fqme()` 2023-05-17 16:43:31 -04:00
Tyler Goodlet a44e926c2f kucoin: handle ws welcome, subs-ack and pong msgs
Previously the subscription response handling was a bit sloppy what with
ignoring the welcome msg; this now correctly expects the correct startup
sequence. Also this avoids warn logging on pong messages by expecting
them in the msg loop and further drops the `KucoinMsg` struct and
instead changes the msg loop to expect `dict`s and only cast to structs
on live feed msgs that we actually process/relay.
2023-05-17 12:30:52 -04:00
Tyler Goodlet d0ba9a0a58 Start draft `conf.toml` "root" config with tsdb contact info 2023-05-17 10:58:12 -04:00
Tyler Goodlet 3294defee1 `fqme` adjustments to marketstore module
Mostly renaming from the old acronym. This also contains necessary
conf.toml loading in order to call `open_storage_client()` which now
does not have default network contact info.
2023-05-17 10:46:32 -04:00
Tyler Goodlet ae049eb84f Pass and use `MktPair` throughout history routines
Previously we were passing the `fqme: str` which isn't as extensive nor
were we able to pass `MktPair` direct to backend history manager-loading
routines (which should be able to rely on always receiving it since
currently `stream_quotes()` is always called first for setup).

This also starts a slight bit of configuration oriented tsdb info
loading (via a new `conf.toml`) such that a user can decide to host
their (marketstore) db on a remote host and our container spawning and
client code will do the right startup automatically based on the config.
|-> Related to this I've added some comments about doing storage
backend module loading which should get actually written out as part of
patches coming in #486 (or something related).

Don't allow overruns again in history context since it seems it was
never a problem?
2023-05-17 10:19:14 -04:00
Tyler Goodlet 5c8a45c64a Fix `MktPair.bs_fqme` to properly strip broker suffix 2023-05-17 09:45:00 -04:00
Tyler Goodlet 07b7d1d229 ib: implement `FeedInit` style quote stream setup
As per the new market info packing schema this patch almost gets it
completely compatible and useful via implementing the `get_mkt_info()`
backend module endpoint B)

There's still some questions around `MktPair.src` since all the contract
search machinery in the ib api isn't expecting a fiat currency in the
symbol key: for ex. `mnq/usd.cme.20230616.ib` has no handling for the
`[/]usd` part. For now i'm just excluding the `.src` since it requires
extra parsing on quotes-feed requests even though this is also currently
breaking forex pairs (idealpro or wtv). I think ideally we do move to
a `dst/src.<venue>.<etc..>` style but it's going to require adjustments
to all the existing crypto backends..

This also allows dropping the old `mk_init_msgs()` closure.
2023-05-16 17:29:07 -04:00
Tyler Goodlet 147e1baee9 Remove typo-ed `sum_tick_vlm` config from all crypto backends 2023-05-16 17:00:15 -04:00
Tyler Goodlet b096ee3b7a Make `FeedInit.shm_write_opts` an empty dict by default 2023-05-16 16:30:30 -04:00
Tyler Goodlet f20e2d6ee2 ib.feed: start drafting out `get_mkt_info()` endpoint 2023-05-15 15:35:57 -04:00
Tyler Goodlet 1263835034 ib.api: make `get_sym_details()` and `get_quote()` mutex methods 2023-05-15 15:35:30 -04:00
Tyler Goodlet 1e1e64f7f9 ib: fix op error when `end_dt` is `None`: the first query 2023-05-15 13:30:34 -04:00
Tyler Goodlet 98c043815a Woops, implement `Symbol.fqme` same a `Mktpair`.. 2023-05-14 20:24:19 -04:00
Tyler Goodlet ebe351e2ee kucoin: raise `DataUnavailable` if we get empty time array at some point? 2023-05-14 15:13:14 -04:00
Tyler Goodlet cfb125beef `.data.feed`: finally solve startup overruns issue
We need to allow overruns during the async multi-broker context spawning
init bc some backends might take longer then others to setup (eg.
binance vs. kucoin) and result in some context (stream) being overrun by
the time we get to the `.open_stream()` phase. Ideally, we can maybe
adjust the concurrent setup to be more of a task-per-provider style to
avoid this in the future - which would also in theory result in
more-immediate per-provider setup in terms showing ready feeds asap.

Also, does a bunch of renaming from fqsn -> fqme and drops the lower
casing of input symbols instead expecting the caller to know what the
data backend it's requesting is going to be able to handle in terms of
symbology.
2023-05-13 17:35:46 -04:00
Tyler Goodlet 1f0db3103d ib.broker: always cast `asset_type` to `str` 2023-05-13 17:27:45 -04:00
Tyler Goodlet 2e8268b53e Allow passing `allow_overruns: bool` to `Services.start_service_task()` 2023-05-13 16:51:11 -04:00
Tyler Goodlet b572cd1b77 kucoin: store fqme -> mktids table
Instead of pre-converting and mapping piker style fqmes to
`KucoinMktPair`s make `Client._pairs` keyed by the kucoin native market
ids and instead also create a `._fqmes2mktids: bidict[str, str]` for
doing lookups to the native pair from the fqme.

Also, adjust any remaining `fqsn` naming to fqme.
2023-05-13 16:45:05 -04:00