Commit Graph

3941 Commits (f25248c87188ac37673d748c52f27353c6473cdb)

Author SHA1 Message Date
Tyler Goodlet 5e2107ff15 Adjust `config.load()` to handle CI git checkout dir, seems they changed it!? 2023-05-26 16:50:15 -04:00
Tyler Goodlet 5f1d0fcb8c `tmpconfdir`: always assert brokers config created 2023-05-26 14:58:59 -04:00
Tyler Goodlet 3b5bd8f43e Ensure quote last price is a `float` 2023-05-26 14:42:35 -04:00
Tyler Goodlet 40c5f39f0d conftest: be explicit about which config we touch 2023-05-26 14:42:09 -04:00
Tyler Goodlet 3d8c1a7b3c ib: don't log-emit ib pp msg when none exists.. 2023-05-26 14:05:32 -04:00
Tyler Goodlet 06cc3ac92c Tidy up ems tests as per some `ruff`in 2023-05-25 18:04:52 -04:00
Tyler Goodlet 4a8e8a32f9 Fix account config loading logic discovered in new test XD 2023-05-25 17:56:14 -04:00
Tyler Goodlet 9bc11d8dd9 Add basic config checking tests 2023-05-25 17:55:20 -04:00
Tyler Goodlet 9c80969fd5 .data.validate: add missing endpoint warnings 2023-05-25 16:01:21 -04:00
Tyler Goodlet da4d344e63 Change to `piker_pin` branch in `tomlkit` fork 2023-05-25 13:53:14 -04:00
goodboy 073ff0103a
Merge pull request #506 from pikers/py311
`python3.11` support!
2023-05-24 19:34:10 -04:00
Tyler Goodlet f0a346dcc3 Some linting fixes after trying out `ruff` 2023-05-24 17:25:23 -04:00
Tyler Goodlet 7381c361cd Strictly drop `LinkedSplits.symbol` B) 2023-05-24 15:42:14 -04:00
Tyler Goodlet 1b577eebf6 Change over the UI layer to use `MktPair`
Including changing to `LinkedSplits.mkt: MktPair` and adding an explicit
setter method for setting it and being sure that nothing breaks
in the display system init!

For this commit we leave in warning access to `LinkedSplits.symbol` but
will remove in following commit.
2023-05-24 15:30:17 -04:00
Tyler Goodlet 39af215d61 kraken: use new `Position.mkt` attr 2023-05-24 15:29:42 -04:00
Tyler Goodlet 35f0520cb0 Drop `Symbol` / `.symbol` support from `.accounting`
Only stuff left was the allocator stuff. Drop the top level subpkg
exports and finally kill off the awkwardly named
`Symbol.lot_size_digits` properties XD

Expose a bunch more util funcs at subpkg top level, do some typing in
allocator method internals.
2023-05-24 15:26:51 -04:00
Tyler Goodlet 738d0ca38b Rename db tests to test_docker_services 2023-05-24 12:30:57 -04:00
Tyler Goodlet bd8e4760d5 Port everything strictly to `Position.mkt` and `Flume.mkt` 2023-05-24 12:16:28 -04:00
Tyler Goodlet 9a063ccb11 ib: Solve lingering bugs for non-vlm contracts
Contract matching in live setup was borked; switch to
`MktPair.dst.atype` matching, don't override the `cmdty` "venue" (a
weird special case) in `get_mkt_info()` otherwise lookup will fail..
2023-05-24 09:11:24 -04:00
Tyler Goodlet e8787d89c6 ib: unset vlm via new `FeedInit.shm_write_opts` field 2023-05-24 08:28:16 -04:00
Tyler Goodlet 8e97814c1f Add "no vlm" indication to `FeedInit`
Stash it for now in the (now mutable by default) `.shm_write_opts` and
have the new `Flume._has_vlm: bool` (only set to false internally by
feed layer) which can be read via new public `.has_vlm()` predicate.
Move out the old `.ui/_fsp` helper logic to this flume method.
2023-05-24 08:25:14 -04:00
Tyler Goodlet e82f7f9012 Skip elasticsearch test for now, container build seems borked? 2023-05-23 22:39:38 -04:00
Tyler Goodlet b44b0915ca ib: i guess only discard `MktPair.src: Asset` on non-forex XD 2023-05-23 19:11:40 -04:00
Tyler Goodlet ff74d47fd5 kucoin: fix fqme or search result key lookups 2023-05-23 16:46:21 -04:00
Tyler Goodlet 6ad8c603d5 More detailed `Position.events` todo 2023-05-23 16:45:58 -04:00
Tyler Goodlet cd55d027c4 Re-implement db tests using new ahab daemons
Avoids the really sloppy flag passing to `open_pikerd()` and allows for
separation of the individual docker daemon starts.

Also add a new `root_conf() -> Path` fixture which will open and load
the `dict` for the new root `conf.toml` file.
2023-05-23 14:16:08 -04:00
Tyler Goodlet d094625bd6 Activate docker daemons via flags using exit stack 2023-05-23 14:16:08 -04:00
Tyler Goodlet e7a172b656 Reimplement marketstore and elasticsearch daemons
Using the new `._ahab.start_ahab_service()` mngr of course, and now
support user config overrides (such that our defaults can be modified by
a keen user, say using a config file, or for testing). This is where the
functionality moved out of the `pikerd` init has been moved - instead of
being triggered by bool flag inputs to that factory.

For marketstore actually support overriding the entire yaml config via
runtime `_yaml_config_str: str` formatting with any passed user dict,
primarily focussing on supporting override of the sockaddrs for testing.
2023-05-23 14:16:02 -04:00
Tyler Goodlet bd919f9d66 _ahab: use `Services` api to spawn docker tasks
Allows for using the `Services.cancel_service()` api for explicit
cancellation in tests and eventually for remote teardown. Change
`.start_ahab()` to an `@acm` `start_ahab_service()` and just yield back
the same values we were returning prior. Also fix the logging (level) to
actually reflect what's passed in - we weren't using the correct name
/ instance from the `.sevice` subpkg..
2023-05-23 14:16:02 -04:00
Tyler Goodlet 611d1ee3fc Drop db flags from pikerd startup 2023-05-23 14:16:02 -04:00
Tyler Goodlet 56b23e1fcc Add docker and elasticsearch to test deps 2023-05-23 14:16:02 -04:00
Tyler Goodlet d3bafb0063 Always prefer a config template if found 2023-05-23 14:16:02 -04:00
Tyler Goodlet 7f246697b4 Remove remaining `fqsn` usage from code base minus backward compats 2023-05-23 14:16:02 -04:00
Tyler Goodlet dd10acbbf9 Replace `Transaction.fqsn` -> `.fqme`
Change over all client (broker) code which constructs transactions
and finally wipe required `.fqsn` usage from `.accounting` B)
2023-05-23 14:15:57 -04:00
Tyler Goodlet 31a00eca94 Rename fqsn -> fqme in ui mods 2023-05-22 12:13:00 -04:00
Tyler Goodlet c93d119873 Move tmpdir creation into separate fixture
Since `.config.load()` was changed to not touch conf files by default
(without explicitly setting `touch_if_dne: bool`), this ensures both the
global module value is set and the `brokers.toml` file exists before
every test.
2023-05-22 12:03:32 -04:00
Tyler Goodlet 588770d034 ib: rename lingering fqsn -> fqme 2023-05-22 12:00:13 -04:00
Tyler Goodlet 2f2d612b5f Add todo to switch to `dst/src` delim 2023-05-22 11:57:37 -04:00
Tyler Goodlet 660a94d610 Don't expect `conf.toml`'s network section
For testing this is particularly true until we offer a template
with whatever (likely localhost) settings planned to ship.
2023-05-22 11:54:36 -04:00
Tyler Goodlet e4e4cacef3 .data.feed: Less stringency with fqme matching
`Flume.mkt.fqme` might not be exactly the same as the local
version now since we've had to add some hacks to certain backends
(cough ib) to handle `MktPair.src` not being set as an `Asset` (yet).
2023-05-22 11:52:36 -04:00
Tyler Goodlet 60a6f3269c ib: use flex report datetime sort
Since `open_trade_ledger()` now requires a sort we pass in a combo of
the std `pendulum.parse()` for API records and a custom flex parser for
flex entries pulled offline.

Add special handling for `MktPair.src` such that when it's a fiat (like
it should always be for most legacy assets) we try to get the fqme
without that `.src` token (i.e. not mnqusd) to avoid breaking
roundtripping of live feed requests (due to new symbology) as well as
the current tsdb table key set..

Do a wholesale renaming of fqsn -> fqme in most of the rest of the
backend modules.
2023-05-22 09:41:44 -04:00
Tyler Goodlet 53003618cb Add longer timeout on brokerd ctx cancel; seems to work? 2023-05-22 00:16:58 -04:00
Tyler Goodlet c6da09f3c6 Add fast(er), time-sorted ledger records
Turns out that reading **and** writing with `tomlkit` is just wayya slow
for large documents like ledger files so move to using the `tomli`
sibling pkg `tomli-w` which seems to much improve on the latency, though
obviously longer run we're likely going to want:
- a better algorithm for only back loading records using as little
  history as possible
- a different serialization format for production maybe something
  like apache parquet?

The only issue with using a non-style-preserving writer is that we don't
necessarily get TOML conf ordering for free (without first ordering it
ourselves), and thus this patch also adds much more general date-time
sorting machinery which is now **required** when using
`open_trades_ledger()` via a `tx_sort: Callable`. By default we now
provide `.accounting._ledger.iter_by_dt()` (exposed in the subpkg mod)
which conducts dynamic "datetime key detection" based parsing of records
based on a `parsers: dict[str, Callabe]` input table. The default should
handle most use cases including all currently supported live backends
(kraken, ib) as well as our paper engine ledger-records format.

Granulars:
- adjust `Position.iter_clears()` to use new `iter_by_dt(key=lambda ..)`
  signature.
- add `tomli-w` to setup and our `tomlkit` fork to requirements file.
- move `.write_config()` to bottom of class defn.
- fix closed pos popping to not error if pp was already popped..
2023-05-18 18:27:54 -04:00
Tyler Goodlet 89d24cfe33 Oof, fix closed position popping by fqme.. 2023-05-18 12:52:34 -04:00
Tyler Goodlet 8d7a9fa19e Make `MktPair.pair()` a meth, allow passing in a delim character 2023-05-18 12:01:30 -04:00
Tyler Goodlet a1a10676cd Go back to `tomllib` for ledger loading, it's wayy faster 2023-05-18 11:27:31 -04:00
Tyler Goodlet 97b2b25256 Avoid import cycle in clearing client 2023-05-18 01:25:04 -04:00
Tyler Goodlet b2bf0b06f2 ib.api: wholesale fqsn -> fqme renames 2023-05-17 16:56:04 -04:00
Tyler Goodlet 907eaa68cb Pass `mkt: MktPair` to `.open_history_client()`
Since porting all backends to the new `FeedInit` + `MktPair` + `Asset`
style init, we can now just directly pass a `MktPair` instance to the
history endpoint(s) since it's always called *after* the live feed
`.stream_quotes()` ep B)

This has a lot of benefits including allowing brokerd backends to have
more flexible, pre-processed market endpoint meta-data that piker has
already validated; makes handling special cases in much more straight
forward as well such as forex pairs from legacy brokers XD

First pass changes all crypto backends to expect this new input, ib will
come next after handling said special cases..
2023-05-17 16:52:15 -04:00
Tyler Goodlet 89e8a834bf Support fqme rendering *without* the src key
Since most (legacy) stock brokers design their symbology without
including the target exchange's source asset name - normally a fiat
currency like USD - this adds an option for rendering market endpoints
without that token for simpler use in backends for such brokers.

As an example IB doesn't expect a `mnq/usd.cme.ib` symbol and instead
presumes that since the CME lists all assets in USD then the source
asset is implied.

Impl details:
- add `MktPair.pair: str` which replaces `.key` as a better name.
- offer a `without_src: bool` to a new `.get_fqme()` getter method
  which will render everything the same minus the src token.
- expose the new flag through both the new `.get_fqme()` and
  `.get_bs_fqme()` methods and wrap those both under the original
  property names `.bs_fqme` and `.fqme`.
2023-05-17 16:47:15 -04:00