Commit Graph

4354 Commits (95ace5acb8c4d26f8f259cf9d83b4b8b1778e81b)

Author SHA1 Message Date
Tyler Goodlet e5fdb33e31 Port cache-`dict` search to new `rapidfuzz` api 2023-10-01 17:46:46 -04:00
Tyler Goodlet 81a8cd1685 binance: always load the `brokers.toml` file since default is `conf.toml` now 2023-10-01 17:37:09 -04:00
Tyler Goodlet a382f01c85 Move tsdb section to `service.tsdb.name` and get host from `.maddrs` 2023-10-01 17:23:39 -04:00
Tyler Goodlet 653348fcd8 Use `.service.find_service()` instead of of `tractor.find_actor()` in pape-eng 2023-10-01 16:10:37 -04:00
Tyler Goodlet e139d2e259 Set `registry_addrs` in CLI (click) context-config
Since `tractor` and our runtime internals is now moved to multihomed semantics,
do the same in the CLI / config entrypoints.

Also, try using the new `tractor.devx.maybe_open_crash_handler()` around
the `pikerd` CLI.
2023-10-01 15:42:31 -04:00
Tyler Goodlet 7258d57c69 Only warn on mismatched `open_registry()` input addrs
When a new (actor) caller opens the registry there are 2 possible cases:
1. - some task already opened the registry during init and set the global
  superset of registrar addrs that are expected to be used,
2. - some task after the init task opens with a subset of addrs.
3. - some task after init opens with a disjoint set - should be an error?

In the 2nd case we don't want to error since the may just not need to
know about other registrar (multi-homed) addrs and thus only needs
specific access - so only warn about the diff in that case. If the
caller is requesting some disjoint set then we still runtime raise.

Adjust `find_service()` to allow a null `registry_addrs` input in which
case we fail over to using whatever pre-set the `Registry.addrs` has;
makes it simple for actors that don't want/need to know about the global
registrar set for their actor tree. Also, always set pass
`tractor.find_actor(only_first=True)` (for now).
2023-10-01 15:36:17 -04:00
Tyler Goodlet 5d081a40d5 Port to new `parse_maddr()` name 2023-09-29 15:20:56 -04:00
Tyler Goodlet fcececce19 Move multi-addr parser mod to `tractor` 2023-09-29 14:33:15 -04:00
Tyler Goodlet b6ac6069fe Temporarily use crash handler around search CLI ep 2023-09-29 14:02:17 -04:00
Tyler Goodlet a98f5877bc ui._exec: use new `get_runtime_vars()` name 2023-09-28 12:31:24 -04:00
Tyler Goodlet 50ddef0985 data.feed: dynamically load `ui._search` mod for headless installs 2023-09-28 12:30:10 -04:00
Tyler Goodlet b1cde3df49 config: make `conf.toml` the default load target 2023-09-28 12:29:07 -04:00
Tyler Goodlet 57010d479d Support multi-homed service actors and multiaddrs
This commit requires an equivalent commit in `tractor` which adds
multi-homed transport server support to the runtime and thus the ability
ability to listen on multiple (embedded protocol) addrs / networks as
well as exposing registry actors similarly. Multiple bind addresses can
now be (bare bones) specified either in the `conf.toml:[network]`
section, or passed on the `pikerd` CLI.

This patch specifically requires the ability to pass a `registry_addrs:
list[tuple]` into `tractor.open_root_actor()` as well as adjusts all
internal runtime routines to do the same, mostly inside the `.service`
pkg.

Further details include:
- adding a new `.service._multiaddr` parser module (which will likely be
  moved into `tractor`'s core) which supports loading lib2p2 style
  "multiaddresses" both from the `conf.toml` and the `pikerd` CLI as
  per,
- reworking the `pikerd` cmd to accept a new `--maddr`/`-m` param that
  accepts multiaddresses.
- adjust the actor-registry subsys to support multi-homing by also
  accepting a list of addrs to its top level API eps.
- various internal name changes to reflect the multi-address interface
  changes throughout.
- non-working CLI tweaks to `piker chart` (ui-client cmds) to begin
  accepting maddrs.
- dropping all elasticsearch and marketstore flags / usage from `pikerd`
  for now since we're planning to drop mkts and elasticsearch will be an
  optional dep in the future.
2023-09-28 12:13:34 -04:00
Tyler Goodlet f94244aad4 Load `network` section from `conf.toml` for service-addr map 2023-09-28 12:04:24 -04:00
Tyler Goodlet 261c331602 Try using `.mkPoetryEnv` instead for devving (dont work yet..) 2023-09-22 16:39:54 -04:00
Tyler Goodlet 3b4a4db7b6 Muck with `develop.nix` to try and hack it with `poetry` venv, go py3.11 2023-09-22 16:39:54 -04:00
Tyler Goodlet ad59a581c7 symcache: passthrough `rapidfuzz.process.extract` kwargs 2023-09-22 15:56:49 -04:00
Tyler Goodlet c312f90c0c kucoin: port to using `rapidfuzz`
Just like the others but also flip to using a `Client.get_mkt_pairs()`
meth name for consistency across clients.
2023-09-22 15:55:19 -04:00
Tyler Goodlet 1a859bc1a2 kraken: drop now unused `rapidfuzz` import 2023-09-22 15:53:03 -04:00
Tyler Goodlet e9887cb611 binance: parse .expiry separate from .venue
Apparently they're being massive cucks and changing their futes pair
schema again now adding a `NEXT_QUARTER` contract type which we weren't
handling at all. The good news is falling back to an old symcache file
would have prevented this from crashing.

Add a new `FutesPair.expiry: str` field so that `.bs_fqme` can simply
call it during the summary FQME-ification output rendering..
2023-09-22 14:48:50 -04:00
Tyler Goodlet 0ba75df877 Add `data.match_from_pairs` fuzzy symbology scanner
A helper for scanning a "pairs table" that most backends should expose
as part of their (internal) symbology set using `rapidfuzz` over
a `dict[str, Struct]` input table.

Also expose the `data.types.Struct` at the subpkg top level.
2023-09-22 13:54:25 -04:00
Tyler Goodlet a97a0ced8c kraken: switch to `rapidfuzz` API 2023-09-21 19:49:10 -04:00
Tyler Goodlet 46d83e9ca9 deribit: switch to `rapidfuzz` API 2023-09-21 19:44:27 -04:00
Tyler Goodlet d4833eba21 binance: switch to `rapidfuzz` API 2023-09-21 19:44:06 -04:00
Tyler Goodlet 14f124164a ib: fix mktpair fallback table: use `Client._con2mkts` to translate..
Previously we were assuming that the `Client._contracts: dict[str,
Contract]` would suffice this directly, which obviously isn't true XD

Also,
- add the `NSE` venue to skip list.
- use new `rapidfuzz.process.extract()` lib API.
- only get con deats for non null exchange names..
2023-09-21 19:14:44 -04:00
Tyler Goodlet 05959eaf70 Always ensure symcache mkt pair entry is valid type 2023-09-19 15:56:47 -04:00
Tyler Goodlet 30d55fdb27 Add `--pdb` support to `piker search` 2023-09-13 12:13:56 -04:00
Tyler Goodlet 2c88ebe697 binance: implement `Client.search_symbols()` using `rapidfuzz`
Change the deats inside the method and have the `brokerd` search task
just call it as needed since we already do internal mem caching on the
lookup table.

APIs changed so we need to make some tweaks as per:
- https://github.com/maxbachmann/RapidFuzz/blob/main/api_differences.md
- https://github.com/maxbachmann/RapidFuzz/blob/main/api_differences.md#differences-in-processor-functions

The main motivation is to get better wheel pkging support (for nixos),
better impl in C++, and a more simply licensed dep.
2023-09-13 11:59:51 -04:00
Tyler Goodlet 4a180019f0 Swap out `fuzzywuzzy` for the newer `rapidfuzz` lib 2023-09-13 11:57:02 -04:00
Tyler Goodlet 4d274b16d8 Attempt to generate .uis deps free lock file
Since `poetry` doesn't seem to actually mark optional group deps as such
in the lock file (!?) manually generate a `poetry.lock` with the
optional groups commented out in the `pyproject.toml`; this is all in
an attempt at trying to make `poetry2nix` build without any UI components
which seem to be the source of much frustration without hacking on p2n
and/or nixpkgs repos..

Further drop all the old build system files including the
setup.py and requirements.txt files.
2023-09-07 14:17:01 -04:00
Tyler Goodlet 481618cc51 kraken: handle ws live trading API symbology
Of course I missed this first try but, we need to use the ws market pair
symbology set (since apparently kraken loves redundancy at least 3 times
XD) when processing transactions that arrive from live clears since it's
an entirely different `LTC/EUR` style key then the `XLTCEUR` style
delivered from the ReST eps..

As part of this:
- add `Client._altnames`, `._wsnames` as `dict[str, Pair]` tables,
  leaving the `._AssetPairs` table as is keyed by the "xname"s.
- Change `Pair.respname: str` -> `.xname` since these keys all just seem
  to have a weird 'X' prefix.
- do the appropriately keyed pair table lookup via a new `api_name_set:
  str` to `norm_trade_records()` and set is correctly in the ws live txn
  handler task.
2023-08-30 16:32:34 -04:00
Tyler Goodlet 778d26067d ib.api: return None on manual quote timeout 2023-08-30 14:56:11 -04:00
Tyler Goodlet e54c3dc523 TOSQUASH 9005335e18: pack empty dict on no flow 2023-08-29 08:45:45 -04:00
Tyler Goodlet ad37cfbe2f Break backfill loop on `end_dt < start_dt` 2023-08-29 08:43:14 -04:00
Tyler Goodlet 8369f557c7 TOSQUASH 2e6b1330f375c310ad: adding .dev / .ui groups 2023-08-25 18:07:15 -04:00
Tyler Goodlet 461764419d ib.api: always key `._contracts` with '.ib' suffix
So that pos msgs from the ems are correctly loaded..
2023-08-25 17:47:30 -04:00
Tyler Goodlet 1002ce1e10 kraken.broker: one last fix to `Position.cumsize`.. 2023-08-25 17:47:30 -04:00
Tyler Goodlet 546049b62f data.history: handle venue-closure gap edge case 2023-08-25 17:47:30 -04:00
Tyler Goodlet e9517cdb02 ib: handle commodity-contract trade records 2023-08-25 17:47:30 -04:00
Tyler Goodlet 2b8cd031e8 By default silence `Client.get_quote()` timeout errors unless caller specifies to raise 2023-08-25 17:47:30 -04:00
Tyler Goodlet 2e6b1330f3 Add `.ui` and `.dev` deps groups via `poetry` Bo
Since we eventually want to allow users to minimally deploy `pikerd`
service-tree (aka distributed cross host) installs, we need to offer
a "headless" deps group. Really this is just the core dep set minus Qt
and some aux search related libs (for now).

The new `.dev` group is for adding hacking and testing tools including
`xonsh` since that will eventually be our REPL of choice more then
likely B)

Oh, and fix the namespace path (was a typo) for the `ledger` CLI and
of course bump the lock file.
2023-08-25 17:47:28 -04:00
Tyler Goodlet 995d1534b6 Drop hard redraws for now 2023-08-25 13:33:59 -04:00
Tyler Goodlet 9d31941d42 order_mode: embedded `Order` maybe be in dict form.. 2023-08-25 13:33:59 -04:00
Tyler Goodlet a695208992 brokers._daemon: drop question-comment about enabling feed module 2023-08-25 13:33:59 -04:00
Tyler Goodlet fed89562dc Import crash handler mngr from `piker.toolz` 2023-08-25 13:33:59 -04:00
Tyler Goodlet 9005335e18 ib: pack empty `dict` on no flow entry 2023-08-25 13:33:59 -04:00
Tyler Goodlet c3f8b089be Drop `.service._ahab` from storage cli runtime mods 2023-08-25 13:33:59 -04:00
Tyler Goodlet 0068119a6d ib: use `asyncio.wait_for()` on ticker first quote; on 3.11 input coros are not allowed.. 2023-08-25 13:33:59 -04:00
Tyler Goodlet 94540ce1cf Pin tomlkit as a path dep for now 2023-08-25 13:13:29 -04:00
Tyler Goodlet ea9a5e524c Factor prefer wheels deps into new `ahot_overrides`
Makes it easier to pass the overrides to multiple p2n functions (like
hopefully `.mkPoetryEnv`). Also, add some commented attempts at using
`mkPoetryEnv` and todo list for "why", remove the `poetry` CLI main
point from the pyproject.toml, bump the poetry lock file.
2023-08-17 15:56:28 -04:00