Commit Graph

173 Commits (wkt/land_broadcast_consumers)

Author SHA1 Message Date
Gud Boi 30df6b83cf .data: isolate cached feed stream consumers
Keep provider root streams private to the cached `Feed` and give every
`maybe_open_feed()` caller, including the first owner, a local
subscription view with copied `Flume` descriptors.

Also,
- key cached resources by the normalized full symbol set and stream
  policy instead of only the first FQME.
- subscribe each `Feed.open_multi_stream()` provider relay and wait for
  all relays to start before yielding the merged channel.
- give each provider relay an independent send-channel clone so one
  provider EOC cannot terminate its siblings.
- document `Feed.pause()` and `Feed.resume()` as shared remote-provider
  controls rather than caller-local cursor operations.
- cover miss/hit ownership, multi-market provider mapping, relay
  fan-out, lexical teardown and cache-key separation.

Prompt-IO: ai/prompt-io/opencode/20260813T202402Z_12b0d66e_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-11 00:46:43 -04:00
Gud Boi 6dfc73d0bf .ib.feed: cache quote channel contexts
Replace the manual `_quote_streams` raw-channel registry with a
symbol-keyed `maybe_open_context()` resource whose lifetime extends to
the final quote consumer.

Give every `open_aio_quote_stream()` caller a lexical child from the
cached `LinkedTaskChannel` broadcaster so concurrent users receive the
same quotes and close independently.

Prompt-IO: ai/prompt-io/opencode/20260813T230744Z_ddac3553_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-11 00:46:43 -04:00
Gud Boi 78e7a91a6e Fix IB `MethodProxy` response routing
`open_client_proxy()` broadcast its one-slot linked channel to an
always-running status relay and an idle method receiver. Status
traffic advanced only the relay, so the first lazy contract lookup
for an uncached order raised `Lagged` before receiving its
response.

Give `relay_client_proxy_messages()` sole receive ownership. Route
tagged responses from concurrent caller tasks into their per-`mid`
slots while each proxy context retains its own channel and relay.

Deats,
- log stale responses with their original method name;
- warn when unresolved-request metadata exceeds its bounded limit;
- drop late responses after caller cancellation;
- preserve the reconnect design note and message-case rationale;
- accept clean channel EOF during proxy teardown;
- cover idle status, concurrency, cancellation and EOF paths.

Prompt-IO: ai/prompt-io/opencode/20260811T205826Z_06d5ea5d_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-11 00:44:55 -04:00
Gud Boi 1fa45dfa98 Fix endpoint config and daemon startup from PR94 review
Keep the shipped `conf.toml` defaults TCP-only for this `tractor`
pin, with an explicit single-transport note and commented UDS
alternatives. Preserve repeated Click `--maddrs` options and treat
a direct string as one endpoint.

Deats,
- Validate daemon bind requests against actual listeners, not
  registrar discovery endpoints; normalize tagged addresses and
  allow OS-assigned TCP ports. Explain registrar ownership and
  automatic client binding at the validation site.
- Resolve explicit or inherited registrar addresses independently
  of other daemon kwargs, consuming overrides before forwarding
  them and keeping discovery and startup on the same registrar.
- Default chart binding to `None` for absent or empty network
  config, and remove the unconditional Binance debugger pause.
- Defer the prematurely cherry-picked `test_datad_spawn` until
  its service API lands. Add offline endpoint/startup regressions
  and a real distinct-registrar/daemon lifecycle check.

Include the human-directed config and registrar/bind explanations
and the direct prompt provenance pair from the review fix pass.

Prompt-IO: ai/prompt-io/20261007T165453Z_bde2457b_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-10 14:20:52 -04:00
Gud Boi 1ba9bdbac0 Preserve exact `piker` logger namespaces
Use `str.removeprefix()` instead of character-set stripping so
subloggers such as `piker.paper` retain their requested namespace.

Also,
- drop stale parent-logger assertions from daemon startup paths
- keep logger setup calls without assigning assertion-only locals
- cover both logger factories and prefix-sensitive module names

Prompt-IO: ai/prompt-io/opencode/20260821T042058Z_b7aca4c0_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi d86f29d416 Update feed test fqmes to use `.<venue>` format
Switch all `test_feeds.py` parametrized fqmes from
legacy `<pair>.<broker>` to the current
`<pair>.<venue>.<broker>` schema (e.g.
`btcusdt.spot.binance`).

Deats,
- update binance fqmes: `btcusdt.binance` ->
  `btcusdt.spot.binance`, same for `ethusdt`.
- update kraken fqmes: `ethusdt.kraken` ->
  `ethusdt.spot.kraken`, `xbtusd.kraken` ->
  `xbtusd.spot.kraken`.
- update cross-broker set similarly.
- comment out old fqmes with `!TODO` to later
  validate raising on bad/legacy formats.

Also,
- reformat `if ci_env and not run_in_ci` to
  multiline style.
- reformat `pytest.skip()` msg to multiline.
- add `?TODO` for symbology helper fn.
- drop stray `await tractor.pause()` in
  `conftest.py`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 69a6edd6f8 Normalize registrar addrs with `wrap_address()`
Canonicalize legacy socket pairs and tagged tuples before storing
or comparing `Registry.addrs`, keeping existing config readable
when tractor emits protocol-tagged forms.

Also,
- broaden registry/runtime addr annotations for both wire shapes
- compare service fixture addrs through tractor's address codec
- use tractor's shared `--ll` pytest option instead of
  registering it twice
- cover mixed legacy/tagged registry state and `pikerd` startup

Prompt-IO: ai/prompt-io/opencode/20260820T163519Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi bbe9deaa62 Port service+tests to latest `tractor` APIs
Continue the `repair_tests`-branch mission (already merged
in this stack's ancestry, see f4c4f1e2 which ported
`conftest.py`) by fixing the remaining drift breakage vs.
`tractor` git `main`; without these NOTHING boots since the
`tractor.Address` port in 604e5fcf.

Deats,
- normalize reg addrs via `wrap_address()` in
  `open_pikerd()` before `.unwrap()`-ing; entries may be
  raw `tuple`s when passed in from (test) client code.
- port `check_for_service()` to `query_actor(regaddr=)`
  (was `arbiter_sockaddr=`) incl. its 2-tuple yield and
  the now-required `open_registry(addrs=)` arg.
- `wait_for_actor(registry_addr=)` + `.chan.raddr.unwrap()`
  raw-tuple compares in `test_runtime_boot` and
  `ensure_service()`.
- update `run_test_w_cancel_method()` for modern `tractor`
  cancel semantics: self-requested sub-service cancels are
  absorbed (no `ContextCancelled` raised to the opener) and
  single-exc groups collapse to a bare KBI.
- `RemoteActorError.boxed_type` (was `.type`) and
  `Position.cumsize` (was `.size`) renames in tests.
- bump the paper-EMS startup budget 9 -> 19s; it includes
  a live (kraken) symbology fetch so needs net headroom.
- woops, add the missing comma in `.deribit.api`'s
  `tractor.trionics` import tuple..

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
2026-10-07 03:10:41 -04:00
Gud Boi 22f166a5c6 Update `conftest.py` for `tractor` runtime API changes
Port test fixtures to match `tractor`'s updated
registry and channel APIs.

Deats,
- use `registry_addrs=[reg_addr]` (list) instead of
  `registry_addr=reg_addr` in `maybe_open_pikerd()`.
- `wait_for_actor()` now takes `registry_addr=`
  kwarg instead of `arbiter_sockaddr=`.
- access `portal.chan` (not `.channel`) and unwrap
  remote addr via `raddr.unwrap()`.
- yield `raddr._host`/`raddr._port` instead of
  tuple-indexing.
- drop random port generation; accept `reg_addr`
  fixture from `tractor`'s builtin pytest plugin.

Also,
- add `reg_addr: tuple` param to `open_test_pikerd()`
  fixture (sourced from `tractor._testing.pytest`).
- type-narrow `reg_addr` to `tuple[str, int|str]`.
- drop unused `import random`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Tyler Goodlet f5850fe5c2 Draft a gt-one-`.fqme`-in-txns/account-file test
To start this is just a shell for the test, there's no checking logic
yet.. put it as `test_accounting.test_ib_account_with_duplicated_mktids()`.
The test is composed for now to be completely runtime-free using only
the offline txn-ledger / symcache / account loading APIs, ideally we
fill in the activated symbology-data-runtime cases once we figure a sane
way to handle incremental symcache updates for backends like IB..

To actually fill the test out with real checks we still need to,
- extract the problem account file from my ib.algopape into the test
  harness data.
- pick some contracts with multiple fqmes despite a single bs_mktid and
  ensure they're aggregated as a single `Position` as well as,
  * ideally de-duplicating txns from the account file section for the
    mkt..
  * warning appropriately about greater-then-one fqme for the bs_mktid
    and providing a way for the ledger re-writing to choose the
    appropriate `<venue>` as the "primary" when the
    data-symbology-runtime is up and possibly use it to incrementally
    update the IB symcache and store offline for next use?
2026-01-07 13:41:07 -05:00
Tyler Goodlet 1a4f8fa76f Drop `open_pps()` from ems tests 2026-01-07 13:41:07 -05:00
Tyler Goodlet 3adbabcba6 Use `pytest` plugin now exposed by `tractor` 2026-01-06 22:27:58 -05:00
Tyler Goodlet ff81e57e73 Spurious first-draft of EG collapsing
Topically, throughout various (seemingly) console-UX-affecting or benign
spots in the code base; nothing that required more intervention beyond
things superficial. A few spots also include `trio.Nursery` ref renames
(always to something with a `tn` in it) and log-level reductions to
quiet (benign) console noise oriented around issues meant to be solved
long..

Note there's still a couple spots i left with the loose-ify flag because
i haven't fully tested them without using the latest version of
`tractor.trionics.collapse_eg()`, but more then likely they should flip
over fine.
2026-01-06 22:27:58 -05:00
Tyler Goodlet 838ddd6e79 Fix type-check assertion in ems test to use `is` 2026-01-06 21:43:59 -05:00
Tyler Goodlet 5d86d336f2 Parametrize account names for offline ledger tests 2023-08-03 17:28:08 -04:00
Tyler Goodlet 803f4a6354 Add first account cumsize test; known to fail Bo 2023-07-14 17:54:13 -04:00
Tyler Goodlet c780164f69 Fix test to use new `load_account()` location 2023-07-12 08:45:55 -04:00
Tyler Goodlet 3fcf44aa52 Skip marketstore docker tests, we're gonna drop it.. 2023-06-27 13:42:08 -04:00
Tyler Goodlet a4d16ec6ab Fix ems tests: add `.spot` venue token to fqme 2023-06-27 13:42:08 -04:00
Tyler Goodlet abd3cefd84 Parametrize ems service test to cancel with API and kbi 2023-05-28 14:28:56 -04:00
Tyler Goodlet 4f67ac0337 Change to new context-cancelled msg contents: pikerd is canceller 2023-05-26 17:16:43 -04:00
Tyler Goodlet 5f1d0fcb8c `tmpconfdir`: always assert brokers config created 2023-05-26 14:58:59 -04:00
Tyler Goodlet 40c5f39f0d conftest: be explicit about which config we touch 2023-05-26 14:42:09 -04:00
Tyler Goodlet 06cc3ac92c Tidy up ems tests as per some `ruff`in 2023-05-25 18:04:52 -04:00
Tyler Goodlet 9bc11d8dd9 Add basic config checking tests 2023-05-25 17:55:20 -04:00
Tyler Goodlet 738d0ca38b Rename db tests to test_docker_services 2023-05-24 12:30:57 -04:00
Tyler Goodlet e82f7f9012 Skip elasticsearch test for now, container build seems borked? 2023-05-23 22:39:38 -04:00
Tyler Goodlet cd55d027c4 Re-implement db tests using new ahab daemons
Avoids the really sloppy flag passing to `open_pikerd()` and allows for
separation of the individual docker daemon starts.

Also add a new `root_conf() -> Path` fixture which will open and load
the `dict` for the new root `conf.toml` file.
2023-05-23 14:16:08 -04:00
Tyler Goodlet c93d119873 Move tmpdir creation into separate fixture
Since `.config.load()` was changed to not touch conf files by default
(without explicitly setting `touch_if_dne: bool`), this ensures both the
global module value is set and the `brokers.toml` file exists before
every test.
2023-05-22 12:03:32 -04:00
Tyler Goodlet 12bfabf056 Expose `.accounting.unpack_fqme()` 2023-05-17 16:43:31 -04:00
Tyler Goodlet 88f3912b2d test_ems: doc out some remaining suites 2023-05-09 14:49:46 -04:00
Tyler Goodlet 1b50bff625 Error test harness if `--pdb` passed without `-s` 2023-05-09 14:49:26 -04:00
Tyler Goodlet adb62dc7b4 Port oustanding parts of codebase to `unpack_fqme()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet b810de3089 Rename fqsn -> fqme in feeds tests 2023-05-09 14:49:26 -04:00
Tyler Goodlet 02eb966a87 Rename ems test mod 2023-05-09 14:49:26 -04:00
Tyler Goodlet 146e0993a9 More explicit test mod docstring 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3b7579990b Link `tractor` debug mode to `pytest` --pdb flag 2023-05-09 14:49:26 -04:00
Tyler Goodlet 7de914d54c Fix bad-fqme test, adjust prices based on buy/sell 2023-05-09 14:49:26 -04:00
Tyler Goodlet 30af91a82c Rewrite order ctl tests as a parametrization
More or less a complete rework which allows passing a detailed
clearing/fills input and allows for *not* rebooting the runtime / ems
between each position check.

Some further enhancements:
- use (unit) fractional sizes to simulate both the more realistic and
  more "complex position calculation" case; since this is crypto.
- add a no-fqme-found test.
- factor cross-session/offline pos storage (pps.toml) checks into
  a `load_and_check_pos()` helper which does all entry loading directly
  from a provided `BrokerdPosition` msg.
- use the new `OrderClient.send()` async api.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 05a33ae634 Make default order size to decimal 2023-05-09 14:49:26 -04:00
Tyler Goodlet b619e4a82d WIP complete rework of paper engine tests
More or less we need to be able to audit not only simple "make trades
check pps.toml files" tests (which btw were great to get started!).

We also need more sophisticated and granular order mgmt and service
config scenarios,

- full e2e EMS msg flow verification
- multi-client (dis)connection scenarios and/or monitoring
- dark order clearing and offline storage
- accounting schema and position calcs detailing

As such, this is the beginning to "modularlizingz" the components needed
in the test harness to this end by breaking up the `OrderClient` control
flows vs. position checking logic so as to allow for more flexible test
scenario cases and likely `pytest` parametrizations over different
transaction sequences.
2023-05-09 14:49:26 -04:00
Tyler Goodlet d67031d9ab Ensure we set the test config dir in the root actor..
Not sure how this worked before but we need to also override the
`piker._config_dir: Path` in the root actor when running in `pytest`; my
guess is something in the old test suite was masking this problem after
the change to passing the dir path down through the runtime vars via
`tractor`?

Also this drops the ems related fixtures/factories since they're
specific enough to define in the clearing engine tests directly.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 96006b2422 Adjust tests to `.clearing._client.OrderClient` type 2023-05-09 14:49:26 -04:00
Tyler Goodlet 9f03484c4d Move all fqsn parsing and `Symbol` to new `accounting._mktinfo 2023-05-09 14:49:25 -04:00
Tyler Goodlet beb6544bad Start a new `.accounting` subpkg, move `.pp` contents there 2023-05-09 14:49:25 -04:00
Tyler Goodlet 8ceaa27872 Add ES client polling to ensure eventual connectivity.. 2023-03-09 18:42:24 -05:00
Tyler Goodlet 97290fcb05 Never drop root perms in test harness 2023-03-09 18:42:24 -05:00
Tyler Goodlet 9a00c45923 Add `log` fixture for easy test plugin 2023-03-09 18:42:24 -05:00
Tyler Goodlet 7cc9911565 Add connection poll loop to es test as well 2023-03-09 15:37:43 -05:00
Tyler Goodlet 79b0db4449 Pass a config `tmp_dir: Path` to the runtime when testing 2023-03-09 15:37:43 -05:00