Compare commits
No commits in common. "wkt/land_datad_history" and "main" have entirely different histories.
wkt/land_d
...
main
|
|
@ -1,143 +0,0 @@
|
||||||
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
|
|
||||||
|
|
||||||
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
|
|
||||||
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
|
|
||||||
|
|
||||||
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
|
|
||||||
|
|
||||||
**User-decided constraints:**
|
|
||||||
|
|
||||||
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
|
|
||||||
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
|
|
||||||
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
|
|
||||||
|
|
||||||
## Target topology
|
|
||||||
|
|
||||||
```
|
|
||||||
pikerd
|
|
||||||
├── datad.ib feed bus, shm writers, tsp history, symbol search
|
|
||||||
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
|
|
||||||
├── emsd
|
|
||||||
│ └── paperboi.ib (paper mode; opens its feed via datad)
|
|
||||||
└── samplerd
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key verified facts (load-bearing)
|
|
||||||
|
|
||||||
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
|
|
||||||
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
|
|
||||||
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
|
|
||||||
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
|
|
||||||
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
|
|
||||||
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
|
|
||||||
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
|
|
||||||
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
|
|
||||||
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
|
|
||||||
|
|
||||||
## Module decision
|
|
||||||
|
|
||||||
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
|
|
||||||
|
|
||||||
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
|
|
||||||
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
|
|
||||||
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
|
|
||||||
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
|
|
||||||
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
|
|
||||||
- `ib`: adjust existing groups so each includes `'api'`
|
|
||||||
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
|
|
||||||
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
|
|
||||||
|
|
||||||
**Gate**: full pytest green, zero behavior change.
|
|
||||||
|
|
||||||
## Stage 1 — introduce `datad` daemon machinery (additive)
|
|
||||||
|
|
||||||
New `piker/data/_daemon.py`:
|
|
||||||
|
|
||||||
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
|
|
||||||
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
|
|
||||||
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
|
|
||||||
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
|
|
||||||
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
|
|
||||||
|
|
||||||
Supporting edits:
|
|
||||||
|
|
||||||
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
|
|
||||||
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
|
|
||||||
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
|
|
||||||
|
|
||||||
**Gate**: suite green + new spawn test; nothing routes through datad yet.
|
|
||||||
|
|
||||||
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
|
|
||||||
|
|
||||||
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
|
|
||||||
|
|
||||||
`piker/clearing/_ems.py`:
|
|
||||||
|
|
||||||
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
|
|
||||||
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
|
|
||||||
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
|
|
||||||
|
|
||||||
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
|
|
||||||
|
|
||||||
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
|
|
||||||
|
|
||||||
## Stage 3 — feed layer hard cutover to datad
|
|
||||||
|
|
||||||
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
|
|
||||||
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
|
|
||||||
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
|
|
||||||
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
|
|
||||||
- `piker/brokers/core.py:33,175` (`symbol_search`)
|
|
||||||
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
|
|
||||||
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
|
|
||||||
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
|
|
||||||
|
|
||||||
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
|
|
||||||
|
|
||||||
## Stage 4 — caps-sec slimming, validation, ib client-id
|
|
||||||
|
|
||||||
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
|
|
||||||
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
|
|
||||||
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
|
|
||||||
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
|
|
||||||
|
|
||||||
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Risk register
|
|
||||||
|
|
||||||
| Risk | Sev | Mitigation |
|
|
||||||
|---|---|---|
|
|
||||||
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
|
|
||||||
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
|
|
||||||
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
|
|
||||||
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
|
|
||||||
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
|
|
||||||
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
|
|
||||||
|
|
||||||
## Verification (per stage gates above, plus end-to-end)
|
|
||||||
|
|
||||||
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
|
|
||||||
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
|
|
||||||
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
|
|
||||||
|
|
||||||
## Critical files
|
|
||||||
|
|
||||||
- **new** `piker/data/_daemon.py` — all datad machinery
|
|
||||||
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
|
|
||||||
- `piker/data/feed.py` — spawn cutover + actor-name assert
|
|
||||||
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
|
|
||||||
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
|
|
||||||
- `piker/data/validate.py` — `get_eps()` helper
|
|
||||||
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
|
|
||||||
- `piker/accounting/cli.py` — keyword-form dialog call
|
|
||||||
- `tests/test_services.py` — inverted + new assertions
|
|
||||||
|
|
@ -1,233 +0,0 @@
|
||||||
# Bounded IB recovery candidate
|
|
||||||
|
|
||||||
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
|
|
||||||
`/home/goodboy/repos/piker`. Base:
|
|
||||||
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
|
|
||||||
Changes are unstaged and uncommitted. No acceptance is implied.
|
|
||||||
|
|
||||||
## Patch selection and provenance
|
|
||||||
|
|
||||||
Keep these existing landing commits:
|
|
||||||
|
|
||||||
| Commit | Reason |
|
|
||||||
| --- | --- |
|
|
||||||
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
|
|
||||||
| `e8e0730a` | Correlate MethodProxy requests and responses |
|
|
||||||
| `7bef50b0` | One response reader, per-request waiters |
|
|
||||||
| `d1571ddf` | Propagate daemon-handler exceptions |
|
|
||||||
| `dc618229` | Isolate cached feed stream consumers |
|
|
||||||
|
|
||||||
Proposed separate implementation boundaries, in this order:
|
|
||||||
|
|
||||||
1. Back out the code and original test from
|
|
||||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
|
|
||||||
This restores the direct-owner `_quote_streams` implementation.
|
|
||||||
Retain the historical prompt records. Replace the removed
|
|
||||||
mocked-cache regression with the real-bridge restart reproducer.
|
|
||||||
This backout is not a claim that arbitrary concurrent direct
|
|
||||||
callers have safe broadcast ownership.
|
|
||||||
2. Backport **only** `piker/brokers/ib/feed.py` from
|
|
||||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
|
|
||||||
Retain its seven history-adapter regressions in
|
|
||||||
`tests/test_ib_history.py`. This includes reset serialization,
|
|
||||||
completion/result signalling and bounded cancellation/failure.
|
|
||||||
It does not include the original SHM/storage orchestration patch.
|
|
||||||
3. Backport **only** `piker/brokers/ib/feed.py` from
|
|
||||||
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
|
|
||||||
Retain `test_empty_history_preserves_request_boundary` and its
|
|
||||||
six cases. This preserves no-data request boundaries and wakes
|
|
||||||
the waiter. It does not include the original history-layer
|
|
||||||
endpoint/filtering changes. Add the real-bridge history recovery
|
|
||||||
case after these adapter changes are present.
|
|
||||||
4. Import the code and original regression from
|
|
||||||
`12b0d66e854247e92cb5a8668be71d816c72d856`.
|
|
||||||
Add a separate real-broadcaster/production-loop regression.
|
|
||||||
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
|
|
||||||
This is a local UX cleanup retaining the existing `None` result.
|
|
||||||
It does not backport the null-grouping/deduplication rewrite in
|
|
||||||
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
|
|
||||||
6. Record cross-boundary regression results and live qualification
|
|
||||||
evidence, distinguishing the known callback failure from passes.
|
|
||||||
|
|
||||||
All three imported source commits were authored by
|
|
||||||
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
|
|
||||||
and the full source SHA when committing each partial backport.
|
|
||||||
Use explicit partial-backport wording, not a whole-commit cherry
|
|
||||||
claim. New real-runtime tests are this session's additional work.
|
|
||||||
|
|
||||||
The current aggregate working diff contains these boundaries in
|
|
||||||
the same feed file. Staging that entire file would combine the
|
|
||||||
first three boundaries. Reconstruct/stage the original per-source
|
|
||||||
patches in the above order when preparing commits.
|
|
||||||
|
|
||||||
## Quote-cache history correction
|
|
||||||
|
|
||||||
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
|
|
||||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
|
|
||||||
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
|
|
||||||
They are equivalent replays, not different implementations.
|
|
||||||
|
|
||||||
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
|
|
||||||
change and retains direct ownership. The original qualification
|
|
||||||
did not validate the combination assembled for the landing.
|
|
||||||
|
|
||||||
After this candidate's edits, `piker/brokers/ib/feed.py` matches
|
|
||||||
the source at `1da9b760` byte-for-byte. Compared with the evaluated
|
|
||||||
tip, only the later schedule/reservation APIs are absent.
|
|
||||||
The EMS code patch has the same stable patch ID as the original
|
|
||||||
`12b0d66e` EMS code patch:
|
|
||||||
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
|
|
||||||
|
|
||||||
## Real-runtime evidence
|
|
||||||
|
|
||||||
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
|
|
||||||
an asyncio-infected child, the installed pinned Tractor bridge,
|
|
||||||
production Piker quote setup, and real IB/eventkit ticker events.
|
|
||||||
Only the external IB connection/input is replaced with replay data.
|
|
||||||
|
|
||||||
Before the quote-cache backout, cancellation after ticker delivery
|
|
||||||
raised `RemoteActorError[TrioTaskExited]` through
|
|
||||||
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
|
|
||||||
reported fatal path.
|
|
||||||
|
|
||||||
After the backout:
|
|
||||||
|
|
||||||
- Two quote generations finish; each subscription is cancelled,
|
|
||||||
each asyncio producer is done and each ticker handler removed.
|
|
||||||
- A second RPC repeats the sequence in the same actor successfully.
|
|
||||||
- The strict callback-cleanliness case still fails: Tractor's
|
|
||||||
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
|
|
||||||
exception handler even though the Trio consumer recovers.
|
|
||||||
|
|
||||||
The history integration case keeps `Client.bars()`, MethodProxy,
|
|
||||||
both relays, `get_bars()`, `wait_on_data_reset()` and
|
|
||||||
`data_reset_hack()` real. It replaces external contract discovery,
|
|
||||||
IB history replies and VNC input. A real IB error event announces
|
|
||||||
farm recovery. The replay rejects retries before recovery and
|
|
||||||
proves one reset followed by completion for both 1s and 60s data.
|
|
||||||
|
|
||||||
The new EMS case runs the production dark-clearing loop over a
|
|
||||||
real Tractor broadcaster: both EMS and a sibling receive a tick,
|
|
||||||
EMS cancellation removes only its subscription, and the sibling
|
|
||||||
continues with an unchanged strict lag policy.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
Unique selected cases: **38 passed, 1 failed**.
|
|
||||||
The failure is intentionally visible, not skipped or xfailed.
|
|
||||||
|
|
||||||
Run from the checkout's working Nix environment:
|
|
||||||
|
|
||||||
```xsh
|
|
||||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
|
|
||||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
|
|
||||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
|
|
||||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
The adapter/proxy/feed/EMS selection initially passed 21 cases;
|
|
||||||
the added real EMS case then passed with its imported case.
|
|
||||||
The bridge file passed two cases and failed callback cleanliness.
|
|
||||||
Logging/startup passed 13 cases; runtime boot passed one with the
|
|
||||||
existing `Registrar.uid` deprecation warning.
|
|
||||||
|
|
||||||
The agent shell inherited another project's environment and lacked
|
|
||||||
native libraries. Tests used the existing Piker `py313` via
|
|
||||||
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
|
|
||||||
libraries from the pinned flake's existing Qt dependency closure,
|
|
||||||
excluding Nix Qt itself and glibc. No environment sync or shellHook
|
|
||||||
ran. Piker resolved to this checkout; Tractor resolved to its
|
|
||||||
installed package in `py313`.
|
|
||||||
|
|
||||||
Workflow-state preparation initially reported changed legacy
|
|
||||||
files/modes after migration. On 2026-10-23 the human authorized
|
|
||||||
reconciliation. Inspection found only two missing legacy guidance
|
|
||||||
snapshots; the neutral copies matched their saved hashes and modes.
|
|
||||||
Restoring those local snapshots resolved preparation without
|
|
||||||
changing archived messages or the original migration receipt.
|
|
||||||
|
|
||||||
## Human live qualification, 2026-10-23
|
|
||||||
|
|
||||||
The human ran these journeys with the installed Tractor pin,
|
|
||||||
the existing IB Gateway and pre-existing NativeDB. The agent did
|
|
||||||
not operate the broker connection, VNC, orders or database.
|
|
||||||
|
|
||||||
- Initial QQQ startup/backfill recovered after pacing error 162
|
|
||||||
with one reset attempt, then displayed live updates. The reset
|
|
||||||
waiter appears to have timed out; the log did not establish an
|
|
||||||
acknowledged farm reset or live quote-stream restart.
|
|
||||||
- Sequential chart exits and reattachment preserved all four
|
|
||||||
daemon PIDs; a read-only process/listener check confirmed this.
|
|
||||||
- The overlapping two-chart journey used `pikerd@961108`,
|
|
||||||
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
|
|
||||||
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
|
|
||||||
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
|
|
||||||
then C closed at 08:18:37. The human reported the exercise worked;
|
|
||||||
survivor updates are a manual observation, while timestamps and
|
|
||||||
unchanged daemon identities corroborate overlap and service reuse.
|
|
||||||
- Startup in that journey hit the explicit single-null-sample
|
|
||||||
breakpoint several times before the charts overlapped. The small
|
|
||||||
debugger cleanup above was applied afterward; this live run does
|
|
||||||
not itself verify that later edit.
|
|
||||||
- Departing charts still report cancellation groups and an FSP
|
|
||||||
receiver-close `ClosedResourceError`, followed by the Qt close
|
|
||||||
handler's self-SIGINT. Peer-disconnected final result warnings
|
|
||||||
remain contained in the persistent services.
|
|
||||||
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
|
|
||||||
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
|
|
||||||
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
|
|
||||||
After the human continued, all three children were reported
|
|
||||||
exited with status 0 and the root terminated at 08:19:45.
|
|
||||||
Cancellation-only RPC error reporting and late cancel-ack
|
|
||||||
diagnostics also remain.
|
|
||||||
|
|
||||||
The human explicitly accepted deferring this daemon teardown fix
|
|
||||||
from #94 and requested loud attribution for the bounded backports.
|
|
||||||
This is an operational smoke result, not clean-shutdown coverage.
|
|
||||||
|
|
||||||
## Downstream reconciliation requirements
|
|
||||||
|
|
||||||
Before landing #94, the human wants an easy replay of the downstream
|
|
||||||
histories intended to follow it. Rehearse against the agreed #94
|
|
||||||
tree in disposable worktrees, then inspect range-diffs, final code
|
|
||||||
and relevant regressions. This note does not authorize rewriting
|
|
||||||
the original branches or changing acceptance markers.
|
|
||||||
|
|
||||||
- `0846cbd4` and `1da9b760` are partial backports. Their imported
|
|
||||||
adapter hunks may already be present, but their history/SHM/storage
|
|
||||||
changes and omitted integration tests remain necessary downstream.
|
|
||||||
Never drop either whole source commit merely because `feed.py`
|
|
||||||
matches or its adapter hunk becomes empty.
|
|
||||||
- `12b0d66e` has source-equivalent EMS code and its imported test,
|
|
||||||
plus an additional local real-broadcaster test. Preserve that extra
|
|
||||||
regression when resolving test-file overlap.
|
|
||||||
- The quote-cache backout intentionally restores direct ownership
|
|
||||||
for this landing. Reintroducing original `4af885d9` downstream
|
|
||||||
requires qualifying cached-owner teardown with the matching
|
|
||||||
Tractor revision; the backout is not a permanent ownership design.
|
|
||||||
- The local single-null breakpoint removal retains legacy behavior.
|
|
||||||
Prefer the full downstream `976db6c3` algorithm when it is replayed;
|
|
||||||
preserving this `return None` guard would lose that later fix.
|
|
||||||
- Real-bridge tests and manual qualification notes are local work,
|
|
||||||
not evidence that omitted portions of the source commits landed.
|
|
||||||
|
|
||||||
Commit messages must name full source SHAs, original authorship,
|
|
||||||
imported paths/tests, omissions, local additions and replay guidance.
|
|
||||||
Use explicit partial-backport language for adapter subsets; do not
|
|
||||||
claim a full cherry-pick or reuse the source's broader fix claims.
|
|
||||||
|
|
||||||
## Remaining decision
|
|
||||||
|
|
||||||
This candidate removes the reproduced fatal quote-restart
|
|
||||||
propagation and imports existing reset fixes. The human exercised
|
|
||||||
live QQQ startup, backfill and concurrent chart consumer lifetimes.
|
|
||||||
Completion-callback and teardown errors remain; the human accepts
|
|
||||||
them as follow-ups outside the bounded #94 landing.
|
|
||||||
The storage/SHM portions of the downstream fixes are not covered by
|
|
||||||
the adapter backport or its integration tests.
|
|
||||||
|
|
||||||
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
|
|
||||||
`1a2d4893` stays a separate follow-up. Further Tractor teardown
|
|
||||||
changes or storage imports require reassessing the bounded landing
|
|
||||||
rather than silently expanding this candidate.
|
|
||||||
|
|
@ -1,71 +0,0 @@
|
||||||
# IB recovery qualification, 2026-11-06
|
|
||||||
|
|
||||||
This records human-operated qualification of the bounded #94
|
|
||||||
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
|
|
||||||
the narrow history catch described below. It supplements the
|
|
||||||
historical evidence in `ai/ib-recovery-candidate.md`.
|
|
||||||
|
|
||||||
## Runtime and dependency source
|
|
||||||
|
|
||||||
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
|
|
||||||
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
|
|
||||||
local checkout during evaluation. The human subsequently published
|
|
||||||
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
|
|
||||||
remote query confirmed the published OID before restoring Piker's
|
|
||||||
Git source and recording the same OID in `uv.lock`.
|
|
||||||
|
|
||||||
Offline lock validation resolves 126 packages successfully. No
|
|
||||||
environment sync was performed. This validates the lock metadata;
|
|
||||||
it does not claim a fresh installation from the published source.
|
|
||||||
|
|
||||||
## Manual observations
|
|
||||||
|
|
||||||
The human operated Gateway, chart windows and paper orders. The
|
|
||||||
agent did not operate live services or the account.
|
|
||||||
|
|
||||||
- Same-feed overlap, close/reopen and paper order submit/cancel
|
|
||||||
worked after the existing order-dialog guard was imported.
|
|
||||||
- Different-feed overlap originally exposed a timed-out one-second
|
|
||||||
MNQ March 2027 gap query. PDB showed `timedout=True` and
|
|
||||||
`out is None`. Its uncaught `NoData` canceled sibling history and
|
|
||||||
quote tasks even though `pikerd` remained alive.
|
|
||||||
- After the narrow catch, the human reported the overlap journeys
|
|
||||||
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
|
|
||||||
live feeds survived the Gateway data reset; closing all charts
|
|
||||||
and reopening with `pikerd` left running restored live updates.
|
|
||||||
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
|
|
||||||
that the intended Gateway data reset was triggered.
|
|
||||||
|
|
||||||
These are manual observations, not automated timing or latency
|
|
||||||
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
|
|
||||||
labels it a connection reset.
|
|
||||||
|
|
||||||
## Narrow history backport
|
|
||||||
|
|
||||||
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
|
|
||||||
`maybe_fill_null_segments()` is copied from
|
|
||||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
|
|
||||||
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
|
|
||||||
remains in place; this does not import broader history repair.
|
|
||||||
|
|
||||||
The new regression uses real null detection and checks that a
|
|
||||||
sibling in the same nursery still runs after either provider
|
|
||||||
exception. Both cases failed before the catch. With it, the two
|
|
||||||
focused history files passed: 17 tests. Earlier bounded-recovery
|
|
||||||
and Tractor qualification results remain separate evidence, not
|
|
||||||
fresh test runs of the final dependency commit tree.
|
|
||||||
|
|
||||||
## Remaining recovery limits
|
|
||||||
|
|
||||||
A full Gateway restart was triggered accidentally through its
|
|
||||||
auto-restart settings dialog. The disconnect delivered the string
|
|
||||||
`Peer closed connection.` to `deliver_trade_events()`. The string
|
|
||||||
branch then reached `err['reqid']`, raising `TypeError`; its `code`
|
|
||||||
local can also retain an earlier event's value. Full Gateway restart
|
|
||||||
recovery is outside this bounded landing.
|
|
||||||
|
|
||||||
The earlier history-failure teardown also exposed a separate
|
|
||||||
Tractor completion-callback assertion after `BrokenResourceError`.
|
|
||||||
The history catch does not establish that this callback race is
|
|
||||||
fixed. Broader history recovery and these disconnect paths remain
|
|
||||||
downstream work.
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-07T16:54:53Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: gpt-6.1-sol
|
|
||||||
git_ref: bde2457b
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user requested fixing all seven local PR94 review findings and
|
|
||||||
exporting the review for human inspection in Tuicr. They asked that a
|
|
||||||
Tractor pin adjustment be considered before adding Piker workarounds.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Correct the endpoint defaults, Click argument shape, daemon binding and
|
|
||||||
registrar option handling, chart fallback, and stray debugger pause.
|
|
||||||
Keep the absent datad implementation's test outside this landing
|
|
||||||
branch. Add offline startup coverage and export original findings
|
|
||||||
without modifying review states. The checked newer Tractor address
|
|
||||||
branch still rejects mixed transports, so leave the dependency pin.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `config/conf.toml`: TCP-only default endpoints.
|
|
||||||
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
|
|
||||||
- `piker/service/_actor_runtime.py`: validate daemon listeners.
|
|
||||||
- `piker/service/_daemon.py`: initialize and consume registrar options.
|
|
||||||
- `piker/ui/cli.py`: fallback to automatic chart binding.
|
|
||||||
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
|
|
||||||
- `tests/test_services.py`: defer test requiring missing datad API.
|
|
||||||
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human identified that datad tests arrived through the agent's
|
|
||||||
mixed cherry-picks and explicitly requested deferring that test. They
|
|
||||||
requested a chart default for existing configs, expanded the task to
|
|
||||||
all review findings, and directed checking whether a newer Tractor pin
|
|
||||||
could reduce the fix scope. No direct manual source edits were observed
|
|
||||||
during these changes; acceptance remains with the human.
|
|
||||||
|
|
||||||
During the human review pass, the human confirmed removing the Binance
|
|
||||||
pause and correcting CLI endpoint parsing. They requested an explicit
|
|
||||||
template note about this pin's single-transport limitation and detailed
|
|
||||||
comments distinguishing registrar discovery endpoints from daemon bind
|
|
||||||
endpoints. Those explanations were added at the human's direction.
|
|
||||||
|
|
@ -1,61 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-07T16:54:53Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: gpt-6.1-sol
|
|
||||||
git_ref: bde2457b
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Fix all findings from the local PR94 review, prefer a Tractor pin
|
|
||||||
change if it reduces work, and export the original review for Tuicr.
|
|
||||||
|
|
||||||
## Generated changes
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- config/conf.toml`
|
|
||||||
|
|
||||||
Use TCP-only defaults, retaining commented UDS alternatives. Both
|
|
||||||
checked Tractor address branch tips still reject mixed transports.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
|
|
||||||
|
|
||||||
Make Click multiaddresses repeatable and normalize single strings.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
|
|
||||||
|
|
||||||
Validate requested daemon bind addresses instead of remote registrars.
|
|
||||||
Normalize accepted endpoints and allow TCP ephemeral port requests.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
|
|
||||||
|
|
||||||
Resolve explicit or inherited registrars independently of other daemon
|
|
||||||
options and consume the explicit keyword before forwarding kwargs.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
|
|
||||||
|
|
||||||
Default chart bind endpoints to None when network config is absent.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
|
||||||
|
|
||||||
Remove the unconditional market-info debugger pause.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
|
|
||||||
|
|
||||||
Remove the prematurely cherry-picked datad test and add offline startup
|
|
||||||
regressions, including a real registrar/child daemon lifecycle check.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
- Startup and logging tests: 13 passed.
|
|
||||||
- Existing runtime boot and tagged registrar tests: 2 passed.
|
|
||||||
- Scoped native Ruff and diff checks pass.
|
|
||||||
- The first new actor test used a removed nursery API; it was corrected
|
|
||||||
to the installed start_actor/Portal.run interface before passing.
|
|
||||||
- Import initially failed without native libraries. Tests use the
|
|
||||||
existing flake's GCC, zlib, Qt and PyQt paths without executing its
|
|
||||||
environment-rebuilding shell hook.
|
|
||||||
- Original seven findings exported locally as Tuicr review-add payloads
|
|
||||||
bound to bde2457b; no session comments or remote PR were changed.
|
|
||||||
|
|
@ -1,45 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-10T23:28:52Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 1fa45dfa
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human rejected keeping the April Tractor pin as a landing baseline.
|
|
||||||
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
|
|
||||||
new rollback adaptations, reuse of original Piker histories where
|
|
||||||
possible, and a candidate prepared within their gateway relocation
|
|
||||||
window. They asked specifically whether the runtime validation review
|
|
||||||
hunk duplicated older endpoint history.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Select the reviewed address branch at an immutable locked revision and
|
|
||||||
preserve all committed application/test compatibility code. Verify the
|
|
||||||
minimal dependency boundary offline and preview existing consumer
|
|
||||||
fixes without importing unrelated datad/backfill changes. Distinguish
|
|
||||||
original-source test results from live teardown acceptance, which is
|
|
||||||
not available in this preparation window.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
|
|
||||||
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
|
|
||||||
preserving Polars 1.36.0 and the rest of the locked graph.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human redirected the task from an older known-good runtime to a
|
|
||||||
modern landing stack and insisted on reusing original history rather
|
|
||||||
than building new compatibility adapters. They proposed stricter
|
|
||||||
exception-group behavior as a hypothesis and set the live-test/time
|
|
||||||
constraints. Those directions materially changed the dependency
|
|
||||||
boundary. No direct manual edits to this final boundary were observed;
|
|
||||||
runtime acceptance and landing remain human-owned.
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-10T23:28:52Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 1fa45dfa
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Undo the April Tractor rollback and select the reviewed
|
|
||||||
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
|
|
||||||
commits to new adaptations, and prepare an offline-verified candidate
|
|
||||||
while live IB verification is deferred for gateway relocation.
|
|
||||||
|
|
||||||
## Generated changes
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
|
||||||
|
|
||||||
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
|
|
||||||
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
|
|
||||||
and test sources byte-for-byte to the committed PR94 tree. Keep the
|
|
||||||
rejected rollback journal only in ignored local experiment reports.
|
|
||||||
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
|
|
||||||
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
|
|
||||||
the resulting graph. Actual imports confirm the reviewed Tractor pin
|
|
||||||
and Trio 0.32.0, not the hypothesized Trio 0.33.
|
|
||||||
|
|
||||||
## Original-history previews
|
|
||||||
|
|
||||||
The existing MethodProxy stack is retained. Read-only merge previews
|
|
||||||
show original `4af885d9` and `b691422b` can be applied without the datad
|
|
||||||
cutover. Original-source regressions run through temporary in-memory
|
|
||||||
overlays against the real pinned runtime: seven cases pass and one
|
|
||||||
cache-hit case fails because `log.at_least_level('info')` looks up INFO
|
|
||||||
in a custom-level mapping. A separate standard logging correction
|
|
||||||
produces eight passing cases. Its patch and execution handoff are saved
|
|
||||||
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
|
|
||||||
cherry-picks or source edits are performed by this boundary.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
- Startup and logging: 13 passed.
|
|
||||||
- Runtime boot and tagged registrar normalization: 2 passed.
|
|
||||||
- Locked sync and source whitespace checks pass.
|
|
||||||
- Original application/test trees are unchanged from PR94 HEAD.
|
|
||||||
- Consumer overlay with explicit logging correction: 8 passed.
|
|
||||||
- Correction patch validates against its exact preview tree using a
|
|
||||||
private temporary index; the real index is untouched.
|
|
||||||
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.
|
|
||||||
|
|
@ -1,45 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
|
||||||
generated_by:
|
|
||||||
harness: codex
|
|
||||||
provider: OpenAI
|
|
||||||
git_ref: 495c0915
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
User reported a different-feed overlap failure and asked whether
|
|
||||||
a fix could be extracted without importing broader history work.
|
|
||||||
PDB confirmed MNQ March 2027 one-second history timed out and
|
|
||||||
returned None, producing NoData. After continuing PDB, sibling
|
|
||||||
history and quote tasks terminated while pikerd remained alive.
|
|
||||||
User requested continuing with the narrow backport.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Partial backport of the NoData/DataUnavailable catch from
|
|
||||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
|
||||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
|
||||||
A new parametrized regression uses real null detection and verifies
|
|
||||||
that a sibling in the same nursery survives the failed repair.
|
|
||||||
|
|
||||||
Both new cases failed before the catch. The focused history suites
|
|
||||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
|
||||||
IB reset churn or the separate Tractor completion-callback assertion.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/tsp/_history.py`: existing downstream exception catch.
|
|
||||||
- `tests/test_history_backfill.py`: sibling survival regression.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
User supplied live reproduction, PDB locals and cancellation logs,
|
|
||||||
and explicitly constrained scope to a small history extraction.
|
|
||||||
The production catch is copied from existing downstream commit
|
|
||||||
0846cbd4, not newly designed here. New regression is agent-authored;
|
|
||||||
no direct human source edits observed during this patch. Review and
|
|
||||||
live qualification remain with the user.
|
|
||||||
|
|
@ -1,22 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
|
||||||
generated_by:
|
|
||||||
harness: codex
|
|
||||||
provider: OpenAI
|
|
||||||
git_ref: 495c0915
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Partial backport of the NoData/DataUnavailable catch from
|
|
||||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
|
||||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
|
||||||
A new parametrized regression uses real null detection and verifies
|
|
||||||
that a sibling in the same nursery survives the failed repair.
|
|
||||||
|
|
||||||
Both new cases failed before the catch. The focused history suites
|
|
||||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
|
||||||
IB reset churn or the separate Tractor completion-callback assertion.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
|
||||||
generated_by:
|
|
||||||
harness: codex
|
|
||||||
provider: OpenAI
|
|
||||||
git_ref: 495c0915
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human requested a commit plan, then a fast-forward landing of
|
|
||||||
the #94 branch and PR updates. They agreed the temporary local
|
|
||||||
Tractor source must be replaced, published piker_pin themselves,
|
|
||||||
and reported "pushed it". They also supplied the manual recovery
|
|
||||||
results and full Gateway restart failure diagnostics.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Restore the Git Tractor source after local qualification and lock
|
|
||||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
|
||||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
|
||||||
Record the human's overlap, data-reset and reopen observations,
|
|
||||||
the narrow history extraction, and full-restart recovery limits.
|
|
||||||
Offline uv lock validation passed without an environment sync.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
|
|
||||||
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
|
|
||||||
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human selected the bounded landing scope, performed live tests,
|
|
||||||
identified the more aggressive reset distinction, and published
|
|
||||||
the tested Tractor ref. The agent applied the metadata edits and
|
|
||||||
wrote the qualification record from those observations. No human
|
|
||||||
acceptance or task-state transitions are inferred by this record.
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
|
||||||
generated_by:
|
|
||||||
harness: codex
|
|
||||||
provider: OpenAI
|
|
||||||
git_ref: 495c0915
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- uv.lock`
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
|
|
||||||
|
|
||||||
Restore the Git Tractor source after local qualification and lock
|
|
||||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
|
||||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
|
||||||
Record the human's overlap, data-reset and reopen observations,
|
|
||||||
the narrow history extraction, and full-restart recovery limits.
|
|
||||||
Offline uv lock validation passed without an environment sync.
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
# AI Prompt I/O Log
|
|
||||||
|
|
||||||
This directory records substantive AI-assisted changes, prompts,
|
|
||||||
original harness identity, and unedited outputs. Human contributors
|
|
||||||
remain accountable for code decisions.
|
|
||||||
|
|
||||||
Records follow the [NLNet generative AI policy][policy]. Each
|
|
||||||
structured entry links its immutable `.raw.md` partner. Code output
|
|
||||||
uses Git diff references to avoid duplication.
|
|
||||||
|
|
||||||
[policy]: https://nlnet.nl/foundation/policies/generativeAI/
|
|
||||||
|
|
@ -1,59 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:08:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Session-initiating instruction (driving all 7 commits in
|
|
||||||
this series):
|
|
||||||
|
|
||||||
> ok i want you to become the distributed runtime and
|
|
||||||
> concurrency expert for this project - namely acquire
|
|
||||||
> a deep understanding of tractor and how it's used.
|
|
||||||
> then i want you to attempt to factor our current
|
|
||||||
> brokerd service daemon into 2 daemons: a brokerd
|
|
||||||
> which only hosts live/paper trading endpoint tasks
|
|
||||||
> [..] a new `datad` subdaemon which in a separate
|
|
||||||
> subactor serves all the data feed service tasks [..]
|
|
||||||
> give me a mega detailed plan on how to approach this,
|
|
||||||
> and a staged approach for the implementation.
|
|
||||||
|
|
||||||
Proximate driver for THIS commit: during the approved
|
|
||||||
plan's stage-0 test gate the agent discovered test
|
|
||||||
subactors were writing into the user's REAL
|
|
||||||
`~/.config/piker/accounting/` files (a bogus paper fill
|
|
||||||
landed in `account.kraken.paper.toml`) because the old
|
|
||||||
test-dir override in `config.get_app_dir()` was
|
|
||||||
commented out and could never work in spawned
|
|
||||||
subactors anyway. Fix applied autonomously under the
|
|
||||||
approved plan; no explicit per-fix user prompt.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Restore `pytest` config-dir isolation by resolving the
|
|
||||||
per-test tmp dir lazily (at conf-path access time) from
|
|
||||||
`tractor` runtime-vars, which propagate down the actor
|
|
||||||
tree; route all conf-path derivation through
|
|
||||||
`config.get_conf_dir()` so the override is effective in
|
|
||||||
every (sub)actor.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/config.py` — add `_maybe_use_test_dir()`;
|
|
||||||
hook in `get_conf_dir()`; route `get_conf_path()` +
|
|
||||||
`load()` mkdir through it
|
|
||||||
- `piker/accounting/_ledger.py` — derive ledger dir
|
|
||||||
via `config.get_conf_dir()` (not the global)
|
|
||||||
- `piker/accounting/_pos.py` — same for account files
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:08:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/config.py`
|
|
||||||
|
|
||||||
Generated: `config._maybe_use_test_dir()` — lazily
|
|
||||||
reads `piker_test_dir` from
|
|
||||||
`tractor.runtime._state._runtime_vars['piker_vars']`
|
|
||||||
(pre-loaded by `open_piker_runtime()` from the
|
|
||||||
`tests.conftest._open_test_pikerd()` overrides) and
|
|
||||||
calls `_override_config_dir()` when set. Hooked at the
|
|
||||||
top of `get_conf_dir()`; `get_conf_path()` and
|
|
||||||
`load()`'s dir-creation rerouted through
|
|
||||||
`get_conf_dir()`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/_pos.py`
|
|
||||||
|
|
||||||
Generated: ledger/account dir derivation switched from
|
|
||||||
the `config._config_dir` module global to
|
|
||||||
`config.get_conf_dir()` + `mkdir(parents=True,
|
|
||||||
exist_ok=True)` for nested tmp-dir creation.
|
|
||||||
|
|
||||||
Key diagnostic reasoning (verbatim from session):
|
|
||||||
|
|
||||||
The old (commented-out) `get_app_dir()` override gated
|
|
||||||
on `'pytest' in sys.modules` which can NEVER work in
|
|
||||||
spawned subactors (fresh procs, no pytest import); as
|
|
||||||
a result test `paperboi`/daemon actors were writing
|
|
||||||
into the user's REAL `~/.config/piker/accounting/`
|
|
||||||
files. Evidence: `account.kraken.paper.toml` gained a
|
|
||||||
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
|
|
||||||
(a test fill), and `trades_kraken_paper.toml` (252B)
|
|
||||||
contained only that fill. The resolution must be
|
|
||||||
checked lazily at config-path access time (NOT import
|
|
||||||
time) since sub-actors only receive runtime-vars once
|
|
||||||
their `tractor` runtime has fully booted.
|
|
||||||
|
|
@ -1,53 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:10:22Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
|
|
||||||
driver: the approved plan's per-stage test gates could
|
|
||||||
not run AT ALL — the branch base was broken vs.
|
|
||||||
`tractor` git `main` (`AttributeError: 'tuple' object
|
|
||||||
has no attribute 'unwrap'` at `pikerd` boot, stale
|
|
||||||
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
|
|
||||||
fixed forward autonomously to (re)establish the gate
|
|
||||||
baseline, continuing the `repair_tests` branch lineage
|
|
||||||
(verified already merged in ancestry via `git cherry`
|
|
||||||
during a user-requested branch-overlap survey).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Port the service layer + test suites to current
|
|
||||||
`tractor` APIs: addr-type normalization in
|
|
||||||
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
|
|
||||||
kwarg renames, modern self-cancel absorption semantics
|
|
||||||
in the cancel-method test harness, exc/position attr
|
|
||||||
renames, a paper-EMS startup-budget bump and a syntax
|
|
||||||
fix in `.deribit.api`.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/service/_actor_runtime.py` — `wrap_address()`
|
|
||||||
normalize before `.unwrap()` in `open_pikerd()`
|
|
||||||
- `piker/service/_registry.py` — `check_for_service()`
|
|
||||||
-> `query_actor(regaddr=)` + 2-tuple yield +
|
|
||||||
`open_registry(addrs=)`
|
|
||||||
- `piker/brokers/deribit/api.py` — missing comma in
|
|
||||||
`tractor.trionics` import tuple
|
|
||||||
- `tests/test_services.py` — `registry_addr=` kwarg,
|
|
||||||
raddr unwraps, cancel-semantics harness rewrite,
|
|
||||||
`fail_after` 9 -> 19s
|
|
||||||
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,56 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:10:22Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
|
||||||
|
|
||||||
Generated: normalize each registry addr via
|
|
||||||
`tractor.discovery._addr.wrap_address()` before
|
|
||||||
`.unwrap()`-ing for the `accept_addrs` bind check —
|
|
||||||
entries may be raw `tuple`s when passed in from (test)
|
|
||||||
client code. Import-path precedent taken from
|
|
||||||
`piker/cli/__init__.py:336`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_registry.py`
|
|
||||||
|
|
||||||
Generated: `check_for_service()` ported to
|
|
||||||
`tractor.query_actor(name, regaddr=...)` (kwarg was
|
|
||||||
`arbiter_sockaddr=`), unpacking the new
|
|
||||||
`(sockaddr, portal)` yield, and passing the
|
|
||||||
now-required `open_registry(addrs=Registry.addrs)`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_ems.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
|
|
||||||
|
|
||||||
Key diagnostic reasoning (verbatim from session):
|
|
||||||
|
|
||||||
- the "DID NOT RAISE ContextCancelled" failure: in this
|
|
||||||
test the client actor IS pikerd (in-proc), and
|
|
||||||
current `tractor` main absorbs a `ContextCancelled`
|
|
||||||
whose canceller is your own actor — self-requested
|
|
||||||
cancels now exit cleanly instead of raising; the
|
|
||||||
'sigint' variant propagates a bare collapsed
|
|
||||||
`KeyboardInterrupt` rather than a
|
|
||||||
`BaseExceptionGroup`.
|
|
||||||
- the hard-coded `trio.fail_after(9)` startup budget is
|
|
||||||
marginal — full stack boot (pikerd -> emsd ->
|
|
||||||
brokerd.kraken -> paperboi + live kraken symbology
|
|
||||||
fetch) occasionally exceeds 9s -> bumped to 19s.
|
|
||||||
- `RemoteActorError.type` -> `.boxed_type`;
|
|
||||||
`Position.size` -> `.cumsize` (the paper engine
|
|
||||||
populates `BrokerdPosition.size` from `pp.cumsize`).
|
|
||||||
- overlap survey (user-requested): all of the
|
|
||||||
`repair_tests` branch commits are already in this
|
|
||||||
stack's ancestry; this commit finishes that branch's
|
|
||||||
port mission (its f4c4f1e2 fixed `conftest.py`'s
|
|
||||||
`arbiter_sockaddr` usage; this fixes the remaining
|
|
||||||
`test_services.py` + `check_for_service()` sites).
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:11:05Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 0" prep. Plan-shaping user
|
|
||||||
decisions captured via in-session Q&A:
|
|
||||||
|
|
||||||
- `datad.<broker>` topology: sibling of
|
|
||||||
`brokerd.<broker>` under `pikerd` (vs. child
|
|
||||||
subactor).
|
|
||||||
- migration: hard cutover, staged by layer (no
|
|
||||||
dual-mode runtime flag).
|
|
||||||
- post-split `brokerd` scope: trading-only +
|
|
||||||
EMS-lazy-spawned (charts/CLI never touch it).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Declare per-daemon-kind backend submod groups
|
|
||||||
(`_datad_mods`/`_brokerd_mods`) in the split-style
|
|
||||||
backends, keyed to the pre-existing
|
|
||||||
`piker.data.validate._eps` contract; add a
|
|
||||||
`validate.get_eps()` introspection helper; ws-import
|
|
||||||
hygiene in `.kraken.broker`. Zero behavior change
|
|
||||||
(`__enable_modules__` unions unchanged).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/validate.py` — add `get_eps()`
|
|
||||||
- `piker/brokers/kraken/__init__.py` — mod groups
|
|
||||||
- `piker/brokers/binance/__init__.py` — mod groups
|
|
||||||
(note: no `symbols.py`; search eps live in `.feed`)
|
|
||||||
- `piker/brokers/deribit/__init__.py` — datad-only
|
|
||||||
groups (no `broker.py` yet)
|
|
||||||
- `piker/brokers/ib/__init__.py` — add `'api'` to the
|
|
||||||
pre-existing `_datad_mods`
|
|
||||||
- `piker/brokers/kraken/broker.py` — import
|
|
||||||
`NoBsWs`/`open_autorecon_ws` from
|
|
||||||
`piker.data._web_bs` directly
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:11:05Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/validate.py`
|
|
||||||
|
|
||||||
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
|
|
||||||
returning the daemon-kind's ep funcs defined by a
|
|
||||||
backend mod, keyed by ep name, missing eps excluded;
|
|
||||||
sourced from the existing `_eps` grouping table
|
|
||||||
(`'middleware' | 'datad' | 'brokerd'`).
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
|
|
||||||
|
|
||||||
Key exploration findings driving the design (from the
|
|
||||||
planning phase, 3 parallel explore agents + 1 architect
|
|
||||||
agent):
|
|
||||||
|
|
||||||
- the codebase already anticipated this split:
|
|
||||||
`validate._eps` groups eps into 'datad' vs 'brokerd'
|
|
||||||
kinds and `ib/__init__.py` already declared
|
|
||||||
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
|
|
||||||
carried the literal TODO "rename the daemon to datad
|
|
||||||
prolly once we split up broker vs. data tasks into
|
|
||||||
separate actors?".
|
|
||||||
- the feared kraken feed<->broker "shared ws state"
|
|
||||||
hazard is mild: `NoBsWs`/`open_autorecon_ws`
|
|
||||||
originate in `piker.data._web_bs` (re-exported via
|
|
||||||
`.kraken.feed`); only `stream_messages` is feed-local
|
|
||||||
and it's a pure parser — in-process imports only,
|
|
||||||
each actor opens its own ws conn.
|
|
||||||
- `enable_modules` gates RPC entry, NOT python imports,
|
|
||||||
so backend trading mods may keep importing feed-side
|
|
||||||
helpers in-process post-split.
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:11:42Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 1": introduce the `datad` daemon
|
|
||||||
machinery additively (nothing routes through it yet).
|
|
||||||
User-decided constraint applied: `datad.<broker>` is a
|
|
||||||
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
|
|
||||||
via the existing `Services` + `maybe_spawn_daemon()`
|
|
||||||
machinery.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
New `piker/data/_daemon.py` hosting the
|
|
||||||
`datad.<broker>` feed-only daemon-actor: lifetime
|
|
||||||
fixture owning the actor-global `_FeedsBus`, init/spawn
|
|
||||||
fns mirroring `.brokers._daemon` conventions and the
|
|
||||||
`samplerd` sub-daemon precedent, plus root-mod
|
|
||||||
registration, `piker.service` re-exports and a spawn
|
|
||||||
test.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/_daemon.py` — NEW:
|
|
||||||
`_setup_persistent_datad()`, `datad_init()`,
|
|
||||||
`spawn_datad()`, `maybe_spawn_datad()`,
|
|
||||||
`_datad_service_mods`
|
|
||||||
- `piker/service/_actor_runtime.py` — add
|
|
||||||
`piker.data._daemon` to `_root_modules`
|
|
||||||
- `piker/service/__init__.py` — re-export spawn eps
|
|
||||||
- `tests/test_services.py` — add `test_datad_spawn`
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:11:42Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/_daemon.py`
|
|
||||||
|
|
||||||
Generated symbols + key design decisions:
|
|
||||||
|
|
||||||
- `_datad_service_mods: list[str]` — datad-always
|
|
||||||
enabled mods, the data-side successor to the old
|
|
||||||
`piker.brokers._daemon._data_mods` set; kept minimal
|
|
||||||
per the caps-sec model.
|
|
||||||
- `_setup_persistent_datad()` — `@tractor.context`
|
|
||||||
lifetime fixture: console-log setup then allocates
|
|
||||||
the actor-global feed bus via
|
|
||||||
`feed.get_feed_bus(brokername, service_nursery)`
|
|
||||||
exactly as the old brokerd fixture did, pinned open
|
|
||||||
with `ctx.started()` + `sleep_forever()`.
|
|
||||||
- `datad_init()` — actor name `f'datad.{brokername}'`;
|
|
||||||
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
|
|
||||||
`infect_asyncio=True`); builds `enable_modules` from
|
|
||||||
`getattr(brokermod, '_datad_mods',
|
|
||||||
getattr(brokermod, '__enable_modules__', []))` as
|
|
||||||
the flat-backend fallback.
|
|
||||||
- `spawn_datad()` — `Services.actor_n.start_actor()` +
|
|
||||||
`Services.start_service_task()` exactly mirroring
|
|
||||||
`spawn_brokerd()`; dedup-composes enable mods via
|
|
||||||
`list(dict.fromkeys(...))`.
|
|
||||||
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
|
|
||||||
service_name=f'datad.{brokername}', ...)`.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
|
||||||
> `git log -1 -p --follow -- piker/service/__init__.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
|
|
||||||
Design rationale (verbatim from session):
|
|
||||||
|
|
||||||
- `_root_modules` must gain `piker.data._daemon` so
|
|
||||||
`pikerd_portal.run(spawn_datad, ...)` resolves in
|
|
||||||
the root.
|
|
||||||
- the `Services`-based impl style deliberately mirrors
|
|
||||||
`spawn_brokerd()` so the eventual `tractor.hilevel`
|
|
||||||
`ServiceMngr` port (see the `service_mng_to_tractor`
|
|
||||||
branch's d8c21d44 prep, surfaced by the
|
|
||||||
user-requested branch-overlap survey) lands
|
|
||||||
symmetrically on both spawn fns.
|
|
||||||
- mod placement (`piker/data/_daemon.py` vs.
|
|
||||||
generalizing `piker.brokers._daemon`) follows the
|
|
||||||
per-subsystem daemon-mod convention
|
|
||||||
(`.clearing._ems`, `.data._sampling`) and resolves
|
|
||||||
the existing TODO at `brokers/_daemon.py:49` ("move
|
|
||||||
this def to the `.data` subpkg").
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:12:26Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 2": decouple the clearing layer
|
|
||||||
from `feed.portals` BEFORE the feed cutover so live
|
|
||||||
trading works at every stage boundary. User-decided
|
|
||||||
constraint applied: post-split `brokerd` is
|
|
||||||
trading-only and spawned LAZILY only by `emsd`'s
|
|
||||||
`open_brokerd_dialog()` path.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Kill the single coupling forcing feed + trading eps
|
|
||||||
into one actor (`Router.open_trade_relays()` pulling
|
|
||||||
its trades portal from `feed.portals[brokermod]`):
|
|
||||||
`open_brokerd_dialog()` now (maybe) spawns/finds
|
|
||||||
`brokerd.<broker>` itself and ONLY when a live
|
|
||||||
trades-ep will actually open; paper mode never touches
|
|
||||||
it. Pre-cutover this is a pure refactor (registry
|
|
||||||
lookup finds the same feed-spawned daemon).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
|
|
||||||
re-sig + inner `acquire_live_portal()`;
|
|
||||||
`Router.maybe_open_brokerd_dialog()` drops `portal`
|
|
||||||
param; `open_trade_relays()` drops the
|
|
||||||
`feed.portals` lookup
|
|
||||||
- `piker/accounting/cli.py` — keyword-form `portal=`
|
|
||||||
override kept for the `piker ledger` ad-hoc actor
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:12:26Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/clearing/_ems.py`
|
|
||||||
|
|
||||||
Generated: new `open_brokerd_dialog()` signature
|
|
||||||
`(brokermod, exec_mode, fqme=None, portal=None,
|
|
||||||
loglevel=None)` with an inner `@acm
|
|
||||||
acquire_live_portal()` that yields the caller-provided
|
|
||||||
`portal` override (the `piker ledger` path) else
|
|
||||||
`maybe_spawn_brokerd(brokermod.name)` — designated THE
|
|
||||||
one place a live, credentialed `brokerd.<broker>` gets
|
|
||||||
booted post-split. The eager
|
|
||||||
`portal.open_context(trades_endpoint, ...)`
|
|
||||||
construction moved inside that block so the paper
|
|
||||||
short-circuit never acquires a live portal.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/accounting/cli.py`
|
|
||||||
|
|
||||||
Key analysis (verbatim from session):
|
|
||||||
|
|
||||||
- `feed.portals` had exactly ONE trading consumer:
|
|
||||||
`piker/clearing/_ems.py:671` (`portal =
|
|
||||||
feed.portals[brokermod]`) — the single coupling
|
|
||||||
forcing feed + trading into one actor.
|
|
||||||
- `piker ledger` (`accounting/cli.py:100-157`) is a
|
|
||||||
hidden consumer: it calls `broker_init()` directly,
|
|
||||||
spawns its own ad-hoc actor and passes the portal
|
|
||||||
positionally — the new signature must keep an
|
|
||||||
explicit `portal:` override param for this path.
|
|
||||||
- stage sequencing: clearing decouple lands BEFORE the
|
|
||||||
feed cutover, otherwise `feed.portals` would hand
|
|
||||||
emsd a `datad` portal and live `open_trade_dialog`
|
|
||||||
RPC would fail.
|
|
||||||
|
|
@ -1,48 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:12:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's "stage 3": the feed-layer hard cutover
|
|
||||||
(user-decided: no dual-mode runtime flag, each stage
|
|
||||||
lands fully working).
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The topology flip: all data-feed consumers route to
|
|
||||||
`datad.<broker>`; the `brokerd` lifetime fixture slims
|
|
||||||
to trading-only; UI/CLI data-flavoured spawn sites
|
|
||||||
repointed; test expectations inverted incl. an explicit
|
|
||||||
negative assert that paper sessions spawn NO
|
|
||||||
`brokerd.<broker>`.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/feed.py` — `open_feed()` ->
|
|
||||||
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
|
|
||||||
assert -> `'datad'`; `install_brokerd_search` ->
|
|
||||||
`install_datad_search`; comment sweep
|
|
||||||
- `piker/brokers/_daemon.py` —
|
|
||||||
`_setup_persistent_brokerd()` slimmed (feed-bus
|
|
||||||
alloc moved to the datad fixture)
|
|
||||||
- `piker/ui/_app.py`, `piker/ui/cli.py`,
|
|
||||||
`piker/brokers/core.py`, `piker/brokers/cli.py`,
|
|
||||||
`piker/ui/kivy/option_chain.py` — spawn-site
|
|
||||||
repoints
|
|
||||||
- `tests/test_services.py` — datad assertions + the
|
|
||||||
no-`brokerd.kraken` negative check
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:12:59Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/data/feed.py`
|
|
||||||
|
|
||||||
Generated: `open_feed()` spawn cutover with the
|
|
||||||
`maybe_spawn_datad` import done "relative-direct" from
|
|
||||||
`._daemon` — NOT via `piker.service` — to dodge a
|
|
||||||
partial-init cycle: when `piker.data.feed` loads as
|
|
||||||
part of `piker.service.__init__` executing its own
|
|
||||||
`piker.data._daemon` import, the service pkg is
|
|
||||||
mid-init and its `maybe_spawn_datad` binding does not
|
|
||||||
exist yet. The `open_feed_bus()` local-state sanity
|
|
||||||
assert flips `'brokerd' in servicename` ->
|
|
||||||
`'datad' in servicename` (the only actor-name assert
|
|
||||||
in the tree, verified by grep).
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
|
||||||
|
|
||||||
Generated: `_setup_persistent_brokerd()` slimmed to
|
|
||||||
console-log setup + pinned-open ctx; drops the bus
|
|
||||||
alloc, the `assert not feed._bus`, the service nursery
|
|
||||||
(backend `open_trade_dialog()` ctxs own their task
|
|
||||||
trees) and the `eg.ExceptionGroup` handler. The
|
|
||||||
`piker ledger` ad-hoc actor enters this same slimmed
|
|
||||||
fixture — exactly what it needs.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/ui/_app.py`
|
|
||||||
> `git log -1 -p --follow -- piker/ui/cli.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/core.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/cli.py`
|
|
||||||
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
|
|
||||||
> `git log -1 -p --follow -- tests/test_services.py`
|
|
||||||
|
|
||||||
Verification (from session): per-suite gates green at
|
|
||||||
this commit (services 5-passed incl. the new negative
|
|
||||||
assert, feeds 3-passed); a headless live smoke
|
|
||||||
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
|
|
||||||
on an alt registry port) confirmed quotes flowing via
|
|
||||||
`datad.kraken` + `samplerd` with `check_for_service(
|
|
||||||
'brokerd.kraken') is None`. Known pre-existing flake
|
|
||||||
documented: `test_multi_fill_positions`' second
|
|
||||||
in-proc runtime boot wedges ~50% (zombie subactor w/
|
|
||||||
unread parent-IPC bytes); reproduced with the split
|
|
||||||
fully reverted so NOT a regression of this work.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:13:44Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Same session-initiating `brokerd`-split instruction (see
|
|
||||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
|
||||||
approved plan's final "stage 4": caps-sec slimming of
|
|
||||||
the (live, credentialed) trading actor + the `ib`
|
|
||||||
dual-daemon `client_id` collision mitigation flagged in
|
|
||||||
the plan's risk register.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
|
|
||||||
spawn fails fast for datad-only backends with a "use
|
|
||||||
paper-mode" error; `ib`'s default api-gw `client_id`
|
|
||||||
gets a per-daemon-kind offset so `datad.ib` +
|
|
||||||
`brokerd.ib` don't collide on connect.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
|
|
||||||
`_brokerd_service_mods`; `broker_init()` reads
|
|
||||||
`_brokerd_mods` (fallback `__enable_modules__`);
|
|
||||||
`spawn_brokerd()` fail-fast via `validate.get_eps()`
|
|
||||||
- `piker/brokers/ib/api.py` — role-based `client_id`
|
|
||||||
offset in `load_aio_clients()`
|
|
||||||
- `piker/cli/__init__.py` — resolved "expose datad"
|
|
||||||
TODO
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:13:44Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); backfilled from the live dev
|
|
||||||
session transcript per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
|
||||||
|
|
||||||
Generated: the fail-fast originally landed in
|
|
||||||
`broker_init()` but was relocated to `spawn_brokerd()`
|
|
||||||
mid-implementation after realizing `piker ledger`
|
|
||||||
calls `broker_init()` directly even for paper accounts
|
|
||||||
on datad-only backends (would have crashed the cli);
|
|
||||||
the service-spawn path is the correct enforcement
|
|
||||||
seam. Error text:
|
|
||||||
|
|
||||||
Backend 'kucoin' offers NO `brokerd` (live
|
|
||||||
order-control) eps!? It is likely a datad-only
|
|
||||||
provider, use paper-mode for clearing instead.
|
|
||||||
|
|
||||||
(verified live via a `trio.run()` unit check.)
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
|
||||||
|
|
||||||
Generated: in `load_aio_clients()`, when `client_id`
|
|
||||||
is the 6116 default: `datad`-named actors offset +16
|
|
||||||
(disjoint from `brokerd`'s linear `client_id + i`
|
|
||||||
retry range), other non-`brokerd` (ad-hoc test/cli)
|
|
||||||
actors +32. Rationale from the plan's risk register:
|
|
||||||
post-split BOTH per-broker daemons connect to the same
|
|
||||||
TWS/gw endpoint; a shared default id collides and
|
|
||||||
burns up to `connect_timeout * retries` (90s) in
|
|
||||||
retry cycles.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/cli/__init__.py`
|
|
||||||
|
|
@ -1,66 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T17:33:09Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
|
||||||
scope: docs
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The session-initiating instruction, verbatim (this doc
|
|
||||||
IS its "mega detailed plan" deliverable):
|
|
||||||
|
|
||||||
> ok i want you to become the distributed runtime and
|
|
||||||
> concurrency expert for this project - namely acquire
|
|
||||||
> a deep understanding of tractor and how it's used.
|
|
||||||
> then i want you to atttempt to factor our current
|
|
||||||
> brokerd service daemon into 2 daemons:
|
|
||||||
>
|
|
||||||
> - a brokerd which only hosts live/paper trading
|
|
||||||
> endoint tasks as defined namely within all
|
|
||||||
> piker/brokers/<backend>/broker.py mods
|
|
||||||
> - a new `datad` subdaemon which in a separate
|
|
||||||
> subactor serves all the data feed service tasks
|
|
||||||
> namely delivered by endpoints in
|
|
||||||
> piker/brokers/<backend>/feed.py (or similar for
|
|
||||||
> less mature backends) and as more rigorously
|
|
||||||
> defined by the validation machinery in
|
|
||||||
> piker.data.validate.
|
|
||||||
>
|
|
||||||
> give me a mega detailed plan on how to approach
|
|
||||||
> this, and a staged approach for the implementation.
|
|
||||||
|
|
||||||
Plan-shaping user decisions (in-session Q&A): sibling
|
|
||||||
topology under `pikerd`; hard cutover staged by layer;
|
|
||||||
trading-only EMS-lazy-spawned `brokerd`. The human
|
|
||||||
staged the doc copy into `ai/claude-code/plans/` and
|
|
||||||
requested this commit after the 7 implementation
|
|
||||||
commits landed.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The staged design doc for the `brokerd` -> (`datad` +
|
|
||||||
`brokerd`) split: context, target supervision topology,
|
|
||||||
load-bearing verified facts, per-stage file-level
|
|
||||||
changes with gates, risk register and an end-to-end
|
|
||||||
verification matrix. Produced via 3 parallel explore
|
|
||||||
agents + 1 architect agent + human Q&A before any code
|
|
||||||
was written; all 7 implementation commits in this
|
|
||||||
branch reference its stages in their provenance
|
|
||||||
entries.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `ai/claude-code/plans/datad_service.md` — the design
|
|
||||||
plan doc (human-staged copy of the AI-authored plan)
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None to the doc content — committed as generated. NB:
|
|
||||||
the implementation deviated from the plan-as-written in
|
|
||||||
4 places, see the raw file's deviation log.
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T17:33:09Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry; the AI output here is the
|
|
||||||
committed doc itself (a design plan, not code) so no
|
|
||||||
verbatim copy is duplicated:
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
|
|
||||||
|
|
||||||
## Implementation deviation log
|
|
||||||
|
|
||||||
Where the landed 7-commit series differs from the plan
|
|
||||||
as written (recorded for reviewer transparency; the
|
|
||||||
plan doc is committed unmodified):
|
|
||||||
|
|
||||||
1. Commit ordering: the `pytest` config-dir isolation
|
|
||||||
fix lands BEFORE the `tractor`-API drift port. The
|
|
||||||
plan's stage gates assumed a runnable+isolated test
|
|
||||||
baseline; per-commit gating exposed that without
|
|
||||||
isolation the paper-EMS test reads the user's real
|
|
||||||
(polluted) `account.kraken.paper.toml` and reddens.
|
|
||||||
2. The plan's "stage 0: full pytest green" gate
|
|
||||||
required first repairing pre-existing branch
|
|
||||||
breakage vs `tractor` git `main` (boot
|
|
||||||
`AttributeError`, stale discovery/exc/position
|
|
||||||
APIs) — that repair became its own commit
|
|
||||||
("Port service+tests to latest `tractor` APIs")
|
|
||||||
rather than plan-stage work.
|
|
||||||
3. Stage-4 fail-fast placement: the plan said
|
|
||||||
`broker_init()` raises on brokerd-ep-less backends;
|
|
||||||
implementation moved the raise to `spawn_brokerd()`
|
|
||||||
since `piker ledger` calls `broker_init()` directly
|
|
||||||
even for paper accounts on datad-only backends.
|
|
||||||
4. `brokers/_daemon.py` change grouping: the
|
|
||||||
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
|
|
||||||
type-only import) landed with the caps-sec slim
|
|
||||||
commit instead of the fixture-slim commit, keeping
|
|
||||||
each intermediate tree import-clean without
|
|
||||||
sub-hunk surgery.
|
|
||||||
|
|
||||||
Also of record: the plan's "Verification" matrix was
|
|
||||||
executed as written (per-suite gates each stage, the
|
|
||||||
headless datad-feed smoke, the kucoin fail-fast unit
|
|
||||||
check); the known pre-existing ~50% second-runtime-boot
|
|
||||||
test wedge was characterized and excluded as a
|
|
||||||
regression via revert-testing.
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
|
||||||
timestamp: 2026-06-10T21:35:49Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
User bug report from live `ib` testing of the
|
|
||||||
(datad|brokerd)-split branch:
|
|
||||||
|
|
||||||
> ok doing some testing and noticing live orders do not
|
|
||||||
> work otb, pikerd show this on live submission,
|
|
||||||
> [pdb traceback: `RuntimeError("Can not order
|
|
||||||
> {symbol}, no live feed?")` at `ib/api.py:1152`
|
|
||||||
> `submit_limit()` with `self._contracts == {}` in the
|
|
||||||
> `brokerd.ib` actor + a `TrioTaskExited` teardown
|
|
||||||
> cascade]
|
|
||||||
|
|
||||||
Follow-up user direction fixing the design intent:
|
|
||||||
|
|
||||||
> i think the main thing here is that [brokerd] should
|
|
||||||
> be able to always submit orders without a live feed
|
|
||||||
> being up and registered in the same subactor right?
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Root cause: pre-split the feed code populated the
|
|
||||||
actor-local `Client._contracts` cache (via
|
|
||||||
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
|
|
||||||
SAME process as order submission; post-split the
|
|
||||||
trading actor's client is never warmed. Fix: lazily
|
|
||||||
qualify + cache the contract per order request in
|
|
||||||
`handle_order_requests()` by running the same
|
|
||||||
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
|
|
||||||
plus per-order error relay (`BrokerdError`) so one bad
|
|
||||||
submission can't crash the whole trades dialog (the
|
|
||||||
`TrioTaskExited` storm was teardown cascade from the
|
|
||||||
original raise).
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/broker.py` — thread `proxies` into
|
|
||||||
`handle_order_requests()`; lazy contract qualify on
|
|
||||||
cache-miss; guard `submit_limit()` w/ `BrokerdError`
|
|
||||||
relay; uncomment the (anticipatory) `get_mkt_info`
|
|
||||||
import
|
|
||||||
- `piker/brokers/ib/api.py` — fix the non-f-string
|
|
||||||
raise msg + document the new qualification contract
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None — committed as generated. Live `ib` order retest
|
|
||||||
performed by the human post-commit.
|
|
||||||
|
|
@ -1,57 +0,0 @@
|
||||||
---
|
|
||||||
model: claude-fable-5[1m]
|
|
||||||
service: claude
|
|
||||||
timestamp: 2026-06-10T21:35:49Z
|
|
||||||
git_ref: datad_service
|
|
||||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
|
||||||
---
|
|
||||||
|
|
||||||
NOTE: diff-ref mode entry (code committed in the same
|
|
||||||
commit as this log); recorded from the live debug
|
|
||||||
session per the `/prompt-io` skill rules.
|
|
||||||
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
|
|
||||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
|
||||||
|
|
||||||
Key diagnostic chain (from session):
|
|
||||||
|
|
||||||
- pdb showed `Client._contracts == {}` inside
|
|
||||||
`brokerd.ib`'s `submit_limit()`; the cache has
|
|
||||||
exactly TWO write sites: `Client.find_contracts()`
|
|
||||||
(api.py, keys `f'{sym}.{exch}.ib'`) and
|
|
||||||
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
|
|
||||||
'nvda.nasdaq' — NO `.ib` suffix).
|
|
||||||
- `BrokerdOrder.symbol` arrives as the bs_fqme form
|
|
||||||
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
|
|
||||||
site produces the key `submit_limit()` reads —
|
|
||||||
ie. pre-split it was the feed's in-proc
|
|
||||||
`get_mkt_info(sym, proxy=proxy)` call keeping orders
|
|
||||||
working, NOT `find_contracts()`.
|
|
||||||
- the existing TODO at `symbols.py:642-644` literally
|
|
||||||
predicted this: "this is going to be problematic
|
|
||||||
if/when we split out the datad vs. brokerd actors
|
|
||||||
since the mktmap lookup table will now be
|
|
||||||
inaccessible.."
|
|
||||||
- instance identity verified: `proxy._aio_ns` IS the
|
|
||||||
same `Client` obj as `_accounts2clients[account]`
|
|
||||||
(both sourced from the `load_aio_clients()` cache via
|
|
||||||
`open_client_proxies()`), so a brokerd-side
|
|
||||||
`get_mkt_info(fqme, proxy=proxies[account])` warms
|
|
||||||
exactly the dict `submit_limit()` reads. It also
|
|
||||||
populates `client._cons2mkts` which the
|
|
||||||
position-audit path (`broker.py` backup-table code)
|
|
||||||
needs in this actor anyway.
|
|
||||||
- the `TrioTaskExited` storm in the user's log
|
|
||||||
(`recv_trade_updates`, `open_aio_client_method_relay`
|
|
||||||
aio tasks) is teardown cascade: the raise crashed
|
|
||||||
`handle_order_requests` -> nursery teardown ripped
|
|
||||||
the trio sides of still-running aio relay tasks.
|
|
||||||
Hence the added per-order try/except ->
|
|
||||||
`BrokerdError` relay hardening so a single bad
|
|
||||||
submission degrades to an EMS error msg instead of
|
|
||||||
killing the backend's entire order-ctl dialog.
|
|
||||||
|
|
||||||
Verification: `tests/test_services.py` (5 passed) +
|
|
||||||
`tests/test_ems.py` (6 passed) regression-green; live
|
|
||||||
`ib` submission retest delegated to the human (needs a
|
|
||||||
running TWS/gw).
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
# AI Prompt I/O Log — claude
|
|
||||||
|
|
||||||
This directory tracks prompt inputs and model
|
|
||||||
outputs for AI-assisted development using
|
|
||||||
`claude` (claude-code CLI).
|
|
||||||
|
|
||||||
## Policy
|
|
||||||
|
|
||||||
Prompt logging follows the
|
|
||||||
[NLNet generative AI policy][nlnet-ai].
|
|
||||||
All substantive AI contributions are logged
|
|
||||||
with:
|
|
||||||
- Model name and version
|
|
||||||
- Timestamps
|
|
||||||
- The prompts that produced the output
|
|
||||||
- Unedited model output (`.raw.md` files)
|
|
||||||
|
|
||||||
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
Entries are created by the `/prompt-io` skill
|
|
||||||
or automatically via `/commit-msg` integration.
|
|
||||||
|
|
||||||
Each commit carrying AI-generated changes links
|
|
||||||
to its provenance entry via a `Prompt-IO:`
|
|
||||||
commit-msg trailer; entries use "diff-ref mode"
|
|
||||||
(pointers into `git log -p` instead of verbatim
|
|
||||||
code copies) to avoid duplicating committed
|
|
||||||
code.
|
|
||||||
|
|
||||||
Human contributors remain accountable for all
|
|
||||||
code decisions. AI-generated content is never
|
|
||||||
presented as human-authored work.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-07-22T00:13:58Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Continue the structured backfill phase after the analysis regressions.
|
|
||||||
Characterize provider exhaustion and genuinely empty frames, then remove
|
|
||||||
completion paths that can hang after child-task exit.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Removed orphan-prone completion events from the structured nursery flow
|
|
||||||
and checked empty provider frames before indexing timestamp endpoints.
|
|
||||||
Added deterministic timeout-bounded regressions for both exit paths.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
|
|
||||||
- `tests/test_history_backfill.py` - deterministic exit regressions
|
|
||||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
|
|
||||||
- unedited response record
|
|
||||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
|
|
||||||
- provenance metadata and response summary
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated changes have not yet been edited by the human.
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
provider: openai
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-07-22T00:13:58Z
|
|
||||||
git_ref: 16f3cd6f
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
|
||||||
|
|
||||||
Removed redundant completion events whose consumers waited only after
|
|
||||||
their owning nurseries had joined. Added empty provider-frame handling
|
|
||||||
before timestamp endpoint indexing so provider exhaustion exits cleanly.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Added bounded, actor-free regressions for `DataUnavailable` and empty
|
|
||||||
provider-frame completion.
|
|
||||||
|
|
||||||
Verification generated with the patch:
|
|
||||||
|
|
||||||
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
|
|
||||||
|
|
||||||
Result: 2 passed.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: ses_0799212ebffe42arY96czXn89F
|
|
||||||
timestamp: 2026-08-11T20:58:26Z
|
|
||||||
git_ref: 06d5ea5d
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Diagnose and fix an IB live-order failure where lazy NVDA contract
|
|
||||||
qualification raised `tractor.trionics.Lagged` after the order reached
|
|
||||||
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
|
|
||||||
searches caused the stale state.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Traced the failure to competing consumers of a one-slot Trio broadcast, not
|
|
||||||
the recent SHM change or stale asyncio client state. Replaced broadcast
|
|
||||||
fan-out with one response/status demultiplexer, retained request-ID
|
|
||||||
correlation, handled cancellation and teardown, documented the task/channel
|
|
||||||
ownership model, warned on diagnostic metadata eviction, and added
|
|
||||||
deterministic proxy regressions.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
|
|
||||||
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
|
|
||||||
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated output follows the user's live failure diagnosis.
|
|
||||||
|
|
@ -1,66 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-11T20:58:26Z
|
|
||||||
git_ref: 06d5ea5d
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user reported that certain IB order entries might have broken and
|
|
||||||
provided a live failure for a first NVDA sell order. EMS delivered the
|
|
||||||
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
|
|
||||||
qualification failed in `MethodProxy._run_method()` with:
|
|
||||||
|
|
||||||
```text
|
|
||||||
tractor.trionics._broadcast.Lagged:
|
|
||||||
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
|
|
||||||
dropped `0` values
|
|
||||||
```
|
|
||||||
|
|
||||||
The user suspected a cancelled symbol-search task might have left stale
|
|
||||||
asyncio state. Cancellation can leave an orphaned method response, but the
|
|
||||||
observed failure was stale Trio broadcast state. `open_client_proxy()`
|
|
||||||
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
|
|
||||||
and an idle method receiver. Status traffic advanced only the relay, so the
|
|
||||||
first method call for an uncached symbol entered an already-lagged receiver.
|
|
||||||
Orders for eagerly cached contracts bypassed this path.
|
|
||||||
|
|
||||||
The immediately preceding sparse-history commit did not modify IB order,
|
|
||||||
symbol, or proxy code; it exposed no direct dependency on this failure.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
|
|
||||||
|
|
||||||
Generated a sole-reader message demultiplexer for each IB client proxy.
|
|
||||||
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
|
|
||||||
status/error traffic, and dispatches `mid`-tagged responses into per-call
|
|
||||||
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
|
|
||||||
late responses to be logged and dropped without skewing later calls. Clean
|
|
||||||
channel EOF is accepted during normal proxy teardown.
|
|
||||||
|
|
||||||
Follow-up review restored the masked reconnect TODO and per-message case
|
|
||||||
rationale, documented the one-reader/many-writer task hierarchy and separate
|
|
||||||
per-proxy channels, and added a warning with request method metadata when the
|
|
||||||
bounded unresolved-request diagnostic table evicts its oldest entry.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
|
|
||||||
|
|
||||||
Generated deterministic regressions for idle status traffic before first
|
|
||||||
qualification, two concurrent calls with reverse-order responses, cancelled
|
|
||||||
calls with late responses followed by a successful call, and graceful
|
|
||||||
channel EOF. Every synchronization wait is timeout-bounded.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
|
||||||
|
|
||||||
Registered the deterministic IB method-proxy target and source mapping.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
........................ [100%]
|
|
||||||
25 passed, 4 warnings in 3.44s
|
|
||||||
```
|
|
||||||
|
|
||||||
The warnings are existing Tractor/Trio deprecations from the EMS test.
|
|
||||||
Adversarial review drove a clean-EOF guard and bounded polling. Final review
|
|
||||||
found no actionable issues. Live IB order validation remains pending and
|
|
||||||
requires restarting `brokerd.ib`.
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: unavailable
|
|
||||||
timestamp: 2026-08-13T20:24:02Z
|
|
||||||
git_ref: 12b0d66e
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Continue the paired piker broadcast-consumer work after Tractor PR 490
|
|
||||||
stabilization. Stop cached feed callers from mutating or consuming shared
|
|
||||||
quote-stream roots, then review, verify and prepare a commit boundary.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Made provider root streams private to the cached `Feed`, gave every caller a
|
|
||||||
local subscribed view, corrected feed cache identity, and made multi-provider
|
|
||||||
relays own synchronized child subscriptions and independent send-channel
|
|
||||||
clones.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
|
|
||||||
subscribed multi-provider relays with independent channel lifetimes.
|
|
||||||
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
|
|
||||||
cache-identity regressions, including sibling survival after provider EOC.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated output follows the requested cached-feed ownership item.
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-13T20:24:02Z
|
|
||||||
git_ref: 12b0d66e
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user asked to continue the paired piker broadcast-consumer work after
|
|
||||||
Tractor PR 490 stabilization. The next item was to stop cached feed callers
|
|
||||||
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
|
|
||||||
review, verify and prepare a commit boundary.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
|
||||||
|
|
||||||
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
|
|
||||||
root streams and every caller, including the first cache owner, receives a
|
|
||||||
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
|
|
||||||
The cache identity now includes normalized full symbol shape and stream policy.
|
|
||||||
|
|
||||||
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
|
|
||||||
of directly consuming a shared stream, and synchronizes relay startup before
|
|
||||||
yielding the merged channel. Each relay owns a cloned send channel so one
|
|
||||||
provider EOC cannot terminate its siblings. The module-level relay returns its
|
|
||||||
live child receiver through `TaskStatus` for typed retention and introspection.
|
|
||||||
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
|
|
||||||
shared by local cache users.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
|
|
||||||
|
|
||||||
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
|
|
||||||
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
|
|
||||||
multi-provider relay fan-out and startup, duplex controls, lexical child
|
|
||||||
closure, sibling survival after provider EOC, cached-root immutability, and
|
|
||||||
full cache-key differentiation.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
... [100%]
|
|
||||||
3 passed in 0.02s
|
|
||||||
|
|
||||||
. [100%]
|
|
||||||
1 passed in 0.01s
|
|
||||||
```
|
|
||||||
|
|
||||||
The second result is the existing EMS broadcast ownership regression. Ruff,
|
|
||||||
E501, compilation and whitespace checks passed. Repeated adversarial review
|
|
||||||
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
|
|
||||||
relay-startup and first-owner asymmetry issues; final review reported no
|
|
||||||
findings.
|
|
||||||
|
|
@ -1,32 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: unavailable
|
|
||||||
timestamp: 2026-08-13T23:07:44Z
|
|
||||||
git_ref: ddac3553
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Continue the piker broadcast-consumer audit after the Deribit ownership commit
|
|
||||||
and prepare the next smallest ownership fix.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
|
|
||||||
context and gave every caller a lexical child from one retained
|
|
||||||
`LinkedTaskChannel` broadcaster root.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
|
|
||||||
ownership.
|
|
||||||
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
|
|
||||||
lifetime regression.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
None - generated output follows the requested broadcast-consumer audit.
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-13T23:07:44Z
|
|
||||||
git_ref: ddac3553
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
The user asked to continue the piker broadcast-consumer audit after committing
|
|
||||||
the Deribit ownership fix. The next isolated item was IB's actor-local
|
|
||||||
`_quote_streams` cache, which retained a raw channel owned by its first caller
|
|
||||||
and constructed competing broadcaster roots for later callers.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
|
||||||
|
|
||||||
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
|
|
||||||
made `open_aio_quote_stream()` cache that context by symbol through
|
|
||||||
`maybe_open_context()`. Every caller now receives a lexical child from the
|
|
||||||
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
|
|
||||||
until its final cache user exits. Removed the manual `_quote_streams` registry
|
|
||||||
and teardown mutation.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
|
|
||||||
|
|
||||||
Added a concurrent ownership regression using real Tractor broadcast state.
|
|
||||||
It proves miss and hit callers use the same symbol/context factory, receive the
|
|
||||||
same quote through distinct children, close independently, and leave the owner
|
|
||||||
live after the hit exits.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
|
|
||||||
```text
|
|
||||||
. [100%]
|
|
||||||
1 passed in 0.01s
|
|
||||||
```
|
|
||||||
|
|
||||||
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
|
|
||||||
Python compilation and whitespace checks pass. Normal package collection
|
|
||||||
remains blocked by the existing Qt environment mismatch.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: tractor-addr-unpacking
|
|
||||||
timestamp: 2026-08-20T16:35:19Z
|
|
||||||
git_ref: 5f631e19
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human asked to implement the Piker compatibility work caused by
|
|
||||||
tractor's canonical tagged-address migration, run the previously
|
|
||||||
deferred tractor verification, and keep runtime-managed WireGuard
|
|
||||||
bindspaces in a later separate PR.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Made Piker normalize legacy and tagged registry inputs before semantic
|
|
||||||
comparison and process-global storage. Updated runtime address typing and
|
|
||||||
test assertions for canonical output, retained legacy caller input, and
|
|
||||||
removed a duplicate pytest log-level option that blocked downstream test
|
|
||||||
collection with current tractor.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/service/_registry.py` - canonical registry normalization and
|
|
||||||
shared address typing.
|
|
||||||
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
|
|
||||||
annotations.
|
|
||||||
- `tests/conftest.py` - canonical fixture comparison and shared pytest
|
|
||||||
option ownership.
|
|
||||||
- `tests/test_services.py` - registry regression and canonical runtime
|
|
||||||
assertions.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human explicitly requested the downstream implementation and chose
|
|
||||||
to keep WireGuard bindspace lifecycle work in a separate future tractor
|
|
||||||
PR. The agent implemented and tested those directions; no direct manual
|
|
||||||
source edits were observed.
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-20T16:35:19Z
|
|
||||||
git_ref: 5f631e19
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
# Raw output - tractor tagged-address compatibility
|
|
||||||
|
|
||||||
The human requested downstream Piker qualification after tractor PR
|
|
||||||
#505 switched transport address emission to canonical tagged tuples.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
|
|
||||||
|
|
||||||
Normalized legacy and tagged registry addresses through tractor's
|
|
||||||
address codec before Piker stores or compares them. Extended service
|
|
||||||
runtime annotations to cover both tuple shapes, updated runtime tests to
|
|
||||||
compare canonical serialized values, and removed Piker's duplicate
|
|
||||||
registration of tractor's shared `--ll` pytest option.
|
|
||||||
|
|
||||||
Ruff passed. The focused normalization regression and Piker runtime boot
|
|
||||||
passed against the local tractor checkout at `5d92595f`. The wider
|
|
||||||
service module reached an unrelated pre-existing logger-name assertion
|
|
||||||
in `maybe_spawn_daemon()` after both address-focused tests passed.
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: tractor-addr-unpacking
|
|
||||||
timestamp: 2026-08-21T04:20:58Z
|
|
||||||
git_ref: b7aca4c0
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human approved refining Piker's logger-name handling after the
|
|
||||||
wider service test exposed a stale parent-logger assertion during the
|
|
||||||
tractor tagged-address compatibility work.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
|
|
||||||
preserving complete requested submodule names. Removed daemon startup
|
|
||||||
assertions that incorrectly required parent logger names and added a
|
|
||||||
regression covering both logger factories and names that the old
|
|
||||||
character-set stripping silently corrupted.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/log.py` - exact relative logger-name derivation.
|
|
||||||
- `piker/service/_daemon.py` - remove stale parent-name assertion.
|
|
||||||
- `piker/data/_daemon.py` - remove stale parent-name assertion.
|
|
||||||
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
|
|
||||||
- `piker/clearing/_ems.py` - remove redundant logger assertion.
|
|
||||||
- `tests/test_logging.py` - package logger namespace regressions.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human explicitly approved this follow-up after reviewing the
|
|
||||||
residual Piker service-suite gap. The agent implemented and verified
|
|
||||||
the refinement; no direct manual source edits were observed.
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
---
|
|
||||||
model: gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-21T04:20:58Z
|
|
||||||
git_ref: b7aca4c0
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
# Raw output - preserve Piker logger namespaces
|
|
||||||
|
|
||||||
The human approved a follow-up refinement after the full Piker service
|
|
||||||
module exposed a stale logger-name assertion while qualifying tractor
|
|
||||||
PR #505.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
|
|
||||||
|
|
||||||
Replaced character-set `str.lstrip()` calls with exact package-prefix
|
|
||||||
removal, removed assertion-only parent-logger expectations from daemon
|
|
||||||
startup paths, and added regression coverage for both logger factories
|
|
||||||
and prefix-sensitive submodule names.
|
|
||||||
|
|
||||||
Ruff passed after removing newly unused logger assignments. All four
|
|
||||||
logger regressions and all six service integration tests passed against
|
|
||||||
the local tractor PR #505 checkout.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: unavailable
|
|
||||||
timestamp: 2026-08-26T22:26:00Z
|
|
||||||
git_ref: 5f631e19
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Diagnose and fix the hanging Binance USD-M perpetual feed for
|
|
||||||
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
|
|
||||||
API and changelog. Ensure unsupported or silent websocket feeds fail
|
|
||||||
loudly into `datad` and chart consumers instead of blocking startup.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Migrates Binance USD-M market data from the retired legacy websocket to
|
|
||||||
the split public and market routes, handles Binance's new aggregate-trade
|
|
||||||
schema, and introduces bounded connection and first-data startup. Direct
|
|
||||||
stream URLs avoid reconnect-time subscription-ACK races, while targeted
|
|
||||||
tests preserve routing and fail-loud behavior.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
|
|
||||||
Binance quote streams.
|
|
||||||
- `piker/brokers/binance/venues.py` - define current USD-M websocket
|
|
||||||
endpoints.
|
|
||||||
- `piker/data/_web_bs.py` - support bounded websocket context startup.
|
|
||||||
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
|
|
||||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
|
|
||||||
retain the generated-output record.
|
|
||||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
|
|
||||||
structured provenance for the code changes.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human explicitly required fail-loud behavior rather than merely
|
|
||||||
updating the endpoint, and supplied a direct CLI reproduction proving
|
|
||||||
both `datad.binance` and the chart caller blocked. Those requirements
|
|
||||||
drove the connection and first-data deadlines and their regressions. The
|
|
||||||
changes remain uncommitted for human review.
|
|
||||||
|
|
@ -1,57 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-26T22:26:00Z
|
|
||||||
git_ref: 5f631e19
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user reported that selecting
|
|
||||||
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
|
|
||||||
websocket but never loaded the feed or downstream chart. They asked
|
|
||||||
OpenCode to compare Binance's API changelog with piker's FQME and feed
|
|
||||||
code, diagnose the failure, and solve it.
|
|
||||||
|
|
||||||
The user clarified that future provider failures must raise loudly
|
|
||||||
instead of silently blocking feed and chart startup. They also confirmed
|
|
||||||
the same hang from
|
|
||||||
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
|
|
||||||
|
|
||||||
## Generated code
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
|
||||||
|
|
||||||
Routes USD-M book-ticker and aggregate-trade data through Binance's new
|
|
||||||
`/public` and `/market` websocket paths, merges both sockets into the
|
|
||||||
existing quote stream, accepts the new `st` aggregate-trade field, and
|
|
||||||
requires both stream classes before declaring startup complete. It also
|
|
||||||
removes the misleading missing-provider warning, derives feed mode from
|
|
||||||
resolved pair metadata, and raises `DataUnavailable` for connection or
|
|
||||||
live-data startup timeouts.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
|
|
||||||
|
|
||||||
Defines Binance's current public and market USD-M websocket endpoints
|
|
||||||
and makes the market endpoint the regular futures websocket API root.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Adds an optional connection deadline to `open_autorecon_ws()` so callers
|
|
||||||
can fail context entry when repeated websocket handshakes never become
|
|
||||||
connected.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
|
||||||
|
|
||||||
Adds regressions for exact Binance stream routing, silent live-data
|
|
||||||
startup, silent websocket connection startup, and the current USD-M
|
|
||||||
aggregate-trade payload schema.
|
|
||||||
|
|
||||||
## Verification output
|
|
||||||
|
|
||||||
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
|
|
||||||
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
|
|
||||||
through piker's merged stream received both required stream classes and
|
|
||||||
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
|
|
||||||
spot.
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
|
|
||||||
timestamp: 2026-08-28T20:03:52Z
|
|
||||||
git_ref: dd676857
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Update the xonsh version used by the repository dev shell before stacking
|
|
||||||
the Binance work, while retaining the dotrc compatibility shim as an
|
|
||||||
immediate workaround for older shells.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
|
|
||||||
The new release supports Python 3.13, handles Git alias completion natively,
|
|
||||||
and works in the updated Nix development shell.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `pyproject.toml` - require xonsh `0.24.2` or newer.
|
|
||||||
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
|
|
||||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
|
|
||||||
the triggering prompt, generated configuration summary, and checks.
|
|
||||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
|
|
||||||
structured provenance for the dependency update.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The user identified old xonsh behavior as the root cause, reported the
|
|
||||||
separate dotrc compatibility shim, and directed that the repository update
|
|
||||||
remain a distinct flake-layer change before the Binance stack continues.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-28T20:03:52Z
|
|
||||||
git_ref: dd676857
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user traced broken multiline conflict-resolution pastes to the old
|
|
||||||
xonsh release installed by the repository dev shell. A separate dotrc
|
|
||||||
agent added a compatibility shim, and the user directed that xonsh still
|
|
||||||
be updated alongside the flake work before continuing the Binance stack.
|
|
||||||
|
|
||||||
## Generated configuration
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
|
||||||
|
|
||||||
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
|
|
||||||
refreshes only xonsh's locked distribution metadata. The selected release
|
|
||||||
supports Python 3.13, includes native Git-config alias completion, and fixes
|
|
||||||
Nix metadata checks.
|
|
||||||
|
|
||||||
## Verification output
|
|
||||||
|
|
||||||
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
|
|
||||||
xonsh from `0.22.4` to `0.24.2`.
|
|
||||||
- Fresh `nix develop` replaces the worktree-local xonsh installation.
|
|
||||||
- `xonsh --version` reports `0.24.2`.
|
|
||||||
- A multi-command xonsh invocation executes Git commands successfully.
|
|
||||||
- `nix flake check --no-build` passes for `x86_64-linux`.
|
|
||||||
- `git diff --check` passes.
|
|
||||||
|
|
||||||
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
|
|
||||||
that its distributions are yanked for an `unpivot` regression.
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
|
||||||
timestamp: 2026-08-29T17:55:42Z
|
|
||||||
git_ref: e09e306a
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
|
|
||||||
stack, then add missing deterministic tests and correctness fixes as a
|
|
||||||
separate follow-up with a complete editor-gated commit plan.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Hardens Binance feed startup around its supported one-symbol contract,
|
|
||||||
direct websocket routes, current provider schemas, and first-quote
|
|
||||||
readiness. Adds deterministic Binance regressions and separates generic
|
|
||||||
reconnect behavior into its own tested commit boundary.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
|
|
||||||
USD-M stream classes, and declare readiness after either quote class.
|
|
||||||
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
|
|
||||||
and documented book-ticker and aggregate-trade schemas.
|
|
||||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
|
|
||||||
the triggering prompt, generated-code summary, and verification output.
|
|
||||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
|
|
||||||
structured provenance for the follow-up.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The user rejected reconstructing already committed Binance work, required
|
|
||||||
preservation through cherry-pick, selected the flake/xonsh stack base, and
|
|
||||||
required the review fixes to remain a separate tested boundary. The user also
|
|
||||||
corrected the workflow to require an automatic `/commit-plan` at every stack
|
|
||||||
review gate. During Tuicr review, the user rejected trade-gated startup,
|
|
||||||
requested a feed-specific initialization error and async-generator TODO,
|
|
||||||
and required generic reconnect changes and tests to land independently.
|
|
||||||
The user then clarified that either L1 or trade data must unblock startup,
|
|
||||||
rather than preferring one stream class or waiting independently for both.
|
|
||||||
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-08-29T17:55:42Z
|
|
||||||
git_ref: e09e306a
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user asked to land recent branch work in small chronological patch sets,
|
|
||||||
including missing tests. For Binance, the user required preserving the
|
|
||||||
existing committed topic patch through cherry-pick and placing newly found
|
|
||||||
correctness repairs in a separate follow-up boundary above the flake/xonsh
|
|
||||||
stack. The user also required every yielded stack layer to include a complete
|
|
||||||
`/commit-plan` with patch-specific checks.
|
|
||||||
|
|
||||||
## Generated code
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
|
||||||
|
|
||||||
Restricts Binance feeds to the supported one-symbol contract, renders direct
|
|
||||||
quote routes, separates L1 liveness from first-trade readiness, and derives
|
|
||||||
the contextual trade timeout from Trio's enclosing effective deadline.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Lets structured cancellation bypass reconnect throttling and ensures handled
|
|
||||||
handshake rejections retry before accessing connection-local cancel state.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
|
||||||
|
|
||||||
Adds deterministic regressions for symbol cardinality, route selection,
|
|
||||||
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
|
|
||||||
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
|
|
||||||
|
|
||||||
## Verification output
|
|
||||||
|
|
||||||
- Targeted Binance tests: `10 passed in 0.26s`.
|
|
||||||
- Ruff passes for all three changed Python files.
|
|
||||||
- `git diff --check` passes.
|
|
||||||
- Independent final review reports no actionable findings.
|
|
||||||
- No live Binance network probe was run.
|
|
||||||
|
|
@ -1,42 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
|
||||||
timestamp: 2026-09-01T02:42:35Z
|
|
||||||
git_ref: e09e306a
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Address the Tuicr review by separating generic websocket reconnect
|
|
||||||
changes from Binance-specific work and testing the real reconnect task
|
|
||||||
hierarchy rather than a reimplementation.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Preserves structured cancellation and safe first-handshake retries in
|
|
||||||
`_web_bs`. Adds three focused regressions in a dedicated websocket test
|
|
||||||
module and packages them as an independent commit boundary.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/_web_bs.py` - restart handled handshake failures and let
|
|
||||||
cancellation bypass the reconnect throttle.
|
|
||||||
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
|
|
||||||
and cancellation task paths with controlled transport boundaries.
|
|
||||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
|
|
||||||
- retain the review prompt, generated-code summary, and verification.
|
|
||||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
|
|
||||||
structured provenance for the generic reconnect boundary.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The user identified that these changes were generic rather than
|
|
||||||
Binance-specific, required a separate commit and test module, requested
|
|
||||||
coverage of the real reconnect hierarchy, and corrected code prose to
|
|
||||||
refer to `trio` consistently. A follow-up review required the production
|
|
||||||
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
|
|
||||||
handler and cite the regression which audits that teardown invariant.
|
|
||||||
|
|
@ -1,33 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-09-01T02:42:35Z
|
|
||||||
git_ref: e09e306a
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user requested that generic `_web_bs` reconnect fixes and tests be
|
|
||||||
separated from the Binance backend patch. The review also required tests
|
|
||||||
to exercise the real reconnect task hierarchy, cite the previous faulty
|
|
||||||
branches, and consistently name `trio` in code prose.
|
|
||||||
|
|
||||||
## Generated code
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Ensures handled handshake failures immediately restart the reconnect
|
|
||||||
loop and lets structured cancellation bypass shielded error throttling.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
|
||||||
|
|
||||||
Adds deterministic scheduler-real regressions for handshake startup
|
|
||||||
timeouts, immediate rejection retries, and cancellation propagation.
|
|
||||||
|
|
||||||
## Verification output
|
|
||||||
|
|
||||||
- Generic and Binance targeted tests: `9 passed in 0.06s`.
|
|
||||||
- Ruff passes for all four changed Python files.
|
|
||||||
- `git diff --check` passes.
|
|
||||||
- No live network probe was run.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
session: 4dc920cf-e82e-4c1e-83db-75a0a9efe82b
|
|
||||||
timestamp: 2026-09-01T23:00:43Z
|
|
||||||
git_ref: e91f41a6
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20260901T230043Z_e91f41a6_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
Update `wkt/land_flake_update` to Tractor's restored `piker_pin` after
|
|
||||||
reusing the existing runtime compatibility commits. Keep stale
|
|
||||||
historical lock snapshots out of the final dependency state.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Updates piker's Python and `msgspec` requirements for current Tractor,
|
|
||||||
then generates one fresh lock at Tractor `piker_pin@0d95a9e6`. The
|
|
||||||
dependency commit completes the compatibility stack already
|
|
||||||
cherry-picked onto the flake branch.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `pyproject.toml` - require Python 3.13 and `msgspec>=0.20.0`.
|
|
||||||
- `uv.lock` - resolve Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`,
|
|
||||||
and current transport dependencies.
|
|
||||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.raw.md`
|
|
||||||
- preserve the prompt, generated-change summary, and resolution notes.
|
|
||||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md` - record
|
|
||||||
structured provenance for the dependency update.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The user identified that Tractor compatibility should land with the
|
|
||||||
flake update rather than the Binance patch, supplied four known source
|
|
||||||
commits, required a broader local-history audit for missing ports, and
|
|
||||||
directed a fresh lock from the restored `piker_pin` instead of replaying
|
|
||||||
historical pins.
|
|
||||||
|
|
@ -1,35 +0,0 @@
|
||||||
---
|
|
||||||
model: openai/gpt-5.6-sol
|
|
||||||
service: opencode
|
|
||||||
timestamp: 2026-09-01T23:00:43Z
|
|
||||||
git_ref: e91f41a6
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The user requested that the flake-update branch absorb the Tractor
|
|
||||||
runtime compatibility work before the Binance branch lands. They
|
|
||||||
directed reuse of existing compatibility commits, exclusion of stale
|
|
||||||
historical lock pins, and a fresh update to Tractor's restored
|
|
||||||
`piker_pin`.
|
|
||||||
|
|
||||||
## Generated changes
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
|
||||||
|
|
||||||
Narrows piker support to Python 3.13 and raises the direct `msgspec`
|
|
||||||
requirement to the version family required by current Tractor.
|
|
||||||
|
|
||||||
> `git diff --cached -- uv.lock`
|
|
||||||
|
|
||||||
Relocks against Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`, and the
|
|
||||||
current transport dependency graph while retaining the existing Xonsh
|
|
||||||
update.
|
|
||||||
|
|
||||||
## Resolution notes
|
|
||||||
|
|
||||||
- Historical lock hunks from reused commits were intentionally omitted.
|
|
||||||
- Tractor runtime compatibility remains represented by the preceding
|
|
||||||
individually preserved commits on `wkt/land_flake_update`.
|
|
||||||
- Project checks are rendered in the commit plan for human execution.
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-21T22:44:21Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 507c75c6
|
|
||||||
scope: config
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261021T224421Z_507c75c6_piker_pin_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human asked to sanity-check the current PR94 landing state against
|
|
||||||
the qualified new Tractor pin before deduplicating branch history. After
|
|
||||||
the initial offline checks passed, the human authorized changing the
|
|
||||||
selector to piker_pin, resolving the lock, and verifying the installed
|
|
||||||
dependency without the temporary local-source PYTHONPATH override.
|
|
||||||
The human explicitly authorized network resolution and a scoped update
|
|
||||||
of Tractor/runtime dependencies while preserving unrelated packages.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Updated the branch selector and lock to published piker_pin@b64c89f0.
|
|
||||||
Installed the exact locked Tractor revision, verified its installed
|
|
||||||
source metadata, and reran 14 offline logger/startup cases without a
|
|
||||||
local Tractor source override. Preserved the rest of the environment.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `pyproject.toml`: select piker_pin and bound Polars below 2.0.
|
|
||||||
- `uv.lock`: lock b64c89f0 and Polars/runtime 1.44.2.
|
|
||||||
- This provenance pair.
|
|
||||||
|
|
||||||
## Verification and limits
|
|
||||||
|
|
||||||
14 distinct offline cases passed against the installed Tractor update
|
|
||||||
with the existing Polars 1.36 installation. The actor.uid deprecation
|
|
||||||
warning remains. No live provider/order testing occurred.
|
|
||||||
|
|
||||||
The initial resolver selected Polars 2.0 and rejected retaining 1.36
|
|
||||||
for its Windows target. The human then requested the latest pre-2.0
|
|
||||||
release instead. PyPI metadata identified non-yanked stable 1.44.2;
|
|
||||||
adding `<2.0.0` to the existing lower bound and relocking succeeded
|
|
||||||
with both polars and polars-runtime-32 at 1.44.2, without platform edits.
|
|
||||||
The raw partner retains the original, superseded 2.0 generation history.
|
|
||||||
|
|
||||||
The scoped install changed only Tractor. Polars 1.44.2 is locked but
|
|
||||||
not installed or data-path qualified yet. The human's existing staged
|
|
||||||
checkpoint is preserved; the pre-2.0 correction is in the worktree.
|
|
||||||
No history cleanup, staging or commit was performed by this agent.
|
|
||||||
|
|
@ -1,45 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-21T22:44:21Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 507c75c6
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
|
||||||
|
|
||||||
Changed the landing branch's Tractor selector from wkt/addr_unpacking
|
|
||||||
to piker_pin and resolved the published branch to
|
|
||||||
b64c89f05a9fc8ac2878551eb5eda36574ebef6a. An offline branch lookup lacked
|
|
||||||
the cached name, so the human explicitly authorized network resolution
|
|
||||||
and a scoped installed-runtime update preserving unrelated packages.
|
|
||||||
|
|
||||||
The resolver advanced Polars and polars-runtime-32 from 1.36.0 to 2.0.0.
|
|
||||||
An attempt to retain both exact 1.36.0 versions was rejected for the
|
|
||||||
resolver's Windows target. The valid lock with the visible Polars bump
|
|
||||||
is retained for review; no platform restriction or source constraint was
|
|
||||||
introduced to bypass the failure.
|
|
||||||
|
|
||||||
Installed only Tractor and its required dependencies with uv pip rather
|
|
||||||
than syncing/pruning the whole environment. The installer reported one
|
|
||||||
package replacement: Tractor a966928d to b64c89f0. Qt distribution versions
|
|
||||||
remained PyQt6 6.10.1, PyQt6-Qt6 6.10.1 and PyQt6-sip 13.10.3.
|
|
||||||
|
|
||||||
Removed PYTHONPATH from verification subprocesses. Piker resolved inside
|
|
||||||
land_flake_update and Tractor resolved from that checkout's py313
|
|
||||||
site-packages. Installed VCS metadata matched the exact locked revision.
|
|
||||||
The temporary LD_LIBRARY_PATH used the binding's bundled Qt libraries
|
|
||||||
and matching native dependencies; no global environment setup changed.
|
|
||||||
|
|
||||||
Verification output:
|
|
||||||
- Logger/non-actor startup: 12 passed, 1 deselected in 0.05s.
|
|
||||||
- test_runtime_boot: 1 passed, 1 warning in 0.04s.
|
|
||||||
- test_daemon_bind_distinct_from_registrar: 1 passed in 0.77s.
|
|
||||||
- Existing warning: actor.uid deprecation; no failure.
|
|
||||||
- git diff --check passed.
|
|
||||||
|
|
||||||
This verifies the installed Tractor pin, not a full sync to all new lock
|
|
||||||
versions. Polars was not upgraded in the environment by the scoped install.
|
|
||||||
No live broker feed, order, branch rewrite, staging or commit was run.
|
|
||||||
|
|
@ -1,67 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-22T21:22:48Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6-astra
|
|
||||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human approved trying the bounded candidate: back out the IB
|
|
||||||
quote-context cache, import adapter-only reset/boundary fixes and
|
|
||||||
the missing EMS subscription fix, preserve source authorship and
|
|
||||||
commit boundaries, and exercise the real Tractor bridge. They
|
|
||||||
required an explicit keep/backout audit and clarification that
|
|
||||||
the evaluated downstream lineage lacked the broadcast fix.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Verified identical stable patch IDs for original 4af885d9 and
|
|
||||||
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
|
|
||||||
in the main checkout. Applied source-equivalent adapter backports
|
|
||||||
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
|
|
||||||
source attribution recorded in `ai/ib-recovery-candidate.md`.
|
|
||||||
|
|
||||||
The real-runtime quote test first reproduced the reported fatal
|
|
||||||
TrioTaskExited propagation with caching enabled. The direct-owner
|
|
||||||
backout restores restart and subsequent RPC survival, but a strict
|
|
||||||
callback-cleanliness test remains red. Real bridge history pacing
|
|
||||||
recovery and real broadcaster dark clearing pass. No live broker,
|
|
||||||
VNC, orders or TSDB operations were performed by the agent.
|
|
||||||
|
|
||||||
On 2026-10-23 the human supplied successful live startup/backfill,
|
|
||||||
sequential reattachment and overlapping chart-close/reopen results,
|
|
||||||
plus the still-failing daemon SIGINT teardown. The human accepted
|
|
||||||
deferring teardown from #94, requested loud partial-backport
|
|
||||||
attribution and required downstream rebase rehearsals before landing.
|
|
||||||
The agent removed only the explicit single-null debugger stop,
|
|
||||||
retaining its legacy result. Downstream `976db6c3` instead rewrites
|
|
||||||
null grouping; the local cleanup does not claim that algorithmic fix.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
|
|
||||||
- `piker/clearing/_ems.py`: original EMS subscription patch.
|
|
||||||
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
|
|
||||||
- `tests/test_ib_history.py`: original adapter regressions only.
|
|
||||||
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
|
|
||||||
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
|
|
||||||
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
|
|
||||||
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
The human required original fixes and authorship, rejected broad
|
|
||||||
speculative integration work, required real concurrency boundaries
|
|
||||||
in tests, and corrected the interpretation of live qualification.
|
|
||||||
They performed the live tests, chose to defer known teardown defects
|
|
||||||
from #94, permitted the minimal debugger cleanup, and requested
|
|
||||||
explicit downstream discrepancies and rebase compatibility checks.
|
|
||||||
The agent applied those directed edits; no direct manual source edits
|
|
||||||
are known in this candidate. The raw partner retains the originally
|
|
||||||
captured output; this structured record includes later contributions.
|
|
||||||
|
|
@ -1,69 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-22T21:22:48Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6-astra
|
|
||||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
|
||||||
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
|
|
||||||
---
|
|
||||||
|
|
||||||
## Human direction
|
|
||||||
|
|
||||||
> yeah i guess this is our best approach strat for sure.
|
|
||||||
|
|
||||||
> so you're saying these 2 are not the same patch?
|
|
||||||
> please tell me no lol
|
|
||||||
|
|
||||||
> so detail which commits to drop vs. keep please in detail so i can
|
|
||||||
> audit your choices this time..
|
|
||||||
|
|
||||||
> let's try it and consider bailing and re-trying if this leads us into
|
|
||||||
> a further (and deeper) sinkhole, bc as it stands right now this whole
|
|
||||||
> endevour has turned into a complete shitfest XD
|
|
||||||
|
|
||||||
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
|
|
||||||
|
|
||||||
## Unedited diagnostic response
|
|
||||||
|
|
||||||
The real-bridge test reproduced the exact fatal path:
|
|
||||||
`maybe_open_context()` teardown raises `TrioTaskExited` back into
|
|
||||||
the quote consumer.
|
|
||||||
|
|
||||||
With only the quote-cache backout, both quote generations complete
|
|
||||||
and their asyncio tasks finish. Tractor still emits
|
|
||||||
`TrioTaskExited` callback errors, so the strict test remains red.
|
|
||||||
That narrows the result: the backout removes the fatal propagation
|
|
||||||
in this reproducer, but doesn’t make bridge teardown clean.
|
|
||||||
|
|
||||||
## Code output pointers
|
|
||||||
|
|
||||||
> `git diff dc618229 -- piker/brokers/ib/feed.py`
|
|
||||||
|
|
||||||
Back out 635314c7's quote owner and import the original adapter-only
|
|
||||||
deltas from 0846cbd4 and 1da9b760.
|
|
||||||
|
|
||||||
> `git diff dc618229 -- piker/clearing/_ems.py`
|
|
||||||
|
|
||||||
Import 12b0d66e's dedicated dark-clearing quote subscription.
|
|
||||||
|
|
||||||
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
|
|
||||||
|
|
||||||
Replace the reverted mock-cache test with a real-runtime restart
|
|
||||||
reproducer; retain original history/EMS tests and add real bridge
|
|
||||||
history recovery and real broadcaster EMS coverage. Newly added
|
|
||||||
test files remain untracked until the human stages them, so the
|
|
||||||
diff pointers include them only after staging or committing.
|
|
||||||
|
|
||||||
## Verification output
|
|
||||||
|
|
||||||
```text
|
|
||||||
21 passed in 0.53s
|
|
||||||
1 failed, 2 passed in 5.53s
|
|
||||||
13 passed in 1.41s
|
|
||||||
1 passed, 1 warning in 0.04s
|
|
||||||
2 passed in 0.03s
|
|
||||||
```
|
|
||||||
|
|
||||||
The last run includes the original EMS case again and one additional
|
|
||||||
real-broadcaster case: 38 unique passes and one strict callback failure.
|
|
||||||
|
|
@ -1,77 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T16:08:46Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 7242dcb9
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261028T160846Z_7242dcb9_websocket_close.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human reported repeated live Binance perp crashes after correct
|
|
||||||
worktree startup: the real trio-websocket reader attempted a Pong
|
|
||||||
while wsproto was LOCAL_CLOSING. They asked whether an existing
|
|
||||||
downstream websocket fix should be reused, then requested:
|
|
||||||
|
|
||||||
> ok add the fix now, follow by a /commit-plan and then i'll test
|
|
||||||
> before saving it ya?
|
|
||||||
|
|
||||||
They approved a real websocket reader/close regression covering
|
|
||||||
recovery and intentional shutdown, and separately authorized
|
|
||||||
reconciling this worktree's legacy workflow bookkeeping while
|
|
||||||
preserving existing archives.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The older `3d83b61f3f7b982e84a4230dba5777f1407c78f1` wrapper
|
|
||||||
suppressed escaped communication failures. The landing replay
|
|
||||||
`97eb8347` of `d50ab6124cd0e1646c82fa4a85d7c13bfff82611`
|
|
||||||
added propagation; it did not restart a reader-context failure.
|
|
||||||
Existing `LocalProtocolError` membership in NoBsWs.recon_errors
|
|
||||||
only protected the receive relay, not reader-nursery context exit.
|
|
||||||
|
|
||||||
Classify these errors around the complete websocket context and
|
|
||||||
retry instead of restoring outer suppression. Place collapse_eg
|
|
||||||
outside that context, invalidate stale connection readiness and
|
|
||||||
socket references, and leave the retry wait cancellable. Preserve
|
|
||||||
unexpected failures and mixed exception groups for propagation.
|
|
||||||
After review, use RuntimeError rather than ValueError for the
|
|
||||||
test-only unexpected-failure sentinel. Its injected exception has
|
|
||||||
no underlying cause. Production recovery logs retain the original
|
|
||||||
traceback and now name the URL; unexpected exceptions use bare
|
|
||||||
raise to preserve identity, existing causes and notes.
|
|
||||||
|
|
||||||
The deterministic seam controls protocol-state ordering in the real
|
|
||||||
client Ping handler. It does not replace the websocket transport,
|
|
||||||
reader, context manager, reconnect task or wsproto exception.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/_web_bs.py`: reader-context failure recovery.
|
|
||||||
- `tests/test_websocket.py`: loopback recovery and two shutdown
|
|
||||||
variants; retain unexpected-failure propagation control.
|
|
||||||
- This provenance pair: live report, lineage and test scope.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
Baseline uses new tests against original HEAD websocket code in an
|
|
||||||
isolated exact-tree checkout. The recovery case fails with the
|
|
||||||
reported Pong-in-LOCAL_CLOSING LocalProtocolError.
|
|
||||||
|
|
||||||
Fixed code passes 13 cases across tests/test_websocket.py and
|
|
||||||
tests/test_binance.py. Exact-boundary verification and native-library
|
|
||||||
configuration are retained in the ignored commit-plan package.
|
|
||||||
Live testing remains human-owned and pending.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
No direct manual source edits observed in this follow-up. The human
|
|
||||||
supplied live failures, corrected the earlier daemon/worktree setup,
|
|
||||||
set the narrow landing scope, requested source-fix reuse, and will
|
|
||||||
test before reviewing and committing. Existing commits are retained.
|
|
||||||
The human requested the RuntimeError sentinel and preservation of
|
|
||||||
underlying error context. The raw partner retains the initial output.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T16:08:46Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 7242dcb9
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
Implemented a focused websocket-context reconnect fix and real
|
|
||||||
loopback reader/close regressions in the Binance landing worktree.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Move exception-group collapse outside the websocket context so
|
|
||||||
reader-nursery exit errors are classified by the reconnect loop.
|
|
||||||
Retry NoBsWs.recon_errors there, clear the stale connected event
|
|
||||||
and socket reference, and honor cancellation through unshielded
|
|
||||||
retry sleep. Retain unexpected-error propagation at the outer API.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
|
||||||
|
|
||||||
Exercise real TCP, handshake, Ping frames, reader dispatch and
|
|
||||||
wsproto Pong validation. A narrow handler gate advances the real
|
|
||||||
protocol state to LOCAL_CLOSING without sending the Close frame,
|
|
||||||
preventing a peer reply from racing the controlled state. Require
|
|
||||||
fresh application data from the replacement connection. Separate
|
|
||||||
variants cancel at the reader gate and at the actual retry wait.
|
|
||||||
Update the existing unexpected-failure control to use ValueError
|
|
||||||
because LocalProtocolError is now intentionally recoverable.
|
|
||||||
|
|
||||||
The isolated original-code/new-test baseline fails with the exact
|
|
||||||
Pong-in-LOCAL_CLOSING error. The fixed isolated tree passes all 13
|
|
||||||
selected Binance/websocket cases. No live provider, broker orders
|
|
||||||
or production daemon operations were performed by the agent.
|
|
||||||
|
|
@ -1,70 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T16:56:38Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: b95e7c79
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261028T165638Z_b95e7c79_reconnect_logging.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human reported that the live Binance Pong-in-LOCAL_CLOSING
|
|
||||||
error now logged a retry without crashing. They requested visible
|
|
||||||
re-establishment reporting under `pikerd -l info --pdb`, and asked
|
|
||||||
to check for existing downstream work before adding a fix.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
The existing `Connection success` INFO message already identifies
|
|
||||||
handshake completion. Feed startup configured piker.data.feed,
|
|
||||||
while shared websocket diagnostics use ._util.log on piker.data.
|
|
||||||
Configure that actual emitter and prefer the daemon's verbosity
|
|
||||||
over a quieter client request for these shared transport tasks.
|
|
||||||
|
|
||||||
Add a readiness INFO message after fixture setup and a first-message
|
|
||||||
INFO report once per connection after actual relay. Include the URL
|
|
||||||
in the receive-timeout reset error. Do not call first raw websocket
|
|
||||||
traffic a parsed quote or proof of restored market-data freshness.
|
|
||||||
|
|
||||||
## Existing-work audit
|
|
||||||
|
|
||||||
Compared `_web_bs.py`, `_util.py`, `piker.log` and daemon logging
|
|
||||||
against the evaluated downstream tip and relevant all-ref histories.
|
|
||||||
`54da2973` contains logging/type cleanup, not this emitter fix.
|
|
||||||
`5adf77a9` / landing equivalents address package-name correctness,
|
|
||||||
but not configuration of the shared data emitter at feed startup.
|
|
||||||
All-ref searches for this actual emitter/readiness/first-relay
|
|
||||||
implementation found no existing commit. The dirty IB history
|
|
||||||
worktree has only an unrelated f-string correction in `_web_bs.py`;
|
|
||||||
the TIPC worktree has no dirty delta in these two feed/websocket
|
|
||||||
files. No downstream work was overwritten or imported wholesale.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/feed.py`: shared transport logger startup/verbosity.
|
|
||||||
- `piker/data/_web_bs.py`: readiness, first-relay and timeout URL.
|
|
||||||
- `tests/test_logging.py`: actual emitter/console startup regression.
|
|
||||||
- `tests/test_websocket.py`: recovery milestones on real loopback.
|
|
||||||
- This prompt provenance pair.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
The new logging regression against original b95e7c79 source fails:
|
|
||||||
the actual websocket emitter remains at WARNING and rejects INFO.
|
|
||||||
It passes with the logger wiring fix. Provider lookup is stopped
|
|
||||||
before network work, while logger factories and console emission
|
|
||||||
are real. Global logger state is restored after the check.
|
|
||||||
|
|
||||||
18 cases pass across tests/test_logging.py, tests/test_websocket.py
|
|
||||||
and tests/test_binance.py. git diff --check passes. Live verification
|
|
||||||
after restarting the daemon with these edits remains human-owned.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
No direct manual source edits observed. The human supplied live
|
|
||||||
recovery evidence, selected INFO verbosity, requested downstream
|
|
||||||
reuse inspection, and retains live-test and commit ownership.
|
|
||||||
|
|
@ -1,38 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T16:56:38Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: b95e7c79
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
Confirmed the existing INFO handshake-success log was hidden by
|
|
||||||
logger wiring: websocket code emits on piker.data, while feed
|
|
||||||
startup configured piker.data.feed. The baseline regression leaves
|
|
||||||
the emitter at WARNING for an INFO daemon and ERROR client.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
|
||||||
|
|
||||||
Configure the actual shared ._util.log namespace, with daemon
|
|
||||||
verbosity taking priority over client verbosity for actor-wide
|
|
||||||
transport diagnostics.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Keep the existing handshake-success message. Add INFO readiness
|
|
||||||
after fixture setup and first-message relay once per connection.
|
|
||||||
Include the URL in the receive-timeout reset diagnostic.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_logging.py tests/test_websocket.py`
|
|
||||||
|
|
||||||
Prove production startup enables real console INFO on the websocket
|
|
||||||
emitter without contacting a provider. Extend the real loopback
|
|
||||||
recovery test to require readiness for both connections and one
|
|
||||||
first-message report for the connection delivering fresh data.
|
|
||||||
|
|
||||||
All 18 targeted logging/Binance/websocket cases passed. No live
|
|
||||||
provider/daemon operations were performed by the agent. First relay
|
|
||||||
is raw websocket traffic; parsed market-data recovery remains a
|
|
||||||
separate live observation.
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T18:08:56Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 032580bf
|
|
||||||
scope: code
|
|
||||||
substantive: true
|
|
||||||
raw_file: 20261028T180856Z_032580bf_timeout_phase.raw.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## Prompt
|
|
||||||
|
|
||||||
The human reported repeated readiness without first-message relay,
|
|
||||||
specifically after first attempting an invalid .usdm. FQME and then
|
|
||||||
the corrected .usdtm. one. They requested distinct log messages for
|
|
||||||
receive starvation versus local backpressure.
|
|
||||||
|
|
||||||
## Response summary
|
|
||||||
|
|
||||||
Add phase-specific WARNING messages for each elapsed relay-loop
|
|
||||||
deadline, with URL, timeout duration, accumulated count and reset
|
|
||||||
threshold. Leave the existing shared receive/send deadline and
|
|
||||||
counter/reset semantics intact. Do not claim that connected/ready
|
|
||||||
sockets alone establish parsed quote recovery.
|
|
||||||
|
|
||||||
Read-only invalid-venue tracing found the usd substring check in
|
|
||||||
Binance get_mkt_info() accepts USDM while retaining it in MktPair.
|
|
||||||
Client.bars() later uses that market identity against canonical
|
|
||||||
USDTM keys. Backend stream startup precedes history startup and
|
|
||||||
sampler consumption. A relationship to the corrected-feed stall
|
|
||||||
still needs runtime evidence; neither venue handling nor task
|
|
||||||
lifetime behavior was changed by this diagnostic patch.
|
|
||||||
|
|
||||||
## Files changed
|
|
||||||
|
|
||||||
- `piker/data/_web_bs.py`: track and report the timed-out operation.
|
|
||||||
- `tests/test_websocket.py`: real silent-source and blocked-sink cases.
|
|
||||||
- This prompt provenance pair.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
15 selected Binance/websocket cases passed. The new cases exercise
|
|
||||||
real TCP, websocket handshake and relay behavior, replacing only
|
|
||||||
log observation. A zero-capacity relay deliberately forces the
|
|
||||||
local-send case; the source-silent case never receives a payload.
|
|
||||||
git diff --check passed. No live provider/daemon operations run.
|
|
||||||
|
|
||||||
## Human edits
|
|
||||||
|
|
||||||
No direct source edits observed. The human supplied the critical
|
|
||||||
invalid-symbol-then-corrected-symbol trigger and requested the
|
|
||||||
phase distinction; live testing and task acceptance remain theirs.
|
|
||||||
|
|
@ -1,34 +0,0 @@
|
||||||
---
|
|
||||||
created_at: 2026-10-28T18:08:56Z
|
|
||||||
generated_by:
|
|
||||||
harness: opencode
|
|
||||||
provider: openai
|
|
||||||
model: openai/gpt-6.1-sol
|
|
||||||
git_ref: 032580bf
|
|
||||||
diff_cmd: git diff HEAD~1..HEAD
|
|
||||||
---
|
|
||||||
|
|
||||||
Added distinct timeout diagnostics for websocket receive starvation
|
|
||||||
versus local relay backpressure, preserving the existing timeout,
|
|
||||||
counter and reconnect threshold policy.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
|
||||||
|
|
||||||
Track whether proxy_msgs() is awaiting ws.get_message() or
|
|
||||||
snd.send(). Report that phase, URL, configured deadline, timeout
|
|
||||||
count and reset threshold for every timeout, before the existing
|
|
||||||
threshold/reset diagnostic. Capture the configured deadline once
|
|
||||||
per operation so the report matches the deadline that elapsed.
|
|
||||||
|
|
||||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
|
||||||
|
|
||||||
Two real loopback variants use a silent server versus one payload
|
|
||||||
and an unconsumed zero-capacity Trio relay. Require the appropriate
|
|
||||||
phase and URL on the actual first timeout. All 15 selected
|
|
||||||
Binance/websocket cases pass. No live provider was contacted.
|
|
||||||
|
|
||||||
Read-only tracing found a permissive invalid-venue path: get_mkt_info
|
|
||||||
accepts USDM via its usd substring check and preserves that venue
|
|
||||||
in MktPair, whereas Client.bars() indexes canonical USDTM pair keys.
|
|
||||||
This identifies a potential late failed-startup path, not proof of
|
|
||||||
the corrected-chart stall or of leaked producer ownership.
|
|
||||||
|
|
@ -1,29 +1,8 @@
|
||||||
[network]
|
[network]
|
||||||
# This Tractor pin supports one transport per actor, not mixed TCP/UDS.
|
|
||||||
# Use the single TCP endpoint shown in each list for this version.
|
|
||||||
# Mixed-transport configs need a future Tractor pin with that support.
|
|
||||||
pikerd = [
|
pikerd = [
|
||||||
# TCP localhost loopback
|
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||||
'/ip4/127.0.0.1/tcp/6116',
|
# '/uds/6116', # TODO std uds socket file
|
||||||
|
|
||||||
# UDS alternative: replace TCP and select the UDS transport.
|
|
||||||
# '/unix/run/user/1000/piker/pikerd.sock',
|
|
||||||
]
|
]
|
||||||
chart = [
|
|
||||||
'/ip4/127.0.0.1/tcp/3003',
|
|
||||||
# '/unix/run/user/1000/piker/chart.sock',
|
|
||||||
]
|
|
||||||
# the service-actor registry endpoint;
|
|
||||||
# other pikerd trees contact this to discover
|
|
||||||
# actors.
|
|
||||||
# XXX if absent, pikerd binds the registry
|
|
||||||
# on its own tpt_bind_addrs.
|
|
||||||
# regd = ['/ip4/127.0.0.1/tcp/6116']
|
|
||||||
|
|
||||||
# chart = [
|
|
||||||
# '/ip4/127.0.0.1/tcp/3333',
|
|
||||||
# '/unix/run/user/1000/piker/chart@3333.sock',
|
|
||||||
# ]
|
|
||||||
|
|
||||||
|
|
||||||
[ui]
|
[ui]
|
||||||
|
|
|
||||||
|
|
@ -2,11 +2,11 @@
|
||||||
"nodes": {
|
"nodes": {
|
||||||
"nixpkgs": {
|
"nixpkgs": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1784497964,
|
"lastModified": 1765779637,
|
||||||
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
|
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||||
"owner": "nixos",
|
"owner": "nixos",
|
||||||
"repo": "nixpkgs",
|
"repo": "nixpkgs",
|
||||||
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
|
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||||
"type": "github"
|
"type": "github"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
|
|
|
||||||
23
flake.nix
23
flake.nix
|
|
@ -22,14 +22,12 @@
|
||||||
|
|
||||||
# do store-path extractions
|
# do store-path extractions
|
||||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||||
zlibStorePath = lib.getLib pkgs.zlib;
|
|
||||||
# ?TODO? can remove below since manual linking not needed?
|
# ?TODO? can remove below since manual linking not needed?
|
||||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||||
|
|
||||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||||
# a major-version-pinned-`cpython`
|
# a major-version-pinned-`cpython`
|
||||||
cpython = "python313";
|
cpython = "python313";
|
||||||
python = pkgs.${cpython};
|
|
||||||
pypkgs = pkgs."${cpython}Packages";
|
pypkgs = pkgs."${cpython}Packages";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
@ -48,7 +46,7 @@
|
||||||
qt6.qtbase
|
qt6.qtbase
|
||||||
|
|
||||||
uv
|
uv
|
||||||
python
|
python313 # ?TODO^ how to set from `cpython` above?
|
||||||
pypkgs.pyqt6
|
pypkgs.pyqt6
|
||||||
pypkgs.pyqt6-sip
|
pypkgs.pyqt6-sip
|
||||||
pypkgs.qtpy
|
pypkgs.qtpy
|
||||||
|
|
@ -72,7 +70,6 @@
|
||||||
|
|
||||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||||
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
|
|
||||||
|
|
||||||
export LD_LIBRARY_PATH
|
export LD_LIBRARY_PATH
|
||||||
|
|
||||||
|
|
@ -89,31 +86,15 @@
|
||||||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||||
|
|
||||||
# ------ uv ------
|
# ------ uv ------
|
||||||
# ignore any venv inherited from an outer checkout
|
|
||||||
unset VIRTUAL_ENV
|
|
||||||
# - always use the ./py313/ venv-subdir
|
# - always use the ./py313/ venv-subdir
|
||||||
export UV_PROJECT_ENVIRONMENT="py313"
|
export UV_PROJECT_ENVIRONMENT="py313"
|
||||||
|
|
||||||
# Rebuild after a flake update changes the Nix Python closure.
|
|
||||||
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
|
|
||||||
nix_python="$(readlink -f "${python}/bin/python3")"
|
|
||||||
if test -e "$venv_python" \
|
|
||||||
&& test "$(readlink -f "$venv_python")" != "$nix_python"
|
|
||||||
then
|
|
||||||
uv venv \
|
|
||||||
--clear \
|
|
||||||
--python "$nix_python" \
|
|
||||||
"$UV_PROJECT_ENVIRONMENT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# sync project-env with all extras
|
# sync project-env with all extras
|
||||||
uv sync --dev --all-extras --no-group lint
|
uv sync --dev --all-extras --no-group lint
|
||||||
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
|
|
||||||
|
|
||||||
# ------ TIPS ------
|
# ------ TIPS ------
|
||||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||||
# run the `nix develop` cmd with,
|
# run the `nix develop` cmd with,
|
||||||
# >> nix develop -c xonsh
|
# >> nix develop -c uv run xonsh
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -324,13 +324,10 @@ def load_ledger(
|
||||||
ldir: Path = (
|
ldir: Path = (
|
||||||
dirpath
|
dirpath
|
||||||
or
|
or
|
||||||
config.get_conf_dir() / 'accounting' / 'ledgers'
|
config._config_dir / 'accounting' / 'ledgers'
|
||||||
)
|
)
|
||||||
if not ldir.is_dir():
|
if not ldir.is_dir():
|
||||||
ldir.mkdir(
|
ldir.mkdir()
|
||||||
parents=True,
|
|
||||||
exist_ok=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
fname = f'trades_{brokername}_{acctid}.toml'
|
fname = f'trades_{brokername}_{acctid}.toml'
|
||||||
fpath: Path = ldir / fname
|
fpath: Path = ldir / fname
|
||||||
|
|
|
||||||
|
|
@ -785,16 +785,9 @@ def load_account(
|
||||||
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
||||||
fn: str = f'account.{brokername}.{acctid}.toml'
|
fn: str = f'account.{brokername}.{acctid}.toml'
|
||||||
|
|
||||||
dirpath: Path = (
|
dirpath: Path = dirpath or (config._config_dir / 'accounting')
|
||||||
dirpath
|
|
||||||
or
|
|
||||||
(config.get_conf_dir() / 'accounting')
|
|
||||||
)
|
|
||||||
if not dirpath.is_dir():
|
if not dirpath.is_dir():
|
||||||
dirpath.mkdir(
|
dirpath.mkdir()
|
||||||
parents=True,
|
|
||||||
exist_ok=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
conf, path = config.load(
|
conf, path = config.load(
|
||||||
path=dirpath / fn,
|
path=dirpath / fn,
|
||||||
|
|
|
||||||
|
|
@ -143,15 +143,12 @@ def sync(
|
||||||
# (what the EMS normally does internall) B)
|
# (what the EMS normally does internall) B)
|
||||||
open_brokerd_dialog(
|
open_brokerd_dialog(
|
||||||
brokermod,
|
brokermod,
|
||||||
|
portal,
|
||||||
exec_mode=(
|
exec_mode=(
|
||||||
'paper'
|
'paper'
|
||||||
if account == 'paper'
|
if account == 'paper'
|
||||||
else 'live'
|
else 'live'
|
||||||
),
|
),
|
||||||
# use our own ad-hoc-spawned actor,
|
|
||||||
# do NOT (spawn and) use the
|
|
||||||
# `brokerd.<broker>` service daemon!
|
|
||||||
portal=portal,
|
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
) as (
|
) as (
|
||||||
brokerd_stream,
|
brokerd_stream,
|
||||||
|
|
|
||||||
|
|
@ -25,8 +25,10 @@ from contextlib import (
|
||||||
)
|
)
|
||||||
from types import ModuleType
|
from types import ModuleType
|
||||||
from typing import (
|
from typing import (
|
||||||
|
TYPE_CHECKING,
|
||||||
AsyncContextManager,
|
AsyncContextManager,
|
||||||
)
|
)
|
||||||
|
import exceptiongroup as eg
|
||||||
|
|
||||||
import tractor
|
import tractor
|
||||||
import trio
|
import trio
|
||||||
|
|
@ -38,20 +40,27 @@ from piker.log import (
|
||||||
from . import _util
|
from . import _util
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from ..data import _FeedsBus
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
log = get_logger(name=__name__)
|
||||||
|
|
||||||
# `brokerd`-actor-always-enabled mods.
|
# `brokerd` enabled modules
|
||||||
|
# TODO: move this def to the `.data` subpkg..
|
||||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||||
# model and should be treated with utmost care! In particular NO
|
# model and should be treated with utmost care!
|
||||||
# `piker.data.*` feed mods should be enabled in this (live,
|
_data_mods: str = [
|
||||||
# credentialed) trading actor; all data-feed serving is the
|
'piker.brokers.core',
|
||||||
# domain of the `datad.<broker>` sibling daemon, see
|
'piker.brokers.data',
|
||||||
# `piker.data._daemon._datad_service_mods`.
|
|
||||||
_brokerd_service_mods: list[str] = [
|
|
||||||
'piker.brokers._daemon',
|
'piker.brokers._daemon',
|
||||||
|
'piker.data',
|
||||||
|
'piker.data.feed',
|
||||||
|
'piker.data._sampling'
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# TODO: we should rename the daemon to datad prolly once we split up
|
||||||
|
# broker vs. data tasks into separate actors?
|
||||||
@tractor.context
|
@tractor.context
|
||||||
async def _setup_persistent_brokerd(
|
async def _setup_persistent_brokerd(
|
||||||
ctx: tractor.Context,
|
ctx: tractor.Context,
|
||||||
|
|
@ -60,15 +69,9 @@ async def _setup_persistent_brokerd(
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Trading-only daemon (lifetime) fixture: console logging
|
Allocate a actor-wide service nursery in ``brokerd``
|
||||||
setup and a pinned-open context for service mgmt.
|
such that feeds can be run in the background persistently by
|
||||||
|
the broker backend as needed.
|
||||||
All data-feed-bus state now lives in the (data-feed-only)
|
|
||||||
`datad.<brokername>` sibling daemon, see
|
|
||||||
`piker.data._daemon._setup_persistent_datad()`; this
|
|
||||||
actor hosts only the backend's `open_trade_dialog()`
|
|
||||||
(live order-control) ep-task(s) which manage their own
|
|
||||||
task trees per `tractor.Context`.
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
# NOTE: we only need to setup logging once (and only) here
|
# NOTE: we only need to setup logging once (and only) here
|
||||||
|
|
@ -77,19 +80,54 @@ async def _setup_persistent_brokerd(
|
||||||
# any further (level) configuration on their own B)
|
# any further (level) configuration on their own B)
|
||||||
actor: tractor.Actor = tractor.current_actor()
|
actor: tractor.Actor = tractor.current_actor()
|
||||||
tll: str = actor.loglevel
|
tll: str = actor.loglevel
|
||||||
get_console_log(
|
log = get_console_log(
|
||||||
level=loglevel or tll,
|
level=loglevel or tll,
|
||||||
name=f'{_util.subsys}.{brokername}',
|
name=f'{_util.subsys}.{brokername}',
|
||||||
with_tractor_log=bool(tll),
|
with_tractor_log=bool(tll),
|
||||||
)
|
)
|
||||||
|
assert log.name == _util.subsys
|
||||||
|
|
||||||
|
# further, set the log level on any broker broker specific
|
||||||
|
# logger instance.
|
||||||
|
|
||||||
|
from piker.data import feed
|
||||||
|
assert not feed._bus
|
||||||
|
|
||||||
|
# allocate a nursery to the bus for spawning background
|
||||||
|
# tasks to service client IPC requests, normally
|
||||||
|
# `tractor.Context` connections to explicitly required
|
||||||
|
# `brokerd` endpoints such as:
|
||||||
|
# - `stream_quotes()`,
|
||||||
|
# - `manage_history()`,
|
||||||
|
# - `allocate_persistent_feed()`,
|
||||||
|
# - `open_symbol_search()`
|
||||||
|
# NOTE: see ep invocation details inside `.data.feed`.
|
||||||
|
try:
|
||||||
|
async with (
|
||||||
|
# tractor.trionics.collapse_eg(),
|
||||||
|
trio.open_nursery() as service_nursery
|
||||||
|
):
|
||||||
|
bus: _FeedsBus = feed.get_feed_bus(
|
||||||
|
brokername,
|
||||||
|
service_nursery,
|
||||||
|
)
|
||||||
|
assert bus is feed._bus
|
||||||
|
|
||||||
# unblock caller
|
# unblock caller
|
||||||
await ctx.started()
|
await ctx.started()
|
||||||
|
|
||||||
# we pin this task to keep the daemon active until the
|
# we pin this task to keep the feeds manager active until the
|
||||||
# parent actor decides to tear it down
|
# parent actor decides to tear it down
|
||||||
await trio.sleep_forever()
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
except eg.ExceptionGroup:
|
||||||
|
# TODO: likely some underlying `brokerd` IPC connection
|
||||||
|
# broke so here we handle a respawn and re-connect attempt!
|
||||||
|
# This likely should pair with development of the OCO task
|
||||||
|
# nusery in dev over @ `tractor` B)
|
||||||
|
# https://github.com/goodboy/tractor/pull/363
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
def broker_init(
|
def broker_init(
|
||||||
brokername: str,
|
brokername: str,
|
||||||
|
|
@ -109,10 +147,8 @@ def broker_init(
|
||||||
|
|
||||||
This includes:
|
This includes:
|
||||||
- load the appropriate <brokername>.py pkg module,
|
- load the appropriate <brokername>.py pkg module,
|
||||||
- reads any declared `_brokerd_mods: list[str]` (falling
|
- reads any declared `__enable_modules__: listr[str]` which will be
|
||||||
back to the full `__enable_modules__` set for
|
passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
|
||||||
not-yet-split backends) which will be passed to
|
|
||||||
`tractor.ActorNursery.start_actor(enable_modules=)`
|
|
||||||
at actor start time,
|
at actor start time,
|
||||||
- deliver a references to the daemon lifetime fixture, which
|
- deliver a references to the daemon lifetime fixture, which
|
||||||
for now is always the `_setup_persistent_brokerd()` context defined
|
for now is always the `_setup_persistent_brokerd()` context defined
|
||||||
|
|
@ -146,15 +182,9 @@ def broker_init(
|
||||||
modpath,
|
modpath,
|
||||||
]
|
]
|
||||||
for submodname in getattr(
|
for submodname in getattr(
|
||||||
brokermod,
|
|
||||||
'_brokerd_mods',
|
|
||||||
# fallback for (flat, less mature) backends which
|
|
||||||
# don't yet declare a daemon-kind mod split.
|
|
||||||
getattr(
|
|
||||||
brokermod,
|
brokermod,
|
||||||
'__enable_modules__',
|
'__enable_modules__',
|
||||||
[],
|
[],
|
||||||
),
|
|
||||||
):
|
):
|
||||||
subpath: str = f'{modpath}.{submodname}'
|
subpath: str = f'{modpath}.{submodname}'
|
||||||
enabled.append(subpath)
|
enabled.append(subpath)
|
||||||
|
|
@ -182,22 +212,6 @@ async def spawn_brokerd(
|
||||||
f'backend: {brokername!r}'
|
f'backend: {brokername!r}'
|
||||||
)
|
)
|
||||||
|
|
||||||
# fail fast on (data-only) backends which don't offer
|
|
||||||
# ANY live order-control eps; the caller should instead
|
|
||||||
# be using paper-mode (and thus never spawning us)!
|
|
||||||
from ..data.validate import get_eps
|
|
||||||
brokerd_eps: dict = get_eps(
|
|
||||||
get_brokermod(brokername),
|
|
||||||
'brokerd',
|
|
||||||
)
|
|
||||||
if not brokerd_eps:
|
|
||||||
raise RuntimeError(
|
|
||||||
f'Backend {brokername!r} offers NO `brokerd` '
|
|
||||||
f'(live order-control) eps!?\n'
|
|
||||||
f'It is likely a datad-only provider, use '
|
|
||||||
f'paper-mode for clearing instead.\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
(
|
(
|
||||||
brokermode,
|
brokermode,
|
||||||
tractor_kwargs,
|
tractor_kwargs,
|
||||||
|
|
@ -219,11 +233,7 @@ async def spawn_brokerd(
|
||||||
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
||||||
portal = await Services.actor_n.start_actor(
|
portal = await Services.actor_n.start_actor(
|
||||||
dname,
|
dname,
|
||||||
enable_modules=list(dict.fromkeys(
|
enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
|
||||||
_brokerd_service_mods
|
|
||||||
+
|
|
||||||
tractor_kwargs.pop('enable_modules')
|
|
||||||
)),
|
|
||||||
debug_mode=Services.debug_mode,
|
debug_mode=Services.debug_mode,
|
||||||
**tractor_kwargs
|
**tractor_kwargs
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -20,17 +20,10 @@ Handy cross-broker utils.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
# from functools import partial
|
# from functools import partial
|
||||||
from typing import (
|
|
||||||
Type,
|
|
||||||
)
|
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import httpx
|
import httpx
|
||||||
import logging
|
import logging
|
||||||
from msgspec import Struct
|
|
||||||
from tractor._exceptions import (
|
|
||||||
reg_err_types,
|
|
||||||
)
|
|
||||||
|
|
||||||
from piker.log import (
|
from piker.log import (
|
||||||
colorize_json,
|
colorize_json,
|
||||||
|
|
@ -104,19 +97,6 @@ class DataThrottle(BrokerError):
|
||||||
'''
|
'''
|
||||||
# TODO: add in throttle metrics/feedback
|
# TODO: add in throttle metrics/feedback
|
||||||
|
|
||||||
class SchemaMismatch(BrokerError):
|
|
||||||
'''
|
|
||||||
Market `Pair` fields mismatch, likely due to provider API update.
|
|
||||||
|
|
||||||
'''
|
|
||||||
|
|
||||||
# auto-register all `BrokerError` subtypes for
|
|
||||||
# tractor IPC exc-marshalling.
|
|
||||||
reg_err_types([
|
|
||||||
BrokerError,
|
|
||||||
*BrokerError.__subclasses__(),
|
|
||||||
])
|
|
||||||
|
|
||||||
|
|
||||||
def resproc(
|
def resproc(
|
||||||
resp: httpx.Response,
|
resp: httpx.Response,
|
||||||
|
|
@ -143,45 +123,3 @@ def resproc(
|
||||||
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
||||||
|
|
||||||
return msg if return_json else resp
|
return msg if return_json else resp
|
||||||
|
|
||||||
|
|
||||||
def get_or_raise_on_pair_schema_mismatch(
|
|
||||||
pair_type: Type[Struct],
|
|
||||||
fields_data: dict,
|
|
||||||
provider_name: str,
|
|
||||||
api_url: str|None = None,
|
|
||||||
) -> Struct:
|
|
||||||
'''
|
|
||||||
Boilerplate helper around assset-`Pair` field schema mismatches,
|
|
||||||
normally due to provider API updates.
|
|
||||||
|
|
||||||
'''
|
|
||||||
try:
|
|
||||||
pair: Struct = pair_type(**fields_data)
|
|
||||||
return pair
|
|
||||||
except TypeError as err:
|
|
||||||
|
|
||||||
from tractor.devx.pformat import ppfmt
|
|
||||||
repr_data: str = ppfmt(fields_data)
|
|
||||||
report: str = (
|
|
||||||
f'Field mismatch we need to codify!\n'
|
|
||||||
f'\n'
|
|
||||||
f'{pair_type!r}({repr_data})'
|
|
||||||
f'\n'
|
|
||||||
f'^^^ {err.args[0]!r} ^^^\n'
|
|
||||||
f'\n'
|
|
||||||
f"Don't panic, prolly {provider_name!r} "
|
|
||||||
f"changed their symbology schema..\n"
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
api_url
|
|
||||||
or
|
|
||||||
(api_url := pair_type._api_url)
|
|
||||||
):
|
|
||||||
report += (
|
|
||||||
f'\n'
|
|
||||||
f'Check out their API docs here:\n'
|
|
||||||
f'{api_url}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
raise SchemaMismatch(report) from err
|
|
||||||
|
|
|
||||||
|
|
@ -52,25 +52,9 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# `brokerd` modules
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
# NOTE: `get_mkt_info` and `open_symbol_search` both live
|
|
||||||
# in `.feed` for this backend (no `symbols.py`).
|
|
||||||
_brokerd_mods: list[str] = [
|
|
||||||
'api',
|
|
||||||
'broker',
|
|
||||||
]
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
|
'broker',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
|
||||||
|
|
@ -49,9 +49,6 @@ from piker import config
|
||||||
from piker.clearing._messages import (
|
from piker.clearing._messages import (
|
||||||
Order,
|
Order,
|
||||||
)
|
)
|
||||||
from piker.brokers._util import (
|
|
||||||
get_or_raise_on_pair_schema_mismatch,
|
|
||||||
)
|
|
||||||
from piker.accounting import (
|
from piker.accounting import (
|
||||||
Asset,
|
Asset,
|
||||||
digits_to_dec,
|
digits_to_dec,
|
||||||
|
|
@ -373,12 +370,20 @@ class Client:
|
||||||
item['filters'] = filters
|
item['filters'] = filters
|
||||||
|
|
||||||
pair_type: Type = PAIRTYPES[venue]
|
pair_type: Type = PAIRTYPES[venue]
|
||||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
try:
|
||||||
pair_type=pair_type,
|
pair: Pair = pair_type(**item)
|
||||||
fields_data=item,
|
except Exception as e:
|
||||||
provider_name='binance',
|
e.add_note(
|
||||||
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
f'\n'
|
||||||
|
f'New or removed field we need to codify!\n'
|
||||||
|
f'pair-type: {pair_type!r}\n'
|
||||||
|
f'\n'
|
||||||
|
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||||
|
f'Check out their API docs here:\n'
|
||||||
|
f'\n'
|
||||||
|
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||||
)
|
)
|
||||||
|
raise
|
||||||
pair_table[pair.symbol.upper()] = pair
|
pair_table[pair.symbol.upper()] = pair
|
||||||
|
|
||||||
# update an additional top-level-cross-venue-table
|
# update an additional top-level-cross-venue-table
|
||||||
|
|
@ -576,8 +581,8 @@ class Client:
|
||||||
self,
|
self,
|
||||||
mkt: MktPair,
|
mkt: MktPair,
|
||||||
|
|
||||||
start_dt: datetime|None = None,
|
start_dt: datetime | None = None,
|
||||||
end_dt: datetime|None = None,
|
end_dt: datetime | None = None,
|
||||||
|
|
||||||
as_np: bool = True,
|
as_np: bool = True,
|
||||||
|
|
||||||
|
|
@ -604,11 +609,7 @@ class Client:
|
||||||
start_time = binance_timestamp(start_dt)
|
start_time = binance_timestamp(start_dt)
|
||||||
end_time = binance_timestamp(end_dt)
|
end_time = binance_timestamp(end_dt)
|
||||||
|
|
||||||
import tractor
|
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
||||||
with tractor.devx.maybe_open_crash_handler():
|
|
||||||
bs_pair: Pair = self._pairs[
|
|
||||||
mkt.bs_fqme.upper()
|
|
||||||
]
|
|
||||||
|
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||||
bars = await self._api(
|
bars = await self._api(
|
||||||
|
|
|
||||||
|
|
@ -22,14 +22,18 @@ from __future__ import annotations
|
||||||
from contextlib import (
|
from contextlib import (
|
||||||
asynccontextmanager as acm,
|
asynccontextmanager as acm,
|
||||||
aclosing,
|
aclosing,
|
||||||
AsyncExitStack,
|
|
||||||
)
|
)
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
from functools import (
|
||||||
|
partial,
|
||||||
|
)
|
||||||
|
import itertools
|
||||||
from pprint import pformat
|
from pprint import pformat
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
AsyncGenerator,
|
AsyncGenerator,
|
||||||
Callable,
|
Callable,
|
||||||
|
Generator,
|
||||||
)
|
)
|
||||||
import time
|
import time
|
||||||
|
|
||||||
|
|
@ -44,7 +48,6 @@ import tractor
|
||||||
from piker.brokers import (
|
from piker.brokers import (
|
||||||
open_cached_client,
|
open_cached_client,
|
||||||
NoData,
|
NoData,
|
||||||
SymbolNotFound,
|
|
||||||
)
|
)
|
||||||
from piker._cacheables import (
|
from piker._cacheables import (
|
||||||
async_lifo_cache,
|
async_lifo_cache,
|
||||||
|
|
@ -56,10 +59,7 @@ from piker.accounting import (
|
||||||
unpack_fqme,
|
unpack_fqme,
|
||||||
)
|
)
|
||||||
from piker.types import Struct
|
from piker.types import Struct
|
||||||
from piker.data.validate import (
|
from piker.data.validate import FeedInit
|
||||||
FeedInit,
|
|
||||||
FeedInitializationError,
|
|
||||||
)
|
|
||||||
from piker.data._web_bs import (
|
from piker.data._web_bs import (
|
||||||
open_autorecon_ws,
|
open_autorecon_ws,
|
||||||
NoBsWs,
|
NoBsWs,
|
||||||
|
|
@ -76,15 +76,13 @@ from .venues import (
|
||||||
Pair,
|
Pair,
|
||||||
FutesPair,
|
FutesPair,
|
||||||
get_api_eps,
|
get_api_eps,
|
||||||
_futes_market_ws,
|
|
||||||
_futes_public_ws,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
log = get_logger(name=__name__)
|
||||||
|
|
||||||
|
|
||||||
class L1(Struct):
|
class L1(Struct):
|
||||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Individual-Symbol-Book-Ticker-Streams
|
# https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
|
||||||
|
|
||||||
update_id: int
|
update_id: int
|
||||||
sym: str
|
sym: str
|
||||||
|
|
@ -109,15 +107,14 @@ class AggTrade(Struct, frozen=True):
|
||||||
T: int # Trade time
|
T: int # Trade time
|
||||||
m: bool # Is the buyer the market maker?
|
m: bool # Is the buyer the market maker?
|
||||||
M: bool|None = None # Ignore
|
M: bool|None = None # Ignore
|
||||||
nq: float|None = None # Normal quantity without RPI-order trades
|
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
||||||
# ^XXX: https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||||
st: int|None = None # 1 for USD-M, 2 for COIN-M
|
|
||||||
|
|
||||||
|
|
||||||
async def stream_messages(
|
async def stream_messages(
|
||||||
ws: NoBsWs,
|
ws: NoBsWs,
|
||||||
|
|
||||||
) -> AsyncGenerator[tuple[str, dict], None]:
|
) -> AsyncGenerator[NoBsWs, dict]:
|
||||||
|
|
||||||
# TODO: match syntax here!
|
# TODO: match syntax here!
|
||||||
msg: dict[str, Any]
|
msg: dict[str, Any]
|
||||||
|
|
@ -206,6 +203,27 @@ async def stream_messages(
|
||||||
yield 'trade', piker_quote
|
yield 'trade', piker_quote
|
||||||
|
|
||||||
|
|
||||||
|
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
||||||
|
'''
|
||||||
|
Create a request subscription packet dict.
|
||||||
|
|
||||||
|
- spot:
|
||||||
|
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
||||||
|
|
||||||
|
- futes:
|
||||||
|
https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||||
|
|
||||||
|
'''
|
||||||
|
return {
|
||||||
|
'method': 'SUBSCRIBE',
|
||||||
|
'params': [
|
||||||
|
f'{pair.lower()}@{sub_name}'
|
||||||
|
for pair in pairs
|
||||||
|
],
|
||||||
|
'id': uid
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||||
# code?!
|
# code?!
|
||||||
@acm
|
@acm
|
||||||
|
|
@ -283,10 +301,6 @@ async def get_mkt_info(
|
||||||
|
|
||||||
# uppercase since kraken bs_mktid is always upper
|
# uppercase since kraken bs_mktid is always upper
|
||||||
if 'binance' not in fqme.lower():
|
if 'binance' not in fqme.lower():
|
||||||
log.warning(
|
|
||||||
f'Missing `.<provider>` part in fqme ??\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
fqme += '.binance'
|
fqme += '.binance'
|
||||||
|
|
||||||
mkt_mode: str = ''
|
mkt_mode: str = ''
|
||||||
|
|
@ -301,12 +315,6 @@ async def get_mkt_info(
|
||||||
venue: str = venue.upper()
|
venue: str = venue.upper()
|
||||||
venue_lower: str = venue.lower()
|
venue_lower: str = venue.lower()
|
||||||
|
|
||||||
if not venue:
|
|
||||||
raise SymbolNotFound(
|
|
||||||
f'Invalid or missing .<venue> part in fqme?\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
# XXX TODO: we should change the usdtm_futes name to just
|
# XXX TODO: we should change the usdtm_futes name to just
|
||||||
# usdm_futes (dropping the tether part) since it turns out that
|
# usdm_futes (dropping the tether part) since it turns out that
|
||||||
# there are indeed USD-tokens OTHER THEN tether being used as
|
# there are indeed USD-tokens OTHER THEN tether being used as
|
||||||
|
|
@ -398,153 +406,54 @@ async def get_mkt_info(
|
||||||
return mkt, pair
|
return mkt, pair
|
||||||
|
|
||||||
|
|
||||||
def get_quote_routes(
|
|
||||||
mkt_mode: str,
|
|
||||||
symbols: list[str],
|
|
||||||
) -> tuple[str, ...]:
|
|
||||||
'''
|
|
||||||
Render websocket URLs for each required Binance stream class.
|
|
||||||
|
|
||||||
'''
|
|
||||||
if len(symbols) != 1:
|
|
||||||
raise ValueError(
|
|
||||||
'Binance quote routes require exactly one symbol; '
|
|
||||||
f'got {symbols!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
symbol: str = symbols[0]
|
|
||||||
sub_names: tuple[str, str] = ('bookTicker', 'aggTrade')
|
|
||||||
if mkt_mode == 'usdtm_futes':
|
|
||||||
return tuple(
|
|
||||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
|
||||||
for wss_url, sub_name in (
|
|
||||||
(_futes_public_ws, sub_names[0]),
|
|
||||||
(_futes_market_ws, sub_names[1]),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
wss_url: str = get_api_eps(mkt_mode)[1]
|
|
||||||
return tuple(
|
|
||||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
|
||||||
for sub_name in sub_names
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def relay_messages(
|
|
||||||
ws: NoBsWs,
|
|
||||||
send_chan: trio.MemorySendChannel,
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Relay one Binance websocket into a merged quote channel.
|
|
||||||
|
|
||||||
'''
|
|
||||||
async with (
|
|
||||||
send_chan,
|
|
||||||
aclosing(stream_messages(ws)) as msg_gen,
|
|
||||||
):
|
|
||||||
async for msg in msg_gen:
|
|
||||||
await send_chan.send(msg)
|
|
||||||
|
|
||||||
|
|
||||||
# TODO: Encapsulate ``relay_messages()`` and ``open_quote_stream()``
|
|
||||||
# inside ``stream_quotes()`` so async-generator closure is owned by
|
|
||||||
# the backend feed task.
|
|
||||||
@acm
|
@acm
|
||||||
async def open_quote_stream(
|
async def subscribe(
|
||||||
mkt_mode: str,
|
ws: NoBsWs,
|
||||||
symbols: list[str],
|
symbols: list[str],
|
||||||
connect_timeout: float = 16,
|
|
||||||
) -> AsyncGenerator[trio.MemoryReceiveChannel, None]:
|
|
||||||
'''
|
|
||||||
Open and merge the websocket routes required by a market.
|
|
||||||
|
|
||||||
'''
|
# defined once at import time to keep a global state B)
|
||||||
quote_routes: tuple[str, ...] = get_quote_routes(
|
iter_subids: Generator[int, None, None] = itertools.count(),
|
||||||
mkt_mode=mkt_mode,
|
|
||||||
symbols=symbols,
|
|
||||||
)
|
|
||||||
|
|
||||||
send, recv = trio.open_memory_channel(616)
|
):
|
||||||
async with AsyncExitStack() as stack:
|
# setup subs
|
||||||
websockets: list[NoBsWs] = []
|
|
||||||
try:
|
|
||||||
for wss_url in quote_routes:
|
|
||||||
ws: NoBsWs = await stack.enter_async_context(
|
|
||||||
open_autorecon_ws(
|
|
||||||
url=wss_url,
|
|
||||||
connect_timeout=connect_timeout,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
websockets.append(ws)
|
|
||||||
|
|
||||||
except (
|
subid: int = next(iter_subids)
|
||||||
TimeoutError,
|
|
||||||
*NoBsWs.recon_errors,
|
|
||||||
) as err:
|
|
||||||
routes_str: str = '\n'.join(quote_routes)
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'Binance websocket connection failed for '
|
|
||||||
f'{symbols!r}:\n'
|
|
||||||
f'{routes_str}'
|
|
||||||
) from err
|
|
||||||
|
|
||||||
async with trio.open_nursery() as tn:
|
# trade data (aka L1)
|
||||||
for ws in websockets:
|
# https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
|
||||||
tn.start_soon(
|
l1_sub = make_sub(symbols, 'bookTicker', subid)
|
||||||
relay_messages,
|
await ws.send_msg(l1_sub)
|
||||||
ws,
|
|
||||||
send.clone(),
|
|
||||||
)
|
|
||||||
|
|
||||||
await send.aclose()
|
# aggregate (each order clear by taker **not** by maker)
|
||||||
async with recv:
|
# trades data:
|
||||||
try:
|
# https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
|
||||||
yield recv
|
agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
|
||||||
finally:
|
await ws.send_msg(agg_trades_sub)
|
||||||
tn.cancel_scope.cancel()
|
|
||||||
|
|
||||||
|
# might get ack from ws server, or maybe some
|
||||||
|
# other msg still in transit..
|
||||||
|
res = await ws.recv_msg()
|
||||||
|
subid: str|None = res.get('id')
|
||||||
|
if subid:
|
||||||
|
assert res['id'] == subid
|
||||||
|
|
||||||
async def wait_for_live_quotes(
|
yield
|
||||||
msg_stream: AsyncGenerator | trio.MemoryReceiveChannel,
|
|
||||||
symbols: list[str],
|
|
||||||
startup_timeout: float = 16,
|
|
||||||
) -> dict:
|
|
||||||
'''
|
|
||||||
Require L1 or trade data before declaring a feed live.
|
|
||||||
|
|
||||||
'''
|
subs = []
|
||||||
if len(symbols) != 1:
|
for sym in symbols:
|
||||||
raise ValueError(
|
subs.append("{sym}@aggTrade")
|
||||||
'Binance quote readiness requires exactly one symbol; '
|
subs.append("{sym}@bookTicker")
|
||||||
f'got {symbols!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
symbol: str = symbols[0]
|
# unsub from all pairs on teardown
|
||||||
msg_iter = aiter(msg_stream)
|
if ws.connected():
|
||||||
|
await ws.send_msg({
|
||||||
|
"method": "UNSUBSCRIBE",
|
||||||
|
"params": subs,
|
||||||
|
"id": subid,
|
||||||
|
})
|
||||||
|
|
||||||
try:
|
# XXX: do we need to ack the unsub?
|
||||||
with trio.fail_after(startup_timeout):
|
# await ws.recv_msg()
|
||||||
while True:
|
|
||||||
typ, quote = await anext(msg_iter)
|
|
||||||
if (
|
|
||||||
typ in {'l1', 'trade'}
|
|
||||||
and
|
|
||||||
quote['symbol'] == symbol
|
|
||||||
):
|
|
||||||
return quote
|
|
||||||
|
|
||||||
except trio.TooSlowError as err:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'Binance quote startup timed out after '
|
|
||||||
f'{startup_timeout}s; '
|
|
||||||
f'no live data received for {symbol}'
|
|
||||||
) from err
|
|
||||||
|
|
||||||
except StopAsyncIteration as err:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'Binance quote streams closed during startup; '
|
|
||||||
f'no live data received for {symbol}'
|
|
||||||
) from err
|
|
||||||
|
|
||||||
|
|
||||||
async def stream_quotes(
|
async def stream_quotes(
|
||||||
|
|
@ -556,46 +465,49 @@ async def stream_quotes(
|
||||||
# startup sync
|
# startup sync
|
||||||
task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
|
task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
if len(symbols) != 1:
|
|
||||||
# ``data.feed.allocate_persistent_feed()`` currently starts
|
|
||||||
# one backend task per symbol. Binance multiplexing remains
|
|
||||||
# deferred until that caller contract can represent
|
|
||||||
# multi-symbol startup.
|
|
||||||
raise FeedInitializationError(
|
|
||||||
'Binance feeds require exactly one symbol; '
|
|
||||||
f'got {symbols!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||||
send_chan as send_chan,
|
send_chan as send_chan,
|
||||||
|
open_cached_client('binance') as client,
|
||||||
):
|
):
|
||||||
|
init_msgs: list[FeedInit] = []
|
||||||
|
for sym in symbols:
|
||||||
mkt: MktPair
|
mkt: MktPair
|
||||||
pair: Pair
|
pair: Pair
|
||||||
mkt, pair = await get_mkt_info(symbols[0])
|
mkt, pair = await get_mkt_info(sym)
|
||||||
mkt_mode: str = (
|
|
||||||
'usdtm_futes'
|
|
||||||
if isinstance(pair, FutesPair)
|
|
||||||
else 'spot'
|
|
||||||
)
|
|
||||||
|
|
||||||
# build out init msgs according to latest spec
|
# build out init msgs according to latest spec
|
||||||
init_msgs: list[FeedInit] = [
|
init_msgs.append(
|
||||||
FeedInit(mkt_info=mkt),
|
FeedInit(mkt_info=mkt)
|
||||||
]
|
|
||||||
|
|
||||||
ws_symbols: list[str] = [mkt.bs_mktid]
|
|
||||||
async with open_quote_stream(
|
|
||||||
mkt_mode=mkt_mode,
|
|
||||||
symbols=ws_symbols,
|
|
||||||
) as msg_stream:
|
|
||||||
quote: dict = await wait_for_live_quotes(
|
|
||||||
msg_stream=msg_stream,
|
|
||||||
symbols=ws_symbols,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
wss_url: str = get_api_eps(client.mkt_mode)[1] # 2nd elem is wss url
|
||||||
|
|
||||||
|
# TODO: for sanity, but remove eventually Xp
|
||||||
|
if 'future' in mkt.type_key:
|
||||||
|
assert 'fstream' in wss_url
|
||||||
|
|
||||||
|
async with (
|
||||||
|
open_autorecon_ws(
|
||||||
|
url=wss_url,
|
||||||
|
fixture=partial(
|
||||||
|
subscribe,
|
||||||
|
symbols=[mkt.bs_mktid],
|
||||||
|
),
|
||||||
|
) as ws,
|
||||||
|
|
||||||
|
# avoid stream-gen closure from breaking trio..
|
||||||
|
aclosing(stream_messages(ws)) as msg_gen,
|
||||||
|
):
|
||||||
|
# log.info('WAITING ON FIRST LIVE QUOTE..')
|
||||||
|
typ, quote = await anext(msg_gen)
|
||||||
|
|
||||||
|
# pull a first quote and deliver
|
||||||
|
while typ != 'trade':
|
||||||
|
typ, quote = await anext(msg_gen)
|
||||||
|
|
||||||
task_status.started((init_msgs, quote))
|
task_status.started((init_msgs, quote))
|
||||||
|
|
||||||
# signal to caller feed is ready for consumption
|
# signal to caller feed is ready for consumption
|
||||||
|
|
@ -611,7 +523,7 @@ async def stream_quotes(
|
||||||
topic: str = mkt.bs_fqme
|
topic: str = mkt.bs_fqme
|
||||||
|
|
||||||
# start streaming
|
# start streaming
|
||||||
async for typ, quote in msg_stream:
|
async for typ, quote in msg_gen:
|
||||||
# period = time.time() - last
|
# period = time.time() - last
|
||||||
# hz = 1/period if period else float('inf')
|
# hz = 1/period if period else float('inf')
|
||||||
# if hz > 60:
|
# if hz > 60:
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,6 @@ Per market data-type definitions and schemas types.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import (
|
from typing import (
|
||||||
ClassVar,
|
|
||||||
Literal,
|
Literal,
|
||||||
)
|
)
|
||||||
from decimal import Decimal
|
from decimal import Decimal
|
||||||
|
|
@ -42,10 +41,7 @@ _spot_ws: str = 'wss://stream.binance.com/ws'
|
||||||
# or this one? ..
|
# or this one? ..
|
||||||
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
||||||
|
|
||||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/
|
# https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||||
# websocket-market-streams/Important-WebSocket-Change-Notice
|
|
||||||
_futes_public_ws: str = f'wss://fstream.{_domain}/public/ws'
|
|
||||||
_futes_market_ws: str = f'wss://fstream.{_domain}/market/ws'
|
|
||||||
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
||||||
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
||||||
|
|
||||||
|
|
@ -82,7 +78,7 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
|
||||||
),
|
),
|
||||||
'usdtm_futes': (
|
'usdtm_futes': (
|
||||||
_futes_url,
|
_futes_url,
|
||||||
_futes_market_ws,
|
_futes_ws,
|
||||||
),
|
),
|
||||||
}[venue]
|
}[venue]
|
||||||
|
|
||||||
|
|
@ -207,8 +203,6 @@ class FutesPair(Pair):
|
||||||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||||
ns_path: str = 'piker.brokers.binance:FutesPair'
|
ns_path: str = 'piker.brokers.binance:FutesPair'
|
||||||
|
|
||||||
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
|
||||||
|
|
||||||
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
||||||
# processing..
|
# processing..
|
||||||
@property
|
@property
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ from piker.log import (
|
||||||
get_logger,
|
get_logger,
|
||||||
)
|
)
|
||||||
from ..service import (
|
from ..service import (
|
||||||
maybe_spawn_datad,
|
maybe_spawn_brokerd,
|
||||||
maybe_open_pikerd,
|
maybe_open_pikerd,
|
||||||
)
|
)
|
||||||
from ..brokers import (
|
from ..brokers import (
|
||||||
|
|
@ -187,7 +187,7 @@ def brokercheck(config, broker):
|
||||||
|
|
||||||
'''
|
'''
|
||||||
async def bcheck_main():
|
async def bcheck_main():
|
||||||
async with maybe_spawn_datad(broker) as portal:
|
async with maybe_spawn_brokerd(broker) as portal:
|
||||||
await portal.run(run_test, broker)
|
await portal.run(run_test, broker)
|
||||||
await portal.cancel_actor()
|
await portal.cancel_actor()
|
||||||
|
|
||||||
|
|
@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
|
||||||
return
|
return
|
||||||
|
|
||||||
async def main(tries):
|
async def main(tries):
|
||||||
async with maybe_spawn_datad(
|
async with maybe_spawn_brokerd(
|
||||||
tries=tries, loglevel=loglevel
|
tries=tries, loglevel=loglevel
|
||||||
) as portal:
|
) as portal:
|
||||||
# run app "main"
|
# run app "main"
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,7 @@ import trio
|
||||||
|
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
from ..service import maybe_spawn_datad
|
from ..service import maybe_spawn_brokerd
|
||||||
from . import open_cached_client
|
from . import open_cached_client
|
||||||
from ..accounting import MktPair
|
from ..accounting import MktPair
|
||||||
|
|
||||||
|
|
@ -172,7 +172,7 @@ async def symbol_search(
|
||||||
# await tractor.devx._debug.maybe_init_greenback()
|
# await tractor.devx._debug.maybe_init_greenback()
|
||||||
# tractor.pause_from_sync()
|
# tractor.pause_from_sync()
|
||||||
|
|
||||||
async with maybe_spawn_datad(
|
async with maybe_spawn_brokerd(
|
||||||
mod.name,
|
mod.name,
|
||||||
infect_asyncio=getattr(
|
infect_asyncio=getattr(
|
||||||
mod,
|
mod,
|
||||||
|
|
|
||||||
|
|
@ -425,7 +425,7 @@ class DataFeed:
|
||||||
async def stream_to_file(
|
async def stream_to_file(
|
||||||
watchlist_name: str,
|
watchlist_name: str,
|
||||||
filename: str,
|
filename: str,
|
||||||
portal: tractor.Portal,
|
portal: tractor._portal.Portal,
|
||||||
tickers: List[str],
|
tickers: List[str],
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
rate: int,
|
rate: int,
|
||||||
|
|
|
||||||
|
|
@ -47,25 +47,13 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# tractor RPC enable arg
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
# NOTE: datad-only backend (no `broker.py` yet)!
|
|
||||||
_brokerd_mods: list[str] = []
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
|
# 'broker',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
'infect_asyncio': True,
|
'infect_asyncio': True,
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,7 @@ from contextlib import (
|
||||||
asynccontextmanager as acm,
|
asynccontextmanager as acm,
|
||||||
)
|
)
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
from functools import partial
|
||||||
import time
|
import time
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
|
|
@ -37,7 +38,7 @@ from rapidfuzz import process as fuzzy
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from tractor.trionics import (
|
from tractor.trionics import (
|
||||||
broadcast_receiver,
|
broadcast_receiver,
|
||||||
maybe_open_context,
|
maybe_open_context
|
||||||
collapse_eg,
|
collapse_eg,
|
||||||
)
|
)
|
||||||
from tractor import to_asyncio
|
from tractor import to_asyncio
|
||||||
|
|
@ -523,12 +524,13 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
|
||||||
|
|
||||||
|
|
||||||
async def aio_price_feed_relay(
|
async def aio_price_feed_relay(
|
||||||
chan: to_asyncio.LinkedTaskChannel,
|
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
|
from_trio: asyncio.Queue,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _trade(data: dict, receipt_timestamp):
|
async def _trade(data: dict, receipt_timestamp):
|
||||||
chan.send_nowait(('trade', {
|
to_trio.send_nowait(('trade', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'last': data,
|
'last': data,
|
||||||
|
|
@ -538,7 +540,7 @@ async def aio_price_feed_relay(
|
||||||
}))
|
}))
|
||||||
|
|
||||||
async def _l1(data: dict, receipt_timestamp):
|
async def _l1(data: dict, receipt_timestamp):
|
||||||
chan.send_nowait(('l1', {
|
to_trio.send_nowait(('l1', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'ticks': [
|
'ticks': [
|
||||||
|
|
@ -568,7 +570,7 @@ async def aio_price_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
chan.started_nowait(None)
|
to_trio.send_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -579,9 +581,11 @@ async def open_price_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
|
partial(
|
||||||
aio_price_feed_relay,
|
aio_price_feed_relay,
|
||||||
fh=fh,
|
fh,
|
||||||
instrument=instrument,
|
instrument
|
||||||
|
)
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
@ -607,9 +611,10 @@ async def maybe_open_price_feed(
|
||||||
|
|
||||||
|
|
||||||
async def aio_order_feed_relay(
|
async def aio_order_feed_relay(
|
||||||
chan: to_asyncio.LinkedTaskChannel,
|
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
|
from_trio: asyncio.Queue,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _fill(data: dict, receipt_timestamp):
|
async def _fill(data: dict, receipt_timestamp):
|
||||||
breakpoint()
|
breakpoint()
|
||||||
|
|
@ -632,7 +637,7 @@ async def aio_order_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
chan.started_nowait(None)
|
to_trio.send_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -643,9 +648,11 @@ async def open_order_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
|
partial(
|
||||||
aio_order_feed_relay,
|
aio_order_feed_relay,
|
||||||
fh=fh,
|
fh,
|
||||||
instrument=instrument,
|
instrument
|
||||||
|
)
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -65,18 +65,17 @@ _brokerd_mods: list[str] = [
|
||||||
]
|
]
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
_datad_mods: list[str] = [
|
||||||
'api',
|
|
||||||
'feed',
|
'feed',
|
||||||
'symbols',
|
'symbols',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
# tractor RPC enable arg
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
__enable_modules__: list[str] = (
|
||||||
_brokerd_mods
|
_brokerd_mods
|
||||||
+
|
+
|
||||||
_datad_mods
|
_datad_mods
|
||||||
))
|
)
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
|
|
|
||||||
|
|
@ -958,26 +958,8 @@ class Client:
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise
|
raise
|
||||||
# XXX: ambiguous/unqualifiable contracts are
|
|
||||||
# returned as `None` entries by
|
|
||||||
# `qualifyContractsAsync()` (which also logs an
|
|
||||||
# "Ambiguous contract" warning listing the
|
|
||||||
# possible matches) so filter them BEFORE any
|
|
||||||
# attr access B)
|
|
||||||
contracts: list[Contract] = [
|
|
||||||
tract
|
|
||||||
for tract in contracts
|
|
||||||
if tract is not None
|
|
||||||
]
|
|
||||||
if not contracts:
|
if not contracts:
|
||||||
raise ValueError(
|
raise ValueError(f"No contract could be found {con}")
|
||||||
f'No (unambiguous) contract could be '
|
|
||||||
f'qualified for {con!r}\n'
|
|
||||||
f'\n'
|
|
||||||
f'If a stonk, you likely need a (more) '
|
|
||||||
f'venue-qualified fqme,\n'
|
|
||||||
f"eg. 'gld.arca.ib' instead of 'gld.ib'\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
# pack all contracts into cache
|
# pack all contracts into cache
|
||||||
for tract in contracts:
|
for tract in contracts:
|
||||||
|
|
@ -1155,16 +1137,10 @@ class Client:
|
||||||
try:
|
try:
|
||||||
con: Contract = self._contracts[symbol]
|
con: Contract = self._contracts[symbol]
|
||||||
except KeyError:
|
except KeyError:
|
||||||
# require that the mkt's contract has been previously
|
# require that the symbol has been previously cached by
|
||||||
# qualified and cached (see `.symbols.get_mkt_info()`
|
# a data feed request - ensure we aren't making orders
|
||||||
# which is run for any feed-init OR lazily by the
|
# against non-known prices.
|
||||||
# order-request handler in `.broker`) - ensures we
|
raise RuntimeError("Can not order {symbol}, no live feed?")
|
||||||
# aren't making orders against unknown contracts.
|
|
||||||
raise RuntimeError(
|
|
||||||
f'Can not order {symbol!r}, '
|
|
||||||
f'no qualified contract cached!?\n'
|
|
||||||
f'_contracts: {list(self._contracts)!r}\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
trade = self.ib.placeOrder(
|
trade = self.ib.placeOrder(
|
||||||
|
|
@ -1355,21 +1331,6 @@ async def load_aio_clients(
|
||||||
ib = None
|
ib = None
|
||||||
client = None
|
client = None
|
||||||
|
|
||||||
# XXX: post (brokerd vs. datad)-split BOTH per-broker
|
|
||||||
# daemons connect to the same API gw/tws endpoint(s); to
|
|
||||||
# avoid `clientId` collisions (and the long conn-timeout
|
|
||||||
# retry cycle they cause) we offset the data-daemon's
|
|
||||||
# default id-range to be disjoint from `brokerd.ib`'s
|
|
||||||
# (which also retries with `client_id + i` increments).
|
|
||||||
if client_id == 6116: # the default from above
|
|
||||||
aname: str = tractor.current_actor().name
|
|
||||||
if 'datad' in aname:
|
|
||||||
client_id += 16
|
|
||||||
# ad-hoc (test/cli) actors get their own range to
|
|
||||||
# avoid clashing with any live daemon-tree's conns.
|
|
||||||
elif 'brokerd' not in aname:
|
|
||||||
client_id += 32
|
|
||||||
|
|
||||||
# attempt to get connection info from config; if no .toml entry
|
# attempt to get connection info from config; if no .toml entry
|
||||||
# exists, we try to load from a default localhost connection.
|
# exists, we try to load from a default localhost connection.
|
||||||
localhost = '127.0.0.1'
|
localhost = '127.0.0.1'
|
||||||
|
|
@ -1623,72 +1584,6 @@ class MethodProxy:
|
||||||
self.event_table = event_table
|
self.event_table = event_table
|
||||||
self._aio_ns = asyncio_ns
|
self._aio_ns = asyncio_ns
|
||||||
|
|
||||||
# request-id counter for correlating each method
|
|
||||||
# call to its (eventual) response; necessary since
|
|
||||||
# the (single, shared) chan has NO other ordering
|
|
||||||
# guarantee whenever a caller task is CANCELLED
|
|
||||||
# (eg. by a search-req timeout) after sending its
|
|
||||||
# request but before consuming the response: the
|
|
||||||
# "orphaned" response otherwise gets mis-delivered
|
|
||||||
# to the next caller causing an off-by-one skew of
|
|
||||||
# every result thereafter!
|
|
||||||
self._mids = itertools.count()
|
|
||||||
self._pending: dict[
|
|
||||||
int,
|
|
||||||
tuple[trio.Event, list[dict]],
|
|
||||||
] = {}
|
|
||||||
self._request_methods: dict[int, str|None] = {}
|
|
||||||
|
|
||||||
def _deliver_method_response(
|
|
||||||
self,
|
|
||||||
msg: dict,
|
|
||||||
|
|
||||||
) -> bool:
|
|
||||||
'''
|
|
||||||
Deliver one correlated response to its waiting caller.
|
|
||||||
|
|
||||||
'''
|
|
||||||
mid: int = msg['mid']
|
|
||||||
meth: str|None = self._request_methods.pop(mid, None)
|
|
||||||
pending = self._pending.get(mid)
|
|
||||||
if pending is None:
|
|
||||||
log.warning(
|
|
||||||
f'Dropping stale method-resp,\n'
|
|
||||||
f'mid: {mid}\n'
|
|
||||||
f'meth: {meth!r}\n'
|
|
||||||
f'(caller prolly got cancelled '
|
|
||||||
f'before its resp?)\n'
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
|
|
||||||
event, slot = pending
|
|
||||||
slot.append(msg)
|
|
||||||
event.set()
|
|
||||||
return True
|
|
||||||
|
|
||||||
def _track_request_method(
|
|
||||||
self,
|
|
||||||
mid: int,
|
|
||||||
meth: str|None,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Retain bounded request metadata for stale-response logs.
|
|
||||||
|
|
||||||
'''
|
|
||||||
self._request_methods[mid] = meth
|
|
||||||
if len(self._request_methods) > 256:
|
|
||||||
oldest_mid: int = next(iter(self._request_methods))
|
|
||||||
oldest_meth: str|None = self._request_methods.pop(
|
|
||||||
oldest_mid
|
|
||||||
)
|
|
||||||
log.warning(
|
|
||||||
f'Evicting unresolved method-request metadata,\n'
|
|
||||||
f'mid: {oldest_mid}\n'
|
|
||||||
f'meth: {oldest_meth!r}\n'
|
|
||||||
f'pending metadata limit: 256\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _run_method(
|
async def _run_method(
|
||||||
self,
|
self,
|
||||||
*,
|
*,
|
||||||
|
|
@ -1701,30 +1596,38 @@ class MethodProxy:
|
||||||
``tractor.to_asyncio`` layer.
|
``tractor.to_asyncio`` layer.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
mid: int = next(self._mids)
|
chan = self.chan
|
||||||
event = trio.Event()
|
await chan.send((meth, kwargs))
|
||||||
slot: list[dict] = []
|
|
||||||
self._pending[mid] = event, slot
|
|
||||||
self._track_request_method(mid, meth)
|
|
||||||
try:
|
|
||||||
await self.chan.send((meth, kwargs, mid))
|
|
||||||
await event.wait()
|
|
||||||
finally:
|
|
||||||
self._pending.pop(mid, None)
|
|
||||||
|
|
||||||
msg: dict = slot[0]
|
while not chan.closed():
|
||||||
match msg:
|
# send through method + ``kwargs: dict`` as pair
|
||||||
# OUR method-call response B)
|
msg = await chan.receive()
|
||||||
case {'result': res}:
|
|
||||||
|
# TODO: implement reconnect functionality like
|
||||||
|
# in our `.data._web_bs.NoBsWs`
|
||||||
|
# try:
|
||||||
|
# msg = await chan.receive()
|
||||||
|
# except ConnectionError:
|
||||||
|
# self.reset()
|
||||||
|
|
||||||
|
# print(f'NEXT MSG: {msg}')
|
||||||
|
|
||||||
|
# TODO: py3.10 ``match:`` syntax B)
|
||||||
|
if 'result' in msg:
|
||||||
|
res = msg.get('result')
|
||||||
return res
|
return res
|
||||||
|
|
||||||
case {'exception': err}:
|
elif 'exception' in msg:
|
||||||
|
err = msg.get('exception')
|
||||||
raise err
|
raise err
|
||||||
|
|
||||||
case _:
|
elif 'error' in msg:
|
||||||
raise RuntimeError(
|
etype, emsg = msg
|
||||||
f'Invalid method response for mid={mid}: {msg!r}'
|
log.warning(f'IB error relay: {emsg}')
|
||||||
)
|
continue
|
||||||
|
|
||||||
|
else:
|
||||||
|
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||||
|
|
||||||
def status_event(
|
def status_event(
|
||||||
self,
|
self,
|
||||||
|
|
@ -1734,7 +1637,7 @@ class MethodProxy:
|
||||||
|
|
||||||
ev = self.event_table.get(pattern)
|
ev = self.event_table.get(pattern)
|
||||||
|
|
||||||
if ev is None or ev.is_set():
|
if not ev or ev.is_set():
|
||||||
# print(f'inserting new data reset event item')
|
# print(f'inserting new data reset event item')
|
||||||
ev = self.event_table[pattern] = trio.Event()
|
ev = self.event_table[pattern] = trio.Event()
|
||||||
|
|
||||||
|
|
@ -1777,31 +1680,22 @@ async def open_aio_client_method_relay(
|
||||||
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
||||||
break
|
break
|
||||||
|
|
||||||
case (meth_name, kwargs, mid):
|
case (meth_name, kwargs):
|
||||||
|
meth_name, kwargs = msg
|
||||||
meth = getattr(client, meth_name)
|
meth = getattr(client, meth_name)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
resp = await meth(**kwargs)
|
resp = await meth(**kwargs)
|
||||||
# echo the msg back tagged with the
|
# echo the msg back
|
||||||
# req-id for caller-side correlation.
|
chan.send_nowait({'result': resp})
|
||||||
chan.send_nowait({
|
|
||||||
'mid': mid,
|
|
||||||
'result': resp,
|
|
||||||
})
|
|
||||||
|
|
||||||
# XXX: relay ALL (non-cancel) errors to the
|
except (
|
||||||
# `trio`-side caller (which re-raises in the
|
RequestError,
|
||||||
# `MethodProxy._run_method()` frame) instead
|
|
||||||
# of crashing this relay task and thus the
|
# TODO: relay all errors to trio?
|
||||||
# whole proxy/dialog; critical post the
|
# BaseException,
|
||||||
# (datad|brokerd)-split where eg. contract
|
) as err:
|
||||||
# qualification failures are expected to be
|
chan.send_nowait({'exception': err})
|
||||||
# caught per-request by order/warmup code!
|
|
||||||
except Exception as err:
|
|
||||||
chan.send_nowait({
|
|
||||||
'mid': mid,
|
|
||||||
'exception': err,
|
|
||||||
})
|
|
||||||
|
|
||||||
case {'error': content}:
|
case {'error': content}:
|
||||||
chan.send_nowait({'exception': content})
|
chan.send_nowait({'exception': content})
|
||||||
|
|
@ -1810,73 +1704,6 @@ async def open_aio_client_method_relay(
|
||||||
raise ValueError(f'Unhandled msg {msg}')
|
raise ValueError(f'Unhandled msg {msg}')
|
||||||
|
|
||||||
|
|
||||||
async def relay_client_proxy_messages(
|
|
||||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
|
||||||
proxy: MethodProxy,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Route one proxy's asyncio-client msgs from a single reader.
|
|
||||||
|
|
||||||
Each `open_client_proxy()` allocates its own linked channel and
|
|
||||||
starts this task in its relay nursery. Any number of Trio caller
|
|
||||||
tasks may concurrently write tagged requests through
|
|
||||||
`MethodProxy._run_method()` and wait on their per-`mid` slots.
|
|
||||||
This task alone reads the channel and dispatches each response to
|
|
||||||
its waiting caller.
|
|
||||||
|
|
||||||
Separate proxy contexts own separate channels and reader tasks;
|
|
||||||
no cross-proxy response broadcast is intended. IB status events
|
|
||||||
still reach each proxy through its own `Client.inline_errors()`
|
|
||||||
handler.
|
|
||||||
|
|
||||||
'''
|
|
||||||
try:
|
|
||||||
while not chan.closed():
|
|
||||||
# TODO: implement reconnect functionality like
|
|
||||||
# in our `.data._web_bs.NoBsWs`
|
|
||||||
# try:
|
|
||||||
msg = await chan.receive()
|
|
||||||
# except ConnectionError:
|
|
||||||
# proxy.reset()
|
|
||||||
|
|
||||||
match msg:
|
|
||||||
# Correlated response from a method call.
|
|
||||||
case {'mid': _}:
|
|
||||||
proxy._deliver_method_response(msg)
|
|
||||||
|
|
||||||
# routine (api-farm conn) status events
|
|
||||||
# relayed inline by `Client.inline_errors()`;
|
|
||||||
# not a response to a method call so just
|
|
||||||
# log at info and keep waiting.
|
|
||||||
case ('event', status_msg):
|
|
||||||
reason = status_msg['reason']
|
|
||||||
event = proxy.event_table.pop(reason, None)
|
|
||||||
if (
|
|
||||||
event is not None
|
|
||||||
and
|
|
||||||
event.statistics().tasks_waiting
|
|
||||||
):
|
|
||||||
log.info(f'Relaying ib status message: {msg}')
|
|
||||||
event.set()
|
|
||||||
|
|
||||||
# Inline API error with no request correlation.
|
|
||||||
case ('error', emsg):
|
|
||||||
log.warning(f'IB error relay: {emsg}')
|
|
||||||
|
|
||||||
# Out-of-band client error with no method `mid`.
|
|
||||||
case {'exception': err}:
|
|
||||||
log.error(
|
|
||||||
f'Uncorrelated IB client error: {err!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
# Preserve visibility into unexpected relay traffic.
|
|
||||||
case _:
|
|
||||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
|
||||||
except trio.EndOfChannel:
|
|
||||||
log.info('IB client proxy relay closed')
|
|
||||||
|
|
||||||
|
|
||||||
@acm
|
@acm
|
||||||
async def open_client_proxy(
|
async def open_client_proxy(
|
||||||
client: Client,
|
client: Client,
|
||||||
|
|
@ -1913,11 +1740,28 @@ async def open_client_proxy(
|
||||||
continue
|
continue
|
||||||
setattr(proxy, name, partial(proxy._run_method, meth=name))
|
setattr(proxy, name, partial(proxy._run_method, meth=name))
|
||||||
|
|
||||||
relay_tn.start_soon(
|
async def relay_events():
|
||||||
relay_client_proxy_messages,
|
|
||||||
chan,
|
async with chan.subscribe() as msg_stream:
|
||||||
proxy,
|
|
||||||
)
|
async for msg in msg_stream:
|
||||||
|
if 'event' not in msg:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# if 'event' in msg:
|
||||||
|
# wake up any system event waiters.
|
||||||
|
etype, status_msg = msg
|
||||||
|
reason = status_msg['reason']
|
||||||
|
|
||||||
|
ev = proxy.event_table.pop(reason, None)
|
||||||
|
|
||||||
|
if ev and ev.statistics().tasks_waiting:
|
||||||
|
log.info(f'Relaying ib status message: {msg}')
|
||||||
|
ev.set()
|
||||||
|
|
||||||
|
continue
|
||||||
|
|
||||||
|
relay_tn.start_soon(relay_events)
|
||||||
|
|
||||||
yield proxy
|
yield proxy
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -89,9 +89,8 @@ from .api import (
|
||||||
MethodProxy,
|
MethodProxy,
|
||||||
)
|
)
|
||||||
from .symbols import (
|
from .symbols import (
|
||||||
cache_contract,
|
|
||||||
con2fqme,
|
con2fqme,
|
||||||
get_mkt_info,
|
# get_mkt_info,
|
||||||
)
|
)
|
||||||
from .ledger import (
|
from .ledger import (
|
||||||
norm_trade_records,
|
norm_trade_records,
|
||||||
|
|
@ -139,7 +138,6 @@ async def handle_order_requests(
|
||||||
ems_order_stream: tractor.MsgStream,
|
ems_order_stream: tractor.MsgStream,
|
||||||
accounts_def: dict[str, str],
|
accounts_def: dict[str, str],
|
||||||
flows: OrderDialogs,
|
flows: OrderDialogs,
|
||||||
proxies: dict[str, MethodProxy],
|
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
|
|
@ -182,58 +180,6 @@ async def handle_order_requests(
|
||||||
if action in {'buy', 'sell'}:
|
if action in {'buy', 'sell'}:
|
||||||
# validate
|
# validate
|
||||||
order = BrokerdOrder(**request_msg)
|
order = BrokerdOrder(**request_msg)
|
||||||
fqme: str = order.symbol
|
|
||||||
|
|
||||||
# XXX: lazily qualify and cache the contract for
|
|
||||||
# this mkt since, post the (datad|brokerd)-split,
|
|
||||||
# this trading-only actor will NOT have had its
|
|
||||||
# (api) `Client._contracts` pre-warmed by any
|
|
||||||
# in-proc feed setup (which now runs in the
|
|
||||||
# `datad.ib` sibling); we run the SAME symbology
|
|
||||||
# resolution ep the feed-side uses so the cache
|
|
||||||
# key (`MktPair.bs_fqme`) matches what
|
|
||||||
# `Client.submit_limit()` looks up.
|
|
||||||
# NOTE: normally a no-op since
|
|
||||||
# `open_trade_dialog()` eagerly pre-qualifies all
|
|
||||||
# already-open pp/order mkts at startup; this
|
|
||||||
# only fires for a brand-new (to this daemon)
|
|
||||||
# mkt's first order.
|
|
||||||
if fqme not in client._contracts:
|
|
||||||
proxy: MethodProxy = proxies[account]
|
|
||||||
try:
|
|
||||||
(
|
|
||||||
mkt,
|
|
||||||
details,
|
|
||||||
) = await get_mkt_info(
|
|
||||||
fqme,
|
|
||||||
proxy=proxy,
|
|
||||||
)
|
|
||||||
# XXX: explicit write since the lifo-cache
|
|
||||||
# may deliver a hit (body skipped) when
|
|
||||||
# another acct/client already qualified
|
|
||||||
# this fqme.
|
|
||||||
cache_contract(
|
|
||||||
client,
|
|
||||||
mkt,
|
|
||||||
details.contract,
|
|
||||||
)
|
|
||||||
except Exception as err:
|
|
||||||
log.exception(
|
|
||||||
f'Failed to qualify contract for\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
await ems_order_stream.send(
|
|
||||||
BrokerdError(
|
|
||||||
oid=oid,
|
|
||||||
symbol=fqme,
|
|
||||||
reason=(
|
|
||||||
f'No contract could be '
|
|
||||||
f'qualified for {fqme!r}:\n'
|
|
||||||
f'{err!r}'
|
|
||||||
),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# XXX: by default 0 tells ``ib_async`` methods that
|
# XXX: by default 0 tells ``ib_async`` methods that
|
||||||
# there is no existing order so ask the client to create
|
# there is no existing order so ask the client to create
|
||||||
|
|
@ -245,34 +191,15 @@ async def handle_order_requests(
|
||||||
reqid = int(reqid)
|
reqid = int(reqid)
|
||||||
|
|
||||||
# call our client api to submit the order
|
# call our client api to submit the order
|
||||||
# NOTE: guard with order-error relay (vs. crashing
|
|
||||||
# this dialog and thus ALL order ctl for the
|
|
||||||
# backend) so one bad submission can't take down
|
|
||||||
# the daemon's clearing loop.
|
|
||||||
try:
|
|
||||||
reqid = client.submit_limit(
|
reqid = client.submit_limit(
|
||||||
oid=order.oid,
|
oid=order.oid,
|
||||||
symbol=fqme,
|
symbol=order.symbol,
|
||||||
price=order.price,
|
price=order.price,
|
||||||
action=order.action,
|
action=order.action,
|
||||||
size=order.size,
|
size=order.size,
|
||||||
account=acct_number,
|
account=acct_number,
|
||||||
reqid=reqid,
|
reqid=reqid,
|
||||||
)
|
)
|
||||||
except Exception as err:
|
|
||||||
log.exception(
|
|
||||||
f'Order submission failed for\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
)
|
|
||||||
await ems_order_stream.send(
|
|
||||||
BrokerdError(
|
|
||||||
oid=oid,
|
|
||||||
symbol=fqme,
|
|
||||||
reason=f'Submission error: {err!r}',
|
|
||||||
)
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
str_reqid: str = str(reqid)
|
str_reqid: str = str(reqid)
|
||||||
if reqid is None:
|
if reqid is None:
|
||||||
err_msg = BrokerdError(
|
err_msg = BrokerdError(
|
||||||
|
|
@ -304,21 +231,20 @@ async def handle_order_requests(
|
||||||
|
|
||||||
async def recv_trade_updates(
|
async def recv_trade_updates(
|
||||||
|
|
||||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
|
||||||
client: Client,
|
client: Client,
|
||||||
|
to_trio: trio.abc.SendChannel,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Receive and relay order control and positioning
|
Receive and relay order control and positioning related events
|
||||||
related events from `ib_async`, pack as tuples and
|
from `ib_async`, pack as tuples and push over mem-chan to our
|
||||||
push over mem-chan to our trio relay task for
|
trio relay task for processing and relay to EMS.
|
||||||
processing and relay to EMS.
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
client.inline_errors(chan)
|
client.inline_errors(to_trio)
|
||||||
|
|
||||||
# sync with trio task
|
# sync with trio task
|
||||||
chan.started_nowait(client.ib)
|
to_trio.send_nowait(client.ib)
|
||||||
|
|
||||||
def push_tradesies(
|
def push_tradesies(
|
||||||
eventkit_obj,
|
eventkit_obj,
|
||||||
|
|
@ -356,7 +282,7 @@ async def recv_trade_updates(
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# emit event name + relevant ibis internal objects
|
# emit event name + relevant ibis internal objects
|
||||||
chan.send_nowait((event_name, emit))
|
to_trio.send_nowait((event_name, emit))
|
||||||
except trio.BrokenResourceError:
|
except trio.BrokenResourceError:
|
||||||
log.exception(f'Disconnected from {eventkit_obj} updates')
|
log.exception(f'Disconnected from {eventkit_obj} updates')
|
||||||
eventkit_obj.disconnect(push_tradesies)
|
eventkit_obj.disconnect(push_tradesies)
|
||||||
|
|
@ -621,14 +547,10 @@ async def open_trade_dialog(
|
||||||
|
|
||||||
) -> AsyncIterator[dict[str, Any]]:
|
) -> AsyncIterator[dict[str, Any]]:
|
||||||
|
|
||||||
# XXX: ONLY adjust the level of this (sub)mod's logger;
|
get_console_log(
|
||||||
# attaching a (stderr) handler (via `get_console_log()`)
|
level=loglevel,
|
||||||
# here would DOUBLE-print every record since the daemon
|
name=__name__,
|
||||||
# fixture (`.._daemon._setup_persistent_brokerd()`)
|
)
|
||||||
# already enables the console handler on the parent
|
|
||||||
# subsys logger which all our records propagate to!
|
|
||||||
if loglevel:
|
|
||||||
log.setLevel(loglevel.upper())
|
|
||||||
|
|
||||||
# task local msg dialog tracking
|
# task local msg dialog tracking
|
||||||
flows = OrderDialogs()
|
flows = OrderDialogs()
|
||||||
|
|
@ -865,64 +787,6 @@ async def open_trade_dialog(
|
||||||
for msg in order_msgs:
|
for msg in order_msgs:
|
||||||
await ems_stream.send(msg)
|
await ems_stream.send(msg)
|
||||||
|
|
||||||
# XXX: eagerly pre-qualify (and cache) the
|
|
||||||
# contracts for all already-open pps and
|
|
||||||
# orders so that (live) order submission
|
|
||||||
# NEVER pays a first-request qualification
|
|
||||||
# delay; any brand-new mkt is still lazily
|
|
||||||
# qualified by `handle_order_requests()` on
|
|
||||||
# its first submission. NOTE: since this
|
|
||||||
# runs BEFORE the order handler task is
|
|
||||||
# even started, no order can clear until
|
|
||||||
# the warmup completes (early reqs just
|
|
||||||
# buffer in the ems IPC stream) B)
|
|
||||||
warmup_fqmes: set[str] = {
|
|
||||||
msg.symbol
|
|
||||||
for msg in all_positions
|
|
||||||
}
|
|
||||||
warmup_fqmes.update(
|
|
||||||
msg.req.symbol
|
|
||||||
for msg in order_msgs
|
|
||||||
)
|
|
||||||
unique_clients: set[Client] = set(
|
|
||||||
aioclients.values()
|
|
||||||
)
|
|
||||||
if (
|
|
||||||
warmup_fqmes
|
|
||||||
and
|
|
||||||
proxies
|
|
||||||
):
|
|
||||||
a_proxy: MethodProxy = next(
|
|
||||||
iter(proxies.values())
|
|
||||||
)
|
|
||||||
for fqme in warmup_fqmes:
|
|
||||||
try:
|
|
||||||
(
|
|
||||||
mkt,
|
|
||||||
details,
|
|
||||||
) = await get_mkt_info(
|
|
||||||
fqme,
|
|
||||||
proxy=a_proxy,
|
|
||||||
)
|
|
||||||
except Exception as err:
|
|
||||||
# XXX: non-fatal; an
|
|
||||||
# un-warmed mkt just falls
|
|
||||||
# back to the lazy qualify
|
|
||||||
# in the order handler.
|
|
||||||
log.warning(
|
|
||||||
f'Failed to pre-qualify\n'
|
|
||||||
f'fqme: {fqme!r}\n'
|
|
||||||
f'err: {err!r}\n'
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
for _client in unique_clients:
|
|
||||||
cache_contract(
|
|
||||||
_client,
|
|
||||||
mkt,
|
|
||||||
details.contract,
|
|
||||||
)
|
|
||||||
|
|
||||||
for client in set(aioclients.values()):
|
for client in set(aioclients.values()):
|
||||||
trade_event_stream: LinkedTaskChannel = await tn.start(
|
trade_event_stream: LinkedTaskChannel = await tn.start(
|
||||||
open_trade_event_stream,
|
open_trade_event_stream,
|
||||||
|
|
@ -936,7 +800,6 @@ async def open_trade_dialog(
|
||||||
ems_stream,
|
ems_stream,
|
||||||
accounts_def,
|
accounts_def,
|
||||||
flows,
|
flows,
|
||||||
proxies,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# allocate event relay tasks for each client connection
|
# allocate event relay tasks for each client connection
|
||||||
|
|
|
||||||
|
|
@ -32,12 +32,14 @@ import time
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
Callable,
|
Callable,
|
||||||
|
TYPE_CHECKING,
|
||||||
)
|
)
|
||||||
|
|
||||||
from async_generator import aclosing
|
from async_generator import aclosing
|
||||||
import ib_async as ibis
|
import ib_async as ibis
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from pendulum import (
|
from pendulum import (
|
||||||
|
now,
|
||||||
from_timestamp,
|
from_timestamp,
|
||||||
Duration,
|
Duration,
|
||||||
duration as mk_duration,
|
duration as mk_duration,
|
||||||
|
|
@ -73,6 +75,9 @@ from ._util import (
|
||||||
)
|
)
|
||||||
from .symbols import get_mkt_info
|
from .symbols import get_mkt_info
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from trio._core._run import Task
|
||||||
|
|
||||||
log = get_logger(
|
log = get_logger(
|
||||||
name=__name__,
|
name=__name__,
|
||||||
)
|
)
|
||||||
|
|
@ -220,11 +225,7 @@ async def open_history_client(
|
||||||
f'mean: {mean}'
|
f'mean: {mean}'
|
||||||
)
|
)
|
||||||
|
|
||||||
# get_bars() uses None internally for an empty reply.
|
# could be trying to retreive bars over weekend
|
||||||
# The backfiller never receives that sentinel: translate
|
|
||||||
# it into NoData, or DataUnavailable at the known start
|
|
||||||
# of the instrument's history. A venue closure can also
|
|
||||||
# yield an empty reply without exhausting all history.
|
|
||||||
if out is None:
|
if out is None:
|
||||||
log.error(
|
log.error(
|
||||||
f"No bars starting at {end_dt!r} !?!?"
|
f"No bars starting at {end_dt!r} !?!?"
|
||||||
|
|
@ -352,7 +353,6 @@ async def wait_on_data_reset(
|
||||||
tuple[
|
tuple[
|
||||||
trio.CancelScope,
|
trio.CancelScope,
|
||||||
trio.Event,
|
trio.Event,
|
||||||
list[bool],
|
|
||||||
]
|
]
|
||||||
] = trio.TASK_STATUS_IGNORED,
|
] = trio.TASK_STATUS_IGNORED,
|
||||||
) -> bool:
|
) -> bool:
|
||||||
|
|
@ -396,9 +396,8 @@ async def wait_on_data_reset(
|
||||||
client: Client = proxy._aio_ns
|
client: Client = proxy._aio_ns
|
||||||
|
|
||||||
done = trio.Event()
|
done = trio.Event()
|
||||||
result: list[bool] = []
|
|
||||||
with trio.move_on_after(timeout) as cs:
|
with trio.move_on_after(timeout) as cs:
|
||||||
task_status.started((cs, done, result))
|
task_status.started((cs, done))
|
||||||
|
|
||||||
log.warning(
|
log.warning(
|
||||||
'Sending DATA RESET request:\n'
|
'Sending DATA RESET request:\n'
|
||||||
|
|
@ -414,7 +413,6 @@ async def wait_on_data_reset(
|
||||||
'NO VNC DETECTED!\n'
|
'NO VNC DETECTED!\n'
|
||||||
'Manually press ctrl-alt-f on your IB java app'
|
'Manually press ctrl-alt-f on your IB java app'
|
||||||
)
|
)
|
||||||
result.append(False)
|
|
||||||
done.set()
|
done.set()
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
@ -429,7 +427,6 @@ async def wait_on_data_reset(
|
||||||
]:
|
]:
|
||||||
await ev.wait()
|
await ev.wait()
|
||||||
log.info(f"{name} DATA RESET")
|
log.info(f"{name} DATA RESET")
|
||||||
result.append(True)
|
|
||||||
done.set()
|
done.set()
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
@ -438,12 +435,12 @@ async def wait_on_data_reset(
|
||||||
'Data reset task canceled?'
|
'Data reset task canceled?'
|
||||||
)
|
)
|
||||||
|
|
||||||
result.append(False)
|
|
||||||
done.set()
|
done.set()
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
_data_reset_lock = trio.Lock()
|
_data_resetter_task: Task | None = None
|
||||||
|
_failed_resets: int = 0
|
||||||
|
|
||||||
|
|
||||||
async def get_bars(
|
async def get_bars(
|
||||||
|
|
@ -459,6 +456,10 @@ async def get_bars(
|
||||||
# how long before we trigger a feed reset (seconds)
|
# how long before we trigger a feed reset (seconds)
|
||||||
feed_reset_timeout: float = 3,
|
feed_reset_timeout: float = 3,
|
||||||
|
|
||||||
|
# how many days to subtract before giving up on further
|
||||||
|
# history queries for instrument, presuming that most don't
|
||||||
|
# not trade for a week XD
|
||||||
|
max_nodatas: int = 6,
|
||||||
max_failed_resets: int = 6,
|
max_failed_resets: int = 6,
|
||||||
|
|
||||||
task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
|
task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
|
||||||
|
|
@ -478,10 +479,8 @@ async def get_bars(
|
||||||
`.ib.api.Client` methods.
|
`.ib.api.Client` methods.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
cancelled_count: int = 0
|
global _data_resetter_task, _failed_resets
|
||||||
failed_resets: int = 0
|
nodatas_count: int = 0
|
||||||
pacing_reset_pending: bool = False
|
|
||||||
pacing_reset_completed: bool = False
|
|
||||||
|
|
||||||
data_cs: trio.CancelScope | None = None
|
data_cs: trio.CancelScope | None = None
|
||||||
result: tuple[
|
result: tuple[
|
||||||
|
|
@ -494,16 +493,10 @@ async def get_bars(
|
||||||
|
|
||||||
async def query():
|
async def query():
|
||||||
|
|
||||||
nonlocal result, data_cs
|
global _failed_resets
|
||||||
nonlocal cancelled_count, failed_resets
|
nonlocal result, data_cs, end_dt, nodatas_count
|
||||||
nonlocal pacing_reset_pending, pacing_reset_completed
|
|
||||||
|
|
||||||
while True:
|
while _failed_resets < max_failed_resets:
|
||||||
if failed_resets >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB history reset failed {failed_resets} '
|
|
||||||
f'times for {fqme}'
|
|
||||||
)
|
|
||||||
try:
|
try:
|
||||||
(
|
(
|
||||||
bars,
|
bars,
|
||||||
|
|
@ -535,10 +528,6 @@ async def get_bars(
|
||||||
)
|
)
|
||||||
# NOTE: REQUIRED to pass back value..
|
# NOTE: REQUIRED to pass back value..
|
||||||
result = None
|
result = None
|
||||||
failed_resets = 0
|
|
||||||
result_ready.set()
|
|
||||||
if data_cs:
|
|
||||||
data_cs.cancel()
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# not enough bars signal, likely due to venue
|
# not enough bars signal, likely due to venue
|
||||||
|
|
@ -598,7 +587,6 @@ async def get_bars(
|
||||||
first_dt,
|
first_dt,
|
||||||
last_dt,
|
last_dt,
|
||||||
)
|
)
|
||||||
failed_resets = 0
|
|
||||||
|
|
||||||
# signal data reset loop parent task
|
# signal data reset loop parent task
|
||||||
result_ready.set()
|
result_ready.set()
|
||||||
|
|
@ -623,21 +611,27 @@ async def get_bars(
|
||||||
# the frame dt index since the upper layer may
|
# the frame dt index since the upper layer may
|
||||||
# be doing so concurrently and we don't want to
|
# be doing so concurrently and we don't want to
|
||||||
# be delivering frames that weren't asked for.
|
# be delivering frames that weren't asked for.
|
||||||
|
# try to decrement start point and look further back
|
||||||
|
# end_dt = end_dt.subtract(seconds=2000)
|
||||||
|
logmsg = "SUBTRACTING DAY from DT index"
|
||||||
|
if end_dt is not None:
|
||||||
|
end_dt = end_dt.subtract(days=1)
|
||||||
|
elif end_dt is None:
|
||||||
|
end_dt = now().subtract(days=1)
|
||||||
|
|
||||||
log.warning(
|
log.warning(
|
||||||
f'NO DATA found ending @ {end_dt}\n'
|
f'NO DATA found ending @ {end_dt}\n'
|
||||||
|
+ logmsg
|
||||||
)
|
)
|
||||||
# Match get_bars()'s empty-bars return: None is
|
|
||||||
# internal to the IB adapter, not its backfill
|
if nodatas_count >= max_nodatas:
|
||||||
# protocol. open_history_client() converts it
|
raise DataUnavailable(
|
||||||
# to NoData (or DataUnavailable at history start).
|
f'Presuming {fqme} has no further history '
|
||||||
# Wake the waiting get_bars() task before exiting
|
f'after {max_nodatas} tries..'
|
||||||
# so it does not mistake this reply for a timeout.
|
)
|
||||||
result = None
|
|
||||||
failed_resets = 0
|
nodatas_count += 1
|
||||||
result_ready.set()
|
continue
|
||||||
if data_cs:
|
|
||||||
data_cs.cancel()
|
|
||||||
return None
|
|
||||||
|
|
||||||
elif (
|
elif (
|
||||||
'API historical data query cancelled'
|
'API historical data query cancelled'
|
||||||
|
|
@ -648,12 +642,6 @@ async def get_bars(
|
||||||
'Query cancelled by IB (:eyeroll:):\n'
|
'Query cancelled by IB (:eyeroll:):\n'
|
||||||
f'{err.message}'
|
f'{err.message}'
|
||||||
)
|
)
|
||||||
cancelled_count += 1
|
|
||||||
if cancelled_count >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB cancelled {cancelled_count} history '
|
|
||||||
f'queries for {fqme}'
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
elif (
|
elif (
|
||||||
|
|
@ -674,7 +662,6 @@ async def get_bars(
|
||||||
)
|
)
|
||||||
|
|
||||||
client = proxy._aio_ns.ib.client
|
client = proxy._aio_ns.ib.client
|
||||||
pacing_reset_pending = True
|
|
||||||
|
|
||||||
# cancel any existing reset task
|
# cancel any existing reset task
|
||||||
if data_cs:
|
if data_cs:
|
||||||
|
|
@ -682,81 +669,69 @@ async def get_bars(
|
||||||
data_cs.cancel()
|
data_cs.cancel()
|
||||||
|
|
||||||
# spawn new data reset task
|
# spawn new data reset task
|
||||||
async with _data_reset_lock:
|
data_cs, reset_done = await tn.start(
|
||||||
(
|
|
||||||
data_cs,
|
|
||||||
reset_done,
|
|
||||||
reset_result,
|
|
||||||
) = await tn.start(
|
|
||||||
partial(
|
partial(
|
||||||
wait_on_data_reset,
|
wait_on_data_reset,
|
||||||
proxy,
|
proxy,
|
||||||
reset_type='connection'
|
reset_type='connection'
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
await reset_done.wait()
|
if reset_done:
|
||||||
if reset_result[0]:
|
_failed_resets = 0
|
||||||
failed_resets = 0
|
|
||||||
else:
|
else:
|
||||||
failed_resets += 1
|
_failed_resets += 1
|
||||||
pacing_reset_pending = False
|
|
||||||
pacing_reset_completed = True
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
else:
|
else:
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
# TODO: make this global across all history task/requests
|
||||||
|
# such that simultaneous symbol queries don't try data resettingn
|
||||||
|
# too fast..
|
||||||
|
unset_resetter: bool = False
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.collapse_eg(),
|
tractor.trionics.collapse_eg(),
|
||||||
trio.open_nursery() as tn
|
trio.open_nursery() as tn
|
||||||
):
|
):
|
||||||
|
|
||||||
# Run history until a result or bounded reset failure.
|
# start history request that we allow
|
||||||
|
# to run indefinitely until a result is acquired
|
||||||
tn.start_soon(query)
|
tn.start_soon(query)
|
||||||
|
|
||||||
# Serialize farm resets across every history requester.
|
# start history reset loop which waits up to the timeout
|
||||||
|
# for a result before triggering a data feed reset.
|
||||||
while not result_ready.is_set():
|
while not result_ready.is_set():
|
||||||
|
|
||||||
with trio.move_on_after(feed_reset_timeout):
|
with trio.move_on_after(feed_reset_timeout):
|
||||||
await result_ready.wait()
|
await result_ready.wait()
|
||||||
break
|
break
|
||||||
|
|
||||||
async with _data_reset_lock:
|
if _data_resetter_task:
|
||||||
if result_ready.is_set():
|
# don't double invoke the reset hack if another
|
||||||
break
|
# requester task already has it covered.
|
||||||
if pacing_reset_completed:
|
|
||||||
pacing_reset_completed = False
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
(
|
else:
|
||||||
data_cs,
|
_data_resetter_task = trio.lowlevel.current_task()
|
||||||
reset_done,
|
unset_resetter: bool = True
|
||||||
reset_result,
|
|
||||||
) = await tn.start(
|
# spawn new data reset task
|
||||||
|
data_cs, reset_done = await tn.start(
|
||||||
partial(
|
partial(
|
||||||
wait_on_data_reset,
|
wait_on_data_reset,
|
||||||
proxy,
|
proxy,
|
||||||
reset_type='data',
|
reset_type='data',
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
# sync wait on reset to complete
|
||||||
await reset_done.wait()
|
await reset_done.wait()
|
||||||
|
|
||||||
if (
|
_data_resetter_task = (
|
||||||
result_ready.is_set()
|
None
|
||||||
or
|
if unset_resetter
|
||||||
reset_result[0]
|
else _data_resetter_task
|
||||||
):
|
|
||||||
failed_resets = 0
|
|
||||||
elif pacing_reset_pending:
|
|
||||||
# Pacing recovery deliberately cancelled this reset
|
|
||||||
# and will consume the retry itself.
|
|
||||||
pass
|
|
||||||
else:
|
|
||||||
failed_resets += 1
|
|
||||||
if failed_resets >= max_failed_resets:
|
|
||||||
raise DataUnavailable(
|
|
||||||
f'IB history reset failed '
|
|
||||||
f'{failed_resets} times for {fqme}'
|
|
||||||
)
|
)
|
||||||
|
assert result
|
||||||
return (
|
return (
|
||||||
result,
|
result,
|
||||||
data_cs is not None,
|
data_cs is not None,
|
||||||
|
|
|
||||||
|
|
@ -639,40 +639,12 @@ async def get_mkt_info(
|
||||||
|
|
||||||
# if possible register the bs_mktid to the just-built
|
# if possible register the bs_mktid to the just-built
|
||||||
# mkt so that it can be retreived by order mode tasks later.
|
# mkt so that it can be retreived by order mode tasks later.
|
||||||
|
# TODO NOTE: this is going to be problematic if/when we split
|
||||||
|
# out the datatd vs. brokerd actors since the mktmap lookup
|
||||||
|
# table will now be inaccessible..
|
||||||
if proxy is not None:
|
if proxy is not None:
|
||||||
cache_contract(
|
client: Client = proxy._aio_ns
|
||||||
proxy._aio_ns,
|
client._contracts[mkt.bs_fqme] = con
|
||||||
mkt,
|
client._cons2mkts[con] = mkt
|
||||||
con,
|
|
||||||
)
|
|
||||||
|
|
||||||
return mkt, details
|
return mkt, details
|
||||||
|
|
||||||
|
|
||||||
def cache_contract(
|
|
||||||
client: Client,
|
|
||||||
mkt: MktPair,
|
|
||||||
con: ibis.Contract,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Register a (qualified) contract + mkt-info pair on the
|
|
||||||
given (api) `Client`'s actor-local lookup tables.
|
|
||||||
|
|
||||||
Cached under BOTH fqme key-forms since consumers vary:
|
|
||||||
- `mkt.bs_fqme` (eg. 'nvda.nasdaq'): read by
|
|
||||||
`Client.submit_limit()` for order requests,
|
|
||||||
- `mkt.fqme` (eg. 'nvda.nasdaq.ib'): read by the
|
|
||||||
fill-time pp-update (symcache-backup-table) path in
|
|
||||||
`.broker.emit_pp_update()`.
|
|
||||||
|
|
||||||
NOTE: post the (datad|brokerd)-actor-split this MUST be
|
|
||||||
run (in the trading actor) for every mkt either eagerly
|
|
||||||
at `.broker.open_trade_dialog()` startup or lazily per
|
|
||||||
order request; there is no in-proc feed setup doing it
|
|
||||||
implicitly anymore!
|
|
||||||
|
|
||||||
'''
|
|
||||||
client._contracts[mkt.bs_fqme] = con
|
|
||||||
client._contracts[mkt.fqme] = con
|
|
||||||
client._cons2mkts[con] = mkt
|
|
||||||
|
|
|
||||||
|
|
@ -66,24 +66,10 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# per-daemon-kind (sub)mod groups: declares which of our
|
# tractor RPC enable arg
|
||||||
# submods host the eps run by each daemon-actor kind as
|
__enable_modules__: list[str] = [
|
||||||
# defined by `piker.data.validate._eps`.
|
|
||||||
_brokerd_mods: list[str] = [
|
|
||||||
'api',
|
'api',
|
||||||
'broker',
|
'broker',
|
||||||
]
|
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
|
||||||
'api',
|
|
||||||
'feed',
|
'feed',
|
||||||
'symbols',
|
'symbols',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
|
||||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
|
||||||
_brokerd_mods
|
|
||||||
+
|
|
||||||
_datad_mods
|
|
||||||
))
|
|
||||||
|
|
|
||||||
|
|
@ -52,7 +52,6 @@ from piker.brokers._util import (
|
||||||
SymbolNotFound,
|
SymbolNotFound,
|
||||||
BrokerError,
|
BrokerError,
|
||||||
DataThrottle,
|
DataThrottle,
|
||||||
get_or_raise_on_pair_schema_mismatch,
|
|
||||||
)
|
)
|
||||||
from piker.accounting import Transaction
|
from piker.accounting import Transaction
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
|
|
@ -118,9 +117,6 @@ class InvalidKey(ValueError):
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
|
||||||
from tractor._exceptions import reg_err_types
|
|
||||||
reg_err_types([InvalidKey])
|
|
||||||
|
|
||||||
|
|
||||||
class Client:
|
class Client:
|
||||||
|
|
||||||
|
|
@ -506,16 +502,7 @@ class Client:
|
||||||
|
|
||||||
# NOTE: always cache in pairs tables for faster lookup
|
# NOTE: always cache in pairs tables for faster lookup
|
||||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||||
# pair = Pair(xname=xkey, **data)
|
pair = Pair(xname=xkey, **data)
|
||||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
|
||||||
pair_type=Pair,
|
|
||||||
fields_data=dict(
|
|
||||||
xname=xkey,
|
|
||||||
**data,
|
|
||||||
),
|
|
||||||
provider_name='kraken',
|
|
||||||
# api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
|
||||||
)
|
|
||||||
|
|
||||||
# register the above `Pair` structs for all
|
# register the above `Pair` structs for all
|
||||||
# key-sets/monikers: a set of 4 (frickin) tables
|
# key-sets/monikers: a set of 4 (frickin) tables
|
||||||
|
|
|
||||||
|
|
@ -67,15 +67,13 @@ from piker.log import (
|
||||||
get_logger,
|
get_logger,
|
||||||
)
|
)
|
||||||
from piker.data import open_symcache
|
from piker.data import open_symcache
|
||||||
from piker.data._web_bs import (
|
|
||||||
open_autorecon_ws,
|
|
||||||
NoBsWs,
|
|
||||||
)
|
|
||||||
from .api import (
|
from .api import (
|
||||||
Client,
|
Client,
|
||||||
BrokerError,
|
BrokerError,
|
||||||
)
|
)
|
||||||
from .feed import (
|
from .feed import (
|
||||||
|
open_autorecon_ws,
|
||||||
|
NoBsWs,
|
||||||
stream_messages,
|
stream_messages,
|
||||||
)
|
)
|
||||||
from .ledger import (
|
from .ledger import (
|
||||||
|
|
@ -99,9 +97,6 @@ MsgUnion = Union[
|
||||||
class TooFastEdit(Exception):
|
class TooFastEdit(Exception):
|
||||||
'Edit requests faster then api submissions'
|
'Edit requests faster then api submissions'
|
||||||
|
|
||||||
from tractor._exceptions import reg_err_types
|
|
||||||
reg_err_types([TooFastEdit])
|
|
||||||
|
|
||||||
|
|
||||||
# TODO: make this wrap the `Client` and `ws` instances
|
# TODO: make this wrap the `Client` and `ws` instances
|
||||||
# and give it methods to submit cancel vs. add vs. edit
|
# and give it methods to submit cancel vs. add vs. edit
|
||||||
|
|
|
||||||
|
|
@ -19,9 +19,6 @@ Symbology defs and search.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
from decimal import Decimal
|
from decimal import Decimal
|
||||||
from typing import (
|
|
||||||
ClassVar,
|
|
||||||
)
|
|
||||||
|
|
||||||
import tractor
|
import tractor
|
||||||
|
|
||||||
|
|
@ -89,14 +86,9 @@ class Pair(Struct):
|
||||||
short_position_limit: float = 0
|
short_position_limit: float = 0
|
||||||
long_position_limit: float = float('inf')
|
long_position_limit: float = float('inf')
|
||||||
|
|
||||||
# TODO, add API note when this was added!
|
|
||||||
execution_venue: str|None = None
|
|
||||||
|
|
||||||
# TODO: should we make this a literal NamespacePath ref?
|
# TODO: should we make this a literal NamespacePath ref?
|
||||||
ns_path: str = 'piker.brokers.kraken:Pair'
|
ns_path: str = 'piker.brokers.kraken:Pair'
|
||||||
|
|
||||||
_api_url: ClassVar[str] = 'https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs'
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def bs_mktid(self) -> str:
|
def bs_mktid(self) -> str:
|
||||||
'''
|
'''
|
||||||
|
|
|
||||||
|
|
@ -95,9 +95,6 @@ _time_frames = {
|
||||||
class QuestradeError(Exception):
|
class QuestradeError(Exception):
|
||||||
"Non-200 OK response code"
|
"Non-200 OK response code"
|
||||||
|
|
||||||
from tractor._exceptions import reg_err_types
|
|
||||||
reg_err_types([QuestradeError])
|
|
||||||
|
|
||||||
|
|
||||||
class ContractsKey(NamedTuple):
|
class ContractsKey(NamedTuple):
|
||||||
symbol: str
|
symbol: str
|
||||||
|
|
|
||||||
|
|
@ -159,11 +159,11 @@ class DarkBook(Struct):
|
||||||
_DEFAULT_SIZE: float = 1.0
|
_DEFAULT_SIZE: float = 1.0
|
||||||
|
|
||||||
|
|
||||||
async def _clear_dark_triggers(
|
async def clear_dark_triggers(
|
||||||
|
|
||||||
router: Router,
|
router: Router,
|
||||||
brokerd_orders_stream: tractor.MsgStream,
|
brokerd_orders_stream: tractor.MsgStream,
|
||||||
quote_stream: trionics.AsyncReceiver,
|
quote_stream: tractor.MsgStream,
|
||||||
broker: str,
|
broker: str,
|
||||||
fqme: str,
|
fqme: str,
|
||||||
|
|
||||||
|
|
@ -182,6 +182,7 @@ async def _clear_dark_triggers(
|
||||||
# - port to the new ringbuf stuff in `tractor.ipc`!
|
# - port to the new ringbuf stuff in `tractor.ipc`!
|
||||||
# - numba all this!
|
# - numba all this!
|
||||||
# - this stream may eventually contain multiple symbols
|
# - this stream may eventually contain multiple symbols
|
||||||
|
quote_stream._raise_on_lag = False
|
||||||
async for quotes in quote_stream:
|
async for quotes in quote_stream:
|
||||||
# start = time.time()
|
# start = time.time()
|
||||||
for sym, quote in quotes.items():
|
for sym, quote in quotes.items():
|
||||||
|
|
@ -313,32 +314,6 @@ async def _clear_dark_triggers(
|
||||||
# print(f'execs scan took: {time.time() - start}')
|
# print(f'execs scan took: {time.time() - start}')
|
||||||
|
|
||||||
|
|
||||||
async def clear_dark_triggers(
|
|
||||||
router: Router,
|
|
||||||
brokerd_orders_stream: tractor.MsgStream,
|
|
||||||
quote_stream: tractor.MsgStream,
|
|
||||||
broker: str,
|
|
||||||
fqme: str,
|
|
||||||
book: DarkBook,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Run dark clearing on a dedicated non-strict quote subscription.
|
|
||||||
|
|
||||||
'''
|
|
||||||
async with quote_stream.subscribe(
|
|
||||||
raise_on_lag=False,
|
|
||||||
) as quotes_stream:
|
|
||||||
await _clear_dark_triggers(
|
|
||||||
router,
|
|
||||||
brokerd_orders_stream,
|
|
||||||
quotes_stream,
|
|
||||||
broker,
|
|
||||||
fqme,
|
|
||||||
book,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TradesRelay(Struct):
|
class TradesRelay(Struct):
|
||||||
|
|
||||||
# for now we keep only a single connection open with
|
# for now we keep only a single connection open with
|
||||||
|
|
@ -360,14 +335,9 @@ class TradesRelay(Struct):
|
||||||
@acm
|
@acm
|
||||||
async def open_brokerd_dialog(
|
async def open_brokerd_dialog(
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
|
portal: tractor.Portal,
|
||||||
exec_mode: str,
|
exec_mode: str,
|
||||||
fqme: str|None = None,
|
fqme: str|None = None,
|
||||||
|
|
||||||
# XXX: explicit (already spawned) trading-actor override,
|
|
||||||
# currently only used by the `piker ledger` cli which
|
|
||||||
# boots its own ad-hoc `brokerd`-like actor; normally we
|
|
||||||
# (lazily) spawn/find the `brokerd.<broker>` daemon here.
|
|
||||||
portal: tractor.Portal|None = None,
|
|
||||||
loglevel: str|None = None,
|
loglevel: str|None = None,
|
||||||
|
|
||||||
) -> tuple[
|
) -> tuple[
|
||||||
|
|
@ -381,10 +351,6 @@ async def open_brokerd_dialog(
|
||||||
paper engine instance depending on live trading support for the
|
paper engine instance depending on live trading support for the
|
||||||
broker backend, configuration, or client code usage.
|
broker backend, configuration, or client code usage.
|
||||||
|
|
||||||
NOTE: this is now the ONE place where a (live, credentialed)
|
|
||||||
`brokerd.<broker>` daemon-actor gets (lazily) booted; pure
|
|
||||||
data/paper sessions should never spawn one!
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
get_console_log(
|
get_console_log(
|
||||||
level=loglevel,
|
level=loglevel,
|
||||||
|
|
@ -450,29 +416,16 @@ async def open_brokerd_dialog(
|
||||||
)
|
)
|
||||||
exec_mode: str = 'paper'
|
exec_mode: str = 'paper'
|
||||||
|
|
||||||
@acm
|
if (
|
||||||
async def acquire_live_portal():
|
trades_endpoint is not None
|
||||||
'''
|
or
|
||||||
Deliver a portal to the (live, credentialed) trading
|
exec_mode != 'paper'
|
||||||
actor hosting the backend's `open_trade_dialog()` ep:
|
):
|
||||||
either the caller-provided override or the (maybe
|
# open live brokerd trades endpoint
|
||||||
lazily spawned) `brokerd.<broker>` service daemon.
|
open_trades_endpoint = portal.open_context(
|
||||||
|
trades_endpoint,
|
||||||
'''
|
|
||||||
if portal is not None:
|
|
||||||
yield portal
|
|
||||||
return
|
|
||||||
|
|
||||||
# XXX: the ONE (normal) place a `brokerd.<broker>`
|
|
||||||
# daemon-actor gets booted in the runtime B)
|
|
||||||
from piker.brokers._daemon import (
|
|
||||||
maybe_spawn_brokerd,
|
|
||||||
)
|
|
||||||
async with maybe_spawn_brokerd(
|
|
||||||
brokermod.name,
|
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
) as live_portal:
|
)
|
||||||
yield live_portal
|
|
||||||
|
|
||||||
@acm
|
@acm
|
||||||
async def maybe_open_paper_ep():
|
async def maybe_open_paper_ep():
|
||||||
|
|
@ -484,14 +437,7 @@ async def open_brokerd_dialog(
|
||||||
return
|
return
|
||||||
|
|
||||||
# open trades-dialog endpoint with backend broker
|
# open trades-dialog endpoint with backend broker
|
||||||
async with (
|
async with open_trades_endpoint as msg:
|
||||||
acquire_live_portal() as live_portal,
|
|
||||||
|
|
||||||
live_portal.open_context(
|
|
||||||
trades_endpoint,
|
|
||||||
loglevel=loglevel,
|
|
||||||
) as msg,
|
|
||||||
):
|
|
||||||
ctx, first = msg
|
ctx, first = msg
|
||||||
|
|
||||||
# runtime indication that the backend can't support live
|
# runtime indication that the backend can't support live
|
||||||
|
|
@ -635,6 +581,7 @@ class Router(Struct):
|
||||||
async def maybe_open_brokerd_dialog(
|
async def maybe_open_brokerd_dialog(
|
||||||
self,
|
self,
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
|
portal: tractor.Portal,
|
||||||
exec_mode: str,
|
exec_mode: str,
|
||||||
fqme: str,
|
fqme: str,
|
||||||
loglevel: str,
|
loglevel: str,
|
||||||
|
|
@ -659,6 +606,7 @@ class Router(Struct):
|
||||||
|
|
||||||
async with open_brokerd_dialog(
|
async with open_brokerd_dialog(
|
||||||
brokermod=brokermod,
|
brokermod=brokermod,
|
||||||
|
portal=portal,
|
||||||
exec_mode=exec_mode,
|
exec_mode=exec_mode,
|
||||||
fqme=fqme,
|
fqme=fqme,
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
|
|
@ -720,6 +668,7 @@ class Router(Struct):
|
||||||
brokername, _, _, _ = unpack_fqme(fqme)
|
brokername, _, _, _ = unpack_fqme(fqme)
|
||||||
brokermod = feed.mods[brokername]
|
brokermod = feed.mods[brokername]
|
||||||
broker = brokermod.name
|
broker = brokermod.name
|
||||||
|
portal = feed.portals[brokermod]
|
||||||
|
|
||||||
# XXX: this should be initial price quote from target provider
|
# XXX: this should be initial price quote from target provider
|
||||||
flume = feed.flumes[fqme]
|
flume = feed.flumes[fqme]
|
||||||
|
|
@ -733,6 +682,7 @@ class Router(Struct):
|
||||||
|
|
||||||
async with self.maybe_open_brokerd_dialog(
|
async with self.maybe_open_brokerd_dialog(
|
||||||
brokermod=brokermod,
|
brokermod=brokermod,
|
||||||
|
portal=portal,
|
||||||
exec_mode=exec_mode,
|
exec_mode=exec_mode,
|
||||||
fqme=fqme,
|
fqme=fqme,
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
|
|
@ -835,10 +785,11 @@ async def _setup_persistent_emsd(
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
if loglevel:
|
if loglevel:
|
||||||
get_console_log(
|
_log = get_console_log(
|
||||||
level=loglevel,
|
level=loglevel,
|
||||||
name=subsys,
|
name=subsys,
|
||||||
)
|
)
|
||||||
|
assert _log.name == 'piker.clearing'
|
||||||
|
|
||||||
global _router
|
global _router
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -27,7 +27,7 @@ from types import ModuleType
|
||||||
import click
|
import click
|
||||||
import trio
|
import trio
|
||||||
import tractor
|
import tractor
|
||||||
from tractor.discovery import parse_endpoints
|
from tractor._multiaddr import parse_maddr
|
||||||
|
|
||||||
from ..log import (
|
from ..log import (
|
||||||
get_console_log,
|
get_console_log,
|
||||||
|
|
@ -46,27 +46,48 @@ log = get_logger('piker.cli')
|
||||||
|
|
||||||
|
|
||||||
def load_trans_eps(
|
def load_trans_eps(
|
||||||
network: dict|None = None,
|
network: dict | None = None,
|
||||||
maddrs: str|tuple[str, ...]|list[str]|None = None,
|
maddrs: list[tuple] | None = None,
|
||||||
|
|
||||||
) -> dict[str, list]:
|
) -> dict[str, dict[str, dict]]:
|
||||||
'''
|
|
||||||
Load transport endpoints from a `[network]` config
|
|
||||||
table or CLI-provided multiaddr strings, delegating
|
|
||||||
to `tractor.discovery.parse_endpoints()`.
|
|
||||||
|
|
||||||
'''
|
# transport-oriented endpoint multi-addresses
|
||||||
if network and not maddrs:
|
eps: dict[
|
||||||
return parse_endpoints(network)
|
str, # service name, eg. `pikerd`, `emsd`..
|
||||||
|
|
||||||
|
# libp2p style multi-addresses parsed into prot layers
|
||||||
|
list[dict[str, str | int]]
|
||||||
|
] = {}
|
||||||
|
|
||||||
|
if (
|
||||||
|
network
|
||||||
|
and
|
||||||
|
not maddrs
|
||||||
|
):
|
||||||
|
# load network section and (attempt to) connect all endpoints
|
||||||
|
# which are reachable B)
|
||||||
|
for key, maddrs in network.items():
|
||||||
|
match key:
|
||||||
|
|
||||||
|
# TODO: resolve table across multiple discov
|
||||||
|
# prots Bo
|
||||||
|
case 'resolv':
|
||||||
|
pass
|
||||||
|
|
||||||
|
case 'pikerd':
|
||||||
|
dname: str = key
|
||||||
|
for maddr in maddrs:
|
||||||
|
layers: dict = parse_maddr(maddr)
|
||||||
|
eps.setdefault(
|
||||||
|
dname,
|
||||||
|
[],
|
||||||
|
).append(layers)
|
||||||
|
|
||||||
elif maddrs:
|
elif maddrs:
|
||||||
if isinstance(maddrs, str):
|
# presume user is manually specifying the root actor ep.
|
||||||
maddrs = [maddrs]
|
eps['pikerd'] = [parse_maddr(maddr)]
|
||||||
return parse_endpoints(
|
|
||||||
{'pikerd': list(maddrs)}
|
|
||||||
)
|
|
||||||
|
|
||||||
return {}
|
return eps
|
||||||
|
|
||||||
|
|
||||||
@click.command()
|
@click.command()
|
||||||
|
|
@ -87,13 +108,13 @@ def load_trans_eps(
|
||||||
help='Enable tractor debug mode',
|
help='Enable tractor debug mode',
|
||||||
)
|
)
|
||||||
@click.option(
|
@click.option(
|
||||||
'--maddrs',
|
'--maddr',
|
||||||
'-m',
|
'-m',
|
||||||
multiple=True,
|
default=None,
|
||||||
help='Multiaddrs to bind or contact',
|
help='Multiaddrs to bind or contact',
|
||||||
)
|
)
|
||||||
def pikerd(
|
def pikerd(
|
||||||
maddrs: tuple[str, ...],
|
maddr: list[str] | None,
|
||||||
loglevel: str,
|
loglevel: str,
|
||||||
tl: bool,
|
tl: bool,
|
||||||
pdb: bool,
|
pdb: bool,
|
||||||
|
|
@ -124,8 +145,7 @@ def pikerd(
|
||||||
))
|
))
|
||||||
|
|
||||||
# service-actor registry endpoint socket-address set
|
# service-actor registry endpoint socket-address set
|
||||||
regaddrs: list = []
|
regaddrs: list[tuple[str, int]] = []
|
||||||
tpt_bind_addrs: list|None = None
|
|
||||||
|
|
||||||
conf, _ = config.load(
|
conf, _ = config.load(
|
||||||
conf_name='conf',
|
conf_name='conf',
|
||||||
|
|
@ -133,8 +153,7 @@ def pikerd(
|
||||||
network: dict = conf.get('network')
|
network: dict = conf.get('network')
|
||||||
if (
|
if (
|
||||||
network is None
|
network is None
|
||||||
and
|
and not maddr
|
||||||
not maddrs
|
|
||||||
):
|
):
|
||||||
regaddrs = [(
|
regaddrs = [(
|
||||||
_default_registry_host,
|
_default_registry_host,
|
||||||
|
|
@ -142,23 +161,15 @@ def pikerd(
|
||||||
)]
|
)]
|
||||||
|
|
||||||
else:
|
else:
|
||||||
from tractor.devx import maybe_open_crash_handler
|
eps: dict = load_trans_eps(
|
||||||
with maybe_open_crash_handler(pdb=pdb):
|
|
||||||
eps: dict[str, list] = load_trans_eps(
|
|
||||||
network,
|
network,
|
||||||
maddrs,
|
maddr,
|
||||||
)
|
|
||||||
|
|
||||||
# pikerd bind addresses (may differ from
|
|
||||||
# registry if regd is separate)
|
|
||||||
tpt_bind_addrs = eps.get('pikerd')
|
|
||||||
|
|
||||||
# registry: dedicated `regd` key, or fall
|
|
||||||
# back to pikerd addrs (pikerd IS the registry)
|
|
||||||
regaddrs = eps.get(
|
|
||||||
'regd',
|
|
||||||
tpt_bind_addrs,
|
|
||||||
)
|
)
|
||||||
|
for layers in eps['pikerd']:
|
||||||
|
regaddrs.append((
|
||||||
|
layers['ipv4']['addr'],
|
||||||
|
layers['tcp']['port'],
|
||||||
|
))
|
||||||
|
|
||||||
from .. import service
|
from .. import service
|
||||||
|
|
||||||
|
|
@ -167,7 +178,6 @@ def pikerd(
|
||||||
async with (
|
async with (
|
||||||
service.open_pikerd(
|
service.open_pikerd(
|
||||||
registry_addrs=regaddrs,
|
registry_addrs=regaddrs,
|
||||||
tpt_bind_addrs=tpt_bind_addrs,
|
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
debug_mode=pdb,
|
debug_mode=pdb,
|
||||||
# enable_transports=['uds'],
|
# enable_transports=['uds'],
|
||||||
|
|
@ -252,6 +262,7 @@ def cli(
|
||||||
|
|
||||||
# TODO: load endpoints from `conf::[network].pikerd`
|
# TODO: load endpoints from `conf::[network].pikerd`
|
||||||
# - pikerd vs. regd, separate registry daemon?
|
# - pikerd vs. regd, separate registry daemon?
|
||||||
|
# - expose datad vs. brokerd?
|
||||||
# - bind emsd with certain perms on public iface?
|
# - bind emsd with certain perms on public iface?
|
||||||
regaddrs: list[tuple[str, int]] = regaddr or [(
|
regaddrs: list[tuple[str, int]] = regaddr or [(
|
||||||
_default_registry_host,
|
_default_registry_host,
|
||||||
|
|
@ -334,7 +345,7 @@ def services(
|
||||||
if not ports:
|
if not ports:
|
||||||
ports: list[int] = [_default_registry_port]
|
ports: list[int] = [_default_registry_port]
|
||||||
|
|
||||||
addr = tractor.discovery._addr.wrap_address(
|
addr = tractor._addr.wrap_address(
|
||||||
addr=(host, ports[0])
|
addr=(host, ports[0])
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -36,7 +36,6 @@ except ModuleNotFoundError:
|
||||||
import tomli as tomllib
|
import tomli as tomllib
|
||||||
|
|
||||||
|
|
||||||
from tractor._exceptions import reg_err_types
|
|
||||||
from .log import get_logger
|
from .log import get_logger
|
||||||
|
|
||||||
log = get_logger('broker-config')
|
log = get_logger('broker-config')
|
||||||
|
|
@ -173,12 +172,6 @@ class ConfigurationError(Exception):
|
||||||
class NoSignature(ConfigurationError):
|
class NoSignature(ConfigurationError):
|
||||||
'No credentials setup for broker backend!'
|
'No credentials setup for broker backend!'
|
||||||
|
|
||||||
# auto-register for tractor IPC exc-marshalling.
|
|
||||||
reg_err_types([
|
|
||||||
ConfigurationError,
|
|
||||||
*ConfigurationError.__subclasses__(),
|
|
||||||
])
|
|
||||||
|
|
||||||
|
|
||||||
def _override_config_dir(
|
def _override_config_dir(
|
||||||
path: str
|
path: str
|
||||||
|
|
@ -187,48 +180,6 @@ def _override_config_dir(
|
||||||
_config_dir = path
|
_config_dir = path
|
||||||
|
|
||||||
|
|
||||||
def _maybe_use_test_dir() -> None:
|
|
||||||
'''
|
|
||||||
When running under the `pytest` harness, override
|
|
||||||
the config dir to the per-test-tmp dir "passed down"
|
|
||||||
the actor tree via `tractor`'s runtime-vars
|
|
||||||
inheritance mechanism.
|
|
||||||
|
|
||||||
See the `tractor_runtime_overrides` usage in our
|
|
||||||
`tests.conftest._open_test_pikerd()` as well as
|
|
||||||
`.service._actor_runtime.open_piker_runtime()` for
|
|
||||||
the root-actor's pre-loading of the var state.
|
|
||||||
|
|
||||||
NOTE: this must be checked lazily at config-path
|
|
||||||
access time (NOT import time) since sub-actors only
|
|
||||||
receive runtime-vars once their `tractor` runtime
|
|
||||||
has fully booted.
|
|
||||||
|
|
||||||
'''
|
|
||||||
global _config_dir
|
|
||||||
import tractor
|
|
||||||
actor = tractor.current_actor(
|
|
||||||
err_on_no_runtime=False,
|
|
||||||
)
|
|
||||||
if actor is None:
|
|
||||||
return
|
|
||||||
|
|
||||||
rvs: dict = tractor.runtime._state._runtime_vars
|
|
||||||
pvars: dict|None = rvs.get('piker_vars')
|
|
||||||
if (
|
|
||||||
pvars
|
|
||||||
and
|
|
||||||
(testdir := pvars.get('piker_test_dir'))
|
|
||||||
):
|
|
||||||
testdirpath = Path(testdir)
|
|
||||||
assert testdirpath.exists(), (
|
|
||||||
f'piker test harness might be borked!?\n'
|
|
||||||
f'testdirpath: {testdirpath!r}\n'
|
|
||||||
)
|
|
||||||
if _config_dir != testdirpath:
|
|
||||||
_override_config_dir(testdirpath)
|
|
||||||
|
|
||||||
|
|
||||||
def _conf_fn_w_ext(
|
def _conf_fn_w_ext(
|
||||||
name: str,
|
name: str,
|
||||||
) -> str:
|
) -> str:
|
||||||
|
|
@ -242,7 +193,6 @@ def get_conf_dir() -> Path:
|
||||||
on the local filesystem.
|
on the local filesystem.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
_maybe_use_test_dir()
|
|
||||||
return _config_dir
|
return _config_dir
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -268,7 +218,7 @@ def get_conf_path(
|
||||||
assert str(conf_name) in _conf_names
|
assert str(conf_name) in _conf_names
|
||||||
|
|
||||||
fn = _conf_fn_w_ext(conf_name)
|
fn = _conf_fn_w_ext(conf_name)
|
||||||
return get_conf_dir() / Path(fn)
|
return _config_dir / Path(fn)
|
||||||
|
|
||||||
|
|
||||||
def repodir() -> Path:
|
def repodir() -> Path:
|
||||||
|
|
@ -313,9 +263,8 @@ def load(
|
||||||
|
|
||||||
'''
|
'''
|
||||||
# create the $HOME/.config/piker dir if dne
|
# create the $HOME/.config/piker dir if dne
|
||||||
conf_dir: Path = get_conf_dir()
|
if not _config_dir.is_dir():
|
||||||
if not conf_dir.is_dir():
|
_config_dir.mkdir(
|
||||||
conf_dir.mkdir(
|
|
||||||
parents=True,
|
parents=True,
|
||||||
exist_ok=True,
|
exist_ok=True,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,312 +0,0 @@
|
||||||
# piker: trading gear for hackers
|
|
||||||
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
|
|
||||||
|
|
||||||
# This program is free software: you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU Affero General Public License as published by
|
|
||||||
# the Free Software Foundation, either version 3 of the License, or
|
|
||||||
# (at your option) any later version.
|
|
||||||
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU Affero General Public License for more details.
|
|
||||||
|
|
||||||
# You should have received a copy of the GNU Affero General Public License
|
|
||||||
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
|
||||||
|
|
||||||
'''
|
|
||||||
Data-daemon-actor "endpoint-hooks": the service task entry
|
|
||||||
points for `datad.<brokername>`, the per-provider real-time
|
|
||||||
and historical market-data feed daemon.
|
|
||||||
|
|
||||||
The (data vs. broker)d-split mirrors the ep-groupings in
|
|
||||||
`piker.data.validate._eps`: this daemon hosts all `'datad'`
|
|
||||||
eps (live quotes, history loading, symbology search) while
|
|
||||||
its `brokerd.<brokername>` sibling hosts only the live
|
|
||||||
order-control (and thus credentialed) `'brokerd'` eps.
|
|
||||||
|
|
||||||
'''
|
|
||||||
from __future__ import annotations
|
|
||||||
from contextlib import (
|
|
||||||
asynccontextmanager as acm,
|
|
||||||
)
|
|
||||||
from types import ModuleType
|
|
||||||
from typing import (
|
|
||||||
TYPE_CHECKING,
|
|
||||||
AsyncContextManager,
|
|
||||||
)
|
|
||||||
|
|
||||||
import tractor
|
|
||||||
import trio
|
|
||||||
|
|
||||||
from piker.log import (
|
|
||||||
get_logger,
|
|
||||||
get_console_log,
|
|
||||||
)
|
|
||||||
from . import _util
|
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from .feed import _FeedsBus
|
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
|
||||||
|
|
||||||
# `datad`-actor-always-enabled mods: the data-side successor
|
|
||||||
# to the old `piker.brokers._daemon._data_mods` set.
|
|
||||||
# NOTE: keeping this list as small as possible is part of
|
|
||||||
# our caps-sec model and should be treated with utmost care!
|
|
||||||
_datad_service_mods: list[str] = [
|
|
||||||
'piker.brokers.core',
|
|
||||||
'piker.brokers.data',
|
|
||||||
'piker.data',
|
|
||||||
'piker.data.feed',
|
|
||||||
'piker.data._sampling',
|
|
||||||
'piker.data._daemon',
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@tractor.context
|
|
||||||
async def _setup_persistent_datad(
|
|
||||||
ctx: tractor.Context,
|
|
||||||
brokername: str,
|
|
||||||
loglevel: str|None = None,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Allocate an actor-wide service nursery in this
|
|
||||||
`datad.<brokername>` actor such that data-feed tasks
|
|
||||||
(shm writer-samplers, history backfillers, symbology
|
|
||||||
loaders) can be run in the background persistently by
|
|
||||||
the provider backend as needed.
|
|
||||||
|
|
||||||
'''
|
|
||||||
# NOTE: we only need to setup logging once (and only)
|
|
||||||
# here since all hosted daemon tasks will reference
|
|
||||||
# this same log instance's (actor local) state and thus
|
|
||||||
# don't require any further (level) configuration on
|
|
||||||
# their own B)
|
|
||||||
actor: tractor.Actor = tractor.current_actor()
|
|
||||||
tll: str = actor.loglevel
|
|
||||||
log = get_console_log(
|
|
||||||
level=loglevel or tll,
|
|
||||||
name=_util.subsys,
|
|
||||||
with_tractor_log=bool(tll),
|
|
||||||
)
|
|
||||||
assert log.name == _util.subsys
|
|
||||||
|
|
||||||
# XXX: ALSO enable console logging for the provider
|
|
||||||
# backend's mod subtree (eg. `piker.brokers.ib.*`)
|
|
||||||
# since (pre the datad|brokerd-split) that was done
|
|
||||||
# by `brokerd`'s fixture; without this all backend
|
|
||||||
# records here fall through to the stdlib's bare
|
|
||||||
# (non-colorized) `logging.lastResort` handler!
|
|
||||||
get_console_log(
|
|
||||||
level=loglevel or tll,
|
|
||||||
name=f'piker.brokers.{brokername}',
|
|
||||||
with_tractor_log=bool(tll),
|
|
||||||
)
|
|
||||||
|
|
||||||
from piker.data import feed
|
|
||||||
assert not feed._bus
|
|
||||||
|
|
||||||
# allocate a nursery to the bus for spawning background
|
|
||||||
# tasks which service client IPC requests, normally
|
|
||||||
# `tractor.Context` connections to explicitly required
|
|
||||||
# `datad` endpoints such as:
|
|
||||||
# - `stream_quotes()`,
|
|
||||||
# - `manage_history()`,
|
|
||||||
# - `allocate_persistent_feed()`,
|
|
||||||
# - `open_symbol_search()`
|
|
||||||
# NOTE: see ep invocation details inside `.data.feed`.
|
|
||||||
async with (
|
|
||||||
trio.open_nursery() as service_nursery
|
|
||||||
):
|
|
||||||
bus: _FeedsBus = feed.get_feed_bus(
|
|
||||||
brokername,
|
|
||||||
service_nursery,
|
|
||||||
)
|
|
||||||
assert bus is feed._bus
|
|
||||||
|
|
||||||
# unblock caller
|
|
||||||
await ctx.started()
|
|
||||||
|
|
||||||
# we pin this task to keep the feeds manager active
|
|
||||||
# until the parent actor decides to tear it down
|
|
||||||
await trio.sleep_forever()
|
|
||||||
|
|
||||||
|
|
||||||
def datad_init(
|
|
||||||
brokername: str,
|
|
||||||
loglevel: str|None = None,
|
|
||||||
|
|
||||||
**start_actor_kwargs,
|
|
||||||
|
|
||||||
) -> tuple[
|
|
||||||
ModuleType,
|
|
||||||
dict,
|
|
||||||
AsyncContextManager,
|
|
||||||
]:
|
|
||||||
'''
|
|
||||||
Given an input broker name, load all named arguments
|
|
||||||
which can be passed for daemon endpoint + context spawn
|
|
||||||
as required in every `datad.<brokername>` (actor)
|
|
||||||
service.
|
|
||||||
|
|
||||||
This includes:
|
|
||||||
- load the appropriate <brokername>.py pkg module,
|
|
||||||
- reads any declared `_datad_mods: list[str]` (falling
|
|
||||||
back to the full `__enable_modules__` set for
|
|
||||||
not-yet-split backends) which will be passed to
|
|
||||||
`tractor.ActorNursery.start_actor(enable_modules=)`
|
|
||||||
at actor start time,
|
|
||||||
- deliver a reference to the daemon lifetime fixture,
|
|
||||||
which for now is always the
|
|
||||||
`_setup_persistent_datad()` context defined above.
|
|
||||||
|
|
||||||
'''
|
|
||||||
from piker.brokers import get_brokermod
|
|
||||||
from .validate import get_eps
|
|
||||||
brokermod = get_brokermod(brokername)
|
|
||||||
modpath: str = brokermod.__name__
|
|
||||||
|
|
||||||
# warn (but don't bail) when the backend is missing
|
|
||||||
# some/all of the `datad` ep contract defined by
|
|
||||||
# `piker.data.validate._eps`.
|
|
||||||
datad_eps: dict = get_eps(brokermod, 'datad')
|
|
||||||
if not datad_eps:
|
|
||||||
log.warning(
|
|
||||||
f'Backend {brokername!r} offers NO `datad` '
|
|
||||||
f'(data-feed) eps!?\n'
|
|
||||||
f'Most feed/chart functionality will be '
|
|
||||||
f'broken for this provider..\n'
|
|
||||||
)
|
|
||||||
|
|
||||||
start_actor_kwargs['name'] = f'datad.{brokername}'
|
|
||||||
|
|
||||||
# XXX CRITICAL: include any backend-declared spawn
|
|
||||||
# kwargs, eg. `{'infect_asyncio': True}` required by
|
|
||||||
# `ib`'s embedded `asyncio`-mode `ib_async` usage!
|
|
||||||
start_actor_kwargs.update(
|
|
||||||
getattr(
|
|
||||||
brokermod,
|
|
||||||
'_spawn_kwargs',
|
|
||||||
{},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
# lookup actor-enabled modules declared by the backend
|
|
||||||
# offering the `datad` endpoint(s).
|
|
||||||
enabled: list[str]
|
|
||||||
enabled = start_actor_kwargs['enable_modules'] = [
|
|
||||||
__name__, # so that eps from THIS mod can be invoked
|
|
||||||
modpath,
|
|
||||||
]
|
|
||||||
for submodname in getattr(
|
|
||||||
brokermod,
|
|
||||||
'_datad_mods',
|
|
||||||
# fallback for (flat, less mature) backends which
|
|
||||||
# don't yet declare a daemon-kind mod split.
|
|
||||||
getattr(
|
|
||||||
brokermod,
|
|
||||||
'__enable_modules__',
|
|
||||||
[],
|
|
||||||
),
|
|
||||||
):
|
|
||||||
subpath: str = f'{modpath}.{submodname}'
|
|
||||||
enabled.append(subpath)
|
|
||||||
|
|
||||||
return (
|
|
||||||
brokermod,
|
|
||||||
start_actor_kwargs, # to `ActorNursery.start_actor()`
|
|
||||||
|
|
||||||
# XXX see impl above; contains all (actor global)
|
|
||||||
# setup/teardown expected in all `datad` actor
|
|
||||||
# instances.
|
|
||||||
_setup_persistent_datad,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def spawn_datad(
|
|
||||||
brokername: str,
|
|
||||||
loglevel: str|None = None,
|
|
||||||
|
|
||||||
**tractor_kwargs,
|
|
||||||
|
|
||||||
) -> bool:
|
|
||||||
|
|
||||||
log.info(
|
|
||||||
f'Spawning data-daemon,\n'
|
|
||||||
f'backend: {brokername!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
(
|
|
||||||
brokermod,
|
|
||||||
tractor_kwargs,
|
|
||||||
daemon_fixture_ep,
|
|
||||||
) = datad_init(
|
|
||||||
brokername,
|
|
||||||
loglevel,
|
|
||||||
**tractor_kwargs,
|
|
||||||
)
|
|
||||||
|
|
||||||
# ask `pikerd` to spawn a new sub-actor and manage it
|
|
||||||
# under its actor nursery
|
|
||||||
from piker.service import Services
|
|
||||||
|
|
||||||
dname: str = tractor_kwargs.pop('name') # f'datad.{brokername}'
|
|
||||||
enable_mods: list[str] = list(dict.fromkeys(
|
|
||||||
_datad_service_mods
|
|
||||||
+
|
|
||||||
tractor_kwargs.pop('enable_modules')
|
|
||||||
))
|
|
||||||
portal = await Services.actor_n.start_actor(
|
|
||||||
dname,
|
|
||||||
enable_modules=enable_mods,
|
|
||||||
debug_mode=Services.debug_mode,
|
|
||||||
**tractor_kwargs,
|
|
||||||
)
|
|
||||||
|
|
||||||
# NOTE: the service mngr expects an already spawned
|
|
||||||
# actor + its portal ref in order to do non-blocking
|
|
||||||
# setup of the `datad` service nursery.
|
|
||||||
await Services.start_service_task(
|
|
||||||
dname,
|
|
||||||
portal,
|
|
||||||
|
|
||||||
# signature of target root-task endpoint
|
|
||||||
daemon_fixture_ep,
|
|
||||||
brokername=brokername,
|
|
||||||
loglevel=loglevel,
|
|
||||||
)
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
@acm
|
|
||||||
async def maybe_spawn_datad(
|
|
||||||
brokername: str,
|
|
||||||
loglevel: str|None = None,
|
|
||||||
|
|
||||||
**pikerd_kwargs,
|
|
||||||
|
|
||||||
) -> tractor.Portal:
|
|
||||||
'''
|
|
||||||
Helper to spawn a datad service *from* a client who
|
|
||||||
wishes to use the sub-actor-daemon but is fine with
|
|
||||||
re-using any existing and contactable `datad`.
|
|
||||||
|
|
||||||
Mas o menos, acts as a cached-actor-getter factory.
|
|
||||||
|
|
||||||
'''
|
|
||||||
from piker.service import maybe_spawn_daemon
|
|
||||||
|
|
||||||
async with maybe_spawn_daemon(
|
|
||||||
service_name=f'datad.{brokername}',
|
|
||||||
service_task_target=spawn_datad,
|
|
||||||
spawn_args={
|
|
||||||
'brokername': brokername,
|
|
||||||
},
|
|
||||||
loglevel=loglevel,
|
|
||||||
|
|
||||||
**pikerd_kwargs,
|
|
||||||
|
|
||||||
) as portal:
|
|
||||||
yield portal
|
|
||||||
|
|
@ -775,7 +775,7 @@ async def sample_and_broadcast(
|
||||||
# channel-ipc-pipe is still up.
|
# channel-ipc-pipe is still up.
|
||||||
if (
|
if (
|
||||||
not chan.connected()
|
not chan.connected()
|
||||||
or ctx.cancel_called
|
or ctx._cancel_called
|
||||||
):
|
):
|
||||||
log.warning(
|
log.warning(
|
||||||
'Dropping broken consumer:\n'
|
'Dropping broken consumer:\n'
|
||||||
|
|
|
||||||
|
|
@ -194,24 +194,15 @@ async def _reconnect_forever(
|
||||||
'''
|
'''
|
||||||
# after so many msg recv timeouts, reset the connection
|
# after so many msg recv timeouts, reset the connection
|
||||||
timeouts: int = 0
|
timeouts: int = 0
|
||||||
first_message: bool = True
|
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
waiting_on: str = 'websocket receive'
|
|
||||||
timeout: float = nobsws._timeout
|
|
||||||
with trio.move_on_after(
|
with trio.move_on_after(
|
||||||
# can be dynamically changed by user code
|
# can be dynamically changed by user code
|
||||||
timeout,
|
nobsws._timeout,
|
||||||
) as cs:
|
) as cs:
|
||||||
try:
|
try:
|
||||||
msg: Any = await ws.get_message()
|
msg: Any = await ws.get_message()
|
||||||
waiting_on = 'local relay send'
|
|
||||||
await snd.send(msg)
|
await snd.send(msg)
|
||||||
if first_message:
|
|
||||||
log.info(
|
|
||||||
f'Websocket first message relayed: {url!r}'
|
|
||||||
)
|
|
||||||
first_message = False
|
|
||||||
except nobsws.recon_errors:
|
except nobsws.recon_errors:
|
||||||
log.exception(
|
log.exception(
|
||||||
f'{src_mod}\n'
|
f'{src_mod}\n'
|
||||||
|
|
@ -227,17 +218,10 @@ async def _reconnect_forever(
|
||||||
|
|
||||||
if cs.cancelled_caught:
|
if cs.cancelled_caught:
|
||||||
timeouts += 1
|
timeouts += 1
|
||||||
log.warning(
|
|
||||||
f'Websocket timeout waiting on {waiting_on}\n'
|
|
||||||
f'url: {url!r}\n'
|
|
||||||
f'timeout: {timeout}s; count: {timeouts}; '
|
|
||||||
f'reset after: {reset_after}\n'
|
|
||||||
)
|
|
||||||
if timeouts > reset_after:
|
if timeouts > reset_after:
|
||||||
log.error(
|
log.error(
|
||||||
f'{src_mod}\n'
|
f'{src_mod}\n'
|
||||||
f'WS feed seems down and slow af.. reconnecting\n'
|
'WS feed seems down and slow af.. reconnecting\n'
|
||||||
f'{url!r}\n'
|
|
||||||
)
|
)
|
||||||
rent_cs.cancel()
|
rent_cs.cancel()
|
||||||
|
|
||||||
|
|
@ -277,8 +261,8 @@ async def _reconnect_forever(
|
||||||
ws: WebSocketConnection
|
ws: WebSocketConnection
|
||||||
try:
|
try:
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.collapse_eg(),
|
|
||||||
open_websocket_url(url) as ws,
|
open_websocket_url(url) as ws,
|
||||||
|
tractor.trionics.collapse_eg(),
|
||||||
trio.open_nursery() as tn,
|
trio.open_nursery() as tn,
|
||||||
):
|
):
|
||||||
cs = nobsws._cs = tn.cancel_scope
|
cs = nobsws._cs = tn.cancel_scope
|
||||||
|
|
@ -311,38 +295,24 @@ async def _reconnect_forever(
|
||||||
|
|
||||||
# indicate to wrapper / opener that we are up and block
|
# indicate to wrapper / opener that we are up and block
|
||||||
# to let tasks run **inside** the ws open block above.
|
# to let tasks run **inside** the ws open block above.
|
||||||
log.info(f'Websocket ready: {url!r}')
|
|
||||||
nobsws._connected.set()
|
nobsws._connected.set()
|
||||||
await trio.sleep_forever()
|
await trio.sleep_forever()
|
||||||
|
|
||||||
except nobsws.recon_errors:
|
except (
|
||||||
# The websocket reader can fail while its context exits,
|
HandshakeError,
|
||||||
# outside proxy_msgs() (e.g. Pong during LOCAL_CLOSING).
|
ConnectionRejected,
|
||||||
# Collapse its nursery group before classifying the error.
|
):
|
||||||
nobsws._connected = trio.Event()
|
log.exception('Retrying connection')
|
||||||
nobsws._ws = None
|
|
||||||
# Unlike the old outer wrapper suppression, restart the
|
|
||||||
# connection. This checkpoint also honors parent shutdown;
|
|
||||||
# never shield a reconnect retry from cancellation.
|
|
||||||
log.exception(f'Retrying connection: {url!r}')
|
|
||||||
await trio.sleep(0.5) # throttle
|
await trio.sleep(0.5) # throttle
|
||||||
continue
|
|
||||||
|
|
||||||
# ``trio.Cancelled`` controls task-tree teardown; it is not a
|
except BaseException as _berr:
|
||||||
# reconnect failure. Let it escape before ``BaseException``
|
berr = _berr
|
||||||
# so nursery cancellation never enters the shielded 200ms
|
|
||||||
# error throttle. Audited by
|
|
||||||
# ``test_reconnect_cancellation_bypasses_error_throttle()``.
|
|
||||||
except trio.Cancelled:
|
|
||||||
raise
|
|
||||||
|
|
||||||
except BaseException:
|
|
||||||
log.exception(
|
log.exception(
|
||||||
'Reconnect-attempt failed ??\n'
|
'Reconnect-attempt failed ??\n'
|
||||||
)
|
)
|
||||||
with trio.CancelScope(shield=True):
|
with trio.CancelScope(shield=True):
|
||||||
await trio.sleep(0.2) # throttle
|
await trio.sleep(0.2) # throttle
|
||||||
raise
|
raise berr
|
||||||
|
|
||||||
#|_ws & nursery block ends
|
#|_ws & nursery block ends
|
||||||
nobsws._connected = trio.Event()
|
nobsws._connected = trio.Event()
|
||||||
|
|
@ -353,7 +323,7 @@ async def _reconnect_forever(
|
||||||
# if wrapper cancelled us, we expect it to also
|
# if wrapper cancelled us, we expect it to also
|
||||||
# have re-assigned a new event
|
# have re-assigned a new event
|
||||||
assert (
|
assert (
|
||||||
nobsws._connected is not None
|
nobsws._connected
|
||||||
and not nobsws._connected.is_set()
|
and not nobsws._connected.is_set()
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -372,7 +342,6 @@ async def open_autorecon_ws(
|
||||||
url: str,
|
url: str,
|
||||||
|
|
||||||
fixture: AsyncContextManager | None = None,
|
fixture: AsyncContextManager | None = None,
|
||||||
connect_timeout: float | None = None,
|
|
||||||
|
|
||||||
# time in sec between msgs received before
|
# time in sec between msgs received before
|
||||||
# we presume connection might need a reset.
|
# we presume connection might need a reset.
|
||||||
|
|
@ -422,19 +391,7 @@ async def open_autorecon_ws(
|
||||||
reset_after=reset_after,
|
reset_after=reset_after,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
try:
|
|
||||||
if connect_timeout is None:
|
|
||||||
await nobsws._connected.wait()
|
await nobsws._connected.wait()
|
||||||
else:
|
|
||||||
with trio.fail_after(connect_timeout):
|
|
||||||
await nobsws._connected.wait()
|
|
||||||
|
|
||||||
except trio.TooSlowError as err:
|
|
||||||
raise TimeoutError(
|
|
||||||
f'Websocket connection timed out after '
|
|
||||||
f'{connect_timeout}s: {url}'
|
|
||||||
) from err
|
|
||||||
|
|
||||||
assert nobsws._cs
|
assert nobsws._cs
|
||||||
assert nobsws.connected()
|
assert nobsws.connected()
|
||||||
try:
|
try:
|
||||||
|
|
@ -447,7 +404,6 @@ async def open_autorecon_ws(
|
||||||
f'Entire ws-channel disconnect due to,\n'
|
f'Entire ws-channel disconnect due to,\n'
|
||||||
f'con_err: {con_err!r}\n'
|
f'con_err: {con_err!r}\n'
|
||||||
)
|
)
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,7 @@
|
||||||
'''
|
'''
|
||||||
Data feed apis and infra.
|
Data feed apis and infra.
|
||||||
|
|
||||||
This module is enabled for ``datad`` daemons and includes mostly
|
This module is enabled for ``brokerd`` daemons and includes mostly
|
||||||
endpoints and middleware to support our real-time, provider agnostic,
|
endpoints and middleware to support our real-time, provider agnostic,
|
||||||
live market quotes layer. Historical data loading and processing is also
|
live market quotes layer. Historical data loading and processing is also
|
||||||
initiated in parts of the feed bus startup but business logic and
|
initiated in parts of the feed bus startup but business logic and
|
||||||
|
|
@ -30,7 +30,6 @@ from collections import (
|
||||||
defaultdict,
|
defaultdict,
|
||||||
abc,
|
abc,
|
||||||
)
|
)
|
||||||
from copy import copy
|
|
||||||
from contextlib import asynccontextmanager as acm
|
from contextlib import asynccontextmanager as acm
|
||||||
from functools import partial
|
from functools import partial
|
||||||
import time
|
import time
|
||||||
|
|
@ -55,11 +54,8 @@ from piker.accounting import (
|
||||||
)
|
)
|
||||||
from piker.types import Struct
|
from piker.types import Struct
|
||||||
from piker.brokers import get_brokermod
|
from piker.brokers import get_brokermod
|
||||||
# NOTE: must be a "relative-direct" import (NOT via
|
from piker.service import (
|
||||||
# `piker.service`) to avoid a partial-init cycle when this
|
maybe_spawn_brokerd,
|
||||||
# mod is loaded as part of `piker.service.__init__`.
|
|
||||||
from ._daemon import (
|
|
||||||
maybe_spawn_datad,
|
|
||||||
)
|
)
|
||||||
from piker.calc import humanize
|
from piker.calc import humanize
|
||||||
from ._util import (
|
from ._util import (
|
||||||
|
|
@ -81,7 +77,7 @@ from ._sampling import (
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from .flows import Flume
|
from .flows import Flume
|
||||||
from tractor.discovery._addr import Address
|
from tractor._addr import Address
|
||||||
from tractor.msg.types import Aid
|
from tractor.msg.types import Aid
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -114,7 +110,7 @@ class _FeedsBus(Struct):
|
||||||
'''
|
'''
|
||||||
Data feeds broadcaster and persistence management.
|
Data feeds broadcaster and persistence management.
|
||||||
|
|
||||||
This is a datad side api used to manager persistent real-time
|
This is a brokerd side api used to manager persistent real-time
|
||||||
streams that can be allocated and left alive indefinitely. A bus is
|
streams that can be allocated and left alive indefinitely. A bus is
|
||||||
associated one-to-one with a particular broker backend where the
|
associated one-to-one with a particular broker backend where the
|
||||||
"bus" refers so a multi-symbol bus where quotes are interleaved in
|
"bus" refers so a multi-symbol bus where quotes are interleaved in
|
||||||
|
|
@ -253,7 +249,7 @@ async def allocate_persistent_feed(
|
||||||
'''
|
'''
|
||||||
Create and maintain a "feed bus" which allocates tasks for real-time
|
Create and maintain a "feed bus" which allocates tasks for real-time
|
||||||
streaming and optional historical data storage per broker/data provider
|
streaming and optional historical data storage per broker/data provider
|
||||||
backend; this normally task runs *in* a `datad` actor.
|
backend; this normally task runs *in* a `brokerd` actor.
|
||||||
|
|
||||||
If none exists, this allocates a ``_FeedsBus`` which manages the
|
If none exists, this allocates a ``_FeedsBus`` which manages the
|
||||||
lifetimes of streaming tasks created for each requested symbol.
|
lifetimes of streaming tasks created for each requested symbol.
|
||||||
|
|
@ -322,8 +318,8 @@ async def allocate_persistent_feed(
|
||||||
# at max capacity.
|
# at max capacity.
|
||||||
# - the same ideas ^ but when a local core is maxxed out (like how
|
# - the same ideas ^ but when a local core is maxxed out (like how
|
||||||
# binance does often with hft XD
|
# binance does often with hft XD
|
||||||
# - if a datad is non-local then we can't just allocate a mem
|
# - if a brokerd is non-local then we can't just allocate a mem
|
||||||
# channel here and have the datad write it, we instead need
|
# channel here and have the brokerd write it, we instead need
|
||||||
# a small streaming machine around the remote feed which can then
|
# a small streaming machine around the remote feed which can then
|
||||||
# do the normal work of sampling and writing shm buffers
|
# do the normal work of sampling and writing shm buffers
|
||||||
# (depending on if we want sampling done on the far end or not?)
|
# (depending on if we want sampling done on the far end or not?)
|
||||||
|
|
@ -488,15 +484,14 @@ async def open_feed_bus(
|
||||||
if loglevel is None:
|
if loglevel is None:
|
||||||
loglevel: str = tractor.current_actor().loglevel
|
loglevel: str = tractor.current_actor().loglevel
|
||||||
|
|
||||||
# Configure the shared piker.data logger used by ._util.log,
|
# XXX: required to propagate ``tractor`` loglevel to piker
|
||||||
# including websocket tasks. A per-client verbosity must not
|
# logging
|
||||||
# demote actor-wide transport diagnostics below daemon verbosity.
|
|
||||||
get_console_log(
|
get_console_log(
|
||||||
level=(tractor.current_actor().loglevel
|
level=(loglevel
|
||||||
or
|
or
|
||||||
loglevel
|
tractor.current_actor().loglevel
|
||||||
),
|
),
|
||||||
name=log.name,
|
name=__name__,
|
||||||
)
|
)
|
||||||
|
|
||||||
# local state sanity checks
|
# local state sanity checks
|
||||||
|
|
@ -504,7 +499,7 @@ async def open_feed_bus(
|
||||||
# (after we also group them in a nice `/dev/shm/piker/` subdir).
|
# (after we also group them in a nice `/dev/shm/piker/` subdir).
|
||||||
# ensure we are who we think we are
|
# ensure we are who we think we are
|
||||||
servicename = tractor.current_actor().name
|
servicename = tractor.current_actor().name
|
||||||
assert 'datad' in servicename
|
assert 'brokerd' in servicename
|
||||||
assert brokername in servicename
|
assert brokername in servicename
|
||||||
|
|
||||||
bus: _FeedsBus = get_feed_bus(brokername)
|
bus: _FeedsBus = get_feed_bus(brokername)
|
||||||
|
|
@ -514,12 +509,12 @@ async def open_feed_bus(
|
||||||
for symbol in symbols:
|
for symbol in symbols:
|
||||||
|
|
||||||
# if no cached feed for this symbol has been created for this
|
# if no cached feed for this symbol has been created for this
|
||||||
# datad yet, start persistent stream and shm writer task in
|
# brokerd yet, start persistent stream and shm writer task in
|
||||||
# service nursery
|
# service nursery
|
||||||
flume = bus.feeds.get(symbol)
|
flume = bus.feeds.get(symbol)
|
||||||
if flume is None:
|
if flume is None:
|
||||||
# allocate a new actor-local stream bus which
|
# allocate a new actor-local stream bus which
|
||||||
# will persist for this `datad`'s service lifetime.
|
# will persist for this `brokerd`'s service lifetime.
|
||||||
async with bus.task_lock:
|
async with bus.task_lock:
|
||||||
await bus.nursery.start(
|
await bus.nursery.start(
|
||||||
partial(
|
partial(
|
||||||
|
|
@ -679,36 +674,6 @@ async def open_feed_bus(
|
||||||
bus.remove_subs(bs_fqme, subs)
|
bus.remove_subs(bs_fqme, subs)
|
||||||
|
|
||||||
|
|
||||||
async def relay_to_common_memchan(
|
|
||||||
stream: (
|
|
||||||
tractor.MsgStream
|
|
||||||
|trionics.BroadcastReceiver
|
|
||||||
),
|
|
||||||
tx: trio.MemorySendChannel[dict[str, Any]],
|
|
||||||
task_status: TaskStatus[
|
|
||||||
trionics.BroadcastReceiver,
|
|
||||||
] = trio.TASK_STATUS_IGNORED,
|
|
||||||
|
|
||||||
) -> None:
|
|
||||||
'''
|
|
||||||
Relay one provider subscription into a common memory channel.
|
|
||||||
|
|
||||||
Deliver the live child receiver through `task_status` so the
|
|
||||||
caller can retain and inspect each provider relay's cursor.
|
|
||||||
|
|
||||||
'''
|
|
||||||
bstream: trionics.BroadcastReceiver
|
|
||||||
async with (
|
|
||||||
stream.subscribe() as bstream,
|
|
||||||
tx,
|
|
||||||
):
|
|
||||||
task_status.started(bstream)
|
|
||||||
|
|
||||||
msg: dict[str, Any]
|
|
||||||
async for msg in bstream:
|
|
||||||
await tx.send(msg)
|
|
||||||
|
|
||||||
|
|
||||||
class Feed(Struct):
|
class Feed(Struct):
|
||||||
'''
|
'''
|
||||||
A per-provider API for client-side consumption from real-time data
|
A per-provider API for client-side consumption from real-time data
|
||||||
|
|
@ -756,53 +721,31 @@ class Feed(Struct):
|
||||||
mods = {name: self.mods[name] for name in brokers}
|
mods = {name: self.mods[name] for name in brokers}
|
||||||
|
|
||||||
if len(mods) == 1:
|
if len(mods) == 1:
|
||||||
# just pass the datad stream directly if only one provider
|
# just pass the brokerd stream directly if only one provider
|
||||||
# was detected.
|
# was detected.
|
||||||
stream: (
|
stream = self.streams[list(brokers)[0]]
|
||||||
tractor.MsgStream
|
|
||||||
|trionics.BroadcastReceiver
|
|
||||||
) = self.streams[list(brokers)[0]]
|
|
||||||
bstream: trionics.BroadcastReceiver
|
|
||||||
async with stream.subscribe() as bstream:
|
async with stream.subscribe() as bstream:
|
||||||
yield bstream
|
yield bstream
|
||||||
return
|
return
|
||||||
|
|
||||||
# start multiplexing task tree
|
# start multiplexing task tree
|
||||||
tx: trio.MemorySendChannel[dict[str, Any]]
|
|
||||||
rx: trio.MemoryReceiveChannel[dict[str, Any]]
|
|
||||||
tx, rx = trio.open_memory_channel(616)
|
tx, rx = trio.open_memory_channel(616)
|
||||||
|
|
||||||
|
async def relay_to_common_memchan(stream: tractor.MsgStream):
|
||||||
|
async with tx:
|
||||||
|
async for msg in stream:
|
||||||
|
await tx.send(msg)
|
||||||
|
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.collapse_eg(),
|
tractor.trionics.collapse_eg(),
|
||||||
trio.open_nursery() as nurse
|
trio.open_nursery() as nurse
|
||||||
):
|
):
|
||||||
# spawn a relay task for each stream so that they all
|
# spawn a relay task for each stream so that they all
|
||||||
# multiplex to a common channel.
|
# multiplex to a common channel.
|
||||||
broker2bstreams: dict[
|
for brokername in mods:
|
||||||
str,
|
stream = self.streams[brokername]
|
||||||
trionics.BroadcastReceiver,
|
nurse.start_soon(relay_to_common_memchan, stream)
|
||||||
] = {}
|
|
||||||
broker: str
|
|
||||||
for broker in mods:
|
|
||||||
stream: (
|
|
||||||
tractor.MsgStream
|
|
||||||
|trionics.BroadcastReceiver
|
|
||||||
) = self.streams[broker]
|
|
||||||
|
|
||||||
relay_tx: trio.MemorySendChannel[
|
|
||||||
dict[str, Any]
|
|
||||||
] = tx.clone()
|
|
||||||
|
|
||||||
bstream: trionics.BroadcastReceiver = await nurse.start(
|
|
||||||
relay_to_common_memchan,
|
|
||||||
stream,
|
|
||||||
relay_tx,
|
|
||||||
)
|
|
||||||
broker2bstreams[broker] = bstream
|
|
||||||
|
|
||||||
# Each relay now owns one `tx` clone. Closing the original
|
|
||||||
# lets `rx` reach EOC only after every provider relay exits.
|
|
||||||
await tx.aclose()
|
|
||||||
try:
|
try:
|
||||||
yield rx
|
yield rx
|
||||||
finally:
|
finally:
|
||||||
|
|
@ -811,29 +754,16 @@ class Feed(Struct):
|
||||||
_max_sample_rate: int = 1
|
_max_sample_rate: int = 1
|
||||||
|
|
||||||
async def pause(self) -> None:
|
async def pause(self) -> None:
|
||||||
'''
|
|
||||||
Pause the shared remote provider context for this cached feed.
|
|
||||||
|
|
||||||
This controls provider publication for every local cache user;
|
|
||||||
it does not pause only this caller's receive cursor.
|
|
||||||
|
|
||||||
'''
|
|
||||||
stream: tractor.MsgStream|trionics.BroadcastReceiver
|
|
||||||
for stream in set(self.streams.values()):
|
for stream in set(self.streams.values()):
|
||||||
await stream.send('pause')
|
await stream.send('pause')
|
||||||
|
|
||||||
async def resume(self) -> None:
|
async def resume(self) -> None:
|
||||||
'''
|
|
||||||
Resume the shared remote provider context for this cached feed.
|
|
||||||
|
|
||||||
'''
|
|
||||||
stream: tractor.MsgStream|trionics.BroadcastReceiver
|
|
||||||
for stream in set(self.streams.values()):
|
for stream in set(self.streams.values()):
|
||||||
await stream.send('resume')
|
await stream.send('resume')
|
||||||
|
|
||||||
|
|
||||||
@acm
|
@acm
|
||||||
async def install_datad_search(
|
async def install_brokerd_search(
|
||||||
|
|
||||||
portal: tractor.Portal,
|
portal: tractor.Portal,
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
|
|
@ -882,22 +812,12 @@ async def maybe_open_feed(
|
||||||
ReceiveChannel[dict[str, Any]],
|
ReceiveChannel[dict[str, Any]],
|
||||||
):
|
):
|
||||||
'''
|
'''
|
||||||
Maybe open a data feed to a ``datad`` daemon only if there is no
|
Maybe open a data to a ``brokerd`` daemon only if there is no
|
||||||
local one for the broker-symbol pair, if one is cached use it wrapped
|
local one for the broker-symbol pair, if one is cached use it wrapped
|
||||||
in a tractor broadcast receiver.
|
in a tractor broadcast receiver.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
cache_key: tuple[
|
fqme = fqmes[0]
|
||||||
tuple[str, ...],
|
|
||||||
float|None,
|
|
||||||
bool,
|
|
||||||
bool,
|
|
||||||
] = (
|
|
||||||
tuple(sorted(fqmes)),
|
|
||||||
kwargs.get('tick_throttle'),
|
|
||||||
kwargs.get('allow_overruns', True),
|
|
||||||
kwargs.get('start_stream', True),
|
|
||||||
)
|
|
||||||
|
|
||||||
async with trionics.maybe_open_context(
|
async with trionics.maybe_open_context(
|
||||||
acm_func=open_feed,
|
acm_func=open_feed,
|
||||||
|
|
@ -910,73 +830,27 @@ async def maybe_open_feed(
|
||||||
'allow_overruns': kwargs.get('allow_overruns', True),
|
'allow_overruns': kwargs.get('allow_overruns', True),
|
||||||
'start_stream': kwargs.get('start_stream', True),
|
'start_stream': kwargs.get('start_stream', True),
|
||||||
},
|
},
|
||||||
key=cache_key,
|
key=fqme,
|
||||||
|
|
||||||
) as (cache_hit, feed):
|
) as (cache_hit, feed):
|
||||||
feed: Feed
|
|
||||||
|
|
||||||
if (
|
if cache_hit:
|
||||||
cache_hit
|
log.info(f'Using cached feed for {fqme}')
|
||||||
and
|
# add a new broadcast subscription for the quote stream
|
||||||
log.at_least_level('info')
|
# if this feed is likely already in use
|
||||||
):
|
|
||||||
log.info(
|
|
||||||
f'Using cached feed for key:\n'
|
|
||||||
f'{cache_key!r}'
|
|
||||||
)
|
|
||||||
|
|
||||||
# The cached `Feed` exclusively owns provider root streams.
|
|
||||||
# Every caller, including the first cache owner, gets a local
|
|
||||||
# child cursor and copied descriptors for its lexical lifetime.
|
|
||||||
broker_names: tuple[str, ...] = tuple(feed.streams)
|
|
||||||
bstreams: tuple[
|
|
||||||
trionics.BroadcastReceiver,
|
|
||||||
...,
|
|
||||||
]
|
|
||||||
async with trionics.gather_contexts(
|
async with trionics.gather_contexts(
|
||||||
mngrs=[
|
mngrs=[stream.subscribe() for stream in feed.streams.values()]
|
||||||
feed.streams[broker].subscribe()
|
|
||||||
for broker in broker_names
|
|
||||||
]
|
|
||||||
) as bstreams:
|
) as bstreams:
|
||||||
broker2bstreams: dict[
|
for bstream, flume in zip(bstreams, feed.flumes.values()):
|
||||||
str,
|
|
||||||
trionics.BroadcastReceiver,
|
|
||||||
] = dict(zip(
|
|
||||||
broker_names,
|
|
||||||
bstreams,
|
|
||||||
strict=True,
|
|
||||||
))
|
|
||||||
flumes: dict[str, Flume] = {}
|
|
||||||
fqme: str
|
|
||||||
flume: Flume
|
|
||||||
for fqme, flume in feed.flumes.items():
|
|
||||||
broker: str = flume.mkt.broker
|
|
||||||
|
|
||||||
bstream: trionics.BroadcastReceiver = (
|
|
||||||
broker2bstreams[broker]
|
|
||||||
)
|
|
||||||
|
|
||||||
# XXX: TODO: horrible hackery that needs fixing..
|
# XXX: TODO: horrible hackery that needs fixing..
|
||||||
# i guess we have to create context proxies?
|
# i guess we have to create context proxies?
|
||||||
root_stream: tractor.MsgStream = feed.streams[broker]
|
bstream._ctx = flume.stream._ctx
|
||||||
|
flume.stream = bstream
|
||||||
|
|
||||||
ctx: tractor.Context = root_stream.ctx
|
yield feed
|
||||||
bstream._ctx = ctx
|
else:
|
||||||
|
yield feed
|
||||||
local_flume: Flume = copy(flume)
|
|
||||||
local_flume.stream = bstream
|
|
||||||
flumes[fqme] = local_flume
|
|
||||||
|
|
||||||
# `feed` is the cache-owned resource and must retain only
|
|
||||||
# provider root streams. A shallow `Feed` copy gives this
|
|
||||||
# caller its own descriptor mappings while intentionally
|
|
||||||
# sharing immutable market metadata, SHM handles and
|
|
||||||
# provider controls for the cache resource's lifetime.
|
|
||||||
local_feed: Feed = copy(feed)
|
|
||||||
local_feed.flumes = flumes
|
|
||||||
local_feed.streams = broker2bstreams
|
|
||||||
yield local_feed
|
|
||||||
|
|
||||||
|
|
||||||
@acm
|
@acm
|
||||||
|
|
@ -1011,14 +885,14 @@ async def open_feed(
|
||||||
providers.setdefault(mod, []).append(bs_fqme)
|
providers.setdefault(mod, []).append(bs_fqme)
|
||||||
feed.mods[mod.name] = mod
|
feed.mods[mod.name] = mod
|
||||||
|
|
||||||
# one actor per datad for now
|
# one actor per brokerd for now
|
||||||
datad_ctxs = []
|
brokerd_ctxs = []
|
||||||
for brokermod, bfqmes in providers.items():
|
for brokermod, bfqmes in providers.items():
|
||||||
|
|
||||||
# if no `datad` for this backend exists yet we spawn
|
# if no `brokerd` for this backend exists yet we spawn
|
||||||
# a daemon actor for it.
|
# a daemon actor for it.
|
||||||
datad_ctxs.append(
|
brokerd_ctxs.append(
|
||||||
maybe_spawn_datad(
|
maybe_spawn_brokerd(
|
||||||
brokermod.name,
|
brokermod.name,
|
||||||
loglevel=loglevel
|
loglevel=loglevel
|
||||||
)
|
)
|
||||||
|
|
@ -1026,7 +900,7 @@ async def open_feed(
|
||||||
|
|
||||||
portals: tuple[tractor.Portal]
|
portals: tuple[tractor.Portal]
|
||||||
async with trionics.gather_contexts(
|
async with trionics.gather_contexts(
|
||||||
datad_ctxs,
|
brokerd_ctxs,
|
||||||
) as portals:
|
) as portals:
|
||||||
|
|
||||||
bus_ctxs: list[AsyncContextManager] = []
|
bus_ctxs: list[AsyncContextManager] = []
|
||||||
|
|
@ -1063,9 +937,9 @@ async def open_feed(
|
||||||
tick_throttle=tick_throttle,
|
tick_throttle=tick_throttle,
|
||||||
|
|
||||||
# XXX: super important to avoid
|
# XXX: super important to avoid
|
||||||
# the datad from some other
|
# the brokerd from some other
|
||||||
# backend overruning the task here
|
# backend overruning the task here
|
||||||
# bc some other datad took longer
|
# bc some other brokerd took longer
|
||||||
# to startup before we hit the `.open_stream()`
|
# to startup before we hit the `.open_stream()`
|
||||||
# loop below XD .. really we should try to do each
|
# loop below XD .. really we should try to do each
|
||||||
# of these stream open sequences sequentially per
|
# of these stream open sequences sequentially per
|
||||||
|
|
@ -1137,7 +1011,7 @@ async def open_feed(
|
||||||
assert stream
|
assert stream
|
||||||
feed.streams[brokermod.name] = stream
|
feed.streams[brokermod.name] = stream
|
||||||
|
|
||||||
# apply `datad`-common stream to each flume
|
# apply `brokerd`-common stream to each flume
|
||||||
# tracking a live market feed from that provider.
|
# tracking a live market feed from that provider.
|
||||||
for fqme, flume in feed.flumes.items():
|
for fqme, flume in feed.flumes.items():
|
||||||
if brokermod.name == flume.mkt.broker:
|
if brokermod.name == flume.mkt.broker:
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,6 @@ from piker.accounting import (
|
||||||
Asset,
|
Asset,
|
||||||
MktPair,
|
MktPair,
|
||||||
)
|
)
|
||||||
from tractor._exceptions import reg_err_types
|
|
||||||
from ._util import log
|
from ._util import log
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -44,8 +43,6 @@ class FeedInitializationError(ValueError):
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
|
||||||
reg_err_types([FeedInitializationError])
|
|
||||||
|
|
||||||
|
|
||||||
class FeedInit(Struct, frozen=True):
|
class FeedInit(Struct, frozen=True):
|
||||||
'''
|
'''
|
||||||
|
|
@ -91,24 +88,6 @@ _eps: dict[str, list[str]] = {
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def get_eps(
|
|
||||||
mod: ModuleType,
|
|
||||||
kind: str, # 'middleware' | 'datad' | 'brokerd'
|
|
||||||
|
|
||||||
) -> dict[str, Callable]:
|
|
||||||
'''
|
|
||||||
Return the daemon-kind's ep funcs defined by the backend
|
|
||||||
`mod`, keyed by ep name; any eps from `_eps[kind]` not
|
|
||||||
implemented by the backend are excluded.
|
|
||||||
|
|
||||||
'''
|
|
||||||
return {
|
|
||||||
name: ep
|
|
||||||
for name in _eps[kind]
|
|
||||||
if (ep := getattr(mod, name, None))
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def validate_backend(
|
def validate_backend(
|
||||||
mod: ModuleType,
|
mod: ModuleType,
|
||||||
syms: list[str],
|
syms: list[str],
|
||||||
|
|
|
||||||
33
piker/log.py
33
piker/log.py
|
|
@ -36,15 +36,6 @@ from pygments import (
|
||||||
_proj_name: str = 'piker'
|
_proj_name: str = 'piker'
|
||||||
|
|
||||||
|
|
||||||
def _relative_name(
|
|
||||||
name: str|None,
|
|
||||||
) -> str|None:
|
|
||||||
if name:
|
|
||||||
return name.removeprefix(f'{_proj_name}.')
|
|
||||||
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def get_logger(
|
def get_logger(
|
||||||
name: str|None = None,
|
name: str|None = None,
|
||||||
**tractor_log_kwargs,
|
**tractor_log_kwargs,
|
||||||
|
|
@ -56,9 +47,17 @@ def get_logger(
|
||||||
See `tractor.log.get_logger()` for details.
|
See `tractor.log.get_logger()` for details.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
pkg_name: str = _proj_name
|
||||||
|
if (
|
||||||
|
name
|
||||||
|
and
|
||||||
|
pkg_name in name
|
||||||
|
):
|
||||||
|
name: str = name.lstrip(f'{_proj_name}.')
|
||||||
|
|
||||||
return tractor.log.get_logger(
|
return tractor.log.get_logger(
|
||||||
name=_relative_name(name),
|
name=name,
|
||||||
pkg_name=_proj_name,
|
pkg_name=pkg_name,
|
||||||
**tractor_log_kwargs,
|
**tractor_log_kwargs,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -82,6 +81,14 @@ def get_console_log(
|
||||||
You do it.. Bp
|
You do it.. Bp
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
pkg_name: str = _proj_name
|
||||||
|
if (
|
||||||
|
name
|
||||||
|
and
|
||||||
|
pkg_name in name
|
||||||
|
):
|
||||||
|
name: str = name.lstrip(f'{_proj_name}.')
|
||||||
|
|
||||||
tll: str|None = None
|
tll: str|None = None
|
||||||
if (
|
if (
|
||||||
with_tractor_log is not False
|
with_tractor_log is not False
|
||||||
|
|
@ -104,8 +111,8 @@ def get_console_log(
|
||||||
|
|
||||||
return tractor.log.get_console_log(
|
return tractor.log.get_console_log(
|
||||||
level=level,
|
level=level,
|
||||||
name=_relative_name(name),
|
name=name,
|
||||||
pkg_name=_proj_name,
|
pkg_name=pkg_name,
|
||||||
**tractor_log_kwargs,
|
**tractor_log_kwargs,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -56,7 +56,3 @@ from ..brokers._daemon import (
|
||||||
spawn_brokerd as spawn_brokerd,
|
spawn_brokerd as spawn_brokerd,
|
||||||
maybe_spawn_brokerd as maybe_spawn_brokerd,
|
maybe_spawn_brokerd as maybe_spawn_brokerd,
|
||||||
)
|
)
|
||||||
from ..data._daemon import (
|
|
||||||
spawn_datad as spawn_datad,
|
|
||||||
maybe_spawn_datad as maybe_spawn_datad,
|
|
||||||
)
|
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue