Compare commits

..

No commits in common. "wkt/land_backfill_history" and "main" have entirely different histories.

150 changed files with 1386 additions and 10417 deletions

View File

@ -1,143 +0,0 @@
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
## Context
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
**User-decided constraints:**
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
## Target topology
```
pikerd
├── datad.ib feed bus, shm writers, tsp history, symbol search
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
├── emsd
│ └── paperboi.ib (paper mode; opens its feed via datad)
└── samplerd
```
## Key verified facts (load-bearing)
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
## Module decision
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
---
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
- `ib`: adjust existing groups so each includes `'api'`
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
**Gate**: full pytest green, zero behavior change.
## Stage 1 — introduce `datad` daemon machinery (additive)
New `piker/data/_daemon.py`:
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
Supporting edits:
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
**Gate**: suite green + new spawn test; nothing routes through datad yet.
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
`piker/clearing/_ems.py`:
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
## Stage 3 — feed layer hard cutover to datad
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
- `piker/brokers/core.py:33,175` (`symbol_search`)
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
## Stage 4 — caps-sec slimming, validation, ib client-id
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
---
## Risk register
| Risk | Sev | Mitigation |
|---|---|---|
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
## Verification (per stage gates above, plus end-to-end)
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
## Critical files
- **new** `piker/data/_daemon.py` — all datad machinery
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
- `piker/data/feed.py` — spawn cutover + actor-name assert
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
- `piker/data/validate.py` — `get_eps()` helper
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
- `piker/accounting/cli.py` — keyword-form dialog call
- `tests/test_services.py` — inverted + new assertions

View File

@ -1,233 +0,0 @@
# Bounded IB recovery candidate
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
`/home/goodboy/repos/piker`. Base:
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
Changes are unstaged and uncommitted. No acceptance is implied.
## Patch selection and provenance
Keep these existing landing commits:
| Commit | Reason |
| --- | --- |
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
| `e8e0730a` | Correlate MethodProxy requests and responses |
| `7bef50b0` | One response reader, per-request waiters |
| `d1571ddf` | Propagate daemon-handler exceptions |
| `dc618229` | Isolate cached feed stream consumers |
Proposed separate implementation boundaries, in this order:
1. Back out the code and original test from
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
This restores the direct-owner `_quote_streams` implementation.
Retain the historical prompt records. Replace the removed
mocked-cache regression with the real-bridge restart reproducer.
This backout is not a claim that arbitrary concurrent direct
callers have safe broadcast ownership.
2. Backport **only** `piker/brokers/ib/feed.py` from
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
Retain its seven history-adapter regressions in
`tests/test_ib_history.py`. This includes reset serialization,
completion/result signalling and bounded cancellation/failure.
It does not include the original SHM/storage orchestration patch.
3. Backport **only** `piker/brokers/ib/feed.py` from
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
Retain `test_empty_history_preserves_request_boundary` and its
six cases. This preserves no-data request boundaries and wakes
the waiter. It does not include the original history-layer
endpoint/filtering changes. Add the real-bridge history recovery
case after these adapter changes are present.
4. Import the code and original regression from
`12b0d66e854247e92cb5a8668be71d816c72d856`.
Add a separate real-broadcaster/production-loop regression.
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
This is a local UX cleanup retaining the existing `None` result.
It does not backport the null-grouping/deduplication rewrite in
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
6. Record cross-boundary regression results and live qualification
evidence, distinguishing the known callback failure from passes.
All three imported source commits were authored by
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
and the full source SHA when committing each partial backport.
Use explicit partial-backport wording, not a whole-commit cherry
claim. New real-runtime tests are this session's additional work.
The current aggregate working diff contains these boundaries in
the same feed file. Staging that entire file would combine the
first three boundaries. Reconstruct/stage the original per-source
patches in the above order when preparing commits.
## Quote-cache history correction
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
They are equivalent replays, not different implementations.
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
change and retains direct ownership. The original qualification
did not validate the combination assembled for the landing.
After this candidate's edits, `piker/brokers/ib/feed.py` matches
the source at `1da9b760` byte-for-byte. Compared with the evaluated
tip, only the later schedule/reservation APIs are absent.
The EMS code patch has the same stable patch ID as the original
`12b0d66e` EMS code patch:
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
## Real-runtime evidence
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
an asyncio-infected child, the installed pinned Tractor bridge,
production Piker quote setup, and real IB/eventkit ticker events.
Only the external IB connection/input is replaced with replay data.
Before the quote-cache backout, cancellation after ticker delivery
raised `RemoteActorError[TrioTaskExited]` through
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
reported fatal path.
After the backout:
- Two quote generations finish; each subscription is cancelled,
each asyncio producer is done and each ticker handler removed.
- A second RPC repeats the sequence in the same actor successfully.
- The strict callback-cleanliness case still fails: Tractor's
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
exception handler even though the Trio consumer recovers.
The history integration case keeps `Client.bars()`, MethodProxy,
both relays, `get_bars()`, `wait_on_data_reset()` and
`data_reset_hack()` real. It replaces external contract discovery,
IB history replies and VNC input. A real IB error event announces
farm recovery. The replay rejects retries before recovery and
proves one reset followed by completion for both 1s and 60s data.
The new EMS case runs the production dark-clearing loop over a
real Tractor broadcaster: both EMS and a sibling receive a tick,
EMS cancellation removes only its subscription, and the sibling
continues with an unchanged strict lag policy.
## Verification
Unique selected cases: **38 passed, 1 failed**.
The failure is intentionally visible, not skipped or xfailed.
Run from the checkout's working Nix environment:
```xsh
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
git diff --check
```
The adapter/proxy/feed/EMS selection initially passed 21 cases;
the added real EMS case then passed with its imported case.
The bridge file passed two cases and failed callback cleanliness.
Logging/startup passed 13 cases; runtime boot passed one with the
existing `Registrar.uid` deprecation warning.
The agent shell inherited another project's environment and lacked
native libraries. Tests used the existing Piker `py313` via
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
libraries from the pinned flake's existing Qt dependency closure,
excluding Nix Qt itself and glibc. No environment sync or shellHook
ran. Piker resolved to this checkout; Tractor resolved to its
installed package in `py313`.
Workflow-state preparation initially reported changed legacy
files/modes after migration. On 2026-10-23 the human authorized
reconciliation. Inspection found only two missing legacy guidance
snapshots; the neutral copies matched their saved hashes and modes.
Restoring those local snapshots resolved preparation without
changing archived messages or the original migration receipt.
## Human live qualification, 2026-10-23
The human ran these journeys with the installed Tractor pin,
the existing IB Gateway and pre-existing NativeDB. The agent did
not operate the broker connection, VNC, orders or database.
- Initial QQQ startup/backfill recovered after pacing error 162
with one reset attempt, then displayed live updates. The reset
waiter appears to have timed out; the log did not establish an
acknowledged farm reset or live quote-stream restart.
- Sequential chart exits and reattachment preserved all four
daemon PIDs; a read-only process/listener check confirmed this.
- The overlapping two-chart journey used `pikerd@961108`,
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
then C closed at 08:18:37. The human reported the exercise worked;
survivor updates are a manual observation, while timestamps and
unchanged daemon identities corroborate overlap and service reuse.
- Startup in that journey hit the explicit single-null-sample
breakpoint several times before the charts overlapped. The small
debugger cleanup above was applied afterward; this live run does
not itself verify that later edit.
- Departing charts still report cancellation groups and an FSP
receiver-close `ClosedResourceError`, followed by the Qt close
handler's self-SIGINT. Peer-disconnected final result warnings
remain contained in the persistent services.
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
After the human continued, all three children were reported
exited with status 0 and the root terminated at 08:19:45.
Cancellation-only RPC error reporting and late cancel-ack
diagnostics also remain.
The human explicitly accepted deferring this daemon teardown fix
from #94 and requested loud attribution for the bounded backports.
This is an operational smoke result, not clean-shutdown coverage.
## Downstream reconciliation requirements
Before landing #94, the human wants an easy replay of the downstream
histories intended to follow it. Rehearse against the agreed #94
tree in disposable worktrees, then inspect range-diffs, final code
and relevant regressions. This note does not authorize rewriting
the original branches or changing acceptance markers.
- `0846cbd4` and `1da9b760` are partial backports. Their imported
adapter hunks may already be present, but their history/SHM/storage
changes and omitted integration tests remain necessary downstream.
Never drop either whole source commit merely because `feed.py`
matches or its adapter hunk becomes empty.
- `12b0d66e` has source-equivalent EMS code and its imported test,
plus an additional local real-broadcaster test. Preserve that extra
regression when resolving test-file overlap.
- The quote-cache backout intentionally restores direct ownership
for this landing. Reintroducing original `4af885d9` downstream
requires qualifying cached-owner teardown with the matching
Tractor revision; the backout is not a permanent ownership design.
- The local single-null breakpoint removal retains legacy behavior.
Prefer the full downstream `976db6c3` algorithm when it is replayed;
preserving this `return None` guard would lose that later fix.
- Real-bridge tests and manual qualification notes are local work,
not evidence that omitted portions of the source commits landed.
Commit messages must name full source SHAs, original authorship,
imported paths/tests, omissions, local additions and replay guidance.
Use explicit partial-backport language for adapter subsets; do not
claim a full cherry-pick or reuse the source's broader fix claims.
## Remaining decision
This candidate removes the reproduced fatal quote-restart
propagation and imports existing reset fixes. The human exercised
live QQQ startup, backfill and concurrent chart consumer lifetimes.
Completion-callback and teardown errors remain; the human accepts
them as follow-ups outside the bounded #94 landing.
The storage/SHM portions of the downstream fixes are not covered by
the adapter backport or its integration tests.
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
`1a2d4893` stays a separate follow-up. Further Tractor teardown
changes or storage imports require reassessing the bounded landing
rather than silently expanding this candidate.

View File

@ -1,71 +0,0 @@
# IB recovery qualification, 2026-11-06
This records human-operated qualification of the bounded #94
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
the narrow history catch described below. It supplements the
historical evidence in `ai/ib-recovery-candidate.md`.
## Runtime and dependency source
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
local checkout during evaluation. The human subsequently published
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
remote query confirmed the published OID before restoring Piker's
Git source and recording the same OID in `uv.lock`.
Offline lock validation resolves 126 packages successfully. No
environment sync was performed. This validates the lock metadata;
it does not claim a fresh installation from the published source.
## Manual observations
The human operated Gateway, chart windows and paper orders. The
agent did not operate live services or the account.
- Same-feed overlap, close/reopen and paper order submit/cancel
worked after the existing order-dialog guard was imported.
- Different-feed overlap originally exposed a timed-out one-second
MNQ March 2027 gap query. PDB showed `timedout=True` and
`out is None`. Its uncaught `NoData` canceled sibling history and
quote tasks even though `pikerd` remained alive.
- After the narrow catch, the human reported the overlap journeys
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
live feeds survived the Gateway data reset; closing all charts
and reopening with `pikerd` left running restored live updates.
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
that the intended Gateway data reset was triggered.
These are manual observations, not automated timing or latency
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
labels it a connection reset.
## Narrow history backport
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
`maybe_fill_null_segments()` is copied from
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
remains in place; this does not import broader history repair.
The new regression uses real null detection and checks that a
sibling in the same nursery still runs after either provider
exception. Both cases failed before the catch. With it, the two
focused history files passed: 17 tests. Earlier bounded-recovery
and Tractor qualification results remain separate evidence, not
fresh test runs of the final dependency commit tree.
## Remaining recovery limits
A full Gateway restart was triggered accidentally through its
auto-restart settings dialog. The disconnect delivered the string
`Peer closed connection.` to `deliver_trade_events()`. The string
branch then reached `err['reqid']`, raising `TypeError`; its `code`
local can also retain an earlier event's value. Full Gateway restart
recovery is outside this bounded landing.
The earlier history-failure teardown also exposed a separate
Tractor completion-callback assertion after `BrokenResourceError`.
The history catch does not establish that this callback race is
fixed. Broader history recovery and these disconnect paths remain
downstream work.

View File

@ -1,52 +0,0 @@
---
created_at: 2026-10-07T16:54:53Z
generated_by:
harness: opencode
provider: openai
model: gpt-6.1-sol
git_ref: bde2457b
scope: code
substantive: true
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
---
## Prompt
The user requested fixing all seven local PR94 review findings and
exporting the review for human inspection in Tuicr. They asked that a
Tractor pin adjustment be considered before adding Piker workarounds.
## Response summary
Correct the endpoint defaults, Click argument shape, daemon binding and
registrar option handling, chart fallback, and stray debugger pause.
Keep the absent datad implementation's test outside this landing
branch. Add offline startup coverage and export original findings
without modifying review states. The checked newer Tractor address
branch still rejects mixed transports, so leave the dependency pin.
## Files changed
- `config/conf.toml`: TCP-only default endpoints.
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
- `piker/service/_actor_runtime.py`: validate daemon listeners.
- `piker/service/_daemon.py`: initialize and consume registrar options.
- `piker/ui/cli.py`: fallback to automatic chart binding.
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
- `tests/test_services.py`: defer test requiring missing datad API.
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
## Human edits
The human identified that datad tests arrived through the agent's
mixed cherry-picks and explicitly requested deferring that test. They
requested a chart default for existing configs, expanded the task to
all review findings, and directed checking whether a newer Tractor pin
could reduce the fix scope. No direct manual source edits were observed
during these changes; acceptance remains with the human.
During the human review pass, the human confirmed removing the Binance
pause and correcting CLI endpoint parsing. They requested an explicit
template note about this pin's single-transport limitation and detailed
comments distinguishing registrar discovery endpoints from daemon bind
endpoints. Those explanations were added at the human's direction.

View File

@ -1,61 +0,0 @@
---
created_at: 2026-10-07T16:54:53Z
generated_by:
harness: opencode
provider: openai
model: gpt-6.1-sol
git_ref: bde2457b
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
Fix all findings from the local PR94 review, prefer a Tractor pin
change if it reduces work, and export the original review for Tuicr.
## Generated changes
> `git diff HEAD~1..HEAD -- config/conf.toml`
Use TCP-only defaults, retaining commented UDS alternatives. Both
checked Tractor address branch tips still reject mixed transports.
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
Make Click multiaddresses repeatable and normalize single strings.
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
Validate requested daemon bind addresses instead of remote registrars.
Normalize accepted endpoints and allow TCP ephemeral port requests.
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
Resolve explicit or inherited registrars independently of other daemon
options and consume the explicit keyword before forwarding kwargs.
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
Default chart bind endpoints to None when network config is absent.
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Remove the unconditional market-info debugger pause.
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
Remove the prematurely cherry-picked datad test and add offline startup
regressions, including a real registrar/child daemon lifecycle check.
## Verification
- Startup and logging tests: 13 passed.
- Existing runtime boot and tagged registrar tests: 2 passed.
- Scoped native Ruff and diff checks pass.
- The first new actor test used a removed nursery API; it was corrected
to the installed start_actor/Portal.run interface before passing.
- Import initially failed without native libraries. Tests use the
existing flake's GCC, zlib, Qt and PyQt paths without executing its
environment-rebuilding shell hook.
- Original seven findings exported locally as Tuicr review-add payloads
bound to bde2457b; no session comments or remote PR were changed.

View File

@ -1,45 +0,0 @@
---
created_at: 2026-10-10T23:28:52Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 1fa45dfa
scope: config
substantive: true
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
---
## Prompt
The human rejected keeping the April Tractor pin as a landing baseline.
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
new rollback adaptations, reuse of original Piker histories where
possible, and a candidate prepared within their gateway relocation
window. They asked specifically whether the runtime validation review
hunk duplicated older endpoint history.
## Response summary
Select the reviewed address branch at an immutable locked revision and
preserve all committed application/test compatibility code. Verify the
minimal dependency boundary offline and preview existing consumer
fixes without importing unrelated datad/backfill changes. Distinguish
original-source test results from live teardown acceptance, which is
not available in this preparation window.
## Files changed
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
preserving Polars 1.36.0 and the rest of the locked graph.
## Human edits
The human redirected the task from an older known-good runtime to a
modern landing stack and insisted on reusing original history rather
than building new compatibility adapters. They proposed stricter
exception-group behavior as a hypothesis and set the live-test/time
constraints. Those directions materially changed the dependency
boundary. No direct manual edits to this final boundary were observed;
runtime acceptance and landing remain human-owned.

View File

@ -1,52 +0,0 @@
---
created_at: 2026-10-10T23:28:52Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 1fa45dfa
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
Undo the April Tractor rollback and select the reviewed
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
commits to new adaptations, and prepare an offline-verified candidate
while live IB verification is deferred for gateway relocation.
## Generated changes
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
and test sources byte-for-byte to the committed PR94 tree. Keep the
rejected rollback journal only in ignored local experiment reports.
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
the resulting graph. Actual imports confirm the reviewed Tractor pin
and Trio 0.32.0, not the hypothesized Trio 0.33.
## Original-history previews
The existing MethodProxy stack is retained. Read-only merge previews
show original `4af885d9` and `b691422b` can be applied without the datad
cutover. Original-source regressions run through temporary in-memory
overlays against the real pinned runtime: seven cases pass and one
cache-hit case fails because `log.at_least_level('info')` looks up INFO
in a custom-level mapping. A separate standard logging correction
produces eight passing cases. Its patch and execution handoff are saved
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
cherry-picks or source edits are performed by this boundary.
## Verification
- Startup and logging: 13 passed.
- Runtime boot and tagged registrar normalization: 2 passed.
- Locked sync and source whitespace checks pass.
- Original application/test trees are unchanged from PR94 HEAD.
- Consumer overlay with explicit logging correction: 8 passed.
- Correction patch validates against its exact preview tree using a
private temporary index; the real index is untouched.
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.

View File

@ -1,45 +0,0 @@
---
created_at: 2026-11-06T18:56:28.621577+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
scope: code
substantive: true
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
---
## Prompt
User reported a different-feed overlap failure and asked whether
a fix could be extracted without importing broader history work.
PDB confirmed MNQ March 2027 one-second history timed out and
returned None, producing NoData. After continuing PDB, sibling
history and quote tasks terminated while pikerd remained alive.
User requested continuing with the narrow backport.
## Response summary
Partial backport of the NoData/DataUnavailable catch from
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
maybe_fill_null_segments(). The existing gap fallback remains in use.
A new parametrized regression uses real null detection and verifies
that a sibling in the same nursery survives the failed repair.
Both new cases failed before the catch. The focused history suites
passed after it: 17 passed. This does not qualify live overlap or fix
IB reset churn or the separate Tractor completion-callback assertion.
## Files changed
- `piker/tsp/_history.py`: existing downstream exception catch.
- `tests/test_history_backfill.py`: sibling survival regression.
## Human edits
User supplied live reproduction, PDB locals and cancellation logs,
and explicitly constrained scope to a small history extraction.
The production catch is copied from existing downstream commit
0846cbd4, not newly designed here. New regression is agent-authored;
no direct human source edits observed during this patch. Review and
live qualification remain with the user.

View File

@ -1,22 +0,0 @@
---
created_at: 2026-11-06T18:56:28.621577+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Partial backport of the NoData/DataUnavailable catch from
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
maybe_fill_null_segments(). The existing gap fallback remains in use.
A new parametrized regression uses real null detection and verifies
that a sibling in the same nursery survives the failed repair.
Both new cases failed before the catch. The focused history suites
passed after it: 17 passed. This does not qualify live overlap or fix
IB reset churn or the separate Tractor completion-callback assertion.

View File

@ -1,41 +0,0 @@
---
created_at: 2026-11-06T20:17:46.622316+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
scope: config
substantive: true
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
---
## Prompt
The human requested a commit plan, then a fast-forward landing of
the #94 branch and PR updates. They agreed the temporary local
Tractor source must be replaced, published piker_pin themselves,
and reported "pushed it". They also supplied the manual recovery
results and full Gateway restart failure diagnostics.
## Response summary
Restore the Git Tractor source after local qualification and lock
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
Record the human's overlap, data-reset and reopen observations,
the narrow history extraction, and full-restart recovery limits.
Offline uv lock validation passed without an environment sync.
## Files changed
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
## Human edits
The human selected the bounded landing scope, performed live tests,
identified the more aggressive reset distinction, and published
the tested Tractor ref. The agent applied the metadata edits and
wrote the qualification record from those observations. No human
acceptance or task-state transitions are inferred by this record.

View File

@ -1,21 +0,0 @@
---
created_at: 2026-11-06T20:17:46.622316+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- pyproject.toml`
> `git diff HEAD~1..HEAD -- uv.lock`
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
Restore the Git Tractor source after local qualification and lock
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
Record the human's overlap, data-reset and reopen observations,
the narrow history extraction, and full-restart recovery limits.
Offline uv lock validation passed without an environment sync.

View File

@ -1,11 +0,0 @@
# AI Prompt I/O Log
This directory records substantive AI-assisted changes, prompts,
original harness identity, and unedited outputs. Human contributors
remain accountable for code decisions.
Records follow the [NLNet generative AI policy][policy]. Each
structured entry links its immutable `.raw.md` partner. Code output
uses Git diff references to avoid duplication.
[policy]: https://nlnet.nl/foundation/policies/generativeAI/

View File

@ -1,59 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:08:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
---
## Prompt
Session-initiating instruction (driving all 7 commits in
this series):
> ok i want you to become the distributed runtime and
> concurrency expert for this project - namely acquire
> a deep understanding of tractor and how it's used.
> then i want you to attempt to factor our current
> brokerd service daemon into 2 daemons: a brokerd
> which only hosts live/paper trading endpoint tasks
> [..] a new `datad` subdaemon which in a separate
> subactor serves all the data feed service tasks [..]
> give me a mega detailed plan on how to approach this,
> and a staged approach for the implementation.
Proximate driver for THIS commit: during the approved
plan's stage-0 test gate the agent discovered test
subactors were writing into the user's REAL
`~/.config/piker/accounting/` files (a bogus paper fill
landed in `account.kraken.paper.toml`) because the old
test-dir override in `config.get_app_dir()` was
commented out and could never work in spawned
subactors anyway. Fix applied autonomously under the
approved plan; no explicit per-fix user prompt.
## Response summary
Restore `pytest` config-dir isolation by resolving the
per-test tmp dir lazily (at conf-path access time) from
`tractor` runtime-vars, which propagate down the actor
tree; route all conf-path derivation through
`config.get_conf_dir()` so the override is effective in
every (sub)actor.
## Files changed
- `piker/config.py` — add `_maybe_use_test_dir()`;
hook in `get_conf_dir()`; route `get_conf_path()` +
`load()` mkdir through it
- `piker/accounting/_ledger.py` — derive ledger dir
via `config.get_conf_dir()` (not the global)
- `piker/accounting/_pos.py` — same for account files
## Human edits
None — committed as generated.

View File

@ -1,46 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:08:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/config.py`
Generated: `config._maybe_use_test_dir()` — lazily
reads `piker_test_dir` from
`tractor.runtime._state._runtime_vars['piker_vars']`
(pre-loaded by `open_piker_runtime()` from the
`tests.conftest._open_test_pikerd()` overrides) and
calls `_override_config_dir()` when set. Hooked at the
top of `get_conf_dir()`; `get_conf_path()` and
`load()`'s dir-creation rerouted through
`get_conf_dir()`.
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
> `git log -1 -p --follow -- piker/accounting/_pos.py`
Generated: ledger/account dir derivation switched from
the `config._config_dir` module global to
`config.get_conf_dir()` + `mkdir(parents=True,
exist_ok=True)` for nested tmp-dir creation.
Key diagnostic reasoning (verbatim from session):
The old (commented-out) `get_app_dir()` override gated
on `'pytest' in sys.modules` which can NEVER work in
spawned subactors (fresh procs, no pytest import); as
a result test `paperboi`/daemon actors were writing
into the user's REAL `~/.config/piker/accounting/`
files. Evidence: `account.kraken.paper.toml` gained a
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
(a test fill), and `trades_kraken_paper.toml` (252B)
contained only that fill. The resolution must be
checked lazily at config-path access time (NOT import
time) since sub-actors only receive runtime-vars once
their `tractor` runtime has fully booted.

View File

@ -1,53 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:10:22Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
driver: the approved plan's per-stage test gates could
not run AT ALL — the branch base was broken vs.
`tractor` git `main` (`AttributeError: 'tuple' object
has no attribute 'unwrap'` at `pikerd` boot, stale
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
fixed forward autonomously to (re)establish the gate
baseline, continuing the `repair_tests` branch lineage
(verified already merged in ancestry via `git cherry`
during a user-requested branch-overlap survey).
## Response summary
Port the service layer + test suites to current
`tractor` APIs: addr-type normalization in
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
kwarg renames, modern self-cancel absorption semantics
in the cancel-method test harness, exc/position attr
renames, a paper-EMS startup-budget bump and a syntax
fix in `.deribit.api`.
## Files changed
- `piker/service/_actor_runtime.py` — `wrap_address()`
normalize before `.unwrap()` in `open_pikerd()`
- `piker/service/_registry.py` — `check_for_service()`
-> `query_actor(regaddr=)` + 2-tuple yield +
`open_registry(addrs=)`
- `piker/brokers/deribit/api.py` — missing comma in
`tractor.trionics` import tuple
- `tests/test_services.py` — `registry_addr=` kwarg,
raddr unwraps, cancel-semantics harness rewrite,
`fail_after` 9 -> 19s
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
## Human edits
None — committed as generated.

View File

@ -1,56 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:10:22Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
Generated: normalize each registry addr via
`tractor.discovery._addr.wrap_address()` before
`.unwrap()`-ing for the `accept_addrs` bind check —
entries may be raw `tuple`s when passed in from (test)
client code. Import-path precedent taken from
`piker/cli/__init__.py:336`.
> `git log -1 -p --follow -- piker/service/_registry.py`
Generated: `check_for_service()` ported to
`tractor.query_actor(name, regaddr=...)` (kwarg was
`arbiter_sockaddr=`), unpacking the new
`(sockaddr, portal)` yield, and passing the
now-required `open_registry(addrs=Registry.addrs)`.
> `git log -1 -p --follow -- tests/test_services.py`
> `git log -1 -p --follow -- tests/test_ems.py`
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
Key diagnostic reasoning (verbatim from session):
- the "DID NOT RAISE ContextCancelled" failure: in this
test the client actor IS pikerd (in-proc), and
current `tractor` main absorbs a `ContextCancelled`
whose canceller is your own actor — self-requested
cancels now exit cleanly instead of raising; the
'sigint' variant propagates a bare collapsed
`KeyboardInterrupt` rather than a
`BaseExceptionGroup`.
- the hard-coded `trio.fail_after(9)` startup budget is
marginal — full stack boot (pikerd -> emsd ->
brokerd.kraken -> paperboi + live kraken symbology
fetch) occasionally exceeds 9s -> bumped to 19s.
- `RemoteActorError.type` -> `.boxed_type`;
`Position.size` -> `.cumsize` (the paper engine
populates `BrokerdPosition.size` from `pp.cumsize`).
- overlap survey (user-requested): all of the
`repair_tests` branch commits are already in this
stack's ancestry; this commit finishes that branch's
port mission (its f4c4f1e2 fixed `conftest.py`'s
`arbiter_sockaddr` usage; this fixes the remaining
`test_services.py` + `check_for_service()` sites).

View File

@ -1,54 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:11:05Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 0" prep. Plan-shaping user
decisions captured via in-session Q&A:
- `datad.<broker>` topology: sibling of
`brokerd.<broker>` under `pikerd` (vs. child
subactor).
- migration: hard cutover, staged by layer (no
dual-mode runtime flag).
- post-split `brokerd` scope: trading-only +
EMS-lazy-spawned (charts/CLI never touch it).
## Response summary
Declare per-daemon-kind backend submod groups
(`_datad_mods`/`_brokerd_mods`) in the split-style
backends, keyed to the pre-existing
`piker.data.validate._eps` contract; add a
`validate.get_eps()` introspection helper; ws-import
hygiene in `.kraken.broker`. Zero behavior change
(`__enable_modules__` unions unchanged).
## Files changed
- `piker/data/validate.py` — add `get_eps()`
- `piker/brokers/kraken/__init__.py` — mod groups
- `piker/brokers/binance/__init__.py` — mod groups
(note: no `symbols.py`; search eps live in `.feed`)
- `piker/brokers/deribit/__init__.py` — datad-only
groups (no `broker.py` yet)
- `piker/brokers/ib/__init__.py` — add `'api'` to the
pre-existing `_datad_mods`
- `piker/brokers/kraken/broker.py` — import
`NoBsWs`/`open_autorecon_ws` from
`piker.data._web_bs` directly
## Human edits
None — committed as generated.

View File

@ -1,46 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:11:05Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/validate.py`
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
returning the daemon-kind's ep funcs defined by a
backend mod, keyed by ep name, missing eps excluded;
sourced from the existing `_eps` grouping table
(`'middleware' | 'datad' | 'brokerd'`).
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
Key exploration findings driving the design (from the
planning phase, 3 parallel explore agents + 1 architect
agent):
- the codebase already anticipated this split:
`validate._eps` groups eps into 'datad' vs 'brokerd'
kinds and `ib/__init__.py` already declared
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
carried the literal TODO "rename the daemon to datad
prolly once we split up broker vs. data tasks into
separate actors?".
- the feared kraken feed<->broker "shared ws state"
hazard is mild: `NoBsWs`/`open_autorecon_ws`
originate in `piker.data._web_bs` (re-exported via
`.kraken.feed`); only `stream_messages` is feed-local
and it's a pure parser — in-process imports only,
each actor opens its own ws conn.
- `enable_modules` gates RPC entry, NOT python imports,
so backend trading mods may keep importing feed-side
helpers in-process post-split.

View File

@ -1,47 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:11:42Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 1": introduce the `datad` daemon
machinery additively (nothing routes through it yet).
User-decided constraint applied: `datad.<broker>` is a
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
via the existing `Services` + `maybe_spawn_daemon()`
machinery.
## Response summary
New `piker/data/_daemon.py` hosting the
`datad.<broker>` feed-only daemon-actor: lifetime
fixture owning the actor-global `_FeedsBus`, init/spawn
fns mirroring `.brokers._daemon` conventions and the
`samplerd` sub-daemon precedent, plus root-mod
registration, `piker.service` re-exports and a spawn
test.
## Files changed
- `piker/data/_daemon.py` — NEW:
`_setup_persistent_datad()`, `datad_init()`,
`spawn_datad()`, `maybe_spawn_datad()`,
`_datad_service_mods`
- `piker/service/_actor_runtime.py` — add
`piker.data._daemon` to `_root_modules`
- `piker/service/__init__.py` — re-export spawn eps
- `tests/test_services.py` — add `test_datad_spawn`
## Human edits
None — committed as generated.

View File

@ -1,60 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:11:42Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/_daemon.py`
Generated symbols + key design decisions:
- `_datad_service_mods: list[str]` — datad-always
enabled mods, the data-side successor to the old
`piker.brokers._daemon._data_mods` set; kept minimal
per the caps-sec model.
- `_setup_persistent_datad()` — `@tractor.context`
lifetime fixture: console-log setup then allocates
the actor-global feed bus via
`feed.get_feed_bus(brokername, service_nursery)`
exactly as the old brokerd fixture did, pinned open
with `ctx.started()` + `sleep_forever()`.
- `datad_init()` — actor name `f'datad.{brokername}'`;
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
`infect_asyncio=True`); builds `enable_modules` from
`getattr(brokermod, '_datad_mods',
getattr(brokermod, '__enable_modules__', []))` as
the flat-backend fallback.
- `spawn_datad()` — `Services.actor_n.start_actor()` +
`Services.start_service_task()` exactly mirroring
`spawn_brokerd()`; dedup-composes enable mods via
`list(dict.fromkeys(...))`.
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
service_name=f'datad.{brokername}', ...)`.
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
> `git log -1 -p --follow -- piker/service/__init__.py`
> `git log -1 -p --follow -- tests/test_services.py`
Design rationale (verbatim from session):
- `_root_modules` must gain `piker.data._daemon` so
`pikerd_portal.run(spawn_datad, ...)` resolves in
the root.
- the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep, surfaced by the
user-requested branch-overlap survey) lands
symmetrically on both spawn fns.
- mod placement (`piker/data/_daemon.py` vs.
generalizing `piker.brokers._daemon`) follows the
per-subsystem daemon-mod convention
(`.clearing._ems`, `.data._sampling`) and resolves
the existing TODO at `brokers/_daemon.py:49` ("move
this def to the `.data` subpkg").

View File

@ -1,47 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:12:26Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 2": decouple the clearing layer
from `feed.portals` BEFORE the feed cutover so live
trading works at every stage boundary. User-decided
constraint applied: post-split `brokerd` is
trading-only and spawned LAZILY only by `emsd`'s
`open_brokerd_dialog()` path.
## Response summary
Kill the single coupling forcing feed + trading eps
into one actor (`Router.open_trade_relays()` pulling
its trades portal from `feed.portals[brokermod]`):
`open_brokerd_dialog()` now (maybe) spawns/finds
`brokerd.<broker>` itself and ONLY when a live
trades-ep will actually open; paper mode never touches
it. Pre-cutover this is a pure refactor (registry
lookup finds the same feed-spawned daemon).
## Files changed
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
re-sig + inner `acquire_live_portal()`;
`Router.maybe_open_brokerd_dialog()` drops `portal`
param; `open_trade_relays()` drops the
`feed.portals` lookup
- `piker/accounting/cli.py` — keyword-form `portal=`
override kept for the `piker ledger` ad-hoc actor
## Human edits
None — committed as generated.

View File

@ -1,43 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:12:26Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/clearing/_ems.py`
Generated: new `open_brokerd_dialog()` signature
`(brokermod, exec_mode, fqme=None, portal=None,
loglevel=None)` with an inner `@acm
acquire_live_portal()` that yields the caller-provided
`portal` override (the `piker ledger` path) else
`maybe_spawn_brokerd(brokermod.name)` — designated THE
one place a live, credentialed `brokerd.<broker>` gets
booted post-split. The eager
`portal.open_context(trades_endpoint, ...)`
construction moved inside that block so the paper
short-circuit never acquires a live portal.
> `git log -1 -p --follow -- piker/accounting/cli.py`
Key analysis (verbatim from session):
- `feed.portals` had exactly ONE trading consumer:
`piker/clearing/_ems.py:671` (`portal =
feed.portals[brokermod]`) — the single coupling
forcing feed + trading into one actor.
- `piker ledger` (`accounting/cli.py:100-157`) is a
hidden consumer: it calls `broker_init()` directly,
spawns its own ad-hoc actor and passes the portal
positionally — the new signature must keep an
explicit `portal:` override param for this path.
- stage sequencing: clearing decouple lands BEFORE the
feed cutover, otherwise `feed.portals` would hand
emsd a `datad` portal and live `open_trade_dialog`
RPC would fail.

View File

@ -1,48 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:12:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 3": the feed-layer hard cutover
(user-decided: no dual-mode runtime flag, each stage
lands fully working).
## Response summary
The topology flip: all data-feed consumers route to
`datad.<broker>`; the `brokerd` lifetime fixture slims
to trading-only; UI/CLI data-flavoured spawn sites
repointed; test expectations inverted incl. an explicit
negative assert that paper sessions spawn NO
`brokerd.<broker>`.
## Files changed
- `piker/data/feed.py` — `open_feed()` ->
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
assert -> `'datad'`; `install_brokerd_search` ->
`install_datad_search`; comment sweep
- `piker/brokers/_daemon.py` —
`_setup_persistent_brokerd()` slimmed (feed-bus
alloc moved to the datad fixture)
- `piker/ui/_app.py`, `piker/ui/cli.py`,
`piker/brokers/core.py`, `piker/brokers/cli.py`,
`piker/ui/kivy/option_chain.py` — spawn-site
repoints
- `tests/test_services.py` — datad assertions + the
no-`brokerd.kraken` negative check
## Human edits
None — committed as generated.

View File

@ -1,54 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:12:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/feed.py`
Generated: `open_feed()` spawn cutover with the
`maybe_spawn_datad` import done "relative-direct" from
`._daemon` — NOT via `piker.service` — to dodge a
partial-init cycle: when `piker.data.feed` loads as
part of `piker.service.__init__` executing its own
`piker.data._daemon` import, the service pkg is
mid-init and its `maybe_spawn_datad` binding does not
exist yet. The `open_feed_bus()` local-state sanity
assert flips `'brokerd' in servicename` ->
`'datad' in servicename` (the only actor-name assert
in the tree, verified by grep).
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
Generated: `_setup_persistent_brokerd()` slimmed to
console-log setup + pinned-open ctx; drops the bus
alloc, the `assert not feed._bus`, the service nursery
(backend `open_trade_dialog()` ctxs own their task
trees) and the `eg.ExceptionGroup` handler. The
`piker ledger` ad-hoc actor enters this same slimmed
fixture — exactly what it needs.
> `git log -1 -p --follow -- piker/ui/_app.py`
> `git log -1 -p --follow -- piker/ui/cli.py`
> `git log -1 -p --follow -- piker/brokers/core.py`
> `git log -1 -p --follow -- piker/brokers/cli.py`
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
> `git log -1 -p --follow -- tests/test_services.py`
Verification (from session): per-suite gates green at
this commit (services 5-passed incl. the new negative
assert, feeds 3-passed); a headless live smoke
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
on an alt registry port) confirmed quotes flowing via
`datad.kraken` + `samplerd` with `check_for_service(
'brokerd.kraken') is None`. Known pre-existing flake
documented: `test_multi_fill_positions`' second
in-proc runtime boot wedges ~50% (zombie subactor w/
unread parent-IPC bytes); reproduced with the split
fully reverted so NOT a regression of this work.

View File

@ -1,43 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:13:44Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's final "stage 4": caps-sec slimming of
the (live, credentialed) trading actor + the `ib`
dual-daemon `client_id` collision mitigation flagged in
the plan's risk register.
## Response summary
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
spawn fails fast for datad-only backends with a "use
paper-mode" error; `ib`'s default api-gw `client_id`
gets a per-daemon-kind offset so `datad.ib` +
`brokerd.ib` don't collide on connect.
## Files changed
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
`_brokerd_service_mods`; `broker_init()` reads
`_brokerd_mods` (fallback `__enable_modules__`);
`spawn_brokerd()` fail-fast via `validate.get_eps()`
- `piker/brokers/ib/api.py` — role-based `client_id`
offset in `load_aio_clients()`
- `piker/cli/__init__.py` — resolved "expose datad"
TODO
## Human edits
None — committed as generated.

View File

@ -1,41 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:13:44Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
Generated: the fail-fast originally landed in
`broker_init()` but was relocated to `spawn_brokerd()`
mid-implementation after realizing `piker ledger`
calls `broker_init()` directly even for paper accounts
on datad-only backends (would have crashed the cli);
the service-spawn path is the correct enforcement
seam. Error text:
Backend 'kucoin' offers NO `brokerd` (live
order-control) eps!? It is likely a datad-only
provider, use paper-mode for clearing instead.
(verified live via a `trio.run()` unit check.)
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
Generated: in `load_aio_clients()`, when `client_id`
is the 6116 default: `datad`-named actors offset +16
(disjoint from `brokerd`'s linear `client_id + i`
retry range), other non-`brokerd` (ad-hoc test/cli)
actors +32. Rationale from the plan's risk register:
post-split BOTH per-broker daemons connect to the same
TWS/gw endpoint; a shared default id collides and
burns up to `connect_timeout * retries` (90s) in
retry cycles.
> `git log -1 -p --follow -- piker/cli/__init__.py`

View File

@ -1,66 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:33:09Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
scope: docs
substantive: true
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
---
## Prompt
The session-initiating instruction, verbatim (this doc
IS its "mega detailed plan" deliverable):
> ok i want you to become the distributed runtime and
> concurrency expert for this project - namely acquire
> a deep understanding of tractor and how it's used.
> then i want you to atttempt to factor our current
> brokerd service daemon into 2 daemons:
>
> - a brokerd which only hosts live/paper trading
> endoint tasks as defined namely within all
> piker/brokers/<backend>/broker.py mods
> - a new `datad` subdaemon which in a separate
> subactor serves all the data feed service tasks
> namely delivered by endpoints in
> piker/brokers/<backend>/feed.py (or similar for
> less mature backends) and as more rigorously
> defined by the validation machinery in
> piker.data.validate.
>
> give me a mega detailed plan on how to approach
> this, and a staged approach for the implementation.
Plan-shaping user decisions (in-session Q&A): sibling
topology under `pikerd`; hard cutover staged by layer;
trading-only EMS-lazy-spawned `brokerd`. The human
staged the doc copy into `ai/claude-code/plans/` and
requested this commit after the 7 implementation
commits landed.
## Response summary
The staged design doc for the `brokerd` -> (`datad` +
`brokerd`) split: context, target supervision topology,
load-bearing verified facts, per-stage file-level
changes with gates, risk register and an end-to-end
verification matrix. Produced via 3 parallel explore
agents + 1 architect agent + human Q&A before any code
was written; all 7 implementation commits in this
branch reference its stages in their provenance
entries.
## Files changed
- `ai/claude-code/plans/datad_service.md` — the design
plan doc (human-staged copy of the AI-authored plan)
## Human edits
None to the doc content — committed as generated. NB:
the implementation deviated from the plan-as-written in
4 places, see the raw file's deviation log.

View File

@ -1,51 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:33:09Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
---
NOTE: diff-ref mode entry; the AI output here is the
committed doc itself (a design plan, not code) so no
verbatim copy is duplicated:
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
## Implementation deviation log
Where the landed 7-commit series differs from the plan
as written (recorded for reviewer transparency; the
plan doc is committed unmodified):
1. Commit ordering: the `pytest` config-dir isolation
fix lands BEFORE the `tractor`-API drift port. The
plan's stage gates assumed a runnable+isolated test
baseline; per-commit gating exposed that without
isolation the paper-EMS test reads the user's real
(polluted) `account.kraken.paper.toml` and reddens.
2. The plan's "stage 0: full pytest green" gate
required first repairing pre-existing branch
breakage vs `tractor` git `main` (boot
`AttributeError`, stale discovery/exc/position
APIs) — that repair became its own commit
("Port service+tests to latest `tractor` APIs")
rather than plan-stage work.
3. Stage-4 fail-fast placement: the plan said
`broker_init()` raises on brokerd-ep-less backends;
implementation moved the raise to `spawn_brokerd()`
since `piker ledger` calls `broker_init()` directly
even for paper accounts on datad-only backends.
4. `brokers/_daemon.py` change grouping: the
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
type-only import) landed with the caps-sec slim
commit instead of the fixture-slim commit, keeping
each intermediate tree import-clean without
sub-hunk surgery.
Also of record: the plan's "Verification" matrix was
executed as written (per-suite gates each stage, the
headless datad-feed smoke, the kucoin fail-fast unit
check); the known pre-existing ~50% second-runtime-boot
test wedge was characterized and excluded as a
regression via revert-testing.

View File

@ -1,60 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T21:35:49Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
---
## Prompt
User bug report from live `ib` testing of the
(datad|brokerd)-split branch:
> ok doing some testing and noticing live orders do not
> work otb, pikerd show this on live submission,
> [pdb traceback: `RuntimeError("Can not order
> {symbol}, no live feed?")` at `ib/api.py:1152`
> `submit_limit()` with `self._contracts == {}` in the
> `brokerd.ib` actor + a `TrioTaskExited` teardown
> cascade]
Follow-up user direction fixing the design intent:
> i think the main thing here is that [brokerd] should
> be able to always submit orders without a live feed
> being up and registered in the same subactor right?
## Response summary
Root cause: pre-split the feed code populated the
actor-local `Client._contracts` cache (via
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
SAME process as order submission; post-split the
trading actor's client is never warmed. Fix: lazily
qualify + cache the contract per order request in
`handle_order_requests()` by running the same
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
plus per-order error relay (`BrokerdError`) so one bad
submission can't crash the whole trades dialog (the
`TrioTaskExited` storm was teardown cascade from the
original raise).
## Files changed
- `piker/brokers/ib/broker.py` — thread `proxies` into
`handle_order_requests()`; lazy contract qualify on
cache-miss; guard `submit_limit()` w/ `BrokerdError`
relay; uncomment the (anticipatory) `get_mkt_info`
import
- `piker/brokers/ib/api.py` — fix the non-f-string
raise msg + document the new qualification contract
## Human edits
None — committed as generated. Live `ib` order retest
performed by the human post-commit.

View File

@ -1,57 +0,0 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T21:35:49Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); recorded from the live debug
session per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
Key diagnostic chain (from session):
- pdb showed `Client._contracts == {}` inside
`brokerd.ib`'s `submit_limit()`; the cache has
exactly TWO write sites: `Client.find_contracts()`
(api.py, keys `f'{sym}.{exch}.ib'`) and
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
'nvda.nasdaq' — NO `.ib` suffix).
- `BrokerdOrder.symbol` arrives as the bs_fqme form
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
site produces the key `submit_limit()` reads —
ie. pre-split it was the feed's in-proc
`get_mkt_info(sym, proxy=proxy)` call keeping orders
working, NOT `find_contracts()`.
- the existing TODO at `symbols.py:642-644` literally
predicted this: "this is going to be problematic
if/when we split out the datad vs. brokerd actors
since the mktmap lookup table will now be
inaccessible.."
- instance identity verified: `proxy._aio_ns` IS the
same `Client` obj as `_accounts2clients[account]`
(both sourced from the `load_aio_clients()` cache via
`open_client_proxies()`), so a brokerd-side
`get_mkt_info(fqme, proxy=proxies[account])` warms
exactly the dict `submit_limit()` reads. It also
populates `client._cons2mkts` which the
position-audit path (`broker.py` backup-table code)
needs in this actor anyway.
- the `TrioTaskExited` storm in the user's log
(`recv_trade_updates`, `open_aio_client_method_relay`
aio tasks) is teardown cascade: the raise crashed
`handle_order_requests` -> nursery teardown ripped
the trio sides of still-running aio relay tasks.
Hence the added per-order try/except ->
`BrokerdError` relay hardening so a single bad
submission degrades to an EMS error msg instead of
killing the backend's entire order-ctl dialog.
Verification: `tests/test_services.py` (5 passed) +
`tests/test_ems.py` (6 passed) regression-green; live
`ib` submission retest delegated to the human (needs a
running TWS/gw).

View File

@ -1,34 +0,0 @@
# AI Prompt I/O Log — claude
This directory tracks prompt inputs and model
outputs for AI-assisted development using
`claude` (claude-code CLI).
## Policy
Prompt logging follows the
[NLNet generative AI policy][nlnet-ai].
All substantive AI contributions are logged
with:
- Model name and version
- Timestamps
- The prompts that produced the output
- Unedited model output (`.raw.md` files)
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
## Usage
Entries are created by the `/prompt-io` skill
or automatically via `/commit-msg` integration.
Each commit carrying AI-generated changes links
to its provenance entry via a `Prompt-IO:`
commit-msg trailer; entries use "diff-ref mode"
(pointers into `git log -p` instead of verbatim
code copies) to avoid duplicating committed
code.
Human contributors remain accountable for all
code decisions. AI-generated content is never
presented as human-authored work.

View File

@ -1,46 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T00:13:57Z
git_ref: 16f3cd6f
scope: code
substantive: true
raw_file: 20260722T001357Z_16f3cd6f_prompt_io.raw.md
---
## Prompt
Continue the gap-checker and history-backfiller repair plan. Start
with deterministic analysis characterization, then fix canonical
sorting, dedupe-derived columns, positive gap detection, and null
grouping.
During review, the user challenged the replacement null-grouping
algorithm because its initial tests did not assert exact endpoints.
The user explicitly required retaining detailed, human-oriented
algorithm documentation such as the existing ASCII segment map.
## Response summary
Added deterministic analysis regressions and corrected normalization
so sorting and duplicate resolution precede derived-column
calculation. Unified null grouping for NumPy and Polars inputs,
documented its exact inclusive-boundary contract with an adapted
visual map, and verified all small null layouts against an independent
state-machine oracle. Retained sorted smart-dedupe output.
## Files changed
- `piker/tsp/_anal.py` - canonical gap and documented null grouping
- `piker/tsp/_dedupe_smart.py` - sorted duplicate resolution
- `tests/test_tsp_analysis.py` - exact and exhaustive regressions
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -1,40 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T00:13:57Z
git_ref: 16f3cd6f
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
Generated canonical gap and null-segment normalization. Derived
datetime and delta columns are recomputed after final sorting and
duplicate resolution, and negative deltas no longer become ordinary
gaps.
NumPy and Polars null grouping share one implementation while
retaining the human-oriented visual map of contiguous zero runs. The
documented contract defines absolute inclusive endpoints, margin
expansion, frame boundary clamping, contiguous-index validation, and
requested-column selection.
> `git diff HEAD~1..HEAD -- piker/tsp/_dedupe_smart.py`
Adjusted smart dedupe to honor sorted output and resolve duplicate
timestamps without retaining stale order.
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
Added deterministic synthetic OHLCV coverage for sorted dedupe, fresh
derived columns, and positive gap detection. Null grouping is checked
against readable endpoint examples and an independent state-machine
oracle over every layout through six rows, three margin widths, and
both NumPy and Polars inputs.
Verification generated with the patch:
`timeout -k 5 30 python -m pytest -q tests/test_tsp_analysis.py`
Result: 11 passed.

View File

@ -1,36 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T00:13:58Z
git_ref: 16f3cd6f
scope: code
substantive: true
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
---
## Prompt
Continue the structured backfill phase after the analysis regressions.
Characterize provider exhaustion and genuinely empty frames, then remove
completion paths that can hang after child-task exit.
## Response summary
Removed orphan-prone completion events from the structured nursery flow
and checked empty provider frames before indexing timestamp endpoints.
Added deterministic timeout-bounded regressions for both exit paths.
## Files changed
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
- `tests/test_history_backfill.py` - deterministic exit regressions
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -1,25 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T00:13:58Z
git_ref: 16f3cd6f
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Removed redundant completion events whose consumers waited only after
their owning nurseries had joined. Added empty provider-frame handling
before timestamp endpoint indexing so provider exhaustion exits cleanly.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Added bounded, actor-free regressions for `DataUnavailable` and empty
provider-frame completion.
Verification generated with the patch:
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
Result: 2 passed.

View File

@ -1,39 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T02:41:33Z
git_ref: 3a4f6737
scope: code
substantive: true
raw_file: 20260722T024133Z_3a4f6737_prompt_io.raw.md
---
## Prompt
The user challenged the analysis commit message's claim that only
positive sample-step gaps are detected and asked whether out-of-order
timestamps were simply ignored. After confirming the missing separate
reporting path, the user instructed the agent to implement the fix.
## Response summary
Separated missing-history gaps from timestamp-ordering errors without
changing the existing `dedupe()` return shape. Added a structured
detector and made normalization warn before sorting removes evidence of
duplicates or reversed timestamps.
## Files changed
- `piker/tsp/_anal.py` - ordering detector and dedupe reporting
- `piker/tsp/__init__.py` - public detector export
- `tests/test_tsp_analysis.py` - exact ordering-error regression
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -1,30 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T02:41:33Z
git_ref: 3a4f6737
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
Added `detect_time_ordering_errors()` to return non-positive timestamp
steps as structured rows. `dedupe()` now warns with those rows before
sorting and duplicate removal erase the original ordering evidence.
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
Exported the ordering detector through the public TSP package.
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
Changed the negative-delta regression to verify exact predecessor,
timestamp, and delta values, confirm exclusion from missing-history
gaps, and require explicit normalization reporting.
Verification generated with the patch:
- `tests/test_tsp_analysis.py`: 11 passed
- Ruff: passed
- `git diff --check`: passed

View File

@ -1,43 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T02:41:34Z
git_ref: 3a4f6737
scope: code
substantive: true
raw_file: 20260722T024134Z_3a4f6737_prompt_io.raw.md
---
## Prompt
The user asked why the in-progress `piker/storage` changes were omitted
from the earlier commit plan. After receiving an inventory of unfinished
Phase 3 requirements, the user instructed the agent to complete the work.
## Response summary
Finished non-destructive, atomic NativeDB updates and integrated them
with history startup and reverse backfill. Expanded tests through two
adversarial review rounds that caught and corrected cross-client lock
blocking, stale temp indexing, discontinuous persisted indexes, startup
snapshot races, missing latest-frame persistence, notification wedges,
and mutable/truncated SHM persistence.
## Files changed
- `piker/storage/__init__.py` - incremental storage contract
- `piker/storage/nativedb.py` - validated atomic merge persistence
- `piker/storage/marketstore/__init__.py` - non-duplicate update mode
- `piker/tsp/_history.py` - provider-delta and startup persistence
- `tests/test_storage_nativedb.py` - NativeDB durability regressions
- `tests/test_history_backfill.py` - persistence lifecycle regressions
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -1,43 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T02:41:34Z
git_ref: 3a4f6737
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/__init__.py piker/storage/nativedb.py piker/storage/marketstore/__init__.py`
Added an incremental storage update contract. NativeDB now validates
canonical numeric OHLCV data, serializes writers both actor-locally and
through cancellable filesystem locks, merges with incoming-wins seam
resolution, regenerates contiguous indexes, reopens and validates a
same-directory temporary parquet, fsyncs it, atomically replaces the
target, fsyncs the directory, and publishes cache only after replacement.
NativeDB indexing ignores lock and crash-left temporary files.
MarketStore uses its non-duplicate write mode for incremental updates.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Separated full provider deltas from capacity-truncated SHM pushes.
Backfill persists before SHM mutation, publishes bounded notifications,
and delays latest-frame publication until the concurrent pre-update
storage snapshot completes.
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py tests/test_history_backfill.py`
Added deterministic coverage for merge preservation, conflict policy,
atomic failures, reopened-candidate validation, cache consistency,
schema and timestamp invariants, contiguous index regeneration,
separate series, stale temp indexing, cancellable file-lock contention,
full provider-delta persistence, latest-frame ordering, and bounded
notification teardown.
Verification generated with the patch:
- deterministic analysis/backfill/storage set: 31 passed
- Ruff: passed
- `git diff --check`: passed
- final adversarial review: no findings

View File

@ -1,40 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: unavailable
timestamp: 2026-07-27T20:47:11Z
git_ref: 84a6d47b
scope: code
substantive: true
raw_file: 20260727T204711Z_84a6d47b_prompt_io.raw.md
---
## Prompt
The user identified that `backfiller_deep_fixes` introduced persistent
`.parquet.lock` files, questioned whether Parquet or Arrow already handled
writer coordination, and clarified that piker's `tractor` structured
concurrency design already provides single-writer ownership through the
persistent `datad` feed task. The user then asked to open the backfiller
worktree and resolve the redundant locking.
## Response summary
Removed redundant actor-local and filesystem locking while preserving
atomic crash-safe Parquet replacement. Replaced the synthetic contention
test with regressions for the actual cross-branch failure: writes create
no lock sidecars, and legacy lock artifacts are ignored during indexing.
## Files changed
- `piker/storage/nativedb.py` - rely on `datad` writer ownership
- `tests/test_storage_nativedb.py` - lock-sidecar regressions
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -1,30 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-27T20:47:11Z
git_ref: 84a6d47b
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
Removed actor-local and filesystem writer locks from NativeDB. The
persistent `datad` feed task already owns writes for each series, while
its timeframe children write distinct files. NativeDB retains validated
temporary writes, file and directory synchronization, and atomic target
replacement for crash durability. The exact `.parquet` suffix filter is
retained so legacy lock sidecars are harmless.
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
Removed the synthetic cross-client lock-contention test. Added regression
coverage proving replacement and incremental writes create no lock
sidecars. Extended indexing coverage with the legacy
`.parquet.lock` artifact that caused upstream `flake_update` to crash.
Verification generated with the patch:
- NativeDB and history regression set: 20 passed
- Ruff: passed
- `git diff --check`: passed

View File

@ -1,42 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-27T21:24:54Z
git_ref: 689df816
scope: code
substantive: true
raw_file: 20260727T212454Z_689df816_prompt_io.raw.md
---
## Prompt
The user reran the known IB FQME on `backfiller_deep_fixes` and reported
that `publish_latest_frame()` still failed because NativeDB required an
`index` absent from IB's provider frame. The user called out the missed
tabular schema boundary and requested an actual end-to-end test suite
rather than storage fakes.
## Response summary
Repaired the IB-provider-to-NativeDB boundary by canonicalizing durable
fields and mapping reloads into provider-specific SHM buffers. Added a
local actor/SHM/Parquet integration regression covering first startup and
restart, plus focused conversion and timestamp-coercion tests. Two review
rounds found and resolved restart hydration and fractional-time hazards.
## Files changed
- `piker/storage/nativedb.py` - canonical storage and SHM field mapping
- `piker/tsp/_anal.py` - name-based Polars-to-NumPy conversion
- `tests/test_history_backfill.py` - actor/SHM/Parquet integration test
- `tests/test_storage_nativedb.py` - schema conversion regressions
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -1,34 +0,0 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-27T21:24:54Z
git_ref: 689df816
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/tsp/_anal.py`
Normalized index-less provider frames to NativeDB's canonical names,
order, and dtypes before merge and publication. Provider-only columns
are excluded from durable files, derived indexes are regenerated, and
fractional timestamps are rejected before integer conversion. Added a
canonical field map for hydrating provider-specific SHM buffers while
leaving provider-only fields at their defaults. Changed Polars-to-NumPy
conversion to select columns by field name rather than position.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py tests/test_storage_nativedb.py`
Added deterministic coverage using IB's actual provider dtype and a real
local `tractor` root actor, IB-typed shared-memory buffers, NativeDB,
Parquet persistence, first-start publication, and restart hydration.
Added focused regressions for reordered/extra Polars columns and
fractional timestamp rejection.
Verification generated with the patch:
- NativeDB and history integration set: 23 passed
- Ruff: passed
- `git diff --check`: passed
- second adversarial review: no findings
- no live gateway or network tests run

View File

@ -1,36 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-08-11T20:58:26Z
git_ref: 06d5ea5d
scope: code
substantive: true
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
---
## Prompt
Diagnose and fix an IB live-order failure where lazy NVDA contract
qualification raised `tractor.trionics.Lagged` after the order reached
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
searches caused the stale state.
## Response summary
Traced the failure to competing consumers of a one-slot Trio broadcast, not
the recent SHM change or stale asyncio client state. Replaced broadcast
fan-out with one response/status demultiplexer, retained request-ID
correlation, handled cancellation and teardown, documented the task/channel
ownership model, warned on diagnostic metadata eviction, and added
deterministic proxy regressions.
## Files changed
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
## Human edits
None - generated output follows the user's live failure diagnosis.

View File

@ -1,66 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-11T20:58:26Z
git_ref: 06d5ea5d
diff_cmd: git diff HEAD~1..HEAD
---
The user reported that certain IB order entries might have broken and
provided a live failure for a first NVDA sell order. EMS delivered the
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
qualification failed in `MethodProxy._run_method()` with:
```text
tractor.trionics._broadcast.Lagged:
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
dropped `0` values
```
The user suspected a cancelled symbol-search task might have left stale
asyncio state. Cancellation can leave an orphaned method response, but the
observed failure was stale Trio broadcast state. `open_client_proxy()`
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
and an idle method receiver. Status traffic advanced only the relay, so the
first method call for an uncached symbol entered an already-lagged receiver.
Orders for eagerly cached contracts bypassed this path.
The immediately preceding sparse-history commit did not modify IB order,
symbol, or proxy code; it exposed no direct dependency on this failure.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
Generated a sole-reader message demultiplexer for each IB client proxy.
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
status/error traffic, and dispatches `mid`-tagged responses into per-call
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
late responses to be logged and dropped without skewing later calls. Clean
channel EOF is accepted during normal proxy teardown.
Follow-up review restored the masked reconnect TODO and per-message case
rationale, documented the one-reader/many-writer task hierarchy and separate
per-proxy channels, and added a warning with request method metadata when the
bounded unresolved-request diagnostic table evicts its oldest entry.
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
Generated deterministic regressions for idle status traffic before first
qualification, two concurrent calls with reverse-order responses, cancelled
calls with late responses followed by a successful call, and graceful
channel EOF. Every synchronization wait is timeout-bounded.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Registered the deterministic IB method-proxy target and source mapping.
Verification output:
```text
........................ [100%]
25 passed, 4 warnings in 3.44s
```
The warnings are existing Tractor/Trio deprecations from the EMS test.
Adversarial review drove a clean-EOF guard and bounded polling. Final review
found no actionable issues. Live IB order validation remains pending and
requires restarting `brokerd.ib`.

View File

@ -1,34 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-13T20:24:02Z
git_ref: 12b0d66e
scope: code
substantive: true
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
---
## Prompt
Continue the paired piker broadcast-consumer work after Tractor PR 490
stabilization. Stop cached feed callers from mutating or consuming shared
quote-stream roots, then review, verify and prepare a commit boundary.
## Response summary
Made provider root streams private to the cached `Feed`, gave every caller a
local subscribed view, corrected feed cache identity, and made multi-provider
relays own synchronized child subscriptions and independent send-channel
clones.
## Files changed
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
subscribed multi-provider relays with independent channel lifetimes.
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
cache-identity regressions, including sibling survival after provider EOC.
## Human edits
None - generated output follows the requested cached-feed ownership item.

View File

@ -1,51 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-13T20:24:02Z
git_ref: 12b0d66e
diff_cmd: git diff HEAD~1..HEAD
---
The user asked to continue the paired piker broadcast-consumer work after
Tractor PR 490 stabilization. The next item was to stop cached feed callers
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
review, verify and prepare a commit boundary.
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
root streams and every caller, including the first cache owner, receives a
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
The cache identity now includes normalized full symbol shape and stream policy.
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
of directly consuming a shared stream, and synchronizes relay startup before
yielding the merged channel. Each relay owns a cloned send channel so one
provider EOC cannot terminate its siblings. The module-level relay returns its
live child receiver through `TaskStatus` for typed retention and introspection.
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
shared by local cache users.
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
multi-provider relay fan-out and startup, duplex controls, lexical child
closure, sibling survival after provider EOC, cached-root immutability, and
full cache-key differentiation.
Verification output:
```text
... [100%]
3 passed in 0.02s
. [100%]
1 passed in 0.01s
```
The second result is the existing EMS broadcast ownership regression. Ruff,
E501, compilation and whitespace checks passed. Repeated adversarial review
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
relay-startup and first-owner asymmetry issues; final review reported no
findings.

View File

@ -1,32 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-13T23:07:44Z
git_ref: ddac3553
scope: code
substantive: true
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
---
## Prompt
Continue the piker broadcast-consumer audit after the Deribit ownership commit
and prepare the next smallest ownership fix.
## Response summary
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
context and gave every caller a lexical child from one retained
`LinkedTaskChannel` broadcaster root.
## Files changed
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
ownership.
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
lifetime regression.
## Human edits
None - generated output follows the requested broadcast-consumer audit.

View File

@ -1,39 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-13T23:07:44Z
git_ref: ddac3553
diff_cmd: git diff HEAD~1..HEAD
---
The user asked to continue the piker broadcast-consumer audit after committing
the Deribit ownership fix. The next isolated item was IB's actor-local
`_quote_streams` cache, which retained a raw channel owned by its first caller
and constructed competing broadcaster roots for later callers.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
made `open_aio_quote_stream()` cache that context by symbol through
`maybe_open_context()`. Every caller now receives a lexical child from the
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
until its final cache user exits. Removed the manual `_quote_streams` registry
and teardown mutation.
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
Added a concurrent ownership regression using real Tractor broadcast state.
It proves miss and hit callers use the same symbol/context factory, receive the
same quote through distinct children, close independently, and leave the owner
live after the hit exits.
Verification output:
```text
. [100%]
1 passed in 0.01s
```
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
Python compilation and whitespace checks pass. Normal package collection
remains blocked by the existing Qt environment mismatch.

View File

@ -1,43 +0,0 @@
---
model: gpt-5.6-sol
service: opencode
session: tractor-addr-unpacking
timestamp: 2026-08-20T16:35:19Z
git_ref: 5f631e19
scope: code
substantive: true
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
---
## Prompt
The human asked to implement the Piker compatibility work caused by
tractor's canonical tagged-address migration, run the previously
deferred tractor verification, and keep runtime-managed WireGuard
bindspaces in a later separate PR.
## Response summary
Made Piker normalize legacy and tagged registry inputs before semantic
comparison and process-global storage. Updated runtime address typing and
test assertions for canonical output, retained legacy caller input, and
removed a duplicate pytest log-level option that blocked downstream test
collection with current tractor.
## Files changed
- `piker/service/_registry.py` - canonical registry normalization and
shared address typing.
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
annotations.
- `tests/conftest.py` - canonical fixture comparison and shared pytest
option ownership.
- `tests/test_services.py` - registry regression and canonical runtime
assertions.
## Human edits
The human explicitly requested the downstream implementation and chose
to keep WireGuard bindspace lifecycle work in a separate future tractor
PR. The agent implemented and tested those directions; no direct manual
source edits were observed.

View File

@ -1,25 +0,0 @@
---
model: gpt-5.6-sol
service: opencode
timestamp: 2026-08-20T16:35:19Z
git_ref: 5f631e19
diff_cmd: git diff HEAD~1..HEAD
---
# Raw output - tractor tagged-address compatibility
The human requested downstream Piker qualification after tractor PR
#505 switched transport address emission to canonical tagged tuples.
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
Normalized legacy and tagged registry addresses through tractor's
address codec before Piker stores or compares them. Extended service
runtime annotations to cover both tuple shapes, updated runtime tests to
compare canonical serialized values, and removed Piker's duplicate
registration of tractor's shared `--ll` pytest option.
Ruff passed. The focused normalization regression and Piker runtime boot
passed against the local tractor checkout at `5d92595f`. The wider
service module reached an unrelated pre-existing logger-name assertion
in `maybe_spawn_daemon()` after both address-focused tests passed.

View File

@ -1,39 +0,0 @@
---
model: gpt-5.6-sol
service: opencode
session: tractor-addr-unpacking
timestamp: 2026-08-21T04:20:58Z
git_ref: b7aca4c0
scope: code
substantive: true
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
---
## Prompt
The human approved refining Piker's logger-name handling after the
wider service test exposed a stale parent-logger assertion during the
tractor tagged-address compatibility work.
## Response summary
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
preserving complete requested submodule names. Removed daemon startup
assertions that incorrectly required parent logger names and added a
regression covering both logger factories and names that the old
character-set stripping silently corrupted.
## Files changed
- `piker/log.py` - exact relative logger-name derivation.
- `piker/service/_daemon.py` - remove stale parent-name assertion.
- `piker/data/_daemon.py` - remove stale parent-name assertion.
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
- `piker/clearing/_ems.py` - remove redundant logger assertion.
- `tests/test_logging.py` - package logger namespace regressions.
## Human edits
The human explicitly approved this follow-up after reviewing the
residual Piker service-suite gap. The agent implemented and verified
the refinement; no direct manual source edits were observed.

View File

@ -1,24 +0,0 @@
---
model: gpt-5.6-sol
service: opencode
timestamp: 2026-08-21T04:20:58Z
git_ref: b7aca4c0
diff_cmd: git diff HEAD~1..HEAD
---
# Raw output - preserve Piker logger namespaces
The human approved a follow-up refinement after the full Piker service
module exposed a stale logger-name assertion while qualifying tractor
PR #505.
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
Replaced character-set `str.lstrip()` calls with exact package-prefix
removal, removed assertion-only parent-logger expectations from daemon
startup paths, and added regression coverage for both logger factories
and prefix-sensitive submodule names.
Ruff passed after removing newly unused logger assignments. All four
logger regressions and all six service integration tests passed against
the local tractor PR #505 checkout.

View File

@ -1,46 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-26T22:26:00Z
git_ref: 5f631e19
scope: code
substantive: true
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
---
## Prompt
Diagnose and fix the hanging Binance USD-M perpetual feed for
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
API and changelog. Ensure unsupported or silent websocket feeds fail
loudly into `datad` and chart consumers instead of blocking startup.
## Response summary
Migrates Binance USD-M market data from the retired legacy websocket to
the split public and market routes, handles Binance's new aggregate-trade
schema, and introduces bounded connection and first-data startup. Direct
stream URLs avoid reconnect-time subscription-ACK races, while targeted
tests preserve routing and fail-loud behavior.
## Files changed
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
Binance quote streams.
- `piker/brokers/binance/venues.py` - define current USD-M websocket
endpoints.
- `piker/data/_web_bs.py` - support bounded websocket context startup.
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
retain the generated-output record.
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
structured provenance for the code changes.
## Human edits
The human explicitly required fail-loud behavior rather than merely
updating the endpoint, and supplied a direct CLI reproduction proving
both `datad.binance` and the chart caller blocked. Those requirements
drove the connection and first-data deadlines and their regressions. The
changes remain uncommitted for human review.

View File

@ -1,57 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-26T22:26:00Z
git_ref: 5f631e19
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user reported that selecting
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
websocket but never loaded the feed or downstream chart. They asked
OpenCode to compare Binance's API changelog with piker's FQME and feed
code, diagnose the failure, and solve it.
The user clarified that future provider failures must raise loudly
instead of silently blocking feed and chart startup. They also confirmed
the same hang from
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
## Generated code
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Routes USD-M book-ticker and aggregate-trade data through Binance's new
`/public` and `/market` websocket paths, merges both sockets into the
existing quote stream, accepts the new `st` aggregate-trade field, and
requires both stream classes before declaring startup complete. It also
removes the misleading missing-provider warning, derives feed mode from
resolved pair metadata, and raises `DataUnavailable` for connection or
live-data startup timeouts.
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
Defines Binance's current public and market USD-M websocket endpoints
and makes the market endpoint the regular futures websocket API root.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Adds an optional connection deadline to `open_autorecon_ws()` so callers
can fail context entry when repeated websocket handshakes never become
connected.
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
Adds regressions for exact Binance stream routing, silent live-data
startup, silent websocket connection startup, and the current USD-M
aggregate-trade payload schema.
## Verification output
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
through piker's merged stream received both required stream classes and
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
spot.

View File

@ -1,37 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
timestamp: 2026-08-28T20:03:52Z
git_ref: dd676857
scope: config
substantive: true
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
---
## Prompt
Update the xonsh version used by the repository dev shell before stacking
the Binance work, while retaining the dotrc compatibility shim as an
immediate workaround for older shells.
## Response summary
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
The new release supports Python 3.13, handles Git alias completion natively,
and works in the updated Nix development shell.
## Files changed
- `pyproject.toml` - require xonsh `0.24.2` or newer.
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
the triggering prompt, generated configuration summary, and checks.
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
structured provenance for the dependency update.
## Human edits
The user identified old xonsh behavior as the root cause, reported the
separate dotrc compatibility shim, and directed that the repository update
remain a distinct flake-layer change before the Binance stack continues.

View File

@ -1,36 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-28T20:03:52Z
git_ref: dd676857
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user traced broken multiline conflict-resolution pastes to the old
xonsh release installed by the repository dev shell. A separate dotrc
agent added a compatibility shim, and the user directed that xonsh still
be updated alongside the flake work before continuing the Binance stack.
## Generated configuration
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
refreshes only xonsh's locked distribution metadata. The selected release
supports Python 3.13, includes native Git-config alias completion, and fixes
Nix metadata checks.
## Verification output
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
xonsh from `0.22.4` to `0.24.2`.
- Fresh `nix develop` replaces the worktree-local xonsh installation.
- `xonsh --version` reports `0.24.2`.
- A multi-command xonsh invocation executes Git commands successfully.
- `nix flake check --no-build` passes for `x86_64-linux`.
- `git diff --check` passes.
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
that its distributions are yanked for an `unpivot` regression.

View File

@ -1,46 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
timestamp: 2026-08-29T17:55:42Z
git_ref: e09e306a
scope: code
substantive: true
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
---
## Prompt
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
stack, then add missing deterministic tests and correctness fixes as a
separate follow-up with a complete editor-gated commit plan.
## Response summary
Hardens Binance feed startup around its supported one-symbol contract,
direct websocket routes, current provider schemas, and first-quote
readiness. Adds deterministic Binance regressions and separates generic
reconnect behavior into its own tested commit boundary.
## Files changed
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
USD-M stream classes, and declare readiness after either quote class.
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
and documented book-ticker and aggregate-trade schemas.
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
the triggering prompt, generated-code summary, and verification output.
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
structured provenance for the follow-up.
## Human edits
The user rejected reconstructing already committed Binance work, required
preservation through cherry-pick, selected the flake/xonsh stack base, and
required the review fixes to remain a separate tested boundary. The user also
corrected the workflow to require an automatic `/commit-plan` at every stack
review gate. During Tuicr review, the user rejected trade-gated startup,
requested a feed-specific initialization error and async-generator TODO,
and required generic reconnect changes and tests to land independently.
The user then clarified that either L1 or trade data must unblock startup,
rather than preferring one stream class or waiting independently for both.

View File

@ -1,43 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-29T17:55:42Z
git_ref: e09e306a
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user asked to land recent branch work in small chronological patch sets,
including missing tests. For Binance, the user required preserving the
existing committed topic patch through cherry-pick and placing newly found
correctness repairs in a separate follow-up boundary above the flake/xonsh
stack. The user also required every yielded stack layer to include a complete
`/commit-plan` with patch-specific checks.
## Generated code
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Restricts Binance feeds to the supported one-symbol contract, renders direct
quote routes, separates L1 liveness from first-trade readiness, and derives
the contextual trade timeout from Trio's enclosing effective deadline.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Lets structured cancellation bypass reconnect throttling and ensures handled
handshake rejections retry before accessing connection-local cancel state.
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
Adds deterministic regressions for symbol cardinality, route selection,
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
## Verification output
- Targeted Binance tests: `10 passed in 0.26s`.
- Ruff passes for all three changed Python files.
- `git diff --check` passes.
- Independent final review reports no actionable findings.
- No live Binance network probe was run.

View File

@ -1,42 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
timestamp: 2026-09-01T02:42:35Z
git_ref: e09e306a
scope: code
substantive: true
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
---
## Prompt
Address the Tuicr review by separating generic websocket reconnect
changes from Binance-specific work and testing the real reconnect task
hierarchy rather than a reimplementation.
## Response summary
Preserves structured cancellation and safe first-handshake retries in
`_web_bs`. Adds three focused regressions in a dedicated websocket test
module and packages them as an independent commit boundary.
## Files changed
- `piker/data/_web_bs.py` - restart handled handshake failures and let
cancellation bypass the reconnect throttle.
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
and cancellation task paths with controlled transport boundaries.
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
- retain the review prompt, generated-code summary, and verification.
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
structured provenance for the generic reconnect boundary.
## Human edits
The user identified that these changes were generic rather than
Binance-specific, required a separate commit and test module, requested
coverage of the real reconnect hierarchy, and corrected code prose to
refer to `trio` consistently. A follow-up review required the production
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
handler and cite the regression which audits that teardown invariant.

View File

@ -1,33 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-09-01T02:42:35Z
git_ref: e09e306a
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user requested that generic `_web_bs` reconnect fixes and tests be
separated from the Binance backend patch. The review also required tests
to exercise the real reconnect task hierarchy, cite the previous faulty
branches, and consistently name `trio` in code prose.
## Generated code
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Ensures handled handshake failures immediately restart the reconnect
loop and lets structured cancellation bypass shielded error throttling.
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
Adds deterministic scheduler-real regressions for handshake startup
timeouts, immediate rejection retries, and cancellation propagation.
## Verification output
- Generic and Binance targeted tests: `9 passed in 0.06s`.
- Ruff passes for all four changed Python files.
- `git diff --check` passes.
- No live network probe was run.

View File

@ -1,41 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 4dc920cf-e82e-4c1e-83db-75a0a9efe82b
timestamp: 2026-09-01T23:00:43Z
git_ref: e91f41a6
scope: config
substantive: true
raw_file: 20260901T230043Z_e91f41a6_prompt_io.raw.md
---
## Prompt
Update `wkt/land_flake_update` to Tractor's restored `piker_pin` after
reusing the existing runtime compatibility commits. Keep stale
historical lock snapshots out of the final dependency state.
## Response summary
Updates piker's Python and `msgspec` requirements for current Tractor,
then generates one fresh lock at Tractor `piker_pin@0d95a9e6`. The
dependency commit completes the compatibility stack already
cherry-picked onto the flake branch.
## Files changed
- `pyproject.toml` - require Python 3.13 and `msgspec>=0.20.0`.
- `uv.lock` - resolve Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`,
and current transport dependencies.
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.raw.md`
- preserve the prompt, generated-change summary, and resolution notes.
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md` - record
structured provenance for the dependency update.
## Human edits
The user identified that Tractor compatibility should land with the
flake update rather than the Binance patch, supplied four known source
commits, required a broader local-history audit for missing ports, and
directed a fresh lock from the restored `piker_pin` instead of replaying
historical pins.

View File

@ -1,35 +0,0 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-09-01T23:00:43Z
git_ref: e91f41a6
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user requested that the flake-update branch absorb the Tractor
runtime compatibility work before the Binance branch lands. They
directed reuse of existing compatibility commits, exclusion of stale
historical lock pins, and a fresh update to Tractor's restored
`piker_pin`.
## Generated changes
> `git diff HEAD~1..HEAD -- pyproject.toml`
Narrows piker support to Python 3.13 and raises the direct `msgspec`
requirement to the version family required by current Tractor.
> `git diff --cached -- uv.lock`
Relocks against Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`, and the
current transport dependency graph while retaining the existing Xonsh
update.
## Resolution notes
- Historical lock hunks from reused commits were intentionally omitted.
- Tractor runtime compatibility remains represented by the preceding
individually preserved commits on `wkt/land_flake_update`.
- Project checks are rendered in the commit plan for human execution.

View File

@ -1,52 +0,0 @@
---
created_at: 2026-10-21T22:44:21Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 507c75c6
scope: config
substantive: true
raw_file: 20261021T224421Z_507c75c6_piker_pin_prompt_io.raw.md
---
## Prompt
The human asked to sanity-check the current PR94 landing state against
the qualified new Tractor pin before deduplicating branch history. After
the initial offline checks passed, the human authorized changing the
selector to piker_pin, resolving the lock, and verifying the installed
dependency without the temporary local-source PYTHONPATH override.
The human explicitly authorized network resolution and a scoped update
of Tractor/runtime dependencies while preserving unrelated packages.
## Response summary
Updated the branch selector and lock to published piker_pin@b64c89f0.
Installed the exact locked Tractor revision, verified its installed
source metadata, and reran 14 offline logger/startup cases without a
local Tractor source override. Preserved the rest of the environment.
## Files changed
- `pyproject.toml`: select piker_pin and bound Polars below 2.0.
- `uv.lock`: lock b64c89f0 and Polars/runtime 1.44.2.
- This provenance pair.
## Verification and limits
14 distinct offline cases passed against the installed Tractor update
with the existing Polars 1.36 installation. The actor.uid deprecation
warning remains. No live provider/order testing occurred.
The initial resolver selected Polars 2.0 and rejected retaining 1.36
for its Windows target. The human then requested the latest pre-2.0
release instead. PyPI metadata identified non-yanked stable 1.44.2;
adding `<2.0.0` to the existing lower bound and relocking succeeded
with both polars and polars-runtime-32 at 1.44.2, without platform edits.
The raw partner retains the original, superseded 2.0 generation history.
The scoped install changed only Tractor. Polars 1.44.2 is locked but
not installed or data-path qualified yet. The human's existing staged
checkpoint is preserved; the pre-2.0 correction is in the worktree.
No history cleanup, staging or commit was performed by this agent.

View File

@ -1,45 +0,0 @@
---
created_at: 2026-10-21T22:44:21Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 507c75c6
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
Changed the landing branch's Tractor selector from wkt/addr_unpacking
to piker_pin and resolved the published branch to
b64c89f05a9fc8ac2878551eb5eda36574ebef6a. An offline branch lookup lacked
the cached name, so the human explicitly authorized network resolution
and a scoped installed-runtime update preserving unrelated packages.
The resolver advanced Polars and polars-runtime-32 from 1.36.0 to 2.0.0.
An attempt to retain both exact 1.36.0 versions was rejected for the
resolver's Windows target. The valid lock with the visible Polars bump
is retained for review; no platform restriction or source constraint was
introduced to bypass the failure.
Installed only Tractor and its required dependencies with uv pip rather
than syncing/pruning the whole environment. The installer reported one
package replacement: Tractor a966928d to b64c89f0. Qt distribution versions
remained PyQt6 6.10.1, PyQt6-Qt6 6.10.1 and PyQt6-sip 13.10.3.
Removed PYTHONPATH from verification subprocesses. Piker resolved inside
land_flake_update and Tractor resolved from that checkout's py313
site-packages. Installed VCS metadata matched the exact locked revision.
The temporary LD_LIBRARY_PATH used the binding's bundled Qt libraries
and matching native dependencies; no global environment setup changed.
Verification output:
- Logger/non-actor startup: 12 passed, 1 deselected in 0.05s.
- test_runtime_boot: 1 passed, 1 warning in 0.04s.
- test_daemon_bind_distinct_from_registrar: 1 passed in 0.77s.
- Existing warning: actor.uid deprecation; no failure.
- git diff --check passed.
This verifies the installed Tractor pin, not a full sync to all new lock
versions. Polars was not upgraded in the environment by the scoped install.
No live broker feed, order, branch rewrite, staging or commit was run.

View File

@ -1,67 +0,0 @@
---
created_at: 2026-10-22T21:22:48Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6-astra
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
scope: code
substantive: true
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
---
## Prompt
The human approved trying the bounded candidate: back out the IB
quote-context cache, import adapter-only reset/boundary fixes and
the missing EMS subscription fix, preserve source authorship and
commit boundaries, and exercise the real Tractor bridge. They
required an explicit keep/backout audit and clarification that
the evaluated downstream lineage lacked the broadcast fix.
## Response summary
Verified identical stable patch IDs for original 4af885d9 and
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
in the main checkout. Applied source-equivalent adapter backports
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
source attribution recorded in `ai/ib-recovery-candidate.md`.
The real-runtime quote test first reproduced the reported fatal
TrioTaskExited propagation with caching enabled. The direct-owner
backout restores restart and subsequent RPC survival, but a strict
callback-cleanliness test remains red. Real bridge history pacing
recovery and real broadcaster dark clearing pass. No live broker,
VNC, orders or TSDB operations were performed by the agent.
On 2026-10-23 the human supplied successful live startup/backfill,
sequential reattachment and overlapping chart-close/reopen results,
plus the still-failing daemon SIGINT teardown. The human accepted
deferring teardown from #94, requested loud partial-backport
attribution and required downstream rebase rehearsals before landing.
The agent removed only the explicit single-null debugger stop,
retaining its legacy result. Downstream `976db6c3` instead rewrites
null grouping; the local cleanup does not claim that algorithmic fix.
## Files changed
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
- `piker/clearing/_ems.py`: original EMS subscription patch.
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
- `tests/test_ib_history.py`: original adapter regressions only.
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
## Human edits
The human required original fixes and authorship, rejected broad
speculative integration work, required real concurrency boundaries
in tests, and corrected the interpretation of live qualification.
They performed the live tests, chose to defer known teardown defects
from #94, permitted the minimal debugger cleanup, and requested
explicit downstream discrepancies and rebase compatibility checks.
The agent applied those directed edits; no direct manual source edits
are known in this candidate. The raw partner retains the originally
captured output; this structured record includes later contributions.

View File

@ -1,69 +0,0 @@
---
created_at: 2026-10-22T21:22:48Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6-astra
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
---
## Human direction
> yeah i guess this is our best approach strat for sure.
> so you're saying these 2 are not the same patch?
> please tell me no lol
> so detail which commits to drop vs. keep please in detail so i can
> audit your choices this time..
> let's try it and consider bailing and re-trying if this leads us into
> a further (and deeper) sinkhole, bc as it stands right now this whole
> endevour has turned into a complete shitfest XD
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
## Unedited diagnostic response
The real-bridge test reproduced the exact fatal path:
`maybe_open_context()` teardown raises `TrioTaskExited` back into
the quote consumer.
With only the quote-cache backout, both quote generations complete
and their asyncio tasks finish. Tractor still emits
`TrioTaskExited` callback errors, so the strict test remains red.
That narrows the result: the backout removes the fatal propagation
in this reproducer, but doesn’t make bridge teardown clean.
## Code output pointers
> `git diff dc618229 -- piker/brokers/ib/feed.py`
Back out 635314c7's quote owner and import the original adapter-only
deltas from 0846cbd4 and 1da9b760.
> `git diff dc618229 -- piker/clearing/_ems.py`
Import 12b0d66e's dedicated dark-clearing quote subscription.
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
Replace the reverted mock-cache test with a real-runtime restart
reproducer; retain original history/EMS tests and add real bridge
history recovery and real broadcaster EMS coverage. Newly added
test files remain untracked until the human stages them, so the
diff pointers include them only after staging or committing.
## Verification output
```text
21 passed in 0.53s
1 failed, 2 passed in 5.53s
13 passed in 1.41s
1 passed, 1 warning in 0.04s
2 passed in 0.03s
```
The last run includes the original EMS case again and one additional
real-broadcaster case: 38 unique passes and one strict callback failure.

View File

@ -1,77 +0,0 @@
---
created_at: 2026-10-28T16:08:46Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 7242dcb9
scope: code
substantive: true
raw_file: 20261028T160846Z_7242dcb9_websocket_close.raw.md
---
## Prompt
The human reported repeated live Binance perp crashes after correct
worktree startup: the real trio-websocket reader attempted a Pong
while wsproto was LOCAL_CLOSING. They asked whether an existing
downstream websocket fix should be reused, then requested:
> ok add the fix now, follow by a /commit-plan and then i'll test
> before saving it ya?
They approved a real websocket reader/close regression covering
recovery and intentional shutdown, and separately authorized
reconciling this worktree's legacy workflow bookkeeping while
preserving existing archives.
## Response summary
The older `3d83b61f3f7b982e84a4230dba5777f1407c78f1` wrapper
suppressed escaped communication failures. The landing replay
`97eb8347` of `d50ab6124cd0e1646c82fa4a85d7c13bfff82611`
added propagation; it did not restart a reader-context failure.
Existing `LocalProtocolError` membership in NoBsWs.recon_errors
only protected the receive relay, not reader-nursery context exit.
Classify these errors around the complete websocket context and
retry instead of restoring outer suppression. Place collapse_eg
outside that context, invalidate stale connection readiness and
socket references, and leave the retry wait cancellable. Preserve
unexpected failures and mixed exception groups for propagation.
After review, use RuntimeError rather than ValueError for the
test-only unexpected-failure sentinel. Its injected exception has
no underlying cause. Production recovery logs retain the original
traceback and now name the URL; unexpected exceptions use bare
raise to preserve identity, existing causes and notes.
The deterministic seam controls protocol-state ordering in the real
client Ping handler. It does not replace the websocket transport,
reader, context manager, reconnect task or wsproto exception.
## Files changed
- `piker/data/_web_bs.py`: reader-context failure recovery.
- `tests/test_websocket.py`: loopback recovery and two shutdown
variants; retain unexpected-failure propagation control.
- This provenance pair: live report, lineage and test scope.
## Verification
Baseline uses new tests against original HEAD websocket code in an
isolated exact-tree checkout. The recovery case fails with the
reported Pong-in-LOCAL_CLOSING LocalProtocolError.
Fixed code passes 13 cases across tests/test_websocket.py and
tests/test_binance.py. Exact-boundary verification and native-library
configuration are retained in the ignored commit-plan package.
Live testing remains human-owned and pending.
## Human edits
No direct manual source edits observed in this follow-up. The human
supplied live failures, corrected the earlier daemon/worktree setup,
set the narrow landing scope, requested source-fix reuse, and will
test before reviewing and committing. Existing commits are retained.
The human requested the RuntimeError sentinel and preservation of
underlying error context. The raw partner retains the initial output.

View File

@ -1,36 +0,0 @@
---
created_at: 2026-10-28T16:08:46Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 7242dcb9
diff_cmd: git diff HEAD~1..HEAD
---
Implemented a focused websocket-context reconnect fix and real
loopback reader/close regressions in the Binance landing worktree.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Move exception-group collapse outside the websocket context so
reader-nursery exit errors are classified by the reconnect loop.
Retry NoBsWs.recon_errors there, clear the stale connected event
and socket reference, and honor cancellation through unshielded
retry sleep. Retain unexpected-error propagation at the outer API.
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
Exercise real TCP, handshake, Ping frames, reader dispatch and
wsproto Pong validation. A narrow handler gate advances the real
protocol state to LOCAL_CLOSING without sending the Close frame,
preventing a peer reply from racing the controlled state. Require
fresh application data from the replacement connection. Separate
variants cancel at the reader gate and at the actual retry wait.
Update the existing unexpected-failure control to use ValueError
because LocalProtocolError is now intentionally recoverable.
The isolated original-code/new-test baseline fails with the exact
Pong-in-LOCAL_CLOSING error. The fixed isolated tree passes all 13
selected Binance/websocket cases. No live provider, broker orders
or production daemon operations were performed by the agent.

View File

@ -1,70 +0,0 @@
---
created_at: 2026-10-28T16:56:38Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: b95e7c79
scope: code
substantive: true
raw_file: 20261028T165638Z_b95e7c79_reconnect_logging.raw.md
---
## Prompt
The human reported that the live Binance Pong-in-LOCAL_CLOSING
error now logged a retry without crashing. They requested visible
re-establishment reporting under `pikerd -l info --pdb`, and asked
to check for existing downstream work before adding a fix.
## Response summary
The existing `Connection success` INFO message already identifies
handshake completion. Feed startup configured piker.data.feed,
while shared websocket diagnostics use ._util.log on piker.data.
Configure that actual emitter and prefer the daemon's verbosity
over a quieter client request for these shared transport tasks.
Add a readiness INFO message after fixture setup and a first-message
INFO report once per connection after actual relay. Include the URL
in the receive-timeout reset error. Do not call first raw websocket
traffic a parsed quote or proof of restored market-data freshness.
## Existing-work audit
Compared `_web_bs.py`, `_util.py`, `piker.log` and daemon logging
against the evaluated downstream tip and relevant all-ref histories.
`54da2973` contains logging/type cleanup, not this emitter fix.
`5adf77a9` / landing equivalents address package-name correctness,
but not configuration of the shared data emitter at feed startup.
All-ref searches for this actual emitter/readiness/first-relay
implementation found no existing commit. The dirty IB history
worktree has only an unrelated f-string correction in `_web_bs.py`;
the TIPC worktree has no dirty delta in these two feed/websocket
files. No downstream work was overwritten or imported wholesale.
## Files changed
- `piker/data/feed.py`: shared transport logger startup/verbosity.
- `piker/data/_web_bs.py`: readiness, first-relay and timeout URL.
- `tests/test_logging.py`: actual emitter/console startup regression.
- `tests/test_websocket.py`: recovery milestones on real loopback.
- This prompt provenance pair.
## Verification
The new logging regression against original b95e7c79 source fails:
the actual websocket emitter remains at WARNING and rejects INFO.
It passes with the logger wiring fix. Provider lookup is stopped
before network work, while logger factories and console emission
are real. Global logger state is restored after the check.
18 cases pass across tests/test_logging.py, tests/test_websocket.py
and tests/test_binance.py. git diff --check passes. Live verification
after restarting the daemon with these edits remains human-owned.
## Human edits
No direct manual source edits observed. The human supplied live
recovery evidence, selected INFO verbosity, requested downstream
reuse inspection, and retains live-test and commit ownership.

View File

@ -1,38 +0,0 @@
---
created_at: 2026-10-28T16:56:38Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: b95e7c79
diff_cmd: git diff HEAD~1..HEAD
---
Confirmed the existing INFO handshake-success log was hidden by
logger wiring: websocket code emits on piker.data, while feed
startup configured piker.data.feed. The baseline regression leaves
the emitter at WARNING for an INFO daemon and ERROR client.
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
Configure the actual shared ._util.log namespace, with daemon
verbosity taking priority over client verbosity for actor-wide
transport diagnostics.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Keep the existing handshake-success message. Add INFO readiness
after fixture setup and first-message relay once per connection.
Include the URL in the receive-timeout reset diagnostic.
> `git diff HEAD~1..HEAD -- tests/test_logging.py tests/test_websocket.py`
Prove production startup enables real console INFO on the websocket
emitter without contacting a provider. Extend the real loopback
recovery test to require readiness for both connections and one
first-message report for the connection delivering fresh data.
All 18 targeted logging/Binance/websocket cases passed. No live
provider/daemon operations were performed by the agent. First relay
is raw websocket traffic; parsed market-data recovery remains a
separate live observation.

View File

@ -1,54 +0,0 @@
---
created_at: 2026-10-28T18:08:56Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 032580bf
scope: code
substantive: true
raw_file: 20261028T180856Z_032580bf_timeout_phase.raw.md
---
## Prompt
The human reported repeated readiness without first-message relay,
specifically after first attempting an invalid .usdm. FQME and then
the corrected .usdtm. one. They requested distinct log messages for
receive starvation versus local backpressure.
## Response summary
Add phase-specific WARNING messages for each elapsed relay-loop
deadline, with URL, timeout duration, accumulated count and reset
threshold. Leave the existing shared receive/send deadline and
counter/reset semantics intact. Do not claim that connected/ready
sockets alone establish parsed quote recovery.
Read-only invalid-venue tracing found the usd substring check in
Binance get_mkt_info() accepts USDM while retaining it in MktPair.
Client.bars() later uses that market identity against canonical
USDTM keys. Backend stream startup precedes history startup and
sampler consumption. A relationship to the corrected-feed stall
still needs runtime evidence; neither venue handling nor task
lifetime behavior was changed by this diagnostic patch.
## Files changed
- `piker/data/_web_bs.py`: track and report the timed-out operation.
- `tests/test_websocket.py`: real silent-source and blocked-sink cases.
- This prompt provenance pair.
## Verification
15 selected Binance/websocket cases passed. The new cases exercise
real TCP, websocket handshake and relay behavior, replacing only
log observation. A zero-capacity relay deliberately forces the
local-send case; the source-silent case never receives a payload.
git diff --check passed. No live provider/daemon operations run.
## Human edits
No direct source edits observed. The human supplied the critical
invalid-symbol-then-corrected-symbol trigger and requested the
phase distinction; live testing and task acceptance remain theirs.

View File

@ -1,34 +0,0 @@
---
created_at: 2026-10-28T18:08:56Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 032580bf
diff_cmd: git diff HEAD~1..HEAD
---
Added distinct timeout diagnostics for websocket receive starvation
versus local relay backpressure, preserving the existing timeout,
counter and reconnect threshold policy.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Track whether proxy_msgs() is awaiting ws.get_message() or
snd.send(). Report that phase, URL, configured deadline, timeout
count and reset threshold for every timeout, before the existing
threshold/reset diagnostic. Capture the configured deadline once
per operation so the report matches the deadline that elapsed.
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
Two real loopback variants use a silent server versus one payload
and an unconsumed zero-capacity Trio relay. Require the appropriate
phase and URL on the actual first timeout. All 15 selected
Binance/websocket cases pass. No live provider was contacted.
Read-only tracing found a permissive invalid-venue path: get_mkt_info
accepts USDM via its usd substring check and preserves that venue
in MktPair, whereas Client.bars() indexes canonical USDTM pair keys.
This identifies a potential late failed-startup path, not proof of
the corrected-chart stall or of leaked producer ownership.

View File

@ -1,29 +1,8 @@
[network] [network]
# This Tractor pin supports one transport per actor, not mixed TCP/UDS.
# Use the single TCP endpoint shown in each list for this version.
# Mixed-transport configs need a future Tractor pin with that support.
pikerd = [ pikerd = [
# TCP localhost loopback '/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
'/ip4/127.0.0.1/tcp/6116', # '/uds/6116', # TODO std uds socket file
# UDS alternative: replace TCP and select the UDS transport.
# '/unix/run/user/1000/piker/pikerd.sock',
] ]
chart = [
'/ip4/127.0.0.1/tcp/3003',
# '/unix/run/user/1000/piker/chart.sock',
]
# the service-actor registry endpoint;
# other pikerd trees contact this to discover
# actors.
# XXX if absent, pikerd binds the registry
# on its own tpt_bind_addrs.
# regd = ['/ip4/127.0.0.1/tcp/6116']
# chart = [
# '/ip4/127.0.0.1/tcp/3333',
# '/unix/run/user/1000/piker/chart@3333.sock',
# ]
[ui] [ui]

View File

@ -2,11 +2,11 @@
"nodes": { "nodes": {
"nixpkgs": { "nixpkgs": {
"locked": { "locked": {
"lastModified": 1784497964, "lastModified": 1765779637,
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=", "narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
"owner": "nixos", "owner": "nixos",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163", "rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
"type": "github" "type": "github"
}, },
"original": { "original": {

View File

@ -22,14 +22,12 @@
# do store-path extractions # do store-path extractions
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase; qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
zlibStorePath = lib.getLib pkgs.zlib;
# ?TODO? can remove below since manual linking not needed? # ?TODO? can remove below since manual linking not needed?
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland; # qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
# XXX NOTE XXX, for now we overlay specific pkgs via # XXX NOTE XXX, for now we overlay specific pkgs via
# a major-version-pinned-`cpython` # a major-version-pinned-`cpython`
cpython = "python313"; cpython = "python313";
python = pkgs.${cpython};
pypkgs = pkgs."${cpython}Packages"; pypkgs = pkgs."${cpython}Packages";
in in
{ {
@ -48,7 +46,7 @@
qt6.qtbase qt6.qtbase
uv uv
python python313 # ?TODO^ how to set from `cpython` above?
pypkgs.pyqt6 pypkgs.pyqt6
pypkgs.pyqt6-sip pypkgs.pyqt6-sip
pypkgs.qtpy pypkgs.qtpy
@ -72,7 +70,6 @@
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.) # link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH" LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH export LD_LIBRARY_PATH
@ -89,31 +86,15 @@
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell" export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
# ------ uv ------ # ------ uv ------
# ignore any venv inherited from an outer checkout
unset VIRTUAL_ENV
# - always use the ./py313/ venv-subdir # - always use the ./py313/ venv-subdir
export UV_PROJECT_ENVIRONMENT="py313" export UV_PROJECT_ENVIRONMENT="py313"
# Rebuild after a flake update changes the Nix Python closure.
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
nix_python="$(readlink -f "${python}/bin/python3")"
if test -e "$venv_python" \
&& test "$(readlink -f "$venv_python")" != "$nix_python"
then
uv venv \
--clear \
--python "$nix_python" \
"$UV_PROJECT_ENVIRONMENT"
fi
# sync project-env with all extras # sync project-env with all extras
uv sync --dev --all-extras --no-group lint uv sync --dev --all-extras --no-group lint
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
# ------ TIPS ------ # ------ TIPS ------
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy) # NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
# run the `nix develop` cmd with, # run the `nix develop` cmd with,
# >> nix develop -c xonsh # >> nix develop -c uv run xonsh
''; '';
}; };
} }

View File

@ -324,13 +324,10 @@ def load_ledger(
ldir: Path = ( ldir: Path = (
dirpath dirpath
or or
config.get_conf_dir() / 'accounting' / 'ledgers' config._config_dir / 'accounting' / 'ledgers'
) )
if not ldir.is_dir(): if not ldir.is_dir():
ldir.mkdir( ldir.mkdir()
parents=True,
exist_ok=True,
)
fname = f'trades_{brokername}_{acctid}.toml' fname = f'trades_{brokername}_{acctid}.toml'
fpath: Path = ldir / fname fpath: Path = ldir / fname

View File

@ -785,16 +785,9 @@ def load_account(
legacy_fn: str = f'pps.{brokername}.{acctid}.toml' legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
fn: str = f'account.{brokername}.{acctid}.toml' fn: str = f'account.{brokername}.{acctid}.toml'
dirpath: Path = ( dirpath: Path = dirpath or (config._config_dir / 'accounting')
dirpath
or
(config.get_conf_dir() / 'accounting')
)
if not dirpath.is_dir(): if not dirpath.is_dir():
dirpath.mkdir( dirpath.mkdir()
parents=True,
exist_ok=True,
)
conf, path = config.load( conf, path = config.load(
path=dirpath / fn, path=dirpath / fn,

View File

@ -143,15 +143,12 @@ def sync(
# (what the EMS normally does internall) B) # (what the EMS normally does internall) B)
open_brokerd_dialog( open_brokerd_dialog(
brokermod, brokermod,
portal,
exec_mode=( exec_mode=(
'paper' 'paper'
if account == 'paper' if account == 'paper'
else 'live' else 'live'
), ),
# use our own ad-hoc-spawned actor,
# do NOT (spawn and) use the
# `brokerd.<broker>` service daemon!
portal=portal,
loglevel=loglevel, loglevel=loglevel,
) as ( ) as (
brokerd_stream, brokerd_stream,

View File

@ -25,8 +25,10 @@ from contextlib import (
) )
from types import ModuleType from types import ModuleType
from typing import ( from typing import (
TYPE_CHECKING,
AsyncContextManager, AsyncContextManager,
) )
import exceptiongroup as eg
import tractor import tractor
import trio import trio
@ -38,20 +40,27 @@ from piker.log import (
from . import _util from . import _util
from . import get_brokermod from . import get_brokermod
if TYPE_CHECKING:
from ..data import _FeedsBus
log = get_logger(name=__name__) log = get_logger(name=__name__)
# `brokerd`-actor-always-enabled mods. # `brokerd` enabled modules
# TODO: move this def to the `.data` subpkg..
# NOTE: keeping this list as small as possible is part of our caps-sec # NOTE: keeping this list as small as possible is part of our caps-sec
# model and should be treated with utmost care! In particular NO # model and should be treated with utmost care!
# `piker.data.*` feed mods should be enabled in this (live, _data_mods: str = [
# credentialed) trading actor; all data-feed serving is the 'piker.brokers.core',
# domain of the `datad.<broker>` sibling daemon, see 'piker.brokers.data',
# `piker.data._daemon._datad_service_mods`.
_brokerd_service_mods: list[str] = [
'piker.brokers._daemon', 'piker.brokers._daemon',
'piker.data',
'piker.data.feed',
'piker.data._sampling'
] ]
# TODO: we should rename the daemon to datad prolly once we split up
# broker vs. data tasks into separate actors?
@tractor.context @tractor.context
async def _setup_persistent_brokerd( async def _setup_persistent_brokerd(
ctx: tractor.Context, ctx: tractor.Context,
@ -60,15 +69,9 @@ async def _setup_persistent_brokerd(
) -> None: ) -> None:
''' '''
Trading-only daemon (lifetime) fixture: console logging Allocate a actor-wide service nursery in ``brokerd``
setup and a pinned-open context for service mgmt. such that feeds can be run in the background persistently by
the broker backend as needed.
All data-feed-bus state now lives in the (data-feed-only)
`datad.<brokername>` sibling daemon, see
`piker.data._daemon._setup_persistent_datad()`; this
actor hosts only the backend's `open_trade_dialog()`
(live order-control) ep-task(s) which manage their own
task trees per `tractor.Context`.
''' '''
# NOTE: we only need to setup logging once (and only) here # NOTE: we only need to setup logging once (and only) here
@ -77,18 +80,53 @@ async def _setup_persistent_brokerd(
# any further (level) configuration on their own B) # any further (level) configuration on their own B)
actor: tractor.Actor = tractor.current_actor() actor: tractor.Actor = tractor.current_actor()
tll: str = actor.loglevel tll: str = actor.loglevel
get_console_log( log = get_console_log(
level=loglevel or tll, level=loglevel or tll,
name=f'{_util.subsys}.{brokername}', name=f'{_util.subsys}.{brokername}',
with_tractor_log=bool(tll), with_tractor_log=bool(tll),
) )
assert log.name == _util.subsys
# unblock caller # further, set the log level on any broker broker specific
await ctx.started() # logger instance.
# we pin this task to keep the daemon active until the from piker.data import feed
# parent actor decides to tear it down assert not feed._bus
await trio.sleep_forever()
# allocate a nursery to the bus for spawning background
# tasks to service client IPC requests, normally
# `tractor.Context` connections to explicitly required
# `brokerd` endpoints such as:
# - `stream_quotes()`,
# - `manage_history()`,
# - `allocate_persistent_feed()`,
# - `open_symbol_search()`
# NOTE: see ep invocation details inside `.data.feed`.
try:
async with (
# tractor.trionics.collapse_eg(),
trio.open_nursery() as service_nursery
):
bus: _FeedsBus = feed.get_feed_bus(
brokername,
service_nursery,
)
assert bus is feed._bus
# unblock caller
await ctx.started()
# we pin this task to keep the feeds manager active until the
# parent actor decides to tear it down
await trio.sleep_forever()
except eg.ExceptionGroup:
# TODO: likely some underlying `brokerd` IPC connection
# broke so here we handle a respawn and re-connect attempt!
# This likely should pair with development of the OCO task
# nusery in dev over @ `tractor` B)
# https://github.com/goodboy/tractor/pull/363
raise
def broker_init( def broker_init(
@ -109,10 +147,8 @@ def broker_init(
This includes: This includes:
- load the appropriate <brokername>.py pkg module, - load the appropriate <brokername>.py pkg module,
- reads any declared `_brokerd_mods: list[str]` (falling - reads any declared `__enable_modules__: listr[str]` which will be
back to the full `__enable_modules__` set for passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
not-yet-split backends) which will be passed to
`tractor.ActorNursery.start_actor(enable_modules=)`
at actor start time, at actor start time,
- deliver a references to the daemon lifetime fixture, which - deliver a references to the daemon lifetime fixture, which
for now is always the `_setup_persistent_brokerd()` context defined for now is always the `_setup_persistent_brokerd()` context defined
@ -147,14 +183,8 @@ def broker_init(
] ]
for submodname in getattr( for submodname in getattr(
brokermod, brokermod,
'_brokerd_mods', '__enable_modules__',
# fallback for (flat, less mature) backends which [],
# don't yet declare a daemon-kind mod split.
getattr(
brokermod,
'__enable_modules__',
[],
),
): ):
subpath: str = f'{modpath}.{submodname}' subpath: str = f'{modpath}.{submodname}'
enabled.append(subpath) enabled.append(subpath)
@ -182,22 +212,6 @@ async def spawn_brokerd(
f'backend: {brokername!r}' f'backend: {brokername!r}'
) )
# fail fast on (data-only) backends which don't offer
# ANY live order-control eps; the caller should instead
# be using paper-mode (and thus never spawning us)!
from ..data.validate import get_eps
brokerd_eps: dict = get_eps(
get_brokermod(brokername),
'brokerd',
)
if not brokerd_eps:
raise RuntimeError(
f'Backend {brokername!r} offers NO `brokerd` '
f'(live order-control) eps!?\n'
f'It is likely a datad-only provider, use '
f'paper-mode for clearing instead.\n'
)
( (
brokermode, brokermode,
tractor_kwargs, tractor_kwargs,
@ -219,11 +233,7 @@ async def spawn_brokerd(
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}' dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
portal = await Services.actor_n.start_actor( portal = await Services.actor_n.start_actor(
dname, dname,
enable_modules=list(dict.fromkeys( enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
_brokerd_service_mods
+
tractor_kwargs.pop('enable_modules')
)),
debug_mode=Services.debug_mode, debug_mode=Services.debug_mode,
**tractor_kwargs **tractor_kwargs
) )

View File

@ -20,17 +20,10 @@ Handy cross-broker utils.
""" """
from __future__ import annotations from __future__ import annotations
# from functools import partial # from functools import partial
from typing import (
Type,
)
import json import json
import httpx import httpx
import logging import logging
from msgspec import Struct
from tractor._exceptions import (
reg_err_types,
)
from piker.log import ( from piker.log import (
colorize_json, colorize_json,
@ -104,19 +97,6 @@ class DataThrottle(BrokerError):
''' '''
# TODO: add in throttle metrics/feedback # TODO: add in throttle metrics/feedback
class SchemaMismatch(BrokerError):
'''
Market `Pair` fields mismatch, likely due to provider API update.
'''
# auto-register all `BrokerError` subtypes for
# tractor IPC exc-marshalling.
reg_err_types([
BrokerError,
*BrokerError.__subclasses__(),
])
def resproc( def resproc(
resp: httpx.Response, resp: httpx.Response,
@ -143,45 +123,3 @@ def resproc(
log.debug(f"Received json contents:\n{colorize_json(msg)}") log.debug(f"Received json contents:\n{colorize_json(msg)}")
return msg if return_json else resp return msg if return_json else resp
def get_or_raise_on_pair_schema_mismatch(
pair_type: Type[Struct],
fields_data: dict,
provider_name: str,
api_url: str|None = None,
) -> Struct:
'''
Boilerplate helper around assset-`Pair` field schema mismatches,
normally due to provider API updates.
'''
try:
pair: Struct = pair_type(**fields_data)
return pair
except TypeError as err:
from tractor.devx.pformat import ppfmt
repr_data: str = ppfmt(fields_data)
report: str = (
f'Field mismatch we need to codify!\n'
f'\n'
f'{pair_type!r}({repr_data})'
f'\n'
f'^^^ {err.args[0]!r} ^^^\n'
f'\n'
f"Don't panic, prolly {provider_name!r} "
f"changed their symbology schema..\n"
)
if (
api_url
or
(api_url := pair_type._api_url)
):
report += (
f'\n'
f'Check out their API docs here:\n'
f'{api_url}\n'
)
raise SchemaMismatch(report) from err

View File

@ -52,25 +52,9 @@ __all__ = [
] ]
# per-daemon-kind (sub)mod groups: declares which of our # `brokerd` modules
# submods host the eps run by each daemon-actor kind as __enable_modules__: list[str] = [
# defined by `piker.data.validate._eps`.
# NOTE: `get_mkt_info` and `open_symbol_search` both live
# in `.feed` for this backend (no `symbols.py`).
_brokerd_mods: list[str] = [
'api',
'broker',
]
_datad_mods: list[str] = [
'api', 'api',
'feed', 'feed',
'broker',
] ]
# tractor RPC enable arg
__enable_modules__: list[str] = list(dict.fromkeys(
_brokerd_mods
+
_datad_mods
))

View File

@ -49,9 +49,6 @@ from piker import config
from piker.clearing._messages import ( from piker.clearing._messages import (
Order, Order,
) )
from piker.brokers._util import (
get_or_raise_on_pair_schema_mismatch,
)
from piker.accounting import ( from piker.accounting import (
Asset, Asset,
digits_to_dec, digits_to_dec,
@ -373,12 +370,20 @@ class Client:
item['filters'] = filters item['filters'] = filters
pair_type: Type = PAIRTYPES[venue] pair_type: Type = PAIRTYPES[venue]
pair: Pair = get_or_raise_on_pair_schema_mismatch( try:
pair_type=pair_type, pair: Pair = pair_type(**item)
fields_data=item, except Exception as e:
provider_name='binance', e.add_note(
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information', f'\n'
) f'New or removed field we need to codify!\n'
f'pair-type: {pair_type!r}\n'
f'\n'
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
f'Check out their API docs here:\n'
f'\n'
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
)
raise
pair_table[pair.symbol.upper()] = pair pair_table[pair.symbol.upper()] = pair
# update an additional top-level-cross-venue-table # update an additional top-level-cross-venue-table
@ -576,8 +581,8 @@ class Client:
self, self,
mkt: MktPair, mkt: MktPair,
start_dt: datetime|None = None, start_dt: datetime | None = None,
end_dt: datetime|None = None, end_dt: datetime | None = None,
as_np: bool = True, as_np: bool = True,
@ -604,11 +609,7 @@ class Client:
start_time = binance_timestamp(start_dt) start_time = binance_timestamp(start_dt)
end_time = binance_timestamp(end_dt) end_time = binance_timestamp(end_dt)
import tractor bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
with tractor.devx.maybe_open_crash_handler():
bs_pair: Pair = self._pairs[
mkt.bs_fqme.upper()
]
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data # https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
bars = await self._api( bars = await self._api(

View File

@ -22,14 +22,18 @@ from __future__ import annotations
from contextlib import ( from contextlib import (
asynccontextmanager as acm, asynccontextmanager as acm,
aclosing, aclosing,
AsyncExitStack,
) )
from datetime import datetime from datetime import datetime
from functools import (
partial,
)
import itertools
from pprint import pformat from pprint import pformat
from typing import ( from typing import (
Any, Any,
AsyncGenerator, AsyncGenerator,
Callable, Callable,
Generator,
) )
import time import time
@ -44,7 +48,6 @@ import tractor
from piker.brokers import ( from piker.brokers import (
open_cached_client, open_cached_client,
NoData, NoData,
SymbolNotFound,
) )
from piker._cacheables import ( from piker._cacheables import (
async_lifo_cache, async_lifo_cache,
@ -56,10 +59,7 @@ from piker.accounting import (
unpack_fqme, unpack_fqme,
) )
from piker.types import Struct from piker.types import Struct
from piker.data.validate import ( from piker.data.validate import FeedInit
FeedInit,
FeedInitializationError,
)
from piker.data._web_bs import ( from piker.data._web_bs import (
open_autorecon_ws, open_autorecon_ws,
NoBsWs, NoBsWs,
@ -76,15 +76,13 @@ from .venues import (
Pair, Pair,
FutesPair, FutesPair,
get_api_eps, get_api_eps,
_futes_market_ws,
_futes_public_ws,
) )
log = get_logger(name=__name__) log = get_logger(name=__name__)
class L1(Struct): class L1(Struct):
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Individual-Symbol-Book-Ticker-Streams # https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
update_id: int update_id: int
sym: str sym: str
@ -109,15 +107,14 @@ class AggTrade(Struct, frozen=True):
T: int # Trade time T: int # Trade time
m: bool # Is the buyer the market maker? m: bool # Is the buyer the market maker?
M: bool|None = None # Ignore M: bool|None = None # Ignore
nq: float|None = None # Normal quantity without RPI-order trades nq: float|None = None # Normal quantity without the trades involving RPI orders
# ^XXX: https://developers.binance.com/docs/derivatives/change-log#2025-12-29 # ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
st: int|None = None # 1 for USD-M, 2 for COIN-M
async def stream_messages( async def stream_messages(
ws: NoBsWs, ws: NoBsWs,
) -> AsyncGenerator[tuple[str, dict], None]: ) -> AsyncGenerator[NoBsWs, dict]:
# TODO: match syntax here! # TODO: match syntax here!
msg: dict[str, Any] msg: dict[str, Any]
@ -206,6 +203,27 @@ async def stream_messages(
yield 'trade', piker_quote yield 'trade', piker_quote
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
'''
Create a request subscription packet dict.
- spot:
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
- futes:
https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
'''
return {
'method': 'SUBSCRIBE',
'params': [
f'{pair.lower()}@{sub_name}'
for pair in pairs
],
'id': uid
}
# TODO, why aren't frame resp `log.info()`s showing in upstream # TODO, why aren't frame resp `log.info()`s showing in upstream
# code?! # code?!
@acm @acm
@ -283,10 +301,6 @@ async def get_mkt_info(
# uppercase since kraken bs_mktid is always upper # uppercase since kraken bs_mktid is always upper
if 'binance' not in fqme.lower(): if 'binance' not in fqme.lower():
log.warning(
f'Missing `.<provider>` part in fqme ??\n'
f'fqme: {fqme!r}\n'
)
fqme += '.binance' fqme += '.binance'
mkt_mode: str = '' mkt_mode: str = ''
@ -301,12 +315,6 @@ async def get_mkt_info(
venue: str = venue.upper() venue: str = venue.upper()
venue_lower: str = venue.lower() venue_lower: str = venue.lower()
if not venue:
raise SymbolNotFound(
f'Invalid or missing .<venue> part in fqme?\n'
f'fqme: {fqme!r}\n'
)
# XXX TODO: we should change the usdtm_futes name to just # XXX TODO: we should change the usdtm_futes name to just
# usdm_futes (dropping the tether part) since it turns out that # usdm_futes (dropping the tether part) since it turns out that
# there are indeed USD-tokens OTHER THEN tether being used as # there are indeed USD-tokens OTHER THEN tether being used as
@ -398,153 +406,54 @@ async def get_mkt_info(
return mkt, pair return mkt, pair
def get_quote_routes(
mkt_mode: str,
symbols: list[str],
) -> tuple[str, ...]:
'''
Render websocket URLs for each required Binance stream class.
'''
if len(symbols) != 1:
raise ValueError(
'Binance quote routes require exactly one symbol; '
f'got {symbols!r}'
)
symbol: str = symbols[0]
sub_names: tuple[str, str] = ('bookTicker', 'aggTrade')
if mkt_mode == 'usdtm_futes':
return tuple(
f'{wss_url}/{symbol.lower()}@{sub_name}'
for wss_url, sub_name in (
(_futes_public_ws, sub_names[0]),
(_futes_market_ws, sub_names[1]),
)
)
wss_url: str = get_api_eps(mkt_mode)[1]
return tuple(
f'{wss_url}/{symbol.lower()}@{sub_name}'
for sub_name in sub_names
)
async def relay_messages(
ws: NoBsWs,
send_chan: trio.MemorySendChannel,
) -> None:
'''
Relay one Binance websocket into a merged quote channel.
'''
async with (
send_chan,
aclosing(stream_messages(ws)) as msg_gen,
):
async for msg in msg_gen:
await send_chan.send(msg)
# TODO: Encapsulate ``relay_messages()`` and ``open_quote_stream()``
# inside ``stream_quotes()`` so async-generator closure is owned by
# the backend feed task.
@acm @acm
async def open_quote_stream( async def subscribe(
mkt_mode: str, ws: NoBsWs,
symbols: list[str], symbols: list[str],
connect_timeout: float = 16,
) -> AsyncGenerator[trio.MemoryReceiveChannel, None]:
'''
Open and merge the websocket routes required by a market.
''' # defined once at import time to keep a global state B)
quote_routes: tuple[str, ...] = get_quote_routes( iter_subids: Generator[int, None, None] = itertools.count(),
mkt_mode=mkt_mode,
symbols=symbols,
)
send, recv = trio.open_memory_channel(616) ):
async with AsyncExitStack() as stack: # setup subs
websockets: list[NoBsWs] = []
try:
for wss_url in quote_routes:
ws: NoBsWs = await stack.enter_async_context(
open_autorecon_ws(
url=wss_url,
connect_timeout=connect_timeout,
)
)
websockets.append(ws)
except ( subid: int = next(iter_subids)
TimeoutError,
*NoBsWs.recon_errors,
) as err:
routes_str: str = '\n'.join(quote_routes)
raise DataUnavailable(
f'Binance websocket connection failed for '
f'{symbols!r}:\n'
f'{routes_str}'
) from err
async with trio.open_nursery() as tn: # trade data (aka L1)
for ws in websockets: # https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
tn.start_soon( l1_sub = make_sub(symbols, 'bookTicker', subid)
relay_messages, await ws.send_msg(l1_sub)
ws,
send.clone(),
)
await send.aclose() # aggregate (each order clear by taker **not** by maker)
async with recv: # trades data:
try: # https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
yield recv agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
finally: await ws.send_msg(agg_trades_sub)
tn.cancel_scope.cancel()
# might get ack from ws server, or maybe some
# other msg still in transit..
res = await ws.recv_msg()
subid: str|None = res.get('id')
if subid:
assert res['id'] == subid
async def wait_for_live_quotes( yield
msg_stream: AsyncGenerator | trio.MemoryReceiveChannel,
symbols: list[str],
startup_timeout: float = 16,
) -> dict:
'''
Require L1 or trade data before declaring a feed live.
''' subs = []
if len(symbols) != 1: for sym in symbols:
raise ValueError( subs.append("{sym}@aggTrade")
'Binance quote readiness requires exactly one symbol; ' subs.append("{sym}@bookTicker")
f'got {symbols!r}'
)
symbol: str = symbols[0] # unsub from all pairs on teardown
msg_iter = aiter(msg_stream) if ws.connected():
await ws.send_msg({
"method": "UNSUBSCRIBE",
"params": subs,
"id": subid,
})
try: # XXX: do we need to ack the unsub?
with trio.fail_after(startup_timeout): # await ws.recv_msg()
while True:
typ, quote = await anext(msg_iter)
if (
typ in {'l1', 'trade'}
and
quote['symbol'] == symbol
):
return quote
except trio.TooSlowError as err:
raise DataUnavailable(
f'Binance quote startup timed out after '
f'{startup_timeout}s; '
f'no live data received for {symbol}'
) from err
except StopAsyncIteration as err:
raise DataUnavailable(
f'Binance quote streams closed during startup; '
f'no live data received for {symbol}'
) from err
async def stream_quotes( async def stream_quotes(
@ -556,46 +465,49 @@ async def stream_quotes(
# startup sync # startup sync
task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED, task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
) -> None: ) -> None:
if len(symbols) != 1:
# ``data.feed.allocate_persistent_feed()`` currently starts
# one backend task per symbol. Binance multiplexing remains
# deferred until that caller contract can represent
# multi-symbol startup.
raise FeedInitializationError(
'Binance feeds require exactly one symbol; '
f'got {symbols!r}'
)
async with ( async with (
tractor.trionics.maybe_raise_from_masking_exc(), tractor.trionics.maybe_raise_from_masking_exc(),
send_chan as send_chan, send_chan as send_chan,
open_cached_client('binance') as client,
): ):
mkt: MktPair init_msgs: list[FeedInit] = []
pair: Pair for sym in symbols:
mkt, pair = await get_mkt_info(symbols[0]) mkt: MktPair
mkt_mode: str = ( pair: Pair
'usdtm_futes' mkt, pair = await get_mkt_info(sym)
if isinstance(pair, FutesPair)
else 'spot'
)
# build out init msgs according to latest spec # build out init msgs according to latest spec
init_msgs: list[FeedInit] = [ init_msgs.append(
FeedInit(mkt_info=mkt), FeedInit(mkt_info=mkt)
]
ws_symbols: list[str] = [mkt.bs_mktid]
async with open_quote_stream(
mkt_mode=mkt_mode,
symbols=ws_symbols,
) as msg_stream:
quote: dict = await wait_for_live_quotes(
msg_stream=msg_stream,
symbols=ws_symbols,
) )
wss_url: str = get_api_eps(client.mkt_mode)[1] # 2nd elem is wss url
# TODO: for sanity, but remove eventually Xp
if 'future' in mkt.type_key:
assert 'fstream' in wss_url
async with (
open_autorecon_ws(
url=wss_url,
fixture=partial(
subscribe,
symbols=[mkt.bs_mktid],
),
) as ws,
# avoid stream-gen closure from breaking trio..
aclosing(stream_messages(ws)) as msg_gen,
):
# log.info('WAITING ON FIRST LIVE QUOTE..')
typ, quote = await anext(msg_gen)
# pull a first quote and deliver
while typ != 'trade':
typ, quote = await anext(msg_gen)
task_status.started((init_msgs, quote)) task_status.started((init_msgs, quote))
# signal to caller feed is ready for consumption # signal to caller feed is ready for consumption
@ -611,7 +523,7 @@ async def stream_quotes(
topic: str = mkt.bs_fqme topic: str = mkt.bs_fqme
# start streaming # start streaming
async for typ, quote in msg_stream: async for typ, quote in msg_gen:
# period = time.time() - last # period = time.time() - last
# hz = 1/period if period else float('inf') # hz = 1/period if period else float('inf')
# if hz > 60: # if hz > 60:

View File

@ -20,7 +20,6 @@ Per market data-type definitions and schemas types.
""" """
from __future__ import annotations from __future__ import annotations
from typing import ( from typing import (
ClassVar,
Literal, Literal,
) )
from decimal import Decimal from decimal import Decimal
@ -42,10 +41,7 @@ _spot_ws: str = 'wss://stream.binance.com/ws'
# or this one? .. # or this one? ..
# 'wss://ws-api.binance.com:443/ws-api/v3', # 'wss://ws-api.binance.com:443/ws-api/v3',
# https://developers.binance.com/docs/derivatives/usds-margined-futures/ # https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
# websocket-market-streams/Important-WebSocket-Change-Notice
_futes_public_ws: str = f'wss://fstream.{_domain}/public/ws'
_futes_market_ws: str = f'wss://fstream.{_domain}/market/ws'
_futes_ws: str = f'wss://fstream.{_domain}/ws' _futes_ws: str = f'wss://fstream.{_domain}/ws'
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws' _auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
@ -82,7 +78,7 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
), ),
'usdtm_futes': ( 'usdtm_futes': (
_futes_url, _futes_url,
_futes_market_ws, _futes_ws,
), ),
}[venue] }[venue]
@ -207,8 +203,6 @@ class FutesPair(Pair):
# NOTE: see `.data._symcache.SymbologyCache.load()` for why # NOTE: see `.data._symcache.SymbologyCache.load()` for why
ns_path: str = 'piker.brokers.binance:FutesPair' ns_path: str = 'piker.brokers.binance:FutesPair'
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
# NOTE: for compat with spot pairs and `MktPair.src: Asset` # NOTE: for compat with spot pairs and `MktPair.src: Asset`
# processing.. # processing..
@property @property

View File

@ -35,7 +35,7 @@ from piker.log import (
get_logger, get_logger,
) )
from ..service import ( from ..service import (
maybe_spawn_datad, maybe_spawn_brokerd,
maybe_open_pikerd, maybe_open_pikerd,
) )
from ..brokers import ( from ..brokers import (
@ -187,7 +187,7 @@ def brokercheck(config, broker):
''' '''
async def bcheck_main(): async def bcheck_main():
async with maybe_spawn_datad(broker) as portal: async with maybe_spawn_brokerd(broker) as portal:
await portal.run(run_test, broker) await portal.run(run_test, broker)
await portal.cancel_actor() await portal.cancel_actor()
@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
return return
async def main(tries): async def main(tries):
async with maybe_spawn_datad( async with maybe_spawn_brokerd(
tries=tries, loglevel=loglevel tries=tries, loglevel=loglevel
) as portal: ) as portal:
# run app "main" # run app "main"

View File

@ -30,7 +30,7 @@ import trio
from piker.log import get_logger from piker.log import get_logger
from . import get_brokermod from . import get_brokermod
from ..service import maybe_spawn_datad from ..service import maybe_spawn_brokerd
from . import open_cached_client from . import open_cached_client
from ..accounting import MktPair from ..accounting import MktPair
@ -172,7 +172,7 @@ async def symbol_search(
# await tractor.devx._debug.maybe_init_greenback() # await tractor.devx._debug.maybe_init_greenback()
# tractor.pause_from_sync() # tractor.pause_from_sync()
async with maybe_spawn_datad( async with maybe_spawn_brokerd(
mod.name, mod.name,
infect_asyncio=getattr( infect_asyncio=getattr(
mod, mod,

View File

@ -425,7 +425,7 @@ class DataFeed:
async def stream_to_file( async def stream_to_file(
watchlist_name: str, watchlist_name: str,
filename: str, filename: str,
portal: tractor.Portal, portal: tractor._portal.Portal,
tickers: List[str], tickers: List[str],
brokermod: ModuleType, brokermod: ModuleType,
rate: int, rate: int,

View File

@ -47,25 +47,13 @@ __all__ = [
] ]
# per-daemon-kind (sub)mod groups: declares which of our # tractor RPC enable arg
# submods host the eps run by each daemon-actor kind as __enable_modules__: list[str] = [
# defined by `piker.data.validate._eps`.
# NOTE: datad-only backend (no `broker.py` yet)!
_brokerd_mods: list[str] = []
_datad_mods: list[str] = [
'api', 'api',
'feed', 'feed',
# 'broker',
] ]
# tractor RPC enable arg
__enable_modules__: list[str] = list(dict.fromkeys(
_brokerd_mods
+
_datad_mods
))
# passed to ``tractor.ActorNursery.start_actor()`` # passed to ``tractor.ActorNursery.start_actor()``
_spawn_kwargs = { _spawn_kwargs = {
'infect_asyncio': True, 'infect_asyncio': True,

View File

@ -23,6 +23,7 @@ from contextlib import (
asynccontextmanager as acm, asynccontextmanager as acm,
) )
from datetime import datetime from datetime import datetime
from functools import partial
import time import time
from typing import ( from typing import (
Any, Any,
@ -37,7 +38,7 @@ from rapidfuzz import process as fuzzy
import numpy as np import numpy as np
from tractor.trionics import ( from tractor.trionics import (
broadcast_receiver, broadcast_receiver,
maybe_open_context, maybe_open_context
collapse_eg, collapse_eg,
) )
from tractor import to_asyncio from tractor import to_asyncio
@ -523,12 +524,13 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
async def aio_price_feed_relay( async def aio_price_feed_relay(
chan: to_asyncio.LinkedTaskChannel,
fh: FeedHandler, fh: FeedHandler,
instrument: Symbol, instrument: Symbol,
from_trio: asyncio.Queue,
to_trio: trio.abc.SendChannel,
) -> None: ) -> None:
async def _trade(data: dict, receipt_timestamp): async def _trade(data: dict, receipt_timestamp):
chan.send_nowait(('trade', { to_trio.send_nowait(('trade', {
'symbol': cb_sym_to_deribit_inst( 'symbol': cb_sym_to_deribit_inst(
str_to_cb_sym(data.symbol)).lower(), str_to_cb_sym(data.symbol)).lower(),
'last': data, 'last': data,
@ -538,7 +540,7 @@ async def aio_price_feed_relay(
})) }))
async def _l1(data: dict, receipt_timestamp): async def _l1(data: dict, receipt_timestamp):
chan.send_nowait(('l1', { to_trio.send_nowait(('l1', {
'symbol': cb_sym_to_deribit_inst( 'symbol': cb_sym_to_deribit_inst(
str_to_cb_sym(data.symbol)).lower(), str_to_cb_sym(data.symbol)).lower(),
'ticks': [ 'ticks': [
@ -568,7 +570,7 @@ async def aio_price_feed_relay(
install_signal_handlers=False) install_signal_handlers=False)
# sync with trio # sync with trio
chan.started_nowait(None) to_trio.send_nowait(None)
await asyncio.sleep(float('inf')) await asyncio.sleep(float('inf'))
@ -579,9 +581,11 @@ async def open_price_feed(
) -> trio.abc.ReceiveStream: ) -> trio.abc.ReceiveStream:
async with maybe_open_feed_handler() as fh: async with maybe_open_feed_handler() as fh:
async with to_asyncio.open_channel_from( async with to_asyncio.open_channel_from(
aio_price_feed_relay, partial(
fh=fh, aio_price_feed_relay,
instrument=instrument, fh,
instrument
)
) as (chan, first): ) as (chan, first):
yield chan yield chan
@ -607,9 +611,10 @@ async def maybe_open_price_feed(
async def aio_order_feed_relay( async def aio_order_feed_relay(
chan: to_asyncio.LinkedTaskChannel,
fh: FeedHandler, fh: FeedHandler,
instrument: Symbol, instrument: Symbol,
from_trio: asyncio.Queue,
to_trio: trio.abc.SendChannel,
) -> None: ) -> None:
async def _fill(data: dict, receipt_timestamp): async def _fill(data: dict, receipt_timestamp):
breakpoint() breakpoint()
@ -632,7 +637,7 @@ async def aio_order_feed_relay(
install_signal_handlers=False) install_signal_handlers=False)
# sync with trio # sync with trio
chan.started_nowait(None) to_trio.send_nowait(None)
await asyncio.sleep(float('inf')) await asyncio.sleep(float('inf'))
@ -643,9 +648,11 @@ async def open_order_feed(
) -> trio.abc.ReceiveStream: ) -> trio.abc.ReceiveStream:
async with maybe_open_feed_handler() as fh: async with maybe_open_feed_handler() as fh:
async with to_asyncio.open_channel_from( async with to_asyncio.open_channel_from(
aio_order_feed_relay, partial(
fh=fh, aio_order_feed_relay,
instrument=instrument, fh,
instrument
)
) as (chan, first): ) as (chan, first):
yield chan yield chan

View File

@ -65,18 +65,17 @@ _brokerd_mods: list[str] = [
] ]
_datad_mods: list[str] = [ _datad_mods: list[str] = [
'api',
'feed', 'feed',
'symbols', 'symbols',
] ]
# tractor RPC enable arg # tractor RPC enable arg
__enable_modules__: list[str] = list(dict.fromkeys( __enable_modules__: list[str] = (
_brokerd_mods _brokerd_mods
+ +
_datad_mods _datad_mods
)) )
# passed to ``tractor.ActorNursery.start_actor()`` # passed to ``tractor.ActorNursery.start_actor()``
_spawn_kwargs = { _spawn_kwargs = {

View File

@ -958,26 +958,8 @@ class Client:
else: else:
raise raise
# XXX: ambiguous/unqualifiable contracts are
# returned as `None` entries by
# `qualifyContractsAsync()` (which also logs an
# "Ambiguous contract" warning listing the
# possible matches) so filter them BEFORE any
# attr access B)
contracts: list[Contract] = [
tract
for tract in contracts
if tract is not None
]
if not contracts: if not contracts:
raise ValueError( raise ValueError(f"No contract could be found {con}")
f'No (unambiguous) contract could be '
f'qualified for {con!r}\n'
f'\n'
f'If a stonk, you likely need a (more) '
f'venue-qualified fqme,\n'
f"eg. 'gld.arca.ib' instead of 'gld.ib'\n"
)
# pack all contracts into cache # pack all contracts into cache
for tract in contracts: for tract in contracts:
@ -1155,16 +1137,10 @@ class Client:
try: try:
con: Contract = self._contracts[symbol] con: Contract = self._contracts[symbol]
except KeyError: except KeyError:
# require that the mkt's contract has been previously # require that the symbol has been previously cached by
# qualified and cached (see `.symbols.get_mkt_info()` # a data feed request - ensure we aren't making orders
# which is run for any feed-init OR lazily by the # against non-known prices.
# order-request handler in `.broker`) - ensures we raise RuntimeError("Can not order {symbol}, no live feed?")
# aren't making orders against unknown contracts.
raise RuntimeError(
f'Can not order {symbol!r}, '
f'no qualified contract cached!?\n'
f'_contracts: {list(self._contracts)!r}\n'
)
try: try:
trade = self.ib.placeOrder( trade = self.ib.placeOrder(
@ -1355,21 +1331,6 @@ async def load_aio_clients(
ib = None ib = None
client = None client = None
# XXX: post (brokerd vs. datad)-split BOTH per-broker
# daemons connect to the same API gw/tws endpoint(s); to
# avoid `clientId` collisions (and the long conn-timeout
# retry cycle they cause) we offset the data-daemon's
# default id-range to be disjoint from `brokerd.ib`'s
# (which also retries with `client_id + i` increments).
if client_id == 6116: # the default from above
aname: str = tractor.current_actor().name
if 'datad' in aname:
client_id += 16
# ad-hoc (test/cli) actors get their own range to
# avoid clashing with any live daemon-tree's conns.
elif 'brokerd' not in aname:
client_id += 32
# attempt to get connection info from config; if no .toml entry # attempt to get connection info from config; if no .toml entry
# exists, we try to load from a default localhost connection. # exists, we try to load from a default localhost connection.
localhost = '127.0.0.1' localhost = '127.0.0.1'
@ -1623,72 +1584,6 @@ class MethodProxy:
self.event_table = event_table self.event_table = event_table
self._aio_ns = asyncio_ns self._aio_ns = asyncio_ns
# request-id counter for correlating each method
# call to its (eventual) response; necessary since
# the (single, shared) chan has NO other ordering
# guarantee whenever a caller task is CANCELLED
# (eg. by a search-req timeout) after sending its
# request but before consuming the response: the
# "orphaned" response otherwise gets mis-delivered
# to the next caller causing an off-by-one skew of
# every result thereafter!
self._mids = itertools.count()
self._pending: dict[
int,
tuple[trio.Event, list[dict]],
] = {}
self._request_methods: dict[int, str|None] = {}
def _deliver_method_response(
self,
msg: dict,
) -> bool:
'''
Deliver one correlated response to its waiting caller.
'''
mid: int = msg['mid']
meth: str|None = self._request_methods.pop(mid, None)
pending = self._pending.get(mid)
if pending is None:
log.warning(
f'Dropping stale method-resp,\n'
f'mid: {mid}\n'
f'meth: {meth!r}\n'
f'(caller prolly got cancelled '
f'before its resp?)\n'
)
return False
event, slot = pending
slot.append(msg)
event.set()
return True
def _track_request_method(
self,
mid: int,
meth: str|None,
) -> None:
'''
Retain bounded request metadata for stale-response logs.
'''
self._request_methods[mid] = meth
if len(self._request_methods) > 256:
oldest_mid: int = next(iter(self._request_methods))
oldest_meth: str|None = self._request_methods.pop(
oldest_mid
)
log.warning(
f'Evicting unresolved method-request metadata,\n'
f'mid: {oldest_mid}\n'
f'meth: {oldest_meth!r}\n'
f'pending metadata limit: 256\n'
)
async def _run_method( async def _run_method(
self, self,
*, *,
@ -1701,30 +1596,38 @@ class MethodProxy:
``tractor.to_asyncio`` layer. ``tractor.to_asyncio`` layer.
''' '''
mid: int = next(self._mids) chan = self.chan
event = trio.Event() await chan.send((meth, kwargs))
slot: list[dict] = []
self._pending[mid] = event, slot
self._track_request_method(mid, meth)
try:
await self.chan.send((meth, kwargs, mid))
await event.wait()
finally:
self._pending.pop(mid, None)
msg: dict = slot[0] while not chan.closed():
match msg: # send through method + ``kwargs: dict`` as pair
# OUR method-call response B) msg = await chan.receive()
case {'result': res}:
# TODO: implement reconnect functionality like
# in our `.data._web_bs.NoBsWs`
# try:
# msg = await chan.receive()
# except ConnectionError:
# self.reset()
# print(f'NEXT MSG: {msg}')
# TODO: py3.10 ``match:`` syntax B)
if 'result' in msg:
res = msg.get('result')
return res return res
case {'exception': err}: elif 'exception' in msg:
err = msg.get('exception')
raise err raise err
case _: elif 'error' in msg:
raise RuntimeError( etype, emsg = msg
f'Invalid method response for mid={mid}: {msg!r}' log.warning(f'IB error relay: {emsg}')
) continue
else:
log.warning(f'UNKNOWN IB MSG: {msg}')
def status_event( def status_event(
self, self,
@ -1734,7 +1637,7 @@ class MethodProxy:
ev = self.event_table.get(pattern) ev = self.event_table.get(pattern)
if ev is None or ev.is_set(): if not ev or ev.is_set():
# print(f'inserting new data reset event item') # print(f'inserting new data reset event item')
ev = self.event_table[pattern] = trio.Event() ev = self.event_table[pattern] = trio.Event()
@ -1777,31 +1680,22 @@ async def open_aio_client_method_relay(
log.info('asyncio `Client` method-proxy SHUTDOWN!') log.info('asyncio `Client` method-proxy SHUTDOWN!')
break break
case (meth_name, kwargs, mid): case (meth_name, kwargs):
meth_name, kwargs = msg
meth = getattr(client, meth_name) meth = getattr(client, meth_name)
try: try:
resp = await meth(**kwargs) resp = await meth(**kwargs)
# echo the msg back tagged with the # echo the msg back
# req-id for caller-side correlation. chan.send_nowait({'result': resp})
chan.send_nowait({
'mid': mid,
'result': resp,
})
# XXX: relay ALL (non-cancel) errors to the except (
# `trio`-side caller (which re-raises in the RequestError,
# `MethodProxy._run_method()` frame) instead
# of crashing this relay task and thus the # TODO: relay all errors to trio?
# whole proxy/dialog; critical post the # BaseException,
# (datad|brokerd)-split where eg. contract ) as err:
# qualification failures are expected to be chan.send_nowait({'exception': err})
# caught per-request by order/warmup code!
except Exception as err:
chan.send_nowait({
'mid': mid,
'exception': err,
})
case {'error': content}: case {'error': content}:
chan.send_nowait({'exception': content}) chan.send_nowait({'exception': content})
@ -1810,73 +1704,6 @@ async def open_aio_client_method_relay(
raise ValueError(f'Unhandled msg {msg}') raise ValueError(f'Unhandled msg {msg}')
async def relay_client_proxy_messages(
chan: tractor.to_asyncio.LinkedTaskChannel,
proxy: MethodProxy,
) -> None:
'''
Route one proxy's asyncio-client msgs from a single reader.
Each `open_client_proxy()` allocates its own linked channel and
starts this task in its relay nursery. Any number of Trio caller
tasks may concurrently write tagged requests through
`MethodProxy._run_method()` and wait on their per-`mid` slots.
This task alone reads the channel and dispatches each response to
its waiting caller.
Separate proxy contexts own separate channels and reader tasks;
no cross-proxy response broadcast is intended. IB status events
still reach each proxy through its own `Client.inline_errors()`
handler.
'''
try:
while not chan.closed():
# TODO: implement reconnect functionality like
# in our `.data._web_bs.NoBsWs`
# try:
msg = await chan.receive()
# except ConnectionError:
# proxy.reset()
match msg:
# Correlated response from a method call.
case {'mid': _}:
proxy._deliver_method_response(msg)
# routine (api-farm conn) status events
# relayed inline by `Client.inline_errors()`;
# not a response to a method call so just
# log at info and keep waiting.
case ('event', status_msg):
reason = status_msg['reason']
event = proxy.event_table.pop(reason, None)
if (
event is not None
and
event.statistics().tasks_waiting
):
log.info(f'Relaying ib status message: {msg}')
event.set()
# Inline API error with no request correlation.
case ('error', emsg):
log.warning(f'IB error relay: {emsg}')
# Out-of-band client error with no method `mid`.
case {'exception': err}:
log.error(
f'Uncorrelated IB client error: {err!r}'
)
# Preserve visibility into unexpected relay traffic.
case _:
log.warning(f'UNKNOWN IB MSG: {msg}')
except trio.EndOfChannel:
log.info('IB client proxy relay closed')
@acm @acm
async def open_client_proxy( async def open_client_proxy(
client: Client, client: Client,
@ -1913,11 +1740,28 @@ async def open_client_proxy(
continue continue
setattr(proxy, name, partial(proxy._run_method, meth=name)) setattr(proxy, name, partial(proxy._run_method, meth=name))
relay_tn.start_soon( async def relay_events():
relay_client_proxy_messages,
chan, async with chan.subscribe() as msg_stream:
proxy,
) async for msg in msg_stream:
if 'event' not in msg:
continue
# if 'event' in msg:
# wake up any system event waiters.
etype, status_msg = msg
reason = status_msg['reason']
ev = proxy.event_table.pop(reason, None)
if ev and ev.statistics().tasks_waiting:
log.info(f'Relaying ib status message: {msg}')
ev.set()
continue
relay_tn.start_soon(relay_events)
yield proxy yield proxy

View File

@ -89,9 +89,8 @@ from .api import (
MethodProxy, MethodProxy,
) )
from .symbols import ( from .symbols import (
cache_contract,
con2fqme, con2fqme,
get_mkt_info, # get_mkt_info,
) )
from .ledger import ( from .ledger import (
norm_trade_records, norm_trade_records,
@ -139,7 +138,6 @@ async def handle_order_requests(
ems_order_stream: tractor.MsgStream, ems_order_stream: tractor.MsgStream,
accounts_def: dict[str, str], accounts_def: dict[str, str],
flows: OrderDialogs, flows: OrderDialogs,
proxies: dict[str, MethodProxy],
) -> None: ) -> None:
@ -182,58 +180,6 @@ async def handle_order_requests(
if action in {'buy', 'sell'}: if action in {'buy', 'sell'}:
# validate # validate
order = BrokerdOrder(**request_msg) order = BrokerdOrder(**request_msg)
fqme: str = order.symbol
# XXX: lazily qualify and cache the contract for
# this mkt since, post the (datad|brokerd)-split,
# this trading-only actor will NOT have had its
# (api) `Client._contracts` pre-warmed by any
# in-proc feed setup (which now runs in the
# `datad.ib` sibling); we run the SAME symbology
# resolution ep the feed-side uses so the cache
# key (`MktPair.bs_fqme`) matches what
# `Client.submit_limit()` looks up.
# NOTE: normally a no-op since
# `open_trade_dialog()` eagerly pre-qualifies all
# already-open pp/order mkts at startup; this
# only fires for a brand-new (to this daemon)
# mkt's first order.
if fqme not in client._contracts:
proxy: MethodProxy = proxies[account]
try:
(
mkt,
details,
) = await get_mkt_info(
fqme,
proxy=proxy,
)
# XXX: explicit write since the lifo-cache
# may deliver a hit (body skipped) when
# another acct/client already qualified
# this fqme.
cache_contract(
client,
mkt,
details.contract,
)
except Exception as err:
log.exception(
f'Failed to qualify contract for\n'
f'fqme: {fqme!r}\n'
)
await ems_order_stream.send(
BrokerdError(
oid=oid,
symbol=fqme,
reason=(
f'No contract could be '
f'qualified for {fqme!r}:\n'
f'{err!r}'
),
)
)
continue
# XXX: by default 0 tells ``ib_async`` methods that # XXX: by default 0 tells ``ib_async`` methods that
# there is no existing order so ask the client to create # there is no existing order so ask the client to create
@ -245,34 +191,15 @@ async def handle_order_requests(
reqid = int(reqid) reqid = int(reqid)
# call our client api to submit the order # call our client api to submit the order
# NOTE: guard with order-error relay (vs. crashing reqid = client.submit_limit(
# this dialog and thus ALL order ctl for the oid=order.oid,
# backend) so one bad submission can't take down symbol=order.symbol,
# the daemon's clearing loop. price=order.price,
try: action=order.action,
reqid = client.submit_limit( size=order.size,
oid=order.oid, account=acct_number,
symbol=fqme, reqid=reqid,
price=order.price, )
action=order.action,
size=order.size,
account=acct_number,
reqid=reqid,
)
except Exception as err:
log.exception(
f'Order submission failed for\n'
f'fqme: {fqme!r}\n'
)
await ems_order_stream.send(
BrokerdError(
oid=oid,
symbol=fqme,
reason=f'Submission error: {err!r}',
)
)
continue
str_reqid: str = str(reqid) str_reqid: str = str(reqid)
if reqid is None: if reqid is None:
err_msg = BrokerdError( err_msg = BrokerdError(
@ -304,21 +231,20 @@ async def handle_order_requests(
async def recv_trade_updates( async def recv_trade_updates(
chan: tractor.to_asyncio.LinkedTaskChannel,
client: Client, client: Client,
to_trio: trio.abc.SendChannel,
) -> None: ) -> None:
''' '''
Receive and relay order control and positioning Receive and relay order control and positioning related events
related events from `ib_async`, pack as tuples and from `ib_async`, pack as tuples and push over mem-chan to our
push over mem-chan to our trio relay task for trio relay task for processing and relay to EMS.
processing and relay to EMS.
''' '''
client.inline_errors(chan) client.inline_errors(to_trio)
# sync with trio task # sync with trio task
chan.started_nowait(client.ib) to_trio.send_nowait(client.ib)
def push_tradesies( def push_tradesies(
eventkit_obj, eventkit_obj,
@ -356,7 +282,7 @@ async def recv_trade_updates(
try: try:
# emit event name + relevant ibis internal objects # emit event name + relevant ibis internal objects
chan.send_nowait((event_name, emit)) to_trio.send_nowait((event_name, emit))
except trio.BrokenResourceError: except trio.BrokenResourceError:
log.exception(f'Disconnected from {eventkit_obj} updates') log.exception(f'Disconnected from {eventkit_obj} updates')
eventkit_obj.disconnect(push_tradesies) eventkit_obj.disconnect(push_tradesies)
@ -621,14 +547,10 @@ async def open_trade_dialog(
) -> AsyncIterator[dict[str, Any]]: ) -> AsyncIterator[dict[str, Any]]:
# XXX: ONLY adjust the level of this (sub)mod's logger; get_console_log(
# attaching a (stderr) handler (via `get_console_log()`) level=loglevel,
# here would DOUBLE-print every record since the daemon name=__name__,
# fixture (`.._daemon._setup_persistent_brokerd()`) )
# already enables the console handler on the parent
# subsys logger which all our records propagate to!
if loglevel:
log.setLevel(loglevel.upper())
# task local msg dialog tracking # task local msg dialog tracking
flows = OrderDialogs() flows = OrderDialogs()
@ -865,64 +787,6 @@ async def open_trade_dialog(
for msg in order_msgs: for msg in order_msgs:
await ems_stream.send(msg) await ems_stream.send(msg)
# XXX: eagerly pre-qualify (and cache) the
# contracts for all already-open pps and
# orders so that (live) order submission
# NEVER pays a first-request qualification
# delay; any brand-new mkt is still lazily
# qualified by `handle_order_requests()` on
# its first submission. NOTE: since this
# runs BEFORE the order handler task is
# even started, no order can clear until
# the warmup completes (early reqs just
# buffer in the ems IPC stream) B)
warmup_fqmes: set[str] = {
msg.symbol
for msg in all_positions
}
warmup_fqmes.update(
msg.req.symbol
for msg in order_msgs
)
unique_clients: set[Client] = set(
aioclients.values()
)
if (
warmup_fqmes
and
proxies
):
a_proxy: MethodProxy = next(
iter(proxies.values())
)
for fqme in warmup_fqmes:
try:
(
mkt,
details,
) = await get_mkt_info(
fqme,
proxy=a_proxy,
)
except Exception as err:
# XXX: non-fatal; an
# un-warmed mkt just falls
# back to the lazy qualify
# in the order handler.
log.warning(
f'Failed to pre-qualify\n'
f'fqme: {fqme!r}\n'
f'err: {err!r}\n'
)
continue
for _client in unique_clients:
cache_contract(
_client,
mkt,
details.contract,
)
for client in set(aioclients.values()): for client in set(aioclients.values()):
trade_event_stream: LinkedTaskChannel = await tn.start( trade_event_stream: LinkedTaskChannel = await tn.start(
open_trade_event_stream, open_trade_event_stream,
@ -936,7 +800,6 @@ async def open_trade_dialog(
ems_stream, ems_stream,
accounts_def, accounts_def,
flows, flows,
proxies,
) )
# allocate event relay tasks for each client connection # allocate event relay tasks for each client connection

View File

@ -32,12 +32,14 @@ import time
from typing import ( from typing import (
Any, Any,
Callable, Callable,
TYPE_CHECKING,
) )
from async_generator import aclosing from async_generator import aclosing
import ib_async as ibis import ib_async as ibis
import numpy as np import numpy as np
from pendulum import ( from pendulum import (
now,
from_timestamp, from_timestamp,
Duration, Duration,
duration as mk_duration, duration as mk_duration,
@ -73,6 +75,9 @@ from ._util import (
) )
from .symbols import get_mkt_info from .symbols import get_mkt_info
if TYPE_CHECKING:
from trio._core._run import Task
log = get_logger( log = get_logger(
name=__name__, name=__name__,
) )
@ -220,11 +225,7 @@ async def open_history_client(
f'mean: {mean}' f'mean: {mean}'
) )
# get_bars() uses None internally for an empty reply. # could be trying to retreive bars over weekend
# The backfiller never receives that sentinel: translate
# it into NoData, or DataUnavailable at the known start
# of the instrument's history. A venue closure can also
# yield an empty reply without exhausting all history.
if out is None: if out is None:
log.error( log.error(
f"No bars starting at {end_dt!r} !?!?" f"No bars starting at {end_dt!r} !?!?"
@ -352,7 +353,6 @@ async def wait_on_data_reset(
tuple[ tuple[
trio.CancelScope, trio.CancelScope,
trio.Event, trio.Event,
list[bool],
] ]
] = trio.TASK_STATUS_IGNORED, ] = trio.TASK_STATUS_IGNORED,
) -> bool: ) -> bool:
@ -396,9 +396,8 @@ async def wait_on_data_reset(
client: Client = proxy._aio_ns client: Client = proxy._aio_ns
done = trio.Event() done = trio.Event()
result: list[bool] = []
with trio.move_on_after(timeout) as cs: with trio.move_on_after(timeout) as cs:
task_status.started((cs, done, result)) task_status.started((cs, done))
log.warning( log.warning(
'Sending DATA RESET request:\n' 'Sending DATA RESET request:\n'
@ -414,7 +413,6 @@ async def wait_on_data_reset(
'NO VNC DETECTED!\n' 'NO VNC DETECTED!\n'
'Manually press ctrl-alt-f on your IB java app' 'Manually press ctrl-alt-f on your IB java app'
) )
result.append(False)
done.set() done.set()
return False return False
@ -429,7 +427,6 @@ async def wait_on_data_reset(
]: ]:
await ev.wait() await ev.wait()
log.info(f"{name} DATA RESET") log.info(f"{name} DATA RESET")
result.append(True)
done.set() done.set()
return True return True
@ -438,12 +435,12 @@ async def wait_on_data_reset(
'Data reset task canceled?' 'Data reset task canceled?'
) )
result.append(False)
done.set() done.set()
return False return False
_data_reset_lock = trio.Lock() _data_resetter_task: Task | None = None
_failed_resets: int = 0
async def get_bars( async def get_bars(
@ -459,6 +456,10 @@ async def get_bars(
# how long before we trigger a feed reset (seconds) # how long before we trigger a feed reset (seconds)
feed_reset_timeout: float = 3, feed_reset_timeout: float = 3,
# how many days to subtract before giving up on further
# history queries for instrument, presuming that most don't
# not trade for a week XD
max_nodatas: int = 6,
max_failed_resets: int = 6, max_failed_resets: int = 6,
task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED, task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
@ -478,10 +479,8 @@ async def get_bars(
`.ib.api.Client` methods. `.ib.api.Client` methods.
''' '''
cancelled_count: int = 0 global _data_resetter_task, _failed_resets
failed_resets: int = 0 nodatas_count: int = 0
pacing_reset_pending: bool = False
pacing_reset_completed: bool = False
data_cs: trio.CancelScope | None = None data_cs: trio.CancelScope | None = None
result: tuple[ result: tuple[
@ -494,16 +493,10 @@ async def get_bars(
async def query(): async def query():
nonlocal result, data_cs global _failed_resets
nonlocal cancelled_count, failed_resets nonlocal result, data_cs, end_dt, nodatas_count
nonlocal pacing_reset_pending, pacing_reset_completed
while True: while _failed_resets < max_failed_resets:
if failed_resets >= max_failed_resets:
raise DataUnavailable(
f'IB history reset failed {failed_resets} '
f'times for {fqme}'
)
try: try:
( (
bars, bars,
@ -535,10 +528,6 @@ async def get_bars(
) )
# NOTE: REQUIRED to pass back value.. # NOTE: REQUIRED to pass back value..
result = None result = None
failed_resets = 0
result_ready.set()
if data_cs:
data_cs.cancel()
return None return None
# not enough bars signal, likely due to venue # not enough bars signal, likely due to venue
@ -598,7 +587,6 @@ async def get_bars(
first_dt, first_dt,
last_dt, last_dt,
) )
failed_resets = 0
# signal data reset loop parent task # signal data reset loop parent task
result_ready.set() result_ready.set()
@ -623,21 +611,27 @@ async def get_bars(
# the frame dt index since the upper layer may # the frame dt index since the upper layer may
# be doing so concurrently and we don't want to # be doing so concurrently and we don't want to
# be delivering frames that weren't asked for. # be delivering frames that weren't asked for.
# try to decrement start point and look further back
# end_dt = end_dt.subtract(seconds=2000)
logmsg = "SUBTRACTING DAY from DT index"
if end_dt is not None:
end_dt = end_dt.subtract(days=1)
elif end_dt is None:
end_dt = now().subtract(days=1)
log.warning( log.warning(
f'NO DATA found ending @ {end_dt}\n' f'NO DATA found ending @ {end_dt}\n'
+ logmsg
) )
# Match get_bars()'s empty-bars return: None is
# internal to the IB adapter, not its backfill if nodatas_count >= max_nodatas:
# protocol. open_history_client() converts it raise DataUnavailable(
# to NoData (or DataUnavailable at history start). f'Presuming {fqme} has no further history '
# Wake the waiting get_bars() task before exiting f'after {max_nodatas} tries..'
# so it does not mistake this reply for a timeout. )
result = None
failed_resets = 0 nodatas_count += 1
result_ready.set() continue
if data_cs:
data_cs.cancel()
return None
elif ( elif (
'API historical data query cancelled' 'API historical data query cancelled'
@ -648,12 +642,6 @@ async def get_bars(
'Query cancelled by IB (:eyeroll:):\n' 'Query cancelled by IB (:eyeroll:):\n'
f'{err.message}' f'{err.message}'
) )
cancelled_count += 1
if cancelled_count >= max_failed_resets:
raise DataUnavailable(
f'IB cancelled {cancelled_count} history '
f'queries for {fqme}'
)
continue continue
elif ( elif (
@ -674,7 +662,6 @@ async def get_bars(
) )
client = proxy._aio_ns.ib.client client = proxy._aio_ns.ib.client
pacing_reset_pending = True
# cancel any existing reset task # cancel any existing reset task
if data_cs: if data_cs:
@ -682,81 +669,69 @@ async def get_bars(
data_cs.cancel() data_cs.cancel()
# spawn new data reset task # spawn new data reset task
async with _data_reset_lock: data_cs, reset_done = await tn.start(
( partial(
data_cs, wait_on_data_reset,
reset_done, proxy,
reset_result, reset_type='connection'
) = await tn.start(
partial(
wait_on_data_reset,
proxy,
reset_type='connection'
)
) )
await reset_done.wait() )
if reset_result[0]: if reset_done:
failed_resets = 0 _failed_resets = 0
else: else:
failed_resets += 1 _failed_resets += 1
pacing_reset_pending = False
pacing_reset_completed = True
continue continue
else: else:
raise raise
# TODO: make this global across all history task/requests
# such that simultaneous symbol queries don't try data resettingn
# too fast..
unset_resetter: bool = False
async with ( async with (
tractor.trionics.collapse_eg(), tractor.trionics.collapse_eg(),
trio.open_nursery() as tn trio.open_nursery() as tn
): ):
# Run history until a result or bounded reset failure. # start history request that we allow
# to run indefinitely until a result is acquired
tn.start_soon(query) tn.start_soon(query)
# Serialize farm resets across every history requester. # start history reset loop which waits up to the timeout
# for a result before triggering a data feed reset.
while not result_ready.is_set(): while not result_ready.is_set():
with trio.move_on_after(feed_reset_timeout): with trio.move_on_after(feed_reset_timeout):
await result_ready.wait() await result_ready.wait()
break break
async with _data_reset_lock: if _data_resetter_task:
if result_ready.is_set(): # don't double invoke the reset hack if another
break # requester task already has it covered.
if pacing_reset_completed: continue
pacing_reset_completed = False
continue
( else:
data_cs, _data_resetter_task = trio.lowlevel.current_task()
reset_done, unset_resetter: bool = True
reset_result,
) = await tn.start( # spawn new data reset task
partial( data_cs, reset_done = await tn.start(
wait_on_data_reset, partial(
proxy, wait_on_data_reset,
reset_type='data', proxy,
) reset_type='data',
) )
await reset_done.wait() )
# sync wait on reset to complete
await reset_done.wait()
if ( _data_resetter_task = (
result_ready.is_set() None
or if unset_resetter
reset_result[0] else _data_resetter_task
): )
failed_resets = 0 assert result
elif pacing_reset_pending:
# Pacing recovery deliberately cancelled this reset
# and will consume the retry itself.
pass
else:
failed_resets += 1
if failed_resets >= max_failed_resets:
raise DataUnavailable(
f'IB history reset failed '
f'{failed_resets} times for {fqme}'
)
return ( return (
result, result,
data_cs is not None, data_cs is not None,

View File

@ -639,40 +639,12 @@ async def get_mkt_info(
# if possible register the bs_mktid to the just-built # if possible register the bs_mktid to the just-built
# mkt so that it can be retreived by order mode tasks later. # mkt so that it can be retreived by order mode tasks later.
# TODO NOTE: this is going to be problematic if/when we split
# out the datatd vs. brokerd actors since the mktmap lookup
# table will now be inaccessible..
if proxy is not None: if proxy is not None:
cache_contract( client: Client = proxy._aio_ns
proxy._aio_ns, client._contracts[mkt.bs_fqme] = con
mkt, client._cons2mkts[con] = mkt
con,
)
return mkt, details return mkt, details
def cache_contract(
client: Client,
mkt: MktPair,
con: ibis.Contract,
) -> None:
'''
Register a (qualified) contract + mkt-info pair on the
given (api) `Client`'s actor-local lookup tables.
Cached under BOTH fqme key-forms since consumers vary:
- `mkt.bs_fqme` (eg. 'nvda.nasdaq'): read by
`Client.submit_limit()` for order requests,
- `mkt.fqme` (eg. 'nvda.nasdaq.ib'): read by the
fill-time pp-update (symcache-backup-table) path in
`.broker.emit_pp_update()`.
NOTE: post the (datad|brokerd)-actor-split this MUST be
run (in the trading actor) for every mkt either eagerly
at `.broker.open_trade_dialog()` startup or lazily per
order request; there is no in-proc feed setup doing it
implicitly anymore!
'''
client._contracts[mkt.bs_fqme] = con
client._contracts[mkt.fqme] = con
client._cons2mkts[con] = mkt

View File

@ -66,24 +66,10 @@ __all__ = [
] ]
# per-daemon-kind (sub)mod groups: declares which of our # tractor RPC enable arg
# submods host the eps run by each daemon-actor kind as __enable_modules__: list[str] = [
# defined by `piker.data.validate._eps`.
_brokerd_mods: list[str] = [
'api', 'api',
'broker', 'broker',
]
_datad_mods: list[str] = [
'api',
'feed', 'feed',
'symbols', 'symbols',
] ]
# tractor RPC enable arg
__enable_modules__: list[str] = list(dict.fromkeys(
_brokerd_mods
+
_datad_mods
))

View File

@ -52,7 +52,6 @@ from piker.brokers._util import (
SymbolNotFound, SymbolNotFound,
BrokerError, BrokerError,
DataThrottle, DataThrottle,
get_or_raise_on_pair_schema_mismatch,
) )
from piker.accounting import Transaction from piker.accounting import Transaction
from piker.log import get_logger from piker.log import get_logger
@ -118,9 +117,6 @@ class InvalidKey(ValueError):
''' '''
from tractor._exceptions import reg_err_types
reg_err_types([InvalidKey])
class Client: class Client:
@ -506,16 +502,7 @@ class Client:
# NOTE: always cache in pairs tables for faster lookup # NOTE: always cache in pairs tables for faster lookup
with tractor.devx.maybe_open_crash_handler(): # as bxerr: with tractor.devx.maybe_open_crash_handler(): # as bxerr:
# pair = Pair(xname=xkey, **data) pair = Pair(xname=xkey, **data)
pair: Pair = get_or_raise_on_pair_schema_mismatch(
pair_type=Pair,
fields_data=dict(
xname=xkey,
**data,
),
provider_name='kraken',
# api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
)
# register the above `Pair` structs for all # register the above `Pair` structs for all
# key-sets/monikers: a set of 4 (frickin) tables # key-sets/monikers: a set of 4 (frickin) tables

View File

@ -67,15 +67,13 @@ from piker.log import (
get_logger, get_logger,
) )
from piker.data import open_symcache from piker.data import open_symcache
from piker.data._web_bs import (
open_autorecon_ws,
NoBsWs,
)
from .api import ( from .api import (
Client, Client,
BrokerError, BrokerError,
) )
from .feed import ( from .feed import (
open_autorecon_ws,
NoBsWs,
stream_messages, stream_messages,
) )
from .ledger import ( from .ledger import (
@ -99,9 +97,6 @@ MsgUnion = Union[
class TooFastEdit(Exception): class TooFastEdit(Exception):
'Edit requests faster then api submissions' 'Edit requests faster then api submissions'
from tractor._exceptions import reg_err_types
reg_err_types([TooFastEdit])
# TODO: make this wrap the `Client` and `ws` instances # TODO: make this wrap the `Client` and `ws` instances
# and give it methods to submit cancel vs. add vs. edit # and give it methods to submit cancel vs. add vs. edit

View File

@ -19,9 +19,6 @@ Symbology defs and search.
''' '''
from decimal import Decimal from decimal import Decimal
from typing import (
ClassVar,
)
import tractor import tractor
@ -89,14 +86,9 @@ class Pair(Struct):
short_position_limit: float = 0 short_position_limit: float = 0
long_position_limit: float = float('inf') long_position_limit: float = float('inf')
# TODO, add API note when this was added!
execution_venue: str|None = None
# TODO: should we make this a literal NamespacePath ref? # TODO: should we make this a literal NamespacePath ref?
ns_path: str = 'piker.brokers.kraken:Pair' ns_path: str = 'piker.brokers.kraken:Pair'
_api_url: ClassVar[str] = 'https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs'
@property @property
def bs_mktid(self) -> str: def bs_mktid(self) -> str:
''' '''

View File

@ -95,9 +95,6 @@ _time_frames = {
class QuestradeError(Exception): class QuestradeError(Exception):
"Non-200 OK response code" "Non-200 OK response code"
from tractor._exceptions import reg_err_types
reg_err_types([QuestradeError])
class ContractsKey(NamedTuple): class ContractsKey(NamedTuple):
symbol: str symbol: str

Some files were not shown because too many files have changed in this diff Show More