Compare commits
66 Commits
main
...
wkt/land_n
| Author | SHA1 | Date |
|---|---|---|
|
|
c1b3e17069 | |
|
|
7cec91e287 | |
|
|
50b84d68f2 | |
|
|
a26d69c269 | |
|
|
8a4a1f67d8 | |
|
|
1b76dd64fd | |
|
|
9851ff886a | |
|
|
9cfe2591d9 | |
|
|
fc532d6acb | |
|
|
61cd084ef2 | |
|
|
13c036d16a | |
|
|
79d7b48fd3 | |
|
|
db3d41a022 | |
|
|
0142476966 | |
|
|
5b3370ddbe | |
|
|
4d3f77f9df | |
|
|
25e63211c6 | |
|
|
d27f953144 | |
|
|
4402d391f0 | |
|
|
29c0fb72b9 | |
|
|
3e3e0fdd46 | |
|
|
b349332e6b | |
|
|
5183d4094f | |
|
|
251987fc4b | |
|
|
63c763800a | |
|
|
7a06a620e2 | |
|
|
3053324f04 | |
|
|
cb5e81d443 | |
|
|
51795a4e07 | |
|
|
495c091546 | |
|
|
2a826fba17 | |
|
|
6575db6463 | |
|
|
3ec447a45a | |
|
|
11aa2eeb1e | |
|
|
9dde7a9867 | |
|
|
02bb193f87 | |
|
|
6f626a9c45 | |
|
|
72d0bdad5e | |
|
|
dc618229bd | |
|
|
635314c757 | |
|
|
d1571ddfc7 | |
|
|
7bef50b0a5 | |
|
|
e8e0730a50 | |
|
|
4133f07340 | |
|
|
507c75c669 | |
|
|
1fa45dfa98 | |
|
|
bde2457bdd | |
|
|
1ba9bdbac0 | |
|
|
d86f29d416 | |
|
|
69a6edd6f8 | |
|
|
bbe9deaa62 | |
|
|
70e8d60326 | |
|
|
2c007d3ada | |
|
|
1fde12d3fd | |
|
|
c347ab08ea | |
|
|
c5155ce814 | |
|
|
d7cefa71f3 | |
|
|
20bfbc2380 | |
|
|
d7f956a566 | |
|
|
d26286f6cc | |
|
|
22f166a5c6 | |
|
|
6c8709d309 | |
|
|
23632d2153 | |
|
|
d85e57334c | |
|
|
6c8d17da75 | |
|
|
1976f6a5de |
|
|
@ -0,0 +1,143 @@
|
||||||
|
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
|
||||||
|
|
||||||
|
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
|
||||||
|
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
|
||||||
|
|
||||||
|
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
|
||||||
|
|
||||||
|
**User-decided constraints:**
|
||||||
|
|
||||||
|
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
|
||||||
|
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
|
||||||
|
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
|
||||||
|
|
||||||
|
## Target topology
|
||||||
|
|
||||||
|
```
|
||||||
|
pikerd
|
||||||
|
├── datad.ib feed bus, shm writers, tsp history, symbol search
|
||||||
|
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
|
||||||
|
├── emsd
|
||||||
|
│ └── paperboi.ib (paper mode; opens its feed via datad)
|
||||||
|
└── samplerd
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key verified facts (load-bearing)
|
||||||
|
|
||||||
|
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
|
||||||
|
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
|
||||||
|
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
|
||||||
|
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
|
||||||
|
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
|
||||||
|
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
|
||||||
|
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
|
||||||
|
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
|
||||||
|
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
|
||||||
|
|
||||||
|
## Module decision
|
||||||
|
|
||||||
|
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
|
||||||
|
|
||||||
|
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
|
||||||
|
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
|
||||||
|
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
|
||||||
|
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
|
||||||
|
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
|
||||||
|
- `ib`: adjust existing groups so each includes `'api'`
|
||||||
|
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
|
||||||
|
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
|
||||||
|
|
||||||
|
**Gate**: full pytest green, zero behavior change.
|
||||||
|
|
||||||
|
## Stage 1 — introduce `datad` daemon machinery (additive)
|
||||||
|
|
||||||
|
New `piker/data/_daemon.py`:
|
||||||
|
|
||||||
|
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
|
||||||
|
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
|
||||||
|
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
|
||||||
|
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
|
||||||
|
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
|
||||||
|
|
||||||
|
Supporting edits:
|
||||||
|
|
||||||
|
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
|
||||||
|
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
|
||||||
|
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
|
||||||
|
|
||||||
|
**Gate**: suite green + new spawn test; nothing routes through datad yet.
|
||||||
|
|
||||||
|
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
|
||||||
|
|
||||||
|
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
|
||||||
|
|
||||||
|
`piker/clearing/_ems.py`:
|
||||||
|
|
||||||
|
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
|
||||||
|
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
|
||||||
|
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
|
||||||
|
|
||||||
|
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
|
||||||
|
|
||||||
|
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
|
||||||
|
|
||||||
|
## Stage 3 — feed layer hard cutover to datad
|
||||||
|
|
||||||
|
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
|
||||||
|
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
|
||||||
|
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
|
||||||
|
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
|
||||||
|
- `piker/brokers/core.py:33,175` (`symbol_search`)
|
||||||
|
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
|
||||||
|
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
|
||||||
|
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
|
||||||
|
|
||||||
|
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
|
||||||
|
|
||||||
|
## Stage 4 — caps-sec slimming, validation, ib client-id
|
||||||
|
|
||||||
|
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
|
||||||
|
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
|
||||||
|
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
|
||||||
|
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
|
||||||
|
|
||||||
|
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risk register
|
||||||
|
|
||||||
|
| Risk | Sev | Mitigation |
|
||||||
|
|---|---|---|
|
||||||
|
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
|
||||||
|
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
|
||||||
|
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
|
||||||
|
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
|
||||||
|
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
|
||||||
|
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
|
||||||
|
|
||||||
|
## Verification (per stage gates above, plus end-to-end)
|
||||||
|
|
||||||
|
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
|
||||||
|
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
|
||||||
|
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
|
||||||
|
|
||||||
|
## Critical files
|
||||||
|
|
||||||
|
- **new** `piker/data/_daemon.py` — all datad machinery
|
||||||
|
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
|
||||||
|
- `piker/data/feed.py` — spawn cutover + actor-name assert
|
||||||
|
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
|
||||||
|
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
|
||||||
|
- `piker/data/validate.py` — `get_eps()` helper
|
||||||
|
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
|
||||||
|
- `piker/accounting/cli.py` — keyword-form dialog call
|
||||||
|
- `tests/test_services.py` — inverted + new assertions
|
||||||
|
|
@ -0,0 +1,233 @@
|
||||||
|
# Bounded IB recovery candidate
|
||||||
|
|
||||||
|
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
|
||||||
|
`/home/goodboy/repos/piker`. Base:
|
||||||
|
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
|
||||||
|
Changes are unstaged and uncommitted. No acceptance is implied.
|
||||||
|
|
||||||
|
## Patch selection and provenance
|
||||||
|
|
||||||
|
Keep these existing landing commits:
|
||||||
|
|
||||||
|
| Commit | Reason |
|
||||||
|
| --- | --- |
|
||||||
|
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
|
||||||
|
| `e8e0730a` | Correlate MethodProxy requests and responses |
|
||||||
|
| `7bef50b0` | One response reader, per-request waiters |
|
||||||
|
| `d1571ddf` | Propagate daemon-handler exceptions |
|
||||||
|
| `dc618229` | Isolate cached feed stream consumers |
|
||||||
|
|
||||||
|
Proposed separate implementation boundaries, in this order:
|
||||||
|
|
||||||
|
1. Back out the code and original test from
|
||||||
|
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
|
||||||
|
This restores the direct-owner `_quote_streams` implementation.
|
||||||
|
Retain the historical prompt records. Replace the removed
|
||||||
|
mocked-cache regression with the real-bridge restart reproducer.
|
||||||
|
This backout is not a claim that arbitrary concurrent direct
|
||||||
|
callers have safe broadcast ownership.
|
||||||
|
2. Backport **only** `piker/brokers/ib/feed.py` from
|
||||||
|
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
|
||||||
|
Retain its seven history-adapter regressions in
|
||||||
|
`tests/test_ib_history.py`. This includes reset serialization,
|
||||||
|
completion/result signalling and bounded cancellation/failure.
|
||||||
|
It does not include the original SHM/storage orchestration patch.
|
||||||
|
3. Backport **only** `piker/brokers/ib/feed.py` from
|
||||||
|
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
|
||||||
|
Retain `test_empty_history_preserves_request_boundary` and its
|
||||||
|
six cases. This preserves no-data request boundaries and wakes
|
||||||
|
the waiter. It does not include the original history-layer
|
||||||
|
endpoint/filtering changes. Add the real-bridge history recovery
|
||||||
|
case after these adapter changes are present.
|
||||||
|
4. Import the code and original regression from
|
||||||
|
`12b0d66e854247e92cb5a8668be71d816c72d856`.
|
||||||
|
Add a separate real-broadcaster/production-loop regression.
|
||||||
|
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
|
||||||
|
This is a local UX cleanup retaining the existing `None` result.
|
||||||
|
It does not backport the null-grouping/deduplication rewrite in
|
||||||
|
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
|
||||||
|
6. Record cross-boundary regression results and live qualification
|
||||||
|
evidence, distinguishing the known callback failure from passes.
|
||||||
|
|
||||||
|
All three imported source commits were authored by
|
||||||
|
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
|
||||||
|
and the full source SHA when committing each partial backport.
|
||||||
|
Use explicit partial-backport wording, not a whole-commit cherry
|
||||||
|
claim. New real-runtime tests are this session's additional work.
|
||||||
|
|
||||||
|
The current aggregate working diff contains these boundaries in
|
||||||
|
the same feed file. Staging that entire file would combine the
|
||||||
|
first three boundaries. Reconstruct/stage the original per-source
|
||||||
|
patches in the above order when preparing commits.
|
||||||
|
|
||||||
|
## Quote-cache history correction
|
||||||
|
|
||||||
|
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
|
||||||
|
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
|
||||||
|
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
|
||||||
|
They are equivalent replays, not different implementations.
|
||||||
|
|
||||||
|
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
|
||||||
|
change and retains direct ownership. The original qualification
|
||||||
|
did not validate the combination assembled for the landing.
|
||||||
|
|
||||||
|
After this candidate's edits, `piker/brokers/ib/feed.py` matches
|
||||||
|
the source at `1da9b760` byte-for-byte. Compared with the evaluated
|
||||||
|
tip, only the later schedule/reservation APIs are absent.
|
||||||
|
The EMS code patch has the same stable patch ID as the original
|
||||||
|
`12b0d66e` EMS code patch:
|
||||||
|
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
|
||||||
|
|
||||||
|
## Real-runtime evidence
|
||||||
|
|
||||||
|
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
|
||||||
|
an asyncio-infected child, the installed pinned Tractor bridge,
|
||||||
|
production Piker quote setup, and real IB/eventkit ticker events.
|
||||||
|
Only the external IB connection/input is replaced with replay data.
|
||||||
|
|
||||||
|
Before the quote-cache backout, cancellation after ticker delivery
|
||||||
|
raised `RemoteActorError[TrioTaskExited]` through
|
||||||
|
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
|
||||||
|
reported fatal path.
|
||||||
|
|
||||||
|
After the backout:
|
||||||
|
|
||||||
|
- Two quote generations finish; each subscription is cancelled,
|
||||||
|
each asyncio producer is done and each ticker handler removed.
|
||||||
|
- A second RPC repeats the sequence in the same actor successfully.
|
||||||
|
- The strict callback-cleanliness case still fails: Tractor's
|
||||||
|
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
|
||||||
|
exception handler even though the Trio consumer recovers.
|
||||||
|
|
||||||
|
The history integration case keeps `Client.bars()`, MethodProxy,
|
||||||
|
both relays, `get_bars()`, `wait_on_data_reset()` and
|
||||||
|
`data_reset_hack()` real. It replaces external contract discovery,
|
||||||
|
IB history replies and VNC input. A real IB error event announces
|
||||||
|
farm recovery. The replay rejects retries before recovery and
|
||||||
|
proves one reset followed by completion for both 1s and 60s data.
|
||||||
|
|
||||||
|
The new EMS case runs the production dark-clearing loop over a
|
||||||
|
real Tractor broadcaster: both EMS and a sibling receive a tick,
|
||||||
|
EMS cancellation removes only its subscription, and the sibling
|
||||||
|
continues with an unchanged strict lag policy.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Unique selected cases: **38 passed, 1 failed**.
|
||||||
|
The failure is intentionally visible, not skipped or xfailed.
|
||||||
|
|
||||||
|
Run from the checkout's working Nix environment:
|
||||||
|
|
||||||
|
```xsh
|
||||||
|
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
|
||||||
|
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
|
||||||
|
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
|
||||||
|
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
The adapter/proxy/feed/EMS selection initially passed 21 cases;
|
||||||
|
the added real EMS case then passed with its imported case.
|
||||||
|
The bridge file passed two cases and failed callback cleanliness.
|
||||||
|
Logging/startup passed 13 cases; runtime boot passed one with the
|
||||||
|
existing `Registrar.uid` deprecation warning.
|
||||||
|
|
||||||
|
The agent shell inherited another project's environment and lacked
|
||||||
|
native libraries. Tests used the existing Piker `py313` via
|
||||||
|
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
|
||||||
|
libraries from the pinned flake's existing Qt dependency closure,
|
||||||
|
excluding Nix Qt itself and glibc. No environment sync or shellHook
|
||||||
|
ran. Piker resolved to this checkout; Tractor resolved to its
|
||||||
|
installed package in `py313`.
|
||||||
|
|
||||||
|
Workflow-state preparation initially reported changed legacy
|
||||||
|
files/modes after migration. On 2026-10-23 the human authorized
|
||||||
|
reconciliation. Inspection found only two missing legacy guidance
|
||||||
|
snapshots; the neutral copies matched their saved hashes and modes.
|
||||||
|
Restoring those local snapshots resolved preparation without
|
||||||
|
changing archived messages or the original migration receipt.
|
||||||
|
|
||||||
|
## Human live qualification, 2026-10-23
|
||||||
|
|
||||||
|
The human ran these journeys with the installed Tractor pin,
|
||||||
|
the existing IB Gateway and pre-existing NativeDB. The agent did
|
||||||
|
not operate the broker connection, VNC, orders or database.
|
||||||
|
|
||||||
|
- Initial QQQ startup/backfill recovered after pacing error 162
|
||||||
|
with one reset attempt, then displayed live updates. The reset
|
||||||
|
waiter appears to have timed out; the log did not establish an
|
||||||
|
acknowledged farm reset or live quote-stream restart.
|
||||||
|
- Sequential chart exits and reattachment preserved all four
|
||||||
|
daemon PIDs; a read-only process/listener check confirmed this.
|
||||||
|
- The overlapping two-chart journey used `pikerd@961108`,
|
||||||
|
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
|
||||||
|
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
|
||||||
|
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
|
||||||
|
then C closed at 08:18:37. The human reported the exercise worked;
|
||||||
|
survivor updates are a manual observation, while timestamps and
|
||||||
|
unchanged daemon identities corroborate overlap and service reuse.
|
||||||
|
- Startup in that journey hit the explicit single-null-sample
|
||||||
|
breakpoint several times before the charts overlapped. The small
|
||||||
|
debugger cleanup above was applied afterward; this live run does
|
||||||
|
not itself verify that later edit.
|
||||||
|
- Departing charts still report cancellation groups and an FSP
|
||||||
|
receiver-close `ClosedResourceError`, followed by the Qt close
|
||||||
|
handler's self-SIGINT. Peer-disconnected final result warnings
|
||||||
|
remain contained in the persistent services.
|
||||||
|
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
|
||||||
|
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
|
||||||
|
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
|
||||||
|
After the human continued, all three children were reported
|
||||||
|
exited with status 0 and the root terminated at 08:19:45.
|
||||||
|
Cancellation-only RPC error reporting and late cancel-ack
|
||||||
|
diagnostics also remain.
|
||||||
|
|
||||||
|
The human explicitly accepted deferring this daemon teardown fix
|
||||||
|
from #94 and requested loud attribution for the bounded backports.
|
||||||
|
This is an operational smoke result, not clean-shutdown coverage.
|
||||||
|
|
||||||
|
## Downstream reconciliation requirements
|
||||||
|
|
||||||
|
Before landing #94, the human wants an easy replay of the downstream
|
||||||
|
histories intended to follow it. Rehearse against the agreed #94
|
||||||
|
tree in disposable worktrees, then inspect range-diffs, final code
|
||||||
|
and relevant regressions. This note does not authorize rewriting
|
||||||
|
the original branches or changing acceptance markers.
|
||||||
|
|
||||||
|
- `0846cbd4` and `1da9b760` are partial backports. Their imported
|
||||||
|
adapter hunks may already be present, but their history/SHM/storage
|
||||||
|
changes and omitted integration tests remain necessary downstream.
|
||||||
|
Never drop either whole source commit merely because `feed.py`
|
||||||
|
matches or its adapter hunk becomes empty.
|
||||||
|
- `12b0d66e` has source-equivalent EMS code and its imported test,
|
||||||
|
plus an additional local real-broadcaster test. Preserve that extra
|
||||||
|
regression when resolving test-file overlap.
|
||||||
|
- The quote-cache backout intentionally restores direct ownership
|
||||||
|
for this landing. Reintroducing original `4af885d9` downstream
|
||||||
|
requires qualifying cached-owner teardown with the matching
|
||||||
|
Tractor revision; the backout is not a permanent ownership design.
|
||||||
|
- The local single-null breakpoint removal retains legacy behavior.
|
||||||
|
Prefer the full downstream `976db6c3` algorithm when it is replayed;
|
||||||
|
preserving this `return None` guard would lose that later fix.
|
||||||
|
- Real-bridge tests and manual qualification notes are local work,
|
||||||
|
not evidence that omitted portions of the source commits landed.
|
||||||
|
|
||||||
|
Commit messages must name full source SHAs, original authorship,
|
||||||
|
imported paths/tests, omissions, local additions and replay guidance.
|
||||||
|
Use explicit partial-backport language for adapter subsets; do not
|
||||||
|
claim a full cherry-pick or reuse the source's broader fix claims.
|
||||||
|
|
||||||
|
## Remaining decision
|
||||||
|
|
||||||
|
This candidate removes the reproduced fatal quote-restart
|
||||||
|
propagation and imports existing reset fixes. The human exercised
|
||||||
|
live QQQ startup, backfill and concurrent chart consumer lifetimes.
|
||||||
|
Completion-callback and teardown errors remain; the human accepts
|
||||||
|
them as follow-ups outside the bounded #94 landing.
|
||||||
|
The storage/SHM portions of the downstream fixes are not covered by
|
||||||
|
the adapter backport or its integration tests.
|
||||||
|
|
||||||
|
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
|
||||||
|
`1a2d4893` stays a separate follow-up. Further Tractor teardown
|
||||||
|
changes or storage imports require reassessing the bounded landing
|
||||||
|
rather than silently expanding this candidate.
|
||||||
|
|
@ -0,0 +1,71 @@
|
||||||
|
# IB recovery qualification, 2026-11-06
|
||||||
|
|
||||||
|
This records human-operated qualification of the bounded #94
|
||||||
|
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
|
||||||
|
the narrow history catch described below. It supplements the
|
||||||
|
historical evidence in `ai/ib-recovery-candidate.md`.
|
||||||
|
|
||||||
|
## Runtime and dependency source
|
||||||
|
|
||||||
|
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
|
||||||
|
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
|
||||||
|
local checkout during evaluation. The human subsequently published
|
||||||
|
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
|
||||||
|
remote query confirmed the published OID before restoring Piker's
|
||||||
|
Git source and recording the same OID in `uv.lock`.
|
||||||
|
|
||||||
|
Offline lock validation resolves 126 packages successfully. No
|
||||||
|
environment sync was performed. This validates the lock metadata;
|
||||||
|
it does not claim a fresh installation from the published source.
|
||||||
|
|
||||||
|
## Manual observations
|
||||||
|
|
||||||
|
The human operated Gateway, chart windows and paper orders. The
|
||||||
|
agent did not operate live services or the account.
|
||||||
|
|
||||||
|
- Same-feed overlap, close/reopen and paper order submit/cancel
|
||||||
|
worked after the existing order-dialog guard was imported.
|
||||||
|
- Different-feed overlap originally exposed a timed-out one-second
|
||||||
|
MNQ March 2027 gap query. PDB showed `timedout=True` and
|
||||||
|
`out is None`. Its uncaught `NoData` canceled sibling history and
|
||||||
|
quote tasks even though `pikerd` remained alive.
|
||||||
|
- After the narrow catch, the human reported the overlap journeys
|
||||||
|
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
|
||||||
|
live feeds survived the Gateway data reset; closing all charts
|
||||||
|
and reopening with `pikerd` left running restored live updates.
|
||||||
|
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
|
||||||
|
that the intended Gateway data reset was triggered.
|
||||||
|
|
||||||
|
These are manual observations, not automated timing or latency
|
||||||
|
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
|
||||||
|
labels it a connection reset.
|
||||||
|
|
||||||
|
## Narrow history backport
|
||||||
|
|
||||||
|
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
|
||||||
|
`maybe_fill_null_segments()` is copied from
|
||||||
|
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
|
||||||
|
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
|
||||||
|
remains in place; this does not import broader history repair.
|
||||||
|
|
||||||
|
The new regression uses real null detection and checks that a
|
||||||
|
sibling in the same nursery still runs after either provider
|
||||||
|
exception. Both cases failed before the catch. With it, the two
|
||||||
|
focused history files passed: 17 tests. Earlier bounded-recovery
|
||||||
|
and Tractor qualification results remain separate evidence, not
|
||||||
|
fresh test runs of the final dependency commit tree.
|
||||||
|
|
||||||
|
## Remaining recovery limits
|
||||||
|
|
||||||
|
A full Gateway restart was triggered accidentally through its
|
||||||
|
auto-restart settings dialog. The disconnect delivered the string
|
||||||
|
`Peer closed connection.` to `deliver_trade_events()`. The string
|
||||||
|
branch then reached `err['reqid']`, raising `TypeError`; its `code`
|
||||||
|
local can also retain an earlier event's value. Full Gateway restart
|
||||||
|
recovery is outside this bounded landing.
|
||||||
|
|
||||||
|
The earlier history-failure teardown also exposed a separate
|
||||||
|
Tractor completion-callback assertion after `BrokenResourceError`.
|
||||||
|
The history catch does not establish that this callback race is
|
||||||
|
fixed. Broader history recovery and these disconnect paths remain
|
||||||
|
downstream work.
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-07T16:54:53Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: gpt-6.1-sol
|
||||||
|
git_ref: bde2457b
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user requested fixing all seven local PR94 review findings and
|
||||||
|
exporting the review for human inspection in Tuicr. They asked that a
|
||||||
|
Tractor pin adjustment be considered before adding Piker workarounds.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Correct the endpoint defaults, Click argument shape, daemon binding and
|
||||||
|
registrar option handling, chart fallback, and stray debugger pause.
|
||||||
|
Keep the absent datad implementation's test outside this landing
|
||||||
|
branch. Add offline startup coverage and export original findings
|
||||||
|
without modifying review states. The checked newer Tractor address
|
||||||
|
branch still rejects mixed transports, so leave the dependency pin.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `config/conf.toml`: TCP-only default endpoints.
|
||||||
|
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
|
||||||
|
- `piker/service/_actor_runtime.py`: validate daemon listeners.
|
||||||
|
- `piker/service/_daemon.py`: initialize and consume registrar options.
|
||||||
|
- `piker/ui/cli.py`: fallback to automatic chart binding.
|
||||||
|
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
|
||||||
|
- `tests/test_services.py`: defer test requiring missing datad API.
|
||||||
|
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human identified that datad tests arrived through the agent's
|
||||||
|
mixed cherry-picks and explicitly requested deferring that test. They
|
||||||
|
requested a chart default for existing configs, expanded the task to
|
||||||
|
all review findings, and directed checking whether a newer Tractor pin
|
||||||
|
could reduce the fix scope. No direct manual source edits were observed
|
||||||
|
during these changes; acceptance remains with the human.
|
||||||
|
|
||||||
|
During the human review pass, the human confirmed removing the Binance
|
||||||
|
pause and correcting CLI endpoint parsing. They requested an explicit
|
||||||
|
template note about this pin's single-transport limitation and detailed
|
||||||
|
comments distinguishing registrar discovery endpoints from daemon bind
|
||||||
|
endpoints. Those explanations were added at the human's direction.
|
||||||
|
|
@ -0,0 +1,61 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-07T16:54:53Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: gpt-6.1-sol
|
||||||
|
git_ref: bde2457b
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Fix all findings from the local PR94 review, prefer a Tractor pin
|
||||||
|
change if it reduces work, and export the original review for Tuicr.
|
||||||
|
|
||||||
|
## Generated changes
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- config/conf.toml`
|
||||||
|
|
||||||
|
Use TCP-only defaults, retaining commented UDS alternatives. Both
|
||||||
|
checked Tractor address branch tips still reject mixed transports.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
|
||||||
|
|
||||||
|
Make Click multiaddresses repeatable and normalize single strings.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
|
||||||
|
|
||||||
|
Validate requested daemon bind addresses instead of remote registrars.
|
||||||
|
Normalize accepted endpoints and allow TCP ephemeral port requests.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
|
||||||
|
|
||||||
|
Resolve explicit or inherited registrars independently of other daemon
|
||||||
|
options and consume the explicit keyword before forwarding kwargs.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
|
||||||
|
|
||||||
|
Default chart bind endpoints to None when network config is absent.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||||
|
|
||||||
|
Remove the unconditional market-info debugger pause.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
|
||||||
|
|
||||||
|
Remove the prematurely cherry-picked datad test and add offline startup
|
||||||
|
regressions, including a real registrar/child daemon lifecycle check.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Startup and logging tests: 13 passed.
|
||||||
|
- Existing runtime boot and tagged registrar tests: 2 passed.
|
||||||
|
- Scoped native Ruff and diff checks pass.
|
||||||
|
- The first new actor test used a removed nursery API; it was corrected
|
||||||
|
to the installed start_actor/Portal.run interface before passing.
|
||||||
|
- Import initially failed without native libraries. Tests use the
|
||||||
|
existing flake's GCC, zlib, Qt and PyQt paths without executing its
|
||||||
|
environment-rebuilding shell hook.
|
||||||
|
- Original seven findings exported locally as Tuicr review-add payloads
|
||||||
|
bound to bde2457b; no session comments or remote PR were changed.
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-10T23:28:52Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 1fa45dfa
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human rejected keeping the April Tractor pin as a landing baseline.
|
||||||
|
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
|
||||||
|
new rollback adaptations, reuse of original Piker histories where
|
||||||
|
possible, and a candidate prepared within their gateway relocation
|
||||||
|
window. They asked specifically whether the runtime validation review
|
||||||
|
hunk duplicated older endpoint history.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Select the reviewed address branch at an immutable locked revision and
|
||||||
|
preserve all committed application/test compatibility code. Verify the
|
||||||
|
minimal dependency boundary offline and preview existing consumer
|
||||||
|
fixes without importing unrelated datad/backfill changes. Distinguish
|
||||||
|
original-source test results from live teardown acceptance, which is
|
||||||
|
not available in this preparation window.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
|
||||||
|
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
|
||||||
|
preserving Polars 1.36.0 and the rest of the locked graph.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human redirected the task from an older known-good runtime to a
|
||||||
|
modern landing stack and insisted on reusing original history rather
|
||||||
|
than building new compatibility adapters. They proposed stricter
|
||||||
|
exception-group behavior as a hypothesis and set the live-test/time
|
||||||
|
constraints. Those directions materially changed the dependency
|
||||||
|
boundary. No direct manual edits to this final boundary were observed;
|
||||||
|
runtime acceptance and landing remain human-owned.
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-10T23:28:52Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 1fa45dfa
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Undo the April Tractor rollback and select the reviewed
|
||||||
|
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
|
||||||
|
commits to new adaptations, and prepare an offline-verified candidate
|
||||||
|
while live IB verification is deferred for gateway relocation.
|
||||||
|
|
||||||
|
## Generated changes
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||||
|
|
||||||
|
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
|
||||||
|
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
|
||||||
|
and test sources byte-for-byte to the committed PR94 tree. Keep the
|
||||||
|
rejected rollback journal only in ignored local experiment reports.
|
||||||
|
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
|
||||||
|
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
|
||||||
|
the resulting graph. Actual imports confirm the reviewed Tractor pin
|
||||||
|
and Trio 0.32.0, not the hypothesized Trio 0.33.
|
||||||
|
|
||||||
|
## Original-history previews
|
||||||
|
|
||||||
|
The existing MethodProxy stack is retained. Read-only merge previews
|
||||||
|
show original `4af885d9` and `b691422b` can be applied without the datad
|
||||||
|
cutover. Original-source regressions run through temporary in-memory
|
||||||
|
overlays against the real pinned runtime: seven cases pass and one
|
||||||
|
cache-hit case fails because `log.at_least_level('info')` looks up INFO
|
||||||
|
in a custom-level mapping. A separate standard logging correction
|
||||||
|
produces eight passing cases. Its patch and execution handoff are saved
|
||||||
|
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
|
||||||
|
cherry-picks or source edits are performed by this boundary.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Startup and logging: 13 passed.
|
||||||
|
- Runtime boot and tagged registrar normalization: 2 passed.
|
||||||
|
- Locked sync and source whitespace checks pass.
|
||||||
|
- Original application/test trees are unchanged from PR94 HEAD.
|
||||||
|
- Consumer overlay with explicit logging correction: 8 passed.
|
||||||
|
- Correction patch validates against its exact preview tree using a
|
||||||
|
private temporary index; the real index is untouched.
|
||||||
|
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: 495c0915
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
User reported a different-feed overlap failure and asked whether
|
||||||
|
a fix could be extracted without importing broader history work.
|
||||||
|
PDB confirmed MNQ March 2027 one-second history timed out and
|
||||||
|
returned None, producing NoData. After continuing PDB, sibling
|
||||||
|
history and quote tasks terminated while pikerd remained alive.
|
||||||
|
User requested continuing with the narrow backport.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Partial backport of the NoData/DataUnavailable catch from
|
||||||
|
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||||
|
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||||
|
A new parametrized regression uses real null detection and verifies
|
||||||
|
that a sibling in the same nursery survives the failed repair.
|
||||||
|
|
||||||
|
Both new cases failed before the catch. The focused history suites
|
||||||
|
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||||
|
IB reset churn or the separate Tractor completion-callback assertion.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/tsp/_history.py`: existing downstream exception catch.
|
||||||
|
- `tests/test_history_backfill.py`: sibling survival regression.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
User supplied live reproduction, PDB locals and cancellation logs,
|
||||||
|
and explicitly constrained scope to a small history extraction.
|
||||||
|
The production catch is copied from existing downstream commit
|
||||||
|
0846cbd4, not newly designed here. New regression is agent-authored;
|
||||||
|
no direct human source edits observed during this patch. Review and
|
||||||
|
live qualification remain with the user.
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: 495c0915
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||||
|
|
||||||
|
Partial backport of the NoData/DataUnavailable catch from
|
||||||
|
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||||
|
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||||
|
A new parametrized regression uses real null detection and verifies
|
||||||
|
that a sibling in the same nursery survives the failed repair.
|
||||||
|
|
||||||
|
Both new cases failed before the catch. The focused history suites
|
||||||
|
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||||
|
IB reset churn or the separate Tractor completion-callback assertion.
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: 495c0915
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human requested a commit plan, then a fast-forward landing of
|
||||||
|
the #94 branch and PR updates. They agreed the temporary local
|
||||||
|
Tractor source must be replaced, published piker_pin themselves,
|
||||||
|
and reported "pushed it". They also supplied the manual recovery
|
||||||
|
results and full Gateway restart failure diagnostics.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Restore the Git Tractor source after local qualification and lock
|
||||||
|
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||||
|
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||||
|
Record the human's overlap, data-reset and reopen observations,
|
||||||
|
the narrow history extraction, and full-restart recovery limits.
|
||||||
|
Offline uv lock validation passed without an environment sync.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
|
||||||
|
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
|
||||||
|
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human selected the bounded landing scope, performed live tests,
|
||||||
|
identified the more aggressive reset distinction, and published
|
||||||
|
the tested Tractor ref. The agent applied the metadata edits and
|
||||||
|
wrote the qualification record from those observations. No human
|
||||||
|
acceptance or task-state transitions are inferred by this record.
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: 495c0915
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- uv.lock`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
|
||||||
|
|
||||||
|
Restore the Git Tractor source after local qualification and lock
|
||||||
|
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||||
|
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||||
|
Record the human's overlap, data-reset and reopen observations,
|
||||||
|
the narrow history extraction, and full-restart recovery limits.
|
||||||
|
Offline uv lock validation passed without an environment sync.
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
# AI Prompt I/O Log
|
||||||
|
|
||||||
|
This directory records substantive AI-assisted changes, prompts,
|
||||||
|
original harness identity, and unedited outputs. Human contributors
|
||||||
|
remain accountable for code decisions.
|
||||||
|
|
||||||
|
Records follow the [NLNet generative AI policy][policy]. Each
|
||||||
|
structured entry links its immutable `.raw.md` partner. Code output
|
||||||
|
uses Git diff references to avoid duplication.
|
||||||
|
|
||||||
|
[policy]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||||
|
|
@ -0,0 +1,59 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:08:59Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Session-initiating instruction (driving all 7 commits in
|
||||||
|
this series):
|
||||||
|
|
||||||
|
> ok i want you to become the distributed runtime and
|
||||||
|
> concurrency expert for this project - namely acquire
|
||||||
|
> a deep understanding of tractor and how it's used.
|
||||||
|
> then i want you to attempt to factor our current
|
||||||
|
> brokerd service daemon into 2 daemons: a brokerd
|
||||||
|
> which only hosts live/paper trading endpoint tasks
|
||||||
|
> [..] a new `datad` subdaemon which in a separate
|
||||||
|
> subactor serves all the data feed service tasks [..]
|
||||||
|
> give me a mega detailed plan on how to approach this,
|
||||||
|
> and a staged approach for the implementation.
|
||||||
|
|
||||||
|
Proximate driver for THIS commit: during the approved
|
||||||
|
plan's stage-0 test gate the agent discovered test
|
||||||
|
subactors were writing into the user's REAL
|
||||||
|
`~/.config/piker/accounting/` files (a bogus paper fill
|
||||||
|
landed in `account.kraken.paper.toml`) because the old
|
||||||
|
test-dir override in `config.get_app_dir()` was
|
||||||
|
commented out and could never work in spawned
|
||||||
|
subactors anyway. Fix applied autonomously under the
|
||||||
|
approved plan; no explicit per-fix user prompt.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Restore `pytest` config-dir isolation by resolving the
|
||||||
|
per-test tmp dir lazily (at conf-path access time) from
|
||||||
|
`tractor` runtime-vars, which propagate down the actor
|
||||||
|
tree; route all conf-path derivation through
|
||||||
|
`config.get_conf_dir()` so the override is effective in
|
||||||
|
every (sub)actor.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/config.py` — add `_maybe_use_test_dir()`;
|
||||||
|
hook in `get_conf_dir()`; route `get_conf_path()` +
|
||||||
|
`load()` mkdir through it
|
||||||
|
- `piker/accounting/_ledger.py` — derive ledger dir
|
||||||
|
via `config.get_conf_dir()` (not the global)
|
||||||
|
- `piker/accounting/_pos.py` — same for account files
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:08:59Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/config.py`
|
||||||
|
|
||||||
|
Generated: `config._maybe_use_test_dir()` — lazily
|
||||||
|
reads `piker_test_dir` from
|
||||||
|
`tractor.runtime._state._runtime_vars['piker_vars']`
|
||||||
|
(pre-loaded by `open_piker_runtime()` from the
|
||||||
|
`tests.conftest._open_test_pikerd()` overrides) and
|
||||||
|
calls `_override_config_dir()` when set. Hooked at the
|
||||||
|
top of `get_conf_dir()`; `get_conf_path()` and
|
||||||
|
`load()`'s dir-creation rerouted through
|
||||||
|
`get_conf_dir()`.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
|
||||||
|
> `git log -1 -p --follow -- piker/accounting/_pos.py`
|
||||||
|
|
||||||
|
Generated: ledger/account dir derivation switched from
|
||||||
|
the `config._config_dir` module global to
|
||||||
|
`config.get_conf_dir()` + `mkdir(parents=True,
|
||||||
|
exist_ok=True)` for nested tmp-dir creation.
|
||||||
|
|
||||||
|
Key diagnostic reasoning (verbatim from session):
|
||||||
|
|
||||||
|
The old (commented-out) `get_app_dir()` override gated
|
||||||
|
on `'pytest' in sys.modules` which can NEVER work in
|
||||||
|
spawned subactors (fresh procs, no pytest import); as
|
||||||
|
a result test `paperboi`/daemon actors were writing
|
||||||
|
into the user's REAL `~/.config/piker/accounting/`
|
||||||
|
files. Evidence: `account.kraken.paper.toml` gained a
|
||||||
|
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
|
||||||
|
(a test fill), and `trades_kraken_paper.toml` (252B)
|
||||||
|
contained only that fill. The resolution must be
|
||||||
|
checked lazily at config-path access time (NOT import
|
||||||
|
time) since sub-actors only receive runtime-vars once
|
||||||
|
their `tractor` runtime has fully booted.
|
||||||
|
|
@ -0,0 +1,53 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:10:22Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
|
||||||
|
driver: the approved plan's per-stage test gates could
|
||||||
|
not run AT ALL — the branch base was broken vs.
|
||||||
|
`tractor` git `main` (`AttributeError: 'tuple' object
|
||||||
|
has no attribute 'unwrap'` at `pikerd` boot, stale
|
||||||
|
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
|
||||||
|
fixed forward autonomously to (re)establish the gate
|
||||||
|
baseline, continuing the `repair_tests` branch lineage
|
||||||
|
(verified already merged in ancestry via `git cherry`
|
||||||
|
during a user-requested branch-overlap survey).
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Port the service layer + test suites to current
|
||||||
|
`tractor` APIs: addr-type normalization in
|
||||||
|
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
|
||||||
|
kwarg renames, modern self-cancel absorption semantics
|
||||||
|
in the cancel-method test harness, exc/position attr
|
||||||
|
renames, a paper-EMS startup-budget bump and a syntax
|
||||||
|
fix in `.deribit.api`.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/service/_actor_runtime.py` — `wrap_address()`
|
||||||
|
normalize before `.unwrap()` in `open_pikerd()`
|
||||||
|
- `piker/service/_registry.py` — `check_for_service()`
|
||||||
|
-> `query_actor(regaddr=)` + 2-tuple yield +
|
||||||
|
`open_registry(addrs=)`
|
||||||
|
- `piker/brokers/deribit/api.py` — missing comma in
|
||||||
|
`tractor.trionics` import tuple
|
||||||
|
- `tests/test_services.py` — `registry_addr=` kwarg,
|
||||||
|
raddr unwraps, cancel-semantics harness rewrite,
|
||||||
|
`fail_after` 9 -> 19s
|
||||||
|
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,56 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:10:22Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
||||||
|
|
||||||
|
Generated: normalize each registry addr via
|
||||||
|
`tractor.discovery._addr.wrap_address()` before
|
||||||
|
`.unwrap()`-ing for the `accept_addrs` bind check —
|
||||||
|
entries may be raw `tuple`s when passed in from (test)
|
||||||
|
client code. Import-path precedent taken from
|
||||||
|
`piker/cli/__init__.py:336`.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/service/_registry.py`
|
||||||
|
|
||||||
|
Generated: `check_for_service()` ported to
|
||||||
|
`tractor.query_actor(name, regaddr=...)` (kwarg was
|
||||||
|
`arbiter_sockaddr=`), unpacking the new
|
||||||
|
`(sockaddr, portal)` yield, and passing the
|
||||||
|
now-required `open_registry(addrs=Registry.addrs)`.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- tests/test_services.py`
|
||||||
|
> `git log -1 -p --follow -- tests/test_ems.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
|
||||||
|
|
||||||
|
Key diagnostic reasoning (verbatim from session):
|
||||||
|
|
||||||
|
- the "DID NOT RAISE ContextCancelled" failure: in this
|
||||||
|
test the client actor IS pikerd (in-proc), and
|
||||||
|
current `tractor` main absorbs a `ContextCancelled`
|
||||||
|
whose canceller is your own actor — self-requested
|
||||||
|
cancels now exit cleanly instead of raising; the
|
||||||
|
'sigint' variant propagates a bare collapsed
|
||||||
|
`KeyboardInterrupt` rather than a
|
||||||
|
`BaseExceptionGroup`.
|
||||||
|
- the hard-coded `trio.fail_after(9)` startup budget is
|
||||||
|
marginal — full stack boot (pikerd -> emsd ->
|
||||||
|
brokerd.kraken -> paperboi + live kraken symbology
|
||||||
|
fetch) occasionally exceeds 9s -> bumped to 19s.
|
||||||
|
- `RemoteActorError.type` -> `.boxed_type`;
|
||||||
|
`Position.size` -> `.cumsize` (the paper engine
|
||||||
|
populates `BrokerdPosition.size` from `pp.cumsize`).
|
||||||
|
- overlap survey (user-requested): all of the
|
||||||
|
`repair_tests` branch commits are already in this
|
||||||
|
stack's ancestry; this commit finishes that branch's
|
||||||
|
port mission (its f4c4f1e2 fixed `conftest.py`'s
|
||||||
|
`arbiter_sockaddr` usage; this fixes the remaining
|
||||||
|
`test_services.py` + `check_for_service()` sites).
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:11:05Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||||
|
approved plan's "stage 0" prep. Plan-shaping user
|
||||||
|
decisions captured via in-session Q&A:
|
||||||
|
|
||||||
|
- `datad.<broker>` topology: sibling of
|
||||||
|
`brokerd.<broker>` under `pikerd` (vs. child
|
||||||
|
subactor).
|
||||||
|
- migration: hard cutover, staged by layer (no
|
||||||
|
dual-mode runtime flag).
|
||||||
|
- post-split `brokerd` scope: trading-only +
|
||||||
|
EMS-lazy-spawned (charts/CLI never touch it).
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Declare per-daemon-kind backend submod groups
|
||||||
|
(`_datad_mods`/`_brokerd_mods`) in the split-style
|
||||||
|
backends, keyed to the pre-existing
|
||||||
|
`piker.data.validate._eps` contract; add a
|
||||||
|
`validate.get_eps()` introspection helper; ws-import
|
||||||
|
hygiene in `.kraken.broker`. Zero behavior change
|
||||||
|
(`__enable_modules__` unions unchanged).
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/validate.py` — add `get_eps()`
|
||||||
|
- `piker/brokers/kraken/__init__.py` — mod groups
|
||||||
|
- `piker/brokers/binance/__init__.py` — mod groups
|
||||||
|
(note: no `symbols.py`; search eps live in `.feed`)
|
||||||
|
- `piker/brokers/deribit/__init__.py` — datad-only
|
||||||
|
groups (no `broker.py` yet)
|
||||||
|
- `piker/brokers/ib/__init__.py` — add `'api'` to the
|
||||||
|
pre-existing `_datad_mods`
|
||||||
|
- `piker/brokers/kraken/broker.py` — import
|
||||||
|
`NoBsWs`/`open_autorecon_ws` from
|
||||||
|
`piker.data._web_bs` directly
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:11:05Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/data/validate.py`
|
||||||
|
|
||||||
|
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
|
||||||
|
returning the daemon-kind's ep funcs defined by a
|
||||||
|
backend mod, keyed by ep name, missing eps excluded;
|
||||||
|
sourced from the existing `_eps` grouping table
|
||||||
|
(`'middleware' | 'datad' | 'brokerd'`).
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
|
||||||
|
|
||||||
|
Key exploration findings driving the design (from the
|
||||||
|
planning phase, 3 parallel explore agents + 1 architect
|
||||||
|
agent):
|
||||||
|
|
||||||
|
- the codebase already anticipated this split:
|
||||||
|
`validate._eps` groups eps into 'datad' vs 'brokerd'
|
||||||
|
kinds and `ib/__init__.py` already declared
|
||||||
|
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
|
||||||
|
carried the literal TODO "rename the daemon to datad
|
||||||
|
prolly once we split up broker vs. data tasks into
|
||||||
|
separate actors?".
|
||||||
|
- the feared kraken feed<->broker "shared ws state"
|
||||||
|
hazard is mild: `NoBsWs`/`open_autorecon_ws`
|
||||||
|
originate in `piker.data._web_bs` (re-exported via
|
||||||
|
`.kraken.feed`); only `stream_messages` is feed-local
|
||||||
|
and it's a pure parser — in-process imports only,
|
||||||
|
each actor opens its own ws conn.
|
||||||
|
- `enable_modules` gates RPC entry, NOT python imports,
|
||||||
|
so backend trading mods may keep importing feed-side
|
||||||
|
helpers in-process post-split.
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:11:42Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||||
|
approved plan's "stage 1": introduce the `datad` daemon
|
||||||
|
machinery additively (nothing routes through it yet).
|
||||||
|
User-decided constraint applied: `datad.<broker>` is a
|
||||||
|
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
|
||||||
|
via the existing `Services` + `maybe_spawn_daemon()`
|
||||||
|
machinery.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
New `piker/data/_daemon.py` hosting the
|
||||||
|
`datad.<broker>` feed-only daemon-actor: lifetime
|
||||||
|
fixture owning the actor-global `_FeedsBus`, init/spawn
|
||||||
|
fns mirroring `.brokers._daemon` conventions and the
|
||||||
|
`samplerd` sub-daemon precedent, plus root-mod
|
||||||
|
registration, `piker.service` re-exports and a spawn
|
||||||
|
test.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/_daemon.py` — NEW:
|
||||||
|
`_setup_persistent_datad()`, `datad_init()`,
|
||||||
|
`spawn_datad()`, `maybe_spawn_datad()`,
|
||||||
|
`_datad_service_mods`
|
||||||
|
- `piker/service/_actor_runtime.py` — add
|
||||||
|
`piker.data._daemon` to `_root_modules`
|
||||||
|
- `piker/service/__init__.py` — re-export spawn eps
|
||||||
|
- `tests/test_services.py` — add `test_datad_spawn`
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,60 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:11:42Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/data/_daemon.py`
|
||||||
|
|
||||||
|
Generated symbols + key design decisions:
|
||||||
|
|
||||||
|
- `_datad_service_mods: list[str]` — datad-always
|
||||||
|
enabled mods, the data-side successor to the old
|
||||||
|
`piker.brokers._daemon._data_mods` set; kept minimal
|
||||||
|
per the caps-sec model.
|
||||||
|
- `_setup_persistent_datad()` — `@tractor.context`
|
||||||
|
lifetime fixture: console-log setup then allocates
|
||||||
|
the actor-global feed bus via
|
||||||
|
`feed.get_feed_bus(brokername, service_nursery)`
|
||||||
|
exactly as the old brokerd fixture did, pinned open
|
||||||
|
with `ctx.started()` + `sleep_forever()`.
|
||||||
|
- `datad_init()` — actor name `f'datad.{brokername}'`;
|
||||||
|
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
|
||||||
|
`infect_asyncio=True`); builds `enable_modules` from
|
||||||
|
`getattr(brokermod, '_datad_mods',
|
||||||
|
getattr(brokermod, '__enable_modules__', []))` as
|
||||||
|
the flat-backend fallback.
|
||||||
|
- `spawn_datad()` — `Services.actor_n.start_actor()` +
|
||||||
|
`Services.start_service_task()` exactly mirroring
|
||||||
|
`spawn_brokerd()`; dedup-composes enable mods via
|
||||||
|
`list(dict.fromkeys(...))`.
|
||||||
|
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
|
||||||
|
service_name=f'datad.{brokername}', ...)`.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
||||||
|
> `git log -1 -p --follow -- piker/service/__init__.py`
|
||||||
|
> `git log -1 -p --follow -- tests/test_services.py`
|
||||||
|
|
||||||
|
Design rationale (verbatim from session):
|
||||||
|
|
||||||
|
- `_root_modules` must gain `piker.data._daemon` so
|
||||||
|
`pikerd_portal.run(spawn_datad, ...)` resolves in
|
||||||
|
the root.
|
||||||
|
- the `Services`-based impl style deliberately mirrors
|
||||||
|
`spawn_brokerd()` so the eventual `tractor.hilevel`
|
||||||
|
`ServiceMngr` port (see the `service_mng_to_tractor`
|
||||||
|
branch's d8c21d44 prep, surfaced by the
|
||||||
|
user-requested branch-overlap survey) lands
|
||||||
|
symmetrically on both spawn fns.
|
||||||
|
- mod placement (`piker/data/_daemon.py` vs.
|
||||||
|
generalizing `piker.brokers._daemon`) follows the
|
||||||
|
per-subsystem daemon-mod convention
|
||||||
|
(`.clearing._ems`, `.data._sampling`) and resolves
|
||||||
|
the existing TODO at `brokers/_daemon.py:49` ("move
|
||||||
|
this def to the `.data` subpkg").
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:12:26Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||||
|
approved plan's "stage 2": decouple the clearing layer
|
||||||
|
from `feed.portals` BEFORE the feed cutover so live
|
||||||
|
trading works at every stage boundary. User-decided
|
||||||
|
constraint applied: post-split `brokerd` is
|
||||||
|
trading-only and spawned LAZILY only by `emsd`'s
|
||||||
|
`open_brokerd_dialog()` path.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Kill the single coupling forcing feed + trading eps
|
||||||
|
into one actor (`Router.open_trade_relays()` pulling
|
||||||
|
its trades portal from `feed.portals[brokermod]`):
|
||||||
|
`open_brokerd_dialog()` now (maybe) spawns/finds
|
||||||
|
`brokerd.<broker>` itself and ONLY when a live
|
||||||
|
trades-ep will actually open; paper mode never touches
|
||||||
|
it. Pre-cutover this is a pure refactor (registry
|
||||||
|
lookup finds the same feed-spawned daemon).
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
|
||||||
|
re-sig + inner `acquire_live_portal()`;
|
||||||
|
`Router.maybe_open_brokerd_dialog()` drops `portal`
|
||||||
|
param; `open_trade_relays()` drops the
|
||||||
|
`feed.portals` lookup
|
||||||
|
- `piker/accounting/cli.py` — keyword-form `portal=`
|
||||||
|
override kept for the `piker ledger` ad-hoc actor
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:12:26Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/clearing/_ems.py`
|
||||||
|
|
||||||
|
Generated: new `open_brokerd_dialog()` signature
|
||||||
|
`(brokermod, exec_mode, fqme=None, portal=None,
|
||||||
|
loglevel=None)` with an inner `@acm
|
||||||
|
acquire_live_portal()` that yields the caller-provided
|
||||||
|
`portal` override (the `piker ledger` path) else
|
||||||
|
`maybe_spawn_brokerd(brokermod.name)` — designated THE
|
||||||
|
one place a live, credentialed `brokerd.<broker>` gets
|
||||||
|
booted post-split. The eager
|
||||||
|
`portal.open_context(trades_endpoint, ...)`
|
||||||
|
construction moved inside that block so the paper
|
||||||
|
short-circuit never acquires a live portal.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/accounting/cli.py`
|
||||||
|
|
||||||
|
Key analysis (verbatim from session):
|
||||||
|
|
||||||
|
- `feed.portals` had exactly ONE trading consumer:
|
||||||
|
`piker/clearing/_ems.py:671` (`portal =
|
||||||
|
feed.portals[brokermod]`) — the single coupling
|
||||||
|
forcing feed + trading into one actor.
|
||||||
|
- `piker ledger` (`accounting/cli.py:100-157`) is a
|
||||||
|
hidden consumer: it calls `broker_init()` directly,
|
||||||
|
spawns its own ad-hoc actor and passes the portal
|
||||||
|
positionally — the new signature must keep an
|
||||||
|
explicit `portal:` override param for this path.
|
||||||
|
- stage sequencing: clearing decouple lands BEFORE the
|
||||||
|
feed cutover, otherwise `feed.portals` would hand
|
||||||
|
emsd a `datad` portal and live `open_trade_dialog`
|
||||||
|
RPC would fail.
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:12:59Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||||
|
approved plan's "stage 3": the feed-layer hard cutover
|
||||||
|
(user-decided: no dual-mode runtime flag, each stage
|
||||||
|
lands fully working).
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
The topology flip: all data-feed consumers route to
|
||||||
|
`datad.<broker>`; the `brokerd` lifetime fixture slims
|
||||||
|
to trading-only; UI/CLI data-flavoured spawn sites
|
||||||
|
repointed; test expectations inverted incl. an explicit
|
||||||
|
negative assert that paper sessions spawn NO
|
||||||
|
`brokerd.<broker>`.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/feed.py` — `open_feed()` ->
|
||||||
|
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
|
||||||
|
assert -> `'datad'`; `install_brokerd_search` ->
|
||||||
|
`install_datad_search`; comment sweep
|
||||||
|
- `piker/brokers/_daemon.py` —
|
||||||
|
`_setup_persistent_brokerd()` slimmed (feed-bus
|
||||||
|
alloc moved to the datad fixture)
|
||||||
|
- `piker/ui/_app.py`, `piker/ui/cli.py`,
|
||||||
|
`piker/brokers/core.py`, `piker/brokers/cli.py`,
|
||||||
|
`piker/ui/kivy/option_chain.py` — spawn-site
|
||||||
|
repoints
|
||||||
|
- `tests/test_services.py` — datad assertions + the
|
||||||
|
no-`brokerd.kraken` negative check
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:12:59Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/data/feed.py`
|
||||||
|
|
||||||
|
Generated: `open_feed()` spawn cutover with the
|
||||||
|
`maybe_spawn_datad` import done "relative-direct" from
|
||||||
|
`._daemon` — NOT via `piker.service` — to dodge a
|
||||||
|
partial-init cycle: when `piker.data.feed` loads as
|
||||||
|
part of `piker.service.__init__` executing its own
|
||||||
|
`piker.data._daemon` import, the service pkg is
|
||||||
|
mid-init and its `maybe_spawn_datad` binding does not
|
||||||
|
exist yet. The `open_feed_bus()` local-state sanity
|
||||||
|
assert flips `'brokerd' in servicename` ->
|
||||||
|
`'datad' in servicename` (the only actor-name assert
|
||||||
|
in the tree, verified by grep).
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
||||||
|
|
||||||
|
Generated: `_setup_persistent_brokerd()` slimmed to
|
||||||
|
console-log setup + pinned-open ctx; drops the bus
|
||||||
|
alloc, the `assert not feed._bus`, the service nursery
|
||||||
|
(backend `open_trade_dialog()` ctxs own their task
|
||||||
|
trees) and the `eg.ExceptionGroup` handler. The
|
||||||
|
`piker ledger` ad-hoc actor enters this same slimmed
|
||||||
|
fixture — exactly what it needs.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/ui/_app.py`
|
||||||
|
> `git log -1 -p --follow -- piker/ui/cli.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/core.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/cli.py`
|
||||||
|
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
|
||||||
|
> `git log -1 -p --follow -- tests/test_services.py`
|
||||||
|
|
||||||
|
Verification (from session): per-suite gates green at
|
||||||
|
this commit (services 5-passed incl. the new negative
|
||||||
|
assert, feeds 3-passed); a headless live smoke
|
||||||
|
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
|
||||||
|
on an alt registry port) confirmed quotes flowing via
|
||||||
|
`datad.kraken` + `samplerd` with `check_for_service(
|
||||||
|
'brokerd.kraken') is None`. Known pre-existing flake
|
||||||
|
documented: `test_multi_fill_positions`' second
|
||||||
|
in-proc runtime boot wedges ~50% (zombie subactor w/
|
||||||
|
unread parent-IPC bytes); reproduced with the split
|
||||||
|
fully reverted so NOT a regression of this work.
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:13:44Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Same session-initiating `brokerd`-split instruction (see
|
||||||
|
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||||
|
approved plan's final "stage 4": caps-sec slimming of
|
||||||
|
the (live, credentialed) trading actor + the `ib`
|
||||||
|
dual-daemon `client_id` collision mitigation flagged in
|
||||||
|
the plan's risk register.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
|
||||||
|
spawn fails fast for datad-only backends with a "use
|
||||||
|
paper-mode" error; `ib`'s default api-gw `client_id`
|
||||||
|
gets a per-daemon-kind offset so `datad.ib` +
|
||||||
|
`brokerd.ib` don't collide on connect.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
|
||||||
|
`_brokerd_service_mods`; `broker_init()` reads
|
||||||
|
`_brokerd_mods` (fallback `__enable_modules__`);
|
||||||
|
`spawn_brokerd()` fail-fast via `validate.get_eps()`
|
||||||
|
- `piker/brokers/ib/api.py` — role-based `client_id`
|
||||||
|
offset in `load_aio_clients()`
|
||||||
|
- `piker/cli/__init__.py` — resolved "expose datad"
|
||||||
|
TODO
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated.
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:13:44Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); backfilled from the live dev
|
||||||
|
session transcript per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
||||||
|
|
||||||
|
Generated: the fail-fast originally landed in
|
||||||
|
`broker_init()` but was relocated to `spawn_brokerd()`
|
||||||
|
mid-implementation after realizing `piker ledger`
|
||||||
|
calls `broker_init()` directly even for paper accounts
|
||||||
|
on datad-only backends (would have crashed the cli);
|
||||||
|
the service-spawn path is the correct enforcement
|
||||||
|
seam. Error text:
|
||||||
|
|
||||||
|
Backend 'kucoin' offers NO `brokerd` (live
|
||||||
|
order-control) eps!? It is likely a datad-only
|
||||||
|
provider, use paper-mode for clearing instead.
|
||||||
|
|
||||||
|
(verified live via a `trio.run()` unit check.)
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
||||||
|
|
||||||
|
Generated: in `load_aio_clients()`, when `client_id`
|
||||||
|
is the 6116 default: `datad`-named actors offset +16
|
||||||
|
(disjoint from `brokerd`'s linear `client_id + i`
|
||||||
|
retry range), other non-`brokerd` (ad-hoc test/cli)
|
||||||
|
actors +32. Rationale from the plan's risk register:
|
||||||
|
post-split BOTH per-broker daemons connect to the same
|
||||||
|
TWS/gw endpoint; a shared default id collides and
|
||||||
|
burns up to `connect_timeout * retries` (90s) in
|
||||||
|
retry cycles.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/cli/__init__.py`
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T17:33:09Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
||||||
|
scope: docs
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The session-initiating instruction, verbatim (this doc
|
||||||
|
IS its "mega detailed plan" deliverable):
|
||||||
|
|
||||||
|
> ok i want you to become the distributed runtime and
|
||||||
|
> concurrency expert for this project - namely acquire
|
||||||
|
> a deep understanding of tractor and how it's used.
|
||||||
|
> then i want you to atttempt to factor our current
|
||||||
|
> brokerd service daemon into 2 daemons:
|
||||||
|
>
|
||||||
|
> - a brokerd which only hosts live/paper trading
|
||||||
|
> endoint tasks as defined namely within all
|
||||||
|
> piker/brokers/<backend>/broker.py mods
|
||||||
|
> - a new `datad` subdaemon which in a separate
|
||||||
|
> subactor serves all the data feed service tasks
|
||||||
|
> namely delivered by endpoints in
|
||||||
|
> piker/brokers/<backend>/feed.py (or similar for
|
||||||
|
> less mature backends) and as more rigorously
|
||||||
|
> defined by the validation machinery in
|
||||||
|
> piker.data.validate.
|
||||||
|
>
|
||||||
|
> give me a mega detailed plan on how to approach
|
||||||
|
> this, and a staged approach for the implementation.
|
||||||
|
|
||||||
|
Plan-shaping user decisions (in-session Q&A): sibling
|
||||||
|
topology under `pikerd`; hard cutover staged by layer;
|
||||||
|
trading-only EMS-lazy-spawned `brokerd`. The human
|
||||||
|
staged the doc copy into `ai/claude-code/plans/` and
|
||||||
|
requested this commit after the 7 implementation
|
||||||
|
commits landed.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
The staged design doc for the `brokerd` -> (`datad` +
|
||||||
|
`brokerd`) split: context, target supervision topology,
|
||||||
|
load-bearing verified facts, per-stage file-level
|
||||||
|
changes with gates, risk register and an end-to-end
|
||||||
|
verification matrix. Produced via 3 parallel explore
|
||||||
|
agents + 1 architect agent + human Q&A before any code
|
||||||
|
was written; all 7 implementation commits in this
|
||||||
|
branch reference its stages in their provenance
|
||||||
|
entries.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `ai/claude-code/plans/datad_service.md` — the design
|
||||||
|
plan doc (human-staged copy of the AI-authored plan)
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None to the doc content — committed as generated. NB:
|
||||||
|
the implementation deviated from the plan-as-written in
|
||||||
|
4 places, see the raw file's deviation log.
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T17:33:09Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry; the AI output here is the
|
||||||
|
committed doc itself (a design plan, not code) so no
|
||||||
|
verbatim copy is duplicated:
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
|
||||||
|
|
||||||
|
## Implementation deviation log
|
||||||
|
|
||||||
|
Where the landed 7-commit series differs from the plan
|
||||||
|
as written (recorded for reviewer transparency; the
|
||||||
|
plan doc is committed unmodified):
|
||||||
|
|
||||||
|
1. Commit ordering: the `pytest` config-dir isolation
|
||||||
|
fix lands BEFORE the `tractor`-API drift port. The
|
||||||
|
plan's stage gates assumed a runnable+isolated test
|
||||||
|
baseline; per-commit gating exposed that without
|
||||||
|
isolation the paper-EMS test reads the user's real
|
||||||
|
(polluted) `account.kraken.paper.toml` and reddens.
|
||||||
|
2. The plan's "stage 0: full pytest green" gate
|
||||||
|
required first repairing pre-existing branch
|
||||||
|
breakage vs `tractor` git `main` (boot
|
||||||
|
`AttributeError`, stale discovery/exc/position
|
||||||
|
APIs) — that repair became its own commit
|
||||||
|
("Port service+tests to latest `tractor` APIs")
|
||||||
|
rather than plan-stage work.
|
||||||
|
3. Stage-4 fail-fast placement: the plan said
|
||||||
|
`broker_init()` raises on brokerd-ep-less backends;
|
||||||
|
implementation moved the raise to `spawn_brokerd()`
|
||||||
|
since `piker ledger` calls `broker_init()` directly
|
||||||
|
even for paper accounts on datad-only backends.
|
||||||
|
4. `brokers/_daemon.py` change grouping: the
|
||||||
|
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
|
||||||
|
type-only import) landed with the caps-sec slim
|
||||||
|
commit instead of the fixture-slim commit, keeping
|
||||||
|
each intermediate tree import-clean without
|
||||||
|
sub-hunk surgery.
|
||||||
|
|
||||||
|
Also of record: the plan's "Verification" matrix was
|
||||||
|
executed as written (per-suite gates each stage, the
|
||||||
|
headless datad-feed smoke, the kucoin fail-fast unit
|
||||||
|
check); the known pre-existing ~50% second-runtime-boot
|
||||||
|
test wedge was characterized and excluded as a
|
||||||
|
regression via revert-testing.
|
||||||
|
|
@ -0,0 +1,60 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||||
|
timestamp: 2026-06-10T21:35:49Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
User bug report from live `ib` testing of the
|
||||||
|
(datad|brokerd)-split branch:
|
||||||
|
|
||||||
|
> ok doing some testing and noticing live orders do not
|
||||||
|
> work otb, pikerd show this on live submission,
|
||||||
|
> [pdb traceback: `RuntimeError("Can not order
|
||||||
|
> {symbol}, no live feed?")` at `ib/api.py:1152`
|
||||||
|
> `submit_limit()` with `self._contracts == {}` in the
|
||||||
|
> `brokerd.ib` actor + a `TrioTaskExited` teardown
|
||||||
|
> cascade]
|
||||||
|
|
||||||
|
Follow-up user direction fixing the design intent:
|
||||||
|
|
||||||
|
> i think the main thing here is that [brokerd] should
|
||||||
|
> be able to always submit orders without a live feed
|
||||||
|
> being up and registered in the same subactor right?
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Root cause: pre-split the feed code populated the
|
||||||
|
actor-local `Client._contracts` cache (via
|
||||||
|
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
|
||||||
|
SAME process as order submission; post-split the
|
||||||
|
trading actor's client is never warmed. Fix: lazily
|
||||||
|
qualify + cache the contract per order request in
|
||||||
|
`handle_order_requests()` by running the same
|
||||||
|
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
|
||||||
|
plus per-order error relay (`BrokerdError`) so one bad
|
||||||
|
submission can't crash the whole trades dialog (the
|
||||||
|
`TrioTaskExited` storm was teardown cascade from the
|
||||||
|
original raise).
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/ib/broker.py` — thread `proxies` into
|
||||||
|
`handle_order_requests()`; lazy contract qualify on
|
||||||
|
cache-miss; guard `submit_limit()` w/ `BrokerdError`
|
||||||
|
relay; uncomment the (anticipatory) `get_mkt_info`
|
||||||
|
import
|
||||||
|
- `piker/brokers/ib/api.py` — fix the non-f-string
|
||||||
|
raise msg + document the new qualification contract
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None — committed as generated. Live `ib` order retest
|
||||||
|
performed by the human post-commit.
|
||||||
|
|
@ -0,0 +1,57 @@
|
||||||
|
---
|
||||||
|
model: claude-fable-5[1m]
|
||||||
|
service: claude
|
||||||
|
timestamp: 2026-06-10T21:35:49Z
|
||||||
|
git_ref: datad_service
|
||||||
|
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
||||||
|
---
|
||||||
|
|
||||||
|
NOTE: diff-ref mode entry (code committed in the same
|
||||||
|
commit as this log); recorded from the live debug
|
||||||
|
session per the `/prompt-io` skill rules.
|
||||||
|
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
|
||||||
|
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
||||||
|
|
||||||
|
Key diagnostic chain (from session):
|
||||||
|
|
||||||
|
- pdb showed `Client._contracts == {}` inside
|
||||||
|
`brokerd.ib`'s `submit_limit()`; the cache has
|
||||||
|
exactly TWO write sites: `Client.find_contracts()`
|
||||||
|
(api.py, keys `f'{sym}.{exch}.ib'`) and
|
||||||
|
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
|
||||||
|
'nvda.nasdaq' — NO `.ib` suffix).
|
||||||
|
- `BrokerdOrder.symbol` arrives as the bs_fqme form
|
||||||
|
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
|
||||||
|
site produces the key `submit_limit()` reads —
|
||||||
|
ie. pre-split it was the feed's in-proc
|
||||||
|
`get_mkt_info(sym, proxy=proxy)` call keeping orders
|
||||||
|
working, NOT `find_contracts()`.
|
||||||
|
- the existing TODO at `symbols.py:642-644` literally
|
||||||
|
predicted this: "this is going to be problematic
|
||||||
|
if/when we split out the datad vs. brokerd actors
|
||||||
|
since the mktmap lookup table will now be
|
||||||
|
inaccessible.."
|
||||||
|
- instance identity verified: `proxy._aio_ns` IS the
|
||||||
|
same `Client` obj as `_accounts2clients[account]`
|
||||||
|
(both sourced from the `load_aio_clients()` cache via
|
||||||
|
`open_client_proxies()`), so a brokerd-side
|
||||||
|
`get_mkt_info(fqme, proxy=proxies[account])` warms
|
||||||
|
exactly the dict `submit_limit()` reads. It also
|
||||||
|
populates `client._cons2mkts` which the
|
||||||
|
position-audit path (`broker.py` backup-table code)
|
||||||
|
needs in this actor anyway.
|
||||||
|
- the `TrioTaskExited` storm in the user's log
|
||||||
|
(`recv_trade_updates`, `open_aio_client_method_relay`
|
||||||
|
aio tasks) is teardown cascade: the raise crashed
|
||||||
|
`handle_order_requests` -> nursery teardown ripped
|
||||||
|
the trio sides of still-running aio relay tasks.
|
||||||
|
Hence the added per-order try/except ->
|
||||||
|
`BrokerdError` relay hardening so a single bad
|
||||||
|
submission degrades to an EMS error msg instead of
|
||||||
|
killing the backend's entire order-ctl dialog.
|
||||||
|
|
||||||
|
Verification: `tests/test_services.py` (5 passed) +
|
||||||
|
`tests/test_ems.py` (6 passed) regression-green; live
|
||||||
|
`ib` submission retest delegated to the human (needs a
|
||||||
|
running TWS/gw).
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
# AI Prompt I/O Log — claude
|
||||||
|
|
||||||
|
This directory tracks prompt inputs and model
|
||||||
|
outputs for AI-assisted development using
|
||||||
|
`claude` (claude-code CLI).
|
||||||
|
|
||||||
|
## Policy
|
||||||
|
|
||||||
|
Prompt logging follows the
|
||||||
|
[NLNet generative AI policy][nlnet-ai].
|
||||||
|
All substantive AI contributions are logged
|
||||||
|
with:
|
||||||
|
- Model name and version
|
||||||
|
- Timestamps
|
||||||
|
- The prompts that produced the output
|
||||||
|
- Unedited model output (`.raw.md` files)
|
||||||
|
|
||||||
|
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Entries are created by the `/prompt-io` skill
|
||||||
|
or automatically via `/commit-msg` integration.
|
||||||
|
|
||||||
|
Each commit carrying AI-generated changes links
|
||||||
|
to its provenance entry via a `Prompt-IO:`
|
||||||
|
commit-msg trailer; entries use "diff-ref mode"
|
||||||
|
(pointers into `git log -p` instead of verbatim
|
||||||
|
code copies) to avoid duplicating committed
|
||||||
|
code.
|
||||||
|
|
||||||
|
Human contributors remain accountable for all
|
||||||
|
code decisions. AI-generated content is never
|
||||||
|
presented as human-authored work.
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-22T00:13:57Z
|
||||||
|
git_ref: 16f3cd6f
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260722T001357Z_16f3cd6f_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continue the gap-checker and history-backfiller repair plan. Start
|
||||||
|
with deterministic analysis characterization, then fix canonical
|
||||||
|
sorting, dedupe-derived columns, positive gap detection, and null
|
||||||
|
grouping.
|
||||||
|
|
||||||
|
During review, the user challenged the replacement null-grouping
|
||||||
|
algorithm because its initial tests did not assert exact endpoints.
|
||||||
|
The user explicitly required retaining detailed, human-oriented
|
||||||
|
algorithm documentation such as the existing ASCII segment map.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Added deterministic analysis regressions and corrected normalization
|
||||||
|
so sorting and duplicate resolution precede derived-column
|
||||||
|
calculation. Unified null grouping for NumPy and Polars inputs,
|
||||||
|
documented its exact inclusive-boundary contract with an adapted
|
||||||
|
visual map, and verified all small null layouts against an independent
|
||||||
|
state-machine oracle. Retained sorted smart-dedupe output.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/tsp/_anal.py` - canonical gap and documented null grouping
|
||||||
|
- `piker/tsp/_dedupe_smart.py` - sorted duplicate resolution
|
||||||
|
- `tests/test_tsp_analysis.py` - exact and exhaustive regressions
|
||||||
|
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not yet been edited by the human.
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-22T00:13:57Z
|
||||||
|
git_ref: 16f3cd6f
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
||||||
|
|
||||||
|
Generated canonical gap and null-segment normalization. Derived
|
||||||
|
datetime and delta columns are recomputed after final sorting and
|
||||||
|
duplicate resolution, and negative deltas no longer become ordinary
|
||||||
|
gaps.
|
||||||
|
|
||||||
|
NumPy and Polars null grouping share one implementation while
|
||||||
|
retaining the human-oriented visual map of contiguous zero runs. The
|
||||||
|
documented contract defines absolute inclusive endpoints, margin
|
||||||
|
expansion, frame boundary clamping, contiguous-index validation, and
|
||||||
|
requested-column selection.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_dedupe_smart.py`
|
||||||
|
|
||||||
|
Adjusted smart dedupe to honor sorted output and resolve duplicate
|
||||||
|
timestamps without retaining stale order.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
||||||
|
|
||||||
|
Added deterministic synthetic OHLCV coverage for sorted dedupe, fresh
|
||||||
|
derived columns, and positive gap detection. Null grouping is checked
|
||||||
|
against readable endpoint examples and an independent state-machine
|
||||||
|
oracle over every layout through six rows, three margin widths, and
|
||||||
|
both NumPy and Polars inputs.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
`timeout -k 5 30 python -m pytest -q tests/test_tsp_analysis.py`
|
||||||
|
|
||||||
|
Result: 11 passed.
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-22T00:13:58Z
|
||||||
|
git_ref: 16f3cd6f
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continue the structured backfill phase after the analysis regressions.
|
||||||
|
Characterize provider exhaustion and genuinely empty frames, then remove
|
||||||
|
completion paths that can hang after child-task exit.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Removed orphan-prone completion events from the structured nursery flow
|
||||||
|
and checked empty provider frames before indexing timestamp endpoints.
|
||||||
|
Added deterministic timeout-bounded regressions for both exit paths.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
|
||||||
|
- `tests/test_history_backfill.py` - deterministic exit regressions
|
||||||
|
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not yet been edited by the human.
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-22T00:13:58Z
|
||||||
|
git_ref: 16f3cd6f
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||||
|
|
||||||
|
Removed redundant completion events whose consumers waited only after
|
||||||
|
their owning nurseries had joined. Added empty provider-frame handling
|
||||||
|
before timestamp endpoint indexing so provider exhaustion exits cleanly.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||||
|
|
||||||
|
Added bounded, actor-free regressions for `DataUnavailable` and empty
|
||||||
|
provider-frame completion.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
|
||||||
|
|
||||||
|
Result: 2 passed.
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-22T02:41:33Z
|
||||||
|
git_ref: 3a4f6737
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260722T024133Z_3a4f6737_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user challenged the analysis commit message's claim that only
|
||||||
|
positive sample-step gaps are detected and asked whether out-of-order
|
||||||
|
timestamps were simply ignored. After confirming the missing separate
|
||||||
|
reporting path, the user instructed the agent to implement the fix.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Separated missing-history gaps from timestamp-ordering errors without
|
||||||
|
changing the existing `dedupe()` return shape. Added a structured
|
||||||
|
detector and made normalization warn before sorting removes evidence of
|
||||||
|
duplicates or reversed timestamps.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/tsp/_anal.py` - ordering detector and dedupe reporting
|
||||||
|
- `piker/tsp/__init__.py` - public detector export
|
||||||
|
- `tests/test_tsp_analysis.py` - exact ordering-error regression
|
||||||
|
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not yet been edited by the human.
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-22T02:41:33Z
|
||||||
|
git_ref: 3a4f6737
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
||||||
|
|
||||||
|
Added `detect_time_ordering_errors()` to return non-positive timestamp
|
||||||
|
steps as structured rows. `dedupe()` now warns with those rows before
|
||||||
|
sorting and duplicate removal erase the original ordering evidence.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
|
||||||
|
|
||||||
|
Exported the ordering detector through the public TSP package.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
||||||
|
|
||||||
|
Changed the negative-delta regression to verify exact predecessor,
|
||||||
|
timestamp, and delta values, confirm exclusion from missing-history
|
||||||
|
gaps, and require explicit normalization reporting.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- `tests/test_tsp_analysis.py`: 11 passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-22T02:41:34Z
|
||||||
|
git_ref: 3a4f6737
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260722T024134Z_3a4f6737_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user asked why the in-progress `piker/storage` changes were omitted
|
||||||
|
from the earlier commit plan. After receiving an inventory of unfinished
|
||||||
|
Phase 3 requirements, the user instructed the agent to complete the work.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Finished non-destructive, atomic NativeDB updates and integrated them
|
||||||
|
with history startup and reverse backfill. Expanded tests through two
|
||||||
|
adversarial review rounds that caught and corrected cross-client lock
|
||||||
|
blocking, stale temp indexing, discontinuous persisted indexes, startup
|
||||||
|
snapshot races, missing latest-frame persistence, notification wedges,
|
||||||
|
and mutable/truncated SHM persistence.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/__init__.py` - incremental storage contract
|
||||||
|
- `piker/storage/nativedb.py` - validated atomic merge persistence
|
||||||
|
- `piker/storage/marketstore/__init__.py` - non-duplicate update mode
|
||||||
|
- `piker/tsp/_history.py` - provider-delta and startup persistence
|
||||||
|
- `tests/test_storage_nativedb.py` - NativeDB durability regressions
|
||||||
|
- `tests/test_history_backfill.py` - persistence lifecycle regressions
|
||||||
|
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not yet been edited by the human.
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-22T02:41:34Z
|
||||||
|
git_ref: 3a4f6737
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/__init__.py piker/storage/nativedb.py piker/storage/marketstore/__init__.py`
|
||||||
|
|
||||||
|
Added an incremental storage update contract. NativeDB now validates
|
||||||
|
canonical numeric OHLCV data, serializes writers both actor-locally and
|
||||||
|
through cancellable filesystem locks, merges with incoming-wins seam
|
||||||
|
resolution, regenerates contiguous indexes, reopens and validates a
|
||||||
|
same-directory temporary parquet, fsyncs it, atomically replaces the
|
||||||
|
target, fsyncs the directory, and publishes cache only after replacement.
|
||||||
|
|
||||||
|
NativeDB indexing ignores lock and crash-left temporary files.
|
||||||
|
MarketStore uses its non-duplicate write mode for incremental updates.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||||
|
|
||||||
|
Separated full provider deltas from capacity-truncated SHM pushes.
|
||||||
|
Backfill persists before SHM mutation, publishes bounded notifications,
|
||||||
|
and delays latest-frame publication until the concurrent pre-update
|
||||||
|
storage snapshot completes.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py tests/test_history_backfill.py`
|
||||||
|
|
||||||
|
Added deterministic coverage for merge preservation, conflict policy,
|
||||||
|
atomic failures, reopened-candidate validation, cache consistency,
|
||||||
|
schema and timestamp invariants, contiguous index regeneration,
|
||||||
|
separate series, stale temp indexing, cancellable file-lock contention,
|
||||||
|
full provider-delta persistence, latest-frame ordering, and bounded
|
||||||
|
notification teardown.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- deterministic analysis/backfill/storage set: 31 passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
- final adversarial review: no findings
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: unavailable
|
||||||
|
timestamp: 2026-07-27T20:47:11Z
|
||||||
|
git_ref: 84a6d47b
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260727T204711Z_84a6d47b_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user identified that `backfiller_deep_fixes` introduced persistent
|
||||||
|
`.parquet.lock` files, questioned whether Parquet or Arrow already handled
|
||||||
|
writer coordination, and clarified that piker's `tractor` structured
|
||||||
|
concurrency design already provides single-writer ownership through the
|
||||||
|
persistent `datad` feed task. The user then asked to open the backfiller
|
||||||
|
worktree and resolve the redundant locking.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Removed redundant actor-local and filesystem locking while preserving
|
||||||
|
atomic crash-safe Parquet replacement. Replaced the synthetic contention
|
||||||
|
test with regressions for the actual cross-branch failure: writes create
|
||||||
|
no lock sidecars, and legacy lock artifacts are ignored during indexing.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/nativedb.py` - rely on `datad` writer ownership
|
||||||
|
- `tests/test_storage_nativedb.py` - lock-sidecar regressions
|
||||||
|
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-27T20:47:11Z
|
||||||
|
git_ref: 84a6d47b
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
|
||||||
|
|
||||||
|
Removed actor-local and filesystem writer locks from NativeDB. The
|
||||||
|
persistent `datad` feed task already owns writes for each series, while
|
||||||
|
its timeframe children write distinct files. NativeDB retains validated
|
||||||
|
temporary writes, file and directory synchronization, and atomic target
|
||||||
|
replacement for crash durability. The exact `.parquet` suffix filter is
|
||||||
|
retained so legacy lock sidecars are harmless.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
|
||||||
|
|
||||||
|
Removed the synthetic cross-client lock-contention test. Added regression
|
||||||
|
coverage proving replacement and incremental writes create no lock
|
||||||
|
sidecars. Extended indexing coverage with the legacy
|
||||||
|
`.parquet.lock` artifact that caused upstream `flake_update` to crash.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- NativeDB and history regression set: 20 passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-27T21:24:54Z
|
||||||
|
git_ref: 689df816
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260727T212454Z_689df816_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user reran the known IB FQME on `backfiller_deep_fixes` and reported
|
||||||
|
that `publish_latest_frame()` still failed because NativeDB required an
|
||||||
|
`index` absent from IB's provider frame. The user called out the missed
|
||||||
|
tabular schema boundary and requested an actual end-to-end test suite
|
||||||
|
rather than storage fakes.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Repaired the IB-provider-to-NativeDB boundary by canonicalizing durable
|
||||||
|
fields and mapping reloads into provider-specific SHM buffers. Added a
|
||||||
|
local actor/SHM/Parquet integration regression covering first startup and
|
||||||
|
restart, plus focused conversion and timestamp-coercion tests. Two review
|
||||||
|
rounds found and resolved restart hydration and fractional-time hazards.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/nativedb.py` - canonical storage and SHM field mapping
|
||||||
|
- `piker/tsp/_anal.py` - name-based Polars-to-NumPy conversion
|
||||||
|
- `tests/test_history_backfill.py` - actor/SHM/Parquet integration test
|
||||||
|
- `tests/test_storage_nativedb.py` - schema conversion regressions
|
||||||
|
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-27T21:24:54Z
|
||||||
|
git_ref: 689df816
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/tsp/_anal.py`
|
||||||
|
|
||||||
|
Normalized index-less provider frames to NativeDB's canonical names,
|
||||||
|
order, and dtypes before merge and publication. Provider-only columns
|
||||||
|
are excluded from durable files, derived indexes are regenerated, and
|
||||||
|
fractional timestamps are rejected before integer conversion. Added a
|
||||||
|
canonical field map for hydrating provider-specific SHM buffers while
|
||||||
|
leaving provider-only fields at their defaults. Changed Polars-to-NumPy
|
||||||
|
conversion to select columns by field name rather than position.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py tests/test_storage_nativedb.py`
|
||||||
|
|
||||||
|
Added deterministic coverage using IB's actual provider dtype and a real
|
||||||
|
local `tractor` root actor, IB-typed shared-memory buffers, NativeDB,
|
||||||
|
Parquet persistence, first-start publication, and restart hydration.
|
||||||
|
Added focused regressions for reordered/extra Polars columns and
|
||||||
|
fractional timestamp rejection.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- NativeDB and history integration set: 23 passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
- second adversarial review: no findings
|
||||||
|
- no live gateway or network tests run
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-07-27T21:59:06Z
|
||||||
|
git_ref: 689df816
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260727T215906Z_689df816_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
After the actor/SHM/Parquet integration test exposed a stale SHM triplet,
|
||||||
|
the user requested leak-cleaner machinery in the pytest harness using
|
||||||
|
Tractor's existing ownership and lifetime patterns.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Added creator-scoped SHM leak tracking to pytest without scanning
|
||||||
|
`/dev/shm` or treating attachments as owned. The fixture cooperates with
|
||||||
|
normal Tractor teardown, verifies POSIX object identity before fallback
|
||||||
|
cleanup, restores process-local token state, and fails after cleaning a
|
||||||
|
leak. Regressions cover the exact pre-actor-registration allocation window,
|
||||||
|
positional creators, and external attachment safety.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `tests/conftest.py` - creator-scoped SHM leak tracking fixture
|
||||||
|
- `tests/test_shm_cleanup.py` - ownership and failure-window regressions
|
||||||
|
- `.claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
- document current-process SHM cleanup behavior
|
||||||
|
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-27T21:59:06Z
|
||||||
|
git_ref: 689df816
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/conftest.py tests/test_shm_cleanup.py .claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
|
||||||
|
Added function-scoped SHM ownership tracking around Tractor's test-process
|
||||||
|
factory. The fixture records only successful creators, removes ownership
|
||||||
|
records during normal Tractor teardown, verifies POSIX object identity
|
||||||
|
before fallback unlink, restores the token-cache baseline, and fails after
|
||||||
|
cleaning any leak. Regressions reproduce allocation failure before actor
|
||||||
|
lifetime registration and prove external attachments remain untouched.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- SHM cleanup regressions: 2 passed
|
||||||
|
- combined SHM, NativeDB, and history set: 25 passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
- final adversarial review: no findings
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
||||||
|
timestamp: 2026-07-28T00:32:47Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260728T003247Z_ed85721c_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Build a safe xonsh workflow for qualifying NativeDB and reverse-backfill
|
||||||
|
correctness against a known gappy IB chart, beginning with
|
||||||
|
`mnq.cme.20260918`. Cover baseline, fresh, restart, append, and expansion
|
||||||
|
testing through implementation, adversarial review, and verification.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Implemented a private disposable xonsh qualification workflow. Evidence is
|
||||||
|
tied to a fixed case, report checksum, snapshot checksum, worktree import,
|
||||||
|
and phase metadata. Added explicit path and symlink safety, source
|
||||||
|
replacement handling, xonsh child environments, and lifecycle preservation
|
||||||
|
comparisons.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `snippets/nativedb_backfill_audit.xsh` - disposable qualification helpers
|
||||||
|
- `docs/manual_backfill_qualification.rst` - operator lifecycle runbook
|
||||||
|
- `tests/test_backfill_audit_snippet.py` - xonsh and comparison regressions
|
||||||
|
- `.claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
- qualification test-harness guidance
|
||||||
|
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-28T00:32:47Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- snippets/nativedb_backfill_audit.xsh docs/manual_backfill_qualification.rst tests/test_backfill_audit_snippet.py .claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
|
||||||
|
Added sourceable xonsh helpers that create a private disposable root, bind
|
||||||
|
child processes to the reviewed worktree, capture checksum-bound phase
|
||||||
|
evidence, safely seed and clear test history, and compare lifecycle
|
||||||
|
preservation.
|
||||||
|
|
||||||
|
The workflow rejects path traversal, pre-existing symlinks, output
|
||||||
|
collisions, mismatched case/report/snapshot evidence, and same-phase
|
||||||
|
comparison. The accompanying runbook covers known-bad replay, fresh
|
||||||
|
backfill, restart, append, gap triage, and expansion cases.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- xonsh qualification helper regressions: passed
|
||||||
|
- xonsh sourceability: passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
|
||||||
|
Adversarial review drove additional evidence binding, worktree executable,
|
||||||
|
environment propagation, source replacement, and descriptor-level safety
|
||||||
|
fixes. Residual concurrent mutation risk is stated explicitly in the private
|
||||||
|
disposable-root runbook.
|
||||||
|
|
@ -0,0 +1,40 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
||||||
|
timestamp: 2026-07-28T00:50:34Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260728T005034Z_ed85721c_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Add reusable read-only tooling to qualify raw NativeDB Parquet against a
|
||||||
|
known gappy IB chart, beginning with `mnq.cme.20260918`. Preserve malformed
|
||||||
|
evidence and distinguish structural validity from unclassified positive
|
||||||
|
gaps without opening or mutating the storage runtime.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Implemented raw Parquet audit reporting with human and JSON output, bounded
|
||||||
|
cadence details, exact integer arithmetic, strict status, collision-safe
|
||||||
|
snapshot capture, and source path protections. Regressions cover malformed
|
||||||
|
schema and values, timestamp precision, gap arithmetic, corrupt bytes,
|
||||||
|
copy failures, symlinks, traversal, and read-only CLI behavior.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/_audit.py` - raw Parquet audit and snapshot implementation
|
||||||
|
- `piker/storage/cli.py` - `piker store audit` command
|
||||||
|
- `tests/test_storage_audit.py` - audit and CLI regressions
|
||||||
|
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-28T00:50:34Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/_audit.py piker/storage/cli.py tests/test_storage_audit.py`
|
||||||
|
|
||||||
|
Added a raw, read-only NativeDB Parquet audit with human and JSON output,
|
||||||
|
bounded cadence evidence, exact integer arithmetic, pre-parse snapshot
|
||||||
|
capture, and strict qualification status.
|
||||||
|
|
||||||
|
The command rejects unsafe FQME paths, symlinked sources and outputs,
|
||||||
|
destination collisions, non-regular source nodes, malformed canonical
|
||||||
|
schema, non-finite values, invalid timestamps, noncanonical indexes, and
|
||||||
|
sub-period cadence. Copy-failure cleanup preserves source bytes while
|
||||||
|
completed corrupt snapshots remain available for diagnosis.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- storage audit and CLI regressions: passed
|
||||||
|
- Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
- final adversarial review: no findings
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
|
||||||
|
timestamp: 2026-07-28T23:27:18Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260728T232718Z_ed85721c_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Diagnose the live `pikerd` failure where backfill publication raised
|
||||||
|
`ValueError: OHLCV timestamps must be finite and positive` while merging the
|
||||||
|
known gappy MNQ NativeDB baseline with fresh IB history. Determine whether to
|
||||||
|
commit the pending audit stack or iterate first.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Traced the primary failure through `publish_latest_frame()` to
|
||||||
|
`NativeStorageClient.update_ohlcv()`. The incoming frame was valid, but the
|
||||||
|
canonicalized persisted baseline retained its epoch-zero row and caused the
|
||||||
|
merged frame to fail validation. Added persisted-row repair before merge and
|
||||||
|
a regression using the observed legacy schema characteristics.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/nativedb.py` - drop non-positive persisted rows during merge
|
||||||
|
- `tests/test_storage_nativedb.py` - live MNQ failure-shape regression
|
||||||
|
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.raw.md`
|
||||||
|
- unedited response record
|
||||||
|
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md`
|
||||||
|
- provenance metadata and response summary
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated changes have not been edited by the human.
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
provider: openai
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-28T23:27:18Z
|
||||||
|
git_ref: ed85721c
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py tests/test_storage_nativedb.py`
|
||||||
|
|
||||||
|
Fixed incremental NativeDB updates against legacy persisted history with
|
||||||
|
non-positive timestamps. Incoming provider frames remain strictly validated,
|
||||||
|
while merge repair discards invalid persisted rows before concat, dedupe,
|
||||||
|
ordering, canonical-index rewrite, and atomic publication.
|
||||||
|
|
||||||
|
The regression reproduces the observed MNQ baseline shape: one epoch-zero
|
||||||
|
row, noncanonical absolute indexes, extra derived columns, and a valid fresh
|
||||||
|
IB frame. It proves the invalid row is removed while all valid old and new
|
||||||
|
bars survive with canonical indexes.
|
||||||
|
|
||||||
|
Verification generated with the patch:
|
||||||
|
|
||||||
|
- NativeDB regressions: 18 passed
|
||||||
|
- history backfill regressions: 6 passed
|
||||||
|
- focused Ruff: passed
|
||||||
|
- `git diff --check`: passed
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-08-11T20:58:26Z
|
||||||
|
git_ref: 06d5ea5d
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Diagnose and fix an IB live-order failure where lazy NVDA contract
|
||||||
|
qualification raised `tractor.trionics.Lagged` after the order reached
|
||||||
|
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
|
||||||
|
searches caused the stale state.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Traced the failure to competing consumers of a one-slot Trio broadcast, not
|
||||||
|
the recent SHM change or stale asyncio client state. Replaced broadcast
|
||||||
|
fan-out with one response/status demultiplexer, retained request-ID
|
||||||
|
correlation, handled cancellation and teardown, documented the task/channel
|
||||||
|
ownership model, warned on diagnostic metadata eviction, and added
|
||||||
|
deterministic proxy regressions.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
|
||||||
|
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
|
||||||
|
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated output follows the user's live failure diagnosis.
|
||||||
|
|
@ -0,0 +1,66 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-11T20:58:26Z
|
||||||
|
git_ref: 06d5ea5d
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
The user reported that certain IB order entries might have broken and
|
||||||
|
provided a live failure for a first NVDA sell order. EMS delivered the
|
||||||
|
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
|
||||||
|
qualification failed in `MethodProxy._run_method()` with:
|
||||||
|
|
||||||
|
```text
|
||||||
|
tractor.trionics._broadcast.Lagged:
|
||||||
|
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
|
||||||
|
dropped `0` values
|
||||||
|
```
|
||||||
|
|
||||||
|
The user suspected a cancelled symbol-search task might have left stale
|
||||||
|
asyncio state. Cancellation can leave an orphaned method response, but the
|
||||||
|
observed failure was stale Trio broadcast state. `open_client_proxy()`
|
||||||
|
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
|
||||||
|
and an idle method receiver. Status traffic advanced only the relay, so the
|
||||||
|
first method call for an uncached symbol entered an already-lagged receiver.
|
||||||
|
Orders for eagerly cached contracts bypassed this path.
|
||||||
|
|
||||||
|
The immediately preceding sparse-history commit did not modify IB order,
|
||||||
|
symbol, or proxy code; it exposed no direct dependency on this failure.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
|
||||||
|
|
||||||
|
Generated a sole-reader message demultiplexer for each IB client proxy.
|
||||||
|
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
|
||||||
|
status/error traffic, and dispatches `mid`-tagged responses into per-call
|
||||||
|
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
|
||||||
|
late responses to be logged and dropped without skewing later calls. Clean
|
||||||
|
channel EOF is accepted during normal proxy teardown.
|
||||||
|
|
||||||
|
Follow-up review restored the masked reconnect TODO and per-message case
|
||||||
|
rationale, documented the one-reader/many-writer task hierarchy and separate
|
||||||
|
per-proxy channels, and added a warning with request method metadata when the
|
||||||
|
bounded unresolved-request diagnostic table evicts its oldest entry.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
|
||||||
|
|
||||||
|
Generated deterministic regressions for idle status traffic before first
|
||||||
|
qualification, two concurrent calls with reverse-order responses, cancelled
|
||||||
|
calls with late responses followed by a successful call, and graceful
|
||||||
|
channel EOF. Every synchronization wait is timeout-bounded.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
|
||||||
|
Registered the deterministic IB method-proxy target and source mapping.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
........................ [100%]
|
||||||
|
25 passed, 4 warnings in 3.44s
|
||||||
|
```
|
||||||
|
|
||||||
|
The warnings are existing Tractor/Trio deprecations from the EMS test.
|
||||||
|
Adversarial review drove a clean-EOF guard and bounded polling. Final review
|
||||||
|
found no actionable issues. Live IB order validation remains pending and
|
||||||
|
requires restarting `brokerd.ib`.
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: unavailable
|
||||||
|
timestamp: 2026-08-13T20:24:02Z
|
||||||
|
git_ref: 12b0d66e
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continue the paired piker broadcast-consumer work after Tractor PR 490
|
||||||
|
stabilization. Stop cached feed callers from mutating or consuming shared
|
||||||
|
quote-stream roots, then review, verify and prepare a commit boundary.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Made provider root streams private to the cached `Feed`, gave every caller a
|
||||||
|
local subscribed view, corrected feed cache identity, and made multi-provider
|
||||||
|
relays own synchronized child subscriptions and independent send-channel
|
||||||
|
clones.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
|
||||||
|
subscribed multi-provider relays with independent channel lifetimes.
|
||||||
|
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
|
||||||
|
cache-identity regressions, including sibling survival after provider EOC.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated output follows the requested cached-feed ownership item.
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-13T20:24:02Z
|
||||||
|
git_ref: 12b0d66e
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
The user asked to continue the paired piker broadcast-consumer work after
|
||||||
|
Tractor PR 490 stabilization. The next item was to stop cached feed callers
|
||||||
|
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
|
||||||
|
review, verify and prepare a commit boundary.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||||
|
|
||||||
|
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
|
||||||
|
root streams and every caller, including the first cache owner, receives a
|
||||||
|
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
|
||||||
|
The cache identity now includes normalized full symbol shape and stream policy.
|
||||||
|
|
||||||
|
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
|
||||||
|
of directly consuming a shared stream, and synchronizes relay startup before
|
||||||
|
yielding the merged channel. Each relay owns a cloned send channel so one
|
||||||
|
provider EOC cannot terminate its siblings. The module-level relay returns its
|
||||||
|
live child receiver through `TaskStatus` for typed retention and introspection.
|
||||||
|
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
|
||||||
|
shared by local cache users.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
|
||||||
|
|
||||||
|
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
|
||||||
|
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
|
||||||
|
multi-provider relay fan-out and startup, duplex controls, lexical child
|
||||||
|
closure, sibling survival after provider EOC, cached-root immutability, and
|
||||||
|
full cache-key differentiation.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
... [100%]
|
||||||
|
3 passed in 0.02s
|
||||||
|
|
||||||
|
. [100%]
|
||||||
|
1 passed in 0.01s
|
||||||
|
```
|
||||||
|
|
||||||
|
The second result is the existing EMS broadcast ownership regression. Ruff,
|
||||||
|
E501, compilation and whitespace checks passed. Repeated adversarial review
|
||||||
|
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
|
||||||
|
relay-startup and first-owner asymmetry issues; final review reported no
|
||||||
|
findings.
|
||||||
|
|
@ -0,0 +1,32 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: unavailable
|
||||||
|
timestamp: 2026-08-13T23:07:44Z
|
||||||
|
git_ref: ddac3553
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continue the piker broadcast-consumer audit after the Deribit ownership commit
|
||||||
|
and prepare the next smallest ownership fix.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
|
||||||
|
context and gave every caller a lexical child from one retained
|
||||||
|
`LinkedTaskChannel` broadcaster root.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
|
||||||
|
ownership.
|
||||||
|
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
|
||||||
|
lifetime regression.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated output follows the requested broadcast-consumer audit.
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-13T23:07:44Z
|
||||||
|
git_ref: ddac3553
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
The user asked to continue the piker broadcast-consumer audit after committing
|
||||||
|
the Deribit ownership fix. The next isolated item was IB's actor-local
|
||||||
|
`_quote_streams` cache, which retained a raw channel owned by its first caller
|
||||||
|
and constructed competing broadcaster roots for later callers.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
||||||
|
|
||||||
|
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
|
||||||
|
made `open_aio_quote_stream()` cache that context by symbol through
|
||||||
|
`maybe_open_context()`. Every caller now receives a lexical child from the
|
||||||
|
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
|
||||||
|
until its final cache user exits. Removed the manual `_quote_streams` registry
|
||||||
|
and teardown mutation.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
|
||||||
|
|
||||||
|
Added a concurrent ownership regression using real Tractor broadcast state.
|
||||||
|
It proves miss and hit callers use the same symbol/context factory, receive the
|
||||||
|
same quote through distinct children, close independently, and leave the owner
|
||||||
|
live after the hit exits.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
. [100%]
|
||||||
|
1 passed in 0.01s
|
||||||
|
```
|
||||||
|
|
||||||
|
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
|
||||||
|
Python compilation and whitespace checks pass. Normal package collection
|
||||||
|
remains blocked by the existing Qt environment mismatch.
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: tractor-addr-unpacking
|
||||||
|
timestamp: 2026-08-20T16:35:19Z
|
||||||
|
git_ref: 5f631e19
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human asked to implement the Piker compatibility work caused by
|
||||||
|
tractor's canonical tagged-address migration, run the previously
|
||||||
|
deferred tractor verification, and keep runtime-managed WireGuard
|
||||||
|
bindspaces in a later separate PR.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Made Piker normalize legacy and tagged registry inputs before semantic
|
||||||
|
comparison and process-global storage. Updated runtime address typing and
|
||||||
|
test assertions for canonical output, retained legacy caller input, and
|
||||||
|
removed a duplicate pytest log-level option that blocked downstream test
|
||||||
|
collection with current tractor.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/service/_registry.py` - canonical registry normalization and
|
||||||
|
shared address typing.
|
||||||
|
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
|
||||||
|
annotations.
|
||||||
|
- `tests/conftest.py` - canonical fixture comparison and shared pytest
|
||||||
|
option ownership.
|
||||||
|
- `tests/test_services.py` - registry regression and canonical runtime
|
||||||
|
assertions.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human explicitly requested the downstream implementation and chose
|
||||||
|
to keep WireGuard bindspace lifecycle work in a separate future tractor
|
||||||
|
PR. The agent implemented and tested those directions; no direct manual
|
||||||
|
source edits were observed.
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-20T16:35:19Z
|
||||||
|
git_ref: 5f631e19
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
# Raw output - tractor tagged-address compatibility
|
||||||
|
|
||||||
|
The human requested downstream Piker qualification after tractor PR
|
||||||
|
#505 switched transport address emission to canonical tagged tuples.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
|
||||||
|
|
||||||
|
Normalized legacy and tagged registry addresses through tractor's
|
||||||
|
address codec before Piker stores or compares them. Extended service
|
||||||
|
runtime annotations to cover both tuple shapes, updated runtime tests to
|
||||||
|
compare canonical serialized values, and removed Piker's duplicate
|
||||||
|
registration of tractor's shared `--ll` pytest option.
|
||||||
|
|
||||||
|
Ruff passed. The focused normalization regression and Piker runtime boot
|
||||||
|
passed against the local tractor checkout at `5d92595f`. The wider
|
||||||
|
service module reached an unrelated pre-existing logger-name assertion
|
||||||
|
in `maybe_spawn_daemon()` after both address-focused tests passed.
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: tractor-addr-unpacking
|
||||||
|
timestamp: 2026-08-21T04:20:58Z
|
||||||
|
git_ref: b7aca4c0
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human approved refining Piker's logger-name handling after the
|
||||||
|
wider service test exposed a stale parent-logger assertion during the
|
||||||
|
tractor tagged-address compatibility work.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
|
||||||
|
preserving complete requested submodule names. Removed daemon startup
|
||||||
|
assertions that incorrectly required parent logger names and added a
|
||||||
|
regression covering both logger factories and names that the old
|
||||||
|
character-set stripping silently corrupted.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/log.py` - exact relative logger-name derivation.
|
||||||
|
- `piker/service/_daemon.py` - remove stale parent-name assertion.
|
||||||
|
- `piker/data/_daemon.py` - remove stale parent-name assertion.
|
||||||
|
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
|
||||||
|
- `piker/clearing/_ems.py` - remove redundant logger assertion.
|
||||||
|
- `tests/test_logging.py` - package logger namespace regressions.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human explicitly approved this follow-up after reviewing the
|
||||||
|
residual Piker service-suite gap. The agent implemented and verified
|
||||||
|
the refinement; no direct manual source edits were observed.
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
---
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-21T04:20:58Z
|
||||||
|
git_ref: b7aca4c0
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
# Raw output - preserve Piker logger namespaces
|
||||||
|
|
||||||
|
The human approved a follow-up refinement after the full Piker service
|
||||||
|
module exposed a stale logger-name assertion while qualifying tractor
|
||||||
|
PR #505.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
|
||||||
|
|
||||||
|
Replaced character-set `str.lstrip()` calls with exact package-prefix
|
||||||
|
removal, removed assertion-only parent-logger expectations from daemon
|
||||||
|
startup paths, and added regression coverage for both logger factories
|
||||||
|
and prefix-sensitive submodule names.
|
||||||
|
|
||||||
|
Ruff passed after removing newly unused logger assignments. All four
|
||||||
|
logger regressions and all six service integration tests passed against
|
||||||
|
the local tractor PR #505 checkout.
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: unavailable
|
||||||
|
timestamp: 2026-08-26T22:26:00Z
|
||||||
|
git_ref: 5f631e19
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Diagnose and fix the hanging Binance USD-M perpetual feed for
|
||||||
|
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
|
||||||
|
API and changelog. Ensure unsupported or silent websocket feeds fail
|
||||||
|
loudly into `datad` and chart consumers instead of blocking startup.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Migrates Binance USD-M market data from the retired legacy websocket to
|
||||||
|
the split public and market routes, handles Binance's new aggregate-trade
|
||||||
|
schema, and introduces bounded connection and first-data startup. Direct
|
||||||
|
stream URLs avoid reconnect-time subscription-ACK races, while targeted
|
||||||
|
tests preserve routing and fail-loud behavior.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
|
||||||
|
Binance quote streams.
|
||||||
|
- `piker/brokers/binance/venues.py` - define current USD-M websocket
|
||||||
|
endpoints.
|
||||||
|
- `piker/data/_web_bs.py` - support bounded websocket context startup.
|
||||||
|
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
|
||||||
|
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
|
||||||
|
retain the generated-output record.
|
||||||
|
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
|
||||||
|
structured provenance for the code changes.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human explicitly required fail-loud behavior rather than merely
|
||||||
|
updating the endpoint, and supplied a direct CLI reproduction proving
|
||||||
|
both `datad.binance` and the chart caller blocked. Those requirements
|
||||||
|
drove the connection and first-data deadlines and their regressions. The
|
||||||
|
changes remain uncommitted for human review.
|
||||||
|
|
@ -0,0 +1,57 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-26T22:26:00Z
|
||||||
|
git_ref: 5f631e19
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user reported that selecting
|
||||||
|
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
|
||||||
|
websocket but never loaded the feed or downstream chart. They asked
|
||||||
|
OpenCode to compare Binance's API changelog with piker's FQME and feed
|
||||||
|
code, diagnose the failure, and solve it.
|
||||||
|
|
||||||
|
The user clarified that future provider failures must raise loudly
|
||||||
|
instead of silently blocking feed and chart startup. They also confirmed
|
||||||
|
the same hang from
|
||||||
|
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
|
||||||
|
|
||||||
|
## Generated code
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||||
|
|
||||||
|
Routes USD-M book-ticker and aggregate-trade data through Binance's new
|
||||||
|
`/public` and `/market` websocket paths, merges both sockets into the
|
||||||
|
existing quote stream, accepts the new `st` aggregate-trade field, and
|
||||||
|
requires both stream classes before declaring startup complete. It also
|
||||||
|
removes the misleading missing-provider warning, derives feed mode from
|
||||||
|
resolved pair metadata, and raises `DataUnavailable` for connection or
|
||||||
|
live-data startup timeouts.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
|
||||||
|
|
||||||
|
Defines Binance's current public and market USD-M websocket endpoints
|
||||||
|
and makes the market endpoint the regular futures websocket API root.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Adds an optional connection deadline to `open_autorecon_ws()` so callers
|
||||||
|
can fail context entry when repeated websocket handshakes never become
|
||||||
|
connected.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||||
|
|
||||||
|
Adds regressions for exact Binance stream routing, silent live-data
|
||||||
|
startup, silent websocket connection startup, and the current USD-M
|
||||||
|
aggregate-trade payload schema.
|
||||||
|
|
||||||
|
## Verification output
|
||||||
|
|
||||||
|
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
|
||||||
|
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
|
||||||
|
through piker's merged stream received both required stream classes and
|
||||||
|
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
|
||||||
|
spot.
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
|
||||||
|
timestamp: 2026-08-28T20:03:52Z
|
||||||
|
git_ref: dd676857
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Update the xonsh version used by the repository dev shell before stacking
|
||||||
|
the Binance work, while retaining the dotrc compatibility shim as an
|
||||||
|
immediate workaround for older shells.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
|
||||||
|
The new release supports Python 3.13, handles Git alias completion natively,
|
||||||
|
and works in the updated Nix development shell.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `pyproject.toml` - require xonsh `0.24.2` or newer.
|
||||||
|
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
|
||||||
|
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
|
||||||
|
the triggering prompt, generated configuration summary, and checks.
|
||||||
|
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
|
||||||
|
structured provenance for the dependency update.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The user identified old xonsh behavior as the root cause, reported the
|
||||||
|
separate dotrc compatibility shim, and directed that the repository update
|
||||||
|
remain a distinct flake-layer change before the Binance stack continues.
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-28T20:03:52Z
|
||||||
|
git_ref: dd676857
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user traced broken multiline conflict-resolution pastes to the old
|
||||||
|
xonsh release installed by the repository dev shell. A separate dotrc
|
||||||
|
agent added a compatibility shim, and the user directed that xonsh still
|
||||||
|
be updated alongside the flake work before continuing the Binance stack.
|
||||||
|
|
||||||
|
## Generated configuration
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||||
|
|
||||||
|
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
|
||||||
|
refreshes only xonsh's locked distribution metadata. The selected release
|
||||||
|
supports Python 3.13, includes native Git-config alias completion, and fixes
|
||||||
|
Nix metadata checks.
|
||||||
|
|
||||||
|
## Verification output
|
||||||
|
|
||||||
|
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
|
||||||
|
xonsh from `0.22.4` to `0.24.2`.
|
||||||
|
- Fresh `nix develop` replaces the worktree-local xonsh installation.
|
||||||
|
- `xonsh --version` reports `0.24.2`.
|
||||||
|
- A multi-command xonsh invocation executes Git commands successfully.
|
||||||
|
- `nix flake check --no-build` passes for `x86_64-linux`.
|
||||||
|
- `git diff --check` passes.
|
||||||
|
|
||||||
|
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
|
||||||
|
that its distributions are yanked for an `unpivot` regression.
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||||
|
timestamp: 2026-08-29T17:55:42Z
|
||||||
|
git_ref: e09e306a
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
|
||||||
|
stack, then add missing deterministic tests and correctness fixes as a
|
||||||
|
separate follow-up with a complete editor-gated commit plan.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Hardens Binance feed startup around its supported one-symbol contract,
|
||||||
|
direct websocket routes, current provider schemas, and first-quote
|
||||||
|
readiness. Adds deterministic Binance regressions and separates generic
|
||||||
|
reconnect behavior into its own tested commit boundary.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
|
||||||
|
USD-M stream classes, and declare readiness after either quote class.
|
||||||
|
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
|
||||||
|
and documented book-ticker and aggregate-trade schemas.
|
||||||
|
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
|
||||||
|
the triggering prompt, generated-code summary, and verification output.
|
||||||
|
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
|
||||||
|
structured provenance for the follow-up.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The user rejected reconstructing already committed Binance work, required
|
||||||
|
preservation through cherry-pick, selected the flake/xonsh stack base, and
|
||||||
|
required the review fixes to remain a separate tested boundary. The user also
|
||||||
|
corrected the workflow to require an automatic `/commit-plan` at every stack
|
||||||
|
review gate. During Tuicr review, the user rejected trade-gated startup,
|
||||||
|
requested a feed-specific initialization error and async-generator TODO,
|
||||||
|
and required generic reconnect changes and tests to land independently.
|
||||||
|
The user then clarified that either L1 or trade data must unblock startup,
|
||||||
|
rather than preferring one stream class or waiting independently for both.
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-29T17:55:42Z
|
||||||
|
git_ref: e09e306a
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user asked to land recent branch work in small chronological patch sets,
|
||||||
|
including missing tests. For Binance, the user required preserving the
|
||||||
|
existing committed topic patch through cherry-pick and placing newly found
|
||||||
|
correctness repairs in a separate follow-up boundary above the flake/xonsh
|
||||||
|
stack. The user also required every yielded stack layer to include a complete
|
||||||
|
`/commit-plan` with patch-specific checks.
|
||||||
|
|
||||||
|
## Generated code
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||||
|
|
||||||
|
Restricts Binance feeds to the supported one-symbol contract, renders direct
|
||||||
|
quote routes, separates L1 liveness from first-trade readiness, and derives
|
||||||
|
the contextual trade timeout from Trio's enclosing effective deadline.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Lets structured cancellation bypass reconnect throttling and ensures handled
|
||||||
|
handshake rejections retry before accessing connection-local cancel state.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||||
|
|
||||||
|
Adds deterministic regressions for symbol cardinality, route selection,
|
||||||
|
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
|
||||||
|
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
|
||||||
|
|
||||||
|
## Verification output
|
||||||
|
|
||||||
|
- Targeted Binance tests: `10 passed in 0.26s`.
|
||||||
|
- Ruff passes for all three changed Python files.
|
||||||
|
- `git diff --check` passes.
|
||||||
|
- Independent final review reports no actionable findings.
|
||||||
|
- No live Binance network probe was run.
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||||
|
timestamp: 2026-09-01T02:42:35Z
|
||||||
|
git_ref: e09e306a
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Address the Tuicr review by separating generic websocket reconnect
|
||||||
|
changes from Binance-specific work and testing the real reconnect task
|
||||||
|
hierarchy rather than a reimplementation.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Preserves structured cancellation and safe first-handshake retries in
|
||||||
|
`_web_bs`. Adds three focused regressions in a dedicated websocket test
|
||||||
|
module and packages them as an independent commit boundary.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/_web_bs.py` - restart handled handshake failures and let
|
||||||
|
cancellation bypass the reconnect throttle.
|
||||||
|
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
|
||||||
|
and cancellation task paths with controlled transport boundaries.
|
||||||
|
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
|
||||||
|
- retain the review prompt, generated-code summary, and verification.
|
||||||
|
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
|
||||||
|
structured provenance for the generic reconnect boundary.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The user identified that these changes were generic rather than
|
||||||
|
Binance-specific, required a separate commit and test module, requested
|
||||||
|
coverage of the real reconnect hierarchy, and corrected code prose to
|
||||||
|
refer to `trio` consistently. A follow-up review required the production
|
||||||
|
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
|
||||||
|
handler and cite the regression which audits that teardown invariant.
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-09-01T02:42:35Z
|
||||||
|
git_ref: e09e306a
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user requested that generic `_web_bs` reconnect fixes and tests be
|
||||||
|
separated from the Binance backend patch. The review also required tests
|
||||||
|
to exercise the real reconnect task hierarchy, cite the previous faulty
|
||||||
|
branches, and consistently name `trio` in code prose.
|
||||||
|
|
||||||
|
## Generated code
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Ensures handled handshake failures immediately restart the reconnect
|
||||||
|
loop and lets structured cancellation bypass shielded error throttling.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||||
|
|
||||||
|
Adds deterministic scheduler-real regressions for handshake startup
|
||||||
|
timeouts, immediate rejection retries, and cancellation propagation.
|
||||||
|
|
||||||
|
## Verification output
|
||||||
|
|
||||||
|
- Generic and Binance targeted tests: `9 passed in 0.06s`.
|
||||||
|
- Ruff passes for all four changed Python files.
|
||||||
|
- `git diff --check` passes.
|
||||||
|
- No live network probe was run.
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: 4dc920cf-e82e-4c1e-83db-75a0a9efe82b
|
||||||
|
timestamp: 2026-09-01T23:00:43Z
|
||||||
|
git_ref: e91f41a6
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260901T230043Z_e91f41a6_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Update `wkt/land_flake_update` to Tractor's restored `piker_pin` after
|
||||||
|
reusing the existing runtime compatibility commits. Keep stale
|
||||||
|
historical lock snapshots out of the final dependency state.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Updates piker's Python and `msgspec` requirements for current Tractor,
|
||||||
|
then generates one fresh lock at Tractor `piker_pin@0d95a9e6`. The
|
||||||
|
dependency commit completes the compatibility stack already
|
||||||
|
cherry-picked onto the flake branch.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `pyproject.toml` - require Python 3.13 and `msgspec>=0.20.0`.
|
||||||
|
- `uv.lock` - resolve Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`,
|
||||||
|
and current transport dependencies.
|
||||||
|
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.raw.md`
|
||||||
|
- preserve the prompt, generated-change summary, and resolution notes.
|
||||||
|
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md` - record
|
||||||
|
structured provenance for the dependency update.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The user identified that Tractor compatibility should land with the
|
||||||
|
flake update rather than the Binance patch, supplied four known source
|
||||||
|
commits, required a broader local-history audit for missing ports, and
|
||||||
|
directed a fresh lock from the restored `piker_pin` instead of replaying
|
||||||
|
historical pins.
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-09-01T23:00:43Z
|
||||||
|
git_ref: e91f41a6
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user requested that the flake-update branch absorb the Tractor
|
||||||
|
runtime compatibility work before the Binance branch lands. They
|
||||||
|
directed reuse of existing compatibility commits, exclusion of stale
|
||||||
|
historical lock pins, and a fresh update to Tractor's restored
|
||||||
|
`piker_pin`.
|
||||||
|
|
||||||
|
## Generated changes
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||||
|
|
||||||
|
Narrows piker support to Python 3.13 and raises the direct `msgspec`
|
||||||
|
requirement to the version family required by current Tractor.
|
||||||
|
|
||||||
|
> `git diff --cached -- uv.lock`
|
||||||
|
|
||||||
|
Relocks against Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`, and the
|
||||||
|
current transport dependency graph while retaining the existing Xonsh
|
||||||
|
update.
|
||||||
|
|
||||||
|
## Resolution notes
|
||||||
|
|
||||||
|
- Historical lock hunks from reused commits were intentionally omitted.
|
||||||
|
- Tractor runtime compatibility remains represented by the preceding
|
||||||
|
individually preserved commits on `wkt/land_flake_update`.
|
||||||
|
- Project checks are rendered in the commit plan for human execution.
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-21T22:44:21Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 507c75c6
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261021T224421Z_507c75c6_piker_pin_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human asked to sanity-check the current PR94 landing state against
|
||||||
|
the qualified new Tractor pin before deduplicating branch history. After
|
||||||
|
the initial offline checks passed, the human authorized changing the
|
||||||
|
selector to piker_pin, resolving the lock, and verifying the installed
|
||||||
|
dependency without the temporary local-source PYTHONPATH override.
|
||||||
|
The human explicitly authorized network resolution and a scoped update
|
||||||
|
of Tractor/runtime dependencies while preserving unrelated packages.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Updated the branch selector and lock to published piker_pin@b64c89f0.
|
||||||
|
Installed the exact locked Tractor revision, verified its installed
|
||||||
|
source metadata, and reran 14 offline logger/startup cases without a
|
||||||
|
local Tractor source override. Preserved the rest of the environment.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `pyproject.toml`: select piker_pin and bound Polars below 2.0.
|
||||||
|
- `uv.lock`: lock b64c89f0 and Polars/runtime 1.44.2.
|
||||||
|
- This provenance pair.
|
||||||
|
|
||||||
|
## Verification and limits
|
||||||
|
|
||||||
|
14 distinct offline cases passed against the installed Tractor update
|
||||||
|
with the existing Polars 1.36 installation. The actor.uid deprecation
|
||||||
|
warning remains. No live provider/order testing occurred.
|
||||||
|
|
||||||
|
The initial resolver selected Polars 2.0 and rejected retaining 1.36
|
||||||
|
for its Windows target. The human then requested the latest pre-2.0
|
||||||
|
release instead. PyPI metadata identified non-yanked stable 1.44.2;
|
||||||
|
adding `<2.0.0` to the existing lower bound and relocking succeeded
|
||||||
|
with both polars and polars-runtime-32 at 1.44.2, without platform edits.
|
||||||
|
The raw partner retains the original, superseded 2.0 generation history.
|
||||||
|
|
||||||
|
The scoped install changed only Tractor. Polars 1.44.2 is locked but
|
||||||
|
not installed or data-path qualified yet. The human's existing staged
|
||||||
|
checkpoint is preserved; the pre-2.0 correction is in the worktree.
|
||||||
|
No history cleanup, staging or commit was performed by this agent.
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-21T22:44:21Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 507c75c6
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||||
|
|
||||||
|
Changed the landing branch's Tractor selector from wkt/addr_unpacking
|
||||||
|
to piker_pin and resolved the published branch to
|
||||||
|
b64c89f05a9fc8ac2878551eb5eda36574ebef6a. An offline branch lookup lacked
|
||||||
|
the cached name, so the human explicitly authorized network resolution
|
||||||
|
and a scoped installed-runtime update preserving unrelated packages.
|
||||||
|
|
||||||
|
The resolver advanced Polars and polars-runtime-32 from 1.36.0 to 2.0.0.
|
||||||
|
An attempt to retain both exact 1.36.0 versions was rejected for the
|
||||||
|
resolver's Windows target. The valid lock with the visible Polars bump
|
||||||
|
is retained for review; no platform restriction or source constraint was
|
||||||
|
introduced to bypass the failure.
|
||||||
|
|
||||||
|
Installed only Tractor and its required dependencies with uv pip rather
|
||||||
|
than syncing/pruning the whole environment. The installer reported one
|
||||||
|
package replacement: Tractor a966928d to b64c89f0. Qt distribution versions
|
||||||
|
remained PyQt6 6.10.1, PyQt6-Qt6 6.10.1 and PyQt6-sip 13.10.3.
|
||||||
|
|
||||||
|
Removed PYTHONPATH from verification subprocesses. Piker resolved inside
|
||||||
|
land_flake_update and Tractor resolved from that checkout's py313
|
||||||
|
site-packages. Installed VCS metadata matched the exact locked revision.
|
||||||
|
The temporary LD_LIBRARY_PATH used the binding's bundled Qt libraries
|
||||||
|
and matching native dependencies; no global environment setup changed.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
- Logger/non-actor startup: 12 passed, 1 deselected in 0.05s.
|
||||||
|
- test_runtime_boot: 1 passed, 1 warning in 0.04s.
|
||||||
|
- test_daemon_bind_distinct_from_registrar: 1 passed in 0.77s.
|
||||||
|
- Existing warning: actor.uid deprecation; no failure.
|
||||||
|
- git diff --check passed.
|
||||||
|
|
||||||
|
This verifies the installed Tractor pin, not a full sync to all new lock
|
||||||
|
versions. Polars was not upgraded in the environment by the scoped install.
|
||||||
|
No live broker feed, order, branch rewrite, staging or commit was run.
|
||||||
|
|
@ -0,0 +1,67 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-22T21:22:48Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6-astra
|
||||||
|
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human approved trying the bounded candidate: back out the IB
|
||||||
|
quote-context cache, import adapter-only reset/boundary fixes and
|
||||||
|
the missing EMS subscription fix, preserve source authorship and
|
||||||
|
commit boundaries, and exercise the real Tractor bridge. They
|
||||||
|
required an explicit keep/backout audit and clarification that
|
||||||
|
the evaluated downstream lineage lacked the broadcast fix.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Verified identical stable patch IDs for original 4af885d9 and
|
||||||
|
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
|
||||||
|
in the main checkout. Applied source-equivalent adapter backports
|
||||||
|
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
|
||||||
|
source attribution recorded in `ai/ib-recovery-candidate.md`.
|
||||||
|
|
||||||
|
The real-runtime quote test first reproduced the reported fatal
|
||||||
|
TrioTaskExited propagation with caching enabled. The direct-owner
|
||||||
|
backout restores restart and subsequent RPC survival, but a strict
|
||||||
|
callback-cleanliness test remains red. Real bridge history pacing
|
||||||
|
recovery and real broadcaster dark clearing pass. No live broker,
|
||||||
|
VNC, orders or TSDB operations were performed by the agent.
|
||||||
|
|
||||||
|
On 2026-10-23 the human supplied successful live startup/backfill,
|
||||||
|
sequential reattachment and overlapping chart-close/reopen results,
|
||||||
|
plus the still-failing daemon SIGINT teardown. The human accepted
|
||||||
|
deferring teardown from #94, requested loud partial-backport
|
||||||
|
attribution and required downstream rebase rehearsals before landing.
|
||||||
|
The agent removed only the explicit single-null debugger stop,
|
||||||
|
retaining its legacy result. Downstream `976db6c3` instead rewrites
|
||||||
|
null grouping; the local cleanup does not claim that algorithmic fix.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
|
||||||
|
- `piker/clearing/_ems.py`: original EMS subscription patch.
|
||||||
|
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
|
||||||
|
- `tests/test_ib_history.py`: original adapter regressions only.
|
||||||
|
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
|
||||||
|
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
|
||||||
|
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
|
||||||
|
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human required original fixes and authorship, rejected broad
|
||||||
|
speculative integration work, required real concurrency boundaries
|
||||||
|
in tests, and corrected the interpretation of live qualification.
|
||||||
|
They performed the live tests, chose to defer known teardown defects
|
||||||
|
from #94, permitted the minimal debugger cleanup, and requested
|
||||||
|
explicit downstream discrepancies and rebase compatibility checks.
|
||||||
|
The agent applied those directed edits; no direct manual source edits
|
||||||
|
are known in this candidate. The raw partner retains the originally
|
||||||
|
captured output; this structured record includes later contributions.
|
||||||
|
|
@ -0,0 +1,69 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-22T21:22:48Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6-astra
|
||||||
|
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||||
|
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
|
||||||
|
---
|
||||||
|
|
||||||
|
## Human direction
|
||||||
|
|
||||||
|
> yeah i guess this is our best approach strat for sure.
|
||||||
|
|
||||||
|
> so you're saying these 2 are not the same patch?
|
||||||
|
> please tell me no lol
|
||||||
|
|
||||||
|
> so detail which commits to drop vs. keep please in detail so i can
|
||||||
|
> audit your choices this time..
|
||||||
|
|
||||||
|
> let's try it and consider bailing and re-trying if this leads us into
|
||||||
|
> a further (and deeper) sinkhole, bc as it stands right now this whole
|
||||||
|
> endevour has turned into a complete shitfest XD
|
||||||
|
|
||||||
|
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
|
||||||
|
|
||||||
|
## Unedited diagnostic response
|
||||||
|
|
||||||
|
The real-bridge test reproduced the exact fatal path:
|
||||||
|
`maybe_open_context()` teardown raises `TrioTaskExited` back into
|
||||||
|
the quote consumer.
|
||||||
|
|
||||||
|
With only the quote-cache backout, both quote generations complete
|
||||||
|
and their asyncio tasks finish. Tractor still emits
|
||||||
|
`TrioTaskExited` callback errors, so the strict test remains red.
|
||||||
|
That narrows the result: the backout removes the fatal propagation
|
||||||
|
in this reproducer, but doesn’t make bridge teardown clean.
|
||||||
|
|
||||||
|
## Code output pointers
|
||||||
|
|
||||||
|
> `git diff dc618229 -- piker/brokers/ib/feed.py`
|
||||||
|
|
||||||
|
Back out 635314c7's quote owner and import the original adapter-only
|
||||||
|
deltas from 0846cbd4 and 1da9b760.
|
||||||
|
|
||||||
|
> `git diff dc618229 -- piker/clearing/_ems.py`
|
||||||
|
|
||||||
|
Import 12b0d66e's dedicated dark-clearing quote subscription.
|
||||||
|
|
||||||
|
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
|
||||||
|
|
||||||
|
Replace the reverted mock-cache test with a real-runtime restart
|
||||||
|
reproducer; retain original history/EMS tests and add real bridge
|
||||||
|
history recovery and real broadcaster EMS coverage. Newly added
|
||||||
|
test files remain untracked until the human stages them, so the
|
||||||
|
diff pointers include them only after staging or committing.
|
||||||
|
|
||||||
|
## Verification output
|
||||||
|
|
||||||
|
```text
|
||||||
|
21 passed in 0.53s
|
||||||
|
1 failed, 2 passed in 5.53s
|
||||||
|
13 passed in 1.41s
|
||||||
|
1 passed, 1 warning in 0.04s
|
||||||
|
2 passed in 0.03s
|
||||||
|
```
|
||||||
|
|
||||||
|
The last run includes the original EMS case again and one additional
|
||||||
|
real-broadcaster case: 38 unique passes and one strict callback failure.
|
||||||
|
|
@ -0,0 +1,77 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T16:08:46Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 7242dcb9
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261028T160846Z_7242dcb9_websocket_close.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human reported repeated live Binance perp crashes after correct
|
||||||
|
worktree startup: the real trio-websocket reader attempted a Pong
|
||||||
|
while wsproto was LOCAL_CLOSING. They asked whether an existing
|
||||||
|
downstream websocket fix should be reused, then requested:
|
||||||
|
|
||||||
|
> ok add the fix now, follow by a /commit-plan and then i'll test
|
||||||
|
> before saving it ya?
|
||||||
|
|
||||||
|
They approved a real websocket reader/close regression covering
|
||||||
|
recovery and intentional shutdown, and separately authorized
|
||||||
|
reconciling this worktree's legacy workflow bookkeeping while
|
||||||
|
preserving existing archives.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
The older `3d83b61f3f7b982e84a4230dba5777f1407c78f1` wrapper
|
||||||
|
suppressed escaped communication failures. The landing replay
|
||||||
|
`97eb8347` of `d50ab6124cd0e1646c82fa4a85d7c13bfff82611`
|
||||||
|
added propagation; it did not restart a reader-context failure.
|
||||||
|
Existing `LocalProtocolError` membership in NoBsWs.recon_errors
|
||||||
|
only protected the receive relay, not reader-nursery context exit.
|
||||||
|
|
||||||
|
Classify these errors around the complete websocket context and
|
||||||
|
retry instead of restoring outer suppression. Place collapse_eg
|
||||||
|
outside that context, invalidate stale connection readiness and
|
||||||
|
socket references, and leave the retry wait cancellable. Preserve
|
||||||
|
unexpected failures and mixed exception groups for propagation.
|
||||||
|
After review, use RuntimeError rather than ValueError for the
|
||||||
|
test-only unexpected-failure sentinel. Its injected exception has
|
||||||
|
no underlying cause. Production recovery logs retain the original
|
||||||
|
traceback and now name the URL; unexpected exceptions use bare
|
||||||
|
raise to preserve identity, existing causes and notes.
|
||||||
|
|
||||||
|
The deterministic seam controls protocol-state ordering in the real
|
||||||
|
client Ping handler. It does not replace the websocket transport,
|
||||||
|
reader, context manager, reconnect task or wsproto exception.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/_web_bs.py`: reader-context failure recovery.
|
||||||
|
- `tests/test_websocket.py`: loopback recovery and two shutdown
|
||||||
|
variants; retain unexpected-failure propagation control.
|
||||||
|
- This provenance pair: live report, lineage and test scope.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Baseline uses new tests against original HEAD websocket code in an
|
||||||
|
isolated exact-tree checkout. The recovery case fails with the
|
||||||
|
reported Pong-in-LOCAL_CLOSING LocalProtocolError.
|
||||||
|
|
||||||
|
Fixed code passes 13 cases across tests/test_websocket.py and
|
||||||
|
tests/test_binance.py. Exact-boundary verification and native-library
|
||||||
|
configuration are retained in the ignored commit-plan package.
|
||||||
|
Live testing remains human-owned and pending.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
No direct manual source edits observed in this follow-up. The human
|
||||||
|
supplied live failures, corrected the earlier daemon/worktree setup,
|
||||||
|
set the narrow landing scope, requested source-fix reuse, and will
|
||||||
|
test before reviewing and committing. Existing commits are retained.
|
||||||
|
The human requested the RuntimeError sentinel and preservation of
|
||||||
|
underlying error context. The raw partner retains the initial output.
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T16:08:46Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 7242dcb9
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
Implemented a focused websocket-context reconnect fix and real
|
||||||
|
loopback reader/close regressions in the Binance landing worktree.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Move exception-group collapse outside the websocket context so
|
||||||
|
reader-nursery exit errors are classified by the reconnect loop.
|
||||||
|
Retry NoBsWs.recon_errors there, clear the stale connected event
|
||||||
|
and socket reference, and honor cancellation through unshielded
|
||||||
|
retry sleep. Retain unexpected-error propagation at the outer API.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||||
|
|
||||||
|
Exercise real TCP, handshake, Ping frames, reader dispatch and
|
||||||
|
wsproto Pong validation. A narrow handler gate advances the real
|
||||||
|
protocol state to LOCAL_CLOSING without sending the Close frame,
|
||||||
|
preventing a peer reply from racing the controlled state. Require
|
||||||
|
fresh application data from the replacement connection. Separate
|
||||||
|
variants cancel at the reader gate and at the actual retry wait.
|
||||||
|
Update the existing unexpected-failure control to use ValueError
|
||||||
|
because LocalProtocolError is now intentionally recoverable.
|
||||||
|
|
||||||
|
The isolated original-code/new-test baseline fails with the exact
|
||||||
|
Pong-in-LOCAL_CLOSING error. The fixed isolated tree passes all 13
|
||||||
|
selected Binance/websocket cases. No live provider, broker orders
|
||||||
|
or production daemon operations were performed by the agent.
|
||||||
|
|
@ -0,0 +1,70 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T16:56:38Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: b95e7c79
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261028T165638Z_b95e7c79_reconnect_logging.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human reported that the live Binance Pong-in-LOCAL_CLOSING
|
||||||
|
error now logged a retry without crashing. They requested visible
|
||||||
|
re-establishment reporting under `pikerd -l info --pdb`, and asked
|
||||||
|
to check for existing downstream work before adding a fix.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
The existing `Connection success` INFO message already identifies
|
||||||
|
handshake completion. Feed startup configured piker.data.feed,
|
||||||
|
while shared websocket diagnostics use ._util.log on piker.data.
|
||||||
|
Configure that actual emitter and prefer the daemon's verbosity
|
||||||
|
over a quieter client request for these shared transport tasks.
|
||||||
|
|
||||||
|
Add a readiness INFO message after fixture setup and a first-message
|
||||||
|
INFO report once per connection after actual relay. Include the URL
|
||||||
|
in the receive-timeout reset error. Do not call first raw websocket
|
||||||
|
traffic a parsed quote or proof of restored market-data freshness.
|
||||||
|
|
||||||
|
## Existing-work audit
|
||||||
|
|
||||||
|
Compared `_web_bs.py`, `_util.py`, `piker.log` and daemon logging
|
||||||
|
against the evaluated downstream tip and relevant all-ref histories.
|
||||||
|
`54da2973` contains logging/type cleanup, not this emitter fix.
|
||||||
|
`5adf77a9` / landing equivalents address package-name correctness,
|
||||||
|
but not configuration of the shared data emitter at feed startup.
|
||||||
|
All-ref searches for this actual emitter/readiness/first-relay
|
||||||
|
implementation found no existing commit. The dirty IB history
|
||||||
|
worktree has only an unrelated f-string correction in `_web_bs.py`;
|
||||||
|
the TIPC worktree has no dirty delta in these two feed/websocket
|
||||||
|
files. No downstream work was overwritten or imported wholesale.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/feed.py`: shared transport logger startup/verbosity.
|
||||||
|
- `piker/data/_web_bs.py`: readiness, first-relay and timeout URL.
|
||||||
|
- `tests/test_logging.py`: actual emitter/console startup regression.
|
||||||
|
- `tests/test_websocket.py`: recovery milestones on real loopback.
|
||||||
|
- This prompt provenance pair.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
The new logging regression against original b95e7c79 source fails:
|
||||||
|
the actual websocket emitter remains at WARNING and rejects INFO.
|
||||||
|
It passes with the logger wiring fix. Provider lookup is stopped
|
||||||
|
before network work, while logger factories and console emission
|
||||||
|
are real. Global logger state is restored after the check.
|
||||||
|
|
||||||
|
18 cases pass across tests/test_logging.py, tests/test_websocket.py
|
||||||
|
and tests/test_binance.py. git diff --check passes. Live verification
|
||||||
|
after restarting the daemon with these edits remains human-owned.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
No direct manual source edits observed. The human supplied live
|
||||||
|
recovery evidence, selected INFO verbosity, requested downstream
|
||||||
|
reuse inspection, and retains live-test and commit ownership.
|
||||||
|
|
@ -0,0 +1,38 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T16:56:38Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: b95e7c79
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
Confirmed the existing INFO handshake-success log was hidden by
|
||||||
|
logger wiring: websocket code emits on piker.data, while feed
|
||||||
|
startup configured piker.data.feed. The baseline regression leaves
|
||||||
|
the emitter at WARNING for an INFO daemon and ERROR client.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||||
|
|
||||||
|
Configure the actual shared ._util.log namespace, with daemon
|
||||||
|
verbosity taking priority over client verbosity for actor-wide
|
||||||
|
transport diagnostics.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Keep the existing handshake-success message. Add INFO readiness
|
||||||
|
after fixture setup and first-message relay once per connection.
|
||||||
|
Include the URL in the receive-timeout reset diagnostic.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_logging.py tests/test_websocket.py`
|
||||||
|
|
||||||
|
Prove production startup enables real console INFO on the websocket
|
||||||
|
emitter without contacting a provider. Extend the real loopback
|
||||||
|
recovery test to require readiness for both connections and one
|
||||||
|
first-message report for the connection delivering fresh data.
|
||||||
|
|
||||||
|
All 18 targeted logging/Binance/websocket cases passed. No live
|
||||||
|
provider/daemon operations were performed by the agent. First relay
|
||||||
|
is raw websocket traffic; parsed market-data recovery remains a
|
||||||
|
separate live observation.
|
||||||
|
|
@ -0,0 +1,54 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T18:08:56Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 032580bf
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261028T180856Z_032580bf_timeout_phase.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human reported repeated readiness without first-message relay,
|
||||||
|
specifically after first attempting an invalid .usdm. FQME and then
|
||||||
|
the corrected .usdtm. one. They requested distinct log messages for
|
||||||
|
receive starvation versus local backpressure.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Add phase-specific WARNING messages for each elapsed relay-loop
|
||||||
|
deadline, with URL, timeout duration, accumulated count and reset
|
||||||
|
threshold. Leave the existing shared receive/send deadline and
|
||||||
|
counter/reset semantics intact. Do not claim that connected/ready
|
||||||
|
sockets alone establish parsed quote recovery.
|
||||||
|
|
||||||
|
Read-only invalid-venue tracing found the usd substring check in
|
||||||
|
Binance get_mkt_info() accepts USDM while retaining it in MktPair.
|
||||||
|
Client.bars() later uses that market identity against canonical
|
||||||
|
USDTM keys. Backend stream startup precedes history startup and
|
||||||
|
sampler consumption. A relationship to the corrected-feed stall
|
||||||
|
still needs runtime evidence; neither venue handling nor task
|
||||||
|
lifetime behavior was changed by this diagnostic patch.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/_web_bs.py`: track and report the timed-out operation.
|
||||||
|
- `tests/test_websocket.py`: real silent-source and blocked-sink cases.
|
||||||
|
- This prompt provenance pair.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
15 selected Binance/websocket cases passed. The new cases exercise
|
||||||
|
real TCP, websocket handshake and relay behavior, replacing only
|
||||||
|
log observation. A zero-capacity relay deliberately forces the
|
||||||
|
local-send case; the source-silent case never receives a payload.
|
||||||
|
git diff --check passed. No live provider/daemon operations run.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
No direct source edits observed. The human supplied the critical
|
||||||
|
invalid-symbol-then-corrected-symbol trigger and requested the
|
||||||
|
phase distinction; live testing and task acceptance remain theirs.
|
||||||
|
|
@ -0,0 +1,34 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-28T18:08:56Z
|
||||||
|
generated_by:
|
||||||
|
harness: opencode
|
||||||
|
provider: openai
|
||||||
|
model: openai/gpt-6.1-sol
|
||||||
|
git_ref: 032580bf
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
Added distinct timeout diagnostics for websocket receive starvation
|
||||||
|
versus local relay backpressure, preserving the existing timeout,
|
||||||
|
counter and reconnect threshold policy.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||||
|
|
||||||
|
Track whether proxy_msgs() is awaiting ws.get_message() or
|
||||||
|
snd.send(). Report that phase, URL, configured deadline, timeout
|
||||||
|
count and reset threshold for every timeout, before the existing
|
||||||
|
threshold/reset diagnostic. Capture the configured deadline once
|
||||||
|
per operation so the report matches the deadline that elapsed.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||||
|
|
||||||
|
Two real loopback variants use a silent server versus one payload
|
||||||
|
and an unconsumed zero-capacity Trio relay. Require the appropriate
|
||||||
|
phase and URL on the actual first timeout. All 15 selected
|
||||||
|
Binance/websocket cases pass. No live provider was contacted.
|
||||||
|
|
||||||
|
Read-only tracing found a permissive invalid-venue path: get_mkt_info
|
||||||
|
accepts USDM via its usd substring check and preserves that venue
|
||||||
|
in MktPair, whereas Client.bars() indexes canonical USDTM pair keys.
|
||||||
|
This identifies a potential late failed-startup path, not proof of
|
||||||
|
the corrected-chart stall or of leaked producer ownership.
|
||||||
|
|
@ -1,8 +1,29 @@
|
||||||
[network]
|
[network]
|
||||||
|
# This Tractor pin supports one transport per actor, not mixed TCP/UDS.
|
||||||
|
# Use the single TCP endpoint shown in each list for this version.
|
||||||
|
# Mixed-transport configs need a future Tractor pin with that support.
|
||||||
pikerd = [
|
pikerd = [
|
||||||
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
# TCP localhost loopback
|
||||||
# '/uds/6116', # TODO std uds socket file
|
'/ip4/127.0.0.1/tcp/6116',
|
||||||
|
|
||||||
|
# UDS alternative: replace TCP and select the UDS transport.
|
||||||
|
# '/unix/run/user/1000/piker/pikerd.sock',
|
||||||
]
|
]
|
||||||
|
chart = [
|
||||||
|
'/ip4/127.0.0.1/tcp/3003',
|
||||||
|
# '/unix/run/user/1000/piker/chart.sock',
|
||||||
|
]
|
||||||
|
# the service-actor registry endpoint;
|
||||||
|
# other pikerd trees contact this to discover
|
||||||
|
# actors.
|
||||||
|
# XXX if absent, pikerd binds the registry
|
||||||
|
# on its own tpt_bind_addrs.
|
||||||
|
# regd = ['/ip4/127.0.0.1/tcp/6116']
|
||||||
|
|
||||||
|
# chart = [
|
||||||
|
# '/ip4/127.0.0.1/tcp/3333',
|
||||||
|
# '/unix/run/user/1000/piker/chart@3333.sock',
|
||||||
|
# ]
|
||||||
|
|
||||||
|
|
||||||
[ui]
|
[ui]
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,285 @@
|
||||||
|
Manual Backfill Qualification
|
||||||
|
=============================
|
||||||
|
|
||||||
|
Purpose
|
||||||
|
-------
|
||||||
|
|
||||||
|
This runbook separates provider coverage, durable NativeDB state,
|
||||||
|
shared-memory hydration, and chart rendering while reproducing gappy
|
||||||
|
history. Work one FQME and timeframe at a time. Preserve the original
|
||||||
|
Parquet first, then qualify both repair of that evidence and a completely
|
||||||
|
fresh backfill.
|
||||||
|
|
||||||
|
The read-only command used throughout is::
|
||||||
|
|
||||||
|
piker store audit FQME --period 60
|
||||||
|
piker store audit FQME --period 60 --json
|
||||||
|
|
||||||
|
It reads raw Parquet directly. It does not open a storage client,
|
||||||
|
canonicalize data, publish cache state, attach SHM, or rewrite history.
|
||||||
|
Positive gaps are deliberately unclassified; an observed interval is not
|
||||||
|
automatically corruption or an expected venue closure.
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
Do not run qualification against normal user storage. The helper below
|
||||||
|
requires a marked disposable root before it will seed, archive, or clear
|
||||||
|
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive
|
||||||
|
and potentially mutating; they are not audit commands.
|
||||||
|
|
||||||
|
Keep the disposable root private and do not rename, replace, or symlink
|
||||||
|
files under it while a helper is running. Stop the chart cleanly before
|
||||||
|
every audit or comparison phase.
|
||||||
|
|
||||||
|
Load The Xonsh Helpers
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Run from the repository root in the same xonsh which will launch the chart
|
||||||
|
or daemon::
|
||||||
|
|
||||||
|
source snippets/nativedb_backfill_audit.xsh
|
||||||
|
|
||||||
|
Choose a disposable root and copy only the provider configuration needed
|
||||||
|
for the case::
|
||||||
|
|
||||||
|
run_root = p'/tmp/piker-backfill/mnq-20260918-60s'
|
||||||
|
source_config = p'~/.config/piker'.expanduser()
|
||||||
|
bfq_init(run_root, source_config)
|
||||||
|
bfq_record_case(
|
||||||
|
run_root,
|
||||||
|
fqme='mnq.cme.20260918.ib',
|
||||||
|
period_s=60,
|
||||||
|
provider='ib',
|
||||||
|
symptom='gappy chart during reverse backfill',
|
||||||
|
bad_start_utc=None,
|
||||||
|
bad_end_utc=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
``bfq_init()`` sets ``XDG_CONFIG_HOME`` in the current xonsh, so every
|
||||||
|
subsequent ``piker`` and ``pikerd`` child inherits the disposable config.
|
||||||
|
The effective storage path is under ``RUN_ROOT/xdg/piker/nativedb``.
|
||||||
|
It also records the exact repository root and ``piker`` executable in the
|
||||||
|
qualification marker, prepends that root to child ``PYTHONPATH``, and
|
||||||
|
verifies that the bound executable exposes ``piker store audit``. Audits
|
||||||
|
fail if that executable imports another tree.
|
||||||
|
|
||||||
|
The copied ``brokers.toml`` can contain credentials. Keep the run root
|
||||||
|
private and remove it manually after evidence is no longer needed.
|
||||||
|
|
||||||
|
Repository Preflight
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
Before every daemon start, verify the code and config inherited by child
|
||||||
|
processes::
|
||||||
|
|
||||||
|
git status --short --branch
|
||||||
|
git rev-parse HEAD
|
||||||
|
python -c 'import piker; print(piker.__file__)'
|
||||||
|
python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])'
|
||||||
|
|
||||||
|
Stop any daemon from the previous phase cleanly. Use a separate registry
|
||||||
|
address in the disposable ``conf.toml`` when a normal Piker stack remains
|
||||||
|
active.
|
||||||
|
|
||||||
|
Known-Bad Replay
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Copy the original file; never move it out of normal storage::
|
||||||
|
|
||||||
|
fqme = 'mnq.cme.20260918.ib'
|
||||||
|
period_s = 60
|
||||||
|
source_parquet = (
|
||||||
|
p'~/.config/piker/nativedb'.expanduser()
|
||||||
|
/f'{fqme}.ohlcv{period_s}s.parquet'
|
||||||
|
)
|
||||||
|
bfq_seed(run_root, fqme, period_s, source_parquet)
|
||||||
|
bfq_audit(run_root, 'baseline', fqme, period_s)
|
||||||
|
|
||||||
|
The baseline phase writes::
|
||||||
|
|
||||||
|
evidence/baseline.json
|
||||||
|
evidence/baseline.parquet
|
||||||
|
evidence/baseline.manual.json
|
||||||
|
|
||||||
|
The JSON checksum must match the read-only snapshot, and the manual sidecar
|
||||||
|
binds that report to its phase and case. Comparison rechecks those bindings
|
||||||
|
before reading either phase. The report records raw schema, row and
|
||||||
|
timestamp counts, non-finite values, ordering, duplicate excess, canonical
|
||||||
|
index state, and every retained positive-gap detail. Files with invalid
|
||||||
|
Parquet encoding can not produce a JSON report, but ``--snapshot`` still
|
||||||
|
preserves their exact bytes before parse failure.
|
||||||
|
|
||||||
|
Launch the chart manually from the same shell and capture logs::
|
||||||
|
|
||||||
|
piker -l info chart mnq.cme.20260918.ib \
|
||||||
|
2>&1 | tee @(run_root / 'logs' / 'repair-chart.log')
|
||||||
|
|
||||||
|
After a clean stop::
|
||||||
|
|
||||||
|
bfq_audit(
|
||||||
|
run_root,
|
||||||
|
'repair',
|
||||||
|
fqme,
|
||||||
|
period_s,
|
||||||
|
require_structural=True,
|
||||||
|
)
|
||||||
|
bfq_compare(run_root, 'baseline', 'repair')
|
||||||
|
|
||||||
|
The comparison fails if structural validity regresses, any original
|
||||||
|
timestamp disappears, chronological endpoints shrink, timestamps become
|
||||||
|
duplicated or unordered, indexes cease to be canonical, or OHLCV becomes
|
||||||
|
non-finite. Provider replacement of matching seam rows is reported
|
||||||
|
separately as ``changed_common_timestamps`` for operator review.
|
||||||
|
|
||||||
|
Fresh Backfill
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Yes, fresh qualification means saving the known-bad evidence and then
|
||||||
|
starting with no active Parquet in the disposable NativeDB. Archive only
|
||||||
|
the disposable file::
|
||||||
|
|
||||||
|
bfq_clear_for_fresh(run_root, fqme, period_s)
|
||||||
|
|
||||||
|
Confirm the printed active path is absent, launch the same chart, wait for
|
||||||
|
bounded provider exhaustion or the intended coverage, then stop cleanly::
|
||||||
|
|
||||||
|
bfq_audit(
|
||||||
|
run_root,
|
||||||
|
'fresh',
|
||||||
|
fqme,
|
||||||
|
period_s,
|
||||||
|
require_structural=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
Do not compare ``baseline`` to ``fresh`` as a timestamp-preservation gate:
|
||||||
|
provider limits may intentionally produce a different initial extent.
|
||||||
|
Treat ``fresh`` as the baseline for restart and append qualification.
|
||||||
|
|
||||||
|
Restart And Append
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Restart the same chart against the fresh persisted file, stop cleanly, and
|
||||||
|
record::
|
||||||
|
|
||||||
|
bfq_audit(
|
||||||
|
run_root,
|
||||||
|
'restart',
|
||||||
|
fqme,
|
||||||
|
period_s,
|
||||||
|
require_structural=True,
|
||||||
|
)
|
||||||
|
bfq_compare(run_root, 'fresh', 'restart')
|
||||||
|
|
||||||
|
Leave a subsequent run active long enough to append new samples, stop, and
|
||||||
|
record::
|
||||||
|
|
||||||
|
bfq_audit(
|
||||||
|
run_root,
|
||||||
|
'append',
|
||||||
|
fqme,
|
||||||
|
period_s,
|
||||||
|
require_structural=True,
|
||||||
|
)
|
||||||
|
bfq_compare(
|
||||||
|
run_root,
|
||||||
|
'restart',
|
||||||
|
'append',
|
||||||
|
require_newer=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
The comparison JSON reports gaps added and removed. Expected closures stay
|
||||||
|
visible until a venue-aware classifier proves their session alignment.
|
||||||
|
|
||||||
|
Gap Layer Triage
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Use the persisted report and chart together:
|
||||||
|
|
||||||
|
=============================== =========================================
|
||||||
|
Observation Suspected layer
|
||||||
|
=============================== =========================================
|
||||||
|
Parquet contiguous, chart gappy SHM hydration, sampling, or chart render
|
||||||
|
Parquet gap, provider has bars provider-delta or storage merge
|
||||||
|
Provider and Parquet both gappy provider omission or venue closure
|
||||||
|
Gap vanishes after restart SHM publication or cache lifecycle
|
||||||
|
Old timestamps disappear destructive persistence regression
|
||||||
|
Newest seam duplicates conflict or dedupe policy
|
||||||
|
=============================== =========================================
|
||||||
|
|
||||||
|
Always record exact UTC gap endpoints. The audit's numeric epoch is
|
||||||
|
authoritative; the ``*_utc`` fields are deterministic display values.
|
||||||
|
|
||||||
|
Expansion Matrix
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Expand only after the fixed FQME passes fresh, restart, and append. Use a
|
||||||
|
new disposable root and case record for every row:
|
||||||
|
|
||||||
|
==================== ========== ========================================
|
||||||
|
Case Period Coverage model
|
||||||
|
==================== ========== ========================================
|
||||||
|
Original IB future 60 seconds CME maintenance and weekend closures
|
||||||
|
Original IB future 1 second high-frequency provider limits
|
||||||
|
Second CME future 60 seconds contract-specific history
|
||||||
|
IB equity 60 seconds regular overnight and weekend closure
|
||||||
|
IB FX 60 seconds 24x5 session behavior
|
||||||
|
Crypto spot/perp 60 seconds continuous-session control
|
||||||
|
Expired future 60 seconds history-only and expiry behavior
|
||||||
|
==================== ========== ========================================
|
||||||
|
|
||||||
|
A sourceable matrix can remain ordinary Python data in xonsh::
|
||||||
|
|
||||||
|
cases = [
|
||||||
|
{
|
||||||
|
'name': 'mnq-60s',
|
||||||
|
'fqme': 'mnq.cme.20260918.ib',
|
||||||
|
'period_s': 60,
|
||||||
|
'provider': 'ib',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'name': 'mnq-1s',
|
||||||
|
'fqme': 'mnq.cme.20260918.ib',
|
||||||
|
'period_s': 1,
|
||||||
|
'provider': 'ib',
|
||||||
|
},
|
||||||
|
]
|
||||||
|
for case in cases:
|
||||||
|
case_root = p'/tmp/piker-backfill' / case['name']
|
||||||
|
bfq_init(case_root, source_config)
|
||||||
|
bfq_record_case(
|
||||||
|
case_root,
|
||||||
|
fqme=case['fqme'],
|
||||||
|
period_s=case['period_s'],
|
||||||
|
provider=case['provider'],
|
||||||
|
symptom='manual expansion case',
|
||||||
|
)
|
||||||
|
|
||||||
|
This loop prepares isolated case roots only. Run the complete seed or
|
||||||
|
fresh, chart, audit, restart, append, and compare lifecycle in each root;
|
||||||
|
directory creation alone is not qualification.
|
||||||
|
|
||||||
|
The chart path may allocate companion 1-second and 60-second files. Audit
|
||||||
|
the selected case path explicitly and retain companion files as secondary
|
||||||
|
evidence; do not imply that the live request isolated one timeframe.
|
||||||
|
|
||||||
|
Interruption Qualification
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
After normal restart is stable, interrupt one phase with graceful Ctrl-C,
|
||||||
|
restart, and compare against the last clean evidence. Provider disconnect,
|
||||||
|
SIGTERM, and deterministic write-barrier failures come next. Avoid timing a
|
||||||
|
SIGKILL by hand; add an explicit failpoint before qualifying atomic replace
|
||||||
|
boundaries.
|
||||||
|
|
||||||
|
Result Interpretation
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
``structural_ok`` means the raw persisted file satisfies durable schema,
|
||||||
|
finite values, positive unique ordered timestamps, canonical index, and
|
||||||
|
period-aligned gaps. ``gap_free`` means no positive gaps were observed when
|
||||||
|
at least two valid timestamps made cadence verifiable; otherwise it is
|
||||||
|
``null``. ``qualification_ok`` combines both and is intentionally strict.
|
||||||
|
|
||||||
|
A structurally valid session-market file can have ``gap_free = false``.
|
||||||
|
Those intervals require provider/session evidence before being marked as
|
||||||
|
expected closures. Unknown calendar state must stay unknown.
|
||||||
|
|
@ -2,11 +2,11 @@
|
||||||
"nodes": {
|
"nodes": {
|
||||||
"nixpkgs": {
|
"nixpkgs": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1765779637,
|
"lastModified": 1784497964,
|
||||||
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
|
||||||
"owner": "nixos",
|
"owner": "nixos",
|
||||||
"repo": "nixpkgs",
|
"repo": "nixpkgs",
|
||||||
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
|
||||||
"type": "github"
|
"type": "github"
|
||||||
},
|
},
|
||||||
"original": {
|
"original": {
|
||||||
|
|
|
||||||
23
flake.nix
23
flake.nix
|
|
@ -22,12 +22,14 @@
|
||||||
|
|
||||||
# do store-path extractions
|
# do store-path extractions
|
||||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||||
|
zlibStorePath = lib.getLib pkgs.zlib;
|
||||||
# ?TODO? can remove below since manual linking not needed?
|
# ?TODO? can remove below since manual linking not needed?
|
||||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||||
|
|
||||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||||
# a major-version-pinned-`cpython`
|
# a major-version-pinned-`cpython`
|
||||||
cpython = "python313";
|
cpython = "python313";
|
||||||
|
python = pkgs.${cpython};
|
||||||
pypkgs = pkgs."${cpython}Packages";
|
pypkgs = pkgs."${cpython}Packages";
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
|
@ -46,7 +48,7 @@
|
||||||
qt6.qtbase
|
qt6.qtbase
|
||||||
|
|
||||||
uv
|
uv
|
||||||
python313 # ?TODO^ how to set from `cpython` above?
|
python
|
||||||
pypkgs.pyqt6
|
pypkgs.pyqt6
|
||||||
pypkgs.pyqt6-sip
|
pypkgs.pyqt6-sip
|
||||||
pypkgs.qtpy
|
pypkgs.qtpy
|
||||||
|
|
@ -70,6 +72,7 @@
|
||||||
|
|
||||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||||
|
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
|
||||||
|
|
||||||
export LD_LIBRARY_PATH
|
export LD_LIBRARY_PATH
|
||||||
|
|
||||||
|
|
@ -86,15 +89,31 @@
|
||||||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||||
|
|
||||||
# ------ uv ------
|
# ------ uv ------
|
||||||
|
# ignore any venv inherited from an outer checkout
|
||||||
|
unset VIRTUAL_ENV
|
||||||
# - always use the ./py313/ venv-subdir
|
# - always use the ./py313/ venv-subdir
|
||||||
export UV_PROJECT_ENVIRONMENT="py313"
|
export UV_PROJECT_ENVIRONMENT="py313"
|
||||||
|
|
||||||
|
# Rebuild after a flake update changes the Nix Python closure.
|
||||||
|
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
|
||||||
|
nix_python="$(readlink -f "${python}/bin/python3")"
|
||||||
|
if test -e "$venv_python" \
|
||||||
|
&& test "$(readlink -f "$venv_python")" != "$nix_python"
|
||||||
|
then
|
||||||
|
uv venv \
|
||||||
|
--clear \
|
||||||
|
--python "$nix_python" \
|
||||||
|
"$UV_PROJECT_ENVIRONMENT"
|
||||||
|
fi
|
||||||
|
|
||||||
# sync project-env with all extras
|
# sync project-env with all extras
|
||||||
uv sync --dev --all-extras --no-group lint
|
uv sync --dev --all-extras --no-group lint
|
||||||
|
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
|
||||||
|
|
||||||
# ------ TIPS ------
|
# ------ TIPS ------
|
||||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||||
# run the `nix develop` cmd with,
|
# run the `nix develop` cmd with,
|
||||||
# >> nix develop -c uv run xonsh
|
# >> nix develop -c xonsh
|
||||||
'';
|
'';
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -324,10 +324,13 @@ def load_ledger(
|
||||||
ldir: Path = (
|
ldir: Path = (
|
||||||
dirpath
|
dirpath
|
||||||
or
|
or
|
||||||
config._config_dir / 'accounting' / 'ledgers'
|
config.get_conf_dir() / 'accounting' / 'ledgers'
|
||||||
)
|
)
|
||||||
if not ldir.is_dir():
|
if not ldir.is_dir():
|
||||||
ldir.mkdir()
|
ldir.mkdir(
|
||||||
|
parents=True,
|
||||||
|
exist_ok=True,
|
||||||
|
)
|
||||||
|
|
||||||
fname = f'trades_{brokername}_{acctid}.toml'
|
fname = f'trades_{brokername}_{acctid}.toml'
|
||||||
fpath: Path = ldir / fname
|
fpath: Path = ldir / fname
|
||||||
|
|
|
||||||
|
|
@ -785,9 +785,16 @@ def load_account(
|
||||||
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
||||||
fn: str = f'account.{brokername}.{acctid}.toml'
|
fn: str = f'account.{brokername}.{acctid}.toml'
|
||||||
|
|
||||||
dirpath: Path = dirpath or (config._config_dir / 'accounting')
|
dirpath: Path = (
|
||||||
|
dirpath
|
||||||
|
or
|
||||||
|
(config.get_conf_dir() / 'accounting')
|
||||||
|
)
|
||||||
if not dirpath.is_dir():
|
if not dirpath.is_dir():
|
||||||
dirpath.mkdir()
|
dirpath.mkdir(
|
||||||
|
parents=True,
|
||||||
|
exist_ok=True,
|
||||||
|
)
|
||||||
|
|
||||||
conf, path = config.load(
|
conf, path = config.load(
|
||||||
path=dirpath / fn,
|
path=dirpath / fn,
|
||||||
|
|
|
||||||
|
|
@ -143,12 +143,15 @@ def sync(
|
||||||
# (what the EMS normally does internall) B)
|
# (what the EMS normally does internall) B)
|
||||||
open_brokerd_dialog(
|
open_brokerd_dialog(
|
||||||
brokermod,
|
brokermod,
|
||||||
portal,
|
|
||||||
exec_mode=(
|
exec_mode=(
|
||||||
'paper'
|
'paper'
|
||||||
if account == 'paper'
|
if account == 'paper'
|
||||||
else 'live'
|
else 'live'
|
||||||
),
|
),
|
||||||
|
# use our own ad-hoc-spawned actor,
|
||||||
|
# do NOT (spawn and) use the
|
||||||
|
# `brokerd.<broker>` service daemon!
|
||||||
|
portal=portal,
|
||||||
loglevel=loglevel,
|
loglevel=loglevel,
|
||||||
) as (
|
) as (
|
||||||
brokerd_stream,
|
brokerd_stream,
|
||||||
|
|
|
||||||
|
|
@ -25,10 +25,8 @@ from contextlib import (
|
||||||
)
|
)
|
||||||
from types import ModuleType
|
from types import ModuleType
|
||||||
from typing import (
|
from typing import (
|
||||||
TYPE_CHECKING,
|
|
||||||
AsyncContextManager,
|
AsyncContextManager,
|
||||||
)
|
)
|
||||||
import exceptiongroup as eg
|
|
||||||
|
|
||||||
import tractor
|
import tractor
|
||||||
import trio
|
import trio
|
||||||
|
|
@ -40,27 +38,20 @@ from piker.log import (
|
||||||
from . import _util
|
from . import _util
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
|
||||||
from ..data import _FeedsBus
|
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
log = get_logger(name=__name__)
|
||||||
|
|
||||||
# `brokerd` enabled modules
|
# `brokerd`-actor-always-enabled mods.
|
||||||
# TODO: move this def to the `.data` subpkg..
|
|
||||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||||
# model and should be treated with utmost care!
|
# model and should be treated with utmost care! In particular NO
|
||||||
_data_mods: str = [
|
# `piker.data.*` feed mods should be enabled in this (live,
|
||||||
'piker.brokers.core',
|
# credentialed) trading actor; all data-feed serving is the
|
||||||
'piker.brokers.data',
|
# domain of the `datad.<broker>` sibling daemon, see
|
||||||
|
# `piker.data._daemon._datad_service_mods`.
|
||||||
|
_brokerd_service_mods: list[str] = [
|
||||||
'piker.brokers._daemon',
|
'piker.brokers._daemon',
|
||||||
'piker.data',
|
|
||||||
'piker.data.feed',
|
|
||||||
'piker.data._sampling'
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# TODO: we should rename the daemon to datad prolly once we split up
|
|
||||||
# broker vs. data tasks into separate actors?
|
|
||||||
@tractor.context
|
@tractor.context
|
||||||
async def _setup_persistent_brokerd(
|
async def _setup_persistent_brokerd(
|
||||||
ctx: tractor.Context,
|
ctx: tractor.Context,
|
||||||
|
|
@ -69,9 +60,15 @@ async def _setup_persistent_brokerd(
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Allocate a actor-wide service nursery in ``brokerd``
|
Trading-only daemon (lifetime) fixture: console logging
|
||||||
such that feeds can be run in the background persistently by
|
setup and a pinned-open context for service mgmt.
|
||||||
the broker backend as needed.
|
|
||||||
|
All data-feed-bus state now lives in the (data-feed-only)
|
||||||
|
`datad.<brokername>` sibling daemon, see
|
||||||
|
`piker.data._daemon._setup_persistent_datad()`; this
|
||||||
|
actor hosts only the backend's `open_trade_dialog()`
|
||||||
|
(live order-control) ep-task(s) which manage their own
|
||||||
|
task trees per `tractor.Context`.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
# NOTE: we only need to setup logging once (and only) here
|
# NOTE: we only need to setup logging once (and only) here
|
||||||
|
|
@ -80,54 +77,19 @@ async def _setup_persistent_brokerd(
|
||||||
# any further (level) configuration on their own B)
|
# any further (level) configuration on their own B)
|
||||||
actor: tractor.Actor = tractor.current_actor()
|
actor: tractor.Actor = tractor.current_actor()
|
||||||
tll: str = actor.loglevel
|
tll: str = actor.loglevel
|
||||||
log = get_console_log(
|
get_console_log(
|
||||||
level=loglevel or tll,
|
level=loglevel or tll,
|
||||||
name=f'{_util.subsys}.{brokername}',
|
name=f'{_util.subsys}.{brokername}',
|
||||||
with_tractor_log=bool(tll),
|
with_tractor_log=bool(tll),
|
||||||
)
|
)
|
||||||
assert log.name == _util.subsys
|
|
||||||
|
|
||||||
# further, set the log level on any broker broker specific
|
|
||||||
# logger instance.
|
|
||||||
|
|
||||||
from piker.data import feed
|
|
||||||
assert not feed._bus
|
|
||||||
|
|
||||||
# allocate a nursery to the bus for spawning background
|
|
||||||
# tasks to service client IPC requests, normally
|
|
||||||
# `tractor.Context` connections to explicitly required
|
|
||||||
# `brokerd` endpoints such as:
|
|
||||||
# - `stream_quotes()`,
|
|
||||||
# - `manage_history()`,
|
|
||||||
# - `allocate_persistent_feed()`,
|
|
||||||
# - `open_symbol_search()`
|
|
||||||
# NOTE: see ep invocation details inside `.data.feed`.
|
|
||||||
try:
|
|
||||||
async with (
|
|
||||||
# tractor.trionics.collapse_eg(),
|
|
||||||
trio.open_nursery() as service_nursery
|
|
||||||
):
|
|
||||||
bus: _FeedsBus = feed.get_feed_bus(
|
|
||||||
brokername,
|
|
||||||
service_nursery,
|
|
||||||
)
|
|
||||||
assert bus is feed._bus
|
|
||||||
|
|
||||||
# unblock caller
|
# unblock caller
|
||||||
await ctx.started()
|
await ctx.started()
|
||||||
|
|
||||||
# we pin this task to keep the feeds manager active until the
|
# we pin this task to keep the daemon active until the
|
||||||
# parent actor decides to tear it down
|
# parent actor decides to tear it down
|
||||||
await trio.sleep_forever()
|
await trio.sleep_forever()
|
||||||
|
|
||||||
except eg.ExceptionGroup:
|
|
||||||
# TODO: likely some underlying `brokerd` IPC connection
|
|
||||||
# broke so here we handle a respawn and re-connect attempt!
|
|
||||||
# This likely should pair with development of the OCO task
|
|
||||||
# nusery in dev over @ `tractor` B)
|
|
||||||
# https://github.com/goodboy/tractor/pull/363
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def broker_init(
|
def broker_init(
|
||||||
brokername: str,
|
brokername: str,
|
||||||
|
|
@ -147,8 +109,10 @@ def broker_init(
|
||||||
|
|
||||||
This includes:
|
This includes:
|
||||||
- load the appropriate <brokername>.py pkg module,
|
- load the appropriate <brokername>.py pkg module,
|
||||||
- reads any declared `__enable_modules__: listr[str]` which will be
|
- reads any declared `_brokerd_mods: list[str]` (falling
|
||||||
passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
|
back to the full `__enable_modules__` set for
|
||||||
|
not-yet-split backends) which will be passed to
|
||||||
|
`tractor.ActorNursery.start_actor(enable_modules=)`
|
||||||
at actor start time,
|
at actor start time,
|
||||||
- deliver a references to the daemon lifetime fixture, which
|
- deliver a references to the daemon lifetime fixture, which
|
||||||
for now is always the `_setup_persistent_brokerd()` context defined
|
for now is always the `_setup_persistent_brokerd()` context defined
|
||||||
|
|
@ -182,9 +146,15 @@ def broker_init(
|
||||||
modpath,
|
modpath,
|
||||||
]
|
]
|
||||||
for submodname in getattr(
|
for submodname in getattr(
|
||||||
|
brokermod,
|
||||||
|
'_brokerd_mods',
|
||||||
|
# fallback for (flat, less mature) backends which
|
||||||
|
# don't yet declare a daemon-kind mod split.
|
||||||
|
getattr(
|
||||||
brokermod,
|
brokermod,
|
||||||
'__enable_modules__',
|
'__enable_modules__',
|
||||||
[],
|
[],
|
||||||
|
),
|
||||||
):
|
):
|
||||||
subpath: str = f'{modpath}.{submodname}'
|
subpath: str = f'{modpath}.{submodname}'
|
||||||
enabled.append(subpath)
|
enabled.append(subpath)
|
||||||
|
|
@ -212,6 +182,22 @@ async def spawn_brokerd(
|
||||||
f'backend: {brokername!r}'
|
f'backend: {brokername!r}'
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# fail fast on (data-only) backends which don't offer
|
||||||
|
# ANY live order-control eps; the caller should instead
|
||||||
|
# be using paper-mode (and thus never spawning us)!
|
||||||
|
from ..data.validate import get_eps
|
||||||
|
brokerd_eps: dict = get_eps(
|
||||||
|
get_brokermod(brokername),
|
||||||
|
'brokerd',
|
||||||
|
)
|
||||||
|
if not brokerd_eps:
|
||||||
|
raise RuntimeError(
|
||||||
|
f'Backend {brokername!r} offers NO `brokerd` '
|
||||||
|
f'(live order-control) eps!?\n'
|
||||||
|
f'It is likely a datad-only provider, use '
|
||||||
|
f'paper-mode for clearing instead.\n'
|
||||||
|
)
|
||||||
|
|
||||||
(
|
(
|
||||||
brokermode,
|
brokermode,
|
||||||
tractor_kwargs,
|
tractor_kwargs,
|
||||||
|
|
@ -233,7 +219,11 @@ async def spawn_brokerd(
|
||||||
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
||||||
portal = await Services.actor_n.start_actor(
|
portal = await Services.actor_n.start_actor(
|
||||||
dname,
|
dname,
|
||||||
enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
|
enable_modules=list(dict.fromkeys(
|
||||||
|
_brokerd_service_mods
|
||||||
|
+
|
||||||
|
tractor_kwargs.pop('enable_modules')
|
||||||
|
)),
|
||||||
debug_mode=Services.debug_mode,
|
debug_mode=Services.debug_mode,
|
||||||
**tractor_kwargs
|
**tractor_kwargs
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -20,10 +20,17 @@ Handy cross-broker utils.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
# from functools import partial
|
# from functools import partial
|
||||||
|
from typing import (
|
||||||
|
Type,
|
||||||
|
)
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import httpx
|
import httpx
|
||||||
import logging
|
import logging
|
||||||
|
from msgspec import Struct
|
||||||
|
from tractor._exceptions import (
|
||||||
|
reg_err_types,
|
||||||
|
)
|
||||||
|
|
||||||
from piker.log import (
|
from piker.log import (
|
||||||
colorize_json,
|
colorize_json,
|
||||||
|
|
@ -97,6 +104,19 @@ class DataThrottle(BrokerError):
|
||||||
'''
|
'''
|
||||||
# TODO: add in throttle metrics/feedback
|
# TODO: add in throttle metrics/feedback
|
||||||
|
|
||||||
|
class SchemaMismatch(BrokerError):
|
||||||
|
'''
|
||||||
|
Market `Pair` fields mismatch, likely due to provider API update.
|
||||||
|
|
||||||
|
'''
|
||||||
|
|
||||||
|
# auto-register all `BrokerError` subtypes for
|
||||||
|
# tractor IPC exc-marshalling.
|
||||||
|
reg_err_types([
|
||||||
|
BrokerError,
|
||||||
|
*BrokerError.__subclasses__(),
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
def resproc(
|
def resproc(
|
||||||
resp: httpx.Response,
|
resp: httpx.Response,
|
||||||
|
|
@ -123,3 +143,45 @@ def resproc(
|
||||||
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
||||||
|
|
||||||
return msg if return_json else resp
|
return msg if return_json else resp
|
||||||
|
|
||||||
|
|
||||||
|
def get_or_raise_on_pair_schema_mismatch(
|
||||||
|
pair_type: Type[Struct],
|
||||||
|
fields_data: dict,
|
||||||
|
provider_name: str,
|
||||||
|
api_url: str|None = None,
|
||||||
|
) -> Struct:
|
||||||
|
'''
|
||||||
|
Boilerplate helper around assset-`Pair` field schema mismatches,
|
||||||
|
normally due to provider API updates.
|
||||||
|
|
||||||
|
'''
|
||||||
|
try:
|
||||||
|
pair: Struct = pair_type(**fields_data)
|
||||||
|
return pair
|
||||||
|
except TypeError as err:
|
||||||
|
|
||||||
|
from tractor.devx.pformat import ppfmt
|
||||||
|
repr_data: str = ppfmt(fields_data)
|
||||||
|
report: str = (
|
||||||
|
f'Field mismatch we need to codify!\n'
|
||||||
|
f'\n'
|
||||||
|
f'{pair_type!r}({repr_data})'
|
||||||
|
f'\n'
|
||||||
|
f'^^^ {err.args[0]!r} ^^^\n'
|
||||||
|
f'\n'
|
||||||
|
f"Don't panic, prolly {provider_name!r} "
|
||||||
|
f"changed their symbology schema..\n"
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
api_url
|
||||||
|
or
|
||||||
|
(api_url := pair_type._api_url)
|
||||||
|
):
|
||||||
|
report += (
|
||||||
|
f'\n'
|
||||||
|
f'Check out their API docs here:\n'
|
||||||
|
f'{api_url}\n'
|
||||||
|
)
|
||||||
|
|
||||||
|
raise SchemaMismatch(report) from err
|
||||||
|
|
|
||||||
|
|
@ -52,9 +52,25 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# `brokerd` modules
|
# per-daemon-kind (sub)mod groups: declares which of our
|
||||||
__enable_modules__: list[str] = [
|
# submods host the eps run by each daemon-actor kind as
|
||||||
|
# defined by `piker.data.validate._eps`.
|
||||||
|
# NOTE: `get_mkt_info` and `open_symbol_search` both live
|
||||||
|
# in `.feed` for this backend (no `symbols.py`).
|
||||||
|
_brokerd_mods: list[str] = [
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
|
||||||
'broker',
|
'broker',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
_datad_mods: list[str] = [
|
||||||
|
'api',
|
||||||
|
'feed',
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# tractor RPC enable arg
|
||||||
|
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||||
|
_brokerd_mods
|
||||||
|
+
|
||||||
|
_datad_mods
|
||||||
|
))
|
||||||
|
|
|
||||||
|
|
@ -49,6 +49,9 @@ from piker import config
|
||||||
from piker.clearing._messages import (
|
from piker.clearing._messages import (
|
||||||
Order,
|
Order,
|
||||||
)
|
)
|
||||||
|
from piker.brokers._util import (
|
||||||
|
get_or_raise_on_pair_schema_mismatch,
|
||||||
|
)
|
||||||
from piker.accounting import (
|
from piker.accounting import (
|
||||||
Asset,
|
Asset,
|
||||||
digits_to_dec,
|
digits_to_dec,
|
||||||
|
|
@ -370,20 +373,12 @@ class Client:
|
||||||
item['filters'] = filters
|
item['filters'] = filters
|
||||||
|
|
||||||
pair_type: Type = PAIRTYPES[venue]
|
pair_type: Type = PAIRTYPES[venue]
|
||||||
try:
|
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
||||||
pair: Pair = pair_type(**item)
|
pair_type=pair_type,
|
||||||
except Exception as e:
|
fields_data=item,
|
||||||
e.add_note(
|
provider_name='binance',
|
||||||
f'\n'
|
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
||||||
f'New or removed field we need to codify!\n'
|
|
||||||
f'pair-type: {pair_type!r}\n'
|
|
||||||
f'\n'
|
|
||||||
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
|
||||||
f'Check out their API docs here:\n'
|
|
||||||
f'\n'
|
|
||||||
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
|
||||||
)
|
)
|
||||||
raise
|
|
||||||
pair_table[pair.symbol.upper()] = pair
|
pair_table[pair.symbol.upper()] = pair
|
||||||
|
|
||||||
# update an additional top-level-cross-venue-table
|
# update an additional top-level-cross-venue-table
|
||||||
|
|
@ -609,7 +604,11 @@ class Client:
|
||||||
start_time = binance_timestamp(start_dt)
|
start_time = binance_timestamp(start_dt)
|
||||||
end_time = binance_timestamp(end_dt)
|
end_time = binance_timestamp(end_dt)
|
||||||
|
|
||||||
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
import tractor
|
||||||
|
with tractor.devx.maybe_open_crash_handler():
|
||||||
|
bs_pair: Pair = self._pairs[
|
||||||
|
mkt.bs_fqme.upper()
|
||||||
|
]
|
||||||
|
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||||
bars = await self._api(
|
bars = await self._api(
|
||||||
|
|
|
||||||
|
|
@ -22,18 +22,14 @@ from __future__ import annotations
|
||||||
from contextlib import (
|
from contextlib import (
|
||||||
asynccontextmanager as acm,
|
asynccontextmanager as acm,
|
||||||
aclosing,
|
aclosing,
|
||||||
|
AsyncExitStack,
|
||||||
)
|
)
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from functools import (
|
|
||||||
partial,
|
|
||||||
)
|
|
||||||
import itertools
|
|
||||||
from pprint import pformat
|
from pprint import pformat
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
AsyncGenerator,
|
AsyncGenerator,
|
||||||
Callable,
|
Callable,
|
||||||
Generator,
|
|
||||||
)
|
)
|
||||||
import time
|
import time
|
||||||
|
|
||||||
|
|
@ -48,6 +44,7 @@ import tractor
|
||||||
from piker.brokers import (
|
from piker.brokers import (
|
||||||
open_cached_client,
|
open_cached_client,
|
||||||
NoData,
|
NoData,
|
||||||
|
SymbolNotFound,
|
||||||
)
|
)
|
||||||
from piker._cacheables import (
|
from piker._cacheables import (
|
||||||
async_lifo_cache,
|
async_lifo_cache,
|
||||||
|
|
@ -59,7 +56,10 @@ from piker.accounting import (
|
||||||
unpack_fqme,
|
unpack_fqme,
|
||||||
)
|
)
|
||||||
from piker.types import Struct
|
from piker.types import Struct
|
||||||
from piker.data.validate import FeedInit
|
from piker.data.validate import (
|
||||||
|
FeedInit,
|
||||||
|
FeedInitializationError,
|
||||||
|
)
|
||||||
from piker.data._web_bs import (
|
from piker.data._web_bs import (
|
||||||
open_autorecon_ws,
|
open_autorecon_ws,
|
||||||
NoBsWs,
|
NoBsWs,
|
||||||
|
|
@ -76,13 +76,15 @@ from .venues import (
|
||||||
Pair,
|
Pair,
|
||||||
FutesPair,
|
FutesPair,
|
||||||
get_api_eps,
|
get_api_eps,
|
||||||
|
_futes_market_ws,
|
||||||
|
_futes_public_ws,
|
||||||
)
|
)
|
||||||
|
|
||||||
log = get_logger(name=__name__)
|
log = get_logger(name=__name__)
|
||||||
|
|
||||||
|
|
||||||
class L1(Struct):
|
class L1(Struct):
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
|
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Individual-Symbol-Book-Ticker-Streams
|
||||||
|
|
||||||
update_id: int
|
update_id: int
|
||||||
sym: str
|
sym: str
|
||||||
|
|
@ -107,14 +109,15 @@ class AggTrade(Struct, frozen=True):
|
||||||
T: int # Trade time
|
T: int # Trade time
|
||||||
m: bool # Is the buyer the market maker?
|
m: bool # Is the buyer the market maker?
|
||||||
M: bool|None = None # Ignore
|
M: bool|None = None # Ignore
|
||||||
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
nq: float|None = None # Normal quantity without RPI-order trades
|
||||||
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
# ^XXX: https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||||
|
st: int|None = None # 1 for USD-M, 2 for COIN-M
|
||||||
|
|
||||||
|
|
||||||
async def stream_messages(
|
async def stream_messages(
|
||||||
ws: NoBsWs,
|
ws: NoBsWs,
|
||||||
|
|
||||||
) -> AsyncGenerator[NoBsWs, dict]:
|
) -> AsyncGenerator[tuple[str, dict], None]:
|
||||||
|
|
||||||
# TODO: match syntax here!
|
# TODO: match syntax here!
|
||||||
msg: dict[str, Any]
|
msg: dict[str, Any]
|
||||||
|
|
@ -203,27 +206,6 @@ async def stream_messages(
|
||||||
yield 'trade', piker_quote
|
yield 'trade', piker_quote
|
||||||
|
|
||||||
|
|
||||||
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
|
||||||
'''
|
|
||||||
Create a request subscription packet dict.
|
|
||||||
|
|
||||||
- spot:
|
|
||||||
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
|
||||||
|
|
||||||
- futes:
|
|
||||||
https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
|
||||||
|
|
||||||
'''
|
|
||||||
return {
|
|
||||||
'method': 'SUBSCRIBE',
|
|
||||||
'params': [
|
|
||||||
f'{pair.lower()}@{sub_name}'
|
|
||||||
for pair in pairs
|
|
||||||
],
|
|
||||||
'id': uid
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||||
# code?!
|
# code?!
|
||||||
@acm
|
@acm
|
||||||
|
|
@ -301,6 +283,10 @@ async def get_mkt_info(
|
||||||
|
|
||||||
# uppercase since kraken bs_mktid is always upper
|
# uppercase since kraken bs_mktid is always upper
|
||||||
if 'binance' not in fqme.lower():
|
if 'binance' not in fqme.lower():
|
||||||
|
log.warning(
|
||||||
|
f'Missing `.<provider>` part in fqme ??\n'
|
||||||
|
f'fqme: {fqme!r}\n'
|
||||||
|
)
|
||||||
fqme += '.binance'
|
fqme += '.binance'
|
||||||
|
|
||||||
mkt_mode: str = ''
|
mkt_mode: str = ''
|
||||||
|
|
@ -315,6 +301,12 @@ async def get_mkt_info(
|
||||||
venue: str = venue.upper()
|
venue: str = venue.upper()
|
||||||
venue_lower: str = venue.lower()
|
venue_lower: str = venue.lower()
|
||||||
|
|
||||||
|
if not venue:
|
||||||
|
raise SymbolNotFound(
|
||||||
|
f'Invalid or missing .<venue> part in fqme?\n'
|
||||||
|
f'fqme: {fqme!r}\n'
|
||||||
|
)
|
||||||
|
|
||||||
# XXX TODO: we should change the usdtm_futes name to just
|
# XXX TODO: we should change the usdtm_futes name to just
|
||||||
# usdm_futes (dropping the tether part) since it turns out that
|
# usdm_futes (dropping the tether part) since it turns out that
|
||||||
# there are indeed USD-tokens OTHER THEN tether being used as
|
# there are indeed USD-tokens OTHER THEN tether being used as
|
||||||
|
|
@ -406,54 +398,153 @@ async def get_mkt_info(
|
||||||
return mkt, pair
|
return mkt, pair
|
||||||
|
|
||||||
|
|
||||||
@acm
|
def get_quote_routes(
|
||||||
async def subscribe(
|
mkt_mode: str,
|
||||||
ws: NoBsWs,
|
|
||||||
symbols: list[str],
|
symbols: list[str],
|
||||||
|
) -> tuple[str, ...]:
|
||||||
|
'''
|
||||||
|
Render websocket URLs for each required Binance stream class.
|
||||||
|
|
||||||
# defined once at import time to keep a global state B)
|
'''
|
||||||
iter_subids: Generator[int, None, None] = itertools.count(),
|
if len(symbols) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
'Binance quote routes require exactly one symbol; '
|
||||||
|
f'got {symbols!r}'
|
||||||
|
)
|
||||||
|
|
||||||
|
symbol: str = symbols[0]
|
||||||
|
sub_names: tuple[str, str] = ('bookTicker', 'aggTrade')
|
||||||
|
if mkt_mode == 'usdtm_futes':
|
||||||
|
return tuple(
|
||||||
|
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||||
|
for wss_url, sub_name in (
|
||||||
|
(_futes_public_ws, sub_names[0]),
|
||||||
|
(_futes_market_ws, sub_names[1]),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
wss_url: str = get_api_eps(mkt_mode)[1]
|
||||||
|
return tuple(
|
||||||
|
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||||
|
for sub_name in sub_names
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def relay_messages(
|
||||||
|
ws: NoBsWs,
|
||||||
|
send_chan: trio.MemorySendChannel,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Relay one Binance websocket into a merged quote channel.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async with (
|
||||||
|
send_chan,
|
||||||
|
aclosing(stream_messages(ws)) as msg_gen,
|
||||||
):
|
):
|
||||||
# setup subs
|
async for msg in msg_gen:
|
||||||
|
await send_chan.send(msg)
|
||||||
|
|
||||||
subid: int = next(iter_subids)
|
|
||||||
|
|
||||||
# trade data (aka L1)
|
# TODO: Encapsulate ``relay_messages()`` and ``open_quote_stream()``
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
|
# inside ``stream_quotes()`` so async-generator closure is owned by
|
||||||
l1_sub = make_sub(symbols, 'bookTicker', subid)
|
# the backend feed task.
|
||||||
await ws.send_msg(l1_sub)
|
@acm
|
||||||
|
async def open_quote_stream(
|
||||||
|
mkt_mode: str,
|
||||||
|
symbols: list[str],
|
||||||
|
connect_timeout: float = 16,
|
||||||
|
) -> AsyncGenerator[trio.MemoryReceiveChannel, None]:
|
||||||
|
'''
|
||||||
|
Open and merge the websocket routes required by a market.
|
||||||
|
|
||||||
# aggregate (each order clear by taker **not** by maker)
|
'''
|
||||||
# trades data:
|
quote_routes: tuple[str, ...] = get_quote_routes(
|
||||||
# https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
|
mkt_mode=mkt_mode,
|
||||||
agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
|
symbols=symbols,
|
||||||
await ws.send_msg(agg_trades_sub)
|
)
|
||||||
|
|
||||||
# might get ack from ws server, or maybe some
|
send, recv = trio.open_memory_channel(616)
|
||||||
# other msg still in transit..
|
async with AsyncExitStack() as stack:
|
||||||
res = await ws.recv_msg()
|
websockets: list[NoBsWs] = []
|
||||||
subid: str|None = res.get('id')
|
try:
|
||||||
if subid:
|
for wss_url in quote_routes:
|
||||||
assert res['id'] == subid
|
ws: NoBsWs = await stack.enter_async_context(
|
||||||
|
open_autorecon_ws(
|
||||||
|
url=wss_url,
|
||||||
|
connect_timeout=connect_timeout,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
websockets.append(ws)
|
||||||
|
|
||||||
yield
|
except (
|
||||||
|
TimeoutError,
|
||||||
|
*NoBsWs.recon_errors,
|
||||||
|
) as err:
|
||||||
|
routes_str: str = '\n'.join(quote_routes)
|
||||||
|
raise DataUnavailable(
|
||||||
|
f'Binance websocket connection failed for '
|
||||||
|
f'{symbols!r}:\n'
|
||||||
|
f'{routes_str}'
|
||||||
|
) from err
|
||||||
|
|
||||||
subs = []
|
async with trio.open_nursery() as tn:
|
||||||
for sym in symbols:
|
for ws in websockets:
|
||||||
subs.append("{sym}@aggTrade")
|
tn.start_soon(
|
||||||
subs.append("{sym}@bookTicker")
|
relay_messages,
|
||||||
|
ws,
|
||||||
|
send.clone(),
|
||||||
|
)
|
||||||
|
|
||||||
# unsub from all pairs on teardown
|
await send.aclose()
|
||||||
if ws.connected():
|
async with recv:
|
||||||
await ws.send_msg({
|
try:
|
||||||
"method": "UNSUBSCRIBE",
|
yield recv
|
||||||
"params": subs,
|
finally:
|
||||||
"id": subid,
|
tn.cancel_scope.cancel()
|
||||||
})
|
|
||||||
|
|
||||||
# XXX: do we need to ack the unsub?
|
|
||||||
# await ws.recv_msg()
|
async def wait_for_live_quotes(
|
||||||
|
msg_stream: AsyncGenerator | trio.MemoryReceiveChannel,
|
||||||
|
symbols: list[str],
|
||||||
|
startup_timeout: float = 16,
|
||||||
|
) -> dict:
|
||||||
|
'''
|
||||||
|
Require L1 or trade data before declaring a feed live.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if len(symbols) != 1:
|
||||||
|
raise ValueError(
|
||||||
|
'Binance quote readiness requires exactly one symbol; '
|
||||||
|
f'got {symbols!r}'
|
||||||
|
)
|
||||||
|
|
||||||
|
symbol: str = symbols[0]
|
||||||
|
msg_iter = aiter(msg_stream)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with trio.fail_after(startup_timeout):
|
||||||
|
while True:
|
||||||
|
typ, quote = await anext(msg_iter)
|
||||||
|
if (
|
||||||
|
typ in {'l1', 'trade'}
|
||||||
|
and
|
||||||
|
quote['symbol'] == symbol
|
||||||
|
):
|
||||||
|
return quote
|
||||||
|
|
||||||
|
except trio.TooSlowError as err:
|
||||||
|
raise DataUnavailable(
|
||||||
|
f'Binance quote startup timed out after '
|
||||||
|
f'{startup_timeout}s; '
|
||||||
|
f'no live data received for {symbol}'
|
||||||
|
) from err
|
||||||
|
|
||||||
|
except StopAsyncIteration as err:
|
||||||
|
raise DataUnavailable(
|
||||||
|
f'Binance quote streams closed during startup; '
|
||||||
|
f'no live data received for {symbol}'
|
||||||
|
) from err
|
||||||
|
|
||||||
|
|
||||||
async def stream_quotes(
|
async def stream_quotes(
|
||||||
|
|
@ -467,46 +558,43 @@ async def stream_quotes(
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
||||||
|
if len(symbols) != 1:
|
||||||
|
# ``data.feed.allocate_persistent_feed()`` currently starts
|
||||||
|
# one backend task per symbol. Binance multiplexing remains
|
||||||
|
# deferred until that caller contract can represent
|
||||||
|
# multi-symbol startup.
|
||||||
|
raise FeedInitializationError(
|
||||||
|
'Binance feeds require exactly one symbol; '
|
||||||
|
f'got {symbols!r}'
|
||||||
|
)
|
||||||
|
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||||
send_chan as send_chan,
|
send_chan as send_chan,
|
||||||
open_cached_client('binance') as client,
|
|
||||||
):
|
):
|
||||||
init_msgs: list[FeedInit] = []
|
|
||||||
for sym in symbols:
|
|
||||||
mkt: MktPair
|
mkt: MktPair
|
||||||
pair: Pair
|
pair: Pair
|
||||||
mkt, pair = await get_mkt_info(sym)
|
mkt, pair = await get_mkt_info(symbols[0])
|
||||||
|
mkt_mode: str = (
|
||||||
# build out init msgs according to latest spec
|
'usdtm_futes'
|
||||||
init_msgs.append(
|
if isinstance(pair, FutesPair)
|
||||||
FeedInit(mkt_info=mkt)
|
else 'spot'
|
||||||
)
|
)
|
||||||
|
|
||||||
wss_url: str = get_api_eps(client.mkt_mode)[1] # 2nd elem is wss url
|
# build out init msgs according to latest spec
|
||||||
|
init_msgs: list[FeedInit] = [
|
||||||
|
FeedInit(mkt_info=mkt),
|
||||||
|
]
|
||||||
|
|
||||||
# TODO: for sanity, but remove eventually Xp
|
ws_symbols: list[str] = [mkt.bs_mktid]
|
||||||
if 'future' in mkt.type_key:
|
async with open_quote_stream(
|
||||||
assert 'fstream' in wss_url
|
mkt_mode=mkt_mode,
|
||||||
|
symbols=ws_symbols,
|
||||||
async with (
|
) as msg_stream:
|
||||||
open_autorecon_ws(
|
quote: dict = await wait_for_live_quotes(
|
||||||
url=wss_url,
|
msg_stream=msg_stream,
|
||||||
fixture=partial(
|
symbols=ws_symbols,
|
||||||
subscribe,
|
)
|
||||||
symbols=[mkt.bs_mktid],
|
|
||||||
),
|
|
||||||
) as ws,
|
|
||||||
|
|
||||||
# avoid stream-gen closure from breaking trio..
|
|
||||||
aclosing(stream_messages(ws)) as msg_gen,
|
|
||||||
):
|
|
||||||
# log.info('WAITING ON FIRST LIVE QUOTE..')
|
|
||||||
typ, quote = await anext(msg_gen)
|
|
||||||
|
|
||||||
# pull a first quote and deliver
|
|
||||||
while typ != 'trade':
|
|
||||||
typ, quote = await anext(msg_gen)
|
|
||||||
|
|
||||||
task_status.started((init_msgs, quote))
|
task_status.started((init_msgs, quote))
|
||||||
|
|
||||||
|
|
@ -523,7 +611,7 @@ async def stream_quotes(
|
||||||
topic: str = mkt.bs_fqme
|
topic: str = mkt.bs_fqme
|
||||||
|
|
||||||
# start streaming
|
# start streaming
|
||||||
async for typ, quote in msg_gen:
|
async for typ, quote in msg_stream:
|
||||||
# period = time.time() - last
|
# period = time.time() - last
|
||||||
# hz = 1/period if period else float('inf')
|
# hz = 1/period if period else float('inf')
|
||||||
# if hz > 60:
|
# if hz > 60:
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,7 @@ Per market data-type definitions and schemas types.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
from typing import (
|
from typing import (
|
||||||
|
ClassVar,
|
||||||
Literal,
|
Literal,
|
||||||
)
|
)
|
||||||
from decimal import Decimal
|
from decimal import Decimal
|
||||||
|
|
@ -41,7 +42,10 @@ _spot_ws: str = 'wss://stream.binance.com/ws'
|
||||||
# or this one? ..
|
# or this one? ..
|
||||||
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
||||||
|
|
||||||
# https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
# https://developers.binance.com/docs/derivatives/usds-margined-futures/
|
||||||
|
# websocket-market-streams/Important-WebSocket-Change-Notice
|
||||||
|
_futes_public_ws: str = f'wss://fstream.{_domain}/public/ws'
|
||||||
|
_futes_market_ws: str = f'wss://fstream.{_domain}/market/ws'
|
||||||
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
||||||
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
||||||
|
|
||||||
|
|
@ -78,7 +82,7 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
|
||||||
),
|
),
|
||||||
'usdtm_futes': (
|
'usdtm_futes': (
|
||||||
_futes_url,
|
_futes_url,
|
||||||
_futes_ws,
|
_futes_market_ws,
|
||||||
),
|
),
|
||||||
}[venue]
|
}[venue]
|
||||||
|
|
||||||
|
|
@ -203,6 +207,8 @@ class FutesPair(Pair):
|
||||||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||||
ns_path: str = 'piker.brokers.binance:FutesPair'
|
ns_path: str = 'piker.brokers.binance:FutesPair'
|
||||||
|
|
||||||
|
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
||||||
|
|
||||||
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
||||||
# processing..
|
# processing..
|
||||||
@property
|
@property
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,7 @@ from piker.log import (
|
||||||
get_logger,
|
get_logger,
|
||||||
)
|
)
|
||||||
from ..service import (
|
from ..service import (
|
||||||
maybe_spawn_brokerd,
|
maybe_spawn_datad,
|
||||||
maybe_open_pikerd,
|
maybe_open_pikerd,
|
||||||
)
|
)
|
||||||
from ..brokers import (
|
from ..brokers import (
|
||||||
|
|
@ -187,7 +187,7 @@ def brokercheck(config, broker):
|
||||||
|
|
||||||
'''
|
'''
|
||||||
async def bcheck_main():
|
async def bcheck_main():
|
||||||
async with maybe_spawn_brokerd(broker) as portal:
|
async with maybe_spawn_datad(broker) as portal:
|
||||||
await portal.run(run_test, broker)
|
await portal.run(run_test, broker)
|
||||||
await portal.cancel_actor()
|
await portal.cancel_actor()
|
||||||
|
|
||||||
|
|
@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
|
||||||
return
|
return
|
||||||
|
|
||||||
async def main(tries):
|
async def main(tries):
|
||||||
async with maybe_spawn_brokerd(
|
async with maybe_spawn_datad(
|
||||||
tries=tries, loglevel=loglevel
|
tries=tries, loglevel=loglevel
|
||||||
) as portal:
|
) as portal:
|
||||||
# run app "main"
|
# run app "main"
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,7 @@ import trio
|
||||||
|
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
from . import get_brokermod
|
from . import get_brokermod
|
||||||
from ..service import maybe_spawn_brokerd
|
from ..service import maybe_spawn_datad
|
||||||
from . import open_cached_client
|
from . import open_cached_client
|
||||||
from ..accounting import MktPair
|
from ..accounting import MktPair
|
||||||
|
|
||||||
|
|
@ -172,7 +172,7 @@ async def symbol_search(
|
||||||
# await tractor.devx._debug.maybe_init_greenback()
|
# await tractor.devx._debug.maybe_init_greenback()
|
||||||
# tractor.pause_from_sync()
|
# tractor.pause_from_sync()
|
||||||
|
|
||||||
async with maybe_spawn_brokerd(
|
async with maybe_spawn_datad(
|
||||||
mod.name,
|
mod.name,
|
||||||
infect_asyncio=getattr(
|
infect_asyncio=getattr(
|
||||||
mod,
|
mod,
|
||||||
|
|
|
||||||
|
|
@ -425,7 +425,7 @@ class DataFeed:
|
||||||
async def stream_to_file(
|
async def stream_to_file(
|
||||||
watchlist_name: str,
|
watchlist_name: str,
|
||||||
filename: str,
|
filename: str,
|
||||||
portal: tractor._portal.Portal,
|
portal: tractor.Portal,
|
||||||
tickers: List[str],
|
tickers: List[str],
|
||||||
brokermod: ModuleType,
|
brokermod: ModuleType,
|
||||||
rate: int,
|
rate: int,
|
||||||
|
|
|
||||||
|
|
@ -47,13 +47,25 @@ __all__ = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
# per-daemon-kind (sub)mod groups: declares which of our
|
||||||
__enable_modules__: list[str] = [
|
# submods host the eps run by each daemon-actor kind as
|
||||||
|
# defined by `piker.data.validate._eps`.
|
||||||
|
# NOTE: datad-only backend (no `broker.py` yet)!
|
||||||
|
_brokerd_mods: list[str] = []
|
||||||
|
|
||||||
|
_datad_mods: list[str] = [
|
||||||
'api',
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
# 'broker',
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# tractor RPC enable arg
|
||||||
|
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||||
|
_brokerd_mods
|
||||||
|
+
|
||||||
|
_datad_mods
|
||||||
|
))
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
'infect_asyncio': True,
|
'infect_asyncio': True,
|
||||||
|
|
|
||||||
|
|
@ -23,7 +23,6 @@ from contextlib import (
|
||||||
asynccontextmanager as acm,
|
asynccontextmanager as acm,
|
||||||
)
|
)
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from functools import partial
|
|
||||||
import time
|
import time
|
||||||
from typing import (
|
from typing import (
|
||||||
Any,
|
Any,
|
||||||
|
|
@ -38,7 +37,7 @@ from rapidfuzz import process as fuzzy
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from tractor.trionics import (
|
from tractor.trionics import (
|
||||||
broadcast_receiver,
|
broadcast_receiver,
|
||||||
maybe_open_context
|
maybe_open_context,
|
||||||
collapse_eg,
|
collapse_eg,
|
||||||
)
|
)
|
||||||
from tractor import to_asyncio
|
from tractor import to_asyncio
|
||||||
|
|
@ -524,13 +523,12 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
|
||||||
|
|
||||||
|
|
||||||
async def aio_price_feed_relay(
|
async def aio_price_feed_relay(
|
||||||
|
chan: to_asyncio.LinkedTaskChannel,
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
from_trio: asyncio.Queue,
|
|
||||||
to_trio: trio.abc.SendChannel,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _trade(data: dict, receipt_timestamp):
|
async def _trade(data: dict, receipt_timestamp):
|
||||||
to_trio.send_nowait(('trade', {
|
chan.send_nowait(('trade', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'last': data,
|
'last': data,
|
||||||
|
|
@ -540,7 +538,7 @@ async def aio_price_feed_relay(
|
||||||
}))
|
}))
|
||||||
|
|
||||||
async def _l1(data: dict, receipt_timestamp):
|
async def _l1(data: dict, receipt_timestamp):
|
||||||
to_trio.send_nowait(('l1', {
|
chan.send_nowait(('l1', {
|
||||||
'symbol': cb_sym_to_deribit_inst(
|
'symbol': cb_sym_to_deribit_inst(
|
||||||
str_to_cb_sym(data.symbol)).lower(),
|
str_to_cb_sym(data.symbol)).lower(),
|
||||||
'ticks': [
|
'ticks': [
|
||||||
|
|
@ -570,7 +568,7 @@ async def aio_price_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
to_trio.send_nowait(None)
|
chan.started_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -581,11 +579,9 @@ async def open_price_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
partial(
|
|
||||||
aio_price_feed_relay,
|
aio_price_feed_relay,
|
||||||
fh,
|
fh=fh,
|
||||||
instrument
|
instrument=instrument,
|
||||||
)
|
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
@ -611,10 +607,9 @@ async def maybe_open_price_feed(
|
||||||
|
|
||||||
|
|
||||||
async def aio_order_feed_relay(
|
async def aio_order_feed_relay(
|
||||||
|
chan: to_asyncio.LinkedTaskChannel,
|
||||||
fh: FeedHandler,
|
fh: FeedHandler,
|
||||||
instrument: Symbol,
|
instrument: Symbol,
|
||||||
from_trio: asyncio.Queue,
|
|
||||||
to_trio: trio.abc.SendChannel,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
async def _fill(data: dict, receipt_timestamp):
|
async def _fill(data: dict, receipt_timestamp):
|
||||||
breakpoint()
|
breakpoint()
|
||||||
|
|
@ -637,7 +632,7 @@ async def aio_order_feed_relay(
|
||||||
install_signal_handlers=False)
|
install_signal_handlers=False)
|
||||||
|
|
||||||
# sync with trio
|
# sync with trio
|
||||||
to_trio.send_nowait(None)
|
chan.started_nowait(None)
|
||||||
|
|
||||||
await asyncio.sleep(float('inf'))
|
await asyncio.sleep(float('inf'))
|
||||||
|
|
||||||
|
|
@ -648,11 +643,9 @@ async def open_order_feed(
|
||||||
) -> trio.abc.ReceiveStream:
|
) -> trio.abc.ReceiveStream:
|
||||||
async with maybe_open_feed_handler() as fh:
|
async with maybe_open_feed_handler() as fh:
|
||||||
async with to_asyncio.open_channel_from(
|
async with to_asyncio.open_channel_from(
|
||||||
partial(
|
|
||||||
aio_order_feed_relay,
|
aio_order_feed_relay,
|
||||||
fh,
|
fh=fh,
|
||||||
instrument
|
instrument=instrument,
|
||||||
)
|
|
||||||
) as (chan, first):
|
) as (chan, first):
|
||||||
yield chan
|
yield chan
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -65,17 +65,18 @@ _brokerd_mods: list[str] = [
|
||||||
]
|
]
|
||||||
|
|
||||||
_datad_mods: list[str] = [
|
_datad_mods: list[str] = [
|
||||||
|
'api',
|
||||||
'feed',
|
'feed',
|
||||||
'symbols',
|
'symbols',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# tractor RPC enable arg
|
# tractor RPC enable arg
|
||||||
__enable_modules__: list[str] = (
|
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||||
_brokerd_mods
|
_brokerd_mods
|
||||||
+
|
+
|
||||||
_datad_mods
|
_datad_mods
|
||||||
)
|
))
|
||||||
|
|
||||||
# passed to ``tractor.ActorNursery.start_actor()``
|
# passed to ``tractor.ActorNursery.start_actor()``
|
||||||
_spawn_kwargs = {
|
_spawn_kwargs = {
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue