Compare commits
No commits in common. "wkt/land_backfill_history" and "main" have entirely different histories.
wkt/land_b
...
main
|
|
@ -1,143 +0,0 @@
|
|||
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
|
||||
|
||||
## Context
|
||||
|
||||
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
|
||||
|
||||
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
|
||||
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
|
||||
|
||||
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
|
||||
|
||||
**User-decided constraints:**
|
||||
|
||||
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
|
||||
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
|
||||
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
|
||||
|
||||
## Target topology
|
||||
|
||||
```
|
||||
pikerd
|
||||
├── datad.ib feed bus, shm writers, tsp history, symbol search
|
||||
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
|
||||
├── emsd
|
||||
│ └── paperboi.ib (paper mode; opens its feed via datad)
|
||||
└── samplerd
|
||||
```
|
||||
|
||||
## Key verified facts (load-bearing)
|
||||
|
||||
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
|
||||
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
|
||||
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
|
||||
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
|
||||
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
|
||||
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
|
||||
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
|
||||
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
|
||||
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
|
||||
|
||||
## Module decision
|
||||
|
||||
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
|
||||
|
||||
---
|
||||
|
||||
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
|
||||
|
||||
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
|
||||
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
|
||||
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
|
||||
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
|
||||
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
|
||||
- `ib`: adjust existing groups so each includes `'api'`
|
||||
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
|
||||
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
|
||||
|
||||
**Gate**: full pytest green, zero behavior change.
|
||||
|
||||
## Stage 1 — introduce `datad` daemon machinery (additive)
|
||||
|
||||
New `piker/data/_daemon.py`:
|
||||
|
||||
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
|
||||
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
|
||||
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
|
||||
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
|
||||
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
|
||||
|
||||
Supporting edits:
|
||||
|
||||
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
|
||||
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
|
||||
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
|
||||
|
||||
**Gate**: suite green + new spawn test; nothing routes through datad yet.
|
||||
|
||||
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
|
||||
|
||||
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
|
||||
|
||||
`piker/clearing/_ems.py`:
|
||||
|
||||
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
|
||||
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
|
||||
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
|
||||
|
||||
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
|
||||
|
||||
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
|
||||
|
||||
## Stage 3 — feed layer hard cutover to datad
|
||||
|
||||
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
|
||||
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
|
||||
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
|
||||
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
|
||||
- `piker/brokers/core.py:33,175` (`symbol_search`)
|
||||
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
|
||||
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
|
||||
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
|
||||
|
||||
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
|
||||
|
||||
## Stage 4 — caps-sec slimming, validation, ib client-id
|
||||
|
||||
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
|
||||
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
|
||||
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
|
||||
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
|
||||
|
||||
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
|
||||
|
||||
---
|
||||
|
||||
## Risk register
|
||||
|
||||
| Risk | Sev | Mitigation |
|
||||
|---|---|---|
|
||||
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
|
||||
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
|
||||
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
|
||||
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
|
||||
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
|
||||
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
|
||||
|
||||
## Verification (per stage gates above, plus end-to-end)
|
||||
|
||||
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
|
||||
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
|
||||
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
|
||||
|
||||
## Critical files
|
||||
|
||||
- **new** `piker/data/_daemon.py` — all datad machinery
|
||||
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
|
||||
- `piker/data/feed.py` — spawn cutover + actor-name assert
|
||||
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
|
||||
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
|
||||
- `piker/data/validate.py` — `get_eps()` helper
|
||||
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
|
||||
- `piker/accounting/cli.py` — keyword-form dialog call
|
||||
- `tests/test_services.py` — inverted + new assertions
|
||||
|
|
@ -1,233 +0,0 @@
|
|||
# Bounded IB recovery candidate
|
||||
|
||||
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
|
||||
`/home/goodboy/repos/piker`. Base:
|
||||
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
|
||||
Changes are unstaged and uncommitted. No acceptance is implied.
|
||||
|
||||
## Patch selection and provenance
|
||||
|
||||
Keep these existing landing commits:
|
||||
|
||||
| Commit | Reason |
|
||||
| --- | --- |
|
||||
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
|
||||
| `e8e0730a` | Correlate MethodProxy requests and responses |
|
||||
| `7bef50b0` | One response reader, per-request waiters |
|
||||
| `d1571ddf` | Propagate daemon-handler exceptions |
|
||||
| `dc618229` | Isolate cached feed stream consumers |
|
||||
|
||||
Proposed separate implementation boundaries, in this order:
|
||||
|
||||
1. Back out the code and original test from
|
||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
|
||||
This restores the direct-owner `_quote_streams` implementation.
|
||||
Retain the historical prompt records. Replace the removed
|
||||
mocked-cache regression with the real-bridge restart reproducer.
|
||||
This backout is not a claim that arbitrary concurrent direct
|
||||
callers have safe broadcast ownership.
|
||||
2. Backport **only** `piker/brokers/ib/feed.py` from
|
||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
|
||||
Retain its seven history-adapter regressions in
|
||||
`tests/test_ib_history.py`. This includes reset serialization,
|
||||
completion/result signalling and bounded cancellation/failure.
|
||||
It does not include the original SHM/storage orchestration patch.
|
||||
3. Backport **only** `piker/brokers/ib/feed.py` from
|
||||
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
|
||||
Retain `test_empty_history_preserves_request_boundary` and its
|
||||
six cases. This preserves no-data request boundaries and wakes
|
||||
the waiter. It does not include the original history-layer
|
||||
endpoint/filtering changes. Add the real-bridge history recovery
|
||||
case after these adapter changes are present.
|
||||
4. Import the code and original regression from
|
||||
`12b0d66e854247e92cb5a8668be71d816c72d856`.
|
||||
Add a separate real-broadcaster/production-loop regression.
|
||||
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
|
||||
This is a local UX cleanup retaining the existing `None` result.
|
||||
It does not backport the null-grouping/deduplication rewrite in
|
||||
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
|
||||
6. Record cross-boundary regression results and live qualification
|
||||
evidence, distinguishing the known callback failure from passes.
|
||||
|
||||
All three imported source commits were authored by
|
||||
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
|
||||
and the full source SHA when committing each partial backport.
|
||||
Use explicit partial-backport wording, not a whole-commit cherry
|
||||
claim. New real-runtime tests are this session's additional work.
|
||||
|
||||
The current aggregate working diff contains these boundaries in
|
||||
the same feed file. Staging that entire file would combine the
|
||||
first three boundaries. Reconstruct/stage the original per-source
|
||||
patches in the above order when preparing commits.
|
||||
|
||||
## Quote-cache history correction
|
||||
|
||||
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
|
||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
|
||||
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
|
||||
They are equivalent replays, not different implementations.
|
||||
|
||||
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
|
||||
change and retains direct ownership. The original qualification
|
||||
did not validate the combination assembled for the landing.
|
||||
|
||||
After this candidate's edits, `piker/brokers/ib/feed.py` matches
|
||||
the source at `1da9b760` byte-for-byte. Compared with the evaluated
|
||||
tip, only the later schedule/reservation APIs are absent.
|
||||
The EMS code patch has the same stable patch ID as the original
|
||||
`12b0d66e` EMS code patch:
|
||||
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
|
||||
|
||||
## Real-runtime evidence
|
||||
|
||||
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
|
||||
an asyncio-infected child, the installed pinned Tractor bridge,
|
||||
production Piker quote setup, and real IB/eventkit ticker events.
|
||||
Only the external IB connection/input is replaced with replay data.
|
||||
|
||||
Before the quote-cache backout, cancellation after ticker delivery
|
||||
raised `RemoteActorError[TrioTaskExited]` through
|
||||
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
|
||||
reported fatal path.
|
||||
|
||||
After the backout:
|
||||
|
||||
- Two quote generations finish; each subscription is cancelled,
|
||||
each asyncio producer is done and each ticker handler removed.
|
||||
- A second RPC repeats the sequence in the same actor successfully.
|
||||
- The strict callback-cleanliness case still fails: Tractor's
|
||||
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
|
||||
exception handler even though the Trio consumer recovers.
|
||||
|
||||
The history integration case keeps `Client.bars()`, MethodProxy,
|
||||
both relays, `get_bars()`, `wait_on_data_reset()` and
|
||||
`data_reset_hack()` real. It replaces external contract discovery,
|
||||
IB history replies and VNC input. A real IB error event announces
|
||||
farm recovery. The replay rejects retries before recovery and
|
||||
proves one reset followed by completion for both 1s and 60s data.
|
||||
|
||||
The new EMS case runs the production dark-clearing loop over a
|
||||
real Tractor broadcaster: both EMS and a sibling receive a tick,
|
||||
EMS cancellation removes only its subscription, and the sibling
|
||||
continues with an unchanged strict lag policy.
|
||||
|
||||
## Verification
|
||||
|
||||
Unique selected cases: **38 passed, 1 failed**.
|
||||
The failure is intentionally visible, not skipped or xfailed.
|
||||
|
||||
Run from the checkout's working Nix environment:
|
||||
|
||||
```xsh
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The adapter/proxy/feed/EMS selection initially passed 21 cases;
|
||||
the added real EMS case then passed with its imported case.
|
||||
The bridge file passed two cases and failed callback cleanliness.
|
||||
Logging/startup passed 13 cases; runtime boot passed one with the
|
||||
existing `Registrar.uid` deprecation warning.
|
||||
|
||||
The agent shell inherited another project's environment and lacked
|
||||
native libraries. Tests used the existing Piker `py313` via
|
||||
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
|
||||
libraries from the pinned flake's existing Qt dependency closure,
|
||||
excluding Nix Qt itself and glibc. No environment sync or shellHook
|
||||
ran. Piker resolved to this checkout; Tractor resolved to its
|
||||
installed package in `py313`.
|
||||
|
||||
Workflow-state preparation initially reported changed legacy
|
||||
files/modes after migration. On 2026-10-23 the human authorized
|
||||
reconciliation. Inspection found only two missing legacy guidance
|
||||
snapshots; the neutral copies matched their saved hashes and modes.
|
||||
Restoring those local snapshots resolved preparation without
|
||||
changing archived messages or the original migration receipt.
|
||||
|
||||
## Human live qualification, 2026-10-23
|
||||
|
||||
The human ran these journeys with the installed Tractor pin,
|
||||
the existing IB Gateway and pre-existing NativeDB. The agent did
|
||||
not operate the broker connection, VNC, orders or database.
|
||||
|
||||
- Initial QQQ startup/backfill recovered after pacing error 162
|
||||
with one reset attempt, then displayed live updates. The reset
|
||||
waiter appears to have timed out; the log did not establish an
|
||||
acknowledged farm reset or live quote-stream restart.
|
||||
- Sequential chart exits and reattachment preserved all four
|
||||
daemon PIDs; a read-only process/listener check confirmed this.
|
||||
- The overlapping two-chart journey used `pikerd@961108`,
|
||||
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
|
||||
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
|
||||
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
|
||||
then C closed at 08:18:37. The human reported the exercise worked;
|
||||
survivor updates are a manual observation, while timestamps and
|
||||
unchanged daemon identities corroborate overlap and service reuse.
|
||||
- Startup in that journey hit the explicit single-null-sample
|
||||
breakpoint several times before the charts overlapped. The small
|
||||
debugger cleanup above was applied afterward; this live run does
|
||||
not itself verify that later edit.
|
||||
- Departing charts still report cancellation groups and an FSP
|
||||
receiver-close `ClosedResourceError`, followed by the Qt close
|
||||
handler's self-SIGINT. Peer-disconnected final result warnings
|
||||
remain contained in the persistent services.
|
||||
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
|
||||
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
|
||||
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
|
||||
After the human continued, all three children were reported
|
||||
exited with status 0 and the root terminated at 08:19:45.
|
||||
Cancellation-only RPC error reporting and late cancel-ack
|
||||
diagnostics also remain.
|
||||
|
||||
The human explicitly accepted deferring this daemon teardown fix
|
||||
from #94 and requested loud attribution for the bounded backports.
|
||||
This is an operational smoke result, not clean-shutdown coverage.
|
||||
|
||||
## Downstream reconciliation requirements
|
||||
|
||||
Before landing #94, the human wants an easy replay of the downstream
|
||||
histories intended to follow it. Rehearse against the agreed #94
|
||||
tree in disposable worktrees, then inspect range-diffs, final code
|
||||
and relevant regressions. This note does not authorize rewriting
|
||||
the original branches or changing acceptance markers.
|
||||
|
||||
- `0846cbd4` and `1da9b760` are partial backports. Their imported
|
||||
adapter hunks may already be present, but their history/SHM/storage
|
||||
changes and omitted integration tests remain necessary downstream.
|
||||
Never drop either whole source commit merely because `feed.py`
|
||||
matches or its adapter hunk becomes empty.
|
||||
- `12b0d66e` has source-equivalent EMS code and its imported test,
|
||||
plus an additional local real-broadcaster test. Preserve that extra
|
||||
regression when resolving test-file overlap.
|
||||
- The quote-cache backout intentionally restores direct ownership
|
||||
for this landing. Reintroducing original `4af885d9` downstream
|
||||
requires qualifying cached-owner teardown with the matching
|
||||
Tractor revision; the backout is not a permanent ownership design.
|
||||
- The local single-null breakpoint removal retains legacy behavior.
|
||||
Prefer the full downstream `976db6c3` algorithm when it is replayed;
|
||||
preserving this `return None` guard would lose that later fix.
|
||||
- Real-bridge tests and manual qualification notes are local work,
|
||||
not evidence that omitted portions of the source commits landed.
|
||||
|
||||
Commit messages must name full source SHAs, original authorship,
|
||||
imported paths/tests, omissions, local additions and replay guidance.
|
||||
Use explicit partial-backport language for adapter subsets; do not
|
||||
claim a full cherry-pick or reuse the source's broader fix claims.
|
||||
|
||||
## Remaining decision
|
||||
|
||||
This candidate removes the reproduced fatal quote-restart
|
||||
propagation and imports existing reset fixes. The human exercised
|
||||
live QQQ startup, backfill and concurrent chart consumer lifetimes.
|
||||
Completion-callback and teardown errors remain; the human accepts
|
||||
them as follow-ups outside the bounded #94 landing.
|
||||
The storage/SHM portions of the downstream fixes are not covered by
|
||||
the adapter backport or its integration tests.
|
||||
|
||||
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
|
||||
`1a2d4893` stays a separate follow-up. Further Tractor teardown
|
||||
changes or storage imports require reassessing the bounded landing
|
||||
rather than silently expanding this candidate.
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
# IB recovery qualification, 2026-11-06
|
||||
|
||||
This records human-operated qualification of the bounded #94
|
||||
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
|
||||
the narrow history catch described below. It supplements the
|
||||
historical evidence in `ai/ib-recovery-candidate.md`.
|
||||
|
||||
## Runtime and dependency source
|
||||
|
||||
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
|
||||
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
|
||||
local checkout during evaluation. The human subsequently published
|
||||
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
|
||||
remote query confirmed the published OID before restoring Piker's
|
||||
Git source and recording the same OID in `uv.lock`.
|
||||
|
||||
Offline lock validation resolves 126 packages successfully. No
|
||||
environment sync was performed. This validates the lock metadata;
|
||||
it does not claim a fresh installation from the published source.
|
||||
|
||||
## Manual observations
|
||||
|
||||
The human operated Gateway, chart windows and paper orders. The
|
||||
agent did not operate live services or the account.
|
||||
|
||||
- Same-feed overlap, close/reopen and paper order submit/cancel
|
||||
worked after the existing order-dialog guard was imported.
|
||||
- Different-feed overlap originally exposed a timed-out one-second
|
||||
MNQ March 2027 gap query. PDB showed `timedout=True` and
|
||||
`out is None`. Its uncaught `NoData` canceled sibling history and
|
||||
quote tasks even though `pikerd` remained alive.
|
||||
- After the narrow catch, the human reported the overlap journeys
|
||||
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
|
||||
live feeds survived the Gateway data reset; closing all charts
|
||||
and reopening with `pikerd` left running restored live updates.
|
||||
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
|
||||
that the intended Gateway data reset was triggered.
|
||||
|
||||
These are manual observations, not automated timing or latency
|
||||
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
|
||||
labels it a connection reset.
|
||||
|
||||
## Narrow history backport
|
||||
|
||||
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
|
||||
`maybe_fill_null_segments()` is copied from
|
||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
|
||||
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
|
||||
remains in place; this does not import broader history repair.
|
||||
|
||||
The new regression uses real null detection and checks that a
|
||||
sibling in the same nursery still runs after either provider
|
||||
exception. Both cases failed before the catch. With it, the two
|
||||
focused history files passed: 17 tests. Earlier bounded-recovery
|
||||
and Tractor qualification results remain separate evidence, not
|
||||
fresh test runs of the final dependency commit tree.
|
||||
|
||||
## Remaining recovery limits
|
||||
|
||||
A full Gateway restart was triggered accidentally through its
|
||||
auto-restart settings dialog. The disconnect delivered the string
|
||||
`Peer closed connection.` to `deliver_trade_events()`. The string
|
||||
branch then reached `err['reqid']`, raising `TypeError`; its `code`
|
||||
local can also retain an earlier event's value. Full Gateway restart
|
||||
recovery is outside this bounded landing.
|
||||
|
||||
The earlier history-failure teardown also exposed a separate
|
||||
Tractor completion-callback assertion after `BrokenResourceError`.
|
||||
The history catch does not establish that this callback race is
|
||||
fixed. Broader history recovery and these disconnect paths remain
|
||||
downstream work.
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-07T16:54:53Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: gpt-6.1-sol
|
||||
git_ref: bde2457b
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested fixing all seven local PR94 review findings and
|
||||
exporting the review for human inspection in Tuicr. They asked that a
|
||||
Tractor pin adjustment be considered before adding Piker workarounds.
|
||||
|
||||
## Response summary
|
||||
|
||||
Correct the endpoint defaults, Click argument shape, daemon binding and
|
||||
registrar option handling, chart fallback, and stray debugger pause.
|
||||
Keep the absent datad implementation's test outside this landing
|
||||
branch. Add offline startup coverage and export original findings
|
||||
without modifying review states. The checked newer Tractor address
|
||||
branch still rejects mixed transports, so leave the dependency pin.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `config/conf.toml`: TCP-only default endpoints.
|
||||
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
|
||||
- `piker/service/_actor_runtime.py`: validate daemon listeners.
|
||||
- `piker/service/_daemon.py`: initialize and consume registrar options.
|
||||
- `piker/ui/cli.py`: fallback to automatic chart binding.
|
||||
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
|
||||
- `tests/test_services.py`: defer test requiring missing datad API.
|
||||
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human identified that datad tests arrived through the agent's
|
||||
mixed cherry-picks and explicitly requested deferring that test. They
|
||||
requested a chart default for existing configs, expanded the task to
|
||||
all review findings, and directed checking whether a newer Tractor pin
|
||||
could reduce the fix scope. No direct manual source edits were observed
|
||||
during these changes; acceptance remains with the human.
|
||||
|
||||
During the human review pass, the human confirmed removing the Binance
|
||||
pause and correcting CLI endpoint parsing. They requested an explicit
|
||||
template note about this pin's single-transport limitation and detailed
|
||||
comments distinguishing registrar discovery endpoints from daemon bind
|
||||
endpoints. Those explanations were added at the human's direction.
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-07T16:54:53Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: gpt-6.1-sol
|
||||
git_ref: bde2457b
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Fix all findings from the local PR94 review, prefer a Tractor pin
|
||||
change if it reduces work, and export the original review for Tuicr.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- config/conf.toml`
|
||||
|
||||
Use TCP-only defaults, retaining commented UDS alternatives. Both
|
||||
checked Tractor address branch tips still reject mixed transports.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
|
||||
|
||||
Make Click multiaddresses repeatable and normalize single strings.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
|
||||
|
||||
Validate requested daemon bind addresses instead of remote registrars.
|
||||
Normalize accepted endpoints and allow TCP ephemeral port requests.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
|
||||
|
||||
Resolve explicit or inherited registrars independently of other daemon
|
||||
options and consume the explicit keyword before forwarding kwargs.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
|
||||
|
||||
Default chart bind endpoints to None when network config is absent.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Remove the unconditional market-info debugger pause.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
|
||||
|
||||
Remove the prematurely cherry-picked datad test and add offline startup
|
||||
regressions, including a real registrar/child daemon lifecycle check.
|
||||
|
||||
## Verification
|
||||
|
||||
- Startup and logging tests: 13 passed.
|
||||
- Existing runtime boot and tagged registrar tests: 2 passed.
|
||||
- Scoped native Ruff and diff checks pass.
|
||||
- The first new actor test used a removed nursery API; it was corrected
|
||||
to the installed start_actor/Portal.run interface before passing.
|
||||
- Import initially failed without native libraries. Tests use the
|
||||
existing flake's GCC, zlib, Qt and PyQt paths without executing its
|
||||
environment-rebuilding shell hook.
|
||||
- Original seven findings exported locally as Tuicr review-add payloads
|
||||
bound to bde2457b; no session comments or remote PR were changed.
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-10T23:28:52Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 1fa45dfa
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human rejected keeping the April Tractor pin as a landing baseline.
|
||||
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
|
||||
new rollback adaptations, reuse of original Piker histories where
|
||||
possible, and a candidate prepared within their gateway relocation
|
||||
window. They asked specifically whether the runtime validation review
|
||||
hunk duplicated older endpoint history.
|
||||
|
||||
## Response summary
|
||||
|
||||
Select the reviewed address branch at an immutable locked revision and
|
||||
preserve all committed application/test compatibility code. Verify the
|
||||
minimal dependency boundary offline and preview existing consumer
|
||||
fixes without importing unrelated datad/backfill changes. Distinguish
|
||||
original-source test results from live teardown acceptance, which is
|
||||
not available in this preparation window.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
|
||||
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
|
||||
preserving Polars 1.36.0 and the rest of the locked graph.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human redirected the task from an older known-good runtime to a
|
||||
modern landing stack and insisted on reusing original history rather
|
||||
than building new compatibility adapters. They proposed stricter
|
||||
exception-group behavior as a hypothesis and set the live-test/time
|
||||
constraints. Those directions materially changed the dependency
|
||||
boundary. No direct manual edits to this final boundary were observed;
|
||||
runtime acceptance and landing remain human-owned.
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-10T23:28:52Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 1fa45dfa
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Undo the April Tractor rollback and select the reviewed
|
||||
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
|
||||
commits to new adaptations, and prepare an offline-verified candidate
|
||||
while live IB verification is deferred for gateway relocation.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
|
||||
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
|
||||
and test sources byte-for-byte to the committed PR94 tree. Keep the
|
||||
rejected rollback journal only in ignored local experiment reports.
|
||||
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
|
||||
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
|
||||
the resulting graph. Actual imports confirm the reviewed Tractor pin
|
||||
and Trio 0.32.0, not the hypothesized Trio 0.33.
|
||||
|
||||
## Original-history previews
|
||||
|
||||
The existing MethodProxy stack is retained. Read-only merge previews
|
||||
show original `4af885d9` and `b691422b` can be applied without the datad
|
||||
cutover. Original-source regressions run through temporary in-memory
|
||||
overlays against the real pinned runtime: seven cases pass and one
|
||||
cache-hit case fails because `log.at_least_level('info')` looks up INFO
|
||||
in a custom-level mapping. A separate standard logging correction
|
||||
produces eight passing cases. Its patch and execution handoff are saved
|
||||
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
|
||||
cherry-picks or source edits are performed by this boundary.
|
||||
|
||||
## Verification
|
||||
|
||||
- Startup and logging: 13 passed.
|
||||
- Runtime boot and tagged registrar normalization: 2 passed.
|
||||
- Locked sync and source whitespace checks pass.
|
||||
- Original application/test trees are unchanged from PR94 HEAD.
|
||||
- Consumer overlay with explicit logging correction: 8 passed.
|
||||
- Correction patch validates against its exact preview tree using a
|
||||
private temporary index; the real index is untouched.
|
||||
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
---
|
||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
User reported a different-feed overlap failure and asked whether
|
||||
a fix could be extracted without importing broader history work.
|
||||
PDB confirmed MNQ March 2027 one-second history timed out and
|
||||
returned None, producing NoData. After continuing PDB, sibling
|
||||
history and quote tasks terminated while pikerd remained alive.
|
||||
User requested continuing with the narrow backport.
|
||||
|
||||
## Response summary
|
||||
|
||||
Partial backport of the NoData/DataUnavailable catch from
|
||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||
A new parametrized regression uses real null detection and verifies
|
||||
that a sibling in the same nursery survives the failed repair.
|
||||
|
||||
Both new cases failed before the catch. The focused history suites
|
||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||
IB reset churn or the separate Tractor completion-callback assertion.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_history.py`: existing downstream exception catch.
|
||||
- `tests/test_history_backfill.py`: sibling survival regression.
|
||||
|
||||
## Human edits
|
||||
|
||||
User supplied live reproduction, PDB locals and cancellation logs,
|
||||
and explicitly constrained scope to a small history extraction.
|
||||
The production catch is copied from existing downstream commit
|
||||
0846cbd4, not newly designed here. New regression is agent-authored;
|
||||
no direct human source edits observed during this patch. Review and
|
||||
live qualification remain with the user.
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
---
|
||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||
|
||||
Partial backport of the NoData/DataUnavailable catch from
|
||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||
A new parametrized regression uses real null detection and verifies
|
||||
that a sibling in the same nursery survives the failed repair.
|
||||
|
||||
Both new cases failed before the catch. The focused history suites
|
||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||
IB reset churn or the separate Tractor completion-callback assertion.
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
---
|
||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested a commit plan, then a fast-forward landing of
|
||||
the #94 branch and PR updates. They agreed the temporary local
|
||||
Tractor source must be replaced, published piker_pin themselves,
|
||||
and reported "pushed it". They also supplied the manual recovery
|
||||
results and full Gateway restart failure diagnostics.
|
||||
|
||||
## Response summary
|
||||
|
||||
Restore the Git Tractor source after local qualification and lock
|
||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||
Record the human's overlap, data-reset and reopen observations,
|
||||
the narrow history extraction, and full-restart recovery limits.
|
||||
Offline uv lock validation passed without an environment sync.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
|
||||
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
|
||||
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the bounded landing scope, performed live tests,
|
||||
identified the more aggressive reset distinction, and published
|
||||
the tested Tractor ref. The agent applied the metadata edits and
|
||||
wrote the qualification record from those observations. No human
|
||||
acceptance or task-state transitions are inferred by this record.
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
---
|
||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- uv.lock`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
|
||||
|
||||
Restore the Git Tractor source after local qualification and lock
|
||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||
Record the human's overlap, data-reset and reopen observations,
|
||||
the narrow history extraction, and full-restart recovery limits.
|
||||
Offline uv lock validation passed without an environment sync.
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
# AI Prompt I/O Log
|
||||
|
||||
This directory records substantive AI-assisted changes, prompts,
|
||||
original harness identity, and unedited outputs. Human contributors
|
||||
remain accountable for code decisions.
|
||||
|
||||
Records follow the [NLNet generative AI policy][policy]. Each
|
||||
structured entry links its immutable `.raw.md` partner. Code output
|
||||
uses Git diff references to avoid duplication.
|
||||
|
||||
[policy]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||
|
|
@ -1,59 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:08:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Session-initiating instruction (driving all 7 commits in
|
||||
this series):
|
||||
|
||||
> ok i want you to become the distributed runtime and
|
||||
> concurrency expert for this project - namely acquire
|
||||
> a deep understanding of tractor and how it's used.
|
||||
> then i want you to attempt to factor our current
|
||||
> brokerd service daemon into 2 daemons: a brokerd
|
||||
> which only hosts live/paper trading endpoint tasks
|
||||
> [..] a new `datad` subdaemon which in a separate
|
||||
> subactor serves all the data feed service tasks [..]
|
||||
> give me a mega detailed plan on how to approach this,
|
||||
> and a staged approach for the implementation.
|
||||
|
||||
Proximate driver for THIS commit: during the approved
|
||||
plan's stage-0 test gate the agent discovered test
|
||||
subactors were writing into the user's REAL
|
||||
`~/.config/piker/accounting/` files (a bogus paper fill
|
||||
landed in `account.kraken.paper.toml`) because the old
|
||||
test-dir override in `config.get_app_dir()` was
|
||||
commented out and could never work in spawned
|
||||
subactors anyway. Fix applied autonomously under the
|
||||
approved plan; no explicit per-fix user prompt.
|
||||
|
||||
## Response summary
|
||||
|
||||
Restore `pytest` config-dir isolation by resolving the
|
||||
per-test tmp dir lazily (at conf-path access time) from
|
||||
`tractor` runtime-vars, which propagate down the actor
|
||||
tree; route all conf-path derivation through
|
||||
`config.get_conf_dir()` so the override is effective in
|
||||
every (sub)actor.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/config.py` — add `_maybe_use_test_dir()`;
|
||||
hook in `get_conf_dir()`; route `get_conf_path()` +
|
||||
`load()` mkdir through it
|
||||
- `piker/accounting/_ledger.py` — derive ledger dir
|
||||
via `config.get_conf_dir()` (not the global)
|
||||
- `piker/accounting/_pos.py` — same for account files
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:08:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/config.py`
|
||||
|
||||
Generated: `config._maybe_use_test_dir()` — lazily
|
||||
reads `piker_test_dir` from
|
||||
`tractor.runtime._state._runtime_vars['piker_vars']`
|
||||
(pre-loaded by `open_piker_runtime()` from the
|
||||
`tests.conftest._open_test_pikerd()` overrides) and
|
||||
calls `_override_config_dir()` when set. Hooked at the
|
||||
top of `get_conf_dir()`; `get_conf_path()` and
|
||||
`load()`'s dir-creation rerouted through
|
||||
`get_conf_dir()`.
|
||||
|
||||
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
|
||||
> `git log -1 -p --follow -- piker/accounting/_pos.py`
|
||||
|
||||
Generated: ledger/account dir derivation switched from
|
||||
the `config._config_dir` module global to
|
||||
`config.get_conf_dir()` + `mkdir(parents=True,
|
||||
exist_ok=True)` for nested tmp-dir creation.
|
||||
|
||||
Key diagnostic reasoning (verbatim from session):
|
||||
|
||||
The old (commented-out) `get_app_dir()` override gated
|
||||
on `'pytest' in sys.modules` which can NEVER work in
|
||||
spawned subactors (fresh procs, no pytest import); as
|
||||
a result test `paperboi`/daemon actors were writing
|
||||
into the user's REAL `~/.config/piker/accounting/`
|
||||
files. Evidence: `account.kraken.paper.toml` gained a
|
||||
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
|
||||
(a test fill), and `trades_kraken_paper.toml` (252B)
|
||||
contained only that fill. The resolution must be
|
||||
checked lazily at config-path access time (NOT import
|
||||
time) since sub-actors only receive runtime-vars once
|
||||
their `tractor` runtime has fully booted.
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:10:22Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
|
||||
driver: the approved plan's per-stage test gates could
|
||||
not run AT ALL — the branch base was broken vs.
|
||||
`tractor` git `main` (`AttributeError: 'tuple' object
|
||||
has no attribute 'unwrap'` at `pikerd` boot, stale
|
||||
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
|
||||
fixed forward autonomously to (re)establish the gate
|
||||
baseline, continuing the `repair_tests` branch lineage
|
||||
(verified already merged in ancestry via `git cherry`
|
||||
during a user-requested branch-overlap survey).
|
||||
|
||||
## Response summary
|
||||
|
||||
Port the service layer + test suites to current
|
||||
`tractor` APIs: addr-type normalization in
|
||||
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
|
||||
kwarg renames, modern self-cancel absorption semantics
|
||||
in the cancel-method test harness, exc/position attr
|
||||
renames, a paper-EMS startup-budget bump and a syntax
|
||||
fix in `.deribit.api`.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/service/_actor_runtime.py` — `wrap_address()`
|
||||
normalize before `.unwrap()` in `open_pikerd()`
|
||||
- `piker/service/_registry.py` — `check_for_service()`
|
||||
-> `query_actor(regaddr=)` + 2-tuple yield +
|
||||
`open_registry(addrs=)`
|
||||
- `piker/brokers/deribit/api.py` — missing comma in
|
||||
`tractor.trionics` import tuple
|
||||
- `tests/test_services.py` — `registry_addr=` kwarg,
|
||||
raddr unwraps, cancel-semantics harness rewrite,
|
||||
`fail_after` 9 -> 19s
|
||||
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:10:22Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
||||
|
||||
Generated: normalize each registry addr via
|
||||
`tractor.discovery._addr.wrap_address()` before
|
||||
`.unwrap()`-ing for the `accept_addrs` bind check —
|
||||
entries may be raw `tuple`s when passed in from (test)
|
||||
client code. Import-path precedent taken from
|
||||
`piker/cli/__init__.py:336`.
|
||||
|
||||
> `git log -1 -p --follow -- piker/service/_registry.py`
|
||||
|
||||
Generated: `check_for_service()` ported to
|
||||
`tractor.query_actor(name, regaddr=...)` (kwarg was
|
||||
`arbiter_sockaddr=`), unpacking the new
|
||||
`(sockaddr, portal)` yield, and passing the
|
||||
now-required `open_registry(addrs=Registry.addrs)`.
|
||||
|
||||
> `git log -1 -p --follow -- tests/test_services.py`
|
||||
> `git log -1 -p --follow -- tests/test_ems.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
|
||||
|
||||
Key diagnostic reasoning (verbatim from session):
|
||||
|
||||
- the "DID NOT RAISE ContextCancelled" failure: in this
|
||||
test the client actor IS pikerd (in-proc), and
|
||||
current `tractor` main absorbs a `ContextCancelled`
|
||||
whose canceller is your own actor — self-requested
|
||||
cancels now exit cleanly instead of raising; the
|
||||
'sigint' variant propagates a bare collapsed
|
||||
`KeyboardInterrupt` rather than a
|
||||
`BaseExceptionGroup`.
|
||||
- the hard-coded `trio.fail_after(9)` startup budget is
|
||||
marginal — full stack boot (pikerd -> emsd ->
|
||||
brokerd.kraken -> paperboi + live kraken symbology
|
||||
fetch) occasionally exceeds 9s -> bumped to 19s.
|
||||
- `RemoteActorError.type` -> `.boxed_type`;
|
||||
`Position.size` -> `.cumsize` (the paper engine
|
||||
populates `BrokerdPosition.size` from `pp.cumsize`).
|
||||
- overlap survey (user-requested): all of the
|
||||
`repair_tests` branch commits are already in this
|
||||
stack's ancestry; this commit finishes that branch's
|
||||
port mission (its f4c4f1e2 fixed `conftest.py`'s
|
||||
`arbiter_sockaddr` usage; this fixes the remaining
|
||||
`test_services.py` + `check_for_service()` sites).
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:11:05Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||
approved plan's "stage 0" prep. Plan-shaping user
|
||||
decisions captured via in-session Q&A:
|
||||
|
||||
- `datad.<broker>` topology: sibling of
|
||||
`brokerd.<broker>` under `pikerd` (vs. child
|
||||
subactor).
|
||||
- migration: hard cutover, staged by layer (no
|
||||
dual-mode runtime flag).
|
||||
- post-split `brokerd` scope: trading-only +
|
||||
EMS-lazy-spawned (charts/CLI never touch it).
|
||||
|
||||
## Response summary
|
||||
|
||||
Declare per-daemon-kind backend submod groups
|
||||
(`_datad_mods`/`_brokerd_mods`) in the split-style
|
||||
backends, keyed to the pre-existing
|
||||
`piker.data.validate._eps` contract; add a
|
||||
`validate.get_eps()` introspection helper; ws-import
|
||||
hygiene in `.kraken.broker`. Zero behavior change
|
||||
(`__enable_modules__` unions unchanged).
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/validate.py` — add `get_eps()`
|
||||
- `piker/brokers/kraken/__init__.py` — mod groups
|
||||
- `piker/brokers/binance/__init__.py` — mod groups
|
||||
(note: no `symbols.py`; search eps live in `.feed`)
|
||||
- `piker/brokers/deribit/__init__.py` — datad-only
|
||||
groups (no `broker.py` yet)
|
||||
- `piker/brokers/ib/__init__.py` — add `'api'` to the
|
||||
pre-existing `_datad_mods`
|
||||
- `piker/brokers/kraken/broker.py` — import
|
||||
`NoBsWs`/`open_autorecon_ws` from
|
||||
`piker.data._web_bs` directly
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:11:05Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/data/validate.py`
|
||||
|
||||
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
|
||||
returning the daemon-kind's ep funcs defined by a
|
||||
backend mod, keyed by ep name, missing eps excluded;
|
||||
sourced from the existing `_eps` grouping table
|
||||
(`'middleware' | 'datad' | 'brokerd'`).
|
||||
|
||||
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
|
||||
|
||||
Key exploration findings driving the design (from the
|
||||
planning phase, 3 parallel explore agents + 1 architect
|
||||
agent):
|
||||
|
||||
- the codebase already anticipated this split:
|
||||
`validate._eps` groups eps into 'datad' vs 'brokerd'
|
||||
kinds and `ib/__init__.py` already declared
|
||||
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
|
||||
carried the literal TODO "rename the daemon to datad
|
||||
prolly once we split up broker vs. data tasks into
|
||||
separate actors?".
|
||||
- the feared kraken feed<->broker "shared ws state"
|
||||
hazard is mild: `NoBsWs`/`open_autorecon_ws`
|
||||
originate in `piker.data._web_bs` (re-exported via
|
||||
`.kraken.feed`); only `stream_messages` is feed-local
|
||||
and it's a pure parser — in-process imports only,
|
||||
each actor opens its own ws conn.
|
||||
- `enable_modules` gates RPC entry, NOT python imports,
|
||||
so backend trading mods may keep importing feed-side
|
||||
helpers in-process post-split.
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:11:42Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||
approved plan's "stage 1": introduce the `datad` daemon
|
||||
machinery additively (nothing routes through it yet).
|
||||
User-decided constraint applied: `datad.<broker>` is a
|
||||
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
|
||||
via the existing `Services` + `maybe_spawn_daemon()`
|
||||
machinery.
|
||||
|
||||
## Response summary
|
||||
|
||||
New `piker/data/_daemon.py` hosting the
|
||||
`datad.<broker>` feed-only daemon-actor: lifetime
|
||||
fixture owning the actor-global `_FeedsBus`, init/spawn
|
||||
fns mirroring `.brokers._daemon` conventions and the
|
||||
`samplerd` sub-daemon precedent, plus root-mod
|
||||
registration, `piker.service` re-exports and a spawn
|
||||
test.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_daemon.py` — NEW:
|
||||
`_setup_persistent_datad()`, `datad_init()`,
|
||||
`spawn_datad()`, `maybe_spawn_datad()`,
|
||||
`_datad_service_mods`
|
||||
- `piker/service/_actor_runtime.py` — add
|
||||
`piker.data._daemon` to `_root_modules`
|
||||
- `piker/service/__init__.py` — re-export spawn eps
|
||||
- `tests/test_services.py` — add `test_datad_spawn`
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,60 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:11:42Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/data/_daemon.py`
|
||||
|
||||
Generated symbols + key design decisions:
|
||||
|
||||
- `_datad_service_mods: list[str]` — datad-always
|
||||
enabled mods, the data-side successor to the old
|
||||
`piker.brokers._daemon._data_mods` set; kept minimal
|
||||
per the caps-sec model.
|
||||
- `_setup_persistent_datad()` — `@tractor.context`
|
||||
lifetime fixture: console-log setup then allocates
|
||||
the actor-global feed bus via
|
||||
`feed.get_feed_bus(brokername, service_nursery)`
|
||||
exactly as the old brokerd fixture did, pinned open
|
||||
with `ctx.started()` + `sleep_forever()`.
|
||||
- `datad_init()` — actor name `f'datad.{brokername}'`;
|
||||
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
|
||||
`infect_asyncio=True`); builds `enable_modules` from
|
||||
`getattr(brokermod, '_datad_mods',
|
||||
getattr(brokermod, '__enable_modules__', []))` as
|
||||
the flat-backend fallback.
|
||||
- `spawn_datad()` — `Services.actor_n.start_actor()` +
|
||||
`Services.start_service_task()` exactly mirroring
|
||||
`spawn_brokerd()`; dedup-composes enable mods via
|
||||
`list(dict.fromkeys(...))`.
|
||||
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
|
||||
service_name=f'datad.{brokername}', ...)`.
|
||||
|
||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
||||
> `git log -1 -p --follow -- piker/service/__init__.py`
|
||||
> `git log -1 -p --follow -- tests/test_services.py`
|
||||
|
||||
Design rationale (verbatim from session):
|
||||
|
||||
- `_root_modules` must gain `piker.data._daemon` so
|
||||
`pikerd_portal.run(spawn_datad, ...)` resolves in
|
||||
the root.
|
||||
- the `Services`-based impl style deliberately mirrors
|
||||
`spawn_brokerd()` so the eventual `tractor.hilevel`
|
||||
`ServiceMngr` port (see the `service_mng_to_tractor`
|
||||
branch's d8c21d44 prep, surfaced by the
|
||||
user-requested branch-overlap survey) lands
|
||||
symmetrically on both spawn fns.
|
||||
- mod placement (`piker/data/_daemon.py` vs.
|
||||
generalizing `piker.brokers._daemon`) follows the
|
||||
per-subsystem daemon-mod convention
|
||||
(`.clearing._ems`, `.data._sampling`) and resolves
|
||||
the existing TODO at `brokers/_daemon.py:49` ("move
|
||||
this def to the `.data` subpkg").
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:12:26Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||
approved plan's "stage 2": decouple the clearing layer
|
||||
from `feed.portals` BEFORE the feed cutover so live
|
||||
trading works at every stage boundary. User-decided
|
||||
constraint applied: post-split `brokerd` is
|
||||
trading-only and spawned LAZILY only by `emsd`'s
|
||||
`open_brokerd_dialog()` path.
|
||||
|
||||
## Response summary
|
||||
|
||||
Kill the single coupling forcing feed + trading eps
|
||||
into one actor (`Router.open_trade_relays()` pulling
|
||||
its trades portal from `feed.portals[brokermod]`):
|
||||
`open_brokerd_dialog()` now (maybe) spawns/finds
|
||||
`brokerd.<broker>` itself and ONLY when a live
|
||||
trades-ep will actually open; paper mode never touches
|
||||
it. Pre-cutover this is a pure refactor (registry
|
||||
lookup finds the same feed-spawned daemon).
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
|
||||
re-sig + inner `acquire_live_portal()`;
|
||||
`Router.maybe_open_brokerd_dialog()` drops `portal`
|
||||
param; `open_trade_relays()` drops the
|
||||
`feed.portals` lookup
|
||||
- `piker/accounting/cli.py` — keyword-form `portal=`
|
||||
override kept for the `piker ledger` ad-hoc actor
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:12:26Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/clearing/_ems.py`
|
||||
|
||||
Generated: new `open_brokerd_dialog()` signature
|
||||
`(brokermod, exec_mode, fqme=None, portal=None,
|
||||
loglevel=None)` with an inner `@acm
|
||||
acquire_live_portal()` that yields the caller-provided
|
||||
`portal` override (the `piker ledger` path) else
|
||||
`maybe_spawn_brokerd(brokermod.name)` — designated THE
|
||||
one place a live, credentialed `brokerd.<broker>` gets
|
||||
booted post-split. The eager
|
||||
`portal.open_context(trades_endpoint, ...)`
|
||||
construction moved inside that block so the paper
|
||||
short-circuit never acquires a live portal.
|
||||
|
||||
> `git log -1 -p --follow -- piker/accounting/cli.py`
|
||||
|
||||
Key analysis (verbatim from session):
|
||||
|
||||
- `feed.portals` had exactly ONE trading consumer:
|
||||
`piker/clearing/_ems.py:671` (`portal =
|
||||
feed.portals[brokermod]`) — the single coupling
|
||||
forcing feed + trading into one actor.
|
||||
- `piker ledger` (`accounting/cli.py:100-157`) is a
|
||||
hidden consumer: it calls `broker_init()` directly,
|
||||
spawns its own ad-hoc actor and passes the portal
|
||||
positionally — the new signature must keep an
|
||||
explicit `portal:` override param for this path.
|
||||
- stage sequencing: clearing decouple lands BEFORE the
|
||||
feed cutover, otherwise `feed.portals` would hand
|
||||
emsd a `datad` portal and live `open_trade_dialog`
|
||||
RPC would fail.
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:12:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||
approved plan's "stage 3": the feed-layer hard cutover
|
||||
(user-decided: no dual-mode runtime flag, each stage
|
||||
lands fully working).
|
||||
|
||||
## Response summary
|
||||
|
||||
The topology flip: all data-feed consumers route to
|
||||
`datad.<broker>`; the `brokerd` lifetime fixture slims
|
||||
to trading-only; UI/CLI data-flavoured spawn sites
|
||||
repointed; test expectations inverted incl. an explicit
|
||||
negative assert that paper sessions spawn NO
|
||||
`brokerd.<broker>`.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/feed.py` — `open_feed()` ->
|
||||
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
|
||||
assert -> `'datad'`; `install_brokerd_search` ->
|
||||
`install_datad_search`; comment sweep
|
||||
- `piker/brokers/_daemon.py` —
|
||||
`_setup_persistent_brokerd()` slimmed (feed-bus
|
||||
alloc moved to the datad fixture)
|
||||
- `piker/ui/_app.py`, `piker/ui/cli.py`,
|
||||
`piker/brokers/core.py`, `piker/brokers/cli.py`,
|
||||
`piker/ui/kivy/option_chain.py` — spawn-site
|
||||
repoints
|
||||
- `tests/test_services.py` — datad assertions + the
|
||||
no-`brokerd.kraken` negative check
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:12:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/data/feed.py`
|
||||
|
||||
Generated: `open_feed()` spawn cutover with the
|
||||
`maybe_spawn_datad` import done "relative-direct" from
|
||||
`._daemon` — NOT via `piker.service` — to dodge a
|
||||
partial-init cycle: when `piker.data.feed` loads as
|
||||
part of `piker.service.__init__` executing its own
|
||||
`piker.data._daemon` import, the service pkg is
|
||||
mid-init and its `maybe_spawn_datad` binding does not
|
||||
exist yet. The `open_feed_bus()` local-state sanity
|
||||
assert flips `'brokerd' in servicename` ->
|
||||
`'datad' in servicename` (the only actor-name assert
|
||||
in the tree, verified by grep).
|
||||
|
||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
||||
|
||||
Generated: `_setup_persistent_brokerd()` slimmed to
|
||||
console-log setup + pinned-open ctx; drops the bus
|
||||
alloc, the `assert not feed._bus`, the service nursery
|
||||
(backend `open_trade_dialog()` ctxs own their task
|
||||
trees) and the `eg.ExceptionGroup` handler. The
|
||||
`piker ledger` ad-hoc actor enters this same slimmed
|
||||
fixture — exactly what it needs.
|
||||
|
||||
> `git log -1 -p --follow -- piker/ui/_app.py`
|
||||
> `git log -1 -p --follow -- piker/ui/cli.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/core.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/cli.py`
|
||||
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
|
||||
> `git log -1 -p --follow -- tests/test_services.py`
|
||||
|
||||
Verification (from session): per-suite gates green at
|
||||
this commit (services 5-passed incl. the new negative
|
||||
assert, feeds 3-passed); a headless live smoke
|
||||
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
|
||||
on an alt registry port) confirmed quotes flowing via
|
||||
`datad.kraken` + `samplerd` with `check_for_service(
|
||||
'brokerd.kraken') is None`. Known pre-existing flake
|
||||
documented: `test_multi_fill_positions`' second
|
||||
in-proc runtime boot wedges ~50% (zombie subactor w/
|
||||
unread parent-IPC bytes); reproduced with the split
|
||||
fully reverted so NOT a regression of this work.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:13:44Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
|
||||
approved plan's final "stage 4": caps-sec slimming of
|
||||
the (live, credentialed) trading actor + the `ib`
|
||||
dual-daemon `client_id` collision mitigation flagged in
|
||||
the plan's risk register.
|
||||
|
||||
## Response summary
|
||||
|
||||
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
|
||||
spawn fails fast for datad-only backends with a "use
|
||||
paper-mode" error; `ib`'s default api-gw `client_id`
|
||||
gets a per-daemon-kind offset so `datad.ib` +
|
||||
`brokerd.ib` don't collide on connect.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
|
||||
`_brokerd_service_mods`; `broker_init()` reads
|
||||
`_brokerd_mods` (fallback `__enable_modules__`);
|
||||
`spawn_brokerd()` fail-fast via `validate.get_eps()`
|
||||
- `piker/brokers/ib/api.py` — role-based `client_id`
|
||||
offset in `load_aio_clients()`
|
||||
- `piker/cli/__init__.py` — resolved "expose datad"
|
||||
TODO
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:13:44Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
|
||||
|
||||
Generated: the fail-fast originally landed in
|
||||
`broker_init()` but was relocated to `spawn_brokerd()`
|
||||
mid-implementation after realizing `piker ledger`
|
||||
calls `broker_init()` directly even for paper accounts
|
||||
on datad-only backends (would have crashed the cli);
|
||||
the service-spawn path is the correct enforcement
|
||||
seam. Error text:
|
||||
|
||||
Backend 'kucoin' offers NO `brokerd` (live
|
||||
order-control) eps!? It is likely a datad-only
|
||||
provider, use paper-mode for clearing instead.
|
||||
|
||||
(verified live via a `trio.run()` unit check.)
|
||||
|
||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
||||
|
||||
Generated: in `load_aio_clients()`, when `client_id`
|
||||
is the 6116 default: `datad`-named actors offset +16
|
||||
(disjoint from `brokerd`'s linear `client_id + i`
|
||||
retry range), other non-`brokerd` (ad-hoc test/cli)
|
||||
actors +32. Rationale from the plan's risk register:
|
||||
post-split BOTH per-broker daemons connect to the same
|
||||
TWS/gw endpoint; a shared default id collides and
|
||||
burns up to `connect_timeout * retries` (90s) in
|
||||
retry cycles.
|
||||
|
||||
> `git log -1 -p --follow -- piker/cli/__init__.py`
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:33:09Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
||||
scope: docs
|
||||
substantive: true
|
||||
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The session-initiating instruction, verbatim (this doc
|
||||
IS its "mega detailed plan" deliverable):
|
||||
|
||||
> ok i want you to become the distributed runtime and
|
||||
> concurrency expert for this project - namely acquire
|
||||
> a deep understanding of tractor and how it's used.
|
||||
> then i want you to atttempt to factor our current
|
||||
> brokerd service daemon into 2 daemons:
|
||||
>
|
||||
> - a brokerd which only hosts live/paper trading
|
||||
> endoint tasks as defined namely within all
|
||||
> piker/brokers/<backend>/broker.py mods
|
||||
> - a new `datad` subdaemon which in a separate
|
||||
> subactor serves all the data feed service tasks
|
||||
> namely delivered by endpoints in
|
||||
> piker/brokers/<backend>/feed.py (or similar for
|
||||
> less mature backends) and as more rigorously
|
||||
> defined by the validation machinery in
|
||||
> piker.data.validate.
|
||||
>
|
||||
> give me a mega detailed plan on how to approach
|
||||
> this, and a staged approach for the implementation.
|
||||
|
||||
Plan-shaping user decisions (in-session Q&A): sibling
|
||||
topology under `pikerd`; hard cutover staged by layer;
|
||||
trading-only EMS-lazy-spawned `brokerd`. The human
|
||||
staged the doc copy into `ai/claude-code/plans/` and
|
||||
requested this commit after the 7 implementation
|
||||
commits landed.
|
||||
|
||||
## Response summary
|
||||
|
||||
The staged design doc for the `brokerd` -> (`datad` +
|
||||
`brokerd`) split: context, target supervision topology,
|
||||
load-bearing verified facts, per-stage file-level
|
||||
changes with gates, risk register and an end-to-end
|
||||
verification matrix. Produced via 3 parallel explore
|
||||
agents + 1 architect agent + human Q&A before any code
|
||||
was written; all 7 implementation commits in this
|
||||
branch reference its stages in their provenance
|
||||
entries.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `ai/claude-code/plans/datad_service.md` — the design
|
||||
plan doc (human-staged copy of the AI-authored plan)
|
||||
|
||||
## Human edits
|
||||
|
||||
None to the doc content — committed as generated. NB:
|
||||
the implementation deviated from the plan-as-written in
|
||||
4 places, see the raw file's deviation log.
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:33:09Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry; the AI output here is the
|
||||
committed doc itself (a design plan, not code) so no
|
||||
verbatim copy is duplicated:
|
||||
|
||||
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
|
||||
|
||||
## Implementation deviation log
|
||||
|
||||
Where the landed 7-commit series differs from the plan
|
||||
as written (recorded for reviewer transparency; the
|
||||
plan doc is committed unmodified):
|
||||
|
||||
1. Commit ordering: the `pytest` config-dir isolation
|
||||
fix lands BEFORE the `tractor`-API drift port. The
|
||||
plan's stage gates assumed a runnable+isolated test
|
||||
baseline; per-commit gating exposed that without
|
||||
isolation the paper-EMS test reads the user's real
|
||||
(polluted) `account.kraken.paper.toml` and reddens.
|
||||
2. The plan's "stage 0: full pytest green" gate
|
||||
required first repairing pre-existing branch
|
||||
breakage vs `tractor` git `main` (boot
|
||||
`AttributeError`, stale discovery/exc/position
|
||||
APIs) — that repair became its own commit
|
||||
("Port service+tests to latest `tractor` APIs")
|
||||
rather than plan-stage work.
|
||||
3. Stage-4 fail-fast placement: the plan said
|
||||
`broker_init()` raises on brokerd-ep-less backends;
|
||||
implementation moved the raise to `spawn_brokerd()`
|
||||
since `piker ledger` calls `broker_init()` directly
|
||||
even for paper accounts on datad-only backends.
|
||||
4. `brokers/_daemon.py` change grouping: the
|
||||
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
|
||||
type-only import) landed with the caps-sec slim
|
||||
commit instead of the fixture-slim commit, keeping
|
||||
each intermediate tree import-clean without
|
||||
sub-hunk surgery.
|
||||
|
||||
Also of record: the plan's "Verification" matrix was
|
||||
executed as written (per-suite gates each stage, the
|
||||
headless datad-feed smoke, the kucoin fail-fast unit
|
||||
check); the known pre-existing ~50% second-runtime-boot
|
||||
test wedge was characterized and excluded as a
|
||||
regression via revert-testing.
|
||||
|
|
@ -1,60 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T21:35:49Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
User bug report from live `ib` testing of the
|
||||
(datad|brokerd)-split branch:
|
||||
|
||||
> ok doing some testing and noticing live orders do not
|
||||
> work otb, pikerd show this on live submission,
|
||||
> [pdb traceback: `RuntimeError("Can not order
|
||||
> {symbol}, no live feed?")` at `ib/api.py:1152`
|
||||
> `submit_limit()` with `self._contracts == {}` in the
|
||||
> `brokerd.ib` actor + a `TrioTaskExited` teardown
|
||||
> cascade]
|
||||
|
||||
Follow-up user direction fixing the design intent:
|
||||
|
||||
> i think the main thing here is that [brokerd] should
|
||||
> be able to always submit orders without a live feed
|
||||
> being up and registered in the same subactor right?
|
||||
|
||||
## Response summary
|
||||
|
||||
Root cause: pre-split the feed code populated the
|
||||
actor-local `Client._contracts` cache (via
|
||||
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
|
||||
SAME process as order submission; post-split the
|
||||
trading actor's client is never warmed. Fix: lazily
|
||||
qualify + cache the contract per order request in
|
||||
`handle_order_requests()` by running the same
|
||||
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
|
||||
plus per-order error relay (`BrokerdError`) so one bad
|
||||
submission can't crash the whole trades dialog (the
|
||||
`TrioTaskExited` storm was teardown cascade from the
|
||||
original raise).
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/broker.py` — thread `proxies` into
|
||||
`handle_order_requests()`; lazy contract qualify on
|
||||
cache-miss; guard `submit_limit()` w/ `BrokerdError`
|
||||
relay; uncomment the (anticipatory) `get_mkt_info`
|
||||
import
|
||||
- `piker/brokers/ib/api.py` — fix the non-f-string
|
||||
raise msg + document the new qualification contract
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated. Live `ib` order retest
|
||||
performed by the human post-commit.
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T21:35:49Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); recorded from the live debug
|
||||
session per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
|
||||
|
||||
Key diagnostic chain (from session):
|
||||
|
||||
- pdb showed `Client._contracts == {}` inside
|
||||
`brokerd.ib`'s `submit_limit()`; the cache has
|
||||
exactly TWO write sites: `Client.find_contracts()`
|
||||
(api.py, keys `f'{sym}.{exch}.ib'`) and
|
||||
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
|
||||
'nvda.nasdaq' — NO `.ib` suffix).
|
||||
- `BrokerdOrder.symbol` arrives as the bs_fqme form
|
||||
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
|
||||
site produces the key `submit_limit()` reads —
|
||||
ie. pre-split it was the feed's in-proc
|
||||
`get_mkt_info(sym, proxy=proxy)` call keeping orders
|
||||
working, NOT `find_contracts()`.
|
||||
- the existing TODO at `symbols.py:642-644` literally
|
||||
predicted this: "this is going to be problematic
|
||||
if/when we split out the datad vs. brokerd actors
|
||||
since the mktmap lookup table will now be
|
||||
inaccessible.."
|
||||
- instance identity verified: `proxy._aio_ns` IS the
|
||||
same `Client` obj as `_accounts2clients[account]`
|
||||
(both sourced from the `load_aio_clients()` cache via
|
||||
`open_client_proxies()`), so a brokerd-side
|
||||
`get_mkt_info(fqme, proxy=proxies[account])` warms
|
||||
exactly the dict `submit_limit()` reads. It also
|
||||
populates `client._cons2mkts` which the
|
||||
position-audit path (`broker.py` backup-table code)
|
||||
needs in this actor anyway.
|
||||
- the `TrioTaskExited` storm in the user's log
|
||||
(`recv_trade_updates`, `open_aio_client_method_relay`
|
||||
aio tasks) is teardown cascade: the raise crashed
|
||||
`handle_order_requests` -> nursery teardown ripped
|
||||
the trio sides of still-running aio relay tasks.
|
||||
Hence the added per-order try/except ->
|
||||
`BrokerdError` relay hardening so a single bad
|
||||
submission degrades to an EMS error msg instead of
|
||||
killing the backend's entire order-ctl dialog.
|
||||
|
||||
Verification: `tests/test_services.py` (5 passed) +
|
||||
`tests/test_ems.py` (6 passed) regression-green; live
|
||||
`ib` submission retest delegated to the human (needs a
|
||||
running TWS/gw).
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
# AI Prompt I/O Log — claude
|
||||
|
||||
This directory tracks prompt inputs and model
|
||||
outputs for AI-assisted development using
|
||||
`claude` (claude-code CLI).
|
||||
|
||||
## Policy
|
||||
|
||||
Prompt logging follows the
|
||||
[NLNet generative AI policy][nlnet-ai].
|
||||
All substantive AI contributions are logged
|
||||
with:
|
||||
- Model name and version
|
||||
- Timestamps
|
||||
- The prompts that produced the output
|
||||
- Unedited model output (`.raw.md` files)
|
||||
|
||||
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||
|
||||
## Usage
|
||||
|
||||
Entries are created by the `/prompt-io` skill
|
||||
or automatically via `/commit-msg` integration.
|
||||
|
||||
Each commit carrying AI-generated changes links
|
||||
to its provenance entry via a `Prompt-IO:`
|
||||
commit-msg trailer; entries use "diff-ref mode"
|
||||
(pointers into `git log -p` instead of verbatim
|
||||
code copies) to avoid duplicating committed
|
||||
code.
|
||||
|
||||
Human contributors remain accountable for all
|
||||
code decisions. AI-generated content is never
|
||||
presented as human-authored work.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-22T00:13:57Z
|
||||
git_ref: 16f3cd6f
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260722T001357Z_16f3cd6f_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the gap-checker and history-backfiller repair plan. Start
|
||||
with deterministic analysis characterization, then fix canonical
|
||||
sorting, dedupe-derived columns, positive gap detection, and null
|
||||
grouping.
|
||||
|
||||
During review, the user challenged the replacement null-grouping
|
||||
algorithm because its initial tests did not assert exact endpoints.
|
||||
The user explicitly required retaining detailed, human-oriented
|
||||
algorithm documentation such as the existing ASCII segment map.
|
||||
|
||||
## Response summary
|
||||
|
||||
Added deterministic analysis regressions and corrected normalization
|
||||
so sorting and duplicate resolution precede derived-column
|
||||
calculation. Unified null grouping for NumPy and Polars inputs,
|
||||
documented its exact inclusive-boundary contract with an adapted
|
||||
visual map, and verified all small null layouts against an independent
|
||||
state-machine oracle. Retained sorted smart-dedupe output.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_anal.py` - canonical gap and documented null grouping
|
||||
- `piker/tsp/_dedupe_smart.py` - sorted duplicate resolution
|
||||
- `tests/test_tsp_analysis.py` - exact and exhaustive regressions
|
||||
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not yet been edited by the human.
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-22T00:13:57Z
|
||||
git_ref: 16f3cd6f
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
||||
|
||||
Generated canonical gap and null-segment normalization. Derived
|
||||
datetime and delta columns are recomputed after final sorting and
|
||||
duplicate resolution, and negative deltas no longer become ordinary
|
||||
gaps.
|
||||
|
||||
NumPy and Polars null grouping share one implementation while
|
||||
retaining the human-oriented visual map of contiguous zero runs. The
|
||||
documented contract defines absolute inclusive endpoints, margin
|
||||
expansion, frame boundary clamping, contiguous-index validation, and
|
||||
requested-column selection.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_dedupe_smart.py`
|
||||
|
||||
Adjusted smart dedupe to honor sorted output and resolve duplicate
|
||||
timestamps without retaining stale order.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
||||
|
||||
Added deterministic synthetic OHLCV coverage for sorted dedupe, fresh
|
||||
derived columns, and positive gap detection. Null grouping is checked
|
||||
against readable endpoint examples and an independent state-machine
|
||||
oracle over every layout through six rows, three margin widths, and
|
||||
both NumPy and Polars inputs.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
`timeout -k 5 30 python -m pytest -q tests/test_tsp_analysis.py`
|
||||
|
||||
Result: 11 passed.
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-22T00:13:58Z
|
||||
git_ref: 16f3cd6f
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the structured backfill phase after the analysis regressions.
|
||||
Characterize provider exhaustion and genuinely empty frames, then remove
|
||||
completion paths that can hang after child-task exit.
|
||||
|
||||
## Response summary
|
||||
|
||||
Removed orphan-prone completion events from the structured nursery flow
|
||||
and checked empty provider frames before indexing timestamp endpoints.
|
||||
Added deterministic timeout-bounded regressions for both exit paths.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
|
||||
- `tests/test_history_backfill.py` - deterministic exit regressions
|
||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not yet been edited by the human.
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-22T00:13:58Z
|
||||
git_ref: 16f3cd6f
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||
|
||||
Removed redundant completion events whose consumers waited only after
|
||||
their owning nurseries had joined. Added empty provider-frame handling
|
||||
before timestamp endpoint indexing so provider exhaustion exits cleanly.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||
|
||||
Added bounded, actor-free regressions for `DataUnavailable` and empty
|
||||
provider-frame completion.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
|
||||
|
||||
Result: 2 passed.
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-22T02:41:33Z
|
||||
git_ref: 3a4f6737
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260722T024133Z_3a4f6737_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user challenged the analysis commit message's claim that only
|
||||
positive sample-step gaps are detected and asked whether out-of-order
|
||||
timestamps were simply ignored. After confirming the missing separate
|
||||
reporting path, the user instructed the agent to implement the fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
Separated missing-history gaps from timestamp-ordering errors without
|
||||
changing the existing `dedupe()` return shape. Added a structured
|
||||
detector and made normalization warn before sorting removes evidence of
|
||||
duplicates or reversed timestamps.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_anal.py` - ordering detector and dedupe reporting
|
||||
- `piker/tsp/__init__.py` - public detector export
|
||||
- `tests/test_tsp_analysis.py` - exact ordering-error regression
|
||||
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not yet been edited by the human.
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-22T02:41:33Z
|
||||
git_ref: 3a4f6737
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
|
||||
|
||||
Added `detect_time_ordering_errors()` to return non-positive timestamp
|
||||
steps as structured rows. `dedupe()` now warns with those rows before
|
||||
sorting and duplicate removal erase the original ordering evidence.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
|
||||
|
||||
Exported the ordering detector through the public TSP package.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
|
||||
|
||||
Changed the negative-delta regression to verify exact predecessor,
|
||||
timestamp, and delta values, confirm exclusion from missing-history
|
||||
gaps, and require explicit normalization reporting.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
- `tests/test_tsp_analysis.py`: 11 passed
|
||||
- Ruff: passed
|
||||
- `git diff --check`: passed
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-22T02:41:34Z
|
||||
git_ref: 3a4f6737
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260722T024134Z_3a4f6737_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user asked why the in-progress `piker/storage` changes were omitted
|
||||
from the earlier commit plan. After receiving an inventory of unfinished
|
||||
Phase 3 requirements, the user instructed the agent to complete the work.
|
||||
|
||||
## Response summary
|
||||
|
||||
Finished non-destructive, atomic NativeDB updates and integrated them
|
||||
with history startup and reverse backfill. Expanded tests through two
|
||||
adversarial review rounds that caught and corrected cross-client lock
|
||||
blocking, stale temp indexing, discontinuous persisted indexes, startup
|
||||
snapshot races, missing latest-frame persistence, notification wedges,
|
||||
and mutable/truncated SHM persistence.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/storage/__init__.py` - incremental storage contract
|
||||
- `piker/storage/nativedb.py` - validated atomic merge persistence
|
||||
- `piker/storage/marketstore/__init__.py` - non-duplicate update mode
|
||||
- `piker/tsp/_history.py` - provider-delta and startup persistence
|
||||
- `tests/test_storage_nativedb.py` - NativeDB durability regressions
|
||||
- `tests/test_history_backfill.py` - persistence lifecycle regressions
|
||||
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not yet been edited by the human.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-22T02:41:34Z
|
||||
git_ref: 3a4f6737
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/storage/__init__.py piker/storage/nativedb.py piker/storage/marketstore/__init__.py`
|
||||
|
||||
Added an incremental storage update contract. NativeDB now validates
|
||||
canonical numeric OHLCV data, serializes writers both actor-locally and
|
||||
through cancellable filesystem locks, merges with incoming-wins seam
|
||||
resolution, regenerates contiguous indexes, reopens and validates a
|
||||
same-directory temporary parquet, fsyncs it, atomically replaces the
|
||||
target, fsyncs the directory, and publishes cache only after replacement.
|
||||
|
||||
NativeDB indexing ignores lock and crash-left temporary files.
|
||||
MarketStore uses its non-duplicate write mode for incremental updates.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||
|
||||
Separated full provider deltas from capacity-truncated SHM pushes.
|
||||
Backfill persists before SHM mutation, publishes bounded notifications,
|
||||
and delays latest-frame publication until the concurrent pre-update
|
||||
storage snapshot completes.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py tests/test_history_backfill.py`
|
||||
|
||||
Added deterministic coverage for merge preservation, conflict policy,
|
||||
atomic failures, reopened-candidate validation, cache consistency,
|
||||
schema and timestamp invariants, contiguous index regeneration,
|
||||
separate series, stale temp indexing, cancellable file-lock contention,
|
||||
full provider-delta persistence, latest-frame ordering, and bounded
|
||||
notification teardown.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
- deterministic analysis/backfill/storage set: 31 passed
|
||||
- Ruff: passed
|
||||
- `git diff --check`: passed
|
||||
- final adversarial review: no findings
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-07-27T20:47:11Z
|
||||
git_ref: 84a6d47b
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260727T204711Z_84a6d47b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user identified that `backfiller_deep_fixes` introduced persistent
|
||||
`.parquet.lock` files, questioned whether Parquet or Arrow already handled
|
||||
writer coordination, and clarified that piker's `tractor` structured
|
||||
concurrency design already provides single-writer ownership through the
|
||||
persistent `datad` feed task. The user then asked to open the backfiller
|
||||
worktree and resolve the redundant locking.
|
||||
|
||||
## Response summary
|
||||
|
||||
Removed redundant actor-local and filesystem locking while preserving
|
||||
atomic crash-safe Parquet replacement. Replaced the synthetic contention
|
||||
test with regressions for the actual cross-branch failure: writes create
|
||||
no lock sidecars, and legacy lock artifacts are ignored during indexing.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/storage/nativedb.py` - rely on `datad` writer ownership
|
||||
- `tests/test_storage_nativedb.py` - lock-sidecar regressions
|
||||
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not been edited by the human.
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-27T20:47:11Z
|
||||
git_ref: 84a6d47b
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
|
||||
|
||||
Removed actor-local and filesystem writer locks from NativeDB. The
|
||||
persistent `datad` feed task already owns writes for each series, while
|
||||
its timeframe children write distinct files. NativeDB retains validated
|
||||
temporary writes, file and directory synchronization, and atomic target
|
||||
replacement for crash durability. The exact `.parquet` suffix filter is
|
||||
retained so legacy lock sidecars are harmless.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
|
||||
|
||||
Removed the synthetic cross-client lock-contention test. Added regression
|
||||
coverage proving replacement and incremental writes create no lock
|
||||
sidecars. Extended indexing coverage with the legacy
|
||||
`.parquet.lock` artifact that caused upstream `flake_update` to crash.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
- NativeDB and history regression set: 20 passed
|
||||
- Ruff: passed
|
||||
- `git diff --check`: passed
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-27T21:24:54Z
|
||||
git_ref: 689df816
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260727T212454Z_689df816_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user reran the known IB FQME on `backfiller_deep_fixes` and reported
|
||||
that `publish_latest_frame()` still failed because NativeDB required an
|
||||
`index` absent from IB's provider frame. The user called out the missed
|
||||
tabular schema boundary and requested an actual end-to-end test suite
|
||||
rather than storage fakes.
|
||||
|
||||
## Response summary
|
||||
|
||||
Repaired the IB-provider-to-NativeDB boundary by canonicalizing durable
|
||||
fields and mapping reloads into provider-specific SHM buffers. Added a
|
||||
local actor/SHM/Parquet integration regression covering first startup and
|
||||
restart, plus focused conversion and timestamp-coercion tests. Two review
|
||||
rounds found and resolved restart hydration and fractional-time hazards.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/storage/nativedb.py` - canonical storage and SHM field mapping
|
||||
- `piker/tsp/_anal.py` - name-based Polars-to-NumPy conversion
|
||||
- `tests/test_history_backfill.py` - actor/SHM/Parquet integration test
|
||||
- `tests/test_storage_nativedb.py` - schema conversion regressions
|
||||
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not been edited by the human.
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-27T21:24:54Z
|
||||
git_ref: 689df816
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/tsp/_anal.py`
|
||||
|
||||
Normalized index-less provider frames to NativeDB's canonical names,
|
||||
order, and dtypes before merge and publication. Provider-only columns
|
||||
are excluded from durable files, derived indexes are regenerated, and
|
||||
fractional timestamps are rejected before integer conversion. Added a
|
||||
canonical field map for hydrating provider-specific SHM buffers while
|
||||
leaving provider-only fields at their defaults. Changed Polars-to-NumPy
|
||||
conversion to select columns by field name rather than position.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py tests/test_storage_nativedb.py`
|
||||
|
||||
Added deterministic coverage using IB's actual provider dtype and a real
|
||||
local `tractor` root actor, IB-typed shared-memory buffers, NativeDB,
|
||||
Parquet persistence, first-start publication, and restart hydration.
|
||||
Added focused regressions for reordered/extra Polars columns and
|
||||
fractional timestamp rejection.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
- NativeDB and history integration set: 23 passed
|
||||
- Ruff: passed
|
||||
- `git diff --check`: passed
|
||||
- second adversarial review: no findings
|
||||
- no live gateway or network tests run
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-08-11T20:58:26Z
|
||||
git_ref: 06d5ea5d
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Diagnose and fix an IB live-order failure where lazy NVDA contract
|
||||
qualification raised `tractor.trionics.Lagged` after the order reached
|
||||
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
|
||||
searches caused the stale state.
|
||||
|
||||
## Response summary
|
||||
|
||||
Traced the failure to competing consumers of a one-slot Trio broadcast, not
|
||||
the recent SHM change or stale asyncio client state. Replaced broadcast
|
||||
fan-out with one response/status demultiplexer, retained request-ID
|
||||
correlation, handled cancellation and teardown, documented the task/channel
|
||||
ownership model, warned on diagnostic metadata eviction, and added
|
||||
deterministic proxy regressions.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
|
||||
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
|
||||
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the user's live failure diagnosis.
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-11T20:58:26Z
|
||||
git_ref: 06d5ea5d
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user reported that certain IB order entries might have broken and
|
||||
provided a live failure for a first NVDA sell order. EMS delivered the
|
||||
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
|
||||
qualification failed in `MethodProxy._run_method()` with:
|
||||
|
||||
```text
|
||||
tractor.trionics._broadcast.Lagged:
|
||||
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
|
||||
dropped `0` values
|
||||
```
|
||||
|
||||
The user suspected a cancelled symbol-search task might have left stale
|
||||
asyncio state. Cancellation can leave an orphaned method response, but the
|
||||
observed failure was stale Trio broadcast state. `open_client_proxy()`
|
||||
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
|
||||
and an idle method receiver. Status traffic advanced only the relay, so the
|
||||
first method call for an uncached symbol entered an already-lagged receiver.
|
||||
Orders for eagerly cached contracts bypassed this path.
|
||||
|
||||
The immediately preceding sparse-history commit did not modify IB order,
|
||||
symbol, or proxy code; it exposed no direct dependency on this failure.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
|
||||
|
||||
Generated a sole-reader message demultiplexer for each IB client proxy.
|
||||
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
|
||||
status/error traffic, and dispatches `mid`-tagged responses into per-call
|
||||
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
|
||||
late responses to be logged and dropped without skewing later calls. Clean
|
||||
channel EOF is accepted during normal proxy teardown.
|
||||
|
||||
Follow-up review restored the masked reconnect TODO and per-message case
|
||||
rationale, documented the one-reader/many-writer task hierarchy and separate
|
||||
per-proxy channels, and added a warning with request method metadata when the
|
||||
bounded unresolved-request diagnostic table evicts its oldest entry.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
|
||||
|
||||
Generated deterministic regressions for idle status traffic before first
|
||||
qualification, two concurrent calls with reverse-order responses, cancelled
|
||||
calls with late responses followed by a successful call, and graceful
|
||||
channel EOF. Every synchronization wait is timeout-bounded.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||
|
||||
Registered the deterministic IB method-proxy target and source mapping.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
........................ [100%]
|
||||
25 passed, 4 warnings in 3.44s
|
||||
```
|
||||
|
||||
The warnings are existing Tractor/Trio deprecations from the EMS test.
|
||||
Adversarial review drove a clean-EOF guard and bounded polling. Final review
|
||||
found no actionable issues. Live IB order validation remains pending and
|
||||
requires restarting `brokerd.ib`.
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-13T20:24:02Z
|
||||
git_ref: 12b0d66e
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the paired piker broadcast-consumer work after Tractor PR 490
|
||||
stabilization. Stop cached feed callers from mutating or consuming shared
|
||||
quote-stream roots, then review, verify and prepare a commit boundary.
|
||||
|
||||
## Response summary
|
||||
|
||||
Made provider root streams private to the cached `Feed`, gave every caller a
|
||||
local subscribed view, corrected feed cache identity, and made multi-provider
|
||||
relays own synchronized child subscriptions and independent send-channel
|
||||
clones.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
|
||||
subscribed multi-provider relays with independent channel lifetimes.
|
||||
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
|
||||
cache-identity regressions, including sibling survival after provider EOC.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the requested cached-feed ownership item.
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-13T20:24:02Z
|
||||
git_ref: 12b0d66e
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user asked to continue the paired piker broadcast-consumer work after
|
||||
Tractor PR 490 stabilization. The next item was to stop cached feed callers
|
||||
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
|
||||
review, verify and prepare a commit boundary.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||
|
||||
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
|
||||
root streams and every caller, including the first cache owner, receives a
|
||||
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
|
||||
The cache identity now includes normalized full symbol shape and stream policy.
|
||||
|
||||
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
|
||||
of directly consuming a shared stream, and synchronizes relay startup before
|
||||
yielding the merged channel. Each relay owns a cloned send channel so one
|
||||
provider EOC cannot terminate its siblings. The module-level relay returns its
|
||||
live child receiver through `TaskStatus` for typed retention and introspection.
|
||||
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
|
||||
shared by local cache users.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
|
||||
|
||||
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
|
||||
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
|
||||
multi-provider relay fan-out and startup, duplex controls, lexical child
|
||||
closure, sibling survival after provider EOC, cached-root immutability, and
|
||||
full cache-key differentiation.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
... [100%]
|
||||
3 passed in 0.02s
|
||||
|
||||
. [100%]
|
||||
1 passed in 0.01s
|
||||
```
|
||||
|
||||
The second result is the existing EMS broadcast ownership regression. Ruff,
|
||||
E501, compilation and whitespace checks passed. Repeated adversarial review
|
||||
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
|
||||
relay-startup and first-owner asymmetry issues; final review reported no
|
||||
findings.
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-13T23:07:44Z
|
||||
git_ref: ddac3553
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the piker broadcast-consumer audit after the Deribit ownership commit
|
||||
and prepare the next smallest ownership fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
|
||||
context and gave every caller a lexical child from one retained
|
||||
`LinkedTaskChannel` broadcaster root.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
|
||||
ownership.
|
||||
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
|
||||
lifetime regression.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the requested broadcast-consumer audit.
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-13T23:07:44Z
|
||||
git_ref: ddac3553
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user asked to continue the piker broadcast-consumer audit after committing
|
||||
the Deribit ownership fix. The next isolated item was IB's actor-local
|
||||
`_quote_streams` cache, which retained a raw channel owned by its first caller
|
||||
and constructed competing broadcaster roots for later callers.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
||||
|
||||
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
|
||||
made `open_aio_quote_stream()` cache that context by symbol through
|
||||
`maybe_open_context()`. Every caller now receives a lexical child from the
|
||||
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
|
||||
until its final cache user exits. Removed the manual `_quote_streams` registry
|
||||
and teardown mutation.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
|
||||
|
||||
Added a concurrent ownership regression using real Tractor broadcast state.
|
||||
It proves miss and hit callers use the same symbol/context factory, receive the
|
||||
same quote through distinct children, close independently, and leave the owner
|
||||
live after the hit exits.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
. [100%]
|
||||
1 passed in 0.01s
|
||||
```
|
||||
|
||||
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
|
||||
Python compilation and whitespace checks pass. Normal package collection
|
||||
remains blocked by the existing Qt environment mismatch.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-20T16:35:19Z
|
||||
git_ref: 5f631e19
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked to implement the Piker compatibility work caused by
|
||||
tractor's canonical tagged-address migration, run the previously
|
||||
deferred tractor verification, and keep runtime-managed WireGuard
|
||||
bindspaces in a later separate PR.
|
||||
|
||||
## Response summary
|
||||
|
||||
Made Piker normalize legacy and tagged registry inputs before semantic
|
||||
comparison and process-global storage. Updated runtime address typing and
|
||||
test assertions for canonical output, retained legacy caller input, and
|
||||
removed a duplicate pytest log-level option that blocked downstream test
|
||||
collection with current tractor.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/service/_registry.py` - canonical registry normalization and
|
||||
shared address typing.
|
||||
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
|
||||
annotations.
|
||||
- `tests/conftest.py` - canonical fixture comparison and shared pytest
|
||||
option ownership.
|
||||
- `tests/test_services.py` - registry regression and canonical runtime
|
||||
assertions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly requested the downstream implementation and chose
|
||||
to keep WireGuard bindspace lifecycle work in a separate future tractor
|
||||
PR. The agent implemented and tested those directions; no direct manual
|
||||
source edits were observed.
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-20T16:35:19Z
|
||||
git_ref: 5f631e19
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - tractor tagged-address compatibility
|
||||
|
||||
The human requested downstream Piker qualification after tractor PR
|
||||
#505 switched transport address emission to canonical tagged tuples.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
|
||||
|
||||
Normalized legacy and tagged registry addresses through tractor's
|
||||
address codec before Piker stores or compares them. Extended service
|
||||
runtime annotations to cover both tuple shapes, updated runtime tests to
|
||||
compare canonical serialized values, and removed Piker's duplicate
|
||||
registration of tractor's shared `--ll` pytest option.
|
||||
|
||||
Ruff passed. The focused normalization regression and Piker runtime boot
|
||||
passed against the local tractor checkout at `5d92595f`. The wider
|
||||
service module reached an unrelated pre-existing logger-name assertion
|
||||
in `maybe_spawn_daemon()` after both address-focused tests passed.
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-21T04:20:58Z
|
||||
git_ref: b7aca4c0
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human approved refining Piker's logger-name handling after the
|
||||
wider service test exposed a stale parent-logger assertion during the
|
||||
tractor tagged-address compatibility work.
|
||||
|
||||
## Response summary
|
||||
|
||||
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
|
||||
preserving complete requested submodule names. Removed daemon startup
|
||||
assertions that incorrectly required parent logger names and added a
|
||||
regression covering both logger factories and names that the old
|
||||
character-set stripping silently corrupted.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/log.py` - exact relative logger-name derivation.
|
||||
- `piker/service/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/data/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/clearing/_ems.py` - remove redundant logger assertion.
|
||||
- `tests/test_logging.py` - package logger namespace regressions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly approved this follow-up after reviewing the
|
||||
residual Piker service-suite gap. The agent implemented and verified
|
||||
the refinement; no direct manual source edits were observed.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-21T04:20:58Z
|
||||
git_ref: b7aca4c0
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - preserve Piker logger namespaces
|
||||
|
||||
The human approved a follow-up refinement after the full Piker service
|
||||
module exposed a stale logger-name assertion while qualifying tractor
|
||||
PR #505.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
|
||||
|
||||
Replaced character-set `str.lstrip()` calls with exact package-prefix
|
||||
removal, removed assertion-only parent-logger expectations from daemon
|
||||
startup paths, and added regression coverage for both logger factories
|
||||
and prefix-sensitive submodule names.
|
||||
|
||||
Ruff passed after removing newly unused logger assignments. All four
|
||||
logger regressions and all six service integration tests passed against
|
||||
the local tractor PR #505 checkout.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-26T22:26:00Z
|
||||
git_ref: 5f631e19
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Diagnose and fix the hanging Binance USD-M perpetual feed for
|
||||
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
|
||||
API and changelog. Ensure unsupported or silent websocket feeds fail
|
||||
loudly into `datad` and chart consumers instead of blocking startup.
|
||||
|
||||
## Response summary
|
||||
|
||||
Migrates Binance USD-M market data from the retired legacy websocket to
|
||||
the split public and market routes, handles Binance's new aggregate-trade
|
||||
schema, and introduces bounded connection and first-data startup. Direct
|
||||
stream URLs avoid reconnect-time subscription-ACK races, while targeted
|
||||
tests preserve routing and fail-loud behavior.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
|
||||
Binance quote streams.
|
||||
- `piker/brokers/binance/venues.py` - define current USD-M websocket
|
||||
endpoints.
|
||||
- `piker/data/_web_bs.py` - support bounded websocket context startup.
|
||||
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
|
||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
|
||||
retain the generated-output record.
|
||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
|
||||
structured provenance for the code changes.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly required fail-loud behavior rather than merely
|
||||
updating the endpoint, and supplied a direct CLI reproduction proving
|
||||
both `datad.binance` and the chart caller blocked. Those requirements
|
||||
drove the connection and first-data deadlines and their regressions. The
|
||||
changes remain uncommitted for human review.
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T22:26:00Z
|
||||
git_ref: 5f631e19
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user reported that selecting
|
||||
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
|
||||
websocket but never loaded the feed or downstream chart. They asked
|
||||
OpenCode to compare Binance's API changelog with piker's FQME and feed
|
||||
code, diagnose the failure, and solve it.
|
||||
|
||||
The user clarified that future provider failures must raise loudly
|
||||
instead of silently blocking feed and chart startup. They also confirmed
|
||||
the same hang from
|
||||
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Routes USD-M book-ticker and aggregate-trade data through Binance's new
|
||||
`/public` and `/market` websocket paths, merges both sockets into the
|
||||
existing quote stream, accepts the new `st` aggregate-trade field, and
|
||||
requires both stream classes before declaring startup complete. It also
|
||||
removes the misleading missing-provider warning, derives feed mode from
|
||||
resolved pair metadata, and raises `DataUnavailable` for connection or
|
||||
live-data startup timeouts.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
|
||||
|
||||
Defines Binance's current public and market USD-M websocket endpoints
|
||||
and makes the market endpoint the regular futures websocket API root.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Adds an optional connection deadline to `open_autorecon_ws()` so callers
|
||||
can fail context entry when repeated websocket handshakes never become
|
||||
connected.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||
|
||||
Adds regressions for exact Binance stream routing, silent live-data
|
||||
startup, silent websocket connection startup, and the current USD-M
|
||||
aggregate-trade payload schema.
|
||||
|
||||
## Verification output
|
||||
|
||||
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
|
||||
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
|
||||
through piker's merged stream received both required stream classes and
|
||||
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
|
||||
spot.
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
|
||||
timestamp: 2026-08-28T20:03:52Z
|
||||
git_ref: dd676857
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Update the xonsh version used by the repository dev shell before stacking
|
||||
the Binance work, while retaining the dotrc compatibility shim as an
|
||||
immediate workaround for older shells.
|
||||
|
||||
## Response summary
|
||||
|
||||
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
|
||||
The new release supports Python 3.13, handles Git alias completion natively,
|
||||
and works in the updated Nix development shell.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml` - require xonsh `0.24.2` or newer.
|
||||
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
|
||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
|
||||
the triggering prompt, generated configuration summary, and checks.
|
||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
|
||||
structured provenance for the dependency update.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified old xonsh behavior as the root cause, reported the
|
||||
separate dotrc compatibility shim, and directed that the repository update
|
||||
remain a distinct flake-layer change before the Binance stack continues.
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T20:03:52Z
|
||||
git_ref: dd676857
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user traced broken multiline conflict-resolution pastes to the old
|
||||
xonsh release installed by the repository dev shell. A separate dotrc
|
||||
agent added a compatibility shim, and the user directed that xonsh still
|
||||
be updated alongside the flake work before continuing the Binance stack.
|
||||
|
||||
## Generated configuration
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
|
||||
refreshes only xonsh's locked distribution metadata. The selected release
|
||||
supports Python 3.13, includes native Git-config alias completion, and fixes
|
||||
Nix metadata checks.
|
||||
|
||||
## Verification output
|
||||
|
||||
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
|
||||
xonsh from `0.22.4` to `0.24.2`.
|
||||
- Fresh `nix develop` replaces the worktree-local xonsh installation.
|
||||
- `xonsh --version` reports `0.24.2`.
|
||||
- A multi-command xonsh invocation executes Git commands successfully.
|
||||
- `nix flake check --no-build` passes for `x86_64-linux`.
|
||||
- `git diff --check` passes.
|
||||
|
||||
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
|
||||
that its distributions are yanked for an `unpivot` regression.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||
timestamp: 2026-08-29T17:55:42Z
|
||||
git_ref: e09e306a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
|
||||
stack, then add missing deterministic tests and correctness fixes as a
|
||||
separate follow-up with a complete editor-gated commit plan.
|
||||
|
||||
## Response summary
|
||||
|
||||
Hardens Binance feed startup around its supported one-symbol contract,
|
||||
direct websocket routes, current provider schemas, and first-quote
|
||||
readiness. Adds deterministic Binance regressions and separates generic
|
||||
reconnect behavior into its own tested commit boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
|
||||
USD-M stream classes, and declare readiness after either quote class.
|
||||
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
|
||||
and documented book-ticker and aggregate-trade schemas.
|
||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
|
||||
the triggering prompt, generated-code summary, and verification output.
|
||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
|
||||
structured provenance for the follow-up.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user rejected reconstructing already committed Binance work, required
|
||||
preservation through cherry-pick, selected the flake/xonsh stack base, and
|
||||
required the review fixes to remain a separate tested boundary. The user also
|
||||
corrected the workflow to require an automatic `/commit-plan` at every stack
|
||||
review gate. During Tuicr review, the user rejected trade-gated startup,
|
||||
requested a feed-specific initialization error and async-generator TODO,
|
||||
and required generic reconnect changes and tests to land independently.
|
||||
The user then clarified that either L1 or trade data must unblock startup,
|
||||
rather than preferring one stream class or waiting independently for both.
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-29T17:55:42Z
|
||||
git_ref: e09e306a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user asked to land recent branch work in small chronological patch sets,
|
||||
including missing tests. For Binance, the user required preserving the
|
||||
existing committed topic patch through cherry-pick and placing newly found
|
||||
correctness repairs in a separate follow-up boundary above the flake/xonsh
|
||||
stack. The user also required every yielded stack layer to include a complete
|
||||
`/commit-plan` with patch-specific checks.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Restricts Binance feeds to the supported one-symbol contract, renders direct
|
||||
quote routes, separates L1 liveness from first-trade readiness, and derives
|
||||
the contextual trade timeout from Trio's enclosing effective deadline.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Lets structured cancellation bypass reconnect throttling and ensures handled
|
||||
handshake rejections retry before accessing connection-local cancel state.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||
|
||||
Adds deterministic regressions for symbol cardinality, route selection,
|
||||
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
|
||||
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
|
||||
|
||||
## Verification output
|
||||
|
||||
- Targeted Binance tests: `10 passed in 0.26s`.
|
||||
- Ruff passes for all three changed Python files.
|
||||
- `git diff --check` passes.
|
||||
- Independent final review reports no actionable findings.
|
||||
- No live Binance network probe was run.
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||
timestamp: 2026-09-01T02:42:35Z
|
||||
git_ref: e09e306a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Address the Tuicr review by separating generic websocket reconnect
|
||||
changes from Binance-specific work and testing the real reconnect task
|
||||
hierarchy rather than a reimplementation.
|
||||
|
||||
## Response summary
|
||||
|
||||
Preserves structured cancellation and safe first-handshake retries in
|
||||
`_web_bs`. Adds three focused regressions in a dedicated websocket test
|
||||
module and packages them as an independent commit boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py` - restart handled handshake failures and let
|
||||
cancellation bypass the reconnect throttle.
|
||||
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
|
||||
and cancellation task paths with controlled transport boundaries.
|
||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
|
||||
- retain the review prompt, generated-code summary, and verification.
|
||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
|
||||
structured provenance for the generic reconnect boundary.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified that these changes were generic rather than
|
||||
Binance-specific, required a separate commit and test module, requested
|
||||
coverage of the real reconnect hierarchy, and corrected code prose to
|
||||
refer to `trio` consistently. A follow-up review required the production
|
||||
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
|
||||
handler and cite the regression which audits that teardown invariant.
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-09-01T02:42:35Z
|
||||
git_ref: e09e306a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested that generic `_web_bs` reconnect fixes and tests be
|
||||
separated from the Binance backend patch. The review also required tests
|
||||
to exercise the real reconnect task hierarchy, cite the previous faulty
|
||||
branches, and consistently name `trio` in code prose.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Ensures handled handshake failures immediately restart the reconnect
|
||||
loop and lets structured cancellation bypass shielded error throttling.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Adds deterministic scheduler-real regressions for handshake startup
|
||||
timeouts, immediate rejection retries, and cancellation propagation.
|
||||
|
||||
## Verification output
|
||||
|
||||
- Generic and Binance targeted tests: `9 passed in 0.06s`.
|
||||
- Ruff passes for all four changed Python files.
|
||||
- `git diff --check` passes.
|
||||
- No live network probe was run.
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: 4dc920cf-e82e-4c1e-83db-75a0a9efe82b
|
||||
timestamp: 2026-09-01T23:00:43Z
|
||||
git_ref: e91f41a6
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20260901T230043Z_e91f41a6_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Update `wkt/land_flake_update` to Tractor's restored `piker_pin` after
|
||||
reusing the existing runtime compatibility commits. Keep stale
|
||||
historical lock snapshots out of the final dependency state.
|
||||
|
||||
## Response summary
|
||||
|
||||
Updates piker's Python and `msgspec` requirements for current Tractor,
|
||||
then generates one fresh lock at Tractor `piker_pin@0d95a9e6`. The
|
||||
dependency commit completes the compatibility stack already
|
||||
cherry-picked onto the flake branch.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml` - require Python 3.13 and `msgspec>=0.20.0`.
|
||||
- `uv.lock` - resolve Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`,
|
||||
and current transport dependencies.
|
||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.raw.md`
|
||||
- preserve the prompt, generated-change summary, and resolution notes.
|
||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md` - record
|
||||
structured provenance for the dependency update.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified that Tractor compatibility should land with the
|
||||
flake update rather than the Binance patch, supplied four known source
|
||||
commits, required a broader local-history audit for missing ports, and
|
||||
directed a fresh lock from the restored `piker_pin` instead of replaying
|
||||
historical pins.
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-09-01T23:00:43Z
|
||||
git_ref: e91f41a6
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested that the flake-update branch absorb the Tractor
|
||||
runtime compatibility work before the Binance branch lands. They
|
||||
directed reuse of existing compatibility commits, exclusion of stale
|
||||
historical lock pins, and a fresh update to Tractor's restored
|
||||
`piker_pin`.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||
|
||||
Narrows piker support to Python 3.13 and raises the direct `msgspec`
|
||||
requirement to the version family required by current Tractor.
|
||||
|
||||
> `git diff --cached -- uv.lock`
|
||||
|
||||
Relocks against Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`, and the
|
||||
current transport dependency graph while retaining the existing Xonsh
|
||||
update.
|
||||
|
||||
## Resolution notes
|
||||
|
||||
- Historical lock hunks from reused commits were intentionally omitted.
|
||||
- Tractor runtime compatibility remains represented by the preceding
|
||||
individually preserved commits on `wkt/land_flake_update`.
|
||||
- Project checks are rendered in the commit plan for human execution.
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-21T22:44:21Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 507c75c6
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261021T224421Z_507c75c6_piker_pin_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked to sanity-check the current PR94 landing state against
|
||||
the qualified new Tractor pin before deduplicating branch history. After
|
||||
the initial offline checks passed, the human authorized changing the
|
||||
selector to piker_pin, resolving the lock, and verifying the installed
|
||||
dependency without the temporary local-source PYTHONPATH override.
|
||||
The human explicitly authorized network resolution and a scoped update
|
||||
of Tractor/runtime dependencies while preserving unrelated packages.
|
||||
|
||||
## Response summary
|
||||
|
||||
Updated the branch selector and lock to published piker_pin@b64c89f0.
|
||||
Installed the exact locked Tractor revision, verified its installed
|
||||
source metadata, and reran 14 offline logger/startup cases without a
|
||||
local Tractor source override. Preserved the rest of the environment.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: select piker_pin and bound Polars below 2.0.
|
||||
- `uv.lock`: lock b64c89f0 and Polars/runtime 1.44.2.
|
||||
- This provenance pair.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
14 distinct offline cases passed against the installed Tractor update
|
||||
with the existing Polars 1.36 installation. The actor.uid deprecation
|
||||
warning remains. No live provider/order testing occurred.
|
||||
|
||||
The initial resolver selected Polars 2.0 and rejected retaining 1.36
|
||||
for its Windows target. The human then requested the latest pre-2.0
|
||||
release instead. PyPI metadata identified non-yanked stable 1.44.2;
|
||||
adding `<2.0.0` to the existing lower bound and relocking succeeded
|
||||
with both polars and polars-runtime-32 at 1.44.2, without platform edits.
|
||||
The raw partner retains the original, superseded 2.0 generation history.
|
||||
|
||||
The scoped install changed only Tractor. Polars 1.44.2 is locked but
|
||||
not installed or data-path qualified yet. The human's existing staged
|
||||
checkpoint is preserved; the pre-2.0 correction is in the worktree.
|
||||
No history cleanup, staging or commit was performed by this agent.
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-21T22:44:21Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 507c75c6
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Changed the landing branch's Tractor selector from wkt/addr_unpacking
|
||||
to piker_pin and resolved the published branch to
|
||||
b64c89f05a9fc8ac2878551eb5eda36574ebef6a. An offline branch lookup lacked
|
||||
the cached name, so the human explicitly authorized network resolution
|
||||
and a scoped installed-runtime update preserving unrelated packages.
|
||||
|
||||
The resolver advanced Polars and polars-runtime-32 from 1.36.0 to 2.0.0.
|
||||
An attempt to retain both exact 1.36.0 versions was rejected for the
|
||||
resolver's Windows target. The valid lock with the visible Polars bump
|
||||
is retained for review; no platform restriction or source constraint was
|
||||
introduced to bypass the failure.
|
||||
|
||||
Installed only Tractor and its required dependencies with uv pip rather
|
||||
than syncing/pruning the whole environment. The installer reported one
|
||||
package replacement: Tractor a966928d to b64c89f0. Qt distribution versions
|
||||
remained PyQt6 6.10.1, PyQt6-Qt6 6.10.1 and PyQt6-sip 13.10.3.
|
||||
|
||||
Removed PYTHONPATH from verification subprocesses. Piker resolved inside
|
||||
land_flake_update and Tractor resolved from that checkout's py313
|
||||
site-packages. Installed VCS metadata matched the exact locked revision.
|
||||
The temporary LD_LIBRARY_PATH used the binding's bundled Qt libraries
|
||||
and matching native dependencies; no global environment setup changed.
|
||||
|
||||
Verification output:
|
||||
- Logger/non-actor startup: 12 passed, 1 deselected in 0.05s.
|
||||
- test_runtime_boot: 1 passed, 1 warning in 0.04s.
|
||||
- test_daemon_bind_distinct_from_registrar: 1 passed in 0.77s.
|
||||
- Existing warning: actor.uid deprecation; no failure.
|
||||
- git diff --check passed.
|
||||
|
||||
This verifies the installed Tractor pin, not a full sync to all new lock
|
||||
versions. Polars was not upgraded in the environment by the scoped install.
|
||||
No live broker feed, order, branch rewrite, staging or commit was run.
|
||||
|
|
@ -1,67 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-22T21:22:48Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6-astra
|
||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human approved trying the bounded candidate: back out the IB
|
||||
quote-context cache, import adapter-only reset/boundary fixes and
|
||||
the missing EMS subscription fix, preserve source authorship and
|
||||
commit boundaries, and exercise the real Tractor bridge. They
|
||||
required an explicit keep/backout audit and clarification that
|
||||
the evaluated downstream lineage lacked the broadcast fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
Verified identical stable patch IDs for original 4af885d9 and
|
||||
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
|
||||
in the main checkout. Applied source-equivalent adapter backports
|
||||
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
|
||||
source attribution recorded in `ai/ib-recovery-candidate.md`.
|
||||
|
||||
The real-runtime quote test first reproduced the reported fatal
|
||||
TrioTaskExited propagation with caching enabled. The direct-owner
|
||||
backout restores restart and subsequent RPC survival, but a strict
|
||||
callback-cleanliness test remains red. Real bridge history pacing
|
||||
recovery and real broadcaster dark clearing pass. No live broker,
|
||||
VNC, orders or TSDB operations were performed by the agent.
|
||||
|
||||
On 2026-10-23 the human supplied successful live startup/backfill,
|
||||
sequential reattachment and overlapping chart-close/reopen results,
|
||||
plus the still-failing daemon SIGINT teardown. The human accepted
|
||||
deferring teardown from #94, requested loud partial-backport
|
||||
attribution and required downstream rebase rehearsals before landing.
|
||||
The agent removed only the explicit single-null debugger stop,
|
||||
retaining its legacy result. Downstream `976db6c3` instead rewrites
|
||||
null grouping; the local cleanup does not claim that algorithmic fix.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
|
||||
- `piker/clearing/_ems.py`: original EMS subscription patch.
|
||||
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
|
||||
- `tests/test_ib_history.py`: original adapter regressions only.
|
||||
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
|
||||
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
|
||||
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
|
||||
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human required original fixes and authorship, rejected broad
|
||||
speculative integration work, required real concurrency boundaries
|
||||
in tests, and corrected the interpretation of live qualification.
|
||||
They performed the live tests, chose to defer known teardown defects
|
||||
from #94, permitted the minimal debugger cleanup, and requested
|
||||
explicit downstream discrepancies and rebase compatibility checks.
|
||||
The agent applied those directed edits; no direct manual source edits
|
||||
are known in this candidate. The raw partner retains the originally
|
||||
captured output; this structured record includes later contributions.
|
||||
|
|
@ -1,69 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-22T21:22:48Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6-astra
|
||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
|
||||
---
|
||||
|
||||
## Human direction
|
||||
|
||||
> yeah i guess this is our best approach strat for sure.
|
||||
|
||||
> so you're saying these 2 are not the same patch?
|
||||
> please tell me no lol
|
||||
|
||||
> so detail which commits to drop vs. keep please in detail so i can
|
||||
> audit your choices this time..
|
||||
|
||||
> let's try it and consider bailing and re-trying if this leads us into
|
||||
> a further (and deeper) sinkhole, bc as it stands right now this whole
|
||||
> endevour has turned into a complete shitfest XD
|
||||
|
||||
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
|
||||
|
||||
## Unedited diagnostic response
|
||||
|
||||
The real-bridge test reproduced the exact fatal path:
|
||||
`maybe_open_context()` teardown raises `TrioTaskExited` back into
|
||||
the quote consumer.
|
||||
|
||||
With only the quote-cache backout, both quote generations complete
|
||||
and their asyncio tasks finish. Tractor still emits
|
||||
`TrioTaskExited` callback errors, so the strict test remains red.
|
||||
That narrows the result: the backout removes the fatal propagation
|
||||
in this reproducer, but doesn’t make bridge teardown clean.
|
||||
|
||||
## Code output pointers
|
||||
|
||||
> `git diff dc618229 -- piker/brokers/ib/feed.py`
|
||||
|
||||
Back out 635314c7's quote owner and import the original adapter-only
|
||||
deltas from 0846cbd4 and 1da9b760.
|
||||
|
||||
> `git diff dc618229 -- piker/clearing/_ems.py`
|
||||
|
||||
Import 12b0d66e's dedicated dark-clearing quote subscription.
|
||||
|
||||
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
|
||||
|
||||
Replace the reverted mock-cache test with a real-runtime restart
|
||||
reproducer; retain original history/EMS tests and add real bridge
|
||||
history recovery and real broadcaster EMS coverage. Newly added
|
||||
test files remain untracked until the human stages them, so the
|
||||
diff pointers include them only after staging or committing.
|
||||
|
||||
## Verification output
|
||||
|
||||
```text
|
||||
21 passed in 0.53s
|
||||
1 failed, 2 passed in 5.53s
|
||||
13 passed in 1.41s
|
||||
1 passed, 1 warning in 0.04s
|
||||
2 passed in 0.03s
|
||||
```
|
||||
|
||||
The last run includes the original EMS case again and one additional
|
||||
real-broadcaster case: 38 unique passes and one strict callback failure.
|
||||
|
|
@ -1,77 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:08:46Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 7242dcb9
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T160846Z_7242dcb9_websocket_close.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported repeated live Binance perp crashes after correct
|
||||
worktree startup: the real trio-websocket reader attempted a Pong
|
||||
while wsproto was LOCAL_CLOSING. They asked whether an existing
|
||||
downstream websocket fix should be reused, then requested:
|
||||
|
||||
> ok add the fix now, follow by a /commit-plan and then i'll test
|
||||
> before saving it ya?
|
||||
|
||||
They approved a real websocket reader/close regression covering
|
||||
recovery and intentional shutdown, and separately authorized
|
||||
reconciling this worktree's legacy workflow bookkeeping while
|
||||
preserving existing archives.
|
||||
|
||||
## Response summary
|
||||
|
||||
The older `3d83b61f3f7b982e84a4230dba5777f1407c78f1` wrapper
|
||||
suppressed escaped communication failures. The landing replay
|
||||
`97eb8347` of `d50ab6124cd0e1646c82fa4a85d7c13bfff82611`
|
||||
added propagation; it did not restart a reader-context failure.
|
||||
Existing `LocalProtocolError` membership in NoBsWs.recon_errors
|
||||
only protected the receive relay, not reader-nursery context exit.
|
||||
|
||||
Classify these errors around the complete websocket context and
|
||||
retry instead of restoring outer suppression. Place collapse_eg
|
||||
outside that context, invalidate stale connection readiness and
|
||||
socket references, and leave the retry wait cancellable. Preserve
|
||||
unexpected failures and mixed exception groups for propagation.
|
||||
After review, use RuntimeError rather than ValueError for the
|
||||
test-only unexpected-failure sentinel. Its injected exception has
|
||||
no underlying cause. Production recovery logs retain the original
|
||||
traceback and now name the URL; unexpected exceptions use bare
|
||||
raise to preserve identity, existing causes and notes.
|
||||
|
||||
The deterministic seam controls protocol-state ordering in the real
|
||||
client Ping handler. It does not replace the websocket transport,
|
||||
reader, context manager, reconnect task or wsproto exception.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py`: reader-context failure recovery.
|
||||
- `tests/test_websocket.py`: loopback recovery and two shutdown
|
||||
variants; retain unexpected-failure propagation control.
|
||||
- This provenance pair: live report, lineage and test scope.
|
||||
|
||||
## Verification
|
||||
|
||||
Baseline uses new tests against original HEAD websocket code in an
|
||||
isolated exact-tree checkout. The recovery case fails with the
|
||||
reported Pong-in-LOCAL_CLOSING LocalProtocolError.
|
||||
|
||||
Fixed code passes 13 cases across tests/test_websocket.py and
|
||||
tests/test_binance.py. Exact-boundary verification and native-library
|
||||
configuration are retained in the ignored commit-plan package.
|
||||
Live testing remains human-owned and pending.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct manual source edits observed in this follow-up. The human
|
||||
supplied live failures, corrected the earlier daemon/worktree setup,
|
||||
set the narrow landing scope, requested source-fix reuse, and will
|
||||
test before reviewing and committing. Existing commits are retained.
|
||||
The human requested the RuntimeError sentinel and preservation of
|
||||
underlying error context. The raw partner retains the initial output.
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:08:46Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 7242dcb9
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Implemented a focused websocket-context reconnect fix and real
|
||||
loopback reader/close regressions in the Binance landing worktree.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Move exception-group collapse outside the websocket context so
|
||||
reader-nursery exit errors are classified by the reconnect loop.
|
||||
Retry NoBsWs.recon_errors there, clear the stale connected event
|
||||
and socket reference, and honor cancellation through unshielded
|
||||
retry sleep. Retain unexpected-error propagation at the outer API.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Exercise real TCP, handshake, Ping frames, reader dispatch and
|
||||
wsproto Pong validation. A narrow handler gate advances the real
|
||||
protocol state to LOCAL_CLOSING without sending the Close frame,
|
||||
preventing a peer reply from racing the controlled state. Require
|
||||
fresh application data from the replacement connection. Separate
|
||||
variants cancel at the reader gate and at the actual retry wait.
|
||||
Update the existing unexpected-failure control to use ValueError
|
||||
because LocalProtocolError is now intentionally recoverable.
|
||||
|
||||
The isolated original-code/new-test baseline fails with the exact
|
||||
Pong-in-LOCAL_CLOSING error. The fixed isolated tree passes all 13
|
||||
selected Binance/websocket cases. No live provider, broker orders
|
||||
or production daemon operations were performed by the agent.
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:56:38Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: b95e7c79
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T165638Z_b95e7c79_reconnect_logging.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported that the live Binance Pong-in-LOCAL_CLOSING
|
||||
error now logged a retry without crashing. They requested visible
|
||||
re-establishment reporting under `pikerd -l info --pdb`, and asked
|
||||
to check for existing downstream work before adding a fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
The existing `Connection success` INFO message already identifies
|
||||
handshake completion. Feed startup configured piker.data.feed,
|
||||
while shared websocket diagnostics use ._util.log on piker.data.
|
||||
Configure that actual emitter and prefer the daemon's verbosity
|
||||
over a quieter client request for these shared transport tasks.
|
||||
|
||||
Add a readiness INFO message after fixture setup and a first-message
|
||||
INFO report once per connection after actual relay. Include the URL
|
||||
in the receive-timeout reset error. Do not call first raw websocket
|
||||
traffic a parsed quote or proof of restored market-data freshness.
|
||||
|
||||
## Existing-work audit
|
||||
|
||||
Compared `_web_bs.py`, `_util.py`, `piker.log` and daemon logging
|
||||
against the evaluated downstream tip and relevant all-ref histories.
|
||||
`54da2973` contains logging/type cleanup, not this emitter fix.
|
||||
`5adf77a9` / landing equivalents address package-name correctness,
|
||||
but not configuration of the shared data emitter at feed startup.
|
||||
All-ref searches for this actual emitter/readiness/first-relay
|
||||
implementation found no existing commit. The dirty IB history
|
||||
worktree has only an unrelated f-string correction in `_web_bs.py`;
|
||||
the TIPC worktree has no dirty delta in these two feed/websocket
|
||||
files. No downstream work was overwritten or imported wholesale.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/feed.py`: shared transport logger startup/verbosity.
|
||||
- `piker/data/_web_bs.py`: readiness, first-relay and timeout URL.
|
||||
- `tests/test_logging.py`: actual emitter/console startup regression.
|
||||
- `tests/test_websocket.py`: recovery milestones on real loopback.
|
||||
- This prompt provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
The new logging regression against original b95e7c79 source fails:
|
||||
the actual websocket emitter remains at WARNING and rejects INFO.
|
||||
It passes with the logger wiring fix. Provider lookup is stopped
|
||||
before network work, while logger factories and console emission
|
||||
are real. Global logger state is restored after the check.
|
||||
|
||||
18 cases pass across tests/test_logging.py, tests/test_websocket.py
|
||||
and tests/test_binance.py. git diff --check passes. Live verification
|
||||
after restarting the daemon with these edits remains human-owned.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct manual source edits observed. The human supplied live
|
||||
recovery evidence, selected INFO verbosity, requested downstream
|
||||
reuse inspection, and retains live-test and commit ownership.
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:56:38Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: b95e7c79
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Confirmed the existing INFO handshake-success log was hidden by
|
||||
logger wiring: websocket code emits on piker.data, while feed
|
||||
startup configured piker.data.feed. The baseline regression leaves
|
||||
the emitter at WARNING for an INFO daemon and ERROR client.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||
|
||||
Configure the actual shared ._util.log namespace, with daemon
|
||||
verbosity taking priority over client verbosity for actor-wide
|
||||
transport diagnostics.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Keep the existing handshake-success message. Add INFO readiness
|
||||
after fixture setup and first-message relay once per connection.
|
||||
Include the URL in the receive-timeout reset diagnostic.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_logging.py tests/test_websocket.py`
|
||||
|
||||
Prove production startup enables real console INFO on the websocket
|
||||
emitter without contacting a provider. Extend the real loopback
|
||||
recovery test to require readiness for both connections and one
|
||||
first-message report for the connection delivering fresh data.
|
||||
|
||||
All 18 targeted logging/Binance/websocket cases passed. No live
|
||||
provider/daemon operations were performed by the agent. First relay
|
||||
is raw websocket traffic; parsed market-data recovery remains a
|
||||
separate live observation.
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T18:08:56Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 032580bf
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T180856Z_032580bf_timeout_phase.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported repeated readiness without first-message relay,
|
||||
specifically after first attempting an invalid .usdm. FQME and then
|
||||
the corrected .usdtm. one. They requested distinct log messages for
|
||||
receive starvation versus local backpressure.
|
||||
|
||||
## Response summary
|
||||
|
||||
Add phase-specific WARNING messages for each elapsed relay-loop
|
||||
deadline, with URL, timeout duration, accumulated count and reset
|
||||
threshold. Leave the existing shared receive/send deadline and
|
||||
counter/reset semantics intact. Do not claim that connected/ready
|
||||
sockets alone establish parsed quote recovery.
|
||||
|
||||
Read-only invalid-venue tracing found the usd substring check in
|
||||
Binance get_mkt_info() accepts USDM while retaining it in MktPair.
|
||||
Client.bars() later uses that market identity against canonical
|
||||
USDTM keys. Backend stream startup precedes history startup and
|
||||
sampler consumption. A relationship to the corrected-feed stall
|
||||
still needs runtime evidence; neither venue handling nor task
|
||||
lifetime behavior was changed by this diagnostic patch.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py`: track and report the timed-out operation.
|
||||
- `tests/test_websocket.py`: real silent-source and blocked-sink cases.
|
||||
- This prompt provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
15 selected Binance/websocket cases passed. The new cases exercise
|
||||
real TCP, websocket handshake and relay behavior, replacing only
|
||||
log observation. A zero-capacity relay deliberately forces the
|
||||
local-send case; the source-silent case never receives a payload.
|
||||
git diff --check passed. No live provider/daemon operations run.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct source edits observed. The human supplied the critical
|
||||
invalid-symbol-then-corrected-symbol trigger and requested the
|
||||
phase distinction; live testing and task acceptance remain theirs.
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
---
|
||||
created_at: 2026-10-28T18:08:56Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 032580bf
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Added distinct timeout diagnostics for websocket receive starvation
|
||||
versus local relay backpressure, preserving the existing timeout,
|
||||
counter and reconnect threshold policy.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Track whether proxy_msgs() is awaiting ws.get_message() or
|
||||
snd.send(). Report that phase, URL, configured deadline, timeout
|
||||
count and reset threshold for every timeout, before the existing
|
||||
threshold/reset diagnostic. Capture the configured deadline once
|
||||
per operation so the report matches the deadline that elapsed.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Two real loopback variants use a silent server versus one payload
|
||||
and an unconsumed zero-capacity Trio relay. Require the appropriate
|
||||
phase and URL on the actual first timeout. All 15 selected
|
||||
Binance/websocket cases pass. No live provider was contacted.
|
||||
|
||||
Read-only tracing found a permissive invalid-venue path: get_mkt_info
|
||||
accepts USDM via its usd substring check and preserves that venue
|
||||
in MktPair, whereas Client.bars() indexes canonical USDTM pair keys.
|
||||
This identifies a potential late failed-startup path, not proof of
|
||||
the corrected-chart stall or of leaked producer ownership.
|
||||
|
|
@ -1,29 +1,8 @@
|
|||
[network]
|
||||
# This Tractor pin supports one transport per actor, not mixed TCP/UDS.
|
||||
# Use the single TCP endpoint shown in each list for this version.
|
||||
# Mixed-transport configs need a future Tractor pin with that support.
|
||||
pikerd = [
|
||||
# TCP localhost loopback
|
||||
'/ip4/127.0.0.1/tcp/6116',
|
||||
|
||||
# UDS alternative: replace TCP and select the UDS transport.
|
||||
# '/unix/run/user/1000/piker/pikerd.sock',
|
||||
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||
# '/uds/6116', # TODO std uds socket file
|
||||
]
|
||||
chart = [
|
||||
'/ip4/127.0.0.1/tcp/3003',
|
||||
# '/unix/run/user/1000/piker/chart.sock',
|
||||
]
|
||||
# the service-actor registry endpoint;
|
||||
# other pikerd trees contact this to discover
|
||||
# actors.
|
||||
# XXX if absent, pikerd binds the registry
|
||||
# on its own tpt_bind_addrs.
|
||||
# regd = ['/ip4/127.0.0.1/tcp/6116']
|
||||
|
||||
# chart = [
|
||||
# '/ip4/127.0.0.1/tcp/3333',
|
||||
# '/unix/run/user/1000/piker/chart@3333.sock',
|
||||
# ]
|
||||
|
||||
|
||||
[ui]
|
||||
|
|
|
|||
|
|
@ -2,11 +2,11 @@
|
|||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1784497964,
|
||||
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
|
||||
"lastModified": 1765779637,
|
||||
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
|
||||
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
|
|||
23
flake.nix
23
flake.nix
|
|
@ -22,14 +22,12 @@
|
|||
|
||||
# do store-path extractions
|
||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||
zlibStorePath = lib.getLib pkgs.zlib;
|
||||
# ?TODO? can remove below since manual linking not needed?
|
||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||
|
||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||
# a major-version-pinned-`cpython`
|
||||
cpython = "python313";
|
||||
python = pkgs.${cpython};
|
||||
pypkgs = pkgs."${cpython}Packages";
|
||||
in
|
||||
{
|
||||
|
|
@ -48,7 +46,7 @@
|
|||
qt6.qtbase
|
||||
|
||||
uv
|
||||
python
|
||||
python313 # ?TODO^ how to set from `cpython` above?
|
||||
pypkgs.pyqt6
|
||||
pypkgs.pyqt6-sip
|
||||
pypkgs.qtpy
|
||||
|
|
@ -72,7 +70,6 @@
|
|||
|
||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
|
|
@ -89,31 +86,15 @@
|
|||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||
|
||||
# ------ uv ------
|
||||
# ignore any venv inherited from an outer checkout
|
||||
unset VIRTUAL_ENV
|
||||
# - always use the ./py313/ venv-subdir
|
||||
export UV_PROJECT_ENVIRONMENT="py313"
|
||||
|
||||
# Rebuild after a flake update changes the Nix Python closure.
|
||||
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
|
||||
nix_python="$(readlink -f "${python}/bin/python3")"
|
||||
if test -e "$venv_python" \
|
||||
&& test "$(readlink -f "$venv_python")" != "$nix_python"
|
||||
then
|
||||
uv venv \
|
||||
--clear \
|
||||
--python "$nix_python" \
|
||||
"$UV_PROJECT_ENVIRONMENT"
|
||||
fi
|
||||
|
||||
# sync project-env with all extras
|
||||
uv sync --dev --all-extras --no-group lint
|
||||
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
|
||||
|
||||
# ------ TIPS ------
|
||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||
# run the `nix develop` cmd with,
|
||||
# >> nix develop -c xonsh
|
||||
# >> nix develop -c uv run xonsh
|
||||
'';
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -324,13 +324,10 @@ def load_ledger(
|
|||
ldir: Path = (
|
||||
dirpath
|
||||
or
|
||||
config.get_conf_dir() / 'accounting' / 'ledgers'
|
||||
config._config_dir / 'accounting' / 'ledgers'
|
||||
)
|
||||
if not ldir.is_dir():
|
||||
ldir.mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
ldir.mkdir()
|
||||
|
||||
fname = f'trades_{brokername}_{acctid}.toml'
|
||||
fpath: Path = ldir / fname
|
||||
|
|
|
|||
|
|
@ -785,16 +785,9 @@ def load_account(
|
|||
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
||||
fn: str = f'account.{brokername}.{acctid}.toml'
|
||||
|
||||
dirpath: Path = (
|
||||
dirpath
|
||||
or
|
||||
(config.get_conf_dir() / 'accounting')
|
||||
)
|
||||
dirpath: Path = dirpath or (config._config_dir / 'accounting')
|
||||
if not dirpath.is_dir():
|
||||
dirpath.mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
dirpath.mkdir()
|
||||
|
||||
conf, path = config.load(
|
||||
path=dirpath / fn,
|
||||
|
|
|
|||
|
|
@ -143,15 +143,12 @@ def sync(
|
|||
# (what the EMS normally does internall) B)
|
||||
open_brokerd_dialog(
|
||||
brokermod,
|
||||
portal,
|
||||
exec_mode=(
|
||||
'paper'
|
||||
if account == 'paper'
|
||||
else 'live'
|
||||
),
|
||||
# use our own ad-hoc-spawned actor,
|
||||
# do NOT (spawn and) use the
|
||||
# `brokerd.<broker>` service daemon!
|
||||
portal=portal,
|
||||
loglevel=loglevel,
|
||||
) as (
|
||||
brokerd_stream,
|
||||
|
|
|
|||
|
|
@ -25,8 +25,10 @@ from contextlib import (
|
|||
)
|
||||
from types import ModuleType
|
||||
from typing import (
|
||||
TYPE_CHECKING,
|
||||
AsyncContextManager,
|
||||
)
|
||||
import exceptiongroup as eg
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
|
|
@ -38,20 +40,27 @@ from piker.log import (
|
|||
from . import _util
|
||||
from . import get_brokermod
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..data import _FeedsBus
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# `brokerd`-actor-always-enabled mods.
|
||||
# `brokerd` enabled modules
|
||||
# TODO: move this def to the `.data` subpkg..
|
||||
# NOTE: keeping this list as small as possible is part of our caps-sec
|
||||
# model and should be treated with utmost care! In particular NO
|
||||
# `piker.data.*` feed mods should be enabled in this (live,
|
||||
# credentialed) trading actor; all data-feed serving is the
|
||||
# domain of the `datad.<broker>` sibling daemon, see
|
||||
# `piker.data._daemon._datad_service_mods`.
|
||||
_brokerd_service_mods: list[str] = [
|
||||
# model and should be treated with utmost care!
|
||||
_data_mods: str = [
|
||||
'piker.brokers.core',
|
||||
'piker.brokers.data',
|
||||
'piker.brokers._daemon',
|
||||
'piker.data',
|
||||
'piker.data.feed',
|
||||
'piker.data._sampling'
|
||||
]
|
||||
|
||||
|
||||
# TODO: we should rename the daemon to datad prolly once we split up
|
||||
# broker vs. data tasks into separate actors?
|
||||
@tractor.context
|
||||
async def _setup_persistent_brokerd(
|
||||
ctx: tractor.Context,
|
||||
|
|
@ -60,15 +69,9 @@ async def _setup_persistent_brokerd(
|
|||
|
||||
) -> None:
|
||||
'''
|
||||
Trading-only daemon (lifetime) fixture: console logging
|
||||
setup and a pinned-open context for service mgmt.
|
||||
|
||||
All data-feed-bus state now lives in the (data-feed-only)
|
||||
`datad.<brokername>` sibling daemon, see
|
||||
`piker.data._daemon._setup_persistent_datad()`; this
|
||||
actor hosts only the backend's `open_trade_dialog()`
|
||||
(live order-control) ep-task(s) which manage their own
|
||||
task trees per `tractor.Context`.
|
||||
Allocate a actor-wide service nursery in ``brokerd``
|
||||
such that feeds can be run in the background persistently by
|
||||
the broker backend as needed.
|
||||
|
||||
'''
|
||||
# NOTE: we only need to setup logging once (and only) here
|
||||
|
|
@ -77,19 +80,54 @@ async def _setup_persistent_brokerd(
|
|||
# any further (level) configuration on their own B)
|
||||
actor: tractor.Actor = tractor.current_actor()
|
||||
tll: str = actor.loglevel
|
||||
get_console_log(
|
||||
log = get_console_log(
|
||||
level=loglevel or tll,
|
||||
name=f'{_util.subsys}.{brokername}',
|
||||
with_tractor_log=bool(tll),
|
||||
)
|
||||
assert log.name == _util.subsys
|
||||
|
||||
# further, set the log level on any broker broker specific
|
||||
# logger instance.
|
||||
|
||||
from piker.data import feed
|
||||
assert not feed._bus
|
||||
|
||||
# allocate a nursery to the bus for spawning background
|
||||
# tasks to service client IPC requests, normally
|
||||
# `tractor.Context` connections to explicitly required
|
||||
# `brokerd` endpoints such as:
|
||||
# - `stream_quotes()`,
|
||||
# - `manage_history()`,
|
||||
# - `allocate_persistent_feed()`,
|
||||
# - `open_symbol_search()`
|
||||
# NOTE: see ep invocation details inside `.data.feed`.
|
||||
try:
|
||||
async with (
|
||||
# tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_nursery
|
||||
):
|
||||
bus: _FeedsBus = feed.get_feed_bus(
|
||||
brokername,
|
||||
service_nursery,
|
||||
)
|
||||
assert bus is feed._bus
|
||||
|
||||
# unblock caller
|
||||
await ctx.started()
|
||||
|
||||
# we pin this task to keep the daemon active until the
|
||||
# we pin this task to keep the feeds manager active until the
|
||||
# parent actor decides to tear it down
|
||||
await trio.sleep_forever()
|
||||
|
||||
except eg.ExceptionGroup:
|
||||
# TODO: likely some underlying `brokerd` IPC connection
|
||||
# broke so here we handle a respawn and re-connect attempt!
|
||||
# This likely should pair with development of the OCO task
|
||||
# nusery in dev over @ `tractor` B)
|
||||
# https://github.com/goodboy/tractor/pull/363
|
||||
raise
|
||||
|
||||
|
||||
def broker_init(
|
||||
brokername: str,
|
||||
|
|
@ -109,10 +147,8 @@ def broker_init(
|
|||
|
||||
This includes:
|
||||
- load the appropriate <brokername>.py pkg module,
|
||||
- reads any declared `_brokerd_mods: list[str]` (falling
|
||||
back to the full `__enable_modules__` set for
|
||||
not-yet-split backends) which will be passed to
|
||||
`tractor.ActorNursery.start_actor(enable_modules=)`
|
||||
- reads any declared `__enable_modules__: listr[str]` which will be
|
||||
passed to `tractor.ActorNursery.start_actor(enabled_modules=<this>)`
|
||||
at actor start time,
|
||||
- deliver a references to the daemon lifetime fixture, which
|
||||
for now is always the `_setup_persistent_brokerd()` context defined
|
||||
|
|
@ -146,15 +182,9 @@ def broker_init(
|
|||
modpath,
|
||||
]
|
||||
for submodname in getattr(
|
||||
brokermod,
|
||||
'_brokerd_mods',
|
||||
# fallback for (flat, less mature) backends which
|
||||
# don't yet declare a daemon-kind mod split.
|
||||
getattr(
|
||||
brokermod,
|
||||
'__enable_modules__',
|
||||
[],
|
||||
),
|
||||
):
|
||||
subpath: str = f'{modpath}.{submodname}'
|
||||
enabled.append(subpath)
|
||||
|
|
@ -182,22 +212,6 @@ async def spawn_brokerd(
|
|||
f'backend: {brokername!r}'
|
||||
)
|
||||
|
||||
# fail fast on (data-only) backends which don't offer
|
||||
# ANY live order-control eps; the caller should instead
|
||||
# be using paper-mode (and thus never spawning us)!
|
||||
from ..data.validate import get_eps
|
||||
brokerd_eps: dict = get_eps(
|
||||
get_brokermod(brokername),
|
||||
'brokerd',
|
||||
)
|
||||
if not brokerd_eps:
|
||||
raise RuntimeError(
|
||||
f'Backend {brokername!r} offers NO `brokerd` '
|
||||
f'(live order-control) eps!?\n'
|
||||
f'It is likely a datad-only provider, use '
|
||||
f'paper-mode for clearing instead.\n'
|
||||
)
|
||||
|
||||
(
|
||||
brokermode,
|
||||
tractor_kwargs,
|
||||
|
|
@ -219,11 +233,7 @@ async def spawn_brokerd(
|
|||
dname: str = tractor_kwargs.pop('name') # f'brokerd.{brokername}'
|
||||
portal = await Services.actor_n.start_actor(
|
||||
dname,
|
||||
enable_modules=list(dict.fromkeys(
|
||||
_brokerd_service_mods
|
||||
+
|
||||
tractor_kwargs.pop('enable_modules')
|
||||
)),
|
||||
enable_modules=_data_mods + tractor_kwargs.pop('enable_modules'),
|
||||
debug_mode=Services.debug_mode,
|
||||
**tractor_kwargs
|
||||
)
|
||||
|
|
|
|||
|
|
@ -20,17 +20,10 @@ Handy cross-broker utils.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
# from functools import partial
|
||||
from typing import (
|
||||
Type,
|
||||
)
|
||||
|
||||
import json
|
||||
import httpx
|
||||
import logging
|
||||
from msgspec import Struct
|
||||
from tractor._exceptions import (
|
||||
reg_err_types,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
colorize_json,
|
||||
|
|
@ -104,19 +97,6 @@ class DataThrottle(BrokerError):
|
|||
'''
|
||||
# TODO: add in throttle metrics/feedback
|
||||
|
||||
class SchemaMismatch(BrokerError):
|
||||
'''
|
||||
Market `Pair` fields mismatch, likely due to provider API update.
|
||||
|
||||
'''
|
||||
|
||||
# auto-register all `BrokerError` subtypes for
|
||||
# tractor IPC exc-marshalling.
|
||||
reg_err_types([
|
||||
BrokerError,
|
||||
*BrokerError.__subclasses__(),
|
||||
])
|
||||
|
||||
|
||||
def resproc(
|
||||
resp: httpx.Response,
|
||||
|
|
@ -143,45 +123,3 @@ def resproc(
|
|||
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
||||
|
||||
return msg if return_json else resp
|
||||
|
||||
|
||||
def get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type: Type[Struct],
|
||||
fields_data: dict,
|
||||
provider_name: str,
|
||||
api_url: str|None = None,
|
||||
) -> Struct:
|
||||
'''
|
||||
Boilerplate helper around assset-`Pair` field schema mismatches,
|
||||
normally due to provider API updates.
|
||||
|
||||
'''
|
||||
try:
|
||||
pair: Struct = pair_type(**fields_data)
|
||||
return pair
|
||||
except TypeError as err:
|
||||
|
||||
from tractor.devx.pformat import ppfmt
|
||||
repr_data: str = ppfmt(fields_data)
|
||||
report: str = (
|
||||
f'Field mismatch we need to codify!\n'
|
||||
f'\n'
|
||||
f'{pair_type!r}({repr_data})'
|
||||
f'\n'
|
||||
f'^^^ {err.args[0]!r} ^^^\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly {provider_name!r} "
|
||||
f"changed their symbology schema..\n"
|
||||
)
|
||||
if (
|
||||
api_url
|
||||
or
|
||||
(api_url := pair_type._api_url)
|
||||
):
|
||||
report += (
|
||||
f'\n'
|
||||
f'Check out their API docs here:\n'
|
||||
f'{api_url}\n'
|
||||
)
|
||||
|
||||
raise SchemaMismatch(report) from err
|
||||
|
|
|
|||
|
|
@ -52,25 +52,9 @@ __all__ = [
|
|||
]
|
||||
|
||||
|
||||
# per-daemon-kind (sub)mod groups: declares which of our
|
||||
# submods host the eps run by each daemon-actor kind as
|
||||
# defined by `piker.data.validate._eps`.
|
||||
# NOTE: `get_mkt_info` and `open_symbol_search` both live
|
||||
# in `.feed` for this backend (no `symbols.py`).
|
||||
_brokerd_mods: list[str] = [
|
||||
'api',
|
||||
'broker',
|
||||
]
|
||||
|
||||
_datad_mods: list[str] = [
|
||||
# `brokerd` modules
|
||||
__enable_modules__: list[str] = [
|
||||
'api',
|
||||
'feed',
|
||||
'broker',
|
||||
]
|
||||
|
||||
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||
_brokerd_mods
|
||||
+
|
||||
_datad_mods
|
||||
))
|
||||
|
|
|
|||
|
|
@ -49,9 +49,6 @@ from piker import config
|
|||
from piker.clearing._messages import (
|
||||
Order,
|
||||
)
|
||||
from piker.brokers._util import (
|
||||
get_or_raise_on_pair_schema_mismatch,
|
||||
)
|
||||
from piker.accounting import (
|
||||
Asset,
|
||||
digits_to_dec,
|
||||
|
|
@ -373,12 +370,20 @@ class Client:
|
|||
item['filters'] = filters
|
||||
|
||||
pair_type: Type = PAIRTYPES[venue]
|
||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type=pair_type,
|
||||
fields_data=item,
|
||||
provider_name='binance',
|
||||
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
||||
try:
|
||||
pair: Pair = pair_type(**item)
|
||||
except Exception as e:
|
||||
e.add_note(
|
||||
f'\n'
|
||||
f'New or removed field we need to codify!\n'
|
||||
f'pair-type: {pair_type!r}\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||
f'Check out their API docs here:\n'
|
||||
f'\n'
|
||||
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||
)
|
||||
raise
|
||||
pair_table[pair.symbol.upper()] = pair
|
||||
|
||||
# update an additional top-level-cross-venue-table
|
||||
|
|
@ -604,11 +609,7 @@ class Client:
|
|||
start_time = binance_timestamp(start_dt)
|
||||
end_time = binance_timestamp(end_dt)
|
||||
|
||||
import tractor
|
||||
with tractor.devx.maybe_open_crash_handler():
|
||||
bs_pair: Pair = self._pairs[
|
||||
mkt.bs_fqme.upper()
|
||||
]
|
||||
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
||||
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||
bars = await self._api(
|
||||
|
|
|
|||
|
|
@ -22,14 +22,18 @@ from __future__ import annotations
|
|||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
aclosing,
|
||||
AsyncExitStack,
|
||||
)
|
||||
from datetime import datetime
|
||||
from functools import (
|
||||
partial,
|
||||
)
|
||||
import itertools
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
AsyncGenerator,
|
||||
Callable,
|
||||
Generator,
|
||||
)
|
||||
import time
|
||||
|
||||
|
|
@ -44,7 +48,6 @@ import tractor
|
|||
from piker.brokers import (
|
||||
open_cached_client,
|
||||
NoData,
|
||||
SymbolNotFound,
|
||||
)
|
||||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
|
|
@ -56,10 +59,7 @@ from piker.accounting import (
|
|||
unpack_fqme,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.data.validate import (
|
||||
FeedInit,
|
||||
FeedInitializationError,
|
||||
)
|
||||
from piker.data.validate import FeedInit
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
|
|
@ -76,15 +76,13 @@ from .venues import (
|
|||
Pair,
|
||||
FutesPair,
|
||||
get_api_eps,
|
||||
_futes_market_ws,
|
||||
_futes_public_ws,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class L1(Struct):
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Individual-Symbol-Book-Ticker-Streams
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
|
||||
|
||||
update_id: int
|
||||
sym: str
|
||||
|
|
@ -109,15 +107,14 @@ class AggTrade(Struct, frozen=True):
|
|||
T: int # Trade time
|
||||
m: bool # Is the buyer the market maker?
|
||||
M: bool|None = None # Ignore
|
||||
nq: float|None = None # Normal quantity without RPI-order trades
|
||||
# ^XXX: https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
st: int|None = None # 1 for USD-M, 2 for COIN-M
|
||||
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
||||
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
|
||||
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
|
||||
) -> AsyncGenerator[tuple[str, dict], None]:
|
||||
) -> AsyncGenerator[NoBsWs, dict]:
|
||||
|
||||
# TODO: match syntax here!
|
||||
msg: dict[str, Any]
|
||||
|
|
@ -206,6 +203,27 @@ async def stream_messages(
|
|||
yield 'trade', piker_quote
|
||||
|
||||
|
||||
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
||||
'''
|
||||
Create a request subscription packet dict.
|
||||
|
||||
- spot:
|
||||
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
||||
|
||||
- futes:
|
||||
https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||
|
||||
'''
|
||||
return {
|
||||
'method': 'SUBSCRIBE',
|
||||
'params': [
|
||||
f'{pair.lower()}@{sub_name}'
|
||||
for pair in pairs
|
||||
],
|
||||
'id': uid
|
||||
}
|
||||
|
||||
|
||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||
# code?!
|
||||
@acm
|
||||
|
|
@ -283,10 +301,6 @@ async def get_mkt_info(
|
|||
|
||||
# uppercase since kraken bs_mktid is always upper
|
||||
if 'binance' not in fqme.lower():
|
||||
log.warning(
|
||||
f'Missing `.<provider>` part in fqme ??\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
fqme += '.binance'
|
||||
|
||||
mkt_mode: str = ''
|
||||
|
|
@ -301,12 +315,6 @@ async def get_mkt_info(
|
|||
venue: str = venue.upper()
|
||||
venue_lower: str = venue.lower()
|
||||
|
||||
if not venue:
|
||||
raise SymbolNotFound(
|
||||
f'Invalid or missing .<venue> part in fqme?\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
|
||||
# XXX TODO: we should change the usdtm_futes name to just
|
||||
# usdm_futes (dropping the tether part) since it turns out that
|
||||
# there are indeed USD-tokens OTHER THEN tether being used as
|
||||
|
|
@ -398,153 +406,54 @@ async def get_mkt_info(
|
|||
return mkt, pair
|
||||
|
||||
|
||||
def get_quote_routes(
|
||||
mkt_mode: str,
|
||||
symbols: list[str],
|
||||
) -> tuple[str, ...]:
|
||||
'''
|
||||
Render websocket URLs for each required Binance stream class.
|
||||
|
||||
'''
|
||||
if len(symbols) != 1:
|
||||
raise ValueError(
|
||||
'Binance quote routes require exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
symbol: str = symbols[0]
|
||||
sub_names: tuple[str, str] = ('bookTicker', 'aggTrade')
|
||||
if mkt_mode == 'usdtm_futes':
|
||||
return tuple(
|
||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||
for wss_url, sub_name in (
|
||||
(_futes_public_ws, sub_names[0]),
|
||||
(_futes_market_ws, sub_names[1]),
|
||||
)
|
||||
)
|
||||
|
||||
wss_url: str = get_api_eps(mkt_mode)[1]
|
||||
return tuple(
|
||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||
for sub_name in sub_names
|
||||
)
|
||||
|
||||
|
||||
async def relay_messages(
|
||||
ws: NoBsWs,
|
||||
send_chan: trio.MemorySendChannel,
|
||||
) -> None:
|
||||
'''
|
||||
Relay one Binance websocket into a merged quote channel.
|
||||
|
||||
'''
|
||||
async with (
|
||||
send_chan,
|
||||
aclosing(stream_messages(ws)) as msg_gen,
|
||||
):
|
||||
async for msg in msg_gen:
|
||||
await send_chan.send(msg)
|
||||
|
||||
|
||||
# TODO: Encapsulate ``relay_messages()`` and ``open_quote_stream()``
|
||||
# inside ``stream_quotes()`` so async-generator closure is owned by
|
||||
# the backend feed task.
|
||||
@acm
|
||||
async def open_quote_stream(
|
||||
mkt_mode: str,
|
||||
async def subscribe(
|
||||
ws: NoBsWs,
|
||||
symbols: list[str],
|
||||
connect_timeout: float = 16,
|
||||
) -> AsyncGenerator[trio.MemoryReceiveChannel, None]:
|
||||
'''
|
||||
Open and merge the websocket routes required by a market.
|
||||
|
||||
'''
|
||||
quote_routes: tuple[str, ...] = get_quote_routes(
|
||||
mkt_mode=mkt_mode,
|
||||
symbols=symbols,
|
||||
)
|
||||
# defined once at import time to keep a global state B)
|
||||
iter_subids: Generator[int, None, None] = itertools.count(),
|
||||
|
||||
send, recv = trio.open_memory_channel(616)
|
||||
async with AsyncExitStack() as stack:
|
||||
websockets: list[NoBsWs] = []
|
||||
try:
|
||||
for wss_url in quote_routes:
|
||||
ws: NoBsWs = await stack.enter_async_context(
|
||||
open_autorecon_ws(
|
||||
url=wss_url,
|
||||
connect_timeout=connect_timeout,
|
||||
)
|
||||
)
|
||||
websockets.append(ws)
|
||||
|
||||
except (
|
||||
TimeoutError,
|
||||
*NoBsWs.recon_errors,
|
||||
) as err:
|
||||
routes_str: str = '\n'.join(quote_routes)
|
||||
raise DataUnavailable(
|
||||
f'Binance websocket connection failed for '
|
||||
f'{symbols!r}:\n'
|
||||
f'{routes_str}'
|
||||
) from err
|
||||
|
||||
async with trio.open_nursery() as tn:
|
||||
for ws in websockets:
|
||||
tn.start_soon(
|
||||
relay_messages,
|
||||
ws,
|
||||
send.clone(),
|
||||
)
|
||||
|
||||
await send.aclose()
|
||||
async with recv:
|
||||
try:
|
||||
yield recv
|
||||
finally:
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
|
||||
async def wait_for_live_quotes(
|
||||
msg_stream: AsyncGenerator | trio.MemoryReceiveChannel,
|
||||
symbols: list[str],
|
||||
startup_timeout: float = 16,
|
||||
) -> dict:
|
||||
'''
|
||||
Require L1 or trade data before declaring a feed live.
|
||||
|
||||
'''
|
||||
if len(symbols) != 1:
|
||||
raise ValueError(
|
||||
'Binance quote readiness requires exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
symbol: str = symbols[0]
|
||||
msg_iter = aiter(msg_stream)
|
||||
|
||||
try:
|
||||
with trio.fail_after(startup_timeout):
|
||||
while True:
|
||||
typ, quote = await anext(msg_iter)
|
||||
if (
|
||||
typ in {'l1', 'trade'}
|
||||
and
|
||||
quote['symbol'] == symbol
|
||||
):
|
||||
return quote
|
||||
# setup subs
|
||||
|
||||
except trio.TooSlowError as err:
|
||||
raise DataUnavailable(
|
||||
f'Binance quote startup timed out after '
|
||||
f'{startup_timeout}s; '
|
||||
f'no live data received for {symbol}'
|
||||
) from err
|
||||
subid: int = next(iter_subids)
|
||||
|
||||
except StopAsyncIteration as err:
|
||||
raise DataUnavailable(
|
||||
f'Binance quote streams closed during startup; '
|
||||
f'no live data received for {symbol}'
|
||||
) from err
|
||||
# trade data (aka L1)
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
|
||||
l1_sub = make_sub(symbols, 'bookTicker', subid)
|
||||
await ws.send_msg(l1_sub)
|
||||
|
||||
# aggregate (each order clear by taker **not** by maker)
|
||||
# trades data:
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
|
||||
agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
|
||||
await ws.send_msg(agg_trades_sub)
|
||||
|
||||
# might get ack from ws server, or maybe some
|
||||
# other msg still in transit..
|
||||
res = await ws.recv_msg()
|
||||
subid: str|None = res.get('id')
|
||||
if subid:
|
||||
assert res['id'] == subid
|
||||
|
||||
yield
|
||||
|
||||
subs = []
|
||||
for sym in symbols:
|
||||
subs.append("{sym}@aggTrade")
|
||||
subs.append("{sym}@bookTicker")
|
||||
|
||||
# unsub from all pairs on teardown
|
||||
if ws.connected():
|
||||
await ws.send_msg({
|
||||
"method": "UNSUBSCRIBE",
|
||||
"params": subs,
|
||||
"id": subid,
|
||||
})
|
||||
|
||||
# XXX: do we need to ack the unsub?
|
||||
# await ws.recv_msg()
|
||||
|
||||
|
||||
async def stream_quotes(
|
||||
|
|
@ -558,44 +467,47 @@ async def stream_quotes(
|
|||
|
||||
) -> None:
|
||||
|
||||
if len(symbols) != 1:
|
||||
# ``data.feed.allocate_persistent_feed()`` currently starts
|
||||
# one backend task per symbol. Binance multiplexing remains
|
||||
# deferred until that caller contract can represent
|
||||
# multi-symbol startup.
|
||||
raise FeedInitializationError(
|
||||
'Binance feeds require exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
async with (
|
||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||
send_chan as send_chan,
|
||||
open_cached_client('binance') as client,
|
||||
):
|
||||
init_msgs: list[FeedInit] = []
|
||||
for sym in symbols:
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(symbols[0])
|
||||
mkt_mode: str = (
|
||||
'usdtm_futes'
|
||||
if isinstance(pair, FutesPair)
|
||||
else 'spot'
|
||||
)
|
||||
mkt, pair = await get_mkt_info(sym)
|
||||
|
||||
# build out init msgs according to latest spec
|
||||
init_msgs: list[FeedInit] = [
|
||||
FeedInit(mkt_info=mkt),
|
||||
]
|
||||
|
||||
ws_symbols: list[str] = [mkt.bs_mktid]
|
||||
async with open_quote_stream(
|
||||
mkt_mode=mkt_mode,
|
||||
symbols=ws_symbols,
|
||||
) as msg_stream:
|
||||
quote: dict = await wait_for_live_quotes(
|
||||
msg_stream=msg_stream,
|
||||
symbols=ws_symbols,
|
||||
init_msgs.append(
|
||||
FeedInit(mkt_info=mkt)
|
||||
)
|
||||
|
||||
wss_url: str = get_api_eps(client.mkt_mode)[1] # 2nd elem is wss url
|
||||
|
||||
# TODO: for sanity, but remove eventually Xp
|
||||
if 'future' in mkt.type_key:
|
||||
assert 'fstream' in wss_url
|
||||
|
||||
async with (
|
||||
open_autorecon_ws(
|
||||
url=wss_url,
|
||||
fixture=partial(
|
||||
subscribe,
|
||||
symbols=[mkt.bs_mktid],
|
||||
),
|
||||
) as ws,
|
||||
|
||||
# avoid stream-gen closure from breaking trio..
|
||||
aclosing(stream_messages(ws)) as msg_gen,
|
||||
):
|
||||
# log.info('WAITING ON FIRST LIVE QUOTE..')
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
# pull a first quote and deliver
|
||||
while typ != 'trade':
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
task_status.started((init_msgs, quote))
|
||||
|
||||
# signal to caller feed is ready for consumption
|
||||
|
|
@ -611,7 +523,7 @@ async def stream_quotes(
|
|||
topic: str = mkt.bs_fqme
|
||||
|
||||
# start streaming
|
||||
async for typ, quote in msg_stream:
|
||||
async for typ, quote in msg_gen:
|
||||
# period = time.time() - last
|
||||
# hz = 1/period if period else float('inf')
|
||||
# if hz > 60:
|
||||
|
|
|
|||
|
|
@ -20,7 +20,6 @@ Per market data-type definitions and schemas types.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
from typing import (
|
||||
ClassVar,
|
||||
Literal,
|
||||
)
|
||||
from decimal import Decimal
|
||||
|
|
@ -42,10 +41,7 @@ _spot_ws: str = 'wss://stream.binance.com/ws'
|
|||
# or this one? ..
|
||||
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
||||
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/
|
||||
# websocket-market-streams/Important-WebSocket-Change-Notice
|
||||
_futes_public_ws: str = f'wss://fstream.{_domain}/public/ws'
|
||||
_futes_market_ws: str = f'wss://fstream.{_domain}/market/ws'
|
||||
# https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
||||
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
||||
|
||||
|
|
@ -82,7 +78,7 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
|
|||
),
|
||||
'usdtm_futes': (
|
||||
_futes_url,
|
||||
_futes_market_ws,
|
||||
_futes_ws,
|
||||
),
|
||||
}[venue]
|
||||
|
||||
|
|
@ -207,8 +203,6 @@ class FutesPair(Pair):
|
|||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||
ns_path: str = 'piker.brokers.binance:FutesPair'
|
||||
|
||||
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
||||
|
||||
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
||||
# processing..
|
||||
@property
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ from piker.log import (
|
|||
get_logger,
|
||||
)
|
||||
from ..service import (
|
||||
maybe_spawn_datad,
|
||||
maybe_spawn_brokerd,
|
||||
maybe_open_pikerd,
|
||||
)
|
||||
from ..brokers import (
|
||||
|
|
@ -187,7 +187,7 @@ def brokercheck(config, broker):
|
|||
|
||||
'''
|
||||
async def bcheck_main():
|
||||
async with maybe_spawn_datad(broker) as portal:
|
||||
async with maybe_spawn_brokerd(broker) as portal:
|
||||
await portal.run(run_test, broker)
|
||||
await portal.cancel_actor()
|
||||
|
||||
|
|
@ -317,7 +317,7 @@ def record(config, rate, name, dhost, filename):
|
|||
return
|
||||
|
||||
async def main(tries):
|
||||
async with maybe_spawn_datad(
|
||||
async with maybe_spawn_brokerd(
|
||||
tries=tries, loglevel=loglevel
|
||||
) as portal:
|
||||
# run app "main"
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ import trio
|
|||
|
||||
from piker.log import get_logger
|
||||
from . import get_brokermod
|
||||
from ..service import maybe_spawn_datad
|
||||
from ..service import maybe_spawn_brokerd
|
||||
from . import open_cached_client
|
||||
from ..accounting import MktPair
|
||||
|
||||
|
|
@ -172,7 +172,7 @@ async def symbol_search(
|
|||
# await tractor.devx._debug.maybe_init_greenback()
|
||||
# tractor.pause_from_sync()
|
||||
|
||||
async with maybe_spawn_datad(
|
||||
async with maybe_spawn_brokerd(
|
||||
mod.name,
|
||||
infect_asyncio=getattr(
|
||||
mod,
|
||||
|
|
|
|||
|
|
@ -425,7 +425,7 @@ class DataFeed:
|
|||
async def stream_to_file(
|
||||
watchlist_name: str,
|
||||
filename: str,
|
||||
portal: tractor.Portal,
|
||||
portal: tractor._portal.Portal,
|
||||
tickers: List[str],
|
||||
brokermod: ModuleType,
|
||||
rate: int,
|
||||
|
|
|
|||
|
|
@ -47,25 +47,13 @@ __all__ = [
|
|||
]
|
||||
|
||||
|
||||
# per-daemon-kind (sub)mod groups: declares which of our
|
||||
# submods host the eps run by each daemon-actor kind as
|
||||
# defined by `piker.data.validate._eps`.
|
||||
# NOTE: datad-only backend (no `broker.py` yet)!
|
||||
_brokerd_mods: list[str] = []
|
||||
|
||||
_datad_mods: list[str] = [
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = [
|
||||
'api',
|
||||
'feed',
|
||||
# 'broker',
|
||||
]
|
||||
|
||||
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||
_brokerd_mods
|
||||
+
|
||||
_datad_mods
|
||||
))
|
||||
|
||||
# passed to ``tractor.ActorNursery.start_actor()``
|
||||
_spawn_kwargs = {
|
||||
'infect_asyncio': True,
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ from contextlib import (
|
|||
asynccontextmanager as acm,
|
||||
)
|
||||
from datetime import datetime
|
||||
from functools import partial
|
||||
import time
|
||||
from typing import (
|
||||
Any,
|
||||
|
|
@ -37,7 +38,7 @@ from rapidfuzz import process as fuzzy
|
|||
import numpy as np
|
||||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
maybe_open_context,
|
||||
maybe_open_context
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor import to_asyncio
|
||||
|
|
@ -523,12 +524,13 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
|
|||
|
||||
|
||||
async def aio_price_feed_relay(
|
||||
chan: to_asyncio.LinkedTaskChannel,
|
||||
fh: FeedHandler,
|
||||
instrument: Symbol,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
) -> None:
|
||||
async def _trade(data: dict, receipt_timestamp):
|
||||
chan.send_nowait(('trade', {
|
||||
to_trio.send_nowait(('trade', {
|
||||
'symbol': cb_sym_to_deribit_inst(
|
||||
str_to_cb_sym(data.symbol)).lower(),
|
||||
'last': data,
|
||||
|
|
@ -538,7 +540,7 @@ async def aio_price_feed_relay(
|
|||
}))
|
||||
|
||||
async def _l1(data: dict, receipt_timestamp):
|
||||
chan.send_nowait(('l1', {
|
||||
to_trio.send_nowait(('l1', {
|
||||
'symbol': cb_sym_to_deribit_inst(
|
||||
str_to_cb_sym(data.symbol)).lower(),
|
||||
'ticks': [
|
||||
|
|
@ -568,7 +570,7 @@ async def aio_price_feed_relay(
|
|||
install_signal_handlers=False)
|
||||
|
||||
# sync with trio
|
||||
chan.started_nowait(None)
|
||||
to_trio.send_nowait(None)
|
||||
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
|
|
@ -579,9 +581,11 @@ async def open_price_feed(
|
|||
) -> trio.abc.ReceiveStream:
|
||||
async with maybe_open_feed_handler() as fh:
|
||||
async with to_asyncio.open_channel_from(
|
||||
partial(
|
||||
aio_price_feed_relay,
|
||||
fh=fh,
|
||||
instrument=instrument,
|
||||
fh,
|
||||
instrument
|
||||
)
|
||||
) as (chan, first):
|
||||
yield chan
|
||||
|
||||
|
|
@ -607,9 +611,10 @@ async def maybe_open_price_feed(
|
|||
|
||||
|
||||
async def aio_order_feed_relay(
|
||||
chan: to_asyncio.LinkedTaskChannel,
|
||||
fh: FeedHandler,
|
||||
instrument: Symbol,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
) -> None:
|
||||
async def _fill(data: dict, receipt_timestamp):
|
||||
breakpoint()
|
||||
|
|
@ -632,7 +637,7 @@ async def aio_order_feed_relay(
|
|||
install_signal_handlers=False)
|
||||
|
||||
# sync with trio
|
||||
chan.started_nowait(None)
|
||||
to_trio.send_nowait(None)
|
||||
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
|
|
@ -643,9 +648,11 @@ async def open_order_feed(
|
|||
) -> trio.abc.ReceiveStream:
|
||||
async with maybe_open_feed_handler() as fh:
|
||||
async with to_asyncio.open_channel_from(
|
||||
partial(
|
||||
aio_order_feed_relay,
|
||||
fh=fh,
|
||||
instrument=instrument,
|
||||
fh,
|
||||
instrument
|
||||
)
|
||||
) as (chan, first):
|
||||
yield chan
|
||||
|
||||
|
|
|
|||
|
|
@ -65,18 +65,17 @@ _brokerd_mods: list[str] = [
|
|||
]
|
||||
|
||||
_datad_mods: list[str] = [
|
||||
'api',
|
||||
'feed',
|
||||
'symbols',
|
||||
]
|
||||
|
||||
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||
__enable_modules__: list[str] = (
|
||||
_brokerd_mods
|
||||
+
|
||||
_datad_mods
|
||||
))
|
||||
)
|
||||
|
||||
# passed to ``tractor.ActorNursery.start_actor()``
|
||||
_spawn_kwargs = {
|
||||
|
|
|
|||
|
|
@ -958,26 +958,8 @@ class Client:
|
|||
|
||||
else:
|
||||
raise
|
||||
# XXX: ambiguous/unqualifiable contracts are
|
||||
# returned as `None` entries by
|
||||
# `qualifyContractsAsync()` (which also logs an
|
||||
# "Ambiguous contract" warning listing the
|
||||
# possible matches) so filter them BEFORE any
|
||||
# attr access B)
|
||||
contracts: list[Contract] = [
|
||||
tract
|
||||
for tract in contracts
|
||||
if tract is not None
|
||||
]
|
||||
if not contracts:
|
||||
raise ValueError(
|
||||
f'No (unambiguous) contract could be '
|
||||
f'qualified for {con!r}\n'
|
||||
f'\n'
|
||||
f'If a stonk, you likely need a (more) '
|
||||
f'venue-qualified fqme,\n'
|
||||
f"eg. 'gld.arca.ib' instead of 'gld.ib'\n"
|
||||
)
|
||||
raise ValueError(f"No contract could be found {con}")
|
||||
|
||||
# pack all contracts into cache
|
||||
for tract in contracts:
|
||||
|
|
@ -1155,16 +1137,10 @@ class Client:
|
|||
try:
|
||||
con: Contract = self._contracts[symbol]
|
||||
except KeyError:
|
||||
# require that the mkt's contract has been previously
|
||||
# qualified and cached (see `.symbols.get_mkt_info()`
|
||||
# which is run for any feed-init OR lazily by the
|
||||
# order-request handler in `.broker`) - ensures we
|
||||
# aren't making orders against unknown contracts.
|
||||
raise RuntimeError(
|
||||
f'Can not order {symbol!r}, '
|
||||
f'no qualified contract cached!?\n'
|
||||
f'_contracts: {list(self._contracts)!r}\n'
|
||||
)
|
||||
# require that the symbol has been previously cached by
|
||||
# a data feed request - ensure we aren't making orders
|
||||
# against non-known prices.
|
||||
raise RuntimeError("Can not order {symbol}, no live feed?")
|
||||
|
||||
try:
|
||||
trade = self.ib.placeOrder(
|
||||
|
|
@ -1355,21 +1331,6 @@ async def load_aio_clients(
|
|||
ib = None
|
||||
client = None
|
||||
|
||||
# XXX: post (brokerd vs. datad)-split BOTH per-broker
|
||||
# daemons connect to the same API gw/tws endpoint(s); to
|
||||
# avoid `clientId` collisions (and the long conn-timeout
|
||||
# retry cycle they cause) we offset the data-daemon's
|
||||
# default id-range to be disjoint from `brokerd.ib`'s
|
||||
# (which also retries with `client_id + i` increments).
|
||||
if client_id == 6116: # the default from above
|
||||
aname: str = tractor.current_actor().name
|
||||
if 'datad' in aname:
|
||||
client_id += 16
|
||||
# ad-hoc (test/cli) actors get their own range to
|
||||
# avoid clashing with any live daemon-tree's conns.
|
||||
elif 'brokerd' not in aname:
|
||||
client_id += 32
|
||||
|
||||
# attempt to get connection info from config; if no .toml entry
|
||||
# exists, we try to load from a default localhost connection.
|
||||
localhost = '127.0.0.1'
|
||||
|
|
@ -1623,72 +1584,6 @@ class MethodProxy:
|
|||
self.event_table = event_table
|
||||
self._aio_ns = asyncio_ns
|
||||
|
||||
# request-id counter for correlating each method
|
||||
# call to its (eventual) response; necessary since
|
||||
# the (single, shared) chan has NO other ordering
|
||||
# guarantee whenever a caller task is CANCELLED
|
||||
# (eg. by a search-req timeout) after sending its
|
||||
# request but before consuming the response: the
|
||||
# "orphaned" response otherwise gets mis-delivered
|
||||
# to the next caller causing an off-by-one skew of
|
||||
# every result thereafter!
|
||||
self._mids = itertools.count()
|
||||
self._pending: dict[
|
||||
int,
|
||||
tuple[trio.Event, list[dict]],
|
||||
] = {}
|
||||
self._request_methods: dict[int, str|None] = {}
|
||||
|
||||
def _deliver_method_response(
|
||||
self,
|
||||
msg: dict,
|
||||
|
||||
) -> bool:
|
||||
'''
|
||||
Deliver one correlated response to its waiting caller.
|
||||
|
||||
'''
|
||||
mid: int = msg['mid']
|
||||
meth: str|None = self._request_methods.pop(mid, None)
|
||||
pending = self._pending.get(mid)
|
||||
if pending is None:
|
||||
log.warning(
|
||||
f'Dropping stale method-resp,\n'
|
||||
f'mid: {mid}\n'
|
||||
f'meth: {meth!r}\n'
|
||||
f'(caller prolly got cancelled '
|
||||
f'before its resp?)\n'
|
||||
)
|
||||
return False
|
||||
|
||||
event, slot = pending
|
||||
slot.append(msg)
|
||||
event.set()
|
||||
return True
|
||||
|
||||
def _track_request_method(
|
||||
self,
|
||||
mid: int,
|
||||
meth: str|None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Retain bounded request metadata for stale-response logs.
|
||||
|
||||
'''
|
||||
self._request_methods[mid] = meth
|
||||
if len(self._request_methods) > 256:
|
||||
oldest_mid: int = next(iter(self._request_methods))
|
||||
oldest_meth: str|None = self._request_methods.pop(
|
||||
oldest_mid
|
||||
)
|
||||
log.warning(
|
||||
f'Evicting unresolved method-request metadata,\n'
|
||||
f'mid: {oldest_mid}\n'
|
||||
f'meth: {oldest_meth!r}\n'
|
||||
f'pending metadata limit: 256\n'
|
||||
)
|
||||
|
||||
async def _run_method(
|
||||
self,
|
||||
*,
|
||||
|
|
@ -1701,30 +1596,38 @@ class MethodProxy:
|
|||
``tractor.to_asyncio`` layer.
|
||||
|
||||
'''
|
||||
mid: int = next(self._mids)
|
||||
event = trio.Event()
|
||||
slot: list[dict] = []
|
||||
self._pending[mid] = event, slot
|
||||
self._track_request_method(mid, meth)
|
||||
try:
|
||||
await self.chan.send((meth, kwargs, mid))
|
||||
await event.wait()
|
||||
finally:
|
||||
self._pending.pop(mid, None)
|
||||
chan = self.chan
|
||||
await chan.send((meth, kwargs))
|
||||
|
||||
msg: dict = slot[0]
|
||||
match msg:
|
||||
# OUR method-call response B)
|
||||
case {'result': res}:
|
||||
while not chan.closed():
|
||||
# send through method + ``kwargs: dict`` as pair
|
||||
msg = await chan.receive()
|
||||
|
||||
# TODO: implement reconnect functionality like
|
||||
# in our `.data._web_bs.NoBsWs`
|
||||
# try:
|
||||
# msg = await chan.receive()
|
||||
# except ConnectionError:
|
||||
# self.reset()
|
||||
|
||||
# print(f'NEXT MSG: {msg}')
|
||||
|
||||
# TODO: py3.10 ``match:`` syntax B)
|
||||
if 'result' in msg:
|
||||
res = msg.get('result')
|
||||
return res
|
||||
|
||||
case {'exception': err}:
|
||||
elif 'exception' in msg:
|
||||
err = msg.get('exception')
|
||||
raise err
|
||||
|
||||
case _:
|
||||
raise RuntimeError(
|
||||
f'Invalid method response for mid={mid}: {msg!r}'
|
||||
)
|
||||
elif 'error' in msg:
|
||||
etype, emsg = msg
|
||||
log.warning(f'IB error relay: {emsg}')
|
||||
continue
|
||||
|
||||
else:
|
||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||
|
||||
def status_event(
|
||||
self,
|
||||
|
|
@ -1734,7 +1637,7 @@ class MethodProxy:
|
|||
|
||||
ev = self.event_table.get(pattern)
|
||||
|
||||
if ev is None or ev.is_set():
|
||||
if not ev or ev.is_set():
|
||||
# print(f'inserting new data reset event item')
|
||||
ev = self.event_table[pattern] = trio.Event()
|
||||
|
||||
|
|
@ -1777,31 +1680,22 @@ async def open_aio_client_method_relay(
|
|||
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
||||
break
|
||||
|
||||
case (meth_name, kwargs, mid):
|
||||
case (meth_name, kwargs):
|
||||
meth_name, kwargs = msg
|
||||
meth = getattr(client, meth_name)
|
||||
|
||||
try:
|
||||
resp = await meth(**kwargs)
|
||||
# echo the msg back tagged with the
|
||||
# req-id for caller-side correlation.
|
||||
chan.send_nowait({
|
||||
'mid': mid,
|
||||
'result': resp,
|
||||
})
|
||||
# echo the msg back
|
||||
chan.send_nowait({'result': resp})
|
||||
|
||||
# XXX: relay ALL (non-cancel) errors to the
|
||||
# `trio`-side caller (which re-raises in the
|
||||
# `MethodProxy._run_method()` frame) instead
|
||||
# of crashing this relay task and thus the
|
||||
# whole proxy/dialog; critical post the
|
||||
# (datad|brokerd)-split where eg. contract
|
||||
# qualification failures are expected to be
|
||||
# caught per-request by order/warmup code!
|
||||
except Exception as err:
|
||||
chan.send_nowait({
|
||||
'mid': mid,
|
||||
'exception': err,
|
||||
})
|
||||
except (
|
||||
RequestError,
|
||||
|
||||
# TODO: relay all errors to trio?
|
||||
# BaseException,
|
||||
) as err:
|
||||
chan.send_nowait({'exception': err})
|
||||
|
||||
case {'error': content}:
|
||||
chan.send_nowait({'exception': content})
|
||||
|
|
@ -1810,73 +1704,6 @@ async def open_aio_client_method_relay(
|
|||
raise ValueError(f'Unhandled msg {msg}')
|
||||
|
||||
|
||||
async def relay_client_proxy_messages(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
proxy: MethodProxy,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Route one proxy's asyncio-client msgs from a single reader.
|
||||
|
||||
Each `open_client_proxy()` allocates its own linked channel and
|
||||
starts this task in its relay nursery. Any number of Trio caller
|
||||
tasks may concurrently write tagged requests through
|
||||
`MethodProxy._run_method()` and wait on their per-`mid` slots.
|
||||
This task alone reads the channel and dispatches each response to
|
||||
its waiting caller.
|
||||
|
||||
Separate proxy contexts own separate channels and reader tasks;
|
||||
no cross-proxy response broadcast is intended. IB status events
|
||||
still reach each proxy through its own `Client.inline_errors()`
|
||||
handler.
|
||||
|
||||
'''
|
||||
try:
|
||||
while not chan.closed():
|
||||
# TODO: implement reconnect functionality like
|
||||
# in our `.data._web_bs.NoBsWs`
|
||||
# try:
|
||||
msg = await chan.receive()
|
||||
# except ConnectionError:
|
||||
# proxy.reset()
|
||||
|
||||
match msg:
|
||||
# Correlated response from a method call.
|
||||
case {'mid': _}:
|
||||
proxy._deliver_method_response(msg)
|
||||
|
||||
# routine (api-farm conn) status events
|
||||
# relayed inline by `Client.inline_errors()`;
|
||||
# not a response to a method call so just
|
||||
# log at info and keep waiting.
|
||||
case ('event', status_msg):
|
||||
reason = status_msg['reason']
|
||||
event = proxy.event_table.pop(reason, None)
|
||||
if (
|
||||
event is not None
|
||||
and
|
||||
event.statistics().tasks_waiting
|
||||
):
|
||||
log.info(f'Relaying ib status message: {msg}')
|
||||
event.set()
|
||||
|
||||
# Inline API error with no request correlation.
|
||||
case ('error', emsg):
|
||||
log.warning(f'IB error relay: {emsg}')
|
||||
|
||||
# Out-of-band client error with no method `mid`.
|
||||
case {'exception': err}:
|
||||
log.error(
|
||||
f'Uncorrelated IB client error: {err!r}'
|
||||
)
|
||||
|
||||
# Preserve visibility into unexpected relay traffic.
|
||||
case _:
|
||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||
except trio.EndOfChannel:
|
||||
log.info('IB client proxy relay closed')
|
||||
|
||||
|
||||
@acm
|
||||
async def open_client_proxy(
|
||||
client: Client,
|
||||
|
|
@ -1913,11 +1740,28 @@ async def open_client_proxy(
|
|||
continue
|
||||
setattr(proxy, name, partial(proxy._run_method, meth=name))
|
||||
|
||||
relay_tn.start_soon(
|
||||
relay_client_proxy_messages,
|
||||
chan,
|
||||
proxy,
|
||||
)
|
||||
async def relay_events():
|
||||
|
||||
async with chan.subscribe() as msg_stream:
|
||||
|
||||
async for msg in msg_stream:
|
||||
if 'event' not in msg:
|
||||
continue
|
||||
|
||||
# if 'event' in msg:
|
||||
# wake up any system event waiters.
|
||||
etype, status_msg = msg
|
||||
reason = status_msg['reason']
|
||||
|
||||
ev = proxy.event_table.pop(reason, None)
|
||||
|
||||
if ev and ev.statistics().tasks_waiting:
|
||||
log.info(f'Relaying ib status message: {msg}')
|
||||
ev.set()
|
||||
|
||||
continue
|
||||
|
||||
relay_tn.start_soon(relay_events)
|
||||
|
||||
yield proxy
|
||||
|
||||
|
|
|
|||
|
|
@ -89,9 +89,8 @@ from .api import (
|
|||
MethodProxy,
|
||||
)
|
||||
from .symbols import (
|
||||
cache_contract,
|
||||
con2fqme,
|
||||
get_mkt_info,
|
||||
# get_mkt_info,
|
||||
)
|
||||
from .ledger import (
|
||||
norm_trade_records,
|
||||
|
|
@ -139,7 +138,6 @@ async def handle_order_requests(
|
|||
ems_order_stream: tractor.MsgStream,
|
||||
accounts_def: dict[str, str],
|
||||
flows: OrderDialogs,
|
||||
proxies: dict[str, MethodProxy],
|
||||
|
||||
) -> None:
|
||||
|
||||
|
|
@ -182,58 +180,6 @@ async def handle_order_requests(
|
|||
if action in {'buy', 'sell'}:
|
||||
# validate
|
||||
order = BrokerdOrder(**request_msg)
|
||||
fqme: str = order.symbol
|
||||
|
||||
# XXX: lazily qualify and cache the contract for
|
||||
# this mkt since, post the (datad|brokerd)-split,
|
||||
# this trading-only actor will NOT have had its
|
||||
# (api) `Client._contracts` pre-warmed by any
|
||||
# in-proc feed setup (which now runs in the
|
||||
# `datad.ib` sibling); we run the SAME symbology
|
||||
# resolution ep the feed-side uses so the cache
|
||||
# key (`MktPair.bs_fqme`) matches what
|
||||
# `Client.submit_limit()` looks up.
|
||||
# NOTE: normally a no-op since
|
||||
# `open_trade_dialog()` eagerly pre-qualifies all
|
||||
# already-open pp/order mkts at startup; this
|
||||
# only fires for a brand-new (to this daemon)
|
||||
# mkt's first order.
|
||||
if fqme not in client._contracts:
|
||||
proxy: MethodProxy = proxies[account]
|
||||
try:
|
||||
(
|
||||
mkt,
|
||||
details,
|
||||
) = await get_mkt_info(
|
||||
fqme,
|
||||
proxy=proxy,
|
||||
)
|
||||
# XXX: explicit write since the lifo-cache
|
||||
# may deliver a hit (body skipped) when
|
||||
# another acct/client already qualified
|
||||
# this fqme.
|
||||
cache_contract(
|
||||
client,
|
||||
mkt,
|
||||
details.contract,
|
||||
)
|
||||
except Exception as err:
|
||||
log.exception(
|
||||
f'Failed to qualify contract for\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
await ems_order_stream.send(
|
||||
BrokerdError(
|
||||
oid=oid,
|
||||
symbol=fqme,
|
||||
reason=(
|
||||
f'No contract could be '
|
||||
f'qualified for {fqme!r}:\n'
|
||||
f'{err!r}'
|
||||
),
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# XXX: by default 0 tells ``ib_async`` methods that
|
||||
# there is no existing order so ask the client to create
|
||||
|
|
@ -245,34 +191,15 @@ async def handle_order_requests(
|
|||
reqid = int(reqid)
|
||||
|
||||
# call our client api to submit the order
|
||||
# NOTE: guard with order-error relay (vs. crashing
|
||||
# this dialog and thus ALL order ctl for the
|
||||
# backend) so one bad submission can't take down
|
||||
# the daemon's clearing loop.
|
||||
try:
|
||||
reqid = client.submit_limit(
|
||||
oid=order.oid,
|
||||
symbol=fqme,
|
||||
symbol=order.symbol,
|
||||
price=order.price,
|
||||
action=order.action,
|
||||
size=order.size,
|
||||
account=acct_number,
|
||||
reqid=reqid,
|
||||
)
|
||||
except Exception as err:
|
||||
log.exception(
|
||||
f'Order submission failed for\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
await ems_order_stream.send(
|
||||
BrokerdError(
|
||||
oid=oid,
|
||||
symbol=fqme,
|
||||
reason=f'Submission error: {err!r}',
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
str_reqid: str = str(reqid)
|
||||
if reqid is None:
|
||||
err_msg = BrokerdError(
|
||||
|
|
@ -304,21 +231,20 @@ async def handle_order_requests(
|
|||
|
||||
async def recv_trade_updates(
|
||||
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
client: Client,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Receive and relay order control and positioning
|
||||
related events from `ib_async`, pack as tuples and
|
||||
push over mem-chan to our trio relay task for
|
||||
processing and relay to EMS.
|
||||
Receive and relay order control and positioning related events
|
||||
from `ib_async`, pack as tuples and push over mem-chan to our
|
||||
trio relay task for processing and relay to EMS.
|
||||
|
||||
'''
|
||||
client.inline_errors(chan)
|
||||
client.inline_errors(to_trio)
|
||||
|
||||
# sync with trio task
|
||||
chan.started_nowait(client.ib)
|
||||
to_trio.send_nowait(client.ib)
|
||||
|
||||
def push_tradesies(
|
||||
eventkit_obj,
|
||||
|
|
@ -356,7 +282,7 @@ async def recv_trade_updates(
|
|||
|
||||
try:
|
||||
# emit event name + relevant ibis internal objects
|
||||
chan.send_nowait((event_name, emit))
|
||||
to_trio.send_nowait((event_name, emit))
|
||||
except trio.BrokenResourceError:
|
||||
log.exception(f'Disconnected from {eventkit_obj} updates')
|
||||
eventkit_obj.disconnect(push_tradesies)
|
||||
|
|
@ -621,14 +547,10 @@ async def open_trade_dialog(
|
|||
|
||||
) -> AsyncIterator[dict[str, Any]]:
|
||||
|
||||
# XXX: ONLY adjust the level of this (sub)mod's logger;
|
||||
# attaching a (stderr) handler (via `get_console_log()`)
|
||||
# here would DOUBLE-print every record since the daemon
|
||||
# fixture (`.._daemon._setup_persistent_brokerd()`)
|
||||
# already enables the console handler on the parent
|
||||
# subsys logger which all our records propagate to!
|
||||
if loglevel:
|
||||
log.setLevel(loglevel.upper())
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
|
||||
# task local msg dialog tracking
|
||||
flows = OrderDialogs()
|
||||
|
|
@ -865,64 +787,6 @@ async def open_trade_dialog(
|
|||
for msg in order_msgs:
|
||||
await ems_stream.send(msg)
|
||||
|
||||
# XXX: eagerly pre-qualify (and cache) the
|
||||
# contracts for all already-open pps and
|
||||
# orders so that (live) order submission
|
||||
# NEVER pays a first-request qualification
|
||||
# delay; any brand-new mkt is still lazily
|
||||
# qualified by `handle_order_requests()` on
|
||||
# its first submission. NOTE: since this
|
||||
# runs BEFORE the order handler task is
|
||||
# even started, no order can clear until
|
||||
# the warmup completes (early reqs just
|
||||
# buffer in the ems IPC stream) B)
|
||||
warmup_fqmes: set[str] = {
|
||||
msg.symbol
|
||||
for msg in all_positions
|
||||
}
|
||||
warmup_fqmes.update(
|
||||
msg.req.symbol
|
||||
for msg in order_msgs
|
||||
)
|
||||
unique_clients: set[Client] = set(
|
||||
aioclients.values()
|
||||
)
|
||||
if (
|
||||
warmup_fqmes
|
||||
and
|
||||
proxies
|
||||
):
|
||||
a_proxy: MethodProxy = next(
|
||||
iter(proxies.values())
|
||||
)
|
||||
for fqme in warmup_fqmes:
|
||||
try:
|
||||
(
|
||||
mkt,
|
||||
details,
|
||||
) = await get_mkt_info(
|
||||
fqme,
|
||||
proxy=a_proxy,
|
||||
)
|
||||
except Exception as err:
|
||||
# XXX: non-fatal; an
|
||||
# un-warmed mkt just falls
|
||||
# back to the lazy qualify
|
||||
# in the order handler.
|
||||
log.warning(
|
||||
f'Failed to pre-qualify\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
f'err: {err!r}\n'
|
||||
)
|
||||
continue
|
||||
|
||||
for _client in unique_clients:
|
||||
cache_contract(
|
||||
_client,
|
||||
mkt,
|
||||
details.contract,
|
||||
)
|
||||
|
||||
for client in set(aioclients.values()):
|
||||
trade_event_stream: LinkedTaskChannel = await tn.start(
|
||||
open_trade_event_stream,
|
||||
|
|
@ -936,7 +800,6 @@ async def open_trade_dialog(
|
|||
ems_stream,
|
||||
accounts_def,
|
||||
flows,
|
||||
proxies,
|
||||
)
|
||||
|
||||
# allocate event relay tasks for each client connection
|
||||
|
|
|
|||
|
|
@ -32,12 +32,14 @@ import time
|
|||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
from async_generator import aclosing
|
||||
import ib_async as ibis
|
||||
import numpy as np
|
||||
from pendulum import (
|
||||
now,
|
||||
from_timestamp,
|
||||
Duration,
|
||||
duration as mk_duration,
|
||||
|
|
@ -73,6 +75,9 @@ from ._util import (
|
|||
)
|
||||
from .symbols import get_mkt_info
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from trio._core._run import Task
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
|
@ -220,11 +225,7 @@ async def open_history_client(
|
|||
f'mean: {mean}'
|
||||
)
|
||||
|
||||
# get_bars() uses None internally for an empty reply.
|
||||
# The backfiller never receives that sentinel: translate
|
||||
# it into NoData, or DataUnavailable at the known start
|
||||
# of the instrument's history. A venue closure can also
|
||||
# yield an empty reply without exhausting all history.
|
||||
# could be trying to retreive bars over weekend
|
||||
if out is None:
|
||||
log.error(
|
||||
f"No bars starting at {end_dt!r} !?!?"
|
||||
|
|
@ -352,7 +353,6 @@ async def wait_on_data_reset(
|
|||
tuple[
|
||||
trio.CancelScope,
|
||||
trio.Event,
|
||||
list[bool],
|
||||
]
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
) -> bool:
|
||||
|
|
@ -396,9 +396,8 @@ async def wait_on_data_reset(
|
|||
client: Client = proxy._aio_ns
|
||||
|
||||
done = trio.Event()
|
||||
result: list[bool] = []
|
||||
with trio.move_on_after(timeout) as cs:
|
||||
task_status.started((cs, done, result))
|
||||
task_status.started((cs, done))
|
||||
|
||||
log.warning(
|
||||
'Sending DATA RESET request:\n'
|
||||
|
|
@ -414,7 +413,6 @@ async def wait_on_data_reset(
|
|||
'NO VNC DETECTED!\n'
|
||||
'Manually press ctrl-alt-f on your IB java app'
|
||||
)
|
||||
result.append(False)
|
||||
done.set()
|
||||
return False
|
||||
|
||||
|
|
@ -429,7 +427,6 @@ async def wait_on_data_reset(
|
|||
]:
|
||||
await ev.wait()
|
||||
log.info(f"{name} DATA RESET")
|
||||
result.append(True)
|
||||
done.set()
|
||||
return True
|
||||
|
||||
|
|
@ -438,12 +435,12 @@ async def wait_on_data_reset(
|
|||
'Data reset task canceled?'
|
||||
)
|
||||
|
||||
result.append(False)
|
||||
done.set()
|
||||
return False
|
||||
|
||||
|
||||
_data_reset_lock = trio.Lock()
|
||||
_data_resetter_task: Task | None = None
|
||||
_failed_resets: int = 0
|
||||
|
||||
|
||||
async def get_bars(
|
||||
|
|
@ -459,6 +456,10 @@ async def get_bars(
|
|||
# how long before we trigger a feed reset (seconds)
|
||||
feed_reset_timeout: float = 3,
|
||||
|
||||
# how many days to subtract before giving up on further
|
||||
# history queries for instrument, presuming that most don't
|
||||
# not trade for a week XD
|
||||
max_nodatas: int = 6,
|
||||
max_failed_resets: int = 6,
|
||||
|
||||
task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
|
||||
|
|
@ -478,10 +479,8 @@ async def get_bars(
|
|||
`.ib.api.Client` methods.
|
||||
|
||||
'''
|
||||
cancelled_count: int = 0
|
||||
failed_resets: int = 0
|
||||
pacing_reset_pending: bool = False
|
||||
pacing_reset_completed: bool = False
|
||||
global _data_resetter_task, _failed_resets
|
||||
nodatas_count: int = 0
|
||||
|
||||
data_cs: trio.CancelScope | None = None
|
||||
result: tuple[
|
||||
|
|
@ -494,16 +493,10 @@ async def get_bars(
|
|||
|
||||
async def query():
|
||||
|
||||
nonlocal result, data_cs
|
||||
nonlocal cancelled_count, failed_resets
|
||||
nonlocal pacing_reset_pending, pacing_reset_completed
|
||||
global _failed_resets
|
||||
nonlocal result, data_cs, end_dt, nodatas_count
|
||||
|
||||
while True:
|
||||
if failed_resets >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB history reset failed {failed_resets} '
|
||||
f'times for {fqme}'
|
||||
)
|
||||
while _failed_resets < max_failed_resets:
|
||||
try:
|
||||
(
|
||||
bars,
|
||||
|
|
@ -535,10 +528,6 @@ async def get_bars(
|
|||
)
|
||||
# NOTE: REQUIRED to pass back value..
|
||||
result = None
|
||||
failed_resets = 0
|
||||
result_ready.set()
|
||||
if data_cs:
|
||||
data_cs.cancel()
|
||||
return None
|
||||
|
||||
# not enough bars signal, likely due to venue
|
||||
|
|
@ -598,7 +587,6 @@ async def get_bars(
|
|||
first_dt,
|
||||
last_dt,
|
||||
)
|
||||
failed_resets = 0
|
||||
|
||||
# signal data reset loop parent task
|
||||
result_ready.set()
|
||||
|
|
@ -623,21 +611,27 @@ async def get_bars(
|
|||
# the frame dt index since the upper layer may
|
||||
# be doing so concurrently and we don't want to
|
||||
# be delivering frames that weren't asked for.
|
||||
# try to decrement start point and look further back
|
||||
# end_dt = end_dt.subtract(seconds=2000)
|
||||
logmsg = "SUBTRACTING DAY from DT index"
|
||||
if end_dt is not None:
|
||||
end_dt = end_dt.subtract(days=1)
|
||||
elif end_dt is None:
|
||||
end_dt = now().subtract(days=1)
|
||||
|
||||
log.warning(
|
||||
f'NO DATA found ending @ {end_dt}\n'
|
||||
+ logmsg
|
||||
)
|
||||
# Match get_bars()'s empty-bars return: None is
|
||||
# internal to the IB adapter, not its backfill
|
||||
# protocol. open_history_client() converts it
|
||||
# to NoData (or DataUnavailable at history start).
|
||||
# Wake the waiting get_bars() task before exiting
|
||||
# so it does not mistake this reply for a timeout.
|
||||
result = None
|
||||
failed_resets = 0
|
||||
result_ready.set()
|
||||
if data_cs:
|
||||
data_cs.cancel()
|
||||
return None
|
||||
|
||||
if nodatas_count >= max_nodatas:
|
||||
raise DataUnavailable(
|
||||
f'Presuming {fqme} has no further history '
|
||||
f'after {max_nodatas} tries..'
|
||||
)
|
||||
|
||||
nodatas_count += 1
|
||||
continue
|
||||
|
||||
elif (
|
||||
'API historical data query cancelled'
|
||||
|
|
@ -648,12 +642,6 @@ async def get_bars(
|
|||
'Query cancelled by IB (:eyeroll:):\n'
|
||||
f'{err.message}'
|
||||
)
|
||||
cancelled_count += 1
|
||||
if cancelled_count >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB cancelled {cancelled_count} history '
|
||||
f'queries for {fqme}'
|
||||
)
|
||||
continue
|
||||
|
||||
elif (
|
||||
|
|
@ -674,7 +662,6 @@ async def get_bars(
|
|||
)
|
||||
|
||||
client = proxy._aio_ns.ib.client
|
||||
pacing_reset_pending = True
|
||||
|
||||
# cancel any existing reset task
|
||||
if data_cs:
|
||||
|
|
@ -682,81 +669,69 @@ async def get_bars(
|
|||
data_cs.cancel()
|
||||
|
||||
# spawn new data reset task
|
||||
async with _data_reset_lock:
|
||||
(
|
||||
data_cs,
|
||||
reset_done,
|
||||
reset_result,
|
||||
) = await tn.start(
|
||||
data_cs, reset_done = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='connection'
|
||||
)
|
||||
)
|
||||
await reset_done.wait()
|
||||
if reset_result[0]:
|
||||
failed_resets = 0
|
||||
if reset_done:
|
||||
_failed_resets = 0
|
||||
else:
|
||||
failed_resets += 1
|
||||
pacing_reset_pending = False
|
||||
pacing_reset_completed = True
|
||||
_failed_resets += 1
|
||||
continue
|
||||
|
||||
else:
|
||||
raise
|
||||
|
||||
# TODO: make this global across all history task/requests
|
||||
# such that simultaneous symbol queries don't try data resettingn
|
||||
# too fast..
|
||||
unset_resetter: bool = False
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
|
||||
# Run history until a result or bounded reset failure.
|
||||
# start history request that we allow
|
||||
# to run indefinitely until a result is acquired
|
||||
tn.start_soon(query)
|
||||
|
||||
# Serialize farm resets across every history requester.
|
||||
# start history reset loop which waits up to the timeout
|
||||
# for a result before triggering a data feed reset.
|
||||
while not result_ready.is_set():
|
||||
|
||||
with trio.move_on_after(feed_reset_timeout):
|
||||
await result_ready.wait()
|
||||
break
|
||||
|
||||
async with _data_reset_lock:
|
||||
if result_ready.is_set():
|
||||
break
|
||||
if pacing_reset_completed:
|
||||
pacing_reset_completed = False
|
||||
if _data_resetter_task:
|
||||
# don't double invoke the reset hack if another
|
||||
# requester task already has it covered.
|
||||
continue
|
||||
|
||||
(
|
||||
data_cs,
|
||||
reset_done,
|
||||
reset_result,
|
||||
) = await tn.start(
|
||||
else:
|
||||
_data_resetter_task = trio.lowlevel.current_task()
|
||||
unset_resetter: bool = True
|
||||
|
||||
# spawn new data reset task
|
||||
data_cs, reset_done = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='data',
|
||||
)
|
||||
)
|
||||
# sync wait on reset to complete
|
||||
await reset_done.wait()
|
||||
|
||||
if (
|
||||
result_ready.is_set()
|
||||
or
|
||||
reset_result[0]
|
||||
):
|
||||
failed_resets = 0
|
||||
elif pacing_reset_pending:
|
||||
# Pacing recovery deliberately cancelled this reset
|
||||
# and will consume the retry itself.
|
||||
pass
|
||||
else:
|
||||
failed_resets += 1
|
||||
if failed_resets >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB history reset failed '
|
||||
f'{failed_resets} times for {fqme}'
|
||||
_data_resetter_task = (
|
||||
None
|
||||
if unset_resetter
|
||||
else _data_resetter_task
|
||||
)
|
||||
assert result
|
||||
return (
|
||||
result,
|
||||
data_cs is not None,
|
||||
|
|
|
|||
|
|
@ -639,40 +639,12 @@ async def get_mkt_info(
|
|||
|
||||
# if possible register the bs_mktid to the just-built
|
||||
# mkt so that it can be retreived by order mode tasks later.
|
||||
# TODO NOTE: this is going to be problematic if/when we split
|
||||
# out the datatd vs. brokerd actors since the mktmap lookup
|
||||
# table will now be inaccessible..
|
||||
if proxy is not None:
|
||||
cache_contract(
|
||||
proxy._aio_ns,
|
||||
mkt,
|
||||
con,
|
||||
)
|
||||
client: Client = proxy._aio_ns
|
||||
client._contracts[mkt.bs_fqme] = con
|
||||
client._cons2mkts[con] = mkt
|
||||
|
||||
return mkt, details
|
||||
|
||||
|
||||
def cache_contract(
|
||||
client: Client,
|
||||
mkt: MktPair,
|
||||
con: ibis.Contract,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Register a (qualified) contract + mkt-info pair on the
|
||||
given (api) `Client`'s actor-local lookup tables.
|
||||
|
||||
Cached under BOTH fqme key-forms since consumers vary:
|
||||
- `mkt.bs_fqme` (eg. 'nvda.nasdaq'): read by
|
||||
`Client.submit_limit()` for order requests,
|
||||
- `mkt.fqme` (eg. 'nvda.nasdaq.ib'): read by the
|
||||
fill-time pp-update (symcache-backup-table) path in
|
||||
`.broker.emit_pp_update()`.
|
||||
|
||||
NOTE: post the (datad|brokerd)-actor-split this MUST be
|
||||
run (in the trading actor) for every mkt either eagerly
|
||||
at `.broker.open_trade_dialog()` startup or lazily per
|
||||
order request; there is no in-proc feed setup doing it
|
||||
implicitly anymore!
|
||||
|
||||
'''
|
||||
client._contracts[mkt.bs_fqme] = con
|
||||
client._contracts[mkt.fqme] = con
|
||||
client._cons2mkts[con] = mkt
|
||||
|
|
|
|||
|
|
@ -66,24 +66,10 @@ __all__ = [
|
|||
]
|
||||
|
||||
|
||||
# per-daemon-kind (sub)mod groups: declares which of our
|
||||
# submods host the eps run by each daemon-actor kind as
|
||||
# defined by `piker.data.validate._eps`.
|
||||
_brokerd_mods: list[str] = [
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = [
|
||||
'api',
|
||||
'broker',
|
||||
]
|
||||
|
||||
_datad_mods: list[str] = [
|
||||
'api',
|
||||
'feed',
|
||||
'symbols',
|
||||
]
|
||||
|
||||
|
||||
# tractor RPC enable arg
|
||||
__enable_modules__: list[str] = list(dict.fromkeys(
|
||||
_brokerd_mods
|
||||
+
|
||||
_datad_mods
|
||||
))
|
||||
|
|
|
|||
|
|
@ -52,7 +52,6 @@ from piker.brokers._util import (
|
|||
SymbolNotFound,
|
||||
BrokerError,
|
||||
DataThrottle,
|
||||
get_or_raise_on_pair_schema_mismatch,
|
||||
)
|
||||
from piker.accounting import Transaction
|
||||
from piker.log import get_logger
|
||||
|
|
@ -118,9 +117,6 @@ class InvalidKey(ValueError):
|
|||
|
||||
'''
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([InvalidKey])
|
||||
|
||||
|
||||
class Client:
|
||||
|
||||
|
|
@ -506,16 +502,7 @@ class Client:
|
|||
|
||||
# NOTE: always cache in pairs tables for faster lookup
|
||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||
# pair = Pair(xname=xkey, **data)
|
||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type=Pair,
|
||||
fields_data=dict(
|
||||
xname=xkey,
|
||||
**data,
|
||||
),
|
||||
provider_name='kraken',
|
||||
# api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
||||
)
|
||||
pair = Pair(xname=xkey, **data)
|
||||
|
||||
# register the above `Pair` structs for all
|
||||
# key-sets/monikers: a set of 4 (frickin) tables
|
||||
|
|
|
|||
|
|
@ -67,15 +67,13 @@ from piker.log import (
|
|||
get_logger,
|
||||
)
|
||||
from piker.data import open_symcache
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
)
|
||||
from .api import (
|
||||
Client,
|
||||
BrokerError,
|
||||
)
|
||||
from .feed import (
|
||||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
stream_messages,
|
||||
)
|
||||
from .ledger import (
|
||||
|
|
@ -99,9 +97,6 @@ MsgUnion = Union[
|
|||
class TooFastEdit(Exception):
|
||||
'Edit requests faster then api submissions'
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([TooFastEdit])
|
||||
|
||||
|
||||
# TODO: make this wrap the `Client` and `ws` instances
|
||||
# and give it methods to submit cancel vs. add vs. edit
|
||||
|
|
|
|||
|
|
@ -19,9 +19,6 @@ Symbology defs and search.
|
|||
|
||||
'''
|
||||
from decimal import Decimal
|
||||
from typing import (
|
||||
ClassVar,
|
||||
)
|
||||
|
||||
import tractor
|
||||
|
||||
|
|
@ -89,14 +86,9 @@ class Pair(Struct):
|
|||
short_position_limit: float = 0
|
||||
long_position_limit: float = float('inf')
|
||||
|
||||
# TODO, add API note when this was added!
|
||||
execution_venue: str|None = None
|
||||
|
||||
# TODO: should we make this a literal NamespacePath ref?
|
||||
ns_path: str = 'piker.brokers.kraken:Pair'
|
||||
|
||||
_api_url: ClassVar[str] = 'https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs'
|
||||
|
||||
@property
|
||||
def bs_mktid(self) -> str:
|
||||
'''
|
||||
|
|
|
|||
|
|
@ -95,9 +95,6 @@ _time_frames = {
|
|||
class QuestradeError(Exception):
|
||||
"Non-200 OK response code"
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([QuestradeError])
|
||||
|
||||
|
||||
class ContractsKey(NamedTuple):
|
||||
symbol: str
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue