piker/ai/ib-recovery-candidate.md

# Bounded IB recovery candidate

Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
`/home/goodboy/repos/piker`. Base:
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
Changes are unstaged and uncommitted. No acceptance is implied.

## Patch selection and provenance

Keep these existing landing commits:

| Commit | Reason |
| --- | --- |
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
| `e8e0730a` | Correlate MethodProxy requests and responses |
| `7bef50b0` | One response reader, per-request waiters |
| `d1571ddf` | Propagate daemon-handler exceptions |
| `dc618229` | Isolate cached feed stream consumers |

Proposed separate implementation boundaries, in this order:

1. Back out the code and original test from
   `635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
   This restores the direct-owner `_quote_streams` implementation.
   Retain the historical prompt records. Replace the removed
   mocked-cache regression with the real-bridge restart reproducer.
   This backout is not a claim that arbitrary concurrent direct
   callers have safe broadcast ownership.
2. Backport **only** `piker/brokers/ib/feed.py` from
   `0846cbd4a4197e47415689cf5e9d146ba3174c32`.
   Retain its seven history-adapter regressions in
   `tests/test_ib_history.py`. This includes reset serialization,
   completion/result signalling and bounded cancellation/failure.
   It does not include the original SHM/storage orchestration patch.
3. Backport **only** `piker/brokers/ib/feed.py` from
   `1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
   Retain `test_empty_history_preserves_request_boundary` and its
   six cases. This preserves no-data request boundaries and wakes
   the waiter. It does not include the original history-layer
   endpoint/filtering changes. Add the real-bridge history recovery
   case after these adapter changes are present.
4. Import the code and original regression from
   `12b0d66e854247e92cb5a8668be71d816c72d856`.
   Add a separate real-broadcaster/production-loop regression.
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
   This is a local UX cleanup retaining the existing `None` result.
   It does not backport the null-grouping/deduplication rewrite in
   `976db6c33c4b8ed07eee3047226417d3928d9cca`.
6. Record cross-boundary regression results and live qualification
   evidence, distinguishing the known callback failure from passes.

All three imported source commits were authored by
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
and the full source SHA when committing each partial backport.
Use explicit partial-backport wording, not a whole-commit cherry
claim. New real-runtime tests are this session's additional work.

The current aggregate working diff contains these boundaries in
the same feed file. Staging that entire file would combine the
first three boundaries. Reconstruct/stage the original per-source
patches in the above order when preparing commits.

## Quote-cache history correction

`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
They are equivalent replays, not different implementations.

`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
change and retains direct ownership. The original qualification
did not validate the combination assembled for the landing.

After this candidate's edits, `piker/brokers/ib/feed.py` matches
the source at `1da9b760` byte-for-byte. Compared with the evaluated
tip, only the later schedule/reservation APIs are absent.
The EMS code patch has the same stable patch ID as the original
`12b0d66e` EMS code patch:
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.

## Real-runtime evidence

`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
an asyncio-infected child, the installed pinned Tractor bridge,
production Piker quote setup, and real IB/eventkit ticker events.
Only the external IB connection/input is replaced with replay data.

Before the quote-cache backout, cancellation after ticker delivery
raised `RemoteActorError[TrioTaskExited]` through
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
reported fatal path.

After the backout:

- Two quote generations finish; each subscription is cancelled,
  each asyncio producer is done and each ticker handler removed.
- A second RPC repeats the sequence in the same actor successfully.
- The strict callback-cleanliness case still fails: Tractor's
  `signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
  exception handler even though the Trio consumer recovers.

The history integration case keeps `Client.bars()`, MethodProxy,
both relays, `get_bars()`, `wait_on_data_reset()` and
`data_reset_hack()` real. It replaces external contract discovery,
IB history replies and VNC input. A real IB error event announces
farm recovery. The replay rejects retries before recovery and
proves one reset followed by completion for both 1s and 60s data.

The new EMS case runs the production dark-clearing loop over a
real Tractor broadcaster: both EMS and a sibling receive a tick,
EMS cancellation removes only its subscription, and the sibling
continues with an unchanged strict lag policy.

## Verification

Unique selected cases: **38 passed, 1 failed**.
The failure is intentionally visible, not skipped or xfailed.

Run from the checkout's working Nix environment:

```xsh
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
git diff --check
```

The adapter/proxy/feed/EMS selection initially passed 21 cases;
the added real EMS case then passed with its imported case.
The bridge file passed two cases and failed callback cleanliness.
Logging/startup passed 13 cases; runtime boot passed one with the
existing `Registrar.uid` deprecation warning.

The agent shell inherited another project's environment and lacked
native libraries. Tests used the existing Piker `py313` via
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
libraries from the pinned flake's existing Qt dependency closure,
excluding Nix Qt itself and glibc. No environment sync or shellHook
ran. Piker resolved to this checkout; Tractor resolved to its
installed package in `py313`.

Workflow-state preparation initially reported changed legacy
files/modes after migration. On 2026-10-23 the human authorized
reconciliation. Inspection found only two missing legacy guidance
snapshots; the neutral copies matched their saved hashes and modes.
Restoring those local snapshots resolved preparation without
changing archived messages or the original migration receipt.

## Human live qualification, 2026-10-23

The human ran these journeys with the installed Tractor pin,
the existing IB Gateway and pre-existing NativeDB. The agent did
not operate the broker connection, VNC, orders or database.

- Initial QQQ startup/backfill recovered after pacing error 162
  with one reset attempt, then displayed live updates. The reset
  waiter appears to have timed out; the log did not establish an
  acknowledged farm reset or live quote-stream restart.
- Sequential chart exits and reattachment preserved all four
  daemon PIDs; a read-only process/listener check confirmed this.
- The overlapping two-chart journey used `pikerd@961108`,
  `brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
  B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
  replacement B started at 08:18:24 and had FSPs up by 08:18:28,
  then C closed at 08:18:37. The human reported the exercise worked;
  survivor updates are a manual observation, while timestamps and
  unchanged daemon identities corroborate overlap and service reuse.
- Startup in that journey hit the explicit single-null-sample
  breakpoint several times before the charts overlapped. The small
  debugger cleanup above was applied afterward; this live run does
  not itself verify that later edit.
- Departing charts still report cancellation groups and an FSP
  receiver-close `ClosedResourceError`, followed by the Qt close
  handler's self-SIGINT. Peer-disconnected final result warnings
  remain contained in the persistent services.
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
  `load_clients_for_trio()` / `_Cache.run_ctx` promoted
  `TrioTaskExited` through `open_trade_dialog()` and entered PDB.
  After the human continued, all three children were reported
  exited with status 0 and the root terminated at 08:19:45.
  Cancellation-only RPC error reporting and late cancel-ack
  diagnostics also remain.

The human explicitly accepted deferring this daemon teardown fix
from #94 and requested loud attribution for the bounded backports.
This is an operational smoke result, not clean-shutdown coverage.

## Downstream reconciliation requirements

Before landing #94, the human wants an easy replay of the downstream
histories intended to follow it. Rehearse against the agreed #94
tree in disposable worktrees, then inspect range-diffs, final code
and relevant regressions. This note does not authorize rewriting
the original branches or changing acceptance markers.

- `0846cbd4` and `1da9b760` are partial backports. Their imported
  adapter hunks may already be present, but their history/SHM/storage
  changes and omitted integration tests remain necessary downstream.
  Never drop either whole source commit merely because `feed.py`
  matches or its adapter hunk becomes empty.
- `12b0d66e` has source-equivalent EMS code and its imported test,
  plus an additional local real-broadcaster test. Preserve that extra
  regression when resolving test-file overlap.
- The quote-cache backout intentionally restores direct ownership
  for this landing. Reintroducing original `4af885d9` downstream
  requires qualifying cached-owner teardown with the matching
  Tractor revision; the backout is not a permanent ownership design.
- The local single-null breakpoint removal retains legacy behavior.
  Prefer the full downstream `976db6c3` algorithm when it is replayed;
  preserving this `return None` guard would lose that later fix.
- Real-bridge tests and manual qualification notes are local work,
  not evidence that omitted portions of the source commits landed.

Commit messages must name full source SHAs, original authorship,
imported paths/tests, omissions, local additions and replay guidance.
Use explicit partial-backport language for adapter subsets; do not
claim a full cherry-pick or reuse the source's broader fix claims.

## Remaining decision

This candidate removes the reproduced fatal quote-restart
propagation and imports existing reset fixes. The human exercised
live QQQ startup, backfill and concurrent chart consumer lifetimes.
Completion-callback and teardown errors remain; the human accepts
them as follow-ups outside the bounded #94 landing.
The storage/SHM portions of the downstream fixes are not covered by
the adapter backport or its integration tests.

`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
`1a2d4893` stays a separate follow-up. Further Tractor teardown
changes or storage imports require reassessing the bounded landing
rather than silently expanding this candidate.