piker/ai/ib-recovery-candidate.md

Bounded IB recovery candidate

Prepared 2026-10-22 on wkt/land_ib_recovery, in the main checkout /home/goodboy/repos/piker. Base: dc618229bd8c9c90aa3160e26677b553a7d472c7. Changes are unstaged and uncommitted. No acceptance is implied.

Patch selection and provenance

Keep these existing landing commits:

Commit	Reason
4133f073 and its ancestors	Packaging, Nix, runtime API and pin work
e8e0730a	Correlate MethodProxy requests and responses
7bef50b0	One response reader, per-request waiters
d1571ddf	Propagate daemon-handler exceptions
dc618229	Isolate cached feed stream consumers

Proposed separate implementation boundaries, in this order:

1. Back out the code and original test from 635314c757c74a1cd95c5cf3dc7dfd03be984b00. This restores the direct-owner _quote_streams implementation. Retain the historical prompt records. Replace the removed mocked-cache regression with the real-bridge restart reproducer. This backout is not a claim that arbitrary concurrent direct callers have safe broadcast ownership.
2. Backport only piker/brokers/ib/feed.py from 0846cbd4a4197e47415689cf5e9d146ba3174c32. Retain its seven history-adapter regressions in tests/test_ib_history.py. This includes reset serialization, completion/result signalling and bounded cancellation/failure. It does not include the original SHM/storage orchestration patch.
3. Backport only piker/brokers/ib/feed.py from 1da9b760a3bc6b96501110cab2b5d09846d42a1d. Retain test_empty_history_preserves_request_boundary and its six cases. This preserves no-data request boundaries and wakes the waiter. It does not include the original history-layer endpoint/filtering changes. Add the real-bridge history recovery case after these adapter changes are present.
4. Import the code and original regression from 12b0d66e854247e92cb5a8668be71d816c72d856. Add a separate real-broadcaster/production-loop regression.
5. Remove the single-null-sample debugger stop in get_null_segs(). This is a local UX cleanup retaining the existing None result. It does not backport the null-grouping/deduplication rewrite in 976db6c33c4b8ed07eee3047226417d3928d9cca.
6. Record cross-boundary regression results and live qualification evidence, distinguishing the known callback failure from passes.

All three imported source commits were authored by goodboy <goodboy_foss@protonmail.com>; preserve that authorship and the full source SHA when committing each partial backport. Use explicit partial-backport wording, not a whole-commit cherry claim. New real-runtime tests are this session’s additional work.

The current aggregate working diff contains these boundaries in the same feed file. Staging that entire file would combine the first three boundaries. Reconstruct/stage the original per-source patches in the above order when preparing commits.

Quote-cache history correction

4af885d90f6e4bcfa963f2f7a1d40436d3c928f3 and landing commit 635314c757c74a1cd95c5cf3dc7dfd03be984b00 have the same stable patch ID: bc54892c25e957be112b135bb9e383ec4dd123c5. They are equivalent replays, not different implementations.

wkt/ib_downstream_pin_eval@e929fdc2 lacks that quote-cache change and retains direct ownership. The original qualification did not validate the combination assembled for the landing.

After this candidate’s edits, piker/brokers/ib/feed.py matches the source at 1da9b760 byte-for-byte. Compared with the evaluated tip, only the later schedule/reservation APIs are absent. The EMS code patch has the same stable patch ID as the original 12b0d66e EMS code patch: 50fee047e94fd67c8d9feab66859fbe3ab25b64c.

Real-runtime evidence

tests/test_ib_quote_restart.py uses real TCP Tractor actors, an asyncio-infected child, the installed pinned Tractor bridge, production Piker quote setup, and real IB/eventkit ticker events. Only the external IB connection/input is replaced with replay data.

Before the quote-cache backout, cancellation after ticker delivery raised RemoteActorError[TrioTaskExited] through maybe_open_context() / _Cache.run_ctx teardown, matching the reported fatal path.

After the backout:

• Two quote generations finish; each subscription is cancelled, each asyncio producer is done and each ticker handler removed.
• A second RPC repeats the sequence in the same actor successfully.
• The strict callback-cleanliness case still fails: Tractor’s signal_trio_when_done() raises TrioTaskExited into asyncio’s exception handler even though the Trio consumer recovers.

The history integration case keeps Client.bars(), MethodProxy, both relays, get_bars(), wait_on_data_reset() and data_reset_hack() real. It replaces external contract discovery, IB history replies and VNC input. A real IB error event announces farm recovery. The replay rejects retries before recovery and proves one reset followed by completion for both 1s and 60s data.

The new EMS case runs the production dark-clearing loop over a real Tractor broadcaster: both EMS and a sibling receive a tick, EMS cancellation removes only its subscription, and the sibling continues with an unchanged strict lag policy.

Verification

Unique selected cases: 38 passed, 1 failed. The failure is intentionally visible, not skipped or xfailed.

Run from the checkout’s working Nix environment:

timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
git diff --check

The adapter/proxy/feed/EMS selection initially passed 21 cases; the added real EMS case then passed with its imported case. The bridge file passed two cases and failed callback cleanliness. Logging/startup passed 13 cases; runtime boot passed one with the existing Registrar.uid deprecation warning.

The agent shell inherited another project’s environment and lacked native libraries. Tests used the existing Piker py313 via /tmp/nix-shell.zfKbeB/opencode/piker_test_env.py, which supplies libraries from the pinned flake’s existing Qt dependency closure, excluding Nix Qt itself and glibc. No environment sync or shellHook ran. Piker resolved to this checkout; Tractor resolved to its installed package in py313.

Workflow-state preparation initially reported changed legacy files/modes after migration. On 2026-10-23 the human authorized reconciliation. Inspection found only two missing legacy guidance snapshots; the neutral copies matched their saved hashes and modes. Restoring those local snapshots resolved preparation without changing archived messages or the original migration receipt.

Human live qualification, 2026-10-23

The human ran these journeys with the installed Tractor pin, the existing IB Gateway and pre-existing NativeDB. The agent did not operate the broker connection, VNC, orders or database.

• Initial QQQ startup/backfill recovered after pacing error 162 with one reset attempt, then displayed live updates. The reset waiter appears to have timed out; the log did not establish an acknowledged farm reset or live quote-stream restart.
• Sequential chart exits and reattachment preserved all four daemon PIDs; a read-only process/listener check confirmed this.
• The overlapping two-chart journey used pikerd@961108, brokerd.ib@964765, samplerd@965900 and emsd@969015. B started at 08:16:31; C at 08:17:33. B closed at 08:18:11, replacement B started at 08:18:24 and had FSPs up by 08:18:28, then C closed at 08:18:37. The human reported the exercise worked; survivor updates are a manual observation, while timestamps and unchanged daemon identities corroborate overlap and service reuse.
• Startup in that journey hit the explicit single-null-sample breakpoint several times before the charts overlapped. The small debugger cleanup above was applied afterward; this live run does not itself verify that later edit.
• Departing charts still report cancellation groups and an FSP receiver-close ClosedResourceError, followed by the Qt close handler’s self-SIGINT. Peer-disconnected final result warnings remain contained in the persistent services.
• Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown: load_clients_for_trio() / _Cache.run_ctx promoted TrioTaskExited through open_trade_dialog() and entered PDB. After the human continued, all three children were reported exited with status 0 and the root terminated at 08:19:45. Cancellation-only RPC error reporting and late cancel-ack diagnostics also remain.

The human explicitly accepted deferring this daemon teardown fix from #94 and requested loud attribution for the bounded backports. This is an operational smoke result, not clean-shutdown coverage.

Downstream reconciliation requirements

Before landing #94, the human wants an easy replay of the downstream histories intended to follow it. Rehearse against the agreed #94 tree in disposable worktrees, then inspect range-diffs, final code and relevant regressions. This note does not authorize rewriting the original branches or changing acceptance markers.

• 0846cbd4 and 1da9b760 are partial backports. Their imported adapter hunks may already be present, but their history/SHM/storage changes and omitted integration tests remain necessary downstream. Never drop either whole source commit merely because feed.py matches or its adapter hunk becomes empty.
• 12b0d66e has source-equivalent EMS code and its imported test, plus an additional local real-broadcaster test. Preserve that extra regression when resolving test-file overlap.
• The quote-cache backout intentionally restores direct ownership for this landing. Reintroducing original 4af885d9 downstream requires qualifying cached-owner teardown with the matching Tractor revision; the backout is not a permanent ownership design.
• The local single-null breakpoint removal retains legacy behavior. Prefer the full downstream 976db6c3 algorithm when it is replayed; preserving this return None guard would lose that later fix.
• Real-bridge tests and manual qualification notes are local work, not evidence that omitted portions of the source commits landed.

Commit messages must name full source SHAs, original authorship, imported paths/tests, omissions, local additions and replay guidance. Use explicit partial-backport language for adapter subsets; do not claim a full cherry-pick or reuse the source’s broader fix claims.

Remaining decision

This candidate removes the reproduced fatal quote-restart propagation and imports existing reset fixes. The human exercised live QQQ startup, backfill and concurrent chart consumer lifetimes. Completion-callback and teardown errors remain; the human accepts them as follow-ups outside the bounded #94 landing. The storage/SHM portions of the downstream fixes are not covered by the adapter backport or its integration tests.

ddac3553 is Deribit-specific and omitted. Qt lifecycle commit 1a2d4893 stays a separate follow-up. Further Tractor teardown changes or storage imports require reassessing the bounded landing rather than silently expanding this candidate.