pikers
/
piker
4
1
Fork 0
Code Issues 20 Pull Requests 13 Packages Projects Releases Wiki Activity

Document bounded IB recovery and downstream replay 

Record exact keep/backout/import provenance for the #94 candidate,
including the quote-cache lineage correction and the distinction
between file-equivalent adapter deltas and partial source commits.

Deats,
- Preserve the original raw prompt record and add the human's
  later scope decisions, corrections and live-test contributions
  to its structured partner.
- Record 38 prior unique offline passes and the known-red bridge
  callback-cleanliness case, without claiming clean teardown.
- Record live QQQ backfill, reattachment and overlapping chart
  close/reopen results. Startup needed debugger continuation;
  the local breakpoint cleanup was applied after that run.
- Retain the cached-client daemon shutdown failure as a human-
  accepted follow-up outside #94, with its exact propagation path.
- Require disposable downstream rebase rehearsals before landing,
  preserving omitted SHM/storage fixes and local regressions.

The commit package records exact per-boundary project checks.
Its structural replay/preflight does not substitute for those
checks, the prior live observations or rebase qualification.

Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
wkt/land_ib_recovery
Gud Boi 2026-10-23 11:36:29 -04:00
parent 11aa2eeb1e
commit 3ec447a45a
3 changed files with 369 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

233
ai/ib-recovery-candidate.md 100644
View File

@ -0,0 +1,233 @@
# Bounded IB recovery candidate
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
`/home/goodboy/repos/piker`. Base:
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
Changes are unstaged and uncommitted. No acceptance is implied.
## Patch selection and provenance
Keep these existing landing commits:
| Commit | Reason |
| --- | --- |
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
| `e8e0730a` | Correlate MethodProxy requests and responses |
| `7bef50b0` | One response reader, per-request waiters |
| `d1571ddf` | Propagate daemon-handler exceptions |
| `dc618229` | Isolate cached feed stream consumers |
Proposed separate implementation boundaries, in this order:
1. Back out the code and original test from
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
This restores the direct-owner `_quote_streams` implementation.
Retain the historical prompt records. Replace the removed
mocked-cache regression with the real-bridge restart reproducer.
This backout is not a claim that arbitrary concurrent direct
callers have safe broadcast ownership.
2. Backport **only** `piker/brokers/ib/feed.py` from
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
Retain its seven history-adapter regressions in
`tests/test_ib_history.py`. This includes reset serialization,
completion/result signalling and bounded cancellation/failure.
It does not include the original SHM/storage orchestration patch.
3. Backport **only** `piker/brokers/ib/feed.py` from
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
Retain `test_empty_history_preserves_request_boundary` and its
six cases. This preserves no-data request boundaries and wakes
the waiter. It does not include the original history-layer
endpoint/filtering changes. Add the real-bridge history recovery
case after these adapter changes are present.
4. Import the code and original regression from
`12b0d66e854247e92cb5a8668be71d816c72d856`.
Add a separate real-broadcaster/production-loop regression.
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
This is a local UX cleanup retaining the existing `None` result.
It does not backport the null-grouping/deduplication rewrite in
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
6. Record cross-boundary regression results and live qualification
evidence, distinguishing the known callback failure from passes.
All three imported source commits were authored by
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
and the full source SHA when committing each partial backport.
Use explicit partial-backport wording, not a whole-commit cherry
claim. New real-runtime tests are this session's additional work.
The current aggregate working diff contains these boundaries in
the same feed file. Staging that entire file would combine the
first three boundaries. Reconstruct/stage the original per-source
patches in the above order when preparing commits.
## Quote-cache history correction
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
They are equivalent replays, not different implementations.
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
change and retains direct ownership. The original qualification
did not validate the combination assembled for the landing.
After this candidate's edits, `piker/brokers/ib/feed.py` matches
the source at `1da9b760` byte-for-byte. Compared with the evaluated
tip, only the later schedule/reservation APIs are absent.
The EMS code patch has the same stable patch ID as the original
`12b0d66e` EMS code patch:
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
## Real-runtime evidence
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
an asyncio-infected child, the installed pinned Tractor bridge,
production Piker quote setup, and real IB/eventkit ticker events.
Only the external IB connection/input is replaced with replay data.
Before the quote-cache backout, cancellation after ticker delivery
raised `RemoteActorError[TrioTaskExited]` through
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
reported fatal path.
After the backout:
- Two quote generations finish; each subscription is cancelled,
each asyncio producer is done and each ticker handler removed.
- A second RPC repeats the sequence in the same actor successfully.
- The strict callback-cleanliness case still fails: Tractor's
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
exception handler even though the Trio consumer recovers.
The history integration case keeps `Client.bars()`, MethodProxy,
both relays, `get_bars()`, `wait_on_data_reset()` and
`data_reset_hack()` real. It replaces external contract discovery,
IB history replies and VNC input. A real IB error event announces
farm recovery. The replay rejects retries before recovery and
proves one reset followed by completion for both 1s and 60s data.
The new EMS case runs the production dark-clearing loop over a
real Tractor broadcaster: both EMS and a sibling receive a tick,
EMS cancellation removes only its subscription, and the sibling
continues with an unchanged strict lag policy.
## Verification
Unique selected cases: **38 passed, 1 failed**.
The failure is intentionally visible, not skipped or xfailed.
Run from the checkout's working Nix environment:
```xsh
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
git diff --check
```
The adapter/proxy/feed/EMS selection initially passed 21 cases;
the added real EMS case then passed with its imported case.
The bridge file passed two cases and failed callback cleanliness.
Logging/startup passed 13 cases; runtime boot passed one with the
existing `Registrar.uid` deprecation warning.
The agent shell inherited another project's environment and lacked
native libraries. Tests used the existing Piker `py313` via
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
libraries from the pinned flake's existing Qt dependency closure,
excluding Nix Qt itself and glibc. No environment sync or shellHook
ran. Piker resolved to this checkout; Tractor resolved to its
installed package in `py313`.
Workflow-state preparation initially reported changed legacy
files/modes after migration. On 2026-10-23 the human authorized
reconciliation. Inspection found only two missing legacy guidance
snapshots; the neutral copies matched their saved hashes and modes.
Restoring those local snapshots resolved preparation without
changing archived messages or the original migration receipt.
## Human live qualification, 2026-10-23
The human ran these journeys with the installed Tractor pin,
the existing IB Gateway and pre-existing NativeDB. The agent did
not operate the broker connection, VNC, orders or database.
- Initial QQQ startup/backfill recovered after pacing error 162
with one reset attempt, then displayed live updates. The reset
waiter appears to have timed out; the log did not establish an
acknowledged farm reset or live quote-stream restart.
- Sequential chart exits and reattachment preserved all four
daemon PIDs; a read-only process/listener check confirmed this.
- The overlapping two-chart journey used `pikerd@961108`,
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
then C closed at 08:18:37. The human reported the exercise worked;
survivor updates are a manual observation, while timestamps and
unchanged daemon identities corroborate overlap and service reuse.
- Startup in that journey hit the explicit single-null-sample
breakpoint several times before the charts overlapped. The small
debugger cleanup above was applied afterward; this live run does
not itself verify that later edit.
- Departing charts still report cancellation groups and an FSP
receiver-close `ClosedResourceError`, followed by the Qt close
handler's self-SIGINT. Peer-disconnected final result warnings
remain contained in the persistent services.
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
After the human continued, all three children were reported
exited with status 0 and the root terminated at 08:19:45.
Cancellation-only RPC error reporting and late cancel-ack
diagnostics also remain.
The human explicitly accepted deferring this daemon teardown fix
from #94 and requested loud attribution for the bounded backports.
This is an operational smoke result, not clean-shutdown coverage.
## Downstream reconciliation requirements
Before landing #94, the human wants an easy replay of the downstream
histories intended to follow it. Rehearse against the agreed #94
tree in disposable worktrees, then inspect range-diffs, final code
and relevant regressions. This note does not authorize rewriting
the original branches or changing acceptance markers.
- `0846cbd4` and `1da9b760` are partial backports. Their imported
adapter hunks may already be present, but their history/SHM/storage
changes and omitted integration tests remain necessary downstream.
Never drop either whole source commit merely because `feed.py`
matches or its adapter hunk becomes empty.
- `12b0d66e` has source-equivalent EMS code and its imported test,
plus an additional local real-broadcaster test. Preserve that extra
regression when resolving test-file overlap.
- The quote-cache backout intentionally restores direct ownership
for this landing. Reintroducing original `4af885d9` downstream
requires qualifying cached-owner teardown with the matching
Tractor revision; the backout is not a permanent ownership design.
- The local single-null breakpoint removal retains legacy behavior.
Prefer the full downstream `976db6c3` algorithm when it is replayed;
preserving this `return None` guard would lose that later fix.
- Real-bridge tests and manual qualification notes are local work,
not evidence that omitted portions of the source commits landed.
Commit messages must name full source SHAs, original authorship,
imported paths/tests, omissions, local additions and replay guidance.
Use explicit partial-backport language for adapter subsets; do not
claim a full cherry-pick or reuse the source's broader fix claims.
## Remaining decision
This candidate removes the reproduced fatal quote-restart
propagation and imports existing reset fixes. The human exercised
live QQQ startup, backfill and concurrent chart consumer lifetimes.
Completion-callback and teardown errors remain; the human accepts
them as follow-ups outside the bounded #94 landing.
The storage/SHM portions of the downstream fixes are not covered by
the adapter backport or its integration tests.
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
`1a2d4893` stays a separate follow-up. Further Tractor teardown
changes or storage imports require reassessing the bounded landing
rather than silently expanding this candidate.

67
ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md 100644
View File

@ -0,0 +1,67 @@
---
created_at: 2026-10-22T21:22:48Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6-astra
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
scope: code
substantive: true
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
---
## Prompt
The human approved trying the bounded candidate: back out the IB
quote-context cache, import adapter-only reset/boundary fixes and
the missing EMS subscription fix, preserve source authorship and
commit boundaries, and exercise the real Tractor bridge. They
required an explicit keep/backout audit and clarification that
the evaluated downstream lineage lacked the broadcast fix.
## Response summary
Verified identical stable patch IDs for original 4af885d9 and
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
in the main checkout. Applied source-equivalent adapter backports
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
source attribution recorded in `ai/ib-recovery-candidate.md`.
The real-runtime quote test first reproduced the reported fatal
TrioTaskExited propagation with caching enabled. The direct-owner
backout restores restart and subsequent RPC survival, but a strict
callback-cleanliness test remains red. Real bridge history pacing
recovery and real broadcaster dark clearing pass. No live broker,
VNC, orders or TSDB operations were performed by the agent.
On 2026-10-23 the human supplied successful live startup/backfill,
sequential reattachment and overlapping chart-close/reopen results,
plus the still-failing daemon SIGINT teardown. The human accepted
deferring teardown from #94, requested loud partial-backport
attribution and required downstream rebase rehearsals before landing.
The agent removed only the explicit single-null debugger stop,
retaining its legacy result. Downstream `976db6c3` instead rewrites
null grouping; the local cleanup does not claim that algorithmic fix.
## Files changed
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
- `piker/clearing/_ems.py`: original EMS subscription patch.
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
- `tests/test_ib_history.py`: original adapter regressions only.
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
## Human edits
The human required original fixes and authorship, rejected broad
speculative integration work, required real concurrency boundaries
in tests, and corrected the interpretation of live qualification.
They performed the live tests, chose to defer known teardown defects
from #94, permitted the minimal debugger cleanup, and requested
explicit downstream discrepancies and rebase compatibility checks.
The agent applied those directed edits; no direct manual source edits
are known in this candidate. The raw partner retains the originally
captured output; this structured record includes later contributions.

69
ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md 100644
View File

@ -0,0 +1,69 @@
---
created_at: 2026-10-22T21:22:48Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6-astra
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
---
## Human direction
> yeah i guess this is our best approach strat for sure.
> so you're saying these 2 are not the same patch?
> please tell me no lol
> so detail which commits to drop vs. keep please in detail so i can
> audit your choices this time..
> let's try it and consider bailing and re-trying if this leads us into
> a further (and deeper) sinkhole, bc as it stands right now this whole
> endevour has turned into a complete shitfest XD
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
## Unedited diagnostic response
The real-bridge test reproduced the exact fatal path:
`maybe_open_context()` teardown raises `TrioTaskExited` back into
the quote consumer.
With only the quote-cache backout, both quote generations complete
and their asyncio tasks finish. Tractor still emits
`TrioTaskExited` callback errors, so the strict test remains red.
That narrows the result: the backout removes the fatal propagation
in this reproducer, but doesn’t make bridge teardown clean.
## Code output pointers
> `git diff dc618229 -- piker/brokers/ib/feed.py`
Back out 635314c7's quote owner and import the original adapter-only
deltas from 0846cbd4 and 1da9b760.
> `git diff dc618229 -- piker/clearing/_ems.py`
Import 12b0d66e's dedicated dark-clearing quote subscription.
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
Replace the reverted mock-cache test with a real-runtime restart
reproducer; retain original history/EMS tests and add real bridge
history recovery and real broadcaster EMS coverage. Newly added
test files remain untracked until the human stages them, so the
diff pointers include them only after staging or committing.
## Verification output
```text
21 passed in 0.53s
1 failed, 2 passed in 5.53s
13 passed in 1.41s
1 passed, 1 warning in 0.04s
2 passed in 0.03s
```
The last run includes the original EMS case again and one additional
real-broadcaster case: 38 unique passes and one strict callback failure.