From e929fdc2d5c105453fd45c68e60045b68d428301 Mon Sep 17 00:00:00 2001 From: goodboy Date: Wed, 21 Oct 2026 16:16:52 -0400 Subject: [PATCH] Record chart disconnect E2E qualification handoff Preserve the manual IB detach, reattach, shared-consumer and startup cancellation evidence as a historical snapshot for the QA worker. Specify independent service ownership and deterministic phase gates, including terminal-order resurrection during second-chart bootstrap. Keep service continuity distinct from clean chart exit and retain the unresolved cancellation, debugger, order-correlation and native-crash findings. The linked Taken tasks own the worker assignments; this document provides supporting context, not acceptance or executed tests. (this patch was generated in some part by `opencode` using `gpt-6.1-sol` (`openai`)) --- .../20261019-chart-disconnect-e2e-handoff.md | 364 ++++++++++++++++++ 1 file changed, 364 insertions(+) create mode 100644 ai/plans/20261019-chart-disconnect-e2e-handoff.md diff --git a/ai/plans/20261019-chart-disconnect-e2e-handoff.md b/ai/plans/20261019-chart-disconnect-e2e-handoff.md new file mode 100644 index 00000000..ae6e1b61 --- /dev/null +++ b/ai/plans/20261019-chart-disconnect-e2e-handoff.md @@ -0,0 +1,364 @@ +# Chart disconnect / reattach E2E handoff + +Recipient: `e2e_qa_mngr` dialog agent. +Date: 2026-10-19. Draft implementation specification, not executed tests. +Source: human-operated IB qualification in `wkt/ib_downstream_pin_eval`. +Authoring harness/model: OpenCode / `openai/gpt-6.1-sol`. + +## Request and intended result + +Human request: "so while i'm going through all dis, i feel like you +should be drafting up equivalent e2e tests to handoff to the +e2e_qa_mngr dlog agent. i know they're working on them testing branches +i likely referred to prior" + +Extend the existing chart qualification work with executable journeys +equivalent to manual trials 1–7 below. Preserve existing task states. +Keep service-isolation and clean-exit assertions independently observable: +an unaffected surviving chart does not make a segfaulting exit successful. + +## Existing foundations inspected + +Paths below are local worktree snapshots; uncommitted files can change. +Verify HEAD, diff, imports and fixture contracts before implementation. + +- `wkt/human_iface_e2e`, HEAD `aadfd1cf`: headless Qt harness foundation. +- `wkt/live_chart_qtractor`, HEAD `1a2d4893`: structured session lifecycle. + Its current dirty work includes paper-mode startup forwarding, runtime + overrides, and live chart tests; HEAD alone does not contain that work. + [test_live_chart.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_live_chart.py:21) + starts a production Kraken chart, checks real SHM/price changes and + records exact actor/SHM ownership. It is explicitly selected through + `--live-provider=kraken` and uses paper EMS without submitting orders. + [test_qtractor.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_qtractor.py:73) + covers sequential structured window closure and runtime restart. + [UI fixtures](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/conftest.py:79) + restore QApplication, widgets, ViewBox registries, config and settings. + [open_test_pikerd](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/conftest.py:452) + supplies isolated config propagation and the plugin registry fixture. +- `wkt/replay_provider_e2e`, HEAD `06fee700`: real replay feed/service + tests with correlated command acknowledgements and event sequences. + [test_provider.py](/home/goodboy/repos/piker/wkts/replay_provider_e2e/tests/replay/test_provider.py:57) + exercises actual datad, sampler, search, pause/resume and SHM teardown. + [Existing E2E direction](/home/goodboy/repos/piker/wkts/replay_provider_e2e/plans/opencode/live-backend-chart-e2e-review.md:3) + prioritizes real public-provider chart journeys; replay is parked until + an observed failure needs controlled input. These disconnect races now + justify a narrow deterministic reproducer, not a replay-first rewrite. + +The live-chart harness reference still describes MainWindow's old SIGINT +behavior, while its branch has the structured lifecycle commit. Reconcile +that stale guidance with current source before selecting commands. + +## Stack being qualified + +- Evaluation Piker HEAD `eed58566226d5476b548c6dd3bb6a44fcd6325f9`, with + dirty modern-dependency and logger-compatibility changes. +- Tractor directory `/home/goodboy/repos/tractor/wkts/address_logging_levels`, + branch `wkt/ctx_final_return_disconnect`, HEAD `0553fcf8`. +- Tractor final-Return guard and endpoint-error control are uncommitted. + [Offline regression](/home/goodboy/repos/tractor/wkts/address_logging_levels/tests/test_context_final_return_disconnect.py) + directly drives real TCP and `_rpc._invoke()`, proves unrelated sibling + survival, and verifies a genuine endpoint-raised TransportClosed still + propagates. This is runtime regression coverage, not chart E2E coverage. +- Recorded focused checks: regression + RPC suite 10 passed; context + stream semantics 65 passed. These are earlier runs, not handoff execution. +- Evaluation runtime: Python 3.13.14, Trio 0.32.0, Qt 6.11.1, PyQt 6.11.0. +- Structured Qtractor lifecycle has NOT been transplanted to evaluation. + Baseline CLI SIGINT behavior must remain testable separately from the + lifecycle-remediated branch. + +## Required topology and observation contract + +Use a suite-owned registrar/pikerd A whose lifetime is independent of B +and C. Each chart must be a separate OS process with its own QApplication +and Tractor root. Do not run concurrent Tractor roots in pytest's process. +The existing in-process single-chart test is useful but does not provide +this persistent external-daemon topology. + +Give A a test-unique registrar address, propagate isolated config to ALL +children before imports, and bind charts to loopback TCP port zero. Pass +discovery and listener addresses separately. Verify actual chart listener +ports are nonzero and distinct, and all roots contact A. Do not copy the +human's registrar `6116`, fixed chart port `3333`, config or accounts. +The manual bind failure came from `[network].chart` explicitly pinning +3333; changing it to port zero let simultaneous charts boot. + +Proposed helper contracts below are implementation requirements, not +existing APIs. Adapt to the manager's current harness instead of adding +another application adapter: + +- A fixture owns A, chart subprocess handles, bounded deadlines, captured + output and an external observation/control channel that outlives charts. +- Capture service identity as actor name + UUID + PID + listener addresses. + Snapshot datad/sampler/EMS/paperboi or brokerd, and provider client IDs or + connection-generation counters when that backend exposes them. +- Identify charts and FSPs by full actor UID plus PID/parent ownership; + names alone are ambiguous with two `chart.fsp_0` actors. +- Readiness means feed/history attached, chart views bound, required FSP + streams active, search handlers ready and EMS entered. Race every wait + against process/session completion and preserve the original exception. +- After detach, perform a fresh service RPC and observe a post-detach + provider quote in the survivor. Process existence or sampler timestamps + alone do not establish service/feed health. +- For deterministic input, match a post-action sequence in feed, SHM and + chart/FSP consumption. For a live provider, await fresh market data with + a realistic deadline; do not require every quote to change price. +- Keep exact created SHM names (including child-created unpublished ones) + and child ownership events. Registry polling alone misses transient + replacement actors and allocations that never reach a ready chart. + +## Journey specifications + +Suggested names describe NEW tests, not existing runnable node IDs. + +### 1. `test_chart_attaches_to_external_pikerd` + +Start A, then B. Establish ready views, nonempty finite history, actual +fresh market updates, running FSPs and configured clearing mode. Record +baseline identities. Assert B did not create a competing registrar or +replacement data service. This setup is reused within each journey, +not across unrelated tests that could contaminate one another. + +### 2–3. `test_last_chart_sigint_then_repeated_reattach` + +With only B attached, send terminal-equivalent SIGINT, await its process +exit, then attach B2 and B3 in sequence to the same A. For each cycle: + +1. Inspect shutdown outcome, exit code/signal and FSP descendants. +2. Assert shared service identities stay constant and fresh RPCs succeed. +3. Assert reattached chart receives post-attach market data and FSP output. +4. Prove no replacement service/provider-client generation was created. + +Use a real process group created for B and its descendants to reproduce +terminal Ctrl-C delivery. Record PID/PGID membership before signaling; +A and the controller must be in different groups. A root-PID-only signal +is a separate variant and must not be called terminal-equivalent. +Run actual MainWindow close/structured shutdown as a separate variant; +calling `request_shutdown()` alone does not cover CLI SIGINT. + +### 4. `test_detach_with_search_results_visible_or_dismissed` + +Parameterize visible results versus dismissed pane. Drive production +search UI (QtBot in an owned chart worker), await a real query result, +then leave results visible or perform the normal dismiss interaction. +Detach, prove daemon/survivor health, reattach and search again. +Do not substitute merely opening the input field for visible results. +Both states have been manually exercised; no causal connection between +pane visibility and intermittent REPL entry has been established. + +### 5. `test_shared_daemon_survives_alternating_chart_departures` + +Parameterize same FQME and two distinct FQMEs under one provider. +Bring B and C fully live, then execute this order: + +1. Close B; assert the SAME C process consumes a post-close quote/FSP update. +2. Reopen B; establish its readiness and fresh data. +3. Close C; assert the SAME replacement B keeps receiving updates. +4. Close the final B, then reattach one chart and verify fresh updates. + +Same-market coverage must prove one departure did not end the shared +market subscription. Different-market coverage must prove per-feed +cleanup did not cancel a sibling feed. Shared feed SHM may legitimately +remain while A owns its cache: assert lifetime according to its owner, +not unconditional SHM disappearance after every chart exit. + +### 6. `test_chart_startup_cancel_preserves_running_consumer` + +Keep ready B and A. Start C and cancel at each gated phase: + +| Phase | Required evidence before cancellation | +| --- | --- | +| Early attachment | Chart actor registered; initial acquisition in flight | +| FSP child bootstrap | Child spawned; imports/registration not completed | +| History publication | History request entered; completion/publication held | +| FSP establishment | Cascade entered; readiness/stream establishment held | + +Use explicit entered/release acknowledgements at real boundaries. +Hold only the designated C operation; do not block the shared datad +event loop or stall B. Ensure cancellation can unwind a held operation. +Where needed use a narrow test seam/controlled provider for that exact +race; timestamps, sleeps and matching an `Entering ...` log are not +deterministic gates. A missed gate is not phase coverage. + +After each abort, prove B receives fresh data, A identities persist, +C-owned RPCs/children are gone, and a new C can fully start. Finally +exercise startup cancellation with no other chart attached to cover the +last-consumer case. Keep each phase's outcome distinct in the report. + +Add a debug-mode bootstrap-abort variant: terminal SIGINT can reach FSP +children during imports, before their IPC registration. The parent must +unwind spawn supervision without attempting an IPC context to itself for +stdio locking. Capture the original bootstrap interruption separately +from any secondary DebugRequestError. Do not classify this mixed failure +as a cancellation-only group or suppress it with cancellation filtering. + +### 7. `test_paper_order_state_survives_chart_reattach` + +Use real paper EMS with isolated writable ledger/account state and a +qualified feed. Submit an unfilled paper limit order, confirm its EMS +ID/state, detach its chart, reattach and verify the same order/state, +then cancel it and observe the terminal cancellation acknowledgement. +Check absence of duplicate submissions/replay, stable clearing service +identities and consistent position state. A startup order-mode widget +alone does not cover this. Credentialed IB paper-account qualification +is a separately selected backend tier, with unique gateway client IDs; +public-feed paper EMS does not prove IB brokerd reconnect behavior. + +Extend IB qualification to a broker-routed limit and an EMS-held dark +order, including a multi-expiry chart. Assert the EMS OID-to-broker reqid +mapping survives reattach, repeated status delivery is idempotent, +market-specific dialogs remain correctly scoped, cancel-all reaches each +applicable order, and terminal cancellation removes the correct dialog. +Repeated status logs alone do not prove duplicate broker submissions; +count canonical order records and broker working orders independently. + +### 7b. `test_cancel_during_second_chart_order_bootstrap` + +New human-observed sequence: B is already live; start C, then repeatedly +submit and immediately cancel orders in B while C is booting. C can +finish with a zombie order registered that is already cancelled in B +and EMS. The first observation resisted several retries; the human then +narrowed the trigger to repeated submit/cancel during C bootstrap. Exact +causative ordering and whether stale state is graphical or dialog-local +remain unproven. Do not substitute a normal fully-ready two-chart cancel. + +Gate C at the actual order snapshot/replay and live-subscription handoff, +keeping B and EMS responsive. Exercise terminal cancellation before, +during and after bootstrap delivery, with multiple distinct OIDs. Release +an older open snapshot after the terminal event in a controlled variant +to test the suspected ordering explicitly, not to assert it was observed. +Require C's canonical order state and graphics to converge to EMS's +terminal state, and remain absent/cancelled after deferred replay and +subsequent live events. Duplicate open delivery must not resurrect a +terminal order. Retain OID, broker reqid, bootstrap phase and event order +in failure output; cover broker-paper and EMS-held dark orders separately. + +## Failure assertions and diagnostics + +Every journey reports both isolation and exit outcomes even if one fails. +Continue collecting A/survivor evidence after chart failure, then fail +with aggregated original errors; avoid first-assertion cleanup masking. + +- Fail clean-exit checks on SIGSEGV/native abort, stuck descendants, + cancel-ack hard-reap, unexpected debugger entry, or cancellation-only + groups relayed as application failures. The native crash is unresolved. +- Capture chart return code, actor IDs, RPC/cid, parent/child relation, + cancellation source and source task, raw stdout/stderr, phase evidence, + SHM ownership, registry/service snapshots and imported code versions. +- Enable fault-handler/native crash evidence in owned subprocesses. + Retain a core identifier/backtrace if available; do not infer the native + fault's root cause solely from the last Python self-SIGINT traceback. +- A `Final context result delivery failed: peer disconnected` warning is + allowed at the guarded send boundary if the actor and sibling work + demonstrably survive. Warning absence does not mean a missed detach. +- Sampler StartAck/CancelAck delivery failure to a departed chart/FSP is + similarly evaluated by target ownership and subsequent sampler health. +- Do not blanket-suppress TransportClosed or BaseExceptionGroup. Mixed + groups and genuine endpoint errors must retain their failure behavior. +- Preserve both failing-baseline and fixed results. Do not label the + current whole journey green because the final-Return guard works. + +## Manual evidence available as of this handoff + +Human-operated observations, not automated results: + +- Last-consumer detach and repeated reattach reused datad UUID + `3d8b3632-2c90-4ad2-bde5-d75d87bf03af`, PID 742944, cached IB client 6132. +- 12:00:27: final-Return guard caught `open_symbol_search` delivery to + departing chart PID 1729338; sampler PID 743912 remained running. +- 12:02:10: B PID 1520477 exited; C PID 1757182 kept receiving live data. +- 12:02:45: replacement B PID 1771731 started with live data. +- 12:03:33: C PID 1757182 exited; human confirmed B remained unaffected. + Datad PID 742944 and sampler PID 743912 retained their identities. +- Same-MNQ alternating-consumer survival is observed. QQQ also booted + alongside MNQ; an equivalent full different-market closure sequence + has not been explicitly reported. +- FSP cancellation-only groups repeatedly became RemoteActorError; + self-SIGINT occurred during Qt close; MNQ exits at 12:00:27 and + 12:03:33 segfaulted. Chart exit is not clean. +- 14:41:26–31: chart `5d1517`, PID 3829973, aborted before order-mode + readiness. Two child bootstrap traces were interrupted during Tractor/ + Trio imports (`ctypes.util.find_library('pthread')`). Spawn cleanup then + attempted stdio locking through a local-actor IPC context, producing + `RuntimeError: Can not open an IPC ctx with the local actor!` and two + DebugRequestError leaves. The chart entered post-mortem. This is a new + bootstrap/debug-lock failure, not merely cancellation-only FSP noise. +- 14:42:32–37: MNQ chart `432913`, PID 3844804, aborted after BOTH + cascade startup profiler exits. This covers near-startup shutdown, + not the precise incomplete-FSP-establishment phase. +- 14:43:17–25: TSLA chart `6235d8`, PID 3855381, aborted while FSP + display startup was still active. Dolla-volume cascade had started and + resynced growing history; no second cascade startup is shown before + Ctrl-C. This is useful partial-FSP/backfill-overlap evidence, not a + deterministic phase gate. Human reports daemon backfill continued. +- TSLA SHM token names datad prefix `006f9855-1f2e-4d`, unlike the earlier + `3d8b3632-2c90-4a`. Human subsequently explained manual pikerd restarts + between network roams because IB roaming recovery remains incomplete. + Do not attribute cross-batch identity changes to chart detachment. +- 14:45:18: TSLA reattach `41a82c`, PID 3882846, reached both cascade + startup completions. Human reports cached-like loading and daemon + survival. Human subsequently confirmed B kept receiving fresh updates + during C aborts and recovered TSLA quotes/bars/indicators looked good. + Startup still emitted no-size/no-yrange rendering messages. +- 14:45:40: TSLA detach again emitted cancellation-only FSP groups and + MainWindow self-SIGINT. No native segfault is shown in these afternoon + excerpts. +- Order trial on `ib.algopaper`: broker-routed sell limit OID + `70fbf5f4-4d3b-41a4-a16f-af92d68e6aac`, reqid `5042`, size 1 at 29935, + `resp='open'`, `exec_mode='live'`. This is IB's paper account through + the live broker route, not Piker's paperboi execution mode. +- EMS-held sell dark order OID `9efd5800-7a50-4383-995b-f75d51d17853`, + size 1 at 29912.50, `resp='dark_open'`, no broker reqid. Human reports + both reappeared after chart reattach. Logs retain the same IDs/states. +- Reattached chart `b1326a`, PID 253598, at 15:27:37 displayed December + 2026 and March 2027 MNQ. Status delivery repeated, with untracked-dialog + and already-set-brokerd_msg warnings. No duplicate broker submission is + established by those repeated logs; replay/dialog correlation needs + focused assertions rather than blanket acceptance or suppression. +- Human used `cc` cancel-all. Initial attempts said no orders cancelled; + later broker details carried reqid `5042`, status `canceled`, filled 0, + remaining 1, but wrapped in `name='error'` / `oid=None`, then emitted + `Received cancel for unsubmitted order None`. Broker cancellation is + observed, clean correlated UI cancellation is not. No terminal dark + cancellation is shown. Human reports positions consistent, no fills. +- 15:29:39: chart PID 253598 segfaulted after Ctrl-C. Native core/GDB + evidence identifies SIP wrapper access during PyQt atexit cleanup. + [Native diagnostics](/home/goodboy/repos/piker/wkts/federated_project_stores/ai/diagnostics/20261019-chart-253598/README.md:1) + are linked from Taken source `piker`, task ID + `piker-qt6-chart-teardown-sigsegv` in the federated store worktree. +- 2026-10-20 repeat: chart `4af4a7`, PID 1080042, replayed broker OID + `6d78315a-0135-44b3-8162-bd7e816d0190`, reqid `4991`, and dark OID + `21f82034-f555-4cbe-a412-ff1a629e5379`. Cancellation again reported + broker status `canceled` wrapped as error / `oid=None`. Human then + restarted the chart and confirmed neither order returned. This is + reported post-cancel absence, not proof of clean status correlation. +- Subsequent second-chart bootstrap trial exposed the zombie-order race + specified in 7b. Preserve this finding alongside successful ordinary + chart restart/cancellation trials; one does not invalidate the other. + +Committed `wkt/fsp_backfill_sync` HEAD `749ca1f0` is an ancestor of the +evaluation baseline (verified with `git merge-base --is-ancestor`). That +does not establish inclusion of any dirty work in that other worktree or +coverage of Tractor process supervision/debug-lock failures. Whether an +older Tractor pin masked cancellation-group behavior rather than fixing +it requires historical reproduction; it is not established by these logs. + +## Suggested implementation order and delivery evidence + +First extend the installed-process topology and same-market alternating +journey; then last-consumer cycles and search-state variants; then gated +startup aborts and paper-order continuity. Reuse the lifecycle fixtures +and live-first direction already owned by the receiving agent. + +The manager should return exact implemented node IDs, fixture ownership, +phase-gate semantics, baseline/fixed test results, independent clean-exit +results and remaining backend-specific gaps. Run headless Qt with the +branch's evidenced flags; live-provider and IB selection remain explicit. +Before choosing the target environment, resolve Piker AND Tractor imports +and account for the evaluation's duplicate pytest `--ll` registration +blocker. No commands for nonexistent proposed nodes are supplied here. + +This handoff was written from source inspection and supplied live logs. +No live services, chart processes, orders or E2E suite were started by +the authoring agent for this drafting task. Delivery to the named dialog +has not been confirmed; the file is ready for that dialog to consume.