21 KiB

Raw Blame History

Chart disconnect / reattach E2E handoff

Recipient: e2e_qa_mngr dialog agent. Date: 2026-10-19. Draft implementation specification, not executed tests. Source: human-operated IB qualification in wkt/ib_downstream_pin_eval. Authoring harness/model: OpenCode / openai/gpt-6.1-sol.

Request and intended result

Human request: “so while i’m going through all dis, i feel like you should be drafting up equivalent e2e tests to handoff to the e2e_qa_mngr dlog agent. i know they’re working on them testing branches i likely referred to prior”

Extend the existing chart qualification work with executable journeys equivalent to manual trials 1–7 below. Preserve existing task states. Keep service-isolation and clean-exit assertions independently observable: an unaffected surviving chart does not make a segfaulting exit successful.

Existing foundations inspected

Paths below are local worktree snapshots; uncommitted files can change. Verify HEAD, diff, imports and fixture contracts before implementation.

wkt/human_iface_e2e, HEAD aadfd1cf: headless Qt harness foundation.
wkt/live_chart_qtractor, HEAD 1a2d4893: structured session lifecycle. Its current dirty work includes paper-mode startup forwarding, runtime overrides, and live chart tests; HEAD alone does not contain that work. test_live_chart.py starts a production Kraken chart, checks real SHM/price changes and records exact actor/SHM ownership. It is explicitly selected through --live-provider=kraken and uses paper EMS without submitting orders. test_qtractor.py covers sequential structured window closure and runtime restart. UI fixtures restore QApplication, widgets, ViewBox registries, config and settings. open_test_pikerd supplies isolated config propagation and the plugin registry fixture.
wkt/replay_provider_e2e, HEAD 06fee700: real replay feed/service tests with correlated command acknowledgements and event sequences. test_provider.py exercises actual datad, sampler, search, pause/resume and SHM teardown. Existing E2E direction prioritizes real public-provider chart journeys; replay is parked until an observed failure needs controlled input. These disconnect races now justify a narrow deterministic reproducer, not a replay-first rewrite.

The live-chart harness reference still describes MainWindow’s old SIGINT behavior, while its branch has the structured lifecycle commit. Reconcile that stale guidance with current source before selecting commands.

Stack being qualified

Evaluation Piker HEAD eed58566226d5476b548c6dd3bb6a44fcd6325f9, with dirty modern-dependency and logger-compatibility changes.
Tractor directory /home/goodboy/repos/tractor/wkts/address_logging_levels, branch wkt/ctx_final_return_disconnect, HEAD 0553fcf8.
Tractor final-Return guard and endpoint-error control are uncommitted. Offline regression directly drives real TCP and _rpc._invoke(), proves unrelated sibling survival, and verifies a genuine endpoint-raised TransportClosed still propagates. This is runtime regression coverage, not chart E2E coverage.
Recorded focused checks: regression + RPC suite 10 passed; context stream semantics 65 passed. These are earlier runs, not handoff execution.
Evaluation runtime: Python 3.13.14, Trio 0.32.0, Qt 6.11.1, PyQt 6.11.0.
Structured Qtractor lifecycle has NOT been transplanted to evaluation. Baseline CLI SIGINT behavior must remain testable separately from the lifecycle-remediated branch.

Required topology and observation contract

Use a suite-owned registrar/pikerd A whose lifetime is independent of B and C. Each chart must be a separate OS process with its own QApplication and Tractor root. Do not run concurrent Tractor roots in pytest’s process. The existing in-process single-chart test is useful but does not provide this persistent external-daemon topology.

Give A a test-unique registrar address, propagate isolated config to ALL children before imports, and bind charts to loopback TCP port zero. Pass discovery and listener addresses separately. Verify actual chart listener ports are nonzero and distinct, and all roots contact A. Do not copy the human’s registrar 6116, fixed chart port 3333, config or accounts. The manual bind failure came from [network].chart explicitly pinning 3333; changing it to port zero let simultaneous charts boot.

Proposed helper contracts below are implementation requirements, not existing APIs. Adapt to the manager’s current harness instead of adding another application adapter:

A fixture owns A, chart subprocess handles, bounded deadlines, captured output and an external observation/control channel that outlives charts.
Capture service identity as actor name + UUID + PID + listener addresses. Snapshot datad/sampler/EMS/paperboi or brokerd, and provider client IDs or connection-generation counters when that backend exposes them.
Identify charts and FSPs by full actor UID plus PID/parent ownership; names alone are ambiguous with two chart.fsp_0 actors.
Readiness means feed/history attached, chart views bound, required FSP streams active, search handlers ready and EMS entered. Race every wait against process/session completion and preserve the original exception.
After detach, perform a fresh service RPC and observe a post-detach provider quote in the survivor. Process existence or sampler timestamps alone do not establish service/feed health.
For deterministic input, match a post-action sequence in feed, SHM and chart/FSP consumption. For a live provider, await fresh market data with a realistic deadline; do not require every quote to change price.
Keep exact created SHM names (including child-created unpublished ones) and child ownership events. Registry polling alone misses transient replacement actors and allocations that never reach a ready chart.

Journey specifications

Suggested names describe NEW tests, not existing runnable node IDs.

1. `test_chart_attaches_to_external_pikerd`

Start A, then B. Establish ready views, nonempty finite history, actual fresh market updates, running FSPs and configured clearing mode. Record baseline identities. Assert B did not create a competing registrar or replacement data service. This setup is reused within each journey, not across unrelated tests that could contaminate one another.

2–3. `test_last_chart_sigint_then_repeated_reattach`

With only B attached, send terminal-equivalent SIGINT, await its process exit, then attach B2 and B3 in sequence to the same A. For each cycle:

Inspect shutdown outcome, exit code/signal and FSP descendants.
Assert shared service identities stay constant and fresh RPCs succeed.
Assert reattached chart receives post-attach market data and FSP output.
Prove no replacement service/provider-client generation was created.

Use a real process group created for B and its descendants to reproduce terminal Ctrl-C delivery. Record PID/PGID membership before signaling; A and the controller must be in different groups. A root-PID-only signal is a separate variant and must not be called terminal-equivalent. Run actual MainWindow close/structured shutdown as a separate variant; calling request_shutdown() alone does not cover CLI SIGINT.

4. `test_detach_with_search_results_visible_or_dismissed`

Parameterize visible results versus dismissed pane. Drive production search UI (QtBot in an owned chart worker), await a real query result, then leave results visible or perform the normal dismiss interaction. Detach, prove daemon/survivor health, reattach and search again. Do not substitute merely opening the input field for visible results. Both states have been manually exercised; no causal connection between pane visibility and intermittent REPL entry has been established.

5. `test_shared_daemon_survives_alternating_chart_departures`

Parameterize same FQME and two distinct FQMEs under one provider. Bring B and C fully live, then execute this order:

Close B; assert the SAME C process consumes a post-close quote/FSP update.
Reopen B; establish its readiness and fresh data.
Close C; assert the SAME replacement B keeps receiving updates.
Close the final B, then reattach one chart and verify fresh updates.

Same-market coverage must prove one departure did not end the shared market subscription. Different-market coverage must prove per-feed cleanup did not cancel a sibling feed. Shared feed SHM may legitimately remain while A owns its cache: assert lifetime according to its owner, not unconditional SHM disappearance after every chart exit.

6. `test_chart_startup_cancel_preserves_running_consumer`

Keep ready B and A. Start C and cancel at each gated phase:

Phase	Required evidence before cancellation
Early attachment	Chart actor registered; initial acquisition in flight
FSP child bootstrap	Child spawned; imports/registration not completed
History publication	History request entered; completion/publication held
FSP establishment	Cascade entered; readiness/stream establishment held

Use explicit entered/release acknowledgements at real boundaries. Hold only the designated C operation; do not block the shared datad event loop or stall B. Ensure cancellation can unwind a held operation. Where needed use a narrow test seam/controlled provider for that exact race; timestamps, sleeps and matching an Entering ... log are not deterministic gates. A missed gate is not phase coverage.

After each abort, prove B receives fresh data, A identities persist, C-owned RPCs/children are gone, and a new C can fully start. Finally exercise startup cancellation with no other chart attached to cover the last-consumer case. Keep each phase’s outcome distinct in the report.

Add a debug-mode bootstrap-abort variant: terminal SIGINT can reach FSP children during imports, before their IPC registration. The parent must unwind spawn supervision without attempting an IPC context to itself for stdio locking. Capture the original bootstrap interruption separately from any secondary DebugRequestError. Do not classify this mixed failure as a cancellation-only group or suppress it with cancellation filtering.

7. `test_paper_order_state_survives_chart_reattach`

Use real paper EMS with isolated writable ledger/account state and a qualified feed. Submit an unfilled paper limit order, confirm its EMS ID/state, detach its chart, reattach and verify the same order/state, then cancel it and observe the terminal cancellation acknowledgement. Check absence of duplicate submissions/replay, stable clearing service identities and consistent position state. A startup order-mode widget alone does not cover this. Credentialed IB paper-account qualification is a separately selected backend tier, with unique gateway client IDs; public-feed paper EMS does not prove IB brokerd reconnect behavior.

Extend IB qualification to a broker-routed limit and an EMS-held dark order, including a multi-expiry chart. Assert the EMS OID-to-broker reqid mapping survives reattach, repeated status delivery is idempotent, market-specific dialogs remain correctly scoped, cancel-all reaches each applicable order, and terminal cancellation removes the correct dialog. Repeated status logs alone do not prove duplicate broker submissions; count canonical order records and broker working orders independently.

7b. `test_cancel_during_second_chart_order_bootstrap`

New human-observed sequence: B is already live; start C, then repeatedly submit and immediately cancel orders in B while C is booting. C can finish with a zombie order registered that is already cancelled in B and EMS. The first observation resisted several retries; the human then narrowed the trigger to repeated submit/cancel during C bootstrap. Exact causative ordering and whether stale state is graphical or dialog-local remain unproven. Do not substitute a normal fully-ready two-chart cancel.

Gate C at the actual order snapshot/replay and live-subscription handoff, keeping B and EMS responsive. Exercise terminal cancellation before, during and after bootstrap delivery, with multiple distinct OIDs. Release an older open snapshot after the terminal event in a controlled variant to test the suspected ordering explicitly, not to assert it was observed. Require C’s canonical order state and graphics to converge to EMS’s terminal state, and remain absent/cancelled after deferred replay and subsequent live events. Duplicate open delivery must not resurrect a terminal order. Retain OID, broker reqid, bootstrap phase and event order in failure output; cover broker-paper and EMS-held dark orders separately.

Failure assertions and diagnostics

Every journey reports both isolation and exit outcomes even if one fails. Continue collecting A/survivor evidence after chart failure, then fail with aggregated original errors; avoid first-assertion cleanup masking.

Fail clean-exit checks on SIGSEGV/native abort, stuck descendants, cancel-ack hard-reap, unexpected debugger entry, or cancellation-only groups relayed as application failures. The native crash is unresolved.
Capture chart return code, actor IDs, RPC/cid, parent/child relation, cancellation source and source task, raw stdout/stderr, phase evidence, SHM ownership, registry/service snapshots and imported code versions.
Enable fault-handler/native crash evidence in owned subprocesses. Retain a core identifier/backtrace if available; do not infer the native fault’s root cause solely from the last Python self-SIGINT traceback.
A Final context result delivery failed: peer disconnected warning is allowed at the guarded send boundary if the actor and sibling work demonstrably survive. Warning absence does not mean a missed detach.
Sampler StartAck/CancelAck delivery failure to a departed chart/FSP is similarly evaluated by target ownership and subsequent sampler health.
Do not blanket-suppress TransportClosed or BaseExceptionGroup. Mixed groups and genuine endpoint errors must retain their failure behavior.
Preserve both failing-baseline and fixed results. Do not label the current whole journey green because the final-Return guard works.

Manual evidence available as of this handoff

Human-operated observations, not automated results:

Last-consumer detach and repeated reattach reused datad UUID 3d8b3632-2c90-4ad2-bde5-d75d87bf03af, PID 742944, cached IB client 6132.
12:00:27: final-Return guard caught open_symbol_search delivery to departing chart PID 1729338; sampler PID 743912 remained running.
12:02:10: B PID 1520477 exited; C PID 1757182 kept receiving live data.
12:02:45: replacement B PID 1771731 started with live data.
12:03:33: C PID 1757182 exited; human confirmed B remained unaffected. Datad PID 742944 and sampler PID 743912 retained their identities.
Same-MNQ alternating-consumer survival is observed. QQQ also booted alongside MNQ; an equivalent full different-market closure sequence has not been explicitly reported.
FSP cancellation-only groups repeatedly became RemoteActorError; self-SIGINT occurred during Qt close; MNQ exits at 12:00:27 and 12:03:33 segfaulted. Chart exit is not clean.
14:41:26–31: chart 5d1517, PID 3829973, aborted before order-mode readiness. Two child bootstrap traces were interrupted during Tractor/ Trio imports (ctypes.util.find_library('pthread')). Spawn cleanup then attempted stdio locking through a local-actor IPC context, producing RuntimeError: Can not open an IPC ctx with the local actor! and two DebugRequestError leaves. The chart entered post-mortem. This is a new bootstrap/debug-lock failure, not merely cancellation-only FSP noise.
14:42:32–37: MNQ chart 432913, PID 3844804, aborted after BOTH cascade startup profiler exits. This covers near-startup shutdown, not the precise incomplete-FSP-establishment phase.
14:43:17–25: TSLA chart 6235d8, PID 3855381, aborted while FSP display startup was still active. Dolla-volume cascade had started and resynced growing history; no second cascade startup is shown before Ctrl-C. This is useful partial-FSP/backfill-overlap evidence, not a deterministic phase gate. Human reports daemon backfill continued.
TSLA SHM token names datad prefix 006f9855-1f2e-4d, unlike the earlier 3d8b3632-2c90-4a. Human subsequently explained manual pikerd restarts between network roams because IB roaming recovery remains incomplete. Do not attribute cross-batch identity changes to chart detachment.
14:45:18: TSLA reattach 41a82c, PID 3882846, reached both cascade startup completions. Human reports cached-like loading and daemon survival. Human subsequently confirmed B kept receiving fresh updates during C aborts and recovered TSLA quotes/bars/indicators looked good. Startup still emitted no-size/no-yrange rendering messages.
14:45:40: TSLA detach again emitted cancellation-only FSP groups and MainWindow self-SIGINT. No native segfault is shown in these afternoon excerpts.
Order trial on ib.algopaper: broker-routed sell limit OID 70fbf5f4-4d3b-41a4-a16f-af92d68e6aac, reqid 5042, size 1 at 29935, resp='open', exec_mode='live'. This is IB’s paper account through the live broker route, not Piker’s paperboi execution mode.
EMS-held sell dark order OID 9efd5800-7a50-4383-995b-f75d51d17853, size 1 at 29912.50, resp='dark_open', no broker reqid. Human reports both reappeared after chart reattach. Logs retain the same IDs/states.
Reattached chart b1326a, PID 253598, at 15:27:37 displayed December 2026 and March 2027 MNQ. Status delivery repeated, with untracked-dialog and already-set-brokerd_msg warnings. No duplicate broker submission is established by those repeated logs; replay/dialog correlation needs focused assertions rather than blanket acceptance or suppression.
Human used cc cancel-all. Initial attempts said no orders cancelled; later broker details carried reqid 5042, status canceled, filled 0, remaining 1, but wrapped in name='error' / oid=None, then emitted Received cancel for unsubmitted order None. Broker cancellation is observed, clean correlated UI cancellation is not. No terminal dark cancellation is shown. Human reports positions consistent, no fills.
15:29:39: chart PID 253598 segfaulted after Ctrl-C. Native core/GDB evidence identifies SIP wrapper access during PyQt atexit cleanup. Native diagnostics are linked from Taken source piker, task ID piker-qt6-chart-teardown-sigsegv in the federated store worktree.
2026-10-20 repeat: chart 4af4a7, PID 1080042, replayed broker OID 6d78315a-0135-44b3-8162-bd7e816d0190, reqid 4991, and dark OID 21f82034-f555-4cbe-a412-ff1a629e5379. Cancellation again reported broker status canceled wrapped as error / oid=None. Human then restarted the chart and confirmed neither order returned. This is reported post-cancel absence, not proof of clean status correlation.
Subsequent second-chart bootstrap trial exposed the zombie-order race specified in 7b. Preserve this finding alongside successful ordinary chart restart/cancellation trials; one does not invalidate the other.

Committed wkt/fsp_backfill_sync HEAD 749ca1f0 is an ancestor of the evaluation baseline (verified with git merge-base --is-ancestor). That does not establish inclusion of any dirty work in that other worktree or coverage of Tractor process supervision/debug-lock failures. Whether an older Tractor pin masked cancellation-group behavior rather than fixing it requires historical reproduction; it is not established by these logs.

Suggested implementation order and delivery evidence

First extend the installed-process topology and same-market alternating journey; then last-consumer cycles and search-state variants; then gated startup aborts and paper-order continuity. Reuse the lifecycle fixtures and live-first direction already owned by the receiving agent.

The manager should return exact implemented node IDs, fixture ownership, phase-gate semantics, baseline/fixed test results, independent clean-exit results and remaining backend-specific gaps. Run headless Qt with the branch’s evidenced flags; live-provider and IB selection remain explicit. Before choosing the target environment, resolve Piker AND Tractor imports and account for the evaluation’s duplicate pytest --ll registration blocker. No commands for nonexistent proposed nodes are supplied here.

This handoff was written from source inspection and supplied live logs. No live services, chart processes, orders or E2E suite were started by the authoring agent for this drafting task. Delivery to the named dialog has not been confirmed; the file is ready for that dialog to consume.

21 KiB Raw Blame History Unescape Escape