piker/ai/plans/20261019-chart-disconnect-e2e-handoff.md

# Chart disconnect / reattach E2E handoff

Recipient: `e2e_qa_mngr` dialog agent.
Date: 2026-10-19. Draft implementation specification, not executed tests.
Source: human-operated IB qualification in `wkt/ib_downstream_pin_eval`.
Authoring harness/model: OpenCode / `openai/gpt-6.1-sol`.

## Request and intended result

Human request: "so while i'm going through all dis, i feel like you
should be drafting up equivalent e2e tests to handoff to the
e2e_qa_mngr dlog agent. i know they're working on them testing branches
i likely referred to prior"

Extend the existing chart qualification work with executable journeys
equivalent to manual trials 1–7 below. Preserve existing task states.
Keep service-isolation and clean-exit assertions independently observable:
an unaffected surviving chart does not make a segfaulting exit successful.

## Existing foundations inspected

Paths below are local worktree snapshots; uncommitted files can change.
Verify HEAD, diff, imports and fixture contracts before implementation.

- `wkt/human_iface_e2e`, HEAD `aadfd1cf`: headless Qt harness foundation.
- `wkt/live_chart_qtractor`, HEAD `1a2d4893`: structured session lifecycle.
  Its current dirty work includes paper-mode startup forwarding, runtime
  overrides, and live chart tests; HEAD alone does not contain that work.
  [test_live_chart.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_live_chart.py:21)
  starts a production Kraken chart, checks real SHM/price changes and
  records exact actor/SHM ownership. It is explicitly selected through
  `--live-provider=kraken` and uses paper EMS without submitting orders.
  [test_qtractor.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_qtractor.py:73)
  covers sequential structured window closure and runtime restart.
  [UI fixtures](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/conftest.py:79)
  restore QApplication, widgets, ViewBox registries, config and settings.
  [open_test_pikerd](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/conftest.py:452)
  supplies isolated config propagation and the plugin registry fixture.
- `wkt/replay_provider_e2e`, HEAD `06fee700`: real replay feed/service
  tests with correlated command acknowledgements and event sequences.
  [test_provider.py](/home/goodboy/repos/piker/wkts/replay_provider_e2e/tests/replay/test_provider.py:57)
  exercises actual datad, sampler, search, pause/resume and SHM teardown.
  [Existing E2E direction](/home/goodboy/repos/piker/wkts/replay_provider_e2e/plans/opencode/live-backend-chart-e2e-review.md:3)
  prioritizes real public-provider chart journeys; replay is parked until
  an observed failure needs controlled input. These disconnect races now
  justify a narrow deterministic reproducer, not a replay-first rewrite.

The live-chart harness reference still describes MainWindow's old SIGINT
behavior, while its branch has the structured lifecycle commit. Reconcile
that stale guidance with current source before selecting commands.

## Stack being qualified

- Evaluation Piker HEAD `eed58566226d5476b548c6dd3bb6a44fcd6325f9`, with
  dirty modern-dependency and logger-compatibility changes.
- Tractor directory `/home/goodboy/repos/tractor/wkts/address_logging_levels`,
  branch `wkt/ctx_final_return_disconnect`, HEAD `0553fcf8`.
- Tractor final-Return guard and endpoint-error control are uncommitted.
  [Offline regression](/home/goodboy/repos/tractor/wkts/address_logging_levels/tests/test_context_final_return_disconnect.py)
  directly drives real TCP and `_rpc._invoke()`, proves unrelated sibling
  survival, and verifies a genuine endpoint-raised TransportClosed still
  propagates. This is runtime regression coverage, not chart E2E coverage.
- Recorded focused checks: regression + RPC suite 10 passed; context
  stream semantics 65 passed. These are earlier runs, not handoff execution.
- Evaluation runtime: Python 3.13.14, Trio 0.32.0, Qt 6.11.1, PyQt 6.11.0.
- Structured Qtractor lifecycle has NOT been transplanted to evaluation.
  Baseline CLI SIGINT behavior must remain testable separately from the
  lifecycle-remediated branch.

## Required topology and observation contract

Use a suite-owned registrar/pikerd A whose lifetime is independent of B
and C. Each chart must be a separate OS process with its own QApplication
and Tractor root. Do not run concurrent Tractor roots in pytest's process.
The existing in-process single-chart test is useful but does not provide
this persistent external-daemon topology.

Give A a test-unique registrar address, propagate isolated config to ALL
children before imports, and bind charts to loopback TCP port zero. Pass
discovery and listener addresses separately. Verify actual chart listener
ports are nonzero and distinct, and all roots contact A. Do not copy the
human's registrar `6116`, fixed chart port `3333`, config or accounts.
The manual bind failure came from `[network].chart` explicitly pinning
3333; changing it to port zero let simultaneous charts boot.

Proposed helper contracts below are implementation requirements, not
existing APIs. Adapt to the manager's current harness instead of adding
another application adapter:

- A fixture owns A, chart subprocess handles, bounded deadlines, captured
  output and an external observation/control channel that outlives charts.
- Capture service identity as actor name + UUID + PID + listener addresses.
  Snapshot datad/sampler/EMS/paperboi or brokerd, and provider client IDs or
  connection-generation counters when that backend exposes them.
- Identify charts and FSPs by full actor UID plus PID/parent ownership;
  names alone are ambiguous with two `chart.fsp_0` actors.
- Readiness means feed/history attached, chart views bound, required FSP
  streams active, search handlers ready and EMS entered. Race every wait
  against process/session completion and preserve the original exception.
- After detach, perform a fresh service RPC and observe a post-detach
  provider quote in the survivor. Process existence or sampler timestamps
  alone do not establish service/feed health.
- For deterministic input, match a post-action sequence in feed, SHM and
  chart/FSP consumption. For a live provider, await fresh market data with
  a realistic deadline; do not require every quote to change price.
- Keep exact created SHM names (including child-created unpublished ones)
  and child ownership events. Registry polling alone misses transient
  replacement actors and allocations that never reach a ready chart.

## Journey specifications

Suggested names describe NEW tests, not existing runnable node IDs.

### 1. `test_chart_attaches_to_external_pikerd`

Start A, then B. Establish ready views, nonempty finite history, actual
fresh market updates, running FSPs and configured clearing mode. Record
baseline identities. Assert B did not create a competing registrar or
replacement data service. This setup is reused within each journey,
not across unrelated tests that could contaminate one another.

### 2–3. `test_last_chart_sigint_then_repeated_reattach`

With only B attached, send terminal-equivalent SIGINT, await its process
exit, then attach B2 and B3 in sequence to the same A. For each cycle:

1. Inspect shutdown outcome, exit code/signal and FSP descendants.
2. Assert shared service identities stay constant and fresh RPCs succeed.
3. Assert reattached chart receives post-attach market data and FSP output.
4. Prove no replacement service/provider-client generation was created.

Use a real process group created for B and its descendants to reproduce
terminal Ctrl-C delivery. Record PID/PGID membership before signaling;
A and the controller must be in different groups. A root-PID-only signal
is a separate variant and must not be called terminal-equivalent.
Run actual MainWindow close/structured shutdown as a separate variant;
calling `request_shutdown()` alone does not cover CLI SIGINT.

### 4. `test_detach_with_search_results_visible_or_dismissed`

Parameterize visible results versus dismissed pane. Drive production
search UI (QtBot in an owned chart worker), await a real query result,
then leave results visible or perform the normal dismiss interaction.
Detach, prove daemon/survivor health, reattach and search again.
Do not substitute merely opening the input field for visible results.
Both states have been manually exercised; no causal connection between
pane visibility and intermittent REPL entry has been established.

### 5. `test_shared_daemon_survives_alternating_chart_departures`

Parameterize same FQME and two distinct FQMEs under one provider.
Bring B and C fully live, then execute this order:

1. Close B; assert the SAME C process consumes a post-close quote/FSP update.
2. Reopen B; establish its readiness and fresh data.
3. Close C; assert the SAME replacement B keeps receiving updates.
4. Close the final B, then reattach one chart and verify fresh updates.

Same-market coverage must prove one departure did not end the shared
market subscription. Different-market coverage must prove per-feed
cleanup did not cancel a sibling feed. Shared feed SHM may legitimately
remain while A owns its cache: assert lifetime according to its owner,
not unconditional SHM disappearance after every chart exit.

### 6. `test_chart_startup_cancel_preserves_running_consumer`

Keep ready B and A. Start C and cancel at each gated phase:

| Phase | Required evidence before cancellation |
| --- | --- |
| Early attachment | Chart actor registered; initial acquisition in flight |
| FSP child bootstrap | Child spawned; imports/registration not completed |
| History publication | History request entered; completion/publication held |
| FSP establishment | Cascade entered; readiness/stream establishment held |

Use explicit entered/release acknowledgements at real boundaries.
Hold only the designated C operation; do not block the shared datad
event loop or stall B. Ensure cancellation can unwind a held operation.
Where needed use a narrow test seam/controlled provider for that exact
race; timestamps, sleeps and matching an `Entering ...` log are not
deterministic gates. A missed gate is not phase coverage.

After each abort, prove B receives fresh data, A identities persist,
C-owned RPCs/children are gone, and a new C can fully start. Finally
exercise startup cancellation with no other chart attached to cover the
last-consumer case. Keep each phase's outcome distinct in the report.

Add a debug-mode bootstrap-abort variant: terminal SIGINT can reach FSP
children during imports, before their IPC registration. The parent must
unwind spawn supervision without attempting an IPC context to itself for
stdio locking. Capture the original bootstrap interruption separately
from any secondary DebugRequestError. Do not classify this mixed failure
as a cancellation-only group or suppress it with cancellation filtering.

### 7. `test_paper_order_state_survives_chart_reattach`

Use real paper EMS with isolated writable ledger/account state and a
qualified feed. Submit an unfilled paper limit order, confirm its EMS
ID/state, detach its chart, reattach and verify the same order/state,
then cancel it and observe the terminal cancellation acknowledgement.
Check absence of duplicate submissions/replay, stable clearing service
identities and consistent position state. A startup order-mode widget
alone does not cover this. Credentialed IB paper-account qualification
is a separately selected backend tier, with unique gateway client IDs;
public-feed paper EMS does not prove IB brokerd reconnect behavior.

Extend IB qualification to a broker-routed limit and an EMS-held dark
order, including a multi-expiry chart. Assert the EMS OID-to-broker reqid
mapping survives reattach, repeated status delivery is idempotent,
market-specific dialogs remain correctly scoped, cancel-all reaches each
applicable order, and terminal cancellation removes the correct dialog.
Repeated status logs alone do not prove duplicate broker submissions;
count canonical order records and broker working orders independently.

### 7b. `test_cancel_during_second_chart_order_bootstrap`

New human-observed sequence: B is already live; start C, then repeatedly
submit and immediately cancel orders in B while C is booting. C can
finish with a zombie order registered that is already cancelled in B
and EMS. The first observation resisted several retries; the human then
narrowed the trigger to repeated submit/cancel during C bootstrap. Exact
causative ordering and whether stale state is graphical or dialog-local
remain unproven. Do not substitute a normal fully-ready two-chart cancel.

Gate C at the actual order snapshot/replay and live-subscription handoff,
keeping B and EMS responsive. Exercise terminal cancellation before,
during and after bootstrap delivery, with multiple distinct OIDs. Release
an older open snapshot after the terminal event in a controlled variant
to test the suspected ordering explicitly, not to assert it was observed.
Require C's canonical order state and graphics to converge to EMS's
terminal state, and remain absent/cancelled after deferred replay and
subsequent live events. Duplicate open delivery must not resurrect a
terminal order. Retain OID, broker reqid, bootstrap phase and event order
in failure output; cover broker-paper and EMS-held dark orders separately.

## Failure assertions and diagnostics

Every journey reports both isolation and exit outcomes even if one fails.
Continue collecting A/survivor evidence after chart failure, then fail
with aggregated original errors; avoid first-assertion cleanup masking.

- Fail clean-exit checks on SIGSEGV/native abort, stuck descendants,
  cancel-ack hard-reap, unexpected debugger entry, or cancellation-only
  groups relayed as application failures. The native crash is unresolved.
- Capture chart return code, actor IDs, RPC/cid, parent/child relation,
  cancellation source and source task, raw stdout/stderr, phase evidence,
  SHM ownership, registry/service snapshots and imported code versions.
- Enable fault-handler/native crash evidence in owned subprocesses.
  Retain a core identifier/backtrace if available; do not infer the native
  fault's root cause solely from the last Python self-SIGINT traceback.
- A `Final context result delivery failed: peer disconnected` warning is
  allowed at the guarded send boundary if the actor and sibling work
  demonstrably survive. Warning absence does not mean a missed detach.
- Sampler StartAck/CancelAck delivery failure to a departed chart/FSP is
  similarly evaluated by target ownership and subsequent sampler health.
- Do not blanket-suppress TransportClosed or BaseExceptionGroup. Mixed
  groups and genuine endpoint errors must retain their failure behavior.
- Preserve both failing-baseline and fixed results. Do not label the
  current whole journey green because the final-Return guard works.

## Manual evidence available as of this handoff

Human-operated observations, not automated results:

- Last-consumer detach and repeated reattach reused datad UUID
  `3d8b3632-2c90-4ad2-bde5-d75d87bf03af`, PID 742944, cached IB client 6132.
- 12:00:27: final-Return guard caught `open_symbol_search` delivery to
  departing chart PID 1729338; sampler PID 743912 remained running.
- 12:02:10: B PID 1520477 exited; C PID 1757182 kept receiving live data.
- 12:02:45: replacement B PID 1771731 started with live data.
- 12:03:33: C PID 1757182 exited; human confirmed B remained unaffected.
  Datad PID 742944 and sampler PID 743912 retained their identities.
- Same-MNQ alternating-consumer survival is observed. QQQ also booted
  alongside MNQ; an equivalent full different-market closure sequence
  has not been explicitly reported.
- FSP cancellation-only groups repeatedly became RemoteActorError;
  self-SIGINT occurred during Qt close; MNQ exits at 12:00:27 and
  12:03:33 segfaulted. Chart exit is not clean.
- 14:41:26–31: chart `5d1517`, PID 3829973, aborted before order-mode
  readiness. Two child bootstrap traces were interrupted during Tractor/
  Trio imports (`ctypes.util.find_library('pthread')`). Spawn cleanup then
  attempted stdio locking through a local-actor IPC context, producing
  `RuntimeError: Can not open an IPC ctx with the local actor!` and two
  DebugRequestError leaves. The chart entered post-mortem. This is a new
  bootstrap/debug-lock failure, not merely cancellation-only FSP noise.
- 14:42:32–37: MNQ chart `432913`, PID 3844804, aborted after BOTH
  cascade startup profiler exits. This covers near-startup shutdown,
  not the precise incomplete-FSP-establishment phase.
- 14:43:17–25: TSLA chart `6235d8`, PID 3855381, aborted while FSP
  display startup was still active. Dolla-volume cascade had started and
  resynced growing history; no second cascade startup is shown before
  Ctrl-C. This is useful partial-FSP/backfill-overlap evidence, not a
  deterministic phase gate. Human reports daemon backfill continued.
- TSLA SHM token names datad prefix `006f9855-1f2e-4d`, unlike the earlier
  `3d8b3632-2c90-4a`. Human subsequently explained manual pikerd restarts
  between network roams because IB roaming recovery remains incomplete.
  Do not attribute cross-batch identity changes to chart detachment.
- 14:45:18: TSLA reattach `41a82c`, PID 3882846, reached both cascade
  startup completions. Human reports cached-like loading and daemon
  survival. Human subsequently confirmed B kept receiving fresh updates
  during C aborts and recovered TSLA quotes/bars/indicators looked good.
  Startup still emitted no-size/no-yrange rendering messages.
- 14:45:40: TSLA detach again emitted cancellation-only FSP groups and
  MainWindow self-SIGINT. No native segfault is shown in these afternoon
  excerpts.
- Order trial on `ib.algopaper`: broker-routed sell limit OID
  `70fbf5f4-4d3b-41a4-a16f-af92d68e6aac`, reqid `5042`, size 1 at 29935,
  `resp='open'`, `exec_mode='live'`. This is IB's paper account through
  the live broker route, not Piker's paperboi execution mode.
- EMS-held sell dark order OID `9efd5800-7a50-4383-995b-f75d51d17853`,
  size 1 at 29912.50, `resp='dark_open'`, no broker reqid. Human reports
  both reappeared after chart reattach. Logs retain the same IDs/states.
- Reattached chart `b1326a`, PID 253598, at 15:27:37 displayed December
  2026 and March 2027 MNQ. Status delivery repeated, with untracked-dialog
  and already-set-brokerd_msg warnings. No duplicate broker submission is
  established by those repeated logs; replay/dialog correlation needs
  focused assertions rather than blanket acceptance or suppression.
- Human used `cc` cancel-all. Initial attempts said no orders cancelled;
  later broker details carried reqid `5042`, status `canceled`, filled 0,
  remaining 1, but wrapped in `name='error'` / `oid=None`, then emitted
  `Received cancel for unsubmitted order None`. Broker cancellation is
  observed, clean correlated UI cancellation is not. No terminal dark
  cancellation is shown. Human reports positions consistent, no fills.
- 15:29:39: chart PID 253598 segfaulted after Ctrl-C. Native core/GDB
  evidence identifies SIP wrapper access during PyQt atexit cleanup.
  [Native diagnostics](/home/goodboy/repos/piker/wkts/federated_project_stores/ai/diagnostics/20261019-chart-253598/README.md:1)
  are linked from Taken source `piker`, task ID
  `piker-qt6-chart-teardown-sigsegv` in the federated store worktree.
- 2026-10-20 repeat: chart `4af4a7`, PID 1080042, replayed broker OID
  `6d78315a-0135-44b3-8162-bd7e816d0190`, reqid `4991`, and dark OID
  `21f82034-f555-4cbe-a412-ff1a629e5379`. Cancellation again reported
  broker status `canceled` wrapped as error / `oid=None`. Human then
  restarted the chart and confirmed neither order returned. This is
  reported post-cancel absence, not proof of clean status correlation.
- Subsequent second-chart bootstrap trial exposed the zombie-order race
  specified in 7b. Preserve this finding alongside successful ordinary
  chart restart/cancellation trials; one does not invalidate the other.

Committed `wkt/fsp_backfill_sync` HEAD `749ca1f0` is an ancestor of the
evaluation baseline (verified with `git merge-base --is-ancestor`). That
does not establish inclusion of any dirty work in that other worktree or
coverage of Tractor process supervision/debug-lock failures. Whether an
older Tractor pin masked cancellation-group behavior rather than fixing
it requires historical reproduction; it is not established by these logs.

## Suggested implementation order and delivery evidence

First extend the installed-process topology and same-market alternating
journey; then last-consumer cycles and search-state variants; then gated
startup aborts and paper-order continuity. Reuse the lifecycle fixtures
and live-first direction already owned by the receiving agent.

The manager should return exact implemented node IDs, fixture ownership,
phase-gate semantics, baseline/fixed test results, independent clean-exit
results and remaining backend-specific gaps. Run headless Qt with the
branch's evidenced flags; live-provider and IB selection remain explicit.
Before choosing the target environment, resolve Piker AND Tractor imports
and account for the evaluation's duplicate pytest `--ll` registration
blocker. No commands for nonexistent proposed nodes are supplied here.

This handoff was written from source inspection and supplied live logs.
No live services, chart processes, orders or E2E suite were started by
the authoring agent for this drafting task. Delivery to the named dialog
has not been confirmed; the file is ready for that dialog to consume.