Record chart disconnect E2E qualification handoff
Preserve the manual IB detach, reattach, shared-consumer and startup cancellation evidence as a historical snapshot for the QA worker. Specify independent service ownership and deterministic phase gates, including terminal-order resurrection during second-chart bootstrap. Keep service continuity distinct from clean chart exit and retain the unresolved cancellation, debugger, order-correlation and native-crash findings. The linked Taken tasks own the worker assignments; this document provides supporting context, not acceptance or executed tests. (this patch was generated in some part by `opencode` using `gpt-6.1-sol` (`openai`))wkt/ib_downstream_pin_eval
parent
29752c509c
commit
e929fdc2d5
|
|
@ -0,0 +1,364 @@
|
|||
# Chart disconnect / reattach E2E handoff
|
||||
|
||||
Recipient: `e2e_qa_mngr` dialog agent.
|
||||
Date: 2026-10-19. Draft implementation specification, not executed tests.
|
||||
Source: human-operated IB qualification in `wkt/ib_downstream_pin_eval`.
|
||||
Authoring harness/model: OpenCode / `openai/gpt-6.1-sol`.
|
||||
|
||||
## Request and intended result
|
||||
|
||||
Human request: "so while i'm going through all dis, i feel like you
|
||||
should be drafting up equivalent e2e tests to handoff to the
|
||||
e2e_qa_mngr dlog agent. i know they're working on them testing branches
|
||||
i likely referred to prior"
|
||||
|
||||
Extend the existing chart qualification work with executable journeys
|
||||
equivalent to manual trials 1–7 below. Preserve existing task states.
|
||||
Keep service-isolation and clean-exit assertions independently observable:
|
||||
an unaffected surviving chart does not make a segfaulting exit successful.
|
||||
|
||||
## Existing foundations inspected
|
||||
|
||||
Paths below are local worktree snapshots; uncommitted files can change.
|
||||
Verify HEAD, diff, imports and fixture contracts before implementation.
|
||||
|
||||
- `wkt/human_iface_e2e`, HEAD `aadfd1cf`: headless Qt harness foundation.
|
||||
- `wkt/live_chart_qtractor`, HEAD `1a2d4893`: structured session lifecycle.
|
||||
Its current dirty work includes paper-mode startup forwarding, runtime
|
||||
overrides, and live chart tests; HEAD alone does not contain that work.
|
||||
[test_live_chart.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_live_chart.py:21)
|
||||
starts a production Kraken chart, checks real SHM/price changes and
|
||||
records exact actor/SHM ownership. It is explicitly selected through
|
||||
`--live-provider=kraken` and uses paper EMS without submitting orders.
|
||||
[test_qtractor.py](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/test_qtractor.py:73)
|
||||
covers sequential structured window closure and runtime restart.
|
||||
[UI fixtures](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/ui/conftest.py:79)
|
||||
restore QApplication, widgets, ViewBox registries, config and settings.
|
||||
[open_test_pikerd](/home/goodboy/repos/piker/wkts/live_chart_qtractor/tests/conftest.py:452)
|
||||
supplies isolated config propagation and the plugin registry fixture.
|
||||
- `wkt/replay_provider_e2e`, HEAD `06fee700`: real replay feed/service
|
||||
tests with correlated command acknowledgements and event sequences.
|
||||
[test_provider.py](/home/goodboy/repos/piker/wkts/replay_provider_e2e/tests/replay/test_provider.py:57)
|
||||
exercises actual datad, sampler, search, pause/resume and SHM teardown.
|
||||
[Existing E2E direction](/home/goodboy/repos/piker/wkts/replay_provider_e2e/plans/opencode/live-backend-chart-e2e-review.md:3)
|
||||
prioritizes real public-provider chart journeys; replay is parked until
|
||||
an observed failure needs controlled input. These disconnect races now
|
||||
justify a narrow deterministic reproducer, not a replay-first rewrite.
|
||||
|
||||
The live-chart harness reference still describes MainWindow's old SIGINT
|
||||
behavior, while its branch has the structured lifecycle commit. Reconcile
|
||||
that stale guidance with current source before selecting commands.
|
||||
|
||||
## Stack being qualified
|
||||
|
||||
- Evaluation Piker HEAD `eed58566226d5476b548c6dd3bb6a44fcd6325f9`, with
|
||||
dirty modern-dependency and logger-compatibility changes.
|
||||
- Tractor directory `/home/goodboy/repos/tractor/wkts/address_logging_levels`,
|
||||
branch `wkt/ctx_final_return_disconnect`, HEAD `0553fcf8`.
|
||||
- Tractor final-Return guard and endpoint-error control are uncommitted.
|
||||
[Offline regression](/home/goodboy/repos/tractor/wkts/address_logging_levels/tests/test_context_final_return_disconnect.py)
|
||||
directly drives real TCP and `_rpc._invoke()`, proves unrelated sibling
|
||||
survival, and verifies a genuine endpoint-raised TransportClosed still
|
||||
propagates. This is runtime regression coverage, not chart E2E coverage.
|
||||
- Recorded focused checks: regression + RPC suite 10 passed; context
|
||||
stream semantics 65 passed. These are earlier runs, not handoff execution.
|
||||
- Evaluation runtime: Python 3.13.14, Trio 0.32.0, Qt 6.11.1, PyQt 6.11.0.
|
||||
- Structured Qtractor lifecycle has NOT been transplanted to evaluation.
|
||||
Baseline CLI SIGINT behavior must remain testable separately from the
|
||||
lifecycle-remediated branch.
|
||||
|
||||
## Required topology and observation contract
|
||||
|
||||
Use a suite-owned registrar/pikerd A whose lifetime is independent of B
|
||||
and C. Each chart must be a separate OS process with its own QApplication
|
||||
and Tractor root. Do not run concurrent Tractor roots in pytest's process.
|
||||
The existing in-process single-chart test is useful but does not provide
|
||||
this persistent external-daemon topology.
|
||||
|
||||
Give A a test-unique registrar address, propagate isolated config to ALL
|
||||
children before imports, and bind charts to loopback TCP port zero. Pass
|
||||
discovery and listener addresses separately. Verify actual chart listener
|
||||
ports are nonzero and distinct, and all roots contact A. Do not copy the
|
||||
human's registrar `6116`, fixed chart port `3333`, config or accounts.
|
||||
The manual bind failure came from `[network].chart` explicitly pinning
|
||||
3333; changing it to port zero let simultaneous charts boot.
|
||||
|
||||
Proposed helper contracts below are implementation requirements, not
|
||||
existing APIs. Adapt to the manager's current harness instead of adding
|
||||
another application adapter:
|
||||
|
||||
- A fixture owns A, chart subprocess handles, bounded deadlines, captured
|
||||
output and an external observation/control channel that outlives charts.
|
||||
- Capture service identity as actor name + UUID + PID + listener addresses.
|
||||
Snapshot datad/sampler/EMS/paperboi or brokerd, and provider client IDs or
|
||||
connection-generation counters when that backend exposes them.
|
||||
- Identify charts and FSPs by full actor UID plus PID/parent ownership;
|
||||
names alone are ambiguous with two `chart.fsp_0` actors.
|
||||
- Readiness means feed/history attached, chart views bound, required FSP
|
||||
streams active, search handlers ready and EMS entered. Race every wait
|
||||
against process/session completion and preserve the original exception.
|
||||
- After detach, perform a fresh service RPC and observe a post-detach
|
||||
provider quote in the survivor. Process existence or sampler timestamps
|
||||
alone do not establish service/feed health.
|
||||
- For deterministic input, match a post-action sequence in feed, SHM and
|
||||
chart/FSP consumption. For a live provider, await fresh market data with
|
||||
a realistic deadline; do not require every quote to change price.
|
||||
- Keep exact created SHM names (including child-created unpublished ones)
|
||||
and child ownership events. Registry polling alone misses transient
|
||||
replacement actors and allocations that never reach a ready chart.
|
||||
|
||||
## Journey specifications
|
||||
|
||||
Suggested names describe NEW tests, not existing runnable node IDs.
|
||||
|
||||
### 1. `test_chart_attaches_to_external_pikerd`
|
||||
|
||||
Start A, then B. Establish ready views, nonempty finite history, actual
|
||||
fresh market updates, running FSPs and configured clearing mode. Record
|
||||
baseline identities. Assert B did not create a competing registrar or
|
||||
replacement data service. This setup is reused within each journey,
|
||||
not across unrelated tests that could contaminate one another.
|
||||
|
||||
### 2–3. `test_last_chart_sigint_then_repeated_reattach`
|
||||
|
||||
With only B attached, send terminal-equivalent SIGINT, await its process
|
||||
exit, then attach B2 and B3 in sequence to the same A. For each cycle:
|
||||
|
||||
1. Inspect shutdown outcome, exit code/signal and FSP descendants.
|
||||
2. Assert shared service identities stay constant and fresh RPCs succeed.
|
||||
3. Assert reattached chart receives post-attach market data and FSP output.
|
||||
4. Prove no replacement service/provider-client generation was created.
|
||||
|
||||
Use a real process group created for B and its descendants to reproduce
|
||||
terminal Ctrl-C delivery. Record PID/PGID membership before signaling;
|
||||
A and the controller must be in different groups. A root-PID-only signal
|
||||
is a separate variant and must not be called terminal-equivalent.
|
||||
Run actual MainWindow close/structured shutdown as a separate variant;
|
||||
calling `request_shutdown()` alone does not cover CLI SIGINT.
|
||||
|
||||
### 4. `test_detach_with_search_results_visible_or_dismissed`
|
||||
|
||||
Parameterize visible results versus dismissed pane. Drive production
|
||||
search UI (QtBot in an owned chart worker), await a real query result,
|
||||
then leave results visible or perform the normal dismiss interaction.
|
||||
Detach, prove daemon/survivor health, reattach and search again.
|
||||
Do not substitute merely opening the input field for visible results.
|
||||
Both states have been manually exercised; no causal connection between
|
||||
pane visibility and intermittent REPL entry has been established.
|
||||
|
||||
### 5. `test_shared_daemon_survives_alternating_chart_departures`
|
||||
|
||||
Parameterize same FQME and two distinct FQMEs under one provider.
|
||||
Bring B and C fully live, then execute this order:
|
||||
|
||||
1. Close B; assert the SAME C process consumes a post-close quote/FSP update.
|
||||
2. Reopen B; establish its readiness and fresh data.
|
||||
3. Close C; assert the SAME replacement B keeps receiving updates.
|
||||
4. Close the final B, then reattach one chart and verify fresh updates.
|
||||
|
||||
Same-market coverage must prove one departure did not end the shared
|
||||
market subscription. Different-market coverage must prove per-feed
|
||||
cleanup did not cancel a sibling feed. Shared feed SHM may legitimately
|
||||
remain while A owns its cache: assert lifetime according to its owner,
|
||||
not unconditional SHM disappearance after every chart exit.
|
||||
|
||||
### 6. `test_chart_startup_cancel_preserves_running_consumer`
|
||||
|
||||
Keep ready B and A. Start C and cancel at each gated phase:
|
||||
|
||||
| Phase | Required evidence before cancellation |
|
||||
| --- | --- |
|
||||
| Early attachment | Chart actor registered; initial acquisition in flight |
|
||||
| FSP child bootstrap | Child spawned; imports/registration not completed |
|
||||
| History publication | History request entered; completion/publication held |
|
||||
| FSP establishment | Cascade entered; readiness/stream establishment held |
|
||||
|
||||
Use explicit entered/release acknowledgements at real boundaries.
|
||||
Hold only the designated C operation; do not block the shared datad
|
||||
event loop or stall B. Ensure cancellation can unwind a held operation.
|
||||
Where needed use a narrow test seam/controlled provider for that exact
|
||||
race; timestamps, sleeps and matching an `Entering ...` log are not
|
||||
deterministic gates. A missed gate is not phase coverage.
|
||||
|
||||
After each abort, prove B receives fresh data, A identities persist,
|
||||
C-owned RPCs/children are gone, and a new C can fully start. Finally
|
||||
exercise startup cancellation with no other chart attached to cover the
|
||||
last-consumer case. Keep each phase's outcome distinct in the report.
|
||||
|
||||
Add a debug-mode bootstrap-abort variant: terminal SIGINT can reach FSP
|
||||
children during imports, before their IPC registration. The parent must
|
||||
unwind spawn supervision without attempting an IPC context to itself for
|
||||
stdio locking. Capture the original bootstrap interruption separately
|
||||
from any secondary DebugRequestError. Do not classify this mixed failure
|
||||
as a cancellation-only group or suppress it with cancellation filtering.
|
||||
|
||||
### 7. `test_paper_order_state_survives_chart_reattach`
|
||||
|
||||
Use real paper EMS with isolated writable ledger/account state and a
|
||||
qualified feed. Submit an unfilled paper limit order, confirm its EMS
|
||||
ID/state, detach its chart, reattach and verify the same order/state,
|
||||
then cancel it and observe the terminal cancellation acknowledgement.
|
||||
Check absence of duplicate submissions/replay, stable clearing service
|
||||
identities and consistent position state. A startup order-mode widget
|
||||
alone does not cover this. Credentialed IB paper-account qualification
|
||||
is a separately selected backend tier, with unique gateway client IDs;
|
||||
public-feed paper EMS does not prove IB brokerd reconnect behavior.
|
||||
|
||||
Extend IB qualification to a broker-routed limit and an EMS-held dark
|
||||
order, including a multi-expiry chart. Assert the EMS OID-to-broker reqid
|
||||
mapping survives reattach, repeated status delivery is idempotent,
|
||||
market-specific dialogs remain correctly scoped, cancel-all reaches each
|
||||
applicable order, and terminal cancellation removes the correct dialog.
|
||||
Repeated status logs alone do not prove duplicate broker submissions;
|
||||
count canonical order records and broker working orders independently.
|
||||
|
||||
### 7b. `test_cancel_during_second_chart_order_bootstrap`
|
||||
|
||||
New human-observed sequence: B is already live; start C, then repeatedly
|
||||
submit and immediately cancel orders in B while C is booting. C can
|
||||
finish with a zombie order registered that is already cancelled in B
|
||||
and EMS. The first observation resisted several retries; the human then
|
||||
narrowed the trigger to repeated submit/cancel during C bootstrap. Exact
|
||||
causative ordering and whether stale state is graphical or dialog-local
|
||||
remain unproven. Do not substitute a normal fully-ready two-chart cancel.
|
||||
|
||||
Gate C at the actual order snapshot/replay and live-subscription handoff,
|
||||
keeping B and EMS responsive. Exercise terminal cancellation before,
|
||||
during and after bootstrap delivery, with multiple distinct OIDs. Release
|
||||
an older open snapshot after the terminal event in a controlled variant
|
||||
to test the suspected ordering explicitly, not to assert it was observed.
|
||||
Require C's canonical order state and graphics to converge to EMS's
|
||||
terminal state, and remain absent/cancelled after deferred replay and
|
||||
subsequent live events. Duplicate open delivery must not resurrect a
|
||||
terminal order. Retain OID, broker reqid, bootstrap phase and event order
|
||||
in failure output; cover broker-paper and EMS-held dark orders separately.
|
||||
|
||||
## Failure assertions and diagnostics
|
||||
|
||||
Every journey reports both isolation and exit outcomes even if one fails.
|
||||
Continue collecting A/survivor evidence after chart failure, then fail
|
||||
with aggregated original errors; avoid first-assertion cleanup masking.
|
||||
|
||||
- Fail clean-exit checks on SIGSEGV/native abort, stuck descendants,
|
||||
cancel-ack hard-reap, unexpected debugger entry, or cancellation-only
|
||||
groups relayed as application failures. The native crash is unresolved.
|
||||
- Capture chart return code, actor IDs, RPC/cid, parent/child relation,
|
||||
cancellation source and source task, raw stdout/stderr, phase evidence,
|
||||
SHM ownership, registry/service snapshots and imported code versions.
|
||||
- Enable fault-handler/native crash evidence in owned subprocesses.
|
||||
Retain a core identifier/backtrace if available; do not infer the native
|
||||
fault's root cause solely from the last Python self-SIGINT traceback.
|
||||
- A `Final context result delivery failed: peer disconnected` warning is
|
||||
allowed at the guarded send boundary if the actor and sibling work
|
||||
demonstrably survive. Warning absence does not mean a missed detach.
|
||||
- Sampler StartAck/CancelAck delivery failure to a departed chart/FSP is
|
||||
similarly evaluated by target ownership and subsequent sampler health.
|
||||
- Do not blanket-suppress TransportClosed or BaseExceptionGroup. Mixed
|
||||
groups and genuine endpoint errors must retain their failure behavior.
|
||||
- Preserve both failing-baseline and fixed results. Do not label the
|
||||
current whole journey green because the final-Return guard works.
|
||||
|
||||
## Manual evidence available as of this handoff
|
||||
|
||||
Human-operated observations, not automated results:
|
||||
|
||||
- Last-consumer detach and repeated reattach reused datad UUID
|
||||
`3d8b3632-2c90-4ad2-bde5-d75d87bf03af`, PID 742944, cached IB client 6132.
|
||||
- 12:00:27: final-Return guard caught `open_symbol_search` delivery to
|
||||
departing chart PID 1729338; sampler PID 743912 remained running.
|
||||
- 12:02:10: B PID 1520477 exited; C PID 1757182 kept receiving live data.
|
||||
- 12:02:45: replacement B PID 1771731 started with live data.
|
||||
- 12:03:33: C PID 1757182 exited; human confirmed B remained unaffected.
|
||||
Datad PID 742944 and sampler PID 743912 retained their identities.
|
||||
- Same-MNQ alternating-consumer survival is observed. QQQ also booted
|
||||
alongside MNQ; an equivalent full different-market closure sequence
|
||||
has not been explicitly reported.
|
||||
- FSP cancellation-only groups repeatedly became RemoteActorError;
|
||||
self-SIGINT occurred during Qt close; MNQ exits at 12:00:27 and
|
||||
12:03:33 segfaulted. Chart exit is not clean.
|
||||
- 14:41:26–31: chart `5d1517`, PID 3829973, aborted before order-mode
|
||||
readiness. Two child bootstrap traces were interrupted during Tractor/
|
||||
Trio imports (`ctypes.util.find_library('pthread')`). Spawn cleanup then
|
||||
attempted stdio locking through a local-actor IPC context, producing
|
||||
`RuntimeError: Can not open an IPC ctx with the local actor!` and two
|
||||
DebugRequestError leaves. The chart entered post-mortem. This is a new
|
||||
bootstrap/debug-lock failure, not merely cancellation-only FSP noise.
|
||||
- 14:42:32–37: MNQ chart `432913`, PID 3844804, aborted after BOTH
|
||||
cascade startup profiler exits. This covers near-startup shutdown,
|
||||
not the precise incomplete-FSP-establishment phase.
|
||||
- 14:43:17–25: TSLA chart `6235d8`, PID 3855381, aborted while FSP
|
||||
display startup was still active. Dolla-volume cascade had started and
|
||||
resynced growing history; no second cascade startup is shown before
|
||||
Ctrl-C. This is useful partial-FSP/backfill-overlap evidence, not a
|
||||
deterministic phase gate. Human reports daemon backfill continued.
|
||||
- TSLA SHM token names datad prefix `006f9855-1f2e-4d`, unlike the earlier
|
||||
`3d8b3632-2c90-4a`. Human subsequently explained manual pikerd restarts
|
||||
between network roams because IB roaming recovery remains incomplete.
|
||||
Do not attribute cross-batch identity changes to chart detachment.
|
||||
- 14:45:18: TSLA reattach `41a82c`, PID 3882846, reached both cascade
|
||||
startup completions. Human reports cached-like loading and daemon
|
||||
survival. Human subsequently confirmed B kept receiving fresh updates
|
||||
during C aborts and recovered TSLA quotes/bars/indicators looked good.
|
||||
Startup still emitted no-size/no-yrange rendering messages.
|
||||
- 14:45:40: TSLA detach again emitted cancellation-only FSP groups and
|
||||
MainWindow self-SIGINT. No native segfault is shown in these afternoon
|
||||
excerpts.
|
||||
- Order trial on `ib.algopaper`: broker-routed sell limit OID
|
||||
`70fbf5f4-4d3b-41a4-a16f-af92d68e6aac`, reqid `5042`, size 1 at 29935,
|
||||
`resp='open'`, `exec_mode='live'`. This is IB's paper account through
|
||||
the live broker route, not Piker's paperboi execution mode.
|
||||
- EMS-held sell dark order OID `9efd5800-7a50-4383-995b-f75d51d17853`,
|
||||
size 1 at 29912.50, `resp='dark_open'`, no broker reqid. Human reports
|
||||
both reappeared after chart reattach. Logs retain the same IDs/states.
|
||||
- Reattached chart `b1326a`, PID 253598, at 15:27:37 displayed December
|
||||
2026 and March 2027 MNQ. Status delivery repeated, with untracked-dialog
|
||||
and already-set-brokerd_msg warnings. No duplicate broker submission is
|
||||
established by those repeated logs; replay/dialog correlation needs
|
||||
focused assertions rather than blanket acceptance or suppression.
|
||||
- Human used `cc` cancel-all. Initial attempts said no orders cancelled;
|
||||
later broker details carried reqid `5042`, status `canceled`, filled 0,
|
||||
remaining 1, but wrapped in `name='error'` / `oid=None`, then emitted
|
||||
`Received cancel for unsubmitted order None`. Broker cancellation is
|
||||
observed, clean correlated UI cancellation is not. No terminal dark
|
||||
cancellation is shown. Human reports positions consistent, no fills.
|
||||
- 15:29:39: chart PID 253598 segfaulted after Ctrl-C. Native core/GDB
|
||||
evidence identifies SIP wrapper access during PyQt atexit cleanup.
|
||||
[Native diagnostics](/home/goodboy/repos/piker/wkts/federated_project_stores/ai/diagnostics/20261019-chart-253598/README.md:1)
|
||||
are linked from Taken source `piker`, task ID
|
||||
`piker-qt6-chart-teardown-sigsegv` in the federated store worktree.
|
||||
- 2026-10-20 repeat: chart `4af4a7`, PID 1080042, replayed broker OID
|
||||
`6d78315a-0135-44b3-8162-bd7e816d0190`, reqid `4991`, and dark OID
|
||||
`21f82034-f555-4cbe-a412-ff1a629e5379`. Cancellation again reported
|
||||
broker status `canceled` wrapped as error / `oid=None`. Human then
|
||||
restarted the chart and confirmed neither order returned. This is
|
||||
reported post-cancel absence, not proof of clean status correlation.
|
||||
- Subsequent second-chart bootstrap trial exposed the zombie-order race
|
||||
specified in 7b. Preserve this finding alongside successful ordinary
|
||||
chart restart/cancellation trials; one does not invalidate the other.
|
||||
|
||||
Committed `wkt/fsp_backfill_sync` HEAD `749ca1f0` is an ancestor of the
|
||||
evaluation baseline (verified with `git merge-base --is-ancestor`). That
|
||||
does not establish inclusion of any dirty work in that other worktree or
|
||||
coverage of Tractor process supervision/debug-lock failures. Whether an
|
||||
older Tractor pin masked cancellation-group behavior rather than fixing
|
||||
it requires historical reproduction; it is not established by these logs.
|
||||
|
||||
## Suggested implementation order and delivery evidence
|
||||
|
||||
First extend the installed-process topology and same-market alternating
|
||||
journey; then last-consumer cycles and search-state variants; then gated
|
||||
startup aborts and paper-order continuity. Reuse the lifecycle fixtures
|
||||
and live-first direction already owned by the receiving agent.
|
||||
|
||||
The manager should return exact implemented node IDs, fixture ownership,
|
||||
phase-gate semantics, baseline/fixed test results, independent clean-exit
|
||||
results and remaining backend-specific gaps. Run headless Qt with the
|
||||
branch's evidenced flags; live-provider and IB selection remain explicit.
|
||||
Before choosing the target environment, resolve Piker AND Tractor imports
|
||||
and account for the evaluation's duplicate pytest `--ll` registration
|
||||
blocker. No commands for nonexistent proposed nodes are supplied here.
|
||||
|
||||
This handoff was written from source inspection and supplied live logs.
|
||||
No live services, chart processes, orders or E2E suite were started by
|
||||
the authoring agent for this drafting task. Delivery to the named dialog
|
||||
has not been confirmed; the file is ready for that dialog to consume.
|
||||
Loading…
Reference in New Issue