piker/plans/opencode/chart-local-gap-overlays.md

Chart-Local Gap Overlays

Goal

Make gap overlays a chart-actor-owned display service while preserving remote annotation as a first-class API. Local startup and UI input avoid an unnecessary IPC round trip; remote tools use a strict msgspec schema and the same renderer/controller.

Ownership Model

GapOverlayMngr is created once in the chart actor and references the actor-global _dss and _annots registries. Each layer key is (owner, id(chart), fqme, timeframe). The chart identity keeps duplicate FQMEs in separate cached displays independent. The reserved chart-local owner is also independent from each remote Context.cid, so remote context teardown cannot remove locally owned graphics.

The controller has one rendering entry point:

1. GapOverlayMngr.apply(SetGapOverlay) replaces an owner layer.
2. GapOverlayMngr.refresh() detects gaps from chart-local SHM and calls apply() directly.
3. AnnotClient.set_gap_overlay() sends the same request through the strict remote_gap_overlays() context.

The generic remote_annotate() context and its rectangle, arrow, text, batch, remove and redraw commands remain unchanged.

Startup And UI

graphics_update_loop() registers each new DisplayState in _dss, then builds the default 60-second local overlay. The focused ChartView handles Ctrl+G through its existing asynchronous keyboard loop. It chooses the focused chart’s 1-second or 60-second timeframe and calls GapOverlayMngr.toggle() directly.

Gap detection reads the structured OHLCV array, excludes invalid timestamps, uses one numpy.diff() pass, and creates Python GapSpec objects only for actual positive timestamp gaps.

Typed IPC

remote_gap_overlays() is separate from remote_annotate() because msgspec cannot strictly decode a union containing both arbitrary legacy dicts and several struct types. Its Tractor payload specification is:

SetGapOverlay | GapOverlay | list[str] | None

SetGapOverlay and GapOverlay are tagged structs. Tractor validates started, stream and return payloads against this union. A lock shared by all FQMEs on one typed stream serializes each send/receive exchange. Each side narrows its public Context.pld_rx decoder to the directional request or response type while receiving. Request IDs let the client discard a late response after an earlier timeout instead of skewing later request/reply pairs.

Refresh Semantics

Startup and every UI transition from hidden to visible recompute the layer from current SHM. Existing redraw handling repositions matching overlays by both FQME and timeframe. Rectangle and arrow geometry update together.

The next production step is to trigger refresh() from an explicit history revision/publication event after a repair or prepend. Polling from the quote display loop is intentionally excluded.

Request/reply serialization and stale-response correlation can move into a generic IPC dialog helper after a second typed consumer establishes the reusable interface. The legacy annotation dict protocol should first migrate to tagged messages before sharing the strict gap context.

Landing Order

The existing ancestry is:

main
  gap_annotator
    datad_service
      backfiller_deep_fixes
      wkt/fix_broadcast_consumers

The recommended integration order is:

1. Review and land the complete 23-commit main..gap_annotator range.
2. Land Tractor PR 490 after its PR 475 dependency, then update the piker Tractor reference used by datad_service.
3. Land datad_service on the new mainline.
4. Land the smaller four-commit broadcast-consumer PR 92.
5. Rebase backfiller_deep_fixes onto the broadcast result and run the NativeDB/SHM/manual chart qualification sequence.
6. Replay the chart-local overlay patch above the qualified backfiller tip.

This order puts the already-reviewed ownership fix ahead of the broader backfiller qualification. The only known overlapping production path is piker/brokers/ib/feed.py; git merge-tree --write-tree reports no current textual conflict, but behavior must still be rechecked.

No branch movement, rebase, merge, commit or push is part of this prototype.

Verification

Run from the prototype worktree with the source-resolving Python environment:

python -m pytest -p no:xonsh -q -x --tb=short --no-header \
    tests/test_gap_overlays.py
python -m ruff check piker/ui/_gaps.py \
    tests/test_gap_overlays.py piker/ui/_remote_ctl.py \
    piker/ui/_display.py piker/ui/_interaction.py \
    piker/ui/_widget.py piker/ui/_annotate.py

Manual chart qualification must confirm startup rendering, focused-timeframe Ctrl+G, remote typed replacement, remote disconnect cleanup, cached symbol switching and refresh after backfill repair.