piker/plans/opencode/chart-local-gap-overlays.md

# Chart-Local Gap Overlays

## Goal

Make gap overlays a chart-actor-owned display service while preserving
remote annotation as a first-class API. Local startup and UI input avoid
an unnecessary IPC round trip; remote tools use a strict `msgspec` schema
and the same renderer/controller.

## Ownership Model

`GapOverlayMngr` is created once in the chart actor and references the
actor-global `_dss` and `_annots` registries. Each layer key is
`(owner, id(chart), fqme, timeframe)`. The chart identity keeps duplicate
FQMEs in separate cached displays independent. The reserved `chart-local`
owner is also independent from each remote `Context.cid`, so remote context
teardown cannot remove locally owned graphics.

The controller has one rendering entry point:

1. `GapOverlayMngr.apply(SetGapOverlay)` replaces an owner layer.
2. `GapOverlayMngr.refresh()` detects gaps from chart-local SHM and calls
   `apply()` directly.
3. `AnnotClient.set_gap_overlay()` sends the same request through the
   strict `remote_gap_overlays()` context.

The generic `remote_annotate()` context and its rectangle, arrow, text,
batch, remove and redraw commands remain unchanged.

## Startup And UI

`graphics_update_loop()` registers each new `DisplayState` in `_dss`, then
builds the default 60-second local overlay. The focused `ChartView` handles
`Ctrl+G` through its existing asynchronous keyboard loop. It chooses the
focused chart's 1-second or 60-second timeframe and calls
`GapOverlayMngr.toggle()` directly.

Gap detection reads the structured OHLCV array, excludes invalid timestamps,
uses one `numpy.diff()` pass, and creates Python `GapSpec` objects only for
actual positive timestamp gaps.

## Typed IPC

`remote_gap_overlays()` is separate from `remote_annotate()` because
`msgspec` cannot strictly decode a union containing both arbitrary legacy
dicts and several struct types. Its Tractor payload specification is:

```python
SetGapOverlay | GapOverlay | list[str] | None
```

`SetGapOverlay` and `GapOverlay` are tagged structs. Tractor validates
started, stream and return payloads against this union. A lock shared by
all FQMEs on one typed stream serializes each send/receive exchange. Each
side narrows its public `Context.pld_rx` decoder to the directional request
or response type while receiving. Request IDs let the client discard a late
response after an earlier timeout instead of skewing later request/reply
pairs.

## Refresh Semantics

Startup and every UI transition from hidden to visible recompute the layer
from current SHM. Existing redraw handling repositions matching overlays by
both FQME and timeframe. Rectangle and arrow geometry update together.

The next production step is to trigger `refresh()` from an explicit history
revision/publication event after a repair or prepend. Polling from the quote
display loop is intentionally excluded.

Request/reply serialization and stale-response correlation can move into a
generic IPC dialog helper after a second typed consumer establishes the
reusable interface. The legacy annotation dict protocol should first migrate
to tagged messages before sharing the strict gap context.

## Landing Order

The existing ancestry is:

```text
main
  gap_annotator
    datad_service
      backfiller_deep_fixes
      wkt/fix_broadcast_consumers
```

The recommended integration order is:

1. Review and land the complete 23-commit `main..gap_annotator` range.
2. Land Tractor PR 490 after its PR 475 dependency, then update the piker
   Tractor reference used by `datad_service`.
3. Land `datad_service` on the new mainline.
4. Land the smaller four-commit broadcast-consumer PR 92.
5. Rebase `backfiller_deep_fixes` onto the broadcast result and run the
   NativeDB/SHM/manual chart qualification sequence.
6. Replay the chart-local overlay patch above the qualified backfiller tip.

This order puts the already-reviewed ownership fix ahead of the broader
backfiller qualification. The only known overlapping production path is
`piker/brokers/ib/feed.py`; `git merge-tree --write-tree` reports no current
textual conflict, but behavior must still be rechecked.

No branch movement, rebase, merge, commit or push is part of this prototype.

## Verification

Run from the prototype worktree with the source-resolving Python environment:

```text
python -m pytest -p no:xonsh -q -x --tb=short --no-header \
    tests/test_gap_overlays.py
python -m ruff check piker/ui/_gaps.py \
    tests/test_gap_overlays.py piker/ui/_remote_ctl.py \
    piker/ui/_display.py piker/ui/_interaction.py \
    piker/ui/_widget.py piker/ui/_annotate.py
```

Manual chart qualification must confirm startup rendering, focused-timeframe
`Ctrl+G`, remote typed replacement, remote disconnect cleanup, cached symbol
switching and refresh after backfill repair.