pikers
/
piker
4
1
Fork 0
Code Issues 20 Pull Requests 12 Packages Projects Releases Wiki Activity

Own gap overlays in chart-local `GapOverlayMngr` 

Detect positive timestamp gaps from chart OHLCV SHM with one
vectorized pass. Route local UI and remote callers through one
`GapOverlayMngr` renderer, and key each layer by owner, chart,
FQME and timeframe so remote ctx teardown cannot remove local
graphics.

Deats,
- add tagged `SetGapOverlay` requests and typed
  `GapOverlay` responses;
- keep `remote_annotate()` intact and serve its typed dialog from
  a dedicated `remote_gap_overlays()` Tractor ctx;
- admit `SetGapOverlay`, `GapOverlay`, `list[str]` and `None` in
  the Tractor payload spec, then strictly narrow requests to
  `SetGapOverlay` and replies to `GapOverlay`;
- serialize shared-stream dialogs and correlate request IDs so
  stale replies cannot skew later exchanges;
- rename `AnnotCtl` / `open_annot_ctl()` to `AnnotClient` /
  `open_annot_client()` across storage and annotation callers;
- enable 60s history overlays at startup and toggle the focused
  1s or 60s chart directly with `Ctrl+G`;
- guard redraws by chart, FQME and timeframe; tolerate empty
  arrays and out-of-range lookups; update rect and arrow geometry;
- cover detector geometry, msgpack roundtrips, schema rejection,
  typed stale-FQME errors and renamed storage seams;
- document ownership, refresh, stack landing and live
  qualification.

Prompt-IO: ai/prompt-io/opencode/20260814T032716Z_2aa66c2c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
chart_local_gapper
Gud Boi 2026-08-14 19:03:11 -04:00
parent 2aa66c2cf6
commit f65df6a138
14 changed files with 1089 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
ai/prompt-io/opencode/20260814T032716Z_2aa66c2c_prompt_io.md 100644
View File

@ -0,0 +1,43 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-14T03:27:16Z
git_ref: 2aa66c2c
scope: code
substantive: true
raw_file: 20260814T032716Z_2aa66c2c_prompt_io.raw.md
---
## Prompt
Prototype chart-actor-owned gap overlays from the current backfiller tip,
with actor-local startup and UI toggling. Preserve remote annotation IPC and
make the gap-specific path strict and `msgspec`-typed.
## Response summary
Adds a chart-local controller and vectorized SHM detector, starts historical
gap overlays with display state, toggles the focused timeframe through the
existing input loop and exposes the same renderer through a strict Tractor
payload context. Keeps all legacy remote annotation commands and separates
local graphics from each remote context's teardown ownership.
## Files changed
- `piker/ui/_gaps.py` - typed messages, detection and local controller.
- `piker/ui/_remote_ctl.py` - IPC clients and chart-actor endpoints.
- `piker/ui/_display.py` - actor-local startup integration.
- `piker/ui/_interaction.py` - focused `Ctrl+G` toggle.
- `piker/ui/_widget.py` - controller reference.
- `piker/ui/_annotate.py` - safe FQME-aware repositioning.
- `piker/tsp/_annotate.py` - typed annotation-client reference.
- `piker/storage/cli.py` - renamed annotation client context.
- `tests/test_gap_overlays.py` - detector and schema regression.
- `tests/test_ldshm.py` - renamed annotation-client test seams.
- `plans/opencode/chart-local-gap-overlays.md` - architecture and landing plan.
## Human edits
None - the work remains an uncommitted disposable-worktree prototype for
human review and live chart qualification.

72
ai/prompt-io/opencode/20260814T032716Z_2aa66c2c_prompt_io.raw.md 100644
View File

@ -0,0 +1,72 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-14T03:27:16Z
git_ref: 2aa66c2c
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user asked what work had been done so far, then authorized autonomous
work while AFK. The target was to scheme and prototype an actor-local gap
annotator from the `backfiller_deep_fixes` tip without mutating existing
branches. The user then clarified twice that the annotation IPC API must
remain available and should become more rigorous and `msgspec`-typed rather
than being removed.
## Generated code
> `git diff HEAD~1..HEAD -- piker/ui/_gaps.py`
Adds tagged gap request/reply structs, render-ready gap specs, vectorized
OHLCV timestamp-gap detection and chart-local overlay ownership.
> `git diff HEAD~1..HEAD -- piker/ui/_remote_ctl.py`
Adds strict Tractor gap IPC, `AnnotClient` stream mapping/locking and
owner-scoped endpoint cleanup while retaining the legacy annotation endpoint.
> `git diff HEAD~1..HEAD -- piker/ui/_display.py`
Starts the local 60-second overlay after display-state registration.
> `git diff HEAD~1..HEAD -- piker/ui/_interaction.py`
Routes `Ctrl+G` from the focused chart's existing async input loop directly
to the local controller.
> `git diff HEAD~1..HEAD -- piker/ui/_widget.py`
Stores the chart actor's local controller reference.
> `git diff HEAD~1..HEAD -- piker/ui/_annotate.py`
Keeps redraw repositioning isolated by FQME and synchronizes arrow geometry.
> `git diff HEAD~1..HEAD -- piker/tsp/_annotate.py piker/storage/cli.py`
Renames the remote annotation control type and context manager to the
client-oriented `AnnotClient` / `open_annot_client` interface.
> `git diff HEAD~1..HEAD -- tests/test_gap_overlays.py`
Adds deterministic detector, directional payload-schema, typed error and
msgpack roundtrip regressions.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Updates storage CLI test seams for the renamed annotation client context.
## Generated architecture output
> `git diff HEAD~1..HEAD -- plans/opencode/chart-local-gap-overlays.md`
Documents ownership, startup, UI, strict IPC, refresh and branch landing
semantics.
## Verification output
Source resolution points to this worktree. The targeted regression passes,
Ruff reports no changed-file findings, `compileall` succeeds, `git diff
--check` succeeds and Tractor's strict gap payload decoder initializes.

8
piker/storage/cli.py
View File

@ -62,7 +62,7 @@ from .nativedb import (
)
if TYPE_CHECKING:
from piker.ui._remote_ctl import AnnotCtl
from piker.ui._remote_ctl import AnnotClient
store = typer.Typer(no_args_is_help=True)
@ -940,9 +940,9 @@ def shm(
async def main() -> bool:
from piker.ui._remote_ctl import (
open_annot_ctl,
open_annot_client,
)
actl: AnnotCtl
actl: AnnotClient
mod: ModuleType
client: StorageClient
async with (
@ -955,7 +955,7 @@ def shm(
mod,
client,
),
open_annot_ctl() as actl,
open_annot_client() as actl,
):
shm_df: pl.DataFrame | None = None
tf2aids: dict[float, dict] = {}

4
piker/tsp/_annotate.py
View File

@ -38,7 +38,7 @@ from piker.toolz.profile import (
from piker.ui._style import get_fonts
if TYPE_CHECKING:
from piker.ui._remote_ctl import AnnotCtl
from piker.ui._remote_ctl import AnnotClient
def humanize_duration(
@ -89,7 +89,7 @@ def humanize_duration(
async def markup_gaps(
fqme: str,
timeframe: float,
actl: AnnotCtl,
actl: AnnotClient,
wdts: pl.DataFrame,
gaps: pl.DataFrame,

49
piker/ui/_annotate.py
View File

@ -581,6 +581,15 @@ class GapAnnotations(GraphicsObject):
'''
# skip reposition if timeframe doesn't match
# (e.g., 1s gaps being repositioned with 60s array)
if (
fqme is not None
and
self._fqme is not None
and
fqme != self._fqme
):
return
if (
timeframe is not None
and
@ -605,6 +614,13 @@ class GapAnnotations(GraphicsObject):
)
return
if not len(array):
log.warning(
'GapAnnotations.reposition() called with an empty '
'array'
)
return
# collect all unique timestamps we need to lookup
timestamps: set[float] = set()
for spec in self._gap_specs:
@ -628,9 +644,14 @@ class GapAnnotations(GraphicsObject):
)
# vectorized bounds check and exact match verification
valid_mask = (
(search_indices < len(array))
& (time_arr[search_indices] == ts_array)
in_bounds = search_indices < len(array)
valid_mask = np.zeros(
len(search_indices),
dtype=bool,
)
valid_mask[in_bounds] = (
time_arr[search_indices[in_bounds]]
== ts_array[in_bounds]
)
valid_indices = search_indices[valid_mask]
@ -702,18 +723,30 @@ class GapAnnotations(GraphicsObject):
# rebuild arrow path with new indices
self._arrow_path.clear()
spec: dict
for spec in self._gap_specs:
time_val = spec.get('time')
time_val: float|None = spec.get('time')
if time_val is None:
continue
arrow_row = time_to_row.get(time_val)
arrow_row: dict[str, float]|None = time_to_row.get(
time_val,
)
if arrow_row is None:
continue
arrow_x = arrow_row['index']
arrow_y = arrow_row['close']
pointing = spec['pointing']
arrow_x: float = arrow_row['index']
start_row: dict[str, float]|None = time_to_row.get(
spec.get('start_time'),
)
arrow_y: float = (
start_row['close']
if start_row is not None
else spec['arrow_y']
)
pointing: str = spec['pointing']
spec['arrow_x'] = arrow_x
spec['arrow_y'] = arrow_y
# create arrow polygon
if pointing == 'down':

24
piker/ui/_display.py
View File

@ -78,6 +78,11 @@ from ._forms import (
FieldsForm,
mk_order_pane_layout,
)
from ._gaps import (
GapOverlay,
GapOverlayMngr,
HIST_GAP_TIMEFRAME,
)
from . import _pg_overrides as pgo
from .order_mode import (
open_order_mode,
@ -487,6 +492,25 @@ async def graphics_update_loop(
# prior view..
from . import _remote_ctl
_remote_ctl._dss.update(dss)
gapman: GapOverlayMngr = _remote_ctl._gapman
godwidget.gapman = gapman
gap_timeframe: float = HIST_GAP_TIMEFRAME
# TODO: expose startup visibility and enabled timeframes through
# the UI config once chart periods are configurable.
fqme: str
ds: DisplayState
for fqme, ds in dss.items():
gap: GapOverlay = gapman.refresh(
ds=ds,
timeframe=gap_timeframe,
)
log.info(
'Chart-local gap overlay ready:\n'
f'fqme: {fqme}\n'
f'timeframe: {gap_timeframe}s\n'
f'gaps: {gap.gap_count}\n'
)
# main real-time quotes update loop
stream: tractor.MsgStream

413
piker/ui/_gaps.py 100644
View File

@ -0,0 +1,413 @@
# piker: trading gear for hackers
# Copyright (C) Tyler Goodlet (in stewardship for pikers)
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Chart-local gap-overlay detection, rendering and IPC messages.
'''
from __future__ import annotations
from typing import (
Literal,
TYPE_CHECKING,
)
import numpy as np
from piker.types import Struct
from ._annotate import GapAnnotations
if TYPE_CHECKING:
from piker.ui.qt import QGraphicsItem
from PyQt6.QtWidgets import QGraphicsScene
from ._chart import ChartPlotWidget
from ._dataviz import Viz
from ._display import DisplayState
# `DisplayState` currently exposes exactly one realtime and one
# historical OHLC chart at these sampling periods.
RT_GAP_TIMEFRAME: float = 1.
HIST_GAP_TIMEFRAME: float = 60.
class GapSpec(Struct, frozen=True):
'''
Render-ready geometry for one OHLC time gap.
'''
start_pos: tuple[float, float]
end_pos: tuple[float, float]
arrow_x: float
arrow_y: float
pointing: Literal['up', 'down']
start_time: float
end_time: float
time: float
color: str = 'dad_blue'
alpha: int = 169
class SetGapOverlay(Struct, frozen=True, tag=True):
'''
Request replacement of one named gap-overlay layer.
`GapOverlayMngr.apply()` accepts this type locally and through
the chart actor's typed IPC endpoint. The receiver assigns
ownership so remote teardown cannot remove chart-local overlays.
'''
fqme: str
timeframe: float
specs: list[GapSpec]
visible: bool = True
request_id: str = ''
class GapOverlay(Struct, frozen=True, tag=True):
'''
Gap-overlay state returned after applying `SetGapOverlay`.
'''
fqme: str
timeframe: float
visible: bool
gap_count: int
aid: int|None = None
error: str|None = None
request_id: str = ''
GapOverlayPayload = (
SetGapOverlay
| GapOverlay
| list[str]
| None
)
def gap_specs_from_ohlcv(
array: np.ndarray,
period_s: float,
) -> list[GapSpec]:
'''
Build render specs for positive timestamp gaps in sorted OHLCV.
Invalid and non-positive timestamps are excluded. Timestamp steps
are detected in one vectorized pass; only actual gaps enter the
small Python object-construction loop.
'''
if len(array) < 2:
return []
times: np.ndarray = array['time'].astype(float, copy=False)
valid: np.ndarray = (
np.isfinite(times)
& (times > 0)
)
rows: np.ndarray = array[valid]
if len(rows) < 2:
return []
times = rows['time'].astype(float, copy=False)
gap_ends: np.ndarray = np.flatnonzero(
np.diff(times) > period_s
) + 1
if not len(gap_ends):
return []
gap_starts: np.ndarray = gap_ends - 1
left: np.ndarray = rows[gap_starts]
right: np.ndarray = rows[gap_ends]
start_xs: np.ndarray = (
left['index'].astype(float, copy=False)
+ 0.9
)
end_xs: np.ndarray = (
right['index'].astype(float, copy=False)
+ 0.1
)
start_ys: np.ndarray = left['close'].astype(
float,
copy=False,
)
end_ys: np.ndarray = right['open'].astype(
float,
copy=False,
)
specs: list[GapSpec] = []
start_x: np.floating
end_x: np.floating
start_y: np.floating
end_y: np.floating
start_time: np.floating
end_time: np.floating
for (
start_x,
end_x,
start_y,
end_y,
start_time,
end_time,
) in zip(
start_xs,
end_xs,
start_ys,
end_ys,
left['time'],
right['time'],
strict=True,
):
specs.append(GapSpec(
start_pos=(float(start_x), float(start_y)),
end_pos=(float(end_x), float(end_y)),
arrow_x=float(end_x - 0.1),
arrow_y=float(start_y),
pointing=(
'down'
if start_y > end_y
else 'up'
),
start_time=float(start_time),
end_time=float(end_time),
time=float(end_time),
))
return specs
class GapOverlayMngr:
'''
Own named gap-overlay layers inside one chart actor.
Local UI calls and remote annotation IPC both dispatch typed
`SetGapOverlay` messages through `GapOverlayMngr.apply()`.
'''
def __init__(
self,
dss: dict[str, DisplayState],
annots: dict[int, QGraphicsItem],
) -> None:
self.dss: dict[str, DisplayState] = dss
self.annots: dict[int, QGraphicsItem] = annots
self._layers: dict[
tuple[ # owner-scoped layer key
str, # owner (`chart-local` or `Context.cid`)
int, # `id(ChartPlotWidget)`
str, # fqme
float, # timeframe
],
int, # `id(GapAnnotations)`
] = {}
self._visible: dict[
tuple[ # actor-local `SetGapOverlay` key
int, # `id(ChartPlotWidget)`
str, # fqme
float, # timeframe
],
bool, # `SetGapOverlay.visible`
] = {}
def apply(
self,
req: SetGapOverlay,
owner: str = 'chart-local',
aids: set[int]|None = None,
ds: DisplayState|None = None,
) -> GapOverlay:
'''
Replace one owner-scoped batch layer from a typed request.
'''
if ds is None:
ds = self.dss.get(req.fqme)
if ds is None:
return GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=False,
gap_count=0,
request_id=req.request_id,
error=(
f'No display state for fqme={req.fqme}'
),
)
try:
chart: ChartPlotWidget = {
HIST_GAP_TIMEFRAME: ds.hist_chart,
RT_GAP_TIMEFRAME: ds.chart,
}[req.timeframe]
except KeyError:
return GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=False,
gap_count=0,
request_id=req.request_id,
error=(
f'No chart for timeframe={req.timeframe}s'
),
)
layer_key: tuple[str, int, str, float] = (
owner,
id(chart),
req.fqme,
req.timeframe,
)
aid: int|None = self._layers.pop(layer_key, None)
if aid is not None:
annot: QGraphicsItem|None = self.annots.pop(
aid,
None,
)
if aids is not None:
aids.discard(aid)
if annot is not None:
scene: QGraphicsScene|None = annot.scene()
if scene:
scene.removeItem(annot)
if owner == 'chart-local':
self._visible[
(id(chart), req.fqme, req.timeframe)
] = req.visible
if (
not req.visible
or not req.specs
):
return GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=req.visible,
gap_count=len(req.specs),
request_id=req.request_id,
)
viz: Viz = chart.get_viz(req.fqme)
gap_specs: list[dict] = []
spec: GapSpec
for spec in req.specs:
gap_specs.append(spec.to_dict())
gaps_item: GapAnnotations = GapAnnotations(
gap_specs=gap_specs,
array=viz.shm.array,
color=req.specs[0].color,
alpha=req.specs[0].alpha,
arrow_size=10,
fqme=req.fqme,
timeframe=req.timeframe,
)
viz.plot.addItem(gaps_item)
gaps_item._chart = chart
new_aid: int = id(gaps_item)
self.annots[new_aid] = gaps_item
self._layers[layer_key] = new_aid
if aids is not None:
aids.add(new_aid)
return GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=True,
gap_count=len(req.specs),
aid=new_aid,
request_id=req.request_id,
)
def refresh(
self,
ds: DisplayState,
timeframe: float,
visible: bool = True,
) -> GapOverlay:
'''
Detect gaps from chart-local SHM and replace the local layer.
'''
viz: Viz = {
HIST_GAP_TIMEFRAME: ds.hist_viz,
RT_GAP_TIMEFRAME: ds.viz,
}[timeframe]
specs: list[GapSpec] = gap_specs_from_ohlcv(
viz.shm.array,
period_s=timeframe,
)
return self.apply(SetGapOverlay(
fqme=ds.fqme,
timeframe=timeframe,
specs=specs,
visible=visible,
), ds=ds)
def toggle(
self,
ds: DisplayState,
timeframe: float,
) -> GapOverlay:
'''
Toggle and refresh one chart-local gap-overlay layer.
'''
chart: ChartPlotWidget = {
HIST_GAP_TIMEFRAME: ds.hist_chart,
RT_GAP_TIMEFRAME: ds.chart,
}[timeframe]
visible: bool = not self._visible.get(
(id(chart), ds.fqme, timeframe),
False,
)
return self.refresh(
ds=ds,
timeframe=timeframe,
visible=visible,
)
def remove_owner(
self,
owner: str,
) -> None:
'''
Remove all graphics and bookkeeping for one IPC owner.
'''
owner_keys: list[tuple[str, int, str, float]] = []
key: tuple[str, int, str, float]
for key in self._layers:
if key[0] == owner:
owner_keys.append(key)
layer_key: tuple[str, int, str, float]
for layer_key in owner_keys:
aid: int = self._layers.pop(layer_key)
annot: QGraphicsItem|None = self.annots.pop(
aid,
None,
)
if annot is not None:
scene: QGraphicsScene|None = annot.scene()
if scene:
scene.removeItem(annot)

35
piker/ui/_interaction.py
View File

@ -61,6 +61,12 @@ from ..toolz import (
from .view_mode import overlay_viewlists
# from ._style import _min_points_to_show
from ._editors import SelectRect
from ._gaps import (
GapOverlay,
GapOverlayMngr,
HIST_GAP_TIMEFRAME,
RT_GAP_TIMEFRAME,
)
from . import _event
if TYPE_CHECKING:
@ -225,6 +231,35 @@ async def handle_viewmode_kb_inputs(
# path necessary?
viz.reset_graphics()
# Ctrl-G is the default chart-local gap-overlay toggle.
# TODO: expose this binding through the UI config.
if (
ctrl
and
key == Qt.Key_G
):
gapman: GapOverlayMngr|None = godw.gapman
if gapman is not None:
fqme: str
ds: DisplayState
for fqme, ds in dss.items():
timeframe: float = (
HIST_GAP_TIMEFRAME
if view._chart is ds.hist_chart
else RT_GAP_TIMEFRAME
)
gap: GapOverlay = gapman.toggle(
ds=ds,
timeframe=timeframe,
)
log.info(
'Toggled chart-local gap overlay:\n'
f'fqme: {fqme}\n'
f'timeframe: {timeframe}s\n'
f'visible: {gap.visible}\n'
f'gaps: {gap.gap_count}\n'
)
# ------ - ------
# SEARCH MODE
# ------ - ------

169
piker/ui/_remote_ctl.py
View File

@ -61,6 +61,12 @@ from ._editors import (
from ._chart import ChartPlotWidget
from ._dataviz import Viz
from ._style import hcolor
from ._gaps import (
GapOverlay,
GapOverlayMngr,
GapOverlayPayload,
SetGapOverlay,
)
log = get_logger(__name__)
@ -96,6 +102,11 @@ EditorsTable = dict[int, ArrowEditor]
_annots: AnnotsTable = {}
_editors: EditorsTable = {}
_gapman = GapOverlayMngr(
dss=_dss,
annots=_annots,
)
def rm_annot(
annot: ArrowEditor|SelectRect|pg.TextItem
) -> bool:
@ -155,6 +166,13 @@ def no_qt_updates(*items):
item.setUpdatesEnabled(True)
# TODO XXX: TYPE THE ENTIRE ANNOTATION IPC API.
# `serve_rc_annots()` still dispatches arbitrary dicts from one large
# match block. Move each operation to tagged request/response
# structs, narrow each receive phase with
# `Context.pld_rx.limit_plds()`, and then reassess whether
# `remote_gap_overlays()` should fold back into the general annotation
# endpoint.
async def serve_rc_annots(
ipc_key: str,
annot_req_stream: MsgStream,
@ -169,6 +187,7 @@ async def serve_rc_annots(
'''
global _editors
msg: dict
async for msg in annot_req_stream:
match msg:
case {
@ -684,7 +703,8 @@ async def serve_rc_annots(
fqme=fqme,
timeframe=timeframe,
)
chart.plotItem.addItem(gaps_item)
gaps_item._chart = chart
viz.plot.addItem(gaps_item)
# register single item for repositioning
aid: int = id(gaps_item)
@ -839,6 +859,8 @@ async def serve_rc_annots(
for aid, annot in annots.items():
# GapAnnotations batch items have .reposition()
if hasattr(annot, 'reposition'):
if annot._chart is not chart:
continue
annot.reposition(
array=arr,
fqme=fqme,
@ -928,17 +950,57 @@ async def remote_annotate(
profiler(f'removed all {len(aids)} annotations')
class AnnotCtl(Struct):
@tractor.context(pld_spec=GapOverlayPayload)
async def remote_gap_overlays(
ctx: Context,
) -> None:
'''
A control for remote "data annotations".
Strictly typed IPC endpoint for batched gap-overlay control.
The generic `remote_annotate()` endpoint remains available for
legacy rectangle, arrow and text commands.
'''
if not _dss:
raise RuntimeError(
'Chart display state is not initialized'
)
owner: str = ctx.cid
aids: set[int] = set()
await ctx.started(list(_dss))
try:
stream: MsgStream
async with ctx.open_stream() as stream:
with ctx.pld_rx.limit_plds(
spec=SetGapOverlay,
):
req: SetGapOverlay
async for req in stream:
resp: GapOverlay = _gapman.apply(
req=req,
owner=owner,
aids=aids,
)
await stream.send(resp)
finally:
_gapman.remove_owner(owner)
class AnnotClient(Struct):
'''
IPC client for remote chart annotations.
You know those "squares they always show in machine vision
UIs.." this API allows you to remotely control stuff like that
in some other graphics actor.
'''
ctx2fqmes: dict[str, str]
ctx2fqmes: dict[str, set[str]]
fqme2ipc: dict[str, MsgStream]
fqme2gap_ipc: dict[str, MsgStream]
_gap_locks: dict[str, trio.Lock]
_annot_stack: AsyncExitStack
# runtime-populated mapping of all annotation
@ -957,6 +1019,55 @@ class AnnotCtl(Struct):
)
return ipc
async def set_gap_overlay(
self,
req: SetGapOverlay,
) -> GapOverlay:
'''
Replace one remote gap-overlay layer using typed IPC.
'''
ipc: MsgStream|None = self.fqme2gap_ipc.get(req.fqme)
if ipc is None:
raise SymbolNotFound(
'No chart actor exposes typed gap-overlay IPC for:\n'
f'{req.fqme}'
)
request_id: str = str(uuid4())
req = SetGapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
specs=req.specs,
visible=req.visible,
request_id=request_id,
)
# TODO: factor this keyed dialog transaction into a generic
# IPC helper once another req/resp client needs stale reply
# correlation and per-stream serialization.
async with self._gap_locks[req.fqme]:
with trio.fail_after(3):
await ipc.send(req)
with ipc.ctx.pld_rx.limit_plds(
spec=GapOverlay,
):
while True:
resp: GapOverlay = await ipc.receive()
if resp.request_id == request_id:
break
log.warning(
'Discarding stale gap-overlay '
'response:\n'
f'expected: {request_id}\n'
f'received: {resp.request_id}\n'
)
if resp.error:
raise RuntimeError(resp.error)
return resp
async def add_rect(
self,
fqme: str,
@ -1243,10 +1354,10 @@ class AnnotCtl(Struct):
@acm
async def open_annot_ctl(
async def open_annot_client(
uid: tuple[str, str] | None = None,
) -> AnnotCtl:
) -> AnnotClient:
# TODO: load connetion to a specific chart actor
# -[ ] pull from either service scan or config
# -[ ] return some kinda client/proxy thinger?
@ -1262,6 +1373,7 @@ async def open_annot_ctl(
) as maybe_portals:
ctx_mngrs: list[AsyncContextManager] = []
gap_ctx_mngrs: list[AsyncContextManager] = []
# TODO: print the current discoverable actor UID set
# here as well?
@ -1274,13 +1386,26 @@ async def open_annot_ctl(
ctx_mngrs.append(
portal.open_context(remote_annotate)
)
gap_ctx_mngrs.append(
portal.open_context(remote_gap_overlays)
)
# Keep the typed gap protocol on its own ctx while legacy
# annotation commands remain arbitrary dicts. A dynamic
# `tractor.msg.limit_plds()` cannot choose the right decoder
# before a mixed stream payload has itself been decoded.
ctx2fqmes: dict[str, set[str]] = {}
fqme2ipc: dict[str, MsgStream] = {}
stream_ctxs: list[AsyncContextManager] = []
gap_ctx2fqmes: dict[str, set[str]] = {}
fqme2gap_ipc: dict[str, MsgStream] = {}
gap_locks: dict[str, trio.Lock] = {}
gap_stream_ctxs: list[AsyncContextManager] = []
async with (
trionics.gather_contexts(ctx_mngrs) as ctxs,
trionics.gather_contexts(gap_ctx_mngrs) as gap_ctxs,
):
for (ctx, fqmes) in ctxs:
stream_ctxs.append(ctx.open_stream())
@ -1291,9 +1416,9 @@ async def open_annot_ctl(
raise ValueError(
f'More then one chart displays {fqme}!?\n'
'Other UI actor info:\n'
f'channel: {other._ctx.chan}]\n'
f'actor uid: {other._ctx.chan.uid}]\n'
f'ctx id: {other._ctx.cid}]\n'
f'channel: {other.ctx.chan}]\n'
f'actor uid: {other.ctx.chan.uid}]\n'
f'ctx id: {other.ctx.cid}]\n'
)
ctx2fqmes.setdefault(
@ -1301,12 +1426,28 @@ async def open_annot_ctl(
set(),
).add(fqme)
async with trionics.gather_contexts(stream_ctxs) as streams:
for (ctx, fqmes) in gap_ctxs:
gap_stream_ctxs.append(ctx.open_stream())
gap_ctx2fqmes[ctx.cid] = set(fqmes)
async with (
trionics.gather_contexts(stream_ctxs) as streams,
trionics.gather_contexts(
gap_stream_ctxs,
) as gap_streams,
):
for stream in streams:
fqmes: set[str] = ctx2fqmes[stream._ctx.cid]
fqmes: set[str] = ctx2fqmes[stream.ctx.cid]
for fqme in fqmes:
fqme2ipc[fqme] = stream
for stream in gap_streams:
fqmes: set[str] = gap_ctx2fqmes[stream.ctx.cid]
stream_lock: trio.Lock = trio.Lock()
for fqme in fqmes:
fqme2gap_ipc[fqme] = stream
gap_locks[fqme] = stream_lock
# NOTE: on graceful teardown we always attempt to
# remove all annots that were created by the
# entering client.
@ -1315,16 +1456,18 @@ async def open_annot_ctl(
# disconnects we always delete all annotations by
# default instaead of expecting the client to?
async with AsyncExitStack() as annots_stack:
client = AnnotCtl(
client: AnnotClient = AnnotClient(
ctx2fqmes=ctx2fqmes,
fqme2ipc=fqme2ipc,
fqme2gap_ipc=fqme2gap_ipc,
_gap_locks=gap_locks,
_annot_stack=annots_stack,
)
yield client
# client exited, measure teardown time
teardown_profiler = Profiler(
msg='Client AnnotCtl teardown',
msg='`AnnotClient` teardown',
disabled=False,
ms_threshold=0.0,
)

4
piker/ui/_widget.py
View File

@ -37,6 +37,7 @@ from piker.ui.qt import (
from ..log import get_logger
if TYPE_CHECKING:
from ._gaps import GapOverlayMngr
from ._search import SearchWidget
from ._chart import (
LinkedSplits,
@ -114,6 +115,7 @@ class GodWidget(QWidget):
# assigned in the startup func `_async_main()`
self._root_n: trio.Nursery = None
self.gapman: GapOverlayMngr|None = None
self._widgets: dict[str, QWidget] = {}
self._resizing: bool = False
@ -348,5 +350,3 @@ class GodWidget(QWidget):
self.rt_linked.resize_sidepanes()
self.hist_linked.resize_sidepanes(from_linked=rt_linked)
self.search.on_resize()

120
plans/opencode/chart-local-gap-overlays.md 100644
View File

@ -0,0 +1,120 @@
# Chart-Local Gap Overlays
## Goal
Make gap overlays a chart-actor-owned display service while preserving
remote annotation as a first-class API. Local startup and UI input avoid
an unnecessary IPC round trip; remote tools use a strict `msgspec` schema
and the same renderer/controller.
## Ownership Model
`GapOverlayMngr` is created once in the chart actor and references the
actor-global `_dss` and `_annots` registries. Each layer key is
`(owner, id(chart), fqme, timeframe)`. The chart identity keeps duplicate
FQMEs in separate cached displays independent. The reserved `chart-local`
owner is also independent from each remote `Context.cid`, so remote context
teardown cannot remove locally owned graphics.
The controller has one rendering entry point:
1. `GapOverlayMngr.apply(SetGapOverlay)` replaces an owner layer.
2. `GapOverlayMngr.refresh()` detects gaps from chart-local SHM and calls
`apply()` directly.
3. `AnnotClient.set_gap_overlay()` sends the same request through the
strict `remote_gap_overlays()` context.
The generic `remote_annotate()` context and its rectangle, arrow, text,
batch, remove and redraw commands remain unchanged.
## Startup And UI
`graphics_update_loop()` registers each new `DisplayState` in `_dss`, then
builds the default 60-second local overlay. The focused `ChartView` handles
`Ctrl+G` through its existing asynchronous keyboard loop. It chooses the
focused chart's 1-second or 60-second timeframe and calls
`GapOverlayMngr.toggle()` directly.
Gap detection reads the structured OHLCV array, excludes invalid timestamps,
uses one `numpy.diff()` pass, and creates Python `GapSpec` objects only for
actual positive timestamp gaps.
## Typed IPC
`remote_gap_overlays()` is separate from `remote_annotate()` because
`msgspec` cannot strictly decode a union containing both arbitrary legacy
dicts and several struct types. Its Tractor payload specification is:
```python
SetGapOverlay | GapOverlay | list[str] | None
```
`SetGapOverlay` and `GapOverlay` are tagged structs. Tractor validates
started, stream and return payloads against this union. A lock shared by
all FQMEs on one typed stream serializes each send/receive exchange. Each
side narrows its public `Context.pld_rx` decoder to the directional request
or response type while receiving. Request IDs let the client discard a late
response after an earlier timeout instead of skewing later request/reply
pairs.
## Refresh Semantics
Startup and every UI transition from hidden to visible recompute the layer
from current SHM. Existing redraw handling repositions matching overlays by
both FQME and timeframe. Rectangle and arrow geometry update together.
The next production step is to trigger `refresh()` from an explicit history
revision/publication event after a repair or prepend. Polling from the quote
display loop is intentionally excluded.
Request/reply serialization and stale-response correlation can move into a
generic IPC dialog helper after a second typed consumer establishes the
reusable interface. The legacy annotation dict protocol should first migrate
to tagged messages before sharing the strict gap context.
## Landing Order
The existing ancestry is:
```text
main
gap_annotator
datad_service
backfiller_deep_fixes
wkt/fix_broadcast_consumers
```
The recommended integration order is:
1. Review and land the complete 23-commit `main..gap_annotator` range.
2. Land Tractor PR 490 after its PR 475 dependency, then update the piker
Tractor reference used by `datad_service`.
3. Land `datad_service` on the new mainline.
4. Land the smaller four-commit broadcast-consumer PR 92.
5. Rebase `backfiller_deep_fixes` onto the broadcast result and run the
NativeDB/SHM/manual chart qualification sequence.
6. Replay the chart-local overlay patch above the qualified backfiller tip.
This order puts the already-reviewed ownership fix ahead of the broader
backfiller qualification. The only known overlapping production path is
`piker/brokers/ib/feed.py`; `git merge-tree --write-tree` reports no current
textual conflict, but behavior must still be rechecked.
No branch movement, rebase, merge, commit or push is part of this prototype.
## Verification
Run from the prototype worktree with the source-resolving Python environment:
```text
python -m pytest -p no:xonsh -q -x --tb=short --no-header \
tests/test_gap_overlays.py
python -m ruff check piker/ui/_gaps.py \
tests/test_gap_overlays.py piker/ui/_remote_ctl.py \
piker/ui/_display.py piker/ui/_interaction.py \
piker/ui/_widget.py piker/ui/_annotate.py
```
Manual chart qualification must confirm startup rendering, focused-timeframe
`Ctrl+G`, remote typed replacement, remote disconnect cleanup, cached symbol
switching and refresh after backfill repair.

27
plans/opencode/chart-local-gap-overlays.summary.md 100644
View File

@ -0,0 +1,27 @@
Own gap overlays inside the chart actor
- Add typed `SetGapOverlay` and `GapOverlay` payloads
- Detect gaps directly from chart-local OHLCV SHM
- Share `GapOverlayMngr` between local UI and remote callers
- Enable 60-second overlays at display startup
- Toggle the focused timeframe directly with `Ctrl+G`
- Preserve legacy `remote_annotate()` commands unchanged
- Isolate local and remote graphics ownership and teardown
- Document the stack landing and manual qualification order
## Deferred
- Trigger refresh from an explicit history revision event
- Run live Qt chart and remote actor qualification
- Review and land the full `gap_annotator` commit range
## Stats
- 10 production/test files changed
- 2 architecture plan files added
- 19 focused regressions passing
- 0 existing branches moved
(this patch was generated in some part by
[`opencode`][opencode-gh])
[opencode-gh]: https://github.com/sst/opencode

148
tests/test_gap_overlays.py 100644
View File

@ -0,0 +1,148 @@
'''
Typed chart-local gap-overlay regressions.
'''
import msgspec
import numpy as np
import pytest
from piker.ui._gaps import (
GapOverlay,
GapOverlayMngr,
GapOverlayPayload,
GapSpec,
SetGapOverlay,
gap_specs_from_ohlcv,
)
def test_gap_specs_and_wire_roundtrip() -> None:
'''
Local detection and remote IPC share one typed request model.
Build a small sorted OHLCV array with one positive timestamp gap,
derive its geometry without a per-row dataframe scan, and
roundtrip the request/reply through msgpack. This proves callers
invoke one chart renderer with one concrete schema.
'''
dtype: np.dtype = np.dtype([
('index', 'i8'),
('time', 'f8'),
('open', 'f8'),
('close', 'f8'),
])
array: np.ndarray = np.array([
(10, 60, 100, 101),
(11, 120, 101, 102),
(12, 300, 95, 96),
(13, 360, 96, 97),
], dtype=dtype)
specs: list[GapSpec] = gap_specs_from_ohlcv(
array,
period_s=60,
)
assert len(specs) == 1
spec: GapSpec = specs[0]
assert spec.start_time == 120
assert spec.end_time == 300
assert spec.start_pos == (11.9, 102)
assert spec.end_pos == (12.1, 95)
assert spec.pointing == 'down'
req: SetGapOverlay = SetGapOverlay(
fqme='mnq.cme.ib',
timeframe=60,
specs=specs,
request_id='req-616',
)
req_wire: bytes = msgspec.msgpack.encode(req)
assert msgspec.msgpack.decode(
req_wire,
type=SetGapOverlay,
) == req
resp: GapOverlay = GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=True,
gap_count=1,
aid=616,
request_id=req.request_id,
)
resp_wire: bytes = msgspec.msgpack.encode(resp)
assert msgspec.msgpack.decode(
resp_wire,
type=GapOverlay,
) == resp
def test_gap_overlay_payload_rejects_other_msgs() -> None:
'''
Reject payloads outside the strict gap-overlay dialog protocol.
The sibling Tractor context installs `GapOverlayPayload` so
legacy annotation dicts cannot enter its stream as partially
validated commands. Encode one such dict and prove the same
msgspec decoder rejects it before endpoint dispatch.
'''
invalid_wire: bytes = msgspec.msgpack.encode({
'cmd': 'SelectRect',
})
with pytest.raises(msgspec.ValidationError):
msgspec.msgpack.decode(
invalid_wire,
type=GapOverlayPayload,
)
req: SetGapOverlay = SetGapOverlay(
fqme='mnq.cme.ib',
timeframe=60,
specs=[],
)
resp: GapOverlay = GapOverlay(
fqme=req.fqme,
timeframe=req.timeframe,
visible=False,
gap_count=0,
)
with pytest.raises(msgspec.ValidationError):
msgspec.msgpack.decode(
msgspec.msgpack.encode(resp),
type=SetGapOverlay,
)
with pytest.raises(msgspec.ValidationError):
msgspec.msgpack.decode(
msgspec.msgpack.encode(req),
type=GapOverlay,
)
def test_gap_overlay_unknown_fqme_returns_typed_error() -> None:
'''
Keep a stale remote FQME request inside the typed dialog.
A chart can unload or replace display state after a client learns
its FQME set. Submit a valid request to an empty controller and
prove it returns `GapOverlay.error` instead of leaking a
`KeyError` which would terminate the whole Tractor context.
'''
gapman: GapOverlayMngr = GapOverlayMngr(
dss={},
annots={},
)
req: SetGapOverlay = SetGapOverlay(
fqme='gone.test',
timeframe=60,
specs=[],
request_id='req-stale',
)
resp: GapOverlay = gapman.apply(req)
assert resp.request_id == req.request_id
assert resp.error == 'No display state for fqme=gone.test'

4
tests/test_ldshm.py
View File

@ -204,7 +204,7 @@ def test_shm_invalid_snapshot_never_reaches_storage(
from piker.ui import _remote_ctl
monkeypatch.setattr(
_remote_ctl,
'open_annot_ctl',
'open_annot_client',
open_annotations,
)
@ -345,7 +345,7 @@ def test_shm_write_without_reload_uses_deduped_markup_frame(
from piker.ui import _remote_ctl
monkeypatch.setattr(
_remote_ctl,
'open_annot_ctl',
'open_annot_client',
open_annotations,
)