Repair IB startup gaps in `ShmArray`

Keep reverse history and stored SHM layout aligned so sparse IB
frames do not expose permanent zero-time reservations.

Deats,
- omit inclusive query endpoints from SHM but retain full storage
- serialize and bound farm resets, blanks, and cancellations
- run the final null sweep after reverse retrieval completes
- require valid timestamp boundaries for synthetic gap rows
- cover the live QQQ 653-row failure and reset interleavings

Prompt-IO: ai/prompt-io/opencode/20260729T204046Z_ad6c560f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
wkt/land_startup_storage
Gud Boi 2026-07-29 16:46:52 -04:00
parent 892898cfd0
commit 0d2c88ce3e
4 changed files with 407 additions and 82 deletions

View File

@ -0,0 +1,37 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 253ade33-07ba-45b5-8839-be35d17ec164
timestamp: 2026-07-29T20:40:46Z
git_ref: ad6c560f
scope: code
substantive: true
raw_file: 20260729T204046Z_ad6c560f_prompt_io.raw.md
---
## Prompt
The user reported that the QQQ 1-second chart still contained a gap after
the `ldshm` fix, asked the agent to inspect it, and authorized continued
implementation after a short pause and shutdown of the live workspace.
## Response summary
Diagnosed a 653-row active RT SHM null reservation caused by inclusive IB
reverse-query overlaps and startup repair orchestration. Prevented extra
physical SHM overlap rows, serialized and bounded IB history recovery,
made blank/error responses terminate, and constrained synthetic repair to
valid interior timestamp boundaries. Added deterministic concurrency and
data-layout regressions.
## Files changed
- `piker/brokers/ib/feed.py` - bound and serialize IB history reset/retry behavior.
- `piker/tsp/_history.py` - prevent SHM overlap skew and safely finalize null repair.
- `tests/test_ib_history.py` - cover blank, cancellation, reset, and pacing interleavings.
- `tests/test_history_backfill.py` - cover overlap and null-repair invariants.
- `.claude/skills/run-tests/test-harness-reference.md` - register deterministic coverage.
## Human edits
None - generated output remains uncommitted.

View File

@ -0,0 +1,72 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-07-29T20:40:46Z
git_ref: ad6c560f
diff_cmd: git diff HEAD~1..HEAD
---
The user reported that QQQ still displayed a gap on the 1-second chart
after the focused `ldshm` fix and asked the agent to inspect it. The user
then paused and resumed implementation while shutting down the live piker
trading workspace.
Read-only live inspection found the active datad generation held 653
contiguous zero-time rows in QQQ's 1-second RT SHM, while durable Parquet
contained no zero rows. The run occupied absolute indexes 168148 through
168800 between timestamps 1785348398 and 1785349051. The stale actor
generation and active 60-second history were separately identified.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Generated history changes that exclude each inclusive reverse-query
endpoint already present in SHM while retaining the full provider frame
for NativeDB merge. Startup null repair now runs after reverse backfill,
uses bounded UI notifications, handles provider no-data without an
interactive pause, fills only exact interior zero groups, and refuses
synthetic timestamps unless both valid boundaries establish the expected
cadence.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
Generated IB history reset changes that make blank responses complete,
bound repeated cancellation and reset failures per request, serialize
timeout and pacing farm resets with one actor-global Trio lock, carry
explicit reset results, and coordinate pacing handoff with queued timeout
waiters.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Generated regressions for inclusive endpoint exclusion, full NativeDB
provider deltas, strict synthetic gap boundaries, empty-provider local
fallback, bounded notifications, and preservation of valid margin rows.
> `git diff HEAD~1..HEAD -- tests/test_ib_history.py`
Generated deterministic regressions for blank IB responses, cancellation
limits, stalled reset cancellation, pacing reset failure limits,
cross-request reset serialization, timeout-to-pacing handoff, and
successful handoff timeout restart.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Registered the deterministic history-backfill target and expanded the IB
history mapping to include `feed.py`.
Verification output:
```text
........................................................................ [ 93%]
..... [100%]
77 passed in 2.12s
```
`git diff --check` passed. Multiple adversarial review rounds identified
and drove fixes for unbounded IB empty/cancellation/reset paths, stale
reset ownership, actor-global failure poisoning, concurrent farm resets,
double-counted pacing handoffs, physical endpoint overlap, and unsafe
synthetic timestamp boundaries. Final review reported no findings.
Live validation was intentionally deferred because the user shut down the
trading workspace. A fresh datad/chart startup is required to regenerate
SHM and verify that QQQ no longer contains the 653-row null run.

View File

@ -206,6 +206,34 @@ async def notify_backfill(
) )
def _synthetic_gap_times(
start_t: float,
right_t: float|None,
count: int,
timeframe: float,
) -> np.ndarray|None:
'''
Build bounded synthetic timestamps without crossing valid data.
'''
times: np.ndarray = (
start_t
+
np.arange(1, count + 1)*timeframe
)
if right_t is None:
return None
if not np.isclose(
right_t,
start_t + (count + 1)*timeframe,
):
return None
return times
async def shm_push_in_between( async def shm_push_in_between(
shm: ShmArray, shm: ShmArray,
to_push: np.ndarray, to_push: np.ndarray,
@ -262,11 +290,11 @@ async def maybe_fill_null_segments(
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED, task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
) -> list[Frame]: ) -> None:
task_status.started() task_status.started()
frame: Frame = shm.array frame: Frame = shm.array.copy()
# TODO, put in parent task/daemon root! # TODO, put in parent task/daemon root!
import greenback import greenback
@ -330,7 +358,6 @@ async def maybe_fill_null_segments(
) )
# ?TODO? do we want to remove the nulls and push # ?TODO? do we want to remove the nulls and push
# the close price here for the gap duration? # the close price here for the gap duration?
await tractor.pause()
break break
if ( if (
@ -343,7 +370,7 @@ async def maybe_fill_null_segments(
f'frame_start_dt: {frame_start_dt!r}\n' f'frame_start_dt: {frame_start_dt!r}\n'
f'backfill_until_dt: {backfill_until_dt!r}\n' f'backfill_until_dt: {backfill_until_dt!r}\n'
) )
await tractor.pause() break
# XXX TODO: pretty sure if i plot tsla, btcusdt.binance # XXX TODO: pretty sure if i plot tsla, btcusdt.binance
# and mnq.cme.ib this causes a Qt crash XXDDD # and mnq.cme.ib this causes a Qt crash XXDDD
@ -365,21 +392,11 @@ async def maybe_fill_null_segments(
# - remember that in the display side, only refersh this # - remember that in the display side, only refersh this
# if the respective history is actually "in view". # if the respective history is actually "in view".
# loop # loop
try: await notify_backfill(
await sampler_stream.send({ sampler_stream,
'broadcast_all': { mkt,
timeframe,
# XXX NOTE XXX: see the )
# `.ui._display.increment_history_view()` if block
# that looks for this info to FORCE a hard viz
# redraw!
'backfilling': (mkt.fqme, timeframe),
},
})
except tractor.ContextCancelled:
# log.exception
await tractor.pause()
raise
# RECHECK for more null-gaps # RECHECK for more null-gaps
frame: Frame = shm.array frame: Frame = shm.array
@ -395,7 +412,7 @@ async def maybe_fill_null_segments(
( (
iabs_slices, iabs_slices,
iabs_zero_rows, iabs_zero_rows,
zero_t, _zero_t,
) = null_segs ) = null_segs
log.warning( log.warning(
f'{len(iabs_slices)} NULL TIME SEGMENTS DETECTED!\n' f'{len(iabs_slices)} NULL TIME SEGMENTS DETECTED!\n'
@ -414,23 +431,61 @@ async def maybe_fill_null_segments(
'close', 'close',
] ]
for istart, istop in iabs_slices: zero_indexes: np.ndarray = np.asarray(iabs_zero_rows)
split_at: np.ndarray = (
# Synthetic fallback also requires valid boundary bars. np.flatnonzero(np.diff(zero_indexes) != 1)
start_t: float = shm._array[istart]['time'] +
end_t: float = shm._array[istop]['time'] 1
if ( )
start_t <= 0 zero_groups: list[np.ndarray] = np.split(
or zero_indexes,
end_t <= start_t split_at,
): )
repaired: int = 0
frame_start: int = int(frame['index'][0])
for zero_group in zero_groups:
istart: int = int(zero_group[0])
istop: int = int(zero_group[-1]) + 1
seed_index: int = istart - 1
if seed_index < frame_start:
log.warning(
f'Can not forward-fill leading SHM nulls:\n'
f'{istart}:{istop}\n'
)
continue continue
# get view into buffer for null-segment seed: np.void = shm._array[seed_index]
gap: np.ndarray = shm._array[istart:istop] start_t: float = seed['time']
if start_t <= 0:
log.warning(
f'Can not forward-fill SHM nulls from '
f'zero-time seed at {seed_index}\n'
)
continue
# copy the oldest OHLC samples forward # Fill only the null rows, preserving both valid margins.
cls: float = shm._array[istart]['close'] gap: np.ndarray = shm._array[istart:istop]
cls: float = seed['close']
frame_end: int = int(frame['index'][-1])
right_t: float|None = (
shm._array[istop]['time']
if istop <= frame_end
else None
)
gap_times: np.ndarray|None = _synthetic_gap_times(
start_t,
right_t,
len(gap),
timeframe,
)
if gap_times is None:
log.warning(
f'Can not safely forward-fill SHM nulls:\n'
f'left={start_t} right={right_t} '
f'rows={len(gap)} period={timeframe}\n'
)
continue
# TODO: how can we mark this range as being a gap tho? # TODO: how can we mark this range as being a gap tho?
# -[ ] maybe pg finally supports nulls in ndarray to # -[ ] maybe pg finally supports nulls in ndarray to
@ -439,30 +494,13 @@ async def maybe_fill_null_segments(
# another col/field to denote? # another col/field to denote?
gap[ohlc_fields] = cls gap[ohlc_fields] = cls
start_t: float = shm._array[istart]['time'] gap['time'] = gap_times
t_diff: float = (istop - istart)*timeframe repaired += len(gap)
gap['time'] = np.arange(
start=start_t,
stop=start_t + t_diff,
step=timeframe,
)
# TODO: reimpl using the new `.ui._remote_ctl` ctx # TODO: reimpl using the new `.ui._remote_ctl` ctx
# ideally using some kinda decent # ideally using some kinda decent
# tractory-reverse-lookup-connnection from some other # tractory-reverse-lookup-connnection from some other
# `Context` type thingy? # `Context` type thingy?
await sampler_stream.send({
'broadcast_all': {
# XXX NOTE XXX: see the
# `.ui._display.increment_history_view()` if block
# that looks for this info to FORCE a hard viz
# redraw!
'backfilling': (mkt.fqme, timeframe),
},
})
# TODO: interatively step through any remaining # TODO: interatively step through any remaining
# time-gaps/null-segments and spawn piecewise backfiller # time-gaps/null-segments and spawn piecewise backfiller
# tasks in a nursery? # tasks in a nursery?
@ -473,6 +511,13 @@ async def maybe_fill_null_segments(
# -[ ] fill algo: do queries in alternating "latest, then # -[ ] fill algo: do queries in alternating "latest, then
# earliest, then latest.. etc?" # earliest, then latest.. etc?"
if repaired:
await notify_backfill(
sampler_stream,
mkt,
timeframe,
)
async def start_backfill( async def start_backfill(
get_hist, get_hist,
@ -706,7 +751,12 @@ async def start_backfill(
array, array,
prepend_until_dt=backfill_until_dt, prepend_until_dt=backfill_until_dt,
) )
to_push: np.ndarray = to_store # The requested end bar is already the oldest published
# sample. Keep it for idempotent storage merge, but do not
# consume another physical SHM row for the overlap.
to_push: np.ndarray = to_store[
to_store['time'] < last_start_dt.timestamp()
]
ln: int = len(to_push) ln: int = len(to_push)
if ln: if ln:
log.info( log.info(
@ -1340,27 +1390,25 @@ async def tsdb_backfill(
log.info(f'Loaded {to_push.shape} datums from storage') log.info(f'Loaded {to_push.shape} datums from storage')
# NOTE: ASYNC-conduct tsdb timestamp gap detection and backfill any # 2nd nursery END
# seemingly missing (null-time) segments..
# TODO: ideally these can never exist! if last_tsdb_dt is not None:
# Repair only after reverse backfill releases providers
# such as IB which permit one history request at a time.
# TODO: ideally these null segments can never exist!
# -[ ] somehow it seems sometimes we're writing zero-ed # -[ ] somehow it seems sometimes we're writing zero-ed
# segments to tsdbs during teardown? # segments to tsdbs during teardown?
# -[ ] can we ensure that the backfiller tasks do this # -[ ] can we ensure that the backfiller tasks do this
# work PREVENTAVELY instead? # work PREVENTAVELY instead?
# -[ ] fill in non-zero epoch time values ALWAYS! # -[ ] fill in non-zero epoch time values ALWAYS!
# await maybe_fill_null_segments( await maybe_fill_null_segments(
await tn.start(partial(
maybe_fill_null_segments,
shm=shm, shm=shm,
timeframe=timeframe, timeframe=timeframe,
get_hist=get_hist, get_hist=get_hist,
sampler_stream=sampler_stream, sampler_stream=sampler_stream,
mkt=mkt, mkt=mkt,
backfill_until_dt=last_tsdb_dt, backfill_until_dt=last_tsdb_dt,
)) )
# 2nd nursery END
# TODO: maybe start history anal and load missing "history # TODO: maybe start history anal and load missing "history
# gaps" via backend.. # gaps" via backend..

View File

@ -33,6 +33,7 @@ from piker.storage.nativedb import (
ohlc_key_map, ohlc_key_map,
) )
from piker.tsp._history import ( from piker.tsp._history import (
_synthetic_gap_times,
maybe_fill_null_segments, maybe_fill_null_segments,
notify_backfill, notify_backfill,
publish_latest_frame, publish_latest_frame,
@ -224,15 +225,22 @@ def test_empty_frame_completes() -> None:
def test_storage_receives_full_provider_delta() -> None: def test_storage_receives_full_provider_delta() -> None:
''' '''
SHM capacity truncation does not truncate durable history. SHM drops the published endpoint without truncating storage.
Reverse history queries include their requested end bar, which is
already the oldest sample in SHM. Publishing the overlap consumed
an extra physical row and made reserved null counts disagree with
elapsed timestamps. Return a frame with the overlap and prove
NativeDB receives the complete provider delta while SHM receives
only bars older than its published boundary.
''' '''
frame = np.zeros( frame = np.zeros(
2, 3,
dtype=np.dtype(def_iohlcv_fields), dtype=np.dtype(def_iohlcv_fields),
) )
frame['index'] = [0, 1] frame['index'] = [0, 1, 2]
frame['time'] = [60, 120] frame['time'] = [60, 120, 180]
events: list[str] = [] events: list[str] = []
@ -274,7 +282,7 @@ def test_storage_receives_full_provider_delta() -> None:
return ( return (
frame, frame,
from_timestamp(60), from_timestamp(60),
from_timestamp(120), from_timestamp(180),
) )
async def main() -> None: async def main() -> None:
@ -286,7 +294,7 @@ def test_storage_receives_full_provider_delta() -> None:
mkt=mkt, mkt=mkt,
shm=shm, shm=shm,
timeframe=60, timeframe=60,
backfill_from_shm_index=1, backfill_from_shm_index=2,
backfill_from_dt=from_timestamp(180), backfill_from_dt=from_timestamp(180),
sampler_stream=Sampler(), sampler_stream=Sampler(),
backfill_until_dt=from_timestamp(60), backfill_until_dt=from_timestamp(60),
@ -295,8 +303,8 @@ def test_storage_receives_full_provider_delta() -> None:
) )
trio.run(main) trio.run(main)
assert shm.pushed[0]['time'].tolist() == [120] assert shm.pushed[0]['time'].tolist() == [60, 120]
assert storage.frames[0]['time'].tolist() == [60, 120] assert storage.frames[0]['time'].tolist() == [60, 120, 180]
assert events == ['storage', 'shm', 'sampler'] assert events == ['storage', 'shm', 'sampler']
@ -476,3 +484,163 @@ def test_backfill_notification_timeout_is_bounded(
) )
trio.run(main) trio.run(main)
def test_synthetic_gap_times_require_valid_right_boundary() -> None:
'''
Synthetic rows must remain strictly between valid boundary bars.
The live QQQ reservation held one more physical zero row than its
timestamp interval could represent. Blind fill would assign the
null the right boundary's timestamp, creating a duplicate. Prove
aligned bounds produce cadence while an overfull segment
is rejected instead of inventing non-monotonic timestamps.
'''
aligned = _synthetic_gap_times(
start_t=60,
right_t=300,
count=3,
timeframe=60,
)
overfull = _synthetic_gap_times(
start_t=60,
right_t=240,
count=3,
timeframe=60,
)
trailing = _synthetic_gap_times(
start_t=60,
right_t=None,
count=3,
timeframe=60,
)
assert aligned is not None
assert aligned.tolist() == [120, 180, 240]
assert overfull is None
assert trailing is None
def test_null_repair_falls_back_without_debug_pause(
monkeypatch: pytest.MonkeyPatch,
) -> None:
'''
Empty provider repair must resolve interior zero rows locally.
QQQ startup exposed 653 zero-time rows between stored history and
an IB frame. The null repair queried IB while reverse backfill
occupied its sole history request slot, then could
enter an interactive debug pause before reaching forward-fill.
Arrange an interior null run, return an empty provider frame, and
wedge sampler notification. A short timeout and fail-fast pause
prove local fallback needs no interactive or IPC progress.
The assertions verify only the
zero rows inherit the preceding close and synthesized timestamps;
both valid boundary rows remain unchanged.
'''
monkeypatch.setattr(
'piker.tsp._history._notify_timeout_s',
0.01,
)
backing = np.zeros(
105,
dtype=np.dtype(def_iohlcv_fields),
)
backing['index'] = np.arange(105)
frame = backing[100:105]
frame['index'] = np.arange(100, 105)
frame['time'] = [60, 0, 0, 0, 300]
frame['open'] = [9, 0, 0, 0, 19]
frame['high'] = [11, 0, 0, 0, 21]
frame['low'] = [8, 0, 0, 0, 18]
frame['close'] = [10, 0, 0, 0, 20]
frame['volume'] = [5, 0, 0, 0, 6]
class Shm:
'''
Expose the test frame through the SHM repair interface.
'''
def __init__(self) -> None:
self._array = backing
@property
def array(self) -> np.ndarray:
'''
Return the readable SHM view.
'''
return self._array[100:105]
class Sampler:
'''
Model a backpressured UI notification stream.
'''
async def send(self, msg) -> None:
'''
Block until the bounded notifier cancels this send.
'''
await trio.sleep_forever()
async def get_hist(*args, **kwargs):
'''
Return no provider bars for the requested null segment.
'''
end_dt = kwargs['end_dt']
empty = np.empty(
0,
dtype=np.dtype(def_iohlcv_fields),
)
return empty, end_dt, end_dt
async def fail_pause() -> None:
'''
Reject interactive debugging in normal repair fallbacks.
'''
raise AssertionError('null repair entered tractor.pause()')
monkeypatch.setattr(tractor, 'pause', fail_pause)
async def main() -> None:
import greenback
async def ensure_portal() -> None:
'''
Avoid installing a greenback portal in this unit test.
'''
monkeypatch.setattr(
greenback,
'ensure_portal',
ensure_portal,
)
with trio.fail_after(0.1):
await maybe_fill_null_segments(
shm=Shm(),
timeframe=60,
get_hist=get_hist,
sampler_stream=Sampler(),
mkt=SimpleNamespace(fqme='qqq.nasdaq.ib'),
backfill_until_dt=from_timestamp(60),
)
trio.run(main)
assert frame['time'].tolist() == [60, 120, 180, 240, 300]
for field in ('open', 'high', 'low', 'close'):
assert frame[field].tolist() == [
frame[field][0],
10,
10,
10,
frame[field][-1],
]
assert frame['volume'].tolist() == [5, 0, 0, 0, 6]