Repair IB startup gaps in `ShmArray`
Keep reverse history and stored SHM layout aligned so sparse IB frames do not expose permanent zero-time reservations. Deats, - omit inclusive query endpoints from SHM but retain full storage - serialize and bound farm resets, blanks, and cancellations - run the final null sweep after reverse retrieval completes - require valid timestamp boundaries for synthetic gap rows - cover the live QQQ 653-row failure and reset interleavings Prompt-IO: ai/prompt-io/opencode/20260729T204046Z_ad6c560f_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))wkt/land_startup_storage
parent
892898cfd0
commit
0d2c88ce3e
|
|
@ -0,0 +1,37 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: 253ade33-07ba-45b5-8839-be35d17ec164
|
||||||
|
timestamp: 2026-07-29T20:40:46Z
|
||||||
|
git_ref: ad6c560f
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260729T204046Z_ad6c560f_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user reported that the QQQ 1-second chart still contained a gap after
|
||||||
|
the `ldshm` fix, asked the agent to inspect it, and authorized continued
|
||||||
|
implementation after a short pause and shutdown of the live workspace.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Diagnosed a 653-row active RT SHM null reservation caused by inclusive IB
|
||||||
|
reverse-query overlaps and startup repair orchestration. Prevented extra
|
||||||
|
physical SHM overlap rows, serialized and bounded IB history recovery,
|
||||||
|
made blank/error responses terminate, and constrained synthetic repair to
|
||||||
|
valid interior timestamp boundaries. Added deterministic concurrency and
|
||||||
|
data-layout regressions.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/brokers/ib/feed.py` - bound and serialize IB history reset/retry behavior.
|
||||||
|
- `piker/tsp/_history.py` - prevent SHM overlap skew and safely finalize null repair.
|
||||||
|
- `tests/test_ib_history.py` - cover blank, cancellation, reset, and pacing interleavings.
|
||||||
|
- `tests/test_history_backfill.py` - cover overlap and null-repair invariants.
|
||||||
|
- `.claude/skills/run-tests/test-harness-reference.md` - register deterministic coverage.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated output remains uncommitted.
|
||||||
|
|
@ -0,0 +1,72 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-07-29T20:40:46Z
|
||||||
|
git_ref: ad6c560f
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
The user reported that QQQ still displayed a gap on the 1-second chart
|
||||||
|
after the focused `ldshm` fix and asked the agent to inspect it. The user
|
||||||
|
then paused and resumed implementation while shutting down the live piker
|
||||||
|
trading workspace.
|
||||||
|
|
||||||
|
Read-only live inspection found the active datad generation held 653
|
||||||
|
contiguous zero-time rows in QQQ's 1-second RT SHM, while durable Parquet
|
||||||
|
contained no zero rows. The run occupied absolute indexes 168148 through
|
||||||
|
168800 between timestamps 1785348398 and 1785349051. The stale actor
|
||||||
|
generation and active 60-second history were separately identified.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||||
|
|
||||||
|
Generated history changes that exclude each inclusive reverse-query
|
||||||
|
endpoint already present in SHM while retaining the full provider frame
|
||||||
|
for NativeDB merge. Startup null repair now runs after reverse backfill,
|
||||||
|
uses bounded UI notifications, handles provider no-data without an
|
||||||
|
interactive pause, fills only exact interior zero groups, and refuses
|
||||||
|
synthetic timestamps unless both valid boundaries establish the expected
|
||||||
|
cadence.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
||||||
|
|
||||||
|
Generated IB history reset changes that make blank responses complete,
|
||||||
|
bound repeated cancellation and reset failures per request, serialize
|
||||||
|
timeout and pacing farm resets with one actor-global Trio lock, carry
|
||||||
|
explicit reset results, and coordinate pacing handoff with queued timeout
|
||||||
|
waiters.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||||
|
|
||||||
|
Generated regressions for inclusive endpoint exclusion, full NativeDB
|
||||||
|
provider deltas, strict synthetic gap boundaries, empty-provider local
|
||||||
|
fallback, bounded notifications, and preservation of valid margin rows.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_ib_history.py`
|
||||||
|
|
||||||
|
Generated deterministic regressions for blank IB responses, cancellation
|
||||||
|
limits, stalled reset cancellation, pacing reset failure limits,
|
||||||
|
cross-request reset serialization, timeout-to-pacing handoff, and
|
||||||
|
successful handoff timeout restart.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
|
||||||
|
Registered the deterministic history-backfill target and expanded the IB
|
||||||
|
history mapping to include `feed.py`.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
..... [100%]
|
||||||
|
77 passed in 2.12s
|
||||||
|
```
|
||||||
|
|
||||||
|
`git diff --check` passed. Multiple adversarial review rounds identified
|
||||||
|
and drove fixes for unbounded IB empty/cancellation/reset paths, stale
|
||||||
|
reset ownership, actor-global failure poisoning, concurrent farm resets,
|
||||||
|
double-counted pacing handoffs, physical endpoint overlap, and unsafe
|
||||||
|
synthetic timestamp boundaries. Final review reported no findings.
|
||||||
|
|
||||||
|
Live validation was intentionally deferred because the user shut down the
|
||||||
|
trading workspace. A fresh datad/chart startup is required to regenerate
|
||||||
|
SHM and verify that QQQ no longer contains the 653-row null run.
|
||||||
|
|
@ -206,6 +206,34 @@ async def notify_backfill(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _synthetic_gap_times(
|
||||||
|
start_t: float,
|
||||||
|
right_t: float|None,
|
||||||
|
count: int,
|
||||||
|
timeframe: float,
|
||||||
|
|
||||||
|
) -> np.ndarray|None:
|
||||||
|
'''
|
||||||
|
Build bounded synthetic timestamps without crossing valid data.
|
||||||
|
|
||||||
|
'''
|
||||||
|
times: np.ndarray = (
|
||||||
|
start_t
|
||||||
|
+
|
||||||
|
np.arange(1, count + 1)*timeframe
|
||||||
|
)
|
||||||
|
if right_t is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if not np.isclose(
|
||||||
|
right_t,
|
||||||
|
start_t + (count + 1)*timeframe,
|
||||||
|
):
|
||||||
|
return None
|
||||||
|
|
||||||
|
return times
|
||||||
|
|
||||||
|
|
||||||
async def shm_push_in_between(
|
async def shm_push_in_between(
|
||||||
shm: ShmArray,
|
shm: ShmArray,
|
||||||
to_push: np.ndarray,
|
to_push: np.ndarray,
|
||||||
|
|
@ -262,11 +290,11 @@ async def maybe_fill_null_segments(
|
||||||
|
|
||||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||||
|
|
||||||
) -> list[Frame]:
|
) -> None:
|
||||||
|
|
||||||
task_status.started()
|
task_status.started()
|
||||||
|
|
||||||
frame: Frame = shm.array
|
frame: Frame = shm.array.copy()
|
||||||
|
|
||||||
# TODO, put in parent task/daemon root!
|
# TODO, put in parent task/daemon root!
|
||||||
import greenback
|
import greenback
|
||||||
|
|
@ -330,7 +358,6 @@ async def maybe_fill_null_segments(
|
||||||
)
|
)
|
||||||
# ?TODO? do we want to remove the nulls and push
|
# ?TODO? do we want to remove the nulls and push
|
||||||
# the close price here for the gap duration?
|
# the close price here for the gap duration?
|
||||||
await tractor.pause()
|
|
||||||
break
|
break
|
||||||
|
|
||||||
if (
|
if (
|
||||||
|
|
@ -343,7 +370,7 @@ async def maybe_fill_null_segments(
|
||||||
f'frame_start_dt: {frame_start_dt!r}\n'
|
f'frame_start_dt: {frame_start_dt!r}\n'
|
||||||
f'backfill_until_dt: {backfill_until_dt!r}\n'
|
f'backfill_until_dt: {backfill_until_dt!r}\n'
|
||||||
)
|
)
|
||||||
await tractor.pause()
|
break
|
||||||
|
|
||||||
# XXX TODO: pretty sure if i plot tsla, btcusdt.binance
|
# XXX TODO: pretty sure if i plot tsla, btcusdt.binance
|
||||||
# and mnq.cme.ib this causes a Qt crash XXDDD
|
# and mnq.cme.ib this causes a Qt crash XXDDD
|
||||||
|
|
@ -365,21 +392,11 @@ async def maybe_fill_null_segments(
|
||||||
# - remember that in the display side, only refersh this
|
# - remember that in the display side, only refersh this
|
||||||
# if the respective history is actually "in view".
|
# if the respective history is actually "in view".
|
||||||
# loop
|
# loop
|
||||||
try:
|
await notify_backfill(
|
||||||
await sampler_stream.send({
|
sampler_stream,
|
||||||
'broadcast_all': {
|
mkt,
|
||||||
|
timeframe,
|
||||||
# XXX NOTE XXX: see the
|
)
|
||||||
# `.ui._display.increment_history_view()` if block
|
|
||||||
# that looks for this info to FORCE a hard viz
|
|
||||||
# redraw!
|
|
||||||
'backfilling': (mkt.fqme, timeframe),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
except tractor.ContextCancelled:
|
|
||||||
# log.exception
|
|
||||||
await tractor.pause()
|
|
||||||
raise
|
|
||||||
|
|
||||||
# RECHECK for more null-gaps
|
# RECHECK for more null-gaps
|
||||||
frame: Frame = shm.array
|
frame: Frame = shm.array
|
||||||
|
|
@ -395,7 +412,7 @@ async def maybe_fill_null_segments(
|
||||||
(
|
(
|
||||||
iabs_slices,
|
iabs_slices,
|
||||||
iabs_zero_rows,
|
iabs_zero_rows,
|
||||||
zero_t,
|
_zero_t,
|
||||||
) = null_segs
|
) = null_segs
|
||||||
log.warning(
|
log.warning(
|
||||||
f'{len(iabs_slices)} NULL TIME SEGMENTS DETECTED!\n'
|
f'{len(iabs_slices)} NULL TIME SEGMENTS DETECTED!\n'
|
||||||
|
|
@ -414,23 +431,61 @@ async def maybe_fill_null_segments(
|
||||||
'close',
|
'close',
|
||||||
]
|
]
|
||||||
|
|
||||||
for istart, istop in iabs_slices:
|
zero_indexes: np.ndarray = np.asarray(iabs_zero_rows)
|
||||||
|
split_at: np.ndarray = (
|
||||||
# Synthetic fallback also requires valid boundary bars.
|
np.flatnonzero(np.diff(zero_indexes) != 1)
|
||||||
start_t: float = shm._array[istart]['time']
|
+
|
||||||
end_t: float = shm._array[istop]['time']
|
1
|
||||||
if (
|
)
|
||||||
start_t <= 0
|
zero_groups: list[np.ndarray] = np.split(
|
||||||
or
|
zero_indexes,
|
||||||
end_t <= start_t
|
split_at,
|
||||||
):
|
)
|
||||||
|
repaired: int = 0
|
||||||
|
frame_start: int = int(frame['index'][0])
|
||||||
|
for zero_group in zero_groups:
|
||||||
|
istart: int = int(zero_group[0])
|
||||||
|
istop: int = int(zero_group[-1]) + 1
|
||||||
|
seed_index: int = istart - 1
|
||||||
|
if seed_index < frame_start:
|
||||||
|
log.warning(
|
||||||
|
f'Can not forward-fill leading SHM nulls:\n'
|
||||||
|
f'{istart}:{istop}\n'
|
||||||
|
)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# get view into buffer for null-segment
|
seed: np.void = shm._array[seed_index]
|
||||||
gap: np.ndarray = shm._array[istart:istop]
|
start_t: float = seed['time']
|
||||||
|
if start_t <= 0:
|
||||||
|
log.warning(
|
||||||
|
f'Can not forward-fill SHM nulls from '
|
||||||
|
f'zero-time seed at {seed_index}\n'
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
# copy the oldest OHLC samples forward
|
# Fill only the null rows, preserving both valid margins.
|
||||||
cls: float = shm._array[istart]['close']
|
gap: np.ndarray = shm._array[istart:istop]
|
||||||
|
cls: float = seed['close']
|
||||||
|
|
||||||
|
frame_end: int = int(frame['index'][-1])
|
||||||
|
right_t: float|None = (
|
||||||
|
shm._array[istop]['time']
|
||||||
|
if istop <= frame_end
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
gap_times: np.ndarray|None = _synthetic_gap_times(
|
||||||
|
start_t,
|
||||||
|
right_t,
|
||||||
|
len(gap),
|
||||||
|
timeframe,
|
||||||
|
)
|
||||||
|
if gap_times is None:
|
||||||
|
log.warning(
|
||||||
|
f'Can not safely forward-fill SHM nulls:\n'
|
||||||
|
f'left={start_t} right={right_t} '
|
||||||
|
f'rows={len(gap)} period={timeframe}\n'
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
# TODO: how can we mark this range as being a gap tho?
|
# TODO: how can we mark this range as being a gap tho?
|
||||||
# -[ ] maybe pg finally supports nulls in ndarray to
|
# -[ ] maybe pg finally supports nulls in ndarray to
|
||||||
|
|
@ -439,30 +494,13 @@ async def maybe_fill_null_segments(
|
||||||
# another col/field to denote?
|
# another col/field to denote?
|
||||||
gap[ohlc_fields] = cls
|
gap[ohlc_fields] = cls
|
||||||
|
|
||||||
start_t: float = shm._array[istart]['time']
|
gap['time'] = gap_times
|
||||||
t_diff: float = (istop - istart)*timeframe
|
repaired += len(gap)
|
||||||
|
|
||||||
gap['time'] = np.arange(
|
|
||||||
start=start_t,
|
|
||||||
stop=start_t + t_diff,
|
|
||||||
step=timeframe,
|
|
||||||
)
|
|
||||||
|
|
||||||
# TODO: reimpl using the new `.ui._remote_ctl` ctx
|
# TODO: reimpl using the new `.ui._remote_ctl` ctx
|
||||||
# ideally using some kinda decent
|
# ideally using some kinda decent
|
||||||
# tractory-reverse-lookup-connnection from some other
|
# tractory-reverse-lookup-connnection from some other
|
||||||
# `Context` type thingy?
|
# `Context` type thingy?
|
||||||
await sampler_stream.send({
|
|
||||||
'broadcast_all': {
|
|
||||||
|
|
||||||
# XXX NOTE XXX: see the
|
|
||||||
# `.ui._display.increment_history_view()` if block
|
|
||||||
# that looks for this info to FORCE a hard viz
|
|
||||||
# redraw!
|
|
||||||
'backfilling': (mkt.fqme, timeframe),
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
# TODO: interatively step through any remaining
|
# TODO: interatively step through any remaining
|
||||||
# time-gaps/null-segments and spawn piecewise backfiller
|
# time-gaps/null-segments and spawn piecewise backfiller
|
||||||
# tasks in a nursery?
|
# tasks in a nursery?
|
||||||
|
|
@ -473,6 +511,13 @@ async def maybe_fill_null_segments(
|
||||||
# -[ ] fill algo: do queries in alternating "latest, then
|
# -[ ] fill algo: do queries in alternating "latest, then
|
||||||
# earliest, then latest.. etc?"
|
# earliest, then latest.. etc?"
|
||||||
|
|
||||||
|
if repaired:
|
||||||
|
await notify_backfill(
|
||||||
|
sampler_stream,
|
||||||
|
mkt,
|
||||||
|
timeframe,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def start_backfill(
|
async def start_backfill(
|
||||||
get_hist,
|
get_hist,
|
||||||
|
|
@ -706,7 +751,12 @@ async def start_backfill(
|
||||||
array,
|
array,
|
||||||
prepend_until_dt=backfill_until_dt,
|
prepend_until_dt=backfill_until_dt,
|
||||||
)
|
)
|
||||||
to_push: np.ndarray = to_store
|
# The requested end bar is already the oldest published
|
||||||
|
# sample. Keep it for idempotent storage merge, but do not
|
||||||
|
# consume another physical SHM row for the overlap.
|
||||||
|
to_push: np.ndarray = to_store[
|
||||||
|
to_store['time'] < last_start_dt.timestamp()
|
||||||
|
]
|
||||||
ln: int = len(to_push)
|
ln: int = len(to_push)
|
||||||
if ln:
|
if ln:
|
||||||
log.info(
|
log.info(
|
||||||
|
|
@ -1340,28 +1390,26 @@ async def tsdb_backfill(
|
||||||
|
|
||||||
log.info(f'Loaded {to_push.shape} datums from storage')
|
log.info(f'Loaded {to_push.shape} datums from storage')
|
||||||
|
|
||||||
# NOTE: ASYNC-conduct tsdb timestamp gap detection and backfill any
|
|
||||||
# seemingly missing (null-time) segments..
|
|
||||||
# TODO: ideally these can never exist!
|
|
||||||
# -[ ] somehow it seems sometimes we're writing zero-ed
|
|
||||||
# segments to tsdbs during teardown?
|
|
||||||
# -[ ] can we ensure that the backfiller tasks do this
|
|
||||||
# work PREVENTAVELY instead?
|
|
||||||
# -[ ] fill in non-zero epoch time values ALWAYS!
|
|
||||||
# await maybe_fill_null_segments(
|
|
||||||
await tn.start(partial(
|
|
||||||
maybe_fill_null_segments,
|
|
||||||
|
|
||||||
shm=shm,
|
|
||||||
timeframe=timeframe,
|
|
||||||
get_hist=get_hist,
|
|
||||||
sampler_stream=sampler_stream,
|
|
||||||
mkt=mkt,
|
|
||||||
backfill_until_dt=last_tsdb_dt,
|
|
||||||
))
|
|
||||||
|
|
||||||
# 2nd nursery END
|
# 2nd nursery END
|
||||||
|
|
||||||
|
if last_tsdb_dt is not None:
|
||||||
|
# Repair only after reverse backfill releases providers
|
||||||
|
# such as IB which permit one history request at a time.
|
||||||
|
# TODO: ideally these null segments can never exist!
|
||||||
|
# -[ ] somehow it seems sometimes we're writing zero-ed
|
||||||
|
# segments to tsdbs during teardown?
|
||||||
|
# -[ ] can we ensure that the backfiller tasks do this
|
||||||
|
# work PREVENTAVELY instead?
|
||||||
|
# -[ ] fill in non-zero epoch time values ALWAYS!
|
||||||
|
await maybe_fill_null_segments(
|
||||||
|
shm=shm,
|
||||||
|
timeframe=timeframe,
|
||||||
|
get_hist=get_hist,
|
||||||
|
sampler_stream=sampler_stream,
|
||||||
|
mkt=mkt,
|
||||||
|
backfill_until_dt=last_tsdb_dt,
|
||||||
|
)
|
||||||
|
|
||||||
# TODO: maybe start history anal and load missing "history
|
# TODO: maybe start history anal and load missing "history
|
||||||
# gaps" via backend..
|
# gaps" via backend..
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ from piker.storage.nativedb import (
|
||||||
ohlc_key_map,
|
ohlc_key_map,
|
||||||
)
|
)
|
||||||
from piker.tsp._history import (
|
from piker.tsp._history import (
|
||||||
|
_synthetic_gap_times,
|
||||||
maybe_fill_null_segments,
|
maybe_fill_null_segments,
|
||||||
notify_backfill,
|
notify_backfill,
|
||||||
publish_latest_frame,
|
publish_latest_frame,
|
||||||
|
|
@ -224,15 +225,22 @@ def test_empty_frame_completes() -> None:
|
||||||
|
|
||||||
def test_storage_receives_full_provider_delta() -> None:
|
def test_storage_receives_full_provider_delta() -> None:
|
||||||
'''
|
'''
|
||||||
SHM capacity truncation does not truncate durable history.
|
SHM drops the published endpoint without truncating storage.
|
||||||
|
|
||||||
|
Reverse history queries include their requested end bar, which is
|
||||||
|
already the oldest sample in SHM. Publishing the overlap consumed
|
||||||
|
an extra physical row and made reserved null counts disagree with
|
||||||
|
elapsed timestamps. Return a frame with the overlap and prove
|
||||||
|
NativeDB receives the complete provider delta while SHM receives
|
||||||
|
only bars older than its published boundary.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
frame = np.zeros(
|
frame = np.zeros(
|
||||||
2,
|
3,
|
||||||
dtype=np.dtype(def_iohlcv_fields),
|
dtype=np.dtype(def_iohlcv_fields),
|
||||||
)
|
)
|
||||||
frame['index'] = [0, 1]
|
frame['index'] = [0, 1, 2]
|
||||||
frame['time'] = [60, 120]
|
frame['time'] = [60, 120, 180]
|
||||||
|
|
||||||
events: list[str] = []
|
events: list[str] = []
|
||||||
|
|
||||||
|
|
@ -274,7 +282,7 @@ def test_storage_receives_full_provider_delta() -> None:
|
||||||
return (
|
return (
|
||||||
frame,
|
frame,
|
||||||
from_timestamp(60),
|
from_timestamp(60),
|
||||||
from_timestamp(120),
|
from_timestamp(180),
|
||||||
)
|
)
|
||||||
|
|
||||||
async def main() -> None:
|
async def main() -> None:
|
||||||
|
|
@ -286,7 +294,7 @@ def test_storage_receives_full_provider_delta() -> None:
|
||||||
mkt=mkt,
|
mkt=mkt,
|
||||||
shm=shm,
|
shm=shm,
|
||||||
timeframe=60,
|
timeframe=60,
|
||||||
backfill_from_shm_index=1,
|
backfill_from_shm_index=2,
|
||||||
backfill_from_dt=from_timestamp(180),
|
backfill_from_dt=from_timestamp(180),
|
||||||
sampler_stream=Sampler(),
|
sampler_stream=Sampler(),
|
||||||
backfill_until_dt=from_timestamp(60),
|
backfill_until_dt=from_timestamp(60),
|
||||||
|
|
@ -295,8 +303,8 @@ def test_storage_receives_full_provider_delta() -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
trio.run(main)
|
trio.run(main)
|
||||||
assert shm.pushed[0]['time'].tolist() == [120]
|
assert shm.pushed[0]['time'].tolist() == [60, 120]
|
||||||
assert storage.frames[0]['time'].tolist() == [60, 120]
|
assert storage.frames[0]['time'].tolist() == [60, 120, 180]
|
||||||
assert events == ['storage', 'shm', 'sampler']
|
assert events == ['storage', 'shm', 'sampler']
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -476,3 +484,163 @@ def test_backfill_notification_timeout_is_bounded(
|
||||||
)
|
)
|
||||||
|
|
||||||
trio.run(main)
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_synthetic_gap_times_require_valid_right_boundary() -> None:
|
||||||
|
'''
|
||||||
|
Synthetic rows must remain strictly between valid boundary bars.
|
||||||
|
|
||||||
|
The live QQQ reservation held one more physical zero row than its
|
||||||
|
timestamp interval could represent. Blind fill would assign the
|
||||||
|
null the right boundary's timestamp, creating a duplicate. Prove
|
||||||
|
aligned bounds produce cadence while an overfull segment
|
||||||
|
is rejected instead of inventing non-monotonic timestamps.
|
||||||
|
|
||||||
|
'''
|
||||||
|
aligned = _synthetic_gap_times(
|
||||||
|
start_t=60,
|
||||||
|
right_t=300,
|
||||||
|
count=3,
|
||||||
|
timeframe=60,
|
||||||
|
)
|
||||||
|
overfull = _synthetic_gap_times(
|
||||||
|
start_t=60,
|
||||||
|
right_t=240,
|
||||||
|
count=3,
|
||||||
|
timeframe=60,
|
||||||
|
)
|
||||||
|
trailing = _synthetic_gap_times(
|
||||||
|
start_t=60,
|
||||||
|
right_t=None,
|
||||||
|
count=3,
|
||||||
|
timeframe=60,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert aligned is not None
|
||||||
|
assert aligned.tolist() == [120, 180, 240]
|
||||||
|
assert overfull is None
|
||||||
|
assert trailing is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_null_repair_falls_back_without_debug_pause(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Empty provider repair must resolve interior zero rows locally.
|
||||||
|
|
||||||
|
QQQ startup exposed 653 zero-time rows between stored history and
|
||||||
|
an IB frame. The null repair queried IB while reverse backfill
|
||||||
|
occupied its sole history request slot, then could
|
||||||
|
enter an interactive debug pause before reaching forward-fill.
|
||||||
|
Arrange an interior null run, return an empty provider frame, and
|
||||||
|
wedge sampler notification. A short timeout and fail-fast pause
|
||||||
|
prove local fallback needs no interactive or IPC progress.
|
||||||
|
The assertions verify only the
|
||||||
|
zero rows inherit the preceding close and synthesized timestamps;
|
||||||
|
both valid boundary rows remain unchanged.
|
||||||
|
|
||||||
|
'''
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'piker.tsp._history._notify_timeout_s',
|
||||||
|
0.01,
|
||||||
|
)
|
||||||
|
|
||||||
|
backing = np.zeros(
|
||||||
|
105,
|
||||||
|
dtype=np.dtype(def_iohlcv_fields),
|
||||||
|
)
|
||||||
|
backing['index'] = np.arange(105)
|
||||||
|
frame = backing[100:105]
|
||||||
|
frame['index'] = np.arange(100, 105)
|
||||||
|
frame['time'] = [60, 0, 0, 0, 300]
|
||||||
|
frame['open'] = [9, 0, 0, 0, 19]
|
||||||
|
frame['high'] = [11, 0, 0, 0, 21]
|
||||||
|
frame['low'] = [8, 0, 0, 0, 18]
|
||||||
|
frame['close'] = [10, 0, 0, 0, 20]
|
||||||
|
frame['volume'] = [5, 0, 0, 0, 6]
|
||||||
|
|
||||||
|
class Shm:
|
||||||
|
'''
|
||||||
|
Expose the test frame through the SHM repair interface.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._array = backing
|
||||||
|
|
||||||
|
@property
|
||||||
|
def array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Return the readable SHM view.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self._array[100:105]
|
||||||
|
|
||||||
|
class Sampler:
|
||||||
|
'''
|
||||||
|
Model a backpressured UI notification stream.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def send(self, msg) -> None:
|
||||||
|
'''
|
||||||
|
Block until the bounded notifier cancels this send.
|
||||||
|
|
||||||
|
'''
|
||||||
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
async def get_hist(*args, **kwargs):
|
||||||
|
'''
|
||||||
|
Return no provider bars for the requested null segment.
|
||||||
|
|
||||||
|
'''
|
||||||
|
end_dt = kwargs['end_dt']
|
||||||
|
empty = np.empty(
|
||||||
|
0,
|
||||||
|
dtype=np.dtype(def_iohlcv_fields),
|
||||||
|
)
|
||||||
|
return empty, end_dt, end_dt
|
||||||
|
|
||||||
|
async def fail_pause() -> None:
|
||||||
|
'''
|
||||||
|
Reject interactive debugging in normal repair fallbacks.
|
||||||
|
|
||||||
|
'''
|
||||||
|
raise AssertionError('null repair entered tractor.pause()')
|
||||||
|
|
||||||
|
monkeypatch.setattr(tractor, 'pause', fail_pause)
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
import greenback
|
||||||
|
|
||||||
|
async def ensure_portal() -> None:
|
||||||
|
'''
|
||||||
|
Avoid installing a greenback portal in this unit test.
|
||||||
|
|
||||||
|
'''
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
greenback,
|
||||||
|
'ensure_portal',
|
||||||
|
ensure_portal,
|
||||||
|
)
|
||||||
|
with trio.fail_after(0.1):
|
||||||
|
await maybe_fill_null_segments(
|
||||||
|
shm=Shm(),
|
||||||
|
timeframe=60,
|
||||||
|
get_hist=get_hist,
|
||||||
|
sampler_stream=Sampler(),
|
||||||
|
mkt=SimpleNamespace(fqme='qqq.nasdaq.ib'),
|
||||||
|
backfill_until_dt=from_timestamp(60),
|
||||||
|
)
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
assert frame['time'].tolist() == [60, 120, 180, 240, 300]
|
||||||
|
for field in ('open', 'high', 'low', 'close'):
|
||||||
|
assert frame[field].tolist() == [
|
||||||
|
frame[field][0],
|
||||||
|
10,
|
||||||
|
10,
|
||||||
|
10,
|
||||||
|
frame[field][-1],
|
||||||
|
]
|
||||||
|
assert frame['volume'].tolist() == [5, 0, 0, 0, 6]
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue