3.0 KiB
The user reported that a completed NVDA 60s backfill left a large zero region in live SHM and asked for immediate inspection before restarting the workspace. The captured buffer contained 42,720 invalid rows at absolute indexes 3065895..3108614. The valid rows on both sides had the identical timestamp 1781271360, proving the region was an erroneous physical-index reservation rather than a real temporal gap.
The user then asked the agent to continue fixing the backfill logic.
git diff HEAD~1..HEAD -- piker/tsp/_history.py
Removed wall-clock-based SHM reservation from tsdb_backfill(). Reverse provider frames now extend the visible ShmArray._first boundary on every prepend, beginning exactly at the current first index after inclusive query endpoints are removed. Added _prepend_tsdb_history() to trim storage rows against the actual earliest provider timestamp and prepend them directly adjacent only after reverse provider retrieval completes.
This preserves sparse venue history as ordinal samples rather than treating every elapsed calendar period as a physical SHM row. It also prevents the old _first + 1 insertion point from overwriting the earliest published bar.
git diff HEAD~1..HEAD -- tests/test_history_backfill.py
Added a full tsdb_backfill() regression modeled on the live NVDA failure. It uses a real ShmArray, NativeDB’s field map, two inclusive reverse-provider frames, and overlapping stored history. Assertions prove the complete timestamp sequence is contiguous and lossless, contains no non-positive timestamps, sends complete provider frames to durable storage, and notifies chart/FSP consumers after the final storage prepend.
Live evidence captured before workspace restart:
rows 1575296 first_index 1568233 last_index 3143528
invalid_count 42720
left boundary
[(3065893, 1781271300, 204.03, 204.82)
(3065894, 1781271360, 204.81, 204.72)
(3065895, 0, 0. , 0. )]
right boundary
[(3108614, 0, 0. , 0. )
(3108615, 1781271360, 204.81, 205.44)
(3108616, 1781271420, 205.46, 205.48)]Verification output:
........................................................................ [ 68%]
................................. [100%]
105 passed in 4.80sAdversarial review caught stale _first semantics and an off-by-one insertion point in the initial patch. The first live restart then exposed an undefined time_key local at the new tsdb_backfill() call site, which crashed datad.ib and propagated to the chart as RemoteActorError. The helper now derives the source timestamp field from its field map, and the test enters the full enclosing lifecycle so the original NameError can not recur unnoticed. A final review also drove explicit publication after stored history is prepended and deterministic final-stage synchronization. Final review found no remaining issues; another live restart remains pending.