5.7 KiB
History segment backfill: first-pass interface plan =================================================
Status and scope
The first implementation queries dated IB schedules and reserves synthetic samples while preparing older stored history for SHM. It does not move published rows, implement automatic segment repair, or change chart rendering. Synthetic ranges are actor-local metadata; parquet remains provider observations and is rescanned on restart.
The existing null-row repair is not the segment-repair implementation. It assumes allocated zero-time slots. Schedule reservations instead have valid synthetic timestamps and explicit external provenance.
Schedule queries
datad.ib exposes feed.get_history_schedule(fqme, start, end) for UTC epoch bounds. The response includes actual coverage, timezone, regular-hours policy, and half-open trading sessions. This wraps Client.history_schedule() through the existing asyncio method proxy. The first pass limits a query to 31 days and times out after 10 seconds. Historical queries must match bars' useRTH=False selection.
Do not extrapolate sessions outside response coverage. Empty schedules, ambiguous DST times, failures, and partial coverage are unknown for allocation purposes. Missing contract/session support is not a closure. A successful schedule establishes possible trading slots, not actual bar counts or proof of trades. Live MNQ coverage remains to be verified.
Startup reservations
After reverse retrieval and before prepending stored history, inspect its newest 32 timestamp gaps within the retained SHM capacity by default. reserve_history_gaps(newest_gaps=32) makes that request budget configurable; zero disables reservations. Query schedules serially through the history client's proxy. Reserve only covered open-session slots, preserving venue closures as time gaps. Large or unsupported ranges remain unresolved. Capacity bounds include synthetic samples; oldest samples may be clipped as with ordinary SHM prepending. No published provider or realtime row is shifted.
feed.get_history_reservations(fqme, timeframe) returns half-open UTC ranges of synthetic rows for this datad generation. Consumers must use this provenance rather than zero volume or flat prices. Index positions are not durable identity. Metadata is published with the synchronous SHM prepend and clipped to its visible timestamps.
The default chart still displays flat placeholders. Compression and synthetic-aware FSP treatment are follow-up consumers of this metadata. Manual whole-SHM export must exclude synthetic ranges before writing provider history; automatic startup writes do not include reservations.
On-demand request contract (proposed) ------------------------------------
A datad context should accept fqme, timeframe, start, end, request_id, and reason. Bounds are half-open UTC timestamps; reasons include interrupted startup, detected gap, and operator selection. CLI and chart selection submit the same request, with no client-local array indexes. The actor validates market identity and sampling grid.
Proposed events are accepted, progress, completed, partial, failed, and cancelled. Progress carries requested/received bounds, rows persisted, slots replaced, remaining synthetic ranges, and failure details. completed means every requested interval received a classified provider outcome, not that every scheduled slot contained a trade.
A provider-owned queue deduplicates overlapping requests and serializes IB history calls. Persist unresolved requests and per-interval outcomes before fetching so actor/client cancellation cannot erase pending work. Distinguish genuine empty replies from timeouts and permission failures. Closing a progress subscription must not implicitly discard queued work; explicit cancellation policy belongs in the request protocol.
Merge real rows into NativeDB first, then replace matching synthetic SHM slots by timestamp without changing indexes. Remove provenance only after successful publication. Revalidate bounds and generation before writes, coordinate sampler/FSP readers, and broadcast a repair interval for downstream cache invalidation. Never use legacy null repair to place an arbitrary provider frame backward over already valid observations.
CLI proposal: piker store backfill FQME --timeframe N --start UTC --end UTC with a preview showing schedules, unknown ranges, and slots. Chart proposal: translate selected x coordinates through the current view's timestamp mapping, preview the interval, and subscribe to the same progress context. Compression must retain inverse timestamp/index mapping for cursor, selection, and annotation behavior.
Deferred work
TODO: replace synthetic rows using the common segment request context. TODO: durable unresolved/checked interval tracking and bounded retries. TODO: automatic scan worker separate from samplerd's live sampling loop. TODO: chart compression and explicit FSP handling of placeholders. TODO: prevent operational whole-SHM exports from persisting placeholders. TODO: cancel the underlying IB schedule request on timeout/disconnect. TODO: batch/cache session queries with coverage and contract identity. TODO: coordinated insertion only if schedule-driven reservation proves insufficient; first pass reports unknown/capacity cases without reindexing.
Validation
Use fake schedule responses for dated timezone conversion, DST ambiguity, partial coverage, closures, empty/error results, and capacity limits. An isolated real-SHM/NativeDB restart test must show that reserved rows are marked synthetic, closures stay compressed, and parquet never gains invented trades. Live gateway qualification is separate from these tests.