piker/docs/history_segment_backfill.rst

History segment backfill: first-pass interface plan
=================================================

Status and scope
----------------

The first implementation queries dated IB schedules and reserves
synthetic samples while preparing older stored history for SHM.
It does not move published rows, implement automatic segment repair,
or change chart rendering. Synthetic ranges are actor-local metadata;
parquet remains provider observations and is rescanned on restart.

The existing null-row repair is not the segment-repair implementation.
It assumes allocated zero-time slots. Schedule reservations instead
have valid synthetic timestamps and explicit external provenance.

Schedule queries
----------------

``datad.ib`` exposes ``feed.get_history_schedule(fqme, start, end)``
for UTC epoch bounds. The response includes actual coverage, timezone,
regular-hours policy, and half-open trading sessions. This wraps
``Client.history_schedule()`` through the existing asyncio method proxy.
The first pass limits a query to 31 days. If IB returns coverage
starting after the requested left edge, double the lookback up to
31 days while retaining the requested end timestamp. All attempts
share one 10-second timeout. A missing right edge cannot be repaired
by widening backwards; return the actual partial coverage so the
allocator leaves that gap unresolved. The same applies when the
31-day lookback still leaves the left edge uncovered.
Historical queries must match bars' ``useRTH=False`` selection.

Do not extrapolate sessions outside response coverage. Empty schedules,
ambiguous DST times, failures, and partial coverage are unknown for
allocation purposes. Missing contract/session support is not a closure.
A successful schedule establishes possible trading slots, not actual
bar counts or proof of trades. Live MNQ coverage remains to be verified.

Startup reservations
--------------------

After reverse retrieval and before prepending stored history, inspect
its newest 32 timestamp gaps within the retained SHM capacity by
default. ``reserve_history_gaps(newest_gaps=32)`` makes that request
budget configurable; zero disables reservations. Query schedules
serially through the history client's proxy. Reserve only
covered open-session slots, preserving venue closures as time gaps.
Large or unsupported ranges remain unresolved. Capacity bounds include
synthetic samples; oldest samples may be clipped as with ordinary SHM
prepending. No published provider or realtime row is shifted.

``feed.get_history_reservations(fqme, timeframe)`` returns half-open UTC
ranges of synthetic rows for this datad generation. Consumers must use
this provenance rather than zero volume or flat prices. Index positions
are not durable identity. Metadata is published with the synchronous SHM
prepend and clipped to its visible timestamps.

The default chart still displays flat placeholders. Compression and
synthetic-aware FSP treatment are follow-up consumers of this metadata.
Manual whole-SHM export must exclude synthetic ranges before writing
provider history; automatic startup writes do not include reservations.

On-demand request contract (proposed)
------------------------------------

A datad context should accept ``fqme``, ``timeframe``, ``start``, ``end``,
``request_id``, and ``reason``. Bounds are half-open UTC timestamps;
reasons include interrupted startup, detected gap, and operator selection.
CLI and chart selection submit the same request, with no client-local
array indexes. The actor validates market identity and sampling grid.

Proposed events are accepted, progress, completed, partial, failed, and
cancelled. Progress carries requested/received bounds, rows persisted,
slots replaced, remaining synthetic ranges, and failure details.
``completed`` means every requested interval received a classified
provider outcome, not that every scheduled slot contained a trade.

A provider-owned queue deduplicates overlapping requests and serializes
IB history calls. Persist unresolved requests and per-interval outcomes
before fetching so actor/client cancellation cannot erase pending work.
Distinguish genuine empty replies from timeouts and permission failures.
Closing a progress subscription must not implicitly discard queued work;
explicit cancellation policy belongs in the request protocol.

Merge real rows into NativeDB first, then replace matching synthetic
SHM slots by timestamp without changing indexes. Remove provenance only
after successful publication. Revalidate bounds and generation before
writes, coordinate sampler/FSP readers, and broadcast a repair interval
for downstream cache invalidation. Never use legacy null repair to place
an arbitrary provider frame backward over already valid observations.

CLI proposal: ``piker store backfill FQME --timeframe N --start UTC
--end UTC`` with a preview showing schedules, unknown ranges, and slots.
Chart proposal: translate selected x coordinates through the current
view's timestamp mapping, preview the interval, and subscribe to the
same progress context. Compression must retain inverse timestamp/index
mapping for cursor, selection, and annotation behavior.

Deferred work
-------------

TODO: replace synthetic rows using the common segment request context.
TODO: durable unresolved/checked interval tracking and bounded retries.
TODO: automatic scan worker separate from samplerd's live sampling loop.
TODO: chart compression and explicit FSP handling of placeholders.
TODO: prevent operational whole-SHM exports from persisting placeholders.
TODO: cancel the underlying IB schedule request on timeout/disconnect.
TODO: batch/cache session queries with coverage and contract identity.
TODO: coordinated insertion only if schedule-driven reservation proves
insufficient; first pass reports unknown/capacity cases without reindexing.

Validation
----------

Use fake schedule responses for dated timezone conversion, DST ambiguity,
partial coverage, closures, empty/error results, and capacity limits.
An isolated real-SHM/NativeDB restart test must show that reserved rows
are marked synthetic, closures stay compressed, and parquet never gains
invented trades. Live gateway qualification is separate from these tests.