114 lines
5.7 KiB
ReStructuredText
114 lines
5.7 KiB
ReStructuredText
History segment backfill: first-pass interface plan
|
|
=================================================
|
|
|
|
Status and scope
|
|
----------------
|
|
|
|
The first implementation queries dated IB schedules and reserves
|
|
synthetic samples while preparing older stored history for SHM.
|
|
It does not move published rows, implement automatic segment repair,
|
|
or change chart rendering. Synthetic ranges are actor-local metadata;
|
|
parquet remains provider observations and is rescanned on restart.
|
|
|
|
The existing null-row repair is not the segment-repair implementation.
|
|
It assumes allocated zero-time slots. Schedule reservations instead
|
|
have valid synthetic timestamps and explicit external provenance.
|
|
|
|
Schedule queries
|
|
----------------
|
|
|
|
``datad.ib`` exposes ``feed.get_history_schedule(fqme, start, end)``
|
|
for UTC epoch bounds. The response includes actual coverage, timezone,
|
|
regular-hours policy, and half-open trading sessions. This wraps
|
|
``Client.history_schedule()`` through the existing asyncio method proxy.
|
|
The first pass limits a query to 31 days and times out after 10 seconds.
|
|
Historical queries must match bars' ``useRTH=False`` selection.
|
|
|
|
Do not extrapolate sessions outside response coverage. Empty schedules,
|
|
ambiguous DST times, failures, and partial coverage are unknown for
|
|
allocation purposes. Missing contract/session support is not a closure.
|
|
A successful schedule establishes possible trading slots, not actual
|
|
bar counts or proof of trades. Live MNQ coverage remains to be verified.
|
|
|
|
Startup reservations
|
|
--------------------
|
|
|
|
After reverse retrieval and before prepending stored history, inspect
|
|
its newest 32 timestamp gaps within the retained SHM capacity by
|
|
default. ``reserve_history_gaps(newest_gaps=32)`` makes that request
|
|
budget configurable; zero disables reservations. Query schedules
|
|
serially through the history client's proxy. Reserve only
|
|
covered open-session slots, preserving venue closures as time gaps.
|
|
Large or unsupported ranges remain unresolved. Capacity bounds include
|
|
synthetic samples; oldest samples may be clipped as with ordinary SHM
|
|
prepending. No published provider or realtime row is shifted.
|
|
|
|
``feed.get_history_reservations(fqme, timeframe)`` returns half-open UTC
|
|
ranges of synthetic rows for this datad generation. Consumers must use
|
|
this provenance rather than zero volume or flat prices. Index positions
|
|
are not durable identity. Metadata is published with the synchronous SHM
|
|
prepend and clipped to its visible timestamps.
|
|
|
|
The default chart still displays flat placeholders. Compression and
|
|
synthetic-aware FSP treatment are follow-up consumers of this metadata.
|
|
Manual whole-SHM export must exclude synthetic ranges before writing
|
|
provider history; automatic startup writes do not include reservations.
|
|
|
|
On-demand request contract (proposed)
|
|
------------------------------------
|
|
|
|
A datad context should accept ``fqme``, ``timeframe``, ``start``, ``end``,
|
|
``request_id``, and ``reason``. Bounds are half-open UTC timestamps;
|
|
reasons include interrupted startup, detected gap, and operator selection.
|
|
CLI and chart selection submit the same request, with no client-local
|
|
array indexes. The actor validates market identity and sampling grid.
|
|
|
|
Proposed events are accepted, progress, completed, partial, failed, and
|
|
cancelled. Progress carries requested/received bounds, rows persisted,
|
|
slots replaced, remaining synthetic ranges, and failure details.
|
|
``completed`` means every requested interval received a classified
|
|
provider outcome, not that every scheduled slot contained a trade.
|
|
|
|
A provider-owned queue deduplicates overlapping requests and serializes
|
|
IB history calls. Persist unresolved requests and per-interval outcomes
|
|
before fetching so actor/client cancellation cannot erase pending work.
|
|
Distinguish genuine empty replies from timeouts and permission failures.
|
|
Closing a progress subscription must not implicitly discard queued work;
|
|
explicit cancellation policy belongs in the request protocol.
|
|
|
|
Merge real rows into NativeDB first, then replace matching synthetic
|
|
SHM slots by timestamp without changing indexes. Remove provenance only
|
|
after successful publication. Revalidate bounds and generation before
|
|
writes, coordinate sampler/FSP readers, and broadcast a repair interval
|
|
for downstream cache invalidation. Never use legacy null repair to place
|
|
an arbitrary provider frame backward over already valid observations.
|
|
|
|
CLI proposal: ``piker store backfill FQME --timeframe N --start UTC
|
|
--end UTC`` with a preview showing schedules, unknown ranges, and slots.
|
|
Chart proposal: translate selected x coordinates through the current
|
|
view's timestamp mapping, preview the interval, and subscribe to the
|
|
same progress context. Compression must retain inverse timestamp/index
|
|
mapping for cursor, selection, and annotation behavior.
|
|
|
|
Deferred work
|
|
-------------
|
|
|
|
TODO: replace synthetic rows using the common segment request context.
|
|
TODO: durable unresolved/checked interval tracking and bounded retries.
|
|
TODO: automatic scan worker separate from samplerd's live sampling loop.
|
|
TODO: chart compression and explicit FSP handling of placeholders.
|
|
TODO: prevent operational whole-SHM exports from persisting placeholders.
|
|
TODO: cancel the underlying IB schedule request on timeout/disconnect.
|
|
TODO: batch/cache session queries with coverage and contract identity.
|
|
TODO: coordinated insertion only if schedule-driven reservation proves
|
|
insufficient; first pass reports unknown/capacity cases without reindexing.
|
|
|
|
Validation
|
|
----------
|
|
|
|
Use fake schedule responses for dated timezone conversion, DST ambiguity,
|
|
partial coverage, closures, empty/error results, and capacity limits.
|
|
An isolated real-SHM/NativeDB restart test must show that reserved rows
|
|
are marked synthetic, closures stay compressed, and parquet never gains
|
|
invented trades. Live gateway qualification is separate from these tests.
|