piker/ai/prompt-io/codex/20260924T185456Z_1da9b760_s...

101 lines
3.8 KiB
Markdown
Raw Normal View History

Use IB schedules to reserve missing SHM history Interrupted backfill can leave stored observations separated by missing trading time. Query dated IB sessions before prepending stored history and reserve synthetic samples only during covered open sessions. Keep venue closures compressed and preserve the indexes of already published rows. Deats, - Add `Client.history_schedule()` with UTC coverage, extended hours, bounded duration, timeout, and rejection of ambiguous DST times. - Keep schedule normalization in `venues` and allocation in `.tsp._reserve`, with the project AGPL headers. - Expose schedule and synthetic-range queries through `datad.ib`. - Bound reservations by available SHM capacity and inspect only the newest 32 gaps by default via `newest_gaps`. Zero disables queries; leave unknown and oversized ranges unresolved. - Record synthetic ranges separately from OHLCV. Automatic storage writes retain provider observations; placeholders copy close. - Group the new tests under `tests/providers`. - Cover request formatting, DST, failure and capacity cases, plus real SHM/NativeDB startup placement with a fake provider. - Document a common CLI/chart segment-repair interface and defer coordinated insertion, repair execution, and reader integration. Validation: 55 targeted tests passed in the working tree; live IB schedule coverage remains unqualified. After the module moves, 33 affected tests passed again; the local review follow-up passed 18 affected cases, including four new request-budget cases. The commit runner repeats lint and isolated regression checks against this exact boundary. Prompt-IO: ai/prompt-io/codex/20260924T185456Z_1da9b760_schedule_prompt_io.md (this patch was generated in some part by `codex` using `gpt-6` (`openai`))
2026-09-24 23:09:22 +00:00
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-24T18:54:56.893490+00:00
git_ref: 1da9b760
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/brokers/ib/_schedule.py`
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_reservations.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
> `git diff HEAD~1..HEAD -- tests/test_ib_schedule.py`
> `git diff HEAD~1..HEAD -- tests/test_history_reservations.py`
> `git diff HEAD~1..HEAD -- docs/history_segment_backfill.rst`
## Generated output
Added Client.history_schedule() over the installed ib_async API,
normalizing dated sessions to UTC with explicit response coverage.
Reject ambiguous DST times, reversed sessions, and overlapping
sessions. Match the existing extended-hours bars policy.
Added startup-only schedule reservations to tsdb_backfill before
prepending older stored history. Reserve open-session sample slots
only where the reply covers the full candidate gap. Return synthetic
UTC ranges separately from OHLCV; copy preceding close into flat
placeholders without writing those placeholders to parquet.
Unknown coverage and capacity failures retain packed time gaps.
Added datad.ib schedule and reservation metadata RPC functions and
a proposed shared CLI/chart segment request protocol. The repair
queue, durable unresolved intervals, chart compression, FSP handling,
and coordinated insertion remain deferred. No live data was repaired.
Verification is complete: 55 targeted tests pass, including the
new schedule and real SHM/parquet tests. I also added checks for
reversed session bounds and the 25-hour day at the autumn DST
change.
The segment API plan uses UTC ranges for both CLI and chart
requests, a provider-owned queue, durable unfinished intervals,
and replacement of reserved slots without reindexing. Live IB
schedule coverage still needs qualification.
## Limits
Only stored-history gaps are reserved before prepend; published rows
are never shifted. Newest 32 gaps, at most 31 days each, are queried
serially with a 10-second schedule timeout. This can delay history
publication. Underlying IB request cancellation is still a TODO.
Manual whole-SHM export requires synthetic-range exclusion before
it can preserve provider-only storage. The chart currently displays
flat placeholders, and synthetic-aware FSP treatment is deferred.
Tests: history backfill 19, IB history 17, method proxy 5, schedule 7,
reservations 7. Sandbox local-socket failures were rerun with approval.
Ruff and git diff --check passed. No live broker requests were made.
## Module-layout review response
Schedule normalization now lives in piker.brokers.ib.venues beside
the existing session helpers. The allocation module is tsp._reserve.
Added the project AGPL headers to the new module and test files.
Earlier diff pointers record the original generated paths; use
the following paths for the reviewed layout:
> `git diff HEAD~1..HEAD -- piker/brokers/ib/venues.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_reserve.py`
## Local review response
The fixes pass all 18 tests in the two affected files.
The new limit tests cover zero, negative, one-gap, and two-gap
budgets, including newest-first selection.
Capacity overflow remains a warning: no partial reservation is
published for that gap, and the original observations remain
available. The comments now explain that fallback explicitly.
Diff pointers above include the review edits to tsp._reserve, both
test modules, and the interface document. This review did not query
a forge or publish replies.
## Provider-test paths
Moved both new test modules under tests/providers as requested.
> `git diff HEAD~1..HEAD -- tests/providers/test_ib_schedule.py`
> `git diff HEAD~1..HEAD -- tests/providers/test_history_reservations.py`