Make bare storage groups and eps render help, so the CLI is
discoverable without memorizing `Typer` option conventions. Add
read-only `series` and `shm` eps for durable `NativeDB` series and
live history buffers, with `Rich`-table or `JSON` output.
Deats,
- preserve exact `(fqme, period)` identities for `NativeDB`
series;
- parse exact `(generation, fqme, kind)` names for SHM buffers;
- reconstruct provider dtypes when attaching to live buffers;
- tolerate buffers disappearing between discovery and attach.
Also,
- report expected and observed cadence separately, plus invalid
rows, ordering defects and ranked gaps;
- add exact `ldshm --shm-name` selection and keep disabled write
and reload paths from reading undefined state or entering `pdb`;
- document the triage layers and cover help, identity parsing,
read-only diagnostics, dtype selection and `ldshm` flow.
Prompt-IO: ai/prompt-io/opencode/20260731T165215Z_0846cbd4_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Do not pass unpublished or corrupt timestamp rows into dedupe,
`NativeDB` replacement writes, or live SHM reload.
Deats,
- infer cadence from finite positive observed steps
- choose the smaller observed step when frequency ties
- skip invalid or degenerate snapshots as a whole
- cover the QQQ zero-slot failure with mutation-failing spies
Prompt-IO: ai/prompt-io/opencode/20260729T185406Z_d359b1cb_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Filter known zero-initialized epoch slots before persisted history
hydrates SHM or merges with a fresh provider frame.
Deats,
- reindex the repaired read view for same-session chart publication
- treat sentinel-only files as empty history for fresh backfill
- reject new first-slot sentinels but preserve modern zero-price bars
- classify raw sentinel evidence separately in `store audit`
Prompt-IO: ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Inspect Parquet bytes directly without opening storage or normalizing
malformed evidence.
Deats,
- report canonical schema, timestamp, index and finite-value state
- retain exact cadence aggregates with bounded endpoint details
- snapshot bytes before parsing and clean up partial copy failures
- reject unsafe FQMEs, links, collisions and non-regular sources
Prompt-IO: ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Keep incoming provider frames strict while dropping non-positive rows
from persisted history before merge validation.
This lets known-bad MNQ history repair itself on the next valid IB
frame without losing valid old bars or publishing noncanonical indexes.
Prompt-IO: ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
IB history frames omit the derived `index` and append `count`.
Normalize provider data to durable names, order, and dtypes before
merge, and reject fractional epoch truncation.
Also,
- map canonical storage fields into provider-specific SHM on reload
- select Polars columns by name instead of position
- exercise first-start and restart through actor, SHM, and Parquet
Prompt-IO: ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
`datad`'s persistent feed task already owns each series writer.
The actor-local and file locks duplicated this SC contract and left
`.parquet.lock` sidecars that upstream indexed as data.
- preserve validated temp writes, `fsync()`, and atomic replacement
- keep exact-suffix indexing for legacy lock and temp sidecars
- prove full and incremental writes create no lock artifacts
Prompt-IO: ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
- merge provider deltas without dropping unaffected history
- validate reopened temp parquet before atomic replacement
- serialize writers with actor-local and cancellable file locks
- publish cache only after disk replacement
- persist full startup/backfill frames before SHM publication
- bound sampler notifications and avoid MarketStore duplicates
Prompt-IO: ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Port internal `tractor._<mod>` references to their
new public or reorganized paths after `tractor`
refactored its subpkg layout.
Deats,
- `tractor._portal.Portal` -> `tractor.Portal`.
- `tractor._supervise.ActorNursery` -> `tractor.ActorNursery`.
- `tractor._multiaddr` -> `tractor.discovery._multiaddr`.
- `tractor._addr` -> `tractor.discovery._addr`.
- `tractor._state._runtime_vars` -> `tractor.runtime._state._runtime_vars`.
- `tractor._state.is_debug_mode()` -> `tractor.runtime._state.is_debug_mode()`.
Files touched: `brokers/data.py`, `cli/__init__.py`, `data/feed.py`,
`service/_actor_runtime.py`, `service/_mngr.py`, `storage/cli.py`,
`tsp/_annotate.py`, `ui/kivy/monitor.py`, `ui/kivy/option_chain.py`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Call `reg_err_types()` for every piker-defined
exception so they can be marshalled and re-raised
across actor boundaries.
Deats,
- `brokers/_util.py`: auto-register `BrokerError` +
all `__subclasses__()` (6 types).
- `config.py`: `ConfigurationError` +
`__subclasses__()` (`NoSignature`).
- `data/validate.py`: `FeedInitializationError`.
- `service/_ahab.py`: `DockerNotStarted`,
`ApplicationLogError`.
- `service/marketstore.py`: `MarketStoreError`.
- `storage/__init__.py`: `TimeseriesNotFound`,
`StorageConnectionError`.
- `brokers/kraken/api.py`: `InvalidKey`.
- `brokers/kraken/broker.py`: `TooFastEdit`.
- `brokers/questrade.py`: `QuestradeError`.
Also,
- uncomment `execution_venue` field on kraken `Pair`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Port all 16 internal import sites from re-exporting
via `piker.data._sharedmem` shim to importing core
shm types directly from `tractor.ipc._shm`.
Deats,
- `ShmArray` now imported from tractor in 10 files.
- `_Token` renamed to `NDToken` everywhere (5 files).
- `attach_shm_array` → `attach_shm_ndarray` at all
call sites.
- `data/__init__.py` sources `ShmArray`,
`get_shm_token` from tractor; keeps
`open/attach_shm_array` as public API aliases.
- Trim shim to only piker-specific wrappers:
`_make_token()`, `maybe_open_shm_array()`,
`try_read()`.
- Drop `Optional` usage in shim, use `|None`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Change `assert aids` to a warning log when no history gaps are found
during `ldshm` gap detection; it is the **ideal case** OBVI. This avoids
crashing the CLI when gap detection finds no issues, which is actually
good news!
Bp
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Namely such that when the previous-df-row by our shm-abs-'index' doesn't
exist we ignore certain cases which are likely due to borked-but-benign
samples written to the tsdb or rt shm buffers prior.
Particularly we now ignore,
- any `dt`/`prev_dt` values which are UNIX-epoch timestamped (val of 0).
- any row-is-first-row in the df; there is no previous.
- any missing previous datum by 'index', in which case we lookup the
`wdts` prior row and use that instead.
* this would indicate a missing sample for the time-step but we can
still detect a "gap" by looking at the prior row, by df-abs-index
`i`, and use its timestamp to determine the period/size of missing
samples (which need to likely still be retrieved).
* in this case i'm leaving in a pause-point for introspecting these
rarer cases when `--pdb` is passed via CLI.
Relatedly in the `piker store` CLI ep,
- add `--pdb` flag to `piker store`, pass it verbatim as `debug_mode`.
- when `times` has only a single row, don't calc a `period_s` median.
- only trace `null_segs` when in debug mode.
- always markup/dedupe gaps for `period_s==60`
Using `claude`, add a `.tsp._dedupe_smart` module that attemps "smarter"
duplicate bars by attempting to distinguish between erroneous bars
partially written during concurrent backfill race conditions vs.
**actual** data quality issues from historical providers.
Problem:
--------
Concurrent writes (live updates vs. backfilling) can result in create
duplicate timestamped ohlcv vars with different values. Some
potential scenarios include,
- a market live feed is cancelled during live update resulting in the
"last" datum being partially updated with all the ticks for the
time step.
- when the feed is rebooted during charting, the backfiller will not
finalize this bar since rn it presumes it should only fill data for
time steps not already in the tsdb storage.
Our current naive `.unique()` approach obvi keeps the incomplete bar
and a "smarter" approach is to compare the provider's final vlm
amount vs. the maybe-cancelled tsdb's bar; a higher vlm value from
the provider likely indicates the cancelled-during-live-write and
**not** a datum discrepancy from said data provider.
Analysis (with `claude`) of `zecusdt` data revealed:
- 1000 duplicate timestamps
- 999 identical bars (pure duplicates from 2022 backfill overlap)
- 1 volume-monotonic conflict (live partial vs backfill complete)
A soln from `claude` -> `tsp._dedupe_smart.dedupe_ohlcv_smart()`
which:
- sorts by vlm **before** deduplication and keep the most complete
bar based on vlm monotonicity as well as the following OHLCV
validation assumptions:
* volume should always increase
* high should be non-decreasing,
* low should be non-increasing
* open should be identical
- Separates valid race conditions from provider data quality issues
and reports and returns both dfs.
Change summary by `claude`:
- `.tsp._dedupe_smart`: new module with validation logic
- `.tsp.__init__`: expose `dedupe_ohlcv_smart()`
- `.storage.cli`: integrate smart dedupe, add logging for:
* duplicate counts (identical vs monotonic races)
* data quality violations (non-monotonic, invalid OHLC ranges)
* warnings for provider data issues
- Remove `assert not diff` (duplicates are valid now)
Verified on `zecusdt`: correctly keeps index 3143645
(volume=287.777) over 3143644 (volume=140.299) for
conflicting 2026-01-16 18:54 UTC bar.
`claude`'s Summary of reasoning
-------------------------------
- volume monotonicity is critical: a bar's volume only increases
during its time window.
- a backfilled bar should always have volume >= live updated.
- violations indicate any of:
* Provider data corruption
* Non-OHLCV aggregation semantics
* Timestamp misalignment
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Since we definitely want to markup gaps that are both data-errors and
just plain old venue closures (time gaps), generalize the `gaps:
pl.DataFrame` loop in a func and call it from the `ldshm` cmd for now.
Some other tweaks to `store ldshm`:
- add `np.median()` failover when detecting (ohlc) ts sample rate.
- use new `tsp.dedupe()` signature.
- differentiate between sample-period size gaps and venue closure gaps
by calling `tsp.detect_time_gaps()` with diff size thresholds.
- add todo for backfilling null-segs when detected.
Apparently `.storage.nativedb.mk_ohlcv_shm_keyed_filepath()` was always
kinda broken if you passed in a `period: float` with an actual non-`int`
to the format string? Fixed it to strictly cast to `int()` before
str-ifying so that you don't get weird `60.0s.parquet` in there..
Further this rejigs the `sotre ldshm` gap correction-annotation loop to,
- use `StorageClient.write_ohlcv()` instead of hackily re-implementing
it.. now that problem from above is fixed!
- use a `needs_correction: bool` var to determine if gap markup and
de-duplictated data should be pushed to the shm buffer,
- go back to using `AnnotCtl.add_rect()` for all detected gaps such that
they all persist (and thus are shown together) until the client
disconnects.
Obvi took a little `.ui` component fixing (as per prior commits) but
this is now a working PoC for gap detection and markup from a remote
(data) non-`chart` actor!
Iface and impl deats from `.ui._remote_ctl`:
- add new `open_annot_ctl()` mngr which attaches to all locally
discoverable chart actors, gathers annot-ctl streams per fqme set, and
delivers a new `AnnotCtl` client which allows adding annotation
rectangles via a `.add_rect()` method.
- also template out some other soon-to-get methods for removing and
modifying pre-exiting annotations on some `ChartView` 💥
- ensure the `chart` CLI subcmd starts the (`qtloops`) guest-mode init
with the `.ui._remote_ctl` module enabled.
- actually use this stuff in the `piker store ldshm` CLI to submit
markup rects around any detected null/time gaps in the tsdb data!
Still lots to do:
- probably colorization of gaps depending on if they're venue
closures (aka real mkt gaps) vs. "missing data" from the backend (aka
timeseries consistency gaps).
- run gap detection and markup as part of the std `.tsp` sub-sys
runtime such that gap annots are a std "built-in" feature of
charting.
- support for epoch time stamp AND abs-shm-index rect x-values
(depending on chart operational state).
Also toss in a poll loop around the `hist_shm: ShmArray` backfill
read-check in the `.data.allocate_persisten_feed()` init to cope with
possible racy-ness from the increased tsdb history loading concurrency
now implemented.
Thinking about just moving all of that module (after a content breakup)
to a new `.piker.tsp` which will mostly depend on the `.data` and
`.storage` sub-pkgs; the idea is to move biz-logic for tsdb IO/mgmt and
orchestration with real-time (shm) buffers and the graphics layer into
a common spot for both manual analysis/research work and better
separation of low level data structure primitives from their higher
level usage.
Add a better `data.history` mod doc string in prep for this move
as well as clean out a bunch of legacy commented cruft from the
`trimeter` and `marketstore` days.
TO CHERRY #486 (if we can)
Yet again these are (going to be) generally useful in the data proc
layer as well as going forward with (possibly) moving the history and
shm rt-processing layer to apache (arrow or other) shared-ds
equivalents.
Includes a rename of `.data._timeseries` -> `.data.tsp` for "time series
processing", making it a public sub-mod; it contains a highly useful set
of data-frame and `numpy.ndarray` ops routines in various subsystems Bo
Think i finally figured out the weird issue without out-of-order OHLC
history getting jammed in the wrong place:
- gap is detected in parquet/offline ts (likely due to a zero dt or
other gap),
- query for history in the gap is made BUT that frame is then inserted
in the shm buffer **at the end** (likely using array int-entry
indexing) which inserts it at the wrong location,
- later this out-of-order frame is written to the storage layer
(parquet) and then is repeated on further reboots with the original
gap causing further queries for the same frame on every history
backfill.
A set of tools useful for detecting these issues and annotating them
nicely on chart part of this patch's intent:
- `dedupe()` will detect any dt gaps, deduplicate datetime rows and
return the de-duplicated df along with gaps table.
- use this in both `piker store anal` such that we potentially
resolve and backfill the gaps correctly if some rows were removed.
- possibly also use this to detect the backfilling error in logic at
the time of backfilling the frame instead of after the fact (which
would require re-writing the shm array from something like `store
ldshm` and would be a manual post-hoc solution, not a fix to the
original issue..
A common usage error is to run `piker anal mnq.cme.ib` where the CLI
passed fqme is not actually fully-qualified (in this case missing an
expiry token) and we get an underlying `FileNotFoundError` from the
`StorageClient.read_ohlcv()` call. In such key misses, scan the existing
`StorageClient._index` for possible matches and report in a `raise from`
the new error.
CHERRY into #486
Since there's a growing list of top level mods which are more or less
utils/tools for working with the runtime; begin to move them into a new
subpkg starting with a new `.toolz.debug`.
Start with,
- a new `open_crash_handller()` for doing breakpoints around blocks that
might error.
- move in what was `piker._profile` into `.toolz.profile` and adjust all
importing appropriately.
Changed from the old `store clone` to instead simply load any shm buffer
matching a user provided `FQME: str` pattern; writing to parquet file is
only done if an explicit option flag is passed by user.
Implement new `iter_dfs_from_shms()` generator which allows interatively
loading both 1m and 1s buffers delivering the `Path`, `ShmArray` and
`polars.DataFrame` instances per matching file B)
Also add a todo for a `NativeStorageClient.clear_range()` method.