Commit Graph

61 Commits (d41b68b80cd60c0d556dcad003a0c9b58fe7af0d)

Author SHA1 Message Date
Gud Boi d41b68b80c Expose storage state through `piker store`
Make bare storage groups and eps render help, so the CLI is
discoverable without memorizing `Typer` option conventions. Add
read-only `series` and `shm` eps for durable `NativeDB` series and
live history buffers, with `Rich`-table or `JSON` output.

Deats,
- preserve exact `(fqme, period)` identities for `NativeDB`
  series;
- parse exact `(generation, fqme, kind)` names for SHM buffers;
- reconstruct provider dtypes when attaching to live buffers;
- tolerate buffers disappearing between discovery and attach.

Also,
- report expected and observed cadence separately, plus invalid
  rows, ordering defects and ranked gaps;
- add exact `ldshm --shm-name` selection and keep disabled write
  and reload paths from reading undefined state or entering `pdb`;
- document the triage layers and cover help, identity parsing,
  read-only diagnostics, dtype selection and `ldshm` flow.

Prompt-IO: ai/prompt-io/opencode/20260731T165215Z_0846cbd4_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-12 09:48:06 -05:00
Gud Boi 892898cfd0 Guard `ldshm` from invalid SHM snapshots
Do not pass unpublished or corrupt timestamp rows into dedupe,
`NativeDB` replacement writes, or live SHM reload.

Deats,
- infer cadence from finite positive observed steps
- choose the smaller observed step when frequency ties
- skip invalid or degenerate snapshots as a whole
- cover the QQQ zero-slot failure with mutation-failing spies

Prompt-IO: ai/prompt-io/opencode/20260729T185406Z_d359b1cb_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:05:35 -05:00
Gud Boi 05e47d0e33 Repair first-slot OHLC sentinels in `NativeDB`
Filter known zero-initialized epoch slots before persisted history
hydrates SHM or merges with a fresh provider frame.

Deats,
- reindex the repaired read view for same-session chart publication
- treat sentinel-only files as empty history for fresh backfill
- reject new first-slot sentinels but preserve modern zero-price bars
- classify raw sentinel evidence separately in `store audit`

Prompt-IO: ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:04:16 -05:00
Gud Boi 7cec91e287 Add raw OHLCV audit to `NativeDB`
Inspect Parquet bytes directly without opening storage or normalizing
malformed evidence.

Deats,
- report canonical schema, timestamp, index and finite-value state
- retain exact cadence aggregates with bounded endpoint details
- snapshot bytes before parsing and clean up partial copy failures
- reject unsafe FQMEs, links, collisions and non-regular sources

Prompt-IO: ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 50b84d68f2 Repair legacy timestamps in `NativeDB` merges
Keep incoming provider frames strict while dropping non-positive rows
from persisted history before merge validation.

This lets known-bad MNQ history repair itself on the next valid IB
frame without losing valid old bars or publishing noncanonical indexes.

Prompt-IO: ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 1b76dd64fd Normalize provider OHLCV for `NativeDB`
IB history frames omit the derived `index` and append `count`.
Normalize provider data to durable names, order, and dtypes before
merge, and reject fractional epoch truncation.

Also,
- map canonical storage fields into provider-specific SHM on reload
- select Polars columns by name instead of position
- exercise first-start and restart through actor, SHM, and Parquet

Prompt-IO: ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:02:36 -05:00
Gud Boi 9851ff886a Drop redundant `NativeDB` writer locks
`datad`'s persistent feed task already owns each series writer.
The actor-local and file locks duplicated this SC contract and left
`.parquet.lock` sidecars that upstream indexed as data.

- preserve validated temp writes, `fsync()`, and atomic replacement
- keep exact-suffix indexing for legacy lock and temp sidecars
- prove full and incremental writes create no lock artifacts

Prompt-IO: ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:33 -05:00
Gud Boi 9cfe2591d9 Make `NativeDB` backfills atomic and non-destructive
- merge provider deltas without dropping unaffected history
- validate reopened temp parquet before atomic replacement
- serialize writers with actor-local and cancellable file locks
- publish cache only after disk replacement
- persist full startup/backfill frames before SHM publication
- bound sampler notifications and avoid MarketStore duplicates

Prompt-IO: ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:32 -05:00
Gud Boi d7cefa71f3 Update `tractor` private-API refs across codebase
Port internal `tractor._<mod>` references to their
new public or reorganized paths after `tractor`
refactored its subpkg layout.

Deats,
- `tractor._portal.Portal` -> `tractor.Portal`.
- `tractor._supervise.ActorNursery` -> `tractor.ActorNursery`.
- `tractor._multiaddr` -> `tractor.discovery._multiaddr`.
- `tractor._addr` -> `tractor.discovery._addr`.
- `tractor._state._runtime_vars` -> `tractor.runtime._state._runtime_vars`.
- `tractor._state.is_debug_mode()` -> `tractor.runtime._state.is_debug_mode()`.

Files touched: `brokers/data.py`, `cli/__init__.py`, `data/feed.py`,
`service/_actor_runtime.py`, `service/_mngr.py`, `storage/cli.py`,
`tsp/_annotate.py`, `ui/kivy/monitor.py`, `ui/kivy/option_chain.py`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi d7f956a566 Register all custom excs with `tractor` IPC
Call `reg_err_types()` for every piker-defined
exception so they can be marshalled and re-raised
across actor boundaries.

Deats,
- `brokers/_util.py`: auto-register `BrokerError` +
  all `__subclasses__()` (6 types).
- `config.py`: `ConfigurationError` +
  `__subclasses__()` (`NoSignature`).
- `data/validate.py`: `FeedInitializationError`.
- `service/_ahab.py`: `DockerNotStarted`,
  `ApplicationLogError`.
- `service/marketstore.py`: `MarketStoreError`.
- `storage/__init__.py`: `TimeseriesNotFound`,
  `StorageConnectionError`.
- `brokers/kraken/api.py`: `InvalidKey`.
- `brokers/kraken/broker.py`: `TooFastEdit`.
- `brokers/questrade.py`: `QuestradeError`.

Also,
- uncomment `execution_venue` field on kraken `Pair`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 0a2059d00f Use `tractor.ipc._shm` types directly across codebase
Port all 16 internal import sites from re-exporting
via `piker.data._sharedmem` shim to importing core
shm types directly from `tractor.ipc._shm`.

Deats,
- `ShmArray` now imported from tractor in 10 files.
- `_Token` renamed to `NDToken` everywhere (5 files).
- `attach_shm_array` → `attach_shm_ndarray` at all
  call sites.
- `data/__init__.py` sources `ShmArray`,
  `get_shm_token` from tractor; keeps
  `open/attach_shm_array` as public API aliases.
- Trim shim to only piker-specific wrappers:
  `_make_token()`, `maybe_open_shm_array()`,
  `try_read()`.
- Drop `Optional` usage in shim, use `|None`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-03-17 18:00:29 -04:00
Gud Boi 964f207150 Replace assert with warn for no-gaps in `.storage.cli`
Change `assert aids` to a warning log when no history gaps are found
during `ldshm` gap detection; it is the **ideal case** OBVI. This avoids
crashing the CLI when gap detection finds no issues, which is actually
good news!

Bp

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-22 22:08:35 -05:00
Gud Boi 9257af02b9 Mv `markup_gaps()` to new `.tsp._annotate` mod 2026-02-22 22:08:34 -05:00
Gud Boi 5f6e24f55c Tolerate various "bad data" cases in `markup_gaps()`
Namely such that when the previous-df-row by our shm-abs-'index' doesn't
exist we ignore certain cases which are likely due to borked-but-benign
samples written to the tsdb or rt shm buffers prior.

Particularly we now ignore,
- any `dt`/`prev_dt` values which are UNIX-epoch timestamped (val of 0).
- any row-is-first-row in the df; there is no previous.
- any missing previous datum by 'index', in which case we lookup the
  `wdts` prior row and use that instead.
  * this would indicate a missing sample for the time-step but we can
    still detect a "gap" by looking at the prior row, by df-abs-index
    `i`, and use its timestamp to determine the period/size of missing
    samples (which need to likely still be retrieved).
  * in this case i'm leaving in a pause-point for introspecting these
    rarer cases when `--pdb` is passed via CLI.

Relatedly in the `piker store` CLI ep,
- add `--pdb` flag to `piker store`, pass it verbatim as `debug_mode`.
- when `times` has only a single row, don't calc a `period_s` median.
- only trace `null_segs` when in debug mode.
- always markup/dedupe gaps for `period_s==60`
2026-02-22 22:08:34 -05:00
Gud Boi d5af471192 Add vlm-based "smart" OHLCV de-duping & bar validation
Using `claude`, add a `.tsp._dedupe_smart` module that attemps "smarter"
duplicate bars by attempting to distinguish between erroneous bars
partially written during concurrent backfill race conditions vs.
**actual** data quality issues from historical providers.

Problem:
--------
Concurrent writes (live updates vs. backfilling) can result in create
duplicate timestamped ohlcv vars with different values. Some
potential scenarios include,

- a market live feed is cancelled during live update resulting in the
  "last" datum being partially updated with all the ticks for the
  time step.
- when the feed is rebooted during charting, the backfiller will not
  finalize this bar since rn it presumes it should only fill data for
  time steps not already in the tsdb storage.

Our current naive  `.unique()` approach obvi keeps the incomplete bar
and a "smarter" approach is to compare the provider's final vlm
amount vs. the maybe-cancelled tsdb's bar; a higher vlm value from
the provider likely indicates the cancelled-during-live-write and
**not** a datum discrepancy from said data provider.

Analysis (with `claude`) of `zecusdt` data revealed:
- 1000 duplicate timestamps
- 999 identical bars (pure duplicates from 2022 backfill overlap)
- 1 volume-monotonic conflict (live partial vs backfill complete)

A soln from `claude` -> `tsp._dedupe_smart.dedupe_ohlcv_smart()`
which:
- sorts by vlm **before** deduplication and keep the most complete
  bar based on vlm monotonicity as well as the following OHLCV
  validation assumptions:
  * volume should always increase
  * high should be non-decreasing,
  * low should be non-increasing
  * open should be identical
- Separates valid race conditions from provider data quality issues
  and reports and returns both dfs.

Change summary by `claude`:
- `.tsp._dedupe_smart`: new module with validation logic
- `.tsp.__init__`: expose `dedupe_ohlcv_smart()`
- `.storage.cli`: integrate smart dedupe, add logging for:
  * duplicate counts (identical vs monotonic races)
  * data quality violations (non-monotonic, invalid OHLC ranges)
  * warnings for provider data issues
- Remove `assert not diff` (duplicates are valid now)

Verified on `zecusdt`: correctly keeps index 3143645
(volume=287.777) over 3143644 (volume=140.299) for
conflicting 2026-01-16 18:54 UTC bar.

`claude`'s Summary of reasoning
-------------------------------
- volume monotonicity is critical: a bar's volume only increases
  during its time window.
- a backfilled bar should always have volume >= live updated.
- violations indicate any of:
  * Provider data corruption
  * Non-OHLCV aggregation semantics
  * Timestamp misalignment

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-02-22 22:08:34 -05:00
Tyler Goodlet f4d9090d6d `.storage.__init__`: code styling updates 2026-02-22 22:08:34 -05:00
Tyler Goodlet c390e87536 `.storage.cli`: collect gap-markup-aids into `tf2aids: dict` prior to pause for introspection 2025-02-19 17:01:24 -05:00
Tyler Goodlet 5e4a6d61c7 Ignore any non-`.parquet` files under `.config/piker/nativedb/` subdir 2025-02-19 17:01:24 -05:00
Tyler Goodlet 52b349fe79 Always reload shm data before annotating gaps, so they line up.. 2024-01-09 15:55:16 -05:00
Tyler Goodlet 6959429af8 Factor gap annotating into new `markup_gaps()`
Since we definitely want to markup gaps that are both data-errors and
just plain old venue closures (time gaps), generalize the `gaps:
pl.DataFrame` loop in a func and call it from the `ldshm` cmd for now.

Some other tweaks to `store ldshm`:
- add `np.median()` failover when detecting (ohlc) ts sample rate.
- use new `tsp.dedupe()` signature.
- differentiate between sample-period size gaps and venue closure gaps
  by calling `tsp.detect_time_gaps()` with diff size thresholds.
- add todo for backfilling null-segs when detected.
2024-01-04 11:01:21 -05:00
Tyler Goodlet a86573b5a2 Fix .parquet filenaming..
Apparently `.storage.nativedb.mk_ohlcv_shm_keyed_filepath()` was always
kinda broken if you passed in a `period: float` with an actual non-`int`
to the format string? Fixed it to strictly cast to `int()` before
str-ifying so that you don't get weird `60.0s.parquet` in there..

Further this rejigs the `sotre ldshm` gap correction-annotation loop to,
- use `StorageClient.write_ohlcv()` instead of hackily re-implementing
  it.. now that problem from above is fixed!
- use a `needs_correction: bool` var to determine if gap markup and
  de-duplictated data should be pushed to the shm buffer,
- go back to using `AnnotCtl.add_rect()` for all detected gaps such that
  they all persist (and thus are shown together) until the client
  disconnects.
2023-12-26 17:14:26 -05:00
Tyler Goodlet b064a5f94d A working remote annotations controller B)
Obvi took a little `.ui` component fixing (as per prior commits) but
this is now a working PoC for gap detection and markup from a remote
(data) non-`chart` actor!

Iface and impl deats from `.ui._remote_ctl`:
- add new `open_annot_ctl()` mngr which attaches to all locally
  discoverable chart actors, gathers annot-ctl streams per fqme set, and
  delivers a new `AnnotCtl` client which allows adding annotation
  rectangles via a `.add_rect()` method.
  - also template out some other soon-to-get methods for removing and
    modifying pre-exiting annotations on some `ChartView` 💥
- ensure the `chart` CLI subcmd starts the (`qtloops`) guest-mode init
  with the `.ui._remote_ctl` module enabled.
- actually use this stuff in the `piker store ldshm` CLI to submit
  markup rects around any detected null/time gaps in the tsdb data!

Still lots to do:
-  probably colorization of gaps depending on if they're venue
   closures (aka real mkt gaps) vs. "missing data" from the backend (aka
   timeseries consistency gaps).
- run gap detection and markup as part of the std `.tsp` sub-sys
   runtime such that gap annots are a std "built-in" feature of
   charting.
- support for epoch time stamp AND abs-shm-index rect x-values
  (depending on chart operational state).
2023-12-22 15:19:20 -05:00
Tyler Goodlet f7cc43ee0b Add pauses to `store anal/ldshm` only on bad segs
Particularly halting before maybe writing the repaired timeseries
history in `store anal` to optionally allow user to avoid writing to
storage.
2023-12-18 11:56:57 -05:00
Tyler Goodlet f5dc21d3f4 Adjust all `.tsp` imports to use new sub-pkg
Also toss in a poll loop around the `hist_shm: ShmArray` backfill
read-check in the `.data.allocate_persisten_feed()`  init to cope with
possible racy-ness from the increased tsdb history loading concurrency
now implemented.
2023-12-18 11:54:28 -05:00
Tyler Goodlet 8989c73a93 Move `iter_dfs_from_shms` into `.data.history`
Thinking about just moving all of that module (after a content breakup)
to a new `.piker.tsp` which will mostly depend on the `.data` and
`.storage` sub-pkgs; the idea is to move biz-logic for tsdb IO/mgmt and
orchestration with real-time (shm) buffers and the graphics layer into
a common spot for both manual analysis/research work and better
separation of low level data structure primitives from their higher
level usage.

Add a better `data.history` mod doc string in prep for this move
as well as clean out a bunch of legacy commented cruft from the
`trimeter` and `marketstore` days.

TO CHERRY #486 (if we can)
2023-12-15 15:53:02 -05:00
Tyler Goodlet c4853a3fee Drop inter-method NL 2023-12-13 09:27:23 -05:00
Tyler Goodlet f274c3db3b Import `np2pl()` from `.data.tsp`
Also toss in todo for a timeseries search CLI cmd which can be handy
when doing offine store mgmt.
2023-12-13 09:25:44 -05:00
Tyler Goodlet f7a8d79b7b Add `NativeStorageClient._cache_df()` use it in `.write_ohlcv()` for caching on writes as well 2023-12-11 20:10:53 -05:00
Tyler Goodlet 49c458710e Move `numpy` <-> `polars` converters into `.data.tsp`
Yet again these are (going to be) generally useful in the data proc
layer as well as going forward with (possibly) moving the history and
shm rt-processing layer to apache (arrow or other) shared-ds
equivalents.
2023-12-11 17:53:31 -05:00
Tyler Goodlet b94582cb35 Move `dedupe()` to `.data.tsp` (so it has pals)
Includes a rename of `.data._timeseries` -> `.data.tsp` for "time series
processing", making it a public sub-mod; it contains a highly useful set
of data-frame and `numpy.ndarray` ops routines in various subsystems Bo
2023-12-11 16:24:27 -05:00
Tyler Goodlet 7311000846 Facepalm, set `was_deduped` as bool not the deduped frame.. 2023-12-11 13:18:10 -05:00
Tyler Goodlet 2eeef2a123 Add `dedupe()` to help with gap detection/resolution
Think i finally figured out the weird issue without out-of-order OHLC
history getting jammed in the wrong place:
- gap is detected in parquet/offline ts (likely due to a zero dt or
  other gap),
- query for history in the gap is made BUT that frame is then inserted
  in the shm buffer **at the end** (likely using array int-entry
  indexing) which inserts it at the wrong location,
- later this out-of-order frame is written to the storage layer
  (parquet) and then is repeated on further reboots with the original
  gap causing further queries for the same frame on every history
  backfill.

A set of tools useful for detecting these issues and annotating them
nicely on chart part of this patch's intent:
- `dedupe()` will detect any dt gaps, deduplicate datetime rows and
  return the de-duplicated df along with gaps table.
- use this in both `piker store anal` such that we potentially
  resolve and backfill the gaps correctly if some rows were removed.
- possibly also use this to detect the backfilling error in logic at
  the time of backfilling the frame instead of after the fact (which
  would require re-writing the shm array from something like `store
  ldshm` and would be a manual post-hoc solution, not a fix to the
  original issue..
2023-12-08 15:11:34 -05:00
Tyler Goodlet b9af6176c5 Factor `TimeseriesNotFound` to top level
TO CHERRY into #486
2023-12-07 12:31:14 -05:00
Tyler Goodlet 22bd83943b .storage: support `store anal --pdb` flag 2023-12-04 13:00:33 -05:00
Tyler Goodlet 239c1c457e Sort fqme suggestions pre-print 2023-12-04 11:34:39 -05:00
Tyler Goodlet 24a54a7085 Add `TimeseriesNotFound` for fqme lookup failures
A common usage error is to run `piker anal mnq.cme.ib` where the CLI
passed fqme is not actually fully-qualified (in this case missing an
expiry token) and we get an underlying `FileNotFoundError` from the
`StorageClient.read_ohlcv()` call. In such key misses, scan the existing
`StorageClient._index` for possible matches and report in a `raise from`
the new error.

CHERRY into #486
2023-12-04 11:22:55 -05:00
Tyler Goodlet a382f01c85 Move tsdb section to `service.tsdb.name` and get host from `.maddrs` 2023-10-01 17:23:39 -04:00
Tyler Goodlet f94244aad4 Load `network` section from `conf.toml` for service-addr map 2023-09-28 12:04:24 -04:00
Tyler Goodlet c3f8b089be Drop `.service._ahab` from storage cli runtime mods 2023-08-25 13:33:59 -04:00
Tyler Goodlet 385561276b Add gap detection into the `store ldshm` cmd 2023-07-26 15:45:55 -04:00
Tyler Goodlet 64329d44e7 Flip `tractor.breakpoint()`s to new `.pause()` 2023-07-26 12:48:19 -04:00
Tyler Goodlet 5e7916a0df Start `piker.toolz` subpkg for all our tooling B)
Since there's a growing list of top level mods which are more or less
utils/tools for working with the runtime; begin to move them into a new
subpkg starting with a new `.toolz.debug`.

Start with,
- a new `open_crash_handller()` for doing breakpoints around blocks that
  might error.
- move in what was `piker._profile` into `.toolz.profile` and adjust all
  importing appropriately.
2023-07-20 15:23:01 -04:00
Tyler Goodlet c9681d0aa2 .nativedb: ignore an `expired/` subdir 2023-07-12 08:45:55 -04:00
Tyler Goodlet d704d631ba Add `store ldshm` subcmd
Changed from the old `store clone` to instead simply load any shm buffer
matching a user provided `FQME: str` pattern; writing to parquet file is
only done if an explicit option flag is passed by user.

Implement new `iter_dfs_from_shms()` generator which allows interatively
loading both 1m and 1s buffers delivering the `Path`, `ShmArray` and
`polars.DataFrame` instances per matching file B)

Also add a todo for a `NativeStorageClient.clear_range()` method.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 58c096bfad Bleh go back to using pdbp for REPL in anal 2023-06-27 13:41:47 -04:00
Tyler Goodlet c1546eb043 Add note about appending parquet files on write 2023-06-27 13:41:47 -04:00
Tyler Goodlet fda7111305 Import from new `.data._timeseries` mod for anal 2023-06-27 13:41:47 -04:00
Tyler Goodlet f25248c871 Add `.data._timeseries` utility mod
Org all the new (time) gap detection routines here and also move in the
`slice_from_time()` epoch -> index converter routine from `._pathops` B)
2023-06-27 13:41:47 -04:00
Tyler Goodlet 54f8a615fc Use `code.interact()` in anal subcmd for now 2023-06-27 13:41:47 -04:00
Tyler Goodlet 2dbcecdac7 Generalize time-gap detector to accept unit and threshold 2023-06-27 13:41:47 -04:00