Simplify `store` series and SHM discovery
Make `store ls` show every NativeDB FQME and its available sample periods by default. Keep `store series` as the detailed view of each exact sub-series, including its period, size and path. Share the same optional FQME filter and numerically ordered payload. Use `store shm` as the canonical volatile-buffer ep and make its default operation a read-only report. Enter the prior persistence workflow only through `--write-parquet`, require that flag before `--reload-parquet-to-shm`. Reject read-only formatting options in repair mode instead of silently ignoring them. Also, - update the qualification docs, debug helper and harness refs; - cover no-flag listings, command help and SHM mode validation; - retain exact generation and object-name selection for stale SHM. Prompt-IO: ai/prompt-io/opencode/20260801T015321Z_74ea2152_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))wkt/backfiller_deep_fixes
parent
74ea2152f9
commit
f0f3c80657
|
|
@ -47,7 +47,7 @@ that's why it's like 1000x faster ya know?"
|
||||||
- Wrap all code symbols: `function()`,
|
- Wrap all code symbols: `function()`,
|
||||||
`ClassName`, `field_name`
|
`ClassName`, `field_name`
|
||||||
- File paths: `piker/ui/_remote_ctl.py`
|
- File paths: `piker/ui/_remote_ctl.py`
|
||||||
- Commands: `git status`, `piker store ldshm`
|
- Commands: `git status`, `piker store shm`
|
||||||
|
|
||||||
**Explain like you're pair programming:**
|
**Explain like you're pair programming:**
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -134,8 +134,8 @@ Require explicit authorization before running:
|
||||||
`tests/test_cli.py` is currently hard-skipped. It is not an active CLI
|
`tests/test_cli.py` is currently hard-skipped. It is not an active CLI
|
||||||
regression gate.
|
regression gate.
|
||||||
|
|
||||||
Never execute `piker store anal` or `piker store ldshm` as tests. They are
|
Never execute `piker store anal` or `piker store shm --write-parquet`
|
||||||
mutating or interactive operational commands.
|
as tests. They are mutating or interactive operational commands.
|
||||||
|
|
||||||
## Project-Specific Flags And Backend Matrix
|
## Project-Specific Flags And Backend Matrix
|
||||||
|
|
||||||
|
|
@ -194,7 +194,7 @@ tests/
|
||||||
test_feeds.py live Binance/Kraken feeds and shared memory
|
test_feeds.py live Binance/Kraken feeds and shared memory
|
||||||
test_ib_history.py deterministic IB history request formatting
|
test_ib_history.py deterministic IB history request formatting
|
||||||
test_history_backfill.py deterministic history/SHM orchestration
|
test_history_backfill.py deterministic history/SHM orchestration
|
||||||
test_ldshm.py ldshm unpublished-slot guard
|
test_ldshm.py SHM unpublished-slot guard
|
||||||
test_questrade.py obsolete credentialed tests; skipped
|
test_questrade.py obsolete credentialed tests; skipped
|
||||||
test_services.py pikerd/datad/feed/EMS actor lifecycle
|
test_services.py pikerd/datad/feed/EMS actor lifecycle
|
||||||
test_store_cli.py storage command help and diagnostics UX
|
test_store_cli.py storage command help and diagnostics UX
|
||||||
|
|
@ -214,7 +214,7 @@ tests/
|
||||||
| `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only |
|
| `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only |
|
||||||
| `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network |
|
| `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network |
|
||||||
| `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM |
|
| `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM |
|
||||||
| `piker/storage/cli.py` ldshm null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
|
| `piker/storage/cli.py` SHM null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
|
||||||
| `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch |
|
| `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch |
|
||||||
| `piker/accounting/` | targeted accounting node | some cases use live/configured state |
|
| `piker/accounting/` | targeted accounting node | some cases use live/configured state |
|
||||||
| `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in |
|
| `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in |
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: ses_0799212ebffe42arY96czXn89F
|
||||||
|
timestamp: 2026-08-01T01:53:21Z
|
||||||
|
git_ref: 74ea2152
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260801T015321Z_74ea2152_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Make `piker store ls` default to all NativeDB series as a compact counterpart
|
||||||
|
to detailed `series`, with sensible no-flag output and a visible help
|
||||||
|
description. Consolidate SHM inspection and repair under canonical `shm`,
|
||||||
|
because attachment to volatile segments is implied, and make inspection the
|
||||||
|
safe default while retaining explicit persistence and reload controls.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Replaced backend-oriented `ls` with compact NativeDB discovery and made both
|
||||||
|
listing commands default to all series. Consolidated SHM behavior under a
|
||||||
|
read-only-by-default `shm` command, guarded explicit repair options, updated
|
||||||
|
executable references and docs, and added command regressions.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/storage/cli.py` - listing defaults and unified SHM command modes.
|
||||||
|
- `tests/test_store_cli.py` - listing, help, and SHM mode regressions.
|
||||||
|
- `tests/test_ldshm.py` - explicit persistence-path regressions.
|
||||||
|
- `docs/manual_backfill_qualification.rst` - revised safe CLI workflow.
|
||||||
|
- `snippets/claude_debug_helper.py` - explicit SHM persistence commands.
|
||||||
|
- `.claude/skills/run-tests/test-harness-reference.md` - safety and test map.
|
||||||
|
- `.claude/skills/piker-slang/examples.md` - canonical command example.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
None - generated output incorporates the user's design refinements.
|
||||||
|
|
@ -0,0 +1,78 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-01T01:53:21Z
|
||||||
|
git_ref: 74ea2152
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
The user requested command-surface refinements after reviewing the new
|
||||||
|
storage CLI. Bare `ls` should default to all series and act as a compact
|
||||||
|
form of detailed `series`; commands should produce sensible default output
|
||||||
|
without requiring trigger flags; and `ls` needed a parent-help description.
|
||||||
|
The user first proposed folding read-only SHM inspection into `ldshm -r`,
|
||||||
|
then clarified that `shm` is the better canonical command because attaching
|
||||||
|
to volatile segments is implied by the resource name.
|
||||||
|
|
||||||
|
The resulting design uses `piker store ls` as a compact NativeDB FQME and
|
||||||
|
period view, while `piker store series` reports each exact sub-series with
|
||||||
|
size and path details. Both default to all records and accept an optional
|
||||||
|
FQME substring.
|
||||||
|
|
||||||
|
`piker store shm FQME` is read-only by default. Persistence and reload enter
|
||||||
|
the prior repair workflow only through explicit `--write-parquet` and
|
||||||
|
`--reload-parquet-to-shm` flags. Reload requires persistence, and read-only
|
||||||
|
formatting flags are rejected in repair mode instead of being ignored.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
|
||||||
|
|
||||||
|
Generated shared NativeDB payload collection, compact and detailed output,
|
||||||
|
numeric period ordering, default-all semantics, the canonical `shm` command,
|
||||||
|
safe mode dispatch, explicit repair validation, and debugger propagation in
|
||||||
|
read-only mode.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_store_cli.py`
|
||||||
|
|
||||||
|
Generated regressions for no-flag listing, parent-help descriptions, numeric
|
||||||
|
period display, default read-only SHM JSON, reload dependencies, and repair
|
||||||
|
versus read-only option conflicts.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
|
||||||
|
|
||||||
|
Adjusted persistence regressions for the renamed command and explicit repair
|
||||||
|
entry while preserving the historical failure rationale.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- docs/manual_backfill_qualification.rst`
|
||||||
|
|
||||||
|
Documented compact versus detailed NativeDB discovery, default read-only SHM
|
||||||
|
inspection, and the explicitly mutating repair invocation.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- snippets/claude_debug_helper.py`
|
||||||
|
|
||||||
|
Updated executable debug helper commands to select explicit SHM persistence.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||||
|
|
||||||
|
Updated operational safety guidance and SHM regression mappings.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .claude/skills/piker-slang/examples.md`
|
||||||
|
|
||||||
|
Updated the command example to the canonical `shm` spelling.
|
||||||
|
|
||||||
|
Verification output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
................................ [100%]
|
||||||
|
104 passed in 5.77s
|
||||||
|
```
|
||||||
|
|
||||||
|
Real command-help rendering was checked for the parent group, `ls`, `series`,
|
||||||
|
and `shm`. Real read-only `ls` and `series --json` output was also exercised
|
||||||
|
against the configured NativeDB. `git diff --check` passed.
|
||||||
|
|
||||||
|
Adversarial review identified and drove fixes for lexicographic period
|
||||||
|
ordering, stale runbook safety wording, ignored mode-specific options, and an
|
||||||
|
explicit-default `--max-gaps 10` validation bypass. Final review found no
|
||||||
|
remaining issues. Real OS-backed SHM attachment and successful repair/reload
|
||||||
|
remain integration-level validation concerns.
|
||||||
|
|
@ -24,8 +24,8 @@ automatically corruption or an expected venue closure.
|
||||||
|
|
||||||
Do not run qualification against normal user storage. The helper below
|
Do not run qualification against normal user storage. The helper below
|
||||||
requires a marked disposable root before it will seed, archive, or clear
|
requires a marked disposable root before it will seed, archive, or clear
|
||||||
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive
|
a Parquet. ``piker store anal`` and ``piker store shm --write-parquet``
|
||||||
and potentially mutating; they are not audit commands.
|
are interactive and potentially mutating; they are not audit commands.
|
||||||
|
|
||||||
Keep the disposable root private and do not rename, replace, or symlink
|
Keep the disposable root private and do not rename, replace, or symlink
|
||||||
files under it while a helper is running. Stop the chart cleanly before
|
files under it while a helper is running. Stop the chart cleanly before
|
||||||
|
|
@ -196,11 +196,13 @@ Gap Layer Triage
|
||||||
List exact durable series before choosing a timeframe, and inspect live
|
List exact durable series before choosing a timeframe, and inspect live
|
||||||
SHM separately::
|
SHM separately::
|
||||||
|
|
||||||
|
piker store ls
|
||||||
piker store series FQME_SUBSTRING
|
piker store series FQME_SUBSTRING
|
||||||
piker store shm FQME --max-gaps 20
|
piker store shm FQME --max-gaps 20
|
||||||
|
|
||||||
Both commands are read-only. ``series`` reports each ``(FQME, period)``
|
All three forms are read-only. ``ls`` summarizes periods by FQME, while
|
||||||
file independently. ``shm`` reports every exact actor generation and
|
``series`` reports each ``(FQME, period)`` file independently. Read-only
|
||||||
|
``shm`` reports every exact actor generation and
|
||||||
``hist``/``rt`` buffer for the FQME, including invalid rows, ordering
|
``hist``/``rt`` buffer for the FQME, including invalid rows, ordering
|
||||||
defects, inferred versus expected cadence, and the largest timestamp gaps.
|
defects, inferred versus expected cadence, and the largest timestamp gaps.
|
||||||
Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer.
|
Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer.
|
||||||
|
|
|
||||||
|
|
@ -51,7 +51,6 @@ from piker import tsp
|
||||||
from piker import config
|
from piker import config
|
||||||
from . import log
|
from . import log
|
||||||
from . import (
|
from . import (
|
||||||
__tsdbs__,
|
|
||||||
open_storage_client,
|
open_storage_client,
|
||||||
StorageClient,
|
StorageClient,
|
||||||
)
|
)
|
||||||
|
|
@ -270,104 +269,14 @@ def _render_audit_report(report: dict) -> None:
|
||||||
console.print(gap_table)
|
console.print(gap_table)
|
||||||
|
|
||||||
|
|
||||||
@store.command(no_args_is_help=True)
|
def _native_series_payload(
|
||||||
def ls(
|
pattern: str|None,
|
||||||
backends: Annotated[
|
|
||||||
list[str]|None,
|
|
||||||
typer.Argument(help='Storage backends to query.'),
|
|
||||||
] = None,
|
|
||||||
all_backends: Annotated[
|
|
||||||
bool,
|
|
||||||
typer.Option(
|
|
||||||
'--all',
|
|
||||||
help='Query every configured storage backend.',
|
|
||||||
),
|
|
||||||
] = False,
|
|
||||||
):
|
|
||||||
from rich.table import Table
|
|
||||||
|
|
||||||
if (
|
) -> tuple[Path, list[dict]]:
|
||||||
all_backends
|
|
||||||
and
|
|
||||||
backends
|
|
||||||
):
|
|
||||||
raise typer.BadParameter(
|
|
||||||
'Pass backend names or --all, not both'
|
|
||||||
)
|
|
||||||
if all_backends:
|
|
||||||
backends = list(__tsdbs__)
|
|
||||||
elif not backends:
|
|
||||||
raise typer.BadParameter('Pass backend names or --all')
|
|
||||||
|
|
||||||
console = Console()
|
|
||||||
|
|
||||||
async def query_all():
|
|
||||||
nonlocal backends
|
|
||||||
|
|
||||||
async with (
|
|
||||||
open_piker_runtime(
|
|
||||||
'tsdb_storage',
|
|
||||||
),
|
|
||||||
):
|
|
||||||
for i, backend in enumerate(backends):
|
|
||||||
table = Table()
|
|
||||||
try:
|
|
||||||
async with open_storage_client(backend=backend) as (
|
|
||||||
mod,
|
|
||||||
client,
|
|
||||||
):
|
|
||||||
table.add_column(f'{mod.name}@{client.address}')
|
|
||||||
keys: list[str] = await client.list_keys()
|
|
||||||
for key in keys:
|
|
||||||
table.add_row(key)
|
|
||||||
|
|
||||||
console.print(table)
|
|
||||||
except Exception:
|
|
||||||
log.error(f'Unable to connect to storage engine: `{backend}`')
|
|
||||||
|
|
||||||
trio.run(query_all)
|
|
||||||
|
|
||||||
|
|
||||||
@store.command(no_args_is_help=True)
|
|
||||||
def series(
|
|
||||||
pattern: Annotated[
|
|
||||||
str|None,
|
|
||||||
typer.Argument(
|
|
||||||
help='Optional case-insensitive FQME substring.',
|
|
||||||
),
|
|
||||||
] = None,
|
|
||||||
all_series: Annotated[
|
|
||||||
bool,
|
|
||||||
typer.Option(
|
|
||||||
'--all',
|
|
||||||
help='List every canonical NativeDB series.',
|
|
||||||
),
|
|
||||||
] = False,
|
|
||||||
json_output: Annotated[
|
|
||||||
bool,
|
|
||||||
typer.Option(
|
|
||||||
'--json',
|
|
||||||
help='Emit machine-readable JSON.',
|
|
||||||
),
|
|
||||||
] = False,
|
|
||||||
) -> None:
|
|
||||||
'''
|
'''
|
||||||
List exact NativeDB series identities without opening a runtime.
|
Collect exact NativeDB series metadata for CLI presentation.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
if (
|
|
||||||
all_series
|
|
||||||
and
|
|
||||||
pattern is not None
|
|
||||||
):
|
|
||||||
raise typer.BadParameter('Pass a pattern or --all, not both')
|
|
||||||
if (
|
|
||||||
not all_series
|
|
||||||
and
|
|
||||||
pattern is None
|
|
||||||
):
|
|
||||||
raise typer.BadParameter('Pass a pattern or --all')
|
|
||||||
|
|
||||||
datadir: Path = config.get_conf_dir() / 'nativedb'
|
datadir: Path = config.get_conf_dir() / 'nativedb'
|
||||||
refs: list[NativeSeriesRef] = list(iter_native_series(datadir))
|
refs: list[NativeSeriesRef] = list(iter_native_series(datadir))
|
||||||
if pattern is not None:
|
if pattern is not None:
|
||||||
|
|
@ -390,6 +299,96 @@ def series(
|
||||||
'path': str(ref.path),
|
'path': str(ref.path),
|
||||||
'size_bytes': size_bytes,
|
'size_bytes': size_bytes,
|
||||||
})
|
})
|
||||||
|
payload.sort(key=lambda item: (
|
||||||
|
item['fqme'],
|
||||||
|
item['period_s'],
|
||||||
|
))
|
||||||
|
|
||||||
|
return datadir, payload
|
||||||
|
|
||||||
|
|
||||||
|
@store.command()
|
||||||
|
def ls(
|
||||||
|
pattern: Annotated[
|
||||||
|
str|None,
|
||||||
|
typer.Argument(
|
||||||
|
help='Optional case-insensitive FQME substring.',
|
||||||
|
),
|
||||||
|
] = None,
|
||||||
|
all_series: Annotated[
|
||||||
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--all',
|
||||||
|
help='List all FQMEs (the default).',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
List NativeDB FQMEs and their available sample periods.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
all_series
|
||||||
|
and
|
||||||
|
pattern is not None
|
||||||
|
):
|
||||||
|
raise typer.BadParameter('Pass a pattern or --all, not both')
|
||||||
|
|
||||||
|
datadir, payload = _native_series_payload(pattern)
|
||||||
|
periods_by_fqme: dict[str, list[int]] = {}
|
||||||
|
for item in payload:
|
||||||
|
periods_by_fqme.setdefault(item['fqme'], []).append(
|
||||||
|
item['period_s']
|
||||||
|
)
|
||||||
|
|
||||||
|
table = Table(title=f'NativeDB series @ {datadir}')
|
||||||
|
table.add_column('FQME')
|
||||||
|
table.add_column('Periods')
|
||||||
|
for fqme, periods in periods_by_fqme.items():
|
||||||
|
table.add_row(
|
||||||
|
fqme,
|
||||||
|
', '.join(
|
||||||
|
f'{period}s'
|
||||||
|
for period in sorted(periods)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
Console().print(table)
|
||||||
|
|
||||||
|
|
||||||
|
@store.command()
|
||||||
|
def series(
|
||||||
|
pattern: Annotated[
|
||||||
|
str|None,
|
||||||
|
typer.Argument(
|
||||||
|
help='Optional case-insensitive FQME substring.',
|
||||||
|
),
|
||||||
|
] = None,
|
||||||
|
all_series: Annotated[
|
||||||
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--all',
|
||||||
|
help='List all series (the default).',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
|
json_output: Annotated[
|
||||||
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--json',
|
||||||
|
help='Emit machine-readable JSON.',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
List detailed NativeDB sub-series without opening a runtime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
all_series
|
||||||
|
and
|
||||||
|
pattern is not None
|
||||||
|
):
|
||||||
|
raise typer.BadParameter('Pass a pattern or --all, not both')
|
||||||
|
datadir, payload = _native_series_payload(pattern)
|
||||||
if json_output:
|
if json_output:
|
||||||
typer.echo(json.dumps(payload, indent=2, sort_keys=True))
|
typer.echo(json.dumps(payload, indent=2, sort_keys=True))
|
||||||
return
|
return
|
||||||
|
|
@ -712,8 +711,7 @@ def anal(
|
||||||
trio.run(main)
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
@store.command('shm', no_args_is_help=True)
|
def _inspect_shm(
|
||||||
def inspect_shm(
|
|
||||||
fqme: str,
|
fqme: str,
|
||||||
shm_name: Annotated[
|
shm_name: Annotated[
|
||||||
str|None,
|
str|None,
|
||||||
|
|
@ -737,6 +735,7 @@ def inspect_shm(
|
||||||
help='Emit machine-readable JSON.',
|
help='Emit machine-readable JSON.',
|
||||||
),
|
),
|
||||||
] = False,
|
] = False,
|
||||||
|
pdb: bool = False,
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Inspect matching OHLCV SHM buffers without mutating them.
|
Inspect matching OHLCV SHM buffers without mutating them.
|
||||||
|
|
@ -765,7 +764,10 @@ def inspect_shm(
|
||||||
reports: list[dict] = []
|
reports: list[dict] = []
|
||||||
|
|
||||||
async def main() -> None:
|
async def main() -> None:
|
||||||
async with open_piker_runtime('shm_inspector'):
|
async with open_piker_runtime(
|
||||||
|
'shm_inspector',
|
||||||
|
debug_mode=pdb,
|
||||||
|
):
|
||||||
for (
|
for (
|
||||||
shmfile,
|
shmfile,
|
||||||
shm,
|
shm,
|
||||||
|
|
@ -840,26 +842,88 @@ def inspect_shm(
|
||||||
|
|
||||||
|
|
||||||
@store.command(no_args_is_help=True)
|
@store.command(no_args_is_help=True)
|
||||||
def ldshm(
|
def shm(
|
||||||
fqme: str,
|
fqme: str,
|
||||||
write_parquet: bool = True,
|
write_parquet: Annotated[
|
||||||
reload_parquet_to_shm: bool = True,
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--write-parquet',
|
||||||
|
help='Persist repaired frames to NativeDB.',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
|
reload_parquet_to_shm: Annotated[
|
||||||
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--reload-parquet-to-shm',
|
||||||
|
help='Reload persisted repairs into SHM.',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
pdb: bool = False, # --pdb passed?
|
pdb: bool = False, # --pdb passed?
|
||||||
|
max_gaps: Annotated[
|
||||||
|
int|None,
|
||||||
|
typer.Option(
|
||||||
|
'--max-gaps',
|
||||||
|
min=0,
|
||||||
|
help=(
|
||||||
|
'Maximum largest gaps in read-only output '
|
||||||
|
'(default: 10).'
|
||||||
|
),
|
||||||
|
show_default=False,
|
||||||
|
),
|
||||||
|
] = None,
|
||||||
|
json_output: Annotated[
|
||||||
|
bool,
|
||||||
|
typer.Option(
|
||||||
|
'--json',
|
||||||
|
help='Emit read-only output as JSON.',
|
||||||
|
),
|
||||||
|
] = False,
|
||||||
shm_name: Annotated[
|
shm_name: Annotated[
|
||||||
str|None,
|
str|None,
|
||||||
typer.Option(
|
typer.Option(
|
||||||
'--shm-name',
|
'--shm-name',
|
||||||
help='Process only this exact SHM object name.',
|
help='Select only this exact SHM object name.',
|
||||||
),
|
),
|
||||||
] = None,
|
] = None,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Linux ONLY: load any fqme file name matching shm buffer from
|
Inspect volatile OHLCV SHM, with explicit repair options.
|
||||||
/dev/shm/ into an OHLCV numpy array and polars DataFrame,
|
|
||||||
optionally write to offline storage via `.parquet` file.
|
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
if (
|
||||||
|
reload_parquet_to_shm
|
||||||
|
and
|
||||||
|
not write_parquet
|
||||||
|
):
|
||||||
|
raise typer.BadParameter(
|
||||||
|
'--reload-parquet-to-shm requires --write-parquet'
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
write_parquet
|
||||||
|
and (
|
||||||
|
json_output
|
||||||
|
or
|
||||||
|
max_gaps is not None
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise typer.BadParameter(
|
||||||
|
'--json and --max-gaps apply only to read-only output'
|
||||||
|
)
|
||||||
|
if not write_parquet:
|
||||||
|
_inspect_shm(
|
||||||
|
fqme,
|
||||||
|
shm_name=shm_name,
|
||||||
|
max_gaps=(
|
||||||
|
10
|
||||||
|
if max_gaps is None
|
||||||
|
else max_gaps
|
||||||
|
),
|
||||||
|
json_output=json_output,
|
||||||
|
pdb=pdb,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
if (
|
if (
|
||||||
shm_name is not None
|
shm_name is not None
|
||||||
and
|
and
|
||||||
|
|
|
||||||
|
|
@ -59,7 +59,10 @@ def expect(
|
||||||
|
|
||||||
def run_pdb_commands(
|
def run_pdb_commands(
|
||||||
commands: list[str],
|
commands: list[str],
|
||||||
initial_cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
initial_cmd: str = (
|
||||||
|
'piker store shm xmrusdt.usdtm.perp.binance '
|
||||||
|
'--write-parquet'
|
||||||
|
),
|
||||||
timeout: int = 30,
|
timeout: int = 30,
|
||||||
print_output: bool = True,
|
print_output: bool = True,
|
||||||
) -> dict[str, str]:
|
) -> dict[str, str]:
|
||||||
|
|
@ -145,7 +148,10 @@ class InteractivePdbSession:
|
||||||
'''
|
'''
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
cmd: str = (
|
||||||
|
'piker store shm xmrusdt.usdtm.perp.binance '
|
||||||
|
'--write-parquet'
|
||||||
|
),
|
||||||
timeout: int = 30,
|
timeout: int = 30,
|
||||||
):
|
):
|
||||||
self.cmd: str = cmd
|
self.cmd: str = cmd
|
||||||
|
|
|
||||||
|
|
@ -21,16 +21,16 @@ from piker.storage import cli as storage_cli
|
||||||
from piker.storage.cli import (
|
from piker.storage.cli import (
|
||||||
_shm_period_and_invalid_count,
|
_shm_period_and_invalid_count,
|
||||||
_summarize_shm_frame,
|
_summarize_shm_frame,
|
||||||
ldshm,
|
shm as shm_cmd,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_ldshm_detects_nulls_without_skewing_period() -> None:
|
def test_shm_detects_nulls_without_skewing_period() -> None:
|
||||||
'''
|
'''
|
||||||
Null SHM slots must stop persistence without skewing cadence.
|
Null SHM slots must stop persistence without skewing cadence.
|
||||||
|
|
||||||
A live QQQ buffer contained an interior run of zero-time rows.
|
A live QQQ buffer contained an interior run of zero-time rows.
|
||||||
``ldshm`` included them in period inference and then sent them to
|
``shm`` included them in period inference and then sent them to
|
||||||
NativeDB, which correctly rejected the replacement. Build a
|
NativeDB, which correctly rejected the replacement. Build a
|
||||||
60-second series with the same interior hole and enough
|
60-second series with the same interior hole and enough
|
||||||
bars around the gap. Prove inspection counts every null while
|
bars around the gap. Prove inspection counts every null while
|
||||||
|
|
@ -54,7 +54,7 @@ def test_ldshm_detects_nulls_without_skewing_period() -> None:
|
||||||
assert invalid_count == 2
|
assert invalid_count == 2
|
||||||
|
|
||||||
|
|
||||||
def test_ldshm_period_is_an_observed_step() -> None:
|
def test_shm_period_is_an_observed_step() -> None:
|
||||||
'''
|
'''
|
||||||
Sparse timestamps must not invent a storage timeframe.
|
Sparse timestamps must not invent a storage timeframe.
|
||||||
|
|
||||||
|
|
@ -73,13 +73,13 @@ def test_ldshm_period_is_an_observed_step() -> None:
|
||||||
assert invalid_count == 0
|
assert invalid_count == 0
|
||||||
|
|
||||||
|
|
||||||
def test_ldshm_counts_every_invalid_timestamp() -> None:
|
def test_shm_counts_every_invalid_timestamp() -> None:
|
||||||
'''
|
'''
|
||||||
Corrupt timestamps stop persistence just like null SHM slots.
|
Corrupt timestamps stop persistence just like null SHM slots.
|
||||||
|
|
||||||
Exact zero denotes an unpublished slot, while negative, NaN, and
|
Exact zero denotes an unpublished slot, while negative, NaN, and
|
||||||
infinite values indicate corruption. NativeDB rejects them all,
|
infinite values indicate corruption. NativeDB rejects them all,
|
||||||
but allowing a replacement attempt would still abort ``ldshm``.
|
but allowing a replacement attempt would still abort ``shm``.
|
||||||
Mix each invalid class into a regular series and prove the same
|
Mix each invalid class into a regular series and prove the same
|
||||||
snapshot inspection counts all four while preserving cadence.
|
snapshot inspection counts all four while preserving cadence.
|
||||||
|
|
||||||
|
|
@ -100,7 +100,7 @@ def test_ldshm_counts_every_invalid_timestamp() -> None:
|
||||||
assert invalid_count == 4
|
assert invalid_count == 4
|
||||||
|
|
||||||
|
|
||||||
def test_ldshm_invalid_snapshot_never_reaches_storage(
|
def test_shm_invalid_snapshot_never_reaches_storage(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
|
|
@ -208,23 +208,25 @@ def test_ldshm_invalid_snapshot_never_reaches_storage(
|
||||||
open_annotations,
|
open_annotations,
|
||||||
)
|
)
|
||||||
|
|
||||||
ldshm('qqq.nasdaq.ib')
|
shm_cmd(
|
||||||
|
'qqq.nasdaq.ib',
|
||||||
|
write_parquet=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize('write_parquet', [False, True])
|
def test_shm_write_without_reload_uses_deduped_markup_frame(
|
||||||
def test_ldshm_no_reload_uses_deduped_markup_frame(
|
|
||||||
tmp_path: Path,
|
tmp_path: Path,
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
write_parquet: bool,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
Disabling SHM reload must not require a reload payload.
|
A Parquet write without SHM reload must not need a reload
|
||||||
|
payload.
|
||||||
|
|
||||||
``ldshm`` created ``new`` only in reload but used it for markup.
|
``ldshm`` created ``new`` only in reload but used it for markup.
|
||||||
No-write and no-reload modes crashed, the latter after a durable
|
Its no-reload mode crashed after a durable write. Arrange one
|
||||||
write. Arrange one gap, exercise
|
gap, explicitly enable persistence with reload disabled, and
|
||||||
write modes with reload disabled, and prove markup receives the
|
prove markup receives the deduplicated frame without any SHM
|
||||||
deduplicated frame without any SHM mutation.
|
mutation.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
array = np.zeros(
|
array = np.zeros(
|
||||||
|
|
@ -347,13 +349,13 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
|
||||||
open_annotations,
|
open_annotations,
|
||||||
)
|
)
|
||||||
|
|
||||||
ldshm(
|
shm_cmd(
|
||||||
'x.test',
|
'x.test',
|
||||||
write_parquet=write_parquet,
|
write_parquet=True,
|
||||||
reload_parquet_to_shm=False,
|
reload_parquet_to_shm=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
assert len(writes) == int(write_parquet)
|
assert len(writes) == 1
|
||||||
assert len(markups) == 1
|
assert len(markups) == 1
|
||||||
assert markups[0]['time'].to_list() == [60, 120, 240]
|
assert markups[0]['time'].to_list() == [60, 120, 240]
|
||||||
|
|
||||||
|
|
@ -366,7 +368,7 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
|
||||||
np.array([60, 60]),
|
np.array([60, 60]),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_ldshm_skips_frame_without_positive_cadence(
|
def test_shm_skips_frame_without_positive_cadence(
|
||||||
times: np.ndarray,
|
times: np.ndarray,
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
|
|
@ -376,7 +378,7 @@ def test_ldshm_skips_frame_without_positive_cadence(
|
||||||
undefined for short buffers. Fully unpublished, one-row, and
|
undefined for short buffers. Fully unpublished, one-row, and
|
||||||
duplicate-only frames could therefore fail before the command's
|
duplicate-only frames could therefore fail before the command's
|
||||||
guard. Exercise each shape and prove inspection
|
guard. Exercise each shape and prove inspection
|
||||||
returns no cadence, which directs ``ldshm`` to skip the buffer.
|
returns no cadence, which directs ``shm`` to skip the buffer.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
period_s, _ = _shm_period_and_invalid_count(times)
|
period_s, _ = _shm_period_and_invalid_count(times)
|
||||||
|
|
@ -390,7 +392,7 @@ def test_shm_buffer_discovery_uses_exact_identities(
|
||||||
'''
|
'''
|
||||||
SHM discovery must not mix substring matches or index companions.
|
SHM discovery must not mix substring matches or index companions.
|
||||||
|
|
||||||
``ldshm`` globbed ``*fqme*`` and could process stale and
|
``shm`` globbed ``*fqme*`` and could process stale and
|
||||||
active objects for another instrument whose name merely contained
|
active objects for another instrument whose name merely contained
|
||||||
requested text. Create two generations, both OHLCV kinds, index
|
requested text. Create two generations, both OHLCV kinds, index
|
||||||
companions, and a substring collision. Prove discovery returns
|
companions, and a substring collision. Prove discovery returns
|
||||||
|
|
@ -551,7 +553,7 @@ def test_shm_parser_rejects_forged_identities(
|
||||||
|
|
||||||
Reject malformed service, generation, and FQME components so a
|
Reject malformed service, generation, and FQME components so a
|
||||||
same-user object that merely resembles OHLCV can not become an
|
same-user object that merely resembles OHLCV can not become an
|
||||||
``ldshm --shm-name`` target.
|
``shm --shm-name`` target.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
assert tsp.parse_shm_buffer_path(Path(name)) is None
|
assert tsp.parse_shm_buffer_path(Path(name)) is None
|
||||||
|
|
|
||||||
|
|
@ -41,7 +41,6 @@ def test_store_group_shows_help_without_arguments() -> None:
|
||||||
'anal',
|
'anal',
|
||||||
'audit',
|
'audit',
|
||||||
'delete',
|
'delete',
|
||||||
'ldshm',
|
|
||||||
'ls',
|
'ls',
|
||||||
'series',
|
'series',
|
||||||
'shm',
|
'shm',
|
||||||
|
|
@ -54,8 +53,8 @@ def test_store_commands_show_help_without_arguments() -> None:
|
||||||
Bare endpoints must not open runtimes or report missing args.
|
Bare endpoints must not open runtimes or report missing args.
|
||||||
|
|
||||||
Every endpoint is discoverable by typing its name once. Exercise
|
Every endpoint is discoverable by typing its name once. Exercise
|
||||||
both required-input and explicitly-triggered listing commands and
|
required-input commands and prove help rendering exits before any
|
||||||
prove help rendering exits before any callback can run.
|
callback can run.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
runner = CliRunner()
|
runner = CliRunner()
|
||||||
|
|
@ -63,9 +62,6 @@ def test_store_commands_show_help_without_arguments() -> None:
|
||||||
'anal',
|
'anal',
|
||||||
'audit',
|
'audit',
|
||||||
'delete',
|
'delete',
|
||||||
'ldshm',
|
|
||||||
'ls',
|
|
||||||
'series',
|
|
||||||
'shm',
|
'shm',
|
||||||
):
|
):
|
||||||
result = runner.invoke(store, [command])
|
result = runner.invoke(store, [command])
|
||||||
|
|
@ -90,23 +86,102 @@ def test_series_lists_exact_native_periods(
|
||||||
nativedb.mkdir()
|
nativedb.mkdir()
|
||||||
one = nativedb / 'qqq.nasdaq.ib.ohlcv1s.parquet'
|
one = nativedb / 'qqq.nasdaq.ib.ohlcv1s.parquet'
|
||||||
sixty = nativedb / 'qqq.nasdaq.ib.ohlcv60s.parquet'
|
sixty = nativedb / 'qqq.nasdaq.ib.ohlcv60s.parquet'
|
||||||
|
three_hundred = nativedb / (
|
||||||
|
'qqq.nasdaq.ib.ohlcv300s.parquet'
|
||||||
|
)
|
||||||
one.write_bytes(b'1')
|
one.write_bytes(b'1')
|
||||||
sixty.write_bytes(b'60')
|
sixty.write_bytes(b'60')
|
||||||
|
three_hundred.write_bytes(b'300')
|
||||||
monkeypatch.setattr(config, 'get_conf_dir', lambda: tmp_path)
|
monkeypatch.setattr(config, 'get_conf_dir', lambda: tmp_path)
|
||||||
|
|
||||||
result = CliRunner().invoke(
|
result = CliRunner().invoke(
|
||||||
store,
|
store,
|
||||||
['series', '--all', '--json'],
|
['series', '--json'],
|
||||||
)
|
)
|
||||||
|
|
||||||
assert result.exit_code == 0
|
assert result.exit_code == 0
|
||||||
payload = json.loads(result.output)
|
payload = json.loads(result.output)
|
||||||
assert [item['period_s'] for item in payload] == [1, 60]
|
assert [item['period_s'] for item in payload] == [1, 60, 300]
|
||||||
assert [item['path'] for item in payload] == [
|
assert [item['path'] for item in payload] == [
|
||||||
str(one),
|
str(one),
|
||||||
str(sixty),
|
str(sixty),
|
||||||
|
str(three_hundred),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
compact = CliRunner().invoke(store, ['ls'])
|
||||||
|
assert compact.exit_code == 0
|
||||||
|
assert 'qqq.nasdaq.ib' in compact.output
|
||||||
|
assert '1s, 60s, 300s' in compact.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_shm_reload_requires_explicit_persistence() -> None:
|
||||||
|
'''
|
||||||
|
SHM reload must not silently opt into durable persistence.
|
||||||
|
|
||||||
|
Reload reads back the Parquet produced by the repair path, so the
|
||||||
|
reload flag alone has no valid input and must not open runtime or
|
||||||
|
storage services. Invoke that invalid combination and prove CLI
|
||||||
|
validation rejects it before inspecting an FQME.
|
||||||
|
|
||||||
|
'''
|
||||||
|
result = CliRunner().invoke(
|
||||||
|
store,
|
||||||
|
['shm', 'qqq.nasdaq.ib', '--reload-parquet-to-shm'],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.exit_code == 2
|
||||||
|
assert 'requires --write-parquet' in result.output
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
'option',
|
||||||
|
[
|
||||||
|
['--json'],
|
||||||
|
['--max-gaps', '10'],
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_shm_repair_rejects_read_only_formatting(
|
||||||
|
option: list[str],
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Repair mode must not silently ignore read-only output options.
|
||||||
|
|
||||||
|
Combining the old repair workflow with the new inspector made
|
||||||
|
``--json`` and an explicitly default-valued ``--max-gaps``
|
||||||
|
appeared accepted even though repair produced logs and entered an
|
||||||
|
interactive pause. Invoke each conflicting option and prove
|
||||||
|
validation stops before runtime or storage can be opened.
|
||||||
|
|
||||||
|
'''
|
||||||
|
result = CliRunner().invoke(
|
||||||
|
store,
|
||||||
|
[
|
||||||
|
'shm',
|
||||||
|
'qqq.nasdaq.ib',
|
||||||
|
'--write-parquet',
|
||||||
|
*option,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.exit_code == 2
|
||||||
|
assert 'apply only to read-only output' in result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_has_description_in_store_help() -> None:
|
||||||
|
'''
|
||||||
|
Compact NativeDB discovery must explain itself in group help.
|
||||||
|
|
||||||
|
The backend-oriented ``ls`` callback had no docstring, leaving
|
||||||
|
its command-map description blank. Render the parent help and
|
||||||
|
prove the compact series purpose is visible without opening the
|
||||||
|
command.
|
||||||
|
|
||||||
|
'''
|
||||||
|
result = CliRunner().invoke(store, [])
|
||||||
|
|
||||||
|
assert result.exit_code == 2
|
||||||
|
assert 'List NativeDB FQMEs' in result.output
|
||||||
|
|
||||||
|
|
||||||
def test_shm_endpoint_reports_immutable_snapshot(
|
def test_shm_endpoint_reports_immutable_snapshot(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
@ -202,7 +277,7 @@ def test_shm_endpoint_reports_immutable_snapshot(
|
||||||
assert payload[0]['invalid_count'] == 1
|
assert payload[0]['invalid_count'] == 1
|
||||||
|
|
||||||
|
|
||||||
def test_ldshm_rejects_unknown_exact_name_before_runtime(
|
def test_shm_rejects_unknown_exact_name_before_runtime(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
|
|
@ -240,7 +315,7 @@ def test_ldshm_rejects_unknown_exact_name_before_runtime(
|
||||||
result = CliRunner().invoke(
|
result = CliRunner().invoke(
|
||||||
store,
|
store,
|
||||||
[
|
[
|
||||||
'ldshm',
|
'shm',
|
||||||
'qqq.nasdaq.ib',
|
'qqq.nasdaq.ib',
|
||||||
'--shm-name',
|
'--shm-name',
|
||||||
'missing',
|
'missing',
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue