Simplify `store` series and SHM discovery
Make `store ls` show every NativeDB FQME and its available sample periods by default. Keep `store series` as the detailed view of each exact sub-series, including its period, size and path. Share the same optional FQME filter and numerically ordered payload. Use `store shm` as the canonical volatile-buffer ep and make its default operation a read-only report. Enter the prior persistence workflow only through `--write-parquet`, require that flag before `--reload-parquet-to-shm`. Reject read-only formatting options in repair mode instead of silently ignoring them. Also, - update the qualification docs, debug helper and harness refs; - cover no-flag listings, command help and SHM mode validation; - retain exact generation and object-name selection for stale SHM. Prompt-IO: ai/prompt-io/opencode/20260801T015321Z_74ea2152_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))wkt/backfiller_deep_fixes
parent
74ea2152f9
commit
f0f3c80657
|
|
@ -47,7 +47,7 @@ that's why it's like 1000x faster ya know?"
|
|||
- Wrap all code symbols: `function()`,
|
||||
`ClassName`, `field_name`
|
||||
- File paths: `piker/ui/_remote_ctl.py`
|
||||
- Commands: `git status`, `piker store ldshm`
|
||||
- Commands: `git status`, `piker store shm`
|
||||
|
||||
**Explain like you're pair programming:**
|
||||
```
|
||||
|
|
|
|||
|
|
@ -134,8 +134,8 @@ Require explicit authorization before running:
|
|||
`tests/test_cli.py` is currently hard-skipped. It is not an active CLI
|
||||
regression gate.
|
||||
|
||||
Never execute `piker store anal` or `piker store ldshm` as tests. They are
|
||||
mutating or interactive operational commands.
|
||||
Never execute `piker store anal` or `piker store shm --write-parquet`
|
||||
as tests. They are mutating or interactive operational commands.
|
||||
|
||||
## Project-Specific Flags And Backend Matrix
|
||||
|
||||
|
|
@ -194,7 +194,7 @@ tests/
|
|||
test_feeds.py live Binance/Kraken feeds and shared memory
|
||||
test_ib_history.py deterministic IB history request formatting
|
||||
test_history_backfill.py deterministic history/SHM orchestration
|
||||
test_ldshm.py ldshm unpublished-slot guard
|
||||
test_ldshm.py SHM unpublished-slot guard
|
||||
test_questrade.py obsolete credentialed tests; skipped
|
||||
test_services.py pikerd/datad/feed/EMS actor lifecycle
|
||||
test_store_cli.py storage command help and diagnostics UX
|
||||
|
|
@ -214,7 +214,7 @@ tests/
|
|||
| `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only |
|
||||
| `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network |
|
||||
| `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM |
|
||||
| `piker/storage/cli.py` ldshm null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
|
||||
| `piker/storage/cli.py` SHM null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
|
||||
| `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch |
|
||||
| `piker/accounting/` | targeted accounting node | some cases use live/configured state |
|
||||
| `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,39 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-08-01T01:53:21Z
|
||||
git_ref: 74ea2152
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260801T015321Z_74ea2152_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Make `piker store ls` default to all NativeDB series as a compact counterpart
|
||||
to detailed `series`, with sensible no-flag output and a visible help
|
||||
description. Consolidate SHM inspection and repair under canonical `shm`,
|
||||
because attachment to volatile segments is implied, and make inspection the
|
||||
safe default while retaining explicit persistence and reload controls.
|
||||
|
||||
## Response summary
|
||||
|
||||
Replaced backend-oriented `ls` with compact NativeDB discovery and made both
|
||||
listing commands default to all series. Consolidated SHM behavior under a
|
||||
read-only-by-default `shm` command, guarded explicit repair options, updated
|
||||
executable references and docs, and added command regressions.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/storage/cli.py` - listing defaults and unified SHM command modes.
|
||||
- `tests/test_store_cli.py` - listing, help, and SHM mode regressions.
|
||||
- `tests/test_ldshm.py` - explicit persistence-path regressions.
|
||||
- `docs/manual_backfill_qualification.rst` - revised safe CLI workflow.
|
||||
- `snippets/claude_debug_helper.py` - explicit SHM persistence commands.
|
||||
- `.claude/skills/run-tests/test-harness-reference.md` - safety and test map.
|
||||
- `.claude/skills/piker-slang/examples.md` - canonical command example.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output incorporates the user's design refinements.
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-01T01:53:21Z
|
||||
git_ref: 74ea2152
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user requested command-surface refinements after reviewing the new
|
||||
storage CLI. Bare `ls` should default to all series and act as a compact
|
||||
form of detailed `series`; commands should produce sensible default output
|
||||
without requiring trigger flags; and `ls` needed a parent-help description.
|
||||
The user first proposed folding read-only SHM inspection into `ldshm -r`,
|
||||
then clarified that `shm` is the better canonical command because attaching
|
||||
to volatile segments is implied by the resource name.
|
||||
|
||||
The resulting design uses `piker store ls` as a compact NativeDB FQME and
|
||||
period view, while `piker store series` reports each exact sub-series with
|
||||
size and path details. Both default to all records and accept an optional
|
||||
FQME substring.
|
||||
|
||||
`piker store shm FQME` is read-only by default. Persistence and reload enter
|
||||
the prior repair workflow only through explicit `--write-parquet` and
|
||||
`--reload-parquet-to-shm` flags. Reload requires persistence, and read-only
|
||||
formatting flags are rejected in repair mode instead of being ignored.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
|
||||
|
||||
Generated shared NativeDB payload collection, compact and detailed output,
|
||||
numeric period ordering, default-all semantics, the canonical `shm` command,
|
||||
safe mode dispatch, explicit repair validation, and debugger propagation in
|
||||
read-only mode.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_store_cli.py`
|
||||
|
||||
Generated regressions for no-flag listing, parent-help descriptions, numeric
|
||||
period display, default read-only SHM JSON, reload dependencies, and repair
|
||||
versus read-only option conflicts.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
|
||||
|
||||
Adjusted persistence regressions for the renamed command and explicit repair
|
||||
entry while preserving the historical failure rationale.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- docs/manual_backfill_qualification.rst`
|
||||
|
||||
Documented compact versus detailed NativeDB discovery, default read-only SHM
|
||||
inspection, and the explicitly mutating repair invocation.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- snippets/claude_debug_helper.py`
|
||||
|
||||
Updated executable debug helper commands to select explicit SHM persistence.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||
|
||||
Updated operational safety guidance and SHM regression mappings.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- .claude/skills/piker-slang/examples.md`
|
||||
|
||||
Updated the command example to the canonical `shm` spelling.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
........................................................................ [ 69%]
|
||||
................................ [100%]
|
||||
104 passed in 5.77s
|
||||
```
|
||||
|
||||
Real command-help rendering was checked for the parent group, `ls`, `series`,
|
||||
and `shm`. Real read-only `ls` and `series --json` output was also exercised
|
||||
against the configured NativeDB. `git diff --check` passed.
|
||||
|
||||
Adversarial review identified and drove fixes for lexicographic period
|
||||
ordering, stale runbook safety wording, ignored mode-specific options, and an
|
||||
explicit-default `--max-gaps 10` validation bypass. Final review found no
|
||||
remaining issues. Real OS-backed SHM attachment and successful repair/reload
|
||||
remain integration-level validation concerns.
|
||||
|
|
@ -24,8 +24,8 @@ automatically corruption or an expected venue closure.
|
|||
|
||||
Do not run qualification against normal user storage. The helper below
|
||||
requires a marked disposable root before it will seed, archive, or clear
|
||||
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive
|
||||
and potentially mutating; they are not audit commands.
|
||||
a Parquet. ``piker store anal`` and ``piker store shm --write-parquet``
|
||||
are interactive and potentially mutating; they are not audit commands.
|
||||
|
||||
Keep the disposable root private and do not rename, replace, or symlink
|
||||
files under it while a helper is running. Stop the chart cleanly before
|
||||
|
|
@ -196,11 +196,13 @@ Gap Layer Triage
|
|||
List exact durable series before choosing a timeframe, and inspect live
|
||||
SHM separately::
|
||||
|
||||
piker store ls
|
||||
piker store series FQME_SUBSTRING
|
||||
piker store shm FQME --max-gaps 20
|
||||
|
||||
Both commands are read-only. ``series`` reports each ``(FQME, period)``
|
||||
file independently. ``shm`` reports every exact actor generation and
|
||||
All three forms are read-only. ``ls`` summarizes periods by FQME, while
|
||||
``series`` reports each ``(FQME, period)`` file independently. Read-only
|
||||
``shm`` reports every exact actor generation and
|
||||
``hist``/``rt`` buffer for the FQME, including invalid rows, ordering
|
||||
defects, inferred versus expected cadence, and the largest timestamp gaps.
|
||||
Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer.
|
||||
|
|
|
|||
|
|
@ -51,7 +51,6 @@ from piker import tsp
|
|||
from piker import config
|
||||
from . import log
|
||||
from . import (
|
||||
__tsdbs__,
|
||||
open_storage_client,
|
||||
StorageClient,
|
||||
)
|
||||
|
|
@ -270,104 +269,14 @@ def _render_audit_report(report: dict) -> None:
|
|||
console.print(gap_table)
|
||||
|
||||
|
||||
@store.command(no_args_is_help=True)
|
||||
def ls(
|
||||
backends: Annotated[
|
||||
list[str]|None,
|
||||
typer.Argument(help='Storage backends to query.'),
|
||||
] = None,
|
||||
all_backends: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--all',
|
||||
help='Query every configured storage backend.',
|
||||
),
|
||||
] = False,
|
||||
):
|
||||
from rich.table import Table
|
||||
def _native_series_payload(
|
||||
pattern: str|None,
|
||||
|
||||
if (
|
||||
all_backends
|
||||
and
|
||||
backends
|
||||
):
|
||||
raise typer.BadParameter(
|
||||
'Pass backend names or --all, not both'
|
||||
)
|
||||
if all_backends:
|
||||
backends = list(__tsdbs__)
|
||||
elif not backends:
|
||||
raise typer.BadParameter('Pass backend names or --all')
|
||||
|
||||
console = Console()
|
||||
|
||||
async def query_all():
|
||||
nonlocal backends
|
||||
|
||||
async with (
|
||||
open_piker_runtime(
|
||||
'tsdb_storage',
|
||||
),
|
||||
):
|
||||
for i, backend in enumerate(backends):
|
||||
table = Table()
|
||||
try:
|
||||
async with open_storage_client(backend=backend) as (
|
||||
mod,
|
||||
client,
|
||||
):
|
||||
table.add_column(f'{mod.name}@{client.address}')
|
||||
keys: list[str] = await client.list_keys()
|
||||
for key in keys:
|
||||
table.add_row(key)
|
||||
|
||||
console.print(table)
|
||||
except Exception:
|
||||
log.error(f'Unable to connect to storage engine: `{backend}`')
|
||||
|
||||
trio.run(query_all)
|
||||
|
||||
|
||||
@store.command(no_args_is_help=True)
|
||||
def series(
|
||||
pattern: Annotated[
|
||||
str|None,
|
||||
typer.Argument(
|
||||
help='Optional case-insensitive FQME substring.',
|
||||
),
|
||||
] = None,
|
||||
all_series: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--all',
|
||||
help='List every canonical NativeDB series.',
|
||||
),
|
||||
] = False,
|
||||
json_output: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--json',
|
||||
help='Emit machine-readable JSON.',
|
||||
),
|
||||
] = False,
|
||||
) -> None:
|
||||
) -> tuple[Path, list[dict]]:
|
||||
'''
|
||||
List exact NativeDB series identities without opening a runtime.
|
||||
Collect exact NativeDB series metadata for CLI presentation.
|
||||
|
||||
'''
|
||||
if (
|
||||
all_series
|
||||
and
|
||||
pattern is not None
|
||||
):
|
||||
raise typer.BadParameter('Pass a pattern or --all, not both')
|
||||
if (
|
||||
not all_series
|
||||
and
|
||||
pattern is None
|
||||
):
|
||||
raise typer.BadParameter('Pass a pattern or --all')
|
||||
|
||||
datadir: Path = config.get_conf_dir() / 'nativedb'
|
||||
refs: list[NativeSeriesRef] = list(iter_native_series(datadir))
|
||||
if pattern is not None:
|
||||
|
|
@ -390,6 +299,96 @@ def series(
|
|||
'path': str(ref.path),
|
||||
'size_bytes': size_bytes,
|
||||
})
|
||||
payload.sort(key=lambda item: (
|
||||
item['fqme'],
|
||||
item['period_s'],
|
||||
))
|
||||
|
||||
return datadir, payload
|
||||
|
||||
|
||||
@store.command()
|
||||
def ls(
|
||||
pattern: Annotated[
|
||||
str|None,
|
||||
typer.Argument(
|
||||
help='Optional case-insensitive FQME substring.',
|
||||
),
|
||||
] = None,
|
||||
all_series: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--all',
|
||||
help='List all FQMEs (the default).',
|
||||
),
|
||||
] = False,
|
||||
) -> None:
|
||||
'''
|
||||
List NativeDB FQMEs and their available sample periods.
|
||||
|
||||
'''
|
||||
if (
|
||||
all_series
|
||||
and
|
||||
pattern is not None
|
||||
):
|
||||
raise typer.BadParameter('Pass a pattern or --all, not both')
|
||||
|
||||
datadir, payload = _native_series_payload(pattern)
|
||||
periods_by_fqme: dict[str, list[int]] = {}
|
||||
for item in payload:
|
||||
periods_by_fqme.setdefault(item['fqme'], []).append(
|
||||
item['period_s']
|
||||
)
|
||||
|
||||
table = Table(title=f'NativeDB series @ {datadir}')
|
||||
table.add_column('FQME')
|
||||
table.add_column('Periods')
|
||||
for fqme, periods in periods_by_fqme.items():
|
||||
table.add_row(
|
||||
fqme,
|
||||
', '.join(
|
||||
f'{period}s'
|
||||
for period in sorted(periods)
|
||||
),
|
||||
)
|
||||
Console().print(table)
|
||||
|
||||
|
||||
@store.command()
|
||||
def series(
|
||||
pattern: Annotated[
|
||||
str|None,
|
||||
typer.Argument(
|
||||
help='Optional case-insensitive FQME substring.',
|
||||
),
|
||||
] = None,
|
||||
all_series: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--all',
|
||||
help='List all series (the default).',
|
||||
),
|
||||
] = False,
|
||||
json_output: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--json',
|
||||
help='Emit machine-readable JSON.',
|
||||
),
|
||||
] = False,
|
||||
) -> None:
|
||||
'''
|
||||
List detailed NativeDB sub-series without opening a runtime.
|
||||
|
||||
'''
|
||||
if (
|
||||
all_series
|
||||
and
|
||||
pattern is not None
|
||||
):
|
||||
raise typer.BadParameter('Pass a pattern or --all, not both')
|
||||
datadir, payload = _native_series_payload(pattern)
|
||||
if json_output:
|
||||
typer.echo(json.dumps(payload, indent=2, sort_keys=True))
|
||||
return
|
||||
|
|
@ -712,8 +711,7 @@ def anal(
|
|||
trio.run(main)
|
||||
|
||||
|
||||
@store.command('shm', no_args_is_help=True)
|
||||
def inspect_shm(
|
||||
def _inspect_shm(
|
||||
fqme: str,
|
||||
shm_name: Annotated[
|
||||
str|None,
|
||||
|
|
@ -737,6 +735,7 @@ def inspect_shm(
|
|||
help='Emit machine-readable JSON.',
|
||||
),
|
||||
] = False,
|
||||
pdb: bool = False,
|
||||
) -> None:
|
||||
'''
|
||||
Inspect matching OHLCV SHM buffers without mutating them.
|
||||
|
|
@ -765,7 +764,10 @@ def inspect_shm(
|
|||
reports: list[dict] = []
|
||||
|
||||
async def main() -> None:
|
||||
async with open_piker_runtime('shm_inspector'):
|
||||
async with open_piker_runtime(
|
||||
'shm_inspector',
|
||||
debug_mode=pdb,
|
||||
):
|
||||
for (
|
||||
shmfile,
|
||||
shm,
|
||||
|
|
@ -840,26 +842,88 @@ def inspect_shm(
|
|||
|
||||
|
||||
@store.command(no_args_is_help=True)
|
||||
def ldshm(
|
||||
def shm(
|
||||
fqme: str,
|
||||
write_parquet: bool = True,
|
||||
reload_parquet_to_shm: bool = True,
|
||||
write_parquet: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--write-parquet',
|
||||
help='Persist repaired frames to NativeDB.',
|
||||
),
|
||||
] = False,
|
||||
reload_parquet_to_shm: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--reload-parquet-to-shm',
|
||||
help='Reload persisted repairs into SHM.',
|
||||
),
|
||||
] = False,
|
||||
pdb: bool = False, # --pdb passed?
|
||||
max_gaps: Annotated[
|
||||
int|None,
|
||||
typer.Option(
|
||||
'--max-gaps',
|
||||
min=0,
|
||||
help=(
|
||||
'Maximum largest gaps in read-only output '
|
||||
'(default: 10).'
|
||||
),
|
||||
show_default=False,
|
||||
),
|
||||
] = None,
|
||||
json_output: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
'--json',
|
||||
help='Emit read-only output as JSON.',
|
||||
),
|
||||
] = False,
|
||||
shm_name: Annotated[
|
||||
str|None,
|
||||
typer.Option(
|
||||
'--shm-name',
|
||||
help='Process only this exact SHM object name.',
|
||||
help='Select only this exact SHM object name.',
|
||||
),
|
||||
] = None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Linux ONLY: load any fqme file name matching shm buffer from
|
||||
/dev/shm/ into an OHLCV numpy array and polars DataFrame,
|
||||
optionally write to offline storage via `.parquet` file.
|
||||
Inspect volatile OHLCV SHM, with explicit repair options.
|
||||
|
||||
'''
|
||||
if (
|
||||
reload_parquet_to_shm
|
||||
and
|
||||
not write_parquet
|
||||
):
|
||||
raise typer.BadParameter(
|
||||
'--reload-parquet-to-shm requires --write-parquet'
|
||||
)
|
||||
if (
|
||||
write_parquet
|
||||
and (
|
||||
json_output
|
||||
or
|
||||
max_gaps is not None
|
||||
)
|
||||
):
|
||||
raise typer.BadParameter(
|
||||
'--json and --max-gaps apply only to read-only output'
|
||||
)
|
||||
if not write_parquet:
|
||||
_inspect_shm(
|
||||
fqme,
|
||||
shm_name=shm_name,
|
||||
max_gaps=(
|
||||
10
|
||||
if max_gaps is None
|
||||
else max_gaps
|
||||
),
|
||||
json_output=json_output,
|
||||
pdb=pdb,
|
||||
)
|
||||
return
|
||||
|
||||
if (
|
||||
shm_name is not None
|
||||
and
|
||||
|
|
|
|||
|
|
@ -59,7 +59,10 @@ def expect(
|
|||
|
||||
def run_pdb_commands(
|
||||
commands: list[str],
|
||||
initial_cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
||||
initial_cmd: str = (
|
||||
'piker store shm xmrusdt.usdtm.perp.binance '
|
||||
'--write-parquet'
|
||||
),
|
||||
timeout: int = 30,
|
||||
print_output: bool = True,
|
||||
) -> dict[str, str]:
|
||||
|
|
@ -145,7 +148,10 @@ class InteractivePdbSession:
|
|||
'''
|
||||
def __init__(
|
||||
self,
|
||||
cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance',
|
||||
cmd: str = (
|
||||
'piker store shm xmrusdt.usdtm.perp.binance '
|
||||
'--write-parquet'
|
||||
),
|
||||
timeout: int = 30,
|
||||
):
|
||||
self.cmd: str = cmd
|
||||
|
|
|
|||
|
|
@ -21,16 +21,16 @@ from piker.storage import cli as storage_cli
|
|||
from piker.storage.cli import (
|
||||
_shm_period_and_invalid_count,
|
||||
_summarize_shm_frame,
|
||||
ldshm,
|
||||
shm as shm_cmd,
|
||||
)
|
||||
|
||||
|
||||
def test_ldshm_detects_nulls_without_skewing_period() -> None:
|
||||
def test_shm_detects_nulls_without_skewing_period() -> None:
|
||||
'''
|
||||
Null SHM slots must stop persistence without skewing cadence.
|
||||
|
||||
A live QQQ buffer contained an interior run of zero-time rows.
|
||||
``ldshm`` included them in period inference and then sent them to
|
||||
``shm`` included them in period inference and then sent them to
|
||||
NativeDB, which correctly rejected the replacement. Build a
|
||||
60-second series with the same interior hole and enough
|
||||
bars around the gap. Prove inspection counts every null while
|
||||
|
|
@ -54,7 +54,7 @@ def test_ldshm_detects_nulls_without_skewing_period() -> None:
|
|||
assert invalid_count == 2
|
||||
|
||||
|
||||
def test_ldshm_period_is_an_observed_step() -> None:
|
||||
def test_shm_period_is_an_observed_step() -> None:
|
||||
'''
|
||||
Sparse timestamps must not invent a storage timeframe.
|
||||
|
||||
|
|
@ -73,13 +73,13 @@ def test_ldshm_period_is_an_observed_step() -> None:
|
|||
assert invalid_count == 0
|
||||
|
||||
|
||||
def test_ldshm_counts_every_invalid_timestamp() -> None:
|
||||
def test_shm_counts_every_invalid_timestamp() -> None:
|
||||
'''
|
||||
Corrupt timestamps stop persistence just like null SHM slots.
|
||||
|
||||
Exact zero denotes an unpublished slot, while negative, NaN, and
|
||||
infinite values indicate corruption. NativeDB rejects them all,
|
||||
but allowing a replacement attempt would still abort ``ldshm``.
|
||||
but allowing a replacement attempt would still abort ``shm``.
|
||||
Mix each invalid class into a regular series and prove the same
|
||||
snapshot inspection counts all four while preserving cadence.
|
||||
|
||||
|
|
@ -100,7 +100,7 @@ def test_ldshm_counts_every_invalid_timestamp() -> None:
|
|||
assert invalid_count == 4
|
||||
|
||||
|
||||
def test_ldshm_invalid_snapshot_never_reaches_storage(
|
||||
def test_shm_invalid_snapshot_never_reaches_storage(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -208,23 +208,25 @@ def test_ldshm_invalid_snapshot_never_reaches_storage(
|
|||
open_annotations,
|
||||
)
|
||||
|
||||
ldshm('qqq.nasdaq.ib')
|
||||
shm_cmd(
|
||||
'qqq.nasdaq.ib',
|
||||
write_parquet=True,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('write_parquet', [False, True])
|
||||
def test_ldshm_no_reload_uses_deduped_markup_frame(
|
||||
def test_shm_write_without_reload_uses_deduped_markup_frame(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
write_parquet: bool,
|
||||
) -> None:
|
||||
'''
|
||||
Disabling SHM reload must not require a reload payload.
|
||||
A Parquet write without SHM reload must not need a reload
|
||||
payload.
|
||||
|
||||
``ldshm`` created ``new`` only in reload but used it for markup.
|
||||
No-write and no-reload modes crashed, the latter after a durable
|
||||
write. Arrange one gap, exercise
|
||||
write modes with reload disabled, and prove markup receives the
|
||||
deduplicated frame without any SHM mutation.
|
||||
Its no-reload mode crashed after a durable write. Arrange one
|
||||
gap, explicitly enable persistence with reload disabled, and
|
||||
prove markup receives the deduplicated frame without any SHM
|
||||
mutation.
|
||||
|
||||
'''
|
||||
array = np.zeros(
|
||||
|
|
@ -347,13 +349,13 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
|
|||
open_annotations,
|
||||
)
|
||||
|
||||
ldshm(
|
||||
shm_cmd(
|
||||
'x.test',
|
||||
write_parquet=write_parquet,
|
||||
write_parquet=True,
|
||||
reload_parquet_to_shm=False,
|
||||
)
|
||||
|
||||
assert len(writes) == int(write_parquet)
|
||||
assert len(writes) == 1
|
||||
assert len(markups) == 1
|
||||
assert markups[0]['time'].to_list() == [60, 120, 240]
|
||||
|
||||
|
|
@ -366,7 +368,7 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
|
|||
np.array([60, 60]),
|
||||
],
|
||||
)
|
||||
def test_ldshm_skips_frame_without_positive_cadence(
|
||||
def test_shm_skips_frame_without_positive_cadence(
|
||||
times: np.ndarray,
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -376,7 +378,7 @@ def test_ldshm_skips_frame_without_positive_cadence(
|
|||
undefined for short buffers. Fully unpublished, one-row, and
|
||||
duplicate-only frames could therefore fail before the command's
|
||||
guard. Exercise each shape and prove inspection
|
||||
returns no cadence, which directs ``ldshm`` to skip the buffer.
|
||||
returns no cadence, which directs ``shm`` to skip the buffer.
|
||||
|
||||
'''
|
||||
period_s, _ = _shm_period_and_invalid_count(times)
|
||||
|
|
@ -390,7 +392,7 @@ def test_shm_buffer_discovery_uses_exact_identities(
|
|||
'''
|
||||
SHM discovery must not mix substring matches or index companions.
|
||||
|
||||
``ldshm`` globbed ``*fqme*`` and could process stale and
|
||||
``shm`` globbed ``*fqme*`` and could process stale and
|
||||
active objects for another instrument whose name merely contained
|
||||
requested text. Create two generations, both OHLCV kinds, index
|
||||
companions, and a substring collision. Prove discovery returns
|
||||
|
|
@ -551,7 +553,7 @@ def test_shm_parser_rejects_forged_identities(
|
|||
|
||||
Reject malformed service, generation, and FQME components so a
|
||||
same-user object that merely resembles OHLCV can not become an
|
||||
``ldshm --shm-name`` target.
|
||||
``shm --shm-name`` target.
|
||||
|
||||
'''
|
||||
assert tsp.parse_shm_buffer_path(Path(name)) is None
|
||||
|
|
|
|||
|
|
@ -41,7 +41,6 @@ def test_store_group_shows_help_without_arguments() -> None:
|
|||
'anal',
|
||||
'audit',
|
||||
'delete',
|
||||
'ldshm',
|
||||
'ls',
|
||||
'series',
|
||||
'shm',
|
||||
|
|
@ -54,8 +53,8 @@ def test_store_commands_show_help_without_arguments() -> None:
|
|||
Bare endpoints must not open runtimes or report missing args.
|
||||
|
||||
Every endpoint is discoverable by typing its name once. Exercise
|
||||
both required-input and explicitly-triggered listing commands and
|
||||
prove help rendering exits before any callback can run.
|
||||
required-input commands and prove help rendering exits before any
|
||||
callback can run.
|
||||
|
||||
'''
|
||||
runner = CliRunner()
|
||||
|
|
@ -63,9 +62,6 @@ def test_store_commands_show_help_without_arguments() -> None:
|
|||
'anal',
|
||||
'audit',
|
||||
'delete',
|
||||
'ldshm',
|
||||
'ls',
|
||||
'series',
|
||||
'shm',
|
||||
):
|
||||
result = runner.invoke(store, [command])
|
||||
|
|
@ -90,23 +86,102 @@ def test_series_lists_exact_native_periods(
|
|||
nativedb.mkdir()
|
||||
one = nativedb / 'qqq.nasdaq.ib.ohlcv1s.parquet'
|
||||
sixty = nativedb / 'qqq.nasdaq.ib.ohlcv60s.parquet'
|
||||
three_hundred = nativedb / (
|
||||
'qqq.nasdaq.ib.ohlcv300s.parquet'
|
||||
)
|
||||
one.write_bytes(b'1')
|
||||
sixty.write_bytes(b'60')
|
||||
three_hundred.write_bytes(b'300')
|
||||
monkeypatch.setattr(config, 'get_conf_dir', lambda: tmp_path)
|
||||
|
||||
result = CliRunner().invoke(
|
||||
store,
|
||||
['series', '--all', '--json'],
|
||||
['series', '--json'],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0
|
||||
payload = json.loads(result.output)
|
||||
assert [item['period_s'] for item in payload] == [1, 60]
|
||||
assert [item['period_s'] for item in payload] == [1, 60, 300]
|
||||
assert [item['path'] for item in payload] == [
|
||||
str(one),
|
||||
str(sixty),
|
||||
str(three_hundred),
|
||||
]
|
||||
|
||||
compact = CliRunner().invoke(store, ['ls'])
|
||||
assert compact.exit_code == 0
|
||||
assert 'qqq.nasdaq.ib' in compact.output
|
||||
assert '1s, 60s, 300s' in compact.output
|
||||
|
||||
|
||||
def test_shm_reload_requires_explicit_persistence() -> None:
|
||||
'''
|
||||
SHM reload must not silently opt into durable persistence.
|
||||
|
||||
Reload reads back the Parquet produced by the repair path, so the
|
||||
reload flag alone has no valid input and must not open runtime or
|
||||
storage services. Invoke that invalid combination and prove CLI
|
||||
validation rejects it before inspecting an FQME.
|
||||
|
||||
'''
|
||||
result = CliRunner().invoke(
|
||||
store,
|
||||
['shm', 'qqq.nasdaq.ib', '--reload-parquet-to-shm'],
|
||||
)
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert 'requires --write-parquet' in result.output
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'option',
|
||||
[
|
||||
['--json'],
|
||||
['--max-gaps', '10'],
|
||||
],
|
||||
)
|
||||
def test_shm_repair_rejects_read_only_formatting(
|
||||
option: list[str],
|
||||
) -> None:
|
||||
'''
|
||||
Repair mode must not silently ignore read-only output options.
|
||||
|
||||
Combining the old repair workflow with the new inspector made
|
||||
``--json`` and an explicitly default-valued ``--max-gaps``
|
||||
appeared accepted even though repair produced logs and entered an
|
||||
interactive pause. Invoke each conflicting option and prove
|
||||
validation stops before runtime or storage can be opened.
|
||||
|
||||
'''
|
||||
result = CliRunner().invoke(
|
||||
store,
|
||||
[
|
||||
'shm',
|
||||
'qqq.nasdaq.ib',
|
||||
'--write-parquet',
|
||||
*option,
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert 'apply only to read-only output' in result.output
|
||||
|
||||
|
||||
def test_ls_has_description_in_store_help() -> None:
|
||||
'''
|
||||
Compact NativeDB discovery must explain itself in group help.
|
||||
|
||||
The backend-oriented ``ls`` callback had no docstring, leaving
|
||||
its command-map description blank. Render the parent help and
|
||||
prove the compact series purpose is visible without opening the
|
||||
command.
|
||||
|
||||
'''
|
||||
result = CliRunner().invoke(store, [])
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert 'List NativeDB FQMEs' in result.output
|
||||
|
||||
|
||||
def test_shm_endpoint_reports_immutable_snapshot(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
|
|
@ -202,7 +277,7 @@ def test_shm_endpoint_reports_immutable_snapshot(
|
|||
assert payload[0]['invalid_count'] == 1
|
||||
|
||||
|
||||
def test_ldshm_rejects_unknown_exact_name_before_runtime(
|
||||
def test_shm_rejects_unknown_exact_name_before_runtime(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
'''
|
||||
|
|
@ -240,7 +315,7 @@ def test_ldshm_rejects_unknown_exact_name_before_runtime(
|
|||
result = CliRunner().invoke(
|
||||
store,
|
||||
[
|
||||
'ldshm',
|
||||
'shm',
|
||||
'qqq.nasdaq.ib',
|
||||
'--shm-name',
|
||||
'missing',
|
||||
|
|
|
|||
Loading…
Reference in New Issue