Simplify `store` series and SHM discovery

Make `store ls` show every NativeDB FQME and its available sample
periods by default. Keep `store series` as the detailed view of
each exact sub-series, including its period, size and path. Share
the same optional FQME filter and numerically ordered payload.

Use `store shm` as the canonical volatile-buffer ep and make its
default operation a read-only report. Enter the prior persistence
workflow only through `--write-parquet`, require that flag before
`--reload-parquet-to-shm`. Reject read-only formatting options in
repair mode instead of silently ignoring them.

Also,
- update the qualification docs, debug helper and harness refs;
- cover no-flag listings, command help and SHM mode validation;
- retain exact generation and object-name selection for stale SHM.

Prompt-IO: ai/prompt-io/opencode/20260801T015321Z_74ea2152_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
wkt/backfiller_deep_fixes
Gud Boi 2026-08-01 00:34:55 -04:00
parent 74ea2152f9
commit f0f3c80657
9 changed files with 415 additions and 149 deletions

View File

@ -47,7 +47,7 @@ that's why it's like 1000x faster ya know?"
- Wrap all code symbols: `function()`, - Wrap all code symbols: `function()`,
`ClassName`, `field_name` `ClassName`, `field_name`
- File paths: `piker/ui/_remote_ctl.py` - File paths: `piker/ui/_remote_ctl.py`
- Commands: `git status`, `piker store ldshm` - Commands: `git status`, `piker store shm`
**Explain like you're pair programming:** **Explain like you're pair programming:**
``` ```

View File

@ -134,8 +134,8 @@ Require explicit authorization before running:
`tests/test_cli.py` is currently hard-skipped. It is not an active CLI `tests/test_cli.py` is currently hard-skipped. It is not an active CLI
regression gate. regression gate.
Never execute `piker store anal` or `piker store ldshm` as tests. They are Never execute `piker store anal` or `piker store shm --write-parquet`
mutating or interactive operational commands. as tests. They are mutating or interactive operational commands.
## Project-Specific Flags And Backend Matrix ## Project-Specific Flags And Backend Matrix
@ -194,7 +194,7 @@ tests/
test_feeds.py live Binance/Kraken feeds and shared memory test_feeds.py live Binance/Kraken feeds and shared memory
test_ib_history.py deterministic IB history request formatting test_ib_history.py deterministic IB history request formatting
test_history_backfill.py deterministic history/SHM orchestration test_history_backfill.py deterministic history/SHM orchestration
test_ldshm.py ldshm unpublished-slot guard test_ldshm.py SHM unpublished-slot guard
test_questrade.py obsolete credentialed tests; skipped test_questrade.py obsolete credentialed tests; skipped
test_services.py pikerd/datad/feed/EMS actor lifecycle test_services.py pikerd/datad/feed/EMS actor lifecycle
test_store_cli.py storage command help and diagnostics UX test_store_cli.py storage command help and diagnostics UX
@ -214,7 +214,7 @@ tests/
| `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only | | `snippets/nativedb_backfill_audit.xsh` | `tests/test_backfill_audit_snippet.py` | disposable paths only |
| `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network | | `piker/brokers/ib/api.py`, `feed.py` history | `tests/test_ib_history.py` | fake client, no network |
| `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM | | `piker/tsp/_history.py` | `tests/test_history_backfill.py` | fake provider/storage/SHM |
| `piker/storage/cli.py` ldshm null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation | | `piker/storage/cli.py` SHM null-slot guard | `tests/test_ldshm.py` | synthetic timestamps, no SHM mutation |
| `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch | | `piker/config.py` | `test_account_file_default_empty` | root-network test has a known mismatch |
| `piker/accounting/` | targeted accounting node | some cases use live/configured state | | `piker/accounting/` | targeted accounting node | some cases use live/configured state |
| `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in | | `piker/ui/_style.py`, `piker/ui/qt.py` | `tests/test_dpi_font.py` | GUI/config-isolated opt-in |

View File

@ -0,0 +1,39 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-08-01T01:53:21Z
git_ref: 74ea2152
scope: code
substantive: true
raw_file: 20260801T015321Z_74ea2152_prompt_io.raw.md
---
## Prompt
Make `piker store ls` default to all NativeDB series as a compact counterpart
to detailed `series`, with sensible no-flag output and a visible help
description. Consolidate SHM inspection and repair under canonical `shm`,
because attachment to volatile segments is implied, and make inspection the
safe default while retaining explicit persistence and reload controls.
## Response summary
Replaced backend-oriented `ls` with compact NativeDB discovery and made both
listing commands default to all series. Consolidated SHM behavior under a
read-only-by-default `shm` command, guarded explicit repair options, updated
executable references and docs, and added command regressions.
## Files changed
- `piker/storage/cli.py` - listing defaults and unified SHM command modes.
- `tests/test_store_cli.py` - listing, help, and SHM mode regressions.
- `tests/test_ldshm.py` - explicit persistence-path regressions.
- `docs/manual_backfill_qualification.rst` - revised safe CLI workflow.
- `snippets/claude_debug_helper.py` - explicit SHM persistence commands.
- `.claude/skills/run-tests/test-harness-reference.md` - safety and test map.
- `.claude/skills/piker-slang/examples.md` - canonical command example.
## Human edits
None - generated output incorporates the user's design refinements.

View File

@ -0,0 +1,78 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-01T01:53:21Z
git_ref: 74ea2152
diff_cmd: git diff HEAD~1..HEAD
---
The user requested command-surface refinements after reviewing the new
storage CLI. Bare `ls` should default to all series and act as a compact
form of detailed `series`; commands should produce sensible default output
without requiring trigger flags; and `ls` needed a parent-help description.
The user first proposed folding read-only SHM inspection into `ldshm -r`,
then clarified that `shm` is the better canonical command because attaching
to volatile segments is implied by the resource name.
The resulting design uses `piker store ls` as a compact NativeDB FQME and
period view, while `piker store series` reports each exact sub-series with
size and path details. Both default to all records and accept an optional
FQME substring.
`piker store shm FQME` is read-only by default. Persistence and reload enter
the prior repair workflow only through explicit `--write-parquet` and
`--reload-parquet-to-shm` flags. Reload requires persistence, and read-only
formatting flags are rejected in repair mode instead of being ignored.
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
Generated shared NativeDB payload collection, compact and detailed output,
numeric period ordering, default-all semantics, the canonical `shm` command,
safe mode dispatch, explicit repair validation, and debugger propagation in
read-only mode.
> `git diff HEAD~1..HEAD -- tests/test_store_cli.py`
Generated regressions for no-flag listing, parent-help descriptions, numeric
period display, default read-only SHM JSON, reload dependencies, and repair
versus read-only option conflicts.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Adjusted persistence regressions for the renamed command and explicit repair
entry while preserving the historical failure rationale.
> `git diff HEAD~1..HEAD -- docs/manual_backfill_qualification.rst`
Documented compact versus detailed NativeDB discovery, default read-only SHM
inspection, and the explicitly mutating repair invocation.
> `git diff HEAD~1..HEAD -- snippets/claude_debug_helper.py`
Updated executable debug helper commands to select explicit SHM persistence.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Updated operational safety guidance and SHM regression mappings.
> `git diff HEAD~1..HEAD -- .claude/skills/piker-slang/examples.md`
Updated the command example to the canonical `shm` spelling.
Verification output:
```text
........................................................................ [ 69%]
................................ [100%]
104 passed in 5.77s
```
Real command-help rendering was checked for the parent group, `ls`, `series`,
and `shm`. Real read-only `ls` and `series --json` output was also exercised
against the configured NativeDB. `git diff --check` passed.
Adversarial review identified and drove fixes for lexicographic period
ordering, stale runbook safety wording, ignored mode-specific options, and an
explicit-default `--max-gaps 10` validation bypass. Final review found no
remaining issues. Real OS-backed SHM attachment and successful repair/reload
remain integration-level validation concerns.

View File

@ -24,8 +24,8 @@ automatically corruption or an expected venue closure.
Do not run qualification against normal user storage. The helper below Do not run qualification against normal user storage. The helper below
requires a marked disposable root before it will seed, archive, or clear requires a marked disposable root before it will seed, archive, or clear
a Parquet. ``piker store anal`` and ``piker store ldshm`` are interactive a Parquet. ``piker store anal`` and ``piker store shm --write-parquet``
and potentially mutating; they are not audit commands. are interactive and potentially mutating; they are not audit commands.
Keep the disposable root private and do not rename, replace, or symlink Keep the disposable root private and do not rename, replace, or symlink
files under it while a helper is running. Stop the chart cleanly before files under it while a helper is running. Stop the chart cleanly before
@ -196,11 +196,13 @@ Gap Layer Triage
List exact durable series before choosing a timeframe, and inspect live List exact durable series before choosing a timeframe, and inspect live
SHM separately:: SHM separately::
piker store ls
piker store series FQME_SUBSTRING piker store series FQME_SUBSTRING
piker store shm FQME --max-gaps 20 piker store shm FQME --max-gaps 20
Both commands are read-only. ``series`` reports each ``(FQME, period)`` All three forms are read-only. ``ls`` summarizes periods by FQME, while
file independently. ``shm`` reports every exact actor generation and ``series`` reports each ``(FQME, period)`` file independently. Read-only
``shm`` reports every exact actor generation and
``hist``/``rt`` buffer for the FQME, including invalid rows, ordering ``hist``/``rt`` buffer for the FQME, including invalid rows, ordering
defects, inferred versus expected cadence, and the largest timestamp gaps. defects, inferred versus expected cadence, and the largest timestamp gaps.
Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer. Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer.

View File

@ -51,7 +51,6 @@ from piker import tsp
from piker import config from piker import config
from . import log from . import log
from . import ( from . import (
__tsdbs__,
open_storage_client, open_storage_client,
StorageClient, StorageClient,
) )
@ -270,104 +269,14 @@ def _render_audit_report(report: dict) -> None:
console.print(gap_table) console.print(gap_table)
@store.command(no_args_is_help=True) def _native_series_payload(
def ls( pattern: str|None,
backends: Annotated[
list[str]|None,
typer.Argument(help='Storage backends to query.'),
] = None,
all_backends: Annotated[
bool,
typer.Option(
'--all',
help='Query every configured storage backend.',
),
] = False,
):
from rich.table import Table
if ( ) -> tuple[Path, list[dict]]:
all_backends
and
backends
):
raise typer.BadParameter(
'Pass backend names or --all, not both'
)
if all_backends:
backends = list(__tsdbs__)
elif not backends:
raise typer.BadParameter('Pass backend names or --all')
console = Console()
async def query_all():
nonlocal backends
async with (
open_piker_runtime(
'tsdb_storage',
),
):
for i, backend in enumerate(backends):
table = Table()
try:
async with open_storage_client(backend=backend) as (
mod,
client,
):
table.add_column(f'{mod.name}@{client.address}')
keys: list[str] = await client.list_keys()
for key in keys:
table.add_row(key)
console.print(table)
except Exception:
log.error(f'Unable to connect to storage engine: `{backend}`')
trio.run(query_all)
@store.command(no_args_is_help=True)
def series(
pattern: Annotated[
str|None,
typer.Argument(
help='Optional case-insensitive FQME substring.',
),
] = None,
all_series: Annotated[
bool,
typer.Option(
'--all',
help='List every canonical NativeDB series.',
),
] = False,
json_output: Annotated[
bool,
typer.Option(
'--json',
help='Emit machine-readable JSON.',
),
] = False,
) -> None:
''' '''
List exact NativeDB series identities without opening a runtime. Collect exact NativeDB series metadata for CLI presentation.
''' '''
if (
all_series
and
pattern is not None
):
raise typer.BadParameter('Pass a pattern or --all, not both')
if (
not all_series
and
pattern is None
):
raise typer.BadParameter('Pass a pattern or --all')
datadir: Path = config.get_conf_dir() / 'nativedb' datadir: Path = config.get_conf_dir() / 'nativedb'
refs: list[NativeSeriesRef] = list(iter_native_series(datadir)) refs: list[NativeSeriesRef] = list(iter_native_series(datadir))
if pattern is not None: if pattern is not None:
@ -390,6 +299,96 @@ def series(
'path': str(ref.path), 'path': str(ref.path),
'size_bytes': size_bytes, 'size_bytes': size_bytes,
}) })
payload.sort(key=lambda item: (
item['fqme'],
item['period_s'],
))
return datadir, payload
@store.command()
def ls(
pattern: Annotated[
str|None,
typer.Argument(
help='Optional case-insensitive FQME substring.',
),
] = None,
all_series: Annotated[
bool,
typer.Option(
'--all',
help='List all FQMEs (the default).',
),
] = False,
) -> None:
'''
List NativeDB FQMEs and their available sample periods.
'''
if (
all_series
and
pattern is not None
):
raise typer.BadParameter('Pass a pattern or --all, not both')
datadir, payload = _native_series_payload(pattern)
periods_by_fqme: dict[str, list[int]] = {}
for item in payload:
periods_by_fqme.setdefault(item['fqme'], []).append(
item['period_s']
)
table = Table(title=f'NativeDB series @ {datadir}')
table.add_column('FQME')
table.add_column('Periods')
for fqme, periods in periods_by_fqme.items():
table.add_row(
fqme,
', '.join(
f'{period}s'
for period in sorted(periods)
),
)
Console().print(table)
@store.command()
def series(
pattern: Annotated[
str|None,
typer.Argument(
help='Optional case-insensitive FQME substring.',
),
] = None,
all_series: Annotated[
bool,
typer.Option(
'--all',
help='List all series (the default).',
),
] = False,
json_output: Annotated[
bool,
typer.Option(
'--json',
help='Emit machine-readable JSON.',
),
] = False,
) -> None:
'''
List detailed NativeDB sub-series without opening a runtime.
'''
if (
all_series
and
pattern is not None
):
raise typer.BadParameter('Pass a pattern or --all, not both')
datadir, payload = _native_series_payload(pattern)
if json_output: if json_output:
typer.echo(json.dumps(payload, indent=2, sort_keys=True)) typer.echo(json.dumps(payload, indent=2, sort_keys=True))
return return
@ -712,8 +711,7 @@ def anal(
trio.run(main) trio.run(main)
@store.command('shm', no_args_is_help=True) def _inspect_shm(
def inspect_shm(
fqme: str, fqme: str,
shm_name: Annotated[ shm_name: Annotated[
str|None, str|None,
@ -737,6 +735,7 @@ def inspect_shm(
help='Emit machine-readable JSON.', help='Emit machine-readable JSON.',
), ),
] = False, ] = False,
pdb: bool = False,
) -> None: ) -> None:
''' '''
Inspect matching OHLCV SHM buffers without mutating them. Inspect matching OHLCV SHM buffers without mutating them.
@ -765,7 +764,10 @@ def inspect_shm(
reports: list[dict] = [] reports: list[dict] = []
async def main() -> None: async def main() -> None:
async with open_piker_runtime('shm_inspector'): async with open_piker_runtime(
'shm_inspector',
debug_mode=pdb,
):
for ( for (
shmfile, shmfile,
shm, shm,
@ -840,26 +842,88 @@ def inspect_shm(
@store.command(no_args_is_help=True) @store.command(no_args_is_help=True)
def ldshm( def shm(
fqme: str, fqme: str,
write_parquet: bool = True, write_parquet: Annotated[
reload_parquet_to_shm: bool = True, bool,
typer.Option(
'--write-parquet',
help='Persist repaired frames to NativeDB.',
),
] = False,
reload_parquet_to_shm: Annotated[
bool,
typer.Option(
'--reload-parquet-to-shm',
help='Reload persisted repairs into SHM.',
),
] = False,
pdb: bool = False, # --pdb passed? pdb: bool = False, # --pdb passed?
max_gaps: Annotated[
int|None,
typer.Option(
'--max-gaps',
min=0,
help=(
'Maximum largest gaps in read-only output '
'(default: 10).'
),
show_default=False,
),
] = None,
json_output: Annotated[
bool,
typer.Option(
'--json',
help='Emit read-only output as JSON.',
),
] = False,
shm_name: Annotated[ shm_name: Annotated[
str|None, str|None,
typer.Option( typer.Option(
'--shm-name', '--shm-name',
help='Process only this exact SHM object name.', help='Select only this exact SHM object name.',
), ),
] = None, ] = None,
) -> None: ) -> None:
''' '''
Linux ONLY: load any fqme file name matching shm buffer from Inspect volatile OHLCV SHM, with explicit repair options.
/dev/shm/ into an OHLCV numpy array and polars DataFrame,
optionally write to offline storage via `.parquet` file.
''' '''
if (
reload_parquet_to_shm
and
not write_parquet
):
raise typer.BadParameter(
'--reload-parquet-to-shm requires --write-parquet'
)
if (
write_parquet
and (
json_output
or
max_gaps is not None
)
):
raise typer.BadParameter(
'--json and --max-gaps apply only to read-only output'
)
if not write_parquet:
_inspect_shm(
fqme,
shm_name=shm_name,
max_gaps=(
10
if max_gaps is None
else max_gaps
),
json_output=json_output,
pdb=pdb,
)
return
if ( if (
shm_name is not None shm_name is not None
and and

View File

@ -59,7 +59,10 @@ def expect(
def run_pdb_commands( def run_pdb_commands(
commands: list[str], commands: list[str],
initial_cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance', initial_cmd: str = (
'piker store shm xmrusdt.usdtm.perp.binance '
'--write-parquet'
),
timeout: int = 30, timeout: int = 30,
print_output: bool = True, print_output: bool = True,
) -> dict[str, str]: ) -> dict[str, str]:
@ -145,7 +148,10 @@ class InteractivePdbSession:
''' '''
def __init__( def __init__(
self, self,
cmd: str = 'piker store ldshm xmrusdt.usdtm.perp.binance', cmd: str = (
'piker store shm xmrusdt.usdtm.perp.binance '
'--write-parquet'
),
timeout: int = 30, timeout: int = 30,
): ):
self.cmd: str = cmd self.cmd: str = cmd

View File

@ -21,16 +21,16 @@ from piker.storage import cli as storage_cli
from piker.storage.cli import ( from piker.storage.cli import (
_shm_period_and_invalid_count, _shm_period_and_invalid_count,
_summarize_shm_frame, _summarize_shm_frame,
ldshm, shm as shm_cmd,
) )
def test_ldshm_detects_nulls_without_skewing_period() -> None: def test_shm_detects_nulls_without_skewing_period() -> None:
''' '''
Null SHM slots must stop persistence without skewing cadence. Null SHM slots must stop persistence without skewing cadence.
A live QQQ buffer contained an interior run of zero-time rows. A live QQQ buffer contained an interior run of zero-time rows.
``ldshm`` included them in period inference and then sent them to ``shm`` included them in period inference and then sent them to
NativeDB, which correctly rejected the replacement. Build a NativeDB, which correctly rejected the replacement. Build a
60-second series with the same interior hole and enough 60-second series with the same interior hole and enough
bars around the gap. Prove inspection counts every null while bars around the gap. Prove inspection counts every null while
@ -54,7 +54,7 @@ def test_ldshm_detects_nulls_without_skewing_period() -> None:
assert invalid_count == 2 assert invalid_count == 2
def test_ldshm_period_is_an_observed_step() -> None: def test_shm_period_is_an_observed_step() -> None:
''' '''
Sparse timestamps must not invent a storage timeframe. Sparse timestamps must not invent a storage timeframe.
@ -73,13 +73,13 @@ def test_ldshm_period_is_an_observed_step() -> None:
assert invalid_count == 0 assert invalid_count == 0
def test_ldshm_counts_every_invalid_timestamp() -> None: def test_shm_counts_every_invalid_timestamp() -> None:
''' '''
Corrupt timestamps stop persistence just like null SHM slots. Corrupt timestamps stop persistence just like null SHM slots.
Exact zero denotes an unpublished slot, while negative, NaN, and Exact zero denotes an unpublished slot, while negative, NaN, and
infinite values indicate corruption. NativeDB rejects them all, infinite values indicate corruption. NativeDB rejects them all,
but allowing a replacement attempt would still abort ``ldshm``. but allowing a replacement attempt would still abort ``shm``.
Mix each invalid class into a regular series and prove the same Mix each invalid class into a regular series and prove the same
snapshot inspection counts all four while preserving cadence. snapshot inspection counts all four while preserving cadence.
@ -100,7 +100,7 @@ def test_ldshm_counts_every_invalid_timestamp() -> None:
assert invalid_count == 4 assert invalid_count == 4
def test_ldshm_invalid_snapshot_never_reaches_storage( def test_shm_invalid_snapshot_never_reaches_storage(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
''' '''
@ -208,23 +208,25 @@ def test_ldshm_invalid_snapshot_never_reaches_storage(
open_annotations, open_annotations,
) )
ldshm('qqq.nasdaq.ib') shm_cmd(
'qqq.nasdaq.ib',
write_parquet=True,
)
@pytest.mark.parametrize('write_parquet', [False, True]) def test_shm_write_without_reload_uses_deduped_markup_frame(
def test_ldshm_no_reload_uses_deduped_markup_frame(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
write_parquet: bool,
) -> None: ) -> None:
''' '''
Disabling SHM reload must not require a reload payload. A Parquet write without SHM reload must not need a reload
payload.
``ldshm`` created ``new`` only in reload but used it for markup. ``ldshm`` created ``new`` only in reload but used it for markup.
No-write and no-reload modes crashed, the latter after a durable Its no-reload mode crashed after a durable write. Arrange one
write. Arrange one gap, exercise gap, explicitly enable persistence with reload disabled, and
write modes with reload disabled, and prove markup receives the prove markup receives the deduplicated frame without any SHM
deduplicated frame without any SHM mutation. mutation.
''' '''
array = np.zeros( array = np.zeros(
@ -347,13 +349,13 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
open_annotations, open_annotations,
) )
ldshm( shm_cmd(
'x.test', 'x.test',
write_parquet=write_parquet, write_parquet=True,
reload_parquet_to_shm=False, reload_parquet_to_shm=False,
) )
assert len(writes) == int(write_parquet) assert len(writes) == 1
assert len(markups) == 1 assert len(markups) == 1
assert markups[0]['time'].to_list() == [60, 120, 240] assert markups[0]['time'].to_list() == [60, 120, 240]
@ -366,7 +368,7 @@ def test_ldshm_no_reload_uses_deduped_markup_frame(
np.array([60, 60]), np.array([60, 60]),
], ],
) )
def test_ldshm_skips_frame_without_positive_cadence( def test_shm_skips_frame_without_positive_cadence(
times: np.ndarray, times: np.ndarray,
) -> None: ) -> None:
''' '''
@ -376,7 +378,7 @@ def test_ldshm_skips_frame_without_positive_cadence(
undefined for short buffers. Fully unpublished, one-row, and undefined for short buffers. Fully unpublished, one-row, and
duplicate-only frames could therefore fail before the command's duplicate-only frames could therefore fail before the command's
guard. Exercise each shape and prove inspection guard. Exercise each shape and prove inspection
returns no cadence, which directs ``ldshm`` to skip the buffer. returns no cadence, which directs ``shm`` to skip the buffer.
''' '''
period_s, _ = _shm_period_and_invalid_count(times) period_s, _ = _shm_period_and_invalid_count(times)
@ -390,7 +392,7 @@ def test_shm_buffer_discovery_uses_exact_identities(
''' '''
SHM discovery must not mix substring matches or index companions. SHM discovery must not mix substring matches or index companions.
``ldshm`` globbed ``*fqme*`` and could process stale and ``shm`` globbed ``*fqme*`` and could process stale and
active objects for another instrument whose name merely contained active objects for another instrument whose name merely contained
requested text. Create two generations, both OHLCV kinds, index requested text. Create two generations, both OHLCV kinds, index
companions, and a substring collision. Prove discovery returns companions, and a substring collision. Prove discovery returns
@ -551,7 +553,7 @@ def test_shm_parser_rejects_forged_identities(
Reject malformed service, generation, and FQME components so a Reject malformed service, generation, and FQME components so a
same-user object that merely resembles OHLCV can not become an same-user object that merely resembles OHLCV can not become an
``ldshm --shm-name`` target. ``shm --shm-name`` target.
''' '''
assert tsp.parse_shm_buffer_path(Path(name)) is None assert tsp.parse_shm_buffer_path(Path(name)) is None

View File

@ -41,7 +41,6 @@ def test_store_group_shows_help_without_arguments() -> None:
'anal', 'anal',
'audit', 'audit',
'delete', 'delete',
'ldshm',
'ls', 'ls',
'series', 'series',
'shm', 'shm',
@ -54,8 +53,8 @@ def test_store_commands_show_help_without_arguments() -> None:
Bare endpoints must not open runtimes or report missing args. Bare endpoints must not open runtimes or report missing args.
Every endpoint is discoverable by typing its name once. Exercise Every endpoint is discoverable by typing its name once. Exercise
both required-input and explicitly-triggered listing commands and required-input commands and prove help rendering exits before any
prove help rendering exits before any callback can run. callback can run.
''' '''
runner = CliRunner() runner = CliRunner()
@ -63,9 +62,6 @@ def test_store_commands_show_help_without_arguments() -> None:
'anal', 'anal',
'audit', 'audit',
'delete', 'delete',
'ldshm',
'ls',
'series',
'shm', 'shm',
): ):
result = runner.invoke(store, [command]) result = runner.invoke(store, [command])
@ -90,23 +86,102 @@ def test_series_lists_exact_native_periods(
nativedb.mkdir() nativedb.mkdir()
one = nativedb / 'qqq.nasdaq.ib.ohlcv1s.parquet' one = nativedb / 'qqq.nasdaq.ib.ohlcv1s.parquet'
sixty = nativedb / 'qqq.nasdaq.ib.ohlcv60s.parquet' sixty = nativedb / 'qqq.nasdaq.ib.ohlcv60s.parquet'
three_hundred = nativedb / (
'qqq.nasdaq.ib.ohlcv300s.parquet'
)
one.write_bytes(b'1') one.write_bytes(b'1')
sixty.write_bytes(b'60') sixty.write_bytes(b'60')
three_hundred.write_bytes(b'300')
monkeypatch.setattr(config, 'get_conf_dir', lambda: tmp_path) monkeypatch.setattr(config, 'get_conf_dir', lambda: tmp_path)
result = CliRunner().invoke( result = CliRunner().invoke(
store, store,
['series', '--all', '--json'], ['series', '--json'],
) )
assert result.exit_code == 0 assert result.exit_code == 0
payload = json.loads(result.output) payload = json.loads(result.output)
assert [item['period_s'] for item in payload] == [1, 60] assert [item['period_s'] for item in payload] == [1, 60, 300]
assert [item['path'] for item in payload] == [ assert [item['path'] for item in payload] == [
str(one), str(one),
str(sixty), str(sixty),
str(three_hundred),
] ]
compact = CliRunner().invoke(store, ['ls'])
assert compact.exit_code == 0
assert 'qqq.nasdaq.ib' in compact.output
assert '1s, 60s, 300s' in compact.output
def test_shm_reload_requires_explicit_persistence() -> None:
'''
SHM reload must not silently opt into durable persistence.
Reload reads back the Parquet produced by the repair path, so the
reload flag alone has no valid input and must not open runtime or
storage services. Invoke that invalid combination and prove CLI
validation rejects it before inspecting an FQME.
'''
result = CliRunner().invoke(
store,
['shm', 'qqq.nasdaq.ib', '--reload-parquet-to-shm'],
)
assert result.exit_code == 2
assert 'requires --write-parquet' in result.output
@pytest.mark.parametrize(
'option',
[
['--json'],
['--max-gaps', '10'],
],
)
def test_shm_repair_rejects_read_only_formatting(
option: list[str],
) -> None:
'''
Repair mode must not silently ignore read-only output options.
Combining the old repair workflow with the new inspector made
``--json`` and an explicitly default-valued ``--max-gaps``
appeared accepted even though repair produced logs and entered an
interactive pause. Invoke each conflicting option and prove
validation stops before runtime or storage can be opened.
'''
result = CliRunner().invoke(
store,
[
'shm',
'qqq.nasdaq.ib',
'--write-parquet',
*option,
],
)
assert result.exit_code == 2
assert 'apply only to read-only output' in result.output
def test_ls_has_description_in_store_help() -> None:
'''
Compact NativeDB discovery must explain itself in group help.
The backend-oriented ``ls`` callback had no docstring, leaving
its command-map description blank. Render the parent help and
prove the compact series purpose is visible without opening the
command.
'''
result = CliRunner().invoke(store, [])
assert result.exit_code == 2
assert 'List NativeDB FQMEs' in result.output
def test_shm_endpoint_reports_immutable_snapshot( def test_shm_endpoint_reports_immutable_snapshot(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
@ -202,7 +277,7 @@ def test_shm_endpoint_reports_immutable_snapshot(
assert payload[0]['invalid_count'] == 1 assert payload[0]['invalid_count'] == 1
def test_ldshm_rejects_unknown_exact_name_before_runtime( def test_shm_rejects_unknown_exact_name_before_runtime(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
''' '''
@ -240,7 +315,7 @@ def test_ldshm_rejects_unknown_exact_name_before_runtime(
result = CliRunner().invoke( result = CliRunner().invoke(
store, store,
[ [
'ldshm', 'shm',
'qqq.nasdaq.ib', 'qqq.nasdaq.ib',
'--shm-name', '--shm-name',
'missing', 'missing',