Manual Backfill Qualification
=============================

Purpose
-------

This runbook separates provider coverage, durable NativeDB state,
shared-memory hydration, and chart rendering while reproducing gappy
history. Work one FQME and timeframe at a time. Preserve the original
Parquet first, then qualify both repair of that evidence and a completely
fresh backfill.

The read-only command used throughout is::

    piker store audit FQME --period 60
    piker store audit FQME --period 60 --json

It reads raw Parquet directly. It does not open a storage client,
canonicalize data, publish cache state, attach SHM, or rewrite history.
Positive gaps are deliberately unclassified; an observed interval is not
automatically corruption or an expected venue closure.

.. warning::

   Do not run qualification against normal user storage. The helper below
   requires a marked disposable root before it will seed, archive, or clear
   a Parquet. ``piker store anal`` and ``piker store shm --write-parquet``
   are interactive and potentially mutating; they are not audit commands.

   Keep the disposable root private and do not rename, replace, or symlink
   files under it while a helper is running. Stop the chart cleanly before
   every audit or comparison phase.

Load The Xonsh Helpers
----------------------

Run from the repository root in the same xonsh which will launch the chart
or daemon::

    source snippets/nativedb_backfill_audit.xsh

Choose a disposable root and copy only the provider configuration needed
for the case::

    run_root = p'/tmp/piker-backfill/mnq-20260918-60s'
    source_config = p'~/.config/piker'.expanduser()
    bfq_init(run_root, source_config)
    bfq_record_case(
        run_root,
        fqme='mnq.cme.20260918.ib',
        period_s=60,
        provider='ib',
        symptom='gappy chart during reverse backfill',
        bad_start_utc=None,
        bad_end_utc=None,
    )

``bfq_init()`` sets ``XDG_CONFIG_HOME`` in the current xonsh, so every
subsequent ``piker`` and ``pikerd`` child inherits the disposable config.
The effective storage path is under ``RUN_ROOT/xdg/piker/nativedb``.
It also records the exact repository root and ``piker`` executable in the
qualification marker, prepends that root to child ``PYTHONPATH``, and
verifies that the bound executable exposes ``piker store audit``. Audits
fail if that executable imports another tree.

The copied ``brokers.toml`` can contain credentials. Keep the run root
private and remove it manually after evidence is no longer needed.

Repository Preflight
--------------------

Before every daemon start, verify the code and config inherited by child
processes::

    git status --short --branch
    git rev-parse HEAD
    python -c 'import piker; print(piker.__file__)'
    python -c 'import os; print(os.environ["XDG_CONFIG_HOME"])'

Stop any daemon from the previous phase cleanly. Use a separate registry
address in the disposable ``conf.toml`` when a normal Piker stack remains
active.

Known-Bad Replay
----------------

Copy the original file; never move it out of normal storage::

    fqme = 'mnq.cme.20260918.ib'
    period_s = 60
    source_parquet = (
        p'~/.config/piker/nativedb'.expanduser()
        /f'{fqme}.ohlcv{period_s}s.parquet'
    )
    bfq_seed(run_root, fqme, period_s, source_parquet)
    bfq_audit(run_root, 'baseline', fqme, period_s)

The baseline phase writes::

    evidence/baseline.json
    evidence/baseline.parquet
    evidence/baseline.manual.json

The JSON checksum must match the read-only snapshot, and the manual sidecar
binds that report to its phase and case. Comparison rechecks those bindings
before reading either phase. The report records raw schema, row and
timestamp counts, non-finite values, ordering, duplicate excess, canonical
index state, and every retained positive-gap detail. Files with invalid
Parquet encoding can not produce a JSON report, but ``--snapshot`` still
preserves their exact bytes before parse failure.

Launch the chart manually from the same shell and capture logs::

    piker -l info chart mnq.cme.20260918.ib \
        2>&1 | tee @(run_root / 'logs' / 'repair-chart.log')

After a clean stop::

    bfq_audit(
        run_root,
        'repair',
        fqme,
        period_s,
        require_structural=True,
    )
    bfq_compare(run_root, 'baseline', 'repair')

The comparison fails if structural validity regresses, any original
timestamp disappears, chronological endpoints shrink, timestamps become
duplicated or unordered, indexes cease to be canonical, or OHLCV becomes
non-finite. Provider replacement of matching seam rows is reported
separately as ``changed_common_timestamps`` for operator review.

Fresh Backfill
--------------

Yes, fresh qualification means saving the known-bad evidence and then
starting with no active Parquet in the disposable NativeDB. Archive only
the disposable file::

    bfq_clear_for_fresh(run_root, fqme, period_s)

Confirm the printed active path is absent, launch the same chart, wait for
bounded provider exhaustion or the intended coverage, then stop cleanly::

    bfq_audit(
        run_root,
        'fresh',
        fqme,
        period_s,
        require_structural=True,
    )

Do not compare ``baseline`` to ``fresh`` as a timestamp-preservation gate:
provider limits may intentionally produce a different initial extent.
Treat ``fresh`` as the baseline for restart and append qualification.

Restart And Append
------------------

Restart the same chart against the fresh persisted file, stop cleanly, and
record::

    bfq_audit(
        run_root,
        'restart',
        fqme,
        period_s,
        require_structural=True,
    )
    bfq_compare(run_root, 'fresh', 'restart')

Leave a subsequent run active long enough to append new samples, stop, and
record::

    bfq_audit(
        run_root,
        'append',
        fqme,
        period_s,
        require_structural=True,
    )
    bfq_compare(
        run_root,
        'restart',
        'append',
        require_newer=True,
    )

The comparison JSON reports gaps added and removed. Expected closures stay
visible until a venue-aware classifier proves their session alignment.

Gap Layer Triage
----------------

List exact durable series before choosing a timeframe, and inspect live
SHM separately::

    piker store ls
    piker store series FQME_SUBSTRING
    piker store shm FQME --max-gaps 20

All three forms are read-only. ``ls`` summarizes periods by FQME, while
``series`` reports each ``(FQME, period)`` file independently. Read-only
``shm`` reports every exact actor generation and
``hist``/``rt`` buffer for the FQME, including invalid rows, ordering
defects, inferred versus expected cadence, and the largest timestamp gaps.
Use ``--shm-name EXACT_NAME`` to inspect only one reported buffer.

Use the persisted report and chart together:

===============================  =========================================
Observation                      Suspected layer
===============================  =========================================
Parquet contiguous, chart gappy  SHM hydration, sampling, or chart render
Parquet gap, provider has bars    provider-delta or storage merge
Provider and Parquet both gappy  provider omission or venue closure
Gap vanishes after restart       SHM publication or cache lifecycle
Old timestamps disappear         destructive persistence regression
Newest seam duplicates           conflict or dedupe policy
===============================  =========================================

Always record exact UTC gap endpoints. The audit's numeric epoch is
authoritative; the ``*_utc`` fields are deterministic display values.

Expansion Matrix
----------------

Expand only after the fixed FQME passes fresh, restart, and append. Use a
new disposable root and case record for every row:

====================  ==========  ========================================
Case                  Period      Coverage model
====================  ==========  ========================================
Original IB future    60 seconds  CME maintenance and weekend closures
Original IB future    1 second    high-frequency provider limits
Second CME future     60 seconds  contract-specific history
IB equity             60 seconds  regular overnight and weekend closure
IB FX                 60 seconds  24x5 session behavior
Crypto spot/perp      60 seconds  continuous-session control
Expired future        60 seconds  history-only and expiry behavior
====================  ==========  ========================================

A sourceable matrix can remain ordinary Python data in xonsh::

    cases = [
        {
            'name': 'mnq-60s',
            'fqme': 'mnq.cme.20260918.ib',
            'period_s': 60,
            'provider': 'ib',
        },
        {
            'name': 'mnq-1s',
            'fqme': 'mnq.cme.20260918.ib',
            'period_s': 1,
            'provider': 'ib',
        },
    ]
    for case in cases:
        case_root = p'/tmp/piker-backfill' / case['name']
        bfq_init(case_root, source_config)
        bfq_record_case(
            case_root,
            fqme=case['fqme'],
            period_s=case['period_s'],
            provider=case['provider'],
            symptom='manual expansion case',
        )

This loop prepares isolated case roots only. Run the complete seed or
fresh, chart, audit, restart, append, and compare lifecycle in each root;
directory creation alone is not qualification.

The chart path may allocate companion 1-second and 60-second files. Audit
the selected case path explicitly and retain companion files as secondary
evidence; do not imply that the live request isolated one timeframe.

Interruption Qualification
--------------------------

After normal restart is stable, interrupt one phase with graceful Ctrl-C,
restart, and compare against the last clean evidence. Provider disconnect,
SIGTERM, and deterministic write-barrier failures come next. Avoid timing a
SIGKILL by hand; add an explicit failpoint before qualifying atomic replace
boundaries.

Result Interpretation
---------------------

``structural_ok`` means the raw persisted file satisfies durable schema,
finite values, positive unique ordered timestamps, canonical index, and
period-aligned gaps. ``gap_free`` means no positive gaps were observed when
at least two valid timestamps made cadence verifiable; otherwise it is
``null``. ``qualification_ok`` combines both and is intentionally strict.

A structurally valid session-market file can have ``gap_free = false``.
Those intervals require provider/session evidence before being marked as
expected closures. Unknown calendar state must stay unknown.
