piker/.taken/current.org

# vim: et sw=2 ts=2
#+TITLE: piker project store
#+TODO: TODO(t) WIP(w!) WAIT(a@) | DONE(d!) NOPE(n@)

# Checkbox states: [ ] open, [-] partial, [x] complete

* piker
:PROPERTIES:
:CATEGORY: piker
:END:
** TODO prs
:PROPERTIES:
:CREATED: [2026-07-15]
:CARRIED: 29
:END:
- [ ] move the below into *** todos with pr/branch links
  * [ ] DPI-aware font scaling + manual kb bindings.
  * [ ] gap_annotator for time gaps.
  * [ ] moar `claude`-y skills,
  * [ ] /lint with ruff
  * [ ] /run-tests
*** WIP qualify known-FQME history backfill
:PROPERTIES:
:CREATED: [2026-07-24]
:CARRIED: 21
:REPO: /home/goodboy/repos/piker
:END:
- [x] preserve the dirty `flake_update` main checkout before
  switching branches
- [x] check out `backfiller_deep_fixes` in the main checkout and
  verify every process imports it
- [ ] complete the below qualification plan
  * [ ] select one known-problem provider, FQME, timeframe, date range,
    and original symptom
  * [ ] preserve and checksum any known-bad parquet
  * [ ] bootstrap `pikerd` with disposable config and NativeDB storage
  * [ ] record baseline timestamps, rows, uniqueness, nulls,
    ordering,
  indexes, gaps, and readability
  * [ ] reproduce fresh-backfill, restart, and append behavior for
    only
  the selected case
  * [ ] convert each reproduced failure into a failing deterministic
    test before fixing it
  * [ ] rerun the exact manual lifecycle after every production
    correction
  * [ ] add deterministic subprocess coverage for process death and
    cross-process locking
  * [ ] exercise Ctrl-C, SIGTERM, SIGKILL, provider interruption, and
    restart recovery
  * [ ] qualify MarketStore and actual sampler backpressure separately
  * [ ] expand to other timeframes and providers only after the
    selected FQME passes
**** TODO fixed-FQME-case backfill qualify
:PROPERTIES:
:CREATED: [2026-07-29]
:CARRIED: 17
:REPO: piker/
:END:
- [ ] follow through using both new files,
  * ./docs/manual_backfill_qualification.rst
  * ./snippets/nativedb_backfill_audit.xsh
- [ ] choose and record one known-problem FQME, timeframe, UTC range,
  and original chart symptom
- [ ] source `snippets/nativedb_backfill_audit.xsh` and initialize a
  private disposable root with only the required provider config
- [ ] run repository and environment preflight at commit `0846cbd4`
- [ ] seed and audit the known-bad baseline, then replay repair and run
  `bfq_compare()` without losing original timestamps
- [ ] clear only the disposable Parquet and qualify a fresh backfill
- [ ] restart against fresh storage and compare `fresh` to `restart`
- [ ] leave one subsequent run active long enough to append, then compare
  `restart` to `append` with `require_newer=True`
- [ ] restart QQQ 1s on fresh datad SHM and verify the former 653-row
  zero-time reservation does not recur
- [ ] preserve chart logs, audit JSON, Parquet snapshots, and exact UTC
  gap endpoints for review
**** TODO review ^qualify^ evidence and close follow-ups
:PROPERTIES:
:CREATED: [2026-07-29]
:CARRIED: 17
:REPO: /piker
:END:
- [ ] verify every qualification process imports commit `0846cbd4`
- [ ] review baseline, repair, fresh, restart, and append reports for
  structural, timestamp, endpoint, and gap regressions
- [ ] distinguish durable Parquet gaps from SHM hydration or chart-only
  gaps using the runbook triage table
- [ ] convert each reproduced failure into a deterministic regression
  before changing production code
- [ ] prepare the final qualification summary and any required patch,
  PR, or worktree-lifecycle follow-ups
*** TODO cross-harness AI skill integration
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:ISSUE: https://www.pikers.dev/pikers/piker/issues/79
:END:
- [ ] review and land the portable piker skill and OpenCode docs patch
- [ ] reconcile the active global `commit-msg` override with the
  repo-local shared skill and style guide
- [ ] verify skill and command discovery after restarting Claude Code
  and OpenCode
- [ ] fold remaining skill-content review into the canonical trackers:
  * [ ] https://www.pikers.dev/pikers/piker/issues/69
  * [ ] https://www.pikers.dev/pikers/piker/issues/79
*** TODO vectorize `gap_annotator` client spec building
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] replace per-row Polars slicing and filtering in
  `piker/tsp/_annotate.py:139` with extracted columns and batched
  lookups
- [ ] preserve missing-prior-row, epoch-skip, direction, label, and
  batch-payload behavior
- [ ] benchmark the 1285-gap case against the handoff in
  `notes_to_self/claudy/gapsannotator_sesh.md`
- [ ] add focused correctness coverage for generated rectangle, arrow,
  and text specs
*** TODO integrate the piker `ruff` lint skill
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] review the clean but unmerged `claudy_writes_ruff_conf` branch
- [ ] validate its `ruff.toml` policy and staged-file checker against
  the current tree
- [ ] port the `/lint` skill to shared Agent Skills frontmatter with
  thin harness-specific commands
- [ ] rebase and prepare the accepted lint/config changes for landing
*** TODO reconcile stale Claude plans and worktrees
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] reconcile `ai/claude-code/plans/shm_port.md` with the implemented
  `tractor.ipc._shm` shim
- [ ] classify `datad_service.md` as an architecture record rather than
  an active implementation plan
- [ ] inspect generated state before retiring the merged
  `wkt/flake_update` and `worktree-ngi0_submission` worktrees
- [ ] preserve the unmerged lint worktree until its branch is reviewed
** TODO chart-local gap overlay follow-ups
:PROPERTIES:
:CREATED: [2026-08-14]
:CARRIED: 7
:REPO: /home/goodboy/repos/piker
:END:
- [ ] trigger `GapOverlayMngr.refresh()` from an explicit history
  revision/publication event after repair or prepend
- [ ] expose gap-overlay UX through UI config
  * [ ] startup visibility and enabled timeframes
  * [ ] default `Ctrl+G` key binding
- [ ] live-qualify chart-local overlay behavior
  * [ ] actor startup and focused 1s/60s toggling
  * [ ] multi-symbol and duplicate-FQME cached charts
  * [ ] secondary-`Viz` placement and backfill repositioning
  * [ ] remote disconnect and owner-scoped graphics cleanup
- [ ] add focused Qt/actor regressions for the qualified lifecycle

** TODO factor typed chart annotation IPC dialogs
:PROPERTIES:
:CREATED: [2026-08-14]
:CARRIED: 7
:REPO: /home/goodboy/repos/piker
:END:
- [ ] exercise the strict `SetGapOverlay`/`GapOverlay`
  dialog under a live Tractor context
  * [ ] wrong-direction and unrelated payload rejection
  * [ ] timeout, cancellation, stale reply, and disconnect recovery
- [ ] factor keyed send/receive locking, fresh request IDs, and stale
  response correlation into a generic IPC dialog helper after a
  second typed consumer establishes the interface
- [ ] migrate legacy `remote_annotate()` dict commands to tagged
  request/response message types
- [ ] decide whether to fold `remote_gap_overlays()` into the general
  annotation endpoint after the legacy protocol is typed
** proj_mgmt
 :PROPERTIES:
 :CATEGORY: project
 :END:
*** TODO ngi0 aug01 funding submit!
  DEADLINE: <2026-08-01 Sat>
 :PROPERTIES:
 :CREATED: [2026-06-01]
:CARRIED: 33
 :END:
- [ ] wip pr: https://github.com/goodboy/tractor/pull/431
  https://nlnet.nl/propose/
  https://nlnet.nl/foundation/policies/generativeAI/
  https://nlnet.nl/commonsfund/
*** TODO MEETING(s)
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
** ui
 :PROPERTIES:
 :CATEGORY: piker.ui
 :END:
*** TODO dpi_auto_calc
 :PROPERTIES:
 :CREATED: [2026-07-15]
:CARRIED: 29
 :END:
- [ ] PUT PRs here
** brokers
 :PROPERTIES:
 :CATEGORY: piker.brokers
 :END:
*** TODO IB: follow-up for search results bs!
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   * https://www.pikers.dev/pikers/piker/issues/85
   - [X] fix the `si.expiry..ib` search result?
   - [ ] introspect why `Client.search_symbols()` is returning the
        empty `.<venue>` case!
       - presumably by something weird in the server-side endpoint?
   - [ ] figure out better tuning of client side fuzzy-filter versus
        server-side versus UI final display pass!
*** TODO ish for port to `uv`,
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   * https://github.com/ib-api-reloaded/ib_async/issues/204
   - [ ] mention to track in #68
*** TODO profiling: try newer libs,
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   - [ ]  https://github.com/pikers/piker/issues/337
** perf/tooling
 :PROPERTIES:
 :CATEGORY: piker.profile
 :END:
- [ ] link existing GH issues for trying new
     libs!

** FSP backfill: branch work
*** TODO review and commit FSP backfill synchronization
:PROPERTIES:
:ID: piker-fsp-backfill-sync-review-and-commit
:CREATED: [2026-09-03]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker/wkts/fsp_backfill_sync
:END:
- [ ] review committed token-targeted history graphics change
  (`fadab3d2`)
- [ ] review and commit the FSP stream-contract boundary
  * yo yo yo
  * `piker/data/__init__.py`
  * `piker/data/ticktools.py`
  * `piker/fsp/_api.py`
  * `piker/fsp/_momo.py`
  * staged portion of `piker/fsp/_volume.py`
- [ ] review and commit the synchronization and live-volume boundary
  * explicit `Flume` RT/history cadence
  * immutable history snapshots and absolute SHM-bound synchronization
  * relevant backfill filtering and non-duplicating destination steps
  * nonnegative sampled and derived volume
  * M4 graphics-cache invalidation and unchanged-update suppression
  * FSP synchronization regressions and Prompt-IO provenance
- [ ] review and commit the `piker-fsp-expert` skill boundary
- [ ] run boundary checks and the final test suite
- [ ] inspect each staged diff and commit message before committing
*** TODO add cumulative M4 downsampling for volume curves
:PROPERTIES:
:ID: piker-cumulative-m4-volume-downsampling
:CREATED: [2026-09-03]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker/wkts/fsp_backfill_sync
:END:
- [ ] define cumulative bucket semantics for step-style data
  * sum all source values represented by each pixel bucket
  * preserve bucket time/index placement and zero-volume gaps
- [ ] add a sum-reducer variant alongside price min/max M4
- [ ] route unit and dollar-volume curves through cumulative downsampling
- [ ] keep price, OHLC, and non-cumulative signal curves on min/max M4
- [ ] preserve incremental appends, view changes, and cache invalidation
- [ ] test sum conservation across zoom levels and partial edge buckets
- [ ] profile the cumulative path against the existing M4 implementation