piker/.taken/current.org

# vim: et sw=2 ts=2
#+TITLE: piker project store
#+TODO: TODO(t) WIP(w!) WAIT(a@) | DONE(d!) NOPE(n@)

# Checkbox states: [ ] open, [-] partial, [x] complete

* piker
:PROPERTIES:
:CATEGORY: piker
:END:
** TODO prs
:PROPERTIES:
:CREATED: [2026-07-15]
:CARRIED: 29
:END:
- [ ] move the below into *** todos with pr/branch links
  * [ ] DPI-aware font scaling + manual kb bindings.
  * [ ] gap_annotator for time gaps.
  * [ ] moar `claude`-y skills,
  * [ ] /lint with ruff
  * [ ] /run-tests
*** WIP qualify known-FQME history backfill
:PROPERTIES:
:CREATED: [2026-07-24]
:CARRIED: 21
:REPO: /home/goodboy/repos/piker
:END:
- [x] preserve the dirty `flake_update` main checkout before
  switching branches
- [x] check out `backfiller_deep_fixes` in the main checkout and
  verify every process imports it
- [ ] complete the below qualification plan
  * [ ] select one known-problem provider, FQME, timeframe, date range,
    and original symptom
  * [ ] preserve and checksum any known-bad parquet
  * [ ] bootstrap `pikerd` with disposable config and NativeDB storage
  * [ ] record baseline timestamps, rows, uniqueness, nulls,
    ordering,
  indexes, gaps, and readability
  * [ ] reproduce fresh-backfill, restart, and append behavior for
    only
  the selected case
  * [ ] convert each reproduced failure into a failing deterministic
    test before fixing it
  * [ ] rerun the exact manual lifecycle after every production
    correction
  * [ ] add deterministic subprocess coverage for process death and
    cross-process locking
  * [ ] exercise Ctrl-C, SIGTERM, SIGKILL, provider interruption, and
    restart recovery
  * [ ] qualify MarketStore and actual sampler backpressure separately
  * [ ] expand to other timeframes and providers only after the
    selected FQME passes
**** TODO fixed-FQME-case backfill qualify
:PROPERTIES:
:CREATED: [2026-07-29]
:CARRIED: 17
:REPO: piker/
:END:
- [ ] follow through using both new files,
  * ./docs/manual_backfill_qualification.rst
  * ./snippets/nativedb_backfill_audit.xsh
- [ ] choose and record one known-problem FQME, timeframe, UTC range,
  and original chart symptom
- [ ] source `snippets/nativedb_backfill_audit.xsh` and initialize a
  private disposable root with only the required provider config
- [ ] run repository and environment preflight at commit `0846cbd4`
- [ ] seed and audit the known-bad baseline, then replay repair and run
  `bfq_compare()` without losing original timestamps
- [ ] clear only the disposable Parquet and qualify a fresh backfill
- [ ] restart against fresh storage and compare `fresh` to `restart`
- [ ] leave one subsequent run active long enough to append, then compare
  `restart` to `append` with `require_newer=True`
- [ ] restart QQQ 1s on fresh datad SHM and verify the former 653-row
  zero-time reservation does not recur
- [ ] preserve chart logs, audit JSON, Parquet snapshots, and exact UTC
  gap endpoints for review
**** TODO review ^qualify^ evidence and close follow-ups
:PROPERTIES:
:CREATED: [2026-07-29]
:CARRIED: 17
:REPO: /piker
:END:
- [ ] verify every qualification process imports commit `0846cbd4`
- [ ] review baseline, repair, fresh, restart, and append reports for
  structural, timestamp, endpoint, and gap regressions
- [ ] distinguish durable Parquet gaps from SHM hydration or chart-only
  gaps using the runbook triage table
- [ ] convert each reproduced failure into a deterministic regression
  before changing production code
- [ ] prepare the final qualification summary and any required patch,
  PR, or worktree-lifecycle follow-ups
*** TODO cross-harness AI skill integration
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:ISSUE: https://www.pikers.dev/pikers/piker/issues/79
:END:
- [ ] review and land the portable piker skill and OpenCode docs patch
- [ ] reconcile the active global `commit-msg` override with the
  repo-local shared skill and style guide
- [ ] verify skill and command discovery after restarting Claude Code
  and OpenCode
- [ ] fold remaining skill-content review into the canonical trackers:
  * [ ] https://www.pikers.dev/pikers/piker/issues/69
  * [ ] https://www.pikers.dev/pikers/piker/issues/79
*** TODO vectorize `gap_annotator` client spec building
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] replace per-row Polars slicing and filtering in
  `piker/tsp/_annotate.py:139` with extracted columns and batched
  lookups
- [ ] preserve missing-prior-row, epoch-skip, direction, label, and
  batch-payload behavior
- [ ] benchmark the 1285-gap case against the handoff in
  `notes_to_self/claudy/gapsannotator_sesh.md`
- [ ] add focused correctness coverage for generated rectangle, arrow,
  and text specs
*** TODO integrate the piker `ruff` lint skill
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] review the clean but unmerged `claudy_writes_ruff_conf` branch
- [ ] validate its `ruff.toml` policy and staged-file checker against
  the current tree
- [ ] port the `/lint` skill to shared Agent Skills frontmatter with
  thin harness-specific commands
- [ ] rebase and prepare the accepted lint/config changes for landing
*** TODO reconcile stale Claude plans and worktrees
:PROPERTIES:
:CREATED: [2026-07-21]
:CARRIED: 24
:REPO: /home/goodboy/repos/piker
:END:
- [ ] reconcile `ai/claude-code/plans/shm_port.md` with the implemented
  `tractor.ipc._shm` shim
- [ ] classify `datad_service.md` as an architecture record rather than
  an active implementation plan
- [ ] inspect generated state before retiring the merged
  `wkt/flake_update` and `worktree-ngi0_submission` worktrees
- [ ] preserve the unmerged lint worktree until its branch is reviewed
** TODO chart-local gap overlay follow-ups
:PROPERTIES:
:CREATED: [2026-08-14]
:CARRIED: 7
:REPO: /home/goodboy/repos/piker
:END:
- [ ] trigger `GapOverlayMngr.refresh()` from an explicit history
  revision/publication event after repair or prepend
- [ ] expose gap-overlay UX through UI config
  * [ ] startup visibility and enabled timeframes
  * [ ] default `Ctrl+G` key binding
- [ ] live-qualify chart-local overlay behavior
  * [ ] actor startup and focused 1s/60s toggling
  * [ ] multi-symbol and duplicate-FQME cached charts
  * [ ] secondary-`Viz` placement and backfill repositioning
  * [ ] remote disconnect and owner-scoped graphics cleanup
- [ ] add focused Qt/actor regressions for the qualified lifecycle

** TODO factor typed chart annotation IPC dialogs
:PROPERTIES:
:CREATED: [2026-08-14]
:CARRIED: 7
:REPO: /home/goodboy/repos/piker
:END:
- [ ] exercise the strict `SetGapOverlay`/`GapOverlay`
  dialog under a live Tractor context
  * [ ] wrong-direction and unrelated payload rejection
  * [ ] timeout, cancellation, stale reply, and disconnect recovery
- [ ] factor keyed send/receive locking, fresh request IDs, and stale
  response correlation into a generic IPC dialog helper after a
  second typed consumer establishes the interface
- [ ] migrate legacy `remote_annotate()` dict commands to tagged
  request/response message types
- [ ] decide whether to fold `remote_gap_overlays()` into the general
  annotation endpoint after the legacy protocol is typed
** proj_mgmt
 :PROPERTIES:
 :CATEGORY: project
 :END:
*** TODO ngi0 aug01 funding submit!
  DEADLINE: <2026-08-01 Sat>
 :PROPERTIES:
 :CREATED: [2026-06-01]
:CARRIED: 33
 :END:
- [ ] wip pr: https://github.com/goodboy/tractor/pull/431
  https://nlnet.nl/propose/
  https://nlnet.nl/foundation/policies/generativeAI/
  https://nlnet.nl/commonsfund/
*** TODO MEETING(s)
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
** ui
 :PROPERTIES:
 :CATEGORY: piker.ui
 :END:
*** TODO dpi_auto_calc
 :PROPERTIES:
 :CREATED: [2026-07-15]
:CARRIED: 29
 :END:
- [ ] PUT PRs here
** brokers
 :PROPERTIES:
 :CATEGORY: piker.brokers
 :END:
*** TODO IB: follow-up for search results bs!
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   * https://www.pikers.dev/pikers/piker/issues/85
   - [X] fix the `si.expiry..ib` search result?
   - [ ] introspect why `Client.search_symbols()` is returning the
        empty `.<venue>` case!
       - presumably by something weird in the server-side endpoint?
   - [ ] figure out better tuning of client side fuzzy-filter versus
        server-side versus UI final display pass!
*** TODO ish for port to `uv`,
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   * https://github.com/ib-api-reloaded/ib_async/issues/204
   - [ ] mention to track in #68
*** TODO profiling: try newer libs,
 :PROPERTIES:
 :CREATED: [2026-05-21]
:CARRIED: 33
 :END:
   - [ ]  https://github.com/pikers/piker/issues/337
** perf/tooling
 :PROPERTIES:
 :CATEGORY: piker.profile
 :END:
- [ ] link existing GH issues for trying new
     libs!

** FSP backfill: branch work
*** TODO review and commit FSP backfill synchronization
:PROPERTIES:
:ID: piker-fsp-backfill-sync-review-and-commit
:CREATED: [2026-09-03]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker/wkts/fsp_backfill_sync
:END:
- [ ] review committed token-targeted history graphics change
  (`fadab3d2`)
- [ ] review and commit the FSP stream-contract boundary
  * yo yo yo
  * `piker/data/__init__.py`
  * `piker/data/ticktools.py`
  * `piker/fsp/_api.py`
  * `piker/fsp/_momo.py`
  * staged portion of `piker/fsp/_volume.py`
- [ ] review and commit the synchronization and live-volume boundary
  * explicit `Flume` RT/history cadence
  * immutable history snapshots and absolute SHM-bound synchronization
  * relevant backfill filtering and non-duplicating destination steps
  * nonnegative sampled and derived volume
  * M4 graphics-cache invalidation and unchanged-update suppression
  * FSP synchronization regressions and Prompt-IO provenance
- [ ] review and commit the `piker-fsp-expert` skill boundary
- [ ] run boundary checks and the final test suite
- [ ] inspect each staged diff and commit message before committing
*** TODO add cumulative M4 downsampling for volume curves
:PROPERTIES:
:ID: piker-cumulative-m4-volume-downsampling
:CREATED: [2026-09-03]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker/wkts/fsp_backfill_sync
:END:
- [ ] define cumulative bucket semantics for step-style data
  * sum all source values represented by each pixel bucket
  * preserve bucket time/index placement and zero-volume gaps
- [ ] add a sum-reducer variant alongside price min/max M4
- [ ] route unit and dollar-volume curves through cumulative downsampling
- [ ] keep price, OHLC, and non-cumulative signal curves on min/max M4
- [ ] preserve incremental appends, view changes, and cache invalidation
- [ ] test sum conservation across zoom levels and partial edge buckets
- [ ] profile the cumulative path against the existing M4 implementation

** TODO review and save the Trend Calculus prototype
:PROPERTIES:
:ID: piker-trend-calculus-prototype-review
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:ISSUE: https://github.com/pikers/piker/issues/187
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Regenerate and review six Trend Calculus behavior boundaries
  on trend_calc_on_jev after workflow commit f148fbcc on
  moar_ai_skillz lands; retain source/test pairs and inspect every
  message before saving.
- [ ] Run each boundary check from the pinned package and the final
  safe regression sequence; the working-tree offline suite passed 253
  tests.
- [ ] Qualify one selected live source through startup, quiet bars,
  skipped wakeups, history repair, replay and owner cancellation.
- [ ] Coordinate inherited stream/backfill work with
  [[id:piker-fsp-backfill-sync-review-and-commit][the existing FSP synchronization task]].

*** TODO split draft TC commit after upstream history is settled
:PROPERTIES:
:ID: piker-trend-calculus-draft-commit-split
:CREATED: [2026-10-06]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:TAKEN_APPEND_OP: 85c6bdbe-7655-4323-b366-2732ac86e16c
:TAKEN_APPEND_REQUEST: 10de3a5f301761e43ba80988385d9eb180d41fc3025ed1cd66255f80e3c9fa98
:AI_DLOG: 01a109d6-d0ec-70d3-b2cf-2112a6780aa3
:AI_WKT: /home/goodboy/repos/piker
:END:
Draft snapshot: =e815fd77cfbf89b241bfabf19bda7b1c5f25c784=.
- [ ] Wait until upstream branches and their landing order are
  settled; preserve a reference to the draft snapshot.
- [ ] Reset/unpack draft commit e815fd77 while retaining the full TC
  changeset and preserving unrelated working-tree and staged work.
- [ ] Prepare a proper commit plan for that changeset with coherent
  behavior boundaries, matching tests, docs and provenance.
- [ ] Run each boundary's checks and review its diff and message
  before recommitting.

** TODO qualify Jev regime judgments against real responses
:PROPERTIES:
:ID: piker-jev-live-regime-qualification
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Once TYPESAFE_API_KEY is available and a live request is
  authorized, send one small synthetic multi-timeframe request
  through the HTTPX/Trio client.
- [ ] Verify the documented request/response schema, model identity,
  distributions, usage, errors and measured end-to-end latency.
- [ ] Evaluate useful regime labels and confidence on held-out trend
  snapshots, including flat, reversing, aligned and conflicting
  timeframes.
- [ ] Measure sustained pacing, coalescing, source freshness and cost
  before enabling realtime judgments.
- [ ] Consider an SDK AnyIO port or Trio backend only after the
  direct HTTP prototype demonstrates useful classifications.

** TODO connect finalized trends to the Jev provider-service
:PROPERTIES:
:ID: piker-jev-provider-service-integration
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Obtain the separate system-1 provider-service repository or
  interface and identify its input, output, lifecycle and ownership
  contracts.
- [ ] Adapt serve_regimes and RegimeClassifier to the service nursery
  and typed publication interface without mixing model labels into
  deterministic trend state.
- [ ] Configure market, periods, model, credentials, request pacing,
  evidence retention and freshness limits explicitly.
- [ ] Verify reconnect, revision/gap invalidation, backpressure,
  stale-response rejection and shutdown with the actual service.

** TODO recover Pine source and check Trend Calculus parity
:PROPERTIES:
:ID: piker-trend-calculus-pine-parity
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:ISSUE: https://github.com/pikers/piker/issues/103
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Recover the human's Pine implementation and record its
  provenance/version alongside the pinned Lua and Scala research.
- [ ] Compare startup seeding, extrema ties, FHLS splitting, opposing
  pivots and recursive reversal orders.
- [ ] Add independent fixtures for meaningful semantic differences;
  distinguish extremum time from confirmation time and mutable-bar
  previews.

** TODO qualify multitimeframe pivots and regime annotations
:PROPERTIES:
:ID: piker-trend-regime-chart-qualification
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Inspect actual 1/5/15-second layers and hover information on a
  selected live market; compare confirmed pivots and regime
  boundaries to finalized source evidence.
- [ ] Verify chart-axis placement, gaps, revisions, hidden charts,
  duplicate cached feeds and owner-scoped cleanup.
- [ ] Qualify model rectangles at response receipt and expiry,
  including stale classifications and source-segment changes.
- [ ] Measure quiet-bar display delay before deciding whether
  publication needs an explicit output notification.
- [ ] Coordinate shared history-overlay lifecycle with the existing
  chart-local gap overlay follow-ups; route any annotation IPC
  factoring to the existing typed-dialog task.

** TODO qualify a native subsecond source for trend regimes
:PROPERTIES:
:ID: piker-trend-native-subsecond-sampling
:CREATED: [2026-10-05]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker
:AI_BRANCH: trend_calc_on_jev
:END:
[[file:/home/goodboy/repos/piker/docs/trend_calculus.rst][Research and prototype guide]]
- [ ] Define the required high-frequency periods and producer
  contract; the numeric/MTF APIs accept fractional periods but the
  current native clock uses whole seconds.
- [ ] Extend or select the source/sampler time grid before enabling
  subsecond chart/service periods; retain strictly increasing
  coordinates and complete aligned buckets.
- [ ] Measure compute, SHM publication, service latency and chart
  refresh under the selected cadence, including gaps, replay and
  cancellation.

** TODO migrate clearing to strict typed messaging
:PROPERTIES:
:ID: piker-typed-clearing-migration
:CREATED: [2026-10-06]
:CARRIED: 0
:REPO: /home/goodboy/repos/piker
:AI_HARNESS: codex
:AI_WORKTREE: /home/goodboy/repos/piker/wkts/typed_clearing_protocol
:AI_BRANCH: wkt/typed_clearing_protocol
:END:
[[file:/home/goodboy/repos/piker/wkts/typed_clearing_protocol/plans/opencode/typed-ems-messaging.md::823][Migration plan and commit boundaries]]

Foundation evidence: plan commit 1793b47c and implementation commit
7fee9b86; human reports both pushed. Strict schemas, validation,
directional stream helpers and the pure OrderState reducer are
implemented. Prior qualification: 65 offline tests and eight IPC
tests passed. Production endpoints still use the legacy protocol.

- [ ] Port paper and its paired EMS consumer together.
  * [ ] Typed startup, requests, events and open-order completion.
  * [ ] Deterministic synthetic ticks and distinct execution IDs.
- [ ] Port Binance, Kraken and IB individually with paired EMS handling.
  * [ ] Shared offline conformance and provider translation fixtures.
  * [ ] Preserve execution identity, signed positions and account mapping.
- [ ] Integrate OrderState into EMS routing and broker-event handling.
  * [ ] Bound pre-ack buffering by age in its asynchronous owner.
  * [ ] Preserve cancel intent, fill deduplication and late-event handling.
- [ ] Migrate client/EMS startup, local queues and UI consumers together.
  * [ ] Replace wire snapshot mutation and dictionary reconstruction.
- [ ] Enforce each migrated endpoint pair with Tractor payload specs.
  * [ ] Narrow receive sets; validate startup, RPC arguments and results.
  * [ ] Reject malformed, wrong-direction and wrong-phase messages.
- [ ] Qualify real adapter and actor behavior offline.
  * [ ] Cancel-before-ack; early, duplicate and late fills.
  * [ ] Empty/populated startup snapshots and two-client order isolation.
  * [ ] Abrupt transport EOF, cancellation and bounded failure teardown.
  * [ ] Repeated lifecycle tests under supported spawn methods.
- [ ] Resolve the duplicate pytest --ll option before a Tractor pin upgrade.
- [ ] Document the position-broadcast isolation limitation and policy.
- [ ] Remove legacy IPC paths after all paired migrations qualify.
- [ ] Propose a reviewable commit plan for each qualified implementation.