pikers
/
piker
4
1
Fork 0
Code Issues 20 Pull Requests 13 Packages Projects Releases Wiki Activity
piker/.taken/current.org

20 KiB
Raw Blame History

piker project store

piker

TODO prs

  • move the below into * todos with pr/branch links

    • DPI-aware font scaling + manual kb bindings.
    • gap_annotator for time gaps.
    • moar `claude`-y skills,
    • /lint with ruff
    • /run-tests

WIP qualify known-FQME history backfill

  • [x] preserve the dirty `flake_update` main checkout before switching branches
  • [x] check out `backfiller_deep_fixes` in the main checkout and verify every process imports it

  • complete the below qualification plan

    • select one known-problem provider, FQME, timeframe, date range, and original symptom
    • preserve and checksum any known-bad parquet
    • bootstrap `pikerd` with disposable config and NativeDB storage
    • record baseline timestamps, rows, uniqueness, nulls, ordering,

    indexes, gaps, and readability

    • reproduce fresh-backfill, restart, and append behavior for only

    the selected case

    • convert each reproduced failure into a failing deterministic test before fixing it
    • rerun the exact manual lifecycle after every production correction
    • add deterministic subprocess coverage for process death and cross-process locking
    • exercise Ctrl-C, SIGTERM, SIGKILL, provider interruption, and restart recovery
    • qualify MarketStore and actual sampler backpressure separately
    • expand to other timeframes and providers only after the selected FQME passes
TODO fixed-FQME-case backfill qualify

  • follow through using both new files,

    • ./docs/manual_backfill_qualification.rst
    • ./snippets/nativedb_backfill_audit.xsh
  • choose and record one known-problem FQME, timeframe, UTC range, and original chart symptom
  • source `snippets/nativedb_backfill_audit.xsh` and initialize a private disposable root with only the required provider config
  • run repository and environment preflight at commit `0846cbd4`
  • seed and audit the known-bad baseline, then replay repair and run `bfq_compare()` without losing original timestamps
  • clear only the disposable Parquet and qualify a fresh backfill
  • restart against fresh storage and compare `fresh` to `restart`
  • leave one subsequent run active long enough to append, then compare `restart` to `append` with `require_newer=True`
  • restart QQQ 1s on fresh datad SHM and verify the former 653-row zero-time reservation does not recur
  • preserve chart logs, audit JSON, Parquet snapshots, and exact UTC gap endpoints for review
TODO review ^qualify^ evidence and close follow-ups
  • verify every qualification process imports commit `0846cbd4`
  • review baseline, repair, fresh, restart, and append reports for structural, timestamp, endpoint, and gap regressions
  • distinguish durable Parquet gaps from SHM hydration or chart-only gaps using the runbook triage table
  • convert each reproduced failure into a deterministic regression before changing production code
  • prepare the final qualification summary and any required patch, PR, or worktree-lifecycle follow-ups

TODO cross-harness AI skill integration

TODO vectorize `gap_annotator` client spec building

  • replace per-row Polars slicing and filtering in `piker/tsp/_annotate.py:139` with extracted columns and batched lookups
  • preserve missing-prior-row, epoch-skip, direction, label, and batch-payload behavior
  • benchmark the 1285-gap case against the handoff in `notes_to_self/claudy/gapsannotator_sesh.md`
  • add focused correctness coverage for generated rectangle, arrow, and text specs

TODO integrate the piker `ruff` lint skill

  • review the clean but unmerged `claudy_writes_ruff_conf` branch
  • validate its `ruff.toml` policy and staged-file checker against the current tree
  • port the `/lint` skill to shared Agent Skills frontmatter with thin harness-specific commands
  • rebase and prepare the accepted lint/config changes for landing

TODO reconcile stale Claude plans and worktrees

  • reconcile `ai/claude-code/plans/shm_port.md` with the implemented `tractor.ipc._shm` shim
  • classify `datad_service.md` as an architecture record rather than an active implementation plan
  • inspect generated state before retiring the merged `wkt/flake_update` and `worktree-ngi0_submission` worktrees
  • preserve the unmerged lint worktree until its branch is reviewed

TODO chart-local gap overlay follow-ups

  • trigger `GapOverlayMngr.refresh()` from an explicit history revision/publication event after repair or prepend

  • expose gap-overlay UX through UI config

    • startup visibility and enabled timeframes
    • default `Ctrl+G` key binding

  • live-qualify chart-local overlay behavior

    • actor startup and focused 1s/60s toggling
    • multi-symbol and duplicate-FQME cached charts
    • secondary-`Viz` placement and backfill repositioning
    • remote disconnect and owner-scoped graphics cleanup
  • add focused Qt/actor regressions for the qualified lifecycle

TODO factor typed chart annotation IPC dialogs

  • exercise the strict `SetGapOverlay`/`GapOverlay` dialog under a live Tractor context

    • wrong-direction and unrelated payload rejection
    • timeout, cancellation, stale reply, and disconnect recovery
  • factor keyed send/receive locking, fresh request IDs, and stale response correlation into a generic IPC dialog helper after a second typed consumer establishes the interface
  • migrate legacy `remote_annotate()` dict commands to tagged request/response message types
  • decide whether to fold `remote_gap_overlays()` into the general annotation endpoint after the legacy protocol is typed

proj_mgmt

TODO ngi0 aug01 funding submit!

DEADLINE: <2026-08-01 Sat>

TODO MEETING(s)

ui

TODO dpi_auto_calc

  • PUT PRs here

brokers

TODO IB: follow-up for search results bs!

  • https://www.pikers.dev/pikers/piker/issues/85
  • fix the `si.expiry..ib` search result?

  • introspect why `Client.search_symbols()` is returning the empty `.<venue>` case!

    • presumably by something weird in the server-side endpoint?
  • figure out better tuning of client side fuzzy-filter versus server-side versus UI final display pass!

TODO ish for port to `uv`,

TODO profiling: try newer libs,

perf/tooling

  • link existing GH issues for trying new libs!

FSP backfill: branch work

TODO review and commit FSP backfill synchronization

  • review committed token-targeted history graphics change (`fadab3d2`)

  • review and commit the FSP stream-contract boundary

    • yo yo yo
    • `piker/data/__init__.py`
    • `piker/data/ticktools.py`
    • `piker/fsp/_api.py`
    • `piker/fsp/_momo.py`
    • staged portion of `piker/fsp/_volume.py`

  • review and commit the synchronization and live-volume boundary

    • explicit `Flume` RT/history cadence
    • immutable history snapshots and absolute SHM-bound synchronization
    • relevant backfill filtering and non-duplicating destination steps
    • nonnegative sampled and derived volume
    • M4 graphics-cache invalidation and unchanged-update suppression
    • FSP synchronization regressions and Prompt-IO provenance
  • review and commit the `piker-fsp-expert` skill boundary
  • run boundary checks and the final test suite
  • inspect each staged diff and commit message before committing

TODO add cumulative M4 downsampling for volume curves

  • define cumulative bucket semantics for step-style data

    • sum all source values represented by each pixel bucket
    • preserve bucket time/index placement and zero-volume gaps
  • add a sum-reducer variant alongside price min/max M4
  • route unit and dollar-volume curves through cumulative downsampling
  • keep price, OHLC, and non-cumulative signal curves on min/max M4
  • preserve incremental appends, view changes, and cache invalidation
  • test sum conservation across zoom levels and partial edge buckets
  • profile the cumulative path against the existing M4 implementation

TODO review and save the Trend Calculus prototype

Research and prototype guide

  • Regenerate and review six Trend Calculus behavior boundaries on trend_calc_on_jev after workflow commit f148fbcc on moar_ai_skillz lands; retain source/test pairs and inspect every message before saving.
  • Run each boundary check from the pinned package and the final safe regression sequence; the working-tree offline suite passed 253 tests.
  • Qualify one selected live source through startup, quiet bars, skipped wakeups, history repair, replay and owner cancellation.
  • Coordinate inherited stream/backfill work with the existing FSP synchronization task.

TODO split draft TC commit after upstream history is settled

Draft snapshot: e815fd77cfbf89b241bfabf19bda7b1c5f25c784.

  • Wait until upstream branches and their landing order are settled; preserve a reference to the draft snapshot.
  • Reset/unpack draft commit e815fd77 while retaining the full TC changeset and preserving unrelated working-tree and staged work.
  • Prepare a proper commit plan for that changeset with coherent behavior boundaries, matching tests, docs and provenance.
  • Run each boundary's checks and review its diff and message before recommitting.

TODO qualify Jev regime judgments against real responses

Research and prototype guide

  • Once TYPESAFE_API_KEY is available and a live request is authorized, send one small synthetic multi-timeframe request through the HTTPX/Trio client.
  • Verify the documented request/response schema, model identity, distributions, usage, errors and measured end-to-end latency.
  • Evaluate useful regime labels and confidence on held-out trend snapshots, including flat, reversing, aligned and conflicting timeframes.
  • Measure sustained pacing, coalescing, source freshness and cost before enabling realtime judgments.
  • Consider an SDK AnyIO port or Trio backend only after the direct HTTP prototype demonstrates useful classifications.

TODO connect finalized trends to the Jev provider-service

Research and prototype guide

  • Obtain the separate system-1 provider-service repository or interface and identify its input, output, lifecycle and ownership contracts.
  • Adapt serve_regimes and RegimeClassifier to the service nursery and typed publication interface without mixing model labels into deterministic trend state.
  • Configure market, periods, model, credentials, request pacing, evidence retention and freshness limits explicitly.
  • Verify reconnect, revision/gap invalidation, backpressure, stale-response rejection and shutdown with the actual service.

TODO recover Pine source and check Trend Calculus parity

Research and prototype guide

  • Recover the human's Pine implementation and record its provenance/version alongside the pinned Lua and Scala research.
  • Compare startup seeding, extrema ties, FHLS splitting, opposing pivots and recursive reversal orders.
  • Add independent fixtures for meaningful semantic differences; distinguish extremum time from confirmation time and mutable-bar previews.

TODO qualify multitimeframe pivots and regime annotations

Research and prototype guide

  • Inspect actual 1/5/15-second layers and hover information on a selected live market; compare confirmed pivots and regime boundaries to finalized source evidence.
  • Verify chart-axis placement, gaps, revisions, hidden charts, duplicate cached feeds and owner-scoped cleanup.
  • Qualify model rectangles at response receipt and expiry, including stale classifications and source-segment changes.
  • Measure quiet-bar display delay before deciding whether publication needs an explicit output notification.
  • Coordinate shared history-overlay lifecycle with the existing chart-local gap overlay follow-ups; route any annotation IPC factoring to the existing typed-dialog task.

TODO qualify a native subsecond source for trend regimes

Research and prototype guide

  • Define the required high-frequency periods and producer contract; the numeric/MTF APIs accept fractional periods but the current native clock uses whole seconds.
  • Extend or select the source/sampler time grid before enabling subsecond chart/service periods; retain strictly increasing coordinates and complete aligned buckets.
  • Measure compute, SHM publication, service latency and chart refresh under the selected cadence, including gaps, replay and cancellation.

TODO migrate clearing to strict typed messaging

Migration plan and commit boundaries

Foundation evidence: plan commit 1793b47c and implementation commit 7fee9b86; human reports both pushed. Strict schemas, validation, directional stream helpers and the pure OrderState reducer are implemented. Prior qualification: 65 offline tests and eight IPC tests passed. Production endpoints still use the legacy protocol.

  • Port paper and its paired EMS consumer together.

    • Typed startup, requests, events and open-order completion.
    • Deterministic synthetic ticks and distinct execution IDs.

  • Port Binance, Kraken and IB individually with paired EMS handling.

    • Shared offline conformance and provider translation fixtures.
    • Preserve execution identity, signed positions and account mapping.

  • Integrate OrderState into EMS routing and broker-event handling.

    • Bound pre-ack buffering by age in its asynchronous owner.
    • Preserve cancel intent, fill deduplication and late-event handling.

  • Migrate client/EMS startup, local queues and UI consumers together.

    • Replace wire snapshot mutation and dictionary reconstruction.

  • Enforce each migrated endpoint pair with Tractor payload specs.

    • Narrow receive sets; validate startup, RPC arguments and results.
    • Reject malformed, wrong-direction and wrong-phase messages.

  • Qualify real adapter and actor behavior offline.

    • Cancel-before-ack; early, duplicate and late fills.
    • Empty/populated startup snapshots and two-client order isolation.
    • Abrupt transport EOF, cancellation and bounded failure teardown.
    • Repeated lifecycle tests under supported spawn methods.
  • Resolve the duplicate pytest –ll option before a Tractor pin upgrade.
  • Document the position-broadcast isolation limitation and policy.
  • Remove legacy IPC paths after all paired migrations qualify.
  • Propose a reviewable commit plan for each qualified implementation.