diff --git a/.agents/skills/piker-fsp-expert/SKILL.md b/.agents/skills/piker-fsp-expert/SKILL.md new file mode 100644 index 00000000..6fffa8cc --- /dev/null +++ b/.agents/skills/piker-fsp-expert/SKILL.md @@ -0,0 +1,78 @@ +--- +name: piker-fsp-expert +description: Piker financial signal processing expertise. Apply when changing or debugging FSP definitions, Cascade execution, derived shared-memory arrays, history/realtime handoff, backfill synchronization, FSP graph composition, online financial algorithms, or FSP chart lifecycle and scaling. +--- + +# Piker FSP Expertise + +Treat financial signal processing as causal, revision-aware processing of +event-time market data, not as generic batch array manipulation. + +## Start Here + +1. Identify the source `Flume`, timeframe, event clock, and writer actor. +2. State whether the operation is historical bootstrap, backfill revision, + current-sample mutation, or sample-step append. +3. Record source and destination absolute SHM bounds before reasoning from + lengths or `ShmArray.index`. +4. Separate compute, SHM publication, IPC wakeup, and graphics costs before + optimizing. +5. Preserve causality, warm-up behavior, missing-data semantics, and output + schema in every implementation and test. + +Read [architecture.md](architecture.md) before modifying runtime code. Read +[signal-processing.md](signal-processing.md) before adding an algorithm. Use +[roadmap-and-research.md](roadmap-and-research.md) for design and dependency +work, and [test-map.md](test-map.md) for verification. + +## Core Contracts + +- `piker.data` owns provider ingestion, source `Flume`s, history publication, + quote fan-out, OHLC mutation, and `samplerd` step events. +- `piker.fsp` owns operator declarations, historical computation, realtime + state updates, and destination SHM publication. +- `piker.ui._fsp.FspAdmin` currently owns FSP worker selection, destination + allocation, cascade lifetime, and chart attachment. +- One logical writer owns each destination FSP array. +- Output event time and absolute indices derive from the source series, not + task scheduling or quote arrival order. +- A source prepend changes historical provenance even when the current sample + index is unchanged. Recompute or patch the affected destination range. +- One-step destination lag is normal between source sampling and destination + append. A destination lead or unexplained bound shift is not. +- Backfill notifications are source and timeframe specific even though the + current sampler transport broadcasts them globally. +- Only graphics backed by a rebuilt destination should refresh. Never use an + FSP update as an unconditional whole-chart redraw request. + +## Editing Workflow + +1. Trace the operator from `@fsp` declaration through `cascade()`, + `connect_streams()`, destination SHM, `FspAdmin`, and every consuming + `Viz`. +2. Write an event-order table covering source writes, bound publication, + notifications, cancellation, recomputation, and destination publication. +3. Add deterministic tests for history and realtime phases independently. +4. Add a controlled interleaving test for every race fix. +5. Profile representative visible ranges and feed rates before adding a + dependency or native boundary. + +## Companion Skills + +- Use `piker-conc-expert` for Tractor contexts, actor topology, cancellation, + service ownership, and cross-actor state. +- Use `timeseries-optimization` for NumPy/Polars lookup and vectorization. +- Use `piker-profiling` for distributed performance measurements. +- Use `pyqtgraph-optimization` for renderer and graphics batching work. +- Use `piker-clearing-expert` when a derived signal reaches EMS or accounting. + +## Do Not + +- Do not hide a historical mismatch merely to stop a resync log. +- Do not infer a complete synchronization state from array length alone. +- Do not run Python loops over historical numeric arrays without evidence. +- Do not apply regular-grid DSP blindly to irregular tick arrival times. +- Do not introduce an external stream runtime that duplicates Tractor before + measuring an isolated protocol or kernel prototype. +- Do not route strategy signals directly around EMS order-state, risk, or + accounting contracts. diff --git a/.agents/skills/piker-fsp-expert/architecture.md b/.agents/skills/piker-fsp-expert/architecture.md new file mode 100644 index 00000000..cf49a25c --- /dev/null +++ b/.agents/skills/piker-fsp-expert/architecture.md @@ -0,0 +1,117 @@ +# FSP Runtime Architecture + +## Current Topology + +```text +provider backend + -> datad feed bus + -> source Flume + -> rt ShmArray (typically 1 second) + -> hist ShmArray (typically 60 seconds) + -> samplerd sample and backfill events + +chart actor + -> FspAdmin + -> chart.fsp_N worker actor + -> cascade() + -> connect_streams() + -> destination Flume / ShmArray + -> targeted Viz updates +``` + +The source path is implemented mainly by `piker.data.feed`, +`piker.data.flows`, `piker.data._sampling`, and `piker.tsp._history`. The FSP +path is implemented by `piker.fsp._api`, `piker.fsp._engine`, and operator +modules such as `_volume` and `_momo`. Worker and graphics lifecycle currently +live in `piker.ui._fsp` and `piker.ui._display`. + +## Operator Contract + +`@fsp` wraps an async generator. Its first yield is historical output computed +from the source array. Later yields are `(field, value)` realtime mutations. +`connect_streams()` converts the first yield into the destination structured +array, aligns time and absolute bounds, then updates the current destination +row from realtime quote events. + +`samplerd` owns row advancement. At a new source sample, the cascade copies or +zeros the previous destination row and aligns destination timestamps to the +source. Quote-time mutations and sample-time appends are distinct operations. + +## SHM Synchronization + +Treat each array as an absolute half-open range: + +```text +[first, last) +``` + +For a normal source history prepend: + +```text +before: src=[10000, 12000) dst=[10000, 12000) +after: src=[ 8000, 12000) dst=[10000, 12000) +``` + +The current step is unchanged, but destination history is missing 2,000 rows. +The destination must be recomputed or incrementally repaired. + +For a normal one-step source lead: + +```text +src=[8000, 12001) dst=[8000, 12000) +``` + +The cascade should append exactly one destination row, not recompute history. + +Use `_first.value` and `_last.value` for synchronization. `ShmArray.index` is +the last bound modulo capacity and is not a complete absolute position. +Snapshot bounds once per check; shared first and last counters do not form a +transactional pair. + +## Backfill Event Ordering + +1. `start_backfill()` receives an older provider frame. +2. `shm_push_in_between()` writes the frame and publishes the earlier first + bound. +3. `notify_backfill()` asks `samplerd` to broadcast the market and timeframe. +4. Relevant cascades treat the event as a history revision. A prepend changes + bounds; an in-place gap repair may not. +5. Each cascade cancels its current compute task and waits for completion. +6. The historical phase recomputes against the newest readable source range. +7. Destination bounds are published and the UI receives an `fsp_update`. +8. Only `Viz` objects backed by that destination token redraw. + +Multiple provider frames may arrive while a recomputation runs. The cascade +must reject mixed-bound bootstrap results, replay in-place revisions, recheck +bounds after restart, and converge on the newest source range. + +## Actor-Local State + +`Fsp._flow_registry`, attached SHM handles, caches, and nurseries are local to +each actor. Never assume worker siblings share warmed registries or Python +objects. Graph dependencies currently rely on startup order, as demonstrated +by `flow_rates` starting only after `dolla_vlm`. + +## Cross-Subsystem Boundaries + +- Data ingest should normalize identity and event-time semantics before an FSP + consumes a stream. +- UI renderers should consume revision/range information without controlling + computation correctness. +- Clearing may consume a derived signal, but EMS remains responsible for order + intent, status, routing, fills, positions, and accounting. +- FSP output exposed as a feed needs the same fan-out, identity, lifecycle, + and backpressure contracts as provider feeds. + +## Architectural Gaps + +The current protocol infers history revisions from shared bounds. A future +revision message should carry at least: + +```text +(fqme, timeframe, revision, first, last, start_ts, end_ts) +``` + +Destination publication should identify the source revision used. This allows +coalescing, range repair, stale-result rejection, and deterministic graph +dependency startup without turning every prepend into a global wakeup. diff --git a/.agents/skills/piker-fsp-expert/roadmap-and-research.md b/.agents/skills/piker-fsp-expert/roadmap-and-research.md new file mode 100644 index 00000000..f6363289 --- /dev/null +++ b/.agents/skills/piker-fsp-expert/roadmap-and-research.md @@ -0,0 +1,104 @@ +# Roadmap and Research + +Issue states change. Re-read each tracker before planning work and preserve +human ownership of issue and checklist state. + +## Guiding Roadmap + +- [piker #270](https://github.com/pikers/piker/issues/270): typed, + stream-composable FSP graph and DSL direction. +- [piker #107](https://github.com/pikers/piker/issues/107): cross-actor shared + memory tick/ring-buffer direction. +- [piker #216](https://github.com/pikers/piker/issues/216): exposing FSP output + through feed-like interfaces. +- [piker #325](https://github.com/pikers/piker/issues/325): faster streaming + y-range computation and overlap with FSP reductions. +- [piker #536](https://github.com/pikers/piker/issues/536): Arrow/Parquet data + ingest and columnar boundaries. +- [piker #98](https://github.com/pikers/piker/issues/98): older realtime feed + architecture discussion. Treat it as likely superseded; compare every idea + against current `Flume`, feed-bus, history, and sampler code. +- [Tractor #339](https://github.com/goodboy/tractor/issues/339): shared-memory + array and localhost IPC evolution. +- [hotbaud issues](https://github.com/guilledk/hotbaud/issues): experimental + high-throughput local transport and ring-buffer work. + +## Near-Term Protocol Research + +Prototype an explicit source revision envelope and measure it with two FSPs +and two overlaid feeds. The prototype should prove source-specific wakeups, +coalescing of consecutive prepends, stale-result rejection, and targeted UI +range updates. Keep the protocol independent of the final IPC transport. + +Characterize historical repair strategies: + +1. Full replay from the new first bound. +2. Prefix-only calculation for pointwise operators. +3. Replay from an operator checkpoint for stateful recurrences. +4. Bounded overlap repair for finite-window operators. + +An FSP declaration eventually needs repair metadata describing locality, +state checkpointability, warm-up/overlap, and whether old inputs can alter all +later outputs. + +## Candidate Libraries + +Candidates are experiments, not dependency recommendations. + +### Bounded Near-Term Prototypes + +- [Bottleneck](https://github.com/pydata/bottleneck): compare rolling min/max, + rank, and moving-window kernels against NumPy/Numba for chart y-ranges and + finite-window FSPs. +- [River](https://github.com/online-ml/river): study stateful online estimator + APIs, drift detectors, and reference semantics. Do not assume its object + model belongs in the hottest tick path. +- [nanoarrow](https://github.com/apache/arrow-nanoarrow): evaluate lightweight + Arrow C data interfaces at native/process boundaries already motivated by + `pyarrow` and piker #536. +- [DuckDB](https://github.com/duckdb/duckdb): use for historical Parquet query, + replay slicing, and validation, outside realtime tick execution. + +### Watchlist + +- [hotbaud](https://github.com/guilledk/hotbaud): track API and correctness + maturity for localhost SHM/ring transport. +- [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2): study zero-copy + pub/sub ownership and lifecycle, but avoid competing with Tractor without a + narrow adapter benchmark. +- [DataFusion Python](https://github.com/apache/datafusion-python): Arrow-native + historical query and expression execution if existing Polars paths cannot + meet a demonstrated need. +- [Feldera](https://github.com/feldera/feldera): study DBSP/incremental-view + semantics for revision-aware graphs; its service/SQL runtime is not a direct + piker integration target. +- [Bytewax](https://github.com/bytewax/bytewax) and + [Arroyo](https://github.com/ArroyoSystems/arroyo): study dataflow semantics, + watermarking, and recovery, not as replacements for the actor tree. + +GPU stacks such as JAX and CuPy are poor defaults for low-batch per-tick work +because transfer, compilation, and scheduling can dominate. Reconsider only +for measured large-batch research or many-market matrix workloads. + +## Benchmark Design + +Report separate distributions for: + +- provider decode and normalization; +- source SHM write and bound publication; +- historical kernel and realtime recurrence; +- actor notification and stream fan-out; +- destination publication; +- visible-range formatting and Qt paint. + +Use p50, p95, p99, maximum, allocation rate, and resident-memory growth. Test +steady state, burst load, backfill overlap, cancellation, and slow consumers. +Throughput without event-time correctness and bounded teardown is not a win. + +## Medium-Term Sequence + +First establish revisioned range messages and operator repair semantics. Then +decouple FSP orchestration from chart ownership and expose destination flows +through feed-like subscriptions. Next benchmark localhost transport adapters +behind the same protocol. Only after those contracts stabilize should graph +placement, fusion, worker scaling, and remote partitioning be automated. diff --git a/.agents/skills/piker-fsp-expert/signal-processing.md b/.agents/skills/piker-fsp-expert/signal-processing.md new file mode 100644 index 00000000..a1f42aca --- /dev/null +++ b/.agents/skills/piker-fsp-expert/signal-processing.md @@ -0,0 +1,103 @@ +# Financial Signal Processing + +## Model the Clock First + +Market inputs can be quote events, trades, book changes, fixed-time bars, or +revised historical bars. Name both event time and processing time. For +irregular events, either use time-aware recurrences or define the resampling +and interpolation policy explicitly. + +Never let backfilled information leak into a value presented as historically +tradable. Distinguish corrected analysis history from the signal values that a +live strategy could have observed. + +## Useful Technique Families + +### Online Moments and Robust Statistics + +Use Welford/Chan recurrences for stable online moments, compensated summation +for long accumulations, and mergeable summaries when partitioning actors. +Rolling median, quantiles, MAD, and winsorized measures are useful under heavy +tails, but exact rolling order statistics need more state than moments. + +### Time-Aware Exponential Filters + +For irregular event intervals, derive decay from elapsed time: + +```text +alpha(dt) = 1 - exp(-dt / tau) +y_t = y_prev + alpha(dt) * (x_t - y_prev) +``` + +Record initialization, session reset, gap, and backfill behavior. A fixed +sample alpha is valid only after an explicit regularization policy. + +### State-Space Models + +Kalman and robust state-space filters fit latent price, spread, volatility, +and lead/lag estimates. Prefer square-root or Joseph-form covariance updates +when conditioning is poor. State revisions after backfill require replay from +a checkpoint or a bounded smoother; silently prepending observations without +replaying state is incorrect. + +### Point Processes and Microstructure + +Trade and quote arrivals are events, not merely sampled amplitudes. Consider +intensity, duration, signed order flow, imbalance, spread, queue change, and +self-excitation models. Validate against provider-specific aggregation and +duplicate/out-of-order behavior before interpreting a statistic. + +### Volatility and Covariance + +Realized variance, bipower variation, EW covariance, and range estimators can +be updated online. Asynchronous cross-market covariance needs synchronization +or estimators designed for non-synchronous observations; naive row alignment +creates lead/lag and Epps-effect artifacts. + +### Spectral and Multiscale Methods + +FFT methods assume a regular grid and a window. Declare detrending, tapering, +overlap, normalization, and latency. For tick data, resample deliberately or +use irregular-time methods. Wavelets and multiresolution filters can separate +horizons, but boundary handling and causal delay must be visible to strategy +code. + +### Change and Anomaly Detection + +CUSUM, Page-Hinkley, sequential likelihood ratios, and robust z-scores are +cheap online tools. Calibrate false alarms under dependence and regime shifts; +do not treat IID thresholds as market guarantees. + +## Numerical Implementation Order + +1. Establish a scalar reference with explicit state and semantics. +2. Vectorize historical bootstrap with NumPy. +3. Keep realtime updates as O(1) recurrences where possible. +4. Use Numba for measured numeric kernels with stable dtypes and no Python + object traffic. +5. Use Polars or Arrow for columnar historical transforms and interchange, + not automatically for each tick. +6. Partition by independent market/operator state before adding threads or a + new distributed runtime. + +## Existing Stack + +- NumPy 2.x: canonical dense and structured-array kernels. +- Numba: compiled CPU loops and recurrences that do not vectorize cleanly. +- Polars: parallel/lazy columnar history preparation and validation. +- PyArrow: columnar interchange, storage, and ingest boundaries. +- Tractor/Trio: structured distributed execution and lifecycle. +- PyQtGraph: visible-range rendering, not a compute scheduler. + +## Algorithm Acceptance Criteria + +Every new FSP should define: + +- input fields, clock, ordering, duplicate, gap, and revision policy; +- output dtype, units, identity, and parameterization; +- warm-up length and initialization bias; +- causal latency and strategy-visible publication point; +- historical/realtime equivalence tolerance; +- reset and session-boundary behavior; +- complexity, state size, and representative throughput; +- replay behavior when historical source data changes. diff --git a/.agents/skills/piker-fsp-expert/test-map.md b/.agents/skills/piker-fsp-expert/test-map.md new file mode 100644 index 00000000..63af82a7 --- /dev/null +++ b/.agents/skills/piker-fsp-expert/test-map.md @@ -0,0 +1,52 @@ +# FSP Test Map + +## Operator Tests + +- Historical output schema, dtype, units, and source timestamp alignment. +- Realtime update equivalence with the final row of a batch recomputation. +- Warm-up, NaN, zero, gap, duplicate, and out-of-order behavior. +- Session reset and elapsed-time behavior. +- Parameter and source identity isolation. + +## Engine Tests + +- Equal absolute source/destination bounds. +- Normal one-step destination lag. +- Destination lead and lag greater than one step. +- Source prepend by one, two, and a full provider frame. +- In-place gap repair with unchanged source bounds. +- Equal lengths with shifted absolute bounds. +- Source change during historical recomputation and convergence afterward. +- Cancellation while the quote stream, sample stream, or compute generator is + blocked. +- Foreign FQME and timeframe backfill broadcasts. + +## Graph and Actor Tests + +- Dependency startup before downstream operator lookup. +- Actor-local flow registry isolation. +- One destination writer and multiple readers. +- Consumer cancellation without orphaned cascade or worker tasks. +- Worker failure propagation and SHM cleanup. +- Feed-like fan-out with a slow or disconnected consumer. + +## UI Tests + +- An `fsp_update` redraws every field visualization sharing its destination + token and no unrelated market, OHLC, or FSP visualization. +- A prepend outside the visible range does not move or re-range the live view. +- A revised value inside the visible range invalidates only required formatter + and graphics caches. +- Hidden charts defer rendering without stopping compute correctness. + +## Performance Tests + +- Historical bootstrap across realistic SHM capacities. +- O(1) realtime recurrence cost under bursty quotes. +- Consecutive provider-frame prepends with two or more cascades. +- Multi-market sampler broadcasts and targeted wakeups. +- Visible-range redraw cost at several units-per-pixel levels. + +Prefer deterministic local arrays and controlled Trio task interleavings. +Use live providers only as a separate integration layer because credentials, +rate limits, sessions, and changing history make poor regression oracles. diff --git a/.claude/skills/piker-fsp-expert b/.claude/skills/piker-fsp-expert new file mode 120000 index 00000000..c5853b51 --- /dev/null +++ b/.claude/skills/piker-fsp-expert @@ -0,0 +1 @@ +../../.agents/skills/piker-fsp-expert \ No newline at end of file