Add `piker-fsp-expert` skill
Document FSP lifecycle, SHM publication, backfill ordering, causal signal-processing semantics, and deterministic tests. Record research directions separately from implementation and expose the package through the relative Claude skill symlink. (this commit msg was generated in some part by `codex` using `gpt-6` (`openai`))wkt/fsp_backfill_sync
parent
224399147d
commit
749ca1f09c
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
name: piker-fsp-expert
|
||||
description: Piker financial signal processing expertise. Apply when changing or debugging FSP definitions, Cascade execution, derived shared-memory arrays, history/realtime handoff, backfill synchronization, FSP graph composition, online financial algorithms, or FSP chart lifecycle and scaling.
|
||||
---
|
||||
|
||||
# Piker FSP Expertise
|
||||
|
||||
Treat financial signal processing as causal, revision-aware processing of
|
||||
event-time market data, not as generic batch array manipulation.
|
||||
|
||||
## Start Here
|
||||
|
||||
1. Identify the source `Flume`, timeframe, event clock, and writer actor.
|
||||
2. State whether the operation is historical bootstrap, backfill revision,
|
||||
current-sample mutation, or sample-step append.
|
||||
3. Record source and destination absolute SHM bounds before reasoning from
|
||||
lengths or `ShmArray.index`.
|
||||
4. Separate compute, SHM publication, IPC wakeup, and graphics costs before
|
||||
optimizing.
|
||||
5. Preserve causality, warm-up behavior, missing-data semantics, and output
|
||||
schema in every implementation and test.
|
||||
|
||||
Read [architecture.md](architecture.md) before modifying runtime code. Read
|
||||
[signal-processing.md](signal-processing.md) before adding an algorithm. Use
|
||||
[roadmap-and-research.md](roadmap-and-research.md) for design and dependency
|
||||
work, and [test-map.md](test-map.md) for verification.
|
||||
|
||||
## Core Contracts
|
||||
|
||||
- `piker.data` owns provider ingestion, source `Flume`s, history publication,
|
||||
quote fan-out, OHLC mutation, and `samplerd` step events.
|
||||
- `piker.fsp` owns operator declarations, historical computation, realtime
|
||||
state updates, and destination SHM publication.
|
||||
- `piker.ui._fsp.FspAdmin` currently owns FSP worker selection, destination
|
||||
allocation, cascade lifetime, and chart attachment.
|
||||
- One logical writer owns each destination FSP array.
|
||||
- Output event time and absolute indices derive from the source series, not
|
||||
task scheduling or quote arrival order.
|
||||
- A source prepend changes historical provenance even when the current sample
|
||||
index is unchanged. Recompute or patch the affected destination range.
|
||||
- One-step destination lag is normal between source sampling and destination
|
||||
append. A destination lead or unexplained bound shift is not.
|
||||
- Backfill notifications are source and timeframe specific even though the
|
||||
current sampler transport broadcasts them globally.
|
||||
- Only graphics backed by a rebuilt destination should refresh. Never use an
|
||||
FSP update as an unconditional whole-chart redraw request.
|
||||
|
||||
## Editing Workflow
|
||||
|
||||
1. Trace the operator from `@fsp` declaration through `cascade()`,
|
||||
`connect_streams()`, destination SHM, `FspAdmin`, and every consuming
|
||||
`Viz`.
|
||||
2. Write an event-order table covering source writes, bound publication,
|
||||
notifications, cancellation, recomputation, and destination publication.
|
||||
3. Add deterministic tests for history and realtime phases independently.
|
||||
4. Add a controlled interleaving test for every race fix.
|
||||
5. Profile representative visible ranges and feed rates before adding a
|
||||
dependency or native boundary.
|
||||
|
||||
## Companion Skills
|
||||
|
||||
- Use `piker-conc-expert` for Tractor contexts, actor topology, cancellation,
|
||||
service ownership, and cross-actor state.
|
||||
- Use `timeseries-optimization` for NumPy/Polars lookup and vectorization.
|
||||
- Use `piker-profiling` for distributed performance measurements.
|
||||
- Use `pyqtgraph-optimization` for renderer and graphics batching work.
|
||||
- Use `piker-clearing-expert` when a derived signal reaches EMS or accounting.
|
||||
|
||||
## Do Not
|
||||
|
||||
- Do not hide a historical mismatch merely to stop a resync log.
|
||||
- Do not infer a complete synchronization state from array length alone.
|
||||
- Do not run Python loops over historical numeric arrays without evidence.
|
||||
- Do not apply regular-grid DSP blindly to irregular tick arrival times.
|
||||
- Do not introduce an external stream runtime that duplicates Tractor before
|
||||
measuring an isolated protocol or kernel prototype.
|
||||
- Do not route strategy signals directly around EMS order-state, risk, or
|
||||
accounting contracts.
|
||||
|
|
@ -0,0 +1,117 @@
|
|||
# FSP Runtime Architecture
|
||||
|
||||
## Current Topology
|
||||
|
||||
```text
|
||||
provider backend
|
||||
-> datad feed bus
|
||||
-> source Flume
|
||||
-> rt ShmArray (typically 1 second)
|
||||
-> hist ShmArray (typically 60 seconds)
|
||||
-> samplerd sample and backfill events
|
||||
|
||||
chart actor
|
||||
-> FspAdmin
|
||||
-> chart.fsp_N worker actor
|
||||
-> cascade()
|
||||
-> connect_streams()
|
||||
-> destination Flume / ShmArray
|
||||
-> targeted Viz updates
|
||||
```
|
||||
|
||||
The source path is implemented mainly by `piker.data.feed`,
|
||||
`piker.data.flows`, `piker.data._sampling`, and `piker.tsp._history`. The FSP
|
||||
path is implemented by `piker.fsp._api`, `piker.fsp._engine`, and operator
|
||||
modules such as `_volume` and `_momo`. Worker and graphics lifecycle currently
|
||||
live in `piker.ui._fsp` and `piker.ui._display`.
|
||||
|
||||
## Operator Contract
|
||||
|
||||
`@fsp` wraps an async generator. Its first yield is historical output computed
|
||||
from the source array. Later yields are `(field, value)` realtime mutations.
|
||||
`connect_streams()` converts the first yield into the destination structured
|
||||
array, aligns time and absolute bounds, then updates the current destination
|
||||
row from realtime quote events.
|
||||
|
||||
`samplerd` owns row advancement. At a new source sample, the cascade copies or
|
||||
zeros the previous destination row and aligns destination timestamps to the
|
||||
source. Quote-time mutations and sample-time appends are distinct operations.
|
||||
|
||||
## SHM Synchronization
|
||||
|
||||
Treat each array as an absolute half-open range:
|
||||
|
||||
```text
|
||||
[first, last)
|
||||
```
|
||||
|
||||
For a normal source history prepend:
|
||||
|
||||
```text
|
||||
before: src=[10000, 12000) dst=[10000, 12000)
|
||||
after: src=[ 8000, 12000) dst=[10000, 12000)
|
||||
```
|
||||
|
||||
The current step is unchanged, but destination history is missing 2,000 rows.
|
||||
The destination must be recomputed or incrementally repaired.
|
||||
|
||||
For a normal one-step source lead:
|
||||
|
||||
```text
|
||||
src=[8000, 12001) dst=[8000, 12000)
|
||||
```
|
||||
|
||||
The cascade should append exactly one destination row, not recompute history.
|
||||
|
||||
Use `_first.value` and `_last.value` for synchronization. `ShmArray.index` is
|
||||
the last bound modulo capacity and is not a complete absolute position.
|
||||
Snapshot bounds once per check; shared first and last counters do not form a
|
||||
transactional pair.
|
||||
|
||||
## Backfill Event Ordering
|
||||
|
||||
1. `start_backfill()` receives an older provider frame.
|
||||
2. `shm_push_in_between()` writes the frame and publishes the earlier first
|
||||
bound.
|
||||
3. `notify_backfill()` asks `samplerd` to broadcast the market and timeframe.
|
||||
4. Relevant cascades treat the event as a history revision. A prepend changes
|
||||
bounds; an in-place gap repair may not.
|
||||
5. Each cascade cancels its current compute task and waits for completion.
|
||||
6. The historical phase recomputes against the newest readable source range.
|
||||
7. Destination bounds are published and the UI receives an `fsp_update`.
|
||||
8. Only `Viz` objects backed by that destination token redraw.
|
||||
|
||||
Multiple provider frames may arrive while a recomputation runs. The cascade
|
||||
must reject mixed-bound bootstrap results, replay in-place revisions, recheck
|
||||
bounds after restart, and converge on the newest source range.
|
||||
|
||||
## Actor-Local State
|
||||
|
||||
`Fsp._flow_registry`, attached SHM handles, caches, and nurseries are local to
|
||||
each actor. Never assume worker siblings share warmed registries or Python
|
||||
objects. Graph dependencies currently rely on startup order, as demonstrated
|
||||
by `flow_rates` starting only after `dolla_vlm`.
|
||||
|
||||
## Cross-Subsystem Boundaries
|
||||
|
||||
- Data ingest should normalize identity and event-time semantics before an FSP
|
||||
consumes a stream.
|
||||
- UI renderers should consume revision/range information without controlling
|
||||
computation correctness.
|
||||
- Clearing may consume a derived signal, but EMS remains responsible for order
|
||||
intent, status, routing, fills, positions, and accounting.
|
||||
- FSP output exposed as a feed needs the same fan-out, identity, lifecycle,
|
||||
and backpressure contracts as provider feeds.
|
||||
|
||||
## Architectural Gaps
|
||||
|
||||
The current protocol infers history revisions from shared bounds. A future
|
||||
revision message should carry at least:
|
||||
|
||||
```text
|
||||
(fqme, timeframe, revision, first, last, start_ts, end_ts)
|
||||
```
|
||||
|
||||
Destination publication should identify the source revision used. This allows
|
||||
coalescing, range repair, stale-result rejection, and deterministic graph
|
||||
dependency startup without turning every prepend into a global wakeup.
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
# Roadmap and Research
|
||||
|
||||
Issue states change. Re-read each tracker before planning work and preserve
|
||||
human ownership of issue and checklist state.
|
||||
|
||||
## Guiding Roadmap
|
||||
|
||||
- [piker #270](https://github.com/pikers/piker/issues/270): typed,
|
||||
stream-composable FSP graph and DSL direction.
|
||||
- [piker #107](https://github.com/pikers/piker/issues/107): cross-actor shared
|
||||
memory tick/ring-buffer direction.
|
||||
- [piker #216](https://github.com/pikers/piker/issues/216): exposing FSP output
|
||||
through feed-like interfaces.
|
||||
- [piker #325](https://github.com/pikers/piker/issues/325): faster streaming
|
||||
y-range computation and overlap with FSP reductions.
|
||||
- [piker #536](https://github.com/pikers/piker/issues/536): Arrow/Parquet data
|
||||
ingest and columnar boundaries.
|
||||
- [piker #98](https://github.com/pikers/piker/issues/98): older realtime feed
|
||||
architecture discussion. Treat it as likely superseded; compare every idea
|
||||
against current `Flume`, feed-bus, history, and sampler code.
|
||||
- [Tractor #339](https://github.com/goodboy/tractor/issues/339): shared-memory
|
||||
array and localhost IPC evolution.
|
||||
- [hotbaud issues](https://github.com/guilledk/hotbaud/issues): experimental
|
||||
high-throughput local transport and ring-buffer work.
|
||||
|
||||
## Near-Term Protocol Research
|
||||
|
||||
Prototype an explicit source revision envelope and measure it with two FSPs
|
||||
and two overlaid feeds. The prototype should prove source-specific wakeups,
|
||||
coalescing of consecutive prepends, stale-result rejection, and targeted UI
|
||||
range updates. Keep the protocol independent of the final IPC transport.
|
||||
|
||||
Characterize historical repair strategies:
|
||||
|
||||
1. Full replay from the new first bound.
|
||||
2. Prefix-only calculation for pointwise operators.
|
||||
3. Replay from an operator checkpoint for stateful recurrences.
|
||||
4. Bounded overlap repair for finite-window operators.
|
||||
|
||||
An FSP declaration eventually needs repair metadata describing locality,
|
||||
state checkpointability, warm-up/overlap, and whether old inputs can alter all
|
||||
later outputs.
|
||||
|
||||
## Candidate Libraries
|
||||
|
||||
Candidates are experiments, not dependency recommendations.
|
||||
|
||||
### Bounded Near-Term Prototypes
|
||||
|
||||
- [Bottleneck](https://github.com/pydata/bottleneck): compare rolling min/max,
|
||||
rank, and moving-window kernels against NumPy/Numba for chart y-ranges and
|
||||
finite-window FSPs.
|
||||
- [River](https://github.com/online-ml/river): study stateful online estimator
|
||||
APIs, drift detectors, and reference semantics. Do not assume its object
|
||||
model belongs in the hottest tick path.
|
||||
- [nanoarrow](https://github.com/apache/arrow-nanoarrow): evaluate lightweight
|
||||
Arrow C data interfaces at native/process boundaries already motivated by
|
||||
`pyarrow` and piker #536.
|
||||
- [DuckDB](https://github.com/duckdb/duckdb): use for historical Parquet query,
|
||||
replay slicing, and validation, outside realtime tick execution.
|
||||
|
||||
### Watchlist
|
||||
|
||||
- [hotbaud](https://github.com/guilledk/hotbaud): track API and correctness
|
||||
maturity for localhost SHM/ring transport.
|
||||
- [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2): study zero-copy
|
||||
pub/sub ownership and lifecycle, but avoid competing with Tractor without a
|
||||
narrow adapter benchmark.
|
||||
- [DataFusion Python](https://github.com/apache/datafusion-python): Arrow-native
|
||||
historical query and expression execution if existing Polars paths cannot
|
||||
meet a demonstrated need.
|
||||
- [Feldera](https://github.com/feldera/feldera): study DBSP/incremental-view
|
||||
semantics for revision-aware graphs; its service/SQL runtime is not a direct
|
||||
piker integration target.
|
||||
- [Bytewax](https://github.com/bytewax/bytewax) and
|
||||
[Arroyo](https://github.com/ArroyoSystems/arroyo): study dataflow semantics,
|
||||
watermarking, and recovery, not as replacements for the actor tree.
|
||||
|
||||
GPU stacks such as JAX and CuPy are poor defaults for low-batch per-tick work
|
||||
because transfer, compilation, and scheduling can dominate. Reconsider only
|
||||
for measured large-batch research or many-market matrix workloads.
|
||||
|
||||
## Benchmark Design
|
||||
|
||||
Report separate distributions for:
|
||||
|
||||
- provider decode and normalization;
|
||||
- source SHM write and bound publication;
|
||||
- historical kernel and realtime recurrence;
|
||||
- actor notification and stream fan-out;
|
||||
- destination publication;
|
||||
- visible-range formatting and Qt paint.
|
||||
|
||||
Use p50, p95, p99, maximum, allocation rate, and resident-memory growth. Test
|
||||
steady state, burst load, backfill overlap, cancellation, and slow consumers.
|
||||
Throughput without event-time correctness and bounded teardown is not a win.
|
||||
|
||||
## Medium-Term Sequence
|
||||
|
||||
First establish revisioned range messages and operator repair semantics. Then
|
||||
decouple FSP orchestration from chart ownership and expose destination flows
|
||||
through feed-like subscriptions. Next benchmark localhost transport adapters
|
||||
behind the same protocol. Only after those contracts stabilize should graph
|
||||
placement, fusion, worker scaling, and remote partitioning be automated.
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
# Financial Signal Processing
|
||||
|
||||
## Model the Clock First
|
||||
|
||||
Market inputs can be quote events, trades, book changes, fixed-time bars, or
|
||||
revised historical bars. Name both event time and processing time. For
|
||||
irregular events, either use time-aware recurrences or define the resampling
|
||||
and interpolation policy explicitly.
|
||||
|
||||
Never let backfilled information leak into a value presented as historically
|
||||
tradable. Distinguish corrected analysis history from the signal values that a
|
||||
live strategy could have observed.
|
||||
|
||||
## Useful Technique Families
|
||||
|
||||
### Online Moments and Robust Statistics
|
||||
|
||||
Use Welford/Chan recurrences for stable online moments, compensated summation
|
||||
for long accumulations, and mergeable summaries when partitioning actors.
|
||||
Rolling median, quantiles, MAD, and winsorized measures are useful under heavy
|
||||
tails, but exact rolling order statistics need more state than moments.
|
||||
|
||||
### Time-Aware Exponential Filters
|
||||
|
||||
For irregular event intervals, derive decay from elapsed time:
|
||||
|
||||
```text
|
||||
alpha(dt) = 1 - exp(-dt / tau)
|
||||
y_t = y_prev + alpha(dt) * (x_t - y_prev)
|
||||
```
|
||||
|
||||
Record initialization, session reset, gap, and backfill behavior. A fixed
|
||||
sample alpha is valid only after an explicit regularization policy.
|
||||
|
||||
### State-Space Models
|
||||
|
||||
Kalman and robust state-space filters fit latent price, spread, volatility,
|
||||
and lead/lag estimates. Prefer square-root or Joseph-form covariance updates
|
||||
when conditioning is poor. State revisions after backfill require replay from
|
||||
a checkpoint or a bounded smoother; silently prepending observations without
|
||||
replaying state is incorrect.
|
||||
|
||||
### Point Processes and Microstructure
|
||||
|
||||
Trade and quote arrivals are events, not merely sampled amplitudes. Consider
|
||||
intensity, duration, signed order flow, imbalance, spread, queue change, and
|
||||
self-excitation models. Validate against provider-specific aggregation and
|
||||
duplicate/out-of-order behavior before interpreting a statistic.
|
||||
|
||||
### Volatility and Covariance
|
||||
|
||||
Realized variance, bipower variation, EW covariance, and range estimators can
|
||||
be updated online. Asynchronous cross-market covariance needs synchronization
|
||||
or estimators designed for non-synchronous observations; naive row alignment
|
||||
creates lead/lag and Epps-effect artifacts.
|
||||
|
||||
### Spectral and Multiscale Methods
|
||||
|
||||
FFT methods assume a regular grid and a window. Declare detrending, tapering,
|
||||
overlap, normalization, and latency. For tick data, resample deliberately or
|
||||
use irregular-time methods. Wavelets and multiresolution filters can separate
|
||||
horizons, but boundary handling and causal delay must be visible to strategy
|
||||
code.
|
||||
|
||||
### Change and Anomaly Detection
|
||||
|
||||
CUSUM, Page-Hinkley, sequential likelihood ratios, and robust z-scores are
|
||||
cheap online tools. Calibrate false alarms under dependence and regime shifts;
|
||||
do not treat IID thresholds as market guarantees.
|
||||
|
||||
## Numerical Implementation Order
|
||||
|
||||
1. Establish a scalar reference with explicit state and semantics.
|
||||
2. Vectorize historical bootstrap with NumPy.
|
||||
3. Keep realtime updates as O(1) recurrences where possible.
|
||||
4. Use Numba for measured numeric kernels with stable dtypes and no Python
|
||||
object traffic.
|
||||
5. Use Polars or Arrow for columnar historical transforms and interchange,
|
||||
not automatically for each tick.
|
||||
6. Partition by independent market/operator state before adding threads or a
|
||||
new distributed runtime.
|
||||
|
||||
## Existing Stack
|
||||
|
||||
- NumPy 2.x: canonical dense and structured-array kernels.
|
||||
- Numba: compiled CPU loops and recurrences that do not vectorize cleanly.
|
||||
- Polars: parallel/lazy columnar history preparation and validation.
|
||||
- PyArrow: columnar interchange, storage, and ingest boundaries.
|
||||
- Tractor/Trio: structured distributed execution and lifecycle.
|
||||
- PyQtGraph: visible-range rendering, not a compute scheduler.
|
||||
|
||||
## Algorithm Acceptance Criteria
|
||||
|
||||
Every new FSP should define:
|
||||
|
||||
- input fields, clock, ordering, duplicate, gap, and revision policy;
|
||||
- output dtype, units, identity, and parameterization;
|
||||
- warm-up length and initialization bias;
|
||||
- causal latency and strategy-visible publication point;
|
||||
- historical/realtime equivalence tolerance;
|
||||
- reset and session-boundary behavior;
|
||||
- complexity, state size, and representative throughput;
|
||||
- replay behavior when historical source data changes.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# FSP Test Map
|
||||
|
||||
## Operator Tests
|
||||
|
||||
- Historical output schema, dtype, units, and source timestamp alignment.
|
||||
- Realtime update equivalence with the final row of a batch recomputation.
|
||||
- Warm-up, NaN, zero, gap, duplicate, and out-of-order behavior.
|
||||
- Session reset and elapsed-time behavior.
|
||||
- Parameter and source identity isolation.
|
||||
|
||||
## Engine Tests
|
||||
|
||||
- Equal absolute source/destination bounds.
|
||||
- Normal one-step destination lag.
|
||||
- Destination lead and lag greater than one step.
|
||||
- Source prepend by one, two, and a full provider frame.
|
||||
- In-place gap repair with unchanged source bounds.
|
||||
- Equal lengths with shifted absolute bounds.
|
||||
- Source change during historical recomputation and convergence afterward.
|
||||
- Cancellation while the quote stream, sample stream, or compute generator is
|
||||
blocked.
|
||||
- Foreign FQME and timeframe backfill broadcasts.
|
||||
|
||||
## Graph and Actor Tests
|
||||
|
||||
- Dependency startup before downstream operator lookup.
|
||||
- Actor-local flow registry isolation.
|
||||
- One destination writer and multiple readers.
|
||||
- Consumer cancellation without orphaned cascade or worker tasks.
|
||||
- Worker failure propagation and SHM cleanup.
|
||||
- Feed-like fan-out with a slow or disconnected consumer.
|
||||
|
||||
## UI Tests
|
||||
|
||||
- An `fsp_update` redraws every field visualization sharing its destination
|
||||
token and no unrelated market, OHLC, or FSP visualization.
|
||||
- A prepend outside the visible range does not move or re-range the live view.
|
||||
- A revised value inside the visible range invalidates only required formatter
|
||||
and graphics caches.
|
||||
- Hidden charts defer rendering without stopping compute correctness.
|
||||
|
||||
## Performance Tests
|
||||
|
||||
- Historical bootstrap across realistic SHM capacities.
|
||||
- O(1) realtime recurrence cost under bursty quotes.
|
||||
- Consecutive provider-frame prepends with two or more cascades.
|
||||
- Multi-market sampler broadcasts and targeted wakeups.
|
||||
- Visible-range redraw cost at several units-per-pixel levels.
|
||||
|
||||
Prefer deterministic local arrays and controlled Trio task interleavings.
|
||||
Use live providers only as a separate integration layer because credentials,
|
||||
rate limits, sessions, and changing history make poor regression oracles.
|
||||
|
|
@ -0,0 +1 @@
|
|||
../../.agents/skills/piker-fsp-expert
|
||||
Loading…
Reference in New Issue