piker/.agents/skills/piker-fsp-expert/roadmap-and-research.md

Roadmap and Research

Issue states change. Re-read each tracker before planning work and preserve human ownership of issue and checklist state.

Guiding Roadmap

• piker #270: typed, stream-composable FSP graph and DSL direction.
• piker #107: cross-actor shared memory tick/ring-buffer direction.
• piker #216: exposing FSP output through feed-like interfaces.
• piker #325: faster streaming y-range computation and overlap with FSP reductions.
• piker #536: Arrow/Parquet data ingest and columnar boundaries.
• piker #98: older realtime feed architecture discussion. Treat it as likely superseded; compare every idea against current Flume, feed-bus, history, and sampler code.
• Tractor #339: shared-memory array and localhost IPC evolution.
• hotbaud issues: experimental high-throughput local transport and ring-buffer work.

Near-Term Protocol Research

Prototype an explicit source revision envelope and measure it with two FSPs and two overlaid feeds. The prototype should prove source-specific wakeups, coalescing of consecutive prepends, stale-result rejection, and targeted UI range updates. Keep the protocol independent of the final IPC transport.

Characterize historical repair strategies:

1. Full replay from the new first bound.
2. Prefix-only calculation for pointwise operators.
3. Replay from an operator checkpoint for stateful recurrences.
4. Bounded overlap repair for finite-window operators.

An FSP declaration eventually needs repair metadata describing locality, state checkpointability, warm-up/overlap, and whether old inputs can alter all later outputs.

Candidate Libraries

Candidates are experiments, not dependency recommendations.

Bounded Near-Term Prototypes

• Bottleneck: compare rolling min/max, rank, and moving-window kernels against NumPy/Numba for chart y-ranges and finite-window FSPs.
• River: study stateful online estimator APIs, drift detectors, and reference semantics. Do not assume its object model belongs in the hottest tick path.
• nanoarrow: evaluate lightweight Arrow C data interfaces at native/process boundaries already motivated by pyarrow and piker #536.
• DuckDB: use for historical Parquet query, replay slicing, and validation, outside realtime tick execution.

Watchlist

• hotbaud: track API and correctness maturity for localhost SHM/ring transport.
• iceoryx2: study zero-copy pub/sub ownership and lifecycle, but avoid competing with Tractor without a narrow adapter benchmark.
• DataFusion Python: Arrow-native historical query and expression execution if existing Polars paths cannot meet a demonstrated need.
• Feldera: study DBSP/incremental-view semantics for revision-aware graphs; its service/SQL runtime is not a direct piker integration target.
• Bytewax and Arroyo: study dataflow semantics, watermarking, and recovery, not as replacements for the actor tree.

GPU stacks such as JAX and CuPy are poor defaults for low-batch per-tick work because transfer, compilation, and scheduling can dominate. Reconsider only for measured large-batch research or many-market matrix workloads.

Benchmark Design

Report separate distributions for:

• provider decode and normalization;
• source SHM write and bound publication;
• historical kernel and realtime recurrence;
• actor notification and stream fan-out;
• destination publication;
• visible-range formatting and Qt paint.

Use p50, p95, p99, maximum, allocation rate, and resident-memory growth. Test steady state, burst load, backfill overlap, cancellation, and slow consumers. Throughput without event-time correctness and bounded teardown is not a win.

Medium-Term Sequence

First establish revisioned range messages and operator repair semantics. Then decouple FSP orchestration from chart ownership and expose destination flows through feed-like subscriptions. Next benchmark localhost transport adapters behind the same protocol. Only after those contracts stabilize should graph placement, fusion, worker scaling, and remote partitioning be automated.