# Roadmap and Research

Issue states change. Re-read each tracker before planning work and preserve
human ownership of issue and checklist state.

## Guiding Roadmap

- [piker #270](https://github.com/pikers/piker/issues/270): typed,
  stream-composable FSP graph and DSL direction.
- [piker #107](https://github.com/pikers/piker/issues/107): cross-actor shared
  memory tick/ring-buffer direction.
- [piker #216](https://github.com/pikers/piker/issues/216): exposing FSP output
  through feed-like interfaces.
- [piker #325](https://github.com/pikers/piker/issues/325): faster streaming
  y-range computation and overlap with FSP reductions.
- [piker #536](https://github.com/pikers/piker/issues/536): Arrow/Parquet data
  ingest and columnar boundaries.
- [piker #98](https://github.com/pikers/piker/issues/98): older realtime feed
  architecture discussion. Treat it as likely superseded; compare every idea
  against current `Flume`, feed-bus, history, and sampler code.
- [Tractor #339](https://github.com/goodboy/tractor/issues/339): shared-memory
  array and localhost IPC evolution.
- [hotbaud issues](https://github.com/guilledk/hotbaud/issues): experimental
  high-throughput local transport and ring-buffer work.

## Near-Term Protocol Research

Prototype an explicit source revision envelope and measure it with two FSPs
and two overlaid feeds. The prototype should prove source-specific wakeups,
coalescing of consecutive prepends, stale-result rejection, and targeted UI
range updates. Keep the protocol independent of the final IPC transport.

Characterize historical repair strategies:

1. Full replay from the new first bound.
2. Prefix-only calculation for pointwise operators.
3. Replay from an operator checkpoint for stateful recurrences.
4. Bounded overlap repair for finite-window operators.

An FSP declaration eventually needs repair metadata describing locality,
state checkpointability, warm-up/overlap, and whether old inputs can alter all
later outputs.

## Candidate Libraries

Candidates are experiments, not dependency recommendations.

### Bounded Near-Term Prototypes

- [Bottleneck](https://github.com/pydata/bottleneck): compare rolling min/max,
  rank, and moving-window kernels against NumPy/Numba for chart y-ranges and
  finite-window FSPs.
- [River](https://github.com/online-ml/river): study stateful online estimator
  APIs, drift detectors, and reference semantics. Do not assume its object
  model belongs in the hottest tick path.
- [nanoarrow](https://github.com/apache/arrow-nanoarrow): evaluate lightweight
  Arrow C data interfaces at native/process boundaries already motivated by
  `pyarrow` and piker #536.
- [DuckDB](https://github.com/duckdb/duckdb): use for historical Parquet query,
  replay slicing, and validation, outside realtime tick execution.

### Watchlist

- [hotbaud](https://github.com/guilledk/hotbaud): track API and correctness
  maturity for localhost SHM/ring transport.
- [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2): study zero-copy
  pub/sub ownership and lifecycle, but avoid competing with Tractor without a
  narrow adapter benchmark.
- [DataFusion Python](https://github.com/apache/datafusion-python): Arrow-native
  historical query and expression execution if existing Polars paths cannot
  meet a demonstrated need.
- [Feldera](https://github.com/feldera/feldera): study DBSP/incremental-view
  semantics for revision-aware graphs; its service/SQL runtime is not a direct
  piker integration target.
- [Bytewax](https://github.com/bytewax/bytewax) and
  [Arroyo](https://github.com/ArroyoSystems/arroyo): study dataflow semantics,
  watermarking, and recovery, not as replacements for the actor tree.

GPU stacks such as JAX and CuPy are poor defaults for low-batch per-tick work
because transfer, compilation, and scheduling can dominate. Reconsider only
for measured large-batch research or many-market matrix workloads.

## Benchmark Design

Report separate distributions for:

- provider decode and normalization;
- source SHM write and bound publication;
- historical kernel and realtime recurrence;
- actor notification and stream fan-out;
- destination publication;
- visible-range formatting and Qt paint.

Use p50, p95, p99, maximum, allocation rate, and resident-memory growth. Test
steady state, burst load, backfill overlap, cancellation, and slow consumers.
Throughput without event-time correctness and bounded teardown is not a win.

## Medium-Term Sequence

First establish revisioned range messages and operator repair semantics. Then
decouple FSP orchestration from chart ownership and expose destination flows
through feed-like subscriptions. Next benchmark localhost transport adapters
behind the same protocol. Only after those contracts stabilize should graph
placement, fusion, worker scaling, and remote partitioning be automated.