105 lines
4.7 KiB
Markdown
105 lines
4.7 KiB
Markdown
# Roadmap and Research
|
|
|
|
Issue states change. Re-read each tracker before planning work and preserve
|
|
human ownership of issue and checklist state.
|
|
|
|
## Guiding Roadmap
|
|
|
|
- [piker #270](https://github.com/pikers/piker/issues/270): typed,
|
|
stream-composable FSP graph and DSL direction.
|
|
- [piker #107](https://github.com/pikers/piker/issues/107): cross-actor shared
|
|
memory tick/ring-buffer direction.
|
|
- [piker #216](https://github.com/pikers/piker/issues/216): exposing FSP output
|
|
through feed-like interfaces.
|
|
- [piker #325](https://github.com/pikers/piker/issues/325): faster streaming
|
|
y-range computation and overlap with FSP reductions.
|
|
- [piker #536](https://github.com/pikers/piker/issues/536): Arrow/Parquet data
|
|
ingest and columnar boundaries.
|
|
- [piker #98](https://github.com/pikers/piker/issues/98): older realtime feed
|
|
architecture discussion. Treat it as likely superseded; compare every idea
|
|
against current `Flume`, feed-bus, history, and sampler code.
|
|
- [Tractor #339](https://github.com/goodboy/tractor/issues/339): shared-memory
|
|
array and localhost IPC evolution.
|
|
- [hotbaud issues](https://github.com/guilledk/hotbaud/issues): experimental
|
|
high-throughput local transport and ring-buffer work.
|
|
|
|
## Near-Term Protocol Research
|
|
|
|
Prototype an explicit source revision envelope and measure it with two FSPs
|
|
and two overlaid feeds. The prototype should prove source-specific wakeups,
|
|
coalescing of consecutive prepends, stale-result rejection, and targeted UI
|
|
range updates. Keep the protocol independent of the final IPC transport.
|
|
|
|
Characterize historical repair strategies:
|
|
|
|
1. Full replay from the new first bound.
|
|
2. Prefix-only calculation for pointwise operators.
|
|
3. Replay from an operator checkpoint for stateful recurrences.
|
|
4. Bounded overlap repair for finite-window operators.
|
|
|
|
An FSP declaration eventually needs repair metadata describing locality,
|
|
state checkpointability, warm-up/overlap, and whether old inputs can alter all
|
|
later outputs.
|
|
|
|
## Candidate Libraries
|
|
|
|
Candidates are experiments, not dependency recommendations.
|
|
|
|
### Bounded Near-Term Prototypes
|
|
|
|
- [Bottleneck](https://github.com/pydata/bottleneck): compare rolling min/max,
|
|
rank, and moving-window kernels against NumPy/Numba for chart y-ranges and
|
|
finite-window FSPs.
|
|
- [River](https://github.com/online-ml/river): study stateful online estimator
|
|
APIs, drift detectors, and reference semantics. Do not assume its object
|
|
model belongs in the hottest tick path.
|
|
- [nanoarrow](https://github.com/apache/arrow-nanoarrow): evaluate lightweight
|
|
Arrow C data interfaces at native/process boundaries already motivated by
|
|
`pyarrow` and piker #536.
|
|
- [DuckDB](https://github.com/duckdb/duckdb): use for historical Parquet query,
|
|
replay slicing, and validation, outside realtime tick execution.
|
|
|
|
### Watchlist
|
|
|
|
- [hotbaud](https://github.com/guilledk/hotbaud): track API and correctness
|
|
maturity for localhost SHM/ring transport.
|
|
- [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2): study zero-copy
|
|
pub/sub ownership and lifecycle, but avoid competing with Tractor without a
|
|
narrow adapter benchmark.
|
|
- [DataFusion Python](https://github.com/apache/datafusion-python): Arrow-native
|
|
historical query and expression execution if existing Polars paths cannot
|
|
meet a demonstrated need.
|
|
- [Feldera](https://github.com/feldera/feldera): study DBSP/incremental-view
|
|
semantics for revision-aware graphs; its service/SQL runtime is not a direct
|
|
piker integration target.
|
|
- [Bytewax](https://github.com/bytewax/bytewax) and
|
|
[Arroyo](https://github.com/ArroyoSystems/arroyo): study dataflow semantics,
|
|
watermarking, and recovery, not as replacements for the actor tree.
|
|
|
|
GPU stacks such as JAX and CuPy are poor defaults for low-batch per-tick work
|
|
because transfer, compilation, and scheduling can dominate. Reconsider only
|
|
for measured large-batch research or many-market matrix workloads.
|
|
|
|
## Benchmark Design
|
|
|
|
Report separate distributions for:
|
|
|
|
- provider decode and normalization;
|
|
- source SHM write and bound publication;
|
|
- historical kernel and realtime recurrence;
|
|
- actor notification and stream fan-out;
|
|
- destination publication;
|
|
- visible-range formatting and Qt paint.
|
|
|
|
Use p50, p95, p99, maximum, allocation rate, and resident-memory growth. Test
|
|
steady state, burst load, backfill overlap, cancellation, and slow consumers.
|
|
Throughput without event-time correctness and bounded teardown is not a win.
|
|
|
|
## Medium-Term Sequence
|
|
|
|
First establish revisioned range messages and operator repair semantics. Then
|
|
decouple FSP orchestration from chart ownership and expose destination flows
|
|
through feed-like subscriptions. Next benchmark localhost transport adapters
|
|
behind the same protocol. Only after those contracts stabilize should graph
|
|
placement, fusion, worker scaling, and remote partitioning be automated.
|