piker/.agents/skills/piker-fsp-expert/architecture.md

# FSP Runtime Architecture

## Current Topology

```text
provider backend
  -> datad feed bus
  -> source Flume
     -> rt ShmArray (typically 1 second)
     -> hist ShmArray (typically 60 seconds)
  -> samplerd sample and backfill events

chart actor
  -> FspAdmin
  -> chart.fsp_N worker actor
     -> cascade()
     -> connect_streams()
     -> destination Flume / ShmArray
  -> targeted Viz updates
```

The source path is implemented mainly by `piker.data.feed`,
`piker.data.flows`, `piker.data._sampling`, and `piker.tsp._history`. The FSP
path is implemented by `piker.fsp._api`, `piker.fsp._engine`, and operator
modules such as `_volume` and `_momo`. Worker and graphics lifecycle currently
live in `piker.ui._fsp` and `piker.ui._display`.

## Operator Contract

`@fsp` wraps an async generator. Its first yield is historical output computed
from the source array. Later yields are `(field, value)` realtime mutations.
`connect_streams()` converts the first yield into the destination structured
array, aligns time and absolute bounds, then updates the current destination
row from realtime quote events.

`samplerd` owns row advancement. At a new source sample, the cascade copies or
zeros the previous destination row and aligns destination timestamps to the
source. Quote-time mutations and sample-time appends are distinct operations.

## SHM Synchronization

Treat each array as an absolute half-open range:

```text
[first, last)
```

For a normal source history prepend:

```text
before: src=[10000, 12000) dst=[10000, 12000)
after:  src=[ 8000, 12000) dst=[10000, 12000)
```

The current step is unchanged, but destination history is missing 2,000 rows.
The destination must be recomputed or incrementally repaired.

For a normal one-step source lead:

```text
src=[8000, 12001) dst=[8000, 12000)
```

The cascade should append exactly one destination row, not recompute history.

Use `_first.value` and `_last.value` for synchronization. `ShmArray.index` is
the last bound modulo capacity and is not a complete absolute position.
Snapshot bounds once per check; shared first and last counters do not form a
transactional pair.

## Backfill Event Ordering

1. `start_backfill()` receives an older provider frame.
2. `shm_push_in_between()` writes the frame and publishes the earlier first
   bound.
3. `notify_backfill()` asks `samplerd` to broadcast the market and timeframe.
4. Relevant cascades treat the event as a history revision. A prepend changes
   bounds; an in-place gap repair may not.
5. Each cascade cancels its current compute task and waits for completion.
6. The historical phase recomputes against the newest readable source range.
7. Destination bounds are published and the UI receives an `fsp_update`.
8. Only `Viz` objects backed by that destination token redraw.

Multiple provider frames may arrive while a recomputation runs. The cascade
must reject mixed-bound bootstrap results, replay in-place revisions, recheck
bounds after restart, and converge on the newest source range.

## Actor-Local State

`Fsp._flow_registry`, attached SHM handles, caches, and nurseries are local to
each actor. Never assume worker siblings share warmed registries or Python
objects. Graph dependencies currently rely on startup order, as demonstrated
by `flow_rates` starting only after `dolla_vlm`.

## Cross-Subsystem Boundaries

- Data ingest should normalize identity and event-time semantics before an FSP
  consumes a stream.
- UI renderers should consume revision/range information without controlling
  computation correctness.
- Clearing may consume a derived signal, but EMS remains responsible for order
  intent, status, routing, fills, positions, and accounting.
- FSP output exposed as a feed needs the same fan-out, identity, lifecycle,
  and backpressure contracts as provider feeds.

## Architectural Gaps

The current protocol infers history revisions from shared bounds. A future
revision message should carry at least:

```text
(fqme, timeframe, revision, first, last, start_ts, end_ts)
```

Destination publication should identify the source revision used. This allows
coalescing, range repair, stale-result rejection, and deterministic graph
dependency startup without turning every prepend into a global wakeup.