4.2 KiB

Raw Blame History

FSP Runtime Architecture

Current Topology

provider backend
  -> datad feed bus
  -> source Flume
     -> rt ShmArray (typically 1 second)
     -> hist ShmArray (typically 60 seconds)
  -> samplerd sample and backfill events

chart actor
  -> FspAdmin
  -> chart.fsp_N worker actor
     -> cascade()
     -> connect_streams()
     -> destination Flume / ShmArray
  -> targeted Viz updates

The source path is implemented mainly by piker.data.feed, piker.data.flows, piker.data._sampling, and piker.tsp._history. The FSP path is implemented by piker.fsp._api, piker.fsp._engine, and operator modules such as _volume and _momo. Worker and graphics lifecycle currently live in piker.ui._fsp and piker.ui._display.

Operator Contract

@fsp wraps an async generator. Its first yield is historical output computed from the source array. Later yields are (field, value) realtime mutations. connect_streams() converts the first yield into the destination structured array, aligns time and absolute bounds, then updates the current destination row from realtime quote events.

samplerd owns row advancement. At a new source sample, the cascade copies or zeros the previous destination row and aligns destination timestamps to the source. Quote-time mutations and sample-time appends are distinct operations.

SHM Synchronization

Treat each array as an absolute half-open range:

[first, last)

For a normal source history prepend:

before: src=[10000, 12000) dst=[10000, 12000)
after:  src=[ 8000, 12000) dst=[10000, 12000)

The current step is unchanged, but destination history is missing 2,000 rows. The destination must be recomputed or incrementally repaired.

For a normal one-step source lead:

src=[8000, 12001) dst=[8000, 12000)

The cascade should append exactly one destination row, not recompute history.

Use _first.value and _last.value for synchronization. ShmArray.index is the last bound modulo capacity and is not a complete absolute position. Snapshot bounds once per check; shared first and last counters do not form a transactional pair.

Backfill Event Ordering

start_backfill() receives an older provider frame.
shm_push_in_between() writes the frame and publishes the earlier first bound.
notify_backfill() asks samplerd to broadcast the market and timeframe.
Relevant cascades treat the event as a history revision. A prepend changes bounds; an in-place gap repair may not.
Each cascade cancels its current compute task and waits for completion.
The historical phase recomputes against the newest readable source range.
Destination bounds are published and the UI receives an fsp_update.
Only Viz objects backed by that destination token redraw.

Multiple provider frames may arrive while a recomputation runs. The cascade must reject mixed-bound bootstrap results, replay in-place revisions, recheck bounds after restart, and converge on the newest source range.

Actor-Local State

Fsp._flow_registry, attached SHM handles, caches, and nurseries are local to each actor. Never assume worker siblings share warmed registries or Python objects. Graph dependencies currently rely on startup order, as demonstrated by flow_rates starting only after dolla_vlm.

Cross-Subsystem Boundaries

Data ingest should normalize identity and event-time semantics before an FSP consumes a stream.
UI renderers should consume revision/range information without controlling computation correctness.
Clearing may consume a derived signal, but EMS remains responsible for order intent, status, routing, fills, positions, and accounting.
FSP output exposed as a feed needs the same fan-out, identity, lifecycle, and backpressure contracts as provider feeds.

Architectural Gaps

The current protocol infers history revisions from shared bounds. A future revision message should carry at least:

(fqme, timeframe, revision, first, last, start_ts, end_ts)

Destination publication should identify the source revision used. This allows coalescing, range repair, stale-result rejection, and deterministic graph dependency startup without turning every prepend into a global wakeup.

4.2 KiB Raw Blame History