30 KiB

Raw Blame History

Trend Calculus: sources and Piker semantics ==========================================

This implementation follows Andrew J. Morgan's FHLS Trend Calculus. It classifies observed price structure; it does not establish predictive accuracy or a profitable trading rule. Research was collected on 2026-10-04/05. The source inventory below records what was found and distinguishes missing material from working code.

Algorithm

Partition an ordered scalar series into disjoint windows containing n >= 2 observations. Summarize each window by its high and low, retaining each extremum's original coordinate. First and second in FHLS mean the earlier and later of these two extrema. They do not mean the opening and closing observation.

For consecutive windows previous and current, define:

trend = sign(
    sign(current.high - previous.high)
    + sign(current.low - previous.low)
)

The implementation uses ordinal comparisons to avoid overflowing the differences. When the result is zero, introduce a synthetic window joining the previous window's second extremum to the current window's first extremum. Compare previous to synthetic, then synthetic to current. Carry the prior direction through unresolved zero comparisons. A complex window can confirm two opposite reversals at the same observation; retaining only its final trend would lose an event.

A confirmed down-to-up change selects the preceding summary's low; an up-to-down change selects its high. Apply the same calculation to the resulting ordered reversal points for the next reversal order. The order counts successive reductions of the pivot stream. It is independent of a wall-clock timeframe such as one or five seconds.

The live recurrence retains one partial window, one completed window and the prior direction per order. Its numeric state is O(orders) regardless of history length or window size. Historical scanning calls the same Numba recurrence. Returning a full historical result necessarily allocates arrays proportional to history length and order count.

A completed window emits at most two reversals. Because each configured window contains at least two observations, two incoming reversals can complete at most one next-order window during that source update. Induction bounds every order to two events per source observation; the fixed event slots therefore do not truncate a larger cascade.

Deliberate semantic choices --------------------------

piker.fsp._trend.TrendCalculator accepts scalar observations, finite strictly increasing timestamps, and strictly increasing nonnegative source indices. Coordinates need not be contiguous. An OHLC candle alone cannot reveal whether its high or low happened first; feeding fabricated high/low order would change the algorithm. The chart FSP therefore uses close prices.

Production startup is neutral. The first completed window establishes the baseline without inventing a pivot. ready becomes true after two completed windows at that order, including when the direction remains zero on flat data. Thus zero is not a calibrated "sideways" regime probability. Readiness and direction are separate information. Higher orders warm up only when enough lower-order pivots exist.

Equal highs and lows in ordinary FHLS windows retain their earliest observation, matching the original Lua implementation. Synthetic bridges have a narrower, distinct rule: when their endpoint values are equal, high belongs to the second endpoint and low to the first. That distinction affects pivot coordinates even when trends agree.

An optional lua_seed=True reproduces the reference's initial comparison with the first window's last observation. It is a seed policy for reference investigation, not a promise to reproduce all Lua command-line startup quirks. Normal FSPs use neutral startup.

NaN clears all recurrence orders and requires fresh warmup. Infinity and ambiguous coordinate ordering are rejected before state changes. The scalar calculator does not guess session boundaries from elapsed time: its caller decides whether a gap is intentional or a reset. Prepending or revising source history requires replay from the new origin; changing a prefix can affect every subsequent reversal.

Causality and output

TrendCalculator.push() returns all newly detected reversals. Each record carries both coordinates:

index, time, price locate the original pivot;
detected_index, detected_time locate the observation that first confirmed it;
direction is the new trend: +1 for a bottom, -1 for a top; order starts at one.

A higher-order pivot retains its original price/time while detection time propagates from the input observation that completed its higher-order window. Strategies must use detection time, not the earlier pivot time, when deciding what information was available.

TrendCalculator.extend() returns per-observation trends and ready arrays. pulses stores only the last detected direction per order and row. pivot_prices and pivot_times retain both possible sides: axis two is top then bottom, with NaN for no event. Use complete push events when the event stream itself is required.

The trend_calculus FSP exposes four orders through trend, trend_2, trend_3 and trend_4. Its ready field counts ready orders. Each order also exposes top_N, top_time_N, bottom_N and bottom_time_N. Pivot prices are published at their detection row, with the paired field retaining pivot time. Its window argument changes the first-order close window only: the four sizes are (window, 2, 2, 2), defaulting to all twos. The numerical and timeframe APIs accept explicit per-order tuples.

Piker's source SHM timestamps name candle starts. Historical rows preceding the last row are finalized; the last row is provisional. Quotes and source sampler steps preview that row from a copy of committed recurrence state. These wakeups inspect authoritative source SHM; they are not additional scalar observations. Each newly finalized row advances committed state once, including when several steps arrive before a wakeup. Sparse pivot fields clear when the destination steps; source revisions require a cascade replay.

For realtime regime consumers, use finalized closes with an explicit interval-end clock. piker.fsp._regime.MultiTimeframeTrend forms epoch-aligned buckets from complete base bars, discards partial startup buckets and resets recurrence after missing base bars. Its snapshots distinguish timeframe period, recursive order, source revision, sequence, gap segment and availability time. An old model response must not overwrite newer source evidence.

The scalar calculator and current SHM trend_calculus FSP count observations without inferring resets from elapsed timestamp gaps; NaN explicitly resets their recurrence. MultiTimeframeTrend instead requires contiguous base bars and starts a new segment after a missing interval. Consequently, the simple one-second FSP and the one-second timeframe tracker can differ across session gaps. That difference is an ingestion policy, not a different FHLS recurrence.

Using the numerical and timeframe APIs

The numerical API has no provider, GUI or credential requirement. This complete example confirms both sides of a complex window:

from piker.fsp._trend import TrendCalculator

calculator = TrendCalculator((2, 2, 2))
prices: list[float] = [1., 2., 4., 3., 2., 5.]
row: int
price: float
for row, price in enumerate(prices):
    update = calculator.push(
        1_700_000_000. + row,
        price,
        index=100 + row,
    )

print(update.trends)
print(update.reversals[
    ['index', 'direction', 'detected_index']
])

The last update has a top at source index 102 and a bottom at 104, both detected at 105. A consumer only learns these events when it receives that update. For batch analysis, pass aligned one-dimensional values, times and optional signed integer indices arrays to calculator.extend(). Use a fresh calculator for an independent historical replay; extend() advances an existing instance.

For wall-clock timeframes, submit each completed base bar once, using its opening timestamp. The tracker converts this to availability at the interval end:

from piker.fsp._regime import MultiTimeframeTrend

tracker = MultiTimeframeTrend(
    'synthetic.test',
    base_period=1.,
    periods=(1., 5.),
    windows=(2, 2, 2),
)
prices: list[float] = [100., 101., 99., 103., 102.] * 4
row: int
price: float
for row, price in enumerate(prices):
    snapshot = tracker.push(
        1_700_000_000. + row,
        price,
        index=100 + row,
    )

state: dict = snapshot.to_state()

These periods describe independent close series; windows still counts observations and reversal points. Subsecond periods must be whole microseconds, and every configured period must be a whole multiple of base_period. Startup midway through a larger bucket does not produce a shortened first observation. On history repair, call tracker.reset() and replay the corrected completed bars.

Source inventory

Project research and earlier implementation work:

Piker #79 is the older trend-detection issue; Piker #103 is the consolidated research entrypoint. It links Morgan's paper, Lua, Scala, a working mathematical manuscript, and a public TradingView/Pine implementation.
Piker #187 explicitly proposes Trend Calculus in the native FSP set and links the planned Python implementation.
Piker #325 connects trend extrema computation with faster chart y-range reductions. Its moving min/max proposal concerns performance; it is not an alternative definition of FHLS.
Python implementation issue #1 requests offline/online support and recurrence updates across timeframes. The setup branch inspected at 30afcfbd8cbb7b84d984fa2e183c6ae392da3479 has a placeholder tc module and Ray moving-average experiments. Those experiments do not implement FHLS reversal detection.

Primary algorithm sources:

Morgan's method document, HowToStudyTrends_v1.03.pdf: the original conceptual account. Inspected PDF SHA-256:
```
2dc506a0449c13c01d2d798f120411df84e5667b611d92204ba0122e2aac7fd5
```
Original Lua program, ByteSumoLtd/TrendCalculus-lua commit 7f54f9a101283cc188dbcd9282292fc3f73de8e9, make/trendcalculus.lua. Its header credits Andrew J. Morgan / ByteSumo Limited and declares GPL version 3 or later. Inspected source SHA-256:
```
c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0
```
Spark TrendCalculus2, commit 3fb6ef23e985d9d6e0414d3b682afbeef51c52e8. The source declares Apache-2.0. It uses earliest equal high and latest equal low, and initializes from a zero dummy point or an excluded first point. Those choices can change reversals and recursive warmup relative to Lua and Piker. Inspected source SHA-256:
```
3d0723836d230c89d99537f1075537c3bbf809cebefd12caa6a1b517e7bc98b5
```
Working mathematical manuscript, main.tex: describes earliest equal low and latest equal high, a fixed initial positive trend and an initial synthetic reversal. These are different from both the Scala code's tie choices and native neutral startup. Its displayed reversal-sign expression also disagrees with the following low/high selection branches. It is a working draft, not a unique executable specification; the executable references and independently checked cases determine the conventions here. Inspected source SHA-256:
```
e7b6fdff06fbde195a81281022cce300135d88e7484a97d1b02d621c5bfc4eab
```
Spark examples provide historical visualization context. The public Pine indicator linked from Piker #103 describes two lagged Donchian channels and their midpoints for visual analysis. That description does not establish recursive FHLS parity. No complete Pine source or the user's separate Pine implementation was recovered in this search, so no Pine parity claim is made.

Local research locations inspected, relative to the user's repos directory:

lns/todo/piker/quant/trend.toml links the Python setup branch, authors, Piker #187 and the Ray SMA example.
lns/the_legacy_tab_nightmare/trading.toml links the working manuscript, Spark comparisons/PRs, examples, and a former local TrendCalculus/notes_redesign_amorgan.pdf. The latter file was not present at its recorded path.
data/bookmarks/bookmarks.html preserves the older ByteSumo Bitbucket project URL. Duplicate bookmarks are not independent algorithm evidence.

A separate filename inventory across /home/goodboy/repos included hidden and ignored files while excluding Git internals, worktrees, virtual environments, dependency directories, caches and build outputs. It found no *.pine or *.pinescript files, TrendCalculus-named source files, or copy of notes_redesign_amorgan.pdf beyond this change's new documentation and fixture.

No earlier native Trend Calculus implementation was found in the searched Piker source, tests, docs, snippets or available research notes. This is a bounded search result, not proof that no private copy or unindexed history exists. User-owned research/task states were left untouched.

Verification and reproducibility -------------------------------

tests/test_trend_reference.py uses a separate batch oracle that materializes complete FHLS windows, inserts synthetic windows and then reduces the expanded sequence. It never calls the production kernel or copies its mutable recurrence state. Coverage includes:

every eight-point sequence over three prices, for three window tuples: 19,683 histories with ties, flat startup and inside/outside windows; direction, readiness, pulses and both pivot sides agree;
another 2,187 exhaustive six-point histories using the optional Lua seed, while keeping readiness tied to two actual complete windows;
48 seeded 512-point random walks across both startup policies, five recursive orders and mixed window sizes, checking every event's source identity and detection coordinates on irregular timestamps and source indices;
actual unmodified Lua output for windows 2, 3, 4 and 7, covering 62 reversals, including original pivot locations and earliest detection prefixes.

The original-Lua fixture is tests/_inputs/trend_calculus_lua.json. It records source/runtime provenance, the complete synthetic inputs, event schema and SHA-256 hashes of each input and full output CSV. To reproduce an output, write the fixture values with header ticker,date,last and rows x,<zero-based-row>,<value>, using LF line endings. Run the pinned program on Lua 5.2.4 with -n<window> -r -H -p1. For detection times, rerun each complete-window prefix and assign newly emitted events to that prefix's last input row. Prefix outputs must preserve all earlier emitted rows. No Lua executable or network is needed for these Python reference tests.

The pinned Lua CLI has observable quirks:

Do not pass -h with its built-in column mapping. That path consumes the first real observation as a second header, shifting every window boundary. -H controls only output headers.
A flat first complete window can fail while reading a nonexistent prior trend during bridge processing.
The simple-window reversal branch suppresses initial output until offset > 2*n + 1; split-window branches use other guards.

The fixture begins with increasing complete windows to establish a common baseline before reversals. It establishes exact parity for those inputs, not universal startup equivalence between programs. The native oracle separately covers the flat and complex startup cases excluded by that reference limitation.

Integration verification is separate from mathematical parity:

tests/test_trend.py exercises committed/preview handoff, quote-free sampler wakeups, indexed publication and cancellation.
tests/test_trend_sampling.py exercises the actual sampler with a controlled clock. Repeated or backward rounded epochs must not append duplicate timestamps, including after realtime bootstrap.
tests/test_trend_shm.py runs actual Tractor shared memory with a separate read-only attachment, nonzero origin, quiet sampling and an equal-bound historical repair. A separate child actor then attaches read-only using the transported token and checks every repaired field. This verifies cross-process visibility after replay, not atomicity during concurrent writes. It needs actor runtime/socket access, but no broker or model service.
tests/test_trend_regimes.py checks complete aligned buckets, fractional clocks, missing intervals and revision replay; tests/test_regime_feed.py checks native source admission, subscription ordering, quote receiver ownership and lifecycle.
tests/test_jev.py uses mocked transport and controlled clocks for response validation, bounded projection, pacing, retry, source invalidation and cancellation while publication is blocked.
tests/test_trend_graphics.py and tests/test_trend_mtf_graphics.py exercise Qt geometry, causal timestamp mapping, revision invalidation, model expiry and teardown.

Performance measurement

The repository example measures compilation/warmup, batch computation, individual committed observations and copied-state previews separately. Run it in the existing Piker Python environment:

py313/bin/python examples/benchmark_trend.py --rows 1000000 --live-rows 20000

One x86-64 run on 2026-10-05 used Python 3.13.14, NumPy 2.3.5 and Numba 0.62.1 with four reversal orders:

Measurement	Observed result
One million-row batch	0.116473 seconds
Batch throughput	8.586 million rows/second
Scalar commit, median/p99	6.963 / 10.670 microseconds
Copy and preview, median/p99	12.223 / 18.835 microseconds
Initial kernel warmup	3.004 seconds
Retained NumPy state	936 bytes

These are reproducible measurements from one machine and workload, not latency guarantees. Retained-state bytes exclude Python objects, compiler caches and returned history arrays. Timings exclude source SHM copies, actor IPC, chart rendering and provider inference. The native sampler currently timestamps periods at or below one second with integer seconds; subsecond end-to-end operation requires a producer with an appropriate event clock, despite the numerical and timeframe APIs supporting fractional-second observations.

TypeSafe/Jev integration -----------------------

The user supplied the TypeSafe model documentation, Python SDK documentation and agent skill. The adapter uses Piker's existing HTTPX/Trio stack against the documented System One HTTP API; installing the separate Python SDK is unnecessary. The default model is pinned to jev-1.13.0. Importing the adapter does not read credentials or open a connection.

build_regime_request(snapshot) produces one Choice question for each ready timeframe, plus an independent alignment question when multiple timeframes are ready. Candidate regimes are uptrend, downtrend, range, transition and unknown. Questions explicitly identify state paths because question identifiers do not reach the model. JevClassification retains every candidate's probability, the selected label, the service's separate confidence value, model/prompt versions, source provenance and receipt time. These semantic judgments do not change the numerical recurrence. Model input uses the most recent 16 closes and 16 reversals per timeframe by default. evidence_limit configures that projection independently of longer chart history; source snapshots stay unchanged.

Inspect a complete synthetic request, then exercise HTTP parsing with explicitly simulated responses:

py313/bin/python examples/trend_regimes.py
py313/bin/python examples/trend_regimes.py --mock

The synthetic example includes quarter-second evidence. Its --live mode sends exactly one synthetic request and reads TYPESAFE_API_KEY from the caller's environment. It consumes provider credit and makes no broker connection. A real provider request has not been verified without an API key; mocked results test the integration, not model quality.

The native live bridge subscribes to the sampler before copying history, excludes the mutable final source row, and emits only the newest snapshot after each captured history range. It catches up skipped sample steps and invalidates old judgments before replaying a prepend, truncation, backfill or lost sampler notice. run_regime_feed() supplies these snapshots to a synchronous callback without requiring a model client. Use serve_regimes() to compose that reader and a Jev worker:

from collections.abc import Awaitable, Callable

import httpx

from piker.fsp._jev import JevClassification, JevClient
from piker.fsp._regime_feed import serve_regimes
from piker.service import open_piker_runtime

async def run_live(
    fqme: str,
    api_key: str,
    publish: Callable[[JevClassification], Awaitable[None]],
) -> None:
    async with (
        open_piker_runtime('trend-regimes'),
        httpx.AsyncClient() as http,
    ):
        await serve_regimes(
            fqme,
            JevClient(http, api_key),
            publish,
            periods=(1., 5., 15.),
            min_interval=1.,
            max_age=5.,
            max_source_lag=2.,
            max_source_age=5.,
        )

Calling this function opens the configured broker feed and sends live inference requests. Supply an existing Piker market identifier and a cancellation-cooperative async consumer. The example periods assume a one-second source; omit periods to derive 1x/5x/15x periods from the actual source. Native subsecond source periods are rejected until the sampler has a compatible clock. A separate correctly timestamped producer can use consume_regime_source() directly.

The worker permits one in-flight request and retains only the newest pending snapshot. min_interval limits request start frequency; max_age limits local submission age, max_source_lag limits event-time lag, and max_source_age filters stale wall-clock source data before submission. Setting the last option to None admits historical replay. Source revision or gap-segment changes revoke old results. A blocked publication is also cancelled when its evidence expires or becomes stale. A downstream consumer must still check provenance on already-delivered messages, which cannot be revoked.

Current source SHM has no transactional revision word. Copy-time bound checks detect appends and origin changes; an in-place correction with unchanged bounds becomes identifiable through its backfill notice. Replay converges to that corrected source. A publication without a local task checkpoint is not an atomic cross-process memory transaction.

Malformed/authentication responses fail explicitly. Transient failures use bounded retries and freshness checks; the adapter bounds HTTP body sizes and timeouts. The owning nursery closes the feed, sampler and in-flight worker on cancellation. Neither the bridge nor the classifier submits trades. Regime rubrics and probability usefulness still need evaluation on recorded market data and real provider responses.

Chart overlays

Preview synthetic multiple-timeframe annotations without a broker, model request or credentials:

py313/bin/python examples/trend_graphics.py --output /tmp/trend.png

The offscreen example accepts --periods 1 5 15 and --order 1 and isolates its Qt/configuration state from the running application.

Enable the native order-one reversal and direction overlay in the existing Piker configuration's [ui] section:

[ui]
trend_calculus = true

This starts one Trend Calculus writer for the chart's realtime source and displays confirmed pivot arrows with translucent direction rectangles. It makes no Jev request and does not read API credentials. Hover text distinguishes the pivot's bar from the later time when the reversal became observable. The mutable last FSP row is excluded from confirmed markers and direction runs.

Within the existing chart owner's nursery, await FspAdmin.open_trend_overlay(order=2) attaches another recursive order while reusing that admin's writer. Optional chart and chart_period_s arguments attach the source evidence to another price chart. Changing the chart period does not recompute the source at that period: its actual realtime sampling period stays explicit.

For independent wall-clock timeframes on the native realtime chart, configure:

[ui]
trend_timeframes = [1, 5, 15]

This option takes precedence over trend_calculus = true. It uses one MultiTimeframeTrend reader and one SnapshotTrendOverlay with separate marker/rectangle layers for each period. It opens no Jev client. await FspAdmin.open_multi_timeframe_trends(...) exposes the same integration programmatically and accepts a recursive order. All periods must be multiples of the actual realtime source period.

Snapshots preserve interval-end pivot and detection times. Markers attach to the base bar that contributed each aggregated close. A bootstrap snapshot contains recent reversals and the latest direction, so its deterministic rectangle history starts with that observed direction. It does not reconstruct earlier direction runs from one snapshot. Skipped complete buckets leave unobserved intervals unpainted. Source revisions and missing-base-bar segments revoke prior annotations.

An existing caller-owned client can share this exact evidence with the model worker inside the chart's nursery:

import trio

from piker.fsp._jev import JevClient, RegimeClassifier
from piker.ui._fsp import FspAdmin
from piker.ui._trend_mtf import SnapshotTrendOverlay

async def attach_classifier(
    admin: FspAdmin,
    client: JevClient,
    nursery: trio.Nursery,
) -> tuple[RegimeClassifier, SnapshotTrendOverlay]:
    worker: RegimeClassifier = RegimeClassifier(client)
    overlay = await admin.open_multi_timeframe_trends(
        periods=(1., 5., 15.),
        on_snapshot=worker.submit,
        on_invalidate=worker.invalidate,
        max_source_age=5.,
        classification_ttl=5.,
    )
    nursery.start_soon(worker.run, overlay.publish_classification)
    return worker, overlay

Graphics retain their longer deterministic history while the client projects bounded model evidence. The source-age gate applies to the optional model callback, so historical chart bootstrap still works. The adapter checks market, revision, segment, sequence and source lag before accepting a model response. Classified runs start at receipt and expire after the configured TTL; they cannot paint future history before the displayed event clock reaches receipt. Cancelling the chart owner removes all layers and stops its sampler reader.

For computations that already publish separate timeframe SHMs, create one piker.ui._trend.TrendOverlay per destination SHM with its own source_period_s, target chart_period_s and order. Mapping uses timestamps and preserves gaps rather than snapping missing observations onto the next candle. Geometry counts are bounded and items are reused instead of adding individual Qt items on every tick.

Call update(force=True) after a matching destination or chart history revision, and remove() when its owner exits. Normal FspAdmin integration routes destination-specific revisions and teardown automatically; direct attachments must retain this ownership. Hidden widgets preserve revision invalidations until their next visible update. Repainting follows quotes and the existing one-second sampler wakeup. If that wakeup precedes asynchronous FSP publication, a quote-free finalization can appear on the next wakeup, roughly one second later; the integration does not emit IPC for every output row.

For model annotations, classification_interval() converts a matching JevClassification into a RegimeInterval. Supply the chart's current market, source revision, timeframe and display end. The interval starts no earlier than the model response's receipt. overlay.set_regimes(intervals) displays an owner-maintained, bounded collection of these intervals. An empty collection clears model rectangles; None restores deterministic direction runs. The caller remains responsible for response ordering and expiry.

30 KiB Raw Blame History