pikers
/
piker
4
1
Fork 0
Code Issues 20 Pull Requests 13 Packages Projects Releases Wiki Activity

Save draft Trend Calculus and Jev prototype 

Save the native FHLS recurrence, causal FSP publication, aligned
multi-timeframe evidence, Trio/HTTPX Jev worker and Qt pivot/regime
overlays as one draft snapshot.

Retain tests, research sources, Lua fixtures, runnable examples,
full license notices and prompt provenance. The latest reviewed
offline suite passed 253 tests with one unrelated actor case
excluded and one existing Tractor warning.

Defer behavioral commit splitting and upstream history integration.
Live Jev quality, provider-service wiring and Pine parity remain
follow-ups in Taken.

Prompt-IO: ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md
Prompt-IO: ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md
Prompt-IO: ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md

(this patch was generated in some part by `codex` using
GPT-6 family models (`openai`); exact model IDs unavailable)
trend_calc_on_jev
Gud Boi 2026-10-06 16:04:41 -04:00
parent eed5856622
commit e815fd77cf
34 changed files with 12677 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
ai/prompt-io/README.md 100644
View File

@ -0,0 +1,23 @@
# AI Prompt I/O Log
This directory tracks prompt inputs and model outputs for
AI-assisted development across harnesses.
## Policy
Prompt logging follows the
[NLNet generative AI policy][nlnet-ai]. Substantive contributions
record the original harness, provider/model when known, timestamps,
the prompting sequence, and an unedited `.raw.md` partner.
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
## Usage
Entries are created by the `prompt-io` skill or its commit-message
integration. Harness-specific records live in their corresponding
subdirectories. Code records can reference the associated Git diff
instead of duplicating source code.
Human contributors retain responsibility for review and acceptance.
AI-generated content is not presented as human-authored work.

118
ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md 100644
View File

@ -0,0 +1,118 @@
---
created_at: 2026-10-05T04:11:32.732262+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
scope: code
substantive: true
raw_file: 20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md
---
## Prompt
The user requested an autonomous search of GitHub issues and local
notes, code, documentation and comments for Trend Calculus research,
then a rigorous Piker-native implementation in the FSP subsystem.
They asked to prepare multiple high-frequency timeframe regimes for
the JEV System-1 provider service, with work bounded by 00:18 Eastern.
They authorized same-caliber subagents and requested, as a bonus,
multiple-timeframe reversal markers and live rectangle annotations.
The user supplied these implementation resources:
- https://docs.typesafe.ai/models
- https://docs.typesafe.ai/sdk/python
- https://docs.typesafe.ai/agent-skill
They offered a Pine implementation but had not supplied it. They
said the API key was still being set up and instructed us to continue
offline. They clarified the workspace concern, disabled fast mode,
and supplied repository rules: preserve human-owned task states,
no autonomous commits or pushes, 69-column prose, Xonsh snippets,
and the Python style skill.
## Response summary
Implemented a bounded Numba recurrence and a four-order native FSP
with provisional-tail isolation, skipped-step recovery, indexed
publication, quiet sampler wakeups, sparse events and history replay.
Added complete aligned multiple-timeframe aggregation, immutable
provenance-rich snapshots, bounded HTTPX/Trio JEV requests and a
paced worker that rejects stale or invalidated results. Added native
feed/service wiring and opt-in Qt markers and regime rectangles,
including actual 1/5/15-second layers and model receipt/expiry rules.
Research identifies pinned Lua/Scala sources, issue discussions,
local notes and the manuscript, with explicit semantic differences.
Original Lua fixtures and an independent materialized-window oracle
check 21,870 exhaustive histories, 48 random recursive streams and
62 original Lua reversals. No complete Pine source was recovered.
The TypeSafe agent skill was installed and applied. No live API key
was read, no model request was sent, no broker session was opened,
and no Git history or human-owned task state was changed.
## Verification
The combined targeted suite passed 252 tests, with one unrelated
remote gap-dialog actor case deselected. The one warning is an
existing Tractor dependency's deprecated Trio Event truth test.
The new SHM test includes an actual separate read-only child actor
after in-place replay. This does not prove concurrent cross-process
write atomicity. Ruff, diff whitespace and 69-column checks passed.
The combined suite covered test_trend, test_trend_reference,
test_trend_shm, test_trend_regimes, test_jev, test_regime_feed,
test_trend_graphics, test_trend_mtf_graphics, test_fsp_sync,
test_fsp_momo and test_gap_overlays. The standard py313 environment
and pytest with the xonsh plugin disabled were used. Socket/SHM
execution required approved sandbox escalation.
The synthetic offscreen chart preview was generated and inspected.
The reproducible benchmark measured roughly 8.586 million batch
rows/second and 6.963 microseconds median scalar update on this host;
these exclude rendering, transport and provider inference. Native
subsecond sampling remains unsupported by the existing clock.
Live provider quality and the user's Pine parity remain unverified.
## Files changed
- `piker/fsp/__init__.py`
- `piker/fsp/_api.py`
- `piker/fsp/_engine.py`
- `piker/ui/_display.py`
- `piker/ui/_fsp.py`
- `piker/fsp/_trend.py`
- `piker/fsp/_trend_fsp.py`
- `piker/fsp/_regime.py`
- `piker/fsp/_jev.py`
- `piker/fsp/_regime_feed.py`
- `piker/ui/_trend.py`
- `piker/ui/_trend_mtf.py`
- `tests/test_trend.py`
- `tests/test_trend_shm.py`
- `tests/test_trend_reference.py`
- `tests/test_trend_regimes.py`
- `tests/test_jev.py`
- `tests/test_regime_feed.py`
- `tests/test_trend_graphics.py`
- `tests/test_trend_mtf_graphics.py`
- `tests/_inputs/trend_calculus_lua.json`
- `examples/benchmark_trend.py`
- `examples/trend_regimes.py`
- `examples/trend_graphics.py`
- `docs/trend_calculus.rst`
- `ai/prompt-io/README.md`
## Human edits
The human supplied the objective, deadline, research resources,
provider choice, optional skill installation, graphics extension,
parallel-agent authorization and offline credential constraint.
Those directions materially determined the implementation and scope.
No direct manual source edits were observed during this work. The
patch is pending human review; acceptance is not inferred. Research,
implementation, review and tests were performed by the root Codex
agent with three same-caliber delegated agents. Existing unrelated
working-tree artifacts were preserved.

106
ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md 100644
View File

@ -0,0 +1,106 @@
---
created_at: 2026-10-05T04:11:32.732262+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
diff_cmd: git diff eed58566 --
---
# Captured output and code references
This record captures selected unedited progress responses and the
code artifact pointers before human source edits. It is not a full
transcript. Exact model and session identifiers were unavailable.
The working branch is trend_calc_on_jev; no commit was authorized.
While files remain untracked, use the no-index references below.
After inclusion in branch history, use the base-to-branch diff.
## Unedited progress excerpts
The cross-process shared-memory test passes, including replay
after an in-place history repair. The JEV feed also preserves
the shared quote policy and catches up cleanly when new bars
arrive during a history copy.
I’m running the combined checks and reviewing the chart preview.
Live JEV verification remains pending your key setup.
The combined offline suite passes: 252 tests, including the
independent algorithm oracle, JEV mock transport, Qt overlays,
existing FSP checks, and the cross-process SHM test.
The chart preview shows separate 1-, 5-, and 15-second reversal
markers and regime shading. I’m checking the final diff and
recording provenance; the work remains uncommitted.
## Unedited verification output
```text
252 passed, 1 deselected, 1 warning in 3.79s
All checks passed!
```
## Code artifact pointers
> `git diff eed58566 -- piker/fsp/__init__.py`
> `git diff eed58566 -- piker/fsp/_api.py`
> `git diff eed58566 -- piker/fsp/_engine.py`
> `git diff eed58566 -- piker/ui/_display.py`
> `git diff eed58566 -- piker/ui/_fsp.py`
> `git diff --no-index /dev/null piker/fsp/_trend.py`
> `git diff --no-index /dev/null piker/fsp/_trend_fsp.py`
> `git diff --no-index /dev/null piker/fsp/_regime.py`
> `git diff --no-index /dev/null piker/fsp/_jev.py`
> `git diff --no-index /dev/null piker/fsp/_regime_feed.py`
> `git diff --no-index /dev/null piker/ui/_trend.py`
> `git diff --no-index /dev/null piker/ui/_trend_mtf.py`
> `git diff --no-index /dev/null tests/test_trend.py`
> `git diff --no-index /dev/null tests/test_trend_shm.py`
> `git diff --no-index /dev/null tests/test_trend_reference.py`
> `git diff --no-index /dev/null tests/test_trend_regimes.py`
> `git diff --no-index /dev/null tests/test_jev.py`
> `git diff --no-index /dev/null tests/test_regime_feed.py`
> `git diff --no-index /dev/null tests/test_trend_graphics.py`
> `git diff --no-index /dev/null tests/test_trend_mtf_graphics.py`
> `git diff --no-index /dev/null tests/_inputs/trend_calculus_lua.json`
> `git diff --no-index /dev/null examples/benchmark_trend.py`
> `git diff --no-index /dev/null examples/trend_regimes.py`
> `git diff --no-index /dev/null examples/trend_graphics.py`
> `git diff --no-index /dev/null docs/trend_calculus.rst`
> `git diff --no-index /dev/null ai/prompt-io/README.md`
The implementation adds the bounded recursive TrendCalculator,
causal history/preview FSP publication, explicit indexed FspSample
updates, coalesced quiet-step wakeups and sparse-event clearing.
MultiTimeframeTrend forms complete aligned buckets; the Jev worker
uses bounded evidence, single-flight requests and freshness checks.
Qt layers render confirmed pivots and bounded regime rectangles.
Independent reference, mock transport and shared-memory tests cover
mathematical and integration behavior. The docs preserve sources,
semantic differences, measured performance and remaining limits.

42
ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md 100644
View File

@ -0,0 +1,42 @@
---
created_at: 2026-10-05T05:02:18.111444+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
scope: code
substantive: true
raw_file: 20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md
---
## Prompt
Continuation of the user's autonomous, deadline-bounded request for
rigorous native Trend Calculus, live regime service preparation and
graphics integration. This supplements the earlier trend record.
## Response summary
Final delegated integration review reproduced duplicate timestamps
in the existing native sampler: seeded [1000,1001] became
[1000,1001,1001], causing the causal calculator to reject the source.
Added a narrow guard for native periods at or below one second and
a controlled-clock regression covering repeated, backward and newer
epochs. Wakeup broadcasts remain unchanged. No refactor or native
subsecond support is claimed. The root reviewed the change and ran
the complete targeted suite: 253 passed, one unrelated remote gap
case deselected, one existing Tractor/Trio deprecation warning.
## Files changed
- `piker/data/_sampling.py`
- `tests/test_trend_sampling.py`
- `docs/trend_calculus.rst`
## Human edits
The original human requirement for reliable native integration
motivated this correction. No additional direct human source edit
was observed. The root and delegated Codex agent implemented and
reviewed it under that scope. Human review and acceptance remain
pending; no commits, broker sessions or model requests were made.

38
ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md 100644
View File

@ -0,0 +1,38 @@
---
created_at: 2026-10-05T05:02:18.111444+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
diff_cmd: git diff eed58566 --
---
# Final integration correction
## Unedited progress output
The sampler fix is verified. The final combined suite passes
253 tests, including repeated and backward clock cases.
Implementation and tests are now frozen; I’m saving the review
patch and final provenance record.
## Unedited verification result
```text
253 passed, 1 deselected, 1 warning in 3.86s
```
## Code pointers
> `git diff eed58566 -- piker/data/_sampling.py`
> `git diff --no-index /dev/null tests/test_trend_sampling.py`
> `git diff --no-index /dev/null docs/trend_calculus.rst`
The actual Sampler.increment_ohlc_buffer could append a duplicate
rounded epoch after realtime startup, or when the clock repeated
or moved backward. The fix retains the existing mutable bar until
a newer epoch exists, preserving broadcasts. The regression drives
the real sampler with Trio MockClock and verifies that its complete
output remains valid input for the strict causal calculator.

79
ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md 100644
View File

@ -0,0 +1,79 @@
---
created_at: 2026-10-05T05:40:14.294712+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
scope: code
substantive: true
raw_file: 20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md
---
## Prompt
The human explicitly requested a complete py-codestyle pass and
proper license headers for every new code module after identifying
that the previous pass was incomplete. They retained autonomous
work and parallel-agent authorization, offline provider constraints,
and human ownership of task states and commits.
## Response summary
Read the repository-deployed py-codestyle skill and shared runtime
review lessons. Root and three agents audited all changed Python
modules, tests and runnable examples. Added complete project AGPL
notices and SPDX identifiers to all 19 new modules, preserving the
Morgan/ByteSumo algorithm attribution. Corrected annotations,
helper contracts, expression layout, named interpolation operands,
and owner-qualified regression documentation. Used narrow F842
suppressions where Ruff cannot connect declarations to isolated
comprehension scopes.
## Verification
Ruff passes across all changed Python modules. All new source and
added old-file lines meet the 69-column rule with no whitespace-only
lines. Every new function and class has the required multiline
contract; function parameters and returns are annotated. The final
offline suite reports 253 passed, one unrelated actor case
excluded, and one existing Tractor deprecation warning. Native SHM
publication, cross-process replay visibility, quiet sampler steps,
Numba reference parity, Qt overlays and mocked Jev protocols pass.
No live model quality, separate provider-service wiring, Pine parity
or cross-process write atomicity is claimed.
## Human edits and follow-ups
The human's style correction materially drove these changes.
No direct manual source edits were observed; acceptance is pending.
The human deferred SDK AnyIO/Trio work until the HTTP prototype is
validated and requested a later Taken-store handoff. Existing task
states, unrelated artifacts, Git index and Git history are preserved.
## Files changed
- `piker/fsp/__init__.py`
- `piker/fsp/_api.py`
- `piker/fsp/_engine.py`
- `piker/ui/_display.py`
- `piker/ui/_fsp.py`
- `piker/data/_sampling.py`
- `piker/fsp/_trend.py`
- `piker/fsp/_trend_fsp.py`
- `piker/fsp/_regime.py`
- `piker/fsp/_jev.py`
- `piker/fsp/_regime_feed.py`
- `piker/ui/_trend.py`
- `piker/ui/_trend_mtf.py`
- `tests/test_trend.py`
- `tests/test_trend_shm.py`
- `tests/test_trend_reference.py`
- `tests/test_trend_regimes.py`
- `tests/test_jev.py`
- `tests/test_regime_feed.py`
- `tests/test_trend_graphics.py`
- `tests/test_trend_mtf_graphics.py`
- `tests/test_trend_sampling.py`
- `examples/benchmark_trend.py`
- `examples/trend_regimes.py`
- `examples/trend_graphics.py`

98
ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md 100644
View File

@ -0,0 +1,98 @@
---
created_at: 2026-10-05T05:40:14.294712+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: eed58566
scope: code
substantive: true
---
## Human instruction (verbatim)
ok since i'm seeing this alot i want to make sure you did a proper /py-codestyle pass and have proper license headers throughout all new code-modules.
## Captured assistant output (verbatim)
The earlier pass wasn’t complete: the tests and examples lacked
license notices, and several modules had only abbreviated ones.
All 19 new Python files now have full AGPL notices and SPDX
identifiers, with the algorithm’s source attribution preserved.
I’m checking the remaining style rules too—annotations,
docstrings, expression layout, and regression explanations.
## Final regression output (verbatim)
```text
253 passed, 1 deselected, 1 warning in 7.93s
```
The warning is the existing Tractor dependency's Trio Event
truth-test deprecation; the unrelated remote gap-dialog actor test
is excluded. The first sandboxed attempt stopped on Trio socket
permissions and was rerun with approved escalation. No broker or
Jev API request was made.
## Code pointers
The associated boundary diffs capture full source changes. The
whole-file commit package includes untracked modules, which a bare
working-tree git diff does not include. Human editing and committing
remain pending; these pointers become valid after the complete
package is committed.
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/__init__.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_api.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_engine.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_display.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_fsp.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/data/_sampling.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend_fsp.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_jev.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime_feed.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend_mtf.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_shm.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_reference.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_regimes.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_jev.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_regime_feed.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_graphics.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_mtf_graphics.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_sampling.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/benchmark_trend.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_regimes.py`
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_graphics.py`
The style pass added full license headers, annotations, multiline
helper docstrings, explicit expression layout and owner-qualified
regression proof descriptions. All recurrence behavior and source
credits are preserved; callback constructor defaults are explicit.

645
docs/trend_calculus.rst 100644
View File

@ -0,0 +1,645 @@
Trend Calculus: sources and Piker semantics
==========================================
This implementation follows Andrew J. Morgan's FHLS Trend Calculus.
It classifies observed price structure; it does not establish
predictive accuracy or a profitable trading rule. Research was
collected on 2026-10-04/05. The source inventory below records what
was found and distinguishes missing material from working code.
Algorithm
---------
Partition an ordered scalar series into disjoint windows containing
``n >= 2`` observations. Summarize each window by its high and low,
retaining each extremum's original coordinate. First and second in
FHLS mean the earlier and later of these two extrema. They do not
mean the opening and closing observation.
For consecutive windows ``previous`` and ``current``, define::
trend = sign(
sign(current.high - previous.high)
+ sign(current.low - previous.low)
)
The implementation uses ordinal comparisons to avoid overflowing
the differences. When the result is zero, introduce a synthetic
window joining the previous window's second extremum to the current
window's first extremum. Compare previous to synthetic, then
synthetic to current. Carry the prior direction through unresolved
zero comparisons. A complex window can confirm two opposite
reversals at the same observation; retaining only its final trend
would lose an event.
A confirmed down-to-up change selects the preceding summary's low;
an up-to-down change selects its high. Apply the same calculation to
the resulting ordered reversal points for the next reversal order.
The order counts successive reductions of the pivot stream. It is
independent of a wall-clock timeframe such as one or five seconds.
The live recurrence retains one partial window, one completed
window and the prior direction per order. Its numeric state is
``O(orders)`` regardless of history length or window size. Historical
scanning calls the same Numba recurrence. Returning a full historical
result necessarily allocates arrays proportional to history length
and order count.
A completed window emits at most two reversals. Because each configured
window contains at least two observations, two incoming reversals can
complete at most one next-order window during that source update.
Induction bounds every order to two events per source observation;
the fixed event slots therefore do not truncate a larger cascade.
Deliberate semantic choices
--------------------------
``piker.fsp._trend.TrendCalculator`` accepts scalar observations,
finite strictly increasing timestamps, and strictly increasing
nonnegative source indices. Coordinates need not be contiguous.
An OHLC candle alone cannot reveal whether its high or low happened
first; feeding fabricated high/low order would change the algorithm.
The chart FSP therefore uses close prices.
Production startup is neutral. The first completed window establishes
the baseline without inventing a pivot. ``ready`` becomes true after
two completed windows at that order, including when the direction
remains zero on flat data. Thus zero is not a calibrated "sideways"
regime probability. Readiness and direction are separate information.
Higher orders warm up only when enough lower-order pivots exist.
Equal highs and lows in ordinary FHLS windows retain their earliest
observation, matching the original Lua implementation. Synthetic
bridges have a narrower, distinct rule: when their endpoint values
are equal, high belongs to the second endpoint and low to the first.
That distinction affects pivot coordinates even when trends agree.
An optional ``lua_seed=True`` reproduces the reference's initial
comparison with the first window's last observation. It is a seed
policy for reference investigation, not a promise to reproduce all
Lua command-line startup quirks. Normal FSPs use neutral startup.
NaN clears all recurrence orders and requires fresh warmup. Infinity
and ambiguous coordinate ordering are rejected before state changes.
The scalar calculator does not guess session boundaries from elapsed
time: its caller decides whether a gap is intentional or a reset.
Prepending or revising source history requires replay from the new
origin; changing a prefix can affect every subsequent reversal.
Causality and output
--------------------
``TrendCalculator.push()`` returns all newly detected reversals.
Each record carries both coordinates:
* ``index``, ``time``, ``price`` locate the original pivot;
* ``detected_index``, ``detected_time`` locate the observation that
first confirmed it;
* ``direction`` is the new trend: ``+1`` for a bottom, ``-1`` for a
top; ``order`` starts at one.
A higher-order pivot retains its original price/time while detection
time propagates from the input observation that completed its
higher-order window. Strategies must use detection time, not the
earlier pivot time, when deciding what information was available.
``TrendCalculator.extend()`` returns per-observation ``trends`` and
``ready`` arrays. ``pulses`` stores only the last detected direction
per order and row. ``pivot_prices`` and ``pivot_times`` retain both
possible sides: axis two is top then bottom, with NaN for no event.
Use complete push events when the event stream itself is required.
The ``trend_calculus`` FSP exposes four orders through ``trend``,
``trend_2``, ``trend_3`` and ``trend_4``. Its ``ready`` field counts
ready orders. Each order also exposes ``top_N``, ``top_time_N``,
``bottom_N`` and ``bottom_time_N``. Pivot prices are published at
their detection row, with the paired field retaining pivot time.
Its ``window`` argument changes the first-order close window only:
the four sizes are ``(window, 2, 2, 2)``, defaulting to all twos.
The numerical and timeframe APIs accept explicit per-order tuples.
Piker's source SHM timestamps name candle starts. Historical rows
preceding the last row are finalized; the last row is provisional.
Quotes and source sampler steps preview that row from a copy of
committed recurrence state. These wakeups inspect authoritative source
SHM; they are not additional scalar observations. Each newly finalized
row advances committed state once, including when several steps arrive
before a wakeup. Sparse pivot fields clear when the destination steps;
source revisions require a cascade replay.
For realtime regime consumers, use finalized closes with an explicit
interval-end clock. ``piker.fsp._regime.MultiTimeframeTrend`` forms
epoch-aligned buckets from complete base bars, discards partial
startup buckets and resets recurrence after missing base bars. Its
snapshots distinguish timeframe period, recursive order, source
revision, sequence, gap segment and availability time. An old model
response must not overwrite newer source evidence.
The scalar calculator and current SHM ``trend_calculus`` FSP count
observations without inferring resets from elapsed timestamp gaps;
NaN explicitly resets their recurrence. ``MultiTimeframeTrend`` instead
requires contiguous base bars and starts a new segment after a missing
interval. Consequently, the simple one-second FSP and the one-second
timeframe tracker can differ across session gaps. That difference is
an ingestion policy, not a different FHLS recurrence.
Using the numerical and timeframe APIs
--------------------------------------
The numerical API has no provider, GUI or credential requirement.
This complete example confirms both sides of a complex window:
.. code-block:: python
from piker.fsp._trend import TrendCalculator
calculator = TrendCalculator((2, 2, 2))
prices: list[float] = [1., 2., 4., 3., 2., 5.]
row: int
price: float
for row, price in enumerate(prices):
update = calculator.push(
1_700_000_000. + row,
price,
index=100 + row,
)
print(update.trends)
print(update.reversals[
['index', 'direction', 'detected_index']
])
The last update has a top at source index 102 and a bottom at 104,
both detected at 105. A consumer only learns these events when it
receives that update. For batch analysis, pass aligned one-dimensional
``values``, ``times`` and optional signed integer ``indices`` arrays
to ``calculator.extend()``. Use a fresh calculator for an independent
historical replay; ``extend()`` advances an existing instance.
For wall-clock timeframes, submit each completed base bar once, using
its opening timestamp. The tracker converts this to availability at
the interval end:
.. code-block:: python
from piker.fsp._regime import MultiTimeframeTrend
tracker = MultiTimeframeTrend(
'synthetic.test',
base_period=1.,
periods=(1., 5.),
windows=(2, 2, 2),
)
prices: list[float] = [100., 101., 99., 103., 102.] * 4
row: int
price: float
for row, price in enumerate(prices):
snapshot = tracker.push(
1_700_000_000. + row,
price,
index=100 + row,
)
state: dict = snapshot.to_state()
These periods describe independent close series; ``windows`` still
counts observations and reversal points. Subsecond periods must be
whole microseconds, and every configured period must be a whole
multiple of ``base_period``. Startup midway through a larger bucket
does not produce a shortened first observation. On history repair,
call ``tracker.reset()`` and replay the corrected completed bars.
Source inventory
----------------
Project research and earlier implementation work:
* `Piker #79`_ is the older trend-detection issue; `Piker #103`_ is
the consolidated research entrypoint. It links Morgan's paper,
Lua, Scala, a working mathematical manuscript, and a public
TradingView/Pine implementation.
* `Piker #187`_ explicitly proposes Trend Calculus in the native FSP
set and links the planned Python implementation.
* `Piker #325`_ connects trend extrema computation with faster chart
y-range reductions. Its moving min/max proposal concerns
performance; it is not an alternative definition of FHLS.
* `Python implementation issue #1`_ requests offline/online support
and recurrence updates across timeframes. The ``setup`` branch
inspected at ``30afcfbd8cbb7b84d984fa2e183c6ae392da3479`` has a
placeholder ``tc`` module and Ray moving-average experiments.
Those experiments do not implement FHLS reversal detection.
Primary algorithm sources:
* `Morgan's method document`_, ``HowToStudyTrends_v1.03.pdf``:
the original conceptual account. Inspected PDF SHA-256::
2dc506a0449c13c01d2d798f120411df84e5667b611d92204ba0122e2aac7fd5
* `Original Lua program`_, ByteSumoLtd/TrendCalculus-lua commit
``7f54f9a101283cc188dbcd9282292fc3f73de8e9``,
``make/trendcalculus.lua``. Its header credits Andrew J. Morgan /
ByteSumo Limited and declares GPL version 3 or later. Inspected
source SHA-256::
c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0
* `Spark TrendCalculus2`_, commit
``3fb6ef23e985d9d6e0414d3b682afbeef51c52e8``. The source declares
Apache-2.0. It uses earliest equal high and latest equal low, and
initializes from a zero dummy point or an excluded first point.
Those choices can change reversals and recursive warmup relative
to Lua and Piker. Inspected source SHA-256::
3d0723836d230c89d99537f1075537c3bbf809cebefd12caa6a1b517e7bc98b5
* `Working mathematical manuscript`_, ``main.tex``: describes
earliest equal low and latest equal high, a fixed initial positive
trend and an initial synthetic reversal. These are different from
both the Scala code's tie choices and native neutral startup.
Its displayed reversal-sign expression also disagrees with the
following low/high selection branches. It is a working draft,
not a unique executable specification; the executable references
and independently checked cases determine the conventions here.
Inspected source SHA-256::
e7b6fdff06fbde195a81281022cce300135d88e7484a97d1b02d621c5bfc4eab
* `Spark examples`_ provide historical visualization context.
The `public Pine indicator`_ linked from Piker #103 describes two
lagged Donchian channels and their midpoints for visual analysis.
That description does not establish recursive FHLS parity. No
complete Pine source or the user's separate Pine implementation
was recovered in this search, so no Pine parity claim is made.
Local research locations inspected, relative to the user's repos
directory:
* ``lns/todo/piker/quant/trend.toml`` links the Python ``setup``
branch, authors, Piker #187 and the Ray SMA example.
* ``lns/the_legacy_tab_nightmare/trading.toml`` links the working
manuscript, Spark comparisons/PRs, examples, and a former local
``TrendCalculus/notes_redesign_amorgan.pdf``. The latter file was
not present at its recorded path.
* ``data/bookmarks/bookmarks.html`` preserves the older ByteSumo
Bitbucket project URL. Duplicate bookmarks are not independent
algorithm evidence.
A separate filename inventory across ``/home/goodboy/repos`` included
hidden and ignored files while excluding Git internals, worktrees,
virtual environments, dependency directories, caches and build outputs.
It found no ``*.pine`` or ``*.pinescript`` files, TrendCalculus-named
source files, or copy of ``notes_redesign_amorgan.pdf`` beyond this
change's new documentation and fixture.
No earlier native Trend Calculus implementation was found in the
searched Piker source, tests, docs, snippets or available research
notes. This is a bounded search result, not proof that no private
copy or unindexed history exists. User-owned research/task states
were left untouched.
Verification and reproducibility
-------------------------------
``tests/test_trend_reference.py`` uses a separate batch oracle that
materializes complete FHLS windows, inserts synthetic windows and
then reduces the expanded sequence. It never calls the production
kernel or copies its mutable recurrence state. Coverage includes:
* every eight-point sequence over three prices, for three window
tuples: 19,683 histories with ties, flat startup and inside/outside
windows; direction, readiness, pulses and both pivot sides agree;
* another 2,187 exhaustive six-point histories using the optional Lua
seed, while keeping readiness tied to two actual complete windows;
* 48 seeded 512-point random walks across both startup policies,
five recursive orders and mixed window sizes, checking every event's
source identity and detection coordinates on irregular timestamps
and source indices;
* actual unmodified Lua output for windows 2, 3, 4 and 7, covering
62 reversals, including original pivot locations and earliest
detection prefixes.
The original-Lua fixture is
``tests/_inputs/trend_calculus_lua.json``. It records source/runtime
provenance, the complete synthetic inputs, event schema and SHA-256
hashes of each input and full output CSV. To reproduce an output,
write the fixture values with header ``ticker,date,last`` and rows
``x,<zero-based-row>,<value>``, using LF line endings. Run the pinned
program on Lua 5.2.4 with ``-n<window> -r -H -p1``. For detection
times, rerun each complete-window prefix and assign newly emitted
events to that prefix's last input row. Prefix outputs must preserve
all earlier emitted rows. No Lua executable or network is needed
for these Python reference tests.
The pinned Lua CLI has observable quirks:
* Do not pass ``-h`` with its built-in column mapping. That path
consumes the first real observation as a second header, shifting
every window boundary. ``-H`` controls only output headers.
* A flat first complete window can fail while reading a nonexistent
prior trend during bridge processing.
* The simple-window reversal branch suppresses initial output until
``offset > 2*n + 1``; split-window branches use other guards.
The fixture begins with increasing complete windows to establish a
common baseline before reversals. It establishes exact parity for
those inputs, not universal startup equivalence between programs.
The native oracle separately covers the flat and complex startup
cases excluded by that reference limitation.
Integration verification is separate from mathematical parity:
* ``tests/test_trend.py`` exercises committed/preview handoff,
quote-free sampler wakeups, indexed publication and cancellation.
* ``tests/test_trend_sampling.py`` exercises the actual sampler with
a controlled clock. Repeated or backward rounded epochs must not
append duplicate timestamps, including after realtime bootstrap.
* ``tests/test_trend_shm.py`` runs actual Tractor shared memory with a
separate read-only attachment, nonzero origin, quiet sampling and
an equal-bound historical repair. A separate child actor then
attaches read-only using the transported token and checks every
repaired field. This verifies cross-process visibility after replay,
not atomicity during concurrent writes. It needs actor runtime/socket
access, but no broker or model service.
* ``tests/test_trend_regimes.py`` checks complete aligned buckets,
fractional clocks, missing intervals and revision replay;
``tests/test_regime_feed.py`` checks native source admission,
subscription ordering, quote receiver ownership and lifecycle.
* ``tests/test_jev.py`` uses mocked transport and controlled clocks
for response validation, bounded projection, pacing, retry, source
invalidation and cancellation while publication is blocked.
* ``tests/test_trend_graphics.py`` and
``tests/test_trend_mtf_graphics.py`` exercise Qt geometry, causal
timestamp mapping, revision invalidation, model expiry and teardown.
Performance measurement
-----------------------
The repository example measures compilation/warmup, batch computation,
individual committed observations and copied-state previews separately.
Run it in the existing Piker Python environment:
.. code-block:: xsh
py313/bin/python examples/benchmark_trend.py --rows 1000000 --live-rows 20000
One x86-64 run on 2026-10-05 used Python 3.13.14, NumPy 2.3.5 and
Numba 0.62.1 with four reversal orders:
=========================== ============================
Measurement Observed result
=========================== ============================
One million-row batch 0.116473 seconds
Batch throughput 8.586 million rows/second
Scalar commit, median/p99 6.963 / 10.670 microseconds
Copy and preview, median/p99 12.223 / 18.835 microseconds
Initial kernel warmup 3.004 seconds
Retained NumPy state 936 bytes
=========================== ============================
These are reproducible measurements from one machine and workload,
not latency guarantees. Retained-state bytes exclude Python objects,
compiler caches and returned history arrays. Timings exclude source
SHM copies, actor IPC, chart rendering and provider inference. The
native sampler currently timestamps periods at or below one second
with integer seconds; subsecond end-to-end operation requires a
producer with an appropriate event clock, despite the numerical and
timeframe APIs supporting fractional-second observations.
TypeSafe/Jev integration
-----------------------
The user supplied the `TypeSafe model documentation`_,
`Python SDK documentation`_ and `agent skill`_. The adapter uses
Piker's existing HTTPX/Trio stack against the documented `System One
HTTP API`_; installing the separate Python SDK is unnecessary. The
default model is pinned to ``jev-1.13.0``. Importing the adapter does
not read credentials or open a connection.
``build_regime_request(snapshot)`` produces one Choice question for
each ready timeframe, plus an independent alignment question when
multiple timeframes are ready. Candidate regimes are ``uptrend``,
``downtrend``, ``range``, ``transition`` and ``unknown``. Questions
explicitly identify state paths because question identifiers do not
reach the model. ``JevClassification`` retains every candidate's
probability, the selected label, the service's separate confidence
value, model/prompt versions, source provenance and receipt time.
These semantic judgments do not change the numerical recurrence.
Model input uses the most recent 16 closes and 16 reversals per
timeframe by default. ``evidence_limit`` configures that projection
independently of longer chart history; source snapshots stay unchanged.
Inspect a complete synthetic request, then exercise HTTP parsing with
explicitly simulated responses:
.. code-block:: xsh
py313/bin/python examples/trend_regimes.py
py313/bin/python examples/trend_regimes.py --mock
The synthetic example includes quarter-second evidence. Its ``--live``
mode sends exactly one synthetic request and reads ``TYPESAFE_API_KEY``
from the caller's environment. It consumes provider credit and makes no
broker connection. A real provider request has not been verified without
an API key; mocked results test the integration, not model quality.
The native live bridge subscribes to the sampler before copying history,
excludes the mutable final source row, and emits only the newest snapshot
after each captured history range. It catches up skipped sample steps
and invalidates old judgments before replaying a prepend, truncation,
backfill or lost sampler notice. ``run_regime_feed()`` supplies these
snapshots to a synchronous callback without requiring a model client.
Use ``serve_regimes()`` to compose that reader and a Jev worker:
.. code-block:: python
from collections.abc import Awaitable, Callable
import httpx
from piker.fsp._jev import JevClassification, JevClient
from piker.fsp._regime_feed import serve_regimes
from piker.service import open_piker_runtime
async def run_live(
fqme: str,
api_key: str,
publish: Callable[[JevClassification], Awaitable[None]],
) -> None:
async with (
open_piker_runtime('trend-regimes'),
httpx.AsyncClient() as http,
):
await serve_regimes(
fqme,
JevClient(http, api_key),
publish,
periods=(1., 5., 15.),
min_interval=1.,
max_age=5.,
max_source_lag=2.,
max_source_age=5.,
)
Calling this function opens the configured broker feed and sends live
inference requests. Supply an existing Piker market identifier and a
cancellation-cooperative async consumer. The example periods assume
a one-second source; omit ``periods`` to derive 1x/5x/15x periods from
the actual source. Native subsecond source periods are rejected until
the sampler has a compatible clock. A separate correctly timestamped
producer can use ``consume_regime_source()`` directly.
The worker permits one in-flight request and retains only the newest
pending snapshot. ``min_interval`` limits request start frequency;
``max_age`` limits local submission age, ``max_source_lag`` limits
event-time lag, and ``max_source_age`` filters stale wall-clock source
data before submission. Setting the last option to ``None`` admits
historical replay. Source revision or gap-segment changes revoke old
results. A blocked publication is also cancelled when its evidence
expires or becomes stale. A downstream consumer must still check
provenance on already-delivered messages, which cannot be revoked.
Current source SHM has no transactional revision word. Copy-time bound
checks detect appends and origin changes; an in-place correction with
unchanged bounds becomes identifiable through its backfill notice.
Replay converges to that corrected source. A publication without a
local task checkpoint is not an atomic cross-process memory transaction.
Malformed/authentication responses fail explicitly. Transient failures
use bounded retries and freshness checks; the adapter bounds HTTP body
sizes and timeouts. The owning nursery closes the feed, sampler and
in-flight worker on cancellation. Neither the bridge nor the classifier
submits trades. Regime rubrics and probability usefulness still need
evaluation on recorded market data and real provider responses.
Chart overlays
--------------
Preview synthetic multiple-timeframe annotations without a broker,
model request or credentials:
.. code-block:: xsh
py313/bin/python examples/trend_graphics.py --output /tmp/trend.png
The offscreen example accepts ``--periods 1 5 15`` and ``--order 1``
and isolates its Qt/configuration state from the running application.
Enable the native order-one reversal and direction overlay in the
existing Piker configuration's ``[ui]`` section::
[ui]
trend_calculus = true
This starts one Trend Calculus writer for the chart's realtime source
and displays confirmed pivot arrows with translucent direction
rectangles. It makes no Jev request and does not read API credentials.
Hover text distinguishes the pivot's bar from the later time when
the reversal became observable. The mutable last FSP row is excluded
from confirmed markers and direction runs.
Within the existing chart owner's nursery,
``await FspAdmin.open_trend_overlay(order=2)`` attaches another
recursive order while reusing that admin's writer. Optional ``chart``
and ``chart_period_s`` arguments attach the source evidence to another
price chart. Changing the chart period does not recompute the source
at that period: its actual realtime sampling period stays explicit.
For independent wall-clock timeframes on the native realtime chart,
configure::
[ui]
trend_timeframes = [1, 5, 15]
This option takes precedence over ``trend_calculus = true``. It uses
one ``MultiTimeframeTrend`` reader and one ``SnapshotTrendOverlay``
with separate marker/rectangle layers for each period. It opens no Jev
client. ``await FspAdmin.open_multi_timeframe_trends(...)`` exposes the
same integration programmatically and accepts a recursive ``order``.
All periods must be multiples of the actual realtime source period.
Snapshots preserve interval-end pivot and detection times. Markers
attach to the base bar that contributed each aggregated close. A
bootstrap snapshot contains recent reversals and the latest direction,
so its deterministic rectangle history starts with that observed
direction. It does not reconstruct earlier direction runs from one
snapshot. Skipped complete buckets leave unobserved intervals unpainted.
Source revisions and missing-base-bar segments revoke prior annotations.
An existing caller-owned client can share this exact evidence with
the model worker inside the chart's nursery:
.. code-block:: python
import trio
from piker.fsp._jev import JevClient, RegimeClassifier
from piker.ui._fsp import FspAdmin
from piker.ui._trend_mtf import SnapshotTrendOverlay
async def attach_classifier(
admin: FspAdmin,
client: JevClient,
nursery: trio.Nursery,
) -> tuple[RegimeClassifier, SnapshotTrendOverlay]:
worker: RegimeClassifier = RegimeClassifier(client)
overlay = await admin.open_multi_timeframe_trends(
periods=(1., 5., 15.),
on_snapshot=worker.submit,
on_invalidate=worker.invalidate,
max_source_age=5.,
classification_ttl=5.,
)
nursery.start_soon(worker.run, overlay.publish_classification)
return worker, overlay
Graphics retain their longer deterministic history while the client
projects bounded model evidence. The source-age gate applies to the
optional model callback, so historical chart bootstrap still works.
The adapter checks market, revision, segment, sequence and source lag
before accepting a model response. Classified runs start at receipt
and expire after the configured TTL; they cannot paint future history
before the displayed event clock reaches receipt. Cancelling the chart
owner removes all layers and stops its sampler reader.
For computations that already publish separate timeframe SHMs, create
one ``piker.ui._trend.TrendOverlay`` per destination SHM with its own
``source_period_s``, target ``chart_period_s`` and ``order``. Mapping
uses timestamps and preserves gaps rather than snapping missing
observations onto the next candle. Geometry counts are bounded and
items are reused instead of adding individual Qt items on every tick.
Call ``update(force=True)`` after a matching destination or chart
history revision, and ``remove()`` when its owner exits. Normal
``FspAdmin`` integration routes destination-specific revisions and
teardown automatically; direct attachments must retain this ownership.
Hidden widgets preserve revision invalidations until their next
visible update. Repainting follows quotes and the existing one-second
sampler wakeup. If that wakeup precedes asynchronous FSP publication,
a quote-free finalization can appear on the next wakeup, roughly one
second later; the integration does not emit IPC for every output row.
For model annotations, ``classification_interval()`` converts a
matching ``JevClassification`` into a ``RegimeInterval``. Supply the
chart's current market, source revision, timeframe and display end.
The interval starts no earlier than the model response's receipt.
``overlay.set_regimes(intervals)`` displays an owner-maintained,
bounded collection of these intervals. An empty collection clears
model rectangles; ``None`` restores deterministic direction runs.
The caller remains responsible for response ordering and expiry.
.. _Piker #79: https://github.com/pikers/piker/issues/79
.. _Piker #103: https://github.com/pikers/piker/issues/103
.. _Piker #187: https://github.com/pikers/piker/issues/187
.. _Piker #325: https://github.com/pikers/piker/issues/325
.. _Python implementation issue #1: https://github.com/bytesumo/TrendCalculus/issues/1
.. _Morgan's method document: https://github.com/bytesumo/TrendCalculus/blob/3d5c598cd3764a40633979c4c6f4188191dd81ef/HowToStudyTrends_v1.03.pdf
.. _Original Lua program: https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua
.. _Spark TrendCalculus2: https://github.com/lamastex/spark-trend-calculus/blob/3fb6ef23e985d9d6e0414d3b682afbeef51c52e8/src/main/scala/org/lamastex/spark/trendcalculus/TrendCalculus2.scala
.. _Working mathematical manuscript: https://github.com/lamastex/working-manuscript-trend-calculus/blob/master/main.tex
.. _Spark examples: https://lamastex.github.io/spark-trend-calculus-examples/
.. _public Pine indicator: https://www.tradingview.com/script/ohSdmhS0-TrendCalculus/
.. _TypeSafe model documentation: https://docs.typesafe.ai/models
.. _Python SDK documentation: https://docs.typesafe.ai/sdk/python
.. _agent skill: https://docs.typesafe.ai/agent-skill
.. _System One HTTP API: https://docs.typesafe.ai/api

135
examples/benchmark_trend.py 100644
View File

@ -0,0 +1,135 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Measure deterministic Trend Calculus processing without a feed.
Timings distinguish kernel warmup, batch scanning, scalar commits,
and uncommitted live previews. They exclude network/model latency.
Run from the Piker environment; no credentials or services are used.
'''
import argparse
import json
import platform
from time import perf_counter
import numba
import numpy as np
from piker.fsp._trend import (
TrendCalculator,
TrendHistory,
TrendUpdate,
)
def benchmark(
rows: int,
live_rows: int,
) -> dict[str, str|int|float|dict[str, float]]:
'''
Return reproducible timings and the retained NumPy state size.
``rows`` controls the ``TrendCalculator.extend()`` batch size.
``live_rows`` controls timed calls to ``TrendCalculator.push()``
and previews made through ``TrendCalculator.copy()``. Assertions
compare directions across all three paths outside timed regions.
'''
rng: np.random.Generator = np.random.default_rng(20261004)
values: np.ndarray = rng.normal(size=rows).cumsum() + 100.
times: np.ndarray = np.arange(rows, dtype=float) * .25
calc: TrendCalculator = TrendCalculator()
start: float = perf_counter()
calc.extend(values[:8], times[:8])
calc.push(times[8], values[8])
warmup: float = perf_counter() - start
start = perf_counter()
full: TrendHistory = TrendCalculator().extend(values, times)
batch: float = perf_counter() - start
calc = TrendCalculator()
elapsed: np.ndarray = np.empty(live_rows)
preview_elapsed: np.ndarray = np.empty(live_rows)
number: int
for number in range(live_rows):
start = perf_counter()
preview: TrendUpdate = calc.copy().push(
times[number], values[number],
)
preview_elapsed[number] = perf_counter() - start
start = perf_counter()
update: TrendUpdate = calc.push(
times[number], values[number],
)
elapsed[number] = perf_counter() - start
assert preview.trends == update.trends
assert update.trends == tuple(full.trends[number])
def percentiles_us(samples: np.ndarray) -> dict[str, float]:
'''
Report median and tail operation durations in microseconds.
'''
return dict(zip(
('p50', 'p95', 'p99'),
np.percentile(samples * 1_000_000, (50, 95, 99)),
))
array: np.ndarray # noqa: F842
return {
'python': platform.python_version(),
'machine': platform.machine(),
'numpy': np.__version__,
'numba': numba.__version__,
'rows': rows,
'live_rows': live_rows,
'orders': len(calc.windows),
'kernel_warmup_s': warmup,
'batch_s': batch,
'batch_rows_per_s': rows / batch,
'scalar_commit_us': percentiles_us(elapsed),
'preview_copy_and_push_us': percentiles_us(preview_elapsed),
'retained_numpy_state_bytes': sum(
array.nbytes for array in (
calc._sizes, calc._bars, calc._counters,
calc._events, calc._counts,
)
),
}
def main() -> None:
'''
Validate CLI sizes and print benchmark measurements as JSON.
'''
parser: argparse.ArgumentParser = argparse.ArgumentParser(
description=__doc__,
)
parser.add_argument('--rows', type=int, default=1_000_000)
parser.add_argument('--live-rows', type=int, default=20_000)
args: argparse.Namespace = parser.parse_args()
if args.rows < 9 or not 1 <= args.live_rows <= args.rows:
parser.error('require rows >= 9 and 1 <= live-rows <= rows')
print(json.dumps(benchmark(args.rows, args.live_rows), indent=2))
if __name__ == '__main__':
main()

235
examples/trend_graphics.py 100644
View File

@ -0,0 +1,235 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Render an explicitly synthetic multiple-timeframe trend PNG.
Run with ``--output /tmp/piker-trend-graphics.png``. This uses the
actual shared tracker and snapshot graphics adapter. It launches
no visible GUI, opens no broker or Jev connection, reads no API key,
and isolates Qt/Piker configuration in a disposable directory.
'''
import argparse
import json
import os
from pathlib import Path
from tempfile import TemporaryDirectory
from types import SimpleNamespace
def render(
output: Path,
*,
periods: tuple[float, ...],
order: int,
samples: int,
) -> dict[str, object]:
'''
Replay synthetic closes through one actual tracker and renderer.
The final OHLC row remains mutable and is never submitted as a
completed observation. Each timeframe retains real aggregation
and TC state, while all plotted marker timestamps identify their
final contributing base bar. Shading is deterministic TC output,
not fabricated model inference.
'''
import numpy as np
import pyqtgraph as pg
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
from piker.ui._trend_mtf import (
SnapshotTrendOverlay,
_SnapshotLayer,
)
from piker.ui.qt import QApplication
app: QApplication = QApplication(['piker-trend-graphics'])
app.setQuitOnLastWindowClosed(False)
steps: np.ndarray = np.arange(samples)
prices: np.ndarray = (
100 + .009 * steps
+ 1.5 * np.sin(steps * .06)
+ .38 * np.sin(steps * .43)
)
rows: np.ndarray = np.empty(samples, dtype=[
('index', 'i8'),
('time', 'f8'),
('close', 'f8'),
('low', 'f8'),
('high', 'f8'),
])
rows['index'] = 1000 + steps
rows['time'] = 1_700_000_100 + steps
rows['close'] = prices
rows['low'] = prices - .12
rows['high'] = prices + .12
shm: SimpleNamespace = SimpleNamespace(
array=rows,
_token='synthetic-demo-ohlcv',
)
widget: pg.PlotWidget = pg.PlotWidget()
widget.resize(1400, 760)
widget.setBackground('#101923')
widget.setTitle(
'SYNTHETIC: Trend Calculus across actual timeframes'
'<br><span style="font-size:11pt;color:#aab7c5">'
'Larger triangles represent slower sampling periods; '
'shading begins at confirmed regime detection.'
'</span>',
color='#dce5ed',
size='16pt',
)
widget.plotItem.titleLabel.setFixedHeight(60)
widget.showGrid(x=True, y=True, alpha=.13)
widget.setLabel('left', 'Synthetic price')
widget.setLabel('bottom', 'Absolute base-bar index (1 second)')
widget.plot(
rows['index'],
prices,
pen=pg.mkPen('#dce5ed', width=1.5),
)
left: int = max(0, samples - 400)
widget.setXRange(1000 + left, 1000 + samples, padding=0)
widget.setYRange(
float(prices[left:].min()) - .6,
float(prices[left:].max()) + .6,
padding=0,
)
viz: SimpleNamespace = SimpleNamespace(
plot=widget.plotItem,
shm=shm,
index_field='index',
)
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.demo',
periods=periods,
windows=(2, 2, 2, 2),
history_limit=256,
)
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
viz,
shm,
fqme='synthetic.demo',
base_period=1,
periods=periods,
order=order,
)
try:
row: np.void
snapshot: RegimeSnapshot
for row in rows[:-1]:
snapshot = tracker.push(
float(row['time']),
float(row['close']),
index=int(row['index']),
)
adapter.submit(snapshot)
legend: pg.LegendItem = widget.addLegend(offset=(12, 12))
period: float
layer: _SnapshotLayer
for period, layer in adapter.layers.items():
legend.addItem(
pg.ScatterPlotItem(
symbol='t1',
size=layer.markers.opts['size'],
brush='#46c7a4',
pen=pg.mkPen(None),
),
f'{period:g}s / order {order}',
)
widget.show()
app.processEvents()
output.parent.mkdir(parents=True, exist_ok=True)
if not widget.grab().save(str(output)):
raise RuntimeError('Qt could not save the requested PNG')
return {
'mode': 'synthetic_native_graphics_no_model_calls',
'output': str(output),
'base_period': 1,
'periods': periods,
'order': order,
'finalized_bars': samples - 1,
'last_event_end': snapshot.as_of,
'layers': {
period: {
'markers': len(layer.points),
'rectangles': len(layer.rectangles.intervals),
}
for period, layer in adapter.layers.items()
},
}
finally:
adapter.remove()
widget.close()
widget.deleteLater()
app.processEvents()
def main() -> None:
'''
Require an output path and isolate offscreen configuration.
'''
parser: argparse.ArgumentParser = argparse.ArgumentParser(
description=__doc__,
)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument(
'--periods', nargs='+', type=float, default=[1, 5, 15],
)
parser.add_argument(
'--order', type=int, choices=range(1, 5), default=1,
)
parser.add_argument('--samples', type=int, default=600)
args: argparse.Namespace = parser.parse_args()
if not 120 <= args.samples <= 10_000:
parser.error('--samples must be between 120 and 10000')
prior: dict[str, str|None] = {
'QT_QPA_PLATFORM': os.environ.get('QT_QPA_PLATFORM'),
'XDG_CONFIG_HOME': os.environ.get('XDG_CONFIG_HOME'),
}
try:
path: str
with TemporaryDirectory(prefix='piker-trend-') as path:
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
os.environ['XDG_CONFIG_HOME'] = path
result: dict[str, object] = render(
args.output.resolve(),
periods=tuple(args.periods),
order=args.order,
samples=args.samples,
)
print(json.dumps(result, indent=2))
finally:
key: str
value: str|None
for key, value in prior.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
if __name__ == '__main__':
main()

165
examples/trend_regimes.py 100644
View File

@ -0,0 +1,165 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Inspect synthetic timeframe evidence and optional Jev inference.
The default prints the exact request without reading credentials or
opening a network connection. ``--mock`` tests the HTTP path locally.
``--live`` sends exactly one synthetic snapshot using the explicitly
configured ``TYPESAFE_API_KEY``. No broker connection is necessary.
'''
import argparse
from dataclasses import asdict
import json
from math import floor, sin
import os
import time
import httpx
import trio
from piker.fsp._jev import (
DEFAULT_MODEL,
JevClassification,
JevClient,
build_regime_request,
)
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
def synthetic_snapshot() -> RegimeSnapshot:
'''
Aggregate quarter-second closes ending before the current time.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.demo',
base_period=.25,
periods=(.25, 1, 5),
windows=(2, 2, 2),
)
count: int = 960
start: float = floor(time.time()) - count * .25
number: int
for number in range(count):
price: float = (
100 + number * .01
+ 2 * sin(number / 12)
+ .6 * sin(number / 3)
)
snapshot: RegimeSnapshot = tracker.push(
start + number * .25, price,
)
return snapshot
def mock_reply(request: httpx.Request) -> httpx.Response:
'''
Demonstrate transport and parsing with explicitly fake labels.
'''
payload: dict = json.loads(request.content)
answers: dict[str, dict] = {}
name: str
question: dict
for name, question in payload['questions'].items():
options: dict = question['criteria']
probabilities: dict[str, float] = {}
option: str
for option in options:
probabilities[option] = float(option == 'unknown')
answers[name] = {
'type': 'choice',
'choice': 'unknown',
'probabilities': probabilities,
'confidence': 1.0,
}
return httpx.Response(200, json={
'model': payload['model'],
'answers': answers,
'usage': {
'input_tokens': 0,
'output_tokens': 0,
},
})
async def classify(
snapshot: RegimeSnapshot,
model: str,
live: bool,
) -> None:
'''
Print one explicitly live or simulated result with provenance.
'''
if live:
key: str = os.environ.get('TYPESAFE_API_KEY', '')
if not key:
raise SystemExit('Set TYPESAFE_API_KEY before --live')
transport: httpx.AsyncBaseTransport|None = None
else:
key = 'offline-demo-key'
transport = httpx.MockTransport(mock_reply)
http: httpx.AsyncClient
async with httpx.AsyncClient(transport=transport) as http:
client: JevClient = JevClient(http, key, model=model)
result: JevClassification = await client.classify(snapshot)
output: dict = {
'mode': 'live_synthetic' if live else 'mock_not_inference',
'classification': asdict(result),
}
print(json.dumps(output, indent=2, allow_nan=False))
def main() -> None:
'''
Require an explicit flag before any credential or network access.
'''
parser: argparse.ArgumentParser = argparse.ArgumentParser(
description=__doc__,
)
mode: argparse._MutuallyExclusiveGroup = (
parser.add_mutually_exclusive_group()
)
mode.add_argument(
'--live', action='store_true',
help='Send one synthetic TypeSafe request (uses API credit)',
)
mode.add_argument(
'--mock', action='store_true',
help='Exercise the adapter locally using simulated answers',
)
parser.add_argument('--model', default=DEFAULT_MODEL)
args: argparse.Namespace = parser.parse_args()
snapshot: RegimeSnapshot = synthetic_snapshot()
if args.live or args.mock:
trio.run(classify, snapshot, args.model, args.live)
else:
print(json.dumps(
build_regime_request(snapshot, model=args.model),
indent=2,
allow_nan=False,
))
if __name__ == '__main__':
main()

6
piker/data/_sampling.py
View File

@ -202,6 +202,12 @@ class Sampler:
if shm_period_s <= 1:
next_t = i_epoch
# Startup may seed a bar one second
# ahead; rounded wall time can repeat or
# move backward. Keep the mutable bar
# until a strictly newer epoch arrives.
if next_t <= t:
continue
# this copies non-std fields (eg. vwap) from the
# last datum

12
piker/fsp/__init__.py
View File

@ -35,6 +35,14 @@ from ._volume import (
flow_rates,
tina_vwap,
)
from ._trend import (
TrendCalculator as TrendCalculator,
TrendHistory as TrendHistory,
TrendUpdate as TrendUpdate,
)
from ._trend_fsp import (
trend_calculus as trend_calculus,
)
__all__: list[str] = [
'cascade',
@ -44,6 +52,10 @@ __all__: list[str] = [
'dolla_vlm',
'flow_rates',
'tina_vwap',
'TrendCalculator',
'TrendHistory',
'TrendUpdate',
'trend_calculus',
]

51
piker/fsp/_api.py
View File

@ -28,6 +28,7 @@ from __future__ import annotations
from typing import (
AsyncGenerator,
Callable,
NamedTuple,
overload,
Protocol,
)
@ -48,7 +49,25 @@ log = get_logger(__name__)
type FspHistory = dict[str, np.ndarray|None]|np.ndarray
type RealtimeValue = np.ndarray|np.number|int|float
type FspYield = FspHistory|tuple[str, RealtimeValue]
class FspSample(NamedTuple):
'''
Publish all FSP fields at an absolute source coordinate.
Unlike a field-only tick update, this record remains valid if the
sampler has not appended its destination row yet.
``FspSample.fields`` are written without task checkpoints.
This is not an atomic memory transaction for readers in other
processes.
'''
index: int
time: float
fields: dict[str, RealtimeValue]
type FspYield = FspHistory|tuple[str, RealtimeValue]|FspSample
type FspStream = AsyncGenerator[FspYield, None]
type FspConfigValue = str|bool|int|float
@ -76,6 +95,7 @@ def _load_builtins() -> dict[NamespacePath, Fsp]:
# import to implicity trigger registration via ``@fsp``
from . import _momo # noqa
from . import _volume # noqa
from . import _trend_fsp # noqa
return _fsp_registry
@ -108,6 +128,8 @@ class Fsp[**P]:
*,
outputs: tuple[str, ...] = (),
display_name: str|None = None,
sample_on_step: bool = False,
event_outputs: tuple[str, ...] = (),
**config: FspConfigValue,
) -> None:
@ -120,6 +142,12 @@ class Fsp[**P]:
self.ns_path: NamespacePath = NamespacePath.from_ref(func)
self.outputs = outputs
# Snapshot processors read SHM on both quote and timer steps.
# Sparse event fields must never be carried into a new row.
self.sample_on_step: bool = sample_on_step
if not set(event_outputs).issubset(outputs):
raise ValueError('event outputs must be declared')
self.event_outputs: tuple[str, ...] = event_outputs
self.config: dict[str, FspConfigValue] = config
# register with declared set.
@ -181,6 +209,8 @@ def fsp[**P](
*,
outputs: tuple[str, ...] = (),
display_name: str|None = None,
sample_on_step: bool = False,
event_outputs: tuple[str, ...] = (),
**config: FspConfigValue,
) -> Fsp[P]:
@ -193,6 +223,8 @@ def fsp[**P](
*,
outputs: tuple[str, ...] = (),
display_name: str|None = None,
sample_on_step: bool = False,
event_outputs: tuple[str, ...] = (),
**config: FspConfigValue,
) -> Callable[[FspFunc[P]], Fsp[P]]:
@ -204,6 +236,8 @@ def fsp[**P](
*,
outputs: tuple[str, ...] = (),
display_name: str|None = None,
sample_on_step: bool = False,
event_outputs: tuple[str, ...] = (),
**config: FspConfigValue,
) -> Fsp[P]|Callable[[FspFunc[P]], Fsp[P]]:
@ -214,6 +248,10 @@ def fsp[**P](
``Fsp``. Parameterized ``@fsp(...)`` returns ``decorate`` first;
Python then passes the decorated function to that closure.
``sample_on_step`` opts into coalesced SHM snapshot wakeups from
quotes and sampler steps; the quote payload is not preserved.
``event_outputs`` names sparse fields cleared to NaN on append.
'''
if wrapped is None:
@ -222,12 +260,21 @@ def fsp[**P](
func,
outputs=outputs,
display_name=display_name,
sample_on_step=sample_on_step,
event_outputs=event_outputs,
**config,
)
return decorate
return Fsp(wrapped, outputs=(wrapped.__name__,))
return Fsp(
wrapped,
outputs=outputs or (wrapped.__name__,),
display_name=display_name,
sample_on_step=sample_on_step,
event_outputs=event_outputs,
**config,
)
def maybe_mk_fsp_shm(

162
piker/fsp/_engine.py
View File

@ -35,6 +35,7 @@ import trio
from trio_typing import TaskStatus
import tractor
from tractor.msg import NamespacePath
from tractor.trionics import Lagged
from piker.types import Struct
from ..log import (
@ -56,6 +57,7 @@ from ._api import (
FspStream,
FspYield,
FspHistory,
FspSample,
_load_builtins,
NDToken,
)
@ -145,6 +147,88 @@ async def filter_quotes_by_sym(
yield quote
@acm
async def _sample_wakeups(
quotes: AsyncIterator[QuoteFrame],
fqme: str,
enabled: bool,
) -> AsyncIterator[
tuple[AsyncIterator[QuoteFrame], Callable[[], None]|None]
]:
'''
Merge quote and sampler wakeups for SHM snapshot processors.
One pending notification is enough: the processor reads every
newly finalized SHM row when woken. Keep this queue outside the
restartable compute task, so history resync cannot lose a step.
Tick-sensitive processors retain their original quote stream.
'''
if not enabled:
yield quotes, None
return
send: trio.MemorySendChannel[QuoteFrame]
receive: trio.MemoryReceiveChannel[QuoteFrame]
send, receive = trio.open_memory_channel[QuoteFrame](1)
def notify() -> None:
'''
Coalesce sampler steps in the local ``send`` channel.
'''
try:
send.send_nowait({fqme: {'ticks': []}})
except trio.WouldBlock:
pass
async def relay() -> None:
'''
Resume after quote overrun and wake the SHM snapshot reader.
'''
iterator: AsyncIterator[QuoteFrame] = aiter(quotes)
while True:
try:
frame: QuoteFrame = await anext(iterator)
except Lagged:
# Dropped quote payloads do not invalidate SHM.
# Wake the snapshot processor to catch up all rows.
notify()
await trio.lowlevel.checkpoint()
continue
except StopAsyncIteration:
return
if fqme in frame:
notify()
nursery: trio.Nursery
async with send, receive, trio.open_nursery() as nursery:
nursery.start_soon(relay)
try:
yield receive, notify
finally:
nursery.cancel_scope.cancel()
def _next_step_output(
previous: np.ndarray,
fsp: Fsp,
zero_on_step: bool,
) -> np.ndarray:
'''
Carry state into a new row without duplicating sparse events.
'''
row: np.ndarray = (
np.zeros_like(previous) if zero_on_step else previous.copy()
)
field: str
for field in fsp.event_outputs:
row[field] = np.nan
return row
class _HistoryShm:
'''
Present an immutable history view before switching to live SHM.
@ -274,6 +358,7 @@ class Cascade(Struct):
complete: trio.Event|None = None
cs: trio.CancelScope|None = None
client_stream: tractor.MsgStream|None = None
sample_ready: trio.Event|None = None
async def resync(self) -> int:
# TODO: adopt an incremental update engine/approach
@ -380,7 +465,16 @@ class Cascade(Struct):
) -> int:
self.bind_func = bind_func
index = await self.tn.start(bind_func)
try:
yield index
finally:
# Stop the current compute task, including after resync,
# before the outer scope closes its source receiver.
if self.cs is not None:
self.cs.cancel()
if self.complete is not None:
with trio.CancelScope(shield=True):
await self.complete.wait()
# TODO: what do we want on teardown/error?
# -[ ] dynamic reconnection after update?
@ -390,6 +484,39 @@ async def _close_fsp_stream(out_stream: FspStream) -> None:
await out_stream.aclose()
async def _write_sample(
casc: Cascade,
sample: FspSample,
) -> bool:
'''
Wait for destination advancement, then publish at source index.
``cascade()`` sets ``Cascade.sample_ready`` after publishing the
new destination bound. Capture that event before checking bounds
so a sampler wakeup cannot be lost. Cancellation during history
resync aborts this wait with the obsolete compute generator.
'''
dst: ShmArray = casc.dst.rt_shm
while True:
ready: trio.Event|None = casc.sample_ready
if ready is None:
ready = casc.sample_ready = trio.Event()
first: int = dst._first.value
last: int = dst._last.value
if sample.index < first:
return False
if sample.index < last:
row: np.void = dst._array[sample.index]
field: str
value: Any
for field, value in sample.fields.items():
row[field] = value
row['time'] = sample.time
return True
await ready.wait()
def _write_history_output(
dst_shm: ShmArray,
history_output: FspHistory,
@ -577,6 +704,9 @@ async def connect_streams(
async for processed in out_stream:
log.debug(f"{func_name}: {processed}")
if isinstance(processed, FspSample):
await _write_sample(casc, processed)
continue
key, output = processed
# dst.array[-1][key] = output
dst_shm.array[[key, 'time']][-1] = (
@ -711,6 +841,10 @@ async def cascade(
) as feed:
flume: Flume = feed.flumes[fqme]
# Cached feed consumers can replace ``Flume.stream`` when
# opening another subscription. Retain this scope's receiver
# before nursery setup gives another consumer a checkpoint.
quote_source: AsyncIterator[QuoteFrame] = flume.stream
# XXX: can't do this since flume.feed will be set XD
# assert flume == src
assert flume.mkt == src.mkt
@ -728,9 +862,17 @@ async def cascade(
profiler(f'{func}: feed up')
func_name: str = func.__name__
tn: trio.Nursery
quote_stream: AsyncIterator[QuoteFrame]
wake_sample: Callable[[], None]|None
async with (
tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console
trio.open_nursery() as tn,
_sample_wakeups(
quote_source,
fqme,
fsp.sample_on_step,
) as (quote_stream, wake_sample),
):
# TODO: might be better to just make a "restart" method where
# the target task is spawned implicitly and then the event is
@ -748,7 +890,7 @@ async def cascade(
connect_streams,
casc=casc,
mkt=mkt,
quote_stream=flume.stream,
quote_stream=quote_stream,
# flumes and shm passthrough
src=src,
@ -777,10 +919,6 @@ async def cascade(
dst_shm: ShmArray = dst.rt_shm
src_shm: ShmArray = src.rt_shm
if zero_on_step:
last = dst.rt_shm.array[-1:]
zeroed = np.zeros(last.shape, dtype=last.dtype)
profiler(f'{func_name}: fsp up')
# sync to client-side actor
@ -842,10 +980,9 @@ async def cascade(
# some metrics like vlm should be reset
# to zero every step.
if zero_on_step:
last = zeroed
else:
last = array[-1:].copy()
last = _next_step_output(
array[-1:], fsp, zero_on_step,
)
dst.rt_shm.push(last)
@ -860,6 +997,13 @@ async def cascade(
dst_shm._array['time'][src_li] = src_lt
dst_shm._array['time'][src_2li] = src_2lt
ready: trio.Event|None = casc.sample_ready
casc.sample_ready = trio.Event()
if ready is not None:
ready.set()
if wake_sample is not None:
wake_sample()
# last2 = dst.array[-2:]
# if (
# last2[-1]['index'] != src_li

777
piker/fsp/_jev.py 100644
View File

@ -0,0 +1,777 @@
# piker: trading gear for hackers
# Copyright (C) Tyler Goodlet (in stewardship of piker0)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Optional, bounded TypeSafe System One judgments over trend evidence.
The HTTP contract is documented at https://docs.typesafe.ai/api .
``JevClient`` uses piker's existing HTTPX/Trio stack. Credentials are
explicit constructor inputs; importing this module opens no network
connection and reads no environment variables. Model judgments never
change deterministic TC state or route orders into the EMS.
'''
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, replace
from email.utils import parsedate_to_datetime
import json
from math import isclose, isfinite
from operator import index as integer_index
import time
from types import MappingProxyType
import httpx
import numpy as np
import trio
from ._regime import RegimeSnapshot, TimeframeRegime
DEFAULT_MODEL: str = 'jev-1.13.0'
PROMPT_VERSION: str = 'piker-regimes-v1'
API_URL: str = 'https://api.typesafe.ai/v1/systemone'
# Centralized human-reviewable rubric. These are descriptive model
# judgments, not new TC rules or validated profitability signals.
REGIME_CRITERIA: MappingProxyType = MappingProxyType({
'uptrend': (
'Persistent upward movement supported by recent closes and '
'ready trend orders; small counter-moves do not dominate.'
),
'downtrend': (
'Persistent downward movement supported by recent closes '
'and ready trend orders; small retracements do not dominate.'
),
'range': (
'Recent movement repeatedly retraces within a bounded range '
'without a persistent direction, or closes remain flat.'
),
'transition': (
'Recent detected reversals and conflicting ready orders '
'suggest a change of regime whose direction is unsettled.'
),
'unknown': (
'Available ready orders and recent observations provide '
'insufficient or contradictory evidence for another label.'
),
})
ALIGNMENT_CRITERIA: MappingProxyType = MappingProxyType({
'aligned_up': 'Ready timeframes consistently support an uptrend',
'aligned_down': (
'Ready timeframes consistently support a downtrend.'
),
'mixed': (
'Ready timeframes disagree in direction or regime, such as '
'a shorter pullback within a longer directional trend.'
),
'unknown': (
'There is insufficient directional evidence to assess '
'agreement across the ready timeframes.'
),
})
@dataclass(frozen=True)
class RegimeAnswer:
'''
Retain a model label and its entire probability distribution.
``RegimeAnswer.period=None`` identifies timeframe alignment.
``RegimeAnswer.probability`` is the selected label's probability;
``RegimeAnswer.confidence`` separately describes concentration.
'''
question: str
period: float|None
label: str
probability: float
confidence: float
probabilities: tuple[tuple[str, float], ...]
@dataclass(frozen=True)
class JevClassification:
'''
Associate semantic judgments with evidence and availability time.
``JevClassification.source_as_of`` is an event-time bar END.
``JevClassification.requested_at`` and
``JevClassification.received_at`` are Unix wall-clock timestamps.
A live chart must not start a classification before receipt.
``JevClassification.latency`` uses a monotonic clock; wall-clock
adjustments do not change the measured request duration.
'''
market: str
revision: int
sequence: int
segment: int
source_as_of: float
requested_at: float
received_at: float
latency: float
model: str
prompt_version: str
answers: tuple[RegimeAnswer, ...]
input_tokens: int
output_tokens: int
class JevError(Exception):
'''
Describe a provider failure without echoing credential values.
'''
class JevProtocolError(JevError):
'''
Reject malformed or semantically inconsistent provider answers.
'''
class JevServiceError(JevError):
'''
Preserve an HTTP status and a bounded retry delay, not its body.
'''
def __init__(
self,
status: int,
retry_after: float = 0,
) -> None:
'''
Retain an HTTP failure status without its response body.
'''
self.status: int = status
self.retry_after: float = retry_after
super().__init__(f'TypeSafe request failed: HTTP {status}')
@property
def retryable(self) -> bool:
'''
Identify transient statuses documented by HTTP and TypeSafe.
'''
return self.status in {408, 429, 500, 502, 503, 504, 529}
def build_regime_request(
snapshot: RegimeSnapshot,
*,
model: str = DEFAULT_MODEL,
evidence_limit: int = 16,
) -> dict:
'''
Batch independent timeframe questions over one immutable state.
Question identifiers are opaque to Jev; each instruction names
its actual state path. Unready timeframes provide context only.
``evidence_limit`` bounds each timeframe's model-input tails
independently of larger histories retained for chart rendering.
'''
if not isinstance(model, str) or not model.strip():
raise ValueError('a model identifier is required')
if (
isinstance(evidence_limit, (bool, np.bool_))
or not 1 <= integer_index(evidence_limit) <= 256
):
raise ValueError('evidence_limit must be in [1, 256]')
frames: list[TimeframeRegime] = []
frame: TimeframeRegime
for frame in snapshot.timeframes:
frames.append(replace(
frame,
recent_closes=frame.recent_closes[-evidence_limit:],
reversals=frame.reversals[-evidence_limit:],
))
evidence: RegimeSnapshot = replace(
snapshot, timeframes=tuple(frames),
)
questions: dict[str, dict] = {}
number: int
for number, frame in enumerate(snapshot.timeframes):
if not frame.ready or not frame.ready[0]:
continue
questions[f'timeframe_{number}'] = {
'type': 'choice',
'instructions': (
f'Which descriptive market regime best fits '
f'`timeframes[{number}]` at its `as_of` time? '
f'Use only its observed recent closes, ready trend '
f'orders and detected reversals. TC order numbers '
f'are recursive reversal scales, not seconds. '
f'Zero is unresolved, not automatically a range. '
f'Ignore unready orders. A pivot was only known at '
f'its detected_time. Do not predict future returns '
f'or make a trading recommendation.'
),
'criteria': dict(REGIME_CRITERIA),
}
if not questions:
raise ValueError('no timeframe has completed TC warm-up')
if len(questions) > 1:
questions['alignment'] = {
'type': 'choice',
'instructions': (
'How do directions across the ready entries in '
'`timeframes` agree at `as_of`? Use only observed '
'closes and ready TC orders. Each timeframe may end '
'earlier than the overall as_of; do not invent '
'partial buckets or rely on other question answers.'
),
'criteria': dict(ALIGNMENT_CRITERIA),
}
return {
'model': model,
'state': evidence.to_state(),
'questions': questions,
}
def _probability(value: object) -> float:
'''
Enforce finite unit-interval probabilities without coercing text.
'''
if (
isinstance(value, bool)
or not isinstance(value, (float, int))
or not isfinite(value)
or not 0 <= value <= 1
):
raise JevProtocolError('invalid probability or confidence')
return float(value)
def _retry_after(value: str|None) -> float:
'''
Interpret Retry-After seconds or an HTTP date with a hard bound.
'''
if value is None:
return 0
try:
delay: float = float(value)
except ValueError:
try:
delay = parsedate_to_datetime(value).timestamp()
delay -= time.time()
except (
TypeError,
ValueError,
OverflowError,
):
return 0
if not isfinite(delay):
return 0
return min(60, max(0, delay))
class JevClient:
'''
Perform one bounded, validated HTTP evaluation on the Trio loop.
The caller owns ``http`` and must close it. Retries and pacing
belong to ``RegimeClassifier``, so a single evaluation never
silently resends stale state. Redirects are disabled to keep
the explicit credential bound to the TypeSafe API origin.
'''
def __init__(
self,
http: httpx.AsyncClient,
api_key: str,
*,
model: str = DEFAULT_MODEL,
timeout: float = 5,
max_request_bytes: int = 65_536,
max_response_bytes: int = 131_072,
evidence_limit: int = 16,
) -> None:
'''
Bind a caller-owned HTTP session, key and request limits.
The caller retains ownership of the supplied ``http`` client.
'''
if not isinstance(api_key, str) or not api_key.strip():
raise ValueError('explicit TypeSafe API key required')
if not isfinite(timeout) or timeout <= 0:
raise ValueError('timeout must be positive and finite')
if max_request_bytes <= 0 or max_response_bytes <= 0:
raise ValueError('byte limits must be positive')
if not isinstance(model, str) or not model.strip():
raise ValueError('a model identifier is required')
if (
isinstance(evidence_limit, (bool, np.bool_))
or not 1 <= integer_index(evidence_limit) <= 256
):
raise ValueError('evidence_limit must be in [1, 256]')
self._http: httpx.AsyncClient = http
self._api_key: str = api_key
self.model: str = model
self.timeout: float = timeout
self.max_request_bytes: int = max_request_bytes
self.max_response_bytes: int = max_response_bytes
self.evidence_limit: int = int(evidence_limit)
async def classify(
self,
snapshot: RegimeSnapshot,
) -> JevClassification:
'''
Validate one complete response against the exact sent rubric.
'''
payload: dict = build_regime_request(
snapshot, model=self.model,
evidence_limit=self.evidence_limit,
)
body: bytes = json.dumps(
payload,
allow_nan=False,
separators=(',', ':'),
).encode()
if len(body) > self.max_request_bytes:
raise ValueError('regime request exceeds byte budget')
requested_at: float = time.time()
started: float = trio.current_time()
data: bytearray = bytearray()
api_key: str = self._api_key
response: httpx.Response
with trio.fail_after(self.timeout):
async with self._http.stream(
'POST',
API_URL,
content=body,
headers={
'Authorization': f'Bearer {api_key}',
'Content-Type': 'application/json',
},
timeout=self.timeout,
follow_redirects=False,
) as response:
if response.status_code != 200:
raise JevServiceError(
response.status_code,
_retry_after(
response.headers.get('retry-after'),
),
)
chunk: bytes
async for chunk in response.aiter_bytes():
data.extend(chunk)
if len(data) > self.max_response_bytes:
raise JevProtocolError(
'response exceeds the byte budget',
)
received_at: float = time.time()
latency: float = trio.current_time() - started
error: ValueError|UnicodeError
try:
decoded: object = json.loads(data)
except (
ValueError,
UnicodeError,
) as error:
raise JevProtocolError('invalid JSON reply') from error
return self._parse(
snapshot,
payload,
decoded,
requested_at,
received_at,
latency,
)
def _parse(
self,
snapshot: RegimeSnapshot,
payload: dict,
decoded: object,
requested_at: float,
received_at: float,
latency: float,
) -> JevClassification:
'''
Reject missing questions, bad labels, or bad distributions.
'''
if not isinstance(decoded, dict):
raise JevProtocolError('response must be an object')
model: object = decoded.get('model')
if not isinstance(model, str) or not model:
raise JevProtocolError('response model is missing')
if (
self.model not in {'jev-latest', 'jev-preview'}
and model != self.model
):
raise JevProtocolError('response violates pinned model')
raw_answers: object = decoded.get('answers')
questions: dict = payload['questions']
if (
not isinstance(raw_answers, dict)
or set(raw_answers) != set(questions)
):
raise JevProtocolError('response question set differs')
answers: list[RegimeAnswer] = []
name: str
question: dict
for name, question in questions.items():
raw: object = raw_answers[name]
if (
not isinstance(raw, dict)
or raw.get('type') != 'choice'
):
raise JevProtocolError('expected a Choice answer')
label: object = raw.get('choice')
probabilities: object = raw.get('probabilities')
if (
not isinstance(probabilities, dict)
or set(probabilities) != set(question['criteria'])
or not isinstance(label, str)
or label not in probabilities
):
raise JevProtocolError('answer options differ')
distribution: dict[str, float] = {}
option: str
probability: object
for option, probability in probabilities.items():
distribution[option] = _probability(probability)
if not isclose(
sum(distribution.values()), 1, abs_tol=1e-5,
):
raise JevProtocolError('probabilities must sum to 1')
if distribution[label] < max(distribution.values()):
raise JevProtocolError('choice is not most probable')
period: float|None = None
if name != 'alignment':
position: int = int(name.removeprefix('timeframe_'))
period = snapshot.timeframes[position].period
answers.append(RegimeAnswer(
question=name,
period=period,
label=label,
probability=distribution[label],
confidence=_probability(raw.get('confidence')),
probabilities=tuple(distribution.items()),
))
usage: object = decoded.get('usage')
if not isinstance(usage, dict):
raise JevProtocolError('token usage is missing')
key: str
for key in ('input_tokens', 'output_tokens'):
count: object = usage.get(key)
if isinstance(count, bool) or not isinstance(count, int):
raise JevProtocolError('token usage must be integer')
if count < 0:
raise JevProtocolError('token usage must be >= 0')
return JevClassification(
market=snapshot.market,
revision=snapshot.revision,
sequence=snapshot.sequence,
segment=snapshot.segment,
source_as_of=snapshot.as_of,
requested_at=requested_at,
received_at=received_at,
latency=latency,
model=model,
prompt_version=PROMPT_VERSION,
answers=tuple(answers),
input_tokens=usage['input_tokens'],
output_tokens=usage['output_tokens'],
)
@dataclass(frozen=True)
class _Submission:
'''
Retain one snapshot and its monotonic local submission time.
'''
snapshot: RegimeSnapshot
submitted: float
class RegimeClassifier:
'''
Coalesce producer updates into one paced, single-flight worker.
``RegimeClassifier.submit()`` replaces
``RegimeClassifier._pending`` without awaiting HTTP or consumers.
Responses must share the current revision/segment and remain
within ``RegimeClassifier.max_source_lag`` event-time seconds and
``RegimeClassifier.max_age`` local seconds of their evidence.
A small nonzero lag permits useful inference on fast feeds;
zero requires the response's exact latest event-time snapshot.
Run one worker in the owning actor's nursery. Cancellation closes
in-flight I/O through HTTPX; the caller owns the HTTP session.
'''
def __init__(
self,
client: JevClient,
*,
min_interval: float = 1,
max_age: float = 5,
max_source_lag: float = 2,
max_retries: int = 2,
) -> None:
'''
Configure bounded request pacing and evidence freshness.
'''
value: float
for value in (min_interval, max_source_lag):
if not isfinite(value) or value < 0:
raise ValueError('interval and lag must be >= 0')
if not isfinite(max_age) or max_age <= 0:
raise ValueError('max_age must be positive and finite')
if (
isinstance(max_retries, bool)
or not 0 <= integer_index(max_retries) <= 8
):
raise ValueError('max_retries must be in [0, 8]')
self.client: JevClient = client
self.min_interval: float = min_interval
self.max_age: float = max_age
self.max_source_lag: float = max_source_lag
self.max_retries: int = max_retries
self.requests: int = 0
self.published: int = 0
self.discarded: int = 0
self.failures: int = 0
self.last_error: Exception|None = None
self._latest: _Submission|None = None
self._pending: _Submission|None = None
self._wake: trio.Event = trio.Event()
self._running: bool = False
self._minimum_revision: int = 0
self._publishing: _Submission|None = None
self._publish_scope: trio.CancelScope|None = None
def invalidate(self, revision: int) -> None:
'''
Revoke older evidence immediately, before source replay.
A feed owner calls this upon receiving a history revision;
it need not wait for a replacement snapshot to become ready.
'''
if (
isinstance(revision, bool)
or integer_index(revision) < 0
):
raise ValueError('revision must be an integer >= 0')
self._minimum_revision = max(
self._minimum_revision, revision,
)
if (
self._pending is not None
and self._pending.snapshot.revision < revision
):
self._pending = None
self._cancel_stale_publication()
def _cancel_stale_publication(self) -> None:
'''
Revoke stale ``RegimeClassifier._publish_scope`` delivery.
``RegimeClassifier._publishing`` retains the exact submission
whose consumer may currently be blocked at an ``await``.
'''
if (
self._publishing is not None
and self._publish_scope is not None
and not self._fresh(self._publishing)
):
self._publish_scope.cancel()
def submit(self, snapshot: RegimeSnapshot) -> bool:
'''
Replace ``RegimeClassifier._pending`` with newer evidence.
Refuse older source coordinates or a different market before
replacing ``RegimeClassifier._latest`` and its pending work.
'''
if snapshot.revision < self._minimum_revision:
return False
latest: _Submission|None = self._latest
if latest is not None:
previous: RegimeSnapshot = latest.snapshot
if snapshot.market != previous.market:
raise ValueError('one classifier owns one market')
if (snapshot.revision, snapshot.sequence) <= (
previous.revision, previous.sequence,
):
return False
if (
snapshot.revision == previous.revision
and snapshot.as_of <= previous.as_of
):
raise ValueError('as_of must advance in a revision')
submission: _Submission = _Submission(
snapshot,
trio.current_time(),
)
self._latest = self._pending = submission
self._cancel_stale_publication()
self._wake.set()
return True
def _fresh(self, submission: _Submission) -> bool:
'''
Check revision, gap segment, source lag, and elapsed age.
'''
latest: _Submission|None = self._latest
assert latest is not None
source: RegimeSnapshot = submission.snapshot
newest: RegimeSnapshot = latest.snapshot
lag: float = newest.as_of - source.as_of
age: float = trio.current_time() - submission.submitted
return (
source.revision >= self._minimum_revision
and source.revision == newest.revision
and source.segment == newest.segment
and 0 <= lag <= self.max_source_lag
and age <= self.max_age
)
async def run(
self,
publish: Callable[[JevClassification], Awaitable[None]],
*,
on_error: Callable[[Exception], Awaitable[None]]|None = None,
task_status: trio.TaskStatus[None] = (
trio.TASK_STATUS_IGNORED
),
) -> None:
'''
Serve until cancelled, with bounded transient retries.
Authentication and malformed-response failures propagate;
transient errors call ``on_error`` and retry newest state.
Retry-After delays have a 60-second cap. Each retry rechecks
freshness; an expired snapshot is never resent.
``publish`` must cooperate with Trio cancellation. Its wait
is cancelled if evidence expires or is invalidated. Consumers
must also validate received message provenance: a delivery
already completed before invalidation cannot be revoked.
'''
if self._running:
raise RuntimeError('classifier worker already running')
self._running = True
task_status.started()
next_request: float = 0
retries: int = 0
try:
while True:
if self._pending is None:
await self._wake.wait()
self._wake = trio.Event()
await trio.sleep_until(next_request)
submission: _Submission|None = self._pending
self._pending = None
if submission is None:
continue
if not self._fresh(submission):
self.discarded += 1
continue
ready: bool = False
frame: TimeframeRegime
for frame in submission.snapshot.timeframes:
ready |= bool(frame.ready and frame.ready[0])
if not ready:
continue
next_request = (
trio.current_time() + self.min_interval
)
self.requests += 1
error: (
httpx.TransportError
|trio.TooSlowError
|JevServiceError
)
try:
result: JevClassification = await (
self.client.classify(submission.snapshot)
)
except (
httpx.TransportError,
trio.TooSlowError,
JevServiceError,
) as error:
self.failures += 1
self.last_error = error
if (
isinstance(error, JevServiceError)
and not error.retryable
):
raise
if on_error is not None:
await on_error(error)
if retries < self.max_retries:
delay: float = min(60, 2 ** retries)
if isinstance(error, JevServiceError):
delay = max(delay, error.retry_after)
next_request = max(
next_request,
trio.current_time() + delay,
)
if self._pending is None:
self._pending = self._latest
retries += 1
else:
retries = 0
continue
retries = 0
self.last_error = None
if self._fresh(submission):
deadline: float = (
submission.submitted + self.max_age
)
scope: trio.CancelScope = trio.CancelScope(
deadline=deadline,
)
with scope:
self._publish_scope = scope
self._publishing = submission
try:
await publish(result)
finally:
self._publishing = None
self._publish_scope = None
if scope.cancelled_caught:
self.discarded += 1
else:
self.published += 1
else:
self.discarded += 1
finally:
self._running = False

422
piker/fsp/_regime.py 100644
View File

@ -0,0 +1,422 @@
# piker: trading gear for hackers
# Copyright (C) Tyler Goodlet (in stewardship of piker0)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Closed-bar, multiple-timeframe trend evidence for regime consumers.
``MultiTimeframeTrend.push()`` receives a FINAL base-bar close and
the bar's START timestamp. Each scalar close is observed at the END
of its interval. Epoch-aligned buckets require every constituent
base bar; neither partial startup buckets nor missing bars become
synthetic observations. A source gap resets all recurrence state.
Wall-clock periods and Trend Calculus observation windows are
independent parameters. This module delegates every trend decision
to ``TrendCalculator``; semantic model labels belong in ``_jev``.
'''
from collections import deque
from dataclasses import asdict, dataclass
from decimal import Decimal
from math import isclose, isfinite, ulp
from operator import index as integer_index
import numpy as np
from ._trend import TrendCalculator, TrendUpdate
def _period_us(period: float) -> int:
'''
Normalize periods to positive integral microseconds.
'''
if isinstance(period, bool) or not isfinite(period):
raise ValueError('period must be finite and positive')
micros: Decimal = Decimal(str(period)) * 1_000_000
nearest: Decimal = micros.to_integral_value()
tolerance: Decimal = Decimal(str(2 * ulp(period))) * 1_000_000
if nearest <= 0 or abs(micros - nearest) > tolerance:
raise ValueError(
'period requires positive whole microseconds',
)
return int(nearest)
@dataclass(frozen=True)
class CloseSample:
'''
A completed bucket close at its observation time.
``CloseSample.index`` is the final contributing base coordinate.
'''
time: float
index: int
price: float
@dataclass(frozen=True)
class TrendReversal:
'''
Preserve both a pivot and when its reversal became observable.
'''
index: int
time: float
price: float
direction: int
order: int
detected_index: int
detected_time: float
@dataclass(frozen=True)
class TimeframeRegime:
'''
Immutable deterministic evidence for one wall-clock timeframe.
``TimeframeRegime.ready`` concerns complete TC windows, never
model confidence. ``TimeframeRegime.as_of`` is the last complete
bucket's end, or ``None``. ``TimeframeRegime.recent_closes`` and
``TimeframeRegime.reversals`` retain bounded causal evidence.
'''
period: float
as_of: float|None
close_count: int
trends: tuple[int, ...]
ready: tuple[bool, ...]
recent_closes: tuple[CloseSample, ...]
reversals: tuple[TrendReversal, ...]
discarded_buckets: int
@dataclass(frozen=True)
class RegimeSnapshot:
'''
Bind all timeframe evidence to one source revision and clock.
``RegimeSnapshot.sequence`` increases on every accepted base bar
within a revision. ``RegimeSnapshot.segment`` increases at source
gaps. ``RegimeSnapshot.as_of`` records the latest base-bar END,
independently of the snapshot's wall-clock arrival time.
'''
market: str
revision: int
sequence: int
segment: int
as_of: float
base_period: float
windows: tuple[int, ...]
timeframes: tuple[TimeframeRegime, ...]
def to_state(self) -> dict:
'''
Build JSON-safe state with explicit timestamp conventions.
'''
state: dict = asdict(self)
state['algorithm'] = 'trend_calculus'
state['price_source'] = 'closed_bar_close'
state['timestamp_convention'] = 'interval_end'
state['trend_codes'] = {
'-1': 'down',
'0': 'unresolved or not ready; consult ready',
'1': 'up',
}
return state
class _Timeframe:
'''
Own one bounded recurrence and its current aggregation bucket.
'''
def __init__(
self,
period_us: int,
factor: int,
windows: tuple[int, ...],
history_limit: int,
) -> None:
'''
Allocate one recurrence and bounded close/reversal tails.
'''
self.period_us: int = period_us
self.factor: int = factor
self.calculator: TrendCalculator = TrendCalculator(windows)
self.closes: deque[CloseSample] = deque(maxlen=history_limit)
self.reversals: deque[TrendReversal] = deque(
maxlen=history_limit,
)
self.update: TrendUpdate|None = None
self.close_count: int = 0
self.bucket: int|None = None
self.count: int = 0
self.discarded: int = 0
def reset(self) -> None:
'''
Reset ``_Timeframe.calculator`` and its bucket provenance.
Clear ``_Timeframe.closes`` and ``_Timeframe.reversals`` on a
source gap so no previous segment survives as evidence.
'''
if self.count:
self.discarded += 1
self.calculator.reset()
self.closes.clear()
self.reversals.clear()
self.update = None
self.close_count = 0
self.bucket = None
self.count = 0
def push(
self,
tick: int,
close: float,
source_index: int,
) -> None:
'''
Advance only after an entire aligned bucket was observed.
'''
bucket: int = tick // self.factor
if bucket != self.bucket:
self.bucket = bucket
self.count = 0
self.count += 1
if (tick + 1) % self.factor:
return
if self.count != self.factor:
self.discarded += 1
self.count = 0
return
end: float = (bucket + 1) * self.period_us / 1_000_000
update: TrendUpdate = self.calculator.push(
end,
close,
index=source_index,
)
self.update = update
self.close_count += 1
self.count = 0
self.closes.append(CloseSample(end, source_index, close))
event: np.void
for event in update.reversals:
self.reversals.append(TrendReversal(
index=int(event['index']),
time=float(event['time']),
price=float(event['price']),
direction=int(event['direction']),
order=int(event['order']),
detected_index=int(event['detected_index']),
detected_time=float(event['detected_time']),
))
def snapshot(self) -> TimeframeRegime:
'''
Detach immutable evidence from this writer's mutable state.
'''
update: TrendUpdate|None = self.update
size: int = len(self.calculator.windows)
return TimeframeRegime(
period=self.period_us / 1_000_000,
as_of=update.time if update else None,
close_count=self.close_count,
trends=update.trends if update else (0,) * size,
ready=update.ready if update else (False,) * size,
recent_closes=tuple(self.closes),
reversals=tuple(self.reversals),
discarded_buckets=self.discarded,
)
class MultiTimeframeTrend:
'''
Aggregate final base bars into causal, independent TC streams.
Call ``reset()`` and replay CLOSED source history after backfill,
corrections, reconnect/session changes, or any source revision.
Never replay the current mutable OHLC row as a final observation.
A missing base interval conservatively resets every timeframe;
a duplicate or out-of-order interval raises without mutation.
Work per update and retained memory are bounded by configured
timeframe/order counts and ``history_limit``, not stream age.
'''
def __init__(
self,
market: str,
*,
base_period: float = 1,
periods: tuple[float, ...] = (1, 5, 15),
windows: tuple[int, ...] = (2, 2, 2),
history_limit: int = 16,
revision: int = 0,
) -> None:
'''
Validate timeframe periods before allocating recurrences.
'''
if not isinstance(market, str) or not market.strip():
raise ValueError('market identity is required')
if not 1 <= len(periods) <= 16:
raise ValueError('configure between 1 and 16 timeframes')
if (
isinstance(history_limit, bool)
or not 1 <= integer_index(history_limit) <= 256
):
raise ValueError('history_limit must be in [1, 256]')
if isinstance(revision, bool) or integer_index(revision) < 0:
raise ValueError('revision must be an integer >= 0')
base_us: int = _period_us(base_period)
periods_us: list[int] = []
period: float
for period in periods:
periods_us.append(_period_us(period))
if len(set(periods_us)) != len(periods_us):
raise ValueError('periods must be unique base multiples')
micros: int
for micros in periods_us:
if micros % base_us:
raise ValueError('periods need whole base multiples')
self.market: str = market
self.base_period: float = base_us / 1_000_000
self.windows: tuple[int, ...] = tuple(windows)
self.revision: int = int(revision)
self.sequence: int = 0
self.segment: int = 0
self._base_us: int = base_us
self._last_tick: int|None = None
self._last_index: int = -1
self._frames: tuple[_Timeframe, ...] = tuple(
_Timeframe(
micros,
micros // base_us,
self.windows,
history_limit,
)
for micros in sorted(periods_us)
)
# Reuse TrendCalculator.windows, already normalized to int,
# so accepted NumPy integer inputs remain JSON-serializable.
self.windows = self._frames[0].calculator.windows
def reset(self, revision: int|None = None) -> None:
'''
Invalidate prior evidence before replaying revised history.
'''
if revision is None:
revision = self.revision + 1
if (
isinstance(revision, bool)
or integer_index(revision) <= self.revision
):
raise ValueError('revision must strictly increase')
frame: _Timeframe
for frame in self._frames:
frame.reset()
frame.discarded = 0
self.revision = int(revision)
self.sequence = 0
self.segment = 0
self._last_tick = None
self._last_index = -1
def push(
self,
time: float,
close: float,
*,
index: int|None = None,
) -> RegimeSnapshot:
'''
Commit a final close with its base-bar START timestamp.
Nonfinite data and unaligned timestamps are rejected before
mutation. Following omitted/invalid bars, the next timestamp
exposes the gap and resets all trend evidence explicitly.
'''
if not isfinite(time) or not isfinite(close):
raise ValueError('bar time and close must be finite')
ticks: float = time / self.base_period
if not isfinite(ticks):
raise ValueError('bar time exceeds grid precision')
tick: int = round(ticks)
aligned: float = tick * self._base_us / 1_000_000
end: float = (tick + 1) * self._base_us / 1_000_000
if not isfinite(end) or end <= aligned:
raise ValueError('bar interval exceeds clock precision')
if not isclose(
time,
aligned,
rel_tol=0,
abs_tol=max(1e-9, 2 * ulp(time)),
):
raise ValueError('bar starts must align to base period')
if self._last_tick is not None and tick <= self._last_tick:
raise ValueError('bar starts must strictly increase')
if index is None:
index = self._last_index + 1
if isinstance(index, (bool, np.bool_)):
raise ValueError('source index needs increasing int64')
coordinate: int = integer_index(index)
if (
not 0 <= coordinate < 2 ** 63
or coordinate <= self._last_index
):
raise ValueError('source index needs increasing int64')
frame: _Timeframe
if (
self._last_tick is not None
and tick != self._last_tick + 1
):
self.segment += 1
for frame in self._frames:
frame.reset()
for frame in self._frames:
frame.push(tick, float(close), coordinate)
self._last_tick = tick
self._last_index = coordinate
self.sequence += 1
return RegimeSnapshot(
market=self.market,
revision=self.revision,
sequence=self.sequence,
segment=self.segment,
as_of=end,
base_period=self.base_period,
windows=self.windows,
timeframes=tuple(
frame.snapshot() for frame in self._frames
),
)

550
piker/fsp/_regime_feed.py 100644
View File

@ -0,0 +1,550 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Bridge finalized native OHLC SHM rows to regime snapshot consumers.
Sampler messages are wakeups and invalidations, not bar values.
This reader copies the authoritative source range and excludes its
mutable final row. Existing SHM has no transactional revision word:
bound checks detect prepend/truncation; in-place corrections rely
on the source's backfill notice and converge by invalidating/replay.
'''
from collections.abc import AsyncIterator, Awaitable, Callable
from dataclasses import dataclass
from operator import index as integer_index
from math import isfinite
import time
from typing import TYPE_CHECKING
import numpy as np
import trio
from tractor.trionics import Lagged
from ._regime import MultiTimeframeTrend, RegimeSnapshot
if TYPE_CHECKING:
from tractor.ipc._shm import ShmArray
from ._jev import JevClassification, JevClient, RegimeClassifier
class SourceChanged(Exception):
'''
Signal changed shared bounds while copying a finalized prefix.
'''
@dataclass(frozen=True)
class ClosedSourceRows:
'''
Keep one detached finalized range and its absolute SHM bounds.
``ClosedSourceRows.last`` includes the mutable source row;
``ClosedSourceRows.rows`` excludes it. ``ClosedSourceRows.start``
is a physical absolute offset in ``ShmArray._array``.
'''
first: int
last: int
start: int
rows: np.ndarray
def _bounds(shm: 'ShmArray') -> tuple[int, int]:
'''
Read absolute shared bounds, never the wrapped ``shm.index``.
'''
return int(shm._first.value), int(shm._last.value)
def copy_closed_rows(
shm: 'ShmArray',
*,
start: int|None = None,
) -> ClosedSourceRows:
'''
Copy a stable finalized range before its consumer can yield.
Invalid numeric data is rejected before recurrence mutation.
``ShmArray._array['index']`` gives absolute source coordinates.
Increasing timestamps may contain gaps: the tracker resets its
segment on those gaps. The source's mutable last row is never
validated or copied, so unfinished OHLC cannot become evidence.
Appends preserve this closed prefix. ``ClosedSourceRows.last``
retains the captured bound so ``consume_regime_source()`` can
process an appended suffix on its next pass.
'''
first: int
last: int
first, last = _bounds(shm)
raw: np.ndarray = shm._array
if not 0 <= first <= last <= len(raw):
if _bounds(shm) != (first, last):
raise SourceChanged('source bounds changed during read')
raise ValueError('source bounds are not a readable range')
stop: int = max(first, last - 1)
if start is None:
start = first
if not first <= start <= stop:
raise SourceChanged('finalized source range moved')
if not {'index', 'time', 'close'} <= set(raw.dtype.names or ()):
raise ValueError('source requires index/time/close fields')
rows: np.ndarray = raw[start:stop][[
'index', 'time', 'close',
]].copy()
current_first: int
current_last: int
current_first, current_last = _bounds(shm)
if current_first != first or current_last < last:
raise SourceChanged('source bounds changed during copy')
indices: np.ndarray = rows['index']
times: np.ndarray = rows['time']
if not np.issubdtype(indices.dtype, np.integer):
raise ValueError('source indices must be integers')
if (
np.any(indices < 0)
or np.any(indices > np.iinfo(np.int64).max)
or np.any(indices[1:] <= indices[:-1])
or not np.isfinite(times).all()
or np.any(times[1:] <= times[:-1])
or not np.isfinite(rows['close']).all()
):
raise ValueError('source has invalid finalized observations')
rows.flags.writeable = False
return ClosedSourceRows(first, last, start, rows)
def _sample_action(
message: dict,
market: str,
base_period: float,
) -> str:
'''
Filter global sampler broadcasts by the actual source identity.
'''
if not isinstance(message, dict):
return 'ignore'
if 'backfilling' in message:
notice: object = message['backfilling']
if (
isinstance(notice, (tuple, list))
and len(notice) == 2
and notice[0] == market
and notice[1] == base_period
):
return 'revision'
return 'ignore'
if message.get('period') == base_period:
return 'step'
return 'ignore'
def source_snapshot_is_current(
snapshot: RegimeSnapshot,
*,
now: float,
max_source_age: float|None = 5,
) -> bool:
'''
Gate model admission by wall-clock source age, independent of UI.
Allow at most one base period of forward clock skew. ``None``
explicitly disables the clock gate for historical replay. Display
consumers can retain history while sharing this gate before Jev.
'''
if max_source_age is None:
return True
if not isfinite(max_source_age) or max_source_age <= 0:
raise ValueError('max_source_age must be positive or None')
if not isfinite(now):
return False
age: float = now - snapshot.as_of
return -snapshot.base_period <= age <= max_source_age
class _Notices:
'''
Coalesce sampling wakeups while retaining revision invalidation.
``_Notices.collect()`` mutates only its notification flags and
``_Notices.revision``, never the TC recurrence.
``_Notices.revise()`` calls ``_Notices.invalidate`` before replay
checkpoints allow an obsolete model response to finish.
'''
def __init__(
self,
tracker: MultiTimeframeTrend,
invalidate: Callable[[int], object]|None,
) -> None:
'''
Bind a tracker revision to its synchronous invalidation hook.
'''
self.tracker: MultiTimeframeTrend = tracker
self.revision: int = tracker.revision
self.invalidate: Callable[[int], object]|None = invalidate
self.dirty: bool = True
self.closed: bool = False
self.wake: trio.Event = trio.Event()
def revise(self) -> None:
'''
Allocate the next generation and revoke its predecessor.
'''
self.revision += 1
if self.invalidate is not None:
self.invalidate(self.revision)
self.dirty = True
self.wake.set()
async def collect(
self,
stream: AsyncIterator[dict],
*,
task_status: trio.TaskStatus[None] = (
trio.TASK_STATUS_IGNORED
),
) -> None:
'''
Preserve backfill/overrun notices during slow bootstrap.
'''
iterator: AsyncIterator[dict] = aiter(stream)
task_status.started()
while True:
try:
message: dict = await anext(iterator)
except Lagged:
# A lost notice can describe an in-place correction;
# unchanged SHM bounds cannot prove consistency.
self.revise()
continue
except StopAsyncIteration:
self.closed = True
self.wake.set()
return
action: str = _sample_action(
message,
self.tracker.market,
self.tracker.base_period,
)
if action == 'revision':
self.revise()
elif action == 'step':
self.dirty = True
self.wake.set()
async def consume_regime_source(
shm: 'ShmArray',
sample_stream: AsyncIterator[dict],
tracker: MultiTimeframeTrend,
submit: Callable[[RegimeSnapshot], object],
*,
invalidate: Callable[[int], object]|None = None,
replay_chunk: int = 512,
) -> None:
'''
Bootstrap/replay closed rows, then follow source sample advances.
Enter the source sampler subscription BEFORE calling this helper.
``submit`` is synchronous and receives only the newest complete
snapshot per captured range, not a request for each history row.
Pass ``RegimeClassifier.submit`` and its ``invalidate`` together
when using Jev. Both callbacks must return without awaiting I/O.
One compute task owns ``tracker``. A second task drains sampler
events into bounded flags during cooperative historical replay.
A prepend, truncation, relevant backfill, or sampler overrun
invalidates the previous revision and rebuilds from full history.
Cancellation closes the collector with the consumer's nursery.
'''
if (
isinstance(replay_chunk, bool)
or integer_index(replay_chunk) < 1
):
raise ValueError('replay_chunk must be a positive integer')
if tracker.sequence:
raise ValueError('source consumer needs an unused tracker')
notices: _Notices = _Notices(tracker, invalidate)
prior_bounds: tuple[int, int]|None = None
copy_failures: int = 0
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
await nursery.start(notices.collect, sample_stream)
try:
while True:
if not notices.dirty:
if notices.closed:
return
await notices.wake.wait()
notices.wake = trio.Event()
notices.dirty = False
revision: int = notices.revision
if revision != tracker.revision:
tracker.reset(revision=revision)
prior_bounds = None
first: int
last: int
first, last = _bounds(shm)
if prior_bounds is not None:
old_first: int
old_last: int
old_first, old_last = prior_bounds
if first != old_first or last < old_last:
notices.revise()
continue
start: int = max(first, old_last - 1)
else:
start = first
try:
source: ClosedSourceRows = copy_closed_rows(
shm, start=start,
)
except SourceChanged:
copy_failures += 1
if copy_failures >= 8:
raise SourceChanged(
'source unstable after 8 copy attempts',
) from None
notices.dirty = True
delay: float = min(.1, .001 * 2 ** copy_failures)
await trio.sleep(delay)
continue
copy_failures = 0
snapshot: RegimeSnapshot|None = None
offset: int
row: np.void
for offset, row in enumerate(source.rows, start=1):
snapshot = tracker.push(
float(row['time']),
float(row['close']),
index=int(row['index']),
)
if offset % replay_chunk == 0:
await trio.lowlevel.checkpoint()
if notices.revision != revision:
break
# Permit queued in-place invalidations to reach the
# collector before admitting bootstrap/replay output.
await trio.lowlevel.checkpoint()
if notices.revision != revision:
notices.dirty = True
continue
current_first: int
current_last: int
current_first, current_last = _bounds(shm)
if (
current_first != source.first
or current_last < source.last
):
notices.revise()
continue
prior_bounds = source.first, source.last
if current_last != source.last:
notices.dirty = True
if snapshot is not None:
submit(snapshot)
finally:
nursery.cancel_scope.cancel()
async def run_regime_feed(
fqme: str,
submit: Callable[[RegimeSnapshot], object],
*,
invalidate: Callable[[int], object]|None = None,
periods: tuple[float, ...]|None = None,
windows: tuple[int, ...] = (2, 2, 2),
history_limit: int = 16,
max_source_age: float|None = 5,
loglevel: str|None = None,
) -> None:
'''
Follow one native realtime Flume in an existing Piker runtime.
This explicit entry point can open a broker data feed. It creates
no Jev client and reads no API key. The native sampler currently
stamps periods <=1s with integer seconds, so this wrapper refuses
subsecond source periods. A separate correct subsecond producer
can call ``consume_regime_source`` directly.
Quote delivery retains the shared feed's default policy. Forcing
a low rate here would throttle later users of its market cache;
this consumer drains quotes and derives only from finalized SHM.
``max_source_age`` bounds wall-clock age of submitted evidence,
separately from the classifier's local queue age. ``None`` allows
historical replay; the default holds stale startup history until
live bars arrive. Source clocks over one base period ahead are
rejected by the same gate when an age limit is enabled.
'''
from ..data._sampling import open_sample_stream
from ..data.feed import Feed, maybe_open_feed
from ..data.flows import Flume
if (
max_source_age is not None
and
(
not isfinite(max_source_age)
or max_source_age <= 0
)
):
raise ValueError('max_source_age must be positive or None')
feed: Feed
async with maybe_open_feed(
fqmes=[fqme],
loglevel=loglevel,
) as feed:
flume: Flume = feed.flumes[fqme]
# maybe_open_feed() replaces the shared Flume.stream on hits.
# Capture our receiver before the sampler context can yield.
quote_stream: AsyncIterator[dict] = flume.stream
period: float = flume.rt_sample_period_s
if period < 1:
raise ValueError('native sampler requires >=1s periods')
if periods is None:
periods = (period, period * 5, period * 15)
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
fqme,
base_period=period,
periods=periods,
windows=windows,
history_limit=history_limit,
)
def submit_fresh(snapshot: RegimeSnapshot) -> None:
'''
Hold stale startup data before admitting live inference.
'''
if source_snapshot_is_current(
snapshot,
now=time.time(),
max_source_age=max_source_age,
):
submit(snapshot)
async def drain_quotes() -> None:
'''
Keep the quote subscription healthy while consuming SHM.
'''
iterator: AsyncIterator[dict] = aiter(quote_stream)
while True:
try:
await anext(iterator)
except Lagged:
# Quotes are discarded; all finalized prices
# still come from source SHM and sampler events.
await trio.lowlevel.checkpoint()
except StopAsyncIteration:
return
stream: AsyncIterator[dict]
nursery: trio.Nursery
async with (
open_sample_stream(
period_s=period,
loglevel=loglevel,
) as stream,
trio.open_nursery() as nursery,
):
nursery.start_soon(drain_quotes)
try:
await consume_regime_source(
flume.rt_shm,
stream,
tracker,
submit_fresh,
invalidate=invalidate,
)
finally:
nursery.cancel_scope.cancel()
async def serve_regimes(
fqme: str,
client: 'JevClient',
publish: Callable[['JevClassification'], Awaitable[None]],
*,
periods: tuple[float, ...]|None = None,
windows: tuple[int, ...] = (2, 2, 2),
history_limit: int = 16,
min_interval: float = 1,
max_age: float = 5,
max_source_lag: float = 2,
max_source_age: float|None = 5,
on_error: Callable[[Exception], Awaitable[None]]|None = None,
loglevel: str|None = None,
) -> 'RegimeClassifier':
'''
Compose a native feed and paced Jev worker in one owned nursery.
This explicit call opens a broker feed and uses the supplied Jev
client for inference. The caller owns its HTTP session and Piker
runtime. Cancellation closes the sampler subscription, quote
drain and in-flight classifier work together. Consumers validate
source provenance again when applying delivered messages.
'''
from ._jev import RegimeClassifier
worker: RegimeClassifier = RegimeClassifier(
client,
min_interval=min_interval,
max_age=max_age,
max_source_lag=max_source_lag,
)
async def classify() -> None:
'''
Bind the optional error callback to this service lifetime.
'''
await worker.run(publish, on_error=on_error)
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(classify)
try:
await run_regime_feed(
fqme,
worker.submit,
invalidate=worker.invalidate,
periods=periods,
windows=windows,
history_limit=history_limit,
max_source_age=max_source_age,
loglevel=loglevel,
)
finally:
nursery.cancel_scope.cancel()
return worker

591
piker/fsp/_trend.py 100644
View File

@ -0,0 +1,591 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
# Algorithm: Andrew J. Morgan / ByteSumo, TrendCalculus (GPLv3+).
# See docs/trend_calculus.rst for sources and semantic differences.
'''
Causal, bounded-state Trend Calculus for scalar observations.
The same compiled recurrence drives historical and live processing.
FHLS means first/high/low/second *extremum*, not open/high/low/close.
'''
from dataclasses import dataclass
from math import isfinite
from operator import index as integer_index
import numpy as np
from numba import njit
_bar_dtype: np.dtype[np.void] = np.dtype([
('high', '<f8'),
('low', '<f8'),
('hi_time', '<f8'),
('lo_time', '<f8'),
('hi_index', '<i8'),
('lo_index', '<i8'),
])
reversal_dtype: np.dtype[np.void] = np.dtype([
('index', '<i8'),
('time', '<f8'),
('price', '<f8'),
('direction', 'i1'),
('order', '<i8'),
('detected_index', '<i8'),
('detected_time', '<f8'),
])
@njit(cache=True)
def _direction(
high: float,
low: float,
prev_high: float,
prev_low: float,
) -> int:
'''
Evaluate Morgan's sign sum without overflowing a subtraction.
'''
delta: int = (
int(high > prev_high) - int(high < prev_high)
+ int(low > prev_low) - int(low < prev_low)
)
return int(delta > 0) - int(delta < 0)
@njit(cache=True)
def _reversal(
previous: np.void,
old_trend: int,
new_trend: int,
level: int,
index: int,
time: float,
events: np.ndarray,
counts: np.ndarray,
) -> None:
'''
Emit a confirmed sign change at its detection observation.
'''
if old_trend * new_trend != -1:
return
slot: int = counts[level]
event: np.void = events[level, slot]
if new_trend == 1:
event['index'] = previous['lo_index']
event['time'] = previous['lo_time']
event['price'] = previous['low']
else:
event['index'] = previous['hi_index']
event['time'] = previous['hi_time']
event['price'] = previous['high']
event['direction'] = new_trend
event['order'] = level + 1
event['detected_index'] = index
event['detected_time'] = time
counts[level] += 1
@njit(cache=True)
def _step(
bars: np.ndarray,
counters: np.ndarray,
sizes: np.ndarray,
index: int,
time: float,
value: float,
events: np.ndarray,
counts: np.ndarray,
lua_seed: bool,
) -> None:
'''
Advance each reversal order using at most two incoming points.
The ``TrendCalculator._bars`` argument uses ``bars[:, 0]`` for
the current window and ``bars[:, 1]`` for its completed
predecessor. ``TrendCalculator._counters`` stores pending sample
count, completed window count, and confirmed direction.
Higher orders consume only the preceding order's reversals.
'''
counts[:] = 0
if np.isnan(value):
bars[:] = np.zeros_like(bars)
counters[:] = 0
return
bridge: np.ndarray = np.empty(1, dtype=_bar_dtype)
level: int
for level in range(len(sizes)):
incoming: int = 1 if level == 0 else counts[level - 1]
if incoming == 0:
break
point: int
for point in range(incoming):
price: float
point_time: float
point_index: int
if level == 0:
price = value
point_time = time
point_index = index
else:
event: np.void = events[level - 1, point]
price = event['price']
point_time = event['time']
point_index = event['index']
current: np.void = bars[level, 0]
previous: np.void = bars[level, 1]
if counters[level, 0] == 0:
current['high'] = current['low'] = price
current['hi_time'] = current['lo_time'] = point_time
current['hi_index'] = point_index
current['lo_index'] = point_index
elif price > current['high']:
current['high'] = price
current['hi_time'] = point_time
current['hi_index'] = point_index
elif price < current['low']:
current['low'] = price
current['lo_time'] = point_time
current['lo_index'] = point_index
counters[level, 0] += 1
if counters[level, 0] < sizes[level]:
continue
counters[level, 0] = 0
counters[level, 1] += 1
if counters[level, 1] == 1:
if lua_seed:
# Lua compares its first window with its final
# point. Native startup leaves direction unknown.
previous['high'] = previous['low'] = price
previous['hi_time'] = point_time
previous['lo_time'] = point_time
previous['hi_index'] = point_index
previous['lo_index'] = point_index
else:
bars[level, 1] = current
continue
trend: int = _direction(
current['high'], current['low'],
previous['high'], previous['low'],
)
old_trend: int = counters[level, 2]
if trend == 0:
# Bridge the later previous extremum to the earlier
# current extremum. Preserve both possible reversals.
first_price: float
first_time: float
first_index: int
if previous['hi_index'] > previous['lo_index']:
first_price = previous['high']
first_time = previous['hi_time']
first_index = previous['hi_index']
else:
first_price = previous['low']
first_time = previous['lo_time']
first_index = previous['lo_index']
second_price: float
second_time: float
second_index: int
if current['hi_index'] < current['lo_index']:
second_price = current['high']
second_time = current['hi_time']
second_index = current['hi_index']
else:
second_price = current['low']
second_time = current['lo_time']
second_index = current['lo_index']
split: np.void = bridge[0]
if first_price > second_price:
split['high'] = first_price
split['hi_time'] = first_time
split['hi_index'] = first_index
split['low'] = second_price
split['lo_time'] = second_time
split['lo_index'] = second_index
else:
split['high'] = second_price
split['hi_time'] = second_time
split['hi_index'] = second_index
split['low'] = first_price
split['lo_time'] = first_time
split['lo_index'] = first_index
split_trend: int = _direction(
split['high'], split['low'],
previous['high'], previous['low'],
) or old_trend
_reversal(
previous, old_trend, split_trend,
level, index, time, events, counts,
)
trend = _direction(
current['high'], current['low'],
split['high'], split['low'],
) or split_trend
_reversal(
split, split_trend, trend,
level, index, time, events, counts,
)
else:
_reversal(
previous, old_trend, trend,
level, index, time, events, counts,
)
counters[level, 2] = trend
bars[level, 1] = current
@njit(cache=True)
def _scan(
values: np.ndarray,
times: np.ndarray,
indices: np.ndarray,
bars: np.ndarray,
counters: np.ndarray,
sizes: np.ndarray,
lua_seed: bool,
) -> tuple[
np.ndarray,
np.ndarray,
np.ndarray,
np.ndarray,
np.ndarray,
]:
'''
Bootstrap with ``_step()``, retaining causal detection pulses.
'''
length: int = len(values)
levels: int = len(sizes)
trends: np.ndarray = np.zeros((length, levels), dtype=np.int8)
ready: np.ndarray = np.zeros((length, levels), dtype=np.bool_)
pulses: np.ndarray = np.zeros((length, levels), dtype=np.int8)
pivot_times: np.ndarray = np.full((length, levels, 2), np.nan)
pivot_prices: np.ndarray = np.full((length, levels, 2), np.nan)
events: np.ndarray = np.empty((levels, 2), dtype=reversal_dtype)
counts: np.ndarray = np.zeros(levels, dtype=np.int64)
row: int
for row in range(length):
_step(
bars, counters, sizes,
indices[row], times[row], values[row],
events, counts, lua_seed,
)
level: int
for level in range(levels):
ready[row, level] = counters[level, 1] >= 2
if ready[row, level]:
trends[row, level] = counters[level, 2]
if counts[level]:
pulses[row, level] = events[
level, counts[level] - 1
]['direction']
slot: int
for slot in range(counts[level]):
event: np.void = events[level, slot]
side: int = int(event['direction'] > 0)
pivot_times[row, level, side] = event['time']
pivot_prices[row, level, side] = event['price']
return trends, ready, pulses, pivot_times, pivot_prices
@dataclass(frozen=True)
class TrendUpdate:
'''
A causal observation: directions and newly confirmed reversals.
``TrendUpdate.ready`` means two complete windows, not statistical
confidence. A ready zero direction means unresolved flat input.
``TrendUpdate.reversals`` has separate pivot and detection
coordinates and can contain two opposing events at the same
order and detection time.
'''
index: int
time: float
trends: tuple[int, ...]
ready: tuple[bool, ...]
reversals: np.ndarray
@dataclass(frozen=True)
class TrendHistory:
'''
Prefix-causal history, indexed by observation and reversal order.
``TrendHistory.pulses`` records the last reversal direction
detected per row and order. Use ``TrendCalculator.push()`` to
retain every event; two reversals at one detection row cannot
fit in one pulse. ``TrendHistory.pivot_times`` and
``TrendHistory.pivot_prices`` retain both sides with shape
(observations, orders, 2), where side 0 is top and 1 is bottom.
Absent events are NaN, never backdated into earlier output rows.
'''
trends: np.ndarray
ready: np.ndarray
pulses: np.ndarray
pivot_times: np.ndarray
pivot_prices: np.ndarray
class TrendCalculator:
'''
Track fixed-size FHLS windows and their recursive reversals.
Each window size counts observations at that order; it is not a
duration. Scalar input avoids inventing intrabar high/low order.
Equal extrema retain their earliest observation (Lua semantics).
Native startup requires two windows and emits no artificial
initial pivot. ``lua_seed=True`` uses the Lua first-window seed
for research; it does not reproduce its output suppression bugs.
NaN resets all orders. Invalid or non-increasing coordinates
and infinite values are rejected before state mutation.
State size is O(number of orders), independent of window sizes
and history length. Instances have one writer and are not shared
between markets or timeframes.
'''
def __init__(
self,
windows: tuple[int, ...] = (2, 2, 2, 2),
*,
lua_seed: bool = False,
) -> None:
'''
Allocate bounded state for each validated window size.
'''
if not windows:
raise ValueError('at least one window size is required')
sizes: list[int] = []
size: int
for size in windows:
if isinstance(size, (bool, np.bool_)):
raise ValueError(
'window size must be an integer >= 2',
)
size = integer_index(size)
if not 2 <= size <= np.iinfo(np.int64).max:
raise ValueError(
'window size exceeds positive int64',
)
sizes.append(size)
self.windows: tuple[int, ...] = tuple(sizes)
self._sizes: np.ndarray = np.array(sizes, dtype=np.int64)
self.lua_seed: bool = lua_seed
self._bars: np.ndarray = np.zeros(
(len(windows), 2), dtype=_bar_dtype,
)
self._counters: np.ndarray = np.zeros(
(len(windows), 3), dtype=np.int64,
)
self._events: np.ndarray = np.empty(
(len(windows), 2), dtype=reversal_dtype,
)
self._counts: np.ndarray = np.zeros(
len(windows), dtype=np.int64,
)
self._last_time: float = -np.inf
self._last_index: int = -1
def copy(self) -> 'TrendCalculator':
'''
Copy bounded recurrence state for an uncommitted preview.
Duplicate ``TrendCalculator._bars`` and
``TrendCalculator._counters`` with the last source
coordinates. Emitted events remain owned by each instance.
'''
clone: TrendCalculator = TrendCalculator(
self.windows, lua_seed=self.lua_seed,
)
clone._bars[:] = self._bars
clone._counters[:] = self._counters
clone._last_time = self._last_time
clone._last_index = self._last_index
return clone
def reset(self) -> None:
'''
Start a new session or revision with no inherited trend.
Clear ``TrendCalculator._bars``,
``TrendCalculator._counters`` and
``TrendCalculator._counts``. Release source constraints in
``TrendCalculator._last_time`` and
``TrendCalculator._last_index``.
'''
self._bars[:] = np.zeros_like(self._bars)
self._counters[:] = 0
self._counts[:] = 0
self._last_time = -np.inf
self._last_index = -1
def _validate(
self,
index: int,
time: float,
value: float,
) -> None:
'''
Refuse ambiguous ordering without consuming an observation.
'''
if not isfinite(time) or time <= self._last_time:
raise ValueError('time must be finite and increasing')
if not 0 <= index <= np.iinfo(np.int64).max:
raise ValueError('index must fit nonnegative int64')
if index <= self._last_index:
raise ValueError('index must be increasing')
if np.isinf(value):
raise ValueError('infinite values are not observations')
def push(
self,
time: float,
value: float,
*,
index: int|None = None,
) -> TrendUpdate:
'''
Commit one scalar sample and return its causal output.
'''
if index is None:
index = self._last_index + 1
if isinstance(index, (bool, np.bool_)):
raise ValueError('index must be a nonnegative integer')
index = integer_index(index)
self._validate(index, time, value)
_step(
self._bars, self._counters, self._sizes,
index, time, value, self._events, self._counts,
self.lua_seed,
)
self._last_time = time
self._last_index = index
ready: np.ndarray = self._counters[:, 1] >= 2
trends: np.ndarray = np.where(
ready, self._counters[:, 2], 0,
)
emitted: np.ndarray = (
np.arange(2)[None, :] < self._counts[:, None]
)
events: np.ndarray = self._events[emitted]
return TrendUpdate(
index=index,
time=time,
trends=tuple(map(int, trends)),
ready=tuple(map(bool, ready)),
reversals=events,
)
def extend(
self,
values: np.ndarray,
times: np.ndarray,
indices: np.ndarray|None = None,
) -> TrendHistory:
'''
Consume a batch with the same live recurrence semantics.
All coordinates are validated before the batch mutates state.
Incomplete windows survive across calls and process restarts
should rebuild them by replaying the same source revision.
'''
values = np.asarray(values, dtype=np.float64)
times = np.asarray(times, dtype=np.float64)
if values.ndim != 1 or times.ndim != 1:
raise ValueError('values and times must be 1D')
if indices is None:
limit: int = np.iinfo(np.int64).max
if self._last_index + len(values) > limit:
raise ValueError('automatic indices overflow int64')
indices = np.arange(
self._last_index + 1,
self._last_index + 1 + len(values),
dtype=np.int64,
)
indices = np.asarray(indices)
if (
indices.ndim != 1
or len(values) != len(times)
or len(values) != len(indices)
):
raise ValueError('values, times, indices must align')
if not np.issubdtype(indices.dtype, np.signedinteger):
raise ValueError('indices require signed integer dtype')
if len(values):
self._validate(int(indices[0]), times[0], values[0])
if (
not np.isfinite(times).all()
or np.isinf(values).any()
or np.any(times[1:] <= times[:-1])
or np.any(indices[1:] <= indices[:-1])
):
raise ValueError('invalid value or non-increasing batch')
trends: np.ndarray
ready: np.ndarray
pulses: np.ndarray
pivot_times: np.ndarray
pivot_prices: np.ndarray
(
trends,
ready,
pulses,
pivot_times,
pivot_prices,
) = _scan(
values, times, indices, self._bars, self._counters,
self._sizes, self.lua_seed,
)
if len(values):
self._last_time = float(times[-1])
self._last_index = int(indices[-1])
return TrendHistory(
trends, ready, pulses, pivot_times, pivot_prices,
)

208
piker/fsp/_trend_fsp.py 100644
View File

@ -0,0 +1,208 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Trend Calculus over Piker's mutable close-price SHM source.
'''
from collections.abc import AsyncIterator
import numpy as np
from tractor.ipc._shm import ShmArray
from ..data.ticktools import FeedQuote
from ._api import FspSample, FspStream, fsp
from ._trend import TrendCalculator, TrendHistory, TrendUpdate
_trend_fields: tuple[str, ...] = (
'trend',
'trend_2',
'trend_3',
'trend_4',
)
order: int
field: str
_outputs: tuple[str, ...] = _trend_fields + ('ready',) + tuple(
field
for order in range(1, 5)
for field in (
f'top_{order}',
f'top_time_{order}',
f'bottom_{order}',
f'bottom_time_{order}',
)
)
def _history_fields(
output: TrendHistory,
) -> dict[str, np.ndarray]:
'''
Convert causal arrays to the fixed SHM field schema.
'''
fields: dict[str, np.ndarray] = {}
level: int
field: str
for level, field in enumerate(_trend_fields):
fields[field] = output.trends[:, level].astype(np.float64)
fields['ready'] = output.ready.sum(axis=1).astype(np.float64)
level: int
for level in range(4):
side: int
label: str
for side, label in enumerate(('top', 'bottom')):
order: int = level + 1
fields[f'{label}_{order}'] = (
output.pivot_prices[:, level, side]
)
fields[f'{label}_time_{order}'] = (
output.pivot_times[:, level, side]
)
return fields
@fsp(
outputs=_outputs,
sample_on_step=True,
event_outputs=_outputs[5:],
)
async def trend_calculus(
source: AsyncIterator[FeedQuote],
ohlcv: ShmArray,
window: int = 2,
) -> FspStream:
'''
Publish four causal reversal orders from scalar bar closes.
``window`` counts first-order closes. Each higher order groups
two reversals from the preceding order into an FHLS window.
History uses finalized closes except for its mutable last row.
Live quotes preview that row from a copy of committed state;
repeated updates never consume a bar twice. Source advancement
commits all newly finalized bars, including skipped quote steps.
``ready`` counts orders having two completed FHLS windows.
``top_N``/``bottom_N`` are pivot prices at their detection row;
matching ``*_time_N`` fields retain the actual extremum time.
NaN means no pivot. Both sides survive split-window reversals.
A source prepend or in-place history revision restarts this
generator through ``Cascade.resync()``. Chart timestamps remain
source bar-open times, and the last row is always provisional.
For service consumption use finalized observations and explicit
bar-close/detection times in ``TrendCalculator`` instead.
'''
calc: TrendCalculator = TrendCalculator((window, 2, 2, 2))
# The engine supplies an immutable bootstrap view. Retain its
# absolute coordinates; live shared bounds may already differ.
history: np.ndarray = ohlcv.array.copy()
first: int = int(history['index'][0]) if len(history) else -1
committed: int = first - 1
output: TrendHistory = calc.extend(
history['close'][:-1],
history['time'][:-1],
history['index'][:-1],
)
if len(history) > 1:
committed = int(history['index'][-2])
tail: TrendHistory = calc.copy().extend(
history['close'][-1:],
history['time'][-1:],
history['index'][-1:],
)
fields: dict[str, np.ndarray] = _history_fields(output)
tail_fields: dict[str, np.ndarray] = _history_fields(tail)
field: str
for field in _outputs:
fields[field] = np.concatenate((
fields[field], tail_fields[field],
))
yield fields
quote: FeedQuote
async for quote in source:
# Quotes are wakeups. The source SHM is authoritative, and a
# quote frame may contain multiple already aggregated trades.
rows: np.ndarray = ohlcv.array
if not len(rows):
continue
source_first: int = int(rows['index'][0])
last: int = int(rows['index'][-1])
if first == -1:
first = source_first
committed = first - 1
if source_first != first or last <= committed:
# Normally the engine restarts first. Avoid using a
# recurrence with a different origin during the handoff.
return
begin: int = committed - first + 1
closed: np.ndarray = rows[begin:-1].copy()
current: np.void = rows[-1].copy()
if len(closed):
finalized: TrendHistory = calc.extend(
closed['close'], closed['time'], closed['index'],
)
committed = int(closed['index'][-1])
# Publish every finalized row, not only the new tail.
# Missing quote wakeups must not discard reversal events
# or leave a prior row with its last provisional close.
closed_fields: dict[str, np.ndarray] = (
_history_fields(finalized)
)
offset: int
row: np.void
for offset, row in enumerate(closed):
final_values: dict[str, float] = {}
field: str
for field in _outputs:
final_values[field] = float(
closed_fields[field][offset],
)
yield FspSample(
index=int(row['index']),
time=float(row['time']),
fields=final_values,
)
preview: TrendUpdate = calc.copy().push(
float(current['time']),
float(current['close']),
index=int(current['index']),
)
values: dict[str, int|float] = dict.fromkeys(
_outputs, np.nan,
)
values.update(zip(_trend_fields, preview.trends))
values['ready'] = sum(preview.ready)
event: np.void
label: str
order: int
for event in preview.reversals:
label = 'bottom' if event['direction'] > 0 else 'top'
order = int(event['order'])
values[f'{label}_{order}'] = float(event['price'])
values[f'{label}_time_{order}'] = float(event['time'])
yield FspSample(
index=preview.index,
time=preview.time,
fields=values,
)

53
piker/ui/_display.py
View File

@ -71,6 +71,7 @@ from ._l1 import L1Labels
from ._style import hcolor
from ._fsp import (
update_fsp_chart,
update_trend_overlays,
start_fsp_displays,
open_vlm_displays,
)
@ -285,6 +286,41 @@ async def increment_history_view(
# check if slow chart needs an x-domain shift and/or
# y-range resize.
if isinstance(msg, dict):
backfill: tuple[str, float]|None = msg.get(
'backfilling',
)
if backfill is not None:
market: str
period_s: float
market, period_s = backfill
if market == ds.fqme:
changed_chart: ChartPlotWidget
for changed_chart in (
ds.chart, ds.hist_chart,
):
hidden: bool = (
changed_chart.linked.isHidden()
)
update_trend_overlays(
changed_chart,
fqme=market,
chart_period_s=period_s,
force=True,
render=not hidden,
)
# Sampler advancement can finalize an FSP row without a
# quote. A writer publishing after this callback appears
# on the next tick; `TrendOverlay._signature` avoids a
# redundant rebuild in `TrendOverlay.update()`.
trend_chart: ChartPlotWidget
for trend_chart in (ds.chart, ds.hist_chart):
update_trend_overlays(
trend_chart,
fqme=ds.fqme,
render=not trend_chart.linked.isHidden(),
)
(
uppx,
liv,
@ -625,6 +661,17 @@ def graphics_update_cycle(
chart = ds.chart
vlm_chart = ds.vlm_chart
# Each market keeps its own destination-backed annotation layers.
# A forced cycle also handles source history coordinate rewrites.
trend_chart: ChartPlotWidget
for trend_chart in (chart, ds.hist_chart):
update_trend_overlays(
trend_chart,
fqme=fqme,
force=trigger_all,
render=not trend_chart.linked.isHidden(),
)
# varz = ds.vars
l1 = ds.l1
flume = ds.flume
@ -1522,11 +1569,17 @@ async def display_symbol_data(
# load (user's) FSP set (otherwise known as "indicators")
# from an input config.
tn.start_soon(
partial(
start_fsp_displays,
rt_linked,
flume,
# loading_sym_key,
loglevel,
show_trend_calculus=(
ui_conf.get('trend_calculus') is True
),
trend_timeframes=ui_conf.get('trend_timeframes'),
),
)
godwidget.resize_all()

420
piker/ui/_fsp.py
View File

@ -23,9 +23,11 @@ Financial signal processing cluster and real-time graphics management.
from __future__ import annotations
from contextlib import asynccontextmanager as acm
from collections.abc import AsyncIterator, Callable
from functools import partial
import inspect
from itertools import cycle
import time
from typing import (
AsyncGenerator,
Any,
@ -47,12 +49,18 @@ from piker.fsp import (
Fsp,
dolla_vlm,
flow_rates,
trend_calculus,
)
from tractor.ipc._shm import (
ShmArray,
NDToken,
)
from piker.data import Flume
from piker.data import Flume, open_sample_stream
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
from piker.fsp._regime_feed import (
consume_regime_source,
source_snapshot_is_current,
)
from piker.data._sharedmem import try_read
from piker.log import get_logger
from piker.toolz import Profiler
@ -72,6 +80,8 @@ from ._forms import (
mk_form,
open_form_input_handling,
)
from ._trend import TrendOverlay
from ._trend_mtf import SnapshotTrendOverlay, _normalize_periods
if TYPE_CHECKING:
from ._dataviz import Viz
@ -80,6 +90,53 @@ if TYPE_CHECKING:
log = get_logger(__name__)
def update_trend_overlays(
chart: ChartPlotWidget,
*,
fqme: str|None = None,
shm_token: NDToken|None = None,
chart_period_s: float|None = None,
force: bool = False,
render: bool = True,
) -> int:
'''
Refresh matching ``ChartPlotWidget._trend_overlays`` layers.
Its keys are ``(fqme, destination_token)`` and each value holds
independently owned reversal orders. Hidden-chart revisions
invalidate their layers without painting; the next quote then
rebuilds even if SHM bounds and the latest sample are unchanged.
'''
layers: dict[tuple[str, NDToken], list[TrendOverlay]] = getattr(
chart, '_trend_overlays', {},
)
updated: int = 0
key: tuple[str, NDToken]
overlays: list[TrendOverlay]
for key, overlays in layers.items():
market: str
token: NDToken
market, token = key
if fqme is not None and market != fqme:
continue
if shm_token is not None and token != shm_token:
continue
overlay: TrendOverlay
for overlay in overlays:
if (
chart_period_s is not None
and overlay.chart_period_s != chart_period_s
):
continue
if force:
overlay.invalidate()
if render and overlay.update():
updated += 1
return updated
def update_fsp_chart(
viz: Viz,
graphics_name: str,
@ -466,6 +523,14 @@ class FspAdmin:
tuple[tractor.MsgStream, ShmArray]
] = {}
self._flow_registry: dict[NDToken, str] = {}
self._trend_engine: tuple[Flume, trio.Event]|None = None
self._trend_start_lock: trio.Lock = trio.Lock()
self._trend_overlays: list[
tuple[ChartPlotWidget, TrendOverlay]
] = []
self._multi_overlay: SnapshotTrendOverlay|None = None
self._multi_tracker: MultiTimeframeTrend|None = None
self._multi_scope: trio.CancelScope|None = None
# TODO: make this a `.src_flume` and add
# a `dst_flume`?
@ -476,6 +541,332 @@ class FspAdmin:
name, portal = next(self._rr_next_actor)
return portal
async def open_trend_overlay(
self,
*,
chart: ChartPlotWidget|None = None,
order: int = 1,
chart_period_s: float|None = None,
loglevel: str = 'info',
) -> TrendOverlay:
'''
Start one TC writer and attach confirmed price annotations.
Several charts or reversal orders reuse this admin's source
engine. The current FSP allocator supplies its realtime SHM
only; another sampling timeframe requires an independently
computed destination, attachable with ``TrendOverlay``.
No Jev request or credential lookup occurs on this path.
'''
if not 1 <= order <= 4:
raise ValueError('order must be 1..4')
if chart is None:
chart = self.linked.chart
fqme: str = self.flume.mkt.fqme
viz: Viz|None = chart.get_viz(fqme)
if viz is None:
raise ValueError(f'No price Viz for {fqme}')
if chart_period_s is None:
if viz.shm._token == self.flume.rt_shm._token:
chart_period_s = self.flume.rt_sample_period_s
elif viz.shm._token == self.flume._hist_shm_token:
chart_period_s = self.flume.hist_sample_period_s
else:
raise ValueError('chart_period_s is required')
if not np.isfinite(chart_period_s) or chart_period_s <= 0:
raise ValueError('chart_period_s must be positive')
async with self._trend_start_lock:
owner: ChartPlotWidget
overlay: TrendOverlay
for owner, overlay in self._trend_overlays:
if (
owner is chart
and type(overlay) is TrendOverlay
and overlay.order == order
):
if overlay.chart_period_s != chart_period_s:
raise ValueError('chart period differs')
return overlay
if self._trend_engine is None:
self._trend_engine = await self.start_engine_task(
trend_calculus,
conf={},
loglevel=loglevel,
)
destination: Flume
started: trio.Event
destination, started = self._trend_engine
await started.wait()
overlay = TrendOverlay(
viz,
destination.rt_shm,
fqme=fqme,
source_period_s=self.flume.rt_sample_period_s,
chart_period_s=chart_period_s,
order=order,
)
try:
overlay.update(force=True)
except BaseException:
overlay.remove()
raise
layers: dict[
tuple[str, NDToken], list[TrendOverlay]
]|None = getattr(chart, '_trend_overlays', None)
if layers is None:
layers = chart._trend_overlays = {}
key: tuple[str, NDToken] = (
fqme, destination.rt_shm._token,
)
layers.setdefault(key, []).append(overlay)
self._trend_overlays.append((chart, overlay))
return overlay
async def open_multi_timeframe_trends(
self,
*,
periods: tuple[float, ...] = (1, 5, 15),
windows: tuple[int, ...] = (2, 2, 2, 2),
order: int = 1,
on_snapshot: Callable[[RegimeSnapshot], object]|None = None,
on_invalidate: Callable[[int], object]|None = None,
max_source_age: float|None = 5,
classification_ttl: float = 5,
max_source_lag: float|None = 2,
) -> SnapshotTrendOverlay:
'''
Follow one finalized source into real multiple-timeframe UI.
The tracker consumes native realtime closes once and shares
its immutable snapshots with all graphics layers and optional
callbacks. Pass a classifier's ``submit`` and ``invalidate``
callbacks together to reuse this same evidence for Jev; no
client or credential is created here. The returned adapter's
``publish_classification`` fits the async classifier worker.
Only the optional model callback uses ``max_source_age``;
historical graphics always render. Explicit ``None`` permits
old model evidence for offline research. Jev requests project
shorter evidence tails without changing graphics retention.
Return after the first finalized snapshot. Empty startup
history waits for a final source bar.
``FspAdmin.close_trend_overlays()`` cancels
``FspAdmin._multi_scope`` and removes its layers from
``ChartPlotWidget._trend_overlays`` on context exit.
'''
if (
max_source_age is not None
and
(
not np.isfinite(max_source_age)
or max_source_age <= 0
)
):
raise ValueError(
'max_source_age must be positive or None',
)
periods = _normalize_periods(
self.flume.rt_sample_period_s, periods,
)
async with self._trend_start_lock:
existing: SnapshotTrendOverlay|None = self._multi_overlay
if existing is not None:
first_layer: TrendOverlay = next(
iter(existing.layers.values()),
)
if (
existing.periods != periods
or first_layer.order != order
or self._multi_tracker.windows != windows
):
raise ValueError('tracker configuration differs')
if (
on_snapshot is not None
or on_invalidate is not None
):
raise ValueError('callbacks already configured')
return existing
chart: ChartPlotWidget = self.linked.chart
fqme: str = self.flume.mkt.fqme
viz: Viz|None = chart.get_viz(fqme)
if viz is None:
raise ValueError(f'No price Viz for {fqme}')
base_period: float = self.flume.rt_sample_period_s
if base_period < 1:
raise ValueError('native sampler requires >=1s bars')
if not 1 <= order <= min(4, len(windows)):
raise ValueError('order exceeds configured windows')
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
fqme,
base_period=base_period,
periods=periods,
windows=windows,
history_limit=256,
)
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
viz,
self.flume.rt_shm,
fqme=fqme,
base_period=base_period,
periods=periods,
chart_period_s=base_period,
order=order,
classification_ttl=classification_ttl,
max_source_lag=max_source_lag,
is_visible=lambda: not self.linked.isHidden(),
)
self._multi_overlay = adapter
self._multi_tracker = tracker
layers: dict[
tuple[str, NDToken], list[TrendOverlay]
]|None = getattr(chart, '_trend_overlays', None)
if layers is None:
layers = chart._trend_overlays = {}
key: tuple[str, NDToken] = (
fqme, self.flume.rt_shm._token,
)
layer: TrendOverlay
for layer in adapter.layers.values():
layers.setdefault(key, []).append(layer)
self._trend_overlays.append((chart, layer))
started: trio.Event = trio.Event()
self.tn.start_soon(
self._run_multi_timeframe_trends,
adapter,
tracker,
started,
on_snapshot,
on_invalidate,
max_source_age,
)
await started.wait()
if adapter.as_of is None and self._multi_scope is None:
raise RuntimeError('source closed before bootstrap')
return adapter
async def _run_multi_timeframe_trends(
self,
adapter: SnapshotTrendOverlay,
tracker: MultiTimeframeTrend,
started: trio.Event,
on_snapshot: Callable[[RegimeSnapshot], object]|None,
on_invalidate: Callable[[int], object]|None,
max_source_age: float|None,
) -> None:
'''
Subscribe before bootstrap and fan out one writer's evidence.
'''
def submit(snapshot: RegimeSnapshot) -> None:
'''
Publish the bridge's completed, current source state.
'''
adapter.submit(snapshot)
if (
on_snapshot is not None
and source_snapshot_is_current(
snapshot,
now=time.time(),
max_source_age=max_source_age,
)
):
on_snapshot(snapshot)
started.set()
def invalidate(revision: int) -> None:
'''
Revoke graphics and optional model work before replay.
'''
adapter.invalidate(revision)
if on_invalidate is not None:
on_invalidate(revision)
scope: trio.CancelScope
with trio.CancelScope() as scope:
self._multi_scope = scope
try:
stream: AsyncIterator[dict]
async with open_sample_stream(
period_s=tracker.base_period,
) as stream:
await consume_regime_source(
self.flume.rt_shm,
stream,
tracker,
submit,
invalidate=invalidate,
)
finally:
self._multi_scope = None
invalidate(adapter.revision + 1)
# Wake a bootstrap waiter if the source closes empty.
started.set()
def refresh_trend_overlays(
self,
shm_token: NDToken,
*,
render: bool,
) -> None:
'''
Route a destination rewrite to this admin's attached charts.
'''
charts: set[ChartPlotWidget] = set()
chart: ChartPlotWidget
overlay: TrendOverlay
for chart, overlay in self._trend_overlays:
if chart in charts:
continue
charts.add(chart)
update_trend_overlays(
chart,
shm_token=shm_token,
force=True,
render=render,
)
def close_trend_overlays(self) -> None:
'''
Remove this admin's layers while preserving sibling owners.
'''
if self._multi_scope is not None:
self._multi_scope.cancel()
if self._multi_overlay is not None:
self._multi_overlay.remove()
self._multi_overlay = None
self._multi_tracker = None
chart: ChartPlotWidget
overlay: TrendOverlay
for chart, overlay in self._trend_overlays:
layers: dict[
tuple[str, NDToken], list[TrendOverlay]
] = chart._trend_overlays
key: tuple[str, NDToken] = (
overlay.fqme, overlay.source._token,
)
owned: list[TrendOverlay] = layers.get(key, [])
if overlay in owned:
owned.remove(overlay)
if not owned:
layers.pop(key, None)
overlay.remove()
self._trend_overlays.clear()
async def open_chain(
self,
@ -540,17 +931,23 @@ class FspAdmin:
async for msg in stream:
info = msg.get('fsp_update')
if info:
token: NDToken = NDToken.from_msg(
dict(info['key']),
)
visible: bool = not self.linked.isHidden()
# if the chart isn't hidden try to update
# the data on screen.
if not self.linked.isHidden():
if visible:
log.debug(
f'Re-syncing graphics for fsp: {ns_path}'
)
update_fsp_vizs(
self.linked,
shm_token=NDToken.from_msg(
dict(info['key']),
),
shm_token=token,
)
self.refresh_trend_overlays(
token,
render=visible,
)
else:
log.info(f'recved unexpected fsp engine msg: {msg}')
@ -710,6 +1107,7 @@ async def open_fsp_admin(
# ??TODO, does this *need* to be inside a finally?
finally:
admin.close_trend_overlays()
# terminate all tasks via signals
for key, entry in admin._registry.items():
_, _, event = entry
@ -1027,6 +1425,8 @@ async def start_fsp_displays(
flume: Flume,
# group_status_key: str,
loglevel: str,
show_trend_calculus: bool = False,
trend_timeframes: tuple[float, ...]|None = None,
) -> None:
'''
@ -1072,6 +1472,13 @@ async def start_fsp_displays(
flume,
) as admin,
):
if trend_timeframes is not None:
await admin.open_multi_timeframe_trends(
periods=tuple(trend_timeframes),
)
elif show_trend_calculus:
await admin.open_trend_overlay(loglevel=loglevel)
statuses: list[trio.Event] = []
for target, conf in fsp_conf.items():
started: trio.Event = await admin.open_fsp_chart(
@ -1091,4 +1498,7 @@ async def start_fsp_displays(
profiler(f'attached to fsp portal: {target}')
# status_cb()
if show_trend_calculus or trend_timeframes is not None:
await trio.sleep_forever()
# blocks on nursery until all fsp actors complete

1009
piker/ui/_trend.py 100644
View File

File diff suppressed because it is too large Load Diff

547
piker/ui/_trend_mtf.py 100644
View File

@ -0,0 +1,547 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Native multiple-timeframe annotations from finalized feed snapshots.
'''
from __future__ import annotations
from collections import deque
from collections.abc import Callable
from math import isfinite
from typing import TYPE_CHECKING
from msgspec.structs import replace
import numpy as np
from piker.fsp._regime import (
RegimeSnapshot,
TimeframeRegime,
_period_us,
)
from ._trend import (
RegimeInterval,
ReversalPoint,
TrendOverlay,
_positive_integer,
times_to_x,
)
if TYPE_CHECKING:
from tractor.ipc._shm import ShmArray
from piker.fsp._jev import JevClassification, RegimeAnswer
from piker.fsp._regime import TrendReversal
from ._dataviz import Viz
def _normalize_periods(
base_period: float,
periods: tuple[float, ...],
) -> tuple[float, ...]:
'''
Match the shared tracker's canonical microsecond durations.
'''
base_us: int = _period_us(base_period)
period: float # noqa: F842
period_us: tuple[int, ...] = tuple(
sorted(_period_us(period) for period in periods)
)
if not 1 <= len(period_us) <= 16:
raise ValueError('configure 1..16 unique timeframes')
if len(set(period_us)) != len(period_us):
raise ValueError('timeframes must be unique')
value: int # noqa: F842
if any(value % base_us for value in period_us):
raise ValueError('timeframes require whole base multiples')
return tuple(value / 1_000_000 for value in period_us)
class _SnapshotLayer(TrendOverlay):
'''
Render one real aggregated timeframe using existing Qt batches.
The source is the actual base OHLC SHM. ``_SnapshotLayer._frame``
supplies computed evidence; no pretend FSP SHM or repeated TC
computation is needed. Snapshot pivots retain interval-end
timestamps while x positions identify their final base bar.
'''
def __init__(
self,
viz: Viz,
source: ShmArray,
*,
fqme: str,
source_period_s: float,
chart_period_s: float,
order: int = 1,
max_points: int = 4096,
max_rectangles: int = 1024,
) -> None:
'''
Initialize a bounded, empty timeframe layer.
'''
super().__init__(
viz,
source,
fqme=fqme,
source_period_s=source_period_s,
chart_period_s=chart_period_s,
order=order,
max_points=max_points,
max_rectangles=max_rectangles,
)
self._frame: TimeframeRegime|None = None
self._base_period: float = self.chart_period_s
self._runs: deque[RegimeInterval] = deque(
maxlen=self.max_rectangles,
)
self._active: bool = False
def clear_evidence(self) -> None:
'''
Revoke old source evidence immediately on revision or gap.
'''
self._frame = None
self._runs.clear()
self._active = False
self._external = None
self._native_intervals.clear()
self._set_points([])
self.rectangles.set_data(np.empty((0, 4)), [])
self.invalidate()
def accept_frame(
self,
frame: TimeframeRegime,
base_period: float,
horizon: float,
) -> None:
'''
Retain only causal live runs supported by received snapshots.
Bootstrap starts at the latest proven direction, since the
snapshot has no earlier trend history. If coalescing skips
complete buckets, end the previous run at its next possible
decision and leave the unknown interval unpainted.
'''
previous: TimeframeRegime|None = self._frame
self._frame = frame
self._base_period = base_period
level: int = self.order - 1
ready: bool = (
frame.as_of is not None
and len(frame.ready) > level
and len(frame.trends) > level
and frame.ready[level]
and abs(frame.trends[level]) == 1
)
label: str = (
'uptrend' if frame.trends[level] > 0 else 'downtrend'
) if len(frame.trends) > level else ''
skipped: bool = (
previous is not None
and frame.close_count > previous.close_count + 1
)
if self._active:
if skipped:
self._runs[-1] = replace(
self._runs[-1],
end_time=previous.as_of + frame.period,
)
self._active = False
elif not ready or self._runs[-1].label != label:
self._runs[-1] = replace(
self._runs[-1],
end_time=(
frame.as_of
if frame.as_of is not None
else horizon - base_period
),
)
self._active = False
if ready:
if self._active:
self._runs[-1] = replace(
self._runs[-1], end_time=horizon,
)
else:
self._runs.append(RegimeInterval(
start_time=frame.as_of,
end_time=horizon,
label=label,
))
self._active = True
self.invalidate()
def _prepare_evidence(
self,
source: np.ndarray,
chart: np.ndarray,
) -> None:
'''
Map immutable endpoint events to their actual source bars.
'''
frame: TimeframeRegime|None = self._frame
if frame is None:
self._set_points([])
self._native_intervals = []
return
events: list[TrendReversal] = []
event: TrendReversal
for event in frame.reversals:
if event.order == self.order:
events.append(event)
events = events[-self.max_points:]
times: np.ndarray = np.fromiter(
(event.time - self._base_period for event in events),
dtype=float,
count=len(events),
)
xs: np.ndarray = times_to_x(
chart,
times,
self.chart_period_s,
self.viz.index_field,
)
points: list[ReversalPoint] = []
x: np.floating
for event, x in zip(events, xs, strict=True):
if not np.isfinite(x):
continue
points.append(ReversalPoint(
x=float(x),
price=event.price,
pivot_time=event.time,
detected_at=event.detected_time,
kind='bottom' if event.direction > 0 else 'top',
order=self.order,
period_s=frame.period,
timestamp_convention='interval_end',
))
self._set_points(points)
self._native_intervals = list(self._runs)
class SnapshotTrendOverlay:
'''
Consume one shared tracker into bounded per-timeframe graphics.
Pass ``submit`` and ``invalidate`` directly to the native
``consume_regime_source`` bridge. Source revision and gap segment
changes synchronously revoke stale annotations; old or duplicate
snapshot sequences cannot repaint newer evidence.
``submit_classification`` is an optional local Jev callback.
It performs no I/O and never reads credentials. Model rectangles
have a finite ``classification_ttl`` after response receipt and
stay absent until the displayed event clock reaches receipt.
'''
def __init__(
self,
viz: Viz,
source: ShmArray,
*,
fqme: str,
base_period: float,
periods: tuple[float, ...] = (1, 5, 15),
chart_period_s: float = 1,
order: int = 1,
history_limit: int = 256,
classification_ttl: float = 5,
max_source_lag: float|None = 2,
is_visible: Callable[[], bool]|None = None,
) -> None:
'''
Allocate two scene items per configured sampling timeframe.
'''
periods = _normalize_periods(base_period, periods)
base_period = _period_us(base_period) / 1_000_000
order = _positive_integer(order, 'order')
history_limit = _positive_integer(
history_limit, 'history_limit',
)
if order > 4:
raise ValueError('order must be 1..4')
if history_limit > 256:
raise ValueError('history_limit must be 1..256')
if (
not isfinite(classification_ttl)
or classification_ttl <= 0
):
raise ValueError('classification_ttl must be positive')
if (
max_source_lag is not None
and
(
not isfinite(max_source_lag)
or max_source_lag < 0
)
):
raise ValueError('max_source_lag must be >=0 or None')
self.fqme: str = fqme
self.base_period: float = base_period
self.periods: tuple[float, ...] = periods
self.classification_ttl: float = classification_ttl
self.max_source_lag: float|None = max_source_lag
self.is_visible: Callable[[], bool] = (
is_visible or (lambda: True)
)
self.revision: int = 0
self.sequence: int = -1
self.segment: int|None = None
self.as_of: float|None = None
self._classification_sequence: int = -1
self._model_runs: dict[float, deque[RegimeInterval]] = {}
self._model_expiry: dict[float, float] = {}
self._removed: bool = False
self.layers: dict[float, _SnapshotLayer] = {}
try:
period: float
for period in periods:
self.layers[period] = _SnapshotLayer(
viz,
source,
fqme=fqme,
source_period_s=period,
chart_period_s=chart_period_s,
order=order,
max_points=history_limit,
max_rectangles=history_limit,
)
self._model_runs[period] = deque(
maxlen=history_limit,
)
except BaseException:
self.remove()
raise
def _clear_evidence(self) -> None:
'''
Clear this owner's deterministic and classified history.
'''
layer: _SnapshotLayer
for layer in self.layers.values():
layer.clear_evidence()
runs: deque[RegimeInterval]
for runs in self._model_runs.values():
runs.clear()
self._model_expiry.clear()
self._classification_sequence = -1
def invalidate(self, revision: int) -> None:
'''
Revoke an obsolete source generation before replay can yield.
'''
if self._removed or revision < self.revision:
return
self.revision = revision
self.sequence = -1
self.segment = None
self.as_of = None
self._clear_evidence()
def submit(self, snapshot: RegimeSnapshot) -> bool:
'''
Accept the latest native evidence and refresh visible layers.
'''
if (
self._removed
or snapshot.market != self.fqme
or snapshot.revision < self.revision
):
return False
if snapshot.base_period != self.base_period:
raise ValueError('snapshot base period differs')
if snapshot.revision > self.revision:
self.invalidate(snapshot.revision)
if snapshot.sequence <= self.sequence:
return False
if self.as_of is not None and snapshot.as_of < self.as_of:
return False
frame: TimeframeRegime # noqa: F842
frames: dict[float, TimeframeRegime] = {
frame.period: frame for frame in snapshot.timeframes
}
if not set(self.periods) <= frames.keys():
raise ValueError('snapshot is missing a timeframe')
if (
self.segment is not None
and snapshot.segment != self.segment
):
self._clear_evidence()
self.segment = snapshot.segment
self.sequence = snapshot.sequence
self.as_of = snapshot.as_of
horizon: float = snapshot.as_of + self.base_period
period: float
layer: _SnapshotLayer
for period, layer in self.layers.items():
layer.accept_frame(
frames[period], self.base_period, horizon,
)
self._extend_model(period, horizon)
if self.is_visible():
layer.update()
return True
def _extend_model(self, period: float, horizon: float) -> None:
'''
Extend the active model interval only through its expiry.
'''
runs: deque[RegimeInterval] = self._model_runs[period]
if not runs:
return
runs[-1] = replace(
runs[-1],
end_time=min(horizon, self._model_expiry[period]),
)
layer: _SnapshotLayer = self.layers[period]
layer._external = tuple(runs)
layer._signature = None
def submit_classification(
self,
result: JevClassification,
) -> bool:
'''
Attach a matching, newly received Jev result without API I/O.
Older requests may finish after newer chart snapshots; source
sequence must increase between accepted model responses and
never exceed the chart's known sequence. Revision and segment
must match exactly. Caller-owned provider freshness checks
still apply before invoking this callback.
'''
if (
self._removed
or self.as_of is None
or result.market != self.fqme
or result.revision != self.revision
or result.segment != self.segment
or result.sequence <= self._classification_sequence
or result.sequence > self.sequence
or not isfinite(result.source_as_of)
or result.source_as_of > self.as_of
or not isfinite(result.received_at)
or (
result.received_at + self.classification_ttl
<= self.as_of
)
):
return False
if (
self.max_source_lag is not None
and (
self.as_of - result.source_as_of
> self.max_source_lag
)
):
return False
answers: list[RegimeAnswer] = []
answer: RegimeAnswer
for answer in result.answers:
if answer.period not in self.layers:
continue
if not isfinite(answer.probability):
return False
if not 0 <= answer.probability <= 1:
return False
answers.append(answer)
if not answers:
return False
available: float = max(
result.source_as_of, result.received_at,
)
horizon: float = self.as_of + self.base_period
expiry: float = available + self.classification_ttl
model: str = result.model
for answer in answers:
period: float = answer.period
runs: deque[RegimeInterval] = self._model_runs[period]
if runs:
runs[-1] = replace(
runs[-1],
end_time=min(
available, self._model_expiry[period],
),
)
runs.append(RegimeInterval(
start_time=result.source_as_of,
end_time=min(horizon, expiry),
label=answer.label,
probability=answer.probability,
source=f'typesafe.ai/{model}',
observed_at=available,
))
self._model_expiry[period] = expiry
layer: _SnapshotLayer = self.layers[period]
layer._external = tuple(runs)
layer._signature = None
if self.is_visible():
layer.update()
self._classification_sequence = result.sequence
return True
def remove(self) -> None:
'''
Remove every timeframe owned by this adapter, idempotently.
'''
if self._removed:
return
layer: _SnapshotLayer
for layer in self.layers.values():
layer.remove()
self._removed = True
async def publish_classification(
self,
result: JevClassification,
) -> None:
'''
Adapt the synchronous graphics callback to classifier.run().
No await occurs inside the mutation, so revision invalidation
cannot interleave halfway through a Qt update.
'''
self.submit_classification(result)

876
tests/_inputs/trend_calculus_lua.json 100644
View File

@ -0,0 +1,876 @@
{
"source": "https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua",
"source_sha256": "c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0",
"source_license": "GPL-3.0-or-later; Copyright 2014-2015 ByteSumo Limited (Andrew J Morgan)",
"fixture_license": "AGPL-3.0-or-later",
"generated_at": "2026-10-05",
"runtime": "Lua 5.2.4",
"command": "lua trendcalculus.lua -n{window} -r -H -p1",
"stdin": "ticker,date,last header followed by x,{zero_based_row},{value} rows, LF-terminated",
"note": "Do not use -h: embedded CSV column mapping already consumes the header and -h drops the first data row.",
"detection": "Run every complete-window prefix through the unmodified CLI. Newly emitted rows get detected_index = prefix_length - 1. Assert output-prefix preservation.",
"event_fields": [
"index",
"price",
"direction",
"detected_index"
],
"cases": [
{
"window": 2,
"values": [
0,
1,
2,
3,
4,
5,
4,
7,
9,
12,
9,
6,
4,
1,
0,
-2,
-1,
-3,
-3,
-3,
-6,
-9,
-8,
-6,
-8,
-9,
-8,
-9,
-7,
-8,
-9,
-10,
-7,
-6,
-6,
-6,
-7,
-7,
-5,
-4,
-6,
-6,
-9,
-7,
-7,
-6,
-4,
-1,
-1,
-4,
-3,
-4,
-6,
-4,
-6,
-6,
-9,
-9,
-7,
-6,
-4,
-7,
-5,
-7,
-5,
-2,
-1,
-3,
-3,
0,
-3,
-3,
-1,
-4,
-2,
1,
1,
-1,
-4,
-1,
0,
1,
1,
-1,
-2,
-1,
1,
3,
6,
8,
8,
8,
10,
8,
8,
5,
5,
2,
1,
1,
-2,
-1,
1,
-2,
1,
-1
],
"events": [
[
9,
12.0,
-1,
11
],
[
21,
-9.0,
1,
23
],
[
23,
-6.0,
-1,
25
],
[
27,
-9.0,
1,
29
],
[
28,
-7.0,
-1,
31
],
[
31,
-10.0,
1,
33
],
[
34,
-6.0,
-1,
37
],
[
36,
-7.0,
1,
39
],
[
39,
-4.0,
-1,
41
],
[
42,
-9.0,
1,
45
],
[
48,
-1.0,
-1,
49
],
[
56,
-9.0,
1,
59
],
[
60,
-4.0,
-1,
63
],
[
63,
-7.0,
1,
65
],
[
69,
0.0,
-1,
71
],
[
70,
-3.0,
1,
73
],
[
72,
-1.0,
-1,
73
],
[
73,
-4.0,
1,
75
],
[
76,
1.0,
-1,
79
],
[
78,
-4.0,
1,
81
],
[
81,
1.0,
-1,
83
],
[
84,
-2.0,
1,
87
],
[
92,
10.0,
-1,
95
],
[
100,
-2.0,
1,
103
]
],
"input_csv_sha256": "8b146558fa0f23f8926404bfeb87e433f02dd253c5a8eed2cf7bec21fa7d62ab",
"output_csv_sha256": "46915ca3dcb7da1b3f9e72a8ece1a8beb032a4f99657798022236150d9c336af"
},
{
"window": 3,
"values": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
11,
14,
16,
19,
19,
17,
14,
15,
17,
19,
18,
16,
18,
19,
17,
16,
19,
19,
18,
15,
16,
16,
15,
14,
17,
16,
17,
14,
14,
17,
20,
18,
21,
20,
22,
21,
20,
18,
21,
20,
23,
22,
19,
19,
18,
19,
18,
18,
21,
24,
23,
21,
23,
26,
26,
28,
25,
28,
31,
30,
27,
24,
24,
26,
26,
28,
31,
33,
35,
33,
34,
32,
32,
31,
28,
29,
30,
33,
33,
31,
34,
33,
30,
29,
28,
29,
31,
33,
33,
33,
30,
29,
28,
28,
30,
33,
34,
35,
35,
34
],
"events": [
[
12,
19.0,
-1,
17
],
[
15,
14.0,
1,
20
],
[
22,
19.0,
-1,
26
],
[
32,
14.0,
1,
35
],
[
33,
17.0,
-1,
38
],
[
36,
14.0,
1,
41
],
[
43,
22.0,
-1,
47
],
[
46,
18.0,
1,
50
],
[
49,
23.0,
-1,
53
],
[
55,
18.0,
1,
59
],
[
67,
31.0,
-1,
71
],
[
70,
24.0,
1,
74
],
[
77,
35.0,
-1,
80
],
[
83,
28.0,
1,
86
],
[
89,
34.0,
-1,
92
],
[
93,
28.0,
1,
98
],
[
96,
33.0,
-1,
101
],
[
101,
28.0,
1,
104
]
],
"input_csv_sha256": "38beb51c87acae40c3226599d6819cd6b9f984b39428ba84d1995257ce2bbc1c",
"output_csv_sha256": "cb847cc137aee4433d9d6b70a1a1c74d02a70325c2ffe771130d6626449a40d4"
},
{
"window": 4,
"values": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
11,
14,
13,
16,
17,
19,
17,
18,
17,
16,
13,
12,
15,
16,
19,
21,
23,
25,
23,
25,
24,
24,
22,
21,
21,
19,
16,
17,
16,
13,
12,
15,
15,
13,
15,
12,
15,
17,
18,
21,
20,
18,
16,
16,
16,
16,
17,
19,
18,
16,
19,
18,
21,
23,
21,
24,
24,
26,
29,
31,
32,
29,
26,
26,
26,
29,
31,
32,
29,
29,
28,
25,
22,
25,
22,
20,
22,
25,
27,
24,
22,
25,
22,
24,
22,
19,
20,
21,
19,
19,
22,
24,
25,
24,
25,
26,
23,
23,
24,
26
],
"events": [
[
17,
19.0,
-1,
23
],
[
23,
12.0,
1,
27
],
[
29,
25.0,
-1,
35
],
[
47,
12.0,
1,
51
],
[
51,
21.0,
-1,
55
],
[
61,
16.0,
1,
67
],
[
72,
32.0,
-1,
79
],
[
74,
26.0,
1,
79
],
[
79,
32.0,
-1,
83
],
[
87,
20.0,
1,
91
],
[
90,
27.0,
-1,
95
],
[
97,
19.0,
1,
103
],
[
107,
26.0,
-1,
111
]
],
"input_csv_sha256": "303cbcb636decdd1a7c6b3841ea696fec0881a95023eeb7af71132a501b53e83",
"output_csv_sha256": "df8be88430783a0669fb9224952fe7a26f4fcf18e3f242e425b7279e75f69850"
},
{
"window": 7,
"values": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
24,
27,
29,
26,
23,
25,
25,
24,
26,
23,
22,
19,
21,
21,
18,
21,
21,
18,
15,
14,
14,
11,
10,
7,
6,
5,
2,
1,
1,
-2,
-2,
-2,
-5,
-4,
-3,
-3,
-1,
-4,
-1,
0,
3,
3,
2,
4,
3,
1,
-1,
-1,
1,
2,
2,
3,
2,
4,
3,
1,
4,
3,
2,
0,
-1,
-3,
-6,
-3,
-2,
-5,
-2,
-1,
1,
-2,
-1,
-1,
-2,
-2,
-5,
-5,
-6,
-4,
-2,
0,
-2,
-1,
-2,
-3,
-1,
1,
3,
5,
3,
2,
-1,
-4,
-7,
-5,
-4,
-5,
-7,
-7,
-7
],
"events": [
[
24,
29.0,
-1,
34
],
[
54,
-5.0,
1,
62
],
[
75,
4.0,
-1,
83
],
[
84,
-6.0,
1,
97
],
[
90,
1.0,
-1,
97
],
[
98,
-6.0,
1,
104
],
[
109,
5.0,
-1,
118
]
],
"input_csv_sha256": "01a6232710be1d4dd606fb077855c424e1b43b17ec602c59b28b6db4f87249db",
"output_csv_sha256": "37fbaaf015d2020eff07e62c2aaeb2de0e4145dea85403c29a98dbaed94cf3d9"
}
]
}

1043
tests/test_jev.py 100644
View File

File diff suppressed because it is too large Load Diff

771
tests/test_regime_feed.py 100644
View File

@ -0,0 +1,771 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Native SHM-to-regime contracts, using synthetic source publications.
'''
from collections.abc import AsyncIterator, Callable
from contextlib import asynccontextmanager
from types import SimpleNamespace
from typing import Any, Never
import json
import httpx
import numpy as np
import pytest
import trio
from tractor.trionics import Lagged
from trio.testing import wait_all_tasks_blocked
from piker.fsp._regime import (
CloseSample,
MultiTimeframeTrend,
RegimeSnapshot,
)
from piker.fsp._regime_feed import (
ClosedSourceRows,
SourceChanged,
consume_regime_source,
copy_closed_rows,
run_regime_feed,
serve_regimes,
)
class FakeSource:
'''
Model absolute bounds independently of mutable backing storage.
'''
def __init__(
self,
first: int = 0,
last: int = 6,
) -> None:
'''
Seed indexed rows independently of their published bounds.
'''
self._first: SimpleNamespace = SimpleNamespace(value=first)
self._last: SimpleNamespace = SimpleNamespace(value=last)
self._array: np.ndarray = np.zeros(128, dtype=[
('index', '<i8'),
('time', '<f8'),
('close', '<f8'),
])
self._array['index'] = np.arange(128)
self._array['time'] = np.arange(128)
self._array['close'] = 100 + np.sin(np.arange(128) / 3)
def test_closed_copy_excludes_mutable_tail_and_detaches_memory(
) -> None:
'''
Treating the mutable source tail as finalized leaks provisional
closes into regime decisions. Make that tail NaN and verify the
detached prefix remains valid, immutable and independent of a
later shared-memory write. Absolute first/last bounds survive.
'''
source: FakeSource = FakeSource(first=2, last=8)
source._array['close'][7] = np.nan
copied: ClosedSourceRows = copy_closed_rows(source)
assert (copied.first, copied.last, copied.start) == (2, 8, 2)
assert len(copied.rows) == 5
assert copied.rows['index'][-1] == 6
assert not copied.rows.flags.writeable
original: float = copied.rows['close'][0]
source._array['close'][2] = 999
assert copied.rows['close'][0] == original
@pytest.mark.parametrize('first,last', [(0, 8), (2, 6)])
def test_changed_bounds_reject_mixed_copy(
first: int,
last: int,
) -> None:
'''
A source prepend or truncation between the first bound read and
array access invalidates the copied range. Publish that change
from a controlled _array property access; the second bound check
must reject the candidate before any tracker can consume it.
'''
class ChangingSource:
'''
Trigger a deterministic source prepend while copying begins.
'''
def __init__(self) -> None:
'''
Retain a prefix whose origin or last bound will change.
'''
self._first: SimpleNamespace = SimpleNamespace(value=2)
self._last: SimpleNamespace = SimpleNamespace(value=8)
self.storage: np.ndarray = FakeSource()._array
@property
def _array(self) -> np.ndarray:
'''
Publish the changed origin after capturing old bounds.
'''
self._first.value = first
self._last.value = last
return self.storage
with pytest.raises(SourceChanged, match='changed during copy'):
copy_closed_rows(ChangingSource())
def test_append_during_copy_preserves_closed_prefix() -> None:
'''
A high-frequency producer may append during every large history
copy. Its old closed prefix remains stable because the captured
mutable tail was excluded. Accept the detached prefix with its
original last bound, then consume the newly closed suffix once
without duplicating observations or starving bootstrap retries.
'''
class AppendingSource:
'''
Publish two rows after the reader captures initial bounds.
'''
def __init__(self) -> None:
'''
Retain closed rows before publishing an appended suffix.
'''
self._first: SimpleNamespace = SimpleNamespace(value=0)
self._last: SimpleNamespace = SimpleNamespace(value=6)
self.storage: np.ndarray = FakeSource()._array
@property
def _array(self) -> np.ndarray:
'''
Preserve the existing prefix while exposing a new tail.
'''
self._last.value = 8
return self.storage
source: AppendingSource = AppendingSource()
copied: ClosedSourceRows = copy_closed_rows(source)
assert (copied.first, copied.last, copied.start) == (0, 6, 0)
assert copied.rows['index'].tolist() == [0, 1, 2, 3, 4]
suffix: ClosedSourceRows = copy_closed_rows(
source, start=copied.last - 1,
)
assert (suffix.first, suffix.last, suffix.start) == (0, 8, 5)
assert suffix.rows['index'].tolist() == [5, 6]
@pytest.mark.parametrize('first,last', [(-1, 6), (8, 6), (0, 129)])
def test_stable_invalid_bounds_fail_instead_of_spinning(
first: int,
last: int,
) -> None:
'''
Stable invalid bounds cannot become a usable source through a
tight retry loop. Treat negative, reversed and out-of-capacity
ranges as source errors, distinguishing them from bounds that
changed during the two-read copy validation window.
'''
with pytest.raises(ValueError, match='readable range'):
copy_closed_rows(FakeSource(first, last))
def test_unsigned_coordinate_overflow_is_rejected_before_cast(
) -> None:
'''
Converting uint64 coordinates to the core's int64 storage can
wrap an oversized value negative. Put that value last in a valid
increasing finalized prefix so ordering checks cannot mask the
overflow; source validation must reject it before replay begins.
'''
source: FakeSource = FakeSource(last=3)
source._array = source._array.astype([
('index', '<u8'),
('time', '<f8'),
('close', '<f8'),
])
source._array['index'][1] = np.uint64(2 ** 63)
with pytest.raises(ValueError, match='invalid finalized'):
copy_closed_rows(source)
@pytest.mark.parametrize('field,value', [
('close', float('nan')),
('close', float('inf')),
('time', float('nan')),
('time', 0),
('index', -1),
])
def test_bad_finalized_source_is_rejected_before_consumption(
field: str,
value: float,
) -> None:
'''
Invalid finalized rows must not partially warm a tracker. Corrupt
a middle row and require validation of the whole detached range
before it can be handed to the recurrence; mutable-tail exclusion
is covered separately because unfinished rows may be invalid.
'''
source: FakeSource = FakeSource()
source._array[field][2] = value
with pytest.raises(ValueError, match='invalid finalized'):
copy_closed_rows(source)
def test_bootstrap_and_skipped_steps_emit_only_latest_snapshot(
) -> None:
'''
A missed sampler wakeup must not skip finalized closes, and a
bootstrap must not create one HTTP job per historical bar.
Load five closed rows, mutate the open sixth row, then append
three rows with one wakeup. Exactly two snapshots must cover the
complete prefixes; duplicate and foreign messages produce none.
'''
published: list[RegimeSnapshot] = []
async def scenario() -> None:
'''
Drive source publication and wakeups in a controlled order.
'''
source: FakeSource = FakeSource()
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', periods=(1, 2),
)
send: trio.MemorySendChannel
receive: trio.MemoryReceiveChannel
send, receive = trio.open_memory_channel(0)
first: trio.Event = trio.Event()
def submit(snapshot: RegimeSnapshot) -> None:
'''
Observe publication synchronously at the admission point.
'''
published.append(snapshot)
first.set()
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(
consume_regime_source,
source, receive, tracker, submit,
)
await first.wait()
source._array['close'][5] = 500
await send.send({'period': 1})
await send.send({'period': 60})
await send.send({'backfilling': ['other.market', 1]})
await send.send({'backfilling': ['test.market', 60]})
await wait_all_tasks_blocked()
assert len(published) == 1
source._last.value = 9
await send.send({'period': 1})
await send.aclose()
trio.run(scenario)
assert len(published) == 2
assert published[0].sequence == 5
assert published[1].sequence == 8
assert published[1].as_of == 8
assert published[1].revision == 0
closes: tuple[CloseSample, ...] = (
published[1].timeframes[0].recent_closes
)
assert closes[5].price == 500
@pytest.mark.parametrize(
'change', ['in_place', 'prepend', 'truncate'],
)
def test_history_changes_revoke_and_replay(change: str) -> None:
'''
Appending revised history to warm recurrence state is incorrect,
including repairs that leave both bounds unchanged. Publish a
source change and verify immediate revision invalidation precedes
its replacement snapshot. The rebuilt result must equal replay
through a fresh tracker over exactly the corrected closed prefix.
'''
published: list[RegimeSnapshot] = []
order: list[tuple[str, int]] = []
async def scenario() -> None:
'''
Compare each revised output with an independent full replay.
'''
source: FakeSource = FakeSource(first=2, last=20)
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', periods=(1, 2),
)
send: trio.MemorySendChannel
receive: trio.MemoryReceiveChannel
send, receive = trio.open_memory_channel(0)
first: trio.Event = trio.Event()
def submit(snapshot: RegimeSnapshot) -> None:
'''
Preserve publication order and source generation.
'''
published.append(snapshot)
order.append(('snapshot', snapshot.revision))
first.set()
def invalidate(revision: int) -> None:
'''
Record when old pending judgments become inadmissible.
'''
order.append(('invalidate', revision))
async def consume() -> None:
'''
Attach both source callbacks to one consumer lifetime.
'''
await consume_regime_source(
source, receive, tracker, submit,
invalidate=invalidate,
)
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(consume)
await first.wait()
if change == 'in_place':
source._array['close'][4] = 200
await send.send({
'backfilling': ['test.market', 1],
})
else:
if change == 'prepend':
source._first.value = 0
else:
source._last.value = 10
await send.send({'period': 1})
await send.aclose()
reference: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', periods=(1, 2), revision=1,
)
row: np.void
for row in copy_closed_rows(source).rows:
expected: RegimeSnapshot = reference.push(
float(row['time']), float(row['close']),
index=int(row['index']),
)
assert published[-1] == expected
trio.run(scenario)
assert order == [
('snapshot', 0), ('invalidate', 1), ('snapshot', 1),
]
def test_revision_during_bootstrap_suppresses_old_publication(
) -> None:
'''
Losing a backfill during bootstrap publishes a mixed-generation
classification that ordinary bound checks cannot detect.
``SignallingTracker.push()`` sets ``entered`` after one row;
``consume_regime_source()`` yields at every replay row so the
``scenario()`` task can repair a close and send its notice.
``_Notices.collect()`` must invalidate that revision before the
replay publishes. One replacement snapshot, its corrected close
and full sequence prove old evidence never reaches ``published``.
'''
published: list[RegimeSnapshot] = []
invalidated: list[int] = []
async def scenario() -> None:
'''
Synchronize repair with the first chunk's cooperative yield.
'''
entered: trio.Event = trio.Event()
class SignallingTracker(MultiTimeframeTrend):
'''
Expose the start of bootstrap without adding I/O to push.
'''
def push(
self,
*args: Any,
**kwargs: Any,
) -> RegimeSnapshot:
'''
Signal after the first historical row is consumed.
'''
snapshot: RegimeSnapshot = super().push(
*args, **kwargs,
)
entered.set()
return snapshot
source: FakeSource = FakeSource(last=100)
tracker: SignallingTracker = SignallingTracker(
'test.market', periods=(1,), history_limit=128,
)
send: trio.MemorySendChannel
receive: trio.MemoryReceiveChannel
send, receive = trio.open_memory_channel(0)
async def consume() -> None:
'''
Yield each row so the controlled repair reaches replay.
'''
await consume_regime_source(
source, receive, tracker, published.append,
invalidate=invalidated.append,
replay_chunk=1,
)
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(consume)
await entered.wait()
source._array['close'][0] = 999
await send.send({'backfilling': ['test.market', 1]})
await send.aclose()
trio.run(scenario)
assert invalidated == [1]
assert len(published) == 1
assert published[0].revision == 1
assert published[0].sequence == 99
assert published[0].timeframes[0].recent_closes[0].price == 999
@pytest.mark.parametrize('quote_lag', [False, True])
@pytest.mark.parametrize('source_age', [None, 0, 60, -10])
def test_native_wrapper_subscribes_filters_age_and_drains_quotes(
monkeypatch: pytest.MonkeyPatch,
quote_lag: bool,
source_age: float|None,
) -> None:
'''
Subscribing after history capture loses in-place revisions in the
startup gap. ``GuardedSource._array`` asserts the subscription is
active before every read. ``sampler_context()`` then replaces
``Flume.stream`` with ``OtherConsumer``; any later lookup of that
shared field fails instead of stealing another consumer's stream.
The independent cases cover these admission contracts:
- ``LaggingQuotes`` raises ``Lagged`` once. It sets ``recovered``
only when the quote drain resumes; SHM processing must survive.
- Old/future source clocks must withhold model evidence even
though submission occurs locally now; no age limit permits
offline replay. ``published`` distinguishes those outcomes.
- ``feed_context()`` rejects a requested quote throttle: feeds
cached solely by market must retain their shared quote policy.
``finish()`` waits on ``read_source`` and, when needed,
``recovered``, then closes sampling after tasks block. Context
exit order proves the native wrapper releases its subscription
before the feed, without any broker, actor service or API key.
'''
order: list[str] = []
subscribed: bool = False
published: list[RegimeSnapshot] = []
read_source: trio.Event
flume_ref: SimpleNamespace|None = None
class GuardedSource:
'''
Require the native sampler subscription before source access.
'''
def __init__(self) -> None:
'''
Seed a readable range whose array access is guarded.
'''
self._first: SimpleNamespace = SimpleNamespace(value=0)
self._last: SimpleNamespace = SimpleNamespace(value=6)
self.storage: np.ndarray = FakeSource()._array
@property
def _array(self) -> np.ndarray:
'''
Detect a source read before entering the subscription.
'''
assert subscribed
order.append('read_source')
read_source.set()
return self.storage
async def scenario() -> None:
'''
Close the synthetic sampler to end the actual wrapper.
'''
nonlocal read_source
read_source = trio.Event()
recovered: trio.Event = trio.Event()
send: trio.MemorySendChannel
receive: trio.MemoryReceiveChannel
send, receive = trio.open_memory_channel(0)
quote_send: trio.MemorySendChannel
quote_receive: trio.MemoryReceiveChannel
quote_send, quote_receive = trio.open_memory_channel(0)
class LaggingQuotes:
'''
Lose one discarded quote before resuming a healthy wait.
'''
def __aiter__(self) -> 'LaggingQuotes':
'''
Return the resumable stream after a Lagged exception.
'''
return self
async def __anext__(self) -> dict:
'''
Force one overrun, then prove the drain resumed.
'''
if 'quote_lagged' not in order:
order.append('quote_lagged')
raise Lagged('synthetic discarded quote')
recovered.set()
await trio.sleep_forever()
@asynccontextmanager
async def feed_context(
**kwargs: object,
) -> AsyncIterator[SimpleNamespace]:
'''
Expose a source Flume with a cancellable quote stream.
'''
nonlocal flume_ref
assert 'tick_throttle' not in kwargs
order.append('open_feed')
flume_ref = SimpleNamespace(
rt_sample_period_s=1,
rt_shm=GuardedSource(),
stream=(
LaggingQuotes() if quote_lag else quote_receive
),
)
yield SimpleNamespace(flumes={
'test.market': flume_ref,
})
await quote_send.aclose()
order.append('close_feed')
@asynccontextmanager
async def sampler_context(
**kwargs: object,
) -> AsyncIterator[trio.MemoryReceiveChannel]:
'''
Mark exactly when the sampler subscription begins.
'''
nonlocal subscribed
subscribed = True
order.append('subscribe')
class OtherConsumer:
'''
Model a shared Flume's receiver ownership change.
'''
def __aiter__(self) -> Never:
'''
Detect consuming a later caller's broadcast view.
'''
pytest.fail('drain stole another quote receiver')
assert flume_ref is not None
flume_ref.stream = OtherConsumer()
try:
yield receive
finally:
subscribed = False
order.append('unsubscribe')
monkeypatch.setattr(
'piker.data.feed.maybe_open_feed', feed_context,
)
monkeypatch.setattr(
'piker.data._sampling.open_sample_stream',
sampler_context,
)
if source_age is not None:
monkeypatch.setattr(
'piker.fsp._regime_feed.time.time',
lambda: 5 + source_age,
)
async def finish() -> None:
'''
Close the sampler after processing and draining quotes.
'''
await read_source.wait()
if quote_lag:
await recovered.wait()
await wait_all_tasks_blocked()
await send.aclose()
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(finish)
await run_regime_feed(
'test.market', published.append,
max_source_age=None if source_age is None else 5,
)
trio.run(scenario)
assert len(published) == (source_age in (None, 0))
assert order.index('subscribe') < order.index('read_source')
assert order[-2:] == ['unsubscribe', 'close_feed']
def test_service_composes_feed_and_classifier_in_one_lifetime(
monkeypatch: pytest.MonkeyPatch,
) -> None:
'''
A prepared provider adapter needs concrete feed-to-classifier
wiring and shared cancellation ownership. Replace the broker
source with a synthetic callback producer but retain the actual
HTTP client, worker and service nursery. One corrected snapshot
must produce one validated judgment; normal source exit must
stop the classifier while leaving the caller's HTTP session open.
'''
from piker.fsp._jev import (
DEFAULT_MODEL,
JevClassification,
JevClient,
RegimeClassifier,
)
published: list[JevClassification] = []
async def scenario() -> None:
'''
End the source only after its judgment is published.
'''
delivered: trio.Event = trio.Event()
async def source(
fqme: str,
submit: Callable[[RegimeSnapshot], object],
*,
invalidate: Callable[[int], object],
**kwargs: object,
) -> None:
'''
Replay one revision through the bound service callbacks.
'''
assert fqme == 'test.market'
assert kwargs['max_source_age'] == 5
invalidate(1)
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
fqme, periods=(1,), revision=1,
)
second: int
for second in range(8):
snapshot: RegimeSnapshot = tracker.push(
second, second,
)
submit(snapshot)
await delivered.wait()
def handler(request: httpx.Request) -> httpx.Response:
'''
Return a documented Choice response without real HTTP.
'''
request_body: dict = json.loads(request.content)
answers: dict[str, dict] = {}
name: str
question: dict
for name, question in request_body['questions'].items():
probabilities: dict[str, float] = {}
label: str
for label in question['criteria']:
probabilities[label] = float(label == 'uptrend')
answers[name] = {
'type': 'choice',
'choice': 'uptrend',
'probabilities': probabilities,
'confidence': 1,
}
return httpx.Response(200, json={
'model': DEFAULT_MODEL,
'answers': answers,
'usage': {
'input_tokens': 100,
'output_tokens': 20,
},
})
async def publish(result: JevClassification) -> None:
'''
Release source shutdown after receiving a typed judgment.
'''
published.append(result)
delivered.set()
monkeypatch.setattr(
'piker.fsp._regime_feed.run_regime_feed', source,
)
http: httpx.AsyncClient
async with httpx.AsyncClient(
transport=httpx.MockTransport(handler),
) as http:
client: JevClient = JevClient(http, 'offline-test-key')
worker: RegimeClassifier = await serve_regimes(
'test.market', client, publish,
)
assert not http.is_closed
assert worker.requests == 1
assert worker.published == 1
assert http.is_closed
trio.run(scenario)
assert len(published) == 1
assert published[0].market == 'test.market'
assert published[0].revision == 1

795
tests/test_trend.py 100644
View File

@ -0,0 +1,795 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Causality, streaming, and publication contracts for Trend Calculus.
'''
from collections.abc import AsyncIterator, Callable
from dataclasses import Field, fields
from types import SimpleNamespace
from typing import Any, cast
import numpy as np
import pytest
import trio
from trio.testing import wait_all_tasks_blocked
from trio_typing import TaskStatus
from piker.data.ticktools import FeedQuote
from piker.fsp._api import (
FspSample,
FspStream,
FspYield,
_load_builtins,
)
from piker.fsp._engine import (
Cascade,
QuoteFrame,
_next_step_output,
_sample_wakeups,
_write_sample,
filter_quotes_by_sym,
)
from piker.fsp._trend import (
TrendCalculator,
TrendHistory,
TrendUpdate,
)
from piker.fsp._trend_fsp import trend_calculus
from tractor.ipc._shm import ShmArray
from tractor.trionics import Lagged
def test_split_preserves_both_reversals() -> None:
'''
A net unchanged direction can contain two opposing reversals.
The final outside window [2, 5] follows the rising window [4, 3].
Bridging its earlier extremum reveals a top at 4 and a bottom at
2, both detected on observation 5. A one-pulse or net-sign-only
implementation loses the top. Assert both pivot coordinates and
the later detection coordinate, including the historical fields.
'''
values: np.ndarray = np.array([1., 2., 4., 3., 2., 5.])
calc: TrendCalculator = TrendCalculator()
calc.extend(values[:-1], np.arange(5.))
update: TrendUpdate = calc.push(5., values[-1])
assert update.trends == (1, 0, 0, 0)
assert update.reversals['index'].tolist() == [2, 4]
assert update.reversals['direction'].tolist() == [-1, 1]
assert update.reversals['price'].tolist() == [4., 2.]
assert update.reversals['detected_index'].tolist() == [5, 5]
assert update.reversals['detected_time'].tolist() == [5., 5.]
history: TrendHistory = TrendCalculator().extend(
values, np.arange(6.),
)
np.testing.assert_array_equal(
history.pivot_prices[-1, 0], [4., 2.],
)
np.testing.assert_array_equal(
history.pivot_times[-1, 0], [2., 4.],
)
assert np.isnan(history.pivot_prices[:-1]).all()
def test_batch_chunking_and_every_prefix() -> None:
'''
Future observations and transport chunking must not alter output.
Materialize a random walk with ties and a missing observation,
then compare every prefix and uneven continuation batches with a
single history scan. Compare pivot detection fields too, because
equal direction arrays alone can hide backdated reversals.
'''
rng: np.random.Generator = np.random.default_rng(18)
values: np.ndarray = (
rng.integers(-2, 3, 80).cumsum().astype(float)
)
values[31] = np.nan
times: np.ndarray = np.arange(80.) * 0.125 + 1000.
indices: np.ndarray = np.arange(80, dtype=np.int64) + 12345
windows: tuple[int, ...] = (3, 2, 4)
full: TrendHistory = TrendCalculator(windows).extend(
values, times, indices,
)
cut: int
field: Field[Any]
for cut in range(81):
prefix: TrendHistory = TrendCalculator(windows).extend(
values[:cut], times[:cut], indices[:cut],
)
for field in fields(TrendHistory):
np.testing.assert_array_equal(
getattr(prefix, field.name),
getattr(full, field.name)[:cut],
)
calc: TrendCalculator = TrendCalculator(windows)
start: int = 0
for cut in (0, 1, 2, 7, 31, 32, 34, 60, 80):
chunk: TrendHistory = calc.extend(
values[start:cut], times[start:cut], indices[start:cut],
)
for field in fields(TrendHistory):
np.testing.assert_array_equal(
getattr(chunk, field.name),
getattr(full, field.name)[start:cut],
)
start = cut
@pytest.mark.parametrize('window', [2, 3, 7])
@pytest.mark.parametrize('direction', [-1, 0, 1])
def test_warmup_flat_and_negative_values(
window: int,
direction: int,
) -> None:
'''
Startup must not invent a trend from a synthetic zero-price bar.
Use negative, monotone or flat values through two full windows.
The first comparable pair establishes readiness and direction;
no sign change or initial pivot exists before that comparison.
'''
values: np.ndarray = (
np.arange(2 * window, dtype=float) * direction - 100
)
result: TrendHistory = TrendCalculator((window,)).extend(
values, np.arange(float(len(values))),
)
assert not result.ready[:-1].any()
assert result.ready[-1, 0]
assert not result.trends[:-1].any()
assert result.trends[-1, 0] == direction
assert np.isnan(result.pivot_prices).all()
def test_nan_resets_hierarchy_without_future_contamination() -> None:
'''
A missing value invalidates every recursive reversal order.
Build nontrivial history, append NaN, and compare the following
observations against a newly constructed calculator. Retaining
even a higher-order partial window would make this replay differ.
'''
rng: np.random.Generator = np.random.default_rng(77)
calc: TrendCalculator = TrendCalculator((2, 2, 2))
calc.extend(rng.normal(size=200), np.arange(200.))
update: TrendUpdate = calc.push(200., np.nan)
assert update.trends == (0, 0, 0)
assert update.ready == (False, False, False)
assert not len(update.reversals)
values: np.ndarray = rng.normal(size=100)
times: np.ndarray = np.arange(201., 301.)
actual: TrendHistory = calc.extend(values, times)
expected: TrendHistory = TrendCalculator((2, 2, 2)).extend(
values, times,
)
field: Field[Any]
for field in fields(TrendHistory):
np.testing.assert_array_equal(
getattr(actual, field.name),
getattr(expected, field.name),
)
@pytest.mark.parametrize(
'time,value,index',
[
(3., 1., 14),
(2., 1., 14),
(4., np.inf, 14),
(np.nan, 1., 14),
(4., 1., 13),
(4., 1., -1),
(4., 1., 2 ** 63),
(4., 1., np.uint64(2 ** 63)),
(4., 1., True),
(4., 1., np.bool_(True)),
],
)
def test_invalid_sample_is_atomic(
time: float,
value: float,
index: int,
) -> None:
'''
Invalid coordinates must not corrupt a partially filled window.
Seed a calculator with four observations, attempt one rejected
sample, and compare continuation to an untouched state copy.
This checks that validation precedes recurrence mutation.
'''
calc: TrendCalculator = TrendCalculator()
calc.extend(np.arange(4.), np.arange(4.), np.arange(10, 14))
untouched: TrendCalculator = calc.copy()
with pytest.raises(ValueError):
calc.push(time, value, index=index)
actual: TrendUpdate = calc.push(4., -1., index=14)
expected: TrendUpdate = untouched.push(4., -1., index=14)
assert actual.trends == expected.trends
np.testing.assert_array_equal(
actual.reversals, expected.reversals,
)
def test_invalid_batch_is_atomic() -> None:
'''
A late malformed sample cannot partly advance a batch operation.
The duplicate final timestamp is detected before the first valid
row is consumed. A subsequent valid replay matches fresh state.
'''
calc: TrendCalculator = TrendCalculator()
with pytest.raises(ValueError):
calc.extend(np.arange(4.), np.array([1., 2., 3., 3.]))
actual: TrendHistory = calc.extend(np.arange(4.), np.arange(4.))
expected: TrendHistory = TrendCalculator().extend(
np.arange(4.), np.arange(4.),
)
np.testing.assert_array_equal(actual.trends, expected.trends)
def test_last_int64_index_is_exact_and_overflow_is_atomic() -> None:
'''
Explicit unsigned coordinates must not wrap in compiled state.
Observe the last representable coordinate using a NumPy uint64.
A subsequent automatic push or batch must reject overflow before
touching the partial window, and retain the exact last index.
'''
calc: TrendCalculator = TrendCalculator()
limit: int = np.iinfo(np.int64).max
result: TrendUpdate = calc.push(
1., 2., index=np.uint64(limit),
)
assert result.index == limit
assert isinstance(result.index, int)
before: np.ndarray = calc._bars.copy()
with pytest.raises(ValueError, match='int64'):
calc.push(2., 3.)
with pytest.raises(ValueError, match='overflow'):
calc.extend(np.array([3.]), np.array([2.]))
np.testing.assert_array_equal(calc._bars, before)
class MemorySource:
'''
Expose mutable source rows with absolute, nonzero SHM bounds.
'''
def __init__(self, values: list[float]) -> None:
'''
Seed mutable rows with a nonzero absolute SHM origin.
'''
self._first: SimpleNamespace = SimpleNamespace(value=37)
self._last: SimpleNamespace = SimpleNamespace(
value=37 + len(values),
)
self._array: np.ndarray = np.zeros(128, dtype=[
('index', '<i8'),
('time', '<f8'),
('close', '<f8'),
])
self._array['index'] = np.arange(128)
self._array['time'] = np.arange(128.) + 1000.
self.array['close'] = values
@property
def array(self) -> np.ndarray:
'''
Expose ``MemorySource._array`` between published bounds.
'''
return self._array[self._first.value:self._last.value]
def assert_sample_matches(
sample: FspSample,
rows: np.ndarray,
) -> None:
'''
Compare every FSP field with a fresh source-prefix replay.
'''
history: TrendHistory = TrendCalculator().extend(
rows['close'], rows['time'], rows['index'],
)
assert sample.index == rows['index'][-1]
assert sample.time == rows['time'][-1]
assert sample.fields['ready'] == history.ready[-1].sum()
level: int
for level in range(4):
order: int = level + 1
name: str = 'trend' if level == 0 else f'trend_{order}'
assert sample.fields[name] == history.trends[-1, level]
side: int
label: str
for side, label in enumerate(('top', 'bottom')):
np.testing.assert_equal(
sample.fields[f'{label}_{order}'],
history.pivot_prices[-1, level, side],
)
np.testing.assert_equal(
sample.fields[f'{label}_time_{order}'],
history.pivot_times[-1, level, side],
)
def test_fsp_preview_mutations_and_skipped_steps() -> None:
'''
Updating an open bar must replace its preview, not consume it.
Repeatedly mutate the current SHM close across both directions,
then advance by two rows before the next quote. Each FspSample
must equal a fresh batch replay, and its absolute row must match
the source. Quote payloads are deliberately empty wakeups so the
source state rather than trade arrivals must drive sampling.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
shm: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.])
async def quotes() -> AsyncIterator[FeedQuote]:
'''
Provide payload-free wakeups for source
snapshot sampling.
'''
while True:
yield {}
stream: FspStream = trend_calculus(
quotes(), cast(ShmArray, shm),
)
try:
historical: FspYield = await anext(stream)
assert isinstance(historical, dict)
assert set(historical) == set(trend_calculus.outputs)
value: np.ndarray|None # noqa: F842
assert all(
value is not None and len(value) == 6
for value in historical.values()
)
close: float
for close in (100., -20., 5., 5.):
shm.array['close'][-1] = close
sample: FspYield = await anext(stream)
assert isinstance(sample, FspSample)
assert_sample_matches(sample, shm.array)
last: int = shm._last.value
shm._array['close'][last:last + 2] = [-4., -5.]
shm._last.value += 2
end: int
for end in (6, 7, 8):
sample: FspYield = await anext(stream)
assert isinstance(sample, FspSample)
assert_sample_matches(sample, shm.array[:end])
finally:
await stream.aclose()
trio.run(main)
def test_fsp_empty_bootstrap_and_source_revision() -> None:
'''
Empty startup must accept later source rows without old state.
After the first sample, prepend changes the source origin. The
old generator must stop; restarting it recomputes the revised
history rather than reusing recurrence state from another origin.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
shm: MemorySource = MemorySource([])
async def quotes() -> AsyncIterator[FeedQuote]:
'''
Provide payload-free wakeups for source
snapshot sampling.
'''
while True:
yield {}
stream: FspStream = trend_calculus(
quotes(), cast(ShmArray, shm),
)
historical: FspYield = await anext(stream)
assert isinstance(historical, dict)
value: np.ndarray|None # noqa: F842
assert all(
value is not None and not len(value)
for value in historical.values()
)
shm._array['close'][37] = 1.
shm._last.value += 1
sample: FspYield = await anext(stream)
assert isinstance(sample, FspSample)
assert_sample_matches(sample, shm.array)
shm._first.value -= 1
with pytest.raises(StopAsyncIteration):
await anext(stream)
restarted: FspStream = trend_calculus(
quotes(), cast(ShmArray, shm),
)
try:
await anext(restarted)
sample: FspYield = await anext(restarted)
assert isinstance(sample, FspSample)
assert_sample_matches(sample, shm.array)
finally:
await restarted.aclose()
trio.run(main)
def test_indexed_sample_waits_for_destination_step() -> None:
'''
A quote can reach the FSP before the sampler appends destination.
Block _write_sample with destination one row behind source and
wait until its task is suspended. Publishing the next destination
bound and setting its event permits one coherent write. The prior
row remains unchanged, proving no accidental previous-bar write.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
array: np.ndarray = np.zeros(16, dtype=[
('time', '<f8'),
('trend', '<f8'),
('ready', '<f8'),
])
dst: SimpleNamespace = SimpleNamespace(
_first=SimpleNamespace(value=7),
_last=SimpleNamespace(value=10),
_array=array,
)
casc: Cascade = cast(Cascade, SimpleNamespace(
dst=SimpleNamespace(rt_shm=dst),
sample_ready=None,
))
complete: trio.Event = trio.Event()
async def publish() -> None:
'''
Wait for the destination bound, then signal publication.
'''
assert await _write_sample(casc, FspSample(
10, 1010., {'trend': -1., 'ready': 2.},
))
complete.set()
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(publish)
await wait_all_tasks_blocked()
assert not complete.is_set()
assert not array['trend'].any()
dst._last.value += 1
assert casc.sample_ready is not None
casc.sample_ready.set()
await complete.wait()
assert tuple(array[9]) == (0., 0., 0.)
assert tuple(array[10]) == (1010., -1., 2.)
trio.run(main)
def test_indexed_sample_wait_is_cancellable() -> None:
'''
History resync must cancel a compute task waiting on sampling.
Suspend indexed publication against an empty destination, cancel
its nursery, then advance that destination. No cancelled sample
may publish into the new source revision after cancellation.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
array: np.ndarray = np.zeros(4, dtype=[
('time', float), ('trend', float),
])
dst: SimpleNamespace = SimpleNamespace(
_first=SimpleNamespace(value=0),
_last=SimpleNamespace(value=0),
_array=array,
)
casc: Cascade = cast(Cascade, SimpleNamespace(
dst=SimpleNamespace(rt_shm=dst),
sample_ready=None,
))
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(
_write_sample, casc, FspSample(0, 1., {'trend': 1.}),
)
await wait_all_tasks_blocked()
nursery.cancel_scope.cancel()
dst._last.value = 1
assert casc.sample_ready is not None
casc.sample_ready.set()
assert not array['trend'].any()
trio.run(main)
def test_builtin_registration() -> None:
'''
Actor-side builtin discovery must resolve the new FSP target.
'''
assert _load_builtins()[trend_calculus.ns_path] is trend_calculus
def test_quote_free_sampler_steps_finalize_without_repeating_pivots(
) -> None:
'''
A quiet feed still finalizes bars and clears old reversal pulses.
Start with an open bar containing two provisional reversals and
advance the source on sampler notifications alone. Mirror the
destination append protocol and run the actual FSP generator and
indexed writer. Every stored field must equal causal replay;
event fields must also be clear before compute catches up.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
source: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.])
first: int = source._first.value
last: int = source._last.value
name: str
dtype: list[tuple[str, str]] = [
('index', '<i8'),
('time', '<f8'),
] + [
(name, '<f8') for name in trend_calculus.outputs
]
array: np.ndarray = np.zeros(128, dtype=dtype)
array['index'] = np.arange(128)
destination: SimpleNamespace = SimpleNamespace(
_first=SimpleNamespace(value=first),
_last=SimpleNamespace(value=last),
_array=array,
)
casc: Cascade = cast(Cascade, SimpleNamespace(
dst=SimpleNamespace(rt_shm=destination),
sample_ready=None,
))
send: trio.MemorySendChannel[QuoteFrame]
quotes: trio.MemoryReceiveChannel[QuoteFrame]
send, quotes = trio.open_memory_channel[QuoteFrame](0)
wakeups: AsyncIterator[QuoteFrame]
notify: Callable[[], None]|None
async with send, quotes, _sample_wakeups(
quotes, 'test.market', True,
) as (wakeups, notify):
stream: FspStream = trend_calculus(
filter_quotes_by_sym('test.market', wakeups),
cast(ShmArray, source),
)
history: FspYield = await anext(stream)
assert isinstance(history, dict)
name: str
values: np.ndarray
for name, values in history.items():
assert values is not None
array[name][first:last] = values
array['time'][first:last] = source.array['time']
async def compute() -> None:
'''
Run until cancellation and record completion on exit.
'''
try:
sample: FspYield
async for sample in stream:
assert isinstance(sample, FspSample)
await _write_sample(casc, sample)
finally:
await stream.aclose()
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
nursery.start_soon(compute)
await wait_all_tasks_blocked()
assert array['top_1'][last - 1] == 4.
assert array['bottom_1'][last - 1] == 2.
close: float
for close in (5., 5., -4., -5.):
last = source._last.value
source._array['close'][last] = close
source._last.value += 1
array[last:last + 1] = _next_step_output(
array[last - 1:last], trend_calculus, False,
)
array['index'][last] = last
array['time'][last] = source.array['time'][-1]
destination._last.value += 1
assert np.isnan(array['top_1'][last])
assert np.isnan(array['bottom_1'][last])
if casc.sample_ready is not None:
casc.sample_ready.set()
casc.sample_ready = trio.Event()
assert notify is not None
# Coalesce a burst without losing finalized rows.
notify()
notify()
await wait_all_tasks_blocked()
offset: int
row: np.void
for offset, row in enumerate(source.array):
name: str
sample: FspSample = FspSample(
int(row['index']), float(row['time']),
{
name: array[name][int(row['index'])]
for name in trend_calculus.outputs
},
)
assert_sample_matches(
sample, source.array[:offset + 1],
)
nursery.cancel_scope.cancel()
trio.run(main)
def test_snapshot_quote_overrun_resumes_source_wakeups() -> None:
'''
Losing irrelevant quote payloads must not kill a SHM processor.
A broadcast receiver raises Lagged and then remains usable.
Simulate that protocol, require a catchup notification, and
deliver another quote afterward to prove the relay resumed.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
send: trio.MemorySendChannel[QuoteFrame|Exception]
receive: trio.MemoryReceiveChannel[QuoteFrame|Exception]
send, receive = trio.open_memory_channel[
QuoteFrame|Exception
](1)
class Quotes:
'''
Inject one overrun without closing the quote receiver.
'''
def __aiter__(self) -> 'Quotes':
'''
Retain this controlled quote receiver
across overruns.
'''
return self
async def __anext__(self) -> QuoteFrame:
'''
Deliver the scheduled frame or
injected receiver error.
'''
value: QuoteFrame|Exception = await receive.receive()
if isinstance(value, Exception):
raise value
return value
wakeups: AsyncIterator[QuoteFrame]
notify: Callable[[], None]|None
async with send, receive, _sample_wakeups(
Quotes(), 'test.market', True,
) as (wakeups, notify):
await send.send(Lagged('synthetic quote overrun'))
assert 'test.market' in await anext(wakeups)
await send.send({'test.market': {'ticks': []}})
assert 'test.market' in await anext(wakeups)
assert notify is not None
trio.run(main)
def test_edge_exit_stops_compute_before_source_receiver_closes(
) -> None:
'''
Normal sampler EOF must not strand or abruptly close compute.
Keep the parent nursery alive beyond open_edge and inspect the
compute task's finalizer immediately after context exit. This
proves cleanup belongs to the edge scope rather than relying on
an outer actor cancellation or a closed receiver exception.
'''
async def main() -> None:
'''
Run the controlled scenario in an isolated Trio lifetime.
'''
cleaned: trio.Event = trio.Event()
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
casc: Cascade = cast(Cascade, SimpleNamespace(
tn=nursery,
cs=None,
complete=None,
bind_func=None,
))
async def compute(
*,
task_status: TaskStatus[int] = (
trio.TASK_STATUS_IGNORED
),
) -> None:
'''
Run until cancellation and record completion on exit.
'''
scope: trio.CancelScope
with trio.CancelScope() as scope:
casc.cs = scope
casc.complete = trio.Event()
task_status.started(42)
try:
await trio.sleep_forever()
finally:
cleaned.set()
casc.complete.set()
index: int
async with Cascade.open_edge(casc, compute) as index:
assert index == 42
assert not cleaned.is_set()
assert cleaned.is_set()
trio.run(main)

929
tests/test_trend_graphics.py 100644
View File

@ -0,0 +1,929 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Trend overlay causality, coordinate and bounded Qt lifetime checks.
'''
from collections.abc import AsyncIterator, Iterator
from contextlib import asynccontextmanager
import os
from types import SimpleNamespace
from typing import Literal
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
import numpy as np
import pyqtgraph as pg
import pytest
import trio
from trio.testing import wait_all_tasks_blocked
from piker.ui._trend import (
RegimeInterval,
ReversalPoint,
TrendOverlay,
classification_interval,
regime_rectangles,
reversal_points,
times_to_x,
trend_intervals,
)
from piker.ui.qt import QApplication, QtGui, QPointF
from piker.fsp._jev import JevClassification, RegimeAnswer
from piker.fsp import Fsp
from piker.ui._fsp import FspAdmin, update_trend_overlays
from piker.ui import _fsp as fsp_ui
def _chart(
times: tuple[float, ...],
first: int = 1000,
) -> np.ndarray:
'''
Make chart bars with absolute indices independent of timestamp.
'''
rows: np.ndarray = np.zeros(len(times), dtype=[
('index', 'i8'),
('time', 'f8'),
('low', 'f8'),
('high', 'f8'),
('close', 'f8'),
])
rows['index'] = np.arange(first, first + len(times))
rows['time'] = times
rows['low'] = 99
rows['high'] = 101
rows['close'] = 100
return rows
def _source(
times: tuple[float, ...],
trends: tuple[float, ...],
) -> np.ndarray:
'''
Make sparse FSP fields, including a deliberately mutable tail.
'''
fields: list[tuple[str, str]] = [
('index', 'i8'),
('time', 'f8'),
('ready', 'f8'),
]
order: int
for order in range(1, 5):
field: str = 'trend' if order == 1 else f'trend_{order}'
fields.append((field, 'f8'))
kind: str
for kind in ('top', 'bottom'):
fields.extend([
(f'{kind}_{order}', 'f8'),
(f'{kind}_time_{order}', 'f8'),
])
rows: np.ndarray = np.empty(len(times), dtype=fields)
name: str
for name in rows.dtype.names:
if name == 'index':
rows[name] = np.arange(5000, 5000 + len(rows))
else:
rows[name] = np.nan
rows['time'] = times
rows['ready'] = 4
rows['trend'] = trends
return rows
@pytest.fixture(scope='session')
def qapp() -> Iterator[QApplication]:
'''
Keep one offscreen application alive without launching a GUI.
'''
app: QApplication|None = QApplication.instance()
if app is None:
app = QApplication(['piker-trend-tests'])
app.setQuitOnLastWindowClosed(False)
yield app
app.processEvents()
@pytest.mark.parametrize('index_field', ['index', 'time'])
def test_timestamp_mapping_preserves_gaps(
index_field: Literal['index', 'time'],
) -> None:
'''
A timestamp inside a missing bucket must not snap to a future
bar or confuse slice offsets with absolute SHM coordinates.
Irregular 10-second bars and off-grid queries exercise both
chart axes; NaNs prove out-of-range observations stay hidden.
'''
rows: np.ndarray = _chart((100, 110, 140))
queries: np.ndarray = np.array((99, 100, 115, 125, 145, 150))
actual: np.ndarray = times_to_x(rows, queries, 10, index_field)
expected: list[float] = (
[np.nan, 1000, 1001.5, np.nan, 1002.5, np.nan]
if index_field == 'index'
else [np.nan, 100, 115, np.nan, 145, np.nan]
)
np.testing.assert_allclose(actual, expected, equal_nan=True)
def test_pivots_confirm_at_later_bar_close() -> None:
'''
Plotting a provisional reversal makes an intrabar repaint look
like a confirmed signal. A confirmed top and mutable-tail bottom
share an earlier pivot time here. Only the top may render, with
its 30-second source detection time and 10-second chart position
retained separately; ``source['index']`` is unrelated on purpose.
'''
source: np.ndarray = _source((100, 130, 160, 190), (0, 1, -1, 1))
source['top_1'][2] = 110
source['top_time_1'][2] = 130
source['bottom_1'][3] = 90
source['bottom_time_1'][3] = 160
chart: np.ndarray = _chart(tuple(range(100, 201, 10)))
points: list[ReversalPoint] = reversal_points(
source,
chart,
source_period_s=30,
chart_period_s=10,
)
assert len(points) == 1
assert points[0].x == 1003
assert points[0].pivot_time == 130
assert points[0].detected_at == 190
assert points[0].period_s == 30
assert points[0].kind == 'top'
def test_reversal_order_is_independent_of_timeframe() -> None:
'''
A second reversal order must use its own pivot fields without
implicitly changing the source sampling period. Distinct order
one and order two prices at the same source time identify which
columns were consumed and prevent order/timeframe conflation.
'''
source: np.ndarray = _source((10, 20, 30), (1, -1, 1))
source['top_1'][1] = 105
source['top_time_1'][1] = 10
source['bottom_2'][1] = 95
source['bottom_time_2'][1] = 10
points: list[ReversalPoint] = reversal_points(
source,
_chart((10, 20, 30)),
source_period_s=10,
chart_period_s=10,
order=2,
)
assert len(points) == 1
assert points[0].price == 95
assert points[0].order == 2
assert points[0].period_s == 10
def test_regime_runs_start_at_confirmation() -> None:
'''
Coloring from a pivot or detection-bar open falsely implies an
earlier available regime. A warm-up row, direction change, and
source gap arrange all run boundaries. Every interval starts at
finalized bar close and the mutable tail's opposite direction
cannot create a new regime.
'''
source: np.ndarray = _source(
(100, 110, 120, 130, 170, 180),
(1, 1, 1, -1, -1, 1),
)
source['ready'][0] = 0
spans: list[RegimeInterval] = trend_intervals(source, 10)
span: RegimeInterval # noqa: F842
assert [(span.start_time, span.end_time) for span in spans] == [
(120, 140),
(140, 150),
(180, 190),
]
assert [span.label for span in spans] == [
'uptrend', 'downtrend', 'downtrend',
]
def test_decimal_subsecond_epochs_do_not_invent_gaps() -> None:
'''
Float64 Unix timestamps cannot represent every 100 ms boundary
exactly; successive differences alternate around 0.1 seconds.
Comparing each difference strictly to the requested period splits
one regime into spurious holes. Realistic epochs prove rounding
preserves one run, while deleting one actual bucket still splits
both regime evidence and chart rectangle geometry.
'''
times: np.ndarray = 1_700_000_000 + np.arange(8) * 0.1
source: np.ndarray = _source(tuple(times), (1,) * 8)
spans: list[RegimeInterval] = trend_intervals(source, 0.1)
assert len(spans) == 1
rectangles: np.ndarray
_: list[RegimeInterval]
rectangles, _ = regime_rectangles(
spans, _chart(tuple(times)), 0.1,
)
assert len(rectangles) == 1
source['time'][4:] += 0.1
spans = trend_intervals(source, 0.1)
assert len(spans) == 2
whole: RegimeInterval = RegimeInterval(
float(times[0]), float(times[-1] + 0.3), 'uptrend',
)
rectangles, _ = regime_rectangles(
[whole], _chart(tuple(source['time'])), 0.1,
)
assert len(rectangles) == 2
@pytest.mark.parametrize('index_field', ['index', 'time'])
def test_delayed_regime_and_chart_gaps(
index_field: Literal['index', 'time'],
) -> None:
'''
A classifier may report an earlier onset after the chart has
advanced. Its rectangle must begin only at response arrival and
must not fill an absent chart bucket. Fractional arrival within
a bar, plus a 20-second hole, proves causal clipping and split
geometry on both time and absolute-index chart axes.
'''
chart: np.ndarray = _chart((100, 110, 140, 150))
span: RegimeInterval = RegimeInterval(
start_time=100,
end_time=155,
label='ranging',
probability=0.8,
source='jev',
observed_at=115,
)
rectangles: np.ndarray
metadata: list[RegimeInterval]
rectangles, metadata = regime_rectangles(
[span], chart, 10, index_field=index_field,
)
expected: list[list[float]] = (
[[1001.5, 99, 0.5, 2], [1002, 99, 1.5, 2]]
if index_field == 'index'
else [[115, 99, 5, 2], [140, 99, 15, 2]]
)
np.testing.assert_allclose(rectangles, expected)
assert metadata == [span, span]
assert metadata[0].probability == 0.8
assert metadata[0].source == 'jev'
def test_empty_and_unavailable_geometry() -> None:
'''
A layer can attach before history arrives, and late classifier
responses can lie beyond retained history. Empty chart arrays
and already-expired intervals must produce empty geometry rather
than indexing the final row or drawing a retrospective rectangle.
'''
span: RegimeInterval = RegimeInterval(10, 20, 'ranging')
empty: np.ndarray
metadata: list[RegimeInterval]
empty, metadata = regime_rectangles([span], _chart(()), 1)
assert empty.shape == (0, 4)
assert metadata == []
span = RegimeInterval(10, 20, 'ranging', observed_at=21)
empty, metadata = regime_rectangles([span], _chart((10, 20)), 1)
assert empty.shape == (0, 4)
assert metadata == []
@pytest.mark.parametrize('period', [0, -1, np.nan, np.inf])
def test_invalid_period_rejected(period: float) -> None:
'''
Invalid sampling periods break binary-search interval semantics
and can create infinite widths. All non-positive/non-finite
period classes are rejected before constructing chart geometry.
'''
with pytest.raises(ValueError, match='period'):
times_to_x(_chart((10, 20)), np.array([10]), period)
@pytest.mark.parametrize(
'market,revision,period,end_time,accepted',
[
('test.market', 3, 5, 130, True),
('other.market', 3, 5, 130, False),
('test.market', 4, 5, 130, False),
('test.market', 3, 15, 130, False),
('test.market', 3, 5, 119, False),
],
ids=['matching', 'market', 'revision', 'period', 'future'],
)
def test_jev_adapter_retains_causal_evidence(
market: str,
revision: int,
period: float,
end_time: float,
accepted: bool,
) -> None:
'''
Late or misrouted classifications must not color an unrelated
market, source revision, or timeframe. A typed Jev response
claims source evidence at 100 but arrives at 120; matching
evidence retains its original probability/model and starts at
receipt time. The rejected cases vary one ownership bound each.
'''
result: JevClassification = JevClassification(
market='test.market',
revision=3,
sequence=40,
segment=2,
source_as_of=100,
requested_at=118,
received_at=120,
latency=2,
model='jev-test',
prompt_version='test',
answers=(RegimeAnswer(
question='regime_5',
period=5,
label='range',
probability=0.82,
confidence=0.7,
probabilities=(('range', 0.82), ('unknown', 0.18)),
),),
input_tokens=100,
output_tokens=20,
)
interval: RegimeInterval|None = classification_interval(
result,
fqme=market,
revision=revision,
period_s=period,
end_time=end_time,
)
if not accepted:
assert interval is None
return
assert interval is not None
assert interval.start_time == 100
assert interval.observed_at == 120
assert interval.label == 'range'
assert interval.probability == 0.82
assert interval.source == 'typesafe.ai/jev-test'
def test_overlay_rebuild_and_owned_teardown(
qapp: QApplication,
) -> None:
'''
Equal-bound backfills rewrite earlier pivot fields without
changing the live tail. An ordinary paint gate intentionally
skips that change; a forced revision must replace stale markers
and rectangles, preserve a sibling layer, and release exactly
the owning layer's two items on repeated cancellation cleanup.
Real Qt items prove scene ownership rather than a mocked API.
'''
source: np.ndarray = _source(
(100, 110, 120, 130, 140), (1, 1, -1, -1, 1),
)
source['top_1'][1] = 105
source['top_time_1'][1] = 100
widget: pg.PlotWidget = pg.PlotWidget()
viz: SimpleNamespace = SimpleNamespace(
plot=widget.plotItem,
shm=SimpleNamespace(array=_chart((100, 110, 120, 130, 140))),
index_field='index',
)
first: TrendOverlay = TrendOverlay(
viz,
SimpleNamespace(array=source),
fqme='first.test',
source_period_s=10,
chart_period_s=10,
)
sibling: TrendOverlay = TrendOverlay(
viz,
SimpleNamespace(array=source),
fqme='second.test',
source_period_s=10,
chart_period_s=10,
)
try:
assert first.update()
assert sibling.update()
assert len(first.points) == 1
assert not first.update()
source['top_1'][1] = np.nan
source['bottom_1'][1] = 95
source['bottom_time_1'][1] = 100
assert not first.update()
assert first.update(force=True)
assert first.points[0].kind == 'bottom'
assert sibling.points[0].kind == 'top'
assert len(widget.plotItem.items) == 4
# A synthetic full-scene paint verifies Qt primitive types
# and brush batching without a GUI or screenshot dependency.
image: QtGui.QImage = QtGui.QImage(
160, 100, QtGui.QImage.Format.Format_ARGB32,
)
image.fill(0)
painter: QtGui.QPainter = QtGui.QPainter(image)
try:
widget.scene().render(painter)
finally:
painter.end()
first.remove()
first.remove()
assert len(widget.plotItem.items) == 2
assert sibling.markers in widget.plotItem.items
assert not first.update(force=True)
finally:
first.remove()
sibling.remove()
widget.close()
widget.deleteLater()
qapp.processEvents()
def test_provisional_updates_reuse_confirmed_markers(
qapp: QApplication,
) -> None:
'''
Quotes may repaint a provisional reversal repeatedly during one
bar. Rebuilding historical markers each time wastes UI time and
can present a temporary reversal as confirmed. Mutate only the
FSP tail, then the chart candle envelope: the first update is
skipped and the second updates rectangles while preserving the
same ``TrendOverlay.points`` list. Changing ``chart['index']``
requires remapping those retained points.
'''
source: np.ndarray = _source((10, 20, 30), (1, 1, -1))
source['top_1'][1] = 105
source['top_time_1'][1] = 10
widget: pg.PlotWidget = pg.PlotWidget()
chart: np.ndarray = _chart((10, 20, 30))
viz: SimpleNamespace = SimpleNamespace(
plot=widget.plotItem,
shm=SimpleNamespace(array=chart),
index_field='index',
)
overlay: TrendOverlay = TrendOverlay(
viz,
SimpleNamespace(array=source),
fqme='live.test',
source_period_s=10,
chart_period_s=10,
)
try:
assert overlay.update()
original: list[ReversalPoint] = overlay.points
source['trend'][-1] = 1
source['bottom_1'][-1] = 91
source['bottom_time_1'][-1] = 20
assert not overlay.update()
assert overlay.points is original
chart['high'][-1] = 109
assert overlay.update()
assert overlay.points is original
assert overlay.rectangles.rectangles[-1, 3] == 10
chart['high'][-1] = 100
assert overlay.update()
assert overlay.points is original
assert overlay.rectangles.rectangles[-1, 3] == 2
chart['index'] += 500
assert overlay.update()
assert overlay.points is not original
assert overlay.points[0].x == 1500
finally:
overlay.remove()
widget.close()
widget.deleteLater()
qapp.processEvents()
def test_overlay_limits_and_classifier_metadata(
qapp: QApplication,
) -> None:
'''
Unbounded event layers can allocate thousands of Qt objects and
delayed classifier results can overwrite deterministic meaning.
Many synthetic pivots exercise the configured retention limits;
replacing rectangles with a Jev result must retain probability,
source, and markers while keeping exactly two scene items.
'''
source: np.ndarray = _source(tuple(range(20)), (1, -1) * 10)
source['top_1'] = 105
source['top_time_1'] = source['time']
widget: pg.PlotWidget = pg.PlotWidget()
viz: SimpleNamespace = SimpleNamespace(
plot=widget.plotItem,
shm=SimpleNamespace(array=_chart(tuple(range(20)))),
index_field='index',
)
overlay: TrendOverlay = TrendOverlay(
viz,
SimpleNamespace(array=source),
fqme='bounded.test',
source_period_s=1,
chart_period_s=1,
max_points=3,
max_rectangles=2,
)
try:
overlay.update()
assert len(overlay.points) == 3
assert overlay.points[0].pivot_time == 16
assert len(overlay.rectangles.rectangles) == 2
assert len(widget.plotItem.items) == 2
classified: RegimeInterval = RegimeInterval(
start_time=15,
end_time=20,
label='ranging',
probability=0.73,
source='jev',
observed_at=18,
)
overlay.set_regimes([classified])
assert len(overlay.points) == 3
assert overlay.rectangles.intervals == [classified]
assert overlay.rectangles.rectangles[0, 0] == 1018
tip: str = overlay.rectangles.tooltip_at(QPointF(1019, 100))
assert 'ranging: 73.0%' in tip
assert '(1s)' in tip
assert 'jev' in tip
assert '00:00:18.000' in tip
overlay.set_regimes([])
assert len(overlay.rectangles.rectangles) == 0
overlay.set_regimes(None)
assert len(overlay.rectangles.rectangles) == 2
tip = overlay.rectangles.tooltip_at(QPointF(1019, 100))
assert 'trend-calculus' in tip
assert '%' not in tip
finally:
overlay.remove()
widget.close()
widget.deleteLater()
qapp.processEvents()
class _PriceChart:
'''
Expose the price-chart API while retaining actual Qt ownership.
'''
_trend_overlays: dict[tuple[str, str], list[TrendOverlay]]
def __init__(self) -> None:
'''
Build a chart whose SHM token identifies its base timeframe.
'''
self.widget: pg.PlotWidget = pg.PlotWidget()
self.viz: SimpleNamespace = SimpleNamespace(
plot=self.widget.plotItem,
shm=SimpleNamespace(
array=_chart((10, 20, 30, 40, 50)),
_token='ohlcv',
),
index_field='index',
)
def get_viz(self, fqme: str) -> SimpleNamespace|None:
'''
Resolve exactly the test market rather than any chart key.
'''
return self.viz if fqme == 'admin.test' else None
def close(self) -> None:
'''
Release the disposable offscreen widget after its layers.
'''
self.widget.close()
self.widget.deleteLater()
async def _admin_bootstrap_scenario(
qapp: QApplication,
monkeypatch: pytest.MonkeyPatch,
) -> None:
'''
Attaching before FSP bootstrap exposes empty history, and opening
several reversal orders must not start multiple SHM writers.
The ``attach_first()`` task calls ``FspAdmin.open_trend_overlay``
while the mocked engine's ``started`` event stays unset.
``wait_all_tasks_blocked()`` reaches that wait deterministically;
one engine call and an empty Qt plot rule out early attachment.
Only setting ``started`` lets ``attached`` confirm publication.
Reopening another order retains one writer, and repeated
``FspAdmin.close_trend_overlays()`` preserves a sibling's items.
This exercises local lifecycle ordering, not actor IPC.
'''
chart: _PriceChart = _PriceChart()
source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5)
source['trend_2'] = 1
destination: SimpleNamespace = SimpleNamespace(
rt_shm=SimpleNamespace(array=source, _token='trend'),
)
flume: SimpleNamespace = SimpleNamespace(
mkt=SimpleNamespace(fqme='admin.test'),
rt_shm=chart.viz.shm,
rt_sample_period_s=10,
_hist_shm_token=None,
)
started: trio.Event = trio.Event()
attached: trio.Event = trio.Event()
calls: list[tuple[str, dict, str]] = []
attached_layers: list[TrendOverlay] = []
async def start_engine(
target: Fsp,
conf: dict,
loglevel: str,
) -> tuple[SimpleNamespace, trio.Event]:
'''
Return an allocated destination before publishing bootstrap.
'''
calls.append((target.name, conf, loglevel))
return destination, started
try:
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
admin: FspAdmin = FspAdmin(
nursery, {}, SimpleNamespace(chart=chart), flume,
)
monkeypatch.setattr(
admin, 'start_engine_task', start_engine,
)
with pytest.raises(ValueError, match='positive'):
await admin.open_trend_overlay(chart_period_s=0)
assert calls == []
async def attach_first() -> None:
'''
Signal only after startup has produced visible data.
'''
layer: TrendOverlay = (
await admin.open_trend_overlay()
)
attached_layers.append(layer)
attached.set()
nursery.start_soon(attach_first)
await wait_all_tasks_blocked()
assert len(calls) == 1
assert not attached.is_set()
assert chart.widget.plotItem.items == []
started.set()
await attached.wait()
first: TrendOverlay = attached_layers[0]
second: TrendOverlay = await admin.open_trend_overlay(
order=2,
)
assert await admin.open_trend_overlay() is first
with pytest.raises(ValueError, match='period differs'):
await admin.open_trend_overlay(chart_period_s=60)
assert second is not first
assert len(calls) == 1
assert calls[0][0] == 'trend_calculus'
assert chart._trend_overlays == {
('admin.test', 'trend'): [first, second],
}
assert len(chart.widget.plotItem.items) == 4
sibling: TrendOverlay = TrendOverlay(
chart.viz,
SimpleNamespace(array=source, _token='other'),
fqme='other.test',
source_period_s=10,
chart_period_s=10,
)
foreign_key: tuple[str, str] = ('other.test', 'other')
chart._trend_overlays[foreign_key] = [sibling]
try:
admin.close_trend_overlays()
admin.close_trend_overlays()
assert len(chart.widget.plotItem.items) == 2
assert chart._trend_overlays == {
('other.test', 'other'): [sibling],
}
finally:
sibling.remove()
finally:
chart.close()
qapp.processEvents()
def test_admin_waits_for_bootstrap_and_reuses_writer(
qapp: QApplication,
monkeypatch: pytest.MonkeyPatch,
) -> None:
'''
Exercise bootstrap publication and owner cleanup in local Trio.
``_admin_bootstrap_scenario()`` controls its ``started`` event
and waits for ``attach_first()`` to block in
``FspAdmin.open_trend_overlay()`` before inspecting the plot.
The unset ``attached`` event and empty plot prove attachment
waits for bootstrap rather than depending on timer luck. A
second order shares the writer; cleanup retains a sibling's
``_PriceChart._trend_overlays`` entry and its two Qt items.
'''
trio.run(_admin_bootstrap_scenario, qapp, monkeypatch)
def test_registry_filters_and_defers_hidden_revisions(
qapp: QApplication,
) -> None:
'''
A destination rewrite while hidden must survive an unchanged
source-tail paint signature when the chart becomes visible.
An equal-bound earlier pivot edit arranges that case. Wrong
market/token/timeframe invalidations leave the layer alone;
a matching hidden notification invalidates without touching Qt
data, and the next ordinary refresh consumes the corrected pivot.
'''
chart: _PriceChart = _PriceChart()
source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5)
source['top_1'][1] = 105
source['top_time_1'][1] = 10
overlay: TrendOverlay = TrendOverlay(
chart.viz,
SimpleNamespace(array=source, _token='trend'),
fqme='admin.test',
source_period_s=10,
chart_period_s=10,
)
chart._trend_overlays = {('admin.test', 'trend'): [overlay]}
try:
assert update_trend_overlays(chart) == 1
original: list[ReversalPoint] = overlay.points
source['top_1'][1] = 109
assert update_trend_overlays(
chart, fqme='other.test', force=True,
) == 0
assert update_trend_overlays(
chart, shm_token='other', force=True,
) == 0
assert update_trend_overlays(
chart, chart_period_s=60, force=True,
) == 0
assert update_trend_overlays(chart) == 0
assert overlay.points is original
admin: FspAdmin = FspAdmin(
None, {}, SimpleNamespace(chart=chart), None,
)
admin._trend_overlays.append((chart, overlay))
admin.refresh_trend_overlays('trend', render=False)
assert overlay.points is original
assert overlay.points[0].price == 105
assert update_trend_overlays(chart) == 1
assert overlay.points[0].price == 109
finally:
overlay.remove()
chart.close()
qapp.processEvents()
def test_admin_context_exit_releases_layers(
qapp: QApplication,
monkeypatch: pytest.MonkeyPatch,
) -> None:
'''
A finished FSP-admin context must not leave its graphics attached
to a reused chart widget. Replace only actor-cluster startup and
engine allocation; ``open_fsp_admin()`` and its Trio nursery run
normal exit with actual Qt items. Empty
``FspAdmin._trend_overlays`` and ``_PriceChart._trend_overlays``
plus the signalled ``complete`` event from
``FspAdmin._registry`` prove graphics teardown and existing
engine shutdown notification both run. This local stub does not
prove remote actor shutdown or shared-memory visibility.
'''
chart: _PriceChart = _PriceChart()
@asynccontextmanager
async def cluster_context(
**kwargs: object,
) -> AsyncIterator[tuple[bool, dict]]:
'''
Supply a local cluster stub without spawning actor processes.
'''
yield False, {}
monkeypatch.setattr(
fsp_ui, 'maybe_open_context', cluster_context,
)
async def scenario() -> None:
'''
Observe cleanup after the actual admin context has exited.
'''
flume: SimpleNamespace = SimpleNamespace(
mkt=SimpleNamespace(fqme='admin.test'),
rt_shm=chart.viz.shm,
rt_sample_period_s=10,
_hist_shm_token=None,
)
source: np.ndarray = _source(
(10, 20, 30, 40, 50), (1,) * 5,
)
started: trio.Event = trio.Event()
started.set()
complete: trio.Event = trio.Event()
async def start_engine(
target: Fsp,
conf: dict,
loglevel: str,
) -> tuple[SimpleNamespace, trio.Event]:
'''
Return an already published, test-owned FSP destination.
'''
return SimpleNamespace(rt_shm=SimpleNamespace(
array=source,
_token='trend',
)), started
admin: FspAdmin
async with fsp_ui.open_fsp_admin(
SimpleNamespace(chart=chart), flume,
) as admin:
monkeypatch.setattr(
admin, 'start_engine_task', start_engine,
)
admin._registry[('test',)] = (None, None, complete)
await admin.open_trend_overlay()
assert len(chart.widget.plotItem.items) == 2
assert chart.widget.plotItem.items == []
assert chart._trend_overlays == {}
assert admin._trend_overlays == []
assert complete.is_set()
try:
trio.run(scenario)
finally:
chart.close()
qapp.processEvents()

611
tests/test_trend_mtf_graphics.py 100644
View File

@ -0,0 +1,611 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Actual shared-tracker graphics, freshness and native feed lifecycle.
'''
from collections import deque
from collections.abc import AsyncIterator, Iterator
from contextlib import asynccontextmanager
from dataclasses import replace
import os
from types import SimpleNamespace
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
import numpy as np
import pyqtgraph as pg
import pytest
import trio
from trio.testing import wait_all_tasks_blocked
from piker.fsp._jev import JevClassification, RegimeAnswer
from piker.fsp._regime import (
MultiTimeframeTrend,
RegimeSnapshot,
TimeframeRegime,
TrendReversal,
)
from piker.ui import _fsp as fsp_ui
from piker.ui._trend import (
RegimeInterval,
ReversalPoint,
TrendOverlay,
)
from piker.ui._trend_mtf import (
SnapshotTrendOverlay,
_SnapshotLayer,
)
from piker.ui.qt import QApplication
class _ArrayShm:
'''
Model published source bounds over deterministic OHLC bars.
'''
def __init__(self, count: int = 600, visible: int = 401) -> None:
'''
Keep an allocated mutable suffix beyond the published range.
'''
self._array: np.ndarray = np.zeros(count, dtype=[
('index', 'i8'),
('time', 'f8'),
('close', 'f8'),
('low', 'f8'),
('high', 'f8'),
])
steps: np.ndarray = np.arange(count)
prices: np.ndarray = (
100 + .015 * steps
+ 2 * np.sin(steps * .09)
+ .5 * np.sin(steps * .4)
)
self._array['index'] = 1000 + steps
self._array['time'] = steps
self._array['close'] = prices
self._array['low'] = prices - .2
self._array['high'] = prices + .2
self._first: SimpleNamespace = SimpleNamespace(value=0)
self._last: SimpleNamespace = SimpleNamespace(value=visible)
self._token: str = 'native-ohlcv'
@property
def array(self) -> np.ndarray:
'''
Expose only the current published SHM range.
'''
return self._array[self._first.value:self._last.value]
class _Chart:
'''
Use a real price plot while replacing actor and SHM allocation.
'''
_trend_overlays: dict[tuple[str, str], list[TrendOverlay]]
def __init__(self, shm: _ArrayShm) -> None:
'''
Bind the actual base source to an offscreen plot.
'''
self.widget: pg.PlotWidget = pg.PlotWidget()
self.viz: SimpleNamespace = SimpleNamespace(
plot=self.widget.plotItem,
shm=shm,
index_field='index',
)
def get_viz(self, fqme: str) -> SimpleNamespace|None:
'''
Resolve the only market owned by this chart.
'''
return self.viz if fqme == 'mtf.test' else None
def close(self) -> None:
'''
Release the disposable Qt widget.
'''
self.widget.close()
self.widget.deleteLater()
@pytest.fixture(scope='session')
def qapp() -> Iterator[QApplication]:
'''
Keep one offscreen application alive for native graphics checks.
'''
app: QApplication|None = QApplication.instance()
if app is None:
app = QApplication(['piker-mtf-tests'])
app.setQuitOnLastWindowClosed(False)
yield app
app.processEvents()
def _replay(
tracker: MultiTimeframeTrend,
shm: _ArrayShm,
overlay: SnapshotTrendOverlay|None = None,
start: int = 0,
) -> RegimeSnapshot:
'''
Replay final closes through the production shared recurrence.
'''
snapshot: RegimeSnapshot
row: np.void
for row in shm.array[start:-1]:
snapshot = tracker.push(
float(row['time']),
float(row['close']),
index=int(row['index']),
)
if overlay is not None:
overlay.submit(snapshot)
return snapshot
def _classification(
snapshot: RegimeSnapshot,
*,
received_at: float|None = None,
) -> JevClassification:
'''
Build explicit synthetic provider output without any HTTP path.
'''
return JevClassification(
market=snapshot.market,
revision=snapshot.revision,
sequence=snapshot.sequence,
segment=snapshot.segment,
source_as_of=snapshot.as_of,
requested_at=snapshot.as_of,
received_at=(
received_at
if received_at is not None
else snapshot.as_of
),
latency=.1,
model='synthetic-jev-test',
prompt_version='test',
answers=(RegimeAnswer(
question='regime_5',
period=5,
label='range',
probability=.8,
confidence=.6,
probabilities=(('range', .8), ('unknown', .2)),
),),
input_tokens=0,
output_tokens=0,
)
def test_bad_limits_do_not_allocate_graphics(
qapp: QApplication,
) -> None:
'''
Reject fractional and boolean bounds before attaching Qt items.
A failed child constructor cannot be cleaned up by the parent
because the parent has not received it yet. Exercise the public
scalar and snapshot constructors against an actual empty plot.
'''
shm: _ArrayShm = _ArrayShm()
chart: _Chart = _Chart(shm)
try:
name: str
for name in ('history_limit', 'order'):
value: float|bool
for value in (1.5, True):
with pytest.raises(ValueError):
SnapshotTrendOverlay(
chart.viz, shm,
fqme='mtf.test', base_period=1,
**{name: value},
)
assert not chart.widget.plotItem.items
for name in ('max_points', 'max_rectangles', 'order'):
value: float|bool
for value in (1.5, True):
with pytest.raises(ValueError):
TrendOverlay(
chart.viz, shm,
fqme='mtf.test', source_period_s=1,
chart_period_s=1,
**{name: value},
)
assert not chart.widget.plotItem.items
finally:
chart.close()
qapp.processEvents()
@pytest.mark.parametrize('periods', [
(1., 5., 15.),
(15.000000000000002, 1.0000000000000002, 5.000000000000001),
])
def test_bootstrap_uses_real_timeframes_and_endpoint_pivots(
qapp: QApplication,
periods: tuple[float, ...],
) -> None:
'''
Sampling 1 s, 5 s and 15 s bars must aggregate one base stream,
rather than relabel identical computations. A real tracker and
scalar sine-wave closes create distinct reversal sets. The
snapshot's pivot end timestamp remains metadata while its x
coordinate equals the final base bar's ``TrendReversal.index``.
Bootstrap supplies no old trend-row history, so rectangles may
begin only at each timeframe's latest proven as_of timestamp.
Harmless decimal rounding must normalize to the tracker's exact
duration keys even if the caller lists timeframes out of order.
'''
shm: _ArrayShm = _ArrayShm()
chart: _Chart = _Chart(shm)
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'mtf.test', periods=periods, history_limit=256,
)
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
chart.viz, shm, fqme='mtf.test', base_period=1,
periods=periods,
)
try:
snapshot: RegimeSnapshot = _replay(tracker, shm)
assert adapter.submit(snapshot)
assert adapter.periods == (1., 5., 15.)
assert len(chart.widget.plotItem.items) == 6
point_counts: list[int] = []
frame: TimeframeRegime
for frame in snapshot.timeframes:
layer: _SnapshotLayer = adapter.layers[frame.period]
event: TrendReversal
events: list[TrendReversal] = [
event for event in frame.reversals
if event.order == 1
]
assert len(layer.points) == len(events)
assert layer.points
point_counts.append(len(layer.points))
point: ReversalPoint
event: TrendReversal
for point, event in zip(
layer.points, events, strict=True,
):
assert point.x == event.index
assert point.pivot_time == event.time
assert point.detected_at == event.detected_time
assert point.timestamp_convention == 'interval_end'
assert layer.rectangles.intervals
span: RegimeInterval # noqa: F842
assert all(
span.start_time >= frame.as_of
for span in layer.rectangles.intervals
)
assert len(set(point_counts)) == 3
finally:
adapter.remove()
chart.close()
qapp.processEvents()
def test_revision_segment_and_sequence_revoke_evidence(
qapp: QApplication,
) -> None:
'''
A corrected source or new gap segment invalidates every previous
pivot and model annotation before replay finishes. The adapter
receives real tracker output, then an explicit generation change.
Old market/revision/sequence snapshots and an earlier segment's
model response must fail acceptance without reviving geometry.
'''
shm: _ArrayShm = _ArrayShm()
chart: _Chart = _Chart(shm)
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
chart.viz, shm, fqme='mtf.test', base_period=1,
)
try:
snapshot: RegimeSnapshot = _replay(tracker, shm)
assert adapter.submit(snapshot)
result: JevClassification = _classification(snapshot)
assert adapter.submit_classification(result)
assert not adapter.submit(snapshot)
assert not adapter.submit(
replace(snapshot, market='foreign'),
)
adapter.invalidate(1)
layer: _SnapshotLayer # noqa: F842
assert all(
not layer.points for layer in adapter.layers.values()
)
assert not adapter.submit(snapshot)
assert not adapter.submit_classification(result)
tracker.reset(revision=1)
replayed: RegimeSnapshot = _replay(tracker, shm)
assert adapter.submit(replayed)
result = _classification(replayed)
assert adapter.submit_classification(result)
after_gap: RegimeSnapshot = tracker.push(
405, 101, index=1405,
)
assert adapter.submit(after_gap)
assert adapter.segment == 1
assert not adapter.submit_classification(result)
assert all(
not layer.points for layer in adapter.layers.values()
)
runs: deque[RegimeInterval] # noqa: F842
assert all(not runs for runs in adapter._model_runs.values())
finally:
adapter.remove()
chart.close()
qapp.processEvents()
def test_coalesced_buckets_leave_unknown_regime_hole(
qapp: QApplication,
) -> None:
'''
Coalesced callbacks reveal only the latest trend classification.
Extending ``_SnapshotLayer._runs`` to the next received snapshot
invents historical knowledge. Skip ten actual finalized buckets
and prove the old interval stops at its next possible decision
while the new run starts only at ``RegimeSnapshot.as_of``.
'''
shm: _ArrayShm = _ArrayShm(visible=101)
chart: _Chart = _Chart(shm)
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
chart.viz, shm, fqme='mtf.test', base_period=1,
)
try:
adapter.submit(_replay(tracker, shm))
previous: float = adapter.as_of
shm._last.value += 10
latest: RegimeSnapshot = _replay(tracker, shm, start=100)
adapter.submit(latest)
runs: deque[RegimeInterval] = adapter.layers[1]._runs
assert len(runs) == 2
assert runs[0].end_time == previous + 1
assert runs[1].start_time == latest.as_of
assert runs[0].end_time < runs[1].start_time
finally:
adapter.remove()
chart.close()
qapp.processEvents()
def test_model_receipt_expiry_and_delayed_queue(
qapp: QApplication,
) -> None:
'''
A Jev response must neither paint before receipt nor claim the
live regime indefinitely. Retain a response arriving 2.5 s
after its source snapshot, advance the real base stream, and
verify its visible interval starts at receipt and stops after
the default 5 s lifetime. Old first-ever responses and expired
queued results are also rejected despite matching provenance.
'''
shm: _ArrayShm = _ArrayShm(visible=201)
chart: _Chart = _Chart(shm)
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
chart.viz, shm, fqme='mtf.test', base_period=1,
)
try:
snapshot: RegimeSnapshot = _replay(tracker, shm)
adapter.submit(snapshot)
old: JevClassification = replace(
_classification(snapshot),
sequence=1,
source_as_of=1,
received_at=200,
)
assert not adapter.submit_classification(old)
expired: JevClassification = _classification(
snapshot, received_at=190,
)
assert not adapter.submit_classification(expired)
result: JevClassification = _classification(
snapshot, received_at=202.5,
)
assert adapter.submit_classification(result)
layer: _SnapshotLayer = adapter.layers[5]
assert len(layer.rectangles.rectangles) == 0
shm._last.value = 212
_replay(tracker, shm, adapter, start=200)
assert layer.rectangles.intervals
span: RegimeInterval = layer.rectangles.intervals[-1]
assert span.observed_at == 202.5
assert span.end_time == 207.5
assert layer.rectangles.rectangles[-1, 0] == 1202.5
assert not adapter.submit_classification(result)
finally:
adapter.remove()
chart.close()
qapp.processEvents()
def test_snapshot_retention_stays_bounded(
qapp: QApplication,
) -> None:
'''
A long-running shared tracker must not accumulate unbounded Qt
objects or regime intervals. Stream hundreds of real snapshots
with a four-event display budget and assert retention at both
the data and scene boundaries after many reversals.
'''
shm: _ArrayShm = _ArrayShm()
chart: _Chart = _Chart(shm)
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
chart.viz,
shm,
fqme='mtf.test',
base_period=1,
history_limit=4,
)
try:
_replay(tracker, shm, adapter)
assert len(chart.widget.plotItem.items) == 6
layer: _SnapshotLayer
for layer in adapter.layers.values():
assert len(layer.points) <= 4
assert len(layer._runs) <= 4
assert len(layer.rectangles.rectangles) <= 4
finally:
adapter.remove()
adapter.remove()
assert chart.widget.plotItem.items == []
chart.close()
qapp.processEvents()
@pytest.mark.parametrize('max_source_age', [5, None])
def test_admin_consumes_native_source_once_and_cancels(
qapp: QApplication,
monkeypatch: pytest.MonkeyPatch,
max_source_age: float|None,
) -> None:
'''
``FspAdmin._run_multi_timeframe_trends()`` must subscribe before
replay, share one tracker across three periods, and invalidate
graphics while model callbacks apply their freshness gate.
The ``sampler()`` stub yields a local Trio receive channel to
the actual ``consume_regime_source()`` task. Explicit append and
equal-bound correction notices, followed by
``wait_all_tasks_blocked()``, prove each notice is consumed
before checking snapshot sequence and revision.
``FspAdmin.close_trend_overlays()`` must cancel the reader:
``closed`` is set only in ``sampler()`` cleanup, before this test
closes either channel endpoint. Empty Qt items and
``_Chart._trend_overlays`` prove owned graphics also detach.
This controls local task ordering without exercising actor IPC.
'''
shm: _ArrayShm = _ArrayShm(visible=201)
chart: _Chart = _Chart(shm)
submitted: list[RegimeSnapshot] = []
invalidated: list[int] = []
subscription_count: list[float] = []
async def scenario() -> None:
'''
Control publication and cancellation with native Trio events.
'''
send: trio.MemorySendChannel[dict]
receive: trio.MemoryReceiveChannel[dict]
send, receive = trio.open_memory_channel(4)
closed: trio.Event = trio.Event()
@asynccontextmanager
async def sampler(
period_s: float,
) -> AsyncIterator[trio.MemoryReceiveChannel[dict]]:
'''
Open exactly one sampler before the real consumer starts.
'''
subscription_count.append(period_s)
try:
yield receive
finally:
closed.set()
monkeypatch.setattr(fsp_ui, 'open_sample_stream', sampler)
linked: SimpleNamespace = SimpleNamespace(
chart=chart,
isHidden=lambda: False,
)
flume: SimpleNamespace = SimpleNamespace(
mkt=SimpleNamespace(fqme='mtf.test'),
rt_shm=shm,
rt_sample_period_s=1,
)
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
admin: fsp_ui.FspAdmin = fsp_ui.FspAdmin(
nursery, {}, linked, flume,
)
adapter: SnapshotTrendOverlay = (
await admin.open_multi_timeframe_trends(
on_snapshot=submitted.append,
on_invalidate=invalidated.append,
max_source_age=max_source_age,
)
)
assert (
await admin.open_multi_timeframe_trends() is adapter
)
assert subscription_count == [1]
assert len(chart.widget.plotItem.items) == 6
assert adapter.sequence == 200
assert bool(submitted) == (max_source_age is None)
shm._last.value += 1
await send.send({'period': 1})
await wait_all_tasks_blocked()
assert adapter.sequence == 201
shm._array['close'][20] += 20
await send.send({'backfilling': ('mtf.test', 1)})
await wait_all_tasks_blocked()
assert invalidated == [1]
assert adapter.revision == 1
assert adapter.sequence == 201
admin.close_trend_overlays()
await closed.wait()
assert chart._trend_overlays == {}
assert chart.widget.plotItem.items == []
await send.aclose()
await receive.aclose()
try:
trio.run(scenario)
finally:
chart.close()
qapp.processEvents()

430
tests/test_trend_reference.py 100644
View File

@ -0,0 +1,430 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Independent batch FHLS oracle and pinned original-Lua observations.
The oracle materializes complete windows and an expanded sequence of
FHLS summaries. It shares neither compiled helpers nor mutable window
state with the production calculator. See docs/trend_calculus.rst.
'''
from dataclasses import dataclass
from itertools import product
import json
from pathlib import Path
from typing import TypedDict
import numpy as np
import pytest
from piker.fsp._trend import (
TrendCalculator,
TrendHistory,
TrendUpdate,
)
class _LuaCase(TypedDict):
'''
Describe the input and observed events of one pinned Lua run.
'''
window: int
values: list[float]
events: list[list[int|float]]
class _LuaFixture(TypedDict):
'''
Type the captured Lua cases consumed by the parity regression.
'''
cases: list[_LuaCase]
@dataclass(frozen=True)
class Point:
'''
Retain a pivot's source position and earliest availability row.
'''
row: int
price: float
available: int
@dataclass(frozen=True)
class Window:
'''
Summarize an entire immutable observation window using extrema.
'''
high: Point
low: Point
available: int
@classmethod
def summarize(
cls: type['Window'],
points: list[Point],
) -> 'Window':
'''
Python min/max retain the earliest equal extremum.
'''
return cls(
high=max(points, key=lambda point: point.price),
low=min(points, key=lambda point: point.price),
available=points[-1].available,
)
def compare(self, previous: 'Window') -> int:
'''
Compare extrema ordinally, avoiding floating subtraction.
'''
if (
(
self.high.price >= previous.high.price
and self.low.price >= previous.low.price
)
and
(
self.high.price > previous.high.price
or self.low.price > previous.low.price
)
):
return 1
if (
(
self.high.price <= previous.high.price
and self.low.price <= previous.low.price
)
and
(
self.high.price < previous.high.price
or self.low.price < previous.low.price
)
):
return -1
return 0
def reference_order(
points: list[Point],
size: int,
*,
lua_seed: bool = False,
) -> tuple[list[tuple[int, int]], list[tuple[Point, int]]]:
'''
Expand complex windows, then derive directions and pivot events.
Direction changes from unknown zero establish a baseline and do
not manufacture a reversal. ``Point.available`` carries detection
availability from the closing observation, including at recursive
orders.
'''
start: int # noqa: F842
windows: list[Window] = [
Window.summarize(points[start:start + size])
for start in range(0, len(points) - size + 1, size)
]
if lua_seed and windows:
last: Point = points[size - 1]
windows.insert(0, Window(last, last, last.available))
if len(windows) < 2:
return [], []
expanded: list[Window] = [windows[0]]
current: Window
previous: Window
for previous, current in zip(windows, windows[1:]):
if current.compare(previous) == 0:
second: Point = max(
(previous.low, previous.high),
key=lambda point: point.row,
)
first: Point = min(
(current.low, current.high),
key=lambda point: point.row,
)
# Lua's synthetic bridge breaks equal-value ties toward
# its second endpoint for high and first for low.
bridge_high: Point
bridge_low: Point
bridge_low, bridge_high = sorted(
[second, first], key=lambda point: point.price,
)
expanded.append(Window(
high=bridge_high,
low=bridge_low,
available=current.available,
))
expanded.append(current)
trend: int = 0
directions: list[tuple[int, int]] = []
events: list[tuple[Point, int]] = []
previous: Window
current: Window
for previous, current in zip(expanded, expanded[1:]):
direction: int = current.compare(previous) or trend
if trend and direction != trend:
pivot: Point = (
previous.low if direction == 1 else previous.high
)
events.append((
Point(pivot.row, pivot.price, current.available),
direction,
))
trend = direction
directions.append((current.available, trend))
return directions, events
def reference_hierarchy(
values: np.ndarray,
sizes: tuple[int, ...],
*,
lua_seed: bool = False,
) -> tuple[
np.ndarray,
np.ndarray,
list[tuple[int, int, int, int]],
]:
'''
Repeatedly reduce whole point sequences, separately per order.
The returned event tuples are pivot row, direction, order and
detection row. Sorting by detection row and order reproduces
causal publication while retaining stable same-order ordering.
'''
row: int
price: np.floating|np.integer # noqa: F842
points: list[Point] = [
Point(row, float(price), row)
for row, price in enumerate(values)
]
trends: np.ndarray = np.zeros(
(len(values), len(sizes)), dtype=np.int8,
)
ready: np.ndarray = np.zeros_like(trends, dtype=np.bool_)
all_events: list[tuple[int, int, int, int]] = []
level: int
size: int
for level, size in enumerate(sizes):
directions: list[tuple[int, int]]
events: list[tuple[Point, int]]
directions, events = reference_order(
points, size, lua_seed=lua_seed,
)
if len(points) >= 2 * size:
ready[points[2 * size - 1].available:, level] = True
row: int
direction: int
for row, direction in directions:
trends[row:, level] = direction
point: Point
direction: int
for point, direction in events:
all_events.append((
point.row, direction, level + 1, point.available,
))
point: Point
direction: int
points = [point for point, direction in events]
all_events.sort(key=lambda event: (event[3], event[2]))
# Public direction remains unresolved until two real windows,
# including when the research seed establishes internal state.
trends[~ready] = 0
return trends, ready, all_events
@pytest.mark.parametrize('sizes', [(2, 2), (3, 2), (4, 3)])
@pytest.mark.parametrize('lua_seed', [False, True])
def test_exhaustive_ternary_histories(
sizes: tuple[int, ...],
lua_seed: bool,
) -> None:
'''
Enumerate ties, inside/outside bars and unresolved flat startup.
Every eight-point sequence over three prices is compared with
independently expanded ``reference_hierarchy()`` windows.
Six-point sequences also exercise the optional Lua seed, without
importing its CLI's output-suppression bugs. Readiness still
requires two actual complete windows. Comparing every
``TrendHistory`` field catches incorrect first/second-extremum
ordering and warmup decisions that agreement between two calls
into the same production kernel cannot detect.
'''
length: int = 6 if lua_seed else 8
sequence: tuple[int, ...]
for sequence in product((-1, 0, 1), repeat=length):
values: np.ndarray = np.array(sequence, dtype=np.float64)
expected: np.ndarray
readiness: np.ndarray
events: list[tuple[int, int, int, int]]
(
expected,
readiness,
events,
) = reference_hierarchy(
values, sizes, lua_seed=lua_seed,
)
actual: TrendHistory = TrendCalculator(
sizes, lua_seed=lua_seed,
).extend(
values, np.arange(len(values), dtype=np.float64),
)
assert np.array_equal(actual.trends, expected), sequence
assert np.array_equal(actual.ready, readiness), sequence
pulses: np.ndarray = np.zeros_like(expected)
prices: np.ndarray = np.full(
(len(values), len(sizes), 2), np.nan,
)
times: np.ndarray = np.full_like(prices, np.nan)
pivot: int
direction: int
order: int
detected: int
for (
pivot,
direction,
order,
detected,
) in events:
side: int = int(direction == 1)
pulses[detected, order - 1] = direction
prices[detected, order - 1, side] = values[pivot]
times[detected, order - 1, side] = pivot
assert np.array_equal(actual.pulses, pulses), sequence
assert np.array_equal(
actual.pivot_prices, prices, equal_nan=True,
), sequence
assert np.array_equal(
actual.pivot_times, times, equal_nan=True,
), sequence
@pytest.mark.parametrize('sizes', [(2,) * 5, (3, 2, 4), (5, 3, 2)])
@pytest.mark.parametrize('seed', range(8))
@pytest.mark.parametrize('lua_seed', [False, True])
def test_random_recursive_events(
sizes: tuple[int, ...],
seed: int,
lua_seed: bool,
) -> None:
'''
Preserve recursive pivot causality across irregular coordinates.
Higher-order pivots occur earlier than their availability, often
sharing one detection row. Rounded random walks create tied and
engulfing windows; ``reference_hierarchy()`` predicts every
event independently of ``TrendCalculator.push()``. Gapped source
indices and nonuniform timestamps prove that propagation retains
source identity and detection time. Field equality and complete
ordered-event equality detect lost or backdated reversals.
'''
rng: np.random.Generator = np.random.default_rng(seed)
values: np.ndarray = np.cumsum(rng.integers(-3, 4, 512))
times: np.ndarray = 1000 + np.cumsum(rng.uniform(.1, 2, 512))
indices: np.ndarray = 800 + np.cumsum(rng.integers(1, 4, 512))
expected: np.ndarray
readiness: np.ndarray
events: list[tuple[int, int, int, int]]
(
expected,
readiness,
events,
) = reference_hierarchy(
values, sizes, lua_seed=lua_seed,
)
calculator: TrendCalculator = TrendCalculator(
sizes, lua_seed=lua_seed,
)
actual_events: list[tuple[int, int, int, int]] = []
row: int
value: np.integer
for row, value in enumerate(values):
update: TrendUpdate = calculator.push(
float(times[row]), float(value), index=int(indices[row]),
)
assert update.trends == tuple(expected[row])
assert update.ready == tuple(readiness[row])
event: np.void
for event in update.reversals:
pivot_row: int = int(np.searchsorted(
indices, event['index'],
))
assert event['price'] == values[pivot_row]
assert event['time'] == times[pivot_row]
assert event['detected_time'] == times[row]
assert event['detected_index'] == indices[row]
assert pivot_row <= row
actual_events.append((
pivot_row, int(event['direction']),
int(event['order']), row,
))
assert actual_events == events
def test_original_lua_fixture() -> None:
'''
Compare production events with actual, unmodified upstream Lua.
The fixture uses increasing initial windows to establish a trend
before reversals, avoiding Lua's documented flat-start crash and
initial simple-reversal suppression. Every later pivot, sign and
earliest detection prefix must agree, including tied extrema.
Comparing complete ``TrendUpdate.reversals`` sequences against
captured Lua output checks an implementation independent of the
Python oracle. Fixture provenance records the exact program and
input hashes; this case does not claim parity at Lua's startup
failures.
'''
path: Path = (
Path(__file__).parent
/ '_inputs'
/ 'trend_calculus_lua.json'
)
fixture: _LuaFixture = json.loads(path.read_text())
case: _LuaCase
for case in fixture['cases']:
calculator: TrendCalculator = TrendCalculator(
(case['window'],),
)
events: list[list[int|float]] = []
row: int
value: float
for row, value in enumerate(case['values']):
update: TrendUpdate = calculator.push(float(row), value)
event: np.void
for event in update.reversals:
events.append([
int(event['index']), float(event['price']),
int(event['direction']),
int(event['detected_index']),
])
assert events == case['events'], case['window']

312
tests/test_trend_regimes.py 100644
View File

@ -0,0 +1,312 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Closed-bar evidence contracts for multiple-timeframe classifiers.
'''
from dataclasses import replace
import json
import numpy as np
import pytest
from piker.fsp._regime import (
MultiTimeframeTrend,
RegimeSnapshot,
TimeframeRegime,
TrendReversal,
)
from piker.fsp._trend import TrendCalculator, TrendUpdate
def test_timeframes_match_independent_closed_bar_calculators(
) -> None:
'''
Aggregating high/low extrema invents their intrabar order and can
change TC reversals. Feed a zigzag scalar close stream through
three periods and compare every prefix with separate calculators
receiving only each complete bucket's final close. Exact trend,
readiness, source-coordinate and detection-time agreement proves
wall-clock aggregation does not change the TC recurrence.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.test', periods=(1, 2, 5), windows=(2, 2),
)
period: int
engines: dict[int, TrendCalculator] = {
period: TrendCalculator((2, 2)) for period in (1, 2, 5)
}
prices: np.ndarray = 100 + np.sin(np.arange(120) / 3) * 10
row: int
price: np.float64
for row, price in enumerate(prices):
snapshot: RegimeSnapshot = tracker.push(
float(row), float(price), index=1000 + row,
)
frame: TimeframeRegime
for frame in snapshot.timeframes:
period: int = int(frame.period)
if (row + 1) % period:
continue
expected: TrendUpdate = engines[period].push(
float(row + 1), float(price), index=1000 + row,
)
assert frame.trends == expected.trends
assert frame.ready == expected.ready
assert frame.as_of == row + 1
assert frame.recent_closes[-1].index == 1000 + row
if len(expected.reversals):
actual: tuple[TrendReversal, ...] = (
frame.reversals[-len(expected.reversals):]
)
got: TrendReversal
event: np.void
for got, event in zip(actual, expected.reversals):
assert got.time == event['time']
assert got.detected_time == (
event['detected_time']
)
assert got.index == event['index']
assert got.direction == event['direction']
def test_partial_startup_bucket_is_never_an_observation() -> None:
'''
Starting a feed mid-bucket must not give the first slow timeframe
a shorter initial observation. Start a five-second stream at t=2
and verify its [0, 5) bucket is explicitly discarded while the
complete [5, 10) bucket yields its final close at t=10 only.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.test', periods=(5,),
)
second: int
for second in range(2, 10):
snapshot: RegimeSnapshot = tracker.push(second, second + 1)
frame: TimeframeRegime = snapshot.timeframes[0]
if second < 9:
assert frame.close_count == 0
assert frame.as_of is None
else:
assert frame.close_count == 1
assert frame.as_of == 10
assert frame.recent_closes[-1].price == 10
assert frame.discarded_buckets == 1
def test_gap_resets_every_timeframe_and_rejects_partial_bucket(
) -> None:
'''
Carrying recurrence state across missing base bars fabricates
continuity and can falsely report a ready regime. Warm two
timeframes, omit t=9, and resume at t=10. The new source segment
must clear all prior evidence, and the interrupted four-second
bucket must not become a valid observation or restore readiness.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.test', periods=(1, 4), windows=(2,),
)
second: int
for second in range(9):
tracker.push(second, second)
snapshot: RegimeSnapshot = tracker.push(10, 20)
assert snapshot.segment == 1
assert snapshot.timeframes[0].close_count == 1
assert snapshot.timeframes[0].ready == (False,)
assert snapshot.timeframes[1].close_count == 0
assert not snapshot.timeframes[1].reversals
snapshot = tracker.push(11, 21)
assert snapshot.timeframes[1].close_count == 0
for second in range(12, 16):
snapshot = tracker.push(second, second)
assert snapshot.timeframes[1].close_count == 1
assert snapshot.timeframes[1].as_of == 16
assert snapshot.timeframes[1].ready == (False,)
def test_subsecond_periods_preserve_epoch_alignment() -> None:
'''
Integer-second resampling silently truncates high-frequency
periods, while float division can select the wrong epoch bucket.
Four quarter-second base bars at a modern Unix epoch must form
exactly one one-second close, with no early slow observation.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'synthetic.test', base_period=.25, periods=(.25, 1),
)
offset: int
for offset in range(4):
snapshot: RegimeSnapshot = tracker.push(
1_700_000_000 + offset / 4,
100 + offset,
)
assert snapshot.timeframes[1].close_count == (offset == 3)
assert snapshot.as_of == 1_700_000_001
assert snapshot.timeframes[1].recent_closes[-1].price == 103
def test_decimal_periods_accept_float_roundoff() -> None:
'''
A calculated duration such as 0.1 * 3 differs from 0.3 by one
float rounding step. Rejecting that representation prevents a
valid integer-microsecond bucket configuration. Normalize only
float-rounding noise and verify three tenth-second samples form
one three-tenth-second observation at the expected endpoint.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', base_period=.1, periods=(.1, .1 * 3),
)
tracker.push(0, 1)
tracker.push(.1, 2)
snapshot: RegimeSnapshot = tracker.push(.2, 3)
assert snapshot.timeframes[1].period == .3
assert snapshot.timeframes[1].as_of == .3
assert snapshot.timeframes[1].close_count == 1
@pytest.mark.parametrize('kwargs', [
{'base_period': 0},
{'base_period': float('nan')},
{'base_period': 0.0000001},
{'base_period': 2, 'periods': (1,)},
{'periods': (1, 1)},
{'periods': ()},
{'periods': (1.5,)},
{'windows': (1,)},
{'history_limit': 0},
{'history_limit': 257},
{'revision': -1},
])
def test_invalid_parameters_reject_ambiguous_sampling(
kwargs: dict,
) -> None:
'''
A timeframe that cannot be built from whole base intervals has
no complete-bucket interpretation. Each invalid configuration
must fail before constructing a usable stream; bounded history
and valid recurrence windows are checked at the same boundary.
'''
with pytest.raises(ValueError):
MultiTimeframeTrend('synthetic.test', **kwargs)
@pytest.mark.parametrize('bar', [
(0, 100, 1),
(-1, 100, 1),
(1.25, 100, 1),
(1, float('nan'), 1),
(1, float('inf'), 1),
(float('inf'), 100, 1),
(1, 100, 0),
(1, 100, 2 ** 63),
(1, 100, np.uint64(2 ** 63)),
(1, 100, np.bool_(True)),
(1e308, 100, 1),
])
def test_invalid_bar_leaves_all_timeframes_unchanged(
bar: tuple[float, float, int|np.integer|np.bool_],
) -> None:
'''
Partial mutation before rejecting a bad bar causes different
timeframes to consume different prefixes. Compare a tracker
subjected to one invalid update with an untouched reference;
accepting the same next bar must produce identical snapshots.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend('test.market')
reference: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market',
)
tracker.push(0, 100)
reference.push(0, 100)
with pytest.raises(ValueError):
tracker.push(bar[0], bar[1], index=bar[2])
assert tracker.push(1, 101) == reference.push(1, 101)
def test_revision_replay_rebuilds_causal_evidence() -> None:
'''
Revised history cannot be prepended to a warm TC recurrence.
Replaying the identical complete stream after reset must retain
every observation and reversal while changing its revision; the
frozen snapshot from the earlier generation must stay unchanged.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', periods=(1, 2), windows=(2, 2),
)
prices: list[int] = [1, 2, 3, 4, 3, 2, 1, 0] * 5
second: int
price: int
for second, price in enumerate(prices):
original: RegimeSnapshot = tracker.push(second, price)
original_state: str = json.dumps(original.to_state())
tracker.reset()
for second, price in enumerate(prices):
replayed: RegimeSnapshot = tracker.push(second, price)
assert replayed == replace(original, revision=1)
assert json.dumps(original.to_state()) == original_state
frame: TimeframeRegime
assert any(frame.reversals for frame in replayed.timeframes)
for frame in replayed.timeframes:
reversal: TrendReversal
for reversal in frame.reversals:
assert reversal.time <= reversal.detected_time
assert reversal.detected_time <= frame.as_of
with pytest.raises(ValueError):
tracker.reset(revision=1)
def test_snapshot_history_is_bounded_and_finite_json() -> None:
'''
Sending unbounded market history makes inference cost grow with
uptime. A long oscillating stream must retain at most the chosen
close/reversal tail per timeframe while preserving total count.
Strict JSON encoding also proves no NaN/Infinity enters state.
Accepted NumPy integer windows must normalize to native integers
so configuration metadata cannot break that same JSON contract.
'''
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
'test.market', periods=(1, 2), history_limit=3,
windows=(np.int64(2), np.int64(2), np.int64(2)),
)
second: int
for second in range(200):
snapshot: RegimeSnapshot = tracker.push(
second,
float(second % 7),
)
frame: TimeframeRegime
for frame in snapshot.timeframes:
assert len(frame.recent_closes) == 3
assert len(frame.reversals) <= 3
assert frame.close_count == 200 / frame.period
decoded: dict = json.loads(json.dumps(
snapshot.to_state(), allow_nan=False,
))
assert decoded['timestamp_convention'] == 'interval_end'
assert decoded['algorithm'] == 'trend_calculus'
assert decoded['windows'] == [2, 2, 2]

149
tests/test_trend_sampling.py 100644
View File

@ -0,0 +1,149 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Native sampler timestamp ordering required by causal processors.
'''
from collections import defaultdict
from collections.abc import Iterator
from types import SimpleNamespace
from unittest.mock import patch
import numpy as np
import trio
from trio.testing import MockClock
from piker.data._sampling import Sampler
from piker.fsp._trend import TrendCalculator, TrendHistory
class SampleSource:
'''
Provide the actual sampler's OHLC and append protocol in memory.
'''
def __init__(self) -> None:
'''
Seed one closed row and the sampler's future open tail.
'''
self._array: np.ndarray = np.zeros(16, dtype=[
('index', '<i8'),
('time', '<f8'),
('open', '<f8'),
('high', '<f8'),
('low', '<f8'),
('close', '<f8'),
('volume', '<f8'),
])
self._array['index'] = np.arange(16)
self._array['time'][:2] = [1000, 1001]
self._array['close'][:2] = [3, 4]
self._write_fields: list[str] = list(
self._array.dtype.names[1:],
)
self.last: int = 2
@property
def array(self) -> np.ndarray:
'''
Expose only the published prefix, including its open tail.
'''
return self._array[:self.last]
def push(self, values: np.ndarray) -> None:
'''
Append OHLC fields while retaining absolute source indices.
'''
end: int = self.last + len(values)
self._array[self._write_fields][self.last:end] = values
self.last = end
def test_sampler_waits_for_strictly_new_epoch() -> None:
'''
Empty realtime feeds seed times round(now) and round(now)+1.
Their first sampler wake can have the same epoch as that future
tail; rounded wall time can also repeat or move backward later.
Appending those epochs makes strict causal FSP bootstrap fail.
Drive the real sampler through all cases with a controlled clock:
keep broadcasting wakeups, append only newer epochs, and verify
the complete output remains valid Trend Calculus input.
'''
async def main() -> None:
'''
Advance virtual time without starting brokers or actors.
'''
source: SampleSource = SampleSource()
epochs: tuple[int, ...] = (
1001, 1001, 1000, 1002, 1002, 1003,
)
clock_values: Iterator[int] = iter(epochs)
counts: list[int] = []
complete: trio.Event = trio.Event()
class ControlledSampler(Sampler):
'''
Isolate sampler state and capture every step broadcast.
'''
ohlcv_shms: dict = {1: [source]}
subscribers: defaultdict = defaultdict(
lambda: [0, set()],
)
@classmethod
async def broadcast(cls, **kwargs: object) -> None:
'''
Stop after all scheduled clock cases are observed.
'''
counts.append(len(source.array))
if len(counts) == len(epochs):
complete.set()
await trio.sleep_forever()
fake_time: SimpleNamespace = SimpleNamespace(
time=lambda: next(clock_values),
)
with patch('piker.data._sampling.time', fake_time):
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
scope: trio.CancelScope = await nursery.start(
ControlledSampler.increment_ohlc_buffer, 1.,
)
await complete.wait()
scope.cancel()
assert counts == [2, 2, 2, 3, 3, 4]
assert source.array['time'].tolist() == [
1000, 1001, 1002, 1003,
]
result: TrendHistory = TrendCalculator().extend(
source.array['close'],
source.array['time'],
source.array['index'],
)
assert len(result.trends) == 4
trio.run(main, clock=MockClock(autojump_threshold=0))

279
tests/test_trend_shm.py 100644
View File

@ -0,0 +1,279 @@
# piker: trading gear for hackers
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
# SPDX-License-Identifier: AGPL-3.0-or-later
# This program is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation, either version 3
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public
# License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
'''
Exercise native Trend Calculus publication through real Tractor SHM.
'''
from collections.abc import AsyncIterator, Callable
from functools import partial
from types import SimpleNamespace
from typing import Any, cast
from uuid import uuid4
import numpy as np
import tractor
from tractor.ipc._shm import (
NDToken,
ShmArray,
attach_shm_ndarray,
open_shm_ndarray,
)
import trio
from trio.testing import wait_all_tasks_blocked
from piker.fsp._engine import (
Cascade,
QuoteFrame,
_next_step_output,
_sample_wakeups,
connect_streams,
)
from piker.fsp._trend import TrendCalculator
from piker.fsp._trend_fsp import _history_fields, trend_calculus
async def _read_in_child(
token: dict[str, Any],
) -> dict[str, Any]:
'''
Attach from an independent actor using the normal token codec.
'''
shm: ShmArray = attach_shm_ndarray(NDToken.from_msg(token))
rows: np.ndarray = shm.array.copy()
fields: dict[str, list] = {}
name: str
for name in rows.dtype.names:
value: int|float # noqa: F842
fields[name] = [
value if np.isfinite(value) else None
for value in rows[name].tolist()
]
return {
'first': shm._first.value,
'last': shm._last.value,
'readonly': not shm.array.flags.writeable,
'fields': fields,
}
def test_real_shm_bootstrap_step_and_in_place_replay() -> None:
'''
Validate actual SHM bounds, field layout and replay visibility.
Bootstrap the real engine against a nonzero source origin and
read its destination through a separate readonly attachment.
Advance a quiet bar through the sampler publication protocol,
then repair an interior close without changing source bounds.
``Cascade.resync()`` must rewrite every affected output and
notify its
consumer. A child actor then attaches using the transported token
and verifies the repaired data independently. All actors and
memory names belong to this test; no broker, model API or
existing Piker daemon is required. The child reads after replay
completes; this proves visibility, not concurrent atomicity.
'''
async def main() -> None:
'''
Own actor lifetimes and SHM names for this replay proof.
'''
suffix: str = uuid4().hex
key: str = f'test_tc_{suffix}'
async with tractor.open_root_actor(
name=key,
tpt_bind_addrs=[('127.0.0.1', 0)],
):
source: ShmArray = open_shm_ndarray(
size=128,
key=f'{key}_source',
append_start_index=37,
dtype=np.dtype([
('index', '<i8'),
('time', '<f8'),
('close', '<f8'),
]),
)
name: str # noqa: F842
destination: ShmArray = open_shm_ndarray(
size=128,
key=f'{key}_output',
append_start_index=0,
dtype=np.dtype([
('index', '<i8'),
('time', '<f8'),
] + [
(name, '<f8')
for name in trend_calculus.outputs
]),
)
seed: np.ndarray = np.zeros(6, dtype=source.array.dtype)
seed['time'] = np.arange(1000., 1006.)
seed['close'] = [1., 2., 4., 3., 2., 5.]
source.push(seed)
reader: ShmArray = attach_shm_ndarray(destination._token)
notices: list[dict] = []
class Sink:
'''
Record resync notices without involving a live UI
stream.
'''
async def send(self, message: dict) -> None:
'''
Retain each engine notice for deterministic
assertions.
'''
notices.append(message)
def check_publication() -> None:
'''
Compare the readonly attachment with causal source
replay.
'''
expected: dict[str, np.ndarray] = _history_fields(
TrendCalculator().extend(
source.array['close'], source.array['time'],
source.array['index'],
),
)
assert reader._first.value == source._first.value
assert reader._last.value == source._last.value
assert not reader.array.flags.writeable
field: str
for field in ('index', 'time'):
np.testing.assert_array_equal(
reader.array[field], source.array[field],
)
field: str
values: np.ndarray
for field, values in expected.items():
np.testing.assert_array_equal(
reader.array[field], values,
)
send: trio.MemorySendChannel[QuoteFrame]
quotes: trio.MemoryReceiveChannel[QuoteFrame]
send, quotes = trio.open_memory_channel[QuoteFrame](0)
market: SimpleNamespace = SimpleNamespace(
fqme='synthetic.test',
)
wakeups: AsyncIterator[QuoteFrame]
notify: Callable[[], None]|None
async with send, quotes, _sample_wakeups(
quotes, market.fqme, True,
) as (wakeups, notify):
nursery: trio.Nursery
async with trio.open_nursery() as nursery:
casc: Cascade = Cascade(
src=SimpleNamespace(
rt_shm=source,
mkt=market,
),
dst=SimpleNamespace(rt_shm=destination),
tn=nursery,
fsp=trend_calculus,
client_stream=cast(
tractor.MsgStream, Sink(),
),
)
compute: Callable = partial(
connect_streams,
casc=casc,
mkt=market,
quote_stream=wakeups,
src=casc.src,
dst=casc.dst,
edge_func=trend_calculus.func,
)
async with casc.open_edge(compute):
await wait_all_tasks_blocked()
check_publication()
assert reader.array['top_1'][-1] == 4.
assert reader.array['bottom_1'][-1] == 2.
step: np.ndarray = source.array[-1:].copy()
step['time'] += 1
source.push(step)
out: np.ndarray = _next_step_output(
destination.array[-1:],
trend_calculus,
False,
)
out['time'] = step['time']
destination.push(out)
if casc.sample_ready is not None:
casc.sample_ready.set()
casc.sample_ready = trio.Event()
assert notify is not None
notify()
await wait_all_tasks_blocked()
check_publication()
source.array['close'][2] = 100.
await casc.resync()
await wait_all_tasks_blocked()
check_publication()
assert len(notices) == 1
info: dict = notices[0]['fsp_update']
assert info['first'] == 37
assert info['last'] == 44
actors: tractor.ActorNursery
async with tractor.open_nursery() as actors:
portal: tractor.Portal = (
await actors.start_actor(
'trend-shm-reader',
enable_modules=[__name__],
)
)
try:
token: dict[str, Any] = (
destination._token.as_msg()
)
remote: dict[str, Any] = (
await portal.run(
_read_in_child,
token=token,
)
)
assert remote['first'] == 37
assert remote['last'] == 44
assert remote['readonly']
fields: tuple[str, ...] = (
reader.array.dtype.names
)
field: str
for field in fields:
np.testing.assert_array_equal(
np.asarray(
remote['fields'][field],
dtype=float,
),
reader.array[field],
)
finally:
await actors.cancel()
assert casc.complete.is_set()
trio.run(main)