diff --git a/ai/prompt-io/README.md b/ai/prompt-io/README.md new file mode 100644 index 00000000..975582c5 --- /dev/null +++ b/ai/prompt-io/README.md @@ -0,0 +1,23 @@ +# AI Prompt I/O Log + +This directory tracks prompt inputs and model outputs for +AI-assisted development across harnesses. + +## Policy + +Prompt logging follows the +[NLNet generative AI policy][nlnet-ai]. Substantive contributions +record the original harness, provider/model when known, timestamps, +the prompting sequence, and an unedited `.raw.md` partner. + +[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/ + +## Usage + +Entries are created by the `prompt-io` skill or its commit-message +integration. Harness-specific records live in their corresponding +subdirectories. Code records can reference the associated Git diff +instead of duplicating source code. + +Human contributors retain responsibility for review and acceptance. +AI-generated content is not presented as human-authored work. diff --git a/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md b/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md new file mode 100644 index 00000000..cce42091 --- /dev/null +++ b/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md @@ -0,0 +1,118 @@ +--- +created_at: 2026-10-05T04:11:32.732262+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +scope: code +substantive: true +raw_file: 20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md +--- + +## Prompt + +The user requested an autonomous search of GitHub issues and local +notes, code, documentation and comments for Trend Calculus research, +then a rigorous Piker-native implementation in the FSP subsystem. +They asked to prepare multiple high-frequency timeframe regimes for +the JEV System-1 provider service, with work bounded by 00:18 Eastern. +They authorized same-caliber subagents and requested, as a bonus, +multiple-timeframe reversal markers and live rectangle annotations. + +The user supplied these implementation resources: + +- https://docs.typesafe.ai/models +- https://docs.typesafe.ai/sdk/python +- https://docs.typesafe.ai/agent-skill + +They offered a Pine implementation but had not supplied it. They +said the API key was still being set up and instructed us to continue +offline. They clarified the workspace concern, disabled fast mode, +and supplied repository rules: preserve human-owned task states, +no autonomous commits or pushes, 69-column prose, Xonsh snippets, +and the Python style skill. + +## Response summary + +Implemented a bounded Numba recurrence and a four-order native FSP +with provisional-tail isolation, skipped-step recovery, indexed +publication, quiet sampler wakeups, sparse events and history replay. +Added complete aligned multiple-timeframe aggregation, immutable +provenance-rich snapshots, bounded HTTPX/Trio JEV requests and a +paced worker that rejects stale or invalidated results. Added native +feed/service wiring and opt-in Qt markers and regime rectangles, +including actual 1/5/15-second layers and model receipt/expiry rules. + +Research identifies pinned Lua/Scala sources, issue discussions, +local notes and the manuscript, with explicit semantic differences. +Original Lua fixtures and an independent materialized-window oracle +check 21,870 exhaustive histories, 48 random recursive streams and +62 original Lua reversals. No complete Pine source was recovered. + +The TypeSafe agent skill was installed and applied. No live API key +was read, no model request was sent, no broker session was opened, +and no Git history or human-owned task state was changed. + +## Verification + +The combined targeted suite passed 252 tests, with one unrelated +remote gap-dialog actor case deselected. The one warning is an +existing Tractor dependency's deprecated Trio Event truth test. +The new SHM test includes an actual separate read-only child actor +after in-place replay. This does not prove concurrent cross-process +write atomicity. Ruff, diff whitespace and 69-column checks passed. + +The combined suite covered test_trend, test_trend_reference, +test_trend_shm, test_trend_regimes, test_jev, test_regime_feed, +test_trend_graphics, test_trend_mtf_graphics, test_fsp_sync, +test_fsp_momo and test_gap_overlays. The standard py313 environment +and pytest with the xonsh plugin disabled were used. Socket/SHM +execution required approved sandbox escalation. + +The synthetic offscreen chart preview was generated and inspected. +The reproducible benchmark measured roughly 8.586 million batch +rows/second and 6.963 microseconds median scalar update on this host; +these exclude rendering, transport and provider inference. Native +subsecond sampling remains unsupported by the existing clock. +Live provider quality and the user's Pine parity remain unverified. + +## Files changed + +- `piker/fsp/__init__.py` +- `piker/fsp/_api.py` +- `piker/fsp/_engine.py` +- `piker/ui/_display.py` +- `piker/ui/_fsp.py` +- `piker/fsp/_trend.py` +- `piker/fsp/_trend_fsp.py` +- `piker/fsp/_regime.py` +- `piker/fsp/_jev.py` +- `piker/fsp/_regime_feed.py` +- `piker/ui/_trend.py` +- `piker/ui/_trend_mtf.py` +- `tests/test_trend.py` +- `tests/test_trend_shm.py` +- `tests/test_trend_reference.py` +- `tests/test_trend_regimes.py` +- `tests/test_jev.py` +- `tests/test_regime_feed.py` +- `tests/test_trend_graphics.py` +- `tests/test_trend_mtf_graphics.py` +- `tests/_inputs/trend_calculus_lua.json` +- `examples/benchmark_trend.py` +- `examples/trend_regimes.py` +- `examples/trend_graphics.py` +- `docs/trend_calculus.rst` +- `ai/prompt-io/README.md` + +## Human edits + +The human supplied the objective, deadline, research resources, +provider choice, optional skill installation, graphics extension, +parallel-agent authorization and offline credential constraint. +Those directions materially determined the implementation and scope. +No direct manual source edits were observed during this work. The +patch is pending human review; acceptance is not inferred. Research, +implementation, review and tests were performed by the root Codex +agent with three same-caliber delegated agents. Existing unrelated +working-tree artifacts were preserved. diff --git a/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md b/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md new file mode 100644 index 00000000..e0452cf6 --- /dev/null +++ b/ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md @@ -0,0 +1,106 @@ +--- +created_at: 2026-10-05T04:11:32.732262+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +diff_cmd: git diff eed58566 -- +--- + +# Captured output and code references + +This record captures selected unedited progress responses and the +code artifact pointers before human source edits. It is not a full +transcript. Exact model and session identifiers were unavailable. +The working branch is trend_calc_on_jev; no commit was authorized. +While files remain untracked, use the no-index references below. +After inclusion in branch history, use the base-to-branch diff. + +## Unedited progress excerpts + +The cross-process shared-memory test passes, including replay +after an in-place history repair. The JEV feed also preserves +the shared quote policy and catches up cleanly when new bars +arrive during a history copy. + +I’m running the combined checks and reviewing the chart preview. +Live JEV verification remains pending your key setup. + +The combined offline suite passes: 252 tests, including the +independent algorithm oracle, JEV mock transport, Qt overlays, +existing FSP checks, and the cross-process SHM test. + +The chart preview shows separate 1-, 5-, and 15-second reversal +markers and regime shading. I’m checking the final diff and +recording provenance; the work remains uncommitted. + +## Unedited verification output + +```text +252 passed, 1 deselected, 1 warning in 3.79s +All checks passed! +``` + +## Code artifact pointers + +> `git diff eed58566 -- piker/fsp/__init__.py` + +> `git diff eed58566 -- piker/fsp/_api.py` + +> `git diff eed58566 -- piker/fsp/_engine.py` + +> `git diff eed58566 -- piker/ui/_display.py` + +> `git diff eed58566 -- piker/ui/_fsp.py` + +> `git diff --no-index /dev/null piker/fsp/_trend.py` + +> `git diff --no-index /dev/null piker/fsp/_trend_fsp.py` + +> `git diff --no-index /dev/null piker/fsp/_regime.py` + +> `git diff --no-index /dev/null piker/fsp/_jev.py` + +> `git diff --no-index /dev/null piker/fsp/_regime_feed.py` + +> `git diff --no-index /dev/null piker/ui/_trend.py` + +> `git diff --no-index /dev/null piker/ui/_trend_mtf.py` + +> `git diff --no-index /dev/null tests/test_trend.py` + +> `git diff --no-index /dev/null tests/test_trend_shm.py` + +> `git diff --no-index /dev/null tests/test_trend_reference.py` + +> `git diff --no-index /dev/null tests/test_trend_regimes.py` + +> `git diff --no-index /dev/null tests/test_jev.py` + +> `git diff --no-index /dev/null tests/test_regime_feed.py` + +> `git diff --no-index /dev/null tests/test_trend_graphics.py` + +> `git diff --no-index /dev/null tests/test_trend_mtf_graphics.py` + +> `git diff --no-index /dev/null tests/_inputs/trend_calculus_lua.json` + +> `git diff --no-index /dev/null examples/benchmark_trend.py` + +> `git diff --no-index /dev/null examples/trend_regimes.py` + +> `git diff --no-index /dev/null examples/trend_graphics.py` + +> `git diff --no-index /dev/null docs/trend_calculus.rst` + +> `git diff --no-index /dev/null ai/prompt-io/README.md` + +The implementation adds the bounded recursive TrendCalculator, +causal history/preview FSP publication, explicit indexed FspSample +updates, coalesced quiet-step wakeups and sparse-event clearing. +MultiTimeframeTrend forms complete aligned buckets; the Jev worker +uses bounded evidence, single-flight requests and freshness checks. +Qt layers render confirmed pivots and bounded regime rectangles. +Independent reference, mock transport and shared-memory tests cover +mathematical and integration behavior. The docs preserve sources, +semantic differences, measured performance and remaining limits. diff --git a/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md b/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md new file mode 100644 index 00000000..6e13305c --- /dev/null +++ b/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md @@ -0,0 +1,42 @@ +--- +created_at: 2026-10-05T05:02:18.111444+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +scope: code +substantive: true +raw_file: 20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md +--- + +## Prompt + +Continuation of the user's autonomous, deadline-bounded request for +rigorous native Trend Calculus, live regime service preparation and +graphics integration. This supplements the earlier trend record. + +## Response summary + +Final delegated integration review reproduced duplicate timestamps +in the existing native sampler: seeded [1000,1001] became +[1000,1001,1001], causing the causal calculator to reject the source. +Added a narrow guard for native periods at or below one second and +a controlled-clock regression covering repeated, backward and newer +epochs. Wakeup broadcasts remain unchanged. No refactor or native +subsecond support is claimed. The root reviewed the change and ran +the complete targeted suite: 253 passed, one unrelated remote gap +case deselected, one existing Tractor/Trio deprecation warning. + +## Files changed + +- `piker/data/_sampling.py` +- `tests/test_trend_sampling.py` +- `docs/trend_calculus.rst` + +## Human edits + +The original human requirement for reliable native integration +motivated this correction. No additional direct human source edit +was observed. The root and delegated Codex agent implemented and +reviewed it under that scope. Human review and acceptance remain +pending; no commits, broker sessions or model requests were made. diff --git a/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md b/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md new file mode 100644 index 00000000..47a51a9f --- /dev/null +++ b/ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md @@ -0,0 +1,38 @@ +--- +created_at: 2026-10-05T05:02:18.111444+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +diff_cmd: git diff eed58566 -- +--- + +# Final integration correction + +## Unedited progress output + +The sampler fix is verified. The final combined suite passes +253 tests, including repeated and backward clock cases. +Implementation and tests are now frozen; I’m saving the review +patch and final provenance record. + +## Unedited verification result + +```text +253 passed, 1 deselected, 1 warning in 3.86s +``` + +## Code pointers + +> `git diff eed58566 -- piker/data/_sampling.py` + +> `git diff --no-index /dev/null tests/test_trend_sampling.py` + +> `git diff --no-index /dev/null docs/trend_calculus.rst` + +The actual Sampler.increment_ohlc_buffer could append a duplicate +rounded epoch after realtime startup, or when the clock repeated +or moved backward. The fix retains the existing mutable bar until +a newer epoch exists, preserving broadcasts. The regression drives +the real sampler with Trio MockClock and verifies that its complete +output remains valid input for the strict causal calculator. diff --git a/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md b/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md new file mode 100644 index 00000000..86c5d02f --- /dev/null +++ b/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md @@ -0,0 +1,79 @@ +--- +created_at: 2026-10-05T05:40:14.294712+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +scope: code +substantive: true +raw_file: 20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md +--- + +## Prompt + +The human explicitly requested a complete py-codestyle pass and +proper license headers for every new code module after identifying +that the previous pass was incomplete. They retained autonomous +work and parallel-agent authorization, offline provider constraints, +and human ownership of task states and commits. + +## Response summary + +Read the repository-deployed py-codestyle skill and shared runtime +review lessons. Root and three agents audited all changed Python +modules, tests and runnable examples. Added complete project AGPL +notices and SPDX identifiers to all 19 new modules, preserving the +Morgan/ByteSumo algorithm attribution. Corrected annotations, +helper contracts, expression layout, named interpolation operands, +and owner-qualified regression documentation. Used narrow F842 +suppressions where Ruff cannot connect declarations to isolated +comprehension scopes. + +## Verification + +Ruff passes across all changed Python modules. All new source and +added old-file lines meet the 69-column rule with no whitespace-only +lines. Every new function and class has the required multiline +contract; function parameters and returns are annotated. The final +offline suite reports 253 passed, one unrelated actor case +excluded, and one existing Tractor deprecation warning. Native SHM +publication, cross-process replay visibility, quiet sampler steps, +Numba reference parity, Qt overlays and mocked Jev protocols pass. +No live model quality, separate provider-service wiring, Pine parity +or cross-process write atomicity is claimed. + +## Human edits and follow-ups + +The human's style correction materially drove these changes. +No direct manual source edits were observed; acceptance is pending. +The human deferred SDK AnyIO/Trio work until the HTTP prototype is +validated and requested a later Taken-store handoff. Existing task +states, unrelated artifacts, Git index and Git history are preserved. + +## Files changed + +- `piker/fsp/__init__.py` +- `piker/fsp/_api.py` +- `piker/fsp/_engine.py` +- `piker/ui/_display.py` +- `piker/ui/_fsp.py` +- `piker/data/_sampling.py` +- `piker/fsp/_trend.py` +- `piker/fsp/_trend_fsp.py` +- `piker/fsp/_regime.py` +- `piker/fsp/_jev.py` +- `piker/fsp/_regime_feed.py` +- `piker/ui/_trend.py` +- `piker/ui/_trend_mtf.py` +- `tests/test_trend.py` +- `tests/test_trend_shm.py` +- `tests/test_trend_reference.py` +- `tests/test_trend_regimes.py` +- `tests/test_jev.py` +- `tests/test_regime_feed.py` +- `tests/test_trend_graphics.py` +- `tests/test_trend_mtf_graphics.py` +- `tests/test_trend_sampling.py` +- `examples/benchmark_trend.py` +- `examples/trend_regimes.py` +- `examples/trend_graphics.py` diff --git a/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md b/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md new file mode 100644 index 00000000..92eba3d6 --- /dev/null +++ b/ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md @@ -0,0 +1,98 @@ +--- +created_at: 2026-10-05T05:40:14.294712+00:00 +generated_by: + harness: codex + provider: OpenAI +git_ref: eed58566 +scope: code +substantive: true +--- + +## Human instruction (verbatim) + +ok since i'm seeing this alot i want to make sure you did a proper /py-codestyle pass and have proper license headers throughout all new code-modules. + +## Captured assistant output (verbatim) + +The earlier pass wasn’t complete: the tests and examples lacked +license notices, and several modules had only abbreviated ones. +All 19 new Python files now have full AGPL notices and SPDX +identifiers, with the algorithm’s source attribution preserved. + +I’m checking the remaining style rules too—annotations, +docstrings, expression layout, and regression explanations. + +## Final regression output (verbatim) + +```text +253 passed, 1 deselected, 1 warning in 7.93s +``` + +The warning is the existing Tractor dependency's Trio Event +truth-test deprecation; the unrelated remote gap-dialog actor test +is excluded. The first sandboxed attempt stopped on Trio socket +permissions and was rerun with approved escalation. No broker or +Jev API request was made. + +## Code pointers + +The associated boundary diffs capture full source changes. The +whole-file commit package includes untracked modules, which a bare +working-tree git diff does not include. Human editing and committing +remain pending; these pointers become valid after the complete +package is committed. + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/__init__.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_api.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_engine.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_display.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_fsp.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/data/_sampling.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend_fsp.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_jev.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime_feed.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend_mtf.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_shm.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_reference.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_regimes.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_jev.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_regime_feed.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_graphics.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_mtf_graphics.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_sampling.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/benchmark_trend.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_regimes.py` + +> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_graphics.py` + +The style pass added full license headers, annotations, multiline +helper docstrings, explicit expression layout and owner-qualified +regression proof descriptions. All recurrence behavior and source +credits are preserved; callback constructor defaults are explicit. diff --git a/docs/trend_calculus.rst b/docs/trend_calculus.rst new file mode 100644 index 00000000..2b3b2aab --- /dev/null +++ b/docs/trend_calculus.rst @@ -0,0 +1,645 @@ +Trend Calculus: sources and Piker semantics +========================================== + +This implementation follows Andrew J. Morgan's FHLS Trend Calculus. +It classifies observed price structure; it does not establish +predictive accuracy or a profitable trading rule. Research was +collected on 2026-10-04/05. The source inventory below records what +was found and distinguishes missing material from working code. + +Algorithm +--------- + +Partition an ordered scalar series into disjoint windows containing +``n >= 2`` observations. Summarize each window by its high and low, +retaining each extremum's original coordinate. First and second in +FHLS mean the earlier and later of these two extrema. They do not +mean the opening and closing observation. + +For consecutive windows ``previous`` and ``current``, define:: + + trend = sign( + sign(current.high - previous.high) + + sign(current.low - previous.low) + ) + +The implementation uses ordinal comparisons to avoid overflowing +the differences. When the result is zero, introduce a synthetic +window joining the previous window's second extremum to the current +window's first extremum. Compare previous to synthetic, then +synthetic to current. Carry the prior direction through unresolved +zero comparisons. A complex window can confirm two opposite +reversals at the same observation; retaining only its final trend +would lose an event. + +A confirmed down-to-up change selects the preceding summary's low; +an up-to-down change selects its high. Apply the same calculation to +the resulting ordered reversal points for the next reversal order. +The order counts successive reductions of the pivot stream. It is +independent of a wall-clock timeframe such as one or five seconds. + +The live recurrence retains one partial window, one completed +window and the prior direction per order. Its numeric state is +``O(orders)`` regardless of history length or window size. Historical +scanning calls the same Numba recurrence. Returning a full historical +result necessarily allocates arrays proportional to history length +and order count. + +A completed window emits at most two reversals. Because each configured +window contains at least two observations, two incoming reversals can +complete at most one next-order window during that source update. +Induction bounds every order to two events per source observation; +the fixed event slots therefore do not truncate a larger cascade. + +Deliberate semantic choices +-------------------------- + +``piker.fsp._trend.TrendCalculator`` accepts scalar observations, +finite strictly increasing timestamps, and strictly increasing +nonnegative source indices. Coordinates need not be contiguous. +An OHLC candle alone cannot reveal whether its high or low happened +first; feeding fabricated high/low order would change the algorithm. +The chart FSP therefore uses close prices. + +Production startup is neutral. The first completed window establishes +the baseline without inventing a pivot. ``ready`` becomes true after +two completed windows at that order, including when the direction +remains zero on flat data. Thus zero is not a calibrated "sideways" +regime probability. Readiness and direction are separate information. +Higher orders warm up only when enough lower-order pivots exist. + +Equal highs and lows in ordinary FHLS windows retain their earliest +observation, matching the original Lua implementation. Synthetic +bridges have a narrower, distinct rule: when their endpoint values +are equal, high belongs to the second endpoint and low to the first. +That distinction affects pivot coordinates even when trends agree. + +An optional ``lua_seed=True`` reproduces the reference's initial +comparison with the first window's last observation. It is a seed +policy for reference investigation, not a promise to reproduce all +Lua command-line startup quirks. Normal FSPs use neutral startup. + +NaN clears all recurrence orders and requires fresh warmup. Infinity +and ambiguous coordinate ordering are rejected before state changes. +The scalar calculator does not guess session boundaries from elapsed +time: its caller decides whether a gap is intentional or a reset. +Prepending or revising source history requires replay from the new +origin; changing a prefix can affect every subsequent reversal. + +Causality and output +-------------------- + +``TrendCalculator.push()`` returns all newly detected reversals. +Each record carries both coordinates: + +* ``index``, ``time``, ``price`` locate the original pivot; +* ``detected_index``, ``detected_time`` locate the observation that + first confirmed it; +* ``direction`` is the new trend: ``+1`` for a bottom, ``-1`` for a + top; ``order`` starts at one. + +A higher-order pivot retains its original price/time while detection +time propagates from the input observation that completed its +higher-order window. Strategies must use detection time, not the +earlier pivot time, when deciding what information was available. + +``TrendCalculator.extend()`` returns per-observation ``trends`` and +``ready`` arrays. ``pulses`` stores only the last detected direction +per order and row. ``pivot_prices`` and ``pivot_times`` retain both +possible sides: axis two is top then bottom, with NaN for no event. +Use complete push events when the event stream itself is required. + +The ``trend_calculus`` FSP exposes four orders through ``trend``, +``trend_2``, ``trend_3`` and ``trend_4``. Its ``ready`` field counts +ready orders. Each order also exposes ``top_N``, ``top_time_N``, +``bottom_N`` and ``bottom_time_N``. Pivot prices are published at +their detection row, with the paired field retaining pivot time. +Its ``window`` argument changes the first-order close window only: +the four sizes are ``(window, 2, 2, 2)``, defaulting to all twos. +The numerical and timeframe APIs accept explicit per-order tuples. + +Piker's source SHM timestamps name candle starts. Historical rows +preceding the last row are finalized; the last row is provisional. +Quotes and source sampler steps preview that row from a copy of +committed recurrence state. These wakeups inspect authoritative source +SHM; they are not additional scalar observations. Each newly finalized +row advances committed state once, including when several steps arrive +before a wakeup. Sparse pivot fields clear when the destination steps; +source revisions require a cascade replay. + +For realtime regime consumers, use finalized closes with an explicit +interval-end clock. ``piker.fsp._regime.MultiTimeframeTrend`` forms +epoch-aligned buckets from complete base bars, discards partial +startup buckets and resets recurrence after missing base bars. Its +snapshots distinguish timeframe period, recursive order, source +revision, sequence, gap segment and availability time. An old model +response must not overwrite newer source evidence. + +The scalar calculator and current SHM ``trend_calculus`` FSP count +observations without inferring resets from elapsed timestamp gaps; +NaN explicitly resets their recurrence. ``MultiTimeframeTrend`` instead +requires contiguous base bars and starts a new segment after a missing +interval. Consequently, the simple one-second FSP and the one-second +timeframe tracker can differ across session gaps. That difference is +an ingestion policy, not a different FHLS recurrence. + +Using the numerical and timeframe APIs +-------------------------------------- + +The numerical API has no provider, GUI or credential requirement. +This complete example confirms both sides of a complex window: + +.. code-block:: python + + from piker.fsp._trend import TrendCalculator + + calculator = TrendCalculator((2, 2, 2)) + prices: list[float] = [1., 2., 4., 3., 2., 5.] + row: int + price: float + for row, price in enumerate(prices): + update = calculator.push( + 1_700_000_000. + row, + price, + index=100 + row, + ) + + print(update.trends) + print(update.reversals[ + ['index', 'direction', 'detected_index'] + ]) + +The last update has a top at source index 102 and a bottom at 104, +both detected at 105. A consumer only learns these events when it +receives that update. For batch analysis, pass aligned one-dimensional +``values``, ``times`` and optional signed integer ``indices`` arrays +to ``calculator.extend()``. Use a fresh calculator for an independent +historical replay; ``extend()`` advances an existing instance. + +For wall-clock timeframes, submit each completed base bar once, using +its opening timestamp. The tracker converts this to availability at +the interval end: + +.. code-block:: python + + from piker.fsp._regime import MultiTimeframeTrend + + tracker = MultiTimeframeTrend( + 'synthetic.test', + base_period=1., + periods=(1., 5.), + windows=(2, 2, 2), + ) + prices: list[float] = [100., 101., 99., 103., 102.] * 4 + row: int + price: float + for row, price in enumerate(prices): + snapshot = tracker.push( + 1_700_000_000. + row, + price, + index=100 + row, + ) + + state: dict = snapshot.to_state() + +These periods describe independent close series; ``windows`` still +counts observations and reversal points. Subsecond periods must be +whole microseconds, and every configured period must be a whole +multiple of ``base_period``. Startup midway through a larger bucket +does not produce a shortened first observation. On history repair, +call ``tracker.reset()`` and replay the corrected completed bars. + +Source inventory +---------------- + +Project research and earlier implementation work: + +* `Piker #79`_ is the older trend-detection issue; `Piker #103`_ is + the consolidated research entrypoint. It links Morgan's paper, + Lua, Scala, a working mathematical manuscript, and a public + TradingView/Pine implementation. +* `Piker #187`_ explicitly proposes Trend Calculus in the native FSP + set and links the planned Python implementation. +* `Piker #325`_ connects trend extrema computation with faster chart + y-range reductions. Its moving min/max proposal concerns + performance; it is not an alternative definition of FHLS. +* `Python implementation issue #1`_ requests offline/online support + and recurrence updates across timeframes. The ``setup`` branch + inspected at ``30afcfbd8cbb7b84d984fa2e183c6ae392da3479`` has a + placeholder ``tc`` module and Ray moving-average experiments. + Those experiments do not implement FHLS reversal detection. + +Primary algorithm sources: + +* `Morgan's method document`_, ``HowToStudyTrends_v1.03.pdf``: + the original conceptual account. Inspected PDF SHA-256:: + + 2dc506a0449c13c01d2d798f120411df84e5667b611d92204ba0122e2aac7fd5 + +* `Original Lua program`_, ByteSumoLtd/TrendCalculus-lua commit + ``7f54f9a101283cc188dbcd9282292fc3f73de8e9``, + ``make/trendcalculus.lua``. Its header credits Andrew J. Morgan / + ByteSumo Limited and declares GPL version 3 or later. Inspected + source SHA-256:: + + c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0 + +* `Spark TrendCalculus2`_, commit + ``3fb6ef23e985d9d6e0414d3b682afbeef51c52e8``. The source declares + Apache-2.0. It uses earliest equal high and latest equal low, and + initializes from a zero dummy point or an excluded first point. + Those choices can change reversals and recursive warmup relative + to Lua and Piker. Inspected source SHA-256:: + + 3d0723836d230c89d99537f1075537c3bbf809cebefd12caa6a1b517e7bc98b5 + +* `Working mathematical manuscript`_, ``main.tex``: describes + earliest equal low and latest equal high, a fixed initial positive + trend and an initial synthetic reversal. These are different from + both the Scala code's tie choices and native neutral startup. + Its displayed reversal-sign expression also disagrees with the + following low/high selection branches. It is a working draft, + not a unique executable specification; the executable references + and independently checked cases determine the conventions here. + Inspected source SHA-256:: + + e7b6fdff06fbde195a81281022cce300135d88e7484a97d1b02d621c5bfc4eab + +* `Spark examples`_ provide historical visualization context. + The `public Pine indicator`_ linked from Piker #103 describes two + lagged Donchian channels and their midpoints for visual analysis. + That description does not establish recursive FHLS parity. No + complete Pine source or the user's separate Pine implementation + was recovered in this search, so no Pine parity claim is made. + +Local research locations inspected, relative to the user's repos +directory: + +* ``lns/todo/piker/quant/trend.toml`` links the Python ``setup`` + branch, authors, Piker #187 and the Ray SMA example. +* ``lns/the_legacy_tab_nightmare/trading.toml`` links the working + manuscript, Spark comparisons/PRs, examples, and a former local + ``TrendCalculus/notes_redesign_amorgan.pdf``. The latter file was + not present at its recorded path. +* ``data/bookmarks/bookmarks.html`` preserves the older ByteSumo + Bitbucket project URL. Duplicate bookmarks are not independent + algorithm evidence. + +A separate filename inventory across ``/home/goodboy/repos`` included +hidden and ignored files while excluding Git internals, worktrees, +virtual environments, dependency directories, caches and build outputs. +It found no ``*.pine`` or ``*.pinescript`` files, TrendCalculus-named +source files, or copy of ``notes_redesign_amorgan.pdf`` beyond this +change's new documentation and fixture. + +No earlier native Trend Calculus implementation was found in the +searched Piker source, tests, docs, snippets or available research +notes. This is a bounded search result, not proof that no private +copy or unindexed history exists. User-owned research/task states +were left untouched. + +Verification and reproducibility +------------------------------- + +``tests/test_trend_reference.py`` uses a separate batch oracle that +materializes complete FHLS windows, inserts synthetic windows and +then reduces the expanded sequence. It never calls the production +kernel or copies its mutable recurrence state. Coverage includes: + +* every eight-point sequence over three prices, for three window + tuples: 19,683 histories with ties, flat startup and inside/outside + windows; direction, readiness, pulses and both pivot sides agree; +* another 2,187 exhaustive six-point histories using the optional Lua + seed, while keeping readiness tied to two actual complete windows; +* 48 seeded 512-point random walks across both startup policies, + five recursive orders and mixed window sizes, checking every event's + source identity and detection coordinates on irregular timestamps + and source indices; +* actual unmodified Lua output for windows 2, 3, 4 and 7, covering + 62 reversals, including original pivot locations and earliest + detection prefixes. + +The original-Lua fixture is +``tests/_inputs/trend_calculus_lua.json``. It records source/runtime +provenance, the complete synthetic inputs, event schema and SHA-256 +hashes of each input and full output CSV. To reproduce an output, +write the fixture values with header ``ticker,date,last`` and rows +``x,,``, using LF line endings. Run the pinned +program on Lua 5.2.4 with ``-n -r -H -p1``. For detection +times, rerun each complete-window prefix and assign newly emitted +events to that prefix's last input row. Prefix outputs must preserve +all earlier emitted rows. No Lua executable or network is needed +for these Python reference tests. + +The pinned Lua CLI has observable quirks: + +* Do not pass ``-h`` with its built-in column mapping. That path + consumes the first real observation as a second header, shifting + every window boundary. ``-H`` controls only output headers. +* A flat first complete window can fail while reading a nonexistent + prior trend during bridge processing. +* The simple-window reversal branch suppresses initial output until + ``offset > 2*n + 1``; split-window branches use other guards. + +The fixture begins with increasing complete windows to establish a +common baseline before reversals. It establishes exact parity for +those inputs, not universal startup equivalence between programs. +The native oracle separately covers the flat and complex startup +cases excluded by that reference limitation. + +Integration verification is separate from mathematical parity: + +* ``tests/test_trend.py`` exercises committed/preview handoff, + quote-free sampler wakeups, indexed publication and cancellation. +* ``tests/test_trend_sampling.py`` exercises the actual sampler with + a controlled clock. Repeated or backward rounded epochs must not + append duplicate timestamps, including after realtime bootstrap. +* ``tests/test_trend_shm.py`` runs actual Tractor shared memory with a + separate read-only attachment, nonzero origin, quiet sampling and + an equal-bound historical repair. A separate child actor then + attaches read-only using the transported token and checks every + repaired field. This verifies cross-process visibility after replay, + not atomicity during concurrent writes. It needs actor runtime/socket + access, but no broker or model service. +* ``tests/test_trend_regimes.py`` checks complete aligned buckets, + fractional clocks, missing intervals and revision replay; + ``tests/test_regime_feed.py`` checks native source admission, + subscription ordering, quote receiver ownership and lifecycle. +* ``tests/test_jev.py`` uses mocked transport and controlled clocks + for response validation, bounded projection, pacing, retry, source + invalidation and cancellation while publication is blocked. +* ``tests/test_trend_graphics.py`` and + ``tests/test_trend_mtf_graphics.py`` exercise Qt geometry, causal + timestamp mapping, revision invalidation, model expiry and teardown. + +Performance measurement +----------------------- + +The repository example measures compilation/warmup, batch computation, +individual committed observations and copied-state previews separately. +Run it in the existing Piker Python environment: + +.. code-block:: xsh + + py313/bin/python examples/benchmark_trend.py --rows 1000000 --live-rows 20000 + +One x86-64 run on 2026-10-05 used Python 3.13.14, NumPy 2.3.5 and +Numba 0.62.1 with four reversal orders: + +=========================== ============================ +Measurement Observed result +=========================== ============================ +One million-row batch 0.116473 seconds +Batch throughput 8.586 million rows/second +Scalar commit, median/p99 6.963 / 10.670 microseconds +Copy and preview, median/p99 12.223 / 18.835 microseconds +Initial kernel warmup 3.004 seconds +Retained NumPy state 936 bytes +=========================== ============================ + +These are reproducible measurements from one machine and workload, +not latency guarantees. Retained-state bytes exclude Python objects, +compiler caches and returned history arrays. Timings exclude source +SHM copies, actor IPC, chart rendering and provider inference. The +native sampler currently timestamps periods at or below one second +with integer seconds; subsecond end-to-end operation requires a +producer with an appropriate event clock, despite the numerical and +timeframe APIs supporting fractional-second observations. + +TypeSafe/Jev integration +----------------------- + +The user supplied the `TypeSafe model documentation`_, +`Python SDK documentation`_ and `agent skill`_. The adapter uses +Piker's existing HTTPX/Trio stack against the documented `System One +HTTP API`_; installing the separate Python SDK is unnecessary. The +default model is pinned to ``jev-1.13.0``. Importing the adapter does +not read credentials or open a connection. + +``build_regime_request(snapshot)`` produces one Choice question for +each ready timeframe, plus an independent alignment question when +multiple timeframes are ready. Candidate regimes are ``uptrend``, +``downtrend``, ``range``, ``transition`` and ``unknown``. Questions +explicitly identify state paths because question identifiers do not +reach the model. ``JevClassification`` retains every candidate's +probability, the selected label, the service's separate confidence +value, model/prompt versions, source provenance and receipt time. +These semantic judgments do not change the numerical recurrence. +Model input uses the most recent 16 closes and 16 reversals per +timeframe by default. ``evidence_limit`` configures that projection +independently of longer chart history; source snapshots stay unchanged. + +Inspect a complete synthetic request, then exercise HTTP parsing with +explicitly simulated responses: + +.. code-block:: xsh + + py313/bin/python examples/trend_regimes.py + py313/bin/python examples/trend_regimes.py --mock + +The synthetic example includes quarter-second evidence. Its ``--live`` +mode sends exactly one synthetic request and reads ``TYPESAFE_API_KEY`` +from the caller's environment. It consumes provider credit and makes no +broker connection. A real provider request has not been verified without +an API key; mocked results test the integration, not model quality. + +The native live bridge subscribes to the sampler before copying history, +excludes the mutable final source row, and emits only the newest snapshot +after each captured history range. It catches up skipped sample steps +and invalidates old judgments before replaying a prepend, truncation, +backfill or lost sampler notice. ``run_regime_feed()`` supplies these +snapshots to a synchronous callback without requiring a model client. +Use ``serve_regimes()`` to compose that reader and a Jev worker: + +.. code-block:: python + + from collections.abc import Awaitable, Callable + + import httpx + + from piker.fsp._jev import JevClassification, JevClient + from piker.fsp._regime_feed import serve_regimes + from piker.service import open_piker_runtime + + async def run_live( + fqme: str, + api_key: str, + publish: Callable[[JevClassification], Awaitable[None]], + ) -> None: + async with ( + open_piker_runtime('trend-regimes'), + httpx.AsyncClient() as http, + ): + await serve_regimes( + fqme, + JevClient(http, api_key), + publish, + periods=(1., 5., 15.), + min_interval=1., + max_age=5., + max_source_lag=2., + max_source_age=5., + ) + +Calling this function opens the configured broker feed and sends live +inference requests. Supply an existing Piker market identifier and a +cancellation-cooperative async consumer. The example periods assume +a one-second source; omit ``periods`` to derive 1x/5x/15x periods from +the actual source. Native subsecond source periods are rejected until +the sampler has a compatible clock. A separate correctly timestamped +producer can use ``consume_regime_source()`` directly. + +The worker permits one in-flight request and retains only the newest +pending snapshot. ``min_interval`` limits request start frequency; +``max_age`` limits local submission age, ``max_source_lag`` limits +event-time lag, and ``max_source_age`` filters stale wall-clock source +data before submission. Setting the last option to ``None`` admits +historical replay. Source revision or gap-segment changes revoke old +results. A blocked publication is also cancelled when its evidence +expires or becomes stale. A downstream consumer must still check +provenance on already-delivered messages, which cannot be revoked. + +Current source SHM has no transactional revision word. Copy-time bound +checks detect appends and origin changes; an in-place correction with +unchanged bounds becomes identifiable through its backfill notice. +Replay converges to that corrected source. A publication without a +local task checkpoint is not an atomic cross-process memory transaction. + +Malformed/authentication responses fail explicitly. Transient failures +use bounded retries and freshness checks; the adapter bounds HTTP body +sizes and timeouts. The owning nursery closes the feed, sampler and +in-flight worker on cancellation. Neither the bridge nor the classifier +submits trades. Regime rubrics and probability usefulness still need +evaluation on recorded market data and real provider responses. + +Chart overlays +-------------- + +Preview synthetic multiple-timeframe annotations without a broker, +model request or credentials: + +.. code-block:: xsh + + py313/bin/python examples/trend_graphics.py --output /tmp/trend.png + +The offscreen example accepts ``--periods 1 5 15`` and ``--order 1`` +and isolates its Qt/configuration state from the running application. + +Enable the native order-one reversal and direction overlay in the +existing Piker configuration's ``[ui]`` section:: + + [ui] + trend_calculus = true + +This starts one Trend Calculus writer for the chart's realtime source +and displays confirmed pivot arrows with translucent direction +rectangles. It makes no Jev request and does not read API credentials. +Hover text distinguishes the pivot's bar from the later time when +the reversal became observable. The mutable last FSP row is excluded +from confirmed markers and direction runs. + +Within the existing chart owner's nursery, +``await FspAdmin.open_trend_overlay(order=2)`` attaches another +recursive order while reusing that admin's writer. Optional ``chart`` +and ``chart_period_s`` arguments attach the source evidence to another +price chart. Changing the chart period does not recompute the source +at that period: its actual realtime sampling period stays explicit. + +For independent wall-clock timeframes on the native realtime chart, +configure:: + + [ui] + trend_timeframes = [1, 5, 15] + +This option takes precedence over ``trend_calculus = true``. It uses +one ``MultiTimeframeTrend`` reader and one ``SnapshotTrendOverlay`` +with separate marker/rectangle layers for each period. It opens no Jev +client. ``await FspAdmin.open_multi_timeframe_trends(...)`` exposes the +same integration programmatically and accepts a recursive ``order``. +All periods must be multiples of the actual realtime source period. + +Snapshots preserve interval-end pivot and detection times. Markers +attach to the base bar that contributed each aggregated close. A +bootstrap snapshot contains recent reversals and the latest direction, +so its deterministic rectangle history starts with that observed +direction. It does not reconstruct earlier direction runs from one +snapshot. Skipped complete buckets leave unobserved intervals unpainted. +Source revisions and missing-base-bar segments revoke prior annotations. + +An existing caller-owned client can share this exact evidence with +the model worker inside the chart's nursery: + +.. code-block:: python + + import trio + + from piker.fsp._jev import JevClient, RegimeClassifier + from piker.ui._fsp import FspAdmin + from piker.ui._trend_mtf import SnapshotTrendOverlay + + async def attach_classifier( + admin: FspAdmin, + client: JevClient, + nursery: trio.Nursery, + ) -> tuple[RegimeClassifier, SnapshotTrendOverlay]: + worker: RegimeClassifier = RegimeClassifier(client) + overlay = await admin.open_multi_timeframe_trends( + periods=(1., 5., 15.), + on_snapshot=worker.submit, + on_invalidate=worker.invalidate, + max_source_age=5., + classification_ttl=5., + ) + nursery.start_soon(worker.run, overlay.publish_classification) + return worker, overlay + +Graphics retain their longer deterministic history while the client +projects bounded model evidence. The source-age gate applies to the +optional model callback, so historical chart bootstrap still works. +The adapter checks market, revision, segment, sequence and source lag +before accepting a model response. Classified runs start at receipt +and expire after the configured TTL; they cannot paint future history +before the displayed event clock reaches receipt. Cancelling the chart +owner removes all layers and stops its sampler reader. + +For computations that already publish separate timeframe SHMs, create +one ``piker.ui._trend.TrendOverlay`` per destination SHM with its own +``source_period_s``, target ``chart_period_s`` and ``order``. Mapping +uses timestamps and preserves gaps rather than snapping missing +observations onto the next candle. Geometry counts are bounded and +items are reused instead of adding individual Qt items on every tick. + +Call ``update(force=True)`` after a matching destination or chart +history revision, and ``remove()`` when its owner exits. Normal +``FspAdmin`` integration routes destination-specific revisions and +teardown automatically; direct attachments must retain this ownership. +Hidden widgets preserve revision invalidations until their next +visible update. Repainting follows quotes and the existing one-second +sampler wakeup. If that wakeup precedes asynchronous FSP publication, +a quote-free finalization can appear on the next wakeup, roughly one +second later; the integration does not emit IPC for every output row. + +For model annotations, ``classification_interval()`` converts a +matching ``JevClassification`` into a ``RegimeInterval``. Supply the +chart's current market, source revision, timeframe and display end. +The interval starts no earlier than the model response's receipt. +``overlay.set_regimes(intervals)`` displays an owner-maintained, +bounded collection of these intervals. An empty collection clears +model rectangles; ``None`` restores deterministic direction runs. +The caller remains responsible for response ordering and expiry. + +.. _Piker #79: https://github.com/pikers/piker/issues/79 +.. _Piker #103: https://github.com/pikers/piker/issues/103 +.. _Piker #187: https://github.com/pikers/piker/issues/187 +.. _Piker #325: https://github.com/pikers/piker/issues/325 +.. _Python implementation issue #1: https://github.com/bytesumo/TrendCalculus/issues/1 +.. _Morgan's method document: https://github.com/bytesumo/TrendCalculus/blob/3d5c598cd3764a40633979c4c6f4188191dd81ef/HowToStudyTrends_v1.03.pdf +.. _Original Lua program: https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua +.. _Spark TrendCalculus2: https://github.com/lamastex/spark-trend-calculus/blob/3fb6ef23e985d9d6e0414d3b682afbeef51c52e8/src/main/scala/org/lamastex/spark/trendcalculus/TrendCalculus2.scala +.. _Working mathematical manuscript: https://github.com/lamastex/working-manuscript-trend-calculus/blob/master/main.tex +.. _Spark examples: https://lamastex.github.io/spark-trend-calculus-examples/ +.. _public Pine indicator: https://www.tradingview.com/script/ohSdmhS0-TrendCalculus/ +.. _TypeSafe model documentation: https://docs.typesafe.ai/models +.. _Python SDK documentation: https://docs.typesafe.ai/sdk/python +.. _agent skill: https://docs.typesafe.ai/agent-skill +.. _System One HTTP API: https://docs.typesafe.ai/api diff --git a/examples/benchmark_trend.py b/examples/benchmark_trend.py new file mode 100644 index 00000000..2abf17f8 --- /dev/null +++ b/examples/benchmark_trend.py @@ -0,0 +1,135 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Measure deterministic Trend Calculus processing without a feed. + +Timings distinguish kernel warmup, batch scanning, scalar commits, +and uncommitted live previews. They exclude network/model latency. +Run from the Piker environment; no credentials or services are used. + +''' +import argparse +import json +import platform +from time import perf_counter + +import numba +import numpy as np + +from piker.fsp._trend import ( + TrendCalculator, + TrendHistory, + TrendUpdate, +) + + +def benchmark( + rows: int, + live_rows: int, +) -> dict[str, str|int|float|dict[str, float]]: + ''' + Return reproducible timings and the retained NumPy state size. + + ``rows`` controls the ``TrendCalculator.extend()`` batch size. + ``live_rows`` controls timed calls to ``TrendCalculator.push()`` + and previews made through ``TrendCalculator.copy()``. Assertions + compare directions across all three paths outside timed regions. + + ''' + rng: np.random.Generator = np.random.default_rng(20261004) + values: np.ndarray = rng.normal(size=rows).cumsum() + 100. + times: np.ndarray = np.arange(rows, dtype=float) * .25 + calc: TrendCalculator = TrendCalculator() + start: float = perf_counter() + calc.extend(values[:8], times[:8]) + calc.push(times[8], values[8]) + warmup: float = perf_counter() - start + + start = perf_counter() + full: TrendHistory = TrendCalculator().extend(values, times) + batch: float = perf_counter() - start + calc = TrendCalculator() + elapsed: np.ndarray = np.empty(live_rows) + preview_elapsed: np.ndarray = np.empty(live_rows) + number: int + for number in range(live_rows): + start = perf_counter() + preview: TrendUpdate = calc.copy().push( + times[number], values[number], + ) + preview_elapsed[number] = perf_counter() - start + start = perf_counter() + update: TrendUpdate = calc.push( + times[number], values[number], + ) + elapsed[number] = perf_counter() - start + assert preview.trends == update.trends + assert update.trends == tuple(full.trends[number]) + + def percentiles_us(samples: np.ndarray) -> dict[str, float]: + ''' + Report median and tail operation durations in microseconds. + + ''' + return dict(zip( + ('p50', 'p95', 'p99'), + np.percentile(samples * 1_000_000, (50, 95, 99)), + )) + + array: np.ndarray # noqa: F842 + return { + 'python': platform.python_version(), + 'machine': platform.machine(), + 'numpy': np.__version__, + 'numba': numba.__version__, + 'rows': rows, + 'live_rows': live_rows, + 'orders': len(calc.windows), + 'kernel_warmup_s': warmup, + 'batch_s': batch, + 'batch_rows_per_s': rows / batch, + 'scalar_commit_us': percentiles_us(elapsed), + 'preview_copy_and_push_us': percentiles_us(preview_elapsed), + 'retained_numpy_state_bytes': sum( + array.nbytes for array in ( + calc._sizes, calc._bars, calc._counters, + calc._events, calc._counts, + ) + ), + } + + +def main() -> None: + ''' + Validate CLI sizes and print benchmark measurements as JSON. + + ''' + parser: argparse.ArgumentParser = argparse.ArgumentParser( + description=__doc__, + ) + parser.add_argument('--rows', type=int, default=1_000_000) + parser.add_argument('--live-rows', type=int, default=20_000) + args: argparse.Namespace = parser.parse_args() + if args.rows < 9 or not 1 <= args.live_rows <= args.rows: + parser.error('require rows >= 9 and 1 <= live-rows <= rows') + print(json.dumps(benchmark(args.rows, args.live_rows), indent=2)) + + +if __name__ == '__main__': + main() diff --git a/examples/trend_graphics.py b/examples/trend_graphics.py new file mode 100644 index 00000000..2ea8cabb --- /dev/null +++ b/examples/trend_graphics.py @@ -0,0 +1,235 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Render an explicitly synthetic multiple-timeframe trend PNG. + +Run with ``--output /tmp/piker-trend-graphics.png``. This uses the +actual shared tracker and snapshot graphics adapter. It launches +no visible GUI, opens no broker or Jev connection, reads no API key, +and isolates Qt/Piker configuration in a disposable directory. + +''' +import argparse +import json +import os +from pathlib import Path +from tempfile import TemporaryDirectory +from types import SimpleNamespace + + +def render( + output: Path, + *, + periods: tuple[float, ...], + order: int, + samples: int, + +) -> dict[str, object]: + ''' + Replay synthetic closes through one actual tracker and renderer. + + The final OHLC row remains mutable and is never submitted as a + completed observation. Each timeframe retains real aggregation + and TC state, while all plotted marker timestamps identify their + final contributing base bar. Shading is deterministic TC output, + not fabricated model inference. + + ''' + import numpy as np + import pyqtgraph as pg + + from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot + from piker.ui._trend_mtf import ( + SnapshotTrendOverlay, + _SnapshotLayer, + ) + from piker.ui.qt import QApplication + + app: QApplication = QApplication(['piker-trend-graphics']) + app.setQuitOnLastWindowClosed(False) + steps: np.ndarray = np.arange(samples) + prices: np.ndarray = ( + 100 + .009 * steps + + 1.5 * np.sin(steps * .06) + + .38 * np.sin(steps * .43) + ) + rows: np.ndarray = np.empty(samples, dtype=[ + ('index', 'i8'), + ('time', 'f8'), + ('close', 'f8'), + ('low', 'f8'), + ('high', 'f8'), + ]) + rows['index'] = 1000 + steps + rows['time'] = 1_700_000_100 + steps + rows['close'] = prices + rows['low'] = prices - .12 + rows['high'] = prices + .12 + shm: SimpleNamespace = SimpleNamespace( + array=rows, + _token='synthetic-demo-ohlcv', + ) + + widget: pg.PlotWidget = pg.PlotWidget() + widget.resize(1400, 760) + widget.setBackground('#101923') + widget.setTitle( + 'SYNTHETIC: Trend Calculus across actual timeframes' + '
' + 'Larger triangles represent slower sampling periods; ' + 'shading begins at confirmed regime detection.' + '', + color='#dce5ed', + size='16pt', + ) + widget.plotItem.titleLabel.setFixedHeight(60) + widget.showGrid(x=True, y=True, alpha=.13) + widget.setLabel('left', 'Synthetic price') + widget.setLabel('bottom', 'Absolute base-bar index (1 second)') + widget.plot( + rows['index'], + prices, + pen=pg.mkPen('#dce5ed', width=1.5), + ) + left: int = max(0, samples - 400) + widget.setXRange(1000 + left, 1000 + samples, padding=0) + widget.setYRange( + float(prices[left:].min()) - .6, + float(prices[left:].max()) + .6, + padding=0, + ) + viz: SimpleNamespace = SimpleNamespace( + plot=widget.plotItem, + shm=shm, + index_field='index', + ) + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.demo', + periods=periods, + windows=(2, 2, 2, 2), + history_limit=256, + ) + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + viz, + shm, + fqme='synthetic.demo', + base_period=1, + periods=periods, + order=order, + ) + try: + row: np.void + snapshot: RegimeSnapshot + for row in rows[:-1]: + snapshot = tracker.push( + float(row['time']), + float(row['close']), + index=int(row['index']), + ) + adapter.submit(snapshot) + + legend: pg.LegendItem = widget.addLegend(offset=(12, 12)) + period: float + layer: _SnapshotLayer + for period, layer in adapter.layers.items(): + legend.addItem( + pg.ScatterPlotItem( + symbol='t1', + size=layer.markers.opts['size'], + brush='#46c7a4', + pen=pg.mkPen(None), + ), + f'{period:g}s / order {order}', + ) + widget.show() + app.processEvents() + output.parent.mkdir(parents=True, exist_ok=True) + if not widget.grab().save(str(output)): + raise RuntimeError('Qt could not save the requested PNG') + return { + 'mode': 'synthetic_native_graphics_no_model_calls', + 'output': str(output), + 'base_period': 1, + 'periods': periods, + 'order': order, + 'finalized_bars': samples - 1, + 'last_event_end': snapshot.as_of, + 'layers': { + period: { + 'markers': len(layer.points), + 'rectangles': len(layer.rectangles.intervals), + } + for period, layer in adapter.layers.items() + }, + } + finally: + adapter.remove() + widget.close() + widget.deleteLater() + app.processEvents() + + +def main() -> None: + ''' + Require an output path and isolate offscreen configuration. + + ''' + parser: argparse.ArgumentParser = argparse.ArgumentParser( + description=__doc__, + ) + parser.add_argument('--output', type=Path, required=True) + parser.add_argument( + '--periods', nargs='+', type=float, default=[1, 5, 15], + ) + parser.add_argument( + '--order', type=int, choices=range(1, 5), default=1, + ) + parser.add_argument('--samples', type=int, default=600) + args: argparse.Namespace = parser.parse_args() + if not 120 <= args.samples <= 10_000: + parser.error('--samples must be between 120 and 10000') + + prior: dict[str, str|None] = { + 'QT_QPA_PLATFORM': os.environ.get('QT_QPA_PLATFORM'), + 'XDG_CONFIG_HOME': os.environ.get('XDG_CONFIG_HOME'), + } + try: + path: str + with TemporaryDirectory(prefix='piker-trend-') as path: + os.environ['QT_QPA_PLATFORM'] = 'offscreen' + os.environ['XDG_CONFIG_HOME'] = path + result: dict[str, object] = render( + args.output.resolve(), + periods=tuple(args.periods), + order=args.order, + samples=args.samples, + ) + print(json.dumps(result, indent=2)) + finally: + key: str + value: str|None + for key, value in prior.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + + +if __name__ == '__main__': + main() diff --git a/examples/trend_regimes.py b/examples/trend_regimes.py new file mode 100644 index 00000000..6797da86 --- /dev/null +++ b/examples/trend_regimes.py @@ -0,0 +1,165 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Inspect synthetic timeframe evidence and optional Jev inference. + +The default prints the exact request without reading credentials or +opening a network connection. ``--mock`` tests the HTTP path locally. +``--live`` sends exactly one synthetic snapshot using the explicitly +configured ``TYPESAFE_API_KEY``. No broker connection is necessary. + +''' +import argparse +from dataclasses import asdict +import json +from math import floor, sin +import os +import time + +import httpx +import trio + +from piker.fsp._jev import ( + DEFAULT_MODEL, + JevClassification, + JevClient, + build_regime_request, +) +from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot + + +def synthetic_snapshot() -> RegimeSnapshot: + ''' + Aggregate quarter-second closes ending before the current time. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.demo', + base_period=.25, + periods=(.25, 1, 5), + windows=(2, 2, 2), + ) + count: int = 960 + start: float = floor(time.time()) - count * .25 + number: int + for number in range(count): + price: float = ( + 100 + number * .01 + + 2 * sin(number / 12) + + .6 * sin(number / 3) + ) + snapshot: RegimeSnapshot = tracker.push( + start + number * .25, price, + ) + return snapshot + + +def mock_reply(request: httpx.Request) -> httpx.Response: + ''' + Demonstrate transport and parsing with explicitly fake labels. + + ''' + payload: dict = json.loads(request.content) + answers: dict[str, dict] = {} + name: str + question: dict + for name, question in payload['questions'].items(): + options: dict = question['criteria'] + probabilities: dict[str, float] = {} + option: str + for option in options: + probabilities[option] = float(option == 'unknown') + answers[name] = { + 'type': 'choice', + 'choice': 'unknown', + 'probabilities': probabilities, + 'confidence': 1.0, + } + return httpx.Response(200, json={ + 'model': payload['model'], + 'answers': answers, + 'usage': { + 'input_tokens': 0, + 'output_tokens': 0, + }, + }) + + +async def classify( + snapshot: RegimeSnapshot, + model: str, + live: bool, +) -> None: + ''' + Print one explicitly live or simulated result with provenance. + + ''' + if live: + key: str = os.environ.get('TYPESAFE_API_KEY', '') + if not key: + raise SystemExit('Set TYPESAFE_API_KEY before --live') + transport: httpx.AsyncBaseTransport|None = None + else: + key = 'offline-demo-key' + transport = httpx.MockTransport(mock_reply) + http: httpx.AsyncClient + async with httpx.AsyncClient(transport=transport) as http: + client: JevClient = JevClient(http, key, model=model) + result: JevClassification = await client.classify(snapshot) + output: dict = { + 'mode': 'live_synthetic' if live else 'mock_not_inference', + 'classification': asdict(result), + } + print(json.dumps(output, indent=2, allow_nan=False)) + + +def main() -> None: + ''' + Require an explicit flag before any credential or network access. + + ''' + parser: argparse.ArgumentParser = argparse.ArgumentParser( + description=__doc__, + ) + mode: argparse._MutuallyExclusiveGroup = ( + parser.add_mutually_exclusive_group() + ) + mode.add_argument( + '--live', action='store_true', + help='Send one synthetic TypeSafe request (uses API credit)', + ) + mode.add_argument( + '--mock', action='store_true', + help='Exercise the adapter locally using simulated answers', + ) + parser.add_argument('--model', default=DEFAULT_MODEL) + args: argparse.Namespace = parser.parse_args() + snapshot: RegimeSnapshot = synthetic_snapshot() + if args.live or args.mock: + trio.run(classify, snapshot, args.model, args.live) + else: + print(json.dumps( + build_regime_request(snapshot, model=args.model), + indent=2, + allow_nan=False, + )) + + +if __name__ == '__main__': + main() diff --git a/piker/data/_sampling.py b/piker/data/_sampling.py index 83afe321..f2dcb684 100644 --- a/piker/data/_sampling.py +++ b/piker/data/_sampling.py @@ -202,6 +202,12 @@ class Sampler: if shm_period_s <= 1: next_t = i_epoch + # Startup may seed a bar one second + # ahead; rounded wall time can repeat or + # move backward. Keep the mutable bar + # until a strictly newer epoch arrives. + if next_t <= t: + continue # this copies non-std fields (eg. vwap) from the # last datum diff --git a/piker/fsp/__init__.py b/piker/fsp/__init__.py index 0651069e..27bcc8fc 100644 --- a/piker/fsp/__init__.py +++ b/piker/fsp/__init__.py @@ -35,6 +35,14 @@ from ._volume import ( flow_rates, tina_vwap, ) +from ._trend import ( + TrendCalculator as TrendCalculator, + TrendHistory as TrendHistory, + TrendUpdate as TrendUpdate, +) +from ._trend_fsp import ( + trend_calculus as trend_calculus, +) __all__: list[str] = [ 'cascade', @@ -44,6 +52,10 @@ __all__: list[str] = [ 'dolla_vlm', 'flow_rates', 'tina_vwap', + 'TrendCalculator', + 'TrendHistory', + 'TrendUpdate', + 'trend_calculus', ] diff --git a/piker/fsp/_api.py b/piker/fsp/_api.py index 6b8d7332..16ebe615 100644 --- a/piker/fsp/_api.py +++ b/piker/fsp/_api.py @@ -28,6 +28,7 @@ from __future__ import annotations from typing import ( AsyncGenerator, Callable, + NamedTuple, overload, Protocol, ) @@ -48,7 +49,25 @@ log = get_logger(__name__) type FspHistory = dict[str, np.ndarray|None]|np.ndarray type RealtimeValue = np.ndarray|np.number|int|float -type FspYield = FspHistory|tuple[str, RealtimeValue] + + +class FspSample(NamedTuple): + ''' + Publish all FSP fields at an absolute source coordinate. + + Unlike a field-only tick update, this record remains valid if the + sampler has not appended its destination row yet. + ``FspSample.fields`` are written without task checkpoints. + This is not an atomic memory transaction for readers in other + processes. + + ''' + index: int + time: float + fields: dict[str, RealtimeValue] + + +type FspYield = FspHistory|tuple[str, RealtimeValue]|FspSample type FspStream = AsyncGenerator[FspYield, None] type FspConfigValue = str|bool|int|float @@ -76,6 +95,7 @@ def _load_builtins() -> dict[NamespacePath, Fsp]: # import to implicity trigger registration via ``@fsp`` from . import _momo # noqa from . import _volume # noqa + from . import _trend_fsp # noqa return _fsp_registry @@ -108,6 +128,8 @@ class Fsp[**P]: *, outputs: tuple[str, ...] = (), display_name: str|None = None, + sample_on_step: bool = False, + event_outputs: tuple[str, ...] = (), **config: FspConfigValue, ) -> None: @@ -120,6 +142,12 @@ class Fsp[**P]: self.ns_path: NamespacePath = NamespacePath.from_ref(func) self.outputs = outputs + # Snapshot processors read SHM on both quote and timer steps. + # Sparse event fields must never be carried into a new row. + self.sample_on_step: bool = sample_on_step + if not set(event_outputs).issubset(outputs): + raise ValueError('event outputs must be declared') + self.event_outputs: tuple[str, ...] = event_outputs self.config: dict[str, FspConfigValue] = config # register with declared set. @@ -181,6 +209,8 @@ def fsp[**P]( *, outputs: tuple[str, ...] = (), display_name: str|None = None, + sample_on_step: bool = False, + event_outputs: tuple[str, ...] = (), **config: FspConfigValue, ) -> Fsp[P]: @@ -193,6 +223,8 @@ def fsp[**P]( *, outputs: tuple[str, ...] = (), display_name: str|None = None, + sample_on_step: bool = False, + event_outputs: tuple[str, ...] = (), **config: FspConfigValue, ) -> Callable[[FspFunc[P]], Fsp[P]]: @@ -204,6 +236,8 @@ def fsp[**P]( *, outputs: tuple[str, ...] = (), display_name: str|None = None, + sample_on_step: bool = False, + event_outputs: tuple[str, ...] = (), **config: FspConfigValue, ) -> Fsp[P]|Callable[[FspFunc[P]], Fsp[P]]: @@ -214,6 +248,10 @@ def fsp[**P]( ``Fsp``. Parameterized ``@fsp(...)`` returns ``decorate`` first; Python then passes the decorated function to that closure. + ``sample_on_step`` opts into coalesced SHM snapshot wakeups from + quotes and sampler steps; the quote payload is not preserved. + ``event_outputs`` names sparse fields cleared to NaN on append. + ''' if wrapped is None: @@ -222,12 +260,21 @@ def fsp[**P]( func, outputs=outputs, display_name=display_name, + sample_on_step=sample_on_step, + event_outputs=event_outputs, **config, ) return decorate - return Fsp(wrapped, outputs=(wrapped.__name__,)) + return Fsp( + wrapped, + outputs=outputs or (wrapped.__name__,), + display_name=display_name, + sample_on_step=sample_on_step, + event_outputs=event_outputs, + **config, + ) def maybe_mk_fsp_shm( diff --git a/piker/fsp/_engine.py b/piker/fsp/_engine.py index 3218fce3..3db7d1d3 100644 --- a/piker/fsp/_engine.py +++ b/piker/fsp/_engine.py @@ -35,6 +35,7 @@ import trio from trio_typing import TaskStatus import tractor from tractor.msg import NamespacePath +from tractor.trionics import Lagged from piker.types import Struct from ..log import ( @@ -56,6 +57,7 @@ from ._api import ( FspStream, FspYield, FspHistory, + FspSample, _load_builtins, NDToken, ) @@ -145,6 +147,88 @@ async def filter_quotes_by_sym( yield quote +@acm +async def _sample_wakeups( + quotes: AsyncIterator[QuoteFrame], + fqme: str, + enabled: bool, +) -> AsyncIterator[ + tuple[AsyncIterator[QuoteFrame], Callable[[], None]|None] +]: + ''' + Merge quote and sampler wakeups for SHM snapshot processors. + + One pending notification is enough: the processor reads every + newly finalized SHM row when woken. Keep this queue outside the + restartable compute task, so history resync cannot lose a step. + Tick-sensitive processors retain their original quote stream. + + ''' + if not enabled: + yield quotes, None + return + + send: trio.MemorySendChannel[QuoteFrame] + receive: trio.MemoryReceiveChannel[QuoteFrame] + send, receive = trio.open_memory_channel[QuoteFrame](1) + + def notify() -> None: + ''' + Coalesce sampler steps in the local ``send`` channel. + + ''' + try: + send.send_nowait({fqme: {'ticks': []}}) + except trio.WouldBlock: + pass + + async def relay() -> None: + ''' + Resume after quote overrun and wake the SHM snapshot reader. + + ''' + iterator: AsyncIterator[QuoteFrame] = aiter(quotes) + while True: + try: + frame: QuoteFrame = await anext(iterator) + except Lagged: + # Dropped quote payloads do not invalidate SHM. + # Wake the snapshot processor to catch up all rows. + notify() + await trio.lowlevel.checkpoint() + continue + except StopAsyncIteration: + return + if fqme in frame: + notify() + + nursery: trio.Nursery + async with send, receive, trio.open_nursery() as nursery: + nursery.start_soon(relay) + try: + yield receive, notify + finally: + nursery.cancel_scope.cancel() + + +def _next_step_output( + previous: np.ndarray, + fsp: Fsp, + zero_on_step: bool, +) -> np.ndarray: + ''' + Carry state into a new row without duplicating sparse events. + + ''' + row: np.ndarray = ( + np.zeros_like(previous) if zero_on_step else previous.copy() + ) + field: str + for field in fsp.event_outputs: + row[field] = np.nan + return row + + class _HistoryShm: ''' Present an immutable history view before switching to live SHM. @@ -274,6 +358,7 @@ class Cascade(Struct): complete: trio.Event|None = None cs: trio.CancelScope|None = None client_stream: tractor.MsgStream|None = None + sample_ready: trio.Event|None = None async def resync(self) -> int: # TODO: adopt an incremental update engine/approach @@ -380,7 +465,16 @@ class Cascade(Struct): ) -> int: self.bind_func = bind_func index = await self.tn.start(bind_func) - yield index + try: + yield index + finally: + # Stop the current compute task, including after resync, + # before the outer scope closes its source receiver. + if self.cs is not None: + self.cs.cancel() + if self.complete is not None: + with trio.CancelScope(shield=True): + await self.complete.wait() # TODO: what do we want on teardown/error? # -[ ] dynamic reconnection after update? @@ -390,6 +484,39 @@ async def _close_fsp_stream(out_stream: FspStream) -> None: await out_stream.aclose() +async def _write_sample( + casc: Cascade, + sample: FspSample, +) -> bool: + ''' + Wait for destination advancement, then publish at source index. + + ``cascade()`` sets ``Cascade.sample_ready`` after publishing the + new destination bound. Capture that event before checking bounds + so a sampler wakeup cannot be lost. Cancellation during history + resync aborts this wait with the obsolete compute generator. + + ''' + dst: ShmArray = casc.dst.rt_shm + while True: + ready: trio.Event|None = casc.sample_ready + if ready is None: + ready = casc.sample_ready = trio.Event() + first: int = dst._first.value + last: int = dst._last.value + if sample.index < first: + return False + if sample.index < last: + row: np.void = dst._array[sample.index] + field: str + value: Any + for field, value in sample.fields.items(): + row[field] = value + row['time'] = sample.time + return True + await ready.wait() + + def _write_history_output( dst_shm: ShmArray, history_output: FspHistory, @@ -577,6 +704,9 @@ async def connect_streams( async for processed in out_stream: log.debug(f"{func_name}: {processed}") + if isinstance(processed, FspSample): + await _write_sample(casc, processed) + continue key, output = processed # dst.array[-1][key] = output dst_shm.array[[key, 'time']][-1] = ( @@ -711,6 +841,10 @@ async def cascade( ) as feed: flume: Flume = feed.flumes[fqme] + # Cached feed consumers can replace ``Flume.stream`` when + # opening another subscription. Retain this scope's receiver + # before nursery setup gives another consumer a checkpoint. + quote_source: AsyncIterator[QuoteFrame] = flume.stream # XXX: can't do this since flume.feed will be set XD # assert flume == src assert flume.mkt == src.mkt @@ -728,9 +862,17 @@ async def cascade( profiler(f'{func}: feed up') func_name: str = func.__name__ + tn: trio.Nursery + quote_stream: AsyncIterator[QuoteFrame] + wake_sample: Callable[[], None]|None async with ( tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console trio.open_nursery() as tn, + _sample_wakeups( + quote_source, + fqme, + fsp.sample_on_step, + ) as (quote_stream, wake_sample), ): # TODO: might be better to just make a "restart" method where # the target task is spawned implicitly and then the event is @@ -748,7 +890,7 @@ async def cascade( connect_streams, casc=casc, mkt=mkt, - quote_stream=flume.stream, + quote_stream=quote_stream, # flumes and shm passthrough src=src, @@ -777,10 +919,6 @@ async def cascade( dst_shm: ShmArray = dst.rt_shm src_shm: ShmArray = src.rt_shm - if zero_on_step: - last = dst.rt_shm.array[-1:] - zeroed = np.zeros(last.shape, dtype=last.dtype) - profiler(f'{func_name}: fsp up') # sync to client-side actor @@ -842,10 +980,9 @@ async def cascade( # some metrics like vlm should be reset # to zero every step. - if zero_on_step: - last = zeroed - else: - last = array[-1:].copy() + last = _next_step_output( + array[-1:], fsp, zero_on_step, + ) dst.rt_shm.push(last) @@ -860,6 +997,13 @@ async def cascade( dst_shm._array['time'][src_li] = src_lt dst_shm._array['time'][src_2li] = src_2lt + ready: trio.Event|None = casc.sample_ready + casc.sample_ready = trio.Event() + if ready is not None: + ready.set() + if wake_sample is not None: + wake_sample() + # last2 = dst.array[-2:] # if ( # last2[-1]['index'] != src_li diff --git a/piker/fsp/_jev.py b/piker/fsp/_jev.py new file mode 100644 index 00000000..f6d37a90 --- /dev/null +++ b/piker/fsp/_jev.py @@ -0,0 +1,777 @@ +# piker: trading gear for hackers +# Copyright (C) Tyler Goodlet (in stewardship of piker0) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Optional, bounded TypeSafe System One judgments over trend evidence. + +The HTTP contract is documented at https://docs.typesafe.ai/api . +``JevClient`` uses piker's existing HTTPX/Trio stack. Credentials are +explicit constructor inputs; importing this module opens no network +connection and reads no environment variables. Model judgments never +change deterministic TC state or route orders into the EMS. + +''' +from collections.abc import Awaitable, Callable +from dataclasses import dataclass, replace +from email.utils import parsedate_to_datetime +import json +from math import isclose, isfinite +from operator import index as integer_index +import time +from types import MappingProxyType + +import httpx +import numpy as np +import trio + +from ._regime import RegimeSnapshot, TimeframeRegime + + +DEFAULT_MODEL: str = 'jev-1.13.0' +PROMPT_VERSION: str = 'piker-regimes-v1' +API_URL: str = 'https://api.typesafe.ai/v1/systemone' + +# Centralized human-reviewable rubric. These are descriptive model +# judgments, not new TC rules or validated profitability signals. +REGIME_CRITERIA: MappingProxyType = MappingProxyType({ + 'uptrend': ( + 'Persistent upward movement supported by recent closes and ' + 'ready trend orders; small counter-moves do not dominate.' + ), + 'downtrend': ( + 'Persistent downward movement supported by recent closes ' + 'and ready trend orders; small retracements do not dominate.' + ), + 'range': ( + 'Recent movement repeatedly retraces within a bounded range ' + 'without a persistent direction, or closes remain flat.' + ), + 'transition': ( + 'Recent detected reversals and conflicting ready orders ' + 'suggest a change of regime whose direction is unsettled.' + ), + 'unknown': ( + 'Available ready orders and recent observations provide ' + 'insufficient or contradictory evidence for another label.' + ), +}) +ALIGNMENT_CRITERIA: MappingProxyType = MappingProxyType({ + 'aligned_up': 'Ready timeframes consistently support an uptrend', + 'aligned_down': ( + 'Ready timeframes consistently support a downtrend.' + ), + 'mixed': ( + 'Ready timeframes disagree in direction or regime, such as ' + 'a shorter pullback within a longer directional trend.' + ), + 'unknown': ( + 'There is insufficient directional evidence to assess ' + 'agreement across the ready timeframes.' + ), +}) + + +@dataclass(frozen=True) +class RegimeAnswer: + ''' + Retain a model label and its entire probability distribution. + + ``RegimeAnswer.period=None`` identifies timeframe alignment. + ``RegimeAnswer.probability`` is the selected label's probability; + ``RegimeAnswer.confidence`` separately describes concentration. + + ''' + question: str + period: float|None + label: str + probability: float + confidence: float + probabilities: tuple[tuple[str, float], ...] + + +@dataclass(frozen=True) +class JevClassification: + ''' + Associate semantic judgments with evidence and availability time. + + ``JevClassification.source_as_of`` is an event-time bar END. + ``JevClassification.requested_at`` and + ``JevClassification.received_at`` are Unix wall-clock timestamps. + A live chart must not start a classification before receipt. + ``JevClassification.latency`` uses a monotonic clock; wall-clock + adjustments do not change the measured request duration. + + ''' + market: str + revision: int + sequence: int + segment: int + source_as_of: float + requested_at: float + received_at: float + latency: float + model: str + prompt_version: str + answers: tuple[RegimeAnswer, ...] + input_tokens: int + output_tokens: int + + +class JevError(Exception): + ''' + Describe a provider failure without echoing credential values. + + ''' + + +class JevProtocolError(JevError): + ''' + Reject malformed or semantically inconsistent provider answers. + + ''' + + +class JevServiceError(JevError): + ''' + Preserve an HTTP status and a bounded retry delay, not its body. + + ''' + def __init__( + self, + status: int, + retry_after: float = 0, + ) -> None: + ''' + Retain an HTTP failure status without its response body. + + ''' + self.status: int = status + self.retry_after: float = retry_after + super().__init__(f'TypeSafe request failed: HTTP {status}') + + @property + def retryable(self) -> bool: + ''' + Identify transient statuses documented by HTTP and TypeSafe. + + ''' + return self.status in {408, 429, 500, 502, 503, 504, 529} + + +def build_regime_request( + snapshot: RegimeSnapshot, + *, + model: str = DEFAULT_MODEL, + evidence_limit: int = 16, +) -> dict: + ''' + Batch independent timeframe questions over one immutable state. + + Question identifiers are opaque to Jev; each instruction names + its actual state path. Unready timeframes provide context only. + ``evidence_limit`` bounds each timeframe's model-input tails + independently of larger histories retained for chart rendering. + + ''' + if not isinstance(model, str) or not model.strip(): + raise ValueError('a model identifier is required') + if ( + isinstance(evidence_limit, (bool, np.bool_)) + or not 1 <= integer_index(evidence_limit) <= 256 + ): + raise ValueError('evidence_limit must be in [1, 256]') + frames: list[TimeframeRegime] = [] + frame: TimeframeRegime + for frame in snapshot.timeframes: + frames.append(replace( + frame, + recent_closes=frame.recent_closes[-evidence_limit:], + reversals=frame.reversals[-evidence_limit:], + )) + evidence: RegimeSnapshot = replace( + snapshot, timeframes=tuple(frames), + ) + questions: dict[str, dict] = {} + number: int + for number, frame in enumerate(snapshot.timeframes): + if not frame.ready or not frame.ready[0]: + continue + questions[f'timeframe_{number}'] = { + 'type': 'choice', + 'instructions': ( + f'Which descriptive market regime best fits ' + f'`timeframes[{number}]` at its `as_of` time? ' + f'Use only its observed recent closes, ready trend ' + f'orders and detected reversals. TC order numbers ' + f'are recursive reversal scales, not seconds. ' + f'Zero is unresolved, not automatically a range. ' + f'Ignore unready orders. A pivot was only known at ' + f'its detected_time. Do not predict future returns ' + f'or make a trading recommendation.' + ), + 'criteria': dict(REGIME_CRITERIA), + } + if not questions: + raise ValueError('no timeframe has completed TC warm-up') + if len(questions) > 1: + questions['alignment'] = { + 'type': 'choice', + 'instructions': ( + 'How do directions across the ready entries in ' + '`timeframes` agree at `as_of`? Use only observed ' + 'closes and ready TC orders. Each timeframe may end ' + 'earlier than the overall as_of; do not invent ' + 'partial buckets or rely on other question answers.' + ), + 'criteria': dict(ALIGNMENT_CRITERIA), + } + return { + 'model': model, + 'state': evidence.to_state(), + 'questions': questions, + } + + +def _probability(value: object) -> float: + ''' + Enforce finite unit-interval probabilities without coercing text. + + ''' + if ( + isinstance(value, bool) + or not isinstance(value, (float, int)) + or not isfinite(value) + or not 0 <= value <= 1 + ): + raise JevProtocolError('invalid probability or confidence') + return float(value) + + +def _retry_after(value: str|None) -> float: + ''' + Interpret Retry-After seconds or an HTTP date with a hard bound. + + ''' + if value is None: + return 0 + try: + delay: float = float(value) + except ValueError: + try: + delay = parsedate_to_datetime(value).timestamp() + delay -= time.time() + except ( + TypeError, + ValueError, + OverflowError, + ): + return 0 + if not isfinite(delay): + return 0 + return min(60, max(0, delay)) + + +class JevClient: + ''' + Perform one bounded, validated HTTP evaluation on the Trio loop. + + The caller owns ``http`` and must close it. Retries and pacing + belong to ``RegimeClassifier``, so a single evaluation never + silently resends stale state. Redirects are disabled to keep + the explicit credential bound to the TypeSafe API origin. + + ''' + def __init__( + self, + http: httpx.AsyncClient, + api_key: str, + *, + model: str = DEFAULT_MODEL, + timeout: float = 5, + max_request_bytes: int = 65_536, + max_response_bytes: int = 131_072, + evidence_limit: int = 16, + ) -> None: + ''' + Bind a caller-owned HTTP session, key and request limits. + + The caller retains ownership of the supplied ``http`` client. + + ''' + if not isinstance(api_key, str) or not api_key.strip(): + raise ValueError('explicit TypeSafe API key required') + if not isfinite(timeout) or timeout <= 0: + raise ValueError('timeout must be positive and finite') + if max_request_bytes <= 0 or max_response_bytes <= 0: + raise ValueError('byte limits must be positive') + if not isinstance(model, str) or not model.strip(): + raise ValueError('a model identifier is required') + if ( + isinstance(evidence_limit, (bool, np.bool_)) + or not 1 <= integer_index(evidence_limit) <= 256 + ): + raise ValueError('evidence_limit must be in [1, 256]') + self._http: httpx.AsyncClient = http + self._api_key: str = api_key + self.model: str = model + self.timeout: float = timeout + self.max_request_bytes: int = max_request_bytes + self.max_response_bytes: int = max_response_bytes + self.evidence_limit: int = int(evidence_limit) + + async def classify( + self, + snapshot: RegimeSnapshot, + ) -> JevClassification: + ''' + Validate one complete response against the exact sent rubric. + + ''' + payload: dict = build_regime_request( + snapshot, model=self.model, + evidence_limit=self.evidence_limit, + ) + body: bytes = json.dumps( + payload, + allow_nan=False, + separators=(',', ':'), + ).encode() + if len(body) > self.max_request_bytes: + raise ValueError('regime request exceeds byte budget') + requested_at: float = time.time() + started: float = trio.current_time() + data: bytearray = bytearray() + api_key: str = self._api_key + response: httpx.Response + with trio.fail_after(self.timeout): + async with self._http.stream( + 'POST', + API_URL, + content=body, + headers={ + 'Authorization': f'Bearer {api_key}', + 'Content-Type': 'application/json', + }, + timeout=self.timeout, + follow_redirects=False, + ) as response: + if response.status_code != 200: + raise JevServiceError( + response.status_code, + _retry_after( + response.headers.get('retry-after'), + ), + ) + chunk: bytes + async for chunk in response.aiter_bytes(): + data.extend(chunk) + if len(data) > self.max_response_bytes: + raise JevProtocolError( + 'response exceeds the byte budget', + ) + received_at: float = time.time() + latency: float = trio.current_time() - started + error: ValueError|UnicodeError + try: + decoded: object = json.loads(data) + except ( + ValueError, + UnicodeError, + ) as error: + raise JevProtocolError('invalid JSON reply') from error + return self._parse( + snapshot, + payload, + decoded, + requested_at, + received_at, + latency, + ) + + def _parse( + self, + snapshot: RegimeSnapshot, + payload: dict, + decoded: object, + requested_at: float, + received_at: float, + latency: float, + ) -> JevClassification: + ''' + Reject missing questions, bad labels, or bad distributions. + + ''' + if not isinstance(decoded, dict): + raise JevProtocolError('response must be an object') + model: object = decoded.get('model') + if not isinstance(model, str) or not model: + raise JevProtocolError('response model is missing') + if ( + self.model not in {'jev-latest', 'jev-preview'} + and model != self.model + ): + raise JevProtocolError('response violates pinned model') + raw_answers: object = decoded.get('answers') + questions: dict = payload['questions'] + if ( + not isinstance(raw_answers, dict) + or set(raw_answers) != set(questions) + ): + raise JevProtocolError('response question set differs') + answers: list[RegimeAnswer] = [] + name: str + question: dict + for name, question in questions.items(): + raw: object = raw_answers[name] + if ( + not isinstance(raw, dict) + or raw.get('type') != 'choice' + ): + raise JevProtocolError('expected a Choice answer') + label: object = raw.get('choice') + probabilities: object = raw.get('probabilities') + if ( + not isinstance(probabilities, dict) + or set(probabilities) != set(question['criteria']) + or not isinstance(label, str) + or label not in probabilities + ): + raise JevProtocolError('answer options differ') + distribution: dict[str, float] = {} + option: str + probability: object + for option, probability in probabilities.items(): + distribution[option] = _probability(probability) + if not isclose( + sum(distribution.values()), 1, abs_tol=1e-5, + ): + raise JevProtocolError('probabilities must sum to 1') + if distribution[label] < max(distribution.values()): + raise JevProtocolError('choice is not most probable') + period: float|None = None + if name != 'alignment': + position: int = int(name.removeprefix('timeframe_')) + period = snapshot.timeframes[position].period + answers.append(RegimeAnswer( + question=name, + period=period, + label=label, + probability=distribution[label], + confidence=_probability(raw.get('confidence')), + probabilities=tuple(distribution.items()), + )) + usage: object = decoded.get('usage') + if not isinstance(usage, dict): + raise JevProtocolError('token usage is missing') + key: str + for key in ('input_tokens', 'output_tokens'): + count: object = usage.get(key) + if isinstance(count, bool) or not isinstance(count, int): + raise JevProtocolError('token usage must be integer') + if count < 0: + raise JevProtocolError('token usage must be >= 0') + return JevClassification( + market=snapshot.market, + revision=snapshot.revision, + sequence=snapshot.sequence, + segment=snapshot.segment, + source_as_of=snapshot.as_of, + requested_at=requested_at, + received_at=received_at, + latency=latency, + model=model, + prompt_version=PROMPT_VERSION, + answers=tuple(answers), + input_tokens=usage['input_tokens'], + output_tokens=usage['output_tokens'], + ) + + +@dataclass(frozen=True) +class _Submission: + ''' + Retain one snapshot and its monotonic local submission time. + + ''' + snapshot: RegimeSnapshot + submitted: float + + +class RegimeClassifier: + ''' + Coalesce producer updates into one paced, single-flight worker. + + ``RegimeClassifier.submit()`` replaces + ``RegimeClassifier._pending`` without awaiting HTTP or consumers. + Responses must share the current revision/segment and remain + within ``RegimeClassifier.max_source_lag`` event-time seconds and + ``RegimeClassifier.max_age`` local seconds of their evidence. + A small nonzero lag permits useful inference on fast feeds; + zero requires the response's exact latest event-time snapshot. + + Run one worker in the owning actor's nursery. Cancellation closes + in-flight I/O through HTTPX; the caller owns the HTTP session. + + ''' + def __init__( + self, + client: JevClient, + *, + min_interval: float = 1, + max_age: float = 5, + max_source_lag: float = 2, + max_retries: int = 2, + ) -> None: + ''' + Configure bounded request pacing and evidence freshness. + + ''' + value: float + for value in (min_interval, max_source_lag): + if not isfinite(value) or value < 0: + raise ValueError('interval and lag must be >= 0') + if not isfinite(max_age) or max_age <= 0: + raise ValueError('max_age must be positive and finite') + if ( + isinstance(max_retries, bool) + or not 0 <= integer_index(max_retries) <= 8 + ): + raise ValueError('max_retries must be in [0, 8]') + self.client: JevClient = client + self.min_interval: float = min_interval + self.max_age: float = max_age + self.max_source_lag: float = max_source_lag + self.max_retries: int = max_retries + self.requests: int = 0 + self.published: int = 0 + self.discarded: int = 0 + self.failures: int = 0 + self.last_error: Exception|None = None + self._latest: _Submission|None = None + self._pending: _Submission|None = None + self._wake: trio.Event = trio.Event() + self._running: bool = False + self._minimum_revision: int = 0 + self._publishing: _Submission|None = None + self._publish_scope: trio.CancelScope|None = None + + def invalidate(self, revision: int) -> None: + ''' + Revoke older evidence immediately, before source replay. + + A feed owner calls this upon receiving a history revision; + it need not wait for a replacement snapshot to become ready. + + ''' + if ( + isinstance(revision, bool) + or integer_index(revision) < 0 + ): + raise ValueError('revision must be an integer >= 0') + self._minimum_revision = max( + self._minimum_revision, revision, + ) + if ( + self._pending is not None + and self._pending.snapshot.revision < revision + ): + self._pending = None + self._cancel_stale_publication() + + def _cancel_stale_publication(self) -> None: + ''' + Revoke stale ``RegimeClassifier._publish_scope`` delivery. + + ``RegimeClassifier._publishing`` retains the exact submission + whose consumer may currently be blocked at an ``await``. + + ''' + if ( + self._publishing is not None + and self._publish_scope is not None + and not self._fresh(self._publishing) + ): + self._publish_scope.cancel() + + def submit(self, snapshot: RegimeSnapshot) -> bool: + ''' + Replace ``RegimeClassifier._pending`` with newer evidence. + + Refuse older source coordinates or a different market before + replacing ``RegimeClassifier._latest`` and its pending work. + + ''' + if snapshot.revision < self._minimum_revision: + return False + latest: _Submission|None = self._latest + if latest is not None: + previous: RegimeSnapshot = latest.snapshot + if snapshot.market != previous.market: + raise ValueError('one classifier owns one market') + if (snapshot.revision, snapshot.sequence) <= ( + previous.revision, previous.sequence, + ): + return False + if ( + snapshot.revision == previous.revision + and snapshot.as_of <= previous.as_of + ): + raise ValueError('as_of must advance in a revision') + submission: _Submission = _Submission( + snapshot, + trio.current_time(), + ) + self._latest = self._pending = submission + self._cancel_stale_publication() + self._wake.set() + return True + + def _fresh(self, submission: _Submission) -> bool: + ''' + Check revision, gap segment, source lag, and elapsed age. + + ''' + latest: _Submission|None = self._latest + assert latest is not None + source: RegimeSnapshot = submission.snapshot + newest: RegimeSnapshot = latest.snapshot + lag: float = newest.as_of - source.as_of + age: float = trio.current_time() - submission.submitted + return ( + source.revision >= self._minimum_revision + and source.revision == newest.revision + and source.segment == newest.segment + and 0 <= lag <= self.max_source_lag + and age <= self.max_age + ) + + async def run( + self, + publish: Callable[[JevClassification], Awaitable[None]], + *, + on_error: Callable[[Exception], Awaitable[None]]|None = None, + task_status: trio.TaskStatus[None] = ( + trio.TASK_STATUS_IGNORED + ), + ) -> None: + ''' + Serve until cancelled, with bounded transient retries. + + Authentication and malformed-response failures propagate; + transient errors call ``on_error`` and retry newest state. + Retry-After delays have a 60-second cap. Each retry rechecks + freshness; an expired snapshot is never resent. + + ``publish`` must cooperate with Trio cancellation. Its wait + is cancelled if evidence expires or is invalidated. Consumers + must also validate received message provenance: a delivery + already completed before invalidation cannot be revoked. + + ''' + if self._running: + raise RuntimeError('classifier worker already running') + self._running = True + task_status.started() + next_request: float = 0 + retries: int = 0 + try: + while True: + if self._pending is None: + await self._wake.wait() + self._wake = trio.Event() + await trio.sleep_until(next_request) + submission: _Submission|None = self._pending + self._pending = None + if submission is None: + continue + if not self._fresh(submission): + self.discarded += 1 + continue + ready: bool = False + frame: TimeframeRegime + for frame in submission.snapshot.timeframes: + ready |= bool(frame.ready and frame.ready[0]) + if not ready: + continue + next_request = ( + trio.current_time() + self.min_interval + ) + self.requests += 1 + error: ( + httpx.TransportError + |trio.TooSlowError + |JevServiceError + ) + try: + result: JevClassification = await ( + self.client.classify(submission.snapshot) + ) + except ( + httpx.TransportError, + trio.TooSlowError, + JevServiceError, + ) as error: + self.failures += 1 + self.last_error = error + if ( + isinstance(error, JevServiceError) + and not error.retryable + ): + raise + if on_error is not None: + await on_error(error) + if retries < self.max_retries: + delay: float = min(60, 2 ** retries) + if isinstance(error, JevServiceError): + delay = max(delay, error.retry_after) + next_request = max( + next_request, + trio.current_time() + delay, + ) + if self._pending is None: + self._pending = self._latest + retries += 1 + else: + retries = 0 + continue + + retries = 0 + self.last_error = None + if self._fresh(submission): + deadline: float = ( + submission.submitted + self.max_age + ) + scope: trio.CancelScope = trio.CancelScope( + deadline=deadline, + ) + with scope: + self._publish_scope = scope + self._publishing = submission + try: + await publish(result) + finally: + self._publishing = None + self._publish_scope = None + if scope.cancelled_caught: + self.discarded += 1 + else: + self.published += 1 + else: + self.discarded += 1 + finally: + self._running = False diff --git a/piker/fsp/_regime.py b/piker/fsp/_regime.py new file mode 100644 index 00000000..dfdad4c7 --- /dev/null +++ b/piker/fsp/_regime.py @@ -0,0 +1,422 @@ +# piker: trading gear for hackers +# Copyright (C) Tyler Goodlet (in stewardship of piker0) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Closed-bar, multiple-timeframe trend evidence for regime consumers. + +``MultiTimeframeTrend.push()`` receives a FINAL base-bar close and +the bar's START timestamp. Each scalar close is observed at the END +of its interval. Epoch-aligned buckets require every constituent +base bar; neither partial startup buckets nor missing bars become +synthetic observations. A source gap resets all recurrence state. + +Wall-clock periods and Trend Calculus observation windows are +independent parameters. This module delegates every trend decision +to ``TrendCalculator``; semantic model labels belong in ``_jev``. + +''' +from collections import deque +from dataclasses import asdict, dataclass +from decimal import Decimal +from math import isclose, isfinite, ulp +from operator import index as integer_index + +import numpy as np + +from ._trend import TrendCalculator, TrendUpdate + + +def _period_us(period: float) -> int: + ''' + Normalize periods to positive integral microseconds. + + ''' + if isinstance(period, bool) or not isfinite(period): + raise ValueError('period must be finite and positive') + micros: Decimal = Decimal(str(period)) * 1_000_000 + nearest: Decimal = micros.to_integral_value() + tolerance: Decimal = Decimal(str(2 * ulp(period))) * 1_000_000 + if nearest <= 0 or abs(micros - nearest) > tolerance: + raise ValueError( + 'period requires positive whole microseconds', + ) + return int(nearest) + + +@dataclass(frozen=True) +class CloseSample: + ''' + A completed bucket close at its observation time. + + ``CloseSample.index`` is the final contributing base coordinate. + + ''' + time: float + index: int + price: float + + +@dataclass(frozen=True) +class TrendReversal: + ''' + Preserve both a pivot and when its reversal became observable. + + ''' + index: int + time: float + price: float + direction: int + order: int + detected_index: int + detected_time: float + + +@dataclass(frozen=True) +class TimeframeRegime: + ''' + Immutable deterministic evidence for one wall-clock timeframe. + + ``TimeframeRegime.ready`` concerns complete TC windows, never + model confidence. ``TimeframeRegime.as_of`` is the last complete + bucket's end, or ``None``. ``TimeframeRegime.recent_closes`` and + ``TimeframeRegime.reversals`` retain bounded causal evidence. + + ''' + period: float + as_of: float|None + close_count: int + trends: tuple[int, ...] + ready: tuple[bool, ...] + recent_closes: tuple[CloseSample, ...] + reversals: tuple[TrendReversal, ...] + discarded_buckets: int + + +@dataclass(frozen=True) +class RegimeSnapshot: + ''' + Bind all timeframe evidence to one source revision and clock. + + ``RegimeSnapshot.sequence`` increases on every accepted base bar + within a revision. ``RegimeSnapshot.segment`` increases at source + gaps. ``RegimeSnapshot.as_of`` records the latest base-bar END, + independently of the snapshot's wall-clock arrival time. + + ''' + market: str + revision: int + sequence: int + segment: int + as_of: float + base_period: float + windows: tuple[int, ...] + timeframes: tuple[TimeframeRegime, ...] + + def to_state(self) -> dict: + ''' + Build JSON-safe state with explicit timestamp conventions. + + ''' + state: dict = asdict(self) + state['algorithm'] = 'trend_calculus' + state['price_source'] = 'closed_bar_close' + state['timestamp_convention'] = 'interval_end' + state['trend_codes'] = { + '-1': 'down', + '0': 'unresolved or not ready; consult ready', + '1': 'up', + } + return state + + +class _Timeframe: + ''' + Own one bounded recurrence and its current aggregation bucket. + + ''' + def __init__( + self, + period_us: int, + factor: int, + windows: tuple[int, ...], + history_limit: int, + ) -> None: + ''' + Allocate one recurrence and bounded close/reversal tails. + + ''' + self.period_us: int = period_us + self.factor: int = factor + self.calculator: TrendCalculator = TrendCalculator(windows) + self.closes: deque[CloseSample] = deque(maxlen=history_limit) + self.reversals: deque[TrendReversal] = deque( + maxlen=history_limit, + ) + self.update: TrendUpdate|None = None + self.close_count: int = 0 + self.bucket: int|None = None + self.count: int = 0 + self.discarded: int = 0 + + def reset(self) -> None: + ''' + Reset ``_Timeframe.calculator`` and its bucket provenance. + + Clear ``_Timeframe.closes`` and ``_Timeframe.reversals`` on a + source gap so no previous segment survives as evidence. + + ''' + if self.count: + self.discarded += 1 + self.calculator.reset() + self.closes.clear() + self.reversals.clear() + self.update = None + self.close_count = 0 + self.bucket = None + self.count = 0 + + def push( + self, + tick: int, + close: float, + source_index: int, + ) -> None: + ''' + Advance only after an entire aligned bucket was observed. + + ''' + bucket: int = tick // self.factor + if bucket != self.bucket: + self.bucket = bucket + self.count = 0 + self.count += 1 + if (tick + 1) % self.factor: + return + if self.count != self.factor: + self.discarded += 1 + self.count = 0 + return + + end: float = (bucket + 1) * self.period_us / 1_000_000 + update: TrendUpdate = self.calculator.push( + end, + close, + index=source_index, + ) + self.update = update + self.close_count += 1 + self.count = 0 + self.closes.append(CloseSample(end, source_index, close)) + event: np.void + for event in update.reversals: + self.reversals.append(TrendReversal( + index=int(event['index']), + time=float(event['time']), + price=float(event['price']), + direction=int(event['direction']), + order=int(event['order']), + detected_index=int(event['detected_index']), + detected_time=float(event['detected_time']), + )) + + def snapshot(self) -> TimeframeRegime: + ''' + Detach immutable evidence from this writer's mutable state. + + ''' + update: TrendUpdate|None = self.update + size: int = len(self.calculator.windows) + return TimeframeRegime( + period=self.period_us / 1_000_000, + as_of=update.time if update else None, + close_count=self.close_count, + trends=update.trends if update else (0,) * size, + ready=update.ready if update else (False,) * size, + recent_closes=tuple(self.closes), + reversals=tuple(self.reversals), + discarded_buckets=self.discarded, + ) + + +class MultiTimeframeTrend: + ''' + Aggregate final base bars into causal, independent TC streams. + + Call ``reset()`` and replay CLOSED source history after backfill, + corrections, reconnect/session changes, or any source revision. + Never replay the current mutable OHLC row as a final observation. + A missing base interval conservatively resets every timeframe; + a duplicate or out-of-order interval raises without mutation. + + Work per update and retained memory are bounded by configured + timeframe/order counts and ``history_limit``, not stream age. + + ''' + def __init__( + self, + market: str, + *, + base_period: float = 1, + periods: tuple[float, ...] = (1, 5, 15), + windows: tuple[int, ...] = (2, 2, 2), + history_limit: int = 16, + revision: int = 0, + ) -> None: + ''' + Validate timeframe periods before allocating recurrences. + + ''' + if not isinstance(market, str) or not market.strip(): + raise ValueError('market identity is required') + if not 1 <= len(periods) <= 16: + raise ValueError('configure between 1 and 16 timeframes') + if ( + isinstance(history_limit, bool) + or not 1 <= integer_index(history_limit) <= 256 + ): + raise ValueError('history_limit must be in [1, 256]') + if isinstance(revision, bool) or integer_index(revision) < 0: + raise ValueError('revision must be an integer >= 0') + base_us: int = _period_us(base_period) + periods_us: list[int] = [] + period: float + for period in periods: + periods_us.append(_period_us(period)) + if len(set(periods_us)) != len(periods_us): + raise ValueError('periods must be unique base multiples') + micros: int + for micros in periods_us: + if micros % base_us: + raise ValueError('periods need whole base multiples') + + self.market: str = market + self.base_period: float = base_us / 1_000_000 + self.windows: tuple[int, ...] = tuple(windows) + self.revision: int = int(revision) + self.sequence: int = 0 + self.segment: int = 0 + self._base_us: int = base_us + self._last_tick: int|None = None + self._last_index: int = -1 + self._frames: tuple[_Timeframe, ...] = tuple( + _Timeframe( + micros, + micros // base_us, + self.windows, + history_limit, + ) + for micros in sorted(periods_us) + ) + # Reuse TrendCalculator.windows, already normalized to int, + # so accepted NumPy integer inputs remain JSON-serializable. + self.windows = self._frames[0].calculator.windows + + def reset(self, revision: int|None = None) -> None: + ''' + Invalidate prior evidence before replaying revised history. + + ''' + if revision is None: + revision = self.revision + 1 + if ( + isinstance(revision, bool) + or integer_index(revision) <= self.revision + ): + raise ValueError('revision must strictly increase') + frame: _Timeframe + for frame in self._frames: + frame.reset() + frame.discarded = 0 + self.revision = int(revision) + self.sequence = 0 + self.segment = 0 + self._last_tick = None + self._last_index = -1 + + def push( + self, + time: float, + close: float, + *, + index: int|None = None, + ) -> RegimeSnapshot: + ''' + Commit a final close with its base-bar START timestamp. + + Nonfinite data and unaligned timestamps are rejected before + mutation. Following omitted/invalid bars, the next timestamp + exposes the gap and resets all trend evidence explicitly. + + ''' + if not isfinite(time) or not isfinite(close): + raise ValueError('bar time and close must be finite') + ticks: float = time / self.base_period + if not isfinite(ticks): + raise ValueError('bar time exceeds grid precision') + tick: int = round(ticks) + aligned: float = tick * self._base_us / 1_000_000 + end: float = (tick + 1) * self._base_us / 1_000_000 + if not isfinite(end) or end <= aligned: + raise ValueError('bar interval exceeds clock precision') + if not isclose( + time, + aligned, + rel_tol=0, + abs_tol=max(1e-9, 2 * ulp(time)), + ): + raise ValueError('bar starts must align to base period') + if self._last_tick is not None and tick <= self._last_tick: + raise ValueError('bar starts must strictly increase') + if index is None: + index = self._last_index + 1 + if isinstance(index, (bool, np.bool_)): + raise ValueError('source index needs increasing int64') + coordinate: int = integer_index(index) + if ( + not 0 <= coordinate < 2 ** 63 + or coordinate <= self._last_index + ): + raise ValueError('source index needs increasing int64') + + frame: _Timeframe + if ( + self._last_tick is not None + and tick != self._last_tick + 1 + ): + self.segment += 1 + for frame in self._frames: + frame.reset() + + for frame in self._frames: + frame.push(tick, float(close), coordinate) + self._last_tick = tick + self._last_index = coordinate + self.sequence += 1 + return RegimeSnapshot( + market=self.market, + revision=self.revision, + sequence=self.sequence, + segment=self.segment, + as_of=end, + base_period=self.base_period, + windows=self.windows, + timeframes=tuple( + frame.snapshot() for frame in self._frames + ), + ) diff --git a/piker/fsp/_regime_feed.py b/piker/fsp/_regime_feed.py new file mode 100644 index 00000000..e42b337e --- /dev/null +++ b/piker/fsp/_regime_feed.py @@ -0,0 +1,550 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Bridge finalized native OHLC SHM rows to regime snapshot consumers. + +Sampler messages are wakeups and invalidations, not bar values. +This reader copies the authoritative source range and excludes its +mutable final row. Existing SHM has no transactional revision word: +bound checks detect prepend/truncation; in-place corrections rely +on the source's backfill notice and converge by invalidating/replay. + +''' +from collections.abc import AsyncIterator, Awaitable, Callable +from dataclasses import dataclass +from operator import index as integer_index +from math import isfinite +import time +from typing import TYPE_CHECKING + +import numpy as np +import trio +from tractor.trionics import Lagged + +from ._regime import MultiTimeframeTrend, RegimeSnapshot + +if TYPE_CHECKING: + from tractor.ipc._shm import ShmArray + from ._jev import JevClassification, JevClient, RegimeClassifier + + +class SourceChanged(Exception): + ''' + Signal changed shared bounds while copying a finalized prefix. + + ''' + + +@dataclass(frozen=True) +class ClosedSourceRows: + ''' + Keep one detached finalized range and its absolute SHM bounds. + + ``ClosedSourceRows.last`` includes the mutable source row; + ``ClosedSourceRows.rows`` excludes it. ``ClosedSourceRows.start`` + is a physical absolute offset in ``ShmArray._array``. + + ''' + first: int + last: int + start: int + rows: np.ndarray + + +def _bounds(shm: 'ShmArray') -> tuple[int, int]: + ''' + Read absolute shared bounds, never the wrapped ``shm.index``. + + ''' + return int(shm._first.value), int(shm._last.value) + + +def copy_closed_rows( + shm: 'ShmArray', + *, + start: int|None = None, +) -> ClosedSourceRows: + ''' + Copy a stable finalized range before its consumer can yield. + + Invalid numeric data is rejected before recurrence mutation. + ``ShmArray._array['index']`` gives absolute source coordinates. + Increasing timestamps may contain gaps: the tracker resets its + segment on those gaps. The source's mutable last row is never + validated or copied, so unfinished OHLC cannot become evidence. + Appends preserve this closed prefix. ``ClosedSourceRows.last`` + retains the captured bound so ``consume_regime_source()`` can + process an appended suffix on its next pass. + + ''' + first: int + last: int + first, last = _bounds(shm) + raw: np.ndarray = shm._array + if not 0 <= first <= last <= len(raw): + if _bounds(shm) != (first, last): + raise SourceChanged('source bounds changed during read') + raise ValueError('source bounds are not a readable range') + stop: int = max(first, last - 1) + if start is None: + start = first + if not first <= start <= stop: + raise SourceChanged('finalized source range moved') + if not {'index', 'time', 'close'} <= set(raw.dtype.names or ()): + raise ValueError('source requires index/time/close fields') + rows: np.ndarray = raw[start:stop][[ + 'index', 'time', 'close', + ]].copy() + current_first: int + current_last: int + current_first, current_last = _bounds(shm) + if current_first != first or current_last < last: + raise SourceChanged('source bounds changed during copy') + indices: np.ndarray = rows['index'] + times: np.ndarray = rows['time'] + if not np.issubdtype(indices.dtype, np.integer): + raise ValueError('source indices must be integers') + if ( + np.any(indices < 0) + or np.any(indices > np.iinfo(np.int64).max) + or np.any(indices[1:] <= indices[:-1]) + or not np.isfinite(times).all() + or np.any(times[1:] <= times[:-1]) + or not np.isfinite(rows['close']).all() + ): + raise ValueError('source has invalid finalized observations') + rows.flags.writeable = False + return ClosedSourceRows(first, last, start, rows) + + +def _sample_action( + message: dict, + market: str, + base_period: float, +) -> str: + ''' + Filter global sampler broadcasts by the actual source identity. + + ''' + if not isinstance(message, dict): + return 'ignore' + if 'backfilling' in message: + notice: object = message['backfilling'] + if ( + isinstance(notice, (tuple, list)) + and len(notice) == 2 + and notice[0] == market + and notice[1] == base_period + ): + return 'revision' + return 'ignore' + if message.get('period') == base_period: + return 'step' + return 'ignore' + + +def source_snapshot_is_current( + snapshot: RegimeSnapshot, + *, + now: float, + max_source_age: float|None = 5, +) -> bool: + ''' + Gate model admission by wall-clock source age, independent of UI. + + Allow at most one base period of forward clock skew. ``None`` + explicitly disables the clock gate for historical replay. Display + consumers can retain history while sharing this gate before Jev. + + ''' + if max_source_age is None: + return True + if not isfinite(max_source_age) or max_source_age <= 0: + raise ValueError('max_source_age must be positive or None') + if not isfinite(now): + return False + age: float = now - snapshot.as_of + return -snapshot.base_period <= age <= max_source_age + + +class _Notices: + ''' + Coalesce sampling wakeups while retaining revision invalidation. + + ``_Notices.collect()`` mutates only its notification flags and + ``_Notices.revision``, never the TC recurrence. + ``_Notices.revise()`` calls ``_Notices.invalidate`` before replay + checkpoints allow an obsolete model response to finish. + + ''' + def __init__( + self, + tracker: MultiTimeframeTrend, + invalidate: Callable[[int], object]|None, + ) -> None: + ''' + Bind a tracker revision to its synchronous invalidation hook. + + ''' + self.tracker: MultiTimeframeTrend = tracker + self.revision: int = tracker.revision + self.invalidate: Callable[[int], object]|None = invalidate + self.dirty: bool = True + self.closed: bool = False + self.wake: trio.Event = trio.Event() + + def revise(self) -> None: + ''' + Allocate the next generation and revoke its predecessor. + + ''' + self.revision += 1 + if self.invalidate is not None: + self.invalidate(self.revision) + self.dirty = True + self.wake.set() + + async def collect( + self, + stream: AsyncIterator[dict], + *, + task_status: trio.TaskStatus[None] = ( + trio.TASK_STATUS_IGNORED + ), + ) -> None: + ''' + Preserve backfill/overrun notices during slow bootstrap. + + ''' + iterator: AsyncIterator[dict] = aiter(stream) + task_status.started() + while True: + try: + message: dict = await anext(iterator) + except Lagged: + # A lost notice can describe an in-place correction; + # unchanged SHM bounds cannot prove consistency. + self.revise() + continue + except StopAsyncIteration: + self.closed = True + self.wake.set() + return + action: str = _sample_action( + message, + self.tracker.market, + self.tracker.base_period, + ) + if action == 'revision': + self.revise() + elif action == 'step': + self.dirty = True + self.wake.set() + + +async def consume_regime_source( + shm: 'ShmArray', + sample_stream: AsyncIterator[dict], + tracker: MultiTimeframeTrend, + submit: Callable[[RegimeSnapshot], object], + *, + invalidate: Callable[[int], object]|None = None, + replay_chunk: int = 512, +) -> None: + ''' + Bootstrap/replay closed rows, then follow source sample advances. + + Enter the source sampler subscription BEFORE calling this helper. + ``submit`` is synchronous and receives only the newest complete + snapshot per captured range, not a request for each history row. + Pass ``RegimeClassifier.submit`` and its ``invalidate`` together + when using Jev. Both callbacks must return without awaiting I/O. + + One compute task owns ``tracker``. A second task drains sampler + events into bounded flags during cooperative historical replay. + A prepend, truncation, relevant backfill, or sampler overrun + invalidates the previous revision and rebuilds from full history. + Cancellation closes the collector with the consumer's nursery. + + ''' + if ( + isinstance(replay_chunk, bool) + or integer_index(replay_chunk) < 1 + ): + raise ValueError('replay_chunk must be a positive integer') + if tracker.sequence: + raise ValueError('source consumer needs an unused tracker') + notices: _Notices = _Notices(tracker, invalidate) + prior_bounds: tuple[int, int]|None = None + copy_failures: int = 0 + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + await nursery.start(notices.collect, sample_stream) + try: + while True: + if not notices.dirty: + if notices.closed: + return + await notices.wake.wait() + notices.wake = trio.Event() + notices.dirty = False + revision: int = notices.revision + if revision != tracker.revision: + tracker.reset(revision=revision) + prior_bounds = None + first: int + last: int + first, last = _bounds(shm) + if prior_bounds is not None: + old_first: int + old_last: int + old_first, old_last = prior_bounds + if first != old_first or last < old_last: + notices.revise() + continue + start: int = max(first, old_last - 1) + else: + start = first + try: + source: ClosedSourceRows = copy_closed_rows( + shm, start=start, + ) + except SourceChanged: + copy_failures += 1 + if copy_failures >= 8: + raise SourceChanged( + 'source unstable after 8 copy attempts', + ) from None + notices.dirty = True + delay: float = min(.1, .001 * 2 ** copy_failures) + await trio.sleep(delay) + continue + copy_failures = 0 + + snapshot: RegimeSnapshot|None = None + offset: int + row: np.void + for offset, row in enumerate(source.rows, start=1): + snapshot = tracker.push( + float(row['time']), + float(row['close']), + index=int(row['index']), + ) + if offset % replay_chunk == 0: + await trio.lowlevel.checkpoint() + if notices.revision != revision: + break + # Permit queued in-place invalidations to reach the + # collector before admitting bootstrap/replay output. + await trio.lowlevel.checkpoint() + if notices.revision != revision: + notices.dirty = True + continue + current_first: int + current_last: int + current_first, current_last = _bounds(shm) + if ( + current_first != source.first + or current_last < source.last + ): + notices.revise() + continue + prior_bounds = source.first, source.last + if current_last != source.last: + notices.dirty = True + if snapshot is not None: + submit(snapshot) + finally: + nursery.cancel_scope.cancel() + + +async def run_regime_feed( + fqme: str, + submit: Callable[[RegimeSnapshot], object], + *, + invalidate: Callable[[int], object]|None = None, + periods: tuple[float, ...]|None = None, + windows: tuple[int, ...] = (2, 2, 2), + history_limit: int = 16, + max_source_age: float|None = 5, + loglevel: str|None = None, +) -> None: + ''' + Follow one native realtime Flume in an existing Piker runtime. + + This explicit entry point can open a broker data feed. It creates + no Jev client and reads no API key. The native sampler currently + stamps periods <=1s with integer seconds, so this wrapper refuses + subsecond source periods. A separate correct subsecond producer + can call ``consume_regime_source`` directly. + + Quote delivery retains the shared feed's default policy. Forcing + a low rate here would throttle later users of its market cache; + this consumer drains quotes and derives only from finalized SHM. + + ``max_source_age`` bounds wall-clock age of submitted evidence, + separately from the classifier's local queue age. ``None`` allows + historical replay; the default holds stale startup history until + live bars arrive. Source clocks over one base period ahead are + rejected by the same gate when an age limit is enabled. + + ''' + from ..data._sampling import open_sample_stream + from ..data.feed import Feed, maybe_open_feed + from ..data.flows import Flume + + if ( + max_source_age is not None + and + ( + not isfinite(max_source_age) + or max_source_age <= 0 + ) + ): + raise ValueError('max_source_age must be positive or None') + feed: Feed + async with maybe_open_feed( + fqmes=[fqme], + loglevel=loglevel, + ) as feed: + flume: Flume = feed.flumes[fqme] + # maybe_open_feed() replaces the shared Flume.stream on hits. + # Capture our receiver before the sampler context can yield. + quote_stream: AsyncIterator[dict] = flume.stream + period: float = flume.rt_sample_period_s + if period < 1: + raise ValueError('native sampler requires >=1s periods') + if periods is None: + periods = (period, period * 5, period * 15) + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + fqme, + base_period=period, + periods=periods, + windows=windows, + history_limit=history_limit, + ) + + def submit_fresh(snapshot: RegimeSnapshot) -> None: + ''' + Hold stale startup data before admitting live inference. + + ''' + if source_snapshot_is_current( + snapshot, + now=time.time(), + max_source_age=max_source_age, + ): + submit(snapshot) + + async def drain_quotes() -> None: + ''' + Keep the quote subscription healthy while consuming SHM. + + ''' + iterator: AsyncIterator[dict] = aiter(quote_stream) + while True: + try: + await anext(iterator) + except Lagged: + # Quotes are discarded; all finalized prices + # still come from source SHM and sampler events. + await trio.lowlevel.checkpoint() + except StopAsyncIteration: + return + + stream: AsyncIterator[dict] + nursery: trio.Nursery + async with ( + open_sample_stream( + period_s=period, + loglevel=loglevel, + ) as stream, + trio.open_nursery() as nursery, + ): + nursery.start_soon(drain_quotes) + try: + await consume_regime_source( + flume.rt_shm, + stream, + tracker, + submit_fresh, + invalidate=invalidate, + ) + finally: + nursery.cancel_scope.cancel() + + +async def serve_regimes( + fqme: str, + client: 'JevClient', + publish: Callable[['JevClassification'], Awaitable[None]], + *, + periods: tuple[float, ...]|None = None, + windows: tuple[int, ...] = (2, 2, 2), + history_limit: int = 16, + min_interval: float = 1, + max_age: float = 5, + max_source_lag: float = 2, + max_source_age: float|None = 5, + on_error: Callable[[Exception], Awaitable[None]]|None = None, + loglevel: str|None = None, +) -> 'RegimeClassifier': + ''' + Compose a native feed and paced Jev worker in one owned nursery. + + This explicit call opens a broker feed and uses the supplied Jev + client for inference. The caller owns its HTTP session and Piker + runtime. Cancellation closes the sampler subscription, quote + drain and in-flight classifier work together. Consumers validate + source provenance again when applying delivered messages. + + ''' + from ._jev import RegimeClassifier + + worker: RegimeClassifier = RegimeClassifier( + client, + min_interval=min_interval, + max_age=max_age, + max_source_lag=max_source_lag, + ) + + async def classify() -> None: + ''' + Bind the optional error callback to this service lifetime. + + ''' + await worker.run(publish, on_error=on_error) + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(classify) + try: + await run_regime_feed( + fqme, + worker.submit, + invalidate=worker.invalidate, + periods=periods, + windows=windows, + history_limit=history_limit, + max_source_age=max_source_age, + loglevel=loglevel, + ) + finally: + nursery.cancel_scope.cancel() + return worker diff --git a/piker/fsp/_trend.py b/piker/fsp/_trend.py new file mode 100644 index 00000000..adcadbac --- /dev/null +++ b/piker/fsp/_trend.py @@ -0,0 +1,591 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +# Algorithm: Andrew J. Morgan / ByteSumo, TrendCalculus (GPLv3+). +# See docs/trend_calculus.rst for sources and semantic differences. + +''' +Causal, bounded-state Trend Calculus for scalar observations. + +The same compiled recurrence drives historical and live processing. +FHLS means first/high/low/second *extremum*, not open/high/low/close. + +''' +from dataclasses import dataclass +from math import isfinite +from operator import index as integer_index + +import numpy as np +from numba import njit + + +_bar_dtype: np.dtype[np.void] = np.dtype([ + ('high', ' int: + ''' + Evaluate Morgan's sign sum without overflowing a subtraction. + + ''' + delta: int = ( + int(high > prev_high) - int(high < prev_high) + + int(low > prev_low) - int(low < prev_low) + ) + return int(delta > 0) - int(delta < 0) + + +@njit(cache=True) +def _reversal( + previous: np.void, + old_trend: int, + new_trend: int, + level: int, + index: int, + time: float, + events: np.ndarray, + counts: np.ndarray, +) -> None: + ''' + Emit a confirmed sign change at its detection observation. + + ''' + if old_trend * new_trend != -1: + return + + slot: int = counts[level] + event: np.void = events[level, slot] + if new_trend == 1: + event['index'] = previous['lo_index'] + event['time'] = previous['lo_time'] + event['price'] = previous['low'] + else: + event['index'] = previous['hi_index'] + event['time'] = previous['hi_time'] + event['price'] = previous['high'] + + event['direction'] = new_trend + event['order'] = level + 1 + event['detected_index'] = index + event['detected_time'] = time + counts[level] += 1 + + +@njit(cache=True) +def _step( + bars: np.ndarray, + counters: np.ndarray, + sizes: np.ndarray, + index: int, + time: float, + value: float, + events: np.ndarray, + counts: np.ndarray, + lua_seed: bool, +) -> None: + ''' + Advance each reversal order using at most two incoming points. + + The ``TrendCalculator._bars`` argument uses ``bars[:, 0]`` for + the current window and ``bars[:, 1]`` for its completed + predecessor. ``TrendCalculator._counters`` stores pending sample + count, completed window count, and confirmed direction. + Higher orders consume only the preceding order's reversals. + + ''' + counts[:] = 0 + if np.isnan(value): + bars[:] = np.zeros_like(bars) + counters[:] = 0 + return + + bridge: np.ndarray = np.empty(1, dtype=_bar_dtype) + level: int + for level in range(len(sizes)): + incoming: int = 1 if level == 0 else counts[level - 1] + if incoming == 0: + break + + point: int + for point in range(incoming): + price: float + point_time: float + point_index: int + if level == 0: + price = value + point_time = time + point_index = index + else: + event: np.void = events[level - 1, point] + price = event['price'] + point_time = event['time'] + point_index = event['index'] + + current: np.void = bars[level, 0] + previous: np.void = bars[level, 1] + if counters[level, 0] == 0: + current['high'] = current['low'] = price + current['hi_time'] = current['lo_time'] = point_time + current['hi_index'] = point_index + current['lo_index'] = point_index + + elif price > current['high']: + current['high'] = price + current['hi_time'] = point_time + current['hi_index'] = point_index + elif price < current['low']: + current['low'] = price + current['lo_time'] = point_time + current['lo_index'] = point_index + + counters[level, 0] += 1 + if counters[level, 0] < sizes[level]: + continue + + counters[level, 0] = 0 + counters[level, 1] += 1 + if counters[level, 1] == 1: + if lua_seed: + # Lua compares its first window with its final + # point. Native startup leaves direction unknown. + previous['high'] = previous['low'] = price + previous['hi_time'] = point_time + previous['lo_time'] = point_time + previous['hi_index'] = point_index + previous['lo_index'] = point_index + + else: + bars[level, 1] = current + continue + + trend: int = _direction( + current['high'], current['low'], + previous['high'], previous['low'], + ) + old_trend: int = counters[level, 2] + if trend == 0: + # Bridge the later previous extremum to the earlier + # current extremum. Preserve both possible reversals. + first_price: float + first_time: float + first_index: int + if previous['hi_index'] > previous['lo_index']: + first_price = previous['high'] + first_time = previous['hi_time'] + first_index = previous['hi_index'] + + else: + first_price = previous['low'] + first_time = previous['lo_time'] + first_index = previous['lo_index'] + + second_price: float + second_time: float + second_index: int + if current['hi_index'] < current['lo_index']: + second_price = current['high'] + second_time = current['hi_time'] + second_index = current['hi_index'] + + else: + second_price = current['low'] + second_time = current['lo_time'] + second_index = current['lo_index'] + + split: np.void = bridge[0] + if first_price > second_price: + split['high'] = first_price + split['hi_time'] = first_time + split['hi_index'] = first_index + split['low'] = second_price + split['lo_time'] = second_time + split['lo_index'] = second_index + + else: + split['high'] = second_price + split['hi_time'] = second_time + split['hi_index'] = second_index + split['low'] = first_price + split['lo_time'] = first_time + split['lo_index'] = first_index + + split_trend: int = _direction( + split['high'], split['low'], + previous['high'], previous['low'], + ) or old_trend + _reversal( + previous, old_trend, split_trend, + level, index, time, events, counts, + ) + trend = _direction( + current['high'], current['low'], + split['high'], split['low'], + ) or split_trend + _reversal( + split, split_trend, trend, + level, index, time, events, counts, + ) + + else: + _reversal( + previous, old_trend, trend, + level, index, time, events, counts, + ) + + counters[level, 2] = trend + bars[level, 1] = current + + +@njit(cache=True) +def _scan( + values: np.ndarray, + times: np.ndarray, + indices: np.ndarray, + bars: np.ndarray, + counters: np.ndarray, + sizes: np.ndarray, + lua_seed: bool, +) -> tuple[ + np.ndarray, + np.ndarray, + np.ndarray, + np.ndarray, + np.ndarray, +]: + ''' + Bootstrap with ``_step()``, retaining causal detection pulses. + + ''' + length: int = len(values) + levels: int = len(sizes) + trends: np.ndarray = np.zeros((length, levels), dtype=np.int8) + ready: np.ndarray = np.zeros((length, levels), dtype=np.bool_) + pulses: np.ndarray = np.zeros((length, levels), dtype=np.int8) + pivot_times: np.ndarray = np.full((length, levels, 2), np.nan) + pivot_prices: np.ndarray = np.full((length, levels, 2), np.nan) + events: np.ndarray = np.empty((levels, 2), dtype=reversal_dtype) + counts: np.ndarray = np.zeros(levels, dtype=np.int64) + row: int + for row in range(length): + _step( + bars, counters, sizes, + indices[row], times[row], values[row], + events, counts, lua_seed, + ) + level: int + for level in range(levels): + ready[row, level] = counters[level, 1] >= 2 + if ready[row, level]: + trends[row, level] = counters[level, 2] + if counts[level]: + pulses[row, level] = events[ + level, counts[level] - 1 + ]['direction'] + slot: int + for slot in range(counts[level]): + event: np.void = events[level, slot] + side: int = int(event['direction'] > 0) + pivot_times[row, level, side] = event['time'] + pivot_prices[row, level, side] = event['price'] + + return trends, ready, pulses, pivot_times, pivot_prices + + +@dataclass(frozen=True) +class TrendUpdate: + ''' + A causal observation: directions and newly confirmed reversals. + + ``TrendUpdate.ready`` means two complete windows, not statistical + confidence. A ready zero direction means unresolved flat input. + ``TrendUpdate.reversals`` has separate pivot and detection + coordinates and can contain two opposing events at the same + order and detection time. + + ''' + index: int + time: float + trends: tuple[int, ...] + ready: tuple[bool, ...] + reversals: np.ndarray + + +@dataclass(frozen=True) +class TrendHistory: + ''' + Prefix-causal history, indexed by observation and reversal order. + + ``TrendHistory.pulses`` records the last reversal direction + detected per row and order. Use ``TrendCalculator.push()`` to + retain every event; two reversals at one detection row cannot + fit in one pulse. ``TrendHistory.pivot_times`` and + ``TrendHistory.pivot_prices`` retain both sides with shape + (observations, orders, 2), where side 0 is top and 1 is bottom. + Absent events are NaN, never backdated into earlier output rows. + + ''' + trends: np.ndarray + ready: np.ndarray + pulses: np.ndarray + pivot_times: np.ndarray + pivot_prices: np.ndarray + + +class TrendCalculator: + ''' + Track fixed-size FHLS windows and their recursive reversals. + + Each window size counts observations at that order; it is not a + duration. Scalar input avoids inventing intrabar high/low order. + Equal extrema retain their earliest observation (Lua semantics). + Native startup requires two windows and emits no artificial + initial pivot. ``lua_seed=True`` uses the Lua first-window seed + for research; it does not reproduce its output suppression bugs. + NaN resets all orders. Invalid or non-increasing coordinates + and infinite values are rejected before state mutation. + + State size is O(number of orders), independent of window sizes + and history length. Instances have one writer and are not shared + between markets or timeframes. + + ''' + def __init__( + self, + windows: tuple[int, ...] = (2, 2, 2, 2), + *, + lua_seed: bool = False, + ) -> None: + ''' + Allocate bounded state for each validated window size. + + ''' + if not windows: + raise ValueError('at least one window size is required') + sizes: list[int] = [] + size: int + for size in windows: + if isinstance(size, (bool, np.bool_)): + raise ValueError( + 'window size must be an integer >= 2', + ) + size = integer_index(size) + if not 2 <= size <= np.iinfo(np.int64).max: + raise ValueError( + 'window size exceeds positive int64', + ) + sizes.append(size) + self.windows: tuple[int, ...] = tuple(sizes) + self._sizes: np.ndarray = np.array(sizes, dtype=np.int64) + self.lua_seed: bool = lua_seed + self._bars: np.ndarray = np.zeros( + (len(windows), 2), dtype=_bar_dtype, + ) + self._counters: np.ndarray = np.zeros( + (len(windows), 3), dtype=np.int64, + ) + self._events: np.ndarray = np.empty( + (len(windows), 2), dtype=reversal_dtype, + ) + self._counts: np.ndarray = np.zeros( + len(windows), dtype=np.int64, + ) + self._last_time: float = -np.inf + self._last_index: int = -1 + + def copy(self) -> 'TrendCalculator': + ''' + Copy bounded recurrence state for an uncommitted preview. + + Duplicate ``TrendCalculator._bars`` and + ``TrendCalculator._counters`` with the last source + coordinates. Emitted events remain owned by each instance. + + ''' + clone: TrendCalculator = TrendCalculator( + self.windows, lua_seed=self.lua_seed, + ) + clone._bars[:] = self._bars + clone._counters[:] = self._counters + clone._last_time = self._last_time + clone._last_index = self._last_index + return clone + + def reset(self) -> None: + ''' + Start a new session or revision with no inherited trend. + + Clear ``TrendCalculator._bars``, + ``TrendCalculator._counters`` and + ``TrendCalculator._counts``. Release source constraints in + ``TrendCalculator._last_time`` and + ``TrendCalculator._last_index``. + + ''' + self._bars[:] = np.zeros_like(self._bars) + self._counters[:] = 0 + self._counts[:] = 0 + self._last_time = -np.inf + self._last_index = -1 + + def _validate( + self, + index: int, + time: float, + value: float, + ) -> None: + ''' + Refuse ambiguous ordering without consuming an observation. + + ''' + if not isfinite(time) or time <= self._last_time: + raise ValueError('time must be finite and increasing') + if not 0 <= index <= np.iinfo(np.int64).max: + raise ValueError('index must fit nonnegative int64') + if index <= self._last_index: + raise ValueError('index must be increasing') + if np.isinf(value): + raise ValueError('infinite values are not observations') + + def push( + self, + time: float, + value: float, + *, + index: int|None = None, + ) -> TrendUpdate: + ''' + Commit one scalar sample and return its causal output. + + ''' + if index is None: + index = self._last_index + 1 + if isinstance(index, (bool, np.bool_)): + raise ValueError('index must be a nonnegative integer') + index = integer_index(index) + self._validate(index, time, value) + _step( + self._bars, self._counters, self._sizes, + index, time, value, self._events, self._counts, + self.lua_seed, + ) + self._last_time = time + self._last_index = index + ready: np.ndarray = self._counters[:, 1] >= 2 + trends: np.ndarray = np.where( + ready, self._counters[:, 2], 0, + ) + emitted: np.ndarray = ( + np.arange(2)[None, :] < self._counts[:, None] + ) + events: np.ndarray = self._events[emitted] + return TrendUpdate( + index=index, + time=time, + trends=tuple(map(int, trends)), + ready=tuple(map(bool, ready)), + reversals=events, + ) + + def extend( + self, + values: np.ndarray, + times: np.ndarray, + indices: np.ndarray|None = None, + ) -> TrendHistory: + ''' + Consume a batch with the same live recurrence semantics. + + All coordinates are validated before the batch mutates state. + Incomplete windows survive across calls and process restarts + should rebuild them by replaying the same source revision. + + ''' + values = np.asarray(values, dtype=np.float64) + times = np.asarray(times, dtype=np.float64) + if values.ndim != 1 or times.ndim != 1: + raise ValueError('values and times must be 1D') + if indices is None: + limit: int = np.iinfo(np.int64).max + if self._last_index + len(values) > limit: + raise ValueError('automatic indices overflow int64') + indices = np.arange( + self._last_index + 1, + self._last_index + 1 + len(values), + dtype=np.int64, + ) + indices = np.asarray(indices) + if ( + indices.ndim != 1 + or len(values) != len(times) + or len(values) != len(indices) + ): + raise ValueError('values, times, indices must align') + if not np.issubdtype(indices.dtype, np.signedinteger): + raise ValueError('indices require signed integer dtype') + if len(values): + self._validate(int(indices[0]), times[0], values[0]) + if ( + not np.isfinite(times).all() + or np.isinf(values).any() + or np.any(times[1:] <= times[:-1]) + or np.any(indices[1:] <= indices[:-1]) + ): + raise ValueError('invalid value or non-increasing batch') + trends: np.ndarray + ready: np.ndarray + pulses: np.ndarray + pivot_times: np.ndarray + pivot_prices: np.ndarray + ( + trends, + ready, + pulses, + pivot_times, + pivot_prices, + ) = _scan( + values, times, indices, self._bars, self._counters, + self._sizes, self.lua_seed, + ) + if len(values): + self._last_time = float(times[-1]) + self._last_index = int(indices[-1]) + return TrendHistory( + trends, ready, pulses, pivot_times, pivot_prices, + ) diff --git a/piker/fsp/_trend_fsp.py b/piker/fsp/_trend_fsp.py new file mode 100644 index 00000000..c6a4e5fa --- /dev/null +++ b/piker/fsp/_trend_fsp.py @@ -0,0 +1,208 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Trend Calculus over Piker's mutable close-price SHM source. + +''' +from collections.abc import AsyncIterator + +import numpy as np +from tractor.ipc._shm import ShmArray + +from ..data.ticktools import FeedQuote +from ._api import FspSample, FspStream, fsp +from ._trend import TrendCalculator, TrendHistory, TrendUpdate + + +_trend_fields: tuple[str, ...] = ( + 'trend', + 'trend_2', + 'trend_3', + 'trend_4', +) +order: int +field: str +_outputs: tuple[str, ...] = _trend_fields + ('ready',) + tuple( + field + for order in range(1, 5) + for field in ( + f'top_{order}', + f'top_time_{order}', + f'bottom_{order}', + f'bottom_time_{order}', + ) +) + + +def _history_fields( + output: TrendHistory, +) -> dict[str, np.ndarray]: + ''' + Convert causal arrays to the fixed SHM field schema. + + ''' + fields: dict[str, np.ndarray] = {} + level: int + field: str + for level, field in enumerate(_trend_fields): + fields[field] = output.trends[:, level].astype(np.float64) + fields['ready'] = output.ready.sum(axis=1).astype(np.float64) + level: int + for level in range(4): + side: int + label: str + for side, label in enumerate(('top', 'bottom')): + order: int = level + 1 + fields[f'{label}_{order}'] = ( + output.pivot_prices[:, level, side] + ) + fields[f'{label}_time_{order}'] = ( + output.pivot_times[:, level, side] + ) + return fields + + +@fsp( + outputs=_outputs, + sample_on_step=True, + event_outputs=_outputs[5:], +) +async def trend_calculus( + source: AsyncIterator[FeedQuote], + ohlcv: ShmArray, + window: int = 2, +) -> FspStream: + ''' + Publish four causal reversal orders from scalar bar closes. + + ``window`` counts first-order closes. Each higher order groups + two reversals from the preceding order into an FHLS window. + History uses finalized closes except for its mutable last row. + Live quotes preview that row from a copy of committed state; + repeated updates never consume a bar twice. Source advancement + commits all newly finalized bars, including skipped quote steps. + ``ready`` counts orders having two completed FHLS windows. + ``top_N``/``bottom_N`` are pivot prices at their detection row; + matching ``*_time_N`` fields retain the actual extremum time. + NaN means no pivot. Both sides survive split-window reversals. + + A source prepend or in-place history revision restarts this + generator through ``Cascade.resync()``. Chart timestamps remain + source bar-open times, and the last row is always provisional. + For service consumption use finalized observations and explicit + bar-close/detection times in ``TrendCalculator`` instead. + + ''' + calc: TrendCalculator = TrendCalculator((window, 2, 2, 2)) + # The engine supplies an immutable bootstrap view. Retain its + # absolute coordinates; live shared bounds may already differ. + history: np.ndarray = ohlcv.array.copy() + first: int = int(history['index'][0]) if len(history) else -1 + committed: int = first - 1 + output: TrendHistory = calc.extend( + history['close'][:-1], + history['time'][:-1], + history['index'][:-1], + ) + if len(history) > 1: + committed = int(history['index'][-2]) + + tail: TrendHistory = calc.copy().extend( + history['close'][-1:], + history['time'][-1:], + history['index'][-1:], + ) + fields: dict[str, np.ndarray] = _history_fields(output) + tail_fields: dict[str, np.ndarray] = _history_fields(tail) + field: str + for field in _outputs: + fields[field] = np.concatenate(( + fields[field], tail_fields[field], + )) + yield fields + + quote: FeedQuote + async for quote in source: + # Quotes are wakeups. The source SHM is authoritative, and a + # quote frame may contain multiple already aggregated trades. + rows: np.ndarray = ohlcv.array + if not len(rows): + continue + source_first: int = int(rows['index'][0]) + last: int = int(rows['index'][-1]) + if first == -1: + first = source_first + committed = first - 1 + if source_first != first or last <= committed: + # Normally the engine restarts first. Avoid using a + # recurrence with a different origin during the handoff. + return + + begin: int = committed - first + 1 + closed: np.ndarray = rows[begin:-1].copy() + current: np.void = rows[-1].copy() + if len(closed): + finalized: TrendHistory = calc.extend( + closed['close'], closed['time'], closed['index'], + ) + committed = int(closed['index'][-1]) + # Publish every finalized row, not only the new tail. + # Missing quote wakeups must not discard reversal events + # or leave a prior row with its last provisional close. + closed_fields: dict[str, np.ndarray] = ( + _history_fields(finalized) + ) + offset: int + row: np.void + for offset, row in enumerate(closed): + final_values: dict[str, float] = {} + field: str + for field in _outputs: + final_values[field] = float( + closed_fields[field][offset], + ) + yield FspSample( + index=int(row['index']), + time=float(row['time']), + fields=final_values, + ) + + preview: TrendUpdate = calc.copy().push( + float(current['time']), + float(current['close']), + index=int(current['index']), + ) + values: dict[str, int|float] = dict.fromkeys( + _outputs, np.nan, + ) + values.update(zip(_trend_fields, preview.trends)) + values['ready'] = sum(preview.ready) + event: np.void + label: str + order: int + for event in preview.reversals: + label = 'bottom' if event['direction'] > 0 else 'top' + order = int(event['order']) + values[f'{label}_{order}'] = float(event['price']) + values[f'{label}_time_{order}'] = float(event['time']) + yield FspSample( + index=preview.index, + time=preview.time, + fields=values, + ) diff --git a/piker/ui/_display.py b/piker/ui/_display.py index 06afd142..1c4f5573 100644 --- a/piker/ui/_display.py +++ b/piker/ui/_display.py @@ -71,6 +71,7 @@ from ._l1 import L1Labels from ._style import hcolor from ._fsp import ( update_fsp_chart, + update_trend_overlays, start_fsp_displays, open_vlm_displays, ) @@ -285,6 +286,41 @@ async def increment_history_view( # check if slow chart needs an x-domain shift and/or # y-range resize. + if isinstance(msg, dict): + backfill: tuple[str, float]|None = msg.get( + 'backfilling', + ) + if backfill is not None: + market: str + period_s: float + market, period_s = backfill + if market == ds.fqme: + changed_chart: ChartPlotWidget + for changed_chart in ( + ds.chart, ds.hist_chart, + ): + hidden: bool = ( + changed_chart.linked.isHidden() + ) + update_trend_overlays( + changed_chart, + fqme=market, + chart_period_s=period_s, + force=True, + render=not hidden, + ) + + # Sampler advancement can finalize an FSP row without a + # quote. A writer publishing after this callback appears + # on the next tick; `TrendOverlay._signature` avoids a + # redundant rebuild in `TrendOverlay.update()`. + trend_chart: ChartPlotWidget + for trend_chart in (ds.chart, ds.hist_chart): + update_trend_overlays( + trend_chart, + fqme=ds.fqme, + render=not trend_chart.linked.isHidden(), + ) ( uppx, liv, @@ -625,6 +661,17 @@ def graphics_update_cycle( chart = ds.chart vlm_chart = ds.vlm_chart + # Each market keeps its own destination-backed annotation layers. + # A forced cycle also handles source history coordinate rewrites. + trend_chart: ChartPlotWidget + for trend_chart in (chart, ds.hist_chart): + update_trend_overlays( + trend_chart, + fqme=fqme, + force=trigger_all, + render=not trend_chart.linked.isHidden(), + ) + # varz = ds.vars l1 = ds.l1 flume = ds.flume @@ -1522,11 +1569,17 @@ async def display_symbol_data( # load (user's) FSP set (otherwise known as "indicators") # from an input config. tn.start_soon( - start_fsp_displays, - rt_linked, - flume, - # loading_sym_key, - loglevel, + partial( + start_fsp_displays, + rt_linked, + flume, + # loading_sym_key, + loglevel, + show_trend_calculus=( + ui_conf.get('trend_calculus') is True + ), + trend_timeframes=ui_conf.get('trend_timeframes'), + ), ) godwidget.resize_all() diff --git a/piker/ui/_fsp.py b/piker/ui/_fsp.py index 4c9e6831..0c65ca16 100644 --- a/piker/ui/_fsp.py +++ b/piker/ui/_fsp.py @@ -23,9 +23,11 @@ Financial signal processing cluster and real-time graphics management. from __future__ import annotations from contextlib import asynccontextmanager as acm +from collections.abc import AsyncIterator, Callable from functools import partial import inspect from itertools import cycle +import time from typing import ( AsyncGenerator, Any, @@ -47,12 +49,18 @@ from piker.fsp import ( Fsp, dolla_vlm, flow_rates, + trend_calculus, ) from tractor.ipc._shm import ( ShmArray, NDToken, ) -from piker.data import Flume +from piker.data import Flume, open_sample_stream +from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot +from piker.fsp._regime_feed import ( + consume_regime_source, + source_snapshot_is_current, +) from piker.data._sharedmem import try_read from piker.log import get_logger from piker.toolz import Profiler @@ -72,6 +80,8 @@ from ._forms import ( mk_form, open_form_input_handling, ) +from ._trend import TrendOverlay +from ._trend_mtf import SnapshotTrendOverlay, _normalize_periods if TYPE_CHECKING: from ._dataviz import Viz @@ -80,6 +90,53 @@ if TYPE_CHECKING: log = get_logger(__name__) +def update_trend_overlays( + chart: ChartPlotWidget, + *, + fqme: str|None = None, + shm_token: NDToken|None = None, + chart_period_s: float|None = None, + force: bool = False, + render: bool = True, + +) -> int: + ''' + Refresh matching ``ChartPlotWidget._trend_overlays`` layers. + + Its keys are ``(fqme, destination_token)`` and each value holds + independently owned reversal orders. Hidden-chart revisions + invalidate their layers without painting; the next quote then + rebuilds even if SHM bounds and the latest sample are unchanged. + + ''' + layers: dict[tuple[str, NDToken], list[TrendOverlay]] = getattr( + chart, '_trend_overlays', {}, + ) + updated: int = 0 + key: tuple[str, NDToken] + overlays: list[TrendOverlay] + for key, overlays in layers.items(): + market: str + token: NDToken + market, token = key + if fqme is not None and market != fqme: + continue + if shm_token is not None and token != shm_token: + continue + overlay: TrendOverlay + for overlay in overlays: + if ( + chart_period_s is not None + and overlay.chart_period_s != chart_period_s + ): + continue + if force: + overlay.invalidate() + if render and overlay.update(): + updated += 1 + return updated + + def update_fsp_chart( viz: Viz, graphics_name: str, @@ -466,6 +523,14 @@ class FspAdmin: tuple[tractor.MsgStream, ShmArray] ] = {} self._flow_registry: dict[NDToken, str] = {} + self._trend_engine: tuple[Flume, trio.Event]|None = None + self._trend_start_lock: trio.Lock = trio.Lock() + self._trend_overlays: list[ + tuple[ChartPlotWidget, TrendOverlay] + ] = [] + self._multi_overlay: SnapshotTrendOverlay|None = None + self._multi_tracker: MultiTimeframeTrend|None = None + self._multi_scope: trio.CancelScope|None = None # TODO: make this a `.src_flume` and add # a `dst_flume`? @@ -476,6 +541,332 @@ class FspAdmin: name, portal = next(self._rr_next_actor) return portal + async def open_trend_overlay( + self, + *, + chart: ChartPlotWidget|None = None, + order: int = 1, + chart_period_s: float|None = None, + loglevel: str = 'info', + + ) -> TrendOverlay: + ''' + Start one TC writer and attach confirmed price annotations. + + Several charts or reversal orders reuse this admin's source + engine. The current FSP allocator supplies its realtime SHM + only; another sampling timeframe requires an independently + computed destination, attachable with ``TrendOverlay``. + No Jev request or credential lookup occurs on this path. + + ''' + if not 1 <= order <= 4: + raise ValueError('order must be 1..4') + if chart is None: + chart = self.linked.chart + fqme: str = self.flume.mkt.fqme + viz: Viz|None = chart.get_viz(fqme) + if viz is None: + raise ValueError(f'No price Viz for {fqme}') + if chart_period_s is None: + if viz.shm._token == self.flume.rt_shm._token: + chart_period_s = self.flume.rt_sample_period_s + elif viz.shm._token == self.flume._hist_shm_token: + chart_period_s = self.flume.hist_sample_period_s + else: + raise ValueError('chart_period_s is required') + if not np.isfinite(chart_period_s) or chart_period_s <= 0: + raise ValueError('chart_period_s must be positive') + + async with self._trend_start_lock: + owner: ChartPlotWidget + overlay: TrendOverlay + for owner, overlay in self._trend_overlays: + if ( + owner is chart + and type(overlay) is TrendOverlay + and overlay.order == order + ): + if overlay.chart_period_s != chart_period_s: + raise ValueError('chart period differs') + return overlay + + if self._trend_engine is None: + self._trend_engine = await self.start_engine_task( + trend_calculus, + conf={}, + loglevel=loglevel, + ) + destination: Flume + started: trio.Event + destination, started = self._trend_engine + await started.wait() + overlay = TrendOverlay( + viz, + destination.rt_shm, + fqme=fqme, + source_period_s=self.flume.rt_sample_period_s, + chart_period_s=chart_period_s, + order=order, + ) + try: + overlay.update(force=True) + except BaseException: + overlay.remove() + raise + + layers: dict[ + tuple[str, NDToken], list[TrendOverlay] + ]|None = getattr(chart, '_trend_overlays', None) + if layers is None: + layers = chart._trend_overlays = {} + key: tuple[str, NDToken] = ( + fqme, destination.rt_shm._token, + ) + layers.setdefault(key, []).append(overlay) + self._trend_overlays.append((chart, overlay)) + return overlay + + async def open_multi_timeframe_trends( + self, + *, + periods: tuple[float, ...] = (1, 5, 15), + windows: tuple[int, ...] = (2, 2, 2, 2), + order: int = 1, + on_snapshot: Callable[[RegimeSnapshot], object]|None = None, + on_invalidate: Callable[[int], object]|None = None, + max_source_age: float|None = 5, + classification_ttl: float = 5, + max_source_lag: float|None = 2, + + ) -> SnapshotTrendOverlay: + ''' + Follow one finalized source into real multiple-timeframe UI. + + The tracker consumes native realtime closes once and shares + its immutable snapshots with all graphics layers and optional + callbacks. Pass a classifier's ``submit`` and ``invalidate`` + callbacks together to reuse this same evidence for Jev; no + client or credential is created here. The returned adapter's + ``publish_classification`` fits the async classifier worker. + Only the optional model callback uses ``max_source_age``; + historical graphics always render. Explicit ``None`` permits + old model evidence for offline research. Jev requests project + shorter evidence tails without changing graphics retention. + + Return after the first finalized snapshot. Empty startup + history waits for a final source bar. + ``FspAdmin.close_trend_overlays()`` cancels + ``FspAdmin._multi_scope`` and removes its layers from + ``ChartPlotWidget._trend_overlays`` on context exit. + + ''' + if ( + max_source_age is not None + and + ( + not np.isfinite(max_source_age) + or max_source_age <= 0 + ) + ): + raise ValueError( + 'max_source_age must be positive or None', + ) + periods = _normalize_periods( + self.flume.rt_sample_period_s, periods, + ) + async with self._trend_start_lock: + existing: SnapshotTrendOverlay|None = self._multi_overlay + if existing is not None: + first_layer: TrendOverlay = next( + iter(existing.layers.values()), + ) + if ( + existing.periods != periods + or first_layer.order != order + or self._multi_tracker.windows != windows + ): + raise ValueError('tracker configuration differs') + if ( + on_snapshot is not None + or on_invalidate is not None + ): + raise ValueError('callbacks already configured') + return existing + + chart: ChartPlotWidget = self.linked.chart + fqme: str = self.flume.mkt.fqme + viz: Viz|None = chart.get_viz(fqme) + if viz is None: + raise ValueError(f'No price Viz for {fqme}') + base_period: float = self.flume.rt_sample_period_s + if base_period < 1: + raise ValueError('native sampler requires >=1s bars') + if not 1 <= order <= min(4, len(windows)): + raise ValueError('order exceeds configured windows') + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + fqme, + base_period=base_period, + periods=periods, + windows=windows, + history_limit=256, + ) + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + viz, + self.flume.rt_shm, + fqme=fqme, + base_period=base_period, + periods=periods, + chart_period_s=base_period, + order=order, + classification_ttl=classification_ttl, + max_source_lag=max_source_lag, + is_visible=lambda: not self.linked.isHidden(), + ) + self._multi_overlay = adapter + self._multi_tracker = tracker + layers: dict[ + tuple[str, NDToken], list[TrendOverlay] + ]|None = getattr(chart, '_trend_overlays', None) + if layers is None: + layers = chart._trend_overlays = {} + key: tuple[str, NDToken] = ( + fqme, self.flume.rt_shm._token, + ) + layer: TrendOverlay + for layer in adapter.layers.values(): + layers.setdefault(key, []).append(layer) + self._trend_overlays.append((chart, layer)) + + started: trio.Event = trio.Event() + self.tn.start_soon( + self._run_multi_timeframe_trends, + adapter, + tracker, + started, + on_snapshot, + on_invalidate, + max_source_age, + ) + await started.wait() + if adapter.as_of is None and self._multi_scope is None: + raise RuntimeError('source closed before bootstrap') + return adapter + + async def _run_multi_timeframe_trends( + self, + adapter: SnapshotTrendOverlay, + tracker: MultiTimeframeTrend, + started: trio.Event, + on_snapshot: Callable[[RegimeSnapshot], object]|None, + on_invalidate: Callable[[int], object]|None, + max_source_age: float|None, + + ) -> None: + ''' + Subscribe before bootstrap and fan out one writer's evidence. + + ''' + def submit(snapshot: RegimeSnapshot) -> None: + ''' + Publish the bridge's completed, current source state. + + ''' + adapter.submit(snapshot) + if ( + on_snapshot is not None + and source_snapshot_is_current( + snapshot, + now=time.time(), + max_source_age=max_source_age, + ) + ): + on_snapshot(snapshot) + started.set() + + def invalidate(revision: int) -> None: + ''' + Revoke graphics and optional model work before replay. + + ''' + adapter.invalidate(revision) + if on_invalidate is not None: + on_invalidate(revision) + + scope: trio.CancelScope + with trio.CancelScope() as scope: + self._multi_scope = scope + try: + stream: AsyncIterator[dict] + async with open_sample_stream( + period_s=tracker.base_period, + ) as stream: + await consume_regime_source( + self.flume.rt_shm, + stream, + tracker, + submit, + invalidate=invalidate, + ) + finally: + self._multi_scope = None + invalidate(adapter.revision + 1) + # Wake a bootstrap waiter if the source closes empty. + started.set() + + def refresh_trend_overlays( + self, + shm_token: NDToken, + *, + render: bool, + + ) -> None: + ''' + Route a destination rewrite to this admin's attached charts. + + ''' + charts: set[ChartPlotWidget] = set() + chart: ChartPlotWidget + overlay: TrendOverlay + for chart, overlay in self._trend_overlays: + if chart in charts: + continue + charts.add(chart) + update_trend_overlays( + chart, + shm_token=shm_token, + force=True, + render=render, + ) + + def close_trend_overlays(self) -> None: + ''' + Remove this admin's layers while preserving sibling owners. + + ''' + if self._multi_scope is not None: + self._multi_scope.cancel() + if self._multi_overlay is not None: + self._multi_overlay.remove() + self._multi_overlay = None + self._multi_tracker = None + chart: ChartPlotWidget + overlay: TrendOverlay + for chart, overlay in self._trend_overlays: + layers: dict[ + tuple[str, NDToken], list[TrendOverlay] + ] = chart._trend_overlays + key: tuple[str, NDToken] = ( + overlay.fqme, overlay.source._token, + ) + owned: list[TrendOverlay] = layers.get(key, []) + if overlay in owned: + owned.remove(overlay) + if not owned: + layers.pop(key, None) + overlay.remove() + self._trend_overlays.clear() + async def open_chain( self, @@ -540,18 +931,24 @@ class FspAdmin: async for msg in stream: info = msg.get('fsp_update') if info: + token: NDToken = NDToken.from_msg( + dict(info['key']), + ) + visible: bool = not self.linked.isHidden() # if the chart isn't hidden try to update # the data on screen. - if not self.linked.isHidden(): + if visible: log.debug( f'Re-syncing graphics for fsp: {ns_path}' ) update_fsp_vizs( self.linked, - shm_token=NDToken.from_msg( - dict(info['key']), - ), + shm_token=token, ) + self.refresh_trend_overlays( + token, + render=visible, + ) else: log.info(f'recved unexpected fsp engine msg: {msg}') @@ -710,6 +1107,7 @@ async def open_fsp_admin( # ??TODO, does this *need* to be inside a finally? finally: + admin.close_trend_overlays() # terminate all tasks via signals for key, entry in admin._registry.items(): _, _, event = entry @@ -1027,6 +1425,8 @@ async def start_fsp_displays( flume: Flume, # group_status_key: str, loglevel: str, + show_trend_calculus: bool = False, + trend_timeframes: tuple[float, ...]|None = None, ) -> None: ''' @@ -1072,6 +1472,13 @@ async def start_fsp_displays( flume, ) as admin, ): + if trend_timeframes is not None: + await admin.open_multi_timeframe_trends( + periods=tuple(trend_timeframes), + ) + elif show_trend_calculus: + await admin.open_trend_overlay(loglevel=loglevel) + statuses: list[trio.Event] = [] for target, conf in fsp_conf.items(): started: trio.Event = await admin.open_fsp_chart( @@ -1091,4 +1498,7 @@ async def start_fsp_displays( profiler(f'attached to fsp portal: {target}') # status_cb() + if show_trend_calculus or trend_timeframes is not None: + await trio.sleep_forever() + # blocks on nursery until all fsp actors complete diff --git a/piker/ui/_trend.py b/piker/ui/_trend.py new file mode 100644 index 00000000..d8fcce1e --- /dev/null +++ b/piker/ui/_trend.py @@ -0,0 +1,1009 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Bounded, chart-local Trend Calculus reversal and regime overlays. + +''' +from __future__ import annotations + +from collections.abc import Sequence +from datetime import datetime, UTC +from operator import index +from typing import Literal, TYPE_CHECKING + +import numpy as np +import pyqtgraph as pg +from pyqtgraph.Qt import internals + +from piker.types import Struct +from .qt import Qt, QtGui, QtWidgets, QRectF, QPointF + +if TYPE_CHECKING: + from tractor.ipc._shm import ShmArray + from piker.fsp._jev import JevClassification, RegimeAnswer + from ._dataviz import Viz + + +class ReversalPoint(Struct, frozen=True): + ''' + Locate a confirmed pivot without losing its detection time. + + ``timestamp_convention`` distinguishes the scalar FSP's bar-open + coordinate from aggregated snapshots' interval-end coordinate. + The reversal becomes knowable only at ``detected_at``, the close + of its detection bar. A marker is historical annotation, not a + trading signal available at the pivot itself. + + ''' + x: float + price: float + pivot_time: float + detected_at: float + kind: Literal['top', 'bottom'] + order: int + period_s: float + timestamp_convention: Literal[ + 'bar_start', 'interval_end', + ] = 'bar_start' + + +class RegimeInterval(Struct, frozen=True): + ''' + Annotate one explicitly sourced, probabilistic regime interval. + + ``observed_at`` records when a classifier response became + available. Graphics clamp ``start_time`` to this timestamp; + a delayed response cannot color earlier bars retroactively. + ``end_time`` is an exclusive, caller-owned event-time bound. + + ''' + start_time: float + end_time: float + label: str + probability: float = 1. + source: str = 'trend-calculus' + observed_at: float|None = None + + +def classification_interval( + result: JevClassification, + *, + fqme: str, + revision: int, + period_s: float, + end_time: float, + +) -> RegimeInterval|None: + ''' + Convert matching Jev evidence into a causal chart annotation. + + The chart owner supplies its current source revision and display + horizon. Wrong-market, stale-revision, future and cross-timeframe + answers do not attach. The provider caller owns expiry and + monotonic sequence filtering; this adapter makes no API calls. + + ''' + _positive_period(period_s) + if not np.isfinite(end_time): + raise ValueError('end_time must be finite') + if result.market != fqme or result.revision != revision: + return None + if ( + not np.isfinite(result.source_as_of) + or not np.isfinite(result.received_at) + ): + return None + available: float = max(result.source_as_of, result.received_at) + if available >= end_time: + return None + + answer: RegimeAnswer + for answer in result.answers: + if answer.period != period_s: + continue + if not 0 <= answer.probability <= 1: + return None + model: str = result.model + return RegimeInterval( + start_time=result.source_as_of, + end_time=end_time, + label=answer.label, + probability=answer.probability, + source=f'typesafe.ai/{model}', + observed_at=available, + ) + return None + + +def _positive_integer(value: int, name: str) -> int: + ''' + Validate allocation limits before attaching any graphics items. + + ''' + if isinstance(value, (bool, np.bool_)): + raise ValueError(f'{name} must be a positive integer') + try: + value = index(value) + except TypeError: + raise ValueError( + f'{name} must be a positive integer', + ) from None + if value < 1: + raise ValueError(f'{name} must be a positive integer') + return value + + +def _positive_period(period_s: float) -> None: + ''' + Require a finite positive sampling period for chart geometry. + + ''' + if not np.isfinite(period_s) or period_s <= 0: + raise ValueError('period must be finite and positive') + + +def _chart_times(array: np.ndarray) -> np.ndarray: + ''' + Validate monotonic chart timestamps before binary searches. + + ''' + times: np.ndarray = array['time'] + if ( + not np.isfinite(times).all() + or np.any(np.diff(times) <= 0) + ): + raise ValueError('chart times must be finite and increasing') + return times + + +def _time_gaps(times: np.ndarray, period_s: float) -> np.ndarray: + ''' + Detect absent buckets without treating epoch rounding as gaps. + + Decimal subsecond periods can alternate by a few timestamp ULPs + when represented as float64 Unix epochs. Only that representable + rounding error is tolerated, not a fraction of the timeframe. + + ''' + slack: np.ndarray = ( + np.spacing(np.abs(times[:-1])) + + np.spacing(np.abs(times[1:])) + ) + return np.diff(times) > period_s + slack + + +def times_to_x( + array: np.ndarray, + times: np.ndarray, + period_s: float, + index_field: Literal['index', 'time'] = 'index', + +) -> np.ndarray: + ''' + Map event times to chart coordinates; absent bars produce NaNs. + + ``array['index']`` contains absolute SHM coordinates, not offsets + into a copied slice. Sub-bar timestamps retain their fractional + positions. Times in missing data never snap onto the next bar. + + ''' + _positive_period(period_s) + if index_field not in ('index', 'time'): + raise ValueError('index_field must be index or time') + queries: np.ndarray = np.asarray(times, dtype=float) + result: np.ndarray = np.full(queries.shape, np.nan) + if not len(array): + return result + + bar_times: np.ndarray = _chart_times(array) + positions: np.ndarray = np.searchsorted( + bar_times, queries, side='right', + ) - 1 + safe: np.ndarray = positions.clip(0, len(array) - 1) + offsets: np.ndarray = queries - bar_times[safe] + valid: np.ndarray = ( + np.isfinite(queries) + & (positions >= 0) + & (offsets >= 0) + & (offsets < period_s) + ) + if index_field == 'time': + result[valid] = queries[valid] + else: + result[valid] = ( + array['index'][safe[valid]] + + offsets[valid] / period_s + ) + return result + + +def reversal_points( + source: np.ndarray, + chart: np.ndarray, + *, + source_period_s: float, + chart_period_s: float, + order: int = 1, + index_field: Literal['index', 'time'] = 'index', + max_points: int = 4096, + +) -> list[ReversalPoint]: + ''' + Extract confirmed pivots from one FSP order and timeframe. + + The final source row is mutable and is always excluded. Each + top/bottom field is sparse on its detection row; its accompanying + timestamp identifies the earlier pivot bar. Higher reversal + orders are distinct from sampling timeframes. + + ''' + _positive_period(source_period_s) + if not 1 <= order <= 4 or max_points < 1: + raise ValueError('order must be 1..4 and limit positive') + rows: np.ndarray = source[:-1] + points: list[ReversalPoint] = [] + kind: Literal['top', 'bottom'] + for kind in ('top', 'bottom'): + prices: np.ndarray = rows[f'{kind}_{order}'] + times: np.ndarray = rows[f'{kind}_time_{order}'] + detections: np.ndarray = rows['time'] + source_period_s + valid: np.ndarray = ( + np.isfinite(prices) + & np.isfinite(times) + & np.isfinite(detections) + & (times <= rows['time']) + ) + positions: np.ndarray = np.flatnonzero(valid)[-max_points:] + xs: np.ndarray = times_to_x( + chart, times[positions], chart_period_s, index_field, + ) + position: np.integer + x: np.floating + for position, x in zip(positions, xs, strict=True): + if not np.isfinite(x): + continue + points.append(ReversalPoint( + x=float(x), + price=float(prices[position]), + pivot_time=float(times[position]), + detected_at=float(detections[position]), + kind=kind, + order=order, + period_s=source_period_s, + )) + + points.sort(key=lambda point: point.detected_at) + return points[-max_points:] + + +def trend_intervals( + source: np.ndarray, + period_s: float, + *, + order: int = 1, + max_intervals: int = 1024, + +) -> list[RegimeInterval]: + ''' + Describe runs of finalized direction from their detection times. + + Every run starts at the close of its first contributing bar. + The mutable final row contributes only the display horizon, not + a trend value. NaNs, warm-up rows and missing time buckets break + runs. Rectangles cover the current display bar after a confirmed + classification until another finalized observation replaces it. + + ''' + _positive_period(period_s) + if not 1 <= order <= 4 or max_intervals < 1: + raise ValueError('order must be 1..4 and limit positive') + if len(source) < 2: + return [] + + _chart_times(source) + rows: np.ndarray = source[:-1] + field: str = 'trend' if order == 1 else f'trend_{order}' + directions: np.ndarray = rows[field] + valid: np.ndarray = ( + np.isfinite(directions) + & (np.abs(directions) == 1) + & (rows['ready'] >= order) + ) + changed: np.ndarray = np.ones(len(rows), dtype=bool) + changed[1:] = ( + (directions[1:] != directions[:-1]) + | (valid[1:] != valid[:-1]) + | _time_gaps(rows['time'], period_s) + ) + starts: np.ndarray = np.flatnonzero(changed) + ends: np.ndarray = np.r_[starts[1:], len(rows)] + retained: np.ndarray = valid[starts] + starts = starts[retained][-max_intervals:] + ends = ends[retained][-max_intervals:] + spans: list[RegimeInterval] = [] + start: np.integer + end: np.integer + for start, end in zip(starts, ends, strict=True): + if not valid[start]: + continue + start_time: float = float(rows['time'][start] + period_s) + # Never extend across a missing source bucket. One additional + # period paints the current chart bar's cell, not a forecast. + end_time: float = float(min( + source['time'][end] + period_s, + rows['time'][end - 1] + 2 * period_s, + )) + if end_time <= start_time: + continue + spans.append(RegimeInterval( + start_time=start_time, + end_time=end_time, + label=( + 'uptrend' if directions[start] > 0 else 'downtrend' + ), + )) + return spans[-max_intervals:] + + +def regime_rectangles( + intervals: Sequence[RegimeInterval], + chart: np.ndarray, + period_s: float, + *, + index_field: Literal['index', 'time'] = 'index', + max_rectangles: int = 1024, + +) -> tuple[np.ndarray, list[RegimeInterval]]: + ''' + Build price-envelope rectangles, clipped to available chart bars. + + Each row is ``(x, y, width, height)``. Missing buckets split a + rectangle so no background fill asserts data in a feed gap. + Returned metadata preserves classifier identity and probability. + + ''' + _positive_period(period_s) + if index_field not in ('index', 'time'): + raise ValueError('index_field must be index or time') + if max_rectangles < 1: + raise ValueError('max_rectangles must be positive') + if not len(chart): + return np.empty((0, 4)), [] + + times: np.ndarray = _chart_times(chart) + bar_ends: np.ndarray = times + period_s + geometry: list[tuple[float, float, float, float]] = [] + metadata: list[RegimeInterval] = [] + interval: RegimeInterval + for interval in intervals[-max_rectangles:]: + start_time: float = interval.start_time + if interval.observed_at is not None: + if not np.isfinite(interval.observed_at): + raise ValueError('observed_at must be finite') + start_time = max(start_time, interval.observed_at) + end_time: float = interval.end_time + if ( + not np.isfinite(start_time) + or not np.isfinite(end_time) + or not np.isfinite(interval.probability) + or not 0 <= interval.probability <= 1 + ): + raise ValueError('invalid regime bounds or probability') + if end_time <= start_time: + continue + + left: int = int(np.searchsorted( + bar_ends, start_time, side='right', + )) + right: int = int(np.searchsorted( + times, end_time, side='left', + )) + if left >= right: + continue + + breaks: np.ndarray = np.flatnonzero( + _time_gaps(times[left:right], period_s) + ) + left + 1 + starts: np.ndarray = np.r_[left, breaks] + ends: np.ndarray = np.r_[breaks, right] + first: np.integer + last: np.integer + for first, last in zip( + starts[-max_rectangles:], + ends[-max_rectangles:], + strict=True, + ): + rows: np.ndarray = chart[first:last] + finite: np.ndarray = ( + np.isfinite(rows['low']) + & np.isfinite(rows['high']) + ) + if not finite.any(): + continue + low: float = float(rows['low'][finite].min()) + high: float = float(rows['high'][finite].max()) + if high < low: + continue + start: float = max(start_time, float(times[first])) + end: float = min( + end_time, float(times[last - 1]) + period_s, + ) + x1: float + x2: float + if index_field == 'time': + x1, x2 = start, end + else: + x1 = float(chart['index'][first]) + ( + start - times[first] + ) / period_s + x2 = float(chart['index'][last - 1]) + ( + end - times[last - 1] + ) / period_s + if x2 <= x1: + continue + height: float = max(high - low, abs(low) * 1e-6, 1e-12) + geometry.append((x1, low, x2 - x1, height)) + metadata.append(interval) + + return ( + np.asarray(geometry[-max_rectangles:]).reshape(-1, 4), + metadata[-max_rectangles:], + ) + + +_colors: dict[str, str] = { + 'uptrend': '#47c99b', + 'downtrend': '#ed6a7a', + 'ranging': '#d5b65b', + 'range': '#d5b65b', + 'transition': '#b58bdd', + 'volatile': '#b58bdd', +} + + +class RegimeRectangles(pg.GraphicsObject): + ''' + Render many regime rectangles using one scene item. + + ``RegimeRectangles._batches`` groups primitive arrays by color + and opacity. Rectangle count is bounded by ``TrendOverlay``; + no per-observation ``QGraphicsItem`` objects are allocated. + + ''' + def __init__(self, period_s: float|None = None) -> None: + ''' + Initialize an empty price-background annotation batch. + + ''' + super().__init__() + self.period_s: float|None = period_s + self._batches: list[ + tuple[internals.PrimitiveArray, QtGui.QBrush] + ] = [] + self._bounds: QRectF = QRectF() + self.intervals: list[RegimeInterval] = [] + self.rectangles: np.ndarray = np.empty((0, 4)) + self.setZValue(-5) + self.setAcceptedMouseButtons(Qt.MouseButton.NoButton) + self.setAcceptHoverEvents(True) + + def set_data( + self, + rectangles: np.ndarray, + intervals: list[RegimeInterval], + + ) -> None: + ''' + Replace ``RegimeRectangles.rectangles`` and Qt item bounds. + + ''' + self.prepareGeometryChange() + self.rectangles = rectangles + self.intervals = intervals + self._batches.clear() + self._bounds = QRectF() + groups: dict[tuple[str, int], list[int]] = {} + index: int + interval: RegimeInterval + for index, interval in enumerate(intervals): + color: str = _colors.get(interval.label, '#92a1b5') + alpha: int = round(12 + 48 * interval.probability) + groups.setdefault((color, alpha), []).append(index) + + key: tuple[str, int] + positions: list[int] + for key, positions in groups.items(): + color: str + alpha: int + color, alpha = key + primitives: internals.PrimitiveArray = ( + internals.PrimitiveArray(QRectF, 4) + ) + primitives.resize(len(positions)) + primitives.ndarray()[:] = rectangles[positions] + brush_color: QtGui.QColor = pg.mkColor(color) + brush_color.setAlpha(alpha) + self._batches.append(( + primitives, pg.mkBrush(brush_color), + )) + + if len(rectangles): + left: float = float(rectangles[:, 0].min()) + bottom: float = float(rectangles[:, 1].min()) + right: float = float(( + rectangles[:, 0] + rectangles[:, 2] + ).max()) + top: float = float(( + rectangles[:, 1] + rectangles[:, 3] + ).max()) + self._bounds = QRectF( + left, bottom, right - left, top - bottom, + ) + self.update() + + def boundingRect(self) -> QRectF: + ''' + Return the data-coordinate envelope of current rectangles. + + ''' + return self._bounds + + def tooltip_at(self, position: QPointF) -> str: + ''' + Describe the interval beneath a data-coordinate position. + + Native direction is deterministic, so it has no model + confidence percentage. External classifications retain their + probability and explicit provider identity in the tooltip. + + ''' + rows: np.ndarray = self.rectangles + x: float = position.x() + y: float = position.y() + hits: np.ndarray = np.flatnonzero( + (rows[:, 0] <= x) + & (rows[:, 0] + rows[:, 2] >= x) + & (rows[:, 1] <= y) + & (rows[:, 1] + rows[:, 3] >= y) + ) + if not len(hits): + return '' + interval: RegimeInterval = self.intervals[int(hits[-1])] + label: str = interval.label + source: str = interval.source + if source != 'trend-calculus': + probability: float = interval.probability + label = f'{label}: {probability:.1%}' + if self.period_s is not None: + period_s: float = self.period_s + label = f'{label} ({period_s:g}s)' + available: float = max( + interval.start_time, + interval.start_time + if interval.observed_at is None + else interval.observed_at, + ) + start: str = datetime.fromtimestamp( + available, UTC, + ).isoformat(timespec='milliseconds') + end: str = datetime.fromtimestamp( + interval.end_time, UTC, + ).isoformat(timespec='milliseconds') + return ( + f'{label}\n' + f'{source}\n' + f'Available: {start}\n' + f'Until: {end}' + ) + + def hoverMoveEvent( + self, + event: QtWidgets.QGraphicsSceneHoverEvent, + + ) -> None: + ''' + Show source and probability without allocating label items. + + ''' + self.setToolTip(self.tooltip_at(event.pos())) + + def paint( + self, + painter: QtGui.QPainter, + option: QtWidgets.QStyleOptionGraphicsItem, + widget: QtWidgets.QWidget|None = None, + + ) -> None: + ''' + Paint each color batch with a single Qt primitive call. + + ''' + painter.setPen(Qt.PenStyle.NoPen) + primitives: internals.PrimitiveArray + brush: QtGui.QBrush + for primitives, brush in self._batches: + painter.setBrush(brush) + painter.drawRects(*primitives.drawargs()) + + +class TrendOverlay: + ''' + Attach one market, source timeframe and reversal order to a Viz. + + ``TrendOverlay.update()`` follows completed FSP samples and + mutable chart prices. The owner must pass ``force=True`` for a + matching destination history revision, including equal-bound gap + repairs, and for historical chart rewrites. + ``TrendOverlay.remove()`` releases both Qt items and may be + called repeatedly during owner cancellation. + + Several instances can share a price chart while retaining + independent timeframes, orders and classifier provenance. + + ''' + def __init__( + self, + viz: Viz, + source: ShmArray, + *, + fqme: str, + source_period_s: float, + chart_period_s: float, + order: int = 1, + max_points: int = 4096, + max_rectangles: int = 1024, + + ) -> None: + ''' + Attach bounded marker and rectangle batches to ``viz.plot``. + + ''' + _positive_period(source_period_s) + _positive_period(chart_period_s) + order = _positive_integer(order, 'order') + max_points = _positive_integer(max_points, 'max_points') + max_rectangles = _positive_integer( + max_rectangles, 'max_rectangles', + ) + if order > 4: + raise ValueError('order must be 1..4') + self.viz: Viz = viz + self.source: ShmArray = source + self.fqme: str = fqme + self.source_period_s: float = source_period_s + self.chart_period_s: float = chart_period_s + self.order: int = order + self.max_points: int = max_points + self.max_rectangles: int = max_rectangles + self._signature: tuple|None = None + self._source_signature: tuple|None = None + self._native_intervals: list[RegimeInterval] = [] + self._live_bounds: list[tuple[int, float, float]] = [] + self._live_prices_valid: bool = False + self._external: tuple[RegimeInterval, ...]|None = None + self._removed: bool = False + self.points: list[ReversalPoint] = [] + self._pens: dict[str, QtGui.QPen] = { + 'top': pg.mkPen(_colors['downtrend']), + 'bottom': pg.mkPen(_colors['uptrend']), + } + self._brushes: dict[str, QtGui.QBrush] = { + 'top': pg.mkBrush(_colors['downtrend']), + 'bottom': pg.mkBrush(_colors['uptrend']), + } + self.markers: pg.ScatterPlotItem = pg.ScatterPlotItem( + pxMode=True, + size=9 + 2 * order + 2 * min(4, max( + 0, np.log2(source_period_s / chart_period_s), + )), + hoverable=True, + tip=self._marker_tip, + ) + self.markers.setZValue(20 + order) + self.rectangles: RegimeRectangles = RegimeRectangles( + period_s=source_period_s, + ) + viz.plot.addItem(self.rectangles, ignoreBounds=True) + viz.plot.addItem(self.markers, ignoreBounds=True) + + @staticmethod + def _marker_tip( + x: float, + y: float, + data: ReversalPoint, + + ) -> str: + ''' + Distinguish pivot coordinates from causal detection time. + + ''' + kind: str = data.kind + order: int = data.order + period: float = data.period_s + pivot: str = datetime.fromtimestamp( + data.pivot_time, UTC, + ).isoformat(timespec='milliseconds') + detected: str = datetime.fromtimestamp( + data.detected_at, UTC, + ).isoformat(timespec='milliseconds') + pivot_label: str = ( + 'Pivot close' + if data.timestamp_convention == 'interval_end' + else 'Pivot bar' + ) + return ( + f'{period:g}s / order {order}: {kind}\n' + f'{pivot_label}: {pivot}\n' + f'Confirmed: {detected}' + ) + + def set_regimes( + self, + intervals: Sequence[RegimeInterval]|None, + + ) -> None: + ''' + Select external classifications, or restore native regimes. + + ``None`` restores deterministic direction rectangles. An + empty sequence clears external regimes while retaining + reversal markers. Each classifier interval retains its + source label and probability in ``rectangles.intervals``. + + ''' + self._external = ( + None + if intervals is None + else tuple(intervals[-self.max_rectangles:]) + ) + self._signature = None + self.update() + + def invalidate(self) -> None: + ''' + Defer a full history rebuild until the next visible update. + + ''' + self._signature = None + self._source_signature = None + + def _update_markers( + self, + source: np.ndarray, + chart: np.ndarray, + + ) -> None: + ''' + Replace confirmed marker data while reusing its Qt styles. + + ''' + self.points = reversal_points( + source, + chart, + source_period_s=self.source_period_s, + chart_period_s=self.chart_period_s, + order=self.order, + index_field=self.viz.index_field, + max_points=self.max_points, + ) + self._set_points(self.points) + + def _set_points(self, points: list[ReversalPoint]) -> None: + ''' + Publish already mapped pivots to the existing marker batch. + + ''' + self.points = points + spots: list[dict[str, object]] = [] + point: ReversalPoint + for point in self.points: + spots.append({ + 'pos': (point.x, point.price), + 'symbol': ( + 't' + if point.kind == 'top' + else 't1' + ), + 'pen': self._pens[point.kind], + 'brush': self._brushes[point.kind], + 'data': point, + }) + self.markers.setData(spots) + + def _prepare_evidence( + self, + source: np.ndarray, + chart: np.ndarray, + + ) -> None: + ''' + Refresh native SHM evidence independently of Qt rendering. + + Snapshot consumers override this boundary and reuse the + bounded marker and rectangle rendering lifecycle below. + + ''' + self._update_markers(source, chart) + self._native_intervals = trend_intervals( + source, + self.source_period_s, + order=self.order, + max_intervals=self.max_rectangles, + ) + + def _cache_live_bounds(self, chart: np.ndarray) -> None: + ''' + Retain finalized price bounds only for active rectangles. + + ``TrendOverlay._live_bounds`` lets quote-time updates replace + the mutable bar's contribution without rescanning history. + It handles current high/low revisions in either direction. + + ''' + self._live_bounds.clear() + if not len(chart): + return + coords: np.ndarray = chart[self.viz.index_field] + last_x: float = float(coords[-1]) + step: float = ( + self.chart_period_s + if self.viz.index_field == 'time' + else 1. + ) + geometry: np.ndarray = self.rectangles.rectangles + active: np.ndarray = np.flatnonzero( + (geometry[:, 0] < last_x + step) + & (geometry[:, 0] + geometry[:, 2] > last_x) + ) + position: np.integer + for position in active: + first: int = max(0, int(np.searchsorted( + coords, geometry[position, 0], side='right', + )) - 1) + rows: np.ndarray = chart[first:-1] + valid: np.ndarray = ( + np.isfinite(rows['low']) + & np.isfinite(rows['high']) + ) + low: float = ( + float(rows['low'][valid].min()) + if valid.any() + else np.inf + ) + high: float = ( + float(rows['high'][valid].max()) + if valid.any() + else -np.inf + ) + self._live_bounds.append((int(position), low, high)) + + def _update_live_rectangles(self, chart: np.ndarray) -> bool: + ''' + Replace the live candle envelope without rebuilding history. + + ''' + geometry: np.ndarray = self.rectangles.rectangles.copy() + changed: bool = False + position: int + closed_low: float + closed_high: float + for ( + position, + closed_low, + closed_high, + ) in self._live_bounds: + low: float = min(closed_low, float(chart['low'][-1])) + high: float = max(closed_high, float(chart['high'][-1])) + height: float = max(high - low, abs(low) * 1e-6, 1e-12) + if ( + geometry[position, 1] != low + or geometry[position, 3] != height + ): + geometry[position, 1] = low + geometry[position, 3] = height + changed = True + if changed: + self.rectangles.set_data( + geometry, self.rectangles.intervals, + ) + return changed + + def update(self, force: bool = False) -> bool: + ''' + Refresh only this layer; return whether geometry changed. + + This gate is a paint optimization, not a revision detector. + Explicit history notifications must pass ``force=True``. + + ''' + if self._removed: + return False + source: np.ndarray = self.source.array + chart: np.ndarray = self.viz.shm.array + source_signature: tuple = ( + len(source), + source[:1].tobytes(), + source[-2:-1].tobytes(), + source['time'][-1:].tobytes(), + len(chart), + chart['time'][:1].tobytes(), + chart['time'][-1:].tobytes(), + chart['index'][:1].tobytes(), + chart['index'][-1:].tobytes(), + self.viz.index_field, + ) + signature: tuple = ( + source_signature, + chart['low'][-1:].tobytes(), + chart['high'][-1:].tobytes(), + ) + if not force and signature == self._signature: + return False + + prices_valid: bool = bool( + len(chart) + and np.isfinite(chart['low'][-1]) + and np.isfinite(chart['high'][-1]) + and chart['low'][-1] <= chart['high'][-1] + ) + if ( + not force + and self._signature is not None + and source_signature == self._source_signature + and prices_valid + and self._live_prices_valid + ): + changed: bool = self._update_live_rectangles(chart) + self._signature = signature + return changed + + if force or source_signature != self._source_signature: + self._prepare_evidence(source, chart) + intervals: Sequence[RegimeInterval] = ( + self._native_intervals + if self._external is None + else self._external + ) + geometry: np.ndarray + metadata: list[RegimeInterval] + geometry, metadata = regime_rectangles( + intervals, + chart, + self.chart_period_s, + index_field=self.viz.index_field, + max_rectangles=self.max_rectangles, + ) + self.rectangles.set_data(geometry, metadata) + self._cache_live_bounds(chart) + self._live_prices_valid = prices_valid + self._source_signature = source_signature + self._signature = signature + return True + + def remove(self) -> None: + ''' + Release this owner's scene items without touching siblings. + + ''' + if self._removed: + return + self.viz.plot.removeItem(self.markers) + self.viz.plot.removeItem(self.rectangles) + self._removed = True + self.points.clear() diff --git a/piker/ui/_trend_mtf.py b/piker/ui/_trend_mtf.py new file mode 100644 index 00000000..d8751656 --- /dev/null +++ b/piker/ui/_trend_mtf.py @@ -0,0 +1,547 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Native multiple-timeframe annotations from finalized feed snapshots. + +''' +from __future__ import annotations + +from collections import deque +from collections.abc import Callable +from math import isfinite +from typing import TYPE_CHECKING + +from msgspec.structs import replace +import numpy as np + +from piker.fsp._regime import ( + RegimeSnapshot, + TimeframeRegime, + _period_us, +) +from ._trend import ( + RegimeInterval, + ReversalPoint, + TrendOverlay, + _positive_integer, + times_to_x, +) + +if TYPE_CHECKING: + from tractor.ipc._shm import ShmArray + from piker.fsp._jev import JevClassification, RegimeAnswer + from piker.fsp._regime import TrendReversal + from ._dataviz import Viz + + +def _normalize_periods( + base_period: float, + periods: tuple[float, ...], + +) -> tuple[float, ...]: + ''' + Match the shared tracker's canonical microsecond durations. + + ''' + base_us: int = _period_us(base_period) + period: float # noqa: F842 + period_us: tuple[int, ...] = tuple( + sorted(_period_us(period) for period in periods) + ) + if not 1 <= len(period_us) <= 16: + raise ValueError('configure 1..16 unique timeframes') + if len(set(period_us)) != len(period_us): + raise ValueError('timeframes must be unique') + value: int # noqa: F842 + if any(value % base_us for value in period_us): + raise ValueError('timeframes require whole base multiples') + return tuple(value / 1_000_000 for value in period_us) + + +class _SnapshotLayer(TrendOverlay): + ''' + Render one real aggregated timeframe using existing Qt batches. + + The source is the actual base OHLC SHM. ``_SnapshotLayer._frame`` + supplies computed evidence; no pretend FSP SHM or repeated TC + computation is needed. Snapshot pivots retain interval-end + timestamps while x positions identify their final base bar. + + ''' + def __init__( + self, + viz: Viz, + source: ShmArray, + *, + fqme: str, + source_period_s: float, + chart_period_s: float, + order: int = 1, + max_points: int = 4096, + max_rectangles: int = 1024, + + ) -> None: + ''' + Initialize a bounded, empty timeframe layer. + + ''' + super().__init__( + viz, + source, + fqme=fqme, + source_period_s=source_period_s, + chart_period_s=chart_period_s, + order=order, + max_points=max_points, + max_rectangles=max_rectangles, + ) + self._frame: TimeframeRegime|None = None + self._base_period: float = self.chart_period_s + self._runs: deque[RegimeInterval] = deque( + maxlen=self.max_rectangles, + ) + self._active: bool = False + + def clear_evidence(self) -> None: + ''' + Revoke old source evidence immediately on revision or gap. + + ''' + self._frame = None + self._runs.clear() + self._active = False + self._external = None + self._native_intervals.clear() + self._set_points([]) + self.rectangles.set_data(np.empty((0, 4)), []) + self.invalidate() + + def accept_frame( + self, + frame: TimeframeRegime, + base_period: float, + horizon: float, + + ) -> None: + ''' + Retain only causal live runs supported by received snapshots. + + Bootstrap starts at the latest proven direction, since the + snapshot has no earlier trend history. If coalescing skips + complete buckets, end the previous run at its next possible + decision and leave the unknown interval unpainted. + + ''' + previous: TimeframeRegime|None = self._frame + self._frame = frame + self._base_period = base_period + level: int = self.order - 1 + ready: bool = ( + frame.as_of is not None + and len(frame.ready) > level + and len(frame.trends) > level + and frame.ready[level] + and abs(frame.trends[level]) == 1 + ) + label: str = ( + 'uptrend' if frame.trends[level] > 0 else 'downtrend' + ) if len(frame.trends) > level else '' + skipped: bool = ( + previous is not None + and frame.close_count > previous.close_count + 1 + ) + if self._active: + if skipped: + self._runs[-1] = replace( + self._runs[-1], + end_time=previous.as_of + frame.period, + ) + self._active = False + elif not ready or self._runs[-1].label != label: + self._runs[-1] = replace( + self._runs[-1], + end_time=( + frame.as_of + if frame.as_of is not None + else horizon - base_period + ), + ) + self._active = False + + if ready: + if self._active: + self._runs[-1] = replace( + self._runs[-1], end_time=horizon, + ) + else: + self._runs.append(RegimeInterval( + start_time=frame.as_of, + end_time=horizon, + label=label, + )) + self._active = True + self.invalidate() + + def _prepare_evidence( + self, + source: np.ndarray, + chart: np.ndarray, + + ) -> None: + ''' + Map immutable endpoint events to their actual source bars. + + ''' + frame: TimeframeRegime|None = self._frame + if frame is None: + self._set_points([]) + self._native_intervals = [] + return + + events: list[TrendReversal] = [] + event: TrendReversal + for event in frame.reversals: + if event.order == self.order: + events.append(event) + events = events[-self.max_points:] + times: np.ndarray = np.fromiter( + (event.time - self._base_period for event in events), + dtype=float, + count=len(events), + ) + xs: np.ndarray = times_to_x( + chart, + times, + self.chart_period_s, + self.viz.index_field, + ) + points: list[ReversalPoint] = [] + x: np.floating + for event, x in zip(events, xs, strict=True): + if not np.isfinite(x): + continue + points.append(ReversalPoint( + x=float(x), + price=event.price, + pivot_time=event.time, + detected_at=event.detected_time, + kind='bottom' if event.direction > 0 else 'top', + order=self.order, + period_s=frame.period, + timestamp_convention='interval_end', + )) + self._set_points(points) + self._native_intervals = list(self._runs) + + +class SnapshotTrendOverlay: + ''' + Consume one shared tracker into bounded per-timeframe graphics. + + Pass ``submit`` and ``invalidate`` directly to the native + ``consume_regime_source`` bridge. Source revision and gap segment + changes synchronously revoke stale annotations; old or duplicate + snapshot sequences cannot repaint newer evidence. + + ``submit_classification`` is an optional local Jev callback. + It performs no I/O and never reads credentials. Model rectangles + have a finite ``classification_ttl`` after response receipt and + stay absent until the displayed event clock reaches receipt. + + ''' + def __init__( + self, + viz: Viz, + source: ShmArray, + *, + fqme: str, + base_period: float, + periods: tuple[float, ...] = (1, 5, 15), + chart_period_s: float = 1, + order: int = 1, + history_limit: int = 256, + classification_ttl: float = 5, + max_source_lag: float|None = 2, + is_visible: Callable[[], bool]|None = None, + + ) -> None: + ''' + Allocate two scene items per configured sampling timeframe. + + ''' + periods = _normalize_periods(base_period, periods) + base_period = _period_us(base_period) / 1_000_000 + order = _positive_integer(order, 'order') + history_limit = _positive_integer( + history_limit, 'history_limit', + ) + if order > 4: + raise ValueError('order must be 1..4') + if history_limit > 256: + raise ValueError('history_limit must be 1..256') + if ( + not isfinite(classification_ttl) + or classification_ttl <= 0 + ): + raise ValueError('classification_ttl must be positive') + if ( + max_source_lag is not None + and + ( + not isfinite(max_source_lag) + or max_source_lag < 0 + ) + ): + raise ValueError('max_source_lag must be >=0 or None') + self.fqme: str = fqme + self.base_period: float = base_period + self.periods: tuple[float, ...] = periods + self.classification_ttl: float = classification_ttl + self.max_source_lag: float|None = max_source_lag + self.is_visible: Callable[[], bool] = ( + is_visible or (lambda: True) + ) + self.revision: int = 0 + self.sequence: int = -1 + self.segment: int|None = None + self.as_of: float|None = None + self._classification_sequence: int = -1 + self._model_runs: dict[float, deque[RegimeInterval]] = {} + self._model_expiry: dict[float, float] = {} + self._removed: bool = False + self.layers: dict[float, _SnapshotLayer] = {} + try: + period: float + for period in periods: + self.layers[period] = _SnapshotLayer( + viz, + source, + fqme=fqme, + source_period_s=period, + chart_period_s=chart_period_s, + order=order, + max_points=history_limit, + max_rectangles=history_limit, + ) + self._model_runs[period] = deque( + maxlen=history_limit, + ) + except BaseException: + self.remove() + raise + + def _clear_evidence(self) -> None: + ''' + Clear this owner's deterministic and classified history. + + ''' + layer: _SnapshotLayer + for layer in self.layers.values(): + layer.clear_evidence() + runs: deque[RegimeInterval] + for runs in self._model_runs.values(): + runs.clear() + self._model_expiry.clear() + self._classification_sequence = -1 + + def invalidate(self, revision: int) -> None: + ''' + Revoke an obsolete source generation before replay can yield. + + ''' + if self._removed or revision < self.revision: + return + self.revision = revision + self.sequence = -1 + self.segment = None + self.as_of = None + self._clear_evidence() + + def submit(self, snapshot: RegimeSnapshot) -> bool: + ''' + Accept the latest native evidence and refresh visible layers. + + ''' + if ( + self._removed + or snapshot.market != self.fqme + or snapshot.revision < self.revision + ): + return False + if snapshot.base_period != self.base_period: + raise ValueError('snapshot base period differs') + if snapshot.revision > self.revision: + self.invalidate(snapshot.revision) + if snapshot.sequence <= self.sequence: + return False + if self.as_of is not None and snapshot.as_of < self.as_of: + return False + frame: TimeframeRegime # noqa: F842 + frames: dict[float, TimeframeRegime] = { + frame.period: frame for frame in snapshot.timeframes + } + if not set(self.periods) <= frames.keys(): + raise ValueError('snapshot is missing a timeframe') + if ( + self.segment is not None + and snapshot.segment != self.segment + ): + self._clear_evidence() + self.segment = snapshot.segment + self.sequence = snapshot.sequence + self.as_of = snapshot.as_of + horizon: float = snapshot.as_of + self.base_period + period: float + layer: _SnapshotLayer + for period, layer in self.layers.items(): + layer.accept_frame( + frames[period], self.base_period, horizon, + ) + self._extend_model(period, horizon) + if self.is_visible(): + layer.update() + return True + + def _extend_model(self, period: float, horizon: float) -> None: + ''' + Extend the active model interval only through its expiry. + + ''' + runs: deque[RegimeInterval] = self._model_runs[period] + if not runs: + return + runs[-1] = replace( + runs[-1], + end_time=min(horizon, self._model_expiry[period]), + ) + layer: _SnapshotLayer = self.layers[period] + layer._external = tuple(runs) + layer._signature = None + + def submit_classification( + self, + result: JevClassification, + + ) -> bool: + ''' + Attach a matching, newly received Jev result without API I/O. + + Older requests may finish after newer chart snapshots; source + sequence must increase between accepted model responses and + never exceed the chart's known sequence. Revision and segment + must match exactly. Caller-owned provider freshness checks + still apply before invoking this callback. + + ''' + if ( + self._removed + or self.as_of is None + or result.market != self.fqme + or result.revision != self.revision + or result.segment != self.segment + or result.sequence <= self._classification_sequence + or result.sequence > self.sequence + or not isfinite(result.source_as_of) + or result.source_as_of > self.as_of + or not isfinite(result.received_at) + or ( + result.received_at + self.classification_ttl + <= self.as_of + ) + ): + return False + if ( + self.max_source_lag is not None + and ( + self.as_of - result.source_as_of + > self.max_source_lag + ) + ): + return False + answers: list[RegimeAnswer] = [] + answer: RegimeAnswer + for answer in result.answers: + if answer.period not in self.layers: + continue + if not isfinite(answer.probability): + return False + if not 0 <= answer.probability <= 1: + return False + answers.append(answer) + if not answers: + return False + + available: float = max( + result.source_as_of, result.received_at, + ) + horizon: float = self.as_of + self.base_period + expiry: float = available + self.classification_ttl + model: str = result.model + for answer in answers: + period: float = answer.period + runs: deque[RegimeInterval] = self._model_runs[period] + if runs: + runs[-1] = replace( + runs[-1], + end_time=min( + available, self._model_expiry[period], + ), + ) + runs.append(RegimeInterval( + start_time=result.source_as_of, + end_time=min(horizon, expiry), + label=answer.label, + probability=answer.probability, + source=f'typesafe.ai/{model}', + observed_at=available, + )) + self._model_expiry[period] = expiry + layer: _SnapshotLayer = self.layers[period] + layer._external = tuple(runs) + layer._signature = None + if self.is_visible(): + layer.update() + self._classification_sequence = result.sequence + return True + + def remove(self) -> None: + ''' + Remove every timeframe owned by this adapter, idempotently. + + ''' + if self._removed: + return + layer: _SnapshotLayer + for layer in self.layers.values(): + layer.remove() + self._removed = True + + async def publish_classification( + self, + result: JevClassification, + + ) -> None: + ''' + Adapt the synchronous graphics callback to classifier.run(). + + No await occurs inside the mutation, so revision invalidation + cannot interleave halfway through a Qt update. + + ''' + self.submit_classification(result) diff --git a/tests/_inputs/trend_calculus_lua.json b/tests/_inputs/trend_calculus_lua.json new file mode 100644 index 00000000..34a57e97 --- /dev/null +++ b/tests/_inputs/trend_calculus_lua.json @@ -0,0 +1,876 @@ +{ + "source": "https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua", + "source_sha256": "c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0", + "source_license": "GPL-3.0-or-later; Copyright 2014-2015 ByteSumo Limited (Andrew J Morgan)", + "fixture_license": "AGPL-3.0-or-later", + "generated_at": "2026-10-05", + "runtime": "Lua 5.2.4", + "command": "lua trendcalculus.lua -n{window} -r -H -p1", + "stdin": "ticker,date,last header followed by x,{zero_based_row},{value} rows, LF-terminated", + "note": "Do not use -h: embedded CSV column mapping already consumes the header and -h drops the first data row.", + "detection": "Run every complete-window prefix through the unmodified CLI. Newly emitted rows get detected_index = prefix_length - 1. Assert output-prefix preservation.", + "event_fields": [ + "index", + "price", + "direction", + "detected_index" + ], + "cases": [ + { + "window": 2, + "values": [ + 0, + 1, + 2, + 3, + 4, + 5, + 4, + 7, + 9, + 12, + 9, + 6, + 4, + 1, + 0, + -2, + -1, + -3, + -3, + -3, + -6, + -9, + -8, + -6, + -8, + -9, + -8, + -9, + -7, + -8, + -9, + -10, + -7, + -6, + -6, + -6, + -7, + -7, + -5, + -4, + -6, + -6, + -9, + -7, + -7, + -6, + -4, + -1, + -1, + -4, + -3, + -4, + -6, + -4, + -6, + -6, + -9, + -9, + -7, + -6, + -4, + -7, + -5, + -7, + -5, + -2, + -1, + -3, + -3, + 0, + -3, + -3, + -1, + -4, + -2, + 1, + 1, + -1, + -4, + -1, + 0, + 1, + 1, + -1, + -2, + -1, + 1, + 3, + 6, + 8, + 8, + 8, + 10, + 8, + 8, + 5, + 5, + 2, + 1, + 1, + -2, + -1, + 1, + -2, + 1, + -1 + ], + "events": [ + [ + 9, + 12.0, + -1, + 11 + ], + [ + 21, + -9.0, + 1, + 23 + ], + [ + 23, + -6.0, + -1, + 25 + ], + [ + 27, + -9.0, + 1, + 29 + ], + [ + 28, + -7.0, + -1, + 31 + ], + [ + 31, + -10.0, + 1, + 33 + ], + [ + 34, + -6.0, + -1, + 37 + ], + [ + 36, + -7.0, + 1, + 39 + ], + [ + 39, + -4.0, + -1, + 41 + ], + [ + 42, + -9.0, + 1, + 45 + ], + [ + 48, + -1.0, + -1, + 49 + ], + [ + 56, + -9.0, + 1, + 59 + ], + [ + 60, + -4.0, + -1, + 63 + ], + [ + 63, + -7.0, + 1, + 65 + ], + [ + 69, + 0.0, + -1, + 71 + ], + [ + 70, + -3.0, + 1, + 73 + ], + [ + 72, + -1.0, + -1, + 73 + ], + [ + 73, + -4.0, + 1, + 75 + ], + [ + 76, + 1.0, + -1, + 79 + ], + [ + 78, + -4.0, + 1, + 81 + ], + [ + 81, + 1.0, + -1, + 83 + ], + [ + 84, + -2.0, + 1, + 87 + ], + [ + 92, + 10.0, + -1, + 95 + ], + [ + 100, + -2.0, + 1, + 103 + ] + ], + "input_csv_sha256": "8b146558fa0f23f8926404bfeb87e433f02dd253c5a8eed2cf7bec21fa7d62ab", + "output_csv_sha256": "46915ca3dcb7da1b3f9e72a8ece1a8beb032a4f99657798022236150d9c336af" + }, + { + "window": 3, + "values": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 11, + 14, + 16, + 19, + 19, + 17, + 14, + 15, + 17, + 19, + 18, + 16, + 18, + 19, + 17, + 16, + 19, + 19, + 18, + 15, + 16, + 16, + 15, + 14, + 17, + 16, + 17, + 14, + 14, + 17, + 20, + 18, + 21, + 20, + 22, + 21, + 20, + 18, + 21, + 20, + 23, + 22, + 19, + 19, + 18, + 19, + 18, + 18, + 21, + 24, + 23, + 21, + 23, + 26, + 26, + 28, + 25, + 28, + 31, + 30, + 27, + 24, + 24, + 26, + 26, + 28, + 31, + 33, + 35, + 33, + 34, + 32, + 32, + 31, + 28, + 29, + 30, + 33, + 33, + 31, + 34, + 33, + 30, + 29, + 28, + 29, + 31, + 33, + 33, + 33, + 30, + 29, + 28, + 28, + 30, + 33, + 34, + 35, + 35, + 34 + ], + "events": [ + [ + 12, + 19.0, + -1, + 17 + ], + [ + 15, + 14.0, + 1, + 20 + ], + [ + 22, + 19.0, + -1, + 26 + ], + [ + 32, + 14.0, + 1, + 35 + ], + [ + 33, + 17.0, + -1, + 38 + ], + [ + 36, + 14.0, + 1, + 41 + ], + [ + 43, + 22.0, + -1, + 47 + ], + [ + 46, + 18.0, + 1, + 50 + ], + [ + 49, + 23.0, + -1, + 53 + ], + [ + 55, + 18.0, + 1, + 59 + ], + [ + 67, + 31.0, + -1, + 71 + ], + [ + 70, + 24.0, + 1, + 74 + ], + [ + 77, + 35.0, + -1, + 80 + ], + [ + 83, + 28.0, + 1, + 86 + ], + [ + 89, + 34.0, + -1, + 92 + ], + [ + 93, + 28.0, + 1, + 98 + ], + [ + 96, + 33.0, + -1, + 101 + ], + [ + 101, + 28.0, + 1, + 104 + ] + ], + "input_csv_sha256": "38beb51c87acae40c3226599d6819cd6b9f984b39428ba84d1995257ce2bbc1c", + "output_csv_sha256": "cb847cc137aee4433d9d6b70a1a1c74d02a70325c2ffe771130d6626449a40d4" + }, + { + "window": 4, + "values": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 11, + 14, + 13, + 16, + 17, + 19, + 17, + 18, + 17, + 16, + 13, + 12, + 15, + 16, + 19, + 21, + 23, + 25, + 23, + 25, + 24, + 24, + 22, + 21, + 21, + 19, + 16, + 17, + 16, + 13, + 12, + 15, + 15, + 13, + 15, + 12, + 15, + 17, + 18, + 21, + 20, + 18, + 16, + 16, + 16, + 16, + 17, + 19, + 18, + 16, + 19, + 18, + 21, + 23, + 21, + 24, + 24, + 26, + 29, + 31, + 32, + 29, + 26, + 26, + 26, + 29, + 31, + 32, + 29, + 29, + 28, + 25, + 22, + 25, + 22, + 20, + 22, + 25, + 27, + 24, + 22, + 25, + 22, + 24, + 22, + 19, + 20, + 21, + 19, + 19, + 22, + 24, + 25, + 24, + 25, + 26, + 23, + 23, + 24, + 26 + ], + "events": [ + [ + 17, + 19.0, + -1, + 23 + ], + [ + 23, + 12.0, + 1, + 27 + ], + [ + 29, + 25.0, + -1, + 35 + ], + [ + 47, + 12.0, + 1, + 51 + ], + [ + 51, + 21.0, + -1, + 55 + ], + [ + 61, + 16.0, + 1, + 67 + ], + [ + 72, + 32.0, + -1, + 79 + ], + [ + 74, + 26.0, + 1, + 79 + ], + [ + 79, + 32.0, + -1, + 83 + ], + [ + 87, + 20.0, + 1, + 91 + ], + [ + 90, + 27.0, + -1, + 95 + ], + [ + 97, + 19.0, + 1, + 103 + ], + [ + 107, + 26.0, + -1, + 111 + ] + ], + "input_csv_sha256": "303cbcb636decdd1a7c6b3841ea696fec0881a95023eeb7af71132a501b53e83", + "output_csv_sha256": "df8be88430783a0669fb9224952fe7a26f4fcf18e3f242e425b7279e75f69850" + }, + { + "window": 7, + "values": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 24, + 27, + 29, + 26, + 23, + 25, + 25, + 24, + 26, + 23, + 22, + 19, + 21, + 21, + 18, + 21, + 21, + 18, + 15, + 14, + 14, + 11, + 10, + 7, + 6, + 5, + 2, + 1, + 1, + -2, + -2, + -2, + -5, + -4, + -3, + -3, + -1, + -4, + -1, + 0, + 3, + 3, + 2, + 4, + 3, + 1, + -1, + -1, + 1, + 2, + 2, + 3, + 2, + 4, + 3, + 1, + 4, + 3, + 2, + 0, + -1, + -3, + -6, + -3, + -2, + -5, + -2, + -1, + 1, + -2, + -1, + -1, + -2, + -2, + -5, + -5, + -6, + -4, + -2, + 0, + -2, + -1, + -2, + -3, + -1, + 1, + 3, + 5, + 3, + 2, + -1, + -4, + -7, + -5, + -4, + -5, + -7, + -7, + -7 + ], + "events": [ + [ + 24, + 29.0, + -1, + 34 + ], + [ + 54, + -5.0, + 1, + 62 + ], + [ + 75, + 4.0, + -1, + 83 + ], + [ + 84, + -6.0, + 1, + 97 + ], + [ + 90, + 1.0, + -1, + 97 + ], + [ + 98, + -6.0, + 1, + 104 + ], + [ + 109, + 5.0, + -1, + 118 + ] + ], + "input_csv_sha256": "01a6232710be1d4dd606fb077855c424e1b43b17ec602c59b28b6db4f87249db", + "output_csv_sha256": "37fbaaf015d2020eff07e62c2aaeb2de0e4145dea85403c29a98dbaed94cf3d9" + } + ] +} diff --git a/tests/test_jev.py b/tests/test_jev.py new file mode 100644 index 00000000..1aef86e0 --- /dev/null +++ b/tests/test_jev.py @@ -0,0 +1,1043 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Offline TypeSafe protocol and controlled Trio scheduling regressions. + +''' +from dataclasses import replace +import json + +import httpx +import numpy as np +import pytest +import trio +from trio.testing import MockClock, wait_all_tasks_blocked + +from piker.fsp._jev import ( + API_URL, + DEFAULT_MODEL, + JevClassification, + JevClient, + JevProtocolError, + JevServiceError, + RegimeClassifier, + build_regime_request, +) +from piker.fsp._regime import ( + MultiTimeframeTrend, + RegimeSnapshot, + TimeframeRegime, +) + + +def _snapshots(count: int = 12) -> list[RegimeSnapshot]: + ''' + Produce complete synthetic evidence without a broker or network. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.test', periods=(1, 2), windows=(2,), + ) + snapshots: list[RegimeSnapshot] = [] + second: int + for second in range(count): + snapshots.append(tracker.push(second, 100 + second)) + return snapshots + + +def _answer_payload(request: httpx.Request) -> dict: + ''' + Simulate the documented Choice response for every sent question. + + ''' + payload: dict = json.loads(request.content) + answers: dict[str, dict] = {} + name: str + question: dict + for name, question in payload['questions'].items(): + labels: list[str] = list(question['criteria']) + probabilities: dict[str, float] = {} + label: str + for label in labels: + probabilities[label] = ( + .8 if label == labels[0] else .2 / (len(labels) - 1) + ) + answers[name] = { + 'type': 'choice', + 'choice': labels[0], + 'confidence': .8, + 'probabilities': probabilities, + } + return { + 'model': DEFAULT_MODEL, + 'answers': answers, + 'usage': { + 'input_tokens': 250, + 'output_tokens': 32, + }, + } + + +def test_request_batches_only_ready_timeframes() -> None: + ''' + Question IDs are not model input, so a timeframe-only identifier + leaves the judgment ambiguous. Verify explicit state paths in + each ready-frame question, one batched alignment question, and + no request before warm-up. The provider never supplies TC state. + + ''' + snapshots: list[RegimeSnapshot] = _snapshots() + with pytest.raises(ValueError, match='warm-up'): + build_regime_request(snapshots[0]) + early: dict = build_regime_request(snapshots[3]) + assert set(early['questions']) == {'timeframe_0'} + ready: dict = build_regime_request(snapshots[-1]) + assert set(ready['questions']) == { + 'timeframe_0', 'timeframe_1', 'alignment', + } + assert ready['model'] == DEFAULT_MODEL + assert 'timeframes[1]' in ( + ready['questions']['timeframe_1']['instructions'] + ) + assert 'unknown' in ready['questions']['timeframe_0']['criteria'] + assert ready['state']['algorithm'] == 'trend_calculus' + + +def test_graphics_snapshot_projects_to_bounded_model_evidence( +) -> None: + ''' + Graphics retains 256 observations for visible pivots, exceeding + the default HTTP request budget if sent directly to Jev. Build + an actual 4500-bar, four-order tracker with that rendering tail; + model preparation must keep only its configured evidence tails + while preserving source provenance, readiness and total counts. + The original rendering snapshot must remain unchanged, and the + projected request must pass through the ordinary HTTP adapter. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 5, 15), windows=(2,) * 4, + history_limit=256, + ) + second: int + for second in range(4500): + snapshot: RegimeSnapshot = tracker.push( + second, 100 + float(np.sin(second / 3)), + ) + full: dict = build_regime_request(snapshot, evidence_limit=256) + projected: dict = build_regime_request(snapshot) + assert len(json.dumps(full).encode()) > 65_536 + assert len(json.dumps(projected).encode()) < 65_536 + assert len(snapshot.timeframes[0].recent_closes) == 256 + assert projected['state']['revision'] == snapshot.revision + assert projected['state']['sequence'] == snapshot.sequence + frame: dict + source: TimeframeRegime + for frame, source in zip( + projected['state']['timeframes'], snapshot.timeframes, + ): + assert len(frame['recent_closes']) <= 16 + assert len(frame['reversals']) <= 16 + assert frame['close_count'] == source.close_count + assert frame['ready'] == source.ready + calls: int = 0 + + def handler(request: httpx.Request) -> httpx.Response: + ''' + Prove projection precedes the client's byte-budget gate. + + ''' + nonlocal calls + calls += 1 + assert len(request.content) <= 65_536 + return httpx.Response(200, json=_answer_payload(request)) + + async def scenario() -> None: + ''' + Exercise normal client admission with the larger UI snapshot. + + ''' + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + client: JevClient = JevClient(http, 'offline-test-key') + result: JevClassification = await client.classify( + snapshot, + ) + assert result.sequence == snapshot.sequence + assert len(result.answers) == 4 + + trio.run(scenario) + assert calls == 1 + + +def test_http_contract_retains_probabilities_and_source_times( +) -> None: + ''' + Collapsing a provider result to a label loses uncertainty and the + source revision needed to reject stale UI updates. MockTransport + inspects the actual HTTP body and bearer header, then responds + using the documented schema. Every probability, source identity, + model version and local availability time must survive parsing. + + ''' + snapshot: RegimeSnapshot = _snapshots()[-1] + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + ''' + Capture the request without permitting network I/O. + + ''' + requests.append(request) + return httpx.Response(200, json=_answer_payload(request)) + + async def scenario() -> None: + ''' + Exercise the same Trio-compatible HTTP path as live calls. + + ''' + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + client: JevClient = JevClient(http, 'offline-test-key') + result: JevClassification = await client.classify( + snapshot, + ) + assert result.market == snapshot.market + assert result.revision == snapshot.revision + assert result.source_as_of == snapshot.as_of + assert result.sequence == snapshot.sequence + assert result.requested_at <= result.received_at + assert result.latency >= 0 + assert result.model == DEFAULT_MODEL + assert result.input_tokens == 250 + assert result.answers[0].period == 1 + assert result.answers[0].probability == .8 + assert result.answers[0].confidence == .8 + assert result.answers[-1].period is None + distribution: dict = dict(result.answers[0].probabilities) + assert sum(distribution.values()) == pytest.approx(1) + + trio.run(scenario) + assert len(requests) == 1 + assert str(requests[0].url) == API_URL + assert requests[0].headers['authorization'] == ( + 'Bearer offline-test-key' + ) + body: dict = json.loads(requests[0].content) + assert body['state']['as_of'] == snapshot.as_of + + +@pytest.mark.parametrize('fault', [ + 'missing_question', + 'unknown_label', + 'missing_probability', + 'nonfinite_probability', + 'negative_probability', + 'wrong_sum', + 'not_maximum', + 'invalid_confidence', + 'wrong_model', + 'negative_usage', + 'boolean_usage', +]) +def test_malformed_response_cannot_publish(fault: str) -> None: + ''' + A typed API can still fail at the network/schema boundary. + Deliberately corrupt one otherwise valid response component and + require a protocol error. This prevents invalid probabilities, + incomplete classifications or unexpected model versions from + entering the same display path as validated semantic judgments. + + ''' + def handler(request: httpx.Request) -> httpx.Response: + ''' + Inject one isolated contract violation into a valid answer. + + ''' + body: dict = _answer_payload(request) + answer: dict = body['answers']['timeframe_0'] + probabilities: dict = answer['probabilities'] + if fault == 'missing_question': + del body['answers']['timeframe_0'] + elif fault == 'unknown_label': + answer['choice'] = 'buy_now' + elif fault == 'missing_probability': + del probabilities['unknown'] + elif fault == 'nonfinite_probability': + probabilities['uptrend'] = 'NaN' + elif fault == 'negative_probability': + probabilities['uptrend'] = -.8 + elif fault == 'wrong_sum': + probabilities['uptrend'] = .5 + elif fault == 'not_maximum': + answer['choice'] = 'downtrend' + elif fault == 'invalid_confidence': + answer['confidence'] = True + elif fault == 'wrong_model': + body['model'] = 'unexpected-model' + elif fault == 'negative_usage': + body['usage']['input_tokens'] = -1 + elif fault == 'boolean_usage': + body['usage']['input_tokens'] = True + return httpx.Response(200, json=body) + + async def scenario() -> None: + ''' + Verify protocol rejection at the HTTP adapter boundary. + + ''' + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + client: JevClient = JevClient(http, 'offline-test-key') + with pytest.raises(JevProtocolError): + await client.classify(_snapshots()[-1]) + + trio.run(scenario) + + +def test_request_and_response_budgets_prevent_unbounded_payloads( +) -> None: + ''' + Bounded history alone does not bound JSON bytes when parameters + or provider output changes. A tiny request limit must fail before + transport invocation; a tiny response limit must reject received + data instead of decoding or publishing an oversized answer. + + ''' + calls: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + ''' + Count transport calls and return an ordinary valid answer. + + ''' + calls.append(request) + return httpx.Response(200, json=_answer_payload(request)) + + async def scenario() -> None: + ''' + Verify the two independent budgets in one owned HTTP session. + + ''' + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + client: JevClient = JevClient( + http, 'offline-test-key', max_request_bytes=10, + ) + with pytest.raises(ValueError, match='byte budget'): + await client.classify(_snapshots()[-1]) + assert not calls + client = JevClient( + http, 'offline-test-key', max_response_bytes=10, + ) + with pytest.raises( + JevProtocolError, match='byte budget', + ): + await client.classify(_snapshots()[-1]) + assert len(calls) == 1 + + trio.run(scenario) + + +def test_worker_coalesces_bursts_and_discards_old_revision() -> None: + ''' + One request per quote can create an unbounded queue, while a + response from before a backfill can repaint corrected history. + The ``handler()`` task sets ``entered`` before waiting on + ``release``. While that request is suspended, ``scenario()`` + replaces ``RegimeClassifier._pending`` with a burst and revision. + Releasing the first request must discard its old evidence and + publish only the replacement. Request count and ``peak`` prove + coalescing and single-flight transport without scheduler timing. + + ''' + requests: list[dict] = [] + published: list[JevClassification] = [] + active: int = 0 + peak: int = 0 + + async def scenario() -> None: + ''' + Control response order with events instead of scheduler luck. + + ''' + entered: trio.Event = trio.Event() + release: trio.Event = trio.Event() + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Keep the first network operation in flight during submit. + + ''' + nonlocal active, peak + active += 1 + peak = max(peak, active) + requests.append(json.loads(request.content)) + try: + if len(requests) == 1: + entered.set() + await release.wait() + return httpx.Response( + 200, json=_answer_payload(request), + ) + finally: + active -= 1 + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=0, + max_source_lag=100, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + async def publish(result: JevClassification) -> None: + ''' + Stop after the fresh result proves publication. + + ''' + published.append(result) + nursery.cancel_scope.cancel() + + nursery.start_soon(worker.run, publish) + await entered.wait() + snapshot: RegimeSnapshot + for snapshot in snapshots[8:]: + worker.submit(snapshot) + newest: RegimeSnapshot = replace( + snapshots[-1], revision=1, sequence=1, + ) + worker.submit(newest) + release.set() + assert worker.discarded == 1 + assert worker.requests == 2 + assert worker.published == 1 + + trio.run(scenario) + assert len(requests) == 2 + assert peak == 1 + assert len(published) == 1 + assert published[0].revision == 1 + assert requests[1]['state']['revision'] == 1 + assert requests[1]['state']['sequence'] == 1 + + +@pytest.mark.parametrize('invalidation', ['lag', 'segment', 'age']) +def test_worker_rejects_stale_results(invalidation: str) -> None: + ''' + Matching ``RegimeSnapshot.revision`` alone does not prove + freshness: a gap, advancing event clock, or slow response can + invalidate evidence. ``handler()`` sets ``entered`` and waits + on ``release`` while ``scenario()`` changes exactly one of those + conditions. Only the second response may publish. ``MockClock`` + controls elapsed age; the two events prove the first response + cannot finish before its evidence becomes obsolete. + + ''' + published: list[JevClassification] = [] + + async def scenario() -> None: + ''' + Delay the first response while replacing source evidence. + + ''' + entered: trio.Event = trio.Event() + release: trio.Event = trio.Event() + calls: int = 0 + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Suspend only the first request at a cancellable event. + + ''' + nonlocal calls + calls += 1 + if calls == 1: + entered.set() + await release.wait() + return httpx.Response(200, json=_answer_payload(request)) + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key', timeout=20), + min_interval=0, + max_age=5, + max_source_lag=1, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + async def publish(result: JevClassification) -> None: + ''' + Exit only after an admissible response arrives. + + ''' + published.append(result) + nursery.cancel_scope.cancel() + + nursery.start_soon(worker.run, publish) + await entered.wait() + newest: RegimeSnapshot = snapshots[8] + if invalidation == 'lag': + newest = snapshots[-1] + elif invalidation == 'segment': + newest = replace(newest, segment=1) + else: + await trio.sleep(6) + worker.submit(newest) + release.set() + assert worker.discarded == 1 + assert published[0].sequence == newest.sequence + assert worker.published == 1 + + trio.run(scenario, clock=MockClock(autojump_threshold=0)) + + +def test_worker_throttles_and_honors_retry_after() -> None: + ''' + Immediate retries amplify rate limits and may resend old market + evidence. Return HTTP 429 with a three-second Retry-After, submit + newer evidence while the worker waits, and use MockClock to + prove the retry respects that delay and contains newest state. + + ''' + sent: list[tuple[float, int]] = [] + errors: list[Exception] = [] + + async def scenario() -> None: + ''' + Advance simulated time only after tasks reach their waits. + + ''' + failed: trio.Event = trio.Event() + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Rate-limit the initial request and accept its successor. + + ''' + body: dict = json.loads(request.content) + sent.append(( + trio.current_time(), body['state']['sequence'], + )) + if len(sent) == 1: + return httpx.Response( + 429, headers={'retry-after': '3'}, + ) + return httpx.Response(200, json=_answer_payload(request)) + + async def on_error(error: Exception) -> None: + ''' + Signal the exact rate-limit failure before new evidence. + + ''' + errors.append(error) + failed.set() + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=2, + max_age=10, + max_source_lag=10, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + async def publish(result: JevClassification) -> None: + ''' + End the test after successful recovery. + + ''' + nursery.cancel_scope.cancel() + + async def run() -> None: + ''' + Bind the worker's optional failure callback. + + ''' + await worker.run(publish, on_error=on_error) + + nursery.start_soon(run) + await failed.wait() + worker.submit(snapshots[-1]) + assert worker.requests == 2 + assert worker.failures == 1 + assert worker.last_error is None + + trio.run(scenario, clock=MockClock(autojump_threshold=0)) + assert sent[1][0] - sent[0][0] >= 3 + assert sent[1][1] == 12 + assert isinstance(errors[0], JevServiceError) + + +def test_cancellation_closes_inflight_work() -> None: + ''' + An HTTP operation detached from its nursery can outlive a chart + and use credentials after shutdown. ``handler()`` signals + ``entered`` before ``trio.sleep_forever()``; ``scenario()`` waits + for that signal before cancelling the worker nursery. The + ``closed`` event proves the transport's finalizer runs before + nursery exit, and ``publish()`` rejects any late output. This + proves cooperative cancellation, not an uninterruptible syscall. + + ''' + async def scenario() -> None: + ''' + Cancel only once the mock request is known to be in flight. + + ''' + entered: trio.Event = trio.Event() + closed: trio.Event = trio.Event() + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Model a cancellable stalled network operation. + + ''' + entered.set() + try: + await trio.sleep_forever() + finally: + closed.set() + + async def publish(result: JevClassification) -> None: + ''' + Reject any publication after cancellation begins. + + ''' + pytest.fail('cancelled request published a result') + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), min_interval=0, + ) + worker.submit(_snapshots()[-1]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(worker.run, publish) + await entered.wait() + nursery.cancel_scope.cancel() + assert closed.is_set() + assert worker.published == 0 + + trio.run(scenario) + + +def test_submit_rejects_wrong_market_and_nonmonotonic_source( +) -> None: + ''' + Sharing a worker across markets or submitting old source evidence + can overwrite its sole pending slot. Feed valid then invalid + identities without starting the worker and prove invalid updates + are refused before any HTTP call can be made. + + ''' + async def scenario() -> None: + ''' + Exercise submission validation on an otherwise idle worker. + + ''' + http: httpx.AsyncClient + async with httpx.AsyncClient() as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + ) + snapshot: RegimeSnapshot = _snapshots()[-1] + assert worker.submit(snapshot) + assert not worker.submit(snapshot) + with pytest.raises(ValueError, match='market'): + worker.submit(replace(snapshot, market='other.test')) + with pytest.raises(ValueError, match='as_of'): + worker.submit(replace(snapshot, sequence=13)) + assert worker.requests == 0 + + trio.run(scenario) + + +def test_worker_publishes_bounded_lag_with_original_provenance( +) -> None: + ''' + Rejecting every response after a faster input update starves live + inference. Permit one second of lag, hold the first response, + then submit its successor. Both may publish, but the older result + must retain its own source sequence and as_of rather than being + relabeled with the newer evidence's coordinates. + + ''' + published: list[JevClassification] = [] + + async def scenario() -> None: + ''' + Force a new source update before the first response returns. + + ''' + entered: trio.Event = trio.Event() + release: trio.Event = trio.Event() + calls: int = 0 + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Hold one response while leaving later requests immediate. + + ''' + nonlocal calls + calls += 1 + if calls == 1: + entered.set() + await release.wait() + return httpx.Response(200, json=_answer_payload(request)) + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=0, + max_source_lag=1, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + async def publish(result: JevClassification) -> None: + ''' + Stop after observing both ordered outputs. + + ''' + published.append(result) + if len(published) == 2: + nursery.cancel_scope.cancel() + + nursery.start_soon(worker.run, publish) + await entered.wait() + worker.submit(snapshots[8]) + release.set() + assert worker.discarded == 0 + assert worker.published == 2 + + trio.run(scenario) + result: JevClassification # noqa: F842 + assert [result.sequence for result in published] == [8, 9] + assert [result.source_as_of for result in published] == [8, 9] + + +def test_worker_transient_retry_count_is_bounded() -> None: + ''' + An unavailable service must not generate an endless retry loop + for one snapshot. Return 503 on every call, allow two retries, + and use MockClock to reach the third failure. Once all tasks are + blocked, the worker must be idle until new evidence arrives; + cancelling it must not reveal another scheduled request. + + ''' + calls: list[float] = [] + + async def scenario() -> None: + ''' + Observe exhausted retries before cancelling the idle worker. + + ''' + exhausted: trio.Event = trio.Event() + errors: int = 0 + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Keep the service unavailable for the whole scenario. + + ''' + calls.append(trio.current_time()) + return httpx.Response(503) + + async def on_error(error: Exception) -> None: + ''' + Signal when the final allowed attempt has failed. + + ''' + nonlocal errors + errors += 1 + if errors == 3: + exhausted.set() + + async def publish(result: JevClassification) -> None: + ''' + Reject fabricated results during provider unavailability. + + ''' + pytest.fail('a failed request must not publish') + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=0, + max_age=100, + max_retries=2, + ) + worker.submit(_snapshots()[-1]) + + async def run() -> None: + ''' + Attach the failure observer to the worker lifetime. + + ''' + await worker.run(publish, on_error=on_error) + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(run) + await exhausted.wait() + await wait_all_tasks_blocked() + assert worker.requests == 3 + assert worker.failures == 3 + nursery.cancel_scope.cancel() + + trio.run(scenario, clock=MockClock(autojump_threshold=0)) + assert calls == [0, 1, 3] + + +def test_worker_drops_expired_pending_work_before_network() -> None: + ''' + Throttling can make a queued snapshot expire even if no request + is in flight. Publish one result, enqueue its successor, then + advance MockClock beyond its five-second age limit but to the + next ten-second request slot. The expired successor must be + dropped before transport invocation, not merely after a reply. + + ''' + clock: MockClock = MockClock() + calls: int = 0 + + async def scenario() -> None: + ''' + Move time only after the worker reaches its throttle wait. + + ''' + first: trio.Event = trio.Event() + + async def handler(request: httpx.Request) -> httpx.Response: + ''' + Count requests independently of worker statistics. + + ''' + nonlocal calls + calls += 1 + return httpx.Response(200, json=_answer_payload(request)) + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=10, + max_age=5, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + + async def publish(result: JevClassification) -> None: + ''' + Submit fresh evidence at the initial publication. + + ''' + assert worker.published == 0 + worker.submit(snapshots[8]) + first.set() + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(worker.run, publish) + await first.wait() + await wait_all_tasks_blocked() + clock.jump(10) + await wait_all_tasks_blocked() + assert worker.requests == 1 + assert worker.discarded == 1 + nursery.cancel_scope.cancel() + + trio.run(scenario, clock=clock) + assert calls == 1 + + +def test_authentication_failure_propagates_without_retry() -> None: + ''' + Repeatedly retrying an invalid credential consumes requests and + hides configuration failures. A 401 response must escape the + worker immediately with a sanitized status-only error; neither + the key nor the server body should appear in its exception text. + + ''' + async def scenario() -> None: + ''' + Return an authentication failure through the real HTTP path. + + ''' + def handler(request: httpx.Request) -> httpx.Response: + ''' + Supply a body that should never be retained in the error. + + ''' + return httpx.Response(401, text='private server details') + + async def publish(result: JevClassification) -> None: + ''' + Reject any result produced without authentication. + + ''' + pytest.fail('unauthenticated request published') + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + ) + worker.submit(_snapshots()[-1]) + caught: pytest.ExceptionInfo[JevServiceError] + with pytest.raises(JevServiceError) as caught: + await worker.run(publish) + assert caught.value.status == 401 + assert 'offline-test-key' not in str(caught.value) + assert 'private server details' not in str(caught.value) + assert worker.requests == 1 + + trio.run(scenario) + + +@pytest.mark.parametrize( + 'invalidation', ['revision', 'segment', 'age'], +) +def test_blocked_publication_is_cancelled_when_evidence_changes( + invalidation: str, +) -> None: + ''' + Checking freshness before awaiting an output consumer is not + enough: a blocked send can deliver obsolete results after source + revision, gap or expiry. ``publish()`` sets ``entered`` before + awaiting ``released``. The ``scenario()`` task then changes the + revision/segment or advances ``MockClock`` past expiry and waits + for ``cancelled`` before releasing the consumer. An empty sink + proves the obsolete delivery was revoked; final provenance and + counters prove only the replacement reaches ``published``. The + event wait models cooperative backpressure, not a stuck syscall. + + ''' + published: list[JevClassification] = [] + + async def scenario() -> None: + ''' + Distinguish a blocked delivery from one already delivered. + + ''' + entered: trio.Event = trio.Event() + released: trio.Event = trio.Event() + cancelled: trio.Event = trio.Event() + attempts: int = 0 + + def handler(request: httpx.Request) -> httpx.Response: + ''' + Complete HTTP immediately; delay the output consumer. + + ''' + return httpx.Response(200, json=_answer_payload(request)) + + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + worker: RegimeClassifier = RegimeClassifier( + JevClient(http, 'offline-test-key'), + min_interval=0, + max_age=5, + ) + snapshots: list[RegimeSnapshot] = _snapshots() + worker.submit(snapshots[7]) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + async def publish(result: JevClassification) -> None: + ''' + Delay the consumer's first result acceptance. + + ''' + nonlocal attempts + attempts += 1 + if attempts == 1: + entered.set() + try: + await released.wait() + except trio.Cancelled: + cancelled.set() + raise + else: + await released.wait() + published.append(result) + nursery.cancel_scope.cancel() + + nursery.start_soon(worker.run, publish) + await entered.wait() + newest: RegimeSnapshot = snapshots[8] + if invalidation == 'revision': + worker.invalidate(1) + newest = replace(newest, revision=1, sequence=1) + assert not worker.submit(snapshots[-1]) + elif invalidation == 'segment': + newest = replace(newest, segment=1) + worker.submit(newest) + else: + await trio.sleep(6) + await cancelled.wait() + assert not published + released.set() + worker.submit(newest) + assert worker.discarded == 1 + assert worker.published == 1 + assert published[0].sequence == newest.sequence + assert published[0].revision == newest.revision + assert published[0].segment == newest.segment + + trio.run(scenario, clock=MockClock(autojump_threshold=0)) diff --git a/tests/test_regime_feed.py b/tests/test_regime_feed.py new file mode 100644 index 00000000..f7eb6de2 --- /dev/null +++ b/tests/test_regime_feed.py @@ -0,0 +1,771 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Native SHM-to-regime contracts, using synthetic source publications. + +''' +from collections.abc import AsyncIterator, Callable +from contextlib import asynccontextmanager +from types import SimpleNamespace +from typing import Any, Never +import json + +import httpx +import numpy as np +import pytest +import trio +from tractor.trionics import Lagged +from trio.testing import wait_all_tasks_blocked + +from piker.fsp._regime import ( + CloseSample, + MultiTimeframeTrend, + RegimeSnapshot, +) +from piker.fsp._regime_feed import ( + ClosedSourceRows, + SourceChanged, + consume_regime_source, + copy_closed_rows, + run_regime_feed, + serve_regimes, +) + + +class FakeSource: + ''' + Model absolute bounds independently of mutable backing storage. + + ''' + def __init__( + self, + first: int = 0, + last: int = 6, + ) -> None: + ''' + Seed indexed rows independently of their published bounds. + + ''' + self._first: SimpleNamespace = SimpleNamespace(value=first) + self._last: SimpleNamespace = SimpleNamespace(value=last) + self._array: np.ndarray = np.zeros(128, dtype=[ + ('index', ' None: + ''' + Treating the mutable source tail as finalized leaks provisional + closes into regime decisions. Make that tail NaN and verify the + detached prefix remains valid, immutable and independent of a + later shared-memory write. Absolute first/last bounds survive. + + ''' + source: FakeSource = FakeSource(first=2, last=8) + source._array['close'][7] = np.nan + copied: ClosedSourceRows = copy_closed_rows(source) + assert (copied.first, copied.last, copied.start) == (2, 8, 2) + assert len(copied.rows) == 5 + assert copied.rows['index'][-1] == 6 + assert not copied.rows.flags.writeable + original: float = copied.rows['close'][0] + source._array['close'][2] = 999 + assert copied.rows['close'][0] == original + + +@pytest.mark.parametrize('first,last', [(0, 8), (2, 6)]) +def test_changed_bounds_reject_mixed_copy( + first: int, + last: int, +) -> None: + ''' + A source prepend or truncation between the first bound read and + array access invalidates the copied range. Publish that change + from a controlled _array property access; the second bound check + must reject the candidate before any tracker can consume it. + + ''' + class ChangingSource: + ''' + Trigger a deterministic source prepend while copying begins. + + ''' + def __init__(self) -> None: + ''' + Retain a prefix whose origin or last bound will change. + + ''' + self._first: SimpleNamespace = SimpleNamespace(value=2) + self._last: SimpleNamespace = SimpleNamespace(value=8) + self.storage: np.ndarray = FakeSource()._array + + @property + def _array(self) -> np.ndarray: + ''' + Publish the changed origin after capturing old bounds. + + ''' + self._first.value = first + self._last.value = last + return self.storage + + with pytest.raises(SourceChanged, match='changed during copy'): + copy_closed_rows(ChangingSource()) + + +def test_append_during_copy_preserves_closed_prefix() -> None: + ''' + A high-frequency producer may append during every large history + copy. Its old closed prefix remains stable because the captured + mutable tail was excluded. Accept the detached prefix with its + original last bound, then consume the newly closed suffix once + without duplicating observations or starving bootstrap retries. + + ''' + class AppendingSource: + ''' + Publish two rows after the reader captures initial bounds. + + ''' + def __init__(self) -> None: + ''' + Retain closed rows before publishing an appended suffix. + + ''' + self._first: SimpleNamespace = SimpleNamespace(value=0) + self._last: SimpleNamespace = SimpleNamespace(value=6) + self.storage: np.ndarray = FakeSource()._array + + @property + def _array(self) -> np.ndarray: + ''' + Preserve the existing prefix while exposing a new tail. + + ''' + self._last.value = 8 + return self.storage + + source: AppendingSource = AppendingSource() + copied: ClosedSourceRows = copy_closed_rows(source) + assert (copied.first, copied.last, copied.start) == (0, 6, 0) + assert copied.rows['index'].tolist() == [0, 1, 2, 3, 4] + suffix: ClosedSourceRows = copy_closed_rows( + source, start=copied.last - 1, + ) + assert (suffix.first, suffix.last, suffix.start) == (0, 8, 5) + assert suffix.rows['index'].tolist() == [5, 6] + + +@pytest.mark.parametrize('first,last', [(-1, 6), (8, 6), (0, 129)]) +def test_stable_invalid_bounds_fail_instead_of_spinning( + first: int, + last: int, +) -> None: + ''' + Stable invalid bounds cannot become a usable source through a + tight retry loop. Treat negative, reversed and out-of-capacity + ranges as source errors, distinguishing them from bounds that + changed during the two-read copy validation window. + + ''' + with pytest.raises(ValueError, match='readable range'): + copy_closed_rows(FakeSource(first, last)) + + +def test_unsigned_coordinate_overflow_is_rejected_before_cast( +) -> None: + ''' + Converting uint64 coordinates to the core's int64 storage can + wrap an oversized value negative. Put that value last in a valid + increasing finalized prefix so ordering checks cannot mask the + overflow; source validation must reject it before replay begins. + + ''' + source: FakeSource = FakeSource(last=3) + source._array = source._array.astype([ + ('index', ' None: + ''' + Invalid finalized rows must not partially warm a tracker. Corrupt + a middle row and require validation of the whole detached range + before it can be handed to the recurrence; mutable-tail exclusion + is covered separately because unfinished rows may be invalid. + + ''' + source: FakeSource = FakeSource() + source._array[field][2] = value + with pytest.raises(ValueError, match='invalid finalized'): + copy_closed_rows(source) + + +def test_bootstrap_and_skipped_steps_emit_only_latest_snapshot( +) -> None: + ''' + A missed sampler wakeup must not skip finalized closes, and a + bootstrap must not create one HTTP job per historical bar. + Load five closed rows, mutate the open sixth row, then append + three rows with one wakeup. Exactly two snapshots must cover the + complete prefixes; duplicate and foreign messages produce none. + + ''' + published: list[RegimeSnapshot] = [] + + async def scenario() -> None: + ''' + Drive source publication and wakeups in a controlled order. + + ''' + source: FakeSource = FakeSource() + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 2), + ) + send: trio.MemorySendChannel + receive: trio.MemoryReceiveChannel + send, receive = trio.open_memory_channel(0) + first: trio.Event = trio.Event() + + def submit(snapshot: RegimeSnapshot) -> None: + ''' + Observe publication synchronously at the admission point. + + ''' + published.append(snapshot) + first.set() + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon( + consume_regime_source, + source, receive, tracker, submit, + ) + await first.wait() + source._array['close'][5] = 500 + await send.send({'period': 1}) + await send.send({'period': 60}) + await send.send({'backfilling': ['other.market', 1]}) + await send.send({'backfilling': ['test.market', 60]}) + await wait_all_tasks_blocked() + assert len(published) == 1 + source._last.value = 9 + await send.send({'period': 1}) + await send.aclose() + trio.run(scenario) + assert len(published) == 2 + assert published[0].sequence == 5 + assert published[1].sequence == 8 + assert published[1].as_of == 8 + assert published[1].revision == 0 + closes: tuple[CloseSample, ...] = ( + published[1].timeframes[0].recent_closes + ) + assert closes[5].price == 500 + + +@pytest.mark.parametrize( + 'change', ['in_place', 'prepend', 'truncate'], +) +def test_history_changes_revoke_and_replay(change: str) -> None: + ''' + Appending revised history to warm recurrence state is incorrect, + including repairs that leave both bounds unchanged. Publish a + source change and verify immediate revision invalidation precedes + its replacement snapshot. The rebuilt result must equal replay + through a fresh tracker over exactly the corrected closed prefix. + + ''' + published: list[RegimeSnapshot] = [] + order: list[tuple[str, int]] = [] + + async def scenario() -> None: + ''' + Compare each revised output with an independent full replay. + + ''' + source: FakeSource = FakeSource(first=2, last=20) + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 2), + ) + send: trio.MemorySendChannel + receive: trio.MemoryReceiveChannel + send, receive = trio.open_memory_channel(0) + first: trio.Event = trio.Event() + + def submit(snapshot: RegimeSnapshot) -> None: + ''' + Preserve publication order and source generation. + + ''' + published.append(snapshot) + order.append(('snapshot', snapshot.revision)) + first.set() + + def invalidate(revision: int) -> None: + ''' + Record when old pending judgments become inadmissible. + + ''' + order.append(('invalidate', revision)) + + async def consume() -> None: + ''' + Attach both source callbacks to one consumer lifetime. + + ''' + await consume_regime_source( + source, receive, tracker, submit, + invalidate=invalidate, + ) + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(consume) + await first.wait() + if change == 'in_place': + source._array['close'][4] = 200 + await send.send({ + 'backfilling': ['test.market', 1], + }) + else: + if change == 'prepend': + source._first.value = 0 + else: + source._last.value = 10 + await send.send({'period': 1}) + await send.aclose() + reference: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 2), revision=1, + ) + row: np.void + for row in copy_closed_rows(source).rows: + expected: RegimeSnapshot = reference.push( + float(row['time']), float(row['close']), + index=int(row['index']), + ) + assert published[-1] == expected + + trio.run(scenario) + assert order == [ + ('snapshot', 0), ('invalidate', 1), ('snapshot', 1), + ] + + +def test_revision_during_bootstrap_suppresses_old_publication( +) -> None: + ''' + Losing a backfill during bootstrap publishes a mixed-generation + classification that ordinary bound checks cannot detect. + ``SignallingTracker.push()`` sets ``entered`` after one row; + ``consume_regime_source()`` yields at every replay row so the + ``scenario()`` task can repair a close and send its notice. + ``_Notices.collect()`` must invalidate that revision before the + replay publishes. One replacement snapshot, its corrected close + and full sequence prove old evidence never reaches ``published``. + + ''' + published: list[RegimeSnapshot] = [] + invalidated: list[int] = [] + + async def scenario() -> None: + ''' + Synchronize repair with the first chunk's cooperative yield. + + ''' + entered: trio.Event = trio.Event() + + class SignallingTracker(MultiTimeframeTrend): + ''' + Expose the start of bootstrap without adding I/O to push. + + ''' + def push( + self, + *args: Any, + **kwargs: Any, + ) -> RegimeSnapshot: + ''' + Signal after the first historical row is consumed. + + ''' + snapshot: RegimeSnapshot = super().push( + *args, **kwargs, + ) + entered.set() + return snapshot + + source: FakeSource = FakeSource(last=100) + tracker: SignallingTracker = SignallingTracker( + 'test.market', periods=(1,), history_limit=128, + ) + send: trio.MemorySendChannel + receive: trio.MemoryReceiveChannel + send, receive = trio.open_memory_channel(0) + + async def consume() -> None: + ''' + Yield each row so the controlled repair reaches replay. + + ''' + await consume_regime_source( + source, receive, tracker, published.append, + invalidate=invalidated.append, + replay_chunk=1, + ) + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(consume) + await entered.wait() + source._array['close'][0] = 999 + await send.send({'backfilling': ['test.market', 1]}) + await send.aclose() + + trio.run(scenario) + assert invalidated == [1] + assert len(published) == 1 + assert published[0].revision == 1 + assert published[0].sequence == 99 + assert published[0].timeframes[0].recent_closes[0].price == 999 + + +@pytest.mark.parametrize('quote_lag', [False, True]) +@pytest.mark.parametrize('source_age', [None, 0, 60, -10]) +def test_native_wrapper_subscribes_filters_age_and_drains_quotes( + monkeypatch: pytest.MonkeyPatch, + quote_lag: bool, + source_age: float|None, +) -> None: + ''' + Subscribing after history capture loses in-place revisions in the + startup gap. ``GuardedSource._array`` asserts the subscription is + active before every read. ``sampler_context()`` then replaces + ``Flume.stream`` with ``OtherConsumer``; any later lookup of that + shared field fails instead of stealing another consumer's stream. + + The independent cases cover these admission contracts: + - ``LaggingQuotes`` raises ``Lagged`` once. It sets ``recovered`` + only when the quote drain resumes; SHM processing must survive. + - Old/future source clocks must withhold model evidence even + though submission occurs locally now; no age limit permits + offline replay. ``published`` distinguishes those outcomes. + - ``feed_context()`` rejects a requested quote throttle: feeds + cached solely by market must retain their shared quote policy. + + ``finish()`` waits on ``read_source`` and, when needed, + ``recovered``, then closes sampling after tasks block. Context + exit order proves the native wrapper releases its subscription + before the feed, without any broker, actor service or API key. + + ''' + order: list[str] = [] + subscribed: bool = False + published: list[RegimeSnapshot] = [] + read_source: trio.Event + flume_ref: SimpleNamespace|None = None + + class GuardedSource: + ''' + Require the native sampler subscription before source access. + + ''' + def __init__(self) -> None: + ''' + Seed a readable range whose array access is guarded. + + ''' + self._first: SimpleNamespace = SimpleNamespace(value=0) + self._last: SimpleNamespace = SimpleNamespace(value=6) + self.storage: np.ndarray = FakeSource()._array + + @property + def _array(self) -> np.ndarray: + ''' + Detect a source read before entering the subscription. + + ''' + assert subscribed + order.append('read_source') + read_source.set() + return self.storage + + async def scenario() -> None: + ''' + Close the synthetic sampler to end the actual wrapper. + + ''' + nonlocal read_source + read_source = trio.Event() + recovered: trio.Event = trio.Event() + send: trio.MemorySendChannel + receive: trio.MemoryReceiveChannel + send, receive = trio.open_memory_channel(0) + quote_send: trio.MemorySendChannel + quote_receive: trio.MemoryReceiveChannel + quote_send, quote_receive = trio.open_memory_channel(0) + + class LaggingQuotes: + ''' + Lose one discarded quote before resuming a healthy wait. + + ''' + def __aiter__(self) -> 'LaggingQuotes': + ''' + Return the resumable stream after a Lagged exception. + + ''' + return self + + async def __anext__(self) -> dict: + ''' + Force one overrun, then prove the drain resumed. + + ''' + if 'quote_lagged' not in order: + order.append('quote_lagged') + raise Lagged('synthetic discarded quote') + recovered.set() + await trio.sleep_forever() + + @asynccontextmanager + async def feed_context( + **kwargs: object, + ) -> AsyncIterator[SimpleNamespace]: + ''' + Expose a source Flume with a cancellable quote stream. + + ''' + nonlocal flume_ref + assert 'tick_throttle' not in kwargs + order.append('open_feed') + flume_ref = SimpleNamespace( + rt_sample_period_s=1, + rt_shm=GuardedSource(), + stream=( + LaggingQuotes() if quote_lag else quote_receive + ), + ) + yield SimpleNamespace(flumes={ + 'test.market': flume_ref, + }) + await quote_send.aclose() + order.append('close_feed') + + @asynccontextmanager + async def sampler_context( + **kwargs: object, + ) -> AsyncIterator[trio.MemoryReceiveChannel]: + ''' + Mark exactly when the sampler subscription begins. + + ''' + nonlocal subscribed + subscribed = True + order.append('subscribe') + + class OtherConsumer: + ''' + Model a shared Flume's receiver ownership change. + + ''' + def __aiter__(self) -> Never: + ''' + Detect consuming a later caller's broadcast view. + + ''' + pytest.fail('drain stole another quote receiver') + + assert flume_ref is not None + flume_ref.stream = OtherConsumer() + try: + yield receive + finally: + subscribed = False + order.append('unsubscribe') + + monkeypatch.setattr( + 'piker.data.feed.maybe_open_feed', feed_context, + ) + monkeypatch.setattr( + 'piker.data._sampling.open_sample_stream', + sampler_context, + ) + if source_age is not None: + monkeypatch.setattr( + 'piker.fsp._regime_feed.time.time', + lambda: 5 + source_age, + ) + + async def finish() -> None: + ''' + Close the sampler after processing and draining quotes. + + ''' + await read_source.wait() + if quote_lag: + await recovered.wait() + await wait_all_tasks_blocked() + await send.aclose() + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(finish) + await run_regime_feed( + 'test.market', published.append, + max_source_age=None if source_age is None else 5, + ) + + trio.run(scenario) + assert len(published) == (source_age in (None, 0)) + assert order.index('subscribe') < order.index('read_source') + assert order[-2:] == ['unsubscribe', 'close_feed'] + + +def test_service_composes_feed_and_classifier_in_one_lifetime( + monkeypatch: pytest.MonkeyPatch, +) -> None: + ''' + A prepared provider adapter needs concrete feed-to-classifier + wiring and shared cancellation ownership. Replace the broker + source with a synthetic callback producer but retain the actual + HTTP client, worker and service nursery. One corrected snapshot + must produce one validated judgment; normal source exit must + stop the classifier while leaving the caller's HTTP session open. + + ''' + from piker.fsp._jev import ( + DEFAULT_MODEL, + JevClassification, + JevClient, + RegimeClassifier, + ) + + published: list[JevClassification] = [] + + async def scenario() -> None: + ''' + End the source only after its judgment is published. + + ''' + delivered: trio.Event = trio.Event() + + async def source( + fqme: str, + submit: Callable[[RegimeSnapshot], object], + *, + invalidate: Callable[[int], object], + **kwargs: object, + ) -> None: + ''' + Replay one revision through the bound service callbacks. + + ''' + assert fqme == 'test.market' + assert kwargs['max_source_age'] == 5 + invalidate(1) + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + fqme, periods=(1,), revision=1, + ) + second: int + for second in range(8): + snapshot: RegimeSnapshot = tracker.push( + second, second, + ) + submit(snapshot) + await delivered.wait() + + def handler(request: httpx.Request) -> httpx.Response: + ''' + Return a documented Choice response without real HTTP. + + ''' + request_body: dict = json.loads(request.content) + answers: dict[str, dict] = {} + name: str + question: dict + for name, question in request_body['questions'].items(): + probabilities: dict[str, float] = {} + label: str + for label in question['criteria']: + probabilities[label] = float(label == 'uptrend') + answers[name] = { + 'type': 'choice', + 'choice': 'uptrend', + 'probabilities': probabilities, + 'confidence': 1, + } + return httpx.Response(200, json={ + 'model': DEFAULT_MODEL, + 'answers': answers, + 'usage': { + 'input_tokens': 100, + 'output_tokens': 20, + }, + }) + + async def publish(result: JevClassification) -> None: + ''' + Release source shutdown after receiving a typed judgment. + + ''' + published.append(result) + delivered.set() + + monkeypatch.setattr( + 'piker.fsp._regime_feed.run_regime_feed', source, + ) + http: httpx.AsyncClient + async with httpx.AsyncClient( + transport=httpx.MockTransport(handler), + ) as http: + client: JevClient = JevClient(http, 'offline-test-key') + worker: RegimeClassifier = await serve_regimes( + 'test.market', client, publish, + ) + assert not http.is_closed + assert worker.requests == 1 + assert worker.published == 1 + assert http.is_closed + + trio.run(scenario) + assert len(published) == 1 + assert published[0].market == 'test.market' + assert published[0].revision == 1 diff --git a/tests/test_trend.py b/tests/test_trend.py new file mode 100644 index 00000000..e4ff87f1 --- /dev/null +++ b/tests/test_trend.py @@ -0,0 +1,795 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Causality, streaming, and publication contracts for Trend Calculus. + +''' +from collections.abc import AsyncIterator, Callable +from dataclasses import Field, fields +from types import SimpleNamespace +from typing import Any, cast + +import numpy as np +import pytest +import trio +from trio.testing import wait_all_tasks_blocked +from trio_typing import TaskStatus + +from piker.data.ticktools import FeedQuote +from piker.fsp._api import ( + FspSample, + FspStream, + FspYield, + _load_builtins, +) +from piker.fsp._engine import ( + Cascade, + QuoteFrame, + _next_step_output, + _sample_wakeups, + _write_sample, + filter_quotes_by_sym, +) +from piker.fsp._trend import ( + TrendCalculator, + TrendHistory, + TrendUpdate, +) +from piker.fsp._trend_fsp import trend_calculus +from tractor.ipc._shm import ShmArray +from tractor.trionics import Lagged + + +def test_split_preserves_both_reversals() -> None: + ''' + A net unchanged direction can contain two opposing reversals. + + The final outside window [2, 5] follows the rising window [4, 3]. + Bridging its earlier extremum reveals a top at 4 and a bottom at + 2, both detected on observation 5. A one-pulse or net-sign-only + implementation loses the top. Assert both pivot coordinates and + the later detection coordinate, including the historical fields. + + ''' + values: np.ndarray = np.array([1., 2., 4., 3., 2., 5.]) + calc: TrendCalculator = TrendCalculator() + calc.extend(values[:-1], np.arange(5.)) + update: TrendUpdate = calc.push(5., values[-1]) + assert update.trends == (1, 0, 0, 0) + assert update.reversals['index'].tolist() == [2, 4] + assert update.reversals['direction'].tolist() == [-1, 1] + assert update.reversals['price'].tolist() == [4., 2.] + assert update.reversals['detected_index'].tolist() == [5, 5] + assert update.reversals['detected_time'].tolist() == [5., 5.] + history: TrendHistory = TrendCalculator().extend( + values, np.arange(6.), + ) + np.testing.assert_array_equal( + history.pivot_prices[-1, 0], [4., 2.], + ) + np.testing.assert_array_equal( + history.pivot_times[-1, 0], [2., 4.], + ) + assert np.isnan(history.pivot_prices[:-1]).all() + + +def test_batch_chunking_and_every_prefix() -> None: + ''' + Future observations and transport chunking must not alter output. + + Materialize a random walk with ties and a missing observation, + then compare every prefix and uneven continuation batches with a + single history scan. Compare pivot detection fields too, because + equal direction arrays alone can hide backdated reversals. + + ''' + rng: np.random.Generator = np.random.default_rng(18) + values: np.ndarray = ( + rng.integers(-2, 3, 80).cumsum().astype(float) + ) + values[31] = np.nan + times: np.ndarray = np.arange(80.) * 0.125 + 1000. + indices: np.ndarray = np.arange(80, dtype=np.int64) + 12345 + windows: tuple[int, ...] = (3, 2, 4) + full: TrendHistory = TrendCalculator(windows).extend( + values, times, indices, + ) + cut: int + field: Field[Any] + for cut in range(81): + prefix: TrendHistory = TrendCalculator(windows).extend( + values[:cut], times[:cut], indices[:cut], + ) + for field in fields(TrendHistory): + np.testing.assert_array_equal( + getattr(prefix, field.name), + getattr(full, field.name)[:cut], + ) + + calc: TrendCalculator = TrendCalculator(windows) + start: int = 0 + for cut in (0, 1, 2, 7, 31, 32, 34, 60, 80): + chunk: TrendHistory = calc.extend( + values[start:cut], times[start:cut], indices[start:cut], + ) + for field in fields(TrendHistory): + np.testing.assert_array_equal( + getattr(chunk, field.name), + getattr(full, field.name)[start:cut], + ) + start = cut + + +@pytest.mark.parametrize('window', [2, 3, 7]) +@pytest.mark.parametrize('direction', [-1, 0, 1]) +def test_warmup_flat_and_negative_values( + window: int, + direction: int, +) -> None: + ''' + Startup must not invent a trend from a synthetic zero-price bar. + + Use negative, monotone or flat values through two full windows. + The first comparable pair establishes readiness and direction; + no sign change or initial pivot exists before that comparison. + + ''' + values: np.ndarray = ( + np.arange(2 * window, dtype=float) * direction - 100 + ) + result: TrendHistory = TrendCalculator((window,)).extend( + values, np.arange(float(len(values))), + ) + assert not result.ready[:-1].any() + assert result.ready[-1, 0] + assert not result.trends[:-1].any() + assert result.trends[-1, 0] == direction + assert np.isnan(result.pivot_prices).all() + + +def test_nan_resets_hierarchy_without_future_contamination() -> None: + ''' + A missing value invalidates every recursive reversal order. + + Build nontrivial history, append NaN, and compare the following + observations against a newly constructed calculator. Retaining + even a higher-order partial window would make this replay differ. + + ''' + rng: np.random.Generator = np.random.default_rng(77) + calc: TrendCalculator = TrendCalculator((2, 2, 2)) + calc.extend(rng.normal(size=200), np.arange(200.)) + update: TrendUpdate = calc.push(200., np.nan) + assert update.trends == (0, 0, 0) + assert update.ready == (False, False, False) + assert not len(update.reversals) + values: np.ndarray = rng.normal(size=100) + times: np.ndarray = np.arange(201., 301.) + actual: TrendHistory = calc.extend(values, times) + expected: TrendHistory = TrendCalculator((2, 2, 2)).extend( + values, times, + ) + field: Field[Any] + for field in fields(TrendHistory): + np.testing.assert_array_equal( + getattr(actual, field.name), + getattr(expected, field.name), + ) + + +@pytest.mark.parametrize( + 'time,value,index', + [ + (3., 1., 14), + (2., 1., 14), + (4., np.inf, 14), + (np.nan, 1., 14), + (4., 1., 13), + (4., 1., -1), + (4., 1., 2 ** 63), + (4., 1., np.uint64(2 ** 63)), + (4., 1., True), + (4., 1., np.bool_(True)), + ], +) +def test_invalid_sample_is_atomic( + time: float, + value: float, + index: int, +) -> None: + ''' + Invalid coordinates must not corrupt a partially filled window. + + Seed a calculator with four observations, attempt one rejected + sample, and compare continuation to an untouched state copy. + This checks that validation precedes recurrence mutation. + + ''' + calc: TrendCalculator = TrendCalculator() + calc.extend(np.arange(4.), np.arange(4.), np.arange(10, 14)) + untouched: TrendCalculator = calc.copy() + with pytest.raises(ValueError): + calc.push(time, value, index=index) + actual: TrendUpdate = calc.push(4., -1., index=14) + expected: TrendUpdate = untouched.push(4., -1., index=14) + assert actual.trends == expected.trends + np.testing.assert_array_equal( + actual.reversals, expected.reversals, + ) + + +def test_invalid_batch_is_atomic() -> None: + ''' + A late malformed sample cannot partly advance a batch operation. + + The duplicate final timestamp is detected before the first valid + row is consumed. A subsequent valid replay matches fresh state. + + ''' + calc: TrendCalculator = TrendCalculator() + with pytest.raises(ValueError): + calc.extend(np.arange(4.), np.array([1., 2., 3., 3.])) + actual: TrendHistory = calc.extend(np.arange(4.), np.arange(4.)) + expected: TrendHistory = TrendCalculator().extend( + np.arange(4.), np.arange(4.), + ) + np.testing.assert_array_equal(actual.trends, expected.trends) + + +def test_last_int64_index_is_exact_and_overflow_is_atomic() -> None: + ''' + Explicit unsigned coordinates must not wrap in compiled state. + + Observe the last representable coordinate using a NumPy uint64. + A subsequent automatic push or batch must reject overflow before + touching the partial window, and retain the exact last index. + + ''' + calc: TrendCalculator = TrendCalculator() + limit: int = np.iinfo(np.int64).max + result: TrendUpdate = calc.push( + 1., 2., index=np.uint64(limit), + ) + assert result.index == limit + assert isinstance(result.index, int) + before: np.ndarray = calc._bars.copy() + with pytest.raises(ValueError, match='int64'): + calc.push(2., 3.) + with pytest.raises(ValueError, match='overflow'): + calc.extend(np.array([3.]), np.array([2.])) + np.testing.assert_array_equal(calc._bars, before) + + +class MemorySource: + ''' + Expose mutable source rows with absolute, nonzero SHM bounds. + + ''' + def __init__(self, values: list[float]) -> None: + ''' + Seed mutable rows with a nonzero absolute SHM origin. + + ''' + self._first: SimpleNamespace = SimpleNamespace(value=37) + self._last: SimpleNamespace = SimpleNamespace( + value=37 + len(values), + ) + self._array: np.ndarray = np.zeros(128, dtype=[ + ('index', ' np.ndarray: + ''' + Expose ``MemorySource._array`` between published bounds. + + ''' + return self._array[self._first.value:self._last.value] + + +def assert_sample_matches( + sample: FspSample, + rows: np.ndarray, +) -> None: + ''' + Compare every FSP field with a fresh source-prefix replay. + + ''' + history: TrendHistory = TrendCalculator().extend( + rows['close'], rows['time'], rows['index'], + ) + assert sample.index == rows['index'][-1] + assert sample.time == rows['time'][-1] + assert sample.fields['ready'] == history.ready[-1].sum() + level: int + for level in range(4): + order: int = level + 1 + name: str = 'trend' if level == 0 else f'trend_{order}' + assert sample.fields[name] == history.trends[-1, level] + side: int + label: str + for side, label in enumerate(('top', 'bottom')): + np.testing.assert_equal( + sample.fields[f'{label}_{order}'], + history.pivot_prices[-1, level, side], + ) + np.testing.assert_equal( + sample.fields[f'{label}_time_{order}'], + history.pivot_times[-1, level, side], + ) + + +def test_fsp_preview_mutations_and_skipped_steps() -> None: + ''' + Updating an open bar must replace its preview, not consume it. + + Repeatedly mutate the current SHM close across both directions, + then advance by two rows before the next quote. Each FspSample + must equal a fresh batch replay, and its absolute row must match + the source. Quote payloads are deliberately empty wakeups so the + source state rather than trade arrivals must drive sampling. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + shm: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.]) + + async def quotes() -> AsyncIterator[FeedQuote]: + ''' + Provide payload-free wakeups for source + snapshot sampling. + + ''' + while True: + yield {} + + stream: FspStream = trend_calculus( + quotes(), cast(ShmArray, shm), + ) + try: + historical: FspYield = await anext(stream) + assert isinstance(historical, dict) + assert set(historical) == set(trend_calculus.outputs) + value: np.ndarray|None # noqa: F842 + assert all( + value is not None and len(value) == 6 + for value in historical.values() + ) + close: float + for close in (100., -20., 5., 5.): + shm.array['close'][-1] = close + sample: FspYield = await anext(stream) + assert isinstance(sample, FspSample) + assert_sample_matches(sample, shm.array) + + last: int = shm._last.value + shm._array['close'][last:last + 2] = [-4., -5.] + shm._last.value += 2 + end: int + for end in (6, 7, 8): + sample: FspYield = await anext(stream) + assert isinstance(sample, FspSample) + assert_sample_matches(sample, shm.array[:end]) + finally: + await stream.aclose() + + trio.run(main) + + +def test_fsp_empty_bootstrap_and_source_revision() -> None: + ''' + Empty startup must accept later source rows without old state. + + After the first sample, prepend changes the source origin. The + old generator must stop; restarting it recomputes the revised + history rather than reusing recurrence state from another origin. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + shm: MemorySource = MemorySource([]) + + async def quotes() -> AsyncIterator[FeedQuote]: + ''' + Provide payload-free wakeups for source + snapshot sampling. + + ''' + while True: + yield {} + + stream: FspStream = trend_calculus( + quotes(), cast(ShmArray, shm), + ) + historical: FspYield = await anext(stream) + assert isinstance(historical, dict) + value: np.ndarray|None # noqa: F842 + assert all( + value is not None and not len(value) + for value in historical.values() + ) + shm._array['close'][37] = 1. + shm._last.value += 1 + sample: FspYield = await anext(stream) + assert isinstance(sample, FspSample) + assert_sample_matches(sample, shm.array) + shm._first.value -= 1 + with pytest.raises(StopAsyncIteration): + await anext(stream) + restarted: FspStream = trend_calculus( + quotes(), cast(ShmArray, shm), + ) + try: + await anext(restarted) + sample: FspYield = await anext(restarted) + assert isinstance(sample, FspSample) + assert_sample_matches(sample, shm.array) + finally: + await restarted.aclose() + + trio.run(main) + + +def test_indexed_sample_waits_for_destination_step() -> None: + ''' + A quote can reach the FSP before the sampler appends destination. + + Block _write_sample with destination one row behind source and + wait until its task is suspended. Publishing the next destination + bound and setting its event permits one coherent write. The prior + row remains unchanged, proving no accidental previous-bar write. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + array: np.ndarray = np.zeros(16, dtype=[ + ('time', ' None: + ''' + Wait for the destination bound, then signal publication. + + ''' + assert await _write_sample(casc, FspSample( + 10, 1010., {'trend': -1., 'ready': 2.}, + )) + complete.set() + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(publish) + await wait_all_tasks_blocked() + assert not complete.is_set() + assert not array['trend'].any() + dst._last.value += 1 + assert casc.sample_ready is not None + casc.sample_ready.set() + await complete.wait() + assert tuple(array[9]) == (0., 0., 0.) + assert tuple(array[10]) == (1010., -1., 2.) + + trio.run(main) + + +def test_indexed_sample_wait_is_cancellable() -> None: + ''' + History resync must cancel a compute task waiting on sampling. + + Suspend indexed publication against an empty destination, cancel + its nursery, then advance that destination. No cancelled sample + may publish into the new source revision after cancellation. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + array: np.ndarray = np.zeros(4, dtype=[ + ('time', float), ('trend', float), + ]) + dst: SimpleNamespace = SimpleNamespace( + _first=SimpleNamespace(value=0), + _last=SimpleNamespace(value=0), + _array=array, + ) + casc: Cascade = cast(Cascade, SimpleNamespace( + dst=SimpleNamespace(rt_shm=dst), + sample_ready=None, + )) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon( + _write_sample, casc, FspSample(0, 1., {'trend': 1.}), + ) + await wait_all_tasks_blocked() + nursery.cancel_scope.cancel() + dst._last.value = 1 + assert casc.sample_ready is not None + casc.sample_ready.set() + assert not array['trend'].any() + + trio.run(main) + + +def test_builtin_registration() -> None: + ''' + Actor-side builtin discovery must resolve the new FSP target. + + ''' + assert _load_builtins()[trend_calculus.ns_path] is trend_calculus + + +def test_quote_free_sampler_steps_finalize_without_repeating_pivots( +) -> None: + ''' + A quiet feed still finalizes bars and clears old reversal pulses. + + Start with an open bar containing two provisional reversals and + advance the source on sampler notifications alone. Mirror the + destination append protocol and run the actual FSP generator and + indexed writer. Every stored field must equal causal replay; + event fields must also be clear before compute catches up. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + source: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.]) + first: int = source._first.value + last: int = source._last.value + name: str + dtype: list[tuple[str, str]] = [ + ('index', ' None: + ''' + Run until cancellation and record completion on exit. + + ''' + try: + sample: FspYield + async for sample in stream: + assert isinstance(sample, FspSample) + await _write_sample(casc, sample) + finally: + await stream.aclose() + + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + nursery.start_soon(compute) + await wait_all_tasks_blocked() + assert array['top_1'][last - 1] == 4. + assert array['bottom_1'][last - 1] == 2. + close: float + for close in (5., 5., -4., -5.): + last = source._last.value + source._array['close'][last] = close + source._last.value += 1 + array[last:last + 1] = _next_step_output( + array[last - 1:last], trend_calculus, False, + ) + array['index'][last] = last + array['time'][last] = source.array['time'][-1] + destination._last.value += 1 + assert np.isnan(array['top_1'][last]) + assert np.isnan(array['bottom_1'][last]) + if casc.sample_ready is not None: + casc.sample_ready.set() + casc.sample_ready = trio.Event() + assert notify is not None + # Coalesce a burst without losing finalized rows. + notify() + notify() + await wait_all_tasks_blocked() + offset: int + row: np.void + for offset, row in enumerate(source.array): + name: str + sample: FspSample = FspSample( + int(row['index']), float(row['time']), + { + name: array[name][int(row['index'])] + for name in trend_calculus.outputs + }, + ) + assert_sample_matches( + sample, source.array[:offset + 1], + ) + nursery.cancel_scope.cancel() + + trio.run(main) + + +def test_snapshot_quote_overrun_resumes_source_wakeups() -> None: + ''' + Losing irrelevant quote payloads must not kill a SHM processor. + + A broadcast receiver raises Lagged and then remains usable. + Simulate that protocol, require a catchup notification, and + deliver another quote afterward to prove the relay resumed. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + send: trio.MemorySendChannel[QuoteFrame|Exception] + receive: trio.MemoryReceiveChannel[QuoteFrame|Exception] + send, receive = trio.open_memory_channel[ + QuoteFrame|Exception + ](1) + + class Quotes: + ''' + Inject one overrun without closing the quote receiver. + + ''' + def __aiter__(self) -> 'Quotes': + ''' + Retain this controlled quote receiver + across overruns. + + ''' + return self + + async def __anext__(self) -> QuoteFrame: + ''' + Deliver the scheduled frame or + injected receiver error. + + ''' + value: QuoteFrame|Exception = await receive.receive() + if isinstance(value, Exception): + raise value + return value + + wakeups: AsyncIterator[QuoteFrame] + notify: Callable[[], None]|None + async with send, receive, _sample_wakeups( + Quotes(), 'test.market', True, + ) as (wakeups, notify): + await send.send(Lagged('synthetic quote overrun')) + assert 'test.market' in await anext(wakeups) + await send.send({'test.market': {'ticks': []}}) + assert 'test.market' in await anext(wakeups) + assert notify is not None + + trio.run(main) + + +def test_edge_exit_stops_compute_before_source_receiver_closes( +) -> None: + ''' + Normal sampler EOF must not strand or abruptly close compute. + + Keep the parent nursery alive beyond open_edge and inspect the + compute task's finalizer immediately after context exit. This + proves cleanup belongs to the edge scope rather than relying on + an outer actor cancellation or a closed receiver exception. + + ''' + async def main() -> None: + ''' + Run the controlled scenario in an isolated Trio lifetime. + + ''' + cleaned: trio.Event = trio.Event() + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + casc: Cascade = cast(Cascade, SimpleNamespace( + tn=nursery, + cs=None, + complete=None, + bind_func=None, + )) + + async def compute( + *, + task_status: TaskStatus[int] = ( + trio.TASK_STATUS_IGNORED + ), + ) -> None: + ''' + Run until cancellation and record completion on exit. + + ''' + scope: trio.CancelScope + with trio.CancelScope() as scope: + casc.cs = scope + casc.complete = trio.Event() + task_status.started(42) + try: + await trio.sleep_forever() + finally: + cleaned.set() + casc.complete.set() + + index: int + async with Cascade.open_edge(casc, compute) as index: + assert index == 42 + assert not cleaned.is_set() + assert cleaned.is_set() + + trio.run(main) diff --git a/tests/test_trend_graphics.py b/tests/test_trend_graphics.py new file mode 100644 index 00000000..fb8b5040 --- /dev/null +++ b/tests/test_trend_graphics.py @@ -0,0 +1,929 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Trend overlay causality, coordinate and bounded Qt lifetime checks. + +''' +from collections.abc import AsyncIterator, Iterator +from contextlib import asynccontextmanager +import os +from types import SimpleNamespace +from typing import Literal + +os.environ['QT_QPA_PLATFORM'] = 'offscreen' + +import numpy as np +import pyqtgraph as pg +import pytest +import trio +from trio.testing import wait_all_tasks_blocked + +from piker.ui._trend import ( + RegimeInterval, + ReversalPoint, + TrendOverlay, + classification_interval, + regime_rectangles, + reversal_points, + times_to_x, + trend_intervals, +) +from piker.ui.qt import QApplication, QtGui, QPointF +from piker.fsp._jev import JevClassification, RegimeAnswer +from piker.fsp import Fsp +from piker.ui._fsp import FspAdmin, update_trend_overlays +from piker.ui import _fsp as fsp_ui + + +def _chart( + times: tuple[float, ...], + first: int = 1000, + +) -> np.ndarray: + ''' + Make chart bars with absolute indices independent of timestamp. + + ''' + rows: np.ndarray = np.zeros(len(times), dtype=[ + ('index', 'i8'), + ('time', 'f8'), + ('low', 'f8'), + ('high', 'f8'), + ('close', 'f8'), + ]) + rows['index'] = np.arange(first, first + len(times)) + rows['time'] = times + rows['low'] = 99 + rows['high'] = 101 + rows['close'] = 100 + return rows + + +def _source( + times: tuple[float, ...], + trends: tuple[float, ...], + +) -> np.ndarray: + ''' + Make sparse FSP fields, including a deliberately mutable tail. + + ''' + fields: list[tuple[str, str]] = [ + ('index', 'i8'), + ('time', 'f8'), + ('ready', 'f8'), + ] + order: int + for order in range(1, 5): + field: str = 'trend' if order == 1 else f'trend_{order}' + fields.append((field, 'f8')) + kind: str + for kind in ('top', 'bottom'): + fields.extend([ + (f'{kind}_{order}', 'f8'), + (f'{kind}_time_{order}', 'f8'), + ]) + + rows: np.ndarray = np.empty(len(times), dtype=fields) + name: str + for name in rows.dtype.names: + if name == 'index': + rows[name] = np.arange(5000, 5000 + len(rows)) + else: + rows[name] = np.nan + rows['time'] = times + rows['ready'] = 4 + rows['trend'] = trends + return rows + + +@pytest.fixture(scope='session') +def qapp() -> Iterator[QApplication]: + ''' + Keep one offscreen application alive without launching a GUI. + + ''' + app: QApplication|None = QApplication.instance() + if app is None: + app = QApplication(['piker-trend-tests']) + app.setQuitOnLastWindowClosed(False) + yield app + app.processEvents() + + +@pytest.mark.parametrize('index_field', ['index', 'time']) +def test_timestamp_mapping_preserves_gaps( + index_field: Literal['index', 'time'], + +) -> None: + ''' + A timestamp inside a missing bucket must not snap to a future + bar or confuse slice offsets with absolute SHM coordinates. + Irregular 10-second bars and off-grid queries exercise both + chart axes; NaNs prove out-of-range observations stay hidden. + + ''' + rows: np.ndarray = _chart((100, 110, 140)) + queries: np.ndarray = np.array((99, 100, 115, 125, 145, 150)) + actual: np.ndarray = times_to_x(rows, queries, 10, index_field) + expected: list[float] = ( + [np.nan, 1000, 1001.5, np.nan, 1002.5, np.nan] + if index_field == 'index' + else [np.nan, 100, 115, np.nan, 145, np.nan] + ) + np.testing.assert_allclose(actual, expected, equal_nan=True) + + +def test_pivots_confirm_at_later_bar_close() -> None: + ''' + Plotting a provisional reversal makes an intrabar repaint look + like a confirmed signal. A confirmed top and mutable-tail bottom + share an earlier pivot time here. Only the top may render, with + its 30-second source detection time and 10-second chart position + retained separately; ``source['index']`` is unrelated on purpose. + + ''' + source: np.ndarray = _source((100, 130, 160, 190), (0, 1, -1, 1)) + source['top_1'][2] = 110 + source['top_time_1'][2] = 130 + source['bottom_1'][3] = 90 + source['bottom_time_1'][3] = 160 + chart: np.ndarray = _chart(tuple(range(100, 201, 10))) + points: list[ReversalPoint] = reversal_points( + source, + chart, + source_period_s=30, + chart_period_s=10, + ) + assert len(points) == 1 + assert points[0].x == 1003 + assert points[0].pivot_time == 130 + assert points[0].detected_at == 190 + assert points[0].period_s == 30 + assert points[0].kind == 'top' + + +def test_reversal_order_is_independent_of_timeframe() -> None: + ''' + A second reversal order must use its own pivot fields without + implicitly changing the source sampling period. Distinct order + one and order two prices at the same source time identify which + columns were consumed and prevent order/timeframe conflation. + + ''' + source: np.ndarray = _source((10, 20, 30), (1, -1, 1)) + source['top_1'][1] = 105 + source['top_time_1'][1] = 10 + source['bottom_2'][1] = 95 + source['bottom_time_2'][1] = 10 + points: list[ReversalPoint] = reversal_points( + source, + _chart((10, 20, 30)), + source_period_s=10, + chart_period_s=10, + order=2, + ) + assert len(points) == 1 + assert points[0].price == 95 + assert points[0].order == 2 + assert points[0].period_s == 10 + + +def test_regime_runs_start_at_confirmation() -> None: + ''' + Coloring from a pivot or detection-bar open falsely implies an + earlier available regime. A warm-up row, direction change, and + source gap arrange all run boundaries. Every interval starts at + finalized bar close and the mutable tail's opposite direction + cannot create a new regime. + + ''' + source: np.ndarray = _source( + (100, 110, 120, 130, 170, 180), + (1, 1, 1, -1, -1, 1), + ) + source['ready'][0] = 0 + spans: list[RegimeInterval] = trend_intervals(source, 10) + span: RegimeInterval # noqa: F842 + assert [(span.start_time, span.end_time) for span in spans] == [ + (120, 140), + (140, 150), + (180, 190), + ] + assert [span.label for span in spans] == [ + 'uptrend', 'downtrend', 'downtrend', + ] + + +def test_decimal_subsecond_epochs_do_not_invent_gaps() -> None: + ''' + Float64 Unix timestamps cannot represent every 100 ms boundary + exactly; successive differences alternate around 0.1 seconds. + Comparing each difference strictly to the requested period splits + one regime into spurious holes. Realistic epochs prove rounding + preserves one run, while deleting one actual bucket still splits + both regime evidence and chart rectangle geometry. + + ''' + times: np.ndarray = 1_700_000_000 + np.arange(8) * 0.1 + source: np.ndarray = _source(tuple(times), (1,) * 8) + spans: list[RegimeInterval] = trend_intervals(source, 0.1) + assert len(spans) == 1 + rectangles: np.ndarray + _: list[RegimeInterval] + rectangles, _ = regime_rectangles( + spans, _chart(tuple(times)), 0.1, + ) + assert len(rectangles) == 1 + + source['time'][4:] += 0.1 + spans = trend_intervals(source, 0.1) + assert len(spans) == 2 + whole: RegimeInterval = RegimeInterval( + float(times[0]), float(times[-1] + 0.3), 'uptrend', + ) + rectangles, _ = regime_rectangles( + [whole], _chart(tuple(source['time'])), 0.1, + ) + assert len(rectangles) == 2 + + +@pytest.mark.parametrize('index_field', ['index', 'time']) +def test_delayed_regime_and_chart_gaps( + index_field: Literal['index', 'time'], + +) -> None: + ''' + A classifier may report an earlier onset after the chart has + advanced. Its rectangle must begin only at response arrival and + must not fill an absent chart bucket. Fractional arrival within + a bar, plus a 20-second hole, proves causal clipping and split + geometry on both time and absolute-index chart axes. + + ''' + chart: np.ndarray = _chart((100, 110, 140, 150)) + span: RegimeInterval = RegimeInterval( + start_time=100, + end_time=155, + label='ranging', + probability=0.8, + source='jev', + observed_at=115, + ) + rectangles: np.ndarray + metadata: list[RegimeInterval] + rectangles, metadata = regime_rectangles( + [span], chart, 10, index_field=index_field, + ) + expected: list[list[float]] = ( + [[1001.5, 99, 0.5, 2], [1002, 99, 1.5, 2]] + if index_field == 'index' + else [[115, 99, 5, 2], [140, 99, 15, 2]] + ) + np.testing.assert_allclose(rectangles, expected) + assert metadata == [span, span] + assert metadata[0].probability == 0.8 + assert metadata[0].source == 'jev' + + +def test_empty_and_unavailable_geometry() -> None: + ''' + A layer can attach before history arrives, and late classifier + responses can lie beyond retained history. Empty chart arrays + and already-expired intervals must produce empty geometry rather + than indexing the final row or drawing a retrospective rectangle. + + ''' + span: RegimeInterval = RegimeInterval(10, 20, 'ranging') + empty: np.ndarray + metadata: list[RegimeInterval] + empty, metadata = regime_rectangles([span], _chart(()), 1) + assert empty.shape == (0, 4) + assert metadata == [] + span = RegimeInterval(10, 20, 'ranging', observed_at=21) + empty, metadata = regime_rectangles([span], _chart((10, 20)), 1) + assert empty.shape == (0, 4) + assert metadata == [] + + +@pytest.mark.parametrize('period', [0, -1, np.nan, np.inf]) +def test_invalid_period_rejected(period: float) -> None: + ''' + Invalid sampling periods break binary-search interval semantics + and can create infinite widths. All non-positive/non-finite + period classes are rejected before constructing chart geometry. + + ''' + with pytest.raises(ValueError, match='period'): + times_to_x(_chart((10, 20)), np.array([10]), period) + + +@pytest.mark.parametrize( + 'market,revision,period,end_time,accepted', + [ + ('test.market', 3, 5, 130, True), + ('other.market', 3, 5, 130, False), + ('test.market', 4, 5, 130, False), + ('test.market', 3, 15, 130, False), + ('test.market', 3, 5, 119, False), + ], + ids=['matching', 'market', 'revision', 'period', 'future'], +) +def test_jev_adapter_retains_causal_evidence( + market: str, + revision: int, + period: float, + end_time: float, + accepted: bool, + +) -> None: + ''' + Late or misrouted classifications must not color an unrelated + market, source revision, or timeframe. A typed Jev response + claims source evidence at 100 but arrives at 120; matching + evidence retains its original probability/model and starts at + receipt time. The rejected cases vary one ownership bound each. + + ''' + result: JevClassification = JevClassification( + market='test.market', + revision=3, + sequence=40, + segment=2, + source_as_of=100, + requested_at=118, + received_at=120, + latency=2, + model='jev-test', + prompt_version='test', + answers=(RegimeAnswer( + question='regime_5', + period=5, + label='range', + probability=0.82, + confidence=0.7, + probabilities=(('range', 0.82), ('unknown', 0.18)), + ),), + input_tokens=100, + output_tokens=20, + ) + interval: RegimeInterval|None = classification_interval( + result, + fqme=market, + revision=revision, + period_s=period, + end_time=end_time, + ) + if not accepted: + assert interval is None + return + assert interval is not None + assert interval.start_time == 100 + assert interval.observed_at == 120 + assert interval.label == 'range' + assert interval.probability == 0.82 + assert interval.source == 'typesafe.ai/jev-test' + + +def test_overlay_rebuild_and_owned_teardown( + qapp: QApplication, + +) -> None: + ''' + Equal-bound backfills rewrite earlier pivot fields without + changing the live tail. An ordinary paint gate intentionally + skips that change; a forced revision must replace stale markers + and rectangles, preserve a sibling layer, and release exactly + the owning layer's two items on repeated cancellation cleanup. + Real Qt items prove scene ownership rather than a mocked API. + + ''' + source: np.ndarray = _source( + (100, 110, 120, 130, 140), (1, 1, -1, -1, 1), + ) + source['top_1'][1] = 105 + source['top_time_1'][1] = 100 + widget: pg.PlotWidget = pg.PlotWidget() + viz: SimpleNamespace = SimpleNamespace( + plot=widget.plotItem, + shm=SimpleNamespace(array=_chart((100, 110, 120, 130, 140))), + index_field='index', + ) + first: TrendOverlay = TrendOverlay( + viz, + SimpleNamespace(array=source), + fqme='first.test', + source_period_s=10, + chart_period_s=10, + ) + sibling: TrendOverlay = TrendOverlay( + viz, + SimpleNamespace(array=source), + fqme='second.test', + source_period_s=10, + chart_period_s=10, + ) + try: + assert first.update() + assert sibling.update() + assert len(first.points) == 1 + assert not first.update() + source['top_1'][1] = np.nan + source['bottom_1'][1] = 95 + source['bottom_time_1'][1] = 100 + assert not first.update() + assert first.update(force=True) + assert first.points[0].kind == 'bottom' + assert sibling.points[0].kind == 'top' + assert len(widget.plotItem.items) == 4 + + # A synthetic full-scene paint verifies Qt primitive types + # and brush batching without a GUI or screenshot dependency. + image: QtGui.QImage = QtGui.QImage( + 160, 100, QtGui.QImage.Format.Format_ARGB32, + ) + image.fill(0) + painter: QtGui.QPainter = QtGui.QPainter(image) + try: + widget.scene().render(painter) + finally: + painter.end() + + first.remove() + first.remove() + assert len(widget.plotItem.items) == 2 + assert sibling.markers in widget.plotItem.items + assert not first.update(force=True) + finally: + first.remove() + sibling.remove() + widget.close() + widget.deleteLater() + qapp.processEvents() + + +def test_provisional_updates_reuse_confirmed_markers( + qapp: QApplication, + +) -> None: + ''' + Quotes may repaint a provisional reversal repeatedly during one + bar. Rebuilding historical markers each time wastes UI time and + can present a temporary reversal as confirmed. Mutate only the + FSP tail, then the chart candle envelope: the first update is + skipped and the second updates rectangles while preserving the + same ``TrendOverlay.points`` list. Changing ``chart['index']`` + requires remapping those retained points. + + ''' + source: np.ndarray = _source((10, 20, 30), (1, 1, -1)) + source['top_1'][1] = 105 + source['top_time_1'][1] = 10 + widget: pg.PlotWidget = pg.PlotWidget() + chart: np.ndarray = _chart((10, 20, 30)) + viz: SimpleNamespace = SimpleNamespace( + plot=widget.plotItem, + shm=SimpleNamespace(array=chart), + index_field='index', + ) + overlay: TrendOverlay = TrendOverlay( + viz, + SimpleNamespace(array=source), + fqme='live.test', + source_period_s=10, + chart_period_s=10, + ) + try: + assert overlay.update() + original: list[ReversalPoint] = overlay.points + source['trend'][-1] = 1 + source['bottom_1'][-1] = 91 + source['bottom_time_1'][-1] = 20 + assert not overlay.update() + assert overlay.points is original + + chart['high'][-1] = 109 + assert overlay.update() + assert overlay.points is original + assert overlay.rectangles.rectangles[-1, 3] == 10 + + chart['high'][-1] = 100 + assert overlay.update() + assert overlay.points is original + assert overlay.rectangles.rectangles[-1, 3] == 2 + + chart['index'] += 500 + assert overlay.update() + assert overlay.points is not original + assert overlay.points[0].x == 1500 + finally: + overlay.remove() + widget.close() + widget.deleteLater() + qapp.processEvents() + + +def test_overlay_limits_and_classifier_metadata( + qapp: QApplication, + +) -> None: + ''' + Unbounded event layers can allocate thousands of Qt objects and + delayed classifier results can overwrite deterministic meaning. + Many synthetic pivots exercise the configured retention limits; + replacing rectangles with a Jev result must retain probability, + source, and markers while keeping exactly two scene items. + + ''' + source: np.ndarray = _source(tuple(range(20)), (1, -1) * 10) + source['top_1'] = 105 + source['top_time_1'] = source['time'] + widget: pg.PlotWidget = pg.PlotWidget() + viz: SimpleNamespace = SimpleNamespace( + plot=widget.plotItem, + shm=SimpleNamespace(array=_chart(tuple(range(20)))), + index_field='index', + ) + overlay: TrendOverlay = TrendOverlay( + viz, + SimpleNamespace(array=source), + fqme='bounded.test', + source_period_s=1, + chart_period_s=1, + max_points=3, + max_rectangles=2, + ) + try: + overlay.update() + assert len(overlay.points) == 3 + assert overlay.points[0].pivot_time == 16 + assert len(overlay.rectangles.rectangles) == 2 + assert len(widget.plotItem.items) == 2 + + classified: RegimeInterval = RegimeInterval( + start_time=15, + end_time=20, + label='ranging', + probability=0.73, + source='jev', + observed_at=18, + ) + overlay.set_regimes([classified]) + assert len(overlay.points) == 3 + assert overlay.rectangles.intervals == [classified] + assert overlay.rectangles.rectangles[0, 0] == 1018 + tip: str = overlay.rectangles.tooltip_at(QPointF(1019, 100)) + assert 'ranging: 73.0%' in tip + assert '(1s)' in tip + assert 'jev' in tip + assert '00:00:18.000' in tip + overlay.set_regimes([]) + assert len(overlay.rectangles.rectangles) == 0 + overlay.set_regimes(None) + assert len(overlay.rectangles.rectangles) == 2 + tip = overlay.rectangles.tooltip_at(QPointF(1019, 100)) + assert 'trend-calculus' in tip + assert '%' not in tip + finally: + overlay.remove() + widget.close() + widget.deleteLater() + qapp.processEvents() + + +class _PriceChart: + ''' + Expose the price-chart API while retaining actual Qt ownership. + + ''' + _trend_overlays: dict[tuple[str, str], list[TrendOverlay]] + + def __init__(self) -> None: + ''' + Build a chart whose SHM token identifies its base timeframe. + + ''' + self.widget: pg.PlotWidget = pg.PlotWidget() + self.viz: SimpleNamespace = SimpleNamespace( + plot=self.widget.plotItem, + shm=SimpleNamespace( + array=_chart((10, 20, 30, 40, 50)), + _token='ohlcv', + ), + index_field='index', + ) + + def get_viz(self, fqme: str) -> SimpleNamespace|None: + ''' + Resolve exactly the test market rather than any chart key. + + ''' + return self.viz if fqme == 'admin.test' else None + + def close(self) -> None: + ''' + Release the disposable offscreen widget after its layers. + + ''' + self.widget.close() + self.widget.deleteLater() + + +async def _admin_bootstrap_scenario( + qapp: QApplication, + monkeypatch: pytest.MonkeyPatch, + +) -> None: + ''' + Attaching before FSP bootstrap exposes empty history, and opening + several reversal orders must not start multiple SHM writers. + The ``attach_first()`` task calls ``FspAdmin.open_trend_overlay`` + while the mocked engine's ``started`` event stays unset. + ``wait_all_tasks_blocked()`` reaches that wait deterministically; + one engine call and an empty Qt plot rule out early attachment. + Only setting ``started`` lets ``attached`` confirm publication. + Reopening another order retains one writer, and repeated + ``FspAdmin.close_trend_overlays()`` preserves a sibling's items. + This exercises local lifecycle ordering, not actor IPC. + + ''' + chart: _PriceChart = _PriceChart() + source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5) + source['trend_2'] = 1 + destination: SimpleNamespace = SimpleNamespace( + rt_shm=SimpleNamespace(array=source, _token='trend'), + ) + flume: SimpleNamespace = SimpleNamespace( + mkt=SimpleNamespace(fqme='admin.test'), + rt_shm=chart.viz.shm, + rt_sample_period_s=10, + _hist_shm_token=None, + ) + started: trio.Event = trio.Event() + attached: trio.Event = trio.Event() + calls: list[tuple[str, dict, str]] = [] + attached_layers: list[TrendOverlay] = [] + + async def start_engine( + target: Fsp, + conf: dict, + loglevel: str, + + ) -> tuple[SimpleNamespace, trio.Event]: + ''' + Return an allocated destination before publishing bootstrap. + + ''' + calls.append((target.name, conf, loglevel)) + return destination, started + + try: + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + admin: FspAdmin = FspAdmin( + nursery, {}, SimpleNamespace(chart=chart), flume, + ) + monkeypatch.setattr( + admin, 'start_engine_task', start_engine, + ) + with pytest.raises(ValueError, match='positive'): + await admin.open_trend_overlay(chart_period_s=0) + assert calls == [] + + async def attach_first() -> None: + ''' + Signal only after startup has produced visible data. + + ''' + layer: TrendOverlay = ( + await admin.open_trend_overlay() + ) + attached_layers.append(layer) + attached.set() + + nursery.start_soon(attach_first) + await wait_all_tasks_blocked() + assert len(calls) == 1 + assert not attached.is_set() + assert chart.widget.plotItem.items == [] + + started.set() + await attached.wait() + first: TrendOverlay = attached_layers[0] + second: TrendOverlay = await admin.open_trend_overlay( + order=2, + ) + assert await admin.open_trend_overlay() is first + with pytest.raises(ValueError, match='period differs'): + await admin.open_trend_overlay(chart_period_s=60) + assert second is not first + assert len(calls) == 1 + assert calls[0][0] == 'trend_calculus' + assert chart._trend_overlays == { + ('admin.test', 'trend'): [first, second], + } + assert len(chart.widget.plotItem.items) == 4 + + sibling: TrendOverlay = TrendOverlay( + chart.viz, + SimpleNamespace(array=source, _token='other'), + fqme='other.test', + source_period_s=10, + chart_period_s=10, + ) + foreign_key: tuple[str, str] = ('other.test', 'other') + chart._trend_overlays[foreign_key] = [sibling] + try: + admin.close_trend_overlays() + admin.close_trend_overlays() + assert len(chart.widget.plotItem.items) == 2 + assert chart._trend_overlays == { + ('other.test', 'other'): [sibling], + } + finally: + sibling.remove() + finally: + chart.close() + qapp.processEvents() + + +def test_admin_waits_for_bootstrap_and_reuses_writer( + qapp: QApplication, + monkeypatch: pytest.MonkeyPatch, + +) -> None: + ''' + Exercise bootstrap publication and owner cleanup in local Trio. + + ``_admin_bootstrap_scenario()`` controls its ``started`` event + and waits for ``attach_first()`` to block in + ``FspAdmin.open_trend_overlay()`` before inspecting the plot. + The unset ``attached`` event and empty plot prove attachment + waits for bootstrap rather than depending on timer luck. A + second order shares the writer; cleanup retains a sibling's + ``_PriceChart._trend_overlays`` entry and its two Qt items. + + ''' + trio.run(_admin_bootstrap_scenario, qapp, monkeypatch) + + +def test_registry_filters_and_defers_hidden_revisions( + qapp: QApplication, + +) -> None: + ''' + A destination rewrite while hidden must survive an unchanged + source-tail paint signature when the chart becomes visible. + An equal-bound earlier pivot edit arranges that case. Wrong + market/token/timeframe invalidations leave the layer alone; + a matching hidden notification invalidates without touching Qt + data, and the next ordinary refresh consumes the corrected pivot. + + ''' + chart: _PriceChart = _PriceChart() + source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5) + source['top_1'][1] = 105 + source['top_time_1'][1] = 10 + overlay: TrendOverlay = TrendOverlay( + chart.viz, + SimpleNamespace(array=source, _token='trend'), + fqme='admin.test', + source_period_s=10, + chart_period_s=10, + ) + chart._trend_overlays = {('admin.test', 'trend'): [overlay]} + try: + assert update_trend_overlays(chart) == 1 + original: list[ReversalPoint] = overlay.points + source['top_1'][1] = 109 + assert update_trend_overlays( + chart, fqme='other.test', force=True, + ) == 0 + assert update_trend_overlays( + chart, shm_token='other', force=True, + ) == 0 + assert update_trend_overlays( + chart, chart_period_s=60, force=True, + ) == 0 + assert update_trend_overlays(chart) == 0 + assert overlay.points is original + + admin: FspAdmin = FspAdmin( + None, {}, SimpleNamespace(chart=chart), None, + ) + admin._trend_overlays.append((chart, overlay)) + admin.refresh_trend_overlays('trend', render=False) + assert overlay.points is original + assert overlay.points[0].price == 105 + assert update_trend_overlays(chart) == 1 + assert overlay.points[0].price == 109 + finally: + overlay.remove() + chart.close() + qapp.processEvents() + + +def test_admin_context_exit_releases_layers( + qapp: QApplication, + monkeypatch: pytest.MonkeyPatch, + +) -> None: + ''' + A finished FSP-admin context must not leave its graphics attached + to a reused chart widget. Replace only actor-cluster startup and + engine allocation; ``open_fsp_admin()`` and its Trio nursery run + normal exit with actual Qt items. Empty + ``FspAdmin._trend_overlays`` and ``_PriceChart._trend_overlays`` + plus the signalled ``complete`` event from + ``FspAdmin._registry`` prove graphics teardown and existing + engine shutdown notification both run. This local stub does not + prove remote actor shutdown or shared-memory visibility. + + ''' + chart: _PriceChart = _PriceChart() + + @asynccontextmanager + async def cluster_context( + **kwargs: object, + + ) -> AsyncIterator[tuple[bool, dict]]: + ''' + Supply a local cluster stub without spawning actor processes. + + ''' + yield False, {} + + monkeypatch.setattr( + fsp_ui, 'maybe_open_context', cluster_context, + ) + + async def scenario() -> None: + ''' + Observe cleanup after the actual admin context has exited. + + ''' + flume: SimpleNamespace = SimpleNamespace( + mkt=SimpleNamespace(fqme='admin.test'), + rt_shm=chart.viz.shm, + rt_sample_period_s=10, + _hist_shm_token=None, + ) + source: np.ndarray = _source( + (10, 20, 30, 40, 50), (1,) * 5, + ) + started: trio.Event = trio.Event() + started.set() + complete: trio.Event = trio.Event() + + async def start_engine( + target: Fsp, + conf: dict, + loglevel: str, + + ) -> tuple[SimpleNamespace, trio.Event]: + ''' + Return an already published, test-owned FSP destination. + + ''' + return SimpleNamespace(rt_shm=SimpleNamespace( + array=source, + _token='trend', + )), started + + admin: FspAdmin + async with fsp_ui.open_fsp_admin( + SimpleNamespace(chart=chart), flume, + ) as admin: + monkeypatch.setattr( + admin, 'start_engine_task', start_engine, + ) + admin._registry[('test',)] = (None, None, complete) + await admin.open_trend_overlay() + assert len(chart.widget.plotItem.items) == 2 + + assert chart.widget.plotItem.items == [] + assert chart._trend_overlays == {} + assert admin._trend_overlays == [] + assert complete.is_set() + + try: + trio.run(scenario) + finally: + chart.close() + qapp.processEvents() diff --git a/tests/test_trend_mtf_graphics.py b/tests/test_trend_mtf_graphics.py new file mode 100644 index 00000000..9260da3e --- /dev/null +++ b/tests/test_trend_mtf_graphics.py @@ -0,0 +1,611 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Actual shared-tracker graphics, freshness and native feed lifecycle. + +''' +from collections import deque +from collections.abc import AsyncIterator, Iterator +from contextlib import asynccontextmanager +from dataclasses import replace +import os +from types import SimpleNamespace + +os.environ['QT_QPA_PLATFORM'] = 'offscreen' + +import numpy as np +import pyqtgraph as pg +import pytest +import trio +from trio.testing import wait_all_tasks_blocked + +from piker.fsp._jev import JevClassification, RegimeAnswer +from piker.fsp._regime import ( + MultiTimeframeTrend, + RegimeSnapshot, + TimeframeRegime, + TrendReversal, +) +from piker.ui import _fsp as fsp_ui +from piker.ui._trend import ( + RegimeInterval, + ReversalPoint, + TrendOverlay, +) +from piker.ui._trend_mtf import ( + SnapshotTrendOverlay, + _SnapshotLayer, +) +from piker.ui.qt import QApplication + + +class _ArrayShm: + ''' + Model published source bounds over deterministic OHLC bars. + + ''' + def __init__(self, count: int = 600, visible: int = 401) -> None: + ''' + Keep an allocated mutable suffix beyond the published range. + + ''' + self._array: np.ndarray = np.zeros(count, dtype=[ + ('index', 'i8'), + ('time', 'f8'), + ('close', 'f8'), + ('low', 'f8'), + ('high', 'f8'), + ]) + steps: np.ndarray = np.arange(count) + prices: np.ndarray = ( + 100 + .015 * steps + + 2 * np.sin(steps * .09) + + .5 * np.sin(steps * .4) + ) + self._array['index'] = 1000 + steps + self._array['time'] = steps + self._array['close'] = prices + self._array['low'] = prices - .2 + self._array['high'] = prices + .2 + self._first: SimpleNamespace = SimpleNamespace(value=0) + self._last: SimpleNamespace = SimpleNamespace(value=visible) + self._token: str = 'native-ohlcv' + + @property + def array(self) -> np.ndarray: + ''' + Expose only the current published SHM range. + + ''' + return self._array[self._first.value:self._last.value] + + +class _Chart: + ''' + Use a real price plot while replacing actor and SHM allocation. + + ''' + _trend_overlays: dict[tuple[str, str], list[TrendOverlay]] + + def __init__(self, shm: _ArrayShm) -> None: + ''' + Bind the actual base source to an offscreen plot. + + ''' + self.widget: pg.PlotWidget = pg.PlotWidget() + self.viz: SimpleNamespace = SimpleNamespace( + plot=self.widget.plotItem, + shm=shm, + index_field='index', + ) + + def get_viz(self, fqme: str) -> SimpleNamespace|None: + ''' + Resolve the only market owned by this chart. + + ''' + return self.viz if fqme == 'mtf.test' else None + + def close(self) -> None: + ''' + Release the disposable Qt widget. + + ''' + self.widget.close() + self.widget.deleteLater() + + +@pytest.fixture(scope='session') +def qapp() -> Iterator[QApplication]: + ''' + Keep one offscreen application alive for native graphics checks. + + ''' + app: QApplication|None = QApplication.instance() + if app is None: + app = QApplication(['piker-mtf-tests']) + app.setQuitOnLastWindowClosed(False) + yield app + app.processEvents() + + +def _replay( + tracker: MultiTimeframeTrend, + shm: _ArrayShm, + overlay: SnapshotTrendOverlay|None = None, + start: int = 0, + +) -> RegimeSnapshot: + ''' + Replay final closes through the production shared recurrence. + + ''' + snapshot: RegimeSnapshot + row: np.void + for row in shm.array[start:-1]: + snapshot = tracker.push( + float(row['time']), + float(row['close']), + index=int(row['index']), + ) + if overlay is not None: + overlay.submit(snapshot) + return snapshot + + +def _classification( + snapshot: RegimeSnapshot, + *, + received_at: float|None = None, + +) -> JevClassification: + ''' + Build explicit synthetic provider output without any HTTP path. + + ''' + return JevClassification( + market=snapshot.market, + revision=snapshot.revision, + sequence=snapshot.sequence, + segment=snapshot.segment, + source_as_of=snapshot.as_of, + requested_at=snapshot.as_of, + received_at=( + received_at + if received_at is not None + else snapshot.as_of + ), + latency=.1, + model='synthetic-jev-test', + prompt_version='test', + answers=(RegimeAnswer( + question='regime_5', + period=5, + label='range', + probability=.8, + confidence=.6, + probabilities=(('range', .8), ('unknown', .2)), + ),), + input_tokens=0, + output_tokens=0, + ) + + +def test_bad_limits_do_not_allocate_graphics( + qapp: QApplication, + +) -> None: + ''' + Reject fractional and boolean bounds before attaching Qt items. + + A failed child constructor cannot be cleaned up by the parent + because the parent has not received it yet. Exercise the public + scalar and snapshot constructors against an actual empty plot. + + ''' + shm: _ArrayShm = _ArrayShm() + chart: _Chart = _Chart(shm) + try: + name: str + for name in ('history_limit', 'order'): + value: float|bool + for value in (1.5, True): + with pytest.raises(ValueError): + SnapshotTrendOverlay( + chart.viz, shm, + fqme='mtf.test', base_period=1, + **{name: value}, + ) + assert not chart.widget.plotItem.items + + for name in ('max_points', 'max_rectangles', 'order'): + value: float|bool + for value in (1.5, True): + with pytest.raises(ValueError): + TrendOverlay( + chart.viz, shm, + fqme='mtf.test', source_period_s=1, + chart_period_s=1, + **{name: value}, + ) + assert not chart.widget.plotItem.items + finally: + chart.close() + qapp.processEvents() + + +@pytest.mark.parametrize('periods', [ + (1., 5., 15.), + (15.000000000000002, 1.0000000000000002, 5.000000000000001), +]) +def test_bootstrap_uses_real_timeframes_and_endpoint_pivots( + qapp: QApplication, + periods: tuple[float, ...], + +) -> None: + ''' + Sampling 1 s, 5 s and 15 s bars must aggregate one base stream, + rather than relabel identical computations. A real tracker and + scalar sine-wave closes create distinct reversal sets. The + snapshot's pivot end timestamp remains metadata while its x + coordinate equals the final base bar's ``TrendReversal.index``. + Bootstrap supplies no old trend-row history, so rectangles may + begin only at each timeframe's latest proven as_of timestamp. + Harmless decimal rounding must normalize to the tracker's exact + duration keys even if the caller lists timeframes out of order. + + ''' + shm: _ArrayShm = _ArrayShm() + chart: _Chart = _Chart(shm) + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'mtf.test', periods=periods, history_limit=256, + ) + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + chart.viz, shm, fqme='mtf.test', base_period=1, + periods=periods, + ) + try: + snapshot: RegimeSnapshot = _replay(tracker, shm) + assert adapter.submit(snapshot) + assert adapter.periods == (1., 5., 15.) + assert len(chart.widget.plotItem.items) == 6 + point_counts: list[int] = [] + frame: TimeframeRegime + for frame in snapshot.timeframes: + layer: _SnapshotLayer = adapter.layers[frame.period] + event: TrendReversal + events: list[TrendReversal] = [ + event for event in frame.reversals + if event.order == 1 + ] + assert len(layer.points) == len(events) + assert layer.points + point_counts.append(len(layer.points)) + point: ReversalPoint + event: TrendReversal + for point, event in zip( + layer.points, events, strict=True, + ): + assert point.x == event.index + assert point.pivot_time == event.time + assert point.detected_at == event.detected_time + assert point.timestamp_convention == 'interval_end' + assert layer.rectangles.intervals + span: RegimeInterval # noqa: F842 + assert all( + span.start_time >= frame.as_of + for span in layer.rectangles.intervals + ) + assert len(set(point_counts)) == 3 + finally: + adapter.remove() + chart.close() + qapp.processEvents() + + +def test_revision_segment_and_sequence_revoke_evidence( + qapp: QApplication, + +) -> None: + ''' + A corrected source or new gap segment invalidates every previous + pivot and model annotation before replay finishes. The adapter + receives real tracker output, then an explicit generation change. + Old market/revision/sequence snapshots and an earlier segment's + model response must fail acceptance without reviving geometry. + + ''' + shm: _ArrayShm = _ArrayShm() + chart: _Chart = _Chart(shm) + tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test') + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + chart.viz, shm, fqme='mtf.test', base_period=1, + ) + try: + snapshot: RegimeSnapshot = _replay(tracker, shm) + assert adapter.submit(snapshot) + result: JevClassification = _classification(snapshot) + assert adapter.submit_classification(result) + assert not adapter.submit(snapshot) + assert not adapter.submit( + replace(snapshot, market='foreign'), + ) + adapter.invalidate(1) + layer: _SnapshotLayer # noqa: F842 + assert all( + not layer.points for layer in adapter.layers.values() + ) + assert not adapter.submit(snapshot) + assert not adapter.submit_classification(result) + + tracker.reset(revision=1) + replayed: RegimeSnapshot = _replay(tracker, shm) + assert adapter.submit(replayed) + result = _classification(replayed) + assert adapter.submit_classification(result) + after_gap: RegimeSnapshot = tracker.push( + 405, 101, index=1405, + ) + assert adapter.submit(after_gap) + assert adapter.segment == 1 + assert not adapter.submit_classification(result) + assert all( + not layer.points for layer in adapter.layers.values() + ) + runs: deque[RegimeInterval] # noqa: F842 + assert all(not runs for runs in adapter._model_runs.values()) + finally: + adapter.remove() + chart.close() + qapp.processEvents() + + +def test_coalesced_buckets_leave_unknown_regime_hole( + qapp: QApplication, + +) -> None: + ''' + Coalesced callbacks reveal only the latest trend classification. + Extending ``_SnapshotLayer._runs`` to the next received snapshot + invents historical knowledge. Skip ten actual finalized buckets + and prove the old interval stops at its next possible decision + while the new run starts only at ``RegimeSnapshot.as_of``. + + ''' + shm: _ArrayShm = _ArrayShm(visible=101) + chart: _Chart = _Chart(shm) + tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test') + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + chart.viz, shm, fqme='mtf.test', base_period=1, + ) + try: + adapter.submit(_replay(tracker, shm)) + previous: float = adapter.as_of + shm._last.value += 10 + latest: RegimeSnapshot = _replay(tracker, shm, start=100) + adapter.submit(latest) + runs: deque[RegimeInterval] = adapter.layers[1]._runs + assert len(runs) == 2 + assert runs[0].end_time == previous + 1 + assert runs[1].start_time == latest.as_of + assert runs[0].end_time < runs[1].start_time + finally: + adapter.remove() + chart.close() + qapp.processEvents() + + +def test_model_receipt_expiry_and_delayed_queue( + qapp: QApplication, + +) -> None: + ''' + A Jev response must neither paint before receipt nor claim the + live regime indefinitely. Retain a response arriving 2.5 s + after its source snapshot, advance the real base stream, and + verify its visible interval starts at receipt and stops after + the default 5 s lifetime. Old first-ever responses and expired + queued results are also rejected despite matching provenance. + + ''' + shm: _ArrayShm = _ArrayShm(visible=201) + chart: _Chart = _Chart(shm) + tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test') + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + chart.viz, shm, fqme='mtf.test', base_period=1, + ) + try: + snapshot: RegimeSnapshot = _replay(tracker, shm) + adapter.submit(snapshot) + old: JevClassification = replace( + _classification(snapshot), + sequence=1, + source_as_of=1, + received_at=200, + ) + assert not adapter.submit_classification(old) + expired: JevClassification = _classification( + snapshot, received_at=190, + ) + assert not adapter.submit_classification(expired) + + result: JevClassification = _classification( + snapshot, received_at=202.5, + ) + assert adapter.submit_classification(result) + layer: _SnapshotLayer = adapter.layers[5] + assert len(layer.rectangles.rectangles) == 0 + shm._last.value = 212 + _replay(tracker, shm, adapter, start=200) + assert layer.rectangles.intervals + span: RegimeInterval = layer.rectangles.intervals[-1] + assert span.observed_at == 202.5 + assert span.end_time == 207.5 + assert layer.rectangles.rectangles[-1, 0] == 1202.5 + assert not adapter.submit_classification(result) + finally: + adapter.remove() + chart.close() + qapp.processEvents() + + +def test_snapshot_retention_stays_bounded( + qapp: QApplication, + +) -> None: + ''' + A long-running shared tracker must not accumulate unbounded Qt + objects or regime intervals. Stream hundreds of real snapshots + with a four-event display budget and assert retention at both + the data and scene boundaries after many reversals. + + ''' + shm: _ArrayShm = _ArrayShm() + chart: _Chart = _Chart(shm) + tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test') + adapter: SnapshotTrendOverlay = SnapshotTrendOverlay( + chart.viz, + shm, + fqme='mtf.test', + base_period=1, + history_limit=4, + ) + try: + _replay(tracker, shm, adapter) + assert len(chart.widget.plotItem.items) == 6 + layer: _SnapshotLayer + for layer in adapter.layers.values(): + assert len(layer.points) <= 4 + assert len(layer._runs) <= 4 + assert len(layer.rectangles.rectangles) <= 4 + finally: + adapter.remove() + adapter.remove() + assert chart.widget.plotItem.items == [] + chart.close() + qapp.processEvents() + + +@pytest.mark.parametrize('max_source_age', [5, None]) +def test_admin_consumes_native_source_once_and_cancels( + qapp: QApplication, + monkeypatch: pytest.MonkeyPatch, + max_source_age: float|None, + +) -> None: + ''' + ``FspAdmin._run_multi_timeframe_trends()`` must subscribe before + replay, share one tracker across three periods, and invalidate + graphics while model callbacks apply their freshness gate. + The ``sampler()`` stub yields a local Trio receive channel to + the actual ``consume_regime_source()`` task. Explicit append and + equal-bound correction notices, followed by + ``wait_all_tasks_blocked()``, prove each notice is consumed + before checking snapshot sequence and revision. + ``FspAdmin.close_trend_overlays()`` must cancel the reader: + ``closed`` is set only in ``sampler()`` cleanup, before this test + closes either channel endpoint. Empty Qt items and + ``_Chart._trend_overlays`` prove owned graphics also detach. + This controls local task ordering without exercising actor IPC. + + ''' + shm: _ArrayShm = _ArrayShm(visible=201) + chart: _Chart = _Chart(shm) + submitted: list[RegimeSnapshot] = [] + invalidated: list[int] = [] + subscription_count: list[float] = [] + + async def scenario() -> None: + ''' + Control publication and cancellation with native Trio events. + + ''' + send: trio.MemorySendChannel[dict] + receive: trio.MemoryReceiveChannel[dict] + send, receive = trio.open_memory_channel(4) + closed: trio.Event = trio.Event() + + @asynccontextmanager + async def sampler( + period_s: float, + + ) -> AsyncIterator[trio.MemoryReceiveChannel[dict]]: + ''' + Open exactly one sampler before the real consumer starts. + + ''' + subscription_count.append(period_s) + try: + yield receive + finally: + closed.set() + + monkeypatch.setattr(fsp_ui, 'open_sample_stream', sampler) + linked: SimpleNamespace = SimpleNamespace( + chart=chart, + isHidden=lambda: False, + ) + flume: SimpleNamespace = SimpleNamespace( + mkt=SimpleNamespace(fqme='mtf.test'), + rt_shm=shm, + rt_sample_period_s=1, + ) + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + admin: fsp_ui.FspAdmin = fsp_ui.FspAdmin( + nursery, {}, linked, flume, + ) + adapter: SnapshotTrendOverlay = ( + await admin.open_multi_timeframe_trends( + on_snapshot=submitted.append, + on_invalidate=invalidated.append, + max_source_age=max_source_age, + ) + ) + assert ( + await admin.open_multi_timeframe_trends() is adapter + ) + assert subscription_count == [1] + assert len(chart.widget.plotItem.items) == 6 + assert adapter.sequence == 200 + assert bool(submitted) == (max_source_age is None) + + shm._last.value += 1 + await send.send({'period': 1}) + await wait_all_tasks_blocked() + assert adapter.sequence == 201 + + shm._array['close'][20] += 20 + await send.send({'backfilling': ('mtf.test', 1)}) + await wait_all_tasks_blocked() + assert invalidated == [1] + assert adapter.revision == 1 + assert adapter.sequence == 201 + admin.close_trend_overlays() + await closed.wait() + assert chart._trend_overlays == {} + assert chart.widget.plotItem.items == [] + await send.aclose() + await receive.aclose() + + try: + trio.run(scenario) + finally: + chart.close() + qapp.processEvents() diff --git a/tests/test_trend_reference.py b/tests/test_trend_reference.py new file mode 100644 index 00000000..1c95c954 --- /dev/null +++ b/tests/test_trend_reference.py @@ -0,0 +1,430 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Independent batch FHLS oracle and pinned original-Lua observations. + +The oracle materializes complete windows and an expanded sequence of +FHLS summaries. It shares neither compiled helpers nor mutable window +state with the production calculator. See docs/trend_calculus.rst. + +''' +from dataclasses import dataclass +from itertools import product +import json +from pathlib import Path +from typing import TypedDict + +import numpy as np +import pytest + +from piker.fsp._trend import ( + TrendCalculator, + TrendHistory, + TrendUpdate, +) + + +class _LuaCase(TypedDict): + ''' + Describe the input and observed events of one pinned Lua run. + + ''' + window: int + values: list[float] + events: list[list[int|float]] + + +class _LuaFixture(TypedDict): + ''' + Type the captured Lua cases consumed by the parity regression. + + ''' + cases: list[_LuaCase] + + +@dataclass(frozen=True) +class Point: + ''' + Retain a pivot's source position and earliest availability row. + + ''' + row: int + price: float + available: int + + +@dataclass(frozen=True) +class Window: + ''' + Summarize an entire immutable observation window using extrema. + + ''' + high: Point + low: Point + available: int + + @classmethod + def summarize( + cls: type['Window'], + points: list[Point], + ) -> 'Window': + ''' + Python min/max retain the earliest equal extremum. + + ''' + return cls( + high=max(points, key=lambda point: point.price), + low=min(points, key=lambda point: point.price), + available=points[-1].available, + ) + + def compare(self, previous: 'Window') -> int: + ''' + Compare extrema ordinally, avoiding floating subtraction. + + ''' + if ( + ( + self.high.price >= previous.high.price + and self.low.price >= previous.low.price + ) + and + ( + self.high.price > previous.high.price + or self.low.price > previous.low.price + ) + ): + return 1 + if ( + ( + self.high.price <= previous.high.price + and self.low.price <= previous.low.price + ) + and + ( + self.high.price < previous.high.price + or self.low.price < previous.low.price + ) + ): + return -1 + return 0 + + +def reference_order( + points: list[Point], + size: int, + *, + lua_seed: bool = False, +) -> tuple[list[tuple[int, int]], list[tuple[Point, int]]]: + ''' + Expand complex windows, then derive directions and pivot events. + + Direction changes from unknown zero establish a baseline and do + not manufacture a reversal. ``Point.available`` carries detection + availability from the closing observation, including at recursive + orders. + + ''' + start: int # noqa: F842 + windows: list[Window] = [ + Window.summarize(points[start:start + size]) + for start in range(0, len(points) - size + 1, size) + ] + if lua_seed and windows: + last: Point = points[size - 1] + windows.insert(0, Window(last, last, last.available)) + if len(windows) < 2: + return [], [] + + expanded: list[Window] = [windows[0]] + current: Window + previous: Window + for previous, current in zip(windows, windows[1:]): + if current.compare(previous) == 0: + second: Point = max( + (previous.low, previous.high), + key=lambda point: point.row, + ) + first: Point = min( + (current.low, current.high), + key=lambda point: point.row, + ) + # Lua's synthetic bridge breaks equal-value ties toward + # its second endpoint for high and first for low. + bridge_high: Point + bridge_low: Point + bridge_low, bridge_high = sorted( + [second, first], key=lambda point: point.price, + ) + expanded.append(Window( + high=bridge_high, + low=bridge_low, + available=current.available, + )) + expanded.append(current) + + trend: int = 0 + directions: list[tuple[int, int]] = [] + events: list[tuple[Point, int]] = [] + previous: Window + current: Window + for previous, current in zip(expanded, expanded[1:]): + direction: int = current.compare(previous) or trend + if trend and direction != trend: + pivot: Point = ( + previous.low if direction == 1 else previous.high + ) + events.append(( + Point(pivot.row, pivot.price, current.available), + direction, + )) + trend = direction + directions.append((current.available, trend)) + return directions, events + + +def reference_hierarchy( + values: np.ndarray, + sizes: tuple[int, ...], + *, + lua_seed: bool = False, +) -> tuple[ + np.ndarray, + np.ndarray, + list[tuple[int, int, int, int]], +]: + ''' + Repeatedly reduce whole point sequences, separately per order. + + The returned event tuples are pivot row, direction, order and + detection row. Sorting by detection row and order reproduces + causal publication while retaining stable same-order ordering. + + ''' + row: int + price: np.floating|np.integer # noqa: F842 + points: list[Point] = [ + Point(row, float(price), row) + for row, price in enumerate(values) + ] + trends: np.ndarray = np.zeros( + (len(values), len(sizes)), dtype=np.int8, + ) + ready: np.ndarray = np.zeros_like(trends, dtype=np.bool_) + all_events: list[tuple[int, int, int, int]] = [] + level: int + size: int + for level, size in enumerate(sizes): + directions: list[tuple[int, int]] + events: list[tuple[Point, int]] + directions, events = reference_order( + points, size, lua_seed=lua_seed, + ) + if len(points) >= 2 * size: + ready[points[2 * size - 1].available:, level] = True + row: int + direction: int + for row, direction in directions: + trends[row:, level] = direction + point: Point + direction: int + for point, direction in events: + all_events.append(( + point.row, direction, level + 1, point.available, + )) + point: Point + direction: int + points = [point for point, direction in events] + all_events.sort(key=lambda event: (event[3], event[2])) + # Public direction remains unresolved until two real windows, + # including when the research seed establishes internal state. + trends[~ready] = 0 + return trends, ready, all_events + + +@pytest.mark.parametrize('sizes', [(2, 2), (3, 2), (4, 3)]) +@pytest.mark.parametrize('lua_seed', [False, True]) +def test_exhaustive_ternary_histories( + sizes: tuple[int, ...], + lua_seed: bool, +) -> None: + ''' + Enumerate ties, inside/outside bars and unresolved flat startup. + + Every eight-point sequence over three prices is compared with + independently expanded ``reference_hierarchy()`` windows. + Six-point sequences also exercise the optional Lua seed, without + importing its CLI's output-suppression bugs. Readiness still + requires two actual complete windows. Comparing every + ``TrendHistory`` field catches incorrect first/second-extremum + ordering and warmup decisions that agreement between two calls + into the same production kernel cannot detect. + + ''' + length: int = 6 if lua_seed else 8 + sequence: tuple[int, ...] + for sequence in product((-1, 0, 1), repeat=length): + values: np.ndarray = np.array(sequence, dtype=np.float64) + expected: np.ndarray + readiness: np.ndarray + events: list[tuple[int, int, int, int]] + ( + expected, + readiness, + events, + ) = reference_hierarchy( + values, sizes, lua_seed=lua_seed, + ) + actual: TrendHistory = TrendCalculator( + sizes, lua_seed=lua_seed, + ).extend( + values, np.arange(len(values), dtype=np.float64), + ) + assert np.array_equal(actual.trends, expected), sequence + assert np.array_equal(actual.ready, readiness), sequence + pulses: np.ndarray = np.zeros_like(expected) + prices: np.ndarray = np.full( + (len(values), len(sizes), 2), np.nan, + ) + times: np.ndarray = np.full_like(prices, np.nan) + pivot: int + direction: int + order: int + detected: int + for ( + pivot, + direction, + order, + detected, + ) in events: + side: int = int(direction == 1) + pulses[detected, order - 1] = direction + prices[detected, order - 1, side] = values[pivot] + times[detected, order - 1, side] = pivot + assert np.array_equal(actual.pulses, pulses), sequence + assert np.array_equal( + actual.pivot_prices, prices, equal_nan=True, + ), sequence + assert np.array_equal( + actual.pivot_times, times, equal_nan=True, + ), sequence + + +@pytest.mark.parametrize('sizes', [(2,) * 5, (3, 2, 4), (5, 3, 2)]) +@pytest.mark.parametrize('seed', range(8)) +@pytest.mark.parametrize('lua_seed', [False, True]) +def test_random_recursive_events( + sizes: tuple[int, ...], + seed: int, + lua_seed: bool, +) -> None: + ''' + Preserve recursive pivot causality across irregular coordinates. + + Higher-order pivots occur earlier than their availability, often + sharing one detection row. Rounded random walks create tied and + engulfing windows; ``reference_hierarchy()`` predicts every + event independently of ``TrendCalculator.push()``. Gapped source + indices and nonuniform timestamps prove that propagation retains + source identity and detection time. Field equality and complete + ordered-event equality detect lost or backdated reversals. + + ''' + rng: np.random.Generator = np.random.default_rng(seed) + values: np.ndarray = np.cumsum(rng.integers(-3, 4, 512)) + times: np.ndarray = 1000 + np.cumsum(rng.uniform(.1, 2, 512)) + indices: np.ndarray = 800 + np.cumsum(rng.integers(1, 4, 512)) + expected: np.ndarray + readiness: np.ndarray + events: list[tuple[int, int, int, int]] + ( + expected, + readiness, + events, + ) = reference_hierarchy( + values, sizes, lua_seed=lua_seed, + ) + calculator: TrendCalculator = TrendCalculator( + sizes, lua_seed=lua_seed, + ) + actual_events: list[tuple[int, int, int, int]] = [] + row: int + value: np.integer + for row, value in enumerate(values): + update: TrendUpdate = calculator.push( + float(times[row]), float(value), index=int(indices[row]), + ) + assert update.trends == tuple(expected[row]) + assert update.ready == tuple(readiness[row]) + event: np.void + for event in update.reversals: + pivot_row: int = int(np.searchsorted( + indices, event['index'], + )) + assert event['price'] == values[pivot_row] + assert event['time'] == times[pivot_row] + assert event['detected_time'] == times[row] + assert event['detected_index'] == indices[row] + assert pivot_row <= row + actual_events.append(( + pivot_row, int(event['direction']), + int(event['order']), row, + )) + assert actual_events == events + + +def test_original_lua_fixture() -> None: + ''' + Compare production events with actual, unmodified upstream Lua. + + The fixture uses increasing initial windows to establish a trend + before reversals, avoiding Lua's documented flat-start crash and + initial simple-reversal suppression. Every later pivot, sign and + earliest detection prefix must agree, including tied extrema. + Comparing complete ``TrendUpdate.reversals`` sequences against + captured Lua output checks an implementation independent of the + Python oracle. Fixture provenance records the exact program and + input hashes; this case does not claim parity at Lua's startup + failures. + + ''' + path: Path = ( + Path(__file__).parent + / '_inputs' + / 'trend_calculus_lua.json' + ) + fixture: _LuaFixture = json.loads(path.read_text()) + case: _LuaCase + for case in fixture['cases']: + calculator: TrendCalculator = TrendCalculator( + (case['window'],), + ) + events: list[list[int|float]] = [] + row: int + value: float + for row, value in enumerate(case['values']): + update: TrendUpdate = calculator.push(float(row), value) + event: np.void + for event in update.reversals: + events.append([ + int(event['index']), float(event['price']), + int(event['direction']), + int(event['detected_index']), + ]) + assert events == case['events'], case['window'] diff --git a/tests/test_trend_regimes.py b/tests/test_trend_regimes.py new file mode 100644 index 00000000..331c4656 --- /dev/null +++ b/tests/test_trend_regimes.py @@ -0,0 +1,312 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Closed-bar evidence contracts for multiple-timeframe classifiers. + +''' +from dataclasses import replace +import json + +import numpy as np +import pytest + +from piker.fsp._regime import ( + MultiTimeframeTrend, + RegimeSnapshot, + TimeframeRegime, + TrendReversal, +) +from piker.fsp._trend import TrendCalculator, TrendUpdate + + +def test_timeframes_match_independent_closed_bar_calculators( +) -> None: + ''' + Aggregating high/low extrema invents their intrabar order and can + change TC reversals. Feed a zigzag scalar close stream through + three periods and compare every prefix with separate calculators + receiving only each complete bucket's final close. Exact trend, + readiness, source-coordinate and detection-time agreement proves + wall-clock aggregation does not change the TC recurrence. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.test', periods=(1, 2, 5), windows=(2, 2), + ) + period: int + engines: dict[int, TrendCalculator] = { + period: TrendCalculator((2, 2)) for period in (1, 2, 5) + } + prices: np.ndarray = 100 + np.sin(np.arange(120) / 3) * 10 + row: int + price: np.float64 + for row, price in enumerate(prices): + snapshot: RegimeSnapshot = tracker.push( + float(row), float(price), index=1000 + row, + ) + frame: TimeframeRegime + for frame in snapshot.timeframes: + period: int = int(frame.period) + if (row + 1) % period: + continue + expected: TrendUpdate = engines[period].push( + float(row + 1), float(price), index=1000 + row, + ) + assert frame.trends == expected.trends + assert frame.ready == expected.ready + assert frame.as_of == row + 1 + assert frame.recent_closes[-1].index == 1000 + row + if len(expected.reversals): + actual: tuple[TrendReversal, ...] = ( + frame.reversals[-len(expected.reversals):] + ) + got: TrendReversal + event: np.void + for got, event in zip(actual, expected.reversals): + assert got.time == event['time'] + assert got.detected_time == ( + event['detected_time'] + ) + assert got.index == event['index'] + assert got.direction == event['direction'] + + +def test_partial_startup_bucket_is_never_an_observation() -> None: + ''' + Starting a feed mid-bucket must not give the first slow timeframe + a shorter initial observation. Start a five-second stream at t=2 + and verify its [0, 5) bucket is explicitly discarded while the + complete [5, 10) bucket yields its final close at t=10 only. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.test', periods=(5,), + ) + second: int + for second in range(2, 10): + snapshot: RegimeSnapshot = tracker.push(second, second + 1) + frame: TimeframeRegime = snapshot.timeframes[0] + if second < 9: + assert frame.close_count == 0 + assert frame.as_of is None + else: + assert frame.close_count == 1 + assert frame.as_of == 10 + assert frame.recent_closes[-1].price == 10 + assert frame.discarded_buckets == 1 + + +def test_gap_resets_every_timeframe_and_rejects_partial_bucket( +) -> None: + ''' + Carrying recurrence state across missing base bars fabricates + continuity and can falsely report a ready regime. Warm two + timeframes, omit t=9, and resume at t=10. The new source segment + must clear all prior evidence, and the interrupted four-second + bucket must not become a valid observation or restore readiness. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.test', periods=(1, 4), windows=(2,), + ) + second: int + for second in range(9): + tracker.push(second, second) + snapshot: RegimeSnapshot = tracker.push(10, 20) + assert snapshot.segment == 1 + assert snapshot.timeframes[0].close_count == 1 + assert snapshot.timeframes[0].ready == (False,) + assert snapshot.timeframes[1].close_count == 0 + assert not snapshot.timeframes[1].reversals + snapshot = tracker.push(11, 21) + assert snapshot.timeframes[1].close_count == 0 + for second in range(12, 16): + snapshot = tracker.push(second, second) + assert snapshot.timeframes[1].close_count == 1 + assert snapshot.timeframes[1].as_of == 16 + assert snapshot.timeframes[1].ready == (False,) + + +def test_subsecond_periods_preserve_epoch_alignment() -> None: + ''' + Integer-second resampling silently truncates high-frequency + periods, while float division can select the wrong epoch bucket. + Four quarter-second base bars at a modern Unix epoch must form + exactly one one-second close, with no early slow observation. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'synthetic.test', base_period=.25, periods=(.25, 1), + ) + offset: int + for offset in range(4): + snapshot: RegimeSnapshot = tracker.push( + 1_700_000_000 + offset / 4, + 100 + offset, + ) + assert snapshot.timeframes[1].close_count == (offset == 3) + assert snapshot.as_of == 1_700_000_001 + assert snapshot.timeframes[1].recent_closes[-1].price == 103 + + +def test_decimal_periods_accept_float_roundoff() -> None: + ''' + A calculated duration such as 0.1 * 3 differs from 0.3 by one + float rounding step. Rejecting that representation prevents a + valid integer-microsecond bucket configuration. Normalize only + float-rounding noise and verify three tenth-second samples form + one three-tenth-second observation at the expected endpoint. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', base_period=.1, periods=(.1, .1 * 3), + ) + tracker.push(0, 1) + tracker.push(.1, 2) + snapshot: RegimeSnapshot = tracker.push(.2, 3) + assert snapshot.timeframes[1].period == .3 + assert snapshot.timeframes[1].as_of == .3 + assert snapshot.timeframes[1].close_count == 1 + + +@pytest.mark.parametrize('kwargs', [ + {'base_period': 0}, + {'base_period': float('nan')}, + {'base_period': 0.0000001}, + {'base_period': 2, 'periods': (1,)}, + {'periods': (1, 1)}, + {'periods': ()}, + {'periods': (1.5,)}, + {'windows': (1,)}, + {'history_limit': 0}, + {'history_limit': 257}, + {'revision': -1}, +]) +def test_invalid_parameters_reject_ambiguous_sampling( + kwargs: dict, +) -> None: + ''' + A timeframe that cannot be built from whole base intervals has + no complete-bucket interpretation. Each invalid configuration + must fail before constructing a usable stream; bounded history + and valid recurrence windows are checked at the same boundary. + + ''' + with pytest.raises(ValueError): + MultiTimeframeTrend('synthetic.test', **kwargs) + + +@pytest.mark.parametrize('bar', [ + (0, 100, 1), + (-1, 100, 1), + (1.25, 100, 1), + (1, float('nan'), 1), + (1, float('inf'), 1), + (float('inf'), 100, 1), + (1, 100, 0), + (1, 100, 2 ** 63), + (1, 100, np.uint64(2 ** 63)), + (1, 100, np.bool_(True)), + (1e308, 100, 1), +]) +def test_invalid_bar_leaves_all_timeframes_unchanged( + bar: tuple[float, float, int|np.integer|np.bool_], +) -> None: + ''' + Partial mutation before rejecting a bad bar causes different + timeframes to consume different prefixes. Compare a tracker + subjected to one invalid update with an untouched reference; + accepting the same next bar must produce identical snapshots. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend('test.market') + reference: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', + ) + tracker.push(0, 100) + reference.push(0, 100) + with pytest.raises(ValueError): + tracker.push(bar[0], bar[1], index=bar[2]) + assert tracker.push(1, 101) == reference.push(1, 101) + + +def test_revision_replay_rebuilds_causal_evidence() -> None: + ''' + Revised history cannot be prepended to a warm TC recurrence. + Replaying the identical complete stream after reset must retain + every observation and reversal while changing its revision; the + frozen snapshot from the earlier generation must stay unchanged. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 2), windows=(2, 2), + ) + prices: list[int] = [1, 2, 3, 4, 3, 2, 1, 0] * 5 + second: int + price: int + for second, price in enumerate(prices): + original: RegimeSnapshot = tracker.push(second, price) + original_state: str = json.dumps(original.to_state()) + tracker.reset() + for second, price in enumerate(prices): + replayed: RegimeSnapshot = tracker.push(second, price) + assert replayed == replace(original, revision=1) + assert json.dumps(original.to_state()) == original_state + frame: TimeframeRegime + assert any(frame.reversals for frame in replayed.timeframes) + for frame in replayed.timeframes: + reversal: TrendReversal + for reversal in frame.reversals: + assert reversal.time <= reversal.detected_time + assert reversal.detected_time <= frame.as_of + with pytest.raises(ValueError): + tracker.reset(revision=1) + + +def test_snapshot_history_is_bounded_and_finite_json() -> None: + ''' + Sending unbounded market history makes inference cost grow with + uptime. A long oscillating stream must retain at most the chosen + close/reversal tail per timeframe while preserving total count. + Strict JSON encoding also proves no NaN/Infinity enters state. + Accepted NumPy integer windows must normalize to native integers + so configuration metadata cannot break that same JSON contract. + + ''' + tracker: MultiTimeframeTrend = MultiTimeframeTrend( + 'test.market', periods=(1, 2), history_limit=3, + windows=(np.int64(2), np.int64(2), np.int64(2)), + ) + second: int + for second in range(200): + snapshot: RegimeSnapshot = tracker.push( + second, + float(second % 7), + ) + frame: TimeframeRegime + for frame in snapshot.timeframes: + assert len(frame.recent_closes) == 3 + assert len(frame.reversals) <= 3 + assert frame.close_count == 200 / frame.period + decoded: dict = json.loads(json.dumps( + snapshot.to_state(), allow_nan=False, + )) + assert decoded['timestamp_convention'] == 'interval_end' + assert decoded['algorithm'] == 'trend_calculus' + assert decoded['windows'] == [2, 2, 2] diff --git a/tests/test_trend_sampling.py b/tests/test_trend_sampling.py new file mode 100644 index 00000000..810e794e --- /dev/null +++ b/tests/test_trend_sampling.py @@ -0,0 +1,149 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Native sampler timestamp ordering required by causal processors. + +''' +from collections import defaultdict +from collections.abc import Iterator +from types import SimpleNamespace +from unittest.mock import patch + +import numpy as np +import trio +from trio.testing import MockClock + +from piker.data._sampling import Sampler +from piker.fsp._trend import TrendCalculator, TrendHistory + + +class SampleSource: + ''' + Provide the actual sampler's OHLC and append protocol in memory. + + ''' + def __init__(self) -> None: + ''' + Seed one closed row and the sampler's future open tail. + + ''' + self._array: np.ndarray = np.zeros(16, dtype=[ + ('index', ' np.ndarray: + ''' + Expose only the published prefix, including its open tail. + + ''' + return self._array[:self.last] + + def push(self, values: np.ndarray) -> None: + ''' + Append OHLC fields while retaining absolute source indices. + + ''' + end: int = self.last + len(values) + self._array[self._write_fields][self.last:end] = values + self.last = end + + +def test_sampler_waits_for_strictly_new_epoch() -> None: + ''' + Empty realtime feeds seed times round(now) and round(now)+1. + Their first sampler wake can have the same epoch as that future + tail; rounded wall time can also repeat or move backward later. + Appending those epochs makes strict causal FSP bootstrap fail. + Drive the real sampler through all cases with a controlled clock: + keep broadcasting wakeups, append only newer epochs, and verify + the complete output remains valid Trend Calculus input. + + ''' + async def main() -> None: + ''' + Advance virtual time without starting brokers or actors. + + ''' + source: SampleSource = SampleSource() + epochs: tuple[int, ...] = ( + 1001, 1001, 1000, 1002, 1002, 1003, + ) + clock_values: Iterator[int] = iter(epochs) + counts: list[int] = [] + complete: trio.Event = trio.Event() + + class ControlledSampler(Sampler): + ''' + Isolate sampler state and capture every step broadcast. + + ''' + ohlcv_shms: dict = {1: [source]} + subscribers: defaultdict = defaultdict( + lambda: [0, set()], + ) + + @classmethod + async def broadcast(cls, **kwargs: object) -> None: + ''' + Stop after all scheduled clock cases are observed. + + ''' + counts.append(len(source.array)) + if len(counts) == len(epochs): + complete.set() + await trio.sleep_forever() + + fake_time: SimpleNamespace = SimpleNamespace( + time=lambda: next(clock_values), + ) + with patch('piker.data._sampling.time', fake_time): + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + scope: trio.CancelScope = await nursery.start( + ControlledSampler.increment_ohlc_buffer, 1., + ) + await complete.wait() + scope.cancel() + + assert counts == [2, 2, 2, 3, 3, 4] + assert source.array['time'].tolist() == [ + 1000, 1001, 1002, 1003, + ] + result: TrendHistory = TrendCalculator().extend( + source.array['close'], + source.array['time'], + source.array['index'], + ) + assert len(result.trends) == 4 + + trio.run(main, clock=MockClock(autojump_threshold=0)) diff --git a/tests/test_trend_shm.py b/tests/test_trend_shm.py new file mode 100644 index 00000000..8f27c158 --- /dev/null +++ b/tests/test_trend_shm.py @@ -0,0 +1,279 @@ +# piker: trading gear for hackers +# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers) +# SPDX-License-Identifier: AGPL-3.0-or-later + +# This program is free software: you can redistribute it and/or +# modify it under the terms of the GNU Affero General Public License +# as published by the Free Software Foundation, either version 3 +# of the License, or (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public +# License along with this program. If not, see +# . + +''' +Exercise native Trend Calculus publication through real Tractor SHM. + +''' +from collections.abc import AsyncIterator, Callable +from functools import partial +from types import SimpleNamespace +from typing import Any, cast +from uuid import uuid4 + +import numpy as np +import tractor +from tractor.ipc._shm import ( + NDToken, + ShmArray, + attach_shm_ndarray, + open_shm_ndarray, +) +import trio +from trio.testing import wait_all_tasks_blocked + +from piker.fsp._engine import ( + Cascade, + QuoteFrame, + _next_step_output, + _sample_wakeups, + connect_streams, +) +from piker.fsp._trend import TrendCalculator +from piker.fsp._trend_fsp import _history_fields, trend_calculus + + +async def _read_in_child( + token: dict[str, Any], +) -> dict[str, Any]: + ''' + Attach from an independent actor using the normal token codec. + + ''' + shm: ShmArray = attach_shm_ndarray(NDToken.from_msg(token)) + rows: np.ndarray = shm.array.copy() + fields: dict[str, list] = {} + name: str + for name in rows.dtype.names: + value: int|float # noqa: F842 + fields[name] = [ + value if np.isfinite(value) else None + for value in rows[name].tolist() + ] + return { + 'first': shm._first.value, + 'last': shm._last.value, + 'readonly': not shm.array.flags.writeable, + 'fields': fields, + } + + +def test_real_shm_bootstrap_step_and_in_place_replay() -> None: + ''' + Validate actual SHM bounds, field layout and replay visibility. + + Bootstrap the real engine against a nonzero source origin and + read its destination through a separate readonly attachment. + Advance a quiet bar through the sampler publication protocol, + then repair an interior close without changing source bounds. + ``Cascade.resync()`` must rewrite every affected output and + notify its + consumer. A child actor then attaches using the transported token + and verifies the repaired data independently. All actors and + memory names belong to this test; no broker, model API or + existing Piker daemon is required. The child reads after replay + completes; this proves visibility, not concurrent atomicity. + + ''' + async def main() -> None: + ''' + Own actor lifetimes and SHM names for this replay proof. + + ''' + suffix: str = uuid4().hex + key: str = f'test_tc_{suffix}' + async with tractor.open_root_actor( + name=key, + tpt_bind_addrs=[('127.0.0.1', 0)], + ): + source: ShmArray = open_shm_ndarray( + size=128, + key=f'{key}_source', + append_start_index=37, + dtype=np.dtype([ + ('index', ' None: + ''' + Retain each engine notice for deterministic + assertions. + + ''' + notices.append(message) + + def check_publication() -> None: + ''' + Compare the readonly attachment with causal source + replay. + + ''' + expected: dict[str, np.ndarray] = _history_fields( + TrendCalculator().extend( + source.array['close'], source.array['time'], + source.array['index'], + ), + ) + assert reader._first.value == source._first.value + assert reader._last.value == source._last.value + assert not reader.array.flags.writeable + field: str + for field in ('index', 'time'): + np.testing.assert_array_equal( + reader.array[field], source.array[field], + ) + field: str + values: np.ndarray + for field, values in expected.items(): + np.testing.assert_array_equal( + reader.array[field], values, + ) + + send: trio.MemorySendChannel[QuoteFrame] + quotes: trio.MemoryReceiveChannel[QuoteFrame] + send, quotes = trio.open_memory_channel[QuoteFrame](0) + market: SimpleNamespace = SimpleNamespace( + fqme='synthetic.test', + ) + wakeups: AsyncIterator[QuoteFrame] + notify: Callable[[], None]|None + async with send, quotes, _sample_wakeups( + quotes, market.fqme, True, + ) as (wakeups, notify): + nursery: trio.Nursery + async with trio.open_nursery() as nursery: + casc: Cascade = Cascade( + src=SimpleNamespace( + rt_shm=source, + mkt=market, + ), + dst=SimpleNamespace(rt_shm=destination), + tn=nursery, + fsp=trend_calculus, + client_stream=cast( + tractor.MsgStream, Sink(), + ), + ) + compute: Callable = partial( + connect_streams, + casc=casc, + mkt=market, + quote_stream=wakeups, + src=casc.src, + dst=casc.dst, + edge_func=trend_calculus.func, + ) + async with casc.open_edge(compute): + await wait_all_tasks_blocked() + check_publication() + assert reader.array['top_1'][-1] == 4. + assert reader.array['bottom_1'][-1] == 2. + + step: np.ndarray = source.array[-1:].copy() + step['time'] += 1 + source.push(step) + out: np.ndarray = _next_step_output( + destination.array[-1:], + trend_calculus, + False, + ) + out['time'] = step['time'] + destination.push(out) + if casc.sample_ready is not None: + casc.sample_ready.set() + casc.sample_ready = trio.Event() + assert notify is not None + notify() + await wait_all_tasks_blocked() + check_publication() + + source.array['close'][2] = 100. + await casc.resync() + await wait_all_tasks_blocked() + check_publication() + assert len(notices) == 1 + info: dict = notices[0]['fsp_update'] + assert info['first'] == 37 + assert info['last'] == 44 + + actors: tractor.ActorNursery + async with tractor.open_nursery() as actors: + portal: tractor.Portal = ( + await actors.start_actor( + 'trend-shm-reader', + enable_modules=[__name__], + ) + ) + try: + token: dict[str, Any] = ( + destination._token.as_msg() + ) + remote: dict[str, Any] = ( + await portal.run( + _read_in_child, + token=token, + ) + ) + assert remote['first'] == 37 + assert remote['last'] == 44 + assert remote['readonly'] + fields: tuple[str, ...] = ( + reader.array.dtype.names + ) + field: str + for field in fields: + np.testing.assert_array_equal( + np.asarray( + remote['fields'][field], + dtype=float, + ), + reader.array[field], + ) + finally: + await actors.cancel() + assert casc.complete.is_set() + + trio.run(main)