Save draft Trend Calculus and Jev prototype
Save the native FHLS recurrence, causal FSP publication, aligned multi-timeframe evidence, Trio/HTTPX Jev worker and Qt pivot/regime overlays as one draft snapshot. Retain tests, research sources, Lua fixtures, runnable examples, full license notices and prompt provenance. The latest reviewed offline suite passed 253 tests with one unrelated actor case excluded and one existing Tractor warning. Defer behavioral commit splitting and upstream history integration. Live Jev quality, provider-service wiring and Pine parity remain follow-ups in Taken. Prompt-IO: ai/prompt-io/codex/20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.md Prompt-IO: ai/prompt-io/codex/20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.md Prompt-IO: ai/prompt-io/codex/20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.md (this patch was generated in some part by `codex` using GPT-6 family models (`openai`); exact model IDs unavailable)trend_calc_on_jev
parent
eed5856622
commit
e815fd77cf
|
|
@ -0,0 +1,23 @@
|
||||||
|
# AI Prompt I/O Log
|
||||||
|
|
||||||
|
This directory tracks prompt inputs and model outputs for
|
||||||
|
AI-assisted development across harnesses.
|
||||||
|
|
||||||
|
## Policy
|
||||||
|
|
||||||
|
Prompt logging follows the
|
||||||
|
[NLNet generative AI policy][nlnet-ai]. Substantive contributions
|
||||||
|
record the original harness, provider/model when known, timestamps,
|
||||||
|
the prompting sequence, and an unedited `.raw.md` partner.
|
||||||
|
|
||||||
|
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Entries are created by the `prompt-io` skill or its commit-message
|
||||||
|
integration. Harness-specific records live in their corresponding
|
||||||
|
subdirectories. Code records can reference the associated Git diff
|
||||||
|
instead of duplicating source code.
|
||||||
|
|
||||||
|
Human contributors retain responsibility for review and acceptance.
|
||||||
|
AI-generated content is not presented as human-authored work.
|
||||||
|
|
@ -0,0 +1,118 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T04:11:32.732262+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261005T041132Z_eed58566_trend_b4fe4d_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The user requested an autonomous search of GitHub issues and local
|
||||||
|
notes, code, documentation and comments for Trend Calculus research,
|
||||||
|
then a rigorous Piker-native implementation in the FSP subsystem.
|
||||||
|
They asked to prepare multiple high-frequency timeframe regimes for
|
||||||
|
the JEV System-1 provider service, with work bounded by 00:18 Eastern.
|
||||||
|
They authorized same-caliber subagents and requested, as a bonus,
|
||||||
|
multiple-timeframe reversal markers and live rectangle annotations.
|
||||||
|
|
||||||
|
The user supplied these implementation resources:
|
||||||
|
|
||||||
|
- https://docs.typesafe.ai/models
|
||||||
|
- https://docs.typesafe.ai/sdk/python
|
||||||
|
- https://docs.typesafe.ai/agent-skill
|
||||||
|
|
||||||
|
They offered a Pine implementation but had not supplied it. They
|
||||||
|
said the API key was still being set up and instructed us to continue
|
||||||
|
offline. They clarified the workspace concern, disabled fast mode,
|
||||||
|
and supplied repository rules: preserve human-owned task states,
|
||||||
|
no autonomous commits or pushes, 69-column prose, Xonsh snippets,
|
||||||
|
and the Python style skill.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Implemented a bounded Numba recurrence and a four-order native FSP
|
||||||
|
with provisional-tail isolation, skipped-step recovery, indexed
|
||||||
|
publication, quiet sampler wakeups, sparse events and history replay.
|
||||||
|
Added complete aligned multiple-timeframe aggregation, immutable
|
||||||
|
provenance-rich snapshots, bounded HTTPX/Trio JEV requests and a
|
||||||
|
paced worker that rejects stale or invalidated results. Added native
|
||||||
|
feed/service wiring and opt-in Qt markers and regime rectangles,
|
||||||
|
including actual 1/5/15-second layers and model receipt/expiry rules.
|
||||||
|
|
||||||
|
Research identifies pinned Lua/Scala sources, issue discussions,
|
||||||
|
local notes and the manuscript, with explicit semantic differences.
|
||||||
|
Original Lua fixtures and an independent materialized-window oracle
|
||||||
|
check 21,870 exhaustive histories, 48 random recursive streams and
|
||||||
|
62 original Lua reversals. No complete Pine source was recovered.
|
||||||
|
|
||||||
|
The TypeSafe agent skill was installed and applied. No live API key
|
||||||
|
was read, no model request was sent, no broker session was opened,
|
||||||
|
and no Git history or human-owned task state was changed.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
The combined targeted suite passed 252 tests, with one unrelated
|
||||||
|
remote gap-dialog actor case deselected. The one warning is an
|
||||||
|
existing Tractor dependency's deprecated Trio Event truth test.
|
||||||
|
The new SHM test includes an actual separate read-only child actor
|
||||||
|
after in-place replay. This does not prove concurrent cross-process
|
||||||
|
write atomicity. Ruff, diff whitespace and 69-column checks passed.
|
||||||
|
|
||||||
|
The combined suite covered test_trend, test_trend_reference,
|
||||||
|
test_trend_shm, test_trend_regimes, test_jev, test_regime_feed,
|
||||||
|
test_trend_graphics, test_trend_mtf_graphics, test_fsp_sync,
|
||||||
|
test_fsp_momo and test_gap_overlays. The standard py313 environment
|
||||||
|
and pytest with the xonsh plugin disabled were used. Socket/SHM
|
||||||
|
execution required approved sandbox escalation.
|
||||||
|
|
||||||
|
The synthetic offscreen chart preview was generated and inspected.
|
||||||
|
The reproducible benchmark measured roughly 8.586 million batch
|
||||||
|
rows/second and 6.963 microseconds median scalar update on this host;
|
||||||
|
these exclude rendering, transport and provider inference. Native
|
||||||
|
subsecond sampling remains unsupported by the existing clock.
|
||||||
|
Live provider quality and the user's Pine parity remain unverified.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/fsp/__init__.py`
|
||||||
|
- `piker/fsp/_api.py`
|
||||||
|
- `piker/fsp/_engine.py`
|
||||||
|
- `piker/ui/_display.py`
|
||||||
|
- `piker/ui/_fsp.py`
|
||||||
|
- `piker/fsp/_trend.py`
|
||||||
|
- `piker/fsp/_trend_fsp.py`
|
||||||
|
- `piker/fsp/_regime.py`
|
||||||
|
- `piker/fsp/_jev.py`
|
||||||
|
- `piker/fsp/_regime_feed.py`
|
||||||
|
- `piker/ui/_trend.py`
|
||||||
|
- `piker/ui/_trend_mtf.py`
|
||||||
|
- `tests/test_trend.py`
|
||||||
|
- `tests/test_trend_shm.py`
|
||||||
|
- `tests/test_trend_reference.py`
|
||||||
|
- `tests/test_trend_regimes.py`
|
||||||
|
- `tests/test_jev.py`
|
||||||
|
- `tests/test_regime_feed.py`
|
||||||
|
- `tests/test_trend_graphics.py`
|
||||||
|
- `tests/test_trend_mtf_graphics.py`
|
||||||
|
- `tests/_inputs/trend_calculus_lua.json`
|
||||||
|
- `examples/benchmark_trend.py`
|
||||||
|
- `examples/trend_regimes.py`
|
||||||
|
- `examples/trend_graphics.py`
|
||||||
|
- `docs/trend_calculus.rst`
|
||||||
|
- `ai/prompt-io/README.md`
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human supplied the objective, deadline, research resources,
|
||||||
|
provider choice, optional skill installation, graphics extension,
|
||||||
|
parallel-agent authorization and offline credential constraint.
|
||||||
|
Those directions materially determined the implementation and scope.
|
||||||
|
No direct manual source edits were observed during this work. The
|
||||||
|
patch is pending human review; acceptance is not inferred. Research,
|
||||||
|
implementation, review and tests were performed by the root Codex
|
||||||
|
agent with three same-caliber delegated agents. Existing unrelated
|
||||||
|
working-tree artifacts were preserved.
|
||||||
|
|
@ -0,0 +1,106 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T04:11:32.732262+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
diff_cmd: git diff eed58566 --
|
||||||
|
---
|
||||||
|
|
||||||
|
# Captured output and code references
|
||||||
|
|
||||||
|
This record captures selected unedited progress responses and the
|
||||||
|
code artifact pointers before human source edits. It is not a full
|
||||||
|
transcript. Exact model and session identifiers were unavailable.
|
||||||
|
The working branch is trend_calc_on_jev; no commit was authorized.
|
||||||
|
While files remain untracked, use the no-index references below.
|
||||||
|
After inclusion in branch history, use the base-to-branch diff.
|
||||||
|
|
||||||
|
## Unedited progress excerpts
|
||||||
|
|
||||||
|
The cross-process shared-memory test passes, including replay
|
||||||
|
after an in-place history repair. The JEV feed also preserves
|
||||||
|
the shared quote policy and catches up cleanly when new bars
|
||||||
|
arrive during a history copy.
|
||||||
|
|
||||||
|
I’m running the combined checks and reviewing the chart preview.
|
||||||
|
Live JEV verification remains pending your key setup.
|
||||||
|
|
||||||
|
The combined offline suite passes: 252 tests, including the
|
||||||
|
independent algorithm oracle, JEV mock transport, Qt overlays,
|
||||||
|
existing FSP checks, and the cross-process SHM test.
|
||||||
|
|
||||||
|
The chart preview shows separate 1-, 5-, and 15-second reversal
|
||||||
|
markers and regime shading. I’m checking the final diff and
|
||||||
|
recording provenance; the work remains uncommitted.
|
||||||
|
|
||||||
|
## Unedited verification output
|
||||||
|
|
||||||
|
```text
|
||||||
|
252 passed, 1 deselected, 1 warning in 3.79s
|
||||||
|
All checks passed!
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code artifact pointers
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/fsp/__init__.py`
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/fsp/_api.py`
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/fsp/_engine.py`
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/ui/_display.py`
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/ui/_fsp.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/fsp/_trend.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/fsp/_trend_fsp.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/fsp/_regime.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/fsp/_jev.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/fsp/_regime_feed.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/ui/_trend.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null piker/ui/_trend_mtf.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_shm.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_reference.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_regimes.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_jev.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_regime_feed.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_graphics.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_mtf_graphics.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/_inputs/trend_calculus_lua.json`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null examples/benchmark_trend.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null examples/trend_regimes.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null examples/trend_graphics.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null docs/trend_calculus.rst`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null ai/prompt-io/README.md`
|
||||||
|
|
||||||
|
The implementation adds the bounded recursive TrendCalculator,
|
||||||
|
causal history/preview FSP publication, explicit indexed FspSample
|
||||||
|
updates, coalesced quiet-step wakeups and sparse-event clearing.
|
||||||
|
MultiTimeframeTrend forms complete aligned buckets; the Jev worker
|
||||||
|
uses bounded evidence, single-flight requests and freshness checks.
|
||||||
|
Qt layers render confirmed pivots and bounded regime rectangles.
|
||||||
|
Independent reference, mock transport and shared-memory tests cover
|
||||||
|
mathematical and integration behavior. The docs preserve sources,
|
||||||
|
semantic differences, measured performance and remaining limits.
|
||||||
|
|
@ -0,0 +1,42 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T05:02:18.111444+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261005T050218Z_eed58566_trend_sampler_5d9102_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Continuation of the user's autonomous, deadline-bounded request for
|
||||||
|
rigorous native Trend Calculus, live regime service preparation and
|
||||||
|
graphics integration. This supplements the earlier trend record.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Final delegated integration review reproduced duplicate timestamps
|
||||||
|
in the existing native sampler: seeded [1000,1001] became
|
||||||
|
[1000,1001,1001], causing the causal calculator to reject the source.
|
||||||
|
Added a narrow guard for native periods at or below one second and
|
||||||
|
a controlled-clock regression covering repeated, backward and newer
|
||||||
|
epochs. Wakeup broadcasts remain unchanged. No refactor or native
|
||||||
|
subsecond support is claimed. The root reviewed the change and ran
|
||||||
|
the complete targeted suite: 253 passed, one unrelated remote gap
|
||||||
|
case deselected, one existing Tractor/Trio deprecation warning.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/data/_sampling.py`
|
||||||
|
- `tests/test_trend_sampling.py`
|
||||||
|
- `docs/trend_calculus.rst`
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The original human requirement for reliable native integration
|
||||||
|
motivated this correction. No additional direct human source edit
|
||||||
|
was observed. The root and delegated Codex agent implemented and
|
||||||
|
reviewed it under that scope. Human review and acceptance remain
|
||||||
|
pending; no commits, broker sessions or model requests were made.
|
||||||
|
|
@ -0,0 +1,38 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T05:02:18.111444+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
diff_cmd: git diff eed58566 --
|
||||||
|
---
|
||||||
|
|
||||||
|
# Final integration correction
|
||||||
|
|
||||||
|
## Unedited progress output
|
||||||
|
|
||||||
|
The sampler fix is verified. The final combined suite passes
|
||||||
|
253 tests, including repeated and backward clock cases.
|
||||||
|
Implementation and tests are now frozen; I’m saving the review
|
||||||
|
patch and final provenance record.
|
||||||
|
|
||||||
|
## Unedited verification result
|
||||||
|
|
||||||
|
```text
|
||||||
|
253 passed, 1 deselected, 1 warning in 3.86s
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code pointers
|
||||||
|
|
||||||
|
> `git diff eed58566 -- piker/data/_sampling.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null tests/test_trend_sampling.py`
|
||||||
|
|
||||||
|
> `git diff --no-index /dev/null docs/trend_calculus.rst`
|
||||||
|
|
||||||
|
The actual Sampler.increment_ohlc_buffer could append a duplicate
|
||||||
|
rounded epoch after realtime startup, or when the clock repeated
|
||||||
|
or moved backward. The fix retains the existing mutable bar until
|
||||||
|
a newer epoch exists, preserving broadcasts. The regression drives
|
||||||
|
the real sampler with Trio MockClock and verifies that its complete
|
||||||
|
output remains valid input for the strict causal calculator.
|
||||||
|
|
@ -0,0 +1,79 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T05:40:14.294712+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20261005T054014Z_eed58566_trend_style_e3216f_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
The human explicitly requested a complete py-codestyle pass and
|
||||||
|
proper license headers for every new code module after identifying
|
||||||
|
that the previous pass was incomplete. They retained autonomous
|
||||||
|
work and parallel-agent authorization, offline provider constraints,
|
||||||
|
and human ownership of task states and commits.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Read the repository-deployed py-codestyle skill and shared runtime
|
||||||
|
review lessons. Root and three agents audited all changed Python
|
||||||
|
modules, tests and runnable examples. Added complete project AGPL
|
||||||
|
notices and SPDX identifiers to all 19 new modules, preserving the
|
||||||
|
Morgan/ByteSumo algorithm attribution. Corrected annotations,
|
||||||
|
helper contracts, expression layout, named interpolation operands,
|
||||||
|
and owner-qualified regression documentation. Used narrow F842
|
||||||
|
suppressions where Ruff cannot connect declarations to isolated
|
||||||
|
comprehension scopes.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Ruff passes across all changed Python modules. All new source and
|
||||||
|
added old-file lines meet the 69-column rule with no whitespace-only
|
||||||
|
lines. Every new function and class has the required multiline
|
||||||
|
contract; function parameters and returns are annotated. The final
|
||||||
|
offline suite reports 253 passed, one unrelated actor case
|
||||||
|
excluded, and one existing Tractor deprecation warning. Native SHM
|
||||||
|
publication, cross-process replay visibility, quiet sampler steps,
|
||||||
|
Numba reference parity, Qt overlays and mocked Jev protocols pass.
|
||||||
|
No live model quality, separate provider-service wiring, Pine parity
|
||||||
|
or cross-process write atomicity is claimed.
|
||||||
|
|
||||||
|
## Human edits and follow-ups
|
||||||
|
|
||||||
|
The human's style correction materially drove these changes.
|
||||||
|
No direct manual source edits were observed; acceptance is pending.
|
||||||
|
The human deferred SDK AnyIO/Trio work until the HTTP prototype is
|
||||||
|
validated and requested a later Taken-store handoff. Existing task
|
||||||
|
states, unrelated artifacts, Git index and Git history are preserved.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `piker/fsp/__init__.py`
|
||||||
|
- `piker/fsp/_api.py`
|
||||||
|
- `piker/fsp/_engine.py`
|
||||||
|
- `piker/ui/_display.py`
|
||||||
|
- `piker/ui/_fsp.py`
|
||||||
|
- `piker/data/_sampling.py`
|
||||||
|
- `piker/fsp/_trend.py`
|
||||||
|
- `piker/fsp/_trend_fsp.py`
|
||||||
|
- `piker/fsp/_regime.py`
|
||||||
|
- `piker/fsp/_jev.py`
|
||||||
|
- `piker/fsp/_regime_feed.py`
|
||||||
|
- `piker/ui/_trend.py`
|
||||||
|
- `piker/ui/_trend_mtf.py`
|
||||||
|
- `tests/test_trend.py`
|
||||||
|
- `tests/test_trend_shm.py`
|
||||||
|
- `tests/test_trend_reference.py`
|
||||||
|
- `tests/test_trend_regimes.py`
|
||||||
|
- `tests/test_jev.py`
|
||||||
|
- `tests/test_regime_feed.py`
|
||||||
|
- `tests/test_trend_graphics.py`
|
||||||
|
- `tests/test_trend_mtf_graphics.py`
|
||||||
|
- `tests/test_trend_sampling.py`
|
||||||
|
- `examples/benchmark_trend.py`
|
||||||
|
- `examples/trend_regimes.py`
|
||||||
|
- `examples/trend_graphics.py`
|
||||||
|
|
@ -0,0 +1,98 @@
|
||||||
|
---
|
||||||
|
created_at: 2026-10-05T05:40:14.294712+00:00
|
||||||
|
generated_by:
|
||||||
|
harness: codex
|
||||||
|
provider: OpenAI
|
||||||
|
git_ref: eed58566
|
||||||
|
scope: code
|
||||||
|
substantive: true
|
||||||
|
---
|
||||||
|
|
||||||
|
## Human instruction (verbatim)
|
||||||
|
|
||||||
|
ok since i'm seeing this alot i want to make sure you did a proper /py-codestyle pass and have proper license headers throughout all new code-modules.
|
||||||
|
|
||||||
|
## Captured assistant output (verbatim)
|
||||||
|
|
||||||
|
The earlier pass wasn’t complete: the tests and examples lacked
|
||||||
|
license notices, and several modules had only abbreviated ones.
|
||||||
|
All 19 new Python files now have full AGPL notices and SPDX
|
||||||
|
identifiers, with the algorithm’s source attribution preserved.
|
||||||
|
|
||||||
|
I’m checking the remaining style rules too—annotations,
|
||||||
|
docstrings, expression layout, and regression explanations.
|
||||||
|
|
||||||
|
## Final regression output (verbatim)
|
||||||
|
|
||||||
|
```text
|
||||||
|
253 passed, 1 deselected, 1 warning in 7.93s
|
||||||
|
```
|
||||||
|
|
||||||
|
The warning is the existing Tractor dependency's Trio Event
|
||||||
|
truth-test deprecation; the unrelated remote gap-dialog actor test
|
||||||
|
is excluded. The first sandboxed attempt stopped on Trio socket
|
||||||
|
permissions and was rerun with approved escalation. No broker or
|
||||||
|
Jev API request was made.
|
||||||
|
|
||||||
|
## Code pointers
|
||||||
|
|
||||||
|
The associated boundary diffs capture full source changes. The
|
||||||
|
whole-file commit package includes untracked modules, which a bare
|
||||||
|
working-tree git diff does not include. Human editing and committing
|
||||||
|
remain pending; these pointers become valid after the complete
|
||||||
|
package is committed.
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/__init__.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_api.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_engine.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_display.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_fsp.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/data/_sampling.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_trend_fsp.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_jev.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/fsp/_regime_feed.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- piker/ui/_trend_mtf.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_shm.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_reference.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_regimes.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_jev.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_regime_feed.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_graphics.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_mtf_graphics.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- tests/test_trend_sampling.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/benchmark_trend.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_regimes.py`
|
||||||
|
|
||||||
|
> `git diff trend_calc_on_jev~7..trend_calc_on_jev -- examples/trend_graphics.py`
|
||||||
|
|
||||||
|
The style pass added full license headers, annotations, multiline
|
||||||
|
helper docstrings, explicit expression layout and owner-qualified
|
||||||
|
regression proof descriptions. All recurrence behavior and source
|
||||||
|
credits are preserved; callback constructor defaults are explicit.
|
||||||
|
|
@ -0,0 +1,645 @@
|
||||||
|
Trend Calculus: sources and Piker semantics
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
This implementation follows Andrew J. Morgan's FHLS Trend Calculus.
|
||||||
|
It classifies observed price structure; it does not establish
|
||||||
|
predictive accuracy or a profitable trading rule. Research was
|
||||||
|
collected on 2026-10-04/05. The source inventory below records what
|
||||||
|
was found and distinguishes missing material from working code.
|
||||||
|
|
||||||
|
Algorithm
|
||||||
|
---------
|
||||||
|
|
||||||
|
Partition an ordered scalar series into disjoint windows containing
|
||||||
|
``n >= 2`` observations. Summarize each window by its high and low,
|
||||||
|
retaining each extremum's original coordinate. First and second in
|
||||||
|
FHLS mean the earlier and later of these two extrema. They do not
|
||||||
|
mean the opening and closing observation.
|
||||||
|
|
||||||
|
For consecutive windows ``previous`` and ``current``, define::
|
||||||
|
|
||||||
|
trend = sign(
|
||||||
|
sign(current.high - previous.high)
|
||||||
|
+ sign(current.low - previous.low)
|
||||||
|
)
|
||||||
|
|
||||||
|
The implementation uses ordinal comparisons to avoid overflowing
|
||||||
|
the differences. When the result is zero, introduce a synthetic
|
||||||
|
window joining the previous window's second extremum to the current
|
||||||
|
window's first extremum. Compare previous to synthetic, then
|
||||||
|
synthetic to current. Carry the prior direction through unresolved
|
||||||
|
zero comparisons. A complex window can confirm two opposite
|
||||||
|
reversals at the same observation; retaining only its final trend
|
||||||
|
would lose an event.
|
||||||
|
|
||||||
|
A confirmed down-to-up change selects the preceding summary's low;
|
||||||
|
an up-to-down change selects its high. Apply the same calculation to
|
||||||
|
the resulting ordered reversal points for the next reversal order.
|
||||||
|
The order counts successive reductions of the pivot stream. It is
|
||||||
|
independent of a wall-clock timeframe such as one or five seconds.
|
||||||
|
|
||||||
|
The live recurrence retains one partial window, one completed
|
||||||
|
window and the prior direction per order. Its numeric state is
|
||||||
|
``O(orders)`` regardless of history length or window size. Historical
|
||||||
|
scanning calls the same Numba recurrence. Returning a full historical
|
||||||
|
result necessarily allocates arrays proportional to history length
|
||||||
|
and order count.
|
||||||
|
|
||||||
|
A completed window emits at most two reversals. Because each configured
|
||||||
|
window contains at least two observations, two incoming reversals can
|
||||||
|
complete at most one next-order window during that source update.
|
||||||
|
Induction bounds every order to two events per source observation;
|
||||||
|
the fixed event slots therefore do not truncate a larger cascade.
|
||||||
|
|
||||||
|
Deliberate semantic choices
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
``piker.fsp._trend.TrendCalculator`` accepts scalar observations,
|
||||||
|
finite strictly increasing timestamps, and strictly increasing
|
||||||
|
nonnegative source indices. Coordinates need not be contiguous.
|
||||||
|
An OHLC candle alone cannot reveal whether its high or low happened
|
||||||
|
first; feeding fabricated high/low order would change the algorithm.
|
||||||
|
The chart FSP therefore uses close prices.
|
||||||
|
|
||||||
|
Production startup is neutral. The first completed window establishes
|
||||||
|
the baseline without inventing a pivot. ``ready`` becomes true after
|
||||||
|
two completed windows at that order, including when the direction
|
||||||
|
remains zero on flat data. Thus zero is not a calibrated "sideways"
|
||||||
|
regime probability. Readiness and direction are separate information.
|
||||||
|
Higher orders warm up only when enough lower-order pivots exist.
|
||||||
|
|
||||||
|
Equal highs and lows in ordinary FHLS windows retain their earliest
|
||||||
|
observation, matching the original Lua implementation. Synthetic
|
||||||
|
bridges have a narrower, distinct rule: when their endpoint values
|
||||||
|
are equal, high belongs to the second endpoint and low to the first.
|
||||||
|
That distinction affects pivot coordinates even when trends agree.
|
||||||
|
|
||||||
|
An optional ``lua_seed=True`` reproduces the reference's initial
|
||||||
|
comparison with the first window's last observation. It is a seed
|
||||||
|
policy for reference investigation, not a promise to reproduce all
|
||||||
|
Lua command-line startup quirks. Normal FSPs use neutral startup.
|
||||||
|
|
||||||
|
NaN clears all recurrence orders and requires fresh warmup. Infinity
|
||||||
|
and ambiguous coordinate ordering are rejected before state changes.
|
||||||
|
The scalar calculator does not guess session boundaries from elapsed
|
||||||
|
time: its caller decides whether a gap is intentional or a reset.
|
||||||
|
Prepending or revising source history requires replay from the new
|
||||||
|
origin; changing a prefix can affect every subsequent reversal.
|
||||||
|
|
||||||
|
Causality and output
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
``TrendCalculator.push()`` returns all newly detected reversals.
|
||||||
|
Each record carries both coordinates:
|
||||||
|
|
||||||
|
* ``index``, ``time``, ``price`` locate the original pivot;
|
||||||
|
* ``detected_index``, ``detected_time`` locate the observation that
|
||||||
|
first confirmed it;
|
||||||
|
* ``direction`` is the new trend: ``+1`` for a bottom, ``-1`` for a
|
||||||
|
top; ``order`` starts at one.
|
||||||
|
|
||||||
|
A higher-order pivot retains its original price/time while detection
|
||||||
|
time propagates from the input observation that completed its
|
||||||
|
higher-order window. Strategies must use detection time, not the
|
||||||
|
earlier pivot time, when deciding what information was available.
|
||||||
|
|
||||||
|
``TrendCalculator.extend()`` returns per-observation ``trends`` and
|
||||||
|
``ready`` arrays. ``pulses`` stores only the last detected direction
|
||||||
|
per order and row. ``pivot_prices`` and ``pivot_times`` retain both
|
||||||
|
possible sides: axis two is top then bottom, with NaN for no event.
|
||||||
|
Use complete push events when the event stream itself is required.
|
||||||
|
|
||||||
|
The ``trend_calculus`` FSP exposes four orders through ``trend``,
|
||||||
|
``trend_2``, ``trend_3`` and ``trend_4``. Its ``ready`` field counts
|
||||||
|
ready orders. Each order also exposes ``top_N``, ``top_time_N``,
|
||||||
|
``bottom_N`` and ``bottom_time_N``. Pivot prices are published at
|
||||||
|
their detection row, with the paired field retaining pivot time.
|
||||||
|
Its ``window`` argument changes the first-order close window only:
|
||||||
|
the four sizes are ``(window, 2, 2, 2)``, defaulting to all twos.
|
||||||
|
The numerical and timeframe APIs accept explicit per-order tuples.
|
||||||
|
|
||||||
|
Piker's source SHM timestamps name candle starts. Historical rows
|
||||||
|
preceding the last row are finalized; the last row is provisional.
|
||||||
|
Quotes and source sampler steps preview that row from a copy of
|
||||||
|
committed recurrence state. These wakeups inspect authoritative source
|
||||||
|
SHM; they are not additional scalar observations. Each newly finalized
|
||||||
|
row advances committed state once, including when several steps arrive
|
||||||
|
before a wakeup. Sparse pivot fields clear when the destination steps;
|
||||||
|
source revisions require a cascade replay.
|
||||||
|
|
||||||
|
For realtime regime consumers, use finalized closes with an explicit
|
||||||
|
interval-end clock. ``piker.fsp._regime.MultiTimeframeTrend`` forms
|
||||||
|
epoch-aligned buckets from complete base bars, discards partial
|
||||||
|
startup buckets and resets recurrence after missing base bars. Its
|
||||||
|
snapshots distinguish timeframe period, recursive order, source
|
||||||
|
revision, sequence, gap segment and availability time. An old model
|
||||||
|
response must not overwrite newer source evidence.
|
||||||
|
|
||||||
|
The scalar calculator and current SHM ``trend_calculus`` FSP count
|
||||||
|
observations without inferring resets from elapsed timestamp gaps;
|
||||||
|
NaN explicitly resets their recurrence. ``MultiTimeframeTrend`` instead
|
||||||
|
requires contiguous base bars and starts a new segment after a missing
|
||||||
|
interval. Consequently, the simple one-second FSP and the one-second
|
||||||
|
timeframe tracker can differ across session gaps. That difference is
|
||||||
|
an ingestion policy, not a different FHLS recurrence.
|
||||||
|
|
||||||
|
Using the numerical and timeframe APIs
|
||||||
|
--------------------------------------
|
||||||
|
|
||||||
|
The numerical API has no provider, GUI or credential requirement.
|
||||||
|
This complete example confirms both sides of a complex window:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from piker.fsp._trend import TrendCalculator
|
||||||
|
|
||||||
|
calculator = TrendCalculator((2, 2, 2))
|
||||||
|
prices: list[float] = [1., 2., 4., 3., 2., 5.]
|
||||||
|
row: int
|
||||||
|
price: float
|
||||||
|
for row, price in enumerate(prices):
|
||||||
|
update = calculator.push(
|
||||||
|
1_700_000_000. + row,
|
||||||
|
price,
|
||||||
|
index=100 + row,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(update.trends)
|
||||||
|
print(update.reversals[
|
||||||
|
['index', 'direction', 'detected_index']
|
||||||
|
])
|
||||||
|
|
||||||
|
The last update has a top at source index 102 and a bottom at 104,
|
||||||
|
both detected at 105. A consumer only learns these events when it
|
||||||
|
receives that update. For batch analysis, pass aligned one-dimensional
|
||||||
|
``values``, ``times`` and optional signed integer ``indices`` arrays
|
||||||
|
to ``calculator.extend()``. Use a fresh calculator for an independent
|
||||||
|
historical replay; ``extend()`` advances an existing instance.
|
||||||
|
|
||||||
|
For wall-clock timeframes, submit each completed base bar once, using
|
||||||
|
its opening timestamp. The tracker converts this to availability at
|
||||||
|
the interval end:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from piker.fsp._regime import MultiTimeframeTrend
|
||||||
|
|
||||||
|
tracker = MultiTimeframeTrend(
|
||||||
|
'synthetic.test',
|
||||||
|
base_period=1.,
|
||||||
|
periods=(1., 5.),
|
||||||
|
windows=(2, 2, 2),
|
||||||
|
)
|
||||||
|
prices: list[float] = [100., 101., 99., 103., 102.] * 4
|
||||||
|
row: int
|
||||||
|
price: float
|
||||||
|
for row, price in enumerate(prices):
|
||||||
|
snapshot = tracker.push(
|
||||||
|
1_700_000_000. + row,
|
||||||
|
price,
|
||||||
|
index=100 + row,
|
||||||
|
)
|
||||||
|
|
||||||
|
state: dict = snapshot.to_state()
|
||||||
|
|
||||||
|
These periods describe independent close series; ``windows`` still
|
||||||
|
counts observations and reversal points. Subsecond periods must be
|
||||||
|
whole microseconds, and every configured period must be a whole
|
||||||
|
multiple of ``base_period``. Startup midway through a larger bucket
|
||||||
|
does not produce a shortened first observation. On history repair,
|
||||||
|
call ``tracker.reset()`` and replay the corrected completed bars.
|
||||||
|
|
||||||
|
Source inventory
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Project research and earlier implementation work:
|
||||||
|
|
||||||
|
* `Piker #79`_ is the older trend-detection issue; `Piker #103`_ is
|
||||||
|
the consolidated research entrypoint. It links Morgan's paper,
|
||||||
|
Lua, Scala, a working mathematical manuscript, and a public
|
||||||
|
TradingView/Pine implementation.
|
||||||
|
* `Piker #187`_ explicitly proposes Trend Calculus in the native FSP
|
||||||
|
set and links the planned Python implementation.
|
||||||
|
* `Piker #325`_ connects trend extrema computation with faster chart
|
||||||
|
y-range reductions. Its moving min/max proposal concerns
|
||||||
|
performance; it is not an alternative definition of FHLS.
|
||||||
|
* `Python implementation issue #1`_ requests offline/online support
|
||||||
|
and recurrence updates across timeframes. The ``setup`` branch
|
||||||
|
inspected at ``30afcfbd8cbb7b84d984fa2e183c6ae392da3479`` has a
|
||||||
|
placeholder ``tc`` module and Ray moving-average experiments.
|
||||||
|
Those experiments do not implement FHLS reversal detection.
|
||||||
|
|
||||||
|
Primary algorithm sources:
|
||||||
|
|
||||||
|
* `Morgan's method document`_, ``HowToStudyTrends_v1.03.pdf``:
|
||||||
|
the original conceptual account. Inspected PDF SHA-256::
|
||||||
|
|
||||||
|
2dc506a0449c13c01d2d798f120411df84e5667b611d92204ba0122e2aac7fd5
|
||||||
|
|
||||||
|
* `Original Lua program`_, ByteSumoLtd/TrendCalculus-lua commit
|
||||||
|
``7f54f9a101283cc188dbcd9282292fc3f73de8e9``,
|
||||||
|
``make/trendcalculus.lua``. Its header credits Andrew J. Morgan /
|
||||||
|
ByteSumo Limited and declares GPL version 3 or later. Inspected
|
||||||
|
source SHA-256::
|
||||||
|
|
||||||
|
c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0
|
||||||
|
|
||||||
|
* `Spark TrendCalculus2`_, commit
|
||||||
|
``3fb6ef23e985d9d6e0414d3b682afbeef51c52e8``. The source declares
|
||||||
|
Apache-2.0. It uses earliest equal high and latest equal low, and
|
||||||
|
initializes from a zero dummy point or an excluded first point.
|
||||||
|
Those choices can change reversals and recursive warmup relative
|
||||||
|
to Lua and Piker. Inspected source SHA-256::
|
||||||
|
|
||||||
|
3d0723836d230c89d99537f1075537c3bbf809cebefd12caa6a1b517e7bc98b5
|
||||||
|
|
||||||
|
* `Working mathematical manuscript`_, ``main.tex``: describes
|
||||||
|
earliest equal low and latest equal high, a fixed initial positive
|
||||||
|
trend and an initial synthetic reversal. These are different from
|
||||||
|
both the Scala code's tie choices and native neutral startup.
|
||||||
|
Its displayed reversal-sign expression also disagrees with the
|
||||||
|
following low/high selection branches. It is a working draft,
|
||||||
|
not a unique executable specification; the executable references
|
||||||
|
and independently checked cases determine the conventions here.
|
||||||
|
Inspected source SHA-256::
|
||||||
|
|
||||||
|
e7b6fdff06fbde195a81281022cce300135d88e7484a97d1b02d621c5bfc4eab
|
||||||
|
|
||||||
|
* `Spark examples`_ provide historical visualization context.
|
||||||
|
The `public Pine indicator`_ linked from Piker #103 describes two
|
||||||
|
lagged Donchian channels and their midpoints for visual analysis.
|
||||||
|
That description does not establish recursive FHLS parity. No
|
||||||
|
complete Pine source or the user's separate Pine implementation
|
||||||
|
was recovered in this search, so no Pine parity claim is made.
|
||||||
|
|
||||||
|
Local research locations inspected, relative to the user's repos
|
||||||
|
directory:
|
||||||
|
|
||||||
|
* ``lns/todo/piker/quant/trend.toml`` links the Python ``setup``
|
||||||
|
branch, authors, Piker #187 and the Ray SMA example.
|
||||||
|
* ``lns/the_legacy_tab_nightmare/trading.toml`` links the working
|
||||||
|
manuscript, Spark comparisons/PRs, examples, and a former local
|
||||||
|
``TrendCalculus/notes_redesign_amorgan.pdf``. The latter file was
|
||||||
|
not present at its recorded path.
|
||||||
|
* ``data/bookmarks/bookmarks.html`` preserves the older ByteSumo
|
||||||
|
Bitbucket project URL. Duplicate bookmarks are not independent
|
||||||
|
algorithm evidence.
|
||||||
|
|
||||||
|
A separate filename inventory across ``/home/goodboy/repos`` included
|
||||||
|
hidden and ignored files while excluding Git internals, worktrees,
|
||||||
|
virtual environments, dependency directories, caches and build outputs.
|
||||||
|
It found no ``*.pine`` or ``*.pinescript`` files, TrendCalculus-named
|
||||||
|
source files, or copy of ``notes_redesign_amorgan.pdf`` beyond this
|
||||||
|
change's new documentation and fixture.
|
||||||
|
|
||||||
|
No earlier native Trend Calculus implementation was found in the
|
||||||
|
searched Piker source, tests, docs, snippets or available research
|
||||||
|
notes. This is a bounded search result, not proof that no private
|
||||||
|
copy or unindexed history exists. User-owned research/task states
|
||||||
|
were left untouched.
|
||||||
|
|
||||||
|
Verification and reproducibility
|
||||||
|
-------------------------------
|
||||||
|
|
||||||
|
``tests/test_trend_reference.py`` uses a separate batch oracle that
|
||||||
|
materializes complete FHLS windows, inserts synthetic windows and
|
||||||
|
then reduces the expanded sequence. It never calls the production
|
||||||
|
kernel or copies its mutable recurrence state. Coverage includes:
|
||||||
|
|
||||||
|
* every eight-point sequence over three prices, for three window
|
||||||
|
tuples: 19,683 histories with ties, flat startup and inside/outside
|
||||||
|
windows; direction, readiness, pulses and both pivot sides agree;
|
||||||
|
* another 2,187 exhaustive six-point histories using the optional Lua
|
||||||
|
seed, while keeping readiness tied to two actual complete windows;
|
||||||
|
* 48 seeded 512-point random walks across both startup policies,
|
||||||
|
five recursive orders and mixed window sizes, checking every event's
|
||||||
|
source identity and detection coordinates on irregular timestamps
|
||||||
|
and source indices;
|
||||||
|
* actual unmodified Lua output for windows 2, 3, 4 and 7, covering
|
||||||
|
62 reversals, including original pivot locations and earliest
|
||||||
|
detection prefixes.
|
||||||
|
|
||||||
|
The original-Lua fixture is
|
||||||
|
``tests/_inputs/trend_calculus_lua.json``. It records source/runtime
|
||||||
|
provenance, the complete synthetic inputs, event schema and SHA-256
|
||||||
|
hashes of each input and full output CSV. To reproduce an output,
|
||||||
|
write the fixture values with header ``ticker,date,last`` and rows
|
||||||
|
``x,<zero-based-row>,<value>``, using LF line endings. Run the pinned
|
||||||
|
program on Lua 5.2.4 with ``-n<window> -r -H -p1``. For detection
|
||||||
|
times, rerun each complete-window prefix and assign newly emitted
|
||||||
|
events to that prefix's last input row. Prefix outputs must preserve
|
||||||
|
all earlier emitted rows. No Lua executable or network is needed
|
||||||
|
for these Python reference tests.
|
||||||
|
|
||||||
|
The pinned Lua CLI has observable quirks:
|
||||||
|
|
||||||
|
* Do not pass ``-h`` with its built-in column mapping. That path
|
||||||
|
consumes the first real observation as a second header, shifting
|
||||||
|
every window boundary. ``-H`` controls only output headers.
|
||||||
|
* A flat first complete window can fail while reading a nonexistent
|
||||||
|
prior trend during bridge processing.
|
||||||
|
* The simple-window reversal branch suppresses initial output until
|
||||||
|
``offset > 2*n + 1``; split-window branches use other guards.
|
||||||
|
|
||||||
|
The fixture begins with increasing complete windows to establish a
|
||||||
|
common baseline before reversals. It establishes exact parity for
|
||||||
|
those inputs, not universal startup equivalence between programs.
|
||||||
|
The native oracle separately covers the flat and complex startup
|
||||||
|
cases excluded by that reference limitation.
|
||||||
|
|
||||||
|
Integration verification is separate from mathematical parity:
|
||||||
|
|
||||||
|
* ``tests/test_trend.py`` exercises committed/preview handoff,
|
||||||
|
quote-free sampler wakeups, indexed publication and cancellation.
|
||||||
|
* ``tests/test_trend_sampling.py`` exercises the actual sampler with
|
||||||
|
a controlled clock. Repeated or backward rounded epochs must not
|
||||||
|
append duplicate timestamps, including after realtime bootstrap.
|
||||||
|
* ``tests/test_trend_shm.py`` runs actual Tractor shared memory with a
|
||||||
|
separate read-only attachment, nonzero origin, quiet sampling and
|
||||||
|
an equal-bound historical repair. A separate child actor then
|
||||||
|
attaches read-only using the transported token and checks every
|
||||||
|
repaired field. This verifies cross-process visibility after replay,
|
||||||
|
not atomicity during concurrent writes. It needs actor runtime/socket
|
||||||
|
access, but no broker or model service.
|
||||||
|
* ``tests/test_trend_regimes.py`` checks complete aligned buckets,
|
||||||
|
fractional clocks, missing intervals and revision replay;
|
||||||
|
``tests/test_regime_feed.py`` checks native source admission,
|
||||||
|
subscription ordering, quote receiver ownership and lifecycle.
|
||||||
|
* ``tests/test_jev.py`` uses mocked transport and controlled clocks
|
||||||
|
for response validation, bounded projection, pacing, retry, source
|
||||||
|
invalidation and cancellation while publication is blocked.
|
||||||
|
* ``tests/test_trend_graphics.py`` and
|
||||||
|
``tests/test_trend_mtf_graphics.py`` exercise Qt geometry, causal
|
||||||
|
timestamp mapping, revision invalidation, model expiry and teardown.
|
||||||
|
|
||||||
|
Performance measurement
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The repository example measures compilation/warmup, batch computation,
|
||||||
|
individual committed observations and copied-state previews separately.
|
||||||
|
Run it in the existing Piker Python environment:
|
||||||
|
|
||||||
|
.. code-block:: xsh
|
||||||
|
|
||||||
|
py313/bin/python examples/benchmark_trend.py --rows 1000000 --live-rows 20000
|
||||||
|
|
||||||
|
One x86-64 run on 2026-10-05 used Python 3.13.14, NumPy 2.3.5 and
|
||||||
|
Numba 0.62.1 with four reversal orders:
|
||||||
|
|
||||||
|
=========================== ============================
|
||||||
|
Measurement Observed result
|
||||||
|
=========================== ============================
|
||||||
|
One million-row batch 0.116473 seconds
|
||||||
|
Batch throughput 8.586 million rows/second
|
||||||
|
Scalar commit, median/p99 6.963 / 10.670 microseconds
|
||||||
|
Copy and preview, median/p99 12.223 / 18.835 microseconds
|
||||||
|
Initial kernel warmup 3.004 seconds
|
||||||
|
Retained NumPy state 936 bytes
|
||||||
|
=========================== ============================
|
||||||
|
|
||||||
|
These are reproducible measurements from one machine and workload,
|
||||||
|
not latency guarantees. Retained-state bytes exclude Python objects,
|
||||||
|
compiler caches and returned history arrays. Timings exclude source
|
||||||
|
SHM copies, actor IPC, chart rendering and provider inference. The
|
||||||
|
native sampler currently timestamps periods at or below one second
|
||||||
|
with integer seconds; subsecond end-to-end operation requires a
|
||||||
|
producer with an appropriate event clock, despite the numerical and
|
||||||
|
timeframe APIs supporting fractional-second observations.
|
||||||
|
|
||||||
|
TypeSafe/Jev integration
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The user supplied the `TypeSafe model documentation`_,
|
||||||
|
`Python SDK documentation`_ and `agent skill`_. The adapter uses
|
||||||
|
Piker's existing HTTPX/Trio stack against the documented `System One
|
||||||
|
HTTP API`_; installing the separate Python SDK is unnecessary. The
|
||||||
|
default model is pinned to ``jev-1.13.0``. Importing the adapter does
|
||||||
|
not read credentials or open a connection.
|
||||||
|
|
||||||
|
``build_regime_request(snapshot)`` produces one Choice question for
|
||||||
|
each ready timeframe, plus an independent alignment question when
|
||||||
|
multiple timeframes are ready. Candidate regimes are ``uptrend``,
|
||||||
|
``downtrend``, ``range``, ``transition`` and ``unknown``. Questions
|
||||||
|
explicitly identify state paths because question identifiers do not
|
||||||
|
reach the model. ``JevClassification`` retains every candidate's
|
||||||
|
probability, the selected label, the service's separate confidence
|
||||||
|
value, model/prompt versions, source provenance and receipt time.
|
||||||
|
These semantic judgments do not change the numerical recurrence.
|
||||||
|
Model input uses the most recent 16 closes and 16 reversals per
|
||||||
|
timeframe by default. ``evidence_limit`` configures that projection
|
||||||
|
independently of longer chart history; source snapshots stay unchanged.
|
||||||
|
|
||||||
|
Inspect a complete synthetic request, then exercise HTTP parsing with
|
||||||
|
explicitly simulated responses:
|
||||||
|
|
||||||
|
.. code-block:: xsh
|
||||||
|
|
||||||
|
py313/bin/python examples/trend_regimes.py
|
||||||
|
py313/bin/python examples/trend_regimes.py --mock
|
||||||
|
|
||||||
|
The synthetic example includes quarter-second evidence. Its ``--live``
|
||||||
|
mode sends exactly one synthetic request and reads ``TYPESAFE_API_KEY``
|
||||||
|
from the caller's environment. It consumes provider credit and makes no
|
||||||
|
broker connection. A real provider request has not been verified without
|
||||||
|
an API key; mocked results test the integration, not model quality.
|
||||||
|
|
||||||
|
The native live bridge subscribes to the sampler before copying history,
|
||||||
|
excludes the mutable final source row, and emits only the newest snapshot
|
||||||
|
after each captured history range. It catches up skipped sample steps
|
||||||
|
and invalidates old judgments before replaying a prepend, truncation,
|
||||||
|
backfill or lost sampler notice. ``run_regime_feed()`` supplies these
|
||||||
|
snapshots to a synchronous callback without requiring a model client.
|
||||||
|
Use ``serve_regimes()`` to compose that reader and a Jev worker:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from piker.fsp._jev import JevClassification, JevClient
|
||||||
|
from piker.fsp._regime_feed import serve_regimes
|
||||||
|
from piker.service import open_piker_runtime
|
||||||
|
|
||||||
|
async def run_live(
|
||||||
|
fqme: str,
|
||||||
|
api_key: str,
|
||||||
|
publish: Callable[[JevClassification], Awaitable[None]],
|
||||||
|
) -> None:
|
||||||
|
async with (
|
||||||
|
open_piker_runtime('trend-regimes'),
|
||||||
|
httpx.AsyncClient() as http,
|
||||||
|
):
|
||||||
|
await serve_regimes(
|
||||||
|
fqme,
|
||||||
|
JevClient(http, api_key),
|
||||||
|
publish,
|
||||||
|
periods=(1., 5., 15.),
|
||||||
|
min_interval=1.,
|
||||||
|
max_age=5.,
|
||||||
|
max_source_lag=2.,
|
||||||
|
max_source_age=5.,
|
||||||
|
)
|
||||||
|
|
||||||
|
Calling this function opens the configured broker feed and sends live
|
||||||
|
inference requests. Supply an existing Piker market identifier and a
|
||||||
|
cancellation-cooperative async consumer. The example periods assume
|
||||||
|
a one-second source; omit ``periods`` to derive 1x/5x/15x periods from
|
||||||
|
the actual source. Native subsecond source periods are rejected until
|
||||||
|
the sampler has a compatible clock. A separate correctly timestamped
|
||||||
|
producer can use ``consume_regime_source()`` directly.
|
||||||
|
|
||||||
|
The worker permits one in-flight request and retains only the newest
|
||||||
|
pending snapshot. ``min_interval`` limits request start frequency;
|
||||||
|
``max_age`` limits local submission age, ``max_source_lag`` limits
|
||||||
|
event-time lag, and ``max_source_age`` filters stale wall-clock source
|
||||||
|
data before submission. Setting the last option to ``None`` admits
|
||||||
|
historical replay. Source revision or gap-segment changes revoke old
|
||||||
|
results. A blocked publication is also cancelled when its evidence
|
||||||
|
expires or becomes stale. A downstream consumer must still check
|
||||||
|
provenance on already-delivered messages, which cannot be revoked.
|
||||||
|
|
||||||
|
Current source SHM has no transactional revision word. Copy-time bound
|
||||||
|
checks detect appends and origin changes; an in-place correction with
|
||||||
|
unchanged bounds becomes identifiable through its backfill notice.
|
||||||
|
Replay converges to that corrected source. A publication without a
|
||||||
|
local task checkpoint is not an atomic cross-process memory transaction.
|
||||||
|
|
||||||
|
Malformed/authentication responses fail explicitly. Transient failures
|
||||||
|
use bounded retries and freshness checks; the adapter bounds HTTP body
|
||||||
|
sizes and timeouts. The owning nursery closes the feed, sampler and
|
||||||
|
in-flight worker on cancellation. Neither the bridge nor the classifier
|
||||||
|
submits trades. Regime rubrics and probability usefulness still need
|
||||||
|
evaluation on recorded market data and real provider responses.
|
||||||
|
|
||||||
|
Chart overlays
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Preview synthetic multiple-timeframe annotations without a broker,
|
||||||
|
model request or credentials:
|
||||||
|
|
||||||
|
.. code-block:: xsh
|
||||||
|
|
||||||
|
py313/bin/python examples/trend_graphics.py --output /tmp/trend.png
|
||||||
|
|
||||||
|
The offscreen example accepts ``--periods 1 5 15`` and ``--order 1``
|
||||||
|
and isolates its Qt/configuration state from the running application.
|
||||||
|
|
||||||
|
Enable the native order-one reversal and direction overlay in the
|
||||||
|
existing Piker configuration's ``[ui]`` section::
|
||||||
|
|
||||||
|
[ui]
|
||||||
|
trend_calculus = true
|
||||||
|
|
||||||
|
This starts one Trend Calculus writer for the chart's realtime source
|
||||||
|
and displays confirmed pivot arrows with translucent direction
|
||||||
|
rectangles. It makes no Jev request and does not read API credentials.
|
||||||
|
Hover text distinguishes the pivot's bar from the later time when
|
||||||
|
the reversal became observable. The mutable last FSP row is excluded
|
||||||
|
from confirmed markers and direction runs.
|
||||||
|
|
||||||
|
Within the existing chart owner's nursery,
|
||||||
|
``await FspAdmin.open_trend_overlay(order=2)`` attaches another
|
||||||
|
recursive order while reusing that admin's writer. Optional ``chart``
|
||||||
|
and ``chart_period_s`` arguments attach the source evidence to another
|
||||||
|
price chart. Changing the chart period does not recompute the source
|
||||||
|
at that period: its actual realtime sampling period stays explicit.
|
||||||
|
|
||||||
|
For independent wall-clock timeframes on the native realtime chart,
|
||||||
|
configure::
|
||||||
|
|
||||||
|
[ui]
|
||||||
|
trend_timeframes = [1, 5, 15]
|
||||||
|
|
||||||
|
This option takes precedence over ``trend_calculus = true``. It uses
|
||||||
|
one ``MultiTimeframeTrend`` reader and one ``SnapshotTrendOverlay``
|
||||||
|
with separate marker/rectangle layers for each period. It opens no Jev
|
||||||
|
client. ``await FspAdmin.open_multi_timeframe_trends(...)`` exposes the
|
||||||
|
same integration programmatically and accepts a recursive ``order``.
|
||||||
|
All periods must be multiples of the actual realtime source period.
|
||||||
|
|
||||||
|
Snapshots preserve interval-end pivot and detection times. Markers
|
||||||
|
attach to the base bar that contributed each aggregated close. A
|
||||||
|
bootstrap snapshot contains recent reversals and the latest direction,
|
||||||
|
so its deterministic rectangle history starts with that observed
|
||||||
|
direction. It does not reconstruct earlier direction runs from one
|
||||||
|
snapshot. Skipped complete buckets leave unobserved intervals unpainted.
|
||||||
|
Source revisions and missing-base-bar segments revoke prior annotations.
|
||||||
|
|
||||||
|
An existing caller-owned client can share this exact evidence with
|
||||||
|
the model worker inside the chart's nursery:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
import trio
|
||||||
|
|
||||||
|
from piker.fsp._jev import JevClient, RegimeClassifier
|
||||||
|
from piker.ui._fsp import FspAdmin
|
||||||
|
from piker.ui._trend_mtf import SnapshotTrendOverlay
|
||||||
|
|
||||||
|
async def attach_classifier(
|
||||||
|
admin: FspAdmin,
|
||||||
|
client: JevClient,
|
||||||
|
nursery: trio.Nursery,
|
||||||
|
) -> tuple[RegimeClassifier, SnapshotTrendOverlay]:
|
||||||
|
worker: RegimeClassifier = RegimeClassifier(client)
|
||||||
|
overlay = await admin.open_multi_timeframe_trends(
|
||||||
|
periods=(1., 5., 15.),
|
||||||
|
on_snapshot=worker.submit,
|
||||||
|
on_invalidate=worker.invalidate,
|
||||||
|
max_source_age=5.,
|
||||||
|
classification_ttl=5.,
|
||||||
|
)
|
||||||
|
nursery.start_soon(worker.run, overlay.publish_classification)
|
||||||
|
return worker, overlay
|
||||||
|
|
||||||
|
Graphics retain their longer deterministic history while the client
|
||||||
|
projects bounded model evidence. The source-age gate applies to the
|
||||||
|
optional model callback, so historical chart bootstrap still works.
|
||||||
|
The adapter checks market, revision, segment, sequence and source lag
|
||||||
|
before accepting a model response. Classified runs start at receipt
|
||||||
|
and expire after the configured TTL; they cannot paint future history
|
||||||
|
before the displayed event clock reaches receipt. Cancelling the chart
|
||||||
|
owner removes all layers and stops its sampler reader.
|
||||||
|
|
||||||
|
For computations that already publish separate timeframe SHMs, create
|
||||||
|
one ``piker.ui._trend.TrendOverlay`` per destination SHM with its own
|
||||||
|
``source_period_s``, target ``chart_period_s`` and ``order``. Mapping
|
||||||
|
uses timestamps and preserves gaps rather than snapping missing
|
||||||
|
observations onto the next candle. Geometry counts are bounded and
|
||||||
|
items are reused instead of adding individual Qt items on every tick.
|
||||||
|
|
||||||
|
Call ``update(force=True)`` after a matching destination or chart
|
||||||
|
history revision, and ``remove()`` when its owner exits. Normal
|
||||||
|
``FspAdmin`` integration routes destination-specific revisions and
|
||||||
|
teardown automatically; direct attachments must retain this ownership.
|
||||||
|
Hidden widgets preserve revision invalidations until their next
|
||||||
|
visible update. Repainting follows quotes and the existing one-second
|
||||||
|
sampler wakeup. If that wakeup precedes asynchronous FSP publication,
|
||||||
|
a quote-free finalization can appear on the next wakeup, roughly one
|
||||||
|
second later; the integration does not emit IPC for every output row.
|
||||||
|
|
||||||
|
For model annotations, ``classification_interval()`` converts a
|
||||||
|
matching ``JevClassification`` into a ``RegimeInterval``. Supply the
|
||||||
|
chart's current market, source revision, timeframe and display end.
|
||||||
|
The interval starts no earlier than the model response's receipt.
|
||||||
|
``overlay.set_regimes(intervals)`` displays an owner-maintained,
|
||||||
|
bounded collection of these intervals. An empty collection clears
|
||||||
|
model rectangles; ``None`` restores deterministic direction runs.
|
||||||
|
The caller remains responsible for response ordering and expiry.
|
||||||
|
|
||||||
|
.. _Piker #79: https://github.com/pikers/piker/issues/79
|
||||||
|
.. _Piker #103: https://github.com/pikers/piker/issues/103
|
||||||
|
.. _Piker #187: https://github.com/pikers/piker/issues/187
|
||||||
|
.. _Piker #325: https://github.com/pikers/piker/issues/325
|
||||||
|
.. _Python implementation issue #1: https://github.com/bytesumo/TrendCalculus/issues/1
|
||||||
|
.. _Morgan's method document: https://github.com/bytesumo/TrendCalculus/blob/3d5c598cd3764a40633979c4c6f4188191dd81ef/HowToStudyTrends_v1.03.pdf
|
||||||
|
.. _Original Lua program: https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua
|
||||||
|
.. _Spark TrendCalculus2: https://github.com/lamastex/spark-trend-calculus/blob/3fb6ef23e985d9d6e0414d3b682afbeef51c52e8/src/main/scala/org/lamastex/spark/trendcalculus/TrendCalculus2.scala
|
||||||
|
.. _Working mathematical manuscript: https://github.com/lamastex/working-manuscript-trend-calculus/blob/master/main.tex
|
||||||
|
.. _Spark examples: https://lamastex.github.io/spark-trend-calculus-examples/
|
||||||
|
.. _public Pine indicator: https://www.tradingview.com/script/ohSdmhS0-TrendCalculus/
|
||||||
|
.. _TypeSafe model documentation: https://docs.typesafe.ai/models
|
||||||
|
.. _Python SDK documentation: https://docs.typesafe.ai/sdk/python
|
||||||
|
.. _agent skill: https://docs.typesafe.ai/agent-skill
|
||||||
|
.. _System One HTTP API: https://docs.typesafe.ai/api
|
||||||
|
|
@ -0,0 +1,135 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Measure deterministic Trend Calculus processing without a feed.
|
||||||
|
|
||||||
|
Timings distinguish kernel warmup, batch scanning, scalar commits,
|
||||||
|
and uncommitted live previews. They exclude network/model latency.
|
||||||
|
Run from the Piker environment; no credentials or services are used.
|
||||||
|
|
||||||
|
'''
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import platform
|
||||||
|
from time import perf_counter
|
||||||
|
|
||||||
|
import numba
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from piker.fsp._trend import (
|
||||||
|
TrendCalculator,
|
||||||
|
TrendHistory,
|
||||||
|
TrendUpdate,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def benchmark(
|
||||||
|
rows: int,
|
||||||
|
live_rows: int,
|
||||||
|
) -> dict[str, str|int|float|dict[str, float]]:
|
||||||
|
'''
|
||||||
|
Return reproducible timings and the retained NumPy state size.
|
||||||
|
|
||||||
|
``rows`` controls the ``TrendCalculator.extend()`` batch size.
|
||||||
|
``live_rows`` controls timed calls to ``TrendCalculator.push()``
|
||||||
|
and previews made through ``TrendCalculator.copy()``. Assertions
|
||||||
|
compare directions across all three paths outside timed regions.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rng: np.random.Generator = np.random.default_rng(20261004)
|
||||||
|
values: np.ndarray = rng.normal(size=rows).cumsum() + 100.
|
||||||
|
times: np.ndarray = np.arange(rows, dtype=float) * .25
|
||||||
|
calc: TrendCalculator = TrendCalculator()
|
||||||
|
start: float = perf_counter()
|
||||||
|
calc.extend(values[:8], times[:8])
|
||||||
|
calc.push(times[8], values[8])
|
||||||
|
warmup: float = perf_counter() - start
|
||||||
|
|
||||||
|
start = perf_counter()
|
||||||
|
full: TrendHistory = TrendCalculator().extend(values, times)
|
||||||
|
batch: float = perf_counter() - start
|
||||||
|
calc = TrendCalculator()
|
||||||
|
elapsed: np.ndarray = np.empty(live_rows)
|
||||||
|
preview_elapsed: np.ndarray = np.empty(live_rows)
|
||||||
|
number: int
|
||||||
|
for number in range(live_rows):
|
||||||
|
start = perf_counter()
|
||||||
|
preview: TrendUpdate = calc.copy().push(
|
||||||
|
times[number], values[number],
|
||||||
|
)
|
||||||
|
preview_elapsed[number] = perf_counter() - start
|
||||||
|
start = perf_counter()
|
||||||
|
update: TrendUpdate = calc.push(
|
||||||
|
times[number], values[number],
|
||||||
|
)
|
||||||
|
elapsed[number] = perf_counter() - start
|
||||||
|
assert preview.trends == update.trends
|
||||||
|
assert update.trends == tuple(full.trends[number])
|
||||||
|
|
||||||
|
def percentiles_us(samples: np.ndarray) -> dict[str, float]:
|
||||||
|
'''
|
||||||
|
Report median and tail operation durations in microseconds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return dict(zip(
|
||||||
|
('p50', 'p95', 'p99'),
|
||||||
|
np.percentile(samples * 1_000_000, (50, 95, 99)),
|
||||||
|
))
|
||||||
|
|
||||||
|
array: np.ndarray # noqa: F842
|
||||||
|
return {
|
||||||
|
'python': platform.python_version(),
|
||||||
|
'machine': platform.machine(),
|
||||||
|
'numpy': np.__version__,
|
||||||
|
'numba': numba.__version__,
|
||||||
|
'rows': rows,
|
||||||
|
'live_rows': live_rows,
|
||||||
|
'orders': len(calc.windows),
|
||||||
|
'kernel_warmup_s': warmup,
|
||||||
|
'batch_s': batch,
|
||||||
|
'batch_rows_per_s': rows / batch,
|
||||||
|
'scalar_commit_us': percentiles_us(elapsed),
|
||||||
|
'preview_copy_and_push_us': percentiles_us(preview_elapsed),
|
||||||
|
'retained_numpy_state_bytes': sum(
|
||||||
|
array.nbytes for array in (
|
||||||
|
calc._sizes, calc._bars, calc._counters,
|
||||||
|
calc._events, calc._counts,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
'''
|
||||||
|
Validate CLI sizes and print benchmark measurements as JSON.
|
||||||
|
|
||||||
|
'''
|
||||||
|
parser: argparse.ArgumentParser = argparse.ArgumentParser(
|
||||||
|
description=__doc__,
|
||||||
|
)
|
||||||
|
parser.add_argument('--rows', type=int, default=1_000_000)
|
||||||
|
parser.add_argument('--live-rows', type=int, default=20_000)
|
||||||
|
args: argparse.Namespace = parser.parse_args()
|
||||||
|
if args.rows < 9 or not 1 <= args.live_rows <= args.rows:
|
||||||
|
parser.error('require rows >= 9 and 1 <= live-rows <= rows')
|
||||||
|
print(json.dumps(benchmark(args.rows, args.live_rows), indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,235 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Render an explicitly synthetic multiple-timeframe trend PNG.
|
||||||
|
|
||||||
|
Run with ``--output /tmp/piker-trend-graphics.png``. This uses the
|
||||||
|
actual shared tracker and snapshot graphics adapter. It launches
|
||||||
|
no visible GUI, opens no broker or Jev connection, reads no API key,
|
||||||
|
and isolates Qt/Piker configuration in a disposable directory.
|
||||||
|
|
||||||
|
'''
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from tempfile import TemporaryDirectory
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
|
||||||
|
def render(
|
||||||
|
output: Path,
|
||||||
|
*,
|
||||||
|
periods: tuple[float, ...],
|
||||||
|
order: int,
|
||||||
|
samples: int,
|
||||||
|
|
||||||
|
) -> dict[str, object]:
|
||||||
|
'''
|
||||||
|
Replay synthetic closes through one actual tracker and renderer.
|
||||||
|
|
||||||
|
The final OHLC row remains mutable and is never submitted as a
|
||||||
|
completed observation. Each timeframe retains real aggregation
|
||||||
|
and TC state, while all plotted marker timestamps identify their
|
||||||
|
final contributing base bar. Shading is deterministic TC output,
|
||||||
|
not fabricated model inference.
|
||||||
|
|
||||||
|
'''
|
||||||
|
import numpy as np
|
||||||
|
import pyqtgraph as pg
|
||||||
|
|
||||||
|
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
|
||||||
|
from piker.ui._trend_mtf import (
|
||||||
|
SnapshotTrendOverlay,
|
||||||
|
_SnapshotLayer,
|
||||||
|
)
|
||||||
|
from piker.ui.qt import QApplication
|
||||||
|
|
||||||
|
app: QApplication = QApplication(['piker-trend-graphics'])
|
||||||
|
app.setQuitOnLastWindowClosed(False)
|
||||||
|
steps: np.ndarray = np.arange(samples)
|
||||||
|
prices: np.ndarray = (
|
||||||
|
100 + .009 * steps
|
||||||
|
+ 1.5 * np.sin(steps * .06)
|
||||||
|
+ .38 * np.sin(steps * .43)
|
||||||
|
)
|
||||||
|
rows: np.ndarray = np.empty(samples, dtype=[
|
||||||
|
('index', 'i8'),
|
||||||
|
('time', 'f8'),
|
||||||
|
('close', 'f8'),
|
||||||
|
('low', 'f8'),
|
||||||
|
('high', 'f8'),
|
||||||
|
])
|
||||||
|
rows['index'] = 1000 + steps
|
||||||
|
rows['time'] = 1_700_000_100 + steps
|
||||||
|
rows['close'] = prices
|
||||||
|
rows['low'] = prices - .12
|
||||||
|
rows['high'] = prices + .12
|
||||||
|
shm: SimpleNamespace = SimpleNamespace(
|
||||||
|
array=rows,
|
||||||
|
_token='synthetic-demo-ohlcv',
|
||||||
|
)
|
||||||
|
|
||||||
|
widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
widget.resize(1400, 760)
|
||||||
|
widget.setBackground('#101923')
|
||||||
|
widget.setTitle(
|
||||||
|
'SYNTHETIC: Trend Calculus across actual timeframes'
|
||||||
|
'<br><span style="font-size:11pt;color:#aab7c5">'
|
||||||
|
'Larger triangles represent slower sampling periods; '
|
||||||
|
'shading begins at confirmed regime detection.'
|
||||||
|
'</span>',
|
||||||
|
color='#dce5ed',
|
||||||
|
size='16pt',
|
||||||
|
)
|
||||||
|
widget.plotItem.titleLabel.setFixedHeight(60)
|
||||||
|
widget.showGrid(x=True, y=True, alpha=.13)
|
||||||
|
widget.setLabel('left', 'Synthetic price')
|
||||||
|
widget.setLabel('bottom', 'Absolute base-bar index (1 second)')
|
||||||
|
widget.plot(
|
||||||
|
rows['index'],
|
||||||
|
prices,
|
||||||
|
pen=pg.mkPen('#dce5ed', width=1.5),
|
||||||
|
)
|
||||||
|
left: int = max(0, samples - 400)
|
||||||
|
widget.setXRange(1000 + left, 1000 + samples, padding=0)
|
||||||
|
widget.setYRange(
|
||||||
|
float(prices[left:].min()) - .6,
|
||||||
|
float(prices[left:].max()) + .6,
|
||||||
|
padding=0,
|
||||||
|
)
|
||||||
|
viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=widget.plotItem,
|
||||||
|
shm=shm,
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.demo',
|
||||||
|
periods=periods,
|
||||||
|
windows=(2, 2, 2, 2),
|
||||||
|
history_limit=256,
|
||||||
|
)
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
viz,
|
||||||
|
shm,
|
||||||
|
fqme='synthetic.demo',
|
||||||
|
base_period=1,
|
||||||
|
periods=periods,
|
||||||
|
order=order,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
row: np.void
|
||||||
|
snapshot: RegimeSnapshot
|
||||||
|
for row in rows[:-1]:
|
||||||
|
snapshot = tracker.push(
|
||||||
|
float(row['time']),
|
||||||
|
float(row['close']),
|
||||||
|
index=int(row['index']),
|
||||||
|
)
|
||||||
|
adapter.submit(snapshot)
|
||||||
|
|
||||||
|
legend: pg.LegendItem = widget.addLegend(offset=(12, 12))
|
||||||
|
period: float
|
||||||
|
layer: _SnapshotLayer
|
||||||
|
for period, layer in adapter.layers.items():
|
||||||
|
legend.addItem(
|
||||||
|
pg.ScatterPlotItem(
|
||||||
|
symbol='t1',
|
||||||
|
size=layer.markers.opts['size'],
|
||||||
|
brush='#46c7a4',
|
||||||
|
pen=pg.mkPen(None),
|
||||||
|
),
|
||||||
|
f'{period:g}s / order {order}',
|
||||||
|
)
|
||||||
|
widget.show()
|
||||||
|
app.processEvents()
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
if not widget.grab().save(str(output)):
|
||||||
|
raise RuntimeError('Qt could not save the requested PNG')
|
||||||
|
return {
|
||||||
|
'mode': 'synthetic_native_graphics_no_model_calls',
|
||||||
|
'output': str(output),
|
||||||
|
'base_period': 1,
|
||||||
|
'periods': periods,
|
||||||
|
'order': order,
|
||||||
|
'finalized_bars': samples - 1,
|
||||||
|
'last_event_end': snapshot.as_of,
|
||||||
|
'layers': {
|
||||||
|
period: {
|
||||||
|
'markers': len(layer.points),
|
||||||
|
'rectangles': len(layer.rectangles.intervals),
|
||||||
|
}
|
||||||
|
for period, layer in adapter.layers.items()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
widget.close()
|
||||||
|
widget.deleteLater()
|
||||||
|
app.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
'''
|
||||||
|
Require an output path and isolate offscreen configuration.
|
||||||
|
|
||||||
|
'''
|
||||||
|
parser: argparse.ArgumentParser = argparse.ArgumentParser(
|
||||||
|
description=__doc__,
|
||||||
|
)
|
||||||
|
parser.add_argument('--output', type=Path, required=True)
|
||||||
|
parser.add_argument(
|
||||||
|
'--periods', nargs='+', type=float, default=[1, 5, 15],
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
'--order', type=int, choices=range(1, 5), default=1,
|
||||||
|
)
|
||||||
|
parser.add_argument('--samples', type=int, default=600)
|
||||||
|
args: argparse.Namespace = parser.parse_args()
|
||||||
|
if not 120 <= args.samples <= 10_000:
|
||||||
|
parser.error('--samples must be between 120 and 10000')
|
||||||
|
|
||||||
|
prior: dict[str, str|None] = {
|
||||||
|
'QT_QPA_PLATFORM': os.environ.get('QT_QPA_PLATFORM'),
|
||||||
|
'XDG_CONFIG_HOME': os.environ.get('XDG_CONFIG_HOME'),
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
path: str
|
||||||
|
with TemporaryDirectory(prefix='piker-trend-') as path:
|
||||||
|
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
|
||||||
|
os.environ['XDG_CONFIG_HOME'] = path
|
||||||
|
result: dict[str, object] = render(
|
||||||
|
args.output.resolve(),
|
||||||
|
periods=tuple(args.periods),
|
||||||
|
order=args.order,
|
||||||
|
samples=args.samples,
|
||||||
|
)
|
||||||
|
print(json.dumps(result, indent=2))
|
||||||
|
finally:
|
||||||
|
key: str
|
||||||
|
value: str|None
|
||||||
|
for key, value in prior.items():
|
||||||
|
if value is None:
|
||||||
|
os.environ.pop(key, None)
|
||||||
|
else:
|
||||||
|
os.environ[key] = value
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
|
|
@ -0,0 +1,165 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Inspect synthetic timeframe evidence and optional Jev inference.
|
||||||
|
|
||||||
|
The default prints the exact request without reading credentials or
|
||||||
|
opening a network connection. ``--mock`` tests the HTTP path locally.
|
||||||
|
``--live`` sends exactly one synthetic snapshot using the explicitly
|
||||||
|
configured ``TYPESAFE_API_KEY``. No broker connection is necessary.
|
||||||
|
|
||||||
|
'''
|
||||||
|
import argparse
|
||||||
|
from dataclasses import asdict
|
||||||
|
import json
|
||||||
|
from math import floor, sin
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import trio
|
||||||
|
|
||||||
|
from piker.fsp._jev import (
|
||||||
|
DEFAULT_MODEL,
|
||||||
|
JevClassification,
|
||||||
|
JevClient,
|
||||||
|
build_regime_request,
|
||||||
|
)
|
||||||
|
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
|
||||||
|
|
||||||
|
|
||||||
|
def synthetic_snapshot() -> RegimeSnapshot:
|
||||||
|
'''
|
||||||
|
Aggregate quarter-second closes ending before the current time.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.demo',
|
||||||
|
base_period=.25,
|
||||||
|
periods=(.25, 1, 5),
|
||||||
|
windows=(2, 2, 2),
|
||||||
|
)
|
||||||
|
count: int = 960
|
||||||
|
start: float = floor(time.time()) - count * .25
|
||||||
|
number: int
|
||||||
|
for number in range(count):
|
||||||
|
price: float = (
|
||||||
|
100 + number * .01
|
||||||
|
+ 2 * sin(number / 12)
|
||||||
|
+ .6 * sin(number / 3)
|
||||||
|
)
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(
|
||||||
|
start + number * .25, price,
|
||||||
|
)
|
||||||
|
return snapshot
|
||||||
|
|
||||||
|
|
||||||
|
def mock_reply(request: httpx.Request) -> httpx.Response:
|
||||||
|
'''
|
||||||
|
Demonstrate transport and parsing with explicitly fake labels.
|
||||||
|
|
||||||
|
'''
|
||||||
|
payload: dict = json.loads(request.content)
|
||||||
|
answers: dict[str, dict] = {}
|
||||||
|
name: str
|
||||||
|
question: dict
|
||||||
|
for name, question in payload['questions'].items():
|
||||||
|
options: dict = question['criteria']
|
||||||
|
probabilities: dict[str, float] = {}
|
||||||
|
option: str
|
||||||
|
for option in options:
|
||||||
|
probabilities[option] = float(option == 'unknown')
|
||||||
|
answers[name] = {
|
||||||
|
'type': 'choice',
|
||||||
|
'choice': 'unknown',
|
||||||
|
'probabilities': probabilities,
|
||||||
|
'confidence': 1.0,
|
||||||
|
}
|
||||||
|
return httpx.Response(200, json={
|
||||||
|
'model': payload['model'],
|
||||||
|
'answers': answers,
|
||||||
|
'usage': {
|
||||||
|
'input_tokens': 0,
|
||||||
|
'output_tokens': 0,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
async def classify(
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
model: str,
|
||||||
|
live: bool,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Print one explicitly live or simulated result with provenance.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if live:
|
||||||
|
key: str = os.environ.get('TYPESAFE_API_KEY', '')
|
||||||
|
if not key:
|
||||||
|
raise SystemExit('Set TYPESAFE_API_KEY before --live')
|
||||||
|
transport: httpx.AsyncBaseTransport|None = None
|
||||||
|
else:
|
||||||
|
key = 'offline-demo-key'
|
||||||
|
transport = httpx.MockTransport(mock_reply)
|
||||||
|
http: httpx.AsyncClient
|
||||||
|
async with httpx.AsyncClient(transport=transport) as http:
|
||||||
|
client: JevClient = JevClient(http, key, model=model)
|
||||||
|
result: JevClassification = await client.classify(snapshot)
|
||||||
|
output: dict = {
|
||||||
|
'mode': 'live_synthetic' if live else 'mock_not_inference',
|
||||||
|
'classification': asdict(result),
|
||||||
|
}
|
||||||
|
print(json.dumps(output, indent=2, allow_nan=False))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
'''
|
||||||
|
Require an explicit flag before any credential or network access.
|
||||||
|
|
||||||
|
'''
|
||||||
|
parser: argparse.ArgumentParser = argparse.ArgumentParser(
|
||||||
|
description=__doc__,
|
||||||
|
)
|
||||||
|
mode: argparse._MutuallyExclusiveGroup = (
|
||||||
|
parser.add_mutually_exclusive_group()
|
||||||
|
)
|
||||||
|
mode.add_argument(
|
||||||
|
'--live', action='store_true',
|
||||||
|
help='Send one synthetic TypeSafe request (uses API credit)',
|
||||||
|
)
|
||||||
|
mode.add_argument(
|
||||||
|
'--mock', action='store_true',
|
||||||
|
help='Exercise the adapter locally using simulated answers',
|
||||||
|
)
|
||||||
|
parser.add_argument('--model', default=DEFAULT_MODEL)
|
||||||
|
args: argparse.Namespace = parser.parse_args()
|
||||||
|
snapshot: RegimeSnapshot = synthetic_snapshot()
|
||||||
|
if args.live or args.mock:
|
||||||
|
trio.run(classify, snapshot, args.model, args.live)
|
||||||
|
else:
|
||||||
|
print(json.dumps(
|
||||||
|
build_regime_request(snapshot, model=args.model),
|
||||||
|
indent=2,
|
||||||
|
allow_nan=False,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
|
|
@ -202,6 +202,12 @@ class Sampler:
|
||||||
|
|
||||||
if shm_period_s <= 1:
|
if shm_period_s <= 1:
|
||||||
next_t = i_epoch
|
next_t = i_epoch
|
||||||
|
# Startup may seed a bar one second
|
||||||
|
# ahead; rounded wall time can repeat or
|
||||||
|
# move backward. Keep the mutable bar
|
||||||
|
# until a strictly newer epoch arrives.
|
||||||
|
if next_t <= t:
|
||||||
|
continue
|
||||||
|
|
||||||
# this copies non-std fields (eg. vwap) from the
|
# this copies non-std fields (eg. vwap) from the
|
||||||
# last datum
|
# last datum
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,14 @@ from ._volume import (
|
||||||
flow_rates,
|
flow_rates,
|
||||||
tina_vwap,
|
tina_vwap,
|
||||||
)
|
)
|
||||||
|
from ._trend import (
|
||||||
|
TrendCalculator as TrendCalculator,
|
||||||
|
TrendHistory as TrendHistory,
|
||||||
|
TrendUpdate as TrendUpdate,
|
||||||
|
)
|
||||||
|
from ._trend_fsp import (
|
||||||
|
trend_calculus as trend_calculus,
|
||||||
|
)
|
||||||
|
|
||||||
__all__: list[str] = [
|
__all__: list[str] = [
|
||||||
'cascade',
|
'cascade',
|
||||||
|
|
@ -44,6 +52,10 @@ __all__: list[str] = [
|
||||||
'dolla_vlm',
|
'dolla_vlm',
|
||||||
'flow_rates',
|
'flow_rates',
|
||||||
'tina_vwap',
|
'tina_vwap',
|
||||||
|
'TrendCalculator',
|
||||||
|
'TrendHistory',
|
||||||
|
'TrendUpdate',
|
||||||
|
'trend_calculus',
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,7 @@ from __future__ import annotations
|
||||||
from typing import (
|
from typing import (
|
||||||
AsyncGenerator,
|
AsyncGenerator,
|
||||||
Callable,
|
Callable,
|
||||||
|
NamedTuple,
|
||||||
overload,
|
overload,
|
||||||
Protocol,
|
Protocol,
|
||||||
)
|
)
|
||||||
|
|
@ -48,7 +49,25 @@ log = get_logger(__name__)
|
||||||
|
|
||||||
type FspHistory = dict[str, np.ndarray|None]|np.ndarray
|
type FspHistory = dict[str, np.ndarray|None]|np.ndarray
|
||||||
type RealtimeValue = np.ndarray|np.number|int|float
|
type RealtimeValue = np.ndarray|np.number|int|float
|
||||||
type FspYield = FspHistory|tuple[str, RealtimeValue]
|
|
||||||
|
|
||||||
|
class FspSample(NamedTuple):
|
||||||
|
'''
|
||||||
|
Publish all FSP fields at an absolute source coordinate.
|
||||||
|
|
||||||
|
Unlike a field-only tick update, this record remains valid if the
|
||||||
|
sampler has not appended its destination row yet.
|
||||||
|
``FspSample.fields`` are written without task checkpoints.
|
||||||
|
This is not an atomic memory transaction for readers in other
|
||||||
|
processes.
|
||||||
|
|
||||||
|
'''
|
||||||
|
index: int
|
||||||
|
time: float
|
||||||
|
fields: dict[str, RealtimeValue]
|
||||||
|
|
||||||
|
|
||||||
|
type FspYield = FspHistory|tuple[str, RealtimeValue]|FspSample
|
||||||
type FspStream = AsyncGenerator[FspYield, None]
|
type FspStream = AsyncGenerator[FspYield, None]
|
||||||
type FspConfigValue = str|bool|int|float
|
type FspConfigValue = str|bool|int|float
|
||||||
|
|
||||||
|
|
@ -76,6 +95,7 @@ def _load_builtins() -> dict[NamespacePath, Fsp]:
|
||||||
# import to implicity trigger registration via ``@fsp``
|
# import to implicity trigger registration via ``@fsp``
|
||||||
from . import _momo # noqa
|
from . import _momo # noqa
|
||||||
from . import _volume # noqa
|
from . import _volume # noqa
|
||||||
|
from . import _trend_fsp # noqa
|
||||||
|
|
||||||
return _fsp_registry
|
return _fsp_registry
|
||||||
|
|
||||||
|
|
@ -108,6 +128,8 @@ class Fsp[**P]:
|
||||||
*,
|
*,
|
||||||
outputs: tuple[str, ...] = (),
|
outputs: tuple[str, ...] = (),
|
||||||
display_name: str|None = None,
|
display_name: str|None = None,
|
||||||
|
sample_on_step: bool = False,
|
||||||
|
event_outputs: tuple[str, ...] = (),
|
||||||
**config: FspConfigValue,
|
**config: FspConfigValue,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -120,6 +142,12 @@ class Fsp[**P]:
|
||||||
|
|
||||||
self.ns_path: NamespacePath = NamespacePath.from_ref(func)
|
self.ns_path: NamespacePath = NamespacePath.from_ref(func)
|
||||||
self.outputs = outputs
|
self.outputs = outputs
|
||||||
|
# Snapshot processors read SHM on both quote and timer steps.
|
||||||
|
# Sparse event fields must never be carried into a new row.
|
||||||
|
self.sample_on_step: bool = sample_on_step
|
||||||
|
if not set(event_outputs).issubset(outputs):
|
||||||
|
raise ValueError('event outputs must be declared')
|
||||||
|
self.event_outputs: tuple[str, ...] = event_outputs
|
||||||
self.config: dict[str, FspConfigValue] = config
|
self.config: dict[str, FspConfigValue] = config
|
||||||
|
|
||||||
# register with declared set.
|
# register with declared set.
|
||||||
|
|
@ -181,6 +209,8 @@ def fsp[**P](
|
||||||
*,
|
*,
|
||||||
outputs: tuple[str, ...] = (),
|
outputs: tuple[str, ...] = (),
|
||||||
display_name: str|None = None,
|
display_name: str|None = None,
|
||||||
|
sample_on_step: bool = False,
|
||||||
|
event_outputs: tuple[str, ...] = (),
|
||||||
**config: FspConfigValue,
|
**config: FspConfigValue,
|
||||||
|
|
||||||
) -> Fsp[P]:
|
) -> Fsp[P]:
|
||||||
|
|
@ -193,6 +223,8 @@ def fsp[**P](
|
||||||
*,
|
*,
|
||||||
outputs: tuple[str, ...] = (),
|
outputs: tuple[str, ...] = (),
|
||||||
display_name: str|None = None,
|
display_name: str|None = None,
|
||||||
|
sample_on_step: bool = False,
|
||||||
|
event_outputs: tuple[str, ...] = (),
|
||||||
**config: FspConfigValue,
|
**config: FspConfigValue,
|
||||||
|
|
||||||
) -> Callable[[FspFunc[P]], Fsp[P]]:
|
) -> Callable[[FspFunc[P]], Fsp[P]]:
|
||||||
|
|
@ -204,6 +236,8 @@ def fsp[**P](
|
||||||
*,
|
*,
|
||||||
outputs: tuple[str, ...] = (),
|
outputs: tuple[str, ...] = (),
|
||||||
display_name: str|None = None,
|
display_name: str|None = None,
|
||||||
|
sample_on_step: bool = False,
|
||||||
|
event_outputs: tuple[str, ...] = (),
|
||||||
**config: FspConfigValue,
|
**config: FspConfigValue,
|
||||||
|
|
||||||
) -> Fsp[P]|Callable[[FspFunc[P]], Fsp[P]]:
|
) -> Fsp[P]|Callable[[FspFunc[P]], Fsp[P]]:
|
||||||
|
|
@ -214,6 +248,10 @@ def fsp[**P](
|
||||||
``Fsp``. Parameterized ``@fsp(...)`` returns ``decorate`` first;
|
``Fsp``. Parameterized ``@fsp(...)`` returns ``decorate`` first;
|
||||||
Python then passes the decorated function to that closure.
|
Python then passes the decorated function to that closure.
|
||||||
|
|
||||||
|
``sample_on_step`` opts into coalesced SHM snapshot wakeups from
|
||||||
|
quotes and sampler steps; the quote payload is not preserved.
|
||||||
|
``event_outputs`` names sparse fields cleared to NaN on append.
|
||||||
|
|
||||||
'''
|
'''
|
||||||
|
|
||||||
if wrapped is None:
|
if wrapped is None:
|
||||||
|
|
@ -222,12 +260,21 @@ def fsp[**P](
|
||||||
func,
|
func,
|
||||||
outputs=outputs,
|
outputs=outputs,
|
||||||
display_name=display_name,
|
display_name=display_name,
|
||||||
|
sample_on_step=sample_on_step,
|
||||||
|
event_outputs=event_outputs,
|
||||||
**config,
|
**config,
|
||||||
)
|
)
|
||||||
|
|
||||||
return decorate
|
return decorate
|
||||||
|
|
||||||
return Fsp(wrapped, outputs=(wrapped.__name__,))
|
return Fsp(
|
||||||
|
wrapped,
|
||||||
|
outputs=outputs or (wrapped.__name__,),
|
||||||
|
display_name=display_name,
|
||||||
|
sample_on_step=sample_on_step,
|
||||||
|
event_outputs=event_outputs,
|
||||||
|
**config,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def maybe_mk_fsp_shm(
|
def maybe_mk_fsp_shm(
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,7 @@ import trio
|
||||||
from trio_typing import TaskStatus
|
from trio_typing import TaskStatus
|
||||||
import tractor
|
import tractor
|
||||||
from tractor.msg import NamespacePath
|
from tractor.msg import NamespacePath
|
||||||
|
from tractor.trionics import Lagged
|
||||||
|
|
||||||
from piker.types import Struct
|
from piker.types import Struct
|
||||||
from ..log import (
|
from ..log import (
|
||||||
|
|
@ -56,6 +57,7 @@ from ._api import (
|
||||||
FspStream,
|
FspStream,
|
||||||
FspYield,
|
FspYield,
|
||||||
FspHistory,
|
FspHistory,
|
||||||
|
FspSample,
|
||||||
_load_builtins,
|
_load_builtins,
|
||||||
NDToken,
|
NDToken,
|
||||||
)
|
)
|
||||||
|
|
@ -145,6 +147,88 @@ async def filter_quotes_by_sym(
|
||||||
yield quote
|
yield quote
|
||||||
|
|
||||||
|
|
||||||
|
@acm
|
||||||
|
async def _sample_wakeups(
|
||||||
|
quotes: AsyncIterator[QuoteFrame],
|
||||||
|
fqme: str,
|
||||||
|
enabled: bool,
|
||||||
|
) -> AsyncIterator[
|
||||||
|
tuple[AsyncIterator[QuoteFrame], Callable[[], None]|None]
|
||||||
|
]:
|
||||||
|
'''
|
||||||
|
Merge quote and sampler wakeups for SHM snapshot processors.
|
||||||
|
|
||||||
|
One pending notification is enough: the processor reads every
|
||||||
|
newly finalized SHM row when woken. Keep this queue outside the
|
||||||
|
restartable compute task, so history resync cannot lose a step.
|
||||||
|
Tick-sensitive processors retain their original quote stream.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not enabled:
|
||||||
|
yield quotes, None
|
||||||
|
return
|
||||||
|
|
||||||
|
send: trio.MemorySendChannel[QuoteFrame]
|
||||||
|
receive: trio.MemoryReceiveChannel[QuoteFrame]
|
||||||
|
send, receive = trio.open_memory_channel[QuoteFrame](1)
|
||||||
|
|
||||||
|
def notify() -> None:
|
||||||
|
'''
|
||||||
|
Coalesce sampler steps in the local ``send`` channel.
|
||||||
|
|
||||||
|
'''
|
||||||
|
try:
|
||||||
|
send.send_nowait({fqme: {'ticks': []}})
|
||||||
|
except trio.WouldBlock:
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def relay() -> None:
|
||||||
|
'''
|
||||||
|
Resume after quote overrun and wake the SHM snapshot reader.
|
||||||
|
|
||||||
|
'''
|
||||||
|
iterator: AsyncIterator[QuoteFrame] = aiter(quotes)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
frame: QuoteFrame = await anext(iterator)
|
||||||
|
except Lagged:
|
||||||
|
# Dropped quote payloads do not invalidate SHM.
|
||||||
|
# Wake the snapshot processor to catch up all rows.
|
||||||
|
notify()
|
||||||
|
await trio.lowlevel.checkpoint()
|
||||||
|
continue
|
||||||
|
except StopAsyncIteration:
|
||||||
|
return
|
||||||
|
if fqme in frame:
|
||||||
|
notify()
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with send, receive, trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(relay)
|
||||||
|
try:
|
||||||
|
yield receive, notify
|
||||||
|
finally:
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
|
||||||
|
|
||||||
|
def _next_step_output(
|
||||||
|
previous: np.ndarray,
|
||||||
|
fsp: Fsp,
|
||||||
|
zero_on_step: bool,
|
||||||
|
) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Carry state into a new row without duplicating sparse events.
|
||||||
|
|
||||||
|
'''
|
||||||
|
row: np.ndarray = (
|
||||||
|
np.zeros_like(previous) if zero_on_step else previous.copy()
|
||||||
|
)
|
||||||
|
field: str
|
||||||
|
for field in fsp.event_outputs:
|
||||||
|
row[field] = np.nan
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
class _HistoryShm:
|
class _HistoryShm:
|
||||||
'''
|
'''
|
||||||
Present an immutable history view before switching to live SHM.
|
Present an immutable history view before switching to live SHM.
|
||||||
|
|
@ -274,6 +358,7 @@ class Cascade(Struct):
|
||||||
complete: trio.Event|None = None
|
complete: trio.Event|None = None
|
||||||
cs: trio.CancelScope|None = None
|
cs: trio.CancelScope|None = None
|
||||||
client_stream: tractor.MsgStream|None = None
|
client_stream: tractor.MsgStream|None = None
|
||||||
|
sample_ready: trio.Event|None = None
|
||||||
|
|
||||||
async def resync(self) -> int:
|
async def resync(self) -> int:
|
||||||
# TODO: adopt an incremental update engine/approach
|
# TODO: adopt an incremental update engine/approach
|
||||||
|
|
@ -380,7 +465,16 @@ class Cascade(Struct):
|
||||||
) -> int:
|
) -> int:
|
||||||
self.bind_func = bind_func
|
self.bind_func = bind_func
|
||||||
index = await self.tn.start(bind_func)
|
index = await self.tn.start(bind_func)
|
||||||
|
try:
|
||||||
yield index
|
yield index
|
||||||
|
finally:
|
||||||
|
# Stop the current compute task, including after resync,
|
||||||
|
# before the outer scope closes its source receiver.
|
||||||
|
if self.cs is not None:
|
||||||
|
self.cs.cancel()
|
||||||
|
if self.complete is not None:
|
||||||
|
with trio.CancelScope(shield=True):
|
||||||
|
await self.complete.wait()
|
||||||
# TODO: what do we want on teardown/error?
|
# TODO: what do we want on teardown/error?
|
||||||
# -[ ] dynamic reconnection after update?
|
# -[ ] dynamic reconnection after update?
|
||||||
|
|
||||||
|
|
@ -390,6 +484,39 @@ async def _close_fsp_stream(out_stream: FspStream) -> None:
|
||||||
await out_stream.aclose()
|
await out_stream.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
async def _write_sample(
|
||||||
|
casc: Cascade,
|
||||||
|
sample: FspSample,
|
||||||
|
) -> bool:
|
||||||
|
'''
|
||||||
|
Wait for destination advancement, then publish at source index.
|
||||||
|
|
||||||
|
``cascade()`` sets ``Cascade.sample_ready`` after publishing the
|
||||||
|
new destination bound. Capture that event before checking bounds
|
||||||
|
so a sampler wakeup cannot be lost. Cancellation during history
|
||||||
|
resync aborts this wait with the obsolete compute generator.
|
||||||
|
|
||||||
|
'''
|
||||||
|
dst: ShmArray = casc.dst.rt_shm
|
||||||
|
while True:
|
||||||
|
ready: trio.Event|None = casc.sample_ready
|
||||||
|
if ready is None:
|
||||||
|
ready = casc.sample_ready = trio.Event()
|
||||||
|
first: int = dst._first.value
|
||||||
|
last: int = dst._last.value
|
||||||
|
if sample.index < first:
|
||||||
|
return False
|
||||||
|
if sample.index < last:
|
||||||
|
row: np.void = dst._array[sample.index]
|
||||||
|
field: str
|
||||||
|
value: Any
|
||||||
|
for field, value in sample.fields.items():
|
||||||
|
row[field] = value
|
||||||
|
row['time'] = sample.time
|
||||||
|
return True
|
||||||
|
await ready.wait()
|
||||||
|
|
||||||
|
|
||||||
def _write_history_output(
|
def _write_history_output(
|
||||||
dst_shm: ShmArray,
|
dst_shm: ShmArray,
|
||||||
history_output: FspHistory,
|
history_output: FspHistory,
|
||||||
|
|
@ -577,6 +704,9 @@ async def connect_streams(
|
||||||
async for processed in out_stream:
|
async for processed in out_stream:
|
||||||
|
|
||||||
log.debug(f"{func_name}: {processed}")
|
log.debug(f"{func_name}: {processed}")
|
||||||
|
if isinstance(processed, FspSample):
|
||||||
|
await _write_sample(casc, processed)
|
||||||
|
continue
|
||||||
key, output = processed
|
key, output = processed
|
||||||
# dst.array[-1][key] = output
|
# dst.array[-1][key] = output
|
||||||
dst_shm.array[[key, 'time']][-1] = (
|
dst_shm.array[[key, 'time']][-1] = (
|
||||||
|
|
@ -711,6 +841,10 @@ async def cascade(
|
||||||
) as feed:
|
) as feed:
|
||||||
|
|
||||||
flume: Flume = feed.flumes[fqme]
|
flume: Flume = feed.flumes[fqme]
|
||||||
|
# Cached feed consumers can replace ``Flume.stream`` when
|
||||||
|
# opening another subscription. Retain this scope's receiver
|
||||||
|
# before nursery setup gives another consumer a checkpoint.
|
||||||
|
quote_source: AsyncIterator[QuoteFrame] = flume.stream
|
||||||
# XXX: can't do this since flume.feed will be set XD
|
# XXX: can't do this since flume.feed will be set XD
|
||||||
# assert flume == src
|
# assert flume == src
|
||||||
assert flume.mkt == src.mkt
|
assert flume.mkt == src.mkt
|
||||||
|
|
@ -728,9 +862,17 @@ async def cascade(
|
||||||
profiler(f'{func}: feed up')
|
profiler(f'{func}: feed up')
|
||||||
|
|
||||||
func_name: str = func.__name__
|
func_name: str = func.__name__
|
||||||
|
tn: trio.Nursery
|
||||||
|
quote_stream: AsyncIterator[QuoteFrame]
|
||||||
|
wake_sample: Callable[[], None]|None
|
||||||
async with (
|
async with (
|
||||||
tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console
|
tractor.trionics.collapse_eg(), # avoid multi-taskc tb in console
|
||||||
trio.open_nursery() as tn,
|
trio.open_nursery() as tn,
|
||||||
|
_sample_wakeups(
|
||||||
|
quote_source,
|
||||||
|
fqme,
|
||||||
|
fsp.sample_on_step,
|
||||||
|
) as (quote_stream, wake_sample),
|
||||||
):
|
):
|
||||||
# TODO: might be better to just make a "restart" method where
|
# TODO: might be better to just make a "restart" method where
|
||||||
# the target task is spawned implicitly and then the event is
|
# the target task is spawned implicitly and then the event is
|
||||||
|
|
@ -748,7 +890,7 @@ async def cascade(
|
||||||
connect_streams,
|
connect_streams,
|
||||||
casc=casc,
|
casc=casc,
|
||||||
mkt=mkt,
|
mkt=mkt,
|
||||||
quote_stream=flume.stream,
|
quote_stream=quote_stream,
|
||||||
|
|
||||||
# flumes and shm passthrough
|
# flumes and shm passthrough
|
||||||
src=src,
|
src=src,
|
||||||
|
|
@ -777,10 +919,6 @@ async def cascade(
|
||||||
dst_shm: ShmArray = dst.rt_shm
|
dst_shm: ShmArray = dst.rt_shm
|
||||||
src_shm: ShmArray = src.rt_shm
|
src_shm: ShmArray = src.rt_shm
|
||||||
|
|
||||||
if zero_on_step:
|
|
||||||
last = dst.rt_shm.array[-1:]
|
|
||||||
zeroed = np.zeros(last.shape, dtype=last.dtype)
|
|
||||||
|
|
||||||
profiler(f'{func_name}: fsp up')
|
profiler(f'{func_name}: fsp up')
|
||||||
|
|
||||||
# sync to client-side actor
|
# sync to client-side actor
|
||||||
|
|
@ -842,10 +980,9 @@ async def cascade(
|
||||||
|
|
||||||
# some metrics like vlm should be reset
|
# some metrics like vlm should be reset
|
||||||
# to zero every step.
|
# to zero every step.
|
||||||
if zero_on_step:
|
last = _next_step_output(
|
||||||
last = zeroed
|
array[-1:], fsp, zero_on_step,
|
||||||
else:
|
)
|
||||||
last = array[-1:].copy()
|
|
||||||
|
|
||||||
dst.rt_shm.push(last)
|
dst.rt_shm.push(last)
|
||||||
|
|
||||||
|
|
@ -860,6 +997,13 @@ async def cascade(
|
||||||
dst_shm._array['time'][src_li] = src_lt
|
dst_shm._array['time'][src_li] = src_lt
|
||||||
dst_shm._array['time'][src_2li] = src_2lt
|
dst_shm._array['time'][src_2li] = src_2lt
|
||||||
|
|
||||||
|
ready: trio.Event|None = casc.sample_ready
|
||||||
|
casc.sample_ready = trio.Event()
|
||||||
|
if ready is not None:
|
||||||
|
ready.set()
|
||||||
|
if wake_sample is not None:
|
||||||
|
wake_sample()
|
||||||
|
|
||||||
# last2 = dst.array[-2:]
|
# last2 = dst.array[-2:]
|
||||||
# if (
|
# if (
|
||||||
# last2[-1]['index'] != src_li
|
# last2[-1]['index'] != src_li
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,777 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) Tyler Goodlet (in stewardship of piker0)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Optional, bounded TypeSafe System One judgments over trend evidence.
|
||||||
|
|
||||||
|
The HTTP contract is documented at https://docs.typesafe.ai/api .
|
||||||
|
``JevClient`` uses piker's existing HTTPX/Trio stack. Credentials are
|
||||||
|
explicit constructor inputs; importing this module opens no network
|
||||||
|
connection and reads no environment variables. Model judgments never
|
||||||
|
change deterministic TC state or route orders into the EMS.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
|
from dataclasses import dataclass, replace
|
||||||
|
from email.utils import parsedate_to_datetime
|
||||||
|
import json
|
||||||
|
from math import isclose, isfinite
|
||||||
|
from operator import index as integer_index
|
||||||
|
import time
|
||||||
|
from types import MappingProxyType
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import numpy as np
|
||||||
|
import trio
|
||||||
|
|
||||||
|
from ._regime import RegimeSnapshot, TimeframeRegime
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_MODEL: str = 'jev-1.13.0'
|
||||||
|
PROMPT_VERSION: str = 'piker-regimes-v1'
|
||||||
|
API_URL: str = 'https://api.typesafe.ai/v1/systemone'
|
||||||
|
|
||||||
|
# Centralized human-reviewable rubric. These are descriptive model
|
||||||
|
# judgments, not new TC rules or validated profitability signals.
|
||||||
|
REGIME_CRITERIA: MappingProxyType = MappingProxyType({
|
||||||
|
'uptrend': (
|
||||||
|
'Persistent upward movement supported by recent closes and '
|
||||||
|
'ready trend orders; small counter-moves do not dominate.'
|
||||||
|
),
|
||||||
|
'downtrend': (
|
||||||
|
'Persistent downward movement supported by recent closes '
|
||||||
|
'and ready trend orders; small retracements do not dominate.'
|
||||||
|
),
|
||||||
|
'range': (
|
||||||
|
'Recent movement repeatedly retraces within a bounded range '
|
||||||
|
'without a persistent direction, or closes remain flat.'
|
||||||
|
),
|
||||||
|
'transition': (
|
||||||
|
'Recent detected reversals and conflicting ready orders '
|
||||||
|
'suggest a change of regime whose direction is unsettled.'
|
||||||
|
),
|
||||||
|
'unknown': (
|
||||||
|
'Available ready orders and recent observations provide '
|
||||||
|
'insufficient or contradictory evidence for another label.'
|
||||||
|
),
|
||||||
|
})
|
||||||
|
ALIGNMENT_CRITERIA: MappingProxyType = MappingProxyType({
|
||||||
|
'aligned_up': 'Ready timeframes consistently support an uptrend',
|
||||||
|
'aligned_down': (
|
||||||
|
'Ready timeframes consistently support a downtrend.'
|
||||||
|
),
|
||||||
|
'mixed': (
|
||||||
|
'Ready timeframes disagree in direction or regime, such as '
|
||||||
|
'a shorter pullback within a longer directional trend.'
|
||||||
|
),
|
||||||
|
'unknown': (
|
||||||
|
'There is insufficient directional evidence to assess '
|
||||||
|
'agreement across the ready timeframes.'
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RegimeAnswer:
|
||||||
|
'''
|
||||||
|
Retain a model label and its entire probability distribution.
|
||||||
|
|
||||||
|
``RegimeAnswer.period=None`` identifies timeframe alignment.
|
||||||
|
``RegimeAnswer.probability`` is the selected label's probability;
|
||||||
|
``RegimeAnswer.confidence`` separately describes concentration.
|
||||||
|
|
||||||
|
'''
|
||||||
|
question: str
|
||||||
|
period: float|None
|
||||||
|
label: str
|
||||||
|
probability: float
|
||||||
|
confidence: float
|
||||||
|
probabilities: tuple[tuple[str, float], ...]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class JevClassification:
|
||||||
|
'''
|
||||||
|
Associate semantic judgments with evidence and availability time.
|
||||||
|
|
||||||
|
``JevClassification.source_as_of`` is an event-time bar END.
|
||||||
|
``JevClassification.requested_at`` and
|
||||||
|
``JevClassification.received_at`` are Unix wall-clock timestamps.
|
||||||
|
A live chart must not start a classification before receipt.
|
||||||
|
``JevClassification.latency`` uses a monotonic clock; wall-clock
|
||||||
|
adjustments do not change the measured request duration.
|
||||||
|
|
||||||
|
'''
|
||||||
|
market: str
|
||||||
|
revision: int
|
||||||
|
sequence: int
|
||||||
|
segment: int
|
||||||
|
source_as_of: float
|
||||||
|
requested_at: float
|
||||||
|
received_at: float
|
||||||
|
latency: float
|
||||||
|
model: str
|
||||||
|
prompt_version: str
|
||||||
|
answers: tuple[RegimeAnswer, ...]
|
||||||
|
input_tokens: int
|
||||||
|
output_tokens: int
|
||||||
|
|
||||||
|
|
||||||
|
class JevError(Exception):
|
||||||
|
'''
|
||||||
|
Describe a provider failure without echoing credential values.
|
||||||
|
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
class JevProtocolError(JevError):
|
||||||
|
'''
|
||||||
|
Reject malformed or semantically inconsistent provider answers.
|
||||||
|
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
class JevServiceError(JevError):
|
||||||
|
'''
|
||||||
|
Preserve an HTTP status and a bounded retry delay, not its body.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
status: int,
|
||||||
|
retry_after: float = 0,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Retain an HTTP failure status without its response body.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.status: int = status
|
||||||
|
self.retry_after: float = retry_after
|
||||||
|
super().__init__(f'TypeSafe request failed: HTTP {status}')
|
||||||
|
|
||||||
|
@property
|
||||||
|
def retryable(self) -> bool:
|
||||||
|
'''
|
||||||
|
Identify transient statuses documented by HTTP and TypeSafe.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self.status in {408, 429, 500, 502, 503, 504, 529}
|
||||||
|
|
||||||
|
|
||||||
|
def build_regime_request(
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
*,
|
||||||
|
model: str = DEFAULT_MODEL,
|
||||||
|
evidence_limit: int = 16,
|
||||||
|
) -> dict:
|
||||||
|
'''
|
||||||
|
Batch independent timeframe questions over one immutable state.
|
||||||
|
|
||||||
|
Question identifiers are opaque to Jev; each instruction names
|
||||||
|
its actual state path. Unready timeframes provide context only.
|
||||||
|
``evidence_limit`` bounds each timeframe's model-input tails
|
||||||
|
independently of larger histories retained for chart rendering.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isinstance(model, str) or not model.strip():
|
||||||
|
raise ValueError('a model identifier is required')
|
||||||
|
if (
|
||||||
|
isinstance(evidence_limit, (bool, np.bool_))
|
||||||
|
or not 1 <= integer_index(evidence_limit) <= 256
|
||||||
|
):
|
||||||
|
raise ValueError('evidence_limit must be in [1, 256]')
|
||||||
|
frames: list[TimeframeRegime] = []
|
||||||
|
frame: TimeframeRegime
|
||||||
|
for frame in snapshot.timeframes:
|
||||||
|
frames.append(replace(
|
||||||
|
frame,
|
||||||
|
recent_closes=frame.recent_closes[-evidence_limit:],
|
||||||
|
reversals=frame.reversals[-evidence_limit:],
|
||||||
|
))
|
||||||
|
evidence: RegimeSnapshot = replace(
|
||||||
|
snapshot, timeframes=tuple(frames),
|
||||||
|
)
|
||||||
|
questions: dict[str, dict] = {}
|
||||||
|
number: int
|
||||||
|
for number, frame in enumerate(snapshot.timeframes):
|
||||||
|
if not frame.ready or not frame.ready[0]:
|
||||||
|
continue
|
||||||
|
questions[f'timeframe_{number}'] = {
|
||||||
|
'type': 'choice',
|
||||||
|
'instructions': (
|
||||||
|
f'Which descriptive market regime best fits '
|
||||||
|
f'`timeframes[{number}]` at its `as_of` time? '
|
||||||
|
f'Use only its observed recent closes, ready trend '
|
||||||
|
f'orders and detected reversals. TC order numbers '
|
||||||
|
f'are recursive reversal scales, not seconds. '
|
||||||
|
f'Zero is unresolved, not automatically a range. '
|
||||||
|
f'Ignore unready orders. A pivot was only known at '
|
||||||
|
f'its detected_time. Do not predict future returns '
|
||||||
|
f'or make a trading recommendation.'
|
||||||
|
),
|
||||||
|
'criteria': dict(REGIME_CRITERIA),
|
||||||
|
}
|
||||||
|
if not questions:
|
||||||
|
raise ValueError('no timeframe has completed TC warm-up')
|
||||||
|
if len(questions) > 1:
|
||||||
|
questions['alignment'] = {
|
||||||
|
'type': 'choice',
|
||||||
|
'instructions': (
|
||||||
|
'How do directions across the ready entries in '
|
||||||
|
'`timeframes` agree at `as_of`? Use only observed '
|
||||||
|
'closes and ready TC orders. Each timeframe may end '
|
||||||
|
'earlier than the overall as_of; do not invent '
|
||||||
|
'partial buckets or rely on other question answers.'
|
||||||
|
),
|
||||||
|
'criteria': dict(ALIGNMENT_CRITERIA),
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
'model': model,
|
||||||
|
'state': evidence.to_state(),
|
||||||
|
'questions': questions,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _probability(value: object) -> float:
|
||||||
|
'''
|
||||||
|
Enforce finite unit-interval probabilities without coercing text.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
isinstance(value, bool)
|
||||||
|
or not isinstance(value, (float, int))
|
||||||
|
or not isfinite(value)
|
||||||
|
or not 0 <= value <= 1
|
||||||
|
):
|
||||||
|
raise JevProtocolError('invalid probability or confidence')
|
||||||
|
return float(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _retry_after(value: str|None) -> float:
|
||||||
|
'''
|
||||||
|
Interpret Retry-After seconds or an HTTP date with a hard bound.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if value is None:
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
delay: float = float(value)
|
||||||
|
except ValueError:
|
||||||
|
try:
|
||||||
|
delay = parsedate_to_datetime(value).timestamp()
|
||||||
|
delay -= time.time()
|
||||||
|
except (
|
||||||
|
TypeError,
|
||||||
|
ValueError,
|
||||||
|
OverflowError,
|
||||||
|
):
|
||||||
|
return 0
|
||||||
|
if not isfinite(delay):
|
||||||
|
return 0
|
||||||
|
return min(60, max(0, delay))
|
||||||
|
|
||||||
|
|
||||||
|
class JevClient:
|
||||||
|
'''
|
||||||
|
Perform one bounded, validated HTTP evaluation on the Trio loop.
|
||||||
|
|
||||||
|
The caller owns ``http`` and must close it. Retries and pacing
|
||||||
|
belong to ``RegimeClassifier``, so a single evaluation never
|
||||||
|
silently resends stale state. Redirects are disabled to keep
|
||||||
|
the explicit credential bound to the TypeSafe API origin.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
http: httpx.AsyncClient,
|
||||||
|
api_key: str,
|
||||||
|
*,
|
||||||
|
model: str = DEFAULT_MODEL,
|
||||||
|
timeout: float = 5,
|
||||||
|
max_request_bytes: int = 65_536,
|
||||||
|
max_response_bytes: int = 131_072,
|
||||||
|
evidence_limit: int = 16,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Bind a caller-owned HTTP session, key and request limits.
|
||||||
|
|
||||||
|
The caller retains ownership of the supplied ``http`` client.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isinstance(api_key, str) or not api_key.strip():
|
||||||
|
raise ValueError('explicit TypeSafe API key required')
|
||||||
|
if not isfinite(timeout) or timeout <= 0:
|
||||||
|
raise ValueError('timeout must be positive and finite')
|
||||||
|
if max_request_bytes <= 0 or max_response_bytes <= 0:
|
||||||
|
raise ValueError('byte limits must be positive')
|
||||||
|
if not isinstance(model, str) or not model.strip():
|
||||||
|
raise ValueError('a model identifier is required')
|
||||||
|
if (
|
||||||
|
isinstance(evidence_limit, (bool, np.bool_))
|
||||||
|
or not 1 <= integer_index(evidence_limit) <= 256
|
||||||
|
):
|
||||||
|
raise ValueError('evidence_limit must be in [1, 256]')
|
||||||
|
self._http: httpx.AsyncClient = http
|
||||||
|
self._api_key: str = api_key
|
||||||
|
self.model: str = model
|
||||||
|
self.timeout: float = timeout
|
||||||
|
self.max_request_bytes: int = max_request_bytes
|
||||||
|
self.max_response_bytes: int = max_response_bytes
|
||||||
|
self.evidence_limit: int = int(evidence_limit)
|
||||||
|
|
||||||
|
async def classify(
|
||||||
|
self,
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
) -> JevClassification:
|
||||||
|
'''
|
||||||
|
Validate one complete response against the exact sent rubric.
|
||||||
|
|
||||||
|
'''
|
||||||
|
payload: dict = build_regime_request(
|
||||||
|
snapshot, model=self.model,
|
||||||
|
evidence_limit=self.evidence_limit,
|
||||||
|
)
|
||||||
|
body: bytes = json.dumps(
|
||||||
|
payload,
|
||||||
|
allow_nan=False,
|
||||||
|
separators=(',', ':'),
|
||||||
|
).encode()
|
||||||
|
if len(body) > self.max_request_bytes:
|
||||||
|
raise ValueError('regime request exceeds byte budget')
|
||||||
|
requested_at: float = time.time()
|
||||||
|
started: float = trio.current_time()
|
||||||
|
data: bytearray = bytearray()
|
||||||
|
api_key: str = self._api_key
|
||||||
|
response: httpx.Response
|
||||||
|
with trio.fail_after(self.timeout):
|
||||||
|
async with self._http.stream(
|
||||||
|
'POST',
|
||||||
|
API_URL,
|
||||||
|
content=body,
|
||||||
|
headers={
|
||||||
|
'Authorization': f'Bearer {api_key}',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
timeout=self.timeout,
|
||||||
|
follow_redirects=False,
|
||||||
|
) as response:
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise JevServiceError(
|
||||||
|
response.status_code,
|
||||||
|
_retry_after(
|
||||||
|
response.headers.get('retry-after'),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
chunk: bytes
|
||||||
|
async for chunk in response.aiter_bytes():
|
||||||
|
data.extend(chunk)
|
||||||
|
if len(data) > self.max_response_bytes:
|
||||||
|
raise JevProtocolError(
|
||||||
|
'response exceeds the byte budget',
|
||||||
|
)
|
||||||
|
received_at: float = time.time()
|
||||||
|
latency: float = trio.current_time() - started
|
||||||
|
error: ValueError|UnicodeError
|
||||||
|
try:
|
||||||
|
decoded: object = json.loads(data)
|
||||||
|
except (
|
||||||
|
ValueError,
|
||||||
|
UnicodeError,
|
||||||
|
) as error:
|
||||||
|
raise JevProtocolError('invalid JSON reply') from error
|
||||||
|
return self._parse(
|
||||||
|
snapshot,
|
||||||
|
payload,
|
||||||
|
decoded,
|
||||||
|
requested_at,
|
||||||
|
received_at,
|
||||||
|
latency,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _parse(
|
||||||
|
self,
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
payload: dict,
|
||||||
|
decoded: object,
|
||||||
|
requested_at: float,
|
||||||
|
received_at: float,
|
||||||
|
latency: float,
|
||||||
|
) -> JevClassification:
|
||||||
|
'''
|
||||||
|
Reject missing questions, bad labels, or bad distributions.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isinstance(decoded, dict):
|
||||||
|
raise JevProtocolError('response must be an object')
|
||||||
|
model: object = decoded.get('model')
|
||||||
|
if not isinstance(model, str) or not model:
|
||||||
|
raise JevProtocolError('response model is missing')
|
||||||
|
if (
|
||||||
|
self.model not in {'jev-latest', 'jev-preview'}
|
||||||
|
and model != self.model
|
||||||
|
):
|
||||||
|
raise JevProtocolError('response violates pinned model')
|
||||||
|
raw_answers: object = decoded.get('answers')
|
||||||
|
questions: dict = payload['questions']
|
||||||
|
if (
|
||||||
|
not isinstance(raw_answers, dict)
|
||||||
|
or set(raw_answers) != set(questions)
|
||||||
|
):
|
||||||
|
raise JevProtocolError('response question set differs')
|
||||||
|
answers: list[RegimeAnswer] = []
|
||||||
|
name: str
|
||||||
|
question: dict
|
||||||
|
for name, question in questions.items():
|
||||||
|
raw: object = raw_answers[name]
|
||||||
|
if (
|
||||||
|
not isinstance(raw, dict)
|
||||||
|
or raw.get('type') != 'choice'
|
||||||
|
):
|
||||||
|
raise JevProtocolError('expected a Choice answer')
|
||||||
|
label: object = raw.get('choice')
|
||||||
|
probabilities: object = raw.get('probabilities')
|
||||||
|
if (
|
||||||
|
not isinstance(probabilities, dict)
|
||||||
|
or set(probabilities) != set(question['criteria'])
|
||||||
|
or not isinstance(label, str)
|
||||||
|
or label not in probabilities
|
||||||
|
):
|
||||||
|
raise JevProtocolError('answer options differ')
|
||||||
|
distribution: dict[str, float] = {}
|
||||||
|
option: str
|
||||||
|
probability: object
|
||||||
|
for option, probability in probabilities.items():
|
||||||
|
distribution[option] = _probability(probability)
|
||||||
|
if not isclose(
|
||||||
|
sum(distribution.values()), 1, abs_tol=1e-5,
|
||||||
|
):
|
||||||
|
raise JevProtocolError('probabilities must sum to 1')
|
||||||
|
if distribution[label] < max(distribution.values()):
|
||||||
|
raise JevProtocolError('choice is not most probable')
|
||||||
|
period: float|None = None
|
||||||
|
if name != 'alignment':
|
||||||
|
position: int = int(name.removeprefix('timeframe_'))
|
||||||
|
period = snapshot.timeframes[position].period
|
||||||
|
answers.append(RegimeAnswer(
|
||||||
|
question=name,
|
||||||
|
period=period,
|
||||||
|
label=label,
|
||||||
|
probability=distribution[label],
|
||||||
|
confidence=_probability(raw.get('confidence')),
|
||||||
|
probabilities=tuple(distribution.items()),
|
||||||
|
))
|
||||||
|
usage: object = decoded.get('usage')
|
||||||
|
if not isinstance(usage, dict):
|
||||||
|
raise JevProtocolError('token usage is missing')
|
||||||
|
key: str
|
||||||
|
for key in ('input_tokens', 'output_tokens'):
|
||||||
|
count: object = usage.get(key)
|
||||||
|
if isinstance(count, bool) or not isinstance(count, int):
|
||||||
|
raise JevProtocolError('token usage must be integer')
|
||||||
|
if count < 0:
|
||||||
|
raise JevProtocolError('token usage must be >= 0')
|
||||||
|
return JevClassification(
|
||||||
|
market=snapshot.market,
|
||||||
|
revision=snapshot.revision,
|
||||||
|
sequence=snapshot.sequence,
|
||||||
|
segment=snapshot.segment,
|
||||||
|
source_as_of=snapshot.as_of,
|
||||||
|
requested_at=requested_at,
|
||||||
|
received_at=received_at,
|
||||||
|
latency=latency,
|
||||||
|
model=model,
|
||||||
|
prompt_version=PROMPT_VERSION,
|
||||||
|
answers=tuple(answers),
|
||||||
|
input_tokens=usage['input_tokens'],
|
||||||
|
output_tokens=usage['output_tokens'],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _Submission:
|
||||||
|
'''
|
||||||
|
Retain one snapshot and its monotonic local submission time.
|
||||||
|
|
||||||
|
'''
|
||||||
|
snapshot: RegimeSnapshot
|
||||||
|
submitted: float
|
||||||
|
|
||||||
|
|
||||||
|
class RegimeClassifier:
|
||||||
|
'''
|
||||||
|
Coalesce producer updates into one paced, single-flight worker.
|
||||||
|
|
||||||
|
``RegimeClassifier.submit()`` replaces
|
||||||
|
``RegimeClassifier._pending`` without awaiting HTTP or consumers.
|
||||||
|
Responses must share the current revision/segment and remain
|
||||||
|
within ``RegimeClassifier.max_source_lag`` event-time seconds and
|
||||||
|
``RegimeClassifier.max_age`` local seconds of their evidence.
|
||||||
|
A small nonzero lag permits useful inference on fast feeds;
|
||||||
|
zero requires the response's exact latest event-time snapshot.
|
||||||
|
|
||||||
|
Run one worker in the owning actor's nursery. Cancellation closes
|
||||||
|
in-flight I/O through HTTPX; the caller owns the HTTP session.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
client: JevClient,
|
||||||
|
*,
|
||||||
|
min_interval: float = 1,
|
||||||
|
max_age: float = 5,
|
||||||
|
max_source_lag: float = 2,
|
||||||
|
max_retries: int = 2,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Configure bounded request pacing and evidence freshness.
|
||||||
|
|
||||||
|
'''
|
||||||
|
value: float
|
||||||
|
for value in (min_interval, max_source_lag):
|
||||||
|
if not isfinite(value) or value < 0:
|
||||||
|
raise ValueError('interval and lag must be >= 0')
|
||||||
|
if not isfinite(max_age) or max_age <= 0:
|
||||||
|
raise ValueError('max_age must be positive and finite')
|
||||||
|
if (
|
||||||
|
isinstance(max_retries, bool)
|
||||||
|
or not 0 <= integer_index(max_retries) <= 8
|
||||||
|
):
|
||||||
|
raise ValueError('max_retries must be in [0, 8]')
|
||||||
|
self.client: JevClient = client
|
||||||
|
self.min_interval: float = min_interval
|
||||||
|
self.max_age: float = max_age
|
||||||
|
self.max_source_lag: float = max_source_lag
|
||||||
|
self.max_retries: int = max_retries
|
||||||
|
self.requests: int = 0
|
||||||
|
self.published: int = 0
|
||||||
|
self.discarded: int = 0
|
||||||
|
self.failures: int = 0
|
||||||
|
self.last_error: Exception|None = None
|
||||||
|
self._latest: _Submission|None = None
|
||||||
|
self._pending: _Submission|None = None
|
||||||
|
self._wake: trio.Event = trio.Event()
|
||||||
|
self._running: bool = False
|
||||||
|
self._minimum_revision: int = 0
|
||||||
|
self._publishing: _Submission|None = None
|
||||||
|
self._publish_scope: trio.CancelScope|None = None
|
||||||
|
|
||||||
|
def invalidate(self, revision: int) -> None:
|
||||||
|
'''
|
||||||
|
Revoke older evidence immediately, before source replay.
|
||||||
|
|
||||||
|
A feed owner calls this upon receiving a history revision;
|
||||||
|
it need not wait for a replacement snapshot to become ready.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
isinstance(revision, bool)
|
||||||
|
or integer_index(revision) < 0
|
||||||
|
):
|
||||||
|
raise ValueError('revision must be an integer >= 0')
|
||||||
|
self._minimum_revision = max(
|
||||||
|
self._minimum_revision, revision,
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
self._pending is not None
|
||||||
|
and self._pending.snapshot.revision < revision
|
||||||
|
):
|
||||||
|
self._pending = None
|
||||||
|
self._cancel_stale_publication()
|
||||||
|
|
||||||
|
def _cancel_stale_publication(self) -> None:
|
||||||
|
'''
|
||||||
|
Revoke stale ``RegimeClassifier._publish_scope`` delivery.
|
||||||
|
|
||||||
|
``RegimeClassifier._publishing`` retains the exact submission
|
||||||
|
whose consumer may currently be blocked at an ``await``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
self._publishing is not None
|
||||||
|
and self._publish_scope is not None
|
||||||
|
and not self._fresh(self._publishing)
|
||||||
|
):
|
||||||
|
self._publish_scope.cancel()
|
||||||
|
|
||||||
|
def submit(self, snapshot: RegimeSnapshot) -> bool:
|
||||||
|
'''
|
||||||
|
Replace ``RegimeClassifier._pending`` with newer evidence.
|
||||||
|
|
||||||
|
Refuse older source coordinates or a different market before
|
||||||
|
replacing ``RegimeClassifier._latest`` and its pending work.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if snapshot.revision < self._minimum_revision:
|
||||||
|
return False
|
||||||
|
latest: _Submission|None = self._latest
|
||||||
|
if latest is not None:
|
||||||
|
previous: RegimeSnapshot = latest.snapshot
|
||||||
|
if snapshot.market != previous.market:
|
||||||
|
raise ValueError('one classifier owns one market')
|
||||||
|
if (snapshot.revision, snapshot.sequence) <= (
|
||||||
|
previous.revision, previous.sequence,
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
if (
|
||||||
|
snapshot.revision == previous.revision
|
||||||
|
and snapshot.as_of <= previous.as_of
|
||||||
|
):
|
||||||
|
raise ValueError('as_of must advance in a revision')
|
||||||
|
submission: _Submission = _Submission(
|
||||||
|
snapshot,
|
||||||
|
trio.current_time(),
|
||||||
|
)
|
||||||
|
self._latest = self._pending = submission
|
||||||
|
self._cancel_stale_publication()
|
||||||
|
self._wake.set()
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _fresh(self, submission: _Submission) -> bool:
|
||||||
|
'''
|
||||||
|
Check revision, gap segment, source lag, and elapsed age.
|
||||||
|
|
||||||
|
'''
|
||||||
|
latest: _Submission|None = self._latest
|
||||||
|
assert latest is not None
|
||||||
|
source: RegimeSnapshot = submission.snapshot
|
||||||
|
newest: RegimeSnapshot = latest.snapshot
|
||||||
|
lag: float = newest.as_of - source.as_of
|
||||||
|
age: float = trio.current_time() - submission.submitted
|
||||||
|
return (
|
||||||
|
source.revision >= self._minimum_revision
|
||||||
|
and source.revision == newest.revision
|
||||||
|
and source.segment == newest.segment
|
||||||
|
and 0 <= lag <= self.max_source_lag
|
||||||
|
and age <= self.max_age
|
||||||
|
)
|
||||||
|
|
||||||
|
async def run(
|
||||||
|
self,
|
||||||
|
publish: Callable[[JevClassification], Awaitable[None]],
|
||||||
|
*,
|
||||||
|
on_error: Callable[[Exception], Awaitable[None]]|None = None,
|
||||||
|
task_status: trio.TaskStatus[None] = (
|
||||||
|
trio.TASK_STATUS_IGNORED
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Serve until cancelled, with bounded transient retries.
|
||||||
|
|
||||||
|
Authentication and malformed-response failures propagate;
|
||||||
|
transient errors call ``on_error`` and retry newest state.
|
||||||
|
Retry-After delays have a 60-second cap. Each retry rechecks
|
||||||
|
freshness; an expired snapshot is never resent.
|
||||||
|
|
||||||
|
``publish`` must cooperate with Trio cancellation. Its wait
|
||||||
|
is cancelled if evidence expires or is invalidated. Consumers
|
||||||
|
must also validate received message provenance: a delivery
|
||||||
|
already completed before invalidation cannot be revoked.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if self._running:
|
||||||
|
raise RuntimeError('classifier worker already running')
|
||||||
|
self._running = True
|
||||||
|
task_status.started()
|
||||||
|
next_request: float = 0
|
||||||
|
retries: int = 0
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
if self._pending is None:
|
||||||
|
await self._wake.wait()
|
||||||
|
self._wake = trio.Event()
|
||||||
|
await trio.sleep_until(next_request)
|
||||||
|
submission: _Submission|None = self._pending
|
||||||
|
self._pending = None
|
||||||
|
if submission is None:
|
||||||
|
continue
|
||||||
|
if not self._fresh(submission):
|
||||||
|
self.discarded += 1
|
||||||
|
continue
|
||||||
|
ready: bool = False
|
||||||
|
frame: TimeframeRegime
|
||||||
|
for frame in submission.snapshot.timeframes:
|
||||||
|
ready |= bool(frame.ready and frame.ready[0])
|
||||||
|
if not ready:
|
||||||
|
continue
|
||||||
|
next_request = (
|
||||||
|
trio.current_time() + self.min_interval
|
||||||
|
)
|
||||||
|
self.requests += 1
|
||||||
|
error: (
|
||||||
|
httpx.TransportError
|
||||||
|
|trio.TooSlowError
|
||||||
|
|JevServiceError
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
result: JevClassification = await (
|
||||||
|
self.client.classify(submission.snapshot)
|
||||||
|
)
|
||||||
|
except (
|
||||||
|
httpx.TransportError,
|
||||||
|
trio.TooSlowError,
|
||||||
|
JevServiceError,
|
||||||
|
) as error:
|
||||||
|
self.failures += 1
|
||||||
|
self.last_error = error
|
||||||
|
if (
|
||||||
|
isinstance(error, JevServiceError)
|
||||||
|
and not error.retryable
|
||||||
|
):
|
||||||
|
raise
|
||||||
|
if on_error is not None:
|
||||||
|
await on_error(error)
|
||||||
|
if retries < self.max_retries:
|
||||||
|
delay: float = min(60, 2 ** retries)
|
||||||
|
if isinstance(error, JevServiceError):
|
||||||
|
delay = max(delay, error.retry_after)
|
||||||
|
next_request = max(
|
||||||
|
next_request,
|
||||||
|
trio.current_time() + delay,
|
||||||
|
)
|
||||||
|
if self._pending is None:
|
||||||
|
self._pending = self._latest
|
||||||
|
retries += 1
|
||||||
|
else:
|
||||||
|
retries = 0
|
||||||
|
continue
|
||||||
|
|
||||||
|
retries = 0
|
||||||
|
self.last_error = None
|
||||||
|
if self._fresh(submission):
|
||||||
|
deadline: float = (
|
||||||
|
submission.submitted + self.max_age
|
||||||
|
)
|
||||||
|
scope: trio.CancelScope = trio.CancelScope(
|
||||||
|
deadline=deadline,
|
||||||
|
)
|
||||||
|
with scope:
|
||||||
|
self._publish_scope = scope
|
||||||
|
self._publishing = submission
|
||||||
|
try:
|
||||||
|
await publish(result)
|
||||||
|
finally:
|
||||||
|
self._publishing = None
|
||||||
|
self._publish_scope = None
|
||||||
|
if scope.cancelled_caught:
|
||||||
|
self.discarded += 1
|
||||||
|
else:
|
||||||
|
self.published += 1
|
||||||
|
else:
|
||||||
|
self.discarded += 1
|
||||||
|
finally:
|
||||||
|
self._running = False
|
||||||
|
|
@ -0,0 +1,422 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) Tyler Goodlet (in stewardship of piker0)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Closed-bar, multiple-timeframe trend evidence for regime consumers.
|
||||||
|
|
||||||
|
``MultiTimeframeTrend.push()`` receives a FINAL base-bar close and
|
||||||
|
the bar's START timestamp. Each scalar close is observed at the END
|
||||||
|
of its interval. Epoch-aligned buckets require every constituent
|
||||||
|
base bar; neither partial startup buckets nor missing bars become
|
||||||
|
synthetic observations. A source gap resets all recurrence state.
|
||||||
|
|
||||||
|
Wall-clock periods and Trend Calculus observation windows are
|
||||||
|
independent parameters. This module delegates every trend decision
|
||||||
|
to ``TrendCalculator``; semantic model labels belong in ``_jev``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections import deque
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
from decimal import Decimal
|
||||||
|
from math import isclose, isfinite, ulp
|
||||||
|
from operator import index as integer_index
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from ._trend import TrendCalculator, TrendUpdate
|
||||||
|
|
||||||
|
|
||||||
|
def _period_us(period: float) -> int:
|
||||||
|
'''
|
||||||
|
Normalize periods to positive integral microseconds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if isinstance(period, bool) or not isfinite(period):
|
||||||
|
raise ValueError('period must be finite and positive')
|
||||||
|
micros: Decimal = Decimal(str(period)) * 1_000_000
|
||||||
|
nearest: Decimal = micros.to_integral_value()
|
||||||
|
tolerance: Decimal = Decimal(str(2 * ulp(period))) * 1_000_000
|
||||||
|
if nearest <= 0 or abs(micros - nearest) > tolerance:
|
||||||
|
raise ValueError(
|
||||||
|
'period requires positive whole microseconds',
|
||||||
|
)
|
||||||
|
return int(nearest)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class CloseSample:
|
||||||
|
'''
|
||||||
|
A completed bucket close at its observation time.
|
||||||
|
|
||||||
|
``CloseSample.index`` is the final contributing base coordinate.
|
||||||
|
|
||||||
|
'''
|
||||||
|
time: float
|
||||||
|
index: int
|
||||||
|
price: float
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TrendReversal:
|
||||||
|
'''
|
||||||
|
Preserve both a pivot and when its reversal became observable.
|
||||||
|
|
||||||
|
'''
|
||||||
|
index: int
|
||||||
|
time: float
|
||||||
|
price: float
|
||||||
|
direction: int
|
||||||
|
order: int
|
||||||
|
detected_index: int
|
||||||
|
detected_time: float
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TimeframeRegime:
|
||||||
|
'''
|
||||||
|
Immutable deterministic evidence for one wall-clock timeframe.
|
||||||
|
|
||||||
|
``TimeframeRegime.ready`` concerns complete TC windows, never
|
||||||
|
model confidence. ``TimeframeRegime.as_of`` is the last complete
|
||||||
|
bucket's end, or ``None``. ``TimeframeRegime.recent_closes`` and
|
||||||
|
``TimeframeRegime.reversals`` retain bounded causal evidence.
|
||||||
|
|
||||||
|
'''
|
||||||
|
period: float
|
||||||
|
as_of: float|None
|
||||||
|
close_count: int
|
||||||
|
trends: tuple[int, ...]
|
||||||
|
ready: tuple[bool, ...]
|
||||||
|
recent_closes: tuple[CloseSample, ...]
|
||||||
|
reversals: tuple[TrendReversal, ...]
|
||||||
|
discarded_buckets: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RegimeSnapshot:
|
||||||
|
'''
|
||||||
|
Bind all timeframe evidence to one source revision and clock.
|
||||||
|
|
||||||
|
``RegimeSnapshot.sequence`` increases on every accepted base bar
|
||||||
|
within a revision. ``RegimeSnapshot.segment`` increases at source
|
||||||
|
gaps. ``RegimeSnapshot.as_of`` records the latest base-bar END,
|
||||||
|
independently of the snapshot's wall-clock arrival time.
|
||||||
|
|
||||||
|
'''
|
||||||
|
market: str
|
||||||
|
revision: int
|
||||||
|
sequence: int
|
||||||
|
segment: int
|
||||||
|
as_of: float
|
||||||
|
base_period: float
|
||||||
|
windows: tuple[int, ...]
|
||||||
|
timeframes: tuple[TimeframeRegime, ...]
|
||||||
|
|
||||||
|
def to_state(self) -> dict:
|
||||||
|
'''
|
||||||
|
Build JSON-safe state with explicit timestamp conventions.
|
||||||
|
|
||||||
|
'''
|
||||||
|
state: dict = asdict(self)
|
||||||
|
state['algorithm'] = 'trend_calculus'
|
||||||
|
state['price_source'] = 'closed_bar_close'
|
||||||
|
state['timestamp_convention'] = 'interval_end'
|
||||||
|
state['trend_codes'] = {
|
||||||
|
'-1': 'down',
|
||||||
|
'0': 'unresolved or not ready; consult ready',
|
||||||
|
'1': 'up',
|
||||||
|
}
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
class _Timeframe:
|
||||||
|
'''
|
||||||
|
Own one bounded recurrence and its current aggregation bucket.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
period_us: int,
|
||||||
|
factor: int,
|
||||||
|
windows: tuple[int, ...],
|
||||||
|
history_limit: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Allocate one recurrence and bounded close/reversal tails.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.period_us: int = period_us
|
||||||
|
self.factor: int = factor
|
||||||
|
self.calculator: TrendCalculator = TrendCalculator(windows)
|
||||||
|
self.closes: deque[CloseSample] = deque(maxlen=history_limit)
|
||||||
|
self.reversals: deque[TrendReversal] = deque(
|
||||||
|
maxlen=history_limit,
|
||||||
|
)
|
||||||
|
self.update: TrendUpdate|None = None
|
||||||
|
self.close_count: int = 0
|
||||||
|
self.bucket: int|None = None
|
||||||
|
self.count: int = 0
|
||||||
|
self.discarded: int = 0
|
||||||
|
|
||||||
|
def reset(self) -> None:
|
||||||
|
'''
|
||||||
|
Reset ``_Timeframe.calculator`` and its bucket provenance.
|
||||||
|
|
||||||
|
Clear ``_Timeframe.closes`` and ``_Timeframe.reversals`` on a
|
||||||
|
source gap so no previous segment survives as evidence.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if self.count:
|
||||||
|
self.discarded += 1
|
||||||
|
self.calculator.reset()
|
||||||
|
self.closes.clear()
|
||||||
|
self.reversals.clear()
|
||||||
|
self.update = None
|
||||||
|
self.close_count = 0
|
||||||
|
self.bucket = None
|
||||||
|
self.count = 0
|
||||||
|
|
||||||
|
def push(
|
||||||
|
self,
|
||||||
|
tick: int,
|
||||||
|
close: float,
|
||||||
|
source_index: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Advance only after an entire aligned bucket was observed.
|
||||||
|
|
||||||
|
'''
|
||||||
|
bucket: int = tick // self.factor
|
||||||
|
if bucket != self.bucket:
|
||||||
|
self.bucket = bucket
|
||||||
|
self.count = 0
|
||||||
|
self.count += 1
|
||||||
|
if (tick + 1) % self.factor:
|
||||||
|
return
|
||||||
|
if self.count != self.factor:
|
||||||
|
self.discarded += 1
|
||||||
|
self.count = 0
|
||||||
|
return
|
||||||
|
|
||||||
|
end: float = (bucket + 1) * self.period_us / 1_000_000
|
||||||
|
update: TrendUpdate = self.calculator.push(
|
||||||
|
end,
|
||||||
|
close,
|
||||||
|
index=source_index,
|
||||||
|
)
|
||||||
|
self.update = update
|
||||||
|
self.close_count += 1
|
||||||
|
self.count = 0
|
||||||
|
self.closes.append(CloseSample(end, source_index, close))
|
||||||
|
event: np.void
|
||||||
|
for event in update.reversals:
|
||||||
|
self.reversals.append(TrendReversal(
|
||||||
|
index=int(event['index']),
|
||||||
|
time=float(event['time']),
|
||||||
|
price=float(event['price']),
|
||||||
|
direction=int(event['direction']),
|
||||||
|
order=int(event['order']),
|
||||||
|
detected_index=int(event['detected_index']),
|
||||||
|
detected_time=float(event['detected_time']),
|
||||||
|
))
|
||||||
|
|
||||||
|
def snapshot(self) -> TimeframeRegime:
|
||||||
|
'''
|
||||||
|
Detach immutable evidence from this writer's mutable state.
|
||||||
|
|
||||||
|
'''
|
||||||
|
update: TrendUpdate|None = self.update
|
||||||
|
size: int = len(self.calculator.windows)
|
||||||
|
return TimeframeRegime(
|
||||||
|
period=self.period_us / 1_000_000,
|
||||||
|
as_of=update.time if update else None,
|
||||||
|
close_count=self.close_count,
|
||||||
|
trends=update.trends if update else (0,) * size,
|
||||||
|
ready=update.ready if update else (False,) * size,
|
||||||
|
recent_closes=tuple(self.closes),
|
||||||
|
reversals=tuple(self.reversals),
|
||||||
|
discarded_buckets=self.discarded,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class MultiTimeframeTrend:
|
||||||
|
'''
|
||||||
|
Aggregate final base bars into causal, independent TC streams.
|
||||||
|
|
||||||
|
Call ``reset()`` and replay CLOSED source history after backfill,
|
||||||
|
corrections, reconnect/session changes, or any source revision.
|
||||||
|
Never replay the current mutable OHLC row as a final observation.
|
||||||
|
A missing base interval conservatively resets every timeframe;
|
||||||
|
a duplicate or out-of-order interval raises without mutation.
|
||||||
|
|
||||||
|
Work per update and retained memory are bounded by configured
|
||||||
|
timeframe/order counts and ``history_limit``, not stream age.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
market: str,
|
||||||
|
*,
|
||||||
|
base_period: float = 1,
|
||||||
|
periods: tuple[float, ...] = (1, 5, 15),
|
||||||
|
windows: tuple[int, ...] = (2, 2, 2),
|
||||||
|
history_limit: int = 16,
|
||||||
|
revision: int = 0,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Validate timeframe periods before allocating recurrences.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isinstance(market, str) or not market.strip():
|
||||||
|
raise ValueError('market identity is required')
|
||||||
|
if not 1 <= len(periods) <= 16:
|
||||||
|
raise ValueError('configure between 1 and 16 timeframes')
|
||||||
|
if (
|
||||||
|
isinstance(history_limit, bool)
|
||||||
|
or not 1 <= integer_index(history_limit) <= 256
|
||||||
|
):
|
||||||
|
raise ValueError('history_limit must be in [1, 256]')
|
||||||
|
if isinstance(revision, bool) or integer_index(revision) < 0:
|
||||||
|
raise ValueError('revision must be an integer >= 0')
|
||||||
|
base_us: int = _period_us(base_period)
|
||||||
|
periods_us: list[int] = []
|
||||||
|
period: float
|
||||||
|
for period in periods:
|
||||||
|
periods_us.append(_period_us(period))
|
||||||
|
if len(set(periods_us)) != len(periods_us):
|
||||||
|
raise ValueError('periods must be unique base multiples')
|
||||||
|
micros: int
|
||||||
|
for micros in periods_us:
|
||||||
|
if micros % base_us:
|
||||||
|
raise ValueError('periods need whole base multiples')
|
||||||
|
|
||||||
|
self.market: str = market
|
||||||
|
self.base_period: float = base_us / 1_000_000
|
||||||
|
self.windows: tuple[int, ...] = tuple(windows)
|
||||||
|
self.revision: int = int(revision)
|
||||||
|
self.sequence: int = 0
|
||||||
|
self.segment: int = 0
|
||||||
|
self._base_us: int = base_us
|
||||||
|
self._last_tick: int|None = None
|
||||||
|
self._last_index: int = -1
|
||||||
|
self._frames: tuple[_Timeframe, ...] = tuple(
|
||||||
|
_Timeframe(
|
||||||
|
micros,
|
||||||
|
micros // base_us,
|
||||||
|
self.windows,
|
||||||
|
history_limit,
|
||||||
|
)
|
||||||
|
for micros in sorted(periods_us)
|
||||||
|
)
|
||||||
|
# Reuse TrendCalculator.windows, already normalized to int,
|
||||||
|
# so accepted NumPy integer inputs remain JSON-serializable.
|
||||||
|
self.windows = self._frames[0].calculator.windows
|
||||||
|
|
||||||
|
def reset(self, revision: int|None = None) -> None:
|
||||||
|
'''
|
||||||
|
Invalidate prior evidence before replaying revised history.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if revision is None:
|
||||||
|
revision = self.revision + 1
|
||||||
|
if (
|
||||||
|
isinstance(revision, bool)
|
||||||
|
or integer_index(revision) <= self.revision
|
||||||
|
):
|
||||||
|
raise ValueError('revision must strictly increase')
|
||||||
|
frame: _Timeframe
|
||||||
|
for frame in self._frames:
|
||||||
|
frame.reset()
|
||||||
|
frame.discarded = 0
|
||||||
|
self.revision = int(revision)
|
||||||
|
self.sequence = 0
|
||||||
|
self.segment = 0
|
||||||
|
self._last_tick = None
|
||||||
|
self._last_index = -1
|
||||||
|
|
||||||
|
def push(
|
||||||
|
self,
|
||||||
|
time: float,
|
||||||
|
close: float,
|
||||||
|
*,
|
||||||
|
index: int|None = None,
|
||||||
|
) -> RegimeSnapshot:
|
||||||
|
'''
|
||||||
|
Commit a final close with its base-bar START timestamp.
|
||||||
|
|
||||||
|
Nonfinite data and unaligned timestamps are rejected before
|
||||||
|
mutation. Following omitted/invalid bars, the next timestamp
|
||||||
|
exposes the gap and resets all trend evidence explicitly.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isfinite(time) or not isfinite(close):
|
||||||
|
raise ValueError('bar time and close must be finite')
|
||||||
|
ticks: float = time / self.base_period
|
||||||
|
if not isfinite(ticks):
|
||||||
|
raise ValueError('bar time exceeds grid precision')
|
||||||
|
tick: int = round(ticks)
|
||||||
|
aligned: float = tick * self._base_us / 1_000_000
|
||||||
|
end: float = (tick + 1) * self._base_us / 1_000_000
|
||||||
|
if not isfinite(end) or end <= aligned:
|
||||||
|
raise ValueError('bar interval exceeds clock precision')
|
||||||
|
if not isclose(
|
||||||
|
time,
|
||||||
|
aligned,
|
||||||
|
rel_tol=0,
|
||||||
|
abs_tol=max(1e-9, 2 * ulp(time)),
|
||||||
|
):
|
||||||
|
raise ValueError('bar starts must align to base period')
|
||||||
|
if self._last_tick is not None and tick <= self._last_tick:
|
||||||
|
raise ValueError('bar starts must strictly increase')
|
||||||
|
if index is None:
|
||||||
|
index = self._last_index + 1
|
||||||
|
if isinstance(index, (bool, np.bool_)):
|
||||||
|
raise ValueError('source index needs increasing int64')
|
||||||
|
coordinate: int = integer_index(index)
|
||||||
|
if (
|
||||||
|
not 0 <= coordinate < 2 ** 63
|
||||||
|
or coordinate <= self._last_index
|
||||||
|
):
|
||||||
|
raise ValueError('source index needs increasing int64')
|
||||||
|
|
||||||
|
frame: _Timeframe
|
||||||
|
if (
|
||||||
|
self._last_tick is not None
|
||||||
|
and tick != self._last_tick + 1
|
||||||
|
):
|
||||||
|
self.segment += 1
|
||||||
|
for frame in self._frames:
|
||||||
|
frame.reset()
|
||||||
|
|
||||||
|
for frame in self._frames:
|
||||||
|
frame.push(tick, float(close), coordinate)
|
||||||
|
self._last_tick = tick
|
||||||
|
self._last_index = coordinate
|
||||||
|
self.sequence += 1
|
||||||
|
return RegimeSnapshot(
|
||||||
|
market=self.market,
|
||||||
|
revision=self.revision,
|
||||||
|
sequence=self.sequence,
|
||||||
|
segment=self.segment,
|
||||||
|
as_of=end,
|
||||||
|
base_period=self.base_period,
|
||||||
|
windows=self.windows,
|
||||||
|
timeframes=tuple(
|
||||||
|
frame.snapshot() for frame in self._frames
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
@ -0,0 +1,550 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Bridge finalized native OHLC SHM rows to regime snapshot consumers.
|
||||||
|
|
||||||
|
Sampler messages are wakeups and invalidations, not bar values.
|
||||||
|
This reader copies the authoritative source range and excludes its
|
||||||
|
mutable final row. Existing SHM has no transactional revision word:
|
||||||
|
bound checks detect prepend/truncation; in-place corrections rely
|
||||||
|
on the source's backfill notice and converge by invalidating/replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from operator import index as integer_index
|
||||||
|
from math import isfinite
|
||||||
|
import time
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import trio
|
||||||
|
from tractor.trionics import Lagged
|
||||||
|
|
||||||
|
from ._regime import MultiTimeframeTrend, RegimeSnapshot
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from tractor.ipc._shm import ShmArray
|
||||||
|
from ._jev import JevClassification, JevClient, RegimeClassifier
|
||||||
|
|
||||||
|
|
||||||
|
class SourceChanged(Exception):
|
||||||
|
'''
|
||||||
|
Signal changed shared bounds while copying a finalized prefix.
|
||||||
|
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ClosedSourceRows:
|
||||||
|
'''
|
||||||
|
Keep one detached finalized range and its absolute SHM bounds.
|
||||||
|
|
||||||
|
``ClosedSourceRows.last`` includes the mutable source row;
|
||||||
|
``ClosedSourceRows.rows`` excludes it. ``ClosedSourceRows.start``
|
||||||
|
is a physical absolute offset in ``ShmArray._array``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
first: int
|
||||||
|
last: int
|
||||||
|
start: int
|
||||||
|
rows: np.ndarray
|
||||||
|
|
||||||
|
|
||||||
|
def _bounds(shm: 'ShmArray') -> tuple[int, int]:
|
||||||
|
'''
|
||||||
|
Read absolute shared bounds, never the wrapped ``shm.index``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return int(shm._first.value), int(shm._last.value)
|
||||||
|
|
||||||
|
|
||||||
|
def copy_closed_rows(
|
||||||
|
shm: 'ShmArray',
|
||||||
|
*,
|
||||||
|
start: int|None = None,
|
||||||
|
) -> ClosedSourceRows:
|
||||||
|
'''
|
||||||
|
Copy a stable finalized range before its consumer can yield.
|
||||||
|
|
||||||
|
Invalid numeric data is rejected before recurrence mutation.
|
||||||
|
``ShmArray._array['index']`` gives absolute source coordinates.
|
||||||
|
Increasing timestamps may contain gaps: the tracker resets its
|
||||||
|
segment on those gaps. The source's mutable last row is never
|
||||||
|
validated or copied, so unfinished OHLC cannot become evidence.
|
||||||
|
Appends preserve this closed prefix. ``ClosedSourceRows.last``
|
||||||
|
retains the captured bound so ``consume_regime_source()`` can
|
||||||
|
process an appended suffix on its next pass.
|
||||||
|
|
||||||
|
'''
|
||||||
|
first: int
|
||||||
|
last: int
|
||||||
|
first, last = _bounds(shm)
|
||||||
|
raw: np.ndarray = shm._array
|
||||||
|
if not 0 <= first <= last <= len(raw):
|
||||||
|
if _bounds(shm) != (first, last):
|
||||||
|
raise SourceChanged('source bounds changed during read')
|
||||||
|
raise ValueError('source bounds are not a readable range')
|
||||||
|
stop: int = max(first, last - 1)
|
||||||
|
if start is None:
|
||||||
|
start = first
|
||||||
|
if not first <= start <= stop:
|
||||||
|
raise SourceChanged('finalized source range moved')
|
||||||
|
if not {'index', 'time', 'close'} <= set(raw.dtype.names or ()):
|
||||||
|
raise ValueError('source requires index/time/close fields')
|
||||||
|
rows: np.ndarray = raw[start:stop][[
|
||||||
|
'index', 'time', 'close',
|
||||||
|
]].copy()
|
||||||
|
current_first: int
|
||||||
|
current_last: int
|
||||||
|
current_first, current_last = _bounds(shm)
|
||||||
|
if current_first != first or current_last < last:
|
||||||
|
raise SourceChanged('source bounds changed during copy')
|
||||||
|
indices: np.ndarray = rows['index']
|
||||||
|
times: np.ndarray = rows['time']
|
||||||
|
if not np.issubdtype(indices.dtype, np.integer):
|
||||||
|
raise ValueError('source indices must be integers')
|
||||||
|
if (
|
||||||
|
np.any(indices < 0)
|
||||||
|
or np.any(indices > np.iinfo(np.int64).max)
|
||||||
|
or np.any(indices[1:] <= indices[:-1])
|
||||||
|
or not np.isfinite(times).all()
|
||||||
|
or np.any(times[1:] <= times[:-1])
|
||||||
|
or not np.isfinite(rows['close']).all()
|
||||||
|
):
|
||||||
|
raise ValueError('source has invalid finalized observations')
|
||||||
|
rows.flags.writeable = False
|
||||||
|
return ClosedSourceRows(first, last, start, rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_action(
|
||||||
|
message: dict,
|
||||||
|
market: str,
|
||||||
|
base_period: float,
|
||||||
|
) -> str:
|
||||||
|
'''
|
||||||
|
Filter global sampler broadcasts by the actual source identity.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isinstance(message, dict):
|
||||||
|
return 'ignore'
|
||||||
|
if 'backfilling' in message:
|
||||||
|
notice: object = message['backfilling']
|
||||||
|
if (
|
||||||
|
isinstance(notice, (tuple, list))
|
||||||
|
and len(notice) == 2
|
||||||
|
and notice[0] == market
|
||||||
|
and notice[1] == base_period
|
||||||
|
):
|
||||||
|
return 'revision'
|
||||||
|
return 'ignore'
|
||||||
|
if message.get('period') == base_period:
|
||||||
|
return 'step'
|
||||||
|
return 'ignore'
|
||||||
|
|
||||||
|
|
||||||
|
def source_snapshot_is_current(
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
*,
|
||||||
|
now: float,
|
||||||
|
max_source_age: float|None = 5,
|
||||||
|
) -> bool:
|
||||||
|
'''
|
||||||
|
Gate model admission by wall-clock source age, independent of UI.
|
||||||
|
|
||||||
|
Allow at most one base period of forward clock skew. ``None``
|
||||||
|
explicitly disables the clock gate for historical replay. Display
|
||||||
|
consumers can retain history while sharing this gate before Jev.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if max_source_age is None:
|
||||||
|
return True
|
||||||
|
if not isfinite(max_source_age) or max_source_age <= 0:
|
||||||
|
raise ValueError('max_source_age must be positive or None')
|
||||||
|
if not isfinite(now):
|
||||||
|
return False
|
||||||
|
age: float = now - snapshot.as_of
|
||||||
|
return -snapshot.base_period <= age <= max_source_age
|
||||||
|
|
||||||
|
|
||||||
|
class _Notices:
|
||||||
|
'''
|
||||||
|
Coalesce sampling wakeups while retaining revision invalidation.
|
||||||
|
|
||||||
|
``_Notices.collect()`` mutates only its notification flags and
|
||||||
|
``_Notices.revision``, never the TC recurrence.
|
||||||
|
``_Notices.revise()`` calls ``_Notices.invalidate`` before replay
|
||||||
|
checkpoints allow an obsolete model response to finish.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
tracker: MultiTimeframeTrend,
|
||||||
|
invalidate: Callable[[int], object]|None,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Bind a tracker revision to its synchronous invalidation hook.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.tracker: MultiTimeframeTrend = tracker
|
||||||
|
self.revision: int = tracker.revision
|
||||||
|
self.invalidate: Callable[[int], object]|None = invalidate
|
||||||
|
self.dirty: bool = True
|
||||||
|
self.closed: bool = False
|
||||||
|
self.wake: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
def revise(self) -> None:
|
||||||
|
'''
|
||||||
|
Allocate the next generation and revoke its predecessor.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.revision += 1
|
||||||
|
if self.invalidate is not None:
|
||||||
|
self.invalidate(self.revision)
|
||||||
|
self.dirty = True
|
||||||
|
self.wake.set()
|
||||||
|
|
||||||
|
async def collect(
|
||||||
|
self,
|
||||||
|
stream: AsyncIterator[dict],
|
||||||
|
*,
|
||||||
|
task_status: trio.TaskStatus[None] = (
|
||||||
|
trio.TASK_STATUS_IGNORED
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Preserve backfill/overrun notices during slow bootstrap.
|
||||||
|
|
||||||
|
'''
|
||||||
|
iterator: AsyncIterator[dict] = aiter(stream)
|
||||||
|
task_status.started()
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
message: dict = await anext(iterator)
|
||||||
|
except Lagged:
|
||||||
|
# A lost notice can describe an in-place correction;
|
||||||
|
# unchanged SHM bounds cannot prove consistency.
|
||||||
|
self.revise()
|
||||||
|
continue
|
||||||
|
except StopAsyncIteration:
|
||||||
|
self.closed = True
|
||||||
|
self.wake.set()
|
||||||
|
return
|
||||||
|
action: str = _sample_action(
|
||||||
|
message,
|
||||||
|
self.tracker.market,
|
||||||
|
self.tracker.base_period,
|
||||||
|
)
|
||||||
|
if action == 'revision':
|
||||||
|
self.revise()
|
||||||
|
elif action == 'step':
|
||||||
|
self.dirty = True
|
||||||
|
self.wake.set()
|
||||||
|
|
||||||
|
|
||||||
|
async def consume_regime_source(
|
||||||
|
shm: 'ShmArray',
|
||||||
|
sample_stream: AsyncIterator[dict],
|
||||||
|
tracker: MultiTimeframeTrend,
|
||||||
|
submit: Callable[[RegimeSnapshot], object],
|
||||||
|
*,
|
||||||
|
invalidate: Callable[[int], object]|None = None,
|
||||||
|
replay_chunk: int = 512,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Bootstrap/replay closed rows, then follow source sample advances.
|
||||||
|
|
||||||
|
Enter the source sampler subscription BEFORE calling this helper.
|
||||||
|
``submit`` is synchronous and receives only the newest complete
|
||||||
|
snapshot per captured range, not a request for each history row.
|
||||||
|
Pass ``RegimeClassifier.submit`` and its ``invalidate`` together
|
||||||
|
when using Jev. Both callbacks must return without awaiting I/O.
|
||||||
|
|
||||||
|
One compute task owns ``tracker``. A second task drains sampler
|
||||||
|
events into bounded flags during cooperative historical replay.
|
||||||
|
A prepend, truncation, relevant backfill, or sampler overrun
|
||||||
|
invalidates the previous revision and rebuilds from full history.
|
||||||
|
Cancellation closes the collector with the consumer's nursery.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
isinstance(replay_chunk, bool)
|
||||||
|
or integer_index(replay_chunk) < 1
|
||||||
|
):
|
||||||
|
raise ValueError('replay_chunk must be a positive integer')
|
||||||
|
if tracker.sequence:
|
||||||
|
raise ValueError('source consumer needs an unused tracker')
|
||||||
|
notices: _Notices = _Notices(tracker, invalidate)
|
||||||
|
prior_bounds: tuple[int, int]|None = None
|
||||||
|
copy_failures: int = 0
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
await nursery.start(notices.collect, sample_stream)
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
if not notices.dirty:
|
||||||
|
if notices.closed:
|
||||||
|
return
|
||||||
|
await notices.wake.wait()
|
||||||
|
notices.wake = trio.Event()
|
||||||
|
notices.dirty = False
|
||||||
|
revision: int = notices.revision
|
||||||
|
if revision != tracker.revision:
|
||||||
|
tracker.reset(revision=revision)
|
||||||
|
prior_bounds = None
|
||||||
|
first: int
|
||||||
|
last: int
|
||||||
|
first, last = _bounds(shm)
|
||||||
|
if prior_bounds is not None:
|
||||||
|
old_first: int
|
||||||
|
old_last: int
|
||||||
|
old_first, old_last = prior_bounds
|
||||||
|
if first != old_first or last < old_last:
|
||||||
|
notices.revise()
|
||||||
|
continue
|
||||||
|
start: int = max(first, old_last - 1)
|
||||||
|
else:
|
||||||
|
start = first
|
||||||
|
try:
|
||||||
|
source: ClosedSourceRows = copy_closed_rows(
|
||||||
|
shm, start=start,
|
||||||
|
)
|
||||||
|
except SourceChanged:
|
||||||
|
copy_failures += 1
|
||||||
|
if copy_failures >= 8:
|
||||||
|
raise SourceChanged(
|
||||||
|
'source unstable after 8 copy attempts',
|
||||||
|
) from None
|
||||||
|
notices.dirty = True
|
||||||
|
delay: float = min(.1, .001 * 2 ** copy_failures)
|
||||||
|
await trio.sleep(delay)
|
||||||
|
continue
|
||||||
|
copy_failures = 0
|
||||||
|
|
||||||
|
snapshot: RegimeSnapshot|None = None
|
||||||
|
offset: int
|
||||||
|
row: np.void
|
||||||
|
for offset, row in enumerate(source.rows, start=1):
|
||||||
|
snapshot = tracker.push(
|
||||||
|
float(row['time']),
|
||||||
|
float(row['close']),
|
||||||
|
index=int(row['index']),
|
||||||
|
)
|
||||||
|
if offset % replay_chunk == 0:
|
||||||
|
await trio.lowlevel.checkpoint()
|
||||||
|
if notices.revision != revision:
|
||||||
|
break
|
||||||
|
# Permit queued in-place invalidations to reach the
|
||||||
|
# collector before admitting bootstrap/replay output.
|
||||||
|
await trio.lowlevel.checkpoint()
|
||||||
|
if notices.revision != revision:
|
||||||
|
notices.dirty = True
|
||||||
|
continue
|
||||||
|
current_first: int
|
||||||
|
current_last: int
|
||||||
|
current_first, current_last = _bounds(shm)
|
||||||
|
if (
|
||||||
|
current_first != source.first
|
||||||
|
or current_last < source.last
|
||||||
|
):
|
||||||
|
notices.revise()
|
||||||
|
continue
|
||||||
|
prior_bounds = source.first, source.last
|
||||||
|
if current_last != source.last:
|
||||||
|
notices.dirty = True
|
||||||
|
if snapshot is not None:
|
||||||
|
submit(snapshot)
|
||||||
|
finally:
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
|
||||||
|
|
||||||
|
async def run_regime_feed(
|
||||||
|
fqme: str,
|
||||||
|
submit: Callable[[RegimeSnapshot], object],
|
||||||
|
*,
|
||||||
|
invalidate: Callable[[int], object]|None = None,
|
||||||
|
periods: tuple[float, ...]|None = None,
|
||||||
|
windows: tuple[int, ...] = (2, 2, 2),
|
||||||
|
history_limit: int = 16,
|
||||||
|
max_source_age: float|None = 5,
|
||||||
|
loglevel: str|None = None,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Follow one native realtime Flume in an existing Piker runtime.
|
||||||
|
|
||||||
|
This explicit entry point can open a broker data feed. It creates
|
||||||
|
no Jev client and reads no API key. The native sampler currently
|
||||||
|
stamps periods <=1s with integer seconds, so this wrapper refuses
|
||||||
|
subsecond source periods. A separate correct subsecond producer
|
||||||
|
can call ``consume_regime_source`` directly.
|
||||||
|
|
||||||
|
Quote delivery retains the shared feed's default policy. Forcing
|
||||||
|
a low rate here would throttle later users of its market cache;
|
||||||
|
this consumer drains quotes and derives only from finalized SHM.
|
||||||
|
|
||||||
|
``max_source_age`` bounds wall-clock age of submitted evidence,
|
||||||
|
separately from the classifier's local queue age. ``None`` allows
|
||||||
|
historical replay; the default holds stale startup history until
|
||||||
|
live bars arrive. Source clocks over one base period ahead are
|
||||||
|
rejected by the same gate when an age limit is enabled.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from ..data._sampling import open_sample_stream
|
||||||
|
from ..data.feed import Feed, maybe_open_feed
|
||||||
|
from ..data.flows import Flume
|
||||||
|
|
||||||
|
if (
|
||||||
|
max_source_age is not None
|
||||||
|
and
|
||||||
|
(
|
||||||
|
not isfinite(max_source_age)
|
||||||
|
or max_source_age <= 0
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise ValueError('max_source_age must be positive or None')
|
||||||
|
feed: Feed
|
||||||
|
async with maybe_open_feed(
|
||||||
|
fqmes=[fqme],
|
||||||
|
loglevel=loglevel,
|
||||||
|
) as feed:
|
||||||
|
flume: Flume = feed.flumes[fqme]
|
||||||
|
# maybe_open_feed() replaces the shared Flume.stream on hits.
|
||||||
|
# Capture our receiver before the sampler context can yield.
|
||||||
|
quote_stream: AsyncIterator[dict] = flume.stream
|
||||||
|
period: float = flume.rt_sample_period_s
|
||||||
|
if period < 1:
|
||||||
|
raise ValueError('native sampler requires >=1s periods')
|
||||||
|
if periods is None:
|
||||||
|
periods = (period, period * 5, period * 15)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
fqme,
|
||||||
|
base_period=period,
|
||||||
|
periods=periods,
|
||||||
|
windows=windows,
|
||||||
|
history_limit=history_limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
def submit_fresh(snapshot: RegimeSnapshot) -> None:
|
||||||
|
'''
|
||||||
|
Hold stale startup data before admitting live inference.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if source_snapshot_is_current(
|
||||||
|
snapshot,
|
||||||
|
now=time.time(),
|
||||||
|
max_source_age=max_source_age,
|
||||||
|
):
|
||||||
|
submit(snapshot)
|
||||||
|
|
||||||
|
async def drain_quotes() -> None:
|
||||||
|
'''
|
||||||
|
Keep the quote subscription healthy while consuming SHM.
|
||||||
|
|
||||||
|
'''
|
||||||
|
iterator: AsyncIterator[dict] = aiter(quote_stream)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
await anext(iterator)
|
||||||
|
except Lagged:
|
||||||
|
# Quotes are discarded; all finalized prices
|
||||||
|
# still come from source SHM and sampler events.
|
||||||
|
await trio.lowlevel.checkpoint()
|
||||||
|
except StopAsyncIteration:
|
||||||
|
return
|
||||||
|
|
||||||
|
stream: AsyncIterator[dict]
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with (
|
||||||
|
open_sample_stream(
|
||||||
|
period_s=period,
|
||||||
|
loglevel=loglevel,
|
||||||
|
) as stream,
|
||||||
|
trio.open_nursery() as nursery,
|
||||||
|
):
|
||||||
|
nursery.start_soon(drain_quotes)
|
||||||
|
try:
|
||||||
|
await consume_regime_source(
|
||||||
|
flume.rt_shm,
|
||||||
|
stream,
|
||||||
|
tracker,
|
||||||
|
submit_fresh,
|
||||||
|
invalidate=invalidate,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
|
||||||
|
|
||||||
|
async def serve_regimes(
|
||||||
|
fqme: str,
|
||||||
|
client: 'JevClient',
|
||||||
|
publish: Callable[['JevClassification'], Awaitable[None]],
|
||||||
|
*,
|
||||||
|
periods: tuple[float, ...]|None = None,
|
||||||
|
windows: tuple[int, ...] = (2, 2, 2),
|
||||||
|
history_limit: int = 16,
|
||||||
|
min_interval: float = 1,
|
||||||
|
max_age: float = 5,
|
||||||
|
max_source_lag: float = 2,
|
||||||
|
max_source_age: float|None = 5,
|
||||||
|
on_error: Callable[[Exception], Awaitable[None]]|None = None,
|
||||||
|
loglevel: str|None = None,
|
||||||
|
) -> 'RegimeClassifier':
|
||||||
|
'''
|
||||||
|
Compose a native feed and paced Jev worker in one owned nursery.
|
||||||
|
|
||||||
|
This explicit call opens a broker feed and uses the supplied Jev
|
||||||
|
client for inference. The caller owns its HTTP session and Piker
|
||||||
|
runtime. Cancellation closes the sampler subscription, quote
|
||||||
|
drain and in-flight classifier work together. Consumers validate
|
||||||
|
source provenance again when applying delivered messages.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from ._jev import RegimeClassifier
|
||||||
|
|
||||||
|
worker: RegimeClassifier = RegimeClassifier(
|
||||||
|
client,
|
||||||
|
min_interval=min_interval,
|
||||||
|
max_age=max_age,
|
||||||
|
max_source_lag=max_source_lag,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def classify() -> None:
|
||||||
|
'''
|
||||||
|
Bind the optional error callback to this service lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
await worker.run(publish, on_error=on_error)
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(classify)
|
||||||
|
try:
|
||||||
|
await run_regime_feed(
|
||||||
|
fqme,
|
||||||
|
worker.submit,
|
||||||
|
invalidate=worker.invalidate,
|
||||||
|
periods=periods,
|
||||||
|
windows=windows,
|
||||||
|
history_limit=history_limit,
|
||||||
|
max_source_age=max_source_age,
|
||||||
|
loglevel=loglevel,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
return worker
|
||||||
|
|
@ -0,0 +1,591 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
# Algorithm: Andrew J. Morgan / ByteSumo, TrendCalculus (GPLv3+).
|
||||||
|
# See docs/trend_calculus.rst for sources and semantic differences.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Causal, bounded-state Trend Calculus for scalar observations.
|
||||||
|
|
||||||
|
The same compiled recurrence drives historical and live processing.
|
||||||
|
FHLS means first/high/low/second *extremum*, not open/high/low/close.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from math import isfinite
|
||||||
|
from operator import index as integer_index
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from numba import njit
|
||||||
|
|
||||||
|
|
||||||
|
_bar_dtype: np.dtype[np.void] = np.dtype([
|
||||||
|
('high', '<f8'),
|
||||||
|
('low', '<f8'),
|
||||||
|
('hi_time', '<f8'),
|
||||||
|
('lo_time', '<f8'),
|
||||||
|
('hi_index', '<i8'),
|
||||||
|
('lo_index', '<i8'),
|
||||||
|
])
|
||||||
|
|
||||||
|
reversal_dtype: np.dtype[np.void] = np.dtype([
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('price', '<f8'),
|
||||||
|
('direction', 'i1'),
|
||||||
|
('order', '<i8'),
|
||||||
|
('detected_index', '<i8'),
|
||||||
|
('detected_time', '<f8'),
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
@njit(cache=True)
|
||||||
|
def _direction(
|
||||||
|
high: float,
|
||||||
|
low: float,
|
||||||
|
prev_high: float,
|
||||||
|
prev_low: float,
|
||||||
|
) -> int:
|
||||||
|
'''
|
||||||
|
Evaluate Morgan's sign sum without overflowing a subtraction.
|
||||||
|
|
||||||
|
'''
|
||||||
|
delta: int = (
|
||||||
|
int(high > prev_high) - int(high < prev_high)
|
||||||
|
+ int(low > prev_low) - int(low < prev_low)
|
||||||
|
)
|
||||||
|
return int(delta > 0) - int(delta < 0)
|
||||||
|
|
||||||
|
|
||||||
|
@njit(cache=True)
|
||||||
|
def _reversal(
|
||||||
|
previous: np.void,
|
||||||
|
old_trend: int,
|
||||||
|
new_trend: int,
|
||||||
|
level: int,
|
||||||
|
index: int,
|
||||||
|
time: float,
|
||||||
|
events: np.ndarray,
|
||||||
|
counts: np.ndarray,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Emit a confirmed sign change at its detection observation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if old_trend * new_trend != -1:
|
||||||
|
return
|
||||||
|
|
||||||
|
slot: int = counts[level]
|
||||||
|
event: np.void = events[level, slot]
|
||||||
|
if new_trend == 1:
|
||||||
|
event['index'] = previous['lo_index']
|
||||||
|
event['time'] = previous['lo_time']
|
||||||
|
event['price'] = previous['low']
|
||||||
|
else:
|
||||||
|
event['index'] = previous['hi_index']
|
||||||
|
event['time'] = previous['hi_time']
|
||||||
|
event['price'] = previous['high']
|
||||||
|
|
||||||
|
event['direction'] = new_trend
|
||||||
|
event['order'] = level + 1
|
||||||
|
event['detected_index'] = index
|
||||||
|
event['detected_time'] = time
|
||||||
|
counts[level] += 1
|
||||||
|
|
||||||
|
|
||||||
|
@njit(cache=True)
|
||||||
|
def _step(
|
||||||
|
bars: np.ndarray,
|
||||||
|
counters: np.ndarray,
|
||||||
|
sizes: np.ndarray,
|
||||||
|
index: int,
|
||||||
|
time: float,
|
||||||
|
value: float,
|
||||||
|
events: np.ndarray,
|
||||||
|
counts: np.ndarray,
|
||||||
|
lua_seed: bool,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Advance each reversal order using at most two incoming points.
|
||||||
|
|
||||||
|
The ``TrendCalculator._bars`` argument uses ``bars[:, 0]`` for
|
||||||
|
the current window and ``bars[:, 1]`` for its completed
|
||||||
|
predecessor. ``TrendCalculator._counters`` stores pending sample
|
||||||
|
count, completed window count, and confirmed direction.
|
||||||
|
Higher orders consume only the preceding order's reversals.
|
||||||
|
|
||||||
|
'''
|
||||||
|
counts[:] = 0
|
||||||
|
if np.isnan(value):
|
||||||
|
bars[:] = np.zeros_like(bars)
|
||||||
|
counters[:] = 0
|
||||||
|
return
|
||||||
|
|
||||||
|
bridge: np.ndarray = np.empty(1, dtype=_bar_dtype)
|
||||||
|
level: int
|
||||||
|
for level in range(len(sizes)):
|
||||||
|
incoming: int = 1 if level == 0 else counts[level - 1]
|
||||||
|
if incoming == 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
point: int
|
||||||
|
for point in range(incoming):
|
||||||
|
price: float
|
||||||
|
point_time: float
|
||||||
|
point_index: int
|
||||||
|
if level == 0:
|
||||||
|
price = value
|
||||||
|
point_time = time
|
||||||
|
point_index = index
|
||||||
|
else:
|
||||||
|
event: np.void = events[level - 1, point]
|
||||||
|
price = event['price']
|
||||||
|
point_time = event['time']
|
||||||
|
point_index = event['index']
|
||||||
|
|
||||||
|
current: np.void = bars[level, 0]
|
||||||
|
previous: np.void = bars[level, 1]
|
||||||
|
if counters[level, 0] == 0:
|
||||||
|
current['high'] = current['low'] = price
|
||||||
|
current['hi_time'] = current['lo_time'] = point_time
|
||||||
|
current['hi_index'] = point_index
|
||||||
|
current['lo_index'] = point_index
|
||||||
|
|
||||||
|
elif price > current['high']:
|
||||||
|
current['high'] = price
|
||||||
|
current['hi_time'] = point_time
|
||||||
|
current['hi_index'] = point_index
|
||||||
|
elif price < current['low']:
|
||||||
|
current['low'] = price
|
||||||
|
current['lo_time'] = point_time
|
||||||
|
current['lo_index'] = point_index
|
||||||
|
|
||||||
|
counters[level, 0] += 1
|
||||||
|
if counters[level, 0] < sizes[level]:
|
||||||
|
continue
|
||||||
|
|
||||||
|
counters[level, 0] = 0
|
||||||
|
counters[level, 1] += 1
|
||||||
|
if counters[level, 1] == 1:
|
||||||
|
if lua_seed:
|
||||||
|
# Lua compares its first window with its final
|
||||||
|
# point. Native startup leaves direction unknown.
|
||||||
|
previous['high'] = previous['low'] = price
|
||||||
|
previous['hi_time'] = point_time
|
||||||
|
previous['lo_time'] = point_time
|
||||||
|
previous['hi_index'] = point_index
|
||||||
|
previous['lo_index'] = point_index
|
||||||
|
|
||||||
|
else:
|
||||||
|
bars[level, 1] = current
|
||||||
|
continue
|
||||||
|
|
||||||
|
trend: int = _direction(
|
||||||
|
current['high'], current['low'],
|
||||||
|
previous['high'], previous['low'],
|
||||||
|
)
|
||||||
|
old_trend: int = counters[level, 2]
|
||||||
|
if trend == 0:
|
||||||
|
# Bridge the later previous extremum to the earlier
|
||||||
|
# current extremum. Preserve both possible reversals.
|
||||||
|
first_price: float
|
||||||
|
first_time: float
|
||||||
|
first_index: int
|
||||||
|
if previous['hi_index'] > previous['lo_index']:
|
||||||
|
first_price = previous['high']
|
||||||
|
first_time = previous['hi_time']
|
||||||
|
first_index = previous['hi_index']
|
||||||
|
|
||||||
|
else:
|
||||||
|
first_price = previous['low']
|
||||||
|
first_time = previous['lo_time']
|
||||||
|
first_index = previous['lo_index']
|
||||||
|
|
||||||
|
second_price: float
|
||||||
|
second_time: float
|
||||||
|
second_index: int
|
||||||
|
if current['hi_index'] < current['lo_index']:
|
||||||
|
second_price = current['high']
|
||||||
|
second_time = current['hi_time']
|
||||||
|
second_index = current['hi_index']
|
||||||
|
|
||||||
|
else:
|
||||||
|
second_price = current['low']
|
||||||
|
second_time = current['lo_time']
|
||||||
|
second_index = current['lo_index']
|
||||||
|
|
||||||
|
split: np.void = bridge[0]
|
||||||
|
if first_price > second_price:
|
||||||
|
split['high'] = first_price
|
||||||
|
split['hi_time'] = first_time
|
||||||
|
split['hi_index'] = first_index
|
||||||
|
split['low'] = second_price
|
||||||
|
split['lo_time'] = second_time
|
||||||
|
split['lo_index'] = second_index
|
||||||
|
|
||||||
|
else:
|
||||||
|
split['high'] = second_price
|
||||||
|
split['hi_time'] = second_time
|
||||||
|
split['hi_index'] = second_index
|
||||||
|
split['low'] = first_price
|
||||||
|
split['lo_time'] = first_time
|
||||||
|
split['lo_index'] = first_index
|
||||||
|
|
||||||
|
split_trend: int = _direction(
|
||||||
|
split['high'], split['low'],
|
||||||
|
previous['high'], previous['low'],
|
||||||
|
) or old_trend
|
||||||
|
_reversal(
|
||||||
|
previous, old_trend, split_trend,
|
||||||
|
level, index, time, events, counts,
|
||||||
|
)
|
||||||
|
trend = _direction(
|
||||||
|
current['high'], current['low'],
|
||||||
|
split['high'], split['low'],
|
||||||
|
) or split_trend
|
||||||
|
_reversal(
|
||||||
|
split, split_trend, trend,
|
||||||
|
level, index, time, events, counts,
|
||||||
|
)
|
||||||
|
|
||||||
|
else:
|
||||||
|
_reversal(
|
||||||
|
previous, old_trend, trend,
|
||||||
|
level, index, time, events, counts,
|
||||||
|
)
|
||||||
|
|
||||||
|
counters[level, 2] = trend
|
||||||
|
bars[level, 1] = current
|
||||||
|
|
||||||
|
|
||||||
|
@njit(cache=True)
|
||||||
|
def _scan(
|
||||||
|
values: np.ndarray,
|
||||||
|
times: np.ndarray,
|
||||||
|
indices: np.ndarray,
|
||||||
|
bars: np.ndarray,
|
||||||
|
counters: np.ndarray,
|
||||||
|
sizes: np.ndarray,
|
||||||
|
lua_seed: bool,
|
||||||
|
) -> tuple[
|
||||||
|
np.ndarray,
|
||||||
|
np.ndarray,
|
||||||
|
np.ndarray,
|
||||||
|
np.ndarray,
|
||||||
|
np.ndarray,
|
||||||
|
]:
|
||||||
|
'''
|
||||||
|
Bootstrap with ``_step()``, retaining causal detection pulses.
|
||||||
|
|
||||||
|
'''
|
||||||
|
length: int = len(values)
|
||||||
|
levels: int = len(sizes)
|
||||||
|
trends: np.ndarray = np.zeros((length, levels), dtype=np.int8)
|
||||||
|
ready: np.ndarray = np.zeros((length, levels), dtype=np.bool_)
|
||||||
|
pulses: np.ndarray = np.zeros((length, levels), dtype=np.int8)
|
||||||
|
pivot_times: np.ndarray = np.full((length, levels, 2), np.nan)
|
||||||
|
pivot_prices: np.ndarray = np.full((length, levels, 2), np.nan)
|
||||||
|
events: np.ndarray = np.empty((levels, 2), dtype=reversal_dtype)
|
||||||
|
counts: np.ndarray = np.zeros(levels, dtype=np.int64)
|
||||||
|
row: int
|
||||||
|
for row in range(length):
|
||||||
|
_step(
|
||||||
|
bars, counters, sizes,
|
||||||
|
indices[row], times[row], values[row],
|
||||||
|
events, counts, lua_seed,
|
||||||
|
)
|
||||||
|
level: int
|
||||||
|
for level in range(levels):
|
||||||
|
ready[row, level] = counters[level, 1] >= 2
|
||||||
|
if ready[row, level]:
|
||||||
|
trends[row, level] = counters[level, 2]
|
||||||
|
if counts[level]:
|
||||||
|
pulses[row, level] = events[
|
||||||
|
level, counts[level] - 1
|
||||||
|
]['direction']
|
||||||
|
slot: int
|
||||||
|
for slot in range(counts[level]):
|
||||||
|
event: np.void = events[level, slot]
|
||||||
|
side: int = int(event['direction'] > 0)
|
||||||
|
pivot_times[row, level, side] = event['time']
|
||||||
|
pivot_prices[row, level, side] = event['price']
|
||||||
|
|
||||||
|
return trends, ready, pulses, pivot_times, pivot_prices
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TrendUpdate:
|
||||||
|
'''
|
||||||
|
A causal observation: directions and newly confirmed reversals.
|
||||||
|
|
||||||
|
``TrendUpdate.ready`` means two complete windows, not statistical
|
||||||
|
confidence. A ready zero direction means unresolved flat input.
|
||||||
|
``TrendUpdate.reversals`` has separate pivot and detection
|
||||||
|
coordinates and can contain two opposing events at the same
|
||||||
|
order and detection time.
|
||||||
|
|
||||||
|
'''
|
||||||
|
index: int
|
||||||
|
time: float
|
||||||
|
trends: tuple[int, ...]
|
||||||
|
ready: tuple[bool, ...]
|
||||||
|
reversals: np.ndarray
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TrendHistory:
|
||||||
|
'''
|
||||||
|
Prefix-causal history, indexed by observation and reversal order.
|
||||||
|
|
||||||
|
``TrendHistory.pulses`` records the last reversal direction
|
||||||
|
detected per row and order. Use ``TrendCalculator.push()`` to
|
||||||
|
retain every event; two reversals at one detection row cannot
|
||||||
|
fit in one pulse. ``TrendHistory.pivot_times`` and
|
||||||
|
``TrendHistory.pivot_prices`` retain both sides with shape
|
||||||
|
(observations, orders, 2), where side 0 is top and 1 is bottom.
|
||||||
|
Absent events are NaN, never backdated into earlier output rows.
|
||||||
|
|
||||||
|
'''
|
||||||
|
trends: np.ndarray
|
||||||
|
ready: np.ndarray
|
||||||
|
pulses: np.ndarray
|
||||||
|
pivot_times: np.ndarray
|
||||||
|
pivot_prices: np.ndarray
|
||||||
|
|
||||||
|
|
||||||
|
class TrendCalculator:
|
||||||
|
'''
|
||||||
|
Track fixed-size FHLS windows and their recursive reversals.
|
||||||
|
|
||||||
|
Each window size counts observations at that order; it is not a
|
||||||
|
duration. Scalar input avoids inventing intrabar high/low order.
|
||||||
|
Equal extrema retain their earliest observation (Lua semantics).
|
||||||
|
Native startup requires two windows and emits no artificial
|
||||||
|
initial pivot. ``lua_seed=True`` uses the Lua first-window seed
|
||||||
|
for research; it does not reproduce its output suppression bugs.
|
||||||
|
NaN resets all orders. Invalid or non-increasing coordinates
|
||||||
|
and infinite values are rejected before state mutation.
|
||||||
|
|
||||||
|
State size is O(number of orders), independent of window sizes
|
||||||
|
and history length. Instances have one writer and are not shared
|
||||||
|
between markets or timeframes.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
windows: tuple[int, ...] = (2, 2, 2, 2),
|
||||||
|
*,
|
||||||
|
lua_seed: bool = False,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Allocate bounded state for each validated window size.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not windows:
|
||||||
|
raise ValueError('at least one window size is required')
|
||||||
|
sizes: list[int] = []
|
||||||
|
size: int
|
||||||
|
for size in windows:
|
||||||
|
if isinstance(size, (bool, np.bool_)):
|
||||||
|
raise ValueError(
|
||||||
|
'window size must be an integer >= 2',
|
||||||
|
)
|
||||||
|
size = integer_index(size)
|
||||||
|
if not 2 <= size <= np.iinfo(np.int64).max:
|
||||||
|
raise ValueError(
|
||||||
|
'window size exceeds positive int64',
|
||||||
|
)
|
||||||
|
sizes.append(size)
|
||||||
|
self.windows: tuple[int, ...] = tuple(sizes)
|
||||||
|
self._sizes: np.ndarray = np.array(sizes, dtype=np.int64)
|
||||||
|
self.lua_seed: bool = lua_seed
|
||||||
|
self._bars: np.ndarray = np.zeros(
|
||||||
|
(len(windows), 2), dtype=_bar_dtype,
|
||||||
|
)
|
||||||
|
self._counters: np.ndarray = np.zeros(
|
||||||
|
(len(windows), 3), dtype=np.int64,
|
||||||
|
)
|
||||||
|
self._events: np.ndarray = np.empty(
|
||||||
|
(len(windows), 2), dtype=reversal_dtype,
|
||||||
|
)
|
||||||
|
self._counts: np.ndarray = np.zeros(
|
||||||
|
len(windows), dtype=np.int64,
|
||||||
|
)
|
||||||
|
self._last_time: float = -np.inf
|
||||||
|
self._last_index: int = -1
|
||||||
|
|
||||||
|
def copy(self) -> 'TrendCalculator':
|
||||||
|
'''
|
||||||
|
Copy bounded recurrence state for an uncommitted preview.
|
||||||
|
|
||||||
|
Duplicate ``TrendCalculator._bars`` and
|
||||||
|
``TrendCalculator._counters`` with the last source
|
||||||
|
coordinates. Emitted events remain owned by each instance.
|
||||||
|
|
||||||
|
'''
|
||||||
|
clone: TrendCalculator = TrendCalculator(
|
||||||
|
self.windows, lua_seed=self.lua_seed,
|
||||||
|
)
|
||||||
|
clone._bars[:] = self._bars
|
||||||
|
clone._counters[:] = self._counters
|
||||||
|
clone._last_time = self._last_time
|
||||||
|
clone._last_index = self._last_index
|
||||||
|
return clone
|
||||||
|
|
||||||
|
def reset(self) -> None:
|
||||||
|
'''
|
||||||
|
Start a new session or revision with no inherited trend.
|
||||||
|
|
||||||
|
Clear ``TrendCalculator._bars``,
|
||||||
|
``TrendCalculator._counters`` and
|
||||||
|
``TrendCalculator._counts``. Release source constraints in
|
||||||
|
``TrendCalculator._last_time`` and
|
||||||
|
``TrendCalculator._last_index``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._bars[:] = np.zeros_like(self._bars)
|
||||||
|
self._counters[:] = 0
|
||||||
|
self._counts[:] = 0
|
||||||
|
self._last_time = -np.inf
|
||||||
|
self._last_index = -1
|
||||||
|
|
||||||
|
def _validate(
|
||||||
|
self,
|
||||||
|
index: int,
|
||||||
|
time: float,
|
||||||
|
value: float,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Refuse ambiguous ordering without consuming an observation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not isfinite(time) or time <= self._last_time:
|
||||||
|
raise ValueError('time must be finite and increasing')
|
||||||
|
if not 0 <= index <= np.iinfo(np.int64).max:
|
||||||
|
raise ValueError('index must fit nonnegative int64')
|
||||||
|
if index <= self._last_index:
|
||||||
|
raise ValueError('index must be increasing')
|
||||||
|
if np.isinf(value):
|
||||||
|
raise ValueError('infinite values are not observations')
|
||||||
|
|
||||||
|
def push(
|
||||||
|
self,
|
||||||
|
time: float,
|
||||||
|
value: float,
|
||||||
|
*,
|
||||||
|
index: int|None = None,
|
||||||
|
) -> TrendUpdate:
|
||||||
|
'''
|
||||||
|
Commit one scalar sample and return its causal output.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if index is None:
|
||||||
|
index = self._last_index + 1
|
||||||
|
if isinstance(index, (bool, np.bool_)):
|
||||||
|
raise ValueError('index must be a nonnegative integer')
|
||||||
|
index = integer_index(index)
|
||||||
|
self._validate(index, time, value)
|
||||||
|
_step(
|
||||||
|
self._bars, self._counters, self._sizes,
|
||||||
|
index, time, value, self._events, self._counts,
|
||||||
|
self.lua_seed,
|
||||||
|
)
|
||||||
|
self._last_time = time
|
||||||
|
self._last_index = index
|
||||||
|
ready: np.ndarray = self._counters[:, 1] >= 2
|
||||||
|
trends: np.ndarray = np.where(
|
||||||
|
ready, self._counters[:, 2], 0,
|
||||||
|
)
|
||||||
|
emitted: np.ndarray = (
|
||||||
|
np.arange(2)[None, :] < self._counts[:, None]
|
||||||
|
)
|
||||||
|
events: np.ndarray = self._events[emitted]
|
||||||
|
return TrendUpdate(
|
||||||
|
index=index,
|
||||||
|
time=time,
|
||||||
|
trends=tuple(map(int, trends)),
|
||||||
|
ready=tuple(map(bool, ready)),
|
||||||
|
reversals=events,
|
||||||
|
)
|
||||||
|
|
||||||
|
def extend(
|
||||||
|
self,
|
||||||
|
values: np.ndarray,
|
||||||
|
times: np.ndarray,
|
||||||
|
indices: np.ndarray|None = None,
|
||||||
|
) -> TrendHistory:
|
||||||
|
'''
|
||||||
|
Consume a batch with the same live recurrence semantics.
|
||||||
|
|
||||||
|
All coordinates are validated before the batch mutates state.
|
||||||
|
Incomplete windows survive across calls and process restarts
|
||||||
|
should rebuild them by replaying the same source revision.
|
||||||
|
|
||||||
|
'''
|
||||||
|
values = np.asarray(values, dtype=np.float64)
|
||||||
|
times = np.asarray(times, dtype=np.float64)
|
||||||
|
if values.ndim != 1 or times.ndim != 1:
|
||||||
|
raise ValueError('values and times must be 1D')
|
||||||
|
if indices is None:
|
||||||
|
limit: int = np.iinfo(np.int64).max
|
||||||
|
if self._last_index + len(values) > limit:
|
||||||
|
raise ValueError('automatic indices overflow int64')
|
||||||
|
indices = np.arange(
|
||||||
|
self._last_index + 1,
|
||||||
|
self._last_index + 1 + len(values),
|
||||||
|
dtype=np.int64,
|
||||||
|
)
|
||||||
|
indices = np.asarray(indices)
|
||||||
|
if (
|
||||||
|
indices.ndim != 1
|
||||||
|
or len(values) != len(times)
|
||||||
|
or len(values) != len(indices)
|
||||||
|
):
|
||||||
|
raise ValueError('values, times, indices must align')
|
||||||
|
if not np.issubdtype(indices.dtype, np.signedinteger):
|
||||||
|
raise ValueError('indices require signed integer dtype')
|
||||||
|
if len(values):
|
||||||
|
self._validate(int(indices[0]), times[0], values[0])
|
||||||
|
if (
|
||||||
|
not np.isfinite(times).all()
|
||||||
|
or np.isinf(values).any()
|
||||||
|
or np.any(times[1:] <= times[:-1])
|
||||||
|
or np.any(indices[1:] <= indices[:-1])
|
||||||
|
):
|
||||||
|
raise ValueError('invalid value or non-increasing batch')
|
||||||
|
trends: np.ndarray
|
||||||
|
ready: np.ndarray
|
||||||
|
pulses: np.ndarray
|
||||||
|
pivot_times: np.ndarray
|
||||||
|
pivot_prices: np.ndarray
|
||||||
|
(
|
||||||
|
trends,
|
||||||
|
ready,
|
||||||
|
pulses,
|
||||||
|
pivot_times,
|
||||||
|
pivot_prices,
|
||||||
|
) = _scan(
|
||||||
|
values, times, indices, self._bars, self._counters,
|
||||||
|
self._sizes, self.lua_seed,
|
||||||
|
)
|
||||||
|
if len(values):
|
||||||
|
self._last_time = float(times[-1])
|
||||||
|
self._last_index = int(indices[-1])
|
||||||
|
return TrendHistory(
|
||||||
|
trends, ready, pulses, pivot_times, pivot_prices,
|
||||||
|
)
|
||||||
|
|
@ -0,0 +1,208 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Trend Calculus over Piker's mutable close-price SHM source.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from tractor.ipc._shm import ShmArray
|
||||||
|
|
||||||
|
from ..data.ticktools import FeedQuote
|
||||||
|
from ._api import FspSample, FspStream, fsp
|
||||||
|
from ._trend import TrendCalculator, TrendHistory, TrendUpdate
|
||||||
|
|
||||||
|
|
||||||
|
_trend_fields: tuple[str, ...] = (
|
||||||
|
'trend',
|
||||||
|
'trend_2',
|
||||||
|
'trend_3',
|
||||||
|
'trend_4',
|
||||||
|
)
|
||||||
|
order: int
|
||||||
|
field: str
|
||||||
|
_outputs: tuple[str, ...] = _trend_fields + ('ready',) + tuple(
|
||||||
|
field
|
||||||
|
for order in range(1, 5)
|
||||||
|
for field in (
|
||||||
|
f'top_{order}',
|
||||||
|
f'top_time_{order}',
|
||||||
|
f'bottom_{order}',
|
||||||
|
f'bottom_time_{order}',
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _history_fields(
|
||||||
|
output: TrendHistory,
|
||||||
|
) -> dict[str, np.ndarray]:
|
||||||
|
'''
|
||||||
|
Convert causal arrays to the fixed SHM field schema.
|
||||||
|
|
||||||
|
'''
|
||||||
|
fields: dict[str, np.ndarray] = {}
|
||||||
|
level: int
|
||||||
|
field: str
|
||||||
|
for level, field in enumerate(_trend_fields):
|
||||||
|
fields[field] = output.trends[:, level].astype(np.float64)
|
||||||
|
fields['ready'] = output.ready.sum(axis=1).astype(np.float64)
|
||||||
|
level: int
|
||||||
|
for level in range(4):
|
||||||
|
side: int
|
||||||
|
label: str
|
||||||
|
for side, label in enumerate(('top', 'bottom')):
|
||||||
|
order: int = level + 1
|
||||||
|
fields[f'{label}_{order}'] = (
|
||||||
|
output.pivot_prices[:, level, side]
|
||||||
|
)
|
||||||
|
fields[f'{label}_time_{order}'] = (
|
||||||
|
output.pivot_times[:, level, side]
|
||||||
|
)
|
||||||
|
return fields
|
||||||
|
|
||||||
|
|
||||||
|
@fsp(
|
||||||
|
outputs=_outputs,
|
||||||
|
sample_on_step=True,
|
||||||
|
event_outputs=_outputs[5:],
|
||||||
|
)
|
||||||
|
async def trend_calculus(
|
||||||
|
source: AsyncIterator[FeedQuote],
|
||||||
|
ohlcv: ShmArray,
|
||||||
|
window: int = 2,
|
||||||
|
) -> FspStream:
|
||||||
|
'''
|
||||||
|
Publish four causal reversal orders from scalar bar closes.
|
||||||
|
|
||||||
|
``window`` counts first-order closes. Each higher order groups
|
||||||
|
two reversals from the preceding order into an FHLS window.
|
||||||
|
History uses finalized closes except for its mutable last row.
|
||||||
|
Live quotes preview that row from a copy of committed state;
|
||||||
|
repeated updates never consume a bar twice. Source advancement
|
||||||
|
commits all newly finalized bars, including skipped quote steps.
|
||||||
|
``ready`` counts orders having two completed FHLS windows.
|
||||||
|
``top_N``/``bottom_N`` are pivot prices at their detection row;
|
||||||
|
matching ``*_time_N`` fields retain the actual extremum time.
|
||||||
|
NaN means no pivot. Both sides survive split-window reversals.
|
||||||
|
|
||||||
|
A source prepend or in-place history revision restarts this
|
||||||
|
generator through ``Cascade.resync()``. Chart timestamps remain
|
||||||
|
source bar-open times, and the last row is always provisional.
|
||||||
|
For service consumption use finalized observations and explicit
|
||||||
|
bar-close/detection times in ``TrendCalculator`` instead.
|
||||||
|
|
||||||
|
'''
|
||||||
|
calc: TrendCalculator = TrendCalculator((window, 2, 2, 2))
|
||||||
|
# The engine supplies an immutable bootstrap view. Retain its
|
||||||
|
# absolute coordinates; live shared bounds may already differ.
|
||||||
|
history: np.ndarray = ohlcv.array.copy()
|
||||||
|
first: int = int(history['index'][0]) if len(history) else -1
|
||||||
|
committed: int = first - 1
|
||||||
|
output: TrendHistory = calc.extend(
|
||||||
|
history['close'][:-1],
|
||||||
|
history['time'][:-1],
|
||||||
|
history['index'][:-1],
|
||||||
|
)
|
||||||
|
if len(history) > 1:
|
||||||
|
committed = int(history['index'][-2])
|
||||||
|
|
||||||
|
tail: TrendHistory = calc.copy().extend(
|
||||||
|
history['close'][-1:],
|
||||||
|
history['time'][-1:],
|
||||||
|
history['index'][-1:],
|
||||||
|
)
|
||||||
|
fields: dict[str, np.ndarray] = _history_fields(output)
|
||||||
|
tail_fields: dict[str, np.ndarray] = _history_fields(tail)
|
||||||
|
field: str
|
||||||
|
for field in _outputs:
|
||||||
|
fields[field] = np.concatenate((
|
||||||
|
fields[field], tail_fields[field],
|
||||||
|
))
|
||||||
|
yield fields
|
||||||
|
|
||||||
|
quote: FeedQuote
|
||||||
|
async for quote in source:
|
||||||
|
# Quotes are wakeups. The source SHM is authoritative, and a
|
||||||
|
# quote frame may contain multiple already aggregated trades.
|
||||||
|
rows: np.ndarray = ohlcv.array
|
||||||
|
if not len(rows):
|
||||||
|
continue
|
||||||
|
source_first: int = int(rows['index'][0])
|
||||||
|
last: int = int(rows['index'][-1])
|
||||||
|
if first == -1:
|
||||||
|
first = source_first
|
||||||
|
committed = first - 1
|
||||||
|
if source_first != first or last <= committed:
|
||||||
|
# Normally the engine restarts first. Avoid using a
|
||||||
|
# recurrence with a different origin during the handoff.
|
||||||
|
return
|
||||||
|
|
||||||
|
begin: int = committed - first + 1
|
||||||
|
closed: np.ndarray = rows[begin:-1].copy()
|
||||||
|
current: np.void = rows[-1].copy()
|
||||||
|
if len(closed):
|
||||||
|
finalized: TrendHistory = calc.extend(
|
||||||
|
closed['close'], closed['time'], closed['index'],
|
||||||
|
)
|
||||||
|
committed = int(closed['index'][-1])
|
||||||
|
# Publish every finalized row, not only the new tail.
|
||||||
|
# Missing quote wakeups must not discard reversal events
|
||||||
|
# or leave a prior row with its last provisional close.
|
||||||
|
closed_fields: dict[str, np.ndarray] = (
|
||||||
|
_history_fields(finalized)
|
||||||
|
)
|
||||||
|
offset: int
|
||||||
|
row: np.void
|
||||||
|
for offset, row in enumerate(closed):
|
||||||
|
final_values: dict[str, float] = {}
|
||||||
|
field: str
|
||||||
|
for field in _outputs:
|
||||||
|
final_values[field] = float(
|
||||||
|
closed_fields[field][offset],
|
||||||
|
)
|
||||||
|
yield FspSample(
|
||||||
|
index=int(row['index']),
|
||||||
|
time=float(row['time']),
|
||||||
|
fields=final_values,
|
||||||
|
)
|
||||||
|
|
||||||
|
preview: TrendUpdate = calc.copy().push(
|
||||||
|
float(current['time']),
|
||||||
|
float(current['close']),
|
||||||
|
index=int(current['index']),
|
||||||
|
)
|
||||||
|
values: dict[str, int|float] = dict.fromkeys(
|
||||||
|
_outputs, np.nan,
|
||||||
|
)
|
||||||
|
values.update(zip(_trend_fields, preview.trends))
|
||||||
|
values['ready'] = sum(preview.ready)
|
||||||
|
event: np.void
|
||||||
|
label: str
|
||||||
|
order: int
|
||||||
|
for event in preview.reversals:
|
||||||
|
label = 'bottom' if event['direction'] > 0 else 'top'
|
||||||
|
order = int(event['order'])
|
||||||
|
values[f'{label}_{order}'] = float(event['price'])
|
||||||
|
values[f'{label}_time_{order}'] = float(event['time'])
|
||||||
|
yield FspSample(
|
||||||
|
index=preview.index,
|
||||||
|
time=preview.time,
|
||||||
|
fields=values,
|
||||||
|
)
|
||||||
|
|
@ -71,6 +71,7 @@ from ._l1 import L1Labels
|
||||||
from ._style import hcolor
|
from ._style import hcolor
|
||||||
from ._fsp import (
|
from ._fsp import (
|
||||||
update_fsp_chart,
|
update_fsp_chart,
|
||||||
|
update_trend_overlays,
|
||||||
start_fsp_displays,
|
start_fsp_displays,
|
||||||
open_vlm_displays,
|
open_vlm_displays,
|
||||||
)
|
)
|
||||||
|
|
@ -285,6 +286,41 @@ async def increment_history_view(
|
||||||
|
|
||||||
# check if slow chart needs an x-domain shift and/or
|
# check if slow chart needs an x-domain shift and/or
|
||||||
# y-range resize.
|
# y-range resize.
|
||||||
|
if isinstance(msg, dict):
|
||||||
|
backfill: tuple[str, float]|None = msg.get(
|
||||||
|
'backfilling',
|
||||||
|
)
|
||||||
|
if backfill is not None:
|
||||||
|
market: str
|
||||||
|
period_s: float
|
||||||
|
market, period_s = backfill
|
||||||
|
if market == ds.fqme:
|
||||||
|
changed_chart: ChartPlotWidget
|
||||||
|
for changed_chart in (
|
||||||
|
ds.chart, ds.hist_chart,
|
||||||
|
):
|
||||||
|
hidden: bool = (
|
||||||
|
changed_chart.linked.isHidden()
|
||||||
|
)
|
||||||
|
update_trend_overlays(
|
||||||
|
changed_chart,
|
||||||
|
fqme=market,
|
||||||
|
chart_period_s=period_s,
|
||||||
|
force=True,
|
||||||
|
render=not hidden,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sampler advancement can finalize an FSP row without a
|
||||||
|
# quote. A writer publishing after this callback appears
|
||||||
|
# on the next tick; `TrendOverlay._signature` avoids a
|
||||||
|
# redundant rebuild in `TrendOverlay.update()`.
|
||||||
|
trend_chart: ChartPlotWidget
|
||||||
|
for trend_chart in (ds.chart, ds.hist_chart):
|
||||||
|
update_trend_overlays(
|
||||||
|
trend_chart,
|
||||||
|
fqme=ds.fqme,
|
||||||
|
render=not trend_chart.linked.isHidden(),
|
||||||
|
)
|
||||||
(
|
(
|
||||||
uppx,
|
uppx,
|
||||||
liv,
|
liv,
|
||||||
|
|
@ -625,6 +661,17 @@ def graphics_update_cycle(
|
||||||
chart = ds.chart
|
chart = ds.chart
|
||||||
vlm_chart = ds.vlm_chart
|
vlm_chart = ds.vlm_chart
|
||||||
|
|
||||||
|
# Each market keeps its own destination-backed annotation layers.
|
||||||
|
# A forced cycle also handles source history coordinate rewrites.
|
||||||
|
trend_chart: ChartPlotWidget
|
||||||
|
for trend_chart in (chart, ds.hist_chart):
|
||||||
|
update_trend_overlays(
|
||||||
|
trend_chart,
|
||||||
|
fqme=fqme,
|
||||||
|
force=trigger_all,
|
||||||
|
render=not trend_chart.linked.isHidden(),
|
||||||
|
)
|
||||||
|
|
||||||
# varz = ds.vars
|
# varz = ds.vars
|
||||||
l1 = ds.l1
|
l1 = ds.l1
|
||||||
flume = ds.flume
|
flume = ds.flume
|
||||||
|
|
@ -1522,11 +1569,17 @@ async def display_symbol_data(
|
||||||
# load (user's) FSP set (otherwise known as "indicators")
|
# load (user's) FSP set (otherwise known as "indicators")
|
||||||
# from an input config.
|
# from an input config.
|
||||||
tn.start_soon(
|
tn.start_soon(
|
||||||
|
partial(
|
||||||
start_fsp_displays,
|
start_fsp_displays,
|
||||||
rt_linked,
|
rt_linked,
|
||||||
flume,
|
flume,
|
||||||
# loading_sym_key,
|
# loading_sym_key,
|
||||||
loglevel,
|
loglevel,
|
||||||
|
show_trend_calculus=(
|
||||||
|
ui_conf.get('trend_calculus') is True
|
||||||
|
),
|
||||||
|
trend_timeframes=ui_conf.get('trend_timeframes'),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
godwidget.resize_all()
|
godwidget.resize_all()
|
||||||
|
|
|
||||||
420
piker/ui/_fsp.py
420
piker/ui/_fsp.py
|
|
@ -23,9 +23,11 @@ Financial signal processing cluster and real-time graphics management.
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from contextlib import asynccontextmanager as acm
|
from contextlib import asynccontextmanager as acm
|
||||||
|
from collections.abc import AsyncIterator, Callable
|
||||||
from functools import partial
|
from functools import partial
|
||||||
import inspect
|
import inspect
|
||||||
from itertools import cycle
|
from itertools import cycle
|
||||||
|
import time
|
||||||
from typing import (
|
from typing import (
|
||||||
AsyncGenerator,
|
AsyncGenerator,
|
||||||
Any,
|
Any,
|
||||||
|
|
@ -47,12 +49,18 @@ from piker.fsp import (
|
||||||
Fsp,
|
Fsp,
|
||||||
dolla_vlm,
|
dolla_vlm,
|
||||||
flow_rates,
|
flow_rates,
|
||||||
|
trend_calculus,
|
||||||
)
|
)
|
||||||
from tractor.ipc._shm import (
|
from tractor.ipc._shm import (
|
||||||
ShmArray,
|
ShmArray,
|
||||||
NDToken,
|
NDToken,
|
||||||
)
|
)
|
||||||
from piker.data import Flume
|
from piker.data import Flume, open_sample_stream
|
||||||
|
from piker.fsp._regime import MultiTimeframeTrend, RegimeSnapshot
|
||||||
|
from piker.fsp._regime_feed import (
|
||||||
|
consume_regime_source,
|
||||||
|
source_snapshot_is_current,
|
||||||
|
)
|
||||||
from piker.data._sharedmem import try_read
|
from piker.data._sharedmem import try_read
|
||||||
from piker.log import get_logger
|
from piker.log import get_logger
|
||||||
from piker.toolz import Profiler
|
from piker.toolz import Profiler
|
||||||
|
|
@ -72,6 +80,8 @@ from ._forms import (
|
||||||
mk_form,
|
mk_form,
|
||||||
open_form_input_handling,
|
open_form_input_handling,
|
||||||
)
|
)
|
||||||
|
from ._trend import TrendOverlay
|
||||||
|
from ._trend_mtf import SnapshotTrendOverlay, _normalize_periods
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from ._dataviz import Viz
|
from ._dataviz import Viz
|
||||||
|
|
@ -80,6 +90,53 @@ if TYPE_CHECKING:
|
||||||
log = get_logger(__name__)
|
log = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def update_trend_overlays(
|
||||||
|
chart: ChartPlotWidget,
|
||||||
|
*,
|
||||||
|
fqme: str|None = None,
|
||||||
|
shm_token: NDToken|None = None,
|
||||||
|
chart_period_s: float|None = None,
|
||||||
|
force: bool = False,
|
||||||
|
render: bool = True,
|
||||||
|
|
||||||
|
) -> int:
|
||||||
|
'''
|
||||||
|
Refresh matching ``ChartPlotWidget._trend_overlays`` layers.
|
||||||
|
|
||||||
|
Its keys are ``(fqme, destination_token)`` and each value holds
|
||||||
|
independently owned reversal orders. Hidden-chart revisions
|
||||||
|
invalidate their layers without painting; the next quote then
|
||||||
|
rebuilds even if SHM bounds and the latest sample are unchanged.
|
||||||
|
|
||||||
|
'''
|
||||||
|
layers: dict[tuple[str, NDToken], list[TrendOverlay]] = getattr(
|
||||||
|
chart, '_trend_overlays', {},
|
||||||
|
)
|
||||||
|
updated: int = 0
|
||||||
|
key: tuple[str, NDToken]
|
||||||
|
overlays: list[TrendOverlay]
|
||||||
|
for key, overlays in layers.items():
|
||||||
|
market: str
|
||||||
|
token: NDToken
|
||||||
|
market, token = key
|
||||||
|
if fqme is not None and market != fqme:
|
||||||
|
continue
|
||||||
|
if shm_token is not None and token != shm_token:
|
||||||
|
continue
|
||||||
|
overlay: TrendOverlay
|
||||||
|
for overlay in overlays:
|
||||||
|
if (
|
||||||
|
chart_period_s is not None
|
||||||
|
and overlay.chart_period_s != chart_period_s
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
if force:
|
||||||
|
overlay.invalidate()
|
||||||
|
if render and overlay.update():
|
||||||
|
updated += 1
|
||||||
|
return updated
|
||||||
|
|
||||||
|
|
||||||
def update_fsp_chart(
|
def update_fsp_chart(
|
||||||
viz: Viz,
|
viz: Viz,
|
||||||
graphics_name: str,
|
graphics_name: str,
|
||||||
|
|
@ -466,6 +523,14 @@ class FspAdmin:
|
||||||
tuple[tractor.MsgStream, ShmArray]
|
tuple[tractor.MsgStream, ShmArray]
|
||||||
] = {}
|
] = {}
|
||||||
self._flow_registry: dict[NDToken, str] = {}
|
self._flow_registry: dict[NDToken, str] = {}
|
||||||
|
self._trend_engine: tuple[Flume, trio.Event]|None = None
|
||||||
|
self._trend_start_lock: trio.Lock = trio.Lock()
|
||||||
|
self._trend_overlays: list[
|
||||||
|
tuple[ChartPlotWidget, TrendOverlay]
|
||||||
|
] = []
|
||||||
|
self._multi_overlay: SnapshotTrendOverlay|None = None
|
||||||
|
self._multi_tracker: MultiTimeframeTrend|None = None
|
||||||
|
self._multi_scope: trio.CancelScope|None = None
|
||||||
|
|
||||||
# TODO: make this a `.src_flume` and add
|
# TODO: make this a `.src_flume` and add
|
||||||
# a `dst_flume`?
|
# a `dst_flume`?
|
||||||
|
|
@ -476,6 +541,332 @@ class FspAdmin:
|
||||||
name, portal = next(self._rr_next_actor)
|
name, portal = next(self._rr_next_actor)
|
||||||
return portal
|
return portal
|
||||||
|
|
||||||
|
async def open_trend_overlay(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
chart: ChartPlotWidget|None = None,
|
||||||
|
order: int = 1,
|
||||||
|
chart_period_s: float|None = None,
|
||||||
|
loglevel: str = 'info',
|
||||||
|
|
||||||
|
) -> TrendOverlay:
|
||||||
|
'''
|
||||||
|
Start one TC writer and attach confirmed price annotations.
|
||||||
|
|
||||||
|
Several charts or reversal orders reuse this admin's source
|
||||||
|
engine. The current FSP allocator supplies its realtime SHM
|
||||||
|
only; another sampling timeframe requires an independently
|
||||||
|
computed destination, attachable with ``TrendOverlay``.
|
||||||
|
No Jev request or credential lookup occurs on this path.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if not 1 <= order <= 4:
|
||||||
|
raise ValueError('order must be 1..4')
|
||||||
|
if chart is None:
|
||||||
|
chart = self.linked.chart
|
||||||
|
fqme: str = self.flume.mkt.fqme
|
||||||
|
viz: Viz|None = chart.get_viz(fqme)
|
||||||
|
if viz is None:
|
||||||
|
raise ValueError(f'No price Viz for {fqme}')
|
||||||
|
if chart_period_s is None:
|
||||||
|
if viz.shm._token == self.flume.rt_shm._token:
|
||||||
|
chart_period_s = self.flume.rt_sample_period_s
|
||||||
|
elif viz.shm._token == self.flume._hist_shm_token:
|
||||||
|
chart_period_s = self.flume.hist_sample_period_s
|
||||||
|
else:
|
||||||
|
raise ValueError('chart_period_s is required')
|
||||||
|
if not np.isfinite(chart_period_s) or chart_period_s <= 0:
|
||||||
|
raise ValueError('chart_period_s must be positive')
|
||||||
|
|
||||||
|
async with self._trend_start_lock:
|
||||||
|
owner: ChartPlotWidget
|
||||||
|
overlay: TrendOverlay
|
||||||
|
for owner, overlay in self._trend_overlays:
|
||||||
|
if (
|
||||||
|
owner is chart
|
||||||
|
and type(overlay) is TrendOverlay
|
||||||
|
and overlay.order == order
|
||||||
|
):
|
||||||
|
if overlay.chart_period_s != chart_period_s:
|
||||||
|
raise ValueError('chart period differs')
|
||||||
|
return overlay
|
||||||
|
|
||||||
|
if self._trend_engine is None:
|
||||||
|
self._trend_engine = await self.start_engine_task(
|
||||||
|
trend_calculus,
|
||||||
|
conf={},
|
||||||
|
loglevel=loglevel,
|
||||||
|
)
|
||||||
|
destination: Flume
|
||||||
|
started: trio.Event
|
||||||
|
destination, started = self._trend_engine
|
||||||
|
await started.wait()
|
||||||
|
overlay = TrendOverlay(
|
||||||
|
viz,
|
||||||
|
destination.rt_shm,
|
||||||
|
fqme=fqme,
|
||||||
|
source_period_s=self.flume.rt_sample_period_s,
|
||||||
|
chart_period_s=chart_period_s,
|
||||||
|
order=order,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
overlay.update(force=True)
|
||||||
|
except BaseException:
|
||||||
|
overlay.remove()
|
||||||
|
raise
|
||||||
|
|
||||||
|
layers: dict[
|
||||||
|
tuple[str, NDToken], list[TrendOverlay]
|
||||||
|
]|None = getattr(chart, '_trend_overlays', None)
|
||||||
|
if layers is None:
|
||||||
|
layers = chart._trend_overlays = {}
|
||||||
|
key: tuple[str, NDToken] = (
|
||||||
|
fqme, destination.rt_shm._token,
|
||||||
|
)
|
||||||
|
layers.setdefault(key, []).append(overlay)
|
||||||
|
self._trend_overlays.append((chart, overlay))
|
||||||
|
return overlay
|
||||||
|
|
||||||
|
async def open_multi_timeframe_trends(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
periods: tuple[float, ...] = (1, 5, 15),
|
||||||
|
windows: tuple[int, ...] = (2, 2, 2, 2),
|
||||||
|
order: int = 1,
|
||||||
|
on_snapshot: Callable[[RegimeSnapshot], object]|None = None,
|
||||||
|
on_invalidate: Callable[[int], object]|None = None,
|
||||||
|
max_source_age: float|None = 5,
|
||||||
|
classification_ttl: float = 5,
|
||||||
|
max_source_lag: float|None = 2,
|
||||||
|
|
||||||
|
) -> SnapshotTrendOverlay:
|
||||||
|
'''
|
||||||
|
Follow one finalized source into real multiple-timeframe UI.
|
||||||
|
|
||||||
|
The tracker consumes native realtime closes once and shares
|
||||||
|
its immutable snapshots with all graphics layers and optional
|
||||||
|
callbacks. Pass a classifier's ``submit`` and ``invalidate``
|
||||||
|
callbacks together to reuse this same evidence for Jev; no
|
||||||
|
client or credential is created here. The returned adapter's
|
||||||
|
``publish_classification`` fits the async classifier worker.
|
||||||
|
Only the optional model callback uses ``max_source_age``;
|
||||||
|
historical graphics always render. Explicit ``None`` permits
|
||||||
|
old model evidence for offline research. Jev requests project
|
||||||
|
shorter evidence tails without changing graphics retention.
|
||||||
|
|
||||||
|
Return after the first finalized snapshot. Empty startup
|
||||||
|
history waits for a final source bar.
|
||||||
|
``FspAdmin.close_trend_overlays()`` cancels
|
||||||
|
``FspAdmin._multi_scope`` and removes its layers from
|
||||||
|
``ChartPlotWidget._trend_overlays`` on context exit.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
max_source_age is not None
|
||||||
|
and
|
||||||
|
(
|
||||||
|
not np.isfinite(max_source_age)
|
||||||
|
or max_source_age <= 0
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise ValueError(
|
||||||
|
'max_source_age must be positive or None',
|
||||||
|
)
|
||||||
|
periods = _normalize_periods(
|
||||||
|
self.flume.rt_sample_period_s, periods,
|
||||||
|
)
|
||||||
|
async with self._trend_start_lock:
|
||||||
|
existing: SnapshotTrendOverlay|None = self._multi_overlay
|
||||||
|
if existing is not None:
|
||||||
|
first_layer: TrendOverlay = next(
|
||||||
|
iter(existing.layers.values()),
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
existing.periods != periods
|
||||||
|
or first_layer.order != order
|
||||||
|
or self._multi_tracker.windows != windows
|
||||||
|
):
|
||||||
|
raise ValueError('tracker configuration differs')
|
||||||
|
if (
|
||||||
|
on_snapshot is not None
|
||||||
|
or on_invalidate is not None
|
||||||
|
):
|
||||||
|
raise ValueError('callbacks already configured')
|
||||||
|
return existing
|
||||||
|
|
||||||
|
chart: ChartPlotWidget = self.linked.chart
|
||||||
|
fqme: str = self.flume.mkt.fqme
|
||||||
|
viz: Viz|None = chart.get_viz(fqme)
|
||||||
|
if viz is None:
|
||||||
|
raise ValueError(f'No price Viz for {fqme}')
|
||||||
|
base_period: float = self.flume.rt_sample_period_s
|
||||||
|
if base_period < 1:
|
||||||
|
raise ValueError('native sampler requires >=1s bars')
|
||||||
|
if not 1 <= order <= min(4, len(windows)):
|
||||||
|
raise ValueError('order exceeds configured windows')
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
fqme,
|
||||||
|
base_period=base_period,
|
||||||
|
periods=periods,
|
||||||
|
windows=windows,
|
||||||
|
history_limit=256,
|
||||||
|
)
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
viz,
|
||||||
|
self.flume.rt_shm,
|
||||||
|
fqme=fqme,
|
||||||
|
base_period=base_period,
|
||||||
|
periods=periods,
|
||||||
|
chart_period_s=base_period,
|
||||||
|
order=order,
|
||||||
|
classification_ttl=classification_ttl,
|
||||||
|
max_source_lag=max_source_lag,
|
||||||
|
is_visible=lambda: not self.linked.isHidden(),
|
||||||
|
)
|
||||||
|
self._multi_overlay = adapter
|
||||||
|
self._multi_tracker = tracker
|
||||||
|
layers: dict[
|
||||||
|
tuple[str, NDToken], list[TrendOverlay]
|
||||||
|
]|None = getattr(chart, '_trend_overlays', None)
|
||||||
|
if layers is None:
|
||||||
|
layers = chart._trend_overlays = {}
|
||||||
|
key: tuple[str, NDToken] = (
|
||||||
|
fqme, self.flume.rt_shm._token,
|
||||||
|
)
|
||||||
|
layer: TrendOverlay
|
||||||
|
for layer in adapter.layers.values():
|
||||||
|
layers.setdefault(key, []).append(layer)
|
||||||
|
self._trend_overlays.append((chart, layer))
|
||||||
|
|
||||||
|
started: trio.Event = trio.Event()
|
||||||
|
self.tn.start_soon(
|
||||||
|
self._run_multi_timeframe_trends,
|
||||||
|
adapter,
|
||||||
|
tracker,
|
||||||
|
started,
|
||||||
|
on_snapshot,
|
||||||
|
on_invalidate,
|
||||||
|
max_source_age,
|
||||||
|
)
|
||||||
|
await started.wait()
|
||||||
|
if adapter.as_of is None and self._multi_scope is None:
|
||||||
|
raise RuntimeError('source closed before bootstrap')
|
||||||
|
return adapter
|
||||||
|
|
||||||
|
async def _run_multi_timeframe_trends(
|
||||||
|
self,
|
||||||
|
adapter: SnapshotTrendOverlay,
|
||||||
|
tracker: MultiTimeframeTrend,
|
||||||
|
started: trio.Event,
|
||||||
|
on_snapshot: Callable[[RegimeSnapshot], object]|None,
|
||||||
|
on_invalidate: Callable[[int], object]|None,
|
||||||
|
max_source_age: float|None,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Subscribe before bootstrap and fan out one writer's evidence.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def submit(snapshot: RegimeSnapshot) -> None:
|
||||||
|
'''
|
||||||
|
Publish the bridge's completed, current source state.
|
||||||
|
|
||||||
|
'''
|
||||||
|
adapter.submit(snapshot)
|
||||||
|
if (
|
||||||
|
on_snapshot is not None
|
||||||
|
and source_snapshot_is_current(
|
||||||
|
snapshot,
|
||||||
|
now=time.time(),
|
||||||
|
max_source_age=max_source_age,
|
||||||
|
)
|
||||||
|
):
|
||||||
|
on_snapshot(snapshot)
|
||||||
|
started.set()
|
||||||
|
|
||||||
|
def invalidate(revision: int) -> None:
|
||||||
|
'''
|
||||||
|
Revoke graphics and optional model work before replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
adapter.invalidate(revision)
|
||||||
|
if on_invalidate is not None:
|
||||||
|
on_invalidate(revision)
|
||||||
|
|
||||||
|
scope: trio.CancelScope
|
||||||
|
with trio.CancelScope() as scope:
|
||||||
|
self._multi_scope = scope
|
||||||
|
try:
|
||||||
|
stream: AsyncIterator[dict]
|
||||||
|
async with open_sample_stream(
|
||||||
|
period_s=tracker.base_period,
|
||||||
|
) as stream:
|
||||||
|
await consume_regime_source(
|
||||||
|
self.flume.rt_shm,
|
||||||
|
stream,
|
||||||
|
tracker,
|
||||||
|
submit,
|
||||||
|
invalidate=invalidate,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
self._multi_scope = None
|
||||||
|
invalidate(adapter.revision + 1)
|
||||||
|
# Wake a bootstrap waiter if the source closes empty.
|
||||||
|
started.set()
|
||||||
|
|
||||||
|
def refresh_trend_overlays(
|
||||||
|
self,
|
||||||
|
shm_token: NDToken,
|
||||||
|
*,
|
||||||
|
render: bool,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Route a destination rewrite to this admin's attached charts.
|
||||||
|
|
||||||
|
'''
|
||||||
|
charts: set[ChartPlotWidget] = set()
|
||||||
|
chart: ChartPlotWidget
|
||||||
|
overlay: TrendOverlay
|
||||||
|
for chart, overlay in self._trend_overlays:
|
||||||
|
if chart in charts:
|
||||||
|
continue
|
||||||
|
charts.add(chart)
|
||||||
|
update_trend_overlays(
|
||||||
|
chart,
|
||||||
|
shm_token=shm_token,
|
||||||
|
force=True,
|
||||||
|
render=render,
|
||||||
|
)
|
||||||
|
|
||||||
|
def close_trend_overlays(self) -> None:
|
||||||
|
'''
|
||||||
|
Remove this admin's layers while preserving sibling owners.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if self._multi_scope is not None:
|
||||||
|
self._multi_scope.cancel()
|
||||||
|
if self._multi_overlay is not None:
|
||||||
|
self._multi_overlay.remove()
|
||||||
|
self._multi_overlay = None
|
||||||
|
self._multi_tracker = None
|
||||||
|
chart: ChartPlotWidget
|
||||||
|
overlay: TrendOverlay
|
||||||
|
for chart, overlay in self._trend_overlays:
|
||||||
|
layers: dict[
|
||||||
|
tuple[str, NDToken], list[TrendOverlay]
|
||||||
|
] = chart._trend_overlays
|
||||||
|
key: tuple[str, NDToken] = (
|
||||||
|
overlay.fqme, overlay.source._token,
|
||||||
|
)
|
||||||
|
owned: list[TrendOverlay] = layers.get(key, [])
|
||||||
|
if overlay in owned:
|
||||||
|
owned.remove(overlay)
|
||||||
|
if not owned:
|
||||||
|
layers.pop(key, None)
|
||||||
|
overlay.remove()
|
||||||
|
self._trend_overlays.clear()
|
||||||
|
|
||||||
async def open_chain(
|
async def open_chain(
|
||||||
self,
|
self,
|
||||||
|
|
||||||
|
|
@ -540,17 +931,23 @@ class FspAdmin:
|
||||||
async for msg in stream:
|
async for msg in stream:
|
||||||
info = msg.get('fsp_update')
|
info = msg.get('fsp_update')
|
||||||
if info:
|
if info:
|
||||||
|
token: NDToken = NDToken.from_msg(
|
||||||
|
dict(info['key']),
|
||||||
|
)
|
||||||
|
visible: bool = not self.linked.isHidden()
|
||||||
# if the chart isn't hidden try to update
|
# if the chart isn't hidden try to update
|
||||||
# the data on screen.
|
# the data on screen.
|
||||||
if not self.linked.isHidden():
|
if visible:
|
||||||
log.debug(
|
log.debug(
|
||||||
f'Re-syncing graphics for fsp: {ns_path}'
|
f'Re-syncing graphics for fsp: {ns_path}'
|
||||||
)
|
)
|
||||||
update_fsp_vizs(
|
update_fsp_vizs(
|
||||||
self.linked,
|
self.linked,
|
||||||
shm_token=NDToken.from_msg(
|
shm_token=token,
|
||||||
dict(info['key']),
|
)
|
||||||
),
|
self.refresh_trend_overlays(
|
||||||
|
token,
|
||||||
|
render=visible,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
log.info(f'recved unexpected fsp engine msg: {msg}')
|
log.info(f'recved unexpected fsp engine msg: {msg}')
|
||||||
|
|
@ -710,6 +1107,7 @@ async def open_fsp_admin(
|
||||||
|
|
||||||
# ??TODO, does this *need* to be inside a finally?
|
# ??TODO, does this *need* to be inside a finally?
|
||||||
finally:
|
finally:
|
||||||
|
admin.close_trend_overlays()
|
||||||
# terminate all tasks via signals
|
# terminate all tasks via signals
|
||||||
for key, entry in admin._registry.items():
|
for key, entry in admin._registry.items():
|
||||||
_, _, event = entry
|
_, _, event = entry
|
||||||
|
|
@ -1027,6 +1425,8 @@ async def start_fsp_displays(
|
||||||
flume: Flume,
|
flume: Flume,
|
||||||
# group_status_key: str,
|
# group_status_key: str,
|
||||||
loglevel: str,
|
loglevel: str,
|
||||||
|
show_trend_calculus: bool = False,
|
||||||
|
trend_timeframes: tuple[float, ...]|None = None,
|
||||||
|
|
||||||
) -> None:
|
) -> None:
|
||||||
'''
|
'''
|
||||||
|
|
@ -1072,6 +1472,13 @@ async def start_fsp_displays(
|
||||||
flume,
|
flume,
|
||||||
) as admin,
|
) as admin,
|
||||||
):
|
):
|
||||||
|
if trend_timeframes is not None:
|
||||||
|
await admin.open_multi_timeframe_trends(
|
||||||
|
periods=tuple(trend_timeframes),
|
||||||
|
)
|
||||||
|
elif show_trend_calculus:
|
||||||
|
await admin.open_trend_overlay(loglevel=loglevel)
|
||||||
|
|
||||||
statuses: list[trio.Event] = []
|
statuses: list[trio.Event] = []
|
||||||
for target, conf in fsp_conf.items():
|
for target, conf in fsp_conf.items():
|
||||||
started: trio.Event = await admin.open_fsp_chart(
|
started: trio.Event = await admin.open_fsp_chart(
|
||||||
|
|
@ -1091,4 +1498,7 @@ async def start_fsp_displays(
|
||||||
profiler(f'attached to fsp portal: {target}')
|
profiler(f'attached to fsp portal: {target}')
|
||||||
# status_cb()
|
# status_cb()
|
||||||
|
|
||||||
|
if show_trend_calculus or trend_timeframes is not None:
|
||||||
|
await trio.sleep_forever()
|
||||||
|
|
||||||
# blocks on nursery until all fsp actors complete
|
# blocks on nursery until all fsp actors complete
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,547 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Native multiple-timeframe annotations from finalized feed snapshots.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import deque
|
||||||
|
from collections.abc import Callable
|
||||||
|
from math import isfinite
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from msgspec.structs import replace
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from piker.fsp._regime import (
|
||||||
|
RegimeSnapshot,
|
||||||
|
TimeframeRegime,
|
||||||
|
_period_us,
|
||||||
|
)
|
||||||
|
from ._trend import (
|
||||||
|
RegimeInterval,
|
||||||
|
ReversalPoint,
|
||||||
|
TrendOverlay,
|
||||||
|
_positive_integer,
|
||||||
|
times_to_x,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from tractor.ipc._shm import ShmArray
|
||||||
|
from piker.fsp._jev import JevClassification, RegimeAnswer
|
||||||
|
from piker.fsp._regime import TrendReversal
|
||||||
|
from ._dataviz import Viz
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_periods(
|
||||||
|
base_period: float,
|
||||||
|
periods: tuple[float, ...],
|
||||||
|
|
||||||
|
) -> tuple[float, ...]:
|
||||||
|
'''
|
||||||
|
Match the shared tracker's canonical microsecond durations.
|
||||||
|
|
||||||
|
'''
|
||||||
|
base_us: int = _period_us(base_period)
|
||||||
|
period: float # noqa: F842
|
||||||
|
period_us: tuple[int, ...] = tuple(
|
||||||
|
sorted(_period_us(period) for period in periods)
|
||||||
|
)
|
||||||
|
if not 1 <= len(period_us) <= 16:
|
||||||
|
raise ValueError('configure 1..16 unique timeframes')
|
||||||
|
if len(set(period_us)) != len(period_us):
|
||||||
|
raise ValueError('timeframes must be unique')
|
||||||
|
value: int # noqa: F842
|
||||||
|
if any(value % base_us for value in period_us):
|
||||||
|
raise ValueError('timeframes require whole base multiples')
|
||||||
|
return tuple(value / 1_000_000 for value in period_us)
|
||||||
|
|
||||||
|
|
||||||
|
class _SnapshotLayer(TrendOverlay):
|
||||||
|
'''
|
||||||
|
Render one real aggregated timeframe using existing Qt batches.
|
||||||
|
|
||||||
|
The source is the actual base OHLC SHM. ``_SnapshotLayer._frame``
|
||||||
|
supplies computed evidence; no pretend FSP SHM or repeated TC
|
||||||
|
computation is needed. Snapshot pivots retain interval-end
|
||||||
|
timestamps while x positions identify their final base bar.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
viz: Viz,
|
||||||
|
source: ShmArray,
|
||||||
|
*,
|
||||||
|
fqme: str,
|
||||||
|
source_period_s: float,
|
||||||
|
chart_period_s: float,
|
||||||
|
order: int = 1,
|
||||||
|
max_points: int = 4096,
|
||||||
|
max_rectangles: int = 1024,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Initialize a bounded, empty timeframe layer.
|
||||||
|
|
||||||
|
'''
|
||||||
|
super().__init__(
|
||||||
|
viz,
|
||||||
|
source,
|
||||||
|
fqme=fqme,
|
||||||
|
source_period_s=source_period_s,
|
||||||
|
chart_period_s=chart_period_s,
|
||||||
|
order=order,
|
||||||
|
max_points=max_points,
|
||||||
|
max_rectangles=max_rectangles,
|
||||||
|
)
|
||||||
|
self._frame: TimeframeRegime|None = None
|
||||||
|
self._base_period: float = self.chart_period_s
|
||||||
|
self._runs: deque[RegimeInterval] = deque(
|
||||||
|
maxlen=self.max_rectangles,
|
||||||
|
)
|
||||||
|
self._active: bool = False
|
||||||
|
|
||||||
|
def clear_evidence(self) -> None:
|
||||||
|
'''
|
||||||
|
Revoke old source evidence immediately on revision or gap.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._frame = None
|
||||||
|
self._runs.clear()
|
||||||
|
self._active = False
|
||||||
|
self._external = None
|
||||||
|
self._native_intervals.clear()
|
||||||
|
self._set_points([])
|
||||||
|
self.rectangles.set_data(np.empty((0, 4)), [])
|
||||||
|
self.invalidate()
|
||||||
|
|
||||||
|
def accept_frame(
|
||||||
|
self,
|
||||||
|
frame: TimeframeRegime,
|
||||||
|
base_period: float,
|
||||||
|
horizon: float,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Retain only causal live runs supported by received snapshots.
|
||||||
|
|
||||||
|
Bootstrap starts at the latest proven direction, since the
|
||||||
|
snapshot has no earlier trend history. If coalescing skips
|
||||||
|
complete buckets, end the previous run at its next possible
|
||||||
|
decision and leave the unknown interval unpainted.
|
||||||
|
|
||||||
|
'''
|
||||||
|
previous: TimeframeRegime|None = self._frame
|
||||||
|
self._frame = frame
|
||||||
|
self._base_period = base_period
|
||||||
|
level: int = self.order - 1
|
||||||
|
ready: bool = (
|
||||||
|
frame.as_of is not None
|
||||||
|
and len(frame.ready) > level
|
||||||
|
and len(frame.trends) > level
|
||||||
|
and frame.ready[level]
|
||||||
|
and abs(frame.trends[level]) == 1
|
||||||
|
)
|
||||||
|
label: str = (
|
||||||
|
'uptrend' if frame.trends[level] > 0 else 'downtrend'
|
||||||
|
) if len(frame.trends) > level else ''
|
||||||
|
skipped: bool = (
|
||||||
|
previous is not None
|
||||||
|
and frame.close_count > previous.close_count + 1
|
||||||
|
)
|
||||||
|
if self._active:
|
||||||
|
if skipped:
|
||||||
|
self._runs[-1] = replace(
|
||||||
|
self._runs[-1],
|
||||||
|
end_time=previous.as_of + frame.period,
|
||||||
|
)
|
||||||
|
self._active = False
|
||||||
|
elif not ready or self._runs[-1].label != label:
|
||||||
|
self._runs[-1] = replace(
|
||||||
|
self._runs[-1],
|
||||||
|
end_time=(
|
||||||
|
frame.as_of
|
||||||
|
if frame.as_of is not None
|
||||||
|
else horizon - base_period
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self._active = False
|
||||||
|
|
||||||
|
if ready:
|
||||||
|
if self._active:
|
||||||
|
self._runs[-1] = replace(
|
||||||
|
self._runs[-1], end_time=horizon,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._runs.append(RegimeInterval(
|
||||||
|
start_time=frame.as_of,
|
||||||
|
end_time=horizon,
|
||||||
|
label=label,
|
||||||
|
))
|
||||||
|
self._active = True
|
||||||
|
self.invalidate()
|
||||||
|
|
||||||
|
def _prepare_evidence(
|
||||||
|
self,
|
||||||
|
source: np.ndarray,
|
||||||
|
chart: np.ndarray,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Map immutable endpoint events to their actual source bars.
|
||||||
|
|
||||||
|
'''
|
||||||
|
frame: TimeframeRegime|None = self._frame
|
||||||
|
if frame is None:
|
||||||
|
self._set_points([])
|
||||||
|
self._native_intervals = []
|
||||||
|
return
|
||||||
|
|
||||||
|
events: list[TrendReversal] = []
|
||||||
|
event: TrendReversal
|
||||||
|
for event in frame.reversals:
|
||||||
|
if event.order == self.order:
|
||||||
|
events.append(event)
|
||||||
|
events = events[-self.max_points:]
|
||||||
|
times: np.ndarray = np.fromiter(
|
||||||
|
(event.time - self._base_period for event in events),
|
||||||
|
dtype=float,
|
||||||
|
count=len(events),
|
||||||
|
)
|
||||||
|
xs: np.ndarray = times_to_x(
|
||||||
|
chart,
|
||||||
|
times,
|
||||||
|
self.chart_period_s,
|
||||||
|
self.viz.index_field,
|
||||||
|
)
|
||||||
|
points: list[ReversalPoint] = []
|
||||||
|
x: np.floating
|
||||||
|
for event, x in zip(events, xs, strict=True):
|
||||||
|
if not np.isfinite(x):
|
||||||
|
continue
|
||||||
|
points.append(ReversalPoint(
|
||||||
|
x=float(x),
|
||||||
|
price=event.price,
|
||||||
|
pivot_time=event.time,
|
||||||
|
detected_at=event.detected_time,
|
||||||
|
kind='bottom' if event.direction > 0 else 'top',
|
||||||
|
order=self.order,
|
||||||
|
period_s=frame.period,
|
||||||
|
timestamp_convention='interval_end',
|
||||||
|
))
|
||||||
|
self._set_points(points)
|
||||||
|
self._native_intervals = list(self._runs)
|
||||||
|
|
||||||
|
|
||||||
|
class SnapshotTrendOverlay:
|
||||||
|
'''
|
||||||
|
Consume one shared tracker into bounded per-timeframe graphics.
|
||||||
|
|
||||||
|
Pass ``submit`` and ``invalidate`` directly to the native
|
||||||
|
``consume_regime_source`` bridge. Source revision and gap segment
|
||||||
|
changes synchronously revoke stale annotations; old or duplicate
|
||||||
|
snapshot sequences cannot repaint newer evidence.
|
||||||
|
|
||||||
|
``submit_classification`` is an optional local Jev callback.
|
||||||
|
It performs no I/O and never reads credentials. Model rectangles
|
||||||
|
have a finite ``classification_ttl`` after response receipt and
|
||||||
|
stay absent until the displayed event clock reaches receipt.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
viz: Viz,
|
||||||
|
source: ShmArray,
|
||||||
|
*,
|
||||||
|
fqme: str,
|
||||||
|
base_period: float,
|
||||||
|
periods: tuple[float, ...] = (1, 5, 15),
|
||||||
|
chart_period_s: float = 1,
|
||||||
|
order: int = 1,
|
||||||
|
history_limit: int = 256,
|
||||||
|
classification_ttl: float = 5,
|
||||||
|
max_source_lag: float|None = 2,
|
||||||
|
is_visible: Callable[[], bool]|None = None,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Allocate two scene items per configured sampling timeframe.
|
||||||
|
|
||||||
|
'''
|
||||||
|
periods = _normalize_periods(base_period, periods)
|
||||||
|
base_period = _period_us(base_period) / 1_000_000
|
||||||
|
order = _positive_integer(order, 'order')
|
||||||
|
history_limit = _positive_integer(
|
||||||
|
history_limit, 'history_limit',
|
||||||
|
)
|
||||||
|
if order > 4:
|
||||||
|
raise ValueError('order must be 1..4')
|
||||||
|
if history_limit > 256:
|
||||||
|
raise ValueError('history_limit must be 1..256')
|
||||||
|
if (
|
||||||
|
not isfinite(classification_ttl)
|
||||||
|
or classification_ttl <= 0
|
||||||
|
):
|
||||||
|
raise ValueError('classification_ttl must be positive')
|
||||||
|
if (
|
||||||
|
max_source_lag is not None
|
||||||
|
and
|
||||||
|
(
|
||||||
|
not isfinite(max_source_lag)
|
||||||
|
or max_source_lag < 0
|
||||||
|
)
|
||||||
|
):
|
||||||
|
raise ValueError('max_source_lag must be >=0 or None')
|
||||||
|
self.fqme: str = fqme
|
||||||
|
self.base_period: float = base_period
|
||||||
|
self.periods: tuple[float, ...] = periods
|
||||||
|
self.classification_ttl: float = classification_ttl
|
||||||
|
self.max_source_lag: float|None = max_source_lag
|
||||||
|
self.is_visible: Callable[[], bool] = (
|
||||||
|
is_visible or (lambda: True)
|
||||||
|
)
|
||||||
|
self.revision: int = 0
|
||||||
|
self.sequence: int = -1
|
||||||
|
self.segment: int|None = None
|
||||||
|
self.as_of: float|None = None
|
||||||
|
self._classification_sequence: int = -1
|
||||||
|
self._model_runs: dict[float, deque[RegimeInterval]] = {}
|
||||||
|
self._model_expiry: dict[float, float] = {}
|
||||||
|
self._removed: bool = False
|
||||||
|
self.layers: dict[float, _SnapshotLayer] = {}
|
||||||
|
try:
|
||||||
|
period: float
|
||||||
|
for period in periods:
|
||||||
|
self.layers[period] = _SnapshotLayer(
|
||||||
|
viz,
|
||||||
|
source,
|
||||||
|
fqme=fqme,
|
||||||
|
source_period_s=period,
|
||||||
|
chart_period_s=chart_period_s,
|
||||||
|
order=order,
|
||||||
|
max_points=history_limit,
|
||||||
|
max_rectangles=history_limit,
|
||||||
|
)
|
||||||
|
self._model_runs[period] = deque(
|
||||||
|
maxlen=history_limit,
|
||||||
|
)
|
||||||
|
except BaseException:
|
||||||
|
self.remove()
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _clear_evidence(self) -> None:
|
||||||
|
'''
|
||||||
|
Clear this owner's deterministic and classified history.
|
||||||
|
|
||||||
|
'''
|
||||||
|
layer: _SnapshotLayer
|
||||||
|
for layer in self.layers.values():
|
||||||
|
layer.clear_evidence()
|
||||||
|
runs: deque[RegimeInterval]
|
||||||
|
for runs in self._model_runs.values():
|
||||||
|
runs.clear()
|
||||||
|
self._model_expiry.clear()
|
||||||
|
self._classification_sequence = -1
|
||||||
|
|
||||||
|
def invalidate(self, revision: int) -> None:
|
||||||
|
'''
|
||||||
|
Revoke an obsolete source generation before replay can yield.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if self._removed or revision < self.revision:
|
||||||
|
return
|
||||||
|
self.revision = revision
|
||||||
|
self.sequence = -1
|
||||||
|
self.segment = None
|
||||||
|
self.as_of = None
|
||||||
|
self._clear_evidence()
|
||||||
|
|
||||||
|
def submit(self, snapshot: RegimeSnapshot) -> bool:
|
||||||
|
'''
|
||||||
|
Accept the latest native evidence and refresh visible layers.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
self._removed
|
||||||
|
or snapshot.market != self.fqme
|
||||||
|
or snapshot.revision < self.revision
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
if snapshot.base_period != self.base_period:
|
||||||
|
raise ValueError('snapshot base period differs')
|
||||||
|
if snapshot.revision > self.revision:
|
||||||
|
self.invalidate(snapshot.revision)
|
||||||
|
if snapshot.sequence <= self.sequence:
|
||||||
|
return False
|
||||||
|
if self.as_of is not None and snapshot.as_of < self.as_of:
|
||||||
|
return False
|
||||||
|
frame: TimeframeRegime # noqa: F842
|
||||||
|
frames: dict[float, TimeframeRegime] = {
|
||||||
|
frame.period: frame for frame in snapshot.timeframes
|
||||||
|
}
|
||||||
|
if not set(self.periods) <= frames.keys():
|
||||||
|
raise ValueError('snapshot is missing a timeframe')
|
||||||
|
if (
|
||||||
|
self.segment is not None
|
||||||
|
and snapshot.segment != self.segment
|
||||||
|
):
|
||||||
|
self._clear_evidence()
|
||||||
|
self.segment = snapshot.segment
|
||||||
|
self.sequence = snapshot.sequence
|
||||||
|
self.as_of = snapshot.as_of
|
||||||
|
horizon: float = snapshot.as_of + self.base_period
|
||||||
|
period: float
|
||||||
|
layer: _SnapshotLayer
|
||||||
|
for period, layer in self.layers.items():
|
||||||
|
layer.accept_frame(
|
||||||
|
frames[period], self.base_period, horizon,
|
||||||
|
)
|
||||||
|
self._extend_model(period, horizon)
|
||||||
|
if self.is_visible():
|
||||||
|
layer.update()
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _extend_model(self, period: float, horizon: float) -> None:
|
||||||
|
'''
|
||||||
|
Extend the active model interval only through its expiry.
|
||||||
|
|
||||||
|
'''
|
||||||
|
runs: deque[RegimeInterval] = self._model_runs[period]
|
||||||
|
if not runs:
|
||||||
|
return
|
||||||
|
runs[-1] = replace(
|
||||||
|
runs[-1],
|
||||||
|
end_time=min(horizon, self._model_expiry[period]),
|
||||||
|
)
|
||||||
|
layer: _SnapshotLayer = self.layers[period]
|
||||||
|
layer._external = tuple(runs)
|
||||||
|
layer._signature = None
|
||||||
|
|
||||||
|
def submit_classification(
|
||||||
|
self,
|
||||||
|
result: JevClassification,
|
||||||
|
|
||||||
|
) -> bool:
|
||||||
|
'''
|
||||||
|
Attach a matching, newly received Jev result without API I/O.
|
||||||
|
|
||||||
|
Older requests may finish after newer chart snapshots; source
|
||||||
|
sequence must increase between accepted model responses and
|
||||||
|
never exceed the chart's known sequence. Revision and segment
|
||||||
|
must match exactly. Caller-owned provider freshness checks
|
||||||
|
still apply before invoking this callback.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
self._removed
|
||||||
|
or self.as_of is None
|
||||||
|
or result.market != self.fqme
|
||||||
|
or result.revision != self.revision
|
||||||
|
or result.segment != self.segment
|
||||||
|
or result.sequence <= self._classification_sequence
|
||||||
|
or result.sequence > self.sequence
|
||||||
|
or not isfinite(result.source_as_of)
|
||||||
|
or result.source_as_of > self.as_of
|
||||||
|
or not isfinite(result.received_at)
|
||||||
|
or (
|
||||||
|
result.received_at + self.classification_ttl
|
||||||
|
<= self.as_of
|
||||||
|
)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
if (
|
||||||
|
self.max_source_lag is not None
|
||||||
|
and (
|
||||||
|
self.as_of - result.source_as_of
|
||||||
|
> self.max_source_lag
|
||||||
|
)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
answers: list[RegimeAnswer] = []
|
||||||
|
answer: RegimeAnswer
|
||||||
|
for answer in result.answers:
|
||||||
|
if answer.period not in self.layers:
|
||||||
|
continue
|
||||||
|
if not isfinite(answer.probability):
|
||||||
|
return False
|
||||||
|
if not 0 <= answer.probability <= 1:
|
||||||
|
return False
|
||||||
|
answers.append(answer)
|
||||||
|
if not answers:
|
||||||
|
return False
|
||||||
|
|
||||||
|
available: float = max(
|
||||||
|
result.source_as_of, result.received_at,
|
||||||
|
)
|
||||||
|
horizon: float = self.as_of + self.base_period
|
||||||
|
expiry: float = available + self.classification_ttl
|
||||||
|
model: str = result.model
|
||||||
|
for answer in answers:
|
||||||
|
period: float = answer.period
|
||||||
|
runs: deque[RegimeInterval] = self._model_runs[period]
|
||||||
|
if runs:
|
||||||
|
runs[-1] = replace(
|
||||||
|
runs[-1],
|
||||||
|
end_time=min(
|
||||||
|
available, self._model_expiry[period],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
runs.append(RegimeInterval(
|
||||||
|
start_time=result.source_as_of,
|
||||||
|
end_time=min(horizon, expiry),
|
||||||
|
label=answer.label,
|
||||||
|
probability=answer.probability,
|
||||||
|
source=f'typesafe.ai/{model}',
|
||||||
|
observed_at=available,
|
||||||
|
))
|
||||||
|
self._model_expiry[period] = expiry
|
||||||
|
layer: _SnapshotLayer = self.layers[period]
|
||||||
|
layer._external = tuple(runs)
|
||||||
|
layer._signature = None
|
||||||
|
if self.is_visible():
|
||||||
|
layer.update()
|
||||||
|
self._classification_sequence = result.sequence
|
||||||
|
return True
|
||||||
|
|
||||||
|
def remove(self) -> None:
|
||||||
|
'''
|
||||||
|
Remove every timeframe owned by this adapter, idempotently.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if self._removed:
|
||||||
|
return
|
||||||
|
layer: _SnapshotLayer
|
||||||
|
for layer in self.layers.values():
|
||||||
|
layer.remove()
|
||||||
|
self._removed = True
|
||||||
|
|
||||||
|
async def publish_classification(
|
||||||
|
self,
|
||||||
|
result: JevClassification,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Adapt the synchronous graphics callback to classifier.run().
|
||||||
|
|
||||||
|
No await occurs inside the mutation, so revision invalidation
|
||||||
|
cannot interleave halfway through a Qt update.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.submit_classification(result)
|
||||||
|
|
@ -0,0 +1,876 @@
|
||||||
|
{
|
||||||
|
"source": "https://github.com/ByteSumoLtd/TrendCalculus-lua/blob/7f54f9a101283cc188dbcd9282292fc3f73de8e9/make/trendcalculus.lua",
|
||||||
|
"source_sha256": "c099fac68c945768c71f7c09804a21777295bbbf2e7dd61db59d9cdfd78f49c0",
|
||||||
|
"source_license": "GPL-3.0-or-later; Copyright 2014-2015 ByteSumo Limited (Andrew J Morgan)",
|
||||||
|
"fixture_license": "AGPL-3.0-or-later",
|
||||||
|
"generated_at": "2026-10-05",
|
||||||
|
"runtime": "Lua 5.2.4",
|
||||||
|
"command": "lua trendcalculus.lua -n{window} -r -H -p1",
|
||||||
|
"stdin": "ticker,date,last header followed by x,{zero_based_row},{value} rows, LF-terminated",
|
||||||
|
"note": "Do not use -h: embedded CSV column mapping already consumes the header and -h drops the first data row.",
|
||||||
|
"detection": "Run every complete-window prefix through the unmodified CLI. Newly emitted rows get detected_index = prefix_length - 1. Assert output-prefix preservation.",
|
||||||
|
"event_fields": [
|
||||||
|
"index",
|
||||||
|
"price",
|
||||||
|
"direction",
|
||||||
|
"detected_index"
|
||||||
|
],
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"window": 2,
|
||||||
|
"values": [
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
4,
|
||||||
|
5,
|
||||||
|
4,
|
||||||
|
7,
|
||||||
|
9,
|
||||||
|
12,
|
||||||
|
9,
|
||||||
|
6,
|
||||||
|
4,
|
||||||
|
1,
|
||||||
|
0,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
-3,
|
||||||
|
-3,
|
||||||
|
-3,
|
||||||
|
-6,
|
||||||
|
-9,
|
||||||
|
-8,
|
||||||
|
-6,
|
||||||
|
-8,
|
||||||
|
-9,
|
||||||
|
-8,
|
||||||
|
-9,
|
||||||
|
-7,
|
||||||
|
-8,
|
||||||
|
-9,
|
||||||
|
-10,
|
||||||
|
-7,
|
||||||
|
-6,
|
||||||
|
-6,
|
||||||
|
-6,
|
||||||
|
-7,
|
||||||
|
-7,
|
||||||
|
-5,
|
||||||
|
-4,
|
||||||
|
-6,
|
||||||
|
-6,
|
||||||
|
-9,
|
||||||
|
-7,
|
||||||
|
-7,
|
||||||
|
-6,
|
||||||
|
-4,
|
||||||
|
-1,
|
||||||
|
-1,
|
||||||
|
-4,
|
||||||
|
-3,
|
||||||
|
-4,
|
||||||
|
-6,
|
||||||
|
-4,
|
||||||
|
-6,
|
||||||
|
-6,
|
||||||
|
-9,
|
||||||
|
-9,
|
||||||
|
-7,
|
||||||
|
-6,
|
||||||
|
-4,
|
||||||
|
-7,
|
||||||
|
-5,
|
||||||
|
-7,
|
||||||
|
-5,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
-3,
|
||||||
|
-3,
|
||||||
|
0,
|
||||||
|
-3,
|
||||||
|
-3,
|
||||||
|
-1,
|
||||||
|
-4,
|
||||||
|
-2,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
-1,
|
||||||
|
-4,
|
||||||
|
-1,
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
-1,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
1,
|
||||||
|
3,
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
10,
|
||||||
|
8,
|
||||||
|
8,
|
||||||
|
5,
|
||||||
|
5,
|
||||||
|
2,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
1,
|
||||||
|
-2,
|
||||||
|
1,
|
||||||
|
-1
|
||||||
|
],
|
||||||
|
"events": [
|
||||||
|
[
|
||||||
|
9,
|
||||||
|
12.0,
|
||||||
|
-1,
|
||||||
|
11
|
||||||
|
],
|
||||||
|
[
|
||||||
|
21,
|
||||||
|
-9.0,
|
||||||
|
1,
|
||||||
|
23
|
||||||
|
],
|
||||||
|
[
|
||||||
|
23,
|
||||||
|
-6.0,
|
||||||
|
-1,
|
||||||
|
25
|
||||||
|
],
|
||||||
|
[
|
||||||
|
27,
|
||||||
|
-9.0,
|
||||||
|
1,
|
||||||
|
29
|
||||||
|
],
|
||||||
|
[
|
||||||
|
28,
|
||||||
|
-7.0,
|
||||||
|
-1,
|
||||||
|
31
|
||||||
|
],
|
||||||
|
[
|
||||||
|
31,
|
||||||
|
-10.0,
|
||||||
|
1,
|
||||||
|
33
|
||||||
|
],
|
||||||
|
[
|
||||||
|
34,
|
||||||
|
-6.0,
|
||||||
|
-1,
|
||||||
|
37
|
||||||
|
],
|
||||||
|
[
|
||||||
|
36,
|
||||||
|
-7.0,
|
||||||
|
1,
|
||||||
|
39
|
||||||
|
],
|
||||||
|
[
|
||||||
|
39,
|
||||||
|
-4.0,
|
||||||
|
-1,
|
||||||
|
41
|
||||||
|
],
|
||||||
|
[
|
||||||
|
42,
|
||||||
|
-9.0,
|
||||||
|
1,
|
||||||
|
45
|
||||||
|
],
|
||||||
|
[
|
||||||
|
48,
|
||||||
|
-1.0,
|
||||||
|
-1,
|
||||||
|
49
|
||||||
|
],
|
||||||
|
[
|
||||||
|
56,
|
||||||
|
-9.0,
|
||||||
|
1,
|
||||||
|
59
|
||||||
|
],
|
||||||
|
[
|
||||||
|
60,
|
||||||
|
-4.0,
|
||||||
|
-1,
|
||||||
|
63
|
||||||
|
],
|
||||||
|
[
|
||||||
|
63,
|
||||||
|
-7.0,
|
||||||
|
1,
|
||||||
|
65
|
||||||
|
],
|
||||||
|
[
|
||||||
|
69,
|
||||||
|
0.0,
|
||||||
|
-1,
|
||||||
|
71
|
||||||
|
],
|
||||||
|
[
|
||||||
|
70,
|
||||||
|
-3.0,
|
||||||
|
1,
|
||||||
|
73
|
||||||
|
],
|
||||||
|
[
|
||||||
|
72,
|
||||||
|
-1.0,
|
||||||
|
-1,
|
||||||
|
73
|
||||||
|
],
|
||||||
|
[
|
||||||
|
73,
|
||||||
|
-4.0,
|
||||||
|
1,
|
||||||
|
75
|
||||||
|
],
|
||||||
|
[
|
||||||
|
76,
|
||||||
|
1.0,
|
||||||
|
-1,
|
||||||
|
79
|
||||||
|
],
|
||||||
|
[
|
||||||
|
78,
|
||||||
|
-4.0,
|
||||||
|
1,
|
||||||
|
81
|
||||||
|
],
|
||||||
|
[
|
||||||
|
81,
|
||||||
|
1.0,
|
||||||
|
-1,
|
||||||
|
83
|
||||||
|
],
|
||||||
|
[
|
||||||
|
84,
|
||||||
|
-2.0,
|
||||||
|
1,
|
||||||
|
87
|
||||||
|
],
|
||||||
|
[
|
||||||
|
92,
|
||||||
|
10.0,
|
||||||
|
-1,
|
||||||
|
95
|
||||||
|
],
|
||||||
|
[
|
||||||
|
100,
|
||||||
|
-2.0,
|
||||||
|
1,
|
||||||
|
103
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"input_csv_sha256": "8b146558fa0f23f8926404bfeb87e433f02dd253c5a8eed2cf7bec21fa7d62ab",
|
||||||
|
"output_csv_sha256": "46915ca3dcb7da1b3f9e72a8ece1a8beb032a4f99657798022236150d9c336af"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"window": 3,
|
||||||
|
"values": [
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
4,
|
||||||
|
5,
|
||||||
|
6,
|
||||||
|
7,
|
||||||
|
8,
|
||||||
|
11,
|
||||||
|
14,
|
||||||
|
16,
|
||||||
|
19,
|
||||||
|
19,
|
||||||
|
17,
|
||||||
|
14,
|
||||||
|
15,
|
||||||
|
17,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
16,
|
||||||
|
18,
|
||||||
|
19,
|
||||||
|
17,
|
||||||
|
16,
|
||||||
|
19,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
15,
|
||||||
|
16,
|
||||||
|
16,
|
||||||
|
15,
|
||||||
|
14,
|
||||||
|
17,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
14,
|
||||||
|
14,
|
||||||
|
17,
|
||||||
|
20,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
20,
|
||||||
|
22,
|
||||||
|
21,
|
||||||
|
20,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
20,
|
||||||
|
23,
|
||||||
|
22,
|
||||||
|
19,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
24,
|
||||||
|
23,
|
||||||
|
21,
|
||||||
|
23,
|
||||||
|
26,
|
||||||
|
26,
|
||||||
|
28,
|
||||||
|
25,
|
||||||
|
28,
|
||||||
|
31,
|
||||||
|
30,
|
||||||
|
27,
|
||||||
|
24,
|
||||||
|
24,
|
||||||
|
26,
|
||||||
|
26,
|
||||||
|
28,
|
||||||
|
31,
|
||||||
|
33,
|
||||||
|
35,
|
||||||
|
33,
|
||||||
|
34,
|
||||||
|
32,
|
||||||
|
32,
|
||||||
|
31,
|
||||||
|
28,
|
||||||
|
29,
|
||||||
|
30,
|
||||||
|
33,
|
||||||
|
33,
|
||||||
|
31,
|
||||||
|
34,
|
||||||
|
33,
|
||||||
|
30,
|
||||||
|
29,
|
||||||
|
28,
|
||||||
|
29,
|
||||||
|
31,
|
||||||
|
33,
|
||||||
|
33,
|
||||||
|
33,
|
||||||
|
30,
|
||||||
|
29,
|
||||||
|
28,
|
||||||
|
28,
|
||||||
|
30,
|
||||||
|
33,
|
||||||
|
34,
|
||||||
|
35,
|
||||||
|
35,
|
||||||
|
34
|
||||||
|
],
|
||||||
|
"events": [
|
||||||
|
[
|
||||||
|
12,
|
||||||
|
19.0,
|
||||||
|
-1,
|
||||||
|
17
|
||||||
|
],
|
||||||
|
[
|
||||||
|
15,
|
||||||
|
14.0,
|
||||||
|
1,
|
||||||
|
20
|
||||||
|
],
|
||||||
|
[
|
||||||
|
22,
|
||||||
|
19.0,
|
||||||
|
-1,
|
||||||
|
26
|
||||||
|
],
|
||||||
|
[
|
||||||
|
32,
|
||||||
|
14.0,
|
||||||
|
1,
|
||||||
|
35
|
||||||
|
],
|
||||||
|
[
|
||||||
|
33,
|
||||||
|
17.0,
|
||||||
|
-1,
|
||||||
|
38
|
||||||
|
],
|
||||||
|
[
|
||||||
|
36,
|
||||||
|
14.0,
|
||||||
|
1,
|
||||||
|
41
|
||||||
|
],
|
||||||
|
[
|
||||||
|
43,
|
||||||
|
22.0,
|
||||||
|
-1,
|
||||||
|
47
|
||||||
|
],
|
||||||
|
[
|
||||||
|
46,
|
||||||
|
18.0,
|
||||||
|
1,
|
||||||
|
50
|
||||||
|
],
|
||||||
|
[
|
||||||
|
49,
|
||||||
|
23.0,
|
||||||
|
-1,
|
||||||
|
53
|
||||||
|
],
|
||||||
|
[
|
||||||
|
55,
|
||||||
|
18.0,
|
||||||
|
1,
|
||||||
|
59
|
||||||
|
],
|
||||||
|
[
|
||||||
|
67,
|
||||||
|
31.0,
|
||||||
|
-1,
|
||||||
|
71
|
||||||
|
],
|
||||||
|
[
|
||||||
|
70,
|
||||||
|
24.0,
|
||||||
|
1,
|
||||||
|
74
|
||||||
|
],
|
||||||
|
[
|
||||||
|
77,
|
||||||
|
35.0,
|
||||||
|
-1,
|
||||||
|
80
|
||||||
|
],
|
||||||
|
[
|
||||||
|
83,
|
||||||
|
28.0,
|
||||||
|
1,
|
||||||
|
86
|
||||||
|
],
|
||||||
|
[
|
||||||
|
89,
|
||||||
|
34.0,
|
||||||
|
-1,
|
||||||
|
92
|
||||||
|
],
|
||||||
|
[
|
||||||
|
93,
|
||||||
|
28.0,
|
||||||
|
1,
|
||||||
|
98
|
||||||
|
],
|
||||||
|
[
|
||||||
|
96,
|
||||||
|
33.0,
|
||||||
|
-1,
|
||||||
|
101
|
||||||
|
],
|
||||||
|
[
|
||||||
|
101,
|
||||||
|
28.0,
|
||||||
|
1,
|
||||||
|
104
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"input_csv_sha256": "38beb51c87acae40c3226599d6819cd6b9f984b39428ba84d1995257ce2bbc1c",
|
||||||
|
"output_csv_sha256": "cb847cc137aee4433d9d6b70a1a1c74d02a70325c2ffe771130d6626449a40d4"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"window": 4,
|
||||||
|
"values": [
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
4,
|
||||||
|
5,
|
||||||
|
6,
|
||||||
|
7,
|
||||||
|
8,
|
||||||
|
9,
|
||||||
|
10,
|
||||||
|
11,
|
||||||
|
11,
|
||||||
|
14,
|
||||||
|
13,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
19,
|
||||||
|
17,
|
||||||
|
18,
|
||||||
|
17,
|
||||||
|
16,
|
||||||
|
13,
|
||||||
|
12,
|
||||||
|
15,
|
||||||
|
16,
|
||||||
|
19,
|
||||||
|
21,
|
||||||
|
23,
|
||||||
|
25,
|
||||||
|
23,
|
||||||
|
25,
|
||||||
|
24,
|
||||||
|
24,
|
||||||
|
22,
|
||||||
|
21,
|
||||||
|
21,
|
||||||
|
19,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
16,
|
||||||
|
13,
|
||||||
|
12,
|
||||||
|
15,
|
||||||
|
15,
|
||||||
|
13,
|
||||||
|
15,
|
||||||
|
12,
|
||||||
|
15,
|
||||||
|
17,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
20,
|
||||||
|
18,
|
||||||
|
16,
|
||||||
|
16,
|
||||||
|
16,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
16,
|
||||||
|
19,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
23,
|
||||||
|
21,
|
||||||
|
24,
|
||||||
|
24,
|
||||||
|
26,
|
||||||
|
29,
|
||||||
|
31,
|
||||||
|
32,
|
||||||
|
29,
|
||||||
|
26,
|
||||||
|
26,
|
||||||
|
26,
|
||||||
|
29,
|
||||||
|
31,
|
||||||
|
32,
|
||||||
|
29,
|
||||||
|
29,
|
||||||
|
28,
|
||||||
|
25,
|
||||||
|
22,
|
||||||
|
25,
|
||||||
|
22,
|
||||||
|
20,
|
||||||
|
22,
|
||||||
|
25,
|
||||||
|
27,
|
||||||
|
24,
|
||||||
|
22,
|
||||||
|
25,
|
||||||
|
22,
|
||||||
|
24,
|
||||||
|
22,
|
||||||
|
19,
|
||||||
|
20,
|
||||||
|
21,
|
||||||
|
19,
|
||||||
|
19,
|
||||||
|
22,
|
||||||
|
24,
|
||||||
|
25,
|
||||||
|
24,
|
||||||
|
25,
|
||||||
|
26,
|
||||||
|
23,
|
||||||
|
23,
|
||||||
|
24,
|
||||||
|
26
|
||||||
|
],
|
||||||
|
"events": [
|
||||||
|
[
|
||||||
|
17,
|
||||||
|
19.0,
|
||||||
|
-1,
|
||||||
|
23
|
||||||
|
],
|
||||||
|
[
|
||||||
|
23,
|
||||||
|
12.0,
|
||||||
|
1,
|
||||||
|
27
|
||||||
|
],
|
||||||
|
[
|
||||||
|
29,
|
||||||
|
25.0,
|
||||||
|
-1,
|
||||||
|
35
|
||||||
|
],
|
||||||
|
[
|
||||||
|
47,
|
||||||
|
12.0,
|
||||||
|
1,
|
||||||
|
51
|
||||||
|
],
|
||||||
|
[
|
||||||
|
51,
|
||||||
|
21.0,
|
||||||
|
-1,
|
||||||
|
55
|
||||||
|
],
|
||||||
|
[
|
||||||
|
61,
|
||||||
|
16.0,
|
||||||
|
1,
|
||||||
|
67
|
||||||
|
],
|
||||||
|
[
|
||||||
|
72,
|
||||||
|
32.0,
|
||||||
|
-1,
|
||||||
|
79
|
||||||
|
],
|
||||||
|
[
|
||||||
|
74,
|
||||||
|
26.0,
|
||||||
|
1,
|
||||||
|
79
|
||||||
|
],
|
||||||
|
[
|
||||||
|
79,
|
||||||
|
32.0,
|
||||||
|
-1,
|
||||||
|
83
|
||||||
|
],
|
||||||
|
[
|
||||||
|
87,
|
||||||
|
20.0,
|
||||||
|
1,
|
||||||
|
91
|
||||||
|
],
|
||||||
|
[
|
||||||
|
90,
|
||||||
|
27.0,
|
||||||
|
-1,
|
||||||
|
95
|
||||||
|
],
|
||||||
|
[
|
||||||
|
97,
|
||||||
|
19.0,
|
||||||
|
1,
|
||||||
|
103
|
||||||
|
],
|
||||||
|
[
|
||||||
|
107,
|
||||||
|
26.0,
|
||||||
|
-1,
|
||||||
|
111
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"input_csv_sha256": "303cbcb636decdd1a7c6b3841ea696fec0881a95023eeb7af71132a501b53e83",
|
||||||
|
"output_csv_sha256": "df8be88430783a0669fb9224952fe7a26f4fcf18e3f242e425b7279e75f69850"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"window": 7,
|
||||||
|
"values": [
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
4,
|
||||||
|
5,
|
||||||
|
6,
|
||||||
|
7,
|
||||||
|
8,
|
||||||
|
9,
|
||||||
|
10,
|
||||||
|
11,
|
||||||
|
12,
|
||||||
|
13,
|
||||||
|
14,
|
||||||
|
15,
|
||||||
|
16,
|
||||||
|
17,
|
||||||
|
18,
|
||||||
|
19,
|
||||||
|
20,
|
||||||
|
21,
|
||||||
|
24,
|
||||||
|
27,
|
||||||
|
29,
|
||||||
|
26,
|
||||||
|
23,
|
||||||
|
25,
|
||||||
|
25,
|
||||||
|
24,
|
||||||
|
26,
|
||||||
|
23,
|
||||||
|
22,
|
||||||
|
19,
|
||||||
|
21,
|
||||||
|
21,
|
||||||
|
18,
|
||||||
|
21,
|
||||||
|
21,
|
||||||
|
18,
|
||||||
|
15,
|
||||||
|
14,
|
||||||
|
14,
|
||||||
|
11,
|
||||||
|
10,
|
||||||
|
7,
|
||||||
|
6,
|
||||||
|
5,
|
||||||
|
2,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
-2,
|
||||||
|
-2,
|
||||||
|
-2,
|
||||||
|
-5,
|
||||||
|
-4,
|
||||||
|
-3,
|
||||||
|
-3,
|
||||||
|
-1,
|
||||||
|
-4,
|
||||||
|
-1,
|
||||||
|
0,
|
||||||
|
3,
|
||||||
|
3,
|
||||||
|
2,
|
||||||
|
4,
|
||||||
|
3,
|
||||||
|
1,
|
||||||
|
-1,
|
||||||
|
-1,
|
||||||
|
1,
|
||||||
|
2,
|
||||||
|
2,
|
||||||
|
3,
|
||||||
|
2,
|
||||||
|
4,
|
||||||
|
3,
|
||||||
|
1,
|
||||||
|
4,
|
||||||
|
3,
|
||||||
|
2,
|
||||||
|
0,
|
||||||
|
-1,
|
||||||
|
-3,
|
||||||
|
-6,
|
||||||
|
-3,
|
||||||
|
-2,
|
||||||
|
-5,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
1,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
-1,
|
||||||
|
-2,
|
||||||
|
-2,
|
||||||
|
-5,
|
||||||
|
-5,
|
||||||
|
-6,
|
||||||
|
-4,
|
||||||
|
-2,
|
||||||
|
0,
|
||||||
|
-2,
|
||||||
|
-1,
|
||||||
|
-2,
|
||||||
|
-3,
|
||||||
|
-1,
|
||||||
|
1,
|
||||||
|
3,
|
||||||
|
5,
|
||||||
|
3,
|
||||||
|
2,
|
||||||
|
-1,
|
||||||
|
-4,
|
||||||
|
-7,
|
||||||
|
-5,
|
||||||
|
-4,
|
||||||
|
-5,
|
||||||
|
-7,
|
||||||
|
-7,
|
||||||
|
-7
|
||||||
|
],
|
||||||
|
"events": [
|
||||||
|
[
|
||||||
|
24,
|
||||||
|
29.0,
|
||||||
|
-1,
|
||||||
|
34
|
||||||
|
],
|
||||||
|
[
|
||||||
|
54,
|
||||||
|
-5.0,
|
||||||
|
1,
|
||||||
|
62
|
||||||
|
],
|
||||||
|
[
|
||||||
|
75,
|
||||||
|
4.0,
|
||||||
|
-1,
|
||||||
|
83
|
||||||
|
],
|
||||||
|
[
|
||||||
|
84,
|
||||||
|
-6.0,
|
||||||
|
1,
|
||||||
|
97
|
||||||
|
],
|
||||||
|
[
|
||||||
|
90,
|
||||||
|
1.0,
|
||||||
|
-1,
|
||||||
|
97
|
||||||
|
],
|
||||||
|
[
|
||||||
|
98,
|
||||||
|
-6.0,
|
||||||
|
1,
|
||||||
|
104
|
||||||
|
],
|
||||||
|
[
|
||||||
|
109,
|
||||||
|
5.0,
|
||||||
|
-1,
|
||||||
|
118
|
||||||
|
]
|
||||||
|
],
|
||||||
|
"input_csv_sha256": "01a6232710be1d4dd606fb077855c424e1b43b17ec602c59b28b6db4f87249db",
|
||||||
|
"output_csv_sha256": "37fbaaf015d2020eff07e62c2aaeb2de0e4145dea85403c29a98dbaed94cf3d9"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,771 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Native SHM-to-regime contracts, using synthetic source publications.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator, Callable
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Any, Never
|
||||||
|
import json
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
import trio
|
||||||
|
from tractor.trionics import Lagged
|
||||||
|
from trio.testing import wait_all_tasks_blocked
|
||||||
|
|
||||||
|
from piker.fsp._regime import (
|
||||||
|
CloseSample,
|
||||||
|
MultiTimeframeTrend,
|
||||||
|
RegimeSnapshot,
|
||||||
|
)
|
||||||
|
from piker.fsp._regime_feed import (
|
||||||
|
ClosedSourceRows,
|
||||||
|
SourceChanged,
|
||||||
|
consume_regime_source,
|
||||||
|
copy_closed_rows,
|
||||||
|
run_regime_feed,
|
||||||
|
serve_regimes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FakeSource:
|
||||||
|
'''
|
||||||
|
Model absolute bounds independently of mutable backing storage.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
first: int = 0,
|
||||||
|
last: int = 6,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Seed indexed rows independently of their published bounds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=first)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(value=last)
|
||||||
|
self._array: np.ndarray = np.zeros(128, dtype=[
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('close', '<f8'),
|
||||||
|
])
|
||||||
|
self._array['index'] = np.arange(128)
|
||||||
|
self._array['time'] = np.arange(128)
|
||||||
|
self._array['close'] = 100 + np.sin(np.arange(128) / 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_closed_copy_excludes_mutable_tail_and_detaches_memory(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Treating the mutable source tail as finalized leaks provisional
|
||||||
|
closes into regime decisions. Make that tail NaN and verify the
|
||||||
|
detached prefix remains valid, immutable and independent of a
|
||||||
|
later shared-memory write. Absolute first/last bounds survive.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: FakeSource = FakeSource(first=2, last=8)
|
||||||
|
source._array['close'][7] = np.nan
|
||||||
|
copied: ClosedSourceRows = copy_closed_rows(source)
|
||||||
|
assert (copied.first, copied.last, copied.start) == (2, 8, 2)
|
||||||
|
assert len(copied.rows) == 5
|
||||||
|
assert copied.rows['index'][-1] == 6
|
||||||
|
assert not copied.rows.flags.writeable
|
||||||
|
original: float = copied.rows['close'][0]
|
||||||
|
source._array['close'][2] = 999
|
||||||
|
assert copied.rows['close'][0] == original
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('first,last', [(0, 8), (2, 6)])
|
||||||
|
def test_changed_bounds_reject_mixed_copy(
|
||||||
|
first: int,
|
||||||
|
last: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A source prepend or truncation between the first bound read and
|
||||||
|
array access invalidates the copied range. Publish that change
|
||||||
|
from a controlled _array property access; the second bound check
|
||||||
|
must reject the candidate before any tracker can consume it.
|
||||||
|
|
||||||
|
'''
|
||||||
|
class ChangingSource:
|
||||||
|
'''
|
||||||
|
Trigger a deterministic source prepend while copying begins.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self) -> None:
|
||||||
|
'''
|
||||||
|
Retain a prefix whose origin or last bound will change.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=2)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(value=8)
|
||||||
|
self.storage: np.ndarray = FakeSource()._array
|
||||||
|
|
||||||
|
@property
|
||||||
|
def _array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Publish the changed origin after capturing old bounds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first.value = first
|
||||||
|
self._last.value = last
|
||||||
|
return self.storage
|
||||||
|
|
||||||
|
with pytest.raises(SourceChanged, match='changed during copy'):
|
||||||
|
copy_closed_rows(ChangingSource())
|
||||||
|
|
||||||
|
|
||||||
|
def test_append_during_copy_preserves_closed_prefix() -> None:
|
||||||
|
'''
|
||||||
|
A high-frequency producer may append during every large history
|
||||||
|
copy. Its old closed prefix remains stable because the captured
|
||||||
|
mutable tail was excluded. Accept the detached prefix with its
|
||||||
|
original last bound, then consume the newly closed suffix once
|
||||||
|
without duplicating observations or starving bootstrap retries.
|
||||||
|
|
||||||
|
'''
|
||||||
|
class AppendingSource:
|
||||||
|
'''
|
||||||
|
Publish two rows after the reader captures initial bounds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self) -> None:
|
||||||
|
'''
|
||||||
|
Retain closed rows before publishing an appended suffix.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=0)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(value=6)
|
||||||
|
self.storage: np.ndarray = FakeSource()._array
|
||||||
|
|
||||||
|
@property
|
||||||
|
def _array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Preserve the existing prefix while exposing a new tail.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._last.value = 8
|
||||||
|
return self.storage
|
||||||
|
|
||||||
|
source: AppendingSource = AppendingSource()
|
||||||
|
copied: ClosedSourceRows = copy_closed_rows(source)
|
||||||
|
assert (copied.first, copied.last, copied.start) == (0, 6, 0)
|
||||||
|
assert copied.rows['index'].tolist() == [0, 1, 2, 3, 4]
|
||||||
|
suffix: ClosedSourceRows = copy_closed_rows(
|
||||||
|
source, start=copied.last - 1,
|
||||||
|
)
|
||||||
|
assert (suffix.first, suffix.last, suffix.start) == (0, 8, 5)
|
||||||
|
assert suffix.rows['index'].tolist() == [5, 6]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('first,last', [(-1, 6), (8, 6), (0, 129)])
|
||||||
|
def test_stable_invalid_bounds_fail_instead_of_spinning(
|
||||||
|
first: int,
|
||||||
|
last: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Stable invalid bounds cannot become a usable source through a
|
||||||
|
tight retry loop. Treat negative, reversed and out-of-capacity
|
||||||
|
ranges as source errors, distinguishing them from bounds that
|
||||||
|
changed during the two-read copy validation window.
|
||||||
|
|
||||||
|
'''
|
||||||
|
with pytest.raises(ValueError, match='readable range'):
|
||||||
|
copy_closed_rows(FakeSource(first, last))
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsigned_coordinate_overflow_is_rejected_before_cast(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Converting uint64 coordinates to the core's int64 storage can
|
||||||
|
wrap an oversized value negative. Put that value last in a valid
|
||||||
|
increasing finalized prefix so ordering checks cannot mask the
|
||||||
|
overflow; source validation must reject it before replay begins.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: FakeSource = FakeSource(last=3)
|
||||||
|
source._array = source._array.astype([
|
||||||
|
('index', '<u8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('close', '<f8'),
|
||||||
|
])
|
||||||
|
source._array['index'][1] = np.uint64(2 ** 63)
|
||||||
|
with pytest.raises(ValueError, match='invalid finalized'):
|
||||||
|
copy_closed_rows(source)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('field,value', [
|
||||||
|
('close', float('nan')),
|
||||||
|
('close', float('inf')),
|
||||||
|
('time', float('nan')),
|
||||||
|
('time', 0),
|
||||||
|
('index', -1),
|
||||||
|
])
|
||||||
|
def test_bad_finalized_source_is_rejected_before_consumption(
|
||||||
|
field: str,
|
||||||
|
value: float,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Invalid finalized rows must not partially warm a tracker. Corrupt
|
||||||
|
a middle row and require validation of the whole detached range
|
||||||
|
before it can be handed to the recurrence; mutable-tail exclusion
|
||||||
|
is covered separately because unfinished rows may be invalid.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: FakeSource = FakeSource()
|
||||||
|
source._array[field][2] = value
|
||||||
|
with pytest.raises(ValueError, match='invalid finalized'):
|
||||||
|
copy_closed_rows(source)
|
||||||
|
|
||||||
|
|
||||||
|
def test_bootstrap_and_skipped_steps_emit_only_latest_snapshot(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A missed sampler wakeup must not skip finalized closes, and a
|
||||||
|
bootstrap must not create one HTTP job per historical bar.
|
||||||
|
Load five closed rows, mutate the open sixth row, then append
|
||||||
|
three rows with one wakeup. Exactly two snapshots must cover the
|
||||||
|
complete prefixes; duplicate and foreign messages produce none.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published: list[RegimeSnapshot] = []
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Drive source publication and wakeups in a controlled order.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: FakeSource = FakeSource()
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', periods=(1, 2),
|
||||||
|
)
|
||||||
|
send: trio.MemorySendChannel
|
||||||
|
receive: trio.MemoryReceiveChannel
|
||||||
|
send, receive = trio.open_memory_channel(0)
|
||||||
|
first: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
def submit(snapshot: RegimeSnapshot) -> None:
|
||||||
|
'''
|
||||||
|
Observe publication synchronously at the admission point.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published.append(snapshot)
|
||||||
|
first.set()
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(
|
||||||
|
consume_regime_source,
|
||||||
|
source, receive, tracker, submit,
|
||||||
|
)
|
||||||
|
await first.wait()
|
||||||
|
source._array['close'][5] = 500
|
||||||
|
await send.send({'period': 1})
|
||||||
|
await send.send({'period': 60})
|
||||||
|
await send.send({'backfilling': ['other.market', 1]})
|
||||||
|
await send.send({'backfilling': ['test.market', 60]})
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert len(published) == 1
|
||||||
|
source._last.value = 9
|
||||||
|
await send.send({'period': 1})
|
||||||
|
await send.aclose()
|
||||||
|
trio.run(scenario)
|
||||||
|
assert len(published) == 2
|
||||||
|
assert published[0].sequence == 5
|
||||||
|
assert published[1].sequence == 8
|
||||||
|
assert published[1].as_of == 8
|
||||||
|
assert published[1].revision == 0
|
||||||
|
closes: tuple[CloseSample, ...] = (
|
||||||
|
published[1].timeframes[0].recent_closes
|
||||||
|
)
|
||||||
|
assert closes[5].price == 500
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
'change', ['in_place', 'prepend', 'truncate'],
|
||||||
|
)
|
||||||
|
def test_history_changes_revoke_and_replay(change: str) -> None:
|
||||||
|
'''
|
||||||
|
Appending revised history to warm recurrence state is incorrect,
|
||||||
|
including repairs that leave both bounds unchanged. Publish a
|
||||||
|
source change and verify immediate revision invalidation precedes
|
||||||
|
its replacement snapshot. The rebuilt result must equal replay
|
||||||
|
through a fresh tracker over exactly the corrected closed prefix.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published: list[RegimeSnapshot] = []
|
||||||
|
order: list[tuple[str, int]] = []
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Compare each revised output with an independent full replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: FakeSource = FakeSource(first=2, last=20)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', periods=(1, 2),
|
||||||
|
)
|
||||||
|
send: trio.MemorySendChannel
|
||||||
|
receive: trio.MemoryReceiveChannel
|
||||||
|
send, receive = trio.open_memory_channel(0)
|
||||||
|
first: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
def submit(snapshot: RegimeSnapshot) -> None:
|
||||||
|
'''
|
||||||
|
Preserve publication order and source generation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published.append(snapshot)
|
||||||
|
order.append(('snapshot', snapshot.revision))
|
||||||
|
first.set()
|
||||||
|
|
||||||
|
def invalidate(revision: int) -> None:
|
||||||
|
'''
|
||||||
|
Record when old pending judgments become inadmissible.
|
||||||
|
|
||||||
|
'''
|
||||||
|
order.append(('invalidate', revision))
|
||||||
|
|
||||||
|
async def consume() -> None:
|
||||||
|
'''
|
||||||
|
Attach both source callbacks to one consumer lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
await consume_regime_source(
|
||||||
|
source, receive, tracker, submit,
|
||||||
|
invalidate=invalidate,
|
||||||
|
)
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(consume)
|
||||||
|
await first.wait()
|
||||||
|
if change == 'in_place':
|
||||||
|
source._array['close'][4] = 200
|
||||||
|
await send.send({
|
||||||
|
'backfilling': ['test.market', 1],
|
||||||
|
})
|
||||||
|
else:
|
||||||
|
if change == 'prepend':
|
||||||
|
source._first.value = 0
|
||||||
|
else:
|
||||||
|
source._last.value = 10
|
||||||
|
await send.send({'period': 1})
|
||||||
|
await send.aclose()
|
||||||
|
reference: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', periods=(1, 2), revision=1,
|
||||||
|
)
|
||||||
|
row: np.void
|
||||||
|
for row in copy_closed_rows(source).rows:
|
||||||
|
expected: RegimeSnapshot = reference.push(
|
||||||
|
float(row['time']), float(row['close']),
|
||||||
|
index=int(row['index']),
|
||||||
|
)
|
||||||
|
assert published[-1] == expected
|
||||||
|
|
||||||
|
trio.run(scenario)
|
||||||
|
assert order == [
|
||||||
|
('snapshot', 0), ('invalidate', 1), ('snapshot', 1),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_revision_during_bootstrap_suppresses_old_publication(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Losing a backfill during bootstrap publishes a mixed-generation
|
||||||
|
classification that ordinary bound checks cannot detect.
|
||||||
|
``SignallingTracker.push()`` sets ``entered`` after one row;
|
||||||
|
``consume_regime_source()`` yields at every replay row so the
|
||||||
|
``scenario()`` task can repair a close and send its notice.
|
||||||
|
``_Notices.collect()`` must invalidate that revision before the
|
||||||
|
replay publishes. One replacement snapshot, its corrected close
|
||||||
|
and full sequence prove old evidence never reaches ``published``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published: list[RegimeSnapshot] = []
|
||||||
|
invalidated: list[int] = []
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Synchronize repair with the first chunk's cooperative yield.
|
||||||
|
|
||||||
|
'''
|
||||||
|
entered: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
class SignallingTracker(MultiTimeframeTrend):
|
||||||
|
'''
|
||||||
|
Expose the start of bootstrap without adding I/O to push.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def push(
|
||||||
|
self,
|
||||||
|
*args: Any,
|
||||||
|
**kwargs: Any,
|
||||||
|
) -> RegimeSnapshot:
|
||||||
|
'''
|
||||||
|
Signal after the first historical row is consumed.
|
||||||
|
|
||||||
|
'''
|
||||||
|
snapshot: RegimeSnapshot = super().push(
|
||||||
|
*args, **kwargs,
|
||||||
|
)
|
||||||
|
entered.set()
|
||||||
|
return snapshot
|
||||||
|
|
||||||
|
source: FakeSource = FakeSource(last=100)
|
||||||
|
tracker: SignallingTracker = SignallingTracker(
|
||||||
|
'test.market', periods=(1,), history_limit=128,
|
||||||
|
)
|
||||||
|
send: trio.MemorySendChannel
|
||||||
|
receive: trio.MemoryReceiveChannel
|
||||||
|
send, receive = trio.open_memory_channel(0)
|
||||||
|
|
||||||
|
async def consume() -> None:
|
||||||
|
'''
|
||||||
|
Yield each row so the controlled repair reaches replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
await consume_regime_source(
|
||||||
|
source, receive, tracker, published.append,
|
||||||
|
invalidate=invalidated.append,
|
||||||
|
replay_chunk=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(consume)
|
||||||
|
await entered.wait()
|
||||||
|
source._array['close'][0] = 999
|
||||||
|
await send.send({'backfilling': ['test.market', 1]})
|
||||||
|
await send.aclose()
|
||||||
|
|
||||||
|
trio.run(scenario)
|
||||||
|
assert invalidated == [1]
|
||||||
|
assert len(published) == 1
|
||||||
|
assert published[0].revision == 1
|
||||||
|
assert published[0].sequence == 99
|
||||||
|
assert published[0].timeframes[0].recent_closes[0].price == 999
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('quote_lag', [False, True])
|
||||||
|
@pytest.mark.parametrize('source_age', [None, 0, 60, -10])
|
||||||
|
def test_native_wrapper_subscribes_filters_age_and_drains_quotes(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
quote_lag: bool,
|
||||||
|
source_age: float|None,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Subscribing after history capture loses in-place revisions in the
|
||||||
|
startup gap. ``GuardedSource._array`` asserts the subscription is
|
||||||
|
active before every read. ``sampler_context()`` then replaces
|
||||||
|
``Flume.stream`` with ``OtherConsumer``; any later lookup of that
|
||||||
|
shared field fails instead of stealing another consumer's stream.
|
||||||
|
|
||||||
|
The independent cases cover these admission contracts:
|
||||||
|
- ``LaggingQuotes`` raises ``Lagged`` once. It sets ``recovered``
|
||||||
|
only when the quote drain resumes; SHM processing must survive.
|
||||||
|
- Old/future source clocks must withhold model evidence even
|
||||||
|
though submission occurs locally now; no age limit permits
|
||||||
|
offline replay. ``published`` distinguishes those outcomes.
|
||||||
|
- ``feed_context()`` rejects a requested quote throttle: feeds
|
||||||
|
cached solely by market must retain their shared quote policy.
|
||||||
|
|
||||||
|
``finish()`` waits on ``read_source`` and, when needed,
|
||||||
|
``recovered``, then closes sampling after tasks block. Context
|
||||||
|
exit order proves the native wrapper releases its subscription
|
||||||
|
before the feed, without any broker, actor service or API key.
|
||||||
|
|
||||||
|
'''
|
||||||
|
order: list[str] = []
|
||||||
|
subscribed: bool = False
|
||||||
|
published: list[RegimeSnapshot] = []
|
||||||
|
read_source: trio.Event
|
||||||
|
flume_ref: SimpleNamespace|None = None
|
||||||
|
|
||||||
|
class GuardedSource:
|
||||||
|
'''
|
||||||
|
Require the native sampler subscription before source access.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self) -> None:
|
||||||
|
'''
|
||||||
|
Seed a readable range whose array access is guarded.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=0)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(value=6)
|
||||||
|
self.storage: np.ndarray = FakeSource()._array
|
||||||
|
|
||||||
|
@property
|
||||||
|
def _array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Detect a source read before entering the subscription.
|
||||||
|
|
||||||
|
'''
|
||||||
|
assert subscribed
|
||||||
|
order.append('read_source')
|
||||||
|
read_source.set()
|
||||||
|
return self.storage
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Close the synthetic sampler to end the actual wrapper.
|
||||||
|
|
||||||
|
'''
|
||||||
|
nonlocal read_source
|
||||||
|
read_source = trio.Event()
|
||||||
|
recovered: trio.Event = trio.Event()
|
||||||
|
send: trio.MemorySendChannel
|
||||||
|
receive: trio.MemoryReceiveChannel
|
||||||
|
send, receive = trio.open_memory_channel(0)
|
||||||
|
quote_send: trio.MemorySendChannel
|
||||||
|
quote_receive: trio.MemoryReceiveChannel
|
||||||
|
quote_send, quote_receive = trio.open_memory_channel(0)
|
||||||
|
|
||||||
|
class LaggingQuotes:
|
||||||
|
'''
|
||||||
|
Lose one discarded quote before resuming a healthy wait.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __aiter__(self) -> 'LaggingQuotes':
|
||||||
|
'''
|
||||||
|
Return the resumable stream after a Lagged exception.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def __anext__(self) -> dict:
|
||||||
|
'''
|
||||||
|
Force one overrun, then prove the drain resumed.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if 'quote_lagged' not in order:
|
||||||
|
order.append('quote_lagged')
|
||||||
|
raise Lagged('synthetic discarded quote')
|
||||||
|
recovered.set()
|
||||||
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def feed_context(
|
||||||
|
**kwargs: object,
|
||||||
|
) -> AsyncIterator[SimpleNamespace]:
|
||||||
|
'''
|
||||||
|
Expose a source Flume with a cancellable quote stream.
|
||||||
|
|
||||||
|
'''
|
||||||
|
nonlocal flume_ref
|
||||||
|
assert 'tick_throttle' not in kwargs
|
||||||
|
order.append('open_feed')
|
||||||
|
flume_ref = SimpleNamespace(
|
||||||
|
rt_sample_period_s=1,
|
||||||
|
rt_shm=GuardedSource(),
|
||||||
|
stream=(
|
||||||
|
LaggingQuotes() if quote_lag else quote_receive
|
||||||
|
),
|
||||||
|
)
|
||||||
|
yield SimpleNamespace(flumes={
|
||||||
|
'test.market': flume_ref,
|
||||||
|
})
|
||||||
|
await quote_send.aclose()
|
||||||
|
order.append('close_feed')
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def sampler_context(
|
||||||
|
**kwargs: object,
|
||||||
|
) -> AsyncIterator[trio.MemoryReceiveChannel]:
|
||||||
|
'''
|
||||||
|
Mark exactly when the sampler subscription begins.
|
||||||
|
|
||||||
|
'''
|
||||||
|
nonlocal subscribed
|
||||||
|
subscribed = True
|
||||||
|
order.append('subscribe')
|
||||||
|
|
||||||
|
class OtherConsumer:
|
||||||
|
'''
|
||||||
|
Model a shared Flume's receiver ownership change.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __aiter__(self) -> Never:
|
||||||
|
'''
|
||||||
|
Detect consuming a later caller's broadcast view.
|
||||||
|
|
||||||
|
'''
|
||||||
|
pytest.fail('drain stole another quote receiver')
|
||||||
|
|
||||||
|
assert flume_ref is not None
|
||||||
|
flume_ref.stream = OtherConsumer()
|
||||||
|
try:
|
||||||
|
yield receive
|
||||||
|
finally:
|
||||||
|
subscribed = False
|
||||||
|
order.append('unsubscribe')
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'piker.data.feed.maybe_open_feed', feed_context,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'piker.data._sampling.open_sample_stream',
|
||||||
|
sampler_context,
|
||||||
|
)
|
||||||
|
if source_age is not None:
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'piker.fsp._regime_feed.time.time',
|
||||||
|
lambda: 5 + source_age,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def finish() -> None:
|
||||||
|
'''
|
||||||
|
Close the sampler after processing and draining quotes.
|
||||||
|
|
||||||
|
'''
|
||||||
|
await read_source.wait()
|
||||||
|
if quote_lag:
|
||||||
|
await recovered.wait()
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
await send.aclose()
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(finish)
|
||||||
|
await run_regime_feed(
|
||||||
|
'test.market', published.append,
|
||||||
|
max_source_age=None if source_age is None else 5,
|
||||||
|
)
|
||||||
|
|
||||||
|
trio.run(scenario)
|
||||||
|
assert len(published) == (source_age in (None, 0))
|
||||||
|
assert order.index('subscribe') < order.index('read_source')
|
||||||
|
assert order[-2:] == ['unsubscribe', 'close_feed']
|
||||||
|
|
||||||
|
|
||||||
|
def test_service_composes_feed_and_classifier_in_one_lifetime(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A prepared provider adapter needs concrete feed-to-classifier
|
||||||
|
wiring and shared cancellation ownership. Replace the broker
|
||||||
|
source with a synthetic callback producer but retain the actual
|
||||||
|
HTTP client, worker and service nursery. One corrected snapshot
|
||||||
|
must produce one validated judgment; normal source exit must
|
||||||
|
stop the classifier while leaving the caller's HTTP session open.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from piker.fsp._jev import (
|
||||||
|
DEFAULT_MODEL,
|
||||||
|
JevClassification,
|
||||||
|
JevClient,
|
||||||
|
RegimeClassifier,
|
||||||
|
)
|
||||||
|
|
||||||
|
published: list[JevClassification] = []
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
End the source only after its judgment is published.
|
||||||
|
|
||||||
|
'''
|
||||||
|
delivered: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
async def source(
|
||||||
|
fqme: str,
|
||||||
|
submit: Callable[[RegimeSnapshot], object],
|
||||||
|
*,
|
||||||
|
invalidate: Callable[[int], object],
|
||||||
|
**kwargs: object,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Replay one revision through the bound service callbacks.
|
||||||
|
|
||||||
|
'''
|
||||||
|
assert fqme == 'test.market'
|
||||||
|
assert kwargs['max_source_age'] == 5
|
||||||
|
invalidate(1)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
fqme, periods=(1,), revision=1,
|
||||||
|
)
|
||||||
|
second: int
|
||||||
|
for second in range(8):
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(
|
||||||
|
second, second,
|
||||||
|
)
|
||||||
|
submit(snapshot)
|
||||||
|
await delivered.wait()
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
'''
|
||||||
|
Return a documented Choice response without real HTTP.
|
||||||
|
|
||||||
|
'''
|
||||||
|
request_body: dict = json.loads(request.content)
|
||||||
|
answers: dict[str, dict] = {}
|
||||||
|
name: str
|
||||||
|
question: dict
|
||||||
|
for name, question in request_body['questions'].items():
|
||||||
|
probabilities: dict[str, float] = {}
|
||||||
|
label: str
|
||||||
|
for label in question['criteria']:
|
||||||
|
probabilities[label] = float(label == 'uptrend')
|
||||||
|
answers[name] = {
|
||||||
|
'type': 'choice',
|
||||||
|
'choice': 'uptrend',
|
||||||
|
'probabilities': probabilities,
|
||||||
|
'confidence': 1,
|
||||||
|
}
|
||||||
|
return httpx.Response(200, json={
|
||||||
|
'model': DEFAULT_MODEL,
|
||||||
|
'answers': answers,
|
||||||
|
'usage': {
|
||||||
|
'input_tokens': 100,
|
||||||
|
'output_tokens': 20,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
async def publish(result: JevClassification) -> None:
|
||||||
|
'''
|
||||||
|
Release source shutdown after receiving a typed judgment.
|
||||||
|
|
||||||
|
'''
|
||||||
|
published.append(result)
|
||||||
|
delivered.set()
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
'piker.fsp._regime_feed.run_regime_feed', source,
|
||||||
|
)
|
||||||
|
http: httpx.AsyncClient
|
||||||
|
async with httpx.AsyncClient(
|
||||||
|
transport=httpx.MockTransport(handler),
|
||||||
|
) as http:
|
||||||
|
client: JevClient = JevClient(http, 'offline-test-key')
|
||||||
|
worker: RegimeClassifier = await serve_regimes(
|
||||||
|
'test.market', client, publish,
|
||||||
|
)
|
||||||
|
assert not http.is_closed
|
||||||
|
assert worker.requests == 1
|
||||||
|
assert worker.published == 1
|
||||||
|
assert http.is_closed
|
||||||
|
|
||||||
|
trio.run(scenario)
|
||||||
|
assert len(published) == 1
|
||||||
|
assert published[0].market == 'test.market'
|
||||||
|
assert published[0].revision == 1
|
||||||
|
|
@ -0,0 +1,795 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Causality, streaming, and publication contracts for Trend Calculus.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator, Callable
|
||||||
|
from dataclasses import Field, fields
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Any, cast
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
import trio
|
||||||
|
from trio.testing import wait_all_tasks_blocked
|
||||||
|
from trio_typing import TaskStatus
|
||||||
|
|
||||||
|
from piker.data.ticktools import FeedQuote
|
||||||
|
from piker.fsp._api import (
|
||||||
|
FspSample,
|
||||||
|
FspStream,
|
||||||
|
FspYield,
|
||||||
|
_load_builtins,
|
||||||
|
)
|
||||||
|
from piker.fsp._engine import (
|
||||||
|
Cascade,
|
||||||
|
QuoteFrame,
|
||||||
|
_next_step_output,
|
||||||
|
_sample_wakeups,
|
||||||
|
_write_sample,
|
||||||
|
filter_quotes_by_sym,
|
||||||
|
)
|
||||||
|
from piker.fsp._trend import (
|
||||||
|
TrendCalculator,
|
||||||
|
TrendHistory,
|
||||||
|
TrendUpdate,
|
||||||
|
)
|
||||||
|
from piker.fsp._trend_fsp import trend_calculus
|
||||||
|
from tractor.ipc._shm import ShmArray
|
||||||
|
from tractor.trionics import Lagged
|
||||||
|
|
||||||
|
|
||||||
|
def test_split_preserves_both_reversals() -> None:
|
||||||
|
'''
|
||||||
|
A net unchanged direction can contain two opposing reversals.
|
||||||
|
|
||||||
|
The final outside window [2, 5] follows the rising window [4, 3].
|
||||||
|
Bridging its earlier extremum reveals a top at 4 and a bottom at
|
||||||
|
2, both detected on observation 5. A one-pulse or net-sign-only
|
||||||
|
implementation loses the top. Assert both pivot coordinates and
|
||||||
|
the later detection coordinate, including the historical fields.
|
||||||
|
|
||||||
|
'''
|
||||||
|
values: np.ndarray = np.array([1., 2., 4., 3., 2., 5.])
|
||||||
|
calc: TrendCalculator = TrendCalculator()
|
||||||
|
calc.extend(values[:-1], np.arange(5.))
|
||||||
|
update: TrendUpdate = calc.push(5., values[-1])
|
||||||
|
assert update.trends == (1, 0, 0, 0)
|
||||||
|
assert update.reversals['index'].tolist() == [2, 4]
|
||||||
|
assert update.reversals['direction'].tolist() == [-1, 1]
|
||||||
|
assert update.reversals['price'].tolist() == [4., 2.]
|
||||||
|
assert update.reversals['detected_index'].tolist() == [5, 5]
|
||||||
|
assert update.reversals['detected_time'].tolist() == [5., 5.]
|
||||||
|
history: TrendHistory = TrendCalculator().extend(
|
||||||
|
values, np.arange(6.),
|
||||||
|
)
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
history.pivot_prices[-1, 0], [4., 2.],
|
||||||
|
)
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
history.pivot_times[-1, 0], [2., 4.],
|
||||||
|
)
|
||||||
|
assert np.isnan(history.pivot_prices[:-1]).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_chunking_and_every_prefix() -> None:
|
||||||
|
'''
|
||||||
|
Future observations and transport chunking must not alter output.
|
||||||
|
|
||||||
|
Materialize a random walk with ties and a missing observation,
|
||||||
|
then compare every prefix and uneven continuation batches with a
|
||||||
|
single history scan. Compare pivot detection fields too, because
|
||||||
|
equal direction arrays alone can hide backdated reversals.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rng: np.random.Generator = np.random.default_rng(18)
|
||||||
|
values: np.ndarray = (
|
||||||
|
rng.integers(-2, 3, 80).cumsum().astype(float)
|
||||||
|
)
|
||||||
|
values[31] = np.nan
|
||||||
|
times: np.ndarray = np.arange(80.) * 0.125 + 1000.
|
||||||
|
indices: np.ndarray = np.arange(80, dtype=np.int64) + 12345
|
||||||
|
windows: tuple[int, ...] = (3, 2, 4)
|
||||||
|
full: TrendHistory = TrendCalculator(windows).extend(
|
||||||
|
values, times, indices,
|
||||||
|
)
|
||||||
|
cut: int
|
||||||
|
field: Field[Any]
|
||||||
|
for cut in range(81):
|
||||||
|
prefix: TrendHistory = TrendCalculator(windows).extend(
|
||||||
|
values[:cut], times[:cut], indices[:cut],
|
||||||
|
)
|
||||||
|
for field in fields(TrendHistory):
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
getattr(prefix, field.name),
|
||||||
|
getattr(full, field.name)[:cut],
|
||||||
|
)
|
||||||
|
|
||||||
|
calc: TrendCalculator = TrendCalculator(windows)
|
||||||
|
start: int = 0
|
||||||
|
for cut in (0, 1, 2, 7, 31, 32, 34, 60, 80):
|
||||||
|
chunk: TrendHistory = calc.extend(
|
||||||
|
values[start:cut], times[start:cut], indices[start:cut],
|
||||||
|
)
|
||||||
|
for field in fields(TrendHistory):
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
getattr(chunk, field.name),
|
||||||
|
getattr(full, field.name)[start:cut],
|
||||||
|
)
|
||||||
|
start = cut
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('window', [2, 3, 7])
|
||||||
|
@pytest.mark.parametrize('direction', [-1, 0, 1])
|
||||||
|
def test_warmup_flat_and_negative_values(
|
||||||
|
window: int,
|
||||||
|
direction: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Startup must not invent a trend from a synthetic zero-price bar.
|
||||||
|
|
||||||
|
Use negative, monotone or flat values through two full windows.
|
||||||
|
The first comparable pair establishes readiness and direction;
|
||||||
|
no sign change or initial pivot exists before that comparison.
|
||||||
|
|
||||||
|
'''
|
||||||
|
values: np.ndarray = (
|
||||||
|
np.arange(2 * window, dtype=float) * direction - 100
|
||||||
|
)
|
||||||
|
result: TrendHistory = TrendCalculator((window,)).extend(
|
||||||
|
values, np.arange(float(len(values))),
|
||||||
|
)
|
||||||
|
assert not result.ready[:-1].any()
|
||||||
|
assert result.ready[-1, 0]
|
||||||
|
assert not result.trends[:-1].any()
|
||||||
|
assert result.trends[-1, 0] == direction
|
||||||
|
assert np.isnan(result.pivot_prices).all()
|
||||||
|
|
||||||
|
|
||||||
|
def test_nan_resets_hierarchy_without_future_contamination() -> None:
|
||||||
|
'''
|
||||||
|
A missing value invalidates every recursive reversal order.
|
||||||
|
|
||||||
|
Build nontrivial history, append NaN, and compare the following
|
||||||
|
observations against a newly constructed calculator. Retaining
|
||||||
|
even a higher-order partial window would make this replay differ.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rng: np.random.Generator = np.random.default_rng(77)
|
||||||
|
calc: TrendCalculator = TrendCalculator((2, 2, 2))
|
||||||
|
calc.extend(rng.normal(size=200), np.arange(200.))
|
||||||
|
update: TrendUpdate = calc.push(200., np.nan)
|
||||||
|
assert update.trends == (0, 0, 0)
|
||||||
|
assert update.ready == (False, False, False)
|
||||||
|
assert not len(update.reversals)
|
||||||
|
values: np.ndarray = rng.normal(size=100)
|
||||||
|
times: np.ndarray = np.arange(201., 301.)
|
||||||
|
actual: TrendHistory = calc.extend(values, times)
|
||||||
|
expected: TrendHistory = TrendCalculator((2, 2, 2)).extend(
|
||||||
|
values, times,
|
||||||
|
)
|
||||||
|
field: Field[Any]
|
||||||
|
for field in fields(TrendHistory):
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
getattr(actual, field.name),
|
||||||
|
getattr(expected, field.name),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
'time,value,index',
|
||||||
|
[
|
||||||
|
(3., 1., 14),
|
||||||
|
(2., 1., 14),
|
||||||
|
(4., np.inf, 14),
|
||||||
|
(np.nan, 1., 14),
|
||||||
|
(4., 1., 13),
|
||||||
|
(4., 1., -1),
|
||||||
|
(4., 1., 2 ** 63),
|
||||||
|
(4., 1., np.uint64(2 ** 63)),
|
||||||
|
(4., 1., True),
|
||||||
|
(4., 1., np.bool_(True)),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_invalid_sample_is_atomic(
|
||||||
|
time: float,
|
||||||
|
value: float,
|
||||||
|
index: int,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Invalid coordinates must not corrupt a partially filled window.
|
||||||
|
|
||||||
|
Seed a calculator with four observations, attempt one rejected
|
||||||
|
sample, and compare continuation to an untouched state copy.
|
||||||
|
This checks that validation precedes recurrence mutation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
calc: TrendCalculator = TrendCalculator()
|
||||||
|
calc.extend(np.arange(4.), np.arange(4.), np.arange(10, 14))
|
||||||
|
untouched: TrendCalculator = calc.copy()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
calc.push(time, value, index=index)
|
||||||
|
actual: TrendUpdate = calc.push(4., -1., index=14)
|
||||||
|
expected: TrendUpdate = untouched.push(4., -1., index=14)
|
||||||
|
assert actual.trends == expected.trends
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
actual.reversals, expected.reversals,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_batch_is_atomic() -> None:
|
||||||
|
'''
|
||||||
|
A late malformed sample cannot partly advance a batch operation.
|
||||||
|
|
||||||
|
The duplicate final timestamp is detected before the first valid
|
||||||
|
row is consumed. A subsequent valid replay matches fresh state.
|
||||||
|
|
||||||
|
'''
|
||||||
|
calc: TrendCalculator = TrendCalculator()
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
calc.extend(np.arange(4.), np.array([1., 2., 3., 3.]))
|
||||||
|
actual: TrendHistory = calc.extend(np.arange(4.), np.arange(4.))
|
||||||
|
expected: TrendHistory = TrendCalculator().extend(
|
||||||
|
np.arange(4.), np.arange(4.),
|
||||||
|
)
|
||||||
|
np.testing.assert_array_equal(actual.trends, expected.trends)
|
||||||
|
|
||||||
|
|
||||||
|
def test_last_int64_index_is_exact_and_overflow_is_atomic() -> None:
|
||||||
|
'''
|
||||||
|
Explicit unsigned coordinates must not wrap in compiled state.
|
||||||
|
|
||||||
|
Observe the last representable coordinate using a NumPy uint64.
|
||||||
|
A subsequent automatic push or batch must reject overflow before
|
||||||
|
touching the partial window, and retain the exact last index.
|
||||||
|
|
||||||
|
'''
|
||||||
|
calc: TrendCalculator = TrendCalculator()
|
||||||
|
limit: int = np.iinfo(np.int64).max
|
||||||
|
result: TrendUpdate = calc.push(
|
||||||
|
1., 2., index=np.uint64(limit),
|
||||||
|
)
|
||||||
|
assert result.index == limit
|
||||||
|
assert isinstance(result.index, int)
|
||||||
|
before: np.ndarray = calc._bars.copy()
|
||||||
|
with pytest.raises(ValueError, match='int64'):
|
||||||
|
calc.push(2., 3.)
|
||||||
|
with pytest.raises(ValueError, match='overflow'):
|
||||||
|
calc.extend(np.array([3.]), np.array([2.]))
|
||||||
|
np.testing.assert_array_equal(calc._bars, before)
|
||||||
|
|
||||||
|
|
||||||
|
class MemorySource:
|
||||||
|
'''
|
||||||
|
Expose mutable source rows with absolute, nonzero SHM bounds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self, values: list[float]) -> None:
|
||||||
|
'''
|
||||||
|
Seed mutable rows with a nonzero absolute SHM origin.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=37)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(
|
||||||
|
value=37 + len(values),
|
||||||
|
)
|
||||||
|
self._array: np.ndarray = np.zeros(128, dtype=[
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('close', '<f8'),
|
||||||
|
])
|
||||||
|
self._array['index'] = np.arange(128)
|
||||||
|
self._array['time'] = np.arange(128.) + 1000.
|
||||||
|
self.array['close'] = values
|
||||||
|
|
||||||
|
@property
|
||||||
|
def array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Expose ``MemorySource._array`` between published bounds.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self._array[self._first.value:self._last.value]
|
||||||
|
|
||||||
|
|
||||||
|
def assert_sample_matches(
|
||||||
|
sample: FspSample,
|
||||||
|
rows: np.ndarray,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Compare every FSP field with a fresh source-prefix replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
history: TrendHistory = TrendCalculator().extend(
|
||||||
|
rows['close'], rows['time'], rows['index'],
|
||||||
|
)
|
||||||
|
assert sample.index == rows['index'][-1]
|
||||||
|
assert sample.time == rows['time'][-1]
|
||||||
|
assert sample.fields['ready'] == history.ready[-1].sum()
|
||||||
|
level: int
|
||||||
|
for level in range(4):
|
||||||
|
order: int = level + 1
|
||||||
|
name: str = 'trend' if level == 0 else f'trend_{order}'
|
||||||
|
assert sample.fields[name] == history.trends[-1, level]
|
||||||
|
side: int
|
||||||
|
label: str
|
||||||
|
for side, label in enumerate(('top', 'bottom')):
|
||||||
|
np.testing.assert_equal(
|
||||||
|
sample.fields[f'{label}_{order}'],
|
||||||
|
history.pivot_prices[-1, level, side],
|
||||||
|
)
|
||||||
|
np.testing.assert_equal(
|
||||||
|
sample.fields[f'{label}_time_{order}'],
|
||||||
|
history.pivot_times[-1, level, side],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fsp_preview_mutations_and_skipped_steps() -> None:
|
||||||
|
'''
|
||||||
|
Updating an open bar must replace its preview, not consume it.
|
||||||
|
|
||||||
|
Repeatedly mutate the current SHM close across both directions,
|
||||||
|
then advance by two rows before the next quote. Each FspSample
|
||||||
|
must equal a fresh batch replay, and its absolute row must match
|
||||||
|
the source. Quote payloads are deliberately empty wakeups so the
|
||||||
|
source state rather than trade arrivals must drive sampling.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.])
|
||||||
|
|
||||||
|
async def quotes() -> AsyncIterator[FeedQuote]:
|
||||||
|
'''
|
||||||
|
Provide payload-free wakeups for source
|
||||||
|
snapshot sampling.
|
||||||
|
|
||||||
|
'''
|
||||||
|
while True:
|
||||||
|
yield {}
|
||||||
|
|
||||||
|
stream: FspStream = trend_calculus(
|
||||||
|
quotes(), cast(ShmArray, shm),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
historical: FspYield = await anext(stream)
|
||||||
|
assert isinstance(historical, dict)
|
||||||
|
assert set(historical) == set(trend_calculus.outputs)
|
||||||
|
value: np.ndarray|None # noqa: F842
|
||||||
|
assert all(
|
||||||
|
value is not None and len(value) == 6
|
||||||
|
for value in historical.values()
|
||||||
|
)
|
||||||
|
close: float
|
||||||
|
for close in (100., -20., 5., 5.):
|
||||||
|
shm.array['close'][-1] = close
|
||||||
|
sample: FspYield = await anext(stream)
|
||||||
|
assert isinstance(sample, FspSample)
|
||||||
|
assert_sample_matches(sample, shm.array)
|
||||||
|
|
||||||
|
last: int = shm._last.value
|
||||||
|
shm._array['close'][last:last + 2] = [-4., -5.]
|
||||||
|
shm._last.value += 2
|
||||||
|
end: int
|
||||||
|
for end in (6, 7, 8):
|
||||||
|
sample: FspYield = await anext(stream)
|
||||||
|
assert isinstance(sample, FspSample)
|
||||||
|
assert_sample_matches(sample, shm.array[:end])
|
||||||
|
finally:
|
||||||
|
await stream.aclose()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fsp_empty_bootstrap_and_source_revision() -> None:
|
||||||
|
'''
|
||||||
|
Empty startup must accept later source rows without old state.
|
||||||
|
|
||||||
|
After the first sample, prepend changes the source origin. The
|
||||||
|
old generator must stop; restarting it recomputes the revised
|
||||||
|
history rather than reusing recurrence state from another origin.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: MemorySource = MemorySource([])
|
||||||
|
|
||||||
|
async def quotes() -> AsyncIterator[FeedQuote]:
|
||||||
|
'''
|
||||||
|
Provide payload-free wakeups for source
|
||||||
|
snapshot sampling.
|
||||||
|
|
||||||
|
'''
|
||||||
|
while True:
|
||||||
|
yield {}
|
||||||
|
|
||||||
|
stream: FspStream = trend_calculus(
|
||||||
|
quotes(), cast(ShmArray, shm),
|
||||||
|
)
|
||||||
|
historical: FspYield = await anext(stream)
|
||||||
|
assert isinstance(historical, dict)
|
||||||
|
value: np.ndarray|None # noqa: F842
|
||||||
|
assert all(
|
||||||
|
value is not None and not len(value)
|
||||||
|
for value in historical.values()
|
||||||
|
)
|
||||||
|
shm._array['close'][37] = 1.
|
||||||
|
shm._last.value += 1
|
||||||
|
sample: FspYield = await anext(stream)
|
||||||
|
assert isinstance(sample, FspSample)
|
||||||
|
assert_sample_matches(sample, shm.array)
|
||||||
|
shm._first.value -= 1
|
||||||
|
with pytest.raises(StopAsyncIteration):
|
||||||
|
await anext(stream)
|
||||||
|
restarted: FspStream = trend_calculus(
|
||||||
|
quotes(), cast(ShmArray, shm),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
await anext(restarted)
|
||||||
|
sample: FspYield = await anext(restarted)
|
||||||
|
assert isinstance(sample, FspSample)
|
||||||
|
assert_sample_matches(sample, shm.array)
|
||||||
|
finally:
|
||||||
|
await restarted.aclose()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_indexed_sample_waits_for_destination_step() -> None:
|
||||||
|
'''
|
||||||
|
A quote can reach the FSP before the sampler appends destination.
|
||||||
|
|
||||||
|
Block _write_sample with destination one row behind source and
|
||||||
|
wait until its task is suspended. Publishing the next destination
|
||||||
|
bound and setting its event permits one coherent write. The prior
|
||||||
|
row remains unchanged, proving no accidental previous-bar write.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
array: np.ndarray = np.zeros(16, dtype=[
|
||||||
|
('time', '<f8'),
|
||||||
|
('trend', '<f8'),
|
||||||
|
('ready', '<f8'),
|
||||||
|
])
|
||||||
|
dst: SimpleNamespace = SimpleNamespace(
|
||||||
|
_first=SimpleNamespace(value=7),
|
||||||
|
_last=SimpleNamespace(value=10),
|
||||||
|
_array=array,
|
||||||
|
)
|
||||||
|
casc: Cascade = cast(Cascade, SimpleNamespace(
|
||||||
|
dst=SimpleNamespace(rt_shm=dst),
|
||||||
|
sample_ready=None,
|
||||||
|
))
|
||||||
|
complete: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
async def publish() -> None:
|
||||||
|
'''
|
||||||
|
Wait for the destination bound, then signal publication.
|
||||||
|
|
||||||
|
'''
|
||||||
|
assert await _write_sample(casc, FspSample(
|
||||||
|
10, 1010., {'trend': -1., 'ready': 2.},
|
||||||
|
))
|
||||||
|
complete.set()
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(publish)
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert not complete.is_set()
|
||||||
|
assert not array['trend'].any()
|
||||||
|
dst._last.value += 1
|
||||||
|
assert casc.sample_ready is not None
|
||||||
|
casc.sample_ready.set()
|
||||||
|
await complete.wait()
|
||||||
|
assert tuple(array[9]) == (0., 0., 0.)
|
||||||
|
assert tuple(array[10]) == (1010., -1., 2.)
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_indexed_sample_wait_is_cancellable() -> None:
|
||||||
|
'''
|
||||||
|
History resync must cancel a compute task waiting on sampling.
|
||||||
|
|
||||||
|
Suspend indexed publication against an empty destination, cancel
|
||||||
|
its nursery, then advance that destination. No cancelled sample
|
||||||
|
may publish into the new source revision after cancellation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
array: np.ndarray = np.zeros(4, dtype=[
|
||||||
|
('time', float), ('trend', float),
|
||||||
|
])
|
||||||
|
dst: SimpleNamespace = SimpleNamespace(
|
||||||
|
_first=SimpleNamespace(value=0),
|
||||||
|
_last=SimpleNamespace(value=0),
|
||||||
|
_array=array,
|
||||||
|
)
|
||||||
|
casc: Cascade = cast(Cascade, SimpleNamespace(
|
||||||
|
dst=SimpleNamespace(rt_shm=dst),
|
||||||
|
sample_ready=None,
|
||||||
|
))
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(
|
||||||
|
_write_sample, casc, FspSample(0, 1., {'trend': 1.}),
|
||||||
|
)
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
dst._last.value = 1
|
||||||
|
assert casc.sample_ready is not None
|
||||||
|
casc.sample_ready.set()
|
||||||
|
assert not array['trend'].any()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_builtin_registration() -> None:
|
||||||
|
'''
|
||||||
|
Actor-side builtin discovery must resolve the new FSP target.
|
||||||
|
|
||||||
|
'''
|
||||||
|
assert _load_builtins()[trend_calculus.ns_path] is trend_calculus
|
||||||
|
|
||||||
|
|
||||||
|
def test_quote_free_sampler_steps_finalize_without_repeating_pivots(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A quiet feed still finalizes bars and clears old reversal pulses.
|
||||||
|
|
||||||
|
Start with an open bar containing two provisional reversals and
|
||||||
|
advance the source on sampler notifications alone. Mirror the
|
||||||
|
destination append protocol and run the actual FSP generator and
|
||||||
|
indexed writer. Every stored field must equal causal replay;
|
||||||
|
event fields must also be clear before compute catches up.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: MemorySource = MemorySource([1., 2., 4., 3., 2., 5.])
|
||||||
|
first: int = source._first.value
|
||||||
|
last: int = source._last.value
|
||||||
|
name: str
|
||||||
|
dtype: list[tuple[str, str]] = [
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
] + [
|
||||||
|
(name, '<f8') for name in trend_calculus.outputs
|
||||||
|
]
|
||||||
|
array: np.ndarray = np.zeros(128, dtype=dtype)
|
||||||
|
array['index'] = np.arange(128)
|
||||||
|
destination: SimpleNamespace = SimpleNamespace(
|
||||||
|
_first=SimpleNamespace(value=first),
|
||||||
|
_last=SimpleNamespace(value=last),
|
||||||
|
_array=array,
|
||||||
|
)
|
||||||
|
casc: Cascade = cast(Cascade, SimpleNamespace(
|
||||||
|
dst=SimpleNamespace(rt_shm=destination),
|
||||||
|
sample_ready=None,
|
||||||
|
))
|
||||||
|
send: trio.MemorySendChannel[QuoteFrame]
|
||||||
|
quotes: trio.MemoryReceiveChannel[QuoteFrame]
|
||||||
|
send, quotes = trio.open_memory_channel[QuoteFrame](0)
|
||||||
|
wakeups: AsyncIterator[QuoteFrame]
|
||||||
|
notify: Callable[[], None]|None
|
||||||
|
async with send, quotes, _sample_wakeups(
|
||||||
|
quotes, 'test.market', True,
|
||||||
|
) as (wakeups, notify):
|
||||||
|
stream: FspStream = trend_calculus(
|
||||||
|
filter_quotes_by_sym('test.market', wakeups),
|
||||||
|
cast(ShmArray, source),
|
||||||
|
)
|
||||||
|
history: FspYield = await anext(stream)
|
||||||
|
assert isinstance(history, dict)
|
||||||
|
name: str
|
||||||
|
values: np.ndarray
|
||||||
|
for name, values in history.items():
|
||||||
|
assert values is not None
|
||||||
|
array[name][first:last] = values
|
||||||
|
array['time'][first:last] = source.array['time']
|
||||||
|
|
||||||
|
async def compute() -> None:
|
||||||
|
'''
|
||||||
|
Run until cancellation and record completion on exit.
|
||||||
|
|
||||||
|
'''
|
||||||
|
try:
|
||||||
|
sample: FspYield
|
||||||
|
async for sample in stream:
|
||||||
|
assert isinstance(sample, FspSample)
|
||||||
|
await _write_sample(casc, sample)
|
||||||
|
finally:
|
||||||
|
await stream.aclose()
|
||||||
|
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
nursery.start_soon(compute)
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert array['top_1'][last - 1] == 4.
|
||||||
|
assert array['bottom_1'][last - 1] == 2.
|
||||||
|
close: float
|
||||||
|
for close in (5., 5., -4., -5.):
|
||||||
|
last = source._last.value
|
||||||
|
source._array['close'][last] = close
|
||||||
|
source._last.value += 1
|
||||||
|
array[last:last + 1] = _next_step_output(
|
||||||
|
array[last - 1:last], trend_calculus, False,
|
||||||
|
)
|
||||||
|
array['index'][last] = last
|
||||||
|
array['time'][last] = source.array['time'][-1]
|
||||||
|
destination._last.value += 1
|
||||||
|
assert np.isnan(array['top_1'][last])
|
||||||
|
assert np.isnan(array['bottom_1'][last])
|
||||||
|
if casc.sample_ready is not None:
|
||||||
|
casc.sample_ready.set()
|
||||||
|
casc.sample_ready = trio.Event()
|
||||||
|
assert notify is not None
|
||||||
|
# Coalesce a burst without losing finalized rows.
|
||||||
|
notify()
|
||||||
|
notify()
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
offset: int
|
||||||
|
row: np.void
|
||||||
|
for offset, row in enumerate(source.array):
|
||||||
|
name: str
|
||||||
|
sample: FspSample = FspSample(
|
||||||
|
int(row['index']), float(row['time']),
|
||||||
|
{
|
||||||
|
name: array[name][int(row['index'])]
|
||||||
|
for name in trend_calculus.outputs
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert_sample_matches(
|
||||||
|
sample, source.array[:offset + 1],
|
||||||
|
)
|
||||||
|
nursery.cancel_scope.cancel()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_quote_overrun_resumes_source_wakeups() -> None:
|
||||||
|
'''
|
||||||
|
Losing irrelevant quote payloads must not kill a SHM processor.
|
||||||
|
|
||||||
|
A broadcast receiver raises Lagged and then remains usable.
|
||||||
|
Simulate that protocol, require a catchup notification, and
|
||||||
|
deliver another quote afterward to prove the relay resumed.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
send: trio.MemorySendChannel[QuoteFrame|Exception]
|
||||||
|
receive: trio.MemoryReceiveChannel[QuoteFrame|Exception]
|
||||||
|
send, receive = trio.open_memory_channel[
|
||||||
|
QuoteFrame|Exception
|
||||||
|
](1)
|
||||||
|
|
||||||
|
class Quotes:
|
||||||
|
'''
|
||||||
|
Inject one overrun without closing the quote receiver.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __aiter__(self) -> 'Quotes':
|
||||||
|
'''
|
||||||
|
Retain this controlled quote receiver
|
||||||
|
across overruns.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def __anext__(self) -> QuoteFrame:
|
||||||
|
'''
|
||||||
|
Deliver the scheduled frame or
|
||||||
|
injected receiver error.
|
||||||
|
|
||||||
|
'''
|
||||||
|
value: QuoteFrame|Exception = await receive.receive()
|
||||||
|
if isinstance(value, Exception):
|
||||||
|
raise value
|
||||||
|
return value
|
||||||
|
|
||||||
|
wakeups: AsyncIterator[QuoteFrame]
|
||||||
|
notify: Callable[[], None]|None
|
||||||
|
async with send, receive, _sample_wakeups(
|
||||||
|
Quotes(), 'test.market', True,
|
||||||
|
) as (wakeups, notify):
|
||||||
|
await send.send(Lagged('synthetic quote overrun'))
|
||||||
|
assert 'test.market' in await anext(wakeups)
|
||||||
|
await send.send({'test.market': {'ticks': []}})
|
||||||
|
assert 'test.market' in await anext(wakeups)
|
||||||
|
assert notify is not None
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
||||||
|
|
||||||
|
def test_edge_exit_stops_compute_before_source_receiver_closes(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Normal sampler EOF must not strand or abruptly close compute.
|
||||||
|
|
||||||
|
Keep the parent nursery alive beyond open_edge and inspect the
|
||||||
|
compute task's finalizer immediately after context exit. This
|
||||||
|
proves cleanup belongs to the edge scope rather than relying on
|
||||||
|
an outer actor cancellation or a closed receiver exception.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Run the controlled scenario in an isolated Trio lifetime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
cleaned: trio.Event = trio.Event()
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
casc: Cascade = cast(Cascade, SimpleNamespace(
|
||||||
|
tn=nursery,
|
||||||
|
cs=None,
|
||||||
|
complete=None,
|
||||||
|
bind_func=None,
|
||||||
|
))
|
||||||
|
|
||||||
|
async def compute(
|
||||||
|
*,
|
||||||
|
task_status: TaskStatus[int] = (
|
||||||
|
trio.TASK_STATUS_IGNORED
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Run until cancellation and record completion on exit.
|
||||||
|
|
||||||
|
'''
|
||||||
|
scope: trio.CancelScope
|
||||||
|
with trio.CancelScope() as scope:
|
||||||
|
casc.cs = scope
|
||||||
|
casc.complete = trio.Event()
|
||||||
|
task_status.started(42)
|
||||||
|
try:
|
||||||
|
await trio.sleep_forever()
|
||||||
|
finally:
|
||||||
|
cleaned.set()
|
||||||
|
casc.complete.set()
|
||||||
|
|
||||||
|
index: int
|
||||||
|
async with Cascade.open_edge(casc, compute) as index:
|
||||||
|
assert index == 42
|
||||||
|
assert not cleaned.is_set()
|
||||||
|
assert cleaned.is_set()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
|
|
@ -0,0 +1,929 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Trend overlay causality, coordinate and bounded Qt lifetime checks.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator, Iterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
import os
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pyqtgraph as pg
|
||||||
|
import pytest
|
||||||
|
import trio
|
||||||
|
from trio.testing import wait_all_tasks_blocked
|
||||||
|
|
||||||
|
from piker.ui._trend import (
|
||||||
|
RegimeInterval,
|
||||||
|
ReversalPoint,
|
||||||
|
TrendOverlay,
|
||||||
|
classification_interval,
|
||||||
|
regime_rectangles,
|
||||||
|
reversal_points,
|
||||||
|
times_to_x,
|
||||||
|
trend_intervals,
|
||||||
|
)
|
||||||
|
from piker.ui.qt import QApplication, QtGui, QPointF
|
||||||
|
from piker.fsp._jev import JevClassification, RegimeAnswer
|
||||||
|
from piker.fsp import Fsp
|
||||||
|
from piker.ui._fsp import FspAdmin, update_trend_overlays
|
||||||
|
from piker.ui import _fsp as fsp_ui
|
||||||
|
|
||||||
|
|
||||||
|
def _chart(
|
||||||
|
times: tuple[float, ...],
|
||||||
|
first: int = 1000,
|
||||||
|
|
||||||
|
) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Make chart bars with absolute indices independent of timestamp.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rows: np.ndarray = np.zeros(len(times), dtype=[
|
||||||
|
('index', 'i8'),
|
||||||
|
('time', 'f8'),
|
||||||
|
('low', 'f8'),
|
||||||
|
('high', 'f8'),
|
||||||
|
('close', 'f8'),
|
||||||
|
])
|
||||||
|
rows['index'] = np.arange(first, first + len(times))
|
||||||
|
rows['time'] = times
|
||||||
|
rows['low'] = 99
|
||||||
|
rows['high'] = 101
|
||||||
|
rows['close'] = 100
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _source(
|
||||||
|
times: tuple[float, ...],
|
||||||
|
trends: tuple[float, ...],
|
||||||
|
|
||||||
|
) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Make sparse FSP fields, including a deliberately mutable tail.
|
||||||
|
|
||||||
|
'''
|
||||||
|
fields: list[tuple[str, str]] = [
|
||||||
|
('index', 'i8'),
|
||||||
|
('time', 'f8'),
|
||||||
|
('ready', 'f8'),
|
||||||
|
]
|
||||||
|
order: int
|
||||||
|
for order in range(1, 5):
|
||||||
|
field: str = 'trend' if order == 1 else f'trend_{order}'
|
||||||
|
fields.append((field, 'f8'))
|
||||||
|
kind: str
|
||||||
|
for kind in ('top', 'bottom'):
|
||||||
|
fields.extend([
|
||||||
|
(f'{kind}_{order}', 'f8'),
|
||||||
|
(f'{kind}_time_{order}', 'f8'),
|
||||||
|
])
|
||||||
|
|
||||||
|
rows: np.ndarray = np.empty(len(times), dtype=fields)
|
||||||
|
name: str
|
||||||
|
for name in rows.dtype.names:
|
||||||
|
if name == 'index':
|
||||||
|
rows[name] = np.arange(5000, 5000 + len(rows))
|
||||||
|
else:
|
||||||
|
rows[name] = np.nan
|
||||||
|
rows['time'] = times
|
||||||
|
rows['ready'] = 4
|
||||||
|
rows['trend'] = trends
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope='session')
|
||||||
|
def qapp() -> Iterator[QApplication]:
|
||||||
|
'''
|
||||||
|
Keep one offscreen application alive without launching a GUI.
|
||||||
|
|
||||||
|
'''
|
||||||
|
app: QApplication|None = QApplication.instance()
|
||||||
|
if app is None:
|
||||||
|
app = QApplication(['piker-trend-tests'])
|
||||||
|
app.setQuitOnLastWindowClosed(False)
|
||||||
|
yield app
|
||||||
|
app.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('index_field', ['index', 'time'])
|
||||||
|
def test_timestamp_mapping_preserves_gaps(
|
||||||
|
index_field: Literal['index', 'time'],
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A timestamp inside a missing bucket must not snap to a future
|
||||||
|
bar or confuse slice offsets with absolute SHM coordinates.
|
||||||
|
Irregular 10-second bars and off-grid queries exercise both
|
||||||
|
chart axes; NaNs prove out-of-range observations stay hidden.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rows: np.ndarray = _chart((100, 110, 140))
|
||||||
|
queries: np.ndarray = np.array((99, 100, 115, 125, 145, 150))
|
||||||
|
actual: np.ndarray = times_to_x(rows, queries, 10, index_field)
|
||||||
|
expected: list[float] = (
|
||||||
|
[np.nan, 1000, 1001.5, np.nan, 1002.5, np.nan]
|
||||||
|
if index_field == 'index'
|
||||||
|
else [np.nan, 100, 115, np.nan, 145, np.nan]
|
||||||
|
)
|
||||||
|
np.testing.assert_allclose(actual, expected, equal_nan=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pivots_confirm_at_later_bar_close() -> None:
|
||||||
|
'''
|
||||||
|
Plotting a provisional reversal makes an intrabar repaint look
|
||||||
|
like a confirmed signal. A confirmed top and mutable-tail bottom
|
||||||
|
share an earlier pivot time here. Only the top may render, with
|
||||||
|
its 30-second source detection time and 10-second chart position
|
||||||
|
retained separately; ``source['index']`` is unrelated on purpose.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source((100, 130, 160, 190), (0, 1, -1, 1))
|
||||||
|
source['top_1'][2] = 110
|
||||||
|
source['top_time_1'][2] = 130
|
||||||
|
source['bottom_1'][3] = 90
|
||||||
|
source['bottom_time_1'][3] = 160
|
||||||
|
chart: np.ndarray = _chart(tuple(range(100, 201, 10)))
|
||||||
|
points: list[ReversalPoint] = reversal_points(
|
||||||
|
source,
|
||||||
|
chart,
|
||||||
|
source_period_s=30,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
assert len(points) == 1
|
||||||
|
assert points[0].x == 1003
|
||||||
|
assert points[0].pivot_time == 130
|
||||||
|
assert points[0].detected_at == 190
|
||||||
|
assert points[0].period_s == 30
|
||||||
|
assert points[0].kind == 'top'
|
||||||
|
|
||||||
|
|
||||||
|
def test_reversal_order_is_independent_of_timeframe() -> None:
|
||||||
|
'''
|
||||||
|
A second reversal order must use its own pivot fields without
|
||||||
|
implicitly changing the source sampling period. Distinct order
|
||||||
|
one and order two prices at the same source time identify which
|
||||||
|
columns were consumed and prevent order/timeframe conflation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source((10, 20, 30), (1, -1, 1))
|
||||||
|
source['top_1'][1] = 105
|
||||||
|
source['top_time_1'][1] = 10
|
||||||
|
source['bottom_2'][1] = 95
|
||||||
|
source['bottom_time_2'][1] = 10
|
||||||
|
points: list[ReversalPoint] = reversal_points(
|
||||||
|
source,
|
||||||
|
_chart((10, 20, 30)),
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
order=2,
|
||||||
|
)
|
||||||
|
assert len(points) == 1
|
||||||
|
assert points[0].price == 95
|
||||||
|
assert points[0].order == 2
|
||||||
|
assert points[0].period_s == 10
|
||||||
|
|
||||||
|
|
||||||
|
def test_regime_runs_start_at_confirmation() -> None:
|
||||||
|
'''
|
||||||
|
Coloring from a pivot or detection-bar open falsely implies an
|
||||||
|
earlier available regime. A warm-up row, direction change, and
|
||||||
|
source gap arrange all run boundaries. Every interval starts at
|
||||||
|
finalized bar close and the mutable tail's opposite direction
|
||||||
|
cannot create a new regime.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source(
|
||||||
|
(100, 110, 120, 130, 170, 180),
|
||||||
|
(1, 1, 1, -1, -1, 1),
|
||||||
|
)
|
||||||
|
source['ready'][0] = 0
|
||||||
|
spans: list[RegimeInterval] = trend_intervals(source, 10)
|
||||||
|
span: RegimeInterval # noqa: F842
|
||||||
|
assert [(span.start_time, span.end_time) for span in spans] == [
|
||||||
|
(120, 140),
|
||||||
|
(140, 150),
|
||||||
|
(180, 190),
|
||||||
|
]
|
||||||
|
assert [span.label for span in spans] == [
|
||||||
|
'uptrend', 'downtrend', 'downtrend',
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_decimal_subsecond_epochs_do_not_invent_gaps() -> None:
|
||||||
|
'''
|
||||||
|
Float64 Unix timestamps cannot represent every 100 ms boundary
|
||||||
|
exactly; successive differences alternate around 0.1 seconds.
|
||||||
|
Comparing each difference strictly to the requested period splits
|
||||||
|
one regime into spurious holes. Realistic epochs prove rounding
|
||||||
|
preserves one run, while deleting one actual bucket still splits
|
||||||
|
both regime evidence and chart rectangle geometry.
|
||||||
|
|
||||||
|
'''
|
||||||
|
times: np.ndarray = 1_700_000_000 + np.arange(8) * 0.1
|
||||||
|
source: np.ndarray = _source(tuple(times), (1,) * 8)
|
||||||
|
spans: list[RegimeInterval] = trend_intervals(source, 0.1)
|
||||||
|
assert len(spans) == 1
|
||||||
|
rectangles: np.ndarray
|
||||||
|
_: list[RegimeInterval]
|
||||||
|
rectangles, _ = regime_rectangles(
|
||||||
|
spans, _chart(tuple(times)), 0.1,
|
||||||
|
)
|
||||||
|
assert len(rectangles) == 1
|
||||||
|
|
||||||
|
source['time'][4:] += 0.1
|
||||||
|
spans = trend_intervals(source, 0.1)
|
||||||
|
assert len(spans) == 2
|
||||||
|
whole: RegimeInterval = RegimeInterval(
|
||||||
|
float(times[0]), float(times[-1] + 0.3), 'uptrend',
|
||||||
|
)
|
||||||
|
rectangles, _ = regime_rectangles(
|
||||||
|
[whole], _chart(tuple(source['time'])), 0.1,
|
||||||
|
)
|
||||||
|
assert len(rectangles) == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('index_field', ['index', 'time'])
|
||||||
|
def test_delayed_regime_and_chart_gaps(
|
||||||
|
index_field: Literal['index', 'time'],
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A classifier may report an earlier onset after the chart has
|
||||||
|
advanced. Its rectangle must begin only at response arrival and
|
||||||
|
must not fill an absent chart bucket. Fractional arrival within
|
||||||
|
a bar, plus a 20-second hole, proves causal clipping and split
|
||||||
|
geometry on both time and absolute-index chart axes.
|
||||||
|
|
||||||
|
'''
|
||||||
|
chart: np.ndarray = _chart((100, 110, 140, 150))
|
||||||
|
span: RegimeInterval = RegimeInterval(
|
||||||
|
start_time=100,
|
||||||
|
end_time=155,
|
||||||
|
label='ranging',
|
||||||
|
probability=0.8,
|
||||||
|
source='jev',
|
||||||
|
observed_at=115,
|
||||||
|
)
|
||||||
|
rectangles: np.ndarray
|
||||||
|
metadata: list[RegimeInterval]
|
||||||
|
rectangles, metadata = regime_rectangles(
|
||||||
|
[span], chart, 10, index_field=index_field,
|
||||||
|
)
|
||||||
|
expected: list[list[float]] = (
|
||||||
|
[[1001.5, 99, 0.5, 2], [1002, 99, 1.5, 2]]
|
||||||
|
if index_field == 'index'
|
||||||
|
else [[115, 99, 5, 2], [140, 99, 15, 2]]
|
||||||
|
)
|
||||||
|
np.testing.assert_allclose(rectangles, expected)
|
||||||
|
assert metadata == [span, span]
|
||||||
|
assert metadata[0].probability == 0.8
|
||||||
|
assert metadata[0].source == 'jev'
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_and_unavailable_geometry() -> None:
|
||||||
|
'''
|
||||||
|
A layer can attach before history arrives, and late classifier
|
||||||
|
responses can lie beyond retained history. Empty chart arrays
|
||||||
|
and already-expired intervals must produce empty geometry rather
|
||||||
|
than indexing the final row or drawing a retrospective rectangle.
|
||||||
|
|
||||||
|
'''
|
||||||
|
span: RegimeInterval = RegimeInterval(10, 20, 'ranging')
|
||||||
|
empty: np.ndarray
|
||||||
|
metadata: list[RegimeInterval]
|
||||||
|
empty, metadata = regime_rectangles([span], _chart(()), 1)
|
||||||
|
assert empty.shape == (0, 4)
|
||||||
|
assert metadata == []
|
||||||
|
span = RegimeInterval(10, 20, 'ranging', observed_at=21)
|
||||||
|
empty, metadata = regime_rectangles([span], _chart((10, 20)), 1)
|
||||||
|
assert empty.shape == (0, 4)
|
||||||
|
assert metadata == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('period', [0, -1, np.nan, np.inf])
|
||||||
|
def test_invalid_period_rejected(period: float) -> None:
|
||||||
|
'''
|
||||||
|
Invalid sampling periods break binary-search interval semantics
|
||||||
|
and can create infinite widths. All non-positive/non-finite
|
||||||
|
period classes are rejected before constructing chart geometry.
|
||||||
|
|
||||||
|
'''
|
||||||
|
with pytest.raises(ValueError, match='period'):
|
||||||
|
times_to_x(_chart((10, 20)), np.array([10]), period)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
'market,revision,period,end_time,accepted',
|
||||||
|
[
|
||||||
|
('test.market', 3, 5, 130, True),
|
||||||
|
('other.market', 3, 5, 130, False),
|
||||||
|
('test.market', 4, 5, 130, False),
|
||||||
|
('test.market', 3, 15, 130, False),
|
||||||
|
('test.market', 3, 5, 119, False),
|
||||||
|
],
|
||||||
|
ids=['matching', 'market', 'revision', 'period', 'future'],
|
||||||
|
)
|
||||||
|
def test_jev_adapter_retains_causal_evidence(
|
||||||
|
market: str,
|
||||||
|
revision: int,
|
||||||
|
period: float,
|
||||||
|
end_time: float,
|
||||||
|
accepted: bool,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Late or misrouted classifications must not color an unrelated
|
||||||
|
market, source revision, or timeframe. A typed Jev response
|
||||||
|
claims source evidence at 100 but arrives at 120; matching
|
||||||
|
evidence retains its original probability/model and starts at
|
||||||
|
receipt time. The rejected cases vary one ownership bound each.
|
||||||
|
|
||||||
|
'''
|
||||||
|
result: JevClassification = JevClassification(
|
||||||
|
market='test.market',
|
||||||
|
revision=3,
|
||||||
|
sequence=40,
|
||||||
|
segment=2,
|
||||||
|
source_as_of=100,
|
||||||
|
requested_at=118,
|
||||||
|
received_at=120,
|
||||||
|
latency=2,
|
||||||
|
model='jev-test',
|
||||||
|
prompt_version='test',
|
||||||
|
answers=(RegimeAnswer(
|
||||||
|
question='regime_5',
|
||||||
|
period=5,
|
||||||
|
label='range',
|
||||||
|
probability=0.82,
|
||||||
|
confidence=0.7,
|
||||||
|
probabilities=(('range', 0.82), ('unknown', 0.18)),
|
||||||
|
),),
|
||||||
|
input_tokens=100,
|
||||||
|
output_tokens=20,
|
||||||
|
)
|
||||||
|
interval: RegimeInterval|None = classification_interval(
|
||||||
|
result,
|
||||||
|
fqme=market,
|
||||||
|
revision=revision,
|
||||||
|
period_s=period,
|
||||||
|
end_time=end_time,
|
||||||
|
)
|
||||||
|
if not accepted:
|
||||||
|
assert interval is None
|
||||||
|
return
|
||||||
|
assert interval is not None
|
||||||
|
assert interval.start_time == 100
|
||||||
|
assert interval.observed_at == 120
|
||||||
|
assert interval.label == 'range'
|
||||||
|
assert interval.probability == 0.82
|
||||||
|
assert interval.source == 'typesafe.ai/jev-test'
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_rebuild_and_owned_teardown(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Equal-bound backfills rewrite earlier pivot fields without
|
||||||
|
changing the live tail. An ordinary paint gate intentionally
|
||||||
|
skips that change; a forced revision must replace stale markers
|
||||||
|
and rectangles, preserve a sibling layer, and release exactly
|
||||||
|
the owning layer's two items on repeated cancellation cleanup.
|
||||||
|
Real Qt items prove scene ownership rather than a mocked API.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source(
|
||||||
|
(100, 110, 120, 130, 140), (1, 1, -1, -1, 1),
|
||||||
|
)
|
||||||
|
source['top_1'][1] = 105
|
||||||
|
source['top_time_1'][1] = 100
|
||||||
|
widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=widget.plotItem,
|
||||||
|
shm=SimpleNamespace(array=_chart((100, 110, 120, 130, 140))),
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
first: TrendOverlay = TrendOverlay(
|
||||||
|
viz,
|
||||||
|
SimpleNamespace(array=source),
|
||||||
|
fqme='first.test',
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
sibling: TrendOverlay = TrendOverlay(
|
||||||
|
viz,
|
||||||
|
SimpleNamespace(array=source),
|
||||||
|
fqme='second.test',
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
assert first.update()
|
||||||
|
assert sibling.update()
|
||||||
|
assert len(first.points) == 1
|
||||||
|
assert not first.update()
|
||||||
|
source['top_1'][1] = np.nan
|
||||||
|
source['bottom_1'][1] = 95
|
||||||
|
source['bottom_time_1'][1] = 100
|
||||||
|
assert not first.update()
|
||||||
|
assert first.update(force=True)
|
||||||
|
assert first.points[0].kind == 'bottom'
|
||||||
|
assert sibling.points[0].kind == 'top'
|
||||||
|
assert len(widget.plotItem.items) == 4
|
||||||
|
|
||||||
|
# A synthetic full-scene paint verifies Qt primitive types
|
||||||
|
# and brush batching without a GUI or screenshot dependency.
|
||||||
|
image: QtGui.QImage = QtGui.QImage(
|
||||||
|
160, 100, QtGui.QImage.Format.Format_ARGB32,
|
||||||
|
)
|
||||||
|
image.fill(0)
|
||||||
|
painter: QtGui.QPainter = QtGui.QPainter(image)
|
||||||
|
try:
|
||||||
|
widget.scene().render(painter)
|
||||||
|
finally:
|
||||||
|
painter.end()
|
||||||
|
|
||||||
|
first.remove()
|
||||||
|
first.remove()
|
||||||
|
assert len(widget.plotItem.items) == 2
|
||||||
|
assert sibling.markers in widget.plotItem.items
|
||||||
|
assert not first.update(force=True)
|
||||||
|
finally:
|
||||||
|
first.remove()
|
||||||
|
sibling.remove()
|
||||||
|
widget.close()
|
||||||
|
widget.deleteLater()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_provisional_updates_reuse_confirmed_markers(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Quotes may repaint a provisional reversal repeatedly during one
|
||||||
|
bar. Rebuilding historical markers each time wastes UI time and
|
||||||
|
can present a temporary reversal as confirmed. Mutate only the
|
||||||
|
FSP tail, then the chart candle envelope: the first update is
|
||||||
|
skipped and the second updates rectangles while preserving the
|
||||||
|
same ``TrendOverlay.points`` list. Changing ``chart['index']``
|
||||||
|
requires remapping those retained points.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source((10, 20, 30), (1, 1, -1))
|
||||||
|
source['top_1'][1] = 105
|
||||||
|
source['top_time_1'][1] = 10
|
||||||
|
widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
chart: np.ndarray = _chart((10, 20, 30))
|
||||||
|
viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=widget.plotItem,
|
||||||
|
shm=SimpleNamespace(array=chart),
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
overlay: TrendOverlay = TrendOverlay(
|
||||||
|
viz,
|
||||||
|
SimpleNamespace(array=source),
|
||||||
|
fqme='live.test',
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
assert overlay.update()
|
||||||
|
original: list[ReversalPoint] = overlay.points
|
||||||
|
source['trend'][-1] = 1
|
||||||
|
source['bottom_1'][-1] = 91
|
||||||
|
source['bottom_time_1'][-1] = 20
|
||||||
|
assert not overlay.update()
|
||||||
|
assert overlay.points is original
|
||||||
|
|
||||||
|
chart['high'][-1] = 109
|
||||||
|
assert overlay.update()
|
||||||
|
assert overlay.points is original
|
||||||
|
assert overlay.rectangles.rectangles[-1, 3] == 10
|
||||||
|
|
||||||
|
chart['high'][-1] = 100
|
||||||
|
assert overlay.update()
|
||||||
|
assert overlay.points is original
|
||||||
|
assert overlay.rectangles.rectangles[-1, 3] == 2
|
||||||
|
|
||||||
|
chart['index'] += 500
|
||||||
|
assert overlay.update()
|
||||||
|
assert overlay.points is not original
|
||||||
|
assert overlay.points[0].x == 1500
|
||||||
|
finally:
|
||||||
|
overlay.remove()
|
||||||
|
widget.close()
|
||||||
|
widget.deleteLater()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_limits_and_classifier_metadata(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Unbounded event layers can allocate thousands of Qt objects and
|
||||||
|
delayed classifier results can overwrite deterministic meaning.
|
||||||
|
Many synthetic pivots exercise the configured retention limits;
|
||||||
|
replacing rectangles with a Jev result must retain probability,
|
||||||
|
source, and markers while keeping exactly two scene items.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: np.ndarray = _source(tuple(range(20)), (1, -1) * 10)
|
||||||
|
source['top_1'] = 105
|
||||||
|
source['top_time_1'] = source['time']
|
||||||
|
widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=widget.plotItem,
|
||||||
|
shm=SimpleNamespace(array=_chart(tuple(range(20)))),
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
overlay: TrendOverlay = TrendOverlay(
|
||||||
|
viz,
|
||||||
|
SimpleNamespace(array=source),
|
||||||
|
fqme='bounded.test',
|
||||||
|
source_period_s=1,
|
||||||
|
chart_period_s=1,
|
||||||
|
max_points=3,
|
||||||
|
max_rectangles=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
overlay.update()
|
||||||
|
assert len(overlay.points) == 3
|
||||||
|
assert overlay.points[0].pivot_time == 16
|
||||||
|
assert len(overlay.rectangles.rectangles) == 2
|
||||||
|
assert len(widget.plotItem.items) == 2
|
||||||
|
|
||||||
|
classified: RegimeInterval = RegimeInterval(
|
||||||
|
start_time=15,
|
||||||
|
end_time=20,
|
||||||
|
label='ranging',
|
||||||
|
probability=0.73,
|
||||||
|
source='jev',
|
||||||
|
observed_at=18,
|
||||||
|
)
|
||||||
|
overlay.set_regimes([classified])
|
||||||
|
assert len(overlay.points) == 3
|
||||||
|
assert overlay.rectangles.intervals == [classified]
|
||||||
|
assert overlay.rectangles.rectangles[0, 0] == 1018
|
||||||
|
tip: str = overlay.rectangles.tooltip_at(QPointF(1019, 100))
|
||||||
|
assert 'ranging: 73.0%' in tip
|
||||||
|
assert '(1s)' in tip
|
||||||
|
assert 'jev' in tip
|
||||||
|
assert '00:00:18.000' in tip
|
||||||
|
overlay.set_regimes([])
|
||||||
|
assert len(overlay.rectangles.rectangles) == 0
|
||||||
|
overlay.set_regimes(None)
|
||||||
|
assert len(overlay.rectangles.rectangles) == 2
|
||||||
|
tip = overlay.rectangles.tooltip_at(QPointF(1019, 100))
|
||||||
|
assert 'trend-calculus' in tip
|
||||||
|
assert '%' not in tip
|
||||||
|
finally:
|
||||||
|
overlay.remove()
|
||||||
|
widget.close()
|
||||||
|
widget.deleteLater()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
class _PriceChart:
|
||||||
|
'''
|
||||||
|
Expose the price-chart API while retaining actual Qt ownership.
|
||||||
|
|
||||||
|
'''
|
||||||
|
_trend_overlays: dict[tuple[str, str], list[TrendOverlay]]
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
'''
|
||||||
|
Build a chart whose SHM token identifies its base timeframe.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
self.viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=self.widget.plotItem,
|
||||||
|
shm=SimpleNamespace(
|
||||||
|
array=_chart((10, 20, 30, 40, 50)),
|
||||||
|
_token='ohlcv',
|
||||||
|
),
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_viz(self, fqme: str) -> SimpleNamespace|None:
|
||||||
|
'''
|
||||||
|
Resolve exactly the test market rather than any chart key.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self.viz if fqme == 'admin.test' else None
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
'''
|
||||||
|
Release the disposable offscreen widget after its layers.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.widget.close()
|
||||||
|
self.widget.deleteLater()
|
||||||
|
|
||||||
|
|
||||||
|
async def _admin_bootstrap_scenario(
|
||||||
|
qapp: QApplication,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Attaching before FSP bootstrap exposes empty history, and opening
|
||||||
|
several reversal orders must not start multiple SHM writers.
|
||||||
|
The ``attach_first()`` task calls ``FspAdmin.open_trend_overlay``
|
||||||
|
while the mocked engine's ``started`` event stays unset.
|
||||||
|
``wait_all_tasks_blocked()`` reaches that wait deterministically;
|
||||||
|
one engine call and an empty Qt plot rule out early attachment.
|
||||||
|
Only setting ``started`` lets ``attached`` confirm publication.
|
||||||
|
Reopening another order retains one writer, and repeated
|
||||||
|
``FspAdmin.close_trend_overlays()`` preserves a sibling's items.
|
||||||
|
This exercises local lifecycle ordering, not actor IPC.
|
||||||
|
|
||||||
|
'''
|
||||||
|
chart: _PriceChart = _PriceChart()
|
||||||
|
source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5)
|
||||||
|
source['trend_2'] = 1
|
||||||
|
destination: SimpleNamespace = SimpleNamespace(
|
||||||
|
rt_shm=SimpleNamespace(array=source, _token='trend'),
|
||||||
|
)
|
||||||
|
flume: SimpleNamespace = SimpleNamespace(
|
||||||
|
mkt=SimpleNamespace(fqme='admin.test'),
|
||||||
|
rt_shm=chart.viz.shm,
|
||||||
|
rt_sample_period_s=10,
|
||||||
|
_hist_shm_token=None,
|
||||||
|
)
|
||||||
|
started: trio.Event = trio.Event()
|
||||||
|
attached: trio.Event = trio.Event()
|
||||||
|
calls: list[tuple[str, dict, str]] = []
|
||||||
|
attached_layers: list[TrendOverlay] = []
|
||||||
|
|
||||||
|
async def start_engine(
|
||||||
|
target: Fsp,
|
||||||
|
conf: dict,
|
||||||
|
loglevel: str,
|
||||||
|
|
||||||
|
) -> tuple[SimpleNamespace, trio.Event]:
|
||||||
|
'''
|
||||||
|
Return an allocated destination before publishing bootstrap.
|
||||||
|
|
||||||
|
'''
|
||||||
|
calls.append((target.name, conf, loglevel))
|
||||||
|
return destination, started
|
||||||
|
|
||||||
|
try:
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
admin: FspAdmin = FspAdmin(
|
||||||
|
nursery, {}, SimpleNamespace(chart=chart), flume,
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
admin, 'start_engine_task', start_engine,
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match='positive'):
|
||||||
|
await admin.open_trend_overlay(chart_period_s=0)
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
async def attach_first() -> None:
|
||||||
|
'''
|
||||||
|
Signal only after startup has produced visible data.
|
||||||
|
|
||||||
|
'''
|
||||||
|
layer: TrendOverlay = (
|
||||||
|
await admin.open_trend_overlay()
|
||||||
|
)
|
||||||
|
attached_layers.append(layer)
|
||||||
|
attached.set()
|
||||||
|
|
||||||
|
nursery.start_soon(attach_first)
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert len(calls) == 1
|
||||||
|
assert not attached.is_set()
|
||||||
|
assert chart.widget.plotItem.items == []
|
||||||
|
|
||||||
|
started.set()
|
||||||
|
await attached.wait()
|
||||||
|
first: TrendOverlay = attached_layers[0]
|
||||||
|
second: TrendOverlay = await admin.open_trend_overlay(
|
||||||
|
order=2,
|
||||||
|
)
|
||||||
|
assert await admin.open_trend_overlay() is first
|
||||||
|
with pytest.raises(ValueError, match='period differs'):
|
||||||
|
await admin.open_trend_overlay(chart_period_s=60)
|
||||||
|
assert second is not first
|
||||||
|
assert len(calls) == 1
|
||||||
|
assert calls[0][0] == 'trend_calculus'
|
||||||
|
assert chart._trend_overlays == {
|
||||||
|
('admin.test', 'trend'): [first, second],
|
||||||
|
}
|
||||||
|
assert len(chart.widget.plotItem.items) == 4
|
||||||
|
|
||||||
|
sibling: TrendOverlay = TrendOverlay(
|
||||||
|
chart.viz,
|
||||||
|
SimpleNamespace(array=source, _token='other'),
|
||||||
|
fqme='other.test',
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
foreign_key: tuple[str, str] = ('other.test', 'other')
|
||||||
|
chart._trend_overlays[foreign_key] = [sibling]
|
||||||
|
try:
|
||||||
|
admin.close_trend_overlays()
|
||||||
|
admin.close_trend_overlays()
|
||||||
|
assert len(chart.widget.plotItem.items) == 2
|
||||||
|
assert chart._trend_overlays == {
|
||||||
|
('other.test', 'other'): [sibling],
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
sibling.remove()
|
||||||
|
finally:
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_waits_for_bootstrap_and_reuses_writer(
|
||||||
|
qapp: QApplication,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Exercise bootstrap publication and owner cleanup in local Trio.
|
||||||
|
|
||||||
|
``_admin_bootstrap_scenario()`` controls its ``started`` event
|
||||||
|
and waits for ``attach_first()`` to block in
|
||||||
|
``FspAdmin.open_trend_overlay()`` before inspecting the plot.
|
||||||
|
The unset ``attached`` event and empty plot prove attachment
|
||||||
|
waits for bootstrap rather than depending on timer luck. A
|
||||||
|
second order shares the writer; cleanup retains a sibling's
|
||||||
|
``_PriceChart._trend_overlays`` entry and its two Qt items.
|
||||||
|
|
||||||
|
'''
|
||||||
|
trio.run(_admin_bootstrap_scenario, qapp, monkeypatch)
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_filters_and_defers_hidden_revisions(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A destination rewrite while hidden must survive an unchanged
|
||||||
|
source-tail paint signature when the chart becomes visible.
|
||||||
|
An equal-bound earlier pivot edit arranges that case. Wrong
|
||||||
|
market/token/timeframe invalidations leave the layer alone;
|
||||||
|
a matching hidden notification invalidates without touching Qt
|
||||||
|
data, and the next ordinary refresh consumes the corrected pivot.
|
||||||
|
|
||||||
|
'''
|
||||||
|
chart: _PriceChart = _PriceChart()
|
||||||
|
source: np.ndarray = _source((10, 20, 30, 40, 50), (1,) * 5)
|
||||||
|
source['top_1'][1] = 105
|
||||||
|
source['top_time_1'][1] = 10
|
||||||
|
overlay: TrendOverlay = TrendOverlay(
|
||||||
|
chart.viz,
|
||||||
|
SimpleNamespace(array=source, _token='trend'),
|
||||||
|
fqme='admin.test',
|
||||||
|
source_period_s=10,
|
||||||
|
chart_period_s=10,
|
||||||
|
)
|
||||||
|
chart._trend_overlays = {('admin.test', 'trend'): [overlay]}
|
||||||
|
try:
|
||||||
|
assert update_trend_overlays(chart) == 1
|
||||||
|
original: list[ReversalPoint] = overlay.points
|
||||||
|
source['top_1'][1] = 109
|
||||||
|
assert update_trend_overlays(
|
||||||
|
chart, fqme='other.test', force=True,
|
||||||
|
) == 0
|
||||||
|
assert update_trend_overlays(
|
||||||
|
chart, shm_token='other', force=True,
|
||||||
|
) == 0
|
||||||
|
assert update_trend_overlays(
|
||||||
|
chart, chart_period_s=60, force=True,
|
||||||
|
) == 0
|
||||||
|
assert update_trend_overlays(chart) == 0
|
||||||
|
assert overlay.points is original
|
||||||
|
|
||||||
|
admin: FspAdmin = FspAdmin(
|
||||||
|
None, {}, SimpleNamespace(chart=chart), None,
|
||||||
|
)
|
||||||
|
admin._trend_overlays.append((chart, overlay))
|
||||||
|
admin.refresh_trend_overlays('trend', render=False)
|
||||||
|
assert overlay.points is original
|
||||||
|
assert overlay.points[0].price == 105
|
||||||
|
assert update_trend_overlays(chart) == 1
|
||||||
|
assert overlay.points[0].price == 109
|
||||||
|
finally:
|
||||||
|
overlay.remove()
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_context_exit_releases_layers(
|
||||||
|
qapp: QApplication,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A finished FSP-admin context must not leave its graphics attached
|
||||||
|
to a reused chart widget. Replace only actor-cluster startup and
|
||||||
|
engine allocation; ``open_fsp_admin()`` and its Trio nursery run
|
||||||
|
normal exit with actual Qt items. Empty
|
||||||
|
``FspAdmin._trend_overlays`` and ``_PriceChart._trend_overlays``
|
||||||
|
plus the signalled ``complete`` event from
|
||||||
|
``FspAdmin._registry`` prove graphics teardown and existing
|
||||||
|
engine shutdown notification both run. This local stub does not
|
||||||
|
prove remote actor shutdown or shared-memory visibility.
|
||||||
|
|
||||||
|
'''
|
||||||
|
chart: _PriceChart = _PriceChart()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def cluster_context(
|
||||||
|
**kwargs: object,
|
||||||
|
|
||||||
|
) -> AsyncIterator[tuple[bool, dict]]:
|
||||||
|
'''
|
||||||
|
Supply a local cluster stub without spawning actor processes.
|
||||||
|
|
||||||
|
'''
|
||||||
|
yield False, {}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
fsp_ui, 'maybe_open_context', cluster_context,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Observe cleanup after the actual admin context has exited.
|
||||||
|
|
||||||
|
'''
|
||||||
|
flume: SimpleNamespace = SimpleNamespace(
|
||||||
|
mkt=SimpleNamespace(fqme='admin.test'),
|
||||||
|
rt_shm=chart.viz.shm,
|
||||||
|
rt_sample_period_s=10,
|
||||||
|
_hist_shm_token=None,
|
||||||
|
)
|
||||||
|
source: np.ndarray = _source(
|
||||||
|
(10, 20, 30, 40, 50), (1,) * 5,
|
||||||
|
)
|
||||||
|
started: trio.Event = trio.Event()
|
||||||
|
started.set()
|
||||||
|
complete: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
async def start_engine(
|
||||||
|
target: Fsp,
|
||||||
|
conf: dict,
|
||||||
|
loglevel: str,
|
||||||
|
|
||||||
|
) -> tuple[SimpleNamespace, trio.Event]:
|
||||||
|
'''
|
||||||
|
Return an already published, test-owned FSP destination.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return SimpleNamespace(rt_shm=SimpleNamespace(
|
||||||
|
array=source,
|
||||||
|
_token='trend',
|
||||||
|
)), started
|
||||||
|
|
||||||
|
admin: FspAdmin
|
||||||
|
async with fsp_ui.open_fsp_admin(
|
||||||
|
SimpleNamespace(chart=chart), flume,
|
||||||
|
) as admin:
|
||||||
|
monkeypatch.setattr(
|
||||||
|
admin, 'start_engine_task', start_engine,
|
||||||
|
)
|
||||||
|
admin._registry[('test',)] = (None, None, complete)
|
||||||
|
await admin.open_trend_overlay()
|
||||||
|
assert len(chart.widget.plotItem.items) == 2
|
||||||
|
|
||||||
|
assert chart.widget.plotItem.items == []
|
||||||
|
assert chart._trend_overlays == {}
|
||||||
|
assert admin._trend_overlays == []
|
||||||
|
assert complete.is_set()
|
||||||
|
|
||||||
|
try:
|
||||||
|
trio.run(scenario)
|
||||||
|
finally:
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
@ -0,0 +1,611 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Actual shared-tracker graphics, freshness and native feed lifecycle.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections import deque
|
||||||
|
from collections.abc import AsyncIterator, Iterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from dataclasses import replace
|
||||||
|
import os
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
os.environ['QT_QPA_PLATFORM'] = 'offscreen'
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pyqtgraph as pg
|
||||||
|
import pytest
|
||||||
|
import trio
|
||||||
|
from trio.testing import wait_all_tasks_blocked
|
||||||
|
|
||||||
|
from piker.fsp._jev import JevClassification, RegimeAnswer
|
||||||
|
from piker.fsp._regime import (
|
||||||
|
MultiTimeframeTrend,
|
||||||
|
RegimeSnapshot,
|
||||||
|
TimeframeRegime,
|
||||||
|
TrendReversal,
|
||||||
|
)
|
||||||
|
from piker.ui import _fsp as fsp_ui
|
||||||
|
from piker.ui._trend import (
|
||||||
|
RegimeInterval,
|
||||||
|
ReversalPoint,
|
||||||
|
TrendOverlay,
|
||||||
|
)
|
||||||
|
from piker.ui._trend_mtf import (
|
||||||
|
SnapshotTrendOverlay,
|
||||||
|
_SnapshotLayer,
|
||||||
|
)
|
||||||
|
from piker.ui.qt import QApplication
|
||||||
|
|
||||||
|
|
||||||
|
class _ArrayShm:
|
||||||
|
'''
|
||||||
|
Model published source bounds over deterministic OHLC bars.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self, count: int = 600, visible: int = 401) -> None:
|
||||||
|
'''
|
||||||
|
Keep an allocated mutable suffix beyond the published range.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._array: np.ndarray = np.zeros(count, dtype=[
|
||||||
|
('index', 'i8'),
|
||||||
|
('time', 'f8'),
|
||||||
|
('close', 'f8'),
|
||||||
|
('low', 'f8'),
|
||||||
|
('high', 'f8'),
|
||||||
|
])
|
||||||
|
steps: np.ndarray = np.arange(count)
|
||||||
|
prices: np.ndarray = (
|
||||||
|
100 + .015 * steps
|
||||||
|
+ 2 * np.sin(steps * .09)
|
||||||
|
+ .5 * np.sin(steps * .4)
|
||||||
|
)
|
||||||
|
self._array['index'] = 1000 + steps
|
||||||
|
self._array['time'] = steps
|
||||||
|
self._array['close'] = prices
|
||||||
|
self._array['low'] = prices - .2
|
||||||
|
self._array['high'] = prices + .2
|
||||||
|
self._first: SimpleNamespace = SimpleNamespace(value=0)
|
||||||
|
self._last: SimpleNamespace = SimpleNamespace(value=visible)
|
||||||
|
self._token: str = 'native-ohlcv'
|
||||||
|
|
||||||
|
@property
|
||||||
|
def array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Expose only the current published SHM range.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self._array[self._first.value:self._last.value]
|
||||||
|
|
||||||
|
|
||||||
|
class _Chart:
|
||||||
|
'''
|
||||||
|
Use a real price plot while replacing actor and SHM allocation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
_trend_overlays: dict[tuple[str, str], list[TrendOverlay]]
|
||||||
|
|
||||||
|
def __init__(self, shm: _ArrayShm) -> None:
|
||||||
|
'''
|
||||||
|
Bind the actual base source to an offscreen plot.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.widget: pg.PlotWidget = pg.PlotWidget()
|
||||||
|
self.viz: SimpleNamespace = SimpleNamespace(
|
||||||
|
plot=self.widget.plotItem,
|
||||||
|
shm=shm,
|
||||||
|
index_field='index',
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_viz(self, fqme: str) -> SimpleNamespace|None:
|
||||||
|
'''
|
||||||
|
Resolve the only market owned by this chart.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self.viz if fqme == 'mtf.test' else None
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
'''
|
||||||
|
Release the disposable Qt widget.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self.widget.close()
|
||||||
|
self.widget.deleteLater()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope='session')
|
||||||
|
def qapp() -> Iterator[QApplication]:
|
||||||
|
'''
|
||||||
|
Keep one offscreen application alive for native graphics checks.
|
||||||
|
|
||||||
|
'''
|
||||||
|
app: QApplication|None = QApplication.instance()
|
||||||
|
if app is None:
|
||||||
|
app = QApplication(['piker-mtf-tests'])
|
||||||
|
app.setQuitOnLastWindowClosed(False)
|
||||||
|
yield app
|
||||||
|
app.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def _replay(
|
||||||
|
tracker: MultiTimeframeTrend,
|
||||||
|
shm: _ArrayShm,
|
||||||
|
overlay: SnapshotTrendOverlay|None = None,
|
||||||
|
start: int = 0,
|
||||||
|
|
||||||
|
) -> RegimeSnapshot:
|
||||||
|
'''
|
||||||
|
Replay final closes through the production shared recurrence.
|
||||||
|
|
||||||
|
'''
|
||||||
|
snapshot: RegimeSnapshot
|
||||||
|
row: np.void
|
||||||
|
for row in shm.array[start:-1]:
|
||||||
|
snapshot = tracker.push(
|
||||||
|
float(row['time']),
|
||||||
|
float(row['close']),
|
||||||
|
index=int(row['index']),
|
||||||
|
)
|
||||||
|
if overlay is not None:
|
||||||
|
overlay.submit(snapshot)
|
||||||
|
return snapshot
|
||||||
|
|
||||||
|
|
||||||
|
def _classification(
|
||||||
|
snapshot: RegimeSnapshot,
|
||||||
|
*,
|
||||||
|
received_at: float|None = None,
|
||||||
|
|
||||||
|
) -> JevClassification:
|
||||||
|
'''
|
||||||
|
Build explicit synthetic provider output without any HTTP path.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return JevClassification(
|
||||||
|
market=snapshot.market,
|
||||||
|
revision=snapshot.revision,
|
||||||
|
sequence=snapshot.sequence,
|
||||||
|
segment=snapshot.segment,
|
||||||
|
source_as_of=snapshot.as_of,
|
||||||
|
requested_at=snapshot.as_of,
|
||||||
|
received_at=(
|
||||||
|
received_at
|
||||||
|
if received_at is not None
|
||||||
|
else snapshot.as_of
|
||||||
|
),
|
||||||
|
latency=.1,
|
||||||
|
model='synthetic-jev-test',
|
||||||
|
prompt_version='test',
|
||||||
|
answers=(RegimeAnswer(
|
||||||
|
question='regime_5',
|
||||||
|
period=5,
|
||||||
|
label='range',
|
||||||
|
probability=.8,
|
||||||
|
confidence=.6,
|
||||||
|
probabilities=(('range', .8), ('unknown', .2)),
|
||||||
|
),),
|
||||||
|
input_tokens=0,
|
||||||
|
output_tokens=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_bad_limits_do_not_allocate_graphics(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Reject fractional and boolean bounds before attaching Qt items.
|
||||||
|
|
||||||
|
A failed child constructor cannot be cleaned up by the parent
|
||||||
|
because the parent has not received it yet. Exercise the public
|
||||||
|
scalar and snapshot constructors against an actual empty plot.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm()
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
try:
|
||||||
|
name: str
|
||||||
|
for name in ('history_limit', 'order'):
|
||||||
|
value: float|bool
|
||||||
|
for value in (1.5, True):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
SnapshotTrendOverlay(
|
||||||
|
chart.viz, shm,
|
||||||
|
fqme='mtf.test', base_period=1,
|
||||||
|
**{name: value},
|
||||||
|
)
|
||||||
|
assert not chart.widget.plotItem.items
|
||||||
|
|
||||||
|
for name in ('max_points', 'max_rectangles', 'order'):
|
||||||
|
value: float|bool
|
||||||
|
for value in (1.5, True):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
TrendOverlay(
|
||||||
|
chart.viz, shm,
|
||||||
|
fqme='mtf.test', source_period_s=1,
|
||||||
|
chart_period_s=1,
|
||||||
|
**{name: value},
|
||||||
|
)
|
||||||
|
assert not chart.widget.plotItem.items
|
||||||
|
finally:
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('periods', [
|
||||||
|
(1., 5., 15.),
|
||||||
|
(15.000000000000002, 1.0000000000000002, 5.000000000000001),
|
||||||
|
])
|
||||||
|
def test_bootstrap_uses_real_timeframes_and_endpoint_pivots(
|
||||||
|
qapp: QApplication,
|
||||||
|
periods: tuple[float, ...],
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Sampling 1 s, 5 s and 15 s bars must aggregate one base stream,
|
||||||
|
rather than relabel identical computations. A real tracker and
|
||||||
|
scalar sine-wave closes create distinct reversal sets. The
|
||||||
|
snapshot's pivot end timestamp remains metadata while its x
|
||||||
|
coordinate equals the final base bar's ``TrendReversal.index``.
|
||||||
|
Bootstrap supplies no old trend-row history, so rectangles may
|
||||||
|
begin only at each timeframe's latest proven as_of timestamp.
|
||||||
|
Harmless decimal rounding must normalize to the tracker's exact
|
||||||
|
duration keys even if the caller lists timeframes out of order.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm()
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'mtf.test', periods=periods, history_limit=256,
|
||||||
|
)
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
chart.viz, shm, fqme='mtf.test', base_period=1,
|
||||||
|
periods=periods,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
snapshot: RegimeSnapshot = _replay(tracker, shm)
|
||||||
|
assert adapter.submit(snapshot)
|
||||||
|
assert adapter.periods == (1., 5., 15.)
|
||||||
|
assert len(chart.widget.plotItem.items) == 6
|
||||||
|
point_counts: list[int] = []
|
||||||
|
frame: TimeframeRegime
|
||||||
|
for frame in snapshot.timeframes:
|
||||||
|
layer: _SnapshotLayer = adapter.layers[frame.period]
|
||||||
|
event: TrendReversal
|
||||||
|
events: list[TrendReversal] = [
|
||||||
|
event for event in frame.reversals
|
||||||
|
if event.order == 1
|
||||||
|
]
|
||||||
|
assert len(layer.points) == len(events)
|
||||||
|
assert layer.points
|
||||||
|
point_counts.append(len(layer.points))
|
||||||
|
point: ReversalPoint
|
||||||
|
event: TrendReversal
|
||||||
|
for point, event in zip(
|
||||||
|
layer.points, events, strict=True,
|
||||||
|
):
|
||||||
|
assert point.x == event.index
|
||||||
|
assert point.pivot_time == event.time
|
||||||
|
assert point.detected_at == event.detected_time
|
||||||
|
assert point.timestamp_convention == 'interval_end'
|
||||||
|
assert layer.rectangles.intervals
|
||||||
|
span: RegimeInterval # noqa: F842
|
||||||
|
assert all(
|
||||||
|
span.start_time >= frame.as_of
|
||||||
|
for span in layer.rectangles.intervals
|
||||||
|
)
|
||||||
|
assert len(set(point_counts)) == 3
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_revision_segment_and_sequence_revoke_evidence(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A corrected source or new gap segment invalidates every previous
|
||||||
|
pivot and model annotation before replay finishes. The adapter
|
||||||
|
receives real tracker output, then an explicit generation change.
|
||||||
|
Old market/revision/sequence snapshots and an earlier segment's
|
||||||
|
model response must fail acceptance without reviving geometry.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm()
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
chart.viz, shm, fqme='mtf.test', base_period=1,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
snapshot: RegimeSnapshot = _replay(tracker, shm)
|
||||||
|
assert adapter.submit(snapshot)
|
||||||
|
result: JevClassification = _classification(snapshot)
|
||||||
|
assert adapter.submit_classification(result)
|
||||||
|
assert not adapter.submit(snapshot)
|
||||||
|
assert not adapter.submit(
|
||||||
|
replace(snapshot, market='foreign'),
|
||||||
|
)
|
||||||
|
adapter.invalidate(1)
|
||||||
|
layer: _SnapshotLayer # noqa: F842
|
||||||
|
assert all(
|
||||||
|
not layer.points for layer in adapter.layers.values()
|
||||||
|
)
|
||||||
|
assert not adapter.submit(snapshot)
|
||||||
|
assert not adapter.submit_classification(result)
|
||||||
|
|
||||||
|
tracker.reset(revision=1)
|
||||||
|
replayed: RegimeSnapshot = _replay(tracker, shm)
|
||||||
|
assert adapter.submit(replayed)
|
||||||
|
result = _classification(replayed)
|
||||||
|
assert adapter.submit_classification(result)
|
||||||
|
after_gap: RegimeSnapshot = tracker.push(
|
||||||
|
405, 101, index=1405,
|
||||||
|
)
|
||||||
|
assert adapter.submit(after_gap)
|
||||||
|
assert adapter.segment == 1
|
||||||
|
assert not adapter.submit_classification(result)
|
||||||
|
assert all(
|
||||||
|
not layer.points for layer in adapter.layers.values()
|
||||||
|
)
|
||||||
|
runs: deque[RegimeInterval] # noqa: F842
|
||||||
|
assert all(not runs for runs in adapter._model_runs.values())
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_coalesced_buckets_leave_unknown_regime_hole(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Coalesced callbacks reveal only the latest trend classification.
|
||||||
|
Extending ``_SnapshotLayer._runs`` to the next received snapshot
|
||||||
|
invents historical knowledge. Skip ten actual finalized buckets
|
||||||
|
and prove the old interval stops at its next possible decision
|
||||||
|
while the new run starts only at ``RegimeSnapshot.as_of``.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm(visible=101)
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
chart.viz, shm, fqme='mtf.test', base_period=1,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
adapter.submit(_replay(tracker, shm))
|
||||||
|
previous: float = adapter.as_of
|
||||||
|
shm._last.value += 10
|
||||||
|
latest: RegimeSnapshot = _replay(tracker, shm, start=100)
|
||||||
|
adapter.submit(latest)
|
||||||
|
runs: deque[RegimeInterval] = adapter.layers[1]._runs
|
||||||
|
assert len(runs) == 2
|
||||||
|
assert runs[0].end_time == previous + 1
|
||||||
|
assert runs[1].start_time == latest.as_of
|
||||||
|
assert runs[0].end_time < runs[1].start_time
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_receipt_expiry_and_delayed_queue(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A Jev response must neither paint before receipt nor claim the
|
||||||
|
live regime indefinitely. Retain a response arriving 2.5 s
|
||||||
|
after its source snapshot, advance the real base stream, and
|
||||||
|
verify its visible interval starts at receipt and stops after
|
||||||
|
the default 5 s lifetime. Old first-ever responses and expired
|
||||||
|
queued results are also rejected despite matching provenance.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm(visible=201)
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
chart.viz, shm, fqme='mtf.test', base_period=1,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
snapshot: RegimeSnapshot = _replay(tracker, shm)
|
||||||
|
adapter.submit(snapshot)
|
||||||
|
old: JevClassification = replace(
|
||||||
|
_classification(snapshot),
|
||||||
|
sequence=1,
|
||||||
|
source_as_of=1,
|
||||||
|
received_at=200,
|
||||||
|
)
|
||||||
|
assert not adapter.submit_classification(old)
|
||||||
|
expired: JevClassification = _classification(
|
||||||
|
snapshot, received_at=190,
|
||||||
|
)
|
||||||
|
assert not adapter.submit_classification(expired)
|
||||||
|
|
||||||
|
result: JevClassification = _classification(
|
||||||
|
snapshot, received_at=202.5,
|
||||||
|
)
|
||||||
|
assert adapter.submit_classification(result)
|
||||||
|
layer: _SnapshotLayer = adapter.layers[5]
|
||||||
|
assert len(layer.rectangles.rectangles) == 0
|
||||||
|
shm._last.value = 212
|
||||||
|
_replay(tracker, shm, adapter, start=200)
|
||||||
|
assert layer.rectangles.intervals
|
||||||
|
span: RegimeInterval = layer.rectangles.intervals[-1]
|
||||||
|
assert span.observed_at == 202.5
|
||||||
|
assert span.end_time == 207.5
|
||||||
|
assert layer.rectangles.rectangles[-1, 0] == 1202.5
|
||||||
|
assert not adapter.submit_classification(result)
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_retention_stays_bounded(
|
||||||
|
qapp: QApplication,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A long-running shared tracker must not accumulate unbounded Qt
|
||||||
|
objects or regime intervals. Stream hundreds of real snapshots
|
||||||
|
with a four-event display budget and assert retention at both
|
||||||
|
the data and scene boundaries after many reversals.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm()
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend('mtf.test')
|
||||||
|
adapter: SnapshotTrendOverlay = SnapshotTrendOverlay(
|
||||||
|
chart.viz,
|
||||||
|
shm,
|
||||||
|
fqme='mtf.test',
|
||||||
|
base_period=1,
|
||||||
|
history_limit=4,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_replay(tracker, shm, adapter)
|
||||||
|
assert len(chart.widget.plotItem.items) == 6
|
||||||
|
layer: _SnapshotLayer
|
||||||
|
for layer in adapter.layers.values():
|
||||||
|
assert len(layer.points) <= 4
|
||||||
|
assert len(layer._runs) <= 4
|
||||||
|
assert len(layer.rectangles.rectangles) <= 4
|
||||||
|
finally:
|
||||||
|
adapter.remove()
|
||||||
|
adapter.remove()
|
||||||
|
assert chart.widget.plotItem.items == []
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('max_source_age', [5, None])
|
||||||
|
def test_admin_consumes_native_source_once_and_cancels(
|
||||||
|
qapp: QApplication,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
max_source_age: float|None,
|
||||||
|
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
``FspAdmin._run_multi_timeframe_trends()`` must subscribe before
|
||||||
|
replay, share one tracker across three periods, and invalidate
|
||||||
|
graphics while model callbacks apply their freshness gate.
|
||||||
|
The ``sampler()`` stub yields a local Trio receive channel to
|
||||||
|
the actual ``consume_regime_source()`` task. Explicit append and
|
||||||
|
equal-bound correction notices, followed by
|
||||||
|
``wait_all_tasks_blocked()``, prove each notice is consumed
|
||||||
|
before checking snapshot sequence and revision.
|
||||||
|
``FspAdmin.close_trend_overlays()`` must cancel the reader:
|
||||||
|
``closed`` is set only in ``sampler()`` cleanup, before this test
|
||||||
|
closes either channel endpoint. Empty Qt items and
|
||||||
|
``_Chart._trend_overlays`` prove owned graphics also detach.
|
||||||
|
This controls local task ordering without exercising actor IPC.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: _ArrayShm = _ArrayShm(visible=201)
|
||||||
|
chart: _Chart = _Chart(shm)
|
||||||
|
submitted: list[RegimeSnapshot] = []
|
||||||
|
invalidated: list[int] = []
|
||||||
|
subscription_count: list[float] = []
|
||||||
|
|
||||||
|
async def scenario() -> None:
|
||||||
|
'''
|
||||||
|
Control publication and cancellation with native Trio events.
|
||||||
|
|
||||||
|
'''
|
||||||
|
send: trio.MemorySendChannel[dict]
|
||||||
|
receive: trio.MemoryReceiveChannel[dict]
|
||||||
|
send, receive = trio.open_memory_channel(4)
|
||||||
|
closed: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def sampler(
|
||||||
|
period_s: float,
|
||||||
|
|
||||||
|
) -> AsyncIterator[trio.MemoryReceiveChannel[dict]]:
|
||||||
|
'''
|
||||||
|
Open exactly one sampler before the real consumer starts.
|
||||||
|
|
||||||
|
'''
|
||||||
|
subscription_count.append(period_s)
|
||||||
|
try:
|
||||||
|
yield receive
|
||||||
|
finally:
|
||||||
|
closed.set()
|
||||||
|
|
||||||
|
monkeypatch.setattr(fsp_ui, 'open_sample_stream', sampler)
|
||||||
|
linked: SimpleNamespace = SimpleNamespace(
|
||||||
|
chart=chart,
|
||||||
|
isHidden=lambda: False,
|
||||||
|
)
|
||||||
|
flume: SimpleNamespace = SimpleNamespace(
|
||||||
|
mkt=SimpleNamespace(fqme='mtf.test'),
|
||||||
|
rt_shm=shm,
|
||||||
|
rt_sample_period_s=1,
|
||||||
|
)
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
admin: fsp_ui.FspAdmin = fsp_ui.FspAdmin(
|
||||||
|
nursery, {}, linked, flume,
|
||||||
|
)
|
||||||
|
adapter: SnapshotTrendOverlay = (
|
||||||
|
await admin.open_multi_timeframe_trends(
|
||||||
|
on_snapshot=submitted.append,
|
||||||
|
on_invalidate=invalidated.append,
|
||||||
|
max_source_age=max_source_age,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
await admin.open_multi_timeframe_trends() is adapter
|
||||||
|
)
|
||||||
|
assert subscription_count == [1]
|
||||||
|
assert len(chart.widget.plotItem.items) == 6
|
||||||
|
assert adapter.sequence == 200
|
||||||
|
assert bool(submitted) == (max_source_age is None)
|
||||||
|
|
||||||
|
shm._last.value += 1
|
||||||
|
await send.send({'period': 1})
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert adapter.sequence == 201
|
||||||
|
|
||||||
|
shm._array['close'][20] += 20
|
||||||
|
await send.send({'backfilling': ('mtf.test', 1)})
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
assert invalidated == [1]
|
||||||
|
assert adapter.revision == 1
|
||||||
|
assert adapter.sequence == 201
|
||||||
|
admin.close_trend_overlays()
|
||||||
|
await closed.wait()
|
||||||
|
assert chart._trend_overlays == {}
|
||||||
|
assert chart.widget.plotItem.items == []
|
||||||
|
await send.aclose()
|
||||||
|
await receive.aclose()
|
||||||
|
|
||||||
|
try:
|
||||||
|
trio.run(scenario)
|
||||||
|
finally:
|
||||||
|
chart.close()
|
||||||
|
qapp.processEvents()
|
||||||
|
|
@ -0,0 +1,430 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Independent batch FHLS oracle and pinned original-Lua observations.
|
||||||
|
|
||||||
|
The oracle materializes complete windows and an expanded sequence of
|
||||||
|
FHLS summaries. It shares neither compiled helpers nor mutable window
|
||||||
|
state with the production calculator. See docs/trend_calculus.rst.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from itertools import product
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TypedDict
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from piker.fsp._trend import (
|
||||||
|
TrendCalculator,
|
||||||
|
TrendHistory,
|
||||||
|
TrendUpdate,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _LuaCase(TypedDict):
|
||||||
|
'''
|
||||||
|
Describe the input and observed events of one pinned Lua run.
|
||||||
|
|
||||||
|
'''
|
||||||
|
window: int
|
||||||
|
values: list[float]
|
||||||
|
events: list[list[int|float]]
|
||||||
|
|
||||||
|
|
||||||
|
class _LuaFixture(TypedDict):
|
||||||
|
'''
|
||||||
|
Type the captured Lua cases consumed by the parity regression.
|
||||||
|
|
||||||
|
'''
|
||||||
|
cases: list[_LuaCase]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Point:
|
||||||
|
'''
|
||||||
|
Retain a pivot's source position and earliest availability row.
|
||||||
|
|
||||||
|
'''
|
||||||
|
row: int
|
||||||
|
price: float
|
||||||
|
available: int
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Window:
|
||||||
|
'''
|
||||||
|
Summarize an entire immutable observation window using extrema.
|
||||||
|
|
||||||
|
'''
|
||||||
|
high: Point
|
||||||
|
low: Point
|
||||||
|
available: int
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def summarize(
|
||||||
|
cls: type['Window'],
|
||||||
|
points: list[Point],
|
||||||
|
) -> 'Window':
|
||||||
|
'''
|
||||||
|
Python min/max retain the earliest equal extremum.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return cls(
|
||||||
|
high=max(points, key=lambda point: point.price),
|
||||||
|
low=min(points, key=lambda point: point.price),
|
||||||
|
available=points[-1].available,
|
||||||
|
)
|
||||||
|
|
||||||
|
def compare(self, previous: 'Window') -> int:
|
||||||
|
'''
|
||||||
|
Compare extrema ordinally, avoiding floating subtraction.
|
||||||
|
|
||||||
|
'''
|
||||||
|
if (
|
||||||
|
(
|
||||||
|
self.high.price >= previous.high.price
|
||||||
|
and self.low.price >= previous.low.price
|
||||||
|
)
|
||||||
|
and
|
||||||
|
(
|
||||||
|
self.high.price > previous.high.price
|
||||||
|
or self.low.price > previous.low.price
|
||||||
|
)
|
||||||
|
):
|
||||||
|
return 1
|
||||||
|
if (
|
||||||
|
(
|
||||||
|
self.high.price <= previous.high.price
|
||||||
|
and self.low.price <= previous.low.price
|
||||||
|
)
|
||||||
|
and
|
||||||
|
(
|
||||||
|
self.high.price < previous.high.price
|
||||||
|
or self.low.price < previous.low.price
|
||||||
|
)
|
||||||
|
):
|
||||||
|
return -1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def reference_order(
|
||||||
|
points: list[Point],
|
||||||
|
size: int,
|
||||||
|
*,
|
||||||
|
lua_seed: bool = False,
|
||||||
|
) -> tuple[list[tuple[int, int]], list[tuple[Point, int]]]:
|
||||||
|
'''
|
||||||
|
Expand complex windows, then derive directions and pivot events.
|
||||||
|
|
||||||
|
Direction changes from unknown zero establish a baseline and do
|
||||||
|
not manufacture a reversal. ``Point.available`` carries detection
|
||||||
|
availability from the closing observation, including at recursive
|
||||||
|
orders.
|
||||||
|
|
||||||
|
'''
|
||||||
|
start: int # noqa: F842
|
||||||
|
windows: list[Window] = [
|
||||||
|
Window.summarize(points[start:start + size])
|
||||||
|
for start in range(0, len(points) - size + 1, size)
|
||||||
|
]
|
||||||
|
if lua_seed and windows:
|
||||||
|
last: Point = points[size - 1]
|
||||||
|
windows.insert(0, Window(last, last, last.available))
|
||||||
|
if len(windows) < 2:
|
||||||
|
return [], []
|
||||||
|
|
||||||
|
expanded: list[Window] = [windows[0]]
|
||||||
|
current: Window
|
||||||
|
previous: Window
|
||||||
|
for previous, current in zip(windows, windows[1:]):
|
||||||
|
if current.compare(previous) == 0:
|
||||||
|
second: Point = max(
|
||||||
|
(previous.low, previous.high),
|
||||||
|
key=lambda point: point.row,
|
||||||
|
)
|
||||||
|
first: Point = min(
|
||||||
|
(current.low, current.high),
|
||||||
|
key=lambda point: point.row,
|
||||||
|
)
|
||||||
|
# Lua's synthetic bridge breaks equal-value ties toward
|
||||||
|
# its second endpoint for high and first for low.
|
||||||
|
bridge_high: Point
|
||||||
|
bridge_low: Point
|
||||||
|
bridge_low, bridge_high = sorted(
|
||||||
|
[second, first], key=lambda point: point.price,
|
||||||
|
)
|
||||||
|
expanded.append(Window(
|
||||||
|
high=bridge_high,
|
||||||
|
low=bridge_low,
|
||||||
|
available=current.available,
|
||||||
|
))
|
||||||
|
expanded.append(current)
|
||||||
|
|
||||||
|
trend: int = 0
|
||||||
|
directions: list[tuple[int, int]] = []
|
||||||
|
events: list[tuple[Point, int]] = []
|
||||||
|
previous: Window
|
||||||
|
current: Window
|
||||||
|
for previous, current in zip(expanded, expanded[1:]):
|
||||||
|
direction: int = current.compare(previous) or trend
|
||||||
|
if trend and direction != trend:
|
||||||
|
pivot: Point = (
|
||||||
|
previous.low if direction == 1 else previous.high
|
||||||
|
)
|
||||||
|
events.append((
|
||||||
|
Point(pivot.row, pivot.price, current.available),
|
||||||
|
direction,
|
||||||
|
))
|
||||||
|
trend = direction
|
||||||
|
directions.append((current.available, trend))
|
||||||
|
return directions, events
|
||||||
|
|
||||||
|
|
||||||
|
def reference_hierarchy(
|
||||||
|
values: np.ndarray,
|
||||||
|
sizes: tuple[int, ...],
|
||||||
|
*,
|
||||||
|
lua_seed: bool = False,
|
||||||
|
) -> tuple[
|
||||||
|
np.ndarray,
|
||||||
|
np.ndarray,
|
||||||
|
list[tuple[int, int, int, int]],
|
||||||
|
]:
|
||||||
|
'''
|
||||||
|
Repeatedly reduce whole point sequences, separately per order.
|
||||||
|
|
||||||
|
The returned event tuples are pivot row, direction, order and
|
||||||
|
detection row. Sorting by detection row and order reproduces
|
||||||
|
causal publication while retaining stable same-order ordering.
|
||||||
|
|
||||||
|
'''
|
||||||
|
row: int
|
||||||
|
price: np.floating|np.integer # noqa: F842
|
||||||
|
points: list[Point] = [
|
||||||
|
Point(row, float(price), row)
|
||||||
|
for row, price in enumerate(values)
|
||||||
|
]
|
||||||
|
trends: np.ndarray = np.zeros(
|
||||||
|
(len(values), len(sizes)), dtype=np.int8,
|
||||||
|
)
|
||||||
|
ready: np.ndarray = np.zeros_like(trends, dtype=np.bool_)
|
||||||
|
all_events: list[tuple[int, int, int, int]] = []
|
||||||
|
level: int
|
||||||
|
size: int
|
||||||
|
for level, size in enumerate(sizes):
|
||||||
|
directions: list[tuple[int, int]]
|
||||||
|
events: list[tuple[Point, int]]
|
||||||
|
directions, events = reference_order(
|
||||||
|
points, size, lua_seed=lua_seed,
|
||||||
|
)
|
||||||
|
if len(points) >= 2 * size:
|
||||||
|
ready[points[2 * size - 1].available:, level] = True
|
||||||
|
row: int
|
||||||
|
direction: int
|
||||||
|
for row, direction in directions:
|
||||||
|
trends[row:, level] = direction
|
||||||
|
point: Point
|
||||||
|
direction: int
|
||||||
|
for point, direction in events:
|
||||||
|
all_events.append((
|
||||||
|
point.row, direction, level + 1, point.available,
|
||||||
|
))
|
||||||
|
point: Point
|
||||||
|
direction: int
|
||||||
|
points = [point for point, direction in events]
|
||||||
|
all_events.sort(key=lambda event: (event[3], event[2]))
|
||||||
|
# Public direction remains unresolved until two real windows,
|
||||||
|
# including when the research seed establishes internal state.
|
||||||
|
trends[~ready] = 0
|
||||||
|
return trends, ready, all_events
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('sizes', [(2, 2), (3, 2), (4, 3)])
|
||||||
|
@pytest.mark.parametrize('lua_seed', [False, True])
|
||||||
|
def test_exhaustive_ternary_histories(
|
||||||
|
sizes: tuple[int, ...],
|
||||||
|
lua_seed: bool,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Enumerate ties, inside/outside bars and unresolved flat startup.
|
||||||
|
|
||||||
|
Every eight-point sequence over three prices is compared with
|
||||||
|
independently expanded ``reference_hierarchy()`` windows.
|
||||||
|
Six-point sequences also exercise the optional Lua seed, without
|
||||||
|
importing its CLI's output-suppression bugs. Readiness still
|
||||||
|
requires two actual complete windows. Comparing every
|
||||||
|
``TrendHistory`` field catches incorrect first/second-extremum
|
||||||
|
ordering and warmup decisions that agreement between two calls
|
||||||
|
into the same production kernel cannot detect.
|
||||||
|
|
||||||
|
'''
|
||||||
|
length: int = 6 if lua_seed else 8
|
||||||
|
sequence: tuple[int, ...]
|
||||||
|
for sequence in product((-1, 0, 1), repeat=length):
|
||||||
|
values: np.ndarray = np.array(sequence, dtype=np.float64)
|
||||||
|
expected: np.ndarray
|
||||||
|
readiness: np.ndarray
|
||||||
|
events: list[tuple[int, int, int, int]]
|
||||||
|
(
|
||||||
|
expected,
|
||||||
|
readiness,
|
||||||
|
events,
|
||||||
|
) = reference_hierarchy(
|
||||||
|
values, sizes, lua_seed=lua_seed,
|
||||||
|
)
|
||||||
|
actual: TrendHistory = TrendCalculator(
|
||||||
|
sizes, lua_seed=lua_seed,
|
||||||
|
).extend(
|
||||||
|
values, np.arange(len(values), dtype=np.float64),
|
||||||
|
)
|
||||||
|
assert np.array_equal(actual.trends, expected), sequence
|
||||||
|
assert np.array_equal(actual.ready, readiness), sequence
|
||||||
|
pulses: np.ndarray = np.zeros_like(expected)
|
||||||
|
prices: np.ndarray = np.full(
|
||||||
|
(len(values), len(sizes), 2), np.nan,
|
||||||
|
)
|
||||||
|
times: np.ndarray = np.full_like(prices, np.nan)
|
||||||
|
pivot: int
|
||||||
|
direction: int
|
||||||
|
order: int
|
||||||
|
detected: int
|
||||||
|
for (
|
||||||
|
pivot,
|
||||||
|
direction,
|
||||||
|
order,
|
||||||
|
detected,
|
||||||
|
) in events:
|
||||||
|
side: int = int(direction == 1)
|
||||||
|
pulses[detected, order - 1] = direction
|
||||||
|
prices[detected, order - 1, side] = values[pivot]
|
||||||
|
times[detected, order - 1, side] = pivot
|
||||||
|
assert np.array_equal(actual.pulses, pulses), sequence
|
||||||
|
assert np.array_equal(
|
||||||
|
actual.pivot_prices, prices, equal_nan=True,
|
||||||
|
), sequence
|
||||||
|
assert np.array_equal(
|
||||||
|
actual.pivot_times, times, equal_nan=True,
|
||||||
|
), sequence
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('sizes', [(2,) * 5, (3, 2, 4), (5, 3, 2)])
|
||||||
|
@pytest.mark.parametrize('seed', range(8))
|
||||||
|
@pytest.mark.parametrize('lua_seed', [False, True])
|
||||||
|
def test_random_recursive_events(
|
||||||
|
sizes: tuple[int, ...],
|
||||||
|
seed: int,
|
||||||
|
lua_seed: bool,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Preserve recursive pivot causality across irregular coordinates.
|
||||||
|
|
||||||
|
Higher-order pivots occur earlier than their availability, often
|
||||||
|
sharing one detection row. Rounded random walks create tied and
|
||||||
|
engulfing windows; ``reference_hierarchy()`` predicts every
|
||||||
|
event independently of ``TrendCalculator.push()``. Gapped source
|
||||||
|
indices and nonuniform timestamps prove that propagation retains
|
||||||
|
source identity and detection time. Field equality and complete
|
||||||
|
ordered-event equality detect lost or backdated reversals.
|
||||||
|
|
||||||
|
'''
|
||||||
|
rng: np.random.Generator = np.random.default_rng(seed)
|
||||||
|
values: np.ndarray = np.cumsum(rng.integers(-3, 4, 512))
|
||||||
|
times: np.ndarray = 1000 + np.cumsum(rng.uniform(.1, 2, 512))
|
||||||
|
indices: np.ndarray = 800 + np.cumsum(rng.integers(1, 4, 512))
|
||||||
|
expected: np.ndarray
|
||||||
|
readiness: np.ndarray
|
||||||
|
events: list[tuple[int, int, int, int]]
|
||||||
|
(
|
||||||
|
expected,
|
||||||
|
readiness,
|
||||||
|
events,
|
||||||
|
) = reference_hierarchy(
|
||||||
|
values, sizes, lua_seed=lua_seed,
|
||||||
|
)
|
||||||
|
calculator: TrendCalculator = TrendCalculator(
|
||||||
|
sizes, lua_seed=lua_seed,
|
||||||
|
)
|
||||||
|
actual_events: list[tuple[int, int, int, int]] = []
|
||||||
|
row: int
|
||||||
|
value: np.integer
|
||||||
|
for row, value in enumerate(values):
|
||||||
|
update: TrendUpdate = calculator.push(
|
||||||
|
float(times[row]), float(value), index=int(indices[row]),
|
||||||
|
)
|
||||||
|
assert update.trends == tuple(expected[row])
|
||||||
|
assert update.ready == tuple(readiness[row])
|
||||||
|
event: np.void
|
||||||
|
for event in update.reversals:
|
||||||
|
pivot_row: int = int(np.searchsorted(
|
||||||
|
indices, event['index'],
|
||||||
|
))
|
||||||
|
assert event['price'] == values[pivot_row]
|
||||||
|
assert event['time'] == times[pivot_row]
|
||||||
|
assert event['detected_time'] == times[row]
|
||||||
|
assert event['detected_index'] == indices[row]
|
||||||
|
assert pivot_row <= row
|
||||||
|
actual_events.append((
|
||||||
|
pivot_row, int(event['direction']),
|
||||||
|
int(event['order']), row,
|
||||||
|
))
|
||||||
|
assert actual_events == events
|
||||||
|
|
||||||
|
|
||||||
|
def test_original_lua_fixture() -> None:
|
||||||
|
'''
|
||||||
|
Compare production events with actual, unmodified upstream Lua.
|
||||||
|
|
||||||
|
The fixture uses increasing initial windows to establish a trend
|
||||||
|
before reversals, avoiding Lua's documented flat-start crash and
|
||||||
|
initial simple-reversal suppression. Every later pivot, sign and
|
||||||
|
earliest detection prefix must agree, including tied extrema.
|
||||||
|
Comparing complete ``TrendUpdate.reversals`` sequences against
|
||||||
|
captured Lua output checks an implementation independent of the
|
||||||
|
Python oracle. Fixture provenance records the exact program and
|
||||||
|
input hashes; this case does not claim parity at Lua's startup
|
||||||
|
failures.
|
||||||
|
|
||||||
|
'''
|
||||||
|
path: Path = (
|
||||||
|
Path(__file__).parent
|
||||||
|
/ '_inputs'
|
||||||
|
/ 'trend_calculus_lua.json'
|
||||||
|
)
|
||||||
|
fixture: _LuaFixture = json.loads(path.read_text())
|
||||||
|
case: _LuaCase
|
||||||
|
for case in fixture['cases']:
|
||||||
|
calculator: TrendCalculator = TrendCalculator(
|
||||||
|
(case['window'],),
|
||||||
|
)
|
||||||
|
events: list[list[int|float]] = []
|
||||||
|
row: int
|
||||||
|
value: float
|
||||||
|
for row, value in enumerate(case['values']):
|
||||||
|
update: TrendUpdate = calculator.push(float(row), value)
|
||||||
|
event: np.void
|
||||||
|
for event in update.reversals:
|
||||||
|
events.append([
|
||||||
|
int(event['index']), float(event['price']),
|
||||||
|
int(event['direction']),
|
||||||
|
int(event['detected_index']),
|
||||||
|
])
|
||||||
|
assert events == case['events'], case['window']
|
||||||
|
|
@ -0,0 +1,312 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Closed-bar evidence contracts for multiple-timeframe classifiers.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from dataclasses import replace
|
||||||
|
import json
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from piker.fsp._regime import (
|
||||||
|
MultiTimeframeTrend,
|
||||||
|
RegimeSnapshot,
|
||||||
|
TimeframeRegime,
|
||||||
|
TrendReversal,
|
||||||
|
)
|
||||||
|
from piker.fsp._trend import TrendCalculator, TrendUpdate
|
||||||
|
|
||||||
|
|
||||||
|
def test_timeframes_match_independent_closed_bar_calculators(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Aggregating high/low extrema invents their intrabar order and can
|
||||||
|
change TC reversals. Feed a zigzag scalar close stream through
|
||||||
|
three periods and compare every prefix with separate calculators
|
||||||
|
receiving only each complete bucket's final close. Exact trend,
|
||||||
|
readiness, source-coordinate and detection-time agreement proves
|
||||||
|
wall-clock aggregation does not change the TC recurrence.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.test', periods=(1, 2, 5), windows=(2, 2),
|
||||||
|
)
|
||||||
|
period: int
|
||||||
|
engines: dict[int, TrendCalculator] = {
|
||||||
|
period: TrendCalculator((2, 2)) for period in (1, 2, 5)
|
||||||
|
}
|
||||||
|
prices: np.ndarray = 100 + np.sin(np.arange(120) / 3) * 10
|
||||||
|
row: int
|
||||||
|
price: np.float64
|
||||||
|
for row, price in enumerate(prices):
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(
|
||||||
|
float(row), float(price), index=1000 + row,
|
||||||
|
)
|
||||||
|
frame: TimeframeRegime
|
||||||
|
for frame in snapshot.timeframes:
|
||||||
|
period: int = int(frame.period)
|
||||||
|
if (row + 1) % period:
|
||||||
|
continue
|
||||||
|
expected: TrendUpdate = engines[period].push(
|
||||||
|
float(row + 1), float(price), index=1000 + row,
|
||||||
|
)
|
||||||
|
assert frame.trends == expected.trends
|
||||||
|
assert frame.ready == expected.ready
|
||||||
|
assert frame.as_of == row + 1
|
||||||
|
assert frame.recent_closes[-1].index == 1000 + row
|
||||||
|
if len(expected.reversals):
|
||||||
|
actual: tuple[TrendReversal, ...] = (
|
||||||
|
frame.reversals[-len(expected.reversals):]
|
||||||
|
)
|
||||||
|
got: TrendReversal
|
||||||
|
event: np.void
|
||||||
|
for got, event in zip(actual, expected.reversals):
|
||||||
|
assert got.time == event['time']
|
||||||
|
assert got.detected_time == (
|
||||||
|
event['detected_time']
|
||||||
|
)
|
||||||
|
assert got.index == event['index']
|
||||||
|
assert got.direction == event['direction']
|
||||||
|
|
||||||
|
|
||||||
|
def test_partial_startup_bucket_is_never_an_observation() -> None:
|
||||||
|
'''
|
||||||
|
Starting a feed mid-bucket must not give the first slow timeframe
|
||||||
|
a shorter initial observation. Start a five-second stream at t=2
|
||||||
|
and verify its [0, 5) bucket is explicitly discarded while the
|
||||||
|
complete [5, 10) bucket yields its final close at t=10 only.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.test', periods=(5,),
|
||||||
|
)
|
||||||
|
second: int
|
||||||
|
for second in range(2, 10):
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(second, second + 1)
|
||||||
|
frame: TimeframeRegime = snapshot.timeframes[0]
|
||||||
|
if second < 9:
|
||||||
|
assert frame.close_count == 0
|
||||||
|
assert frame.as_of is None
|
||||||
|
else:
|
||||||
|
assert frame.close_count == 1
|
||||||
|
assert frame.as_of == 10
|
||||||
|
assert frame.recent_closes[-1].price == 10
|
||||||
|
assert frame.discarded_buckets == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_gap_resets_every_timeframe_and_rejects_partial_bucket(
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Carrying recurrence state across missing base bars fabricates
|
||||||
|
continuity and can falsely report a ready regime. Warm two
|
||||||
|
timeframes, omit t=9, and resume at t=10. The new source segment
|
||||||
|
must clear all prior evidence, and the interrupted four-second
|
||||||
|
bucket must not become a valid observation or restore readiness.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.test', periods=(1, 4), windows=(2,),
|
||||||
|
)
|
||||||
|
second: int
|
||||||
|
for second in range(9):
|
||||||
|
tracker.push(second, second)
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(10, 20)
|
||||||
|
assert snapshot.segment == 1
|
||||||
|
assert snapshot.timeframes[0].close_count == 1
|
||||||
|
assert snapshot.timeframes[0].ready == (False,)
|
||||||
|
assert snapshot.timeframes[1].close_count == 0
|
||||||
|
assert not snapshot.timeframes[1].reversals
|
||||||
|
snapshot = tracker.push(11, 21)
|
||||||
|
assert snapshot.timeframes[1].close_count == 0
|
||||||
|
for second in range(12, 16):
|
||||||
|
snapshot = tracker.push(second, second)
|
||||||
|
assert snapshot.timeframes[1].close_count == 1
|
||||||
|
assert snapshot.timeframes[1].as_of == 16
|
||||||
|
assert snapshot.timeframes[1].ready == (False,)
|
||||||
|
|
||||||
|
|
||||||
|
def test_subsecond_periods_preserve_epoch_alignment() -> None:
|
||||||
|
'''
|
||||||
|
Integer-second resampling silently truncates high-frequency
|
||||||
|
periods, while float division can select the wrong epoch bucket.
|
||||||
|
Four quarter-second base bars at a modern Unix epoch must form
|
||||||
|
exactly one one-second close, with no early slow observation.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'synthetic.test', base_period=.25, periods=(.25, 1),
|
||||||
|
)
|
||||||
|
offset: int
|
||||||
|
for offset in range(4):
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(
|
||||||
|
1_700_000_000 + offset / 4,
|
||||||
|
100 + offset,
|
||||||
|
)
|
||||||
|
assert snapshot.timeframes[1].close_count == (offset == 3)
|
||||||
|
assert snapshot.as_of == 1_700_000_001
|
||||||
|
assert snapshot.timeframes[1].recent_closes[-1].price == 103
|
||||||
|
|
||||||
|
|
||||||
|
def test_decimal_periods_accept_float_roundoff() -> None:
|
||||||
|
'''
|
||||||
|
A calculated duration such as 0.1 * 3 differs from 0.3 by one
|
||||||
|
float rounding step. Rejecting that representation prevents a
|
||||||
|
valid integer-microsecond bucket configuration. Normalize only
|
||||||
|
float-rounding noise and verify three tenth-second samples form
|
||||||
|
one three-tenth-second observation at the expected endpoint.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', base_period=.1, periods=(.1, .1 * 3),
|
||||||
|
)
|
||||||
|
tracker.push(0, 1)
|
||||||
|
tracker.push(.1, 2)
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(.2, 3)
|
||||||
|
assert snapshot.timeframes[1].period == .3
|
||||||
|
assert snapshot.timeframes[1].as_of == .3
|
||||||
|
assert snapshot.timeframes[1].close_count == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('kwargs', [
|
||||||
|
{'base_period': 0},
|
||||||
|
{'base_period': float('nan')},
|
||||||
|
{'base_period': 0.0000001},
|
||||||
|
{'base_period': 2, 'periods': (1,)},
|
||||||
|
{'periods': (1, 1)},
|
||||||
|
{'periods': ()},
|
||||||
|
{'periods': (1.5,)},
|
||||||
|
{'windows': (1,)},
|
||||||
|
{'history_limit': 0},
|
||||||
|
{'history_limit': 257},
|
||||||
|
{'revision': -1},
|
||||||
|
])
|
||||||
|
def test_invalid_parameters_reject_ambiguous_sampling(
|
||||||
|
kwargs: dict,
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
A timeframe that cannot be built from whole base intervals has
|
||||||
|
no complete-bucket interpretation. Each invalid configuration
|
||||||
|
must fail before constructing a usable stream; bounded history
|
||||||
|
and valid recurrence windows are checked at the same boundary.
|
||||||
|
|
||||||
|
'''
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
MultiTimeframeTrend('synthetic.test', **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('bar', [
|
||||||
|
(0, 100, 1),
|
||||||
|
(-1, 100, 1),
|
||||||
|
(1.25, 100, 1),
|
||||||
|
(1, float('nan'), 1),
|
||||||
|
(1, float('inf'), 1),
|
||||||
|
(float('inf'), 100, 1),
|
||||||
|
(1, 100, 0),
|
||||||
|
(1, 100, 2 ** 63),
|
||||||
|
(1, 100, np.uint64(2 ** 63)),
|
||||||
|
(1, 100, np.bool_(True)),
|
||||||
|
(1e308, 100, 1),
|
||||||
|
])
|
||||||
|
def test_invalid_bar_leaves_all_timeframes_unchanged(
|
||||||
|
bar: tuple[float, float, int|np.integer|np.bool_],
|
||||||
|
) -> None:
|
||||||
|
'''
|
||||||
|
Partial mutation before rejecting a bad bar causes different
|
||||||
|
timeframes to consume different prefixes. Compare a tracker
|
||||||
|
subjected to one invalid update with an untouched reference;
|
||||||
|
accepting the same next bar must produce identical snapshots.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend('test.market')
|
||||||
|
reference: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market',
|
||||||
|
)
|
||||||
|
tracker.push(0, 100)
|
||||||
|
reference.push(0, 100)
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
tracker.push(bar[0], bar[1], index=bar[2])
|
||||||
|
assert tracker.push(1, 101) == reference.push(1, 101)
|
||||||
|
|
||||||
|
|
||||||
|
def test_revision_replay_rebuilds_causal_evidence() -> None:
|
||||||
|
'''
|
||||||
|
Revised history cannot be prepended to a warm TC recurrence.
|
||||||
|
Replaying the identical complete stream after reset must retain
|
||||||
|
every observation and reversal while changing its revision; the
|
||||||
|
frozen snapshot from the earlier generation must stay unchanged.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', periods=(1, 2), windows=(2, 2),
|
||||||
|
)
|
||||||
|
prices: list[int] = [1, 2, 3, 4, 3, 2, 1, 0] * 5
|
||||||
|
second: int
|
||||||
|
price: int
|
||||||
|
for second, price in enumerate(prices):
|
||||||
|
original: RegimeSnapshot = tracker.push(second, price)
|
||||||
|
original_state: str = json.dumps(original.to_state())
|
||||||
|
tracker.reset()
|
||||||
|
for second, price in enumerate(prices):
|
||||||
|
replayed: RegimeSnapshot = tracker.push(second, price)
|
||||||
|
assert replayed == replace(original, revision=1)
|
||||||
|
assert json.dumps(original.to_state()) == original_state
|
||||||
|
frame: TimeframeRegime
|
||||||
|
assert any(frame.reversals for frame in replayed.timeframes)
|
||||||
|
for frame in replayed.timeframes:
|
||||||
|
reversal: TrendReversal
|
||||||
|
for reversal in frame.reversals:
|
||||||
|
assert reversal.time <= reversal.detected_time
|
||||||
|
assert reversal.detected_time <= frame.as_of
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
tracker.reset(revision=1)
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_history_is_bounded_and_finite_json() -> None:
|
||||||
|
'''
|
||||||
|
Sending unbounded market history makes inference cost grow with
|
||||||
|
uptime. A long oscillating stream must retain at most the chosen
|
||||||
|
close/reversal tail per timeframe while preserving total count.
|
||||||
|
Strict JSON encoding also proves no NaN/Infinity enters state.
|
||||||
|
Accepted NumPy integer windows must normalize to native integers
|
||||||
|
so configuration metadata cannot break that same JSON contract.
|
||||||
|
|
||||||
|
'''
|
||||||
|
tracker: MultiTimeframeTrend = MultiTimeframeTrend(
|
||||||
|
'test.market', periods=(1, 2), history_limit=3,
|
||||||
|
windows=(np.int64(2), np.int64(2), np.int64(2)),
|
||||||
|
)
|
||||||
|
second: int
|
||||||
|
for second in range(200):
|
||||||
|
snapshot: RegimeSnapshot = tracker.push(
|
||||||
|
second,
|
||||||
|
float(second % 7),
|
||||||
|
)
|
||||||
|
frame: TimeframeRegime
|
||||||
|
for frame in snapshot.timeframes:
|
||||||
|
assert len(frame.recent_closes) == 3
|
||||||
|
assert len(frame.reversals) <= 3
|
||||||
|
assert frame.close_count == 200 / frame.period
|
||||||
|
decoded: dict = json.loads(json.dumps(
|
||||||
|
snapshot.to_state(), allow_nan=False,
|
||||||
|
))
|
||||||
|
assert decoded['timestamp_convention'] == 'interval_end'
|
||||||
|
assert decoded['algorithm'] == 'trend_calculus'
|
||||||
|
assert decoded['windows'] == [2, 2, 2]
|
||||||
|
|
@ -0,0 +1,149 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Native sampler timestamp ordering required by causal processors.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections import defaultdict
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import trio
|
||||||
|
from trio.testing import MockClock
|
||||||
|
|
||||||
|
from piker.data._sampling import Sampler
|
||||||
|
from piker.fsp._trend import TrendCalculator, TrendHistory
|
||||||
|
|
||||||
|
|
||||||
|
class SampleSource:
|
||||||
|
'''
|
||||||
|
Provide the actual sampler's OHLC and append protocol in memory.
|
||||||
|
|
||||||
|
'''
|
||||||
|
def __init__(self) -> None:
|
||||||
|
'''
|
||||||
|
Seed one closed row and the sampler's future open tail.
|
||||||
|
|
||||||
|
'''
|
||||||
|
self._array: np.ndarray = np.zeros(16, dtype=[
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('open', '<f8'),
|
||||||
|
('high', '<f8'),
|
||||||
|
('low', '<f8'),
|
||||||
|
('close', '<f8'),
|
||||||
|
('volume', '<f8'),
|
||||||
|
])
|
||||||
|
self._array['index'] = np.arange(16)
|
||||||
|
self._array['time'][:2] = [1000, 1001]
|
||||||
|
self._array['close'][:2] = [3, 4]
|
||||||
|
self._write_fields: list[str] = list(
|
||||||
|
self._array.dtype.names[1:],
|
||||||
|
)
|
||||||
|
self.last: int = 2
|
||||||
|
|
||||||
|
@property
|
||||||
|
def array(self) -> np.ndarray:
|
||||||
|
'''
|
||||||
|
Expose only the published prefix, including its open tail.
|
||||||
|
|
||||||
|
'''
|
||||||
|
return self._array[:self.last]
|
||||||
|
|
||||||
|
def push(self, values: np.ndarray) -> None:
|
||||||
|
'''
|
||||||
|
Append OHLC fields while retaining absolute source indices.
|
||||||
|
|
||||||
|
'''
|
||||||
|
end: int = self.last + len(values)
|
||||||
|
self._array[self._write_fields][self.last:end] = values
|
||||||
|
self.last = end
|
||||||
|
|
||||||
|
|
||||||
|
def test_sampler_waits_for_strictly_new_epoch() -> None:
|
||||||
|
'''
|
||||||
|
Empty realtime feeds seed times round(now) and round(now)+1.
|
||||||
|
Their first sampler wake can have the same epoch as that future
|
||||||
|
tail; rounded wall time can also repeat or move backward later.
|
||||||
|
Appending those epochs makes strict causal FSP bootstrap fail.
|
||||||
|
Drive the real sampler through all cases with a controlled clock:
|
||||||
|
keep broadcasting wakeups, append only newer epochs, and verify
|
||||||
|
the complete output remains valid Trend Calculus input.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Advance virtual time without starting brokers or actors.
|
||||||
|
|
||||||
|
'''
|
||||||
|
source: SampleSource = SampleSource()
|
||||||
|
epochs: tuple[int, ...] = (
|
||||||
|
1001, 1001, 1000, 1002, 1002, 1003,
|
||||||
|
)
|
||||||
|
clock_values: Iterator[int] = iter(epochs)
|
||||||
|
counts: list[int] = []
|
||||||
|
complete: trio.Event = trio.Event()
|
||||||
|
|
||||||
|
class ControlledSampler(Sampler):
|
||||||
|
'''
|
||||||
|
Isolate sampler state and capture every step broadcast.
|
||||||
|
|
||||||
|
'''
|
||||||
|
ohlcv_shms: dict = {1: [source]}
|
||||||
|
subscribers: defaultdict = defaultdict(
|
||||||
|
lambda: [0, set()],
|
||||||
|
)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def broadcast(cls, **kwargs: object) -> None:
|
||||||
|
'''
|
||||||
|
Stop after all scheduled clock cases are observed.
|
||||||
|
|
||||||
|
'''
|
||||||
|
counts.append(len(source.array))
|
||||||
|
if len(counts) == len(epochs):
|
||||||
|
complete.set()
|
||||||
|
await trio.sleep_forever()
|
||||||
|
|
||||||
|
fake_time: SimpleNamespace = SimpleNamespace(
|
||||||
|
time=lambda: next(clock_values),
|
||||||
|
)
|
||||||
|
with patch('piker.data._sampling.time', fake_time):
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
scope: trio.CancelScope = await nursery.start(
|
||||||
|
ControlledSampler.increment_ohlc_buffer, 1.,
|
||||||
|
)
|
||||||
|
await complete.wait()
|
||||||
|
scope.cancel()
|
||||||
|
|
||||||
|
assert counts == [2, 2, 2, 3, 3, 4]
|
||||||
|
assert source.array['time'].tolist() == [
|
||||||
|
1000, 1001, 1002, 1003,
|
||||||
|
]
|
||||||
|
result: TrendHistory = TrendCalculator().extend(
|
||||||
|
source.array['close'],
|
||||||
|
source.array['time'],
|
||||||
|
source.array['index'],
|
||||||
|
)
|
||||||
|
assert len(result.trends) == 4
|
||||||
|
|
||||||
|
trio.run(main, clock=MockClock(autojump_threshold=0))
|
||||||
|
|
@ -0,0 +1,279 @@
|
||||||
|
# piker: trading gear for hackers
|
||||||
|
# Copyright (C) 2018-present Tyler Goodlet (in stewardship of pikers)
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||||
|
|
||||||
|
# This program is free software: you can redistribute it and/or
|
||||||
|
# modify it under the terms of the GNU Affero General Public License
|
||||||
|
# as published by the Free Software Foundation, either version 3
|
||||||
|
# of the License, or (at your option) any later version.
|
||||||
|
|
||||||
|
# This program is distributed in the hope that it will be useful,
|
||||||
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||||
|
# GNU Affero General Public License for more details.
|
||||||
|
|
||||||
|
# You should have received a copy of the GNU Affero General Public
|
||||||
|
# License along with this program. If not, see
|
||||||
|
# <https://www.gnu.org/licenses/>.
|
||||||
|
|
||||||
|
'''
|
||||||
|
Exercise native Trend Calculus publication through real Tractor SHM.
|
||||||
|
|
||||||
|
'''
|
||||||
|
from collections.abc import AsyncIterator, Callable
|
||||||
|
from functools import partial
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Any, cast
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import tractor
|
||||||
|
from tractor.ipc._shm import (
|
||||||
|
NDToken,
|
||||||
|
ShmArray,
|
||||||
|
attach_shm_ndarray,
|
||||||
|
open_shm_ndarray,
|
||||||
|
)
|
||||||
|
import trio
|
||||||
|
from trio.testing import wait_all_tasks_blocked
|
||||||
|
|
||||||
|
from piker.fsp._engine import (
|
||||||
|
Cascade,
|
||||||
|
QuoteFrame,
|
||||||
|
_next_step_output,
|
||||||
|
_sample_wakeups,
|
||||||
|
connect_streams,
|
||||||
|
)
|
||||||
|
from piker.fsp._trend import TrendCalculator
|
||||||
|
from piker.fsp._trend_fsp import _history_fields, trend_calculus
|
||||||
|
|
||||||
|
|
||||||
|
async def _read_in_child(
|
||||||
|
token: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
'''
|
||||||
|
Attach from an independent actor using the normal token codec.
|
||||||
|
|
||||||
|
'''
|
||||||
|
shm: ShmArray = attach_shm_ndarray(NDToken.from_msg(token))
|
||||||
|
rows: np.ndarray = shm.array.copy()
|
||||||
|
fields: dict[str, list] = {}
|
||||||
|
name: str
|
||||||
|
for name in rows.dtype.names:
|
||||||
|
value: int|float # noqa: F842
|
||||||
|
fields[name] = [
|
||||||
|
value if np.isfinite(value) else None
|
||||||
|
for value in rows[name].tolist()
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
'first': shm._first.value,
|
||||||
|
'last': shm._last.value,
|
||||||
|
'readonly': not shm.array.flags.writeable,
|
||||||
|
'fields': fields,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_shm_bootstrap_step_and_in_place_replay() -> None:
|
||||||
|
'''
|
||||||
|
Validate actual SHM bounds, field layout and replay visibility.
|
||||||
|
|
||||||
|
Bootstrap the real engine against a nonzero source origin and
|
||||||
|
read its destination through a separate readonly attachment.
|
||||||
|
Advance a quiet bar through the sampler publication protocol,
|
||||||
|
then repair an interior close without changing source bounds.
|
||||||
|
``Cascade.resync()`` must rewrite every affected output and
|
||||||
|
notify its
|
||||||
|
consumer. A child actor then attaches using the transported token
|
||||||
|
and verifies the repaired data independently. All actors and
|
||||||
|
memory names belong to this test; no broker, model API or
|
||||||
|
existing Piker daemon is required. The child reads after replay
|
||||||
|
completes; this proves visibility, not concurrent atomicity.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def main() -> None:
|
||||||
|
'''
|
||||||
|
Own actor lifetimes and SHM names for this replay proof.
|
||||||
|
|
||||||
|
'''
|
||||||
|
suffix: str = uuid4().hex
|
||||||
|
key: str = f'test_tc_{suffix}'
|
||||||
|
async with tractor.open_root_actor(
|
||||||
|
name=key,
|
||||||
|
tpt_bind_addrs=[('127.0.0.1', 0)],
|
||||||
|
):
|
||||||
|
source: ShmArray = open_shm_ndarray(
|
||||||
|
size=128,
|
||||||
|
key=f'{key}_source',
|
||||||
|
append_start_index=37,
|
||||||
|
dtype=np.dtype([
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
('close', '<f8'),
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
name: str # noqa: F842
|
||||||
|
destination: ShmArray = open_shm_ndarray(
|
||||||
|
size=128,
|
||||||
|
key=f'{key}_output',
|
||||||
|
append_start_index=0,
|
||||||
|
dtype=np.dtype([
|
||||||
|
('index', '<i8'),
|
||||||
|
('time', '<f8'),
|
||||||
|
] + [
|
||||||
|
(name, '<f8')
|
||||||
|
for name in trend_calculus.outputs
|
||||||
|
]),
|
||||||
|
)
|
||||||
|
seed: np.ndarray = np.zeros(6, dtype=source.array.dtype)
|
||||||
|
seed['time'] = np.arange(1000., 1006.)
|
||||||
|
seed['close'] = [1., 2., 4., 3., 2., 5.]
|
||||||
|
source.push(seed)
|
||||||
|
reader: ShmArray = attach_shm_ndarray(destination._token)
|
||||||
|
notices: list[dict] = []
|
||||||
|
|
||||||
|
class Sink:
|
||||||
|
'''
|
||||||
|
Record resync notices without involving a live UI
|
||||||
|
stream.
|
||||||
|
|
||||||
|
'''
|
||||||
|
async def send(self, message: dict) -> None:
|
||||||
|
'''
|
||||||
|
Retain each engine notice for deterministic
|
||||||
|
assertions.
|
||||||
|
|
||||||
|
'''
|
||||||
|
notices.append(message)
|
||||||
|
|
||||||
|
def check_publication() -> None:
|
||||||
|
'''
|
||||||
|
Compare the readonly attachment with causal source
|
||||||
|
replay.
|
||||||
|
|
||||||
|
'''
|
||||||
|
expected: dict[str, np.ndarray] = _history_fields(
|
||||||
|
TrendCalculator().extend(
|
||||||
|
source.array['close'], source.array['time'],
|
||||||
|
source.array['index'],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
assert reader._first.value == source._first.value
|
||||||
|
assert reader._last.value == source._last.value
|
||||||
|
assert not reader.array.flags.writeable
|
||||||
|
field: str
|
||||||
|
for field in ('index', 'time'):
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
reader.array[field], source.array[field],
|
||||||
|
)
|
||||||
|
field: str
|
||||||
|
values: np.ndarray
|
||||||
|
for field, values in expected.items():
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
reader.array[field], values,
|
||||||
|
)
|
||||||
|
|
||||||
|
send: trio.MemorySendChannel[QuoteFrame]
|
||||||
|
quotes: trio.MemoryReceiveChannel[QuoteFrame]
|
||||||
|
send, quotes = trio.open_memory_channel[QuoteFrame](0)
|
||||||
|
market: SimpleNamespace = SimpleNamespace(
|
||||||
|
fqme='synthetic.test',
|
||||||
|
)
|
||||||
|
wakeups: AsyncIterator[QuoteFrame]
|
||||||
|
notify: Callable[[], None]|None
|
||||||
|
async with send, quotes, _sample_wakeups(
|
||||||
|
quotes, market.fqme, True,
|
||||||
|
) as (wakeups, notify):
|
||||||
|
nursery: trio.Nursery
|
||||||
|
async with trio.open_nursery() as nursery:
|
||||||
|
casc: Cascade = Cascade(
|
||||||
|
src=SimpleNamespace(
|
||||||
|
rt_shm=source,
|
||||||
|
mkt=market,
|
||||||
|
),
|
||||||
|
dst=SimpleNamespace(rt_shm=destination),
|
||||||
|
tn=nursery,
|
||||||
|
fsp=trend_calculus,
|
||||||
|
client_stream=cast(
|
||||||
|
tractor.MsgStream, Sink(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
compute: Callable = partial(
|
||||||
|
connect_streams,
|
||||||
|
casc=casc,
|
||||||
|
mkt=market,
|
||||||
|
quote_stream=wakeups,
|
||||||
|
src=casc.src,
|
||||||
|
dst=casc.dst,
|
||||||
|
edge_func=trend_calculus.func,
|
||||||
|
)
|
||||||
|
async with casc.open_edge(compute):
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
check_publication()
|
||||||
|
assert reader.array['top_1'][-1] == 4.
|
||||||
|
assert reader.array['bottom_1'][-1] == 2.
|
||||||
|
|
||||||
|
step: np.ndarray = source.array[-1:].copy()
|
||||||
|
step['time'] += 1
|
||||||
|
source.push(step)
|
||||||
|
out: np.ndarray = _next_step_output(
|
||||||
|
destination.array[-1:],
|
||||||
|
trend_calculus,
|
||||||
|
False,
|
||||||
|
)
|
||||||
|
out['time'] = step['time']
|
||||||
|
destination.push(out)
|
||||||
|
if casc.sample_ready is not None:
|
||||||
|
casc.sample_ready.set()
|
||||||
|
casc.sample_ready = trio.Event()
|
||||||
|
assert notify is not None
|
||||||
|
notify()
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
check_publication()
|
||||||
|
|
||||||
|
source.array['close'][2] = 100.
|
||||||
|
await casc.resync()
|
||||||
|
await wait_all_tasks_blocked()
|
||||||
|
check_publication()
|
||||||
|
assert len(notices) == 1
|
||||||
|
info: dict = notices[0]['fsp_update']
|
||||||
|
assert info['first'] == 37
|
||||||
|
assert info['last'] == 44
|
||||||
|
|
||||||
|
actors: tractor.ActorNursery
|
||||||
|
async with tractor.open_nursery() as actors:
|
||||||
|
portal: tractor.Portal = (
|
||||||
|
await actors.start_actor(
|
||||||
|
'trend-shm-reader',
|
||||||
|
enable_modules=[__name__],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
token: dict[str, Any] = (
|
||||||
|
destination._token.as_msg()
|
||||||
|
)
|
||||||
|
remote: dict[str, Any] = (
|
||||||
|
await portal.run(
|
||||||
|
_read_in_child,
|
||||||
|
token=token,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert remote['first'] == 37
|
||||||
|
assert remote['last'] == 44
|
||||||
|
assert remote['readonly']
|
||||||
|
fields: tuple[str, ...] = (
|
||||||
|
reader.array.dtype.names
|
||||||
|
)
|
||||||
|
field: str
|
||||||
|
for field in fields:
|
||||||
|
np.testing.assert_array_equal(
|
||||||
|
np.asarray(
|
||||||
|
remote['fields'][field],
|
||||||
|
dtype=float,
|
||||||
|
),
|
||||||
|
reader.array[field],
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
await actors.cancel()
|
||||||
|
assert casc.complete.is_set()
|
||||||
|
|
||||||
|
trio.run(main)
|
||||||
Loading…
Reference in New Issue