Compare commits
46 Commits
main
...
wkt/land_b
| Author | SHA1 | Date |
|---|---|---|
|
|
3e3e0fdd46 | |
|
|
b349332e6b | |
|
|
5183d4094f | |
|
|
251987fc4b | |
|
|
63c763800a | |
|
|
7a06a620e2 | |
|
|
3053324f04 | |
|
|
cb5e81d443 | |
|
|
51795a4e07 | |
|
|
495c091546 | |
|
|
2a826fba17 | |
|
|
6575db6463 | |
|
|
3ec447a45a | |
|
|
11aa2eeb1e | |
|
|
9dde7a9867 | |
|
|
02bb193f87 | |
|
|
6f626a9c45 | |
|
|
72d0bdad5e | |
|
|
dc618229bd | |
|
|
635314c757 | |
|
|
d1571ddfc7 | |
|
|
7bef50b0a5 | |
|
|
e8e0730a50 | |
|
|
4133f07340 | |
|
|
507c75c669 | |
|
|
1fa45dfa98 | |
|
|
bde2457bdd | |
|
|
1ba9bdbac0 | |
|
|
d86f29d416 | |
|
|
69a6edd6f8 | |
|
|
bbe9deaa62 | |
|
|
70e8d60326 | |
|
|
2c007d3ada | |
|
|
1fde12d3fd | |
|
|
c347ab08ea | |
|
|
c5155ce814 | |
|
|
d7cefa71f3 | |
|
|
20bfbc2380 | |
|
|
d7f956a566 | |
|
|
d26286f6cc | |
|
|
22f166a5c6 | |
|
|
6c8709d309 | |
|
|
23632d2153 | |
|
|
d85e57334c | |
|
|
6c8d17da75 | |
|
|
1976f6a5de |
|
|
@ -0,0 +1,233 @@
|
|||
# Bounded IB recovery candidate
|
||||
|
||||
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
|
||||
`/home/goodboy/repos/piker`. Base:
|
||||
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
|
||||
Changes are unstaged and uncommitted. No acceptance is implied.
|
||||
|
||||
## Patch selection and provenance
|
||||
|
||||
Keep these existing landing commits:
|
||||
|
||||
| Commit | Reason |
|
||||
| --- | --- |
|
||||
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
|
||||
| `e8e0730a` | Correlate MethodProxy requests and responses |
|
||||
| `7bef50b0` | One response reader, per-request waiters |
|
||||
| `d1571ddf` | Propagate daemon-handler exceptions |
|
||||
| `dc618229` | Isolate cached feed stream consumers |
|
||||
|
||||
Proposed separate implementation boundaries, in this order:
|
||||
|
||||
1. Back out the code and original test from
|
||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
|
||||
This restores the direct-owner `_quote_streams` implementation.
|
||||
Retain the historical prompt records. Replace the removed
|
||||
mocked-cache regression with the real-bridge restart reproducer.
|
||||
This backout is not a claim that arbitrary concurrent direct
|
||||
callers have safe broadcast ownership.
|
||||
2. Backport **only** `piker/brokers/ib/feed.py` from
|
||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
|
||||
Retain its seven history-adapter regressions in
|
||||
`tests/test_ib_history.py`. This includes reset serialization,
|
||||
completion/result signalling and bounded cancellation/failure.
|
||||
It does not include the original SHM/storage orchestration patch.
|
||||
3. Backport **only** `piker/brokers/ib/feed.py` from
|
||||
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
|
||||
Retain `test_empty_history_preserves_request_boundary` and its
|
||||
six cases. This preserves no-data request boundaries and wakes
|
||||
the waiter. It does not include the original history-layer
|
||||
endpoint/filtering changes. Add the real-bridge history recovery
|
||||
case after these adapter changes are present.
|
||||
4. Import the code and original regression from
|
||||
`12b0d66e854247e92cb5a8668be71d816c72d856`.
|
||||
Add a separate real-broadcaster/production-loop regression.
|
||||
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
|
||||
This is a local UX cleanup retaining the existing `None` result.
|
||||
It does not backport the null-grouping/deduplication rewrite in
|
||||
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
|
||||
6. Record cross-boundary regression results and live qualification
|
||||
evidence, distinguishing the known callback failure from passes.
|
||||
|
||||
All three imported source commits were authored by
|
||||
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
|
||||
and the full source SHA when committing each partial backport.
|
||||
Use explicit partial-backport wording, not a whole-commit cherry
|
||||
claim. New real-runtime tests are this session's additional work.
|
||||
|
||||
The current aggregate working diff contains these boundaries in
|
||||
the same feed file. Staging that entire file would combine the
|
||||
first three boundaries. Reconstruct/stage the original per-source
|
||||
patches in the above order when preparing commits.
|
||||
|
||||
## Quote-cache history correction
|
||||
|
||||
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
|
||||
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
|
||||
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
|
||||
They are equivalent replays, not different implementations.
|
||||
|
||||
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
|
||||
change and retains direct ownership. The original qualification
|
||||
did not validate the combination assembled for the landing.
|
||||
|
||||
After this candidate's edits, `piker/brokers/ib/feed.py` matches
|
||||
the source at `1da9b760` byte-for-byte. Compared with the evaluated
|
||||
tip, only the later schedule/reservation APIs are absent.
|
||||
The EMS code patch has the same stable patch ID as the original
|
||||
`12b0d66e` EMS code patch:
|
||||
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
|
||||
|
||||
## Real-runtime evidence
|
||||
|
||||
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
|
||||
an asyncio-infected child, the installed pinned Tractor bridge,
|
||||
production Piker quote setup, and real IB/eventkit ticker events.
|
||||
Only the external IB connection/input is replaced with replay data.
|
||||
|
||||
Before the quote-cache backout, cancellation after ticker delivery
|
||||
raised `RemoteActorError[TrioTaskExited]` through
|
||||
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
|
||||
reported fatal path.
|
||||
|
||||
After the backout:
|
||||
|
||||
- Two quote generations finish; each subscription is cancelled,
|
||||
each asyncio producer is done and each ticker handler removed.
|
||||
- A second RPC repeats the sequence in the same actor successfully.
|
||||
- The strict callback-cleanliness case still fails: Tractor's
|
||||
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
|
||||
exception handler even though the Trio consumer recovers.
|
||||
|
||||
The history integration case keeps `Client.bars()`, MethodProxy,
|
||||
both relays, `get_bars()`, `wait_on_data_reset()` and
|
||||
`data_reset_hack()` real. It replaces external contract discovery,
|
||||
IB history replies and VNC input. A real IB error event announces
|
||||
farm recovery. The replay rejects retries before recovery and
|
||||
proves one reset followed by completion for both 1s and 60s data.
|
||||
|
||||
The new EMS case runs the production dark-clearing loop over a
|
||||
real Tractor broadcaster: both EMS and a sibling receive a tick,
|
||||
EMS cancellation removes only its subscription, and the sibling
|
||||
continues with an unchanged strict lag policy.
|
||||
|
||||
## Verification
|
||||
|
||||
Unique selected cases: **38 passed, 1 failed**.
|
||||
The failure is intentionally visible, not skipped or xfailed.
|
||||
|
||||
Run from the checkout's working Nix environment:
|
||||
|
||||
```xsh
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
|
||||
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The adapter/proxy/feed/EMS selection initially passed 21 cases;
|
||||
the added real EMS case then passed with its imported case.
|
||||
The bridge file passed two cases and failed callback cleanliness.
|
||||
Logging/startup passed 13 cases; runtime boot passed one with the
|
||||
existing `Registrar.uid` deprecation warning.
|
||||
|
||||
The agent shell inherited another project's environment and lacked
|
||||
native libraries. Tests used the existing Piker `py313` via
|
||||
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
|
||||
libraries from the pinned flake's existing Qt dependency closure,
|
||||
excluding Nix Qt itself and glibc. No environment sync or shellHook
|
||||
ran. Piker resolved to this checkout; Tractor resolved to its
|
||||
installed package in `py313`.
|
||||
|
||||
Workflow-state preparation initially reported changed legacy
|
||||
files/modes after migration. On 2026-10-23 the human authorized
|
||||
reconciliation. Inspection found only two missing legacy guidance
|
||||
snapshots; the neutral copies matched their saved hashes and modes.
|
||||
Restoring those local snapshots resolved preparation without
|
||||
changing archived messages or the original migration receipt.
|
||||
|
||||
## Human live qualification, 2026-10-23
|
||||
|
||||
The human ran these journeys with the installed Tractor pin,
|
||||
the existing IB Gateway and pre-existing NativeDB. The agent did
|
||||
not operate the broker connection, VNC, orders or database.
|
||||
|
||||
- Initial QQQ startup/backfill recovered after pacing error 162
|
||||
with one reset attempt, then displayed live updates. The reset
|
||||
waiter appears to have timed out; the log did not establish an
|
||||
acknowledged farm reset or live quote-stream restart.
|
||||
- Sequential chart exits and reattachment preserved all four
|
||||
daemon PIDs; a read-only process/listener check confirmed this.
|
||||
- The overlapping two-chart journey used `pikerd@961108`,
|
||||
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
|
||||
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
|
||||
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
|
||||
then C closed at 08:18:37. The human reported the exercise worked;
|
||||
survivor updates are a manual observation, while timestamps and
|
||||
unchanged daemon identities corroborate overlap and service reuse.
|
||||
- Startup in that journey hit the explicit single-null-sample
|
||||
breakpoint several times before the charts overlapped. The small
|
||||
debugger cleanup above was applied afterward; this live run does
|
||||
not itself verify that later edit.
|
||||
- Departing charts still report cancellation groups and an FSP
|
||||
receiver-close `ClosedResourceError`, followed by the Qt close
|
||||
handler's self-SIGINT. Peer-disconnected final result warnings
|
||||
remain contained in the persistent services.
|
||||
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
|
||||
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
|
||||
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
|
||||
After the human continued, all three children were reported
|
||||
exited with status 0 and the root terminated at 08:19:45.
|
||||
Cancellation-only RPC error reporting and late cancel-ack
|
||||
diagnostics also remain.
|
||||
|
||||
The human explicitly accepted deferring this daemon teardown fix
|
||||
from #94 and requested loud attribution for the bounded backports.
|
||||
This is an operational smoke result, not clean-shutdown coverage.
|
||||
|
||||
## Downstream reconciliation requirements
|
||||
|
||||
Before landing #94, the human wants an easy replay of the downstream
|
||||
histories intended to follow it. Rehearse against the agreed #94
|
||||
tree in disposable worktrees, then inspect range-diffs, final code
|
||||
and relevant regressions. This note does not authorize rewriting
|
||||
the original branches or changing acceptance markers.
|
||||
|
||||
- `0846cbd4` and `1da9b760` are partial backports. Their imported
|
||||
adapter hunks may already be present, but their history/SHM/storage
|
||||
changes and omitted integration tests remain necessary downstream.
|
||||
Never drop either whole source commit merely because `feed.py`
|
||||
matches or its adapter hunk becomes empty.
|
||||
- `12b0d66e` has source-equivalent EMS code and its imported test,
|
||||
plus an additional local real-broadcaster test. Preserve that extra
|
||||
regression when resolving test-file overlap.
|
||||
- The quote-cache backout intentionally restores direct ownership
|
||||
for this landing. Reintroducing original `4af885d9` downstream
|
||||
requires qualifying cached-owner teardown with the matching
|
||||
Tractor revision; the backout is not a permanent ownership design.
|
||||
- The local single-null breakpoint removal retains legacy behavior.
|
||||
Prefer the full downstream `976db6c3` algorithm when it is replayed;
|
||||
preserving this `return None` guard would lose that later fix.
|
||||
- Real-bridge tests and manual qualification notes are local work,
|
||||
not evidence that omitted portions of the source commits landed.
|
||||
|
||||
Commit messages must name full source SHAs, original authorship,
|
||||
imported paths/tests, omissions, local additions and replay guidance.
|
||||
Use explicit partial-backport language for adapter subsets; do not
|
||||
claim a full cherry-pick or reuse the source's broader fix claims.
|
||||
|
||||
## Remaining decision
|
||||
|
||||
This candidate removes the reproduced fatal quote-restart
|
||||
propagation and imports existing reset fixes. The human exercised
|
||||
live QQQ startup, backfill and concurrent chart consumer lifetimes.
|
||||
Completion-callback and teardown errors remain; the human accepts
|
||||
them as follow-ups outside the bounded #94 landing.
|
||||
The storage/SHM portions of the downstream fixes are not covered by
|
||||
the adapter backport or its integration tests.
|
||||
|
||||
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
|
||||
`1a2d4893` stays a separate follow-up. Further Tractor teardown
|
||||
changes or storage imports require reassessing the bounded landing
|
||||
rather than silently expanding this candidate.
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# IB recovery qualification, 2026-11-06
|
||||
|
||||
This records human-operated qualification of the bounded #94
|
||||
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
|
||||
the narrow history catch described below. It supplements the
|
||||
historical evidence in `ai/ib-recovery-candidate.md`.
|
||||
|
||||
## Runtime and dependency source
|
||||
|
||||
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
|
||||
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
|
||||
local checkout during evaluation. The human subsequently published
|
||||
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
|
||||
remote query confirmed the published OID before restoring Piker's
|
||||
Git source and recording the same OID in `uv.lock`.
|
||||
|
||||
Offline lock validation resolves 126 packages successfully. No
|
||||
environment sync was performed. This validates the lock metadata;
|
||||
it does not claim a fresh installation from the published source.
|
||||
|
||||
## Manual observations
|
||||
|
||||
The human operated Gateway, chart windows and paper orders. The
|
||||
agent did not operate live services or the account.
|
||||
|
||||
- Same-feed overlap, close/reopen and paper order submit/cancel
|
||||
worked after the existing order-dialog guard was imported.
|
||||
- Different-feed overlap originally exposed a timed-out one-second
|
||||
MNQ March 2027 gap query. PDB showed `timedout=True` and
|
||||
`out is None`. Its uncaught `NoData` canceled sibling history and
|
||||
quote tasks even though `pikerd` remained alive.
|
||||
- After the narrow catch, the human reported the overlap journeys
|
||||
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
|
||||
live feeds survived the Gateway data reset; closing all charts
|
||||
and reopening with `pikerd` left running restored live updates.
|
||||
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
|
||||
that the intended Gateway data reset was triggered.
|
||||
|
||||
These are manual observations, not automated timing or latency
|
||||
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
|
||||
labels it a connection reset.
|
||||
|
||||
## Narrow history backport
|
||||
|
||||
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
|
||||
`maybe_fill_null_segments()` is copied from
|
||||
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
|
||||
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
|
||||
remains in place; this does not import broader history repair.
|
||||
|
||||
The new regression uses real null detection and checks that a
|
||||
sibling in the same nursery still runs after either provider
|
||||
exception. Both cases failed before the catch. With it, the two
|
||||
focused history files passed: 17 tests. Earlier bounded-recovery
|
||||
and Tractor qualification results remain separate evidence, not
|
||||
fresh test runs of the final dependency commit tree.
|
||||
|
||||
## Remaining recovery limits
|
||||
|
||||
A full Gateway restart was triggered accidentally through its
|
||||
auto-restart settings dialog. The disconnect delivered the string
|
||||
`Peer closed connection.` to `deliver_trade_events()`. The string
|
||||
branch then reached `err['reqid']`, raising `TypeError`; its `code`
|
||||
local can also retain an earlier event's value. Full Gateway restart
|
||||
recovery is outside this bounded landing.
|
||||
|
||||
The earlier history-failure teardown also exposed a separate
|
||||
Tractor completion-callback assertion after `BrokenResourceError`.
|
||||
The history catch does not establish that this callback race is
|
||||
fixed. Broader history recovery and these disconnect paths remain
|
||||
downstream work.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
created_at: 2026-10-07T16:54:53Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: gpt-6.1-sol
|
||||
git_ref: bde2457b
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested fixing all seven local PR94 review findings and
|
||||
exporting the review for human inspection in Tuicr. They asked that a
|
||||
Tractor pin adjustment be considered before adding Piker workarounds.
|
||||
|
||||
## Response summary
|
||||
|
||||
Correct the endpoint defaults, Click argument shape, daemon binding and
|
||||
registrar option handling, chart fallback, and stray debugger pause.
|
||||
Keep the absent datad implementation's test outside this landing
|
||||
branch. Add offline startup coverage and export original findings
|
||||
without modifying review states. The checked newer Tractor address
|
||||
branch still rejects mixed transports, so leave the dependency pin.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `config/conf.toml`: TCP-only default endpoints.
|
||||
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
|
||||
- `piker/service/_actor_runtime.py`: validate daemon listeners.
|
||||
- `piker/service/_daemon.py`: initialize and consume registrar options.
|
||||
- `piker/ui/cli.py`: fallback to automatic chart binding.
|
||||
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
|
||||
- `tests/test_services.py`: defer test requiring missing datad API.
|
||||
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human identified that datad tests arrived through the agent's
|
||||
mixed cherry-picks and explicitly requested deferring that test. They
|
||||
requested a chart default for existing configs, expanded the task to
|
||||
all review findings, and directed checking whether a newer Tractor pin
|
||||
could reduce the fix scope. No direct manual source edits were observed
|
||||
during these changes; acceptance remains with the human.
|
||||
|
||||
During the human review pass, the human confirmed removing the Binance
|
||||
pause and correcting CLI endpoint parsing. They requested an explicit
|
||||
template note about this pin's single-transport limitation and detailed
|
||||
comments distinguishing registrar discovery endpoints from daemon bind
|
||||
endpoints. Those explanations were added at the human's direction.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
created_at: 2026-10-07T16:54:53Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: gpt-6.1-sol
|
||||
git_ref: bde2457b
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Fix all findings from the local PR94 review, prefer a Tractor pin
|
||||
change if it reduces work, and export the original review for Tuicr.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- config/conf.toml`
|
||||
|
||||
Use TCP-only defaults, retaining commented UDS alternatives. Both
|
||||
checked Tractor address branch tips still reject mixed transports.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
|
||||
|
||||
Make Click multiaddresses repeatable and normalize single strings.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
|
||||
|
||||
Validate requested daemon bind addresses instead of remote registrars.
|
||||
Normalize accepted endpoints and allow TCP ephemeral port requests.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
|
||||
|
||||
Resolve explicit or inherited registrars independently of other daemon
|
||||
options and consume the explicit keyword before forwarding kwargs.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
|
||||
|
||||
Default chart bind endpoints to None when network config is absent.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Remove the unconditional market-info debugger pause.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
|
||||
|
||||
Remove the prematurely cherry-picked datad test and add offline startup
|
||||
regressions, including a real registrar/child daemon lifecycle check.
|
||||
|
||||
## Verification
|
||||
|
||||
- Startup and logging tests: 13 passed.
|
||||
- Existing runtime boot and tagged registrar tests: 2 passed.
|
||||
- Scoped native Ruff and diff checks pass.
|
||||
- The first new actor test used a removed nursery API; it was corrected
|
||||
to the installed start_actor/Portal.run interface before passing.
|
||||
- Import initially failed without native libraries. Tests use the
|
||||
existing flake's GCC, zlib, Qt and PyQt paths without executing its
|
||||
environment-rebuilding shell hook.
|
||||
- Original seven findings exported locally as Tuicr review-add payloads
|
||||
bound to bde2457b; no session comments or remote PR were changed.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
created_at: 2026-10-10T23:28:52Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 1fa45dfa
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human rejected keeping the April Tractor pin as a landing baseline.
|
||||
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
|
||||
new rollback adaptations, reuse of original Piker histories where
|
||||
possible, and a candidate prepared within their gateway relocation
|
||||
window. They asked specifically whether the runtime validation review
|
||||
hunk duplicated older endpoint history.
|
||||
|
||||
## Response summary
|
||||
|
||||
Select the reviewed address branch at an immutable locked revision and
|
||||
preserve all committed application/test compatibility code. Verify the
|
||||
minimal dependency boundary offline and preview existing consumer
|
||||
fixes without importing unrelated datad/backfill changes. Distinguish
|
||||
original-source test results from live teardown acceptance, which is
|
||||
not available in this preparation window.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
|
||||
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
|
||||
preserving Polars 1.36.0 and the rest of the locked graph.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human redirected the task from an older known-good runtime to a
|
||||
modern landing stack and insisted on reusing original history rather
|
||||
than building new compatibility adapters. They proposed stricter
|
||||
exception-group behavior as a hypothesis and set the live-test/time
|
||||
constraints. Those directions materially changed the dependency
|
||||
boundary. No direct manual edits to this final boundary were observed;
|
||||
runtime acceptance and landing remain human-owned.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
created_at: 2026-10-10T23:28:52Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 1fa45dfa
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Undo the April Tractor rollback and select the reviewed
|
||||
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
|
||||
commits to new adaptations, and prepare an offline-verified candidate
|
||||
while live IB verification is deferred for gateway relocation.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
|
||||
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
|
||||
and test sources byte-for-byte to the committed PR94 tree. Keep the
|
||||
rejected rollback journal only in ignored local experiment reports.
|
||||
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
|
||||
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
|
||||
the resulting graph. Actual imports confirm the reviewed Tractor pin
|
||||
and Trio 0.32.0, not the hypothesized Trio 0.33.
|
||||
|
||||
## Original-history previews
|
||||
|
||||
The existing MethodProxy stack is retained. Read-only merge previews
|
||||
show original `4af885d9` and `b691422b` can be applied without the datad
|
||||
cutover. Original-source regressions run through temporary in-memory
|
||||
overlays against the real pinned runtime: seven cases pass and one
|
||||
cache-hit case fails because `log.at_least_level('info')` looks up INFO
|
||||
in a custom-level mapping. A separate standard logging correction
|
||||
produces eight passing cases. Its patch and execution handoff are saved
|
||||
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
|
||||
cherry-picks or source edits are performed by this boundary.
|
||||
|
||||
## Verification
|
||||
|
||||
- Startup and logging: 13 passed.
|
||||
- Runtime boot and tagged registrar normalization: 2 passed.
|
||||
- Locked sync and source whitespace checks pass.
|
||||
- Original application/test trees are unchanged from PR94 HEAD.
|
||||
- Consumer overlay with explicit logging correction: 8 passed.
|
||||
- Correction patch validates against its exact preview tree using a
|
||||
private temporary index; the real index is untouched.
|
||||
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
User reported a different-feed overlap failure and asked whether
|
||||
a fix could be extracted without importing broader history work.
|
||||
PDB confirmed MNQ March 2027 one-second history timed out and
|
||||
returned None, producing NoData. After continuing PDB, sibling
|
||||
history and quote tasks terminated while pikerd remained alive.
|
||||
User requested continuing with the narrow backport.
|
||||
|
||||
## Response summary
|
||||
|
||||
Partial backport of the NoData/DataUnavailable catch from
|
||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||
A new parametrized regression uses real null detection and verifies
|
||||
that a sibling in the same nursery survives the failed repair.
|
||||
|
||||
Both new cases failed before the catch. The focused history suites
|
||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||
IB reset churn or the separate Tractor completion-callback assertion.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_history.py`: existing downstream exception catch.
|
||||
- `tests/test_history_backfill.py`: sibling survival regression.
|
||||
|
||||
## Human edits
|
||||
|
||||
User supplied live reproduction, PDB locals and cancellation logs,
|
||||
and explicitly constrained scope to a small history extraction.
|
||||
The production catch is copied from existing downstream commit
|
||||
0846cbd4, not newly designed here. New regression is agent-authored;
|
||||
no direct human source edits observed during this patch. Review and
|
||||
live qualification remain with the user.
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
---
|
||||
created_at: 2026-11-06T18:56:28.621577+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||
|
||||
Partial backport of the NoData/DataUnavailable catch from
|
||||
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
|
||||
maybe_fill_null_segments(). The existing gap fallback remains in use.
|
||||
A new parametrized regression uses real null detection and verifies
|
||||
that a sibling in the same nursery survives the failed repair.
|
||||
|
||||
Both new cases failed before the catch. The focused history suites
|
||||
passed after it: 17 passed. This does not qualify live overlap or fix
|
||||
IB reset churn or the separate Tractor completion-callback assertion.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human requested a commit plan, then a fast-forward landing of
|
||||
the #94 branch and PR updates. They agreed the temporary local
|
||||
Tractor source must be replaced, published piker_pin themselves,
|
||||
and reported "pushed it". They also supplied the manual recovery
|
||||
results and full Gateway restart failure diagnostics.
|
||||
|
||||
## Response summary
|
||||
|
||||
Restore the Git Tractor source after local qualification and lock
|
||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||
Record the human's overlap, data-reset and reopen observations,
|
||||
the narrow history extraction, and full-restart recovery limits.
|
||||
Offline uv lock validation passed without an environment sync.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
|
||||
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
|
||||
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human selected the bounded landing scope, performed live tests,
|
||||
identified the more aggressive reset distinction, and published
|
||||
the tested Tractor ref. The agent applied the metadata edits and
|
||||
wrote the qualification record from those observations. No human
|
||||
acceptance or task-state transitions are inferred by this record.
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
---
|
||||
created_at: 2026-11-06T20:17:46.622316+00:00
|
||||
generated_by:
|
||||
harness: codex
|
||||
provider: OpenAI
|
||||
git_ref: 495c0915
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- uv.lock`
|
||||
|
||||
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
|
||||
|
||||
Restore the Git Tractor source after local qualification and lock
|
||||
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
|
||||
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
|
||||
Record the human's overlap, data-reset and reopen observations,
|
||||
the narrow history extraction, and full-restart recovery limits.
|
||||
Offline uv lock validation passed without an environment sync.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# AI Prompt I/O Log
|
||||
|
||||
This directory records substantive AI-assisted changes, prompts,
|
||||
original harness identity, and unedited outputs. Human contributors
|
||||
remain accountable for code decisions.
|
||||
|
||||
Records follow the [NLNet generative AI policy][policy]. Each
|
||||
structured entry links its immutable `.raw.md` partner. Code output
|
||||
uses Git diff references to avoid duplication.
|
||||
|
||||
[policy]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:08:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Session-initiating instruction (driving all 7 commits in
|
||||
this series):
|
||||
|
||||
> ok i want you to become the distributed runtime and
|
||||
> concurrency expert for this project - namely acquire
|
||||
> a deep understanding of tractor and how it's used.
|
||||
> then i want you to attempt to factor our current
|
||||
> brokerd service daemon into 2 daemons: a brokerd
|
||||
> which only hosts live/paper trading endpoint tasks
|
||||
> [..] a new `datad` subdaemon which in a separate
|
||||
> subactor serves all the data feed service tasks [..]
|
||||
> give me a mega detailed plan on how to approach this,
|
||||
> and a staged approach for the implementation.
|
||||
|
||||
Proximate driver for THIS commit: during the approved
|
||||
plan's stage-0 test gate the agent discovered test
|
||||
subactors were writing into the user's REAL
|
||||
`~/.config/piker/accounting/` files (a bogus paper fill
|
||||
landed in `account.kraken.paper.toml`) because the old
|
||||
test-dir override in `config.get_app_dir()` was
|
||||
commented out and could never work in spawned
|
||||
subactors anyway. Fix applied autonomously under the
|
||||
approved plan; no explicit per-fix user prompt.
|
||||
|
||||
## Response summary
|
||||
|
||||
Restore `pytest` config-dir isolation by resolving the
|
||||
per-test tmp dir lazily (at conf-path access time) from
|
||||
`tractor` runtime-vars, which propagate down the actor
|
||||
tree; route all conf-path derivation through
|
||||
`config.get_conf_dir()` so the override is effective in
|
||||
every (sub)actor.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/config.py` — add `_maybe_use_test_dir()`;
|
||||
hook in `get_conf_dir()`; route `get_conf_path()` +
|
||||
`load()` mkdir through it
|
||||
- `piker/accounting/_ledger.py` — derive ledger dir
|
||||
via `config.get_conf_dir()` (not the global)
|
||||
- `piker/accounting/_pos.py` — same for account files
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:08:59Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/config.py`
|
||||
|
||||
Generated: `config._maybe_use_test_dir()` — lazily
|
||||
reads `piker_test_dir` from
|
||||
`tractor.runtime._state._runtime_vars['piker_vars']`
|
||||
(pre-loaded by `open_piker_runtime()` from the
|
||||
`tests.conftest._open_test_pikerd()` overrides) and
|
||||
calls `_override_config_dir()` when set. Hooked at the
|
||||
top of `get_conf_dir()`; `get_conf_path()` and
|
||||
`load()`'s dir-creation rerouted through
|
||||
`get_conf_dir()`.
|
||||
|
||||
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
|
||||
> `git log -1 -p --follow -- piker/accounting/_pos.py`
|
||||
|
||||
Generated: ledger/account dir derivation switched from
|
||||
the `config._config_dir` module global to
|
||||
`config.get_conf_dir()` + `mkdir(parents=True,
|
||||
exist_ok=True)` for nested tmp-dir creation.
|
||||
|
||||
Key diagnostic reasoning (verbatim from session):
|
||||
|
||||
The old (commented-out) `get_app_dir()` override gated
|
||||
on `'pytest' in sys.modules` which can NEVER work in
|
||||
spawned subactors (fresh procs, no pytest import); as
|
||||
a result test `paperboi`/daemon actors were writing
|
||||
into the user's REAL `~/.config/piker/accounting/`
|
||||
files. Evidence: `account.kraken.paper.toml` gained a
|
||||
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
|
||||
(a test fill), and `trades_kraken_paper.toml` (252B)
|
||||
contained only that fill. The resolution must be
|
||||
checked lazily at config-path access time (NOT import
|
||||
time) since sub-actors only receive runtime-vars once
|
||||
their `tractor` runtime has fully booted.
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
|
||||
timestamp: 2026-06-10T17:10:22Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Same session-initiating `brokerd`-split instruction (see
|
||||
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
|
||||
driver: the approved plan's per-stage test gates could
|
||||
not run AT ALL — the branch base was broken vs.
|
||||
`tractor` git `main` (`AttributeError: 'tuple' object
|
||||
has no attribute 'unwrap'` at `pikerd` boot, stale
|
||||
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
|
||||
fixed forward autonomously to (re)establish the gate
|
||||
baseline, continuing the `repair_tests` branch lineage
|
||||
(verified already merged in ancestry via `git cherry`
|
||||
during a user-requested branch-overlap survey).
|
||||
|
||||
## Response summary
|
||||
|
||||
Port the service layer + test suites to current
|
||||
`tractor` APIs: addr-type normalization in
|
||||
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
|
||||
kwarg renames, modern self-cancel absorption semantics
|
||||
in the cancel-method test harness, exc/position attr
|
||||
renames, a paper-EMS startup-budget bump and a syntax
|
||||
fix in `.deribit.api`.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/service/_actor_runtime.py` — `wrap_address()`
|
||||
normalize before `.unwrap()` in `open_pikerd()`
|
||||
- `piker/service/_registry.py` — `check_for_service()`
|
||||
-> `query_actor(regaddr=)` + 2-tuple yield +
|
||||
`open_registry(addrs=)`
|
||||
- `piker/brokers/deribit/api.py` — missing comma in
|
||||
`tractor.trionics` import tuple
|
||||
- `tests/test_services.py` — `registry_addr=` kwarg,
|
||||
raddr unwraps, cancel-semantics harness rewrite,
|
||||
`fail_after` 9 -> 19s
|
||||
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
|
||||
|
||||
## Human edits
|
||||
|
||||
None — committed as generated.
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
---
|
||||
model: claude-fable-5[1m]
|
||||
service: claude
|
||||
timestamp: 2026-06-10T17:10:22Z
|
||||
git_ref: datad_service
|
||||
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
|
||||
---
|
||||
|
||||
NOTE: diff-ref mode entry (code committed in the same
|
||||
commit as this log); backfilled from the live dev
|
||||
session transcript per the `/prompt-io` skill rules.
|
||||
|
||||
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
|
||||
|
||||
Generated: normalize each registry addr via
|
||||
`tractor.discovery._addr.wrap_address()` before
|
||||
`.unwrap()`-ing for the `accept_addrs` bind check —
|
||||
entries may be raw `tuple`s when passed in from (test)
|
||||
client code. Import-path precedent taken from
|
||||
`piker/cli/__init__.py:336`.
|
||||
|
||||
> `git log -1 -p --follow -- piker/service/_registry.py`
|
||||
|
||||
Generated: `check_for_service()` ported to
|
||||
`tractor.query_actor(name, regaddr=...)` (kwarg was
|
||||
`arbiter_sockaddr=`), unpacking the new
|
||||
`(sockaddr, portal)` yield, and passing the
|
||||
now-required `open_registry(addrs=Registry.addrs)`.
|
||||
|
||||
> `git log -1 -p --follow -- tests/test_services.py`
|
||||
> `git log -1 -p --follow -- tests/test_ems.py`
|
||||
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
|
||||
|
||||
Key diagnostic reasoning (verbatim from session):
|
||||
|
||||
- the "DID NOT RAISE ContextCancelled" failure: in this
|
||||
test the client actor IS pikerd (in-proc), and
|
||||
current `tractor` main absorbs a `ContextCancelled`
|
||||
whose canceller is your own actor — self-requested
|
||||
cancels now exit cleanly instead of raising; the
|
||||
'sigint' variant propagates a bare collapsed
|
||||
`KeyboardInterrupt` rather than a
|
||||
`BaseExceptionGroup`.
|
||||
- the hard-coded `trio.fail_after(9)` startup budget is
|
||||
marginal — full stack boot (pikerd -> emsd ->
|
||||
brokerd.kraken -> paperboi + live kraken symbology
|
||||
fetch) occasionally exceeds 9s -> bumped to 19s.
|
||||
- `RemoteActorError.type` -> `.boxed_type`;
|
||||
`Position.size` -> `.cumsize` (the paper engine
|
||||
populates `BrokerdPosition.size` from `pp.cumsize`).
|
||||
- overlap survey (user-requested): all of the
|
||||
`repair_tests` branch commits are already in this
|
||||
stack's ancestry; this commit finishes that branch's
|
||||
port mission (its f4c4f1e2 fixed `conftest.py`'s
|
||||
`arbiter_sockaddr` usage; this fixes the remaining
|
||||
`test_services.py` + `check_for_service()` sites).
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
# AI Prompt I/O Log — claude
|
||||
|
||||
This directory tracks prompt inputs and model
|
||||
outputs for AI-assisted development using
|
||||
`claude` (claude-code CLI).
|
||||
|
||||
## Policy
|
||||
|
||||
Prompt logging follows the
|
||||
[NLNet generative AI policy][nlnet-ai].
|
||||
All substantive AI contributions are logged
|
||||
with:
|
||||
- Model name and version
|
||||
- Timestamps
|
||||
- The prompts that produced the output
|
||||
- Unedited model output (`.raw.md` files)
|
||||
|
||||
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
|
||||
|
||||
## Usage
|
||||
|
||||
Entries are created by the `/prompt-io` skill
|
||||
or automatically via `/commit-msg` integration.
|
||||
|
||||
Each commit carrying AI-generated changes links
|
||||
to its provenance entry via a `Prompt-IO:`
|
||||
commit-msg trailer; entries use "diff-ref mode"
|
||||
(pointers into `git log -p` instead of verbatim
|
||||
code copies) to avoid duplicating committed
|
||||
code.
|
||||
|
||||
Human contributors remain accountable for all
|
||||
code decisions. AI-generated content is never
|
||||
presented as human-authored work.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-07-22T00:13:58Z
|
||||
git_ref: 16f3cd6f
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the structured backfill phase after the analysis regressions.
|
||||
Characterize provider exhaustion and genuinely empty frames, then remove
|
||||
completion paths that can hang after child-task exit.
|
||||
|
||||
## Response summary
|
||||
|
||||
Removed orphan-prone completion events from the structured nursery flow
|
||||
and checked empty provider frames before indexing timestamp endpoints.
|
||||
Added deterministic timeout-bounded regressions for both exit paths.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
|
||||
- `tests/test_history_backfill.py` - deterministic exit regressions
|
||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
|
||||
- unedited response record
|
||||
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
|
||||
- provenance metadata and response summary
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated changes have not yet been edited by the human.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
provider: openai
|
||||
service: opencode
|
||||
timestamp: 2026-07-22T00:13:58Z
|
||||
git_ref: 16f3cd6f
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
|
||||
|
||||
Removed redundant completion events whose consumers waited only after
|
||||
their owning nurseries had joined. Added empty provider-frame handling
|
||||
before timestamp endpoint indexing so provider exhaustion exits cleanly.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
|
||||
|
||||
Added bounded, actor-free regressions for `DataUnavailable` and empty
|
||||
provider-frame completion.
|
||||
|
||||
Verification generated with the patch:
|
||||
|
||||
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
|
||||
|
||||
Result: 2 passed.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: ses_0799212ebffe42arY96czXn89F
|
||||
timestamp: 2026-08-11T20:58:26Z
|
||||
git_ref: 06d5ea5d
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Diagnose and fix an IB live-order failure where lazy NVDA contract
|
||||
qualification raised `tractor.trionics.Lagged` after the order reached
|
||||
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
|
||||
searches caused the stale state.
|
||||
|
||||
## Response summary
|
||||
|
||||
Traced the failure to competing consumers of a one-slot Trio broadcast, not
|
||||
the recent SHM change or stale asyncio client state. Replaced broadcast
|
||||
fan-out with one response/status demultiplexer, retained request-ID
|
||||
correlation, handled cancellation and teardown, documented the task/channel
|
||||
ownership model, warned on diagnostic metadata eviction, and added
|
||||
deterministic proxy regressions.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
|
||||
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
|
||||
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the user's live failure diagnosis.
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-11T20:58:26Z
|
||||
git_ref: 06d5ea5d
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user reported that certain IB order entries might have broken and
|
||||
provided a live failure for a first NVDA sell order. EMS delivered the
|
||||
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
|
||||
qualification failed in `MethodProxy._run_method()` with:
|
||||
|
||||
```text
|
||||
tractor.trionics._broadcast.Lagged:
|
||||
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
|
||||
dropped `0` values
|
||||
```
|
||||
|
||||
The user suspected a cancelled symbol-search task might have left stale
|
||||
asyncio state. Cancellation can leave an orphaned method response, but the
|
||||
observed failure was stale Trio broadcast state. `open_client_proxy()`
|
||||
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
|
||||
and an idle method receiver. Status traffic advanced only the relay, so the
|
||||
first method call for an uncached symbol entered an already-lagged receiver.
|
||||
Orders for eagerly cached contracts bypassed this path.
|
||||
|
||||
The immediately preceding sparse-history commit did not modify IB order,
|
||||
symbol, or proxy code; it exposed no direct dependency on this failure.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
|
||||
|
||||
Generated a sole-reader message demultiplexer for each IB client proxy.
|
||||
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
|
||||
status/error traffic, and dispatches `mid`-tagged responses into per-call
|
||||
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
|
||||
late responses to be logged and dropped without skewing later calls. Clean
|
||||
channel EOF is accepted during normal proxy teardown.
|
||||
|
||||
Follow-up review restored the masked reconnect TODO and per-message case
|
||||
rationale, documented the one-reader/many-writer task hierarchy and separate
|
||||
per-proxy channels, and added a warning with request method metadata when the
|
||||
bounded unresolved-request diagnostic table evicts its oldest entry.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
|
||||
|
||||
Generated deterministic regressions for idle status traffic before first
|
||||
qualification, two concurrent calls with reverse-order responses, cancelled
|
||||
calls with late responses followed by a successful call, and graceful
|
||||
channel EOF. Every synchronization wait is timeout-bounded.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
|
||||
|
||||
Registered the deterministic IB method-proxy target and source mapping.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
........................ [100%]
|
||||
25 passed, 4 warnings in 3.44s
|
||||
```
|
||||
|
||||
The warnings are existing Tractor/Trio deprecations from the EMS test.
|
||||
Adversarial review drove a clean-EOF guard and bounded polling. Final review
|
||||
found no actionable issues. Live IB order validation remains pending and
|
||||
requires restarting `brokerd.ib`.
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-13T20:24:02Z
|
||||
git_ref: 12b0d66e
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the paired piker broadcast-consumer work after Tractor PR 490
|
||||
stabilization. Stop cached feed callers from mutating or consuming shared
|
||||
quote-stream roots, then review, verify and prepare a commit boundary.
|
||||
|
||||
## Response summary
|
||||
|
||||
Made provider root streams private to the cached `Feed`, gave every caller a
|
||||
local subscribed view, corrected feed cache identity, and made multi-provider
|
||||
relays own synchronized child subscriptions and independent send-channel
|
||||
clones.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
|
||||
subscribed multi-provider relays with independent channel lifetimes.
|
||||
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
|
||||
cache-identity regressions, including sibling survival after provider EOC.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the requested cached-feed ownership item.
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-13T20:24:02Z
|
||||
git_ref: 12b0d66e
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user asked to continue the paired piker broadcast-consumer work after
|
||||
Tractor PR 490 stabilization. The next item was to stop cached feed callers
|
||||
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
|
||||
review, verify and prepare a commit boundary.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||
|
||||
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
|
||||
root streams and every caller, including the first cache owner, receives a
|
||||
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
|
||||
The cache identity now includes normalized full symbol shape and stream policy.
|
||||
|
||||
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
|
||||
of directly consuming a shared stream, and synchronizes relay startup before
|
||||
yielding the merged channel. Each relay owns a cloned send channel so one
|
||||
provider EOC cannot terminate its siblings. The module-level relay returns its
|
||||
live child receiver through `TaskStatus` for typed retention and introspection.
|
||||
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
|
||||
shared by local cache users.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
|
||||
|
||||
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
|
||||
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
|
||||
multi-provider relay fan-out and startup, duplex controls, lexical child
|
||||
closure, sibling survival after provider EOC, cached-root immutability, and
|
||||
full cache-key differentiation.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
... [100%]
|
||||
3 passed in 0.02s
|
||||
|
||||
. [100%]
|
||||
1 passed in 0.01s
|
||||
```
|
||||
|
||||
The second result is the existing EMS broadcast ownership regression. Ruff,
|
||||
E501, compilation and whitespace checks passed. Repeated adversarial review
|
||||
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
|
||||
relay-startup and first-owner asymmetry issues; final review reported no
|
||||
findings.
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-13T23:07:44Z
|
||||
git_ref: ddac3553
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Continue the piker broadcast-consumer audit after the Deribit ownership commit
|
||||
and prepare the next smallest ownership fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
|
||||
context and gave every caller a lexical child from one retained
|
||||
`LinkedTaskChannel` broadcaster root.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
|
||||
ownership.
|
||||
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
|
||||
lifetime regression.
|
||||
|
||||
## Human edits
|
||||
|
||||
None - generated output follows the requested broadcast-consumer audit.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-13T23:07:44Z
|
||||
git_ref: ddac3553
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
The user asked to continue the piker broadcast-consumer audit after committing
|
||||
the Deribit ownership fix. The next isolated item was IB's actor-local
|
||||
`_quote_streams` cache, which retained a raw channel owned by its first caller
|
||||
and constructed competing broadcaster roots for later callers.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
|
||||
|
||||
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
|
||||
made `open_aio_quote_stream()` cache that context by symbol through
|
||||
`maybe_open_context()`. Every caller now receives a lexical child from the
|
||||
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
|
||||
until its final cache user exits. Removed the manual `_quote_streams` registry
|
||||
and teardown mutation.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
|
||||
|
||||
Added a concurrent ownership regression using real Tractor broadcast state.
|
||||
It proves miss and hit callers use the same symbol/context factory, receive the
|
||||
same quote through distinct children, close independently, and leave the owner
|
||||
live after the hit exits.
|
||||
|
||||
Verification output:
|
||||
|
||||
```text
|
||||
. [100%]
|
||||
1 passed in 0.01s
|
||||
```
|
||||
|
||||
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
|
||||
Python compilation and whitespace checks pass. Normal package collection
|
||||
remains blocked by the existing Qt environment mismatch.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-20T16:35:19Z
|
||||
git_ref: 5f631e19
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked to implement the Piker compatibility work caused by
|
||||
tractor's canonical tagged-address migration, run the previously
|
||||
deferred tractor verification, and keep runtime-managed WireGuard
|
||||
bindspaces in a later separate PR.
|
||||
|
||||
## Response summary
|
||||
|
||||
Made Piker normalize legacy and tagged registry inputs before semantic
|
||||
comparison and process-global storage. Updated runtime address typing and
|
||||
test assertions for canonical output, retained legacy caller input, and
|
||||
removed a duplicate pytest log-level option that blocked downstream test
|
||||
collection with current tractor.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/service/_registry.py` - canonical registry normalization and
|
||||
shared address typing.
|
||||
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
|
||||
annotations.
|
||||
- `tests/conftest.py` - canonical fixture comparison and shared pytest
|
||||
option ownership.
|
||||
- `tests/test_services.py` - registry regression and canonical runtime
|
||||
assertions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly requested the downstream implementation and chose
|
||||
to keep WireGuard bindspace lifecycle work in a separate future tractor
|
||||
PR. The agent implemented and tested those directions; no direct manual
|
||||
source edits were observed.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-20T16:35:19Z
|
||||
git_ref: 5f631e19
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - tractor tagged-address compatibility
|
||||
|
||||
The human requested downstream Piker qualification after tractor PR
|
||||
#505 switched transport address emission to canonical tagged tuples.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
|
||||
|
||||
Normalized legacy and tagged registry addresses through tractor's
|
||||
address codec before Piker stores or compares them. Extended service
|
||||
runtime annotations to cover both tuple shapes, updated runtime tests to
|
||||
compare canonical serialized values, and removed Piker's duplicate
|
||||
registration of tractor's shared `--ll` pytest option.
|
||||
|
||||
Ruff passed. The focused normalization regression and Piker runtime boot
|
||||
passed against the local tractor checkout at `5d92595f`. The wider
|
||||
service module reached an unrelated pre-existing logger-name assertion
|
||||
in `maybe_spawn_daemon()` after both address-focused tests passed.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
session: tractor-addr-unpacking
|
||||
timestamp: 2026-08-21T04:20:58Z
|
||||
git_ref: b7aca4c0
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human approved refining Piker's logger-name handling after the
|
||||
wider service test exposed a stale parent-logger assertion during the
|
||||
tractor tagged-address compatibility work.
|
||||
|
||||
## Response summary
|
||||
|
||||
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
|
||||
preserving complete requested submodule names. Removed daemon startup
|
||||
assertions that incorrectly required parent logger names and added a
|
||||
regression covering both logger factories and names that the old
|
||||
character-set stripping silently corrupted.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/log.py` - exact relative logger-name derivation.
|
||||
- `piker/service/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/data/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
|
||||
- `piker/clearing/_ems.py` - remove redundant logger assertion.
|
||||
- `tests/test_logging.py` - package logger namespace regressions.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly approved this follow-up after reviewing the
|
||||
residual Piker service-suite gap. The agent implemented and verified
|
||||
the refinement; no direct manual source edits were observed.
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
model: gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-21T04:20:58Z
|
||||
git_ref: b7aca4c0
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
# Raw output - preserve Piker logger namespaces
|
||||
|
||||
The human approved a follow-up refinement after the full Piker service
|
||||
module exposed a stale logger-name assertion while qualifying tractor
|
||||
PR #505.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
|
||||
|
||||
Replaced character-set `str.lstrip()` calls with exact package-prefix
|
||||
removal, removed assertion-only parent-logger expectations from daemon
|
||||
startup paths, and added regression coverage for both logger factories
|
||||
and prefix-sensitive submodule names.
|
||||
|
||||
Ruff passed after removing newly unused logger assignments. All four
|
||||
logger regressions and all six service integration tests passed against
|
||||
the local tractor PR #505 checkout.
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: unavailable
|
||||
timestamp: 2026-08-26T22:26:00Z
|
||||
git_ref: 5f631e19
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Diagnose and fix the hanging Binance USD-M perpetual feed for
|
||||
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
|
||||
API and changelog. Ensure unsupported or silent websocket feeds fail
|
||||
loudly into `datad` and chart consumers instead of blocking startup.
|
||||
|
||||
## Response summary
|
||||
|
||||
Migrates Binance USD-M market data from the retired legacy websocket to
|
||||
the split public and market routes, handles Binance's new aggregate-trade
|
||||
schema, and introduces bounded connection and first-data startup. Direct
|
||||
stream URLs avoid reconnect-time subscription-ACK races, while targeted
|
||||
tests preserve routing and fail-loud behavior.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
|
||||
Binance quote streams.
|
||||
- `piker/brokers/binance/venues.py` - define current USD-M websocket
|
||||
endpoints.
|
||||
- `piker/data/_web_bs.py` - support bounded websocket context startup.
|
||||
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
|
||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
|
||||
retain the generated-output record.
|
||||
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
|
||||
structured provenance for the code changes.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human explicitly required fail-loud behavior rather than merely
|
||||
updating the endpoint, and supplied a direct CLI reproduction proving
|
||||
both `datad.binance` and the chart caller blocked. Those requirements
|
||||
drove the connection and first-data deadlines and their regressions. The
|
||||
changes remain uncommitted for human review.
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-26T22:26:00Z
|
||||
git_ref: 5f631e19
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user reported that selecting
|
||||
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
|
||||
websocket but never loaded the feed or downstream chart. They asked
|
||||
OpenCode to compare Binance's API changelog with piker's FQME and feed
|
||||
code, diagnose the failure, and solve it.
|
||||
|
||||
The user clarified that future provider failures must raise loudly
|
||||
instead of silently blocking feed and chart startup. They also confirmed
|
||||
the same hang from
|
||||
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Routes USD-M book-ticker and aggregate-trade data through Binance's new
|
||||
`/public` and `/market` websocket paths, merges both sockets into the
|
||||
existing quote stream, accepts the new `st` aggregate-trade field, and
|
||||
requires both stream classes before declaring startup complete. It also
|
||||
removes the misleading missing-provider warning, derives feed mode from
|
||||
resolved pair metadata, and raises `DataUnavailable` for connection or
|
||||
live-data startup timeouts.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
|
||||
|
||||
Defines Binance's current public and market USD-M websocket endpoints
|
||||
and makes the market endpoint the regular futures websocket API root.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Adds an optional connection deadline to `open_autorecon_ws()` so callers
|
||||
can fail context entry when repeated websocket handshakes never become
|
||||
connected.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||
|
||||
Adds regressions for exact Binance stream routing, silent live-data
|
||||
startup, silent websocket connection startup, and the current USD-M
|
||||
aggregate-trade payload schema.
|
||||
|
||||
## Verification output
|
||||
|
||||
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
|
||||
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
|
||||
through piker's merged stream received both required stream classes and
|
||||
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
|
||||
spot.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
|
||||
timestamp: 2026-08-28T20:03:52Z
|
||||
git_ref: dd676857
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Update the xonsh version used by the repository dev shell before stacking
|
||||
the Binance work, while retaining the dotrc compatibility shim as an
|
||||
immediate workaround for older shells.
|
||||
|
||||
## Response summary
|
||||
|
||||
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
|
||||
The new release supports Python 3.13, handles Git alias completion natively,
|
||||
and works in the updated Nix development shell.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml` - require xonsh `0.24.2` or newer.
|
||||
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
|
||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
|
||||
the triggering prompt, generated configuration summary, and checks.
|
||||
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
|
||||
structured provenance for the dependency update.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified old xonsh behavior as the root cause, reported the
|
||||
separate dotrc compatibility shim, and directed that the repository update
|
||||
remain a distinct flake-layer change before the Binance stack continues.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-28T20:03:52Z
|
||||
git_ref: dd676857
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user traced broken multiline conflict-resolution pastes to the old
|
||||
xonsh release installed by the repository dev shell. A separate dotrc
|
||||
agent added a compatibility shim, and the user directed that xonsh still
|
||||
be updated alongside the flake work before continuing the Binance stack.
|
||||
|
||||
## Generated configuration
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
|
||||
refreshes only xonsh's locked distribution metadata. The selected release
|
||||
supports Python 3.13, includes native Git-config alias completion, and fixes
|
||||
Nix metadata checks.
|
||||
|
||||
## Verification output
|
||||
|
||||
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
|
||||
xonsh from `0.22.4` to `0.24.2`.
|
||||
- Fresh `nix develop` replaces the worktree-local xonsh installation.
|
||||
- `xonsh --version` reports `0.24.2`.
|
||||
- A multi-command xonsh invocation executes Git commands successfully.
|
||||
- `nix flake check --no-build` passes for `x86_64-linux`.
|
||||
- `git diff --check` passes.
|
||||
|
||||
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
|
||||
that its distributions are yanked for an `unpivot` regression.
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||
timestamp: 2026-08-29T17:55:42Z
|
||||
git_ref: e09e306a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
|
||||
stack, then add missing deterministic tests and correctness fixes as a
|
||||
separate follow-up with a complete editor-gated commit plan.
|
||||
|
||||
## Response summary
|
||||
|
||||
Hardens Binance feed startup around its supported one-symbol contract,
|
||||
direct websocket routes, current provider schemas, and first-quote
|
||||
readiness. Adds deterministic Binance regressions and separates generic
|
||||
reconnect behavior into its own tested commit boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
|
||||
USD-M stream classes, and declare readiness after either quote class.
|
||||
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
|
||||
and documented book-ticker and aggregate-trade schemas.
|
||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
|
||||
the triggering prompt, generated-code summary, and verification output.
|
||||
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
|
||||
structured provenance for the follow-up.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user rejected reconstructing already committed Binance work, required
|
||||
preservation through cherry-pick, selected the flake/xonsh stack base, and
|
||||
required the review fixes to remain a separate tested boundary. The user also
|
||||
corrected the workflow to require an automatic `/commit-plan` at every stack
|
||||
review gate. During Tuicr review, the user rejected trade-gated startup,
|
||||
requested a feed-specific initialization error and async-generator TODO,
|
||||
and required generic reconnect changes and tests to land independently.
|
||||
The user then clarified that either L1 or trade data must unblock startup,
|
||||
rather than preferring one stream class or waiting independently for both.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-08-29T17:55:42Z
|
||||
git_ref: e09e306a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user asked to land recent branch work in small chronological patch sets,
|
||||
including missing tests. For Binance, the user required preserving the
|
||||
existing committed topic patch through cherry-pick and placing newly found
|
||||
correctness repairs in a separate follow-up boundary above the flake/xonsh
|
||||
stack. The user also required every yielded stack layer to include a complete
|
||||
`/commit-plan` with patch-specific checks.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
|
||||
|
||||
Restricts Binance feeds to the supported one-symbol contract, renders direct
|
||||
quote routes, separates L1 liveness from first-trade readiness, and derives
|
||||
the contextual trade timeout from Trio's enclosing effective deadline.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Lets structured cancellation bypass reconnect throttling and ensures handled
|
||||
handshake rejections retry before accessing connection-local cancel state.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
|
||||
|
||||
Adds deterministic regressions for symbol cardinality, route selection,
|
||||
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
|
||||
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
|
||||
|
||||
## Verification output
|
||||
|
||||
- Targeted Binance tests: `10 passed in 0.26s`.
|
||||
- Ruff passes for all three changed Python files.
|
||||
- `git diff --check` passes.
|
||||
- Independent final review reports no actionable findings.
|
||||
- No live Binance network probe was run.
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
|
||||
timestamp: 2026-09-01T02:42:35Z
|
||||
git_ref: e09e306a
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Address the Tuicr review by separating generic websocket reconnect
|
||||
changes from Binance-specific work and testing the real reconnect task
|
||||
hierarchy rather than a reimplementation.
|
||||
|
||||
## Response summary
|
||||
|
||||
Preserves structured cancellation and safe first-handshake retries in
|
||||
`_web_bs`. Adds three focused regressions in a dedicated websocket test
|
||||
module and packages them as an independent commit boundary.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py` - restart handled handshake failures and let
|
||||
cancellation bypass the reconnect throttle.
|
||||
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
|
||||
and cancellation task paths with controlled transport boundaries.
|
||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
|
||||
- retain the review prompt, generated-code summary, and verification.
|
||||
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
|
||||
structured provenance for the generic reconnect boundary.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified that these changes were generic rather than
|
||||
Binance-specific, required a separate commit and test module, requested
|
||||
coverage of the real reconnect hierarchy, and corrected code prose to
|
||||
refer to `trio` consistently. A follow-up review required the production
|
||||
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
|
||||
handler and cite the regression which audits that teardown invariant.
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-09-01T02:42:35Z
|
||||
git_ref: e09e306a
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested that generic `_web_bs` reconnect fixes and tests be
|
||||
separated from the Binance backend patch. The review also required tests
|
||||
to exercise the real reconnect task hierarchy, cite the previous faulty
|
||||
branches, and consistently name `trio` in code prose.
|
||||
|
||||
## Generated code
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Ensures handled handshake failures immediately restart the reconnect
|
||||
loop and lets structured cancellation bypass shielded error throttling.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Adds deterministic scheduler-real regressions for handshake startup
|
||||
timeouts, immediate rejection retries, and cancellation propagation.
|
||||
|
||||
## Verification output
|
||||
|
||||
- Generic and Binance targeted tests: `9 passed in 0.06s`.
|
||||
- Ruff passes for all four changed Python files.
|
||||
- `git diff --check` passes.
|
||||
- No live network probe was run.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
session: 4dc920cf-e82e-4c1e-83db-75a0a9efe82b
|
||||
timestamp: 2026-09-01T23:00:43Z
|
||||
git_ref: e91f41a6
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20260901T230043Z_e91f41a6_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
Update `wkt/land_flake_update` to Tractor's restored `piker_pin` after
|
||||
reusing the existing runtime compatibility commits. Keep stale
|
||||
historical lock snapshots out of the final dependency state.
|
||||
|
||||
## Response summary
|
||||
|
||||
Updates piker's Python and `msgspec` requirements for current Tractor,
|
||||
then generates one fresh lock at Tractor `piker_pin@0d95a9e6`. The
|
||||
dependency commit completes the compatibility stack already
|
||||
cherry-picked onto the flake branch.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml` - require Python 3.13 and `msgspec>=0.20.0`.
|
||||
- `uv.lock` - resolve Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`,
|
||||
and current transport dependencies.
|
||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.raw.md`
|
||||
- preserve the prompt, generated-change summary, and resolution notes.
|
||||
- `ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md` - record
|
||||
structured provenance for the dependency update.
|
||||
|
||||
## Human edits
|
||||
|
||||
The user identified that Tractor compatibility should land with the
|
||||
flake update rather than the Binance patch, supplied four known source
|
||||
commits, required a broader local-history audit for missing ports, and
|
||||
directed a fresh lock from the restored `piker_pin` instead of replaying
|
||||
historical pins.
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
model: openai/gpt-5.6-sol
|
||||
service: opencode
|
||||
timestamp: 2026-09-01T23:00:43Z
|
||||
git_ref: e91f41a6
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The user requested that the flake-update branch absorb the Tractor
|
||||
runtime compatibility work before the Binance branch lands. They
|
||||
directed reuse of existing compatibility commits, exclusion of stale
|
||||
historical lock pins, and a fresh update to Tractor's restored
|
||||
`piker_pin`.
|
||||
|
||||
## Generated changes
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml`
|
||||
|
||||
Narrows piker support to Python 3.13 and raises the direct `msgspec`
|
||||
requirement to the version family required by current Tractor.
|
||||
|
||||
> `git diff --cached -- uv.lock`
|
||||
|
||||
Relocks against Tractor `piker_pin@0d95a9e6`, `msgspec 0.21.1`, and the
|
||||
current transport dependency graph while retaining the existing Xonsh
|
||||
update.
|
||||
|
||||
## Resolution notes
|
||||
|
||||
- Historical lock hunks from reused commits were intentionally omitted.
|
||||
- Tractor runtime compatibility remains represented by the preceding
|
||||
individually preserved commits on `wkt/land_flake_update`.
|
||||
- Project checks are rendered in the commit plan for human execution.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
created_at: 2026-10-21T22:44:21Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 507c75c6
|
||||
scope: config
|
||||
substantive: true
|
||||
raw_file: 20261021T224421Z_507c75c6_piker_pin_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human asked to sanity-check the current PR94 landing state against
|
||||
the qualified new Tractor pin before deduplicating branch history. After
|
||||
the initial offline checks passed, the human authorized changing the
|
||||
selector to piker_pin, resolving the lock, and verifying the installed
|
||||
dependency without the temporary local-source PYTHONPATH override.
|
||||
The human explicitly authorized network resolution and a scoped update
|
||||
of Tractor/runtime dependencies while preserving unrelated packages.
|
||||
|
||||
## Response summary
|
||||
|
||||
Updated the branch selector and lock to published piker_pin@b64c89f0.
|
||||
Installed the exact locked Tractor revision, verified its installed
|
||||
source metadata, and reran 14 offline logger/startup cases without a
|
||||
local Tractor source override. Preserved the rest of the environment.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pyproject.toml`: select piker_pin and bound Polars below 2.0.
|
||||
- `uv.lock`: lock b64c89f0 and Polars/runtime 1.44.2.
|
||||
- This provenance pair.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
14 distinct offline cases passed against the installed Tractor update
|
||||
with the existing Polars 1.36 installation. The actor.uid deprecation
|
||||
warning remains. No live provider/order testing occurred.
|
||||
|
||||
The initial resolver selected Polars 2.0 and rejected retaining 1.36
|
||||
for its Windows target. The human then requested the latest pre-2.0
|
||||
release instead. PyPI metadata identified non-yanked stable 1.44.2;
|
||||
adding `<2.0.0` to the existing lower bound and relocking succeeded
|
||||
with both polars and polars-runtime-32 at 1.44.2, without platform edits.
|
||||
The raw partner retains the original, superseded 2.0 generation history.
|
||||
|
||||
The scoped install changed only Tractor. Polars 1.44.2 is locked but
|
||||
not installed or data-path qualified yet. The human's existing staged
|
||||
checkpoint is preserved; the pre-2.0 correction is in the worktree.
|
||||
No history cleanup, staging or commit was performed by this agent.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
created_at: 2026-10-21T22:44:21Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 507c75c6
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
|
||||
|
||||
Changed the landing branch's Tractor selector from wkt/addr_unpacking
|
||||
to piker_pin and resolved the published branch to
|
||||
b64c89f05a9fc8ac2878551eb5eda36574ebef6a. An offline branch lookup lacked
|
||||
the cached name, so the human explicitly authorized network resolution
|
||||
and a scoped installed-runtime update preserving unrelated packages.
|
||||
|
||||
The resolver advanced Polars and polars-runtime-32 from 1.36.0 to 2.0.0.
|
||||
An attempt to retain both exact 1.36.0 versions was rejected for the
|
||||
resolver's Windows target. The valid lock with the visible Polars bump
|
||||
is retained for review; no platform restriction or source constraint was
|
||||
introduced to bypass the failure.
|
||||
|
||||
Installed only Tractor and its required dependencies with uv pip rather
|
||||
than syncing/pruning the whole environment. The installer reported one
|
||||
package replacement: Tractor a966928d to b64c89f0. Qt distribution versions
|
||||
remained PyQt6 6.10.1, PyQt6-Qt6 6.10.1 and PyQt6-sip 13.10.3.
|
||||
|
||||
Removed PYTHONPATH from verification subprocesses. Piker resolved inside
|
||||
land_flake_update and Tractor resolved from that checkout's py313
|
||||
site-packages. Installed VCS metadata matched the exact locked revision.
|
||||
The temporary LD_LIBRARY_PATH used the binding's bundled Qt libraries
|
||||
and matching native dependencies; no global environment setup changed.
|
||||
|
||||
Verification output:
|
||||
- Logger/non-actor startup: 12 passed, 1 deselected in 0.05s.
|
||||
- test_runtime_boot: 1 passed, 1 warning in 0.04s.
|
||||
- test_daemon_bind_distinct_from_registrar: 1 passed in 0.77s.
|
||||
- Existing warning: actor.uid deprecation; no failure.
|
||||
- git diff --check passed.
|
||||
|
||||
This verifies the installed Tractor pin, not a full sync to all new lock
|
||||
versions. Polars was not upgraded in the environment by the scoped install.
|
||||
No live broker feed, order, branch rewrite, staging or commit was run.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
created_at: 2026-10-22T21:22:48Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6-astra
|
||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261022T212248Z_dc618229_ib_recovery_prompt_io.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human approved trying the bounded candidate: back out the IB
|
||||
quote-context cache, import adapter-only reset/boundary fixes and
|
||||
the missing EMS subscription fix, preserve source authorship and
|
||||
commit boundaries, and exercise the real Tractor bridge. They
|
||||
required an explicit keep/backout audit and clarification that
|
||||
the evaluated downstream lineage lacked the broadcast fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
Verified identical stable patch IDs for original 4af885d9 and
|
||||
landing replay 635314c7. Created `wkt/land_ib_recovery` at dc618229
|
||||
in the main checkout. Applied source-equivalent adapter backports
|
||||
from 0846cbd4 and 1da9b760 and the EMS code from 12b0d66e, with
|
||||
source attribution recorded in `ai/ib-recovery-candidate.md`.
|
||||
|
||||
The real-runtime quote test first reproduced the reported fatal
|
||||
TrioTaskExited propagation with caching enabled. The direct-owner
|
||||
backout restores restart and subsequent RPC survival, but a strict
|
||||
callback-cleanliness test remains red. Real bridge history pacing
|
||||
recovery and real broadcaster dark clearing pass. No live broker,
|
||||
VNC, orders or TSDB operations were performed by the agent.
|
||||
|
||||
On 2026-10-23 the human supplied successful live startup/backfill,
|
||||
sequential reattachment and overlapping chart-close/reopen results,
|
||||
plus the still-failing daemon SIGINT teardown. The human accepted
|
||||
deferring teardown from #94, requested loud partial-backport
|
||||
attribution and required downstream rebase rehearsals before landing.
|
||||
The agent removed only the explicit single-null debugger stop,
|
||||
retaining its legacy result. Downstream `976db6c3` instead rewrites
|
||||
null grouping; the local cleanup does not claim that algorithmic fix.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/brokers/ib/feed.py`: source-equivalent backout/backports.
|
||||
- `piker/clearing/_ems.py`: original EMS subscription patch.
|
||||
- `tests/test_ib_broadcast.py`: removed with the reverted behavior.
|
||||
- `tests/test_ib_history.py`: original adapter regressions only.
|
||||
- `tests/test_ems_broadcast.py`: imported and real-broadcaster tests.
|
||||
- `tests/test_ib_quote_restart.py`: real Tractor boundary tests.
|
||||
- `piker/tsp/_anal.py`: local single-null breakpoint removal only.
|
||||
- `ai/ib-recovery-candidate.md`: provenance and qualification limits.
|
||||
|
||||
## Human edits
|
||||
|
||||
The human required original fixes and authorship, rejected broad
|
||||
speculative integration work, required real concurrency boundaries
|
||||
in tests, and corrected the interpretation of live qualification.
|
||||
They performed the live tests, chose to defer known teardown defects
|
||||
from #94, permitted the minimal debugger cleanup, and requested
|
||||
explicit downstream discrepancies and rebase compatibility checks.
|
||||
The agent applied those directed edits; no direct manual source edits
|
||||
are known in this candidate. The raw partner retains the originally
|
||||
captured output; this structured record includes later contributions.
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
created_at: 2026-10-22T21:22:48Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6-astra
|
||||
git_ref: dc618229bd8c9c90aa3160e26677b553a7d472c7
|
||||
diff_cmd: git diff dc618229 -- piker/brokers/ib/feed.py piker/clearing/_ems.py tests
|
||||
---
|
||||
|
||||
## Human direction
|
||||
|
||||
> yeah i guess this is our best approach strat for sure.
|
||||
|
||||
> so you're saying these 2 are not the same patch?
|
||||
> please tell me no lol
|
||||
|
||||
> so detail which commits to drop vs. keep please in detail so i can
|
||||
> audit your choices this time..
|
||||
|
||||
> let's try it and consider bailing and re-trying if this leads us into
|
||||
> a further (and deeper) sinkhole, bc as it stands right now this whole
|
||||
> endevour has turned into a complete shitfest XD
|
||||
|
||||
> you didn't respond to my ask about the wkt/ib_downstream_pin_eval stuff and it missing the bcast rx commit either
|
||||
|
||||
## Unedited diagnostic response
|
||||
|
||||
The real-bridge test reproduced the exact fatal path:
|
||||
`maybe_open_context()` teardown raises `TrioTaskExited` back into
|
||||
the quote consumer.
|
||||
|
||||
With only the quote-cache backout, both quote generations complete
|
||||
and their asyncio tasks finish. Tractor still emits
|
||||
`TrioTaskExited` callback errors, so the strict test remains red.
|
||||
That narrows the result: the backout removes the fatal propagation
|
||||
in this reproducer, but doesn’t make bridge teardown clean.
|
||||
|
||||
## Code output pointers
|
||||
|
||||
> `git diff dc618229 -- piker/brokers/ib/feed.py`
|
||||
|
||||
Back out 635314c7's quote owner and import the original adapter-only
|
||||
deltas from 0846cbd4 and 1da9b760.
|
||||
|
||||
> `git diff dc618229 -- piker/clearing/_ems.py`
|
||||
|
||||
Import 12b0d66e's dedicated dark-clearing quote subscription.
|
||||
|
||||
> `git diff dc618229 -- tests/test_ib_broadcast.py tests/test_ib_quote_restart.py tests/test_ib_history.py tests/test_ems_broadcast.py`
|
||||
|
||||
Replace the reverted mock-cache test with a real-runtime restart
|
||||
reproducer; retain original history/EMS tests and add real bridge
|
||||
history recovery and real broadcaster EMS coverage. Newly added
|
||||
test files remain untracked until the human stages them, so the
|
||||
diff pointers include them only after staging or committing.
|
||||
|
||||
## Verification output
|
||||
|
||||
```text
|
||||
21 passed in 0.53s
|
||||
1 failed, 2 passed in 5.53s
|
||||
13 passed in 1.41s
|
||||
1 passed, 1 warning in 0.04s
|
||||
2 passed in 0.03s
|
||||
```
|
||||
|
||||
The last run includes the original EMS case again and one additional
|
||||
real-broadcaster case: 38 unique passes and one strict callback failure.
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:08:46Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 7242dcb9
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T160846Z_7242dcb9_websocket_close.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported repeated live Binance perp crashes after correct
|
||||
worktree startup: the real trio-websocket reader attempted a Pong
|
||||
while wsproto was LOCAL_CLOSING. They asked whether an existing
|
||||
downstream websocket fix should be reused, then requested:
|
||||
|
||||
> ok add the fix now, follow by a /commit-plan and then i'll test
|
||||
> before saving it ya?
|
||||
|
||||
They approved a real websocket reader/close regression covering
|
||||
recovery and intentional shutdown, and separately authorized
|
||||
reconciling this worktree's legacy workflow bookkeeping while
|
||||
preserving existing archives.
|
||||
|
||||
## Response summary
|
||||
|
||||
The older `3d83b61f3f7b982e84a4230dba5777f1407c78f1` wrapper
|
||||
suppressed escaped communication failures. The landing replay
|
||||
`97eb8347` of `d50ab6124cd0e1646c82fa4a85d7c13bfff82611`
|
||||
added propagation; it did not restart a reader-context failure.
|
||||
Existing `LocalProtocolError` membership in NoBsWs.recon_errors
|
||||
only protected the receive relay, not reader-nursery context exit.
|
||||
|
||||
Classify these errors around the complete websocket context and
|
||||
retry instead of restoring outer suppression. Place collapse_eg
|
||||
outside that context, invalidate stale connection readiness and
|
||||
socket references, and leave the retry wait cancellable. Preserve
|
||||
unexpected failures and mixed exception groups for propagation.
|
||||
After review, use RuntimeError rather than ValueError for the
|
||||
test-only unexpected-failure sentinel. Its injected exception has
|
||||
no underlying cause. Production recovery logs retain the original
|
||||
traceback and now name the URL; unexpected exceptions use bare
|
||||
raise to preserve identity, existing causes and notes.
|
||||
|
||||
The deterministic seam controls protocol-state ordering in the real
|
||||
client Ping handler. It does not replace the websocket transport,
|
||||
reader, context manager, reconnect task or wsproto exception.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py`: reader-context failure recovery.
|
||||
- `tests/test_websocket.py`: loopback recovery and two shutdown
|
||||
variants; retain unexpected-failure propagation control.
|
||||
- This provenance pair: live report, lineage and test scope.
|
||||
|
||||
## Verification
|
||||
|
||||
Baseline uses new tests against original HEAD websocket code in an
|
||||
isolated exact-tree checkout. The recovery case fails with the
|
||||
reported Pong-in-LOCAL_CLOSING LocalProtocolError.
|
||||
|
||||
Fixed code passes 13 cases across tests/test_websocket.py and
|
||||
tests/test_binance.py. Exact-boundary verification and native-library
|
||||
configuration are retained in the ignored commit-plan package.
|
||||
Live testing remains human-owned and pending.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct manual source edits observed in this follow-up. The human
|
||||
supplied live failures, corrected the earlier daemon/worktree setup,
|
||||
set the narrow landing scope, requested source-fix reuse, and will
|
||||
test before reviewing and committing. Existing commits are retained.
|
||||
The human requested the RuntimeError sentinel and preservation of
|
||||
underlying error context. The raw partner retains the initial output.
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:08:46Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 7242dcb9
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Implemented a focused websocket-context reconnect fix and real
|
||||
loopback reader/close regressions in the Binance landing worktree.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Move exception-group collapse outside the websocket context so
|
||||
reader-nursery exit errors are classified by the reconnect loop.
|
||||
Retry NoBsWs.recon_errors there, clear the stale connected event
|
||||
and socket reference, and honor cancellation through unshielded
|
||||
retry sleep. Retain unexpected-error propagation at the outer API.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Exercise real TCP, handshake, Ping frames, reader dispatch and
|
||||
wsproto Pong validation. A narrow handler gate advances the real
|
||||
protocol state to LOCAL_CLOSING without sending the Close frame,
|
||||
preventing a peer reply from racing the controlled state. Require
|
||||
fresh application data from the replacement connection. Separate
|
||||
variants cancel at the reader gate and at the actual retry wait.
|
||||
Update the existing unexpected-failure control to use ValueError
|
||||
because LocalProtocolError is now intentionally recoverable.
|
||||
|
||||
The isolated original-code/new-test baseline fails with the exact
|
||||
Pong-in-LOCAL_CLOSING error. The fixed isolated tree passes all 13
|
||||
selected Binance/websocket cases. No live provider, broker orders
|
||||
or production daemon operations were performed by the agent.
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:56:38Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: b95e7c79
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T165638Z_b95e7c79_reconnect_logging.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported that the live Binance Pong-in-LOCAL_CLOSING
|
||||
error now logged a retry without crashing. They requested visible
|
||||
re-establishment reporting under `pikerd -l info --pdb`, and asked
|
||||
to check for existing downstream work before adding a fix.
|
||||
|
||||
## Response summary
|
||||
|
||||
The existing `Connection success` INFO message already identifies
|
||||
handshake completion. Feed startup configured piker.data.feed,
|
||||
while shared websocket diagnostics use ._util.log on piker.data.
|
||||
Configure that actual emitter and prefer the daemon's verbosity
|
||||
over a quieter client request for these shared transport tasks.
|
||||
|
||||
Add a readiness INFO message after fixture setup and a first-message
|
||||
INFO report once per connection after actual relay. Include the URL
|
||||
in the receive-timeout reset error. Do not call first raw websocket
|
||||
traffic a parsed quote or proof of restored market-data freshness.
|
||||
|
||||
## Existing-work audit
|
||||
|
||||
Compared `_web_bs.py`, `_util.py`, `piker.log` and daemon logging
|
||||
against the evaluated downstream tip and relevant all-ref histories.
|
||||
`54da2973` contains logging/type cleanup, not this emitter fix.
|
||||
`5adf77a9` / landing equivalents address package-name correctness,
|
||||
but not configuration of the shared data emitter at feed startup.
|
||||
All-ref searches for this actual emitter/readiness/first-relay
|
||||
implementation found no existing commit. The dirty IB history
|
||||
worktree has only an unrelated f-string correction in `_web_bs.py`;
|
||||
the TIPC worktree has no dirty delta in these two feed/websocket
|
||||
files. No downstream work was overwritten or imported wholesale.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/feed.py`: shared transport logger startup/verbosity.
|
||||
- `piker/data/_web_bs.py`: readiness, first-relay and timeout URL.
|
||||
- `tests/test_logging.py`: actual emitter/console startup regression.
|
||||
- `tests/test_websocket.py`: recovery milestones on real loopback.
|
||||
- This prompt provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
The new logging regression against original b95e7c79 source fails:
|
||||
the actual websocket emitter remains at WARNING and rejects INFO.
|
||||
It passes with the logger wiring fix. Provider lookup is stopped
|
||||
before network work, while logger factories and console emission
|
||||
are real. Global logger state is restored after the check.
|
||||
|
||||
18 cases pass across tests/test_logging.py, tests/test_websocket.py
|
||||
and tests/test_binance.py. git diff --check passes. Live verification
|
||||
after restarting the daemon with these edits remains human-owned.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct manual source edits observed. The human supplied live
|
||||
recovery evidence, selected INFO verbosity, requested downstream
|
||||
reuse inspection, and retains live-test and commit ownership.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
---
|
||||
created_at: 2026-10-28T16:56:38Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: b95e7c79
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Confirmed the existing INFO handshake-success log was hidden by
|
||||
logger wiring: websocket code emits on piker.data, while feed
|
||||
startup configured piker.data.feed. The baseline regression leaves
|
||||
the emitter at WARNING for an INFO daemon and ERROR client.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
|
||||
|
||||
Configure the actual shared ._util.log namespace, with daemon
|
||||
verbosity taking priority over client verbosity for actor-wide
|
||||
transport diagnostics.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Keep the existing handshake-success message. Add INFO readiness
|
||||
after fixture setup and first-message relay once per connection.
|
||||
Include the URL in the receive-timeout reset diagnostic.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_logging.py tests/test_websocket.py`
|
||||
|
||||
Prove production startup enables real console INFO on the websocket
|
||||
emitter without contacting a provider. Extend the real loopback
|
||||
recovery test to require readiness for both connections and one
|
||||
first-message report for the connection delivering fresh data.
|
||||
|
||||
All 18 targeted logging/Binance/websocket cases passed. No live
|
||||
provider/daemon operations were performed by the agent. First relay
|
||||
is raw websocket traffic; parsed market-data recovery remains a
|
||||
separate live observation.
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
---
|
||||
created_at: 2026-10-28T18:08:56Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 032580bf
|
||||
scope: code
|
||||
substantive: true
|
||||
raw_file: 20261028T180856Z_032580bf_timeout_phase.raw.md
|
||||
---
|
||||
|
||||
## Prompt
|
||||
|
||||
The human reported repeated readiness without first-message relay,
|
||||
specifically after first attempting an invalid .usdm. FQME and then
|
||||
the corrected .usdtm. one. They requested distinct log messages for
|
||||
receive starvation versus local backpressure.
|
||||
|
||||
## Response summary
|
||||
|
||||
Add phase-specific WARNING messages for each elapsed relay-loop
|
||||
deadline, with URL, timeout duration, accumulated count and reset
|
||||
threshold. Leave the existing shared receive/send deadline and
|
||||
counter/reset semantics intact. Do not claim that connected/ready
|
||||
sockets alone establish parsed quote recovery.
|
||||
|
||||
Read-only invalid-venue tracing found the usd substring check in
|
||||
Binance get_mkt_info() accepts USDM while retaining it in MktPair.
|
||||
Client.bars() later uses that market identity against canonical
|
||||
USDTM keys. Backend stream startup precedes history startup and
|
||||
sampler consumption. A relationship to the corrected-feed stall
|
||||
still needs runtime evidence; neither venue handling nor task
|
||||
lifetime behavior was changed by this diagnostic patch.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `piker/data/_web_bs.py`: track and report the timed-out operation.
|
||||
- `tests/test_websocket.py`: real silent-source and blocked-sink cases.
|
||||
- This prompt provenance pair.
|
||||
|
||||
## Verification
|
||||
|
||||
15 selected Binance/websocket cases passed. The new cases exercise
|
||||
real TCP, websocket handshake and relay behavior, replacing only
|
||||
log observation. A zero-capacity relay deliberately forces the
|
||||
local-send case; the source-silent case never receives a payload.
|
||||
git diff --check passed. No live provider/daemon operations run.
|
||||
|
||||
## Human edits
|
||||
|
||||
No direct source edits observed. The human supplied the critical
|
||||
invalid-symbol-then-corrected-symbol trigger and requested the
|
||||
phase distinction; live testing and task acceptance remain theirs.
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
created_at: 2026-10-28T18:08:56Z
|
||||
generated_by:
|
||||
harness: opencode
|
||||
provider: openai
|
||||
model: openai/gpt-6.1-sol
|
||||
git_ref: 032580bf
|
||||
diff_cmd: git diff HEAD~1..HEAD
|
||||
---
|
||||
|
||||
Added distinct timeout diagnostics for websocket receive starvation
|
||||
versus local relay backpressure, preserving the existing timeout,
|
||||
counter and reconnect threshold policy.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
|
||||
|
||||
Track whether proxy_msgs() is awaiting ws.get_message() or
|
||||
snd.send(). Report that phase, URL, configured deadline, timeout
|
||||
count and reset threshold for every timeout, before the existing
|
||||
threshold/reset diagnostic. Capture the configured deadline once
|
||||
per operation so the report matches the deadline that elapsed.
|
||||
|
||||
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
|
||||
|
||||
Two real loopback variants use a silent server versus one payload
|
||||
and an unconsumed zero-capacity Trio relay. Require the appropriate
|
||||
phase and URL on the actual first timeout. All 15 selected
|
||||
Binance/websocket cases pass. No live provider was contacted.
|
||||
|
||||
Read-only tracing found a permissive invalid-venue path: get_mkt_info
|
||||
accepts USDM via its usd substring check and preserves that venue
|
||||
in MktPair, whereas Client.bars() indexes canonical USDTM pair keys.
|
||||
This identifies a potential late failed-startup path, not proof of
|
||||
the corrected-chart stall or of leaked producer ownership.
|
||||
|
|
@ -1,8 +1,29 @@
|
|||
[network]
|
||||
# This Tractor pin supports one transport per actor, not mixed TCP/UDS.
|
||||
# Use the single TCP endpoint shown in each list for this version.
|
||||
# Mixed-transport configs need a future Tractor pin with that support.
|
||||
pikerd = [
|
||||
'/ipv4/127.0.0.1/tcp/6116', # std localhost daemon-actor tree
|
||||
# '/uds/6116', # TODO std uds socket file
|
||||
# TCP localhost loopback
|
||||
'/ip4/127.0.0.1/tcp/6116',
|
||||
|
||||
# UDS alternative: replace TCP and select the UDS transport.
|
||||
# '/unix/run/user/1000/piker/pikerd.sock',
|
||||
]
|
||||
chart = [
|
||||
'/ip4/127.0.0.1/tcp/3003',
|
||||
# '/unix/run/user/1000/piker/chart.sock',
|
||||
]
|
||||
# the service-actor registry endpoint;
|
||||
# other pikerd trees contact this to discover
|
||||
# actors.
|
||||
# XXX if absent, pikerd binds the registry
|
||||
# on its own tpt_bind_addrs.
|
||||
# regd = ['/ip4/127.0.0.1/tcp/6116']
|
||||
|
||||
# chart = [
|
||||
# '/ip4/127.0.0.1/tcp/3333',
|
||||
# '/unix/run/user/1000/piker/chart@3333.sock',
|
||||
# ]
|
||||
|
||||
|
||||
[ui]
|
||||
|
|
|
|||
|
|
@ -2,11 +2,11 @@
|
|||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1765779637,
|
||||
"narHash": "sha256-KJ2wa/BLSrTqDjbfyNx70ov/HdgNBCBBSQP3BIzKnv4=",
|
||||
"lastModified": 1784497964,
|
||||
"narHash": "sha256-vlHUuqAcbcH2RKmHbPiuQzbv1pnzzavXnI62RD0bqCU=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "1306659b587dc277866c7b69eb97e5f07864d8c4",
|
||||
"rev": "241313f4e8e508cb9b13278c2b0fa25b9ca27163",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
|
|
|
|||
23
flake.nix
23
flake.nix
|
|
@ -22,12 +22,14 @@
|
|||
|
||||
# do store-path extractions
|
||||
qt6baseStorePath = lib.getLib pkgs.qt6.qtbase;
|
||||
zlibStorePath = lib.getLib pkgs.zlib;
|
||||
# ?TODO? can remove below since manual linking not needed?
|
||||
# qt6QtWaylandStorePath = lib.getLib pkgs.qt6.qtwayland;
|
||||
|
||||
# XXX NOTE XXX, for now we overlay specific pkgs via
|
||||
# a major-version-pinned-`cpython`
|
||||
cpython = "python313";
|
||||
python = pkgs.${cpython};
|
||||
pypkgs = pkgs."${cpython}Packages";
|
||||
in
|
||||
{
|
||||
|
|
@ -46,7 +48,7 @@
|
|||
qt6.qtbase
|
||||
|
||||
uv
|
||||
python313 # ?TODO^ how to set from `cpython` above?
|
||||
python
|
||||
pypkgs.pyqt6
|
||||
pypkgs.pyqt6-sip
|
||||
pypkgs.qtpy
|
||||
|
|
@ -70,6 +72,7 @@
|
|||
|
||||
# link-in c++ stdlib for various AOT-ext-pkgs (numpy, etc.)
|
||||
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH"
|
||||
LD_LIBRARY_PATH="${zlibStorePath}/lib:$LD_LIBRARY_PATH"
|
||||
|
||||
export LD_LIBRARY_PATH
|
||||
|
||||
|
|
@ -86,15 +89,31 @@
|
|||
export QT_WAYLAND_SHELL_INTEGRATION="xdg-shell"
|
||||
|
||||
# ------ uv ------
|
||||
# ignore any venv inherited from an outer checkout
|
||||
unset VIRTUAL_ENV
|
||||
# - always use the ./py313/ venv-subdir
|
||||
export UV_PROJECT_ENVIRONMENT="py313"
|
||||
|
||||
# Rebuild after a flake update changes the Nix Python closure.
|
||||
venv_python="$UV_PROJECT_ENVIRONMENT/bin/python"
|
||||
nix_python="$(readlink -f "${python}/bin/python3")"
|
||||
if test -e "$venv_python" \
|
||||
&& test "$(readlink -f "$venv_python")" != "$nix_python"
|
||||
then
|
||||
uv venv \
|
||||
--clear \
|
||||
--python "$nix_python" \
|
||||
"$UV_PROJECT_ENVIRONMENT"
|
||||
fi
|
||||
|
||||
# sync project-env with all extras
|
||||
uv sync --dev --all-extras --no-group lint
|
||||
source "$UV_PROJECT_ENVIRONMENT/bin/activate"
|
||||
|
||||
# ------ TIPS ------
|
||||
# NOTE, to launch the py-venv installed `xonsh` (like @goodboy)
|
||||
# run the `nix develop` cmd with,
|
||||
# >> nix develop -c uv run xonsh
|
||||
# >> nix develop -c xonsh
|
||||
'';
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -324,10 +324,13 @@ def load_ledger(
|
|||
ldir: Path = (
|
||||
dirpath
|
||||
or
|
||||
config._config_dir / 'accounting' / 'ledgers'
|
||||
config.get_conf_dir() / 'accounting' / 'ledgers'
|
||||
)
|
||||
if not ldir.is_dir():
|
||||
ldir.mkdir()
|
||||
ldir.mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
|
||||
fname = f'trades_{brokername}_{acctid}.toml'
|
||||
fpath: Path = ldir / fname
|
||||
|
|
|
|||
|
|
@ -785,9 +785,16 @@ def load_account(
|
|||
legacy_fn: str = f'pps.{brokername}.{acctid}.toml'
|
||||
fn: str = f'account.{brokername}.{acctid}.toml'
|
||||
|
||||
dirpath: Path = dirpath or (config._config_dir / 'accounting')
|
||||
dirpath: Path = (
|
||||
dirpath
|
||||
or
|
||||
(config.get_conf_dir() / 'accounting')
|
||||
)
|
||||
if not dirpath.is_dir():
|
||||
dirpath.mkdir()
|
||||
dirpath.mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
|
||||
conf, path = config.load(
|
||||
path=dirpath / fn,
|
||||
|
|
|
|||
|
|
@ -80,12 +80,11 @@ async def _setup_persistent_brokerd(
|
|||
# any further (level) configuration on their own B)
|
||||
actor: tractor.Actor = tractor.current_actor()
|
||||
tll: str = actor.loglevel
|
||||
log = get_console_log(
|
||||
get_console_log(
|
||||
level=loglevel or tll,
|
||||
name=f'{_util.subsys}.{brokername}',
|
||||
with_tractor_log=bool(tll),
|
||||
)
|
||||
assert log.name == _util.subsys
|
||||
|
||||
# further, set the log level on any broker broker specific
|
||||
# logger instance.
|
||||
|
|
|
|||
|
|
@ -20,10 +20,17 @@ Handy cross-broker utils.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
# from functools import partial
|
||||
from typing import (
|
||||
Type,
|
||||
)
|
||||
|
||||
import json
|
||||
import httpx
|
||||
import logging
|
||||
from msgspec import Struct
|
||||
from tractor._exceptions import (
|
||||
reg_err_types,
|
||||
)
|
||||
|
||||
from piker.log import (
|
||||
colorize_json,
|
||||
|
|
@ -97,6 +104,19 @@ class DataThrottle(BrokerError):
|
|||
'''
|
||||
# TODO: add in throttle metrics/feedback
|
||||
|
||||
class SchemaMismatch(BrokerError):
|
||||
'''
|
||||
Market `Pair` fields mismatch, likely due to provider API update.
|
||||
|
||||
'''
|
||||
|
||||
# auto-register all `BrokerError` subtypes for
|
||||
# tractor IPC exc-marshalling.
|
||||
reg_err_types([
|
||||
BrokerError,
|
||||
*BrokerError.__subclasses__(),
|
||||
])
|
||||
|
||||
|
||||
def resproc(
|
||||
resp: httpx.Response,
|
||||
|
|
@ -123,3 +143,45 @@ def resproc(
|
|||
log.debug(f"Received json contents:\n{colorize_json(msg)}")
|
||||
|
||||
return msg if return_json else resp
|
||||
|
||||
|
||||
def get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type: Type[Struct],
|
||||
fields_data: dict,
|
||||
provider_name: str,
|
||||
api_url: str|None = None,
|
||||
) -> Struct:
|
||||
'''
|
||||
Boilerplate helper around assset-`Pair` field schema mismatches,
|
||||
normally due to provider API updates.
|
||||
|
||||
'''
|
||||
try:
|
||||
pair: Struct = pair_type(**fields_data)
|
||||
return pair
|
||||
except TypeError as err:
|
||||
|
||||
from tractor.devx.pformat import ppfmt
|
||||
repr_data: str = ppfmt(fields_data)
|
||||
report: str = (
|
||||
f'Field mismatch we need to codify!\n'
|
||||
f'\n'
|
||||
f'{pair_type!r}({repr_data})'
|
||||
f'\n'
|
||||
f'^^^ {err.args[0]!r} ^^^\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly {provider_name!r} "
|
||||
f"changed their symbology schema..\n"
|
||||
)
|
||||
if (
|
||||
api_url
|
||||
or
|
||||
(api_url := pair_type._api_url)
|
||||
):
|
||||
report += (
|
||||
f'\n'
|
||||
f'Check out their API docs here:\n'
|
||||
f'{api_url}\n'
|
||||
)
|
||||
|
||||
raise SchemaMismatch(report) from err
|
||||
|
|
|
|||
|
|
@ -49,6 +49,9 @@ from piker import config
|
|||
from piker.clearing._messages import (
|
||||
Order,
|
||||
)
|
||||
from piker.brokers._util import (
|
||||
get_or_raise_on_pair_schema_mismatch,
|
||||
)
|
||||
from piker.accounting import (
|
||||
Asset,
|
||||
digits_to_dec,
|
||||
|
|
@ -370,20 +373,12 @@ class Client:
|
|||
item['filters'] = filters
|
||||
|
||||
pair_type: Type = PAIRTYPES[venue]
|
||||
try:
|
||||
pair: Pair = pair_type(**item)
|
||||
except Exception as e:
|
||||
e.add_note(
|
||||
f'\n'
|
||||
f'New or removed field we need to codify!\n'
|
||||
f'pair-type: {pair_type!r}\n'
|
||||
f'\n'
|
||||
f"Don't panic, prolly stupid binance changed their symbology schema again..\n"
|
||||
f'Check out their API docs here:\n'
|
||||
f'\n'
|
||||
f'https://binance-docs.github.io/apidocs/spot/en/#exchange-information\n'
|
||||
)
|
||||
raise
|
||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type=pair_type,
|
||||
fields_data=item,
|
||||
provider_name='binance',
|
||||
api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
||||
)
|
||||
pair_table[pair.symbol.upper()] = pair
|
||||
|
||||
# update an additional top-level-cross-venue-table
|
||||
|
|
@ -581,8 +576,8 @@ class Client:
|
|||
self,
|
||||
mkt: MktPair,
|
||||
|
||||
start_dt: datetime | None = None,
|
||||
end_dt: datetime | None = None,
|
||||
start_dt: datetime|None = None,
|
||||
end_dt: datetime|None = None,
|
||||
|
||||
as_np: bool = True,
|
||||
|
||||
|
|
@ -609,7 +604,11 @@ class Client:
|
|||
start_time = binance_timestamp(start_dt)
|
||||
end_time = binance_timestamp(end_dt)
|
||||
|
||||
bs_pair: Pair = self._pairs[mkt.bs_fqme.upper()]
|
||||
import tractor
|
||||
with tractor.devx.maybe_open_crash_handler():
|
||||
bs_pair: Pair = self._pairs[
|
||||
mkt.bs_fqme.upper()
|
||||
]
|
||||
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data
|
||||
bars = await self._api(
|
||||
|
|
|
|||
|
|
@ -22,18 +22,14 @@ from __future__ import annotations
|
|||
from contextlib import (
|
||||
asynccontextmanager as acm,
|
||||
aclosing,
|
||||
AsyncExitStack,
|
||||
)
|
||||
from datetime import datetime
|
||||
from functools import (
|
||||
partial,
|
||||
)
|
||||
import itertools
|
||||
from pprint import pformat
|
||||
from typing import (
|
||||
Any,
|
||||
AsyncGenerator,
|
||||
Callable,
|
||||
Generator,
|
||||
)
|
||||
import time
|
||||
|
||||
|
|
@ -48,6 +44,7 @@ import tractor
|
|||
from piker.brokers import (
|
||||
open_cached_client,
|
||||
NoData,
|
||||
SymbolNotFound,
|
||||
)
|
||||
from piker._cacheables import (
|
||||
async_lifo_cache,
|
||||
|
|
@ -59,7 +56,10 @@ from piker.accounting import (
|
|||
unpack_fqme,
|
||||
)
|
||||
from piker.types import Struct
|
||||
from piker.data.validate import FeedInit
|
||||
from piker.data.validate import (
|
||||
FeedInit,
|
||||
FeedInitializationError,
|
||||
)
|
||||
from piker.data._web_bs import (
|
||||
open_autorecon_ws,
|
||||
NoBsWs,
|
||||
|
|
@ -76,13 +76,15 @@ from .venues import (
|
|||
Pair,
|
||||
FutesPair,
|
||||
get_api_eps,
|
||||
_futes_market_ws,
|
||||
_futes_public_ws,
|
||||
)
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
|
||||
class L1(Struct):
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#individual-symbol-book-ticker-streams
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/websocket-market-streams/Individual-Symbol-Book-Ticker-Streams
|
||||
|
||||
update_id: int
|
||||
sym: str
|
||||
|
|
@ -107,14 +109,15 @@ class AggTrade(Struct, frozen=True):
|
|||
T: int # Trade time
|
||||
m: bool # Is the buyer the market maker?
|
||||
M: bool|None = None # Ignore
|
||||
nq: float|None = None # Normal quantity without the trades involving RPI orders
|
||||
# ^XXX https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
nq: float|None = None # Normal quantity without RPI-order trades
|
||||
# ^XXX: https://developers.binance.com/docs/derivatives/change-log#2025-12-29
|
||||
st: int|None = None # 1 for USD-M, 2 for COIN-M
|
||||
|
||||
|
||||
async def stream_messages(
|
||||
ws: NoBsWs,
|
||||
|
||||
) -> AsyncGenerator[NoBsWs, dict]:
|
||||
) -> AsyncGenerator[tuple[str, dict], None]:
|
||||
|
||||
# TODO: match syntax here!
|
||||
msg: dict[str, Any]
|
||||
|
|
@ -203,27 +206,6 @@ async def stream_messages(
|
|||
yield 'trade', piker_quote
|
||||
|
||||
|
||||
def make_sub(pairs: list[str], sub_name: str, uid: int) -> dict[str, str]:
|
||||
'''
|
||||
Create a request subscription packet dict.
|
||||
|
||||
- spot:
|
||||
https://binance-docs.github.io/apidocs/spot/en/#live-subscribing-unsubscribing-to-streams
|
||||
|
||||
- futes:
|
||||
https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||
|
||||
'''
|
||||
return {
|
||||
'method': 'SUBSCRIBE',
|
||||
'params': [
|
||||
f'{pair.lower()}@{sub_name}'
|
||||
for pair in pairs
|
||||
],
|
||||
'id': uid
|
||||
}
|
||||
|
||||
|
||||
# TODO, why aren't frame resp `log.info()`s showing in upstream
|
||||
# code?!
|
||||
@acm
|
||||
|
|
@ -301,6 +283,10 @@ async def get_mkt_info(
|
|||
|
||||
# uppercase since kraken bs_mktid is always upper
|
||||
if 'binance' not in fqme.lower():
|
||||
log.warning(
|
||||
f'Missing `.<provider>` part in fqme ??\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
fqme += '.binance'
|
||||
|
||||
mkt_mode: str = ''
|
||||
|
|
@ -315,6 +301,12 @@ async def get_mkt_info(
|
|||
venue: str = venue.upper()
|
||||
venue_lower: str = venue.lower()
|
||||
|
||||
if not venue:
|
||||
raise SymbolNotFound(
|
||||
f'Invalid or missing .<venue> part in fqme?\n'
|
||||
f'fqme: {fqme!r}\n'
|
||||
)
|
||||
|
||||
# XXX TODO: we should change the usdtm_futes name to just
|
||||
# usdm_futes (dropping the tether part) since it turns out that
|
||||
# there are indeed USD-tokens OTHER THEN tether being used as
|
||||
|
|
@ -406,54 +398,153 @@ async def get_mkt_info(
|
|||
return mkt, pair
|
||||
|
||||
|
||||
@acm
|
||||
async def subscribe(
|
||||
ws: NoBsWs,
|
||||
def get_quote_routes(
|
||||
mkt_mode: str,
|
||||
symbols: list[str],
|
||||
) -> tuple[str, ...]:
|
||||
'''
|
||||
Render websocket URLs for each required Binance stream class.
|
||||
|
||||
# defined once at import time to keep a global state B)
|
||||
iter_subids: Generator[int, None, None] = itertools.count(),
|
||||
'''
|
||||
if len(symbols) != 1:
|
||||
raise ValueError(
|
||||
'Binance quote routes require exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
):
|
||||
# setup subs
|
||||
symbol: str = symbols[0]
|
||||
sub_names: tuple[str, str] = ('bookTicker', 'aggTrade')
|
||||
if mkt_mode == 'usdtm_futes':
|
||||
return tuple(
|
||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||
for wss_url, sub_name in (
|
||||
(_futes_public_ws, sub_names[0]),
|
||||
(_futes_market_ws, sub_names[1]),
|
||||
)
|
||||
)
|
||||
|
||||
subid: int = next(iter_subids)
|
||||
wss_url: str = get_api_eps(mkt_mode)[1]
|
||||
return tuple(
|
||||
f'{wss_url}/{symbol.lower()}@{sub_name}'
|
||||
for sub_name in sub_names
|
||||
)
|
||||
|
||||
# trade data (aka L1)
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#symbol-order-book-ticker
|
||||
l1_sub = make_sub(symbols, 'bookTicker', subid)
|
||||
await ws.send_msg(l1_sub)
|
||||
|
||||
# aggregate (each order clear by taker **not** by maker)
|
||||
# trades data:
|
||||
# https://binance-docs.github.io/apidocs/spot/en/#aggregate-trade-streams
|
||||
agg_trades_sub = make_sub(symbols, 'aggTrade', subid)
|
||||
await ws.send_msg(agg_trades_sub)
|
||||
async def relay_messages(
|
||||
ws: NoBsWs,
|
||||
send_chan: trio.MemorySendChannel,
|
||||
) -> None:
|
||||
'''
|
||||
Relay one Binance websocket into a merged quote channel.
|
||||
|
||||
# might get ack from ws server, or maybe some
|
||||
# other msg still in transit..
|
||||
res = await ws.recv_msg()
|
||||
subid: str|None = res.get('id')
|
||||
if subid:
|
||||
assert res['id'] == subid
|
||||
'''
|
||||
async with (
|
||||
send_chan,
|
||||
aclosing(stream_messages(ws)) as msg_gen,
|
||||
):
|
||||
async for msg in msg_gen:
|
||||
await send_chan.send(msg)
|
||||
|
||||
yield
|
||||
|
||||
subs = []
|
||||
for sym in symbols:
|
||||
subs.append("{sym}@aggTrade")
|
||||
subs.append("{sym}@bookTicker")
|
||||
# TODO: Encapsulate ``relay_messages()`` and ``open_quote_stream()``
|
||||
# inside ``stream_quotes()`` so async-generator closure is owned by
|
||||
# the backend feed task.
|
||||
@acm
|
||||
async def open_quote_stream(
|
||||
mkt_mode: str,
|
||||
symbols: list[str],
|
||||
connect_timeout: float = 16,
|
||||
) -> AsyncGenerator[trio.MemoryReceiveChannel, None]:
|
||||
'''
|
||||
Open and merge the websocket routes required by a market.
|
||||
|
||||
# unsub from all pairs on teardown
|
||||
if ws.connected():
|
||||
await ws.send_msg({
|
||||
"method": "UNSUBSCRIBE",
|
||||
"params": subs,
|
||||
"id": subid,
|
||||
})
|
||||
'''
|
||||
quote_routes: tuple[str, ...] = get_quote_routes(
|
||||
mkt_mode=mkt_mode,
|
||||
symbols=symbols,
|
||||
)
|
||||
|
||||
# XXX: do we need to ack the unsub?
|
||||
# await ws.recv_msg()
|
||||
send, recv = trio.open_memory_channel(616)
|
||||
async with AsyncExitStack() as stack:
|
||||
websockets: list[NoBsWs] = []
|
||||
try:
|
||||
for wss_url in quote_routes:
|
||||
ws: NoBsWs = await stack.enter_async_context(
|
||||
open_autorecon_ws(
|
||||
url=wss_url,
|
||||
connect_timeout=connect_timeout,
|
||||
)
|
||||
)
|
||||
websockets.append(ws)
|
||||
|
||||
except (
|
||||
TimeoutError,
|
||||
*NoBsWs.recon_errors,
|
||||
) as err:
|
||||
routes_str: str = '\n'.join(quote_routes)
|
||||
raise DataUnavailable(
|
||||
f'Binance websocket connection failed for '
|
||||
f'{symbols!r}:\n'
|
||||
f'{routes_str}'
|
||||
) from err
|
||||
|
||||
async with trio.open_nursery() as tn:
|
||||
for ws in websockets:
|
||||
tn.start_soon(
|
||||
relay_messages,
|
||||
ws,
|
||||
send.clone(),
|
||||
)
|
||||
|
||||
await send.aclose()
|
||||
async with recv:
|
||||
try:
|
||||
yield recv
|
||||
finally:
|
||||
tn.cancel_scope.cancel()
|
||||
|
||||
|
||||
async def wait_for_live_quotes(
|
||||
msg_stream: AsyncGenerator | trio.MemoryReceiveChannel,
|
||||
symbols: list[str],
|
||||
startup_timeout: float = 16,
|
||||
) -> dict:
|
||||
'''
|
||||
Require L1 or trade data before declaring a feed live.
|
||||
|
||||
'''
|
||||
if len(symbols) != 1:
|
||||
raise ValueError(
|
||||
'Binance quote readiness requires exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
symbol: str = symbols[0]
|
||||
msg_iter = aiter(msg_stream)
|
||||
|
||||
try:
|
||||
with trio.fail_after(startup_timeout):
|
||||
while True:
|
||||
typ, quote = await anext(msg_iter)
|
||||
if (
|
||||
typ in {'l1', 'trade'}
|
||||
and
|
||||
quote['symbol'] == symbol
|
||||
):
|
||||
return quote
|
||||
|
||||
except trio.TooSlowError as err:
|
||||
raise DataUnavailable(
|
||||
f'Binance quote startup timed out after '
|
||||
f'{startup_timeout}s; '
|
||||
f'no live data received for {symbol}'
|
||||
) from err
|
||||
|
||||
except StopAsyncIteration as err:
|
||||
raise DataUnavailable(
|
||||
f'Binance quote streams closed during startup; '
|
||||
f'no live data received for {symbol}'
|
||||
) from err
|
||||
|
||||
|
||||
async def stream_quotes(
|
||||
|
|
@ -465,49 +556,46 @@ async def stream_quotes(
|
|||
# startup sync
|
||||
task_status: TaskStatus[tuple[dict, dict]] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
) -> None:
|
||||
|
||||
if len(symbols) != 1:
|
||||
# ``data.feed.allocate_persistent_feed()`` currently starts
|
||||
# one backend task per symbol. Binance multiplexing remains
|
||||
# deferred until that caller contract can represent
|
||||
# multi-symbol startup.
|
||||
raise FeedInitializationError(
|
||||
'Binance feeds require exactly one symbol; '
|
||||
f'got {symbols!r}'
|
||||
)
|
||||
|
||||
async with (
|
||||
tractor.trionics.maybe_raise_from_masking_exc(),
|
||||
send_chan as send_chan,
|
||||
open_cached_client('binance') as client,
|
||||
):
|
||||
init_msgs: list[FeedInit] = []
|
||||
for sym in symbols:
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(sym)
|
||||
mkt: MktPair
|
||||
pair: Pair
|
||||
mkt, pair = await get_mkt_info(symbols[0])
|
||||
mkt_mode: str = (
|
||||
'usdtm_futes'
|
||||
if isinstance(pair, FutesPair)
|
||||
else 'spot'
|
||||
)
|
||||
|
||||
# build out init msgs according to latest spec
|
||||
init_msgs.append(
|
||||
FeedInit(mkt_info=mkt)
|
||||
# build out init msgs according to latest spec
|
||||
init_msgs: list[FeedInit] = [
|
||||
FeedInit(mkt_info=mkt),
|
||||
]
|
||||
|
||||
ws_symbols: list[str] = [mkt.bs_mktid]
|
||||
async with open_quote_stream(
|
||||
mkt_mode=mkt_mode,
|
||||
symbols=ws_symbols,
|
||||
) as msg_stream:
|
||||
quote: dict = await wait_for_live_quotes(
|
||||
msg_stream=msg_stream,
|
||||
symbols=ws_symbols,
|
||||
)
|
||||
|
||||
wss_url: str = get_api_eps(client.mkt_mode)[1] # 2nd elem is wss url
|
||||
|
||||
# TODO: for sanity, but remove eventually Xp
|
||||
if 'future' in mkt.type_key:
|
||||
assert 'fstream' in wss_url
|
||||
|
||||
async with (
|
||||
open_autorecon_ws(
|
||||
url=wss_url,
|
||||
fixture=partial(
|
||||
subscribe,
|
||||
symbols=[mkt.bs_mktid],
|
||||
),
|
||||
) as ws,
|
||||
|
||||
# avoid stream-gen closure from breaking trio..
|
||||
aclosing(stream_messages(ws)) as msg_gen,
|
||||
):
|
||||
# log.info('WAITING ON FIRST LIVE QUOTE..')
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
# pull a first quote and deliver
|
||||
while typ != 'trade':
|
||||
typ, quote = await anext(msg_gen)
|
||||
|
||||
task_status.started((init_msgs, quote))
|
||||
|
||||
# signal to caller feed is ready for consumption
|
||||
|
|
@ -523,7 +611,7 @@ async def stream_quotes(
|
|||
topic: str = mkt.bs_fqme
|
||||
|
||||
# start streaming
|
||||
async for typ, quote in msg_gen:
|
||||
async for typ, quote in msg_stream:
|
||||
# period = time.time() - last
|
||||
# hz = 1/period if period else float('inf')
|
||||
# if hz > 60:
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ Per market data-type definitions and schemas types.
|
|||
"""
|
||||
from __future__ import annotations
|
||||
from typing import (
|
||||
ClassVar,
|
||||
Literal,
|
||||
)
|
||||
from decimal import Decimal
|
||||
|
|
@ -41,7 +42,10 @@ _spot_ws: str = 'wss://stream.binance.com/ws'
|
|||
# or this one? ..
|
||||
# 'wss://ws-api.binance.com:443/ws-api/v3',
|
||||
|
||||
# https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams
|
||||
# https://developers.binance.com/docs/derivatives/usds-margined-futures/
|
||||
# websocket-market-streams/Important-WebSocket-Change-Notice
|
||||
_futes_public_ws: str = f'wss://fstream.{_domain}/public/ws'
|
||||
_futes_market_ws: str = f'wss://fstream.{_domain}/market/ws'
|
||||
_futes_ws: str = f'wss://fstream.{_domain}/ws'
|
||||
_auth_futes_ws: str = 'wss://fstream-auth.{_domain}/ws'
|
||||
|
||||
|
|
@ -78,7 +82,7 @@ def get_api_eps(venue: MarketType) -> tuple[str, str]:
|
|||
),
|
||||
'usdtm_futes': (
|
||||
_futes_url,
|
||||
_futes_ws,
|
||||
_futes_market_ws,
|
||||
),
|
||||
}[venue]
|
||||
|
||||
|
|
@ -203,6 +207,8 @@ class FutesPair(Pair):
|
|||
# NOTE: see `.data._symcache.SymbologyCache.load()` for why
|
||||
ns_path: str = 'piker.brokers.binance:FutesPair'
|
||||
|
||||
_api_url: ClassVar[str] = 'https://binance-docs.github.io/apidocs/spot/en/#exchange-information'
|
||||
|
||||
# NOTE: for compat with spot pairs and `MktPair.src: Asset`
|
||||
# processing..
|
||||
@property
|
||||
|
|
|
|||
|
|
@ -425,7 +425,7 @@ class DataFeed:
|
|||
async def stream_to_file(
|
||||
watchlist_name: str,
|
||||
filename: str,
|
||||
portal: tractor._portal.Portal,
|
||||
portal: tractor.Portal,
|
||||
tickers: List[str],
|
||||
brokermod: ModuleType,
|
||||
rate: int,
|
||||
|
|
|
|||
|
|
@ -23,7 +23,6 @@ from contextlib import (
|
|||
asynccontextmanager as acm,
|
||||
)
|
||||
from datetime import datetime
|
||||
from functools import partial
|
||||
import time
|
||||
from typing import (
|
||||
Any,
|
||||
|
|
@ -38,7 +37,7 @@ from rapidfuzz import process as fuzzy
|
|||
import numpy as np
|
||||
from tractor.trionics import (
|
||||
broadcast_receiver,
|
||||
maybe_open_context
|
||||
maybe_open_context,
|
||||
collapse_eg,
|
||||
)
|
||||
from tractor import to_asyncio
|
||||
|
|
@ -524,13 +523,12 @@ async def maybe_open_feed_handler() -> trio.abc.ReceiveStream:
|
|||
|
||||
|
||||
async def aio_price_feed_relay(
|
||||
chan: to_asyncio.LinkedTaskChannel,
|
||||
fh: FeedHandler,
|
||||
instrument: Symbol,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
) -> None:
|
||||
async def _trade(data: dict, receipt_timestamp):
|
||||
to_trio.send_nowait(('trade', {
|
||||
chan.send_nowait(('trade', {
|
||||
'symbol': cb_sym_to_deribit_inst(
|
||||
str_to_cb_sym(data.symbol)).lower(),
|
||||
'last': data,
|
||||
|
|
@ -540,7 +538,7 @@ async def aio_price_feed_relay(
|
|||
}))
|
||||
|
||||
async def _l1(data: dict, receipt_timestamp):
|
||||
to_trio.send_nowait(('l1', {
|
||||
chan.send_nowait(('l1', {
|
||||
'symbol': cb_sym_to_deribit_inst(
|
||||
str_to_cb_sym(data.symbol)).lower(),
|
||||
'ticks': [
|
||||
|
|
@ -570,7 +568,7 @@ async def aio_price_feed_relay(
|
|||
install_signal_handlers=False)
|
||||
|
||||
# sync with trio
|
||||
to_trio.send_nowait(None)
|
||||
chan.started_nowait(None)
|
||||
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
|
|
@ -581,11 +579,9 @@ async def open_price_feed(
|
|||
) -> trio.abc.ReceiveStream:
|
||||
async with maybe_open_feed_handler() as fh:
|
||||
async with to_asyncio.open_channel_from(
|
||||
partial(
|
||||
aio_price_feed_relay,
|
||||
fh,
|
||||
instrument
|
||||
)
|
||||
aio_price_feed_relay,
|
||||
fh=fh,
|
||||
instrument=instrument,
|
||||
) as (chan, first):
|
||||
yield chan
|
||||
|
||||
|
|
@ -611,10 +607,9 @@ async def maybe_open_price_feed(
|
|||
|
||||
|
||||
async def aio_order_feed_relay(
|
||||
chan: to_asyncio.LinkedTaskChannel,
|
||||
fh: FeedHandler,
|
||||
instrument: Symbol,
|
||||
from_trio: asyncio.Queue,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
) -> None:
|
||||
async def _fill(data: dict, receipt_timestamp):
|
||||
breakpoint()
|
||||
|
|
@ -637,7 +632,7 @@ async def aio_order_feed_relay(
|
|||
install_signal_handlers=False)
|
||||
|
||||
# sync with trio
|
||||
to_trio.send_nowait(None)
|
||||
chan.started_nowait(None)
|
||||
|
||||
await asyncio.sleep(float('inf'))
|
||||
|
||||
|
|
@ -648,11 +643,9 @@ async def open_order_feed(
|
|||
) -> trio.abc.ReceiveStream:
|
||||
async with maybe_open_feed_handler() as fh:
|
||||
async with to_asyncio.open_channel_from(
|
||||
partial(
|
||||
aio_order_feed_relay,
|
||||
fh,
|
||||
instrument
|
||||
)
|
||||
aio_order_feed_relay,
|
||||
fh=fh,
|
||||
instrument=instrument,
|
||||
) as (chan, first):
|
||||
yield chan
|
||||
|
||||
|
|
|
|||
|
|
@ -1584,6 +1584,72 @@ class MethodProxy:
|
|||
self.event_table = event_table
|
||||
self._aio_ns = asyncio_ns
|
||||
|
||||
# request-id counter for correlating each method
|
||||
# call to its (eventual) response; necessary since
|
||||
# the (single, shared) chan has NO other ordering
|
||||
# guarantee whenever a caller task is CANCELLED
|
||||
# (eg. by a search-req timeout) after sending its
|
||||
# request but before consuming the response: the
|
||||
# "orphaned" response otherwise gets mis-delivered
|
||||
# to the next caller causing an off-by-one skew of
|
||||
# every result thereafter!
|
||||
self._mids = itertools.count()
|
||||
self._pending: dict[
|
||||
int,
|
||||
tuple[trio.Event, list[dict]],
|
||||
] = {}
|
||||
self._request_methods: dict[int, str|None] = {}
|
||||
|
||||
def _deliver_method_response(
|
||||
self,
|
||||
msg: dict,
|
||||
|
||||
) -> bool:
|
||||
'''
|
||||
Deliver one correlated response to its waiting caller.
|
||||
|
||||
'''
|
||||
mid: int = msg['mid']
|
||||
meth: str|None = self._request_methods.pop(mid, None)
|
||||
pending = self._pending.get(mid)
|
||||
if pending is None:
|
||||
log.warning(
|
||||
f'Dropping stale method-resp,\n'
|
||||
f'mid: {mid}\n'
|
||||
f'meth: {meth!r}\n'
|
||||
f'(caller prolly got cancelled '
|
||||
f'before its resp?)\n'
|
||||
)
|
||||
return False
|
||||
|
||||
event, slot = pending
|
||||
slot.append(msg)
|
||||
event.set()
|
||||
return True
|
||||
|
||||
def _track_request_method(
|
||||
self,
|
||||
mid: int,
|
||||
meth: str|None,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Retain bounded request metadata for stale-response logs.
|
||||
|
||||
'''
|
||||
self._request_methods[mid] = meth
|
||||
if len(self._request_methods) > 256:
|
||||
oldest_mid: int = next(iter(self._request_methods))
|
||||
oldest_meth: str|None = self._request_methods.pop(
|
||||
oldest_mid
|
||||
)
|
||||
log.warning(
|
||||
f'Evicting unresolved method-request metadata,\n'
|
||||
f'mid: {oldest_mid}\n'
|
||||
f'meth: {oldest_meth!r}\n'
|
||||
f'pending metadata limit: 256\n'
|
||||
)
|
||||
|
||||
async def _run_method(
|
||||
self,
|
||||
*,
|
||||
|
|
@ -1596,38 +1662,30 @@ class MethodProxy:
|
|||
``tractor.to_asyncio`` layer.
|
||||
|
||||
'''
|
||||
chan = self.chan
|
||||
await chan.send((meth, kwargs))
|
||||
mid: int = next(self._mids)
|
||||
event = trio.Event()
|
||||
slot: list[dict] = []
|
||||
self._pending[mid] = event, slot
|
||||
self._track_request_method(mid, meth)
|
||||
try:
|
||||
await self.chan.send((meth, kwargs, mid))
|
||||
await event.wait()
|
||||
finally:
|
||||
self._pending.pop(mid, None)
|
||||
|
||||
while not chan.closed():
|
||||
# send through method + ``kwargs: dict`` as pair
|
||||
msg = await chan.receive()
|
||||
|
||||
# TODO: implement reconnect functionality like
|
||||
# in our `.data._web_bs.NoBsWs`
|
||||
# try:
|
||||
# msg = await chan.receive()
|
||||
# except ConnectionError:
|
||||
# self.reset()
|
||||
|
||||
# print(f'NEXT MSG: {msg}')
|
||||
|
||||
# TODO: py3.10 ``match:`` syntax B)
|
||||
if 'result' in msg:
|
||||
res = msg.get('result')
|
||||
msg: dict = slot[0]
|
||||
match msg:
|
||||
# OUR method-call response B)
|
||||
case {'result': res}:
|
||||
return res
|
||||
|
||||
elif 'exception' in msg:
|
||||
err = msg.get('exception')
|
||||
case {'exception': err}:
|
||||
raise err
|
||||
|
||||
elif 'error' in msg:
|
||||
etype, emsg = msg
|
||||
log.warning(f'IB error relay: {emsg}')
|
||||
continue
|
||||
|
||||
else:
|
||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||
case _:
|
||||
raise RuntimeError(
|
||||
f'Invalid method response for mid={mid}: {msg!r}'
|
||||
)
|
||||
|
||||
def status_event(
|
||||
self,
|
||||
|
|
@ -1637,7 +1695,7 @@ class MethodProxy:
|
|||
|
||||
ev = self.event_table.get(pattern)
|
||||
|
||||
if not ev or ev.is_set():
|
||||
if ev is None or ev.is_set():
|
||||
# print(f'inserting new data reset event item')
|
||||
ev = self.event_table[pattern] = trio.Event()
|
||||
|
||||
|
|
@ -1680,22 +1738,31 @@ async def open_aio_client_method_relay(
|
|||
log.info('asyncio `Client` method-proxy SHUTDOWN!')
|
||||
break
|
||||
|
||||
case (meth_name, kwargs):
|
||||
meth_name, kwargs = msg
|
||||
case (meth_name, kwargs, mid):
|
||||
meth = getattr(client, meth_name)
|
||||
|
||||
try:
|
||||
resp = await meth(**kwargs)
|
||||
# echo the msg back
|
||||
chan.send_nowait({'result': resp})
|
||||
# echo the msg back tagged with the
|
||||
# req-id for caller-side correlation.
|
||||
chan.send_nowait({
|
||||
'mid': mid,
|
||||
'result': resp,
|
||||
})
|
||||
|
||||
except (
|
||||
RequestError,
|
||||
|
||||
# TODO: relay all errors to trio?
|
||||
# BaseException,
|
||||
) as err:
|
||||
chan.send_nowait({'exception': err})
|
||||
# XXX: relay ALL (non-cancel) errors to the
|
||||
# `trio`-side caller (which re-raises in the
|
||||
# `MethodProxy._run_method()` frame) instead
|
||||
# of crashing this relay task and thus the
|
||||
# whole proxy/dialog; critical post the
|
||||
# (datad|brokerd)-split where eg. contract
|
||||
# qualification failures are expected to be
|
||||
# caught per-request by order/warmup code!
|
||||
except Exception as err:
|
||||
chan.send_nowait({
|
||||
'mid': mid,
|
||||
'exception': err,
|
||||
})
|
||||
|
||||
case {'error': content}:
|
||||
chan.send_nowait({'exception': content})
|
||||
|
|
@ -1704,6 +1771,73 @@ async def open_aio_client_method_relay(
|
|||
raise ValueError(f'Unhandled msg {msg}')
|
||||
|
||||
|
||||
async def relay_client_proxy_messages(
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
proxy: MethodProxy,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Route one proxy's asyncio-client msgs from a single reader.
|
||||
|
||||
Each `open_client_proxy()` allocates its own linked channel and
|
||||
starts this task in its relay nursery. Any number of Trio caller
|
||||
tasks may concurrently write tagged requests through
|
||||
`MethodProxy._run_method()` and wait on their per-`mid` slots.
|
||||
This task alone reads the channel and dispatches each response to
|
||||
its waiting caller.
|
||||
|
||||
Separate proxy contexts own separate channels and reader tasks;
|
||||
no cross-proxy response broadcast is intended. IB status events
|
||||
still reach each proxy through its own `Client.inline_errors()`
|
||||
handler.
|
||||
|
||||
'''
|
||||
try:
|
||||
while not chan.closed():
|
||||
# TODO: implement reconnect functionality like
|
||||
# in our `.data._web_bs.NoBsWs`
|
||||
# try:
|
||||
msg = await chan.receive()
|
||||
# except ConnectionError:
|
||||
# proxy.reset()
|
||||
|
||||
match msg:
|
||||
# Correlated response from a method call.
|
||||
case {'mid': _}:
|
||||
proxy._deliver_method_response(msg)
|
||||
|
||||
# routine (api-farm conn) status events
|
||||
# relayed inline by `Client.inline_errors()`;
|
||||
# not a response to a method call so just
|
||||
# log at info and keep waiting.
|
||||
case ('event', status_msg):
|
||||
reason = status_msg['reason']
|
||||
event = proxy.event_table.pop(reason, None)
|
||||
if (
|
||||
event is not None
|
||||
and
|
||||
event.statistics().tasks_waiting
|
||||
):
|
||||
log.info(f'Relaying ib status message: {msg}')
|
||||
event.set()
|
||||
|
||||
# Inline API error with no request correlation.
|
||||
case ('error', emsg):
|
||||
log.warning(f'IB error relay: {emsg}')
|
||||
|
||||
# Out-of-band client error with no method `mid`.
|
||||
case {'exception': err}:
|
||||
log.error(
|
||||
f'Uncorrelated IB client error: {err!r}'
|
||||
)
|
||||
|
||||
# Preserve visibility into unexpected relay traffic.
|
||||
case _:
|
||||
log.warning(f'UNKNOWN IB MSG: {msg}')
|
||||
except trio.EndOfChannel:
|
||||
log.info('IB client proxy relay closed')
|
||||
|
||||
|
||||
@acm
|
||||
async def open_client_proxy(
|
||||
client: Client,
|
||||
|
|
@ -1740,28 +1874,11 @@ async def open_client_proxy(
|
|||
continue
|
||||
setattr(proxy, name, partial(proxy._run_method, meth=name))
|
||||
|
||||
async def relay_events():
|
||||
|
||||
async with chan.subscribe() as msg_stream:
|
||||
|
||||
async for msg in msg_stream:
|
||||
if 'event' not in msg:
|
||||
continue
|
||||
|
||||
# if 'event' in msg:
|
||||
# wake up any system event waiters.
|
||||
etype, status_msg = msg
|
||||
reason = status_msg['reason']
|
||||
|
||||
ev = proxy.event_table.pop(reason, None)
|
||||
|
||||
if ev and ev.statistics().tasks_waiting:
|
||||
log.info(f'Relaying ib status message: {msg}')
|
||||
ev.set()
|
||||
|
||||
continue
|
||||
|
||||
relay_tn.start_soon(relay_events)
|
||||
relay_tn.start_soon(
|
||||
relay_client_proxy_messages,
|
||||
chan,
|
||||
proxy,
|
||||
)
|
||||
|
||||
yield proxy
|
||||
|
||||
|
|
|
|||
|
|
@ -231,20 +231,21 @@ async def handle_order_requests(
|
|||
|
||||
async def recv_trade_updates(
|
||||
|
||||
chan: tractor.to_asyncio.LinkedTaskChannel,
|
||||
client: Client,
|
||||
to_trio: trio.abc.SendChannel,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Receive and relay order control and positioning related events
|
||||
from `ib_async`, pack as tuples and push over mem-chan to our
|
||||
trio relay task for processing and relay to EMS.
|
||||
Receive and relay order control and positioning
|
||||
related events from `ib_async`, pack as tuples and
|
||||
push over mem-chan to our trio relay task for
|
||||
processing and relay to EMS.
|
||||
|
||||
'''
|
||||
client.inline_errors(to_trio)
|
||||
client.inline_errors(chan)
|
||||
|
||||
# sync with trio task
|
||||
to_trio.send_nowait(client.ib)
|
||||
chan.started_nowait(client.ib)
|
||||
|
||||
def push_tradesies(
|
||||
eventkit_obj,
|
||||
|
|
@ -282,7 +283,7 @@ async def recv_trade_updates(
|
|||
|
||||
try:
|
||||
# emit event name + relevant ibis internal objects
|
||||
to_trio.send_nowait((event_name, emit))
|
||||
chan.send_nowait((event_name, emit))
|
||||
except trio.BrokenResourceError:
|
||||
log.exception(f'Disconnected from {eventkit_obj} updates')
|
||||
eventkit_obj.disconnect(push_tradesies)
|
||||
|
|
|
|||
|
|
@ -32,14 +32,12 @@ import time
|
|||
from typing import (
|
||||
Any,
|
||||
Callable,
|
||||
TYPE_CHECKING,
|
||||
)
|
||||
|
||||
from async_generator import aclosing
|
||||
import ib_async as ibis
|
||||
import numpy as np
|
||||
from pendulum import (
|
||||
now,
|
||||
from_timestamp,
|
||||
Duration,
|
||||
duration as mk_duration,
|
||||
|
|
@ -75,9 +73,6 @@ from ._util import (
|
|||
)
|
||||
from .symbols import get_mkt_info
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from trio._core._run import Task
|
||||
|
||||
log = get_logger(
|
||||
name=__name__,
|
||||
)
|
||||
|
|
@ -225,7 +220,11 @@ async def open_history_client(
|
|||
f'mean: {mean}'
|
||||
)
|
||||
|
||||
# could be trying to retreive bars over weekend
|
||||
# get_bars() uses None internally for an empty reply.
|
||||
# The backfiller never receives that sentinel: translate
|
||||
# it into NoData, or DataUnavailable at the known start
|
||||
# of the instrument's history. A venue closure can also
|
||||
# yield an empty reply without exhausting all history.
|
||||
if out is None:
|
||||
log.error(
|
||||
f"No bars starting at {end_dt!r} !?!?"
|
||||
|
|
@ -353,6 +352,7 @@ async def wait_on_data_reset(
|
|||
tuple[
|
||||
trio.CancelScope,
|
||||
trio.Event,
|
||||
list[bool],
|
||||
]
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
) -> bool:
|
||||
|
|
@ -396,8 +396,9 @@ async def wait_on_data_reset(
|
|||
client: Client = proxy._aio_ns
|
||||
|
||||
done = trio.Event()
|
||||
result: list[bool] = []
|
||||
with trio.move_on_after(timeout) as cs:
|
||||
task_status.started((cs, done))
|
||||
task_status.started((cs, done, result))
|
||||
|
||||
log.warning(
|
||||
'Sending DATA RESET request:\n'
|
||||
|
|
@ -413,6 +414,7 @@ async def wait_on_data_reset(
|
|||
'NO VNC DETECTED!\n'
|
||||
'Manually press ctrl-alt-f on your IB java app'
|
||||
)
|
||||
result.append(False)
|
||||
done.set()
|
||||
return False
|
||||
|
||||
|
|
@ -427,6 +429,7 @@ async def wait_on_data_reset(
|
|||
]:
|
||||
await ev.wait()
|
||||
log.info(f"{name} DATA RESET")
|
||||
result.append(True)
|
||||
done.set()
|
||||
return True
|
||||
|
||||
|
|
@ -435,12 +438,12 @@ async def wait_on_data_reset(
|
|||
'Data reset task canceled?'
|
||||
)
|
||||
|
||||
result.append(False)
|
||||
done.set()
|
||||
return False
|
||||
|
||||
|
||||
_data_resetter_task: Task | None = None
|
||||
_failed_resets: int = 0
|
||||
_data_reset_lock = trio.Lock()
|
||||
|
||||
|
||||
async def get_bars(
|
||||
|
|
@ -456,10 +459,6 @@ async def get_bars(
|
|||
# how long before we trigger a feed reset (seconds)
|
||||
feed_reset_timeout: float = 3,
|
||||
|
||||
# how many days to subtract before giving up on further
|
||||
# history queries for instrument, presuming that most don't
|
||||
# not trade for a week XD
|
||||
max_nodatas: int = 6,
|
||||
max_failed_resets: int = 6,
|
||||
|
||||
task_status: TaskStatus[trio.CancelScope] = trio.TASK_STATUS_IGNORED,
|
||||
|
|
@ -479,8 +478,10 @@ async def get_bars(
|
|||
`.ib.api.Client` methods.
|
||||
|
||||
'''
|
||||
global _data_resetter_task, _failed_resets
|
||||
nodatas_count: int = 0
|
||||
cancelled_count: int = 0
|
||||
failed_resets: int = 0
|
||||
pacing_reset_pending: bool = False
|
||||
pacing_reset_completed: bool = False
|
||||
|
||||
data_cs: trio.CancelScope | None = None
|
||||
result: tuple[
|
||||
|
|
@ -493,10 +494,16 @@ async def get_bars(
|
|||
|
||||
async def query():
|
||||
|
||||
global _failed_resets
|
||||
nonlocal result, data_cs, end_dt, nodatas_count
|
||||
nonlocal result, data_cs
|
||||
nonlocal cancelled_count, failed_resets
|
||||
nonlocal pacing_reset_pending, pacing_reset_completed
|
||||
|
||||
while _failed_resets < max_failed_resets:
|
||||
while True:
|
||||
if failed_resets >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB history reset failed {failed_resets} '
|
||||
f'times for {fqme}'
|
||||
)
|
||||
try:
|
||||
(
|
||||
bars,
|
||||
|
|
@ -528,6 +535,10 @@ async def get_bars(
|
|||
)
|
||||
# NOTE: REQUIRED to pass back value..
|
||||
result = None
|
||||
failed_resets = 0
|
||||
result_ready.set()
|
||||
if data_cs:
|
||||
data_cs.cancel()
|
||||
return None
|
||||
|
||||
# not enough bars signal, likely due to venue
|
||||
|
|
@ -587,6 +598,7 @@ async def get_bars(
|
|||
first_dt,
|
||||
last_dt,
|
||||
)
|
||||
failed_resets = 0
|
||||
|
||||
# signal data reset loop parent task
|
||||
result_ready.set()
|
||||
|
|
@ -611,27 +623,21 @@ async def get_bars(
|
|||
# the frame dt index since the upper layer may
|
||||
# be doing so concurrently and we don't want to
|
||||
# be delivering frames that weren't asked for.
|
||||
# try to decrement start point and look further back
|
||||
# end_dt = end_dt.subtract(seconds=2000)
|
||||
logmsg = "SUBTRACTING DAY from DT index"
|
||||
if end_dt is not None:
|
||||
end_dt = end_dt.subtract(days=1)
|
||||
elif end_dt is None:
|
||||
end_dt = now().subtract(days=1)
|
||||
|
||||
log.warning(
|
||||
f'NO DATA found ending @ {end_dt}\n'
|
||||
+ logmsg
|
||||
)
|
||||
|
||||
if nodatas_count >= max_nodatas:
|
||||
raise DataUnavailable(
|
||||
f'Presuming {fqme} has no further history '
|
||||
f'after {max_nodatas} tries..'
|
||||
)
|
||||
|
||||
nodatas_count += 1
|
||||
continue
|
||||
# Match get_bars()'s empty-bars return: None is
|
||||
# internal to the IB adapter, not its backfill
|
||||
# protocol. open_history_client() converts it
|
||||
# to NoData (or DataUnavailable at history start).
|
||||
# Wake the waiting get_bars() task before exiting
|
||||
# so it does not mistake this reply for a timeout.
|
||||
result = None
|
||||
failed_resets = 0
|
||||
result_ready.set()
|
||||
if data_cs:
|
||||
data_cs.cancel()
|
||||
return None
|
||||
|
||||
elif (
|
||||
'API historical data query cancelled'
|
||||
|
|
@ -642,6 +648,12 @@ async def get_bars(
|
|||
'Query cancelled by IB (:eyeroll:):\n'
|
||||
f'{err.message}'
|
||||
)
|
||||
cancelled_count += 1
|
||||
if cancelled_count >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB cancelled {cancelled_count} history '
|
||||
f'queries for {fqme}'
|
||||
)
|
||||
continue
|
||||
|
||||
elif (
|
||||
|
|
@ -662,6 +674,7 @@ async def get_bars(
|
|||
)
|
||||
|
||||
client = proxy._aio_ns.ib.client
|
||||
pacing_reset_pending = True
|
||||
|
||||
# cancel any existing reset task
|
||||
if data_cs:
|
||||
|
|
@ -669,69 +682,81 @@ async def get_bars(
|
|||
data_cs.cancel()
|
||||
|
||||
# spawn new data reset task
|
||||
data_cs, reset_done = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='connection'
|
||||
async with _data_reset_lock:
|
||||
(
|
||||
data_cs,
|
||||
reset_done,
|
||||
reset_result,
|
||||
) = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='connection'
|
||||
)
|
||||
)
|
||||
)
|
||||
if reset_done:
|
||||
_failed_resets = 0
|
||||
await reset_done.wait()
|
||||
if reset_result[0]:
|
||||
failed_resets = 0
|
||||
else:
|
||||
_failed_resets += 1
|
||||
failed_resets += 1
|
||||
pacing_reset_pending = False
|
||||
pacing_reset_completed = True
|
||||
continue
|
||||
|
||||
else:
|
||||
raise
|
||||
|
||||
# TODO: make this global across all history task/requests
|
||||
# such that simultaneous symbol queries don't try data resettingn
|
||||
# too fast..
|
||||
unset_resetter: bool = False
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as tn
|
||||
):
|
||||
|
||||
# start history request that we allow
|
||||
# to run indefinitely until a result is acquired
|
||||
# Run history until a result or bounded reset failure.
|
||||
tn.start_soon(query)
|
||||
|
||||
# start history reset loop which waits up to the timeout
|
||||
# for a result before triggering a data feed reset.
|
||||
# Serialize farm resets across every history requester.
|
||||
while not result_ready.is_set():
|
||||
|
||||
with trio.move_on_after(feed_reset_timeout):
|
||||
await result_ready.wait()
|
||||
break
|
||||
|
||||
if _data_resetter_task:
|
||||
# don't double invoke the reset hack if another
|
||||
# requester task already has it covered.
|
||||
continue
|
||||
async with _data_reset_lock:
|
||||
if result_ready.is_set():
|
||||
break
|
||||
if pacing_reset_completed:
|
||||
pacing_reset_completed = False
|
||||
continue
|
||||
|
||||
else:
|
||||
_data_resetter_task = trio.lowlevel.current_task()
|
||||
unset_resetter: bool = True
|
||||
|
||||
# spawn new data reset task
|
||||
data_cs, reset_done = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='data',
|
||||
(
|
||||
data_cs,
|
||||
reset_done,
|
||||
reset_result,
|
||||
) = await tn.start(
|
||||
partial(
|
||||
wait_on_data_reset,
|
||||
proxy,
|
||||
reset_type='data',
|
||||
)
|
||||
)
|
||||
)
|
||||
# sync wait on reset to complete
|
||||
await reset_done.wait()
|
||||
await reset_done.wait()
|
||||
|
||||
_data_resetter_task = (
|
||||
None
|
||||
if unset_resetter
|
||||
else _data_resetter_task
|
||||
)
|
||||
assert result
|
||||
if (
|
||||
result_ready.is_set()
|
||||
or
|
||||
reset_result[0]
|
||||
):
|
||||
failed_resets = 0
|
||||
elif pacing_reset_pending:
|
||||
# Pacing recovery deliberately cancelled this reset
|
||||
# and will consume the retry itself.
|
||||
pass
|
||||
else:
|
||||
failed_resets += 1
|
||||
if failed_resets >= max_failed_resets:
|
||||
raise DataUnavailable(
|
||||
f'IB history reset failed '
|
||||
f'{failed_resets} times for {fqme}'
|
||||
)
|
||||
return (
|
||||
result,
|
||||
data_cs is not None,
|
||||
|
|
|
|||
|
|
@ -52,6 +52,7 @@ from piker.brokers._util import (
|
|||
SymbolNotFound,
|
||||
BrokerError,
|
||||
DataThrottle,
|
||||
get_or_raise_on_pair_schema_mismatch,
|
||||
)
|
||||
from piker.accounting import Transaction
|
||||
from piker.log import get_logger
|
||||
|
|
@ -117,6 +118,9 @@ class InvalidKey(ValueError):
|
|||
|
||||
'''
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([InvalidKey])
|
||||
|
||||
|
||||
class Client:
|
||||
|
||||
|
|
@ -502,7 +506,16 @@ class Client:
|
|||
|
||||
# NOTE: always cache in pairs tables for faster lookup
|
||||
with tractor.devx.maybe_open_crash_handler(): # as bxerr:
|
||||
pair = Pair(xname=xkey, **data)
|
||||
# pair = Pair(xname=xkey, **data)
|
||||
pair: Pair = get_or_raise_on_pair_schema_mismatch(
|
||||
pair_type=Pair,
|
||||
fields_data=dict(
|
||||
xname=xkey,
|
||||
**data,
|
||||
),
|
||||
provider_name='kraken',
|
||||
# api_url='https://binance-docs.github.io/apidocs/spot/en/#exchange-information',
|
||||
)
|
||||
|
||||
# register the above `Pair` structs for all
|
||||
# key-sets/monikers: a set of 4 (frickin) tables
|
||||
|
|
|
|||
|
|
@ -97,6 +97,9 @@ MsgUnion = Union[
|
|||
class TooFastEdit(Exception):
|
||||
'Edit requests faster then api submissions'
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([TooFastEdit])
|
||||
|
||||
|
||||
# TODO: make this wrap the `Client` and `ws` instances
|
||||
# and give it methods to submit cancel vs. add vs. edit
|
||||
|
|
|
|||
|
|
@ -19,6 +19,9 @@ Symbology defs and search.
|
|||
|
||||
'''
|
||||
from decimal import Decimal
|
||||
from typing import (
|
||||
ClassVar,
|
||||
)
|
||||
|
||||
import tractor
|
||||
|
||||
|
|
@ -86,9 +89,14 @@ class Pair(Struct):
|
|||
short_position_limit: float = 0
|
||||
long_position_limit: float = float('inf')
|
||||
|
||||
# TODO, add API note when this was added!
|
||||
execution_venue: str|None = None
|
||||
|
||||
# TODO: should we make this a literal NamespacePath ref?
|
||||
ns_path: str = 'piker.brokers.kraken:Pair'
|
||||
|
||||
_api_url: ClassVar[str] = 'https://docs.kraken.com/api/docs/rest-api/get-tradable-asset-pairs'
|
||||
|
||||
@property
|
||||
def bs_mktid(self) -> str:
|
||||
'''
|
||||
|
|
|
|||
|
|
@ -95,6 +95,9 @@ _time_frames = {
|
|||
class QuestradeError(Exception):
|
||||
"Non-200 OK response code"
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([QuestradeError])
|
||||
|
||||
|
||||
class ContractsKey(NamedTuple):
|
||||
symbol: str
|
||||
|
|
|
|||
|
|
@ -159,11 +159,11 @@ class DarkBook(Struct):
|
|||
_DEFAULT_SIZE: float = 1.0
|
||||
|
||||
|
||||
async def clear_dark_triggers(
|
||||
async def _clear_dark_triggers(
|
||||
|
||||
router: Router,
|
||||
brokerd_orders_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.MsgStream,
|
||||
quote_stream: trionics.AsyncReceiver,
|
||||
broker: str,
|
||||
fqme: str,
|
||||
|
||||
|
|
@ -182,7 +182,6 @@ async def clear_dark_triggers(
|
|||
# - port to the new ringbuf stuff in `tractor.ipc`!
|
||||
# - numba all this!
|
||||
# - this stream may eventually contain multiple symbols
|
||||
quote_stream._raise_on_lag = False
|
||||
async for quotes in quote_stream:
|
||||
# start = time.time()
|
||||
for sym, quote in quotes.items():
|
||||
|
|
@ -314,6 +313,32 @@ async def clear_dark_triggers(
|
|||
# print(f'execs scan took: {time.time() - start}')
|
||||
|
||||
|
||||
async def clear_dark_triggers(
|
||||
router: Router,
|
||||
brokerd_orders_stream: tractor.MsgStream,
|
||||
quote_stream: tractor.MsgStream,
|
||||
broker: str,
|
||||
fqme: str,
|
||||
book: DarkBook,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Run dark clearing on a dedicated non-strict quote subscription.
|
||||
|
||||
'''
|
||||
async with quote_stream.subscribe(
|
||||
raise_on_lag=False,
|
||||
) as quotes_stream:
|
||||
await _clear_dark_triggers(
|
||||
router,
|
||||
brokerd_orders_stream,
|
||||
quotes_stream,
|
||||
broker,
|
||||
fqme,
|
||||
book,
|
||||
)
|
||||
|
||||
|
||||
class TradesRelay(Struct):
|
||||
|
||||
# for now we keep only a single connection open with
|
||||
|
|
@ -785,11 +810,10 @@ async def _setup_persistent_emsd(
|
|||
) -> None:
|
||||
|
||||
if loglevel:
|
||||
_log = get_console_log(
|
||||
get_console_log(
|
||||
level=loglevel,
|
||||
name=subsys,
|
||||
)
|
||||
assert _log.name == 'piker.clearing'
|
||||
|
||||
global _router
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ from types import ModuleType
|
|||
import click
|
||||
import trio
|
||||
import tractor
|
||||
from tractor._multiaddr import parse_maddr
|
||||
from tractor.discovery import parse_endpoints
|
||||
|
||||
from ..log import (
|
||||
get_console_log,
|
||||
|
|
@ -46,48 +46,27 @@ log = get_logger('piker.cli')
|
|||
|
||||
|
||||
def load_trans_eps(
|
||||
network: dict | None = None,
|
||||
maddrs: list[tuple] | None = None,
|
||||
network: dict|None = None,
|
||||
maddrs: str|tuple[str, ...]|list[str]|None = None,
|
||||
|
||||
) -> dict[str, dict[str, dict]]:
|
||||
) -> dict[str, list]:
|
||||
'''
|
||||
Load transport endpoints from a `[network]` config
|
||||
table or CLI-provided multiaddr strings, delegating
|
||||
to `tractor.discovery.parse_endpoints()`.
|
||||
|
||||
# transport-oriented endpoint multi-addresses
|
||||
eps: dict[
|
||||
str, # service name, eg. `pikerd`, `emsd`..
|
||||
|
||||
# libp2p style multi-addresses parsed into prot layers
|
||||
list[dict[str, str | int]]
|
||||
] = {}
|
||||
|
||||
if (
|
||||
network
|
||||
and
|
||||
not maddrs
|
||||
):
|
||||
# load network section and (attempt to) connect all endpoints
|
||||
# which are reachable B)
|
||||
for key, maddrs in network.items():
|
||||
match key:
|
||||
|
||||
# TODO: resolve table across multiple discov
|
||||
# prots Bo
|
||||
case 'resolv':
|
||||
pass
|
||||
|
||||
case 'pikerd':
|
||||
dname: str = key
|
||||
for maddr in maddrs:
|
||||
layers: dict = parse_maddr(maddr)
|
||||
eps.setdefault(
|
||||
dname,
|
||||
[],
|
||||
).append(layers)
|
||||
'''
|
||||
if network and not maddrs:
|
||||
return parse_endpoints(network)
|
||||
|
||||
elif maddrs:
|
||||
# presume user is manually specifying the root actor ep.
|
||||
eps['pikerd'] = [parse_maddr(maddr)]
|
||||
if isinstance(maddrs, str):
|
||||
maddrs = [maddrs]
|
||||
return parse_endpoints(
|
||||
{'pikerd': list(maddrs)}
|
||||
)
|
||||
|
||||
return eps
|
||||
return {}
|
||||
|
||||
|
||||
@click.command()
|
||||
|
|
@ -108,13 +87,13 @@ def load_trans_eps(
|
|||
help='Enable tractor debug mode',
|
||||
)
|
||||
@click.option(
|
||||
'--maddr',
|
||||
'--maddrs',
|
||||
'-m',
|
||||
default=None,
|
||||
multiple=True,
|
||||
help='Multiaddrs to bind or contact',
|
||||
)
|
||||
def pikerd(
|
||||
maddr: list[str] | None,
|
||||
maddrs: tuple[str, ...],
|
||||
loglevel: str,
|
||||
tl: bool,
|
||||
pdb: bool,
|
||||
|
|
@ -145,7 +124,8 @@ def pikerd(
|
|||
))
|
||||
|
||||
# service-actor registry endpoint socket-address set
|
||||
regaddrs: list[tuple[str, int]] = []
|
||||
regaddrs: list = []
|
||||
tpt_bind_addrs: list|None = None
|
||||
|
||||
conf, _ = config.load(
|
||||
conf_name='conf',
|
||||
|
|
@ -153,7 +133,8 @@ def pikerd(
|
|||
network: dict = conf.get('network')
|
||||
if (
|
||||
network is None
|
||||
and not maddr
|
||||
and
|
||||
not maddrs
|
||||
):
|
||||
regaddrs = [(
|
||||
_default_registry_host,
|
||||
|
|
@ -161,15 +142,23 @@ def pikerd(
|
|||
)]
|
||||
|
||||
else:
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddr,
|
||||
)
|
||||
for layers in eps['pikerd']:
|
||||
regaddrs.append((
|
||||
layers['ipv4']['addr'],
|
||||
layers['tcp']['port'],
|
||||
))
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
with maybe_open_crash_handler(pdb=pdb):
|
||||
eps: dict[str, list] = load_trans_eps(
|
||||
network,
|
||||
maddrs,
|
||||
)
|
||||
|
||||
# pikerd bind addresses (may differ from
|
||||
# registry if regd is separate)
|
||||
tpt_bind_addrs = eps.get('pikerd')
|
||||
|
||||
# registry: dedicated `regd` key, or fall
|
||||
# back to pikerd addrs (pikerd IS the registry)
|
||||
regaddrs = eps.get(
|
||||
'regd',
|
||||
tpt_bind_addrs,
|
||||
)
|
||||
|
||||
from .. import service
|
||||
|
||||
|
|
@ -178,6 +167,7 @@ def pikerd(
|
|||
async with (
|
||||
service.open_pikerd(
|
||||
registry_addrs=regaddrs,
|
||||
tpt_bind_addrs=tpt_bind_addrs,
|
||||
loglevel=loglevel,
|
||||
debug_mode=pdb,
|
||||
# enable_transports=['uds'],
|
||||
|
|
@ -345,7 +335,7 @@ def services(
|
|||
if not ports:
|
||||
ports: list[int] = [_default_registry_port]
|
||||
|
||||
addr = tractor._addr.wrap_address(
|
||||
addr = tractor.discovery._addr.wrap_address(
|
||||
addr=(host, ports[0])
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,7 @@ except ModuleNotFoundError:
|
|||
import tomli as tomllib
|
||||
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
from .log import get_logger
|
||||
|
||||
log = get_logger('broker-config')
|
||||
|
|
@ -172,6 +173,12 @@ class ConfigurationError(Exception):
|
|||
class NoSignature(ConfigurationError):
|
||||
'No credentials setup for broker backend!'
|
||||
|
||||
# auto-register for tractor IPC exc-marshalling.
|
||||
reg_err_types([
|
||||
ConfigurationError,
|
||||
*ConfigurationError.__subclasses__(),
|
||||
])
|
||||
|
||||
|
||||
def _override_config_dir(
|
||||
path: str
|
||||
|
|
@ -180,6 +187,48 @@ def _override_config_dir(
|
|||
_config_dir = path
|
||||
|
||||
|
||||
def _maybe_use_test_dir() -> None:
|
||||
'''
|
||||
When running under the `pytest` harness, override
|
||||
the config dir to the per-test-tmp dir "passed down"
|
||||
the actor tree via `tractor`'s runtime-vars
|
||||
inheritance mechanism.
|
||||
|
||||
See the `tractor_runtime_overrides` usage in our
|
||||
`tests.conftest._open_test_pikerd()` as well as
|
||||
`.service._actor_runtime.open_piker_runtime()` for
|
||||
the root-actor's pre-loading of the var state.
|
||||
|
||||
NOTE: this must be checked lazily at config-path
|
||||
access time (NOT import time) since sub-actors only
|
||||
receive runtime-vars once their `tractor` runtime
|
||||
has fully booted.
|
||||
|
||||
'''
|
||||
global _config_dir
|
||||
import tractor
|
||||
actor = tractor.current_actor(
|
||||
err_on_no_runtime=False,
|
||||
)
|
||||
if actor is None:
|
||||
return
|
||||
|
||||
rvs: dict = tractor.runtime._state._runtime_vars
|
||||
pvars: dict|None = rvs.get('piker_vars')
|
||||
if (
|
||||
pvars
|
||||
and
|
||||
(testdir := pvars.get('piker_test_dir'))
|
||||
):
|
||||
testdirpath = Path(testdir)
|
||||
assert testdirpath.exists(), (
|
||||
f'piker test harness might be borked!?\n'
|
||||
f'testdirpath: {testdirpath!r}\n'
|
||||
)
|
||||
if _config_dir != testdirpath:
|
||||
_override_config_dir(testdirpath)
|
||||
|
||||
|
||||
def _conf_fn_w_ext(
|
||||
name: str,
|
||||
) -> str:
|
||||
|
|
@ -193,6 +242,7 @@ def get_conf_dir() -> Path:
|
|||
on the local filesystem.
|
||||
|
||||
'''
|
||||
_maybe_use_test_dir()
|
||||
return _config_dir
|
||||
|
||||
|
||||
|
|
@ -218,7 +268,7 @@ def get_conf_path(
|
|||
assert str(conf_name) in _conf_names
|
||||
|
||||
fn = _conf_fn_w_ext(conf_name)
|
||||
return _config_dir / Path(fn)
|
||||
return get_conf_dir() / Path(fn)
|
||||
|
||||
|
||||
def repodir() -> Path:
|
||||
|
|
@ -263,8 +313,9 @@ def load(
|
|||
|
||||
'''
|
||||
# create the $HOME/.config/piker dir if dne
|
||||
if not _config_dir.is_dir():
|
||||
_config_dir.mkdir(
|
||||
conf_dir: Path = get_conf_dir()
|
||||
if not conf_dir.is_dir():
|
||||
conf_dir.mkdir(
|
||||
parents=True,
|
||||
exist_ok=True,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -775,7 +775,7 @@ async def sample_and_broadcast(
|
|||
# channel-ipc-pipe is still up.
|
||||
if (
|
||||
not chan.connected()
|
||||
or ctx._cancel_called
|
||||
or ctx.cancel_called
|
||||
):
|
||||
log.warning(
|
||||
'Dropping broken consumer:\n'
|
||||
|
|
|
|||
|
|
@ -194,15 +194,24 @@ async def _reconnect_forever(
|
|||
'''
|
||||
# after so many msg recv timeouts, reset the connection
|
||||
timeouts: int = 0
|
||||
first_message: bool = True
|
||||
|
||||
while True:
|
||||
waiting_on: str = 'websocket receive'
|
||||
timeout: float = nobsws._timeout
|
||||
with trio.move_on_after(
|
||||
# can be dynamically changed by user code
|
||||
nobsws._timeout,
|
||||
timeout,
|
||||
) as cs:
|
||||
try:
|
||||
msg: Any = await ws.get_message()
|
||||
waiting_on = 'local relay send'
|
||||
await snd.send(msg)
|
||||
if first_message:
|
||||
log.info(
|
||||
f'Websocket first message relayed: {url!r}'
|
||||
)
|
||||
first_message = False
|
||||
except nobsws.recon_errors:
|
||||
log.exception(
|
||||
f'{src_mod}\n'
|
||||
|
|
@ -218,10 +227,17 @@ async def _reconnect_forever(
|
|||
|
||||
if cs.cancelled_caught:
|
||||
timeouts += 1
|
||||
log.warning(
|
||||
f'Websocket timeout waiting on {waiting_on}\n'
|
||||
f'url: {url!r}\n'
|
||||
f'timeout: {timeout}s; count: {timeouts}; '
|
||||
f'reset after: {reset_after}\n'
|
||||
)
|
||||
if timeouts > reset_after:
|
||||
log.error(
|
||||
f'{src_mod}\n'
|
||||
'WS feed seems down and slow af.. reconnecting\n'
|
||||
f'WS feed seems down and slow af.. reconnecting\n'
|
||||
f'{url!r}\n'
|
||||
)
|
||||
rent_cs.cancel()
|
||||
|
||||
|
|
@ -261,8 +277,8 @@ async def _reconnect_forever(
|
|||
ws: WebSocketConnection
|
||||
try:
|
||||
async with (
|
||||
open_websocket_url(url) as ws,
|
||||
tractor.trionics.collapse_eg(),
|
||||
open_websocket_url(url) as ws,
|
||||
trio.open_nursery() as tn,
|
||||
):
|
||||
cs = nobsws._cs = tn.cancel_scope
|
||||
|
|
@ -295,24 +311,38 @@ async def _reconnect_forever(
|
|||
|
||||
# indicate to wrapper / opener that we are up and block
|
||||
# to let tasks run **inside** the ws open block above.
|
||||
log.info(f'Websocket ready: {url!r}')
|
||||
nobsws._connected.set()
|
||||
await trio.sleep_forever()
|
||||
|
||||
except (
|
||||
HandshakeError,
|
||||
ConnectionRejected,
|
||||
):
|
||||
log.exception('Retrying connection')
|
||||
except nobsws.recon_errors:
|
||||
# The websocket reader can fail while its context exits,
|
||||
# outside proxy_msgs() (e.g. Pong during LOCAL_CLOSING).
|
||||
# Collapse its nursery group before classifying the error.
|
||||
nobsws._connected = trio.Event()
|
||||
nobsws._ws = None
|
||||
# Unlike the old outer wrapper suppression, restart the
|
||||
# connection. This checkpoint also honors parent shutdown;
|
||||
# never shield a reconnect retry from cancellation.
|
||||
log.exception(f'Retrying connection: {url!r}')
|
||||
await trio.sleep(0.5) # throttle
|
||||
continue
|
||||
|
||||
except BaseException as _berr:
|
||||
berr = _berr
|
||||
# ``trio.Cancelled`` controls task-tree teardown; it is not a
|
||||
# reconnect failure. Let it escape before ``BaseException``
|
||||
# so nursery cancellation never enters the shielded 200ms
|
||||
# error throttle. Audited by
|
||||
# ``test_reconnect_cancellation_bypasses_error_throttle()``.
|
||||
except trio.Cancelled:
|
||||
raise
|
||||
|
||||
except BaseException:
|
||||
log.exception(
|
||||
'Reconnect-attempt failed ??\n'
|
||||
)
|
||||
with trio.CancelScope(shield=True):
|
||||
await trio.sleep(0.2) # throttle
|
||||
raise berr
|
||||
raise
|
||||
|
||||
#|_ws & nursery block ends
|
||||
nobsws._connected = trio.Event()
|
||||
|
|
@ -323,7 +353,7 @@ async def _reconnect_forever(
|
|||
# if wrapper cancelled us, we expect it to also
|
||||
# have re-assigned a new event
|
||||
assert (
|
||||
nobsws._connected
|
||||
nobsws._connected is not None
|
||||
and not nobsws._connected.is_set()
|
||||
)
|
||||
|
||||
|
|
@ -342,6 +372,7 @@ async def open_autorecon_ws(
|
|||
url: str,
|
||||
|
||||
fixture: AsyncContextManager | None = None,
|
||||
connect_timeout: float | None = None,
|
||||
|
||||
# time in sec between msgs received before
|
||||
# we presume connection might need a reset.
|
||||
|
|
@ -391,7 +422,19 @@ async def open_autorecon_ws(
|
|||
reset_after=reset_after,
|
||||
)
|
||||
)
|
||||
await nobsws._connected.wait()
|
||||
try:
|
||||
if connect_timeout is None:
|
||||
await nobsws._connected.wait()
|
||||
else:
|
||||
with trio.fail_after(connect_timeout):
|
||||
await nobsws._connected.wait()
|
||||
|
||||
except trio.TooSlowError as err:
|
||||
raise TimeoutError(
|
||||
f'Websocket connection timed out after '
|
||||
f'{connect_timeout}s: {url}'
|
||||
) from err
|
||||
|
||||
assert nobsws._cs
|
||||
assert nobsws.connected()
|
||||
try:
|
||||
|
|
@ -404,6 +447,7 @@ async def open_autorecon_ws(
|
|||
f'Entire ws-channel disconnect due to,\n'
|
||||
f'con_err: {con_err!r}\n'
|
||||
)
|
||||
raise
|
||||
|
||||
|
||||
'''
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ from collections import (
|
|||
defaultdict,
|
||||
abc,
|
||||
)
|
||||
from copy import copy
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from functools import partial
|
||||
import time
|
||||
|
|
@ -77,7 +78,7 @@ from ._sampling import (
|
|||
|
||||
if TYPE_CHECKING:
|
||||
from .flows import Flume
|
||||
from tractor._addr import Address
|
||||
from tractor.discovery._addr import Address
|
||||
from tractor.msg.types import Aid
|
||||
|
||||
|
||||
|
|
@ -484,14 +485,15 @@ async def open_feed_bus(
|
|||
if loglevel is None:
|
||||
loglevel: str = tractor.current_actor().loglevel
|
||||
|
||||
# XXX: required to propagate ``tractor`` loglevel to piker
|
||||
# logging
|
||||
# Configure the shared piker.data logger used by ._util.log,
|
||||
# including websocket tasks. A per-client verbosity must not
|
||||
# demote actor-wide transport diagnostics below daemon verbosity.
|
||||
get_console_log(
|
||||
level=(loglevel
|
||||
level=(tractor.current_actor().loglevel
|
||||
or
|
||||
tractor.current_actor().loglevel
|
||||
loglevel
|
||||
),
|
||||
name=__name__,
|
||||
name=log.name,
|
||||
)
|
||||
|
||||
# local state sanity checks
|
||||
|
|
@ -674,6 +676,36 @@ async def open_feed_bus(
|
|||
bus.remove_subs(bs_fqme, subs)
|
||||
|
||||
|
||||
async def relay_to_common_memchan(
|
||||
stream: (
|
||||
tractor.MsgStream
|
||||
|trionics.BroadcastReceiver
|
||||
),
|
||||
tx: trio.MemorySendChannel[dict[str, Any]],
|
||||
task_status: TaskStatus[
|
||||
trionics.BroadcastReceiver,
|
||||
] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> None:
|
||||
'''
|
||||
Relay one provider subscription into a common memory channel.
|
||||
|
||||
Deliver the live child receiver through `task_status` so the
|
||||
caller can retain and inspect each provider relay's cursor.
|
||||
|
||||
'''
|
||||
bstream: trionics.BroadcastReceiver
|
||||
async with (
|
||||
stream.subscribe() as bstream,
|
||||
tx,
|
||||
):
|
||||
task_status.started(bstream)
|
||||
|
||||
msg: dict[str, Any]
|
||||
async for msg in bstream:
|
||||
await tx.send(msg)
|
||||
|
||||
|
||||
class Feed(Struct):
|
||||
'''
|
||||
A per-provider API for client-side consumption from real-time data
|
||||
|
|
@ -723,29 +755,51 @@ class Feed(Struct):
|
|||
if len(mods) == 1:
|
||||
# just pass the brokerd stream directly if only one provider
|
||||
# was detected.
|
||||
stream = self.streams[list(brokers)[0]]
|
||||
stream: (
|
||||
tractor.MsgStream
|
||||
|trionics.BroadcastReceiver
|
||||
) = self.streams[list(brokers)[0]]
|
||||
bstream: trionics.BroadcastReceiver
|
||||
async with stream.subscribe() as bstream:
|
||||
yield bstream
|
||||
return
|
||||
|
||||
# start multiplexing task tree
|
||||
tx: trio.MemorySendChannel[dict[str, Any]]
|
||||
rx: trio.MemoryReceiveChannel[dict[str, Any]]
|
||||
tx, rx = trio.open_memory_channel(616)
|
||||
|
||||
async def relay_to_common_memchan(stream: tractor.MsgStream):
|
||||
async with tx:
|
||||
async for msg in stream:
|
||||
await tx.send(msg)
|
||||
|
||||
async with (
|
||||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as nurse
|
||||
):
|
||||
# spawn a relay task for each stream so that they all
|
||||
# multiplex to a common channel.
|
||||
for brokername in mods:
|
||||
stream = self.streams[brokername]
|
||||
nurse.start_soon(relay_to_common_memchan, stream)
|
||||
broker2bstreams: dict[
|
||||
str,
|
||||
trionics.BroadcastReceiver,
|
||||
] = {}
|
||||
broker: str
|
||||
for broker in mods:
|
||||
stream: (
|
||||
tractor.MsgStream
|
||||
|trionics.BroadcastReceiver
|
||||
) = self.streams[broker]
|
||||
|
||||
relay_tx: trio.MemorySendChannel[
|
||||
dict[str, Any]
|
||||
] = tx.clone()
|
||||
|
||||
bstream: trionics.BroadcastReceiver = await nurse.start(
|
||||
relay_to_common_memchan,
|
||||
stream,
|
||||
relay_tx,
|
||||
)
|
||||
broker2bstreams[broker] = bstream
|
||||
|
||||
# Each relay now owns one `tx` clone. Closing the original
|
||||
# lets `rx` reach EOC only after every provider relay exits.
|
||||
await tx.aclose()
|
||||
try:
|
||||
yield rx
|
||||
finally:
|
||||
|
|
@ -754,10 +808,23 @@ class Feed(Struct):
|
|||
_max_sample_rate: int = 1
|
||||
|
||||
async def pause(self) -> None:
|
||||
'''
|
||||
Pause the shared remote provider context for this cached feed.
|
||||
|
||||
This controls provider publication for every local cache user;
|
||||
it does not pause only this caller's receive cursor.
|
||||
|
||||
'''
|
||||
stream: tractor.MsgStream|trionics.BroadcastReceiver
|
||||
for stream in set(self.streams.values()):
|
||||
await stream.send('pause')
|
||||
|
||||
async def resume(self) -> None:
|
||||
'''
|
||||
Resume the shared remote provider context for this cached feed.
|
||||
|
||||
'''
|
||||
stream: tractor.MsgStream|trionics.BroadcastReceiver
|
||||
for stream in set(self.streams.values()):
|
||||
await stream.send('resume')
|
||||
|
||||
|
|
@ -817,7 +884,17 @@ async def maybe_open_feed(
|
|||
in a tractor broadcast receiver.
|
||||
|
||||
'''
|
||||
fqme = fqmes[0]
|
||||
cache_key: tuple[
|
||||
tuple[str, ...],
|
||||
float|None,
|
||||
bool,
|
||||
bool,
|
||||
] = (
|
||||
tuple(sorted(fqmes)),
|
||||
kwargs.get('tick_throttle'),
|
||||
kwargs.get('allow_overruns', True),
|
||||
kwargs.get('start_stream', True),
|
||||
)
|
||||
|
||||
async with trionics.maybe_open_context(
|
||||
acm_func=open_feed,
|
||||
|
|
@ -830,27 +907,73 @@ async def maybe_open_feed(
|
|||
'allow_overruns': kwargs.get('allow_overruns', True),
|
||||
'start_stream': kwargs.get('start_stream', True),
|
||||
},
|
||||
key=fqme,
|
||||
key=cache_key,
|
||||
|
||||
) as (cache_hit, feed):
|
||||
feed: Feed
|
||||
|
||||
if cache_hit:
|
||||
log.info(f'Using cached feed for {fqme}')
|
||||
# add a new broadcast subscription for the quote stream
|
||||
# if this feed is likely already in use
|
||||
if (
|
||||
cache_hit
|
||||
and
|
||||
log.at_least_level('info')
|
||||
):
|
||||
log.info(
|
||||
f'Using cached feed for key:\n'
|
||||
f'{cache_key!r}'
|
||||
)
|
||||
|
||||
async with trionics.gather_contexts(
|
||||
mngrs=[stream.subscribe() for stream in feed.streams.values()]
|
||||
) as bstreams:
|
||||
for bstream, flume in zip(bstreams, feed.flumes.values()):
|
||||
# XXX: TODO: horrible hackery that needs fixing..
|
||||
# i guess we have to create context proxies?
|
||||
bstream._ctx = flume.stream._ctx
|
||||
flume.stream = bstream
|
||||
# The cached `Feed` exclusively owns provider root streams.
|
||||
# Every caller, including the first cache owner, gets a local
|
||||
# child cursor and copied descriptors for its lexical lifetime.
|
||||
broker_names: tuple[str, ...] = tuple(feed.streams)
|
||||
bstreams: tuple[
|
||||
trionics.BroadcastReceiver,
|
||||
...,
|
||||
]
|
||||
async with trionics.gather_contexts(
|
||||
mngrs=[
|
||||
feed.streams[broker].subscribe()
|
||||
for broker in broker_names
|
||||
]
|
||||
) as bstreams:
|
||||
broker2bstreams: dict[
|
||||
str,
|
||||
trionics.BroadcastReceiver,
|
||||
] = dict(zip(
|
||||
broker_names,
|
||||
bstreams,
|
||||
strict=True,
|
||||
))
|
||||
flumes: dict[str, Flume] = {}
|
||||
fqme: str
|
||||
flume: Flume
|
||||
for fqme, flume in feed.flumes.items():
|
||||
broker: str = flume.mkt.broker
|
||||
|
||||
yield feed
|
||||
else:
|
||||
yield feed
|
||||
bstream: trionics.BroadcastReceiver = (
|
||||
broker2bstreams[broker]
|
||||
)
|
||||
|
||||
# XXX: TODO: horrible hackery that needs fixing..
|
||||
# i guess we have to create context proxies?
|
||||
root_stream: tractor.MsgStream = feed.streams[broker]
|
||||
|
||||
ctx: tractor.Context = root_stream.ctx
|
||||
bstream._ctx = ctx
|
||||
|
||||
local_flume: Flume = copy(flume)
|
||||
local_flume.stream = bstream
|
||||
flumes[fqme] = local_flume
|
||||
|
||||
# `feed` is the cache-owned resource and must retain only
|
||||
# provider root streams. A shallow `Feed` copy gives this
|
||||
# caller its own descriptor mappings while intentionally
|
||||
# sharing immutable market metadata, SHM handles and
|
||||
# provider controls for the cache resource's lifetime.
|
||||
local_feed: Feed = copy(feed)
|
||||
local_feed.flumes = flumes
|
||||
local_feed.streams = broker2bstreams
|
||||
yield local_feed
|
||||
|
||||
|
||||
@acm
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ from piker.accounting import (
|
|||
Asset,
|
||||
MktPair,
|
||||
)
|
||||
from tractor._exceptions import reg_err_types
|
||||
from ._util import log
|
||||
|
||||
|
||||
|
|
@ -43,6 +44,8 @@ class FeedInitializationError(ValueError):
|
|||
|
||||
'''
|
||||
|
||||
reg_err_types([FeedInitializationError])
|
||||
|
||||
|
||||
class FeedInit(Struct, frozen=True):
|
||||
'''
|
||||
|
|
|
|||
33
piker/log.py
33
piker/log.py
|
|
@ -36,6 +36,15 @@ from pygments import (
|
|||
_proj_name: str = 'piker'
|
||||
|
||||
|
||||
def _relative_name(
|
||||
name: str|None,
|
||||
) -> str|None:
|
||||
if name:
|
||||
return name.removeprefix(f'{_proj_name}.')
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_logger(
|
||||
name: str|None = None,
|
||||
**tractor_log_kwargs,
|
||||
|
|
@ -47,17 +56,9 @@ def get_logger(
|
|||
See `tractor.log.get_logger()` for details.
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
return tractor.log.get_logger(
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
name=_relative_name(name),
|
||||
pkg_name=_proj_name,
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
|
||||
|
|
@ -81,14 +82,6 @@ def get_console_log(
|
|||
You do it.. Bp
|
||||
|
||||
'''
|
||||
pkg_name: str = _proj_name
|
||||
if (
|
||||
name
|
||||
and
|
||||
pkg_name in name
|
||||
):
|
||||
name: str = name.lstrip(f'{_proj_name}.')
|
||||
|
||||
tll: str|None = None
|
||||
if (
|
||||
with_tractor_log is not False
|
||||
|
|
@ -111,8 +104,8 @@ def get_console_log(
|
|||
|
||||
return tractor.log.get_console_log(
|
||||
level=level,
|
||||
name=name,
|
||||
pkg_name=pkg_name,
|
||||
name=_relative_name(name),
|
||||
pkg_name=_proj_name,
|
||||
**tractor_log_kwargs,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ from ._registry import ( # noqa
|
|||
_tractor_kwargs,
|
||||
_default_reg_addr,
|
||||
open_registry,
|
||||
RegistryAddress,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -58,7 +59,8 @@ def get_runtime_vars() -> dict[str, Any]:
|
|||
@acm
|
||||
async def open_piker_runtime(
|
||||
name: str,
|
||||
registry_addrs: list[tuple[str, int]] = [],
|
||||
registry_addrs: list[RegistryAddress] = [],
|
||||
tpt_bind_addrs: list|None = None,
|
||||
|
||||
enable_modules: list[str] = [],
|
||||
loglevel: str|None = None,
|
||||
|
|
@ -76,7 +78,7 @@ async def open_piker_runtime(
|
|||
|
||||
) -> tuple[
|
||||
tractor.Actor,
|
||||
list[tuple[str, int]],
|
||||
list[RegistryAddress],
|
||||
]:
|
||||
'''
|
||||
Start a piker actor who's runtime will automatically sync with
|
||||
|
|
@ -91,7 +93,7 @@ async def open_piker_runtime(
|
|||
try:
|
||||
actor = tractor.current_actor()
|
||||
except tractor._exceptions.NoRuntime:
|
||||
tractor._state._runtime_vars[
|
||||
tractor.runtime._state._runtime_vars[
|
||||
'piker_vars'
|
||||
] = tractor_runtime_overrides
|
||||
|
||||
|
|
@ -112,6 +114,7 @@ async def open_piker_runtime(
|
|||
|
||||
# passed through to `open_root_actor`
|
||||
registry_addrs=registry_addrs,
|
||||
tpt_bind_addrs=tpt_bind_addrs,
|
||||
name=name,
|
||||
start_method=start_method,
|
||||
loglevel=loglevel,
|
||||
|
|
@ -165,7 +168,8 @@ _root_modules: list[str] = [
|
|||
|
||||
@acm
|
||||
async def open_pikerd(
|
||||
registry_addrs: list[tuple[str, int]],
|
||||
registry_addrs: list[RegistryAddress],
|
||||
tpt_bind_addrs: list|None = None,
|
||||
loglevel: str|None = None,
|
||||
|
||||
# XXX: you should pretty much never want debug mode
|
||||
|
|
@ -198,6 +202,7 @@ async def open_pikerd(
|
|||
loglevel=loglevel,
|
||||
debug_mode=debug_mode,
|
||||
registry_addrs=registry_addrs,
|
||||
tpt_bind_addrs=tpt_bind_addrs,
|
||||
|
||||
**kwargs,
|
||||
|
||||
|
|
@ -209,8 +214,44 @@ async def open_pikerd(
|
|||
tractor.trionics.collapse_eg(),
|
||||
trio.open_nursery() as service_tn,
|
||||
):
|
||||
for addr in reg_addrs:
|
||||
if addr not in root_actor.accept_addrs:
|
||||
# `reg_addrs` locates the registrar used for actor discovery;
|
||||
# `tpt_bind_addrs` requests this daemon's own IPC listeners.
|
||||
# An existing registrar can live elsewhere, so its endpoints
|
||||
# must not be checked against `root_actor.accept_addrs`.
|
||||
bind_addrs = tpt_bind_addrs
|
||||
if bind_addrs is None:
|
||||
# Only a registrar actor owns the discovery endpoints.
|
||||
# A client with no explicit bind request lets Tractor pick
|
||||
# its listener; there is no caller-specified bind to check.
|
||||
bind_addrs = (
|
||||
reg_addrs if root_actor.is_registrar else []
|
||||
)
|
||||
|
||||
# `root_actor.accept_addrs` contains the actual bound endpoints.
|
||||
# Normalize both sides with Tractor's address API so legacy
|
||||
# pairs and tagged transport tuples compare in the same form.
|
||||
accept_addrs = [
|
||||
tractor.discovery._addr.wrap_address(addr).unwrap()
|
||||
for addr in root_actor.accept_addrs
|
||||
]
|
||||
for addr in bind_addrs:
|
||||
# normalize to a wrapped `tractor` addr-type;
|
||||
# entries may be raw `tuple`s when passed in
|
||||
# from (test) client code.
|
||||
wladdr = tractor.discovery._addr.wrap_address(
|
||||
addr,
|
||||
)
|
||||
uaddr: tuple = wladdr.unwrap()
|
||||
bound = uaddr in accept_addrs
|
||||
if wladdr.proto_key == 'tcp' and uaddr[-1] == 0:
|
||||
# TCP port zero asks the OS for an ephemeral listener.
|
||||
# Its actual port differs from the request, but the
|
||||
# tagged transport and host must still match exactly.
|
||||
bound = any(
|
||||
actual[:-1] == uaddr[:-1]
|
||||
for actual in accept_addrs
|
||||
)
|
||||
if not bound:
|
||||
raise RuntimeError(
|
||||
f'`pikerd` failed to bind on {addr}!\n'
|
||||
'Maybe you have another daemon already running?'
|
||||
|
|
@ -258,13 +299,13 @@ async def open_pikerd(
|
|||
|
||||
@acm
|
||||
async def maybe_open_pikerd(
|
||||
registry_addrs: list[tuple[str, int]] | None = None,
|
||||
registry_addrs: list[RegistryAddress] | None = None,
|
||||
|
||||
loglevel: str | None = None,
|
||||
**kwargs,
|
||||
|
||||
) -> (
|
||||
tractor._portal.Portal
|
||||
tractor.Portal
|
||||
|ClassVar[Services]
|
||||
):
|
||||
'''
|
||||
|
|
@ -292,7 +333,7 @@ async def maybe_open_pikerd(
|
|||
# async with open_portal(chan) as arb_portal:
|
||||
# yield arb_portal
|
||||
|
||||
registry_addrs: list[tuple[str, int]] = (
|
||||
registry_addrs: list[RegistryAddress] = (
|
||||
registry_addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
|
|
|
|||
|
|
@ -49,6 +49,7 @@ from requests.exceptions import (
|
|||
ReadTimeout,
|
||||
)
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
from piker.log import (
|
||||
get_console_log,
|
||||
get_logger,
|
||||
|
|
@ -66,6 +67,11 @@ class DockerNotStarted(Exception):
|
|||
class ApplicationLogError(Exception):
|
||||
'App in container reported an error in logs'
|
||||
|
||||
reg_err_types([
|
||||
DockerNotStarted,
|
||||
ApplicationLogError,
|
||||
])
|
||||
|
||||
|
||||
@acm
|
||||
async def open_docker(
|
||||
|
|
|
|||
|
|
@ -72,7 +72,6 @@ async def maybe_spawn_daemon(
|
|||
level=loglevel,
|
||||
name=__name__,
|
||||
)
|
||||
assert log.name == 'piker.service'
|
||||
|
||||
# serialize access to this section to avoid
|
||||
# 2 or more tasks racing to create a daemon
|
||||
|
|
@ -80,9 +79,21 @@ async def maybe_spawn_daemon(
|
|||
await lock.acquire()
|
||||
|
||||
try:
|
||||
# Discovery and `maybe_open_pikerd()` must use the same
|
||||
# registrar list, independently of other runtime options.
|
||||
# Consume an override here: startup forwards it explicitly
|
||||
# below, so leaving it in `pikerd_kwargs` duplicates the keyword.
|
||||
registry_addrs = pikerd_kwargs.pop('registry_addrs', None)
|
||||
if registry_addrs is None:
|
||||
# Use the registrar addresses inherited by this actor.
|
||||
rtvs: dict = tractor.get_runtime_vars()
|
||||
registry_addrs: list[tuple] = list(
|
||||
map(tuple, rtvs['_registry_addrs'])
|
||||
)
|
||||
|
||||
async with find_service(
|
||||
service_name,
|
||||
registry_addrs=[('127.0.0.1', 6116)],
|
||||
registry_addrs=registry_addrs,
|
||||
) as portal:
|
||||
if portal is not None:
|
||||
lock.release()
|
||||
|
|
@ -99,6 +110,7 @@ async def maybe_spawn_daemon(
|
|||
# process tree
|
||||
async with maybe_open_pikerd(
|
||||
loglevel=loglevel,
|
||||
registry_addrs=registry_addrs,
|
||||
**pikerd_kwargs,
|
||||
|
||||
) as pikerd_portal:
|
||||
|
|
@ -142,7 +154,65 @@ async def maybe_spawn_daemon(
|
|||
async with tractor.wait_for_actor(service_name) as portal:
|
||||
lock.release()
|
||||
yield portal
|
||||
await portal.cancel_actor()
|
||||
# --- ---- ---
|
||||
# XXX NOTE XXX
|
||||
# --- ---- ---
|
||||
# DO NOT PUT A `portal.cancel_actor()` here (as was prior)!
|
||||
#
|
||||
# Doing so will cause an "out-of-band" ctxc
|
||||
# (`tractor.ContextCancelled`) to be raised inside the
|
||||
# `ServiceMngr.open_context_in_task()`'s call to
|
||||
# `ctx.wait_for_result()` AND the internal self-ctxc
|
||||
# "graceful capture" WILL NOT CATCH IT!
|
||||
#
|
||||
# This can cause certain types of operations to raise
|
||||
# that ctxc BEFORE THEY `return`, resulting in
|
||||
# a "false-negative" ctxc being raised when really
|
||||
# nothing actually failed, other then our semantic
|
||||
# "failure" to suppress an expected, graceful,
|
||||
# self-cancel scenario..
|
||||
#
|
||||
# bUt wHy duZ It WorK lIKe dis..
|
||||
# ------------------------------
|
||||
# from the perspective of the `tractor.Context` this
|
||||
# cancel request was conducted "out of band" since
|
||||
# `Context.cancel()` was never called and thus the
|
||||
# `._cancel_called: bool` was never set. Despite the
|
||||
# remote `.canceller` being set to `pikerd` (i.e. the
|
||||
# same `Actor.uid` of the raising service-mngr task) the
|
||||
# service-task's ctx itself was never marked as having
|
||||
# requested cancellation and thus still raises the ctxc
|
||||
# bc it was unaware of any such request.
|
||||
#
|
||||
# How to make grokin these cases easier tho?
|
||||
# ------------------------------------------
|
||||
# Because `Portal.cancel_actor()` was called it requests
|
||||
# "full-`Actor`-runtime-cancellation" of it's peer
|
||||
# process which IS NOT THE SAME as a single inter-actor
|
||||
# RPC task cancelling its local context with a remote
|
||||
# peer `Task` in that same peer process.
|
||||
#
|
||||
# ?TODO? It might be better if we do one (or all) of the
|
||||
# following:
|
||||
#
|
||||
# -[ ] at least set a special message for the
|
||||
# `ContextCancelled` when raised locally by the
|
||||
# unaware ctx task such that we check for the
|
||||
# `.canceller` being *our `Actor`* and in the case
|
||||
# where `Context._cancel_called == False` we specially
|
||||
# note that this is likely an "out-of-band"
|
||||
# runtime-cancel request triggered by some call to
|
||||
# `Portal.cancel_actor()`, possibly even reporting the
|
||||
# exact LOC of that caller by tracking it inside our
|
||||
# portal-type?
|
||||
# -[ ] possibly add another field `ContextCancelled` like
|
||||
# maybe a,
|
||||
# `.request_type: Literal['os', 'proc', 'actor',
|
||||
# 'ctx']` type thing which would allow immediately
|
||||
# being able to tell what kind of cancellation caused
|
||||
# the unexpected ctxc?
|
||||
# -[ ] REMOVE THIS COMMENT, once we've settled on how to
|
||||
# better augment `tractor` to be more explicit on this!
|
||||
|
||||
except BaseException as _err:
|
||||
err = _err
|
||||
|
|
@ -152,11 +222,13 @@ async def maybe_spawn_daemon(
|
|||
lock.statistics().owner is current_task()
|
||||
):
|
||||
log.exception(
|
||||
f'Releasing stale lock after crash..?'
|
||||
f'Releasing stale lock after crash..?\n'
|
||||
f'\n'
|
||||
f'{err!r}\n'
|
||||
)
|
||||
lock.release()
|
||||
raise err
|
||||
|
||||
raise
|
||||
|
||||
|
||||
async def spawn_emsd(
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ log = get_logger(name=__name__)
|
|||
# new actors and supervises them to completion?
|
||||
class Services:
|
||||
|
||||
actor_n: tractor._supervise.ActorNursery
|
||||
actor_n: tractor.ActorNursery
|
||||
service_n: trio.Nursery
|
||||
debug_mode: bool # tractor sub-actor debug mode flag
|
||||
service_tasks: dict[
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ from contextlib import (
|
|||
)
|
||||
from typing import (
|
||||
Any,
|
||||
TypeAlias,
|
||||
)
|
||||
|
||||
import tractor
|
||||
|
|
@ -32,11 +33,23 @@ from tractor import (
|
|||
Actor,
|
||||
Portal,
|
||||
)
|
||||
from tractor.discovery._addr import (
|
||||
LegacyUnwrappedAddress,
|
||||
UnwrappedAddress,
|
||||
wrap_address,
|
||||
)
|
||||
|
||||
from piker.log import get_logger
|
||||
|
||||
log = get_logger(name=__name__)
|
||||
|
||||
# Piker config still accepts legacy socket pairs while tractor now
|
||||
# emits tagged `UnwrappedAddress` values.
|
||||
RegistryAddress: TypeAlias = (
|
||||
LegacyUnwrappedAddress
|
||||
|UnwrappedAddress
|
||||
)
|
||||
|
||||
# TODO? default path-space for UDS registry?
|
||||
# [ ] needs to be Xplatform tho!
|
||||
# _default_registry_path: Path = (
|
||||
|
|
@ -46,10 +59,7 @@ log = get_logger(name=__name__)
|
|||
|
||||
_default_registry_host: str = '127.0.0.1'
|
||||
_default_registry_port: int = 6116
|
||||
_default_reg_addr: tuple[
|
||||
str,
|
||||
int, # |str TODO, once we support UDS, see above.
|
||||
] = (
|
||||
_default_reg_addr: RegistryAddress = (
|
||||
_default_registry_host,
|
||||
_default_registry_port,
|
||||
)
|
||||
|
|
@ -63,24 +73,37 @@ _registry: Registry | None = None
|
|||
class Registry:
|
||||
# TODO: should this be a set or should we complain
|
||||
# on duplicates?
|
||||
addrs: list[tuple[str, int]] = []
|
||||
addrs: list[RegistryAddress] = []
|
||||
|
||||
# TODO: table of uids to sockaddrs
|
||||
peers: dict[
|
||||
tuple[str, str],
|
||||
tuple[str, int],
|
||||
RegistryAddress,
|
||||
] = {}
|
||||
|
||||
|
||||
_tractor_kwargs: dict[str, Any] = {}
|
||||
|
||||
|
||||
def _normalize_addresses(
|
||||
addrs: list[RegistryAddress],
|
||||
) -> list[RegistryAddress]:
|
||||
'''
|
||||
Normalize legacy and tagged addresses through tractor's codec.
|
||||
|
||||
'''
|
||||
return [
|
||||
wrap_address(addr).unwrap()
|
||||
for addr in addrs
|
||||
]
|
||||
|
||||
|
||||
@acm
|
||||
async def open_registry(
|
||||
addrs: list[tuple[str, int]],
|
||||
addrs: list[RegistryAddress],
|
||||
ensure_exists: bool = True,
|
||||
|
||||
) -> list[tuple[str, int]]:
|
||||
) -> list[RegistryAddress]:
|
||||
'''
|
||||
Open the service-actor-discovery registry by returning a set of
|
||||
tranport socket-addrs to registrar actors which may be
|
||||
|
|
@ -92,9 +115,12 @@ async def open_registry(
|
|||
actor: Actor = tractor.current_actor()
|
||||
aid: msg.Aid = actor.aid
|
||||
uid: tuple[str, str] = aid.uid
|
||||
preset_reg_addrs: list[
|
||||
tuple[str, int]
|
||||
] = Registry.addrs
|
||||
addrs = _normalize_addresses(addrs)
|
||||
preset_reg_addrs: list[RegistryAddress] = (
|
||||
_normalize_addresses(Registry.addrs)
|
||||
)
|
||||
if preset_reg_addrs:
|
||||
Registry.addrs = preset_reg_addrs
|
||||
if (
|
||||
preset_reg_addrs
|
||||
and
|
||||
|
|
@ -102,9 +128,10 @@ async def open_registry(
|
|||
):
|
||||
if preset_reg_addrs != addrs:
|
||||
# if any(addr in preset_reg_addrs for addr in addrs):
|
||||
diff: set[
|
||||
tuple[str, int]
|
||||
] = set(preset_reg_addrs) - set(addrs)
|
||||
diff: set[RegistryAddress] = (
|
||||
set(preset_reg_addrs)
|
||||
-set(addrs)
|
||||
)
|
||||
if diff:
|
||||
log.warning(
|
||||
f'`{uid}` requested only subset of registrars: {addrs}\n'
|
||||
|
|
@ -123,7 +150,9 @@ async def open_registry(
|
|||
and
|
||||
not Registry.addrs
|
||||
):
|
||||
Registry.addrs.extend(actor.reg_addrs)
|
||||
Registry.addrs.extend(
|
||||
_normalize_addresses(actor.reg_addrs)
|
||||
)
|
||||
|
||||
if (
|
||||
ensure_exists
|
||||
|
|
@ -139,13 +168,13 @@ async def open_registry(
|
|||
Registry.addrs = (
|
||||
addrs
|
||||
or
|
||||
[_default_reg_addr]
|
||||
_normalize_addresses([_default_reg_addr])
|
||||
)
|
||||
|
||||
# NOTE: only spot this seems currently used is inside
|
||||
# `.ui._exec` which is the (eventual qtloops) bootstrapping
|
||||
# with guest mode.
|
||||
reg_addrs: list[tuple[str, str|int]] = Registry.addrs
|
||||
reg_addrs: list[RegistryAddress] = Registry.addrs
|
||||
# !TODO, a struct-API to stringently allow this only in special
|
||||
# cases?
|
||||
# -> better would be to have some way to (atomically) rewrite
|
||||
|
|
@ -159,13 +188,13 @@ async def open_registry(
|
|||
# next (set of) calls will apply whatever new one is passed
|
||||
# in.
|
||||
if was_set:
|
||||
Registry.addrs = None
|
||||
Registry.addrs = []
|
||||
|
||||
|
||||
@acm
|
||||
async def find_service(
|
||||
service_name: str,
|
||||
registry_addrs: list[tuple[str, int]] | None = None,
|
||||
registry_addrs: list[RegistryAddress] | None = None,
|
||||
|
||||
first_only: bool = True,
|
||||
|
||||
|
|
@ -175,7 +204,7 @@ async def find_service(
|
|||
| None
|
||||
):
|
||||
# try:
|
||||
reg_addrs: list[tuple[str, int|str]]
|
||||
reg_addrs: list[RegistryAddress]
|
||||
async with open_registry(
|
||||
addrs=(
|
||||
registry_addrs
|
||||
|
|
@ -219,16 +248,19 @@ async def find_service(
|
|||
|
||||
async def check_for_service(
|
||||
service_name: str,
|
||||
) -> None|tuple[str, int]:
|
||||
) -> None|RegistryAddress:
|
||||
'''
|
||||
Service daemon "liveness" predicate.
|
||||
|
||||
'''
|
||||
async with (
|
||||
open_registry(ensure_exists=False) as reg_addr,
|
||||
open_registry(
|
||||
addrs=Registry.addrs,
|
||||
ensure_exists=False,
|
||||
) as reg_addrs,
|
||||
tractor.query_actor(
|
||||
service_name,
|
||||
arbiter_sockaddr=reg_addr,
|
||||
) as sockaddr,
|
||||
regaddr=reg_addrs[0],
|
||||
) as (sockaddr, _),
|
||||
):
|
||||
return sockaddr
|
||||
|
|
|
|||
|
|
@ -382,6 +382,9 @@ def quote_to_marketstore_structarray(
|
|||
class MarketStoreError(Exception):
|
||||
"Generic marketstore client error"
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([MarketStoreError])
|
||||
|
||||
|
||||
# def err_on_resp(response: dict) -> None:
|
||||
# """Raise any errors found in responses from client request.
|
||||
|
|
|
|||
|
|
@ -151,6 +151,13 @@ class StorageConnectionError(ConnectionError):
|
|||
|
||||
'''
|
||||
|
||||
from tractor._exceptions import reg_err_types
|
||||
reg_err_types([
|
||||
TimeseriesNotFound,
|
||||
StorageConnectionError,
|
||||
])
|
||||
|
||||
|
||||
def get_storagemod(
|
||||
name: str,
|
||||
|
||||
|
|
|
|||
|
|
@ -301,7 +301,7 @@ def ldshm(
|
|||
# TODO: call null-seg fixer somehow?
|
||||
if null_segs:
|
||||
|
||||
if tractor._state.is_debug_mode():
|
||||
if tractor.runtime._state.is_debug_mode():
|
||||
await tractor.pause()
|
||||
# async with (
|
||||
# trio.open_nursery() as tn,
|
||||
|
|
|
|||
|
|
@ -276,15 +276,8 @@ def get_null_segs(
|
|||
absi_zdiff: np.ndarray = np.diff(absi_zeros)
|
||||
|
||||
if zero_t.size < 2:
|
||||
try:
|
||||
breakpoint()
|
||||
except RuntimeError:
|
||||
# XXX, if greenback not active from
|
||||
# piker store ldshm cmd..
|
||||
log.exception(
|
||||
"Can't debug single-sample null!\n"
|
||||
)
|
||||
|
||||
# Keep the legacy single-null result without a REPL stop.
|
||||
# Full null grouping is handled downstream in 976db6c3.
|
||||
return None
|
||||
|
||||
# scan for all frame-indices where the
|
||||
|
|
|
|||
|
|
@ -168,7 +168,7 @@ async def markup_gaps(
|
|||
prev_r: pl.DataFrame = prev_row_by_i
|
||||
|
||||
# debug any missing pre-row
|
||||
if tractor._state.is_debug_mode():
|
||||
if tractor.runtime._state.is_debug_mode():
|
||||
await tractor.pause()
|
||||
|
||||
istart: int = prev_r['index'][0]
|
||||
|
|
|
|||
|
|
@ -202,12 +202,11 @@ async def maybe_fill_null_segments(
|
|||
mkt: MktPair,
|
||||
backfill_until_dt: datetime,
|
||||
|
||||
task_status: TaskStatus[trio.Event] = trio.TASK_STATUS_IGNORED,
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> list[Frame]:
|
||||
|
||||
null_segs_detected = trio.Event()
|
||||
task_status.started(null_segs_detected)
|
||||
task_status.started()
|
||||
|
||||
frame: Frame = shm.array
|
||||
|
||||
|
|
@ -240,15 +239,26 @@ async def maybe_fill_null_segments(
|
|||
await tractor.pause()
|
||||
break
|
||||
|
||||
(
|
||||
array,
|
||||
next_start_dt,
|
||||
next_end_dt,
|
||||
) = await get_hist(
|
||||
timeframe,
|
||||
start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
)
|
||||
try:
|
||||
(
|
||||
array,
|
||||
next_start_dt,
|
||||
next_end_dt,
|
||||
) = await get_hist(
|
||||
timeframe,
|
||||
start_dt=start_dt,
|
||||
end_dt=end_dt,
|
||||
)
|
||||
except (
|
||||
NoData,
|
||||
DataUnavailable,
|
||||
) as exc:
|
||||
log.warning(
|
||||
f'Provider could not repair SHM gap:\n'
|
||||
f'{start_dt} -> {end_dt}\n'
|
||||
f'{exc}\n'
|
||||
)
|
||||
break
|
||||
|
||||
if array.size == 0:
|
||||
log.warning(
|
||||
|
|
@ -308,7 +318,6 @@ async def maybe_fill_null_segments(
|
|||
await tractor.pause()
|
||||
raise
|
||||
|
||||
null_segs_detected.set()
|
||||
# RECHECK for more null-gaps
|
||||
frame: Frame = shm.array
|
||||
null_segs: tuple | None = get_null_segs(
|
||||
|
|
@ -407,15 +416,13 @@ async def start_backfill(
|
|||
storage: StorageClient|None = None,
|
||||
write_tsdb: bool = True,
|
||||
|
||||
task_status: TaskStatus[tuple] = trio.TASK_STATUS_IGNORED,
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
|
||||
) -> int:
|
||||
|
||||
# let caller unblock and deliver latest history frame
|
||||
# and use to signal that backfilling the shm gap until
|
||||
# the tsdb end is complete!
|
||||
bf_done = trio.Event()
|
||||
task_status.started(bf_done)
|
||||
# Let the caller place storage history while this task
|
||||
# continues reverse retrieval in the owning nursery.
|
||||
task_status.started()
|
||||
|
||||
# based on the sample step size, maybe load a certain amount history
|
||||
update_start_on_prepend: bool = False
|
||||
|
|
@ -535,6 +542,15 @@ async def start_backfill(
|
|||
# charge of solving such faults? yes, right?
|
||||
return
|
||||
|
||||
if array.size == 0:
|
||||
log.warning(
|
||||
f'Provider {mod.name!r} returned an empty frame\n'
|
||||
f'fqme: {mkt.fqme}\n'
|
||||
f'timeframe: {timeframe}\n'
|
||||
f'end_dt: {end_dt_param}\n'
|
||||
)
|
||||
break
|
||||
|
||||
time: np.ndarray = array['time']
|
||||
assert (
|
||||
time[0]
|
||||
|
|
@ -791,10 +807,6 @@ async def start_backfill(
|
|||
# memory...
|
||||
# await sampler_stream.send('broadcast_all')
|
||||
|
||||
# short-circuit (for now)
|
||||
bf_done.set()
|
||||
|
||||
|
||||
# NOTE: originally this was used to cope with a tsdb (marketstore)
|
||||
# which could not delivery very large frames of history over gRPC
|
||||
# (thanks goolag) due to corruption issues.
|
||||
|
|
@ -1181,7 +1193,7 @@ async def tsdb_backfill(
|
|||
trio.open_nursery() as tn,
|
||||
):
|
||||
|
||||
bf_done: trio.Event = await tn.start(
|
||||
await tn.start(
|
||||
partial(
|
||||
start_backfill,
|
||||
get_hist=get_hist,
|
||||
|
|
@ -1201,8 +1213,6 @@ async def tsdb_backfill(
|
|||
write_tsdb=True,
|
||||
)
|
||||
)
|
||||
nulls_detected: trio.Event|None = None
|
||||
|
||||
if last_tsdb_dt is not None:
|
||||
|
||||
# calc the index from which the tsdb data should be
|
||||
|
|
@ -1279,7 +1289,7 @@ async def tsdb_backfill(
|
|||
# work PREVENTAVELY instead?
|
||||
# -[ ] fill in non-zero epoch time values ALWAYS!
|
||||
# await maybe_fill_null_segments(
|
||||
nulls_detected: trio.Event = await tn.start(partial(
|
||||
await tn.start(partial(
|
||||
maybe_fill_null_segments,
|
||||
|
||||
shm=shm,
|
||||
|
|
@ -1292,11 +1302,6 @@ async def tsdb_backfill(
|
|||
|
||||
# 2nd nursery END
|
||||
|
||||
# TODO: who would want to?
|
||||
if nulls_detected:
|
||||
await nulls_detected.wait()
|
||||
|
||||
await bf_done.wait()
|
||||
# TODO: maybe start history anal and load missing "history
|
||||
# gaps" via backend..
|
||||
|
||||
|
|
|
|||
|
|
@ -527,7 +527,7 @@ async def graphics_update_loop(
|
|||
continue
|
||||
|
||||
ic = fast_chart.view._in_interact
|
||||
if ic:
|
||||
if ic is not None:
|
||||
fast_chart.pause_all_feeds()
|
||||
log.debug(
|
||||
f'Pausing chart updaates during interaction\n'
|
||||
|
|
|
|||
|
|
@ -581,7 +581,7 @@ class ChartView(ViewBox):
|
|||
self.order_mode: bool = False
|
||||
|
||||
self.setFocusPolicy(QtCore.Qt.StrongFocus)
|
||||
self._in_interact: trio.Event | None = None
|
||||
self._in_interact: trio.Event|None = None
|
||||
self._interact_stack: ExitStack = ExitStack()
|
||||
|
||||
# TODO: probably just assign this whenever a new `PlotItem` is
|
||||
|
|
@ -619,7 +619,7 @@ class ChartView(ViewBox):
|
|||
to any waiters.
|
||||
|
||||
'''
|
||||
if self._in_interact:
|
||||
if self._in_interact is not None:
|
||||
try:
|
||||
self._interact_stack.close()
|
||||
self.chart.resume_all_feeds()
|
||||
|
|
@ -1154,7 +1154,7 @@ class ChartView(ViewBox):
|
|||
str,
|
||||
tuple[float, float],
|
||||
] | None = None,
|
||||
|
||||
|
||||
):
|
||||
profiler = Profiler(
|
||||
msg=f'ChartView.interact_graphics_cycle() for {self.name}',
|
||||
|
|
|
|||
118
piker/ui/cli.py
118
piker/ui/cli.py
|
|
@ -22,7 +22,10 @@ import os
|
|||
import click
|
||||
import tractor
|
||||
|
||||
from ..cli import cli
|
||||
from ..cli import (
|
||||
cli,
|
||||
load_trans_eps,
|
||||
)
|
||||
from .. import watchlists as wl
|
||||
from ..service import maybe_spawn_brokerd
|
||||
|
||||
|
|
@ -159,67 +162,63 @@ def chart(
|
|||
needed if not discovered via [network] config.
|
||||
|
||||
'''
|
||||
# eg. ``--profile 3`` reports profiling for anything slower then 3 ms.
|
||||
if profile is not None:
|
||||
from .. import _profile
|
||||
_profile._pg_profile = True
|
||||
_profile.ms_slower_then = float(profile)
|
||||
|
||||
# Qt UI entrypoint
|
||||
from ._app import _main
|
||||
|
||||
for symbol in symbols:
|
||||
if '.' not in symbol:
|
||||
click.echo(click.style(
|
||||
f'symbol: {symbol} must have a {symbol}.<provider> suffix',
|
||||
fg='red',
|
||||
))
|
||||
return
|
||||
|
||||
# global opts
|
||||
# brokernames: list[str] = config['brokers']
|
||||
brokermods = config['brokermods']
|
||||
assert brokermods
|
||||
tractorloglevel = config['tractorloglevel']
|
||||
pikerloglevel = config['loglevel']
|
||||
|
||||
maddrs: list[tuple[str, int]] = config.get(
|
||||
'maddrs',
|
||||
[],
|
||||
)
|
||||
|
||||
# if maddrs:
|
||||
# from tractor._multiaddr import parse_maddr
|
||||
# for addr in maddrs:
|
||||
# breakpoint()
|
||||
# layers: dict = parse_maddr(addr)
|
||||
|
||||
regaddrs: list[tuple[str, int]] = config.get(
|
||||
'registry_addrs',
|
||||
[],
|
||||
)
|
||||
|
||||
from ..config import load
|
||||
conf, _ = load(
|
||||
conf_name='conf',
|
||||
)
|
||||
network: dict = conf.get('network')
|
||||
if network:
|
||||
from ..cli import load_trans_eps
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddrs,
|
||||
)
|
||||
for layers in eps['pikerd']:
|
||||
regaddrs.append((
|
||||
layers['ipv4']['addr'],
|
||||
layers['tcp']['port'],
|
||||
))
|
||||
|
||||
# breakpoint()
|
||||
from tractor.devx import maybe_open_crash_handler
|
||||
pdb: bool = config['pdb']
|
||||
with maybe_open_crash_handler(pdb=pdb):
|
||||
# eg. ``--profile 3`` reports profiling for anything slower then 3 ms.
|
||||
if profile is not None:
|
||||
from .. import _profile
|
||||
_profile._pg_profile = True
|
||||
_profile.ms_slower_then = float(profile)
|
||||
|
||||
# Qt UI entrypoint
|
||||
from ._app import _main
|
||||
|
||||
for symbol in symbols:
|
||||
if '.' not in symbol:
|
||||
click.echo(click.style(
|
||||
f'symbol: {symbol} must have a {symbol}.<provider> suffix',
|
||||
fg='red',
|
||||
))
|
||||
return
|
||||
|
||||
# global opts
|
||||
# brokernames: list[str] = config['brokers']
|
||||
brokermods = config['brokermods']
|
||||
assert brokermods
|
||||
tractorloglevel = config['tractorloglevel']
|
||||
pikerloglevel = config['loglevel']
|
||||
|
||||
# !TODO, allow declaring bind addr via maddr!
|
||||
from ..config import load
|
||||
conf, _ = load(
|
||||
conf_name='conf',
|
||||
)
|
||||
|
||||
regaddrs: list = config.get(
|
||||
'registry_addrs',
|
||||
[],
|
||||
)
|
||||
chart_eps: list|None = None
|
||||
network: dict = conf.get('network')
|
||||
if network:
|
||||
eps: dict = load_trans_eps(
|
||||
network,
|
||||
maddrs=None,
|
||||
)
|
||||
|
||||
# registry addrs: prefer `regd`, fall back
|
||||
# to `pikerd` eps
|
||||
reg_eps = eps.get(
|
||||
'regd',
|
||||
eps.get('pikerd')
|
||||
)
|
||||
assert reg_eps
|
||||
regaddrs = [
|
||||
addr.unwrap() for addr in reg_eps
|
||||
]
|
||||
chart_eps = eps.get('chart')
|
||||
|
||||
_main(
|
||||
syms=symbols,
|
||||
brokermods=brokermods,
|
||||
|
|
@ -229,6 +228,7 @@ def chart(
|
|||
'loglevel': tractorloglevel,
|
||||
'name': 'chart',
|
||||
'registry_addrs': list(set(regaddrs)),
|
||||
'tpt_bind_addrs': chart_eps,
|
||||
'enable_modules': [
|
||||
|
||||
# remote data-view annotations Bo
|
||||
|
|
|
|||
|
|
@ -167,7 +167,7 @@ async def stream_symbol_selection():
|
|||
|
||||
async def _async_main(
|
||||
name: str,
|
||||
portal: tractor._portal.Portal,
|
||||
portal: tractor.Portal,
|
||||
symbols: List[str],
|
||||
brokermod: ModuleType,
|
||||
loglevel: str = 'info',
|
||||
|
|
|
|||
|
|
@ -436,7 +436,7 @@ class OptionChain(object):
|
|||
|
||||
|
||||
async def new_chain_ui(
|
||||
portal: tractor._portal.Portal,
|
||||
portal: tractor.Portal,
|
||||
symbol: str,
|
||||
brokermod: types.ModuleType,
|
||||
rate: int = 1,
|
||||
|
|
|
|||
|
|
@ -1022,13 +1022,22 @@ async def open_order_mode(
|
|||
started.set()
|
||||
|
||||
for oid, msg in ems_dialog_msgs.items():
|
||||
|
||||
# HACK ALERT: ensure a resp field is filled out since
|
||||
# techincally the call below expects a ``Status``. TODO:
|
||||
# parse into proper ``Status`` equivalents ems-side?
|
||||
# msg.setdefault('resp', msg['broker_details']['resp'])
|
||||
# msg.setdefault('oid', msg['broker_details']['oid'])
|
||||
msg['brokerd_msg'] = msg
|
||||
ya_msg: dict = msg.setdefault(
|
||||
'brokerd_msg',
|
||||
msg,
|
||||
)
|
||||
if msg is not ya_msg:
|
||||
log.warning(
|
||||
f'A `.brokerd_msg` was already set for ems-dialog msg?\n'
|
||||
f'oid: {oid!r}\n'
|
||||
f'ya_msg: {ya_msg!r}\n'
|
||||
f'msg: {ya_msg!r}\n'
|
||||
)
|
||||
|
||||
await process_trade_msg(
|
||||
mode,
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ name = "piker"
|
|||
version = "0.1.0a0dev0"
|
||||
description = "trading gear for hackers"
|
||||
authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
|
||||
requires-python = ">=3.12, <3.14"
|
||||
requires-python = ">=3.13, <3.14"
|
||||
license = "AGPL-3.0-or-later"
|
||||
readme = "README.rst"
|
||||
keywords = [
|
||||
|
|
@ -39,7 +39,6 @@ classifiers = [
|
|||
"Operating System :: POSIX :: Linux",
|
||||
"Programming Language :: Python :: Implementation :: CPython",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Intended Audience :: Financial and Insurance Industry",
|
||||
"Intended Audience :: Science/Research",
|
||||
|
|
@ -53,7 +52,7 @@ dependencies = [
|
|||
"colorama >=0.4.6, <0.5.0",
|
||||
"colorlog >=6.7.0, <7.0.0",
|
||||
"numpy>=2.0",
|
||||
"polars >=0.20.6",
|
||||
"polars >=0.20.6, <2.0.0",
|
||||
"polars-fuzzy-match>=0.1.5",
|
||||
"pygments >=2.16.1, <3.0.0",
|
||||
"rich >=13.5.2, <14.0.0",
|
||||
|
|
@ -62,13 +61,13 @@ dependencies = [
|
|||
"trio-util >=0.7.0, <0.8.0",
|
||||
"trio-websocket >=0.10.3, <0.11.0",
|
||||
"typer >=0.9.0, <1.0.0",
|
||||
"trio >=0.27",
|
||||
"trio >=0.34.0",
|
||||
"pendulum",
|
||||
"httpx >=0.27.0, <0.28.0",
|
||||
"cryptofeed >=2.4.0, <3.0.0",
|
||||
"pyarrow>=18.0.0",
|
||||
"websockets ==12.0",
|
||||
"msgspec>=0.19.0,<0.20",
|
||||
"msgspec>=0.20.0",
|
||||
"tractor",
|
||||
"tomlkit",
|
||||
"trio-typing>=0.10.0",
|
||||
|
|
@ -132,7 +131,7 @@ repl = [
|
|||
"greenback >=1.1.1, <2.0.0",
|
||||
|
||||
# @goodboy's preferred console toolz
|
||||
"xonsh>=0.22.2",
|
||||
"xonsh>=0.24.2",
|
||||
"prompt-toolkit ==3.0.40",
|
||||
"pyperclip>=1.9.0",
|
||||
|
||||
|
|
@ -203,7 +202,7 @@ pyvnc = { git = "https://github.com/regulad/pyvnc.git" }
|
|||
# xonsh = { git = 'https://github.com/xonsh/xonsh.git', branch = 'main' }
|
||||
|
||||
# XXX since, we're like, always hacking new shite all-the-time. Bp
|
||||
tractor = { git = "https://github.com/goodboy/tractor.git", branch ="main" }
|
||||
tractor = { git = "https://github.com/goodboy/tractor.git", branch = "piker_pin" }
|
||||
# tractor = { git = "https://pikers.dev/goodboy/tractor", branch = "piker_pin" }
|
||||
# ------ goodboy ------
|
||||
# hackin dev-envs, usually there's something new he's hackin in..
|
||||
|
|
|
|||
|
|
@ -22,8 +22,6 @@ pytest_plugins: tuple[str] = (
|
|||
|
||||
|
||||
def pytest_addoption(parser):
|
||||
parser.addoption("--ll", action="store", dest='loglevel',
|
||||
default=None, help="logging level to set when testing")
|
||||
parser.addoption("--confdir", default=None,
|
||||
help="Use a practice API account")
|
||||
|
||||
|
|
@ -92,8 +90,7 @@ def log(
|
|||
@acm
|
||||
async def _open_test_pikerd(
|
||||
tmpconfdir: str,
|
||||
|
||||
reg_addr: tuple[str, int] | None = None,
|
||||
reg_addr: tuple[str, int|str],
|
||||
loglevel: str = 'warning',
|
||||
debug_mode: bool = False,
|
||||
|
||||
|
|
@ -113,16 +110,10 @@ async def _open_test_pikerd(
|
|||
to boot the root actor / tractor runtime.
|
||||
|
||||
'''
|
||||
import random
|
||||
from piker.service import maybe_open_pikerd
|
||||
|
||||
if reg_addr is None:
|
||||
port = random.randint(6e3, 7e3)
|
||||
reg_addr = ('127.0.0.1', port)
|
||||
|
||||
async with (
|
||||
maybe_open_pikerd(
|
||||
registry_addr=reg_addr,
|
||||
registry_addrs=[reg_addr],
|
||||
loglevel=loglevel,
|
||||
|
||||
tractor_runtime_overrides={
|
||||
|
|
@ -139,13 +130,19 @@ async def _open_test_pikerd(
|
|||
|
||||
async with tractor.wait_for_actor(
|
||||
'pikerd',
|
||||
arbiter_sockaddr=reg_addr,
|
||||
registry_addr=reg_addr,
|
||||
) as portal:
|
||||
raddr = portal.channel.raddr
|
||||
assert raddr == reg_addr
|
||||
raddr = portal.chan.raddr
|
||||
uw_raddr: tuple = raddr.unwrap()
|
||||
canonical_addr: tuple = (
|
||||
tractor.discovery._addr.wrap_address(
|
||||
reg_addr,
|
||||
).unwrap()
|
||||
)
|
||||
assert uw_raddr == canonical_addr
|
||||
yield (
|
||||
raddr[0],
|
||||
raddr[1],
|
||||
raddr._host,
|
||||
raddr._port,
|
||||
portal,
|
||||
service_manager,
|
||||
)
|
||||
|
|
@ -202,7 +199,10 @@ def open_test_pikerd(
|
|||
request: pytest.FixtureRequest,
|
||||
tmp_path: Path,
|
||||
tmpconfdir: Path,
|
||||
|
||||
# XXX from `tractor._testing.pytest` plugin
|
||||
loglevel: str,
|
||||
reg_addr: tuple,
|
||||
):
|
||||
|
||||
tmpconfdir_str: str = str(tmpconfdir)
|
||||
|
|
@ -236,10 +236,13 @@ def open_test_pikerd(
|
|||
# bwitout clobbering each other's config state.
|
||||
tmpconfdir=tmpconfdir_str,
|
||||
|
||||
# bind in level from fixture, which is itself set by
|
||||
# `--ll <value>` cli flag.
|
||||
# NOTE these come verbatim from `tractor`'s builtin plugin!
|
||||
#
|
||||
# per-tpt compat registrar address.
|
||||
reg_addr=reg_addr,
|
||||
# bind in level from fixture.
|
||||
# (can be set with `--ll <value>` flag to `pytest`).
|
||||
loglevel=loglevel,
|
||||
|
||||
debug_mode=debug_mode,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,251 @@
|
|||
'''
|
||||
Binance websocket routing and feed-startup regressions.
|
||||
|
||||
'''
|
||||
import pytest
|
||||
import trio
|
||||
|
||||
from piker.brokers._util import DataUnavailable
|
||||
from piker.brokers.binance import feed as binance_feed
|
||||
from piker.brokers.binance.feed import (
|
||||
get_quote_routes,
|
||||
stream_messages,
|
||||
wait_for_live_quotes,
|
||||
)
|
||||
from piker.brokers.binance.venues import (
|
||||
_futes_market_ws,
|
||||
_futes_public_ws,
|
||||
_spot_ws,
|
||||
)
|
||||
from piker.data.validate import FeedInitializationError
|
||||
|
||||
|
||||
def test_usdtm_websocket_routes() -> None:
|
||||
'''
|
||||
Binance retired its legacy USD-M ``/ws`` route in April 2026.
|
||||
|
||||
Market and high-frequency public traffic now use separate base
|
||||
paths. Verify piker retains both routes so a
|
||||
future endpoint edit cannot silently send a valid subscription to
|
||||
a socket which never publishes that traffic class.
|
||||
|
||||
'''
|
||||
routes: tuple[str, ...] = get_quote_routes(
|
||||
mkt_mode='usdtm_futes',
|
||||
symbols=['NVDAUSDT'],
|
||||
)
|
||||
assert routes == (
|
||||
f'{_futes_public_ws}/nvdausdt@bookTicker',
|
||||
f'{_futes_market_ws}/nvdausdt@aggTrade',
|
||||
)
|
||||
assert get_quote_routes(
|
||||
mkt_mode='spot',
|
||||
symbols=['BTCUSDT'],
|
||||
) == (
|
||||
f'{_spot_ws}/btcusdt@bookTicker',
|
||||
f'{_spot_ws}/btcusdt@aggTrade',
|
||||
)
|
||||
|
||||
|
||||
def test_silent_binance_feed_startup_fails() -> None:
|
||||
'''
|
||||
A retired Binance websocket accepted piker's connection but sent
|
||||
no first data, leaving ``stream_quotes()`` blocked before
|
||||
``task_status.started()`` and hanging every chart consumer.
|
||||
|
||||
Keep an empty receive channel open to reproduce a connected but
|
||||
silent socket. The controlled deadline proves startup raises
|
||||
``DataUnavailable`` with the absent stream classes instead of
|
||||
waiting indefinitely for a first quote.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
send, recv = trio.open_memory_channel(1)
|
||||
async with send, recv:
|
||||
with pytest.raises(
|
||||
DataUnavailable,
|
||||
match='no live data received for NVDAUSDT',
|
||||
):
|
||||
await wait_for_live_quotes(
|
||||
msg_stream=recv,
|
||||
symbols=['NVDAUSDT'],
|
||||
startup_timeout=0.01,
|
||||
)
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def test_usdtm_aggregate_trade_stream_type() -> None:
|
||||
'''
|
||||
Binance added ``st`` to USD-M aggregate trades during its June
|
||||
2026 UM/CM websocket integration. The first valid NVDAUSDT trade
|
||||
then raised ``TypeError`` while constructing piker's strict
|
||||
``AggTrade`` struct, terminating the feed relay during startup.
|
||||
|
||||
Feed one representative USD-M frame through ``stream_messages()``
|
||||
and assert its trade quote is emitted. This proves the new stream
|
||||
discriminator is accepted without weakening parsing of the fields
|
||||
used by downstream consumers.
|
||||
|
||||
'''
|
||||
class MockWs:
|
||||
def __init__(self) -> None:
|
||||
self._msgs = iter([{
|
||||
'e': 'aggTrade',
|
||||
'E': 1787782452314,
|
||||
'a': 8430770,
|
||||
's': 'NVDAUSDT',
|
||||
'p': '217.91000',
|
||||
'q': '0.92',
|
||||
'nq': '0.92',
|
||||
'f': 20677382,
|
||||
'l': 20677382,
|
||||
'T': 1787782452162,
|
||||
'm': True,
|
||||
'st': 1,
|
||||
}])
|
||||
|
||||
def __aiter__(self):
|
||||
return self
|
||||
|
||||
async def __anext__(self) -> dict:
|
||||
try:
|
||||
return next(self._msgs)
|
||||
except StopIteration:
|
||||
raise StopAsyncIteration from None
|
||||
|
||||
async def main() -> None:
|
||||
msg_gen = stream_messages(MockWs())
|
||||
typ, quote = await anext(msg_gen)
|
||||
assert typ == 'trade'
|
||||
assert quote['symbol'] == 'NVDAUSDT'
|
||||
assert quote['last'] == 217.91
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def test_usdtm_book_ticker_stream_type() -> None:
|
||||
'''
|
||||
Binance USD-M book-ticker frames include event and transaction
|
||||
metadata in addition to the update ID and four L1 value fields.
|
||||
|
||||
Feed the documented USD-M response shape through
|
||||
``stream_messages()``. The emitted bid and ask ticks prove the
|
||||
parser recognizes the real provider schema rather than a reduced
|
||||
test-only event which could conceal route incompatibility.
|
||||
|
||||
'''
|
||||
class MockWs:
|
||||
def __init__(self) -> None:
|
||||
self._msgs = iter([{
|
||||
'e': 'bookTicker',
|
||||
'u': 400900217,
|
||||
'E': 1568014460893,
|
||||
'T': 1568014460891,
|
||||
's': 'BNBUSDT',
|
||||
'b': '25.35190000',
|
||||
'B': '31.21000000',
|
||||
'a': '25.36520000',
|
||||
'A': '40.66000000',
|
||||
}])
|
||||
|
||||
def __aiter__(self):
|
||||
return self
|
||||
|
||||
async def __anext__(self) -> dict:
|
||||
try:
|
||||
return next(self._msgs)
|
||||
except StopIteration:
|
||||
raise StopAsyncIteration from None
|
||||
|
||||
async def main() -> None:
|
||||
msg_gen = stream_messages(MockWs())
|
||||
typ, quote = await anext(msg_gen)
|
||||
assert typ == 'l1'
|
||||
assert quote['symbol'] == 'BNBUSDT'
|
||||
assert quote['ticks'][0] == {
|
||||
'type': 'bid',
|
||||
'price': 25.3519,
|
||||
'size': 31.21,
|
||||
}
|
||||
assert quote['ticks'][2] == {
|
||||
'type': 'ask',
|
||||
'price': 25.3652,
|
||||
'size': 40.66,
|
||||
}
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def test_binance_feed_rejects_non_single_symbol_requests() -> None:
|
||||
'''
|
||||
Piker currently allocates one backend feed task per symbol, but
|
||||
Binance previously accepted empty or multi-symbol requests and
|
||||
either rendered invalid routes or silently retained one market.
|
||||
|
||||
Exercise both invalid cardinalities against the pure route helper
|
||||
and public backend boundary. The helper ``ValueError`` proves no
|
||||
malformed URLs can be rendered, while contextual
|
||||
``FeedInitializationError`` errors prove ``stream_quotes()``
|
||||
rejects each request before market lookup, websocket opening, or
|
||||
task startup.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
for symbols in (
|
||||
[],
|
||||
['NVDAUSDT', 'TSLAUSDT'],
|
||||
):
|
||||
with pytest.raises(
|
||||
ValueError,
|
||||
match='exactly one symbol',
|
||||
):
|
||||
get_quote_routes(
|
||||
mkt_mode='usdtm_futes',
|
||||
symbols=symbols,
|
||||
)
|
||||
|
||||
with pytest.raises(
|
||||
FeedInitializationError,
|
||||
match='exactly one symbol',
|
||||
):
|
||||
await binance_feed.stream_quotes(
|
||||
send_chan=None,
|
||||
symbols=symbols,
|
||||
feed_is_live=trio.Event(),
|
||||
)
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'typ',
|
||||
['l1', 'trade'],
|
||||
)
|
||||
def test_first_quote_starts_feed(typ: str) -> None:
|
||||
'''
|
||||
Waiting independently for both L1 and trade data blocked startup
|
||||
whenever either stream class remained quiet after the other proved
|
||||
the merged Binance route was live.
|
||||
|
||||
Parameterize the first frame across both stream classes and keep
|
||||
the merged channel open without a second producer. Returning that
|
||||
exact frame under the startup timeout proves either L1-first or
|
||||
trade-first delivery independently establishes feed readiness.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
send, recv = trio.open_memory_channel(1)
|
||||
first_quote: dict = {
|
||||
'symbol': 'NVDAUSDT',
|
||||
}
|
||||
async with send, recv:
|
||||
await send.send((typ, first_quote))
|
||||
quote: dict = await wait_for_live_quotes(
|
||||
msg_stream=recv,
|
||||
symbols=['NVDAUSDT'],
|
||||
startup_timeout=0.01,
|
||||
)
|
||||
assert quote is first_quote
|
||||
|
||||
trio.run(main)
|
||||
|
|
@ -151,7 +151,7 @@ def load_and_check_pos(
|
|||
# is the same the fqme.
|
||||
pp: Position = table.pps[ppmsg.symbol]
|
||||
|
||||
assert ppmsg.size == pp.size
|
||||
assert ppmsg.size == pp.cumsize
|
||||
assert ppmsg.avg_price == pp.ppu
|
||||
|
||||
yield pp
|
||||
|
|
@ -179,7 +179,7 @@ def test_ems_err_on_bad_broker(
|
|||
# NOTE: emsd should error on the actor's enabled modules
|
||||
# import phase, when looking for a backend named `doggy`.
|
||||
except tractor.RemoteActorError as re:
|
||||
assert re.type is ModuleNotFoundError
|
||||
assert re.boxed_type is ModuleNotFoundError
|
||||
|
||||
run_and_tollerate_cancels(load_bad_fqme)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,133 @@
|
|||
'''
|
||||
EMS broadcast ownership regressions.
|
||||
|
||||
'''
|
||||
from contextlib import asynccontextmanager as acm
|
||||
|
||||
import trio
|
||||
from trio.testing import wait_all_tasks_blocked
|
||||
from tractor.trionics import broadcast_receiver
|
||||
|
||||
from piker.clearing import _ems
|
||||
|
||||
|
||||
def test_dark_clearing_owns_non_strict_subscription(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
'''
|
||||
Dark clearing must not mutate another feed consumer's lag policy.
|
||||
|
||||
`clear_dark_triggers()` previously assigned `_raise_on_lag` on
|
||||
whichever root or temporary receiver was stored in `Flume.stream`.
|
||||
That changed private shared state and made behavior depend on feed
|
||||
cache ownership. Supply a stream whose subscription records the
|
||||
public lag-policy argument and yields a distinct child. Replace
|
||||
the hot trigger loop with a checkpointing probe, then prove the
|
||||
non-strict child is entered, passed to the core, and closed only
|
||||
after processing completes while the source remains untouched.
|
||||
|
||||
'''
|
||||
events: list[tuple] = []
|
||||
child = object()
|
||||
|
||||
class QuoteStream:
|
||||
'''
|
||||
Record public subscription ownership without quote traffic.
|
||||
|
||||
'''
|
||||
@acm
|
||||
async def subscribe(self, raise_on_lag: bool = True):
|
||||
'''
|
||||
Yield one dedicated child for the wrapper lifetime.
|
||||
|
||||
'''
|
||||
events.append(('enter', raise_on_lag))
|
||||
try:
|
||||
yield child
|
||||
finally:
|
||||
events.append(('exit', raise_on_lag))
|
||||
|
||||
async def clear_core(*args) -> None:
|
||||
'''
|
||||
Prove the child remains owned while processing runs.
|
||||
|
||||
'''
|
||||
events.append(('core', args[2] is child))
|
||||
await trio.lowlevel.checkpoint()
|
||||
|
||||
monkeypatch.setattr(
|
||||
_ems,
|
||||
'_clear_dark_triggers',
|
||||
clear_core,
|
||||
)
|
||||
|
||||
async def main() -> None:
|
||||
stream = QuoteStream()
|
||||
await _ems.clear_dark_triggers(
|
||||
object(),
|
||||
object(),
|
||||
stream,
|
||||
'ib',
|
||||
'nvda.nasdaq.ib',
|
||||
object(),
|
||||
)
|
||||
assert not hasattr(stream, '_raise_on_lag')
|
||||
|
||||
trio.run(main)
|
||||
assert events == [
|
||||
('enter', False),
|
||||
('core', True),
|
||||
('exit', False),
|
||||
]
|
||||
|
||||
|
||||
def test_dark_clearing_with_real_broadcast() -> None:
|
||||
'''
|
||||
Dark clearing must consume quotes without owning sibling lifetime.
|
||||
|
||||
Exercise the production trigger loop and real Tractor broadcaster
|
||||
together. Deliver one trade tick to both EMS and a sibling cursor;
|
||||
DarkBook.lasts proves EMS processed it. Cancel only the EMS task
|
||||
and verify its subscription disappears while the sibling still
|
||||
receives the next tick with its strict lag policy intact.
|
||||
|
||||
The book has no order triggers, so this requires no broker or
|
||||
account. Unlike the imported wrapper test, no stream or clearing
|
||||
function is replaced.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
symbol = 'qqq.nasdaq.ib'
|
||||
book = _ems.DarkBook(broker='ib', triggers={}, lasts={})
|
||||
tx, rx = trio.open_memory_channel(8)
|
||||
root = broadcast_receiver(rx, 8)
|
||||
quote = {symbol: {'ticks': [{'type': 'trade', 'price': 100}]}}
|
||||
|
||||
with trio.fail_after(2):
|
||||
async with tx, rx, root, root.subscribe() as sibling:
|
||||
async with trio.open_nursery() as nursery:
|
||||
nursery.start_soon(
|
||||
_ems.clear_dark_triggers,
|
||||
None,
|
||||
None,
|
||||
root,
|
||||
'ib',
|
||||
symbol,
|
||||
book,
|
||||
)
|
||||
await wait_all_tasks_blocked()
|
||||
assert len(root._state.subs) == 3
|
||||
await tx.send(quote)
|
||||
assert await sibling.receive() == quote
|
||||
await wait_all_tasks_blocked()
|
||||
assert book.lasts[symbol] == 100
|
||||
assert root._raise_on_lag
|
||||
assert sibling._raise_on_lag
|
||||
nursery.cancel_scope.cancel()
|
||||
|
||||
assert len(root._state.subs) == 2
|
||||
assert not sibling._closed
|
||||
await tx.send(quote)
|
||||
assert await sibling.receive() == quote
|
||||
|
||||
trio.run(main)
|
||||
|
|
@ -0,0 +1,347 @@
|
|||
'''
|
||||
Cached feed broadcast ownership regressions.
|
||||
|
||||
'''
|
||||
from contextlib import asynccontextmanager as acm
|
||||
from types import SimpleNamespace
|
||||
|
||||
import tractor
|
||||
import trio
|
||||
import pytest
|
||||
|
||||
from piker.data import feed as feed_mod
|
||||
from piker.data.flows import Flume
|
||||
|
||||
|
||||
class FakeMsgStream:
|
||||
'''
|
||||
Duplex test stream backed by a real Tractor broadcaster.
|
||||
|
||||
'''
|
||||
def __init__(self, broker: str) -> None:
|
||||
self.broker: str = broker
|
||||
self._ctx: object = object()
|
||||
self._tx, rx = trio.open_memory_channel(8)
|
||||
self._broadcaster = tractor.trionics.broadcast_receiver(
|
||||
rx,
|
||||
8,
|
||||
)
|
||||
self.children: list[
|
||||
tractor.trionics.BroadcastReceiver,
|
||||
] = []
|
||||
self.controls: list[str] = []
|
||||
|
||||
@property
|
||||
def ctx(self) -> object:
|
||||
'''
|
||||
Expose the public context ref provided by `MsgStream.ctx`.
|
||||
|
||||
'''
|
||||
return self._ctx
|
||||
|
||||
@acm
|
||||
async def subscribe(self):
|
||||
'''
|
||||
Yield a real child receiver with duplex send delegation.
|
||||
|
||||
'''
|
||||
child: tractor.trionics.BroadcastReceiver
|
||||
async with self._broadcaster.subscribe() as child:
|
||||
child.send = self.send
|
||||
self.children.append(child)
|
||||
yield child
|
||||
|
||||
async def send(self, msg: str) -> None:
|
||||
'''
|
||||
Record a pause or resume control sent through a child.
|
||||
|
||||
'''
|
||||
self.controls.append(msg)
|
||||
|
||||
async def push(self, msg: dict) -> None:
|
||||
'''
|
||||
Send one provider quote into the broadcast source.
|
||||
|
||||
'''
|
||||
await self._tx.send(msg)
|
||||
|
||||
async def close_source(self) -> None:
|
||||
'''
|
||||
End this provider without closing sibling relay channels.
|
||||
|
||||
'''
|
||||
await self._tx.aclose()
|
||||
|
||||
|
||||
def make_feed() -> tuple[
|
||||
feed_mod.Feed,
|
||||
dict[str, FakeMsgStream],
|
||||
]:
|
||||
'''
|
||||
Build a real `Feed` and `Flume` graph for two providers.
|
||||
|
||||
'''
|
||||
roots: dict[str, FakeMsgStream] = {
|
||||
broker: FakeMsgStream(broker)
|
||||
for broker in ('ib', 'deribit')
|
||||
}
|
||||
flumes: dict[str, Flume] = {
|
||||
fqme: Flume(
|
||||
mkt=SimpleNamespace(broker=broker),
|
||||
first_quote={},
|
||||
_rt_shm_token=SimpleNamespace(),
|
||||
stream=roots[broker],
|
||||
)
|
||||
for fqme, broker in (
|
||||
('nvda.nasdaq.ib', 'ib'),
|
||||
('aapl.nasdaq.ib', 'ib'),
|
||||
('btc-usd.deribit', 'deribit'),
|
||||
)
|
||||
}
|
||||
return (
|
||||
feed_mod.Feed(
|
||||
mods={
|
||||
broker: SimpleNamespace(name=broker)
|
||||
for broker in roots
|
||||
},
|
||||
portals={},
|
||||
flumes=flumes,
|
||||
streams=roots,
|
||||
status={},
|
||||
),
|
||||
roots,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('cache_hit', [False, True])
|
||||
def test_cached_feed_returns_local_streams(
|
||||
monkeypatch,
|
||||
cache_hit: bool,
|
||||
) -> None:
|
||||
'''
|
||||
A cached feed caller must not consume or replace shared streams.
|
||||
|
||||
`maybe_open_feed()` previously paired provider subscriptions with
|
||||
`Feed.flumes.values()` using `zip()` and installed each child on the
|
||||
cached `Flume`. One caller therefore exposed its lexical receiver
|
||||
to every holder of the shared `Feed`; after exit those descriptors
|
||||
pointed at closed children. The positional pairing also dropped a
|
||||
market whenever one provider supplied multiple flumes.
|
||||
|
||||
Model two real provider broadcasters with two IB markets and one
|
||||
Deribit market. Enter both first-owner and cache-hit paths, then
|
||||
prove the yielded `Feed`, its `Feed.streams` mapping and all
|
||||
`Flume`s are caller-local views. Exercise provider-context-wide
|
||||
pause/resume and `Feed.open_multi_stream()` to show duplex sends
|
||||
still reach the roots while each relay owns another child. Finally
|
||||
prove every lexical child closes and all cached descriptors still
|
||||
point at live roots.
|
||||
|
||||
'''
|
||||
cached_feed: feed_mod.Feed
|
||||
roots: dict[str, FakeMsgStream]
|
||||
cached_feed, roots = make_feed()
|
||||
cached_flumes: dict[str, Flume] = cached_feed.flumes
|
||||
|
||||
@acm
|
||||
async def maybe_open_context(**kwargs):
|
||||
'''
|
||||
Return the controlled shared feed as a cache hit.
|
||||
|
||||
'''
|
||||
yield cache_hit, cached_feed
|
||||
|
||||
monkeypatch.setattr(
|
||||
feed_mod.trionics,
|
||||
'maybe_open_context',
|
||||
maybe_open_context,
|
||||
)
|
||||
|
||||
local_feed: feed_mod.Feed|None = None
|
||||
local_streams: dict[
|
||||
str,
|
||||
tractor.trionics.BroadcastReceiver,
|
||||
]|None = None
|
||||
|
||||
async def main() -> None:
|
||||
nonlocal local_feed, local_streams
|
||||
feed: feed_mod.Feed
|
||||
async with feed_mod.maybe_open_feed(
|
||||
list(cached_flumes),
|
||||
) as feed:
|
||||
local_feed = feed
|
||||
local_streams = feed.streams
|
||||
assert feed is not cached_feed
|
||||
assert feed.streams is not roots
|
||||
assert feed.flumes is not cached_flumes
|
||||
|
||||
broker: str
|
||||
stream: tractor.trionics.BroadcastReceiver
|
||||
for broker, stream in feed.streams.items():
|
||||
assert stream is roots[broker].children[0]
|
||||
assert stream._ctx is roots[broker].ctx
|
||||
|
||||
fqme: str
|
||||
flume: Flume
|
||||
for fqme, flume in feed.flumes.items():
|
||||
cached: Flume = cached_flumes[fqme]
|
||||
|
||||
stream: tractor.trionics.BroadcastReceiver = (
|
||||
feed.streams[flume.mkt.broker]
|
||||
)
|
||||
|
||||
assert flume is not cached
|
||||
assert flume.stream is stream
|
||||
assert cached.stream is roots[flume.mkt.broker]
|
||||
|
||||
assert (
|
||||
feed.flumes['nvda.nasdaq.ib'].stream
|
||||
is feed.flumes['aapl.nasdaq.ib'].stream
|
||||
)
|
||||
|
||||
await feed.pause()
|
||||
await feed.resume()
|
||||
assert all(
|
||||
root.controls == ['pause', 'resume']
|
||||
for root in roots.values()
|
||||
)
|
||||
|
||||
stream: trio.MemoryReceiveChannel
|
||||
async with feed.open_multi_stream() as stream:
|
||||
assert all(
|
||||
len(child._state.subs) == 3
|
||||
for child in feed.streams.values()
|
||||
)
|
||||
await roots['ib'].push({'provider': 'ib'})
|
||||
await roots['deribit'].push({
|
||||
'provider': 'deribit',
|
||||
})
|
||||
with trio.fail_after(1):
|
||||
msgs = {
|
||||
(await stream.receive())['provider'],
|
||||
(await stream.receive())['provider'],
|
||||
}
|
||||
assert msgs == {'ib', 'deribit'}
|
||||
|
||||
# One provider's EOC must close only its `tx` clone;
|
||||
# the sibling relay continues publishing to `stream`.
|
||||
await roots['ib'].close_source()
|
||||
with trio.fail_after(1):
|
||||
while (
|
||||
len(feed.streams['ib']._state.subs)
|
||||
!= 2
|
||||
):
|
||||
await trio.lowlevel.checkpoint()
|
||||
|
||||
await roots['deribit'].push({
|
||||
'provider': 'deribit-after-ib-eoc',
|
||||
})
|
||||
msg: dict
|
||||
msg = await stream.receive()
|
||||
|
||||
assert msg['provider'] == 'deribit-after-ib-eoc'
|
||||
|
||||
assert all(
|
||||
len(child._state.subs) == 2
|
||||
for child in feed.streams.values()
|
||||
)
|
||||
|
||||
trio.run(main)
|
||||
|
||||
assert local_feed is not None
|
||||
assert local_streams is not None
|
||||
assert all(stream._closed for stream in local_streams.values())
|
||||
assert all(
|
||||
flume.stream is roots[flume.mkt.broker]
|
||||
for flume in cached_flumes.values()
|
||||
)
|
||||
assert all(
|
||||
len(root._broadcaster._state.subs) == 1
|
||||
for root in roots.values()
|
||||
)
|
||||
|
||||
|
||||
def test_feed_cache_key_includes_shape_and_policy(
|
||||
monkeypatch,
|
||||
) -> None:
|
||||
'''
|
||||
Feed cache identity must include every resource-defining input.
|
||||
|
||||
The old `fqmes[0]` key aliased `[A]` with `[A, B]` and ignored
|
||||
`tick_throttle`, `allow_overruns` and `start_stream`. Whichever
|
||||
caller entered first silently determined later feed shape and
|
||||
stream behavior, including reuse of a history-only remote feed for
|
||||
a live quote request.
|
||||
|
||||
Capture the key passed to `maybe_open_context()` for reordered and
|
||||
varied requests. Symbol order must normalize to one key, while the
|
||||
full symbol set and each stream policy must produce distinct keys.
|
||||
|
||||
'''
|
||||
feed: feed_mod.Feed = feed_mod.Feed(
|
||||
mods={},
|
||||
portals={},
|
||||
flumes={},
|
||||
streams={},
|
||||
status={},
|
||||
)
|
||||
keys: list[tuple] = []
|
||||
|
||||
@acm
|
||||
async def maybe_open_context(**kwargs):
|
||||
'''
|
||||
Capture cache identity without opening a remote feed.
|
||||
|
||||
'''
|
||||
keys.append(kwargs['key'])
|
||||
yield False, feed
|
||||
|
||||
@acm
|
||||
async def gather_contexts(mngrs):
|
||||
'''
|
||||
Permit the metadata-only empty feed used by this key probe.
|
||||
|
||||
'''
|
||||
assert not mngrs
|
||||
yield ()
|
||||
|
||||
monkeypatch.setattr(
|
||||
feed_mod.trionics,
|
||||
'maybe_open_context',
|
||||
maybe_open_context,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
feed_mod.trionics,
|
||||
'gather_contexts',
|
||||
gather_contexts,
|
||||
)
|
||||
|
||||
async def open_once(
|
||||
fqmes: list[str],
|
||||
**kwargs,
|
||||
) -> None:
|
||||
async with feed_mod.maybe_open_feed(
|
||||
fqmes,
|
||||
**kwargs,
|
||||
):
|
||||
pass
|
||||
|
||||
async def main() -> None:
|
||||
await open_once(['a.ib', 'b.ib'])
|
||||
await open_once(['b.ib', 'a.ib'])
|
||||
await open_once(['a.ib'])
|
||||
await open_once(['a.ib', 'b.ib'], tick_throttle=10)
|
||||
await open_once(
|
||||
['a.ib', 'b.ib'],
|
||||
allow_overruns=False,
|
||||
)
|
||||
await open_once(
|
||||
['a.ib', 'b.ib'],
|
||||
start_stream=False,
|
||||
)
|
||||
|
||||
trio.run(main)
|
||||
|
||||
assert keys[0] == keys[1]
|
||||
assert len(set(keys[2:])) == 4
|
||||
assert keys[0] not in keys[2:]
|
||||
|
|
@ -23,13 +23,35 @@ from piker.accounting import (
|
|||
'fqmes',
|
||||
[
|
||||
# binance
|
||||
(100, {'btcusdt.binance', 'ethusdt.binance'}, False),
|
||||
(100, {
|
||||
# !TODO, write a suite which validates raising against
|
||||
# bad/legacy fqmes such as this!
|
||||
# 'btcusdt.binance',
|
||||
'btcusdt.spot.binance',
|
||||
'ethusdt.spot.binance',
|
||||
}, False),
|
||||
|
||||
# kraken
|
||||
(20, {'ethusdt.kraken', 'xbtusd.kraken'}, True),
|
||||
(20, {
|
||||
# !TODO, write a suite which validates raising against
|
||||
# bad/legacy fqmes such as this!
|
||||
# 'ethusdt.kraken',
|
||||
# 'xbtusd.kraken',
|
||||
|
||||
'ethusdt.spot.kraken',
|
||||
'xbtusd.spot.kraken',
|
||||
}, True),
|
||||
|
||||
# binance + kraken
|
||||
(100, {'btcusdt.binance', 'xbtusd.kraken'}, False),
|
||||
(100, {
|
||||
# !TODO, write a suite which validates raising against
|
||||
# bad/legacy fqmes such as this!
|
||||
# 'btcusdt.binance',
|
||||
# 'xbtusd.kraken',
|
||||
|
||||
'btcusdt.spot.binance',
|
||||
'xbtusd.spot.kraken',
|
||||
}, False),
|
||||
],
|
||||
ids=lambda param: f'quotes={param[0]}@fqmes={param[1]}',
|
||||
)
|
||||
|
|
@ -48,12 +70,17 @@ def test_multi_fqsn_feed(
|
|||
|
||||
if (
|
||||
ci_env
|
||||
and not run_in_ci
|
||||
and
|
||||
not run_in_ci
|
||||
):
|
||||
pytest.skip('Skipping CI disabled test due to feed restrictions')
|
||||
pytest.skip(
|
||||
'CI-disabled-test due to live-feed restrictions'
|
||||
)
|
||||
|
||||
brokers = set()
|
||||
for fqme in fqmes:
|
||||
# ?TODO, add this unpack + normalize check to a symbology
|
||||
# helper fn?
|
||||
brokername, *_ = unpack_fqme(fqme)
|
||||
brokers.add(brokername)
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,153 @@
|
|||
'''
|
||||
Deterministic history-backfill regressions.
|
||||
|
||||
'''
|
||||
from functools import partial
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
from pendulum import datetime
|
||||
import pytest
|
||||
import trio
|
||||
from trio import TaskStatus
|
||||
|
||||
from piker.brokers import (
|
||||
DataUnavailable,
|
||||
NoData,
|
||||
)
|
||||
from piker.data._source import def_iohlcv_fields
|
||||
from piker.tsp._history import (
|
||||
maybe_fill_null_segments,
|
||||
start_backfill,
|
||||
)
|
||||
|
||||
|
||||
def run_exhausted_backfill(get_hist) -> None:
|
||||
'''
|
||||
Run a provider-exhaustion case without actor services.
|
||||
|
||||
'''
|
||||
async def main() -> None:
|
||||
with trio.fail_after(0.5):
|
||||
async with trio.open_nursery() as nursery:
|
||||
await nursery.start(partial(
|
||||
start_backfill,
|
||||
get_hist=get_hist,
|
||||
def_frame_duration=None,
|
||||
mod=SimpleNamespace(name='fake'),
|
||||
mkt=SimpleNamespace(fqme='x.fake'),
|
||||
shm=object(),
|
||||
timeframe=60,
|
||||
backfill_from_shm_index=100,
|
||||
backfill_from_dt=datetime(2026, 1, 2),
|
||||
sampler_stream=object(),
|
||||
backfill_until_dt=datetime(2026, 1, 1),
|
||||
storage=None,
|
||||
write_tsdb=False,
|
||||
))
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def test_data_unavailable_completes() -> None:
|
||||
'''
|
||||
Provider exhaustion exits without orphaned completion waits.
|
||||
|
||||
'''
|
||||
async def get_hist(*args, **kwargs):
|
||||
raise DataUnavailable('history exhausted')
|
||||
|
||||
run_exhausted_backfill(get_hist)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('error_type', [
|
||||
NoData,
|
||||
DataUnavailable,
|
||||
])
|
||||
def test_gap_repair_failure_preserves_sibling(
|
||||
error_type: type[Exception],
|
||||
) -> None:
|
||||
'''
|
||||
An unavailable gap query must not cancel sibling feed tasks.
|
||||
|
||||
Use real null detection and a provider that raises on its first
|
||||
query. After repair returns, release a sibling waiting in the
|
||||
same nursery and require its acknowledgement. This checks local
|
||||
failure containment, not Gateway reset or quote reconnection.
|
||||
|
||||
'''
|
||||
frame: np.ndarray = np.zeros(
|
||||
5,
|
||||
dtype=np.dtype(def_iohlcv_fields),
|
||||
)
|
||||
frame['index'] = np.arange(5)
|
||||
frame['time'] = datetime(2026, 11, 6).timestamp() + np.arange(5)
|
||||
frame['time'][1:3] = 0
|
||||
frame['close'][0] = 100
|
||||
calls: list[dict] = []
|
||||
|
||||
async def get_hist(timeframe: float, **kwargs) -> None:
|
||||
assert timeframe == 1
|
||||
assert kwargs['start_dt'] < kwargs['end_dt']
|
||||
calls.append(kwargs)
|
||||
raise error_type('provider cannot repair gap')
|
||||
|
||||
async def main() -> None:
|
||||
repaired: trio.Event = trio.Event()
|
||||
resume: trio.Event = trio.Event()
|
||||
survived: trio.Event = trio.Event()
|
||||
send, receive = trio.open_memory_channel(1)
|
||||
|
||||
async def sibling(
|
||||
task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
|
||||
) -> None:
|
||||
task_status.started()
|
||||
await resume.wait()
|
||||
survived.set()
|
||||
|
||||
async def repair() -> None:
|
||||
await maybe_fill_null_segments(
|
||||
shm=SimpleNamespace(array=frame, _array=frame),
|
||||
timeframe=1,
|
||||
get_hist=get_hist,
|
||||
sampler_stream=send,
|
||||
mkt=SimpleNamespace(fqme='mnq.cme.fake'),
|
||||
backfill_until_dt=datetime(2026, 11, 5),
|
||||
)
|
||||
repaired.set()
|
||||
|
||||
with trio.fail_after(1):
|
||||
async with trio.open_nursery() as nursery:
|
||||
await nursery.start(sibling)
|
||||
nursery.start_soon(repair)
|
||||
await repaired.wait()
|
||||
resume.set()
|
||||
await survived.wait()
|
||||
nursery.cancel_scope.cancel()
|
||||
|
||||
assert len(calls) == 1
|
||||
assert receive.receive_nowait() == {
|
||||
'broadcast_all': {
|
||||
'backfilling': ('mnq.cme.fake', 1),
|
||||
},
|
||||
}
|
||||
assert (frame['time'] > 0).all()
|
||||
|
||||
trio.run(main)
|
||||
|
||||
|
||||
def test_empty_frame_completes() -> None:
|
||||
'''
|
||||
An empty provider frame is checked before endpoint indexing.
|
||||
|
||||
'''
|
||||
frame = np.empty(
|
||||
0,
|
||||
dtype=np.dtype(def_iohlcv_fields),
|
||||
)
|
||||
|
||||
async def get_hist(*args, **kwargs):
|
||||
end_dt = kwargs['end_dt']
|
||||
return frame, end_dt, end_dt
|
||||
|
||||
run_exhausted_backfill(get_hist)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue