Commit Graph

21 Commits (wkt/land_binance_usdm_feed)

Author SHA1 Message Date
Gud Boi 7952eb0634 Distinguish websocket receive and relay timeouts
The `proxy_msgs()` deadline covers both `ws.get_message()` and
`snd.send()`. Repeated socket-ready reports with no first relay
could therefore mean either a silent source or a blocked local
consumer; the old generic reset log could not distinguish them.

Track the currently awaited operation and report its phase, URL,
deadline, accumulated timeout count and reset threshold on each
timeout. Snapshot `NoBsWs._timeout` once per iteration so the log
reflects the deadline that actually elapsed. Keep the existing
shared deadline, counter behavior and reconnect policy intact.

Add two real loopback regressions: a silent websocket server and
one payload blocked on an unconsumed zero-capacity Trio relay.
Require the matching timeout phase and URL before the reset budget
is exhausted. The 15 selected Binance/websocket cases pass.

The human observes this after invalid `.usdm.` startup followed
by corrected `.usdtm.` input. Metadata accepts the typo but later
history lookup expects canonical keys. Record that investigation
as context; this diagnostic patch does not fix FQME validation,
failed-startup cleanup or prove the cause of corrected-feed stalls.

Prompt-IO: ai/prompt-io/opencode/20261028T180856Z_032580bf_timeout_phase.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-28 14:35:57 -04:00
Gud Boi 032580bfa5 .data: expose websocket reconnect progress at INFO
An INFO pikerd still hid websocket connection-success messages.
The shared ._util.log emitter is piker.data, but open_feed_bus()
configured piker.data.feed. Errors reached bare stderr while
INFO diagnostics remained disabled on the actual emitter.

Configure the shared logger by its exact log.name and prefer
the actor's loglevel over a quieter per-client request for these
actor-wide transport diagnostics. Preserve the selected client
level passed to provider feed startup.

Also,
- retain the existing handshake-success report and add URL-tagged
  readiness after fixture setup and first-message relay once per
  connection, including replacement sockets.
- include the URL in receive-timeout reset errors so Binance's
  independent bookTicker and aggTrade routes are distinguishable.
- prove real console INFO visibility through production feed
  startup, stopping before provider lookup, then restore logging
  state; the original code fails with the emitter at WARNING.
- extend the loopback recovery test to verify both readiness
  reports and the first relay from the replacement connection.

First-message relay is raw websocket traffic, not proof that
parsed market data or chart updates have recovered. The 18
logging/Binance/websocket cases pass; live reporting qualification
remains separate.

This is a local follow-up, not a downstream cherry-pick. The
54da2973 websocket cleanup and 5adf77a9 namespace fix do not
configure this shared emitter. Relevant local histories and
dirty IB-history/TIPC deltas contain no equivalent wiring fix.

Prompt-IO: ai/prompt-io/opencode/20261028T165638Z_b95e7c79_reconnect_logging.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-28 13:13:26 -04:00
Gud Boi b95e7c796e .data: reconnect after websocket reader close errors
A Binance Ping racing local socket closure makes trio-websocket's
reader send a Pong while wsproto is LOCAL_CLOSING. That reader
nursery error escapes proxy_msgs(), then tears down brokerd and
pikerd instead of restoring quote delivery.

Classify NoBsWs.recon_errors around the entire websocket context,
with collapse_eg() outside it so a singleton reader exit group
reaches the reconnect handler. Invalidate the stale connected
event and socket reference before retrying. Keep retry sleep
unshielded so parent cancellation stops recovery promptly.

Deats,
- Add real loopback websocket coverage: TCP, handshake, Ping,
  reader dispatch and wsproto's actual Pong validation.
- Control only the first client Ping's protocol-state ordering,
  withholding its Close frame to prevent a peer response racing
  the deterministic LOCAL_CLOSING reproducer.
- Require fresh data on a replacement connection, plus exactly
  one connection when cancelling at the reader or retry boundary.
- Use RuntimeError for the unexpected-failure control:
  LocalProtocolError is now a recoverable connection failure.
- Preserve the original unexpected exception with bare raise,
  and assert its identity in the test. Log recoverable errors
  with the websocket URL and the underlying traceback.

This is a local follow-up, not another downstream cherry-pick.
The earlier wrapper suppressed escaped communication failures;
97eb8347 deliberately re-raised them. Retain that outer error
visibility rather than restoring suppression without recovery.
The isolated baseline reproduces the live Pong failure; all 13
selected Binance/websocket cases pass with this fix. Human live
qualification remains separate from these loopback regressions.

Follow-up-to: 97eb834799
Original-equivalent: d50ab6124c
Related-history: 3d83b61f3f
Prompt-IO: ai/prompt-io/opencode/20261028T160846Z_7242dcb9_websocket_close.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-28 12:48:06 -04:00
Gud Boi 7242dcb9e1 Harden Binance USD-M feed startup
- reject non-single-symbol feeds with `FeedInitializationError`
- route book and trade data across the correct USD-M endpoints
- accept the first matching L1 or trade quote as feed readiness
- preserve transport and startup-timeout failure context
- cover provider msg schemas and both readiness classes

Prompt-IO: ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-28 10:45:47 -04:00
Gud Boi 97eb834799 Fix reconnect startup in `_web_bs`
- restart handled handshake failures before touching cancel state
- propagate `trio.Cancelled` without shielded retry throttling
- bound initial connection waits and expose transport failures
- cover real reconnect task paths in `test_websocket.py`

Prompt-IO: ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-28 10:45:47 -04:00
Gud Boi a51002eb70 Fix Binance USD-M websocket routing
Binance retired its legacy futures websocket and split quote
traffic across `/public` and `/market`. Route `bookTicker` and
`aggTrade` through dedicated URLs, merge both sockets, and accept
the new `st` stream-type field.

Bound connection and first-quote startup so failed or silent
streams raise `DataUnavailable` instead of hanging `datad` and
chart consumers.

Also,
- derive feed mode from resolved `FutesPair` metadata
- keep direct URL subscriptions stable across reconnects
- cover USD-M routes, payloads, and startup timeouts

Prompt-IO: ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-28 10:45:47 -04:00
Gud Boi 3ec447a45a Document bounded IB recovery and downstream replay
Record exact keep/backout/import provenance for the #94 candidate,
including the quote-cache lineage correction and the distinction
between file-equivalent adapter deltas and partial source commits.

Deats,
- Preserve the original raw prompt record and add the human's
  later scope decisions, corrections and live-test contributions
  to its structured partner.
- Record 38 prior unique offline passes and the known-red bridge
  callback-cleanliness case, without claiming clean teardown.
- Record live QQQ backfill, reattachment and overlapping chart
  close/reopen results. Startup needed debugger continuation;
  the local breakpoint cleanup was applied after that run.
- Retain the cached-client daemon shutdown failure as a human-
  accepted follow-up outside #94, with its exact propagation path.
- Require disposable downstream rebase rehearsals before landing,
  preserving omitted SHM/storage fixes and local regressions.

The commit package records exact per-boundary project checks.
Its structural replay/preflight does not substitute for those
checks, the prior live observations or rebase qualification.

Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:36:29 -04:00
Gud Boi dc618229bd .data: isolate cached feed stream consumers
Keep provider root streams private to the cached `Feed` and give every
`maybe_open_feed()` caller, including the first owner, a local
subscription view with copied `Flume` descriptors.

Also,
- key cached resources by the normalized full symbol set and stream
  policy instead of only the first FQME.
- subscribe each `Feed.open_multi_stream()` provider relay and wait for
  all relays to start before yielding the merged channel.
- give each provider relay an independent send-channel clone so one
  provider EOC cannot terminate its siblings.
- document `Feed.pause()` and `Feed.resume()` as shared remote-provider
  controls rather than caller-local cursor operations.
- cover miss/hit ownership, multi-market provider mapping, relay
  fan-out, lexical teardown and cache-key separation.

Prompt-IO: ai/prompt-io/opencode/20260813T202402Z_12b0d66e_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 635314c757 .ib.feed: cache quote channel contexts
Replace the manual `_quote_streams` raw-channel registry with a
symbol-keyed `maybe_open_context()` resource whose lifetime extends to
the final quote consumer.

Give every `open_aio_quote_stream()` caller a lexical child from the
cached `LinkedTaskChannel` broadcaster so concurrent users receive the
same quotes and close independently.

Prompt-IO: ai/prompt-io/opencode/20260813T230744Z_ddac3553_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 7bef50b0a5 Fix IB `MethodProxy` response routing
`open_client_proxy()` broadcast its one-slot linked channel to an
always-running status relay and an idle method receiver. Status
traffic advanced only the relay, so the first lazy contract lookup
for an uncached order raised `Lagged` before receiving its
response.

Give `relay_client_proxy_messages()` sole receive ownership. Route
tagged responses from concurrent caller tasks into their per-`mid`
slots while each proxy context retains its own channel and relay.

Deats,
- log stale responses with their original method name;
- warn when unresolved-request metadata exceeds its bounded limit;
- drop late responses after caller cancellation;
- preserve the reconnect design note and message-case rationale;
- accept clean channel EOF during proxy teardown;
- cover idle status, concurrency, cancellation and EOF paths.

Prompt-IO: ai/prompt-io/opencode/20260811T205826Z_06d5ea5d_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 4133f07340 Select `piker_pin` and keep `polars` below 2.0
Replace the address-unpacking branch selector with `piker_pin` and
lock the published branch at b64c89f0. Verify the installed Git
revision without a local-source `PYTHONPATH` override.

Bound Polars below 2.0 and resolve polars/runtime 1.44.2, the latest
non-yanked stable 1.x release, instead of taking a major upgrade as
part of the Tractor landing. This resolves the universal lock where
retaining the older 1.36 versions was rejected for a Windows target.

The scoped install changed only Tractor; 14 offline logger/startup
cases passed with the existing Polars 1.36 installation. A full sync
and Polars 1.44.2 data-path qualification remain outstanding.

Prompt-IO: ai/prompt-io/opencode/20261021T224421Z_507c75c6_piker_pin_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-21 19:16:14 -04:00
Gud Boi 507c75c669 Use `tractor`'s reviewed address-unpacking branch
Select `wkt/addr_unpacking` in `pyproject.toml` and lock `tractor`
at `a966928dead4cbf369ad244834ce9dbe988ac8ca` in `uv.lock`.
Keep the existing PR94 application/test tree as the landing base
for the original downstream consumer fixes.

Also,
- resolve `setproctitle` 1.3.8 while retaining `polars` 1.36.0,
- record the dependency selection and original-history previews
  with their Prompt-IO provenance.

Prompt-IO: ai/prompt-io/20261010T232852Z_1fa45dfa_prompt_io.md

(this patch was generated in some part by `opencode`
using `gpt-6.1-sol` (`openai`))
2026-10-11 00:43:06 -04:00
Gud Boi 1fa45dfa98 Fix endpoint config and daemon startup from PR94 review
Keep the shipped `conf.toml` defaults TCP-only for this `tractor`
pin, with an explicit single-transport note and commented UDS
alternatives. Preserve repeated Click `--maddrs` options and treat
a direct string as one endpoint.

Deats,
- Validate daemon bind requests against actual listeners, not
  registrar discovery endpoints; normalize tagged addresses and
  allow OS-assigned TCP ports. Explain registrar ownership and
  automatic client binding at the validation site.
- Resolve explicit or inherited registrar addresses independently
  of other daemon kwargs, consuming overrides before forwarding
  them and keeping discovery and startup on the same registrar.
- Default chart binding to `None` for absent or empty network
  config, and remove the unconditional Binance debugger pause.
- Defer the prematurely cherry-picked `test_datad_spawn` until
  its service API lands. Add offline endpoint/startup regressions
  and a real distinct-registrar/daemon lifecycle check.

Include the human-directed config and registrar/bind explanations
and the direct prompt provenance pair from the review fix pass.

Prompt-IO: ai/prompt-io/20261007T165453Z_bde2457b_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-10 14:20:52 -04:00
Gud Boi bde2457bdd Bump `tractor` and `msgspec` for Python 3.13
- require Python 3.13 across package metadata and the lock
- advance `tractor` from `e77198bb` to `piker_pin@0d95a9e6`
- raise `msgspec` from 0.19.0 to 0.21.1
- lock current multi-address transport deps
- retain `xonsh` 0.24.2 from the flake-update stack

Prompt-IO: ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 1ba9bdbac0 Preserve exact `piker` logger namespaces
Use `str.removeprefix()` instead of character-set stripping so
subloggers such as `piker.paper` retain their requested namespace.

Also,
- drop stale parent-logger assertions from daemon startup paths
- keep logger setup calls without assigning assertion-only locals
- cover both logger factories and prefix-sensitive module names

Prompt-IO: ai/prompt-io/opencode/20260821T042058Z_b7aca4c0_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 69a6edd6f8 Normalize registrar addrs with `wrap_address()`
Canonicalize legacy socket pairs and tagged tuples before storing
or comparing `Registry.addrs`, keeping existing config readable
when tractor emits protocol-tagged forms.

Also,
- broaden registry/runtime addr annotations for both wire shapes
- compare service fixture addrs through tractor's address codec
- use tractor's shared `--ll` pytest option instead of
  registering it twice
- cover mixed legacy/tagged registry state and `pikerd` startup

Prompt-IO: ai/prompt-io/opencode/20260820T163519Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi bbe9deaa62 Port service+tests to latest `tractor` APIs
Continue the `repair_tests`-branch mission (already merged
in this stack's ancestry, see f4c4f1e2 which ported
`conftest.py`) by fixing the remaining drift breakage vs.
`tractor` git `main`; without these NOTHING boots since the
`tractor.Address` port in 604e5fcf.

Deats,
- normalize reg addrs via `wrap_address()` in
  `open_pikerd()` before `.unwrap()`-ing; entries may be
  raw `tuple`s when passed in from (test) client code.
- port `check_for_service()` to `query_actor(regaddr=)`
  (was `arbiter_sockaddr=`) incl. its 2-tuple yield and
  the now-required `open_registry(addrs=)` arg.
- `wait_for_actor(registry_addr=)` + `.chan.raddr.unwrap()`
  raw-tuple compares in `test_runtime_boot` and
  `ensure_service()`.
- update `run_test_w_cancel_method()` for modern `tractor`
  cancel semantics: self-requested sub-service cancels are
  absorbed (no `ContextCancelled` raised to the opener) and
  single-exc groups collapse to a bare KBI.
- `RemoteActorError.boxed_type` (was `.type`) and
  `Position.cumsize` (was `.size`) renames in tests.
- bump the paper-EMS startup budget 9 -> 19s; it includes
  a live (kraken) symbology fetch so needs net headroom.
- woops, add the missing comma in `.deribit.api`'s
  `tractor.trionics` import tuple..

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
2026-10-07 03:10:41 -04:00
Gud Boi 70e8d60326 Fix `pytest` config-dir isolation in subactors
The old (commented-out) `get_app_dir()` override gated on
`'pytest' in sys.modules` which can NEVER work in spawned
subactors (fresh procs, no pytest import); as a result test
`paperboi`/daemon actors were writing into the user's REAL
`~/.config/piker/accounting/` files.. friggin yikes.

Deats,
- add `config._maybe_use_test_dir()` which lazily (at
  conf-path access time, NOT import time) reads the
  `piker_test_dir` entry from
  `tractor.runtime._state._runtime_vars['piker_vars']` as
  pre-loaded by `open_piker_runtime()` from the
  `tests.conftest._open_test_pikerd()` overrides.
- hook it in `get_conf_dir()` and route `get_conf_path()`
  + `load()`'s mkdir through `get_conf_dir()`.
- route `.accounting._ledger` / `._pos` dir derivation
  through `config.get_conf_dir()` (was reading the
  `_config_dir` global directly, bypassing the override);
  also `mkdir(parents=True, exist_ok=True)` for nested
  tmp-dir creation.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
2026-10-07 03:10:41 -04:00
Gud Boi 6c8709d309 Bump `xonsh` to 0.24.2
Use the current xonsh release in the Nix-managed REPL environment.

Deats,
- require xonsh 0.24.2 for Python 3.13 and Nix fixes;
- refresh only xonsh's source and wheel lock metadata;
- retain dotrc compatibility for shells outside this project;
- record Prompt-IO provenance for the dependency update.

Prompt-IO: ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 45d0f003fc Move Piker skills to `.agents` discovery
- keep repository-owned skill sources provider-neutral
- retain Claude discovery through portable relative links
- drop tracked local settings and the superseded style guide

Prompt-IO: ai/prompt-io/opencode/20260908T043134Z_f148fbcc_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-5.6-sol` (`openai`))
2026-09-08 00:35:01 -04:00
Gud Boi f148fbcc23 Add neutral `.ai` workflow config
Select the provider-neutral backend and port repo-owned commit-msg
and test-harness refs into `.ai`.

Also,
- ignore local `.agents` links and generated runtime state.
- retain legacy Claude archives for recovery.
- record Prompt-IO provenance for the migration.

Prompt-IO: ai/prompt-io/opencode/20260907T214000Z_9115d24_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-09-07 19:07:00 -04:00