Commit Graph

57 Commits (38d7aa1acdf954ba7a66c02e3467c59822d8a51e)

Author SHA1 Message Date
Gud Boi 84292898ac Retry incomplete IB schedule coverage
A one-day MNQ schedule reply can start in the next session,
leaving the gap's earlier trading hours uncovered. Widen
`Client.history_schedule()` lookbacks up to 31 days, keeping the
requested end fixed and sharing one timeout across attempts.
Preserve unknown coverage when widening cannot cover the range.
Share `SECONDS_PER_DAY` between query and reservation arithmetic,
with one explanation of elapsed versus calendar/session days.

Replay the captured MNQ replies through the allocator: reserve
51,824 candidate slots while excluding the maintenance closure.
Cover bounded retries and timeout cancellation, and document the
query contract. This does not repair existing SHM or parquet.

Prompt-IO: ai/prompt-io/codex/20260924T232405Z_441358e2_schedule_coverage_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:30:25 -05:00
Gud Boi 81e8ba3ca8 Use IB schedules to reserve missing SHM history
Interrupted backfill can leave stored observations separated by
missing trading time. Query dated IB sessions before prepending
stored history and reserve synthetic samples only during covered
open sessions. Keep venue closures compressed and preserve the
indexes of already published rows.

Deats,
- Add `Client.history_schedule()` with UTC coverage, extended hours,
  bounded duration, timeout, and rejection of ambiguous DST times.
- Keep schedule normalization in `venues` and allocation in
  `.tsp._reserve`, with the project AGPL headers.
- Expose schedule and synthetic-range queries through `datad.ib`.
- Bound reservations by available SHM capacity and inspect only the
  newest 32 gaps by default via `newest_gaps`. Zero disables
  queries; leave unknown and oversized ranges unresolved.
- Record synthetic ranges separately from OHLCV. Automatic storage
  writes retain provider observations; placeholders copy close.
- Group the new tests under `tests/providers`.
- Cover request formatting, DST, failure and capacity cases, plus
  real SHM/NativeDB startup placement with a fake provider.
- Document a common CLI/chart segment-repair interface and defer
  coordinated insertion, repair execution, and reader integration.

Validation: 55 targeted tests passed in the working tree; live IB
schedule coverage remains unqualified. After the module moves,
33 affected tests passed again; the local review follow-up passed
18 affected cases, including four new request-budget cases.
The commit runner repeats
lint and isolated regression checks against this exact boundary.

Prompt-IO: ai/prompt-io/codex/20260924T185456Z_1da9b760_schedule_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:30:24 -05:00
Gud Boi 430a7591a9 Factor and bound synthetic SHM null repair
Extract `_copy_close_to_null_rows()` with explicit mutation
bounds, searchable close-copy comments, and preserved activity
fields. Type and document `maybe_fill_null_segments()` including
its distinction from packed timestamp-gap repair.

Disable relative epoch tolerance when validating synthetic
stamps: an overfull reservation must not duplicate the right
boundary. Cover modern epochs, typed empty replies, packed gaps,
and unchanged rejected inputs.

Prompt-IO: ai/prompt-io/codex/20260924T165757Z_1da9b760_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:29:30 -05:00
Gud Boi f106caf59c Preserve IB history request bounds on no data
Complete HMDS no-data replies without silently retrying a day
older. Let `open_history_client()` classify the original request
and leave frame stepping to `start_backfill()`.

Distinguish `request_end_dt` from `oldest_shm_dt` so the first
frame after an empty interval retains its unwritten end bar.
Inline the storage time cutoff and drop the unused two-direction
`diff_history()` helper. Explain both dates and filtering rules.

Add request-boundary tests and real SHM/NativeDB regressions:
all 2,004 supplied bars survive, while the closure stays empty.

Prompt-IO: ai/prompt-io/codex/20260923T164656Z_efa72980_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:29:28 -05:00
Gud Boi 5fc15b6610 Plan gap-checker and backfiller repairs
Lay out a correctness-first path from deterministic regressions
through normalized gap detection, structured backfill completion,
and crash-safe `NativeDB` persistence.

Deats,
- separate checker, storage, provider, and annotation gates
- record confirmed current defects and data invariants
- define commit boundaries and timeout-wrapped verification

Prompt-IO: ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 17:25:55 -05:00
Gud Boi 5f75a37923 Fix IB idle timer arithmetic for packet nanoseconds
Override `NonShittyWrapper._setTimer()` to compute packet silence
in seconds from integer epoch nanoseconds, avoiding upstream's
`datetime` subtraction error after `tcpDataArrived()`. Delegate
pre-traffic datetime and sentinel handling upstream.

Preserve the packet timestamp on expiry and clear the timer before
`timeoutEvent` emission so listeners can safely rearm it. Add three
offline regressions under `tests/providers/` for rescheduling, single
expiry, datetime and sentinel states, disabling, and reentrant
listener rearming. Document the wrapper overrides on the class.

This fixes an explicitly armed packet-idle timer; heartbeat, default
arming, and reconnect behavior are outside this change.

Prompt-IO: ai/prompt-io/opencode/20260924_ib_idle_timer.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:58 -05:00
Gud Boi 350e2ccd4a Raise on IB socket disconnects in proxy and trades
Propagate the authoritative `disconnectedEvent` as a
`ConnectionError` so proxy callers and trade consumers fail loudly
instead of waiting on a dead socket. Preserve the last `apiError`
text as the reason without treating numeric farm/order errors as
socket failures.

Deats,
- Disconnect the one-shot callback before sending the exception.
- Raise when `recv_trade_updates()` loses its socket, and relay
  disconnect exceptions through both Trio-side consumers.
- Exercise real `eventkit` error/disconnect callbacks and preserve
  the supplied commission fill while testing bridge termination.
- Record the aborted integration stage as historical provenance;
  its test passes do not validate this post-abort boundary.

Prompt-IO: ai/prompt-io/opencode/20260916_ib_connectivity_integration.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:58 -05:00
Gud Boi 2347ad6198 Fix IB fill accounting by execution identity
Keep each commission callback's exact `Fill` paired with its
report, so overlapping executions on one contract cannot borrow
each other's fees. Cache positions by account and `conId`, queue
commissioned fills by execution ID until a position arrives, and
accept zero commissions. Reject mismatched execution/report IDs.

Keep `Execution.execId` authoritative when flattening ledger
entries; an empty commission ID must not erase a valid fill. Add
regressions for empty commission IDs, reversed fee arrival, and
positions arriving before or after overlapping executions.

The human live-tested fills and reports the accounting fix works.
Exact-boundary automated checks remain pending; the earlier nine
passes covered the mixed working tree. Included Prompt-IO logs
retain whole-session context, including separate roaming work.

Prompt-IO: ai/prompt-io/opencode/20260916T194052Z_749ca1f0_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:57 -05:00
Gud Boi e4d9f43ffd Fix string errors in `.ib.broker` relay
Normalize structured and raw IB errors without indexing strings as
message dictionaries.

Deats,
- match each error form before request-id handling;
- parse `[code N]` and `[Errno N]` prefixes without crashing;
- cover raw errors through `deliver_trade_events()`.

Prompt-IO: ai/prompt-io/opencode/20260828T063126Z_1606122b_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 17:13:23 -05:00
Gud Boi e3f2e9ce2a Fix sparse IB history prepends in `ShmArray`
IB returns trading samples, not one row per wall-clock period.
Converting elapsed time to a physical prepend offset left 42,720
zero rows in NVDA's 60s buffer even though both valid margins
shared the same timestamp.

Use the actual earliest provider sample as the NativeDB boundary.
After reverse retrieval, prepend older stored rows directly beside
it.

Deats,
- start reverse insertion at the current first index;
- update `ShmArray._first` after every provider prepend;
- trim provider/NativeDB overlap by its timestamp;
- publish the final storage prepend to chart and FSP consumers;
- derive the source `time` field from `ohlc_key_map`;
- cover the full lifecycle with a real multi-frame `ShmArray`.

Prompt-IO: ai/prompt-io/opencode/20260805T195001Z_f0f3c806_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 15:47:52 -05:00
Gud Boi e197e90005 Simplify `store` series and SHM discovery
Make `store ls` show every NativeDB FQME and its available sample
periods by default. Keep `store series` as the detailed view of
each exact sub-series, including its period, size and path. Share
the same optional FQME filter and numerically ordered payload.

Use `store shm` as the canonical volatile-buffer ep and make its
default operation a read-only report. Enter the prior persistence
workflow only through `--write-parquet`, require that flag before
`--reload-parquet-to-shm`. Reject read-only formatting options in
repair mode instead of silently ignoring them.

Also,
- update the qualification docs, debug helper and harness refs;
- cover no-flag listings, command help and SHM mode validation;
- retain exact generation and object-name selection for stale SHM.

Prompt-IO: ai/prompt-io/opencode/20260801T015321Z_74ea2152_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-12 09:48:09 -05:00
Gud Boi d41b68b80c Expose storage state through `piker store`
Make bare storage groups and eps render help, so the CLI is
discoverable without memorizing `Typer` option conventions. Add
read-only `series` and `shm` eps for durable `NativeDB` series and
live history buffers, with `Rich`-table or `JSON` output.

Deats,
- preserve exact `(fqme, period)` identities for `NativeDB`
  series;
- parse exact `(generation, fqme, kind)` names for SHM buffers;
- reconstruct provider dtypes when attaching to live buffers;
- tolerate buffers disappearing between discovery and attach.

Also,
- report expected and observed cadence separately, plus invalid
  rows, ordering defects and ranked gaps;
- add exact `ldshm --shm-name` selection and keep disabled write
  and reload paths from reading undefined state or entering `pdb`;
- document the triage layers and cover help, identity parsing,
  read-only diagnostics, dtype selection and `ldshm` flow.

Prompt-IO: ai/prompt-io/opencode/20260731T165215Z_0846cbd4_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-12 09:48:06 -05:00
Gud Boi 0d2c88ce3e Repair IB startup gaps in `ShmArray`
Keep reverse history and stored SHM layout aligned so sparse IB
frames do not expose permanent zero-time reservations.

Deats,
- omit inclusive query endpoints from SHM but retain full storage
- serialize and bound farm resets, blanks, and cancellations
- run the final null sweep after reverse retrieval completes
- require valid timestamp boundaries for synthetic gap rows
- cover the live QQQ 653-row failure and reset interleavings

Prompt-IO: ai/prompt-io/opencode/20260729T204046Z_ad6c560f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:35:47 -05:00
Gud Boi 892898cfd0 Guard `ldshm` from invalid SHM snapshots
Do not pass unpublished or corrupt timestamp rows into dedupe,
`NativeDB` replacement writes, or live SHM reload.

Deats,
- infer cadence from finite positive observed steps
- choose the smaller observed step when frequency ties
- skip invalid or degenerate snapshots as a whole
- cover the QQQ zero-slot failure with mutation-failing spies

Prompt-IO: ai/prompt-io/opencode/20260729T185406Z_d359b1cb_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:05:35 -05:00
Gud Boi 05e47d0e33 Repair first-slot OHLC sentinels in `NativeDB`
Filter known zero-initialized epoch slots before persisted history
hydrates SHM or merges with a fresh provider frame.

Deats,
- reindex the repaired read view for same-session chart publication
- treat sentinel-only files as empty history for fresh backfill
- reject new first-slot sentinels but preserve modern zero-price bars
- classify raw sentinel evidence separately in `store audit`

Prompt-IO: ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:04:16 -05:00
Gud Boi ad61d6689a Use IB UTC syntax for history `endDateTime`
Serialize aware reverse-fill boundaries as `YYYYMMDD-HH:MM:SS`
before handing them to `ib_async`.

This avoids Gateway error 10314 from its trailing-`UTC` format and
drops the unfinished EST comparison breakpoint from `Client.bars()`.

Prompt-IO: ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:04:05 -05:00
Gud Boi c1b3e17069 Add `xonsh` backfill qualification workflow
Drive known-bad, fresh, restart and append checks from a private
disposable NativeDB root.

Deats,
- bind child config and imports to the marked worktree root
- preserve checksum-matched reports, snapshots and case metadata
- compare timestamp coverage, seam values and persisted structure
- document manual chart runs, gap triage and the expansion matrix

Prompt-IO: ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:36:45 -05:00
Gud Boi 7cec91e287 Add raw OHLCV audit to `NativeDB`
Inspect Parquet bytes directly without opening storage or normalizing
malformed evidence.

Deats,
- report canonical schema, timestamp, index and finite-value state
- retain exact cadence aggregates with bounded endpoint details
- snapshot bytes before parsing and clean up partial copy failures
- reject unsafe FQMEs, links, collisions and non-regular sources

Prompt-IO: ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 50b84d68f2 Repair legacy timestamps in `NativeDB` merges
Keep incoming provider frames strict while dropping non-positive rows
from persisted history before merge validation.

This lets known-bad MNQ history repair itself on the next valid IB
frame without losing valid old bars or publishing noncanonical indexes.

Prompt-IO: ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi a26d69c269 Clean test-owned SHM leaks in `pytest`
Track successful `create=True` calls through Tractor's
current-process factory instead of scanning `/dev/shm` or trusting
the attachment cache.

- cooperate with normal actor unlink and restore token-cache state
- verify POSIX object identity before fallback cleanup
- fail after cleaning exact surviving segments
- reproduce the pre-`current_actor()` allocation leak

Prompt-IO: ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 1b76dd64fd Normalize provider OHLCV for `NativeDB`
IB history frames omit the derived `index` and append `count`.
Normalize provider data to durable names, order, and dtypes before
merge, and reject fractional epoch truncation.

Also,
- map canonical storage fields into provider-specific SHM on reload
- select Polars columns by name instead of position
- exercise first-start and restart through actor, SHM, and Parquet

Prompt-IO: ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:02:36 -05:00
Gud Boi 9851ff886a Drop redundant `NativeDB` writer locks
`datad`'s persistent feed task already owns each series writer.
The actor-local and file locks duplicated this SC contract and left
`.parquet.lock` sidecars that upstream indexed as data.

- preserve validated temp writes, `fsync()`, and atomic replacement
- keep exact-suffix indexing for legacy lock and temp sidecars
- prove full and incremental writes create no lock artifacts

Prompt-IO: ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:33 -05:00
Gud Boi 9cfe2591d9 Make `NativeDB` backfills atomic and non-destructive
- merge provider deltas without dropping unaffected history
- validate reopened temp parquet before atomic replacement
- serialize writers with actor-local and cancellable file locks
- publish cache only after disk replacement
- persist full startup/backfill frames before SHM publication
- bound sampler notifications and avoid MarketStore duplicates

Prompt-IO: ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:32 -05:00
Gud Boi fc532d6acb .tsp: report non-positive timestamp steps
- keep positive missing-history gaps distinct from ordering faults
- return duplicate and reversed timestamp rows as structured data
- warn before normalization removes the original ordering evidence

Prompt-IO: ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 14:32:23 -05:00
Gud Boi 61cd084ef2 .tsp: normalize gaps after sorted dedupe
- recompute datetime and delta cols after final timestamp ordering
- detect only positive sample-step gaps
- document inclusive null grouping with an updated visual map
- check all small null layouts against an independent test oracle
- honor selected null cols and reject discontinuous frame indexes
- share exact null grouping across NumPy and Polars

Prompt-IO: ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 14:32:20 -05:00
Gud Boi 0142476966 .ib.broker: lazily qualify contracts on order req
Post (datad|brokerd)-split the trading actor's
`Client._contracts` cache is never warmed by in-proc
feed setup (that now happens in the `datad.ib` sibling)
so ALL live submissions failed with "no live feed?" at
`Client.submit_limit()`; `brokerd` must be able to
submit orders without any feed registered in its own
subactor.

Deats,
- thread the acct `proxies` table into
  `handle_order_requests()` and, on a `_contracts`
  cache-miss for the req's fqme, lazily run the same
  `get_mkt_info(fqme, proxy=...)` symbology ep the
  feed-side uses; it writes the `mkt.bs_fqme` key
  `submit_limit()` looks up (and warms `_cons2mkts`
  which the position-audit path also needs) on exactly
  the same aio `Client` instance.
- guard `submit_limit()` w/ a try/except ->
  `BrokerdError` relay so a single bad submission
  degrades to an EMS error msg instead of crashing the
  dialog (and causing the `TrioTaskExited` teardown
  storm seen in testing).
- fix the (non-f-string..) raise msg in
  `Client.submit_limit()` and doc the new lazy-qualify
  contract; the bug was foretold by the TODO in
  `.symbols.get_mkt_info()` B)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 5b3370ddbe Add `datad`-split design plan + provenance docs
Commit the AI-authored design doc driving this branch's
7-commit `brokerd` -> (`datad` + `brokerd`) split; all
prior commits' `Prompt-IO:` entries reference its
stages so this makes those refs resolvable in-repo for
PR review.

Also,
- add the doc's own prompt-io entry pair (scope: docs)
  incl. a 4-item implementation-deviation log vs. the
  plan as written.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 4d3f77f9df brokerd: slim RPC caps + `ib` client-id offset
Caps-sec tightening now that `brokerd` is trading-only: NO
`piker.data.*` (feed) mods are RPC-enabled in the (live,
credentialed) trading actor anymore.

Deats,
- drop `_data_mods` for a minimal `_brokerd_service_mods`
  (just `piker.brokers._daemon`); dedup-compose with the
  backend's set in `spawn_brokerd()`.
- `broker_init()` reads the backend's `_brokerd_mods`
  (fallback: `__enable_modules__` for flat backends).
- fail fast in `spawn_brokerd()` via `validate.get_eps()`
  when a backend offers NO live order-ctl eps (eg.
  `kucoin`, `deribit`) -> tells the caller to use
  paper-mode instead of booting a dead actor; analogous
  warning in `datad_init()` for datad-ep-less backends.
- offset `ib`'s default `client_id` per daemon-kind in
  `load_aio_clients()`: post-split BOTH `datad.ib` and
  `brokerd.ib` connect to the same gw/tws endpoint and the
  shared default (6116 + linear retry incrs) would collide
  and burn the full conn-timeout retry cycle; datad gets
  +16, ad-hoc (test/cli) actors +32.
- drop the import-cleanup leftovers (`exceptiongroup`,
  `_FeedsBus` type-only import) and the now-resolved
  "expose datad" TODO in `.cli`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 25e63211c6 .data: cut feed layer over to `datad` actors
The topology flip: all data-feed consumers now route to the
new `datad.<broker>` sibling daemon; `brokerd` becomes
trading-only and is ONLY ever booted lazily by `emsd`'s
`open_brokerd_dialog()` (see prior commit). Chart-only and
paper sessions run with zero (live, credentialed) `brokerd`
procs B)

Deats,
- `open_feed()` -> `maybe_spawn_datad()` (NB: imported
  relative-direct from `._daemon` to dodge a partial-init
  cycle via `piker.service`); flip the `open_feed_bus()`
  actor-name assert to `'datad'`; comment sweep.
- slim `_setup_persistent_brokerd()` to a trading-only
  fixture: console logging + pinned-open ctx; the feed-bus
  alloc moves to `_setup_persistent_datad()` and backend
  `open_trade_dialog()` ctxs own their own task trees.
  (the `piker ledger` ad-hoc actor enters this same slimmed
  fixture - exactly what it needs.)
- repoint data-flavoured spawn sites to `maybe_spawn_datad`:
  `.ui._app` symbol-search (+ rename
  `install_brokerd_search` -> `install_datad_search`),
  `.brokers.core.symbol_search()`, `.brokers.cli`
  `brokercheck`/`record`, legacy kivy `.ui.cli` +
  `option_chain`'s `wait_for_actor()`.
- invert `tests.test_services` expectations: feed/EMS-paper
  flows must spawn `datad.kraken` and `paperboi.kraken`
  with an explicit negative assert that NO `brokerd.kraken`
  service task exists.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi d27f953144 .clearing: lazily spawn `brokerd` from `emsd`
Drop the ONE coupling that forces feed + trading eps into
the same actor: `Router.open_trade_relays()` pulling its
trades-dialog portal from `feed.portals[brokermod]`.
Instead `open_brokerd_dialog()` now (maybe) spawns/finds
`brokerd.<broker>` itself via `maybe_spawn_brokerd()` and
ONLY when a live trades-ep will actually be opened; the
paper-mode short-circuit never touches it, so post
feed-cutover paper sessions will run with zero `brokerd`
procs.

Pre-cutover this is a pure refactor: the registry lookup
just finds the same feed-spawned daemon.

Deats,
- new `open_brokerd_dialog()` sig: portal acquisition moves
  inside via an `acquire_live_portal()` helper; keep an
  explicit `portal: Portal|None` override for the
  `piker ledger` cli which boots its own ad-hoc actor.
- `Router.maybe_open_brokerd_dialog()` drops its `portal`
  param; `open_trade_relays()` drops the `feed.portals`
  lookup entirely.
- `.accounting.cli`: pass `portal=` by keyword.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 4402d391f0 Add `datad` daemon machinery to `.data`
First half of the `brokerd` split: a new per-provider
data-feed-only daemon-actor `datad.<broker>` to (soon) host
all `validate._eps['datad']` eps (live quotes, history
loading, symbology search) leaving `brokerd` for live order
ctl only. Purely additive; nothing routes through it yet.

Deats,
- new `piker.data._daemon` mod mirroring the
  `.brokers._daemon` conventions (and the `samplerd`
  sub-daemon precedent):
  - `_setup_persistent_datad()` lifetime fixture owning the
    actor-global `_FeedsBus` alloc.
  - `datad_init()` building `enable_modules` from the
    backend's `_datad_mods` (falling back to
    `__enable_modules__` for not-yet-split backends) and
    copying `_spawn_kwargs` (critical for `ib`'s
    `infect_asyncio`).
  - `spawn_datad()`/`maybe_spawn_datad()` wrapping
    `Services` + `maybe_spawn_daemon()`.
- add `piker.data._daemon` to `_root_modules` so `pikerd`
  can run `spawn_datad()` requests.
- re-export the spawn eps from `piker.service`.
- add `test_datad_spawn` verifying actor boot + service
  registration via `ensure_service('datad.kraken')`.

Note the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep work) lands symmetrically on both.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
2026-11-09 17:05:39 -05:00
Gud Boi 29c0fb72b9 Declare per-daemon-kind backend mod groups
Prep for the `brokerd` -> (`datad` + `brokerd`) actor split
by having each (split-style) backend declare which of its
submods host which daemon-kind's eps, exactly per the
`piker.data.validate._eps` groupings; `ib` already had
`_brokerd_mods`/`_datad_mods` so extend the convention to
`kraken`, `binance` and `deribit` (and add `'api'` to ib's
datad set since both kinds need the `Client` layer).

`__enable_modules__` stays as the (deduped) union so this
is a ZERO behavior change; flat backends (`kucoin` etc.)
just don't declare the split yet.

Also,
- add `validate.get_eps()` returning a backend's defined
  eps per daemon-kind for spawn-time introspection.
- import `NoBsWs`/`open_autorecon_ws` from
  `piker.data._web_bs` directly in `.kraken.broker` (they
  were only re-exported via `.kraken.feed`) so the trading
  mod doesn't depend on the feed mod for ws primitives.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
2026-11-09 17:02:44 -05:00
Gud Boi b349332e6b Distinguish websocket receive and relay timeouts
The `proxy_msgs()` deadline covers both `ws.get_message()` and
`snd.send()`. Repeated socket-ready reports with no first relay
could therefore mean either a silent source or a blocked local
consumer; the old generic reset log could not distinguish them.

Track the currently awaited operation and report its phase, URL,
deadline, accumulated timeout count and reset threshold on each
timeout. Snapshot `NoBsWs._timeout` once per iteration so the log
reflects the deadline that actually elapsed. Keep the existing
shared deadline, counter behavior and reconnect policy intact.

Add two real loopback regressions: a silent websocket server and
one payload blocked on an unconsumed zero-capacity Trio relay.
Require the matching timeout phase and URL before the reset budget
is exhausted. The 15 selected Binance/websocket cases pass.

The human observes this after invalid `.usdm.` startup followed
by corrected `.usdtm.` input. Metadata accepts the typo but later
history lookup expects canonical keys. Record that investigation
as context; this diagnostic patch does not fix FQME validation,
failed-startup cleanup or prove the cause of corrected-feed stalls.

Prompt-IO: ai/prompt-io/opencode/20261028T180856Z_032580bf_timeout_phase.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 5183d4094f .data: expose websocket reconnect progress at INFO
An INFO pikerd still hid websocket connection-success messages.
The shared ._util.log emitter is piker.data, but open_feed_bus()
configured piker.data.feed. Errors reached bare stderr while
INFO diagnostics remained disabled on the actual emitter.

Configure the shared logger by its exact log.name and prefer
the actor's loglevel over a quieter per-client request for these
actor-wide transport diagnostics. Preserve the selected client
level passed to provider feed startup.

Also,
- retain the existing handshake-success report and add URL-tagged
  readiness after fixture setup and first-message relay once per
  connection, including replacement sockets.
- include the URL in receive-timeout reset errors so Binance's
  independent bookTicker and aggTrade routes are distinguishable.
- prove real console INFO visibility through production feed
  startup, stopping before provider lookup, then restore logging
  state; the original code fails with the emitter at WARNING.
- extend the loopback recovery test to verify both readiness
  reports and the first relay from the replacement connection.

First-message relay is raw websocket traffic, not proof that
parsed market data or chart updates have recovered. The 18
logging/Binance/websocket cases pass; live reporting qualification
remains separate.

This is a local follow-up, not a downstream cherry-pick. The
54da2973 websocket cleanup and 5adf77a9 namespace fix do not
configure this shared emitter. Relevant local histories and
dirty IB-history/TIPC deltas contain no equivalent wiring fix.

Prompt-IO: ai/prompt-io/opencode/20261028T165638Z_b95e7c79_reconnect_logging.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 251987fc4b .data: reconnect after websocket reader close errors
A Binance Ping racing local socket closure makes trio-websocket's
reader send a Pong while wsproto is LOCAL_CLOSING. That reader
nursery error escapes proxy_msgs(), then tears down brokerd and
pikerd instead of restoring quote delivery.

Classify NoBsWs.recon_errors around the entire websocket context,
with collapse_eg() outside it so a singleton reader exit group
reaches the reconnect handler. Invalidate the stale connected
event and socket reference before retrying. Keep retry sleep
unshielded so parent cancellation stops recovery promptly.

Deats,
- Add real loopback websocket coverage: TCP, handshake, Ping,
  reader dispatch and wsproto's actual Pong validation.
- Control only the first client Ping's protocol-state ordering,
  withholding its Close frame to prevent a peer response racing
  the deterministic LOCAL_CLOSING reproducer.
- Require fresh data on a replacement connection, plus exactly
  one connection when cancelling at the reader or retry boundary.
- Use RuntimeError for the unexpected-failure control:
  LocalProtocolError is now a recoverable connection failure.
- Preserve the original unexpected exception with bare raise,
  and assert its identity in the test. Log recoverable errors
  with the websocket URL and the underlying traceback.

This is a local follow-up, not another downstream cherry-pick.
The earlier wrapper suppressed escaped communication failures;
97eb8347 deliberately re-raised them. Retain that outer error
visibility rather than restoring suppression without recovery.
The isolated baseline reproduces the live Pong failure; all 13
selected Binance/websocket cases pass with this fix. Human live
qualification remains separate from these loopback regressions.

Follow-up-to: 97eb834799
Original-equivalent: d50ab6124c
Related-history: 3d83b61f3f
Prompt-IO: ai/prompt-io/opencode/20261028T160846Z_7242dcb9_websocket_close.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 63c763800a Harden Binance USD-M feed startup
- reject non-single-symbol feeds with `FeedInitializationError`
- route book and trade data across the correct USD-M endpoints
- accept the first matching L1 or trade quote as feed readiness
- preserve transport and startup-timeout failure context
- cover provider msg schemas and both readiness classes

Prompt-IO: ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 7a06a620e2 Fix reconnect startup in `_web_bs`
- restart handled handshake failures before touching cancel state
- propagate `trio.Cancelled` without shielded retry throttling
- bound initial connection waits and expose transport failures
- cover real reconnect task paths in `test_websocket.py`

Prompt-IO: ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 3053324f04 Fix Binance USD-M websocket routing
Binance retired its legacy futures websocket and split quote
traffic across `/public` and `/market`. Route `bookTicker` and
`aggTrade` through dedicated URLs, merge both sockets, and accept
the new `st` stream-type field.

Bound connection and first-quote startup so failed or silent
streams raise `DataUnavailable` instead of hanging `datad` and
chart consumers.

Also,
- derive feed mode from resolved `FutesPair` metadata
- keep direct URL subscriptions stable across reconnects
- cover USD-M routes, payloads, and startup timeouts

Prompt-IO: ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi cb5e81d443 Pin qualified `tractor` and require `trio>=0.34.0`
Advance the `piker_pin` lock from `b64c89f0` to the published
`8fe574c1`, retaining the evaluated Trio 0.34.0 release. Require
Trio >=0.34.0 for the updated cancellation APIs. Restore the Git
source used before local editable evaluation; no environment sync
is included.

Record the human's overlapping charts, Gateway data reset and
close/reopen qualification. Keep the full Gateway restart string
error and Tractor teardown callback assertion explicit as separate
recovery limits. Offline lock validation resolves 126 packages;
this is not a fresh-install qualification claim.

Prompt-IO: ai/prompt-io/20261106T201746Z_495c0915_612dfc_prompt_io.md

(this patch was generated in some part by `codex` (`openai`))
2026-11-06 15:21:33 -05:00
Gud Boi 51795a4e07 Contain provider failures during SHM gap repair
Partially backport only the `get_hist()` exception catch from
0846cbd4a4, authored by goodboy
<goodboy_foss@protonmail.com>. Keep the existing gap fallback.

A timed-out IB query raises `NoData` from null-segment repair.
Letting it escape cancels sibling history and live quote tasks
while `pikerd` can remain alive. Catch `NoData` and
`DataUnavailable` locally and stop querying this repair pass.

Add a regression using real null detection and a waiting sibling
in the same nursery. Both provider failures must return without
canceling the sibling. The two cases failed before this catch;
the focused history files passed 17 tests with it.

Partial-backport-of: 0846cbd4a4
Prompt-IO: ai/prompt-io/20261106T185628Z_495c0915_215e9b_prompt_io.md

(this patch was generated in some part by `codex` (`openai`))
2026-11-06 15:20:58 -05:00
Gud Boi 2a826fba17 `.tsp._history`: drop orphaned backfill events
- let structured nursery joins own child-task completion
- stop empty provider frames before timestamp endpoint indexing
- cover provider exhaustion and empty frames with bounded tests

Prompt-IO: ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))

(cherry picked from commit 931f22d235)
2026-11-06 13:41:07 -05:00
Gud Boi 3ec447a45a Document bounded IB recovery and downstream replay
Record exact keep/backout/import provenance for the #94 candidate,
including the quote-cache lineage correction and the distinction
between file-equivalent adapter deltas and partial source commits.

Deats,
- Preserve the original raw prompt record and add the human's
  later scope decisions, corrections and live-test contributions
  to its structured partner.
- Record 38 prior unique offline passes and the known-red bridge
  callback-cleanliness case, without claiming clean teardown.
- Record live QQQ backfill, reattachment and overlapping chart
  close/reopen results. Startup needed debugger continuation;
  the local breakpoint cleanup was applied after that run.
- Retain the cached-client daemon shutdown failure as a human-
  accepted follow-up outside #94, with its exact propagation path.
- Require disposable downstream rebase rehearsals before landing,
  preserving omitted SHM/storage fixes and local regressions.

The commit package records exact per-boundary project checks.
Its structural replay/preflight does not substitute for those
checks, the prior live observations or rebase qualification.

Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:36:29 -04:00
Gud Boi dc618229bd .data: isolate cached feed stream consumers
Keep provider root streams private to the cached `Feed` and give every
`maybe_open_feed()` caller, including the first owner, a local
subscription view with copied `Flume` descriptors.

Also,
- key cached resources by the normalized full symbol set and stream
  policy instead of only the first FQME.
- subscribe each `Feed.open_multi_stream()` provider relay and wait for
  all relays to start before yielding the merged channel.
- give each provider relay an independent send-channel clone so one
  provider EOC cannot terminate its siblings.
- document `Feed.pause()` and `Feed.resume()` as shared remote-provider
  controls rather than caller-local cursor operations.
- cover miss/hit ownership, multi-market provider mapping, relay
  fan-out, lexical teardown and cache-key separation.

Prompt-IO: ai/prompt-io/opencode/20260813T202402Z_12b0d66e_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 635314c757 .ib.feed: cache quote channel contexts
Replace the manual `_quote_streams` raw-channel registry with a
symbol-keyed `maybe_open_context()` resource whose lifetime extends to
the final quote consumer.

Give every `open_aio_quote_stream()` caller a lexical child from the
cached `LinkedTaskChannel` broadcaster so concurrent users receive the
same quotes and close independently.

Prompt-IO: ai/prompt-io/opencode/20260813T230744Z_ddac3553_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 7bef50b0a5 Fix IB `MethodProxy` response routing
`open_client_proxy()` broadcast its one-slot linked channel to an
always-running status relay and an idle method receiver. Status
traffic advanced only the relay, so the first lazy contract lookup
for an uncached order raised `Lagged` before receiving its
response.

Give `relay_client_proxy_messages()` sole receive ownership. Route
tagged responses from concurrent caller tasks into their per-`mid`
slots while each proxy context retains its own channel and relay.

Deats,
- log stale responses with their original method name;
- warn when unresolved-request metadata exceeds its bounded limit;
- drop late responses after caller cancellation;
- preserve the reconnect design note and message-case rationale;
- accept clean channel EOF during proxy teardown;
- cover idle status, concurrency, cancellation and EOF paths.

Prompt-IO: ai/prompt-io/opencode/20260811T205826Z_06d5ea5d_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 4133f07340 Select `piker_pin` and keep `polars` below 2.0
Replace the address-unpacking branch selector with `piker_pin` and
lock the published branch at b64c89f0. Verify the installed Git
revision without a local-source `PYTHONPATH` override.

Bound Polars below 2.0 and resolve polars/runtime 1.44.2, the latest
non-yanked stable 1.x release, instead of taking a major upgrade as
part of the Tractor landing. This resolves the universal lock where
retaining the older 1.36 versions was rejected for a Windows target.

The scoped install changed only Tractor; 14 offline logger/startup
cases passed with the existing Polars 1.36 installation. A full sync
and Polars 1.44.2 data-path qualification remain outstanding.

Prompt-IO: ai/prompt-io/opencode/20261021T224421Z_507c75c6_piker_pin_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-21 19:16:14 -04:00
Gud Boi 507c75c669 Use `tractor`'s reviewed address-unpacking branch
Select `wkt/addr_unpacking` in `pyproject.toml` and lock `tractor`
at `a966928dead4cbf369ad244834ce9dbe988ac8ca` in `uv.lock`.
Keep the existing PR94 application/test tree as the landing base
for the original downstream consumer fixes.

Also,
- resolve `setproctitle` 1.3.8 while retaining `polars` 1.36.0,
- record the dependency selection and original-history previews
  with their Prompt-IO provenance.

Prompt-IO: ai/prompt-io/20261010T232852Z_1fa45dfa_prompt_io.md

(this patch was generated in some part by `opencode`
using `gpt-6.1-sol` (`openai`))
2026-10-11 00:43:06 -04:00
Gud Boi 1fa45dfa98 Fix endpoint config and daemon startup from PR94 review
Keep the shipped `conf.toml` defaults TCP-only for this `tractor`
pin, with an explicit single-transport note and commented UDS
alternatives. Preserve repeated Click `--maddrs` options and treat
a direct string as one endpoint.

Deats,
- Validate daemon bind requests against actual listeners, not
  registrar discovery endpoints; normalize tagged addresses and
  allow OS-assigned TCP ports. Explain registrar ownership and
  automatic client binding at the validation site.
- Resolve explicit or inherited registrar addresses independently
  of other daemon kwargs, consuming overrides before forwarding
  them and keeping discovery and startup on the same registrar.
- Default chart binding to `None` for absent or empty network
  config, and remove the unconditional Binance debugger pause.
- Defer the prematurely cherry-picked `test_datad_spawn` until
  its service API lands. Add offline endpoint/startup regressions
  and a real distinct-registrar/daemon lifecycle check.

Include the human-directed config and registrar/bind explanations
and the direct prompt provenance pair from the review fix pass.

Prompt-IO: ai/prompt-io/20261007T165453Z_bde2457b_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-10 14:20:52 -04:00
Gud Boi bde2457bdd Bump `tractor` and `msgspec` for Python 3.13
- require Python 3.13 across package metadata and the lock
- advance `tractor` from `e77198bb` to `piker_pin@0d95a9e6`
- raise `msgspec` from 0.19.0 to 0.21.1
- lock current multi-address transport deps
- retain `xonsh` 0.24.2 from the flake-update stack

Prompt-IO: ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 1ba9bdbac0 Preserve exact `piker` logger namespaces
Use `str.removeprefix()` instead of character-set stripping so
subloggers such as `piker.paper` retain their requested namespace.

Also,
- drop stale parent-logger assertions from daemon startup paths
- keep logger setup calls without assigning assertion-only locals
- cover both logger factories and prefix-sensitive module names

Prompt-IO: ai/prompt-io/opencode/20260821T042058Z_b7aca4c0_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00