Compare commits

..

108 Commits

Author SHA1 Message Date
Gud Boi e187c50081 Plan `pytest-qt` chart UI audit layers
Distill the gap-overlay E2E findings into a staged audit plan for
the full PyQtGraph iface. Keep Qt and graphics real while replacing
feeds, services and provider search with deterministic seams.

Deats,
- define the headless PyQt6 env and pytest-qt fixture ownership;
- layer primitive, input, SHM, full-window and actor checks;
- guard window SIGINT, provider search and overlapping symbol boot;
- require signal/postcondition waits and lifecycle leak assertions;
- split reviewable PRs before distilling a repo-local UI skill.

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 2987d17202 Test gap dialogs through a real `tractor` actor
Schema checks do not exercise ctx discovery, directional decoding,
stream setup, request correlation or endpoint-finally cleanup.

Boot an isolated child actor around `remote_gap_overlays()` and
drive it through `AnnotClient`. Cancel one published request, leave
its reply queued and prove the next request discards that stale
msg.

Also,
- verify generated request IDs and remote-owner cleanup;
- document automated coverage and broker-backed manual boundaries.

Prompt-IO: ai/prompt-io/opencode/20260817T222036Z_f65df6a1_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 81373f6a2e Fix focused gap routing in `ChartView`
The keyboard loop treated every display not owning the focused
history chart as a realtime target. One `Ctrl+G` could therefore
toggle unrelated cached symbols.

Match the focused chart explicitly before selecting its timeframe.
Factor startup registration and toggle routing into testable
helpers, and use scoped PyQt6 enums throughout the input path.

Deats,
- preserve independent 60s startup layers per display;
- toggle only the focused 1s or 60s chart;
- drive the production event relay with a real `QKeyEvent`.

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 6e458e4d46 Fix Qt indexing in `GapAnnotations.reposition()`
`prepareGeometryChange()` ran after the gap rects and arrow specs
were mutated. Qt could retain the item's old scene index even
though `boundingRect()` returned its new prepended-history
position.

Invalidate cached bounds and notify Qt before changing any
geometry. Cover both scene lookup positions with a real offscreen
item.

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 47e82b8cac Fix `PlotItem` cleanup in `GapOverlayMngr`
`PlotItem.addItem()` records overlays outside the Qt scene. Direct
scene removal leaves stale plot refs and range bookkeeping behind.

Track each annotation's owning plot and remove it through the same
API used for insertion. Exclude pixel-sized arrows from data
bounds.

Deats,
- centralize replacement, hiding and owner teardown;
- retain scene removal as a defensive fallback;
- cover real Qt registries and duplicate-FQME chart identity.

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 772c581cb1 Own gap overlays in chart-local `GapOverlayMngr`
Detect positive timestamp gaps from chart OHLCV SHM with one
vectorized pass. Route local UI and remote callers through one
`GapOverlayMngr` renderer, and key each layer by owner, chart,
FQME and timeframe so remote ctx teardown cannot remove local
graphics.

Deats,
- add tagged `SetGapOverlay` requests and typed
  `GapOverlay` responses;
- keep `remote_annotate()` intact and serve its typed dialog from
  a dedicated `remote_gap_overlays()` Tractor ctx;
- admit `SetGapOverlay`, `GapOverlay`, `list[str]` and `None` in
  the Tractor payload spec, then strictly narrow requests to
  `SetGapOverlay` and replies to `GapOverlay`;
- serialize shared-stream dialogs and correlate request IDs so
  stale replies cannot skew later exchanges;
- rename `AnnotCtl` / `open_annot_ctl()` to `AnnotClient` /
  `open_annot_client()` across storage and annotation callers;
- enable 60s history overlays at startup and toggle the focused
  1s or 60s chart directly with `Ctrl+G`;
- guard redraws by chart, FQME and timeframe; tolerate empty
  arrays and out-of-range lookups; update rect and arrow geometry;
- cover detector geometry, msgpack roundtrips, schema rejection,
  typed stale-FQME errors and renamed storage seams;
- document ownership, refresh, stack landing and live
  qualification.

Prompt-IO: ai/prompt-io/opencode/20260814T032716Z_2aa66c2c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 31670417f0 Preserve gap-annotation profiling handoff
Track the untracked Claude session note with its benchmark, client
bottleneck, and follow-up optimization sketch intact.

(this commit msg was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-14 16:04:10 -05:00
Gud Boi 8b726bc533 Pin `pg` at latest official `0.14.0` release
Keep in masked GH sources lines for easy hackin against upstream
`master` branch when needed as well!
2026-11-14 16:04:10 -05:00
Gud Boi d5b6da1949 Add batch-submit API for gap annotations
Introduce `AnnotCtl.add_batch()` and `serve_rc_annots()` batch
handler to submit 1000s of gaps in single IPC msg instead of
per-annot round-trips. Server builds `GapAnnotations` from specs
and handles vectorized timestamp-to-index lookups.

Deats,
- add `'cmd': 'batch'` handler in `serve_rc_annots()`
- vectorized timestamp lookup via `np.searchsorted()` + masking
- build `gap_specs: list[dict]` from rect+arrow specs client-side
- create single `GapAnnotations` item for all gaps server-side
- handle `GapAnnotations.reposition()` in redraw handler
- add profiling to batch path for perf measurement
- support optional individual arrows for A/B comparison

Also,
- refactor `markup_gaps()` to collect specs + single batch call
- add `no_qt_updates()` context mgr for batch render ops
- add profiling to annotation teardown path
- add `GapAnnotations` case to `rm_annot()` match block

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 16:04:10 -05:00
Gud Boi a03884196c Add a `GapAnnotations` path-renderer
For a ~1000x perf gain says ol' claudy, our boi who wrote this entire
patch! Bo

Introduce `GapAnnotations` in `.ui._annotate` for batch-rendering gap
rects/arrows instead of individual `QGraphicsItem` instances. Uses
upstream's `pyqtgraph.Qt.internals.PrimitiveArray` for rects and
a `QPainterPath` for arrows. This API-replicates our prior annotator's
in view shape-graphics but now using (what we're dubbing)
"single-array-multiple-graphics" tech much like our `.ui._curve`
extensions to `pg` B)

Impl deats,
- batch draw ~1000 gaps in single paint call vs 1000 items
- arrows render in scene coords to maintain pixel size on zoom
- add vectorized timestamp-to-index lookup for repositioning
- cache bounding rect, rebuild on `reposition()` calls
- match `SelectRect` + `ArrowItem` visual style/colors
- skip reposition when timeframe doesn't match gap's period

Other,
- fix typo in `LevelMarker` docstring: "graphich" -> "graphic"
- reflow docstring in `qgo_draw_markers()` to 67 char limit

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 16:04:10 -05:00
Gud Boi 2946435999 Bump to latest official `pyqtgraph` release 2026-11-14 16:04:10 -05:00
Tyler Goodlet a7c5dc2f2b Uhh ok, apparently this osenv var is just idiotic?
See https://www.pikers.dev/pikers/piker/issues/50#issuecomment-1041 for
deatz.
2026-11-14 15:41:51 -05:00
Gud Boi 379f757832 Gate `size_to_values()` on macOS in `_axes.py`
NOTE, this reversion was discovered as needed by @goodboy after
extensively manually testing the new zoom-by-font-size feats introduced
alongside macOS support.

Use class-body `if _friggin_macos:` branching to
conditionally define `size_to_values()` for both
`PriceAxis` and `DynamicDateAxis` — macOS gets the
new `_updateWidth()`/`_updateHeight()` + geometry
recalc path, other platforms fall back to the
original `setWidth()`/`setHeight()` calls.

Deats,
- add `platform` import and module-level
  `_friggin_macos: bool` flag.
- `PriceAxis.size_to_values()`: macOS branch calls
  `_updateWidth()` + `updateGeometry()`; else branch
  uses `self.setWidth(self.typical_br.width())`.
- `DynamicDateAxis.size_to_values()`: macOS branch
  calls `_updateHeight()` + `updateGeometry()`; else
  uses `self.setHeight(self.typical_br.height() + 1)`.
- reorder imports: `platform` before `typing`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 15:41:51 -05:00
Gud Boi bcca507462 Improve styling and logging for UI font-size zoom
Refine zoom methods in `MainWindow` and font helpers
in `_style` to return `px_size` up the call chain and
log detailed zoom state on each change.

Deats,
- make `_set_qfont_px_size()` return `self.px_size`.
- make `configure_to_dpi()` and `_config_fonts_to_screen()`
  return the new `px_size` up through the call chain.
- add `font_size` to `log.info()` in `zoom_in()`,
  `zoom_out()`, and `reset_zoom()` alongside
  `zoom_step` and `zoom_level(%)`.
- reformat `has_ctrl`/`_has_shift` bitwise checks and
  key-match tuples to multiline style.
- comment out `Shift` modifier requirement for zoom
  hotkeys (now `Ctrl`-only).
- comment out unused `mn_dpi` and `dpi` locals.

Also,
- convert all single-line docstrings to `'''` multiline
  style across zoom and font methods.
- rewrap `configure_to_dpi()` docstring to 67 chars.
- move `from . import _style` to module-level import
  in `_window.py`.
- drop unused `screen` binding in `boundingRect()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 15:41:51 -05:00
di1ara 7ff028ad78 improve ui zoom defaults 2026-11-14 15:41:51 -05:00
Gud Boi a440f47254 Fix chart axis scaling on UI zoom level change
Again a patch (vibed) from our very own @dnks
(just a commit msg reworking using his new `/commit-msg` skill added by
@goodboy B)

Deats,
- add `Axis.update_fonts()` to recalculate tick font, text offset,
  bounding rect and `pyqtgraph`'s internal text-width/height tracking
  after a zoom change; store `_typical_max_str` at init for later reuse.
- rework `PriceAxis.size_to_values()` and
  `DynamicDateAxis.size_to_values()` to use pyqtgraph's
  `_updateWidth()`/`_updateHeight()` with `updateGeometry()` instead of
  raw `setWidth()`/ `setHeight()` so auto-expand constraints are
  respected.
- fix `GlobalZoomEventFilter` to mask out `KeypadModifier` and
  explicitly require both Ctrl+Shift, letting plain Ctrl+Plus/Minus pass
  through to chart zoom.
- add `_update_chart_axes()` to walk all plot-item axes during
  `_apply_zoom()` and call `splits.resize_sidepanes()` to sync subplot
  widths.

(this commit msg, and likely patch, was generated in some part by
[`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 15:41:51 -05:00
Gud Boi a31a014d4d Add global UI font-size zoom scaling (from @dnks)
Add `Ctrl+Shift+Plus/Minus/0` shortcuts for zooming all
UI widget font sizes via a `GlobalZoomEventFilter`
installed at the `QApplication` level.

Deats,
- `.ui._window`: add `GlobalZoomEventFilter` event
  filter class and `MainWindow.zoom_in/out/reset_zoom()`
  methods that reconfigure `DpiAwareFont` with a
  `zoom_level` multiplier then propagate to all child
  widgets.
- `.ui._style`: extend `DpiAwareFont.configure_to_dpi()`
  and `_config_fonts_to_screen()` to accept a
  `zoom_level` float multiplier; cast `px_size` to `int`.
- `.ui._forms`: add `update_fonts()` to `Edit`,
  `Selection`, `FieldsForm` and `FillStatusBar` for
  stylesheet regen.
- `.ui._label`: add `FormatLabel.update_font()` method.
- `.ui._position`: add `SettingsPane.update_fonts()`.
- `.ui._search`: add `update_fonts()` to `CompleterView`
  and `SearchWidget`.
- `.ui._exec`: install the zoom filter on window show.
- `.ui.qt`: import `QObject` from `PyQt6`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 15:41:51 -05:00
Tyler Goodlet f367d72a3f Add `.xsh` script mentioned in gitea #50
Note since it's actually `xonsh` code run with either,
- most pedantically: `xonsh ./snippets/calc_ppi.xsh`
- or relying on how shebang: `./snippets/calc_ppi.xsh`
  * an sheboom.
2026-11-14 15:41:51 -05:00
Tyler Goodlet 8b7b444737 Reorder imports in `qt_screen_info.py` ??
For wtv reason on nixos importing `pyqtgraph` first is causing `numpy`
to fail import?? No idea, but likely something to do with recent
`flake.nix`'s ld-lib-linking with `<nixpkgs>` marlarky?
2026-11-14 15:41:51 -05:00
Tyler Goodlet 3c30af9b23 Add some Qt DPI extras to `qt_screen_info.py`
- set `QT_USE_PHYSICAL_DPI='1'` env var for Qt6 high-DPI
  * we likely want to do this in `piker.ui` as well!
- move `pxr` calc from widget to per-screen in loop.
- add `unscaled_size` calc using `pxr * size`.
- switch from `.availableGeometry()` to `.geometry()` for full
  bounds.
- shorten output labels, add `!r` repr formatting
- add Qt6 DPI rounding policy TODO with doc links

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-11-14 15:41:51 -05:00
Tyler Goodlet 312bf66ed9 Re-fmt and `.info()` the `.configure_to_dpi()` DPI calcs for now 2026-11-14 15:41:51 -05:00
di1ara 73633ce129 fixed spacing 2026-11-14 15:41:51 -05:00
di1ara 38d7aa1acd fixed pytest test for dpi font auto calculation 2026-11-14 15:41:51 -05:00
di1ara 8904b15e9a added pytest, moved dependencies 2026-11-14 15:41:51 -05:00
di1ara 7e6f29397c fix DpiAwareFont default size calculation 2026-11-14 15:41:51 -05:00
Gud Boi 84292898ac Retry incomplete IB schedule coverage
A one-day MNQ schedule reply can start in the next session,
leaving the gap's earlier trading hours uncovered. Widen
`Client.history_schedule()` lookbacks up to 31 days, keeping the
requested end fixed and sharing one timeout across attempts.
Preserve unknown coverage when widening cannot cover the range.
Share `SECONDS_PER_DAY` between query and reservation arithmetic,
with one explanation of elapsed versus calendar/session days.

Replay the captured MNQ replies through the allocator: reserve
51,824 candidate slots while excluding the maintenance closure.
Cover bounded retries and timeout cancellation, and document the
query contract. This does not repair existing SHM or parquet.

Prompt-IO: ai/prompt-io/codex/20260924T232405Z_441358e2_schedule_coverage_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:30:25 -05:00
Gud Boi 81e8ba3ca8 Use IB schedules to reserve missing SHM history
Interrupted backfill can leave stored observations separated by
missing trading time. Query dated IB sessions before prepending
stored history and reserve synthetic samples only during covered
open sessions. Keep venue closures compressed and preserve the
indexes of already published rows.

Deats,
- Add `Client.history_schedule()` with UTC coverage, extended hours,
  bounded duration, timeout, and rejection of ambiguous DST times.
- Keep schedule normalization in `venues` and allocation in
  `.tsp._reserve`, with the project AGPL headers.
- Expose schedule and synthetic-range queries through `datad.ib`.
- Bound reservations by available SHM capacity and inspect only the
  newest 32 gaps by default via `newest_gaps`. Zero disables
  queries; leave unknown and oversized ranges unresolved.
- Record synthetic ranges separately from OHLCV. Automatic storage
  writes retain provider observations; placeholders copy close.
- Group the new tests under `tests/providers`.
- Cover request formatting, DST, failure and capacity cases, plus
  real SHM/NativeDB startup placement with a fake provider.
- Document a common CLI/chart segment-repair interface and defer
  coordinated insertion, repair execution, and reader integration.

Validation: 55 targeted tests passed in the working tree; live IB
schedule coverage remains unqualified. After the module moves,
33 affected tests passed again; the local review follow-up passed
18 affected cases, including four new request-budget cases.
The commit runner repeats
lint and isolated regression checks against this exact boundary.

Prompt-IO: ai/prompt-io/codex/20260924T185456Z_1da9b760_schedule_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:30:24 -05:00
Gud Boi 430a7591a9 Factor and bound synthetic SHM null repair
Extract `_copy_close_to_null_rows()` with explicit mutation
bounds, searchable close-copy comments, and preserved activity
fields. Type and document `maybe_fill_null_segments()` including
its distinction from packed timestamp-gap repair.

Disable relative epoch tolerance when validating synthetic
stamps: an overfull reservation must not duplicate the right
boundary. Cover modern epochs, typed empty replies, packed gaps,
and unchanged rejected inputs.

Prompt-IO: ai/prompt-io/codex/20260924T165757Z_1da9b760_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:29:30 -05:00
Gud Boi f106caf59c Preserve IB history request bounds on no data
Complete HMDS no-data replies without silently retrying a day
older. Let `open_history_client()` classify the original request
and leave frame stepping to `start_backfill()`.

Distinguish `request_end_dt` from `oldest_shm_dt` so the first
frame after an empty interval retains its unwritten end bar.
Inline the storage time cutoff and drop the unused two-direction
`diff_history()` helper. Explain both dates and filtering rules.

Add request-boundary tests and real SHM/NativeDB regressions:
all 2,004 supplied bars survive, while the closure stays empty.

Prompt-IO: ai/prompt-io/codex/20260923T164656Z_efa72980_prompt_io.md

(this patch was generated in some part by `codex` using
`gpt-6` (`openai`))
2026-11-13 17:29:28 -05:00
Gud Boi 315bfcdf5a Plan known-FQME backfill qualification
- preserve dirty main-tree work while transferring branch ownership
- hold provider, FQME, timeframe, and symptom fixed while debugging
- capture read-only baselines before disposable live runs
- turn each reproduced failure into deterministic coverage
- qualify interruption and restart behavior after the normal lifecycle

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 17:25:55 -05:00
Gud Boi 5fc15b6610 Plan gap-checker and backfiller repairs
Lay out a correctness-first path from deterministic regressions
through normalized gap detection, structured backfill completion,
and crash-safe `NativeDB` persistence.

Deats,
- separate checker, storage, provider, and annotation gates
- record confirmed current defects and data invariants
- define commit boundaries and timeout-wrapped verification

Prompt-IO: ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 17:25:55 -05:00
Gud Boi 5f75a37923 Fix IB idle timer arithmetic for packet nanoseconds
Override `NonShittyWrapper._setTimer()` to compute packet silence
in seconds from integer epoch nanoseconds, avoiding upstream's
`datetime` subtraction error after `tcpDataArrived()`. Delegate
pre-traffic datetime and sentinel handling upstream.

Preserve the packet timestamp on expiry and clear the timer before
`timeoutEvent` emission so listeners can safely rearm it. Add three
offline regressions under `tests/providers/` for rescheduling, single
expiry, datetime and sentinel states, disabling, and reentrant
listener rearming. Document the wrapper overrides on the class.

This fixes an explicitly armed packet-idle timer; heartbeat, default
arming, and reconnect behavior are outside this change.

Prompt-IO: ai/prompt-io/opencode/20260924_ib_idle_timer.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:58 -05:00
Gud Boi 350e2ccd4a Raise on IB socket disconnects in proxy and trades
Propagate the authoritative `disconnectedEvent` as a
`ConnectionError` so proxy callers and trade consumers fail loudly
instead of waiting on a dead socket. Preserve the last `apiError`
text as the reason without treating numeric farm/order errors as
socket failures.

Deats,
- Disconnect the one-shot callback before sending the exception.
- Raise when `recv_trade_updates()` loses its socket, and relay
  disconnect exceptions through both Trio-side consumers.
- Exercise real `eventkit` error/disconnect callbacks and preserve
  the supplied commission fill while testing bridge termination.
- Record the aborted integration stage as historical provenance;
  its test passes do not validate this post-abort boundary.

Prompt-IO: ai/prompt-io/opencode/20260916_ib_connectivity_integration.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:58 -05:00
Gud Boi 2347ad6198 Fix IB fill accounting by execution identity
Keep each commission callback's exact `Fill` paired with its
report, so overlapping executions on one contract cannot borrow
each other's fees. Cache positions by account and `conId`, queue
commissioned fills by execution ID until a position arrives, and
accept zero commissions. Reject mismatched execution/report IDs.

Keep `Execution.execId` authoritative when flattening ledger
entries; an empty commission ID must not erase a valid fill. Add
regressions for empty commission IDs, reversed fee arrival, and
positions arriving before or after overlapping executions.

The human live-tested fills and reports the accounting fix works.
Exact-boundary automated checks remain pending; the earlier nine
passes covered the mixed working tree. Included Prompt-IO logs
retain whole-session context, including separate roaming work.

Prompt-IO: ai/prompt-io/opencode/20260916T194052Z_749ca1f0_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-11-13 17:17:57 -05:00
Gud Boi e4d9f43ffd Fix string errors in `.ib.broker` relay
Normalize structured and raw IB errors without indexing strings as
message dictionaries.

Deats,
- match each error form before request-id handling;
- parse `[code N]` and `[Errno N]` prefixes without crashing;
- cover raw errors through `deliver_trade_events()`.

Prompt-IO: ai/prompt-io/opencode/20260828T063126Z_1606122b_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 17:13:23 -05:00
Gud Boi e3f2e9ce2a Fix sparse IB history prepends in `ShmArray`
IB returns trading samples, not one row per wall-clock period.
Converting elapsed time to a physical prepend offset left 42,720
zero rows in NVDA's 60s buffer even though both valid margins
shared the same timestamp.

Use the actual earliest provider sample as the NativeDB boundary.
After reverse retrieval, prepend older stored rows directly beside
it.

Deats,
- start reverse insertion at the current first index;
- update `ShmArray._first` after every provider prepend;
- trim provider/NativeDB overlap by its timestamp;
- publish the final storage prepend to chart and FSP consumers;
- derive the source `time` field from `ohlc_key_map`;
- cover the full lifecycle with a real multi-frame `ShmArray`.

Prompt-IO: ai/prompt-io/opencode/20260805T195001Z_f0f3c806_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-13 15:47:52 -05:00
Gud Boi e197e90005 Simplify `store` series and SHM discovery
Make `store ls` show every NativeDB FQME and its available sample
periods by default. Keep `store series` as the detailed view of
each exact sub-series, including its period, size and path. Share
the same optional FQME filter and numerically ordered payload.

Use `store shm` as the canonical volatile-buffer ep and make its
default operation a read-only report. Enter the prior persistence
workflow only through `--write-parquet`, require that flag before
`--reload-parquet-to-shm`. Reject read-only formatting options in
repair mode instead of silently ignoring them.

Also,
- update the qualification docs, debug helper and harness refs;
- cover no-flag listings, command help and SHM mode validation;
- retain exact generation and object-name selection for stale SHM.

Prompt-IO: ai/prompt-io/opencode/20260801T015321Z_74ea2152_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-12 09:48:09 -05:00
Gud Boi d41b68b80c Expose storage state through `piker store`
Make bare storage groups and eps render help, so the CLI is
discoverable without memorizing `Typer` option conventions. Add
read-only `series` and `shm` eps for durable `NativeDB` series and
live history buffers, with `Rich`-table or `JSON` output.

Deats,
- preserve exact `(fqme, period)` identities for `NativeDB`
  series;
- parse exact `(generation, fqme, kind)` names for SHM buffers;
- reconstruct provider dtypes when attaching to live buffers;
- tolerate buffers disappearing between discovery and attach.

Also,
- report expected and observed cadence separately, plus invalid
  rows, ordering defects and ranked gaps;
- add exact `ldshm --shm-name` selection and keep disabled write
  and reload paths from reading undefined state or entering `pdb`;
- document the triage layers and cover help, identity parsing,
  read-only diagnostics, dtype selection and `ldshm` flow.

Prompt-IO: ai/prompt-io/opencode/20260731T165215Z_0846cbd4_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-12 09:48:06 -05:00
Gud Boi 0d2c88ce3e Repair IB startup gaps in `ShmArray`
Keep reverse history and stored SHM layout aligned so sparse IB
frames do not expose permanent zero-time reservations.

Deats,
- omit inclusive query endpoints from SHM but retain full storage
- serialize and bound farm resets, blanks, and cancellations
- run the final null sweep after reverse retrieval completes
- require valid timestamp boundaries for synthetic gap rows
- cover the live QQQ 653-row failure and reset interleavings

Prompt-IO: ai/prompt-io/opencode/20260729T204046Z_ad6c560f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:35:47 -05:00
Gud Boi 892898cfd0 Guard `ldshm` from invalid SHM snapshots
Do not pass unpublished or corrupt timestamp rows into dedupe,
`NativeDB` replacement writes, or live SHM reload.

Deats,
- infer cadence from finite positive observed steps
- choose the smaller observed step when frequency ties
- skip invalid or degenerate snapshots as a whole
- cover the QQQ zero-slot failure with mutation-failing spies

Prompt-IO: ai/prompt-io/opencode/20260729T185406Z_d359b1cb_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:05:35 -05:00
Gud Boi 05e47d0e33 Repair first-slot OHLC sentinels in `NativeDB`
Filter known zero-initialized epoch slots before persisted history
hydrates SHM or merges with a fresh provider frame.

Deats,
- reindex the repaired read view for same-session chart publication
- treat sentinel-only files as empty history for fresh backfill
- reject new first-slot sentinels but preserve modern zero-price bars
- classify raw sentinel evidence separately in `store audit`

Prompt-IO: ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:04:16 -05:00
Gud Boi ad61d6689a Use IB UTC syntax for history `endDateTime`
Serialize aware reverse-fill boundaries as `YYYYMMDD-HH:MM:SS`
before handing them to `ib_async`.

This avoids Gateway error 10314 from its trailing-`UTC` format and
drops the unfinished EST comparison breakpoint from `Client.bars()`.

Prompt-IO: ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 18:04:05 -05:00
Gud Boi c1b3e17069 Add `xonsh` backfill qualification workflow
Drive known-bad, fresh, restart and append checks from a private
disposable NativeDB root.

Deats,
- bind child config and imports to the marked worktree root
- preserve checksum-matched reports, snapshots and case metadata
- compare timestamp coverage, seam values and persisted structure
- document manual chart runs, gap triage and the expansion matrix

Prompt-IO: ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:36:45 -05:00
Gud Boi 7cec91e287 Add raw OHLCV audit to `NativeDB`
Inspect Parquet bytes directly without opening storage or normalizing
malformed evidence.

Deats,
- report canonical schema, timestamp, index and finite-value state
- retain exact cadence aggregates with bounded endpoint details
- snapshot bytes before parsing and clean up partial copy failures
- reject unsafe FQMEs, links, collisions and non-regular sources

Prompt-IO: ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 50b84d68f2 Repair legacy timestamps in `NativeDB` merges
Keep incoming provider frames strict while dropping non-positive rows
from persisted history before merge validation.

This lets known-bad MNQ history repair itself on the next valid IB
frame without losing valid old bars or publishing noncanonical indexes.

Prompt-IO: ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi a26d69c269 Clean test-owned SHM leaks in `pytest`
Track successful `create=True` calls through Tractor's
current-process factory instead of scanning `/dev/shm` or trusting
the attachment cache.

- cooperate with normal actor unlink and restore token-cache state
- verify POSIX object identity before fallback cleanup
- fail after cleaning exact surviving segments
- reproduce the pre-`current_actor()` allocation leak

Prompt-IO: ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-11 11:24:45 -05:00
Gud Boi 8a4a1f67d8 Defer SHM null repair without valid time bounds
Require positive, increasing timestamp bounds before querying a SHM
gap or applying synthetic fallback. Defer unbounded edge runs
without entering the debugger or rewriting their rows, and continue
checking later segments.

Cover leading, trailing and entirely null frames with provider,
debugger and notification calls forbidden; assert the complete
frame stays unchanged.

(this patch was generated in some part by `Codex` using
`GPT-6` (`OpenAI`))
2026-11-11 11:18:11 -05:00
Gud Boi 1b76dd64fd Normalize provider OHLCV for `NativeDB`
IB history frames omit the derived `index` and append `count`.
Normalize provider data to durable names, order, and dtypes before
merge, and reject fractional epoch truncation.

Also,
- map canonical storage fields into provider-specific SHM on reload
- select Polars columns by name instead of position
- exercise first-start and restart through actor, SHM, and Parquet

Prompt-IO: ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:02:36 -05:00
Gud Boi 9851ff886a Drop redundant `NativeDB` writer locks
`datad`'s persistent feed task already owns each series writer.
The actor-local and file locks duplicated this SC contract and left
`.parquet.lock` sidecars that upstream indexed as data.

- preserve validated temp writes, `fsync()`, and atomic replacement
- keep exact-suffix indexing for legacy lock and temp sidecars
- prove full and incremental writes create no lock artifacts

Prompt-IO: ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:33 -05:00
Gud Boi 9cfe2591d9 Make `NativeDB` backfills atomic and non-destructive
- merge provider deltas without dropping unaffected history
- validate reopened temp parquet before atomic replacement
- serialize writers with actor-local and cancellable file locks
- publish cache only after disk replacement
- persist full startup/backfill frames before SHM publication
- bound sampler notifications and avoid MarketStore duplicates

Prompt-IO: ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 16:00:32 -05:00
Gud Boi fc532d6acb .tsp: report non-positive timestamp steps
- keep positive missing-history gaps distinct from ordering faults
- return duplicate and reversed timestamp rows as structured data
- warn before normalization removes the original ordering evidence

Prompt-IO: ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 14:32:23 -05:00
Gud Boi 61cd084ef2 .tsp: normalize gaps after sorted dedupe
- recompute datetime and delta cols after final timestamp ordering
- detect only positive sample-step gaps
- document inclusive null grouping with an updated visual map
- check all small null layouts against an independent test oracle
- honor selected null cols and reject discontinuous frame indexes
- share exact null grouping across NumPy and Polars

Prompt-IO: ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-10 14:32:20 -05:00
Gud Boi 13c036d16a Fix `datad` log naming; rm duplicate `brokername` 2026-11-10 12:26:56 -05:00
Gud Boi 79d7b48fd3 datad: console-log the backend's mod subtree
Pre the (datad|brokerd)-split this was done by `brokerd`'s
daemon fixture, so ALSO `get_console_log()` the provider
backend's mod subtree (eg. `piker.brokers.ib.*`) in
`_setup_persistent_datad()`; without it all backend records
emitted in the `datad` actor fall through to the stdlib's
bare (non-colorized) `logging.lastResort` handler.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-11-09 17:06:36 -05:00
Gud Boi db3d41a022 .ib.broker: eagerly pre-qualify pp/order mkts
Follow-up to the lazy order-req qualify: pre-qualify (and
cache) contracts for ALL already-open pps and orders during
`open_trade_dialog()` startup so live submissions NEVER pay
a first-request qualification delay; the warmup runs before
the order handler task starts so early reqs just buffer in
the ems IPC stream. Any brand-new mkt still lazily
qualifies on its first submission.

Deats,
- factor the `Client` lookup-table writes out of
  `.symbols.get_mkt_info()` into a new `cache_contract()`
  helper which now ALSO keys `_contracts` by `mkt.fqme`
  (read by the fill-time pp-update path in
  `emit_pp_update()`) alongside `mkt.bs_fqme` (read by
  `Client.submit_limit()`); resolves the old "post-split
  mktmap lookup" TODO.
- explicitly `cache_contract()` in
  `handle_order_requests()` since the lifo-cache may
  deliver a hit (body skipped) when another acct/client
  already qualified the fqme.
- filter `None` entries (ambiguous contracts) from
  `qualifyContractsAsync()` results in
  `Client.find_contracts()` before any attr access + raise
  a better "use a (more) venue-qualified fqme" error msg.
- relay ALL (non-cancel) errors from the aio method-relay
  task to the `trio`-side caller instead of crashing the
  whole proxy/dialog; critical post-`datad`-split where eg.
  qualification failures are expected to be caught
  per-request by order/warmup code.
- handle inline `('event', ...)` api-farm status msgs in
  `MethodProxy._run_method()` at info-level instead of the
  "UNKNOWN IB MSG" warning.
- only `log.setLevel()` in `open_trade_dialog()`;
  attaching a handler via `get_console_log()` double-prints
  every record since the daemon fixture already enables the
  console handler on the parent subsys logger.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-11-09 17:06:33 -05:00
Gud Boi 0142476966 .ib.broker: lazily qualify contracts on order req
Post (datad|brokerd)-split the trading actor's
`Client._contracts` cache is never warmed by in-proc
feed setup (that now happens in the `datad.ib` sibling)
so ALL live submissions failed with "no live feed?" at
`Client.submit_limit()`; `brokerd` must be able to
submit orders without any feed registered in its own
subactor.

Deats,
- thread the acct `proxies` table into
  `handle_order_requests()` and, on a `_contracts`
  cache-miss for the req's fqme, lazily run the same
  `get_mkt_info(fqme, proxy=...)` symbology ep the
  feed-side uses; it writes the `mkt.bs_fqme` key
  `submit_limit()` looks up (and warms `_cons2mkts`
  which the position-audit path also needs) on exactly
  the same aio `Client` instance.
- guard `submit_limit()` w/ a try/except ->
  `BrokerdError` relay so a single bad submission
  degrades to an EMS error msg instead of crashing the
  dialog (and causing the `TrioTaskExited` teardown
  storm seen in testing).
- fix the (non-f-string..) raise msg in
  `Client.submit_limit()` and doc the new lazy-qualify
  contract; the bug was foretold by the TODO in
  `.symbols.get_mkt_info()` B)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 5b3370ddbe Add `datad`-split design plan + provenance docs
Commit the AI-authored design doc driving this branch's
7-commit `brokerd` -> (`datad` + `brokerd`) split; all
prior commits' `Prompt-IO:` entries reference its
stages so this makes those refs resolvable in-repo for
PR review.

Also,
- add the doc's own prompt-io entry pair (scope: docs)
  incl. a 4-item implementation-deviation log vs. the
  plan as written.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 4d3f77f9df brokerd: slim RPC caps + `ib` client-id offset
Caps-sec tightening now that `brokerd` is trading-only: NO
`piker.data.*` (feed) mods are RPC-enabled in the (live,
credentialed) trading actor anymore.

Deats,
- drop `_data_mods` for a minimal `_brokerd_service_mods`
  (just `piker.brokers._daemon`); dedup-compose with the
  backend's set in `spawn_brokerd()`.
- `broker_init()` reads the backend's `_brokerd_mods`
  (fallback: `__enable_modules__` for flat backends).
- fail fast in `spawn_brokerd()` via `validate.get_eps()`
  when a backend offers NO live order-ctl eps (eg.
  `kucoin`, `deribit`) -> tells the caller to use
  paper-mode instead of booting a dead actor; analogous
  warning in `datad_init()` for datad-ep-less backends.
- offset `ib`'s default `client_id` per daemon-kind in
  `load_aio_clients()`: post-split BOTH `datad.ib` and
  `brokerd.ib` connect to the same gw/tws endpoint and the
  shared default (6116 + linear retry incrs) would collide
  and burn the full conn-timeout retry cycle; datad gets
  +16, ad-hoc (test/cli) actors +32.
- drop the import-cleanup leftovers (`exceptiongroup`,
  `_FeedsBus` type-only import) and the now-resolved
  "expose datad" TODO in `.cli`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 25e63211c6 .data: cut feed layer over to `datad` actors
The topology flip: all data-feed consumers now route to the
new `datad.<broker>` sibling daemon; `brokerd` becomes
trading-only and is ONLY ever booted lazily by `emsd`'s
`open_brokerd_dialog()` (see prior commit). Chart-only and
paper sessions run with zero (live, credentialed) `brokerd`
procs B)

Deats,
- `open_feed()` -> `maybe_spawn_datad()` (NB: imported
  relative-direct from `._daemon` to dodge a partial-init
  cycle via `piker.service`); flip the `open_feed_bus()`
  actor-name assert to `'datad'`; comment sweep.
- slim `_setup_persistent_brokerd()` to a trading-only
  fixture: console logging + pinned-open ctx; the feed-bus
  alloc moves to `_setup_persistent_datad()` and backend
  `open_trade_dialog()` ctxs own their own task trees.
  (the `piker ledger` ad-hoc actor enters this same slimmed
  fixture - exactly what it needs.)
- repoint data-flavoured spawn sites to `maybe_spawn_datad`:
  `.ui._app` symbol-search (+ rename
  `install_brokerd_search` -> `install_datad_search`),
  `.brokers.core.symbol_search()`, `.brokers.cli`
  `brokercheck`/`record`, legacy kivy `.ui.cli` +
  `option_chain`'s `wait_for_actor()`.
- invert `tests.test_services` expectations: feed/EMS-paper
  flows must spawn `datad.kraken` and `paperboi.kraken`
  with an explicit negative assert that NO `brokerd.kraken`
  service task exists.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi d27f953144 .clearing: lazily spawn `brokerd` from `emsd`
Drop the ONE coupling that forces feed + trading eps into
the same actor: `Router.open_trade_relays()` pulling its
trades-dialog portal from `feed.portals[brokermod]`.
Instead `open_brokerd_dialog()` now (maybe) spawns/finds
`brokerd.<broker>` itself via `maybe_spawn_brokerd()` and
ONLY when a live trades-ep will actually be opened; the
paper-mode short-circuit never touches it, so post
feed-cutover paper sessions will run with zero `brokerd`
procs.

Pre-cutover this is a pure refactor: the registry lookup
just finds the same feed-spawned daemon.

Deats,
- new `open_brokerd_dialog()` sig: portal acquisition moves
  inside via an `acquire_live_portal()` helper; keep an
  explicit `portal: Portal|None` override for the
  `piker ledger` cli which boots its own ad-hoc actor.
- `Router.maybe_open_brokerd_dialog()` drops its `portal`
  param; `open_trade_relays()` drops the `feed.portals`
  lookup entirely.
- `.accounting.cli`: pass `portal=` by keyword.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
2026-11-09 17:05:43 -05:00
Gud Boi 4402d391f0 Add `datad` daemon machinery to `.data`
First half of the `brokerd` split: a new per-provider
data-feed-only daemon-actor `datad.<broker>` to (soon) host
all `validate._eps['datad']` eps (live quotes, history
loading, symbology search) leaving `brokerd` for live order
ctl only. Purely additive; nothing routes through it yet.

Deats,
- new `piker.data._daemon` mod mirroring the
  `.brokers._daemon` conventions (and the `samplerd`
  sub-daemon precedent):
  - `_setup_persistent_datad()` lifetime fixture owning the
    actor-global `_FeedsBus` alloc.
  - `datad_init()` building `enable_modules` from the
    backend's `_datad_mods` (falling back to
    `__enable_modules__` for not-yet-split backends) and
    copying `_spawn_kwargs` (critical for `ib`'s
    `infect_asyncio`).
  - `spawn_datad()`/`maybe_spawn_datad()` wrapping
    `Services` + `maybe_spawn_daemon()`.
- add `piker.data._daemon` to `_root_modules` so `pikerd`
  can run `spawn_datad()` requests.
- re-export the spawn eps from `piker.service`.
- add `test_datad_spawn` verifying actor boot + service
  registration via `ensure_service('datad.kraken')`.

Note the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep work) lands symmetrically on both.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
2026-11-09 17:05:39 -05:00
Gud Boi 29c0fb72b9 Declare per-daemon-kind backend mod groups
Prep for the `brokerd` -> (`datad` + `brokerd`) actor split
by having each (split-style) backend declare which of its
submods host which daemon-kind's eps, exactly per the
`piker.data.validate._eps` groupings; `ib` already had
`_brokerd_mods`/`_datad_mods` so extend the convention to
`kraken`, `binance` and `deribit` (and add `'api'` to ib's
datad set since both kinds need the `Client` layer).

`__enable_modules__` stays as the (deduped) union so this
is a ZERO behavior change; flat backends (`kucoin` etc.)
just don't declare the split yet.

Also,
- add `validate.get_eps()` returning a backend's defined
  eps per daemon-kind for spawn-time introspection.
- import `NoBsWs`/`open_autorecon_ws` from
  `piker.data._web_bs` directly in `.kraken.broker` (they
  were only re-exported via `.kraken.feed`) so the trading
  mod doesn't depend on the feed mod for ws primitives.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
2026-11-09 17:02:44 -05:00
Gud Boi 3e3e0fdd46 Use public `tractor.Context.cancel_called` property 2026-11-06 15:49:43 -05:00
Gud Boi b349332e6b Distinguish websocket receive and relay timeouts
The `proxy_msgs()` deadline covers both `ws.get_message()` and
`snd.send()`. Repeated socket-ready reports with no first relay
could therefore mean either a silent source or a blocked local
consumer; the old generic reset log could not distinguish them.

Track the currently awaited operation and report its phase, URL,
deadline, accumulated timeout count and reset threshold on each
timeout. Snapshot `NoBsWs._timeout` once per iteration so the log
reflects the deadline that actually elapsed. Keep the existing
shared deadline, counter behavior and reconnect policy intact.

Add two real loopback regressions: a silent websocket server and
one payload blocked on an unconsumed zero-capacity Trio relay.
Require the matching timeout phase and URL before the reset budget
is exhausted. The 15 selected Binance/websocket cases pass.

The human observes this after invalid `.usdm.` startup followed
by corrected `.usdtm.` input. Metadata accepts the typo but later
history lookup expects canonical keys. Record that investigation
as context; this diagnostic patch does not fix FQME validation,
failed-startup cleanup or prove the cause of corrected-feed stalls.

Prompt-IO: ai/prompt-io/opencode/20261028T180856Z_032580bf_timeout_phase.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 5183d4094f .data: expose websocket reconnect progress at INFO
An INFO pikerd still hid websocket connection-success messages.
The shared ._util.log emitter is piker.data, but open_feed_bus()
configured piker.data.feed. Errors reached bare stderr while
INFO diagnostics remained disabled on the actual emitter.

Configure the shared logger by its exact log.name and prefer
the actor's loglevel over a quieter per-client request for these
actor-wide transport diagnostics. Preserve the selected client
level passed to provider feed startup.

Also,
- retain the existing handshake-success report and add URL-tagged
  readiness after fixture setup and first-message relay once per
  connection, including replacement sockets.
- include the URL in receive-timeout reset errors so Binance's
  independent bookTicker and aggTrade routes are distinguishable.
- prove real console INFO visibility through production feed
  startup, stopping before provider lookup, then restore logging
  state; the original code fails with the emitter at WARNING.
- extend the loopback recovery test to verify both readiness
  reports and the first relay from the replacement connection.

First-message relay is raw websocket traffic, not proof that
parsed market data or chart updates have recovered. The 18
logging/Binance/websocket cases pass; live reporting qualification
remains separate.

This is a local follow-up, not a downstream cherry-pick. The
54da2973 websocket cleanup and 5adf77a9 namespace fix do not
configure this shared emitter. Relevant local histories and
dirty IB-history/TIPC deltas contain no equivalent wiring fix.

Prompt-IO: ai/prompt-io/opencode/20261028T165638Z_b95e7c79_reconnect_logging.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 251987fc4b .data: reconnect after websocket reader close errors
A Binance Ping racing local socket closure makes trio-websocket's
reader send a Pong while wsproto is LOCAL_CLOSING. That reader
nursery error escapes proxy_msgs(), then tears down brokerd and
pikerd instead of restoring quote delivery.

Classify NoBsWs.recon_errors around the entire websocket context,
with collapse_eg() outside it so a singleton reader exit group
reaches the reconnect handler. Invalidate the stale connected
event and socket reference before retrying. Keep retry sleep
unshielded so parent cancellation stops recovery promptly.

Deats,
- Add real loopback websocket coverage: TCP, handshake, Ping,
  reader dispatch and wsproto's actual Pong validation.
- Control only the first client Ping's protocol-state ordering,
  withholding its Close frame to prevent a peer response racing
  the deterministic LOCAL_CLOSING reproducer.
- Require fresh data on a replacement connection, plus exactly
  one connection when cancelling at the reader or retry boundary.
- Use RuntimeError for the unexpected-failure control:
  LocalProtocolError is now a recoverable connection failure.
- Preserve the original unexpected exception with bare raise,
  and assert its identity in the test. Log recoverable errors
  with the websocket URL and the underlying traceback.

This is a local follow-up, not another downstream cherry-pick.
The earlier wrapper suppressed escaped communication failures;
97eb8347 deliberately re-raised them. Retain that outer error
visibility rather than restoring suppression without recovery.
The isolated baseline reproduces the live Pong failure; all 13
selected Binance/websocket cases pass with this fix. Human live
qualification remains separate from these loopback regressions.

Follow-up-to: 97eb834799
Original-equivalent: d50ab6124c
Related-history: 3d83b61f3f
Prompt-IO: ai/prompt-io/opencode/20261028T160846Z_7242dcb9_websocket_close.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 63c763800a Harden Binance USD-M feed startup
- reject non-single-symbol feeds with `FeedInitializationError`
- route book and trade data across the correct USD-M endpoints
- accept the first matching L1 or trade quote as feed readiness
- preserve transport and startup-timeout failure context
- cover provider msg schemas and both readiness classes

Prompt-IO: ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 7a06a620e2 Fix reconnect startup in `_web_bs`
- restart handled handshake failures before touching cancel state
- propagate `trio.Cancelled` without shielded retry throttling
- bound initial connection waits and expose transport failures
- cover real reconnect task paths in `test_websocket.py`

Prompt-IO: ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi 3053324f04 Fix Binance USD-M websocket routing
Binance retired its legacy futures websocket and split quote
traffic across `/public` and `/market`. Route `bookTicker` and
`aggTrade` through dedicated URLs, merge both sockets, and accept
the new `st` stream-type field.

Bound connection and first-quote startup so failed or silent
streams raise `DataUnavailable` instead of hanging `datad` and
chart consumers.

Also,
- derive feed mode from resolved `FutesPair` metadata
- keep direct URL subscriptions stable across reconnects
- cover USD-M routes, payloads, and startup timeouts

Prompt-IO: ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-11-06 15:40:58 -05:00
Gud Boi cb5e81d443 Pin qualified `tractor` and require `trio>=0.34.0`
Advance the `piker_pin` lock from `b64c89f0` to the published
`8fe574c1`, retaining the evaluated Trio 0.34.0 release. Require
Trio >=0.34.0 for the updated cancellation APIs. Restore the Git
source used before local editable evaluation; no environment sync
is included.

Record the human's overlapping charts, Gateway data reset and
close/reopen qualification. Keep the full Gateway restart string
error and Tractor teardown callback assertion explicit as separate
recovery limits. Offline lock validation resolves 126 packages;
this is not a fresh-install qualification claim.

Prompt-IO: ai/prompt-io/20261106T201746Z_495c0915_612dfc_prompt_io.md

(this patch was generated in some part by `codex` (`openai`))
2026-11-06 15:21:33 -05:00
Gud Boi 51795a4e07 Contain provider failures during SHM gap repair
Partially backport only the `get_hist()` exception catch from
0846cbd4a4, authored by goodboy
<goodboy_foss@protonmail.com>. Keep the existing gap fallback.

A timed-out IB query raises `NoData` from null-segment repair.
Letting it escape cancels sibling history and live quote tasks
while `pikerd` can remain alive. Catch `NoData` and
`DataUnavailable` locally and stop querying this repair pass.

Add a regression using real null detection and a waiting sibling
in the same nursery. Both provider failures must return without
canceling the sibling. The two cases failed before this catch;
the focused history files passed 17 tests with it.

Partial-backport-of: 0846cbd4a4
Prompt-IO: ai/prompt-io/20261106T185628Z_495c0915_215e9b_prompt_io.md

(this patch was generated in some part by `codex` (`openai`))
2026-11-06 15:20:58 -05:00
Gud Boi 495c091546 Guard `brokerd_msg` set in order-mode dialog loop
Use `msg.setdefault('brokerd_msg', msg)` instead of blind assignment and
log a warning when the field was already populated.

Specifically, this avoids a self-reference field recursion which causes
crashes when using `tractor.devx.pformat.ppfmt()`..

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

(cherry picked from commit 8cefc1bdf8)
2026-11-06 13:46:36 -05:00
Gud Boi 2a826fba17 `.tsp._history`: drop orphaned backfill events
- let structured nursery joins own child-task completion
- stop empty provider frames before timestamp endpoint indexing
- cover provider exhaustion and empty frames with bounded tests

Prompt-IO: ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))

(cherry picked from commit 931f22d235)
2026-11-06 13:41:07 -05:00
Gud Boi 6575db6463 Adjust to `trio>=0.34.0` invalid `Event` truthy checks
(cherry picked from commit 63ae23aa4ff229eb89cd970da132e997570c0afb)
2026-11-06 13:10:50 -05:00
Gud Boi 3ec447a45a Document bounded IB recovery and downstream replay
Record exact keep/backout/import provenance for the #94 candidate,
including the quote-cache lineage correction and the distinction
between file-equivalent adapter deltas and partial source commits.

Deats,
- Preserve the original raw prompt record and add the human's
  later scope decisions, corrections and live-test contributions
  to its structured partner.
- Record 38 prior unique offline passes and the known-red bridge
  callback-cleanliness case, without claiming clean teardown.
- Record live QQQ backfill, reattachment and overlapping chart
  close/reopen results. Startup needed debugger continuation;
  the local breakpoint cleanup was applied after that run.
- Retain the cached-client daemon shutdown failure as a human-
  accepted follow-up outside #94, with its exact propagation path.
- Require disposable downstream rebase rehearsals before landing,
  preserving omitted SHM/storage fixes and local regressions.

The commit package records exact per-boundary project checks.
Its structural replay/preflight does not substitute for those
checks, the prior live observations or rebase qualification.

Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:36:29 -04:00
Gud Boi 11aa2eeb1e .tsp: drop the single-null debugger stop
Avoid interactive startup pauses when `get_null_segs()` sees one
zero-time row. Remove the explicit breakpoint and its now-unused
`RuntimeError` handler, preserving the legacy `None` result.

LOCAL UX CLEANUP -- NOT the downstream null-grouping fix.
The related source below removes this branch as part of a much
larger grouping/deduplication rewrite. None of that algorithm,
its index validation or its oracle coverage is backported here.

Downstream replay,
Prefer the complete downstream implementation. Do not preserve
this legacy single-null early return against the later rewrite.
The small inline comment records that temporary discrepancy.

Related-downstream: 976db6c33c
Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:36:11 -04:00
Gud Boi 9dde7a9867 .clearing: backport owned dark quote subscriptions
Import the source-equivalent EMS code and its original wrapper
regression. `clear_dark_triggers()` owns a non-strict child from
`quote_stream.subscribe()` for the lifetime of its clearing loop,
instead of changing a shared receiver's private lag policy.

LOCAL ADDITION,
Add a second regression using the production clearing loop and a
real broadcaster. Deliver a tick to EMS and a sibling, cancel EMS,
and prove the sibling remains usable with strict lag policy.
No order trigger or live broker is involved in this test.

Downstream replay,
The EMS code has the source patch's stable patch ID. The test file
also contains the local real-broadcaster case, so the complete
commit is not patch-identical. Preserve that extra coverage when
reconciling an already-applied EMS delta.

Backport-of: 12b0d66e85
Source-author: goodboy <goodboy_foss@protonmail.com>
Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:36:02 -04:00
Gud Boi 02bb193f87 .ib.feed: backport no-data request boundaries
PARTIAL BACKPORT -- IB adapter and six boundary cases only.
The source also fixes history-layer endpoint/filtering behavior;
those changes and its SHM/NativeDB regressions are NOT imported.

Import the exact `piker/brokers/ib/feed.py` delta. Complete HMDS
no-data replies at the requested date instead of silently walking
back a day. Wake the waiter and let `open_history_client()` turn
the internal empty sentinel into `NoData` or `DataUnavailable`.

Deats,
- Retain the original parametrized request-boundary test body and
  its six cases, adding only its required module import.
- Add LOCAL real-bridge history recovery coverage after the reset
  backport: production proxy/relay/reset tasks, replayed external
  IB/VNC input, and a real farm event before retrying 1s/60s bars.
- Leave `request_end_dt` versus `oldest_shm_dt`, storage cutoff
  filtering, `diff_history()` removal and full-bar retention
  coverage to the downstream history-layer commit.

Downstream replay,
Retain those omitted changes even if the adapter hunks vanish.
Preserve the local real-bridge regression when reconciling tests.
The resulting adapter file matches the source byte-for-byte;
that equivalence does not apply to the whole source commit.

Partial-backport-of: 1da9b760a3
Source-author: goodboy <goodboy_foss@protonmail.com>
Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 11:35:31 -04:00
Gud Boi 6f626a9c45 .ib.feed: backport bounded reset coordination
PARTIAL BACKPORT -- IB adapter and seven adapter regressions only.
The source commit also repairs startup gaps in `ShmArray`; this
patch does NOT deliver that complete history/storage fix.

Import the exact `piker/brokers/ib/feed.py` delta: serialize farm
resets, publish completion and result separately, keep failure
counters per request, bound IB cancellation retries, and complete
empty replies without leaving the parent waiting forever.

Deats,
- Extract seven existing cases into `tests/test_ib_history.py`.
  Their function bodies match the source; the module header and
  imports are reduced for this older landing tree.
- Omit `piker/tsp/_history.py`: inclusive endpoint handling, SHM
  layout, final null-sweep ordering and synthetic gap boundaries.
- Omit the source's history-layer regressions, harness update and
  historical prompt files. The source SHA retains that provenance.

Downstream replay,
Only the adapter hunks and these seven cases are already present.
Retain the omitted history/SHM changes and their regressions; do
NOT skip the whole source commit when its adapter delta is empty.

Partial-backport-of: 0846cbd4a4
Source-author: goodboy <goodboy_foss@protonmail.com>
Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 10:50:20 -04:00
Gud Boi 72d0bdad5e Back out cached IB quote ownership
Reverse the quote-owner code and mocked-cache test from the
landing replay below. Keep its historical prompt records and
restore the direct-owner `_quote_streams` implementation for #94.

With the pinned `tractor`, the cached owner turns quote restart
teardown into a normal bridge exit, then propagates its injected
`TrioTaskExited` into the consumer and kills the broker actor.
The evaluated downstream branch did not contain this cache patch.

LOCAL DIVERGENCE,
- This is a targeted backout, not a replacement broadcast design.
- Add a real infected-actor quote restart regression, including a
  second RPC on the same actor and subscription/task cleanup.
- Keep the strict callback-cleanliness case visibly red: actor
  survival does not establish clean asyncio completion callbacks.

Downstream replay,
Reintroducing the original cache patch requires qualifying its
owner teardown with the matching `tractor` fix. Do not interpret
this backout as rejecting lexical broadcast subscriptions or as
proving arbitrary concurrent direct-owner callers are safe.

Reverts-code-and-test: 635314c757
Original-equivalent: 4af885d90f
Source-author: goodboy <goodboy_foss@protonmail.com>
Prompt-IO: ai/prompt-io/opencode/20261022T212248Z_dc618229_ib_recovery_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6-astra` (`openai`))
2026-10-23 10:45:03 -04:00
Gud Boi dc618229bd .data: isolate cached feed stream consumers
Keep provider root streams private to the cached `Feed` and give every
`maybe_open_feed()` caller, including the first owner, a local
subscription view with copied `Flume` descriptors.

Also,
- key cached resources by the normalized full symbol set and stream
  policy instead of only the first FQME.
- subscribe each `Feed.open_multi_stream()` provider relay and wait for
  all relays to start before yielding the merged channel.
- give each provider relay an independent send-channel clone so one
  provider EOC cannot terminate its siblings.
- document `Feed.pause()` and `Feed.resume()` as shared remote-provider
  controls rather than caller-local cursor operations.
- cover miss/hit ownership, multi-market provider mapping, relay
  fan-out, lexical teardown and cache-key separation.

Prompt-IO: ai/prompt-io/opencode/20260813T202402Z_12b0d66e_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi 635314c757 .ib.feed: cache quote channel contexts
Replace the manual `_quote_streams` raw-channel registry with a
symbol-keyed `maybe_open_context()` resource whose lifetime extends to
the final quote consumer.

Give every `open_aio_quote_stream()` caller a lexical child from the
cached `LinkedTaskChannel` broadcaster so concurrent users receive the
same quotes and close independently.

Prompt-IO: ai/prompt-io/opencode/20260813T230744Z_ddac3553_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi d1571ddfc7 Always re-raise in `maybe_spawn_daemon()` handler
Move bare `raise` outside the `if lock.owner` guard so the error
propagates regardless of whether the stale-lock branch runs. Also add
a blank-line separator in the crash log msg.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-21 20:50:38 -04:00
Gud Boi 7bef50b0a5 Fix IB `MethodProxy` response routing
`open_client_proxy()` broadcast its one-slot linked channel to an
always-running status relay and an idle method receiver. Status
traffic advanced only the relay, so the first lazy contract lookup
for an uncached order raised `Lagged` before receiving its
response.

Give `relay_client_proxy_messages()` sole receive ownership. Route
tagged responses from concurrent caller tasks into their per-`mid`
slots while each proxy context retains its own channel and relay.

Deats,
- log stale responses with their original method name;
- warn when unresolved-request metadata exceeds its bounded limit;
- drop late responses after caller cancellation;
- preserve the reconnect design note and message-case rationale;
- accept clean channel EOF during proxy teardown;
- cover idle status, concurrency, cancellation and EOF paths.

Prompt-IO: ai/prompt-io/opencode/20260811T205826Z_06d5ea5d_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-21 20:50:38 -04:00
Gud Boi e8e0730a50 .ib.api: req-id correlate `MethodProxy` calls
The (single, shared) relay chan has NO ordering guarantee
when a caller task is cancelled (eg. by a search-req
timeout) after sending its request but before consuming the
response: the "orphaned" response gets mis-delivered to the
next caller, off-by-one skewing every result thereafter!

Deats,
- tag each request with a `mid` from an `itertools.count()`
  on the proxy and echo it back in both the result and
  exception resps from `open_aio_client_method_relay()`.
- drop (w/ a warning) any stale resp whose `mid` doesn't
  match the current caller's and keep waiting for ours.
- rewrite the resp-wait loop as a `match:` (resolving the
  old "py3.10 syntax" TODO) incl. the prior inline
  `('error', ...)`/`('event', ...)` relay cases.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-10-21 20:50:38 -04:00
Gud Boi 4133f07340 Select `piker_pin` and keep `polars` below 2.0
Replace the address-unpacking branch selector with `piker_pin` and
lock the published branch at b64c89f0. Verify the installed Git
revision without a local-source `PYTHONPATH` override.

Bound Polars below 2.0 and resolve polars/runtime 1.44.2, the latest
non-yanked stable 1.x release, instead of taking a major upgrade as
part of the Tractor landing. This resolves the universal lock where
retaining the older 1.36 versions was rejected for a Windows target.

The scoped install changed only Tractor; 14 offline logger/startup
cases passed with the existing Polars 1.36 installation. A full sync
and Polars 1.44.2 data-path qualification remain outstanding.

Prompt-IO: ai/prompt-io/opencode/20261021T224421Z_507c75c6_piker_pin_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-21 19:16:14 -04:00
Gud Boi 507c75c669 Use `tractor`'s reviewed address-unpacking branch
Select `wkt/addr_unpacking` in `pyproject.toml` and lock `tractor`
at `a966928dead4cbf369ad244834ce9dbe988ac8ca` in `uv.lock`.
Keep the existing PR94 application/test tree as the landing base
for the original downstream consumer fixes.

Also,
- resolve `setproctitle` 1.3.8 while retaining `polars` 1.36.0,
- record the dependency selection and original-history previews
  with their Prompt-IO provenance.

Prompt-IO: ai/prompt-io/20261010T232852Z_1fa45dfa_prompt_io.md

(this patch was generated in some part by `opencode`
using `gpt-6.1-sol` (`openai`))
2026-10-11 00:43:06 -04:00
Gud Boi 1fa45dfa98 Fix endpoint config and daemon startup from PR94 review
Keep the shipped `conf.toml` defaults TCP-only for this `tractor`
pin, with an explicit single-transport note and commented UDS
alternatives. Preserve repeated Click `--maddrs` options and treat
a direct string as one endpoint.

Deats,
- Validate daemon bind requests against actual listeners, not
  registrar discovery endpoints; normalize tagged addresses and
  allow OS-assigned TCP ports. Explain registrar ownership and
  automatic client binding at the validation site.
- Resolve explicit or inherited registrar addresses independently
  of other daemon kwargs, consuming overrides before forwarding
  them and keeping discovery and startup on the same registrar.
- Default chart binding to `None` for absent or empty network
  config, and remove the unconditional Binance debugger pause.
- Defer the prematurely cherry-picked `test_datad_spawn` until
  its service API lands. Add offline endpoint/startup regressions
  and a real distinct-registrar/daemon lifecycle check.

Include the human-directed config and registrar/bind explanations
and the direct prompt provenance pair from the review fix pass.

Prompt-IO: ai/prompt-io/20261007T165453Z_bde2457b_prompt_io.md

(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
2026-10-10 14:20:52 -04:00
Gud Boi bde2457bdd Bump `tractor` and `msgspec` for Python 3.13
- require Python 3.13 across package metadata and the lock
- advance `tractor` from `e77198bb` to `piker_pin@0d95a9e6`
- raise `msgspec` from 0.19.0 to 0.21.1
- lock current multi-address transport deps
- retain `xonsh` 0.24.2 from the flake-update stack

Prompt-IO: ai/prompt-io/opencode/20260901T230043Z_e91f41a6_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 1ba9bdbac0 Preserve exact `piker` logger namespaces
Use `str.removeprefix()` instead of character-set stripping so
subloggers such as `piker.paper` retain their requested namespace.

Also,
- drop stale parent-logger assertions from daemon startup paths
- keep logger setup calls without assigning assertion-only locals
- cover both logger factories and prefix-sensitive module names

Prompt-IO: ai/prompt-io/opencode/20260821T042058Z_b7aca4c0_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi d86f29d416 Update feed test fqmes to use `.<venue>` format
Switch all `test_feeds.py` parametrized fqmes from
legacy `<pair>.<broker>` to the current
`<pair>.<venue>.<broker>` schema (e.g.
`btcusdt.spot.binance`).

Deats,
- update binance fqmes: `btcusdt.binance` ->
  `btcusdt.spot.binance`, same for `ethusdt`.
- update kraken fqmes: `ethusdt.kraken` ->
  `ethusdt.spot.kraken`, `xbtusd.kraken` ->
  `xbtusd.spot.kraken`.
- update cross-broker set similarly.
- comment out old fqmes with `!TODO` to later
  validate raising on bad/legacy formats.

Also,
- reformat `if ci_env and not run_in_ci` to
  multiline style.
- reformat `pytest.skip()` msg to multiline.
- add `?TODO` for symbology helper fn.
- drop stray `await tractor.pause()` in
  `conftest.py`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 69a6edd6f8 Normalize registrar addrs with `wrap_address()`
Canonicalize legacy socket pairs and tagged tuples before storing
or comparing `Registry.addrs`, keeping existing config readable
when tractor emits protocol-tagged forms.

Also,
- broaden registry/runtime addr annotations for both wire shapes
- compare service fixture addrs through tractor's address codec
- use tractor's shared `--ll` pytest option instead of
  registering it twice
- cover mixed legacy/tagged registry state and `pikerd` startup

Prompt-IO: ai/prompt-io/opencode/20260820T163519Z_5f631e19_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi bbe9deaa62 Port service+tests to latest `tractor` APIs
Continue the `repair_tests`-branch mission (already merged
in this stack's ancestry, see f4c4f1e2 which ported
`conftest.py`) by fixing the remaining drift breakage vs.
`tractor` git `main`; without these NOTHING boots since the
`tractor.Address` port in 604e5fcf.

Deats,
- normalize reg addrs via `wrap_address()` in
  `open_pikerd()` before `.unwrap()`-ing; entries may be
  raw `tuple`s when passed in from (test) client code.
- port `check_for_service()` to `query_actor(regaddr=)`
  (was `arbiter_sockaddr=`) incl. its 2-tuple yield and
  the now-required `open_registry(addrs=)` arg.
- `wait_for_actor(registry_addr=)` + `.chan.raddr.unwrap()`
  raw-tuple compares in `test_runtime_boot` and
  `ensure_service()`.
- update `run_test_w_cancel_method()` for modern `tractor`
  cancel semantics: self-requested sub-service cancels are
  absorbed (no `ContextCancelled` raised to the opener) and
  single-exc groups collapse to a bare KBI.
- `RemoteActorError.boxed_type` (was `.type`) and
  `Position.cumsize` (was `.size`) renames in tests.
- bump the paper-EMS startup budget 9 -> 19s; it includes
  a live (kraken) symbology fetch so needs net headroom.
- woops, add the missing comma in `.deribit.api`'s
  `tractor.trionics` import tuple..

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
2026-10-07 03:10:41 -04:00
Gud Boi 70e8d60326 Fix `pytest` config-dir isolation in subactors
The old (commented-out) `get_app_dir()` override gated on
`'pytest' in sys.modules` which can NEVER work in spawned
subactors (fresh procs, no pytest import); as a result test
`paperboi`/daemon actors were writing into the user's REAL
`~/.config/piker/accounting/` files.. friggin yikes.

Deats,
- add `config._maybe_use_test_dir()` which lazily (at
  conf-path access time, NOT import time) reads the
  `piker_test_dir` entry from
  `tractor.runtime._state._runtime_vars['piker_vars']` as
  pre-loaded by `open_piker_runtime()` from the
  `tests.conftest._open_test_pikerd()` overrides.
- hook it in `get_conf_dir()` and route `get_conf_path()`
  + `load()`'s mkdir through `get_conf_dir()`.
- route `.accounting._ledger` / `._pos` dir derivation
  through `config.get_conf_dir()` (was reading the
  `_config_dir` global directly, bypassing the override);
  also `mkdir(parents=True, exist_ok=True)` for nested
  tmp-dir creation.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Prompt-IO: ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
2026-10-07 03:10:41 -04:00
Gud Boi 2c007d3ada Add `tpt_bind_addrs` and separate registry eps
Thread a `tpt_bind_addrs` param through
`open_piker_runtime()` and `open_pikerd()` so
pikerd's bind addrs can differ from the registry
endpoint (support a dedicated `regd` service).
Simplify `load_trans_eps()` to delegate entirely
to `tractor.discovery.parse_endpoints()`.

Deats,
- `conf.toml`: fix maddr prefixes to proper `/ip4/` and `/unix/`, add
  `chart` endpoints section, add commented `regd` example.
- `cli/__init__.py`: replace `parse_maddr` with `parse_endpoints`,
  rename `--maddr` -> `--maddrs`, parse `regd` key from eps falling back
  to `pikerd` addrs.
- `_actor_runtime.py`: thread `tpt_bind_addrs` through runtime open fns
  to `open_root_actor`.
- `ui/cli.py`: activate `network` config parsing for chart CLI, extract
  `chart` eps for bind and `regd`/`pikerd` for registry, pass
  `tpt_bind_addrs` through runtime config.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 1fde12d3fd Use `tractor.Address` for endpoint resolution
Replace manual `layers['ipv4']['addr']` / `layers['tcp']['port']` tuple
extraction with direct `tractor.Address` objects returned from
`load_trans_eps()`. In `open_pikerd()` call `addr.unwrap()` for the
raw-tuple comparison against `root_actor.accept_addrs`.

Deats,
- `conf.toml`: update maddr prefix `/ipv4/` ->
  `/ipv/`, add commented UDS socket path example.
- `cli/__init__.py`: wrap endpoint loading in
  `maybe_open_crash_handler`, append `addr`
  objects directly to `regaddrs`.
- `ui/cli.py`: restructure `chart()` body into
  `maybe_open_crash_handler` scope, switch to
  `registry_addrs` from config, comment out the
  `network`-based `load_trans_eps` path (WIP
  `multiaddr` transition).
- `_actor_runtime.py`: use `addr.unwrap()` for
  accept-addr membership check.
- `uv.lock`: add `multiaddr >= 0.2.0` and its
  transitive deps.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Tyler Goodlet c347ab08ea Drop `.cancel_actor()` from `maybe_spawn_daemon()`
Since `tractor`'s new and improved inter-actor cancellation semantics
are much more pedantic, AND bc we use the `ServiceMngr` for spawning
service actors on-demand, the caller of `maybe_spawn_daemon()` should
NEVER conduct a so called "out of band" `Actor`-runtime cancel request
since this is precisely the job of our `ServiceMngr` XD

Add a super in depth note explaining the underlying issue and adding
a todo list of how we should prolly augment `tractor` to make such cases
easier to grok and fix in the future!
2026-10-07 03:10:41 -04:00
Gud Boi c5155ce814 Swap `open_channel_from()` yield-pair order
Port deribit and IB `asyncio` bridge callables to the new
`to_asyncio.open_channel_from()` signature where the `LinkedTaskChannel`
is the first param and `started_nowait()` replaces the old
`to_trio.send_nowait()` sync handshake.

Deats,
- deribit `api.py`: update `aio_price_feed_relay()` and
  `aio_order_feed_relay()` signatures to take `chan: LinkedTaskChannel`
  as first arg; drop `from_trio`/`to_trio` params; replace
  `to_trio.send_nowait()` with `chan.send_nowait()` and
  `chan.started_nowait()`.
- drop `functools.partial()` wrapping in both `open_price_feed()` and
  `open_order_feed()`; pass `fh=`/`instrument=` as kwargs directly.
- IB `broker.py`: same `chan` + `started_nowait()` port for
  `recv_trade_updates()`.

Other styling,
- rewrap `recv_trade_updates()` docstring to 67 chars.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi d7cefa71f3 Update `tractor` private-API refs across codebase
Port internal `tractor._<mod>` references to their
new public or reorganized paths after `tractor`
refactored its subpkg layout.

Deats,
- `tractor._portal.Portal` -> `tractor.Portal`.
- `tractor._supervise.ActorNursery` -> `tractor.ActorNursery`.
- `tractor._multiaddr` -> `tractor.discovery._multiaddr`.
- `tractor._addr` -> `tractor.discovery._addr`.
- `tractor._state._runtime_vars` -> `tractor.runtime._state._runtime_vars`.
- `tractor._state.is_debug_mode()` -> `tractor.runtime._state.is_debug_mode()`.

Files touched: `brokers/data.py`, `cli/__init__.py`, `data/feed.py`,
`service/_actor_runtime.py`, `service/_mngr.py`, `storage/cli.py`,
`tsp/_annotate.py`, `ui/kivy/monitor.py`, `ui/kivy/option_chain.py`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 20bfbc2380 Use pre-set reg addrs in `maybe_spawn_daemon()`
Pull actor-runtime `registry_addrs` from (the new)
`tractor.get_runtime_vars()` (over the previous hardcoding of
`('127.0.0.1', 6116)`..)) so that underlying `find_service()` and
`maybe_open_pikerd()` calls use the local actor's assigned registrar
endpoints.

Note, this is particularly necessary to get the `pytest` harness workin
again alongside any local running `pikerd` instance(s).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi d7f956a566 Register all custom excs with `tractor` IPC
Call `reg_err_types()` for every piker-defined
exception so they can be marshalled and re-raised
across actor boundaries.

Deats,
- `brokers/_util.py`: auto-register `BrokerError` +
  all `__subclasses__()` (6 types).
- `config.py`: `ConfigurationError` +
  `__subclasses__()` (`NoSignature`).
- `data/validate.py`: `FeedInitializationError`.
- `service/_ahab.py`: `DockerNotStarted`,
  `ApplicationLogError`.
- `service/marketstore.py`: `MarketStoreError`.
- `storage/__init__.py`: `TimeseriesNotFound`,
  `StorageConnectionError`.
- `brokers/kraken/api.py`: `InvalidKey`.
- `brokers/kraken/broker.py`: `TooFastEdit`.
- `brokers/questrade.py`: `QuestradeError`.

Also,
- uncomment `execution_venue` field on kraken `Pair`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi d26286f6cc Factor `Pair` schema-mismatch handling to `_util`
Add `get_or_raise_on_pair_schema_mismatch()` helper
and `SchemaMismatch` error type in `brokers._util`
to standardize the "provider changed their API" error
reporting across backends.

Deats,
- add `SchemaMismatch(BrokerError)` exc type.
- `get_or_raise_on_pair_schema_mismatch()`: catch
  `TypeError` on `Pair` ctor, build `ppfmt()`-ed
  report with provider name, fall back to
  `pair_type._api_url` if no explicit URL passed,
  then raise `SchemaMismatch`.
- binance `api.py`: replace inline `try/except` +
  `e.add_note()` with the new helper.
- kraken `api.py`: replace bare `Pair(...)` ctor
  with the new helper inside crash handler.

Also,
- add `_api_url: ClassVar[str]` to binance
  `FutesPair` and kraken `Pair` structs.
- binance `feed.py`: warn on missing `.<provider>`
  in fqme; raise `SymbolNotFound` on empty venue.
- reformat `start_dt`/`end_dt` unions to
  `datetime|None` style in binance `Client`.
- wrap binance `_pairs` lookup in
  `maybe_open_crash_handler()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 22f166a5c6 Update `conftest.py` for `tractor` runtime API changes
Port test fixtures to match `tractor`'s updated
registry and channel APIs.

Deats,
- use `registry_addrs=[reg_addr]` (list) instead of
  `registry_addr=reg_addr` in `maybe_open_pikerd()`.
- `wait_for_actor()` now takes `registry_addr=`
  kwarg instead of `arbiter_sockaddr=`.
- access `portal.chan` (not `.channel`) and unwrap
  remote addr via `raddr.unwrap()`.
- yield `raddr._host`/`raddr._port` instead of
  tuple-indexing.
- drop random port generation; accept `reg_addr`
  fixture from `tractor`'s builtin pytest plugin.

Also,
- add `reg_addr: tuple` param to `open_test_pikerd()`
  fixture (sourced from `tractor._testing.pytest`).
- type-narrow `reg_addr` to `tuple[str, int|str]`.
- drop unused `import random`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
2026-10-07 03:10:41 -04:00
Gud Boi 6c8709d309 Bump `xonsh` to 0.24.2
Use the current xonsh release in the Nix-managed REPL environment.

Deats,
- require xonsh 0.24.2 for Python 3.13 and Nix fixes;
- refresh only xonsh's source and wheel lock metadata;
- retain dotrc compatibility for shells outside this project;
- record Prompt-IO provenance for the dependency update.

Prompt-IO: ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 23632d2153 Bump `flake.lock` after dev-shell fixes
Advance `nixpkgs` to `241313f4`; the refreshed closure passes Qt,
`pyqtgraph` and wksmod-path import checks.

(this commit msg was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi d85e57334c Rebuild stale `uv` envs after flake updates
`uv sync` accepts an existing same-minor Python even when its Nix
store path belongs to an older closure. That can leave a glibc 2.40
interpreter loading Qt built against glibc 2.42.

- derive the Python package once from `cpython`
- clear `py313` when its interpreter target differs from the flake
- ignore inherited `VIRTUAL_ENV` before syncing

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 6c8d17da75 Fix `flake.nix` dev-shell install
The refreshed Nix closure no longer supplies `libz.so.1` to the
`uv` NumPy wheel, and leaves the synced `py313` env off `PATH`.

- add Nix `zlib` to `LD_LIBRARY_PATH` for native Python deps
- activate `py313` after `uv sync` so `pyqtgraph`, `piker` and
  `xonsh` resolve from the locked env

(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
2026-10-07 03:10:41 -04:00
Gud Boi 1976f6a5de Bump `flake.lock` to avoid nix warnings..
However this breaks `pyqtgraph` due to missing `PyQtX` libs??
2026-10-07 03:10:41 -04:00
239 changed files with 25926 additions and 1981 deletions

View File

@ -47,7 +47,7 @@ that's why it's like 1000x faster ya know?"
- Wrap all code symbols: `function()`, - Wrap all code symbols: `function()`,
`ClassName`, `field_name` `ClassName`, `field_name`
- File paths: `piker/ui/_remote_ctl.py` - File paths: `piker/ui/_remote_ctl.py`
- Commands: `git status`, `piker store ldshm` - Commands: `git status`, `piker store shm`
**Explain like you're pair programming:** **Explain like you're pair programming:**
``` ```

View File

@ -0,0 +1,143 @@
# Split `brokerd.<broker>` into trading-only `brokerd` + new `datad.<broker>`
## Context
Today a single `brokerd.<broker>` actor hosts BOTH concerns:
- **data feed service tasks**: the `_FeedsBus` + `open_feed_bus()` ep (`piker/data/feed.py:464`), per-symbol `allocate_persistent_feed()` tasks (shm writers via `sample_and_broadcast()`, history backfill via `piker.tsp`), plus backend eps `stream_quotes`, `open_history_client`, `open_symbol_search`, `get_mkt_info` from `piker/brokers/<backend>/feed.py`/`symbols.py`,
- **live order-control tasks**: `open_trade_dialog()` from `piker/brokers/<backend>/broker.py`, driven by `emsd`.
The codebase already anticipates this split: `piker/data/validate.py:70-91` groups backend eps into `'datad'` vs `'brokerd'` kinds, `piker/brokers/ib/__init__.py:62-70` already declares `_brokerd_mods`/`_datad_mods`, and `piker/brokers/_daemon.py:62` carries the literal TODO *"rename the daemon to datad prolly once we split up broker vs. data tasks into separate actors?"*. This work executes that split.
**User-decided constraints:**
- `datad.<broker>` is a **sibling** of `brokerd.<broker>` under `pikerd`, spawned via the existing `Services` + `maybe_spawn_daemon()` machinery (`piker/service/_daemon.py:46`).
- **Hard cutover, staged by layer** — no dual-mode runtime flag; every stage lands fully working.
- Post-split `brokerd` is **trading-only and lazily spawned solely by emsd's** `open_brokerd_dialog` path; UI/CLI/feed code never spawns it. Chart-only and paper sessions run with **zero** brokerd processes.
## Target topology
```
pikerd
├── datad.ib feed bus, shm writers, tsp history, symbol search
├── brokerd.ib open_trade_dialog only (EMS-spawned, lazy)
├── emsd
│ └── paperboi.ib (paper mode; opens its feed via datad)
└── samplerd
```
## Key verified facts (load-bearing)
- `feed.portals` has exactly ONE trading consumer: `piker/clearing/_ems.py:671` (`portal = feed.portals[brokermod]` → handed to `open_brokerd_dialog`). This is the single coupling forcing feed + trading into one actor.
- `assert 'brokerd' in servicename` at `piker/data/feed.py:502` is the only actor-name assert in the tree.
- `piker ledger` (`piker/accounting/cli.py:100-157`) is a hidden consumer: it calls `broker_init()` directly, spawns its own ad-hoc actor, enters `_setup_persistent_brokerd`, then calls `open_brokerd_dialog(brokermod, portal, ...)` **with an explicit portal** — the new signature must keep a `portal:` override param.
- kraken's feed→broker coupling is mild: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` (really from `piker.data._web_bs`) and `stream_messages` (pure parser) from `feed.py`. In-process imports only — `enable_modules` gates RPC, not imports. No live shared state; each actor opens its own WS.
- ib: default `client_id=6116` (`ib/api.py:1320`) with linear retry `clientId=client_id + i` (`:1403`). Two ib actors connecting concurrently will collide → needs a role-based id offset. Both daemons need `_spawn_kwargs={'infect_asyncio': True}` (copied by both init fns).
- binance has no `symbols.py` (`get_mkt_info`/`open_symbol_search` live in `binance/feed.py`); kucoin/questrade/robinhood are flat single-file with NO `__enable_modules__`; deribit has no `broker.py`.
- `tests/test_services.py:80,185` assert `brokerd` actor names incl. the paper-mode flow asserting `brokerd.kraken` spawns (`:229,:237`) — must invert post-split.
- `_root_modules` (`piker/service/_actor_runtime.py:156`) must gain the new datad daemon mod so `pikerd_portal.run(spawn_datad, ...)` resolves.
- `piker/brokers/core.py:175` `symbol_search` does `portal.run(search_w_brokerd)` where the target fn lives in `piker.brokers.core` → that mod must stay RPC-enabled in datad.
## Module decision
New file **`piker/data/_daemon.py`** hosts all datad machinery (`_setup_persistent_datad`, `datad_init`, `spawn_datad`, `maybe_spawn_datad`, `_datad_service_mods`). Mirrors the `piker.brokers._daemon` / `piker.data._sampling` (samplerd) per-subsystem convention and satisfies the existing TODO at `brokers/_daemon.py:49` ("move this def to the `.data` subpkg"). `piker.brokers._daemon` keeps brokerd-only code, slimmed. Do NOT over-DRY the two ~40-line init fns into a shared factory yet (samplerd precedent accepts the duplication).
---
## Stage 0 — prep: backend module grouping + ep introspection (no topology change)
1. `piker/data/validate.py`: add a `get_eps(mod, kind) -> dict[str, Callable]` helper returning the backend's defined eps for a daemon-kind from `_eps` (missing eps excluded). Used later by both init fns + fail-fast checks.
2. Declare daemon-kind module groups in each split backend's `__init__.py`, keeping `__enable_modules__` as the (deduped) union so behavior is unchanged:
- `kraken`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed', 'symbols']`
- `binance`: `_brokerd_mods = ['api', 'broker']`, `_datad_mods = ['api', 'feed']`
- `deribit`: `_brokerd_mods = []`, `_datad_mods = ['api', 'feed']`
- `ib`: adjust existing groups so each includes `'api'`
- flat backends (kucoin etc.): no attrs — init fns fall back to enabling just `modpath` (existing behavior).
3. Hygiene: `kraken/broker.py:77-81` imports `NoBsWs`/`open_autorecon_ws` from `piker.data._web_bs` directly (keep the `stream_messages` import from `.feed`).
**Gate**: full pytest green, zero behavior change.
## Stage 1 — introduce `datad` daemon machinery (additive)
New `piker/data/_daemon.py`:
- `_datad_service_mods: list[str]` — datad-always-enabled mods (successor to the data side of `_data_mods` from `brokers/_daemon.py:52`): `['piker.brokers.core', 'piker.brokers.data', 'piker.data', 'piker.data.feed', 'piker.data._sampling', 'piker.data._daemon']`.
- `_setup_persistent_datad(ctx, brokername, loglevel)` — `@tractor.context` fixture; logging boilerplate (as `_setup_persistent_brokerd:81-88`), then allocates the actor-global feed bus exactly as the brokerd fixture does today (`brokers/_daemon.py:105-121`): `assert not feed._bus`, open service nursery, `feed.get_feed_bus(brokername, service_nursery)`, `ctx.started()`, `sleep_forever()`.
- `datad_init(brokername, ...)` — mirrors `broker_init()` (`brokers/_daemon.py:132`): actor name `f'datad.{brokername}'`, copies backend `_spawn_kwargs` (**critical for ib infect_asyncio**), builds `enable_modules` from `getattr(brokermod, '_datad_mods', getattr(brokermod, '__enable_modules__', []))`.
- `spawn_datad(brokername, ...)` — mirrors `spawn_brokerd()` (`brokers/_daemon.py:202`): `Services.actor_n.start_actor(dname, enable_modules=_datad_service_mods + backend_mods, ...)` + `Services.start_service_task(dname, portal, _setup_persistent_datad, ...)`.
- `maybe_spawn_datad(brokername, ...)` — wraps `maybe_spawn_daemon(service_name=f'datad.{brokername}', service_task_target=spawn_datad, ...)` exactly like `maybe_spawn_brokerd` (`brokers/_daemon.py:256`).
Supporting edits:
- `piker/service/_actor_runtime.py:156`: add `'piker.data._daemon'` to `_root_modules` (keep `'piker.brokers._daemon'`).
- `piker/service/__init__.py`: re-export `spawn_datad`/`maybe_spawn_datad` next to the brokerd ones (`:56-57`).
- `tests/test_services.py`: new `test_datad_spawn` — `open_test_pikerd` + `maybe_spawn_datad('kraken')` + `ensure_service('datad.kraken')`. (Do NOT route a feed through it yet — `open_feed_bus`'s assert still says brokerd.)
**Gate**: suite green + new spawn test; nothing routes through datad yet.
## Stage 2 — clearing layer: emsd self-spawns brokerd (decouple from `feed.portals`)
Sequenced BEFORE the feed cutover so live trading works at every boundary (otherwise `feed.portals` would hand emsd a datad portal).
`piker/clearing/_ems.py`:
1. `open_brokerd_dialog()` (`:336`) new signature: `(brokermod, exec_mode, fqme=None, portal: tractor.Portal|None = None, loglevel=None)`. Internals: keep trades-ep detection (`:400-417`); acquire the brokerd actor ONLY when a live ep will actually open, via a small inner `@acm _acquire_live_portal()` that yields the passed `portal` if given (the `piker ledger` path) else `async with maybe_spawn_brokerd(brokermod.name, loglevel=loglevel)` — **the single place brokerd boots post-split**. Move the eager `portal.open_context(trades_endpoint, ...)` construction (`:425`) inside that block; the paper short-circuit (`:432`) never touches it.
2. `Router.maybe_open_brokerd_dialog()` (`:581`) — drop the `portal` param; `Router.open_trade_relays()` (`:640`) — delete `portal = feed.portals[brokermod]` (`:671`) and the pass-through (`:685`).
3. `piker/accounting/cli.py:144`: switch to keyword form `open_brokerd_dialog(brokermod, exec_mode=..., portal=portal, loglevel=...)`.
Pre-cutover this is a pure refactor: emsd's `maybe_spawn_brokerd` just *finds* the already-running brokerd via the registry.
**Gate**: `tests/test_ems.py` + services suite green; manual paper order on kraken; `piker ledger` smoke.
## Stage 3 — feed layer hard cutover to datad
1. `piker/data/feed.py`: import `maybe_spawn_datad` (replacing `maybe_spawn_brokerd`, `:56-58`); `open_feed()` `:895` → `maybe_spawn_datad(...)` (rename `brokerd_ctxs` → `datad_ctxs`); `open_feed_bus()` `:502` → `assert 'datad' in servicename`; comment sweep.
2. `piker/brokers/_daemon.py` `_setup_persistent_brokerd()`: slim to trading-only fixture — logging setup, `ctx.started()`, `sleep_forever()`. Drop the bus alloc, `assert not feed._bus`, the service nursery (backend `open_trade_dialog` ctxs own their task trees), the `eg.ExceptionGroup` handler, and the stale `_FeedsBus` TYPE_CHECKING import. (`piker ledger`'s ad-hoc actor enters this same slimmed fixture — exactly what it needs.)
3. Repoint remaining data-flavored spawn sites `maybe_spawn_brokerd` → `maybe_spawn_datad`:
- `piker/ui/_app.py:40,57` (symbol search; optionally rename `install_brokerd_search` → `install_datad_search`, def at `piker/data/feed.py:766`)
- `piker/brokers/core.py:33,175` (`symbol_search`)
- `piker/brokers/cli.py:38,190,320` (`brokercheck`, `record`)
- `piker/ui/cli.py:30,72,121` (legacy kivy monitor/optschain) + best-effort `piker/ui/kivy/option_chain.py:498` `wait_for_actor('brokerd')` → `'datad'`
4. `tests/test_services.py`: `:80,:185` `actor_name = 'datad'`; paper-mode test asserts `datad.kraken` + `paperboi.kraken` + `emsd` AND adds the headline negative check: `assert 'brokerd.kraken' not in services.service_tasks`.
**Gate**: full suite with inverted assertions; manual: chart on binance/kraken shows `datad.<broker>` in `piker services` and NO brokerd; paper order fills; symbol search works; history backfill sane.
## Stage 4 — caps-sec slimming, validation, ib client-id
1. `piker/brokers/_daemon.py`: delete `_data_mods` (`:52`); `spawn_brokerd()` `:236` → `enable_modules=tractor_kwargs.pop('enable_modules')`; `broker_init()` `:184` reads `getattr(brokermod, '_brokerd_mods', getattr(brokermod, '__enable_modules__', []))`. Resulting brokerd enable set has no `piker.data.*` at all.
2. Fail-fast ep validation via Stage-0 `validate.get_eps()`: `broker_init` raises a clear error when `get_eps(mod, 'brokerd')` is empty (e.g. "kucoin is datad-only — use paper mode"); `datad_init` warns analogously.
3. ib client-id mitigation in `load_aio_clients()` (`ib/api.py:~1316`): role-based offset when `client_id` is the 6116 default (e.g. `+16` when `'datad' in tractor.current_actor().name`), optionally configurable via `brokers.toml` keys.
4. Docs/comment sweep: resolve `brokers/_daemon.py:62` TODO, `piker/cli/__init__.py:253` TODO, daemon list in `piker/service/__init__.py:21` docstring.
**Gate**: full suite; audit greps clean: `grep -rn maybe_spawn_brokerd piker/` hits only `clearing/_ems.py`, `service/__init__.py`, `brokers/_daemon.py`; no spawn-path `'brokerd'` literals left in `piker/data`, `piker/ui`, `piker/brokers/cli.py`.
---
## Risk register
| Risk | Sev | Mitigation |
|---|---|---|
| ib: `datad.ib` + `brokerd.ib` both connect to TWS/gw; `client_id=6116` collision burns `connect_timeout` retries | High (live ib only) | Stage 4 role-based id offset; both daemons inherit `infect_asyncio` via `_spawn_kwargs` copy in both init fns |
| `piker ledger` ad-hoc actor path silently broken by signature change | Med | `portal:` override kept on `open_brokerd_dialog` (Stage 2); explicit smoke test |
| datad-only backends (kucoin/deribit) → useless brokerd spawn | Low | already covered: `open_brokerd_dialog` forces `exec_mode='paper'` when no trades ep (`_ems.py:413-417`) BEFORE the lazy spawn; Stage 4 fail-fast |
| brokerd in-process symbology/ledger needs (ib `broker.py` imports `.symbols`; kraken uses `stream_messages`) | None | verified pure in-process imports; `enable_modules` gates RPC only |
| paper-mode test asserts brokerd spawns (`test_services.py:237`) | Low | Stage 3 deliberately inverts it |
| doubled per-broker clients (HTTP/WS, symcache loads) | Low | each actor already opens its own WS today; symcache is disk-read-mostly |
## Verification (per stage gates above, plus end-to-end)
- `pytest tests/test_services.py tests/test_feeds.py tests/test_ems.py` at every stage (kraken/binance public endpoints, no creds needed).
- Manual smoke matrix post-Stage-3: `pikerd -l info` + `piker chart btcusdt.spot.binance` → `piker services` shows `datad.binance`, no brokerd; submit paper order → `paperboi.binance` appears, still no brokerd; symbol search; `piker ledger <broker>.paper`.
- If ib creds/gateway available: live ib chart + small live order to exercise dual-actor client-id path (Stage 4).
## Critical files
- **new** `piker/data/_daemon.py` — all datad machinery
- `piker/brokers/_daemon.py` — slimmed brokerd fixture/init/spawn
- `piker/data/feed.py` — spawn cutover + actor-name assert
- `piker/clearing/_ems.py` — emsd lazy brokerd spawn (`open_brokerd_dialog`, `Router`)
- `piker/service/_actor_runtime.py`, `piker/service/__init__.py` — root mods + re-exports
- `piker/data/validate.py` — `get_eps()` helper
- backend `__init__.py`s (kraken/binance/deribit/ib) — `_datad_mods`/`_brokerd_mods`
- `piker/accounting/cli.py` — keyword-form dialog call
- `tests/test_services.py` — inverted + new assertions

View File

@ -0,0 +1,233 @@
# Bounded IB recovery candidate
Prepared 2026-10-22 on `wkt/land_ib_recovery`, in the main checkout
`/home/goodboy/repos/piker`. Base:
`dc618229bd8c9c90aa3160e26677b553a7d472c7`.
Changes are unstaged and uncommitted. No acceptance is implied.
## Patch selection and provenance
Keep these existing landing commits:
| Commit | Reason |
| --- | --- |
| `4133f073` and its ancestors | Packaging, Nix, runtime API and pin work |
| `e8e0730a` | Correlate MethodProxy requests and responses |
| `7bef50b0` | One response reader, per-request waiters |
| `d1571ddf` | Propagate daemon-handler exceptions |
| `dc618229` | Isolate cached feed stream consumers |
Proposed separate implementation boundaries, in this order:
1. Back out the code and original test from
`635314c757c74a1cd95c5cf3dc7dfd03be984b00`.
This restores the direct-owner `_quote_streams` implementation.
Retain the historical prompt records. Replace the removed
mocked-cache regression with the real-bridge restart reproducer.
This backout is not a claim that arbitrary concurrent direct
callers have safe broadcast ownership.
2. Backport **only** `piker/brokers/ib/feed.py` from
`0846cbd4a4197e47415689cf5e9d146ba3174c32`.
Retain its seven history-adapter regressions in
`tests/test_ib_history.py`. This includes reset serialization,
completion/result signalling and bounded cancellation/failure.
It does not include the original SHM/storage orchestration patch.
3. Backport **only** `piker/brokers/ib/feed.py` from
`1da9b760a3bc6b96501110cab2b5d09846d42a1d`.
Retain `test_empty_history_preserves_request_boundary` and its
six cases. This preserves no-data request boundaries and wakes
the waiter. It does not include the original history-layer
endpoint/filtering changes. Add the real-bridge history recovery
case after these adapter changes are present.
4. Import the code and original regression from
`12b0d66e854247e92cb5a8668be71d816c72d856`.
Add a separate real-broadcaster/production-loop regression.
5. Remove the single-null-sample debugger stop in `get_null_segs()`.
This is a local UX cleanup retaining the existing `None` result.
It does not backport the null-grouping/deduplication rewrite in
`976db6c33c4b8ed07eee3047226417d3928d9cca`.
6. Record cross-boundary regression results and live qualification
evidence, distinguishing the known callback failure from passes.
All three imported source commits were authored by
`goodboy <goodboy_foss@protonmail.com>`; preserve that authorship
and the full source SHA when committing each partial backport.
Use explicit partial-backport wording, not a whole-commit cherry
claim. New real-runtime tests are this session's additional work.
The current aggregate working diff contains these boundaries in
the same feed file. Staging that entire file would combine the
first three boundaries. Reconstruct/stage the original per-source
patches in the above order when preparing commits.
## Quote-cache history correction
`4af885d90f6e4bcfa963f2f7a1d40436d3c928f3` and landing commit
`635314c757c74a1cd95c5cf3dc7dfd03be984b00` have the same stable
patch ID: `bc54892c25e957be112b135bb9e383ec4dd123c5`.
They are equivalent replays, not different implementations.
`wkt/ib_downstream_pin_eval@e929fdc2` lacks that quote-cache
change and retains direct ownership. The original qualification
did not validate the combination assembled for the landing.
After this candidate's edits, `piker/brokers/ib/feed.py` matches
the source at `1da9b760` byte-for-byte. Compared with the evaluated
tip, only the later schedule/reservation APIs are absent.
The EMS code patch has the same stable patch ID as the original
`12b0d66e` EMS code patch:
`50fee047e94fd67c8d9feab66859fbe3ab25b64c`.
## Real-runtime evidence
`tests/test_ib_quote_restart.py` uses real TCP Tractor actors,
an asyncio-infected child, the installed pinned Tractor bridge,
production Piker quote setup, and real IB/eventkit ticker events.
Only the external IB connection/input is replaced with replay data.
Before the quote-cache backout, cancellation after ticker delivery
raised `RemoteActorError[TrioTaskExited]` through
`maybe_open_context()` / `_Cache.run_ctx` teardown, matching the
reported fatal path.
After the backout:
- Two quote generations finish; each subscription is cancelled,
each asyncio producer is done and each ticker handler removed.
- A second RPC repeats the sequence in the same actor successfully.
- The strict callback-cleanliness case still fails: Tractor's
`signal_trio_when_done()` raises `TrioTaskExited` into asyncio's
exception handler even though the Trio consumer recovers.
The history integration case keeps `Client.bars()`, MethodProxy,
both relays, `get_bars()`, `wait_on_data_reset()` and
`data_reset_hack()` real. It replaces external contract discovery,
IB history replies and VNC input. A real IB error event announces
farm recovery. The replay rejects retries before recovery and
proves one reset followed by completion for both 1s and 60s data.
The new EMS case runs the production dark-clearing loop over a
real Tractor broadcaster: both EMS and a sibling receive a tick,
EMS cancellation removes only its subscription, and the sibling
continues with an unchanged strict lag policy.
## Verification
Unique selected cases: **38 passed, 1 failed**.
The failure is intentionally visible, not skipped or xfailed.
Run from the checkout's working Nix environment:
```xsh
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_ib_history.py tests/test_ems_broadcast.py tests/test_ib_method_proxy.py tests/test_feed_broadcast.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q --tb=short tests/test_ib_quote_restart.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_logging.py tests/test_startup.py
timeout -k 5 45 py313/bin/python -m pytest -p no:xonsh -q -x --tb=short tests/test_services.py::test_runtime_boot
git diff --check
```
The adapter/proxy/feed/EMS selection initially passed 21 cases;
the added real EMS case then passed with its imported case.
The bridge file passed two cases and failed callback cleanliness.
Logging/startup passed 13 cases; runtime boot passed one with the
existing `Registrar.uid` deprecation warning.
The agent shell inherited another project's environment and lacked
native libraries. Tests used the existing Piker `py313` via
`/tmp/nix-shell.zfKbeB/opencode/piker_test_env.py`, which supplies
libraries from the pinned flake's existing Qt dependency closure,
excluding Nix Qt itself and glibc. No environment sync or shellHook
ran. Piker resolved to this checkout; Tractor resolved to its
installed package in `py313`.
Workflow-state preparation initially reported changed legacy
files/modes after migration. On 2026-10-23 the human authorized
reconciliation. Inspection found only two missing legacy guidance
snapshots; the neutral copies matched their saved hashes and modes.
Restoring those local snapshots resolved preparation without
changing archived messages or the original migration receipt.
## Human live qualification, 2026-10-23
The human ran these journeys with the installed Tractor pin,
the existing IB Gateway and pre-existing NativeDB. The agent did
not operate the broker connection, VNC, orders or database.
- Initial QQQ startup/backfill recovered after pacing error 162
with one reset attempt, then displayed live updates. The reset
waiter appears to have timed out; the log did not establish an
acknowledged farm reset or live quote-stream restart.
- Sequential chart exits and reattachment preserved all four
daemon PIDs; a read-only process/listener check confirmed this.
- The overlapping two-chart journey used `pikerd@961108`,
`brokerd.ib@964765`, `samplerd@965900` and `emsd@969015`.
B started at 08:16:31; C at 08:17:33. B closed at 08:18:11,
replacement B started at 08:18:24 and had FSPs up by 08:18:28,
then C closed at 08:18:37. The human reported the exercise worked;
survivor updates are a manual observation, while timestamps and
unchanged daemon identities corroborate overlap and service reuse.
- Startup in that journey hit the explicit single-null-sample
breakpoint several times before the charts overlapped. The small
debugger cleanup above was applied afterward; this live run does
not itself verify that later edit.
- Departing charts still report cancellation groups and an FSP
receiver-close `ClosedResourceError`, followed by the Qt close
handler's self-SIGINT. Peer-disconnected final result warnings
remain contained in the persistent services.
- Daemon SIGINT at 08:19:43 reproduced cached shared-client teardown:
`load_clients_for_trio()` / `_Cache.run_ctx` promoted
`TrioTaskExited` through `open_trade_dialog()` and entered PDB.
After the human continued, all three children were reported
exited with status 0 and the root terminated at 08:19:45.
Cancellation-only RPC error reporting and late cancel-ack
diagnostics also remain.
The human explicitly accepted deferring this daemon teardown fix
from #94 and requested loud attribution for the bounded backports.
This is an operational smoke result, not clean-shutdown coverage.
## Downstream reconciliation requirements
Before landing #94, the human wants an easy replay of the downstream
histories intended to follow it. Rehearse against the agreed #94
tree in disposable worktrees, then inspect range-diffs, final code
and relevant regressions. This note does not authorize rewriting
the original branches or changing acceptance markers.
- `0846cbd4` and `1da9b760` are partial backports. Their imported
adapter hunks may already be present, but their history/SHM/storage
changes and omitted integration tests remain necessary downstream.
Never drop either whole source commit merely because `feed.py`
matches or its adapter hunk becomes empty.
- `12b0d66e` has source-equivalent EMS code and its imported test,
plus an additional local real-broadcaster test. Preserve that extra
regression when resolving test-file overlap.
- The quote-cache backout intentionally restores direct ownership
for this landing. Reintroducing original `4af885d9` downstream
requires qualifying cached-owner teardown with the matching
Tractor revision; the backout is not a permanent ownership design.
- The local single-null breakpoint removal retains legacy behavior.
Prefer the full downstream `976db6c3` algorithm when it is replayed;
preserving this `return None` guard would lose that later fix.
- Real-bridge tests and manual qualification notes are local work,
not evidence that omitted portions of the source commits landed.
Commit messages must name full source SHAs, original authorship,
imported paths/tests, omissions, local additions and replay guidance.
Use explicit partial-backport language for adapter subsets; do not
claim a full cherry-pick or reuse the source's broader fix claims.
## Remaining decision
This candidate removes the reproduced fatal quote-restart
propagation and imports existing reset fixes. The human exercised
live QQQ startup, backfill and concurrent chart consumer lifetimes.
Completion-callback and teardown errors remain; the human accepts
them as follow-ups outside the bounded #94 landing.
The storage/SHM portions of the downstream fixes are not covered by
the adapter backport or its integration tests.
`ddac3553` is Deribit-specific and omitted. Qt lifecycle commit
`1a2d4893` stays a separate follow-up. Further Tractor teardown
changes or storage imports require reassessing the bounded landing
rather than silently expanding this candidate.

View File

@ -0,0 +1,71 @@
# IB recovery qualification, 2026-11-06
This records human-operated qualification of the bounded #94
landing on `wkt/land_ib_recovery`, starting from `495c0915` plus
the narrow history catch described below. It supplements the
historical evidence in `ai/ib-recovery-candidate.md`.
## Runtime and dependency source
The evaluated environment uses Python 3.13, Trio 0.34.0 and Tractor
`8fe574c1e447358dafd8ebf5ada3b0714c688f78`. Tractor was an editable
local checkout during evaluation. The human subsequently published
that exact commit as GitHub's `goodboy/tractor:piker_pin`; a read-only
remote query confirmed the published OID before restoring Piker's
Git source and recording the same OID in `uv.lock`.
Offline lock validation resolves 126 packages successfully. No
environment sync was performed. This validates the lock metadata;
it does not claim a fresh installation from the published source.
## Manual observations
The human operated Gateway, chart windows and paper orders. The
agent did not operate live services or the account.
- Same-feed overlap, close/reopen and paper order submit/cancel
worked after the existing order-dialog guard was imported.
- Different-feed overlap originally exposed a timed-out one-second
MNQ March 2027 gap query. PDB showed `timedout=True` and
`out is None`. Its uncaught `NoData` canceled sibling history and
quote tasks even though `pikerd` remained alive.
- After the narrow catch, the human reported the overlap journeys
working, then confirmed cases 5 and 6 with Ctrl+Alt+F: overlapping
live feeds survived the Gateway data reset; closing all charts
and reopening with `pikerd` left running restored live updates.
- Earlier Ctrl+Shift+R attempts left feeds live, but do not prove
that the intended Gateway data reset was triggered.
These are manual observations, not automated timing or latency
measurements. Ctrl+Alt+R was not qualified here; Piker's utility
labels it a connection reset.
## Narrow history backport
Only the `NoData`/`DataUnavailable` catch around `get_hist()` in
`maybe_fill_null_segments()` is copied from
`0846cbd4a4197e47415689cf5e9d146ba3174c32`, authored by
`goodboy <goodboy_foss@protonmail.com>`. The existing gap fallback
remains in place; this does not import broader history repair.
The new regression uses real null detection and checks that a
sibling in the same nursery still runs after either provider
exception. Both cases failed before the catch. With it, the two
focused history files passed: 17 tests. Earlier bounded-recovery
and Tractor qualification results remain separate evidence, not
fresh test runs of the final dependency commit tree.
## Remaining recovery limits
A full Gateway restart was triggered accidentally through its
auto-restart settings dialog. The disconnect delivered the string
`Peer closed connection.` to `deliver_trade_events()`. The string
branch then reached `err['reqid']`, raising `TypeError`; its `code`
local can also retain an earlier event's value. Full Gateway restart
recovery is outside this bounded landing.
The earlier history-failure teardown also exposed a separate
Tractor completion-callback assertion after `BrokenResourceError`.
The history catch does not establish that this callback race is
fixed. Broader history recovery and these disconnect paths remain
downstream work.

View File

@ -0,0 +1,52 @@
---
created_at: 2026-10-07T16:54:53Z
generated_by:
harness: opencode
provider: openai
model: gpt-6.1-sol
git_ref: bde2457b
scope: code
substantive: true
raw_file: 20261007T165453Z_bde2457b_prompt_io.raw.md
---
## Prompt
The user requested fixing all seven local PR94 review findings and
exporting the review for human inspection in Tuicr. They asked that a
Tractor pin adjustment be considered before adding Piker workarounds.
## Response summary
Correct the endpoint defaults, Click argument shape, daemon binding and
registrar option handling, chart fallback, and stray debugger pause.
Keep the absent datad implementation's test outside this landing
branch. Add offline startup coverage and export original findings
without modifying review states. The checked newer Tractor address
branch still rejects mixed transports, so leave the dependency pin.
## Files changed
- `config/conf.toml`: TCP-only default endpoints.
- `piker/cli/__init__.py`: repeatable and intact multiaddresses.
- `piker/service/_actor_runtime.py`: validate daemon listeners.
- `piker/service/_daemon.py`: initialize and consume registrar options.
- `piker/ui/cli.py`: fallback to automatic chart binding.
- `piker/brokers/binance/feed.py`: remove unconditional debugger entry.
- `tests/test_services.py`: defer test requiring missing datad API.
- `tests/test_startup.py`: CLI/config and service lifecycle regressions.
## Human edits
The human identified that datad tests arrived through the agent's
mixed cherry-picks and explicitly requested deferring that test. They
requested a chart default for existing configs, expanded the task to
all review findings, and directed checking whether a newer Tractor pin
could reduce the fix scope. No direct manual source edits were observed
during these changes; acceptance remains with the human.
During the human review pass, the human confirmed removing the Binance
pause and correcting CLI endpoint parsing. They requested an explicit
template note about this pin's single-transport limitation and detailed
comments distinguishing registrar discovery endpoints from daemon bind
endpoints. Those explanations were added at the human's direction.

View File

@ -0,0 +1,61 @@
---
created_at: 2026-10-07T16:54:53Z
generated_by:
harness: opencode
provider: openai
model: gpt-6.1-sol
git_ref: bde2457b
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
Fix all findings from the local PR94 review, prefer a Tractor pin
change if it reduces work, and export the original review for Tuicr.
## Generated changes
> `git diff HEAD~1..HEAD -- config/conf.toml`
Use TCP-only defaults, retaining commented UDS alternatives. Both
checked Tractor address branch tips still reject mixed transports.
> `git diff HEAD~1..HEAD -- piker/cli/__init__.py`
Make Click multiaddresses repeatable and normalize single strings.
> `git diff HEAD~1..HEAD -- piker/service/_actor_runtime.py`
Validate requested daemon bind addresses instead of remote registrars.
Normalize accepted endpoints and allow TCP ephemeral port requests.
> `git diff HEAD~1..HEAD -- piker/service/_daemon.py`
Resolve explicit or inherited registrars independently of other daemon
options and consume the explicit keyword before forwarding kwargs.
> `git diff HEAD~1..HEAD -- piker/ui/cli.py`
Default chart bind endpoints to None when network config is absent.
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Remove the unconditional market-info debugger pause.
> `git diff HEAD~1..HEAD -- tests/test_services.py tests/test_startup.py`
Remove the prematurely cherry-picked datad test and add offline startup
regressions, including a real registrar/child daemon lifecycle check.
## Verification
- Startup and logging tests: 13 passed.
- Existing runtime boot and tagged registrar tests: 2 passed.
- Scoped native Ruff and diff checks pass.
- The first new actor test used a removed nursery API; it was corrected
to the installed start_actor/Portal.run interface before passing.
- Import initially failed without native libraries. Tests use the
existing flake's GCC, zlib, Qt and PyQt paths without executing its
environment-rebuilding shell hook.
- Original seven findings exported locally as Tuicr review-add payloads
bound to bde2457b; no session comments or remote PR were changed.

View File

@ -0,0 +1,45 @@
---
created_at: 2026-10-10T23:28:52Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 1fa45dfa
scope: config
substantive: true
raw_file: 20261010T232852Z_1fa45dfa_prompt_io.raw.md
---
## Prompt
The human rejected keeping the April Tractor pin as a landing baseline.
They requested the reviewed `wkt/addr_unpacking` branch, removal of the
new rollback adaptations, reuse of original Piker histories where
possible, and a candidate prepared within their gateway relocation
window. They asked specifically whether the runtime validation review
hunk duplicated older endpoint history.
## Response summary
Select the reviewed address branch at an immutable locked revision and
preserve all committed application/test compatibility code. Verify the
minimal dependency boundary offline and preview existing consumer
fixes without importing unrelated datad/backfill changes. Distinguish
original-source test results from live teardown acceptance, which is
not available in this preparation window.
## Files changed
- `pyproject.toml`: select Tractor `wkt/addr_unpacking`.
- `uv.lock`: resolve reviewed `a966928d` and setproctitle 1.3.8 while
preserving Polars 1.36.0 and the rest of the locked graph.
## Human edits
The human redirected the task from an older known-good runtime to a
modern landing stack and insisted on reusing original history rather
than building new compatibility adapters. They proposed stricter
exception-group behavior as a hypothesis and set the live-test/time
constraints. Those directions materially changed the dependency
boundary. No direct manual edits to this final boundary were observed;
runtime acceptance and landing remain human-owned.

View File

@ -0,0 +1,52 @@
---
created_at: 2026-10-10T23:28:52Z
generated_by:
harness: opencode
provider: openai
model: openai/gpt-6.1-sol
git_ref: 1fa45dfa
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
Undo the April Tractor rollback and select the reviewed
`wkt/addr_unpacking` branch. Prefer original downstream compatibility
commits to new adaptations, and prepare an offline-verified candidate
while live IB verification is deferred for gateway relocation.
## Generated changes
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
Change Tractor's source from `piker_pin` to `wkt/addr_unpacking` and
lock it at `a966928dead4cbf369ad244834ce9dbe988ac8ca`. Restore application
and test sources byte-for-byte to the committed PR94 tree. Keep the
rejected rollback journal only in ignored local experiment reports.
Preserve Polars 1.36.0 by retaining its exact existing lock blocks;
setproctitle re-resolves from 1.3.7 to 1.3.8. `uv sync --locked` accepts
the resulting graph. Actual imports confirm the reviewed Tractor pin
and Trio 0.32.0, not the hypothesized Trio 0.33.
## Original-history previews
The existing MethodProxy stack is retained. Read-only merge previews
show original `4af885d9` and `b691422b` can be applied without the datad
cutover. Original-source regressions run through temporary in-memory
overlays against the real pinned runtime: seven cases pass and one
cache-hit case fails because `log.at_least_level('info')` looks up INFO
in a custom-level mapping. A separate standard logging correction
produces eight passing cases. Its patch and execution handoff are saved
under ignored `.ai/code-review/reports/pr94_modern_stack/`; no consumer
cherry-picks or source edits are performed by this boundary.
## Verification
- Startup and logging: 13 passed.
- Runtime boot and tagged registrar normalization: 2 passed.
- Locked sync and source whitespace checks pass.
- Original application/test trees are unchanged from PR94 HEAD.
- Consumer overlay with explicit logging correction: 8 passed.
- Correction patch validates against its exact preview tree using a
private temporary index; the real index is untouched.
- Live IB startup, SIGINT and surviving-pikerd reattachment are pending.

View File

@ -0,0 +1,45 @@
---
created_at: 2026-11-06T18:56:28.621577+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
scope: code
substantive: true
raw_file: 20261106T185628Z_495c0915_215e9b_prompt_io.raw.md
---
## Prompt
User reported a different-feed overlap failure and asked whether
a fix could be extracted without importing broader history work.
PDB confirmed MNQ March 2027 one-second history timed out and
returned None, producing NoData. After continuing PDB, sibling
history and quote tasks terminated while pikerd remained alive.
User requested continuing with the narrow backport.
## Response summary
Partial backport of the NoData/DataUnavailable catch from
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
maybe_fill_null_segments(). The existing gap fallback remains in use.
A new parametrized regression uses real null detection and verifies
that a sibling in the same nursery survives the failed repair.
Both new cases failed before the catch. The focused history suites
passed after it: 17 passed. This does not qualify live overlap or fix
IB reset churn or the separate Tractor completion-callback assertion.
## Files changed
- `piker/tsp/_history.py`: existing downstream exception catch.
- `tests/test_history_backfill.py`: sibling survival regression.
## Human edits
User supplied live reproduction, PDB locals and cancellation logs,
and explicitly constrained scope to a small history extraction.
The production catch is copied from existing downstream commit
0846cbd4, not newly designed here. New regression is agent-authored;
no direct human source edits observed during this patch. Review and
live qualification remain with the user.

View File

@ -0,0 +1,22 @@
---
created_at: 2026-11-06T18:56:28.621577+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Partial backport of the NoData/DataUnavailable catch from
0846cbd4a4197e47415689cf5e9d146ba3174c32 into
maybe_fill_null_segments(). The existing gap fallback remains in use.
A new parametrized regression uses real null detection and verifies
that a sibling in the same nursery survives the failed repair.
Both new cases failed before the catch. The focused history suites
passed after it: 17 passed. This does not qualify live overlap or fix
IB reset churn or the separate Tractor completion-callback assertion.

View File

@ -0,0 +1,41 @@
---
created_at: 2026-11-06T20:17:46.622316+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
scope: config
substantive: true
raw_file: 20261106T201746Z_495c0915_612dfc_prompt_io.raw.md
---
## Prompt
The human requested a commit plan, then a fast-forward landing of
the #94 branch and PR updates. They agreed the temporary local
Tractor source must be replaced, published piker_pin themselves,
and reported "pushed it". They also supplied the manual recovery
results and full Gateway restart failure diagnostics.
## Response summary
Restore the Git Tractor source after local qualification and lock
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
Record the human's overlap, data-reset and reopen observations,
the narrow history extraction, and full-restart recovery limits.
Offline uv lock validation passed without an environment sync.
## Files changed
- `pyproject.toml`: Trio minimum; restore existing Tractor Git source.
- `uv.lock`: published Tractor OID and evaluated Trio 0.34.0.
- `ai/ib-recovery-qualification-20261106.md`: scoped evidence.
## Human edits
The human selected the bounded landing scope, performed live tests,
identified the more aggressive reset distinction, and published
the tested Tractor ref. The agent applied the metadata edits and
wrote the qualification record from those observations. No human
acceptance or task-state transitions are inferred by this record.

View File

@ -0,0 +1,21 @@
---
created_at: 2026-11-06T20:17:46.622316+00:00
generated_by:
harness: codex
provider: OpenAI
git_ref: 495c0915
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- pyproject.toml`
> `git diff HEAD~1..HEAD -- uv.lock`
> `git diff HEAD~1..HEAD -- ai/ib-recovery-qualification-20261106.md`
Restore the Git Tractor source after local qualification and lock
its published piker_pin at 8fe574c1e447358dafd8ebf5ada3b0714c688f78.
Retain the evaluated Trio 0.34.0 lock and require Trio >=0.34.0.
Record the human's overlap, data-reset and reopen observations,
the narrow history extraction, and full-restart recovery limits.
Offline uv lock validation passed without an environment sync.

View File

@ -0,0 +1,11 @@
# AI Prompt I/O Log
This directory records substantive AI-assisted changes, prompts,
original harness identity, and unedited outputs. Human contributors
remain accountable for code decisions.
Records follow the [NLNet generative AI policy][policy]. Each
structured entry links its immutable `.raw.md` partner. Code output
uses Git diff references to avoid duplication.
[policy]: https://nlnet.nl/foundation/policies/generativeAI/

View File

@ -0,0 +1,59 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:08:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T170859Z_75cefe10_prompt_io.raw.md
---
## Prompt
Session-initiating instruction (driving all 7 commits in
this series):
> ok i want you to become the distributed runtime and
> concurrency expert for this project - namely acquire
> a deep understanding of tractor and how it's used.
> then i want you to attempt to factor our current
> brokerd service daemon into 2 daemons: a brokerd
> which only hosts live/paper trading endpoint tasks
> [..] a new `datad` subdaemon which in a separate
> subactor serves all the data feed service tasks [..]
> give me a mega detailed plan on how to approach this,
> and a staged approach for the implementation.
Proximate driver for THIS commit: during the approved
plan's stage-0 test gate the agent discovered test
subactors were writing into the user's REAL
`~/.config/piker/accounting/` files (a bogus paper fill
landed in `account.kraken.paper.toml`) because the old
test-dir override in `config.get_app_dir()` was
commented out and could never work in spawned
subactors anyway. Fix applied autonomously under the
approved plan; no explicit per-fix user prompt.
## Response summary
Restore `pytest` config-dir isolation by resolving the
per-test tmp dir lazily (at conf-path access time) from
`tractor` runtime-vars, which propagate down the actor
tree; route all conf-path derivation through
`config.get_conf_dir()` so the override is effective in
every (sub)actor.
## Files changed
- `piker/config.py` — add `_maybe_use_test_dir()`;
hook in `get_conf_dir()`; route `get_conf_path()` +
`load()` mkdir through it
- `piker/accounting/_ledger.py` — derive ledger dir
via `config.get_conf_dir()` (not the global)
- `piker/accounting/_pos.py` — same for account files
## Human edits
None — committed as generated.

View File

@ -0,0 +1,46 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:08:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T170859Z_75cefe10_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/config.py`
Generated: `config._maybe_use_test_dir()` — lazily
reads `piker_test_dir` from
`tractor.runtime._state._runtime_vars['piker_vars']`
(pre-loaded by `open_piker_runtime()` from the
`tests.conftest._open_test_pikerd()` overrides) and
calls `_override_config_dir()` when set. Hooked at the
top of `get_conf_dir()`; `get_conf_path()` and
`load()`'s dir-creation rerouted through
`get_conf_dir()`.
> `git log -1 -p --follow -- piker/accounting/_ledger.py`
> `git log -1 -p --follow -- piker/accounting/_pos.py`
Generated: ledger/account dir derivation switched from
the `config._config_dir` module global to
`config.get_conf_dir()` + `mkdir(parents=True,
exist_ok=True)` for nested tmp-dir creation.
Key diagnostic reasoning (verbatim from session):
The old (commented-out) `get_app_dir()` override gated
on `'pytest' in sys.modules` which can NEVER work in
spawned subactors (fresh procs, no pytest import); as
a result test `paperboi`/daemon actors were writing
into the user's REAL `~/.config/piker/accounting/`
files. Evidence: `account.kraken.paper.toml` gained a
single 0.001 xbtusdt clear stamped 2026-06-09T18:47Z
(a test fill), and `trades_kraken_paper.toml` (252B)
contained only that fill. The resolution must be
checked lazily at config-path access time (NOT import
time) since sub-actors only receive runtime-vars once
their `tractor` runtime has fully booted.

View File

@ -0,0 +1,53 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:10:22Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171022Z_4485f2b9_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`). Proximate
driver: the approved plan's per-stage test gates could
not run AT ALL — the branch base was broken vs.
`tractor` git `main` (`AttributeError: 'tuple' object
has no attribute 'unwrap'` at `pikerd` boot, stale
`arbiter_sockaddr`/`.type`/`.size` API refs). The agent
fixed forward autonomously to (re)establish the gate
baseline, continuing the `repair_tests` branch lineage
(verified already merged in ancestry via `git cherry`
during a user-requested branch-overlap survey).
## Response summary
Port the service layer + test suites to current
`tractor` APIs: addr-type normalization in
`open_pikerd()`, `query_actor()`/`wait_for_actor()`
kwarg renames, modern self-cancel absorption semantics
in the cancel-method test harness, exc/position attr
renames, a paper-EMS startup-budget bump and a syntax
fix in `.deribit.api`.
## Files changed
- `piker/service/_actor_runtime.py` — `wrap_address()`
normalize before `.unwrap()` in `open_pikerd()`
- `piker/service/_registry.py` — `check_for_service()`
-> `query_actor(regaddr=)` + 2-tuple yield +
`open_registry(addrs=)`
- `piker/brokers/deribit/api.py` — missing comma in
`tractor.trionics` import tuple
- `tests/test_services.py` — `registry_addr=` kwarg,
raddr unwraps, cancel-semantics harness rewrite,
`fail_after` 9 -> 19s
- `tests/test_ems.py` — `.boxed_type`, `pp.cumsize`
## Human edits
None — committed as generated.

View File

@ -0,0 +1,56 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:10:22Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
Generated: normalize each registry addr via
`tractor.discovery._addr.wrap_address()` before
`.unwrap()`-ing for the `accept_addrs` bind check —
entries may be raw `tuple`s when passed in from (test)
client code. Import-path precedent taken from
`piker/cli/__init__.py:336`.
> `git log -1 -p --follow -- piker/service/_registry.py`
Generated: `check_for_service()` ported to
`tractor.query_actor(name, regaddr=...)` (kwarg was
`arbiter_sockaddr=`), unpacking the new
`(sockaddr, portal)` yield, and passing the
now-required `open_registry(addrs=Registry.addrs)`.
> `git log -1 -p --follow -- tests/test_services.py`
> `git log -1 -p --follow -- tests/test_ems.py`
> `git log -1 -p --follow -- piker/brokers/deribit/api.py`
Key diagnostic reasoning (verbatim from session):
- the "DID NOT RAISE ContextCancelled" failure: in this
test the client actor IS pikerd (in-proc), and
current `tractor` main absorbs a `ContextCancelled`
whose canceller is your own actor — self-requested
cancels now exit cleanly instead of raising; the
'sigint' variant propagates a bare collapsed
`KeyboardInterrupt` rather than a
`BaseExceptionGroup`.
- the hard-coded `trio.fail_after(9)` startup budget is
marginal — full stack boot (pikerd -> emsd ->
brokerd.kraken -> paperboi + live kraken symbology
fetch) occasionally exceeds 9s -> bumped to 19s.
- `RemoteActorError.type` -> `.boxed_type`;
`Position.size` -> `.cumsize` (the paper engine
populates `BrokerdPosition.size` from `pp.cumsize`).
- overlap survey (user-requested): all of the
`repair_tests` branch commits are already in this
stack's ancestry; this commit finishes that branch's
port mission (its f4c4f1e2 fixed `conftest.py`'s
`arbiter_sockaddr` usage; this fixes the remaining
`test_services.py` + `check_for_service()` sites).

View File

@ -0,0 +1,54 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:11:05Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171105Z_bc6e18d7_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 0" prep. Plan-shaping user
decisions captured via in-session Q&A:
- `datad.<broker>` topology: sibling of
`brokerd.<broker>` under `pikerd` (vs. child
subactor).
- migration: hard cutover, staged by layer (no
dual-mode runtime flag).
- post-split `brokerd` scope: trading-only +
EMS-lazy-spawned (charts/CLI never touch it).
## Response summary
Declare per-daemon-kind backend submod groups
(`_datad_mods`/`_brokerd_mods`) in the split-style
backends, keyed to the pre-existing
`piker.data.validate._eps` contract; add a
`validate.get_eps()` introspection helper; ws-import
hygiene in `.kraken.broker`. Zero behavior change
(`__enable_modules__` unions unchanged).
## Files changed
- `piker/data/validate.py` — add `get_eps()`
- `piker/brokers/kraken/__init__.py` — mod groups
- `piker/brokers/binance/__init__.py` — mod groups
(note: no `symbols.py`; search eps live in `.feed`)
- `piker/brokers/deribit/__init__.py` — datad-only
groups (no `broker.py` yet)
- `piker/brokers/ib/__init__.py` — add `'api'` to the
pre-existing `_datad_mods`
- `piker/brokers/kraken/broker.py` — import
`NoBsWs`/`open_autorecon_ws` from
`piker.data._web_bs` directly
## Human edits
None — committed as generated.

View File

@ -0,0 +1,46 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:11:05Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171105Z_bc6e18d7_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/validate.py`
Generated: `get_eps(mod, kind) -> dict[str, Callable]`
returning the daemon-kind's ep funcs defined by a
backend mod, keyed by ep name, missing eps excluded;
sourced from the existing `_eps` grouping table
(`'middleware' | 'datad' | 'brokerd'`).
> `git log -1 -p --follow -- piker/brokers/kraken/__init__.py`
> `git log -1 -p --follow -- piker/brokers/binance/__init__.py`
> `git log -1 -p --follow -- piker/brokers/deribit/__init__.py`
> `git log -1 -p --follow -- piker/brokers/ib/__init__.py`
> `git log -1 -p --follow -- piker/brokers/kraken/broker.py`
Key exploration findings driving the design (from the
planning phase, 3 parallel explore agents + 1 architect
agent):
- the codebase already anticipated this split:
`validate._eps` groups eps into 'datad' vs 'brokerd'
kinds and `ib/__init__.py` already declared
`_brokerd_mods`/`_datad_mods`; `brokers/_daemon.py`
carried the literal TODO "rename the daemon to datad
prolly once we split up broker vs. data tasks into
separate actors?".
- the feared kraken feed<->broker "shared ws state"
hazard is mild: `NoBsWs`/`open_autorecon_ws`
originate in `piker.data._web_bs` (re-exported via
`.kraken.feed`); only `stream_messages` is feed-local
and it's a pure parser — in-process imports only,
each actor opens its own ws conn.
- `enable_modules` gates RPC entry, NOT python imports,
so backend trading mods may keep importing feed-side
helpers in-process post-split.

View File

@ -0,0 +1,47 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:11:42Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171142Z_119d2c04_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 1": introduce the `datad` daemon
machinery additively (nothing routes through it yet).
User-decided constraint applied: `datad.<broker>` is a
SIBLING of `brokerd.<broker>` under `pikerd`, spawned
via the existing `Services` + `maybe_spawn_daemon()`
machinery.
## Response summary
New `piker/data/_daemon.py` hosting the
`datad.<broker>` feed-only daemon-actor: lifetime
fixture owning the actor-global `_FeedsBus`, init/spawn
fns mirroring `.brokers._daemon` conventions and the
`samplerd` sub-daemon precedent, plus root-mod
registration, `piker.service` re-exports and a spawn
test.
## Files changed
- `piker/data/_daemon.py` — NEW:
`_setup_persistent_datad()`, `datad_init()`,
`spawn_datad()`, `maybe_spawn_datad()`,
`_datad_service_mods`
- `piker/service/_actor_runtime.py` — add
`piker.data._daemon` to `_root_modules`
- `piker/service/__init__.py` — re-export spawn eps
- `tests/test_services.py` — add `test_datad_spawn`
## Human edits
None — committed as generated.

View File

@ -0,0 +1,60 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:11:42Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/_daemon.py`
Generated symbols + key design decisions:
- `_datad_service_mods: list[str]` — datad-always
enabled mods, the data-side successor to the old
`piker.brokers._daemon._data_mods` set; kept minimal
per the caps-sec model.
- `_setup_persistent_datad()` — `@tractor.context`
lifetime fixture: console-log setup then allocates
the actor-global feed bus via
`feed.get_feed_bus(brokername, service_nursery)`
exactly as the old brokerd fixture did, pinned open
with `ctx.started()` + `sleep_forever()`.
- `datad_init()` — actor name `f'datad.{brokername}'`;
copies backend `_spawn_kwargs` (CRITICAL for `ib`'s
`infect_asyncio=True`); builds `enable_modules` from
`getattr(brokermod, '_datad_mods',
getattr(brokermod, '__enable_modules__', []))` as
the flat-backend fallback.
- `spawn_datad()` — `Services.actor_n.start_actor()` +
`Services.start_service_task()` exactly mirroring
`spawn_brokerd()`; dedup-composes enable mods via
`list(dict.fromkeys(...))`.
- `maybe_spawn_datad()` — wraps `maybe_spawn_daemon(
service_name=f'datad.{brokername}', ...)`.
> `git log -1 -p --follow -- piker/service/_actor_runtime.py`
> `git log -1 -p --follow -- piker/service/__init__.py`
> `git log -1 -p --follow -- tests/test_services.py`
Design rationale (verbatim from session):
- `_root_modules` must gain `piker.data._daemon` so
`pikerd_portal.run(spawn_datad, ...)` resolves in
the root.
- the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep, surfaced by the
user-requested branch-overlap survey) lands
symmetrically on both spawn fns.
- mod placement (`piker/data/_daemon.py` vs.
generalizing `piker.brokers._daemon`) follows the
per-subsystem daemon-mod convention
(`.clearing._ems`, `.data._sampling`) and resolves
the existing TODO at `brokers/_daemon.py:49` ("move
this def to the `.data` subpkg").

View File

@ -0,0 +1,47 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:12:26Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171226Z_64181219_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 2": decouple the clearing layer
from `feed.portals` BEFORE the feed cutover so live
trading works at every stage boundary. User-decided
constraint applied: post-split `brokerd` is
trading-only and spawned LAZILY only by `emsd`'s
`open_brokerd_dialog()` path.
## Response summary
Kill the single coupling forcing feed + trading eps
into one actor (`Router.open_trade_relays()` pulling
its trades portal from `feed.portals[brokermod]`):
`open_brokerd_dialog()` now (maybe) spawns/finds
`brokerd.<broker>` itself and ONLY when a live
trades-ep will actually open; paper mode never touches
it. Pre-cutover this is a pure refactor (registry
lookup finds the same feed-spawned daemon).
## Files changed
- `piker/clearing/_ems.py` — `open_brokerd_dialog()`
re-sig + inner `acquire_live_portal()`;
`Router.maybe_open_brokerd_dialog()` drops `portal`
param; `open_trade_relays()` drops the
`feed.portals` lookup
- `piker/accounting/cli.py` — keyword-form `portal=`
override kept for the `piker ledger` ad-hoc actor
## Human edits
None — committed as generated.

View File

@ -0,0 +1,43 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:12:26Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171226Z_64181219_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/clearing/_ems.py`
Generated: new `open_brokerd_dialog()` signature
`(brokermod, exec_mode, fqme=None, portal=None,
loglevel=None)` with an inner `@acm
acquire_live_portal()` that yields the caller-provided
`portal` override (the `piker ledger` path) else
`maybe_spawn_brokerd(brokermod.name)` — designated THE
one place a live, credentialed `brokerd.<broker>` gets
booted post-split. The eager
`portal.open_context(trades_endpoint, ...)`
construction moved inside that block so the paper
short-circuit never acquires a live portal.
> `git log -1 -p --follow -- piker/accounting/cli.py`
Key analysis (verbatim from session):
- `feed.portals` had exactly ONE trading consumer:
`piker/clearing/_ems.py:671` (`portal =
feed.portals[brokermod]`) — the single coupling
forcing feed + trading into one actor.
- `piker ledger` (`accounting/cli.py:100-157`) is a
hidden consumer: it calls `broker_init()` directly,
spawns its own ad-hoc actor and passes the portal
positionally — the new signature must keep an
explicit `portal:` override param for this path.
- stage sequencing: clearing decouple lands BEFORE the
feed cutover, otherwise `feed.portals` would hand
emsd a `datad` portal and live `open_trade_dialog`
RPC would fail.

View File

@ -0,0 +1,48 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:12:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171259Z_59d5d9a6_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's "stage 3": the feed-layer hard cutover
(user-decided: no dual-mode runtime flag, each stage
lands fully working).
## Response summary
The topology flip: all data-feed consumers route to
`datad.<broker>`; the `brokerd` lifetime fixture slims
to trading-only; UI/CLI data-flavoured spawn sites
repointed; test expectations inverted incl. an explicit
negative assert that paper sessions spawn NO
`brokerd.<broker>`.
## Files changed
- `piker/data/feed.py` — `open_feed()` ->
`maybe_spawn_datad()`; `open_feed_bus()` actor-name
assert -> `'datad'`; `install_brokerd_search` ->
`install_datad_search`; comment sweep
- `piker/brokers/_daemon.py` —
`_setup_persistent_brokerd()` slimmed (feed-bus
alloc moved to the datad fixture)
- `piker/ui/_app.py`, `piker/ui/cli.py`,
`piker/brokers/core.py`, `piker/brokers/cli.py`,
`piker/ui/kivy/option_chain.py` — spawn-site
repoints
- `tests/test_services.py` — datad assertions + the
no-`brokerd.kraken` negative check
## Human edits
None — committed as generated.

View File

@ -0,0 +1,54 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:12:59Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171259Z_59d5d9a6_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/data/feed.py`
Generated: `open_feed()` spawn cutover with the
`maybe_spawn_datad` import done "relative-direct" from
`._daemon` — NOT via `piker.service` — to dodge a
partial-init cycle: when `piker.data.feed` loads as
part of `piker.service.__init__` executing its own
`piker.data._daemon` import, the service pkg is
mid-init and its `maybe_spawn_datad` binding does not
exist yet. The `open_feed_bus()` local-state sanity
assert flips `'brokerd' in servicename` ->
`'datad' in servicename` (the only actor-name assert
in the tree, verified by grep).
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
Generated: `_setup_persistent_brokerd()` slimmed to
console-log setup + pinned-open ctx; drops the bus
alloc, the `assert not feed._bus`, the service nursery
(backend `open_trade_dialog()` ctxs own their task
trees) and the `eg.ExceptionGroup` handler. The
`piker ledger` ad-hoc actor enters this same slimmed
fixture — exactly what it needs.
> `git log -1 -p --follow -- piker/ui/_app.py`
> `git log -1 -p --follow -- piker/ui/cli.py`
> `git log -1 -p --follow -- piker/brokers/core.py`
> `git log -1 -p --follow -- piker/brokers/cli.py`
> `git log -1 -p --follow -- piker/ui/kivy/option_chain.py`
> `git log -1 -p --follow -- tests/test_services.py`
Verification (from session): per-suite gates green at
this commit (services 5-passed incl. the new negative
assert, feeds 3-passed); a headless live smoke
(`maybe_open_pikerd` + `open_feed(['xbtusdt.kraken'])`
on an alt registry port) confirmed quotes flowing via
`datad.kraken` + `samplerd` with `check_for_service(
'brokerd.kraken') is None`. Known pre-existing flake
documented: `test_multi_fill_positions`' second
in-proc runtime boot wedges ~50% (zombie subactor w/
unread parent-IPC bytes); reproduced with the split
fully reverted so NOT a regression of this work.

View File

@ -0,0 +1,43 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:13:44Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T171344Z_eee19de0_prompt_io.raw.md
---
## Prompt
Same session-initiating `brokerd`-split instruction (see
`20260610T170859Z_75cefe10_prompt_io.md`); this is the
approved plan's final "stage 4": caps-sec slimming of
the (live, credentialed) trading actor + the `ib`
dual-daemon `client_id` collision mitigation flagged in
the plan's risk register.
## Response summary
`brokerd` loses ALL `piker.data.*` (feed) RPC mods;
spawn fails fast for datad-only backends with a "use
paper-mode" error; `ib`'s default api-gw `client_id`
gets a per-daemon-kind offset so `datad.ib` +
`brokerd.ib` don't collide on connect.
## Files changed
- `piker/brokers/_daemon.py` — `_data_mods` -> minimal
`_brokerd_service_mods`; `broker_init()` reads
`_brokerd_mods` (fallback `__enable_modules__`);
`spawn_brokerd()` fail-fast via `validate.get_eps()`
- `piker/brokers/ib/api.py` — role-based `client_id`
offset in `load_aio_clients()`
- `piker/cli/__init__.py` — resolved "expose datad"
TODO
## Human edits
None — committed as generated.

View File

@ -0,0 +1,41 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:13:44Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T171344Z_eee19de0_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); backfilled from the live dev
session transcript per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/brokers/_daemon.py`
Generated: the fail-fast originally landed in
`broker_init()` but was relocated to `spawn_brokerd()`
mid-implementation after realizing `piker ledger`
calls `broker_init()` directly even for paper accounts
on datad-only backends (would have crashed the cli);
the service-spawn path is the correct enforcement
seam. Error text:
Backend 'kucoin' offers NO `brokerd` (live
order-control) eps!? It is likely a datad-only
provider, use paper-mode for clearing instead.
(verified live via a `trio.run()` unit check.)
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
Generated: in `load_aio_clients()`, when `client_id`
is the 6116 default: `datad`-named actors offset +16
(disjoint from `brokerd`'s linear `client_id + i`
retry range), other non-`brokerd` (ad-hoc test/cli)
actors +32. Rationale from the plan's risk register:
post-split BOTH per-broker daemons connect to the same
TWS/gw endpoint; a shared default id collides and
burns up to `connect_timeout * retries` (90s) in
retry cycles.
> `git log -1 -p --follow -- piker/cli/__init__.py`

View File

@ -0,0 +1,66 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T17:33:09Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
scope: docs
substantive: true
raw_file: 20260610T173309Z_f15f8178_prompt_io.raw.md
---
## Prompt
The session-initiating instruction, verbatim (this doc
IS its "mega detailed plan" deliverable):
> ok i want you to become the distributed runtime and
> concurrency expert for this project - namely acquire
> a deep understanding of tractor and how it's used.
> then i want you to atttempt to factor our current
> brokerd service daemon into 2 daemons:
>
> - a brokerd which only hosts live/paper trading
> endoint tasks as defined namely within all
> piker/brokers/<backend>/broker.py mods
> - a new `datad` subdaemon which in a separate
> subactor serves all the data feed service tasks
> namely delivered by endpoints in
> piker/brokers/<backend>/feed.py (or similar for
> less mature backends) and as more rigorously
> defined by the validation machinery in
> piker.data.validate.
>
> give me a mega detailed plan on how to approach
> this, and a staged approach for the implementation.
Plan-shaping user decisions (in-session Q&A): sibling
topology under `pikerd`; hard cutover staged by layer;
trading-only EMS-lazy-spawned `brokerd`. The human
staged the doc copy into `ai/claude-code/plans/` and
requested this commit after the 7 implementation
commits landed.
## Response summary
The staged design doc for the `brokerd` -> (`datad` +
`brokerd`) split: context, target supervision topology,
load-bearing verified facts, per-stage file-level
changes with gates, risk register and an end-to-end
verification matrix. Produced via 3 parallel explore
agents + 1 architect agent + human Q&A before any code
was written; all 7 implementation commits in this
branch reference its stages in their provenance
entries.
## Files changed
- `ai/claude-code/plans/datad_service.md` — the design
plan doc (human-staged copy of the AI-authored plan)
## Human edits
None to the doc content — committed as generated. NB:
the implementation deviated from the plan-as-written in
4 places, see the raw file's deviation log.

View File

@ -0,0 +1,51 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T17:33:09Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T173309Z_f15f8178_prompt_io.md
---
NOTE: diff-ref mode entry; the AI output here is the
committed doc itself (a design plan, not code) so no
verbatim copy is duplicated:
> `git log -1 -p --follow -- ai/claude-code/plans/datad_service.md`
## Implementation deviation log
Where the landed 7-commit series differs from the plan
as written (recorded for reviewer transparency; the
plan doc is committed unmodified):
1. Commit ordering: the `pytest` config-dir isolation
fix lands BEFORE the `tractor`-API drift port. The
plan's stage gates assumed a runnable+isolated test
baseline; per-commit gating exposed that without
isolation the paper-EMS test reads the user's real
(polluted) `account.kraken.paper.toml` and reddens.
2. The plan's "stage 0: full pytest green" gate
required first repairing pre-existing branch
breakage vs `tractor` git `main` (boot
`AttributeError`, stale discovery/exc/position
APIs) — that repair became its own commit
("Port service+tests to latest `tractor` APIs")
rather than plan-stage work.
3. Stage-4 fail-fast placement: the plan said
`broker_init()` raises on brokerd-ep-less backends;
implementation moved the raise to `spawn_brokerd()`
since `piker ledger` calls `broker_init()` directly
even for paper accounts on datad-only backends.
4. `brokers/_daemon.py` change grouping: the
import-cleanup hunks (`exceptiongroup`, `_FeedsBus`
type-only import) landed with the caps-sec slim
commit instead of the fixture-slim commit, keeping
each intermediate tree import-clean without
sub-hunk surgery.
Also of record: the plan's "Verification" matrix was
executed as written (per-suite gates each stage, the
headless datad-feed smoke, the kucoin fail-fast unit
check); the known pre-existing ~50% second-runtime-boot
test wedge was characterized and excluded as a
regression via revert-testing.

View File

@ -0,0 +1,60 @@
---
model: claude-fable-5[1m]
service: claude
session: 32d15f9a-b2d3-4c26-bdc9-190219141a25
timestamp: 2026-06-10T21:35:49Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
scope: code
substantive: true
raw_file: 20260610T213549Z_f084e899_prompt_io.raw.md
---
## Prompt
User bug report from live `ib` testing of the
(datad|brokerd)-split branch:
> ok doing some testing and noticing live orders do not
> work otb, pikerd show this on live submission,
> [pdb traceback: `RuntimeError("Can not order
> {symbol}, no live feed?")` at `ib/api.py:1152`
> `submit_limit()` with `self._contracts == {}` in the
> `brokerd.ib` actor + a `TrioTaskExited` teardown
> cascade]
Follow-up user direction fixing the design intent:
> i think the main thing here is that [brokerd] should
> be able to always submit orders without a live feed
> being up and registered in the same subactor right?
## Response summary
Root cause: pre-split the feed code populated the
actor-local `Client._contracts` cache (via
`get_mkt_info()` writing `mkt.bs_fqme` keys) in the
SAME process as order submission; post-split the
trading actor's client is never warmed. Fix: lazily
qualify + cache the contract per order request in
`handle_order_requests()` by running the same
`get_mkt_info(fqme, proxy=...)` ep the feed side uses,
plus per-order error relay (`BrokerdError`) so one bad
submission can't crash the whole trades dialog (the
`TrioTaskExited` storm was teardown cascade from the
original raise).
## Files changed
- `piker/brokers/ib/broker.py` — thread `proxies` into
`handle_order_requests()`; lazy contract qualify on
cache-miss; guard `submit_limit()` w/ `BrokerdError`
relay; uncomment the (anticipatory) `get_mkt_info`
import
- `piker/brokers/ib/api.py` — fix the non-f-string
raise msg + document the new qualification contract
## Human edits
None — committed as generated. Live `ib` order retest
performed by the human post-commit.

View File

@ -0,0 +1,57 @@
---
model: claude-fable-5[1m]
service: claude
timestamp: 2026-06-10T21:35:49Z
git_ref: datad_service
diff_cmd: git log -1 -p --follow -- ai/prompt-io/claude/20260610T213549Z_f084e899_prompt_io.md
---
NOTE: diff-ref mode entry (code committed in the same
commit as this log); recorded from the live debug
session per the `/prompt-io` skill rules.
> `git log -1 -p --follow -- piker/brokers/ib/broker.py`
> `git log -1 -p --follow -- piker/brokers/ib/api.py`
Key diagnostic chain (from session):
- pdb showed `Client._contracts == {}` inside
`brokerd.ib`'s `submit_limit()`; the cache has
exactly TWO write sites: `Client.find_contracts()`
(api.py, keys `f'{sym}.{exch}.ib'`) and
`symbols.get_mkt_info()` (key `mkt.bs_fqme`, eg.
'nvda.nasdaq' — NO `.ib` suffix).
- `BrokerdOrder.symbol` arrives as the bs_fqme form
('nvda.nasdaq') so ONLY the `get_mkt_info()` write
site produces the key `submit_limit()` reads —
ie. pre-split it was the feed's in-proc
`get_mkt_info(sym, proxy=proxy)` call keeping orders
working, NOT `find_contracts()`.
- the existing TODO at `symbols.py:642-644` literally
predicted this: "this is going to be problematic
if/when we split out the datad vs. brokerd actors
since the mktmap lookup table will now be
inaccessible.."
- instance identity verified: `proxy._aio_ns` IS the
same `Client` obj as `_accounts2clients[account]`
(both sourced from the `load_aio_clients()` cache via
`open_client_proxies()`), so a brokerd-side
`get_mkt_info(fqme, proxy=proxies[account])` warms
exactly the dict `submit_limit()` reads. It also
populates `client._cons2mkts` which the
position-audit path (`broker.py` backup-table code)
needs in this actor anyway.
- the `TrioTaskExited` storm in the user's log
(`recv_trade_updates`, `open_aio_client_method_relay`
aio tasks) is teardown cascade: the raise crashed
`handle_order_requests` -> nursery teardown ripped
the trio sides of still-running aio relay tasks.
Hence the added per-order try/except ->
`BrokerdError` relay hardening so a single bad
submission degrades to an EMS error msg instead of
killing the backend's entire order-ctl dialog.
Verification: `tests/test_services.py` (5 passed) +
`tests/test_ems.py` (6 passed) regression-green; live
`ib` submission retest delegated to the human (needs a
running TWS/gw).

View File

@ -0,0 +1,34 @@
# AI Prompt I/O Log — claude
This directory tracks prompt inputs and model
outputs for AI-assisted development using
`claude` (claude-code CLI).
## Policy
Prompt logging follows the
[NLNet generative AI policy][nlnet-ai].
All substantive AI contributions are logged
with:
- Model name and version
- Timestamps
- The prompts that produced the output
- Unedited model output (`.raw.md` files)
[nlnet-ai]: https://nlnet.nl/foundation/policies/generativeAI/
## Usage
Entries are created by the `/prompt-io` skill
or automatically via `/commit-msg` integration.
Each commit carrying AI-generated changes links
to its provenance entry via a `Prompt-IO:`
commit-msg trailer; entries use "diff-ref mode"
(pointers into `git log -p` instead of verbatim
code copies) to avoid duplicating committed
code.
Human contributors remain accountable for all
code decisions. AI-generated content is never
presented as human-authored work.

View File

@ -0,0 +1,64 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-23T16:46:56.830897+00:00
git_ref: efa72980
session: 01a0a5b9-f872-7ca3-ab8a-d1cffc14e895
scope: code
substantive: true
raw_file: 20260923T164656Z_efa72980_prompt_io.raw.md
---
## Prompt
Investigate existing branch fixes before repairing IB day-skipping.
Create an isolated worktree on ib_connectivity_harden and add a test
that checks actual persisted data, leaving live gap repair for later.
## Response summary
Preserve IB request boundaries on HMDS no-data responses. Retain the
existing earliest-history classification and bounded reset handling.
Separate requested and published boundaries during backfill so the
first response after an empty interval does not lose its end bar.
Prove supplied bars survive both SHM publication and a fresh parquet
load while a simulated venue closure remains empty.
## Files changed
- `piker/brokers/ib/feed.py`: complete HMDS no-data requests unchanged.
- `piker/tsp/_history.py`: filter overlap against published history.
- `tests/test_ib_history.py`: six empty-response boundary cases.
- `tests/test_history_backfill.py`: two SHM/parquet integration cases.
## Human edits
The human identified the live gap, required prior-branch discovery,
selected ib_connectivity_harden as the base, and requested stronger
end-to-end evidence. They corrected overstated reproduction claims,
insisted on plain explanations, and pointed out that valid venue
closures must remain gaps. Existing data repair was explicitly deferred.
No direct human source edits have been observed; changes await review.
## Review follow-up
The human's tuicr review asked whether None was part of the backfill
protocol and requested plain explanations of the two dates and overlap
filter. Clarified the IB-internal empty-response sentinel and its
translation into NoData/DataUnavailable before reaching the backfiller.
Renamed last_start_dt to request_end_dt and published_start_dt to
oldest_shm_dt. Added a concrete empty-interval example at the filter and
explained that only successful SHM writes move oldest_shm_dt.
These changes clarify naming and documentation without changing the
previously tested request or publication behavior.
## History-filter review follow-up
The human flagged the unused append_until_dt argument and confusing
append/prepend terminology in diff_history(). The helper had one
caller, using only prepend_until_dt. Removed the helper and inlined
its inclusive lower-time cutoff beside the SHM duplicate filter.
The human's nearby TODO comment is resolved by this removal; the
replacement comment describes the retained timestamps directly.
The storage comparison remains >= backfill_until_dt; the SHM
comparison remains < oldest_shm_dt.

View File

@ -0,0 +1,87 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-23T16:46:56.830897+00:00
git_ref: efa72980
diff_cmd: git diff efa72980 -- piker/brokers/ib/feed.py piker/tsp/_history.py tests/test_ib_history.py tests/test_history_backfill.py
---
## Prompt
The human requested the IB request-boundary repair in a new worktree
based on ib_connectivity_harden and a test exercising real persistence.
The existing MNQ parquet gap is outside this repair scope.
## Code output
> `git diff efa72980 -- piker/brokers/ib/feed.py`
> `git diff efa72980 -- piker/tsp/_history.py`
> `git diff efa72980 -- tests/test_ib_history.py`
> `git diff efa72980 -- tests/test_history_backfill.py`
get_bars() now completes HMDS no-data responses without changing the
requested date. open_history_client() retains its NoData and earliest
history classification. start_backfill() tracks the published boundary
separately from the request cursor after empty intervals.
## Unedited progress output
The new test failed on the original code as intended: the
temporary parquet contained only 4 bars instead of 2,004.
The day-skip caused 2,000 available bars to be omitted.
The test also includes a simulated venue closure that must remain
a gap. It checks that we fetch available bars without inventing
data for an empty interval.
The IB fix now preserves all 2,004 bars in parquet. The SHM
assertion exposed a second bug: after stepping over an empty
interval, the backfiller drops the last bar of the next response
as though it were already published.
I’ll separate the request date from the published-data boundary
so the same test can verify that both memory and disk retain
every available bar.
## Validation
Using the existing fsp_backfill_sync/py313 interpreter, with cwd in
ib_history_request_bounds and the Piker import checked against that cwd:
- tests/test_history_backfill.py: 11 passed in 1.92s.
- tests/test_ib_history.py: 17 passed in 0.36s.
- Ruff on all four changed Python files: all checks passed.
- git diff --check: passed.
The integration test uses simulated IB bars responses, real IB history
adapter and backfill logic, real SHM, and temporary NativeDB parquet.
It checks a fresh disk load. It does not test a live gateway, actor RPC,
or Qt rendering, and does not prove the origin of the user's live gap.
## Unedited review response
`None` is internal to the IB adapter. The backfiller receives
`NoData`, or `DataUnavailable` when the requested date has reached
the known beginning of history. I’ll make that distinction explicit.
The two dates answer different questions: “where should the next
history query end?” and “what is the oldest bar already in SHM?”
An empty reply changes only the first. I’ll rename them to
`request_end_dt` and `oldest_shm_dt` and explain the filtering with
a concrete example.
> `git diff efa72980 -- piker/brokers/ib/feed.py piker/tsp/_history.py`
## History-filter review follow-up
The human flagged the unused append_until_dt argument and confusing
append/prepend terminology in diff_history(). The helper had one
caller, using only prepend_until_dt. Removed the helper and inlined
its inclusive lower-time cutoff beside the SHM duplicate filter.
The human's nearby TODO comment is resolved by this removal; the
replacement comment describes the retained timestamps directly.
The storage comparison remains >= backfill_until_dt; the SHM
comparison remains < oldest_shm_dt.

View File

@ -0,0 +1,51 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-24T16:57:57.342784+00:00
git_ref: 1da9b760
scope: code
substantive: true
raw_file: 20260924T165757Z_1da9b760_prompt_io.raw.md
---
## Prompt
Document and test the existing null-segment repair, including its
flat-bar fallback, while distinguishing packed timestamp gaps.
## Response summary
Documented SHM-only repair, provider placement assumptions, fallback
fields, notifications, and concurrency limits. Added boundary and
no-history coverage and fixed epoch-dependent comparison tolerance
which could synthesize a duplicate timestamp at the right margin.
## Files changed
- piker/tsp/_history.py: helper contract and absolute tolerance.
- tests/test_history_backfill.py: four additional parameter/case runs.
## Human edits
The human identified cancellation as a source of stored gaps,
correctly described packed indexes and insertion costs, pointed to
maybe_fill_null_segments(), and requested immediate documentation
and tests. No direct source edits observed during this change;
the patch remains uncommitted for review.
## Close-copy factoring follow-up
The human requested stronger internal types, a reusable close-copy
helper, searchable comments for copying close and synthesizing
stamps, and research into historical venue closure queries.
Extracted _copy_close_to_null_rows(), preserving non-OHLC fields and
rejecting occupied slots before mutation. Added explicit callback,
array, segment-index and date annotations. Preserved the existing
provenance TODO and suggested avoiding volume sentinels.
Four helper cases cover successful fill and unchanged rejected
inputs. Backfill suite: 19 passed in 2.01s; Ruff and diff check pass.
IB documents historical SCHEDULE requests separately from contract
hours. The installed ib_async exposes reqHistoricalScheduleAsync.
No live schedule request or venue behavior change was performed.
Source: https://interactivebrokers.github.io/tws-api/historical_bars.html

View File

@ -0,0 +1,48 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-24T16:57:57.342784+00:00
git_ref: 1da9b760
diff_cmd: git diff 1da9b760 -- piker/tsp/_history.py tests/test_history_backfill.py
---
## Prompt
The human requested documentation and tests for
maybe_fill_null_segments(), identifying its flat OHLC fallback
and asking how packed SHM could accept later recovered bars.
## Code output
> `git diff 1da9b760 -- piker/tsp/_history.py`
> `git diff 1da9b760 -- tests/test_history_backfill.py`
Document the existing slot-based repair and its limitations.
Extend fallback coverage to NoData and DataUnavailable; verify
the observed packed MNQ gap remains unchanged. Reproduce the
relative-tolerance failure at a modern epoch, then disable relative
tolerance in the synthetic timestamp boundary check.
## Validation
Before the fix: modern-epoch boundary case failed, zero-epoch
case passed. The overfull synthetic run duplicated the right bar.
After the fix: 15 backfill tests passed in 1.99s.
## Close-copy factoring follow-up
The human requested stronger internal types, a reusable close-copy
helper, searchable comments for copying close and synthesizing
stamps, and research into historical venue closure queries.
Extracted _copy_close_to_null_rows(), preserving non-OHLC fields and
rejecting occupied slots before mutation. Added explicit callback,
array, segment-index and date annotations. Preserved the existing
provenance TODO and suggested avoiding volume sentinels.
Four helper cases cover successful fill and unchanged rejected
inputs. Backfill suite: 19 passed in 2.01s; Ruff and diff check pass.
IB documents historical SCHEDULE requests separately from contract
hours. The installed ib_async exposes reqHistoricalScheduleAsync.
No live schedule request or venue behavior change was performed.
Source: https://interactivebrokers.github.io/tws-api/historical_bars.html

View File

@ -0,0 +1,91 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-24T18:54:56.893490+00:00
git_ref: 1da9b760
scope: code
substantive: true
raw_file: 20260924T185456Z_1da9b760_schedule_prompt_io.raw.md
---
## Prompt
The human requested: first freeze a commit-plan for the existing
close-copy/null-repair changes, then implement ib_async historical
schedule support with tests, use it for datad.ib history-buffer gap
spacing, and plan a common on-demand segment interface for CLI and
chart selection. Avoid reindexing in this first pass and record
fallback insertion only as a TODO.
## Response summary
Implemented historical schedule normalization, bounded startup
reservations, explicit actor-local synthetic ranges, and two read
RPC functions. Added deterministic adapter tests and real SHM/parquet
startup coverage. Wrote the segment-backfill interface plan.
This entry covers the new schedule work only. The earlier frozen
null-repair plan has its own provenance. The shared _history.py file
contains changes from both scopes until the earlier commit is made.
## Human edits
The human identified cancellation during backfill as the reproducible
trigger, explained the packed-index insertion tradeoff, requested
reuse of venue closure knowledge, and chose placeholders to avoid
expensive rewrites. They required explicit placeholder annotation,
proposed later graphics compression, and asked to defer insertion
fallbacks. They also requested the existing null-repair commit plan
before the new schedule work. These are human-directed architecture
and scope choices; no acceptance or task completion state is inferred.
## Validation and limits
55 targeted tests passed; Ruff and diff checks passed. No live IB
qualification was performed. Reservations are synthetic, not recovered
observations. Repair execution, persistence of unfinished requests,
chart compression, FSP handling, export filtering, and request-level
IB cancellation remain follow-up work documented in the plan.
## Files changed
- `piker/brokers/ib/venues.py`
- `piker/brokers/ib/api.py`
- `piker/brokers/ib/feed.py`
- `piker/tsp/_reserve.py`
- `piker/tsp/_history.py`
- `tests/providers/test_ib_schedule.py`
- `tests/providers/test_history_reservations.py`
- `docs/history_segment_backfill.rst`
## Pre-commit module review
The human requested placing historical schedule normalization in the
existing venues.py module, naming the allocation module _reserve.py,
and adding the missing project license headers. Applied those moves,
updated imports and test patch targets, and added the standard AGPL
headers to the new allocation module and both new test modules.
## Local tuicr review follow-up
The human supplied eleven comments requesting a configurable newest
gap count, caller and metadata references, explanations of guards
and provider-specific limits, multiline boolean style, capacity
fallback rationale, an explanation of numpy split boundaries, and
clearer test cases, fixture types, and module docstrings.
Added newest_gaps=32, with zero disabling queries and negative values
rejected. Documented tsdb_backfill ownership and the local/RPC range
accessors. Explained recoverable overflow and half-open synthetic
range grouping. Typed tmp_path as pathlib.Path and monkeypatch as
pytest.MonkeyPatch. Added four request-budget cases.
Verification: reservation tests 11 passed; schedule tests 7 passed;
Ruff and git diff --check passed. Changes remain unstaged for review.
No forge review or CI state was queried; this was local feedback.
## Provider-test layout review
The human requested grouping both new history/schedule test modules
under tests/providers. Moved both files without changing test logic;
they retain the shared tests/conftest.py fixtures.

View File

@ -0,0 +1,100 @@
---
model: gpt-6 (exact variant not exposed)
service: codex
timestamp: 2026-09-24T18:54:56.893490+00:00
git_ref: 1da9b760
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/brokers/ib/_schedule.py`
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_reservations.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
> `git diff HEAD~1..HEAD -- tests/test_ib_schedule.py`
> `git diff HEAD~1..HEAD -- tests/test_history_reservations.py`
> `git diff HEAD~1..HEAD -- docs/history_segment_backfill.rst`
## Generated output
Added Client.history_schedule() over the installed ib_async API,
normalizing dated sessions to UTC with explicit response coverage.
Reject ambiguous DST times, reversed sessions, and overlapping
sessions. Match the existing extended-hours bars policy.
Added startup-only schedule reservations to tsdb_backfill before
prepending older stored history. Reserve open-session sample slots
only where the reply covers the full candidate gap. Return synthetic
UTC ranges separately from OHLCV; copy preceding close into flat
placeholders without writing those placeholders to parquet.
Unknown coverage and capacity failures retain packed time gaps.
Added datad.ib schedule and reservation metadata RPC functions and
a proposed shared CLI/chart segment request protocol. The repair
queue, durable unresolved intervals, chart compression, FSP handling,
and coordinated insertion remain deferred. No live data was repaired.
Verification is complete: 55 targeted tests pass, including the
new schedule and real SHM/parquet tests. I also added checks for
reversed session bounds and the 25-hour day at the autumn DST
change.
The segment API plan uses UTC ranges for both CLI and chart
requests, a provider-owned queue, durable unfinished intervals,
and replacement of reserved slots without reindexing. Live IB
schedule coverage still needs qualification.
## Limits
Only stored-history gaps are reserved before prepend; published rows
are never shifted. Newest 32 gaps, at most 31 days each, are queried
serially with a 10-second schedule timeout. This can delay history
publication. Underlying IB request cancellation is still a TODO.
Manual whole-SHM export requires synthetic-range exclusion before
it can preserve provider-only storage. The chart currently displays
flat placeholders, and synthetic-aware FSP treatment is deferred.
Tests: history backfill 19, IB history 17, method proxy 5, schedule 7,
reservations 7. Sandbox local-socket failures were rerun with approval.
Ruff and git diff --check passed. No live broker requests were made.
## Module-layout review response
Schedule normalization now lives in piker.brokers.ib.venues beside
the existing session helpers. The allocation module is tsp._reserve.
Added the project AGPL headers to the new module and test files.
Earlier diff pointers record the original generated paths; use
the following paths for the reviewed layout:
> `git diff HEAD~1..HEAD -- piker/brokers/ib/venues.py`
> `git diff HEAD~1..HEAD -- piker/tsp/_reserve.py`
## Local review response
The fixes pass all 18 tests in the two affected files.
The new limit tests cover zero, negative, one-gap, and two-gap
budgets, including newest-first selection.
Capacity overflow remains a warning: no partial reservation is
published for that gap, and the original observations remain
available. The comments now explain that fallback explicitly.
Diff pointers above include the review edits to tsp._reserve, both
test modules, and the interface document. This review did not query
a forge or publish replies.
## Provider-test paths
Moved both new test modules under tests/providers as requested.
> `git diff HEAD~1..HEAD -- tests/providers/test_ib_schedule.py`
> `git diff HEAD~1..HEAD -- tests/providers/test_history_reservations.py`

View File

@ -0,0 +1,45 @@
---
model: gpt-6
service: codex
timestamp: 2026-09-24T23:24:05.525363+00:00
git_ref: 441358e2
scope: code
substantive: true
raw_file: 20260924T232405Z_441358e2_schedule_coverage_prompt_io.raw.md
---
## Prompt
User: "do it", approving the proposed bounded wider-window retry
for incomplete schedule coverage and its MNQ regression test.
Previous live qualification showed a one-day request began after
the gap's left edge; a wider query covered it and excluded the
one-hour venue closure.
## Response summary
Added bounded backwards widening under one timeout, preserving actual
coverage and fixed end bounds. Added captured-response allocation,
unknown-coverage, and retry-timeout tests, and updated API documentation.
39 selected tests and Ruff passed. Live data was not changed.
## Files changed
- piker/brokers/ib/api.py — bounded schedule coverage retries.
- tests/providers/test_ib_schedule.py — captured MNQ and retry regressions.
- docs/history_segment_backfill.rst — retry contract and limits.
## Human edits
The user established schedule-driven allocation, explicit unknown
coverage, and avoiding reindexing as the design constraints, supplied
the MNQ gap, and authorized this focused coverage fix. No direct human
source edits occurred during this patch; review is pending.
## Follow-up review
The user requested factoring and documenting the repeated 86400
conversion. Added piker.calc.SECONDS_PER_DAY, documented elapsed
versus calendar/session days once, and reused it in the IB query
and reservation span check. Named the client retry limit locally.
Schedule tests: 11 passed; Ruff passed after this adjustment.

View File

@ -0,0 +1,66 @@
---
model: gpt-6
service: codex
timestamp: 2026-09-24T23:24:05.525363+00:00
git_ref: 441358e2
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
> `git diff HEAD~1..HEAD -- tests/providers/test_ib_schedule.py`
> `git diff HEAD~1..HEAD -- docs/history_segment_backfill.rst`
Client.history_schedule() widens incomplete left coverage by doubling
numDays up to 31 while preserving endDateTime and actual returned
coverage. All attempts share an asyncio timeout. Missing right coverage
returns unchanged because extending the lookback cannot repair it.
Tests replay the captured MNQ schedules through the real client and
reservation allocator, check bounded incomplete coverage and retry
cancellation under one timeout. Documentation describes these limits.
## User-facing output
The captured MNQ case now passes through the real client method
and allocator: the request widens from one day to two, then
reserves 51,824 slots while leaving the venue closure compressed.
The tests also cover permanently incomplete replies and a timeout
during the retry. I’m running the related regression suites now.
All 39 selected tests passed, and Ruff passed. The reservation
suite needed a retry outside the sandbox because Trio’s local
wakeup socket was blocked.
The fix and regression tests are ready for review. I’m recording
the prompt provenance; live SHM and parquet remain unchanged.
## Validation commands and results
Worktree-local py313 in the existing Nix environment; import piker
resolved inside this worktree.
- py313/bin/python -m pytest -p no:xonsh -q -x --tb=short --no-header tests/providers/test_ib_schedule.py tests/providers/test_history_reservations.py
Sandbox: 11 passed then PermissionError creating Trio wakeup socket.
Escalated rerun: 22 passed in 1.02s; fake IB, disposable test SHM.
- py313/bin/python -m pytest -p no:xonsh -q -x --tb=short --no-header tests/test_ib_history.py
17 passed in 0.69s.
- ruff check piker/brokers/ib/api.py tests/providers/test_ib_schedule.py
All checks passed (existing deprecated configuration warning).
- git diff --check: passed.
No live service restart, live history repair, staging, or commit.
## Follow-up user review and generated changes
User: "feels like this same magic # from the prior commit should be
factored and commented once? `num_days: int = max(1, ceil(seconds /
86400))` other then that all looks decent"
> `git diff HEAD~1..HEAD -- piker/calc.py piker/tsp/_reserve.py`
Added SECONDS_PER_DAY for both consumers, with one comment explaining
fixed elapsed days versus calendar days and trading sessions. Named
the client's 31-day retry limit locally. After this change the
schedule suite passed 11 tests and Ruff passed all four Python files.

View File

@ -0,0 +1,41 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-21T21:50:55Z
git_ref: c6ec3d41
scope: docs
substantive: true
raw_file: 20260721T215055Z_c6ec3d41_prompt_io.raw.md
---
## Prompt
The user asked to boot a plan to continue work on the
time-series gap checker and bug fixing of the history
backfiller. The user authorized prompt-I/O capture for the
plan documentation after the repository audit.
## Response summary
Produced a bounded repair plan based on current source,
historical issue caches, existing profiling notes, and a
test-coverage audit. The plan prioritizes reproducible
correctness failures and storage durability before venue
classification, annotation performance, or manual repair
UX.
## Files changed
- `plans/opencode/gap-checker-backfiller-repair.md` -
phased implementation and verification plan
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260721T215055Z_c6ec3d41_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated plan has not yet been edited by the
human.

View File

@ -0,0 +1,26 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-21T21:50:55Z
git_ref: c6ec3d41
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff -- plans/opencode/gap-checker-backfiller-repair.md`
Generated a phased implementation plan for continuing
piker's gap checker and history backfiller repairs. The
plan separates data correctness from annotation
performance and orders work by dependency: deterministic
tests, canonical frame/gap normalization, structured
backfill completion, non-destructive atomic NativeDB
writes, exact null repair, provider/venue classification,
annotation vectorization, and crash qualification.
The plan is grounded in current code paths, local issue
caches `#62`, `#71`, and `#75`, and the existing 1,285-gap
profiling handoff. It records confirmed present defects,
design invariants, per-phase acceptance gates, test
commands, commit boundaries, and explicitly deferred UX
and provider work.

View File

@ -0,0 +1,46 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T00:13:57Z
git_ref: 16f3cd6f
scope: code
substantive: true
raw_file: 20260722T001357Z_16f3cd6f_prompt_io.raw.md
---
## Prompt
Continue the gap-checker and history-backfiller repair plan. Start
with deterministic analysis characterization, then fix canonical
sorting, dedupe-derived columns, positive gap detection, and null
grouping.
During review, the user challenged the replacement null-grouping
algorithm because its initial tests did not assert exact endpoints.
The user explicitly required retaining detailed, human-oriented
algorithm documentation such as the existing ASCII segment map.
## Response summary
Added deterministic analysis regressions and corrected normalization
so sorting and duplicate resolution precede derived-column
calculation. Unified null grouping for NumPy and Polars inputs,
documented its exact inclusive-boundary contract with an adapted
visual map, and verified all small null layouts against an independent
state-machine oracle. Retained sorted smart-dedupe output.
## Files changed
- `piker/tsp/_anal.py` - canonical gap and documented null grouping
- `piker/tsp/_dedupe_smart.py` - sorted duplicate resolution
- `tests/test_tsp_analysis.py` - exact and exhaustive regressions
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T001357Z_16f3cd6f_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -0,0 +1,40 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T00:13:57Z
git_ref: 16f3cd6f
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
Generated canonical gap and null-segment normalization. Derived
datetime and delta columns are recomputed after final sorting and
duplicate resolution, and negative deltas no longer become ordinary
gaps.
NumPy and Polars null grouping share one implementation while
retaining the human-oriented visual map of contiguous zero runs. The
documented contract defines absolute inclusive endpoints, margin
expansion, frame boundary clamping, contiguous-index validation, and
requested-column selection.
> `git diff HEAD~1..HEAD -- piker/tsp/_dedupe_smart.py`
Adjusted smart dedupe to honor sorted output and resolve duplicate
timestamps without retaining stale order.
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
Added deterministic synthetic OHLCV coverage for sorted dedupe, fresh
derived columns, and positive gap detection. Null grouping is checked
against readable endpoint examples and an independent state-machine
oracle over every layout through six rows, three margin widths, and
both NumPy and Polars inputs.
Verification generated with the patch:
`timeout -k 5 30 python -m pytest -q tests/test_tsp_analysis.py`
Result: 11 passed.

View File

@ -0,0 +1,36 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T00:13:58Z
git_ref: 16f3cd6f
scope: code
substantive: true
raw_file: 20260722T001358Z_16f3cd6f_prompt_io.raw.md
---
## Prompt
Continue the structured backfill phase after the analysis regressions.
Characterize provider exhaustion and genuinely empty frames, then remove
completion paths that can hang after child-task exit.
## Response summary
Removed orphan-prone completion events from the structured nursery flow
and checked empty provider frames before indexing timestamp endpoints.
Added deterministic timeout-bounded regressions for both exit paths.
## Files changed
- `piker/tsp/_history.py` - structured completion and empty-frame fixes
- `tests/test_history_backfill.py` - deterministic exit regressions
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T001358Z_16f3cd6f_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -0,0 +1,25 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T00:13:58Z
git_ref: 16f3cd6f
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Removed redundant completion events whose consumers waited only after
their owning nurseries had joined. Added empty provider-frame handling
before timestamp endpoint indexing so provider exhaustion exits cleanly.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Added bounded, actor-free regressions for `DataUnavailable` and empty
provider-frame completion.
Verification generated with the patch:
`timeout -k 5 60 python -m pytest -q tests/test_history_backfill.py`
Result: 2 passed.

View File

@ -0,0 +1,39 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T02:41:33Z
git_ref: 3a4f6737
scope: code
substantive: true
raw_file: 20260722T024133Z_3a4f6737_prompt_io.raw.md
---
## Prompt
The user challenged the analysis commit message's claim that only
positive sample-step gaps are detected and asked whether out-of-order
timestamps were simply ignored. After confirming the missing separate
reporting path, the user instructed the agent to implement the fix.
## Response summary
Separated missing-history gaps from timestamp-ordering errors without
changing the existing `dedupe()` return shape. Added a structured
detector and made normalization warn before sorting removes evidence of
duplicates or reversed timestamps.
## Files changed
- `piker/tsp/_anal.py` - ordering detector and dedupe reporting
- `piker/tsp/__init__.py` - public detector export
- `tests/test_tsp_analysis.py` - exact ordering-error regression
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T024133Z_3a4f6737_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -0,0 +1,30 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T02:41:33Z
git_ref: 3a4f6737
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/tsp/_anal.py`
Added `detect_time_ordering_errors()` to return non-positive timestamp
steps as structured rows. `dedupe()` now warns with those rows before
sorting and duplicate removal erase the original ordering evidence.
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
Exported the ordering detector through the public TSP package.
> `git diff HEAD~1..HEAD -- tests/test_tsp_analysis.py`
Changed the negative-delta regression to verify exact predecessor,
timestamp, and delta values, confirm exclusion from missing-history
gaps, and require explicit normalization reporting.
Verification generated with the patch:
- `tests/test_tsp_analysis.py`: 11 passed
- Ruff: passed
- `git diff --check`: passed

View File

@ -0,0 +1,43 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-22T02:41:34Z
git_ref: 3a4f6737
scope: code
substantive: true
raw_file: 20260722T024134Z_3a4f6737_prompt_io.raw.md
---
## Prompt
The user asked why the in-progress `piker/storage` changes were omitted
from the earlier commit plan. After receiving an inventory of unfinished
Phase 3 requirements, the user instructed the agent to complete the work.
## Response summary
Finished non-destructive, atomic NativeDB updates and integrated them
with history startup and reverse backfill. Expanded tests through two
adversarial review rounds that caught and corrected cross-client lock
blocking, stale temp indexing, discontinuous persisted indexes, startup
snapshot races, missing latest-frame persistence, notification wedges,
and mutable/truncated SHM persistence.
## Files changed
- `piker/storage/__init__.py` - incremental storage contract
- `piker/storage/nativedb.py` - validated atomic merge persistence
- `piker/storage/marketstore/__init__.py` - non-duplicate update mode
- `piker/tsp/_history.py` - provider-delta and startup persistence
- `tests/test_storage_nativedb.py` - NativeDB durability regressions
- `tests/test_history_backfill.py` - persistence lifecycle regressions
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260722T024134Z_3a4f6737_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not yet been edited by the human.

View File

@ -0,0 +1,43 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-22T02:41:34Z
git_ref: 3a4f6737
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/__init__.py piker/storage/nativedb.py piker/storage/marketstore/__init__.py`
Added an incremental storage update contract. NativeDB now validates
canonical numeric OHLCV data, serializes writers both actor-locally and
through cancellable filesystem locks, merges with incoming-wins seam
resolution, regenerates contiguous indexes, reopens and validates a
same-directory temporary parquet, fsyncs it, atomically replaces the
target, fsyncs the directory, and publishes cache only after replacement.
NativeDB indexing ignores lock and crash-left temporary files.
MarketStore uses its non-duplicate write mode for incremental updates.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Separated full provider deltas from capacity-truncated SHM pushes.
Backfill persists before SHM mutation, publishes bounded notifications,
and delays latest-frame publication until the concurrent pre-update
storage snapshot completes.
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py tests/test_history_backfill.py`
Added deterministic coverage for merge preservation, conflict policy,
atomic failures, reopened-candidate validation, cache consistency,
schema and timestamp invariants, contiguous index regeneration,
separate series, stale temp indexing, cancellable file-lock contention,
full provider-delta persistence, latest-frame ordering, and bounded
notification teardown.
Verification generated with the patch:
- deterministic analysis/backfill/storage set: 31 passed
- Ruff: passed
- `git diff --check`: passed
- final adversarial review: no findings

View File

@ -0,0 +1,40 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: unavailable
timestamp: 2026-07-27T20:47:11Z
git_ref: 84a6d47b
scope: code
substantive: true
raw_file: 20260727T204711Z_84a6d47b_prompt_io.raw.md
---
## Prompt
The user identified that `backfiller_deep_fixes` introduced persistent
`.parquet.lock` files, questioned whether Parquet or Arrow already handled
writer coordination, and clarified that piker's `tractor` structured
concurrency design already provides single-writer ownership through the
persistent `datad` feed task. The user then asked to open the backfiller
worktree and resolve the redundant locking.
## Response summary
Removed redundant actor-local and filesystem locking while preserving
atomic crash-safe Parquet replacement. Replaced the synthetic contention
test with regressions for the actual cross-branch failure: writes create
no lock sidecars, and legacy lock artifacts are ignored during indexing.
## Files changed
- `piker/storage/nativedb.py` - rely on `datad` writer ownership
- `tests/test_storage_nativedb.py` - lock-sidecar regressions
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260727T204711Z_84a6d47b_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,30 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-27T20:47:11Z
git_ref: 84a6d47b
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
Removed actor-local and filesystem writer locks from NativeDB. The
persistent `datad` feed task already owns writes for each series, while
its timeframe children write distinct files. NativeDB retains validated
temporary writes, file and directory synchronization, and atomic target
replacement for crash durability. The exact `.parquet` suffix filter is
retained so legacy lock sidecars are harmless.
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
Removed the synthetic cross-client lock-contention test. Added regression
coverage proving replacement and incremental writes create no lock
sidecars. Extended indexing coverage with the legacy
`.parquet.lock` artifact that caused upstream `flake_update` to crash.
Verification generated with the patch:
- NativeDB and history regression set: 20 passed
- Ruff: passed
- `git diff --check`: passed

View File

@ -0,0 +1,42 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-27T21:24:54Z
git_ref: 689df816
scope: code
substantive: true
raw_file: 20260727T212454Z_689df816_prompt_io.raw.md
---
## Prompt
The user reran the known IB FQME on `backfiller_deep_fixes` and reported
that `publish_latest_frame()` still failed because NativeDB required an
`index` absent from IB's provider frame. The user called out the missed
tabular schema boundary and requested an actual end-to-end test suite
rather than storage fakes.
## Response summary
Repaired the IB-provider-to-NativeDB boundary by canonicalizing durable
fields and mapping reloads into provider-specific SHM buffers. Added a
local actor/SHM/Parquet integration regression covering first startup and
restart, plus focused conversion and timestamp-coercion tests. Two review
rounds found and resolved restart hydration and fractional-time hazards.
## Files changed
- `piker/storage/nativedb.py` - canonical storage and SHM field mapping
- `piker/tsp/_anal.py` - name-based Polars-to-NumPy conversion
- `tests/test_history_backfill.py` - actor/SHM/Parquet integration test
- `tests/test_storage_nativedb.py` - schema conversion regressions
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260727T212454Z_689df816_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,34 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-27T21:24:54Z
git_ref: 689df816
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/tsp/_anal.py`
Normalized index-less provider frames to NativeDB's canonical names,
order, and dtypes before merge and publication. Provider-only columns
are excluded from durable files, derived indexes are regenerated, and
fractional timestamps are rejected before integer conversion. Added a
canonical field map for hydrating provider-specific SHM buffers while
leaving provider-only fields at their defaults. Changed Polars-to-NumPy
conversion to select columns by field name rather than position.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py tests/test_storage_nativedb.py`
Added deterministic coverage using IB's actual provider dtype and a real
local `tractor` root actor, IB-typed shared-memory buffers, NativeDB,
Parquet persistence, first-start publication, and restart hydration.
Added focused regressions for reordered/extra Polars columns and
fractional timestamp rejection.
Verification generated with the patch:
- NativeDB and history integration set: 23 passed
- Ruff: passed
- `git diff --check`: passed
- second adversarial review: no findings
- no live gateway or network tests run

View File

@ -0,0 +1,41 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-27T21:59:06Z
git_ref: 689df816
scope: code
substantive: true
raw_file: 20260727T215906Z_689df816_prompt_io.raw.md
---
## Prompt
After the actor/SHM/Parquet integration test exposed a stale SHM triplet,
the user requested leak-cleaner machinery in the pytest harness using
Tractor's existing ownership and lifetime patterns.
## Response summary
Added creator-scoped SHM leak tracking to pytest without scanning
`/dev/shm` or treating attachments as owned. The fixture cooperates with
normal Tractor teardown, verifies POSIX object identity before fallback
cleanup, restores process-local token state, and fails after cleaning a
leak. Regressions cover the exact pre-actor-registration allocation window,
positional creators, and external attachment safety.
## Files changed
- `tests/conftest.py` - creator-scoped SHM leak tracking fixture
- `tests/test_shm_cleanup.py` - ownership and failure-window regressions
- `.claude/skills/run-tests/test-harness-reference.md`
- document current-process SHM cleanup behavior
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260727T215906Z_689df816_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,25 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-27T21:59:06Z
git_ref: 689df816
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- tests/conftest.py tests/test_shm_cleanup.py .claude/skills/run-tests/test-harness-reference.md`
Added function-scoped SHM ownership tracking around Tractor's test-process
factory. The fixture records only successful creators, removes ownership
records during normal Tractor teardown, verifies POSIX object identity
before fallback unlink, restores the token-cache baseline, and fails after
cleaning any leak. Regressions reproduce allocation failure before actor
lifetime registration and prove external attachments remain untouched.
Verification generated with the patch:
- SHM cleanup regressions: 2 passed
- combined SHM, NativeDB, and history set: 25 passed
- Ruff: passed
- `git diff --check`: passed
- final adversarial review: no findings

View File

@ -0,0 +1,42 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
timestamp: 2026-07-28T00:32:47Z
git_ref: ed85721c
scope: code
substantive: true
raw_file: 20260728T003247Z_ed85721c_prompt_io.raw.md
---
## Prompt
Build a safe xonsh workflow for qualifying NativeDB and reverse-backfill
correctness against a known gappy IB chart, beginning with
`mnq.cme.20260918`. Cover baseline, fresh, restart, append, and expansion
testing through implementation, adversarial review, and verification.
## Response summary
Implemented a private disposable xonsh qualification workflow. Evidence is
tied to a fixed case, report checksum, snapshot checksum, worktree import,
and phase metadata. Added explicit path and symlink safety, source
replacement handling, xonsh child environments, and lifecycle preservation
comparisons.
## Files changed
- `snippets/nativedb_backfill_audit.xsh` - disposable qualification helpers
- `docs/manual_backfill_qualification.rst` - operator lifecycle runbook
- `tests/test_backfill_audit_snippet.py` - xonsh and comparison regressions
- `.claude/skills/run-tests/test-harness-reference.md`
- qualification test-harness guidance
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260728T003247Z_ed85721c_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,32 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-28T00:32:47Z
git_ref: ed85721c
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- snippets/nativedb_backfill_audit.xsh docs/manual_backfill_qualification.rst tests/test_backfill_audit_snippet.py .claude/skills/run-tests/test-harness-reference.md`
Added sourceable xonsh helpers that create a private disposable root, bind
child processes to the reviewed worktree, capture checksum-bound phase
evidence, safely seed and clear test history, and compare lifecycle
preservation.
The workflow rejects path traversal, pre-existing symlinks, output
collisions, mismatched case/report/snapshot evidence, and same-phase
comparison. The accompanying runbook covers known-bad replay, fresh
backfill, restart, append, gap triage, and expansion cases.
Verification generated with the patch:
- xonsh qualification helper regressions: passed
- xonsh sourceability: passed
- Ruff: passed
- `git diff --check`: passed
Adversarial review drove additional evidence binding, worktree executable,
environment propagation, source replacement, and descriptor-level safety
fixes. Residual concurrent mutation risk is stated explicitly in the private
disposable-root runbook.

View File

@ -0,0 +1,40 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
timestamp: 2026-07-28T00:50:34Z
git_ref: ed85721c
scope: code
substantive: true
raw_file: 20260728T005034Z_ed85721c_prompt_io.raw.md
---
## Prompt
Add reusable read-only tooling to qualify raw NativeDB Parquet against a
known gappy IB chart, beginning with `mnq.cme.20260918`. Preserve malformed
evidence and distinguish structural validity from unclassified positive
gaps without opening or mutating the storage runtime.
## Response summary
Implemented raw Parquet audit reporting with human and JSON output, bounded
cadence details, exact integer arithmetic, strict status, collision-safe
snapshot capture, and source path protections. Regressions cover malformed
schema and values, timestamp precision, gap arithmetic, corrupt bytes,
copy failures, symlinks, traversal, and read-only CLI behavior.
## Files changed
- `piker/storage/_audit.py` - raw Parquet audit and snapshot implementation
- `piker/storage/cli.py` - `piker store audit` command
- `tests/test_storage_audit.py` - audit and CLI regressions
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260728T005034Z_ed85721c_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,27 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-28T00:50:34Z
git_ref: ed85721c
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/_audit.py piker/storage/cli.py tests/test_storage_audit.py`
Added a raw, read-only NativeDB Parquet audit with human and JSON output,
bounded cadence evidence, exact integer arithmetic, pre-parse snapshot
capture, and strict qualification status.
The command rejects unsafe FQME paths, symlinked sources and outputs,
destination collisions, non-regular source nodes, malformed canonical
schema, non-finite values, invalid timestamps, noncanonical indexes, and
sub-period cadence. Copy-failure cleanup preserves source bytes while
completed corrupt snapshots remain available for diagnosis.
Verification generated with the patch:
- storage audit and CLI regressions: passed
- Ruff: passed
- `git diff --check`: passed
- final adversarial review: no findings

View File

@ -0,0 +1,39 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
timestamp: 2026-07-28T23:27:18Z
git_ref: ed85721c
scope: code
substantive: true
raw_file: 20260728T232718Z_ed85721c_prompt_io.raw.md
---
## Prompt
Diagnose the live `pikerd` failure where backfill publication raised
`ValueError: OHLCV timestamps must be finite and positive` while merging the
known gappy MNQ NativeDB baseline with fresh IB history. Determine whether to
commit the pending audit stack or iterate first.
## Response summary
Traced the primary failure through `publish_latest_frame()` to
`NativeStorageClient.update_ohlcv()`. The incoming frame was valid, but the
canonicalized persisted baseline retained its epoch-zero row and caused the
merged frame to fail validation. Added persisted-row repair before merge and
a regression using the observed legacy schema characteristics.
## Files changed
- `piker/storage/nativedb.py` - drop non-positive persisted rows during merge
- `tests/test_storage_nativedb.py` - live MNQ failure-shape regression
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260728T232718Z_ed85721c_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,27 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-28T23:27:18Z
git_ref: ed85721c
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py tests/test_storage_nativedb.py`
Fixed incremental NativeDB updates against legacy persisted history with
non-positive timestamps. Incoming provider frames remain strictly validated,
while merge repair discards invalid persisted rows before concat, dedupe,
ordering, canonical-index rewrite, and atomic publication.
The regression reproduces the observed MNQ baseline shape: one epoch-zero
row, noncanonical absolute indexes, extra derived columns, and a valid fresh
IB frame. It proves the invalid row is removed while all valid old and new
bars survive with canonical indexes.
Verification generated with the patch:
- NativeDB regressions: 18 passed
- history backfill regressions: 6 passed
- focused Ruff: passed
- `git diff --check`: passed

View File

@ -0,0 +1,40 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
timestamp: 2026-07-29T03:41:06Z
git_ref: 922a5df8
scope: code
substantive: true
raw_file: 20260729T034106Z_922a5df8_prompt_io.raw.md
---
## Prompt
Diagnose the live IB reverse-fill failure where
`reqHistoricalDataAsync()` raised API error 10314 because its end date,
time, or timezone was invalid.
## Response summary
Traced the failure to `Client.bars()`, where an unfinished EST conversion
left the original datetime for `ib_async` to serialize as
`YYYYMMDD HH:MM:SS UTC`. Converted aware boundaries explicitly to UTC and
passed IB's accepted `YYYYMMDD-HH:MM:SS` string while retaining a blank
latest-request boundary. Added exact request-argument regressions.
## Files changed
- `piker/brokers/ib/api.py` - explicit UTC-dash history boundary format
- `tests/test_ib_history.py` - IB request-format regressions
- `.claude/skills/run-tests/test-harness-reference.md`
- deterministic IB history test mapping
- `ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260729T034106Z_922a5df8_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,25 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-29T03:41:06Z
git_ref: 922a5df8
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py tests/test_ib_history.py .claude/skills/run-tests/test-harness-reference.md`
Fixed IB reverse-history requests to serialize aware end boundaries with
IB's explicit UTC-dash syntax, `YYYYMMDD-HH:MM:SS`. This bypasses
`ib_async`'s trailing-`UTC` representation, which the live Gateway rejected
with API error 10314, and removes the unfinished EST comparison breakpoint.
The regression captures the exact `reqHistoricalDataAsync()` argument for
latest, stdlib-aware, Pendulum UTC, and America/New_York inputs.
Verification generated with the patch:
- IB history request regressions: 4 passed
- focused Ruff: passed
- `git diff --check`: passed
- adversarial review: no findings

View File

@ -0,0 +1,40 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
session: 24fb9765-a550-4570-8350-f0fc9b7e17db
timestamp: 2026-07-29T04:17:23Z
git_ref: ce33deb6
scope: code
substantive: true
raw_file: 20260729T041723Z_ce33deb6_prompt_io.raw.md
---
## Prompt
Diagnose why the chart still showed a zero first OHLCV row after the
epoch-zero timestamp repair, causing auto-y-ranging to include zero.
## Response summary
Audited live 60s and 1s NativeDB files and found one exact 60s first-slot
sentinel at epoch 60 with a completely zero OHLCV payload. Added narrowly
scoped read and merge repair, matching incoming validation, sentinel-only
startup handling, canonical reindexing, and explicit audit classification.
Preserved modern all-zero bars as unclassified evidence rather than assuming
all zero-priced instruments are corrupt.
## Files changed
- `piker/storage/nativedb.py` - first-slot sentinel repair and validation
- `piker/storage/_audit.py` - sentinel classification and zero-row evidence
- `tests/test_storage_nativedb.py` - hydration, merge, and boundary regressions
- `tests/test_storage_audit.py` - sentinel and modern-zero audit regressions
- `ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.raw.md`
- unedited response record
- `ai/prompt-io/opencode/20260729T041723Z_ce33deb6_prompt_io.md`
- provenance metadata and response summary
## Human edits
None - generated changes have not been edited by the human.

View File

@ -0,0 +1,31 @@
---
model: gpt-5.6-sol
provider: openai
service: opencode
timestamp: 2026-07-29T04:17:23Z
git_ref: ce33deb6
diff_cmd: git diff HEAD~1..HEAD
---
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py piker/storage/_audit.py tests/test_storage_nativedb.py tests/test_storage_audit.py`
Fixed the remaining legacy first-slot sentinel observed in live MNQ 60s
history: `time=60` with zero open, high, low, close, and volume. NativeDB now
repairs the exact sentinel before read hydration and merge, reindexes the
filtered view, and persists the repaired frame on the next valid update.
The repair remains narrowly scoped. Modern all-zero OHLC bars stay valid for
spreads or synthetic instruments, null-containing malformed rows still fail
validation, new writes can not recreate the first-slot sentinel, and a file
containing only sentinels loads as no history so fresh backfill can proceed.
The raw audit reports total all-zero price rows, classifies first-slot
sentinels as structural violations, and leaves later all-zero rows visible as
unclassified warnings.
Verification generated with the patch:
- NativeDB, audit, history, xonsh, and IB regressions: 61 passed
- focused Ruff: passed
- `git diff --check`: passed
- final adversarial review: no findings

View File

@ -0,0 +1,34 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 6409e3bb-1a9a-4ada-b6c4-c554115143e9
timestamp: 2026-07-29T18:54:06Z
git_ref: d359b1cb
scope: code
substantive: true
raw_file: 20260729T185406Z_d359b1cb_prompt_io.raw.md
---
## Prompt
The user reported that QQQ 60-second zeros appeared as a flat line after
reload, clarified that the outstanding concern was the prior `ldshm`
failure, and instructed the agent to implement the more focused fix.
## Response summary
Changed `ldshm` to classify invalid SHM snapshots before downstream
analysis. Cadence is inferred only from observed finite positive steps;
buffers containing non-finite or non-positive timestamps, or lacking a
positive cadence, are skipped rather than partially replacement-written
or reloaded. Added deterministic helper and command-level regressions.
## Files changed
- `piker/storage/cli.py` - inspect cadence and reject invalid SHM snapshots before mutation.
- `tests/test_ldshm.py` - cover QQQ-like zeros, corrupt timestamps, cadence selection, degenerate buffers, and mutation prevention.
- `.claude/skills/run-tests/test-harness-reference.md` - register the deterministic regression target.
## Human edits
None - generated output remains uncommitted.

View File

@ -0,0 +1,58 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-07-29T18:54:06Z
git_ref: d359b1cb
diff_cmd: git diff HEAD~1..HEAD
---
The user clarified that the QQQ 60-second chart now displayed a flat
forward-filled segment, then asked whether the previously reported
`piker store ldshm` failure had been fixed. After being told it had not,
the user instructed: "ok but i read above you have a more focussed fix
todo now? so have at it!"
The implementation focused on the exact failure boundary. It does not
filter invalid rows into a partial replacement and does not mutate live
SHM. Instead, `ldshm` inspects one captured Polars snapshot, infers its
cadence from observed finite positive timestamp steps, and skips the
entire buffer before analysis, persistence, or reload when any timestamp
is non-finite or non-positive.
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
Generated `_shm_period_and_invalid_count()` and integrated its result in
`ldshm`. The helper selects the most frequent observed positive step,
using the smallest observed step as the deterministic tie-break. Short
or cadence-degenerate buffers are skipped. Invalid snapshots are logged
and cannot reach deduplication, NativeDB replacement, or SHM reload.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Generated deterministic regressions for the QQQ-shaped interior zero
run, observed-step cadence selection, all invalid timestamp classes,
short and degenerate buffers, and command-level prevention of dedupe,
storage writes, and reload. The command-level test uses an immutable
Polars snapshot and failing spies, without live SHM mutation or timing.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Added the deterministic `ldshm` regression target and its change-to-test
mapping to the repository test harness reference.
Verification output:
```text
.......................................................... [100%]
58 passed in 0.70s
```
`git diff --check` passed. Ruff was unavailable in the worktree's
existing environment. Three adversarial review passes found and drove
fixes for partial-replacement data loss, invented median timeframes,
mixed SHM observations, and pre-guard null-analysis failures. The final
review reported no remaining findings.
The patch intentionally does not claim that `ldshm` is safe to run as a
general concurrent repair writer. It handles invalid live snapshots by
refusing mutation rather than weakening NativeDB validation.

View File

@ -0,0 +1,37 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 253ade33-07ba-45b5-8839-be35d17ec164
timestamp: 2026-07-29T20:40:46Z
git_ref: ad6c560f
scope: code
substantive: true
raw_file: 20260729T204046Z_ad6c560f_prompt_io.raw.md
---
## Prompt
The user reported that the QQQ 1-second chart still contained a gap after
the `ldshm` fix, asked the agent to inspect it, and authorized continued
implementation after a short pause and shutdown of the live workspace.
## Response summary
Diagnosed a 653-row active RT SHM null reservation caused by inclusive IB
reverse-query overlaps and startup repair orchestration. Prevented extra
physical SHM overlap rows, serialized and bounded IB history recovery,
made blank/error responses terminate, and constrained synthetic repair to
valid interior timestamp boundaries. Added deterministic concurrency and
data-layout regressions.
## Files changed
- `piker/brokers/ib/feed.py` - bound and serialize IB history reset/retry behavior.
- `piker/tsp/_history.py` - prevent SHM overlap skew and safely finalize null repair.
- `tests/test_ib_history.py` - cover blank, cancellation, reset, and pacing interleavings.
- `tests/test_history_backfill.py` - cover overlap and null-repair invariants.
- `.claude/skills/run-tests/test-harness-reference.md` - register deterministic coverage.
## Human edits
None - generated output remains uncommitted.

View File

@ -0,0 +1,72 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-07-29T20:40:46Z
git_ref: ad6c560f
diff_cmd: git diff HEAD~1..HEAD
---
The user reported that QQQ still displayed a gap on the 1-second chart
after the focused `ldshm` fix and asked the agent to inspect it. The user
then paused and resumed implementation while shutting down the live piker
trading workspace.
Read-only live inspection found the active datad generation held 653
contiguous zero-time rows in QQQ's 1-second RT SHM, while durable Parquet
contained no zero rows. The run occupied absolute indexes 168148 through
168800 between timestamps 1785348398 and 1785349051. The stale actor
generation and active 60-second history were separately identified.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Generated history changes that exclude each inclusive reverse-query
endpoint already present in SHM while retaining the full provider frame
for NativeDB merge. Startup null repair now runs after reverse backfill,
uses bounded UI notifications, handles provider no-data without an
interactive pause, fills only exact interior zero groups, and refuses
synthetic timestamps unless both valid boundaries establish the expected
cadence.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
Generated IB history reset changes that make blank responses complete,
bound repeated cancellation and reset failures per request, serialize
timeout and pacing farm resets with one actor-global Trio lock, carry
explicit reset results, and coordinate pacing handoff with queued timeout
waiters.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Generated regressions for inclusive endpoint exclusion, full NativeDB
provider deltas, strict synthetic gap boundaries, empty-provider local
fallback, bounded notifications, and preservation of valid margin rows.
> `git diff HEAD~1..HEAD -- tests/test_ib_history.py`
Generated deterministic regressions for blank IB responses, cancellation
limits, stalled reset cancellation, pacing reset failure limits,
cross-request reset serialization, timeout-to-pacing handoff, and
successful handoff timeout restart.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Registered the deterministic history-backfill target and expanded the IB
history mapping to include `feed.py`.
Verification output:
```text
........................................................................ [ 93%]
..... [100%]
77 passed in 2.12s
```
`git diff --check` passed. Multiple adversarial review rounds identified
and drove fixes for unbounded IB empty/cancellation/reset paths, stale
reset ownership, actor-global failure poisoning, concurrent farm resets,
double-counted pacing handoffs, physical endpoint overlap, and unsafe
synthetic timestamp boundaries. Final review reported no findings.
Live validation was intentionally deferred because the user shut down the
trading workspace. A fresh datad/chart startup is required to regenerate
SHM and verify that QQQ no longer contains the 653-row null run.

View File

@ -0,0 +1,41 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-07-31T16:52:15Z
git_ref: 0846cbd4
scope: code
substantive: true
raw_file: 20260731T165215Z_0846cbd4_prompt_io.raw.md
---
## Prompt
Evaluate and update `piker store` so bare groups and endpoints immediately
show help; prepare for eventual optional fuzzy selection in `anal` and
`ldshm`; aggregate recent time-series diagnostics into CLI endpoints. The
user selected the safe-foundation scope.
## Response summary
Added immediate help UX, exact NativeDB and SHM identities, read-only
`series` and `shm` diagnostics, selector-ready exact SHM names, and robust
discovery/attachment. Fixed related `ldshm` option and no-match failures.
Deferred optional `fzf` and mutation-default changes.
## Files changed
- `piker/storage/cli.py` - help UX and read-only series/SHM diagnostics.
- `piker/storage/nativedb.py` - canonical compound series identities.
- `piker/tsp/_history.py` - exact SHM discovery and dtype resolution.
- `piker/data/_sharedmem.py` - attach-only existing-segment helper.
- `piker/tsp/__init__.py` - SHM identity helper exports.
- `tests/test_store_cli.py` - command UX and endpoint regressions.
- `tests/test_ldshm.py` - SHM diagnostics and operational regressions.
- `tests/test_storage_nativedb.py` - series parsing/index regressions.
- `docs/manual_backfill_qualification.rst` - triage command documentation.
- `.claude/skills/run-tests/test-harness-reference.md` - test mapping.
## Human edits
None - generated output remains uncommitted.

View File

@ -0,0 +1,93 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-07-31T16:52:15Z
git_ref: 0846cbd4
diff_cmd: git diff HEAD~1..HEAD
---
The user requested an evaluation and possible update of `piker store`,
including immediate help output when a group or endpoint is invoked with
no arguments. The user also proposed eventual optional `fzf` selection for
`anal` and `ldshm`, and authorized aggregating the recent time-series and
SHM debugging work into CLI endpoints. After reviewing three scopes, the
user selected the recommended safe foundation rather than implementing
interactive fuzzy selection or changing mutation defaults immediately.
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
Generated no-argument help behavior for the `store` group and every
endpoint. Added read-only `series` and `shm` commands, JSON and Rich output,
exact `--shm-name` plumbing, SHM cadence/invalid/order/gap summaries, and
safe corrupt-epoch formatting. Fixed existing `ldshm` no-write/no-reload
control flow and no-match debugger behavior.
> `git diff HEAD~1..HEAD -- piker/storage/nativedb.py`
Generated canonical `NativeSeriesRef` parsing and deterministic listing.
Changed the internal index to preserve `(fqme, period)` identities while
keeping unique-FQME compatibility listing and cardinality. Hardened
malformed, sidecar, Unicode-period, symlink, and traversal handling.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Generated exact `ShmBufferRef` parsing and stable discovery, replacing
unsafe substring globbing. Added dtype resolution that mirrors history
allocation, exact-ref attachment, disappearance handling, and optional
exact-name iteration.
> `git diff HEAD~1..HEAD -- piker/data/_sharedmem.py`
Generated attach-only SHM access which never creates a segment when a
discovered object disappears.
> `git diff HEAD~1..HEAD -- piker/tsp/__init__.py`
Exported selector-ready SHM identity, parsing, listing, and dtype helpers.
> `git diff HEAD~1..HEAD -- tests/test_store_cli.py`
Generated command-level help, exact durable-series JSON, read-only SHM JSON,
and unknown exact-name regressions.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Generated SHM identity, summary, ordering, dtype, forged-name, disabled
mutation-flag, and gap-markup regressions.
> `git diff HEAD~1..HEAD -- tests/test_storage_nativedb.py`
Generated compound index, canonical parser, malformed filename, and dotted
FQME regressions.
> `git diff HEAD~1..HEAD -- docs/manual_backfill_qualification.rst`
Documented `store series` and `store shm` as separate read-only durable and
live-memory triage layers.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Registered the deterministic storage CLI UX target and mapping.
Verification output:
```text
........................................................................ [ 71%]
............................. [100%]
101 passed in 1.72s
```
Real `piker store` and `piker store audit` invocations printed complete
help without `--help`, exiting with Click's normal no-args help status 2.
`git diff --check` passed. Ruff was unavailable in the existing worktree
environment.
Adversarial review drove fixes for provider dtype reconstruction, observed
versus expected cadence, exact SHM grammar, discovery races, Typer direct
defaults, malformed NativeDB periods, compound cardinality compatibility,
undefined `ldshm` reload payloads, no-match debugger hangs, ordering
corruption reporting, and dtype precedence. Final review found no issues.
Optional external `fzf` or built-in numbered selection remains deferred.
The exact series and SHM identities added here are the intended foundation
for that later patch.

View File

@ -0,0 +1,39 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-08-01T01:53:21Z
git_ref: 74ea2152
scope: code
substantive: true
raw_file: 20260801T015321Z_74ea2152_prompt_io.raw.md
---
## Prompt
Make `piker store ls` default to all NativeDB series as a compact counterpart
to detailed `series`, with sensible no-flag output and a visible help
description. Consolidate SHM inspection and repair under canonical `shm`,
because attachment to volatile segments is implied, and make inspection the
safe default while retaining explicit persistence and reload controls.
## Response summary
Replaced backend-oriented `ls` with compact NativeDB discovery and made both
listing commands default to all series. Consolidated SHM behavior under a
read-only-by-default `shm` command, guarded explicit repair options, updated
executable references and docs, and added command regressions.
## Files changed
- `piker/storage/cli.py` - listing defaults and unified SHM command modes.
- `tests/test_store_cli.py` - listing, help, and SHM mode regressions.
- `tests/test_ldshm.py` - explicit persistence-path regressions.
- `docs/manual_backfill_qualification.rst` - revised safe CLI workflow.
- `snippets/claude_debug_helper.py` - explicit SHM persistence commands.
- `.claude/skills/run-tests/test-harness-reference.md` - safety and test map.
- `.claude/skills/piker-slang/examples.md` - canonical command example.
## Human edits
None - generated output incorporates the user's design refinements.

View File

@ -0,0 +1,78 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-01T01:53:21Z
git_ref: 74ea2152
diff_cmd: git diff HEAD~1..HEAD
---
The user requested command-surface refinements after reviewing the new
storage CLI. Bare `ls` should default to all series and act as a compact
form of detailed `series`; commands should produce sensible default output
without requiring trigger flags; and `ls` needed a parent-help description.
The user first proposed folding read-only SHM inspection into `ldshm -r`,
then clarified that `shm` is the better canonical command because attaching
to volatile segments is implied by the resource name.
The resulting design uses `piker store ls` as a compact NativeDB FQME and
period view, while `piker store series` reports each exact sub-series with
size and path details. Both default to all records and accept an optional
FQME substring.
`piker store shm FQME` is read-only by default. Persistence and reload enter
the prior repair workflow only through explicit `--write-parquet` and
`--reload-parquet-to-shm` flags. Reload requires persistence, and read-only
formatting flags are rejected in repair mode instead of being ignored.
> `git diff HEAD~1..HEAD -- piker/storage/cli.py`
Generated shared NativeDB payload collection, compact and detailed output,
numeric period ordering, default-all semantics, the canonical `shm` command,
safe mode dispatch, explicit repair validation, and debugger propagation in
read-only mode.
> `git diff HEAD~1..HEAD -- tests/test_store_cli.py`
Generated regressions for no-flag listing, parent-help descriptions, numeric
period display, default read-only SHM JSON, reload dependencies, and repair
versus read-only option conflicts.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Adjusted persistence regressions for the renamed command and explicit repair
entry while preserving the historical failure rationale.
> `git diff HEAD~1..HEAD -- docs/manual_backfill_qualification.rst`
Documented compact versus detailed NativeDB discovery, default read-only SHM
inspection, and the explicitly mutating repair invocation.
> `git diff HEAD~1..HEAD -- snippets/claude_debug_helper.py`
Updated executable debug helper commands to select explicit SHM persistence.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Updated operational safety guidance and SHM regression mappings.
> `git diff HEAD~1..HEAD -- .claude/skills/piker-slang/examples.md`
Updated the command example to the canonical `shm` spelling.
Verification output:
```text
........................................................................ [ 69%]
................................ [100%]
104 passed in 5.77s
```
Real command-help rendering was checked for the parent group, `ls`, `series`,
and `shm`. Real read-only `ls` and `series --json` output was also exercised
against the configured NativeDB. `git diff --check` passed.
Adversarial review identified and drove fixes for lexicographic period
ordering, stale runbook safety wording, ignored mode-specific options, and an
explicit-default `--max-gaps 10` validation bypass. Final review found no
remaining issues. Real OS-backed SHM attachment and successful repair/reload
remain integration-level validation concerns.

View File

@ -0,0 +1,33 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-08-05T19:50:01Z
git_ref: f0f3c806
scope: code
substantive: true
raw_file: 20260805T195001Z_f0f3c806_prompt_io.raw.md
---
## Prompt
Inspect the completed NVDA 60s SHM backfill before workspace restart, explain
the persistent zero region, and continue fixing the indexing logic so sparse
IB history does not leave physical null reservations.
## Response summary
Captured a 42,720-row live SHM null span whose valid margins shared the same
timestamp. Replaced wall-clock row reservation with actual provider-boundary
placement, fixed stale SHM first-index, endpoint insertion and field-map
semantics, published the final storage prepend, and added a full multi-frame
`tsdb_backfill()` regression using a real `ShmArray`.
## Files changed
- `piker/tsp/_history.py` - contiguous provider and NativeDB prepends.
- `tests/test_history_backfill.py` - sparse multi-frame SHM regression.
## Human edits
None - generated output incorporates the user's live observations.

View File

@ -0,0 +1,71 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-05T19:50:01Z
git_ref: f0f3c806
diff_cmd: git diff HEAD~1..HEAD
---
The user reported that a completed NVDA 60s backfill left a large zero
region in live SHM and asked for immediate inspection before restarting the
workspace. The captured buffer contained 42,720 invalid rows at absolute
indexes `3065895..3108614`. The valid rows on both sides had the identical
timestamp `1781271360`, proving the region was an erroneous physical-index
reservation rather than a real temporal gap.
The user then asked the agent to continue fixing the backfill logic.
> `git diff HEAD~1..HEAD -- piker/tsp/_history.py`
Removed wall-clock-based SHM reservation from `tsdb_backfill()`. Reverse
provider frames now extend the visible `ShmArray._first` boundary on every
prepend, beginning exactly at the current first index after inclusive query
endpoints are removed. Added `_prepend_tsdb_history()` to trim storage rows
against the actual earliest provider timestamp and prepend them directly
adjacent only after reverse provider retrieval completes.
This preserves sparse venue history as ordinal samples rather than treating
every elapsed calendar period as a physical SHM row. It also prevents the
old `_first + 1` insertion point from overwriting the earliest published bar.
> `git diff HEAD~1..HEAD -- tests/test_history_backfill.py`
Added a full `tsdb_backfill()` regression modeled on the live NVDA failure.
It uses a real `ShmArray`, NativeDB's field map, two inclusive reverse-provider
frames, and overlapping stored history. Assertions prove the complete
timestamp sequence is contiguous and lossless, contains no non-positive
timestamps, sends complete provider frames to durable storage, and notifies
chart/FSP consumers after the final storage prepend.
Live evidence captured before workspace restart:
```text
rows 1575296 first_index 1568233 last_index 3143528
invalid_count 42720
left boundary
[(3065893, 1781271300, 204.03, 204.82)
(3065894, 1781271360, 204.81, 204.72)
(3065895, 0, 0. , 0. )]
right boundary
[(3108614, 0, 0. , 0. )
(3108615, 1781271360, 204.81, 205.44)
(3108616, 1781271420, 205.46, 205.48)]
```
Verification output:
```text
........................................................................ [ 68%]
................................. [100%]
105 passed in 4.80s
```
Adversarial review caught stale `_first` semantics and an off-by-one
insertion point in the initial patch. The first live restart then exposed an
undefined `time_key` local at the new `tsdb_backfill()` call site, which
crashed `datad.ib` and propagated to the chart as `RemoteActorError`. The
helper now derives the source timestamp field from its field map, and the
test enters the full enclosing lifecycle so the original `NameError` can not
recur unnoticed. A final review also drove explicit publication after stored
history is prepended and deterministic final-stage synchronization. Final
review found no remaining issues; another live restart remains pending.

View File

@ -0,0 +1,36 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: ses_0799212ebffe42arY96czXn89F
timestamp: 2026-08-11T20:58:26Z
git_ref: 06d5ea5d
scope: code
substantive: true
raw_file: 20260811T205826Z_06d5ea5d_prompt_io.raw.md
---
## Prompt
Diagnose and fix an IB live-order failure where lazy NVDA contract
qualification raised `tractor.trionics.Lagged` after the order reached
`brokerd.ib`. Determine whether recent history changes or cancelled symbol
searches caused the stale state.
## Response summary
Traced the failure to competing consumers of a one-slot Trio broadcast, not
the recent SHM change or stale asyncio client state. Replaced broadcast
fan-out with one response/status demultiplexer, retained request-ID
correlation, handled cancellation and teardown, documented the task/channel
ownership model, warned on diagnostic metadata eviction, and added
deterministic proxy regressions.
## Files changed
- `piker/brokers/ib/api.py` - sole-reader IB proxy response routing.
- `tests/test_ib_method_proxy.py` - status, concurrency, cancellation and EOF regressions.
- `.claude/skills/run-tests/test-harness-reference.md` - proxy test mapping.
## Human edits
None - generated output follows the user's live failure diagnosis.

View File

@ -0,0 +1,66 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-11T20:58:26Z
git_ref: 06d5ea5d
diff_cmd: git diff HEAD~1..HEAD
---
The user reported that certain IB order entries might have broken and
provided a live failure for a first NVDA sell order. EMS delivered the
request to `brokerd.ib`, but lazy `get_mkt_info('nvda.nasdaq')` contract
qualification failed in `MethodProxy._run_method()` with:
```text
tractor.trionics._broadcast.Lagged:
Task `piker.brokers.ib.broker.handle_order_requests` overrun and
dropped `0` values
```
The user suspected a cancelled symbol-search task might have left stale
asyncio state. Cancellation can leave an orphaned method response, but the
observed failure was stale Trio broadcast state. `open_client_proxy()`
broadcast a one-slot `LinkedTaskChannel` to an always-running IB status relay
and an idle method receiver. Status traffic advanced only the relay, so the
first method call for an uncached symbol entered an already-lagged receiver.
Orders for eagerly cached contracts bypassed this path.
The immediately preceding sparse-history commit did not modify IB order,
symbol, or proxy code; it exposed no direct dependency on this failure.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/api.py`
Generated a sole-reader message demultiplexer for each IB client proxy.
`relay_client_proxy_messages()` owns every asyncio-to-Trio receive, handles
status/error traffic, and dispatches `mid`-tagged responses into per-call
`MethodProxy` wait slots. Cancelled calls remove their pending slot, causing
late responses to be logged and dropped without skewing later calls. Clean
channel EOF is accepted during normal proxy teardown.
Follow-up review restored the masked reconnect TODO and per-message case
rationale, documented the one-reader/many-writer task hierarchy and separate
per-proxy channels, and added a warning with request method metadata when the
bounded unresolved-request diagnostic table evicts its oldest entry.
> `git diff HEAD~1..HEAD -- tests/test_ib_method_proxy.py`
Generated deterministic regressions for idle status traffic before first
qualification, two concurrent calls with reverse-order responses, cancelled
calls with late responses followed by a successful call, and graceful
channel EOF. Every synchronization wait is timeout-bounded.
> `git diff HEAD~1..HEAD -- .claude/skills/run-tests/test-harness-reference.md`
Registered the deterministic IB method-proxy target and source mapping.
Verification output:
```text
........................ [100%]
25 passed, 4 warnings in 3.44s
```
The warnings are existing Tractor/Trio deprecations from the EMS test.
Adversarial review drove a clean-EOF guard and bounded polling. Final review
found no actionable issues. Live IB order validation remains pending and
requires restarting `brokerd.ib`.

View File

@ -0,0 +1,34 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-13T20:24:02Z
git_ref: 12b0d66e
scope: code
substantive: true
raw_file: 20260813T202402Z_12b0d66e_prompt_io.raw.md
---
## Prompt
Continue the paired piker broadcast-consumer work after Tractor PR 490
stabilization. Stop cached feed callers from mutating or consuming shared
quote-stream roots, then review, verify and prepare a commit boundary.
## Response summary
Made provider root streams private to the cached `Feed`, gave every caller a
local subscribed view, corrected feed cache identity, and made multi-provider
relays own synchronized child subscriptions and independent send-channel
clones.
## Files changed
- `piker/data/feed.py` - caller-local feed views, complete cache keys, and
subscribed multi-provider relays with independent channel lifetimes.
- `tests/test_feed_broadcast.py` - real broadcast ownership, lifecycle and
cache-identity regressions, including sibling survival after provider EOC.
## Human edits
None - generated output follows the requested cached-feed ownership item.

View File

@ -0,0 +1,51 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-13T20:24:02Z
git_ref: 12b0d66e
diff_cmd: git diff HEAD~1..HEAD
---
The user asked to continue the paired piker broadcast-consumer work after
Tractor PR 490 stabilization. The next item was to stop cached feed callers
from mutating or consuming shared `Feed`/`Flume` quote-stream roots, then
review, verify and prepare a commit boundary.
> `git diff HEAD~1..HEAD -- piker/data/feed.py`
Changed `maybe_open_feed()` so the cached `Feed` exclusively owns provider
root streams and every caller, including the first cache owner, receives a
caller-local `Feed`/`Flume` view backed by one child subscription per provider.
The cache identity now includes normalized full symbol shape and stream policy.
Changed `Feed.open_multi_stream()` so every provider relay subscribes instead
of directly consuming a shared stream, and synchronizes relay startup before
yielding the merged channel. Each relay owns a cloned send channel so one
provider EOC cannot terminate its siblings. The module-level relay returns its
live child receiver through `TaskStatus` for typed retention and introspection.
Documented `Feed.pause()`/`Feed.resume()` as provider-context-wide controls
shared by local cache users.
> `git diff HEAD~1..HEAD -- tests/test_feed_broadcast.py`
Added real `Feed`, `Flume`, and Tractor broadcast regressions covering both
cache-miss owner and cache-hit paths, one-provider-many-market mapping,
multi-provider relay fan-out and startup, duplex controls, lexical child
closure, sibling survival after provider EOC, cached-root immutability, and
full cache-key differentiation.
Verification output:
```text
... [100%]
3 passed in 0.02s
. [100%]
1 passed in 0.01s
```
The second result is the existing EMS broadcast ownership regression. Ruff,
E501, compilation and whitespace checks passed. Repeated adversarial review
found and resolved shared `Feed.streams`, incomplete cache-key, fake-lifecycle,
relay-startup and first-owner asymmetry issues; final review reported no
findings.

View File

@ -0,0 +1,32 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-13T23:07:44Z
git_ref: ddac3553
scope: code
substantive: true
raw_file: 20260813T230744Z_ddac3553_prompt_io.raw.md
---
## Prompt
Continue the piker broadcast-consumer audit after the Deribit ownership commit
and prepare the next smallest ownership fix.
## Response summary
Replaced IB's manual raw quote-channel registry with a symbol-keyed cached
context and gave every caller a lexical child from one retained
`LinkedTaskChannel` broadcaster root.
## Files changed
- `piker/brokers/ib/feed.py` - cached raw-channel lifetime and lexical child
ownership.
- `tests/test_ib_broadcast.py` - concurrent fan-out, cache-key and child
lifetime regression.
## Human edits
None - generated output follows the requested broadcast-consumer audit.

View File

@ -0,0 +1,39 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-13T23:07:44Z
git_ref: ddac3553
diff_cmd: git diff HEAD~1..HEAD
---
The user asked to continue the piker broadcast-consumer audit after committing
the Deribit ownership fix. The next isolated item was IB's actor-local
`_quote_streams` cache, which retained a raw channel owned by its first caller
and constructed competing broadcaster roots for later callers.
> `git diff HEAD~1..HEAD -- piker/brokers/ib/feed.py`
Factored `_open_aio_quote_channel()` as the raw asyncio-channel context and
made `open_aio_quote_stream()` cache that context by symbol through
`maybe_open_context()`. Every caller now receives a lexical child from the
retained `LinkedTaskChannel` broadcaster, while the raw source remains alive
until its final cache user exits. Removed the manual `_quote_streams` registry
and teardown mutation.
> `git diff HEAD~1..HEAD -- tests/test_ib_broadcast.py`
Added a concurrent ownership regression using real Tractor broadcast state.
It proves miss and hit callers use the same symbol/context factory, receive the
same quote through distinct children, close independently, and leave the owner
live after the hit exits.
Verification output:
```text
. [100%]
1 passed in 0.01s
```
The new test passes through a source-isolated IB feed import. Test Ruff, E501,
Python compilation and whitespace checks pass. Normal package collection
remains blocked by the existing Qt environment mismatch.

View File

@ -0,0 +1,43 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-14T03:27:16Z
git_ref: 2aa66c2c
scope: code
substantive: true
raw_file: 20260814T032716Z_2aa66c2c_prompt_io.raw.md
---
## Prompt
Prototype chart-actor-owned gap overlays from the current backfiller tip,
with actor-local startup and UI toggling. Preserve remote annotation IPC and
make the gap-specific path strict and `msgspec`-typed.
## Response summary
Adds a chart-local controller and vectorized SHM detector, starts historical
gap overlays with display state, toggles the focused timeframe through the
existing input loop and exposes the same renderer through a strict Tractor
payload context. Keeps all legacy remote annotation commands and separates
local graphics from each remote context's teardown ownership.
## Files changed
- `piker/ui/_gaps.py` - typed messages, detection and local controller.
- `piker/ui/_remote_ctl.py` - IPC clients and chart-actor endpoints.
- `piker/ui/_display.py` - actor-local startup integration.
- `piker/ui/_interaction.py` - focused `Ctrl+G` toggle.
- `piker/ui/_widget.py` - controller reference.
- `piker/ui/_annotate.py` - safe FQME-aware repositioning.
- `piker/tsp/_annotate.py` - typed annotation-client reference.
- `piker/storage/cli.py` - renamed annotation client context.
- `tests/test_gap_overlays.py` - detector and schema regression.
- `tests/test_ldshm.py` - renamed annotation-client test seams.
- `plans/opencode/chart-local-gap-overlays.md` - architecture and landing plan.
## Human edits
None - the work remains an uncommitted disposable-worktree prototype for
human review and live chart qualification.

View File

@ -0,0 +1,72 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-14T03:27:16Z
git_ref: 2aa66c2c
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user asked what work had been done so far, then authorized autonomous
work while AFK. The target was to scheme and prototype an actor-local gap
annotator from the `backfiller_deep_fixes` tip without mutating existing
branches. The user then clarified twice that the annotation IPC API must
remain available and should become more rigorous and `msgspec`-typed rather
than being removed.
## Generated code
> `git diff HEAD~1..HEAD -- piker/ui/_gaps.py`
Adds tagged gap request/reply structs, render-ready gap specs, vectorized
OHLCV timestamp-gap detection and chart-local overlay ownership.
> `git diff HEAD~1..HEAD -- piker/ui/_remote_ctl.py`
Adds strict Tractor gap IPC, `AnnotClient` stream mapping/locking and
owner-scoped endpoint cleanup while retaining the legacy annotation endpoint.
> `git diff HEAD~1..HEAD -- piker/ui/_display.py`
Starts the local 60-second overlay after display-state registration.
> `git diff HEAD~1..HEAD -- piker/ui/_interaction.py`
Routes `Ctrl+G` from the focused chart's existing async input loop directly
to the local controller.
> `git diff HEAD~1..HEAD -- piker/ui/_widget.py`
Stores the chart actor's local controller reference.
> `git diff HEAD~1..HEAD -- piker/ui/_annotate.py`
Keeps redraw repositioning isolated by FQME and synchronizes arrow geometry.
> `git diff HEAD~1..HEAD -- piker/tsp/_annotate.py piker/storage/cli.py`
Renames the remote annotation control type and context manager to the
client-oriented `AnnotClient` / `open_annot_client` interface.
> `git diff HEAD~1..HEAD -- tests/test_gap_overlays.py`
Adds deterministic detector, directional payload-schema, typed error and
msgpack roundtrip regressions.
> `git diff HEAD~1..HEAD -- tests/test_ldshm.py`
Updates storage CLI test seams for the renamed annotation client context.
## Generated architecture output
> `git diff HEAD~1..HEAD -- plans/opencode/chart-local-gap-overlays.md`
Documents ownership, startup, UI, strict IPC, refresh and branch landing
semantics.
## Verification output
Source resolution points to this worktree. The targeted regression passes,
Ruff reports no changed-file findings, `compileall` succeeds, `git diff
--check` succeeds and Tractor's strict gap payload decoder initializes.

View File

@ -0,0 +1,39 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-17T22:20:36Z
git_ref: f65df6a1
scope: code
substantive: true
raw_file: 20260817T222036Z_f65df6a1_prompt_io.raw.md
---
## Prompt
Add end-to-end coverage for chart-local gap annotations in the existing
gap-overlay worktree without disturbing concurrent manual backfiller testing
in the root checkout.
## Response summary
Adds real offscreen Qt and real Tractor actor integration coverage. The tests
drive startup registration, PyQtGraph insertion/removal, focused Ctrl-G input,
history-prepend repositioning, duplicate-FQME chart identity, typed endpoint
exchange and cancellation with stale-response recovery. Defects exposed by
those tests are fixed in manager cleanup, Qt geometry notification and PyQt6
event routing.
## Files changed
- `piker/ui/_gaps.py` - correct PyQtGraph ownership and bounds behavior.
- `piker/ui/_annotate.py` - notify Qt before reposition geometry changes.
- `piker/ui/_display.py` - expose deterministic startup registration.
- `piker/ui/_interaction.py` - focus toggles and PyQt6 event registration.
- `tests/test_gap_overlays.py` - real Qt and Tractor integration coverage.
- `plans/opencode/chart-local-gap-overlays.md` - update coverage boundaries.
- `plans/opencode/chart-local-gap-overlays.summary.md` - update test totals.
## Human edits
None - the changes remain uncommitted for human review.

View File

@ -0,0 +1,59 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-17T22:20:36Z
git_ref: f65df6a1
diff_cmd: git diff f65df6a1..HEAD
---
## Prompt
The user asked whether the chart-local gap annotation feature had complete
end-to-end coverage, then requested implementation in the existing gap-overlay
worktree while manual NativeDB/backfiller qualification continued in the root
checkout. The user initially declined a tests-only Prompt-IO entry. Real Qt
and Tractor integration tests then exposed production lifecycle and PyQt6
event-routing defects, making the resulting patch substantive code work and
requiring this full provenance entry.
## Generated code
> `git diff HEAD~1..HEAD -- piker/ui/_gaps.py`
Removes manager-owned graphics through their `PlotItem`, retains owning-plot
registrations, and excludes pixel-sized gap arrows from automatic data bounds.
> `git diff HEAD~1..HEAD -- piker/ui/_annotate.py`
Moves Qt geometry-change notification ahead of rectangle and arrow mutation so
scene spatial indexing follows history-prepend repositioning.
> `git diff HEAD~1..HEAD -- piker/ui/_display.py`
Extracts `_register_gap_overlays()` as the deterministic chart startup seam
which registers display states and renders default historical gap layers.
> `git diff HEAD~1..HEAD -- piker/ui/_interaction.py`
Adds focused-chart toggle routing, updates the gap keyboard path to PyQt6 event
enums, and keeps unrelated cached chart states untouched.
> `git diff HEAD~1..HEAD -- tests/test_gap_overlays.py`
Adds offscreen real-Qt manager, startup, keyboard, cached-chart and reposition
coverage plus a real Tractor actor endpoint/client dialog with deterministic
cancellation and stale-response recovery.
## Generated architecture output
> `git diff HEAD~1..HEAD -- plans/opencode/chart-local-gap-overlays.md plans/opencode/chart-local-gap-overlays.summary.md`
Records the automated integration boundary and remaining broker-backed manual
qualification requirements.
## Verification output
The source-isolated non-actor and related suites pass 24 tests with one actor
case deselected. The isolated actor case passes with one upstream Tractor/Trio
deprecation warning. Ruff, compileall, import resolution and whitespace checks
pass. Independent review reports no remaining P1/P2 findings.

View File

@ -0,0 +1,43 @@
---
model: gpt-5.6-sol
service: opencode
session: tractor-addr-unpacking
timestamp: 2026-08-20T16:35:19Z
git_ref: 5f631e19
scope: code
substantive: true
raw_file: 20260820T163519Z_5f631e19_prompt_io.raw.md
---
## Prompt
The human asked to implement the Piker compatibility work caused by
tractor's canonical tagged-address migration, run the previously
deferred tractor verification, and keep runtime-managed WireGuard
bindspaces in a later separate PR.
## Response summary
Made Piker normalize legacy and tagged registry inputs before semantic
comparison and process-global storage. Updated runtime address typing and
test assertions for canonical output, retained legacy caller input, and
removed a duplicate pytest log-level option that blocked downstream test
collection with current tractor.
## Files changed
- `piker/service/_registry.py` - canonical registry normalization and
shared address typing.
- `piker/service/_actor_runtime.py` - tagged-compatible runtime API
annotations.
- `tests/conftest.py` - canonical fixture comparison and shared pytest
option ownership.
- `tests/test_services.py` - registry regression and canonical runtime
assertions.
## Human edits
The human explicitly requested the downstream implementation and chose
to keep WireGuard bindspace lifecycle work in a separate future tractor
PR. The agent implemented and tested those directions; no direct manual
source edits were observed.

View File

@ -0,0 +1,25 @@
---
model: gpt-5.6-sol
service: opencode
timestamp: 2026-08-20T16:35:19Z
git_ref: 5f631e19
diff_cmd: git diff HEAD~1..HEAD
---
# Raw output - tractor tagged-address compatibility
The human requested downstream Piker qualification after tractor PR
#505 switched transport address emission to canonical tagged tuples.
> `git diff HEAD~1..HEAD -- piker/service/_registry.py piker/service/_actor_runtime.py tests/conftest.py tests/test_services.py`
Normalized legacy and tagged registry addresses through tractor's
address codec before Piker stores or compares them. Extended service
runtime annotations to cover both tuple shapes, updated runtime tests to
compare canonical serialized values, and removed Piker's duplicate
registration of tractor's shared `--ll` pytest option.
Ruff passed. The focused normalization regression and Piker runtime boot
passed against the local tractor checkout at `5d92595f`. The wider
service module reached an unrelated pre-existing logger-name assertion
in `maybe_spawn_daemon()` after both address-focused tests passed.

View File

@ -0,0 +1,39 @@
---
model: gpt-5.6-sol
service: opencode
session: tractor-addr-unpacking
timestamp: 2026-08-21T04:20:58Z
git_ref: b7aca4c0
scope: code
substantive: true
raw_file: 20260821T042058Z_b7aca4c0_prompt_io.raw.md
---
## Prompt
The human approved refining Piker's logger-name handling after the
wider service test exposed a stale parent-logger assertion during the
tractor tagged-address compatibility work.
## Response summary
Changed Piker's logger helpers to remove only an exact `piker.` prefix,
preserving complete requested submodule names. Removed daemon startup
assertions that incorrectly required parent logger names and added a
regression covering both logger factories and names that the old
character-set stripping silently corrupted.
## Files changed
- `piker/log.py` - exact relative logger-name derivation.
- `piker/service/_daemon.py` - remove stale parent-name assertion.
- `piker/data/_daemon.py` - remove stale parent-name assertion.
- `piker/brokers/_daemon.py` - remove stale parent-name assertion.
- `piker/clearing/_ems.py` - remove redundant logger assertion.
- `tests/test_logging.py` - package logger namespace regressions.
## Human edits
The human explicitly approved this follow-up after reviewing the
residual Piker service-suite gap. The agent implemented and verified
the refinement; no direct manual source edits were observed.

View File

@ -0,0 +1,24 @@
---
model: gpt-5.6-sol
service: opencode
timestamp: 2026-08-21T04:20:58Z
git_ref: b7aca4c0
diff_cmd: git diff HEAD~1..HEAD
---
# Raw output - preserve Piker logger namespaces
The human approved a follow-up refinement after the full Piker service
module exposed a stale logger-name assertion while qualifying tractor
PR #505.
> `git diff HEAD~1..HEAD -- piker/log.py piker/service/_daemon.py piker/data/_daemon.py piker/brokers/_daemon.py piker/clearing/_ems.py tests/test_logging.py`
Replaced character-set `str.lstrip()` calls with exact package-prefix
removal, removed assertion-only parent-logger expectations from daemon
startup paths, and added regression coverage for both logger factories
and prefix-sensitive submodule names.
Ruff passed after removing newly unused logger assignments. All four
logger regressions and all six service integration tests passed against
the local tractor PR #505 checkout.

View File

@ -0,0 +1,46 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: unavailable
timestamp: 2026-08-26T22:26:00Z
git_ref: 5f631e19
scope: code
substantive: true
raw_file: 20260826T222600Z_5f631e19_prompt_io.raw.md
---
## Prompt
Diagnose and fix the hanging Binance USD-M perpetual feed for
`NVDAUSDT.USDTM.PERP.BINANCE` by comparing piker with Binance's current
API and changelog. Ensure unsupported or silent websocket feeds fail
loudly into `datad` and chart consumers instead of blocking startup.
## Response summary
Migrates Binance USD-M market data from the retired legacy websocket to
the split public and market routes, handles Binance's new aggregate-trade
schema, and introduces bounded connection and first-data startup. Direct
stream URLs avoid reconnect-time subscription-ACK races, while targeted
tests preserve routing and fail-loud behavior.
## Files changed
- `piker/brokers/binance/feed.py` - route, merge, normalize, and bound
Binance quote streams.
- `piker/brokers/binance/venues.py` - define current USD-M websocket
endpoints.
- `piker/data/_web_bs.py` - support bounded websocket context startup.
- `tests/test_binance.py` - cover routes, schemas, and startup failures.
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.raw.md` -
retain the generated-output record.
- `ai/prompt-io/opencode/20260826T222600Z_5f631e19_prompt_io.md` - record
structured provenance for the code changes.
## Human edits
The human explicitly required fail-loud behavior rather than merely
updating the endpoint, and supplied a direct CLI reproduction proving
both `datad.binance` and the chart caller blocked. Those requirements
drove the connection and first-data deadlines and their regressions. The
changes remain uncommitted for human review.

View File

@ -0,0 +1,57 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-26T22:26:00Z
git_ref: 5f631e19
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user reported that selecting
`NVDAUSDT.USDTM.PERP.BINANCE` connected Binance's USD-M
websocket but never loaded the feed or downstream chart. They asked
OpenCode to compare Binance's API changelog with piker's FQME and feed
code, diagnose the failure, and solve it.
The user clarified that future provider failures must raise loudly
instead of silently blocking feed and chart startup. They also confirmed
the same hang from
`piker -l info --pdb chart nvdausdt.usdtm.perp.binance`.
## Generated code
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Routes USD-M book-ticker and aggregate-trade data through Binance's new
`/public` and `/market` websocket paths, merges both sockets into the
existing quote stream, accepts the new `st` aggregate-trade field, and
requires both stream classes before declaring startup complete. It also
removes the misleading missing-provider warning, derives feed mode from
resolved pair metadata, and raises `DataUnavailable` for connection or
live-data startup timeouts.
> `git diff HEAD~1..HEAD -- piker/brokers/binance/venues.py`
Defines Binance's current public and market USD-M websocket endpoints
and makes the market endpoint the regular futures websocket API root.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Adds an optional connection deadline to `open_autorecon_ws()` so callers
can fail context entry when repeated websocket handshakes never become
connected.
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
Adds regressions for exact Binance stream routing, silent live-data
startup, silent websocket connection startup, and the current USD-M
aggregate-trade payload schema.
## Verification output
`pytest -q tests/test_binance.py` passes all four tests. Compile checks,
focused Ruff fatal-error checks, and `git diff --check` pass. Live probes
through piker's merged stream received both required stream classes and
returned normalized trade quotes for `NVDAUSDT` USD-M and `BTCUSDT`
spot.

View File

@ -0,0 +1,35 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: pid-469264
timestamp: 2026-08-28T06:31:26Z
git_ref: 1606122b
scope: code
substantive: true
raw_file: 20260828T063126Z_1606122b_prompt_io.raw.md
---
## Prompt
Review the staged IB broker patch before committing it. After the review
identified a deterministic error-parser crash, continue with the available
next steps while leaving the typed EMS worktree and Ruff change untouched.
## Response summary
Repaired bracketed IB API error parsing by splitting on the matched error
prefix. Added a deterministic regression that routes bracketed and plain
strings through the real trade-event relay and verifies their EMS error
messages.
## Files changed
- `piker/brokers/ib/broker.py` - safely extracts bracketed API error codes.
- `tests/test_ib_broker.py` - covers bracketed and plain error relay inputs.
## Human edits
The human required review before the root commit, accepted the identified
parser defect as the next work item, and explicitly authorized continuation.
The pre-existing staged IB refactor is human-authored; the agent applied only
the parser repair and focused regression described above.

View File

@ -0,0 +1,30 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-28T06:31:26Z
git_ref: fix_symcache_search
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The human asked for a review of the root checkout's staged IB patch before
committing it. The review found that bracketed API error strings call
`str.partition()` without its required separator. The human then instructed
the agent to continue with available next steps, or ask only if unsure.
## Response
> `git diff HEAD~1..HEAD -- piker/brokers/ib/broker.py`
The generated parser repair uses the recognized `[code ` or `[Errno ` prefix
to extract the numeric code without changing the surrounding event relay.
> `git diff HEAD~1..HEAD -- tests/test_ib_broker.py`
The generated regression drives bracketed and plain strings through
`deliver_trade_events()` and verifies that one `BrokerdError` reaches EMS
with the expected reason and unknown request ID.
The typed EMS worktree and the unrelated Ruff policy change were left
untouched. No commit was created.

View File

@ -0,0 +1,37 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: 3e2eb8c9-6cfb-49a5-bf1a-1c0c2c2a4ddd
timestamp: 2026-08-28T20:03:52Z
git_ref: dd676857
scope: config
substantive: true
raw_file: 20260828T200352Z_dd676857_prompt_io.raw.md
---
## Prompt
Update the xonsh version used by the repository dev shell before stacking
the Binance work, while retaining the dotrc compatibility shim as an
immediate workaround for older shells.
## Response summary
Raises the xonsh dependency floor to `0.24.2` and refreshes its lock entry.
The new release supports Python 3.13, handles Git alias completion natively,
and works in the updated Nix development shell.
## Files changed
- `pyproject.toml` - require xonsh `0.24.2` or newer.
- `uv.lock` - lock xonsh `0.24.2` wheels and source metadata.
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.raw.md` - retain
the triggering prompt, generated configuration summary, and checks.
- `ai/prompt-io/opencode/20260828T200352Z_dd676857_prompt_io.md` - record
structured provenance for the dependency update.
## Human edits
The user identified old xonsh behavior as the root cause, reported the
separate dotrc compatibility shim, and directed that the repository update
remain a distinct flake-layer change before the Binance stack continues.

View File

@ -0,0 +1,36 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-28T20:03:52Z
git_ref: dd676857
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user traced broken multiline conflict-resolution pastes to the old
xonsh release installed by the repository dev shell. A separate dotrc
agent added a compatibility shim, and the user directed that xonsh still
be updated alongside the flake work before continuing the Binance stack.
## Generated configuration
> `git diff HEAD~1..HEAD -- pyproject.toml uv.lock`
Raises the repository's xonsh requirement from `0.22.2` to `0.24.2` and
refreshes only xonsh's locked distribution metadata. The selected release
supports Python 3.13, includes native Git-config alias completion, and fixes
Nix metadata checks.
## Verification output
- `uv lock --upgrade-package xonsh` resolves 111 packages and updates only
xonsh from `0.22.4` to `0.24.2`.
- Fresh `nix develop` replaces the worktree-local xonsh installation.
- `xonsh --version` reports `0.24.2`.
- A multi-command xonsh invocation executes Git commands successfully.
- `nix flake check --no-build` passes for `x86_64-linux`.
- `git diff --check` passes.
The existing Polars `1.36.0` lock remains unchanged despite uv reporting
that its distributions are yanked for an `unpivot` regression.

View File

@ -0,0 +1,46 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
timestamp: 2026-08-29T17:55:42Z
git_ref: e09e306a
scope: code
substantive: true
raw_file: 20260829T175542Z_e09e306a_prompt_io.raw.md
---
## Prompt
Preserve the existing Binance USD-M commit above the reviewed flake/xonsh
stack, then add missing deterministic tests and correctness fixes as a
separate follow-up with a complete editor-gated commit plan.
## Response summary
Hardens Binance feed startup around its supported one-symbol contract,
direct websocket routes, current provider schemas, and first-quote
readiness. Adds deterministic Binance regressions and separates generic
reconnect behavior into its own tested commit boundary.
## Files changed
- `piker/brokers/binance/feed.py` - validate feed cardinality, route
USD-M stream classes, and declare readiness after either quote class.
- `tests/test_binance.py` - cover routes, first-quote startup, cardinality,
and documented book-ticker and aggregate-trade schemas.
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.raw.md` - retain
the triggering prompt, generated-code summary, and verification output.
- `ai/prompt-io/opencode/20260829T175542Z_e09e306a_prompt_io.md` - record
structured provenance for the follow-up.
## Human edits
The user rejected reconstructing already committed Binance work, required
preservation through cherry-pick, selected the flake/xonsh stack base, and
required the review fixes to remain a separate tested boundary. The user also
corrected the workflow to require an automatic `/commit-plan` at every stack
review gate. During Tuicr review, the user rejected trade-gated startup,
requested a feed-specific initialization error and async-generator TODO,
and required generic reconnect changes and tests to land independently.
The user then clarified that either L1 or trade data must unblock startup,
rather than preferring one stream class or waiting independently for both.

View File

@ -0,0 +1,43 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-08-29T17:55:42Z
git_ref: e09e306a
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user asked to land recent branch work in small chronological patch sets,
including missing tests. For Binance, the user required preserving the
existing committed topic patch through cherry-pick and placing newly found
correctness repairs in a separate follow-up boundary above the flake/xonsh
stack. The user also required every yielded stack layer to include a complete
`/commit-plan` with patch-specific checks.
## Generated code
> `git diff HEAD~1..HEAD -- piker/brokers/binance/feed.py`
Restricts Binance feeds to the supported one-symbol contract, renders direct
quote routes, separates L1 liveness from first-trade readiness, and derives
the contextual trade timeout from Trio's enclosing effective deadline.
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Lets structured cancellation bypass reconnect throttling and ensures handled
handshake rejections retry before accessing connection-local cancel state.
> `git diff HEAD~1..HEAD -- tests/test_binance.py`
Adds deterministic regressions for symbol cardinality, route selection,
silent startup, immediate rejection, cancellation cleanup, aggregate-trade
schema evolution, delayed trades, and finite/infinite/tiny deadlines.
## Verification output
- Targeted Binance tests: `10 passed in 0.26s`.
- Ruff passes for all three changed Python files.
- `git diff --check` passes.
- Independent final review reports no actionable findings.
- No live Binance network probe was run.

View File

@ -0,0 +1,42 @@
---
model: openai/gpt-5.6-sol
service: opencode
session: c3a1b1ba-65c6-4c8e-bd12-e596031b0eb0
timestamp: 2026-09-01T02:42:35Z
git_ref: e09e306a
scope: code
substantive: true
raw_file: 20260901T024235Z_e09e306a_prompt_io.raw.md
---
## Prompt
Address the Tuicr review by separating generic websocket reconnect
changes from Binance-specific work and testing the real reconnect task
hierarchy rather than a reimplementation.
## Response summary
Preserves structured cancellation and safe first-handshake retries in
`_web_bs`. Adds three focused regressions in a dedicated websocket test
module and packages them as an independent commit boundary.
## Files changed
- `piker/data/_web_bs.py` - restart handled handshake failures and let
cancellation bypass the reconnect throttle.
- `tests/test_websocket.py` - exercise real reconnect startup, retry,
and cancellation task paths with controlled transport boundaries.
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.raw.md`
- retain the review prompt, generated-code summary, and verification.
- `ai/prompt-io/opencode/20260901T024235Z_e09e306a_prompt_io.md` - record
structured provenance for the generic reconnect boundary.
## Human edits
The user identified that these changes were generic rather than
Binance-specific, required a separate commit and test module, requested
coverage of the real reconnect hierarchy, and corrected code prose to
refer to `trio` consistently. A follow-up review required the production
code to explain why `trio.Cancelled` bypasses the broad reconnect-error
handler and cite the regression which audits that teardown invariant.

View File

@ -0,0 +1,33 @@
---
model: openai/gpt-5.6-sol
service: opencode
timestamp: 2026-09-01T02:42:35Z
git_ref: e09e306a
diff_cmd: git diff HEAD~1..HEAD
---
## Prompt
The user requested that generic `_web_bs` reconnect fixes and tests be
separated from the Binance backend patch. The review also required tests
to exercise the real reconnect task hierarchy, cite the previous faulty
branches, and consistently name `trio` in code prose.
## Generated code
> `git diff HEAD~1..HEAD -- piker/data/_web_bs.py`
Ensures handled handshake failures immediately restart the reconnect
loop and lets structured cancellation bypass shielded error throttling.
> `git diff HEAD~1..HEAD -- tests/test_websocket.py`
Adds deterministic scheduler-real regressions for handshake startup
timeouts, immediate rejection retries, and cancellation propagation.
## Verification output
- Generic and Binance targeted tests: `9 passed in 0.06s`.
- Ruff passes for all four changed Python files.
- `git diff --check` passes.
- No live network probe was run.

Some files were not shown because too many files have changed in this diff Show More