Extend the `col_sym_key` asset-type check in `start_backfill()`
to also exclude crypto-denominated futures (where `src` is
`'crypto_currency'` and `dst` is `'future'`) from the
`without_src=True` fqme path.
Also in `.brokers.binance` backend (it being the guilty culprit in the
discovery of this bug; and why i touched styling this code),
- reformat `make_sub()` fn sig to multiline style in
`.binance.feed`.
- add backtick around `dict` in `make_sub()` docstring.
- reformat `or` conditionals to multiline style in
`.binance.feed.get_mkt_info()`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add guards for empty-array and zero-bar-diff cases in the TSP backfill
loops to avoid crashes and allow graceful loop termination.
In `maybe_fill_null_segments()`,
- add `array.size == 0` guard in `maybe_fill_null_segments()` to detect
valid (venue closure) gaps from the backend; add a warning + bp
+ break for this case.
- add TODO that we should likely be filling nulls with the close price
for the gap's duration.
In `start_backfill()`,
- expand the "0 bars after diff" warning msg with
`backfill_until_dt` and `end_dt_param` context.
- mask the `await tractor.pause()` and add a `break` to avoid blocking
the backfill loop.
(this commit msg was generated in some part by
[`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Shorten SHM buffer key names to fit Darwin's 31
char filename limit by hashing the `fqme`/content
portion with `md5` and truncating to 8 hex chars.
Deats,
- `.fsp._api`: replace `piker.{actor}[{uuid}].{sym}`
format with `{uuid[:8]}_{hash}.fsp`
- `.tsp._history`: add `platform.system()` check to
conditionally shorten `.hist`/`.rt` keys on Darwin
while keeping the full key format on Linux
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
NOTE that this patch was orig by @dnks and broken out from a larger
commit which added unnecessary/out-of-scope changes we didn't end up
requiring.
Add `loglevel` param propagation across the data feed and sampling
subsystems to enable proper console log setup in downstream (distibuted)
subactor tasks. This ensures sampler and history-mgmt tasks receive the
same loglevel as their parent `.data.feed` tasks.
Deats,
- add `loglevel: str|None` param to `register_with_sampler()`,
`maybe_open_samplerd()`, and `open_sample_stream()`.
- pass `loglevel` through to `get_console_log()` in
`register_with_sampler()` with fallback to actor `loglevel`.
- use `partial()` in `allocate_persistent_feed()` to pass
`loglevel` to `manage_history()` at task-start.
- add `loglevel` param to `manage_history()` with default
`'warning'` and pass through to `open_sample_stream()` from there.
- capture `loglevel` var in `brokers.cli.search()` and pass to
`symbol_search()` call.
Also,
- drop blank lines in fn sigs for consistency with piker style.
- add debug bp in `open_feed()` when `loglevel != 'info'`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Change most sub-modules to use `get_logger(name=__name__)` for
per-leaf-module `log` instances vs previous subpkg-level/shared refs.
Primary changes,
- import `get_[console_]logger()` from top-level `piker.log` across leaf
mods.
- change any `<subsys>._util.log` logger-instances as well (though this
approach should no longer be used since it masks the endpoint module's
emissions.
Also,
- add a defaulted `loglevel: str` param to all `open_trade_dialog()`
endpoints, anticipating it being passed in by `.clearing`-engine.
- call `get_console_log(level=loglevel, name=__name__)` in each trade
dialog ep to enable per-`brokerd`-backend console writing.
- drop `get_logger` from `.brokers.__all__` exports
- fix type annotations: `str|None` vs `str | None`
- add TODOs for,
* comments in `._util` about multi-subsys logging
* `.accounting.__init__` about console log setup
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add frame-gap detection when `frame_last_dt < end_dt_param` to
warn about potential venue closures or missing data during the
backfill loop in `start_backfill()`.
Deats,
- add `frame_last_dt < end_dt_param` check after frame recv
- log warnings with EST-converted timestamps for clarity
- add `await tractor.pause()` for REPL-investigation on gaps
- add TODO comment about venue closure hour checking
- capture `_until_was_none` walrus var for null-check clarity
- add `last_time` assertion for `time[-1] == next_end_dt`
- rename `_daterr` to `nodata` with `_nodata` capture
Also,
- import `pendulum.timezone` and create `est` tz instance
- change `get_logger()` import from `.data._util` to `.log`
- add parens around `(next_prepend_index - ln) < 0` check
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add timeout-gated wait for `feed_is_live: trio.Event` before passing shm
tokens to `open_sample_stream()`; skip registering shm-buffers with the
sampler if the feed doesn't "go live" within a new timeout.
The main motivation here is to avoid the sampler incrementing shm-array
bufs when the mkt-venue is closed so that a trailing "same price"
line/bars isn't updated/rendered in the chart's view when unnecessary.
Deats,
- add `wait_for_live_timeout: float = 0.5` param to `manage_history()`
- warn-log the fqme when timeout triggers
- add error log for invalid `frame_start_dt` comparisons to
`maybe_fill_null_segments()`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Enable the previously commented-out dedupe-and-write logic in
`start_backfill()` to ensure tsdb stays clean of duplicate
entries.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Prevent `ValueError` from negative prepend index in
`start_backfill()` by checking buffer space before push
attempts. Truncate incoming frame if needed and stop gracefully
when buffer full.
Also,
- add pre-push capacity check with frame truncation logic
- stop backfill when `next_prepend_index <= 0`
- log warnings for capacity exceeded and buffer-full conditions
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
For now by REPLing them and raising an RTE inside `.ib.feed` as well as
tracing any such cases that make it (from other providers) up to the
`.tsp._history` layer during null-segment backfilling.
Namely insertion writes which over-fill the shm buffer past the latest
tsdb sample via `.tsp._history.shm_push_in_between()`.
Deats,
- check earliest `to_push` timestamp and enter pause point if it's
earlier then the tsdb's `backfill_until_dt` stamp.
- requires actually passing the `backfill_until_dt: datetime` thru,
* `get_null_segs()`
* `maybe_fill_null_segments()`
* `shm_push_in_between()` (obvi XD)
The `await feed_is_live.wait()` is more or less pointless and would only
cause slower startup afaig (as-far-as-i-grok) so i'm masking it here.
This also removes the final `strict_exception_groups=False` use from the
non-tests code base, flipping to the `tractor.trionics` collapser once
and for all!
Gud Boi
Tyler Goodlet