First half of the `brokerd` split: a new per-provider
data-feed-only daemon-actor `datad.<broker>` to (soon) host
all `validate._eps['datad']` eps (live quotes, history
loading, symbology search) leaving `brokerd` for live order
ctl only. Purely additive; nothing routes through it yet.
Deats,
- new `piker.data._daemon` mod mirroring the
`.brokers._daemon` conventions (and the `samplerd`
sub-daemon precedent):
- `_setup_persistent_datad()` lifetime fixture owning the
actor-global `_FeedsBus` alloc.
- `datad_init()` building `enable_modules` from the
backend's `_datad_mods` (falling back to
`__enable_modules__` for not-yet-split backends) and
copying `_spawn_kwargs` (critical for `ib`'s
`infect_asyncio`).
- `spawn_datad()`/`maybe_spawn_datad()` wrapping
`Services` + `maybe_spawn_daemon()`.
- add `piker.data._daemon` to `_root_modules` so `pikerd`
can run `spawn_datad()` requests.
- re-export the spawn eps from `piker.service`.
- add `test_datad_spawn` verifying actor boot + service
registration via `ensure_service('datad.kraken')`.
Note the `Services`-based impl style deliberately mirrors
`spawn_brokerd()` so the eventual `tractor.hilevel`
`ServiceMngr` port (see the `service_mng_to_tractor`
branch's d8c21d44 prep work) lands symmetrically on both.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T171142Z_119d2c04_prompt_io.md
Move bare `raise` outside the `if lock.owner` guard so the error
propagates regardless of whether the stale-lock branch runs. Also add
a blank-line separator in the crash log msg.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Keep the shipped `conf.toml` defaults TCP-only for this `tractor`
pin, with an explicit single-transport note and commented UDS
alternatives. Preserve repeated Click `--maddrs` options and treat
a direct string as one endpoint.
Deats,
- Validate daemon bind requests against actual listeners, not
registrar discovery endpoints; normalize tagged addresses and
allow OS-assigned TCP ports. Explain registrar ownership and
automatic client binding at the validation site.
- Resolve explicit or inherited registrar addresses independently
of other daemon kwargs, consuming overrides before forwarding
them and keeping discovery and startup on the same registrar.
- Default chart binding to `None` for absent or empty network
config, and remove the unconditional Binance debugger pause.
- Defer the prematurely cherry-picked `test_datad_spawn` until
its service API lands. Add offline endpoint/startup regressions
and a real distinct-registrar/daemon lifecycle check.
Include the human-directed config and registrar/bind explanations
and the direct prompt provenance pair from the review fix pass.
Prompt-IO: ai/prompt-io/20261007T165453Z_bde2457b_prompt_io.md
(this patch was generated in some part by `opencode` using
`gpt-6.1-sol` (`openai`))
Use `str.removeprefix()` instead of character-set stripping so
subloggers such as `piker.paper` retain their requested namespace.
Also,
- drop stale parent-logger assertions from daemon startup paths
- keep logger setup calls without assigning assertion-only locals
- cover both logger factories and prefix-sensitive module names
Prompt-IO: ai/prompt-io/opencode/20260821T042058Z_b7aca4c0_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Canonicalize legacy socket pairs and tagged tuples before storing
or comparing `Registry.addrs`, keeping existing config readable
when tractor emits protocol-tagged forms.
Also,
- broaden registry/runtime addr annotations for both wire shapes
- compare service fixture addrs through tractor's address codec
- use tractor's shared `--ll` pytest option instead of
registering it twice
- cover mixed legacy/tagged registry state and `pikerd` startup
Prompt-IO: ai/prompt-io/opencode/20260820T163519Z_5f631e19_prompt_io.md
(this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))
Continue the `repair_tests`-branch mission (already merged
in this stack's ancestry, see f4c4f1e2 which ported
`conftest.py`) by fixing the remaining drift breakage vs.
`tractor` git `main`; without these NOTHING boots since the
`tractor.Address` port in 604e5fcf.
Deats,
- normalize reg addrs via `wrap_address()` in
`open_pikerd()` before `.unwrap()`-ing; entries may be
raw `tuple`s when passed in from (test) client code.
- port `check_for_service()` to `query_actor(regaddr=)`
(was `arbiter_sockaddr=`) incl. its 2-tuple yield and
the now-required `open_registry(addrs=)` arg.
- `wait_for_actor(registry_addr=)` + `.chan.raddr.unwrap()`
raw-tuple compares in `test_runtime_boot` and
`ensure_service()`.
- update `run_test_w_cancel_method()` for modern `tractor`
cancel semantics: self-requested sub-service cancels are
absorbed (no `ContextCancelled` raised to the opener) and
single-exc groups collapse to a bare KBI.
- `RemoteActorError.boxed_type` (was `.type`) and
`Position.cumsize` (was `.size`) renames in tests.
- bump the paper-EMS startup budget 9 -> 19s; it includes
a live (kraken) symbology fetch so needs net headroom.
- woops, add the missing comma in `.deribit.api`'s
`tractor.trionics` import tuple..
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompt-IO: ai/prompt-io/claude/20260610T171022Z_4485f2b9_prompt_io.md
Thread a `tpt_bind_addrs` param through
`open_piker_runtime()` and `open_pikerd()` so
pikerd's bind addrs can differ from the registry
endpoint (support a dedicated `regd` service).
Simplify `load_trans_eps()` to delegate entirely
to `tractor.discovery.parse_endpoints()`.
Deats,
- `conf.toml`: fix maddr prefixes to proper `/ip4/` and `/unix/`, add
`chart` endpoints section, add commented `regd` example.
- `cli/__init__.py`: replace `parse_maddr` with `parse_endpoints`,
rename `--maddr` -> `--maddrs`, parse `regd` key from eps falling back
to `pikerd` addrs.
- `_actor_runtime.py`: thread `tpt_bind_addrs` through runtime open fns
to `open_root_actor`.
- `ui/cli.py`: activate `network` config parsing for chart CLI, extract
`chart` eps for bind and `regd`/`pikerd` for registry, pass
`tpt_bind_addrs` through runtime config.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Replace manual `layers['ipv4']['addr']` / `layers['tcp']['port']` tuple
extraction with direct `tractor.Address` objects returned from
`load_trans_eps()`. In `open_pikerd()` call `addr.unwrap()` for the
raw-tuple comparison against `root_actor.accept_addrs`.
Deats,
- `conf.toml`: update maddr prefix `/ipv4/` ->
`/ipv/`, add commented UDS socket path example.
- `cli/__init__.py`: wrap endpoint loading in
`maybe_open_crash_handler`, append `addr`
objects directly to `regaddrs`.
- `ui/cli.py`: restructure `chart()` body into
`maybe_open_crash_handler` scope, switch to
`registry_addrs` from config, comment out the
`network`-based `load_trans_eps` path (WIP
`multiaddr` transition).
- `_actor_runtime.py`: use `addr.unwrap()` for
accept-addr membership check.
- `uv.lock`: add `multiaddr >= 0.2.0` and its
transitive deps.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Since `tractor`'s new and improved inter-actor cancellation semantics
are much more pedantic, AND bc we use the `ServiceMngr` for spawning
service actors on-demand, the caller of `maybe_spawn_daemon()` should
NEVER conduct a so called "out of band" `Actor`-runtime cancel request
since this is precisely the job of our `ServiceMngr` XD
Add a super in depth note explaining the underlying issue and adding
a todo list of how we should prolly augment `tractor` to make such cases
easier to grok and fix in the future!
Port internal `tractor._<mod>` references to their
new public or reorganized paths after `tractor`
refactored its subpkg layout.
Deats,
- `tractor._portal.Portal` -> `tractor.Portal`.
- `tractor._supervise.ActorNursery` -> `tractor.ActorNursery`.
- `tractor._multiaddr` -> `tractor.discovery._multiaddr`.
- `tractor._addr` -> `tractor.discovery._addr`.
- `tractor._state._runtime_vars` -> `tractor.runtime._state._runtime_vars`.
- `tractor._state.is_debug_mode()` -> `tractor.runtime._state.is_debug_mode()`.
Files touched: `brokers/data.py`, `cli/__init__.py`, `data/feed.py`,
`service/_actor_runtime.py`, `service/_mngr.py`, `storage/cli.py`,
`tsp/_annotate.py`, `ui/kivy/monitor.py`, `ui/kivy/option_chain.py`.
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Pull actor-runtime `registry_addrs` from (the new)
`tractor.get_runtime_vars()` (over the previous hardcoding of
`('127.0.0.1', 6116)`..)) so that underlying `find_service()` and
`maybe_open_pikerd()` calls use the local actor's assigned registrar
endpoints.
Note, this is particularly necessary to get the `pytest` harness workin
again alongside any local running `pikerd` instance(s).
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Call `reg_err_types()` for every piker-defined
exception so they can be marshalled and re-raised
across actor boundaries.
Deats,
- `brokers/_util.py`: auto-register `BrokerError` +
all `__subclasses__()` (6 types).
- `config.py`: `ConfigurationError` +
`__subclasses__()` (`NoSignature`).
- `data/validate.py`: `FeedInitializationError`.
- `service/_ahab.py`: `DockerNotStarted`,
`ApplicationLogError`.
- `service/marketstore.py`: `MarketStoreError`.
- `storage/__init__.py`: `TimeseriesNotFound`,
`StorageConnectionError`.
- `brokers/kraken/api.py`: `InvalidKey`.
- `brokers/kraken/broker.py`: `TooFastEdit`.
- `brokers/questrade.py`: `QuestradeError`.
Also,
- uncomment `execution_venue` field on kraken `Pair`.
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Change all `.service` sub-modules to use `get_logger(name=__name__)`
for per-submod instances vs a shared `._util.log`.
Deats,
- import `get_logger()` and `get_console_log()` from top-level
`piker.log` instead of `._util` for all.
- drop `log` and `get_console_log()` partial from `._util`.
- add `name=subsys` kwarg to `get_console_log()` call in
`_actor_runtime.maybe_open_pikerd()`.
- add `name='piker.service'` to `get_console_log()` in
`_ahab.open_ahabd()`.
- change default `loglevel` from `None` to `'cancel'` in
`_ahab.open_ahabd()`.
- add sanity check: `assert log.name == 'piker.service'` in
`_daemon.maybe_spawn_daemon()`.
- change `print()` -> `log.info()` in `_registry.find_service()`.
- drop stray `from piker.service._util import log` import in
`brokers._daemon.spawn_brokerd()`.
Styling/cleanups,
- drop blank lines from various fn sigs.
- do more sin-ws union type annots.
- add more multiline style to `or` expressions in `_actor_runtime` and
`_registry`.
- update `._util` docstring with TODO about `import`-time console
log setup.
- add TODO comments in `_registry` about UDS registry support.
- use `.aid.uid` from actor in `_registry.open_registry()`.
- add intermediate var `reg_addrs` in `_registry.open_registry()` (bc
i was tracing rtvs value issues in `tractor`).
- add `pformat` import to `.elastic` (code path is currently
not used but figured might as well appease the linter..)
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Change most sub-modules to use `get_logger(name=__name__)` for
per-leaf-module `log` instances vs previous subpkg-level/shared refs.
Primary changes,
- import `get_[console_]logger()` from top-level `piker.log` across leaf
mods.
- change any `<subsys>._util.log` logger-instances as well (though this
approach should no longer be used since it masks the endpoint module's
emissions.
Also,
- add a defaulted `loglevel: str` param to all `open_trade_dialog()`
endpoints, anticipating it being passed in by `.clearing`-engine.
- call `get_console_log(level=loglevel, name=__name__)` in each trade
dialog ep to enable per-`brokerd`-backend console writing.
- drop `get_logger` from `.brokers.__all__` exports
- fix type annotations: `str|None` vs `str | None`
- add TODOs for,
* comments in `._util` about multi-subsys logging
* `.accounting.__init__` about console log setup
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Update logging helpers to use new `tractor.log` API with `pkg_name=`
kwarg and add optional `tractor` "root logger" enabling.
Deats,
- change `piker.log.get_logger()` to use `pkg_name=` vs `_root_name=`.
- add `**tractor_log_kwargs` passthrough to both wrapper fns.
- add `with_tractor_log: bool` toggle to `.get_console_log()`.
- strip `'piker.'` prefix from logger names when present to avoid
newly added `tractor.get_logger()` warnings.
Surroundingly,
- add `subsys` import to `.clearing._ems` for log name
- update all `get_console_log()` calls to use `level=` kwarg
- add assertion checks for logger names in `_setup_persistent_emsd()`
Additionally,,
- fix all type annotations: `str|None` vs `str | None`.
- add multiline style to conditional in `.cli.services()`.
- drop unused `Optional` import from `._actor_runtime`.
- drop a few "blank lines" in various function sigs.
Warning: this patch will require an equivalent dev-commit at the time of
writing in `tractor` itself, for now the `piker_pin` branch should be
sufficient to avoid breakage 🙏!
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
k
Namely changes for the `registry_addrs: list`, enable_transports: list`
and related `tractor._addr` primitive requirements.
Other updates include,
- passing `maybe_enable_greenback=True`,
- additional exc logging around `pikerd` syncing/booting,
- changing to newer `Context.wait_for_result()`,
- dropping (unnecessary?) `maybe_open_crash_handler()` around `pikerd` ep.
Using `tractor.trionics.collapse_eg()` as needed to avoid, at the least,
crash-worthy (in debug-mode REPL-ing terms) nested cancellation egs that
exhibit on SIGINT/ctl-c of each "app" (chart & daemon).
Also a bit of renaming of all `trio.Nursery`s to `tn`, the new "task
nursery" shorthand-var-name being used in all our other `tractor`
related projects.
Since service daemon actors may be cancelled remotely by clients (who
maybe also requested said daemon-actor's spawn in the first place) we
specifically ignore `tractor.ContextCancelled`s from the `ctx.wait()`
inside `Services.start_service_task()` to avoid crashing the service
mngr, and thus for now `pikerd`, (which **does** happen now due to
updated and more explicit remote cancellation semantics implemented in
`tractor`) since the `.canceller` field is not going to match the
`pikerd` uid in such cases!
This explicit check makes sense since the `Services` mngr is built to
allow remote requests to "spawn-n-supervise service actors" where the
services can remain persistent but also cancelled later as requested. We
may want to consider only allowing cancellation by actors who requested
spawn in the future tho?
Also change to more explicit imports to `tractor` types for annots
throughout the sub-pkg.
Since certain actors (even if client-like) will want to augment their
module set to provide remote features (such as our new rc annotation
msg-prot for `Qt`-chart actors) we need to ensure we merge in any input
`enable_modules: list[str]` to the value passed to the underlying
`tractor` spawning api. Previously we were passing `_root_modules` as
this value by name, but now instead we simply `list.extend()` that into
whatever is in the `kwargs` both in `open_piker_runtime()` and
`open_pikerd()`.
Since it probably IS sane to just assume a root-actor-as-registrar
listening on the localhost as a default, AND allows NOT expecting every
caller of `open_piker_runtime()` to not have to pass an addr set XD
This makes a bucha CLI shit work again after breakage due to no
default..
When a new (actor) caller opens the registry there are 2 possible cases:
1. - some task already opened the registry during init and set the global
superset of registrar addrs that are expected to be used,
2. - some task after the init task opens with a subset of addrs.
3. - some task after init opens with a disjoint set - should be an error?
In the 2nd case we don't want to error since the may just not need to
know about other registrar (multi-homed) addrs and thus only needs
specific access - so only warn about the diff in that case. If the
caller is requesting some disjoint set then we still runtime raise.
Adjust `find_service()` to allow a null `registry_addrs` input in which
case we fail over to using whatever pre-set the `Registry.addrs` has;
makes it simple for actors that don't want/need to know about the global
registrar set for their actor tree. Also, always set pass
`tractor.find_actor(only_first=True)` (for now).
This commit requires an equivalent commit in `tractor` which adds
multi-homed transport server support to the runtime and thus the ability
ability to listen on multiple (embedded protocol) addrs / networks as
well as exposing registry actors similarly. Multiple bind addresses can
now be (bare bones) specified either in the `conf.toml:[network]`
section, or passed on the `pikerd` CLI.
This patch specifically requires the ability to pass a `registry_addrs:
list[tuple]` into `tractor.open_root_actor()` as well as adjusts all
internal runtime routines to do the same, mostly inside the `.service`
pkg.
Further details include:
- adding a new `.service._multiaddr` parser module (which will likely be
moved into `tractor`'s core) which supports loading lib2p2 style
"multiaddresses" both from the `conf.toml` and the `pikerd` CLI as
per,
- reworking the `pikerd` cmd to accept a new `--maddr`/`-m` param that
accepts multiaddresses.
- adjust the actor-registry subsys to support multi-homing by also
accepting a list of addrs to its top level API eps.
- various internal name changes to reflect the multi-address interface
changes throughout.
- non-working CLI tweaks to `piker chart` (ui-client cmds) to begin
accepting maddrs.
- dropping all elasticsearch and marketstore flags / usage from `pikerd`
for now since we're planning to drop mkts and elasticsearch will be an
optional dep in the future.
To kick off our (tsdb) storage backends this adds our first implementing
a new `Storage(Protocol)` client interface. Going foward, the top level
`.storage` pkg-module will now expose backend agnostic APIs and helpers
whilst specific backend implementations will adhere to that middle-ware
layer.
Deats:
- add `.storage.marketstore.Storage` as the first client implementation,
moving all needed (import) dependencies out from
`.service.marketstore` as well as `.ohlc_key_map` and `get_client()`.
- move root `conf.toml` loading from `.data.history` into
`.storage.__init__.open_storage_client()` which now takes in a `name:
str` and does all the work of loading the correct backend module, its
config, and determining if a service-instance can be contacted and
a client loaded; in the case where this fails we raise a new
`StorageConnectionError`.
- add a new `.storage.get_storagemod()` just like we have for brokers.
- make `open_storage_client()` also return the backend module such that
the history-data layer can make backend specific calls as needed (eg.
ohlc_key_map).
- fall back to a basic non-tsdb backfill when `open_storage_client()`
raises the new connection error.
The plan is to offer multiple tsdb and other storage backends (for
a variety of use cases) and expose them similarly to how we do for
broker and data providers B)
Using the new `._ahab.start_ahab_service()` mngr of course, and now
support user config overrides (such that our defaults can be modified by
a keen user, say using a config file, or for testing). This is where the
functionality moved out of the `pikerd` init has been moved - instead of
being triggered by bool flag inputs to that factory.
For marketstore actually support overriding the entire yaml config via
runtime `_yaml_config_str: str` formatting with any passed user dict,
primarily focussing on supporting override of the sockaddrs for testing.
Allows for using the `Services.cancel_service()` api for explicit
cancellation in tests and eventually for remote teardown. Change
`.start_ahab()` to an `@acm` `start_ahab_service()` and just yield back
the same values we were returning prior. Also fix the logging (level) to
actually reflect what's passed in - we weren't using the correct name
/ instance from the `.sevice` subpkg..
Mostly renaming from the old acronym. This also contains necessary
conf.toml loading in order to call `open_storage_client()` which now
does not have default network contact info.
Since it's a bit weird having service specific implementation details
inside the general service `._daemon` mod, and since i'd mentioned
trying this re-org; let's do it B)
Requires enabling the new mod in both `pikerd` and `brokerd` and
obviously a bit more runtime-loading of the service modules in the
`brokerd` service eps to avoid import cycles.
Also moved `_setup_persistent_brokerd()` into the new mod since the
naming would place it there even though the implementation really
wouldn't (longer run) since we want to split up `.data.feed` layer
backend-invoked eps into a separate actor eventually from the "actual"
`brokerd` which will be the actor running **only** the trade control eps
(eg. trades_dialogue()` and friends).
If you want a sub-actor to write console logs (with the right level) the
`get_console_log()` call has to be made somewhere during service task
startup. Previously this wasn't well formalized nor used (depending on
daemon) so passing `loglevel` to the service's root-task-endpoint (eg.
`_setup_persistent_brokerd()`) encourages that the daemon's logging is
configured during init according to the spawner's requesting logging
config. The previous `get_console_log()` call happening inside
`maybe_spawn_daemon()` wasn't actually doing anything in the target
daemon XD, so obviously remove that and instead passthrough loglevel
to the ctx endpoints and service manager methods.
This broke non-disti-mode actor tree spawn / runtime, seemingly because
the cli entrypoint for a `piker chart` also sends these values down
through the call stack independently? Pretty sure we don't need to send
the `enable_modules` from the chart actor anyway.
Needed to move the startup sequence inside the `try:` block to guarantee
we always do the (now shielded) `.cancel()` call if we get a cancel
during startup.
Also, support an optional `started_afunc` field in the config if
backends want to just provide a one-off blocking async func to sync
container startup. Add a `drop_root_perms: bool` to allow persisting
sudo perms for testing or dyanmic container spawning purposes.