41 KiB

Raw Blame History

Typed EMS and Brokerd Messaging

October 2026 implementation baseline

This revision carries the August 28 design into the September Piker baseline (eed58566). Preserve IB execution-ID accounting (5eb4fc9d) and socket-disconnect propagation (efa72980) while changing payloads. The installed Tractor is 5c98ab1f; qualification must distinguish that April revision from local main (5cb600a8) and the September context-fault development stack. Do not test against a dirty upstream checkout without recording its changes.

The complete contract includes local client queues, EMS startup and service readiness, both order-control streams, quote normalization for dark/paper execution, and typed internal trigger/dialog records. Provider-native APIs and persistent ledger formats retain their own normalization boundaries. Runtime handles are not wire messages.

Use explicit Struct validation before local state mutation or sending; annotations and Struct construction do not validate field values. Receive decoding proves wire shape, not the sender’s Python type: a correctly tagged mapping has the same encoding as a Struct.

Endpoint-wide unions need separate startup, receive-direction, and final-result checks. Validate RPC arguments explicitly until Tractor issue #514 supplies signature-derived Start contracts. Never admit Any or raw diagnostic mappings into the clearing wire schema.

Frozen snapshots must use replacement rather than the inherited Struct.copy(update=...), which mutates its source. Keep mutable EMS dialog state separate from published messages and retain execution IDs for fill deduplication. Position messages remain explicit summaries; they must never become synthetic entries in durable trade accounting.

Port each boundary’s producers and consumers together before enabling its strict receive set. Qualify schema errors, wrong-direction and wrong-phase messages, startup completion, cancel-before-ack, reordered and repeated fills, transport failure, and bounded teardown with offline provider fixtures and real local actors. Live order submission is outside this qualification.

Qualified dependency baseline and first review boundary

The first boundary introduces ._schema, local validation in ._codec, directional streams and lifecycle checks in ._protocol, and a pure ._state.OrderState reducer. Existing endpoints keep their original ._messages imports until their paired migration. This is infrastructure qualification, not a claim that paper or any live adapter is ported.

The installed baseline is Python 3.13.14, msgspec 0.19.0, Trio 0.32.0, and Tractor 5c98ab1fb6e2cb7781aa75845c7f323a52853c46. Its existing payload APIs support this boundary without a dependency update. All eight actor contract tests also pass against clean local Tractor main at 5cb600a84b6e9fcac68bad04a487b83640b3e5fb.

The first set has 65 schema/state/phase tests and eight real IPC tests. These tests cover protocol fixtures and the new reducer; adapter-level conformance, feed-driven paper clearing, abrupt transport EOF, and other spawn backends remain qualification work for subsequent ports.

That local main adds --ll to Tractor’s pytest plugin; Piker’s current tests/conftest.py adds the same option. The ordinary downstream invocation therefore fails before collection. The isolated API check uses --noconftest -p tractor._testing.pytest on the same test file. Resolve that harness collision before upgrading Piker’s dependency pin. Do not conflate this isolated API qualification with a whole-suite pass.

Relevant unmerged Tractor work remains a separate qualification input: pld_dec_refinements (6bd685ad), auto_codecs (45508252), wkt/context_fault_status (9547c2f9), and wkt/ipc_fault_regressions (fe9eb738). The context-fault worktree has uncommitted transport-error handling changes; it is not a clean test baseline. The IPC-fault branch includes 192411c4 for transport EOF propagation. Typed Start arguments remain upstream work; validate them explicitly when migrating the actual clearing contexts.

Purpose

Replace EMS’s dictionary-shaped actor traffic with explicit, tagged msgspec.Struct protocols and modern Tractor payload enforcement on both actor boundaries:

order client <-> emsd <-> brokerd.<backend> / paperboi.<backend>

The broker boundary is part of the formal protocol, not an implementation detail behind the EMS endpoint. Paper, Binance, Kraken, and IB must all emit the same startup type, consume the same request union, and produce the same event union before strict Tractor payload limits are enabled.

This plan deliberately normalizes schemas and backend behavior before adding pld_spec. Enabling transport validation first would only turn existing status drift, startup-shape drift, and raw dictionary reconstruction into hard actor failures without first defining the intended behavior.

Scope

This work includes:

typed client-to-EMS requests and EMS-to-client events;
typed EMS-to-broker requests and broker-to-EMS events;
one typed broker startup payload shared by live and paper endpoints;
one typed EMS startup payload;
stable tagged unions suitable for native msgspec decoding;
normalized status, identifier, startup, and metadata semantics;
per-direction Context.pld_rx.limit_plds() enforcement;
backend-neutral conformance tests for paper, Binance, Kraken, and IB;
deterministic actor-free tests and real Tractor actor/IPC tests;
bounded handling of acknowledgement, status, cancellation, and teardown races;
removal of raw dictionary dispatch after all producers have migrated.

This work does not include:

typing provider-native websocket or REST payloads inside each backend;
changing the datad/brokerd actor split;
spawning brokerd from any location other than open_brokerd_dialog();
redesigning accounting or replacing BrokerdPosition with the full accounting Position type in the same patch series;
live order submission in CI;
preserving undocumented dictionary IPC between actors running different Piker revisions. The actor tree is treated as a same-checkout deployment.

Current Contract Inventory

Client and EMS boundary

Location	Startup payload	Stream received	Stream sent	Current issue
`_emsd_main()`	tuple of positions map, accounts list, dialogs map	dictionaries reconstructed as `Order` or `Cancel`	`Status` and `BrokerdPosition`, often decoded as dictionaries by clients	no payload spec, tuple startup, raw matching, loose nested fields
`open_ems()` / `OrderClient`	destructures tuple	`Status` and `BrokerdPosition` dictionaries	annotated as `Order \| dict`, but implementation assumes struct attributes	public shape and wire shape are conflated

EMS and broker boundary

Endpoint	Startup payload	Stream received by backend	Stream sent by backend	Current issue
paper `open_trade_dialog()`	`(list[BrokerdPosition], ['paper'])`	raw `BrokerdOrder` / `BrokerdCancel` dictionaries	ack, status, fill, error, position structs	raw dispatch and network-dependent simulator setup
Binance `open_trade_dialog()`	`'paper'` sentinel or `(positions, accounts)`	raw request dictionaries	structs plus client-side `Status` for existing orders	mixed protocol layers and `fill` status drift
Kraken `open_trade_dialog()`	`'paper'` sentinel or `(positions, accounts)`	raw request dictionaries	structs plus client-side `Status` for existing orders	`filled` status drift and race workarounds
IB `open_trade_dialog()`	`(positions, tuple(accounts))`	raw request dictionaries	structs plus client-side `Status` for existing orders	status can race ack and internal event types leak into sequencing

The shared broker boundary already exists operationally: all four endpoints are long-lived Tractor contexts opened by open_brokerd_dialog(). The upgrade formalizes that de facto interface instead of allowing each backend to define its own wire dialect.

Target Protocol

Tagging and decoding policy

Every union member must have an explicit stable tag. Use one common tag field, called type, with explicit protocol values such as order, cancel, broker_ack, status, broker_fill, broker_error, position, and open_order. Broker requests have distinct broker_order and broker_cancel tags so decoding cannot confuse the two directions. Do not rely on Python class names as wire tags.

All newly formalized wire structs should use:

tag='<stable value>' and tag_field='type';
frozen=True for wire snapshots;
kw_only=True where it improves constructor safety;
immutable tuple defaults; factories for mutable internal state;
finite Literal values for protocol enums;
forbid_unknown_fields=True from schema introduction; adopt the new schema only when both endpoints can meet it.

The tag is the dispatch discriminator. Fields such as action, status, and resp retain only domain meaning and are not overloaded as message type tags. Mutable EMS dialog bookkeeping must use a separate internal state type. Do not make a wire struct mutable merely because current code mutates and re-sends the same Status or Order instance.

Define shared scalar aliases and use them consistently:

RequestId = int | str
FillId = str

Client-to-EMS requests

Define:

EmsRequest = Order | Cancel

Normalize the request structs as follows:

Order.action is exactly buy | sell | alert.
Order.exec_mode is exactly dark | live.
Order.oid, symbol, and account remain required strings.
Order.price remains Decimal; tests must prove transport round-tripping.
Order.size remains signed float for this migration. Resolving the duplicated side/action representation is a later domain change.
Cancel has its own tag; action='cancel' can be removed once no local UI matching depends on it.
Actor IPC accepts structs only. Any user-facing convenience constructor for mappings must validate and construct a struct before entering the stream.

EMS-to-client events

Define:

EmsEvent = Status | BrokerdPosition

Retain BrokerdPosition as the wire-level position summary for this series, but document it as shared by the broker and client event protocols. A later accounting patch may rename it to PositionSummary without coupling that change to IPC enforcement.

Normalize Status so that:

resp is exactly pending | open | dark_open | triggered | fill | closed | canceled | error;
req is a required Order, never an unvalidated nested dictionary; an uncorrelated backend error must not fabricate a client dialog;
brokerd_request is BrokerdRequest | None and records the command that caused the status transition;
brokerd_event is BrokerdEvent | None and records the backend response;
the current mixed-purpose brokerd_msg field is removed after callers have migrated; neither replacement accepts an untyped dictionary;
an error is represented by Status(resp='error', ...) or by a distinct tagged client event, not the current incomplete Error(Status) subclass;
internal mutable dialog state is not the same object sent repeatedly over the wire. Build a status snapshot when broadcasting so prior history and multi-client delivery cannot be changed by later in-place mutation.

EMS-to-broker requests

Define:

BrokerdRequest = BrokerdOrder | BrokerdCancel

Normalize both variants so that:

they have explicit broker_order and broker_cancel protocol tags independent of action;
oid always identifies the EMS dialog;
BrokerdOrder.reqid is RequestId | None because allocation policy differs by backend; BrokerdCancel.reqid is required, and EMS retains cancel intent until acknowledgement supplies that identity;
account and symbol validation failures produce a typed BrokerdError rather than logging and continuing silently;
updates remain BrokerdOrder with an existing reqid; a separate modify variant is deferred unless conformance tests prove the current distinction ambiguous.

Broker-to-EMS events

Define:

BrokerdEvent = (
    BrokerdOrderAck
    | BrokerdStatus
    | BrokerdFill
    | BrokerdError
    | BrokerdPosition
    | BrokerdOpenOrder
    | BrokerdOpenOrdersComplete
)

BrokerdOpenOrder replaces the current backend-to-EMS use of client-facing Status. It carries the provider request id, account, source backend, and a fully typed embedded Order. EMS alone translates this event into a client Status and registers it in the dialog book.

Require oid on acknowledgements. Status and fill events identify the order by (account, reqid) within their broker session; EMS retains that correlation across edits and terminal events. For an external order, the adapter creates a collision-safe stable oid from broker, account, and provider reqid before emitting BrokerdOpenOrder. Its embedded Order.oid carries that value. EMS registers it without allocating a reverse-message identity. Service errors may omit oid; a typed rejected request can identify a submission which never received a provider ID.

BrokerdFill has a required fill_id: FillId. Use the provider execution id or provider event sequence. The four current implementations must use Binance trade id, Kraken transaction id, IB execution id, and a distinct paper-ledger transaction id respectively. Price/size/time are not a safe deduplication key because two legitimate executions may share them. A future provider with no execution identity must define an adapter session-id plus monotonic sequence and document that deduplication does not survive brokerd restart; it may not silently synthesize identity from equal-valued fill fields.

Use one broker status vocabulary:

BrokerdOrderState = Literal[
    'pending',
    'open',
    'canceled',
    'closed',
]

Fills are events, not status values. A partial execution emits a BrokerdFill and may then emit BrokerdStatus(status='open', ...). A complete execution emits the final BrokerdFill followed by BrokerdStatus(status='closed', remaining=0). Binance fill, Kraken filled, and IB provider status names must be translated inside their backend adapters and must never cross the brokerd IPC boundary.

Startup payloads

Replace every tuple and string sentinel with tagged startup structs.

BrokerdStartup contains:

mode: Literal['live', 'paper', 'paper_fallback'];
broker: str;
accounts: tuple[str, ...];
positions: tuple[BrokerdPosition, ...].

A live endpoint without credentials returns BrokerdStartup(mode='paper_fallback', broker=..., accounts=(), positions=()). open_brokerd_dialog() then opens the paper endpoint, which returns the same type with mode='paper', its paper account, and persisted positions. The fallback value is a typed delegation result, not an active broker snapshot: it is exempt from account/position and open-order-completion requirements, and its context returns after startup. This removes the raw 'paper' union branch without hiding fallback behavior.

Existing open orders remain BrokerdOpenOrder stream events because some providers discover them only after authenticated stream subscription. The contract must state that startup positions/accounts are a complete snapshot, while open-order synchronization completes through typed stream events. Every active live or paper backend emits exactly one BrokerdOpenOrdersComplete after the initial set, including when it found zero orders, so EMS can distinguish completion from delay and can order initial client readiness deterministically. A paper_fallback delegation does not open a stream and does not emit this event; the subsequently opened paper endpoint does.

EmsStartup contains:

broker: str;
positions: tuple[BrokerdPosition, ...];
accounts: tuple[str, ...];
dialogs: tuple[Status, ...].

open_ems() may continue returning the existing nested lookup maps to callers, but it builds them locally from EmsStartup. The public convenience shape is therefore separated from the wire contract.

Treat BrokerdStartup as the brokerd equivalent of datad’s FeedInit, but do not reproduce validate_backend()’s long-lived dictionary compatibility branches. Add a broker initialization validator beside the clearing protocol that checks backend name, unique/non-empty accounts, position account membership, position.broker == startup.broker, unique (broker, account, symbol) position keys, and mode-specific requirements. Give malformed startup its own registered initialization error so failures retain a useful remote exception type before strict Tractor decoding takes over. Each account name must be non-empty; the accounts collection may be empty only for paper_fallback. Paper sessions announce exactly ('paper',). EMS readiness also requires at least one account.

Replace open_brokerd_dialog()’s manual probing with one broker order-control endpoint resolver. It may use get_eps(mod, 'brokerd') as input, but it must select exactly one of open_trade_dialog or the temporary legacy trades_dialogue alias and must not confuse unrelated norm_trade support with a trading dialog. The runtime validator proves startup; the conformance suite proves request/event behavior. Neither mechanism should import or enable datad modules inside the credentialed broker actor.

Payload aliases and Tractor limits

The endpoint-wide payload aliases include startup, stream traffic, and the final None result:

EmsPayload = EmsStartup | EmsRequest | EmsEvent | None
BrokerdPayload = BrokerdStartup | BrokerdRequest | BrokerdEvent | None

Apply broad endpoint specs and narrow receive-side specs as follows:

Actor side	Endpoint `pld_spec`	Narrow receive spec
emsd `_emsd_main()`	`EmsPayload`	`EmsRequest` around the client command loop
order client	inherited from endpoint	`EmsEvent` around trade event receives
backend/paper `open_trade_dialog()`	`BrokerdPayload`	`BrokerdRequest` around backend request loops
emsd broker relay	inherited from endpoint	`BrokerdEvent` around broker event receives

The endpoint payload spec supplies the broad decoder used for Started, stream, and return payloads. ctx.started(..., validate_pld_spec=True) can also perform local started-value validation. Ordinary MsgStream.send() does not validate locally: malformed stream data raises tractor.MsgTypeError when the peer receives/decodes it. ctx.pld_rx.limit_plds(spec=...) narrows that receiving direction so a valid message from the wrong direction still fails there.

Do not add either decorator spec until the corresponding boundary has no remaining legacy sender. The final enforcement must follow the proven pattern in remote_gap_overlays() and Tractor’s own payload-limiting tests.

Provider metadata

Provider-native diagnostics remain inside adapters and their logs. The clearing wire schema has no raw-data extension point or Any field. Source backend, broker timestamp, symbol, account, order ID, request ID, and execution ID have explicit typed fields. Errors can carry a typed request. Position summaries contain no mutable accounting history.

Ordering and State Semantics

Typed messages alone do not repair ordering defects. Define these invariants before enforcement:

A backend sends exactly one BrokerdOrderAck for each accepted new order.
The ack establishes oid <-> reqid before status/fill events are exposed to EMS whenever the provider API permits that ordering.
A backend adapter that receives provider events before it can send the ack buffers them by reqid and flushes them after ack.
EMS keeps a bounded defensive pre-ack buffer because actor scheduling can still expose ordering bugs. The first reducer bounds event count and fails loudly on overflow without dropping retained executions. Add wall-clock expiry in its asynchronous owner when porting EMS; the pure reducer does not claim an age bound.
A cancel received before ack marks the dialog cancel_called. After replaying buffered events, the first ack sends one typed BrokerdCancel if the dialog is still active.
Duplicate ack, fill, and terminal status events are idempotent or rejected deterministically. Fill deduplication uses fill_id. They never resurrect a closed dialog.
closed and canceled are terminal. A late fill may be recorded and reported according to an explicit test, but it cannot recreate mutable active state.
Existing external orders enter through BrokerdOpenOrder with the backend’s stable namespaced oid and are registered before broadcast.
Initial open-order discovery is complete only after BrokerdOpenOrdersComplete; client readiness cannot infer completion from an empty or temporarily quiet stream.
Order status is routed by dialog and FQME. Account authorization and position privacy remain a separately scoped subscription design because _emsd_main() currently receives no account authorization input. Tests in this series document the current position broadcast and must not claim unauthorized-account isolation.

Implementation Phases

Phase 0: Characterize the Existing Boundary

Add tests that record current intended behavior before restructuring:

new order, update, cancel, ack, open, partial fill, close, and position flow;
cancel-before-ack;
status-before-ack;
existing provider order discovery;
bad account, zero size, and backend submission error as typed domain errors;
bad action/tag/field type as protocol decode failures, not BrokerdError;
startup with live credentials, no credentials, paper account, persisted positions, and no positions;
Binance startup containing a negative/short position;
IB startup containing at least two accounts with distinct contracts;
zero, one, and multiple existing orders followed by the explicit snapshot completion event;
disconnect while requests or events are in flight.

Use fake streams and fake provider clients; do not require public market data or credentials. Capture known undesirable outcomes as explicit expected failures only when a same-phase fix would obscure the baseline.

Gate: every behavior changed later has a deterministic assertion describing the old behavior and the target behavior.

Phase 1: Define and Audit the Wire Schemas

Introduce the strict schema in piker/clearing/_schema.py alongside the legacy ._messages imports. Qualify it independently before moving each producer/consumer pair together. Remove the legacy module after every endpoint and public consumer has migrated; do not add dictionary bridges.

Replace mutable literal defaults with factories. Resolve the accidental tuple default on BrokerdStatus.account. Remove or complete the unusable Error.from_status() path. Add explicit startup types and BrokerdOpenOrder. Add the broker initialization error/validator without a legacy dictionary fallback.

Inventory every current mutation of Order, Status, and embedded requests. Introduce a mutable internal dialog-state record and construct frozen wire snapshots from it. Replace mutation-based Struct.copy(update=...) usage with new-instance replacement that is proven to work for frozen structs.

Add actor-free serialization tests using the same msgpack semantics as Tractor:

each concrete type round-trips with exact type equality;
each union decodes to the correct concrete type by tag;
nested Order, broker event, position, Decimal, and optional identifier fields round-trip;
unknown tag, missing required field, wrong literal, wrong primitive type, malformed nested struct, and duplicate/ambiguous tag fail;
unknown fields fail from the first use of the new schema;
request and event aliases do not accept each other’s members.

Gate: all protocol unions have unique tags and can be decoded without manual dictionary inspection.

Phase 2: Build a Backend Conformance Harness

Create reusable conformance helpers that exercise a backend request handler with a memory-backed stream and fake provider client. The harness supplies the same typed request cases and validates only typed events. Every request and emitted event must be encoded and decoded through its declared union before assertion; isinstance checks alone are insufficient because direct msgspec.Struct construction does not validate annotated field types.

The shared suite must verify:

startup is one BrokerdStartup, including typed fallback-to-paper;
malformed startup fails the shared broker initialization validator;
new order produces one correlated ack;
update preserves oid and reqid;
cancel uses the correlated request id;
invalid account/size produces BrokerdError without killing the dialog;
invalid action/tag/type is rejected by codec or Tractor before the handler;
provider statuses normalize to the four broker states;
partial and complete fills use BrokerdFill plus normalized statuses;
emitted positions and existing orders are typed;
every active live/paper startup emits one open-order completion event, even for zero orders; paper_fallback delegates without one;
duplicate fills are rejected by fill_id, while distinct equal-valued fills remain valid;
provider-native dictionaries never cross the test stream;
cancellation closes all handler tasks and leaves no blocked sender;
one malformed request fails at the protocol boundary once limits are enabled.

Separate backend-neutral contract assertions from provider translation fixtures. Provider fixtures should be the smallest recorded websocket/API payload required to prove normalization. Include regressions for Binance’s tuple-shaped error reason, string fill size, short startup position, and float-vs-Decimal open-order price; Kraken’s filled status and open-order price; and IB multi-account startup association.

Gate: the harness can fail a deliberately nonconforming fake backend for each contract rule.

Phase 3: Port the Paper Engine as the Reference Backend

Make paper open_trade_dialog() the reference implementation of BrokerdPayload:

return BrokerdStartup;
consume BrokerdRequest instances directly;
emit only BrokerdEvent instances;
remove BrokerdOrder(**request_msg) and dictionary pattern dispatch;
allocate a distinct deterministic fill_id/transaction id instead of reusing oid as tid;
separate its order protocol tests from market-feed simulation;
allow deterministic synthetic ticks to drive fill tests without a public feed.

Do not enable endpoint pld_spec yet if live backends share code paths that still send legacy payloads.

Gate: the complete broker conformance suite passes for paper without network, credentials, sleeps, or teardown suppression.

Phase 4: Port Live Backend Adapters Individually

Port one backend per independently reviewable change. A backend is complete only when its provider translation fixtures and shared conformance suite pass.

Recommended order:

Binance: replace raw request matching, type startup/fallback, replace existing-order Status with BrokerdOpenOrder, and map fill to fill plus open/closed events.
Kraken: replace raw request matching, type startup/fallback, replace existing-order Status, map filled to fill plus closed, and formalize the too-fast edit/cancel behavior.
IB: replace raw request indexing, type startup, replace existing-order Status, and put an explicit sequencing gate between asyncio provider events and ack publication.

For every backend, core fields are extracted before constructing the event. The provider’s original message stays inside its adapter and logs. Keep brokerd’s enabled RPC module set minimal and do not reintroduce datad modules into the credentialed actor.

Each adapter must preserve a provider execution id as fill_id, emit the initial open-order completion event, and populate a stable namespaced oid for external orders. Binance must retain negative positions in startup. IB startup must associate each position with the correct account/client rather than loop state from the last account.

Gate: paper, Binance, Kraken, and IB all pass the same offline conformance contract; no backend emits a dictionary or client-facing Status to EMS.

Phase 5: Convert EMS Broker Translation and State

Change open_brokerd_dialog() to consume BrokerdStartup and return a typed relay object. Remove startup tuple/string branching and BrokerdPosition(**msg) reconstruction.

Remove the current client_ready dependency from initial broker-event processing. Start the broker relay as soon as its stream opens, consume and reduce BrokerdOpenOrder events through BrokerdOpenOrdersComplete, and only then construct EmsStartup and call _emsd_main().started(). The resulting startup dialogs contain the completed initial snapshot. Events arriving before the first client stream exists remain in dialog state/history for startup or later delivery rather than blocking the broker sender or being discarded.

Split the current large translation loop into a small typed event reducer plus async effects:

dispatch with class patterns, not mapping patterns;
update correlation/dialog state in a deterministic function or narrowly scoped state object;
keep that state mutable internally but emit only new frozen Status snapshots;
return typed client events and typed broker requests to send;
keep stream broadcast and logging in the async shell;
implement bounded pre-ack buffering and terminal-state idempotence;
translate BrokerdOpenOrder to Status in one place;
stop reading routing fields from broker_details.

The reducer must be directly testable with event sequences and no actors.

Gate: all order-state permutations pass actor-free tests, and translate_and_relay_brokerd_events() contains no struct reconstruction from dictionaries.

Phase 6: Convert the Client Boundary

Change process_client_order_cmds() to receive EmsRequest directly and use class-pattern dispatch. Return EmsStartup from _emsd_main() and construct the existing open_ems() convenience maps client-side.

Make OrderClient.send() and send_nowait() accept EmsRequest only. If a mapping convenience API is required by a known caller, put it in an explicit constructor that performs local msgspec validation; do not put dictionaries on the actor stream. Audit UI consumers and tests so receive loops work with already-decoded Status and BrokerdPosition instances.

Replace OrderClient.update*() and UI mutations with construction of a new Order. Audit mutations of Status.req, Status.resp, and embedded orders in order mode and clearing code; all should update internal state or replace a wire value, never mutate a frozen payload.

Gate: there are no Order(**cmd), Cancel(**cmd), Status(**msg), or BrokerdPosition(**msg) conversions on the EMS client boundary.

Phase 7: Enable Tractor Payload Enforcement

Enable one boundary at a time:

Add BrokerdPayload to every paper/live open_trade_dialog() decorator.
Limit backend receive scopes to BrokerdRequest.
Limit EMS’s broker receive scope to BrokerdEvent.
Add EmsPayload to _emsd_main().
Limit EMS’s client receive scope to EmsRequest.
Limit order-client receive scopes to EmsEvent.
Verify every producer satisfies the already-strict wire schemas.

At each step, test invalid startup, stream, wrong-direction, and return payloads. Assert where decoding occurs, tractor.MsgTypeError provenance, context cancellation, and bounded teardown rather than assuming every sender raises locally. Add an actor-free metadata test that each paper/Binance/Kraken/ IB endpoint’s _tractor_context_meta['pld_spec'] resolves to BrokerdPayload and _emsd_main() resolves to EmsPayload.

Gate: no valid wrong-direction type or malformed payload reaches business logic, and malformed traffic cannot wedge the actor tree.

Phase 8: Real Actor and Lifecycle Qualification

Add test-only Tractor context endpoints in a dedicated test fixture module to exercise actual encoding, process boundaries, and cancellation. Do not mock MsgStream in this layer.

The real-actor matrix includes:

valid startup and bidirectional exchange for both protocol families;
exact concrete types observed after remote decode;
invalid Started, stream send, wrong-direction send, and return payload;
optional local Started validation and peer receive-side stream/return validation;
child failure before and after ctx.started();
client cancellation while backend send is blocked;
backend cancellation while EMS waits for an event;
simultaneous stream close and terminal status;
repeated open/close cycles with no stale subscribers, tasks, actors, or SHM;
two clients with distinct dialogs and no cross-delivery of order status;
deterministic ack/status and cancel/ack race barriers;
brokerd restart/reconnect policy, explicitly failing if unsupported rather than hanging.

Use Trio events/channels as barriers. Do not use timing sleeps to force races. Use bounded fail_after() scopes around every scenario. Do not use run_and_tollerate_cancels() for these tests; expected cancellation must be asserted precisely.

Gate: the offline real-actor suite passes repeatedly under the repository’s supported Tractor spawn methods and fails loudly within a bounded timeout.

Phase 9: Remove Compatibility Paths

After all strict tests pass:

delete mapping-pattern IPC dispatch and Struct(**wire_dict) conversions;
delete the 'paper' startup sentinel and startup tuple handling;
delete backend emission of client Status;
delete obsolete status aliases such as fill and filled;
delete legacy trades_dialogue lookup if no backend implements it;
remove stale dictionary type annotations and compatibility comments;
remove expected failures introduced in Phase 0;
update backend author documentation with the protocol aliases and conformance command.

Gate: searches for the removed patterns are empty or have an explicit local, non-IPC justification.

Test Layout

Use focused flat modules consistent with the existing test tree:

Test module	Responsibility	Network
`tests/test_ems_messages.py`	union tags, codec round-trips, invalid schemas, strict fields	never
`tests/test_brokerd_protocol.py`	shared backend conformance and provider translation fixtures	never
`tests/test_ems_state.py`	reducer transitions, correlation, races, idempotence, routing	never
`tests/test_ems_ipc.py`	real Tractor actor payload limits and lifecycle	never
`tests/test_ems.py`	public `open_ems()` behavior, persistence, paper integration	deterministic by default
backend-specific live qualification	authenticated smoke tests only	opt-in

The default CI lane runs every actor-free and offline actor test. Authenticated or public-network checks use explicit markers and are never the only coverage for a protocol behavior.

Required Test Matrix

Schema validation

every valid concrete message;
every startup mode;
int and str request identifiers;
zero, positive, and negative sizes where semantically permitted;
Decimal prices;
optional and populated typed request/event causes;
unknown/missing/wrong tags;
unknown fields rejected from schema introduction;
wrong literal and primitive field types;
malformed nested order/event/position;
a request sent in the event direction and vice versa.

Dialog transitions

submit -> ack -> pending -> open;
submit -> ack -> fill -> open -> fill -> closed;
submit -> ack -> canceled;
submit -> cancel -> ack -> broker cancel -> canceled;
submit -> provider error;
status -> ack reordered delivery;
fill -> ack reordered delivery;
duplicate ack;
duplicate fill_id and distinct equal-valued fills;
duplicate and conflicting terminal statuses;
late fill after close;
update before and after ack;
cancel and update for unknown dialog;
existing external open order;
position update with and without an active order.

Backend conformance

paper, Binance, Kraken, and IB request acceptance;
account rejection;
provider status normalization;
provider exception to BrokerdError conversion;
open-order conversion;
zero-order and populated open-order snapshot completion;
startup account/position conversion;
Binance negative-position startup and IB multi-account association;
provider metadata isolation;
backend handler cancellation and stream closure.

Actor semantics

valid typed startup and streams;
local Started and peer receive-side MsgTypeError provenance;
malformed startup cancels context cleanly;
malformed stream traffic cancels both ends cleanly;
caller and callee cancellation;
no swallowed teardown errors;
no actor/task/channel leaks;
repeated execution to expose scheduling-sensitive failures.

Routing and isolation

event reaches the originating dialog subscribers;
unrelated FQME subscriber does not receive it;
current all-client position broadcast is explicitly characterized as a known limitation pending an account authorization/subscription schema;
disconnected clients are removed from subscriber/dialog tables;
reconnect receives only the documented startup snapshot and active dialogs.

CI and Qualification Gates

Run gates in increasing cost order:

schema and reducer unit tests;
backend conformance tests with fake provider clients;
real Tractor IPC tests;
deterministic paper EMS integration;
the broader clearing/service regression set;
repeated race tests under each supported spawn method;
opt-in live read-only startup for Binance, Kraken, and IB;
manually authorized live order qualification with minimal size.

No live credential or public-network result can waive an offline failure. Never place an order as part of default CI.

Commit Boundaries

Keep the implementation reviewable and bisectable:

add the refreshed dependency baseline, strict schemas, pure reducer, lifecycle guards, and offline/actor protocol qualification;
characterize and port paper together with its EMS consumer;
port each live adapter and its paired EMS handling with provider translation fixtures and execution-identity tests;
migrate client/EMS and UI consumers together;
enforce each migrated endpoint pair with directional limits and lifecycle/race/failure qualification;
remove legacy paths and document backend requirements.

Every commit must leave deterministic tests green. The first boundary keeps legacy endpoints operational while independently qualifying the new protocol. Subsequent migrations move both ends together; no commit requires a raw-dictionary bridge into the strict schema.

Rollback Strategy

Revert a backend port with its paired EMS changes. Once enforcement begins, include the matching decorator and receive limits in that rollback; never leave one actor side strict while restoring a dictionary sender.

No long-lived persisted wire data requires a schema migration. Accounting and ledger files remain outside this protocol change. If a strictness regression is discovered, repair the producer or roll back the complete endpoint pair. Do not widen the schema to accept arbitrary dictionaries or unknown fields.

Exit Criteria

The formal EMS messaging upgrade is complete when:

all client/EMS and EMS/backend IPC payloads are tagged structs;
paper, Binance, Kraken, and IB share one broker startup/request/event contract;
both Tractor context families declare broad pld_spec aliases and narrow each receive direction;
startup, stream, and return payload failures produce tested MsgTypeError behavior;
no broker status outside pending/open/canceled/closed crosses IPC;
fills and existing orders use dedicated variants;
no business logic reconstructs wire dictionaries into message structs;
ack/status/cancel races are deterministic and bounded;
multi-client order routing is tested and the unresolved position-broadcast privacy limitation is explicit;
default EMS tests run without network or credentials;
actor teardown failures are asserted rather than tolerated;
backend authors have one conformance suite and protocol definition to follow.

Primary References

piker/clearing/_messages.py: current EMS and broker message structs.
piker/clearing/_ems.py: broker dialog opening, translation, client command processing, and _emsd_main().
piker/clearing/_client.py: public EMS client and startup handling.
piker/clearing/_paper_engine.py: paper reference backend.
piker/brokers/binance/broker.py: Binance request and event adapter.
piker/brokers/kraken/broker.py: Kraken request and event adapter.
piker/brokers/ib/broker.py: IB request and event adapter.
piker/data/validate.py: existing datad-side FeedInit backend validation direction to improve upon rather than duplicate.
piker/ui/_remote_ctl.py: in-tree pld_spec and per-direction limit_plds() precedent.
piker/ui/_gaps.py: in-tree tagged union precedent.
tests/test_ems.py: current network-dependent clearing coverage and empty order tests.
/home/goodboy/repos/tractor/tests/msg/test_pldrx_limiting.py: canonical payload limiting and MsgTypeError behavior.
/home/goodboy/repos/tractor/tests/msg/test_ext_types_msgspec.py: canonical msgspec codec behavior.
/home/goodboy/repos/tractor/tests/test_context_stream_semantics.py: context stream cancellation and serialization semantics.

Relevant issue/design threads:

41 KiB Raw Blame History Unescape Escape

Typed EMS and Brokerd Messaging

October 2026 implementation baseline

Qualified dependency baseline and first review boundary

Purpose

Scope

Current Contract Inventory

Client and EMS boundary

EMS and broker boundary

Target Protocol

Tagging and decoding policy

Client-to-EMS requests

EMS-to-client events

EMS-to-broker requests

Broker-to-EMS events

Startup payloads

Payload aliases and Tractor limits

Provider metadata

Ordering and State Semantics

Implementation Phases

Phase 0: Characterize the Existing Boundary

Phase 1: Define and Audit the Wire Schemas

Phase 2: Build a Backend Conformance Harness

Phase 3: Port the Paper Engine as the Reference Backend

Phase 4: Port Live Backend Adapters Individually

Phase 5: Convert EMS Broker Translation and State

Phase 6: Convert the Client Boundary

Phase 7: Enable Tractor Payload Enforcement

Phase 8: Real Actor and Lifecycle Qualification

Phase 9: Remove Compatibility Paths

Test Layout

Required Test Matrix

Schema validation

Dialog transitions

Backend conformance

Actor semantics

Routing and isolation

CI and Qualification Gates

Commit Boundaries

Rollback Strategy

Exit Criteria

Primary References

41 KiB

Raw Blame History