piker/plans/opencode/typed-ems-messaging.md

# Typed EMS and Brokerd Messaging

## October 2026 implementation baseline

This revision carries the August 28 design into the September Piker
baseline (`eed58566`). Preserve IB execution-ID accounting (`5eb4fc9d`)
and socket-disconnect propagation (`efa72980`) while changing payloads.
The installed Tractor is `5c98ab1f`; qualification must distinguish
that April revision from local `main` (`5cb600a8`) and the September
context-fault development stack. Do not test against a dirty upstream
checkout without recording its changes.

The complete contract includes local client queues, EMS startup and
service readiness, both order-control streams, quote normalization for
dark/paper execution, and typed internal trigger/dialog records.
Provider-native APIs and persistent ledger formats retain their own
normalization boundaries. Runtime handles are not wire messages.

Use explicit Struct validation before local state mutation or sending;
annotations and Struct construction do not validate field values.
Receive decoding proves wire shape, not the sender's Python type: a
correctly tagged mapping has the same encoding as a Struct.

Endpoint-wide unions need separate startup, receive-direction, and
final-result checks. Validate RPC arguments explicitly until Tractor
issue #514 supplies signature-derived `Start` contracts. Never admit
`Any` or raw diagnostic mappings into the clearing wire schema.

Frozen snapshots must use replacement rather than the inherited
`Struct.copy(update=...)`, which mutates its source. Keep mutable EMS
dialog state separate from published messages and retain execution IDs
for fill deduplication. Position messages remain explicit summaries;
they must never become synthetic entries in durable trade accounting.

Port each boundary's producers and consumers together before enabling
its strict receive set. Qualify schema errors, wrong-direction and
wrong-phase messages, startup completion, cancel-before-ack, reordered
and repeated fills, transport failure, and bounded teardown with
offline provider fixtures and real local actors. Live order submission
is outside this qualification.


### Qualified dependency baseline and first review boundary

The first boundary introduces `._schema`, local validation in `._codec`,
directional streams and lifecycle checks in `._protocol`, and a pure
`._state.OrderState` reducer. Existing endpoints keep their original
`._messages` imports until their paired migration. This is infrastructure
qualification, not a claim that paper or any live adapter is ported.

The installed baseline is Python 3.13.14, msgspec 0.19.0, Trio 0.32.0,
and Tractor `5c98ab1fb6e2cb7781aa75845c7f323a52853c46`. Its existing
payload APIs support this boundary without a dependency update.
All eight actor contract tests also pass against clean local Tractor
`main` at `5cb600a84b6e9fcac68bad04a487b83640b3e5fb`.

The first set has 65 schema/state/phase tests and eight real IPC tests.
These tests cover protocol fixtures and the new reducer; adapter-level
conformance, feed-driven paper clearing, abrupt transport EOF, and
other spawn backends remain qualification work for subsequent ports.

That local main adds `--ll` to Tractor's pytest plugin; Piker's current
`tests/conftest.py` adds the same option. The ordinary downstream
invocation therefore fails before collection. The isolated API check
uses `--noconftest -p tractor._testing.pytest` on the same test file.
Resolve that harness collision before upgrading Piker's dependency pin.
Do not conflate this isolated API qualification with a whole-suite pass.

Relevant unmerged Tractor work remains a separate qualification input:
`pld_dec_refinements` (`6bd685ad`), `auto_codecs` (`45508252`),
`wkt/context_fault_status` (`9547c2f9`), and
`wkt/ipc_fault_regressions` (`fe9eb738`). The context-fault worktree has
uncommitted transport-error handling changes; it is not a clean test
baseline. The IPC-fault branch includes `192411c4` for transport EOF
propagation. Typed Start arguments remain upstream work; validate them
explicitly when migrating the actual clearing contexts.

## Purpose

Replace EMS's dictionary-shaped actor traffic with explicit, tagged
`msgspec.Struct` protocols and modern Tractor payload enforcement on both
actor boundaries:

```text
order client <-> emsd <-> brokerd.<backend> / paperboi.<backend>
```

The broker boundary is part of the formal protocol, not an implementation
detail behind the EMS endpoint. Paper, Binance, Kraken, and IB must all emit
the same startup type, consume the same request union, and produce the same
event union before strict Tractor payload limits are enabled.

This plan deliberately normalizes schemas and backend behavior before adding
`pld_spec`. Enabling transport validation first would only turn existing
status drift, startup-shape drift, and raw dictionary reconstruction into
hard actor failures without first defining the intended behavior.

## Scope

This work includes:

- typed client-to-EMS requests and EMS-to-client events;
- typed EMS-to-broker requests and broker-to-EMS events;
- one typed broker startup payload shared by live and paper endpoints;
- one typed EMS startup payload;
- stable tagged unions suitable for native `msgspec` decoding;
- normalized status, identifier, startup, and metadata semantics;
- per-direction `Context.pld_rx.limit_plds()` enforcement;
- backend-neutral conformance tests for paper, Binance, Kraken, and IB;
- deterministic actor-free tests and real Tractor actor/IPC tests;
- bounded handling of acknowledgement, status, cancellation, and teardown
  races;
- removal of raw dictionary dispatch after all producers have migrated.

This work does not include:

- typing provider-native websocket or REST payloads inside each backend;
- changing the datad/brokerd actor split;
- spawning brokerd from any location other than `open_brokerd_dialog()`;
- redesigning accounting or replacing `BrokerdPosition` with the full
  accounting `Position` type in the same patch series;
- live order submission in CI;
- preserving undocumented dictionary IPC between actors running different
  Piker revisions. The actor tree is treated as a same-checkout deployment.

## Current Contract Inventory

### Client and EMS boundary

| Location | Startup payload | Stream received | Stream sent | Current issue |
| --- | --- | --- | --- | --- |
| `_emsd_main()` | tuple of positions map, accounts list, dialogs map | dictionaries reconstructed as `Order` or `Cancel` | `Status` and `BrokerdPosition`, often decoded as dictionaries by clients | no payload spec, tuple startup, raw matching, loose nested fields |
| `open_ems()` / `OrderClient` | destructures tuple | `Status` and `BrokerdPosition` dictionaries | annotated as `Order | dict`, but implementation assumes struct attributes | public shape and wire shape are conflated |

### EMS and broker boundary

| Endpoint | Startup payload | Stream received by backend | Stream sent by backend | Current issue |
| --- | --- | --- | --- | --- |
| paper `open_trade_dialog()` | `(list[BrokerdPosition], ['paper'])` | raw `BrokerdOrder` / `BrokerdCancel` dictionaries | ack, status, fill, error, position structs | raw dispatch and network-dependent simulator setup |
| Binance `open_trade_dialog()` | `'paper'` sentinel or `(positions, accounts)` | raw request dictionaries | structs plus client-side `Status` for existing orders | mixed protocol layers and `fill` status drift |
| Kraken `open_trade_dialog()` | `'paper'` sentinel or `(positions, accounts)` | raw request dictionaries | structs plus client-side `Status` for existing orders | `filled` status drift and race workarounds |
| IB `open_trade_dialog()` | `(positions, tuple(accounts))` | raw request dictionaries | structs plus client-side `Status` for existing orders | status can race ack and internal event types leak into sequencing |

The shared broker boundary already exists operationally: all four endpoints
are long-lived Tractor contexts opened by `open_brokerd_dialog()`. The upgrade
formalizes that de facto interface instead of allowing each backend to define
its own wire dialect.

## Target Protocol

### Tagging and decoding policy

Every union member must have an explicit stable tag. Use one common tag field,
called `type`, with explicit protocol values such as `order`, `cancel`,
`broker_ack`, `status`, `broker_fill`, `broker_error`, `position`, and
`open_order`. Broker requests have distinct `broker_order` and
`broker_cancel` tags so decoding cannot confuse the two directions. Do not rely on
Python class names as wire tags.

All newly formalized wire structs should use:

- `tag='<stable value>'` and `tag_field='type'`;
- `frozen=True` for wire snapshots;
- `kw_only=True` where it improves constructor safety;
- immutable tuple defaults; factories for mutable internal state;
- finite `Literal` values for protocol enums;
- `forbid_unknown_fields=True` from schema introduction; adopt the new
  schema only when both endpoints can meet it.

The tag is the dispatch discriminator. Fields such as `action`, `status`, and
`resp` retain only domain meaning and are not overloaded as message type tags.
Mutable EMS dialog bookkeeping must use a separate internal state type. Do not
make a wire struct mutable merely because current code mutates and re-sends the
same `Status` or `Order` instance.

Define shared scalar aliases and use them consistently:

```python
RequestId = int | str
FillId = str
```

### Client-to-EMS requests

Define:

```python
EmsRequest = Order | Cancel
```

Normalize the request structs as follows:

- `Order.action` is exactly `buy | sell | alert`.
- `Order.exec_mode` is exactly `dark | live`.
- `Order.oid`, `symbol`, and `account` remain required strings.
- `Order.price` remains `Decimal`; tests must prove transport round-tripping.
- `Order.size` remains signed `float` for this migration. Resolving the
  duplicated side/action representation is a later domain change.
- `Cancel` has its own tag; `action='cancel'` can be removed once no local UI
  matching depends on it.
- Actor IPC accepts structs only. Any user-facing convenience constructor for
  mappings must validate and construct a struct before entering the stream.

### EMS-to-client events

Define:

```python
EmsEvent = Status | BrokerdPosition
```

Retain `BrokerdPosition` as the wire-level position summary for this series,
but document it as shared by the broker and client event protocols. A later
accounting patch may rename it to `PositionSummary` without coupling that
change to IPC enforcement.

Normalize `Status` so that:

- `resp` is exactly `pending | open | dark_open | triggered | fill | closed |
  canceled | error`;
- `req` is a required `Order`, never an unvalidated nested dictionary;
  an uncorrelated backend error must not fabricate a client dialog;
- `brokerd_request` is `BrokerdRequest | None` and records the command that
  caused the status transition;
- `brokerd_event` is `BrokerdEvent | None` and records the backend response;
- the current mixed-purpose `brokerd_msg` field is removed after callers have
  migrated; neither replacement accepts an untyped dictionary;
- an error is represented by `Status(resp='error', ...)` or by a distinct
  tagged client event, not the current incomplete `Error(Status)` subclass;
- internal mutable dialog state is not the same object sent repeatedly over
  the wire. Build a status snapshot when broadcasting so prior history and
  multi-client delivery cannot be changed by later in-place mutation.

### EMS-to-broker requests

Define:

```python
BrokerdRequest = BrokerdOrder | BrokerdCancel
```

Normalize both variants so that:

- they have explicit `broker_order` and `broker_cancel` protocol tags
  independent of `action`;
- `oid` always identifies the EMS dialog;
- `BrokerdOrder.reqid` is `RequestId | None` because allocation policy
  differs by backend; `BrokerdCancel.reqid` is required, and EMS retains
  cancel intent until acknowledgement supplies that identity;
- account and symbol validation failures produce a typed `BrokerdError`
  rather than logging and continuing silently;
- updates remain `BrokerdOrder` with an existing `reqid`; a separate modify
  variant is deferred unless conformance tests prove the current distinction
  ambiguous.

### Broker-to-EMS events

Define:

```python
BrokerdEvent = (
    BrokerdOrderAck
    | BrokerdStatus
    | BrokerdFill
    | BrokerdError
    | BrokerdPosition
    | BrokerdOpenOrder
    | BrokerdOpenOrdersComplete
)
```

`BrokerdOpenOrder` replaces the current backend-to-EMS use of client-facing
`Status`. It carries the provider request id, account, source backend, and a
fully typed embedded `Order`. EMS alone translates this event into a client
`Status` and registers it in the dialog book.

Require `oid` on acknowledgements. Status and fill events identify the
order by `(account, reqid)` within their broker session; EMS retains that
correlation across edits and terminal events. For an external order, the
adapter creates a collision-safe stable `oid` from broker, account, and
provider `reqid` before emitting `BrokerdOpenOrder`. Its embedded
`Order.oid` carries that value. EMS registers it without allocating a
reverse-message identity. Service errors may omit `oid`; a typed rejected
request can identify a submission which never received a provider ID.

`BrokerdFill` has a required `fill_id: FillId`. Use the provider execution id
or provider event sequence. The four current implementations must use Binance
trade id, Kraken transaction id, IB execution id, and a distinct paper-ledger
transaction id respectively. Price/size/time are not a safe deduplication key
because two legitimate executions may share them. A future provider with no
execution identity must define an adapter session-id plus monotonic sequence
and document that deduplication does not survive brokerd restart; it may not
silently synthesize identity from equal-valued fill fields.

Use one broker status vocabulary:

```python
BrokerdOrderState = Literal[
    'pending',
    'open',
    'canceled',
    'closed',
]
```

Fills are events, not status values. A partial execution emits a
`BrokerdFill` and may then emit `BrokerdStatus(status='open', ...)`. A complete
execution emits the final `BrokerdFill` followed by
`BrokerdStatus(status='closed', remaining=0)`. Binance `fill`, Kraken
`filled`, and IB provider status names must be translated inside their
backend adapters and must never cross the brokerd IPC boundary.

### Startup payloads

Replace every tuple and string sentinel with tagged startup structs.

`BrokerdStartup` contains:

- `mode: Literal['live', 'paper', 'paper_fallback']`;
- `broker: str`;
- `accounts: tuple[str, ...]`;
- `positions: tuple[BrokerdPosition, ...]`.

A live endpoint without credentials returns
`BrokerdStartup(mode='paper_fallback', broker=..., accounts=(), positions=())`.
`open_brokerd_dialog()` then opens the paper endpoint, which returns the same
type with `mode='paper'`, its paper account, and persisted positions. The
fallback value is a typed delegation result, not an active broker snapshot: it
is exempt from account/position and open-order-completion requirements, and
its context returns after startup. This removes the raw `'paper'` union branch
without hiding fallback behavior.

Existing open orders remain `BrokerdOpenOrder` stream events because some
providers discover them only after authenticated stream subscription. The
contract must state that startup positions/accounts are a complete snapshot,
while open-order synchronization completes through typed stream events. Every
active live or paper backend emits exactly one `BrokerdOpenOrdersComplete`
after the initial set, including when it found zero orders, so EMS can
distinguish completion from delay and can order initial client readiness
deterministically. A `paper_fallback` delegation does not open a stream and
does not emit this event; the subsequently opened paper endpoint does.

`EmsStartup` contains:

- `broker: str`;
- `positions: tuple[BrokerdPosition, ...]`;
- `accounts: tuple[str, ...]`;
- `dialogs: tuple[Status, ...]`.

`open_ems()` may continue returning the existing nested lookup maps to callers,
but it builds them locally from `EmsStartup`. The public convenience shape is
therefore separated from the wire contract.

Treat `BrokerdStartup` as the brokerd equivalent of datad's `FeedInit`, but do
not reproduce `validate_backend()`'s long-lived dictionary compatibility
branches. Add a broker initialization validator beside the clearing protocol
that checks backend name, unique/non-empty accounts, position account
membership, `position.broker == startup.broker`, unique
`(broker, account, symbol)` position keys, and mode-specific requirements.
Give malformed startup its own registered initialization error so failures
retain a useful remote exception type before strict Tractor decoding takes
over. Each account name must be non-empty; the accounts collection may be
empty only for `paper_fallback`. Paper sessions announce exactly
`('paper',)`. EMS readiness also requires at least one account.

Replace `open_brokerd_dialog()`'s manual probing with one broker order-control
endpoint resolver. It may use `get_eps(mod, 'brokerd')` as input, but it must
select exactly one of `open_trade_dialog` or the temporary legacy
`trades_dialogue` alias and must not confuse unrelated `norm_trade` support
with a trading dialog. The runtime validator proves startup; the conformance
suite proves request/event behavior. Neither mechanism should import or enable
datad modules inside the credentialed broker actor.

### Payload aliases and Tractor limits

The endpoint-wide payload aliases include startup, stream traffic, and the
final `None` result:

```python
EmsPayload = EmsStartup | EmsRequest | EmsEvent | None
BrokerdPayload = BrokerdStartup | BrokerdRequest | BrokerdEvent | None
```

Apply broad endpoint specs and narrow receive-side specs as follows:

| Actor side | Endpoint `pld_spec` | Narrow receive spec |
| --- | --- | --- |
| emsd `_emsd_main()` | `EmsPayload` | `EmsRequest` around the client command loop |
| order client | inherited from endpoint | `EmsEvent` around trade event receives |
| backend/paper `open_trade_dialog()` | `BrokerdPayload` | `BrokerdRequest` around backend request loops |
| emsd broker relay | inherited from endpoint | `BrokerdEvent` around broker event receives |

The endpoint payload spec supplies the broad decoder used for `Started`, stream,
and return payloads. `ctx.started(..., validate_pld_spec=True)` can also perform
local started-value validation. Ordinary `MsgStream.send()` does not validate
locally: malformed stream data raises `tractor.MsgTypeError` when the peer
receives/decodes it. `ctx.pld_rx.limit_plds(spec=...)` narrows that receiving
direction so a valid message from the wrong direction still fails there.

Do not add either decorator spec until the corresponding boundary has no
remaining legacy sender. The final enforcement must follow the proven pattern
in `remote_gap_overlays()` and Tractor's own payload-limiting tests.

### Provider metadata

Provider-native diagnostics remain inside adapters and their logs. The
clearing wire schema has no raw-data extension point or `Any` field.
Source backend, broker timestamp, symbol, account, order ID, request ID,
and execution ID have explicit typed fields. Errors can carry a typed
request. Position summaries contain no mutable accounting history.

## Ordering and State Semantics

Typed messages alone do not repair ordering defects. Define these invariants
before enforcement:

1. A backend sends exactly one `BrokerdOrderAck` for each accepted new order.
2. The ack establishes `oid <-> reqid` before status/fill events are exposed
   to EMS whenever the provider API permits that ordering.
3. A backend adapter that receives provider events before it can send the ack
   buffers them by `reqid` and flushes them after ack.
4. EMS keeps a bounded defensive pre-ack buffer because actor scheduling
   can still expose ordering bugs. The first reducer bounds event count
   and fails loudly on overflow without dropping retained executions.
   Add wall-clock expiry in its asynchronous owner when porting EMS;
   the pure reducer does not claim an age bound.
5. A cancel received before ack marks the dialog `cancel_called`. After
   replaying buffered events, the first ack sends one typed
   `BrokerdCancel` if the dialog is still active.
6. Duplicate ack, fill, and terminal status events are idempotent or rejected
   deterministically. Fill deduplication uses `fill_id`. They never resurrect
   a closed dialog.
7. `closed` and `canceled` are terminal. A late fill may be recorded and
   reported according to an explicit test, but it cannot recreate mutable
   active state.
8. Existing external orders enter through `BrokerdOpenOrder` with the backend's
   stable namespaced `oid` and are registered before broadcast.
9. Initial open-order discovery is complete only after
   `BrokerdOpenOrdersComplete`; client readiness cannot infer completion from
   an empty or temporarily quiet stream.
10. Order status is routed by dialog and FQME. Account authorization and
    position privacy remain a separately scoped subscription design because
    `_emsd_main()` currently receives no account authorization input. Tests in
    this series document the current position broadcast and must not claim
    unauthorized-account isolation.

## Implementation Phases

### Phase 0: Characterize the Existing Boundary

Add tests that record current intended behavior before restructuring:

- new order, update, cancel, ack, open, partial fill, close, and position flow;
- cancel-before-ack;
- status-before-ack;
- existing provider order discovery;
- bad account, zero size, and backend submission error as typed domain errors;
- bad action/tag/field type as protocol decode failures, not `BrokerdError`;
- startup with live credentials, no credentials, paper account, persisted
  positions, and no positions;
- Binance startup containing a negative/short position;
- IB startup containing at least two accounts with distinct contracts;
- zero, one, and multiple existing orders followed by the explicit snapshot
  completion event;
- disconnect while requests or events are in flight.

Use fake streams and fake provider clients; do not require public market data
or credentials. Capture known undesirable outcomes as explicit expected
failures only when a same-phase fix would obscure the baseline.

Gate: every behavior changed later has a deterministic assertion describing
the old behavior and the target behavior.

### Phase 1: Define and Audit the Wire Schemas

Introduce the strict schema in `piker/clearing/_schema.py` alongside the
legacy `._messages` imports. Qualify it independently before moving each
producer/consumer pair together. Remove the legacy module after every
endpoint and public consumer has migrated; do not add dictionary bridges.

Replace mutable literal defaults with factories. Resolve the accidental tuple
default on `BrokerdStatus.account`. Remove or complete the unusable
`Error.from_status()` path. Add explicit startup types and
`BrokerdOpenOrder`. Add the broker initialization error/validator without a
legacy dictionary fallback.

Inventory every current mutation of `Order`, `Status`, and embedded requests.
Introduce a mutable internal dialog-state record and construct frozen wire
snapshots from it. Replace mutation-based `Struct.copy(update=...)` usage with
new-instance replacement that is proven to work for frozen structs.

Add actor-free serialization tests using the same msgpack semantics as
Tractor:

- each concrete type round-trips with exact type equality;
- each union decodes to the correct concrete type by tag;
- nested `Order`, broker event, position, `Decimal`, and optional
  identifier fields round-trip;
- unknown tag, missing required field, wrong literal, wrong primitive type,
  malformed nested struct, and duplicate/ambiguous tag fail;
- unknown fields fail from the first use of the new schema;
- request and event aliases do not accept each other's members.

Gate: all protocol unions have unique tags and can be decoded without manual
dictionary inspection.

### Phase 2: Build a Backend Conformance Harness

Create reusable conformance helpers that exercise a backend request handler
with a memory-backed stream and fake provider client. The harness supplies the
same typed request cases and validates only typed events. Every request and
emitted event must be encoded and decoded through its declared union before
assertion; `isinstance` checks alone are insufficient because direct
`msgspec.Struct` construction does not validate annotated field types.

The shared suite must verify:

- startup is one `BrokerdStartup`, including typed fallback-to-paper;
- malformed startup fails the shared broker initialization validator;
- new order produces one correlated ack;
- update preserves `oid` and `reqid`;
- cancel uses the correlated request id;
- invalid account/size produces `BrokerdError` without killing the dialog;
- invalid action/tag/type is rejected by codec or Tractor before the handler;
- provider statuses normalize to the four broker states;
- partial and complete fills use `BrokerdFill` plus normalized statuses;
- emitted positions and existing orders are typed;
- every active live/paper startup emits one open-order completion event, even
  for zero orders; `paper_fallback` delegates without one;
- duplicate fills are rejected by `fill_id`, while distinct equal-valued fills
  remain valid;
- provider-native dictionaries never cross the test stream;
- cancellation closes all handler tasks and leaves no blocked sender;
- one malformed request fails at the protocol boundary once limits are
  enabled.

Separate backend-neutral contract assertions from provider translation
fixtures. Provider fixtures should be the smallest recorded websocket/API
payload required to prove normalization. Include regressions for Binance's
tuple-shaped error reason, string fill size, short startup position, and
float-vs-`Decimal` open-order price; Kraken's `filled` status and open-order
price; and IB multi-account startup association.

Gate: the harness can fail a deliberately nonconforming fake backend for each
contract rule.

### Phase 3: Port the Paper Engine as the Reference Backend

Make paper `open_trade_dialog()` the reference implementation of
`BrokerdPayload`:

- return `BrokerdStartup`;
- consume `BrokerdRequest` instances directly;
- emit only `BrokerdEvent` instances;
- remove `BrokerdOrder(**request_msg)` and dictionary pattern dispatch;
- allocate a distinct deterministic `fill_id`/transaction id instead of
  reusing `oid` as `tid`;
- separate its order protocol tests from market-feed simulation;
- allow deterministic synthetic ticks to drive fill tests without a public
  feed.

Do not enable endpoint `pld_spec` yet if live backends share code paths that
still send legacy payloads.

Gate: the complete broker conformance suite passes for paper without network,
credentials, sleeps, or teardown suppression.

### Phase 4: Port Live Backend Adapters Individually

Port one backend per independently reviewable change. A backend is complete
only when its provider translation fixtures and shared conformance suite pass.

Recommended order:

1. Binance: replace raw request matching, type startup/fallback, replace
   existing-order `Status` with `BrokerdOpenOrder`, and map `fill` to fill plus
   `open/closed` events.
2. Kraken: replace raw request matching, type startup/fallback, replace
   existing-order `Status`, map `filled` to fill plus `closed`, and formalize
   the too-fast edit/cancel behavior.
3. IB: replace raw request indexing, type startup, replace existing-order
   `Status`, and put an explicit sequencing gate between asyncio provider
   events and ack publication.

For every backend, core fields are extracted before constructing the event.
The provider's original message stays inside its adapter and logs.
Keep brokerd's enabled RPC module set minimal and do not reintroduce
datad modules into the credentialed actor.

Each adapter must preserve a provider execution id as `fill_id`, emit the
initial open-order completion event, and populate a stable namespaced `oid` for
external orders. Binance must retain negative positions in startup. IB startup
must associate each position with the correct account/client rather than loop
state from the last account.

Gate: paper, Binance, Kraken, and IB all pass the same offline conformance
contract; no backend emits a dictionary or client-facing `Status` to EMS.

### Phase 5: Convert EMS Broker Translation and State

Change `open_brokerd_dialog()` to consume `BrokerdStartup` and return a typed
relay object. Remove startup tuple/string branching and
`BrokerdPosition(**msg)` reconstruction.

Remove the current `client_ready` dependency from initial broker-event
processing. Start the broker relay as soon as its stream opens, consume and
reduce `BrokerdOpenOrder` events through `BrokerdOpenOrdersComplete`, and only
then construct `EmsStartup` and call `_emsd_main().started()`. The resulting
startup dialogs contain the completed initial snapshot. Events arriving before
the first client stream exists remain in dialog state/history for startup or
later delivery rather than blocking the broker sender or being discarded.

Split the current large translation loop into a small typed event reducer plus
async effects:

- dispatch with class patterns, not mapping patterns;
- update correlation/dialog state in a deterministic function or narrowly
  scoped state object;
- keep that state mutable internally but emit only new frozen `Status`
  snapshots;
- return typed client events and typed broker requests to send;
- keep stream broadcast and logging in the async shell;
- implement bounded pre-ack buffering and terminal-state idempotence;
- translate `BrokerdOpenOrder` to `Status` in one place;
- stop reading routing fields from `broker_details`.

The reducer must be directly testable with event sequences and no actors.

Gate: all order-state permutations pass actor-free tests, and
`translate_and_relay_brokerd_events()` contains no struct reconstruction from
dictionaries.

### Phase 6: Convert the Client Boundary

Change `process_client_order_cmds()` to receive `EmsRequest` directly and use
class-pattern dispatch. Return `EmsStartup` from `_emsd_main()` and construct
the existing `open_ems()` convenience maps client-side.

Make `OrderClient.send()` and `send_nowait()` accept `EmsRequest` only. If a
mapping convenience API is required by a known caller, put it in an explicit
constructor that performs local msgspec validation; do not put dictionaries on
the actor stream. Audit UI consumers and tests so receive loops work with
already-decoded `Status` and `BrokerdPosition` instances.

Replace `OrderClient.update*()` and UI mutations with construction of a new
`Order`. Audit mutations of `Status.req`, `Status.resp`, and embedded orders in
order mode and clearing code; all should update internal state or replace a
wire value, never mutate a frozen payload.

Gate: there are no `Order(**cmd)`, `Cancel(**cmd)`, `Status(**msg)`, or
`BrokerdPosition(**msg)` conversions on the EMS client boundary.

### Phase 7: Enable Tractor Payload Enforcement

Enable one boundary at a time:

1. Add `BrokerdPayload` to every paper/live `open_trade_dialog()` decorator.
2. Limit backend receive scopes to `BrokerdRequest`.
3. Limit EMS's broker receive scope to `BrokerdEvent`.
4. Add `EmsPayload` to `_emsd_main()`.
5. Limit EMS's client receive scope to `EmsRequest`.
6. Limit order-client receive scopes to `EmsEvent`.
7. Verify every producer satisfies the already-strict wire schemas.

At each step, test invalid startup, stream, wrong-direction, and return
payloads. Assert where decoding occurs, `tractor.MsgTypeError` provenance,
context cancellation, and bounded teardown rather than assuming every sender
raises locally. Add an actor-free metadata test that each paper/Binance/Kraken/
IB endpoint's `_tractor_context_meta['pld_spec']` resolves to `BrokerdPayload`
and `_emsd_main()` resolves to `EmsPayload`.

Gate: no valid wrong-direction type or malformed payload reaches business
logic, and malformed traffic cannot wedge the actor tree.

### Phase 8: Real Actor and Lifecycle Qualification

Add test-only Tractor context endpoints in a dedicated test fixture module to
exercise actual encoding, process boundaries, and cancellation. Do not mock
`MsgStream` in this layer.

The real-actor matrix includes:

- valid startup and bidirectional exchange for both protocol families;
- exact concrete types observed after remote decode;
- invalid `Started`, stream send, wrong-direction send, and return payload;
- optional local `Started` validation and peer receive-side stream/return
  validation;
- child failure before and after `ctx.started()`;
- client cancellation while backend send is blocked;
- backend cancellation while EMS waits for an event;
- simultaneous stream close and terminal status;
- repeated open/close cycles with no stale subscribers, tasks, actors, or SHM;
- two clients with distinct dialogs and no cross-delivery of order status;
- deterministic ack/status and cancel/ack race barriers;
- brokerd restart/reconnect policy, explicitly failing if unsupported rather
  than hanging.

Use Trio events/channels as barriers. Do not use timing sleeps to force races.
Use bounded `fail_after()` scopes around every scenario. Do not use
`run_and_tollerate_cancels()` for these tests; expected cancellation must be
asserted precisely.

Gate: the offline real-actor suite passes repeatedly under the repository's
supported Tractor spawn methods and fails loudly within a bounded timeout.

### Phase 9: Remove Compatibility Paths

After all strict tests pass:

- delete mapping-pattern IPC dispatch and `Struct(**wire_dict)` conversions;
- delete the `'paper'` startup sentinel and startup tuple handling;
- delete backend emission of client `Status`;
- delete obsolete status aliases such as `fill` and `filled`;
- delete legacy `trades_dialogue` lookup if no backend implements it;
- remove stale dictionary type annotations and compatibility comments;
- remove expected failures introduced in Phase 0;
- update backend author documentation with the protocol aliases and
  conformance command.

Gate: searches for the removed patterns are empty or have an explicit local,
non-IPC justification.

## Test Layout

Use focused flat modules consistent with the existing test tree:

| Test module | Responsibility | Network |
| --- | --- | --- |
| `tests/test_ems_messages.py` | union tags, codec round-trips, invalid schemas, strict fields | never |
| `tests/test_brokerd_protocol.py` | shared backend conformance and provider translation fixtures | never |
| `tests/test_ems_state.py` | reducer transitions, correlation, races, idempotence, routing | never |
| `tests/test_ems_ipc.py` | real Tractor actor payload limits and lifecycle | never |
| `tests/test_ems.py` | public `open_ems()` behavior, persistence, paper integration | deterministic by default |
| backend-specific live qualification | authenticated smoke tests only | opt-in |

The default CI lane runs every actor-free and offline actor test. Authenticated
or public-network checks use explicit markers and are never the only coverage
for a protocol behavior.

## Required Test Matrix

### Schema validation

- every valid concrete message;
- every startup mode;
- `int` and `str` request identifiers;
- zero, positive, and negative sizes where semantically permitted;
- `Decimal` prices;
- optional and populated typed request/event causes;
- unknown/missing/wrong tags;
- unknown fields rejected from schema introduction;
- wrong literal and primitive field types;
- malformed nested order/event/position;
- a request sent in the event direction and vice versa.

### Dialog transitions

- submit -> ack -> pending -> open;
- submit -> ack -> fill -> open -> fill -> closed;
- submit -> ack -> canceled;
- submit -> cancel -> ack -> broker cancel -> canceled;
- submit -> provider error;
- status -> ack reordered delivery;
- fill -> ack reordered delivery;
- duplicate ack;
- duplicate `fill_id` and distinct equal-valued fills;
- duplicate and conflicting terminal statuses;
- late fill after close;
- update before and after ack;
- cancel and update for unknown dialog;
- existing external open order;
- position update with and without an active order.

### Backend conformance

- paper, Binance, Kraken, and IB request acceptance;
- account rejection;
- provider status normalization;
- provider exception to `BrokerdError` conversion;
- open-order conversion;
- zero-order and populated open-order snapshot completion;
- startup account/position conversion;
- Binance negative-position startup and IB multi-account association;
- provider metadata isolation;
- backend handler cancellation and stream closure.

### Actor semantics

- valid typed startup and streams;
- local `Started` and peer receive-side `MsgTypeError` provenance;
- malformed startup cancels context cleanly;
- malformed stream traffic cancels both ends cleanly;
- caller and callee cancellation;
- no swallowed teardown errors;
- no actor/task/channel leaks;
- repeated execution to expose scheduling-sensitive failures.

### Routing and isolation

- event reaches the originating dialog subscribers;
- unrelated FQME subscriber does not receive it;
- current all-client position broadcast is explicitly characterized as a
  known limitation pending an account authorization/subscription schema;
- disconnected clients are removed from subscriber/dialog tables;
- reconnect receives only the documented startup snapshot and active dialogs.

## CI and Qualification Gates

Run gates in increasing cost order:

1. schema and reducer unit tests;
2. backend conformance tests with fake provider clients;
3. real Tractor IPC tests;
4. deterministic paper EMS integration;
5. the broader clearing/service regression set;
6. repeated race tests under each supported spawn method;
7. opt-in live read-only startup for Binance, Kraken, and IB;
8. manually authorized live order qualification with minimal size.

No live credential or public-network result can waive an offline failure.
Never place an order as part of default CI.

## Commit Boundaries

Keep the implementation reviewable and bisectable:

1. add the refreshed dependency baseline, strict schemas, pure reducer,
   lifecycle guards, and offline/actor protocol qualification;
2. characterize and port paper together with its EMS consumer;
3. port each live adapter and its paired EMS handling with provider
   translation fixtures and execution-identity tests;
4. migrate client/EMS and UI consumers together;
5. enforce each migrated endpoint pair with directional limits and
   lifecycle/race/failure qualification;
6. remove legacy paths and document backend requirements.

Every commit must leave deterministic tests green. The first boundary
keeps legacy endpoints operational while independently qualifying the
new protocol. Subsequent migrations move both ends together; no commit
requires a raw-dictionary bridge into the strict schema.

## Rollback Strategy

Revert a backend port with its paired EMS changes. Once enforcement
begins, include the matching decorator and receive limits in that
rollback; never leave one actor side strict while restoring a
dictionary sender.

No long-lived persisted wire data requires a schema migration. Accounting and
ledger files remain outside this protocol change. If a strictness regression
is discovered, repair the producer or roll back the complete endpoint pair.
Do not widen the schema to accept arbitrary dictionaries or unknown fields.

## Exit Criteria

The formal EMS messaging upgrade is complete when:

- all client/EMS and EMS/backend IPC payloads are tagged structs;
- paper, Binance, Kraken, and IB share one broker startup/request/event
  contract;
- both Tractor context families declare broad `pld_spec` aliases and narrow
  each receive direction;
- startup, stream, and return payload failures produce tested
  `MsgTypeError` behavior;
- no broker status outside `pending/open/canceled/closed` crosses IPC;
- fills and existing orders use dedicated variants;
- no business logic reconstructs wire dictionaries into message structs;
- ack/status/cancel races are deterministic and bounded;
- multi-client order routing is tested and the unresolved position-broadcast
  privacy limitation is explicit;
- default EMS tests run without network or credentials;
- actor teardown failures are asserted rather than tolerated;
- backend authors have one conformance suite and protocol definition to follow.

## Primary References

- `piker/clearing/_messages.py`: current EMS and broker message structs.
- `piker/clearing/_ems.py`: broker dialog opening, translation, client command
  processing, and `_emsd_main()`.
- `piker/clearing/_client.py`: public EMS client and startup handling.
- `piker/clearing/_paper_engine.py`: paper reference backend.
- `piker/brokers/binance/broker.py`: Binance request and event adapter.
- `piker/brokers/kraken/broker.py`: Kraken request and event adapter.
- `piker/brokers/ib/broker.py`: IB request and event adapter.
- `piker/data/validate.py`: existing datad-side `FeedInit` backend validation
  direction to improve upon rather than duplicate.
- `piker/ui/_remote_ctl.py`: in-tree `pld_spec` and per-direction
  `limit_plds()` precedent.
- `piker/ui/_gaps.py`: in-tree tagged union precedent.
- `tests/test_ems.py`: current network-dependent clearing coverage and empty
  order tests.
- `/home/goodboy/repos/tractor/tests/msg/test_pldrx_limiting.py`: canonical
  payload limiting and `MsgTypeError` behavior.
- `/home/goodboy/repos/tractor/tests/msg/test_ext_types_msgspec.py`: canonical
  msgspec codec behavior.
- `/home/goodboy/repos/tractor/tests/test_context_stream_semantics.py`:
  context stream cancellation and serialization semantics.


Relevant issue/design threads:

- https://github.com/goodboy/tractor/issues/196
- https://github.com/goodboy/tractor/issues/365
- https://github.com/goodboy/tractor/issues/514
- https://github.com/goodboy/tractor/pull/376
- https://github.com/pikers/piker/issues/514
- https://github.com/pikers/piker/issues/297
- https://github.com/pikers/piker/issues/466