---
name: piker-clearing-expert
description: >
  Piker clearing, EMS/order-control, accounting, position, and
  trade-ledger expertise. Apply when changing or debugging Order,
  Status, Brokerd* protocols, live or paper fills, dark orders,
  order IDs, TransactionLedger persistence, Account/Position/PPU,
  FQME/bs_mktid identity, SymbologyCache lookups, or broker trade
  normalization.
compatibility: >
  Requires a piker checkout and familiarity with Python, trio,
  tractor, msgspec Structs, and backend-specific trading APIs.
metadata:
  author: goodboy
  version: "1.0"
---

# Piker Clearing Expert

Use this mental model before touching order control or accounting:

```text
OrderClient / UI
  -> emsd routes intent and owns order-dialog state
  -> live brokerd OR paperboi executes and reports events
  -> executing backend owns fill-to-accounting integration
  -> emsd translates and broadcasts broker events
  -> UI consumes order state and position summaries
```

The EMS is not the trade ledger. It correlates IDs, runs dark
predicates, and relays lifecycle events. Paper and accounting-aware
live backends may normalize fills into transactions, update local
ledger/account state, and emit `BrokerdPosition` summaries. Inspect
each live backend; this integration is not uniform.

## Runtime ownership

- `OrderClient` owns client intent and local sent-order tracking.
- `emsd` owns `Router`, `DarkBook`, active `Status` dialogs,
  subscribers, and `oid <-> reqid` correlation.
- `brokerd.<broker>` owns credentialed live order control.
- `paperboi.<broker>` owns simulated execution and paper accounting.
- `TransactionLedger` owns trade-record persistence, not order state.
- `Account` owns positions keyed by backend-system market identity.
- UI `Position.update_from_msg()` is a summary reset, not durable
  transaction accounting.

For daemon placement, context streams, cancellation, and actor-local
state, also load `piker-conc-expert`.

## Event lifecycle

1. A client sends `Order` or `Cancel` through `OrderClient`.
2. `process_client_order_cmds()` creates or updates EMS `Status`.
3. EMS routes a `BrokerdOrder` or stores a dark trigger predicate.
4. The executing backend acknowledges with `BrokerdOrderAck`.
5. EMS records the current `oid <-> reqid` relation.
6. Accepted, ack-correlated status/fill events become client updates.
7. The backend integrates fills into accounting when supported.
8. A resulting `BrokerdPosition` is relayed through EMS to clients.

Cancellation may arrive before acknowledgement. Preserve
`Status.cancel_called`; once the ack supplies `reqid`, EMS can issue
the deferred `BrokerdCancel`.

## Identifier namespaces

Never substitute one namespace merely because two values happen to
match in a backend:

| ID | Owner and invariant |
|---|---|
| `oid` | Stable client/EMS order-dialog ID across submit, modify, cancel |
| `reqid` | Backend order-control ID established by an ack; it may change |
| venue order ID | Native API identity; may differ from both IDs above |
| `tid` | Stable clear ID used to deduplicate position events and TOML keys |
| `fqme` | Piker-normalized market address and symcache lookup key |
| `bs_fqme` | FQME without its final broker suffix |
| `bs_mktid` | Backend-system market key and `Account.pps` key |

Normalize every durably persisted `tid` to a stable string. Although
`Transaction.tid` permits integers, TOML mapping keys do not.

Paper transactions commonly set `Transaction.bs_mktid` to the full
FQME. The embedded `Position.mkt.bs_mktid` can still retain the
provider-native market ID. Check the concrete object and table owner
before asserting key equality.

## Accounting invariants

- `Transaction.size` is signed; preserve side semantics during
  broker normalization.
- `Transaction` is the normalized in-memory interchange. Live ledger
  records may remain backend-native and normalize through
  `mod.norm_trade()` when read.
- `Position._events` is keyed by `tid`; `add_clear()` is the
  idempotence boundary.
- `Position.ppu` and `Position.cumsize` are derived from transaction
  events, not independent mutable truth.
- Normalized accounting transactions need a stable, non-null
  `bs_mktid`; `Account.pps` uses it as the position key.
- `Account.update_from_ledger()` resolves market metadata from
  `SymbologyCache.mktmaps` or its explicit fallback table.
- UI position messages are projections. Do not feed their synthetic
  reset event back into durable broker accounting.

## Persistence contracts

Trade ledgers live under the accounting ledger directory as
`trades_<broker>_<account>.toml`. Top-level keys are transaction IDs;
record schemas are backend-specific unless the paper engine wrote a
normalized transaction dictionary.

`open_trade_ledger()` writes on context exit when its data comparison
detects a change or `rewrite=True`. Its snapshot is shallow, so nested
in-place record mutation can evade dirty detection. Explicit writes can
also fail inside fill handlers. Treat all persistence points as part of
the live execution path.

Accounts persist active positions. `dump_active()` separates open and
net-zero positions. Do not assume `minimized_clears()` safely preserves
size/PPU: current sign-transition and longer net-zero slicing lacks
round-trip coverage and can retain the wrong event subset.

## Market identity and symcache

`SymbologyCache` has three distinct key-addressed mappings:

- `assets`: provider asset ID to `Asset`;
- `pairs`: native `bs_mktid` to backend pair `Struct`;
- `mktmaps`: searchable FQME or explicit alias to normalized
  `MktPair`.

Search mapping keys, then resolve values. RapidFuzz treats mapping
values as choices; passing `mktmaps` directly makes it operate on
`MktPair` objects. Preserve in-memory aliases because IB can store both
native IDs and FQMEs for one `MktPair`. Do not persist such aliases
without fixing reload: `from_dict()` currently requires every key to
equal `mkt.fqme`.

## Change workflow

Before editing:

1. Identify the owning layer: client, EMS, executor, accounting, or UI.
2. Write down every ID/key transition across that boundary.
3. Trace both normal and cancel-before-ack event ordering.
4. Determine whether records are native, normalized, or summaries.
5. Check actor-local caches and context-exit writes.
6. Add a deterministic regression at the narrowest broken contract.
7. Run live/backend tests only when their credentials and side effects
   are explicitly authorized.

Do not fix an accounting defect by adding state to EMS, or fix an EMS
correlation defect by rewriting durable transaction history.

## Canonical source map

- `piker/clearing/_messages.py`: client and brokerd protocol types.
- `piker/clearing/_client.py`: `OrderClient` and `open_ems()`.
- `piker/clearing/_ems.py`: router, dark book, ID mapping, translation.
- `piker/clearing/_paper_engine.py`: simulated execution/accounting.
- `piker/accounting/_ledger.py`: transactions and ledger persistence.
- `piker/accounting/_pos.py`: positions, accounts, PPU event state.
- `piker/accounting/_mktinfo.py`: `Asset`, `MktPair`, FQME schema.
- `piker/data/_symcache.py`: provider symbology mappings and search.
- `piker/accounting/calc.py`: transaction ordering and PPU/PnL math.
- `piker/brokers/*/broker.py`: backend order-control integration.
- `piker/brokers/*/ledger.py`: backend trade normalization.

See [architecture.md](architecture.md) for complete flows,
[gotchas.md](gotchas.md) for symptom-driven diagnosis, and
[test-map.md](test-map.md) for verification choices.