piker/.agents/skills/piker-clearing-expert/architecture.md

# Clearing And Accounting Architecture

## Protocol direction

Client-facing messages and broker-facing messages intentionally differ.

```text
client -> EMS
  Order, Cancel

EMS -> executor
  BrokerdOrder, BrokerdCancel

executor -> EMS
  BrokerdOrderAck, BrokerdStatus, BrokerdFill,
  BrokerdError, BrokerdPosition

EMS -> subscribed clients
  Status, BrokerdPosition
```

`Order` carries the client `oid`, account, market, price, size, action,
and execution mode. `BrokerdOrderAck` establishes the executor's
`reqid`; subsequent broker messages should carry that ID so EMS can
resolve the client dialog.

`Status` is mutable current dialog state, not a history envelope. Exact
client states include `pending`, `open`, `dark_open`, `triggered`,
`fill`, `closed`, `canceled`, and `error`. Audit existing translation
before adding a state; backend `BrokerdStatus` vocabulary is narrower
and current producers are not fully conformant.

## EMS routing

`_emsd_main()` serves each client/market context. The actor-local
`Router` owns books, subscribers, order dialogs, and cached live trades
relays. Paper mode opens per-symbol relay contexts under the same broker
key, a current overwrite/pop hazard for concurrent paper symbols.

`open_brokerd_dialog()` is the session live-versus-paper selection
point:

- live execution talks to `brokerd.<broker>`;
- paper execution talks to `paperboi.<broker>`;
- chart-only and paper sessions should not boot credentialed brokerd.

Do not infer paper execution from each order's `exec_mode`. Session
mode selects the executor; per-order mode primarily distinguishes dark
from immediately routed orders, despite incomplete literal annotations.

`process_client_order_cmds()` handles submit, modify, cancel, dark,
and alert intent. It updates EMS dialog state before routing broker
commands. `translate_and_relay_brokerd_events()` applies acks, statuses,
fills, errors, and positions to dialogs and subscribers.

The `DarkBook` stores local predicates and watches quote ticks. When a
predicate triggers, `clear_dark_triggers()` emits the live broker order
and updates the existing client dialog rather than inventing a second
order identity.

## Ack and cancel ordering

Required actionable ordering is submit, ack, open, zero or more fills,
then terminal status. EMS handles cancel-before-ack, but it does not
buffer arbitrary status-before-ack events; those currently fall through
as unhandled. Backends must ack before actionable statuses.

The important race is:

```text
client Cancel(oid)
  -> EMS has no reqid yet
  -> mark Status.cancel_called
BrokerdOrderAck(oid, reqid)
  -> install mapping
  -> send deferred BrokerdCancel(reqid)
```

Do not drop a pre-ack cancel and do not send a broker cancel with an
`oid` unless the backend explicitly uses that value as its `reqid`.

## Paper execution

`PaperBoi` maintains simulated orders, opens the market feed, and fills
orders when L1 ask/bid quotes cross their prices. `trade`/`last` ticks
zip buy and sell iterators, so a lone resting side currently may not
fill from those tick types. `fake_fill()` is the core fill-to-account
path:

1. emit `BrokerdFill`;
2. emit terminal `BrokerdStatus` for the simulated order;
3. construct a signed normalized `Transaction`;
4. `TransactionLedger.update_from_t()`;
5. `Account.update_from_ledger()`;
6. persist the ledger and account;
7. emit `BrokerdPosition` with current size and PPU.

The protocol notifications and persistence occur in one task but are
not a database transaction. If persistence fails after fill/status
publication, restart/reconciliation behavior matters. Do not reorder
the sequence casually.

Partial fills currently reuse `oid` as `tid`. A later partial therefore
overwrites the paper ledger record and is rejected by position-event
dedupe. Fix unique fill IDs before relying on paper partial-fill math.

`open_trade_dialog()` opens symcache, paper ledger, account, and feed
resources under structured concurrency. An exception from any nested
resource, including ledger context exit, cancels the dialog and its feed
stream.

## Live backend variation

There is no universal live-broker accounting implementation:

- IB normalizes execution/commission events, updates ledger/account
  state, and emits position updates.
- Kraken has backend trade normalization and position reconstruction,
  but its live path writes account state without inserting every new
  trade into `TransactionLedger`.
- Binance currently relays venue order/position events and does not
  share every local `TransactionLedger`/`Account` path.

Inspect the backend's `broker.py`, `ledger.py`, and `norm_trade()` before
moving generic logic. A paper behavior is not automatically a live
backend contract.

## Ledger normalization

`TransactionLedger.data` is the durable record mapping. For paper it
contains transaction-like dictionaries. For live brokers it can contain
raw provider records.

`iter_txns()` sorts records and chooses normalization:

- paper uses `_paper_engine.norm_trade()`;
- live accounts use `brokermod.norm_trade()`.

Normalizer output should carry datetime, signed size, price, cost,
FQME, backend market ID, and transaction ID. Some normalizers return
`None` for unusable records; `iter_txns()` does not filter it, while
`to_txns()` does. Audit direct iterator consumers.

`write_config()` sorts records for stable persistence. Paper rewrite
logic migrates old symbol fields and qualifies legacy FQMEs through the
symcache. Search can return no matches; callers which select the first
result need an explicit fallback or a descriptive failure policy.

## Position event model

`Position` holds a market and a transaction-event mapping. New clears
are deduplicated by `tid`. Current size and PPU are recomputed from the
event sequence.

`Account.update_from_ledger()`:

1. receives normalized transactions;
2. resolves each market through `mktmaps` or a fallback table;
3. looks up or creates the position by `Transaction.bs_mktid`;
4. adds unseen clears;
5. returns visited positions, including duplicates whose clear was
   rejected by `add_clear()`.

`minimized_clears()` intends to discard irrelevant history after
net-zero or sign transitions, but current slicing is not proven safe
and can reconstruct the wrong active size/PPU. Treat it as a known bug
surface until sign-transition round trips are covered.

`Position.update_from_msg()` serves UI summaries by replacing local
event state with a synthetic clear representing reported size and PPU.
That projection is intentionally lossy.

## Symbology contracts

`MktPair.fqme` is a normalized endpoint address assembled from pair,
venue, expiry/contract information, and broker. `MktPair.bs_fqme`
drops the broker suffix. `MktPair.bs_mktid` is the provider-native
market identity.

`SymbologyCache.load()` attempts to obtain provider assets and pair
structs, calls `get_mkt_info()`, and populates conventional mappings.
Assets are optional; missing/empty pair support returns early.

```text
assets[provider_asset_id] -> Asset
pairs[pair.bs_mktid]      -> backend Pair Struct
mktmaps[mkt.fqme]         -> MktPair
```

Some backends add aliases such as `mktmaps[bs_mktid]`. Search must
return the matched mapping key rather than rebuilding a key from the
value, otherwise aliases silently collapse. Current serialization then
writes aliases which reload rejects because `from_dict()` asserts each
key equals `mkt.fqme`.

## Persistence boundaries

`open_trade_ledger()` loads account TOML and compares a shallow snapshot
in `finally`. Top-level additions/replacements trigger writes; nested
in-place mutation may not. `open_account()` similarly mediates account
state. These contexts run inside actor tasks, so serialization errors
are runtime failures, not merely offline maintenance failures.

When changing schemas:

- preserve backend-native records unless migration is explicit;
- keep old paper field migration deterministic;
- avoid fuzzy qualification when an exact `bs_mktid`/FQME exists;
- test context exit, not only in-memory updates;
- inspect emitted positions after disk round trips.