piker/.agents/skills/piker-clearing-expert/architecture.md

Clearing And Accounting Architecture

Protocol direction

Client-facing messages and broker-facing messages intentionally differ.

client -> EMS
  Order, Cancel

EMS -> executor
  BrokerdOrder, BrokerdCancel

executor -> EMS
  BrokerdOrderAck, BrokerdStatus, BrokerdFill,
  BrokerdError, BrokerdPosition

EMS -> subscribed clients
  Status, BrokerdPosition

Order carries the client oid, account, market, price, size, action, and execution mode. BrokerdOrderAck establishes the executor’s reqid; subsequent broker messages should carry that ID so EMS can resolve the client dialog.

Status is mutable current dialog state, not a history envelope. Exact client states include pending, open, dark_open, triggered, fill, closed, canceled, and error. Audit existing translation before adding a state; backend BrokerdStatus vocabulary is narrower and current producers are not fully conformant.

EMS routing

_emsd_main() serves each client/market context. The actor-local Router owns books, subscribers, order dialogs, and cached live trades relays. Paper mode opens per-symbol relay contexts under the same broker key, a current overwrite/pop hazard for concurrent paper symbols.

open_brokerd_dialog() is the session live-versus-paper selection point:

• live execution talks to brokerd.<broker>;
• paper execution talks to paperboi.<broker>;
• chart-only and paper sessions should not boot credentialed brokerd.

Do not infer paper execution from each order’s exec_mode. Session mode selects the executor; per-order mode primarily distinguishes dark from immediately routed orders, despite incomplete literal annotations.

process_client_order_cmds() handles submit, modify, cancel, dark, and alert intent. It updates EMS dialog state before routing broker commands. translate_and_relay_brokerd_events() applies acks, statuses, fills, errors, and positions to dialogs and subscribers.

The DarkBook stores local predicates and watches quote ticks. When a predicate triggers, clear_dark_triggers() emits the live broker order and updates the existing client dialog rather than inventing a second order identity.

Ack and cancel ordering

Required actionable ordering is submit, ack, open, zero or more fills, then terminal status. EMS handles cancel-before-ack, but it does not buffer arbitrary status-before-ack events; those currently fall through as unhandled. Backends must ack before actionable statuses.

The important race is:

client Cancel(oid)
  -> EMS has no reqid yet
  -> mark Status.cancel_called
BrokerdOrderAck(oid, reqid)
  -> install mapping
  -> send deferred BrokerdCancel(reqid)

Do not drop a pre-ack cancel and do not send a broker cancel with an oid unless the backend explicitly uses that value as its reqid.

Paper execution

PaperBoi maintains simulated orders, opens the market feed, and fills orders when L1 ask/bid quotes cross their prices. trade/last ticks zip buy and sell iterators, so a lone resting side currently may not fill from those tick types. fake_fill() is the core fill-to-account path:

1. emit BrokerdFill;
2. emit terminal BrokerdStatus for the simulated order;
3. construct a signed normalized Transaction;
4. TransactionLedger.update_from_t();
5. Account.update_from_ledger();
6. persist the ledger and account;
7. emit BrokerdPosition with current size and PPU.

The protocol notifications and persistence occur in one task but are not a database transaction. If persistence fails after fill/status publication, restart/reconciliation behavior matters. Do not reorder the sequence casually.

Partial fills currently reuse oid as tid. A later partial therefore overwrites the paper ledger record and is rejected by position-event dedupe. Fix unique fill IDs before relying on paper partial-fill math.

open_trade_dialog() opens symcache, paper ledger, account, and feed resources under structured concurrency. An exception from any nested resource, including ledger context exit, cancels the dialog and its feed stream.

Live backend variation

There is no universal live-broker accounting implementation:

• IB normalizes execution/commission events, updates ledger/account state, and emits position updates.
• Kraken has backend trade normalization and position reconstruction, but its live path writes account state without inserting every new trade into TransactionLedger.
• Binance currently relays venue order/position events and does not share every local TransactionLedger/Account path.

Inspect the backend’s broker.py, ledger.py, and norm_trade() before moving generic logic. A paper behavior is not automatically a live backend contract.

Ledger normalization

TransactionLedger.data is the durable record mapping. For paper it contains transaction-like dictionaries. For live brokers it can contain raw provider records.

iter_txns() sorts records and chooses normalization:

• paper uses _paper_engine.norm_trade();
• live accounts use brokermod.norm_trade().

Normalizer output should carry datetime, signed size, price, cost, FQME, backend market ID, and transaction ID. Some normalizers return None for unusable records; iter_txns() does not filter it, while to_txns() does. Audit direct iterator consumers.

write_config() sorts records for stable persistence. Paper rewrite logic migrates old symbol fields and qualifies legacy FQMEs through the symcache. Search can return no matches; callers which select the first result need an explicit fallback or a descriptive failure policy.

Position event model

Position holds a market and a transaction-event mapping. New clears are deduplicated by tid. Current size and PPU are recomputed from the event sequence.

Account.update_from_ledger():

1. receives normalized transactions;
2. resolves each market through mktmaps or a fallback table;
3. looks up or creates the position by Transaction.bs_mktid;
4. adds unseen clears;
5. returns visited positions, including duplicates whose clear was rejected by add_clear().

minimized_clears() intends to discard irrelevant history after net-zero or sign transitions, but current slicing is not proven safe and can reconstruct the wrong active size/PPU. Treat it as a known bug surface until sign-transition round trips are covered.

Position.update_from_msg() serves UI summaries by replacing local event state with a synthetic clear representing reported size and PPU. That projection is intentionally lossy.

Symbology contracts

MktPair.fqme is a normalized endpoint address assembled from pair, venue, expiry/contract information, and broker. MktPair.bs_fqme drops the broker suffix. MktPair.bs_mktid is the provider-native market identity.

SymbologyCache.load() attempts to obtain provider assets and pair structs, calls get_mkt_info(), and populates conventional mappings. Assets are optional; missing/empty pair support returns early.

assets[provider_asset_id] -> Asset
pairs[pair.bs_mktid]      -> backend Pair Struct
mktmaps[mkt.fqme]         -> MktPair

Some backends add aliases such as mktmaps[bs_mktid]. Search must return the matched mapping key rather than rebuilding a key from the value, otherwise aliases silently collapse. Current serialization then writes aliases which reload rejects because from_dict() asserts each key equals mkt.fqme.

Persistence boundaries

open_trade_ledger() loads account TOML and compares a shallow snapshot in finally. Top-level additions/replacements trigger writes; nested in-place mutation may not. open_account() similarly mediates account state. These contexts run inside actor tasks, so serialization errors are runtime failures, not merely offline maintenance failures.

When changing schemas:

• preserve backend-native records unless migration is explicit;
• keep old paper field migration deterministic;
• avoid fuzzy qualification when an exact bs_mktid/FQME exists;
• test context exit, not only in-memory updates;
• inspect emitted positions after disk round trips.