211 lines
7.9 KiB
Markdown
211 lines
7.9 KiB
Markdown
# Clearing And Accounting Architecture
|
|
|
|
## Protocol direction
|
|
|
|
Client-facing messages and broker-facing messages intentionally differ.
|
|
|
|
```text
|
|
client -> EMS
|
|
Order, Cancel
|
|
|
|
EMS -> executor
|
|
BrokerdOrder, BrokerdCancel
|
|
|
|
executor -> EMS
|
|
BrokerdOrderAck, BrokerdStatus, BrokerdFill,
|
|
BrokerdError, BrokerdPosition
|
|
|
|
EMS -> subscribed clients
|
|
Status, BrokerdPosition
|
|
```
|
|
|
|
`Order` carries the client `oid`, account, market, price, size, action,
|
|
and execution mode. `BrokerdOrderAck` establishes the executor's
|
|
`reqid`; subsequent broker messages should carry that ID so EMS can
|
|
resolve the client dialog.
|
|
|
|
`Status` is mutable current dialog state, not a history envelope. Exact
|
|
client states include `pending`, `open`, `dark_open`, `triggered`,
|
|
`fill`, `closed`, `canceled`, and `error`. Audit existing translation
|
|
before adding a state; backend `BrokerdStatus` vocabulary is narrower
|
|
and current producers are not fully conformant.
|
|
|
|
## EMS routing
|
|
|
|
`_emsd_main()` serves each client/market context. The actor-local
|
|
`Router` owns books, subscribers, order dialogs, and cached live trades
|
|
relays. Paper mode opens per-symbol relay contexts under the same broker
|
|
key, a current overwrite/pop hazard for concurrent paper symbols.
|
|
|
|
`open_brokerd_dialog()` is the session live-versus-paper selection
|
|
point:
|
|
|
|
- live execution talks to `brokerd.<broker>`;
|
|
- paper execution talks to `paperboi.<broker>`;
|
|
- chart-only and paper sessions should not boot credentialed brokerd.
|
|
|
|
Do not infer paper execution from each order's `exec_mode`. Session
|
|
mode selects the executor; per-order mode primarily distinguishes dark
|
|
from immediately routed orders, despite incomplete literal annotations.
|
|
|
|
`process_client_order_cmds()` handles submit, modify, cancel, dark,
|
|
and alert intent. It updates EMS dialog state before routing broker
|
|
commands. `translate_and_relay_brokerd_events()` applies acks, statuses,
|
|
fills, errors, and positions to dialogs and subscribers.
|
|
|
|
The `DarkBook` stores local predicates and watches quote ticks. When a
|
|
predicate triggers, `clear_dark_triggers()` emits the live broker order
|
|
and updates the existing client dialog rather than inventing a second
|
|
order identity.
|
|
|
|
## Ack and cancel ordering
|
|
|
|
Required actionable ordering is submit, ack, open, zero or more fills,
|
|
then terminal status. EMS handles cancel-before-ack, but it does not
|
|
buffer arbitrary status-before-ack events; those currently fall through
|
|
as unhandled. Backends must ack before actionable statuses.
|
|
|
|
The important race is:
|
|
|
|
```text
|
|
client Cancel(oid)
|
|
-> EMS has no reqid yet
|
|
-> mark Status.cancel_called
|
|
BrokerdOrderAck(oid, reqid)
|
|
-> install mapping
|
|
-> send deferred BrokerdCancel(reqid)
|
|
```
|
|
|
|
Do not drop a pre-ack cancel and do not send a broker cancel with an
|
|
`oid` unless the backend explicitly uses that value as its `reqid`.
|
|
|
|
## Paper execution
|
|
|
|
`PaperBoi` maintains simulated orders, opens the market feed, and fills
|
|
orders when L1 ask/bid quotes cross their prices. `trade`/`last` ticks
|
|
zip buy and sell iterators, so a lone resting side currently may not
|
|
fill from those tick types. `fake_fill()` is the core fill-to-account
|
|
path:
|
|
|
|
1. emit `BrokerdFill`;
|
|
2. emit terminal `BrokerdStatus` for the simulated order;
|
|
3. construct a signed normalized `Transaction`;
|
|
4. `TransactionLedger.update_from_t()`;
|
|
5. `Account.update_from_ledger()`;
|
|
6. persist the ledger and account;
|
|
7. emit `BrokerdPosition` with current size and PPU.
|
|
|
|
The protocol notifications and persistence occur in one task but are
|
|
not a database transaction. If persistence fails after fill/status
|
|
publication, restart/reconciliation behavior matters. Do not reorder
|
|
the sequence casually.
|
|
|
|
Partial fills currently reuse `oid` as `tid`. A later partial therefore
|
|
overwrites the paper ledger record and is rejected by position-event
|
|
dedupe. Fix unique fill IDs before relying on paper partial-fill math.
|
|
|
|
`open_trade_dialog()` opens symcache, paper ledger, account, and feed
|
|
resources under structured concurrency. An exception from any nested
|
|
resource, including ledger context exit, cancels the dialog and its feed
|
|
stream.
|
|
|
|
## Live backend variation
|
|
|
|
There is no universal live-broker accounting implementation:
|
|
|
|
- IB normalizes execution/commission events, updates ledger/account
|
|
state, and emits position updates.
|
|
- Kraken has backend trade normalization and position reconstruction,
|
|
but its live path writes account state without inserting every new
|
|
trade into `TransactionLedger`.
|
|
- Binance currently relays venue order/position events and does not
|
|
share every local `TransactionLedger`/`Account` path.
|
|
|
|
Inspect the backend's `broker.py`, `ledger.py`, and `norm_trade()` before
|
|
moving generic logic. A paper behavior is not automatically a live
|
|
backend contract.
|
|
|
|
## Ledger normalization
|
|
|
|
`TransactionLedger.data` is the durable record mapping. For paper it
|
|
contains transaction-like dictionaries. For live brokers it can contain
|
|
raw provider records.
|
|
|
|
`iter_txns()` sorts records and chooses normalization:
|
|
|
|
- paper uses `_paper_engine.norm_trade()`;
|
|
- live accounts use `brokermod.norm_trade()`.
|
|
|
|
Normalizer output should carry datetime, signed size, price, cost,
|
|
FQME, backend market ID, and transaction ID. Some normalizers return
|
|
`None` for unusable records; `iter_txns()` does not filter it, while
|
|
`to_txns()` does. Audit direct iterator consumers.
|
|
|
|
`write_config()` sorts records for stable persistence. Paper rewrite
|
|
logic migrates old symbol fields and qualifies legacy FQMEs through the
|
|
symcache. Search can return no matches; callers which select the first
|
|
result need an explicit fallback or a descriptive failure policy.
|
|
|
|
## Position event model
|
|
|
|
`Position` holds a market and a transaction-event mapping. New clears
|
|
are deduplicated by `tid`. Current size and PPU are recomputed from the
|
|
event sequence.
|
|
|
|
`Account.update_from_ledger()`:
|
|
|
|
1. receives normalized transactions;
|
|
2. resolves each market through `mktmaps` or a fallback table;
|
|
3. looks up or creates the position by `Transaction.bs_mktid`;
|
|
4. adds unseen clears;
|
|
5. returns visited positions, including duplicates whose clear was
|
|
rejected by `add_clear()`.
|
|
|
|
`minimized_clears()` intends to discard irrelevant history after
|
|
net-zero or sign transitions, but current slicing is not proven safe
|
|
and can reconstruct the wrong active size/PPU. Treat it as a known bug
|
|
surface until sign-transition round trips are covered.
|
|
|
|
`Position.update_from_msg()` serves UI summaries by replacing local
|
|
event state with a synthetic clear representing reported size and PPU.
|
|
That projection is intentionally lossy.
|
|
|
|
## Symbology contracts
|
|
|
|
`MktPair.fqme` is a normalized endpoint address assembled from pair,
|
|
venue, expiry/contract information, and broker. `MktPair.bs_fqme`
|
|
drops the broker suffix. `MktPair.bs_mktid` is the provider-native
|
|
market identity.
|
|
|
|
`SymbologyCache.load()` attempts to obtain provider assets and pair
|
|
structs, calls `get_mkt_info()`, and populates conventional mappings.
|
|
Assets are optional; missing/empty pair support returns early.
|
|
|
|
```text
|
|
assets[provider_asset_id] -> Asset
|
|
pairs[pair.bs_mktid] -> backend Pair Struct
|
|
mktmaps[mkt.fqme] -> MktPair
|
|
```
|
|
|
|
Some backends add aliases such as `mktmaps[bs_mktid]`. Search must
|
|
return the matched mapping key rather than rebuilding a key from the
|
|
value, otherwise aliases silently collapse. Current serialization then
|
|
writes aliases which reload rejects because `from_dict()` asserts each
|
|
key equals `mkt.fqme`.
|
|
|
|
## Persistence boundaries
|
|
|
|
`open_trade_ledger()` loads account TOML and compares a shallow snapshot
|
|
in `finally`. Top-level additions/replacements trigger writes; nested
|
|
in-place mutation may not. `open_account()` similarly mediates account
|
|
state. These contexts run inside actor tasks, so serialization errors
|
|
are runtime failures, not merely offline maintenance failures.
|
|
|
|
When changing schemas:
|
|
|
|
- preserve backend-native records unless migration is explicit;
|
|
- keep old paper field migration deterministic;
|
|
- avoid fuzzy qualification when an exact `bs_mktid`/FQME exists;
|
|
- test context exit, not only in-memory updates;
|
|
- inspect emitted positions after disk round trips.
|