175 lines
7.4 KiB
Markdown
175 lines
7.4 KiB
Markdown
---
|
|
name: piker-clearing-expert
|
|
description: >
|
|
Piker clearing, EMS/order-control, accounting, position, and
|
|
trade-ledger expertise. Apply when changing or debugging Order,
|
|
Status, Brokerd* protocols, live or paper fills, dark orders,
|
|
order IDs, TransactionLedger persistence, Account/Position/PPU,
|
|
FQME/bs_mktid identity, SymbologyCache lookups, or broker trade
|
|
normalization.
|
|
compatibility: >
|
|
Requires a piker checkout and familiarity with Python, trio,
|
|
tractor, msgspec Structs, and backend-specific trading APIs.
|
|
metadata:
|
|
author: goodboy
|
|
version: "1.0"
|
|
---
|
|
|
|
# Piker Clearing Expert
|
|
|
|
Use this mental model before touching order control or accounting:
|
|
|
|
```text
|
|
OrderClient / UI
|
|
-> emsd routes intent and owns order-dialog state
|
|
-> live brokerd OR paperboi executes and reports events
|
|
-> executing backend owns fill-to-accounting integration
|
|
-> emsd translates and broadcasts broker events
|
|
-> UI consumes order state and position summaries
|
|
```
|
|
|
|
The EMS is not the trade ledger. It correlates IDs, runs dark
|
|
predicates, and relays lifecycle events. Paper and accounting-aware
|
|
live backends may normalize fills into transactions, update local
|
|
ledger/account state, and emit `BrokerdPosition` summaries. Inspect
|
|
each live backend; this integration is not uniform.
|
|
|
|
## Runtime ownership
|
|
|
|
- `OrderClient` owns client intent and local sent-order tracking.
|
|
- `emsd` owns `Router`, `DarkBook`, active `Status` dialogs,
|
|
subscribers, and `oid <-> reqid` correlation.
|
|
- `brokerd.<broker>` owns credentialed live order control.
|
|
- `paperboi.<broker>` owns simulated execution and paper accounting.
|
|
- `TransactionLedger` owns trade-record persistence, not order state.
|
|
- `Account` owns positions keyed by backend-system market identity.
|
|
- UI `Position.update_from_msg()` is a summary reset, not durable
|
|
transaction accounting.
|
|
|
|
For daemon placement, context streams, cancellation, and actor-local
|
|
state, also load `piker-conc-expert`.
|
|
|
|
## Event lifecycle
|
|
|
|
1. A client sends `Order` or `Cancel` through `OrderClient`.
|
|
2. `process_client_order_cmds()` creates or updates EMS `Status`.
|
|
3. EMS routes a `BrokerdOrder` or stores a dark trigger predicate.
|
|
4. The executing backend acknowledges with `BrokerdOrderAck`.
|
|
5. EMS records the current `oid <-> reqid` relation.
|
|
6. Accepted, ack-correlated status/fill events become client updates.
|
|
7. The backend integrates fills into accounting when supported.
|
|
8. A resulting `BrokerdPosition` is relayed through EMS to clients.
|
|
|
|
Cancellation may arrive before acknowledgement. Preserve
|
|
`Status.cancel_called`; once the ack supplies `reqid`, EMS can issue
|
|
the deferred `BrokerdCancel`.
|
|
|
|
## Identifier namespaces
|
|
|
|
Never substitute one namespace merely because two values happen to
|
|
match in a backend:
|
|
|
|
| ID | Owner and invariant |
|
|
|---|---|
|
|
| `oid` | Stable client/EMS order-dialog ID across submit, modify, cancel |
|
|
| `reqid` | Backend order-control ID established by an ack; it may change |
|
|
| venue order ID | Native API identity; may differ from both IDs above |
|
|
| `tid` | Stable clear ID used to deduplicate position events and TOML keys |
|
|
| `fqme` | Piker-normalized market address and symcache lookup key |
|
|
| `bs_fqme` | FQME without its final broker suffix |
|
|
| `bs_mktid` | Backend-system market key and `Account.pps` key |
|
|
|
|
Normalize every durably persisted `tid` to a stable string. Although
|
|
`Transaction.tid` permits integers, TOML mapping keys do not.
|
|
|
|
Paper transactions commonly set `Transaction.bs_mktid` to the full
|
|
FQME. The embedded `Position.mkt.bs_mktid` can still retain the
|
|
provider-native market ID. Check the concrete object and table owner
|
|
before asserting key equality.
|
|
|
|
## Accounting invariants
|
|
|
|
- `Transaction.size` is signed; preserve side semantics during
|
|
broker normalization.
|
|
- `Transaction` is the normalized in-memory interchange. Live ledger
|
|
records may remain backend-native and normalize through
|
|
`mod.norm_trade()` when read.
|
|
- `Position._events` is keyed by `tid`; `add_clear()` is the
|
|
idempotence boundary.
|
|
- `Position.ppu` and `Position.cumsize` are derived from transaction
|
|
events, not independent mutable truth.
|
|
- Normalized accounting transactions need a stable, non-null
|
|
`bs_mktid`; `Account.pps` uses it as the position key.
|
|
- `Account.update_from_ledger()` resolves market metadata from
|
|
`SymbologyCache.mktmaps` or its explicit fallback table.
|
|
- UI position messages are projections. Do not feed their synthetic
|
|
reset event back into durable broker accounting.
|
|
|
|
## Persistence contracts
|
|
|
|
Trade ledgers live under the accounting ledger directory as
|
|
`trades_<broker>_<account>.toml`. Top-level keys are transaction IDs;
|
|
record schemas are backend-specific unless the paper engine wrote a
|
|
normalized transaction dictionary.
|
|
|
|
`open_trade_ledger()` writes on context exit when its data comparison
|
|
detects a change or `rewrite=True`. Its snapshot is shallow, so nested
|
|
in-place record mutation can evade dirty detection. Explicit writes can
|
|
also fail inside fill handlers. Treat all persistence points as part of
|
|
the live execution path.
|
|
|
|
Accounts persist active positions. `dump_active()` separates open and
|
|
net-zero positions. Do not assume `minimized_clears()` safely preserves
|
|
size/PPU: current sign-transition and longer net-zero slicing lacks
|
|
round-trip coverage and can retain the wrong event subset.
|
|
|
|
## Market identity and symcache
|
|
|
|
`SymbologyCache` has three distinct key-addressed mappings:
|
|
|
|
- `assets`: provider asset ID to `Asset`;
|
|
- `pairs`: native `bs_mktid` to backend pair `Struct`;
|
|
- `mktmaps`: searchable FQME or explicit alias to normalized
|
|
`MktPair`.
|
|
|
|
Search mapping keys, then resolve values. RapidFuzz treats mapping
|
|
values as choices; passing `mktmaps` directly makes it operate on
|
|
`MktPair` objects. Preserve in-memory aliases because IB can store both
|
|
native IDs and FQMEs for one `MktPair`. Do not persist such aliases
|
|
without fixing reload: `from_dict()` currently requires every key to
|
|
equal `mkt.fqme`.
|
|
|
|
## Change workflow
|
|
|
|
Before editing:
|
|
|
|
1. Identify the owning layer: client, EMS, executor, accounting, or UI.
|
|
2. Write down every ID/key transition across that boundary.
|
|
3. Trace both normal and cancel-before-ack event ordering.
|
|
4. Determine whether records are native, normalized, or summaries.
|
|
5. Check actor-local caches and context-exit writes.
|
|
6. Add a deterministic regression at the narrowest broken contract.
|
|
7. Run live/backend tests only when their credentials and side effects
|
|
are explicitly authorized.
|
|
|
|
Do not fix an accounting defect by adding state to EMS, or fix an EMS
|
|
correlation defect by rewriting durable transaction history.
|
|
|
|
## Canonical source map
|
|
|
|
- `piker/clearing/_messages.py`: client and brokerd protocol types.
|
|
- `piker/clearing/_client.py`: `OrderClient` and `open_ems()`.
|
|
- `piker/clearing/_ems.py`: router, dark book, ID mapping, translation.
|
|
- `piker/clearing/_paper_engine.py`: simulated execution/accounting.
|
|
- `piker/accounting/_ledger.py`: transactions and ledger persistence.
|
|
- `piker/accounting/_pos.py`: positions, accounts, PPU event state.
|
|
- `piker/accounting/_mktinfo.py`: `Asset`, `MktPair`, FQME schema.
|
|
- `piker/data/_symcache.py`: provider symbology mappings and search.
|
|
- `piker/accounting/calc.py`: transaction ordering and PPU/PnL math.
|
|
- `piker/brokers/*/broker.py`: backend order-control integration.
|
|
- `piker/brokers/*/ledger.py`: backend trade normalization.
|
|
|
|
See [architecture.md](architecture.md) for complete flows,
|
|
[gotchas.md](gotchas.md) for symptom-driven diagnosis, and
|
|
[test-map.md](test-map.md) for verification choices.
|