Add `piker-clearing-expert` skill
Capture EMS/order-control ownership, identifier boundaries, ledger and position invariants, backend variation, and current known hazards. Store the canonical bundle under `.agents/skills/` and bridge Claude discovery through a portable relative link. Also, - add architecture, debugging, and test-map references; - document the provider-neutral source convention; - ignore the canonical `wkts/` runtime root. Prompt-IO: ai/prompt-io/opencode/20260828T021411Z_6ca6ab2a_prompt_io.md (this patch was generated in some part by `opencode` using `gpt-5.6-sol` (`openai`))wkt/fsp_backfill_sync
parent
7b51927f6b
commit
1606122b45
|
|
@ -0,0 +1,174 @@
|
||||||
|
---
|
||||||
|
name: piker-clearing-expert
|
||||||
|
description: >
|
||||||
|
Piker clearing, EMS/order-control, accounting, position, and
|
||||||
|
trade-ledger expertise. Apply when changing or debugging Order,
|
||||||
|
Status, Brokerd* protocols, live or paper fills, dark orders,
|
||||||
|
order IDs, TransactionLedger persistence, Account/Position/PPU,
|
||||||
|
FQME/bs_mktid identity, SymbologyCache lookups, or broker trade
|
||||||
|
normalization.
|
||||||
|
compatibility: >
|
||||||
|
Requires a piker checkout and familiarity with Python, trio,
|
||||||
|
tractor, msgspec Structs, and backend-specific trading APIs.
|
||||||
|
metadata:
|
||||||
|
author: goodboy
|
||||||
|
version: "1.0"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Piker Clearing Expert
|
||||||
|
|
||||||
|
Use this mental model before touching order control or accounting:
|
||||||
|
|
||||||
|
```text
|
||||||
|
OrderClient / UI
|
||||||
|
-> emsd routes intent and owns order-dialog state
|
||||||
|
-> live brokerd OR paperboi executes and reports events
|
||||||
|
-> executing backend owns fill-to-accounting integration
|
||||||
|
-> emsd translates and broadcasts broker events
|
||||||
|
-> UI consumes order state and position summaries
|
||||||
|
```
|
||||||
|
|
||||||
|
The EMS is not the trade ledger. It correlates IDs, runs dark
|
||||||
|
predicates, and relays lifecycle events. Paper and accounting-aware
|
||||||
|
live backends may normalize fills into transactions, update local
|
||||||
|
ledger/account state, and emit `BrokerdPosition` summaries. Inspect
|
||||||
|
each live backend; this integration is not uniform.
|
||||||
|
|
||||||
|
## Runtime ownership
|
||||||
|
|
||||||
|
- `OrderClient` owns client intent and local sent-order tracking.
|
||||||
|
- `emsd` owns `Router`, `DarkBook`, active `Status` dialogs,
|
||||||
|
subscribers, and `oid <-> reqid` correlation.
|
||||||
|
- `brokerd.<broker>` owns credentialed live order control.
|
||||||
|
- `paperboi.<broker>` owns simulated execution and paper accounting.
|
||||||
|
- `TransactionLedger` owns trade-record persistence, not order state.
|
||||||
|
- `Account` owns positions keyed by backend-system market identity.
|
||||||
|
- UI `Position.update_from_msg()` is a summary reset, not durable
|
||||||
|
transaction accounting.
|
||||||
|
|
||||||
|
For daemon placement, context streams, cancellation, and actor-local
|
||||||
|
state, also load `piker-conc-expert`.
|
||||||
|
|
||||||
|
## Event lifecycle
|
||||||
|
|
||||||
|
1. A client sends `Order` or `Cancel` through `OrderClient`.
|
||||||
|
2. `process_client_order_cmds()` creates or updates EMS `Status`.
|
||||||
|
3. EMS routes a `BrokerdOrder` or stores a dark trigger predicate.
|
||||||
|
4. The executing backend acknowledges with `BrokerdOrderAck`.
|
||||||
|
5. EMS records the current `oid <-> reqid` relation.
|
||||||
|
6. Accepted, ack-correlated status/fill events become client updates.
|
||||||
|
7. The backend integrates fills into accounting when supported.
|
||||||
|
8. A resulting `BrokerdPosition` is relayed through EMS to clients.
|
||||||
|
|
||||||
|
Cancellation may arrive before acknowledgement. Preserve
|
||||||
|
`Status.cancel_called`; once the ack supplies `reqid`, EMS can issue
|
||||||
|
the deferred `BrokerdCancel`.
|
||||||
|
|
||||||
|
## Identifier namespaces
|
||||||
|
|
||||||
|
Never substitute one namespace merely because two values happen to
|
||||||
|
match in a backend:
|
||||||
|
|
||||||
|
| ID | Owner and invariant |
|
||||||
|
|---|---|
|
||||||
|
| `oid` | Stable client/EMS order-dialog ID across submit, modify, cancel |
|
||||||
|
| `reqid` | Backend order-control ID established by an ack; it may change |
|
||||||
|
| venue order ID | Native API identity; may differ from both IDs above |
|
||||||
|
| `tid` | Stable clear ID used to deduplicate position events and TOML keys |
|
||||||
|
| `fqme` | Piker-normalized market address and symcache lookup key |
|
||||||
|
| `bs_fqme` | FQME without its final broker suffix |
|
||||||
|
| `bs_mktid` | Backend-system market key and `Account.pps` key |
|
||||||
|
|
||||||
|
Normalize every durably persisted `tid` to a stable string. Although
|
||||||
|
`Transaction.tid` permits integers, TOML mapping keys do not.
|
||||||
|
|
||||||
|
Paper transactions commonly set `Transaction.bs_mktid` to the full
|
||||||
|
FQME. The embedded `Position.mkt.bs_mktid` can still retain the
|
||||||
|
provider-native market ID. Check the concrete object and table owner
|
||||||
|
before asserting key equality.
|
||||||
|
|
||||||
|
## Accounting invariants
|
||||||
|
|
||||||
|
- `Transaction.size` is signed; preserve side semantics during
|
||||||
|
broker normalization.
|
||||||
|
- `Transaction` is the normalized in-memory interchange. Live ledger
|
||||||
|
records may remain backend-native and normalize through
|
||||||
|
`mod.norm_trade()` when read.
|
||||||
|
- `Position._events` is keyed by `tid`; `add_clear()` is the
|
||||||
|
idempotence boundary.
|
||||||
|
- `Position.ppu` and `Position.cumsize` are derived from transaction
|
||||||
|
events, not independent mutable truth.
|
||||||
|
- Normalized accounting transactions need a stable, non-null
|
||||||
|
`bs_mktid`; `Account.pps` uses it as the position key.
|
||||||
|
- `Account.update_from_ledger()` resolves market metadata from
|
||||||
|
`SymbologyCache.mktmaps` or its explicit fallback table.
|
||||||
|
- UI position messages are projections. Do not feed their synthetic
|
||||||
|
reset event back into durable broker accounting.
|
||||||
|
|
||||||
|
## Persistence contracts
|
||||||
|
|
||||||
|
Trade ledgers live under the accounting ledger directory as
|
||||||
|
`trades_<broker>_<account>.toml`. Top-level keys are transaction IDs;
|
||||||
|
record schemas are backend-specific unless the paper engine wrote a
|
||||||
|
normalized transaction dictionary.
|
||||||
|
|
||||||
|
`open_trade_ledger()` writes on context exit when its data comparison
|
||||||
|
detects a change or `rewrite=True`. Its snapshot is shallow, so nested
|
||||||
|
in-place record mutation can evade dirty detection. Explicit writes can
|
||||||
|
also fail inside fill handlers. Treat all persistence points as part of
|
||||||
|
the live execution path.
|
||||||
|
|
||||||
|
Accounts persist active positions. `dump_active()` separates open and
|
||||||
|
net-zero positions. Do not assume `minimized_clears()` safely preserves
|
||||||
|
size/PPU: current sign-transition and longer net-zero slicing lacks
|
||||||
|
round-trip coverage and can retain the wrong event subset.
|
||||||
|
|
||||||
|
## Market identity and symcache
|
||||||
|
|
||||||
|
`SymbologyCache` has three distinct key-addressed mappings:
|
||||||
|
|
||||||
|
- `assets`: provider asset ID to `Asset`;
|
||||||
|
- `pairs`: native `bs_mktid` to backend pair `Struct`;
|
||||||
|
- `mktmaps`: searchable FQME or explicit alias to normalized
|
||||||
|
`MktPair`.
|
||||||
|
|
||||||
|
Search mapping keys, then resolve values. RapidFuzz treats mapping
|
||||||
|
values as choices; passing `mktmaps` directly makes it operate on
|
||||||
|
`MktPair` objects. Preserve in-memory aliases because IB can store both
|
||||||
|
native IDs and FQMEs for one `MktPair`. Do not persist such aliases
|
||||||
|
without fixing reload: `from_dict()` currently requires every key to
|
||||||
|
equal `mkt.fqme`.
|
||||||
|
|
||||||
|
## Change workflow
|
||||||
|
|
||||||
|
Before editing:
|
||||||
|
|
||||||
|
1. Identify the owning layer: client, EMS, executor, accounting, or UI.
|
||||||
|
2. Write down every ID/key transition across that boundary.
|
||||||
|
3. Trace both normal and cancel-before-ack event ordering.
|
||||||
|
4. Determine whether records are native, normalized, or summaries.
|
||||||
|
5. Check actor-local caches and context-exit writes.
|
||||||
|
6. Add a deterministic regression at the narrowest broken contract.
|
||||||
|
7. Run live/backend tests only when their credentials and side effects
|
||||||
|
are explicitly authorized.
|
||||||
|
|
||||||
|
Do not fix an accounting defect by adding state to EMS, or fix an EMS
|
||||||
|
correlation defect by rewriting durable transaction history.
|
||||||
|
|
||||||
|
## Canonical source map
|
||||||
|
|
||||||
|
- `piker/clearing/_messages.py`: client and brokerd protocol types.
|
||||||
|
- `piker/clearing/_client.py`: `OrderClient` and `open_ems()`.
|
||||||
|
- `piker/clearing/_ems.py`: router, dark book, ID mapping, translation.
|
||||||
|
- `piker/clearing/_paper_engine.py`: simulated execution/accounting.
|
||||||
|
- `piker/accounting/_ledger.py`: transactions and ledger persistence.
|
||||||
|
- `piker/accounting/_pos.py`: positions, accounts, PPU event state.
|
||||||
|
- `piker/accounting/_mktinfo.py`: `Asset`, `MktPair`, FQME schema.
|
||||||
|
- `piker/data/_symcache.py`: provider symbology mappings and search.
|
||||||
|
- `piker/accounting/calc.py`: transaction ordering and PPU/PnL math.
|
||||||
|
- `piker/brokers/*/broker.py`: backend order-control integration.
|
||||||
|
- `piker/brokers/*/ledger.py`: backend trade normalization.
|
||||||
|
|
||||||
|
See [architecture.md](architecture.md) for complete flows,
|
||||||
|
[gotchas.md](gotchas.md) for symptom-driven diagnosis, and
|
||||||
|
[test-map.md](test-map.md) for verification choices.
|
||||||
|
|
@ -0,0 +1,210 @@
|
||||||
|
# Clearing And Accounting Architecture
|
||||||
|
|
||||||
|
## Protocol direction
|
||||||
|
|
||||||
|
Client-facing messages and broker-facing messages intentionally differ.
|
||||||
|
|
||||||
|
```text
|
||||||
|
client -> EMS
|
||||||
|
Order, Cancel
|
||||||
|
|
||||||
|
EMS -> executor
|
||||||
|
BrokerdOrder, BrokerdCancel
|
||||||
|
|
||||||
|
executor -> EMS
|
||||||
|
BrokerdOrderAck, BrokerdStatus, BrokerdFill,
|
||||||
|
BrokerdError, BrokerdPosition
|
||||||
|
|
||||||
|
EMS -> subscribed clients
|
||||||
|
Status, BrokerdPosition
|
||||||
|
```
|
||||||
|
|
||||||
|
`Order` carries the client `oid`, account, market, price, size, action,
|
||||||
|
and execution mode. `BrokerdOrderAck` establishes the executor's
|
||||||
|
`reqid`; subsequent broker messages should carry that ID so EMS can
|
||||||
|
resolve the client dialog.
|
||||||
|
|
||||||
|
`Status` is mutable current dialog state, not a history envelope. Exact
|
||||||
|
client states include `pending`, `open`, `dark_open`, `triggered`,
|
||||||
|
`fill`, `closed`, `canceled`, and `error`. Audit existing translation
|
||||||
|
before adding a state; backend `BrokerdStatus` vocabulary is narrower
|
||||||
|
and current producers are not fully conformant.
|
||||||
|
|
||||||
|
## EMS routing
|
||||||
|
|
||||||
|
`_emsd_main()` serves each client/market context. The actor-local
|
||||||
|
`Router` owns books, subscribers, order dialogs, and cached live trades
|
||||||
|
relays. Paper mode opens per-symbol relay contexts under the same broker
|
||||||
|
key, a current overwrite/pop hazard for concurrent paper symbols.
|
||||||
|
|
||||||
|
`open_brokerd_dialog()` is the session live-versus-paper selection
|
||||||
|
point:
|
||||||
|
|
||||||
|
- live execution talks to `brokerd.<broker>`;
|
||||||
|
- paper execution talks to `paperboi.<broker>`;
|
||||||
|
- chart-only and paper sessions should not boot credentialed brokerd.
|
||||||
|
|
||||||
|
Do not infer paper execution from each order's `exec_mode`. Session
|
||||||
|
mode selects the executor; per-order mode primarily distinguishes dark
|
||||||
|
from immediately routed orders, despite incomplete literal annotations.
|
||||||
|
|
||||||
|
`process_client_order_cmds()` handles submit, modify, cancel, dark,
|
||||||
|
and alert intent. It updates EMS dialog state before routing broker
|
||||||
|
commands. `translate_and_relay_brokerd_events()` applies acks, statuses,
|
||||||
|
fills, errors, and positions to dialogs and subscribers.
|
||||||
|
|
||||||
|
The `DarkBook` stores local predicates and watches quote ticks. When a
|
||||||
|
predicate triggers, `clear_dark_triggers()` emits the live broker order
|
||||||
|
and updates the existing client dialog rather than inventing a second
|
||||||
|
order identity.
|
||||||
|
|
||||||
|
## Ack and cancel ordering
|
||||||
|
|
||||||
|
Required actionable ordering is submit, ack, open, zero or more fills,
|
||||||
|
then terminal status. EMS handles cancel-before-ack, but it does not
|
||||||
|
buffer arbitrary status-before-ack events; those currently fall through
|
||||||
|
as unhandled. Backends must ack before actionable statuses.
|
||||||
|
|
||||||
|
The important race is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
client Cancel(oid)
|
||||||
|
-> EMS has no reqid yet
|
||||||
|
-> mark Status.cancel_called
|
||||||
|
BrokerdOrderAck(oid, reqid)
|
||||||
|
-> install mapping
|
||||||
|
-> send deferred BrokerdCancel(reqid)
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not drop a pre-ack cancel and do not send a broker cancel with an
|
||||||
|
`oid` unless the backend explicitly uses that value as its `reqid`.
|
||||||
|
|
||||||
|
## Paper execution
|
||||||
|
|
||||||
|
`PaperBoi` maintains simulated orders, opens the market feed, and fills
|
||||||
|
orders when L1 ask/bid quotes cross their prices. `trade`/`last` ticks
|
||||||
|
zip buy and sell iterators, so a lone resting side currently may not
|
||||||
|
fill from those tick types. `fake_fill()` is the core fill-to-account
|
||||||
|
path:
|
||||||
|
|
||||||
|
1. emit `BrokerdFill`;
|
||||||
|
2. emit terminal `BrokerdStatus` for the simulated order;
|
||||||
|
3. construct a signed normalized `Transaction`;
|
||||||
|
4. `TransactionLedger.update_from_t()`;
|
||||||
|
5. `Account.update_from_ledger()`;
|
||||||
|
6. persist the ledger and account;
|
||||||
|
7. emit `BrokerdPosition` with current size and PPU.
|
||||||
|
|
||||||
|
The protocol notifications and persistence occur in one task but are
|
||||||
|
not a database transaction. If persistence fails after fill/status
|
||||||
|
publication, restart/reconciliation behavior matters. Do not reorder
|
||||||
|
the sequence casually.
|
||||||
|
|
||||||
|
Partial fills currently reuse `oid` as `tid`. A later partial therefore
|
||||||
|
overwrites the paper ledger record and is rejected by position-event
|
||||||
|
dedupe. Fix unique fill IDs before relying on paper partial-fill math.
|
||||||
|
|
||||||
|
`open_trade_dialog()` opens symcache, paper ledger, account, and feed
|
||||||
|
resources under structured concurrency. An exception from any nested
|
||||||
|
resource, including ledger context exit, cancels the dialog and its feed
|
||||||
|
stream.
|
||||||
|
|
||||||
|
## Live backend variation
|
||||||
|
|
||||||
|
There is no universal live-broker accounting implementation:
|
||||||
|
|
||||||
|
- IB normalizes execution/commission events, updates ledger/account
|
||||||
|
state, and emits position updates.
|
||||||
|
- Kraken has backend trade normalization and position reconstruction,
|
||||||
|
but its live path writes account state without inserting every new
|
||||||
|
trade into `TransactionLedger`.
|
||||||
|
- Binance currently relays venue order/position events and does not
|
||||||
|
share every local `TransactionLedger`/`Account` path.
|
||||||
|
|
||||||
|
Inspect the backend's `broker.py`, `ledger.py`, and `norm_trade()` before
|
||||||
|
moving generic logic. A paper behavior is not automatically a live
|
||||||
|
backend contract.
|
||||||
|
|
||||||
|
## Ledger normalization
|
||||||
|
|
||||||
|
`TransactionLedger.data` is the durable record mapping. For paper it
|
||||||
|
contains transaction-like dictionaries. For live brokers it can contain
|
||||||
|
raw provider records.
|
||||||
|
|
||||||
|
`iter_txns()` sorts records and chooses normalization:
|
||||||
|
|
||||||
|
- paper uses `_paper_engine.norm_trade()`;
|
||||||
|
- live accounts use `brokermod.norm_trade()`.
|
||||||
|
|
||||||
|
Normalizer output should carry datetime, signed size, price, cost,
|
||||||
|
FQME, backend market ID, and transaction ID. Some normalizers return
|
||||||
|
`None` for unusable records; `iter_txns()` does not filter it, while
|
||||||
|
`to_txns()` does. Audit direct iterator consumers.
|
||||||
|
|
||||||
|
`write_config()` sorts records for stable persistence. Paper rewrite
|
||||||
|
logic migrates old symbol fields and qualifies legacy FQMEs through the
|
||||||
|
symcache. Search can return no matches; callers which select the first
|
||||||
|
result need an explicit fallback or a descriptive failure policy.
|
||||||
|
|
||||||
|
## Position event model
|
||||||
|
|
||||||
|
`Position` holds a market and a transaction-event mapping. New clears
|
||||||
|
are deduplicated by `tid`. Current size and PPU are recomputed from the
|
||||||
|
event sequence.
|
||||||
|
|
||||||
|
`Account.update_from_ledger()`:
|
||||||
|
|
||||||
|
1. receives normalized transactions;
|
||||||
|
2. resolves each market through `mktmaps` or a fallback table;
|
||||||
|
3. looks up or creates the position by `Transaction.bs_mktid`;
|
||||||
|
4. adds unseen clears;
|
||||||
|
5. returns visited positions, including duplicates whose clear was
|
||||||
|
rejected by `add_clear()`.
|
||||||
|
|
||||||
|
`minimized_clears()` intends to discard irrelevant history after
|
||||||
|
net-zero or sign transitions, but current slicing is not proven safe
|
||||||
|
and can reconstruct the wrong active size/PPU. Treat it as a known bug
|
||||||
|
surface until sign-transition round trips are covered.
|
||||||
|
|
||||||
|
`Position.update_from_msg()` serves UI summaries by replacing local
|
||||||
|
event state with a synthetic clear representing reported size and PPU.
|
||||||
|
That projection is intentionally lossy.
|
||||||
|
|
||||||
|
## Symbology contracts
|
||||||
|
|
||||||
|
`MktPair.fqme` is a normalized endpoint address assembled from pair,
|
||||||
|
venue, expiry/contract information, and broker. `MktPair.bs_fqme`
|
||||||
|
drops the broker suffix. `MktPair.bs_mktid` is the provider-native
|
||||||
|
market identity.
|
||||||
|
|
||||||
|
`SymbologyCache.load()` attempts to obtain provider assets and pair
|
||||||
|
structs, calls `get_mkt_info()`, and populates conventional mappings.
|
||||||
|
Assets are optional; missing/empty pair support returns early.
|
||||||
|
|
||||||
|
```text
|
||||||
|
assets[provider_asset_id] -> Asset
|
||||||
|
pairs[pair.bs_mktid] -> backend Pair Struct
|
||||||
|
mktmaps[mkt.fqme] -> MktPair
|
||||||
|
```
|
||||||
|
|
||||||
|
Some backends add aliases such as `mktmaps[bs_mktid]`. Search must
|
||||||
|
return the matched mapping key rather than rebuilding a key from the
|
||||||
|
value, otherwise aliases silently collapse. Current serialization then
|
||||||
|
writes aliases which reload rejects because `from_dict()` asserts each
|
||||||
|
key equals `mkt.fqme`.
|
||||||
|
|
||||||
|
## Persistence boundaries
|
||||||
|
|
||||||
|
`open_trade_ledger()` loads account TOML and compares a shallow snapshot
|
||||||
|
in `finally`. Top-level additions/replacements trigger writes; nested
|
||||||
|
in-place mutation may not. `open_account()` similarly mediates account
|
||||||
|
state. These contexts run inside actor tasks, so serialization errors
|
||||||
|
are runtime failures, not merely offline maintenance failures.
|
||||||
|
|
||||||
|
When changing schemas:
|
||||||
|
|
||||||
|
- preserve backend-native records unless migration is explicit;
|
||||||
|
- keep old paper field migration deterministic;
|
||||||
|
- avoid fuzzy qualification when an exact `bs_mktid`/FQME exists;
|
||||||
|
- test context exit, not only in-memory updates;
|
||||||
|
- inspect emitted positions after disk round trips.
|
||||||
|
|
@ -0,0 +1,146 @@
|
||||||
|
# Clearing And Ledger Gotchas
|
||||||
|
|
||||||
|
## Trade dialog dies immediately after a fill
|
||||||
|
|
||||||
|
Likely cause: an explicit ledger/account write in the fill handler, or
|
||||||
|
a later context-exit write, raised after event publication.
|
||||||
|
|
||||||
|
Inspect, in order:
|
||||||
|
|
||||||
|
1. the first non-cancellation exception in the actor traceback;
|
||||||
|
2. `PaperBoi.fake_fill()` or the live backend fill handler;
|
||||||
|
3. explicit and context-exit `write_config()` calls;
|
||||||
|
4. FQME/`bs_mktid` qualification and TOML-compatible values;
|
||||||
|
5. the persisted ledger/account files for a partial update.
|
||||||
|
|
||||||
|
Tractor cancellation logs are often the consequence. Root-cause the
|
||||||
|
inner serialization, lookup, or normalization exception first.
|
||||||
|
|
||||||
|
## `MktPair` has no `len()` in RapidFuzz
|
||||||
|
|
||||||
|
Cause: a mapping of string keys to `MktPair` values was passed directly
|
||||||
|
to `rapidfuzz.process.extract()`. Mapping values become fuzzy choices.
|
||||||
|
|
||||||
|
Fix: search a list/view of string keys and resolve matched keys back
|
||||||
|
through the original mapping. Do not add sequence methods to `MktPair`
|
||||||
|
and do not use `processor=str` when aliases must be preserved.
|
||||||
|
|
||||||
|
## Fuzzy qualification raises `IndexError`
|
||||||
|
|
||||||
|
Cause: the caller selected the first match from an empty result after
|
||||||
|
the score cutoff rejected every key.
|
||||||
|
|
||||||
|
Decide explicitly whether to retain the input identity, reject the
|
||||||
|
record with a descriptive exception, or obtain missing market metadata.
|
||||||
|
Do not lower the cutoff blindly; the wrong market corrupts accounting.
|
||||||
|
|
||||||
|
## Cancel is ignored or sent with the wrong ID
|
||||||
|
|
||||||
|
Cause: confusion between client `oid` and backend `reqid`, commonly
|
||||||
|
during cancel-before-ack ordering.
|
||||||
|
|
||||||
|
Verify the ack installed both mapping directions and that
|
||||||
|
`Status.cancel_called` survives until `reqid` exists. Log both IDs and
|
||||||
|
the actor/backend account at every transition.
|
||||||
|
|
||||||
|
## Fills appear but positions do not move
|
||||||
|
|
||||||
|
Check these boundaries:
|
||||||
|
|
||||||
|
- the backend emitted `BrokerdFill` but never normalized a transaction;
|
||||||
|
- transaction size lost its buy/sell sign;
|
||||||
|
- `tid` collided with an existing position event;
|
||||||
|
- `Transaction.bs_mktid` selected a different `Account.pps` key;
|
||||||
|
- `mktmaps` lacked the exact transaction identity;
|
||||||
|
- position publication happened before accounting update;
|
||||||
|
- the backend intentionally reports venue positions instead of local
|
||||||
|
ledger-derived positions.
|
||||||
|
|
||||||
|
## Duplicate or inflated position size
|
||||||
|
|
||||||
|
Do not deduplicate by price/time alone. Confirm stable transaction IDs,
|
||||||
|
chronological sorting, and `Position.add_clear()` idempotence. Broker
|
||||||
|
execution IDs and order IDs are not interchangeable.
|
||||||
|
|
||||||
|
For partial fills, several `tid`s may correctly share one order
|
||||||
|
`reqid`. Collapsing them loses clears; replaying one `tid` twice
|
||||||
|
inflates nothing only if the dedupe boundary is preserved.
|
||||||
|
|
||||||
|
Current paper fills use `tid=oid`, so later partials overwrite the
|
||||||
|
ledger record and are deduplicated out of the position. Treat paper
|
||||||
|
partial fills as broken until each clear receives a stable unique ID.
|
||||||
|
|
||||||
|
## Paper and live state disagree
|
||||||
|
|
||||||
|
Paper uses normalized local transactions and can use the full FQME as
|
||||||
|
its account position key. A live backend may use native IDs, raw ledger
|
||||||
|
records, or venue-reported positions.
|
||||||
|
|
||||||
|
Compare normalized `Transaction`s and emitted `BrokerdPosition`s, not
|
||||||
|
raw TOML dictionaries across backends.
|
||||||
|
|
||||||
|
## Account appears under the wrong name
|
||||||
|
|
||||||
|
Separate UI/EMS account aliases from backend account IDs and ledger file
|
||||||
|
names. Backends may prefix account names for routing while persistence
|
||||||
|
uses an unprefixed native account key.
|
||||||
|
|
||||||
|
Trace account identity through `open_ems()`, broker dialog startup,
|
||||||
|
ledger opening, and `BrokerdPosition.account` before changing naming.
|
||||||
|
|
||||||
|
## A module-global cache is unexpectedly empty
|
||||||
|
|
||||||
|
Symcache, clients, contracts, and order tables are actor-local. Data
|
||||||
|
loaded in `datad` is not automatically available in `brokerd` or
|
||||||
|
`paperboi`. Also, `_symcache._caches` is currently read but never
|
||||||
|
populated, so do not assume repeated `open_symcache()` calls hit it.
|
||||||
|
|
||||||
|
Load `piker-conc-expert` and audit actor-local globals, async caches,
|
||||||
|
and dialog startup warming. Never rely on an import side effect from a
|
||||||
|
sibling actor.
|
||||||
|
|
||||||
|
## Teardown emits a cancellation storm
|
||||||
|
|
||||||
|
Find the first application exception. Structured cancellation then
|
||||||
|
closes feed streams, broker streams, and nested contexts. Avoid masking
|
||||||
|
the originating exception with broad cancellation handling.
|
||||||
|
|
||||||
|
For stream ownership and context cancellation semantics, use
|
||||||
|
`piker-conc-expert`.
|
||||||
|
|
||||||
|
## Broker status is logged as unhandled
|
||||||
|
|
||||||
|
EMS currently accepts only ack-correlated `open`, `closed`, and
|
||||||
|
`canceled` broker statuses. Declared `pending` and backend-produced
|
||||||
|
`fill`/`filled` variants fall through, as do statuses arriving before
|
||||||
|
ack. Verify producer vocabulary and ack ordering before adding another
|
||||||
|
consumer branch.
|
||||||
|
|
||||||
|
## Another client sees order or position events
|
||||||
|
|
||||||
|
Current translation broadcasts order updates by FQME and positions to
|
||||||
|
all attached clients; `Router.dialogs` is not used as an event privacy
|
||||||
|
filter. Treat subscriber isolation as incomplete, not guaranteed.
|
||||||
|
|
||||||
|
## Debug breadcrumb
|
||||||
|
|
||||||
|
Capture one row per event with:
|
||||||
|
|
||||||
|
```text
|
||||||
|
actor, backend, account, fqme, bs_mktid,
|
||||||
|
oid, reqid, venue_order_id, tid,
|
||||||
|
message_type, status, action, signed_size, price, timestamp
|
||||||
|
```
|
||||||
|
|
||||||
|
Then compare:
|
||||||
|
|
||||||
|
1. client command order;
|
||||||
|
2. EMS status history;
|
||||||
|
3. broker event order;
|
||||||
|
4. normalized transactions;
|
||||||
|
5. position event IDs and computed size/PPU;
|
||||||
|
6. ledger/account disk state;
|
||||||
|
7. emitted position summaries.
|
||||||
|
|
||||||
|
This separates protocol loss, identity skew, accounting math, and
|
||||||
|
persistence failure without guessing from UI state.
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
# Clearing And Accounting Test Map
|
||||||
|
|
||||||
|
## Deterministic first-pass tests
|
||||||
|
|
||||||
|
Use the narrowest test that owns the broken contract:
|
||||||
|
|
||||||
|
| Area | First target |
|
||||||
|
|---|---|
|
||||||
|
| Ledger/account persistence | targeted node in `tests/test_accounting.py` |
|
||||||
|
| EMS bad-backend handling | `tests/test_ems.py::test_ems_err_on_bad_broker` |
|
||||||
|
| IB request/response routing | `tests/test_ib_method_proxy.py` |
|
||||||
|
| IB history normalization | `tests/test_ib_history.py` |
|
||||||
|
| Service/actor startup | targeted node in `tests/test_services.py` |
|
||||||
|
|
||||||
|
Known missing high-value regressions include paper partial-fill IDs,
|
||||||
|
status-before-ack handling, `minimized_clears()` sign-transition round
|
||||||
|
trips, nested ledger dirty detection, and symcache alias reload.
|
||||||
|
|
||||||
|
For `SymbologyCache.search()` and paper ledger qualification, cover:
|
||||||
|
|
||||||
|
- real `Asset` and `MktPair` values;
|
||||||
|
- a native-ID alias and canonical FQME key;
|
||||||
|
- the actual `TransactionLedger.write_config()` path;
|
||||||
|
- persisted TOML fields after the round trip;
|
||||||
|
- no use of live APIs or user configuration.
|
||||||
|
|
||||||
|
## Side-effectful tests
|
||||||
|
|
||||||
|
Do not run these without explicit authorization and the repository test
|
||||||
|
harness guidance:
|
||||||
|
|
||||||
|
- paper EMS tests which open live market/symbology feeds;
|
||||||
|
- live Binance/Kraken feed suites;
|
||||||
|
- IB account tests using configured accounts;
|
||||||
|
- tests which write tracked ledger/account fixtures;
|
||||||
|
- broker dialogs requiring credentials or venue connectivity.
|
||||||
|
|
||||||
|
After any accounting test using `tests/_inputs`, inspect its diff. Some
|
||||||
|
context managers can rewrite fixture ledgers on exit.
|
||||||
|
|
||||||
|
## Regression design
|
||||||
|
|
||||||
|
Prefer a direct contract regression over a full actor test when the
|
||||||
|
failure is synchronous and deterministic. Use an actor-level test when
|
||||||
|
the bug depends on:
|
||||||
|
|
||||||
|
- ack/cancel or fill/status interleaving;
|
||||||
|
- context cancellation or stream ownership;
|
||||||
|
- actor-local state/cache separation;
|
||||||
|
- daemon selection or startup;
|
||||||
|
- publication ordering across EMS and broker streams.
|
||||||
|
|
||||||
|
Every regression should document the original failure, triggering
|
||||||
|
state, violated invariant, arrangement, and proof of the fix.
|
||||||
|
|
||||||
|
## Verification layers
|
||||||
|
|
||||||
|
1. `git diff --cached --check` for the exact staged boundary.
|
||||||
|
2. Ruff with the repository's intended rule profile.
|
||||||
|
3. A deterministic unit/filesystem regression.
|
||||||
|
4. Adjacent accounting or EMS tests which do not require live state.
|
||||||
|
5. Explicitly authorized backend/actor integration tests.
|
||||||
|
|
||||||
|
Do not interpret a passing ellipsis-body or hard-skipped test as
|
||||||
|
behavioral coverage. Read the selected node before using it as a gate.
|
||||||
|
|
||||||
|
## Manual reconciliation check
|
||||||
|
|
||||||
|
For production-only failures, compare one fill end to end:
|
||||||
|
|
||||||
|
```text
|
||||||
|
BrokerdFill
|
||||||
|
-> normalized Transaction
|
||||||
|
-> ledger record
|
||||||
|
-> Position._events[tid]
|
||||||
|
-> Position.cumsize / ppu
|
||||||
|
-> account TOML
|
||||||
|
-> BrokerdPosition
|
||||||
|
-> UI summary
|
||||||
|
```
|
||||||
|
|
||||||
|
All identities, sign, price, and account values should remain explainable
|
||||||
|
at each arrow. A mismatch identifies the owning boundary for the next
|
||||||
|
focused test.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
../../.agents/skills/piker-clearing-expert
|
||||||
|
|
@ -211,6 +211,7 @@ docs/conversations/
|
||||||
# BEGIN ai.skillz: runtime:open-wkt
|
# BEGIN ai.skillz: runtime:open-wkt
|
||||||
.claude/wkts/
|
.claude/wkts/
|
||||||
claude_wkts
|
claude_wkts
|
||||||
|
/wkts/
|
||||||
# END ai.skillz: runtime:open-wkt
|
# END ai.skillz: runtime:open-wkt
|
||||||
|
|
||||||
# BEGIN ai.skillz: direct:symlink:claude:open-wkt
|
# BEGIN ai.skillz: direct:symlink:claude:open-wkt
|
||||||
|
|
|
||||||
13
ai/README.md
13
ai/README.md
|
|
@ -21,11 +21,14 @@ track new integration ideas and proposals in
|
||||||
|
|
||||||
## Shared Skills
|
## Shared Skills
|
||||||
|
|
||||||
Repo-specific skills use the portable Agent Skills
|
Provider-neutral skill bundles should use the portable Agent Skills
|
||||||
frontmatter subset and live under `.claude/skills/` as a
|
subset and live under `.agents/skills/` as their single source.
|
||||||
single source. Claude Code discovers that directory
|
OpenCode discovers that project directory natively. Relative links in
|
||||||
natively, and OpenCode discovers the same project skills
|
provider directories such as `.claude/skills/` may bridge clients which
|
||||||
without copies or generated wrappers.
|
do not yet scan `.agents/skills/`; never copy the skill body.
|
||||||
|
|
||||||
|
Existing repo skills under `.claude/skills/` predate this convention
|
||||||
|
and can migrate separately without coupling their history to new skills.
|
||||||
|
|
||||||
Harness-specific command files should only delegate to a
|
Harness-specific command files should only delegate to a
|
||||||
shared skill. They must not duplicate the skill body.
|
shared skill. They must not duplicate the skill body.
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,7 @@ integration for piker's shared coding-harness skills.
|
||||||
| Skill | Invocable | Description |
|
| Skill | Invocable | Description |
|
||||||
|-------|-----------|-------------|
|
|-------|-----------|-------------|
|
||||||
| [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
|
| [`commit-msg`](#commit-msg) | `/commit-msg` | Generate piker-style commit messages |
|
||||||
|
| `piker-clearing-expert` | auto | Clearing, order-control, and accounting invariants |
|
||||||
| `piker-conc-expert` | auto | Actor-tree and structured-concurrency invariants |
|
| `piker-conc-expert` | auto | Actor-tree and structured-concurrency invariants |
|
||||||
| `piker-profiling` | auto | `Profiler` API patterns for perf work |
|
| `piker-profiling` | auto | `Profiler` API patterns for perf work |
|
||||||
| `piker-slang` | auto | Communication style + slang guide |
|
| `piker-slang` | auto | Communication style + slang guide |
|
||||||
|
|
@ -19,10 +20,12 @@ Skills marked **auto** are background knowledge
|
||||||
applied automatically when Claude detects relevance.
|
applied automatically when Claude detects relevance.
|
||||||
Only `commit-msg` is user-invoked via slash command.
|
Only `commit-msg` is user-invoked via slash command.
|
||||||
|
|
||||||
Portable skill source files live under
|
New portable skill source files live under
|
||||||
`.claude/skills/<skill-name>/SKILL.md` and are shared with
|
`.agents/skills/<skill-name>/SKILL.md`. Relative links under
|
||||||
OpenCode. Claude-specific behavior belongs in command or
|
`.claude/skills/` expose them to Claude Code without copying bodies.
|
||||||
settings files, not the shared skill bodies.
|
Existing pre-migration skills can remain under `.claude/skills/`.
|
||||||
|
Claude-specific behavior belongs in command or settings files, not
|
||||||
|
shared skill bodies.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,16 @@
|
||||||
# OpenCode Integration
|
# OpenCode Integration
|
||||||
|
|
||||||
[OpenCode](https://opencode.ai/) uses piker's shared
|
[OpenCode](https://opencode.ai/) discovers provider-neutral skills
|
||||||
repo-specific skills directly from `.claude/skills/`.
|
directly from `.agents/skills/` and pre-migration piker skills from
|
||||||
There is no second copy of each skill to keep in sync.
|
`.claude/skills/`. There is no second copy of each skill to keep in
|
||||||
|
sync.
|
||||||
|
|
||||||
## Available Skills
|
## Available Skills
|
||||||
|
|
||||||
| Skill | Activation | Specialization |
|
| Skill | Activation | Specialization |
|
||||||
|-------|------------|----------------|
|
|-------|------------|----------------|
|
||||||
| `commit-msg` | `/commit-msg` or explicit request | Piker commit-message style and artifacts |
|
| `commit-msg` | `/commit-msg` or explicit request | Piker commit-message style and artifacts |
|
||||||
|
| `piker-clearing-expert` | automatic | Clearing, order-control, and accounting invariants |
|
||||||
| `piker-conc-expert` | automatic | `tractor` actor topology, RPC, and cancellation |
|
| `piker-conc-expert` | automatic | `tractor` actor topology, RPC, and cancellation |
|
||||||
| `piker-profiling` | automatic | Cross-actor `Profiler` instrumentation |
|
| `piker-profiling` | automatic | Cross-actor `Profiler` instrumentation |
|
||||||
| `piker-slang` | automatic | Project communication style |
|
| `piker-slang` | automatic | Project communication style |
|
||||||
|
|
@ -49,3 +51,5 @@ hot-reloaded into an active session.
|
||||||
hardcode Claude Code attribution in shared workflows.
|
hardcode Claude Code attribution in shared workflows.
|
||||||
- Keep supporting references beside each shared skill so
|
- Keep supporting references beside each shared skill so
|
||||||
relative links resolve in every harness.
|
relative links resolve in every harness.
|
||||||
|
- Put new provider-neutral bundles under `.agents/skills/`; use
|
||||||
|
relative provider links only when a harness requires one.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
session: opencode-pid-469264
|
||||||
|
timestamp: 2026-08-28T02:14:11Z
|
||||||
|
git_ref: 6ca6ab2a
|
||||||
|
scope: config
|
||||||
|
substantive: true
|
||||||
|
raw_file: 20260828T021411Z_6ca6ab2a_prompt_io.raw.md
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Move the fix out of the root checkout into an isolated worktree and
|
||||||
|
include a Piker-specific clearing, accounting, order-control, and
|
||||||
|
trade-ledger expert skill in the worktree's commit plan.
|
||||||
|
Use a cross-harness, AI-provider-neutral standard location for its
|
||||||
|
canonical files instead of making one provider directory authoritative.
|
||||||
|
|
||||||
|
## Response summary
|
||||||
|
|
||||||
|
Created a portable `piker-clearing-expert` skill bundle under the
|
||||||
|
provider-neutral `.agents/skills/` convention, with a relative Claude
|
||||||
|
compatibility link. It covers subsystem ownership, protocols,
|
||||||
|
identities, persistence, backend variation, debugging hazards, and
|
||||||
|
test strategy. Added the canonical `wkts/` runtime ignore.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `.gitignore` - ignores the canonical root-level worktree directory.
|
||||||
|
- `.agents/skills/piker-clearing-expert/SKILL.md` - core expertise.
|
||||||
|
- `.agents/skills/piker-clearing-expert/architecture.md` - deep flows.
|
||||||
|
- `.agents/skills/piker-clearing-expert/gotchas.md` - diagnosis.
|
||||||
|
- `.agents/skills/piker-clearing-expert/test-map.md` - test map.
|
||||||
|
- `.claude/skills/piker-clearing-expert` - relative Claude bridge.
|
||||||
|
- `ai/README.md` - defines the provider-neutral source convention.
|
||||||
|
- `ai/claude-code/README.md` - advertises the shared skill to Claude.
|
||||||
|
- `ai/opencode/README.md` - advertises the shared skill to OpenCode.
|
||||||
|
|
||||||
|
## Human edits
|
||||||
|
|
||||||
|
The human rejected planning in the dirty root checkout, required a
|
||||||
|
dedicated worktree, expanded the work to include a durable expert
|
||||||
|
skill, and then directed its canonical source into a cross-provider
|
||||||
|
standard location. The agent applied those directed scope and layout
|
||||||
|
changes; no direct manual source edit was identified.
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
---
|
||||||
|
model: openai/gpt-5.6-sol
|
||||||
|
service: opencode
|
||||||
|
timestamp: 2026-08-28T02:14:11Z
|
||||||
|
git_ref: 6ca6ab2a
|
||||||
|
diff_cmd: git diff HEAD~1..HEAD
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt
|
||||||
|
|
||||||
|
Move the symcache/accounting fix into an isolated worktree and, as
|
||||||
|
part of its commit plan, distill a Piker-specific skill for deep
|
||||||
|
clearing, accounting, order-control, and trade-ledger expertise.
|
||||||
|
A follow-up required the skill's canonical files to begin moving into
|
||||||
|
a cross-harness, AI-provider-neutral standard location.
|
||||||
|
|
||||||
|
## Response
|
||||||
|
|
||||||
|
Created the shared repository skill bundle under the provider-neutral
|
||||||
|
`.agents/skills/piker-clearing-expert/` location. Its main skill defines
|
||||||
|
activation cues,
|
||||||
|
runtime ownership, event and identifier contracts, accounting and
|
||||||
|
persistence invariants, market identity rules, an editing workflow,
|
||||||
|
and a canonical source map.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .agents/skills/piker-clearing-expert/SKILL.md`
|
||||||
|
|
||||||
|
Added a detailed architecture reference for EMS routing, ack/cancel
|
||||||
|
ordering, paper and live execution, ledger normalization, position
|
||||||
|
events, symbology, and context-managed persistence.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .agents/skills/piker-clearing-expert/architecture.md`
|
||||||
|
|
||||||
|
Added symptom-driven gotchas and debugging breadcrumbs for actor
|
||||||
|
teardown, ID skew, position reconciliation, duplicate fills,
|
||||||
|
paper/live differences, and RapidFuzz mapping misuse.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .agents/skills/piker-clearing-expert/gotchas.md`
|
||||||
|
|
||||||
|
Added a deterministic-to-live test map and regression-design guidance.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .agents/skills/piker-clearing-expert/test-map.md`
|
||||||
|
|
||||||
|
Updated the worktree runtime ignore block to include the canonical
|
||||||
|
root-level `wkts/` directory required by the worktree lifecycle.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .gitignore`
|
||||||
|
|
||||||
|
Added a relative Claude compatibility link and documented the
|
||||||
|
provider-neutral source convention in the shared and harness-specific
|
||||||
|
inventories.
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- .claude/skills/piker-clearing-expert`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- ai/README.md`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- ai/claude-code/README.md`
|
||||||
|
|
||||||
|
> `git diff HEAD~1..HEAD -- ai/opencode/README.md`
|
||||||
|
|
||||||
|
The skill cross-links `piker-conc-expert` for actor-runtime mechanics
|
||||||
|
instead of duplicating structured-concurrency guidance.
|
||||||
Loading…
Reference in New Issue