Commit Graph

4181 Commits (fff610fa8d2206e55cc02992afec18fe3f2c6262)

Author SHA1 Message Date
Tyler Goodlet fff610fa8d Fix `PositionTracker.pane` attr resolve bug.. 2023-08-02 17:33:02 -04:00
Tyler Goodlet 7ecf2bd89a Guess exit transaction costs for BEP prediction
In order to attempt giving the user a realistic prediction for a BEP per
txn we need to model what the (worst case) anticipated exit txn costs
will be during the equivalent, paired entries. For now we use a simple
"symmetric cost prediction" model where we assume the exit costs will be
simply the same as the enter txn costs and thus on every entry we apply
2x the enter txn cost; on exit txns we then unroll these predictions by
keeping a cumulative sum of the cost-per-unit and reversing the charges
based on applying that mean to the current exit txn's size. Once
unrolled we apply the actual exit txn cost received from the
broker-provider.
2023-08-02 17:25:23 -04:00
Tyler Goodlet 1e3a4ca36d Drop commented, now deprecated edge case notes 🏄 2023-08-01 15:49:56 -04:00
Tyler Goodlet b6a705852d Handle txn costs in BEP, factor enter/exit blocks and df row assignments B) 2023-08-01 15:42:30 -04:00
Tyler Goodlet 29bab02c64 Pass sync code flag in flex report processor 2023-08-01 09:12:52 -04:00
Tyler Goodlet 85ae180f8f Factor df conversion into lone routine: `ledger_to_dfs()` 2023-07-31 17:48:03 -04:00
Tyler Goodlet 5d24b5defb Swap branch order for enter/exit
Also fix bug since we always need to reset cum_pos_pnl
after a `exit_to_zero` case.
2023-07-31 17:32:49 -04:00
Tyler Goodlet 100be54641 data.history: add TODO for non-zero epochs and some typing 2023-07-31 17:21:11 -04:00
Tyler Goodlet a088ebf5e2 Use inf row/col repr for debugging atm 2023-07-31 17:18:28 -04:00
Tyler Goodlet b37a447595 Implement PPU and BEP and inject the ledger frames
Since it appears impossible to compute the recurrence relations for PPU
(at least sanely) without using embedded `polars.List` elements, this
instead just implements price-per-unit and break-even-price calcs
doing a plain-ol-for-loop imperative approach with logic branching.

I burned wayy too much time trying to implement this in some kinda
`polars` DF native way without luck, so hopefuly someone smarter can
come in and make it work at some point xD

Resolves a related bullet in #515
2023-07-31 16:01:31 -04:00
Tyler Goodlet b1edaf0639 First draft position accounting with `polars`
Took a little while to get right using declarative style but it's
finally workin and seems (mostly correct B)

Computes the ppu (price per unit) using the PnL since last
net-zero-cumsize (aka the pnl from open to close) and uses it to calc
the pnl-per-exit trade (using the ppu).

Next up, bep (break even price both) per position and maybe since
ledger start or an arbitrary ref point?
2023-07-29 21:02:59 -04:00
Tyler Goodlet 385561276b Add gap detection into the `store ldshm` cmd 2023-07-26 15:45:55 -04:00
Tyler Goodlet d94ab9d5b2 order_mode: Only send cancels for dialogs that still exist 2023-07-26 15:43:48 -04:00
Tyler Goodlet 08e8990fe3 Do single `ShmArray.array` read on zero-time filtering 2023-07-26 15:41:04 -04:00
Tyler Goodlet 2c6ae5d994 Drop the `gap_dt_unit: str` column
We don't need it in `detect_time_gaps()` since doing straight up
datetime diffs in `polars` already has a humanized `str` representation
but with higher precision like '2d 1h 24m 1s' B)
2023-07-26 15:37:59 -04:00
Tyler Goodlet f1289ccce2 ib: Oof, right need to create ledger entries too.. 2023-07-26 14:55:17 -04:00
Tyler Goodlet 7802febd20 Backfill history gaps with pre-gap close 2023-07-26 12:56:06 -04:00
Tyler Goodlet 64329d44e7 Flip `tractor.breakpoint()`s to new `.pause()` 2023-07-26 12:48:19 -04:00
Tyler Goodlet bd0af7a4c0 kucoin: facepalm, use correct pair fields for price/size ticks 2023-07-26 12:44:41 -04:00
Tyler Goodlet 618c461bfb binance: always upper case venue and expiry tokens
Since we need `.get_mkt_info()` to remain symmetric across calls with
different fqme inputs, and binance generally uses upper case for it's
symbology keys, we always upper the FQME related tokens for both
symcaching and general search purposes.

Also don't set `_atype` on mkt pairs since it should be fully handled
via the dst asset loading in `Client._cache_pairs()`.
2023-07-26 12:44:29 -04:00
Tyler Goodlet c00cf41541 kraken: `norm_trade()` now much accept an optional symcache 2023-07-26 12:40:58 -04:00
Tyler Goodlet 4436342d33 Change ui stuff to use new `Position.cumsize` attr name 2023-07-26 12:40:09 -04:00
Tyler Goodlet 58cf7ce10e Add `norm_trade()` ep to validator warnings 2023-07-26 12:39:08 -04:00
Tyler Goodlet 9fbb75ce7f Remove piker.trionics; already factored into `tractor` 2023-07-26 12:38:25 -04:00
Tyler Goodlet d0f72bf269 Wrap symcache loading into `.from_scratch()`
Since we need it both when explicitly reloading **and**
whenever either the file or data in the file doesn't exist.
2023-07-26 12:27:26 -04:00
Tyler Goodlet 188508575a Utilize the new `_mktmap_table` input in paper engine
In cases where a brokerd backend doesn't yet support a symcache we need
to do manual `.get_mkt_info()` queries and stash them in a table that we
pass in for the mkt failover lookup to `Account.update_from_ledger()`.
Set the `PaperBoi._mkts` to this table for use on real-time ledger
writes in `.fake_fill()`.
2023-07-26 12:21:27 -04:00
Tyler Goodlet bebc817d19 Partition ledger data frames by `bs_mktid`
Since some backends are going to have the issue of supporting multiple
venues for a given "position distinguishing instrument", like IB, we
can't presume that every `Position` can be uniquely keyed by
a `MktPair.fqme` (since the venue part can change and still be the same
"pair" relationship in accounting terms) so instead presume the
"backend system's market id" is the unique key (at least for now)
instead of the fqme.

More practically we use the `bs_mktid` to groupby-partition the per
pair DFs from the trades ledger and attempt to scan-match the input
fqme (in `ledger disect` cli) against the fqme column values set.
2023-07-26 12:13:54 -04:00
Tyler Goodlet 1d35747fbf Always clear `Position._events` in `.from_msg()`..
Not sure why i ever thought it would work otherwise but, obviously if
you're replicating a `Position` from a **summary** (IPC) msg we
need to wipe any prior clearing events from the events history..
The main use for this loading mechanism is precisely if you don't have
local access to the txn ledger and need to represent a position from
a summary 🤦

Also, never bother with ledger file fqme "rewriting" if the backend has
no symcache support (yet) since obviously there's then no symbol set to
search for a better key xD
2023-07-26 12:10:26 -04:00
Tyler Goodlet e344bdbf1b ib: rework trade handling, take ib position sizes as gospel
Instead of casting to `dict`s and rewriting event names in the
`push_tradesies()` handler, be transparent with event names (also
defining and piker-equivalent mapping them in a redefined `_statuses`
table) and types
passing them directly to the `deliver_trade_events()` task and generally
make event handler blocks much easier to grok with type annotations. To
deal with the causality dilemma of *when to emit a pos msg* due to
needing all of `execDetailsEvent, commissionReportEvent, positionEvent`
but having no guarantee on received order, we implement a small task
`clears: dict[Contract, tuple[Position, Fill]]` tracker table and (as
before) only emit a position event once the "cost" can be accessed for
the fill. We now ALWAYS relay any `Position` update from IB directly to
ensure (at least) the cumsize is correct (since it appears we still have
ongoing issues with computing this correctly via `.accounting.Position`
updates..).

Further related adjustments:
- load (fiat) balances and startup positions into a new `IbAcnt` struct.
- change `update_and_audit_pos_msg()` to blindly forward ib position
  event updates for the **the size** since it should always be
  considered the true gospel for accounting!
  - drop ib-has-no-position handling since it should never occur..
- move `update_ledger_from_api_trades()` to the `.ledger` submod and do
  processing of ib_insync `Fill` related objects instead of dict-casted
  versions instead doing the casting in
  `api_trades_to_ledger_entries()`.
- `norm_trade()`: add `symcache.mktmaps[bs_mktid] = mkt` in since it
  turns out API (and sometimes FLEX) records don't contain the listing
  exchange/venue thus making it impossible to map an asset pair in the
  "position sense" (i.e. over multiple venues: qqq.nasdaq, qqq.arca,
  qqq.directedge) to an fqme when doing offline ledger processing;
  instead use frickin IB's internal int-id so there's no discrepancy.
  - also much better handle futures mkt trade flex records such that
    parsed `MktPair.fqme` is consistent.
2023-07-25 20:28:54 -04:00
Tyler Goodlet b33be86b2f ib: fill out contract tables in `.get_mkt_info()`
Since getting a global symcache result from the API is basically
impossible, we ad-hoc fill out the needed client tables on demand per
client code queries to the mkt info EP.

Also, use `unpack_fqme()` in fqme (search) pattern parser instead of
hacky `str.partition()`.
2023-07-25 16:43:08 -04:00
Tyler Goodlet 50b221f788 ib: rework client-internal contract caching
Add new `Client` attr tables to better stash `Contract` lookup results
normally mapped from some in put FQME;

- `._contracts: dict[str, Contract]` for any input pattern (fqme).
- `._cons: dict[str, Contract] = {}` for the `.conId: int` inputs.
- `_cons2mkts: bidict[Contract, MktPair]` for mapping back and forth
  between ib and piker internal pair types.

Further,
- type out as many ib_insync internal types as possible mostly for
  contract related objects.
- change `Client.trades()` -> `.get_fills()` and return directly the
  result from `IB.fill()`.
2023-07-25 16:42:15 -04:00
Tyler Goodlet 897c20bd4a Moar `.accounting` tweaks
- start flipping over internals to `Position.cumsize`
- allow passing in a `_mktmap_table` to `Account.update_from_ledger()`
  for cases where the caller wants to per-call-dyamically insert the
  `MktPair` via a one-off table (cough IB).
- use `polars.from_dicts()` in `.calc.open_ledger_dfs()`. and wrap the
  whole func in a new `toolz.open_crash_handler()`.
2023-07-21 23:48:53 -04:00
Tyler Goodlet 759ebe71e9 Allow disabling symcache load via kwarg as well 2023-07-20 15:27:46 -04:00
Tyler Goodlet e88913e1f3 .data._pathops: drop profiler imports, fix some naming to appease `ruff` 2023-07-20 15:27:22 -04:00
Tyler Goodlet 5e7916a0df Start `piker.toolz` subpkg for all our tooling B)
Since there's a growing list of top level mods which are more or less
utils/tools for working with the runtime; begin to move them into a new
subpkg starting with a new `.toolz.debug`.

Start with,
- a new `open_crash_handller()` for doing breakpoints around blocks that
  might error.
- move in what was `piker._profile` into `.toolz.profile` and adjust all
  importing appropriately.
2023-07-20 15:23:01 -04:00
Tyler Goodlet 5eb310cac9 ib: more fixes to try and get positioning correct..
Define and bind in the `tx_sort()` routine to be used by
`open_trade_ledger()` when datetime sorting trade records.

Further deats:
- always use the IB reported position size (since apparently our ledger
  based accounting is getting rekt on occasion..).
- better ib pos msg formatting when there's mismatches with the piker
  equivalent.
- never emit zero-size pos msgs (in terms of strict ib pos sizing) since
  when there's piker ledger sizing errors we'll send the wrong thing to
  the ems and its clients..
2023-07-19 16:46:36 -04:00
Tyler Goodlet 8a10cbf6ab Change `Position.clearsdict()` -> `.clearsitems()`
Since apparently rendering to dict from a sorted generator func clearly
doesn't preserve the order when using a `dict`-comprehension.. Further,
there's really no reason to strictly return a `dict`. Adjust
`.calc.ppu()` to make the return value instead a `list[tuple[str,
dict]]`; this results in the current df cumsum values matching the
original impl and the existing `binance.paper` unit tests now passing XD

Other details that fix a variety of nonsense..
- adjust all `.clearsitems()` consumers to the new list output.
- use `str(pendulum.now())` in `Position.from_msg()` since adding
  multiples with an `unknown` str will obviously discard them, facepalm.
- fix `.calc.ppu()` to NOT short circuit when `accum_size` is 0; it's
  been causing all sorts of incorrect size outputs in the clearing
  table.. lel, this is what fixed the unit test!
2023-07-18 21:00:19 -04:00
Tyler Goodlet fe78277948 ib: add new `.symbols` sub-mod
Move in the obvious things XD
- all the specially defined venue tables from `.api`.
- some parser funcs: `con2fqme()` and `parse_patt2fqme()`.
- the `get_mkt_info()` and `open_symbol_search()` broker eps.
- the `_asset_type_map` table which converts to `.accounting.Asset`
  compat keys for each contract/security.
2023-07-17 18:30:11 -04:00
Tyler Goodlet 9e87b6515b ib: be symcache compat by using bypass attr
Since there's no easy way to support it yet, we bypass symbology caching
in for now and instead allow the `ib.ledger` routines to fill in
`MktPair` and `Asset` entries ad-hoc for the purposes of txn ledger
processing.
2023-07-17 17:31:34 -04:00
Tyler Goodlet a05a82486d Log a warning on no symcache support in a backend 2023-07-17 17:31:12 -04:00
Tyler Goodlet e4731eff10 Fix `Position.expiry == None` bug 2023-07-17 17:27:22 -04:00
Tyler Goodlet dfa13afe22 Allow backends to "bypass" symcache loading
Some backends like `ib` don't have an obvious (nor practical) way to
easily download the entire symbology set available from all its mkt
venues. For such backends loading might require a non-std approach (like
using the contract search from some input mkt-key set) and can't be
expected to necessarily be supported out of the box. As such, allow
annotating a broker sub-pkg module with a `_no_symcache: bool = True`
attr which will make `open_symcache()` yield early with an empty
`SymbologyCache` instance for use by the caller to fill in the mkt and
assets tables in whatever ad-hoc way desired.
2023-07-17 17:12:40 -04:00
Tyler Goodlet 912f1bc635 .kraken: start new `.symbols` submod and move symcache and search stuff there 2023-07-17 16:20:11 -04:00
Tyler Goodlet 82fd785646 Adjust default `[binance]` config to use paper and disable testnets 2023-07-17 14:58:15 -04:00
Tyler Goodlet 71d0097dc7 Switch to `Position.cumsize` in tracker and order mode mods 2023-07-17 13:51:30 -04:00
Tyler Goodlet 8fb667686f Open symcaches as part of per-backend search spawning 2023-07-17 01:24:45 -04:00
Tyler Goodlet 2dab0e2e56 Expose `.data._symcache` stuff at subpkg toplevel
The list is `open_symcache()`, `get_symcache()`, `SymbologyCache`, and
`Stuct` which seems more or less fine to make part of the public
namespace. Also, make `._timeseries.t_unit` an instance of literal to make
`ruff` happy?
2023-07-17 01:20:52 -04:00
Tyler Goodlet e8025d0985 .data.types.Struct: by default include non-members from `.to_dict()`.. 2023-07-16 21:32:36 -04:00
Tyler Goodlet 430309b5dc .accounting: type `Transaction.etype` as a `Literal`
Start working out the set of possible "txn types" we want to define in
a simple set.

Relates to #510
2023-07-16 21:22:15 -04:00
Tyler Goodlet 4c5507301e kraken: be symcache compatible!
This was more involved then expected but on the bright side, is going to
help drive a more general `Account` update/processing/loading API
providing for all the high-level txn update methods needed for any
backend to generically update the participant's account *state* via
an input ledger/txn set B)

Key changes to enable `SymbologyCache` compat:
- adjust `Client` pairs / assets lookup tables to include a duplicate
  keying of all assets and "asset pairs" using the (chitty) default key
  set that kraken ships which is NOT the `.altname` no `.wsname` keys;
  the "default ReST response keys" i guess?
  - `._AssetPairs` and `._Assets` are *these ^* rest-key sets delivered
    verbatim from the endpoint responses,
  - `._pairs` and `._assets` the equivalent value-sets keyed by piker
    style FQME-looking keys (now provided via the new
    `.kraken.symbols.Pair.bs_fqme: str` and the delivered `'altname'`
    field (for assets) respectively.
- re-implement `.get_assets()` and `.get_mkt_pairs()` to appropriately
  delegate to internal methods and these new (multi-keyed) tables to
  deliver the cacheable set of symbology info.
- adjust `.feed.get_mkt_info()` to handle parsing of both fqme-style and
  wtv(-the-shit-stupid) kraken key set a caller passes via
  a key-matches-first-table-style-scan after pre-processing the
  input `fqme: str`; also do the `Asset` lookups from the new
  `Pair.bs_dst/src_asset: str` fields which should always map correctly
  to an internal asset entry delivered by `Client.get_assets()`.

Dirty impl deatz:
- add new `.kraken.symbols` and move the newly refined `Pair` there.
- add `.kraken.ledger` and move in the factored out ledger processing
  routines.
- also move out what was the `has_pp()` and large chung of nested-ish
  looking acnt-position verification logic blocks into a new
  `verify_balances()` B)
2023-07-16 21:21:53 -04:00