Commit Graph

4358 Commits (b7883325a9e1ce9136f757c2fe9d3c8bbe597949)

Author SHA1 Message Date
Tyler Goodlet 8f40e522ef Add handy `DiffDump`ing for our `.types.Struct`
So you can do a `Struct1` - `Struct2` and we dump a little diff `list`
of tuples for anal on the REPL B)

Prolly can be broken out into it's own micro-patch?
2023-07-12 08:45:55 -04:00
Tyler Goodlet 87185cf8bb Drop `config` get/set/del apis.. 2023-07-12 08:45:55 -04:00
Tyler Goodlet ff267890d1 Change cached-client hit msg to runtime level 2023-07-12 08:45:55 -04:00
Tyler Goodlet 749401e500 .accounting: expose new names at pkg top level 2023-07-12 08:45:55 -04:00
Tyler Goodlet 3704e2ceac Call `open_ledger_dfs()` for `disect` sub-cmd
Drop all the old `polars` (groupby + agg related) mangling to get a df
per fqme by delegating to the new routine and add in the `.cumsum()`ing
(per frame) as a first start on computing pps using dfs instead of
python dicts + loops as in `ppu()`.
2023-07-12 08:45:55 -04:00
Tyler Goodlet 8f1983fd8e Move df loading into `calc.load_ledger_dfs()`
To isolate it from the ledger/account mods and bc it is actually for
doing (eventual) position calcs / anal, might as well put it in this
mod. Add in the old-masked `ensure_state()` method content in case we
want to use it later for testing. Also tighten up the parser loading
inside `dyn_parse_to_dt()`.
2023-07-12 08:45:55 -04:00
Tyler Goodlet f5d4f58610 `Account` api update and refine
Rename `open_pps()` -> `open_account()` for obvious reasons as well as
expect a bit tighter integration with `SymbologyCache` and consequently
`LedgerTransaction` in order to drop `Transaction.sym: MktPair`
dependence when compiling / allocating new `Position`s from a ledger.

Also we drop a bunch of  prior attrs and do some cleaning,
- `Position.first_clear_dt` we no longer sort during insert.
- `._clears` now replaces by `._events` table.
- drop the now masked `.ensure_state()` method (eventually moved to
  `.calc` submod for maybe-later-use).
- drop `.sym=` from all remaining txns init calls.
- clean out the `Position.add_clear()` method and only add the provided
  txn directly to the `._events` table.

Improve some `Account` docs and interface:
- fill out the main type descr.
- add the backend broker modules as `Account.mod` allowing to drop
  `.brokername` as input and instead wrap as a `@property`.
- make `.update_from_trans()` now a new `.update_from_ledger()` and
  expect either of a `TransactionLedger` (user-dict) or a dict of txns;
  in the latter case if we have not been also passed a symcache as input
  then runtime error since the symcache is necessary to allocate
  positions.
  - also, delegate to `TransactionLedger.iter_txns()` instead of
    a manual datetime sorted iter-loop.
  - drop all the clears datetime don't-insert-if-earlier-then-first
    logic.
- rename `.to_toml()` -> `.prep_toml()`.
- drop old `PpTable` alias.
- rename `load_pps_from_ledger()` -> `load_account_from_ledger()` and
  make it only deliver the account instance and also move out all the
  `polars.DataFrame` related stuff (to `.calc`).

And tweak some account clears table formatting,
- store datetimes as TOML native equivs.
- drop `be_price` fixing.
- obvsly drop `.ensure_state()` call to pps.
2023-07-12 08:45:55 -04:00
Tyler Goodlet 0e94e89373 Finally, just drop `Transaction.sym`
Turns out we don't really need it directly for most "txn processing" AND
if we do it's usually related to some `Account`-ing related calcs; which
means we can instead just rely on the new `SymbologyCache` lookup to get
it when needed. So, basically just get rid of it and rely instead on the
`.fqme` to be the god-key to getting `MktPair` info (from the cache).

Further, extend the `TransactionLedger` to contain much more info on the
pertaining backend:
- `.mod` mapping to the (pkg) py mod.
- `.filepath` pointing to the actual ledger TOML file.
- `_symcache` for doing any needed asset or mkt lookup as mentioned
  above.
- rename `.iter_trans()` -> `.iter_txns()` and allow passing in
  a symcache or using the init provided one.
  - rename `.to_trans()` similarly.
- delegate paper account txn processing to the `.clearing._paper_engine`
  mod's `norm_trade()` (and expect this similarly from other backends!)
- use new `SymbologyCache.search()` to find the best but
  un-fully-qualified fqme for a given `txdict` being processed when
  writing a config (aka always try to expand to the most verbose `.fqme`
  possible).
- add a `rewrite: bool` control to `open_trade_ledger()`.
2023-07-12 08:45:55 -04:00
Tyler Goodlet 520414a096 Oof, fix `.size` tick msg encode.. 2023-07-12 08:45:55 -04:00
Tyler Goodlet ddc5f2b441 Use `MktPair.from_msg()` in symcache
Since we now fully support interchange-as-dict-msg, use the msg codec
API and drop manual `Asset` unpacking. Also, wrap `get_symcache()` in
a `pdbp` crash handler block for now B)
2023-07-12 08:45:55 -04:00
Tyler Goodlet 3994fd8384 Also handle `Decimal` interchange in `MktPair` msg-ification 2023-07-12 08:45:55 -04:00
Tyler Goodlet 13f231b926 Decode cached mkts and assets back to structs B)
As part of loading the cache we can now fill the asset sub-tables:
`.mktmaps` and `.assets` with their deserialized struct instances!
In theory this might be possible for the backend defined `Pair` structs
as well but we need to figure out probably an endpoint to offer
the conversion?

Also, add a `SymbologyCache.search()` which allows sync code to scan the
existing (known via cache) symbol set just like how async code can use the
(much slower) `open_symbol_search()` ctx endpoint 💥
2023-07-12 08:45:55 -04:00
Tyler Goodlet 309b91676d Finally, support full `MktPair` + `Asset` msgs
Previously we weren't necessarily serializing mkt pairs (for IPC msging)
entirely bc the assets `.src/.dst` were being sent just by their
str-names. This now properly supports fully serializing `Asset`s as
`dict`-msgs such that use of `MktPair.to_dict()` can be transmitted over
`tractor.MsgStream`s and deserialized entirely back to struct from on
the receiver end.

Deats:
- implement `Asset.to_dict()` and `.from_msg()`
- adjust `MktPair.to_dict()` and `.from_msg()` to use these methods.
  - drop all the hacky "if .src/.dst is str" handling.
- add better `MktPair.from_fqme()` input handling for expiry and venue;
  ensure that either can be extracted from passed fqme *and* if so they
  are also popped from any duplicate passed in `**kwargs**`.
2023-07-12 08:45:55 -04:00
Tyler Goodlet c8c28df62f Much (much) better symbology cache refinements
For starters rename the cache type to `SymbologyCache` and fill out its
interface to include an (async) `.reload()` which can be used to populate
the in-mem asset-table sets such that any tractor-runtime task can
actually directly call it. Use a symcache file name schema of
`_cache/<backend>.symcache.toml`.

Dirtier deatz:
- make `.open_symcache()` a `@cm` such that it can be used from sync code
  and will actually call `trio.run()` in the case where it needs to do a
  full (re)load; also don't write on exit only on reloads.
- add `.get_symcache()` a simple non-ctx-mngr reader which again can
  mostly be called willy-nilly from sync code without the full runtime
  being up (but likely will only work if symcache files already exist
  for the backend).
2023-07-12 08:45:55 -04:00
Tyler Goodlet 005023275e Add a symbology cache subsys
New mod is `.data._symcache` and it needs backend clients to declare
`Client.get_assets()` and `.get_mkt_pairs()` to generate the cache files
which now go in the config dir under `_cache/`.
2023-07-12 08:45:55 -04:00
Tyler Goodlet 05af2b3e64 Rework `.accounting.Position` calcs to prep for `polars`
We're probably going to move to implementing all accounting using
`polars.DataFrame` and friends and thus this rejig preps for a much more
"stateless" implementation of our `Position` type and its internal
pos-accounting metrics: `ppu` and `cumsize`.

Summary:
- wrt to `._pos.Position`:
  - rename `.size`/`.accum_size` to `.cumsize` to be more in line
    with `polars.DataFrame.cumsum()`.
  - make `Position.expiry` delegate to the underlying `.mkt: MktPair`
    handling (hopefully) all edge cases..
  - change over to a new `._events: dict[str, Transaction]` in prep
    for #510 (and friends) and enforce a new `Transaction.etype: str`
    which is by default `clear`.
  - add `.iter_by_type()` which iterates, filters and sorts the
    entries in `._events` from above.
  - add `Position.clearsdict()` which returns the dict-ified and
    datetime-sorted table which can more-or-less be stored in the
    toml account file.
  - add `.minimized_clears()` a new (and close) version of the old
    method which always grabs at least one clear before
    a position-side-polarity-change.
  - mask-drop `.ensure_state()` since there is no more `.size`/`.price`
    state vars (per say) as we always re-calc the ppu and cumsize from
    the clears records on every read.
  - `.add_clear` no longer does bisec insorting since all sorting is
    done on position properties *reads*.
  - move the PPU (price per unit) calculator to a new `.accounting.calcs`
    as well as add in the `iter_by_dt()` clearing transaction sorted
    iterator.
    - also make some fixes to this to handle both lists of `Transaction`
      as well as `dict`s as before.

- start rename of `PpTable` -> `Account` and make a note about adding
  a `.balances` table.
- always `float()` the transaction size/price values since it seems if
  they get processed as `tomlkit.Integer` there's some suuper weird
  double negative on read-then-write to the clears table?
  - something like `cumsize = -1` -> `cumsize = --1` !?!?
- make `load_pps_from_ledger()` work again but now includes some very
  very first draft `polars` df processing from a transaction ledger.
  - use this from the `accounting.cli.disect` subcmd which is also in
    *super early draft* mode ;)
- obviously as mentioned in the `Position` section, add the new `.calcs`
  module with a `.ppu()` calculator func B)
2023-07-12 08:45:55 -04:00
Tyler Goodlet 745c144314 ib.feed: handle fiat (forex) pairs with `Asset`
Also finally adds full `FeedInit` and `MktPair` support for this backend
by handling:
- all "currency" fields for each `Contract` by constructing
  and `Asset` and setting the `MktPair.src` with `.atype='fiat'`.
- always render the `MktPair.src` name in the `.fqme` for fiat pairs
  (aka forex) but never for other instruments.
2023-07-12 08:45:55 -04:00
Tyler Goodlet 10ebc855e4 ib: fully handle `MktPair.src` and `.dst` in ledger loading
In an effort to properly support fiat pairs (aka forex) as well as more
generally insert a fully-qualified `MktPair` in for the
`Transaction.sys`. Note that there's a bit of special handling for API
`Contract`s-as-dict records vs. flex-report-from-xml equivalents.
2023-07-12 08:45:55 -04:00
Tyler Goodlet c0929c042a ib: fix `Client.trades()` return type annot 2023-07-12 08:45:55 -04:00
Tyler Goodlet 9748b22d34 Always include the src asset for (parquet file names) for fiat pairs 2023-07-12 08:45:55 -04:00
Tyler Goodlet 3ff9fb3e10 clearing._messages: add todo to drop the `BrokedPosition` msg 2023-07-12 08:45:55 -04:00
Tyler Goodlet 75f01e22d7 Drop `Position.expiry`, delegate to `.mkt: MktPair`
No point having duplicate data when we already stash the `expiry` on the
mkt info type and can just read it (and cast to `datetime` obj).

Further this fixes a regression caused by converting `._clears` to
a list by adding a `._events: dict[str, Transaction]` which prevents
double entering transactions based on checking the events table for the
existing id.. Further add a sanity check that all events are popped
(for now) after serializing the clearing table for the toml account
file.

In the longer run, ideally we don't have the separate sequences ._clears
and ._events by choosing a better data structure (sorted unique set of
mkt events) maybe a specially used `polars.DataFrame` (which we kind
need eventually anyway)?
2023-07-12 08:45:55 -04:00
Tyler Goodlet 87d6115954 Add src asset name ignore via `MktPair._fqme_without_src: bool` 2023-07-12 08:45:55 -04:00
Tyler Goodlet c780164f69 Fix test to use new `load_account()` location 2023-07-12 08:45:55 -04:00
Tyler Goodlet 482403c887 Expose `.accounting.load_account()` 2023-07-12 08:45:55 -04:00
Ebisu 2ac8191722 discrepancy between live/testnet urls 2023-07-12 01:49:17 +02:00
Tyler Goodlet 35af5f11fa binance: Map `use_testnet` to off by default (since data feeds) 2023-06-30 20:20:14 -04:00
Tyler Goodlet a7ec59862a binance: Map `use_testnet` to off by default (since data feeds) 2023-06-30 20:17:02 -04:00
Tyler Goodlet ad4847cbac basic bot: iter latest ticks first to decide new submission price per quote 2023-06-27 15:47:23 -04:00
Tyler Goodlet da07685e8b Use `iterticks()` to filter to clears, get first price manually before submit.. 2023-06-27 15:47:23 -04:00
Tyler Goodlet f1eb76d29f Drop prints, break on latest clear match tick 2023-06-27 15:47:23 -04:00
Tyler Goodlet 46b22958f0 basic bot: add real-time price trailer (task) that keeps bid price 0.0005% below last clear value 2023-06-27 15:47:23 -04:00
Tyler Goodlet 57399e4f5d basic bot: drop registry addr and connect to default pikerd 2023-06-27 15:47:23 -04:00
Tyler Goodlet 5690595064 basic bot: set unix fileformat, add KBI handling to cancel order submission 2023-06-27 15:47:23 -04:00
Tyler Goodlet 63a6c6efde Add a super basic "order bot" example B)
Shows how to boot the piker runtime, submit an order to the ems, cancel
said order right away. NOTE, this uses piker's built in paper engine but
can be easily tweaked to use a live backend at the user's whim.
2023-06-27 15:47:23 -04:00
Tyler Goodlet f2fff5a5fa ib._ledger: move trades transaction processing helpers into new module 2023-06-27 15:47:05 -04:00
Tyler Goodlet c0d575c009 Change `Position.clears` -> `._clears[list[dict]]`
When you look at usage we don't end up really needing clear entries to
be keyed by their `Transaction.tid`, instead it's much more important to
ensure the time sorted order of trade-clearing transactions such that
position properties such as the size and ppu are calculated correctly.
Thus, this instead simplified the `.clears` table to a list of clear
dict entries making a bunch of things simpler:
- object form `Position._clears` compared to the offline TOML schema
  (saved in account files) is now data-structure-symmetrical.
- `Position.add_clear()` now uses `bisect.insort()` to
  datetime-field-sort-insert into the *list* which saves having to worry
  about sorting on every sequence *read*.

Further deats:
- adjust `.accounting._ledger.iter_by_dt()` to expect an input `list`.
- change `Position.iter_clears()` to iterate only the clearing entry
  dicts without yielding a key/tid; no more tuples.
- drop `Position.to_dict()` since parent `Struct` already implements it.
2023-06-27 15:47:05 -04:00
Tyler Goodlet 66d402b80e Load ledger records into `pl.DataFrame` for `disect`-tion 2023-06-27 15:47:05 -04:00
Tyler Goodlet ea270d3396 .data.ticktools: add reverse flag, better docs
Since it may be handy to get the latest ticks first, add a `reverse:
bool` to `iterticks()` and add some cleaner logic and a proper doc
string to `frame_ticks()`.
2023-06-27 15:47:05 -04:00
Tyler Goodlet 621634b5a2 Move `frame_ticks()` and tick-type defs into `.ticktools` 2023-06-27 15:47:05 -04:00
Tyler Goodlet eacc59226f rename `.data._normalize` -> `.ticktools` 2023-06-27 15:47:05 -04:00
Tyler Goodlet 7b4472e37e data._sampling.frame_ticks(): slight rework to generalize 2023-06-27 15:47:05 -04:00
Tyler Goodlet 4a8eafabb8 Never key error on bad flow pops.. 2023-06-27 13:48:03 -04:00
Tyler Goodlet e7e7919a43 Ensure paper engine logger is `piker.clearing` instance.. 2023-06-27 13:48:03 -04:00
Tyler Goodlet cdf9105d0d Export `Flume` and `Feed` from `piker.data` 2023-06-27 13:48:03 -04:00
Tyler Goodlet 49e67d5f36 Always add a paper (account) entry to order mode init
Allows for tracking paper engine orders despite the ems not necessarily
being opened by the current order mode instance (UI) in "paper"
execution mode; useful for tracking bots/strats running against the same
EMS daemon.
2023-06-27 13:48:03 -04:00
Tyler Goodlet 85fa87fe6f Update the `_emsd_main()` doc task tree layout 2023-06-27 13:48:03 -04:00
Tyler Goodlet 249b091c2f binance: better bad account in order request error msg 2023-06-27 13:48:03 -04:00
Tyler Goodlet 2d291bd2c3 ib: expose `.broker.norm_trade_records()` from pkg 2023-06-27 13:42:08 -04:00
Tyler Goodlet cf1f4bed75 Move `.accounting` related config loaders to subpkg
Like you'd think:
- `load_ledger()` -> ._ledger
- `load_accounrt()` -> ._pos

Also fixup the old `load_pps_from_ledger()` and expose it from a new
`.accounting.cli.disect` cli cmd for trying to figure out why pp calcs
are totally mucked on stupid ib..
2023-06-27 13:42:08 -04:00