Commit Graph

3376 Commits (d0ba9a0a587ff9815307179e288525304063b226)

Author SHA1 Message Date
Tyler Goodlet aa5f25231a `ib`: never override existing ledger records
If user has loaded from a flex report then we don't want the API records
from the same period to override those; instead just update with any
missing fields from the API schema.

Also, always `str`-ify the contract id (what is set for the `.bs_mktid`
*before* packing into transaction type to ensure when serialized to
`pps.toml` there are no discrepancies at the codec level.. smh
2023-05-09 14:49:26 -04:00
Tyler Goodlet f3049016d6 `ib`: drop use of `_account2clients` in `load_clients_for_trio()`
Instead adjust `load_aio_clients()` to only reload clients detected as
non-loaded or disconnected (2 birds), and avoid use of the global module
table which could result in stale disconnected clients persisting on
multiple `brokerd` client reconnects, resulting in error.
2023-05-09 14:49:26 -04:00
Tyler Goodlet 16e11d447c Move toml table decoder to separate mod 2023-05-09 14:49:26 -04:00
Tyler Goodlet 199a5e8b38 `ib`: stick exc handler around client connection erros 2023-05-09 14:49:26 -04:00
Tyler Goodlet 59b095b2d5 `kraken`: heh, use `trio_util` for trades streamz tooo XD 2023-05-09 14:49:26 -04:00
Tyler Goodlet c59ec77d9c WIP: refactor ib pp load init 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3e5da64571 Cache contract lookups from `Client.get_con()` 2023-05-09 14:49:26 -04:00
Tyler Goodlet 1c576d72d1 Dump `Position`s as pformatted dicts for now.. 2023-05-09 14:49:26 -04:00
Tyler Goodlet ea42f66b54 Use common `.brokers` logger in most backends 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2ae9576cd8 Add common logger instance for `.brokers` 2023-05-09 14:49:26 -04:00
Tyler Goodlet a462de6f2d Use a single log for entire `.service` subsys 2023-05-09 14:49:26 -04:00
Tyler Goodlet 3bf48ab597 Use a single log for entire `.clearing` subsys 2023-05-09 14:49:26 -04:00
Tyler Goodlet 2454dda18f Use `MktPair` attr `.size_tick` in charting 2023-05-09 14:49:26 -04:00
Tyler Goodlet 7498cbb5f4 Use `Struct.copy()` with update dict for `Order` from staged 2023-05-09 14:49:25 -04:00
Tyler Goodlet 581782800d Rename `Client.send_update()` -> `.update_nowait()` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 069466218e Use `str(cmd.symbol)` for fqme on cancels, add `_nowait()` method names 2023-05-09 14:49:25 -04:00
Tyler Goodlet fd9e484b55 Add `.__str__()` to mktpair and symbol types, fix `MktPair.fqme` token order 2023-05-09 14:49:25 -04:00
Tyler Goodlet 406565f74d Rename `fqsn` -> `fqme` in paper engine 2023-05-09 14:49:25 -04:00
Tyler Goodlet 6272cae8d4 Drop more `Optional` usage on our `Struct` 2023-05-09 14:49:25 -04:00
Tyler Goodlet dc2332c980 '`kraken`: finally, use new `MktPair` in `'mkt_info'` init msg field!' 2023-05-09 14:49:25 -04:00
Tyler Goodlet 7be85a882b Drop use of legacy `Symbol.broker_info` in display startup 2023-05-09 14:49:25 -04:00
Tyler Goodlet b6df83a0e9 Typecast `OrderMode.staged.symbol: str` before `.copy()`! 2023-05-09 14:49:25 -04:00
Tyler Goodlet d62fb655eb `kraken`: parse our source asset key and set on `MktPair.src: str` 2023-05-09 14:49:25 -04:00
Tyler Goodlet a9778e4001 Always cast `Order.symbol: str` for now
To make nested `msgspec.Struct`s work we need to tell the codec that the
`.symbol` is some struct def, since we don't really need to enforce that
(yet) we're just going to enc/dec as `str` until we further formalize
and/or need something more complex.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 580165f2f4 Expect new `MktPair.tick_size: Decimal` attr in ems 2023-05-09 14:49:25 -04:00
Tyler Goodlet 0f3041724b Use `MktPair` for `Flume.symbol` when used by backend
Initial attempt at getting the sampling and shm layer to use the new mkt
info meta-data type. Draft out a potential `BackendInitMsg:
msgspec.Struct` for validating the init msg returned from the
`stream_quotes()` start value; obvs don't actually use it yet.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 1d08ee6d01 `.clearing`: broad rename of `fqsn` -> `fqme` 2023-05-09 14:49:25 -04:00
Tyler Goodlet d4a5a3057c Add `MktPair.suffix: str` read from contract info
To be compat with the `Symbol` (for now) and generally allow for reading
the (derivative) contract specific part of the fqme. Adjust
`contract_info: list[str]` and make `src: str = ''` by default.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 452cd7db8a Optionally load `MktPair` in `Flume`s 2023-05-09 14:49:25 -04:00
Tyler Goodlet 2cc80d53ca First stage port of `.data.feed` to `MktPair`
Add `MktPair` handling block for when a backend delivers
a `mkt_info`-field containing init msg. Adjust the original
`Symbol`-style `'symbol_info'` msg processing to do `Decimal` defaults
and convert to `MktPair` including slapping in a hacky `_atype: str`
field XD

General initial name changes to `bs_mktid` and `_fqme` throughout!
2023-05-09 14:49:25 -04:00
Tyler Goodlet 7eb0b1d249 Comment about `Struct.typecast()` conflict with frozen instances 2023-05-09 14:49:25 -04:00
Tyler Goodlet 589b3f4201 Default `pps.toml` precision fields to `Decimal`
For `price_tick` and `size_tick` we read in `str` and decimal-ize
and now correctly fail over to default values of the same type..
Also, always treat `bs_mktid` field as a `str` in TOML form.

Drop the strange `clears: dict` var from the loading code (not sure why
that was left in smh) and better name `toml_clears_list` for the
TOML-loaded-pre-transaction sequence.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 6d5d9731ed Implement `MktPair.from_msg()` constructor
Handle case where `'dst'` field is just a `str` (in which case delegate to
`.from_fqme()`) as well as do `Asset` loading and use our
`Struct.copy()` to enforce type-casting to (for eg. `Decimal`s) such
that we'll now capture typing errors despite IPC transport.

Change `Symbol.tick_size` and `.lot_tick_size` defaults to decimal
for proper casting and type `MktPair.atype: str` since `msgspec` can't
cast to `AssetTypeName` without special handling..
2023-05-09 14:49:25 -04:00
Tyler Goodlet 25363ebd2e `ib`: deliver mkt precision info as `Decimal` 2023-05-09 14:49:25 -04:00
Tyler Goodlet b9c7e1b0c7 `binance`: deliver mkt precision info as `Decimal` 2023-05-09 14:49:25 -04:00
Tyler Goodlet ea9ea4a6d7 Rename `float_digits()` -> `dec_digits()`, since decimal. 2023-05-09 14:49:25 -04:00
Tyler Goodlet 76cd5519b3 Fix `Symbol.tick_size_digits`, add `.price/size_tick` props 2023-05-09 14:49:25 -04:00
Tyler Goodlet 677a6fc113 Cast to float from decimal for level line y-increment
Qt only accepts `float` to it's APIs obvs..
2023-05-09 14:49:25 -04:00
Tyler Goodlet 99199905b6 Add parity mapping from altnames back to themsevles in `Client._ntable` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 55b6cba31e Encode a `mktpair` field if passed in msg by caller 2023-05-09 14:49:25 -04:00
Tyler Goodlet 17b976eb88 Use `MktPair` building `Position` objects in `PpTable.update_from_trans()` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 7a8e615fa6 Explicitly decode tick sizes as decimal for symbol loading in `Flume` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 335e8d10d4 Cast back to float from decimal for cursor y-increment 2023-05-09 14:49:25 -04:00
Tyler Goodlet 6431071b2a Pass old fields in sym info init msg section 2023-05-09 14:49:25 -04:00
Tyler Goodlet 8fdff8769d Ensure `Symbol` tick sizes are decoded as `Decimal`.. 2023-05-09 14:49:25 -04:00
Tyler Goodlet 66782d29d1 `kraken`: use `Client.mkt_info()` in quotes feed init msg 2023-05-09 14:49:25 -04:00
Tyler Goodlet cfbba9e0b3 Add `MktPair._atype` for back-compat, always `str(.dst)` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 7aba290541 `kraken`: use `MktPair` in trasactions 2023-05-09 14:49:25 -04:00
Tyler Goodlet da10422160 `kraken`: add `Client.mkt_info()`
Allows building a `MktPair` from the backend specific `Pair` for
eventual use in the data feed layer. Also adds `Pair.price/tick_size` to
get to the expected tick precision info format.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 9e2eff507e Drop shm logging levels to debug over warning 2023-05-09 14:49:25 -04:00
Tyler Goodlet 71fc8b95dd Flip to `.bs_mktid` in `ib` and `kraken` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 72c97d4672 Handle read and write of `pps.toml` using `MktPair`
Add a logic branch for now that switches on an instance check.
Generally swap over all `Position.symbol` and `Transaction.sym` refs to
`MktPair`. Do a wholesale rename of all `.bsuid` var names to
`.bs_mktid`.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 7b28c7a43f Prep for dropping `Transaction.sym`
Instead let's name it `.sys` for "system", the thing we use to conduct
the "transactions" ..

Also rename `.bsuid` -> `.bs_mktid` for "backend system market id`
which is more explicit, easier to remember and read.
2023-05-09 14:49:25 -04:00
Tyler Goodlet cf9442f4d5 Further refinement and shimming of `MktPair`
Prepping to entirely replace `Symbol`; this adds a buncha docs/comments,
better implementation for representing and parsing the FQME: "fully
qualified market endpoint".

Deatz:
- make `.src` an optional field until we figure out how we're going
  to support loading source assets from all backends sensibly..
- implement `MktPair.fqme: str` (what was previously called `fqsn`)
  using a new util func: `maybe_cons_tokens()`.
- `Symbol.brokers` and expect only `.broker` usage.
- remap anything with `fqsn` in the name to `fqme` with aliases from the
  old name.
- implement `unpack_fqme()` with `match:` syntax B)
- add `MktPair.tick_size_digits`, `.lot_size_digits`, `.fqsn`, `.key` for
  backward compat.
- make all fqme generation related fields empty `str`s by default.
- add `MktPair.resolved: bool` a flag indicating whether or not `.dst`
  is an `Asset` instance or just a string and, `.bs_mktid` the field
  to hold the "backend system market id" per broker.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 85ddfc0f2d Drop use of `mk_fqsn()` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 56f736e7ca Drop use of `Symbol.brokers` everywhere 2023-05-09 14:49:25 -04:00
Tyler Goodlet 63304f535c Start to prep `Transaction` for `MktPair`.. 2023-05-09 14:49:25 -04:00
Tyler Goodlet 2583706b35 Port `accounting._pos` to new `Symbol` simplifications 2023-05-09 14:49:25 -04:00
Tyler Goodlet 65a7853cf3 Delegate to new `.accounting._mktinfo._derivs` from `ui._positioning` 2023-05-09 14:49:25 -04:00
Tyler Goodlet 69c9ecc5e3 `kraken`: write `pps.toml` on updates for now 2023-05-09 14:49:25 -04:00
Tyler Goodlet 3be53540c1 `kraken`: pack `Asset` into local client cache
Try out using our new internal type for storing info about kraken's asset
infos now stored in the `Client.assets: dict[str, Asset]` table. Handle
a server error when requesting such info msgs.
2023-05-09 14:49:25 -04:00
Tyler Goodlet a44b6f7c2f `ib`: adjust to new simplified `Symbol`
Drop usage of removed methods and attrs and only pass in the
`.tick_size: Decimal` value during construction.
2023-05-09 14:49:25 -04:00
Tyler Goodlet e65f3f84b9 Drop `Symbol.front_fqsn()` usage from chart, fsp and clearing stuff 2023-05-09 14:49:25 -04:00
Tyler Goodlet acc5af1fdb Drop `Symbol.front_feed()` usage from order mode 2023-05-09 14:49:25 -04:00
Tyler Goodlet 91dda3020e Simplify `Symbol` extend `MktPair`, add `Asset`
Drop everything we can in terms of methods and attrs from `Symbol`:
- kill `.tokens()`, `.front_feed()`, `.tokens()`, `.nearest_tick()`,
  `.front_fqsn()`, instead moving logic from these methods into
  dependents (and obviously removing any usage from rest of code base,
  coming in follow up commits).
- rename `.quantize_size()` -> `.quantize()`.
- re-implement `.brokers`, `.lot_size_digits`, `.tick_size_digits` as
  `@property` methods; for the latter two, allows us to minimize to only
  accepting min tick decimal values on alternative constructor class
  methods and to drop the equivalent instance vars.
- map `_fqsn` related variable names to new and preferred `_fqme`.

We also juggle around some utility functions, moving limited precision
related `decimal.Decimal` routines to the top of module and soon-to-be
legacy `fqsn` related routines to the bottom.

`MktPair` draft type extensions:
- drop requirements for `src_type`, and offer the optional `.dst_type`
  field as either a `str` or (new `typing.Literal`) `AssetTypeName`.
- define an equivalent `.quantize()` as (re)defined in `Symbol` but with
  `quantity_type: str` field which specifies whether to use the price or
  the size precision.
- add a lot more docs, a `.key` property for the "symbol" name, draft
  property for a `.fqme: str`
- allow `.src` and `.dst` to be of type `str | Asset`

Add a new `Asset` to capture "things which can be used in markets and/or
transactions" XD
- defines a `.name`, `.atype: AssetTypeName` a financial category tag, `tx_tick:
  Decimal` the precision limit for transactions and of course
  a `.quantime()` method for doing accounting arithmetic on a given tech
  stack.
- define the `atype: AssetTypeName` type as a finite set of `str`s
  expected to be used in various ways for default settings in other
  parts of the data and order control layers..
2023-05-09 14:49:25 -04:00
Tyler Goodlet 9f03484c4d Move all fqsn parsing and `Symbol` to new `accounting._mktinfo 2023-05-09 14:49:25 -04:00
Tyler Goodlet 22622e1c01 `ib`: (cukcit) just presume a stonk if we can read type from existing ledger.. 2023-05-09 14:49:25 -04:00
Tyler Goodlet f549de7c88 Break out old `.pp` components into submods: `._ledger` and `._pos` 2023-05-09 14:49:25 -04:00
Tyler Goodlet beb6544bad Start a new `.accounting` subpkg, move `.pp` contents there 2023-05-09 14:49:25 -04:00
Tyler Goodlet d01fdbf981 '`kraken`: fix pos loading using `digits_to_dec()` to pair info
Our issue was not having the correct value set on each
`Symbol.lot_tick_size`.. and then doing PPU calcs with the default set
for legacy mkts..

Also,
- actually write `pps.toml` on broker mode exit.
- drop `get_likely_pair()` and import from pp module.
2023-05-09 14:49:25 -04:00
Tyler Goodlet badc30baae Add an inverse of `float_digits()`: `digits_to_dec() 2023-05-09 14:49:25 -04:00
Tyler Goodlet 4f36a03df2 Ensure clearing table entries are time-sorted..
Not sure how this worked before but, the PPU calculation critically
requires that the order of clearing transactions are in the correct
chronological order! Fix this by sorting `trans: dict[str, Transaction]`
in the `PpTable.update_from_trans()` method.

Also, move the `get_likely_pair()` parser from the `kraken` backend here
for future use particularly when we revamp the asset-transaction
processing layer.
2023-05-09 14:49:25 -04:00
Tyler Goodlet 0d9acb1cb0 numpy: drop `numpy.float` in py311 2023-05-04 12:01:59 -04:00
jaredgoldman 3836f7d458 Run autopep8, add default case for message stream match case 2023-04-21 21:16:14 -04:00
jaredgoldman ae3f6696a7 Fix type hinting for stream_messages return type 2023-04-21 20:40:23 -04:00
jaredgoldman a06a4f67cc Remove unused timeframe var from open_history_client 2023-04-21 17:17:47 -04:00
jaredgoldman a69c8a8b44 Uncomment loglevel 2023-04-20 18:51:13 -04:00
jaredgoldman efad49ec5b Raise ValueError if no config is found when sending authenticated headers 2023-04-19 14:58:28 -04:00
jaredgoldman d772fe45c0 Comment out unused args 2023-04-19 14:55:58 -04:00
jaredgoldman 6f91c2932d Type bars data dict 2023-04-19 14:49:28 -04:00
jaredgoldman d07a73cf70 Add type annotation for open_ping_task' 2023-04-19 14:47:19 -04:00
jaredgoldman fcdddadec1 Use singlequotes 2023-04-18 10:42:30 -04:00
jaredgoldman 9fcfb8d780 More linting fixes 2023-04-18 10:39:47 -04:00
jaredgoldman 37ce04ca9a Linting fixes 2023-04-18 10:19:59 -04:00
jaredgoldman a109a8bf67 Add linting fixes 2023-04-18 09:51:50 -04:00
jaredgoldman b01771be1b Add comments to kucoin->piker bar conversion 2023-04-16 10:46:22 -04:00
jaredgoldman 0e4095c947 Don't yield ws from the ping task 2023-04-16 10:45:05 -04:00
jaredgoldman dae56baeba Refactor streaming logic to be less nested and readable 2023-04-16 10:12:29 -04:00
jaredgoldman 9706803220 Refactor streaming logic to be less nested and readable 2023-04-16 10:11:17 -04:00
jaredgoldman 8403d8a482 Simplify numpy mapping logic 2023-04-15 21:05:25 -04:00
Tyler Goodlet a111819667 Few fixes after review to get running again B)
- use `Struct.copy()` for frozen type
- fix `BrokerConfig` delegation attr lookups
- bit of linting according to `flake8`
2023-04-14 19:05:19 -04:00
jaredgoldman 4f576b6f36 Fix typo with ts vars 2023-04-13 22:37:17 -04:00
jaredgoldman 672c01f13a Use trade_data_ts for trade message receival 2023-04-13 22:35:21 -04:00
jaredgoldman f67ffeb70f Remove extra Noen check on msg.get 2023-04-13 22:34:04 -04:00
jaredgoldman 1b1e35d32d Add comment explaining waiting for first trade quote 2023-04-13 22:28:44 -04:00
jaredgoldman 9f5dfe8501 Remove anext() comment 2023-04-13 22:27:56 -04:00
jaredgoldman 11bd2e2f65 Use datetime | none instead of Optional[datetime] in get_bars 2023-04-13 22:04:43 -04:00
jaredgoldman ebfd490a1a Cache instead of get pairs in symbol search 2023-04-13 22:02:13 -04:00
jaredgoldman 89bb124728 Remove old comments normalize arguents and improve pair fetching log 2023-04-13 22:00:41 -04:00
jaredgoldman 63e34cf595 Typecast config, add type hint to pair in init message creation and turn init msg vals into floats 2023-04-13 21:57:54 -04:00
jaredgoldman 92f372dcc8 Use proper value for init message 2023-04-13 21:52:40 -04:00
jaredgoldman b00abd0e51 Add a fail case ws token request 2023-04-13 21:48:17 -04:00
jaredgoldman 52a015d927 Remove typo in binance 2023-04-12 21:40:58 -04:00
jaredgoldman 2c82b2aba9 Remove breakpoint in binance 2023-04-12 20:43:28 -04:00
jaredgoldman ff0f8dfaca Improve client._get_ws_token docstring 2023-04-12 20:37:10 -04:00
jaredgoldman ace04af21a Use anext() in kucoin stream_quotes 2023-04-12 20:25:35 -04:00
jaredgoldman d2f3a79c09 Use pendulum for header timestamp,
type hint cleanup
2023-04-12 19:48:46 -04:00
jaredgoldman bedbbc3025 Only diff trade time 2023-04-12 19:48:46 -04:00
jaredgoldman 6e55f6706f Format condition for filtering and add link to docs explaining need for filtering in the first case 2023-04-12 19:48:46 -04:00
jaredgoldman d1b0608c88 Remove breakpoint 2023-04-12 19:48:46 -04:00
jaredgoldman 3bed3a64c3 Implement duplicate filtering at message level 2023-04-12 19:48:46 -04:00
jaredgoldman 93e7d54c5e Add api doc links to _get_bars def 2023-04-12 19:48:46 -04:00
jaredgoldman 9db84e8029 Remove norm_pairs method and do all normalization in initial _get_pairs call 2023-04-12 19:48:46 -04:00
jaredgoldman ea21656624 Don't cache pairs in _get_pairs call 2023-04-12 19:48:46 -04:00
jaredgoldman 5a0d29c774 Add ws token api doc link 2023-04-12 19:48:46 -04:00
jaredgoldman 13df3e70d5 Refactor sign gen into one line 2023-04-12 19:48:46 -04:00
jaredgoldman 208a8e5d7a Remove unecessary config vars 2023-04-12 19:48:46 -04:00
jaredgoldman ca937dff5e Add api doc links in structs 2023-04-12 19:48:46 -04:00
jaredgoldman c68fcf7e1c Remove extra line from docstrings 2023-04-12 19:48:46 -04:00
jaredgoldman 48c3b333b2 Format imports with parenthesis 2023-04-12 19:48:46 -04:00
jaredgoldman b71f6b6c67 Strip uneccesary data from ticks in l1 data feed 2023-04-12 19:48:46 -04:00
jaredgoldman 54cf648d74 Ensure sub logging dict attritbutes will be there 2023-04-12 19:48:46 -04:00
jaredgoldman 68d0327d41 Remove breakpoints, simplify backoff logic 2023-04-12 19:48:46 -04:00
jaredgoldman 68a06093e9 Format and ensure we're only grabbing the most closest bid and ask 2023-04-12 19:48:46 -04:00
jaredgoldman 52aadb374b Add L1 data feed and correct history issue 2023-04-12 19:48:46 -04:00
jaredgoldman dfd030a6aa Remove float conversion of key_id again 2023-04-12 19:48:46 -04:00
jaredgoldman 788e158d9f Stop still converting datetime to float 2023-04-12 19:48:46 -04:00
jaredgoldman 81890a39d9 Leave datetimes alone! 2023-04-12 19:48:46 -04:00
jaredgoldman ae170f2645 Add more informative logs on startup 2023-04-12 19:48:46 -04:00
jaredgoldman e2e5191ded Remove breaking useless condition for determining if res is list of ohlc values 2023-04-12 19:48:46 -04:00
jaredgoldman dcbb7fa64f Remove float conversion for config key id 2023-04-12 19:48:46 -04:00
jaredgoldman 32107d0ac3 Strengthen retry case and add comments 2023-04-12 19:48:46 -04:00
jaredgoldman 7bdebd47d1 Add exponential retry case for history client 2023-04-12 19:48:46 -04:00
jaredgoldman ac31bca181 Make broker creds/auth optional 2023-04-12 19:48:46 -04:00
jaredgoldman 52070c00f9 Remove typo 2023-04-12 19:48:46 -04:00
jaredgoldman 5ff0cc7905 Cast/validate streamed messages
Update comments

Minor formatting

Minor formatting
2023-04-12 19:48:46 -04:00
jaredgoldman 6ad1e3da38 Correct typo in license 2023-04-12 19:48:46 -04:00
jaredgoldman 9bf6f557ed Label private methods accordingly, remove cryptofeeds module 2023-04-12 19:48:46 -04:00
jaredgoldman 50e1070004 More cleanup, add comments re sub func 2023-04-12 19:48:46 -04:00
jaredgoldman 1c4c19b351 Clean up broker code,
Add typecasting for messages/rt-data and historcal user trades
ensure we're fetching all history
add multi-symbol support
'
2023-04-12 19:48:46 -04:00
jaredgoldman 199a70880c Spawn background ping task 2023-04-12 19:48:46 -04:00
jaredgoldman b14b323068 Remove breakpoint in web_bs,
ensure we only unsub if ws is connected
2023-04-12 19:48:46 -04:00
jaredgoldman a3c7bec576 Implement working message streaming 2023-04-12 19:48:46 -04:00
jaredgoldman ac34ca7cad Add sub method to flow
Stash for checkout of master
2023-04-12 19:48:46 -04:00
jaredgoldman ade2c32adb Succesfully connect to kucoin ws 2023-04-12 19:48:46 -04:00
jaredgoldman 109e7d7b43 Add back static API version in headers 2023-04-12 19:48:46 -04:00
jaredgoldman 1a655b7e39 Ensure we're passing the correct api version to the header builder,
make headers a default arg
2023-04-12 19:48:46 -04:00
jaredgoldman cda045f123 Abstract header gen to seperate function 2023-04-12 19:48:46 -04:00
jaredgoldman 7074ca7713 Implement Kucoin auth and last trades call 2023-04-12 19:48:46 -04:00
Tyler Goodlet 8e91e215b3 WIP - ensure `asyncio` pumps the event loop each send 2023-04-12 19:48:46 -04:00
jaredgoldman c751c36a8b Update trade message format 2023-04-12 19:48:46 -04:00
jaredgoldman ad9d645782 WIP - setup basic history and streaming client 2023-04-12 19:48:46 -04:00
jaredgoldman c96d4387c5 Start adding history client 2023-04-12 19:48:46 -04:00
jaredgoldman 5fdec8012d Add cryptofeeds data feed module,
Add Kucoin backend client
wip
2023-04-12 19:48:46 -04:00
Tyler Goodlet 609b91e848 Try out `@trio_util.async_generator` for streaming
Apparently it will likely fix our `trio`-cancel-scopes-corrupted crash
when we try to let our `._web_bs.NoBsWs` do reconnect logic around
the asyn-generator implemented data-feed streaming routines in `binance`
and `kraken`.  See the project docs for deatz; obvs we add the lib as
a dep.
2023-03-20 12:54:48 -04:00
Tyler Goodlet 78eb784091 Stick `try:` outside all `xdotool` subproc calls 2023-03-13 15:36:45 -04:00
Tyler Goodlet 973e4b5f44 `binance`: wrap streamer async-gen in `aclosing()` 2023-03-13 15:36:29 -04:00
Tyler Goodlet 9197e6decb `binance`: use built-in `anext()` add note about new ws ep URL 2023-03-13 15:36:29 -04:00
Tyler Goodlet 889e920796 Short-circuit rendering on no 1d-data; avoid m4 layer crash 2023-03-13 12:18:54 -04:00
Tyler Goodlet 1aab9f1f81 Actually yes, we need to handle empty in-view range.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5c697de58e Presume never handling not-in-view case for minor curves 2023-03-10 18:20:22 -05:00
Tyler Goodlet 3066b1541e Handle (shorter supported) minor-curve not-in-view
Solve this by always scaling the y-range for the major/target curve
*before* the final overlay scaling loop; this implicitly always solve
the case where the major series is the only one in view.

Tidy up debug print formatting and add some loop-end demarcation comment
lines.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 32339cb41a Always show a minimum bars during startup
This is particularly more "good looking" when we boot with a pair that
doesn't have historical 1s OHLC and thus the fast chart is empty from
outset. In this case it's a lot nicer to be already zoomed to
a comfortable preset number of "datums in view" even when the history
isn't yet filled in.

Adjusts the chart display `Viz.default_view()` startup to explicitly
ensure this happens via the `do_min_bars=True` flag B)
2023-03-10 18:20:22 -05:00
Tyler Goodlet 12e196a6f7 Catch `KeyError` on bcast errors which pop the sub
Not sure how i missed this (and left in handling of `list.remove()` and
it ever worked for that?) after the `samplerd` impl in 5ec1a72 but, this
adjusts the remove-broken-subscriber loop to catch the correct
`set.remove()` exception type on a missing (likely already removed)
subscription entry.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8a87e5f390 Remove leftover debug print in cache reset meth 2023-03-10 18:20:22 -05:00
Tyler Goodlet 5958acebe1 Add (commented) draft 1min OHLC time index logging
For the purposes of eventually trying to resolve last-step indexing
synchronization (an intermittent but still existing) issue(s) that can
happen due to races during history frame query and shm writing during
startup. In fact, here we drop all `hist_viz` info queries from the main
display loop for now anticipating that this code will either be removed
or improved later.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8d1c713a5a Always pass step to `slice_from_time()` in view mode
Again, as per the signature change, never expect implicit time step
calcs from overlay processing/machinery code. Also, extend the debug
printing (yet again) to include better details around
"rescale-due-to-minor-range-out-of-view" cases and a detailed msg for
the transform/scaling calculation (inputs/outputs), particularly for the
cases when one of the curves has a lesser support.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 32926747c6 Always pass `step` to `slice_from_time()` in the `Viz`
As per the change to `slice_from_time()` this ensures this `Viz` always
passes its self-calculated time indexing step size to the time slicing
routine(s).

Further this contains a slight impl tweak to `.scalars_from_index()` to
slice the actual view range from `xref` to `Viz.ViewState.xrange[1]` and
then reading the corresponding `yref` from the first entry in that
array; this should be no slower in theory and makes way for further
caching of x-read-range to `ViewState` opportunities later.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 712f1a47a0 Require `step: float` input to `slice_from_time()`
There's been way too many issues when trying to calculate this
dynamically from the input array, so just expect the caller to know what
it's doing and don't bother with ever hitting the error case of
calculating and incorrect value internally.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 51f3733487 Handle "target-is-shorter-then-pinned" case
When the target pinning curve (by default, the dispersion major) is
shorter then the pinned curve, we need to make sure we find still find
the x-intersect for computing returns scalars! Use `Viz.i_from_t()` to
accomplish this as well and, augment that method with a `return_y: bool`
to allow the caller to also retrieve the equivalent y-value at the
requested input time `t: float` for convenience.

Also tweak a few more internals around the 'loglin_ref_to_curve'
method:
- only solve / adjust for the above case when the major's xref is
  detected as being "earlier" in time the current minor's.
- pop the major viz entry from the overlay table ahead of time to avoid
  a needless iteration and simplify the transform calc phase loop to
  avoid handling that needless cycle B)
- add much better "organized" debug printing with more clear headers
  around which "phase"/loop the message pertains and well as more
  explicit details in terms of x and y-range values on each cycle of
  each loop.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 4bb580ae60 Don't `@lru_cache` on `Viz.i_from_t()`, since view state.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 05aee4a311 Tweak debug printing to display y-mxmn per viz 2023-03-10 18:20:22 -05:00
Tyler Goodlet fc98d66ffc Fix curve up-sampling on `'r'` hotkey
Previously when very zoomed out and using the `'r'` hotkey the
interaction handler loop wouldn't trigger a re-(up)sampling to get
a more detailed curve graphic and instead the previous downsampled
(under-detailed) graphic would show. Fix that by ensuring we yield back
to the Qt event loop and do at least a couple render cycles with paired
`.interact_graphics_cycle()` calls.

Further this flips the `.start/signal_ic()` methods to use the new
`.reset_graphics_caches()` ctr-mngr method.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 57d56c4791 Facepalm: set `Viz.ViewState.yrange` even on cache hits.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7e6e04b7e2 Drop remaining usage of `ChartPlotWidget.default_view()`
Instead delegate directly to `Viz.default_view()` throughout charting
startup and interaction handlers.

Also add a `ChartPlotWidget.reset_graphics_caches()` context mngr which
resets all managed graphics object's cacheing modes on enter and
restores them on exit for simplified use in interaction handling code.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 12bee716c2 Add `do_min_bars: bool` flag to `Viz.default_view()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 6690bd4576 Drop remaining non-usage of `ChartPlotWidget.maxmin()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 9c8bd9b8ce Expand mxmn view y-margins back to 0.06 2023-03-10 18:20:22 -05:00
Tyler Goodlet eea850450a Handle yrange not set on view vase for vlm fsp plot 2023-03-10 18:20:22 -05:00
Tyler Goodlet f7dfe57090 Disable coordinate caching during interaction
This finally seems to mitigate all the "smearing" and "jitter" artifacts
when using Qt's "coordinate cache" graphics-mode:

- whenever we're in a mouse interaction (as per calls to
  `ChartView.start/signal_ic()`) we simply disable the caching mode (set
  `.NoCache` until the interaction is complete.
- only do this (for now) during a pan since it doesn't seem to be an
  issue when zooming?
- ensure disabling all `Viz.graphics` and `.ds_graphics` to be agnostic
  to any case where there's both a zoom and a pan simultaneously (not
  that it's easy to do manually XD) as well as solving the problem
  whenever an OHLC series is in traced-and-downsampled mode (during low
  zoom).

Impl deatz:
- rename `ChartView._ic` -> `._in_interact: trio.Event`
- add `.ChartView._interact_stack: ExitStack` which  we use to open.
  and close the `FlowGraphics.reset_cache()` mngrs from mouse handlers.
- drop all the commented per-subtype overrides for `.cache_mode: int`.
- write up much better doc strings for `FlattenedOHLC` and `StepCurve`
  including some very basic ASCII-art diagrams.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 9b960594aa Add per-chart `Viz`/overlay graphics iterator method 2023-03-10 18:20:22 -05:00
Tyler Goodlet 75642929e3 Move cache-reset ctx mngr to parent type: `FlowGraphics.reset_cache()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet eda283f059 Fix focal min calc after switching to `Viz.datums_range()`.. 2023-03-10 18:20:22 -05:00
Tyler Goodlet 77401a94fb Simplify `FlowGraphics.x_last()` logics 2023-03-10 18:20:22 -05:00
Tyler Goodlet 75807f4a96 Rename overlay technique var to `method` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 94f0ef13ef Repair x-label datetime labels when in array-index mode 2023-03-10 18:20:22 -05:00
Tyler Goodlet 7579863068 Skip overlay handling when `N < 2` are detected 2023-03-10 18:20:22 -05:00
Tyler Goodlet 993bb47138 Drop passing overlay method from viewbox to view-mode handler 2023-03-10 18:20:22 -05:00
Tyler Goodlet 8c392fda60 Drop a bunch of commented/uneeded cruft 2023-03-10 18:20:22 -05:00
Tyler Goodlet 45e97dd4c8 Solve a final minor-should-rescale edge case
When the minor has the same scaling as the major in a given direction we
should still do back-scaling against the major-target and previous
minors to avoid strange edge cases where only the target-major might not
be shifted correctly to show an matched intersect point? More or less
this just meant making the y-mxmn checks interval-inclusive with
`>=`/`<=` operators.

Also adds a shite ton of detailed comments throughout the pin-to-target
method blocks and moves the final major y-range call outside the final
`scaled: dict` loop.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 01ea706644 Better doc string, use `Viz.vs: ViewState` 2023-03-10 18:20:22 -05:00
Tyler Goodlet c1ea8552ac Back-rescale previous (minor) curves from latest
For the "pin to target major/target curve" overlay method, this finally
solves the longstanding issue of ensuring that any new minor curve,
which requires and increase in the major/target curve y-range, also
re-scales all previously scaled minor curves retroactively. Thus we now
guarantee that all minor curves are correctly "pinned" to their
target/major on their earliest available datum **and** are all kept in
view.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 6601dea8cc Support "pin-to-target-curve" overlay method again
Yah yah, i know it's the same as before (the N > 2 curves case with
out-of-range-minor rescaling the previously scaled curves isn't fixed
yet...) but, this is a much better and optional implementation in less
code. Further we're now better leveraging various new cached properties
and methods on `Viz`.

We now handle different `overlay_technique: str` options using `match:`
syntax in the 2ndary scaling loop, stash the returns scalars per curve
in `overlay_table`, and store and iterate the curves by dispersion
measure sort order.

Further wrt "pin-to-target-curve" mode, which currently still pins to
the largest measured dispersion curve in the overlay set:

- pop major Ci overlay table entries at start for sub-calcs usage when
  handling the "minor requires major rescale after pin" case.
- (finally) correctly rescale the major curve y-mxmn to whatever the
  latest minor overlay curve by calcing the inverse transform from the
  minor *at that point*:
  - the intersect point being that which the minor has starts support on
    the major's x-domain* using the new `Viz.scalars_from_index()` and,
  - checking that the minor is not out of range (versus what the major's
    transform calcs it to be, in which case,
  - calc the inverse transform from the current out-of-range minor and
    use it to project the new y-mxmn for the major/target based on the
    same intersect-reference point in the x-domain used by the minor.
  - always handle the target-major Ci specially by only setting the
    `mx_ymn` / `mx_ymn` value when iterating that entry in the overlay
    table.
  - add todos around also doing the last sub-sub bullet for all previously
    major-transform scaled minor overlays (this is coming next..i hope).
- add a final 3rd overlay loop which goes through a final `scaled: dict`
  to apply all range values to each view; this is where we will
  eventually solve that last edge case of an out-of-range minor's
  scaling needing to be used to rescale already scaled minors XD
2023-03-10 18:20:22 -05:00
Tyler Goodlet 4d11c5c89c Add cached dispersion methods to `Viz`
In an effort to make overlay calcs cleaner and leverage caching of view
range -> dispersion measures, this adds the following new methods:

- `._dispersion()` an lru cached returns scalar calculator given input
  y-range and y-ref values.
- `.disp_from_range()` which calls the above method and returns variable
  output depending on requested calc `method: str`.
- `.i_from_t()` a currently unused cached method for slicing the
  in-view's array index from time stamp (though not working yet due to
  needing to parameterize the cache by the input `.vs.xrange`).

Further refinements/adjustments:
- rename `.view_state: ViewState` -> `.vs`.
- drop the `.bars_range()` method as it's no longer used anywhere else
  in the code base.
- always set the `ViewState.in_view: np.ndarray` inside `.read()`.
- return the start array index (from slice) and `yref` value @ `xref`
  from `.scalars_from_index()` to aid with "pin to curve" rescaling
  caused by out-of-range pinned-minor curves.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 29418e9655 Avoid index-from-time slicing including gaps
Not sure why this was ever allowed but, for slicing to the sample
*before* whatever target time stamp is passed in we should definitely
not return the prior index as for the slice start since that might
include a very large gap prior to whatever sample is scanned to have
the earliest matching time stamp.

This was essential to fixing overlay intersect points searching in our
``ui.view_mode`` machinery..
2023-03-10 18:20:22 -05:00
Tyler Goodlet 8fd5c67f2a Drop last lingering usage of `Viz.bars_range()` 2023-03-10 18:20:22 -05:00
Tyler Goodlet 62e0889bf5 Add `Viz.view_state: ViewState`
Adds a small struct which is used to track the most recently viewed
data's x/y ranges as well as the last `Viz.read()` "in view" array data
for fast access by chart related graphics processing code, namely view
mode overlay handling.

Also adds new `Viz` interfaces:
- `Viz.ds_yrange: tuple[float, float]' which replaces the previous
  `.yrange` (now set by `.datums_range()` on manual y-range calcs) so
  that the m4 downsampler can set this field specifically and then it
  get used (when available) by `Viz.maxmin()`.
- `Viz.scalars_from_index()` a new returns-scalar generator which can be
  used to calc the up and down returns values (used for scaling overlay
  y-ranges) from an input `xref` x-domain index which maps to some
  `Ci(xref) = yref`.
2023-03-10 18:20:22 -05:00
Tyler Goodlet 26690b061b Make slow chart a teensie bit smaller 2023-03-10 18:20:22 -05:00
Tyler Goodlet 98b7d78476 Drop (now) unused major curve mx/mn variables 2023-03-10 18:20:22 -05:00
Tyler Goodlet 35c40e825a Move overlay transform logic to new `.ui.view_mode`
It was getting waayy to long to be jammed in a method XD

This moves all the chart-viz iteration and transform logic into a new
`piker.ui.view_mode.overlay_viewlists()` core routine which will make it
a lot nicer for,

- AOT compilation via `numba` / `cython` / `mypyc`.
- decoupling from the `pyqtgraph.ViewBox` APIs if we ever decide to get
  crazy and go without another graphics engine.
- keeping your head clear when trying to rework the code B)
2023-03-10 18:20:22 -05:00