Commit Graph

3973 Commits (f81ea64cab21cbf1baaa169820479ba5fab8bf73)

Author SHA1 Message Date
Tyler Goodlet 94733c4a0b A PoC tsdb prototype: `parqdb` using `polars`
Turns out just (over)writing `.parquet` files with >= 1M datums is like
less then a second, and we can likely speed up appends using
`fastparquet` (usage coming soon).

Includes:
- a new `clone` CLI subcmd to test this all out by ad-hoc copy of
  (literally hardcoded to a daemon-actor specific shm allocation X) an
  existing `/dev/shm/<ShmArray>` and push to `.parquet` file.
  - code to convert from our `ShmArray.array: np.ndarray` ->
    `polars.DataFrame` (thanks SO).
  - timing checks around the file IO and np -> polars conversion.
- a `read` subcmd which i was using to test the sync `pymarketstore`
  client against our async one to see if the issues from
  https://github.com/pikers/piker/issues/443 were resolved, but nope!
2023-06-27 13:41:47 -04:00
Tyler Goodlet 7d1cc47db9 ROFL, even using `pymarketstore`'s json-RPC it's borked..
Turns out trying to switch to the old sync client and going back to
using the old json-RPC API (after having had to patch the upstream repo
to not import gRPC machinery to avoid crashes..) I'm basically getting
the exact same issues.

New tinkering results does possibly tell some new stuff:
- the EOF error seems to indeed be due to trying fetch records which haven't been
  written (properly) - like asking for a `end=<epoch_int>` that is
  earlier then the earliest record.
- the "snappy input corrupt" error seems to have something to do with
  the `Params.end` field not being an `int` and/or the int precision not
  being chosen correctly?
  - toying with this a bunch manually shows that the internals of the
    client (particularly `.build_query()` stuff) is parsing/calcing the
    `Epoch` and `Nanoseconds` values out incorrectly.. which is likely
    part of the problem.
  - we also changed `anyio_marketstore.MarketStoreclient.build_query()`
    logic when removing `pandas` a while back, which also seems to be
    part of the problem on the async side, however reverting those
    changes also didn't fix the issue entirely; likely something else
    more subtle going on (maybe with the write vs. read `Epoch` field
    type we pass?).

Despite all this malarky, we're already underway more or less obsoleting
this whole thing with a much less complex approach of using apache
parquet files and modern filesystem tools to get a more flexible and
numerics-native dataframe-oriented tsdb B)
2023-06-27 13:41:47 -04:00
Tyler Goodlet 9859f601ca Invert data provider's OHLCV field defs
Turns out the reason we were originally making the `time: float` column in our
ohlcv arrays was bc that's what **only** ib uses XD (and/or 🤦)

Instead we changed the default field type to be an `int` (which is also
more correct to avoid `float` rounding/precision discrepancies) and thus
**do not need to override it** in all other (crypto) backends (except
`ib`). Now we only do the customization (via `._ohlc_dtype`) to `float`
only for `ib` for now (though pretty sure we can also not do that
eventually as well..)!
2023-06-27 13:41:47 -04:00
Tyler Goodlet af64152640 .data.history: update to new naming
-> `._source.def_iohlcv_fields`
-> `.storage.StorageClient`
2023-06-27 13:41:47 -04:00
Tyler Goodlet bf21d2e329 Rename default OHLCV `np.dtype` descriptions
Use `def_iohlcv_fields` for a name and instead of copying and inserting
the index field pop it for the non-index version. Drop creating
`np.dtype()` instances since `numpy`'s apis accept both input forms so
this is simpler on our end.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 848577488e Add public config dir getter 2023-06-27 13:41:47 -04:00
Tyler Goodlet e82538eded .data: export ohlc dtypes at top level 2023-06-27 13:41:47 -04:00
Tyler Goodlet 8ccb8b0744 kucoin: drop shm-array `numpy` dtype def, our default is the same 2023-06-27 13:41:47 -04:00
Tyler Goodlet e83de2906f Relegate old marketstore cli eps to masked module 2023-06-27 13:41:47 -04:00
Tyler Goodlet 33c464524b Lower the paper engine order-cancel latency 2023-06-27 13:41:47 -04:00
Tyler Goodlet cb774e5a5d Re-implement `piker store` CLI with `typer`
Turns out you can mix and match `click` with `typer` so this moves what
was the `.data.cli` stuff into `storage.cli` and uses the integration
api to make it all work B)

New subcmd: `piker store`
- add `piker store ls` which lists all fqme keyed time-series from backend.
- add `store delete` to remove any such key->time-series.
  - now uses a nursery for multi-timeframe concurrency B)

Mask out all the old `marketstore` specific subcmds for now (streaming,
ingest, storesh, etc..) in anticipation of moving them into
a subpkg-module and make sure to import the sub-cmd module in our top
level cli package.

Other `.storage` api tweaks:
- drop the reraising with custom error (for now).
- rename `Storage` -> `StorageClient` (or should it be API?).
2023-06-27 13:41:47 -04:00
Tyler Goodlet 1ec9b0565f Move `.data.cli` to `.storage.cli` 2023-06-27 13:41:47 -04:00
Tyler Goodlet 7ab97fb21d Add marketstore client as storage-backend module
To kick off our (tsdb) storage backends this adds our first implementing
a new `Storage(Protocol)` client interface. Going foward, the top level
`.storage` pkg-module will now expose backend agnostic APIs and helpers
whilst specific backend implementations will adhere to that middle-ware
layer.

Deats:
- add `.storage.marketstore.Storage` as the first client implementation,
  moving all needed (import) dependencies out from
  `.service.marketstore` as well as `.ohlc_key_map` and `get_client()`.
- move root `conf.toml` loading from `.data.history` into
  `.storage.__init__.open_storage_client()` which now takes in a `name:
  str` and does all the work of loading the correct backend module, its
  config, and determining if a service-instance can be contacted and
  a client loaded; in the case where this fails we raise a new
  `StorageConnectionError`.
- add a new `.storage.get_storagemod()` just like we have for brokers.
- make `open_storage_client()` also return the backend module such that
  the history-data layer can make backend specific calls as needed (eg.
  ohlc_key_map).
- fall back to a basic non-tsdb backfill when `open_storage_client()`
  raises the new connection error.
2023-06-27 13:41:47 -04:00
Tyler Goodlet 29211b200d Start `piker.storage` subsys: cross-(ts)db middlewares
The plan is to offer multiple tsdb and other storage backends (for
a variety of use cases) and expose them similarly to how we do for
broker and data providers B)
2023-06-27 13:41:47 -04:00
Tyler Goodlet ae8358a5e7 Tidy up unused imports and doc string 2023-06-27 13:32:18 -04:00
Tyler Goodlet 00a51c0288 Use new `msgspec.structs` api for `.typecast()` 2023-06-27 13:26:52 -04:00
Tyler Goodlet 994564f923 Just warn-print when annots are str values? 2023-06-27 13:26:52 -04:00
Tyler Goodlet 12172cc5cd Make `.data.types.Struct.typecast()` work via type lookup from `builtins` 2023-06-27 13:26:52 -04:00
goodboy a65910c732
Merge pull request #523 from ebisu4/master
get font style from main config
2023-06-27 13:25:11 -04:00
ebisu4 949fa9fbb9
Merge pull request #1 from pikers/fix_custom_font_settings
Fix reading font size from user config on linux
2023-06-21 10:53:46 +02:00
Tyler Goodlet 4b77de5e2d Fix reading font size from user config
Was borked on linux if you didn't provide the setting in `conf.toml` due
to some logic errors. Fix that by rejigging `DpiAwareFont` internal
variables:

- add new `._font_size_calc_key: str` which was the old `._font_size`
  and is only used when no explicit font size is set by the user in the
  `conf.toml` config:
  - this is the "key" that is used to lookup a calculation function
    which attempts to compute a best fit font size given the measured
    system displays DPI settings and dimensions.
- make the `._font_size: int` the **actual** font size integer that is
  cached and passed to `Qt` to set the size.
  - this is overridden by user config now if defined.
- change the input kwarg `font_size: str` to the constructor to better
  change the input kwarg `font_size: str` to the constructor to better
  named private `_font_size_key: str` which gets set to the new
  `._font_size_calc_key`.

Also, adjust all client code which instantiates `DpiAwareFont` to use
the new `_font_size_key` kwarg input so nothing breaks XD
2023-06-19 15:13:01 -04:00
Ebisu d660376206 get font style from main config 2023-06-19 00:10:37 +02:00
goodboy 201b0d99c1
Merge pull request #518 from pikers/fix_price_label_digits
Fix price label precision as `MktPair.price_tick_digits`
2023-05-31 10:59:30 -04:00
Tyler Goodlet c27da99e12 Fix price label precision as `MktPair.price_tick_digits`
Was only really borked for higher-precision but lower priced assets
(like TLOS or peeneez) which have a `MktPair.price_tick_digits >= 2`.

The issue was using the wrong attr, the `size_tick_digits`..
2023-05-31 10:36:20 -04:00
goodboy e51ba404fc
Merge pull request #489 from pikers/rekt_pps
Rekt pps? problem? => `piker.accounting`
2023-05-28 15:41:50 -04:00
Tyler Goodlet abd3cefd84 Parametrize ems service test to cancel with API and kbi 2023-05-28 14:28:56 -04:00
Tyler Goodlet f6549fcb62 Always allocate a new `OrderClient` per `open_ems()` call 2023-05-28 14:05:03 -04:00
Tyler Goodlet 41aa87f847 Fix `_digits` attr names in order mode.. 2023-05-28 13:13:43 -04:00
Tyler Goodlet d6331ce9e1 Add nonlocal annots to satisfy ruff 2023-05-28 12:41:14 -04:00
Tyler Goodlet 4f67ac0337 Change to new context-cancelled msg contents: pikerd is canceller 2023-05-26 17:16:43 -04:00
Tyler Goodlet 024cf8b8c2 add in `[kucoin]` section to brokers conf 2023-05-26 16:51:11 -04:00
Tyler Goodlet 9ec664f7c8 Drop elastic search container build for now since we're also skipping the test 2023-05-26 16:50:53 -04:00
Tyler Goodlet 5e2107ff15 Adjust `config.load()` to handle CI git checkout dir, seems they changed it!? 2023-05-26 16:50:15 -04:00
Tyler Goodlet 5f1d0fcb8c `tmpconfdir`: always assert brokers config created 2023-05-26 14:58:59 -04:00
Tyler Goodlet 3b5bd8f43e Ensure quote last price is a `float` 2023-05-26 14:42:35 -04:00
Tyler Goodlet 40c5f39f0d conftest: be explicit about which config we touch 2023-05-26 14:42:09 -04:00
Tyler Goodlet 3d8c1a7b3c ib: don't log-emit ib pp msg when none exists.. 2023-05-26 14:05:32 -04:00
Tyler Goodlet 06cc3ac92c Tidy up ems tests as per some `ruff`in 2023-05-25 18:04:52 -04:00
Tyler Goodlet 4a8e8a32f9 Fix account config loading logic discovered in new test XD 2023-05-25 17:56:14 -04:00
Tyler Goodlet 9bc11d8dd9 Add basic config checking tests 2023-05-25 17:55:20 -04:00
Tyler Goodlet 9c80969fd5 .data.validate: add missing endpoint warnings 2023-05-25 16:01:21 -04:00
Tyler Goodlet da4d344e63 Change to `piker_pin` branch in `tomlkit` fork 2023-05-25 13:53:14 -04:00
goodboy 073ff0103a
Merge pull request #506 from pikers/py311
`python3.11` support!
2023-05-24 19:34:10 -04:00
Tyler Goodlet f0a346dcc3 Some linting fixes after trying out `ruff` 2023-05-24 17:25:23 -04:00
Tyler Goodlet 7381c361cd Strictly drop `LinkedSplits.symbol` B) 2023-05-24 15:42:14 -04:00
Tyler Goodlet 1b577eebf6 Change over the UI layer to use `MktPair`
Including changing to `LinkedSplits.mkt: MktPair` and adding an explicit
setter method for setting it and being sure that nothing breaks
in the display system init!

For this commit we leave in warning access to `LinkedSplits.symbol` but
will remove in following commit.
2023-05-24 15:30:17 -04:00
Tyler Goodlet 39af215d61 kraken: use new `Position.mkt` attr 2023-05-24 15:29:42 -04:00
Tyler Goodlet 35f0520cb0 Drop `Symbol` / `.symbol` support from `.accounting`
Only stuff left was the allocator stuff. Drop the top level subpkg
exports and finally kill off the awkwardly named
`Symbol.lot_size_digits` properties XD

Expose a bunch more util funcs at subpkg top level, do some typing in
allocator method internals.
2023-05-24 15:26:51 -04:00
Tyler Goodlet 738d0ca38b Rename db tests to test_docker_services 2023-05-24 12:30:57 -04:00
Tyler Goodlet bd8e4760d5 Port everything strictly to `Position.mkt` and `Flume.mkt` 2023-05-24 12:16:28 -04:00