Catch using `Sampler.bcast_errors` where possible

In all other possible IPC disconnect handling blocks. Also more
comprehensive typing throughout `uniform_rate_send()`.

README.rst

@@ -1,162 +1,161 @@
 piker
 -----
-trading gear for hackers.
+trading gear for hackers
 
 |gh_actions|
 
 .. |gh_actions| image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fpikers%2Fpiker%2Fbadge&style=popout-square
     :target: https://actions-badge.atrox.dev/piker/pikers/goto
 
-``piker`` is a broker agnostic, next-gen FOSS toolset for real-time
+``piker`` is a broker agnostic, next-gen FOSS toolset and runtime for
-computational trading targeted at `hardcore Linux users <comp_trader>`_ .
+real-time computational trading targeted at `hardcore Linux users
+<comp_trader>`_ .
 
-we use as much bleeding edge tech as possible including (but not limited to):
+we use much bleeding edge tech including (but not limited to):
 
 - latest python for glue_
-- trio_ & tractor_ for our distributed, multi-core, real-time streaming
+- uv_ for packaging and distribution
-  `structured concurrency`_ runtime B)
+- trio_ & tractor_ for our distributed `structured concurrency`_ runtime
-- Qt_ for pristine high performance UIs
+- Qt_ for pristine low latency UIs
-- pyqtgraph_ for real-time charting
+- pyqtgraph_ (which we've extended) for real-time charting and graphics
-- ``polars`` ``numpy`` and ``numba`` for `fast numerics`_
+- ``polars`` ``numpy`` and ``numba`` for redic `fast numerics`_
-- `apache arrow and parquet`_ for time series history management
+- `apache arrow and parquet`_ for time-series storage
-  persistence and sharing
-- (prototyped) techtonicdb_ for L2 book storage
 
-.. |travis| image:: https://img.shields.io/travis/pikers/piker/master.svg
+potential projects we might integrate with soon,
-    :target: https://travis-ci.org/pikers/piker
+
+- (already prototyped in ) techtonicdb_ for L2 book storage
+
+.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
+.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
+.. _uv: https://docs.astral.sh/uv/
 .. _trio: https://github.com/python-trio/trio
 .. _tractor: https://github.com/goodboy/tractor
 .. _structured concurrency: https://trio.discourse.group/
-.. _marketstore: https://github.com/alpacahq/marketstore
-.. _techtonicdb: https://github.com/0b01/tectonicdb
 .. _Qt: https://www.qt.io/
 .. _pyqtgraph: https://github.com/pyqtgraph/pyqtgraph
-.. _glue: https://numpy.org/doc/stable/user/c-info.python-as-glue.html#using-python-as-glue
 .. _apache arrow and parquet: https://arrow.apache.org/faq/
 .. _fast numerics: https://zerowithdot.com/python-numpy-and-pandas-performance/
-.. _comp_trader: https://jfaleiro.wordpress.com/2019/10/09/computational-trader/
+.. _techtonicdb: https://github.com/0b01/tectonicdb
 
 
-focus and features:
+focus and feats:
-*******************
+****************
-- 100% federated: your code, your hardware, your data feeds, your broker fills.
+fitting with these tenets, we're always open to new
-- zero web: low latency, native software that doesn't try to re-invent the OS
+framework/lib/service interop suggestions and ideas!
-- maximal **privacy**: prevent brokers and mms from knowing your
-  planz; smack their spreads with dark volume.
-- zero clutter: modal, context oriented UIs that echew minimalism, reduce
-  thought noise and encourage un-emotion.
-- first class parallelism: built from the ground up on next-gen structured concurrency
-  primitives.
-- traders first: broker/exchange/asset-class agnostic
-- systems grounded: real-time financial signal processing that will
-  make any queuing or DSP eng juice their shorts.
-- non-tina UX: sleek, powerful keyboard driven interaction with expected use in tiling wms
-- data collaboration: every process and protocol is multi-host scalable.
-- fight club ready: zero interest in adoption by suits; no corporate friendly license, ever.
 
-fitting with these tenets, we're always open to new framework suggestions and ideas.
+- **100% federated**:
+  your code, your hardware, your data feeds, your broker fills.
 
-building the best looking, most reliable, keyboard friendly trading
+- **zero web**:
-platform is the dream; join the cause.
+  low latency as a prime objective, native UIs and modern IPC
+  protocols without trying to re-invent the "OS-as-an-app"..
+
+- **maximal privacy**:
+  prevent brokers and mms from knowing your planz; smack their
+  spreads with dark volume from a VPN tunnel.
+
+- **zero clutter**:
+  modal, context oriented UIs that echew minimalism, reduce thought
+  noise and encourage un-emotion.
+
+- **first class parallelism**:
+  built from the ground up on a next-gen structured concurrency
+  supervision sys.
+
+- **traders first**:
+  broker/exchange/venue/asset-class/money-sys agnostic
+
+- **systems grounded**:
+  real-time financial signal processing (fsp) that will make any
+  queuing or DSP eng juice their shorts.
+
+- **non-tina UX**:
+  sleek, powerful keyboard driven interaction with expected use in
+  tiling wms (or maybe even a DDE).
+
+- **data collab at scale**:
+  every actor-process and protocol is multi-host aware.
+
+- **fight club ready**:
+  zero interest in adoption by suits; no corporate friendly license,
+  ever.
+
+building the hottest looking, fastest, most reliable, keyboard
+friendly FOSS trading platform is the dream; join the cause.
 
 
-sane install with `poetry`
+a sane install with `uv`
-**************************
+************************
-TODO!
+bc why install with `python` when you can faster with `rust` ::
 
-
+    uv lock
-rigorous install on ``nixos`` using ``poetry2nix``
-**************************************************
-TODO!
 
 
 hacky install on nixos
 **********************
-`NixOS` is our core devs' distro of choice for which we offer
+``NixOS`` is our core devs' distro of choice for which we offer
 a stringently defined development shell envoirment that can be loaded with::
 
-    nix-shell develop.nix
+    nix-shell default.nix
-
-this will setup the required python environment to run piker, make sure to
-run::
-
-    pip install -r requirements.txt -e .
-
-once after loading the shell
 
 
-install wild-west style via `pip`
+start a chart
-*********************************
+*************
-``piker`` is currently under heavy pre-alpha development and as such
+run a realtime OHLCV chart stand-alone::
-should be cloned from this repo and hacked on directly.
 
-for a development install::
+    piker -l info chart btcusdt.spot.binance xmrusdt.spot.kraken
 
-    git clone git@github.com:pikers/piker.git
+this runs a chart UI (with 1m sampled OHLCV) and shows 2 spot markets from 2 diff cexes
-    cd piker
+overlayed on the same graph. Use of `piker` without first starting
-    virtualenv env
+a daemon (`pikerd` - see below) means there is an implicit spawning of the
-    source ./env/bin/activate
+multi-actor-runtime (implemented as a `tractor` app).
-    pip install -r requirements.txt -e .
+
+For additional subsystem feats available through our chart UI see the
+various sub-readmes:
+
+- order control using a mouse-n-keyboard UX B)
+- cross venue market-pair (what most call "symbol") search, select, overlay Bo
+- financial-signal-processing (`piker.fsp`) write-n-reload to sub-chart BO
+- src-asset derivatives scan for anal, like the infamous "max pain" XO
 
 
-check out our charts
+spawn a daemon standalone
-********************
+*************************
-bet you weren't expecting this from the foss::
+we call the root actor-process the ``pikerd``. it can be (and is
-
+recommended normally to be) started separately from the ``piker
-    piker -l info -b kraken -b binance chart btcusdt.binance --pdb
+chart`` program::
-
-
-this runs the main chart (currently with 1m sampled OHLC) in in debug
-mode and you can practice paper trading using the following
-micro-manual:
-
-``order_mode`` (
-    edge triggered activation by any of the following keys,
-    ``mouse-click`` on y-level to submit at that price
-    ):
-
-    - ``f``/ ``ctl-f`` to stage buy
-    - ``d``/ ``ctl-d`` to stage sell
-    - ``a`` to stage alert
-
-
-``search_mode`` (
-    ``ctl-l`` or ``ctl-space`` to open,
-    ``ctl-c`` or ``ctl-space`` to close
-    ) :
-
-    - begin typing to have symbol search automatically lookup
-      symbols from all loaded backend (broker) providers
-    - arrow keys and mouse click to navigate selection
-    - vi-like ``ctl-[hjkl]`` for navigation
-
-
-you can also configure your position allocation limits from the
-sidepane.
-
-
-run in distributed mode
-***********************
-start the service manager and data feed daemon in the background and
-connect to it::
 
     pikerd -l info --pdb
 
+the daemon does nothing until a ``piker``-client (like ``piker
+chart``) connects and requests some particular sub-system. for
+a connecting chart ``pikerd`` will spawn and manage at least,
 
-connect your chart::
+- a data-feed daemon: ``datad`` which does all the work of comms with
+  the backend provider (in this case the ``binance`` cex).
+- a paper-trading engine instance, ``paperboi.binance``, (if no live
+  account has been configured) which allows for auto/manual order
+  control against the live quote stream.
 
-    piker -l info -b kraken -b binance chart xmrusdt.binance --pdb
+*using* an actor-service (aka micro-daemon) manager which dynamically
+supervises various sub-subsystems-as-services throughout the ``piker``
+runtime-stack.
 
+now you can (implicitly) connect your chart::
 
-enjoy persistent real-time data feeds tied to daemon lifetime. the next
+    piker chart btcusdt.spot.binance
-time you spawn a chart it will load much faster since the data feed has
+
-been cached and is now always running live in the background until you
+since ``pikerd`` was started separately you can now enjoy a persistent
-kill ``pikerd``.
+real-time data stream tied to the daemon-tree's lifetime. i.e. the next
+time you spawn a chart it will obviously not only load much faster
+(since the underlying ``datad.binance`` is left running with its
+in-memory IPC data structures) but also the data-feed and any order
+mgmt states should be persistent until you finally cancel ``pikerd``.
 
 
 if anyone asks you what this project is about
 *********************************************
-you don't talk about it.
+you don't talk about it; just use it.
 
 
 how do i get involved?
@@ -166,6 +165,15 @@ enter the matrix.
 
 how come there ain't that many docs
 ***********************************
-suck it up, learn the code; no one is trying to sell you on anything.
+i mean we want/need them but building the core right has been higher
-also, we need lotsa help so if you want to start somewhere and can't
+prio then marketting (and likely will stay that way Bp).
-necessarily write serious code, this might be the place for you!
+
+soo, suck it up bc,
+
+- no one is trying to sell you on anything
+- learning the code base is prolly way more valuable
+- the UI/UXs are intended to be "intuitive" for any hacker..
+
+we obviously need tonz help so if you want to start somewhere and
+can't necessarily write "advanced" concurrent python/rust code, this
+helping document literally anything might be the place for you!

default.nix

@@ -0,0 +1,134 @@
+with (import <nixpkgs> {});
+let
+  glibStorePath = lib.getLib glib;
+  zlibStorePath = lib.getLib zlib;
+  zstdStorePath = lib.getLib zstd;
+  dbusStorePath = lib.getLib dbus;
+  libGLStorePath = lib.getLib libGL;
+  freetypeStorePath = lib.getLib freetype;
+  qt6baseStorePath = lib.getLib qt6.qtbase;
+  fontconfigStorePath = lib.getLib fontconfig;
+  libxkbcommonStorePath = lib.getLib libxkbcommon;
+  xcbutilcursorStorePath = lib.getLib xcb-util-cursor;
+
+  qtpyStorePath = lib.getLib python312Packages.qtpy;
+  pyqt6StorePath = lib.getLib python312Packages.pyqt6;
+  pyqt6SipStorePath = lib.getLib python312Packages.pyqt6-sip;
+  rapidfuzzStorePath = lib.getLib python312Packages.rapidfuzz;
+  qdarkstyleStorePath = lib.getLib python312Packages.qdarkstyle;
+
+  xorgLibX11StorePath = lib.getLib xorg.libX11;
+  xorgLibxcbStorePath = lib.getLib xorg.libxcb;
+  xorgxcbutilwmStorePath = lib.getLib xorg.xcbutilwm;
+  xorgxcbutilimageStorePath = lib.getLib xorg.xcbutilimage;
+  xorgxcbutilerrorsStorePath = lib.getLib xorg.xcbutilerrors;
+  xorgxcbutilkeysymsStorePath = lib.getLib xorg.xcbutilkeysyms;
+  xorgxcbutilrenderutilStorePath = lib.getLib xorg.xcbutilrenderutil;
+in
+stdenv.mkDerivation {
+  name = "piker-qt6-uv";
+  buildInputs = [
+    # System requirements.
+    glib
+    zlib
+    dbus
+    zstd
+    libGL
+    freetype
+    qt6.qtbase
+    libgcc.lib
+    fontconfig
+    libxkbcommon
+
+    # Xorg requirements
+    xcb-util-cursor
+    xorg.libxcb
+    xorg.libX11
+    xorg.xcbutilwm
+    xorg.xcbutilimage
+    xorg.xcbutilerrors
+    xorg.xcbutilkeysyms
+    xorg.xcbutilrenderutil
+
+    # Python requirements.
+    python312Full
+    python312Packages.uv
+    python312Packages.qdarkstyle
+    python312Packages.rapidfuzz
+    python312Packages.pyqt6
+    python312Packages.qtpy
+  ];
+  src = null;
+  shellHook = ''
+    set -e
+
+    # Set the Qt plugin path
+    # export QT_DEBUG_PLUGINS=1
+
+    QTBASE_PATH="${qt6baseStorePath}/lib"
+    QT_PLUGIN_PATH="$QTBASE_PATH/qt-6/plugins"
+    QT_QPA_PLATFORM_PLUGIN_PATH="$QT_PLUGIN_PATH/platforms"
+
+    LIB_GCC_PATH="${libgcc.lib}/lib"
+    GLIB_PATH="${glibStorePath}/lib"
+    ZSTD_PATH="${zstdStorePath}/lib"
+    ZLIB_PATH="${zlibStorePath}/lib"
+    DBUS_PATH="${dbusStorePath}/lib"
+    LIBGL_PATH="${libGLStorePath}/lib"
+    FREETYPE_PATH="${freetypeStorePath}/lib"
+    FONTCONFIG_PATH="${fontconfigStorePath}/lib"
+    LIB_XKB_COMMON_PATH="${libxkbcommonStorePath}/lib"
+
+    XCB_UTIL_CURSOR_PATH="${xcbutilcursorStorePath}/lib"
+    XORG_LIB_X11_PATH="${xorgLibX11StorePath}/lib"
+    XORG_LIB_XCB_PATH="${xorgLibxcbStorePath}/lib"
+    XORG_XCB_UTIL_IMAGE_PATH="${xorgxcbutilimageStorePath}/lib"
+    XORG_XCB_UTIL_WM_PATH="${xorgxcbutilwmStorePath}/lib"
+    XORG_XCB_UTIL_RENDER_UTIL_PATH="${xorgxcbutilrenderutilStorePath}/lib"
+    XORG_XCB_UTIL_KEYSYMS_PATH="${xorgxcbutilkeysymsStorePath}/lib"
+    XORG_XCB_UTIL_ERRORS_PATH="${xorgxcbutilerrorsStorePath}/lib"
+
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QTBASE_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_PLUGIN_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$QT_QPA_PLATFORM_PLUGIN_PATH"
+
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_GCC_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$DBUS_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GLIB_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZLIB_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ZSTD_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIBGL_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FONTCONFIG_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$FREETYPE_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$LIB_XKB_COMMON_PATH"
+
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XCB_UTIL_CURSOR_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_X11_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_LIB_XCB_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_IMAGE_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_WM_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_RENDER_UTIL_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_KEYSYMS_PATH"
+    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$XORG_XCB_UTIL_ERRORS_PATH"
+
+    export LD_LIBRARY_PATH
+
+    RPDFUZZ_PATH="${rapidfuzzStorePath}/lib/python3.12/site-packages"
+    QDRKSTYLE_PATH="${qdarkstyleStorePath}/lib/python3.12/site-packages"
+    QTPY_PATH="${qtpyStorePath}/lib/python3.12/site-packages"
+    PYQT6_PATH="${pyqt6StorePath}/lib/python3.12/site-packages"
+    PYQT6_SIP_PATH="${pyqt6SipStorePath}/lib/python3.12/site-packages"
+
+    PATCH="$PATCH:$RPDFUZZ_PATH"
+    PATCH="$PATCH:$QDRKSTYLE_PATH"
+    PATCH="$PATCH:$QTPY_PATH"
+    PATCH="$PATCH:$PYQT6_PATH"
+    PATCH="$PATCH:$PYQT6_SIP_PATH"
+
+    export PATCH
+
+    # Install deps
+    uv lock
+
+  '';
+}

piker/brokers/__init__.py

@@ -98,13 +98,14 @@ async def open_cached_client(
     If one has not been setup do it and cache it.
 
     '''
-    brokermod = get_brokermod(brokername)
+    brokermod: ModuleType = get_brokermod(brokername)
+
+    # TODO: make abstract or `typing.Protocol`
+    # client: Client
     async with maybe_open_context(
         acm_func=brokermod.get_client,
         kwargs=kwargs,
-
     ) as (cache_hit, client):
-
         if cache_hit:
             log.runtime(f'Reusing existing {client}')
 

piker/brokers/core.py

@@ -22,7 +22,9 @@ routines should be primitive data types where possible.
 """
 import inspect
 from types import ModuleType
-from typing import List, Dict, Any, Optional
+from typing import (
+    Any,
+)
 
 import trio
 
@@ -34,8 +36,10 @@ from ..accounting import MktPair
 
 
 async def api(brokername: str, methname: str, **kwargs) -> dict:
-    """Make (proxy through) a broker API call by name and return its result.
+    '''
-    """
+    Make (proxy through) a broker API call by name and return its result.
+
+    '''
     brokermod = get_brokermod(brokername)
     async with brokermod.get_client() as client:
         meth = getattr(client, methname, None)
@@ -62,10 +66,14 @@ async def api(brokername: str, methname: str, **kwargs) -> dict:
 
 async def stocks_quote(
     brokermod: ModuleType,
-    tickers: List[str]
+    tickers: list[str]
-) -> Dict[str, Dict[str, Any]]:
+
-    """Return quotes dict for ``tickers``.
+) -> dict[str, dict[str, Any]]:
-    """
+    '''
+    Return a `dict` of snapshot quotes for the provided input
+    `tickers`: a `list` of fqmes.
+
+    '''
     async with brokermod.get_client() as client:
         return await client.quote(tickers)
 
@@ -74,13 +82,15 @@ async def stocks_quote(
 async def option_chain(
     brokermod: ModuleType,
     symbol: str,
-    date: Optional[str] = None,
+    date: str|None = None,
-) -> Dict[str, Dict[str, Dict[str, Any]]]:
+) -> dict[str, dict[str, dict[str, Any]]]:
-    """Return option chain for ``symbol`` for ``date``.
+    '''
+    Return option chain for ``symbol`` for ``date``.
 
     By default all expiries are returned. If ``date`` is provided
     then contract quotes for that single expiry are returned.
-    """
+
+    '''
     async with brokermod.get_client() as client:
         if date:
             id = int((await client.tickers2ids([symbol]))[symbol])
@@ -98,7 +108,7 @@ async def option_chain(
 # async def contracts(
 #     brokermod: ModuleType,
 #     symbol: str,
-# ) -> Dict[str, Dict[str, Dict[str, Any]]]:
+# ) -> dict[str, dict[str, dict[str, Any]]]:
 #     """Return option contracts (all expiries) for ``symbol``.
 #     """
 #     async with brokermod.get_client() as client:
@@ -110,15 +120,24 @@ async def bars(
     brokermod: ModuleType,
     symbol: str,
     **kwargs,
-) -> Dict[str, Dict[str, Dict[str, Any]]]:
+) -> dict[str, dict[str, dict[str, Any]]]:
-    """Return option contracts (all expiries) for ``symbol``.
+    '''
-    """
+    Return option contracts (all expiries) for ``symbol``.
+
+    '''
     async with brokermod.get_client() as client:
         return await client.bars(symbol, **kwargs)
 
 
-async def search_w_brokerd(name: str, pattern: str) -> dict:
+async def search_w_brokerd(
+    name: str,
+    pattern: str,
+) -> dict:
 
+    # TODO: WHY NOT WORK!?!
+    # when we `step` through the next block?
+    # import tractor
+    # await tractor.pause()
     async with open_cached_client(name) as client:
 
         # TODO: support multiple asset type concurrent searches.
@@ -130,12 +149,12 @@ async def symbol_search(
     pattern: str,
     **kwargs,
 
-) -> Dict[str, Dict[str, Dict[str, Any]]]:
+) -> dict[str, dict[str, dict[str, Any]]]:
     '''
     Return symbol info from broker.
 
     '''
-    results = []
+    results: list[str] = []
 
     async def search_backend(
         brokermod: ModuleType
@@ -143,6 +162,13 @@ async def symbol_search(
 
         brokername: str = mod.name
 
+        # TODO: figure this the FUCK OUT
+        # -> ok so obvi in the root actor any async task that's
+        # spawned outside the main tractor-root-actor task needs to
+        # call this..
+        # await tractor.devx._debug.maybe_init_greenback()
+        # tractor.pause_from_sync()
+
         async with maybe_spawn_brokerd(
             mod.name,
             infect_asyncio=getattr(
@@ -162,7 +188,6 @@ async def symbol_search(
             ))
 
     async with trio.open_nursery() as n:
-
         for mod in brokermods:
             n.start_soon(search_backend, mod.name)
 
@@ -172,11 +197,13 @@ async def symbol_search(
 async def mkt_info(
     brokermod: ModuleType,
     fqme: str,
+
     **kwargs,
 
 ) -> MktPair:
     '''
-    Return MktPair info from broker including src and dst assets.
+    Return the `piker.accounting.MktPair` info struct from a given
+    backend broker tradable src/dst asset pair.
 
     '''
     async with open_cached_client(brokermod.name) as client:

piker/brokers/kucoin.py

@@ -62,7 +62,7 @@ from piker._cacheables import (
 )
 from piker.log import get_logger
 from piker.data.validate import FeedInit
-from piker.types import Struct
+from piker.types import Struct  # NOTE, this is already a `tractor.msg.Struct`
 from piker.data import (
     def_iohlcv_fields,
     match_from_pairs,
@@ -98,9 +98,18 @@ class KucoinMktPair(Struct, frozen=True):
     def size_tick(self) -> Decimal:
         return Decimal(str(self.quoteMinSize))
 
+    callauctionFirstStageStartTime: None|float
+    callauctionIsEnabled: bool
+    callauctionPriceCeiling: float|None
+    callauctionPriceFloor: float|None
+    callauctionSecondStageStartTime: float|None
+    callauctionThirdStageStartTime: float|None
+
     enableTrading: bool
+    feeCategory: int
     feeCurrency: str
     isMarginEnabled: bool
+    makerFeeCoefficient: float
     market: str
     minFunds: float
     name: str
@@ -110,11 +119,10 @@ class KucoinMktPair(Struct, frozen=True):
     quoteIncrement: float
     quoteMaxSize: float
     quoteMinSize: float
-    symbol: str  # our bs_mktid, kucoin's internal id
-    feeCategory: int
-    makerFeeCoefficient: float
-    takerFeeCoefficient: float
     st: bool
+    symbol: str  # our bs_mktid, kucoin's internal id
+    takerFeeCoefficient: float
+    tradingStartTime: float|None
 
 
 class AccountTrade(Struct, frozen=True):
@@ -396,7 +404,13 @@ class Client:
         pairs: dict[str, KucoinMktPair] = {}
         fqmes2mktids: bidict[str, str] = bidict()
         for item in entries:
-            pair = pairs[item['name']] = KucoinMktPair(**item)
+            try:
+                pair = pairs[item['name']] = KucoinMktPair(**item)
+            except TypeError as te:
+                raise TypeError(
+                    '`KucoinMktPair` and reponse fields do not match ??\n'
+                    f'{KucoinMktPair.fields_diff(item)}\n'
+                ) from te
             fqmes2mktids[
                 item['name'].lower().replace('-', '')
             ] = pair.name

piker/clearing/README.rst

@@ -0,0 +1,49 @@
+piker.clearing
+______________
+trade execution-n-control subsys for both live and paper trading as
+well as algo-trading manual override/interaction across any backend
+broker and data provider.
+
+avail UIs
+*********
+
+order ctl
+---------
+the `piker.clearing` subsys is exposed mainly though
+the `piker chart` GUI as a "chart trader" style UX and
+is automatically enabled whenever a chart is opened.
+
+.. ^TODO, more prose here!
+
+the "manual" order control features are exposed via the
+`piker.ui.order_mode` API and can pretty much always be
+used (at least) in simulated-trading mode, aka "paper"-mode, and
+the micro-manual is as follows:
+
+``order_mode`` (
+    edge triggered activation by any of the following keys,
+    ``mouse-click`` on y-level to submit at that price
+    ):
+
+    - ``f``/ ``ctl-f`` to stage buy
+    - ``d``/ ``ctl-d`` to stage sell
+    - ``a`` to stage alert
+
+
+``search_mode`` (
+    ``ctl-l`` or ``ctl-space`` to open,
+    ``ctl-c`` or ``ctl-space`` to close
+    ) :
+
+    - begin typing to have symbol search automatically lookup
+      symbols from all loaded backend (broker) providers
+    - arrow keys and mouse click to navigate selection
+    - vi-like ``ctl-[hjkl]`` for navigation
+
+
+position (pp) mgmt
+------------------
+you can also configure your position allocation limits from the
+sidepane.
+
+.. ^TODO, explain and provide tut once more refined!

piker/clearing/_client.py

@@ -168,7 +168,6 @@ class OrderClient(Struct):
 
 
 async def relay_orders_from_sync_code(
-
     client: OrderClient,
     symbol_key: str,
     to_ems_stream: tractor.MsgStream,
@@ -242,6 +241,11 @@ async def open_ems(
 
     async with maybe_open_emsd(
         broker,
+        # XXX NOTE, LOL so this determines the daemon `emsd` loglevel
+        # then FYI.. that's kinda wrong no?
+        # -[ ] shouldn't it be set by `pikerd -l` or no?
+        # -[ ] would make a lot more sense to have a subsys ctl for
+        #     levels.. like `-l emsd.info` or something?
         loglevel=loglevel,
     ) as portal:
 

piker/clearing/_paper_engine.py

@@ -297,6 +297,8 @@ class PaperBoi(Struct):
 
         # transmit pp msg to ems
         pp: Position = self.acnt.pps[bs_mktid]
+        # TODO, this will break if `require_only=True` was passed to
+        # `.update_from_ledger()`
 
         pp_msg = BrokerdPosition(
             broker=self.broker,

piker/clearing/_util.py

@@ -30,6 +30,7 @@ subsys: str = 'piker.clearing'
 
 log = get_logger(subsys)
 
+# TODO, oof doesn't this ignore the `loglevel` then???
 get_console_log = partial(
     get_console_log,
     name=subsys,

piker/data/_sampling.py

@@ -95,6 +95,12 @@ class Sampler:
     # history loading.
     incr_task_cs: trio.CancelScope | None = None
 
+    bcast_errors: tuple[Exception] = (
+        trio.BrokenResourceError,
+        trio.ClosedResourceError,
+        trio.EndOfChannel,
+    )
+
     # holds all the ``tractor.Context`` remote subscriptions for
     # a particular sample period increment event: all subscribers are
     # notified on a step.
@@ -258,14 +264,15 @@ class Sampler:
         subs: set
         last_ts, subs = pair
 
-        task = trio.lowlevel.current_task()
+        # NOTE, for debugging pub-sub issues
-        log.debug(
+        # task = trio.lowlevel.current_task()
-            f'SUBS {self.subscribers}\n'
+        # log.debug(
-            f'PAIR {pair}\n'
+        #     f'AlL-SUBS@{period_s!r}: {self.subscribers}\n'
-            f'TASK: {task}: {id(task)}\n'
+        #     f'PAIR: {pair}\n'
-            f'broadcasting {period_s} -> {last_ts}\n'
+        #     f'TASK: {task}: {id(task)}\n'
-            # f'consumers: {subs}'
+        #     f'broadcasting {period_s} -> {last_ts}\n'
-        )
+        #     f'consumers: {subs}'
+        # )
         borked: set[MsgStream] = set()
         sent: set[MsgStream] = set()
         while True:
@@ -282,12 +289,11 @@ class Sampler:
                         await stream.send(msg)
                         sent.add(stream)
 
-                    except (
+                    except self.bcast_errors as err:
-                        trio.BrokenResourceError,
-                        trio.ClosedResourceError
-                    ):
                         log.error(
-                            f'{stream._ctx.chan.uid} dropped connection'
+                            f'Connection dropped for IPC ctx\n'
+                            f'{stream._ctx}\n\n'
+                            f'Due to {type(err)}'
                         )
                         borked.add(stream)
                 else:
@@ -394,7 +400,8 @@ async def register_with_sampler(
                 finally:
                     if (
                         sub_for_broadcasts
-                        and subs
+                        and
+                        subs
                     ):
                         try:
                             subs.remove(stream)
@@ -561,8 +568,7 @@ async def open_sample_stream(
 
 
 async def sample_and_broadcast(
-
+    bus: _FeedsBus,
-    bus: _FeedsBus,  # noqa
     rt_shm: ShmArray,
     hist_shm: ShmArray,
     quote_stream: trio.abc.ReceiveChannel,
@@ -582,11 +588,33 @@ async def sample_and_broadcast(
 
     overruns = Counter()
 
+    # NOTE, only used for debugging live-data-feed issues, though
+    # this should be resolved more correctly in the future using the
+    # new typed-msgspec feats of `tractor`!
+    #
+    # XXX, a multiline nested `dict` formatter (since rn quote-msgs
+    # are just that).
+    # pfmt: Callable[[str], str] = mk_repr()
+
     # iterate stream delivered by broker
     async for quotes in quote_stream:
         # print(quotes)
 
-        # TODO: ``numba`` this!
+        # XXX WARNING XXX only enable for debugging bc ow can cost
+        # ALOT of perf with HF-feedz!!!
+        #
+        # log.info(
+        #     'Rx live quotes:\n'
+        #     f'{pfmt(quotes)}'
+        # )
+
+        # TODO,
+        # -[ ] `numba` or `cython`-nize this loop possibly?
+        #  |_alternatively could we do it in rust somehow by upacking
+        #    arrow msgs instead of using `msgspec`?
+        # -[ ] use `msgspec.Struct` support in new typed-msging from
+        #     `tractor` to ensure only allowed msgs are transmitted?
+        #
         for broker_symbol, quote in quotes.items():
             # TODO: in theory you can send the IPC msg *before* writing
             # to the sharedmem array to decrease latency, however, that
@@ -659,6 +687,21 @@ async def sample_and_broadcast(
             sub_key: str = broker_symbol.lower()
             subs: set[Sub] = bus.get_subs(sub_key)
 
+            # TODO, figure out how to make this useful whilst
+            # incoporating feed "pausing" ..
+            #
+            # if not subs:
+            #     all_bs_fqmes: list[str] = list(
+            #         bus._subscribers.keys()
+            #     )
+            #     log.warning(
+            #         f'No subscribers for {brokername!r} live-quote ??\n'
+            #         f'broker_symbol: {broker_symbol}\n\n'
+
+            #         f'Maybe the backend-sys symbol does not match one of,\n'
+            #         f'{pfmt(all_bs_fqmes)}\n'
+            #     )
+
             # NOTE: by default the broker backend doesn't append
             # it's own "name" into the fqme schema (but maybe it
             # should?) so we have to manually generate the correct
@@ -728,18 +771,14 @@ async def sample_and_broadcast(
                         if lags > 10:
                             await tractor.pause()
 
-                except (
+                except Sampler.bcast_errors as ipc_err:
-                    trio.BrokenResourceError,
-                    trio.ClosedResourceError,
-                    trio.EndOfChannel,
-                ):
                     ctx: Context = ipc._ctx
                     chan: Channel = ctx.chan
                     if ctx:
                         log.warning(
-                            'Dropped `brokerd`-quotes-feed connection:\n'
+                            f'Dropped `brokerd`-feed for {broker_symbol!r} due to,\n'
-                            f'{broker_symbol}:'
+                            f'x>) {ctx.cid}@{chan.uid}'
-                            f'{ctx.cid}@{chan.uid}'
+                            f'|_{ipc_err!r}\n\n'
                         )
                     if sub.throttle_rate:
                         assert ipc._closed
@@ -756,12 +795,11 @@ async def sample_and_broadcast(
 
 
 async def uniform_rate_send(
-
     rate: float,
     quote_stream: trio.abc.ReceiveChannel,
     stream: MsgStream,
 
-    task_status: TaskStatus = trio.TASK_STATUS_IGNORED,
+    task_status: TaskStatus[None] = trio.TASK_STATUS_IGNORED,
 
 ) -> None:
     '''
@@ -779,13 +817,16 @@ async def uniform_rate_send(
     https://gist.github.com/njsmith/7ea44ec07e901cb78ebe1dd8dd846cb9
 
     '''
-    # TODO: compute the approx overhead latency per cycle
+    # ?TODO? dynamically compute the **actual** approx overhead latency per cycle
-    left_to_sleep = throttle_period = 1/rate - 0.000616
+    # instead of this magic # bidinezz?
+    throttle_period: float = 1/rate - 0.000616
+    left_to_sleep: float = throttle_period
 
     # send cycle state
+    first_quote: dict|None
     first_quote = last_quote = None
-    last_send = time.time()
+    last_send: float = time.time()
-    diff = 0
+    diff: float = 0
 
     task_status.started()
     ticks_by_type: dict[
@@ -796,22 +837,28 @@ async def uniform_rate_send(
     clear_types = _tick_groups['clears']
 
     while True:
-
         # compute the remaining time to sleep for this throttled cycle
-        left_to_sleep = throttle_period - diff
+        left_to_sleep: float = throttle_period - diff
 
         if left_to_sleep > 0:
+            cs: trio.CancelScope
             with trio.move_on_after(left_to_sleep) as cs:
+                sym: str
+                last_quote: dict
                 try:
                     sym, last_quote = await quote_stream.receive()
                 except trio.EndOfChannel:
-                    log.exception(f"feed for {stream} ended?")
+                    log.exception(
+                        f'Live stream for feed for ended?\n'
+                        f'<=c\n'
+                        f'  |_[{stream!r}\n'
+                    )
                     break
 
-                diff = time.time() - last_send
+                diff: float = time.time() - last_send
 
                 if not first_quote:
-                    first_quote = last_quote
+                    first_quote: float = last_quote
                     # first_quote['tbt'] = ticks_by_type
 
                 if (throttle_period - diff) > 0:
@@ -872,7 +919,9 @@ async def uniform_rate_send(
         # TODO: now if only we could sync this to the display
         # rate timing exactly lul
         try:
-            await stream.send({sym: first_quote})
+            await stream.send({
+                sym: first_quote
+            })
         except tractor.RemoteActorError as rme:
             if rme.type is not tractor._exceptions.StreamOverrun:
                 raise
@@ -883,19 +932,28 @@ async def uniform_rate_send(
                 f'{sym}:{ctx.cid}@{chan.uid}'
             )
 
+        # NOTE: any of these can be raised by `tractor`'s IPC
+        # transport-layer and we want to be highly resilient
+        # to consumers which crash or lose network connection.
+        # I.e. we **DO NOT** want to crash and propagate up to
+        # ``pikerd`` these kinds of errors!
         except (
-            # NOTE: any of these can be raised by ``tractor``'s IPC
-            # transport-layer and we want to be highly resilient
-            # to consumers which crash or lose network connection.
-            # I.e. we **DO NOT** want to crash and propagate up to
-            # ``pikerd`` these kinds of errors!
-            trio.ClosedResourceError,
-            trio.BrokenResourceError,
             ConnectionResetError,
-        ):
+        ) + Sampler.bcast_errors as ipc_err:
-            # if the feed consumer goes down then drop
+            match ipc_err:
-            # out of this rate limiter
+                case trio.EndOfChannel():
-            log.warning(f'{stream} closed')
+                    log.info(
+                        f'{stream} terminated by peer,\n'
+                        f'{ipc_err!r}'
+                    )
+                case _:
+                    # if the feed consumer goes down then drop
+                    # out of this rate limiter
+                    log.warning(
+                        f'{stream} closed due to,\n'
+                        f'{ipc_err!r}'
+                    )
+
             await stream.aclose()
             return
 

piker/data/_symcache.py

@@ -31,6 +31,7 @@ from pathlib import Path
 from pprint import pformat
 from typing import (
     Any,
+    Callable,
     Sequence,
     Hashable,
     TYPE_CHECKING,
@@ -56,7 +57,7 @@ from piker.brokers import (
 )
 
 if TYPE_CHECKING:
-    from ..accounting import (
+    from piker.accounting import (
         Asset,
         MktPair,
     )
@@ -149,57 +150,68 @@ class SymbologyCache(Struct):
                     'Implement `Client.get_assets()`!'
                 )
 
-            if get_mkt_pairs := getattr(client, 'get_mkt_pairs', None):
+            get_mkt_pairs: Callable|None = getattr(
-
+                client,
-                pairs: dict[str, Struct] = await get_mkt_pairs()
+                'get_mkt_pairs',
-                for bs_fqme, pair in pairs.items():
+                None,
-
+            )
-                    # NOTE: every backend defined pair should
+            if not get_mkt_pairs:
-                    # declare it's ns path for roundtrip
-                    # serialization lookup.
-                    if not getattr(pair, 'ns_path', None):
-                        raise TypeError(
-                            f'Pair-struct for {self.mod.name} MUST define a '
-                            '`.ns_path: str`!\n'
-                            f'{pair}'
-                        )
-
-                    entry = await self.mod.get_mkt_info(pair.bs_fqme)
-                    if not entry:
-                        continue
-
-                    mkt: MktPair
-                    pair: Struct
-                    mkt, _pair = entry
-                    assert _pair is pair, (
-                        f'`{self.mod.name}` backend probably has a '
-                        'keying-symmetry problem between the pair-`Struct` '
-                        'returned from `Client.get_mkt_pairs()`and the '
-                        'module level endpoint: `.get_mkt_info()`\n\n'
-                        "Here's the struct diff:\n"
-                        f'{_pair - pair}'
-                    )
-                    # NOTE XXX: this means backends MUST implement
-                    # a `Struct.bs_mktid: str` field to provide
-                    # a native-keyed map to their own symbol
-                    # set(s).
-                    self.pairs[pair.bs_mktid] = pair
-
-                    # NOTE: `MktPair`s are keyed here using piker's
-                    # internal FQME schema so that search,
-                    # accounting and feed init can be accomplished
-                    # a sane, uniform, normalized basis.
-                    self.mktmaps[mkt.fqme] = mkt
-
-                self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
-                    pair,
-                )
-
-            else:
                 log.warning(
                     'No symbology cache `Pair` support for `{provider}`..\n'
                     'Implement `Client.get_mkt_pairs()`!'
                 )
+                return self
+
+            pairs: dict[str, Struct] = await get_mkt_pairs()
+            if not pairs:
+                log.warning(
+                    'No pairs from intial {provider!r} sym-cache request?\n\n'
+                    '`Client.get_mkt_pairs()` -> {pairs!r} ?'
+                )
+                return self
+
+            for bs_fqme, pair in pairs.items():
+                if not getattr(pair, 'ns_path', None):
+                    # XXX: every backend defined pair must declare
+                    # a `.ns_path: tractor.NamespacePath` to enable
+                    # roundtrip serialization lookup from a local
+                    # cache file.
+                    raise TypeError(
+                        f'Pair-struct for {self.mod.name} MUST define a '
+                        '`.ns_path: str`!\n\n'
+                        f'{pair!r}'
+                    )
+
+                entry = await self.mod.get_mkt_info(pair.bs_fqme)
+                if not entry:
+                    continue
+
+                mkt: MktPair
+                pair: Struct
+                mkt, _pair = entry
+                assert _pair is pair, (
+                    f'`{self.mod.name}` backend probably has a '
+                    'keying-symmetry problem between the pair-`Struct` '
+                    'returned from `Client.get_mkt_pairs()`and the '
+                    'module level endpoint: `.get_mkt_info()`\n\n'
+                    "Here's the struct diff:\n"
+                    f'{_pair - pair}'
+                )
+                # NOTE XXX: this means backends MUST implement
+                # a `Struct.bs_mktid: str` field to provide
+                # a native-keyed map to their own symbol
+                # set(s).
+                self.pairs[pair.bs_mktid] = pair
+
+                # NOTE: `MktPair`s are keyed here using piker's
+                # internal FQME schema so that search,
+                # accounting and feed init can be accomplished
+                # a sane, uniform, normalized basis.
+                self.mktmaps[mkt.fqme] = mkt
+
+            self.pair_ns_path: str = tractor.msg.NamespacePath.from_ref(
+                pair,
+            )
 
         return self
 

piker/data/feed.py

@@ -786,7 +786,6 @@ async def install_brokerd_search(
 
 @acm
 async def maybe_open_feed(
-
     fqmes: list[str],
     loglevel: str | None = None,
 
@@ -840,13 +839,12 @@ async def maybe_open_feed(
 
 @acm
 async def open_feed(
-
     fqmes: list[str],
 
-    loglevel: str | None = None,
+    loglevel: str|None = None,
     allow_overruns: bool = True,
     start_stream: bool = True,
-    tick_throttle: float | None = None,  # Hz
+    tick_throttle: float|None = None,  # Hz
 
     allow_remote_ctl_ui: bool = False,
 

piker/data/flows.py

@@ -36,10 +36,10 @@ from ._sharedmem import (
     ShmArray,
     _Token,
 )
+from piker.accounting import MktPair
 
 if TYPE_CHECKING:
-    from ..accounting import MktPair
+    from piker.data.feed import Feed
-    from .feed import Feed
 
 
 class Flume(Struct):
@@ -82,7 +82,7 @@ class Flume(Struct):
 
     # TODO: do we need this really if we can pull the `Portal` from
     # ``tractor``'s internals?
-    feed: Feed | None = None
+    feed: Feed|None = None
 
     @property
     def rt_shm(self) -> ShmArray:

piker/data/validate.py

@@ -113,9 +113,9 @@ def validate_backend(
             )
             if ep is None:
                 log.warning(
-                    f'Provider backend {mod.name} is missing '
+                    f'Provider backend {mod.name!r} is missing '
-                    f'{daemon_name} support :(\n'
+                    f'{daemon_name!r} support?\n'
-                    f'The following endpoint is missing: {name}'
+                    f'|_module endpoint-func missing: {name!r}\n'
                 )
 
     inits: list[

piker/log.py

@@ -19,6 +19,10 @@ Log like a forester!
 """
 import logging
 import json
+import reprlib
+from typing import (
+    Callable,
+)
 
 import tractor
 from pygments import (
@@ -84,3 +88,29 @@ def colorize_json(
         # likeable styles: algol_nu, tango, monokai
         formatters.TerminalTrueColorFormatter(style=style)
     )
+
+
+# TODO, eventually defer to the version in `modden` once
+# it becomes a dep!
+def mk_repr(
+    **repr_kws,
+) -> Callable[[str], str]:
+    '''
+    Allocate and deliver a `repr.Repr` instance with provided input
+    settings using the std-lib's `reprlib` mod,
+     * https://docs.python.org/3/library/reprlib.html
+
+    ------ Ex. ------
+    An up to 6-layer-nested `dict` as multi-line:
+    - https://stackoverflow.com/a/79102479
+    - https://docs.python.org/3/library/reprlib.html#reprlib.Repr.maxlevel
+
+    '''
+    def_kws: dict[str, int] = dict(
+        indent=2,
+        maxlevel=6,  # recursion levels
+        maxstring=66,  # match editor line-len limit
+    )
+    def_kws |= repr_kws
+    reprr = reprlib.Repr(**def_kws)
+    return reprr.repr

piker/storage/cli.py

@@ -386,6 +386,8 @@ def ldshm(
             open_annot_ctl() as actl,
         ):
             shm_df: pl.DataFrame | None = None
+            tf2aids: dict[float, dict] = {}
+
             for (
                 shmfile,
                 shm,
@@ -526,16 +528,17 @@ def ldshm(
                             new_df,
                             step_gaps,
                         )
-
                         # last chance manual overwrites in REPL
-                        await tractor.pause()
+                        # await tractor.pause()
                         assert aids
+                        tf2aids[period_s] = aids
 
                 else:
                     # allow interaction even when no ts problems.
-                    await tractor.pause()
+                    assert not diff
-                    # assert not diff
 
+            await tractor.pause()
+            log.info('Exiting TSP shm anal-izer!')
 
             if shm_df is None:
                 log.error(

piker/storage/nativedb.py

@@ -161,7 +161,13 @@ class NativeStorageClient:
 
     def index_files(self):
         for path in self._datadir.iterdir():
-            if path.name in {'borked', 'expired',}:
+            if (
+                path.is_dir()
+                or
+                '.parquet' not in str(path)
+                # or
+                # path.name in {'borked', 'expired',}
+            ):
                 continue
 
             key: str = path.name.rstrip('.parquet')

piker/tsp/_anal.py

@@ -616,6 +616,18 @@ def detect_price_gaps(
     # ])
     ...
 
+# TODO: probably just use the null_segs impl above?
+def detect_vlm_gaps(
+    df: pl.DataFrame,
+    col: str = 'volume',
+
+) -> pl.DataFrame:
+
+    vnull: pl.DataFrame = w_dts.filter(
+        pl.col(col) == 0
+    )
+    return vnull
+
 
 def dedupe(
     src_df: pl.DataFrame,
@@ -626,7 +638,6 @@ def dedupe(
 
 ) -> tuple[
     pl.DataFrame,  # with dts
-    pl.DataFrame,  # gaps
     pl.DataFrame,  # with deduplicated dts (aka gap/repeat removal)
     int,  # len diff between input and deduped
 ]:
@@ -639,19 +650,22 @@ def dedupe(
     '''
     wdts: pl.DataFrame = with_dts(src_df)
 
-    # maybe sort on any time field
+    deduped = wdts
-    if sort:
-        wdts = wdts.sort(by='time')
-        # TODO: detect out-of-order segments which were corrected!
-        # -[ ] report in log msg
-        # -[ ] possibly return segment sections which were moved?
 
     # remove duplicated datetime samples/sections
     deduped: pl.DataFrame = wdts.unique(
-        subset=['dt'],
+        # subset=['dt'],
+        subset=['time'],
         maintain_order=True,
     )
 
+    # maybe sort on any time field
+    if sort:
+        deduped = deduped.sort(by='time')
+        # TODO: detect out-of-order segments which were corrected!
+        # -[ ] report in log msg
+        # -[ ] possibly return segment sections which were moved?
+
     diff: int = (
         wdts.height
         -

piker/types.py

@@ -21,230 +21,4 @@ Extensions to built-in or (heavily used but 3rd party) friend-lib
 types.
 
 '''
-from __future__ import annotations
+from tractor.msg import Struct as Struct
-from collections import UserList
-from pprint import (
-    saferepr,
-)
-from typing import Any
-
-from msgspec import (
-    msgpack,
-    Struct as _Struct,
-    structs,
-)
-
-
-class DiffDump(UserList):
-    '''
-    Very simple list delegator that repr() dumps (presumed) tuple
-    elements of the form `tuple[str, Any, Any]` in a nice
-    multi-line readable form for analyzing `Struct` diffs.
-
-    '''
-    def __repr__(self) -> str:
-        if not len(self):
-            return super().__repr__()
-
-        # format by displaying item pair's ``repr()`` on multiple,
-        # indented lines such that they are more easily visually
-        # comparable when printed to console when printed to
-        # console.
-        repstr: str = '[\n'
-        for k, left, right in self:
-            repstr += (
-                f'({k},\n'
-                f'\t{repr(left)},\n'
-                f'\t{repr(right)},\n'
-                ')\n'
-            )
-        repstr += ']\n'
-        return repstr
-
-
-class Struct(
-    _Struct,
-
-    # https://jcristharif.com/msgspec/structs.html#tagged-unions
-    # tag='pikerstruct',
-    # tag=True,
-):
-    '''
-    A "human friendlier" (aka repl buddy) struct subtype.
-
-    '''
-    def _sin_props(self) -> Iterator[
-        tuple[
-            structs.FieldIinfo,
-            str,
-            Any,
-        ]
-    ]:
-        '''
-        Iterate over all non-@property fields of this struct.
-
-        '''
-        fi: structs.FieldInfo
-        for fi in structs.fields(self):
-            key: str = fi.name
-            val: Any = getattr(self, key)
-            yield fi, key, val
-
-    def to_dict(
-        self,
-        include_non_members: bool = True,
-
-    ) -> dict:
-        '''
-        Like it sounds.. direct delegation to:
-        https://jcristharif.com/msgspec/api.html#msgspec.structs.asdict
-
-        BUT, by default we pop all non-member (aka not defined as
-        struct fields) fields by default.
-
-        '''
-        asdict: dict = structs.asdict(self)
-        if include_non_members:
-            return asdict
-
-        # only return a dict of the struct members
-        # which were provided as input, NOT anything
-        # added as type-defined `@property` methods!
-        sin_props: dict = {}
-        fi: structs.FieldInfo
-        for fi, k, v in self._sin_props():
-            sin_props[k] = asdict[k]
-
-        return sin_props
-
-    def pformat(
-        self,
-        field_indent: int = 2,
-        indent: int = 0,
-
-    ) -> str:
-        '''
-        Recursion-safe `pprint.pformat()` style formatting of
-        a `msgspec.Struct` for sane reading by a human using a REPL.
-
-        '''
-        # global whitespace indent
-        ws: str = ' '*indent
-
-        # field whitespace indent
-        field_ws: str = ' '*(field_indent + indent)
-
-        # qtn: str = ws + self.__class__.__qualname__
-        qtn: str = self.__class__.__qualname__
-
-        obj_str: str = ''  # accumulator
-        fi: structs.FieldInfo
-        k: str
-        v: Any
-        for fi, k, v in self._sin_props():
-
-            # TODO: how can we prefer `Literal['option1',  'option2,
-            # ..]` over .__name__ == `Literal` but still get only the
-            # latter for simple types like `str | int | None` etc..?
-            ft: type = fi.type
-            typ_name: str = getattr(ft, '__name__', str(ft))
-
-            # recurse to get sub-struct's `.pformat()` output Bo
-            if isinstance(v, Struct):
-                val_str: str =  v.pformat(
-                    indent=field_indent + indent,
-                    field_indent=indent + field_indent,
-                )
-
-            else:  # the `pprint` recursion-safe format:
-                # https://docs.python.org/3.11/library/pprint.html#pprint.saferepr
-                val_str: str = saferepr(v)
-
-            obj_str += (field_ws + f'{k}: {typ_name} = {val_str},\n')
-
-        return (
-            f'{qtn}(\n'
-            f'{obj_str}'
-            f'{ws})'
-        )
-
-    # TODO: use a pprint.PrettyPrinter instance around ONLY rendering
-    # inside a known tty?
-    # def __repr__(self) -> str:
-    #     ...
-
-    # __str__ = __repr__ = pformat
-    __repr__ = pformat
-
-    def copy(
-        self,
-        update: dict | None = None,
-
-    ) -> Struct:
-        '''
-        Validate-typecast all self defined fields, return a copy of
-        us with all such fields.
-
-        NOTE: This is kinda like the default behaviour in
-        `pydantic.BaseModel` except a copy of the object is
-        returned making it compat with `frozen=True`.
-
-        '''
-        if update:
-            for k, v in update.items():
-                setattr(self, k, v)
-
-        # NOTE: roundtrip serialize to validate
-        # - enode to msgpack binary format,
-        # - decode that back to a struct.
-        return msgpack.Decoder(type=type(self)).decode(
-            msgpack.Encoder().encode(self)
-        )
-
-    def typecast(
-        self,
-
-        # TODO: allow only casting a named subset?
-        # fields: set[str] | None = None,
-
-    ) -> None:
-        '''
-        Cast all fields using their declared type annotations
-        (kinda like what `pydantic` does by default).
-
-        NOTE: this of course won't work on frozen types, use
-        ``.copy()`` above in such cases.
-
-        '''
-        # https://jcristharif.com/msgspec/api.html#msgspec.structs.fields
-        fi: structs.FieldInfo
-        for fi in structs.fields(self):
-            setattr(
-                self,
-                fi.name,
-                fi.type(getattr(self, fi.name)),
-            )
-
-    def __sub__(
-        self,
-        other: Struct,
-
-    ) -> DiffDump[tuple[str, Any, Any]]:
-        '''
-        Compare fields/items key-wise and return a ``DiffDump``
-        for easy visual REPL comparison B)
-
-        '''
-        diffs: DiffDump[tuple[str, Any, Any]] = DiffDump()
-        for fi in structs.fields(self):
-            attr_name: str = fi.name
-            ours: Any = getattr(self, attr_name)
-            theirs: Any = getattr(other, attr_name)
-            if ours != theirs:
-                diffs.append((
-                    attr_name,
-                    ours,
-                    theirs,
-                ))
-
-        return diffs

pyproject.toml

@@ -15,8 +15,8 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 [build-system]
-requires = ["poetry-core"]
+requires = ["hatchling"]
-build-backend = "poetry.core.masonry.api"
+build-backend = "hatchling.build"
 
 # ------ - ------
 
@@ -34,121 +34,114 @@ ignore = []
 
 # ------ - ------
 
-[tool.poetry]
-name = "piker"
-version = "0.1.0.alpha0.dev0"
-description = "trading gear for hackers"
-authors = ["Tyler Goodlet <goodboy_foss@protonmail.com>"]
-license = "AGPLv3"
-readme = "README.rst"
-
-# ------ - ------
-
-[tool.poetry.dependencies]
-async-generator = "^1.10"
-attrs = "^23.1.0"
-bidict = "^0.22.1"
-colorama = "^0.4.6"
-colorlog = "^6.7.0"
-cython = "^3.0.0"
-greenback = "^1.1.1"
-ib-insync = "^0.9.86"
-msgspec = "^0.18.0"
-numba = "^0.59.0"
-numpy = "^1.25"
-polars = "^0.18.13"
-pygments = "^2.16.1"
-python = ">=3.11, <3.13"
-rich = "^13.5.2"
-# setuptools = "^68.0.0"
-tomli = "^2.0.1"
-tomli-w = "^1.0.0"
-trio-util = "^0.7.0"
-trio-websocket = "^0.10.3"
-typer = "^0.9.0"
-rapidfuzz = "^3.5.2"
-pdbp = "^1.5.0"
-trio = "^0.24"
-pendulum = "^3.0.0"
-httpx = "^0.27.0"
-
-[tool.poetry.dependencies.tractor]
-develop = true
-git = 'https://github.com/goodboy/tractor.git'
-branch = 'asyncio_debugger_support'
-# path = "../tractor"
-
-[tool.poetry.dependencies.asyncvnc]
-git = 'https://github.com/pikers/asyncvnc.git'
-branch = 'main'
-
-[tool.poetry.dependencies.tomlkit]
-develop = true
-git = 'https://github.com/pikers/tomlkit.git'
-branch = 'piker_pin'
-# path = "../tomlkit/"
-
-[tool.poetry.group.uis]
-optional = true
-[tool.poetry.group.uis.dependencies]
-# https://python-poetry.org/docs/managing-dependencies/#dependency-groups
-# TODO: make sure the levenshtein shit compiles on nix..
-# rapidfuzz = {extras = ["speedup"], version = "^0.18.0"}
-rapidfuzz = "^3.2.0"
-qdarkstyle = ">=3.0.2"
-pyqtgraph = { git = 'https://github.com/pikers/pyqtgraph.git' }
-
-# ------ - ------
-pyqt6 = "^6.7.0"
-
-[tool.poetry.group.dev]
-optional = true
-[tool.poetry.group.dev.dependencies]
-# testing / CI
-pytest = "^6.0.0"
-elasticsearch = "^8.9.0"
-xonsh = "^0.14.2"
-prompt-toolkit = "3.0.40"
-
-# console ehancements and eventually remote debugging
-# extras/helpers.
-# TODO: add a toolset that makes debugging a `pikerd` service
-# (tree) easy to hack on directly using more or less the local env:
-# - xonsh + xxh
-# - rsyscall + pdbp
-# - actor runtime control console like BEAM/OTP
-
-# ------ - ------
-
-# TODO: add an `--only daemon` group for running non-ui / pikerd
-# service tree in distributed mode B)
-# https://python-poetry.org/docs/managing-dependencies/#installing-group-dependencies
-# [tool.poetry.group.daemon.dependencies]
-
-[tool.poetry.scripts]
-piker = 'piker.cli:cli'
-pikerd = 'piker.cli:pikerd'
-ledger = 'piker.accounting.cli:ledger'
-
-
 [project]
-keywords=[
+name = "piker"
-  "async",
+version = "0.1.0a0dev0"
-  "trading",
+description = "trading gear for hackers"
-  "finance",
+authors = [{ name = "Tyler Goodlet", email = "goodboy_foss@protonmail.com" }]
-  "quant",
+requires-python = ">=3.12, <3.13"
-  "charting",
+license = "AGPL-3.0-or-later"
+readme = "README.rst"
+keywords = [
+    "async",
+    "trading",
+    "finance",
+    "quant",
+    "charting",
 ]
-classifiers=[
+classifiers = [
-  'Development Status :: 3 - Alpha',
+    "Development Status :: 3 - Alpha",
-  "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
+    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
-  'Operating System :: POSIX :: Linux',
+    "Operating System :: POSIX :: Linux",
-  "Programming Language :: Python :: Implementation :: CPython",
+    "Programming Language :: Python :: Implementation :: CPython",
-  "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3 :: Only",
-  "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.11",
-  "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.12",
-  'Intended Audience :: Financial and Insurance Industry',
+    "Intended Audience :: Financial and Insurance Industry",
-  'Intended Audience :: Science/Research',
+    "Intended Audience :: Science/Research",
-  'Intended Audience :: Developers',
+    "Intended Audience :: Developers",
-  'Intended Audience :: Education',
+    "Intended Audience :: Education",
 ]
+dependencies = [
+    "async-generator >=1.10, <2.0.0",
+    "attrs >=23.1.0, <24.0.0",
+    "bidict >=0.22.1, <0.23.0",
+    "colorama >=0.4.6, <0.5.0",
+    "colorlog >=6.7.0, <7.0.0",
+    "ib-insync >=0.9.86, <0.10.0",
+    "numba >=0.59.0, <0.60.0",
+    "numpy >=1.25, <2.0",
+    "polars >=0.18.13, <0.19.0",
+    "pygments >=2.16.1, <3.0.0",
+    "rich >=13.5.2, <14.0.0",
+    "tomli >=2.0.1, <3.0.0",
+    "tomli-w >=1.0.0, <2.0.0",
+    "trio-util >=0.7.0, <0.8.0",
+    "trio-websocket >=0.10.3, <0.11.0",
+    "typer >=0.9.0, <1.0.0",
+    "rapidfuzz >=3.5.2, <4.0.0",
+    "pdbp >=1.5.0, <2.0.0",
+    "trio >=0.24, <0.25",
+    "pendulum >=3.0.0, <4.0.0",
+    "httpx >=0.27.0, <0.28.0",
+    "cryptofeed >=2.4.0, <3.0.0",
+    "pyarrow >=17.0.0, <18.0.0",
+    "websockets ==12.0",
+    "msgspec",
+    "tractor",
+    "asyncvnc",
+    "tomlkit",
+]
+
+[project.optional-dependencies]
+uis = [
+    # https://docs.astral.sh/uv/concepts/projects/dependencies/#optional-dependencies
+    # TODO: make sure the levenshtein shit compiles on nix..
+    # rapidfuzz = {extras = ["speedup"], version = "^0.18.0"}
+    "rapidfuzz >=3.2.0, <4.0.0",
+    "qdarkstyle >=3.0.2, <4.0.0",
+    "pyqt6 >=6.7.0, <7.0.0",
+    "pyqtgraph",
+
+    # ------ - ------
+
+    # TODO: add an `--only daemon` group for running non-ui / pikerd
+    # service tree in distributed mode B)
+    # https://docs.astral.sh/uv/concepts/projects/dependencies/#optional-dependencies
+    # [project.optional-dependencies]
+]
+
+[dependency-groups]
+dev = [
+    "pytest >=6.0.0, <7.0.0",
+    "elasticsearch >=8.9.0, <9.0.0",
+    "xonsh >=0.14.2, <0.15.0",
+    "prompt-toolkit ==3.0.40",
+    "cython >=3.0.0, <4.0.0",
+    "greenback >=1.1.1, <2.0.0",
+    # console ehancements and eventually remote debugging
+    # extras/helpers.
+    # TODO: add a toolset that makes debugging a `pikerd` service
+    # (tree) easy to hack on directly using more or less the local env:
+    # - xonsh + xxh
+    # - rsyscall + pdbp
+    # - actor runtime control console like BEAM/OTP
+]
+
+[project.scripts]
+piker = "piker.cli:cli"
+pikerd = "piker.cli:pikerd"
+ledger = "piker.accounting.cli:ledger"
+
+[tool.hatch.build.targets.sdist]
+include = ["piker"]
+
+[tool.hatch.build.targets.wheel]
+include = ["piker"]
+
+[tool.uv.sources]
+pyqtgraph = { git = "https://github.com/pikers/pyqtgraph.git" }
+asyncvnc = { git = "https://github.com/pikers/asyncvnc.git", branch = "main" }
+tomlkit = { git = "https://github.com/pikers/tomlkit.git", branch ="piker_pin" }
+msgspec = { git = "https://github.com/jcrist/msgspec.git" }
+tractor = { path = "../tractor", editable = true }